x86_64: add config options to optimize for newer AMD processors
PATCH 1/1 x86_64: add config options to optimize for newer AMD processors
From: Austin S. Hemmelgarn <[email protected]>
This patch adds Kconfig options to allow optimization for AMD family 10h, AMD Bulldozer, and AMD Piledriver derived CPU's in version 3.12-rc2. These provide noticeable improvements over the K8 config option, and allow the kernel to take full advantage of AMD specific instruction set extensions, such as ABM, LZCNT, and POPCNT.
---
arch/x86/Kconfig.cpu 2013-09-28 21:31:35.782173832 -0400 renamed from arch/x86/Kconfig.cpu 2013-09-23 18:41:09.000000000 -0400 +54 -3
@@ -155,9 +155,9 @@ config MK7 flags to GCC. config MK8 - bool "Opteron/Athlon64/Hammer/K8" + bool "Old Opteron/Athlon64/Hammer/K8" ---help--- - Select this for an AMD Opteron or Athlon64 Hammer-family processor. + Select this for an AMD old 64-bit Opteron or Athlon64 Hammer-family processor. Enables use of some extended instructions, and passes appropriate optimization flags to GCC.
@@ -269,6 +269,57 @@ config MATOM accordingly optimized code. Use a recent GCC with specific Atom support in order to fully benefit from selecting this option. +config MAMDFAM10 + bool "AMD Family 10h (Athlon II, Phenom II, and Opteron)" + depends on X86_64 + ---help--- + + Select this for AMD Family 10h processors. + This includes Athlon II, Phenom II, early third-generation + Opterons, and a number of other Socket AM2, AM2+, AM3, and + Socket F processors. CPU's in this series show cpu family + 16 in /proc/cpuinfo. Note that a kernel compiled with this + option may not run correctly on non-AMD processors. + +config MBULLDOZER + bool "AMD Bulldozer (FX and Opteron)" + depends on X86_64 + ---help--- + + Select this for AMD Buldozer microarchitecture processors. + This includes the following CPUs: + FX-41x0 + FX-61x0 + FX-6200 + FX-81x0 + Opteron 32xx + Opteron 42xx + Opteron 62xx + + Note that a kernel compiled with this option may not run + correctly on non-Bulldozer derived processors, if it runs at all. + +config MPILEDRIVER + bool "AMD Piledriver (FX, APU, and Opteron)" + depends on X86_64 + ---help--- + + Select this for AMD Piledriver microarchitecture processors. + This includes the Following CPUs: + 'Trinity' APUs + 'Richland' APUs + FX-43xx + FX-63xx + FX-83xx + FX-9370 + FX-9590 + Opteron 33xx + Opteron 43xx + Opteron 63xx + + Note that a kernel compiled with this option may not run + correctly on non-Piledriver derived processors, if it runs at all. + config GENERIC_CPU bool "Generic-x86-64" depends on X86_64
@@ -300,7 +351,7 @@ config X86_INTERNODE_CACHE_SHIFT config X86_L1_CACHE_SHIFT int default "7" if MPENTIUM4 || MPSC - default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MATOM || MVIAC7 || X86_GENERIC || GENERIC_CPU + default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MATOM || MVIAC7 || MAMDFAM10 || MBULLDOZER || MPILEDRIVER || X86_GENERIC || GENERIC_CPU default "4" if MELAN || M486 || MGEODEGX1 default "5" if MWINCHIP3D || MWINCHIPC6 || MCRUSOE || MEFFICEON || MCYRIXIII || MK6 || MPENTIUMIII || MPENTIUMII || M686 || M586MMX || M586TSC || M586 || MVIAC3_2 || MGEODE_LX
arch/x86/Makefile 2013-09-28 21:20:41.869697978 -0400 renamed from arch/x86/Makefile 2013-09-23 18:41:09.000000000 -0400 +6 -0
@@ -68,6 +68,12 @@ else $(call cc-option,-march=core2,$(call cc-option,-mtune=generic)) cflags-$(CONFIG_MATOM) += $(call cc-option,-march=atom) \ $(call cc-option,-mtune=atom,$(call cc-option,-mtune=generic)) + cflags-$(CONFIG_AMDFAM10) += $(call cc-option,-march=amdfam10) \ + $(call cc-option,-mtune=amdfam10,$(call cc-option,-mtune=generic)) + cflags-$(CONFIG_MBULLCOZER) += $(call cc-option,-march=bdver1) \ + $(call cc-option,-mtune=bdver1,$(call cc-option,-mtune=generic)) + cflags-$(CONFIG_MPILEDRIVER) += $(call cc-option,-march=bdver2) \ + $(call cc-option,-mtune=bdver2,$(call cc-option,-mtune=generic)) cflags-$(CONFIG_GENERIC_CPU) += $(call cc-option,-mtune=generic) KBUILD_CFLAGS += $(cflags-y)
Borislav Petkov
maintainerQuoted 7 lines · Austin S Hemmelgarn
From: Austin S. Hemmelgarn <[email protected]>
This patch adds
Kconfigoptions to allow optimization for AMD family 10h, AMD Bulldozer, and AMD Piledriver derived CPU's in version 3.12-rc2. These provide noticeable improvements over the K8 config option, and allow the kernel to take full advantage of AMD specific instruction set extensions, such as ABM, LZCNT, and POPCNT.
A patch like that keeps popping up every couple of months. Please show us those noticeable improvements because the guy last time failed to do so.
And also, keep in mind that distro kernels are built with CONFIG_GENERIC_CPU so CPU-specific kernels are used only by the small percentage of people who actually do build their own kernels.
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 15 lines · Borislav Petkov
Quoted 8 lines · Austin S Hemmelgarn
From: Austin S. Hemmelgarn <[email protected]>
This patch adds
Kconfigoptions to allow optimization for AMD family 10h, AMD Bulldozer, and AMD Piledriver derived CPU's in version 3.12-rc2. These provide noticeable improvements over the K8 config option, and allow the kernel to take full advantage of AMD specific instruction set extensions, such as ABM, LZCNT, and POPCNT.A patch like that keeps popping up every couple of months. Please show us those noticeable improvements because the guy last time failed to do so.
Quoted 7 lines
And also, keep in mind that distro kernels are built with
CONFIG_GENERIC_CPUso CPU-specific kernels are used only by the small percentage of people who actually do build their own kernels.Thanks.
While I understand that you want decisive proof that it provides an improvement, does it specifically matter if the option is unused by most people and doesn't result in a negative performance hit when used?
I will, however, do some intensive profiling over the next week and get back to you about exact improvements. Just in the past few days of using it, I have noticed much better response time in time sensitive user-space code (mostly emulators and multimedia apps).
Borislav Petkov
maintainerQuoted 4 lines · Austin S Hemmelgarn
While I understand that you want decisive proof that it provides an improvement, does it specifically matter if the option is unused by most people and doesn't result in a negative performance hit when used?
Just having the option for no good reason at all is a no-no.
Quoted 4 lines
I will, however, do some intensive profiling over the next week and get back to you about exact improvements. Just in the past few days of using it, I have noticed much better response time in time sensitive user-space code (mostly emulators and multimedia apps).
Yes, we need concrete benchmark numbers from a kernel built with MK8 or even GENERIC vs one built with those new options, most likely with the one for the box you'll be running the benchmarks on. And no microbenchmarks please.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 8 lines · Borislav Petkov
Quoted 4 lines · Austin S Hemmelgarn
While I understand that you want decisive proof that it provides an improvement, does it specifically matter if the option is unused by most people and doesn't result in a negative performance hit when used?
Just having the option for no good reason at all is a no-no.
I'm not saying that should just be included without substantiation, I simply mean that the reason to include it (as far as I am concerned) is that it doesn't break anything and provides something useful that isn't in the kernel already.
Despite this I am still happy to benchmark it to provide some more concrete proof.
Borislav Petkov
maintainerQuoted 4 lines · Austin S Hemmelgarn
I'm not saying that should just be included without substantiation, I simply mean that the reason to include it (as far as I am concerned) is that it doesn't break anything and provides something useful that isn't in the kernel already.
If it doesn't bring any performance improvement - and I don't want to rain on your parade but I think it won't, at least not enough to warrant a serious look - there's absolutely no reason to add it.
But hey, I'm always open to surprises! So surprise me!
:-)
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 6 lines · Borislav Petkov
Quoted 4 lines · Austin S Hemmelgarn
I'm not saying that should just be included without substantiation, I simply mean that the reason to include it (as far as I am concerned) is that it doesn't break anything and provides something useful that isn't in the kernel already.
Quoted 8 lines
If it doesn't bring any performance improvement - and I don't want to rain on your parade but I think it won't, at least not enough to warrant a serious look - there's absolutely no reason to add it.
But hey, I'm always open to surprises! So surprise me!
:-)
Sorry it took me so long to get back to you about this.
After doing some intensive testing, it appears that there is some improvement, but it is mostly in the scheduling code and syscalls. Most sysbench benchmarks only show improvement when running with more threads than processor cores; however, build jobs appear to be much improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes 22 minutes and 57 seconds on a kernel with CONFIG_MK8, 21 minutes and 35 seconds on a kernel with CONFIG_GENERIC, and 19 minutes and 11 seconds on a kernel with CONFIG_PILEDRIVER. I see similar results for a build of GCC 4.7 (45m1s, 41m39s, and 38m56s(. I would love to test it on my Athlon II system, but I can't afford to have any appreciable load on that system because I'm using it as a router/firewall, based on a look at the low level stuff though, I believe that it would still be at least a bit better with CONFIG_MAMDFAM10 than with CONFIG_GENERIC, and certainly better than CONFIG_MK8 (there are some big differences in the execution pipeline between K8 and family 10h).
I don't know about you, but that sure seems to be a worthwhile performance increase to me.
Quoted 5 lines · Austin S Hemmelgarn
improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes
22 minutes and 57 seconds on a kernel with
CONFIG_MK8, 21 minutes and 35 seconds on a kernel withCONFIG_GENERIC, and 19 minutes and 11 seconds on a kernel withCONFIG_PILEDRIVER.
That's certainly noticeable. Surprisingly so. What makes MK8 so bad in particular, I wonder?
Just out of interest, have you done any profiles on the kernel cost here to see what it is that makes such a big difference. Because normally on a kernel build, I see most of the overhead in path lookup. But that's only true for otherwise optimized builds that don't have system call auditing etc debugging that spreads the costs out over everything..
Linus
Borislav Petkov
maintainerQuoted 11 lines · Linus Torvalds
Quoted 5 lines · Austin S Hemmelgarn
improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes
22 minutes and 57 seconds on a kernel with
CONFIG_MK8, 21 minutes and 35 seconds on a kernel withCONFIG_GENERIC, and 19 minutes and 11 seconds on a kernel withCONFIG_PILEDRIVER.That's certainly noticeable. Surprisingly so. What makes MK8 so bad in particular, I wonder?
Quoted 6 lines
Just out of interest, have you done any profiles on the kernel cost here to see what it is that makes such a big difference. Because normally on a kernel build, I see most of the overhead in path lookup. But that's only true for otherwise optimized builds that don't have system call auditing etc debugging that spreads the costs out over everything..
Yeah, I was having some doubts about the numbers above so I ran my own benchmarking, machine is a Piledriver box:
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0
and I don't really see any of those improvements above. Actually, -march=bdver2 is even slightly worse in comparison to mk8.
And the workload is of building a config specific to that machine but allmodconfig looks very similar, the numbers being simply higher.
$ zgrep MK8 /proc/config.gz CONFIG_MK8=y
/home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage
Performance counter stats for 'make -s -j64 bzImage' (10 runs):
1081808.628840 task-clock # 7.996 CPUs utilized ( +- 0.06% ) [100.00%]
1,203,753 context-switches # 0.001 M/sec ( +- 0.04% ) [100.00%]
48,748 cpu-migrations # 0.045 K/sec ( +- 0.59% ) [100.00%]
31,145,439 page-faults # 0.029 M/sec ( +- 0.00% )
3,836,736,801,500 cycles # 3.547 GHz ( +- 0.03% ) [100.00%]
957,386,966,493 stalled-cycles-frontend # 24.95% frontend cycles idle ( +- 0.06% ) [100.00%]
218,581,249,251 stalled-cycles-backend # 5.70% backend cycles idle ( +- 0.06% ) [100.00%]
2,466,632,641,972 instructions # 0.64 insns per cycle
# 0.39 stalled cycles per insn ( +- 0.00% ) [100.00%]
537,749,333,838 branches # 497.084 M/sec ( +- 0.00% ) [100.00%]
27,802,940,176 branch-misses # 5.17% of all branches ( +- 0.00% )135.292843025 seconds time elapsed ( +- 0.06% )
$ zgrep PILEDRIVER /proc/config.gz CONFIG_MPILEDRIVER=y
/home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage
Performance counter stats for 'make -s -j64 bzImage' (10 runs):
1085723.230470 task-clock # 7.996 CPUs utilized ( +- 0.10% ) [100.00%]
1,204,355 context-switches # 0.001 M/sec ( +- 0.10% ) [100.00%]
49,143 cpu-migrations # 0.045 K/sec ( +- 0.76% ) [100.00%]
31,196,575 page-faults # 0.029 M/sec ( +- 0.00% )
3,851,255,065,133 cycles # 3.547 GHz ( +- 0.02% ) [100.00%]
958,840,197,117 stalled-cycles-frontend # 24.90% frontend cycles idle ( +- 0.09% ) [100.00%]
220,260,399,411 stalled-cycles-backend # 5.72% backend cycles idle ( +- 0.04% ) [100.00%]
2,466,701,295,156 instructions # 0.64 insns per cycle
# 0.39 stalled cycles per insn ( +- 0.00% ) [100.00%]
537,992,040,195 branches # 495.515 M/sec ( +- 0.00% ) [100.00%]
27,860,290,286 branch-misses # 5.18% of all branches ( +- 0.00% )135.784111961 seconds time elapsed ( +- 0.10% )
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 74 lines · Linus Torvalds
Quoted 11 lines
Quoted 5 lines · Austin S Hemmelgarn
improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes
22 minutes and 57 seconds on a kernel with
CONFIG_MK8, 21 minutes and 35 seconds on a kernel withCONFIG_GENERIC, and 19 minutes and 11 seconds on a kernel withCONFIG_PILEDRIVER.That's certainly noticeable. Surprisingly so. What makes MK8 so bad in particular, I wonder?
Quoted 6 lines
Just out of interest, have you done any profiles on the kernel cost here to see what it is that makes such a big difference. Because normally on a kernel build, I see most of the overhead in path lookup. But that's only true for otherwise optimized builds that don't have system call auditing etc debugging that spreads the costs out over everything..
Yeah, I was having some doubts about the numbers above so I ran my own benchmarking, machine is a Piledriver box:
vendor_id: AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0and I don't really see any of those improvements above. Actually, -march=bdver2 is even slightly worse in comparison to mk8.
And the workload is of building a config specific to that machine but allmodconfig looks very similar, the numbers being simply higher.
$ zgrep MK8 /proc/config.gz CONFIG_MK8=y
/home/boris/bin/perfstat --repeat 10 -a --sync --pre/home/boris/kernel/pre-build-kernel.shmake -s -j64 bzImagePerformance counter stats for 'make -s -j64 bzImage' (10 runs):
1081808.628840 task-clock # 7.996 CPUs utilized ( +- 0.06% ) [100.00%] 1,203,753 context-switches # 0.001 M/sec ( +- 0.04% ) [100.00%] 48,748 cpu-migrations # 0.045 K/sec ( +- 0.59% ) [100.00%] 31,145,439 page-faults # 0.029 M/sec ( +- 0.00% ) 3,836,736,801,500 cycles # 3.547 GHz ( +- 0.03% ) [100.00%] 957,386,966,493 stalled-cycles-frontend # 24.95% frontend cycles idle ( +- 0.06% ) [100.00%] 218,581,249,251 stalled-cycles-backend # 5.70% backend cycles idle ( +- 0.06% ) [100.00%] 2,466,632,641,972 instructions # 0.64 insns per cycle # 0.39 stalled cycles per insn ( +- 0.00% ) [100.00%] 537,749,333,838 branches # 497.084 M/sec ( +- 0.00% ) [100.00%] 27,802,940,176 branch-misses # 5.17% of all branches ( +- 0.00% )135.292843025 seconds time elapsed ( +- 0.06% )
$ zgrep PILEDRIVER /proc/config.gz CONFIG_MPILEDRIVER=y
/home/boris/bin/perfstat --repeat 10 -a --sync --pre/home/boris/kernel/pre-build-kernel.shmake -s -j64 bzImagePerformance counter stats for 'make -s -j64 bzImage' (10 runs):
1085723.230470 task-clock # 7.996 CPUs utilized ( +- 0.10% ) [100.00%] 1,204,355 context-switches # 0.001 M/sec ( +- 0.10% ) [100.00%] 49,143 cpu-migrations # 0.045 K/sec ( +- 0.76% ) [100.00%] 31,196,575 page-faults # 0.029 M/sec ( +- 0.00% ) 3,851,255,065,133 cycles # 3.547 GHz ( +- 0.02% ) [100.00%] 958,840,197,117 stalled-cycles-frontend # 24.90% frontend cycles idle ( +- 0.09% ) [100.00%] 220,260,399,411 stalled-cycles-backend # 5.72% backend cycles idle ( +- 0.04% ) [100.00%] 2,466,701,295,156 instructions # 0.64 insns per cycle # 0.39 stalled cycles per insn ( +- 0.00% ) [100.00%] 537,992,040,195 branches # 495.515 M/sec ( +- 0.00% ) [100.00%] 27,860,290,286 branch-misses # 5.18% of all branches ( +- 0.00% )
Quoted 2 lines
135.784111961 seconds time elapsed ( +- 0.10% )
Part of the difference between our results may be that I have my entire userspace built with -mtune=bdver2, so less of the time is spent in userspace. Also, the part about using many more threads than cpu cores was with regards to sysbench, not the kernel build, for that I just used 8 jobs in make.
With regards to the differences shown above relative to CONFIG_MK8, that does actually make sense; with CONFIG_MK8, gcc makes very minimal use of extension instructions (afaik, only MMX, SSE, and 3Dnow!), this improves performance slightly on bulldozer derivatives because there are only half as many SSE and FP units as CPU cores (and the scheduler isn't as smart as it could be with regards to that, but that is something for another patch as far as I am concerned).
Borislav Petkov
maintainerQuoted 3 lines · Austin S Hemmelgarn
Part of the difference between our results may be that I have my entire userspace built with -mtune=bdver2, so less of the time is spent in userspace.
That still cannot explain the huge difference between building on a mk8 vs bdver2 kernel. Provided your userspace is the same and only the kernels are different, I don't see how that happens.
Btw, you could run my workload on your machine, once on an mk8 kernel and once on a bdver2 one.
The contents of the pre-build-kernel.sh script are:
--
#!/bin/bash
make -s clean
echo 3 > /proc/sys/vm/drop_caches
--
Of course, you need to run this as root.
Quoted 3 lines
Also, the part about using many more threads than cpu cores was with regards to sysbench, not the kernel build, for that I just used 8 jobs in make.
Overcommitting gcc threads does not bring a really a big difference with a kernel build.
Quoted 5 lines
With regards to the differences shown above relative to
CONFIG_MK8, that does actually make sense; withCONFIG_MK8, gcc makes very minimal use of extension instructions (afaik, only MMX, SSE, and 3Dnow!), this improves performance slightly on bulldozer derivatives because there are only half as many SSE and FP units as CPU cores (and the
yes, the FPU is shared between the cores of a compute unit on BD.
Quoted 2 lines
scheduler isn't as smart as it could be with regards to that, but that is something for another patch as far as I am concerned).
Again, with unchanging userspace and only differing kernels, you shouldn't be seeing such a big difference. Besides, the kernel itself makes very little, if at all, use of FPU instructions.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--