Skip to contentSkip to conversation

x86_64: add config options to optimize for newer AMD processors

PATCH In review Austin S Hemmelgarn 3 participants 11 messages Sep 29, 2013 – Oct 3, 2013 1 message not in the archive x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86_64: add config options to optimize for newer AMD processors

From: Austin S. Hemmelgarn <[email protected]>

This patch adds Kconfig options to allow optimization for AMD family 10h, AMD Bulldozer, and AMD Piledriver derived CPU's in version 3.12-rc2. These provide noticeable improvements over the K8 config option, and allow the kernel to take full advantage of AMD specific instruction set extensions, such as ABM, LZCNT, and POPCNT.

---

arch/x86/Kconfig.cpu 2013-09-28 21:31:35.782173832 -0400 renamed from arch/x86/Kconfig.cpu 2013-09-23 18:41:09.000000000 -0400 +54 -3
diff -uprN linux/arch/x86/Kconfig.cpu linux-new/arch/x86/Kconfig.cpu
@@ -155,9 +155,9 @@ config MK7
          flags to GCC.
 
 config MK8
-       bool "Opteron/Athlon64/Hammer/K8"
+       bool "Old Opteron/Athlon64/Hammer/K8"
        ---help---
-         Select this for an AMD Opteron or Athlon64 Hammer-family processor.
+         Select this for an AMD old 64-bit Opteron or Athlon64 Hammer-family processor.
          Enables use of some extended instructions, and passes appropriate
          optimization flags to GCC.
 
@@ -269,6 +269,57 @@ config MATOM
          accordingly optimized code. Use a recent GCC with specific Atom
          support in order to fully benefit from selecting this option.
 
+config MAMDFAM10
+       bool "AMD Family 10h (Athlon II, Phenom II, and Opteron)"
+       depends on X86_64
+       ---help---
+
+         Select this for AMD Family 10h processors.
+         This includes Athlon II, Phenom II, early third-generation
+         Opterons, and a number of other Socket AM2, AM2+, AM3, and
+         Socket F processors.  CPU's in this series show cpu family
+         16 in /proc/cpuinfo.  Note that a kernel compiled with this
+         option may not run correctly on non-AMD processors.
+
+config MBULLDOZER
+       bool "AMD Bulldozer (FX and Opteron)"
+       depends on X86_64
+       ---help---
+
+         Select this for AMD Buldozer microarchitecture processors.
+         This includes the following CPUs:
+           FX-41x0
+           FX-61x0
+           FX-6200
+           FX-81x0
+           Opteron 32xx
+           Opteron 42xx
+           Opteron 62xx
+
+         Note that a kernel compiled with this option may not run
+         correctly on non-Bulldozer derived processors, if it runs at all.
+
+config MPILEDRIVER
+       bool "AMD Piledriver (FX, APU, and Opteron)"
+       depends on X86_64
+       ---help---
+
+         Select this for AMD Piledriver microarchitecture processors.
+         This includes the Following CPUs:
+           'Trinity' APUs
+           'Richland' APUs
+           FX-43xx
+           FX-63xx
+           FX-83xx
+           FX-9370
+           FX-9590
+           Opteron 33xx
+           Opteron 43xx
+           Opteron 63xx
+
+         Note that a kernel compiled with this option may not run
+         correctly on non-Piledriver derived processors, if it runs at all.
+
 config GENERIC_CPU
        bool "Generic-x86-64"
        depends on X86_64
@@ -300,7 +351,7 @@ config X86_INTERNODE_CACHE_SHIFT
 config X86_L1_CACHE_SHIFT
        int
        default "7" if MPENTIUM4 || MPSC
-       default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MATOM || MVIAC7 || X86_GENERIC || GENERIC_CPU
+       default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MATOM || MVIAC7 || MAMDFAM10 || MBULLDOZER || MPILEDRIVER || X86_GENERIC || GENERIC_CPU
        default "4" if MELAN || M486 || MGEODEGX1
        default "5" if MWINCHIP3D || MWINCHIPC6 || MCRUSOE || MEFFICEON || MCYRIXIII || MK6 || MPENTIUMIII || MPENTIUMII || M686 || M586MMX || M586TSC || M586 || MVIAC3_2 || MGEODE_LX
 
arch/x86/Makefile 2013-09-28 21:20:41.869697978 -0400 renamed from arch/x86/Makefile 2013-09-23 18:41:09.000000000 -0400 +6 -0
diff -uprN linux/arch/x86/Makefile linux-new/arch/x86/Makefile
@@ -68,6 +68,12 @@ else
                 $(call cc-option,-march=core2,$(call cc-option,-mtune=generic))
        cflags-$(CONFIG_MATOM) += $(call cc-option,-march=atom) \
                $(call cc-option,-mtune=atom,$(call cc-option,-mtune=generic))
+        cflags-$(CONFIG_AMDFAM10) += $(call cc-option,-march=amdfam10) \
+                $(call cc-option,-mtune=amdfam10,$(call cc-option,-mtune=generic))
+        cflags-$(CONFIG_MBULLCOZER) += $(call cc-option,-march=bdver1) \
+                $(call cc-option,-mtune=bdver1,$(call cc-option,-mtune=generic))
+        cflags-$(CONFIG_MPILEDRIVER) += $(call cc-option,-march=bdver2) \
+                $(call cc-option,-mtune=bdver2,$(call cc-option,-mtune=generic))
         cflags-$(CONFIG_GENERIC_CPU) += $(call cc-option,-mtune=generic)
         KBUILD_CFLAGS += $(cflags-y)
 
maintainer
#1 lore raw
Quoted 7 lines · Austin S Hemmelgarn

From: Austin S. Hemmelgarn <[email protected]>

This patch adds Kconfig options to allow optimization for AMD family 10h, AMD Bulldozer, and AMD Piledriver derived CPU's in version 3.12-rc2. These provide noticeable improvements over the K8 config option, and allow the kernel to take full advantage of AMD specific instruction set extensions, such as ABM, LZCNT, and POPCNT.

A patch like that keeps popping up every couple of months. Please show us those noticeable improvements because the guy last time failed to do so.

And also, keep in mind that distro kernels are built with CONFIG_GENERIC_CPU so CPU-specific kernels are used only by the small percentage of people who actually do build their own kernels.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#2 lore raw
Quoted 15 lines · Borislav Petkov
Quoted 8 lines · Austin S Hemmelgarn

From: Austin S. Hemmelgarn <[email protected]>

This patch adds Kconfig options to allow optimization for AMD family 10h, AMD Bulldozer, and AMD Piledriver derived CPU's in version 3.12-rc2. These provide noticeable improvements over the K8 config option, and allow the kernel to take full advantage of AMD specific instruction set extensions, such as ABM, LZCNT, and POPCNT.

A patch like that keeps popping up every couple of months. Please show us those noticeable improvements because the guy last time failed to do so.

Quoted 7 lines

And also, keep in mind that distro kernels are built with CONFIG_GENERIC_CPU so CPU-specific kernels are used only by the small percentage of people who actually do build their own kernels.

Thanks.

While I understand that you want decisive proof that it provides an improvement, does it specifically matter if the option is unused by most people and doesn't result in a negative performance hit when used?

I will, however, do some intensive profiling over the next week and get back to you about exact improvements. Just in the past few days of using it, I have noticed much better response time in time sensitive user-space code (mostly emulators and multimedia apps).

maintainer
#3 lore raw
Quoted 4 lines · Austin S Hemmelgarn

While I understand that you want decisive proof that it provides an improvement, does it specifically matter if the option is unused by most people and doesn't result in a negative performance hit when used?

Just having the option for no good reason at all is a no-no.

Quoted 4 lines

I will, however, do some intensive profiling over the next week and get back to you about exact improvements. Just in the past few days of using it, I have noticed much better response time in time sensitive user-space code (mostly emulators and multimedia apps).

Yes, we need concrete benchmark numbers from a kernel built with MK8 or even GENERIC vs one built with those new options, most likely with the one for the box you'll be running the benchmarks on. And no microbenchmarks please.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#4 lore raw
Quoted 8 lines · Borislav Petkov
Quoted 4 lines · Austin S Hemmelgarn

While I understand that you want decisive proof that it provides an improvement, does it specifically matter if the option is unused by most people and doesn't result in a negative performance hit when used?

Just having the option for no good reason at all is a no-no.

I'm not saying that should just be included without substantiation, I simply mean that the reason to include it (as far as I am concerned) is that it doesn't break anything and provides something useful that isn't in the kernel already.

Despite this I am still happy to benchmark it to provide some more concrete proof.

maintainer
#5 lore raw
Quoted 4 lines · Austin S Hemmelgarn

I'm not saying that should just be included without substantiation, I simply mean that the reason to include it (as far as I am concerned) is that it doesn't break anything and provides something useful that isn't in the kernel already.

If it doesn't bring any performance improvement - and I don't want to rain on your parade but I think it won't, at least not enough to warrant a serious look - there's absolutely no reason to add it.

But hey, I'm always open to surprises! So surprise me!

:-)

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#6 lore raw
Quoted 6 lines · Borislav Petkov
Quoted 4 lines · Austin S Hemmelgarn

I'm not saying that should just be included without substantiation, I simply mean that the reason to include it (as far as I am concerned) is that it doesn't break anything and provides something useful that isn't in the kernel already.

Quoted 8 lines

If it doesn't bring any performance improvement - and I don't want to rain on your parade but I think it won't, at least not enough to warrant a serious look - there's absolutely no reason to add it.

But hey, I'm always open to surprises! So surprise me!

:-)

Sorry it took me so long to get back to you about this.

After doing some intensive testing, it appears that there is some improvement, but it is mostly in the scheduling code and syscalls. Most sysbench benchmarks only show improvement when running with more threads than processor cores; however, build jobs appear to be much improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes 22 minutes and 57 seconds on a kernel with CONFIG_MK8, 21 minutes and 35 seconds on a kernel with CONFIG_GENERIC, and 19 minutes and 11 seconds on a kernel with CONFIG_PILEDRIVER. I see similar results for a build of GCC 4.7 (45m1s, 41m39s, and 38m56s(. I would love to test it on my Athlon II system, but I can't afford to have any appreciable load on that system because I'm using it as a router/firewall, based on a look at the low level stuff though, I believe that it would still be at least a bit better with CONFIG_MAMDFAM10 than with CONFIG_GENERIC, and certainly better than CONFIG_MK8 (there are some big differences in the execution pipeline between K8 and family 10h).

I don't know about you, but that sure seems to be a worthwhile performance increase to me.

#8 lore raw
In reply to a message not in the archive
Quoted 5 lines · Austin S Hemmelgarn

improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes

22 minutes and 57 seconds on a kernel with CONFIG_MK8, 21 minutes and 35 seconds on a kernel with CONFIG_GENERIC, and 19 minutes and 11 seconds on a kernel with CONFIG_PILEDRIVER.

That's certainly noticeable. Surprisingly so. What makes MK8 so bad in particular, I wonder?

Just out of interest, have you done any profiles on the kernel cost here to see what it is that makes such a big difference. Because normally on a kernel build, I see most of the overhead in path lookup. But that's only true for otherwise optimized builds that don't have system call auditing etc debugging that spreads the costs out over everything..

Linus

maintainer
#9 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Austin S Hemmelgarn

improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes

22 minutes and 57 seconds on a kernel with CONFIG_MK8, 21 minutes and 35 seconds on a kernel with CONFIG_GENERIC, and 19 minutes and 11 seconds on a kernel with CONFIG_PILEDRIVER.

That's certainly noticeable. Surprisingly so. What makes MK8 so bad in particular, I wonder?

Quoted 6 lines

Just out of interest, have you done any profiles on the kernel cost here to see what it is that makes such a big difference. Because normally on a kernel build, I see most of the overhead in path lookup. But that's only true for otherwise optimized builds that don't have system call auditing etc debugging that spreads the costs out over everything..

Yeah, I was having some doubts about the numbers above so I ran my own benchmarking, machine is a Piledriver box:

vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0

and I don't really see any of those improvements above. Actually, -march=bdver2 is even slightly worse in comparison to mk8.

And the workload is of building a config specific to that machine but allmodconfig looks very similar, the numbers being simply higher.

$ zgrep MK8 /proc/config.gz
CONFIG_MK8=y

/home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage

Performance counter stats for 'make -s -j64 bzImage' (10 runs):

    1081808.628840 task-clock                #    7.996 CPUs utilized            ( +-  0.06% ) [100.00%]
         1,203,753 context-switches          #    0.001 M/sec                    ( +-  0.04% ) [100.00%]
            48,748 cpu-migrations            #    0.045 K/sec                    ( +-  0.59% ) [100.00%]
        31,145,439 page-faults               #    0.029 M/sec                    ( +-  0.00% )
 3,836,736,801,500 cycles                    #    3.547 GHz                      ( +-  0.03% ) [100.00%]
   957,386,966,493 stalled-cycles-frontend   #   24.95% frontend cycles idle     ( +-  0.06% ) [100.00%]
   218,581,249,251 stalled-cycles-backend    #    5.70% backend  cycles idle     ( +-  0.06% ) [100.00%]
 2,466,632,641,972 instructions              #    0.64  insns per cycle
                                             #    0.39  stalled cycles per insn  ( +-  0.00% ) [100.00%]
   537,749,333,838 branches                  #  497.084 M/sec                    ( +-  0.00% ) [100.00%]
    27,802,940,176 branch-misses             #    5.17% of all branches          ( +-  0.00% )

135.292843025 seconds time elapsed ( +- 0.06% )

$ zgrep PILEDRIVER /proc/config.gz
CONFIG_MPILEDRIVER=y

/home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage

Performance counter stats for 'make -s -j64 bzImage' (10 runs):

    1085723.230470 task-clock                #    7.996 CPUs utilized            ( +-  0.10% ) [100.00%]
         1,204,355 context-switches          #    0.001 M/sec                    ( +-  0.10% ) [100.00%]
            49,143 cpu-migrations            #    0.045 K/sec                    ( +-  0.76% ) [100.00%]
        31,196,575 page-faults               #    0.029 M/sec                    ( +-  0.00% )
 3,851,255,065,133 cycles                    #    3.547 GHz                      ( +-  0.02% ) [100.00%]
   958,840,197,117 stalled-cycles-frontend   #   24.90% frontend cycles idle     ( +-  0.09% ) [100.00%]
   220,260,399,411 stalled-cycles-backend    #    5.72% backend  cycles idle     ( +-  0.04% ) [100.00%]
 2,466,701,295,156 instructions              #    0.64  insns per cycle
                                             #    0.39  stalled cycles per insn  ( +-  0.00% ) [100.00%]
   537,992,040,195 branches                  #  495.515 M/sec                    ( +-  0.00% ) [100.00%]
    27,860,290,286 branch-misses             #    5.18% of all branches          ( +-  0.00% )

135.784111961 seconds time elapsed ( +- 0.10% )

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#10 lore raw
Quoted 74 lines · Linus Torvalds
Quoted 11 lines
Quoted 5 lines · Austin S Hemmelgarn

improved. Building kernel 3.12-rc2 with allmodconfig using 8 jobs on a FX-8320 takes

22 minutes and 57 seconds on a kernel with CONFIG_MK8, 21 minutes and 35 seconds on a kernel with CONFIG_GENERIC, and 19 minutes and 11 seconds on a kernel with CONFIG_PILEDRIVER.

That's certainly noticeable. Surprisingly so. What makes MK8 so bad in particular, I wonder?

Quoted 6 lines

Just out of interest, have you done any profiles on the kernel cost here to see what it is that makes such a big difference. Because normally on a kernel build, I see most of the overhead in path lookup. But that's only true for otherwise optimized builds that don't have system call auditing etc debugging that spreads the costs out over everything..

Yeah, I was having some doubts about the numbers above so I ran my own benchmarking, machine is a Piledriver box:

vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0

and I don't really see any of those improvements above. Actually, -march=bdver2 is even slightly worse in comparison to mk8.

And the workload is of building a config specific to that machine but allmodconfig looks very similar, the numbers being simply higher.

$ zgrep MK8 /proc/config.gz
CONFIG_MK8=y

/home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage

Performance counter stats for 'make -s -j64 bzImage' (10 runs):

    1081808.628840 task-clock                #    7.996 CPUs utilized            ( +-  0.06% ) [100.00%]
         1,203,753 context-switches          #    0.001 M/sec                    ( +-  0.04% ) [100.00%]
            48,748 cpu-migrations            #    0.045 K/sec                    ( +-  0.59% ) [100.00%]
        31,145,439 page-faults               #    0.029 M/sec                    ( +-  0.00% )
 3,836,736,801,500 cycles                    #    3.547 GHz                      ( +-  0.03% ) [100.00%]
   957,386,966,493 stalled-cycles-frontend   #   24.95% frontend cycles idle     ( +-  0.06% ) [100.00%]
   218,581,249,251 stalled-cycles-backend    #    5.70% backend  cycles idle     ( +-  0.06% ) [100.00%]
 2,466,632,641,972 instructions              #    0.64  insns per cycle
                                             #    0.39  stalled cycles per insn  ( +-  0.00% ) [100.00%]
   537,749,333,838 branches                  #  497.084 M/sec                    ( +-  0.00% ) [100.00%]
    27,802,940,176 branch-misses             #    5.17% of all branches          ( +-  0.00% )

135.292843025 seconds time elapsed ( +- 0.06% )

$ zgrep PILEDRIVER /proc/config.gz
CONFIG_MPILEDRIVER=y

/home/boris/bin/perf stat --repeat 10 -a --sync --pre /home/boris/kernel/pre-build-kernel.sh make -s -j64 bzImage

Performance counter stats for 'make -s -j64 bzImage' (10 runs):

    1085723.230470 task-clock                #    7.996 CPUs utilized            ( +-  0.10% ) [100.00%]
         1,204,355 context-switches          #    0.001 M/sec                    ( +-  0.10% ) [100.00%]
            49,143 cpu-migrations            #    0.045 K/sec                    ( +-  0.76% ) [100.00%]
        31,196,575 page-faults               #    0.029 M/sec                    ( +-  0.00% )
 3,851,255,065,133 cycles                    #    3.547 GHz                      ( +-  0.02% ) [100.00%]
   958,840,197,117 stalled-cycles-frontend   #   24.90% frontend cycles idle     ( +-  0.09% ) [100.00%]
   220,260,399,411 stalled-cycles-backend    #    5.72% backend  cycles idle     ( +-  0.04% ) [100.00%]
 2,466,701,295,156 instructions              #    0.64  insns per cycle
                                             #    0.39  stalled cycles per insn  ( +-  0.00% ) [100.00%]
   537,992,040,195 branches                  #  495.515 M/sec                    ( +-  0.00% ) [100.00%]
    27,860,290,286 branch-misses             #    5.18% of all branches          ( +-  0.00% )
Quoted 2 lines

135.784111961 seconds time elapsed ( +- 0.10% )

Part of the difference between our results may be that I have my entire userspace built with -mtune=bdver2, so less of the time is spent in userspace. Also, the part about using many more threads than cpu cores was with regards to sysbench, not the kernel build, for that I just used 8 jobs in make.

With regards to the differences shown above relative to CONFIG_MK8, that does actually make sense; with CONFIG_MK8, gcc makes very minimal use of extension instructions (afaik, only MMX, SSE, and 3Dnow!), this improves performance slightly on bulldozer derivatives because there are only half as many SSE and FP units as CPU cores (and the scheduler isn't as smart as it could be with regards to that, but that is something for another patch as far as I am concerned).

maintainer
#11 lore raw
Quoted 3 lines · Austin S Hemmelgarn

Part of the difference between our results may be that I have my entire userspace built with -mtune=bdver2, so less of the time is spent in userspace.

That still cannot explain the huge difference between building on a mk8 vs bdver2 kernel. Provided your userspace is the same and only the kernels are different, I don't see how that happens.

Btw, you could run my workload on your machine, once on an mk8 kernel and once on a bdver2 one.

The contents of the pre-build-kernel.sh script are:

--
#!/bin/bash

make -s clean
echo 3 > /proc/sys/vm/drop_caches
--

Of course, you need to run this as root.

Quoted 3 lines

Also, the part about using many more threads than cpu cores was with regards to sysbench, not the kernel build, for that I just used 8 jobs in make.

Overcommitting gcc threads does not bring a really a big difference with a kernel build.

Quoted 5 lines

With regards to the differences shown above relative to CONFIG_MK8, that does actually make sense; with CONFIG_MK8, gcc makes very minimal use of extension instructions (afaik, only MMX, SSE, and 3Dnow!), this improves performance slightly on bulldozer derivatives because there are only half as many SSE and FP units as CPU cores (and the

yes, the FPU is shared between the cores of a compute unit on BD.

Quoted 2 lines

scheduler isn't as smart as it could be with regards to that, but that is something for another patch as far as I am concerned).

Again, with unchanging userspace and only differing kernels, you shouldn't be seeing such a big difference. Besides, the kernel itself makes very little, if at all, use of FPU instructions.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--