Kconfig: default to CC_OPTIMIZE_FOR_PERFORMANCE_O3 for gcc >= 10
Subsystems (2)
PATCH 1/1 Kconfig: default to CC_OPTIMIZE_FOR_PERFORMANCE_O3 for gcc >= 10
GCC 10 appears to have changed -O2 in order to make compilation time faster when using -flto, seemingly at the expense of performance, in particular with regards to how the inliner works. Since -O3 these days shouldn't have the same set of bugs as 10 years ago, this commit defaults new kernel compiles to -O3 when using gcc >= 10.
--- init/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
init/Kconfig +2 -1
@@ -1245,7 +1245,8 @@ config BOOT_CONFIG choice prompt "Compiler optimization level" - default CC_OPTIMIZE_FOR_PERFORMANCE + default CC_OPTIMIZE_FOR_PERFORMANCE_O3 if GCC_VERSION >= 100000 + default CC_OPTIMIZE_FOR_PERFORMANCE if (GCC_VERSION < 100000 || CC_IS_CLANG) config CC_OPTIMIZE_FOR_PERFORMANCE bool "Optimize for performance (-O2)"
Signature · 1 line
2.26.2
Quoted 5 lines · Jason A. Donenfeld
GCC 10 appears to have changed -O2 in order to make compilation time faster when using -flto, seemingly at the expense of performance, in particular with regards to how the inliner works. Since -O3 these days shouldn't have the same set of bugs as 10 years ago, this commit defaults new kernel compiles to -O3 when using gcc >= 10.
Would be nice to get some GCC person's feedback on this. But in general, I think you're right in that O3 isn't the code-gen disaster it used to be.
Quoted 22 lines
Signed-off-by Jason A. Donenfeld <[email protected]>--- init/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)init/Kconfig +2
-1@@ -1245,7 +1245,8 @@ config BOOT_CONFIG choice prompt "Compiler optimization level" - default CC_OPTIMIZE_FOR_PERFORMANCE + default CC_OPTIMIZE_FOR_PERFORMANCE_O3 if GCC_VERSION >= 100000 + default CC_OPTIMIZE_FOR_PERFORMANCE if (GCC_VERSION < 100000 || CC_IS_CLANG) config CC_OPTIMIZE_FOR_PERFORMANCE bool "Optimize for performance (-O2)"Signature · 1 line
2.26.2
Quoted 28 lines · Jason A. Donenfeld
GCC 10 appears to have changed -O2 in order to make compilation time faster when using -flto, seemingly at the expense of performance, in particular with regards to how the inliner works. Since -O3 these days shouldn't have the same set of bugs as 10 years ago, this commit defaults new kernel compiles to -O3 when using gcc >= 10.
Signed-off-by Jason A. Donenfeld <[email protected]>--- init/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)init/Kconfig +2
-1@@ -1245,7 +1245,8 @@ config BOOT_CONFIG choice prompt "Compiler optimization level" - default CC_OPTIMIZE_FOR_PERFORMANCE + default CC_OPTIMIZE_FOR_PERFORMANCE_O3 if GCC_VERSION >= 100000 + default CC_OPTIMIZE_FOR_PERFORMANCE if (GCC_VERSION < 100000 || CC_IS_CLANG) config CC_OPTIMIZE_FOR_PERFORMANCE bool "Optimize for performance (-O2)"Signature · 1 line
2.26.2
Should we untangle -O3 from depending on ARC first maybe?
Signature · 3 lines
Best regards,
Oleksandr Natalenko (post-factum)
Principal Software Maintenance EngineerQuoted 2 lines · Oleksandr Natalenko
Should we untangle -O3 from depending on ARC first maybe?
Oh, hah, good point. Yes, I'll do that for a v2, but will wait another day for feedback first.
Jason
Quoted 4 lines · Jason A. Donenfeld
Quoted 1 line
Should we untangle -O3 from depending on ARC first maybe?
Oh, hah, good point. Yes, I'll do that for a v2, but will wait another day for feedback first.
Just keep in mind that my previous attempt [1] failed because of too many false positive warnings despite -O3 really uncovered a couple of bugs in the codebase.
Lets hope your attempt will be more successfull. I'll happily offer my review tag ;).
Also Cc'ing Andrew who (IIRC) tried to took my sumbission and Arnd who tried to clean up the mess afterwards.
[1] https://lore.kernel.org/lkml/[email protected]/
Signature · 3 lines
Best regards,
Oleksandr Natalenko (post-factum)
Principal Software Maintenance EngineerQuoted 10 lines · Oleksandr Natalenko
Quoted 4 lines · Jason A. Donenfeld
Quoted 1 line
Should we untangle -O3 from depending on ARC first maybe?
Oh, hah, good point. Yes, I'll do that for a v2, but will wait another day for feedback first.
Just keep in mind that my previous attempt [1] failed because of too many false positive warnings despite -O3 really uncovered a couple of bugs in the codebase.
I think my warning fixes were mostly picked up in the meantime, but if there are any remaining, they would be mixed in with random other fixes in my testing tree, so it's hard to know for sure.
I also want to hear the feedback from the gcc developers about what the general recommendations are between O2 and O3, and how they may have changed over time. According to the gcc-10 documentation, the difference between -O2 and -O3 is exactly this set of options:
-fgcse-after-reload
-fipa-cp-clone
-floop-interchange
-floop-unroll-and-jam
-fpeel-loops
-fpredictive-commoning
-fsplit-loops
-fsplit-paths
-ftree-loop-distribution
-ftree-loop-vectorize
-ftree-partial-pre
-ftree-slp-vectorize
-funswitch-loops
-fvect-cost-model
-fvect-cost-model=dynamic
-fversion-loops-for-strides
It's a relatively short list, so someone familiar with the options could perhaps look into whether we want to change the default for all of them, or if it makes sense to be more selective.
Personally, I'm more interested in improving compile speed of the kernel and eventually supporting -Og or some variant of it for my own build testing, but of course I also want to make sure that the other optimization levels do not produce warnings, and -Og leads to more problems than -O3 at the moment.
Arnd
Quoted 28 lines · Arnd Bergmann
Quoted 10 lines · Oleksandr Natalenko
Quoted 4 lines · Jason A. Donenfeld
Quoted 1 line
Should we untangle -O3 from depending on ARC first maybe?
Oh, hah, good point. Yes, I'll do that for a v2, but will wait another day for feedback first.
Just keep in mind that my previous attempt [1] failed because of too many false positive warnings despite -O3 really uncovered a couple of bugs in the codebase.
I think my warning fixes were mostly picked up in the meantime, but if there are any remaining, they would be mixed in with random other fixes in my testing tree, so it's hard to know for sure.
I also want to hear the feedback from the gcc developers about what the general recommendations are between O2 and O3, and how they may have changed over time. According to the gcc-10 documentation, the difference between -O2 and -O3 is exactly this set of options:
(quoted diff) +0
-6-fgcse-after-reload -fipa-cp-clone -floop-interchange -floop-unroll-and-jam -fpeel-loops -fpredictive-commoning
Review
Quoted 10 lines
(quoted diff) +0
-10-fsplit-loops -fsplit-paths -ftree-loop-distribution -ftree-loop-vectorize -ftree-partial-pre -ftree-slp-vectorize -funswitch-loops -fvect-cost-model -fvect-cost-model=dynamic -fversion-loops-for-strides
The other significant thing -- and what prompted this patchset -- is it looks like gcc 10 has lowered the inlining degree for -O2, and put gcc 9's inlining parameters from -O2 into gcc-10's -O3.
Quoted 5 lines · Jason A. Donenfeld
On Fri, May 8, 2020 at 5:56 AM Arnd Bergmann <[email protected]> wrote:
The other significant thing -- and what prompted this patchset -- is it looks like gcc 10 has lowered the inlining degree for -O2, and put gcc 9's inlining parameters from -O2 into gcc-10's -O3.
I suspect it is more complicated than that, as there are a number of parameters that determine inlining decisions. It's also not clear whether the ones for -O3 are generally better than the ones with -O2, or if it's just that whatever changed caused a few surprises but is otherwise preferable.
Did you see regressions in specific modules, or just a general slowdown or growth in object size as the result?
Arnd
Quoted 1 line · Arnd Bergmann
Personally, I'm more interested in improving compile speed of the kernel
Any opinion on precompiled header support?
Quoted 5 lines · Joe Perches
Quoted 1 line · Arnd Bergmann
Personally, I'm more interested in improving compile speed of the kernel
Any opinion on precompiled header support?
I have not tried it. IIRC precompiled headers usually work best for projects that have a large header with all the global declarations that gets included everywhere, while Linux has always tried (with different amounts of success) to minimize the number of headers that get included per file.
Arnd
From: Joe Perches
Quoted 5 lines
Sent: 08 May 2020 16:06
Quoted 1 line · Arnd Bergmann
Personally, I'm more interested in improving compile speed of the kernel
Any opinion on precompiled header support?
When ever I've been anywhere near it it is always a disaster. It may make sense for C++ where there is lots of complicated code to parse in .h files. Parsing C headers is usually easier.
One this I have done that significantly speeds up .h file processing is to take the long list of '-I directory' parameters that are passed to the compiler and copy the first version of each file into a separate 'object headers' directory. This saves the compiler doing lots of 'failed opens'.
If each fragment makefile lists its 'public' headers make can generate dependency rules that do the copies.
FWIW make is much faster if you delete all the builtin and suffix rules and rely on explicit rules for each file.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 8 lines · David Laight
From: Joe Perches
Quoted 5 lines
Sent: 08 May 2020 16:06
Quoted 1 line · Arnd Bergmann
Personally, I'm more interested in improving compile speed of the kernel
Any opinion on precompiled header support?
When ever I've been anywhere near it it is always a disaster.
A disaster? Why?
For a large commercial c only project, it worked well by reducing a combined multi-include file, similar to kernel.h here, to a single file.
That was before SSDs though and the file open times might have been rather larger then.
From: Joe Perches
Quoted 6 lines
Sent: 10 May 2020 18:45
Quoted 3 lines · David Laight
From: Joe Perches
Quoted 2 lines
Sent: 08 May 2020 16:06
On Fri, 2020-05-08 at 13:49 +0200, Arnd Bergmann wrote:
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Personally, I'm more interested in improving compile speed of the kernel
Any opinion on precompiled header support?
When ever I've been anywhere near it it is always a disaster.
A disaster? Why?
The only time I've had systems that used them they always got out of step with the headers - probable due to #define changes. If auto-generated by the compiler then parallel makes also give problems.
Quoted 3 lines
For a large commercial c only project, it worked well by reducing a combined multi-include file, similar to
kernel.hhere, to a single file.
Certainly reducing the number of directories searched can make a big difference.
I've also compiled .so by merging all the sources into a single file.
Quoted 2 lines
That was before SSDs though and the file open times might have been rather larger then.
The real killer is lots of directory names in the -I <paths> especially over NFS.
I've also looked at system call stats during a kernel compile. open() dominated and my 'gut feeling' was that most were failing opens.
I also suspect that modern compilers remember that an include file contained an include guard - and don't even both looking for it a second time.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 18 lines · David Laight
From: Joe Perches
Quoted 5 lines
Sent: 08 May 2020 16:06
Quoted 1 line · Arnd Bergmann
Personally, I'm more interested in improving compile speed of the kernel
Any opinion on precompiled header support?
When ever I've been anywhere near it it is always a disaster. It may make sense for C++ where there is lots of complicated code to parse in .h files. Parsing C headers is usually easier.
One this I have done that significantly speeds up .h file processing is to take the long list of '-I directory' parameters that are passed to the compiler and copy the first version of each file into a separate 'object headers' directory. This saves the compiler doing lots of 'failed opens'.
Quoted 5 lines
If each fragment makefile lists its 'public' headers make can generate dependency rules that do the copies.
FWIW make is much faster if you delete all the builtin and suffix rules and rely on explicit rules for each file.
Kbuild disables Make's builtin rules at least.
# Do not use make's built-in rules and variables
# (this increases performance and avoids hard-to-debug behaviour) MAKEFLAGS += -rR
--
Best Regards
Masahiro Yamada
PATCH v2 1/1 Kconfig: default to CC_OPTIMIZE_FOR_PERFORMANCE_O3 for gcc >= 10
In reply to Oleksandr NatalenkoGCC 10 appears to have changed -O2 in order to make compilation time faster when using -flto, seemingly at the expense of performance, in particular with regards to how the inliner works. Since -O3 these days shouldn't have the same set of bugs as 10 years ago, this commit defaults new kernel compiles to -O3 when using gcc >= 10.
---
Changes v1->v2:
- [Oleksandr] Remove O3 dependency on ARC.
init/Kconfig | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
init/Kconfig +2 -2
@@ -1245,7 +1245,8 @@ config BOOT_CONFIG choice prompt "Compiler optimization level" - default CC_OPTIMIZE_FOR_PERFORMANCE + default CC_OPTIMIZE_FOR_PERFORMANCE_O3 if GCC_VERSION >= 100000 + default CC_OPTIMIZE_FOR_PERFORMANCE if (GCC_VERSION < 100000 || CC_IS_CLANG) config CC_OPTIMIZE_FOR_PERFORMANCE bool "Optimize for performance (-O2)"
@@ -1256,7 +1257,6 @@ config CC_OPTIMIZE_FOR_PERFORMANCE config CC_OPTIMIZE_FOR_PERFORMANCE_O3 bool "Optimize more for performance (-O3)" - depends on ARC imply CC_DISABLE_WARN_MAYBE_UNINITIALIZED # avoid false positives help Choosing this option will pass "-O3" to your compiler to optimize
Signature · 1 line
2.26.2
Quoted 6 lines · Jason A. Donenfeld
GCC 10 appears to have changed -O2 in order to make compilation time faster when using -flto, seemingly at the expense of performance, in particular with regards to how the inliner works. Since -O3 these days shouldn't have the same set of bugs as 10 years ago, this commit defaults new kernel compiles to -O3 when using gcc >= 10.
I'm not convinced this is sensible.
-O3 historically does bad things with gcc. Including bad things for performance. It traditionally makes code larger and often SLOWER.
And I don't mean slower to compile (although that's an issue). I mean actually generating slower code.
Things like trying to unroll loops etc makes very little sense in the kernel, where we very seldom have high loop counts for pretty much anything.
There's a reason -O3 isn't even offered as an option.
Maybe things have changed, and maybe they've improved. But I'd like to see actual numbers for something like this.
Not inlining as aggressively is not necessarily a bad thing. It can be, of course. But I've actually also done gcc bugreports about gcc inlining too much, and generating _worse_ code as a result (ie inlinging things that were behind an "if (unlikely())" test, and causing the likely path to grow a stack fram and stack spills as a result).
So just "O3 inlines more" is not a valid argument.
Linus
Quoted 7 lines · Linus Torvalds
Not inlining as aggressively is not necessarily a bad thing. It can be, of course. But I've actually also done gcc bugreports about gcc inlining too much, and generating _worse_ code as a result (ie inlinging things that were behind an "if (
unlikely())" test, and causing the likely path to grow a stack fram and stack spills as a result).
In case people care, the bugzilla case I mentioned is this one:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=49194
with example code on why it's actively wrong to inline.
Obviously, in the kernel, we can fix the obvious cases with "noinline" and "always_inline", but those take care of the outliers. Having a compiler that does reasonably well by default is a good thing, and that very much includes *not* inlining mindlessly.
Linus
Quoted 5 lines · Linus Torvalds
There's a reason -O3 isn't even offered as an option.
Maybe things have changed, and maybe they've improved. But I'd like to see actual numbers for something like this.
Quoted 8 lines
Not inlining as aggressively is not necessarily a bad thing. It can be, of course. But I've actually also done gcc bugreports about gcc inlining too much, and generating _worse_ code as a result (ie inlinging things that were behind an "if (
unlikely())" test, and causing the likely path to grow a stack fram and stack spills as a result).So just "O3 inlines more" is not a valid argument.
Alright. It might be possible to produce some benchmarks, and then isolate the precise inlining parameter that makes the difference, and include that for gcc-10. But you made a compelling argument in that old gcc bug report about not going down the finicky rabbit hole of gcc inlining switches that seem to change meaning between releases, which is persuasive.
The other possibility would be if -O3 actually isn't bad like it used to be and the codegen is markedly better, alongside some numbers to back it up. I'm not presently making that argument and don't have those numbers, but perhaps others who were interested in this patch for other reasons do have strong arguments there and want to chime in. Otherwise, no problem dropping this.
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Jason A. Donenfeld
GCC 10 appears to have changed -O2 in order to make compilation time faster when using -flto, seemingly at the expense of performance, in particular with regards to how the inliner works. Since -O3 these days shouldn't have the same set of bugs as 10 years ago, this commit defaults new kernel compiles to -O3 when using gcc >= 10.
I'm not convinced this is sensible.
Note the real thing that changed for GCC 10 at -O2 is that -O2 now includes -finline-functions which means GCC considers inlining of functions not marked with 'inline' at -O2. To counter code-size growth and tune that back to previous levels the inlining limits in effect at -O2 have been lowered.
Note this has been done based on analyzing larger C++ code and obviously not because the kernel would benefit (IIRC kernel folks like 'inline' to behave as written and thus rather may dislike the change to default to -finline-functions).
Quoted 11 lines
-O3 historically does bad things with gcc. Including bad things for performance. It traditionally makes code larger and often SLOWER.
And I don't mean slower to compile (although that's an issue). I mean actually generating slower code.
Things like trying to unroll loops etc makes very little sense in the kernel, where we very seldom have high loop counts for pretty much anything.
There's a reason -O3 isn't even offered as an option.
And I think that's completely sensible. I would not recommend to use -O3 for the kernel. Somehow feeding back profile data might help - though getting such data at all and with enough coverage is probably hard.
As you said in the followup I wouldn't recommend tweaking GCCs defaults for the various --param affecting inlining. The behavior with this is not consistent across releases.
Richard.
Quoted 14 lines
Maybe things have changed, and maybe they've improved. But I'd like to see actual numbers for something like this.
Not inlining as aggressively is not necessarily a bad thing. It can be, of course. But I've actually also done gcc bugreports about gcc inlining too much, and generating _worse_ code as a result (ie inlinging things that were behind an "if (
unlikely())" test, and causing the likely path to grow a stack fram and stack spills as a result).So just "O3 inlines more" is not a valid argument.
Linus
Signature · 3 lines
Richard Biener <[email protected]> SUSE Software Solutions Germany GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany; GF: Felix Imendörffer; HRB 36809 (AG Nuernberg)
PATCH 1/1 Kconfig: default to CC_OPTIMIZE_FOR_PERFORMANCE_O3 for gcc >= 10
In reply to Jason A. DonenfeldQuoted 1 line
GCC 10 appears to have changed -O2 in order to make compilation time
faster when using -flto, seemingly at the expense of performance, in particular with regards to how the inliner works. Since -O3 these days shouldn't have the same set of bugs as 10 years ago, this commit defaults new kernel compiles to -O3 when using gcc >= 10.
It's a strong "no" from me.
1) Aside from rare Gentoo users no one has extensively tested -O3 with the kernel - even Gentoo defaults to -O2 for kernel compilation
2) -O3 _always_ bloats the code by a large amount which means both vmlinux/bzImage and modules will become bigger, and slower to load from the disk
3) -O3 does _not_ necessarily makes the code run faster
4) If GCC10 has removed certain options for the -O2 optimization level you could just readded them as compilation flags without forcing -O3 by default on everyone
5) If you still insist on -O3 I guess everyone would be happy if you just made two KConfig options:
OPTIMIZE_O2 (-O2)OPTIMIZE_O3_EVEN_MOAR (-O3)
Best regards,
Artem