x86: Use -m-omit-leaf-frame-pointer to shrink text size
PATCH 1/1 x86: Use -m-omit-leaf-frame-pointer to shrink text size
This patch turns on -momit-leaf-frame-pointer on x86 builds and thus shrinks .text noticeably. On a defconfig-ish kernel:
text data bss dec hex filename
9843902 1935808 3649536 15429246 eb6e7e vmlinux.before
9813764 1935792 3649536 15399092 eaf8b4 vmlinux.after
That's 0.3% off text size.
The actual win is larger than this percentage suggests: many small, hot helper functions such as find_next_bit(), do_raw_spin_lock() or most of the list_*() functions are leaf functions and are now shorter by 2 instructions.
Probably a good chunk of the framepointers related runtime overhead on common workloads is eliminated via this patch, as small leaf functions execute more often than larger parent functions.
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
--- arch/x86/Makefile | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/Makefile +8 -0
@@ -72,6 +72,14 @@ else KBUILD_CFLAGS += -maccumulate-outgoing-args endif +# +# This shrinks many small functions, we don't actually +# need their frame pointer, in backtraces the RIP will +# identify the function and the stack frame walker will +# find the parent function: +# +KBUILD_CFLAGS += $(call cc-option,-momit-leaf-frame-pointer) + ifdef CONFIG_CC_STACKPROTECTOR cc_has_sp := $(srctree)/scripts/gcc-x86_$(BITS)-has-stack-protector.sh ifeq ($(shell $(CONFIG_SHELL) $(cc_has_sp) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)
Quoted 20 lines · On Fri, 16 Dec 2011 09:19:16 +0100 Ingo Molnar <[email protected]> wrote:
This patch turns on -momit-leaf-frame-pointer on x86 builds and thus shrinks .text noticeably. On a defconfig-ish kernel:
text data bss dec hex filename 9843902 1935808 3649536 15429246 eb6e7e vmlinux.before 9813764 1935792 3649536 15399092 eaf8b4 vmlinux.afterThat's 0.3% off text size.
The actual win is larger than this percentage suggests: many small, hot helper functions such as
find_next_bit(),do_raw_spin_lock()or most of the list_*() functions are leaf functions and are now shorter by 2 instructions.Probably a good chunk of the framepointers related runtime overhead on common workloads is eliminated via this patch, as small leaf functions execute more often than larger parent functions.
Quoted 4 lines
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
The only problem I can think of (apart from tickling gcc bugs) is that it might break __builtin_return_address(n) for n>0 with frame pointers enabled? The only code I can find which does this is drivers/isdn/hardware/mISDN/ and ftrace.
Quoted 27 lines · Andrew Morton
Quoted 20 lines · On Fri, 16 Dec 2011 09:19:16 +0100 Ingo Molnar <[email protected]> wrote:
This patch turns on -momit-leaf-frame-pointer on x86 builds and thus shrinks .text noticeably. On a defconfig-ish kernel:
text data bss dec hex filename 9843902 1935808 3649536 15429246 eb6e7e vmlinux.before 9813764 1935792 3649536 15399092 eaf8b4 vmlinux.afterThat's 0.3% off text size.
The actual win is larger than this percentage suggests: many small, hot helper functions such as
find_next_bit(),do_raw_spin_lock()or most of the list_*() functions are leaf functions and are now shorter by 2 instructions.Probably a good chunk of the framepointers related runtime overhead on common workloads is eliminated via this patch, as small leaf functions execute more often than larger parent functions.
Quoted 4 lines
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
Quoted 4 lines
The only problem I can think of (apart from tickling gcc bugs) is that it might break
__builtin_return_address(n)for n>0 with frame pointers enabled? The only code I can find which does this isdrivers/isdn/hardware/mISDN/and ftrace.
Well, AFAICS it won't really 'break' it but behave as if the leaf function got inlined into the parent function. I think we can live with that.
Thanks,
Ingo
Quoted 6 lines · Ingo Molnar
[...]
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
Hm, noticed one complication while looking at annotated assembly code in perf top. Code doing function calls from within asm() is incorrectly marked 'leaf' by GCC:
ffffffff812b82d8 <arch_local_save_flags>:
ffffffff812b82d8: ff 14 25 00 d9 c1 81 callq *0xffffffff81c1d900 ffffffff812b82df: c3 retq
So all the paravirt details will have to be fixed, so that GCC is able to see that there's a real function call done inside. Jeremy, Konrad?
Thanks,
Ingo
Quoted 12 lines · Ingo Molnar
Quoted 6 lines · Ingo Molnar
[...]
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
Hm, noticed one complication while looking at annotated assembly code in perf top. Code doing function calls from within
asm()is incorrectly marked 'leaf' by GCC:
Quoted 7 lines
ffffffff812b82d8<arch_local_save_flags>:ffffffff812b82d8: ff 14 25 00 d9 c1 81 callq *0xffffffff81c1d900 ffffffff812b82df: c3 retqSo all the paravirt details will have to be fixed, so that GCC is able to see that there's a real function call done inside. Jeremy, Konrad?
Um. So the issue is that a function that contains only pvops looks like it's a leaf to gcc and it does some leaf-function optimisation?
How can we tell gcc the asm contains a call, or otherwise suppress the "leaf function" classification?
The alternative is to just make it a plain C-level indirect call, but then we'd lose all the patching and callee-save optimisations.
Any suggestions?
Thanks,
J
Quoted 24 lines · Jeremy Fitzhardinge
Quoted 12 lines · Ingo Molnar
Quoted 6 lines · Ingo Molnar
[...]
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
Hm, noticed one complication while looking at annotated assembly code in perf top. Code doing function calls from within
asm()is incorrectly marked 'leaf' by GCC:Quoted 7 lines
ffffffff812b82d8<arch_local_save_flags>:ffffffff812b82d8: ff 14 25 00 d9 c1 81 callq *0xffffffff81c1d900 ffffffff812b82df: c3 retqSo all the paravirt details will have to be fixed, so that GCC is able to see that there's a real function call done inside. Jeremy, Konrad?
Um. So the issue is that a function that contains only pvops looks like it's a leaf to gcc and it does some leaf-function optimisation?
Quoted 7 lines
How can we tell gcc the asm contains a call, or otherwise suppress the "leaf function" classification?
The alternative is to just make it a plain C-level indirect call, but then we'd lose all the patching and callee-save optimisations.
Any suggestions?
Added Richard Henderson to CC.
I only found the function __attribute__((leaf)) to explicitly mark a function as being a leaf function, but the documentation doesn't list the inverse of that to explicitly mark it as _not_ being one.
I haven't done a git grep on the gcc sources yet since I seem to have misplaced my gcc.git tree.
Quoted 2 lines · Peter Zijlstra
Quoted 2 lines
How can we tell gcc the asm contains a call, or otherwise suppress the "leaf function" classification?
You can't at present.
Quoted 3 lines
I only found the function __attribute__((leaf)) to explicitly mark a function as being a leaf function, but the documentation doesn't list the inverse of that to explicitly mark it as _not_ being one.
In any case, "leaf" doesn't do what you think it does -- see the docs. I told Honza at the time that he was overloading existing terminology without purpose, but no one came up with a better name before the release, so it stayed that way. AFAIK, no one uses it because no one understands it.
Do you have a proposal for what you'd like the extension to look like? The only thing that comes to mind atm is "call" in the clobber section. This is not likely to get written before gcc 4.8 though...
r~
Quoted 10 lines
Quoted 1 line
Quoted 1 line · Jeremy Fitzhardinge
Quoted 8 lines · Ingo Molnar
Quoted 6 lines · Ingo Molnar
[...]
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
Are you sure about that even if the leaf function uses rBP for a different purpose?
Quoted 11 lines
Quoted 11 lines
Hm, noticed one complication while looking at annotated assembly code in perf top. Code doing function calls from within
asm()is incorrectly marked 'leaf' by GCC:
ffffffff812b82d8<arch_local_save_flags>:ffffffff812b82d8: ff 14 25 00 d9 c1 81 callq *0xffffffff81c1d900 ffffffff812b82df: c3 retqSo all the paravirt details will have to be fixed, so that GCC is able to see that there's a real function call done inside. Jeremy, Konrad?
If the above is not a problem, wouldn't this simply result in a skipped function layer?
Also, iirc it's not just pv-ops that uses calls within asm()-s.
Quoted 5 lines
Um. So the issue is that a function that contains only pvops looks like it's a leaf to gcc and it does some leaf-function optimisation?
How can we tell gcc the asm contains a call, or otherwise suppress the "leaf function" classification?
I'm afraid you can't without adding code (i.e. a dummy function call).
Jan
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · Jeremy Fitzhardinge
Quoted 4 lines · Ingo Molnar
Quoted 2 lines · Ingo Molnar
[...]
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
Are you sure about that even if the leaf function uses rBP for a different purpose?
Well, i assumed that GCC does not mess with %bp in leaf functions - a frame pointer is barely useful if it's destroyed spuriously in leaf functions.
A quick grep of the assembly appears to support that assumption:
$ objdump -d vmlinux | grep ',%rbp$' | cut -d: -f2- | sort | uniq -c | sort -n | tail -10
3 48 89 d5 mov %rdx,%rbp
3 4c 89 cd mov %r9,%rbp
4 48 0f 45 e8 cmovne %rax,%rbp
4 48 83 cd ff or $0xffffffffffffffff,%rbp
5 4c 89 dd mov %r11,%rbp
7 48 21 fd and %rdi,%rbp
10 48 d3 e5 shl %cl,%rbp
14 48 85 ed test %rbp,%rbp
14 48 8b 6c 24 20 mov 0x20(%rsp),%rbp
31042 48 89 e5 mov %rsp,%rbp%rbp is not touched, except in a few special assembly glue/entry pieces of code.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Hm, noticed one complication while looking at annotated assembly code in perf top. Code doing function calls from within
asm()is incorrectly marked 'leaf' by GCC:
ffffffff812b82d8<arch_local_save_flags>:ffffffff812b82d8: ff 14 25 00 d9 c1 81 callq *0xffffffff81c1d900 ffffffff812b82df: c3 retq
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
So all the paravirt details will have to be fixed, so that GCC is able to see that there's a real function call done inside. Jeremy, Konrad?
If the above is not a problem, wouldn't this simply result in a skipped function layer?
Yeah - i guess we can live with that, as long as the frame pointer chain is otherwise usable and walkable.
Thanks,
Ingo
H. Peter Anvin
reviewerQuoted 8 lines · Ingo Molnar
Quoted 3 lines
Are you sure about that even if the leaf function uses rBP for a different purpose?
Well, i assumed that GCC does not mess with %bp in leaf functions - a frame pointer is barely useful if it's destroyed spuriously in leaf functions.
We should verify that explicitly. gcc has every "right" to treat it as a normal callee-saved register, but I think it is a very low priority register in gcc's register allocation scheme.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 20 lines · Ingo Molnar
This patch turns on -momit-leaf-frame-pointer on x86 builds and thus shrinks .text noticeably. On a defconfig-ish kernel:
text data bss dec hex filename 9843902 1935808 3649536 15429246 eb6e7e vmlinux.before 9813764 1935792 3649536 15399092 eaf8b4 vmlinux.afterThat's 0.3% off text size.
The actual win is larger than this percentage suggests: many small, hot helper functions such as
find_next_bit(),do_raw_spin_lock()or most of the list_*() functions are leaf functions and are now shorter by 2 instructions.Probably a good chunk of the framepointers related runtime overhead on common workloads is eliminated via this patch, as small leaf functions execute more often than larger parent functions.
Quoted 4 lines
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
Probably not actually. We are going to miss the parent of those leaf functions all the time in the stacktrace.
Consider an irq interrupting the following chain:
spin_lock() -> raw_spin_lock() -> do_raw_spin_lock()
And we do a stacktrace on top of the interrupted regs.
What we we do typically is to include the regs->ip as a first entry (like in perf) or we make it obvious in a bug stacktrace. Then we purely walk through regs->bp (perf) or we walk the stack and validate with regs->bp (bug stacktraces)
If do_raw_spin_lock() is a leaf function, we have the following happening:
1) dump regs->ip = do_raw_spin_lock()
2) then use regs->bp to find the return address, but bp
has been saved in the parent, the return address is the one of the parent, which is = spin_lock() and not raw_spin_lock()
We are more lucky with the paranoid stack walking made for bug reports because we at least find the return address somehow of do_raw_spin_lock() but it will appear with the "?" because it won't be validated by the frame pointer.
I'm not sure we can work around that, unless we can find fast ways to identify which functions are concerned by this ripped frame pointer while we are unwinding, in which case we can use some black magic. And still, we can do something reliable only if we ensure the leaf function has no stackframe (otherwise we can't reliably find its return address).
Quoted 25 lines
Signed-off-by Ingo Molnar <[email protected]>--- arch/x86/Makefile | 8 ++++++++ 1 file changed, 8 insertions(+)Index: linux/arch/x86/Makefile
===================================================================arch/x86/Makefile +8
-0@@ -72,6 +72,14 @@ else KBUILD_CFLAGS += -maccumulate-outgoing-args endif +# +# This shrinks many small functions, we don't actually +# need their frame pointer, in backtraces the RIP will +# identify the function and the stack frame walker will +# find the parent function: +# +KBUILD_CFLAGS += $(call cc-option,-momit-leaf-frame-pointer) + ifdef CONFIG_CC_STACKPROTECTOR cc_has_sp := $(srctree)/scripts/gcc-x86_$(BITS)-has-stack-protector.sh ifeq ($(shell $(CONFIG_SHELL) $(cc_has_sp) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)
Quoted 20 lines · Ingo Molnar
This patch turns on -momit-leaf-frame-pointer on x86 builds and thus shrinks .text noticeably. On a defconfig-ish kernel:
text data bss dec hex filename 9843902 1935808 3649536 15429246 eb6e7e vmlinux.before 9813764 1935792 3649536 15399092 eaf8b4 vmlinux.afterThat's 0.3% off text size.
The actual win is larger than this percentage suggests: many small, hot helper functions such as
find_next_bit(),do_raw_spin_lock()or most of the list_*() functions are leaf functions and are now shorter by 2 instructions.Probably a good chunk of the framepointers related runtime overhead on common workloads is eliminated via this patch, as small leaf functions execute more often than larger parent functions.
Quoted 4 lines
The call-chains are still intact for quality backtraces and for call-chain profiling (perf record -g), as the backtrace walker can deduct the full backtrace from the RIP of a leaf function and the parent chain.
In the case of linked stacks (like frame pointer linking interrupt to exception stack) we may also miss a leaf function (and not its parent).
Now perhaps we can live with all that, I don't know.
Quoted 25 lines
Signed-off-by Ingo Molnar <[email protected]>--- arch/x86/Makefile | 8 ++++++++ 1 file changed, 8 insertions(+)Index: linux/arch/x86/Makefile
===================================================================arch/x86/Makefile +8
-0@@ -72,6 +72,14 @@ else KBUILD_CFLAGS += -maccumulate-outgoing-args endif +# +# This shrinks many small functions, we don't actually +# need their frame pointer, in backtraces the RIP will +# identify the function and the stack frame walker will +# find the parent function: +# +KBUILD_CFLAGS += $(call cc-option,-momit-leaf-frame-pointer) + ifdef CONFIG_CC_STACKPROTECTOR cc_has_sp := $(srctree)/scripts/gcc-x86_$(BITS)-has-stack-protector.sh ifeq ($(shell $(CONFIG_SHELL) $(cc_has_sp) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)