Tracers vs. CALLER_ADDR on PowerPC
Subsystems (5)
v5 cover Tracers vs. CALLER_ADDR on PowerPC
Hi all,
In v5 there are few fixes suggested by Sam Ravnborg:
- Use 8 spaces for indentation in Makefile;
- Rename ARCH_HAS_NORMAL_FRAME_POINTERS to HAVE_NORMAL_FRAME_POINTER;
Thanks,
---
Preface for those who don't know or forgot what the problem is:
Gcc frame pointers do nothing useful on PowerPC (they're harmful, actually), and thus lib/Kconfig.debug makes CONFIG_FRAME_POINTER unselectable on PPC targets, but CALLER_ADDR macros are available only with CONFIG_FRAME_POINTER, therefore tracing is completely useless on PowerPC:
[...]
<idle>-0 0X.h3 2us+: 0:140:R + [000] 1733:120:S mvtsd <idle>-0 0X.h3 9us+: 0 (0) <idle>-0 0X..3 72us : 0 (0) <idle>-0 0X..3 73us : 0:140:R ==> [000] 1733:120:R mvtsd
While it should look like this:
[...]
<idle>-0 0X.h3 2us+: 0:140:R + [000] 1740:120:S mvtsd <idle>-0 0X.h3 9us+: hrtimer_wakeup (__run_hrtimer) <idle>-0 0X..3 87us : cpu_idle (__got2_end) <idle>-0 0X..3 89us : 0:140:R ==> [000] 1740:120:R mvtsd
I've tried to fix the issue via expanding the #ifdef in the ftrace.h:
http://lkml.org/lkml/2009/1/31/141
Then Steven Rostedt suggested to implement something more generic, i.e. HAVE_NORMAL_FRAME_POINTERS Kconfig symbol.
I found a way to solve the problem w/o additional symbols, but with some Makefile magic (http://lkml.org/lkml/2009/2/4/273).
But because of top-level Makefile issues on other arches
(http://lkml.org/lkml/2009/2/14/89) I had to abandon the approach.
So, this patch set combines Steven Rostedt's idea and a small Makefile change, so that now only top-level Makefile has to know about the new symbol, and the rest of the kernel can stay with using CONFIG_FRAME_POINTER.
I'm aware of
commit c79a61f55773d2519fd0525bf58385f7d20752d3
Author: Uwe Kleine-Koenig <[email protected]>
Date: Fri Feb 27 21:30:03 2009 +0100
tracing: make CALLER_ADDRx overwriteable
But I think the patch set is still applicable, considering that it removes gcc bug workaround in a nice way, and makes CONFIG_FRAME_POINTER available on PowerPC, thus other code can rely on that.
If not, I can just fill-in the asm/ftrace.h for PowerPC.
Signature · 3 lines
Anton Vorontsov email: [email protected] irc://irc.freenode.net/bd2
PATCH v5 1/3 powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
This patch introduces HAVE_NORMAL_FRAME_POINTER Kconfig symbol. When defined, the top level Makefile won't add -fno-omit-frame-pointer cflag (the flag is useless for PowerPC kernels, and also makes gcc generate wrong code).
Also move ARCH_WANT_FRAME_POINTERS's help text.
--- Makefile | 7 +++++-- arch/powerpc/Kconfig | 1 + lib/Kconfig.debug | 16 ++++++++++------ 3 files changed, 16 insertions(+), 8 deletions(-)
Makefile +5 -2
@@ -543,9 +543,12 @@ KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector) endif ifdef CONFIG_FRAME_POINTER -KBUILD_CFLAGS += -fno-omit-frame-pointer -fno-optimize-sibling-calls + KBUILD_CFLAGS += -fno-optimize-sibling-calls + ifndef CONFIG_HAVE_NORMAL_FRAME_POINTER + KBUILD_CFLAGS += -fno-omit-frame-pointer + endif else -KBUILD_CFLAGS += -fomit-frame-pointer + KBUILD_CFLAGS += -fomit-frame-pointer endif ifdef CONFIG_DEBUG_INFO
arch/powerpc/Kconfig +1 -0
@@ -119,6 +119,7 @@ config PPC select HAVE_KPROBES select HAVE_ARCH_KGDB select HAVE_KRETPROBES + select HAVE_NORMAL_FRAME_POINTER select HAVE_ARCH_TRACEHOOK select HAVE_LMB select HAVE_DMA_ATTRS if PPC64
lib/Kconfig.debug +10 -6
@@ -660,20 +660,24 @@ config DEBUG_NOTIFIERS This is a relatively cheap check but if you care about maximum performance, say N. -# -# Select this config option from the architecture Kconfig, if it -# it is preferred to always offer frame pointers as a config -# option on the architecture (regardless of KERNEL_DEBUG): -# config ARCH_WANT_FRAME_POINTERS bool help + Select this config option from the architecture Kconfig, if it + it is preferred to always offer frame pointers as a config + option on the architecture (regardless of KERNEL_DEBUG). + +config HAVE_NORMAL_FRAME_POINTER + bool + help + Architectures should select this symbol if their ABI implies + having a frame pointer. config FRAME_POINTER bool "Compile the kernel with frame pointers" depends on DEBUG_KERNEL && \ (CRIS || M68K || M68KNOMMU || FRV || UML || \ - AVR32 || SUPERH || BLACKFIN || MN10300) || \ + AVR32 || SUPERH || BLACKFIN || MN10300 || PPC) || \ ARCH_WANT_FRAME_POINTERS default y if (DEBUG_INFO && UML) || ARCH_WANT_FRAME_POINTERS help
Signature · 1 line
1.6.2.2
PATCH v5 2/3 powerpc: Remove -fno-omit-frame-pointer workarounds
In reply to Anton VorontsovThe workarounds aren't needed any longer since the top level Makefile doesn't pass -fno-omit-frame-pointer cflag for PowerPC builds.
--- arch/powerpc/Makefile | 5 ----- arch/powerpc/kernel/Makefile | 12 ++++++------ arch/powerpc/platforms/powermac/Makefile | 2 +- lib/Kconfig.debug | 6 +++--- 4 files changed, 10 insertions(+), 15 deletions(-)
arch/powerpc/Makefile +0 -5
@@ -120,11 +120,6 @@ ifeq ($(CONFIG_6xx),y) KBUILD_CFLAGS += -mcpu=powerpc endif -# Work around a gcc code-gen bug with -fno-omit-frame-pointer. -ifeq ($(CONFIG_FUNCTION_TRACER),y) -KBUILD_CFLAGS += -mno-sched-epilog -endif - cpu-as-$(CONFIG_4xx) += -Wa,-m405 cpu-as-$(CONFIG_6xx) += -Wa,-maltivec cpu-as-$(CONFIG_POWER4) += -Wa,-maltivec
arch/powerpc/kernel/Makefile +6 -6
@@ -14,14 +14,14 @@ endif ifdef CONFIG_FUNCTION_TRACER # Do not trace early boot code -CFLAGS_REMOVE_cputable.o = -pg -mno-sched-epilog -CFLAGS_REMOVE_prom_init.o = -pg -mno-sched-epilog -CFLAGS_REMOVE_btext.o = -pg -mno-sched-epilog -CFLAGS_REMOVE_prom.o = -pg -mno-sched-epilog +CFLAGS_REMOVE_cputable.o = -pg +CFLAGS_REMOVE_prom_init.o = -pg +CFLAGS_REMOVE_btext.o = -pg +CFLAGS_REMOVE_prom.o = -pg # do not trace tracer code -CFLAGS_REMOVE_ftrace.o = -pg -mno-sched-epilog +CFLAGS_REMOVE_ftrace.o = -pg # timers used by tracing -CFLAGS_REMOVE_time.o = -pg -mno-sched-epilog +CFLAGS_REMOVE_time.o = -pg endif obj-y := cputable.o ptrace.o syscalls.o \
arch/powerpc/platforms/powermac/Makefile +1 -1
@@ -2,7 +2,7 @@ CFLAGS_bootx_init.o += -fPIC ifdef CONFIG_FUNCTION_TRACER # Do not trace early boot code -CFLAGS_REMOVE_bootx_init.o = -pg -mno-sched-epilog +CFLAGS_REMOVE_bootx_init.o = -pg endif obj-y += pic.o setup.o time.o feature.o pci.o \
lib/Kconfig.debug +3 -3
@@ -492,7 +492,7 @@ config LOCKDEP bool depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT select STACKTRACE - select FRAME_POINTER if !X86 && !MIPS && !PPC && !ARM_UNWIND && !S390 + select FRAME_POINTER if !X86 && !MIPS && !ARM_UNWIND && !S390 select KALLSYMS select KALLSYMS_ALL
@@ -865,13 +865,13 @@ config FAULT_INJECTION_STACKTRACE_FILTER depends on FAULT_INJECTION_DEBUG_FS && STACKTRACE_SUPPORT depends on !X86_64 select STACKTRACE - select FRAME_POINTER if !PPC && !S390 + select FRAME_POINTER if !S390 help Provide stacktrace filter for fault-injection capabilities config LATENCYTOP bool "Latency measuring infrastructure" - select FRAME_POINTER if !MIPS && !PPC && !S390 + select FRAME_POINTER if !MIPS && !S390 select KALLSYMS select KALLSYMS_ALL select STACKTRACE
Signature · 1 line
1.6.2.2
PATCH v5 3/3 tracing: Tracers that use CALLER_ADDR macros should select FRAME_POINTER
In reply to Anton VorontsovIrqsoff, switch and preempt tracers use CALLER_ADDR macros, so they should select FRAME_POINTER. Otherwise traces are meaningless.
--- kernel/trace/Kconfig | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/trace/Kconfig +3 -0
@@ -122,6 +122,7 @@ config IRQSOFF_TRACER select TRACE_IRQFLAGS select TRACING select TRACER_MAX_TRACE + select FRAME_POINTER help This option measures the time spent in irqs-off critical sections, with microsecond accuracy.
@@ -143,6 +144,7 @@ config PREEMPT_TRACER depends on PREEMPT select TRACING select TRACER_MAX_TRACE + select FRAME_POINTER help This option measures the time spent in preemption off critical sections, with microsecond accuracy.
@@ -171,6 +173,7 @@ config SCHED_TRACER select TRACING select CONTEXT_SWITCH_TRACER select TRACER_MAX_TRACE + select FRAME_POINTER help This tracer tracks the latency of the highest priority task to be scheduled in, starting from the point it has woken up.
Signature · 1 line
1.6.2.2
Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
In reply to Anton VorontsovQuoted 7 lines
This patch introduces
HAVE_NORMAL_FRAME_POINTERKconfigsymbol. When defined, the top levelMakefilewon't add -fno-omit-frame-pointer cflag (the flag is useless for PowerPC kernels, and also makes gcc generate wrong code).+++ b/arch/powerpc/Kconfig
lib/Kconfig.debug config DEBUG_NOTIFIERS · line 674
Quoted 7 lines
lib/Kconfig.debug +6
-0+ select HAVE_NORMAL_FRAME_POINTER +config HAVE_NORMAL_FRAME_POINTER + bool + help + Architectures should select this symbol if their ABI implies + having a frame pointer.
I am totally confused what you call a frame pointer here. None of the relevant PowerPC ABIs have a frame pointer separate from the stack pointer; the compiler can create one, of course. A better config symbol name and help text would help understand this patch :-)
Segher
Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
Quoted 9 lines · Segher Boessenkool
Quoted 8 lines
This patch introduces
HAVE_NORMAL_FRAME_POINTERKconfigsymbol. When defined, the top levelMakefilewon't add -fno-omit-frame-pointer cflag (the flag is useless for PowerPC kernels, and also makes gcc generate wrong code).+++ b/arch/powerpc/Kconfig
arch/powerpc/Kconfig +1
-0+ select HAVE_NORMAL_FRAME_POINTER
Quoted 11 lines
lib/Kconfig.debug config DEBUG_NOTIFIERS · line 674
Quoted 5 lines
lib/Kconfig.debug +5
-0+config HAVE_NORMAL_FRAME_POINTER + bool + help + Architectures should select this symbol if their ABI implies + having a frame pointer.I am totally confused what you call a frame pointer here. None of the relevant PowerPC ABIs have a frame pointer separate from the stack pointer; the compiler can create one, of course. A better config symbol name and help text would help understand this patch :-)
Yeah, I agree. This needs a better description. I only know what's going on because I was there for the start of the discussion.
But just to be sure, this is what I think is happening.
When we add "-pg" to gcc, it automatically causes frame pointers to be used.
But with PPC, it always has frame pointers and there's no problem.
But with Linux, when you add CONFIG_FRAME_POINTER, it automatically
adds: -fno-omit-frame-pointer. Thus the config will add "-fomit-frame-pointer" when CONFIG_FRAME_POINTER is not set, or it will add "-fno-omit-frame-pointer" when it is set.
The problem with PPC is that "-fno-omit-frame-pointer" is buggy and causes gcc to produce bad code.
Perhaps a better name would be:
HAVE_FRAME_POINTER_AS_DEFAULT
??
Or am I totally wrong in my analysis?
-- Steve
Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
Quoted 7 lines
Yeah, I agree. This needs a better description. I only know what's going on because I was there for the start of the discussion.
But just to be sure, this is what I think is happening.
When we add "-pg" to gcc, it automatically causes frame pointers to be used.
Nope, it does no such thing.
Quoted 1 line
But with PPC, it always has frame pointers and there's no problem.
Well, what do you call a "frame pointer"? In the general meaning of "some register that points to the incoming function arguments and the function local variables", PowerPC can _usually_ use GPR1, the stack pointer (and indeed it is called "stack frame pointer" in the ABI). In the more narrow meaning of "what GCC calls the frame pointer", "the thing that -fomit-frame-pointer optimises away" -- on PowerPC (and many other targets), -fomit-frame-pointer is the *default* when optimisation is enabled!
There is a GCC bug here though: it won't allow both -pg and -fomit-frame-pointer be set at the command line at the same time, even on targets where that is not problematic.
Quoted 7 lines
But with Linux, when you add
CONFIG_FRAME_POINTER, it automatically
adds: -fno-omit-frame-pointer. Thus the config will add "-fomit-frame-pointer" whenCONFIG_FRAME_POINTERis not set, or it will add "-fno-omit-frame-pointer" when it is set.The problem with PPC is that "-fno-omit-frame-pointer" is buggy and causes gcc to produce bad code.
It's a deeper problem that is only _exposed_ by -fno-o-f-p (and can be hidden by -mno-sched-epilog in the one spot where it hit us).
Quoted 3 lines
Perhaps a better name would be:
HAVE_FRAME_POINTER_AS_DEFAULT
NO_NO_OMIT_FRAME_POINTER ? Or better, just never use -fno-o-f-p, I don't see why you would ever need it.
Segher
Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
Quoted 10 lines · Segher Boessenkool
Quoted 8 lines
Yeah, I agree. This needs a better description. I only know what's going
on because I was there for the start of the discussion.But just to be sure, this is what I think is happening.
When we add "-pg" to gcc, it automatically causes frame pointers to be used.
Nope, it does no such thing.
Well, mcount is expected to be able to get to not just who called mcount, but also the parent of that function. The way mcount is implemented does not let you do that. If mcount was the first thing to be called in a function, then it would have been perfect. We could get to the caller, its parent, and even the parameters. But unfortunately, mcount is called after the stack is set up. Thus, without frame pointers (the way to find a previous frame) there's no way (on some archs) to find the parent. Nor can we figure out the parameters, which really sucks.
Quoted 28 lines
Quoted 2 lines
But with PPC, it always has frame pointers and there's no problem.
Well, what do you call a "frame pointer"? In the general meaning of "some register that points to the incoming function arguments and the function local variables", PowerPC can _usually_ use GPR1, the stack pointer (and indeed it is called "stack frame pointer" in the ABI). In the more narrow meaning of "what GCC calls the frame pointer", "the thing that -fomit-frame-pointer optimises away" -- on PowerPC (and many other targets), -fomit-frame-pointer is the *default* when optimisation is enabled!
There is a GCC bug here though: it won't allow both -pg and -fomit-frame-pointer be set at the command line at the same time, even on targets where that is not problematic.
Quoted 8 lines
But with Linux, when you add
CONFIG_FRAME_POINTER, it automatically
adds: -fno-omit-frame-pointer. Thus the config will add "-fomit-frame-pointer" whenCONFIG_FRAME_POINTERis not set, or it will
add "-fno-omit-frame-pointer" when it is set.The problem with PPC is that "-fno-omit-frame-pointer" is buggy and causes gcc to produce bad code.
It's a deeper problem that is only _exposed_ by -fno-o-f-p (and can be hidden by -mno-sched-epilog in the one spot where it hit us).
Quoted 6 lines
Quoted 3 lines
Perhaps a better name would be:
HAVE_FRAME_POINTER_AS_DEFAULT
NO_NO_OMIT_FRAME_POINTER? Or better, just never use -fno-o-f-p, I don't see why you would ever need it.
Because on x86_64 it gives better back traces. x86_64 has no way to get to the previous frames without it. There's code to use other debug metadata to get back tracing, but for uses of things like the stack tracer, we need to be able to use the actual stack frames.
As you said above, -fomit-frame-pointer is default when we optimize, and that is how the kernel is built. If we optimize on x86_64 and do not use -fno-omit-frame-pointer, the stack tracer is useless.
-- Steve
Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
When we add "-pg" to gcc, it automatically causes frame pointers to be used.
Nope, it does no such thing.
Quoted 9 lines
Well, mcount is expected to be able to get to not just who called mcount, but also the parent of that function. The way mcount is implemented does not let you do that. If mcount was the first thing to be called in a function, then it would have been perfect. We could get to the caller, its parent, and even the parameters. But unfortunately, mcount is called after the stack is set up. Thus, without frame pointers (the way to find a previous frame) there's no way (on some archs) to find the parent. Nor can we figure out the parameters, which really sucks.
Yes, and this is (supposedly) why GCC does not like seeing -pg and -fomit-frame-pointer at the same time -- because that cannot work *on some architectures*. These are the same architectures that do not enable -fomit-frame-pointer automatically at -O.
Quoted 11 lines
Quoted 2 lines
NO_NO_OMIT_FRAME_POINTER? Or better, just never use -fno-o-f-p, I don't see why you would ever need it.Because on
x86_64it gives better back traces.x86_64has no way to get to the previous frames without it. There's code to use other debug metadata to get back tracing, but for uses of things like the stack tracer, we need to be able to use the actual stack frames.As you said above, -fomit-frame-pointer is default when we optimize, and that is how the kernel is built. If we optimize on
x86_64and do not use -fno-omit-frame-pointer, the stack tracer is useless.
No. -fomit-frame-pointer is only the default when optimising on archs/ABIs where it doesn't hinder debugging and -pg and all that goodness; specifically, you do not get it by default on x86, not at any optimisation level.
Segher
Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
Quoted 37 lines · Segher Boessenkool
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
When we add "-pg" to gcc, it automatically causes frame pointers to be
used.Nope, it does no such thing.
Quoted 10 lines
Well, mcount is expected to be able to get to not just who called mcount, but also the parent of that function. The way mcount is implemented does not let you do that. If mcount was the first thing to be called in a function, then it would have been perfect. We could get to the caller, its parent, and even the parameters. But unfortunately, mcount is called after the stack is set up. Thus, without frame pointers
(the way to find a previous frame) there's no way (on some archs) to find the parent. Nor can we figure out the parameters, which really sucks.Yes, and this is (supposedly) why GCC does not like seeing -pg and -fomit-frame-pointer at the same time -- because that cannot work *on some architectures*. These are the same architectures that do not enable -fomit-frame-pointer automatically at -O.
Quoted 9 lines
Quoted 2 lines
NO_NO_OMIT_FRAME_POINTER? Or better, just never use -fno-o-f-p, I don't see why you would ever need it.Because on
x86_64it gives better back traces.x86_64has no way to get
to the previous frames without it. There's code to use other debug metadata to get back tracing, but for uses of things like the stack tracer, we need to be able to use the actual stack frames.Quoted 5 lines
As you said above, -fomit-frame-pointer is default when we optimize, and
that is how the kernel is built. If we optimize onx86_64and do not use
-fno-omit-frame-pointer, the stack tracer is useless.
Quoted 4 lines
No. -fomit-frame-pointer is only the default when optimising on archs/ABIs where it doesn't hinder debugging and -pg and all that goodness; specifically, you do not get it by default on x86, not at any optimisation level.
OK, so what's the status with this patch series?
I don't want to pull it in unless I have an ack from Sam, and now there's issues with having -fno-omit-frame-pointer. Should we add a patch instead that simply removes that?
If we eliminate the -fno-omit-frame-pointer, would that solve the PPC problem? And would it cause any other issues with other archs?
-- Steve
Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
Quoted 8 lines · Steven Rostedt
OK, so what's the status with this patch series?
I don't want to pull it in unless I have an ack from Sam, and now there's issues with having -fno-omit-frame-pointer. Should we add a patch instead that simply removes that?
If we eliminate the -fno-omit-frame-pointer, would that solve the PPC problem? And would it cause any other issues with other archs?
Well, the patch looks fine to me to be honest I'm not sure what the complaint is above...
IE. On arch that define the new HAVE_NORMAL_FRAME_POINTER (let's just call it HAVE_IMPLIED_FRAME_POINTER then :-) we just don't do -fno-omit-frame-pointer and avoid triggering the bug...
Segher, what are we missing here ?
Ben.
Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER
Quoted 1 line
Segher, what are we missing here ?
The symbol name is misleading, and the help text is incorrect.
With those things fixed, I won't oppose the patch; it would be nice if the logic here was cleaned up to avoid all the double negations, but I'm not going to do it, so I won't ask others to either.
Segher