Skip to contentSkip to conversation

Tracers vs. CALLER_ADDR on PowerPC

PATCH In review Anton Vorontsov 4 participants 12 messages May 2, 2009 – May 18, 2009

Subsystems (5)
PowerPC Kbuild Library code Tracing +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v5 cover Tracers vs. CALLER_ADDR on PowerPC

Hi all,

In v5 there are few fixes suggested by Sam Ravnborg:

- Use 8 spaces for indentation in Makefile;
- Rename ARCH_HAS_NORMAL_FRAME_POINTERS to HAVE_NORMAL_FRAME_POINTER;

Thanks,

---

Preface for those who don't know or forgot what the problem is:

Gcc frame pointers do nothing useful on PowerPC (they're harmful, actually), and thus lib/Kconfig.debug makes CONFIG_FRAME_POINTER unselectable on PPC targets, but CALLER_ADDR macros are available only with CONFIG_FRAME_POINTER, therefore tracing is completely useless on PowerPC:

[...]

  <idle>-0       0X.h3    2us+:      0:140:R   + [000]  1733:120:S mvtsd
  <idle>-0       0X.h3    9us+: 0 (0)
  <idle>-0       0X..3   72us : 0 (0)
  <idle>-0       0X..3   73us :      0:140:R ==> [000]  1733:120:R mvtsd

While it should look like this:

[...]

  <idle>-0       0X.h3    2us+:      0:140:R   + [000]  1740:120:S mvtsd
  <idle>-0       0X.h3    9us+: hrtimer_wakeup (__run_hrtimer)
  <idle>-0       0X..3   87us : cpu_idle (__got2_end)
  <idle>-0       0X..3   89us :      0:140:R ==> [000]  1740:120:R mvtsd

I've tried to fix the issue via expanding the #ifdef in the ftrace.h:
http://lkml.org/lkml/2009/1/31/141

Then Steven Rostedt suggested to implement something more generic, i.e. HAVE_NORMAL_FRAME_POINTERS Kconfig symbol.

I found a way to solve the problem w/o additional symbols, but with some Makefile magic (http://lkml.org/lkml/2009/2/4/273).
But because of top-level Makefile issues on other arches
(http://lkml.org/lkml/2009/2/14/89) I had to abandon the approach.

So, this patch set combines Steven Rostedt's idea and a small Makefile change, so that now only top-level Makefile has to know about the new symbol, and the rest of the kernel can stay with using CONFIG_FRAME_POINTER.

I'm aware of

commit c79a61f55773d2519fd0525bf58385f7d20752d3
Author: Uwe Kleine-Koenig <[email protected]>
Date: Fri Feb 27 21:30:03 2009 +0100

tracing: make CALLER_ADDRx overwriteable

But I think the patch set is still applicable, considering that it removes gcc bug workaround in a nice way, and makes CONFIG_FRAME_POINTER available on PowerPC, thus other code can rely on that.

If not, I can just fill-in the asm/ftrace.h for PowerPC.

Signature · 3 lines
Anton Vorontsov
email: [email protected]
irc://irc.freenode.net/bd2
#1 lore raw

PATCH v5 1/3 powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

This patch introduces HAVE_NORMAL_FRAME_POINTER Kconfig symbol. When defined, the top level Makefile won't add -fno-omit-frame-pointer cflag (the flag is useless for PowerPC kernels, and also makes gcc generate wrong code).

Also move ARCH_WANT_FRAME_POINTERS's help text.

---
 Makefile             |    7 +++++--
 arch/powerpc/Kconfig |    1 +
 lib/Kconfig.debug    |   16 ++++++++++------
 3 files changed, 16 insertions(+), 8 deletions(-)
Makefile +5 -2
index b18afad..bfa58ea 100644
@@ -543,9 +543,12 @@ KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
 endif
 
 ifdef CONFIG_FRAME_POINTER
-KBUILD_CFLAGS	+= -fno-omit-frame-pointer -fno-optimize-sibling-calls
+        KBUILD_CFLAGS += -fno-optimize-sibling-calls
+        ifndef CONFIG_HAVE_NORMAL_FRAME_POINTER
+                KBUILD_CFLAGS += -fno-omit-frame-pointer
+        endif
 else
-KBUILD_CFLAGS	+= -fomit-frame-pointer
+        KBUILD_CFLAGS += -fomit-frame-pointer
 endif
 
 ifdef CONFIG_DEBUG_INFO
arch/powerpc/Kconfig +1 -0
index 4c78045..b29a6c3 100644
@@ -119,6 +119,7 @@ config PPC
 	select HAVE_KPROBES
 	select HAVE_ARCH_KGDB
 	select HAVE_KRETPROBES
+	select HAVE_NORMAL_FRAME_POINTER
 	select HAVE_ARCH_TRACEHOOK
 	select HAVE_LMB
 	select HAVE_DMA_ATTRS if PPC64
lib/Kconfig.debug +10 -6
index 444002b..1f1633b 100644
@@ -660,20 +660,24 @@ config DEBUG_NOTIFIERS
 	  This is a relatively cheap check but if you care about maximum
 	  performance, say N.
 
-#
-# Select this config option from the architecture Kconfig, if it
-# it is preferred to always offer frame pointers as a config
-# option on the architecture (regardless of KERNEL_DEBUG):
-#
 config ARCH_WANT_FRAME_POINTERS
 	bool
 	help
+	  Select this config option from the architecture Kconfig, if it
+	  it is preferred to always offer frame pointers as a config
+	  option on the architecture (regardless of KERNEL_DEBUG).
+
+config HAVE_NORMAL_FRAME_POINTER
+	bool
+	help
+	  Architectures should select this symbol if their ABI implies
+	  having a frame pointer.
 
 config FRAME_POINTER
 	bool "Compile the kernel with frame pointers"
 	depends on DEBUG_KERNEL && \
 		(CRIS || M68K || M68KNOMMU || FRV || UML || \
-		 AVR32 || SUPERH || BLACKFIN || MN10300) || \
+		 AVR32 || SUPERH || BLACKFIN || MN10300 || PPC) || \
 		ARCH_WANT_FRAME_POINTERS
 	default y if (DEBUG_INFO && UML) || ARCH_WANT_FRAME_POINTERS
 	help
Signature · 1 line
1.6.2.2
#2 lore raw

Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

Quoted 7 lines

This patch introduces HAVE_NORMAL_FRAME_POINTER Kconfig symbol. When defined, the top level Makefile won't add -fno-omit-frame-pointer cflag (the flag is useless for PowerPC kernels, and also makes gcc generate wrong code).

+++ b/arch/powerpc/Kconfig

lib/Kconfig.debug config DEBUG_NOTIFIERS · line 674

Quoted 7 lines
lib/Kconfig.debug +6 -0
+	select HAVE_NORMAL_FRAME_POINTER
 
+config HAVE_NORMAL_FRAME_POINTER
+	bool
+	help
+	  Architectures should select this symbol if their ABI implies
+	  having a frame pointer.

I am totally confused what you call a frame pointer here. None of the relevant PowerPC ABIs have a frame pointer separate from the stack pointer; the compiler can create one, of course. A better config symbol name and help text would help understand this patch :-)

Segher

#3 lore raw

Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

Quoted 9 lines · Segher Boessenkool
Quoted 8 lines

This patch introduces HAVE_NORMAL_FRAME_POINTER Kconfig symbol. When defined, the top level Makefile won't add -fno-omit-frame-pointer cflag (the flag is useless for PowerPC kernels, and also makes gcc generate wrong code).

+++ b/arch/powerpc/Kconfig

arch/powerpc/Kconfig +1 -0
+	select HAVE_NORMAL_FRAME_POINTER
Quoted 11 lines

lib/Kconfig.debug config DEBUG_NOTIFIERS · line 674

Quoted 5 lines
lib/Kconfig.debug +5 -0
+config HAVE_NORMAL_FRAME_POINTER
+	bool
+	help
+	  Architectures should select this symbol if their ABI implies
+	  having a frame pointer.

I am totally confused what you call a frame pointer here. None of the relevant PowerPC ABIs have a frame pointer separate from the stack pointer; the compiler can create one, of course. A better config symbol name and help text would help understand this patch :-)

Yeah, I agree. This needs a better description. I only know what's going on because I was there for the start of the discussion.

But just to be sure, this is what I think is happening.

When we add "-pg" to gcc, it automatically causes frame pointers to be used.

But with PPC, it always has frame pointers and there's no problem.

But with Linux, when you add CONFIG_FRAME_POINTER, it automatically
adds: -fno-omit-frame-pointer. Thus the config will add "-fomit-frame-pointer" when CONFIG_FRAME_POINTER is not set, or it will add "-fno-omit-frame-pointer" when it is set.

The problem with PPC is that "-fno-omit-frame-pointer" is buggy and causes gcc to produce bad code.

Perhaps a better name would be:

HAVE_FRAME_POINTER_AS_DEFAULT

??

Or am I totally wrong in my analysis?

-- Steve

#4 lore raw

Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

Quoted 7 lines

Yeah, I agree. This needs a better description. I only know what's going on because I was there for the start of the discussion.

But just to be sure, this is what I think is happening.

When we add "-pg" to gcc, it automatically causes frame pointers to be used.

Nope, it does no such thing.

Quoted 1 line

But with PPC, it always has frame pointers and there's no problem.

Well, what do you call a "frame pointer"? In the general meaning of "some register that points to the incoming function arguments and the function local variables", PowerPC can _usually_ use GPR1, the stack pointer (and indeed it is called "stack frame pointer" in the ABI). In the more narrow meaning of "what GCC calls the frame pointer", "the thing that -fomit-frame-pointer optimises away" -- on PowerPC (and many other targets), -fomit-frame-pointer is the *default* when optimisation is enabled!

There is a GCC bug here though: it won't allow both -pg and -fomit-frame-pointer be set at the command line at the same time, even on targets where that is not problematic.

Quoted 7 lines

But with Linux, when you add CONFIG_FRAME_POINTER, it automatically
adds: -fno-omit-frame-pointer. Thus the config will add "-fomit-frame-pointer" when CONFIG_FRAME_POINTER is not set, or it will add "-fno-omit-frame-pointer" when it is set.

The problem with PPC is that "-fno-omit-frame-pointer" is buggy and causes gcc to produce bad code.

It's a deeper problem that is only _exposed_ by -fno-o-f-p (and can be hidden by -mno-sched-epilog in the one spot where it hit us).

Quoted 3 lines

Perhaps a better name would be:

HAVE_FRAME_POINTER_AS_DEFAULT

NO_NO_OMIT_FRAME_POINTER ? Or better, just never use -fno-o-f-p, I don't see why you would ever need it.

Segher

#5 lore raw

Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

Quoted 10 lines · Segher Boessenkool
Quoted 8 lines

Yeah, I agree. This needs a better description. I only know what's going
on because I was there for the start of the discussion.

But just to be sure, this is what I think is happening.

When we add "-pg" to gcc, it automatically causes frame pointers to be used.

Nope, it does no such thing.

Well, mcount is expected to be able to get to not just who called mcount, but also the parent of that function. The way mcount is implemented does not let you do that. If mcount was the first thing to be called in a function, then it would have been perfect. We could get to the caller, its parent, and even the parameters. But unfortunately, mcount is called after the stack is set up. Thus, without frame pointers (the way to find a previous frame) there's no way (on some archs) to find the parent. Nor can we figure out the parameters, which really sucks.

Quoted 28 lines
Quoted 2 lines

But with PPC, it always has frame pointers and there's no problem.

Well, what do you call a "frame pointer"? In the general meaning of "some register that points to the incoming function arguments and the function local variables", PowerPC can _usually_ use GPR1, the stack pointer (and indeed it is called "stack frame pointer" in the ABI). In the more narrow meaning of "what GCC calls the frame pointer", "the thing that -fomit-frame-pointer optimises away" -- on PowerPC (and many other targets), -fomit-frame-pointer is the *default* when optimisation is enabled!

There is a GCC bug here though: it won't allow both -pg and -fomit-frame-pointer be set at the command line at the same time, even on targets where that is not problematic.

Quoted 8 lines

But with Linux, when you add CONFIG_FRAME_POINTER, it automatically
adds: -fno-omit-frame-pointer. Thus the config will add "-fomit-frame-pointer" when CONFIG_FRAME_POINTER is not set, or it will
add "-fno-omit-frame-pointer" when it is set.

The problem with PPC is that "-fno-omit-frame-pointer" is buggy and causes gcc to produce bad code.

It's a deeper problem that is only _exposed_ by -fno-o-f-p (and can be hidden by -mno-sched-epilog in the one spot where it hit us).

Quoted 6 lines
Quoted 3 lines

Perhaps a better name would be:

HAVE_FRAME_POINTER_AS_DEFAULT

NO_NO_OMIT_FRAME_POINTER ? Or better, just never use -fno-o-f-p, I don't see why you would ever need it.

Because on x86_64 it gives better back traces. x86_64 has no way to get to the previous frames without it. There's code to use other debug metadata to get back tracing, but for uses of things like the stack tracer, we need to be able to use the actual stack frames.

As you said above, -fomit-frame-pointer is default when we optimize, and that is how the kernel is built. If we optimize on x86_64 and do not use -fno-omit-frame-pointer, the stack tracer is useless.

-- Steve

#6 lore raw

Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

When we add "-pg" to gcc, it automatically causes frame pointers to be used.

Nope, it does no such thing.

Quoted 9 lines

Well, mcount is expected to be able to get to not just who called mcount, but also the parent of that function. The way mcount is implemented does not let you do that. If mcount was the first thing to be called in a function, then it would have been perfect. We could get to the caller, its parent, and even the parameters. But unfortunately, mcount is called after the stack is set up. Thus, without frame pointers (the way to find a previous frame) there's no way (on some archs) to find the parent. Nor can we figure out the parameters, which really sucks.

Yes, and this is (supposedly) why GCC does not like seeing -pg and -fomit-frame-pointer at the same time -- because that cannot work *on some architectures*. These are the same architectures that do not enable -fomit-frame-pointer automatically at -O.

Quoted 11 lines
Quoted 2 lines

NO_NO_OMIT_FRAME_POINTER ? Or better, just never use -fno-o-f-p, I don't see why you would ever need it.

Because on x86_64 it gives better back traces. x86_64 has no way to get to the previous frames without it. There's code to use other debug metadata to get back tracing, but for uses of things like the stack tracer, we need to be able to use the actual stack frames.

As you said above, -fomit-frame-pointer is default when we optimize, and that is how the kernel is built. If we optimize on x86_64 and do not use -fno-omit-frame-pointer, the stack tracer is useless.

No. -fomit-frame-pointer is only the default when optimising on archs/ABIs where it doesn't hinder debugging and -pg and all that goodness; specifically, you do not get it by default on x86, not at any optimisation level.

Segher

#7 lore raw

Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

Quoted 37 lines · Segher Boessenkool
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

When we add "-pg" to gcc, it automatically causes frame pointers to be
used.

Nope, it does no such thing.

Quoted 10 lines

Well, mcount is expected to be able to get to not just who called mcount, but also the parent of that function. The way mcount is implemented does not let you do that. If mcount was the first thing to be called in a function, then it would have been perfect. We could get to the caller, its parent, and even the parameters. But unfortunately, mcount is called after the stack is set up. Thus, without frame pointers
(the way to find a previous frame) there's no way (on some archs) to find the parent. Nor can we figure out the parameters, which really sucks.

Yes, and this is (supposedly) why GCC does not like seeing -pg and -fomit-frame-pointer at the same time -- because that cannot work *on some architectures*. These are the same architectures that do not enable -fomit-frame-pointer automatically at -O.

Quoted 9 lines
Quoted 2 lines

NO_NO_OMIT_FRAME_POINTER ? Or better, just never use -fno-o-f-p, I don't see why you would ever need it.

Because on x86_64 it gives better back traces. x86_64 has no way to get
to the previous frames without it. There's code to use other debug metadata to get back tracing, but for uses of things like the stack tracer, we need to be able to use the actual stack frames.

Quoted 5 lines

As you said above, -fomit-frame-pointer is default when we optimize, and
that is how the kernel is built. If we optimize on x86_64 and do not use
-fno-omit-frame-pointer, the stack tracer is useless.

Quoted 4 lines

No. -fomit-frame-pointer is only the default when optimising on archs/ABIs where it doesn't hinder debugging and -pg and all that goodness; specifically, you do not get it by default on x86, not at any optimisation level.

OK, so what's the status with this patch series?

I don't want to pull it in unless I have an ack from Sam, and now there's issues with having -fno-omit-frame-pointer. Should we add a patch instead that simply removes that?

If we eliminate the -fno-omit-frame-pointer, would that solve the PPC problem? And would it cause any other issues with other archs?

-- Steve

#8 lore raw

Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

Quoted 8 lines · Steven Rostedt

OK, so what's the status with this patch series?

I don't want to pull it in unless I have an ack from Sam, and now there's issues with having -fno-omit-frame-pointer. Should we add a patch instead that simply removes that?

If we eliminate the -fno-omit-frame-pointer, would that solve the PPC problem? And would it cause any other issues with other archs?

Well, the patch looks fine to me to be honest I'm not sure what the complaint is above...

IE. On arch that define the new HAVE_NORMAL_FRAME_POINTER (let's just call it HAVE_IMPLIED_FRAME_POINTER then :-) we just don't do -fno-omit-frame-pointer and avoid triggering the bug...

Segher, what are we missing here ?

Ben.

#9 lore raw

Re: [PATCH 1/3] powerpc, Makefile: Make it possible to safely select CONFIG_FRAME_POINTER

Quoted 1 line

Segher, what are we missing here ?

The symbol name is misleading, and the help text is incorrect.

With those things fixed, I won't oppose the patch; it would be nice if the logic here was cleaned up to avoid all the double negations, but I'm not going to do it, so I won't ask others to either.

Segher

#10 lore raw

PATCH v5 2/3 powerpc: Remove -fno-omit-frame-pointer workarounds

In reply to Anton Vorontsov

The workarounds aren't needed any longer since the top level Makefile doesn't pass -fno-omit-frame-pointer cflag for PowerPC builds.

---
 arch/powerpc/Makefile                    |    5 -----
 arch/powerpc/kernel/Makefile             |   12 ++++++------
 arch/powerpc/platforms/powermac/Makefile |    2 +-
 lib/Kconfig.debug                        |    6 +++---
 4 files changed, 10 insertions(+), 15 deletions(-)
arch/powerpc/Makefile +0 -5
index 551fc58..1dd7748 100644
@@ -120,11 +120,6 @@ ifeq ($(CONFIG_6xx),y)
 KBUILD_CFLAGS		+= -mcpu=powerpc
 endif
 
-# Work around a gcc code-gen bug with -fno-omit-frame-pointer.
-ifeq ($(CONFIG_FUNCTION_TRACER),y)
-KBUILD_CFLAGS		+= -mno-sched-epilog
-endif
-
 cpu-as-$(CONFIG_4xx)		+= -Wa,-m405
 cpu-as-$(CONFIG_6xx)		+= -Wa,-maltivec
 cpu-as-$(CONFIG_POWER4)		+= -Wa,-maltivec
arch/powerpc/kernel/Makefile +6 -6
index 71901fb..fdbca81 100644
@@ -14,14 +14,14 @@ endif
 
 ifdef CONFIG_FUNCTION_TRACER
 # Do not trace early boot code
-CFLAGS_REMOVE_cputable.o = -pg -mno-sched-epilog
-CFLAGS_REMOVE_prom_init.o = -pg -mno-sched-epilog
-CFLAGS_REMOVE_btext.o = -pg -mno-sched-epilog
-CFLAGS_REMOVE_prom.o = -pg -mno-sched-epilog
+CFLAGS_REMOVE_cputable.o = -pg
+CFLAGS_REMOVE_prom_init.o = -pg
+CFLAGS_REMOVE_btext.o = -pg
+CFLAGS_REMOVE_prom.o = -pg
 # do not trace tracer code
-CFLAGS_REMOVE_ftrace.o = -pg -mno-sched-epilog
+CFLAGS_REMOVE_ftrace.o = -pg
 # timers used by tracing
-CFLAGS_REMOVE_time.o = -pg -mno-sched-epilog
+CFLAGS_REMOVE_time.o = -pg
 endif
 
 obj-y				:= cputable.o ptrace.o syscalls.o \
arch/powerpc/platforms/powermac/Makefile +1 -1
index 50f1693..0eb8781 100644
@@ -2,7 +2,7 @@ CFLAGS_bootx_init.o  		+= -fPIC
 
 ifdef CONFIG_FUNCTION_TRACER
 # Do not trace early boot code
-CFLAGS_REMOVE_bootx_init.o = -pg -mno-sched-epilog
+CFLAGS_REMOVE_bootx_init.o = -pg
 endif
 
 obj-y				+= pic.o setup.o time.o feature.o pci.o \
lib/Kconfig.debug +3 -3
index 1f1633b..cffbfc4 100644
@@ -492,7 +492,7 @@ config LOCKDEP
 	bool
 	depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
 	select STACKTRACE
-	select FRAME_POINTER if !X86 && !MIPS && !PPC && !ARM_UNWIND && !S390
+	select FRAME_POINTER if !X86 && !MIPS && !ARM_UNWIND && !S390
 	select KALLSYMS
 	select KALLSYMS_ALL
 
@@ -865,13 +865,13 @@ config FAULT_INJECTION_STACKTRACE_FILTER
 	depends on FAULT_INJECTION_DEBUG_FS && STACKTRACE_SUPPORT
 	depends on !X86_64
 	select STACKTRACE
-	select FRAME_POINTER if !PPC && !S390
+	select FRAME_POINTER if !S390
 	help
 	  Provide stacktrace filter for fault-injection capabilities
 
 config LATENCYTOP
 	bool "Latency measuring infrastructure"
-	select FRAME_POINTER if !MIPS && !PPC && !S390
+	select FRAME_POINTER if !MIPS && !S390
 	select KALLSYMS
 	select KALLSYMS_ALL
 	select STACKTRACE
Signature · 1 line
1.6.2.2
#11 lore raw

PATCH v5 3/3 tracing: Tracers that use CALLER_ADDR macros should select FRAME_POINTER

In reply to Anton Vorontsov

Irqsoff, switch and preempt tracers use CALLER_ADDR macros, so they should select FRAME_POINTER. Otherwise traces are meaningless.

---
 kernel/trace/Kconfig |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/trace/Kconfig +3 -0
index 450d3c2..ad44791 100644
@@ -122,6 +122,7 @@ config IRQSOFF_TRACER
 	select TRACE_IRQFLAGS
 	select TRACING
 	select TRACER_MAX_TRACE
+	select FRAME_POINTER
 	help
 	  This option measures the time spent in irqs-off critical
 	  sections, with microsecond accuracy.
@@ -143,6 +144,7 @@ config PREEMPT_TRACER
 	depends on PREEMPT
 	select TRACING
 	select TRACER_MAX_TRACE
+	select FRAME_POINTER
 	help
 	  This option measures the time spent in preemption off critical
 	  sections, with microsecond accuracy.
@@ -171,6 +173,7 @@ config SCHED_TRACER
 	select TRACING
 	select CONTEXT_SWITCH_TRACER
 	select TRACER_MAX_TRACE
+	select FRAME_POINTER
 	help
 	  This tracer tracks the latency of the highest priority task
 	  to be scheduled in, starting from the point it has woken up.
Signature · 1 line
1.6.2.2