Skip to contentSkip to conversation

ftrace: fixes for PPC

PATCH Applied Steven Rostedt 6 participants 20 messages Sep 6, 2008 – Oct 20, 2008

Subsystems (4)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover ftrace: fixes for PPC

I spent the day chasing a bug that would hang PPC on boot up when ftrace is configured in. I found that it was simply a stupid bug I did to handle the non MCOUNT_RECORD case. Since I was testing only on x86, and the MCOUNT_RECORD is automatically set for dynamic ftrace if it is available, I did not test the case where MCOUNT_RECORD was not set.

I have not finished porting MCOUNT_RECORD to PPC, but have found that it has caused some issues for archs that do not support it yet.

This patch series handles these cases.

-- Steve

author maintainer
#1 lore raw

PATCH 1/2 ftrace: use ftrace_release for all dynamic ftrace functions

ftrace_release is necessary for all uses of dynamic ftrace and not just the archs that have CONFIG_FTRACE_MCOUNT_RECORD defined.

---
 include/linux/ftrace.h |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
include/linux/ftrace.h +5 -4
Index: linux-tip.git/include/linux/ftrace.h
===================================================================
@@ -77,8 +77,10 @@ extern void mcount_call(void);
 
 extern int skip_trace(unsigned long ip);
 
-void ftrace_disable_daemon(void);
-void ftrace_enable_daemon(void);
+extern void ftrace_release(void *start, unsigned long size);
+
+extern void ftrace_disable_daemon(void);
+extern void ftrace_enable_daemon(void);
 
 #else
 # define skip_trace(ip)				({ 0; })
@@ -86,6 +88,7 @@ void ftrace_enable_daemon(void);
 # define ftrace_set_filter(buf, len, reset)	do { } while (0)
 # define ftrace_disable_daemon()		do { } while (0)
 # define ftrace_enable_daemon()			do { } while (0)
+static inline void ftrace_release(void *start, unsigned long size) { }
 #endif /* CONFIG_DYNAMIC_FTRACE */
 
 /* totally disable ftrace - can not re-enable after this */
@@ -199,12 +202,10 @@ static inline void ftrace_dump(void) { }
 #ifdef CONFIG_FTRACE_MCOUNT_RECORD
 extern void ftrace_init(void);
 extern void ftrace_init_module(unsigned long *start, unsigned long *end);
-extern void ftrace_release(void *start, unsigned long size);
 #else
 static inline void ftrace_init(void) { }
 static inline void
 ftrace_init_module(unsigned long *start, unsigned long *end) { }
-static inline void ftrace_release(void *start, unsigned long size) { }
 #endif
 
 
author maintainer
#2 lore raw

PATCH 2/2 ftrace: fix unlocking of hash

In reply to Steven Rostedt

This must be brown paper bag week for Steven Rostedt!

While working on ftrace for PPC, I discovered that the hash locking done when CONFIG_FTRACE_MCOUNT_RECORD is not set, is totally incorrect.

With a cut and paste error, I had the hash lock macro to lock for both hash_lock _and_ hash_unlock!

This bug did not affect x86 since this bug was introduced when CONFIG_FTRACE_MCOUNT_RECORD was added to x86.

---
 kernel/trace/ftrace.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/trace/ftrace.c +2 -1
Index: linux-tip.git/kernel/trace/ftrace.c
===================================================================
@@ -170,7 +170,8 @@ static int __unregister_ftrace_function(
  */
 static DEFINE_SPINLOCK(ftrace_hash_lock);
 #define ftrace_hash_lock(flags)	  spin_lock_irqsave(&ftrace_hash_lock, flags)
-#define ftrace_hash_unlock(flags) spin_lock_irqsave(&ftrace_hash_lock, flags)
+#define ftrace_hash_unlock(flags) \
+			spin_unlock_irqrestore(&ftrace_hash_lock, flags)
 #else
 /* This is protected via the ftrace_lock with MCOUNT_RECORD. */
 #define ftrace_hash_lock(flags)   do { (void)(flags); } while (0)
Applied
#3 lore raw
In reply to Steven Rostedt
Quoted 10 lines · Steven Rostedt

I spent the day chasing a bug that would hang PPC on boot up when ftrace is configured in. I found that it was simply a stupid bug I did to handle the non MCOUNT_RECORD case. Since I was testing only on x86, and the MCOUNT_RECORD is automatically set for dynamic ftrace if it is available, I did not test the case where MCOUNT_RECORD was not set.

I have not finished porting MCOUNT_RECORD to PPC, but have found that it has caused some issues for archs that do not support it yet.

This patch series handles these cases.

applied to tip/tracing/ftrace, thanks Steve.

Ingo

#4 lore raw
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Steven Rostedt

I spent the day chasing a bug that would hang PPC on boot up when ftrace is configured in. I found that it was simply a stupid bug I did to handle the non MCOUNT_RECORD case. Since I was testing only on x86, and the MCOUNT_RECORD is automatically set for dynamic ftrace if it is available, I did not test the case where MCOUNT_RECORD was not set.

Quoted 6 lines
Quoted 4 lines

I have not finished porting MCOUNT_RECORD to PPC, but have found that it has caused some issues for archs that do not support it yet.

This patch series handles these cases.

applied to tip/tracing/ftrace, thanks Steve.

Are these two merged yet? I just spent the better part of the morning trying to figure out why various Fedora kernels based on 2.6.27-rcX and 2.6.27 final hung on my G5 and finally got one booting with FTRACE disabled.

josh

author maintainer
#5 lore raw
Quoted 13 lines · Josh Boyer
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Steven Rostedt

I spent the day chasing a bug that would hang PPC on boot up when ftrace is configured in. I found that it was simply a stupid bug I did to handle the non MCOUNT_RECORD case. Since I was testing only on x86, and the MCOUNT_RECORD is automatically set for dynamic ftrace if it is available, I did not test the case where MCOUNT_RECORD was not set.

I have not finished porting MCOUNT_RECORD to PPC, but have found that it has caused some issues for archs that do not support it yet.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

This patch series handles these cases.

applied to tip/tracing/ftrace, thanks Steve.

Are these two merged yet? I just spent the better part of the morning trying to figure out why various Fedora kernels based on 2.6.27-rcX and 2.6.27 final hung on my G5 and finally got one booting with FTRACE disabled.

My powerbook does not boot with ftrace enabled. I'm currently debugging it. When I get it working, you'll see patches.

They will be to linux-tip though, and not for the current mainline (unless the code in linux-tip gets merged).

-- Steve

#6 lore raw
Quoted 20 lines · Steven Rostedt
Quoted 17 lines · Josh Boyer
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Steven Rostedt

I spent the day chasing a bug that would hang PPC on boot up when ftrace is configured in. I found that it was simply a stupid bug I did to handle the non MCOUNT_RECORD case. Since I was testing only on x86, and the MCOUNT_RECORD is automatically set for dynamic ftrace if it is available, I did not test the case where MCOUNT_RECORD was not set.

I have not finished porting MCOUNT_RECORD to PPC, but have found that it has caused some issues for archs that do not support it yet.

This patch series handles these cases.

applied to tip/tracing/ftrace, thanks Steve.

Quoted 7 lines
Quoted 4 lines

Are these two merged yet? I just spent the better part of the morning trying to figure out why various Fedora kernels based on 2.6.27-rcX and 2.6.27 final hung on my G5 and finally got one booting with FTRACE disabled.

My powerbook does not boot with ftrace enabled. I'm currently debugging it. When I get it working, you'll see patches.

Well, that's why I asked. You sent 2 patches out over a month ago that don't appear to have shown up in any Linus or PowerPC tree.

Quoted 2 lines

They will be to linux-tip though, and not for the current mainline (unless the code in linux-tip gets merged).

If linux-tip is some x86 topic branch, I think that's the wrong place for fixes that are needed on non-x86 arches.

josh

author maintainer
#7 lore raw
Quoted 6 lines · Josh Boyer
Quoted 3 lines

My powerbook does not boot with ftrace enabled. I'm currently debugging it. When I get it working, you'll see patches.

Well, that's why I asked. You sent 2 patches out over a month ago that don't appear to have shown up in any Linus or PowerPC tree.

OK, I'll see if those patches fixes my problem.

Quoted 6 lines
Quoted 3 lines

They will be to linux-tip though, and not for the current mainline (unless the code in linux-tip gets merged).

If linux-tip is some x86 topic branch, I think that's the wrong place for fixes that are needed on non-x86 arches.

linux-tip has been used by more that x86 specific changes. It's where we have been developing ftrace itself.

My original port to PPC first landed into linux-tip.

-- Steve

author maintainer
#8 lore raw
In reply to Josh Boyer
Quoted 3 lines · Josh Boyer

Well, that's why I asked. You sent 2 patches out over a month ago that don't appear to have shown up in any Linus or PowerPC tree.

Oh, the patches I sent on here, are not to solve this issue. It was actually solving issues in linux-tip itself.

I'm still looking into the cause for ftrace not to boot on PPC.

-- Steve

#9 lore raw
Quoted 8 lines · On Thu, 16 Oct 2008 14:45:15 -0400 (EDT) Steven Rostedt <[email protected]>…
Quoted 3 lines · Josh Boyer

Well, that's why I asked. You sent 2 patches out over a month ago that don't appear to have shown up in any Linus or PowerPC tree.

Oh, the patches I sent on here, are not to solve this issue. It was actually solving issues in linux-tip itself.

Totally confused as to what linux-tip is, but ok.

Quoted 1 line

I'm still looking into the cause for ftrace not to boot on PPC.

There were issues with -pg and some other compile flag on PPC at one point. I think you worked that out with Ben, but I don't recall.

Anyway, if you want a tester let me know. It seems 2.6.27.1 should be fine since FTRACE was disabled, but for .28-rc1 it would be cool if it worked :).

josh

author maintainer
#10 lore raw
Quoted 11 lines · Josh Boyer
Quoted 8 lines · Josh Boyer
Quoted 3 lines

Well, that's why I asked. You sent 2 patches out over a month ago that don't appear to have shown up in any Linus or PowerPC tree.

Oh, the patches I sent on here, are not to solve this issue. It was actually solving issues in linux-tip itself.

Totally confused as to what linux-tip is, but ok.

Quoted 8 lines
Quoted 1 line

I'm still looking into the cause for ftrace not to boot on PPC.

There were issues with -pg and some other compile flag on PPC at one point. I think you worked that out with Ben, but I don't recall.

Anyway, if you want a tester let me know. It seems 2.6.27.1 should be fine since FTRACE was disabled, but for .28-rc1 it would be cool if it worked :).

Hi Josh,

I've been looking deeper at the code for PPC. I realized that my PPC64 box that I've been testing on did not use modules. While looking at the module code it dawned on me the dynamic ftrace needs a bit of work. This is because the way modules are handled in PPC (and other architectures as well). The jmps used by mcount is a 24 bit jump. Since the modules are loaded farther than 24bits away, a trampoline is needed.

A bit of rework is needed in the ftrace infrastructure to handle the trampoline. Too much work to go into 28. I'll start working on code that can hopefully be ready and tested for 29. It's not that major of a change, but since the merge window for 28 has already been opened, we would like to get a bit more testing in before we hand it over to Linus.

-- Steve

#11 lore raw
Quoted 4 lines · Steven Rostedt
Quoted 3 lines

Anyway, if you want a tester let me know. It seems 2.6.27.1 should be fine since FTRACE was disabled, but for .28-rc1 it would be cool if it worked :).

Quoted 8 lines

Hi Josh,

I've been looking deeper at the code for PPC. I realized that my PPC64 box that I've been testing on did not use modules. While looking at the module code it dawned on me the dynamic ftrace needs a bit of work. This is because the way modules are handled in PPC (and other architectures as well). The jmps used by mcount is a 24 bit jump. Since the modules are loaded farther than 24bits away, a trampoline is needed.

Ah, indeed.

Quoted 5 lines

A bit of rework is needed in the ftrace infrastructure to handle the trampoline. Too much work to go into 28. I'll start working on code that can hopefully be ready and tested for 29. It's not that major of a change, but since the merge window for 28 has already been opened, we would like to get a bit more testing in before we hand it over to Linus.

That seems like a good plan. Should we mark dynamic ftrace as X86 only in Kconfig for .28 to prevent people from inadvertently setting it?

josh

author maintainer
#12 lore raw
Quoted 16 lines · Josh Boyer
Quoted 4 lines · Steven Rostedt
Quoted 3 lines

Anyway, if you want a tester let me know. It seems 2.6.27.1 should be fine since FTRACE was disabled, but for .28-rc1 it would be cool if it worked :).

Quoted 8 lines

Hi Josh,

I've been looking deeper at the code for PPC. I realized that my PPC64 box that I've been testing on did not use modules. While looking at the module code it dawned on me the dynamic ftrace needs a bit of work. This is because the way modules are handled in PPC (and other architectures as well). The jmps used by mcount is a 24 bit jump. Since the modules are loaded farther than 24bits away, a trampoline is needed.

Ah, indeed.

Quoted 8 lines
Quoted 5 lines

A bit of rework is needed in the ftrace infrastructure to handle the trampoline. Too much work to go into 28. I'll start working on code that can hopefully be ready and tested for 29. It's not that major of a change, but since the merge window for 28 has already been opened, we would like to get a bit more testing in before we hand it over to Linus.

That seems like a good plan. Should we mark dynamic ftrace as X86 only in Kconfig for .28 to prevent people from inadvertently setting it?

Well, we probably should disable it for PPC at least. I don't know if sparc has the same issue.

David?

-- Steve

#13 lore raw
In reply to Josh Boyer

2008/10/20 Josh Boyer <[email protected]>:

Quoted 2 lines

That seems like a good plan. Should we mark dynamic ftrace as X86 only in Kconfig for .28 to prevent people from inadvertently setting it?

Hi Josh.

There are other arch that support ftrace as well like sh or sparc64 (I'm currently working on
an implementation for mips).
So the choice would be better between waiting for a fix or disable dynamic ftrace
on Kconfig only for PowerPC.

author maintainer
#14 lore raw
Quoted 6 lines · Fr?d?ric Weisbecker

There are other arch that support ftrace as well like sh or sparc64 (I'm currently working on
an implementation for mips).
So the choice would be better between waiting for a fix or disable dynamic ftrace
on Kconfig only for PowerPC.

Hi Frederic,

I believe MIPS has the same issues as PPC. Doesn't it use a trampoline too? I want to make the generic code handle trampolines a bit better. This is one of the problems with developing in x86. It avoids a lot of the issues that other archs might have.

-- Steve

#15 lore raw

2008/10/20 Steven Rostedt <[email protected]>:

Quoted 2 lines

I believe MIPS has the same issues as PPC. Doesn't it use a trampoline too?

No it doesn't seem to. If I'm not wrong, MIPS uses only the elf relocation table to relocate
its addresses.

Quoted 1 line

I want to make the generic code handle trampolines a bit better.

Do you think there could be a generic approach to handles these trampolines?

author maintainer
#16 lore raw

PATCH 1/1 ftrace: powerpc: remove startup functions from tracing

In reply to Josh Boyer

[
Josh, can you see if this allows you to boot on PowerPC? It worked on my powerbook.
]

The early init code in PowerPC is not mapped to their final locations and all jumps and memory references must be done with relative jumps and accesses.

The lib files in the powerpc directory are called in early boot, and since mcount will perform direct access to memory, the lib files need not be traced.

---
 arch/powerpc/lib/Makefile |    5 +++++
 1 file changed, 5 insertions(+)
arch/powerpc/lib/Makefile +5 -0
Index: linux-compile.git/arch/powerpc/lib/Makefile
===================================================================
@@ -2,6 +2,11 @@
 # Makefile for ppc-specific library files..
 #
 
+ifdef CONFIG_FTRACE
+ORIG_CFLAGS := $(KBUILD_CFLAGS)
+KBUILD_CFLAGS = $(subst -pg,,$(ORIG_CFLAGS))
+endif
+
 ifeq ($(CONFIG_PPC64),y)
 EXTRA_CFLAGS		+= -mno-minimal-toc
 endif
#17 lore raw

Re: [PATCH] ftrace: powerpc: remove startup functions from tracing

Quoted 8 lines · Steven Rostedt

The early init code in PowerPC is not mapped to their final locations and all jumps and memory references must be done with relative jumps and accesses.

The lib files in the powerpc directory are called in early boot, and since mcount will perform direct access to memory, the lib files need not be traced.

This is annoying though, because that means things like memcpy, copy_to_from_user etc... can't be traced. On the other hand a lot of that is asm and already doesn't call mcount.

Cheers,
Ben.

author maintainer
#18 lore raw

Re: [PATCH] ftrace: powerpc: remove startup functions from tracing

Quoted 6 lines · Benjamin Herrenschmidt
Quoted 5 lines · Steven Rostedt

The early init code in PowerPC is not mapped to their final locations and all jumps and memory references must be done with relative jumps and accesses.

Quoted 7 lines
Quoted 3 lines

The lib files in the powerpc directory are called in early boot, and since mcount will perform direct access to memory, the lib files need not be traced.

This is annoying though, because that means things like memcpy, copy_to_from_user etc... can't be traced. On the other hand a lot of that is asm and already doesn't call mcount.

Yeah, I know. I was going to pick and choose which files in there should be converted, but then I saw that they were mostly asm, and it seemed to be better safe than sorry.

Sure, we could probably bring it down a bit. But I'm a bit paranoid it may cause someone else not to boot because of something else being called at early boot up.

But don't worry. When we get MCOUNT_RECORD ported to PPC this no longer becomes an issue, and we can simply do a
"ifndef CONFIG_FTRACE_MCOUNT_RECORD" around the -pg removal in the Makefile. (or what ever the Makefile syntax is for config options)

-- Steve

#19 lore raw
In reply to Josh Boyer
Quoted 4 lines · Josh Boyer

Are these two merged yet? I just spent the better part of the morning trying to figure out why various Fedora kernels based on 2.6.27-rcX and 2.6.27 final hung on my G5 and finally got one booting with FTRACE disabled.

Until recently I could boot my G5 with FTRACE enabled, but it totally screwed up load balancing.

As of a couple days ago, -tip hung on boot due to the following genirq issue:

kernel/irq/chip.c +1 -1
index 570d1ea..1c178ae 100644
@@ -132,7 +132,7 @@ int set_irq_type(unsigned int irq, unsigned int type)
   		return 0;
 
   	spin_lock_irqsave(&desc->lock, flags);
-	ret = __irq_set_trigger(desc, irq, flags);
+	ret = __irq_set_trigger(desc, irq, type);
   	spin_unlock_irqrestore(&desc->lock, flags);
   	return ret;
   }

Chris