Skip to contentSkip to conversation

microblaze: Support FRAME_POINTER for better backtrace

RFC In review Steven J. Magnani 3 participants 8 messages Feb 25, 2010 – Mar 11, 2010 Microblaze architecture

lore
Share & tools
lore
#0 lore raw

RFC 1/1 microblaze: Support FRAME_POINTER for better backtrace

Add a FRAME_POINTER option and when it is enabled, use frame pointers to walk the stack during a backtrace dump. This eliminates printout of confusing "function calls" corresponding to stack values that look like they might be return addresses, but aren't.

This patch is dependent upon
[PATCH] microblaze: Begin stack dump with caller of dump_stack()

I'm not certain whether the MMU compiler generates frame pointers the same way as the noMMU compiler I am using. I'm also not sure what all the ramifications of providing FRAME_POINTER are. It looks like tracing functionality makes use of it. Need someone familiar with these areas to comment on the patch.

---

arch/microblaze/Kconfig.debug +7 -0
diff -uprN a/arch/microblaze/Kconfig.debug b/arch/microblaze/Kconfig.debug
@@ -26,4 +26,11 @@ config DEBUG_BOOTMEM
 	depends on DEBUG_KERNEL
 	bool "Debug BOOTMEM initialization"
 
+config FRAME_POINTER
+	bool "Use frame pointers"
+	default n
+	help
+	  If you say N here, the resulting kernel will be slightly smaller and
+	  faster. However, stack dumps will be much harder to interpret.
+
 endmenu
arch/microblaze/kernel/traps.c +9 -1
diff -uprN a/arch/microblaze/kernel/traps.c b/arch/microblaze/kernel/traps.c
@@ -8,6 +8,7 @@
  * for more details.
  */
 
+#include <generated/autoconf.h>
 #include <linux/kernel.h>
 #include <linux/kallsyms.h>
 #include <linux/module.h>
@@ -44,7 +45,7 @@ void show_trace(struct task_struct *task
 	printk(KERN_NOTICE "\n");
 #endif
 	while (!kstack_end(stack)) {
-		addr = *stack++;
+		addr = *stack;
 		/*
 		 * If the address is either in the text segment of the
 		 * kernel, or in the region which contains vmalloc'ed
@@ -55,6 +56,13 @@ void show_trace(struct task_struct *task
 		 */
 		if (kernel_text_address(addr))
 			print_ip_sym(addr);
+
+#if defined(CONFIG_FRAME_POINTER)
+		/* Fetch the caller's frame pointer */
+		stack = (unsigned long *) stack[7];
+#else
+		stack++;
+#endif
 	}
 	printk(KERN_NOTICE "\n");
 
maintainer
#1 lore raw
Quoted 7 lines · Steven J. Magnani

Add a FRAME_POINTER option and when it is enabled, use frame pointers to walk the stack during a backtrace dump. This eliminates printout of confusing "function calls" corresponding to stack values that look like they might be return addresses, but aren't.

This patch is dependent upon
[PATCH] microblaze: Begin stack dump with caller of dump_stack()

yes.

Quoted 4 lines

I'm not certain whether the MMU compiler generates frame pointers the same
way as the noMMU compiler I am using. I'm also not sure what all the ramifications of providing FRAME_POINTER are. It looks like tracing functionality makes use of it. Need someone familiar with these areas to comment on the patch.

Firstly I was surprise that you create any frame pointer solution but
1. It is not frame pointer because Microblaze not use it
2. it is just one optimization which could help but IMHO not. Your patch expects that every stack frame size has 7/8 (doesn't matter right now) items but that's not correct expectation. (Do objdump from vmlinux and look at cpu_idle, prom_add_property and others) - that's why I think that your patch won't work.
3. The next question is, if we can expect that every function record has at least 7/8 items. If yes than look at my function below.
4. One more thing is that function still use kernel_text_address() which is silly because we are still not sure if the address there is correct or not. It is just checking and if we are using, it is just mean that there is any expectation which is not correct.

I did some testing several months/weeks ago and I tried to solve it in the same way as you. But found that it is based on any expectation which
is not correct.

I know that stack tracing is pain but I am not convinced that this is way how to solve it. :-(

Michal

Quoted 7 lines

---
diff -uprN a/arch/microblaze/Kconfig.debug b/arch/microblaze/Kconfig.debug

arch/microblaze/Kconfig.debug config DEBUG_BOOTMEM · line 35

Quoted 10 lines
arch/microblaze/Kconfig.debug +7 -0
@@ -26,4 +26,11 @@ config DEBUG_BOOTMEM
 	depends on DEBUG_KERNEL
 	bool "Debug BOOTMEM initialization"
 
+config FRAME_POINTER
+	bool "Use frame pointers"
+	default n
+	help
+	  If you say N here, the resulting kernel will be slightly smaller and
+	  faster. However, stack dumps will be much harder to interpret.
+

depends on !MMU

arch/microblaze/kernel/traps.c line 11

Quoted 9 lines
arch/microblaze/Kconfig.debug +0 -0
 endmenu

diff -uprN a/arch/microblaze/kernel/traps.c b/arch/microblaze/kernel/traps.c

arch/microblaze/kernel/traps.c +1 -0
@@ -8,6 +8,7 @@
  * for more details.
  */
 
+#include <generated/autoconf.h>

why? I don't think that this is necessary.

Quoted 13 lines
arch/microblaze/kernel/traps.c +1 -1
 #include <linux/kernel.h>
 #include <linux/kallsyms.h>
 #include <linux/module.h>
@@ -44,7 +45,7 @@ void show_trace(struct task_struct *task
 	printk(KERN_NOTICE "\n");
 #endif
 	while (!kstack_end(stack)) {
-		addr = *stack++;
+		addr = *stack;
 		/*
 		 * If the address is either in the text segment of the
 		 * kernel, or in the region which contains vmalloc'ed

arch/microblaze/kernel/traps.c void show_trace(struct task_struct *task · line 62

Quoted 7 lines
arch/microblaze/kernel/traps.c +4 -0
@@ -55,6 +56,13 @@ void show_trace(struct task_struct *task
 		 */
 		if (kernel_text_address(addr))
 			print_ip_sym(addr);
+
+#if defined(CONFIG_FRAME_POINTER)
+		/* Fetch the caller's frame pointer */
+		stack = (unsigned long *) stack[7];

If is calculation correct then some comments, why you use number 7, will be necessary.

arch/microblaze/kernel/traps.c void show_trace(struct task_struct *task · line 68

Quoted 7 lines
arch/microblaze/kernel/traps.c +3 -0
+#else
+		stack++;
+#endif
 	}
 	printk(KERN_NOTICE "\n");
 

Look at this code which should be better than yours.

if (kernel_text_address(addr)) {

  			print_ip_sym(addr);
			/* Fetch the caller's frame pointer */
			stack = (unsigned long *) stack[7];
		}
		stack++;
Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#2 lore raw

Michal,

Quoted 2 lines · Michal Simek

Firstly I was surprise that you create any frame pointer solution but
1. It is not frame pointer because Microblaze not use it

Can you explain this in different words? I'm not following you. The code compiles differently in these two cases:

ifdef CONFIG_FRAME_POINTER
KBUILD_CFLAGS += -fno-omit-frame-pointer -fno-optimize-sibling-calls
else
KBUILD_CFLAGS += -fomit-frame-pointer
endif

Empirically speaking, the former causes the compiler to use r19 to hold the frame pointer. The latter allows the compiler to use r19 for other purposes.

Quoted 5 lines

2. it is just one optimization which could help but IMHO not. Your patch expects that every stack frame size has 7/8 (doesn't matter right now) items but that's not correct expectation. (Do objdump from vmlinux and look at cpu_idle, prom_add_property and others) - that's why I think that your patch won't work.

The patch expects only that frames involved in a backtrace are _at least_ 8 words deep and that the frame pointer is always the 8th word of the frame (index 7).

In my build, cpu_idle() begins like this:

 4b8:   3021ffd8        addik   r1, r1, -40
 4bc:   fa61001c        swi     r19, r1, 28
 4c0:   f9e10000        swi     r15, r1, 0

...which is a frame of 40 bytes, and a frame pointer stored 7 words into the frame. prom_add_property() has a frame of 48 bytes and a frame pointer stored 7 words in.

Now, disable_hlt() has a runt frame of only 8 bytes when compiled with -fno-omit-frame-pointer. But it is a leaf function and should never show up in a backtrace, at least in a noMMU kernel. For MMU I suppose it's possible for a leaf function to oops. I don't know the implications of that.

Although the examples you cite don't prove your point, in looking more closely, I see that there _are_ non-leaf functions where the frame pointer is being placed elsewhere, for example do_one_initcall():

20000064:       3021ffc0        addik   r1, r1, -64
20000068:       fa61002c        swi     r19, r1, 44

This of course is a killer. I wonder if this is something that could be changed in the Microblaze gcc someday.

Quoted 6 lines

3. The next question is, if we can expect that every function record has at least 7/8 items. If yes than look at my function below.
4. One more thing is that function still use kernel_text_address() which is silly because we are still not sure if the address there is correct or not. It is just checking and if we are using, it is just mean that there is any expectation which is not correct.

It may be that the function can be improved further; that's beyond the scope of what I was trying to accomplish.

Quoted 26 lines
Quoted 5 lines

---
diff -uprN a/arch/microblaze/Kconfig.debug b/arch/microblaze/Kconfig.debug

arch/microblaze/Kconfig.debug config DEBUG_BOOTMEM · line 35

Quoted 10 lines
arch/microblaze/Kconfig.debug +7 -0
@@ -26,4 +26,11 @@ config DEBUG_BOOTMEM
 	depends on DEBUG_KERNEL
 	bool "Debug BOOTMEM initialization"
 
+config FRAME_POINTER
+	bool "Use frame pointers"
+	default n
+	help
+	  If you say N here, the resulting kernel will be slightly smaller and
+	  faster. However, stack dumps will be much harder to interpret.
+

depends on !MMU

Quoted 8 lines
arch/microblaze/Kconfig.debug +0 -0
 endmenu

diff -uprN a/arch/microblaze/kernel/traps.c b/arch/microblaze/kernel/traps.c

arch/microblaze/kernel/traps.c +0 -0
@@ -8,6 +8,7 @@
  * for more details.
  */
 
Quoted 3 lines

arch/microblaze/kernel/traps.c line 11

Quoted 1 line
arch/microblaze/kernel/traps.c +1 -0
+#include <generated/autoconf.h>

why? I don't think that this is necessary.

Otherwise CONFIG_FRAME_POINTER is always undefined.

Quoted 35 lines
Quoted 14 lines
arch/microblaze/kernel/traps.c +1 -1
 #include <linux/kernel.h>
 #include <linux/kallsyms.h>
 #include <linux/module.h>
@@ -44,7 +45,7 @@ void show_trace(struct task_struct *task
 	printk(KERN_NOTICE "\n");
 #endif
 	while (!kstack_end(stack)) {
-		addr = *stack++;
+		addr = *stack;
 		/*
 		 * If the address is either in the text segment of the
 		 * kernel, or in the region which contains vmalloc'ed

arch/microblaze/kernel/traps.c void show_trace(struct task_struct *task · line 62

Quoted 7 lines
arch/microblaze/kernel/traps.c +4 -0
@@ -55,6 +56,13 @@ void show_trace(struct task_struct *task
 		 */
 		if (kernel_text_address(addr))
 			print_ip_sym(addr);
+
+#if defined(CONFIG_FRAME_POINTER)
+		/* Fetch the caller's frame pointer */
+		stack = (unsigned long *) stack[7];

If is calculation correct then some comments, why you use number 7, will be necessary.

arch/microblaze/kernel/traps.c void show_trace(struct task_struct *task · line 68

Quoted 7 lines
arch/microblaze/kernel/traps.c +3 -0
+#else
+		stack++;
+#endif
 	}
 	printk(KERN_NOTICE "\n");
 

Look at this code which should be better than yours.

Quoted 5 lines
   		if (kernel_text_address(addr)) {
   			print_ip_sym(addr);
			/* Fetch the caller's frame pointer */
			stack = (unsigned long *) stack[7];
		}

There would need to be an 'else' here. Also, once the code goes off the frame pointer rail, it should stay off.

Quoted 2 lines
		stack++;

Regards,
------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
www.digidescorp.com Earthling, return my space modulator!"

#include <standard.disclaimer>

#3 lore raw
Quoted 31 lines · Steven J. Magnani
Quoted 5 lines · Michal Simek

2. it is just one optimization which could help but IMHO not. Your patch expects that every stack frame size has 7/8 (doesn't matter right now) items but that's not correct expectation. (Do objdump from vmlinux and look at cpu_idle, prom_add_property and others) - that's why I think that your patch won't work.

The patch expects only that frames involved in a backtrace are _at least_ 8 words deep and that the frame pointer is always the 8th word of the frame (index 7).

In my build, cpu_idle() begins like this:

 4b8:   3021ffd8        addik   r1, r1, -40
 4bc:   fa61001c        swi     r19, r1, 28
 4c0:   f9e10000        swi     r15, r1, 0

...which is a frame of 40 bytes, and a frame pointer stored 7 words into the frame. prom_add_property() has a frame of 48 bytes and a frame pointer stored 7 words in.

Now, disable_hlt() has a runt frame of only 8 bytes when compiled with -fno-omit-frame-pointer. But it is a leaf function and should never show up in a backtrace, at least in a noMMU kernel. For MMU I suppose it's possible for a leaf function to oops. I don't know the implications of that.

Although the examples you cite don't prove your point, in looking more closely, I see that there _are_ non-leaf functions where the frame pointer is being placed elsewhere, for example do_one_initcall():

Quoted 6 lines
20000064:       3021ffc0        addik   r1, r1, -64
20000068:       fa61002c        swi     r19, r1, 44

This of course is a killer. I wonder if this is something that could be changed in the Microblaze gcc someday.

This doesn't look that bad compared to what some of the other architectures have to deal with. If the frame pointer is always setup using these addik/swi pairs then you can trivially scan an arbitrary number of instructions attempting to match before giving up. We do similar things for sh64 where we also have to figure out how stack frames were created in order to roll them back.

In any event, take a look at arch/sh/kernel/cpu/sh5/unwind.c, you should be able to use a similar scheme without the need for undue complexity.

#4 lore raw

Paul -

Quoted 10 lines · Paul Mundt

This doesn't look that bad compared to what some of the other architectures have to deal with. If the frame pointer is always setup using these addik/swi pairs then you can trivially scan an arbitrary number of instructions attempting to match before giving up. We do similar things for sh64 where we also have to figure out how stack frames were created in order to roll them back.

In any event, take a look at arch/sh/kernel/cpu/sh5/unwind.c, you should be able to use a similar scheme without the need for undue complexity.

Thanks for the tip. This looks manageable. I had thought to search for instructions that create frames but didn't think working backwards from return addresses was a good idea. Using kallsyms to get the "top" of each function is a nice way around that.

Regards,
------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
www.digidescorp.com Earthling, return my space modulator!"

#include <standard.disclaimer>

#5 lore raw

I've had some time to bake a Microblaze implementation of this and have come up with a question...

Quoted 8 lines · Steven J. Magnani
Quoted 7 lines · Paul Mundt

This doesn't look that bad compared to what some of the other architectures have to deal with. If the frame pointer is always setup using these addik/swi pairs then you can trivially scan an arbitrary number of instructions attempting to match before giving up. We do similar things for sh64 where we also have to figure out how stack frames were created in order to roll them back.

Quoted 8 lines
Quoted 3 lines

In any event, take a look at arch/sh/kernel/cpu/sh5/unwind.c, you should be able to use a similar scheme without the need for undue complexity.

Thanks for the tip. This looks manageable. I had thought to search for instructions that create frames but didn't think working backwards from return addresses was a good idea. Using kallsyms to get the "top" of each function is a nice way around that.

The kallsyms approach to unwinding depends on as many functions as possible being in the table, for two reasons:

* The unwind can't get past an address that kallsyms can't identify
* A mis-lookup, such as reporting a nearby global function when the actual address lies in a static function, can send the unwind off the rails

To get static (local) functions into kallsyms, I've found that I need to specify -fno-unit-at-a-time to Microblaze gcc 4.1.2. I suspect that particular option has other side-effects, but despite much searching I couldn't find any mention of a gcc option that would do what I need. My x86_64 gcc (4.4.1) seems to emit local symbols by default even with -funit-at-a-time.

Any suggestions?

------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
www.digidescorp.com Earthling, return my space modulator!"

#include <standard.disclaimer>

#6 lore raw
Quoted 29 lines · Steven J. Magnani

I've had some time to bake a Microblaze implementation of this and have come up with a question...

Quoted 8 lines · Steven J. Magnani
Quoted 7 lines · Paul Mundt

This doesn't look that bad compared to what some of the other architectures have to deal with. If the frame pointer is always setup using these addik/swi pairs then you can trivially scan an arbitrary number of instructions attempting to match before giving up. We do similar things for sh64 where we also have to figure out how stack frames were created in order to roll them back.

Quoted 8 lines
Quoted 3 lines

In any event, take a look at arch/sh/kernel/cpu/sh5/unwind.c, you should be able to use a similar scheme without the need for undue complexity.

Thanks for the tip. This looks manageable. I had thought to search for instructions that create frames but didn't think working backwards from return addresses was a good idea. Using kallsyms to get the "top" of each function is a nice way around that.

The kallsyms approach to unwinding depends on as many functions as possible being in the table, for two reasons:

* The unwind can't get past an address that kallsyms can't identify
* A mis-lookup, such as reporting a nearby global function when the actual address lies in a static function, can send the unwind off the rails

Quoted 9 lines

To get static (local) functions into kallsyms, I've found that I need to specify -fno-unit-at-a-time to Microblaze gcc 4.1.2. I suspect that particular option has other side-effects, but despite much searching I couldn't find any mention of a gcc option that would do what I need. My x86_64 gcc (4.4.1) seems to emit local symbols by default even with -funit-at-a-time.

Any suggestions?

Local functions under kallsyms are a bit of a bother, particularly since there are bound to be namespace collisions that the linker will never see. You would also want to set KALLSYMS_ALL if you are valuing debugging over space savings, which seems to be the case if you are considering disabling unit-at-a-time.

On the kprobes side there was an attempt by jejb to prefix local symbols with their file and line numbers to differentiate the symbols, but while this might work in the case of manual probe insertion, it's certainly not suitable for the unwinder. Nothing seems to have happened with that though, so the point is rather moot regardless.

The lookup not being exact is certainly a problem, but I haven't found a good workaround for that either. Potentially inaccurate call stacks are still wildly preferential to not being able to unwind past the first frame.

#7 lore raw
In reply to Steven J. Magnani
Quoted 5 lines · Steven J. Magnani

diff -uprN a/arch/microblaze/Kconfig.debug b/arch/microblaze/Kconfig.debug

arch/microblaze/Kconfig.debug +0 -0
@@ -26,4 +26,11 @@ config DEBUG_BOOTMEM
 	depends on DEBUG_KERNEL

arch/microblaze/Kconfig.debug config DEBUG_BOOTMEM · line 36

Quoted 10 lines
arch/microblaze/Kconfig.debug +7 -0
 	bool "Debug BOOTMEM initialization"
 
+config FRAME_POINTER
+	bool "Use frame pointers"
+	default n
+	help
+	  If you say N here, the resulting kernel will be slightly smaller and
+	  faster. However, stack dumps will be much harder to interpret.
+
 endmenu

This isn't necessary, as it already exists in lib/Kconfig.debug. You can add your architecture to the list there, or if you want it default enabled you can also select ARCH_WANT_FRAME_POINTERS.

arch/microblaze/kernel/traps.c line 14

Quoted 11 lines

diff -uprN a/arch/microblaze/kernel/traps.c b/arch/microblaze/kernel/traps.c

arch/microblaze/kernel/traps.c +1 -0
@@ -8,6 +8,7 @@
  * for more details.
  */
 
+#include <generated/autoconf.h>
 #include <linux/kernel.h>
 #include <linux/kallsyms.h>
 #include <linux/module.h>

No need for this either, it's pulled in by Kbuild.