microblaze: Support FRAME_POINTER for better backtrace
RFC 1/1 microblaze: Support FRAME_POINTER for better backtrace
Add a FRAME_POINTER option and when it is enabled, use frame pointers to walk the stack during a backtrace dump. This eliminates printout of confusing "function calls" corresponding to stack values that look like they might be return addresses, but aren't.
This patch is dependent upon
[PATCH] microblaze: Begin stack dump with caller of dump_stack()
I'm not certain whether the MMU compiler generates frame pointers the same way as the noMMU compiler I am using. I'm also not sure what all the ramifications of providing FRAME_POINTER are. It looks like tracing functionality makes use of it. Need someone familiar with these areas to comment on the patch.
---
arch/microblaze/Kconfig.debug +7 -0
@@ -26,4 +26,11 @@ config DEBUG_BOOTMEM depends on DEBUG_KERNEL bool "Debug BOOTMEM initialization" +config FRAME_POINTER + bool "Use frame pointers" + default n + help + If you say N here, the resulting kernel will be slightly smaller and + faster. However, stack dumps will be much harder to interpret. + endmenu
arch/microblaze/kernel/traps.c +9 -1
@@ -8,6 +8,7 @@* for more details. */ +#include <generated/autoconf.h> #include <linux/kernel.h> #include <linux/kallsyms.h> #include <linux/module.h>
@@ -44,7 +45,7 @@ void show_trace(struct task_struct *taskprintk (KERN_NOTICE "\n" );#endif while (!kstack_end (stack)) { - addr = *stack++; + addr = *stack;/* * If the address is either in the text segment of the * kernel, or in the region which contains vmalloc'ed
@@ -55,6 +56,13 @@ void show_trace(struct task_struct *task*/ if (kernel_text_address (addr))print_ip_sym (addr); + +#if defined(CONFIG_FRAME_POINTER ) +/* Fetch the caller's frame pointer */ + stack = (unsigned long *) stack[7 ]; +#else + stack++; +#endif }printk (KERN_NOTICE "\n" );
Michal Simek
maintainerQuoted 7 lines · Steven J. Magnani
Add a
FRAME_POINTERoption and when it is enabled, use frame pointers to walk the stack during a backtrace dump. This eliminates printout of confusing "function calls" corresponding to stack values that look like they might be return addresses, but aren't.This patch is dependent upon
[PATCH] microblaze: Begin stack dump with caller ofdump_stack()
yes.
Quoted 4 lines
I'm not certain whether the MMU compiler generates frame pointers the same
way as the noMMU compiler I am using. I'm also not sure what all the ramifications of providingFRAME_POINTERare. It looks like tracing functionality makes use of it. Need someone familiar with these areas to comment on the patch.
Firstly I was surprise that you create any frame pointer solution but
1. It is not frame pointer because Microblaze not use it
2. it is just one optimization which could help but IMHO not. Your patch expects that every stack frame size has 7/8 (doesn't matter right now) items but that's not correct expectation. (Do objdump from vmlinux and look at cpu_idle, prom_add_property and others) - that's why I think that your patch won't work.
3. The next question is, if we can expect that every function record has at least 7/8 items. If yes than look at my function below.
4. One more thing is that function still use kernel_text_address() which is silly because we are still not sure if the address there is correct or not. It is just checking and if we are using, it is just mean that there is any expectation which is not correct.
I did some testing several months/weeks ago and I tried to solve it in the same way as you. But found that it is based on any expectation which
is not correct.
I know that stack tracing is pain but I am not convinced that this is way how to solve it. :-(
Michal
Quoted 7 lines
Signed-off-by Steven J. Magnani <[email protected]>---
diff -uprN a/arch/microblaze/Kconfig.debug b/arch/microblaze/Kconfig.debug
arch/microblaze/Kconfig.debug config DEBUG_BOOTMEM · line 35
Quoted 10 lines
arch/microblaze/Kconfig.debug +7
-0@@ -26,4 +26,11 @@ config DEBUG_BOOTMEM depends on DEBUG_KERNEL bool "Debug BOOTMEM initialization" +config FRAME_POINTER + bool "Use frame pointers" + default n + help + If you say N here, the resulting kernel will be slightly smaller and + faster. However, stack dumps will be much harder to interpret. +
depends on !MMU
arch/microblaze/kernel/traps.c line 11
Quoted 9 lines
arch/microblaze/Kconfig.debug +0
-0endmenudiff -uprN
a/arch/microblaze/kernel/traps.cb/arch/microblaze/kernel/traps.carch/microblaze/kernel/traps.c +1
-0@@ -8,6 +8,7 @@* for more details. */ +#include <generated/autoconf.h>
why? I don't think that this is necessary.
Quoted 13 lines
arch/microblaze/kernel/traps.c +1
-1#include <linux/kernel.h> #include <linux/kallsyms.h> #include <linux/module.h> @@ -44,7 +45,7 @@ void show_trace(struct task_struct *taskprintk (KERN_NOTICE "\n" );#endif while (!kstack_end (stack)) { - addr = *stack++; + addr = *stack;/* * If the address is either in the text segment of the * kernel, or in the region which contains vmalloc'ed
arch/microblaze/kernel/traps.c void show_trace(struct task_struct *task · line 62
Quoted 7 lines
arch/microblaze/kernel/traps.c +4
-0@@ -55,6 +56,13 @@ void show_trace(struct task_struct *task*/ if (kernel_text_address (addr))print_ip_sym (addr); + +#if defined(CONFIG_FRAME_POINTER ) +/* Fetch the caller's frame pointer */ + stack = (unsigned long *) stack[7 ];
If is calculation correct then some comments, why you use number 7, will be necessary.
arch/microblaze/kernel/traps.c void show_trace(struct task_struct *task · line 68
Quoted 7 lines
arch/microblaze/kernel/traps.c +3
-0+#else + stack++; +#endif }printk (KERN_NOTICE "\n" );
Look at this code which should be better than yours.
if (kernel_text_address(addr)) {
print_ip_sym(addr); /* Fetch the caller's frame pointer */ stack = (unsigned long *) stack[7]; } stack++;
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Michal,
Quoted 2 lines · Michal Simek
Firstly I was surprise that you create any frame pointer solution but
1. It is not frame pointer because Microblaze not use it
Can you explain this in different words? I'm not following you. The code compiles differently in these two cases:
ifdef CONFIG_FRAME_POINTER
KBUILD_CFLAGS += -fno-omit-frame-pointer -fno-optimize-sibling-calls
else
KBUILD_CFLAGS += -fomit-frame-pointer
endif
Empirically speaking, the former causes the compiler to use r19 to hold the frame pointer. The latter allows the compiler to use r19 for other purposes.
Quoted 5 lines
2. it is just one optimization which could help but IMHO not. Your patch expects that every stack frame size has 7/8 (doesn't matter right now) items but that's not correct expectation. (Do objdump from vmlinux and look at
cpu_idle,prom_add_propertyand others) - that's why I think that your patch won't work.
The patch expects only that frames involved in a backtrace are _at least_ 8 words deep and that the frame pointer is always the 8th word of the frame (index 7).
In my build, cpu_idle() begins like this:
4b8: 3021ffd8 addik r1, r1, -40 4bc: fa61001c swi r19, r1, 28 4c0: f9e10000 swi r15, r1, 0
...which is a frame of 40 bytes, and a frame pointer stored 7 words into the frame. prom_add_property() has a frame of 48 bytes and a frame pointer stored 7 words in.
Now, disable_hlt() has a runt frame of only 8 bytes when compiled with -fno-omit-frame-pointer. But it is a leaf function and should never show up in a backtrace, at least in a noMMU kernel. For MMU I suppose it's possible for a leaf function to oops. I don't know the implications of that.
Although the examples you cite don't prove your point, in looking more closely, I see that there _are_ non-leaf functions where the frame pointer is being placed elsewhere, for example do_one_initcall():
20000064: 3021ffc0 addik r1, r1, -64 20000068: fa61002c swi r19, r1, 44
This of course is a killer. I wonder if this is something that could be changed in the Microblaze gcc someday.
Quoted 6 lines
3. The next question is, if we can expect that every function record has at least 7/8 items. If yes than look at my function below.
4. One more thing is that function still usekernel_text_address()which is silly because we are still not sure if the address there is correct or not. It is just checking and if we are using, it is just mean that there is any expectation which is not correct.
It may be that the function can be improved further; that's beyond the scope of what I was trying to accomplish.
Quoted 26 lines
Quoted 5 lines
---
diff -uprN a/arch/microblaze/Kconfig.debug b/arch/microblaze/Kconfig.debugarch/microblaze/Kconfig.debug config DEBUG_BOOTMEM · line 35
Quoted 10 lines
arch/microblaze/Kconfig.debug +7
-0@@ -26,4 +26,11 @@ config DEBUG_BOOTMEM depends on DEBUG_KERNEL bool "Debug BOOTMEM initialization" +config FRAME_POINTER + bool "Use frame pointers" + default n + help + If you say N here, the resulting kernel will be slightly smaller and + faster. However, stack dumps will be much harder to interpret. +depends on !MMU
Quoted 8 lines
arch/microblaze/Kconfig.debug +0
-0endmenudiff -uprN
a/arch/microblaze/kernel/traps.cb/arch/microblaze/kernel/traps.carch/microblaze/kernel/traps.c +0
-0@@ -8,6 +8,7 @@* for more details. */
Quoted 3 lines
arch/microblaze/kernel/traps.c line 11
Quoted 1 line
arch/microblaze/kernel/traps.c +1
-0+#include <generated/autoconf.h> why? I don't think that this is necessary.
Otherwise CONFIG_FRAME_POINTER is always undefined.
Quoted 35 lines
Quoted 14 lines
arch/microblaze/kernel/traps.c +1
-1#include <linux/kernel.h> #include <linux/kallsyms.h> #include <linux/module.h> @@ -44,7 +45,7 @@ void show_trace(struct task_struct *taskprintk (KERN_NOTICE "\n" );#endif while (!kstack_end (stack)) { - addr = *stack++; + addr = *stack;/* * If the address is either in the text segment of the * kernel, or in the region which contains vmalloc'ed arch/microblaze/kernel/traps.c void show_trace(struct task_struct *task · line 62
Quoted 7 lines
arch/microblaze/kernel/traps.c +4
-0@@ -55,6 +56,13 @@ void show_trace(struct task_struct *task*/ if (kernel_text_address (addr))print_ip_sym (addr); + +#if defined(CONFIG_FRAME_POINTER ) +/* Fetch the caller's frame pointer */ + stack = (unsigned long *) stack[7 ];If is calculation correct then some comments, why you use number 7, will be necessary.
arch/microblaze/kernel/traps.c void show_trace(struct task_struct *task · line 68
Quoted 7 lines
arch/microblaze/kernel/traps.c +3
-0+#else + stack++; +#endif }printk (KERN_NOTICE "\n" );Look at this code which should be better than yours.
Quoted 5 lines
if (kernel_text_address(addr)) {
print_ip_sym(addr);
/* Fetch the caller's frame pointer */
stack = (unsigned long *) stack[7];
}There would need to be an 'else' here. Also, once the code goes off the frame pointer rail, it should stay off.
Quoted 2 lines
stack++;
Regards,
------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
www.digidescorp.com Earthling, return my space modulator!"
#include <standard.disclaimer>
Quoted 31 lines · Steven J. Magnani
Quoted 5 lines · Michal Simek
2. it is just one optimization which could help but IMHO not. Your patch expects that every stack frame size has 7/8 (doesn't matter right now) items but that's not correct expectation. (Do objdump from vmlinux and look at
cpu_idle,prom_add_propertyand others) - that's why I think that your patch won't work.The patch expects only that frames involved in a backtrace are
_atleast_8 words deep and that the frame pointer is always the 8th word of the frame (index 7).In my build,
cpu_idle()begins like this:4b8: 3021ffd8 addik r1, r1, -40 4bc: fa61001c swi r19, r1, 28 4c0: f9e10000 swi r15, r1, 0...which is a frame of 40 bytes, and a frame pointer stored 7 words into the frame.
prom_add_property()has a frame of 48 bytes and a frame pointer stored 7 words in.Now,
disable_hlt()has a runt frame of only 8 bytes when compiled with -fno-omit-frame-pointer. But it is a leaf function and should never show up in a backtrace, at least in a noMMU kernel. For MMU I suppose it's possible for a leaf function to oops. I don't know the implications of that.Although the examples you cite don't prove your point, in looking more closely, I see that there _are_ non-leaf functions where the frame pointer is being placed elsewhere, for example
do_one_initcall():
Quoted 6 lines
20000064: 3021ffc0 addik r1, r1, -64 20000068: fa61002c swi r19, r1, 44This of course is a killer. I wonder if this is something that could be changed in the Microblaze gcc someday.
This doesn't look that bad compared to what some of the other architectures have to deal with. If the frame pointer is always setup using these addik/swi pairs then you can trivially scan an arbitrary number of instructions attempting to match before giving up. We do similar things for sh64 where we also have to figure out how stack frames were created in order to roll them back.
In any event, take a look at arch/sh/kernel/cpu/sh5/unwind.c, you should be able to use a similar scheme without the need for undue complexity.
Paul -
Quoted 10 lines · Paul Mundt
This doesn't look that bad compared to what some of the other architectures have to deal with. If the frame pointer is always setup using these addik/swi pairs then you can trivially scan an arbitrary number of instructions attempting to match before giving up. We do similar things for sh64 where we also have to figure out how stack frames were created in order to roll them back.
In any event, take a look at
arch/sh/kernel/cpu/sh5/unwind.c, you should be able to use a similar scheme without the need for undue complexity.
Thanks for the tip. This looks manageable. I had thought to search for instructions that create frames but didn't think working backwards from return addresses was a good idea. Using kallsyms to get the "top" of each function is a nice way around that.
Regards,
------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
www.digidescorp.com Earthling, return my space modulator!"
#include <standard.disclaimer>
I've had some time to bake a Microblaze implementation of this and have come up with a question...
Quoted 8 lines · Steven J. Magnani
Quoted 7 lines · Paul Mundt
This doesn't look that bad compared to what some of the other architectures have to deal with. If the frame pointer is always setup using these addik/swi pairs then you can trivially scan an arbitrary number of instructions attempting to match before giving up. We do similar things for sh64 where we also have to figure out how stack frames were created in order to roll them back.
Quoted 8 lines
Quoted 3 lines
In any event, take a look at
arch/sh/kernel/cpu/sh5/unwind.c, you should be able to use a similar scheme without the need for undue complexity.Thanks for the tip. This looks manageable. I had thought to search for instructions that create frames but didn't think working backwards from return addresses was a good idea. Using kallsyms to get the "top" of each function is a nice way around that.
The kallsyms approach to unwinding depends on as many functions as possible being in the table, for two reasons:
* The unwind can't get past an address that kallsyms can't identify
* A mis-lookup, such as reporting a nearby global function when the actual address lies in a static function, can send the unwind off the rails
To get static (local) functions into kallsyms, I've found that I need to specify -fno-unit-at-a-time to Microblaze gcc 4.1.2. I suspect that particular option has other side-effects, but despite much searching I couldn't find any mention of a gcc option that would do what I need. My x86_64 gcc (4.4.1) seems to emit local symbols by default even with -funit-at-a-time.
Any suggestions?
------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
www.digidescorp.com Earthling, return my space modulator!"
#include <standard.disclaimer>
Quoted 29 lines · Steven J. Magnani
I've had some time to bake a Microblaze implementation of this and have come up with a question...
Quoted 8 lines · Steven J. Magnani
Quoted 7 lines · Paul Mundt
This doesn't look that bad compared to what some of the other architectures have to deal with. If the frame pointer is always setup using these addik/swi pairs then you can trivially scan an arbitrary number of instructions attempting to match before giving up. We do similar things for sh64 where we also have to figure out how stack frames were created in order to roll them back.
Quoted 8 lines
Quoted 3 lines
In any event, take a look at
arch/sh/kernel/cpu/sh5/unwind.c, you should be able to use a similar scheme without the need for undue complexity.Thanks for the tip. This looks manageable. I had thought to search for instructions that create frames but didn't think working backwards from return addresses was a good idea. Using kallsyms to get the "top" of each function is a nice way around that.
The kallsyms approach to unwinding depends on as many functions as possible being in the table, for two reasons:
* The unwind can't get past an address that kallsyms can't identify
* A mis-lookup, such as reporting a nearby global function when the actual address lies in a static function, can send the unwind off the rails
Quoted 9 lines
To get static (local) functions into kallsyms, I've found that I need to specify -fno-unit-at-a-time to Microblaze gcc 4.1.2. I suspect that particular option has other side-effects, but despite much searching I couldn't find any mention of a gcc option that would do what I need. My
x86_64gcc (4.4.1) seems to emit local symbols by default even with -funit-at-a-time.Any suggestions?
Local functions under kallsyms are a bit of a bother, particularly since there are bound to be namespace collisions that the linker will never see. You would also want to set KALLSYMS_ALL if you are valuing debugging over space savings, which seems to be the case if you are considering disabling unit-at-a-time.
On the kprobes side there was an attempt by jejb to prefix local symbols with their file and line numbers to differentiate the symbols, but while this might work in the case of manual probe insertion, it's certainly not suitable for the unwinder. Nothing seems to have happened with that though, so the point is rather moot regardless.
The lookup not being exact is certainly a problem, but I haven't found a good workaround for that either. Potentially inaccurate call stacks are still wildly preferential to not being able to unwind past the first frame.
Quoted 5 lines · Steven J. Magnani
diff -uprN a/arch/microblaze/Kconfig.debug b/arch/microblaze/Kconfig.debug
arch/microblaze/Kconfig.debug +0
-0@@ -26,4 +26,11 @@ config DEBUG_BOOTMEM depends on DEBUG_KERNEL
arch/microblaze/Kconfig.debug config DEBUG_BOOTMEM · line 36
Quoted 10 lines
arch/microblaze/Kconfig.debug +7
-0bool "Debug BOOTMEM initialization" +config FRAME_POINTER + bool "Use frame pointers" + default n + help + If you say N here, the resulting kernel will be slightly smaller and + faster. However, stack dumps will be much harder to interpret. + endmenu
This isn't necessary, as it already exists in lib/Kconfig.debug. You can add your architecture to the list there, or if you want it default enabled you can also select ARCH_WANT_FRAME_POINTERS.
arch/microblaze/kernel/traps.c line 14
Quoted 11 lines
diff -uprN
a/arch/microblaze/kernel/traps.cb/arch/microblaze/kernel/traps.carch/microblaze/kernel/traps.c +1
-0@@ -8,6 +8,7 @@* for more details. */ +#include <generated/autoconf.h> #include <linux/kernel.h> #include <linux/kallsyms.h> #include <linux/module.h>
No need for this either, it's pulled in by Kbuild.