how to get a process backtrace from kernel gdb?
I'm running gdb on vmlinux connected to a remote target (2.4 kernel). I have the task_struct address of 'current' and other processes. Is it possible to get a symbolic stack trace of the kernel stack? Where is the kernel stack located? I tried to print (task_struct->reg29)[13]. Is this the PC?
thanks
srini
PS. I broke into gdb using a hotkey on the serial console; so the gdb backtrace shows the serial driver.
Quoted 5 lines · Srinivas Kommu
I'm running gdb on vmlinux connected to a remote target (2.4 kernel). I have the
task_structaddress of 'current' and other processes. Is it possible to get a symbolic stack trace of the kernel stack? Where is the kernel stack located? I tried to print (task_struct->reg29)[13]. Is this the PC?
I assume you meant task_struct->reg29)[31] which is the address at which the process is going to resume execution when it's time has arrived. But in most cases this address isn't terribly interesting. So we have two cases here:
o $31 pointing to ret_from_fork
This is a new born process which will begin it's active life of execution on a CPU at ret_from_fork. This is where the resume function will jump to which may well be not the scheduler function.
o Otherwise:
The thread is regaining the CPU and the resume() call is going to return straight to it's caller, kernel/sched.c:context_switch() which inlined into schedule() and what we actually want is schedule's caller, so we dig through the scheduler's stack frame. To make things more entertaining the stack frame will change with configuration options and compiler used, so we basically have to disassemble the stackframe.
For get_wchan We repeat that exercise in the cases we one of the other scheduler functions may have called schedule() until we reach a non-schedule function. So now we have a pointer that actually points to something interesting, a driver or other kernel subsystem.
The whole thing is a bit of a mindbender. Why? The scheduler is designed to deliver best possible performance and we've not compromized on performance to make the job of backtracing any easier.
Quoted 2 lines
PS. I broke into gdb using a hotkey on the serial console; so the gdb backtrace shows the serial driver.
In which you need to be extra careful about the validity of any pointer you might encounter.
Ralf
Are there any existing method/code to get a reliable back trace dump for oops? Kgdb is not usable here because I have no serial port. Do I need to copy gdb code to analysis the stack?
If I am not wrong, dump_stack() just print out any stack address falled in the text section, and that will make the result hard to understand. For example, I often find some interrupt functions mixed with normal function names and repeated functions, e.g.
8e1c7968: redzone 1: 0x0, redzone 2: 0x170fc2a5.
slab error in cache_free_debugcheck(): cache `dentry_cache': double free, or memory nCall Trace:
[<8035de48>] cache_free_debugcheck+0x1dc/0x2d4 [<8035de1c>] cache_free_debugcheck+0x1b0/0x2d4 [<8035f35c>] kmem_cache_free+0x60/0x248 [<8033f7e4>] __rcu_process_callbacks+0xdc/0x360 /* between kmem_cache_free? */ [<8035f35c>] kmem_cache_free+0x60/0x248 [<80300e10>] ll_cputimer_irq+0xc/0x14 [<8033f7e4>] __rcu_process_callbacks+0xdc/0x360 [<804c1480>] int_timer_do+0x1c/0x30 [<8033fa8c>] rcu_process_callbacks+0x24/0x48 [<803338d8>] run_timer_softirq+0x128/0x244 [<804c1480>] int_timer_do+0x1c/0x30 [<8032ee50>] tasklet_action+0xd4/0x1b8 [<8032e7e4>] __do_softirq+0x94/0x15c [<8032e93c>] do_softirq+0x90/0xbc [<80303814>] do_IRQ+0x24/0x34 [<8035e220>] cache_alloc_debugcheck_after+0x17c/0x1b4 /* do_IRQ happens inside debugcheck? */ [<8035e220>] cache_alloc_debugcheck_after+0x17c/0x1b4 [<80300e10>] ll_cputimer_irq+0xc/0x14 [<803a961c>] d_alloc+0x34/0x22c [<8035e5e8>] kmem_cache_alloc+0x128/0x220 [<803a961c>] d_alloc+0x34/0x22c /* where is the ROOT of current trace... ?? */ [<80478660>] memset_partial+0x44/0x6c [<8039a808>] __link_path_walk+0x790/0x1a38 [<8039a870>] __link_path_walk+0x7f8/0x1a38 [<8035d8a0>] check_poison_obj+0x40/0x238 [<8039bb90>] link_path_walk+0xe0/0x438 [<8035e174>] cache_alloc_debugcheck_after+0xd0/0x1b4 [<80380114>] get_unused_fd+0x158/0x20c [<8039bfc8>] path_lookup+0xe0/0x3e8 [<8038235c>] get_empty_filp+0x5c/0x110 [<8038235c>] get_empty_filp+0x5c/0x110 [<8039d2f8>] open_namei+0xa4/0xdf4 [<8038235c>] get_empty_filp+0x5c/0x110 [<8037fee4>] filp_open+0x74/0xe0 [<80398bdc>] getname+0x2c/0x13c [<80380114>] get_unused_fd+0x158/0x20c [<803802e0>] do_sys_open+0x84/0x144 [<80303814>] do_IRQ+0x24/0x34 [<8030c5a4>] stack_done+0x20/0x3c [<80478670>] memset_partial+0x54/0x6c
8e1c78d4: redzone 1: 0x170fc2a5, redzone 2: 0x10.
IOPortBase = 2af34000
Quoted 44 lines · Ralf Baechle
Quoted 5 lines · Srinivas Kommu
I'm running gdb on vmlinux connected to a remote target (2.4 kernel). I have the
task_structaddress of 'current' and other processes. Is it possible to get a symbolic stack trace of the kernel stack? Where is the kernel stack located? I tried to print (task_struct->reg29)[13]. Is this the PC?I assume you meant task_struct->reg29)[31] which is the address at which the process is going to resume execution when it's time has arrived. But in most cases this address isn't terribly interesting. So we have two cases here:
o $31 pointing to
ret_from_fork
This is a new born process which will begin it's active life of execution on a CPU atret_from_fork. This is where the resume function will jump to which may well be not the scheduler function.
o Otherwise:
The thread is regaining the CPU and theresume()call is going to return straight to it's caller, kernel/sched.c:context_switch() which inlined intoschedule()and what we actually want is schedule's caller, so we dig through the scheduler's stack frame. To make things more entertaining the stack frame will change with configuration options and compiler used, so we basically have to disassemble the stackframe.For
get_wchanWe repeat that exercise in the cases we one of the other scheduler functions may have calledschedule()until we reach a non-schedule function. So now we have a pointer that actually points to something interesting, a driver or other kernel subsystem.The whole thing is a bit of a mindbender. Why? The scheduler is designed to deliver best possible performance and we've not compromized on performance to make the job of backtracing any easier.
Quoted 2 lines
PS. I broke into gdb using a hotkey on the serial console; so the gdb backtrace shows the serial driver.
In which you need to be extra careful about the validity of any pointer you might encounter.
Ralf
Quoted 3 lines · Fuxin Zhang
Are there any existing method/code to get a reliable back trace dump for oops? Kgdb is not usable here because I have no serial port. Do I need to copy gdb code to analysis the stack?
By all practical means the method used by get_wchan() in my previous posting is safe. A method that is also theoretically correct is a little harder but could be constructed based on the gcc generated DWARF 2 debugging info. IA64 does this for example.
Quoted 4 lines
If I am not wrong,
dump_stack()just print out any stack address falled in the text section, and that will make the result hard to understand. For example, I often find some interrupt functions mixed with normal function names and repeated functions, e.g.
You're absolutely right, dump_stack really needs to be fixed to be more useful. As it is it's generating alot of confusing output. There are two possible approaches here:
a) Analyze the code with a simple frame unwinder along the lines of the get_wchan implementation. Simple but possibly fragile as compilers continue to improve.
b) Bite the bullet and use the DWARD 2 frame unwind info and code. More complicated but will be a solid and correct solution albeit larger so not acceptable for small embedded devices.
I fear we may have to do both, 2) as the prefered solution and 1) as the fallback solution.
Ralf
Quoted 10 lines
a) Analyze the code with a simple frame unwinder along the lines of the
get_wchanimplementation. Simple but possibly fragile as compilers continue to improve.
b) Bite the bullet and use the DWARD 2 frame unwind info and code. More complicated but will be a solid and correct solution albeit larger so not acceptable for small embedded devices.I fear we may have to do both, 2) as the prefered solution and 1) as the fallback solution.
OK, thanks. I will have a look at IA64's implementation and port it if possible.
Quoted 5 lines
Ralf
Quoted 11 lines · Fuxin Zhang
Quoted 9 lines
a) Analyze the code with a simple frame unwinder along the lines of the
get_wchanimplementation. Simple but possibly fragile as compilers continue to improve.
b) Bite the bullet and use the DWARD 2 frame unwind info and code. More complicated but will be a solid and correct solution albeit larger so not acceptable for small embedded devices.I fear we may have to do both, 2) as the prefered solution and 1) as the fallback solution.
OK, thanks. I will have a look at IA64's implementation and port it if possible.
Thanks alot!
Ralf