Accessing Thread Information in kernel crash dumps with ddd+gdb
Subsystems (2)
Hey Guys:
I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed with the visibility of local variables on the threads/tasks not currently running on CPUs. Both crash, and the gdb macros that you guys wrote, show the most important stuff but I'd prefer to be able to see everything with gdb/ddd as I can with kgdb; including all local variables and formal parameters at each stack frame.
A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp and $sp from the saved information in the U-block to view a process. I wish gdb would allow be to run your macros, btt for example, and extract the stackp from task.thread.esp assign it temporally to $sp for the current task,
do the backtrace command and see everything. Changing $sp and $fp for a while
like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to browse the
stack formals and locals would be nice. Just doing a 'set write on' isn't sufficient,
gdb wants a process and I can't see to satisfy it with simply setting the current
thread.
I was wondering if any of you guys have been thinking of anything like this and had and hacks or ideas on how to see the locals and formals for all tasks.
One thought I had was a minor hack of the kexec code to do something like your gdb macros
and walk thru the task list and then append a ELF Notes, like done by crash_save_this_cpu(),
for each task. I have no idea if gdb has a limit on the number of elf_prstatus structures
that can be provided. I suppose I'd leave it a KEXEC config variable to enable this, as
some would argue that it's not as save as simply saving the regs for the active CPUs.
This would leave 'info threads' with gdb similar to 'ps' with crash and virtually identical
to the experience with kgdb.
-piet
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Ingo,
It appears you already did a
thread-aware coredump
back in Oct 2002:
http://lwn.net/Articles/12035/
Your patch appears to dump the registers for all task. What happen with your idea?
-piet
Quoted 31 lines
Hey Guys:
I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed with the visibility of local variables on the threads/tasks not currently running on CPUs. Both crash, and the gdb macros that you guys wrote, show the most important stuff but I'd prefer to be able to see everything with gdb/ddd as I can with kgdb; including all local variables and formal parameters at each stack frame.
A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp
and $sp from the saved information in the U-block to view a process.
I wish gdb would allow be to run your macros, btt for example, and extract the stackp from task.thread.esp assign it temporally to $sp for the current task, do the backtrace command and see everything. Changing $sp and $fp for a while like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to browse the stack formals and locals would be nice. Just doing a 'set write on' isn't sufficient, gdb wants a process and I can't see to satisfy it with simply setting the current thread.I was wondering if any of you guys have been thinking of anything like this and had and hacks or ideas on how to see the locals and formals for all tasks.
One thought I had was a minor hack of the kexec code to do something like your gdb macros and walk thru the task list and then append a ELF Notes, like done by
crash_save_this_cpu(), for each task. I have no idea if gdb has a limit on the number ofelf_prstatusstructures that can be provided. I suppose I'd leave it a KEXEC config variable to enable this, as some would argue that it's not as save as simply saving the regs for the active CPUs. This would leave 'info threads' with gdb similar to 'ps' with crash and virtually identical to the experience with kgdb.
Quoted 2 lines
-piet
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 29 lines · Piet Delaney
Hey Guys:
I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed with the visibility of local variables on the threads/tasks not currently running on CPUs. Both crash, and the gdb macros that you guys wrote, show the most important stuff but I'd prefer to be able to see everything with gdb/ddd as I can with kgdb; including all local variables and formal parameters at each stack frame.
A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp and $sp from the saved information in the U-block to view a process. I wish gdb would allow be to run your macros, btt for example, and extract the stackp from task.thread.esp assign it temporally to $sp for the current task,
do the backtrace command and see everything. Changing $sp and $fp for a while
like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to browse the
stack formals and locals would be nice. Just doing a 'set write on' isn't sufficient,
gdb wants a process and I can't see to satisfy it with simply setting the current
thread.I was wondering if any of you guys have been thinking of anything like this and had and hacks or ideas on how to see the locals and formals for all tasks.
Quoted 13 lines
One thought I had was a minor hack of the kexec code to do something like your gdb macros
and walk thru the task list and then append a ELF Notes, like done bycrash_save_this_cpu(),
for each task. I have no idea if gdb has a limit on the number ofelf_prstatusstructures
that can be provided. I suppose I'd leave it a KEXEC config variable to enable this, as
some would argue that it's not as save as simply saving the regs for the active CPUs.
This would leave 'info threads' with gdb similar to 'ps' with crash and virtually identical
to the experience with kgdb.
IIUC, you are suggesting that we create elf notes even for non-active tasks in vmcore. We should not be doing that.
- It is not safe to traverse through task list after system has crashed.
- We reserve the memory for elf notes at system boot. At that time we have no idea how many task system will have at the time of crash.
I think following can be a way forward for your requirement.
- Either gdb should provide a SunOS kind of facility where one can provide stack pointer and switch the task context. ( I don't know if there is already a way to do that).
- Or one can write a user space tool, which parses original vmcore, walks through task list, prepare elf notes for all the tasks and emit a new vmcore which is fetched to gdb.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 36 lines · Vivek Goyal
Quoted 23 lines · Piet Delaney
Hey Guys:
I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed with the visibility of local variables on the threads/tasks not currently running on CPUs. Both crash, and the gdb macros that you guys wrote, show the most important stuff but I'd prefer to be able to see everything with gdb/ddd as I can with kgdb; including all local variables and formal parameters at each stack frame.
A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp
and $sp from the saved information in the U-block to view a process.
I wish gdb would allow be to run your macros, btt for example, and extract the stackp from task.thread.esp assign it temporally to $sp for the current task, do the backtrace command and see everything. Changing $sp and $fp for a while like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to browse the stack formals and locals would be nice. Just doing a 'set write on' isn't sufficient, gdb wants a process and I can't see to satisfy it with simply setting the current thread.I was wondering if any of you guys have been thinking of anything like this and had and hacks or ideas on how to see the locals and formals for all tasks.
Quoted 9 lines
One thought I had was a minor hack of the kexec code to do something like your gdb macros and walk thru the task list and then append a ELF Notes, like done by
crash_save_this_cpu(), for each task. I have no idea if gdb has a limit on the number ofelf_prstatusstructures that can be provided. I suppose I'd leave it a KEXEC config variable to enable this, as some would argue that it's not as save as simply saving the regs for the active CPUs. This would leave 'info threads' with gdb similar to 'ps' with crash and virtually identical to the experience with kgdb.
Quoted 3 lines
IIUC, you are suggesting that we create elf notes even for non-active tasks in vmcore.
Yep.
Quoted 5 lines
We should not be doing that.
- It is not safe to traverse through task list after system has crashed.
I agree it's not 100% safe, but for many developers it's a risk worth taking. For example we make a lot of changes in the TCP/IP stack for implementing a proxy for filtering network traffic. Virtually all bugs are ones we make in the networking code and having a precise view of each stack would be helpful. It's extremely unlikely that the task structures have been whacked in our case; and likely for many other developers.
Quoted 4 lines
- We reserve the memory for elf notes at system boot. At that time we have no idea how many task system will have at the time of crash.
How about reserving memory with each task structure for the ELF notes? With Solaris I reserved a page for each CPU to be able to store the register windows during a stack overflow so I could map it in during the stack overflow. It's not unreasonable to allocate the space with the task; especially if it's a kernel config option. If no one used it we could remove it and go for an alternate approach.
Quoted 7 lines
I think following can be a way forward for your requirement.
- Either gdb should provide a SunOS kind of facility where one can provide stack pointer and switch the task context. ( I don't know if there is already a way to do that).
I'll talk on the gdb mailing list about it. Perhaps we could initially get it in as a developer maintance function to allow the registers to be changed for a thread in a core dump.
Quoted 5 lines
- Or one can write a user space tool, which parses original vmcore, walks through task list, prepare elf notes for all the tasks and emit a new vmcore which is fetched to gdb.
Sounds difficult. I was talking to Dave Anderson about having crash provide task info to his embedded gdb process but he wasn't supportive of that approach.
Having a peace of code that has to walk thru the task list and modify the elf notes seems a lot harder than just having the kernel do it. More code that has to be maintained.
Isn't there precedence with other kernels like freebsd and Solaris having the core dumps provide task information? If it was conditional we could see just how many developers find the risk of causing a problem during the crash dump a issue.
Looks to me like the size of the ELF notes is quite small and wouldn't be a significant burden to add to the task structure.
-piet
Quoted 4 lines
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 7 lines · Piet Delaney
I agree it's not 100% safe, but for many developers it's a risk worth taking. For example we make a lot of changes in the TCP/IP stack for implementing a proxy for filtering network traffic. Virtually all bugs are ones we make in the networking code and having a precise view of each stack would be helpful. It's extremely unlikely that the task structures have been whacked in our case; and likely for many other developers.
Kernel crash dumps in development?
Relying on debuggers in kernel development?
There is something that smells horribly like the debug it until it works version of code development instead of stopping and understanding what the code does.
Quoted 2 lines
How about reserving memory with each task structure for the ELF notes?
Doesn't fly we need to know where the memory is before the kernel crashes. You have just required the task list walker you are trying to avoid be present to gather up your ELF notes. If you need the walker anyway there is no point in generating the notes a priori.
Quoted 6 lines
With Solaris I reserved a page for each CPU to be
able to store the register windows during a stack overflow so I could map it in during the stack overflow. It's not unreasonable to allocate the space with the task; especially if it's a kernel config option. If no one used it we could remove it and go for an alternate approach.
In some sense we have already been where you are suggesting. Having a nice crash dump facility in the kernel proved to fail to capture crash dumps in real world failure scenarios.
Quoted 19 lines
I think following can be a way forward for your requirement.
- Either gdb should provide a SunOS kind of facility where one can provide stack pointer and switch the task context. ( I don't know if there is already a way to do that).
I'll talk on the gdb mailing list about it. Perhaps we could initially get it in as a developer maintance function to allow the registers to be changed for a thread in a core dump.
- Or one can write a user space tool, which parses original vmcore, walks through task list, prepare elf notes for all the tasks and emit a new vmcore which is fetched to gdb.
Quoted 7 lines
Sounds difficult. I was talking to Dave Anderson about having crash provide task info to his embedded gdb process but he wasn't supportive of that approach.
Having a peace of code that has to walk thru the task list and modify the elf notes seems a lot harder than just having the kernel do it. More code that has to be maintained.
Totally Wrong.
It is critical that the kexec on panic code path remains as small as possible to maintain reliability.
Doing just about anything elsewhere is less work then on that code path because we need to assume just about any part of the kernel is busted and wrong.
Further there is no significant benefit (except possibly the code being more visible to kernel developers) to doing the work in the kernel. If that is a real concern placing a debug utility in the kernel source that finds the task list and walks it should provide the same benefit.
Debuggers do have perfect information about the structure layouts, as normal dwarf2 debugging information can record it.
So forget the notion of doing this on the kexec on panic code path and see if you can make one of one of Vivek's ideas fly. If nothing is busted walking the task list is just a stupid linked list walk easy, and not at all fatal if you get it wrong in user space, and easy to fix. A mysterious failure that takes weeks of work if you are not perfectly paranoid in the kexec on panic code path.
Eric
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 3 lines · Vivek Goyal
On Thu, Apr 17, 2008 at 05:16:55PM -0700, Piet Delaney wrote:
I was thinking over lunch that the largest risk to whacking the task list might be a stack overflows. Perhaps a list of crash notes could be maintained with the allocation/freeing of the task structures. With the notes being so small it might not be too bad just to allocate an array based on some configurable maximum. Being able to dump say up to 1000 task notes would be more that sufficient for our systems. The wasted memory of allocating more memory than actually needed would be a small cost to ensure well debugged kernels.
-piet
Quoted 46 lines
Quoted 23 lines
Hey Guys:
I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed with the visibility of local variables on the threads/tasks not currently running on CPUs. Both crash, and the gdb macros that you guys wrote, show the most important stuff but I'd prefer to be able to see everything with gdb/ddd as I can with kgdb; including all local variables and formal parameters at each stack frame.
A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp
and $sp from the saved information in the U-block to view a process.
I wish gdb would allow be to run your macros, btt for example, and extract the stackp from task.thread.esp assign it temporally to $sp for the current task, do the backtrace command and see everything. Changing $sp and $fp for a while like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to browse the stack formals and locals would be nice. Just doing a 'set write on' isn't sufficient, gdb wants a process and I can't see to satisfy it with simply setting the current thread.I was wondering if any of you guys have been thinking of anything like this and had and hacks or ideas on how to see the locals and formals for all tasks.
Quoted 9 lines
One thought I had was a minor hack of the kexec code to do something like your gdb macros and walk thru the task list and then append a ELF Notes, like done by
crash_save_this_cpu(), for each task. I have no idea if gdb has a limit on the number ofelf_prstatusstructures that can be provided. I suppose I'd leave it a KEXEC config variable to enable this, as some would argue that it's not as save as simply saving the regs for the active CPUs. This would leave 'info threads' with gdb similar to 'ps' with crash and virtually identical to the experience with kgdb.IIUC, you are suggesting that we create elf notes even for non-active tasks in vmcore. We should not be doing that.
- It is not safe to traverse through task list after system has crashed.
- We reserve the memory for elf notes at system boot. At that time we have no idea how many task system will have at the time of crash.I think following can be a way forward for your requirement.
- Either gdb should provide a SunOS kind of facility where one can provide stack pointer and switch the task context. ( I don't know if there is already a way to do that).
Quoted 8 lines
- Or one can write a user space tool, which parses original vmcore, walks through task list, prepare elf notes for all the tasks and emit a new vmcore which is fetched to gdb.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec