Tool for creating full vmcore
Subsystems (2)
Hello,
I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.
I was trying to use /proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.
The tool is at https://github.com/cminyard/kdump-tool
It can create a physical memory coredump (like the kdump tool) and a virtual memory coredump (either from a physical memory one, or directly from /proc/vmcore and /dev/mem).
Two kernel patches are in the "kernel-patches" directory of the tool, on the master branch:
One adds the valid physical memory ranges to the notes in the core file. Memory doesn't always start from zero, some systems have more than one OS running on the same hardware, and memory often has holes and places that are bad to read from. It seems prudent to pass in these ranges so the coredump is only the memory required. It also adds the physical address of the kernel page directory to the notes, so it can parse the page tables.
The other is MIPS specific. You need a whole bunch of information about the configuration to parse the MIPS page tables.
One cool thing this should be able to do is create a coredump for userspace processes. You can look into kernel memory to find the page table and pass that in to the tool. It should be able to extract the process' resident memory from a physical coredump. Of course, it will only dump resident memory, anything on disk will not be there.
Is this interesting to the kdump community? I'd like to include it in kexec, if possible.
Thanks,
-corey
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 5 lines · Corey Minyard
Hello,
I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.
I am wondering where is it useful. So you find using gdb on kernel core give you better experience as compared to crash which is fully tailored to kernel debugging only?
What about dump filtering? Now a days we use makedumpfile by default and makedumpfile has its own format for dump files which crash understands.
I guess gdb will not work with makedumpfile format files. If that's the case, that means this tool will work with unfiltered core files and generate unfiltered core files. I think working with unfiltered core files is not very practical now a days given large amount of RAM machines typically tend to have.
Thanks
Vivek
Quoted 35 lines
I was trying to use
/proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.The tool is at https://github.com/cminyard/kdump-tool
It can create a physical memory coredump (like the kdump tool) and a virtual memory coredump (either from a physical memory one, or directly from
/proc/vmcoreand/dev/mem).Two kernel patches are in the "kernel-patches" directory of the tool, on the master branch:
One adds the valid physical memory ranges to the notes in the core file. Memory doesn't always start from zero, some systems have more than one OS running on the same hardware, and memory often has holes and places that are bad to read from. It seems prudent to pass in these ranges so the coredump is only the memory required. It also adds the physical address of the kernel page directory to the notes, so it can parse the page tables.
The other is MIPS specific. You need a whole bunch of information about the configuration to parse the MIPS page tables.
One cool thing this should be able to do is create a coredump for userspace processes. You can look into kernel memory to find the page table and pass that in to the tool. It should be able to extract the process' resident memory from a physical coredump. Of course, it will only dump resident memory, anything on disk will not be there.
Is this interesting to the kdump community? I'd like to include it in kexec, if possible.
Quoted 8 lines
Thanks,
-corey
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
A message that is not in the archive (sent off-list or not yet received): CADq4U7+K-Zi2Ckb-mfDWFf50Oq7hbTWR6H92=Jx=STqWSJu89g@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Oops, forgot to cc kexec list. Doing it now.
Quoted 27 lines · Vivek Goyal
CCing kexec list. Please don't drop the list from the conversation.
Quoted 2 lines · Corey Minyard
Well, I would like to use crash, but it does not support cross debugging and everything we do is cross development.
Hmm.., I will let Dave Anderson comment on that.
Quoted 4 lines
I'm not familiar with filtered core files, but it would be easy enough to just dump kernel memory. It's not a big deal for what I do, we don't deal with huge machines. I suppose that's coming, though.
Well, makedumpfile utility does the kernel filtering and that utility has been growing over time. So for a specific use case it might be trivial but when you try to make it generic, we end up the size of makedumpfile.
So this is like trying to regenerate kcore file out of vmcore where there are ELF headers for vmalloc and vmemmap areas?
Given the fact that this tool is specifically generating kcore, I guess one possile name for the tool could be vmcore-kcore.
I saw some references to
/dev/oldmemin git./dev/oldmemhas been deprecated, so you can remove all the code dealing with that and just rely on/proc/vmcorebeing there.
Quoted 80 lines
I guess if code is small enough, then it might be useful to keep it in a direcotry say kexec-tools/vmcore-kcore. But if it turns out to be big, it might be better to maintain it as a separate project (like makedumpfile).
Thanks
VivekQuoted 72 lines
-corey
Quoted 68 lines · Vivek Goyal
Quoted 5 lines · Corey Minyard
Hello,
I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.
I am wondering where is it useful. So you find using gdb on kernel core give you better experience as compared to crash which is fully tailored to kernel debugging only?
What about dump filtering? Now a days we use makedumpfile by default and makedumpfile has its own format for dump files which crash understands.
I guess gdb will not work with makedumpfile format files. If that's the case, that means this tool will work with unfiltered core files and generate unfiltered core files. I think working with unfiltered core files is not very practical now a days given large amount of RAM machines typically tend to have.
Thanks
VivekQuoted 43 lines
I was trying to use
/proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.The tool is at https://github.com/cminyard/kdump-tool
It can create a physical memory coredump (like the kdump tool) and a virtual memory coredump (either from a physical memory one, or directly from
/proc/vmcoreand/dev/mem).Two kernel patches are in the "kernel-patches" directory of the tool, on the master branch:
One adds the valid physical memory ranges to the notes in the core file. Memory doesn't always start from zero, some systems have more than one OS running on the same hardware, and memory often has holes and places that are bad to read from. It seems prudent to pass in these ranges so the coredump is only the memory required. It also adds the physical address of the kernel page directory to the notes, so it can parse the page tables.
The other is MIPS specific. You need a whole bunch of information about the configuration to parse the MIPS page tables.
One cool thing this should be able to do is create a coredump for userspace processes. You can look into kernel memory to find the page table and pass that in to the tool. It should be able to extract the process' resident memory from a physical coredump. Of course, it will only dump resident memory, anything on disk will not be there.
Is this interesting to the kdump community? I'd like to include it in kexec, if possible.
Thanks,
-corey
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 1 line · Vivek Goyal
Oops, forgot to cc kexec list. Doing it now.
Yeah, I forgot that, too.
Quoted 8 lines · Vivek Goyal
Quoted 6 lines
CCing kexec list. Please don't drop the list from the conversation.
Quoted 2 lines · Corey Minyard
Well, I would like to use crash, but it does not support cross debugging and everything we do is cross development.
Hmm.., I will let Dave Anderson comment on that.
Dave has already commented on it several times. It's not going to happen.
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
I'm not familiar with filtered core files, but it would be easy enough to just dump kernel memory. It's not a big deal for what I do, we don't deal with huge machines. I suppose that's coming, though.
Well, makedumpfile utility does the kernel filtering and that utility has been growing over time. So for a specific use case it might be trivial but when you try to make it generic, we end up the size of makedumpfile.
So this is like trying to regenerate kcore file out of vmcore where there are ELF headers for vmalloc and vmemmap areas?
Basically, that's it. It walks the page tables and generates an ELF file from that, so it will pick up anything in the page tables. It's theoretically possible to walk user page tables, too. That means there's arch-specific code. It's pretty trivial for everything but MIPS, which implements page tables in software and has too many options.
Oh, I forgot to mention earlier that i386 is not tested yet. x86_64, ARM, and MIPS are tested. I need to get to i386 soon.
Quoted 3 lines
Quoted 3 lines
Given the fact that this tool is specifically generating kcore, I guess one possile name for the tool could be vmcore-kcore.
Yeah, I've been leaning somewhere in that direction for the name.
Quoted 4 lines
Quoted 4 lines
I saw some references to
/dev/oldmemin git./dev/oldmemhas been deprecated, so you can remove all the code dealing with that and just rely on/proc/vmcorebeing there.
Yes, I found that out and removed it a while ago. Makes sense.
Quoted 5 lines
Quoted 5 lines
I guess if code is small enough, then it might be useful to keep it in a direcotry say kexec-tools/vmcore-kcore. But if it turns out to be big, it might be better to maintain it as a separate project (like makedumpfile).
Yeah, it's a little big, but not huge. Most of the code is the elf part, which is in it's own file and could be a separate library. I'm fine either way. It will get bigger as new arch-specific code is added.
Thanks,
-corey
Quoted 63 lines
Quoted 63 lines
Thanks
VivekQuoted 60 lines
-corey
Quoted 57 lines · Vivek Goyal
Quoted 5 lines · Corey Minyard
Hello,
I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.
I am wondering where is it useful. So you find using gdb on kernel core give you better experience as compared to crash which is fully tailored to kernel debugging only?
What about dump filtering? Now a days we use makedumpfile by default and makedumpfile has its own format for dump files which crash understands.
I guess gdb will not work with makedumpfile format files. If that's the case, that means this tool will work with unfiltered core files and generate unfiltered core files. I think working with unfiltered core files is not very practical now a days given large amount of RAM machines typically tend to have.
Thanks
VivekQuoted 34 lines
I was trying to use
/proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.The tool is at https://github.com/cminyard/kdump-tool
It can create a physical memory coredump (like the kdump tool) and a virtual memory coredump (either from a physical memory one, or directly from
/proc/vmcoreand/dev/mem).Two kernel patches are in the "kernel-patches" directory of the tool, on the master branch:
One adds the valid physical memory ranges to the notes in the core file. Memory doesn't always start from zero, some systems have more than one OS running on the same hardware, and memory often has holes and places that are bad to read from. It seems prudent to pass in these ranges so the coredump is only the memory required. It also adds the physical address of the kernel page directory to the notes, so it can parse the page tables.
The other is MIPS specific. You need a whole bunch of information about the configuration to parse the MIPS page tables.
One cool thing this should be able to do is create a coredump for userspace processes. You can look into kernel memory to find the page table and pass that in to the tool. It should be able to extract the process' resident memory from a physical coredump. Of course, it will only dump resident memory, anything on disk will not be there.
Is this interesting to the kdump community? I'd like to include it in kexec, if possible.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks,
-corey
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
On Thu, Feb 13, 2014 at 10:45:16AM -0600, Corey Minyard wrote:
[..]
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
I guess if code is small enough, then it might be useful to keep it in a direcotry say kexec-tools/vmcore-kcore. But if it turns out to be big, it might be better to maintain it as a separate project (like makedumpfile).
Yeah, it's a little big, but not huge. Most of the code is the elf part, which is in it's own file and could be a separate library. I'm fine either way. It will get bigger as new arch-specific code is added.
This does sound like significant code when other arch support is added. I am beginning to think if it better to manage it as a separate project in separate git tree.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello,
Quoted 6 lines · Corey Minyard
Hello,
I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.
Quoted 6 lines
I was trying to use
/proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.The tool is at https://github.com/cminyard/kdump-tool
Aside from the other opinions already expressed, I think that the name collision is unfortunate and will cause problems.
kdump-tools is all over Debian and Ubuntu. I would suggest to rename your project to something else.
Kind regards,
...Louis
Signature · 9 lines
Louis Bouchard Software engineer, Cloud & Sustaining eng. Canonical Ltd Ubuntu support: http://canonical.com/support _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec