Skip to contentSkip to conversation

Tool for creating full vmcore

DISCUSSION Corey Minyard 3 participants 6 messages Feb 12, 2014 – Feb 13, 2014 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.

I was trying to use /proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.

The tool is at https://github.com/cminyard/kdump-tool

It can create a physical memory coredump (like the kdump tool) and a virtual memory coredump (either from a physical memory one, or directly from /proc/vmcore and /dev/mem).

Two kernel patches are in the "kernel-patches" directory of the tool, on the master branch:

One adds the valid physical memory ranges to the notes in the core file. Memory doesn't always start from zero, some systems have more than one OS running on the same hardware, and memory often has holes and places that are bad to read from. It seems prudent to pass in these ranges so the coredump is only the memory required. It also adds the physical address of the kernel page directory to the notes, so it can parse the page tables.

The other is MIPS specific. You need a whole bunch of information about the configuration to parse the MIPS page tables.

One cool thing this should be able to do is create a coredump for userspace processes. You can look into kernel memory to find the page table and pass that in to the tool. It should be able to extract the process' resident memory from a physical coredump. Of course, it will only dump resident memory, anything on disk will not be there.

Is this interesting to the kdump community? I'd like to include it in kexec, if possible.

Thanks,

-corey

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#1 lore raw
Quoted 5 lines · Corey Minyard

Hello,

I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.

I am wondering where is it useful. So you find using gdb on kernel core give you better experience as compared to crash which is fully tailored to kernel debugging only?

What about dump filtering? Now a days we use makedumpfile by default and makedumpfile has its own format for dump files which crash understands.

I guess gdb will not work with makedumpfile format files. If that's the case, that means this tool will work with unfiltered core files and generate unfiltered core files. I think working with unfiltered core files is not very practical now a days given large amount of RAM machines typically tend to have.

Thanks
Vivek

Quoted 35 lines

I was trying to use /proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.

The tool is at https://github.com/cminyard/kdump-tool

It can create a physical memory coredump (like the kdump tool) and a virtual memory coredump (either from a physical memory one, or directly from /proc/vmcore and /dev/mem).

Two kernel patches are in the "kernel-patches" directory of the tool, on the master branch:

One adds the valid physical memory ranges to the notes in the core file. Memory doesn't always start from zero, some systems have more than one OS running on the same hardware, and memory often has holes and places that are bad to read from. It seems prudent to pass in these ranges so the coredump is only the memory required. It also adds the physical address of the kernel page directory to the notes, so it can parse the page tables.

The other is MIPS specific. You need a whole bunch of information about the configuration to parse the MIPS page tables.

One cool thing this should be able to do is create a coredump for userspace processes. You can look into kernel memory to find the page table and pass that in to the tool. It should be able to extract the process' resident memory from a physical coredump. Of course, it will only dump resident memory, anything on disk will not be there.

Is this interesting to the kdump community? I'd like to include it in kexec, if possible.

Quoted 8 lines

Thanks,

-corey

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw
In reply to Corey Minyard

Hello,

Quoted 6 lines · Corey Minyard

Hello,

I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.

Quoted 6 lines

I was trying to use /proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.

The tool is at https://github.com/cminyard/kdump-tool

Aside from the other opinions already expressed, I think that the name collision is unfortunate and will cause problems.

kdump-tools is all over Debian and Ubuntu. I would suggest to rename your project to something else.

Kind regards,

...Louis

Signature · 9 lines
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu support: http://canonical.com/support

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#4 lore raw
In reply to a message not in the archive

Oops, forgot to cc kexec list. Doing it now.

Quoted 27 lines · Vivek Goyal

CCing kexec list. Please don't drop the list from the conversation.

Quoted 2 lines · Corey Minyard

Well, I would like to use crash, but it does not support cross debugging and everything we do is cross development.

Hmm.., I will let Dave Anderson comment on that.

Quoted 4 lines

I'm not familiar with filtered core files, but it would be easy enough to just dump kernel memory. It's not a big deal for what I do, we don't deal with huge machines. I suppose that's coming, though.

Well, makedumpfile utility does the kernel filtering and that utility has been growing over time. So for a specific use case it might be trivial but when you try to make it generic, we end up the size of makedumpfile.

So this is like trying to regenerate kcore file out of vmcore where there are ELF headers for vmalloc and vmemmap areas?

Given the fact that this tool is specifically generating kcore, I guess one possile name for the tool could be vmcore-kcore.

I saw some references to /dev/oldmem in git. /dev/oldmem has been deprecated, so you can remove all the code dealing with that and just rely on /proc/vmcore being there.

Quoted 80 lines

I guess if code is small enough, then it might be useful to keep it in a direcotry say kexec-tools/vmcore-kcore. But if it turns out to be big, it might be better to maintain it as a separate project (like makedumpfile).

Thanks
Vivek

Quoted 72 lines

-corey

Quoted 68 lines · Vivek Goyal
Quoted 5 lines · Corey Minyard

Hello,

I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.

I am wondering where is it useful. So you find using gdb on kernel core give you better experience as compared to crash which is fully tailored to kernel debugging only?

What about dump filtering? Now a days we use makedumpfile by default and makedumpfile has its own format for dump files which crash understands.

I guess gdb will not work with makedumpfile format files. If that's the case, that means this tool will work with unfiltered core files and generate unfiltered core files. I think working with unfiltered core files is not very practical now a days given large amount of RAM machines typically tend to have.

Thanks
Vivek

Quoted 43 lines

I was trying to use /proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.

The tool is at https://github.com/cminyard/kdump-tool

It can create a physical memory coredump (like the kdump tool) and a virtual memory coredump (either from a physical memory one, or directly from /proc/vmcore and /dev/mem).

Two kernel patches are in the "kernel-patches" directory of the tool, on the master branch:

One adds the valid physical memory ranges to the notes in the core file. Memory doesn't always start from zero, some systems have more than one OS running on the same hardware, and memory often has holes and places that are bad to read from. It seems prudent to pass in these ranges so the coredump is only the memory required. It also adds the physical address of the kernel page directory to the notes, so it can parse the page tables.

The other is MIPS specific. You need a whole bunch of information about the configuration to parse the MIPS page tables.

One cool thing this should be able to do is create a coredump for userspace processes. You can look into kernel memory to find the page table and pass that in to the tool. It should be able to extract the process' resident memory from a physical coredump. Of course, it will only dump resident memory, anything on disk will not be there.

Is this interesting to the kdump community? I'd like to include it in kexec, if possible.

Thanks,

-corey

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw
Quoted 1 line · Vivek Goyal

Oops, forgot to cc kexec list. Doing it now.

Yeah, I forgot that, too.

Quoted 8 lines · Vivek Goyal
Quoted 6 lines

CCing kexec list. Please don't drop the list from the conversation.

Quoted 2 lines · Corey Minyard

Well, I would like to use crash, but it does not support cross debugging and everything we do is cross development.

Hmm.., I will let Dave Anderson comment on that.

Dave has already commented on it several times. It's not going to happen.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

I'm not familiar with filtered core files, but it would be easy enough to just dump kernel memory. It's not a big deal for what I do, we don't deal with huge machines. I suppose that's coming, though.

Well, makedumpfile utility does the kernel filtering and that utility has been growing over time. So for a specific use case it might be trivial but when you try to make it generic, we end up the size of makedumpfile.

So this is like trying to regenerate kcore file out of vmcore where there are ELF headers for vmalloc and vmemmap areas?

Basically, that's it. It walks the page tables and generates an ELF file from that, so it will pick up anything in the page tables. It's theoretically possible to walk user page tables, too. That means there's arch-specific code. It's pretty trivial for everything but MIPS, which implements page tables in software and has too many options.

Oh, I forgot to mention earlier that i386 is not tested yet. x86_64, ARM, and MIPS are tested. I need to get to i386 soon.

Quoted 3 lines
Quoted 3 lines

Given the fact that this tool is specifically generating kcore, I guess one possile name for the tool could be vmcore-kcore.

Yeah, I've been leaning somewhere in that direction for the name.

Quoted 4 lines
Quoted 4 lines

I saw some references to /dev/oldmem in git. /dev/oldmem has been deprecated, so you can remove all the code dealing with that and just rely on /proc/vmcore being there.

Yes, I found that out and removed it a while ago. Makes sense.

Quoted 5 lines
Quoted 5 lines

I guess if code is small enough, then it might be useful to keep it in a direcotry say kexec-tools/vmcore-kcore. But if it turns out to be big, it might be better to maintain it as a separate project (like makedumpfile).

Yeah, it's a little big, but not huge. Most of the code is the elf part, which is in it's own file and could be a separate library. I'm fine either way. It will get bigger as new arch-specific code is added.

Thanks,

-corey

Quoted 63 lines
Quoted 63 lines

Thanks
Vivek

Quoted 60 lines

-corey

Quoted 57 lines · Vivek Goyal
Quoted 5 lines · Corey Minyard

Hello,

I've written a tool that takes an coredump of physical kernel memory and converts it to a coredump of virtual kernel memory that can be directly used by gdb to debug a kernel.

I am wondering where is it useful. So you find using gdb on kernel core give you better experience as compared to crash which is fully tailored to kernel debugging only?

What about dump filtering? Now a days we use makedumpfile by default and makedumpfile has its own format for dump files which crash understands.

I guess gdb will not work with makedumpfile format files. If that's the case, that means this tool will work with unfiltered core files and generate unfiltered core files. I think working with unfiltered core files is not very practical now a days given large amount of RAM machines typically tend to have.

Thanks
Vivek

Quoted 34 lines

I was trying to use /proc/vmcore, but that did not include any of vmalloc memory or vmemmap, which made it kind of useless. I thought about adding all that to the vmcore, but that didn't seem like the proper way to do things.

The tool is at https://github.com/cminyard/kdump-tool

It can create a physical memory coredump (like the kdump tool) and a virtual memory coredump (either from a physical memory one, or directly from /proc/vmcore and /dev/mem).

Two kernel patches are in the "kernel-patches" directory of the tool, on the master branch:

One adds the valid physical memory ranges to the notes in the core file. Memory doesn't always start from zero, some systems have more than one OS running on the same hardware, and memory often has holes and places that are bad to read from. It seems prudent to pass in these ranges so the coredump is only the memory required. It also adds the physical address of the kernel page directory to the notes, so it can parse the page tables.

The other is MIPS specific. You need a whole bunch of information about the configuration to parse the MIPS page tables.

One cool thing this should be able to do is create a coredump for userspace processes. You can look into kernel memory to find the page table and pass that in to the tool. It should be able to extract the process' resident memory from a physical coredump. Of course, it will only dump resident memory, anything on disk will not be there.

Is this interesting to the kdump community? I'd like to include it in kexec, if possible.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Thanks,

-corey

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

On Thu, Feb 13, 2014 at 10:45:16AM -0600, Corey Minyard wrote:

[..]

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

I guess if code is small enough, then it might be useful to keep it in a direcotry say kexec-tools/vmcore-kcore. But if it turns out to be big, it might be better to maintain it as a separate project (like makedumpfile).

Yeah, it's a little big, but not huge. Most of the code is the elf part, which is in it's own file and could be a separate library. I'm fine either way. It will get bigger as new arch-specific code is added.

This does sound like significant code when other arch support is added. I am beginning to think if it better to manage it as a separate project in separate git tree.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec