Skip to contentSkip to conversation

tools to dump guest memory and generate core file

DISCUSSION david ahern 3 participants 6 messages Mar 17, 2008 – Mar 19, 2008 KVM

lore
Share & tools
lore
#0 lore raw

Does anyone know of tools that can dump memory for a qemu guest with addresses as seen by the guest and generate a core file? For instance, say you know the guest is running a 32-bit linux kernel with a 1G/3G split. Then you would want to dump 1G of memory starting 0xc0000000 and create an ELF core file. The core file could then be analyzed using tools like crash (or gdb for the truly adventurous).

I know VMware can take a snapshot and generate such a core file. Does a similar tool exist for qemu? I see that the qemu console supports a raw memory dump, so it should be possible. (Note I am not talking about a core file of the qemu process, but rather a core file based on guest memory addressing.)

david

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

#1 lore raw
Quoted 7 lines · david ahern

Does anyone know of tools that can dump memory for a qemu guest with addresses as seen by the guest and generate a core file? For instance, say you know the guest is running a 32-bit linux kernel with a 1G/3G split. Then you would want to dump 1G of memory starting 0xc0000000 and create an ELF core file. The core file could then be analyzed using tools like crash (or gdb for the truly adventurous).

Quoted 6 lines

I know VMware can take a snapshot and generate such a core file. Does a similar tool exist for qemu? I see that the qemu console supports a raw memory dump, so it should be possible. (Note I am not talking about a core file of the qemu process, but rather a core file based on guest memory addressing.)

You might try connecting with gdb and using the generate-core-file command.

Signature · 7 lines
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
#2 lore raw

Attaching gdb to qemu you work with addresses as seen by the qemu process; the idea is to work with addresses as seen inside the guest.

For example, in the qemu console you can examine guest kernel memory such as task structs using guest kernel based addresses:

(qemu) x /128w 0xc0327a80
00000000c0327a80: 0x00000000 0xc039a000 0x00000002 0x00000000 00000000c0327a90: 0x00000000 0xffffffff 0x0000008c 0x00000078 00000000c0327aa0: 0xc0327aa0 0xc0327aa0 0x00000000 0x00000000 00000000c0327ab0: 0xffffff9b 0xdae71a00 0x003d098c 0xdae71a00 00000000c0327ac0: 0x003d098c 0x00000000 0x00000000 0xffffffff
...

where 0xc0327a80 is the address of the first task (init_task symbol). This form is really painful to decipher much less follow the task list.

Now, if you attach gdb to the qemu process,

gdb /usr/local/bin/qemu-system-x86_64 2346

and run that same memory dump request you get:

(gdb) x /128w 0xc0327a80
0xc0327a80: Cannot access memory at address 0xc0327a80

because from qemu's perspective 0xc0327a80 is not a valid address.

Ideally it would be much more convenient to dump the guest memory based on its view of addresses and then run crash. e.g.,

crash> foreach task

PID: 0 TASK: c0327a80 CPU: 0 COMMAND: "swapper"
struct task_struct {
state = 0,
thread_info = 0xc039a000,
usage = {
counter = 2
...

Does that make sense?

Such a tool would be really useful to figure out why for example a guest appears to have locked up or why it is sucking up host cpu time.

david

Quoted 22 lines · Avi Kivity
Quoted 11 lines · david ahern

Does anyone know of tools that can dump memory for a qemu guest with addresses
as seen by the guest and generate a core file? For instance, say you know the
guest is running a 32-bit linux kernel with a 1G/3G split. Then you would want
to dump 1G of memory starting 0xc0000000 and create an ELF core file. The core
file could then be analyzed using tools like crash (or gdb for the truly adventurous).

Quoted 9 lines

I know VMware can take a snapshot and generate such a core file. Does a similar
tool exist for qemu? I see that the qemu console supports a raw memory dump, so
it should be possible. (Note I am not talking about a core file of the qemu
process, but rather a core file based on guest memory addressing.)

Quoted 2 lines

You might try connecting with gdb and using the generate-core-file command.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

#3 lore raw
Quoted 14 lines · david ahern

Attaching gdb to qemu you work with addresses as seen by the qemu process; the idea is to work with addresses as seen inside the guest.

For example, in the qemu console you can examine guest kernel memory such as task structs using guest kernel based addresses:

(qemu) x /128w 0xc0327a80
00000000c0327a80: 0x00000000 0xc039a000 0x00000002 0x00000000 00000000c0327a90: 0x00000000 0xffffffff 0x0000008c 0x00000078 00000000c0327aa0: 0xc0327aa0 0xc0327aa0 0x00000000 0x00000000 00000000c0327ab0: 0xffffff9b 0xdae71a00 0x003d098c 0xdae71a00 00000000c0327ac0: 0x003d098c 0x00000000 0x00000000 0xffffffff
...

Quoted 8 lines

where 0xc0327a80 is the address of the first task (init_task symbol). This form is really painful to decipher much less follow the task list.

Now, if you attach gdb to the qemu process,

gdb /usr/local/bin/qemu-system-x86_64 2346

I meant connecting to the gdb stub in qemu that represents the guest:

(gdb) target remote localhost:1234

Of course, it means starting qemu with the gdb stub enabled. We might add a monitor command to start it after the fact.

Signature · 7 lines
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
#4 lore raw
Quoted 11 lines · Avi Kivity
Quoted 9 lines · david ahern

Attaching gdb to qemu you work with addresses as seen by the qemu process; the idea is to work with addresses as seen inside the guest.

Now, if you attach gdb to the qemu process,

gdb /usr/local/bin/qemu-system-x86_64 2346

Quoted 7 lines

I meant connecting to the gdb stub in qemu that represents the guest:

(gdb) target remote localhost:1234

Of course, it means starting qemu with the gdb stub enabled. We might add a monitor command to start it after the fact.

gdbserver is the monitor command you're looking for:
(qemu) help gdbserver
gdbserver [port] -- start gdbserver session (default port=1234)

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

#5 lore raw

That's what I was looking for.

thanks,
david

Quoted 14 lines · Uri Lublin
Quoted 13 lines · Avi Kivity
Quoted 9 lines · david ahern

Attaching gdb to qemu you work with addresses as seen by the qemu process; the
idea is to work with addresses as seen inside the guest.

Now, if you attach gdb to the qemu process,

gdb /usr/local/bin/qemu-system-x86_64 2346

I meant connecting to the gdb stub in qemu that represents the guest:

Quoted 10 lines
Quoted 5 lines

(gdb) target remote localhost:1234

Of course, it means starting qemu with the gdb stub enabled. We might add a monitor command to start it after the fact.

gdbserver is the monitor command you're looking for:
(qemu) help gdbserver
gdbserver [port] -- start gdbserver session (default port=1234)

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/