Skip to contentSkip to conversation

uniquely identifying KDUMP files that originate from QEMU

DISCUSSION Laszlo Ersek 6 participants 24 messages Nov 11, 2014 – Nov 12, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Alas, the QEMU notes are not present in the KDUMP-format vmcores that QEMU produces (they cannot be), hence crash has no way to tell apart such files from those generated by genuine kdump. As an end result, "crash" cannot automatically find the phys_base of OVMF-based Linux vmcores.

Dave suggested that a new flag, or a special phys_base value (like ~0UL) be introduced as a distinguishing mark for QEMU-produced kdumps.

Implementing this in QEMU wouldn't be hard. The big question is compatibility -- whose analysis tools would be broken by a (phys_base == ~0UL) setting, or by a new flag?

Note that this change would affect SeaBIOS-based vmcores too. QEMU can't (and shouldn't) discriminate the vmcores it dumps based on guest firmware. (If QEMU did that, then it might as well try to figure out the real phys_base value, which is clearly out of scope for qemu. One of the selling points of the paging=false dump is that it doesn't involve parsing guest RAM.)

Thanks
Laszlo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#1 lore raw
Quoted 8 lines · Laszlo Ersek

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

Quoted 7 lines

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

Presumably this is also not going to work for machines other than the x86 PC, where physical memory may well not start at address zero...

-- PMM

#2 lore raw
In reply to Laszlo Ersek
Quoted 32 lines · On Tue, 11 Nov 2014 12:22:52 +0100 Laszlo Ersek <[email protected]> wrote:

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Quoted 7 lines

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Alas, the QEMU notes are not present in the KDUMP-format vmcores that QEMU produces (they cannot be),

Why? Since KDUMP format version 4, the complete ELF notes can be stored in the file (see offset_note, size_note fields in the sub-header).

Petr Tesarik

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw
In reply to Laszlo Ersek
Quoted 30 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

Quoted 6 lines

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

What advantages does KDUMP have over ELF?

Thanks,
Chris

Signature · 3 lines
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#3 lore raw
In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>
Subject: Re: uniquely identifying KDUMP files that originate from QEMU
Date: Tue, 11 Nov 2014 13:09:13 +0100

Quoted 40 lines
Quoted 37 lines · On Tue, 11 Nov 2014 12:22:52 +0100 Laszlo Ersek <[email protected]> wrote:

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Quoted 6 lines
Quoted 2 lines

Alas, the QEMU notes are not present in the KDUMP-format vmcores that QEMU produces (they cannot be),

Why? Since KDUMP format version 4, the complete ELF notes can be stored in the file (see offset_note, size_note fields in the sub-header).

Yes, the QEMU notes is present in kdump-compressed format. But phys_base cannot be calculated only from qemu-side. We cannot do more than the efforts crash utility does for workaround. So, the phys_base value in kdump-sub header is now designed to have 0 now.

Anyway, phys_base is kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.

One ad-hoc but simple way is to put phys_base value as part of VMCOREINFO note information on kernel.

Although there has already been a similar one in VMCOREINFO, like

arch/x86/kernel/
==
void arch_crash_save_vmcoreinfo(void)
{
VMCOREINFO_SYMBOL(phys_base); <---- This
VMCOREINFO_SYMBOL(init_level4_pgt);

...
==

this is meangless, because this value is a virtual address assigned to phys_base symbol. To refer to the value of phys_base itself, we need the phys_base value we are about to get now.

So, instead, if we change this to save the value, not value of symbol phys_base, we can get phys_base from the VMCOREINFO.

The VMCOREINFO consists simply of string. So it's easy to search vmcore for it e.g. using strings and grep like this:

$ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A 100
VMCOREINFO
OSRELEASE=3.10.0-121.el7.x86_64
PAGESIZE=4096
...
SYMBOL(phys_base)=ffffffff818e5010  <-- though this is address of phys_base now...
SYMBOL(init_level4_pgt)=ffffffff818de000
SYMBOL(node_data)=ffffffff819f1cc0
LENGTH(node_data)=1024
CRASHTIME=1399460394
...

This should also be useful to get phys_base of 2nd kernel, which is inherently relocated kernel from a vmcore generated using qemu dump.

This is far from well-designed from qemu's point of view, but it would be manually easier to get phys_base than now.

Obviously, the VMCOREINFO is available only if CONFIG_KEXEC is enabled. Other users cannot use this.

--
Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw
Quoted 15 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…

From: Petr Tesarik <[email protected]>
Subject: Re: uniquely identifying KDUMP files that originate from QEMU
Date: Tue, 11 Nov 2014 13:09:13 +0100

Quoted 2 lines

On Tue, 11 Nov 2014 12:22:52 +0100
Laszlo Ersek <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 7 lines

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Alas, the QEMU notes are not present in the KDUMP-format vmcores that QEMU produces (they cannot be),

Quoted 6 lines
Quoted 3 lines

Why? Since KDUMP format version 4, the complete ELF notes can be stored in the file (see offset_note, size_note fields in the sub-header).

Yes, the QEMU notes is present in kdump-compressed format. But phys_base cannot be calculated only from qemu-side. We cannot do more

Yes, this part is obvious. I was referring to this sentence: "Alas, the QEMU notes are not present in the KDUMP-format vmcores." My understanding was that crash cannot detect a KDUMP file created by QEMU, and so it does not apply the workaround. Sorry for confusion if this was not your problem.

Quoted 6 lines

than the efforts crash utility does for workaround. So, the phys_base value in kdump-sub header is now designed to have 0 now.

Anyway, phys_base is kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.

Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.

Quoted 2 lines

One ad-hoc but simple way is to put phys_base value as part of VMCOREINFO note information on kernel.

YES! In fact, this has been on my TODO list for a few weeks now.

Quoted 9 lines

Although there has already been a similar one in VMCOREINFO, like

arch/x86/kernel/
==
void arch_crash_save_vmcoreinfo(void)
{
VMCOREINFO_SYMBOL(phys_base); <---- This
VMCOREINFO_SYMBOL(init_level4_pgt);

Quoted 5 lines

...
==

this is meangless, because this value is a virtual address assigned to phys_base symbol.

Yes, again. I have already done some research and *nobody* needs the actual symbol value. For example, makedumpfile only checks if the symbol exists and sets phys_base to 0 unconditionally if not. That's so wrong...

Quoted 5 lines

To refer to the value of phys_base itself, we need the phys_base value we are about to get now.

So, instead, if we change this to save the value, not value of symbol phys_base, we can get phys_base from the VMCOREINFO.

Yes, please do that. It should be sufficient to replace this line in kernel's arch/x86/kernel/machine_kexec_64.c:

	VMCOREINFO_SYMBOL(phys_base);

with:

	VMCOREINFO_NUMBER(phys_base);
Quoted 4 lines

The VMCOREINFO consists simply of string. So it's easy to search vmcore for it e.g. using strings and grep like this:

$ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A 100

If vmcore-3.10.0-121.el7.x86_64 is a standard kernel ELF dump file, you can actually run elfutil's "readelf -n" on it and get the VMCOREINFO directly (or use my libkdumpfile library to read the kernel core file, see https://github.com/ptesarik/libkdumpfile).

If it is simply a QEMU dump file (without the VMCOREINFO ELF note), then running strings on it seems like the only sensible workaround. I tried to solve a similar problem in kdumpid
(http://sourceforge.net/projects/kdumpid/), and best I could do is very similar to the workaround in the crash utility (scanning physical memory for something that looks like kernel text).

Petr T

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#12 lore raw
In reply to Christopher Covington
Quoted 31 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

Quoted 8 lines
Quoted 6 lines

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

HTH,
Petr T

#13 lore raw
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Quoted 4 lines
Quoted 1 line

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

How does QEMU figure out the kernel version information?

Thanks,
Chris

Signature · 3 lines
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#14 lore raw
Quoted 42 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)

Quoted 1 line

How does QEMU figure out the kernel version information?

Good question. Who can answer this part?

Petr T

#15 lore raw
Quoted 50 lines · Petr Tesarik
Quoted 47 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 45 lines · Petr Tesarik
Quoted 31 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

Quoted 8 lines
Quoted 6 lines

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

Quoted 7 lines
Quoted 1 line

What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)

Understood on the ELF.xz approach, but why couldn't each page (or maybe a configurable size) be a SHF_COMPRESSED section?

Thanks,
Chris

Signature · 3 lines
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#18 lore raw
In reply to Petr Tesarik
Quoted 45 lines · Petr Tesarik
Quoted 42 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.

The difference in speed of dumping is stunning.

Quoted 11 lines
Quoted 1 line

What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)

Quoted 1 line

How does QEMU figure out the kernel version information?

Good question. Who can answer this part?

I can.

(Apologies for being a bit non-responsive, I'm swamped. I figured I'd let the discussion unfold a bit between the kdump experts.)

So, QEMU doesn't figure out the kernel version information. It just dumps the guest-physical frames, and that's it.

I linked the code before that populates the kdump header. The links and function names are still visible above.

Thanks
Laszlo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#16 lore raw
In reply to Christopher Covington

V Wed, 12 Nov 2014 08:28:54 -0500 Christopher Covington <[email protected]> napsáno:

Quoted 53 lines · Petr Tesarik
Quoted 52 lines
Quoted 48 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Quoted 4 lines
Quoted 1 line

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

Quoted 8 lines
Quoted 5 lines

Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)

Understood on the ELF.xz approach, but why couldn't each page (or maybe a configurable size) be a SHF_COMPRESSED section?

A machine with 64TB of RAM (already manufactured by SGI) has 17,179,869,184 pages. When KDUMP (or, actually diskdump) format was invented, ELF files could have at most 2^16 = 65,536 program headers. Since then, ELF specification has been extended (PN_XNUM), so the number of sections can be stored in the sh_info field of the first ELF section, but that only increases the number of possible sections to 2^32 = 4,294,967,296.

Yes, we could divide memory into larger chunks than pages, but:

1. you're probably the first one to have the idea, and
2. this is easy if you save the complete RAM content, but not quite that easy if some pages should be filtered out (makedumpfile).

There are a few other (minor) points, e.g.:

* Each program header consumes 56 bytes in ELF64, while a single bit is sufficient in KDUMP compressed files to tell if the corresponding page is stored or not.

* SHF_COMPRESSED currently supports only zlib compression, which is rather slow. KDUMP supports zlib, lzo and snappy.

* Support for KDUMP files is already present in the crash utility, while I don't think there is any support for SHF_COMPRESSED segments.

In short, SHF_COMPRESSED looks like a viable alternative, but right now KDUMP is the better choice in terms of features and interoperability.

Just my two cents,
Petr T

#23 lore raw
In reply to Christopher Covington
Quoted 31 lines · Christopher Covington
Quoted 30 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

Quoted 8 lines
Quoted 6 lines

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

What advantages does KDUMP have over ELF?

This has been discussed, but I'd like to give a short perspective from personal experience.

The more obvious advantage is the smaller size, due to (a) per-page compression (which preserves random-access for "crash"), and (b) zero page sharing. A smaller dump file is easier to store, and easier to upload if you're requesting assitance with debugging.

The perhaps less obvious advantage is the speed at which qemu writes the dump. We're talking orders of magnitude, especially on rotational media. This is because lzo and snappy are *incredibly* fast (put differently:
they incur very little CPU penalty for the same guest RAM size). The CPU penalty is actually so small that in almost all cases the dumping procedure stays IO-bound (in my experience: even on an SSD!). Now combine that with a potential reduction of 4GB -> 256MB in size: that's a sixteen-fold speedup.

(I'm allowed to praise this qemu feature, I didn't write it. :))

Thanks
Laszlo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#17 lore raw
In reply to Christopher Covington
Quoted 53 lines · Christopher Covington
Quoted 52 lines · Petr Tesarik
Quoted 48 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Quoted 4 lines
Quoted 1 line

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

Quoted 8 lines
Quoted 5 lines

Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)

Understood on the ELF.xz approach, but why couldn't each page (or maybe a configurable size) be a SHF_COMPRESSED section?

Perhaps it could, technically -- it's just not how Qiao Nuohan implemented the feature. I didn't research the background for this.

Thanks
Laszlo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#19 lore raw
In reply to Laszlo Ersek

Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time.

Quoted 48 lines · Laszlo Ersek
Quoted 47 lines · Petr Tesarik
Quoted 44 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 43 lines · Petr Tesarik
Quoted 39 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

Quoted 6 lines

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

What advantages does KDUMP have over ELF?

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.

The difference in speed of dumping is stunning.

Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?

Thanks,
Chris

Signature · 3 lines
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#5 lore raw
In reply to Petr Tesarik
Quoted 9 lines · Petr Tesarik
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…

Anyway, phys_base is kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.

Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.

Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.

Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs? It certainly can, it already does. It might need more heuristics than in when the guest kernel offers those bits of info on a plate, but those heuristics already work. We just need to tell "crash" that this particular box, flying in the air, has been launched by qemu, so that "crash" enable its heuristics.

Thanks
Laszlo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#20 lore raw
In reply to Christopher Covington
Quoted 55 lines · Christopher Covington

Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time.

Quoted 51 lines · Laszlo Ersek
Quoted 49 lines · Petr Tesarik
Quoted 46 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 45 lines · Petr Tesarik
Quoted 31 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

Quoted 8 lines
Quoted 6 lines

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

Quoted 6 lines
Quoted 4 lines

Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.

The difference in speed of dumping is stunning.

Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?

Sorry, I don't know what SHT_NOBITS is.

Laszlo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#21 lore raw
Quoted 56 lines · Laszlo Ersek
Quoted 55 lines · Christopher Covington

Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time.

Quoted 51 lines · Laszlo Ersek
Quoted 49 lines · Petr Tesarik
Quoted 46 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 45 lines · Petr Tesarik
Quoted 31 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

Quoted 8 lines
Quoted 6 lines

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.

The difference in speed of dumping is stunning.

Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?

Sorry, I don't know what SHT_NOBITS is.

My newbie understanding is that SHT_NOBITS is the section type of the .bss section in an everyday executable. It's what makes the section only need a header table entry in the ELF file, and not sh_size (Size) worth of zeros. The ELF loader will essentially zero memory beginning at sh_addr (Address) for sh_size (Size).

http://stackoverflow.com/questions/22855320/address-space-of-a-bss-section-space-in-elf-file

http://people.redhat.com/mpolacek/src/devconf2012.pdf

Chris

Signature · 3 lines
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#6 lore raw
In reply to Laszlo Ersek

V Wed, 12 Nov 2014 15:50:32 +0100
Laszlo Ersek <[email protected]> napsáno:

Quoted 11 lines · Petr Tesarik
Quoted 9 lines
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…

Anyway, phys_base is kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.

Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.

Quoted 8 lines

Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.

Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs?

If only you could save the _complete_ state of the CPU... For example the content of CR3 would be quite useful.

Petr T

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw

adding back a few CC's because this discussion is useful

Quoted 19 lines · Petr Tesarik

V Wed, 12 Nov 2014 15:50:32 +0100
Laszlo Ersek <[email protected]> napsáno:

Quoted 16 lines · Petr Tesarik
Quoted 9 lines
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…

Anyway, phys_base is kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.

Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.

Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.

Quoted 6 lines
Quoted 3 lines

Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs?

If only you could save the _complete_ state of the CPU... For example the content of CR3 would be quite useful.

(1) CR3 is already saved, in both the ELF and the kdump compressed formats.

- ELF case:

qmp_dump_guest_memory() [dump.c]
create_vmcore()

    dump_begin()
      write_elf64_notes()

        loop from 1 to #vcpu:
          cpu_write_elf64_note() [qom/cpu.c]
            x86_64_write_elf64_note() [target-i386/arch_dump.c]
              writes "CORE"

        loop from 1 to #vcpu:
          cpu_write_elf64_qemunote() [qom/cpu.c]
            x86_cpu_write_elf64_qemunote() [target-i386/arch_dump.c]
              cpu_write_qemu_note()
                qemu_get_cpustate()
                  s->cr[3] = env->cr[3]; <---------- here
                writes "QEMU"

Hence, the information is part of the QEMU note.

- kdump case:

qmp_dump_guest_memory() [dump.c]
create_kdump_vmcore()

    write_dump_header()
      create_header64()
        write_elf64_notes()
          [... same as above ...]

The trick here is that the note-writer functions use a callback function for actually outputting the data. So while in the ELF case the stuff goes directly to a file, in the kdump case the notes are first saved in a memory buffer, and then later saved in the file at offset KdumpSubHeader64.offset_note. (... Which is then represented in the flattened file format of course.)

So, the information is there in both cases.

(2) Dave -- this just made me realize that the QEMU note is *already* there in the kdump file as well; pointed-to by KdumpSubHeader64.offset_note, for a length of KdumpSubHeader64.note_size.

From your other email
<http://thread.gmane.org/gmane.linux.kernel.kexec/12787/focus=12797>:

Quoted 17 lines
    sub_header_kdump: 1c9cff0
             phys_base: 0
            dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
                 split: 0
             start_pfn: (unused)
               end_pfn: (unused)
     offset_vmcoreinfo: 0 (0x0)
       size_vmcoreinfo: 0 (0x0)
           offset_note: 4200 (0x1068)       <----------- here
             size_note: 3232 (0xca0)        <-----------
    num_prstatus_notes: 4
             notes_buf: 1c9e000
              notes[0]: 1c9e000
              notes[1]: 1c9e164
              notes[2]: 1c9e2c8
              notes[3]: 1c9e42c
    NT_PRSTATUS_offset: 1068
Quoted 8 lines
                        11cc
                        1330
                        1494
      offset_eraseinfo: 0 (0x0)
        size_eraseinfo: 0 (0x0)
          start_pfn_64: (unused)
            end_pfn_64: (unused)
          max_mapnr_64: 1245184 (0x130000)

Can you fetch that in "crash"? If you can, then there's nothing to do on the qemu side (and I'll have to apologize for spamming a bunch of lists :/).

I think "crash" already iterates over all of the notes in the note buffer, but skips everything different from NT_PRSTATUS.

(3) Regarding the structure of the notes, we have to consider the placement of the notes and their internal structure. The placement is different between the ELF and the KDUMP file format. The internal structure of the notes is identical between the two file formats.

For example, for a 4 VCPU guest, you end up with note names like

CORE
CORE
CORE
CORE
QEMU
QEMU
QEMU
QEMU

All of these are Elf64_Nhdr structures. The CORE ones have type NT_PRSTATUS, and the QEMU ones have type 0.

(3a) The placement in the ELF file is already handled by "crash". Each note "simply" gets its own ELF note segment/section.

(3b) In the kdump file, the Elf64_Nhdr structures (8 pieces in total, in the above example -- 4x CORE, 4x QEMU) are concatenated in that order, and finally stored at "offset_note".

(3c) Regarding the internal structure of the notes. The CORE ones are already known and handled. The QEMU notes have the following structure:

Quoted 5 lines

Elf64_Nhdr:
n_namesz: 5 ("QEMU")
n_descsz: 432
n_type: 0 (?)

          000001b000000001 0000000000000000
            |------||------| |--------------|
            size    version  rax
Quoted 1 line
          0000000000000000 0000000000000000
            |--------------| |--------------|
            rbx              rcx
Quoted 1 line
          0000000000000000 0000000000000001
            |--------------| |--------------|
            rdx              rsi
Quoted 1 line
          ffffffff81dd5228 ffffffff81a01ec8
            |--------------| |--------------|
            rdi              rsp
Quoted 1 line
          ffffffff81a01ec8 0000000000000000
            |--------------| |--------------|
            rbp              r8
Quoted 1 line
          0000000000000000 00000013911d5f29
            |--------------| |--------------|
            r9               r10
Quoted 1 line
          0000000000000000 ffffffff81c00480
            |--------------| |--------------|
            r11              r12
Quoted 1 line
          0000000000000000 ffffffffffffffff
            |--------------| |--------------|
            r13              r14
Quoted 1 line
          000000000309f000 ffffffff810375ab
            |--------------| |--------------|
            r15              rip
Quoted 1 line
          0000000000000246 ffffffff00000010
            |--------------| |------||------|
            rflags           cs/lim  cs/sel
Quoted 1 line
          0000000000a09b00 0000000000000000
            |------||------| |--------------|
            cs/pad  cs/flags cs/base
Quoted 1 line
          ffffffff00000018 0000000000c09300
            |------||------| |------||------|
            ds/lim  ds/sel   ds/pad  ds/flags
Quoted 1 line
          0000000000000000 ffffffff00000018
            |--------------| |------||------|
            ds/base          es/lim  es/sel
Quoted 1 line
          0000000000c09300 0000000000000000
            |------||------| |--------------|
            es/pad  es/flags es/base
Quoted 1 line
          ffffffff00000000 0000000000000000
            |------||------| |------||------|
            fs/lim  fs/sel   fs/pad  fs/flags
Quoted 1 line
          0000000000000000 ffffffff00000000
            |--------------| |------||------|
            fs/base          gs/lim  gs/sel
Quoted 1 line
          0000000000000000 ffff880003200000
            |------||------| |--------------|
            gs/pad  gs/flags gs/base
Quoted 1 line
          ffffffff00000018 0000000000c09300
            |------||------| |------||------|
            ss/lim  ss/sel   ss/pad  ss/flags
Quoted 1 line
          0000000000000000 ffffffff00000000
            |--------------| |------||------|
            ss/base          ldt...
Quoted 1 line
          0000000000000000 0000000000000000
            |------||------| |--------------|
                                       ...ldt
Quoted 1 line
          0000208700000040 0000000000008b00
            |------||------| |------||------|
            tr...
Quoted 1 line
          ffff880003213b40 0000007f00000000
            |--------------| |------||------|
                       ...tr gdt...
Quoted 1 line
          0000000000000000 ffff880003204000
            |------||------| |--------------|
                                       ...gdt
Quoted 1 line
          00000fff00000000 0000000000000000
            |------||------| |------||------|
            idt...
Quoted 1 line
          ffffffff81dd2000 000000008005003b
            |--------------| |--------------|
                      ...idt cr0
Quoted 1 line
          0000000000000000 0000000001b2e000
            |--------------| |--------------|
            cr1              cr2
Quoted 1 line
          0000000007b18000 00000000000006f0
            |--------------| |--------------|
            cr3              cr4

From "target-i386/arch_dump.c":

Quoted 21 lines
struct QEMUCPUSegment {
    uint32_t selector;
    uint32_t limit;
    uint32_t flags;
    uint32_t pad;
    uint64_t base;
};

typedef struct QEMUCPUSegment QEMUCPUSegment;

struct QEMUCPUState {
    uint32_t version;
    uint32_t size;
    uint64_t rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp;
    uint64_t r8, r9, r10, r11, r12, r13, r14, r15;
    uint64_t rip, rflags;
    QEMUCPUSegment cs, ds, es, fs, gs, ss;
    QEMUCPUSegment ldt, tr, gdt, idt;
    uint64_t cr[5];
};
Quoted 1 line

typedef struct QEMUCPUState QEMUCPUState;

Summary: I think the info is all there.

Thanks
Laszlo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw

----- Original Message -----

Quoted 105 lines

adding back a few CC's because this discussion is useful

Quoted 19 lines · Petr Tesarik

V Wed, 12 Nov 2014 15:50:32 +0100
Laszlo Ersek <[email protected]> napsáno:

Quoted 16 lines · Petr Tesarik
Quoted 9 lines
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…

Anyway, phys_base is kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.

Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.

Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.

Quoted 6 lines
Quoted 3 lines

Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs?

If only you could save the _complete_ state of the CPU... For example the content of CR3 would be quite useful.

(1) CR3 is already saved, in both the ELF and the kdump compressed formats.

- ELF case:

qmp_dump_guest_memory() [dump.c]
create_vmcore()

    dump_begin()
      write_elf64_notes()

        loop from 1 to #vcpu:
          cpu_write_elf64_note() [qom/cpu.c]
            x86_64_write_elf64_note() [target-i386/arch_dump.c]
              writes "CORE"

        loop from 1 to #vcpu:
          cpu_write_elf64_qemunote() [qom/cpu.c]
            x86_cpu_write_elf64_qemunote() [target-i386/arch_dump.c]
              cpu_write_qemu_note()
                qemu_get_cpustate()
                  s->cr[3] = env->cr[3]; <---------- here
                writes "QEMU"

Hence, the information is part of the QEMU note.

- kdump case:

qmp_dump_guest_memory() [dump.c]
create_kdump_vmcore()

    write_dump_header()
      create_header64()
        write_elf64_notes()
          [... same as above ...]

The trick here is that the note-writer functions use a callback function for actually outputting the data. So while in the ELF case the stuff goes directly to a file, in the kdump case the notes are first saved in a memory buffer, and then later saved in the file at offset KdumpSubHeader64.offset_note. (... Which is then represented in the flattened file format of course.)

So, the information is there in both cases.

(2) Dave -- this just made me realize that the QEMU note is *already* there in the kdump file as well; pointed-to by KdumpSubHeader64.offset_note, for a length of KdumpSubHeader64.note_size.

From your other email
<http://thread.gmane.org/gmane.linux.kernel.kexec/12787/focus=12797>:

Quoted 17 lines
    sub_header_kdump: 1c9cff0
             phys_base: 0
            dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
                 split: 0
             start_pfn: (unused)
               end_pfn: (unused)
     offset_vmcoreinfo: 0 (0x0)
       size_vmcoreinfo: 0 (0x0)
           offset_note: 4200 (0x1068)       <----------- here
             size_note: 3232 (0xca0)        <-----------
    num_prstatus_notes: 4
             notes_buf: 1c9e000
              notes[0]: 1c9e000
              notes[1]: 1c9e164
              notes[2]: 1c9e2c8
              notes[3]: 1c9e42c
    NT_PRSTATUS_offset: 1068
Quoted 8 lines
                        11cc
                        1330
                        1494
      offset_eraseinfo: 0 (0x0)
        size_eraseinfo: 0 (0x0)
          start_pfn_64: (unused)
            end_pfn_64: (unused)
          max_mapnr_64: 1245184 (0x130000)
Quoted 2 lines

Can you fetch that in "crash"? If you can, then there's nothing to do on the qemu side (and I'll have to apologize for spamming a bunch of lists :/).

Sure enough...

I was just playing with process_el64_notes() to check/read the note name strings, and noticed that I can certainly see them. But as you noted, only the NT_PRSTATUS notes are stored in the "notes[]" array. so I was under the impression that the QEMU notes were completely missing.

That being the case -- we're pretty much done!

I'll put a patch in the next upstream release of crash.

Thanks,
Dave

Quoted 169 lines

I think "crash" already iterates over all of the notes in the note buffer, but skips everything different from NT_PRSTATUS.

(3) Regarding the structure of the notes, we have to consider the placement of the notes and their internal structure. The placement is different between the ELF and the KDUMP file format. The internal structure of the notes is identical between the two file formats.

For example, for a 4 VCPU guest, you end up with note names like

CORE
CORE
CORE
CORE
QEMU
QEMU
QEMU
QEMU

All of these are Elf64_Nhdr structures. The CORE ones have type NT_PRSTATUS, and the QEMU ones have type 0.

(3a) The placement in the ELF file is already handled by "crash". Each note "simply" gets its own ELF note segment/section.

(3b) In the kdump file, the Elf64_Nhdr structures (8 pieces in total, in the above example -- 4x CORE, 4x QEMU) are concatenated in that order, and finally stored at "offset_note".

(3c) Regarding the internal structure of the notes. The CORE ones are already known and handled. The QEMU notes have the following structure:

Quoted 5 lines

Elf64_Nhdr:
n_namesz: 5 ("QEMU")
n_descsz: 432
n_type: 0 (?)

          000001b000000001 0000000000000000
            |------||------| |--------------|
            size    version  rax
Quoted 1 line
          0000000000000000 0000000000000000
            |--------------| |--------------|
            rbx              rcx
Quoted 1 line
          0000000000000000 0000000000000001
            |--------------| |--------------|
            rdx              rsi
Quoted 1 line
          ffffffff81dd5228 ffffffff81a01ec8
            |--------------| |--------------|
            rdi              rsp
Quoted 1 line
          ffffffff81a01ec8 0000000000000000
            |--------------| |--------------|
            rbp              r8
Quoted 1 line
          0000000000000000 00000013911d5f29
            |--------------| |--------------|
            r9               r10
Quoted 1 line
          0000000000000000 ffffffff81c00480
            |--------------| |--------------|
            r11              r12
Quoted 1 line
          0000000000000000 ffffffffffffffff
            |--------------| |--------------|
            r13              r14
Quoted 1 line
          000000000309f000 ffffffff810375ab
            |--------------| |--------------|
            r15              rip
Quoted 1 line
          0000000000000246 ffffffff00000010
            |--------------| |------||------|
            rflags           cs/lim  cs/sel
Quoted 1 line
          0000000000a09b00 0000000000000000
            |------||------| |--------------|
            cs/pad  cs/flags cs/base
Quoted 1 line
          ffffffff00000018 0000000000c09300
            |------||------| |------||------|
            ds/lim  ds/sel   ds/pad  ds/flags
Quoted 1 line
          0000000000000000 ffffffff00000018
            |--------------| |------||------|
            ds/base          es/lim  es/sel
Quoted 1 line
          0000000000c09300 0000000000000000
            |------||------| |--------------|
            es/pad  es/flags es/base
Quoted 1 line
          ffffffff00000000 0000000000000000
            |------||------| |------||------|
            fs/lim  fs/sel   fs/pad  fs/flags
Quoted 1 line
          0000000000000000 ffffffff00000000
            |--------------| |------||------|
            fs/base          gs/lim  gs/sel
Quoted 1 line
          0000000000000000 ffff880003200000
            |------||------| |--------------|
            gs/pad  gs/flags gs/base
Quoted 1 line
          ffffffff00000018 0000000000c09300
            |------||------| |------||------|
            ss/lim  ss/sel   ss/pad  ss/flags
Quoted 1 line
          0000000000000000 ffffffff00000000
            |--------------| |------||------|
            ss/base          ldt...
Quoted 1 line
          0000000000000000 0000000000000000
            |------||------| |--------------|
                                       ...ldt
Quoted 1 line
          0000208700000040 0000000000008b00
            |------||------| |------||------|
            tr...
Quoted 1 line
          ffff880003213b40 0000007f00000000
            |--------------| |------||------|
                       ...tr gdt...
Quoted 1 line
          0000000000000000 ffff880003204000
            |------||------| |--------------|
                                       ...gdt
Quoted 1 line
          00000fff00000000 0000000000000000
            |------||------| |------||------|
            idt...
Quoted 1 line
          ffffffff81dd2000 000000008005003b
            |--------------| |--------------|
                      ...idt cr0
Quoted 1 line
          0000000000000000 0000000001b2e000
            |--------------| |--------------|
            cr1              cr2
Quoted 1 line
          0000000007b18000 00000000000006f0
            |--------------| |--------------|
            cr3              cr4

From "target-i386/arch_dump.c":

Quoted 21 lines
struct QEMUCPUSegment {
    uint32_t selector;
    uint32_t limit;
    uint32_t flags;
    uint32_t pad;
    uint64_t base;
};

typedef struct QEMUCPUSegment QEMUCPUSegment;

struct QEMUCPUState {
    uint32_t version;
    uint32_t size;
    uint64_t rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp;
    uint64_t r8, r9, r10, r11, r12, r13, r14, r15;
    uint64_t rip, rflags;
    QEMUCPUSegment cs, ds, es, fs, gs, ss;
    QEMUCPUSegment ldt, tr, gdt, idt;
    uint64_t cr[5];
};
Quoted 8 lines
Quoted 1 line

typedef struct QEMUCPUState QEMUCPUState;

Summary: I think the info is all there.

Thanks
Laszlo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#22 lore raw
In reply to Christopher Covington
Quoted 60 lines · On Wed, 12 Nov 2014 10:43:59 -0500 Christopher Covington <[email protected]>…
Quoted 59 lines · Laszlo Ersek
Quoted 58 lines · Christopher Covington

Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time.

Quoted 54 lines · Laszlo Ersek
Quoted 45 lines · Petr Tesarik
Quoted 42 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek

(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).

Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

The difference in speed of dumping is stunning.

Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?

Sorry, I don't know what SHT_NOBITS is.

My newbie understanding is that SHT_NOBITS is the section type of the .bss section in an everyday executable.

Heh, yes and no. Let's clarify a few things.

First, a Linux kernel dump (or a QEMU ELF dump) does not contain any sections. It only contains program headers. The reason is that program headers can specify both the virtual address and the physical address. BTW this feature is used to determine the physical base of the Linux kernel when dumping via kexec. Sections can only specify the virtual address. Of course, program headers can achieve an effect similar to
SHT_NOBITS: by specifying a larger memory size than file size.

Now, all this does not mean you can't create a new standard that uses sections instead (in fact, Xen DomU dumps already use ELF sections). But of course, this new standard won't be understood by the existing tools until somebody (you?) adds support for it. Same thing is true for the SHF_COMPRESSED section flag.

Anyway, reuse of the zero page is a minor point. The main difference in speed comes from using a faster compression algorithm (LZO or snappy). Of course, nothing prevents you from using those algorithms in ELF compressed sections, but you'd have to extend the ELF standard first, adding magic numbers for these algorithms. To me it sounds like a long way to go.

In short, there's no technical reason why ELF couldn't achieve results similar to KDUMP. It's "merely" not implemented, and for sure, I'm not going to push all the necessary changes. ;-)

Petr T

#10 lore raw
In reply to Laszlo Ersek
Quoted 22 lines · On Wed, 12 Nov 2014 21:30:20 +0100 Laszlo Ersek <[email protected]> wrote:

adding back a few CC's because this discussion is useful

Quoted 19 lines · Petr Tesarik

V Wed, 12 Nov 2014 15:50:32 +0100
Laszlo Ersek <[email protected]> napsáno:

Quoted 16 lines · Petr Tesarik
Quoted 9 lines
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…

Anyway, phys_base is kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.

Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.

Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs?

If only you could save the _complete_ state of the CPU... For example the content of CR3 would be quite useful.

(1) CR3 is already saved, in both the ELF and the kdump compressed formats.

Sweet. :-)

So, there's no need for any heuristics. Since CR3 gives the physical address of the PML4 table, I can use it to translate __START_KERNEL_map (0xffffffff80000000UL on all Linux kernels since introduction of x86_64) to a physical address and compute phys_base from that.

In fact, QEMU could do the same if you can live with hardcoding a Linux-kernel specific constant into the tool...

Petr T

#9 lore raw
In reply to Dave Anderson

----- Original Message -----

Quoted 3 lines
Quoted 3 lines

Can you fetch that in "crash"? If you can, then there's nothing to do on the qemu side (and I'll have to apologize for spamming a bunch of lists :/).

Well, let's be clear -- I was the one who put you up to it...

But no apology is required -- and in fact, if today's discussion results in the "phys_base" vmcoreinfo issue being resolved in the kernel, we'll all be eternally grateful.

Thanks again,
Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec