uniquely identifying KDUMP files that originate from QEMU
Subsystems (2)
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "phys_base" field. Refer to dump.c, functions create_header32(), create_header64(), and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text "0").
http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a phys_base up to 32MB, in 1MB steps.
Alas, the QEMU notes are not present in the KDUMP-format vmcores that QEMU produces (they cannot be), hence crash has no way to tell apart such files from those generated by genuine kdump. As an end result, "crash" cannot automatically find the phys_base of OVMF-based Linux vmcores.
Dave suggested that a new flag, or a special phys_base value (like ~0UL) be introduced as a distinguishing mark for QEMU-produced kdumps.
Implementing this in QEMU wouldn't be hard. The big question is compatibility -- whose analysis tools would be broken by a (phys_base == ~0UL) setting, or by a new flag?
Note that this change would affect SeaBIOS-based vmcores too. QEMU can't (and shouldn't) discriminate the vmcores it dumps based on guest firmware. (If QEMU did that, then it might as well try to figure out the real phys_base value, which is clearly out of scope for qemu. One of the selling points of the paging=false dump is that it doesn't involve parsing guest RAM.)
Thanks
Laszlo
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 8 lines · Laszlo Ersek
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
Quoted 7 lines
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
Presumably this is also not going to work for machines other than the x86 PC, where physical memory may well not start at address zero...
-- PMM
Quoted 32 lines · On Tue, 11 Nov 2014 12:22:52 +0100 Laszlo Ersek <[email protected]> wrote:
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Quoted 7 lines
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.Alas, the QEMU notes are not present in the KDUMP-format vmcores that QEMU produces (they cannot be),
Why? Since KDUMP format version 4, the complete ELF notes can be stored in the file (see offset_note, size_note fields in the sub-header).
Petr Tesarik
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 30 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
Quoted 6 lines
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.
What advantages does KDUMP have over ELF?
Thanks,
Chris
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
From: Petr Tesarik <[email protected]>
Subject: Re: uniquely identifying KDUMP files that originate from QEMU
Date: Tue, 11 Nov 2014 13:09:13 +0100
Quoted 40 lines
Quoted 37 lines · On Tue, 11 Nov 2014 12:22:52 +0100 Laszlo Ersek <[email protected]> wrote:
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.
Quoted 6 lines
Quoted 2 lines
Alas, the QEMU notes are not present in the KDUMP-format vmcores that QEMU produces (they cannot be),
Why? Since KDUMP format version 4, the complete ELF notes can be stored in the file (see
offset_note,size_notefields in the sub-header).
Yes, the QEMU notes is present in kdump-compressed format. But phys_base cannot be calculated only from qemu-side. We cannot do more than the efforts crash utility does for workaround. So, the phys_base value in kdump-sub header is now designed to have 0 now.
Anyway, phys_base is kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.
One ad-hoc but simple way is to put phys_base value as part of VMCOREINFO note information on kernel.
Although there has already been a similar one in VMCOREINFO, like
arch/x86/kernel/
==
void arch_crash_save_vmcoreinfo(void)
{
VMCOREINFO_SYMBOL(phys_base); <---- This
VMCOREINFO_SYMBOL(init_level4_pgt);
...
==
this is meangless, because this value is a virtual address assigned to phys_base symbol. To refer to the value of phys_base itself, we need the phys_base value we are about to get now.
So, instead, if we change this to save the value, not value of symbol phys_base, we can get phys_base from the VMCOREINFO.
The VMCOREINFO consists simply of string. So it's easy to search vmcore for it e.g. using strings and grep like this:
$ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A 100 VMCOREINFO OSRELEASE=3.10.0-121.el7.x86_64 PAGESIZE=4096 ... SYMBOL(phys_base)=ffffffff818e5010 <-- though this is address of phys_base now... SYMBOL(init_level4_pgt)=ffffffff818de000 SYMBOL(node_data)=ffffffff819f1cc0 LENGTH(node_data)=1024 CRASHTIME=1399460394 ...
This should also be useful to get phys_base of 2nd kernel, which is inherently relocated kernel from a vmcore generated using qemu dump.
This is far from well-designed from qemu's point of view, but it would be manually easier to get phys_base than now.
Obviously, the VMCOREINFO is available only if CONFIG_KEXEC is enabled. Other users cannot use this.
--
Thanks.
HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 15 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…
From: Petr Tesarik <[email protected]>
Subject: Re: uniquely identifying KDUMP files that originate from QEMU
Date: Tue, 11 Nov 2014 13:09:13 +0100Quoted 2 lines
On Tue, 11 Nov 2014 12:22:52 +0100
Laszlo Ersek <[email protected]> wrote:[...]
Quoted 8 lines
Quoted 7 lines
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.Alas, the QEMU notes are not present in the KDUMP-format vmcores that QEMU produces (they cannot be),
Quoted 6 lines
Quoted 3 lines
Why? Since KDUMP format version 4, the complete ELF notes can be stored in the file (see
offset_note,size_notefields in the sub-header).Yes, the QEMU notes is present in kdump-compressed format. But
phys_basecannot be calculated only from qemu-side. We cannot do more
Yes, this part is obvious. I was referring to this sentence: "Alas, the QEMU notes are not present in the KDUMP-format vmcores." My understanding was that crash cannot detect a KDUMP file created by QEMU, and so it does not apply the workaround. Sorry for confusion if this was not your problem.
Quoted 6 lines
than the efforts crash utility does for workaround. So, the
phys_basevalue in kdump-sub header is now designed to have 0 now.Anyway,
phys_baseis kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.
Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.
Quoted 2 lines
One ad-hoc but simple way is to put
phys_basevalue as part of VMCOREINFO note information on kernel.
YES! In fact, this has been on my TODO list for a few weeks now.
Quoted 9 lines
Although there has already been a similar one in VMCOREINFO, like
arch/x86/kernel/
==
voidarch_crash_save_vmcoreinfo(void)
{
VMCOREINFO_SYMBOL(phys_base); <---- This
VMCOREINFO_SYMBOL(init_level4_pgt);
Quoted 5 lines
...
==this is meangless, because this value is a virtual address assigned to
phys_basesymbol.
Yes, again. I have already done some research and *nobody* needs the actual symbol value. For example, makedumpfile only checks if the symbol exists and sets phys_base to 0 unconditionally if not. That's so wrong...
Quoted 5 lines
To refer to the value of
phys_baseitself, we need thephys_basevalue we are about to get now.So, instead, if we change this to save the value, not value of symbol
phys_base, we can getphys_basefrom the VMCOREINFO.
Yes, please do that. It should be sufficient to replace this line in kernel's arch/x86/kernel/machine_kexec_64.c:
VMCOREINFO_SYMBOL(phys_base);
with:
VMCOREINFO_NUMBER(phys_base);
Quoted 4 lines
The VMCOREINFO consists simply of string. So it's easy to search vmcore for it e.g. using strings and grep like this:
$ strings vmcore-3.10.0-121.el7.x86_64 | grep -E ".*VMCOREINFO.*" -A 100
If vmcore-3.10.0-121.el7.x86_64 is a standard kernel ELF dump file, you can actually run elfutil's "readelf -n" on it and get the VMCOREINFO directly (or use my libkdumpfile library to read the kernel core file, see https://github.com/ptesarik/libkdumpfile).
If it is simply a QEMU dump file (without the VMCOREINFO ELF note), then running strings on it seems like the only sensible workaround. I tried to solve a similar problem in kdumpid
(http://sourceforge.net/projects/kdumpid/), and best I could do is very similar to the workaround in the crash utility (scanning physical memory for something that looks like kernel text).
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 31 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
Quoted 8 lines
Quoted 6 lines
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
HTH,
Petr T
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.
Quoted 4 lines
Quoted 1 line
What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?
How does QEMU figure out the kernel version information?
Thanks,
Chris
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 42 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
What if the ELF dumper used
SHF_COMPRESSEDor could dump an ELF.xz?
Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)
Quoted 1 line
How does QEMU figure out the kernel version information?
Good question. Who can answer this part?
Petr T
Quoted 50 lines · Petr Tesarik
Quoted 47 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 45 lines · Petr Tesarik
Quoted 31 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
Quoted 8 lines
Quoted 6 lines
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
Quoted 7 lines
Quoted 1 line
What if the ELF dumper used
SHF_COMPRESSEDor could dump an ELF.xz?Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)
Understood on the ELF.xz approach, but why couldn't each page (or maybe a configurable size) be a SHF_COMPRESSED section?
Thanks,
Chris
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 45 lines · Petr Tesarik
Quoted 42 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.
The difference in speed of dumping is stunning.
Quoted 11 lines
Quoted 1 line
What if the ELF dumper used
SHF_COMPRESSEDor could dump an ELF.xz?Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)
Quoted 1 line
How does QEMU figure out the kernel version information?
Good question. Who can answer this part?
I can.
(Apologies for being a bit non-responsive, I'm swamped. I figured I'd let the discussion unfold a bit between the kdump experts.)
So, QEMU doesn't figure out the kernel version information. It just dumps the guest-physical frames, and that's it.
I linked the code before that populates the kdump header. The links and function names are still visible above.
Thanks
Laszlo
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
V Wed, 12 Nov 2014 08:28:54 -0500 Christopher Covington <[email protected]> napsáno:
Quoted 53 lines · Petr Tesarik
Quoted 52 lines
Quoted 48 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.Quoted 4 lines
Quoted 1 line
What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
What if the ELF dumper used
SHF_COMPRESSEDor could dump an ELF.xz?
Quoted 8 lines
Quoted 5 lines
Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)
Understood on the ELF.xz approach, but why couldn't each page (or maybe a configurable size) be a
SHF_COMPRESSEDsection?
A machine with 64TB of RAM (already manufactured by SGI) has 17,179,869,184 pages. When KDUMP (or, actually diskdump) format was invented, ELF files could have at most 2^16 = 65,536 program headers. Since then, ELF specification has been extended (PN_XNUM), so the number of sections can be stored in the sh_info field of the first ELF section, but that only increases the number of possible sections to 2^32 = 4,294,967,296.
Yes, we could divide memory into larger chunks than pages, but:
1. you're probably the first one to have the idea, and
2. this is easy if you save the complete RAM content, but not quite that easy if some pages should be filtered out (makedumpfile).
There are a few other (minor) points, e.g.:
* Each program header consumes 56 bytes in ELF64, while a single bit is sufficient in KDUMP compressed files to tell if the corresponding page is stored or not.
* SHF_COMPRESSED currently supports only zlib compression, which is rather slow. KDUMP supports zlib, lzo and snappy.
* Support for KDUMP files is already present in the crash utility, while I don't think there is any support for SHF_COMPRESSED segments.
In short, SHF_COMPRESSED looks like a viable alternative, but right now KDUMP is the better choice in terms of features and interoperability.
Just my two cents,
Petr T
Quoted 31 lines · Christopher Covington
Quoted 30 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
Quoted 8 lines
Quoted 6 lines
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.What advantages does KDUMP have over ELF?
This has been discussed, but I'd like to give a short perspective from personal experience.
The more obvious advantage is the smaller size, due to (a) per-page compression (which preserves random-access for "crash"), and (b) zero page sharing. A smaller dump file is easier to store, and easier to upload if you're requesting assitance with debugging.
The perhaps less obvious advantage is the speed at which qemu writes the dump. We're talking orders of magnitude, especially on rotational media. This is because lzo and snappy are *incredibly* fast (put differently:
they incur very little CPU penalty for the same guest RAM size). The CPU penalty is actually so small that in almost all cases the dumping procedure stays IO-bound (in my experience: even on an SSD!). Now combine that with a potential reduction of 4GB -> 256MB in size: that's a sixteen-fold speedup.
(I'm allowed to praise this qemu feature, I didn't write it. :))
Thanks
Laszlo
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 53 lines · Christopher Covington
Quoted 52 lines · Petr Tesarik
Quoted 48 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.Quoted 4 lines
Quoted 1 line
What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
What if the ELF dumper used
SHF_COMPRESSEDor could dump an ELF.xz?
Quoted 8 lines
Quoted 5 lines
Not the same thing. With KDUMP, each page is compressed separately, so if a utility like crash needs a page from the middle, it can find it and unpack it immediately. If we had an ELF.xz, then the whole file must be unpacked before it can be used. And unpacking a few terabytes takes ... a while. ;-)
Understood on the ELF.xz approach, but why couldn't each page (or maybe a configurable size) be a
SHF_COMPRESSEDsection?
Perhaps it could, technically -- it's just not how Qiao Nuohan implemented the feature. I didn't research the background for this.
Thanks
Laszlo
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time.
Quoted 48 lines · Laszlo Ersek
Quoted 47 lines · Petr Tesarik
Quoted 44 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 43 lines · Petr Tesarik
Quoted 39 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
Quoted 6 lines
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.What advantages does KDUMP have over ELF?
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.
The difference in speed of dumping is stunning.
Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?
Thanks,
Chris
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 9 lines · Petr Tesarik
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…
Anyway,
phys_baseis kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.
Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.
Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs? It certainly can, it already does. It might need more heuristics than in when the guest kernel offers those bits of info on a plate, but those heuristics already work. We just need to tell "crash" that this particular box, flying in the air, has been launched by qemu, so that "crash" enable its heuristics.
Thanks
Laszlo
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 55 lines · Christopher Covington
Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time.
Quoted 51 lines · Laszlo Ersek
Quoted 49 lines · Petr Tesarik
Quoted 46 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 45 lines · Petr Tesarik
Quoted 31 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
Quoted 8 lines
Quoted 6 lines
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
Quoted 6 lines
Quoted 4 lines
Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.
The difference in speed of dumping is stunning.
Would you expect using
SHT_NOBITSto give a similar speedup to the ELF dumper?
Sorry, I don't know what SHT_NOBITS is.
Laszlo
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 56 lines · Laszlo Ersek
Quoted 55 lines · Christopher Covington
Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time.
Quoted 51 lines · Laszlo Ersek
Quoted 49 lines · Petr Tesarik
Quoted 46 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 45 lines · Petr Tesarik
Quoted 31 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 30 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
Quoted 8 lines
Quoted 6 lines
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.
The difference in speed of dumping is stunning.
Would you expect using
SHT_NOBITSto give a similar speedup to the ELF dumper?Sorry, I don't know what
SHT_NOBITSis.
My newbie understanding is that SHT_NOBITS is the section type of the .bss section in an everyday executable. It's what makes the section only need a header table entry in the ELF file, and not sh_size (Size) worth of zeros. The ELF loader will essentially zero memory beginning at sh_addr (Address) for sh_size (Size).
http://stackoverflow.com/questions/22855320/address-space-of-a-bss-section-space-in-elf-file
http://people.redhat.com/mpolacek/src/devconf2012.pdf
Chris
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
V Wed, 12 Nov 2014 15:50:32 +0100
Laszlo Ersek <[email protected]> napsáno:
Quoted 11 lines · Petr Tesarik
Quoted 9 lines
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…
Anyway,
phys_baseis kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.
Quoted 8 lines
Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.
Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs?
If only you could save the _complete_ state of the CPU... For example the content of CR3 would be quite useful.
Petr T
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
adding back a few CC's because this discussion is useful
Quoted 19 lines · Petr Tesarik
V Wed, 12 Nov 2014 15:50:32 +0100
Laszlo Ersek <[email protected]> napsáno:Quoted 16 lines · Petr Tesarik
Quoted 9 lines
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…
Anyway,
phys_baseis kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.
Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.
Quoted 6 lines
Quoted 3 lines
Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs?
If only you could save the _complete_ state of the CPU... For example the content of CR3 would be quite useful.
(1) CR3 is already saved, in both the ELF and the kdump compressed formats.
- ELF case:
qmp_dump_guest_memory() [dump.c]
create_vmcore()
dump_begin()
write_elf64_notes()
loop from 1 to #vcpu:
cpu_write_elf64_note() [qom/cpu.c]
x86_64_write_elf64_note() [target-i386/arch_dump.c]
writes "CORE"
loop from 1 to #vcpu:
cpu_write_elf64_qemunote() [qom/cpu.c]
x86_cpu_write_elf64_qemunote() [target-i386/arch_dump.c]
cpu_write_qemu_note()
qemu_get_cpustate()
s->cr[3] = env->cr[3]; <---------- here
writes "QEMU"Hence, the information is part of the QEMU note.
- kdump case:
qmp_dump_guest_memory() [dump.c]
create_kdump_vmcore()
write_dump_header()
create_header64()
write_elf64_notes()
[... same as above ...]The trick here is that the note-writer functions use a callback function for actually outputting the data. So while in the ELF case the stuff goes directly to a file, in the kdump case the notes are first saved in a memory buffer, and then later saved in the file at offset KdumpSubHeader64.offset_note. (... Which is then represented in the flattened file format of course.)
So, the information is there in both cases.
(2) Dave -- this just made me realize that the QEMU note is *already* there in the kdump file as well; pointed-to by KdumpSubHeader64.offset_note, for a length of KdumpSubHeader64.note_size.
From your other email
<http://thread.gmane.org/gmane.linux.kernel.kexec/12787/focus=12797>:
Quoted 17 lines
sub_header_kdump: 1c9cff0
phys_base: 0
dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 0 (0x0)
size_vmcoreinfo: 0 (0x0)
offset_note: 4200 (0x1068) <----------- here
size_note: 3232 (0xca0) <-----------
num_prstatus_notes: 4
notes_buf: 1c9e000
notes[0]: 1c9e000
notes[1]: 1c9e164
notes[2]: 1c9e2c8
notes[3]: 1c9e42c
NT_PRSTATUS_offset: 1068Quoted 8 lines
11cc
1330
1494
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 1245184 (0x130000)Can you fetch that in "crash"? If you can, then there's nothing to do on the qemu side (and I'll have to apologize for spamming a bunch of lists :/).
I think "crash" already iterates over all of the notes in the note buffer, but skips everything different from NT_PRSTATUS.
(3) Regarding the structure of the notes, we have to consider the placement of the notes and their internal structure. The placement is different between the ELF and the KDUMP file format. The internal structure of the notes is identical between the two file formats.
For example, for a 4 VCPU guest, you end up with note names like
CORE
CORE
CORE
CORE
QEMU
QEMU
QEMU
QEMU
All of these are Elf64_Nhdr structures. The CORE ones have type NT_PRSTATUS, and the QEMU ones have type 0.
(3a) The placement in the ELF file is already handled by "crash". Each note "simply" gets its own ELF note segment/section.
(3b) In the kdump file, the Elf64_Nhdr structures (8 pieces in total, in the above example -- 4x CORE, 4x QEMU) are concatenated in that order, and finally stored at "offset_note".
(3c) Regarding the internal structure of the notes. The CORE ones are already known and handled. The QEMU notes have the following structure:
Quoted 5 lines
Elf64_Nhdr:n_namesz: 5 ("QEMU")n_descsz: 432
n_type: 0 (?)000001b000000001 0000000000000000
|------||------| |--------------|
size version raxQuoted 1 line
0000000000000000 0000000000000000
|--------------| |--------------|
rbx rcxQuoted 1 line
0000000000000000 0000000000000001
|--------------| |--------------|
rdx rsiQuoted 1 line
ffffffff81dd5228 ffffffff81a01ec8
|--------------| |--------------|
rdi rspQuoted 1 line
ffffffff81a01ec8 0000000000000000
|--------------| |--------------|
rbp r8Quoted 1 line
0000000000000000 00000013911d5f29
|--------------| |--------------|
r9 r10Quoted 1 line
0000000000000000 ffffffff81c00480
|--------------| |--------------|
r11 r12Quoted 1 line
0000000000000000 ffffffffffffffff
|--------------| |--------------|
r13 r14Quoted 1 line
000000000309f000 ffffffff810375ab
|--------------| |--------------|
r15 ripQuoted 1 line
0000000000000246 ffffffff00000010
|--------------| |------||------|
rflags cs/lim cs/selQuoted 1 line
0000000000a09b00 0000000000000000
|------||------| |--------------|
cs/pad cs/flags cs/baseQuoted 1 line
ffffffff00000018 0000000000c09300
|------||------| |------||------|
ds/lim ds/sel ds/pad ds/flagsQuoted 1 line
0000000000000000 ffffffff00000018
|--------------| |------||------|
ds/base es/lim es/selQuoted 1 line
0000000000c09300 0000000000000000
|------||------| |--------------|
es/pad es/flags es/baseQuoted 1 line
ffffffff00000000 0000000000000000
|------||------| |------||------|
fs/lim fs/sel fs/pad fs/flagsQuoted 1 line
0000000000000000 ffffffff00000000
|--------------| |------||------|
fs/base gs/lim gs/selQuoted 1 line
0000000000000000 ffff880003200000
|------||------| |--------------|
gs/pad gs/flags gs/baseQuoted 1 line
ffffffff00000018 0000000000c09300
|------||------| |------||------|
ss/lim ss/sel ss/pad ss/flagsQuoted 1 line
0000000000000000 ffffffff00000000
|--------------| |------||------|
ss/base ldt...Quoted 1 line
0000000000000000 0000000000000000
|------||------| |--------------|
...ldtQuoted 1 line
0000208700000040 0000000000008b00
|------||------| |------||------|
tr...Quoted 1 line
ffff880003213b40 0000007f00000000
|--------------| |------||------|
...tr gdt...Quoted 1 line
0000000000000000 ffff880003204000
|------||------| |--------------|
...gdtQuoted 1 line
00000fff00000000 0000000000000000
|------||------| |------||------|
idt...Quoted 1 line
ffffffff81dd2000 000000008005003b
|--------------| |--------------|
...idt cr0Quoted 1 line
0000000000000000 0000000001b2e000
|--------------| |--------------|
cr1 cr2Quoted 1 line
0000000007b18000 00000000000006f0
|--------------| |--------------|
cr3 cr4From "target-i386/arch_dump.c":
Quoted 21 lines
struct QEMUCPUSegment { uint32_t selector; uint32_t limit; uint32_t flags; uint32_t pad; uint64_t base; };typedef struct QEMUCPUSegment QEMUCPUSegment;
struct QEMUCPUState { uint32_t version; uint32_t size; uint64_t rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp; uint64_t r8, r9, r10, r11, r12, r13, r14, r15; uint64_t rip, rflags; QEMUCPUSegment cs, ds, es, fs, gs, ss; QEMUCPUSegment ldt, tr, gdt, idt; uint64_t cr[5]; };
Quoted 1 line
typedef struct QEMUCPUState QEMUCPUState;
Summary: I think the info is all there.
Thanks
Laszlo
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
----- Original Message -----
Quoted 105 lines
adding back a few CC's because this discussion is useful
Quoted 19 lines · Petr Tesarik
V Wed, 12 Nov 2014 15:50:32 +0100
Laszlo Ersek <[email protected]> napsáno:Quoted 16 lines · Petr Tesarik
Quoted 9 lines
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…
Anyway,
phys_baseis kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.
Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.
Quoted 6 lines
Quoted 3 lines
Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs?
If only you could save the _complete_ state of the CPU... For example the content of CR3 would be quite useful.
(1) CR3 is already saved, in both the ELF and the kdump compressed formats.
- ELF case:
qmp_dump_guest_memory()[dump.c]
create_vmcore()dump_begin() write_elf64_notes() loop from 1 to #vcpu: cpu_write_elf64_note() [qom/cpu.c] x86_64_write_elf64_note() [target-i386/arch_dump.c] writes "CORE" loop from 1 to #vcpu: cpu_write_elf64_qemunote() [qom/cpu.c] x86_cpu_write_elf64_qemunote() [target-i386/arch_dump.c] cpu_write_qemu_note() qemu_get_cpustate() s->cr[3] = env->cr[3]; <---------- here writes "QEMU"Hence, the information is part of the QEMU note.
- kdump case:
qmp_dump_guest_memory()[dump.c]
create_kdump_vmcore()write_dump_header() create_header64() write_elf64_notes() [... same as above ...]The trick here is that the note-writer functions use a callback function for actually outputting the data. So while in the ELF case the stuff goes directly to a file, in the kdump case the notes are first saved in a memory buffer, and then later saved in the file at offset KdumpSubHeader64.offset_note. (... Which is then represented in the flattened file format of course.)
So, the information is there in both cases.
(2) Dave -- this just made me realize that the QEMU note is *already* there in the kdump file as well; pointed-to by KdumpSubHeader64.offset_note, for a length of KdumpSubHeader64.note_size.
From your other email
<http://thread.gmane.org/gmane.linux.kernel.kexec/12787/focus=12797>:Quoted 17 lines
sub_header_kdump: 1c9cff0 phys_base: 0 dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO) split: 0 start_pfn: (unused) end_pfn: (unused) offset_vmcoreinfo: 0 (0x0) size_vmcoreinfo: 0 (0x0) offset_note: 4200 (0x1068) <----------- here size_note: 3232 (0xca0) <----------- num_prstatus_notes: 4 notes_buf: 1c9e000 notes[0]: 1c9e000 notes[1]: 1c9e164 notes[2]: 1c9e2c8 notes[3]: 1c9e42c NT_PRSTATUS_offset: 1068Quoted 8 lines
11cc 1330 1494 offset_eraseinfo: 0 (0x0) size_eraseinfo: 0 (0x0) start_pfn_64: (unused) end_pfn_64: (unused) max_mapnr_64: 1245184 (0x130000)
Quoted 2 lines
Can you fetch that in "crash"? If you can, then there's nothing to do on the qemu side (and I'll have to apologize for spamming a bunch of lists :/).
Sure enough...
I was just playing with process_el64_notes() to check/read the note name strings, and noticed that I can certainly see them. But as you noted, only the NT_PRSTATUS notes are stored in the "notes[]" array. so I was under the impression that the QEMU notes were completely missing.
That being the case -- we're pretty much done!
I'll put a patch in the next upstream release of crash.
Thanks,
Dave
Quoted 169 lines
I think "crash" already iterates over all of the notes in the note buffer, but skips everything different from
NT_PRSTATUS.(3) Regarding the structure of the notes, we have to consider the placement of the notes and their internal structure. The placement is different between the ELF and the KDUMP file format. The internal structure of the notes is identical between the two file formats.
For example, for a 4 VCPU guest, you end up with note names like
CORE
CORE
CORE
CORE
QEMU
QEMU
QEMU
QEMUAll of these are
Elf64_Nhdrstructures. The CORE ones have typeNT_PRSTATUS, and the QEMU ones have type 0.(3a) The placement in the ELF file is already handled by "crash". Each note "simply" gets its own ELF note segment/section.
(3b) In the kdump file, the
Elf64_Nhdrstructures (8 pieces in total, in the above example -- 4x CORE, 4x QEMU) are concatenated in that order, and finally stored at "offset_note".(3c) Regarding the internal structure of the notes. The CORE ones are already known and handled. The QEMU notes have the following structure:
Quoted 5 lines
Elf64_Nhdr:n_namesz: 5 ("QEMU")n_descsz: 432
n_type: 0 (?)000001b000000001 0000000000000000|------||------| |--------------| size version raxQuoted 1 line
0000000000000000 0000000000000000|--------------| |--------------| rbx rcxQuoted 1 line
0000000000000000 0000000000000001|--------------| |--------------| rdx rsiQuoted 1 line
ffffffff81dd5228 ffffffff81a01ec8|--------------| |--------------| rdi rspQuoted 1 line
ffffffff81a01ec8 0000000000000000|--------------| |--------------| rbp r8Quoted 1 line
0000000000000000 00000013911d5f29|--------------| |--------------| r9 r10Quoted 1 line
0000000000000000 ffffffff81c00480|--------------| |--------------| r11 r12Quoted 1 line
0000000000000000 ffffffffffffffff|--------------| |--------------| r13 r14Quoted 1 line
000000000309f000 ffffffff810375ab|--------------| |--------------| r15 ripQuoted 1 line
0000000000000246 ffffffff00000010|--------------| |------||------| rflags cs/lim cs/selQuoted 1 line
0000000000a09b00 0000000000000000|------||------| |--------------| cs/pad cs/flags cs/baseQuoted 1 line
ffffffff00000018 0000000000c09300|------||------| |------||------| ds/lim ds/sel ds/pad ds/flagsQuoted 1 line
0000000000000000 ffffffff00000018|--------------| |------||------| ds/base es/lim es/selQuoted 1 line
0000000000c09300 0000000000000000|------||------| |--------------| es/pad es/flags es/baseQuoted 1 line
ffffffff00000000 0000000000000000|------||------| |------||------| fs/lim fs/sel fs/pad fs/flagsQuoted 1 line
0000000000000000 ffffffff00000000|--------------| |------||------| fs/base gs/lim gs/selQuoted 1 line
0000000000000000 ffff880003200000|------||------| |--------------| gs/pad gs/flags gs/baseQuoted 1 line
ffffffff00000018 0000000000c09300|------||------| |------||------| ss/lim ss/sel ss/pad ss/flagsQuoted 1 line
0000000000000000 ffffffff00000000|--------------| |------||------| ss/base ldt...Quoted 1 line
0000000000000000 0000000000000000|------||------| |--------------| ...ldtQuoted 1 line
0000208700000040 0000000000008b00|------||------| |------||------| tr...Quoted 1 line
ffff880003213b40 0000007f00000000|--------------| |------||------| ...tr gdt...Quoted 1 line
0000000000000000 ffff880003204000|------||------| |--------------| ...gdtQuoted 1 line
00000fff00000000 0000000000000000|------||------| |------||------| idt...Quoted 1 line
ffffffff81dd2000 000000008005003b|--------------| |--------------| ...idt cr0Quoted 1 line
0000000000000000 0000000001b2e000|--------------| |--------------| cr1 cr2Quoted 1 line
0000000007b18000 00000000000006f0|--------------| |--------------| cr3 cr4From "
target-i386/arch_dump.c":Quoted 21 lines
struct QEMUCPUSegment { uint32_t selector; uint32_t limit; uint32_t flags; uint32_t pad; uint64_t base; };typedef struct QEMUCPUSegment QEMUCPUSegment;
struct QEMUCPUState { uint32_t version; uint32_t size; uint64_t rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp; uint64_t r8, r9, r10, r11, r12, r13, r14, r15; uint64_t rip, rflags; QEMUCPUSegment cs, ds, es, fs, gs, ss; QEMUCPUSegment ldt, tr, gdt, idt; uint64_t cr[5]; };
Quoted 8 lines
Quoted 1 line
typedef struct QEMUCPUState QEMUCPUState;
Summary: I think the info is all there.
Thanks
Laszlo
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 60 lines · On Wed, 12 Nov 2014 10:43:59 -0500 Christopher Covington <[email protected]>…
Quoted 59 lines · Laszlo Ersek
Quoted 58 lines · Christopher Covington
Thanks Petr and Laszlo for entertaining my questions. I've got one last one if you have the time.
Quoted 54 lines · Laszlo Ersek
Quoted 45 lines · Petr Tesarik
Quoted 42 lines · On Wed, 12 Nov 2014 08:18:04 -0500 Christopher Covington <[email protected]>…
Quoted 41 lines · Petr Tesarik
Quoted 38 lines · On Tue, 11 Nov 2014 12:27:44 -0500 Christopher Covington <[email protected]>…
Quoted 36 lines · Laszlo Ersek
(Note: I'm not subscribed to either qemu-devel or the kexec list; please keep me CC'd.)
QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib, kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
The resultant vmcore is usually analyzed with the "crash" utility.
The original tool producing such files is kdump. Unlike the procedure performed by QEMU, kdump runs from *within* the guest (under a kexec'd kdump kernel), and has more information about the original guest kernel state (which is being dumped) than QEMU. To QEMU, the guest kernel state is opaque.
For this reason, the kdump preparation logic in QEMU hardcodes a number of fields in the kdump header. The direct issue is the "
phys_base" field. Refer todump.c, functionscreate_header32(),create_header64(), and "include/sysemu/dump.h", macroPHYS_BASE(with the replacement text "0").http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
This works in most cases, because the guest Linux kernel indeed tends to be loaded at guest-phys address 0. However, when the guest Linux kernel is booted on top of OVMF (which has a somewhat unusual UEFI memory map), then the guest Linux kernel is loaded at 16MB, thereby getting out of sync with the phys_base=0 setting visible in the KDUMP header.
This trips up the "crash" utility.
Dave worked around the issue in "crash" for ELF format dumps -- "crash" can identify QEMU as the originator of the vmcore by finding the QEMU notes in the ELF vmcore. If those are present, then "crash" employs a heuristic, probing for a
phys_baseup to 32MB, in 1MB steps.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
What advantages does KDUMP have over ELF?
It's smaller (data is compressed), and it contains a header with some useful information (e.g. the crashed kernel's version and release).
Another advantage is that all zero-filled pages are represented in the kdump file by one shared zero page.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
The difference in speed of dumping is stunning.
Would you expect using
SHT_NOBITSto give a similar speedup to the ELF dumper?Sorry, I don't know what
SHT_NOBITSis.My newbie understanding is that
SHT_NOBITSis the section type of the .bss section in an everyday executable.
Heh, yes and no. Let's clarify a few things.
First, a Linux kernel dump (or a QEMU ELF dump) does not contain any sections. It only contains program headers. The reason is that program headers can specify both the virtual address and the physical address. BTW this feature is used to determine the physical base of the Linux kernel when dumping via kexec. Sections can only specify the virtual address. Of course, program headers can achieve an effect similar toSHT_NOBITS: by specifying a larger memory size than file size.
Now, all this does not mean you can't create a new standard that uses sections instead (in fact, Xen DomU dumps already use ELF sections). But of course, this new standard won't be understood by the existing tools until somebody (you?) adds support for it. Same thing is true for the SHF_COMPRESSED section flag.
Anyway, reuse of the zero page is a minor point. The main difference in speed comes from using a faster compression algorithm (LZO or snappy). Of course, nothing prevents you from using those algorithms in ELF compressed sections, but you'd have to extend the ELF standard first, adding magic numbers for these algorithms. To me it sounds like a long way to go.
In short, there's no technical reason why ELF couldn't achieve results similar to KDUMP. It's "merely" not implemented, and for sure, I'm not going to push all the necessary changes. ;-)
Petr T
Quoted 22 lines · On Wed, 12 Nov 2014 21:30:20 +0100 Laszlo Ersek <[email protected]> wrote:
adding back a few CC's because this discussion is useful
Quoted 19 lines · Petr Tesarik
V Wed, 12 Nov 2014 15:50:32 +0100
Laszlo Ersek <[email protected]> napsáno:Quoted 16 lines · Petr Tesarik
Quoted 9 lines
Quoted 3 lines · On Wed, 12 Nov 2014 12:08:38 +0900 (JST) HATAYAMA Daisuke…
Anyway,
phys_baseis kernel information. To make it available for qemu side, there's need to prepare a mechanism for qemu to have any access to it.Yes. I wonder if you can have access without some sort of co-operation from the guest kernel itself. I guess not.
Propagating any kind of additional information from the guest kernel (which is unprivileged and potentially malicious) to the host-side qemu process (which is by definition more privileged, although still confined by various measures) is something we'd explicitly like to avoid.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Think of it like this. I throw a physical box at you, running Linux, that has frozen in time. Can "crash" work with nothing else but the contents of the memory, and information about the CPUs?
If only you could save the _complete_ state of the CPU... For example the content of CR3 would be quite useful.
(1) CR3 is already saved, in both the ELF and the kdump compressed formats.
Sweet. :-)
So, there's no need for any heuristics. Since CR3 gives the physical address of the PML4 table, I can use it to translate __START_KERNEL_map (0xffffffff80000000UL on all Linux kernels since introduction of x86_64) to a physical address and compute phys_base from that.
In fact, QEMU could do the same if you can live with hardcoding a Linux-kernel specific constant into the tool...
Petr T
----- Original Message -----
Quoted 3 lines
Quoted 3 lines
Can you fetch that in "crash"? If you can, then there's nothing to do on the qemu side (and I'll have to apologize for spamming a bunch of lists :/).
Well, let's be clear -- I was the one who put you up to it...
But no apology is required -- and in fact, if today's discussion results in the "phys_base" vmcoreinfo issue being resolved in the kernel, we'll all be eternally grateful.
Thanks again,
Dave
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec