Export offsets of VMCS fields as note information for kdump
Subsystems (6)
v2 cover Export offsets of VMCS fields as note information for kdump
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Here are two usercases for two features that we want.
1) Create guest machine's crash dumpfile from host machine's crash dumpfile
In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.
As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.
The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.
Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.
So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.
Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.
2) Get offsets of VMCS information on the CPU running on the host machine
If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.
TODO:
1. In kexec-tools, get VMCSINFO via sysfs and dump it as note information into vmcore.
2. Dump VMCS region of each guest vcpu and VMCSINFO into qemu-process core file. To do this, we will modify kernel core dumper, gdb gcore and crash gcore.
3. Dump guest image from the qemu-process core file into a vmcore.
Changelog for v1 to v2:
1. The VMCSINFO now has a simple binary <field><encoded offset> format, as below:
+-------------+--------------------------+
| Byte offset | Contents |
+-------------+--------------------------+
| 0 | VMCS revision identifier |
+-------------+--------------------------+
| 4 | <field><encoded offset> |
+-------------+--------------------------+
| 16 | <field><encoded offset> |
+-------------+--------------------------+
......The first 32 bits of VMCSINFO contains the VMCS revision identifier. The remainder of VMCSINFO is used for <field><encoded offset> sets. Each set takes 12 bytes: field occupys 4 bytes and its corresponding encoded offset occupys 8 bytes.
Encoded offsets are raw values read by vmcs_read{16, 64, 32, l}, and they are all unsigned extended to 8 bytes for each <field><encoded offset> set will have the same size.
We do not decode offsets here. The decoding work is delayed in userspace tools for more flexible handling.
And here are two examples of the new VMCSINFO:
Processor: Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
VMCSINFO contains:
<0000000d> --> VMCS revision id = 0xd
<00004000><0000000001840180> --> OFFSET(PIN_BASED_VM_EXEC_CONTROL) = 0x01840180
<00004002><0000000001940190> --> OFFSET(CPU_BASED_VM_EXEC_CONTROL) = 0x01940190
<0000401e><000000000fe40fe0> --> OFFSET(SECONDARY_VM_EXEC_CONTROL) = 0x0fe40fe0
<0000400c><0000000001e401e0> --> OFFSET(VM_EXIT_CONTROLS) = 0x01e401e0
...... Processor: Intel(R) Xeon(R) CPU E7540 @ 2.00GHz (24 cores)
VMCSINFO contains:
<0000000e> --> VMCS revision id = 0xe
<00004000><0000000005540550> --> OFFSET(PIN_BASED_VM_EXEC_CONTROL) = 0x05540550
<00004002><0000000005440540> --> OFFSET(CPU_BASED_VM_EXEC_CONTROL) = 0x05440540
<0000401e><00000000054c0548> --> OFFSET(SECONDARY_VM_EXEC_CONTROL) = 0x054c0548
<0000400c><00000000057c0578> --> OFFSET(VM_EXIT_CONTROLS) = 0x057c0578
......2. Add a new kernel module *vmcsinfo-intel* for filling VMCSINFO instead of putting it in module kvm-intel. The new module is auto-loaded when the vmx cpufeature is detected and it depends on module kvm-intel. *Loading and unloading this module will have no side effect on the running guests.*
3. The sysfs file vmcsinfo is splitted into 2 files:
/sys/kernel/vmcsinfo: shows physical address of VMCSINFO note information. /sys/kernel/vmcsinfo_maxsize: shows max size of VMCSINFO.
4. A new Documentation/ABI entry is added for vmcsinfo and vmcsinfo_maxsize.
5. Do not update VMCSINFO note when the kernel is panicked.
zhangyanfei (5):
x86: Add helper variables and functions to hold VMCSINFO
KVM: Export symbols for module vmcsinfo-intel
KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
ksysfs: Export VMCSINFO via sysfs
Documentation: Add ABI entry for sysfs file vmcsinfo and
vmcsinfo_maxsize
Documentation/ABI/testing/sysfs-kernel-vmcsinfo | 16 + arch/x86/include/asm/vmcsinfo.h | 34 ++ arch/x86/include/asm/vmx.h | 133 ++++++++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/vmcsinfo.c | 79 +++++ arch/x86/kvm/Kconfig | 11 + arch/x86/kvm/Makefile | 3 + arch/x86/kvm/vmcsinfo.c | 402 +++++++++++++++++++++++ arch/x86/kvm/vmx.c | 151 ++------- include/linux/kvm_host.h | 3 + kernel/ksysfs.c | 29 ++ virt/kvm/kvm_main.c | 8 +- 12 files changed, 740 insertions(+), 131 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-vmcsinfo create mode 100644 arch/x86/include/asm/vmcsinfo.h create mode 100644 arch/x86/kernel/vmcsinfo.c create mode 100644 arch/x86/kvm/vmcsinfo.c
PATCH v2 1/5 x86: Add helper variables and functions to hold VMCSINFO
This patch provides a set of variables to hold the VMCSINFO and also some helper functions to help fill the VMCSINFO.
--- arch/x86/include/asm/vmcsinfo.h | 34 +++++++++++++++++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/vmcsinfo.c | 79 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 115 insertions(+), 0 deletions(-) create mode 100644 arch/x86/include/asm/vmcsinfo.h create mode 100644 arch/x86/kernel/vmcsinfo.c
arch/x86/include/asm/vmcsinfo.h new file +34 -0
@@ -0,0 +1,34 @@ +#ifndef _ASM_X86_VMCSINFO_H +#define _ASM_X86_VMCSINFO_H + +#ifndef __ASSEMBLY__ +#include <linux/types.h> +#include <linux/elf.h> + +/* +* Currently, 1 page is enough for vmcsinfo. +*/ +#define VMCSINFO_BYTES (4096) +#define VMCSINFO_NOTE_NAME "VMCSINFO" +#define VMCSINFO_NOTE_NAME_BYTES ALIGN(sizeof(VMCSINFO_NOTE_NAME), 4) +#define VMCSINFO_NOTE_HEAD_BYTES ALIGN(sizeof(struct elf_note), 4) +#define VMCSINFO_NOTE_SIZE (VMCSINFO_NOTE_HEAD_BYTES*2 \ + + VMCSINFO_BYTES \ + + VMCSINFO_NOTE_NAME_BYTES) + +extern size_t vmcsinfo_size; +extern size_t vmcsinfo_max_size; +extern unsigned char vmcsinfo_data[VMCSINFO_BYTES ]; + +extern void update_vmcsinfo_note (void ); +extern void vmcsinfo_append_id (u32 ); +extern void vmcsinfo_append_field (u32 ,u64 ); +extern unsigned long paddr_vmcsinfo_note (void ); + +#define VMCSINFO_REVISION_ID (id) \ + vmcsinfo_append_id(id) +#define VMCSINFO_FIELD (field, offset) \ + vmcsinfo_append_field(field, offset) + +#endif /* __ASSEMBLY__ */ +#endif /* _ASM_X86_VMCSINFO_H */
arch/x86/kernel/Makefile +2 -0
@@ -102,6 +102,8 @@ obj-$(CONFIG_X86_CHECK_BIOS_CORRUPTION) += check.o obj-$(CONFIG_SWIOTLB) += pci-swiotlb.o obj-$(CONFIG_OF) += devicetree.o +obj-y += vmcsinfo.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/vmcsinfo.c new file +79 -0
79 lines not on this page · raw message
Signature · 1 line
1.7.1
Re: [PATCH v2 1/5] x86: Add helper variables and functions to hold VMCSINFO
Quoted 2 lines · zhangyanfei
This patch provides a set of variables to hold the VMCSINFO and also some helper functions to help fill the VMCSINFO.
Need to document the format.
Quoted 4 lines
arch/x86/kernel/vmcsinfo.c new file +4
-0+void vmcsinfo_append_id (u32 id) +{ +size_t r; +
arch/x86/kernel/vmcsinfo.c line 61
Quoted 10 lines
arch/x86/kernel/vmcsinfo.c new file +10
-0+ r =sizeof (id); +if (r + vmcsinfo_size > vmcsinfo_max_size) +return ; + +memcpy (&vmcsinfo_data[vmcsinfo_size], &id, r); + vmcsinfo_size += r; +} +EXPORT_SYMBOL_GPL (vmcsinfo_append_id); + +void vmcsinfo_append_field (u32 field,u64 offset)
Why u64? It's guaranteed to fit within a page.
arch/x86/kernel/vmcsinfo.c line 72
Quoted 11 lines
arch/x86/kernel/vmcsinfo.c new file +11
-0+{ +size_t r; + + r =sizeof (field) +sizeof (offset); +if (r + vmcsinfo_size > vmcsinfo_max_size) +return ; + +memcpy (&vmcsinfo_data[vmcsinfo_size], &field,sizeof (field)); + vmcsinfo_size +=sizeof (field); +memcpy (&vmcsinfo_data[vmcsinfo_size], &offset,sizeof (offset)); + vmcsinfo_size +=sizeof (offset);
Instead of this vmcsinfo_data, how about a struct with fields for the revision ID and field count, and an array for the fields? Should be a lot simpler.
Quoted 8 lines
arch/x86/kernel/vmcsinfo.c new file +7
-0+} +EXPORT_SYMBOL_GPL (vmcsinfo_append_field); + +unsigned long paddr_vmcsinfo_note (void ) +{ +return __pa ((unsigned long )(char *)&vmcsinfo_note); +}
Signature · 1 line
error compiling committee.c: too many arguments to function
PATCH v2 2/5 KVM: Export symbols for module vmcsinfo-intel
In reply to zhangyanfeiA new module named vmcsinfo-intel is used to fill VMCSINFO. And this module depends on kvm-intel and kvm module. So we should export some symbols of kvm-intel and kvm module that are needed by vmcsinfo-intel.
--- arch/x86/include/asm/vmx.h | 133 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx.c | 151 +++++++------------------------------------- include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 8 +- 4 files changed, 164 insertions(+), 131 deletions(-)
arch/x86/include/asm/vmx.h +133 -0
@@ -27,6 +27,8 @@#include <linux/types.h> +#include <asm/kvm_host.h> +/* * Definitions of Primary Processor-Based VM-Execution Controls. */
@@ -481,4 +483,135 @@ enum vm_instruction_error_number { VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID = 28, }; +#define __ex (x) __kvm_handle_fault_on_reboot(x) +#define __ex_clear (x, reg) \ + ____kvm_handle_fault_on_reboot(x, "xor " reg " , " reg) + +struct vmcs { +u32 revision_id; +u32 abort; +char data[0 ]; +}; + +struct vmcs_config { +int size; +int order; +u32 revision_id; +u32 pin_based_exec_ctrl; +u32 cpu_based_exec_ctrl; +u32 cpu_based_2nd_exec_ctrl; +u32 vmexit_ctrl; +u32 vmentry_ctrl; +}; + +extern struct vmcs_config vmcs_config; + +DECLARE_PER_CPU(struct vmcs *, vmxarea); +DECLARE_PER_CPU(struct vmcs *, current_vmcs); + +struct vmcs *alloc_vmcs (void ); +void kvm_cpu_vmxon (u64 ); +void kvm_cpu_vmxoff (void ); +void vmcs_load (struct vmcs *); +void vmcs_write_control_field (unsigned long ,u32 ); +void vmcs_clear (struct vmcs *); +void free_vmcs (struct vmcs *); + +static __always_inline unsigned long vmcs_readl (unsigned long field) +{ +unsigned long value; + + asm volatile (__ex_clear(ASM_VMX_VMREAD_RDX_RAX, "%0") + : "=a"(value) : "d"(field) : "cc"); +return value; +} + +static __always_inline u16 vmcs_read16(unsigned long field) +{ +return vmcs_readl (field); +} + +static __always_inline u32 vmcs_read32(unsigned long field) +{ +return vmcs_readl (field); +} + +static __always_inline u64 vmcs_read64(unsigned long field) +{ +#ifdef CONFIG_X86_64 +return vmcs_readl (field); +#else +return vmcs_readl (field) | ((u64 )vmcs_readl (field+1 ) <<32 ); +#endif +} + +static inline bool cpu_has_vmx_msr_bitmap (void ) +{ +return vmcs_config.cpu_based_exec_ctrl &CPU_BASED_USE_MSR_BITMAPS ; +} + +static inline bool cpu_has_vmx_tpr_shadow (void ) +{ +return vmcs_config.cpu_based_exec_ctrl &CPU_BASED_TPR_SHADOW ; +} + +static inline bool cpu_has_secondary_exec_ctrls (void ) +{ +return vmcs_config.cpu_based_exec_ctrl & +CPU_BASED_ACTIVATE_SECONDARY_CONTROLS ; +} + +static inline bool cpu_has_vmx_virtualize_apic_accesses (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl & +SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; +} + +static inline bool cpu_has_vmx_flexpriority (void ) +{ +return cpu_has_vmx_tpr_shadow () && +cpu_has_vmx_virtualize_apic_accesses (); +} + +static inline bool cpu_has_vmx_ept (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl & +SECONDARY_EXEC_ENABLE_EPT ; +} + +static inline bool cpu_has_vmx_unrestricted_guest (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl & +SECONDARY_EXEC_UNRESTRICTED_GUEST ; +} + +static inline bool cpu_has_vmx_ple (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl & +SECONDARY_EXEC_PAUSE_LOOP_EXITING ; +} + +static inline bool cpu_has_vmx_vpid (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl & +SECONDARY_EXEC_ENABLE_VPID ; +} + +static inline bool cpu_has_vmx_rdtscp (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl & +SECONDARY_EXEC_RDTSCP ; +} + +static inline bool cpu_has_virtual_nmis (void ) +{ +return vmcs_config.pin_based_exec_ctrl &PIN_BASED_VIRTUAL_NMIS ; +} + +static inline bool cpu_has_vmx_wbinvd_exit (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl & +SECONDARY_EXEC_WBINVD_EXITING ; +} +#endif
arch/x86/kvm/vmx.c +24 -127
269 lines not on this page · raw message
include/linux/kvm_host.h +3 -0
@@ -80,6 +80,9 @@ enum kvm_bus {KVM_NR_BUSES }; +int hardware_enable_all (void ); +void hardware_disable_all (void ); +int kvm_io_bus_write (struct kvm *kvm,enum kvm_bus bus_idx,gpa_t addr,int len,const void *val); int kvm_io_bus_read(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr, int len,
virt/kvm/kvm_main.c +4 -4
@@ -90,8 +90,6 @@ static long kvm_vcpu_ioctl(struct file *file, unsigned int ioctl,static long kvm_vcpu_compat_ioctl (struct file *file,unsigned int ioctl,unsigned long arg);#endif -static int hardware_enable_all (void ); -static void hardware_disable_all (void );static void kvm_io_bus_destroy (struct kvm_io_bus *bus);
@@ -2286,14 +2284,15 @@ static void hardware_disable_all_nolock(void)on_each_cpu (hardware_disable_nolock,NULL ,1 ); } -static void hardware_disable_all (void ) +void hardware_disable_all (void ) {raw_spin_lock (&kvm_lock);hardware_disable_all_nolock ();raw_spin_unlock (&kvm_lock); } +EXPORT_SYMBOL_GPL (hardware_disable_all); -static int hardware_enable_all (void ) +int hardware_enable_all (void ) {int r =0 ;
@@ -2314,6 +2313,7 @@ static int hardware_enable_all(void)return r; } +EXPORT_SYMBOL_GPL (hardware_enable_all);static int kvm_cpu_hotplug (struct notifier_block *notifier,unsigned long val,void *v)
Signature · 1 line
1.7.1
PATCH v2 3/5 KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
In reply to zhangyanfeiThis patch implements a new module named vmcsinfo-intel. The module fills VMCSINFO with the VMCS revision identifier, and encoded offsets of VMCS fields.
Note, offsets of fields below will not be filled into VMCSINFO:
1. fields defined in Intel specification (Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C) but not defined in *vmcs_field*.
2. fields don't exist because their corresponding control bits are not set.
--- arch/x86/kvm/Kconfig | 11 ++ arch/x86/kvm/Makefile | 3 + arch/x86/kvm/vmcsinfo.c | 402 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 416 insertions(+), 0 deletions(-) create mode 100644 arch/x86/kvm/vmcsinfo.c
arch/x86/kvm/Kconfig +11 -0
@@ -62,6 +62,17 @@ config KVM_INTEL To compile this as a module, choose M here: the module will be called kvm-intel. +config VMCSINFO_INTEL + tristate "Export VMCSINFO for Intel processors" + depends on KVM_INTEL + ---help--- + Provides support for exporting VMCSINFO on Intel processors equipped + with the VT extensions. The VMCSINFO contains a VMCS revision + identifier and offsets of VMCS fields. + + To compile this as a module, choose M here: the module + will be called vmcsinfo-intel. + config KVM_AMD tristate "KVM for AMD processors support" depends on KVM
arch/x86/kvm/Makefile +3 -0
@@ -4,6 +4,7 @@ ccflags-y += -Ivirt/kvm -Iarch/x86/kvm CFLAGS_x86.o := -I. CFLAGS_svm.o := -I. CFLAGS_vmx.o := -I. +CFLAGS_vmcsinfo.o := -I. kvm-y += $(addprefix ../../../virt/kvm/, kvm_main.o ioapic.o \ coalesced_mmio.o irq_comm.o eventfd.o \
@@ -15,7 +16,9 @@ kvm-y += x86.o mmu.o emulate.o i8259.o irq.o lapic.o \ i8254.o timer.o cpuid.o pmu.o kvm-intel-y += vmx.o kvm-amd-y += svm.o +vmcsinfo-intel-y += vmcsinfo.o obj-$(CONFIG_KVM) += kvm.o obj-$(CONFIG_KVM_INTEL) += kvm-intel.o obj-$(CONFIG_KVM_AMD) += kvm-amd.o +obj-$(CONFIG_VMCSINFO_INTEL) += vmcsinfo-intel.o
arch/x86/kvm/vmcsinfo.c new file +402 -0
402 lines not on this page · raw message
Signature · 6 lines
1.7.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v2 3/5] KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
Quoted 11 lines · zhangyanfei
This patch implements a new module named vmcsinfo-intel. The module fills VMCSINFO with the VMCS revision identifier, and encoded offsets of VMCS fields.
Note, offsets of fields below will not be filled into VMCSINFO:
1. fields defined in Intel specification (Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C) but not defined in *vmcs_field*.
2. fields don't exist because their corresponding control bits are not set.
arch/x86/kvm/vmcsinfo.c line 67
Quoted 6 lines
arch/x86/kvm/vmcsinfo.c new file +6
-0+ +/* +* We separate these five control fields from other fields +* because some fields only exist on processors that support +* the 1-setting of control bits in the five control fields. +*/
I thought this was checked only during VMENTRY. So perhaps you don't need this special casing.
In fact you might be able to
// pre-fill vmcs with patterns
for (i = 0; i < 64k; ++i)
if (vmcs_read_checking(i, &pattern)) {
// decode pattern
} else
// field does not exist (VM Instruction error 12), ignorewith no knowledge of the control fields, or of any field name.
Quoted 15 lines
arch/x86/kvm/vmcsinfo.c new file +15
-0+ +/* +* The format of VMCSINFO is given below: +* +-------------+--------------------------+ +* | Byte offset | Contents | +* +-------------+--------------------------+ +* | 0 | VMCS revision identifier | +* +-------------+--------------------------+ +* | 4 | <field><encoded offset> | +* +-------------+--------------------------+ +* | 16 | <field><encoded offset> | +* +-------------+--------------------------+ +* ...... +* +* The first 32 bits of VMCSINFO contains the VMCS revision
arch/x86/kvm/vmcsinfo.c line 320
Quoted 10 lines
arch/x86/kvm/vmcsinfo.c new file +10
-0+* identifier. +* The remainder of VMCSINFO is used for <field><encoded offset> +* sets. Each set takes 12 bytes: field occupys 4 bytes +* and its corresponding encoded offset occupys 8 bytes. +* +* Encoded offsets are raw values read by vmcs_read{16, 64, 32, l}, +* and they are all unsigned extended to 8 bytes for each +* <field><encoded offset> set has the same size. +* We do not decode offsets here. The decoding work is delayed +* in userspace tools.
It's better to do the decoding here, or no one will know how to do it. Also have an nfields field.
Signature · 8 lines
error compiling committee.c: too many arguments to function _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v2 3/5] KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
From: Avi Kivity <[email protected]>
Subject: Re: [PATCH v2 3/5] KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
Date: Thu, 14 Jun 2012 16:37:58 +0300
Quoted 1 line · zhangyanfei
<cut>
Quoted 20 lines
Quoted 20 lines
arch/x86/kvm/vmcsinfo.c new file +20
-0+ +/* +* The format of VMCSINFO is given below: +* +-------------+--------------------------+ +* | Byte offset | Contents | +* +-------------+--------------------------+ +* | 0 | VMCS revision identifier | +* +-------------+--------------------------+ +* | 4 | <field><encoded offset> | +* +-------------+--------------------------+ +* | 16 | <field><encoded offset> | +* +-------------+--------------------------+ +* ...... +* +* The first 32 bits of VMCSINFO contains the VMCS revision +* identifier. +* The remainder of VMCSINFO is used for <field><encoded offset> +* sets. Each set takes 12 bytes: field occupys 4 bytes +* and its corresponding encoded offset occupys 8 bytes. +*
Quoted 8 lines
Quoted 5 lines
+ * Encoded offsets are raw values read by vmcs_read{16, 64, 32, l},
+ * and they are all unsigned extended to 8 bytes for each
+ * <field><encoded offset> set has the same size.
+ * We do not decode offsets here. The decoding work is delayed
+ * in userspace tools.It's better to do the decoding here, or no one will know how to do it. Also have an nfields field.
We did so because actual internal format is unkown; it could possibly be encrypted, although unlikely. We thought processing such unkown data should have been done in userland debugging tools. But decoding them here is no problem; the decode is of simple shift operations and has no risk affecting system status, there's only possibility to display broken values, which is same as the case displaying the encrypted values for users.
FYI, the assumptions behind the reverse enginerring method are:
- For each field, the corresponding offset is unique; IOW, arbitrary two offsets for the corresponding two fields are different each other.
- Field size remains 8 bytes, 16 bytes, 32 bytes even on vmcs region.
- Each field is 8 byte alighed on vmcs region.
- Some fields may be big endition on vmcs region.
- We found this fact under development. We give up if the data is modified more drastically.
- offset < vmcs region size
Thanks.
HATAYAMA, Daisuke
PATCH v2 4/5 ksysfs: Export VMCSINFO via sysfs
In reply to zhangyanfeiThis patch creates two sysfs files to export where VMCSINFO is allocated and what maximum size of VMCSINFO is, as below:
$ cat /sys/kernel/vmcsinfo
1cb88a0
$ cat /sys/kernel/vmcsinfo_maxsize
1000
/sys/kernel/vmcsinfo shows the physical address of VMCSINFO,
while /sys/kernel/vmcsinfo_maxsize shows the max size of VMCSINFO.--- kernel/ksysfs.c | 29 +++++++++++++++++++++++++++++ 1 files changed, 29 insertions(+), 0 deletions(-)
kernel/ksysfs.c +29 -0
@@ -18,6 +18,8 @@#include <linux/stat.h> #include <linux/sched.h> #include <linux/capability.h> +#include <asm/vmcsinfo.h> +#include <asm/virtext.h> #define KERNEL_ATTR_RO (_name) \ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
@@ -133,6 +135,29 @@ KERNEL_ATTR_RO(vmcoreinfo);#endif /* CONFIG_KEXEC */ +#ifdef CONFIG_X86 +static ssize_t vmcsinfo_show (struct kobject *kobj, +struct kobj_attribute *attr,char *buf) +{ +if (cpu_has_vmx ()) +return sprintf (buf,"%lx\n" , +paddr_vmcsinfo_note ()); +return 0 ; +} +KERNEL_ATTR_RO (vmcsinfo); + +static ssize_t vmcsinfo_maxsize_show (struct kobject *kobj, +struct kobj_attribute *attr,char *buf) +{ +if (cpu_has_vmx ()) +return sprintf (buf,"%x\n" , + (unsigned int )vmcsinfo_max_size); +return 0 ; +} +KERNEL_ATTR_RO (vmcsinfo_maxsize); + +#endif /* CONFIG_X86 */ +/* whether file capabilities are enabled */ static ssize_t fscaps_show (struct kobject *kobj,struct kobj_attribute *attr,char *buf)
@@ -182,6 +207,10 @@ static struct attribute * kernel_attrs[] = { &kexec_crash_size_attr.attr, &vmcoreinfo_attr.attr, #endif +#ifdef CONFIG_X86 + &vmcsinfo_attr.attr, + &vmcsinfo_maxsize_attr.attr, +#endifNULL };
Signature · 1 line
1.7.1
PATCH v2 5/5 Documentation: Add ABI entry for sysfs file vmcsinfo and vmcsinfo_maxsize
In reply to zhangyanfeiWe create two new sysfs files, vmcsinfo and vmcsinfo_maxsize. And here we add an Documentation/ABI entry for them.
--- Documentation/ABI/testing/sysfs-kernel-vmcsinfo | 16 ++++++++++++++++ 1 files changed, 16 insertions(+), 0 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-vmcsinfo
Documentation/ABI/testing/sysfs-kernel-vmcsinfo new file +16 -0
@@ -0,0 +1,16 @@ +What: /sys/kernel/vmcsinfo +Date: April 2012 +KernelVersion: 3.4.0 +Contact: Zhang Yanfei <[email protected]> +Description + Shows physical address of VMCSINFO. VMCSINFO contains + the VMCS revision identifier and encoded offsets of fields + in VMCS data on Intel processors equipped with the VT + extensions. + +What: /sys/kernel/vmcsinfo_maxsize +Date: April 2012 +KernelVersion: 3.4.0 +Contact: Zhang Yanfei <[email protected]> +Description + Shows max size of VMCSINFO.
Signature · 1 line
1.7.1
Re: [PATCH v2 5/5] Documentation: Add ABI entry for sysfs file vmcsinfo and vmcsinfo_maxsize
Quoted 17 lines · zhangyanfei
We create two new sysfs files, vmcsinfo and
vmcsinfo_maxsize. And here we add anDocumentation/ABIentry for them.Documentation/ABI/testing/sysfs-kernel-vmcsinfo new file +6
-0@@ -0,0 +1,16 @@ +What: /sys/kernel/vmcsinfo +Date: April 2012 +KernelVersion: 3.4.0 +Contact: Zhang Yanfei <[email protected]> +Description + Shows physical address of VMCSINFO. VMCSINFO contains
Documentation/ABI/testing/sysfs-kernel-vmcsinfo line 16
Quoted 10 lines
Documentation/ABI/testing/sysfs-kernel-vmcsinfo new file +10
-0+ the VMCS revision identifier and encoded offsets of fields + in VMCS data on Intel processors equipped with the VT + extensions. + +What: /sys/kernel/vmcsinfo_maxsize +Date: April 2012 +KernelVersion: 3.4.0 +Contact: Zhang Yanfei <[email protected]> +Description + Shows max size of VMCSINFO.
This describes the cpu, so maybe /sys/devices/cpu is a better place for these files.
Would it make sense to expose the actual fields?
that is, have /sys/devices/cpu/vmcs/0806 contain the offset of GUEST_DS_SELECTOR.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 23 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Here are two usercases for two features that we want.
1) Create guest machine's crash dumpfile from host machine's crash dumpfile
In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.
As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.
Quoted 3 lines
The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.
Do you mean, that a heartbeat failure in the guest lead to host panic?
My expectation is that a problem in the guest will cause the guest to panic and perhaps produce a dump; the host will remain up.
Quoted 11 lines
Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.
So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.
Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.
Quoted 7 lines
2) Get offsets of VMCS information on the CPU running on the host machine
If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.
Shouldn't sadump then expose the VMCS offsets? Perhaps bundling them into its dump file?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 24 lines · Avi Kivity
Quoted 23 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Here are two usercases for two features that we want.
1) Create guest machine's crash dumpfile from host machine's crash dumpfile
In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.
As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.
Quoted 9 lines
Quoted 3 lines
The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.
Do you mean, that a heartbeat failure in the guest lead to host panic?
My expectation is that a problem in the guest will cause the guest to panic and perhaps produce a dump; the host will remain up.
The point is that before our investigation, we didn't know which side leads to this buggy situation. Maybe a bug in host machine or the guest machine itself causes a heartbeat failure.
So we want to get both host machine's crash dump and guest machine's crash dump *at the same time*. Then we could use userspace tools to get guest machine crash dump from host machine's and analyse them separately to find which side causes the problem.
Quoted 13 lines
Quoted 13 lines
Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.
So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.
Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.
2) Get offsets of VMCS information on the CPU running on the host machine
Quoted 9 lines
Quoted 5 lines
If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.
Shouldn't sadump then expose the VMCS offsets? Perhaps bundling them into its dump file?
Firmware-based crash dump doesn't concern the os running on the machine. So it will not do any os handling when machine crashes.
Thanks
Zhang Yanfei
Quoted 31 lines · Yanfei Zhang
Quoted 30 lines · Avi Kivity
Quoted 23 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Here are two usercases for two features that we want.
1) Create guest machine's crash dumpfile from host machine's crash dumpfile
In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.
As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.
Quoted 3 lines
The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.
Do you mean, that a heartbeat failure in the guest lead to host panic?
Quoted 7 lines
Quoted 3 lines
My expectation is that a problem in the guest will cause the guest to panic and perhaps produce a dump; the host will remain up.
The point is that before our investigation, we didn't know which side leads to this buggy situation. Maybe a bug in host machine or the guest machine itself causes a heartbeat failure.
How can a guest bug cause a host panic?
Quoted 5 lines
So we want to get both host machine's crash dump and guest machine's crash dump *at the same time*. Then we could use userspace tools to get guest machine crash dump from host machine's and analyse them separately to find which side causes the problem.
If the guest caused the problem, there would be no panic; therefore there was a host bug.
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.
So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.
Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.
2) Get offsets of VMCS information on the CPU running on the host machine
If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.
Quoted 6 lines
Quoted 3 lines
Shouldn't sadump then expose the VMCS offsets? Perhaps bundling them into its dump file?
Firmware-based crash dump doesn't concern the os running on the machine. So it will not do any os handling when machine crashes.
Seems to me the VMCS offsets are OS independent.
Signature · 7 lines
error compiling committee.c: too many arguments to function _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 42 lines · Avi Kivity
Quoted 31 lines · Yanfei Zhang
Quoted 30 lines · Avi Kivity
Quoted 23 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Here are two usercases for two features that we want.
1) Create guest machine's crash dumpfile from host machine's crash dumpfile
In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.
As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.
Quoted 3 lines
The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.
Do you mean, that a heartbeat failure in the guest lead to host panic?
Quoted 7 lines
Quoted 3 lines
My expectation is that a problem in the guest will cause the guest to panic and perhaps produce a dump; the host will remain up.
The point is that before our investigation, we didn't know which side leads to this buggy situation. Maybe a bug in host machine or the guest machine itself causes a heartbeat failure.
How can a guest bug cause a host panic?
Quoted 9 lines
Quoted 5 lines
So we want to get both host machine's crash dump and guest machine's crash dump *at the same time*. Then we could use userspace tools to get guest machine crash dump from host machine's and analyse them separately to find which side causes the problem.
If the guest caused the problem, there would be no panic; therefore there was a host bug.
Yes, a guest bug cannot cause a host panic. When heartbeat stops in guest machine, we could trigger the host dump mechanism to work. This is because we want to get the status of both host and guest machine at the same time when heartbeat stops in guest machine. Then we can look for bug reasons from both host machine's and guest machine's views.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.
So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.
Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.
2) Get offsets of VMCS information on the CPU running on the host machine
If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Shouldn't sadump then expose the VMCS offsets? Perhaps bundling them into its dump file?
Firmware-based crash dump doesn't concern the os running on the machine. So it will not do any os handling when machine crashes.
Seems to me the VMCS offsets are OS independent.
Hmm, you mean we could get VMCS offsets in sadump itself?
But I think if we just export VMCS offsets in kernel, we could use the current existing dump tools with no or just very tiny change. I think this could be a more general mechanism than making changes in all kinds of dump tools.
Thanks
Zhang Yanfei
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 53 lines · Yanfei Zhang
Quoted 42 lines · Avi Kivity
Quoted 31 lines · Yanfei Zhang
Quoted 30 lines · Avi Kivity
Quoted 23 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Here are two usercases for two features that we want.
1) Create guest machine's crash dumpfile from host machine's crash dumpfile
In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.
As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.
Quoted 3 lines
The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.
Do you mean, that a heartbeat failure in the guest lead to host panic?
Quoted 7 lines
Quoted 3 lines
My expectation is that a problem in the guest will cause the guest to panic and perhaps produce a dump; the host will remain up.
The point is that before our investigation, we didn't know which side leads to this buggy situation. Maybe a bug in host machine or the guest machine itself causes a heartbeat failure.
How can a guest bug cause a host panic?
Quoted 9 lines
Quoted 5 lines
So we want to get both host machine's crash dump and guest machine's crash dump *at the same time*. Then we could use userspace tools to get guest machine crash dump from host machine's and analyse them separately to find which side causes the problem.
If the guest caused the problem, there would be no panic; therefore there was a host bug.
Quoted 5 lines
Yes, a guest bug cannot cause a host panic. When heartbeat stops in guest machine, we could trigger the host dump mechanism to work. This is because we want to get the status of both host and guest machine at the same time when heartbeat stops in guest machine. Then we can look for bug reasons from both host machine's and guest machine's views.
That sounds like a bad idea. Can you explain in what situation it makes sense for a guest to stop the host (and all other guests running on it) rather than just restarting the failed services (on the host or other guests)?
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 13 lines
Quoted 13 lines
Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.
So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.
Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.
2) Get offsets of VMCS information on the CPU running on the host machine
Quoted 9 lines
Quoted 5 lines
If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.
Shouldn't sadump then expose the VMCS offsets? Perhaps bundling them into its dump file?
Firmware-based crash dump doesn't concern the os running on the machine. So it will not do any os handling when machine crashes.
Quoted 6 lines
Quoted 2 lines
Seems to me the VMCS offsets are OS independent.
Hmm, you mean we could get VMCS offsets in sadump itself?
But I think if we just export VMCS offsets in kernel, we could use the current existing dump tools with no or just very tiny change. I think this could be a more general mechanism than making changes in all kinds of dump tools.
The sadump tool generates a core file with the OS image, right? Can it not attach the offsets to a note, just like you propose for kdump?
Signature · 7 lines
error compiling committee.c: too many arguments to function _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 60 lines · Avi Kivity
Quoted 53 lines · Yanfei Zhang
Quoted 42 lines · Avi Kivity
Quoted 31 lines · Yanfei Zhang
Quoted 30 lines · Avi Kivity
Quoted 23 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Here are two usercases for two features that we want.
1) Create guest machine's crash dumpfile from host machine's crash dumpfile
In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.
As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.
Quoted 3 lines
The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.
Do you mean, that a heartbeat failure in the guest lead to host panic?
Quoted 7 lines
Quoted 3 lines
My expectation is that a problem in the guest will cause the guest to panic and perhaps produce a dump; the host will remain up.
The point is that before our investigation, we didn't know which side leads to this buggy situation. Maybe a bug in host machine or the guest machine itself causes a heartbeat failure.
How can a guest bug cause a host panic?
Quoted 9 lines
Quoted 5 lines
So we want to get both host machine's crash dump and guest machine's crash dump *at the same time*. Then we could use userspace tools to get guest machine crash dump from host machine's and analyse them separately to find which side causes the problem.
If the guest caused the problem, there would be no panic; therefore there was a host bug.
Quoted 5 lines
Yes, a guest bug cannot cause a host panic. When heartbeat stops in guest machine, we could trigger the host dump mechanism to work. This is because we want to get the status of both host and guest machine at the same time when heartbeat stops in guest machine. Then we can look for bug reasons from both host machine's and guest machine's views.
Quoted 5 lines
That sounds like a bad idea. Can you explain in what situation it makes sense for a guest to stop the host (and all other guests running on it) rather than just restarting the failed services (on the host or other guests)?
We never do this on customer's environment which maybe a host with many guests running on it. We do this on another environment to reproduce the buggy situation; or we do this in testing phase on development environment towards production one on the customer's site.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.
So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.
Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.
2) Get offsets of VMCS information on the CPU running on the host machine
If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.
Quoted 6 lines
Quoted 3 lines
Shouldn't sadump then expose the VMCS offsets? Perhaps bundling them into its dump file?
Firmware-based crash dump doesn't concern the os running on the machine. So it will not do any os handling when machine crashes.
Seems to me the VMCS offsets are OS independent.
Quoted 8 lines
Quoted 4 lines
Hmm, you mean we could get VMCS offsets in sadump itself?
But I think if we just export VMCS offsets in kernel, we could use the current existing dump tools with no or just very tiny change. I think this could be a more general mechanism than making changes in all kinds of dump tools.The sadump tool generates a core file with the OS image, right? Can it not attach the offsets to a note, just like you propose for kdump?
Both are right.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello Avi,
Quoted 105 lines · Yanfei Zhang
Quoted 60 lines · Avi Kivity
Quoted 53 lines · Yanfei Zhang
Quoted 42 lines · Avi Kivity
Quoted 31 lines · Yanfei Zhang
Quoted 30 lines · Avi Kivity
Quoted 23 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Here are two usercases for two features that we want.
1) Create guest machine's crash dumpfile from host machine's crash dumpfile
In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.
As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.
Quoted 3 lines
The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.
Do you mean, that a heartbeat failure in the guest lead to host panic?
Quoted 7 lines
Quoted 3 lines
My expectation is that a problem in the guest will cause the guest to panic and perhaps produce a dump; the host will remain up.
The point is that before our investigation, we didn't know which side leads to this buggy situation. Maybe a bug in host machine or the guest machine itself causes a heartbeat failure.
How can a guest bug cause a host panic?
Quoted 9 lines
Quoted 5 lines
So we want to get both host machine's crash dump and guest machine's crash dump *at the same time*. Then we could use userspace tools to get guest machine crash dump from host machine's and analyse them separately to find which side causes the problem.
If the guest caused the problem, there would be no panic; therefore there was a host bug.
Quoted 5 lines
Yes, a guest bug cannot cause a host panic. When heartbeat stops in guest machine, we could trigger the host dump mechanism to work. This is because we want to get the status of both host and guest machine at the same time when heartbeat stops in guest machine. Then we can look for bug reasons from both host machine's and guest machine's views.
Quoted 5 lines
That sounds like a bad idea. Can you explain in what situation it makes sense for a guest to stop the host (and all other guests running on it) rather than just restarting the failed services (on the host or other guests)?
We never do this on customer's environment which maybe a host with many guests running on it. We do this on another environment to reproduce the buggy situation; or we do this in testing phase on development environment towards production one on the customer's site.
Quoted 33 lines
Quoted 32 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.
So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.
Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.
2) Get offsets of VMCS information on the CPU running on the host machine
If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Shouldn't sadump then expose the VMCS offsets? Perhaps bundling them into its dump file?
Firmware-based crash dump doesn't concern the os running on the machine. So it will not do any os handling when machine crashes.
Seems to me the VMCS offsets are OS independent.
Hmm, you mean we could get VMCS offsets in sadump itself?
But I think if we just export VMCS offsets in kernel, we could use the current existing dump tools with no or just very tiny change. I think this could be a more general mechanism than making changes in all kinds of dump tools.
Quoted 5 lines
Quoted 3 lines
The sadump tool generates a core file with the OS image, right? Can it not attach the offsets to a note, just like you propose for kdump?
Both are right.
Dou you have any comments about this patch set?
Thanks
Zhang Yanfei
Quoted 2 lines · Yanfei Zhang
Dou you have any comments about this patch set?
I still have a hard time understanding why it is needed. If the host crashes, there is no reason to look at guest state; the host should survive no matter what the guest does.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 2 lines · Yanfei Zhang
Dou you have any comments about this patch set?
I still have a hard time understanding why it is needed. If the host crashes, there is no reason to look at guest state; the host should survive no matter what the guest does.
OK. Let me summarize it.
1. Why is this patch needed? (Our requirement)
We once came to a buggy situation: a host scheduler bug caused guest machine's vcpu stopped for a long time and then led to heartbeat stop (host is still running).
we want to have an efficient way to make the bug analysis when we come to the similar situation where guest machine doesn't work well due to something of host machine's,
Because we should debug both host machine's and guest machine's sides to look for the reasons, so we want to get both host machine's crash dump and guest machine's crash dump at the same time when the buggy situation remains.
2. What will we do?
If this bug was found on customer's environment, we have two ways to avoid affecting other guest machines running on the same host. First, we could do bug analysis on another environment to reproduce the buggy situation; Second, we could migrate other guest machines to other hosts.
After the buggy situation is reproduced, we panic the host *manually*. Then we could use userland tools to get guest machine's crash dump from host machine's with the feature provided by this patch set. Finally we could analyse them separately to find which side causes the problem.
Thanks
Zhang Yanfei
Hello Avi,
Quoted 32 lines · Yanfei Zhang
Quoted 9 lines · Avi Kivity
Quoted 2 lines · Yanfei Zhang
Dou you have any comments about this patch set?
I still have a hard time understanding why it is needed. If the host crashes, there is no reason to look at guest state; the host should survive no matter what the guest does.
OK. Let me summarize it.
1. Why is this patch needed? (Our requirement)
We once came to a buggy situation: a host scheduler bug caused guest machine's vcpu stopped for a long time and then led to heartbeat stop (host is still running).
we want to have an efficient way to make the bug analysis when we come to the similar situation where guest machine doesn't work well due to something of host machine's,
Because we should debug both host machine's and guest machine's sides to look for the reasons, so we want to get both host machine's crash dump and guest machine's crash dump at the same time when the buggy situation remains.
2. What will we do?
If this bug was found on customer's environment, we have two ways to avoid affecting other guest machines running on the same host. First, we could do bug analysis on another environment to reproduce the buggy situation; Second, we could migrate other guest machines to other hosts.
Quoted 5 lines
After the buggy situation is reproduced, we panic the host *manually*. Then we could use userland tools to get guest machine's crash dump from host machine's with the feature provided by this patch set. Finally we could analyse them separately to find which side causes the problem.
Could you please tell me your attitude towards this patch?
And here is a new case from the LinuxCon Japan:
Developers from Hitach are now developing a new livedump mechanism for the same reason as ours. They have come to the situation *many times* that guest machines crashed due to host's failures, in particular, under development.
So they develop this mechanism to get crash dump while retaining the buggy situation between host and guest machine. The difference between theirs and ours is whether or not to use the feature on _customer's running machine_.
Thanks
Zhang Yanfei
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 1 line · Yanfei Zhang
Hello Avi,
Sorry about the delay...
Quoted 20 lines
Quoted 18 lines · Yanfei Zhang
Quoted 9 lines · Avi Kivity
Quoted 2 lines · Yanfei Zhang
Dou you have any comments about this patch set?
I still have a hard time understanding why it is needed. If the host crashes, there is no reason to look at guest state; the host should survive no matter what the guest does.
OK. Let me summarize it.
1. Why is this patch needed? (Our requirement)
We once came to a buggy situation: a host scheduler bug caused guest machine's vcpu stopped for a long time and then led to heartbeat stop (host is still running).
Quoted 6 lines
Quoted 6 lines
we want to have an efficient way to make the bug analysis when we come to the similar situation where guest machine doesn't work well due to something of host machine's,
Because we should debug both host machine's and guest machine's sides to look for the reasons, so we want to get both host machine's crash dump and guest machine's crash dump at the same time when the buggy situation remains.
I would argue that there are two separate bugs here: (1) a host bug which caused the scheduling delay (2) putting a heartbeat service on a virtualized guests with no real time guarantees.
But I understand your situation.
Quoted 7 lines
Quoted 7 lines
2. What will we do?
If this bug was found on customer's environment, we have two ways to avoid affecting other guest machines running on the same host. First, we could do bug analysis on another environment to reproduce the buggy situation; Second, we could migrate other guest machines to other hosts.
You could also use tracing (there's the latency tracer and the scheduler tracepoints) to debug this on a live system.
Quoted 8 lines
Quoted 6 lines
After the buggy situation is reproduced, we panic the host *manually*. Then we could use userland tools to get guest machine's crash dump from host machine's with the feature provided by this patch set. Finally we could analyse them separately to find which side causes the problem.
Could you please tell me your attitude towards this patch?
I still dislike it conceptually. But let me do a technical review of the latest version.
Quoted 5 lines
And here is a new case from the LinuxCon Japan:
Developers from Hitach are now developing a new livedump mechanism for the same reason as ours. They have come to the situation *many times* that guest machines crashed due to host's failures, in particular, under development.
This has happened to me as well, possible even more times :). I don't use crash dumps for debugging but different people may use different techniques.
Quoted 3 lines
So they develop this mechanism to get crash dump while retaining the buggy situation between host and guest machine. The difference between theirs and ours is whether or not to use the feature on _customer's running
machine_.
Signature · 8 lines
error compiling committee.c: too many arguments to function _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Hi, Avi, Yanfei
I'm YOSHIDA Masanori from Hitachi, a developer of livedump.
Quoted 9 lines
Quoted 5 lines
And here is a new case from the LinuxCon Japan:
Developers from Hitach are now developing a new livedump mechanism for the same reason as ours. They have come to the situation *many times* that guest machines crashed due to host's failures, in particular, under development.
This has happened to me as well, possible even more times . I don't use crash dumps for debugging but different people may use different techniques.
As Yanfei's introduction, I'm developing livedump for cases where guests crash due to host's failures.
Especially in very important systems, it is strongly requested to identify the root cause of any failure even if it is very rare. For this purpose, crash dumps must be obtained. Therefore, we think livedump technique must be applied to the virtualization system on that kind of area.
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
After the buggy situation is reproduced, we panic the host *manually*. Then we could use userland tools to get guest machine's crash dump from host machine's with the feature provided by this patch set. Finally we could analyse them separately to find which side causes the problem.
Could you please tell me your attitude towards this patch?
I still dislike it conceptually. But let me do a technical review of the latest version.
Actually, current implementation of livedump is just a core part, which dumps only the image of kernel space. And I'd like to expand it to obtain guest image at the same time too.
Also for this situation, VMCSINFO seems necessary to be exported.
Thanks,
YOSHIDA Masanori
Yokohama Research Laboratory,
Hitachi, Ltd.
Quoted 8 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Perhaps I'm wrong, but this solution seems much, much more dynamic than it needs to be.
The VMCS offsets aren't going to change between different boots on the same CPU, unless perhaps the microcode has been updated.
So you can have the VMCS offset dumping be a manually-loaded module. Build a database mapping from (CPUID, microcode revision) -> (VMCSINFO). There's no need for anything beyond the (CPUID, microcode revision) to be put in the kdump, since your offline processing of a kdump can then look up the rest.
It means you don't have to interact with the vmx module at all, and no extra modules or code have to be loaded on the millions of Linux machines that won't need the functionality.
Quoted 23 lines · Eric Northup
Quoted 8 lines · zhangyanfei
This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs to kexec-tools just like VMCOREINFO.
Perhaps I'm wrong, but this solution seems much, much more dynamic than it needs to be.
The VMCS offsets aren't going to change between different boots on the same CPU, unless perhaps the microcode has been updated.
So you can have the VMCS offset dumping be a manually-loaded module. Build a database mapping from (CPUID, microcode revision) -> (VMCSINFO). There's no need for anything beyond the (CPUID, microcode revision) to be put in the kdump, since your offline processing of a kdump can then look up the rest.
Quoted 4 lines
It means you don't have to interact with the vmx module at all, and no extra modules or code have to be loaded on the millions of Linux machines that won't need the functionality.
We have considered this way, but there are two issues:
1) vmx resource is unique for a single cpu, and it's risky to grab it forcibly on the environment where kvm module is used, in particular on customer's environment. To do this safely, kvm support is needed.
2) It highly costs to prepare each cpu to each customer environment to collect vmcsinfo. After all, there are various environments on our customer's.
Our patch provides a module, so those who doesn't want this feature can just stop it being auto-loaded when system starts up.
Thanks
Zhang Yanfei
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 1 line · Yanfei Zhang
于 2012年05月22日 02:58, Eric Northup 写道:
[...]
Quoted 5 lines
Quoted 5 lines
So you can have the VMCS offset dumping be a manually-loaded module. Build a database mapping from (CPUID, microcode revision) -> (VMCSINFO). There's no need for anything beyond the (CPUID, microcode revision) to be put in the kdump, since your offline processing of a kdump can then look up the rest.
[...]
Quoted 5 lines
We have considered this way, but there are two issues:
1) vmx resource is unique for a single cpu, and it's risky to grab it forcibly on the environment where kvm module is used, in particular on customer's environment. To do this safely, kvm support is needed.
It's not risky: you just have to make sure that no one else is going to use the VMCS on your CPU while you're running. You can disable preemption and then save the old VMCS pointer from the CPU (see the VMPTRST instructions). Load your temporary VMCS pointer, discover the fields, then restore the original VMCS pointer. Then re-enable preemption and you're done.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec