Export offsets of VMCS fields as note information for kdump

This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs (/sys/devices/system/cpu/vmcs/) to kexec-tools.

Here are two usercases for two features that we want.

1) Create guest machine's crash dumpfile from host machine's crash dumpfile

In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.

As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.

The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.

Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.

So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.

Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.

2) Get offsets of VMCS information on the CPU running on the host machine

If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.

TODO:
1. In kexec-tools, get VMCSINFO via sysfs and dump it as note information into vmcore.
2. Dump VMCS region of each guest vcpu and VMCSINFO into qemu-process core file. To do this, we will modify kernel core dumper, gdb gcore and crash gcore.
3. Dump guest image from the qemu-process core file into a vmcore.

Changelog from v3 to v4:
1. All the variables and functions are moved to vmcsinfo-intel module.
2. Add a new sysfs interface /sys/devices/system/cpu/vmcs_id to export vmcs revision identifier. And origial sysfs interface is changed from /sys/devices/cpu/vmcs to /sys/devices/system/cpu/vmcs. Thanks Greg KH for his helpful comments about sysfs.

Changelog from v2 to v3:
1. New VMCSINFO format.
Now the VMCSINFO is mainly made up of an array that contains all vmcs fields' offsets. The offsets aren't encoded because we decode them in the module itself. If some field doesn't exist or its offset cannot be decoded correctly, the offset in the array is just set to zero.
2. New sysfs interface and Documentation/ABI entry.
We expose the actual fields in /sys/devices/cpu/vmcs instead of just exporting the address of VMCSINFO in /sys/kernel/vmcsinfo.
For example, /sys/devices/cpu/vmcs/0800 contains the offset of GUEST_DS_SELECTOR. 0800 is the encoding of GUEST_DS_SELECTOR. Accordingly, ABI entry in Documentation is changed from sysfs-kernel-vmcsinfo to sysfs-devices-cpu-vmcs.

Changelog from v1 to v2:
1. The VMCSINFO now has a simple binary <field><encoded offset> format, as below:

     +-------------+--------------------------+
     | Byte offset | Contents                 |
     +-------------+--------------------------+
     | 0           | VMCS revision identifier |
     +-------------+--------------------------+
     | 4           | <field><encoded offset>  |
     +-------------+--------------------------+
     | 16          | <field><encoded offset>  |
     +-------------+--------------------------+
     ......

The first 32 bits of VMCSINFO contains the VMCS revision identifier. The remainder of VMCSINFO is used for <field><encoded offset> sets. Each set takes 12 bytes: field occupys 4 bytes and its corresponding encoded offset occupys 8 bytes.

Encoded offsets are raw values read by vmcs_read{16, 64, 32, l}, and they are all unsigned extended to 8 bytes for each <field><encoded offset> set will have the same size.
We do not decode offsets here. The decoding work is delayed in userspace tools for more flexible handling.

And here are two examples of the new VMCSINFO:
Processor: Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
VMCSINFO contains:

     <0000000d>                   --> VMCS revision id = 0xd
     <00004000><0000000001840180> --> OFFSET(PIN_BASED_VM_EXEC_CONTROL) = 0x01840180
     <00004002><0000000001940190> --> OFFSET(CPU_BASED_VM_EXEC_CONTROL) = 0x01940190
     <0000401e><000000000fe40fe0> --> OFFSET(SECONDARY_VM_EXEC_CONTROL) = 0x0fe40fe0
     <0000400c><0000000001e401e0> --> OFFSET(VM_EXIT_CONTROLS) = 0x01e401e0
     ......

Processor: Intel(R) Xeon(R) CPU E7540 @ 2.00GHz (24 cores)
VMCSINFO contains:

     <0000000e>                   --> VMCS revision id = 0xe 
     <00004000><0000000005540550> --> OFFSET(PIN_BASED_VM_EXEC_CONTROL) = 0x05540550
     <00004002><0000000005440540> --> OFFSET(CPU_BASED_VM_EXEC_CONTROL) = 0x05440540
     <0000401e><00000000054c0548> --> OFFSET(SECONDARY_VM_EXEC_CONTROL) = 0x054c0548
     <0000400c><00000000057c0578> --> OFFSET(VM_EXIT_CONTROLS) = 0x057c0578
     ......

2. Add a new kernel module *vmcsinfo-intel* for filling VMCSINFO instead of putting it in module kvm-intel. The new module is auto-loaded when the vmx cpufeature is detected and it depends on module kvm-intel. *Loading and unloading this module will have no side effect on the running guests.*
3. The sysfs file vmcsinfo is splitted into 2 files:
/sys/kernel/vmcsinfo: shows physical address of VMCSINFO note information. /sys/kernel/vmcsinfo_maxsize: shows max size of VMCSINFO.
4. A new Documentation/ABI entry is added for vmcsinfo and vmcsinfo_maxsize.
5. Do not update VMCSINFO note when the kernel is panicked.

zhangyanfei (3):
KVM: Export symbols for module vmcsinfo-intel
KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
Documentation: Add ABI entry for vmcs sysfs interface.

 Documentation/ABI/testing/sysfs-devices-system-cpu |   21 +
 arch/x86/include/asm/vmx.h                         |   73 +++
 arch/x86/kvm/Kconfig                               |   11 +
 arch/x86/kvm/Makefile                              |    3 +
 arch/x86/kvm/vmcsinfo.c                            |  586 ++++++++++++++++++++
 arch/x86/kvm/vmx.c                                 |   81 +---
 include/linux/kvm_host.h                           |    3 +
 virt/kvm/kvm_main.c                                |    8 +-
 8 files changed, 714 insertions(+), 72 deletions(-)
 create mode 100644 arch/x86/kvm/vmcsinfo.c

A new module named vmcsinfo-intel is used to fill VMCSINFO. And this module depends on kvm-intel and kvm module. So we should export some symbols of kvm-intel and kvm module that are needed by vmcsinfo-intel.

---
 arch/x86/include/asm/vmx.h |   73 +++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx.c         |   81 +++++++-------------------------------------
 include/linux/kvm_host.h   |    3 ++
 virt/kvm/kvm_main.c        |    8 ++--
 4 files changed, 93 insertions(+), 72 deletions(-)

index 31f180c..1044e2e 100644
@@ -26,6 +26,7 @@
  */
 
 #include <linux/types.h>
+#include <linux/kvm_host.h>
 
 /*
  * Definitions of Primary Processor-Based VM-Execution Controls.

@@ -481,4 +482,76 @@ enum vm_instruction_error_number {
 	VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID = 28,
 };
 
+#define __ex(x) __kvm_handle_fault_on_reboot(x)
+#define __ex_clear(x, reg) \
+	____kvm_handle_fault_on_reboot(x, "xor " reg " , " reg)
+
+struct vmcs {
+	u32 revision_id;
+	u32 abort;
+	char data[0];
+};
+
+struct vmcs_config {
+	int size;
+	int order;
+	u32 revision_id;
+	u32 pin_based_exec_ctrl;
+	u32 cpu_based_exec_ctrl;
+	u32 cpu_based_2nd_exec_ctrl;
+	u32 vmexit_ctrl;
+	u32 vmentry_ctrl;
+};
+
+extern struct vmcs_config vmcs_config;
+
+DECLARE_PER_CPU(struct vmcs *, current_vmcs);
+
+enum vmcs_field_type {
+	VMCS_FIELD_TYPE_U16 = 0,
+	VMCS_FIELD_TYPE_U64 = 1,
+	VMCS_FIELD_TYPE_U32 = 2,
+	VMCS_FIELD_TYPE_NATURAL_WIDTH = 3
+};
+
+static inline int vmcs_field_type(unsigned long field)
+{
+	if (0x1 & field)	/* the *_HIGH fields are all 32 bit */
+		return VMCS_FIELD_TYPE_U32;
+	return (field >> 13) & 0x3 ;
+}
+
+static __always_inline unsigned long vmcs_readl(unsigned long field)
+{
+	unsigned long value;
+
+	asm volatile (__ex_clear(ASM_VMX_VMREAD_RDX_RAX, "%0")
+		      : "=a"(value) : "d"(field) : "cc");
+	return value;
+}
+
+static __always_inline u16 vmcs_read16(unsigned long field)
+{
+	return vmcs_readl(field);
+}
+
+static __always_inline u32 vmcs_read32(unsigned long field)
+{
+	return vmcs_readl(field);
+}
+
+static __always_inline u64 vmcs_read64(unsigned long field)
+{
+#ifdef CONFIG_X86_64
+	return vmcs_readl(field);
+#else
+	return vmcs_readl(field) | ((u64)vmcs_readl(field+1) << 32);
+#endif
+}
+
+struct vmcs *alloc_vmcs(void);
+void vmcs_load(struct vmcs *);
+void vmcs_clear(struct vmcs *);
+void free_vmcs(struct vmcs *);
+
 #endif

index c446435..0930fd9 100644
@@ -95,6 +95,9 @@ enum kvm_bus {
 	KVM_NR_BUSES
 };
 
+int hardware_enable_all(void);
+void hardware_disable_all(void);
+
 int kvm_io_bus_write(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
 		     int len, const void *val);
 int kvm_io_bus_read(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr, int len,

index 7e14068..26fd04d 100644
@@ -90,8 +90,6 @@ static long kvm_vcpu_ioctl(struct file *file, unsigned int ioctl,
 static long kvm_vcpu_compat_ioctl(struct file *file, unsigned int ioctl,
 				  unsigned long arg);
 #endif
-static int hardware_enable_all(void);
-static void hardware_disable_all(void);
 
 static void kvm_io_bus_destroy(struct kvm_io_bus *bus);
 

@@ -2330,14 +2328,15 @@ static void hardware_disable_all_nolock(void)
 		on_each_cpu(hardware_disable_nolock, NULL, 1);
 }
 
-static void hardware_disable_all(void)
+void hardware_disable_all(void)
 {
 	raw_spin_lock(&kvm_lock);
 	hardware_disable_all_nolock();
 	raw_spin_unlock(&kvm_lock);
 }
+EXPORT_SYMBOL_GPL(hardware_disable_all);
 
-static int hardware_enable_all(void)
+int hardware_enable_all(void)
 {
 	int r = 0;
 

@@ -2358,6 +2357,7 @@ static int hardware_enable_all(void)
 
 	return r;
 }
+EXPORT_SYMBOL_GPL(hardware_enable_all);
 
 static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val,
 			   void *v)

1.7.1

This patch implements a new module named vmcsinfo-intel. The module fills VMCSINFO with the VMCS revision identifier, and offsets of VMCS fields.

Note, offsets of fields that defined in Intel specification (Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C) but not defined in *vmcs_field* will not be filled in VMCSINFO. And, some fields may be unsupported in some machines, in these machines, corresponding offsets will be zero.

Besides, this patch also exports vmcs revision identifier via /sys/devices/system/cpu/vmcs_id and offsets of fields via /sys/devices/system/cpu/vmcs/.
Individual offsets are contained in subfiles named by the filed's encoding, e.g.: /sys/devices/cpu/vmcs/0800

---
 arch/x86/kvm/Kconfig    |   11 +
 arch/x86/kvm/Makefile   |    3 +
 arch/x86/kvm/vmcsinfo.c |  586 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 600 insertions(+), 0 deletions(-)
 create mode 100644 arch/x86/kvm/vmcsinfo.c

index a28f338..1dd64b1 100644
@@ -63,6 +63,17 @@ config KVM_INTEL
 	  To compile this as a module, choose M here: the module
 	  will be called kvm-intel.
 
+config VMCSINFO_INTEL
+	tristate "Export VMCSINFO for Intel processors"
+	depends on KVM_INTEL
+	---help---
+	  Provides support for exporting VMCSINFO on Intel processors equipped
+	  with the VT extensions. The VMCSINFO contains a VMCS revision
+	  identifier and offsets of VMCS fields.
+
+	  To compile this as a module, choose M here: the module
+	  will be called vmcsinfo-intel.
+
 config KVM_AMD
 	tristate "KVM for AMD processors support"
 	depends on KVM

index 4f579e8..12a1ef6 100644
@@ -4,6 +4,7 @@ ccflags-y += -Ivirt/kvm -Iarch/x86/kvm
 CFLAGS_x86.o := -I.
 CFLAGS_svm.o := -I.
 CFLAGS_vmx.o := -I.
+CFLAGS_vmcsinfo.o := -I.
 
 kvm-y			+= $(addprefix ../../../virt/kvm/, kvm_main.o ioapic.o \
 				coalesced_mmio.o irq_comm.o eventfd.o \

@@ -15,7 +16,9 @@ kvm-y			+= x86.o mmu.o emulate.o i8259.o irq.o lapic.o \
 			   i8254.o timer.o cpuid.o pmu.o
 kvm-intel-y		+= vmx.o
 kvm-amd-y		+= svm.o
+vmcsinfo-intel-y	+= vmcsinfo.o
 
 obj-$(CONFIG_KVM)	+= kvm.o
 obj-$(CONFIG_KVM_INTEL)	+= kvm-intel.o
 obj-$(CONFIG_KVM_AMD)	+= kvm-amd.o
+obj-$(CONFIG_VMCSINFO_INTEL)	+= vmcsinfo-intel.o

1.7.1

---
 Documentation/ABI/testing/sysfs-devices-system-cpu |   21 ++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)

index 5dab364..6efbd6c 100644
@@ -9,6 +9,27 @@ Description:
 
 		/sys/devices/system/cpu/cpu#/
 
+What:		/sys/devices/system/cpu/vmcs_id
+Date:		June 2012
+KernelVersion:	3.5.0
+Contact:	Zhang Yanfei <[email protected]>
+Description:
+		Vmcs revision identifier in Intel cpu. The value enables
+		software to avoid using a VMCS region formatted for one
+		processor on a processor that uses a different format.
+
+What:		/sys/devices/system/cpu/vmcs/
+Date:		June 2012
+KernelVersion:	3.5.0
+Contact:	Zhang Yanfei <[email protected]>
+Description:
+		A collection of vmcs fields' offsets for Intel cpu.
+
+		Individual offsets are contained in subfiles named by
+		the filed's encoding, e.g.:
+
+		/sys/devices/system/cpu/vmcs/0800
+
 What:		/sys/devices/system/cpu/kernel_max
 		/sys/devices/system/cpu/offline
 		/sys/devices/system/cpu/online

1.7.1

From: Yanfei Zhang <[email protected]>
Subject: [PATCH v4 2/3] KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
Date: Wed, 4 Jul 2012 18:05:19 +0800

According to the discussion starting from

http://lkml.indiana.edu/hypermail/linux/kernel/1105.3/00749.html

system can be composed of CPUs with different steppings or differnet microcode revisions. Becase of the nature that it's hided in the specification, I suspect layout of vmcs could change across different steppings or microcode revisions. Then, the interface needs to be changed as per-cpu like

/sys/devices/cpu/cpu0/vmcs/0800
/sys/devices/cpu/cpu1/vmcs/0800
...
/sys/devices/cpu/cpuN/vmcs/0800

Also, processing of vmcsinfo initialization needs to be done per cpu, and can be triggered when cpu is added not when kvm module is loaded.

Thanks.
HATAYAMA, Daisuke

At 07/06/2012 04:04 PM, HATAYAMA Daisuke Wrote:

IIRC, kvm can not work in such environment. The vcpu can run on different cpu. If the cpu's vmcs is different, I don't know what will happen. So do we need to support for such environment now? I think that if kvm can not work in such environment, we should not provide vmcs information for each physical cpu.

Thanks
Wen Congyang

From: Wen Congyang <[email protected]>
Subject: Re: [PATCH v4 2/3] KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
Date: Fri, 6 Jul 2012 16:25:23 +0800

I think so too. The design can be kept very simple if kvm doesn't support such case, and it would be good news. Is it true?

Thanks.
HATAYAMA, Daisuke

From: Wen Congyang <[email protected]>
Subject: Re: [PATCH v4 2/3] KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
Date: Fri, 6 Jul 2012 16:25:23 +0800

Ah, I noticed my basic confusion: if it --- vcpu can run on cpus with differnet VMCS revision --- were possible, this vmcsinfo would be unnecessary because it would mean processer could read VMCS data with revision different from its own one or some kind of reverse engineering for convertion of differnet VMCS data were done.

I think kvm could probably work if only processors that have the same VMCS revision are assigned to a single guest. But considering the VMCS nature, such processor with differnet revision seems unlikely to be used on host machine.

Thanks.
HATAYAMA, Daisuke

From: Yanfei Zhang <[email protected]>
Subject: [PATCH v4 2/3] KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
Date: Wed, 4 Jul 2012 18:05:19 +0800

I think /sys/devices/system/cpu/vmcs/id is more natural, which also belongs to vmcs.

<cut>