Skip to contentSkip to conversation

Export offsets of VMCS fields as note information for kdump

PATCH Superseded Yanfei Zhang 3 participants 12 messages Jun 27, 2012 – Jul 3, 2012

Subsystems (6)
x86 KVM Driver core KVM x86 +2 more subsystems

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Export offsets of VMCS fields as note information for kdump

This patch set exports offsets of VMCS fields as note information for kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retrieve runtime state of guest machine image, such as registers, in host machine's crash dump as VMCS format. The problem is that VMCS internal is hidden by Intel in its specification. So, we slove this problem by reverse engineering implemented in this patch set. The VMCSINFO is exported via sysfs (/sys/devices/cpu/vmcs/) to kexec-tools.

Here are two usercases for two features that we want.

1) Create guest machine's crash dumpfile from host machine's crash dumpfile

In general, we want to use this feature on failure analysis for the system where the processing depends on the communication between host and guest machines to look into the system from both machines's viewpoints.

As a concrete situation, consider where there's heartbeat monitoring feature on the guest machine's side, where we need to determine in which machine side the cause of heartbeat stop lies. In our actual experiments, we encountered such situation and we found the cause of the bug was in host's process schedular so guest machine's vcpu stopped for a long time and then led to heartbeat stop.

The module that judges heartbeat stop is on guest machine, so we need to debug guest machine's data. But if the cause lies in host machine side, we need to look into host machine's crash dump.

Without this feature, we first create guest machine's dump and then create host mahine's, but there's only a short time between two processings, during which it's unlikely that buggy situation remains.

So, we think the feature is useful to debug both guest machine's and host machine's sides at the same time, and expect we can make failure analysis efficiently.

Of course, we believe this feature is commonly useful on the situation where guest machine doesn't work well due to something of host machine's.

2) Get offsets of VMCS information on the CPU running on the host machine

If kdump doesn't work well, then it means we cannot use kvm API to get register values of guest machine and they are still left on its vmcs region. In the case, we use crash dump mechanism running outside of linux kernel, such as sadump, a firmware-based crash dump. Then VMCS information is then necessary.

TODO:
1. In kexec-tools, get VMCSINFO via sysfs and dump it as note information into vmcore.
2. Dump VMCS region of each guest vcpu and VMCSINFO into qemu-process core file. To do this, we will modify kernel core dumper, gdb gcore and crash gcore.
3. Dump guest image from the qemu-process core file into a vmcore.

Changelog from v2 to v3:
1. New VMCSINFO format.
Now the VMCSINFO is mainly made up of an array that contains all vmcs fields' offsets. The offsets aren't encoded because we decode them in the module itself. If some field doesn't exist or its offset cannot be decoded correctly, the offset in the array is just set to zero.
2. New sysfs interface and Documentation/ABI entry.
We expose the actual fields in /sys/devices/cpu/vmcs instead of just exporting the address of VMCSINFO in /sys/kernel/vmcsinfo.
For example, /sys/devices/cpu/vmcs/0800 contains the offset of GUEST_DS_SELECTOR. 0800 is the encoding of GUEST_DS_SELECTOR. Accordingly, ABI entry in Documentation is changed from sysfs-kernel-vmcsinfo to sysfs-devices-cpu-vmcs.

Changelog from v1 to v2:
1. The VMCSINFO now has a simple binary <field><encoded offset> format, as below:

     +-------------+--------------------------+
     | Byte offset | Contents                 |
     +-------------+--------------------------+
     | 0           | VMCS revision identifier |
     +-------------+--------------------------+
     | 4           | <field><encoded offset>  |
     +-------------+--------------------------+
     | 16          | <field><encoded offset>  |
     +-------------+--------------------------+
     ......

The first 32 bits of VMCSINFO contains the VMCS revision identifier. The remainder of VMCSINFO is used for <field><encoded offset> sets. Each set takes 12 bytes: field occupys 4 bytes and its corresponding encoded offset occupys 8 bytes.

Encoded offsets are raw values read by vmcs_read{16, 64, 32, l}, and they are all unsigned extended to 8 bytes for each <field><encoded offset> set will have the same size.
We do not decode offsets here. The decoding work is delayed in userspace tools for more flexible handling.

And here are two examples of the new VMCSINFO:
Processor: Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
VMCSINFO contains:

     <0000000d>                   --> VMCS revision id = 0xd
     <00004000><0000000001840180> --> OFFSET(PIN_BASED_VM_EXEC_CONTROL) = 0x01840180
     <00004002><0000000001940190> --> OFFSET(CPU_BASED_VM_EXEC_CONTROL) = 0x01940190
     <0000401e><000000000fe40fe0> --> OFFSET(SECONDARY_VM_EXEC_CONTROL) = 0x0fe40fe0
     <0000400c><0000000001e401e0> --> OFFSET(VM_EXIT_CONTROLS) = 0x01e401e0
     ......

Processor: Intel(R) Xeon(R) CPU E7540 @ 2.00GHz (24 cores)
VMCSINFO contains:

     <0000000e>                   --> VMCS revision id = 0xe 
     <00004000><0000000005540550> --> OFFSET(PIN_BASED_VM_EXEC_CONTROL) = 0x05540550
     <00004002><0000000005440540> --> OFFSET(CPU_BASED_VM_EXEC_CONTROL) = 0x05440540
     <0000401e><00000000054c0548> --> OFFSET(SECONDARY_VM_EXEC_CONTROL) = 0x054c0548
     <0000400c><00000000057c0578> --> OFFSET(VM_EXIT_CONTROLS) = 0x057c0578
     ......

2. Add a new kernel module *vmcsinfo-intel* for filling VMCSINFO instead of putting it in module kvm-intel. The new module is auto-loaded when the vmx cpufeature is detected and it depends on module kvm-intel. *Loading and unloading this module will have no side effect on the running guests.*
3. The sysfs file vmcsinfo is splitted into 2 files:
/sys/kernel/vmcsinfo: shows physical address of VMCSINFO note information. /sys/kernel/vmcsinfo_maxsize: shows max size of VMCSINFO.
4. A new Documentation/ABI entry is added for vmcsinfo and vmcsinfo_maxsize.
5. Do not update VMCSINFO note when the kernel is panicked.

zhangyanfei (5):
x86: Add helper variables and functions to hold VMCSINFO
KVM: Export symbols for module vmcsinfo-intel
KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO
Sysfs: Export VMCSINFO via sysfs
Documentation: Add ABI entry for vmcs sysfs interface.

 Documentation/ABI/testing/sysfs-devices-cpu-vmcs |   11 +
 arch/x86/include/asm/vmcsinfo.h                  |  219 +++++++++++++
 arch/x86/include/asm/vmx.h                       |  231 +++++---------
 arch/x86/kernel/Makefile                         |    1 +
 arch/x86/kernel/vmcsinfo.c                       |  381 ++++++++++++++++++++++
 arch/x86/kvm/Kconfig                             |   11 +
 arch/x86/kvm/Makefile                            |    3 +
 arch/x86/kvm/vmcsinfo.c                          |  198 +++++++++++
 arch/x86/kvm/vmx.c                               |   81 +----
 drivers/base/core.c                              |   13 +
 include/linux/kvm_host.h                         |    3 +
 virt/kvm/kvm_main.c                              |    8 +-
 12 files changed, 932 insertions(+), 228 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-cpu-vmcs
 create mode 100644 arch/x86/include/asm/vmcsinfo.h
 create mode 100644 arch/x86/kernel/vmcsinfo.c
 create mode 100644 arch/x86/kvm/vmcsinfo.c
#1 lore raw

PATCH v3 1/5 x86: Add helper variables and functions to hold VMCSINFO

This patch provides a set of variables to hold the VMCSINFO and also some helper functions to help fill the VMCSINFO.

---
 arch/x86/include/asm/vmcsinfo.h |  219 ++++++++++++++++++++++
 arch/x86/include/asm/vmx.h      |  158 +----------------
 arch/x86/kernel/Makefile        |    1 +
 arch/x86/kernel/vmcsinfo.c      |  381 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 603 insertions(+), 156 deletions(-)
 create mode 100644 arch/x86/include/asm/vmcsinfo.h
 create mode 100644 arch/x86/kernel/vmcsinfo.c
arch/x86/include/asm/vmcsinfo.h new file +219 -0
new file mode 100644
index 0000000..4b9f56b
@@ -0,0 +1,219 @@
+#ifndef _ASM_X86_VMCSINFO_H
+#define _ASM_X86_VMCSINFO_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+#include <linux/elf.h>
+#include <linux/device.h>
+
+/* VMCS Encodings */
+enum vmcs_field {
+	VIRTUAL_PROCESSOR_ID            = 0x00000000,
+	GUEST_ES_SELECTOR               = 0x00000800,
+	GUEST_CS_SELECTOR               = 0x00000802,
+	GUEST_SS_SELECTOR               = 0x00000804,
+	GUEST_DS_SELECTOR               = 0x00000806,
+	GUEST_FS_SELECTOR               = 0x00000808,
+	GUEST_GS_SELECTOR               = 0x0000080a,
+	GUEST_LDTR_SELECTOR             = 0x0000080c,
+	GUEST_TR_SELECTOR               = 0x0000080e,
+	HOST_ES_SELECTOR                = 0x00000c00,
+	HOST_CS_SELECTOR                = 0x00000c02,
+	HOST_SS_SELECTOR                = 0x00000c04,
+	HOST_DS_SELECTOR                = 0x00000c06,
+	HOST_FS_SELECTOR                = 0x00000c08,
+	HOST_GS_SELECTOR                = 0x00000c0a,
+	HOST_TR_SELECTOR                = 0x00000c0c,
+	IO_BITMAP_A                     = 0x00002000,
+	IO_BITMAP_A_HIGH                = 0x00002001,
+	IO_BITMAP_B                     = 0x00002002,
+	IO_BITMAP_B_HIGH                = 0x00002003,
+	MSR_BITMAP                      = 0x00002004,
+	MSR_BITMAP_HIGH                 = 0x00002005,
+	VM_EXIT_MSR_STORE_ADDR          = 0x00002006,
+	VM_EXIT_MSR_STORE_ADDR_HIGH     = 0x00002007,
+	VM_EXIT_MSR_LOAD_ADDR           = 0x00002008,
+	VM_EXIT_MSR_LOAD_ADDR_HIGH      = 0x00002009,
+	VM_ENTRY_MSR_LOAD_ADDR          = 0x0000200a,
+	VM_ENTRY_MSR_LOAD_ADDR_HIGH     = 0x0000200b,
+	TSC_OFFSET                      = 0x00002010,
+	TSC_OFFSET_HIGH                 = 0x00002011,
+	VIRTUAL_APIC_PAGE_ADDR          = 0x00002012,
+	VIRTUAL_APIC_PAGE_ADDR_HIGH     = 0x00002013,
+	APIC_ACCESS_ADDR		= 0x00002014,
+	APIC_ACCESS_ADDR_HIGH		= 0x00002015,
+	EPT_POINTER                     = 0x0000201a,
+	EPT_POINTER_HIGH                = 0x0000201b,
+	GUEST_PHYSICAL_ADDRESS          = 0x00002400,
+	GUEST_PHYSICAL_ADDRESS_HIGH     = 0x00002401,
+	VMCS_LINK_POINTER               = 0x00002800,
+	VMCS_LINK_POINTER_HIGH          = 0x00002801,
+	GUEST_IA32_DEBUGCTL             = 0x00002802,
+	GUEST_IA32_DEBUGCTL_HIGH        = 0x00002803,
+	GUEST_IA32_PAT			= 0x00002804,
+	GUEST_IA32_PAT_HIGH		= 0x00002805,
+	GUEST_IA32_EFER			= 0x00002806,
+	GUEST_IA32_EFER_HIGH		= 0x00002807,
+	GUEST_IA32_PERF_GLOBAL_CTRL	= 0x00002808,
+	GUEST_IA32_PERF_GLOBAL_CTRL_HIGH= 0x00002809,
+	GUEST_PDPTR0                    = 0x0000280a,
+	GUEST_PDPTR0_HIGH               = 0x0000280b,
+	GUEST_PDPTR1                    = 0x0000280c,
+	GUEST_PDPTR1_HIGH               = 0x0000280d,
+	GUEST_PDPTR2                    = 0x0000280e,
+	GUEST_PDPTR2_HIGH               = 0x0000280f,
+	GUEST_PDPTR3                    = 0x00002810,
+	GUEST_PDPTR3_HIGH               = 0x00002811,
+	HOST_IA32_PAT			= 0x00002c00,
+	HOST_IA32_PAT_HIGH		= 0x00002c01,
+	HOST_IA32_EFER			= 0x00002c02,
+	HOST_IA32_EFER_HIGH		= 0x00002c03,
+	HOST_IA32_PERF_GLOBAL_CTRL	= 0x00002c04,
+	HOST_IA32_PERF_GLOBAL_CTRL_HIGH	= 0x00002c05,
+	PIN_BASED_VM_EXEC_CONTROL       = 0x00004000,
+	CPU_BASED_VM_EXEC_CONTROL       = 0x00004002,
+	EXCEPTION_BITMAP                = 0x00004004,
+	PAGE_FAULT_ERROR_CODE_MASK      = 0x00004006,
+	PAGE_FAULT_ERROR_CODE_MATCH     = 0x00004008,
+	CR3_TARGET_COUNT                = 0x0000400a,
+	VM_EXIT_CONTROLS                = 0x0000400c,
+	VM_EXIT_MSR_STORE_COUNT         = 0x0000400e,
+	VM_EXIT_MSR_LOAD_COUNT          = 0x00004010,
+	VM_ENTRY_CONTROLS               = 0x00004012,
+	VM_ENTRY_MSR_LOAD_COUNT         = 0x00004014,
+	VM_ENTRY_INTR_INFO_FIELD        = 0x00004016,
+	VM_ENTRY_EXCEPTION_ERROR_CODE   = 0x00004018,
+	VM_ENTRY_INSTRUCTION_LEN        = 0x0000401a,
+	TPR_THRESHOLD                   = 0x0000401c,
+	SECONDARY_VM_EXEC_CONTROL       = 0x0000401e,
+	PLE_GAP                         = 0x00004020,
+	PLE_WINDOW                      = 0x00004022,
+	VM_INSTRUCTION_ERROR            = 0x00004400,
+	VM_EXIT_REASON                  = 0x00004402,
+	VM_EXIT_INTR_INFO               = 0x00004404,
+	VM_EXIT_INTR_ERROR_CODE         = 0x00004406,
+	IDT_VECTORING_INFO_FIELD        = 0x00004408,
+	IDT_VECTORING_ERROR_CODE        = 0x0000440a,
+	VM_EXIT_INSTRUCTION_LEN         = 0x0000440c,
+	VMX_INSTRUCTION_INFO            = 0x0000440e,
+	GUEST_ES_LIMIT                  = 0x00004800,
+	GUEST_CS_LIMIT                  = 0x00004802,
+	GUEST_SS_LIMIT                  = 0x00004804,
+	GUEST_DS_LIMIT                  = 0x00004806,
+	GUEST_FS_LIMIT                  = 0x00004808,
+	GUEST_GS_LIMIT                  = 0x0000480a,
+	GUEST_LDTR_LIMIT                = 0x0000480c,
+	GUEST_TR_LIMIT                  = 0x0000480e,
+	GUEST_GDTR_LIMIT                = 0x00004810,
+	GUEST_IDTR_LIMIT                = 0x00004812,
+	GUEST_ES_AR_BYTES               = 0x00004814,
+	GUEST_CS_AR_BYTES               = 0x00004816,
+	GUEST_SS_AR_BYTES               = 0x00004818,
+	GUEST_DS_AR_BYTES               = 0x0000481a,
+	GUEST_FS_AR_BYTES               = 0x0000481c,
+	GUEST_GS_AR_BYTES               = 0x0000481e,
+	GUEST_LDTR_AR_BYTES             = 0x00004820,
+	GUEST_TR_AR_BYTES               = 0x00004822,
+	GUEST_INTERRUPTIBILITY_INFO     = 0x00004824,
+	GUEST_ACTIVITY_STATE            = 0X00004826,
+	GUEST_SYSENTER_CS               = 0x0000482A,
+	HOST_IA32_SYSENTER_CS           = 0x00004c00,
+	CR0_GUEST_HOST_MASK             = 0x00006000,
+	CR4_GUEST_HOST_MASK             = 0x00006002,
+	CR0_READ_SHADOW                 = 0x00006004,
+	CR4_READ_SHADOW                 = 0x00006006,
+	CR3_TARGET_VALUE0               = 0x00006008,
+	CR3_TARGET_VALUE1               = 0x0000600a,
+	CR3_TARGET_VALUE2               = 0x0000600c,
+	CR3_TARGET_VALUE3               = 0x0000600e,
+	EXIT_QUALIFICATION              = 0x00006400,
+	GUEST_LINEAR_ADDRESS            = 0x0000640a,
+	GUEST_CR0                       = 0x00006800,
+	GUEST_CR3                       = 0x00006802,
+	GUEST_CR4                       = 0x00006804,
+	GUEST_ES_BASE                   = 0x00006806,
+	GUEST_CS_BASE                   = 0x00006808,
+	GUEST_SS_BASE                   = 0x0000680a,
+	GUEST_DS_BASE                   = 0x0000680c,
+	GUEST_FS_BASE                   = 0x0000680e,
+	GUEST_GS_BASE                   = 0x00006810,
+	GUEST_LDTR_BASE                 = 0x00006812,
+	GUEST_TR_BASE                   = 0x00006814,
+	GUEST_GDTR_BASE                 = 0x00006816,
+	GUEST_IDTR_BASE                 = 0x00006818,
+	GUEST_DR7                       = 0x0000681a,
+	GUEST_RSP                       = 0x0000681c,
+	GUEST_RIP                       = 0x0000681e,
+	GUEST_RFLAGS                    = 0x00006820,
+	GUEST_PENDING_DBG_EXCEPTIONS    = 0x00006822,
+	GUEST_SYSENTER_ESP              = 0x00006824,
+	GUEST_SYSENTER_EIP              = 0x00006826,
+	HOST_CR0                        = 0x00006c00,
+	HOST_CR3                        = 0x00006c02,
+	HOST_CR4                        = 0x00006c04,
+	HOST_FS_BASE                    = 0x00006c06,
+	HOST_GS_BASE                    = 0x00006c08,
+	HOST_TR_BASE                    = 0x00006c0a,
+	HOST_GDTR_BASE                  = 0x00006c0c,
+	HOST_IDTR_BASE                  = 0x00006c0e,
+	HOST_IA32_SYSENTER_ESP          = 0x00006c10,
+	HOST_IA32_SYSENTER_EIP          = 0x00006c12,
+	HOST_RSP                        = 0x00006c14,
+	HOST_RIP                        = 0x00006c16,
+};
+
+/*
+ * vmcs field offsets.
+ */
+struct vmcsinfo {
+	u32 vmcs_revision_id;
+	int filled;
+	u16 vmcs_field_to_offset_table[HOST_RIP + 1];
+};
+
+#define VMCSINFO_NOTE_NAME         "VMCSINFO"
+#define VMCSINFO_NOTE_NAME_BYTES   ALIGN(sizeof(VMCSINFO_NOTE_NAME), 4)
+#define VMCSINFO_NOTE_HEAD_BYTES   ALIGN(sizeof(struct elf_note), 4)
+#define VMCSINFO_NOTE_SIZE         (VMCSINFO_NOTE_HEAD_BYTES*2 \
+				    + sizeof(struct vmcsinfo) \
+				    + VMCSINFO_NOTE_NAME_BYTES)
+
+extern struct vmcsinfo vmcsinfo;
+#define VMCSINFO_MAX_FIELD \
+	ARRAY_SIZE(vmcsinfo.vmcs_field_to_offset_table)
+
+extern void update_vmcsinfo_note(void);
+extern int vmcs_sysfs_add(struct device *);
+extern void vmcs_sysfs_remove(struct device *);
+
+static inline void vmcsinfo_revision_id(u32 id)
+{
+	vmcsinfo.vmcs_revision_id = id;
+}
+
+static inline void vmcsinfo_field(unsigned long field, u16 offset)
+{
+	if (field < VMCSINFO_MAX_FIELD)
+		vmcsinfo.vmcs_field_to_offset_table[field] = offset;
+}
+
+static inline int vmcsinfo_is_filled(void)
+{
+	return vmcsinfo.filled;
+}
+
+static inline void vmcsinfo_filled(void)
+{
+	vmcsinfo.filled = 1;
+}
+
+static inline short get_vmcs_field_offset(unsigned long field)
+{
+	if (field >= VMCSINFO_MAX_FIELD ||
+	    vmcsinfo.vmcs_field_to_offset_table[field] == 0)
+		return -1;
+	return vmcsinfo.vmcs_field_to_offset_table[field];
+}
+
+#endif /* __ASSEMBLY__ */
+#endif /* _ASM_X86_VMCSINFO_H */
arch/x86/include/asm/vmx.h +2 -156

170 lines not on this page · raw message

arch/x86/kernel/Makefile +1 -0
index 8215e56..2c41f93 100644
@@ -99,6 +99,7 @@ obj-$(CONFIG_X86_CHECK_BIOS_CORRUPTION) += check.o
 obj-$(CONFIG_SWIOTLB)			+= pci-swiotlb.o
 obj-$(CONFIG_OF)			+= devicetree.o
 obj-$(CONFIG_UPROBES)			+= uprobes.o
+obj-y					+= vmcsinfo.o
 
 ###
 # 64 bit specific files
arch/x86/kernel/vmcsinfo.c new file +381 -0

381 lines not on this page · raw message

Signature · 1 line
1.7.1
#2 lore raw

Re: [PATCH v3 1/5] x86: Add helper variables and functions to hold VMCSINFO

From: Yanfei Zhang <[email protected]>
Subject: [PATCH v3 1/5] x86: Add helper variables and functions to hold VMCSINFO
Date: Wed, 27 Jun 2012 16:51:58 +0800

Quoted 30 lines

This patch provides a set of variables to hold the VMCSINFO and also some helper functions to help fill the VMCSINFO.

---
 arch/x86/include/asm/vmcsinfo.h |  219 ++++++++++++++++++++++
 arch/x86/include/asm/vmx.h      |  158 +----------------
 arch/x86/kernel/Makefile        |    1 +
 arch/x86/kernel/vmcsinfo.c      |  381 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 603 insertions(+), 156 deletions(-)
 create mode 100644 arch/x86/include/asm/vmcsinfo.h
 create mode 100644 arch/x86/kernel/vmcsinfo.c
arch/x86/include/asm/vmcsinfo.h new file +11 -0
new file mode 100644
index 0000000..4b9f56b
@@ -0,0 +1,219 @@
+#ifndef _ASM_X86_VMCSINFO_H
+#define _ASM_X86_VMCSINFO_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+#include <linux/elf.h>
+#include <linux/device.h>
+
+/* VMCS Encodings */
+enum vmcs_field {
+	VIRTUAL_PROCESSOR_ID            = 0x00000000,

<cut>

arch/x86/include/asm/vmcsinfo.h line 171

Quoted 10 lines
arch/x86/include/asm/vmcsinfo.h new file +10 -0
+	HOST_RIP                        = 0x00006c16,
+};
+
+/*
+ * vmcs field offsets.
+ */
+struct vmcsinfo {
+	u32 vmcs_revision_id;
+	int filled;
+	u16 vmcs_field_to_offset_table[HOST_RIP + 1];

HOST_RIP is so large that this array becomes large. Also there are unused elements in this array because field encoding is not indexed constantly.

Instead, how about defining the numbr of vmcs fields, 152?, as a specific constant, indexing each fields using integers and newly preparing index_to_field_table[]?

Thanks.
HATAYAMA, Daisuke

#3 lore raw

PATCH v3 2/5 KVM: Export symbols for module vmcsinfo-intel

In reply to Yanfei Zhang

A new module named vmcsinfo-intel is used to fill VMCSINFO. And this module depends on kvm-intel and kvm module. So we should export some symbols of kvm-intel and kvm module that are needed by vmcsinfo-intel.

---
 arch/x86/include/asm/vmx.h |   73 +++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx.c         |   81 +++++++-------------------------------------
 include/linux/kvm_host.h   |    3 ++
 virt/kvm/kvm_main.c        |    8 ++--
 4 files changed, 93 insertions(+), 72 deletions(-)
arch/x86/include/asm/vmx.h +73 -0
index c364219..03b7ae3 100644
@@ -26,6 +26,7 @@
  */
 
 #include <linux/types.h>
+#include <linux/kvm_host.h>
 
 #include <asm/vmcsinfo.h>
 
@@ -327,4 +328,76 @@ enum vm_instruction_error_number {
 	VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID = 28,
 };
 
+#define __ex(x) __kvm_handle_fault_on_reboot(x)
+#define __ex_clear(x, reg) \
+	____kvm_handle_fault_on_reboot(x, "xor " reg " , " reg)
+
+struct vmcs {
+	u32 revision_id;
+	u32 abort;
+	char data[0];
+};
+
+struct vmcs_config {
+	int size;
+	int order;
+	u32 revision_id;
+	u32 pin_based_exec_ctrl;
+	u32 cpu_based_exec_ctrl;
+	u32 cpu_based_2nd_exec_ctrl;
+	u32 vmexit_ctrl;
+	u32 vmentry_ctrl;
+};
+
+extern struct vmcs_config vmcs_config;
+
+DECLARE_PER_CPU(struct vmcs *, current_vmcs);
+
+enum vmcs_field_type {
+	VMCS_FIELD_TYPE_U16 = 0,
+	VMCS_FIELD_TYPE_U64 = 1,
+	VMCS_FIELD_TYPE_U32 = 2,
+	VMCS_FIELD_TYPE_NATURAL_WIDTH = 3
+};
+
+static inline int vmcs_field_type(unsigned long field)
+{
+	if (0x1 & field)	/* the *_HIGH fields are all 32 bit */
+		return VMCS_FIELD_TYPE_U32;
+	return (field >> 13) & 0x3 ;
+}
+
+static __always_inline unsigned long vmcs_readl(unsigned long field)
+{
+	unsigned long value;
+
+	asm volatile (__ex_clear(ASM_VMX_VMREAD_RDX_RAX, "%0")
+		      : "=a"(value) : "d"(field) : "cc");
+	return value;
+}
+
+static __always_inline u16 vmcs_read16(unsigned long field)
+{
+	return vmcs_readl(field);
+}
+
+static __always_inline u32 vmcs_read32(unsigned long field)
+{
+	return vmcs_readl(field);
+}
+
+static __always_inline u64 vmcs_read64(unsigned long field)
+{
+#ifdef CONFIG_X86_64
+	return vmcs_readl(field);
+#else
+	return vmcs_readl(field) | ((u64)vmcs_readl(field+1) << 32);
+#endif
+}
+
+struct vmcs *alloc_vmcs(void);
+void vmcs_load(struct vmcs *);
+void vmcs_clear(struct vmcs *);
+void free_vmcs(struct vmcs *);
+
 #endif
arch/x86/kvm/vmx.c +13 -68

160 lines not on this page · raw message

include/linux/kvm_host.h +3 -0
index c446435..0930fd9 100644
@@ -95,6 +95,9 @@ enum kvm_bus {
 	KVM_NR_BUSES
 };
 
+int hardware_enable_all(void);
+void hardware_disable_all(void);
+
 int kvm_io_bus_write(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
 		     int len, const void *val);
 int kvm_io_bus_read(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr, int len,
virt/kvm/kvm_main.c +4 -4
index 7e14068..26fd04d 100644
@@ -90,8 +90,6 @@ static long kvm_vcpu_ioctl(struct file *file, unsigned int ioctl,
 static long kvm_vcpu_compat_ioctl(struct file *file, unsigned int ioctl,
 				  unsigned long arg);
 #endif
-static int hardware_enable_all(void);
-static void hardware_disable_all(void);
 
 static void kvm_io_bus_destroy(struct kvm_io_bus *bus);
 
@@ -2330,14 +2328,15 @@ static void hardware_disable_all_nolock(void)
 		on_each_cpu(hardware_disable_nolock, NULL, 1);
 }
 
-static void hardware_disable_all(void)
+void hardware_disable_all(void)
 {
 	raw_spin_lock(&kvm_lock);
 	hardware_disable_all_nolock();
 	raw_spin_unlock(&kvm_lock);
 }
+EXPORT_SYMBOL_GPL(hardware_disable_all);
 
-static int hardware_enable_all(void)
+int hardware_enable_all(void)
 {
 	int r = 0;
 
@@ -2358,6 +2357,7 @@ static int hardware_enable_all(void)
 
 	return r;
 }
+EXPORT_SYMBOL_GPL(hardware_enable_all);
 
 static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val,
 			   void *v)
Signature · 1 line
1.7.1
#4 lore raw

PATCH v3 3/5 KVM-INTEL: Add new module vmcsinfo-intel to fill VMCSINFO

In reply to Yanfei Zhang

This patch implements a new module named vmcsinfo-intel. The module fills VMCSINFO with the VMCS revision identifier, and offsets of VMCS fields.

Note, offsets of fields below will not be filled into VMCSINFO:
1. fields defined in Intel specification (Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C) but not defined in *vmcs_field*.
2. fields unsupported.

---
 arch/x86/kvm/Kconfig    |   11 +++
 arch/x86/kvm/Makefile   |    3 +
 arch/x86/kvm/vmcsinfo.c |  198 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 212 insertions(+), 0 deletions(-)
 create mode 100644 arch/x86/kvm/vmcsinfo.c
arch/x86/kvm/Kconfig +11 -0
index a28f338..1dd64b1 100644
@@ -63,6 +63,17 @@ config KVM_INTEL
 	  To compile this as a module, choose M here: the module
 	  will be called kvm-intel.
 
+config VMCSINFO_INTEL
+	tristate "Export VMCSINFO for Intel processors"
+	depends on KVM_INTEL
+	---help---
+	  Provides support for exporting VMCSINFO on Intel processors equipped
+	  with the VT extensions. The VMCSINFO contains a VMCS revision
+	  identifier and offsets of VMCS fields.
+
+	  To compile this as a module, choose M here: the module
+	  will be called vmcsinfo-intel.
+
 config KVM_AMD
 	tristate "KVM for AMD processors support"
 	depends on KVM
arch/x86/kvm/Makefile +3 -0
index 4f579e8..12a1ef6 100644
@@ -4,6 +4,7 @@ ccflags-y += -Ivirt/kvm -Iarch/x86/kvm
 CFLAGS_x86.o := -I.
 CFLAGS_svm.o := -I.
 CFLAGS_vmx.o := -I.
+CFLAGS_vmcsinfo.o := -I.
 
 kvm-y			+= $(addprefix ../../../virt/kvm/, kvm_main.o ioapic.o \
 				coalesced_mmio.o irq_comm.o eventfd.o \
@@ -15,7 +16,9 @@ kvm-y			+= x86.o mmu.o emulate.o i8259.o irq.o lapic.o \
 			   i8254.o timer.o cpuid.o pmu.o
 kvm-intel-y		+= vmx.o
 kvm-amd-y		+= svm.o
+vmcsinfo-intel-y	+= vmcsinfo.o
 
 obj-$(CONFIG_KVM)	+= kvm.o
 obj-$(CONFIG_KVM_INTEL)	+= kvm-intel.o
 obj-$(CONFIG_KVM_AMD)	+= kvm-amd.o
+obj-$(CONFIG_VMCSINFO_INTEL)	+= vmcsinfo-intel.o
arch/x86/kvm/vmcsinfo.c new file +198 -0

198 lines not on this page · raw message

Signature · 1 line
1.7.1
#5 lore raw

PATCH v3 4/5 Sysfs: Export VMCSINFO via sysfs

In reply to Yanfei Zhang

This patch export offsets of fields via /sys/devices/cpu/vmcs/. Individual offsets are contained in subfiles named by the filed's encoding, e.g.: /sys/devices/cpu/vmcs/0800

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +13 -0
index 346be8b..dd05ee7 100644
@@ -26,6 +26,7 @@
 #include <linux/async.h>
 #include <linux/pm_runtime.h>
 #include <linux/netdevice.h>
+#include <asm/vmcsinfo.h>
 
 #include "base.h"
 #include "power/power.h"
@@ -1038,6 +1039,11 @@ int device_add(struct device *dev)
 	error = dpm_sysfs_add(dev);
 	if (error)
 		goto DPMError;
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+	error = vmcs_sysfs_add(dev);
+	if (error)
+		goto VMCSError;
+#endif
 	device_pm_add(dev);
 
 	/* Notify clients of device addition.  This call must come
@@ -1069,6 +1075,10 @@ int device_add(struct device *dev)
 done:
 	put_device(dev);
 	return error;
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+ VMCSError:
+	dpm_sysfs_remove(dev);
+#endif
  DPMError:
 	bus_remove_device(dev);
  BusError:
@@ -1171,6 +1181,9 @@ void device_del(struct device *dev)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DEL_DEVICE, dev);
 	device_pm_remove(dev);
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+	vmcs_sysfs_remove(dev);
+#endif
 	dpm_sysfs_remove(dev);
 	if (parent)
 		klist_del(&dev->p->knode_parent);
Signature · 1 line
1.7.1
maintainer
#6 lore raw

Re: [PATCH v3 4/5] Sysfs: Export VMCSINFO via sysfs

Quoted 14 lines · Yanfei Zhang

This patch export offsets of fields via /sys/devices/cpu/vmcs/. Individual offsets are contained in subfiles named by the filed's encoding, e.g.: /sys/devices/cpu/vmcs/0800

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +0 -0
index 346be8b..dd05ee7 100644

drivers/base/core.c line 29

Quoted 4 lines
drivers/base/core.c +1 -0
@@ -26,6 +26,7 @@
 #include <linux/async.h>
 #include <linux/pm_runtime.h>
 #include <linux/netdevice.h>
+#include <asm/vmcsinfo.h>

Did you just break the build on all other arches? Not nice.

drivers/base/core.c int device_add(struct device *dev) · line 1046

Quoted 9 lines
drivers/base/core.c +5 -0
@@ -1038,6 +1039,11 @@ int device_add(struct device *dev)
 	error = dpm_sysfs_add(dev);
 	if (error)
 		goto DPMError;
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+	error = vmcs_sysfs_add(dev);
+	if (error)
+		goto VMCSError;
+#endif

Oh my no, that's no way to ever do this, you know better than that, please fix.

greg k-h

#7 lore raw

Re: [PATCH v3 4/5] Sysfs: Export VMCSINFO via sysfs

Quoted 35 lines · Greg KH
Quoted 14 lines · Yanfei Zhang

This patch export offsets of fields via /sys/devices/cpu/vmcs/. Individual offsets are contained in subfiles named by the filed's encoding, e.g.: /sys/devices/cpu/vmcs/0800

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +0 -0
index 346be8b..dd05ee7 100644

drivers/base/core.c line 29

Quoted 4 lines
drivers/base/core.c +1 -0
@@ -26,6 +26,7 @@
 #include <linux/async.h>
 #include <linux/pm_runtime.h>
 #include <linux/netdevice.h>
+#include <asm/vmcsinfo.h>

Did you just break the build on all other arches? Not nice.

drivers/base/core.c int device_add(struct device *dev) · line 1046

Quoted 9 lines
drivers/base/core.c +5 -0
@@ -1038,6 +1039,11 @@ int device_add(struct device *dev)
 	error = dpm_sysfs_add(dev);
 	if (error)
 		goto DPMError;
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+	error = vmcs_sysfs_add(dev);
+	if (error)
+		goto VMCSError;
+#endif

Oh my no, that's no way to ever do this, you know better than that, please fix.

Quoted 2 lines

greg k-h

Sorry for my thoughtless, Here is the new patch.

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +13 -0
index 346be8b..7b5266a 100644
@@ -30,6 +30,13 @@
 #include "base.h"
 #include "power/power.h"
 
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+#include <asm/vmcsinfo.h>
+#else
+static inline int vmcs_sysfs_add(struct device *dev) { return 0; }
+static inline void vmcs_sysfs_remove(struct device *dev) { }
+#endif
+
 #ifdef CONFIG_SYSFS_DEPRECATED
 #ifdef CONFIG_SYSFS_DEPRECATED_V2
 long sysfs_deprecated = 1;
@@ -1038,6 +1045,9 @@ int device_add(struct device *dev)
 	error = dpm_sysfs_add(dev);
 	if (error)
 		goto DPMError;
+	error = vmcs_sysfs_add(dev);
+	if (error)
+		goto VMCSError;
 	device_pm_add(dev);
 
 	/* Notify clients of device addition.  This call must come
@@ -1069,6 +1079,8 @@ int device_add(struct device *dev)
 done:
 	put_device(dev);
 	return error;
+ VMCSError:
+	dpm_sysfs_remove(dev);
  DPMError:
 	bus_remove_device(dev);
  BusError:
@@ -1171,6 +1183,7 @@ void device_del(struct device *dev)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DEL_DEVICE, dev);
 	device_pm_remove(dev);
+	vmcs_sysfs_remove(dev);
 	dpm_sysfs_remove(dev);
 	if (parent)
 		klist_del(&dev->p->knode_parent);
Signature · 1 line
1.7.1
maintainer
#8 lore raw

Re: [PATCH v3 4/5] Sysfs: Export VMCSINFO via sysfs

Quoted 50 lines · Yanfei Zhang
Quoted 35 lines · Greg KH
Quoted 14 lines · Yanfei Zhang

This patch export offsets of fields via /sys/devices/cpu/vmcs/. Individual offsets are contained in subfiles named by the filed's encoding, e.g.: /sys/devices/cpu/vmcs/0800

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +0 -0
index 346be8b..dd05ee7 100644

drivers/base/core.c line 29

Quoted 4 lines
drivers/base/core.c +1 -0
@@ -26,6 +26,7 @@
 #include <linux/async.h>
 #include <linux/pm_runtime.h>
 #include <linux/netdevice.h>
+#include <asm/vmcsinfo.h>

Did you just break the build on all other arches? Not nice.

drivers/base/core.c int device_add(struct device *dev) · line 1046

Quoted 9 lines
drivers/base/core.c +5 -0
@@ -1038,6 +1039,11 @@ int device_add(struct device *dev)
 	error = dpm_sysfs_add(dev);
 	if (error)
 		goto DPMError;
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+	error = vmcs_sysfs_add(dev);
+	if (error)
+		goto VMCSError;
+#endif

Oh my no, that's no way to ever do this, you know better than that, please fix.

Quoted 2 lines

greg k-h

Sorry for my thoughtless, Here is the new patch.

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +0 -0
index 346be8b..7b5266a 100644

drivers/base/core.c line 38

Quoted 9 lines
drivers/base/core.c +6 -0
@@ -30,6 +30,13 @@
 #include "base.h"
 #include "power/power.h"
 
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+#include <asm/vmcsinfo.h>
+#else
+static inline int vmcs_sysfs_add(struct device *dev) { return 0; }
+static inline void vmcs_sysfs_remove(struct device *dev) { }
+#endif

{sigh} No, again, you know better, don't do this.

greg k-h

maintainer
#9 lore raw

Re: [PATCH v3 4/5] Sysfs: Export VMCSINFO via sysfs

Quoted 54 lines · Greg KH
Quoted 53 lines · Yanfei Zhang
Quoted 35 lines · Greg KH
Quoted 14 lines · Yanfei Zhang

This patch export offsets of fields via /sys/devices/cpu/vmcs/. Individual offsets are contained in subfiles named by the filed's encoding, e.g.: /sys/devices/cpu/vmcs/0800

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +0 -0
index 346be8b..dd05ee7 100644

drivers/base/core.c line 29

Quoted 4 lines
drivers/base/core.c +1 -0
@@ -26,6 +26,7 @@
 #include <linux/async.h>
 #include <linux/pm_runtime.h>
 #include <linux/netdevice.h>
+#include <asm/vmcsinfo.h>

Did you just break the build on all other arches? Not nice.

drivers/base/core.c int device_add(struct device *dev) · line 1046

Quoted 9 lines
drivers/base/core.c +5 -0
@@ -1038,6 +1039,11 @@ int device_add(struct device *dev)
 	error = dpm_sysfs_add(dev);
 	if (error)
 		goto DPMError;
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+	error = vmcs_sysfs_add(dev);
+	if (error)
+		goto VMCSError;
+#endif

Oh my no, that's no way to ever do this, you know better than that, please fix.

Quoted 2 lines

greg k-h

Sorry for my thoughtless, Here is the new patch.

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +0 -0
index 346be8b..7b5266a 100644
@@ -30,6 +30,13 @@
 #include "base.h"
 #include "power/power.h"
 
Quoted 8 lines

drivers/base/core.c line 38

Quoted 6 lines
drivers/base/core.c +6 -0
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+#include <asm/vmcsinfo.h>
+#else
+static inline int vmcs_sysfs_add(struct device *dev) { return 0; }
+static inline void vmcs_sysfs_remove(struct device *dev) { }
+#endif

{sigh} No, again, you know better, don't do this.

Ok, as others have rightly pointed out, this wasn't the most helpful review comment, sorry about that.

In Linux, we don't put ifdefs in .c files, we put them in .h files. See many examples of this all over the place. That's my main complaints the past two times of this patch.

But, for this, I would question why you even want / need to do this in the drivers/base/core/ file in the first place. Shouldn't it be in some arch or cpu specific file instead that already handles the cpu files?

thanks,

greg k-h

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw

Re: [PATCH v3 4/5] Sysfs: Export VMCSINFO via sysfs

Quoted 71 lines · Greg KH
Quoted 54 lines · Greg KH
Quoted 53 lines · Yanfei Zhang
Quoted 35 lines · Greg KH
Quoted 14 lines · Yanfei Zhang

This patch export offsets of fields via /sys/devices/cpu/vmcs/. Individual offsets are contained in subfiles named by the filed's encoding, e.g.: /sys/devices/cpu/vmcs/0800

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +0 -0
index 346be8b..dd05ee7 100644

drivers/base/core.c line 29

Quoted 4 lines
drivers/base/core.c +1 -0
@@ -26,6 +26,7 @@
 #include <linux/async.h>
 #include <linux/pm_runtime.h>
 #include <linux/netdevice.h>
+#include <asm/vmcsinfo.h>

Did you just break the build on all other arches? Not nice.

drivers/base/core.c int device_add(struct device *dev) · line 1046

Quoted 9 lines
drivers/base/core.c +5 -0
@@ -1038,6 +1039,11 @@ int device_add(struct device *dev)
 	error = dpm_sysfs_add(dev);
 	if (error)
 		goto DPMError;
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+	error = vmcs_sysfs_add(dev);
+	if (error)
+		goto VMCSError;
+#endif

Oh my no, that's no way to ever do this, you know better than that, please fix.

Quoted 2 lines

greg k-h

Sorry for my thoughtless, Here is the new patch.

---
 drivers/base/core.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/base/core.c +0 -0
index 346be8b..7b5266a 100644
@@ -30,6 +30,13 @@
 #include "base.h"
 #include "power/power.h"
 
Quoted 8 lines

drivers/base/core.c line 38

Quoted 6 lines
drivers/base/core.c +6 -0
+#if defined(CONFIG_KVM_INTEL) || defined(CONFIG_KVM_INTEL_MODULE)
+#include <asm/vmcsinfo.h>
+#else
+static inline int vmcs_sysfs_add(struct device *dev) { return 0; }
+static inline void vmcs_sysfs_remove(struct device *dev) { }
+#endif

{sigh} No, again, you know better, don't do this.

Ok, as others have rightly pointed out, this wasn't the most helpful review comment, sorry about that.

In Linux, we don't put ifdefs in .c files, we put them in .h files. See many examples of this all over the place. That's my main complaints the past two times of this patch.

Quoted 8 lines

But, for this, I would question why you even want / need to do this in the drivers/base/core/ file in the first place. Shouldn't it be in some arch or cpu specific file instead that already handles the cpu files?

thanks,

greg k-h

Many thanks. I have moved the code to my vmcsinfo_intel module. Thanks again for your helpful comment.

Thanks
Zhang Yanfei

#11 lore raw

PATCH v3 5/5 Documentation: Add ABI entry for vmcs sysfs interface

In reply to Yanfei Zhang
---
 Documentation/ABI/testing/sysfs-devices-cpu-vmcs |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-cpu-vmcs
Documentation/ABI/testing/sysfs-devices-cpu-vmcs new file +11 -0
new file mode 100644
index 0000000..0846b07
@@ -0,0 +1,11 @@
+What:		/sys/devices/cpu/vmcs/
+Date:		June 2012
+KernelVersion:	3.5.0
+Contact:	Zhang Yanfei <[email protected]>
+Description:
+		A collection of vmcs fields' offsets for Intel cpu.
+
+		Individual offsets are contained in subfiles named by
+		the filed's encoding, e.g.:
+
+		/sys/devices/cpu/vmcs/0800
Signature · 1 line
1.7.1