Skip to contentSkip to conversation

Intel SPEC CTRL virtualization support

RFC PATCH In review Zhang Chen 6 participants 28 messages Dec 10, 2022 – Jan 15, 2023

Subsystems (4)
lore
Share & tools
lore

Zhang Chen

author
#0 lore raw

cover Intel SPEC CTRL virtualization support

This series introduces "virtual IA32_SPEC_CTRL" support. Here are introduction and use cases of this new feature.

### Virtual IA32_SPEC_CTRL

Virtual IA32_SPEC_CTRL [1] is a new VMX feature on Intel CPUs. This feature allows VMM to fix some bits of IA32_SPEC_CTRL MSR even when the MSR is pass-thru'd to a guest.

### Use cases of virtual IA32_SPEC_CTRL

Software mitigations like Retpoline and software BHB-clearing sequence depend on CPU microarchitectures. And guest cannot know exactly the underlying microarchitecture. When a guest is migrated between processors of different microarchitectures, software mitigations which work perfectly on previous microachitecture may be not effective on the new one. To fix the problem, some hardware mitigations should be used in conjunction with software mitigations. Using virtual IA32_SPEC_CTRL, VMM can enforce hardware mitigations transparently to guests and avoid those hardware mitigations being accidentally disabled when guest changes IA32_SPEC_CTRL MSR.

### Intention of this series

This series adds the capability of enforcing hardware mitigations for guests transparently and efficiently (i.e., without intecepting IA32_SPEC_CTRL MSR accesses) to kvm. The capability can be used to solve the VM migration issue in a pool consisting of processors of different microarchitectures.

Specifically, below are two target scenarios of this series:

Scenario 1: If retpoline is used by a VM to mitigate IMBTI in CPL0, VMM
can set RRSBA_DIS_S on parts enumerates RRSBA. Note that the VM is presented with a microarchitecture doesn't enumerate RRSBA.

Scenario 2: If a VM uses software BHB-clearing sequence on transitions
into CPL0 to mitigate BHI, VMM can use "virtual IA32_SPEC_CTRL" to set BHI_DIS_S on newer hardware which doesn't enumerate BHI_NO.

Intel defines some virtual MSRs for guests to notify VMM of software mitigations in use. Such information allows VMM to enable hardware mitigations only when necessary (i.e., VMs care about the vulnerability and are using software mitigiation) to minimize the performance impact to other VMs. The support of the virtual MSRs also added in this series.

### Organization of this series

This series is developed based on v6.1-rc7 (commit ef4d3ea40565a781c25847e9cb96c1bd9f462bc6)

1. Patch 1 introduce Intel SPEC_CTRL BHI related definition.
2. Patch 2/3 advertises {RRSBA/BHI}_CTRL for retpoline/BHB-clearing sequence mitigations for KVM.
3. Patch 4 introduce Intel virtual IA32_SPEC_CTRL
4. Patch 5-7 virtualize a set of virtual MSRs for guests to communicate software mitigations in use.
5. Patch 8-9 enable "virtual IA32_SPEC_CTRL" VMX feature with virtual MSRs support in KVM.

### Documentation

Refer to below link for more information:
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html

Pawan Gupta (1):
x86/bugs: Use Virtual MSRs to request hardware mitigations

Zhang Chen (8):
x86/speculation: Introduce Intel SPEC_CTRL BHI related definition
KVM: x86: Add a kvm-only leaf for RRSBA_CTRL
KVM: x86: Add a kvm-only leaf for BHI_CTRL x86/kvm/vmx: Virtualize Intel IA32_SPEC_CTRL
kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest
MSR_IA32_ARCH_CAPABILITIES
kvm/x86: Add MSR_VIRTUAL_MITIGATION_ENUM/CTRL emulation x86/kvm/vmx: Initialize SPEC_CTRL MASK for RRSBA x86/kvm/vmx: Initialize SPEC_CTRL MASK for BHI

 arch/x86/include/asm/msr-index.h       |  29 +++++++
 arch/x86/include/asm/vmx.h             |   5 ++
 arch/x86/include/asm/vmxfeatures.h     |   2 +
 arch/x86/kernel/cpu/bugs.c             |  24 ++++++
 arch/x86/kvm/cpuid.c                   |   4 +
 arch/x86/kvm/reverse_cpuid.h           |   9 ++
 arch/x86/kvm/vmx/capabilities.h        |   5 ++
 arch/x86/kvm/vmx/vmx.c                 | 109 ++++++++++++++++++++++++-
 arch/x86/kvm/vmx/vmx.h                 |  27 +++++-
 arch/x86/kvm/x86.c                     |  23 +++++-
 tools/arch/x86/include/asm/msr-index.h |   6 ++
 11 files changed, 238 insertions(+), 5 deletions(-)
Signature · 1 line
2.25.1
author
#1 lore raw

RFC PATCH 1/9 x86/speculation: Introduce Intel SPEC_CTRL BHI related definition

Define BHI_NO bit and new control of BHI hardware mitigation in IA32_SPEC_CTRL. These definitions are used by following KVM patches to determine whether to enforce BHI hardware mitigiations for guests transparently.

BHI_NO means the processor isn't vulnernable to BHI attacks. BHI_DIS_S is a new indirect predictor control. Once enabled, BHI_DIS_S prevents predicted targets of indirect branches executed in CPL0/1/2 from being selected based on branch history from branches executed in CPL3. While set in the VMX root, it also prevents predicted targets executed in CPL0 from being selected based on branch history from branches executed in a VMX non-root.

Branch History Injection (BHI) describes a specific form of intra-mode BTI, where an attacker may manipulate branch history before transitioning from user to supervisor mode (or from VMX non-root/guest to root mode) in an effort to cause an indirect branch predictor to select a specific predictor entry for an indirect branch, and a disclosure gadget at the predicted target will transiently execute. This may be possible since the relevant branch history may contain branches taken in previous security contexts, and in particular, in other predictor modes.

Refer to below link for more information:
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html

---
 arch/x86/include/asm/msr-index.h       | 6 ++++++
 tools/arch/x86/include/asm/msr-index.h | 6 ++++++
 2 files changed, 12 insertions(+)
arch/x86/include/asm/msr-index.h +6 -0
index 4a2af82553e4..1143ac9400c3 100644
@@ -53,6 +53,8 @@
 #define SPEC_CTRL_SSBD			BIT(SPEC_CTRL_SSBD_SHIFT)	/* Speculative Store Bypass Disable */
 #define SPEC_CTRL_RRSBA_DIS_S_SHIFT	6	   /* Disable RRSBA behavior */
 #define SPEC_CTRL_RRSBA_DIS_S		BIT(SPEC_CTRL_RRSBA_DIS_S_SHIFT)
+#define SPEC_CTRL_BHI_DIS_S_SHIFT	10	   /* Enable BHI_DIS_S behavior */
+#define SPEC_CTRL_BHI_DIS_S		BIT(SPEC_CTRL_BHI_DIS_S_SHIFT)
 
 #define MSR_IA32_PRED_CMD		0x00000049 /* Prediction Command */
 #define PRED_CMD_IBPB			BIT(0)	   /* Indirect Branch Prediction Barrier */
@@ -150,6 +152,10 @@
 						 * are restricted to targets in
 						 * kernel.
 						 */
+#define ARCH_CAP_BHI_NO			BIT(20)	/*
+						 * Not susceptible to Branch History
+						 * Injection.
+						 */
 #define ARCH_CAP_PBRSB_NO		BIT(24)	/*
 						 * Not susceptible to Post-Barrier
 						 * Return Stack Buffer Predictions.
tools/arch/x86/include/asm/msr-index.h +6 -0
index f17ade084720..aed18b76dee0 100644
@@ -53,6 +53,8 @@
 #define SPEC_CTRL_SSBD			BIT(SPEC_CTRL_SSBD_SHIFT)	/* Speculative Store Bypass Disable */
 #define SPEC_CTRL_RRSBA_DIS_S_SHIFT	6	   /* Disable RRSBA behavior */
 #define SPEC_CTRL_RRSBA_DIS_S		BIT(SPEC_CTRL_RRSBA_DIS_S_SHIFT)
+#define SPEC_CTRL_BHI_DIS_S_SHIFT	10         /* Enable BHI_DIS_S behavior */
+#define SPEC_CTRL_BHI_DIS_S		BIT(SPEC_CTRL_BHI_DIS_S_SHIFT)
 
 #define MSR_IA32_PRED_CMD		0x00000049 /* Prediction Command */
 #define PRED_CMD_IBPB			BIT(0)	   /* Indirect Branch Prediction Barrier */
@@ -150,6 +152,10 @@
 						 * are restricted to targets in
 						 * kernel.
 						 */
+#define ARCH_CAP_BHI_NO			BIT(20) /*
+						 * Not susceptible to Branch History
+						 * Injection.
+						 */
 #define ARCH_CAP_PBRSB_NO		BIT(24)	/*
 						 * Not susceptible to Post-Barrier
 						 * Return Stack Buffer Predictions.
Signature · 1 line
2.25.1
author
#2 lore raw

RFC PATCH 2/9 KVM: x86: Add a kvm-only leaf for RRSBA_CTRL

In reply to Zhang Chen

KVM needs to check if guests can see RRSBA_CTRL. If a guest is using retpoline and cannot see RRSBA_CTRL and the host enumerates RRSBA, KVM is responsible for setting RRSBA_DIS_S for the guest.

This allows VM migration from parts doesn't enumerates RRSBA to those that enumerate RRSBA.

---
 arch/x86/kvm/cpuid.c         | 4 ++++
 arch/x86/kvm/reverse_cpuid.h | 7 +++++++
 2 files changed, 11 insertions(+)
arch/x86/kvm/cpuid.c +4 -0
index 62bc7a01cecc..8d45bc0b4b7c 100644
@@ -668,6 +668,10 @@ void kvm_set_cpu_caps(void)
 		SF(SGX1) | SF(SGX2)
 	);
 
+	kvm_cpu_cap_init_scattered(CPUID_7_2_EDX,
+		SF(RRSBA_CTRL)
+	);
+
 	kvm_cpu_cap_mask(CPUID_8000_0001_ECX,
 		F(LAHF_LM) | F(CMP_LEGACY) | 0 /*SVM*/ | 0 /* ExtApicSpace */ |
 		F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) |
arch/x86/kvm/reverse_cpuid.h +7 -0
index a19d473d0184..4c38ed61c505 100644
@@ -13,6 +13,7 @@
  */
 enum kvm_only_cpuid_leafs {
 	CPUID_12_EAX	 = NCAPINTS,
+	CPUID_7_2_EDX,
 	NR_KVM_CPU_CAPS,
 
 	NKVMCAPINTS = NR_KVM_CPU_CAPS - NCAPINTS,
@@ -24,6 +25,9 @@ enum kvm_only_cpuid_leafs {
 #define KVM_X86_FEATURE_SGX1		KVM_X86_FEATURE(CPUID_12_EAX, 0)
 #define KVM_X86_FEATURE_SGX2		KVM_X86_FEATURE(CPUID_12_EAX, 1)
 
+/* Intel-defined sub-features, CPUID level 0x00000007:2 (EDX)*/
+#define KVM_X86_FEATURE_RRSBA_CTRL	KVM_X86_FEATURE(CPUID_7_2_EDX, 2)
+
 struct cpuid_reg {
 	u32 function;
 	u32 index;
@@ -46,6 +50,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
 	[CPUID_8000_0007_EBX] = {0x80000007, 0, CPUID_EBX},
 	[CPUID_7_EDX]         = {         7, 0, CPUID_EDX},
 	[CPUID_7_1_EAX]       = {         7, 1, CPUID_EAX},
+	[CPUID_7_2_EDX]       = {         7, 2, CPUID_EDX},
 	[CPUID_12_EAX]        = {0x00000012, 0, CPUID_EAX},
 	[CPUID_8000_001F_EAX] = {0x8000001f, 0, CPUID_EAX},
 };
@@ -78,6 +83,8 @@ static __always_inline u32 __feature_translate(int x86_feature)
 		return KVM_X86_FEATURE_SGX1;
 	else if (x86_feature == X86_FEATURE_SGX2)
 		return KVM_X86_FEATURE_SGX2;
+	else if (x86_feature == X86_FEATURE_RRSBA_CTRL)
+		return KVM_X86_FEATURE_RRSBA_CTRL;
 
 	return x86_feature;
 }
Signature · 1 line
2.25.1
#3 lore raw

Re: [RFC PATCH 2/9] KVM: x86: Add a kvm-only leaf for RRSBA_CTRL

Quoted 5 lines · Zhang Chen

KVM needs to check if guests can see RRSBA_CTRL. If a guest is using retpoline and cannot see RRSBA_CTRL and the host enumerates RRSBA, KVM is responsible for setting RRSBA_DIS_S for the guest.

This allows VM migration from parts doesn't enumerates RRSBA to those

perhaps s/parts doesn't enumerates/parts that do not enumerate/ reads better?

Quoted 1 line

that enumerate RRSBA.

and here maybe s/that enumerate RRSBA/do/ ?

Thanks and BR,
Ricardo

#4 lore raw

RE: [RFC PATCH 2/9] KVM: x86: Add a kvm-only leaf for RRSBA_CTRL

Quoted 16 lines

-----Original Message-----
From: Ricardo Neri <[email protected]>
Sent: Thursday, December 15, 2022 5:34 AM
To: Zhang, Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Gao, Chao <[email protected]>; Pawan Gupta <[email protected]>; Paolo Bonzini <[email protected]>; Christopherson,, Sean <[email protected]>; H. Peter Anvin <[email protected]>; Dave Hansen <[email protected]>; Borislav Petkov <[email protected]>; Ingo Molnar <[email protected]>; Thomas Gleixner <[email protected]>
Subject: Re: [RFC PATCH 2/9] KVM: x86: Add a kvm-only leaf for RRSBA_CTRL

On Sun, Dec 11, 2022 at 12:00:39AM +0800, Zhang Chen wrote:

Quoted 8 lines
Quoted 5 lines

KVM needs to check if guests can see RRSBA_CTRL. If a guest is using retpoline and cannot see RRSBA_CTRL and the host enumerates RRSBA, KVM is responsible for setting RRSBA_DIS_S for the guest.

This allows VM migration from parts doesn't enumerates RRSBA to those

perhaps s/parts doesn't enumerates/parts that do not enumerate/ reads better?

OK.

Quoted 4 lines
Quoted 2 lines

that enumerate RRSBA.

and here maybe s/that enumerate RRSBA/do/ ?

Looks good to me. Thanks for your comments.
I will fix patch2/3 in the next version.

Thanks
Chen

Quoted 3 lines

Thanks and BR,
Ricardo

author
#5 lore raw

RFC PATCH 3/9 KVM: x86: Add a kvm-only leaf for BHI_CTRL

In reply to Zhang Chen

KVM needs to check if guests can see BHI_CTRL. If a guest is using BHB-clearing sequence and cannot see BHI_CTRL and the host enumerates BHI, KVM is responsible for setting BHI_DIS_S for the guest.

This allows VM migration from parts doesn't enumerate BHI to those that enumerate BHI.

---
 arch/x86/kvm/cpuid.c         | 2 +-
 arch/x86/kvm/reverse_cpuid.h | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/cpuid.c +1 -1
index 8d45bc0b4b7c..91af27cc57e5 100644
@@ -669,7 +669,7 @@ void kvm_set_cpu_caps(void)
 	);
 
 	kvm_cpu_cap_init_scattered(CPUID_7_2_EDX,
-		SF(RRSBA_CTRL)
+		SF(RRSBA_CTRL) | F(BHI_CTRL)
 	);
 
 	kvm_cpu_cap_mask(CPUID_8000_0001_ECX,
arch/x86/kvm/reverse_cpuid.h +2 -0
index 4c38ed61c505..cf4e209ce2f6 100644
@@ -27,6 +27,8 @@ enum kvm_only_cpuid_leafs {
 
 /* Intel-defined sub-features, CPUID level 0x00000007:2 (EDX)*/
 #define KVM_X86_FEATURE_RRSBA_CTRL	KVM_X86_FEATURE(CPUID_7_2_EDX, 2)
+/* X86_FEATURE_BHI_CTRL only used by KVM */
+#define X86_FEATURE_BHI_CTRL		KVM_X86_FEATURE(CPUID_7_2_EDX, 4)
 
 struct cpuid_reg {
 	u32 function;
Signature · 1 line
2.25.1
#6 lore raw

Re: [RFC PATCH 3/9] KVM: x86: Add a kvm-only leaf for BHI_CTRL

Quoted 6 lines · Zhang Chen

KVM needs to check if guests can see BHI_CTRL. If a guest is using BHB-clearing sequence and cannot see BHI_CTRL and the host enumerates BHI, KVM is responsible for setting BHI_DIS_S for the guest.

This allows VM migration from parts doesn't enumerate BHI to those that enumerate BHI.

Similar comment as in patch 2.

author
#7 lore raw

RFC PATCH 4/9 x86/kvm/vmx: Virtualize Intel IA32_SPEC_CTRL

In reply to Zhang Chen

Currently KVM disables interception of IA32_SPEC_CTRL after a non-0 is written to IA32_SPEC_CTRL by guest. Then, guest is allowed to write any value to hardware.

"virtualize IA32_SPEC_CTRL" is a new tertiary vm-exec control. It provides VMM the capability to restrict the value of IA32_SPEC_CTRL in hardware even if the MSR isn't intercepted (e.g., prevent VMs from changing some bits in IA32_SPEC_CTRL) in an efficient way.

Two new fields are added to VMCS,
IA32_SPEC_CTRL_MASK: Setting a bit in this field prevents guest software from modifying the corresponding bit in the IA32_SPEC_CTRL MSR
IA32_SPEC_CTRL_SHADOW: This field contains the value that guest software expects to be in the IA32_SPEC_CTRL MSR

In VMX non-root mode, when IA32_SPEC_CTRL isn't intercepted by VMM, guest's accesses to IA32_SPEC_CTRL is virtualized by the processor according to the two new fields:
RDMSR IA32_SPEC_CTRL returns the shadow value.
WRMSR IA32_SPEC_CTRL writes EDX:EAX to the shadow field and calculate a new value according to the guest value (EDX:EAX), current value of IA32_SPEC_CTRL MSR and the IA32_SPEC_CTRL_MASK field (specifically, (cur_val & mask) | (guest_val & ~mask)) and write it to IA32_SPEC_CTRL MSR.

Enable "virtual IA32_SPEC_CTRL" if it is supported. With "virtual IA32_SPEC_CTRL" enabled, IA32_SPEC_CTRL MSR value seen from guest p.o.v is different from the value in hardware when guest is running. We refer to the two values as below:

1. effective value of IA32_SPEC_CTRL. This value is the one programmed in hardware when the vCPU is running.
2. shadow value of IA32_SPEC_CTRL. This value is returned when rdmsr is used inside a guest to read IA32_SPEC_CTRL. This value doesn't affect CPU's enablement of indirect predictor controls.

In KVM, vmx->spec_ctrl always stores the effective value of IA32_SPEC_CTRL when guest is running (even when "virtual IA32_SPEC_CTRL" is disabled. In this case, the shadow value is equal to the effective one). When "virtual IA32_SPEC_CTRL" is enabled, the shadow value of IA32_SPEC_CTRL is stored in IA32_SPEC_CTRL_SHADOW field in VMCS.

IA32_SPEC_CTRL_MASK is always 0 for now. It means all supported bits in hardware is allowed to be toggled by guest's wrmsr. The mask will be changed by following patches.

Note "virtual IA32_SPEC_CTRL" is now used by VMM to enforce some bits of IA32_SPEC_CTRL to 1 (i.e., enabled some HW mitigations transparently for guests). In theory, VMM can disable some HW mitigations behind guests. But to keep this series simple, we leave that for future work.

---
 arch/x86/include/asm/vmx.h         |  5 +++++
 arch/x86/include/asm/vmxfeatures.h |  2 ++
 arch/x86/kvm/vmx/capabilities.h    |  5 +++++
 arch/x86/kvm/vmx/vmx.c             | 30 ++++++++++++++++++++++++++++--
 arch/x86/kvm/vmx/vmx.h             | 10 +++++++++-
 5 files changed, 49 insertions(+), 3 deletions(-)
arch/x86/include/asm/vmx.h +5 -0
index 498dc600bd5c..c2efdad491c1 100644
@@ -81,6 +81,7 @@
  * Definitions of Tertiary Processor-Based VM-Execution Controls.
  */
 #define TERTIARY_EXEC_IPI_VIRT			VMCS_CONTROL_BIT(IPI_VIRT)
+#define TERTIARY_EXEC_VIRT_SPEC_CTRL		VMCS_CONTROL_BIT(VIRT_SPEC_CTRL)
 
 #define PIN_BASED_EXT_INTR_MASK                 VMCS_CONTROL_BIT(INTR_EXITING)
 #define PIN_BASED_NMI_EXITING                   VMCS_CONTROL_BIT(NMI_EXITING)
@@ -233,6 +234,10 @@ enum vmcs_field {
 	TERTIARY_VM_EXEC_CONTROL_HIGH	= 0x00002035,
 	PID_POINTER_TABLE		= 0x00002042,
 	PID_POINTER_TABLE_HIGH		= 0x00002043,
+	IA32_SPEC_CTRL_MASK             = 0x0000204A,
+	IA32_SPEC_CTRL_MASK_HIGH        = 0x0000204B,
+	IA32_SPEC_CTRL_SHADOW           = 0x0000204C,
+	IA32_SPEC_CTRL_SHADOW_HIGH      = 0x0000204D,
 	GUEST_PHYSICAL_ADDRESS          = 0x00002400,
 	GUEST_PHYSICAL_ADDRESS_HIGH     = 0x00002401,
 	VMCS_LINK_POINTER               = 0x00002800,
arch/x86/include/asm/vmxfeatures.h +2 -0
index c6a7eed03914..d3b7237d9c42 100644
@@ -89,4 +89,6 @@
 
 /* Tertiary Processor-Based VM-Execution Controls, word 3 */
 #define VMX_FEATURE_IPI_VIRT		( 3*32+  4) /* Enable IPI virtualization */
+#define VMX_FEATURE_VIRT_SPEC_CTRL	( 3*32+  7) /* Virtualize IA32_SPEC_CTRL */
+
 #endif /* _ASM_X86_VMXFEATURES_H */
arch/x86/kvm/vmx/capabilities.h +5 -0
index 07254314f3dd..a9a0adcd403b 100644
@@ -138,6 +138,11 @@ static inline bool cpu_has_tertiary_exec_ctrls(void)
 		CPU_BASED_ACTIVATE_TERTIARY_CONTROLS;
 }
 
+static inline bool cpu_has_virt_spec_ctrl(void)
+{
+	return vmcs_config.cpu_based_3rd_exec_ctrl & TERTIARY_EXEC_VIRT_SPEC_CTRL;
+}
+
 static inline bool cpu_has_vmx_virtualize_apic_accesses(void)
 {
 	return vmcs_config.cpu_based_2nd_exec_ctrl &
arch/x86/kvm/vmx/vmx.c +28 -2
index 63247c57c72c..407061b369b4 100644
@@ -1898,7 +1898,10 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		    !guest_has_spec_ctrl_msr(vcpu))
 			return 1;
 
-		msr_info->data = to_vmx(vcpu)->spec_ctrl;
+		if (cpu_has_virt_spec_ctrl())
+			msr_info->data = vmcs_read64(IA32_SPEC_CTRL_SHADOW);
+		else
+			msr_info->data = to_vmx(vcpu)->spec_ctrl;
 		break;
 	case MSR_IA32_SYSENTER_CS:
 		msr_info->data = vmcs_read32(GUEST_SYSENTER_CS);
@@ -2160,10 +2163,22 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		    !guest_has_spec_ctrl_msr(vcpu))
 			return 1;
 
-		if (kvm_spec_ctrl_test_value(data))
+		if (kvm_spec_ctrl_test_value(data | vmx->spec_ctrl_mask))
 			return 1;
 
 		vmx->spec_ctrl = data;
+
+		if (cpu_has_virt_spec_ctrl()) {
+			vmcs_write64(IA32_SPEC_CTRL_SHADOW, data);
+			/*
+			 * Some bits are allowed to be toggled by guest.
+			 * Update the effective value of IA32_SPEC_CTRL
+			 * MSR according to the value written by guest
+			 * but keep bits in the mask set.
+			 */
+			vmx->spec_ctrl = data | vmx->spec_ctrl_mask;
+		}
+
 		if (!data)
 			break;
 
@@ -4673,6 +4688,11 @@ static void init_vmcs(struct vcpu_vmx *vmx)
 	if (cpu_has_vmx_xsaves())
 		vmcs_write64(XSS_EXIT_BITMAP, VMX_XSS_EXIT_BITMAP);
 
+	if (cpu_has_virt_spec_ctrl()) {
+		vmcs_write64(IA32_SPEC_CTRL_SHADOW, 0);
+		vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
+	}
+
 	if (enable_pml) {
 		vmcs_write64(PML_ADDRESS, page_to_phys(vmx->pml_pg));
 		vmcs_write16(GUEST_PML_INDEX, PML_ENTITY_NUM - 1);
@@ -4738,6 +4758,12 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
 		__vmx_vcpu_reset(vcpu);
 
 	vmx->rmode.vm86_active = 0;
+
+	if (cpu_has_virt_spec_ctrl()) {
+		vmx->spec_ctrl_mask = 0;
+		vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
+		vmcs_write64(IA32_SPEC_CTRL_SHADOW, 0);
+	}
 	vmx->spec_ctrl = 0;
 
 	vmx->msr_ia32_umwait_control = 0;
arch/x86/kvm/vmx/vmx.h +9 -1
index a3da84f4ea45..c5a41ae14237 100644
@@ -290,7 +290,14 @@ struct vcpu_vmx {
 	u64		      msr_guest_kernel_gs_base;
 #endif
 
+	/* The value of hardware IA32_SPEC_CTRL MSR when guest is running */
 	u64		      spec_ctrl;
+	/*
+	 * The bits KVM doesn't allow guest to toggle.
+	 * A bit set in the mask should always be set in guest
+	 * IA32_SPEC_CTRL_MSR.
+	 */
+	u64		      spec_ctrl_mask;
 	u32		      msr_ia32_umwait_control;
 
 	/*
@@ -589,7 +596,8 @@ static inline u8 vmx_get_rvi(void)
 
 #define KVM_REQUIRED_VMX_TERTIARY_VM_EXEC_CONTROL 0
 #define KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL			\
-	(TERTIARY_EXEC_IPI_VIRT)
+	(TERTIARY_EXEC_IPI_VIRT |					\
+	TERTIARY_EXEC_VIRT_SPEC_CTRL)
 
 #define BUILD_CONTROLS_SHADOW(lname, uname, bits)						\
 static inline void lname##_controls_set(struct vcpu_vmx *vmx, u##bits val)			\
Signature · 1 line
2.25.1
author
#8 lore raw

RFC PATCH 5/9 x86/bugs: Use Virtual MSRs to request hardware mitigations

In reply to Zhang Chen

From: Pawan Gupta <[email protected]>

Guests that have different family/model than the host may not be aware of hardware mitigations(such as RRSBA_DIS_S) available on host. This is particularly true when guests migrate. To solve this problem Intel processors have added a virtual MSR interface through which guests can report their mitigation status and request VMM to deploy relevant hardware mitigations.

Use this virtualized MSR interface to request relevant hardware controls for retpoline mitigation.

---
 arch/x86/include/asm/msr-index.h | 23 +++++++++++++++++++++++
 arch/x86/kernel/cpu/bugs.c       | 24 ++++++++++++++++++++++++
 2 files changed, 47 insertions(+)
arch/x86/include/asm/msr-index.h +23 -0
index 1143ac9400c3..1166b472377c 100644
@@ -165,6 +165,7 @@
 						 * IA32_XAPIC_DISABLE_STATUS MSR
 						 * supported
 						 */
+#define ARCH_CAP_VIRTUAL_ENUM		BIT(63)	/* MSR_VIRTUAL_ENUMERATION supported */
 
 #define MSR_IA32_FLUSH_CMD		0x0000010b
 #define L1D_FLUSH			BIT(0)	/*
@@ -1062,6 +1063,28 @@
 #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
 #define MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS (1ULL << 29)
 #define MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE   0x1F
+
+/* Intel virtual MSRs */
+#define MSR_VIRTUAL_ENUMERATION			0x50000000
+#define VIRT_ENUM_MITIGATION_CTRL_SUPPORT	BIT(0)	/*
+							 * Mitigation ctrl via virtual
+							 * MSRs supported
+							 */
+
+#define MSR_VIRTUAL_MITIGATION_ENUM		0x50000001
+#define MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT	BIT(0)	/* VMM supports BHI_DIS_S */
+#define MITI_ENUM_RETPOLINE_S_SUPPORT		BIT(1)	/* VMM supports RRSBA_DIS_S */
+
+#define MSR_VIRTUAL_MITIGATION_CTRL		0x50000002
+#define MITI_CTRL_BHB_CLEAR_SEQ_S_USED		BIT(0)	/*
+							 * Request VMM to deploy
+							 * BHI_DIS_S mitigation
+							 */
+#define MITI_CTRL_RETPOLINE_S_USED		BIT(1)	/*
+							 * Request VMM to deploy
+							 * RRSBA_DIS_S mitigation
+							 */
+
 /* AMD-V MSRs */
 
 #define MSR_VM_CR                       0xc0010114
arch/x86/kernel/cpu/bugs.c +24 -0
index 3e3230cccaa7..a9e869f568ee 100644
@@ -1379,6 +1379,28 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
 	dump_stack();
 }
 
+/* Speculation control using virtualized MSRs */
+static void __init spec_ctrl_setup_virtualized_msr(void)
+{
+	u64 msr_virt_enum, msr_mitigation_enum, msr_mitigation_ctrl;
+
+	if (!(x86_read_arch_cap_msr() & ARCH_CAP_VIRTUAL_ENUM))
+		return;
+
+	rdmsrl(MSR_VIRTUAL_ENUMERATION, msr_virt_enum);
+	if (!(msr_virt_enum & VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+		return;
+
+	rdmsrl(MSR_VIRTUAL_MITIGATION_ENUM, msr_mitigation_enum);
+	/* When retpoline is being used, request relevant hardware controls */
+	if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
+	    msr_mitigation_enum & MITI_ENUM_RETPOLINE_S_SUPPORT) {
+		rdmsrl(MSR_VIRTUAL_MITIGATION_CTRL, msr_mitigation_ctrl);
+		msr_mitigation_ctrl |= MITI_CTRL_RETPOLINE_S_USED;
+		wrmsrl(MSR_VIRTUAL_MITIGATION_CTRL, msr_mitigation_ctrl);
+	}
+}
+
 static void __init spectre_v2_select_mitigation(void)
 {
 	enum spectre_v2_mitigation_cmd cmd = spectre_v2_parse_cmdline();
@@ -1485,6 +1507,8 @@ static void __init spectre_v2_select_mitigation(void)
 	    mode == SPECTRE_V2_RETPOLINE)
 		spec_ctrl_disable_kernel_rrsba();
 
+	spec_ctrl_setup_virtualized_msr();
+
 	spectre_v2_enabled = mode;
 	pr_info("%s\n", spectre_v2_strings[mode]);
 
Signature · 1 line
2.25.1
reviewer
#9 lore raw

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 93 lines · Zhang Chen

From: Pawan Gupta <[email protected]>

Guests that have different family/model than the host may not be aware of hardware mitigations(such as RRSBA_DIS_S) available on host. This is particularly true when guests migrate. To solve this problem Intel processors have added a virtual MSR interface through which guests can report their mitigation status and request VMM to deploy relevant hardware mitigations.

Use this virtualized MSR interface to request relevant hardware controls for retpoline mitigation.

---
 arch/x86/include/asm/msr-index.h | 23 +++++++++++++++++++++++
 arch/x86/kernel/cpu/bugs.c       | 24 ++++++++++++++++++++++++
 2 files changed, 47 insertions(+)
arch/x86/include/asm/msr-index.h +23 -0
index 1143ac9400c3..1166b472377c 100644
@@ -165,6 +165,7 @@
 						 * IA32_XAPIC_DISABLE_STATUS MSR
 						 * supported
 						 */
+#define ARCH_CAP_VIRTUAL_ENUM		BIT(63)	/* MSR_VIRTUAL_ENUMERATION supported */
 
 #define MSR_IA32_FLUSH_CMD		0x0000010b
 #define L1D_FLUSH			BIT(0)	/*
@@ -1062,6 +1063,28 @@
 #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
 #define MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS (1ULL << 29)
 #define MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE   0x1F
+
+/* Intel virtual MSRs */
+#define MSR_VIRTUAL_ENUMERATION			0x50000000
+#define VIRT_ENUM_MITIGATION_CTRL_SUPPORT	BIT(0)	/*
+							 * Mitigation ctrl via virtual
+							 * MSRs supported
+							 */
+
+#define MSR_VIRTUAL_MITIGATION_ENUM		0x50000001
+#define MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT	BIT(0)	/* VMM supports BHI_DIS_S */
+#define MITI_ENUM_RETPOLINE_S_SUPPORT		BIT(1)	/* VMM supports RRSBA_DIS_S */
+
+#define MSR_VIRTUAL_MITIGATION_CTRL		0x50000002
+#define MITI_CTRL_BHB_CLEAR_SEQ_S_USED		BIT(0)	/*
+							 * Request VMM to deploy
+							 * BHI_DIS_S mitigation
+							 */
+#define MITI_CTRL_RETPOLINE_S_USED		BIT(1)	/*
+							 * Request VMM to deploy
+							 * RRSBA_DIS_S mitigation
+							 */
+
 /* AMD-V MSRs */
 
 #define MSR_VM_CR                       0xc0010114
arch/x86/kernel/cpu/bugs.c +22 -0
index 3e3230cccaa7..a9e869f568ee 100644
@@ -1379,6 +1379,28 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
 	dump_stack();
 }
 
+/* Speculation control using virtualized MSRs */
+static void __init spec_ctrl_setup_virtualized_msr(void)
+{
+	u64 msr_virt_enum, msr_mitigation_enum, msr_mitigation_ctrl;
+
+	if (!(x86_read_arch_cap_msr() & ARCH_CAP_VIRTUAL_ENUM))
+		return;
+
+	rdmsrl(MSR_VIRTUAL_ENUMERATION, msr_virt_enum);
+	if (!(msr_virt_enum & VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+		return;
+
+	rdmsrl(MSR_VIRTUAL_MITIGATION_ENUM, msr_mitigation_enum);
+	/* When retpoline is being used, request relevant hardware controls */
+	if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
+	    msr_mitigation_enum & MITI_ENUM_RETPOLINE_S_SUPPORT) {
+		rdmsrl(MSR_VIRTUAL_MITIGATION_CTRL, msr_mitigation_ctrl);
+		msr_mitigation_ctrl |= MITI_CTRL_RETPOLINE_S_USED;
+		wrmsrl(MSR_VIRTUAL_MITIGATION_CTRL, msr_mitigation_ctrl);
+	}
+}
+
 static void __init spectre_v2_select_mitigation(void)
 {
 	enum spectre_v2_mitigation_cmd cmd = spectre_v2_parse_cmdline();

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1510

Quoted 4 lines
arch/x86/kernel/cpu/bugs.c +1 -0
@@ -1485,6 +1507,8 @@ static void __init spectre_v2_select_mitigation(void)
 	    mode == SPECTRE_V2_RETPOLINE)
 		spec_ctrl_disable_kernel_rrsba();
 
+	spec_ctrl_setup_virtualized_msr();

I think this also needs to be called during secondary CPU initialization.

#10 lore raw

RE: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 124 lines

-----Original Message-----
From: Pawan Gupta <[email protected]>
Sent: Tuesday, December 13, 2022 4:24 AM
To: Zhang, Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Gao, Chao <[email protected]>; Paolo Bonzini <[email protected]>; Christopherson,, Sean <[email protected]>; H. Peter Anvin <[email protected]>; Dave Hansen <[email protected]>; Borislav Petkov <[email protected]>; Ingo Molnar <[email protected]>; Thomas Gleixner <[email protected]>
Subject: Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 24 lines · Zhang Chen

From: Pawan Gupta <[email protected]>

Guests that have different family/model than the host may not be aware of hardware mitigations(such as RRSBA_DIS_S) available on host. This is particularly true when guests migrate. To solve this problem Intel processors have added a virtual MSR interface through which guests can report their mitigation status and request VMM to deploy relevant hardware mitigations.

Use this virtualized MSR interface to request relevant hardware controls for retpoline mitigation.

---
 arch/x86/include/asm/msr-index.h | 23 +++++++++++++++++++++++
 arch/x86/kernel/cpu/bugs.c       | 24 ++++++++++++++++++++++++
 2 files changed, 47 insertions(+)
arch/x86/include/asm/msr-index.h +0 -0
index 1143ac9400c3..1166b472377c 100644

arch/x86/include/asm/msr-index.h line 165

Quoted 1 line
arch/x86/include/asm/msr-index.h +0 -0
@@ -165,6 +165,7 @@
 						 *

IA32_XAPIC_DISABLE_STATUS MSR

arch/x86/include/asm/msr-index.h

Quoted 3 lines
arch/x86/include/asm/msr-index.h +1 -0
 						 * supported
 						 */
+#define ARCH_CAP_VIRTUAL_ENUM		BIT(63)	/*

MSR_VIRTUAL_ENUMERATION supported */

arch/x86/include/asm/msr-index.h line 1064

Quoted 6 lines
arch/x86/include/asm/msr-index.h +0 -0
 #define MSR_IA32_FLUSH_CMD		0x0000010b
 #define L1D_FLUSH			BIT(0)	/*
@@ -1062,6 +1063,28 @@
 #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
 #define MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS (1ULL <<

29)

arch/x86/include/asm/msr-index.h

Quoted 6 lines
arch/x86/include/asm/msr-index.h +5 -0
 #define MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE   0x1F
+
+/* Intel virtual MSRs */
+#define MSR_VIRTUAL_ENUMERATION			0x50000000
+#define VIRT_ENUM_MITIGATION_CTRL_SUPPORT	BIT(0)	/*
+							 * Mitigation ctrl via

virtual

arch/x86/include/asm/msr-index.h

Quoted 5 lines
arch/x86/include/asm/msr-index.h +5 -0
+							 * MSRs supported
+							 */
+
+#define MSR_VIRTUAL_MITIGATION_ENUM		0x50000001
+#define MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT	BIT(0)	/* VMM

supports BHI_DIS_S */

arch/x86/include/asm/msr-index.h line 1076

Quoted 1 line
arch/x86/include/asm/msr-index.h +1 -0
+#define MITI_ENUM_RETPOLINE_S_SUPPORT		BIT(1)	/*

VMM supports RRSBA_DIS_S */

arch/x86/include/asm/msr-index.h

Quoted 4 lines
arch/x86/include/asm/msr-index.h +4 -0
+
+#define MSR_VIRTUAL_MITIGATION_CTRL		0x50000002
+#define MITI_CTRL_BHB_CLEAR_SEQ_S_USED		BIT(0)	/*
+							 * Request VMM to

deploy

arch/x86/include/asm/msr-index.h line 1081

Quoted 1 line
arch/x86/include/asm/msr-index.h +1 -0
+							 * BHI_DIS_S

mitigation

arch/x86/include/asm/msr-index.h

Quoted 3 lines
arch/x86/include/asm/msr-index.h +3 -0
+							 */
+#define MITI_CTRL_RETPOLINE_S_USED		BIT(1)	/*
+							 * Request VMM to

deploy

arch/x86/include/asm/msr-index.h line 1085

Quoted 1 line
arch/x86/include/asm/msr-index.h +1 -0
+							 * RRSBA_DIS_S

mitigation

arch/x86/kernel/cpu/bugs.c static void __init

Quoted 10 lines
arch/x86/include/asm/msr-index.h +2 -0
+							 */
+
 /* AMD-V MSRs */
 
 #define MSR_VM_CR                       0xc0010114
arch/x86/kernel/cpu/bugs.c +0 -0
index 3e3230cccaa7..a9e869f568ee 100644
@@ -1379,6 +1379,28 @@ static void __init

spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_

Quoted 7 lines
arch/x86/kernel/cpu/bugs.c +4 -0
 	dump_stack();
 }
 
+/* Speculation control using virtualized MSRs */ static void __init
+spec_ctrl_setup_virtualized_msr(void)
+{
+	u64 msr_virt_enum, msr_mitigation_enum, msr_mitigation_ctrl;

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_

Quoted 10 lines
arch/x86/kernel/cpu/bugs.c +10 -0
+
+	if (!(x86_read_arch_cap_msr() & ARCH_CAP_VIRTUAL_ENUM))
+		return;
+
+	rdmsrl(MSR_VIRTUAL_ENUMERATION, msr_virt_enum);
+	if (!(msr_virt_enum & VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+		return;
+
+	rdmsrl(MSR_VIRTUAL_MITIGATION_ENUM, msr_mitigation_enum);
+	/* When retpoline is being used, request relevant hardware controls

*/

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_

Quoted 3 lines
arch/x86/kernel/cpu/bugs.c +3 -0
+	if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
+	    msr_mitigation_enum & MITI_ENUM_RETPOLINE_S_SUPPORT) {
+		rdmsrl(MSR_VIRTUAL_MITIGATION_CTRL,

msr_mitigation_ctrl);

arch/x86/kernel/cpu/bugs.c static void __init · line 1400

Quoted 2 lines
arch/x86/kernel/cpu/bugs.c +2 -0
+		msr_mitigation_ctrl |= MITI_CTRL_RETPOLINE_S_USED;
+		wrmsrl(MSR_VIRTUAL_MITIGATION_CTRL,

msr_mitigation_ctrl);

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_ · line 1406

Quoted 6 lines
arch/x86/kernel/cpu/bugs.c +3 -0
+	}
+}
+
 static void __init spectre_v2_select_mitigation(void)
 {
 	enum spectre_v2_mitigation_cmd cmd = spectre_v2_parse_cmdline();

@@

Quoted 7 lines

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1510

Quoted 5 lines

-1485,6 +1507,8 @@ static void __init spectre_v2_select_mitigation(void)

arch/x86/kernel/cpu/bugs.c +1 -0
 	    mode == SPECTRE_V2_RETPOLINE)
 		spec_ctrl_disable_kernel_rrsba();
 
+	spec_ctrl_setup_virtualized_msr();

I think this also needs to be called during secondary CPU initialization.

Yes, we need setup virtual MSRs for each CPU.
I will add the " spec_ctrl_setup_virtualized_msr()" to the "start_secondary()" in the arch/x86/kernel/smpboot.c next version.

Thanks
Chen

#11 lore raw

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

In reply to Zhang Chen
Quoted 6 lines · Zhang Chen

From: Pawan Gupta <[email protected]>

Guests that have different family/model than the host may not be aware of hardware mitigations(such as RRSBA_DIS_S) available on host. This is particularly true when guests migrate. To solve this problem Intel processors have added a virtual MSR interface

Is there any actual "processor" support here? To me, this looks like Intel is foisting a paravirt interface on KVM and other hypervisors without collaborating with said hypervisors' developers and maintainers.

I get that some of the mitigations are vendor specific, but things like RETPOLINE aren't vendor specific. I haven't followed all of the mitigation stuff very closely, but I wouldn't be surprised if there are mitigations now or in the future that are common across architectures, e.g. arm64 and x86-64. Intel doing its own thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting multiple paravirt interfaces for very similar things, without having any control over the APIs. That's all kinds of backwards.

And having to wait for Intel to roll out new documentation when software inevitably comes up with some clever new mitigation doesn't exactly fill my heart with joy.

#12 lore raw

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 9 lines · Sean Christopherson
Quoted 6 lines · Zhang Chen

From: Pawan Gupta <[email protected]>

Guests that have different family/model than the host may not be aware of hardware mitigations(such as RRSBA_DIS_S) available on host. This is particularly true when guests migrate. To solve this problem Intel processors have added a virtual MSR interface

Is there any actual "processor" support here?

No.

Quoted 11 lines

To me, this looks like Intel is
foisting a paravirt interface on KVM and other hypervisors without collaborating with said hypervisors' developers and maintainers.

I get that some of the mitigations are vendor specific, but things like RETPOLINE aren't vendor specific. I haven't followed all of the mitigation stuff very closely, but I wouldn't be surprised if there are mitigations now or in the future that are common across architectures, e.g. arm64 and x86-64. Intel doing its own thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting multiple paravirt interfaces for very similar things, without having any control over the APIs. That's all kinds of backwards.

But if the interface is defined by KVM rather than Intel, it will likely end up with different interfaces for different VMMs, then Linux guest needs to support all of them. And KVM needs to implement Hyper-V's and Xen's interface to support Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and complicates KVM/Linux in a similar way as multiple paravirt interfaces.

The use case of this paravirt interface is specific to Intel CPU microarchitecture. Supporting multiple paravirt interfaces may not happen in the near future if there is no use case for AMD and arm64.

Quoted 3 lines

And having to wait for Intel to roll out new documentation when software inevitably comes up with some clever new mitigation doesn't exactly fill my heart with joy.

There should be some misunderstanding.

A bit for a software mitigation will be added if
1. the software mitigation works well on existing processors. And
2. there will be a new processor on which the software mitigation won't be fully effective (due to some microarchiecture change).

IOW, the new documentation comes along with a new microarchitecture (new processor) rather than software mitigations.

#13 lore raw

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 12 lines · Chao Gao
Quoted 10 lines · Sean Christopherson

To me, this looks like Intel is foisting a paravirt interface on KVM and other hypervisors without collaborating with said hypervisors' developers and maintainers.

I get that some of the mitigations are vendor specific, but things like RETPOLINE aren't vendor specific. I haven't followed all of the mitigation stuff very closely, but I wouldn't be surprised if there are mitigations now or in the future that are common across architectures, e.g. arm64 and x86-64. Intel doing its own thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting multiple paravirt interfaces for very similar things, without having any control over the APIs. That's all kinds of backwards.

Quoted 5 lines

But if the interface is defined by KVM rather than Intel, it will likely end up with different interfaces for different VMMs, then Linux guest needs to support all of them. And KVM needs to implement Hyper-V's and Xen's interface to support Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and complicates KVM/Linux in a similar way as multiple paravirt interfaces.

I never said the PV interfaces should be defined by KVM. I 100% agree that any one hypervisor defining its own interface will suffer the same problem.

I think having a PV interface for coordinating mitigations between host and guest is a great idea. What I don't like is tying the interface to "hardware" and defining the interface without even trying to collaborate with others.

Quoted 1 line

The use case of this paravirt interface is specific to Intel CPU microarchitecture.

Well yeah, because the interface was designed only to work for Intel CPUs.

Quoted 2 lines

Supporting multiple paravirt interfaces may not happen in the near future if there is no use case for AMD and arm64.

I'll take that bet. The vast majority of problems that are solved by PV interfaces are common to all architectures and vendors, e.g. steal time, PV spinlocks, async page faults, directed yield, confidential VMs (GHCB vs. GHCI), etc. I highly doubt Intel is the only hardware vendor that will ever benefit from paravirtualizing mitigations.

#14 lore raw

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Chao Gao
Quoted 10 lines · Sean Christopherson

To me, this looks like Intel is foisting a paravirt interface on KVM and other hypervisors without collaborating with said hypervisors' developers and maintainers.

I get that some of the mitigations are vendor specific, but things like RETPOLINE aren't vendor specific. I haven't followed all of the mitigation stuff very closely, but I wouldn't be surprised if there are mitigations now or in the future that are common across architectures, e.g. arm64 and x86-64. Intel doing its own thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting multiple paravirt interfaces for very similar things, without having any control over the APIs. That's all kinds of backwards.

Quoted 8 lines
Quoted 5 lines

But if the interface is defined by KVM rather than Intel, it will likely end up with different interfaces for different VMMs, then Linux guest needs to support all of them. And KVM needs to implement Hyper-V's and Xen's interface to support Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and complicates KVM/Linux in a similar way as multiple paravirt interfaces.

I never said the PV interfaces should be defined by KVM. I 100% agree that any one hypervisor defining its own interface will suffer the same problem.

I am thinking there are only two options:

1. Intel defines the interface.
2. Every VMM defines its own interface.

Any middle ground between the two options?

Quoted 3 lines

I think having a PV interface for coordinating mitigations between host and guest is a great idea. What I don't like is tying the interface to "hardware" and defining

Do you think something below is better than the intel-defined interface? add a new KVM_CAP_* and a KVM_FEATURE_* in hypervisor CPUID leaf to enumerate the interface. And add a new virtual MSR 0x4b564dxx for guest to report in-use software mitigations and assign one bit for each software mitigation. On MSR write emulation, call into vmx code to enable some hardware mitigations if the corresponding software mitigations are not effective on host.

I am afraid it is late to switch to above approach because e.g., if Hyper-V decides to support the intel-defined interface, KVM probably needs to support it for Hyper-V guests.

Can we emulate IA32_ARCH_CAPABILITIES for AMD guests so that AMD can re-use intel-defined interface?

Quoted 1 line

the interface without even trying to collaborate with others.

I will pass on this to related Intel folks and ask them to collaborate with KVM community on future PV interfaces.

#15 lore raw

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 23 lines · Chao Gao
Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Chao Gao
Quoted 10 lines · Sean Christopherson

To me, this looks like Intel is foisting a paravirt interface on KVM and other hypervisors without collaborating with said hypervisors' developers and maintainers.

I get that some of the mitigations are vendor specific, but things like RETPOLINE aren't vendor specific. I haven't followed all of the mitigation stuff very closely, but I wouldn't be surprised if there are mitigations now or in the future that are common across architectures, e.g. arm64 and x86-64. Intel doing its own thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting multiple paravirt interfaces for very similar things, without having any control over the APIs. That's all kinds of backwards.

Quoted 8 lines
Quoted 5 lines

But if the interface is defined by KVM rather than Intel, it will likely end up with different interfaces for different VMMs, then Linux guest needs to support all of them. And KVM needs to implement Hyper-V's and Xen's interface to support Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and complicates KVM/Linux in a similar way as multiple paravirt interfaces.

I never said the PV interfaces should be defined by KVM. I 100% agree that any one hypervisor defining its own interface will suffer the same problem.

Quoted 6 lines

I am thinking there are only two options:

1. Intel defines the interface.
2. Every VMM defines its own interface.

Any middle ground between the two options?

Work with other x86 hardware vendors to define a common interface? Ask hypervisor developers to define a common, extensible interface?

Maybe it turns out that it's impossible to abstract anything away and everything ends up being vendor-specific anyways, but not even trying to provide a common interace is extremely frustrating, especially since all this mitigation stuff has been going on for ~5 years. There's been plenty of time to establish relationships and points of contact.

Quoted 4 lines
Quoted 2 lines

I think having a PV interface for coordinating mitigations between host and guest is a great idea. What I don't like is tying the interface to "hardware" and defining

Do you think something below is better than the intel-defined interface?

No, KVM doing its own thing would only exacerbate the issue.

Quoted 9 lines

add a new KVM_CAP_* and a KVM_FEATURE_* in hypervisor CPUID leaf to enumerate the interface. And add a new virtual MSR 0x4b564dxx for guest to report in-use software mitigations and assign one bit for each software mitigation. On MSR write emulation, call into vmx code to enable some hardware mitigations if the corresponding software mitigations are not effective on host.

I am afraid it is late to switch to above approach because e.g., if Hyper-V decides to support the intel-defined interface, KVM probably needs to support it for Hyper-V guests.

Hence my frustration.

#16 lore raw

RE: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 59 lines

-----Original Message-----
From: Sean Christopherson <[email protected]>
Sent: Friday, December 23, 2022 2:32 AM
To: Gao, Chao <[email protected]>
Cc: Zhang, Chen <[email protected]>; [email protected]; linux- [email protected]; [email protected]; Pawan Gupta <[email protected]>; Paolo Bonzini <[email protected]>; H. Peter Anvin <[email protected]>; Dave Hansen <[email protected]>; Borislav Petkov <[email protected]>; Ingo Molnar <[email protected]>; Thomas Gleixner <[email protected]>
Subject: Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

Quoted 5 lines · Chao Gao
Quoted 4 lines · Sean Christopherson
Quoted 3 lines · Chao Gao
Quoted 2 lines · Sean Christopherson

To me, this looks like Intel is foisting a paravirt interface on KVM and other hypervisors without collaborating with said hypervisors'

developers and maintainers.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I get that some of the mitigations are vendor specific, but things like RETPOLINE aren't vendor specific. I haven't followed all of the mitigation stuff very closely, but I wouldn't be surprised if there are mitigations now or in the future that are common across architectures, e.g. arm64 and x86-64. Intel doing its own thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting multiple paravirt interfaces for very similar things, without

having any control over the APIs. That's all kinds of backwards.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

But if the interface is defined by KVM rather than Intel, it will likely end up with different interfaces for different VMMs, then Linux guest needs to support all of them. And KVM needs to implement Hyper-V's and Xen's interface to support Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and

complicates KVM/Linux in a similar way as multiple paravirt interfaces.

Quoted 3 lines
Quoted 3 lines

I never said the PV interfaces should be defined by KVM. I 100% agree that any one hypervisor defining its own interface will suffer the same

problem.

Quoted 7 lines

I am thinking there are only two options:

1. Intel defines the interface.
2. Every VMM defines its own interface.

Any middle ground between the two options?

Work with other x86 hardware vendors to define a common interface? Ask hypervisor developers to define a common, extensible interface?

Maybe it turns out that it's impossible to abstract anything away and everything ends up being vendor-specific anyways, but not even trying to provide a common interace is extremely frustrating, especially since all this mitigation stuff has been going on for ~5 years. There's been plenty of time to establish relationships and points of contact.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I think having a PV interface for coordinating mitigations between host and guest is a great idea. What I don't like is tying the interface to "hardware" and defining

Do you think something below is better than the intel-defined interface?

No, KVM doing its own thing would only exacerbate the issue.

Hi Sean,

Rethink about it and synced with Chao, we didn't think of a better solution than Intel-defined interface. If you have no more suggestions/comments here, please review the rest of patches from this series when you have time. I will try to address comments and push this series to RFC V2.

Thanks
Chen

Quoted 13 lines
Quoted 6 lines

add a new KVM_CAP_* and a KVM_FEATURE_* in hypervisor CPUID leaf to enumerate the interface. And add a new virtual MSR 0x4b564dxx for guest to report in-use software mitigations and assign one bit for each software mitigation. On MSR write emulation, call into vmx code to enable some hardware mitigations if the corresponding software

mitigations are not effective on host.

Quoted 4 lines

I am afraid it is late to switch to above approach because e.g., if Hyper-V decides to support the intel-defined interface, KVM probably needs to support it for Hyper-V guests.

Hence my frustration.

author
#17 lore raw

RFC PATCH 6/9 kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

In reply to Zhang Chen

Add the 63 bit in MSR_IA32_ARCH_CAPABILITIES for enable the virtual MSRs. Virtual MSRs can allow guests to notify VMM whether or not they are using specific software mitigation, allowing a VMM to enable there hardware control only where necessary.
As Intel spec defination, expose virtual MSR for guest.
Make guest have ability to check virtual MSR 0x50000000.

---
 arch/x86/kvm/vmx/vmx.c | 15 +++++++++++++++
 arch/x86/kvm/vmx/vmx.h |  1 +
 arch/x86/kvm/x86.c     | 16 +++++++++++++++-
 3 files changed, 31 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +15 -0
index 407061b369b4..6ed6b743be0e 100644
@@ -2001,6 +2001,12 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_IA32_DEBUGCTLMSR:
 		msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL);
 		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		if (!msr_info->host_initiated &&
+		    !(vcpu->arch.arch_capabilities & ARCH_CAP_VIRTUAL_ENUM))
+			return 1;
+		msr_info->data = vmx->msr_virtual_enumeration;
+		break;
 	default:
 	find_uret_msr:
 		msr = vmx_find_uret_msr(vmx, msr_info->index);
@@ -2375,6 +2381,15 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		}
 		ret = kvm_set_msr_common(vcpu, msr_info);
 		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		if (msr_info->host_initiated &&
+		    !(vcpu->arch.arch_capabilities & ARCH_CAP_VIRTUAL_ENUM))
+			return 1;
+		if (data & ~VIRT_ENUM_MITIGATION_CTRL_SUPPORT)
+			return 1;
+		vmx->msr_virtual_enumeration = data &
+						VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
+		break;
 
 	default:
 	find_uret_msr:
arch/x86/kvm/vmx/vmx.h +1 -0
index c5a41ae14237..fc873cf45f70 100644
@@ -298,6 +298,7 @@ struct vcpu_vmx {
 	 * IA32_SPEC_CTRL_MSR.
 	 */
 	u64		      spec_ctrl_mask;
+	u64		      msr_virtual_enumeration;
 	u32		      msr_ia32_umwait_control;
 
 	/*
arch/x86/kvm/x86.c +15 -1
index 2835bd796639..6be0a3f1281f 100644
@@ -1532,6 +1532,8 @@ static const u32 emulated_msrs_all[] = {
 	MSR_IA32_VMX_EPT_VPID_CAP,
 	MSR_IA32_VMX_VMFUNC,
 
+	MSR_VIRTUAL_ENUMERATION,
+
 	MSR_K7_HWCR,
 	MSR_KVM_POLL_CONTROL,
 };
@@ -1567,6 +1569,7 @@ static const u32 msr_based_features_all[] = {
 	MSR_IA32_UCODE_REV,
 	MSR_IA32_ARCH_CAPABILITIES,
 	MSR_IA32_PERF_CAPABILITIES,
+	MSR_VIRTUAL_ENUMERATION,
 };
 
 static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all)];
@@ -1588,7 +1591,8 @@ static unsigned int num_msr_based_features;
 	 ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO | ARCH_CAP_MDS_NO | \
 	 ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \
 	 ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \
-	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO)
+	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | \
+	 ARCH_CAP_VIRTUAL_ENUM)
 
 static u64 kvm_get_arch_capabilities(void)
 {
@@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
 	 */
 	data |= ARCH_CAP_PSCHANGE_MC_NO;
 
+	/*
+	 * Virtual MSRs can allow guests to notify VMM whether or not
+	 * they are using specific software mitigation, allowing a VMM
+	 * to enable there hardware control only where necessary.
+	 */
+	data |= ARCH_CAP_VIRTUAL_ENUM;
+
 	/*
 	 * If we're doing cache flushes (either "always" or "cond")
 	 * we will do one whenever the guest does a vmlaunch/vmresume.
@@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct kvm_msr_entry *msr)
 	case MSR_IA32_UCODE_REV:
 		rdmsrl_safe(msr->index, &msr->data);
 		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
+		break;
 	default:
 		return static_call(kvm_x86_get_msr_feature)(msr);
 	}
Signature · 1 line
2.25.1
#18 lore raw

Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

Quoted 91 lines · Zhang Chen

Add the 63 bit in MSR_IA32_ARCH_CAPABILITIES for enable the virtual MSRs. Virtual MSRs can allow guests to notify VMM whether or not they are using specific software mitigation, allowing a VMM to enable there hardware control only where necessary.
As Intel spec defination, expose virtual MSR for guest.
Make guest have ability to check virtual MSR 0x50000000.

---
  arch/x86/kvm/vmx/vmx.c | 15 +++++++++++++++
  arch/x86/kvm/vmx/vmx.h |  1 +
  arch/x86/kvm/x86.c     | 16 +++++++++++++++-
  3 files changed, 31 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +15 -0
index 407061b369b4..6ed6b743be0e 100644
@@ -2001,6 +2001,12 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
  	case MSR_IA32_DEBUGCTLMSR:
  		msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL);
  		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		if (!msr_info->host_initiated &&
+		    !(vcpu->arch.arch_capabilities & ARCH_CAP_VIRTUAL_ENUM))
+			return 1;
+		msr_info->data = vmx->msr_virtual_enumeration;
+		break;
  	default:
  	find_uret_msr:
  		msr = vmx_find_uret_msr(vmx, msr_info->index);
@@ -2375,6 +2381,15 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
  		}
  		ret = kvm_set_msr_common(vcpu, msr_info);
  		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		if (msr_info->host_initiated &&
+		    !(vcpu->arch.arch_capabilities & ARCH_CAP_VIRTUAL_ENUM))
+			return 1;
+		if (data & ~VIRT_ENUM_MITIGATION_CTRL_SUPPORT)
+			return 1;
+		vmx->msr_virtual_enumeration = data &
+						VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
+		break;
 
  	default:
  	find_uret_msr:
arch/x86/kvm/vmx/vmx.h +1 -0
index c5a41ae14237..fc873cf45f70 100644
@@ -298,6 +298,7 @@ struct vcpu_vmx {
  	 * IA32_SPEC_CTRL_MSR.
  	 */
  	u64		      spec_ctrl_mask;
+	u64		      msr_virtual_enumeration;
  	u32		      msr_ia32_umwait_control;
 
  	/*
arch/x86/kvm/x86.c +5 -1
index 2835bd796639..6be0a3f1281f 100644
@@ -1532,6 +1532,8 @@ static const u32 emulated_msrs_all[] = {
  	MSR_IA32_VMX_EPT_VPID_CAP,
  	MSR_IA32_VMX_VMFUNC,
 
+	MSR_VIRTUAL_ENUMERATION,
+
  	MSR_K7_HWCR,
  	MSR_KVM_POLL_CONTROL,
  };
@@ -1567,6 +1569,7 @@ static const u32 msr_based_features_all[] = {
  	MSR_IA32_UCODE_REV,
  	MSR_IA32_ARCH_CAPABILITIES,
  	MSR_IA32_PERF_CAPABILITIES,
+	MSR_VIRTUAL_ENUMERATION,
  };
 
  static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all)];
@@ -1588,7 +1591,8 @@ static unsigned int num_msr_based_features;
  	 ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO | ARCH_CAP_MDS_NO | \
  	 ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \
  	 ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \
-	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO)
+	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | \
+	 ARCH_CAP_VIRTUAL_ENUM)
 
  static u64 kvm_get_arch_capabilities(void)
  {

arch/x86/kvm/x86.c static u64 kvm_get_arch_capabilities(void) · line 1619

Quoted 9 lines
arch/x86/kvm/x86.c +6 -0
@@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
  	 */
  	data |= ARCH_CAP_PSCHANGE_MC_NO;
 
+	/*
+	 * Virtual MSRs can allow guests to notify VMM whether or not
+	 * they are using specific software mitigation, allowing a VMM
+	 * to enable there hardware control only where necessary.
+	 */
+	data |= ARCH_CAP_VIRTUAL_ENUM;

IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires platform support.

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct kvm_msr_entry *msr) · line 1672

Quoted 10 lines
arch/x86/kvm/x86.c +3 -0
+
  	/*
  	 * If we're doing cache flushes (either "always" or "cond")
  	 * we will do one whenever the guest does a vmlaunch/vmresume.
@@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct kvm_msr_entry *msr)
  	case MSR_IA32_UCODE_REV:
  		rdmsrl_safe(msr->index, &msr->data);
  		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;

Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the feature.

Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
+		break;
  	default:
  		return static_call(kvm_x86_get_msr_feature)(msr);
  	}
#19 lore raw

RE: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

Quoted 122 lines

-----Original Message-----
From: Yang, Weijiang <[email protected]>
Sent: Wednesday, December 21, 2022 12:04 PM
To: Zhang, Chen <[email protected]>
Cc: Gao, Chao <[email protected]>; Pawan Gupta <[email protected]>; Paolo Bonzini <[email protected]>; Christopherson,, Sean <[email protected]>; H. Peter Anvin <[email protected]>; Dave Hansen <[email protected]>; Borislav Petkov <[email protected]>; Ingo Molnar <[email protected]>; Thomas Gleixner <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

arch/x86/kvm/vmx/vmx.c static int vmx_get_msr(struct kvm_vcpu *vcpu,

Quoted 19 lines · Zhang Chen

Add the 63 bit in MSR_IA32_ARCH_CAPABILITIES for enable the virtual MSRs. Virtual MSRs can allow guests to notify VMM whether or not they are using specific software mitigation, allowing a VMM to enable there hardware control only where necessary.
As Intel spec defination, expose virtual MSR for guest.
Make guest have ability to check virtual MSR 0x50000000.

---
  arch/x86/kvm/vmx/vmx.c | 15 +++++++++++++++
  arch/x86/kvm/vmx/vmx.h |  1 +
  arch/x86/kvm/x86.c     | 16 +++++++++++++++-
  3 files changed, 31 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c index renamed from arch/x86/kvm/vmx/vmx.c +0 -0

407061b369b4..6ed6b743be0e 100644

arch/x86/kvm/vmx/vmx.c +0 -0
@@ -2001,6 +2001,12 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu,

struct msr_data *msr_info)

arch/x86/kvm/vmx/vmx.c static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)

Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +3 -0
  	case MSR_IA32_DEBUGCTLMSR:
  		msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL);
  		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		if (!msr_info->host_initiated &&
+		    !(vcpu->arch.arch_capabilities &

ARCH_CAP_VIRTUAL_ENUM))

arch/x86/kvm/vmx/vmx.c static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)

Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +3 -0
+			return 1;
+		msr_info->data = vmx->msr_virtual_enumeration;
+		break;
  	default:
  	find_uret_msr:
  		msr = vmx_find_uret_msr(vmx, msr_info->index); @@ -2375,6

+2381,15

Quoted 1 line

@@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data

*msr_info)

arch/x86/kvm/vmx/vmx.c static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)

Quoted 6 lines
  		}
  		ret = kvm_set_msr_common(vcpu, msr_info);
  		break;
arch/x86/kvm/vmx/vmx.c +3 -0
+	case MSR_VIRTUAL_ENUMERATION:
+		if (msr_info->host_initiated &&
+		    !(vcpu->arch.arch_capabilities &

ARCH_CAP_VIRTUAL_ENUM))

arch/x86/kvm/vmx/vmx.c static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)

Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +5 -0
+			return 1;
+		if (data & ~VIRT_ENUM_MITIGATION_CTRL_SUPPORT)
+			return 1;
+		vmx->msr_virtual_enumeration = data &
+
	VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
Quoted 38 lines
arch/x86/kvm/vmx/vmx.c +1 -0
+		break;
 
  	default:
  	find_uret_msr:
arch/x86/kvm/vmx/vmx.h index renamed from arch/x86/kvm/vmx/vmx.h +0 -0

c5a41ae14237..fc873cf45f70 100644

arch/x86/kvm/vmx/vmx.h +1 -0
@@ -298,6 +298,7 @@ struct vcpu_vmx {
  	 * IA32_SPEC_CTRL_MSR.
  	 */
  	u64		      spec_ctrl_mask;
+	u64		      msr_virtual_enumeration;
  	u32		      msr_ia32_umwait_control;
 
  	/*
arch/x86/kvm/x86.c index renamed from arch/x86/kvm/x86.c +0 -0

2835bd796639..6be0a3f1281f 100644

arch/x86/kvm/x86.c +3 -0
@@ -1532,6 +1532,8 @@ static const u32 emulated_msrs_all[] = {
  	MSR_IA32_VMX_EPT_VPID_CAP,
  	MSR_IA32_VMX_VMFUNC,
 
+	MSR_VIRTUAL_ENUMERATION,
+
  	MSR_K7_HWCR,
  	MSR_KVM_POLL_CONTROL,
  };
@@ -1567,6 +1569,7 @@ static const u32 msr_based_features_all[] = {
  	MSR_IA32_UCODE_REV,
  	MSR_IA32_ARCH_CAPABILITIES,
  	MSR_IA32_PERF_CAPABILITIES,
+	MSR_VIRTUAL_ENUMERATION,
  };
 
  static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all)];

arch/x86/kvm/x86.c static unsigned int num_msr_based_features; · line 1591

Quoted 1 line
arch/x86/kvm/x86.c +0 -0
@@ -1588,7 +1591,8 @@ static unsigned int num_msr_based_features;
  	 ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO |

ARCH_CAP_MDS_NO | \

arch/x86/kvm/x86.c static unsigned int num_msr_based_features; · line 1592

Quoted 1 line
arch/x86/kvm/x86.c +0 -0
  	 ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR |

ARCH_CAP_TAA_NO | \

arch/x86/kvm/x86.c static unsigned int num_msr_based_features; · line 1593

Quoted 1 line
arch/x86/kvm/x86.c +0 -0
  	 ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO |

ARCH_CAP_PSDP_NO | \

arch/x86/kvm/x86.c static unsigned int num_msr_based_features; · line 1594

Quoted 2 lines
arch/x86/kvm/x86.c +1 -1
-	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO)
+	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO |

\

Quoted 8 lines
arch/x86/kvm/x86.c +1 -0
+	 ARCH_CAP_VIRTUAL_ENUM)
 
  static u64 kvm_get_arch_capabilities(void)
  {
@@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
  	 */
  	data |= ARCH_CAP_PSCHANGE_MC_NO;
 
Quoted 10 lines

arch/x86/kvm/x86.c static u64 kvm_get_arch_capabilities(void) · line 1619

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+	/*
+	 * Virtual MSRs can allow guests to notify VMM whether or not
+	 * they are using specific software mitigation, allowing a VMM
+	 * to enable there hardware control only where necessary.
+	 */
+	data |= ARCH_CAP_VIRTUAL_ENUM;

IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires platform support.

Intel defined the virtual MSRs for software mitigations for all platforms. KVM should be unconditionally opened it for the software mitigation in migration pools. For example migration from the old platform to the new platform.
Please check the Software Mitigations in Migration Pools section in documents:
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html

Quoted 8 lines

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct

Quoted 7 lines
arch/x86/kvm/x86.c +1 -0
+
  	/*
  	 * If we're doing cache flushes (either "always" or "cond")
  	 * we will do one whenever the guest does a vmlaunch/vmresume.
@@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct

kvm_msr_entry *msr)

Quoted 9 lines

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct kvm_msr_entry *msr) · line 1672

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
  	case MSR_IA32_UCODE_REV:
  		rdmsrl_safe(msr->index, &msr->data);
  		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;

Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the feature.

Refer to the above comments.

Thanks
Chen

Quoted 6 lines
Quoted 6 lines
arch/x86/kvm/x86.c +1 -0
+		break;
  	default:
  		return static_call(kvm_x86_get_msr_feature)(msr);
  	}
#20 lore raw

Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

On 12/29/2022 10:58 AM, Zhang, Chen wrote:

[...]

Quoted 18 lines
Quoted 9 lines

\

Quoted 8 lines
arch/x86/kvm/x86.c +1 -0
+	 ARCH_CAP_VIRTUAL_ENUM)
 
   static u64 kvm_get_arch_capabilities(void)
   {
@@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
   	 */
   	data |= ARCH_CAP_PSCHANGE_MC_NO;
 
Quoted 9 lines

arch/x86/kvm/x86.c static u64 kvm_get_arch_capabilities(void) · line 1619

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+	/*
+	 * Virtual MSRs can allow guests to notify VMM whether or not
+	 * they are using specific software mitigation, allowing a VMM
+	 * to enable there hardware control only where necessary.
+	 */
+	data |= ARCH_CAP_VIRTUAL_ENUM;

IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires platform support.

Quoted 5 lines

Intel defined the virtual MSRs for software mitigations for all platforms. KVM should be unconditionally opened it for the software mitigation in migration pools. For example migration from the old platform to the new platform.
Please check the Software Mitigations in Migration Pools section in documents:
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html

If this series running on old platforms, how VMM can set specific vmcs fields,

e.g., “virtualize IA32_SPEC_CTRL” VM-execution control, to mitigate guest issues?

Quoted 26 lines
Quoted 8 lines

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct

Quoted 7 lines
arch/x86/kvm/x86.c +1 -0
+
   	/*
   	 * If we're doing cache flushes (either "always" or "cond")
   	 * we will do one whenever the guest does a vmlaunch/vmresume.
@@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct

kvm_msr_entry *msr)

Quoted 8 lines

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct kvm_msr_entry *msr) · line 1672

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
   	case MSR_IA32_UCODE_REV:
   		rdmsrl_safe(msr->index, &msr->data);
   		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;

Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the feature.

Refer to the above comments.

Thanks
Chen

Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/x86.c +1 -0
+		break;
   	default:
   		return static_call(kvm_x86_get_msr_feature)(msr);
   	}
#21 lore raw

RE: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

Quoted 45 lines

-----Original Message-----
From: Yang, Weijiang <[email protected]>
Sent: Thursday, December 29, 2022 3:03 PM
To: Zhang, Chen <[email protected]>
Cc: Gao, Chao <[email protected]>; Pawan Gupta <[email protected]>; Paolo Bonzini <[email protected]>; Christopherson,, Sean <[email protected]>; H. Peter Anvin <[email protected]>; Dave Hansen <[email protected]>; Borislav Petkov <[email protected]>; Ingo Molnar <[email protected]>; Thomas Gleixner <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

On 12/29/2022 10:58 AM, Zhang, Chen wrote:

[...]

Quoted 18 lines
Quoted 9 lines

\

Quoted 8 lines
arch/x86/kvm/x86.c +1 -0
+	 ARCH_CAP_VIRTUAL_ENUM)
 
   static u64 kvm_get_arch_capabilities(void)
   {
@@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
   	 */
   	data |= ARCH_CAP_PSCHANGE_MC_NO;
 
Quoted 9 lines

arch/x86/kvm/x86.c static u64 kvm_get_arch_capabilities(void) · line 1619

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+	/*
+	 * Virtual MSRs can allow guests to notify VMM whether or not
+	 * they are using specific software mitigation, allowing a VMM
+	 * to enable there hardware control only where necessary.
+	 */
+	data |= ARCH_CAP_VIRTUAL_ENUM;

IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires platform support.

Quoted 2 lines

Intel defined the virtual MSRs for software mitigations for all platforms. KVM should be unconditionally opened it for the software mitigation in

migration pools.

Quoted 2 lines

For example migration from the old platform to the new platform. Please check the Software Mitigations in Migration Pools section in

documents:

Quoted 8 lines
Quoted 3 lines

https://www.intel.com/content/www/us/en/developer/articles/technical/s
oftware-security-guidance/technical-documentation/branch-history-injec tion.html

If this series running on old platforms, how VMM can set specific vmcs fields,

e.g., “virtualize IA32_SPEC_CTRL” VM-execution control, to mitigate guest issues?

Enable the virtual MSRs does not means to enable the “virtualize IA32_SPEC_CTRL”. KVM will check "cpu_has_virt_spec_ctrl()" before set specific VMCS.

Thanks
Chen

Quoted 28 lines
Quoted 28 lines
Quoted 10 lines

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct

Quoted 9 lines
arch/x86/kvm/x86.c +1 -0
+
   	/*
   	 * If we're doing cache flushes (either "always" or "cond")
   	 * we will do one whenever the guest does a vmlaunch/vmresume.
@@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct

kvm_msr_entry *msr)

Quoted 8 lines

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct kvm_msr_entry *msr) · line 1672

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
   	case MSR_IA32_UCODE_REV:
   		rdmsrl_safe(msr->index, &msr->data);
   		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;

Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the feature.

Refer to the above comments.

Thanks
Chen

Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/x86.c +1 -0
+		break;
   	default:
   		return static_call(kvm_x86_get_msr_feature)(msr);
   	}
#22 lore raw

Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

Quoted 46 lines · Chen
Quoted 46 lines

-----Original Message-----
From: Yang, Weijiang <[email protected]>
Sent: Thursday, December 29, 2022 3:03 PM
To: Zhang, Chen <[email protected]>
Cc: Gao, Chao <[email protected]>; Pawan Gupta <[email protected]>; Paolo Bonzini <[email protected]>; Christopherson,, Sean <[email protected]>; H. Peter Anvin <[email protected]>; Dave Hansen <[email protected]>; Borislav Petkov <[email protected]>; Ingo Molnar <[email protected]>; Thomas Gleixner <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

On 12/29/2022 10:58 AM, Zhang, Chen wrote:

[...]

Quoted 17 lines
Quoted 9 lines

\

Quoted 8 lines
arch/x86/kvm/x86.c +1 -0
+	 ARCH_CAP_VIRTUAL_ENUM)
 
    static u64 kvm_get_arch_capabilities(void)
    {
@@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
    	 */
    	data |= ARCH_CAP_PSCHANGE_MC_NO;
 
Quoted 8 lines

arch/x86/kvm/x86.c static u64 kvm_get_arch_capabilities(void) · line 1619

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+	/*
+	 * Virtual MSRs can allow guests to notify VMM whether or not
+	 * they are using specific software mitigation, allowing a VMM
+	 * to enable there hardware control only where necessary.
+	 */
+	data |= ARCH_CAP_VIRTUAL_ENUM;

IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires platform support.

Quoted 2 lines

Intel defined the virtual MSRs for software mitigations for all platforms. KVM should be unconditionally opened it for the software mitigation in

migration pools.

Quoted 2 lines

For example migration from the old platform to the new platform. Please check the Software Mitigations in Migration Pools section in

documents:

Quoted 3 lines

https://www.intel.com/content/www/us/en/developer/articles/technical/s
oftware-security-guidance/technical-documentation/branch-history-injec tion.html

Quoted 6 lines
Quoted 4 lines

If this series running on old platforms, how VMM can set specific vmcs fields,

e.g., “virtualize IA32_SPEC_CTRL” VM-execution control, to mitigate guest issues?

Enable the virtual MSRs does not means to enable the “virtualize IA32_SPEC_CTRL”. KVM will check "cpu_has_virt_spec_ctrl()" before set specific VMCS.

What I'm concerned is, if the feature is not supported on some platforms, unconditionally

exposing to guest would make it wrongly "think"  the feature is supported on HW, but actually

WMM can do nothing to mitigate issues. Read through the patch set, looks like guest cannot

even get any errors during feature configuration in this case. So I suggest add more

sanity checks. Did I  miss something?

Quoted 27 lines

Thanks
Chen

Quoted 23 lines
Quoted 23 lines
Quoted 7 lines

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct

Quoted 6 lines
arch/x86/kvm/x86.c +1 -0
+
    	/*
    	 * If we're doing cache flushes (either "always" or "cond")
    	 * we will do one whenever the guest does a vmlaunch/vmresume.
@@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct

kvm_msr_entry *msr)

Quoted 7 lines

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct kvm_msr_entry *msr) · line 1672

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
    	case MSR_IA32_UCODE_REV:
    		rdmsrl_safe(msr->index, &msr->data);
    		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;

Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the feature.

Refer to the above comments.

Thanks
Chen

Quoted 4 lines
Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
+		break;
    	default:
    		return static_call(kvm_x86_get_msr_feature)(msr);
    	}
#23 lore raw

RE: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

Quoted 83 lines

-----Original Message-----
From: Yang, Weijiang <[email protected]>
Sent: Thursday, December 29, 2022 4:39 PM
To: Zhang, Chen <[email protected]>
Cc: Gao, Chao <[email protected]>; Pawan Gupta <[email protected]>; Paolo Bonzini <[email protected]>; Christopherson,, Sean <[email protected]>; H. Peter Anvin <[email protected]>; Dave Hansen <[email protected]>; Borislav Petkov <[email protected]>; Ingo Molnar <[email protected]>; Thomas Gleixner <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

Quoted 49 lines · Chen
Quoted 49 lines

-----Original Message-----
From: Yang, Weijiang <[email protected]>
Sent: Thursday, December 29, 2022 3:03 PM
To: Zhang, Chen <[email protected]>
Cc: Gao, Chao <[email protected]>; Pawan Gupta <[email protected]>; Paolo Bonzini <[email protected]>; Christopherson,, Sean <[email protected]>; H. Peter Anvin <[email protected]>; Dave Hansen <[email protected]>; Borislav Petkov <[email protected]>; Ingo Molnar <[email protected]>; Thomas Gleixner <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

On 12/29/2022 10:58 AM, Zhang, Chen wrote:

[...]

Quoted 17 lines
Quoted 9 lines

\

Quoted 8 lines
arch/x86/kvm/x86.c +1 -0
+	 ARCH_CAP_VIRTUAL_ENUM)
 
    static u64 kvm_get_arch_capabilities(void)
    {
@@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
    	 */
    	data |= ARCH_CAP_PSCHANGE_MC_NO;
 
Quoted 8 lines

arch/x86/kvm/x86.c static u64 kvm_get_arch_capabilities(void) · line 1619

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+	/*
+	 * Virtual MSRs can allow guests to notify VMM whether or not
+	 * they are using specific software mitigation, allowing a VMM
+	 * to enable there hardware control only where necessary.
+	 */
+	data |= ARCH_CAP_VIRTUAL_ENUM;

IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires platform support.

Quoted 3 lines

Intel defined the virtual MSRs for software mitigations for all platforms. KVM should be unconditionally opened it for the software mitigation in

migration pools.

Quoted 2 lines

For example migration from the old platform to the new platform. Please check the Software Mitigations in Migration Pools section in

documents:

Quoted 5 lines

https://www.intel.com/content/www/us/en/developer/articles/technical
/s oftware-security-guidance/technical-documentation/branch-history-inj ec
tion.html

Quoted 6 lines
Quoted 5 lines

If this series running on old platforms, how VMM can set specific vmcs fields,

e.g., “virtualize IA32_SPEC_CTRL” VM-execution control, to mitigate guest issues?

Enable the virtual MSRs does not means to enable the “virtualize

IA32_SPEC_CTRL”.

Quoted 1 line

KVM will check "cpu_has_virt_spec_ctrl()" before set specific VMCS.

What I'm concerned is, if the feature is not supported on some platforms, unconditionally

exposing to guest would make it wrongly "think"  the feature is supported on HW, but actually

Quoted 7 lines

WMM can do nothing to mitigate issues. Read through the patch set, looks like guest cannot

even get any errors during feature configuration in this case. So I suggest add more

sanity checks. Did I  miss something?

Understand your concern, but I think no need to concern about it. Although we have unconditionally turned on virtual MSRs in all platforms, The detailed virtual MSR bits still can tell guest what feature is supported on HW. QEMU will setup the MSR_VIRTUAL_MITIGATION_ENUM(0x50000001) bits
Like the BHB_CLEAR_SEQ_S_SUPPORT/RETPOLINE_S_SUPPORT before boot VMs.

Thanks
Chen

Quoted 30 lines
Quoted 11 lines

Thanks
Chen

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/x86.c +1 -0
+
    	/*
    	 * If we're doing cache flushes (either "always" or "cond")
    	 * we will do one whenever the guest does a

vmlaunch/vmresume.

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 9 lines

Review static int kvm_get_msr_feature(struct

Quoted 1 line
(quoted diff) +0 -0
@@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct

kvm_msr_entry *msr)

arch/x86/kvm/x86.c static int kvm_get_msr_feature(struct kvm_msr_entry *msr) · line 1672

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
    	case MSR_IA32_UCODE_REV:
    		rdmsrl_safe(msr->index, &msr->data);
    		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;

Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the feature.

Refer to the above comments.

Thanks
Chen

Quoted 4 lines
Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
+		break;
    	default:
    		return static_call(kvm_x86_get_msr_feature)(msr);
    	}
author
#24 lore raw

RFC PATCH 7/9 kvm/x86: Add MSR_VIRTUAL_MITIGATION_ENUM/CTRL emulation

In reply to Zhang Chen

Introduce Intel virtual MSR_VIRTUAL_MITIGATION_ENUM(0x50000001) and MSR_VIRTUAL_MITIGATION_CTRL(0x50000002).
The MSR_VIRTUAL_MITIGATION_ENUM to tell guest about supported mitigations and enable the MSR_VIRTUAL_MITIGATION_CTRL virtual MSRs for guest, VMM will help to setup virtual spec ctrl mask for SPEC_CTRL_RRSBA_DIS_S, SPEC_CTRL_BHI_DIS_S as guest's needs.

---
 arch/x86/kvm/vmx/vmx.h | 16 ++++++++++++++++
 arch/x86/kvm/x86.c     |  7 +++++++
 2 files changed, 23 insertions(+)
arch/x86/kvm/vmx/vmx.h +16 -0
index fc873cf45f70..6abda05cc426 100644
@@ -29,6 +29,10 @@
 #endif
 
 #define MAX_NR_LOADSTORE_MSRS	8
+#define MITI_ENUM_SUPPORTED	(MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT |          \
+				MITI_ENUM_RETPOLINE_S_SUPPORT)
+#define MITI_CTRL_USED		(MITI_CTRL_BHB_CLEAR_SEQ_S_USED |                 \
+				MITI_CTRL_RETPOLINE_S_USED)
 
 struct vmx_msrs {
 	unsigned int		nr;
@@ -301,6 +305,18 @@ struct vcpu_vmx {
 	u64		      msr_virtual_enumeration;
 	u32		      msr_ia32_umwait_control;
 
+	/*
+	 * Guest read only, Only available if MITIGATION_CTRL_SUPPORT
+	 * is enumerated.
+	 */
+	u64		      msr_virtual_mitigation_enum;
+
+	/*
+	 * Read/Write, Only available if MITIGATION_CTRL_SUPPORT
+	 * is enumerated.
+	 */
+	u64		      msr_virtual_mitigation_ctrl;
+
 	/*
 	 * loaded_vmcs points to the VMCS currently used in this vcpu. For a
 	 * non-nested (L1) guest, it always points to vmcs01. For a nested
arch/x86/kvm/x86.c +7 -0
index 6be0a3f1281f..f6c314def6a8 100644
@@ -1533,6 +1533,8 @@ static const u32 emulated_msrs_all[] = {
 	MSR_IA32_VMX_VMFUNC,
 
 	MSR_VIRTUAL_ENUMERATION,
+	MSR_VIRTUAL_MITIGATION_ENUM,
+	MSR_VIRTUAL_MITIGATION_CTRL,
 
 	MSR_K7_HWCR,
 	MSR_KVM_POLL_CONTROL,
@@ -1570,6 +1572,7 @@ static const u32 msr_based_features_all[] = {
 	MSR_IA32_ARCH_CAPABILITIES,
 	MSR_IA32_PERF_CAPABILITIES,
 	MSR_VIRTUAL_ENUMERATION,
+	MSR_VIRTUAL_MITIGATION_ENUM,
 };
 
 static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all)];
@@ -1671,6 +1674,10 @@ static int kvm_get_msr_feature(struct kvm_msr_entry *msr)
 	case MSR_VIRTUAL_ENUMERATION:
 		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
 		break;
+	case MSR_VIRTUAL_MITIGATION_ENUM:
+		msr->data = MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT |
+			    MITI_ENUM_RETPOLINE_S_SUPPORT;
+		break;
 	default:
 		return static_call(kvm_x86_get_msr_feature)(msr);
 	}
Signature · 1 line
2.25.1
author
#25 lore raw

RFC PATCH 8/9 x86/kvm/vmx: Initialize SPEC_CTRL MASK for RRSBA

In reply to Zhang Chen

VMMs can address mitigations issues in migration pool by applying the needed controls whenever the guest is operating on a newer processor. If a guest is using retpoline to mitigate intra-mode BTI in CPL0, the VMM can set RRSBA_DIS_S when the guest runs on hardware which enumerates RRSBA.

---
 arch/x86/kvm/vmx/vmx.c | 57 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 56 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +56 -1
index 6ed6b743be0e..fb0f3b1639b9 100644
@@ -2007,6 +2007,20 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 			return 1;
 		msr_info->data = vmx->msr_virtual_enumeration;
 		break;
+	case MSR_VIRTUAL_MITIGATION_ENUM:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		msr_info->data = vmx->msr_virtual_mitigation_enum;
+		break;
+	case MSR_VIRTUAL_MITIGATION_CTRL:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		msr_info->data = vmx->msr_virtual_mitigation_ctrl;
+		break;
 	default:
 	find_uret_msr:
 		msr = vmx_find_uret_msr(vmx, msr_info->index);
@@ -2056,7 +2070,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	struct vmx_uret_msr *msr;
 	int ret = 0;
 	u32 msr_index = msr_info->index;
-	u64 data = msr_info->data;
+	u64 data = msr_info->data, arch_msr;
 	u32 index;
 
 	switch (msr_index) {
@@ -2390,6 +2404,46 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		vmx->msr_virtual_enumeration = data &
 						VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
 		break;
+	case MSR_VIRTUAL_MITIGATION_ENUM:
+		if (msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		if (data & ~MITI_ENUM_SUPPORTED)
+			return 1;
+		vmx->msr_virtual_mitigation_enum = data;
+		break;
+	case MSR_VIRTUAL_MITIGATION_CTRL:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		if (data & ~MITI_CTRL_USED)
+			return 1;
+
+		if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES))
+			rdmsrl(MSR_IA32_ARCH_CAPABILITIES, arch_msr);
+
+		if (data & MITI_CTRL_RETPOLINE_S_USED &&
+		    boot_cpu_has(X86_FEATURE_RRSBA_CTRL) &&
+		    arch_msr & ARCH_CAP_RRSBA)
+			vmx->spec_ctrl_mask |= SPEC_CTRL_RRSBA_DIS_S;
+		else
+			vmx->spec_ctrl_mask &= ~SPEC_CTRL_RRSBA_DIS_S;
+
+		if (cpu_has_virt_spec_ctrl()) {
+			vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
+		} else if (vmx->spec_ctrl_mask) {
+			pr_err_once("Virtual spec ctrl is missing. Cannot keep "
+					"bits in %llx always set\n",
+					vmx->spec_ctrl_mask);
+			vmx->spec_ctrl_mask = 0;
+		}
+
+		vmx->spec_ctrl = vmx->spec_ctrl | vmx->spec_ctrl_mask;
+
+		vmx->msr_virtual_mitigation_ctrl = data;
+		break;
 
 	default:
 	find_uret_msr:
@@ -4774,6 +4828,7 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
 
 	vmx->rmode.vm86_active = 0;
 
+	vmx->msr_virtual_mitigation_ctrl = 0;
 	if (cpu_has_virt_spec_ctrl()) {
 		vmx->spec_ctrl_mask = 0;
 		vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
Signature · 1 line
2.25.1
#26 lore raw

Re: [RFC PATCH 8/9] x86/kvm/vmx: Initialize SPEC_CTRL MASK for RRSBA

Quoted 67 lines · Zhang Chen

VMMs can address mitigations issues in migration pool by applying the needed controls whenever the guest is operating on a newer processor. If a guest is using retpoline to mitigate intra-mode BTI in CPL0, the VMM can set RRSBA_DIS_S when the guest runs on hardware which enumerates RRSBA.

---
arch/x86/kvm/vmx/vmx.c | 57 +++++++++++++++++++++++++++++++++++++++++-

1 file changed, 56 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +32 -1
index 6ed6b743be0e..fb0f3b1639b9 100644
@@ -2007,6 +2007,20 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 			return 1;
 		msr_info->data = vmx->msr_virtual_enumeration;
 		break;
+	case MSR_VIRTUAL_MITIGATION_ENUM:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		msr_info->data = vmx->msr_virtual_mitigation_enum;
+		break;
+	case MSR_VIRTUAL_MITIGATION_CTRL:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		msr_info->data = vmx->msr_virtual_mitigation_ctrl;
+		break;
 	default:
 	find_uret_msr:
 		msr = vmx_find_uret_msr(vmx, msr_info->index);
@@ -2056,7 +2070,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	struct vmx_uret_msr *msr;
 	int ret = 0;
 	u32 msr_index = msr_info->index;
-	u64 data = msr_info->data;
+	u64 data = msr_info->data, arch_msr;
 	u32 index;
 
 	switch (msr_index) {
@@ -2390,6 +2404,46 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		vmx->msr_virtual_enumeration = data &
 						VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
 		break;
+	case MSR_VIRTUAL_MITIGATION_ENUM:
+		if (msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		if (data & ~MITI_ENUM_SUPPORTED)
+			return 1;
+		vmx->msr_virtual_mitigation_enum = data;
+		break;
+	case MSR_VIRTUAL_MITIGATION_CTRL:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		if (data & ~MITI_CTRL_USED)
+			return 1;
+
 

arch/x86/kvm/vmx/vmx.c static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 2434

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +9 -0
 
+		if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES))
+			rdmsrl(MSR_IA32_ARCH_CAPABILITIES, arch_msr);
+
+		if (data & MITI_CTRL_RETPOLINE_S_USED &&
+		    boot_cpu_has(X86_FEATURE_RRSBA_CTRL) &&
+		    arch_msr & ARCH_CAP_RRSBA)
+			vmx->spec_ctrl_mask |= SPEC_CTRL_RRSBA_DIS_S;
+		else
+			vmx->spec_ctrl_mask &= ~SPEC_CTRL_RRSBA_DIS_S;

IIUC, only above chunk belongs to this patch. Other changes should be moved to patch 7, which deals with MSRs emulation.

author
#27 lore raw

RFC PATCH 9/9 x86/kvm/vmx: Initialize SPEC_CTRL MASK for BHI

In reply to Zhang Chen

VMMs can address mitigations issues in migration pool by applying the needed controls whenever the guest is operating on a newer processor. If a guest is using the BHB-clearing sequence on transitions into CPL0 to mitigate BHI, the VMM can use the “virtual IA32_SPEC_CTRL” VM-execution control to set BHI_DIS_S on newer hardware which does not enumerate BHI_NO.

---
 arch/x86/kvm/vmx/vmx.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/kvm/vmx/vmx.c +7 -0
index fb0f3b1639b9..980d1ace9718 100644
@@ -2431,6 +2431,13 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		else
 			vmx->spec_ctrl_mask &= ~SPEC_CTRL_RRSBA_DIS_S;
 
+		if (data & MITI_CTRL_BHB_CLEAR_SEQ_S_USED &&
+		    kvm_cpu_cap_has(X86_FEATURE_BHI_CTRL) &&
+		    !(arch_msr & ARCH_CAP_BHI_NO))
+			vmx->spec_ctrl_mask |= SPEC_CTRL_BHI_DIS_S;
+		else
+			vmx->spec_ctrl_mask &= ~SPEC_CTRL_BHI_DIS_S;
+
 		if (cpu_has_virt_spec_ctrl()) {
 			vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
 		} else if (vmx->spec_ctrl_mask) {
Signature · 1 line
2.25.1