Skip to contentSkip to conversation

Unify IBRS virtualization

PATCH Applied Yosry Ahmed 3 participants 13 messages Feb 21, 2025 – Apr 25, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Unify IBRS virtualization

To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.

Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.

I am not sure if this series would land through the kvm-x86 tree or the tip/x86 tree.

Yosry Ahmed (3):
x86/cpufeatures: Define X86_FEATURE_AMD_IBRS_SAME_MODE
KVM: x86: Propagate AMD's IbrsSameMode to the guest
KVM: x86: Generalize IBRS virtualization on emulated VM-exit

 arch/x86/include/asm/cpufeatures.h       |  1 +
 arch/x86/kvm/cpuid.c                     |  1 +
 arch/x86/kvm/svm/nested.c                |  2 ++
 arch/x86/kvm/vmx/nested.c                | 11 +----------
 arch/x86/kvm/x86.h                       | 18 ++++++++++++++++++
 tools/arch/x86/include/asm/cpufeatures.h |  1 +
 6 files changed, 24 insertions(+), 10 deletions(-)
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
#1 lore raw

PATCH 1/3 x86/cpufeatures: Define X86_FEATURE_AMD_IBRS_SAME_MODE

Per the APM [1]:

Some processors, identified by CPUID Fn8000_0008_EBX[IbrsSameMode] (bit 19) = 1, provide additional speculation limits. For these processors, when IBRS is set, indirect branch predictions are not influenced by any prior indirect branches, regardless of mode (CPL and guest/host) and regardless of whether the prior indirect branches occurred before or after the setting of IBRS. This is referred to as Same Mode IBRS.

Define this feature bit, which will be used by KVM to determine if an IBPB is required on nested VM-exits in SVM.

[1] AMD64 Architecture Programmer's Manual Pub. 40332, Rev 4.08 - April

    2024, Volume 2, 3.2.9 Speculation Control MSRs
---
 arch/x86/include/asm/cpufeatures.h       | 1 +
 tools/arch/x86/include/asm/cpufeatures.h | 1 +
 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 645aa360628da..46af88357ac89 100644
@@ -344,6 +344,7 @@
 #define X86_FEATURE_AMD_IBRS		(13*32+14) /* Indirect Branch Restricted Speculation */
 #define X86_FEATURE_AMD_STIBP		(13*32+15) /* Single Thread Indirect Branch Predictors */
 #define X86_FEATURE_AMD_STIBP_ALWAYS_ON	(13*32+17) /* Single Thread Indirect Branch Predictors always-on preferred */
+#define X86_FEATURE_AMD_IBRS_SAME_MODE	(13*32+19) /* Indirect Branch Restricted Speculation same mode protection*/
 #define X86_FEATURE_AMD_PPIN		(13*32+23) /* "amd_ppin" Protected Processor Inventory Number */
 #define X86_FEATURE_AMD_SSBD		(13*32+24) /* Speculative Store Bypass Disable */
 #define X86_FEATURE_VIRT_SSBD		(13*32+25) /* "virt_ssbd" Virtualized Speculative Store Bypass Disable */
tools/arch/x86/include/asm/cpufeatures.h +1 -0
index 17b6590748c00..c99d626357c1e 100644
@@ -344,6 +344,7 @@
 #define X86_FEATURE_AMD_IBRS		(13*32+14) /* Indirect Branch Restricted Speculation */
 #define X86_FEATURE_AMD_STIBP		(13*32+15) /* Single Thread Indirect Branch Predictors */
 #define X86_FEATURE_AMD_STIBP_ALWAYS_ON	(13*32+17) /* Single Thread Indirect Branch Predictors always-on preferred */
+#define X86_FEATURE_AMD_IBRS_SAME_MODE (13*32+19) /* Indirect Branch Restricted Speculation same mode protection*/
 #define X86_FEATURE_AMD_PPIN		(13*32+23) /* "amd_ppin" Protected Processor Inventory Number */
 #define X86_FEATURE_AMD_SSBD		(13*32+24) /* Speculative Store Bypass Disable */
 #define X86_FEATURE_VIRT_SSBD		(13*32+25) /* "virt_ssbd" Virtualized Speculative Store Bypass Disable */
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
#2 lore raw

PATCH 2/3 KVM: x86: Propagate AMD's IbrsSameMode to the guest

In reply to Yosry Ahmed

If IBRS provides same mode (kernel/user or host/guest) protection on the host, then by definition it also provides same mode protection in the guest. In fact, all different modes from the guest's perspective are the same mode from the host's perspective anyway.

Propagate IbrsSameMode to the guests.

---
 arch/x86/kvm/cpuid.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/kvm/cpuid.c +1 -0
index edef30359c198..05d7bbfbb8885 100644
@@ -1103,6 +1103,7 @@ void kvm_set_cpu_caps(void)
 		F(AMD_SSB_NO),
 		F(AMD_STIBP),
 		F(AMD_STIBP_ALWAYS_ON),
+		F(AMD_IBRS_SAME_MODE),
 		F(AMD_PSFD),
 		F(AMD_IBPB_RET),
 	);
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
#3 lore raw

PATCH 3/3 KVM: x86: Generalize IBRS virtualization on emulated VM-exit

In reply to Yosry Ahmed

Commit 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.

AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.

Generalize the handling in nested_vmx_vmexit() by moving it into a generic function, add the AMD handling, and use it in nested_svm_vmexit() too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.

---
 arch/x86/kvm/svm/nested.c |  2 ++
 arch/x86/kvm/vmx/nested.c | 11 +----------
 arch/x86/kvm/x86.h        | 18 ++++++++++++++++++
 3 files changed, 21 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
index d77b094d9a4d6..61b73ff30807e 100644
@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 
 	nested_svm_copy_common_state(svm->nested.vmcb02.ptr, svm->vmcb01.ptr);
 
+	kvm_nested_vmexit_handle_spec_ctrl(vcpu);
+
 	svm_switch_vmcb(svm, &svm->vmcb01);
 
 	/*
arch/x86/kvm/vmx/nested.c +1 -10
index 8a7af02d466e9..453d52a6e836a 100644
@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 
 	vmx_switch_vmcs(vcpu, &vmx->vmcs01);
 
-	/*
-	 * If IBRS is advertised to the vCPU, KVM must flush the indirect
-	 * branch predictors when transitioning from L2 to L1, as L1 expects
-	 * hardware (KVM in this case) to provide separate predictor modes.
-	 * Bare metal isolates VMX root (host) from VMX non-root (guest), but
-	 * doesn't isolate different VMCSs, i.e. in this case, doesn't provide
-	 * separate modes for L2 vs L1.
-	 */
-	if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL))
-		indirect_branch_prediction_barrier();
+	kvm_nested_vmexit_handle_spec_ctrl(vcpu);
 
 	/* Update any VMCS fields that might have changed while L2 ran */
 	vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
arch/x86/kvm/x86.h +18 -0
index 7a87c5fc57f1b..008c8d381c253 100644
@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu)
 	kvm_x86_ops.nested_ops->leave_nested(vcpu);
 }
 
+/*
+ * If IBRS is advertised to the vCPU, KVM must flush the indirect branch
+ * predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in
+ * this case) to provide separate predictor modes.  Bare metal isolates the host
+ * from the guest, but doesn't isolate different guests from one another (in
+ * this case L1 and L2). The exception is if bare metal supports same mode IBRS,
+ * which offers protection within the same mode, and hence protects L1 from L2.
+ */
+static inline void kvm_nested_vmexit_handle_spec_ctrl(struct kvm_vcpu *vcpu)
+{
+	if (cpu_feature_enabled(X86_FEATURE_AMD_IBRS_SAME_MODE))
+		return;
+
+	if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) ||
+	    guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBRS))
+		indirect_branch_prediction_barrier();
+}
+
 static inline bool kvm_vcpu_has_run(struct kvm_vcpu *vcpu)
 {
 	return vcpu->arch.last_vmentry_cpu != -1;
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Reviewed-by on v1 3/3
#4 lore raw

Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit

Quoted 65 lines · Yosry Ahmed

Commit 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.

AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.

Generalize the handling in nested_vmx_vmexit() by moving it into a generic function, add the AMD handling, and use it in nested_svm_vmexit() too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.

---
 arch/x86/kvm/svm/nested.c |  2 ++
 arch/x86/kvm/vmx/nested.c | 11 +----------
 arch/x86/kvm/x86.h        | 18 ++++++++++++++++++
 3 files changed, 21 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
index d77b094d9a4d6..61b73ff30807e 100644
@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 
        nested_svm_copy_common_state(svm->nested.vmcb02.ptr, svm->vmcb01.ptr);
 
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
+
        svm_switch_vmcb(svm, &svm->vmcb01);
 
        /*
arch/x86/kvm/vmx/nested.c +1 -10
index 8a7af02d466e9..453d52a6e836a 100644
@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 
        vmx_switch_vmcs(vcpu, &vmx->vmcs01);
 
-       /*
-        * If IBRS is advertised to the vCPU, KVM must flush the indirect
-        * branch predictors when transitioning from L2 to L1, as L1 expects
-        * hardware (KVM in this case) to provide separate predictor modes.
-        * Bare metal isolates VMX root (host) from VMX non-root (guest), but
-        * doesn't isolate different VMCSs, i.e. in this case, doesn't provide
-        * separate modes for L2 vs L1.
-        */
-       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL))
-               indirect_branch_prediction_barrier();
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
 
        /* Update any VMCS fields that might have changed while L2 ran */
        vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
arch/x86/kvm/x86.h +0 -0
index 7a87c5fc57f1b..008c8d381c253 100644
@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu)
        kvm_x86_ops.nested_ops->leave_nested(vcpu);
 }

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127

Quoted 10 lines
arch/x86/kvm/x86.h +9 -0
 
+/*
+ * If IBRS is advertised to the vCPU, KVM must flush the indirect branch
+ * predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in
+ * this case) to provide separate predictor modes.  Bare metal isolates the host
+ * from the guest, but doesn't isolate different guests from one another (in
+ * this case L1 and L2). The exception is if bare metal supports same mode IBRS,
+ * which offers protection within the same mode, and hence protects L1 from L2.
+ */
+static inline void kvm_nested_vmexit_handle_spec_ctrl(struct kvm_vcpu *vcpu)

Maybe just kvm_nested_vmexit_handle_ibrs?

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133

Quoted 6 lines
arch/x86/kvm/x86.h +6 -0
+{
+       if (cpu_feature_enabled(X86_FEATURE_AMD_IBRS_SAME_MODE))
+               return;
+
+       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) ||
+           guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBRS))

This is a bit conservative, but I don't think there's any ROI in being more pedantic.

For the series,

Quoted 9 lines
arch/x86/kvm/x86.h +3 -1
+               indirect_branch_prediction_barrier();
+}
+
 static inline bool kvm_vcpu_has_run(struct kvm_vcpu *vcpu)
 {
        return vcpu->arch.last_vmentry_cpu != -1;
--

2.48.1.601.g30ceb7b040-goog

#5 lore raw

Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit

Quoted 67 lines · Jim Mattson
Quoted 66 lines · Yosry Ahmed

Commit 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.

AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.

Generalize the handling in nested_vmx_vmexit() by moving it into a generic function, add the AMD handling, and use it in nested_svm_vmexit() too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.

---
 arch/x86/kvm/svm/nested.c |  2 ++
 arch/x86/kvm/vmx/nested.c | 11 +----------
 arch/x86/kvm/x86.h        | 18 ++++++++++++++++++
 3 files changed, 21 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
index d77b094d9a4d6..61b73ff30807e 100644
@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 
        nested_svm_copy_common_state(svm->nested.vmcb02.ptr, svm->vmcb01.ptr);
 
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
+
        svm_switch_vmcb(svm, &svm->vmcb01);
 
        /*
arch/x86/kvm/vmx/nested.c +1 -10
index 8a7af02d466e9..453d52a6e836a 100644
@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 
        vmx_switch_vmcs(vcpu, &vmx->vmcs01);
 
-       /*
-        * If IBRS is advertised to the vCPU, KVM must flush the indirect
-        * branch predictors when transitioning from L2 to L1, as L1 expects
-        * hardware (KVM in this case) to provide separate predictor modes.
-        * Bare metal isolates VMX root (host) from VMX non-root (guest), but
-        * doesn't isolate different VMCSs, i.e. in this case, doesn't provide
-        * separate modes for L2 vs L1.
-        */
-       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL))
-               indirect_branch_prediction_barrier();
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
 
        /* Update any VMCS fields that might have changed while L2 ran */
        vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
arch/x86/kvm/x86.h +0 -0
index 7a87c5fc57f1b..008c8d381c253 100644
@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu)
        kvm_x86_ops.nested_ops->leave_nested(vcpu);
 }
 
Quoted 11 lines

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127

Quoted 9 lines
arch/x86/kvm/x86.h +9 -0
+/*
+ * If IBRS is advertised to the vCPU, KVM must flush the indirect branch
+ * predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in
+ * this case) to provide separate predictor modes.  Bare metal isolates the host
+ * from the guest, but doesn't isolate different guests from one another (in
+ * this case L1 and L2). The exception is if bare metal supports same mode IBRS,
+ * which offers protection within the same mode, and hence protects L1 from L2.
+ */
+static inline void kvm_nested_vmexit_handle_spec_ctrl(struct kvm_vcpu *vcpu)

Maybe just kvm_nested_vmexit_handle_ibrs?

I was trying to use a generic name to accomodate any future handling needed for non-IBRS speculation control virtualization. But I could just be overthinking. Happy to take whatever name is agreed upon in during reviews.

Quoted 10 lines

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133

Quoted 7 lines
arch/x86/kvm/x86.h +6 -0
+{
+       if (cpu_feature_enabled(X86_FEATURE_AMD_IBRS_SAME_MODE))
+               return;
+
+       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) ||
+           guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBRS))

This is a bit conservative, but I don't think there's any ROI in being more pedantic.

Could you elaborate on this?

Is this about doing the IBPB even if L1 does not actually execute an IBRS? I thought about this for a bit, but otherwise we'd have to intercept the MSR write IIUC, and I am not sure if that's better. Also, that's what we are already doing so I just kept it as-is.

Or maybe about whether we need this on AMD only with AUTOIBRS? The APM is a bit unclear to me in this regard, but I believe may be needed even for 'normal' IBRS.

Quoted 4 lines

For the series,

Thanks!

Quoted 10 lines
Quoted 10 lines
arch/x86/kvm/x86.h +3 -1
+               indirect_branch_prediction_barrier();
+}
+
 static inline bool kvm_vcpu_has_run(struct kvm_vcpu *vcpu)
 {
        return vcpu->arch.last_vmentry_cpu != -1;
--

2.48.1.601.g30ceb7b040-goog

#6 lore raw

Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit

Quoted 99 lines · Yosry Ahmed
Quoted 67 lines · Jim Mattson
Quoted 66 lines · Yosry Ahmed

Commit 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.

AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.

Generalize the handling in nested_vmx_vmexit() by moving it into a generic function, add the AMD handling, and use it in nested_svm_vmexit() too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.

---
 arch/x86/kvm/svm/nested.c |  2 ++
 arch/x86/kvm/vmx/nested.c | 11 +----------
 arch/x86/kvm/x86.h        | 18 ++++++++++++++++++
 3 files changed, 21 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
index d77b094d9a4d6..61b73ff30807e 100644
@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 
        nested_svm_copy_common_state(svm->nested.vmcb02.ptr, svm->vmcb01.ptr);
 
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
+
        svm_switch_vmcb(svm, &svm->vmcb01);
 
        /*
arch/x86/kvm/vmx/nested.c +1 -10
index 8a7af02d466e9..453d52a6e836a 100644
@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 
        vmx_switch_vmcs(vcpu, &vmx->vmcs01);
 
-       /*
-        * If IBRS is advertised to the vCPU, KVM must flush the indirect
-        * branch predictors when transitioning from L2 to L1, as L1 expects
-        * hardware (KVM in this case) to provide separate predictor modes.
-        * Bare metal isolates VMX root (host) from VMX non-root (guest), but
-        * doesn't isolate different VMCSs, i.e. in this case, doesn't provide
-        * separate modes for L2 vs L1.
-        */
-       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL))
-               indirect_branch_prediction_barrier();
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
 
        /* Update any VMCS fields that might have changed while L2 ran */
        vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
arch/x86/kvm/x86.h +0 -0
index 7a87c5fc57f1b..008c8d381c253 100644
@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu)
        kvm_x86_ops.nested_ops->leave_nested(vcpu);
 }
 
Quoted 11 lines

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127

Quoted 9 lines
arch/x86/kvm/x86.h +9 -0
+/*
+ * If IBRS is advertised to the vCPU, KVM must flush the indirect branch
+ * predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in
+ * this case) to provide separate predictor modes.  Bare metal isolates the host
+ * from the guest, but doesn't isolate different guests from one another (in
+ * this case L1 and L2). The exception is if bare metal supports same mode IBRS,
+ * which offers protection within the same mode, and hence protects L1 from L2.
+ */
+static inline void kvm_nested_vmexit_handle_spec_ctrl(struct kvm_vcpu *vcpu)

Maybe just kvm_nested_vmexit_handle_ibrs?

I was trying to use a generic name to accomodate any future handling needed for non-IBRS speculation control virtualization. But I could just be overthinking. Happy to take whatever name is agreed upon in during reviews.

Quoted 10 lines

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133

Quoted 7 lines
arch/x86/kvm/x86.h +6 -0
+{
+       if (cpu_feature_enabled(X86_FEATURE_AMD_IBRS_SAME_MODE))
+               return;
+
+       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) ||
+           guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBRS))

This is a bit conservative, but I don't think there's any ROI in being more pedantic.

Could you elaborate on this?

Quoted 8 lines

Is this about doing the IBPB even if L1 does not actually execute an IBRS? I thought about this for a bit, but otherwise we'd have to intercept the MSR write IIUC, and I am not sure if that's better. Also, that's what we are already doing so I just kept it as-is.

Or maybe about whether we need this on AMD only with AUTOIBRS? The APM is a bit unclear to me in this regard, but I believe may be needed even for 'normal' IBRS.

If IA32_SPEC_CTRL.IBRS is clear at emulated VM-exit, then this IBPB is unnecessary.

However, since the host (L1) is running in a de-privileged prediction domain, simply setting IA32_SPEC_CTRL.IBRS in the future won't protect it from the guest (L2) that just exited. If we don't eagerly perform an IBPB now, then L0 would have to intercept WRMSR(IA32_SPEC_CTRL) from L1 so that we can issue an IBPB in the future, if L1 ever sets IA32_SPEC_CTRL.IBRS.

Eagerly performing an IBPB now seems like the better option.

Quoted 17 lines
Quoted 4 lines

For the series,

Thanks!

Quoted 10 lines
Quoted 10 lines
arch/x86/kvm/x86.h +3 -1
+               indirect_branch_prediction_barrier();
+}
+
 static inline bool kvm_vcpu_has_run(struct kvm_vcpu *vcpu)
 {
        return vcpu->arch.last_vmentry_cpu != -1;
--

2.48.1.601.g30ceb7b040-goog

#7 lore raw

Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit

Quoted 112 lines · Jim Mattson
Quoted 99 lines · Yosry Ahmed
Quoted 67 lines · Jim Mattson
Quoted 66 lines · Yosry Ahmed

Commit 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.

AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.

Generalize the handling in nested_vmx_vmexit() by moving it into a generic function, add the AMD handling, and use it in nested_svm_vmexit() too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.

---
 arch/x86/kvm/svm/nested.c |  2 ++
 arch/x86/kvm/vmx/nested.c | 11 +----------
 arch/x86/kvm/x86.h        | 18 ++++++++++++++++++
 3 files changed, 21 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
index d77b094d9a4d6..61b73ff30807e 100644
@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 
        nested_svm_copy_common_state(svm->nested.vmcb02.ptr, svm->vmcb01.ptr);
 
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
+
        svm_switch_vmcb(svm, &svm->vmcb01);
 
        /*
arch/x86/kvm/vmx/nested.c +1 -10
index 8a7af02d466e9..453d52a6e836a 100644
@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 
        vmx_switch_vmcs(vcpu, &vmx->vmcs01);
 
-       /*
-        * If IBRS is advertised to the vCPU, KVM must flush the indirect
-        * branch predictors when transitioning from L2 to L1, as L1 expects
-        * hardware (KVM in this case) to provide separate predictor modes.
-        * Bare metal isolates VMX root (host) from VMX non-root (guest), but
-        * doesn't isolate different VMCSs, i.e. in this case, doesn't provide
-        * separate modes for L2 vs L1.
-        */
-       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL))
-               indirect_branch_prediction_barrier();
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
 
        /* Update any VMCS fields that might have changed while L2 ran */
        vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
arch/x86/kvm/x86.h +0 -0
index 7a87c5fc57f1b..008c8d381c253 100644
@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu)
        kvm_x86_ops.nested_ops->leave_nested(vcpu);
 }
 
Quoted 11 lines

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127

Quoted 9 lines
arch/x86/kvm/x86.h +9 -0
+/*
+ * If IBRS is advertised to the vCPU, KVM must flush the indirect branch
+ * predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in
+ * this case) to provide separate predictor modes.  Bare metal isolates the host
+ * from the guest, but doesn't isolate different guests from one another (in
+ * this case L1 and L2). The exception is if bare metal supports same mode IBRS,
+ * which offers protection within the same mode, and hence protects L1 from L2.
+ */
+static inline void kvm_nested_vmexit_handle_spec_ctrl(struct kvm_vcpu *vcpu)

Maybe just kvm_nested_vmexit_handle_ibrs?

I was trying to use a generic name to accomodate any future handling needed for non-IBRS speculation control virtualization. But I could just be overthinking. Happy to take whatever name is agreed upon in during reviews.

Quoted 10 lines

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133

Quoted 7 lines
arch/x86/kvm/x86.h +6 -0
+{
+       if (cpu_feature_enabled(X86_FEATURE_AMD_IBRS_SAME_MODE))
+               return;
+
+       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) ||
+           guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBRS))

This is a bit conservative, but I don't think there's any ROI in being more pedantic.

Could you elaborate on this?

Quoted 8 lines

Is this about doing the IBPB even if L1 does not actually execute an IBRS? I thought about this for a bit, but otherwise we'd have to intercept the MSR write IIUC, and I am not sure if that's better. Also, that's what we are already doing so I just kept it as-is.

Or maybe about whether we need this on AMD only with AUTOIBRS? The APM is a bit unclear to me in this regard, but I believe may be needed even for 'normal' IBRS.

If IA32_SPEC_CTRL.IBRS is clear at emulated VM-exit, then this IBPB is unnecessary.

Quoted 6 lines

However, since the host (L1) is running in a de-privileged prediction domain, simply setting IA32_SPEC_CTRL.IBRS in the future won't protect it from the guest (L2) that just exited. If we don't eagerly perform an IBPB now, then L0 would have to intercept WRMSR(IA32_SPEC_CTRL) from L1 so that we can issue an IBPB in the future, if L1 ever sets IA32_SPEC_CTRL.IBRS.

Right, that's what I meant by "we'd have to intercept the MSR write" above, but I didn't put it as eloquently as you just did :)

We'd also need to have different handling for eIBRS/AUTOIBRS. It would basically be:

if eIBRS/AUTOIBRS is enabled by L1:
- Do not intercept IBRS MSR writes
- Always IBPB on emulated VM-exits (unless IbrsSameMode).
else if IBRS is advertised to L1:
- Intercept IBRS MSR writes and do an IBPB.
- Do not IBPB on emulated VM-exits.

We'd basically have two modes of IBRS virtualization and we'd need to switch between them at runtime according to L1's setting of eIBRS/AUTOIBRS.

We can simplify it if we always intercept IBRS MSR writes assuming L1 won't do them with eIBRS/AUTOIBRS anyway, so this becomes:

- On emulated VM-exits, IBPB if eIBRS/AUTOIBRS is enabled (unless IbrsSameMode).
- On IBRS MSR writes, do an IBPB.

Simpler, but not sure if it buys us much.

Quoted 2 lines

Eagerly performing an IBPB now seems like the better option.

So yeah I definitely agree, unless we get regression reports caused by the IBPB on emulated VM-exits, and the MSR write interception turns out to be an improvement.

#8 lore raw

Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit

Quoted 145 lines · Yosry Ahmed
Quoted 112 lines · Jim Mattson
Quoted 99 lines · Yosry Ahmed
Quoted 67 lines · Jim Mattson
Quoted 66 lines · Yosry Ahmed

Commit 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.

AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.

Generalize the handling in nested_vmx_vmexit() by moving it into a generic function, add the AMD handling, and use it in nested_svm_vmexit() too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.

---
 arch/x86/kvm/svm/nested.c |  2 ++
 arch/x86/kvm/vmx/nested.c | 11 +----------
 arch/x86/kvm/x86.h        | 18 ++++++++++++++++++
 3 files changed, 21 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
index d77b094d9a4d6..61b73ff30807e 100644
@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 
        nested_svm_copy_common_state(svm->nested.vmcb02.ptr, svm->vmcb01.ptr);
 
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
+
        svm_switch_vmcb(svm, &svm->vmcb01);
 
        /*
arch/x86/kvm/vmx/nested.c +1 -10
index 8a7af02d466e9..453d52a6e836a 100644
@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 
        vmx_switch_vmcs(vcpu, &vmx->vmcs01);
 
-       /*
-        * If IBRS is advertised to the vCPU, KVM must flush the indirect
-        * branch predictors when transitioning from L2 to L1, as L1 expects
-        * hardware (KVM in this case) to provide separate predictor modes.
-        * Bare metal isolates VMX root (host) from VMX non-root (guest), but
-        * doesn't isolate different VMCSs, i.e. in this case, doesn't provide
-        * separate modes for L2 vs L1.
-        */
-       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL))
-               indirect_branch_prediction_barrier();
+       kvm_nested_vmexit_handle_spec_ctrl(vcpu);
 
        /* Update any VMCS fields that might have changed while L2 ran */
        vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
arch/x86/kvm/x86.h +0 -0
index 7a87c5fc57f1b..008c8d381c253 100644
@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu)
        kvm_x86_ops.nested_ops->leave_nested(vcpu);
 }
 
Quoted 11 lines

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127

Quoted 9 lines
arch/x86/kvm/x86.h +9 -0
+/*
+ * If IBRS is advertised to the vCPU, KVM must flush the indirect branch
+ * predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in
+ * this case) to provide separate predictor modes.  Bare metal isolates the host
+ * from the guest, but doesn't isolate different guests from one another (in
+ * this case L1 and L2). The exception is if bare metal supports same mode IBRS,
+ * which offers protection within the same mode, and hence protects L1 from L2.
+ */
+static inline void kvm_nested_vmexit_handle_spec_ctrl(struct kvm_vcpu *vcpu)

Maybe just kvm_nested_vmexit_handle_ibrs?

I was trying to use a generic name to accomodate any future handling needed for non-IBRS speculation control virtualization. But I could just be overthinking. Happy to take whatever name is agreed upon in during reviews.

Quoted 10 lines

arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133

Quoted 7 lines
arch/x86/kvm/x86.h +6 -0
+{
+       if (cpu_feature_enabled(X86_FEATURE_AMD_IBRS_SAME_MODE))
+               return;
+
+       if (guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) ||
+           guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBRS))

This is a bit conservative, but I don't think there's any ROI in being more pedantic.

Could you elaborate on this?

Quoted 8 lines

Is this about doing the IBPB even if L1 does not actually execute an IBRS? I thought about this for a bit, but otherwise we'd have to intercept the MSR write IIUC, and I am not sure if that's better. Also, that's what we are already doing so I just kept it as-is.

Or maybe about whether we need this on AMD only with AUTOIBRS? The APM is a bit unclear to me in this regard, but I believe may be needed even for 'normal' IBRS.

If IA32_SPEC_CTRL.IBRS is clear at emulated VM-exit, then this IBPB is unnecessary.

Quoted 6 lines

However, since the host (L1) is running in a de-privileged prediction domain, simply setting IA32_SPEC_CTRL.IBRS in the future won't protect it from the guest (L2) that just exited. If we don't eagerly perform an IBPB now, then L0 would have to intercept WRMSR(IA32_SPEC_CTRL) from L1 so that we can issue an IBPB in the future, if L1 ever sets IA32_SPEC_CTRL.IBRS.

Right, that's what I meant by "we'd have to intercept the MSR write" above, but I didn't put it as eloquently as you just did :)

We'd also need to have different handling for eIBRS/AUTOIBRS. It would basically be:

if eIBRS/AUTOIBRS is enabled by L1:
- Do not intercept IBRS MSR writes
- Always IBPB on emulated VM-exits (unless IbrsSameMode).
else if IBRS is advertised to L1:
- Intercept IBRS MSR writes and do an IBPB.
- Do not IBPB on emulated VM-exits.

We'd basically have two modes of IBRS virtualization and we'd need to switch between them at runtime according to L1's setting of eIBRS/AUTOIBRS.

We can simplify it if we always intercept IBRS MSR writes assuming L1 won't do them with eIBRS/AUTOIBRS anyway, so this becomes:

- On emulated VM-exits, IBPB if eIBRS/AUTOIBRS is enabled (unless IbrsSameMode).
- On IBRS MSR writes, do an IBPB.

Quoted 8 lines

Simpler, but not sure if it buys us much.

Quoted 2 lines

Eagerly performing an IBPB now seems like the better option.

So yeah I definitely agree, unless we get regression reports caused by the IBPB on emulated VM-exits, and the MSR write interception turns out to be an improvement.

For "normal" configurations, I would expect L1 to set eIBRS or autoIBRS on modern CPUs. In that case, the IBPB is required. However, if L1 is a Linux OS booted with mitigations=off, the IBPB is probably gratuitous.

On the Intel side, KVM tries to do some optimization of the case where the guest doesn't use IA32_SPEC_CTRL. Since writes to IA32_SPEC_CTRL are intercepted until the first non-zero value is written, I suppose you could skip the IBPB when writes to IA32_SPEC_CTRL are intercepted, and then just blindly perform an IBPB when disabling the intercept in vmx_set_msr() if nested is enabled. Again, though, I don't think this is worth it.

#9 lore raw
In reply to Yosry Ahmed
Quoted 10 lines · Yosry Ahmed

To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.

Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.

I am not sure if this series would land through the kvm-x86 tree or the tip/x86 tree.

Sean, any thoughts about this (or general feedback about this series)?

Quoted 17 lines

Yosry Ahmed (3):
x86/cpufeatures: Define X86_FEATURE_AMD_IBRS_SAME_MODE
KVM: x86: Propagate AMD's IbrsSameMode to the guest
KVM: x86: Generalize IBRS virtualization on emulated VM-exit

 arch/x86/include/asm/cpufeatures.h       |  1 +
 arch/x86/kvm/cpuid.c                     |  1 +
 arch/x86/kvm/svm/nested.c                |  2 ++
 arch/x86/kvm/vmx/nested.c                | 11 +----------
 arch/x86/kvm/x86.h                       | 18 ++++++++++++++++++
 tools/arch/x86/include/asm/cpufeatures.h |  1 +
 6 files changed, 24 insertions(+), 10 deletions(-)
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
#10 lore raw
Quoted 9 lines · Yosry Ahmed
Quoted 8 lines · Yosry Ahmed

To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.

Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.

Quoted 4 lines
Quoted 2 lines

I am not sure if this series would land through the kvm-x86 tree or the tip/x86 tree.

Sean, any thoughts about this (or general feedback about this series)?

No feedback, I just you and Jim to get mitigation stuff right far more than I trust myself :-)

I'm planning on grabbing this for 6.16.

#11 lore raw
Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Yosry Ahmed
Quoted 10 lines · Yosry Ahmed

To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.

Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.

I am not sure if this series would land through the kvm-x86 tree or the tip/x86 tree.

Quoted 6 lines
Quoted 1 line

Sean, any thoughts about this (or general feedback about this series)?

No feedback, I just you and Jim to get mitigation stuff right far more than I trust myself :-)

I'm planning on grabbing this for 6.16.

Awesome, thanks!

maintainer Applied
#12 lore raw
In reply to Yosry Ahmed
Quoted 9 lines · Yosry Ahmed

To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.

Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.

[...]

Applied to kvm-x86 misc, thanks!

[1/3] x86/cpufeatures: Define X86_FEATURE_AMD_IBRS_SAME_MODE

      commit: 9a7cb00a8ff7380a09fa75287a3f2642c472d562
[2/3] KVM: x86: Propagate AMD's IbrsSameMode to the guest
      commit: 65ca2872015c232d6743b497e3c08ff96596b917
[3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit
      commit: 656d9624bd21d35499eaa5ee97fda6def62901c8

--
https://github.com/kvm-x86/linux/tree/next