Unify IBRS virtualization
cover Unify IBRS virtualization
To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.
Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.
I am not sure if this series would land through the kvm-x86 tree or the tip/x86 tree.
Yosry Ahmed (3):
x86/cpufeatures: Define X86_FEATURE_AMD_IBRS_SAME_MODE
KVM: x86: Propagate AMD's IbrsSameMode to the guest
KVM: x86: Generalize IBRS virtualization on emulated VM-exit
arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kvm/cpuid.c | 1 + arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/vmx/nested.c | 11 +---------- arch/x86/kvm/x86.h | 18 ++++++++++++++++++ tools/arch/x86/include/asm/cpufeatures.h | 1 + 6 files changed, 24 insertions(+), 10 deletions(-)
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
PATCH 1/3 x86/cpufeatures: Define X86_FEATURE_AMD_IBRS_SAME_MODE
Per the APM [1]:
Some processors, identified by CPUID Fn8000_0008_EBX[IbrsSameMode] (bit 19) = 1, provide additional speculation limits. For these processors, when IBRS is set, indirect branch predictions are not influenced by any prior indirect branches, regardless of mode (CPL and guest/host) and regardless of whether the prior indirect branches occurred before or after the setting of IBRS. This is referred to as Same Mode IBRS.
Define this feature bit, which will be used by KVM to determine if an IBPB is required on nested VM-exits in SVM.
[1] AMD64 Architecture Programmer's Manual Pub. 40332, Rev 4.08 - April
2024, Volume 2, 3.2.9 Speculation Control MSRs
--- arch/x86/include/asm/cpufeatures.h | 1 + tools/arch/x86/include/asm/cpufeatures.h | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -344,6 +344,7 @@#define X86_FEATURE_AMD_IBRS (13*32+14)/* Indirect Branch Restricted Speculation */ #define X86_FEATURE_AMD_STIBP (13*32+15)/* Single Thread Indirect Branch Predictors */ #define X86_FEATURE_AMD_STIBP_ALWAYS_ON (13*32+17)/* Single Thread Indirect Branch Predictors always-on preferred */ +#define X86_FEATURE_AMD_IBRS_SAME_MODE (13*32+19)/* Indirect Branch Restricted Speculation same mode protection*/ #define X86_FEATURE_AMD_PPIN (13*32+23)/* "amd_ppin" Protected Processor Inventory Number */ #define X86_FEATURE_AMD_SSBD (13*32+24)/* Speculative Store Bypass Disable */ #define X86_FEATURE_VIRT_SSBD (13*32+25)/* "virt_ssbd" Virtualized Speculative Store Bypass Disable */
tools/arch/x86/include/asm/cpufeatures.h +1 -0
@@ -344,6 +344,7 @@#define X86_FEATURE_AMD_IBRS (13*32+14)/* Indirect Branch Restricted Speculation */ #define X86_FEATURE_AMD_STIBP (13*32+15)/* Single Thread Indirect Branch Predictors */ #define X86_FEATURE_AMD_STIBP_ALWAYS_ON (13*32+17)/* Single Thread Indirect Branch Predictors always-on preferred */ +#define X86_FEATURE_AMD_IBRS_SAME_MODE (13*32+19)/* Indirect Branch Restricted Speculation same mode protection*/ #define X86_FEATURE_AMD_PPIN (13*32+23)/* "amd_ppin" Protected Processor Inventory Number */ #define X86_FEATURE_AMD_SSBD (13*32+24)/* Speculative Store Bypass Disable */ #define X86_FEATURE_VIRT_SSBD (13*32+25)/* "virt_ssbd" Virtualized Speculative Store Bypass Disable */
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
PATCH 2/3 KVM: x86: Propagate AMD's IbrsSameMode to the guest
In reply to Yosry AhmedIf IBRS provides same mode (kernel/user or host/guest) protection on the host, then by definition it also provides same mode protection in the guest. In fact, all different modes from the guest's perspective are the same mode from the host's perspective anyway.
Propagate IbrsSameMode to the guests.
--- arch/x86/kvm/cpuid.c | 1 + 1 file changed, 1 insertion(+)
arch/x86/kvm/cpuid.c +1 -0
@@ -1103,6 +1103,7 @@ void kvm_set_cpu_caps(void)F (AMD_SSB_NO ),F (AMD_STIBP ),F (AMD_STIBP_ALWAYS_ON ), +F (AMD_IBRS_SAME_MODE ),F (AMD_PSFD ), F(AMD_IBPB_RET), );
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
PATCH 3/3 KVM: x86: Generalize IBRS virtualization on emulated VM-exit
In reply to Yosry AhmedCommit 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.
AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.
Generalize the handling in nested_vmx_vmexit() by moving it into a generic function, add the AMD handling, and use it in nested_svm_vmexit() too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.
--- arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/vmx/nested.c | 11 +---------- arch/x86/kvm/x86.h | 18 ++++++++++++++++++ 3 files changed, 21 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)nested_svm_copy_common_state (svm->nested.vmcb02.ptr, svm->vmcb01.ptr); +kvm_nested_vmexit_handle_spec_ctrl (vcpu); +svm_switch_vmcb (svm, &svm->vmcb01);/*
arch/x86/kvm/vmx/nested.c +1 -10
@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmx_switch_vmcs (vcpu, &vmx->vmcs01); -/* -* If IBRS is advertised to the vCPU, KVM must flush the indirect -* branch predictors when transitioning from L2 to L1, as L1 expects -* hardware (KVM in this case) to provide separate predictor modes. -* Bare metal isolates VMX root (host) from VMX non-root (guest), but -* doesn't isolate different VMCSs, i.e. in this case, doesn't provide -* separate modes for L2 vs L1. -*/ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL )) -indirect_branch_prediction_barrier (); +kvm_nested_vmexit_handle_spec_ctrl (vcpu);/* Update any VMCS fields that might have changed while L2 ran */ vmcs_write32 (VM_EXIT_MSR_LOAD_COUNT , vmx->msr_autoload.host.nr);
arch/x86/kvm/x86.h +18 -0
@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) kvm_x86_ops.nested_ops->leave_nested (vcpu); } +/* +* If IBRS is advertised to the vCPU, KVM must flush the indirect branch +* predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in +* this case) to provide separate predictor modes. Bare metal isolates the host +* from the guest, but doesn't isolate different guests from one another (in +* this case L1 and L2). The exception is if bare metal supports same mode IBRS, +* which offers protection within the same mode, and hence protects L1 from L2. +*/ +static inline void kvm_nested_vmexit_handle_spec_ctrl (struct kvm_vcpu *vcpu) +{ +if (cpu_feature_enabled (X86_FEATURE_AMD_IBRS_SAME_MODE )) +return ; + +if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ) || +guest_cpu_cap_has (vcpu,X86_FEATURE_AMD_IBRS )) +indirect_branch_prediction_barrier (); +} +static inline bool kvm_vcpu_has_run (struct kvm_vcpu *vcpu) {return vcpu->arch.last_vmentry_cpu !=-1 ;
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit
Quoted 65 lines · Yosry Ahmed
Commit
2e7eab81425a("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.
Generalize the handling in
nested_vmx_vmexit()by moving it into a generic function, add the AMD handling, and use it innested_svm_vmexit()too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/vmx/nested.c | 11 +---------- arch/x86/kvm/x86.h | 18 ++++++++++++++++++ 3 files changed, 21 insertions(+), 10 deletions(-)arch/x86/kvm/svm/nested.c +2
-0@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)nested_svm_copy_common_state (svm->nested.vmcb02.ptr, svm->vmcb01.ptr); +kvm_nested_vmexit_handle_spec_ctrl (vcpu); +svm_switch_vmcb (svm, &svm->vmcb01);/* arch/x86/kvm/vmx/nested.c +1
-10@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmx_switch_vmcs (vcpu, &vmx->vmcs01); -/* -* If IBRS is advertised to the vCPU, KVM must flush the indirect -* branch predictors when transitioning from L2 to L1, as L1 expects -* hardware (KVM in this case) to provide separate predictor modes. -* Bare metal isolates VMX root (host) from VMX non-root (guest), but -* doesn't isolate different VMCSs, i.e. in this case, doesn't provide -* separate modes for L2 vs L1. -*/ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL )) -indirect_branch_prediction_barrier (); +kvm_nested_vmexit_handle_spec_ctrl (vcpu);/* Update any VMCS fields that might have changed while L2 ran */ vmcs_write32 (VM_EXIT_MSR_LOAD_COUNT , vmx->msr_autoload.host.nr);arch/x86/kvm/x86.h +0
-0@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) kvm_x86_ops.nested_ops->leave_nested (vcpu); }
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127
Quoted 10 lines
arch/x86/kvm/x86.h +9
-0+/* +* If IBRS is advertised to the vCPU, KVM must flush the indirect branch +* predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in +* this case) to provide separate predictor modes. Bare metal isolates the host +* from the guest, but doesn't isolate different guests from one another (in +* this case L1 and L2). The exception is if bare metal supports same mode IBRS, +* which offers protection within the same mode, and hence protects L1 from L2. +*/ +static inline void kvm_nested_vmexit_handle_spec_ctrl (struct kvm_vcpu *vcpu)
Maybe just kvm_nested_vmexit_handle_ibrs?
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133
Quoted 6 lines
arch/x86/kvm/x86.h +6
-0+{ +if (cpu_feature_enabled (X86_FEATURE_AMD_IBRS_SAME_MODE )) +return ; + +if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ) || +guest_cpu_cap_has (vcpu,X86_FEATURE_AMD_IBRS ))
This is a bit conservative, but I don't think there's any ROI in being more pedantic.
For the series,
Quoted 9 lines
arch/x86/kvm/x86.h +3
-1+indirect_branch_prediction_barrier (); +} +static inline bool kvm_vcpu_has_run (struct kvm_vcpu *vcpu) {return vcpu->arch.last_vmentry_cpu !=-1 ; --2.48.1.601.g30ceb7b040-goog
Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit
Quoted 67 lines · Jim Mattson
Quoted 66 lines · Yosry Ahmed
Commit
2e7eab81425a("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.
Generalize the handling in
nested_vmx_vmexit()by moving it into a generic function, add the AMD handling, and use it innested_svm_vmexit()too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/vmx/nested.c | 11 +---------- arch/x86/kvm/x86.h | 18 ++++++++++++++++++ 3 files changed, 21 insertions(+), 10 deletions(-)arch/x86/kvm/svm/nested.c +2
-0@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)nested_svm_copy_common_state (svm->nested.vmcb02.ptr, svm->vmcb01.ptr); +kvm_nested_vmexit_handle_spec_ctrl (vcpu); +svm_switch_vmcb (svm, &svm->vmcb01);/* arch/x86/kvm/vmx/nested.c +1
-10@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmx_switch_vmcs (vcpu, &vmx->vmcs01); -/* -* If IBRS is advertised to the vCPU, KVM must flush the indirect -* branch predictors when transitioning from L2 to L1, as L1 expects -* hardware (KVM in this case) to provide separate predictor modes. -* Bare metal isolates VMX root (host) from VMX non-root (guest), but -* doesn't isolate different VMCSs, i.e. in this case, doesn't provide -* separate modes for L2 vs L1. -*/ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL )) -indirect_branch_prediction_barrier (); +kvm_nested_vmexit_handle_spec_ctrl (vcpu);/* Update any VMCS fields that might have changed while L2 ran */ vmcs_write32 (VM_EXIT_MSR_LOAD_COUNT , vmx->msr_autoload.host.nr);arch/x86/kvm/x86.h +0
-0@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) kvm_x86_ops.nested_ops->leave_nested (vcpu); }
Quoted 11 lines
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127
Quoted 9 lines
arch/x86/kvm/x86.h +9
-0+/* +* If IBRS is advertised to the vCPU, KVM must flush the indirect branch +* predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in +* this case) to provide separate predictor modes. Bare metal isolates the host +* from the guest, but doesn't isolate different guests from one another (in +* this case L1 and L2). The exception is if bare metal supports same mode IBRS, +* which offers protection within the same mode, and hence protects L1 from L2. +*/ +static inline void kvm_nested_vmexit_handle_spec_ctrl (struct kvm_vcpu *vcpu)Maybe just
kvm_nested_vmexit_handle_ibrs?
I was trying to use a generic name to accomodate any future handling needed for non-IBRS speculation control virtualization. But I could just be overthinking. Happy to take whatever name is agreed upon in during reviews.
Quoted 10 lines
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133
Quoted 7 lines
arch/x86/kvm/x86.h +6
-0+{ +if (cpu_feature_enabled (X86_FEATURE_AMD_IBRS_SAME_MODE )) +return ; + +if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ) || +guest_cpu_cap_has (vcpu,X86_FEATURE_AMD_IBRS ))This is a bit conservative, but I don't think there's any ROI in being more pedantic.
Could you elaborate on this?
Is this about doing the IBPB even if L1 does not actually execute an IBRS? I thought about this for a bit, but otherwise we'd have to intercept the MSR write IIUC, and I am not sure if that's better. Also, that's what we are already doing so I just kept it as-is.
Or maybe about whether we need this on AMD only with AUTOIBRS? The APM is a bit unclear to me in this regard, but I believe may be needed even for 'normal' IBRS.
Quoted 4 lines
For the series,
Reviewed-by Jim Mattson <[email protected]>
Thanks!
Quoted 10 lines
Quoted 10 lines
arch/x86/kvm/x86.h +3
-1+indirect_branch_prediction_barrier (); +} +static inline bool kvm_vcpu_has_run (struct kvm_vcpu *vcpu) {return vcpu->arch.last_vmentry_cpu !=-1 ; --2.48.1.601.g30ceb7b040-goog
Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit
Quoted 99 lines · Yosry Ahmed
Quoted 67 lines · Jim Mattson
Quoted 66 lines · Yosry Ahmed
Commit
2e7eab81425a("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.
Generalize the handling in
nested_vmx_vmexit()by moving it into a generic function, add the AMD handling, and use it innested_svm_vmexit()too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/vmx/nested.c | 11 +---------- arch/x86/kvm/x86.h | 18 ++++++++++++++++++ 3 files changed, 21 insertions(+), 10 deletions(-)arch/x86/kvm/svm/nested.c +2
-0@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)nested_svm_copy_common_state (svm->nested.vmcb02.ptr, svm->vmcb01.ptr); +kvm_nested_vmexit_handle_spec_ctrl (vcpu); +svm_switch_vmcb (svm, &svm->vmcb01);/* arch/x86/kvm/vmx/nested.c +1
-10@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmx_switch_vmcs (vcpu, &vmx->vmcs01); -/* -* If IBRS is advertised to the vCPU, KVM must flush the indirect -* branch predictors when transitioning from L2 to L1, as L1 expects -* hardware (KVM in this case) to provide separate predictor modes. -* Bare metal isolates VMX root (host) from VMX non-root (guest), but -* doesn't isolate different VMCSs, i.e. in this case, doesn't provide -* separate modes for L2 vs L1. -*/ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL )) -indirect_branch_prediction_barrier (); +kvm_nested_vmexit_handle_spec_ctrl (vcpu);/* Update any VMCS fields that might have changed while L2 ran */ vmcs_write32 (VM_EXIT_MSR_LOAD_COUNT , vmx->msr_autoload.host.nr);arch/x86/kvm/x86.h +0
-0@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) kvm_x86_ops.nested_ops->leave_nested (vcpu); }Quoted 11 lines
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127
Quoted 9 lines
arch/x86/kvm/x86.h +9
-0+/* +* If IBRS is advertised to the vCPU, KVM must flush the indirect branch +* predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in +* this case) to provide separate predictor modes. Bare metal isolates the host +* from the guest, but doesn't isolate different guests from one another (in +* this case L1 and L2). The exception is if bare metal supports same mode IBRS, +* which offers protection within the same mode, and hence protects L1 from L2. +*/ +static inline void kvm_nested_vmexit_handle_spec_ctrl (struct kvm_vcpu *vcpu)Maybe just
kvm_nested_vmexit_handle_ibrs?I was trying to use a generic name to accomodate any future handling needed for non-IBRS speculation control virtualization. But I could just be overthinking. Happy to take whatever name is agreed upon in during reviews.
Quoted 10 lines
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133
Quoted 7 lines
arch/x86/kvm/x86.h +6
-0+{ +if (cpu_feature_enabled (X86_FEATURE_AMD_IBRS_SAME_MODE )) +return ; + +if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ) || +guest_cpu_cap_has (vcpu,X86_FEATURE_AMD_IBRS ))This is a bit conservative, but I don't think there's any ROI in being more pedantic.
Could you elaborate on this?
Quoted 8 lines
Is this about doing the IBPB even if L1 does not actually execute an IBRS? I thought about this for a bit, but otherwise we'd have to intercept the MSR write IIUC, and I am not sure if that's better. Also, that's what we are already doing so I just kept it as-is.
Or maybe about whether we need this on AMD only with AUTOIBRS? The APM is a bit unclear to me in this regard, but I believe may be needed even for 'normal' IBRS.
If IA32_SPEC_CTRL.IBRS is clear at emulated VM-exit, then this IBPB is unnecessary.
However, since the host (L1) is running in a de-privileged prediction domain, simply setting IA32_SPEC_CTRL.IBRS in the future won't protect it from the guest (L2) that just exited. If we don't eagerly perform an IBPB now, then L0 would have to intercept WRMSR(IA32_SPEC_CTRL) from L1 so that we can issue an IBPB in the future, if L1 ever sets IA32_SPEC_CTRL.IBRS.
Eagerly performing an IBPB now seems like the better option.
Quoted 17 lines
Quoted 4 lines
For the series,
Reviewed-by Jim Mattson <[email protected]>Thanks!
Quoted 10 lines
Quoted 10 lines
arch/x86/kvm/x86.h +3
-1+indirect_branch_prediction_barrier (); +} +static inline bool kvm_vcpu_has_run (struct kvm_vcpu *vcpu) {return vcpu->arch.last_vmentry_cpu !=-1 ; --2.48.1.601.g30ceb7b040-goog
Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit
Quoted 112 lines · Jim Mattson
Quoted 99 lines · Yosry Ahmed
Quoted 67 lines · Jim Mattson
Quoted 66 lines · Yosry Ahmed
Commit
2e7eab81425a("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.
Generalize the handling in
nested_vmx_vmexit()by moving it into a generic function, add the AMD handling, and use it innested_svm_vmexit()too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/vmx/nested.c | 11 +---------- arch/x86/kvm/x86.h | 18 ++++++++++++++++++ 3 files changed, 21 insertions(+), 10 deletions(-)arch/x86/kvm/svm/nested.c +2
-0@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)nested_svm_copy_common_state (svm->nested.vmcb02.ptr, svm->vmcb01.ptr); +kvm_nested_vmexit_handle_spec_ctrl (vcpu); +svm_switch_vmcb (svm, &svm->vmcb01);/* arch/x86/kvm/vmx/nested.c +1
-10@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmx_switch_vmcs (vcpu, &vmx->vmcs01); -/* -* If IBRS is advertised to the vCPU, KVM must flush the indirect -* branch predictors when transitioning from L2 to L1, as L1 expects -* hardware (KVM in this case) to provide separate predictor modes. -* Bare metal isolates VMX root (host) from VMX non-root (guest), but -* doesn't isolate different VMCSs, i.e. in this case, doesn't provide -* separate modes for L2 vs L1. -*/ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL )) -indirect_branch_prediction_barrier (); +kvm_nested_vmexit_handle_spec_ctrl (vcpu);/* Update any VMCS fields that might have changed while L2 ran */ vmcs_write32 (VM_EXIT_MSR_LOAD_COUNT , vmx->msr_autoload.host.nr);arch/x86/kvm/x86.h +0
-0@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) kvm_x86_ops.nested_ops->leave_nested (vcpu); }Quoted 11 lines
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127
Quoted 9 lines
arch/x86/kvm/x86.h +9
-0+/* +* If IBRS is advertised to the vCPU, KVM must flush the indirect branch +* predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in +* this case) to provide separate predictor modes. Bare metal isolates the host +* from the guest, but doesn't isolate different guests from one another (in +* this case L1 and L2). The exception is if bare metal supports same mode IBRS, +* which offers protection within the same mode, and hence protects L1 from L2. +*/ +static inline void kvm_nested_vmexit_handle_spec_ctrl (struct kvm_vcpu *vcpu)Maybe just
kvm_nested_vmexit_handle_ibrs?I was trying to use a generic name to accomodate any future handling needed for non-IBRS speculation control virtualization. But I could just be overthinking. Happy to take whatever name is agreed upon in during reviews.
Quoted 10 lines
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133
Quoted 7 lines
arch/x86/kvm/x86.h +6
-0+{ +if (cpu_feature_enabled (X86_FEATURE_AMD_IBRS_SAME_MODE )) +return ; + +if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ) || +guest_cpu_cap_has (vcpu,X86_FEATURE_AMD_IBRS ))This is a bit conservative, but I don't think there's any ROI in being more pedantic.
Could you elaborate on this?
Quoted 8 lines
Is this about doing the IBPB even if L1 does not actually execute an IBRS? I thought about this for a bit, but otherwise we'd have to intercept the MSR write IIUC, and I am not sure if that's better. Also, that's what we are already doing so I just kept it as-is.
Or maybe about whether we need this on AMD only with AUTOIBRS? The APM is a bit unclear to me in this regard, but I believe may be needed even for 'normal' IBRS.
If IA32_SPEC_CTRL.IBRS is clear at emulated VM-exit, then this IBPB is unnecessary.
Quoted 6 lines
However, since the host (L1) is running in a de-privileged prediction domain, simply setting IA32_SPEC_CTRL.IBRS in the future won't protect it from the guest (L2) that just exited. If we don't eagerly perform an IBPB now, then L0 would have to intercept
WRMSR(IA32_SPEC_CTRL)from L1 so that we can issue an IBPB in the future, if L1 ever sets IA32_SPEC_CTRL.IBRS.
Right, that's what I meant by "we'd have to intercept the MSR write" above, but I didn't put it as eloquently as you just did :)
We'd also need to have different handling for eIBRS/AUTOIBRS. It would basically be:
if eIBRS/AUTOIBRS is enabled by L1:
- Do not intercept IBRS MSR writes
- Always IBPB on emulated VM-exits (unless IbrsSameMode).
else if IBRS is advertised to L1:
- Intercept IBRS MSR writes and do an IBPB.
- Do not IBPB on emulated VM-exits.
We'd basically have two modes of IBRS virtualization and we'd need to switch between them at runtime according to L1's setting of eIBRS/AUTOIBRS.
We can simplify it if we always intercept IBRS MSR writes assuming L1 won't do them with eIBRS/AUTOIBRS anyway, so this becomes:
- On emulated VM-exits, IBPB if eIBRS/AUTOIBRS is enabled (unless IbrsSameMode).
- On IBRS MSR writes, do an IBPB.
Simpler, but not sure if it buys us much.
Quoted 2 lines
Eagerly performing an IBPB now seems like the better option.
So yeah I definitely agree, unless we get regression reports caused by the IBPB on emulated VM-exits, and the MSR write interception turns out to be an improvement.
Re: [PATCH 3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit
Quoted 145 lines · Yosry Ahmed
Quoted 112 lines · Jim Mattson
Quoted 99 lines · Yosry Ahmed
Quoted 67 lines · Jim Mattson
Quoted 66 lines · Yosry Ahmed
Commit
2e7eab81425a("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS") added an IBPB in the emulated VM-exit path on Intel to properly virtualize IBRS by providing separate predictor modes for L1 and L2.AMD requires similar handling, except when IbrsSameMode is enumerated by the host CPU (which is the case on most/all AMD CPUs). With IbrsSameMode, hardware IBRS is sufficient and no extra handling is needed from KVM.
Generalize the handling in
nested_vmx_vmexit()by moving it into a generic function, add the AMD handling, and use it innested_svm_vmexit()too. The main reason for using a generic function is to have a single place to park the huge comment about virtualizing IBRS.Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/vmx/nested.c | 11 +---------- arch/x86/kvm/x86.h | 18 ++++++++++++++++++ 3 files changed, 21 insertions(+), 10 deletions(-)arch/x86/kvm/svm/nested.c +2
-0@@ -1041,6 +1041,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)nested_svm_copy_common_state (svm->nested.vmcb02.ptr, svm->vmcb01.ptr); +kvm_nested_vmexit_handle_spec_ctrl (vcpu); +svm_switch_vmcb (svm, &svm->vmcb01);/* arch/x86/kvm/vmx/nested.c +1
-10@@ -5018,16 +5018,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmx_switch_vmcs (vcpu, &vmx->vmcs01); -/* -* If IBRS is advertised to the vCPU, KVM must flush the indirect -* branch predictors when transitioning from L2 to L1, as L1 expects -* hardware (KVM in this case) to provide separate predictor modes. -* Bare metal isolates VMX root (host) from VMX non-root (guest), but -* doesn't isolate different VMCSs, i.e. in this case, doesn't provide -* separate modes for L2 vs L1. -*/ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL )) -indirect_branch_prediction_barrier (); +kvm_nested_vmexit_handle_spec_ctrl (vcpu);/* Update any VMCS fields that might have changed while L2 ran */ vmcs_write32 (VM_EXIT_MSR_LOAD_COUNT , vmx->msr_autoload.host.nr);arch/x86/kvm/x86.h +0
-0@@ -116,6 +116,24 @@ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) kvm_x86_ops.nested_ops->leave_nested (vcpu); }Quoted 11 lines
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 127
Quoted 9 lines
arch/x86/kvm/x86.h +9
-0+/* +* If IBRS is advertised to the vCPU, KVM must flush the indirect branch +* predictors when transitioning from L2 to L1, as L1 expects hardware (KVM in +* this case) to provide separate predictor modes. Bare metal isolates the host +* from the guest, but doesn't isolate different guests from one another (in +* this case L1 and L2). The exception is if bare metal supports same mode IBRS, +* which offers protection within the same mode, and hence protects L1 from L2. +*/ +static inline void kvm_nested_vmexit_handle_spec_ctrl (struct kvm_vcpu *vcpu)Maybe just
kvm_nested_vmexit_handle_ibrs?I was trying to use a generic name to accomodate any future handling needed for non-IBRS speculation control virtualization. But I could just be overthinking. Happy to take whatever name is agreed upon in during reviews.
Quoted 10 lines
arch/x86/kvm/x86.h static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) · line 133
Quoted 7 lines
arch/x86/kvm/x86.h +6
-0+{ +if (cpu_feature_enabled (X86_FEATURE_AMD_IBRS_SAME_MODE )) +return ; + +if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ) || +guest_cpu_cap_has (vcpu,X86_FEATURE_AMD_IBRS ))This is a bit conservative, but I don't think there's any ROI in being more pedantic.
Could you elaborate on this?
Quoted 8 lines
Is this about doing the IBPB even if L1 does not actually execute an IBRS? I thought about this for a bit, but otherwise we'd have to intercept the MSR write IIUC, and I am not sure if that's better. Also, that's what we are already doing so I just kept it as-is.
Or maybe about whether we need this on AMD only with AUTOIBRS? The APM is a bit unclear to me in this regard, but I believe may be needed even for 'normal' IBRS.
If IA32_SPEC_CTRL.IBRS is clear at emulated VM-exit, then this IBPB is unnecessary.
Quoted 6 lines
However, since the host (L1) is running in a de-privileged prediction domain, simply setting IA32_SPEC_CTRL.IBRS in the future won't protect it from the guest (L2) that just exited. If we don't eagerly perform an IBPB now, then L0 would have to intercept
WRMSR(IA32_SPEC_CTRL)from L1 so that we can issue an IBPB in the future, if L1 ever sets IA32_SPEC_CTRL.IBRS.Right, that's what I meant by "we'd have to intercept the MSR write" above, but I didn't put it as eloquently as you just did :)
We'd also need to have different handling for eIBRS/AUTOIBRS. It would basically be:
if eIBRS/AUTOIBRS is enabled by L1:
- Do not intercept IBRS MSR writes
- Always IBPB on emulated VM-exits (unless IbrsSameMode).
else if IBRS is advertised to L1:
- Intercept IBRS MSR writes and do an IBPB.
- Do not IBPB on emulated VM-exits.We'd basically have two modes of IBRS virtualization and we'd need to switch between them at runtime according to L1's setting of eIBRS/AUTOIBRS.
We can simplify it if we always intercept IBRS MSR writes assuming L1 won't do them with eIBRS/AUTOIBRS anyway, so this becomes:
- On emulated VM-exits, IBPB if eIBRS/AUTOIBRS is enabled (unless IbrsSameMode).
- On IBRS MSR writes, do an IBPB.
Quoted 8 lines
Simpler, but not sure if it buys us much.
Quoted 2 lines
Eagerly performing an IBPB now seems like the better option.
So yeah I definitely agree, unless we get regression reports caused by the IBPB on emulated VM-exits, and the MSR write interception turns out to be an improvement.
For "normal" configurations, I would expect L1 to set eIBRS or autoIBRS on modern CPUs. In that case, the IBPB is required. However, if L1 is a Linux OS booted with mitigations=off, the IBPB is probably gratuitous.
On the Intel side, KVM tries to do some optimization of the case where the guest doesn't use IA32_SPEC_CTRL. Since writes to IA32_SPEC_CTRL are intercepted until the first non-zero value is written, I suppose you could skip the IBPB when writes to IA32_SPEC_CTRL are intercepted, and then just blindly perform an IBPB when disabling the intercept in vmx_set_msr() if nested is enabled. Again, though, I don't think this is worth it.
Quoted 10 lines · Yosry Ahmed
To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.
Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.
I am not sure if this series would land through the kvm-x86 tree or the tip/x86 tree.
Sean, any thoughts about this (or general feedback about this series)?
Quoted 17 lines
Yosry Ahmed (3):
x86/cpufeatures: DefineX86_FEATURE_AMD_IBRS_SAME_MODE
KVM: x86: Propagate AMD's IbrsSameMode to the guest
KVM: x86: Generalize IBRS virtualization on emulated VM-exitarch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kvm/cpuid.c | 1 + arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/vmx/nested.c | 11 +---------- arch/x86/kvm/x86.h | 18 ++++++++++++++++++ tools/arch/x86/include/asm/cpufeatures.h | 1 + 6 files changed, 24 insertions(+), 10 deletions(-)Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Sean Christopherson
maintainerQuoted 9 lines · Yosry Ahmed
Quoted 8 lines · Yosry Ahmed
To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.
Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.
Quoted 4 lines
Quoted 2 lines
I am not sure if this series would land through the kvm-x86 tree or the tip/x86 tree.
Sean, any thoughts about this (or general feedback about this series)?
No feedback, I just you and Jim to get mitigation stuff right far more than I trust myself :-)
I'm planning on grabbing this for 6.16.
Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Yosry Ahmed
Quoted 10 lines · Yosry Ahmed
To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.
Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.
I am not sure if this series would land through the kvm-x86 tree or the tip/x86 tree.
Quoted 6 lines
Quoted 1 line
Sean, any thoughts about this (or general feedback about this series)?
No feedback, I just you and Jim to get mitigation stuff right far more than I trust myself :-)
I'm planning on grabbing this for 6.16.
Awesome, thanks!
Quoted 9 lines · Yosry Ahmed
To properly virtualize IBRS on Intel, an IBPB is executed on emulated VM-exits to provide separate predictor modes for L1 and L2.
Similar handling is theoretically needed for AMD, unless IbrsSameMode is enumerated by the CPU (which should be the case for most/all CPUs anyway). For correctness and clarity, this series generalizes the handling to apply for both Intel and AMD as needed.
[...]
Applied to kvm-x86 misc, thanks!
[1/3] x86/cpufeatures: Define X86_FEATURE_AMD_IBRS_SAME_MODE
commit: 9a7cb00a8ff7380a09fa75287a3f2642c472d562
[2/3] KVM: x86: Propagate AMD's IbrsSameMode to the guest
commit: 65ca2872015c232d6743b497e3c08ff96596b917
[3/3] KVM: x86: Generalize IBRS virtualization on emulated VM-exit
commit: 656d9624bd21d35499eaa5ee97fda6def62901c8