IBPB cleanups and a fixup
cover IBPB cleanups and a fixup
This series removes X86_FEATURE_USE_IBPB, and fixes a KVM nVMX bug in the process. The motivation is mostly the confusing name of X86_FEATURE_USE_IBPB, which sounds like it controls IBPBs in general, but it only controls IBPBs for spectre_v2_mitigation. A side effect of this confusion is the nVMX bug, where virtualizing IBRS correctly depends on the spectre_v2_user mitigation.
The feature bit is mostly redundant, except in controlling the IBPB in the vCPU load path. For that, a separate static branch is introduced, similar to switch_mm_*_ibpb.
I wanted to do more, but decided to stay conservative. I was mainly hoping to merge indirect_branch_prediction_barrier() with entry_ibpb() to have a single IBPB primitive that always stuffs the RSB if the IBPB doesn't, but this would add some overhead in paths that currently use indirect_branch_prediction_barrier(), and I was not sure if that's acceptable.
For the record, my measurements of the latency of indirect_branch_prediction_barrier() and entry_ibpb() on Rome and Milan (both do not have X86_FEATURE_AMD_IBPB_RET) are as follows:
Rome:
400ns (indirect_branch_prediction_barrier) vs 500ns (entry_ibpb)
Milan:
220ns (indirect_branch_prediction_barrier) vs 280ns (entry_ibpb)
I also wanted to move controlling the IBPB on vCPU load from being under spectre_v2_user to spectre_v2, because "user" in a lot of mitigation contexts does not include VMs.
Just laying out these thoughts in case others have any comments.
Yosry Ahmed (6):
x86/bugs: Move the X86_FEATURE_USE_IBPB check into callers x86/mm: Remove X86_FEATURE_USE_IBPB checks in cond_mitigation() x86/bugs: Remove the X86_FEATURE_USE_IBPB check in ib_prctl_set() x86/bugs: Use a static branch to guard IBPB on vCPU load
KVM: nVMX: Always use IBPB to properly virtualize IBRS x86/bugs: Remove X86_FEATURE_USE_IBPB
arch/x86/include/asm/cpufeatures.h | 1 - arch/x86/include/asm/nospec-branch.h | 4 +++- arch/x86/kernel/cpu/bugs.c | 7 +++++-- arch/x86/kvm/svm/svm.c | 3 ++- arch/x86/kvm/vmx/vmx.c | 3 ++- arch/x86/mm/tlb.c | 3 +-- tools/arch/x86/include/asm/cpufeatures.h | 1 - 7 files changed, 13 insertions(+), 9 deletions(-)
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
PATCH 1/6 x86/bugs: Move the X86_FEATURE_USE_IBPB check into callers
indirect_branch_prediction_barrier() only performs the MSR write if X86_FEATURE_USE_IBPB is set, using alternative_msr_write(). In preparation for removing X86_FEATURE_USE_IBPB, move the feature check into the callers so that they can be addressed one-by-one, and use X86_FEATURE_IBPB instead to guard the MSR write.
--- arch/x86/include/asm/nospec-branch.h | 2 +- arch/x86/kernel/cpu/bugs.c | 2 +- arch/x86/kvm/svm/svm.c | 3 ++- arch/x86/kvm/vmx/nested.c | 3 ++- arch/x86/kvm/vmx/vmx.c | 3 ++- arch/x86/mm/tlb.c | 7 ++++--- 6 files changed, 12 insertions(+), 8 deletions(-)
arch/x86/include/asm/nospec-branch.h +1 -1
@@ -515,7 +515,7 @@ extern u64 x86_pred_cmd;static inline void indirect_branch_prediction_barrier (void ) { -alternative_msr_write (MSR_IA32_PRED_CMD , x86_pred_cmd,X86_FEATURE_USE_IBPB ); +alternative_msr_write (MSR_IA32_PRED_CMD , x86_pred_cmd,X86_FEATURE_IBPB ); }/* The Intel SPEC CTRL MSR base value cache */
arch/x86/kernel/cpu/bugs.c +1 -1
@@ -2272,7 +2272,7 @@ static int ib_prctl_set(struct task_struct *task, unsigned long ctrl)if (ctrl ==PR_SPEC_FORCE_DISABLE )task_set_spec_ib_force_disable (task);task_update_spec_tif (task); -if (task == current) +if (task == current &&cpu_feature_enabled (X86_FEATURE_USE_IBPB ))indirect_branch_prediction_barrier ();break ;default :
arch/x86/kvm/svm/svm.c +2 -1
@@ -1559,7 +1559,8 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (sd->current_vmcb != svm->vmcb) { sd->current_vmcb = svm->vmcb; -if (!cpu_feature_enabled (X86_FEATURE_IBPB_ON_VMEXIT )) +if (!cpu_feature_enabled (X86_FEATURE_IBPB_ON_VMEXIT ) && +cpu_feature_enabled (X86_FEATURE_USE_IBPB ))indirect_branch_prediction_barrier (); }if (kvm_vcpu_apicv_active (vcpu))
arch/x86/kvm/vmx/nested.c +2 -1
@@ -5026,7 +5026,8 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,* doesn't isolate different VMCSs, i.e. in this case, doesn't provide * separate modes for L2 vs L1. */ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL )) +if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ) && +cpu_feature_enabled (X86_FEATURE_USE_IBPB ))indirect_branch_prediction_barrier ();/* Update any VMCS fields that might have changed while L2 ran */
arch/x86/kvm/vmx/vmx.c +2 -1
@@ -1478,7 +1478,8 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu,* may switch the active VMCS multiple times). */ if (!buddy ||WARN_ON_ONCE (buddy->vmcs != prev)) -indirect_branch_prediction_barrier (); +if (cpu_feature_enabled (X86_FEATURE_USE_IBPB )) +indirect_branch_prediction_barrier (); }if (!already_loaded) {
arch/x86/mm/tlb.c +4 -3
@@ -437,7 +437,8 @@ static void cond_mitigation(struct task_struct *next)* both have the IBPB bit set. */ if (next_mm != prev_mm && - (next_mm | prev_mm) &LAST_USER_MM_IBPB ) + (next_mm | prev_mm) &LAST_USER_MM_IBPB && +cpu_feature_enabled (X86_FEATURE_USE_IBPB ))indirect_branch_prediction_barrier (); }
@@ -447,8 +448,8 @@ static void cond_mitigation(struct task_struct *next)* different context than the user space task which ran * last on this CPU. */ -if ((prev_mm & ~LAST_USER_MM_SPEC_MASK ) != - (unsigned long )next->mm) +if ((prev_mm & ~LAST_USER_MM_SPEC_MASK ) != (unsigned long )next->mm && +cpu_feature_enabled (X86_FEATURE_USE_IBPB ))indirect_branch_prediction_barrier (); }
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Sean Christopherson
maintainerRe: [PATCH 1/6] x86/bugs: Move the X86_FEATURE_USE_IBPB check into callers
arch/x86/kvm/vmx/vmx.c void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu, · line 1481
Quoted 10 lines · Yosry Ahmed
arch/x86/kvm/vmx/vmx.c +1
-1@@ -1478,7 +1478,8 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu,* may switch the active VMCS multiple times). */ if (!buddy ||WARN_ON_ONCE (buddy->vmcs != prev)) -indirect_branch_prediction_barrier (); +if (cpu_feature_enabled (X86_FEATURE_USE_IBPB ))
Combine this into a single if-statement, to make it readable and because as-is the outer if would need curly braces.
And since this check will stay around in the form of a static_branch, I vote to check it first so that the checks on "buddy" are elided if vcpu_load_ibpb is disabled. That'll mean the WARN_ON_ONCE() won't fire if we have a bug and someone is running with mitigations disabled, but I'm a-ok with that.
Re: [PATCH 1/6] x86/bugs: Move the X86_FEATURE_USE_IBPB check into callers
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Yosry Ahmed
arch/x86/kvm/vmx/vmx.c +0
-0index 6c56d5235f0f3..729a8ee24037b 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -1478,7 +1478,8 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu, * may switch the active VMCS multiple times). */ if (!buddy ||WARN_ON_ONCE (buddy->vmcs != prev)) -indirect_branch_prediction_barrier ();
Quoted 8 lines
Quoted 1 line
+ if (
cpu_feature_enabled(X86_FEATURE_USE_IBPB))Combine this into a single if-statement, to make it readable and because as-is the outer if would need curly braces.
And since this check will stay around in the form of astatic_branch, I vote to check it first so that the checks on "buddy" are elided ifvcpu_load_ibpbis disabled. That'll mean theWARN_ON_ONCE()won't fire if we have a bug and someone is running with mitigations disabled, but I'm a-ok with that.
SGTM, will do that in the next version. Thanks!
PATCH 2/6 x86/mm: Remove X86_FEATURE_USE_IBPB checks in cond_mitigation()
In reply to Yosry AhmedThe check is performed when either switch_mm_cond_ibpb or switch_mm_always_ibpb is set. In both cases, X86_FEATURE_USE_IBPB is always set. Remove the redundant check.
--- arch/x86/mm/tlb.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
arch/x86/mm/tlb.c +2 -4
@@ -437,8 +437,7 @@ static void cond_mitigation(struct task_struct *next)* both have the IBPB bit set. */ if (next_mm != prev_mm && - (next_mm | prev_mm) &LAST_USER_MM_IBPB && -cpu_feature_enabled (X86_FEATURE_USE_IBPB )) + (next_mm | prev_mm) &LAST_USER_MM_IBPB )indirect_branch_prediction_barrier (); }
@@ -448,8 +447,7 @@ static void cond_mitigation(struct task_struct *next)* different context than the user space task which ran * last on this CPU. */ -if ((prev_mm & ~LAST_USER_MM_SPEC_MASK ) != (unsigned long )next->mm && -cpu_feature_enabled (X86_FEATURE_USE_IBPB )) +if ((prev_mm & ~LAST_USER_MM_SPEC_MASK ) != (unsigned long )next->mm)indirect_branch_prediction_barrier (); }
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
PATCH 3/6 x86/bugs: Remove the X86_FEATURE_USE_IBPB check in ib_prctl_set()
In reply to Yosry AhmedIf X86_FEATURE_USE_IBPB is not set, then both spectre_v2_user_ibpb and spectre_v2_user_stibp are set to SPECTRE_V2_USER_NONE in spectre_v2_user_select_mitigation(). Since ib_prctl_set() already checks for this before performing the IBPB, the X86_FEATURE_USE_IBPB check is redundant. Remove it.
--- arch/x86/kernel/cpu/bugs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/bugs.c +1 -1
@@ -2272,7 +2272,7 @@ static int ib_prctl_set(struct task_struct *task, unsigned long ctrl)if (ctrl ==PR_SPEC_FORCE_DISABLE )task_set_spec_ib_force_disable (task);task_update_spec_tif (task); -if (task == current &&cpu_feature_enabled (X86_FEATURE_USE_IBPB )) +if (task == current)indirect_branch_prediction_barrier ();break ;default :
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
PATCH 4/6 x86/bugs: Use a static branch to guard IBPB on vCPU load
In reply to Yosry AhmedInstead of using X86_FEATURE_USE_IBPB to guard the IBPB execution in the vCPU load path, introduce a static branch, similar to switch_mm_*_ibpb.
This makes it obvious in spectre_v2_user_select_mitigation() what exactly is being toggled, instead of the unclear X86_FEATURE_USE_IBPB (which will be shortly removed). It also provides more fine-grained control, making it simpler to change/add paths that control the IBPB in the vCPU load path without affecting other IBPBs.
--- arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 5 +++++ arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- 4 files changed, 9 insertions(+), 2 deletions(-)
arch/x86/include/asm/nospec-branch.h +2 -0
@@ -552,6 +552,8 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp);DECLARE_STATIC_KEY_FALSE (switch_mm_cond_ibpb);DECLARE_STATIC_KEY_FALSE (switch_mm_always_ibpb); +DECLARE_STATIC_KEY_FALSE (vcpu_load_ibpb); +DECLARE_STATIC_KEY_FALSE (mds_idle_clear);DECLARE_STATIC_KEY_FALSE (switch_mm_cond_l1d_flush);
arch/x86/kernel/cpu/bugs.c +5 -0
@@ -113,6 +113,10 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);/* Control unconditional IBPB in switch_mm() */ DEFINE_STATIC_KEY_FALSE (switch_mm_always_ibpb); +/* Control IBPB on vCPU load */ +DEFINE_STATIC_KEY_FALSE (vcpu_load_ibpb); +EXPORT_SYMBOL_GPL (vcpu_load_ibpb); +/* Control MDS CPU buffer clear before idling (halt, mwait) */ DEFINE_STATIC_KEY_FALSE (mds_idle_clear);EXPORT_SYMBOL_GPL (mds_idle_clear);
@@ -1365,6 +1369,7 @@ spectre_v2_user_select_mitigation(void)/* Initialize Indirect Branch Prediction Barrier */ if (boot_cpu_has (X86_FEATURE_IBPB )) {setup_force_cpu_cap (X86_FEATURE_USE_IBPB ); +static_branch_enable (&vcpu_load_ibpb); spectre_v2_user_ibpb = mode;switch (cmd) {
arch/x86/kvm/svm/svm.c +1 -1
@@ -1560,7 +1560,7 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu) sd->current_vmcb = svm->vmcb;if (!cpu_feature_enabled (X86_FEATURE_IBPB_ON_VMEXIT ) && -cpu_feature_enabled (X86_FEATURE_USE_IBPB )) +static_branch_likely (&vcpu_load_ibpb))indirect_branch_prediction_barrier (); }if (kvm_vcpu_apicv_active (vcpu))
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -1478,7 +1478,7 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu,* may switch the active VMCS multiple times). */ if (!buddy ||WARN_ON_ONCE (buddy->vmcs != prev)) -if (cpu_feature_enabled (X86_FEATURE_USE_IBPB )) +if (static_branch_likely (&vcpu_load_ibpb))indirect_branch_prediction_barrier (); }
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Sean Christopherson
maintainerRe: [PATCH 4/6] x86/bugs: Use a static branch to guard IBPB on vCPU load
Quoted 22 lines · Yosry Ahmed
Instead of using
X86_FEATURE_USE_IBPBto guard the IBPB execution in the vCPU load path, introduce a static branch, similar to switch_mm_*_ibpb.This makes it obvious in
spectre_v2_user_select_mitigation()what exactly is being toggled, instead of the unclearX86_FEATURE_USE_IBPB(which will be shortly removed). It also provides more fine-grained control, making it simpler to change/add paths that control the IBPB in the vCPU load path without affecting other IBPBs.Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 5 +++++ arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- 4 files changed, 9 insertions(+), 2 deletions(-)arch/x86/include/asm/nospec-branch.h +0
-0
arch/x86/include/asm/nospec-branch.h DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp); · line 555
Quoted 4 lines
arch/x86/include/asm/nospec-branch.h +1
-0@@ -552,6 +552,8 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp);DECLARE_STATIC_KEY_FALSE (switch_mm_cond_ibpb);DECLARE_STATIC_KEY_FALSE (switch_mm_always_ibpb); +DECLARE_STATIC_KEY_FALSE (vcpu_load_ibpb);
How about ibpb_on_vcpu_load? To make it easy for readers to understand exactly what the knob controls.
Re: [PATCH 4/6] x86/bugs: Use a static branch to guard IBPB on vCPU load
Quoted 22 lines · Sean Christopherson
Quoted 20 lines · Yosry Ahmed
Instead of using
X86_FEATURE_USE_IBPBto guard the IBPB execution in the
vCPU load path, introduce a static branch, similar to switch_mm_*_ibpb.This makes it obvious in
spectre_v2_user_select_mitigation()what exactly is being toggled, instead of the unclearX86_FEATURE_USE_IBPB(which will be shortly removed). It also provides more fine-grained control, making it simpler to change/add paths that control the IBPB in the vCPU load path without affecting other IBPBs.Signed-off-by Yosry Ahmed <[email protected]>---
arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 5 +++++ arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- 4 files changed, 9 insertions(+), 2 deletions(-)
Quoted 7 lines
Quoted 7 lines
diff --git
a/arch/x86/include/asm/nospec-branch.hb/arch/x86/include/asm/nospec-branch.hindex7cbb76a2434b9..a22836c5fb338100644
---a/arch/x86/include/asm/nospec-branch.h
+++b/arch/x86/include/asm/nospec-branch.h
@@ -552,6 +552,8 @@DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp);DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);
DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);
Quoted 1 line
Quoted 1 line
+DECLARE_STATIC_KEY_FALSE(vcpu_load_ibpb);
Quoted 4 lines
Quoted 1 line
How about
ibpb_on_vcpu_load? To make it easy for readers to understand exactly what the knob controls.
I was trying to remain consistent with the existing static branches' names, but I am fine with ibpb_on_vcpu_load if others don't object.
Sean Christopherson
maintainerRe: [PATCH 4/6] x86/bugs: Use a static branch to guard IBPB on vCPU load
Quoted 32 lines · Yosry Ahmed
Quoted 22 lines · Sean Christopherson
Quoted 20 lines · Yosry Ahmed
Instead of using
X86_FEATURE_USE_IBPBto guard the IBPB execution in the
vCPU load path, introduce a static branch, similar to switch_mm_*_ibpb.This makes it obvious in
spectre_v2_user_select_mitigation()what exactly is being toggled, instead of the unclearX86_FEATURE_USE_IBPB(which will be shortly removed). It also provides more fine-grained control, making it simpler to change/add paths that control the IBPB in the vCPU load path without affecting other IBPBs.Signed-off-by Yosry Ahmed <[email protected]>---
arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 5 +++++ arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- 4 files changed, 9 insertions(+), 2 deletions(-)Quoted 7 lines
Quoted 7 lines
diff --git
a/arch/x86/include/asm/nospec-branch.hb/arch/x86/include/asm/nospec-branch.hindex7cbb76a2434b9..a22836c5fb338100644
---a/arch/x86/include/asm/nospec-branch.h
+++b/arch/x86/include/asm/nospec-branch.h
@@ -552,6 +552,8 @@DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp);DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);
DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);Quoted 1 line
Quoted 1 line
Quoted 8 lines
+DECLARE_STATIC_KEY_FALSE(vcpu_load_ibpb);
Quoted 4 lines
Quoted 1 line
How about
ibpb_on_vcpu_load? To make it easy for readers to understand exactly what the knob controls.I was trying to remain consistent with the existing static branches' names, but I am fine with
ibpb_on_vcpu_loadif others don't object.
I assumed as much :-) I'm ok with vcpu_load_ibpb if that's what others prefer.
Re: [PATCH 4/6] x86/bugs: Use a static branch to guard IBPB on vCPU load
Quoted 36 lines · Sean Christopherson
Quoted 35 lines · Yosry Ahmed
Quoted 22 lines · Sean Christopherson
Quoted 20 lines · Yosry Ahmed
Instead of using
X86_FEATURE_USE_IBPBto guard the IBPB execution in the
vCPU load path, introduce a static branch, similar to switch_mm_*_ibpb.This makes it obvious in
spectre_v2_user_select_mitigation()what exactly is being toggled, instead of the unclearX86_FEATURE_USE_IBPB(which will be shortly removed). It also provides more fine-grained control, making it simpler to change/add paths that control the IBPB in the vCPU load path without affecting other IBPBs.Signed-off-by Yosry Ahmed <[email protected]>---
arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 5 +++++ arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- 4 files changed, 9 insertions(+), 2 deletions(-)Quoted 7 lines
Quoted 7 lines
diff --git
a/arch/x86/include/asm/nospec-branch.hb/arch/x86/include/asm/nospec-branch.hindex7cbb76a2434b9..a22836c5fb338100644
---a/arch/x86/include/asm/nospec-branch.h
+++b/arch/x86/include/asm/nospec-branch.h
@@ -552,6 +552,8 @@DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp);DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);
DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);Quoted 1 line
Quoted 1 line
+DECLARE_STATIC_KEY_FALSE(vcpu_load_ibpb);
Quoted 2 lines
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
How about
ibpb_on_vcpu_load? To make it easy for readers to understand exactly what the knob controls.I was trying to remain consistent with the existing static branches' names, but I am fine with
ibpb_on_vcpu_loadif others don't object.I assumed as much :-) I'm ok with
vcpu_load_ibpbif that's what others prefer.
To be honest looking at this again I think I prefer consistency, so if you don't mind and others don't chime in I'd rather keep it as-is.
Alternatively I can rename all the static branches (e.g. ibpb_always_on_switch_mm) :P
Sean Christopherson
maintainerRe: [PATCH 4/6] x86/bugs: Use a static branch to guard IBPB on vCPU load
Quoted 40 lines · Yosry Ahmed
Quoted 39 lines · Sean Christopherson
Quoted 38 lines · Yosry Ahmed
Quoted 22 lines · Sean Christopherson
Quoted 20 lines · Yosry Ahmed
Instead of using
X86_FEATURE_USE_IBPBto guard the IBPB execution in the
vCPU load path, introduce a static branch, similar to switch_mm_*_ibpb.This makes it obvious in
spectre_v2_user_select_mitigation()what exactly is being toggled, instead of the unclearX86_FEATURE_USE_IBPB(which will be shortly removed). It also provides more fine-grained control, making it simpler to change/add paths that control the IBPB in the vCPU load path without affecting other IBPBs.Signed-off-by Yosry Ahmed <[email protected]>---
arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 5 +++++ arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- 4 files changed, 9 insertions(+), 2 deletions(-)Quoted 7 lines
Quoted 7 lines
diff --git
a/arch/x86/include/asm/nospec-branch.hb/arch/x86/include/asm/nospec-branch.hindex7cbb76a2434b9..a22836c5fb338100644
---a/arch/x86/include/asm/nospec-branch.h
+++b/arch/x86/include/asm/nospec-branch.h
@@ -552,6 +552,8 @@DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp);DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);
DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);Quoted 1 line
Quoted 1 line
+DECLARE_STATIC_KEY_FALSE(vcpu_load_ibpb);
Quoted 4 lines
Quoted 1 line
How about
ibpb_on_vcpu_load? To make it easy for readers to understand exactly what the knob controls.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I was trying to remain consistent with the existing static branches' names, but I am fine with
ibpb_on_vcpu_loadif others don't object.I assumed as much :-) I'm ok with
vcpu_load_ibpbif that's what others prefer.To be honest looking at this again I think I prefer consistency, so if you don't mind and others don't chime in I'd rather keep it as-is.
Works for me.
Actually, looking at the names again, wouldn't "switch_vcpu_ibpb" be better? KVM doesn't do IBPB on every vCPU load or even every VMCS load, only when a different vCPU is being loaded.
Quoted 2 lines
Alternatively I can rename all the static branches (e.g.
ibpb_always_on_switch_mm) :P
LOL, also works for me.
Re: [PATCH 4/6] x86/bugs: Use a static branch to guard IBPB on vCPU load
Quoted 46 lines · Sean Christopherson
Quoted 45 lines · Yosry Ahmed
Quoted 43 lines · Sean Christopherson
Quoted 32 lines · Yosry Ahmed
Quoted 22 lines · Sean Christopherson
Quoted 20 lines · Yosry Ahmed
Instead of using
X86_FEATURE_USE_IBPBto guard the IBPB execution in the
vCPU load path, introduce a static branch, similar to switch_mm_*_ibpb.This makes it obvious in
spectre_v2_user_select_mitigation()what exactly is being toggled, instead of the unclearX86_FEATURE_USE_IBPB(which will be shortly removed). It also provides more fine-grained control, making it simpler to change/add paths that control the IBPB in the vCPU load path without affecting other IBPBs.Signed-off-by Yosry Ahmed <[email protected]>---
arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/kernel/cpu/bugs.c | 5 +++++ arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- 4 files changed, 9 insertions(+), 2 deletions(-)Quoted 7 lines
Quoted 7 lines
diff --git
a/arch/x86/include/asm/nospec-branch.hb/arch/x86/include/asm/nospec-branch.hindex7cbb76a2434b9..a22836c5fb338100644
---a/arch/x86/include/asm/nospec-branch.h
+++b/arch/x86/include/asm/nospec-branch.h
@@ -552,6 +552,8 @@DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp);DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);
DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);Quoted 1 line
Quoted 1 line
Quoted 8 lines
+DECLARE_STATIC_KEY_FALSE(vcpu_load_ibpb);
Quoted 4 lines
Quoted 1 line
How about
ibpb_on_vcpu_load? To make it easy for readers to understand exactly what the knob controls.I was trying to remain consistent with the existing static branches' names, but I am fine with
ibpb_on_vcpu_loadif others don't object.I assumed as much :-) I'm ok with
vcpu_load_ibpbif that's what others prefer.
Quoted 8 lines
Quoted 2 lines
To be honest looking at this again I think I prefer consistency, so if you don't mind and others don't chime in I'd rather keep it as-is.
Works for me.
Actually, looking at the names again, wouldn't "
switch_vcpu_ibpb" be better? KVM doesn't do IBPB on every vCPU load or even every VMCS load, only when a different vCPU is being loaded.
Sold :)
Quoted 5 lines
Quoted 3 lines
Alternatively I can rename all the static branches (e.g.
ibpb_always_on_switch_mm) :PLOL, also works for me.
PATCH 5/6 KVM: nVMX: Always use IBPB to properly virtualize IBRS
In reply to Yosry AhmedOn synthesized nested VM-exits in VMX, an IBPB is performed if IBRS is advertised to the guest to properly provide separate prediction domains for L1 and L2. However, this is currently conditional on X86_FEATURE_USE_IBPB, which depends on the host spectre_v2_user mitigation.
In short, if spectre_v2_user=no, IBRS is not virtualized correctly and L1 becomes suspectible to attacks from L2. Fix this by performing the IBPB regardless of X86_FEATURE_USE_IBPB.
--- arch/x86/kvm/vmx/nested.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kvm/vmx/nested.c +1 -2
@@ -5026,8 +5026,7 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,* doesn't isolate different VMCSs, i.e. in this case, doesn't provide * separate modes for L2 vs L1. */ -if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ) && -cpu_feature_enabled (X86_FEATURE_USE_IBPB )) +if (guest_cpu_cap_has (vcpu,X86_FEATURE_SPEC_CTRL ))indirect_branch_prediction_barrier ();/* Update any VMCS fields that might have changed while L2 ran */
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Re: [PATCH 5/6] KVM: nVMX: Always use IBPB to properly virtualize IBRS
Quoted 9 lines · Yosry Ahmed
On synthesized nested VM-exits in VMX, an IBPB is performed if IBRS is advertised to the guest to properly provide separate prediction domains for L1 and L2. However, this is currently conditional on
X86_FEATURE_USE_IBPB, which depends on the hostspectre_v2_usermitigation.In short, if spectre_v2_user=no, IBRS is not virtualized correctly and L1 becomes suspectible to attacks from L2. Fix this by performing the
Nit: susceptible.
Quoted 4 lines
IBPB regardless of
X86_FEATURE_USE_IBPB.Fixes 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS")Signed-off-by Yosry Ahmed <[email protected]>
Argh! No doubt, I was burned once again by assuming that a function name (indirect_branch_prediction_barrier) was actually descriptive.
Re: [PATCH 5/6] KVM: nVMX: Always use IBPB to properly virtualize IBRS
In reply to Yosry AhmedQuoted 6 lines · Yosry Ahmed
On synthesized nested VM-exits in VMX, an IBPB is performed if IBRS is advertised to the guest to properly provide separate prediction domains for L1 and L2. However, this is currently conditional on
X86_FEATURE_USE_IBPB, which depends on the hostspectre_v2_usermitigation.
Quoted 7 lines
In short, if spectre_v2_user=no, IBRS is not virtualized correctly and L1 becomes suspectible to attacks from L2. Fix this by performing the IBPB regardless of
X86_FEATURE_USE_IBPB.Fixes 2e7eab81425a ("KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS")Signed-off-by Yosry Ahmed <[email protected]>---
PATCH 6/6 x86/bugs: Remove X86_FEATURE_USE_IBPB
In reply to Yosry AhmedX86_FEATURE_USE_IBPB was introduced in commit 2961298efe1e ("x86/cpufeatures: Clean up Spectre v2 related CPUID flags") to have separate flags for when the CPU supports IBPB (i.e. X86_FEATURE_IBPB) and when an IBPB is actually used to mitigate Spectre v2.
Ever since then, the uses of IBPB expanded. The name became confusing because it does not control all IBPB executions in the kernel. Furthermore, because its name is generic and it's buried within indirect_branch_prediction_barrier(), it's easy to use it not knowing that it is specific to Spectre v2.
X86_FEATURE_USE_IBPB is no longer needed because all the IBPB executions it used to control are now controlled through other means (e.g. switch_mm_*_ibpb static branches). Remove the unused feature bit.
--- arch/x86/include/asm/cpufeatures.h | 1 - arch/x86/kernel/cpu/bugs.c | 1 - tools/arch/x86/include/asm/cpufeatures.h | 1 - 3 files changed, 3 deletions(-)
arch/x86/include/asm/cpufeatures.h +0 -1
@@ -210,7 +210,6 @@#define X86_FEATURE_MBA ( 7*32+18)/* "mba" Memory Bandwidth Allocation */ #define X86_FEATURE_RSB_CTXSW ( 7*32+19)/* Fill RSB on context switches */ #define X86_FEATURE_PERFMON_V2 ( 7*32+20)/* "perfmon_v2" AMD Performance Monitoring Version 2 */ -#define X86_FEATURE_USE_IBPB ( 7*32+21)/* Indirect Branch Prediction Barrier enabled */ #define X86_FEATURE_USE_IBRS_FW ( 7*32+22)/* Use IBRS during runtime firmware calls */ #define X86_FEATURE_SPEC_STORE_BYPASS_DISABLE ( 7*32+23)/* Disable Speculative Store Bypass. */ #define X86_FEATURE_LS_CFG_SSBD ( 7*32+24)/* AMD SSBD implementation via LS_CFG MSR */
arch/x86/kernel/cpu/bugs.c +0 -1
@@ -1368,7 +1368,6 @@ spectre_v2_user_select_mitigation(void)/* Initialize Indirect Branch Prediction Barrier */ if (boot_cpu_has (X86_FEATURE_IBPB )) { -setup_force_cpu_cap (X86_FEATURE_USE_IBPB );static_branch_enable (&vcpu_load_ibpb); spectre_v2_user_ibpb = mode;
tools/arch/x86/include/asm/cpufeatures.h +0 -1
@@ -210,7 +210,6 @@#define X86_FEATURE_MBA ( 7*32+18)/* "mba" Memory Bandwidth Allocation */ #define X86_FEATURE_RSB_CTXSW ( 7*32+19)/* Fill RSB on context switches */ #define X86_FEATURE_PERFMON_V2 ( 7*32+20)/* "perfmon_v2" AMD Performance Monitoring Version 2 */ -#define X86_FEATURE_USE_IBPB ( 7*32+21)/* Indirect Branch Prediction Barrier enabled */ #define X86_FEATURE_USE_IBRS_FW ( 7*32+22)/* Use IBRS during runtime firmware calls */ #define X86_FEATURE_SPEC_STORE_BYPASS_DISABLE ( 7*32+23)/* Disable Speculative Store Bypass. */ #define X86_FEATURE_LS_CFG_SSBD ( 7*32+24)/* AMD SSBD implementation via LS_CFG MSR */
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Josh Poimboeuf
maintainerQuoted 10 lines · Yosry Ahmed
This series removes
X86_FEATURE_USE_IBPB, and fixes a KVM nVMX bug in the process. The motivation is mostly the confusing name ofX86_FEATURE_USE_IBPB, which sounds like it controls IBPBs in general, but it only controls IBPBs forspectre_v2_mitigation. A side effect of this confusion is the nVMX bug, where virtualizing IBRS correctly depends on thespectre_v2_usermitigation.The feature bit is mostly redundant, except in controlling the IBPB in the vCPU load path. For that, a separate static branch is introduced, similar to switch_mm_*_ibpb.
Thanks for doing this. A few months ago I was working on patches to fix the same thing but I got preempted multiple times over.
Quoted 6 lines
I wanted to do more, but decided to stay conservative. I was mainly hoping to merge
indirect_branch_prediction_barrier()withentry_ibpb()to have a single IBPB primitive that always stuffs the RSB if the IBPB doesn't, but this would add some overhead in paths that currently useindirect_branch_prediction_barrier(), and I was not sure if that's acceptable.
We always rely on IBPB clearing RSB, so yes, I'd say that's definitely needed. In fact I had a patch to do exactly that, with it ending up like this:
static inline void indirect_branch_prediction_barrier(void)
{
asm volatile(ALTERNATIVE("", "call write_ibpb", X86_FEATURE_IBPB)
: ASM_CALL_CONSTRAINT
: : "rax", "rcx", "rdx", "memory");
}I also renamed "entry_ibpb" -> "write_ibpb" since it's no longer just for entry code.
Signature · 1 line
Josh
Quoted 22 lines · Josh Poimboeuf
Quoted 10 lines · Yosry Ahmed
This series removes
X86_FEATURE_USE_IBPB, and fixes a KVM nVMX bug in the process. The motivation is mostly the confusing name ofX86_FEATURE_USE_IBPB, which sounds like it controls IBPBs in general, but it only controls IBPBs forspectre_v2_mitigation. A side effect of this confusion is the nVMX bug, where virtualizing IBRS correctly depends on thespectre_v2_usermitigation.The feature bit is mostly redundant, except in controlling the IBPB in the vCPU load path. For that, a separate static branch is introduced, similar to switch_mm_*_ibpb.
Thanks for doing this. A few months ago I was working on patches to fix the same thing but I got preempted multiple times over.
Quoted 6 lines
I wanted to do more, but decided to stay conservative. I was mainly hoping to merge
indirect_branch_prediction_barrier()withentry_ibpb()to have a single IBPB primitive that always stuffs the RSB if the IBPB doesn't, but this would add some overhead in paths that currently useindirect_branch_prediction_barrier(), and I was not sure if that's acceptable.
Quoted 3 lines
We always rely on IBPB clearing RSB, so yes, I'd say that's definitely needed. In fact I had a patch to do exactly that, with it ending up like this:
I was mainly concerned about the overhead this adds, but if it's a requirement then yes we should do it.
Quoted 10 lines
static inline void
indirect_branch_prediction_barrier(void){ asm volatile(ALTERNATIVE("", "call write_ibpb", X86_FEATURE_IBPB) : ASM_CALL_CONSTRAINT : : "rax", "rcx", "rdx", "memory"); }I also renamed "
entry_ibpb" -> "write_ibpb" since it's no longer just for entry code.
Do you want me to add this in this series or do you want to do it on top of it? If you have a patch lying around I can also include it as-is.
Quoted 7 lines · Yosry Ahmed
Quoted 7 lines
static inline void
indirect_branch_prediction_barrier(void){ asm volatile(ALTERNATIVE("", "call write_ibpb", X86_FEATURE_IBPB) : ASM_CALL_CONSTRAINT : : "rax", "rcx", "rdx", "memory"); }
Quoted 5 lines
Quoted 2 lines
I also renamed "
entry_ibpb" -> "write_ibpb" since it's no longer just for entry code.Do you want me to add this in this series or do you want to do it on top of it? If you have a patch lying around I can also include it as-is.
Your patches are already an improvement and can be taken as-is:
I'll try to dust off my patches soon and rebase them on yours.
Signature · 1 line
Josh
Quoted 11 lines · Josh Poimboeuf
Quoted 10 lines · Yosry Ahmed
Quoted 9 lines
static inline void
indirect_branch_prediction_barrier(void){ asm volatile(ALTERNATIVE("", "call write_ibpb", X86_FEATURE_IBPB) : ASM_CALL_CONSTRAINT : : "rax", "rcx", "rdx", "memory"); }I also renamed "
entry_ibpb" -> "write_ibpb" since it's no longer just for entry code.
Quoted 8 lines
Quoted 2 lines
Do you want me to add this in this series or do you want to do it on top of it? If you have a patch lying around I can also include it as-is.
Your patches are already an improvement and can be taken as-is:
Acked-by Josh Poimboeuf <[email protected]>I'll try to dust off my patches soon and rebase them on yours.
SGTM, thanks!
Quoted 3 lines
Signature · 2 lines
-- Josh