KVM: VMX: remove LFENCE in vmx_spec_ctrl_restore_host()
PATCH 1/1 KVM: VMX: remove LFENCE in vmx_spec_ctrl_restore_host()
Remove barrier_nospec(), which translates to LFENCE, in vmx_spec_ctrl_restore_host() as RSB-barriers (as defined by [1]) already exist prior to this point.
This LFENCE was added on commit fc02735b14ff ("KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS") in the 5.19 timeframe; however, commit 2b1299322016 ("x86/speculation: Add RSB VM Exit protections") in 6.0 timeframe added a LFENCE for X86_FEATURE_RSB_VMEXIT_LITE was added directly in vmx_vmexit, prior to CALL vmx_spec_ctrl_restore_host.
For posterity, vmx_spec_ctrl_restore_host also will execute WRMSR to IA32_SPEC_CTRL for X86_FEATURE_KERNEL_IBRS or when guest/host MSR value does not match, which serves as an additional RSB-barrier.
--- arch/x86/kvm/vmx/vmx.c | 2 -- 1 file changed, 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +0 -2
@@ -7130,8 +7130,6 @@ void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,if (cpu_feature_enabled (X86_FEATURE_KERNEL_IBRS ) || vmx->spec_ctrl != hostval)native_wrmsrl (MSR_IA32_SPEC_CTRL , hostval); - -barrier_nospec (); }static fastpath_t vmx_exit_handlers_fastpath (struct kvm_vcpu *vcpu)
Signature · 1 line
2.30.1 (Apple Git-130)
Quoted 10 lines · Jon Kohler
Remove
barrier_nospec(), which translates to LFENCE, invmx_spec_ctrl_restore_host()as RSB-barriers (as defined by [1]) already exist prior to this point.This LFENCE was added on commit
fc02735b14ff("KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS") in the 5.19 timeframe; however, commit2b1299322016("x86/speculation: Add RSB VM Exit protections") in 6.0 timeframe added a LFENCE forX86_FEATURE_RSB_VMEXIT_LITEwas added directly invmx_vmexit, prior to CALLvmx_spec_ctrl_restore_host.
Quoted 8 lines
For posterity,
vmx_spec_ctrl_restore_hostalso will execute WRMSR toIA32_SPEC_CTRLforX86_FEATURE_KERNEL_IBRSor when guest/host MSR value does not match, which serves as an additional RSB-barrier.Fixes fc02735b14ff ("KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS")Fixes 2b1299322016 ("x86/speculation: Add RSB VM Exit protections")
Sorry, I knew I should have put a comment there.
The goal of this barrier_nospec() is to prevent speculative execution from bypassing the SPEC_CTRL write (due to misprediction of the conditional branch, Spectre v1 style). Otherwise the next indirect branch or unbalanced RET could be an attack target.
So any previous LFENCEs before that conditional branch won't help here.
Signature · 1 line
Josh
Quoted 24 lines · Josh Poimboeuf
Quoted 10 lines · Jon Kohler
Remove
barrier_nospec(), which translates to LFENCE, invmx_spec_ctrl_restore_host()as RSB-barriers (as defined by [1]) already exist prior to this point.This LFENCE was added on commit
fc02735b14ff("KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS") in the 5.19 timeframe; however, commit2b1299322016("x86/speculation: Add RSB VM Exit protections") in 6.0 timeframe added a LFENCE forX86_FEATURE_RSB_VMEXIT_LITEwas added directly invmx_vmexit, prior to CALLvmx_spec_ctrl_restore_host.Quoted 8 lines
For posterity,
vmx_spec_ctrl_restore_hostalso will execute WRMSR toIA32_SPEC_CTRLforX86_FEATURE_KERNEL_IBRSor when guest/host MSR value does not match, which serves as an additional RSB-barrier.Fixes fc02735b14ff ("KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS")Fixes 2b1299322016 ("x86/speculation: Add RSB VM Exit protections")
Quoted 11 lines
Sorry, I knew I should have put a comment there.
The goal of this
barrier_nospec()is to prevent speculative execution from bypassing theSPEC_CTRLwrite (due to misprediction of the conditional branch, Spectre v1 style). Otherwise the next indirect branch or unbalanced RET could be an attack target.So any previous LFENCEs before that conditional branch won't help here.
Signature · 1 line
Josh
Ah interesting. Ok, to be clear, thats a guest -> host attack, correct? And such an attack would not at all be thwarted by the first CALL retire + LFENCE that was added on commit 2b1299322016 ("x86/speculation: Add RSB VM Exit protections”)? Sorry to be long winded, just wanting to triple check because the aforementioned commit was added slightly after the original one, and I want to make extra sure that they aren’t solving the same thing.
If that is indeed the case, does that commit need to be revisited at all?
Or are we saying that this Intel vulnerability needs *two* LFENCE’s to keep the host secure?
If that’s the case, that’s fine, and I’m happy to transform this commit into some comments in this area to capture the issue for future onlookers?
Quoted 11 lines · Jon Kohler
Quoted 6 lines
The goal of this
barrier_nospec()is to prevent speculative execution from bypassing theSPEC_CTRLwrite (due to misprediction of the conditional branch, Spectre v1 style). Otherwise the next indirect branch or unbalanced RET could be an attack target.So any previous LFENCEs before that conditional branch won't help here.
Ah interesting. Ok, to be clear, thats a guest -> host attack, correct? And such an attack would not at all be thwarted by the first CALL retire + LFENCE that was added on commit
2b1299322016("x86/speculation: Add RSB VM Exit protections”)?
Right.
Quoted 8 lines
Sorry to be long winded, just wanting to triple check because the aforementioned commit was added slightly after the original one, and I want to make extra sure that they aren’t solving the same thing.
If that is indeed the case, does that commit need to be revisited at all?
Or are we saying that this Intel vulnerability needs *two* LFENCE’s to keep the host secure?
The first LFENCE (FILL_RETURN_BUFFER) forces the CALL to retire so the RSB stuff is guaranteed to take effect before the next unbalanced RET can be speculatively executed.
The second LFENCE (vmx_spec_ctrl_restore_host) forces the conditional branch to retire so the SPEC_CTRL write (potential IBRS/eIBRS enablement) is guaranteed to take effect before the next indirect branch and/or unbalanced RET can be speculatively executed.
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them. So it should be fine to combine them into a single LFENCE after both.
You could for example just remove the first LFENCE. But only for that usage site, i.e. not for other users of FILL_RETURN_BUFFER.
Or, remove them both and add an LFENCE in vmx_vmexit() right after the call to vmx_spec_ctrl_restore_host(). That might be clearer. Then it could have a comment describing its dual purposes.
Signature · 1 line
Josh
Quoted 2 lines · Josh Poimboeuf
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them.
How lucky are you feeling?
You're in C at this point, which means the compiler could have emitted a call to mem{cpy,cmp}() in place of a simple assignment/comparison.
~Andrew
Quoted 8 lines · Andrew Cooper
Quoted 2 lines · Josh Poimboeuf
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them.
How lucky are you feeling?
You're in C at this point, which means the compiler could have emitted a call to mem{cpy,cmp}() in place of a simple assignment/comparison.
But it's only unbalanced RETs we're concerned about, per Intel.
Signature · 1 line
Josh
Quoted 8 lines · Andrew Cooper
Quoted 2 lines · Josh Poimboeuf
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them.
How lucky are you feeling?
You're in C at this point, which means the compiler could have emitted a call to mem{cpy,cmp}() in place of a simple assignment/comparison.
Moving the second LFENCE to the else part of WRMSR should be possible? So that the serialization can be achived either by WRMSR or LFENCE. This saves an LFENCE when host and guest value of MSR_SPEC_CTRL differ.
---
arch/x86/kvm/vmx/vmx.c +2 -2
@@ -7157,8 +7157,8 @@ void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,if (cpu_feature_enabled (X86_FEATURE_KERNEL_IBRS ) || vmx->spec_ctrl != hostval)native_wrmsrl (MSR_IA32_SPEC_CTRL , hostval); - -barrier_nospec (); +else +barrier_nospec (); }static fastpath_t vmx_exit_handlers_fastpath (struct kvm_vcpu *vcpu)
On Wed, May 31, 2023 at 06:24:29PM -0700, Pawan Gupta wrote:
## 2023-05-31
Quoted 5 lines · Andrew Cooper
Quoted 4 lines · Josh Poimboeuf
Quoted 2 lines
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them.
Quoted 8 lines
Quoted 4 lines
How lucky are you feeling?
You're in C at this point, which means the compiler could have emitted a call to mem{cpy,cmp}() in place of a simple assignment/comparison.
Moving the second LFENCE to the else part of WRMSR should be possible? So that the serialization can be achived either by WRMSR or LFENCE. This saves an LFENCE when host and guest value of
MSR_SPEC_CTRLdiffer.
Yes. Though in practice it might not make much of a difference. With wrmsr+lfence, the lfence has nothing to do so it might be almost instantaneous anyway.
Signature · 1 line
Josh
Quoted 17 lines · Josh Poimboeuf
On Wed, May 31, 2023 at 06:24:29PM -0700, Pawan Gupta wrote:
## 2023-05-31
Quoted 10 lines · Andrew Cooper
Quoted 8 lines · Josh Poimboeuf
Quoted 2 lines
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them.
How lucky are you feeling?
You're in C at this point, which means the compiler could have emitted a call to mem{cpy,cmp}() in place of a simple assignment/comparison.
Quoted 10 lines
Quoted 3 lines
Moving the second LFENCE to the else part of WRMSR should be possible? So that the serialization can be achived either by WRMSR or LFENCE. This saves an LFENCE when host and guest value of
MSR_SPEC_CTRLdiffer.Yes. Though in practice it might not make much of a difference. With wrmsr+lfence, the lfence has nothing to do so it might be almost instantaneous anyway.
Signature · 1 line
Josh
Coming back to this, what if we hoisted call vmx_spec_ctrl_restore_host above FILL_RETURN_BUFFER, and dropped this LFENCE as I did here?
That way, we wouldn’t have to mess with the internal LFENCE in nospec-branch.h, and that would act as the “final line of defense” LFENCE.
Would that be acceptable? Or does FILL_RETURN_BUFFER *need* to occur before any sort of calls no matter what?
Thanks,
Jon
Quoted 28 lines · Jon Kohler
Quoted 17 lines · Josh Poimboeuf
On Wed, May 31, 2023 at 06:24:29PM -0700, Pawan Gupta wrote:
## 2023-05-31
Quoted 10 lines · Andrew Cooper
Quoted 8 lines · Josh Poimboeuf
Quoted 2 lines
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them.
How lucky are you feeling?
You're in C at this point, which means the compiler could have emitted a call to mem{cpy,cmp}() in place of a simple assignment/comparison.
Quoted 10 lines
Quoted 3 lines
Moving the second LFENCE to the else part of WRMSR should be possible? So that the serialization can be achived either by WRMSR or LFENCE. This saves an LFENCE when host and guest value of
MSR_SPEC_CTRLdiffer.Yes. Though in practice it might not make much of a difference. With wrmsr+lfence, the lfence has nothing to do so it might be almost instantaneous anyway.
Signature · 1 line
Josh
Quoted 8 lines
Coming back to this, what if we hoisted call
vmx_spec_ctrl_restore_hostaboveFILL_RETURN_BUFFER, and dropped this LFENCE as I did here?That way, we wouldn’t have to mess with the internal LFENCE in
nospec-branch.h, and that would act as the “final line of defense” LFENCE.Would that be acceptable? Or does
FILL_RETURN_BUFFER*need* to occur before any sort of calls no matter what?
If we go by Intel's statement that only unbalanced RETs are a concern, that *might* be ok as long as there's a nice comment above the FILL_RETURN_BUFFER usage site describing the two purposes for the LFENCE.
However, based on Andy's concerns, which I've discussed with him privately (but I'm not qualified to agree or disagree with), we may want to just convert vmx_spec_ctrl_restore_host() to asm. Better safe than sorry. My original implementation of that function was actually asm. I can try to dig up that code.
Signature · 1 line
Josh
Quoted 47 lines · Josh Poimboeuf
Quoted 28 lines · Jon Kohler
Quoted 17 lines · Josh Poimboeuf
On Wed, May 31, 2023 at 06:24:29PM -0700, Pawan Gupta wrote:
## 2023-05-31
Quoted 10 lines · Andrew Cooper
Quoted 8 lines · Josh Poimboeuf
Quoted 2 lines
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them.
How lucky are you feeling?
You're in C at this point, which means the compiler could have emitted a call to mem{cpy,cmp}() in place of a simple assignment/comparison.
Quoted 10 lines
Quoted 3 lines
Moving the second LFENCE to the else part of WRMSR should be possible? So that the serialization can be achived either by WRMSR or LFENCE. This saves an LFENCE when host and guest value of
MSR_SPEC_CTRLdiffer.Yes. Though in practice it might not make much of a difference. With wrmsr+lfence, the lfence has nothing to do so it might be almost instantaneous anyway.
Signature · 1 line
JoshQuoted 8 lines
Coming back to this, what if we hoisted call
vmx_spec_ctrl_restore_hostaboveFILL_RETURN_BUFFER, and dropped this LFENCE as I did here?That way, we wouldn’t have to mess with the internal LFENCE in
nospec-branch.h, and that would act as the “final line of defense” LFENCE.Would that be acceptable? Or does
FILL_RETURN_BUFFER*need* to occur before any sort of calls no matter what?If we go by Intel's statement that only unbalanced RETs are a concern, that *might* be ok as long as there's a nice comment above the
FILL_RETURN_BUFFERusage site describing the two purposes for the LFENCE.
Quoted 8 lines
However, based on Andy's concerns, which I've discussed with him privately (but I'm not qualified to agree or disagree with), we may want to just convert
vmx_spec_ctrl_restore_host()to asm. Better safe than sorry. My original implementation of that function was actually asm. I can try to dig up that code.Signature · 1 line
Josh
RE ASM - that’d be nice, given that the restore guest pre-vmentry is now ASM, would match nicely. Then all the code is in one spot, in one file, in one language.
If you could dig that up so we didn’t have to start from scratch, that’d be great (imho).
Jon
Quoted 39 lines · Jon Kohler
Quoted 39 lines · Josh Poimboeuf
Quoted 34 lines · Jon Kohler
Quoted 17 lines · Josh Poimboeuf
On Wed, May 31, 2023 at 06:24:29PM -0700, Pawan Gupta wrote:
## 2023-05-31
Quoted 10 lines · Andrew Cooper
Quoted 8 lines · Josh Poimboeuf
Quoted 2 lines
So each LFENCE has a distinct purpose. That said, there are no indirect branches or unbalanced RETs between them.
How lucky are you feeling?
You're in C at this point, which means the compiler could have emitted a call to mem{cpy,cmp}() in place of a simple assignment/comparison.
Quoted 10 lines
Quoted 3 lines
Moving the second LFENCE to the else part of WRMSR should be possible? So that the serialization can be achived either by WRMSR or LFENCE. This saves an LFENCE when host and guest value of
MSR_SPEC_CTRLdiffer.Yes. Though in practice it might not make much of a difference. With wrmsr+lfence, the lfence has nothing to do so it might be almost instantaneous anyway.
Signature · 1 line
JoshComing back to this, what if we hoisted call
vmx_spec_ctrl_restore_hostaboveFILL_RETURN_BUFFER, and dropped this LFENCE as I did here?That way, we wouldn’t have to mess with the internal LFENCE in
nospec-branch.h, and that would act as the “final line of defense” LFENCE.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Would that be acceptable? Or does
FILL_RETURN_BUFFER*need* to occur before any sort of calls no matter what?If we go by Intel's statement that only unbalanced RETs are a concern, that *might* be ok as long as there's a nice comment above the
FILL_RETURN_BUFFERusage site describing the two purposes for the LFENCE.
We would then need FILL_RETURN_BUFFER to unconditionally execute LFENCE to account for wrmsr branch misprediction. Currently LFENCE is not executed for !X86_BUG_EIBRS_PBRSB.
Quoted 5 lines
Quoted 5 lines
However, based on Andy's concerns, which I've discussed with him privately (but I'm not qualified to agree or disagree with), we may want to just convert
vmx_spec_ctrl_restore_host()to asm. Better safe than sorry. My original implementation of that function was actually asm. I can try to dig up that code.
Note:
VMexit
CALL
RET
RET <---- This is also a problem if the first call hasn't retired yet.
LFENCE
Converting vmx_spec_ctrl_restore_host() to ASM should be able to take care of this.
Sean Christopherson
maintainerQuoted 34 lines · Pawan Gupta
Quoted 13 lines · Jon Kohler
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Yes. Though in practice it might not make much of a difference. With wrmsr+lfence, the lfence has nothing to do so it might be almost instantaneous anyway.
Signature · 1 line
JoshComing back to this, what if we hoisted call
vmx_spec_ctrl_restore_hostaboveFILL_RETURN_BUFFER, and dropped this LFENCE as I did here?That way, we wouldn’t have to mess with the internal LFENCE in
nospec-branch.h, and that would act as the “final line of defense” LFENCE.Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Would that be acceptable? Or does
FILL_RETURN_BUFFER*need* to occur before any sort of calls no matter what?If we go by Intel's statement that only unbalanced RETs are a concern, that *might* be ok as long as there's a nice comment above the
FILL_RETURN_BUFFERusage site describing the two purposes for the LFENCE.We would then need
FILL_RETURN_BUFFERto unconditionally execute LFENCE to account for wrmsr branch misprediction. Currently LFENCE is not executed for !X86_BUG_EIBRS_PBRSB.Quoted 5 lines
Quoted 5 lines
However, based on Andy's concerns, which I've discussed with him privately (but I'm not qualified to agree or disagree with), we may want to just convert
vmx_spec_ctrl_restore_host()to asm. Better safe than sorry. My original implementation of that function was actually asm. I can try to dig up that code.Note:
Quoted 8 lines
VMexit
CALL
RET
RET <---- This is also a problem if the first call hasn't retired yet.
LFENCEConverting
vmx_spec_ctrl_restore_host()to ASM should be able to take care of this.
Is there an actual bug here, or are we just micro-optimizing something that may or may not need additional optimization? Unless there's a bug to be fixed, moving code into ASM and increasing complexity doesn't seem worthwhile.
Quoted 45 lines · Sean Christopherson
Quoted 40 lines · Pawan Gupta
Quoted 13 lines · Jon Kohler
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Yes. Though in practice it might not make much of a difference. With wrmsr+lfence, the lfence has nothing to do so it might be almost instantaneous anyway.
Signature · 1 line
JoshComing back to this, what if we hoisted call
vmx_spec_ctrl_restore_hostaboveFILL_RETURN_BUFFER, and dropped this LFENCE as I did here?That way, we wouldn’t have to mess with the internal LFENCE in
nospec-branch.h, and that would act as the “final line of defense” LFENCE.Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Would that be acceptable? Or does
FILL_RETURN_BUFFER*need* to occur before any sort of calls no matter what?If we go by Intel's statement that only unbalanced RETs are a concern, that *might* be ok as long as there's a nice comment above the
FILL_RETURN_BUFFERusage site describing the two purposes for the LFENCE.We would then need
FILL_RETURN_BUFFERto unconditionally execute LFENCE to account for wrmsr branch misprediction. Currently LFENCE is not executed for !X86_BUG_EIBRS_PBRSB.Quoted 5 lines
Quoted 5 lines
However, based on Andy's concerns, which I've discussed with him privately (but I'm not qualified to agree or disagree with), we may want to just convert
vmx_spec_ctrl_restore_host()to asm. Better safe than sorry. My original implementation of that function was actually asm. I can try to dig up that code.Note:
VMexit
CALL
RET
RET <---- This is also a problem if the first call hasn't retired yet.
LFENCE
Quoted 6 lines
Quoted 2 lines
Converting
vmx_spec_ctrl_restore_host()to ASM should be able to take care of this.Is there an actual bug here, or are we just micro-optimizing something that may or may not need additional optimization? Unless there's a bug to be fixed, moving code into ASM and increasing complexity doesn't seem worthwhile.
The (slight) bug here is that on systems where the host != guest spec ctrl, they get hit with LFENCE + WRMSR to SPEC_CTRL + LFENCE, when in reality they should just get LFENCE + WRMSR to SPEC_CTRL and thats it.
That would be satisfied with Pawan’s suggestion to move the LFENCE into the else condition in the last branch of vmx_spec_ctrl_restore_host().
The optimization on top of that would be to see if we could whack that 2x LFENCE down to 1x LFENCE. Just feels wrong to have 2x LFENCE’s in the critical path, even if the second one ends up being fairly snappy because of the first one (and/or the WRMSR).
So to recap, fixing “the bug” does not require moving to ASM. Optimizing the 2x LFENCE into 1x LFENCE (probably) requires going to ASM from the sounds of it?
Quoted 3 lines · Sean Christopherson
Is there an actual bug here, or are we just micro-optimizing something that may or may not need additional optimization? Unless there's a bug to be fixed, moving code into ASM and increasing complexity doesn't seem worthwhile.
I can't really argue with that. FWIW, here's the (completely untested) patch.
---8<---
From: Josh Poimboeuf <[email protected]>
Subject: [PATCH] KVM: VMX: Convert vmx_spec_ctrl_restore_host() to assembly
Convert vmx_spec_ctrl_restore_host() to assembly. This allows the removal of a redundant LFENCE. It also "feels" safer as it doesn't allow the compiler to insert any surprises. And it's more symmetrical with the pre-vmentry SPEC_CTRL handling, which is also done in assembly.
--- arch/x86/kvm/vmx/vmenter.S | 71 ++++++++++++++++++++++++++++++++------ arch/x86/kvm/vmx/vmx.c | 25 -------------- arch/x86/kvm/vmx/vmx.h | 1 - 3 files changed, 61 insertions(+), 36 deletions(-)
arch/x86/kvm/vmx/vmenter.S +61 -10
@@ -108,7 +108,7 @@ SYM_FUNC_START(__vmx_vcpu_run) lea (%_ASM_SP), %_ASM_ARG2 call vmx_update_host_rsp - ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL + ALTERNATIVE "jmp .Lguest_spec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL /* * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the
@@ -122,13 +122,13 @@ SYM_FUNC_START(__vmx_vcpu_run) movl VMX_spec_ctrl(%_ASM_DI), %edi movl PER_CPU_VAR(x86_spec_ctrl_current), %esi cmp %edi, %esi - je .Lspec_ctrl_done + je .Lguest_spec_ctrl_done mov $MSR_IA32_SPEC_CTRL, %ecx xor %edx, %edx mov %edi, %eax wrmsr -.Lspec_ctrl_done: +.Lguest_spec_ctrl_done: /* * Since vmentry is serializing on affected CPUs, there's no need for
@@ -253,9 +253,65 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) #endif /* - * IMPORTANT: RSB filling and SPEC_CTRL handling must be done before - * the first unbalanced RET after vmexit! + * IMPORTANT: The below SPEC_CTRL handling and RSB filling must be done + * before the first RET after vmexit! + */ + + ALTERNATIVE "jmp .Lhost_spec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL + + pop %_ASM_SI /* @flags */ + pop %_ASM_DI /* @vmx */ + + /* + * if (flags & VMX_RUN_SAVE_SPEC_CTRL) + * vmx->spec_ctrl = __rdmsr(MSR_IA32_SPEC_CTRL); + */ + test $VMX_RUN_SAVE_SPEC_CTRL, %_ASM_SI + jz .Lhost_spec_ctrl_check + + mov $MSR_IA32_SPEC_CTRL, %ecx + rdmsr + mov %eax, VMX_spec_ctrl(%_ASM_DI) + +.Lhost_spec_ctrl_check: + /* + * If the guest/host SPEC_CTRL values differ, restore the host value. * + * For legacy IBRS, the IBRS bit always needs to be written after + * transitioning from a less privileged predictor mode, regardless of + * whether the guest/host values differ. + * + * if (cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS) || + * vmx->spec_ctrl != this_cpu_read(x86_spec_ctrl_current)) + * native_wrmsrl(MSR_IA32_SPEC_CTRL, hostval); + */ + ALTERNATIVE "", "jmp .Lhost_spec_ctrl_write", X86_FEATURE_KERNEL_IBRS + movl VMX_spec_ctrl(%_ASM_DI), %edi + movl PER_CPU_VAR(x86_spec_ctrl_current), %esi + cmp %edi, %esi + je .Lhost_spec_ctrl_done_lfence + +.Lhost_spec_ctrl_write: + mov $MSR_IA32_SPEC_CTRL, %ecx + xor %edx, %edx + mov %esi, %eax + wrmsr + +.Lhost_spec_ctrl_done_lfence: + /* + * This ensures that speculative execution doesn't incorrectly bypass + * the above SPEC_CTRL wrmsr by mispredicting the 'je'. + * + * It's only needed if the below FILL_RETURN_BUFFER doesn't do an + * LFENCE. Thus the X86_FEATURE_RSB_VMEXIT{_LITE} alternatives. + */ + ALTERNATIVE_2 "lfence", \ + "", X86_FEATURE_RSB_VMEXIT, \ + "", X86_FEATURE_RSB_VMEXIT_LITE + +.Lhost_spec_ctrl_done: + + /* * For retpoline or IBRS, RSB filling is needed to prevent poisoned RSB * entries and (in some cases) RSB underflow. *
@@ -267,11 +323,6 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) FILL_RETURN_BUFFER %_ASM_CX, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT,\ X86_FEATURE_RSB_VMEXIT_LITE - pop %_ASM_ARG2 /* @flags */ - pop %_ASM_ARG1 /* @vmx */ - - call vmx_spec_ctrl_restore_host - /* Put return value in AX */ mov %_ASM_BX, %_ASM_AX
arch/x86/kvm/vmx/vmx.c +0 -25
@@ -7109,31 +7109,6 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) } } -void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, -unsigned int flags) -{ -u64 hostval =this_cpu_read (x86_spec_ctrl_current); - -if (!cpu_feature_enabled (X86_FEATURE_MSR_SPEC_CTRL )) -return ; - -if (flags &VMX_RUN_SAVE_SPEC_CTRL ) - vmx->spec_ctrl =__rdmsr (MSR_IA32_SPEC_CTRL ); - -/* -* If the guest/host SPEC_CTRL values differ, restore the host value. -* -* For legacy IBRS, the IBRS bit always needs to be written after -* transitioning from a less privileged predictor mode, regardless of -* whether the guest/host values differ. -*/ -if (cpu_feature_enabled (X86_FEATURE_KERNEL_IBRS ) || - vmx->spec_ctrl != hostval) -native_wrmsrl (MSR_IA32_SPEC_CTRL , hostval); - -barrier_nospec (); -} -static fastpath_t vmx_exit_handlers_fastpath (struct kvm_vcpu *vcpu) {switch (to_vmx (vcpu)->exit_reason.basic) {
arch/x86/kvm/vmx/vmx.h +0 -1
@@ -420,7 +420,6 @@ void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu);struct vmx_uret_msr *vmx_find_uret_msr (struct vcpu_vmx *vmx,u32 msr);void pt_update_intercept_for_msr (struct kvm_vcpu *vcpu);void vmx_update_host_rsp (struct vcpu_vmx *vmx,unsigned long host_rsp); -void vmx_spec_ctrl_restore_host (struct vcpu_vmx *vmx,unsigned int flags);unsigned int __vmx_vcpu_run_flags (struct vcpu_vmx *vmx);bool __vmx_vcpu_run (struct vcpu_vmx *vmx,unsigned long *regs,unsigned int flags);
Signature · 1 line
2.40.1
Quoted 14 lines · Josh Poimboeuf
Quoted 8 lines · Jon Kohler
Coming back to this, what if we hoisted call
vmx_spec_ctrl_restore_hostaboveFILL_RETURN_BUFFER, and dropped this LFENCE as I did here?That way, we wouldn’t have to mess with the internal LFENCE in
nospec-branch.h, and that would act as the “final line of defense” LFENCE.Would that be acceptable? Or does
FILL_RETURN_BUFFER*need* to occur before any sort of calls no matter what?If we go by Intel's statement that only unbalanced RETs are a concern, that *might* be ok as long as there's a nice comment above the
FILL_RETURN_BUFFERusage site describing the two purposes for the LFENCE.
Quoted 5 lines
However, based on Andy's concerns, which I've discussed with him privately (but I'm not qualified to agree or disagree with), we may want to just convert
vmx_spec_ctrl_restore_host()to asm. Better safe than sorry. My original implementation of that function was actually asm. I can try to dig up that code.
Lemme see if I can remember the whole safely position. I've just gone through a years worth of notes and conversations, including the following gems:
From Intel: "And then on top of that, the LFENCE in vmx_spec_ctrl_restore_host would save you. Fingers crossed." From me: "The how and why is important. Not for right now, but for the years to come when all of this logic inevitably gets tweaked/refactored/moved".
Date-check says 11 months...
__vmx_vcpu_run() is a leaf function as far as the kernel stack is concerned, so to a first approximation, it behaves as such:
VMEXIT
RET
The RET gets a prediction from the top of the RSB (a guest controlled value), but during execute the prediction is discovered to be wrong so a resteer occurs, causing it to restart from __vmx_vcpu_run()'s caller.
Skipping the middle-ground with a 32-entry RSB stuffling loop, we have the following behaviour on eIBRS parts:
VMEXIT (flush RSB if IBRS was 1 in guest)
Restore Host MSR_SPEC_CTRL (flush RSB if IBRS was 0 in guest) RET
where the RET (in theory) encounters an RSB-empty condition and falls back to an indirect prediction.
PBRSB is a missing corner case in the hardware RSB flush, which is only corrected after one CALL instruction architecturally retires.
The problem with mitigating however is that it is heavily entangled with BCBS, so I refer you to
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/analysis-speculative-execution-side-channels.html#inpage-nav-undefined-1-undefined
which describes one example of how RETs can go wrong.
The critical observation is that while for PBRSB it's the first unbalanced RET which causes problem, as soon as *any* RET has executed and got a bad resteer (even temporarily), you've lost all ability to contain the damage. So, to protect against PBRSB, one CALL instruction must retire before *any* RET instruction executes.
Pawan's patch to turn the unconditional lfence into an else-lfence should be safe seeing as Intel's guidance
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/post-barrier-return-stack-buffer-predictions.html
explicitly says you can use the WRMSR to restore the host MSR_SPEC_CTRL value as the speculation barrier.
But, the safety of vmx_spec_ctrl_restore_host() in the first place depends on the early return never ever becoming a conditional, and the compiler never emitting a call to memcpy()/memset()/whatever behind your back - something which is not prohibited by noinstr.
I hope this clears things up.
~Andrew
On Tue, Jun 06, 2023 at 01:20:52AM +0100, Andrew Cooper wrote:
<clip very useful summary which belongs in git somewhere>
Quoted 2 lines
But, the safety of
vmx_spec_ctrl_restore_host()in the first place depends on the early return never ever becoming a conditional,
Good point. And that would be easier to overlook in C.
Quoted 2 lines
and the compiler never emitting a call to memcpy()/memset()/whatever behind your back - something which is not prohibited by noinstr.
Au contraire, objtool has checking for that:
if (state->noinstr && state->instr <= 0 &&
!noinstr_call_dest(file, insn, insn_call_dest(insn))) {
WARN_INSN(insn, "call to %s() leaves .noinstr.text section", call_dest_name(insn));
return 1;
}Regardless, despite being the person who wrote this thing in C to begin with, I believe asm really is a better fit due to the delicate and precise nature of the mitigations.
Signature · 1 line
Josh
Quoted 10 lines · Jon Kohler
Remove
barrier_nospec(), which translates to LFENCE, invmx_spec_ctrl_restore_host()as RSB-barriers (as defined by [1]) already exist prior to this point.This LFENCE was added on commit
fc02735b14ff("KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS") in the 5.19 timeframe; however, commit2b1299322016("x86/speculation: Add RSB VM Exit protections") in 6.0 timeframe added a LFENCE forX86_FEATURE_RSB_VMEXIT_LITEwas added directly invmx_vmexit, prior to CALLvmx_spec_ctrl_restore_host.
Quoted 5 lines
For posterity,
vmx_spec_ctrl_restore_hostalso will execute WRMSR toIA32_SPEC_CTRLforX86_FEATURE_KERNEL_IBRSor when guest/host MSR value does not match, which serves as an additional RSB-barrier.
Yeah, unfortunately PBRSB is insidious.
From memory (please correct me if I'm wrong), the required safety property is this: After a VMExit (if IBRS was set prior to exit) or the write to MSR_SPEC_CTRL setting IBRS (if IBRS was not set prior to exit), one single CALL instruction must architecturally retire before any RET instructions execute (speculatively).
There are several ways to arrange this, but they all basically boil down to having some serialising instruction between the first CALL and first RET on any reachable path from VMExit.
~Andrew
Quoted 25 lines · Andrew Cooper
Quoted 10 lines · Jon Kohler
Remove
barrier_nospec(), which translates to LFENCE, invmx_spec_ctrl_restore_host()as RSB-barriers (as defined by [1]) already exist prior to this point.This LFENCE was added on commit
fc02735b14ff("KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS") in the 5.19 timeframe; however, commit2b1299322016("x86/speculation: Add RSB VM Exit protections") in 6.0 timeframe added a LFENCE forX86_FEATURE_RSB_VMEXIT_LITEwas added directly invmx_vmexit, prior to CALLvmx_spec_ctrl_restore_host.Quoted 5 lines
For posterity,
vmx_spec_ctrl_restore_hostalso will execute WRMSR toIA32_SPEC_CTRLforX86_FEATURE_KERNEL_IBRSor when guest/host MSR value does not match, which serves as an additional RSB-barrier.Yeah, unfortunately PBRSB is insidious.
From memory (please correct me if I'm wrong), the required safety property is this: After a VMExit (if IBRS was set prior to exit) or the write to
MSR_SPEC_CTRLsetting IBRS (if IBRS was not set prior to exit), one single CALL instruction must architecturally retire before any RET instructions execute (speculatively).
Quoted 3 lines
There are several ways to arrange this, but they all basically boil down to having some serialising instruction between the first CALL and first RET on any reachable path from VMExit.
The document says the problem is *unbalanced* RET, i.e. RSB underflow.
So the mitigation needs a single RSB stuff (i.e., unbalanced CALL) and then an LFENCE anytime before the next unbalanced RET.
Signature · 1 line
Josh