KVM: VMX: Unify L1D flush for L1TF
Sean Christopherson
maintainerv3 cover KVM: VMX: Unify L1D flush for L1TF
Unify the L1D cache flushing done to mitigate L1TF under the per-CPU variable, as the per-vCPU variable has been superfluous for quite some time.
Patch 1 fixes a bug (I think it's a bug?) I found when poking around the code. If L1D flushes are conditional and KVM skips an L1D flush on VM-Enter, then arguably KVM should flush CPU buffers based on other mitigations.
Patches 2-3 bury the L1TF L1D flushing under CONFIG_CPU_MITIGATIONS, partly because it's absurd that KVM doesn't honor CONFIG_CPU_MITIGATIONS for that case, partly because it simplifies unifying the tracking code (helps obviate the need for a stub).
Patch 4 is Brendan's patch and the main goal of the mini-series.
v3:
- Put the "raw" variant in KVM, dress it up with KVM's "request" terminology, and add a comment explaining why _KVM_ knows its usage doesn't need to disable virtualization.
- Add the prep patches.
v2:
- https://lore.kernel.org/all/[email protected]
- Moved the bit back to irq_stat
- Fixed DEBUG_PREEMPT issues by adding a _raw variant
v1: https://lore.kernel.org/r/[email protected]
Brendan Jackman (1):
KVM: x86: Unify L1TF flushing under per-CPU variable
Sean Christopherson (3):
KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
KVM: VMX: Bundle all L1 data cache flush mitigation code together
KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n
arch/x86/include/asm/hardirq.h | 4 +- arch/x86/include/asm/kvm_host.h | 3 - arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 222 ++++++++++++++++++-------------- arch/x86/kvm/x86.c | 6 +- arch/x86/kvm/x86.h | 14 ++ 7 files changed, 144 insertions(+), 109 deletions(-)
base-commit: f222788458c8a7753d43befef2769cd282dc008e
Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
Sean Christopherson
maintainerPATCH v3 1/4 KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Note, the flaw goes back to the introduction of the MDS mitigation. The MDS mitigation was inadvertently fixed by commit 43fb862de8f6 ("KVM/VMX:
Move VERW closer to VMentry for MDS mitigation"), but previous kernels that flush CPU buffers in vmx_vcpu_enter_exit() are affected.
--- arch/x86/kvm/vmx/vmx.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/vmx.c +7 -5
@@ -6663,7 +6663,7 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)* information but as all relevant affected CPUs have 32KiB L1D cache size * there is no point in doing so. */ -static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) +static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu) {int size =PAGE_SIZE <<L1D_CACHE_ORDER ;
@@ -6691,14 +6691,14 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)kvm_clear_cpu_l1tf_flush_l1d ();if (!flush_l1d) -return ; +return false; } vcpu->stat.l1d_flush++;if (static_cpu_has (X86_FEATURE_FLUSH_L1D )) {native_wrmsrq (MSR_IA32_FLUSH_CMD ,L1D_FLUSH ); -return ; +return true; } asm volatile(
@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) :: [flush_pages] "r" (vmx_l1d_flush_pages), [size] "r" (size) : "eax", "ebx", "ecx", "edx"); +return true; }void vmx_update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr)
@@ -7330,8 +7331,9 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,* and is affected by MMIO Stale Data. In such cases mitigation in only * needed against an MMIO capable guest. */ -if (static_branch_unlikely (&vmx_l1d_should_flush)) -vmx_l1d_flush (vcpu); +if (static_branch_unlikely (&vmx_l1d_should_flush) && +vmx_l1d_flush (vcpu)) + ;else if (static_branch_unlikely (&cpu_buf_vm_clear) && (flags &VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO ))x86_clear_cpu_buffers ();
Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 4 lines · Sean Christopherson
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug. If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.
Quoted 42 lines
MDS mitigation was inadvertently fixed by commit
43fb862de8f6("KVM/VMX:
Move VERW closer to VMentry for MDS mitigation"), but previous kernels that flush CPU buffers invmx_vcpu_enter_exit()are affected.Fixes 650b68a0622f ("x86/kvm/vmx: Add MDS protection when L1D Flush is not active")Cc Pawan Gupta <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)arch/x86/kvm/vmx/vmx.c +3
-3@@ -6663,7 +6663,7 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)* information but as all relevant affected CPUs have 32KiB L1D cache size * there is no point in doing so. */ -static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) +static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu) {int size =PAGE_SIZE <<L1D_CACHE_ORDER ;@@ -6691,14 +6691,14 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)kvm_clear_cpu_l1tf_flush_l1d ();if (!flush_l1d) -return ; +return false; } vcpu->stat.l1d_flush++;if (static_cpu_has (X86_FEATURE_FLUSH_L1D )) {native_wrmsrq (MSR_IA32_FLUSH_CMD ,L1D_FLUSH ); -return ; +return true; } asm volatile(
arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725
Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +1
-0@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) :: [flush_pages] "r" (vmx_l1d_flush_pages), [size] "r" (size) : "eax", "ebx", "ecx", "edx"); + return true;
The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)
Anyway, if this is an issue, it's orthogonal to this patch.
Quoted 15 lines
arch/x86/kvm/vmx/vmx.c +3
-2}void vmx_update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr)@@ -7330,8 +7331,9 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,* and is affected by MMIO Stale Data. In such cases mitigation in only * needed against an MMIO capable guest. */ -if (static_branch_unlikely (&vmx_l1d_should_flush)) -vmx_l1d_flush (vcpu); +if (static_branch_unlikely (&vmx_l1d_should_flush) && +vmx_l1d_flush (vcpu)) + ;else if (static_branch_unlikely (&cpu_buf_vm_clear) && (flags &VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO ))x86_clear_cpu_buffers ();
Sean Christopherson
maintainerRe: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 11 lines · Brendan Jackman
Quoted 4 lines · Sean Christopherson
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug.
Yeah, that's where I'm at as well.
Quoted 2 lines
If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.
...
Quoted 11 lines
arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1
-0@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) :: [flush_pages] "r" (vmx_l1d_flush_pages), [size] "r" (size) : "eax", "ebx", "ecx", "edx"); + return true;The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)
Quoted 1 line
Anyway, if this is an issue, it's orthogonal to this patch.
Pawan, any idea?
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 31 lines · Sean Christopherson
Quoted 11 lines · Brendan Jackman
Quoted 4 lines · Sean Christopherson
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug.
Yeah, that's where I'm at as well.
Quoted 2 lines
If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.
...
Quoted 11 lines
arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1
-0@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) :: [flush_pages] "r" (vmx_l1d_flush_pages), [size] "r" (size) : "eax", "ebx", "ecx", "edx"); + return true;The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)
Quoted 3 lines
Quoted 1 line
Anyway, if this is an issue, it's orthogonal to this patch.
Pawan, any idea?
I want to say yes, but let me first confirm this internally and get back to you.
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 21 lines · Pawan Gupta
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Brendan Jackman
Quoted 4 lines · Sean Christopherson
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug.
Yeah, that's where I'm at as well.
Quoted 2 lines
If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.
...
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1
-0@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) :: [flush_pages] "r" (vmx_l1d_flush_pages), [size] "r" (size) : "eax", "ebx", "ecx", "edx"); + return true;The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)
Anyways, I have asked for a confirmation if the sequence works for MMIO stale data also. I will update once I get a response.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Anyway, if this is an issue, it's orthogonal to this patch.
Pawan, any idea?
I want to say yes, but let me first confirm this internally and get back to you.
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 7 lines · Pawan Gupta
...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)
KVM started reporting MD_CLEAR to userspace in Linux v5.2, but it didn't report L1D_FLUSH to userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 10 lines · Jim Mattson
Quoted 7 lines · Pawan Gupta
...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)
Quoted 4 lines
KVM started reporting
MD_CLEARto userspace in Linux v5.2, but it didn't reportL1D_FLUSHto userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.
Shouldn't only the L0 hypervisor be doing the L1D_FLUSH?
kvm_get_arch_capabilities()
{
...
/*
* If we're doing cache flushes (either "always" or "cond")
* we will do one whenever the guest does a vmlaunch/vmresume.
* If an outer hypervisor is doing the cache flush for us
* (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that
* capability to the guest too, and if EPT is disabled we're not
* vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will
* require a nested hypervisor to do a flush of its own. */
if (l1tf_vmx_mitigation != VMENTER_L1D_FLUSH_NEVER)
data |= ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 19 lines · Pawan Gupta
Quoted 10 lines · Jim Mattson
Quoted 7 lines · Pawan Gupta
...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)Quoted 4 lines
KVM started reporting
MD_CLEARto userspace in Linux v5.2, but it didn't reportL1D_FLUSHto userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.Shouldn't only the L0 hypervisor be doing the
L1D_FLUSH?
Quoted 15 lines
kvm_get_arch_capabilities()
{
...
/*
* If we're doing cache flushes (either "always" or "cond")
* we will do one whenever the guest does a vmlaunch/vmresume.
* If an outer hypervisor is doing the cache flush for us
* (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that
* capability to the guest too, and if EPT is disabled we're not
* vulnerable. Overall, onlyVMENTER_L1D_FLUSH_NEVERwill
* require a nested hypervisor to do a flush of its own. */
if (l1tf_vmx_mitigation!=VMENTER_L1D_FLUSH_NEVER)
data |=ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;
Unless L0 has chosen L1D_FLUSH_NEVER. :)
On GCE's L1TF-vulnerable hosts, we actually do an L1D flush at ASI entry rather than VM-entry. ASI entries are two orders of magnitude less frequent than VM-entries, so we get comparable protection to L1D_FLUSH_ALWAYS at a fraction of the cost.
At the moment, we still do an L1D flush on emulated VM-entry, but that's just because we have historically advertised IA32_ARCH_CAPABILITIES.SKIP_L1DFL_VMENTRY to L1.
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 39 lines · Jim Mattson
Quoted 19 lines · Pawan Gupta
Quoted 10 lines · Jim Mattson
Quoted 7 lines · Pawan Gupta
...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)Quoted 4 lines
KVM started reporting
MD_CLEARto userspace in Linux v5.2, but it didn't reportL1D_FLUSHto userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.Shouldn't only the L0 hypervisor be doing the
L1D_FLUSH?Quoted 15 lines
kvm_get_arch_capabilities()
{
...
/*
* If we're doing cache flushes (either "always" or "cond")
* we will do one whenever the guest does a vmlaunch/vmresume.
* If an outer hypervisor is doing the cache flush for us
* (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that
* capability to the guest too, and if EPT is disabled we're not
* vulnerable. Overall, onlyVMENTER_L1D_FLUSH_NEVERwill
* require a nested hypervisor to do a flush of its own. */
if (l1tf_vmx_mitigation!=VMENTER_L1D_FLUSH_NEVER)
data |=ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;Unless L0 has chosen
L1D_FLUSH_NEVER. :)
Quoted 8 lines
On GCE's L1TF-vulnerable hosts, we actually do an L1D flush at ASI entry rather than VM-entry. ASI entries are two orders of magnitude less frequent than VM-entries, so we get comparable protection to
L1D_FLUSH_ALWAYSat a fraction of the cost.At the moment, we still do an L1D flush on emulated VM-entry, but that's just because we have historically advertised IA32_ARCH_CAPABILITIES.SKIP_L1DFL_VMENTRY to L1.
Thanks for the background.
I still don't see the problem, CPUs that are vulnerable to L1TF are also vulnerable to MDS. So, they don't set mmio_stale_data_clear, instead they set X86_FEATURE_CLEAR_CPU_BUF and execute VERW in __vmx_vcpu_run() regardless of whether L1D_FLUSH was done.
But, I agree it is best to decouple L1D flush and MMIO Stale Data to be avoid any confusion.
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 45 lines · Pawan Gupta
Quoted 44 lines · Jim Mattson
Quoted 19 lines · Pawan Gupta
Quoted 10 lines · Jim Mattson
Quoted 7 lines · Pawan Gupta
...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)Quoted 4 lines
KVM started reporting
MD_CLEARto userspace in Linux v5.2, but it didn't reportL1D_FLUSHto userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.Shouldn't only the L0 hypervisor be doing the
L1D_FLUSH?Quoted 15 lines
kvm_get_arch_capabilities()
{
...
/*
* If we're doing cache flushes (either "always" or "cond")
* we will do one whenever the guest does a vmlaunch/vmresume.
* If an outer hypervisor is doing the cache flush for us
* (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that
* capability to the guest too, and if EPT is disabled we're not
* vulnerable. Overall, onlyVMENTER_L1D_FLUSH_NEVERwill
* require a nested hypervisor to do a flush of its own. */
if (l1tf_vmx_mitigation!=VMENTER_L1D_FLUSH_NEVER)
data |=ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;Unless L0 has chosen
L1D_FLUSH_NEVER. :)On GCE's L1TF-vulnerable hosts, we actually do an L1D flush at ASI entry rather than VM-entry. ASI entries are two orders of magnitude less frequent than VM-entries, so we get comparable protection to
L1D_FLUSH_ALWAYSat a fraction of the cost.
Quoted 8 lines
Quoted 3 lines
At the moment, we still do an L1D flush on emulated VM-entry, but that's just because we have historically advertised IA32_ARCH_CAPABILITIES.SKIP_L1DFL_VMENTRY to L1.
Thanks for the background.
I still don't see the problem, CPUs that are vulnerable to L1TF are also vulnerable to MDS. So, they don't set
mmio_stale_data_clear, instead they
Sorry I meant cpu_buf_vm_clear instead of mmio_stale_data_clear (I was looking at a slightly older kernel).
Quoted 5 lines
set
X86_FEATURE_CLEAR_CPU_BUFand execute VERW in__vmx_vcpu_run()regardless of whetherL1D_FLUSHwas done.But, I agree it is best to decouple L1D flush and MMIO Stale Data to be avoid any confusion.
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
In reply to Pawan GuptaQuoted 32 lines · Pawan Gupta
Quoted 31 lines · Sean Christopherson
Quoted 11 lines · Brendan Jackman
Quoted 4 lines · Sean Christopherson
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug.
Yeah, that's where I'm at as well.
Quoted 2 lines
If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.
...
Quoted 11 lines
arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1
-0@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) :: [flush_pages] "r" (vmx_l1d_flush_pages), [size] "r" (size) : "eax", "ebx", "ecx", "edx"); + return true;The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Anyway, if this is an issue, it's orthogonal to this patch.
Pawan, any idea?
I want to say yes, but let me first confirm this internally and get back to you.
The software sequence for L1D flush was not validated to mitigate MMIO Stale Data. To be on safer side, it is better to not rely on the sequence.
OTOH, if a user has not updated the microcode to mitigate L1TF, the system will not have the microcode to mitigate MMIO Stale Data either, because the microcode for MMIO Stale Data was released after L1TF. Also I am not aware of any CPUs that are vulnerable to L1TF and vulnerable to MMIO Stale Data only(not MDS).
So in practice, decoupling L1D flush and MMIO Stale Data won't have any practical impact on functionality, and makes MMIO Stale Data mitigation consistent with MDS mitigation. I hope that makes things clear.
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
In reply to Sean ChristophersonQuoted 6 lines · Sean Christopherson
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Note, the flaw goes back to the introduction of the MDS mitigation.
I don't think it is a flaw. If L1D flush was skipped because VMexit did not touch any interested data, then there shouldn't be any need to flush CPU buffers.
Secondly, when L1D flush is skipped, flushing MDS affected buffers is of no use, because the data could still be extracted from L1D cache using L1TF. Isn't it?
Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 8 lines · Pawan Gupta
Quoted 6 lines · Sean Christopherson
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Note, the flaw goes back to the introduction of the MDS mitigation.
Quoted 7 lines
I don't think it is a flaw. If L1D flush was skipped because VMexit did not touch any interested data, then there shouldn't be any need to flush CPU buffers.
Secondly, when L1D flush is skipped, flushing MDS affected buffers is of no use, because the data could still be extracted from L1D cache using L1TF. Isn't it?
This is assuming an equivalence between what L1TF and MMIO Stale Data exploits can do, that isn't really captured in the code/documentation IMO. This probably felt much more obvious when the vulns were new...
I dunno, in the end this definitely doesn't seem like a terrifying big deal, I'm not saying the current behaviour is crazy or anything, it's just slightly surprising and people with sophisticated opinions about this might not be getting what they think they are out of the default setup.
But I have no evidence that these sophisticated dissidents actually exist, maybe just adding commentary about this rationale is more than good enough here.
Sean Christopherson
maintainerRe: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
Quoted 7 lines · Brendan Jackman
Quoted 6 lines · Pawan Gupta
Quoted 5 lines · Sean Christopherson
If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Note, the flaw goes back to the introduction of the MDS mitigation.
I don't think it is a flaw. If L1D flush was skipped because VMexit did not touch any interested data, then there shouldn't be any need to flush CPU buffers.
But as Brendan alludes to below, that assumes certain aspects of L1TF and MDS are equal. Obliterating the L1D is far more costly than flushing CPU buffers, as evidenced by the much more conditional flushing for L1TF. My read of the L1TF mitigation is that the conditional flushing is that it's a compromise between performance and security. Skipping the flush doesn't necessarily mean nothing interesting was accessed, it just means that KVM didn't hit any of the flows where a large amount of interesting data was guaranteed to have been accessed.
Quoted 7 lines
Quoted 3 lines
Secondly, when L1D flush is skipped, flushing MDS affected buffers is of no use, because the data could still be extracted from L1D cache using L1TF. Isn't it?
This is assuming an equivalence between what L1TF and MMIO Stale Data exploits can do, that isn't really captured in the code/documentation IMO.
And again, the cost. To fully mitigate L1TF, KVM would need to flush on every entry, but that completely tanks performance. But that doesn't
Quoted 7 lines
This probably felt much more obvious when the vulns were new...
I dunno, in the end this definitely doesn't seem like a terrifying big deal, I'm not saying the current behaviour is crazy or anything, it's just slightly surprising and people with sophisticated opinions about this might not be getting what they think they are out of the default setup.
Ya. I highly doubt this particular combination matters in practice, but I don't like surprises. And I find it surprising that the behavior of KVM's mitigation for MMIO Stale Data changes based on whether or not the L1TF mitigation is enabled.
Quoted 3 lines
But I have no evidence that these sophisticated dissidents actually exist, maybe just adding commentary about this rationale is more than good enough here.
Sean Christopherson
maintainerPATCH v3 2/4 KVM: VMX: Bundle all L1 data cache flush mitigation code together
In reply to Sean ChristophersonMove vmx_l1d_flush(), vmx_cleanup_l1d_flush(), and the vmentry_l1d_flush param code up in vmx.c so that all of the L1 data cache flushing code is bundled together. This will allow conditioning the mitigation code on CONFIG_CPU_MITIGATIONS=y with minimal #ifdefs.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.c | 176 ++++++++++++++++++++--------------------- 1 file changed, 88 insertions(+), 88 deletions(-)
arch/x86/kvm/vmx/vmx.c +88 -88
@@ -302,6 +302,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)return 0 ; } +static void vmx_cleanup_l1d_flush (void ) +{ +if (vmx_l1d_flush_pages) { +free_pages ((unsigned long )vmx_l1d_flush_pages,L1D_CACHE_ORDER ); + vmx_l1d_flush_pages =NULL ; + } +/* Restore state so sysfs ignores VMX */ + l1tf_vmx_mitigation =VMENTER_L1D_FLUSH_AUTO ; +} +static int vmentry_l1d_flush_parse (const char *s) {unsigned int i;
@@ -352,6 +362,84 @@ static int vmentry_l1d_flush_get(char *s, const struct kernel_param *kp)return sysfs_emit (s,"%s\n" , vmentry_l1d_param[l1tf_vmx_mitigation].option); } +/* +* Software based L1D cache flush which is used when microcode providing +* the cache control MSR is not loaded. +* +* The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to +* flush it is required to read in 64 KiB because the replacement algorithm +* is not exactly LRU. This could be sized at runtime via topology +* information but as all relevant affected CPUs have 32KiB L1D cache size +* there is no point in doing so. +*/ +static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu) +{ +int size =PAGE_SIZE <<L1D_CACHE_ORDER ; + +/* +* This code is only executed when the flush mode is 'cond' or +* 'always' +*/ +if (static_branch_likely (&vmx_l1d_flush_cond)) { +bool flush_l1d; + +/* +* Clear the per-vcpu flush bit, it gets set again if the vCPU +* is reloaded, i.e. if the vCPU is scheduled out or if KVM +* exits to userspace, or if KVM reaches one of the unsafe +* VMEXIT handlers, e.g. if KVM calls into the emulator. +*/ + flush_l1d = vcpu->arch.l1tf_flush_l1d; + vcpu->arch.l1tf_flush_l1d = false; + +/* +* Clear the per-cpu flush bit, it gets set again from +* the interrupt handlers. +*/ + flush_l1d |=kvm_get_cpu_l1tf_flush_l1d (); +kvm_clear_cpu_l1tf_flush_l1d (); + +if (!flush_l1d) +return false; + } + + vcpu->stat.l1d_flush++; + +if (static_cpu_has (X86_FEATURE_FLUSH_L1D )) { +native_wrmsrq (MSR_IA32_FLUSH_CMD ,L1D_FLUSH ); +return true; + } + + asmvolatile ( +/* First ensure the pages are in the TLB */ +"xorl %%eax, %%eax\n" +".Lpopulate_tlb:\n\t" +"movzbl (%[flush_pages], %%" _ASM_AX"), %%ecx\n\t" +"addl $4096, %%eax\n\t" +"cmpl %%eax, %[size]\n\t" +"jne .Lpopulate_tlb\n\t" +"xorl %%eax, %%eax\n\t" +"cpuid\n\t" +/* Now fill the cache */ +"xorl %%eax, %%eax\n" +".Lfill_cache:\n" +"movzbl (%[flush_pages], %%" _ASM_AX"), %%ecx\n\t" +"addl $64, %%eax\n\t" +"cmpl %%eax, %[size]\n\t" +"jne .Lfill_cache\n\t" +"lfence\n" + :: [flush_pages]"r" (vmx_l1d_flush_pages), + [size]"r" (size) + :"eax" ,"ebx" ,"ecx" ,"edx" ); +return true; +} + +static const struct kernel_param_ops vmentry_l1d_flush_ops = { + .set = vmentry_l1d_flush_set, + .get = vmentry_l1d_flush_get, +}; +module_param_cb (vmentry_l1d_flush, &vmentry_l1d_flush_ops,NULL ,0644 ); + static __always_inline void vmx_disable_fb_clear(struct vcpu_vmx *vmx) {u64 msr;
@@ -404,12 +492,6 @@ static void vmx_update_fb_clear_dis(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx) vmx->disable_fb_clear = false; } -static const struct kernel_param_ops vmentry_l1d_flush_ops = { - .set = vmentry_l1d_flush_set, - .get = vmentry_l1d_flush_get, -}; -module_param_cb (vmentry_l1d_flush, &vmentry_l1d_flush_ops,NULL ,0644 ); -static u32 vmx_segment_access_rights (struct kvm_segment *var);void vmx_vmexit (void );
@@ -6653,78 +6735,6 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)return ret; } -/* -* Software based L1D cache flush which is used when microcode providing -* the cache control MSR is not loaded. -* -* The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to -* flush it is required to read in 64 KiB because the replacement algorithm -* is not exactly LRU. This could be sized at runtime via topology -* information but as all relevant affected CPUs have 32KiB L1D cache size -* there is no point in doing so. -*/ -static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu) -{ -int size =PAGE_SIZE <<L1D_CACHE_ORDER ; - -/* -* This code is only executed when the flush mode is 'cond' or -* 'always' -*/ -if (static_branch_likely (&vmx_l1d_flush_cond)) { -bool flush_l1d; - -/* -* Clear the per-vcpu flush bit, it gets set again if the vCPU -* is reloaded, i.e. if the vCPU is scheduled out or if KVM -* exits to userspace, or if KVM reaches one of the unsafe -* VMEXIT handlers, e.g. if KVM calls into the emulator. -*/ - flush_l1d = vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d = false; - -/* -* Clear the per-cpu flush bit, it gets set again from -* the interrupt handlers. -*/ - flush_l1d |=kvm_get_cpu_l1tf_flush_l1d (); -kvm_clear_cpu_l1tf_flush_l1d (); - -if (!flush_l1d) -return false; - } - - vcpu->stat.l1d_flush++; - -if (static_cpu_has (X86_FEATURE_FLUSH_L1D )) { -native_wrmsrq (MSR_IA32_FLUSH_CMD ,L1D_FLUSH ); -return true; - } - - asmvolatile ( -/* First ensure the pages are in the TLB */ -"xorl %%eax, %%eax\n" -".Lpopulate_tlb:\n\t" -"movzbl (%[flush_pages], %%" _ASM_AX"), %%ecx\n\t" -"addl $4096, %%eax\n\t" -"cmpl %%eax, %[size]\n\t" -"jne .Lpopulate_tlb\n\t" -"xorl %%eax, %%eax\n\t" -"cpuid\n\t" -/* Now fill the cache */ -"xorl %%eax, %%eax\n" -".Lfill_cache:\n" -"movzbl (%[flush_pages], %%" _ASM_AX"), %%ecx\n\t" -"addl $64, %%eax\n\t" -"cmpl %%eax, %[size]\n\t" -"jne .Lfill_cache\n\t" -"lfence\n" - :: [flush_pages]"r" (vmx_l1d_flush_pages), - [size]"r" (size) - :"eax" ,"ebx" ,"ecx" ,"edx" ); -return true; -} -void vmx_update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr) {struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);
@@ -8673,16 +8683,6 @@ __init int vmx_hardware_setup(void)return r; } -static void vmx_cleanup_l1d_flush (void ) -{ -if (vmx_l1d_flush_pages) { -free_pages ((unsigned long )vmx_l1d_flush_pages,L1D_CACHE_ORDER ); - vmx_l1d_flush_pages =NULL ; - } -/* Restore state so sysfs ignores VMX */ - l1tf_vmx_mitigation =VMENTER_L1D_FLUSH_AUTO ; -} -void vmx_exit (void ) { allow_smaller_maxphyaddr = false;
Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
Re: [PATCH v3 2/4] KVM: VMX: Bundle all L1 data cache flush mitigation code together
Quoted 8 lines · Sean Christopherson
Move
vmx_l1d_flush(),vmx_cleanup_l1d_flush(), and thevmentry_l1d_flushparam code up invmx.cso that all of the L1 data cache flushing code is bundled together. This will allow conditioning the mitigation code onCONFIG_CPU_MITIGATIONS=ywith minimal #ifdefs.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>
(Git says no changed lines)
Sean Christopherson
maintainerPATCH v3 3/4 KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n
In reply to Sean ChristophersonDisable support for flushing the L1 data cache to mitigate L1TF if CPU mitigations are disabled for the entire kernel. KVM's mitigation of L1TF is in no way special enough to justify ignoring CONFIG_CPU_MITIGATIONS=n.
Deliberately use CPU_MITIGATIONS instead of the more precise MITIGATION_L1TF, as MITIGATION_L1TF only controls the default behavior, i.e. CONFIG_MITIGATION_L1TF=n doesn't completely disable L1TF mitigations in the kernel.
Keep the vmentry_l1d_flush module param to avoid breaking existing setups, and leverage the .set path to alert the user to the fact that vmentry_l1d_flush will be ignored. Don't bother validating the incoming value; if an admin misconfigures vmentry_l1d_flush, the fact that the bad configuration won't be detected when running with CONFIG_CPU_MITIGATIONS=n is likely the least of their worries.
--- arch/x86/include/asm/hardirq.h | 4 +-- arch/x86/kvm/vmx/vmx.c | 56 ++++++++++++++++++++++++++-------- 2 files changed, 46 insertions(+), 14 deletions(-)
arch/x86/include/asm/hardirq.h +2 -2
@@ -5,7 +5,7 @@#include <linux/threads.h> typedef struct { -#if IS_ENABLED(CONFIG_KVM_INTEL) +#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL) u8 kvm_cpu_l1tf_flush_l1d; #endif unsigned int __nmi_count; /* arch dependent */
@@ -68,7 +68,7 @@ extern u64 arch_irq_stat(void);DECLARE_PER_CPU_CACHE_HOT (u16, __softirq_pending);#define local_softirq_pending_ref __softirq_pending -#if IS_ENABLED (CONFIG_KVM_INTEL ) +#if IS_ENABLED (CONFIG_CPU_MITIGATIONS ) &&IS_ENABLED (CONFIG_KVM_INTEL )/* * This function is called from noinstr interrupt contexts * and must be inlined to not get instrumentation.
arch/x86/kvm/vmx/vmx.c +44 -12
104 lines not on this page · raw message
Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
kernel test robot
build botRe: [PATCH v3 3/4] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n
kernel test robot noticed the following build errors:
Hi Sean,
kernel test robot noticed the following build errors:
[auto build test ERROR on f222788458c8a7753d43befef2769cd282dc008e]
url: https://github.com/intel-lab-lkp/linux/commits/Sean-Christopherson/KVM-VMX-Flush-CPU-buffers-as-needed-if-L1D-cache-flush-is-skipped/20251017-040642 base: f222788458c8a7753d43befef2769cd282dc008e patch link: https://lore.kernel.org/r/20251016200417.97003-4-seanjc%40google.com patch subject: [PATCH v3 3/4] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n config: x86_64-kexec (https://download.01.org/0day-ci/archive/20251018/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251018/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> arch/x86/kvm/vmx/vmx.c:313:29: error: too many arguments to function call, expected 0, have 1
313 | return vmx_setup_l1d_flush(vmentry_l1d_flush_param);
| ~~~~~~~~~~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~ arch/x86/kvm/vmx/vmx.c:306:12: note: 'vmx_setup_l1d_flush' declared here
306 | static int vmx_setup_l1d_flush(void)
| ^
1 error generated.
vim +313 arch/x86/kvm/vmx/vmx.c
305
306 static int vmx_setup_l1d_flush(void)
307 {
308 /*
309 * Hand the parameter mitigation value in which was stored in the pre
310 * module init parser. If no parameter was given, it will contain
311 * 'auto' which will be turned into the default 'cond' mitigation mode.
312 */
> 313 return vmx_setup_l1d_flush(vmentry_l1d_flush_param);
314 }
315
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v3 3/4] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n
In reply to Sean Christophersonarch/x86/kvm/vmx/vmx.c static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) · line 313
Quoted 12 lines · Sean Christopherson
arch/x86/kvm/vmx/vmx.c +8
-0@@ -302,6 +303,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)return 0 ; } +static int vmx_setup_l1d_flush (void ) +{ +/* +* Hand the parameter mitigation value in which was stored in the pre +* module init parser. If no parameter was given, it will contain +* 'auto' which will be turned into the default 'cond' mitigation mode. +*/ +return vmx_setup_l1d_flush (vmentry_l1d_flush_param);
A likely typo here, it should be:
return __vmx_setup_l1d_flush(vmentry_l1d_flush_param);
Sean Christopherson
maintainerRe: [PATCH v3 3/4] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n
Quoted 16 lines · Pawan Gupta
arch/x86/kvm/vmx/vmx.c static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) · line 313
Quoted 12 lines · Sean Christopherson
arch/x86/kvm/vmx/vmx.c +8
-0@@ -302,6 +303,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)return 0 ; } +static int vmx_setup_l1d_flush (void ) +{ +/* +* Hand the parameter mitigation value in which was stored in the pre +* module init parser. If no parameter was given, it will contain +* 'auto' which will be turned into the default 'cond' mitigation mode. +*/ +return vmx_setup_l1d_flush (vmentry_l1d_flush_param);A likely typo here, it should be:
Quoted 1 line
return __vmx_setup_l1d_flush(vmentry_l1d_flush_param);
Argh, I have a feeling I clobbered my branch with a --force push, as I remember fixing this exact problem. Or maybe I saw Brendan's struggles and thought, "hold my beer!" :-D
Sean Christopherson
maintainerPATCH v3 4/4 KVM: x86: Unify L1TF flushing under per-CPU variable
In reply to Sean ChristophersonFrom: Brendan Jackman <[email protected]>
Currently the tracking of the need to flush L1D for L1TF is tracked by two bits: one per-CPU and one per-vCPU.
The per-vCPU bit is always set when the vCPU shows up on a core, so there is no interesting state that's truly per-vCPU. Indeed, this is a requirement, since L1D is a part of the physical CPU.
So simplify this by combining the two bits.
The vCPU bit was being written from preemption-enabled regions. To play nice with those cases, wrap all calls from KVM and use a raw write so that request a flush with preemption enabled doesn't trigger what would effectively be DEBUG_PREEMPT false positives. Preemption doesn't need to be disabled, as kvm_arch_vcpu_load() will mark the new CPU as needing a flush if the vCPU task is migrated, or if userspace runs the vCPU on a different task.
--- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 20 +++++--------------- arch/x86/kvm/x86.c | 6 +++--- arch/x86/kvm/x86.h | 14 ++++++++++++++ 6 files changed, 24 insertions(+), 23 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -3
@@ -1055,9 +1055,6 @@ struct kvm_vcpu_arch {/* be preempted when it's in kernel-mode(cpl=0) */ bool preempted_in_kernel; -/* Flush the L1 Data cache for L1TF mitigation on VMENTER */ -bool l1tf_flush_l1d; -/* Host CPU on which VM-entry was most recently attempted */ int last_vmentry_cpu;
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -4859,7 +4859,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code,*/ BUILD_BUG_ON (lower_32_bits (PFERR_SYNTHETIC_MASK )); - vcpu->arch.l1tf_flush_l1d = true; +kvm_request_l1tf_flush_l1d ();if (!flags) {trace_kvm_page_fault (vcpu, fault_address, error_code);
arch/x86/kvm/vmx/nested.c +1 -1
@@ -3880,7 +3880,7 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch) goto vmentry_failed;/* Hide L1D cache contents from the nested guest. */ - vcpu->arch.l1tf_flush_l1d = true; +kvm_request_l1tf_flush_l1d ();/* * Must happen outside of nested_vmx_enter_non_root_mode() as it will
arch/x86/kvm/vmx/vmx.c +5 -15
@@ -395,26 +395,16 @@ static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu)* 'always' */ if (static_branch_likely (&vmx_l1d_flush_cond)) { -bool flush_l1d; -/* -* Clear the per-vcpu flush bit, it gets set again if the vCPU +* Clear the per-cpu flush bit, it gets set again if the vCPU * is reloaded, i.e. if the vCPU is scheduled out or if KVM * exits to userspace, or if KVM reaches one of the unsafe -* VMEXIT handlers, e.g. if KVM calls into the emulator. +* VMEXIT handlers, e.g. if KVM calls into the emulator, +* or from the interrupt handlers. */ - flush_l1d = vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d = false; - -/* -* Clear the per-cpu flush bit, it gets set again from -* the interrupt handlers. -*/ - flush_l1d |=kvm_get_cpu_l1tf_flush_l1d (); +if (!kvm_get_cpu_l1tf_flush_l1d ()) +return ;kvm_clear_cpu_l1tf_flush_l1d (); - -if (!flush_l1d) -return false; } vcpu->stat.l1d_flush++;
arch/x86/kvm/x86.c +3 -3
@@ -5189,7 +5189,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); - vcpu->arch.l1tf_flush_l1d = true; +kvm_request_l1tf_flush_l1d ();if (vcpu->scheduled_out && pmu->version && pmu->event_count) { pmu->need_cleanup = true;
@@ -7999,7 +7999,7 @@ int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception) {/* kvm_write_guest_virt_system can pull in tons of pages. */ - vcpu->arch.l1tf_flush_l1d = true; +kvm_request_l1tf_flush_l1d ();return kvm_write_guest_virt_helper (addr, val, bytes, vcpu,PFERR_WRITE_MASK , exception);
@@ -9395,7 +9395,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,return handle_emulation_failure (vcpu, emulation_type); } - vcpu->arch.l1tf_flush_l1d = true; +kvm_request_l1tf_flush_l1d ();if (!(emulation_type &EMULTYPE_NO_DECODE )) {kvm_clear_exception_queue (vcpu);
arch/x86/kvm/x86.h +14 -0
@@ -420,6 +420,20 @@ static inline bool kvm_check_has_quirk(struct kvm *kvm, u64 quirk)return !(kvm->arch.disabled_quirks & quirk); } +static __always_inline void kvm_request_l1tf_flush_l1d(void) +{ +#if IS_ENABLED (CONFIG_CPU_MITIGATIONS ) &&IS_ENABLED (CONFIG_KVM_INTEL ) +/* +* Use a raw write to set the per-CPU flag, as KVM will ensure a flush +* even if preemption is currently enabled.. If the current vCPU task +* is migrated to a different CPU (or userspace runs the vCPU on a +* different task) before the next VM-Entry, then kvm_arch_vcpu_load() +* will request a flush on the new CPU. +*/ +raw_cpu_write (irq_stat.kvm_cpu_l1tf_flush_l1d,1 ); +#endif +} +void kvm_inject_realmode_interrupt (struct kvm_vcpu *vcpu,int irq,int inc_eip);u64 get_kvmclock_ns (struct kvm *kvm);
Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
Re: [PATCH v3 4/4] KVM: x86: Unify L1TF flushing under per-CPU variable
Quoted 17 lines · Sean Christopherson
arch/x86/kvm/vmx/vmx.c +3
-4@@ -395,26 +395,16 @@ static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu)* 'always' */ if (static_branch_likely (&vmx_l1d_flush_cond)) { -bool flush_l1d; -/* -* Clear the per-vcpu flush bit, it gets set again if the vCPU +* Clear the per-cpu flush bit, it gets set again if the vCPU * is reloaded, i.e. if the vCPU is scheduled out or if KVM * exits to userspace, or if KVM reaches one of the unsafe -* VMEXIT handlers, e.g. if KVM calls into the emulator. +* VMEXIT handlers, e.g. if KVM calls into the emulator, +* or from the interrupt handlers. */
arch/x86/kvm/vmx/vmx.c static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 406
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +2
-8- flush_l1d = vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d = false; - -/* -* Clear the per-cpu flush bit, it gets set again from -* the interrupt handlers. -*/ - flush_l1d |=kvm_get_cpu_l1tf_flush_l1d (); +if (!kvm_get_cpu_l1tf_flush_l1d ()) +return ;
This should be returning false here.
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +0
-3kvm_clear_cpu_l1tf_flush_l1d (); - -if (!flush_l1d) -return false; }