Skip to contentSkip to conversation

KVM: VMX: Unify L1D flush for L1TF

PATCH In review Sean Christopherson 5 participants 23 messages Oct 16, 2025 – Oct 28, 2025

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

v3 cover KVM: VMX: Unify L1D flush for L1TF

Unify the L1D cache flushing done to mitigate L1TF under the per-CPU variable, as the per-vCPU variable has been superfluous for quite some time.

Patch 1 fixes a bug (I think it's a bug?) I found when poking around the code. If L1D flushes are conditional and KVM skips an L1D flush on VM-Enter, then arguably KVM should flush CPU buffers based on other mitigations.

Patches 2-3 bury the L1TF L1D flushing under CONFIG_CPU_MITIGATIONS, partly because it's absurd that KVM doesn't honor CONFIG_CPU_MITIGATIONS for that case, partly because it simplifies unifying the tracking code (helps obviate the need for a stub).

Patch 4 is Brendan's patch and the main goal of the mini-series.

v3:
- Put the "raw" variant in KVM, dress it up with KVM's "request" terminology, and add a comment explaining why _KVM_ knows its usage doesn't need to disable virtualization.
- Add the prep patches.

v2:
- https://lore.kernel.org/all/[email protected]
- Moved the bit back to irq_stat
- Fixed DEBUG_PREEMPT issues by adding a _raw variant

v1: https://lore.kernel.org/r/[email protected]

Brendan Jackman (1):
KVM: x86: Unify L1TF flushing under per-CPU variable

Sean Christopherson (3):
KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped
KVM: VMX: Bundle all L1 data cache flush mitigation code together
KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n

 arch/x86/include/asm/hardirq.h  |   4 +-
 arch/x86/include/asm/kvm_host.h |   3 -
 arch/x86/kvm/mmu/mmu.c          |   2 +-
 arch/x86/kvm/vmx/nested.c       |   2 +-
 arch/x86/kvm/vmx/vmx.c          | 222 ++++++++++++++++++--------------
 arch/x86/kvm/x86.c              |   6 +-
 arch/x86/kvm/x86.h              |  14 ++
 7 files changed, 144 insertions(+), 109 deletions(-)

base-commit: f222788458c8a7753d43befef2769cd282dc008e

Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
author maintainer
#1 lore raw

PATCH v3 1/4 KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Note, the flaw goes back to the introduction of the MDS mitigation. The MDS mitigation was inadvertently fixed by commit 43fb862de8f6 ("KVM/VMX:
Move VERW closer to VMentry for MDS mitigation"), but previous kernels that flush CPU buffers in vmx_vcpu_enter_exit() are affected.

Fixes 650b68a0622f ("x86/kvm/vmx: Add MDS protection when L1D Flush is not active")
Cc Pawan Gupta <[email protected]>
---
 arch/x86/kvm/vmx/vmx.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/vmx.c +7 -5
index f87c216d976d..ce556d5dc39b 100644
@@ -6663,7 +6663,7 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
  * information but as all relevant affected CPUs have 32KiB L1D cache size
  * there is no point in doing so.
  */
-static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
+static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu)
 {
 	int size = PAGE_SIZE << L1D_CACHE_ORDER;
 
@@ -6691,14 +6691,14 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
 		kvm_clear_cpu_l1tf_flush_l1d();
 
 		if (!flush_l1d)
-			return;
+			return false;
 	}
 
 	vcpu->stat.l1d_flush++;
 
 	if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
 		native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
-		return;
+		return true;
 	}
 
 	asm volatile(
@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
 		:: [flush_pages] "r" (vmx_l1d_flush_pages),
 		    [size] "r" (size)
 		: "eax", "ebx", "ecx", "edx");
+	return true;
 }
 
 void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)
@@ -7330,8 +7331,9 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,
 	 * and is affected by MMIO Stale Data. In such cases mitigation in only
 	 * needed against an MMIO capable guest.
 	 */
-	if (static_branch_unlikely(&vmx_l1d_should_flush))
-		vmx_l1d_flush(vcpu);
+	if (static_branch_unlikely(&vmx_l1d_should_flush) &&
+	    vmx_l1d_flush(vcpu))
+		;
 	else if (static_branch_unlikely(&cpu_buf_vm_clear) &&
 		 (flags & VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO))
 		x86_clear_cpu_buffers();
Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
Reviewed-by on v3 1/4
#2 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 4 lines · Sean Christopherson

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug. If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.

Quoted 42 lines

MDS mitigation was inadvertently fixed by commit 43fb862de8f6 ("KVM/VMX:
Move VERW closer to VMentry for MDS mitigation"), but previous kernels that flush CPU buffers in vmx_vcpu_enter_exit() are affected.

Fixes 650b68a0622f ("x86/kvm/vmx: Add MDS protection when L1D Flush is not active")
Cc Pawan Gupta <[email protected]>
---
 arch/x86/kvm/vmx/vmx.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/vmx.c +3 -3
index f87c216d976d..ce556d5dc39b 100644
@@ -6663,7 +6663,7 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
  * information but as all relevant affected CPUs have 32KiB L1D cache size
  * there is no point in doing so.
  */
-static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
+static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu)
 {
 	int size = PAGE_SIZE << L1D_CACHE_ORDER;
 
@@ -6691,14 +6691,14 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
 		kvm_clear_cpu_l1tf_flush_l1d();
 
 		if (!flush_l1d)
-			return;
+			return false;
 	}
 
 	vcpu->stat.l1d_flush++;
 
 	if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
 		native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
-		return;
+		return true;
 	}
 
 	asm volatile(

arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
 		:: [flush_pages] "r" (vmx_l1d_flush_pages),
 		    [size] "r" (size)
 		: "eax", "ebx", "ecx", "edx");
+	return true;

The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)

Anyway, if this is an issue, it's orthogonal to this patch.

Quoted 15 lines
arch/x86/kvm/vmx/vmx.c +3 -2
 }
 
 void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)
@@ -7330,8 +7331,9 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,
 	 * and is affected by MMIO Stale Data. In such cases mitigation in only
 	 * needed against an MMIO capable guest.
 	 */
-	if (static_branch_unlikely(&vmx_l1d_should_flush))
-		vmx_l1d_flush(vcpu);
+	if (static_branch_unlikely(&vmx_l1d_should_flush) &&
+	    vmx_l1d_flush(vcpu))
+		;
 	else if (static_branch_unlikely(&cpu_buf_vm_clear) &&
 		 (flags & VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO))
 		x86_clear_cpu_buffers();
author maintainer
#3 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 11 lines · Brendan Jackman
Quoted 4 lines · Sean Christopherson

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug.

Yeah, that's where I'm at as well.

Quoted 2 lines

If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.

...

Quoted 11 lines

arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725

Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
 		:: [flush_pages] "r" (vmx_l1d_flush_pages),
 		    [size] "r" (size)
 		: "eax", "ebx", "ecx", "edx");
+	return true;

The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)

Quoted 1 line

Anyway, if this is an issue, it's orthogonal to this patch.

Pawan, any idea?

#4 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 31 lines · Sean Christopherson
Quoted 11 lines · Brendan Jackman
Quoted 4 lines · Sean Christopherson

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug.

Yeah, that's where I'm at as well.

Quoted 2 lines

If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.

...

Quoted 11 lines

arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725

Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
 		:: [flush_pages] "r" (vmx_l1d_flush_pages),
 		    [size] "r" (size)
 		: "eax", "ebx", "ecx", "edx");
+	return true;

The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)

Quoted 3 lines
Quoted 1 line

Anyway, if this is an issue, it's orthogonal to this patch.

Pawan, any idea?

I want to say yes, but let me first confirm this internally and get back to you.

#5 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 21 lines · Pawan Gupta
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Brendan Jackman
Quoted 4 lines · Sean Christopherson

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug.

Yeah, that's where I'm at as well.

Quoted 2 lines

If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.

...

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725

Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
 		:: [flush_pages] "r" (vmx_l1d_flush_pages),
 		    [size] "r" (size)
 		: "eax", "ebx", "ecx", "edx");
+	return true;

The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)

Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)

Anyways, I have asked for a confirmation if the sequence works for MMIO stale data also. I will update once I get a response.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Anyway, if this is an issue, it's orthogonal to this patch.

Pawan, any idea?

I want to say yes, but let me first confirm this internally and get back to you.

#6 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 7 lines · Pawan Gupta

...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)

KVM started reporting MD_CLEAR to userspace in Linux v5.2, but it didn't report L1D_FLUSH to userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.

#7 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 10 lines · Jim Mattson
Quoted 7 lines · Pawan Gupta

...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)

Quoted 4 lines

KVM started reporting MD_CLEAR to userspace in Linux v5.2, but it didn't report L1D_FLUSH to userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.

Shouldn't only the L0 hypervisor be doing the L1D_FLUSH?

kvm_get_arch_capabilities()
{
...
/*
* If we're doing cache flushes (either "always" or "cond")
* we will do one whenever the guest does a vmlaunch/vmresume.
* If an outer hypervisor is doing the cache flush for us
* (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that
* capability to the guest too, and if EPT is disabled we're not
* vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will
* require a nested hypervisor to do a flush of its own. */
if (l1tf_vmx_mitigation != VMENTER_L1D_FLUSH_NEVER)
data |= ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;

#8 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 19 lines · Pawan Gupta
Quoted 10 lines · Jim Mattson
Quoted 7 lines · Pawan Gupta

...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)

Quoted 4 lines

KVM started reporting MD_CLEAR to userspace in Linux v5.2, but it didn't report L1D_FLUSH to userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.

Shouldn't only the L0 hypervisor be doing the L1D_FLUSH?

Quoted 15 lines

kvm_get_arch_capabilities()
{
...
/*
* If we're doing cache flushes (either "always" or "cond")
* we will do one whenever the guest does a vmlaunch/vmresume.
* If an outer hypervisor is doing the cache flush for us
* (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that
* capability to the guest too, and if EPT is disabled we're not
* vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will
* require a nested hypervisor to do a flush of its own. */
if (l1tf_vmx_mitigation != VMENTER_L1D_FLUSH_NEVER)
data |= ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;

Unless L0 has chosen L1D_FLUSH_NEVER. :)

On GCE's L1TF-vulnerable hosts, we actually do an L1D flush at ASI entry rather than VM-entry. ASI entries are two orders of magnitude less frequent than VM-entries, so we get comparable protection to L1D_FLUSH_ALWAYS at a fraction of the cost.

At the moment, we still do an L1D flush on emulated VM-entry, but that's just because we have historically advertised IA32_ARCH_CAPABILITIES.SKIP_L1DFL_VMENTRY to L1.

#9 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 39 lines · Jim Mattson
Quoted 19 lines · Pawan Gupta
Quoted 10 lines · Jim Mattson
Quoted 7 lines · Pawan Gupta

...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)

Quoted 4 lines

KVM started reporting MD_CLEAR to userspace in Linux v5.2, but it didn't report L1D_FLUSH to userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.

Shouldn't only the L0 hypervisor be doing the L1D_FLUSH?

Quoted 15 lines

kvm_get_arch_capabilities()
{
...
/*
* If we're doing cache flushes (either "always" or "cond")
* we will do one whenever the guest does a vmlaunch/vmresume.
* If an outer hypervisor is doing the cache flush for us
* (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that
* capability to the guest too, and if EPT is disabled we're not
* vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will
* require a nested hypervisor to do a flush of its own. */
if (l1tf_vmx_mitigation != VMENTER_L1D_FLUSH_NEVER)
data |= ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;

Unless L0 has chosen L1D_FLUSH_NEVER. :)

Quoted 8 lines

On GCE's L1TF-vulnerable hosts, we actually do an L1D flush at ASI entry rather than VM-entry. ASI entries are two orders of magnitude less frequent than VM-entries, so we get comparable protection to L1D_FLUSH_ALWAYS at a fraction of the cost.

At the moment, we still do an L1D flush on emulated VM-entry, but that's just because we have historically advertised IA32_ARCH_CAPABILITIES.SKIP_L1DFL_VMENTRY to L1.

Thanks for the background.

I still don't see the problem, CPUs that are vulnerable to L1TF are also vulnerable to MDS. So, they don't set mmio_stale_data_clear, instead they set X86_FEATURE_CLEAR_CPU_BUF and execute VERW in __vmx_vcpu_run() regardless of whether L1D_FLUSH was done.

But, I agree it is best to decouple L1D flush and MMIO Stale Data to be avoid any confusion.

#10 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 45 lines · Pawan Gupta
Quoted 44 lines · Jim Mattson
Quoted 19 lines · Pawan Gupta
Quoted 10 lines · Jim Mattson
Quoted 7 lines · Pawan Gupta

...
Thinking more on this, the software sequence is only invoked when the system doesn't have the L1D flushing feature added by a microcode update. In such a case system is not expected to have a flushing VERW either, which was introduced after L1TF. Also, the admin needs to have a very good reason for not updating the microcode for 5+ years :-)

Quoted 4 lines

KVM started reporting MD_CLEAR to userspace in Linux v5.2, but it didn't report L1D_FLUSH to userspace until Linux v6.4, so there are plenty of virtual CPUs with a flushing VERW that don't have the L1D flushing feature.

Shouldn't only the L0 hypervisor be doing the L1D_FLUSH?

Quoted 15 lines

kvm_get_arch_capabilities()
{
...
/*
* If we're doing cache flushes (either "always" or "cond")
* we will do one whenever the guest does a vmlaunch/vmresume.
* If an outer hypervisor is doing the cache flush for us
* (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that
* capability to the guest too, and if EPT is disabled we're not
* vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will
* require a nested hypervisor to do a flush of its own. */
if (l1tf_vmx_mitigation != VMENTER_L1D_FLUSH_NEVER)
data |= ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;

Unless L0 has chosen L1D_FLUSH_NEVER. :)

On GCE's L1TF-vulnerable hosts, we actually do an L1D flush at ASI entry rather than VM-entry. ASI entries are two orders of magnitude less frequent than VM-entries, so we get comparable protection to L1D_FLUSH_ALWAYS at a fraction of the cost.

Quoted 8 lines
Quoted 3 lines

At the moment, we still do an L1D flush on emulated VM-entry, but that's just because we have historically advertised IA32_ARCH_CAPABILITIES.SKIP_L1DFL_VMENTRY to L1.

Thanks for the background.

I still don't see the problem, CPUs that are vulnerable to L1TF are also vulnerable to MDS. So, they don't set mmio_stale_data_clear, instead they

Sorry I meant cpu_buf_vm_clear instead of mmio_stale_data_clear (I was looking at a slightly older kernel).

Quoted 5 lines

set X86_FEATURE_CLEAR_CPU_BUF and execute VERW in __vmx_vcpu_run() regardless of whether L1D_FLUSH was done.

But, I agree it is best to decouple L1D flush and MMIO Stale Data to be avoid any confusion.

#11 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

In reply to Pawan Gupta
Quoted 32 lines · Pawan Gupta
Quoted 31 lines · Sean Christopherson
Quoted 11 lines · Brendan Jackman
Quoted 4 lines · Sean Christopherson

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Presumably the assumption here was that the L1TF conditionality is good enough for the MMIO stale data vuln too? I'm not qualified to assess if that assumption is true, but also even if it's a good one it's definitely not obvious to users that the mitigation you pick for L1TF has this side-effect. So I think I'm on board with calling this a bug.

Yeah, that's where I'm at as well.

Quoted 2 lines

If anyone turns out to be depending on the current behaviour for performance I think they should probably add it back as a separate flag.

...

Quoted 11 lines

arch/x86/kvm/vmx/vmx.c static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 6725

Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -6722,6 +6722,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
 		:: [flush_pages] "r" (vmx_l1d_flush_pages),
 		    [size] "r" (size)
 		: "eax", "ebx", "ecx", "edx");
+	return true;

The comment in the caller says the L1D flush "includes CPU buffer clear to mitigate MDS" - do we actually know that this software sequence mitigates the MMIO stale data vuln like the verw does? (Do we even know if it mitigates MDS?)

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Anyway, if this is an issue, it's orthogonal to this patch.

Pawan, any idea?

I want to say yes, but let me first confirm this internally and get back to you.

The software sequence for L1D flush was not validated to mitigate MMIO Stale Data. To be on safer side, it is better to not rely on the sequence.

OTOH, if a user has not updated the microcode to mitigate L1TF, the system will not have the microcode to mitigate MMIO Stale Data either, because the microcode for MMIO Stale Data was released after L1TF. Also I am not aware of any CPUs that are vulnerable to L1TF and vulnerable to MMIO Stale Data only(not MDS).

So in practice, decoupling L1D flush and MMIO Stale Data won't have any practical impact on functionality, and makes MMIO Stale Data mitigation consistent with MDS mitigation. I hope that makes things clear.

#12 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

In reply to Sean Christopherson
Quoted 6 lines · Sean Christopherson

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Note, the flaw goes back to the introduction of the MDS mitigation.

I don't think it is a flaw. If L1D flush was skipped because VMexit did not touch any interested data, then there shouldn't be any need to flush CPU buffers.

Secondly, when L1D flush is skipped, flushing MDS affected buffers is of no use, because the data could still be extracted from L1D cache using L1TF. Isn't it?

#13 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 8 lines · Pawan Gupta
Quoted 6 lines · Sean Christopherson

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Note, the flaw goes back to the introduction of the MDS mitigation.

Quoted 7 lines

I don't think it is a flaw. If L1D flush was skipped because VMexit did not touch any interested data, then there shouldn't be any need to flush CPU buffers.

Secondly, when L1D flush is skipped, flushing MDS affected buffers is of no use, because the data could still be extracted from L1D cache using L1TF. Isn't it?

This is assuming an equivalence between what L1TF and MMIO Stale Data exploits can do, that isn't really captured in the code/documentation IMO. This probably felt much more obvious when the vulns were new...

I dunno, in the end this definitely doesn't seem like a terrifying big deal, I'm not saying the current behaviour is crazy or anything, it's just slightly surprising and people with sophisticated opinions about this might not be getting what they think they are out of the default setup.

But I have no evidence that these sophisticated dissidents actually exist, maybe just adding commentary about this rationale is more than good enough here.

author maintainer
#14 lore raw

Re: [PATCH v3 1/4] KVM: VMX: Flush CPU buffers as needed if L1D cache flush is skipped

Quoted 7 lines · Brendan Jackman
Quoted 6 lines · Pawan Gupta
Quoted 5 lines · Sean Christopherson

If the L1D flush for L1TF is conditionally enabled, flush CPU buffers to mitigate MMIO Stale Data as needed if KVM skips the L1D flush, e.g. because none of the "heavy" paths that trigger an L1D flush were tripped since the last VM-Enter.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Note, the flaw goes back to the introduction of the MDS mitigation.

I don't think it is a flaw. If L1D flush was skipped because VMexit did not touch any interested data, then there shouldn't be any need to flush CPU buffers.

But as Brendan alludes to below, that assumes certain aspects of L1TF and MDS are equal. Obliterating the L1D is far more costly than flushing CPU buffers, as evidenced by the much more conditional flushing for L1TF. My read of the L1TF mitigation is that the conditional flushing is that it's a compromise between performance and security. Skipping the flush doesn't necessarily mean nothing interesting was accessed, it just means that KVM didn't hit any of the flows where a large amount of interesting data was guaranteed to have been accessed.

Quoted 7 lines
Quoted 3 lines

Secondly, when L1D flush is skipped, flushing MDS affected buffers is of no use, because the data could still be extracted from L1D cache using L1TF. Isn't it?

This is assuming an equivalence between what L1TF and MMIO Stale Data exploits can do, that isn't really captured in the code/documentation IMO.

And again, the cost. To fully mitigate L1TF, KVM would need to flush on every entry, but that completely tanks performance. But that doesn't

Quoted 7 lines

This probably felt much more obvious when the vulns were new...

I dunno, in the end this definitely doesn't seem like a terrifying big deal, I'm not saying the current behaviour is crazy or anything, it's just slightly surprising and people with sophisticated opinions about this might not be getting what they think they are out of the default setup.

Ya. I highly doubt this particular combination matters in practice, but I don't like surprises. And I find it surprising that the behavior of KVM's mitigation for MMIO Stale Data changes based on whether or not the L1TF mitigation is enabled.

Quoted 3 lines

But I have no evidence that these sophisticated dissidents actually exist, maybe just adding commentary about this rationale is more than good enough here.

author maintainer
#15 lore raw

PATCH v3 2/4 KVM: VMX: Bundle all L1 data cache flush mitigation code together

In reply to Sean Christopherson

Move vmx_l1d_flush(), vmx_cleanup_l1d_flush(), and the vmentry_l1d_flush param code up in vmx.c so that all of the L1 data cache flushing code is bundled together. This will allow conditioning the mitigation code on CONFIG_CPU_MITIGATIONS=y with minimal #ifdefs.

No functional change intended.

---
 arch/x86/kvm/vmx/vmx.c | 176 ++++++++++++++++++++---------------------
 1 file changed, 88 insertions(+), 88 deletions(-)
arch/x86/kvm/vmx/vmx.c +88 -88
index ce556d5dc39b..cd8ae1b2ae55 100644
@@ -302,6 +302,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)
 	return 0;
 }
 
+static void vmx_cleanup_l1d_flush(void)
+{
+	if (vmx_l1d_flush_pages) {
+		free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER);
+		vmx_l1d_flush_pages = NULL;
+	}
+	/* Restore state so sysfs ignores VMX */
+	l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
+}
+
 static int vmentry_l1d_flush_parse(const char *s)
 {
 	unsigned int i;
@@ -352,6 +362,84 @@ static int vmentry_l1d_flush_get(char *s, const struct kernel_param *kp)
 	return sysfs_emit(s, "%s\n", vmentry_l1d_param[l1tf_vmx_mitigation].option);
 }
 
+/*
+ * Software based L1D cache flush which is used when microcode providing
+ * the cache control MSR is not loaded.
+ *
+ * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to
+ * flush it is required to read in 64 KiB because the replacement algorithm
+ * is not exactly LRU. This could be sized at runtime via topology
+ * information but as all relevant affected CPUs have 32KiB L1D cache size
+ * there is no point in doing so.
+ */
+static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu)
+{
+	int size = PAGE_SIZE << L1D_CACHE_ORDER;
+
+	/*
+	 * This code is only executed when the flush mode is 'cond' or
+	 * 'always'
+	 */
+	if (static_branch_likely(&vmx_l1d_flush_cond)) {
+		bool flush_l1d;
+
+		/*
+		 * Clear the per-vcpu flush bit, it gets set again if the vCPU
+		 * is reloaded, i.e. if the vCPU is scheduled out or if KVM
+		 * exits to userspace, or if KVM reaches one of the unsafe
+		 * VMEXIT handlers, e.g. if KVM calls into the emulator.
+		 */
+		flush_l1d = vcpu->arch.l1tf_flush_l1d;
+		vcpu->arch.l1tf_flush_l1d = false;
+
+		/*
+		 * Clear the per-cpu flush bit, it gets set again from
+		 * the interrupt handlers.
+		 */
+		flush_l1d |= kvm_get_cpu_l1tf_flush_l1d();
+		kvm_clear_cpu_l1tf_flush_l1d();
+
+		if (!flush_l1d)
+			return false;
+	}
+
+	vcpu->stat.l1d_flush++;
+
+	if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
+		native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
+		return true;
+	}
+
+	asm volatile(
+		/* First ensure the pages are in the TLB */
+		"xorl	%%eax, %%eax\n"
+		".Lpopulate_tlb:\n\t"
+		"movzbl	(%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
+		"addl	$4096, %%eax\n\t"
+		"cmpl	%%eax, %[size]\n\t"
+		"jne	.Lpopulate_tlb\n\t"
+		"xorl	%%eax, %%eax\n\t"
+		"cpuid\n\t"
+		/* Now fill the cache */
+		"xorl	%%eax, %%eax\n"
+		".Lfill_cache:\n"
+		"movzbl	(%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
+		"addl	$64, %%eax\n\t"
+		"cmpl	%%eax, %[size]\n\t"
+		"jne	.Lfill_cache\n\t"
+		"lfence\n"
+		:: [flush_pages] "r" (vmx_l1d_flush_pages),
+		    [size] "r" (size)
+		: "eax", "ebx", "ecx", "edx");
+	return true;
+}
+
+static const struct kernel_param_ops vmentry_l1d_flush_ops = {
+	.set = vmentry_l1d_flush_set,
+	.get = vmentry_l1d_flush_get,
+};
+module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644);
+
 static __always_inline void vmx_disable_fb_clear(struct vcpu_vmx *vmx)
 {
 	u64 msr;
@@ -404,12 +492,6 @@ static void vmx_update_fb_clear_dis(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx)
 		vmx->disable_fb_clear = false;
 }
 
-static const struct kernel_param_ops vmentry_l1d_flush_ops = {
-	.set = vmentry_l1d_flush_set,
-	.get = vmentry_l1d_flush_get,
-};
-module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644);
-
 static u32 vmx_segment_access_rights(struct kvm_segment *var);
 
 void vmx_vmexit(void);
@@ -6653,78 +6735,6 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 	return ret;
 }
 
-/*
- * Software based L1D cache flush which is used when microcode providing
- * the cache control MSR is not loaded.
- *
- * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to
- * flush it is required to read in 64 KiB because the replacement algorithm
- * is not exactly LRU. This could be sized at runtime via topology
- * information but as all relevant affected CPUs have 32KiB L1D cache size
- * there is no point in doing so.
- */
-static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu)
-{
-	int size = PAGE_SIZE << L1D_CACHE_ORDER;
-
-	/*
-	 * This code is only executed when the flush mode is 'cond' or
-	 * 'always'
-	 */
-	if (static_branch_likely(&vmx_l1d_flush_cond)) {
-		bool flush_l1d;
-
-		/*
-		 * Clear the per-vcpu flush bit, it gets set again if the vCPU
-		 * is reloaded, i.e. if the vCPU is scheduled out or if KVM
-		 * exits to userspace, or if KVM reaches one of the unsafe
-		 * VMEXIT handlers, e.g. if KVM calls into the emulator.
-		 */
-		flush_l1d = vcpu->arch.l1tf_flush_l1d;
-		vcpu->arch.l1tf_flush_l1d = false;
-
-		/*
-		 * Clear the per-cpu flush bit, it gets set again from
-		 * the interrupt handlers.
-		 */
-		flush_l1d |= kvm_get_cpu_l1tf_flush_l1d();
-		kvm_clear_cpu_l1tf_flush_l1d();
-
-		if (!flush_l1d)
-			return false;
-	}
-
-	vcpu->stat.l1d_flush++;
-
-	if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
-		native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
-		return true;
-	}
-
-	asm volatile(
-		/* First ensure the pages are in the TLB */
-		"xorl	%%eax, %%eax\n"
-		".Lpopulate_tlb:\n\t"
-		"movzbl	(%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
-		"addl	$4096, %%eax\n\t"
-		"cmpl	%%eax, %[size]\n\t"
-		"jne	.Lpopulate_tlb\n\t"
-		"xorl	%%eax, %%eax\n\t"
-		"cpuid\n\t"
-		/* Now fill the cache */
-		"xorl	%%eax, %%eax\n"
-		".Lfill_cache:\n"
-		"movzbl	(%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
-		"addl	$64, %%eax\n\t"
-		"cmpl	%%eax, %[size]\n\t"
-		"jne	.Lfill_cache\n\t"
-		"lfence\n"
-		:: [flush_pages] "r" (vmx_l1d_flush_pages),
-		    [size] "r" (size)
-		: "eax", "ebx", "ecx", "edx");
-	return true;
-}
-
 void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
@@ -8673,16 +8683,6 @@ __init int vmx_hardware_setup(void)
 	return r;
 }
 
-static void vmx_cleanup_l1d_flush(void)
-{
-	if (vmx_l1d_flush_pages) {
-		free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER);
-		vmx_l1d_flush_pages = NULL;
-	}
-	/* Restore state so sysfs ignores VMX */
-	l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
-}
-
 void vmx_exit(void)
 {
 	allow_smaller_maxphyaddr = false;
Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
Reviewed-by on v3 2/4
#16 lore raw

Re: [PATCH v3 2/4] KVM: VMX: Bundle all L1 data cache flush mitigation code together

Quoted 8 lines · Sean Christopherson

Move vmx_l1d_flush(), vmx_cleanup_l1d_flush(), and the vmentry_l1d_flush param code up in vmx.c so that all of the L1 data cache flushing code is bundled together. This will allow conditioning the mitigation code on CONFIG_CPU_MITIGATIONS=y with minimal #ifdefs.

No functional change intended.

(Git says no changed lines)

author maintainer
#17 lore raw

PATCH v3 3/4 KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n

In reply to Sean Christopherson

Disable support for flushing the L1 data cache to mitigate L1TF if CPU mitigations are disabled for the entire kernel. KVM's mitigation of L1TF is in no way special enough to justify ignoring CONFIG_CPU_MITIGATIONS=n.

Deliberately use CPU_MITIGATIONS instead of the more precise MITIGATION_L1TF, as MITIGATION_L1TF only controls the default behavior, i.e. CONFIG_MITIGATION_L1TF=n doesn't completely disable L1TF mitigations in the kernel.

Keep the vmentry_l1d_flush module param to avoid breaking existing setups, and leverage the .set path to alert the user to the fact that vmentry_l1d_flush will be ignored. Don't bother validating the incoming value; if an admin misconfigures vmentry_l1d_flush, the fact that the bad configuration won't be detected when running with CONFIG_CPU_MITIGATIONS=n is likely the least of their worries.

---
 arch/x86/include/asm/hardirq.h |  4 +--
 arch/x86/kvm/vmx/vmx.c         | 56 ++++++++++++++++++++++++++--------
 2 files changed, 46 insertions(+), 14 deletions(-)
arch/x86/include/asm/hardirq.h +2 -2
index f00c09ffe6a9..6b6d472baa0b 100644
@@ -5,7 +5,7 @@
 #include <linux/threads.h>
 
 typedef struct {
-#if IS_ENABLED(CONFIG_KVM_INTEL)
+#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL)
 	u8	     kvm_cpu_l1tf_flush_l1d;
 #endif
 	unsigned int __nmi_count;	/* arch dependent */
@@ -68,7 +68,7 @@ extern u64 arch_irq_stat(void);
 DECLARE_PER_CPU_CACHE_HOT(u16, __softirq_pending);
 #define local_softirq_pending_ref       __softirq_pending
 
-#if IS_ENABLED(CONFIG_KVM_INTEL)
+#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL)
 /*
  * This function is called from noinstr interrupt contexts
  * and must be inlined to not get instrumentation.
arch/x86/kvm/vmx/vmx.c +44 -12

104 lines not on this page · raw message

Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
#18 lore raw

Re: [PATCH v3 3/4] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n

kernel test robot noticed the following build errors:

Hi Sean,

kernel test robot noticed the following build errors:

[auto build test ERROR on f222788458c8a7753d43befef2769cd282dc008e]

url:    https://github.com/intel-lab-lkp/linux/commits/Sean-Christopherson/KVM-VMX-Flush-CPU-buffers-as-needed-if-L1D-cache-flush-is-skipped/20251017-040642
base:   f222788458c8a7753d43befef2769cd282dc008e
patch link:    https://lore.kernel.org/r/20251016200417.97003-4-seanjc%40google.com
patch subject: [PATCH v3 3/4] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n
config: x86_64-kexec (https://download.01.org/0day-ci/archive/20251018/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251018/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> arch/x86/kvm/vmx/vmx.c:313:29: error: too many arguments to function call, expected 0, have 1
     313 |         return vmx_setup_l1d_flush(vmentry_l1d_flush_param);
         |                ~~~~~~~~~~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~

arch/x86/kvm/vmx/vmx.c:306:12: note: 'vmx_setup_l1d_flush' declared here
306 | static int vmx_setup_l1d_flush(void)
| ^
1 error generated.

vim +313 arch/x86/kvm/vmx/vmx.c

305
306 static int vmx_setup_l1d_flush(void)
307 {
308 /*
309 * Hand the parameter mitigation value in which was stored in the pre
310 * module init parser. If no parameter was given, it will contain
311 * 'auto' which will be turned into the default 'cond' mitigation mode.
312 */

 > 313		return vmx_setup_l1d_flush(vmentry_l1d_flush_param);

314 }
315

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#19 lore raw

Re: [PATCH v3 3/4] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n

In reply to Sean Christopherson

arch/x86/kvm/vmx/vmx.c static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) · line 313

Quoted 12 lines · Sean Christopherson
arch/x86/kvm/vmx/vmx.c +8 -0
@@ -302,6 +303,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)
 	return 0;
 }
 
+static int vmx_setup_l1d_flush(void)
+{
+	/*
+	 * Hand the parameter mitigation value in which was stored in the pre
+	 * module init parser. If no parameter was given, it will contain
+	 * 'auto' which will be turned into the default 'cond' mitigation mode.
+	 */
+	return vmx_setup_l1d_flush(vmentry_l1d_flush_param);

A likely typo here, it should be:

	return __vmx_setup_l1d_flush(vmentry_l1d_flush_param);
author maintainer
#20 lore raw

Re: [PATCH v3 3/4] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n

Quoted 16 lines · Pawan Gupta

arch/x86/kvm/vmx/vmx.c static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) · line 313

Quoted 12 lines · Sean Christopherson
arch/x86/kvm/vmx/vmx.c +8 -0
@@ -302,6 +303,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)
 	return 0;
 }
 
+static int vmx_setup_l1d_flush(void)
+{
+	/*
+	 * Hand the parameter mitigation value in which was stored in the pre
+	 * module init parser. If no parameter was given, it will contain
+	 * 'auto' which will be turned into the default 'cond' mitigation mode.
+	 */
+	return vmx_setup_l1d_flush(vmentry_l1d_flush_param);

A likely typo here, it should be:

Quoted 1 line
	return __vmx_setup_l1d_flush(vmentry_l1d_flush_param);

Argh, I have a feeling I clobbered my branch with a --force push, as I remember fixing this exact problem. Or maybe I saw Brendan's struggles and thought, "hold my beer!" :-D

author maintainer
#21 lore raw

PATCH v3 4/4 KVM: x86: Unify L1TF flushing under per-CPU variable

In reply to Sean Christopherson

From: Brendan Jackman <[email protected]>

Currently the tracking of the need to flush L1D for L1TF is tracked by two bits: one per-CPU and one per-vCPU.

The per-vCPU bit is always set when the vCPU shows up on a core, so there is no interesting state that's truly per-vCPU. Indeed, this is a requirement, since L1D is a part of the physical CPU.

So simplify this by combining the two bits.

The vCPU bit was being written from preemption-enabled regions. To play nice with those cases, wrap all calls from KVM and use a raw write so that request a flush with preemption enabled doesn't trigger what would effectively be DEBUG_PREEMPT false positives. Preemption doesn't need to be disabled, as kvm_arch_vcpu_load() will mark the new CPU as needing a flush if the vCPU task is migrated, or if userspace runs the vCPU on a different task.

[sean: put raw write in KVM instead of in a hardirq.h variant]
---
 arch/x86/include/asm/kvm_host.h |  3 ---
 arch/x86/kvm/mmu/mmu.c          |  2 +-
 arch/x86/kvm/vmx/nested.c       |  2 +-
 arch/x86/kvm/vmx/vmx.c          | 20 +++++---------------
 arch/x86/kvm/x86.c              |  6 +++---
 arch/x86/kvm/x86.h              | 14 ++++++++++++++
 6 files changed, 24 insertions(+), 23 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -3
index 48598d017d6f..fcdc65ab13d8 100644
@@ -1055,9 +1055,6 @@ struct kvm_vcpu_arch {
 	/* be preempted when it's in kernel-mode(cpl=0) */
 	bool preempted_in_kernel;
 
-	/* Flush the L1 Data cache for L1TF mitigation on VMENTER */
-	bool l1tf_flush_l1d;
-
 	/* Host CPU on which VM-entry was most recently attempted */
 	int last_vmentry_cpu;
 
arch/x86/kvm/mmu/mmu.c +1 -1
index 18d69d48bc55..4e016582adc7 100644
@@ -4859,7 +4859,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code,
 	 */
 	BUILD_BUG_ON(lower_32_bits(PFERR_SYNTHETIC_MASK));
 
-	vcpu->arch.l1tf_flush_l1d = true;
+	kvm_request_l1tf_flush_l1d();
 	if (!flags) {
 		trace_kvm_page_fault(vcpu, fault_address, error_code);
 
arch/x86/kvm/vmx/nested.c +1 -1
index 3fca63a261f5..468a013d9ef3 100644
@@ -3880,7 +3880,7 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)
 		goto vmentry_failed;
 
 	/* Hide L1D cache contents from the nested guest.  */
-	vcpu->arch.l1tf_flush_l1d = true;
+	kvm_request_l1tf_flush_l1d();
 
 	/*
 	 * Must happen outside of nested_vmx_enter_non_root_mode() as it will
arch/x86/kvm/vmx/vmx.c +5 -15
index e91d99211efe..0347d321a86e 100644
@@ -395,26 +395,16 @@ static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu)
 	 * 'always'
 	 */
 	if (static_branch_likely(&vmx_l1d_flush_cond)) {
-		bool flush_l1d;
-
 		/*
-		 * Clear the per-vcpu flush bit, it gets set again if the vCPU
+		 * Clear the per-cpu flush bit, it gets set again if the vCPU
 		 * is reloaded, i.e. if the vCPU is scheduled out or if KVM
 		 * exits to userspace, or if KVM reaches one of the unsafe
-		 * VMEXIT handlers, e.g. if KVM calls into the emulator.
+		 * VMEXIT handlers, e.g. if KVM calls into the emulator,
+		 * or from the interrupt handlers.
 		 */
-		flush_l1d = vcpu->arch.l1tf_flush_l1d;
-		vcpu->arch.l1tf_flush_l1d = false;
-
-		/*
-		 * Clear the per-cpu flush bit, it gets set again from
-		 * the interrupt handlers.
-		 */
-		flush_l1d |= kvm_get_cpu_l1tf_flush_l1d();
+		if (!kvm_get_cpu_l1tf_flush_l1d())
+			return;
 		kvm_clear_cpu_l1tf_flush_l1d();
-
-		if (!flush_l1d)
-			return false;
 	}
 
 	vcpu->stat.l1d_flush++;
arch/x86/kvm/x86.c +3 -3
index b4b5d2d09634..851f078cd5ca 100644
@@ -5189,7 +5189,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 {
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 
-	vcpu->arch.l1tf_flush_l1d = true;
+	kvm_request_l1tf_flush_l1d();
 
 	if (vcpu->scheduled_out && pmu->version && pmu->event_count) {
 		pmu->need_cleanup = true;
@@ -7999,7 +7999,7 @@ int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val,
 				unsigned int bytes, struct x86_exception *exception)
 {
 	/* kvm_write_guest_virt_system can pull in tons of pages. */
-	vcpu->arch.l1tf_flush_l1d = true;
+	kvm_request_l1tf_flush_l1d();
 
 	return kvm_write_guest_virt_helper(addr, val, bytes, vcpu,
 					   PFERR_WRITE_MASK, exception);
@@ -9395,7 +9395,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		return handle_emulation_failure(vcpu, emulation_type);
 	}
 
-	vcpu->arch.l1tf_flush_l1d = true;
+	kvm_request_l1tf_flush_l1d();
 
 	if (!(emulation_type & EMULTYPE_NO_DECODE)) {
 		kvm_clear_exception_queue(vcpu);
arch/x86/kvm/x86.h +14 -0
index f3dc77f006f9..cd67ccbb747f 100644
@@ -420,6 +420,20 @@ static inline bool kvm_check_has_quirk(struct kvm *kvm, u64 quirk)
 	return !(kvm->arch.disabled_quirks & quirk);
 }
 
+static __always_inline void kvm_request_l1tf_flush_l1d(void)
+{
+#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL)
+	/*
+	 * Use a raw write to set the per-CPU flag, as KVM will ensure a flush
+	 * even if preemption is currently enabled..  If the current vCPU task
+	 * is migrated to a different CPU (or userspace runs the vCPU on a
+	 * different task) before the next VM-Entry, then kvm_arch_vcpu_load()
+	 * will request a flush on the new CPU.
+	 */
+	raw_cpu_write(irq_stat.kvm_cpu_l1tf_flush_l1d, 1);
+#endif
+}
+
 void kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc_eip);
 
 u64 get_kvmclock_ns(struct kvm *kvm);
Signature · 1 line
2.51.0.858.gf9c4a03a3a-goog
#22 lore raw

Re: [PATCH v3 4/4] KVM: x86: Unify L1TF flushing under per-CPU variable

Quoted 17 lines · Sean Christopherson
arch/x86/kvm/vmx/vmx.c +3 -4
@@ -395,26 +395,16 @@ static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu)
 	 * 'always'
 	 */
 	if (static_branch_likely(&vmx_l1d_flush_cond)) {
-		bool flush_l1d;
-
 		/*
-		 * Clear the per-vcpu flush bit, it gets set again if the vCPU
+		 * Clear the per-cpu flush bit, it gets set again if the vCPU
 		 * is reloaded, i.e. if the vCPU is scheduled out or if KVM
 		 * exits to userspace, or if KVM reaches one of the unsafe
-		 * VMEXIT handlers, e.g. if KVM calls into the emulator.
+		 * VMEXIT handlers, e.g. if KVM calls into the emulator,
+		 * or from the interrupt handlers.
 		 */

arch/x86/kvm/vmx/vmx.c static noinstr bool vmx_l1d_flush(struct kvm_vcpu *vcpu) · line 406

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +2 -8
-		flush_l1d = vcpu->arch.l1tf_flush_l1d;
-		vcpu->arch.l1tf_flush_l1d = false;
-
-		/*
-		 * Clear the per-cpu flush bit, it gets set again from
-		 * the interrupt handlers.
-		 */
-		flush_l1d |= kvm_get_cpu_l1tf_flush_l1d();
+		if (!kvm_get_cpu_l1tf_flush_l1d())
+			return;

This should be returning false here.

Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +0 -3
 		kvm_clear_cpu_l1tf_flush_l1d();
-
-		if (!flush_l1d)
-			return false;
 	}