Skip to contentSkip to conversation

KVM: nVMX: Fix ept=n bugs where KVM runs L2 with guest CR3

PATCH Applied Sean Christopherson 3 participants 6 messages Jun 12 – Jul 14

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

v2 cover KVM: nVMX: Fix ept=n bugs where KVM runs L2 with guest CR3

Fix two bugs where KVM can run L2 with a guest-controlled CR3. The underlying flaw dates back to commit f087a02941fe ("KVM: nVMX: Stash L1's CR3 in vmcs01.GUEST_CR3 on nested entry w/o EPT"). Past me claimed:

Smashing vmcs01.GUEST_CR3 is safe because nested VM-Exits, and the unwind, reset KVM's MMU, i.e. vmcs01.GUEST_CR3 is guaranteed to be overwritten with a shadow CR3 prior to re-entering L1.

which was and is true, _if_ a nested VM-Exit or the unwind is reached. If KVM fails directly, vmcs01.GUEST_CR3 will be left pointing at L1's actual CR3, i.e. KVM will run with legacy shadow paging a guest-controlled CR3, which is... not good.

v2:
- Use kvm_read_cr3() to read vcpu->arch.cr3. [Sashiko]
- Skip consistency check if reading vTPR fails. [Jim]

v1: https://lore.kernel.org/all/[email protected]

Sean Christopherson (2):
KVM: nVMX: Move vTPR vs. TPR Threshold consistency check into "normal"
checks
KVM: nVMX: Don't use vmcs01.GUEST_CR3 to snapshot L1's CR3 when EPT is
disabled

 arch/x86/kvm/vmx/nested.c | 87 +++++++++++++++++----------------------
 arch/x86/kvm/vmx/vmx.h    |  7 ++++
 2 files changed, 44 insertions(+), 50 deletions(-)

base-commit: de3a35be92d2391ece4bf3143ef2887192625fd0

Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
author maintainer
#1 lore raw

PATCH v2 1/2 KVM: nVMX: Move vTPR vs. TPR Threshold consistency check into "normal" checks

Move the off-by-default consistency check for vmcs12.tpr_threshold vs. the virtual APIC vTPR into the "normal" controls checks, as waiting until KVM has loaded some amount of state is unnecessary and actively dangerous. Specifically, failure to unwind vmcs01.GUEST_CR3 to KVM's value when EPT is disabled results in KVM running L1 with an L1-controlled CR3, not with KVM's CR3!

Alternatively, KVM could simply reset the MMU to force a reload of vmcs01.GUEST_CR3, but the _only_ reason the check was shoved into a "late" flow was to wait until the vmcs12 pages were retrieved. Rather than build up more crusty code, simply access vTPR using a regular guest memory access (performance isn't a concern). To circumvent the restrictions that led to KVM deferring nested_get_vmcs12_pages(), (a) use a VM-scoped API to read guest memory so that it always hits non-SMM memslots (for RSM), and (b) skip the check (since its off-by-default anyways) when the vCPU doesn't want to run, i.e. when userspace is restoring/stuffing state.

If reading guest memory fails, simply skip the consistency check, as KVM's de facto ABI is that VMX instruction accesses to non-existent memory get PCI Bus Error semantics, where reads return 0xFFs. And if vTPR=0xFF, then the vTPR is guaranteed to be greater than or equal to TPR_THRESHOLD.

Fixes 1100e4910ad2 ("KVM: nVMX: Add an off-by-default module param to WARN on missed consistency checks")
---
 arch/x86/kvm/vmx/nested.c | 66 +++++++++++++++++----------------------
 1 file changed, 29 insertions(+), 37 deletions(-)
arch/x86/kvm/vmx/nested.c +29 -37
index b2c851cc7d5c..199b866072c0 100644
@@ -582,6 +582,9 @@ static int nested_vmx_check_msr_bitmap_controls(struct kvm_vcpu *vcpu,
 static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu,
 						struct vmcs12 *vmcs12)
 {
+	gpa_t vtpr_gpa = vmcs12->virtual_apic_page_addr + APIC_TASKPRI;
+	u32 vtpr;
+
 	if (!nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW))
 		return 0;
 
@@ -591,6 +594,32 @@ static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu,
 	if (CC(!nested_cpu_has_vid(vmcs12) && vmcs12->tpr_threshold >> 4))
 		return -EINVAL;
 
+	/*
+	 * Do the illegal vTPR vs. TPR Threshold consistency check if and only
+	 * if KVM is configured to WARN on missed consistency checks, otherwise
+	 * it's a waste of time.  KVM needs to rely on hardware to fully detect
+	 * an illegal combination due to the vTPR being writable by L1 at all
+	 * times (it's an in-memory value, not a VMCS field).  I.e. even if the
+	 * check passes now, it might fail at the actual VM-Enter.
+	 *
+	 * If reading guest memory fails, skip the check as KVM's de facto ABI
+	 * for VMX instruction accesses to non-existent memory is to provide
+	 * PCI Bus Error semantics (reads return 0xFFs), in which case the vTPR
+	 * is guaranteed to greater than or equal to the threshold.
+	 *
+	 * Note!  Deliberately use the VM-scoped API when reading guest memory,
+	 * to ensure the read doesn't hit SMRAM when restoring L2 state on RSM,
+	 * and only perform the check when in KVM_RUN, to avoid a false failure
+	 * if userspace hasn't yet configured memslots during state restore.
+	 */
+	if (warn_on_missed_cc && vcpu->wants_to_run &&
+	    nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) &&
+	    !nested_cpu_has_vid(vmcs12) &&
+	    !nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES) &&
+	    !kvm_read_guest(vcpu->kvm, vtpr_gpa, &vtpr, sizeof(vtpr)) &&
+	    CC((vmcs12->tpr_threshold & GENMASK(3, 0)) > ((vtpr >> 4) & GENMASK(3, 0))))
+		return -EINVAL;
+
 	return 0;
 }
 
@@ -3115,38 +3144,6 @@ static int nested_vmx_check_controls(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
-static int nested_vmx_check_controls_late(struct kvm_vcpu *vcpu,
-					  struct vmcs12 *vmcs12)
-{
-	void *vapic = to_vmx(vcpu)->nested.virtual_apic_map.hva;
-	u32 vtpr = vapic ? (*(u32 *)(vapic + APIC_TASKPRI)) >> 4 : 0;
-
-	/*
-	 * Don't bother with the consistency checks if KVM isn't configured to
-	 * WARN on missed consistency checks, as KVM needs to rely on hardware
-	 * to fully detect an illegal vTPR vs. TRP Threshold combination due to
-	 * the vTPR being writable by L1 at all times (it's an in-memory value,
-	 * not a VMCS field).  I.e. even if the check passes now, it might fail
-	 * at the actual VM-Enter.
-	 *
-	 * Keying off the module param also allows treating an invalid vAPIC
-	 * mapping as a consistency check failure without increasing the risk
-	 * of breaking a "real" VM.
-	 */
-	if (!warn_on_missed_cc)
-		return 0;
-
-	if ((exec_controls_get(to_vmx(vcpu)) & CPU_BASED_TPR_SHADOW) &&
-	    nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) &&
-	    !nested_cpu_has_vid(vmcs12) &&
-	    !nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES) &&
-	    (CC(!vapic) ||
-	     CC((vmcs12->tpr_threshold & GENMASK(3, 0)) > (vtpr & GENMASK(3, 0)))))
-		return -EINVAL;
-
-	return 0;
-}
-
 static int nested_vmx_check_address_space_size(struct kvm_vcpu *vcpu,
 				       struct vmcs12 *vmcs12)
 {
@@ -3696,11 +3693,6 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
 			return NVMX_VMENTRY_KVM_INTERNAL_ERROR;
 		}
 
-		if (nested_vmx_check_controls_late(vcpu, vmcs12)) {
-			vmx_switch_vmcs(vcpu, &vmx->vmcs01);
-			return NVMX_VMENTRY_VMFAIL;
-		}
-
 		if (nested_vmx_check_guest_state(vcpu, vmcs12,
 						 &entry_failure_code)) {
 			exit_reason.basic = EXIT_REASON_INVALID_STATE;
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
author maintainer
#2 lore raw

PATCH v2 2/2 KVM: nVMX: Don't use vmcs01.GUEST_CR3 to snapshot L1's CR3 when EPT is disabled

In reply to Sean Christopherson

Add a dedicated field in "struct nested_vmx" to track L1's pre-VM-Enter CR3 instead of using vmcs01.GUEST_CR3, which isn't anywhere near as safe as the comment purports it to be. E.g. in addition to the warn_on_missed_cc bug (that was fixed by relocating the consistency check), if getting vmcs12 pages (during actual nested VM-Entry) fails and EPT is disabled (in KVM), KVM will return control to userspace with vmcs01.GUEST_CR3 holding a guest- controlled value.

Alternatively, KVM could force a reload of vmcs01.GUEST_CR3 by resetting the MMU context in the error path, but as above, the safety of the vmcs01 approach is extremely questionable, e.g. it took all of ~4 months for the code to break.

Fixes 671ddc700fd0 ("KVM: nVMX: Don't leak L1 MMIO regions to L2")
Cc Jim Mattson <[email protected]>
---
 arch/x86/kvm/vmx/nested.c | 21 ++++++++-------------
 arch/x86/kvm/vmx/vmx.h    |  7 +++++++
 2 files changed, 15 insertions(+), 13 deletions(-)
arch/x86/kvm/vmx/nested.c +8 -13
index 199b866072c0..7a2251061bfa 100644
@@ -3669,19 +3669,14 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
 				    &vmx->nested.pre_vmenter_ssp_tbl);
 
 	/*
-	 * Overwrite vmcs01.GUEST_CR3 with L1's CR3 if EPT is disabled.  In the
-	 * event of a "late" VM-Fail, i.e. a VM-Fail detected by hardware but
-	 * not KVM, KVM must unwind its software model to the pre-VM-Entry host
-	 * state.  When EPT is disabled, GUEST_CR3 holds KVM's shadow CR3, not
-	 * L1's "real" CR3, which causes nested_vmx_restore_host_state() to
-	 * corrupt vcpu->arch.cr3.  Stuffing vmcs01.GUEST_CR3 results in the
-	 * unwind naturally setting arch.cr3 to the correct value.  Smashing
-	 * vmcs01.GUEST_CR3 is safe because nested VM-Exits, and the unwind,
-	 * reset KVM's MMU, i.e. vmcs01.GUEST_CR3 is guaranteed to be
-	 * overwritten with a shadow CR3 prior to re-entering L1.
+	 * Stash L1's CR3, so that in the event of a "late" VM-Fail, i.e. a
+	 * VM-Fail detected by hardware but not KVM, KVM can unwind its
+	 * software model to the pre-VM-Entry host state.  When EPT is
+	 * disabled, GUEST_CR3 holds KVM's shadow CR3, not L1's "real" CR3,
+	 * and so simply restoring from vmcs01.GUEST_CR3 would corrupt
+	 * vcpu->arch.cr3.
 	 */
-	if (!enable_ept)
-		vmcs_writel(GUEST_CR3, vcpu->arch.cr3);
+	vmx->nested.pre_vmenter_cr3 = kvm_read_cr3(vcpu);
 
 	vmx_switch_vmcs(vcpu, &vmx->nested.vmcs02);
 
@@ -4993,7 +4988,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)
 	vmx_set_cr4(vcpu, vmcs_readl(CR4_READ_SHADOW));
 
 	nested_ept_uninit_mmu_context(vcpu);
-	vcpu->arch.cr3 = vmcs_readl(GUEST_CR3);
+	vcpu->arch.cr3 = vmx->nested.pre_vmenter_cr3;
 	kvm_register_mark_available(vcpu, VCPU_REG_CR3);
 
 	/*
arch/x86/kvm/vmx/vmx.h +7 -0
index de9de0d2016c..dc8517f15bc4 100644
@@ -159,6 +159,13 @@ struct nested_vmx {
 	bool has_preemption_timer_deadline;
 	bool preemption_timer_expired;
 
+	/*
+	 * Used to restore L1's CR3 if hardware detects a VM-Fail Consistency
+	 * Check that KVM does not, in which case KVM needs to unwind CR3 back
+	 * to its pre-VM-Enter state, NOT to vmcs01.HOST_CR3.
+	 */
+	unsigned long pre_vmenter_cr3;
+
 	/*
 	 * Used to snapshot MSRs that are conditionally loaded on VM-Enter in
 	 * order to propagate the guest's pre-VM-Enter value into vmcs02.  For
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
#3 lore raw

Re: [PATCH v2 2/2] KVM: nVMX: Don't use vmcs01.GUEST_CR3 to snapshot L1's CR3 when EPT is disabled

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The pre_vmenter_cr3 field is not serialized during live migration, causing L1's CR3 to be permanently lost and corrupted if a late VM-Fail occurs on the destination. --

commit 2c3bbd56634c27ee75bdd6af8c48278025e7dcb4
Author: Sean Christopherson <[email protected]>

KVM: nVMX: Don't use vmcs01.GUEST_CR3 to snapshot L1's CR3 when EPT is disabled

Add a dedicated field in "struct nested_vmx" to track L1's pre-VM-Enter CR3 instead of using vmcs01.GUEST_CR3.

Quoted 4 lines
arch/x86/kvm/vmx/nested.c +0 -0
index 199b866072c0f..7a2251061bfa7 100644

[ ... ]

Quoted 14 lines
arch/x86/kvm/vmx/nested.c +0 -10
@@ -3669,19 +3669,14 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
 				    &vmx->nested.pre_vmenter_ssp_tbl);
 
 	/*
-	 * Overwrite vmcs01.GUEST_CR3 with L1's CR3 if EPT is disabled.  In the
-	 * event of a "late" VM-Fail, i.e. a VM-Fail detected by hardware but
-	 * not KVM, KVM must unwind its software model to the pre-VM-Entry host
-	 * state.  When EPT is disabled, GUEST_CR3 holds KVM's shadow CR3, not
-	 * L1's "real" CR3, which causes nested_vmx_restore_host_state() to
-	 * corrupt vcpu->arch.cr3.  Stuffing vmcs01.GUEST_CR3 results in the
-	 * unwind naturally setting arch.cr3 to the correct value.  Smashing
-	 * vmcs01.GUEST_CR3 is safe because nested VM-Exits, and the unwind,
-	 * reset KVM's MMU, i.e. vmcs01.GUEST_CR3 is guaranteed to be
-	 * overwritten with a shadow CR3 prior to re-entering L1.

arch/x86/kvm/vmx/nested.c enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, · line 3679

Quoted 10 lines
arch/x86/kvm/vmx/nested.c +7 -2
+	 * Stash L1's CR3, so that in the event of a "late" VM-Fail, i.e. a
+	 * VM-Fail detected by hardware but not KVM, KVM can unwind its
+	 * software model to the pre-VM-Entry host state.  When EPT is
+	 * disabled, GUEST_CR3 holds KVM's shadow CR3, not L1's "real" CR3,
+	 * and so simply restoring from vmcs01.GUEST_CR3 would corrupt
+	 * vcpu->arch.cr3.
 	 */
-	if (!enable_ept)
-		vmcs_writel(GUEST_CR3, vcpu->arch.cr3);
+	vmx->nested.pre_vmenter_cr3 = kvm_read_cr3(vcpu);

[Severity: High]
This is a pre-existing issue, but could this corrupt L1's CR3 during live migration if a late VM-Fail occurs on the destination?

When migrating a nested VM that is currently in L2 (or has a pending nested run), KVM_SET_NESTED_STATE calls nested_vmx_enter_non_root_mode() with from_vmentry=false.

At this point, kvm_read_cr3() either holds L2's CR3 (if KVM_SET_SREGS was already called by userspace) or the default initial CR3, but not L1's CR3.

[ ... ]

arch/x86/kvm/vmx/nested.c static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu) · line 4992

Quoted 7 lines
arch/x86/kvm/vmx/nested.c +1 -1
@@ -4993,7 +4988,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)
 	vmx_set_cr4(vcpu, vmcs_readl(CR4_READ_SHADOW));
 
 	nested_ept_uninit_mmu_context(vcpu);
-	vcpu->arch.cr3 = vmcs_readl(GUEST_CR3);
+	vcpu->arch.cr3 = vmx->nested.pre_vmenter_cr3;
 	kvm_register_mark_available(vcpu, VCPU_REG_CR3);

If a late hardware VM-Fail then occurs when KVM attempts to enter the guest on the destination, nested_vmx_restore_host_state() will unwind L1's state by setting vcpu->arch.cr3 to the value stashed above.

Since pre_vmenter_cr3 is not serialized as part of the nested state, doesn't this clobber L1's CR3 with L2's CR3 or zero, potentially leading to memory corruption or a kernel panic in L1?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
maintainer
#4 lore raw

Re: [PATCH v2 2/2] KVM: nVMX: Don't use vmcs01.GUEST_CR3 to snapshot L1's CR3 when EPT is disabled

Quoted 4 lines · On 6/12/26 17:16, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The pre_vmenter_cr3 field is not serialized during live migration, causing L1's CR3 to be permanently lost and corrupted if a late VM-Fail occurs on the destination.

Since this patch is basically the same issue that you pointed out for the PDPTRs, and both need the same change to get/set_nested_state, if you don't mind I'll pick these two, fold them together with the PDPTR patches, and send them tomorrow.

Paolo

author maintainer Applied
#5 lore raw
In reply to Sean Christopherson
Quoted 9 lines · Sean Christopherson

Fix two bugs where KVM can run L2 with a guest-controlled CR3. The underlying flaw dates back to commit f087a02941fe ("KVM: nVMX: Stash L1's CR3 in vmcs01.GUEST_CR3 on nested entry w/o EPT"). Past me claimed:

Smashing vmcs01.GUEST_CR3 is safe because nested VM-Exits, and the unwind, reset KVM's MMU, i.e. vmcs01.GUEST_CR3 is guaranteed to be overwritten with a shadow CR3 prior to re-entering L1.

[...]

Applied to kvm-x86 fixes, thanks!

[1/2] KVM: nVMX: Move vTPR vs. TPR Threshold consistency check into "normal" checks

      https://github.com/kvm-x86/linux/commit/ebdac7554abb
[2/2] KVM: nVMX: Don't use vmcs01.GUEST_CR3 to snapshot L1's CR3 when EPT is disabled
      https://github.com/kvm-x86/linux/commit/3e6dd2b9b7b2

--
https://github.com/kvm-x86/linux/tree/next