Skip to contentSkip to conversation

KVM: x86: Fastpath userspace exit fix and hardening

PATCH Applied Sean Christopherson 3 participants 7 messages Apr 23 – May 19

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

cover KVM: x86: Fastpath userspace exit fix and hardening

Fix a found-by-inspection bug where KVM could fail to flush the PML buffer (VMX) or fail to grab the most recet CR0/CR3 (SVM) prior to exiting to userspace, which could prove to be fatal if that exit is that last exit prior to save/restore, e.g. the last exit before userspace commits to migrating the VM.

Patch 2 is related hardening, i.e. isn't fixing any existing bugs (AFAIK).

Sean Christopherson (2):
KVM: x86: Ensure vendor's exit handler runs before fastpath userspace
exits
KVM: SVM: Refresh vcpu->arch.cr{0,3} prior to invoking fastpath
handler

 arch/x86/kvm/svm/svm.c | 15 ++++++++-------
 arch/x86/kvm/vmx/vmx.c |  3 +++
 arch/x86/kvm/x86.c     |  3 ---
 3 files changed, 11 insertions(+), 10 deletions(-)

base-commit: 85f871f6ba46f20d7fbc0b016b4db648c33220dd

Signature · 1 line
2.54.0.545.g6539524ca2-goog
author maintainer
#1 lore raw

PATCH 1/2 KVM: x86: Ensure vendor's exit handler runs before fastpath userspace exits

Move the handling of fastpath userspace exits into vendor code to ensure KVM runs vendor specific operations that need to run before userspace gains control of the vCPU. E.g. for VMX (and soon to be for SVM as well), KVM needs to flush the PML buffer prior to exiting to userspace, otherwise any memory written by the final KVM_RUN might never be flagged as dirty.

Note, waiting to snapshot CR0 and CR3 until svm_handle_exit() is flawed in general, as that risks consuming stale state in a fastpath handler. That will be addressed in a future change.

Fixes f7f39c50edb9 ("KVM: x86: Exit to userspace if fastpath triggers one on instruction skip")
Cc Nikunj A. Dadhania <[email protected]>
---
 arch/x86/kvm/svm/svm.c | 3 +++
 arch/x86/kvm/vmx/vmx.c | 3 +++
 arch/x86/kvm/x86.c     | 3 ---
 3 files changed, 6 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/svm.c +3 -0
index e7fdd7a9c280..eb351ca4dd82 100644
@@ -3652,6 +3652,9 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 			vcpu->arch.cr3 = svm->vmcb->save.cr3;
 	}
 
+	if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE))
+		return 0;
+
 	if (is_guest_mode(vcpu)) {
 		int vmexit;
 
arch/x86/kvm/vmx/vmx.c +3 -0
index a29896a9ef14..4cb355ecfe46 100644
@@ -6687,6 +6687,9 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 	if (enable_pml && !is_guest_mode(vcpu))
 		vmx_flush_pml_buffer(vcpu);
 
+	if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE))
+		return 0;
+
 	/*
 	 * KVM should never reach this point with a pending nested VM-Enter.
 	 * More specifically, short-circuiting VM-Entry to emulate L2 due to
arch/x86/kvm/x86.c +0 -3
index 0a1b63c63d1a..9ad7ec3bf0f1 100644
@@ -11602,9 +11602,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	if (vcpu->arch.apic_attention)
 		kvm_lapic_sync_from_vapic(vcpu);
 
-	if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE))
-		return 0;
-
 	r = kvm_x86_call(handle_exit)(vcpu, exit_fastpath);
 	return r;
 
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] KVM: x86: Ensure vendor's exit handler runs before fastpath userspace exits

Quoted 6 lines · Sean Christopherson

Move the handling of fastpath userspace exits into vendor code to ensure KVM runs vendor specific operations that need to run before userspace gains control of the vCPU. E.g. for VMX (and soon to be for SVM as well), KVM needs to flush the PML buffer prior to exiting to userspace, otherwise any memory written by the final KVM_RUN might never be flagged as dirty.

Quoted 9 lines

Note, waiting to snapshot CR0 and CR3 until svm_handle_exit() is flawed in general, as that risks consuming stale state in a fastpath handler. That will be addressed in a future change.

Fixes f7f39c50edb9 ("KVM: x86: Exit to userspace if fastpath triggers one on instruction skip")
Cc Nikunj A. Dadhania <[email protected]>
Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] KVM: x86: Ensure vendor's exit handler runs before fastpath userspace exits

In reply to Sean Christopherson
Quoted 6 lines · Sean Christopherson

Move the handling of fastpath userspace exits into vendor code to ensure KVM runs vendor specific operations that need to run before userspace gains control of the vCPU. E.g. for VMX (and soon to be for SVM as well), KVM needs to flush the PML buffer prior to exiting to userspace, otherwise any memory written by the final KVM_RUN might never be flagged as dirty.

Quoted 8 lines

Note, waiting to snapshot CR0 and CR3 until svm_handle_exit() is flawed in general, as that risks consuming stale state in a fastpath handler. That will be addressed in a future change.

Fixes f7f39c50edb9 ("KVM: x86: Exit to userspace if fastpath triggers one on instruction skip")
Cc Nikunj A. Dadhania <[email protected]>
author maintainer
#4 lore raw

PATCH 2/2 KVM: SVM: Refresh vcpu->arch.cr{0,3} prior to invoking fastpath handler

In reply to Sean Christopherson

Refresh KVM's copies of CR0 and CR3 from the VMCB prior to (potentially) invoking a fastpath handler to ensure that KVM doesn't consume stale state. While it's unlikely KVM will ever consume CR3 or CR0.{TS,MP} in the fastpath, grabbing the values from the VMCB is inexpensive, i.e. the risk of subtle bugs far outweighs the reward of deferring reads for a small subset of VM-Exits.

Note, KVM doesn't currently consume CR3 or CR0.{TS,MP} in the fastpath, as KVM requires next_rip to be valid (i.e. KVM doesn't read CR3 to decode the instruction), CR0.MP is never consumed, and CR0.TS is only consumed by the full emulator.

---
 arch/x86/kvm/svm/svm.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
arch/x86/kvm/svm/svm.c +6 -8
index eb351ca4dd82..df0bd132edf7 100644
@@ -3644,14 +3644,6 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_run *kvm_run = vcpu->run;
 
-	/* SEV-ES guests must use the CR write traps to track CR registers. */
-	if (!is_sev_es_guest(vcpu)) {
-		if (!svm_is_intercept(svm, INTERCEPT_CR0_WRITE))
-			vcpu->arch.cr0 = svm->vmcb->save.cr0;
-		if (npt_enabled)
-			vcpu->arch.cr3 = svm->vmcb->save.cr3;
-	}
-
 	if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE))
 		return 0;
 
@@ -4505,11 +4497,17 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 	if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
 		x86_spec_ctrl_restore_host(svm->virt_spec_ctrl);
 
+	/* SEV-ES guests must use the CR write traps to track CR registers. */
 	if (!is_sev_es_guest(vcpu)) {
 		vcpu->arch.cr2 = svm->vmcb->save.cr2;
 		vcpu->arch.regs[VCPU_REGS_RAX] = svm->vmcb->save.rax;
 		vcpu->arch.regs[VCPU_REGS_RSP] = svm->vmcb->save.rsp;
 		vcpu->arch.regs[VCPU_REGS_RIP] = svm->vmcb->save.rip;
+
+		if (!svm_is_intercept(svm, INTERCEPT_CR0_WRITE))
+			vcpu->arch.cr0 = svm->vmcb->save.cr0;
+		if (npt_enabled)
+			vcpu->arch.cr3 = svm->vmcb->save.cr3;
 	}
 	vcpu->arch.regs_dirty = 0;
 
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] KVM: SVM: Refresh vcpu->arch.cr{0,3} prior to invoking fastpath handler

Quoted 6 lines · Sean Christopherson

Refresh KVM's copies of CR0 and CR3 from the VMCB prior to (potentially) invoking a fastpath handler to ensure that KVM doesn't consume stale state. While it's unlikely KVM will ever consume CR3 or CR0.{TS,MP} in the fastpath, grabbing the values from the VMCB is inexpensive, i.e. the risk of subtle bugs far outweighs the reward of deferring reads for a small subset of VM-Exits.

This also keeps CR0/CR3 current during the fast-reenter loop in vcpu_enter_guest(), which previously skipped svm_handle_exit() entirely.

Quoted 7 lines

Note, KVM doesn't currently consume CR3 or CR0.{TS,MP} in the fastpath, as KVM requires next_rip to be valid (i.e. KVM doesn't read CR3 to decode the instruction), CR0.MP is never consumed, and CR0.TS is only consumed by the full emulator.

author maintainer Applied
#6 lore raw
In reply to Sean Christopherson
Quoted 9 lines · Sean Christopherson

Fix a found-by-inspection bug where KVM could fail to flush the PML buffer (VMX) or fail to grab the most recet CR0/CR3 (SVM) prior to exiting to userspace, which could prove to be fatal if that exit is that last exit prior to save/restore, e.g. the last exit before userspace commits to migrating the VM.

Patch 2 is related hardening, i.e. isn't fixing any existing bugs (AFAIK).

[...]

Applied to kvm-x86 misc, thanks!

[1/2] KVM: x86: Ensure vendor's exit handler runs before fastpath userspace exits

      https://github.com/kvm-x86/linux/commit/0ffedf43910e
[2/2] KVM: SVM: Refresh vcpu->arch.cr{0,3} prior to invoking fastpath handler
      https://github.com/kvm-x86/linux/commit/b21525756e82

--
https://github.com/kvm-x86/linux/tree/next