Skip to contentSkip to conversation

KVM: x86: Cleanup #MC and XCR0/XSS/PKRU handling

PATCH Applied Sean Christopherson 5 participants 19 messages Oct 30, 2025 – Nov 18, 2025

Subsystems (4)

v1 of 2 Applied Latest is v2 →

lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

cover KVM: x86: Cleanup #MC and XCR0/XSS/PKRU handling

This series is the result of the recent PUCK discussion[*] on optimizing the XCR0/XSS loads that are currently done on every VM-Enter and VM-Exit. My initial thought that swapping XCR0/XSS outside of the fastpath was spot on; turns out the only reason they're swapped in the fastpath is because of a hack-a-fix that papered over an egregious #MC handling bug where the kernel #MC handler would call schedule() from an atomic context. The resulting #GP due to trying to swap FPU state with a guest XCR0/XSS was "fixed" by loading the host values before handling #MCs from the guest.

Thankfully, the #MC mess has long since been cleaned up, so it's once again safe to swap XCR0/XSS outside of the fastpath (but when IRQs are disabled!).

As for what may be contributing to the SAP HANA performance improvements when enabling PKU, my instincts again appear to be spot on. As predicted, the fastpath savings are ~300 cycles on Intel (~500 on AMD). I.e. if the guest is literally doing _nothing_ but generating fastpath exits, it will see a ~%25 improvement. There's basically zero chance the uplift seen with enabling PKU is dues to eliding XCR0 loads; my guess is that the guest actualy uses protection keys to optimize something.

Why does kvm_load_guest_xsave_state() show up in perf? Probably because it's the only visible symbol other than vmx_vmexit() (and vmx_vcpu_run() when not hammering the fastpath). E.g. running perf top on a running VM instance yields these numbers with various guest workloads (the middle one is running mmu_stress_test in the guest, which hammers on mmu_lock in L0). But other than doing INVD (handled in the fastpath) in a tight loop, there's no perceived perf improvement from the guest.

Overhead  Shared Object       Symbol
  15.65%  [kernel]            [k] vmx_vmexit
   6.78%  [kernel]            [k] kvm_vcpu_halt
   5.15%  [kernel]            [k] __srcu_read_lock
   4.73%  [kernel]            [k] kvm_load_guest_xsave_state
   4.69%  [kernel]            [k] __srcu_read_unlock
   4.65%  [kernel]            [k] read_tsc
   4.44%  [kernel]            [k] vmx_sync_pir_to_irr
   4.03%  [kernel]            [k] kvm_apic_has_interrupt
  45.52%  [kernel]            [k] queued_spin_lock_slowpath
  24.40%  [kernel]            [k] vmx_vmexit
   2.84%  [kernel]            [k] queued_write_lock_slowpath
   1.92%  [kernel]            [k] vmx_vcpu_run
   1.40%  [kernel]            [k] vcpu_run
   1.00%  [kernel]            [k] kvm_load_guest_xsave_state
   0.84%  [kernel]            [k] kvm_load_host_xsave_state
   0.72%  [kernel]            [k] mmu_try_to_unsync_pages
   0.68%  [kernel]            [k] __srcu_read_lock
   0.65%  [kernel]            [k] try_get_folio
  17.78%  [kernel]            [k] vmx_vmexit
   5.08%  [kernel]            [k] vmx_vcpu_run
   4.24%  [kernel]            [k] vcpu_run
   4.21%  [kernel]            [k] _raw_spin_lock_irqsave
   2.99%  [kernel]            [k] kvm_load_guest_xsave_state
   2.51%  [kernel]            [k] rcu_note_context_switch
   2.47%  [kernel]            [k] ktime_get_update_offsets_now
   2.21%  [kernel]            [k] kvm_load_host_xsave_state
   2.16%  [kernel]            [k] fput

[*] https://drive.google.com/corp/drive/folders/1DCdvqFGudQc7pxXjM7f35vXogTf9uhD4

Sean Christopherson (4):
KVM: SVM: Handle #MCs in guest outside of fastpath
KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath
KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run
loop
KVM: x86: Load guest/host PKRU outside of the fastpath run loop

 arch/x86/kvm/svm/svm.c  | 20 ++++++++--------
 arch/x86/kvm/vmx/main.c | 13 ++++++++++-
 arch/x86/kvm/vmx/tdx.c  |  3 ---
 arch/x86/kvm/vmx/vmx.c  |  7 ------
 arch/x86/kvm/x86.c      | 51 ++++++++++++++++++++++++++++-------------
 arch/x86/kvm/x86.h      |  2 --
 6 files changed, 56 insertions(+), 40 deletions(-)

base-commit: 4cc167c50eb19d44ac7e204938724e685e3d8057

Signature · 1 line
2.51.1.930.gacf6e81ea2-goog
author maintainer
#1 lore raw

PATCH 1/4 KVM: SVM: Handle #MCs in guest outside of fastpath

Handle Machine Checks (#MC) that happen in the guest (by forwarding them to the host) outside of KVM's fastpath so that as much host state as possible is re-loaded before invoking the kernel's #MC handler. The only requirement is that KVM invokes the #MC handler before enabling IRQs (and even that could _probably_ be relaxed to handling #MCs before enabling preemption).

Waiting to handle #MCs until "more" host state is loaded hardens KVM against flaws in the #MC handler, which has historically been quite brittle. E.g. prior to commit 5567d11c21a1 ("x86/mce: Send #MC singal from task work"), the #MC code could trigger a schedule() with IRQs and preemption disabled. That led to a KVM hack-a-fix in commit 1811d979c716 ("x86/kvm: move kvm_load/put_guest_xcr0 into atomic context").

Note, except for #MCs on VM-Enter, VMX already handles #MCs outside of the fastpath.

---
 arch/x86/kvm/svm/svm.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/svm.c +9 -9
index f14709a511aa..e8b158f73c79 100644
@@ -4335,14 +4335,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 
 	vcpu->arch.regs_avail &= ~SVM_REGS_LAZY_LOAD_SET;
 
-	/*
-	 * We need to handle MC intercepts here before the vcpu has a chance to
-	 * change the physical cpu
-	 */
-	if (unlikely(svm->vmcb->control.exit_code ==
-		     SVM_EXIT_EXCP_BASE + MC_VECTOR))
-		svm_handle_mce(vcpu);
-
 	trace_kvm_exit(vcpu, KVM_ISA_SVM);
 
 	svm_complete_interrupts(vcpu);
@@ -4631,8 +4623,16 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
 
 static void svm_handle_exit_irqoff(struct kvm_vcpu *vcpu)
 {
-	if (to_svm(vcpu)->vmcb->control.exit_code == SVM_EXIT_INTR)
+	switch (to_svm(vcpu)->vmcb->control.exit_code) {
+	case SVM_EXIT_EXCP_BASE + MC_VECTOR:
+		svm_handle_mce(vcpu);
+		break;
+	case SVM_EXIT_INTR:
 		vcpu->arch.at_instruction_boundary = true;
+		break;
+	default:
+		break;
+	}
 }
 
 static void svm_setup_mce(struct kvm_vcpu *vcpu)
Signature · 1 line
2.51.1.930.gacf6e81ea2-goog
author maintainer
#2 lore raw

PATCH 2/4 KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath

In reply to Sean Christopherson

Handle Machine Checks (#MC) that happen on VM-Enter (VMX or TDX) outside of KVM's fastpath so that as much host state as possible is re-loaded before invoking the kernel's #MC handler. The only requirement is that KVM invokes the #MC handler before enabling IRQs (and even that could _probably_ be related to handling #MCs before enabling preemption).

Waiting to handle #MCs until "more" host state is loaded hardens KVM against flaws in the #MC handler, which has historically been quite brittle. E.g. prior to commit 5567d11c21a1 ("x86/mce: Send #MC singal from task work"), the #MC code could trigger a schedule() with IRQs and preemption disabled. That led to a KVM hack-a-fix in commit 1811d979c716 ("x86/kvm: move kvm_load/put_guest_xcr0 into atomic context").

---
 arch/x86/kvm/vmx/main.c | 13 ++++++++++++-
 arch/x86/kvm/vmx/tdx.c  |  3 ---
 arch/x86/kvm/vmx/vmx.c  |  3 ---
 3 files changed, 12 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/main.c +12 -1
index 0eb2773b2ae2..1beaec5b9727 100644
@@ -608,6 +608,17 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 	vmx_load_mmu_pgd(vcpu, root_hpa, pgd_level);
 }
 
+static void vt_handle_exit_irqoff(struct kvm_vcpu *vcpu)
+{
+	if (unlikely((u16)vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
+		kvm_machine_check();
+
+	if (is_td_vcpu(vcpu))
+		return;
+
+	return vmx_handle_exit_irqoff(vcpu);
+}
+
 static void vt_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask)
 {
 	if (is_td_vcpu(vcpu))
@@ -969,7 +980,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.load_mmu_pgd = vt_op(load_mmu_pgd),
 
 	.check_intercept = vmx_check_intercept,
-	.handle_exit_irqoff = vmx_handle_exit_irqoff,
+	.handle_exit_irqoff = vt_op(handle_exit_irqoff),
 
 	.update_cpu_dirty_logging = vt_op(update_cpu_dirty_logging),
 
arch/x86/kvm/vmx/tdx.c +0 -3
index 326db9b9c567..a2f6ba3268d1 100644
@@ -1069,9 +1069,6 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 	if (unlikely((tdx->vp_enter_ret & TDX_SW_ERROR) == TDX_SW_ERROR))
 		return EXIT_FASTPATH_NONE;
 
-	if (unlikely(vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
-		kvm_machine_check();
-
 	trace_kvm_exit(vcpu, KVM_ISA_VMX);
 
 	if (unlikely(tdx_failed_vmentry(vcpu)))
arch/x86/kvm/vmx/vmx.c +0 -3
index 1021d3b65ea0..123dae8cf46b 100644
@@ -7527,9 +7527,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 	if (unlikely(vmx->fail))
 		return EXIT_FASTPATH_NONE;
 
-	if (unlikely((u16)vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
-		kvm_machine_check();
-
 	trace_kvm_exit(vcpu, KVM_ISA_VMX);
 
 	if (unlikely(vmx_get_exit_reason(vcpu).failed_vmentry))
Signature · 1 line
2.51.1.930.gacf6e81ea2-goog
#3 lore raw

Re: [PATCH 2/4] KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath

Hi,

Quoted 6 lines · Sean Christopherson
arch/x86/kvm/vmx/main.c +0 -0
@@ -608,6 +608,17 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 	vmx_load_mmu_pgd(vcpu, root_hpa, pgd_level);
 }
 

arch/x86/kvm/vmx/main.c static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · line 620

Quoted 10 lines
arch/x86/kvm/vmx/main.c +10 -0
+static void vt_handle_exit_irqoff(struct kvm_vcpu *vcpu)
+{
+	if (unlikely((u16)vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
+		kvm_machine_check();
+
+	if (is_td_vcpu(vcpu))
+		return;
+
+	return vmx_handle_exit_irqoff(vcpu);
+}

I bisected kvm-x86/next down to this change for a TDX guest not booting and host producing errors like:

watchdog: CPU118: Watchdog detected hard LOCKUP on cpu 118

Dropping the is_td_vcpu(vcpu) check above fixes the issue. Earlier the call for vmx_handle_exit_irqoff() was unconditional.

Probably the (u16) cast above can be dropped too? It was never used for TDX looking at the patch.

Regards,

Tony

author maintainer
#4 lore raw

Re: [PATCH 2/4] KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath

Quoted 23 lines · Tony Lindgren

Hi,

Quoted 6 lines · Sean Christopherson
arch/x86/kvm/vmx/main.c +0 -0
@@ -608,6 +608,17 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 	vmx_load_mmu_pgd(vcpu, root_hpa, pgd_level);
 }
 

arch/x86/kvm/vmx/main.c static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · line 620

Quoted 10 lines
arch/x86/kvm/vmx/main.c +10 -0
+static void vt_handle_exit_irqoff(struct kvm_vcpu *vcpu)
+{
+	if (unlikely((u16)vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
+		kvm_machine_check();
+
+	if (is_td_vcpu(vcpu))
+		return;
+
+	return vmx_handle_exit_irqoff(vcpu);
+}

I bisected kvm-x86/next down to this change for a TDX guest not booting and host producing errors like:

Quoted 4 lines

watchdog: CPU118: Watchdog detected hard LOCKUP on cpu 118

Dropping the is_td_vcpu(vcpu) check above fixes the issue. Earlier the call for vmx_handle_exit_irqoff() was unconditional.

Ugh, once you see it, it's obvious. Sorry :-(

I'll drop the entire series and send a v2. There's only one other patch that I already sent the "thank you" for, so I think it's worth unwinding to avoid breaking bisection for TDX (and because the diff can be very different).

Lightly tested, but I think this patch can instead be:

arch/x86/kvm/vmx/tdx.c +0 -3
index 163f854a39f2..6d41d2fc8043 100644
@@ -1063,9 +1063,6 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
        if (unlikely((tdx->vp_enter_ret & TDX_SW_ERROR) == TDX_SW_ERROR))
                return EXIT_FASTPATH_NONE;
 
-       if (unlikely(vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
-               kvm_machine_check();
-
        trace_kvm_exit(vcpu, KVM_ISA_VMX);
 
        if (unlikely(tdx_failed_vmentry(vcpu)))
arch/x86/kvm/vmx/vmx.c +11 -5
index d98107a7bdaa..d1117da5463f 100644
@@ -7035,10 +7035,19 @@ void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu)
        if (to_vt(vcpu)->emulation_required)
                return;
 
-       if (vmx_get_exit_reason(vcpu).basic == EXIT_REASON_EXTERNAL_INTERRUPT)
+       switch (vmx_get_exit_reason(vcpu).basic) {
+       case EXIT_REASON_EXTERNAL_INTERRUPT:
                handle_external_interrupt_irqoff(vcpu, vmx_get_intr_info(vcpu));
-       else if (vmx_get_exit_reason(vcpu).basic == EXIT_REASON_EXCEPTION_NMI)
+               break;
+       case EXIT_REASON_EXCEPTION_NMI:
                handle_exception_irqoff(vcpu, vmx_get_intr_info(vcpu));
+               break;
+       case EXIT_REASON_MCE_DURING_VMENTRY:
+               kvm_machine_check();
+               break;
+       default:
+               break;
+       }
 }
 
 /*
@@ -7501,9 +7510,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
        if (unlikely(vmx->fail))
                return EXIT_FASTPATH_NONE;
 
-       if (unlikely((u16)vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
-               kvm_machine_check();
-
        trace_kvm_exit(vcpu, KVM_ISA_VMX);
 
        if (unlikely(vmx_get_exit_reason(vcpu).failed_vmentry))
#5 lore raw

Re: [PATCH 2/4] KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath

Quoted 26 lines · Sean Christopherson
Quoted 25 lines · Tony Lindgren

Hi,

Quoted 6 lines · Sean Christopherson
arch/x86/kvm/vmx/main.c +0 -0
@@ -608,6 +608,17 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 	vmx_load_mmu_pgd(vcpu, root_hpa, pgd_level);
 }
 

arch/x86/kvm/vmx/main.c static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · line 620

Quoted 10 lines
arch/x86/kvm/vmx/main.c +10 -0
+static void vt_handle_exit_irqoff(struct kvm_vcpu *vcpu)
+{
+	if (unlikely((u16)vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
+		kvm_machine_check();
+
+	if (is_td_vcpu(vcpu))
+		return;
+
+	return vmx_handle_exit_irqoff(vcpu);
+}

I bisected kvm-x86/next down to this change for a TDX guest not booting and host producing errors like:

watchdog: CPU118: Watchdog detected hard LOCKUP on cpu 118

Quoted 8 lines
Quoted 2 lines

Dropping the is_td_vcpu(vcpu) check above fixes the issue. Earlier the call for vmx_handle_exit_irqoff() was unconditional.

Ugh, once you see it, it's obvious. Sorry :-(

I'll drop the entire series and send a v2. There's only one other patch that I already sent the "thank you" for, so I think it's worth unwinding to avoid breaking bisection for TDX (and because the diff can be very different).

OK thanks.

Quoted 43 lines

Lightly tested, but I think this patch can instead be:

arch/x86/kvm/vmx/tdx.c +0 -3
index 163f854a39f2..6d41d2fc8043 100644
@@ -1063,9 +1063,6 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
        if (unlikely((tdx->vp_enter_ret & TDX_SW_ERROR) == TDX_SW_ERROR))
                return EXIT_FASTPATH_NONE;
 
-       if (unlikely(vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
-               kvm_machine_check();
-
        trace_kvm_exit(vcpu, KVM_ISA_VMX);
 
        if (unlikely(tdx_failed_vmentry(vcpu)))
arch/x86/kvm/vmx/vmx.c +11 -2
index d98107a7bdaa..d1117da5463f 100644
@@ -7035,10 +7035,19 @@ void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu)
        if (to_vt(vcpu)->emulation_required)
                return;
 
-       if (vmx_get_exit_reason(vcpu).basic == EXIT_REASON_EXTERNAL_INTERRUPT)
+       switch (vmx_get_exit_reason(vcpu).basic) {
+       case EXIT_REASON_EXTERNAL_INTERRUPT:
                handle_external_interrupt_irqoff(vcpu, vmx_get_intr_info(vcpu));
-       else if (vmx_get_exit_reason(vcpu).basic == EXIT_REASON_EXCEPTION_NMI)
+               break;
+       case EXIT_REASON_EXCEPTION_NMI:
                handle_exception_irqoff(vcpu, vmx_get_intr_info(vcpu));
+               break;
+       case EXIT_REASON_MCE_DURING_VMENTRY:
+               kvm_machine_check();
+               break;
+       default:
+               break;
+       }
 }
 
 /*

arch/x86/kvm/vmx/vmx.c fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · line 7515

Quoted 9 lines
arch/x86/kvm/vmx/vmx.c +0 -3
@@ -7501,9 +7510,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
        if (unlikely(vmx->fail))
                return EXIT_FASTPATH_NONE;
 
-       if (unlikely((u16)vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
-               kvm_machine_check();
-
        trace_kvm_exit(vcpu, KVM_ISA_VMX);
 
        if (unlikely(vmx_get_exit_reason(vcpu).failed_vmentry))

Looks good to me.

Regards,

Tony

#6 lore raw

Re: [PATCH 2/4] KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath

In reply to Tony Lindgren
Quoted 10 lines · Tony Lindgren

I bisected kvm-x86/next down to this change for a TDX guest not booting and host producing errors like:

watchdog: CPU118: Watchdog detected hard LOCKUP on cpu 118

Dropping the is_td_vcpu(vcpu) check above fixes the issue. Earlier the call for vmx_handle_exit_irqoff() was unconditional.

Probably the (u16) cast above can be dropped too? It was never used for TDX looking at the patch.

Ah! Thanks for picking this up. I had almost got there but lost my TDX machine for a bit.

author maintainer
#7 lore raw

PATCH 3/4 KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run loop

In reply to Sean Christopherson

Move KVM's swapping of XFEATURE masks, i.e. XCR0 and XSS, out of the fastpath loop now that the guts of the #MC handler runs in task context, i.e. won't invoke schedule() with preemption disabled and clobber state (or crash the kernel) due to trying to context switch XSTATE with a mix of host and guest state.

For all intents and purposes, this reverts commit 1811d979c716 ("x86/kvm:
move kvm_load/put_guest_xcr0 into atomic context"), which papered over an egregious bug/flaw in the #MC handler where it would do schedule() even though IRQs are disabled. E.g. the call stack from the commit:

kvm_load_guest_xcr0
...
kvm_x86_ops->run(vcpu)
vmx_vcpu_run
vmx_complete_atomic_exit
kvm_machine_check
do_machine_check
do_memory_failure
memory_failure
lock_page

Commit 1811d979c716 "fixed" the immediate issue of XRSTORS exploding, but completely ignored that scheduling out a vCPU task while IRQs and preemption is wildly broken. Thankfully, commit 5567d11c21a1 ("x86/mce:
Send #MC singal from task work") (somewhat incidentally?) fixed that flaw by pushing the meat of the work to the user-return path, i.e. to task context.

KVM has also hardened itself against #MC goofs by moving #MC forwarding to kvm_x86_ops.handle_exit_irqoff(), i.e. out of the fastpath. While that's by no means a robust fix, restoring as much state as possible before handling the #MC will hopefully provide some measure of protection in the event that #MC handling goes off the rails again.

Note, KVM always intercepts XCR0 writes for vCPUs without protected state, e.g. there's no risk of consuming a stale XCR0 when determining if a PKRU update is needed; kvm_load_host_xfeatures() only reads, and never writes, vcpu->arch.xcr0.

Deferring the XCR0 and XSS loads shaves ~300 cycles off the fastpath for Intel, and ~500 cycles for AMD. E.g. using INVD in KVM-Unit-Test's vmexit.c, which an extra hack to enable CR4.OXSAVE, latency numbers for AMD Turin go from ~2000 => 1500, and for Intel Emerald Rapids, go from ~1300 => ~1000.

---
 arch/x86/kvm/x86.c | 39 ++++++++++++++++++++++++++-------------
 1 file changed, 26 insertions(+), 13 deletions(-)
arch/x86/kvm/x86.c +26 -13
index b4b5d2d09634..b5c2879e3330 100644
@@ -1203,13 +1203,12 @@ void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw);
 
-void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
+static void kvm_load_guest_xfeatures(struct kvm_vcpu *vcpu)
 {
 	if (vcpu->arch.guest_state_protected)
 		return;
 
 	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
 		if (vcpu->arch.xcr0 != kvm_host.xcr0)
 			xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
 
@@ -1217,6 +1216,27 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
 		    vcpu->arch.ia32_xss != kvm_host.xss)
 			wrmsrq(MSR_IA32_XSS, vcpu->arch.ia32_xss);
 	}
+}
+
+static void kvm_load_host_xfeatures(struct kvm_vcpu *vcpu)
+{
+	if (vcpu->arch.guest_state_protected)
+		return;
+
+	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
+		if (vcpu->arch.xcr0 != kvm_host.xcr0)
+			xsetbv(XCR_XFEATURE_ENABLED_MASK, kvm_host.xcr0);
+
+		if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
+		    vcpu->arch.ia32_xss != kvm_host.xss)
+			wrmsrq(MSR_IA32_XSS, kvm_host.xss);
+	}
+}
+
+void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
+{
+	if (vcpu->arch.guest_state_protected)
+		return;
 
 	if (cpu_feature_enabled(X86_FEATURE_PKU) &&
 	    vcpu->arch.pkru != vcpu->arch.host_pkru &&
@@ -1238,17 +1258,6 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
 		if (vcpu->arch.pkru != vcpu->arch.host_pkru)
 			wrpkru(vcpu->arch.host_pkru);
 	}
-
-	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
-		if (vcpu->arch.xcr0 != kvm_host.xcr0)
-			xsetbv(XCR_XFEATURE_ENABLED_MASK, kvm_host.xcr0);
-
-		if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
-		    vcpu->arch.ia32_xss != kvm_host.xss)
-			wrmsrq(MSR_IA32_XSS, kvm_host.xss);
-	}
-
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_load_host_xsave_state);
 
@@ -11292,6 +11301,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	if (vcpu->arch.guest_fpu.xfd_err)
 		wrmsrq(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);
 
+	kvm_load_guest_xfeatures(vcpu);
+
 	if (unlikely(vcpu->arch.switch_db_regs &&
 		     !(vcpu->arch.switch_db_regs & KVM_DEBUGREG_AUTO_SWITCH))) {
 		set_debugreg(DR7_FIXED_1, 7);
@@ -11378,6 +11389,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	vcpu->mode = OUTSIDE_GUEST_MODE;
 	smp_wmb();
 
+	kvm_load_host_xfeatures(vcpu);
+
 	/*
 	 * Sync xfd before calling handle_exit_irqoff() which may
 	 * rely on the fact that guest_fpu::xfd is up-to-date (e.g.
Signature · 1 line
2.51.1.930.gacf6e81ea2-goog
#8 lore raw

Re: [PATCH 3/4] KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run loop

On 10/31/2025 6:42 AM, Sean Christopherson wrote:
[...]

Quoted 22 lines
arch/x86/kvm/x86.c +7 -2
-void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
+static void kvm_load_guest_xfeatures(struct kvm_vcpu *vcpu)
  {
  	if (vcpu->arch.guest_state_protected)
  		return;
 
  	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
  		if (vcpu->arch.xcr0 != kvm_host.xcr0)
  			xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
@@ -1217,6 +1216,27 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
  		    vcpu->arch.ia32_xss != kvm_host.xss)
  			wrmsrq(MSR_IA32_XSS, vcpu->arch.ia32_xss);
  	}
+}
+
+static void kvm_load_host_xfeatures(struct kvm_vcpu *vcpu)
+{
+	if (vcpu->arch.guest_state_protected)
+		return;

arch/x86/kvm/x86.c void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · line 1234

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+
+	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
+		if (vcpu->arch.xcr0 != kvm_host.xcr0)
+			xsetbv(XCR_XFEATURE_ENABLED_MASK, kvm_host.xcr0);
+
+		if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
+		    vcpu->arch.ia32_xss != kvm_host.xss)
+			wrmsrq(MSR_IA32_XSS, kvm_host.xss);
+	}
+}

kvm_load_guest_xfeatures() and kvm_load_host_xfeatures() are almost the same except for the guest values VS. host values to set.
I am wondering if it is worth adding a helper to dedup the code, like:

static void kvm_load_xfeatures(struct kvm_vcpu *vcpu, u64 xcr0, u64 xss) {
        if (vcpu->arch.guest_state_protected)
                return;

        if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) { if (vcpu->arch.xcr0 != kvm_host.xcr0) xsetbv(XCR_XFEATURE_ENABLED_MASK, xcr0);

                if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) && vcpu->arch.ia32_xss != kvm_host.xss) wrmsrq(MSR_IA32_XSS, xss);
        }
}

author maintainer
#9 lore raw

Re: [PATCH 3/4] KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run loop

Quoted 47 lines · Binbin Wu

On 10/31/2025 6:42 AM, Sean Christopherson wrote:
[...]

Quoted 19 lines
arch/x86/kvm/x86.c +7 -2
-void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
+static void kvm_load_guest_xfeatures(struct kvm_vcpu *vcpu)
  {
  	if (vcpu->arch.guest_state_protected)
  		return;
  	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
  		if (vcpu->arch.xcr0 != kvm_host.xcr0)
  			xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
@@ -1217,6 +1216,27 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
  		    vcpu->arch.ia32_xss != kvm_host.xss)
  			wrmsrq(MSR_IA32_XSS, vcpu->arch.ia32_xss);
  	}
+}
+
+static void kvm_load_host_xfeatures(struct kvm_vcpu *vcpu)
+{
+	if (vcpu->arch.guest_state_protected)
+		return;

arch/x86/kvm/x86.c void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · line 1234

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+
+	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
+		if (vcpu->arch.xcr0 != kvm_host.xcr0)
+			xsetbv(XCR_XFEATURE_ENABLED_MASK, kvm_host.xcr0);
+
+		if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
+		    vcpu->arch.ia32_xss != kvm_host.xss)
+			wrmsrq(MSR_IA32_XSS, kvm_host.xss);
+	}
+}

kvm_load_guest_xfeatures() and kvm_load_host_xfeatures() are almost the same except for the guest values VS. host values to set.
I am wondering if it is worth adding a helper to dedup the code, like:

static void kvm_load_xfeatures(struct kvm_vcpu *vcpu, u64 xcr0, u64 xss) {
        if (vcpu->arch.guest_state_protected)
                return;

        if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) { if (vcpu->arch.xcr0 != kvm_host.xcr0) xsetbv(XCR_XFEATURE_ENABLED_MASK, xcr0);

Quoted 5 lines

                if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) && vcpu->arch.ia32_xss != kvm_host.xss) wrmsrq(MSR_IA32_XSS, xss);
        }
}

Nice! I like it. Want to send a proper patch (relative to this series)? Or I can turn the above into a patch with a Suggested-by. Either way works for me.

#10 lore raw

Re: [PATCH 3/4] KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run loop

Quoted 46 lines · Sean Christopherson
Quoted 45 lines · Binbin Wu

On 10/31/2025 6:42 AM, Sean Christopherson wrote:
[...]

Quoted 19 lines
arch/x86/kvm/x86.c +7 -2
-void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
+static void kvm_load_guest_xfeatures(struct kvm_vcpu *vcpu)
   {
   	if (vcpu->arch.guest_state_protected)
   		return;
   	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
   		if (vcpu->arch.xcr0 != kvm_host.xcr0)
   			xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
@@ -1217,6 +1216,27 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
   		    vcpu->arch.ia32_xss != kvm_host.xss)
   			wrmsrq(MSR_IA32_XSS, vcpu->arch.ia32_xss);
   	}
+}
+
+static void kvm_load_host_xfeatures(struct kvm_vcpu *vcpu)
+{
+	if (vcpu->arch.guest_state_protected)
+		return;

arch/x86/kvm/x86.c void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · line 1234

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+
+	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
+		if (vcpu->arch.xcr0 != kvm_host.xcr0)
+			xsetbv(XCR_XFEATURE_ENABLED_MASK, kvm_host.xcr0);
+
+		if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
+		    vcpu->arch.ia32_xss != kvm_host.xss)
+			wrmsrq(MSR_IA32_XSS, kvm_host.xss);
+	}
+}

kvm_load_guest_xfeatures() and kvm_load_host_xfeatures() are almost the same except for the guest values VS. host values to set.
I am wondering if it is worth adding a helper to dedup the code, like:

static void kvm_load_xfeatures(struct kvm_vcpu *vcpu, u64 xcr0, u64 xss) {
        if (vcpu->arch.guest_state_protected)
                return;

        if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) { if (vcpu->arch.xcr0 != kvm_host.xcr0) xsetbv(XCR_XFEATURE_ENABLED_MASK, xcr0);

Quoted 8 lines
Quoted 5 lines

                if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) && vcpu->arch.ia32_xss != kvm_host.xss) wrmsrq(MSR_IA32_XSS, xss);
        }
}

Nice! I like it. Want to send a proper patch (relative to this series)? Or I can turn the above into a patch with a Suggested-by. Either way works for me.

I can send a patch.

author maintainer
#11 lore raw

PATCH 4/4 KVM: x86: Load guest/host PKRU outside of the fastpath run loop

In reply to Sean Christopherson

Move KVM's swapping of PKRU outside of the fastpath loop, as there is no KVM code anywhere in the fastpath that accesses guest/userspace memory, i.e. that can consume protection keys.

As documented by commit 1be0e61c1f25 ("KVM, pkeys: save/restore PKRU when guest/host switches"), KVM just needs to ensure the host's PKRU is loaded when KVM (or the kernel at-large) may access userspace memory. And at the time of commit 1be0e61c1f25, KVM didn't have a fastpath, and PKU was strictly contained to VMX, i.e. there was no reason to swap PKRU outside of vmx_vcpu_run().

Over time, the "need" to swap PKRU close to VM-Enter was likely falsely solidified by the association with XFEATUREs in commit 37486135d3a7 ("KVM: x86: Fix pkru save/restore when guest CR4.PKE=0, move it to x86.c"), and XFEATURE swapping was in turn moved close to VM-Enter/VM-Exit as a KVM hack-a-fix ution for an #MC handler bug by commit 1811d979c716 ("x86/kvm: move kvm_load/put_guest_xcr0 into atomic context").

Deferring the PKRU loads shaves ~40 cycles off the fastpath for Intel, and ~60 cycles for AMD. E.g. using INVD in KVM-Unit-Test's vmexit.c, with extra hacks to enable CR4.PKE and PKRU=(-1u & ~0x3), latency numbers for AMD Turin go from ~1560 => ~1500, and for Intel Emerald Rapids, go from ~810 => ~770.

---
 arch/x86/kvm/svm/svm.c |  2 --
 arch/x86/kvm/vmx/vmx.c |  4 ----
 arch/x86/kvm/x86.c     | 14 ++++++++++----
 arch/x86/kvm/x86.h     |  2 --
 4 files changed, 10 insertions(+), 12 deletions(-)
arch/x86/kvm/svm/svm.c +0 -2
index e8b158f73c79..e1fb853c263c 100644
@@ -4260,7 +4260,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 		svm_set_dr6(vcpu, DR6_ACTIVE_LOW);
 
 	clgi();
-	kvm_load_guest_xsave_state(vcpu);
 
 	/*
 	 * Hardware only context switches DEBUGCTL if LBR virtualization is
@@ -4303,7 +4302,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 	    vcpu->arch.host_debugctl != svm->vmcb->save.dbgctl)
 		update_debugctlmsr(vcpu->arch.host_debugctl);
 
-	kvm_load_host_xsave_state(vcpu);
 	stgi();
 
 	/* Any pending NMI will happen here */
arch/x86/kvm/vmx/vmx.c +0 -4
index 123dae8cf46b..55d637cea84a 100644
@@ -7465,8 +7465,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 	if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
 		vmx_set_interrupt_shadow(vcpu, 0);
 
-	kvm_load_guest_xsave_state(vcpu);
-
 	pt_guest_enter(vmx);
 
 	atomic_switch_perf_msrs(vmx);
@@ -7510,8 +7508,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 
 	pt_guest_exit(vmx);
 
-	kvm_load_host_xsave_state(vcpu);
-
 	if (is_guest_mode(vcpu)) {
 		/*
 		 * Track VMLAUNCH/VMRESUME that have made past guest state
arch/x86/kvm/x86.c +10 -4
index b5c2879e3330..6924006f0796 100644
@@ -1233,7 +1233,7 @@ static void kvm_load_host_xfeatures(struct kvm_vcpu *vcpu)
 	}
 }
 
-void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
+static void kvm_load_guest_pkru(struct kvm_vcpu *vcpu)
 {
 	if (vcpu->arch.guest_state_protected)
 		return;
@@ -1244,9 +1244,8 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
 	     kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE)))
 		wrpkru(vcpu->arch.pkru);
 }
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_load_guest_xsave_state);
 
-void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
+static void kvm_load_host_pkru(struct kvm_vcpu *vcpu)
 {
 	if (vcpu->arch.guest_state_protected)
 		return;
@@ -1259,7 +1258,6 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
 			wrpkru(vcpu->arch.host_pkru);
 	}
 }
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_load_host_xsave_state);
 
 #ifdef CONFIG_X86_64
 static inline u64 kvm_guest_supported_xfd(struct kvm_vcpu *vcpu)
@@ -11331,6 +11329,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 	guest_timing_enter_irqoff();
 
+	/*
+	 * Swap PKRU with hardware breakpoints disabled to minimize the number
+	 * of flows where non-KVM code can run with guest state loaded.
+	 */
+	kvm_load_guest_pkru(vcpu);
+
 	for (;;) {
 		/*
 		 * Assert that vCPU vs. VM APICv state is consistent.  An APICv
@@ -11359,6 +11363,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		++vcpu->stat.exits;
 	}
 
+	kvm_load_host_pkru(vcpu);
+
 	/*
 	 * Do this here before restoring debug registers on the host.  And
 	 * since we do this before handling the vmexit, a DR access vmexit
arch/x86/kvm/x86.h +0 -2
index f3dc77f006f9..24c754b0db2e 100644
@@ -622,8 +622,6 @@ static inline void kvm_machine_check(void)
 #endif
 }
 
-void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu);
-void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu);
 int kvm_spec_ctrl_test_value(u64 value);
 int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r,
 			      struct x86_exception *e);
Signature · 1 line
2.51.1.930.gacf6e81ea2-goog
#12 lore raw

Re: [PATCH 4/4] KVM: x86: Load guest/host PKRU outside of the fastpath run loop

Quoted 116 lines · Sean Christopherson

!-------------------------------------------------------------------|
CAUTION: External Email

|-------------------------------------------------------------------!

Move KVM's swapping of PKRU outside of the fastpath loop, as there is no KVM code anywhere in the fastpath that accesses guest/userspace memory, i.e. that can consume protection keys.

As documented by commit 1be0e61c1f25 ("KVM, pkeys: save/restore PKRU when guest/host switches"), KVM just needs to ensure the host's PKRU is loaded when KVM (or the kernel at-large) may access userspace memory. And at the time of commit 1be0e61c1f25, KVM didn't have a fastpath, and PKU was strictly contained to VMX, i.e. there was no reason to swap PKRU outside of vmx_vcpu_run().

Over time, the "need" to swap PKRU close to VM-Enter was likely falsely solidified by the association with XFEATUREs in commit 37486135d3a7 ("KVM: x86: Fix pkru save/restore when guest CR4.PKE=0, move it to x86.c"), and XFEATURE swapping was in turn moved close to VM-Enter/VM-Exit as a KVM hack-a-fix ution for an #MC handler bug by commit 1811d979c716 ("x86/kvm: move kvm_load/put_guest_xcr0 into atomic context").

Deferring the PKRU loads shaves ~40 cycles off the fastpath for Intel, and ~60 cycles for AMD. E.g. using INVD in KVM-Unit-Test's vmexit.c, with extra hacks to enable CR4.PKE and PKRU=(-1u & ~0x3), latency numbers for AMD Turin go from ~1560 => ~1500, and for Intel Emerald Rapids, go from ~810 => ~770.

---

arch/x86/kvm/svm/svm.c |  2 --
arch/x86/kvm/vmx/vmx.c |  4 ----
arch/x86/kvm/x86.c     | 14 ++++++++++----
arch/x86/kvm/x86.h     |  2 --
4 files changed, 10 insertions(+), 12 deletions(-)
arch/x86/kvm/svm/svm.c +0 -2
index e8b158f73c79..e1fb853c263c 100644
@@ -4260,7 +4260,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 svm_set_dr6(vcpu, DR6_ACTIVE_LOW);
 
 clgi();
- kvm_load_guest_xsave_state(vcpu);
 
 /*
 * Hardware only context switches DEBUGCTL if LBR virtualization is
@@ -4303,7 +4302,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
   vcpu->arch.host_debugctl != svm->vmcb->save.dbgctl)
 update_debugctlmsr(vcpu->arch.host_debugctl);
 
- kvm_load_host_xsave_state(vcpu);
 stgi();
 
 /* Any pending NMI will happen here */
arch/x86/kvm/vmx/vmx.c +0 -4
index 123dae8cf46b..55d637cea84a 100644
@@ -7465,8 +7465,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
 vmx_set_interrupt_shadow(vcpu, 0);
 
- kvm_load_guest_xsave_state(vcpu);
-
 pt_guest_enter(vmx);
 
 atomic_switch_perf_msrs(vmx);
@@ -7510,8 +7508,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 
 pt_guest_exit(vmx);
 
- kvm_load_host_xsave_state(vcpu);
-
 if (is_guest_mode(vcpu)) {
 /*
 * Track VMLAUNCH/VMRESUME that have made past guest state
arch/x86/kvm/x86.c +2 -4
index b5c2879e3330..6924006f0796 100644
@@ -1233,7 +1233,7 @@ static void kvm_load_host_xfeatures(struct kvm_vcpu *vcpu)
 }
 }
 
-void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
+static void kvm_load_guest_pkru(struct kvm_vcpu *vcpu)
 {
 if (vcpu->arch.guest_state_protected)
 return;
@@ -1244,9 +1244,8 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
    kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE)))
 wrpkru(vcpu->arch.pkru);
 }
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_load_guest_xsave_state);
 
-void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
+static void kvm_load_host_pkru(struct kvm_vcpu *vcpu)
 {
 if (vcpu->arch.guest_state_protected)
 return;
@@ -1259,7 +1258,6 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
 wrpkru(vcpu->arch.host_pkru);
 }
 }
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_load_host_xsave_state);
 
 #ifdef CONFIG_X86_64
 static inline u64 kvm_guest_supported_xfd(struct kvm_vcpu *vcpu)

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 11337

Quoted 9 lines
arch/x86/kvm/x86.c +6 -0
@@ -11331,6 +11329,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 guest_timing_enter_irqoff();
 
+ /*
+ * Swap PKRU with hardware breakpoints disabled to minimize the number
+ * of flows where non-KVM code can run with guest state loaded.
+ */
+ kvm_load_guest_pkru(vcpu);
+

I was mocking this up after PUCK, and went down a similar-ish path, but was thinking it might be interesting to have an x86 op called something to the effect of “prepare_switch_to_guest_irqoff” and “prepare_switch_to_host_irqoff”, which might make for a place to nestle any other sort of “needs to be done in atomic context but doesn’t need to be done in the fast path” sort of stuff (if any).

One other one that caught my eye was the cr3 stuff that was moved out a while ago, but then moved back with 1a7158101.

I haven’t gone through absolutely everything else in that tight loop code (and didn’t get a chance to do the same for SVM code), but figured I’d put the idea out there to see what you think.

To be clear, I’m totally OK with the series as-is, just thinking about perhaps future ways to incrementally optimize here?

Quoted 28 lines
arch/x86/kvm/x86.c +3 -0
 for (;;) {
 /*
 * Assert that vCPU vs. VM APICv state is consistent.  An APICv
@@ -11359,6 +11363,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
++vcpu->stat.exits;
 }
 
+ kvm_load_host_pkru(vcpu);
+
 /*
 * Do this here before restoring debug registers on the host.  And
 * since we do this before handling the vmexit, a DR access vmexit
arch/x86/kvm/x86.h +0 -2
index f3dc77f006f9..24c754b0db2e 100644
@@ -622,8 +622,6 @@ static inline void kvm_machine_check(void)
 #endif
 }
 
-void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu);
-void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu);
 int kvm_spec_ctrl_test_value(u64 value);
 int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r,
     struct x86_exception *e);
Signature · 1 line
2.51.1.930.gacf6e81ea2-goog
author maintainer
#13 lore raw

Re: [PATCH 4/4] KVM: x86: Load guest/host PKRU outside of the fastpath run loop

Quoted 8 lines · Jon Kohler
Quoted 7 lines · Sean Christopherson

+ /*
+ * Swap PKRU with hardware breakpoints disabled to minimize the number
+ * of flows where non-KVM code can run with guest state loaded.
+ */
+ kvm_load_guest_pkru(vcpu);
+

Quoted 5 lines

I was mocking this up after PUCK, and went down a similar-ish path, but was thinking it might be interesting to have an x86 op called something to the effect of “prepare_switch_to_guest_irqoff” and “prepare_switch_to_host_irqoff”, which might make for a place to nestle any other sort of “needs to be done in atomic context but doesn’t need to be done in the fast path” sort of stuff (if any).

Hmm, I would say I'm flat out opposed to generic hooks of that nature. For anything that _needs_ to be modified with IRQs disabled, the ordering will matter greatly. E.g. we already have kvm_x86_ops.sync_pir_to_irr(), and that _must_ run before kvm_vcpu_exit_request() if it triggers a late request.

And I also want to push for as much stuff as possible to be handled in common x86, i.e. I want to actively encourage landing things like PKU and CET support in common x86 instead of implementing support in one vendor and then having to churn a pile of code to later move it to

Quoted 9 lines

One other one that caught my eye was the cr3 stuff that was moved out a while ago, but then moved back with 1a7158101.

I haven’t gone through absolutely everything else in that tight loop code (and didn’t get a chance to do the same for SVM code), but figured I’d put the idea out there to see what you think.

To be clear, I’m totally OK with the series as-is, just thinking about perhaps future ways to incrementally optimize here?

To some extent, we're going to hit diminishing returns. E.g. one of the reasons I did a straight revert in commit 1a71581012dd is that were talking about a handful of cycles difference. E.g. as measured from the guest, eliding the CR3+CR4 checks shaves 3-5 cycles. From the host side it _looks_ like more (~20 cycles), but it's hard to even measure accurately because just doing RDTSC affects the results.

For SVM, I don't see any obvious candidates. E.g. pre_sev_run() has some code that only needs to be done on the first iteration, but checking a flag or doing a static CALL+RET is going to be just as costly as what's already there.

In short, the only flows that will benefit are relatively slow flows and/or flows that aren't easily predicted by the CPU. E.g. __get_current_cr3_fast() and cr4_read_shadow() require CALL+RET and might not be super predictable? But even they are on the cusp of "who cares".

And that needs to be balanced against the probability of introducing bugs. E.g. this code _could_ be done only on the first iteration:

	if (vmx->ple_window_dirty) {
		vmx->ple_window_dirty = false;
		vmcs_write32(PLE_WINDOW, vmx->ple_window);
	}

but (a) checking vmx->ple_window_dirty is going to be super predictable after the first iteration, (b) handling PLE exits in the fastpath would break things, and
(c) _if_ we want to optimize that code, it can/should be simply moved to vmx_prepare_switch_to_guest() (but outside of the guest_state_loaded check).

All that said, I'm not totally opposed to shaving cycles. Now that @run_flags is a thing, it's actually trivially easy to optimize the CR3/CR4 checks (famous last words):

arch/x86/include/asm/kvm_host.h +1 -0
index 48598d017d6f..5cc1f0168b8a 100644
@@ -1709,6 +1709,7 @@ enum kvm_x86_run_flags {
        KVM_RUN_FORCE_IMMEDIATE_EXIT    = BIT(0),
        KVM_RUN_LOAD_GUEST_DR6          = BIT(1),
        KVM_RUN_LOAD_DEBUGCTL           = BIT(2),
+       KVM_RUN_IS_FIRST_ITERATION      = BIT(3),
 };
 
 struct kvm_x86_ops {
arch/x86/kvm/vmx/vmx.c +20 -14
index 55d637cea84a..3deb20b8d0c5 100644
@@ -7439,22 +7439,28 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
                vmx_reload_guest_debugctl(vcpu);
 
        /*
-        * Refresh vmcs.HOST_CR3 if necessary.  This must be done immediately
-        * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time
-        * it switches back to the current->mm, which can occur in KVM context
-        * when switching to a temporary mm to patch kernel code, e.g. if KVM
-        * toggles a static key while handling a VM-Exit.
+        * Refresh vmcs.HOST_CR3 if necessary.  This must be done after IRQs
+        * are disabled, i.e. not when preparing to switch to the guest, as the
+        * the kernel may load a new ASID (PCID) any time it switches back to
+        * the current->mm, which can occur in KVM context when switching to a
+        * temporary mm to patch kernel code, e.g. if KVM toggles a static key
+        * while handling a VM-Exit.
+        *
+        * Refresh host CR3 and CR4 only on the first iteration of the inner
+        * loop, as modifying CR3 or CR4 from NMI context is not allowed.
         */
-       cr3 = __get_current_cr3_fast();
-       if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) {
-               vmcs_writel(HOST_CR3, cr3);
-               vmx->loaded_vmcs->host_state.cr3 = cr3;
-       }
+       if (run_flags & KVM_RUN_IS_FIRST_ITERATION) {
+               cr3 = __get_current_cr3_fast();
+               if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) {
+                       vmcs_writel(HOST_CR3, cr3);
+                       vmx->loaded_vmcs->host_state.cr3 = cr3;
+               }
 
-       cr4 = cr4_read_shadow();
-       if (unlikely(cr4 != vmx->loaded_vmcs->host_state.cr4)) {
-               vmcs_writel(HOST_CR4, cr4);
-               vmx->loaded_vmcs->host_state.cr4 = cr4;
+               cr4 = cr4_read_shadow();
+               if (unlikely(cr4 != vmx->loaded_vmcs->host_state.cr4)) {
+                       vmcs_writel(HOST_CR4, cr4);
+                       vmx->loaded_vmcs->host_state.cr4 = cr4;
+               }
        }
 
        /* When single-stepping over STI and MOV SS, we must clear the
arch/x86/kvm/x86.c +1 -1
index 6924006f0796..bff08f58c29a 100644
@@ -11286,7 +11286,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
                goto cancel_injection;
        }
 
-       run_flags = 0;
+       run_flags = KVM_RUN_IS_FIRST_ITERATION;
        if (req_immediate_exit) {
                run_flags |= KVM_RUN_FORCE_IMMEDIATE_EXIT;
                kvm_make_request(KVM_REQ_EVENT, vcpu);
#14 lore raw

Re: [PATCH 4/4] KVM: x86: Load guest/host PKRU outside of the fastpath run loop

Quoted 9 lines · Sean Christopherson

Hmm, I would say I'm flat out opposed to generic hooks of that nature. For anything that _needs_ to be modified with IRQs disabled, the ordering will matter greatly. E.g. we already have kvm_x86_ops.sync_pir_to_irr(), and that _must_ run before kvm_vcpu_exit_request() if it triggers a late request.

Quoted 4 lines

And I also want to push for as much stuff as possible to be handled in common x86, i.e. I want to actively encourage landing things like PKU and CET support in common x86 instead of implementing support in one vendor and then having to churn a pile of code to later move it to

Fair, agreed having things common-ized helps everyone

Quoted 3 lines

All that said, I'm not totally opposed to shaving cycles. Now that @run_flags is a thing, it's actually trivially easy to optimize the CR3/CR4 checks (famous last words):

A cycle saved is a cycle earned, perhaps? :)

arch/x86/include/asm/kvm_host.h enum kvm_x86_run_flags { · line 1713

Quoted 10 lines
arch/x86/include/asm/kvm_host.h +1 -0
index 48598d017d6f..5cc1f0168b8a 100644
@@ -1709,6 +1709,7 @@ enum kvm_x86_run_flags {
       KVM_RUN_FORCE_IMMEDIATE_EXIT    = BIT(0),
       KVM_RUN_LOAD_GUEST_DR6          = BIT(1),
       KVM_RUN_LOAD_DEBUGCTL           = BIT(2),
+       KVM_RUN_IS_FIRST_ITERATION      = BIT(3),
 };

I like this approach, as it makes the code easier to grok what we want and when

Quoted 63 lines
arch/x86/include/asm/kvm_host.h +0 -0
 struct kvm_x86_ops {
arch/x86/kvm/vmx/vmx.c +20 -14
index 55d637cea84a..3deb20b8d0c5 100644
@@ -7439,22 +7439,28 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
               vmx_reload_guest_debugctl(vcpu);
 
       /*
-        * Refresh vmcs.HOST_CR3 if necessary.  This must be done immediately
-        * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time
-        * it switches back to the current->mm, which can occur in KVM context
-        * when switching to a temporary mm to patch kernel code, e.g. if KVM
-        * toggles a static key while handling a VM-Exit.
+        * Refresh vmcs.HOST_CR3 if necessary.  This must be done after IRQs
+        * are disabled, i.e. not when preparing to switch to the guest, as the
+        * the kernel may load a new ASID (PCID) any time it switches back to
+        * the current->mm, which can occur in KVM context when switching to a
+        * temporary mm to patch kernel code, e.g. if KVM toggles a static key
+        * while handling a VM-Exit.
+        *
+        * Refresh host CR3 and CR4 only on the first iteration of the inner
+        * loop, as modifying CR3 or CR4 from NMI context is not allowed.
        */
-       cr3 = __get_current_cr3_fast();
-       if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) {
-               vmcs_writel(HOST_CR3, cr3);
-               vmx->loaded_vmcs->host_state.cr3 = cr3;
-       }
+       if (run_flags & KVM_RUN_IS_FIRST_ITERATION) {
+               cr3 = __get_current_cr3_fast();
+               if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) {
+                       vmcs_writel(HOST_CR3, cr3);
+                       vmx->loaded_vmcs->host_state.cr3 = cr3;
+               }
 
-       cr4 = cr4_read_shadow();
-       if (unlikely(cr4 != vmx->loaded_vmcs->host_state.cr4)) {
-               vmcs_writel(HOST_CR4, cr4);
-               vmx->loaded_vmcs->host_state.cr4 = cr4;
+               cr4 = cr4_read_shadow();
+               if (unlikely(cr4 != vmx->loaded_vmcs->host_state.cr4)) {
+                       vmcs_writel(HOST_CR4, cr4);
+                       vmx->loaded_vmcs->host_state.cr4 = cr4;
+               }
       }
 
       /* When single-stepping over STI and MOV SS, we must clear the
arch/x86/kvm/x86.c +1 -1
index 6924006f0796..bff08f58c29a 100644
@@ -11286,7 +11286,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
               goto cancel_injection;
       }
 
-       run_flags = 0;
+       run_flags = KVM_RUN_IS_FIRST_ITERATION;
       if (req_immediate_exit) {
               run_flags |= KVM_RUN_FORCE_IMMEDIATE_EXIT;
               kvm_make_request(KVM_REQ_EVENT, vcpu);
Reviewed-by on the v1 series
#15 lore raw
In reply to Sean Christopherson
Quoted 18 lines · Sean Christopherson

!-------------------------------------------------------------------|
CAUTION: External Email

|-------------------------------------------------------------------!

This series is the result of the recent PUCK discussion[*] on optimizing the XCR0/XSS loads that are currently done on every VM-Enter and VM-Exit. My initial thought that swapping XCR0/XSS outside of the fastpath was spot on; turns out the only reason they're swapped in the fastpath is because of a hack-a-fix that papered over an egregious #MC handling bug where the kernel #MC handler would call schedule() from an atomic context. The resulting #GP due to trying to swap FPU state with a guest XCR0/XSS was "fixed" by loading the host values before handling #MCs from the guest.

Quoted 2 lines

Thankfully, the #MC mess has long since been cleaned up, so it's once again safe to swap XCR0/XSS outside of the fastpath (but when IRQs are disabled!).

Thank you for doing the diligence on this, I appreciate it!

Quoted 7 lines

As for what may be contributing to the SAP HANA performance improvements when enabling PKU, my instincts again appear to be spot on. As predicted, the fastpath savings are ~300 cycles on Intel (~500 on AMD). I.e. if the guest is literally doing _nothing_ but generating fastpath exits, it will see a ~%25 improvement. There's basically zero chance the uplift seen with enabling PKU is dues to eliding XCR0 loads; my guess is that the guest actualy uses protection keys to optimize something.

Every little bit counts, thats a healthy percentage speedup for fast path stuff, especially on AMD.

Quoted 7 lines

Why does kvm_load_guest_xsave_state() show up in perf? Probably because it's the only visible symbol other than vmx_vmexit() (and vmx_vcpu_run() when not hammering the fastpath). E.g. running perf top on a running VM instance yields these numbers with various guest workloads (the middle one is running mmu_stress_test in the guest, which hammers on mmu_lock in L0). But other than doing INVD (handled in the fastpath) in a tight loop, there's no perceived perf improvement from the guest.

nit: it’d be nice if these bits were labeled with what they were from (the middle one you called out above, but what’s the first and third one)

Quoted 50 lines

Overhead Shared Object Symbol

 15.65%  [kernel]            [k] vmx_vmexit
  6.78%  [kernel]            [k] kvm_vcpu_halt
  5.15%  [kernel]            [k] __srcu_read_lock
  4.73%  [kernel]            [k] kvm_load_guest_xsave_state
  4.69%  [kernel]            [k] __srcu_read_unlock
  4.65%  [kernel]            [k] read_tsc
  4.44%  [kernel]            [k] vmx_sync_pir_to_irr
  4.03%  [kernel]            [k] kvm_apic_has_interrupt
 45.52%  [kernel]            [k] queued_spin_lock_slowpath
 24.40%  [kernel]            [k] vmx_vmexit
  2.84%  [kernel]            [k] queued_write_lock_slowpath
  1.92%  [kernel]            [k] vmx_vcpu_run
  1.40%  [kernel]            [k] vcpu_run
  1.00%  [kernel]            [k] kvm_load_guest_xsave_state
  0.84%  [kernel]            [k] kvm_load_host_xsave_state
  0.72%  [kernel]            [k] mmu_try_to_unsync_pages
  0.68%  [kernel]            [k] __srcu_read_lock
  0.65%  [kernel]            [k] try_get_folio
 17.78%  [kernel]            [k] vmx_vmexit
  5.08%  [kernel]            [k] vmx_vcpu_run
  4.24%  [kernel]            [k] vcpu_run
  4.21%  [kernel]            [k] _raw_spin_lock_irqsave
  2.99%  [kernel]            [k] kvm_load_guest_xsave_state
  2.51%  [kernel]            [k] rcu_note_context_switch
  2.47%  [kernel]            [k] ktime_get_update_offsets_now
  2.21%  [kernel]            [k] kvm_load_host_xsave_state
  2.16%  [kernel]            [k] fput

[*] https://drive.google.com/drive/folders/1DCdvqFGudQc7pxXjM7f35vXogTf9uhD4

Sean Christopherson (4):
KVM: SVM: Handle #MCs in guest outside of fastpath
KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath
KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run
loop
KVM: x86: Load guest/host PKRU outside of the fastpath run loop

arch/x86/kvm/svm/svm.c  | 20 ++++++++--------
arch/x86/kvm/vmx/main.c | 13 ++++++++++-
arch/x86/kvm/vmx/tdx.c  |  3 ---
arch/x86/kvm/vmx/vmx.c  |  7 ------
arch/x86/kvm/x86.c      | 51 ++++++++++++++++++++++++++++-------------
arch/x86/kvm/x86.h      |  2 --
6 files changed, 56 insertions(+), 40 deletions(-)
Quoted 4 lines

base-commit: 4cc167c50eb19d44ac7e204938724e685e3d8057

Signature · 1 line
2.51.1.930.gacf6e81ea2-goog

Had one conversation starter comment on patch 4, but otherwise, LGTM for the entire series, thanks again for the help!

maintainer Reviewed-by on the v1 series
#16 lore raw
In reply to Sean Christopherson
Quoted 6 lines · Sean Christopherson

Sean Christopherson (4):
  KVM: SVM: Handle #MCs in guest outside of fastpath
  KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath
  KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run loop
  KVM: x86: Load guest/host PKRU outside of the fastpath run loop

Interesting analysis.

author maintainer Applied
#17 lore raw
In reply to Sean Christopherson
Quoted 10 lines · Sean Christopherson

This series is the result of the recent PUCK discussion[*] on optimizing the XCR0/XSS loads that are currently done on every VM-Enter and VM-Exit. My initial thought that swapping XCR0/XSS outside of the fastpath was spot on; turns out the only reason they're swapped in the fastpath is because of a hack-a-fix that papered over an egregious #MC handling bug where the kernel #MC handler would call schedule() from an atomic context. The resulting #GP due to trying to swap FPU state with a guest XCR0/XSS was "fixed" by loading the host values before handling #MCs from the guest.

[...]

Applied to kvm-x86 misc, thanks!

[1/4] KVM: SVM: Handle #MCs in guest outside of fastpath

      https://github.com/kvm-x86/linux/commit/6e640bb5caab
[2/4] KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath
      https://github.com/kvm-x86/linux/commit/8934c592bcbf
[3/4] KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run loop
      https://github.com/kvm-x86/linux/commit/3377a9233d30
[4/4] KVM: x86: Load guest/host PKRU outside of the fastpath run loop
      https://github.com/kvm-x86/linux/commit/7df3021b622f

--
https://github.com/kvm-x86/linux/tree/next

author maintainer
#18 lore raw
Quoted 14 lines · Sean Christopherson
Quoted 10 lines · Sean Christopherson

This series is the result of the recent PUCK discussion[*] on optimizing the XCR0/XSS loads that are currently done on every VM-Enter and VM-Exit. My initial thought that swapping XCR0/XSS outside of the fastpath was spot on; turns out the only reason they're swapped in the fastpath is because of a hack-a-fix that papered over an egregious #MC handling bug where the kernel #MC handler would call schedule() from an atomic context. The resulting #GP due to trying to swap FPU state with a guest XCR0/XSS was "fixed" by loading the host values before handling #MCs from the guest.

[...]

Applied to kvm-x86 misc, thanks!

Quoted 8 lines

[1/4] KVM: SVM: Handle #MCs in guest outside of fastpath

      https://github.com/kvm-x86/linux/commit/6e640bb5caab
[2/4] KVM: VMX: Handle #MCs on VM-Enter/TD-Enter outside of the fastpath
      https://github.com/kvm-x86/linux/commit/8934c592bcbf
[3/4] KVM: x86: Load guest/host XCR0 and XSS outside of the fastpath run loop
      https://github.com/kvm-x86/linux/commit/3377a9233d30
[4/4] KVM: x86: Load guest/host PKRU outside of the fastpath run loop
      https://github.com/kvm-x86/linux/commit/7df3021b622f

I've dropped these for now as patch 2 broke TDX. I'll send a v2 shortly.