Skip to contentSkip to conversation

KVM: TDX: TD vcpu enter/exit

PATCH In review Adrian Hunter 4 participants 38 messages Jan 29, 2025 – Mar 10, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover KVM: TDX: TD vcpu enter/exit

Hi

Changes in V2:

Note, in addition to addressing replies to version 1, there are changes suggested by Sean here:

    https://lore.kernel.org/r/[email protected]

Patches are re-based on kvm-coco-queue plus:

latest TDX MMU Part 2 series
latest TDX vCPU/VM creation series

x86/virt/tdx: Make tdh_vp_enter() noinstr
KVM: x86: Allow the use of kvm_load_host_xsave_state() with
guest_state_protected
KVM: TDX: Set arch.has_protected_state to true
KVM: VMX: Move common fields of struct vcpu_{vmx,tdx} to a struct
New patches

KVM: TDX: Implement TDX vcpu enter/exit path
Move VCPU_TD_STATE_INITIALIZED check to tdx_vcpu_pre_run() (Xiaoyao) Check TD_STATE_RUNNABLE also in tdx_vcpu_pre_run() (Yan) Add back 'noinstr' for tdx_vcpu_enter_exit() (Sean)
Add WARN_ON_ONCE if force_immediate_exit (Sean)
Add vp_enter_args to vcpu_tdx to store the input/output arguments for tdh_vp_enter().
Don't copy arguments to/from vcpu->arch.regs[] unconditionally. (Sean)

KVM: TDX: vcpu_run: save/restore host state(host kernel gs)
Use 1 variable named 'guest_state_loaded' to track host state save/restore (Sean)
Rebased due to moving guest_state_loaded/msr_host_kernel_gs_base to struct vcpu_vt.

KVM: TDX: restore host xsave state when exit from the guest TD
Drop PT and CET feature flags (Chao)
Use cpu_feature_enabled() instead of static_cpu_has() (Chao) Restore PKRU only if the host value differs from defined exit value (Chao)
Use defined masks to separate XFAM bits into XCR0/XSS (Adrian) Use existing kvm_load_host_xsave_state() in place of tdx_restore_host_xsave_state() by defining guest CR4, XCR0, XSS and PKRU (Sean)
Do not enter if vital guest state is invalid (Adrian)

KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr
KVM: TDX: restore user ret MSRs
No changes

KVM: TDX: Disable support for TSX and WAITPKG
KVM: TDX: Save and restore IA32_DEBUGCTL
New patches

KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior
Moved from TDX "the rest" to "TD vcpu enter/exit"

This patch series introduces callbacks to facilitate the entry of a TD VCPU and the corresponding save/restore of host state.

The "outstanding things still to do" from V1 are now all done. Refer V1 cover letter:

https://lore.kernel.org/[email protected]

So the patches are ready for hand off to Paolo.

Overview

A TD VCPU is entered via the SEAMCALL TDH.VP.ENTER. The TDX Module manages the save/restore of guest state and, in conjunction with the SEAMCALL interface, handles certain aspects of host state. However, there are specific elements of the host state that require additional attention, as detailed in the Intel TDX ABI documentation for TDH.VP.ENTER.

TDX is quite different from VMX in this regard. For VMX, the host VMM is heavily involved in restoring, managing and saving guest CPU state, whereas for TDX this is handled by the TDX Module. In that way, the TDX Module can protect the confidentiality and integrity of TD CPU state.

The TDX Module does not save/restore all host CPU state because the host VMM can do it more efficiently and selectively. CPU state referred to below is host CPU state. Often values are already held in memory so no explicit save is needed, and restoration may not be needed if the kernel is not using a feature.

Key Details

Argument Passing: Similar to other SEAMCALLs, TDH.VP.ENTER passes arguments through General Purpose Registers (GPRs). For the special case of the TD guest invoking TDG.VP.VMCALL, nearly any GPR can be used, as well as XMM0 to XMM15. Notably, RBP is not used, and Linux mandates the TDX Module feature NO_RBP_MOD, which is enforced elsewhere. Additionally, XMM registers are not required for the existing Guest Hypervisor Communication Interface and are handled by existing KVM code should they be modified by the guest.

Debug Register Handling: After TDH.VP.ENTER returns, registers DR0, DR1, DR2, DR3, DR6, and DR7 are set to their architectural INIT values. Existing KVM code already handles the restoration of host values as needed, refer vcpu_enter_guest() which calls hw_breakpoint_restore().

MSR Restoration: Certain Model-Specific Registers (MSRs) need to be restored post TDH.VP.ENTER. The Intel TDX ABI documentation provides a detailed list in the msr_preservation.json file. Most MSRs do not require restoration if the guest is not utilizing the corresponding feature. The following features are currently assumed to be unsupported, and their MSRs are not restored:

  PERFMON            (TD ATTRIBUTES[63])
  LBRs               (XFAM[15])
  User Interrupts    (XFAM[14])
  Intel PT           (XFAM[8])
  CET                (XFAM[11-12])

Remaining host MSR/Register Handling:

MSR IA32_XFD is already restored by KVM, refer to kvm_put_guest_fpu(). The TDX Module sets MSR IA32_XFD_ERR to its RESET value (0) which is fine for the kernel.

MSR IA32_DEBUGCTL is addressed in patch "KVM: TDX: Save and restore IA32_DEBUGCTL".

MSR IA32_UARCH_MISC_CTL is not utilized by the kernel, so it is fine if the TDX Module sets it to it's RESET value.

MSR IA32_KERNEL_GS_BASE is addressed in patch "KVM: TDX: vcpu_run:
save/restore host state (host kernel gs)".

MSRs IA32_XSS and XCRO are handled in patch "KVM: TDX: restore host xsave state when exiting from the guest TD".

MSRs IA32_STAR, IA32_LSTAR, IA32_FMASK, and IA32_TSC_AUX are handled in patch "KVM: TDX: restore user ret MSRs".

MSR IA32_TSX_CTRL and MSR IA32_UMWAIT_CONTROL are addressed in patch "KVM: TDX: Disable support for TSX and WAITPKG"

Protecting Host CPU State

These patches completely protect host CPU state. The host CPU state that can be affected is determined by the TD configuration, especially XFAM, TD attributes and CPUID.

Should any vital guest information be found to conflict with TD configuration, VCPU entry is blocked by checks in tdx_guest_state_is_invalid().

Base

This series is based on kvm-coco-queue plus:

latest TDX MMU Part 2 series
latest TDX vCPU/VM creation series

Due to "KVM: VMX: Move common fields of struct", subsequent patches require changes to use new struct vcpu_vt, refer to the full KVM branch below.

It requires TDX module 1.5.06.00.0744 [1], or later. This is due to removal of the workarounds for the lack of the NO_RBP_MOD feature required by the kernel. Now NO_RBP_MOD is enabled (in VM/vCPU creation patches), and this particular version of the TDX module has a required NO_RBP_MOD related bug fix.
A working edk2 commit is 95d8a1c ("UnitTestFrameworkPkg: Use TianoCore mirror of subhook submodule").

Testing

The series has been tested as part of the development branch for the TDX base series. The testing consisted of TDX kvm-unit-tests and booting a Linux TD, and TDX enhanced KVM selftests.

The full KVM branch is here:

https://github.com/intel/tdx/tree/tdx_kvm_dev-2025-01-28.2

Matching QEMU: TBD

[1] https://github.com/intel/tdx-module/releases/tag/TDX_1.5.06

Adrian Hunter (4):
x86/virt/tdx: Make tdh_vp_enter() noinstr
KVM: TDX: Set arch.has_protected_state to true
KVM: TDX: Disable support for TSX and WAITPKG
KVM: TDX: Save and restore IA32_DEBUGCTL

Binbin Wu (1):
KVM: VMX: Move common fields of struct vcpu_{vmx,tdx} to a struct

Chao Gao (1):
KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr

Isaku Yamahata (5):
KVM: TDX: Implement TDX vcpu enter/exit path
KVM: TDX: vcpu_run: save/restore host state(host kernel gs)
KVM: TDX: restore host xsave state when exit from the guest TD
KVM: TDX: restore user ret MSRs
KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior

Sean Christopherson (1):
KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

 arch/x86/include/asm/kvm_host.h  |  12 +-
 arch/x86/kvm/svm/svm.c           |   7 +-
 arch/x86/kvm/vmx/common.h        |  68 +++++++++++
 arch/x86/kvm/vmx/main.c          |  48 +++++++-
 arch/x86/kvm/vmx/nested.c        |  10 +-
 arch/x86/kvm/vmx/posted_intr.c   |  18 +--
 arch/x86/kvm/vmx/tdx.c           | 258 +++++++++++++++++++++++++++++++++++++--
 arch/x86/kvm/vmx/tdx.h           |  19 +--
 arch/x86/kvm/vmx/vmx.c           |  99 +++++++--------
 arch/x86/kvm/vmx/vmx.h           | 104 ++++++----------
 arch/x86/kvm/vmx/x86_ops.h       |  11 ++
 arch/x86/kvm/x86.c               |  46 +++++--
 arch/x86/virt/vmx/tdx/seamcall.S |   3 +
 arch/x86/virt/vmx/tdx/tdx.c      |   2 +-
 14 files changed, 535 insertions(+), 170 deletions(-)

Regards
Adrian

#1 lore raw

PATCH v2 1/12 x86/virt/tdx: Make tdh_vp_enter() noinstr

Make tdh_vp_enter() noinstr because KVM requires VM entry to be noinstr for 2 reasons:
1. The use of context tracking via guest_state_enter_irqoff() and guest_state_exit_irqoff()
2. The need to avoid IRET between VM-exit and NMI handling in order to avoid prematurely releasing NMI inhibit.

Consequently make __seamcall_saved_ret() noinstr also. Currently tdh_vp_enter() is the only caller of __seamcall_saved_ret().

---
TD vcpu enter/exit v2:
- New patch

---
 arch/x86/virt/vmx/tdx/seamcall.S | 3 +++
 arch/x86/virt/vmx/tdx/tdx.c      | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
arch/x86/virt/vmx/tdx/seamcall.S +3 -0
index 5b1f2286aea9..6854c52c374b 100644
@@ -41,6 +41,9 @@ SYM_FUNC_START(__seamcall_ret)
 	TDX_MODULE_CALL host=1 ret=1
 SYM_FUNC_END(__seamcall_ret)
 
+/* KVM requires non-instrumentable __seamcall_saved_ret() for TDH.VP.ENTER */
+.section .noinstr.text, "ax"
+
 /*
  * __seamcall_saved_ret() - Host-side interface functions to SEAM software
  * (the P-SEAMLDR or the TDX module), with saving output registers to the
arch/x86/virt/vmx/tdx/tdx.c +1 -1
index 4a010e65276d..1515c467dd86 100644
@@ -1511,7 +1511,7 @@ static void tdx_clflush_page(struct page *page)
 	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
 }
 
-u64 tdh_vp_enter(struct tdx_vp *td, struct tdx_module_args *args)
+noinstr u64 tdh_vp_enter(struct tdx_vp *td, struct tdx_module_args *args)
 {
 	args->rcx = tdx_tdvpr_pa(td);
 
Signature · 1 line
2.43.0
maintainer
#2 lore raw

Re: [PATCH V2 01/12] x86/virt/tdx: Make tdh_vp_enter() noinstr

Quoted 11 lines · Adrian Hunter

Make tdh_vp_enter() noinstr because KVM requires VM entry to be noinstr for 2 reasons:
1. The use of context tracking via guest_state_enter_irqoff() and guest_state_exit_irqoff()
2. The need to avoid IRET between VM-exit and NMI handling in order to avoid prematurely releasing NMI inhibit.

Consequently make __seamcall_saved_ret() noinstr also. Currently tdh_vp_enter() is the only caller of __seamcall_saved_ret().

This can be squashed into "x86/virt/tdx: Add SEAMCALL wrapper to enter/exit TDX guest"; I did that in kvm-coco-queue.

Paolo

Quoted 35 lines

---
TD vcpu enter/exit v2:
- New patch

---
  arch/x86/virt/vmx/tdx/seamcall.S | 3 +++
  arch/x86/virt/vmx/tdx/tdx.c      | 2 +-
  2 files changed, 4 insertions(+), 1 deletion(-)
arch/x86/virt/vmx/tdx/seamcall.S +3 -0
index 5b1f2286aea9..6854c52c374b 100644
@@ -41,6 +41,9 @@ SYM_FUNC_START(__seamcall_ret)
  	TDX_MODULE_CALL host=1 ret=1
  SYM_FUNC_END(__seamcall_ret)
 
+/* KVM requires non-instrumentable __seamcall_saved_ret() for TDH.VP.ENTER */
+.section .noinstr.text, "ax"
+
  /*
   * __seamcall_saved_ret() - Host-side interface functions to SEAM software
   * (the P-SEAMLDR or the TDX module), with saving output registers to the
arch/x86/virt/vmx/tdx/tdx.c +1 -1
index 4a010e65276d..1515c467dd86 100644
@@ -1511,7 +1511,7 @@ static void tdx_clflush_page(struct page *page)
  	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
  }
 
-u64 tdh_vp_enter(struct tdx_vp *td, struct tdx_module_args *args)
+noinstr u64 tdh_vp_enter(struct tdx_vp *td, struct tdx_module_args *args)
  {
  	args->rcx = tdx_tdvpr_pa(td);
 
#3 lore raw

Re: [PATCH V2 01/12] x86/virt/tdx: Make tdh_vp_enter() noinstr

Quoted 8 lines · Paolo Bonzini
Quoted 7 lines · Adrian Hunter

Make tdh_vp_enter() noinstr because KVM requires VM entry to be noinstr for 2 reasons:
  1. The use of context tracking via guest_state_enter_irqoff() and guest_state_exit_irqoff()
  2. The need to avoid IRET between VM-exit and NMI handling in order to avoid prematurely releasing NMI inhibit.

Quoted 6 lines
Quoted 4 lines

Consequently make __seamcall_saved_ret() noinstr also. Currently tdh_vp_enter() is the only caller of __seamcall_saved_ret().

This can be squashed into "x86/virt/tdx: Add SEAMCALL wrapper to enter/exit TDX guest"; I did that in kvm-coco-queue.

We have re-based on kvm-coco-queue so we in-sync on this.

#4 lore raw

PATCH v2 2/12 KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

In reply to Adrian Hunter

From: Sean Christopherson <[email protected]>

Allow the use of kvm_load_host_xsave_state() with vcpu->arch.guest_state_protected == true. This will allow TDX to reuse kvm_load_host_xsave_state() instead of creating its own version.

For consistency, amend kvm_load_guest_xsave_state() also.

Ensure that guest state that kvm_load_host_xsave_state() depends upon, such as MSR_IA32_XSS, cannot be changed by user space, if guest_state_protected.

[Adrian: wrote commit message]

---
TD vcpu enter/exit v2:
- New patch

---
 arch/x86/kvm/svm/svm.c |  7 +++++--
 arch/x86/kvm/x86.c     | 18 +++++++++++-------
 2 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/svm.c +5 -2
index 7640a84e554a..b4bcfe15ad5e 100644
@@ -4253,7 +4253,9 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
 		svm_set_dr6(svm, DR6_ACTIVE_LOW);
 
 	clgi();
-	kvm_load_guest_xsave_state(vcpu);
+
+	if (!vcpu->arch.guest_state_protected)
+		kvm_load_guest_xsave_state(vcpu);
 
 	kvm_wait_lapic_expire(vcpu);
 
@@ -4282,7 +4284,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
 	if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
 		kvm_before_interrupt(vcpu, KVM_HANDLING_NMI);
 
-	kvm_load_host_xsave_state(vcpu);
+	if (!vcpu->arch.guest_state_protected)
+		kvm_load_host_xsave_state(vcpu);
 	stgi();
 
 	/* Any pending NMI will happen here */
arch/x86/kvm/x86.c +11 -7
index bbb6b7f40b3a..5cf9f023fd4b 100644
@@ -1169,11 +1169,9 @@ EXPORT_SYMBOL_GPL(kvm_lmsw);
 
 void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
 {
-	if (vcpu->arch.guest_state_protected)
-		return;
+	WARN_ON_ONCE(vcpu->arch.guest_state_protected);
 
 	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
 		if (vcpu->arch.xcr0 != kvm_host.xcr0)
 			xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
 
@@ -1192,13 +1190,11 @@ EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
 
 void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
 {
-	if (vcpu->arch.guest_state_protected)
-		return;
-
 	if (cpu_feature_enabled(X86_FEATURE_PKU) &&
 	    ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
 	     kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE))) {
-		vcpu->arch.pkru = rdpkru();
+		if (!vcpu->arch.guest_state_protected)
+			vcpu->arch.pkru = rdpkru();
 		if (vcpu->arch.pkru != vcpu->arch.host_pkru)
 			wrpkru(vcpu->arch.host_pkru);
 	}
@@ -3916,6 +3912,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		if (!msr_info->host_initiated &&
 		    !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES))
 			return 1;
+
+		if (vcpu->arch.guest_state_protected)
+			return 1;
+
 		/*
 		 * KVM supports exposing PT to the guest, but does not support
 		 * IA32_XSS[bit 8]. Guests have to use RDMSR/WRMSR rather than
@@ -4375,6 +4375,10 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		if (!msr_info->host_initiated &&
 		    !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES))
 			return 1;
+
+		if (vcpu->arch.guest_state_protected)
+			return 1;
+
 		msr_info->data = vcpu->arch.ia32_xss;
 		break;
 	case MSR_K7_CLK_CTL:
Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

Quoted 70 lines · Adrian Hunter

From: Sean Christopherson <[email protected]>

Allow the use of kvm_load_host_xsave_state() with vcpu->arch.guest_state_protected == true. This will allow TDX to reuse kvm_load_host_xsave_state() instead of creating its own version.

For consistency, amend kvm_load_guest_xsave_state() also.

Ensure that guest state that kvm_load_host_xsave_state() depends upon, such as MSR_IA32_XSS, cannot be changed by user space, if guest_state_protected.

[Adrian: wrote commit message]

---
TD vcpu enter/exit v2:
- New patch

---
  arch/x86/kvm/svm/svm.c |  7 +++++--
  arch/x86/kvm/x86.c     | 18 +++++++++++-------
  2 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/svm.c +5 -2
index 7640a84e554a..b4bcfe15ad5e 100644
@@ -4253,7 +4253,9 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
  		svm_set_dr6(svm, DR6_ACTIVE_LOW);
 
  	clgi();
-	kvm_load_guest_xsave_state(vcpu);
+
+	if (!vcpu->arch.guest_state_protected)
+		kvm_load_guest_xsave_state(vcpu);
 
  	kvm_wait_lapic_expire(vcpu);
 
@@ -4282,7 +4284,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
  	if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
  		kvm_before_interrupt(vcpu, KVM_HANDLING_NMI);
 
-	kvm_load_host_xsave_state(vcpu);
+	if (!vcpu->arch.guest_state_protected)
+		kvm_load_host_xsave_state(vcpu);
  	stgi();
 
  	/* Any pending NMI will happen here */
arch/x86/kvm/x86.c +1 -3
index bbb6b7f40b3a..5cf9f023fd4b 100644
@@ -1169,11 +1169,9 @@ EXPORT_SYMBOL_GPL(kvm_lmsw);
 
  void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
  {
-	if (vcpu->arch.guest_state_protected)
-		return;
+	WARN_ON_ONCE(vcpu->arch.guest_state_protected);
 
  	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
  		if (vcpu->arch.xcr0 != kvm_host.xcr0)
  			xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
 
@@ -1192,13 +1190,11 @@ EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
 
  void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)

arch/x86/kvm/x86.c EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state); · line 1197

Quoted 10 lines
arch/x86/kvm/x86.c +2 -4
  {
-	if (vcpu->arch.guest_state_protected)
-		return;
-
  	if (cpu_feature_enabled(X86_FEATURE_PKU) &&
  	    ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) ||
  	     kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE))) {
-		vcpu->arch.pkru = rdpkru();
+		if (!vcpu->arch.guest_state_protected)
+			vcpu->arch.pkru = rdpkru();

this needs justification.

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3918

Quoted 13 lines
arch/x86/kvm/x86.c +4 -0
  		if (vcpu->arch.pkru != vcpu->arch.host_pkru)
  			wrpkru(vcpu->arch.host_pkru);
  	}
@@ -3916,6 +3912,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
  		if (!msr_info->host_initiated &&
  		    !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES))
  			return 1;
+
+		if (vcpu->arch.guest_state_protected)
+			return 1;
+

this and below change need to be a separate patch. So that we can discuss independently.

I see no reason to make MSR_IA32_XSS special than other MSRs. When guest_state_protected, most of the MSRs that aren't emulated by KVM are inaccessible by KVM.

Quoted 14 lines
arch/x86/kvm/x86.c +4 -0
  		/*
  		 * KVM supports exposing PT to the guest, but does not support
  		 * IA32_XSS[bit 8]. Guests have to use RDMSR/WRMSR rather than
@@ -4375,6 +4375,10 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
  		if (!msr_info->host_initiated &&
  		    !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES))
  			return 1;
+
+		if (vcpu->arch.guest_state_protected)
+			return 1;
+
  		msr_info->data = vcpu->arch.ia32_xss;
  		break;
  	case MSR_K7_CLK_CTL:
#6 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

Quoted 66 lines · Xiaoyao Li
Quoted 65 lines · Adrian Hunter

From: Sean Christopherson <[email protected]>

Allow the use of kvm_load_host_xsave_state() with vcpu->arch.guest_state_protected == true. This will allow TDX to reuse kvm_load_host_xsave_state() instead of creating its own version.

For consistency, amend kvm_load_guest_xsave_state() also.

Ensure that guest state that kvm_load_host_xsave_state() depends upon, such as MSR_IA32_XSS, cannot be changed by user space, if guest_state_protected.

[Adrian: wrote commit message]

---
TD vcpu enter/exit v2:
  - New patch
---
  arch/x86/kvm/svm/svm.c |  7 +++++--
  arch/x86/kvm/x86.c     | 18 +++++++++++-------

  2 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/svm.c +3 -1
index 7640a84e554a..b4bcfe15ad5e 100644
@@ -4253,7 +4253,9 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
           svm_set_dr6(svm, DR6_ACTIVE_LOW);
         clgi();
-    kvm_load_guest_xsave_state(vcpu);
+
+    if (!vcpu->arch.guest_state_protected)
+        kvm_load_guest_xsave_state(vcpu);
         kvm_wait_lapic_expire(vcpu);
   @@ -4282,7 +4284,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
       if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
           kvm_before_interrupt(vcpu, KVM_HANDLING_NMI);

  -    kvm_load_host_xsave_state(vcpu);
+    if (!vcpu->arch.guest_state_protected)
+        kvm_load_host_xsave_state(vcpu);
      stgi();
        /* Any pending NMI will happen here */

arch/x86/kvm/x86.c +1 -6
index bbb6b7f40b3a..5cf9f023fd4b 100644
@@ -1169,11 +1169,9 @@ EXPORT_SYMBOL_GPL(kvm_lmsw);
     void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
   {
-    if (vcpu->arch.guest_state_protected)
-        return;
+    WARN_ON_ONCE(vcpu->arch.guest_state_protected);
         if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
           if (vcpu->arch.xcr0 != kvm_host.xcr0)
               xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
   @@ -1192,13 +1190,11 @@ EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
     void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
   {
-    if (vcpu->arch.guest_state_protected)
-        return;
-
Quoted 8 lines
Quoted 6 lines

      if (cpu_feature_enabled(X86_FEATURE_PKU) && ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE))) { -        vcpu->arch.pkru = rdpkru();
+        if (!vcpu->arch.guest_state_protected) +            vcpu->arch.pkru = rdpkru();

this needs justification.

It was proposed by Sean here:

	https://lore.kernel.org/all/[email protected]/

which is part of the email thread referenced by the "Link:" tag above

Quoted 17 lines
Quoted 10 lines

          if (vcpu->arch.pkru != vcpu->arch.host_pkru) wrpkru(vcpu->arch.host_pkru);
      }

arch/x86/kvm/x86.c +0 -0
@@ -3916,6 +3912,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)

          if (!msr_info->host_initiated && !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) return 1;

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3918

Quoted 4 lines
arch/x86/kvm/x86.c +4 -0
+
+        if (vcpu->arch.guest_state_protected)
+            return 1;
+

this and below change need to be a separate patch. So that we can discuss independently.

Quoted 1 line

I see no reason to make MSR_IA32_XSS special than other MSRs. When guest_state_protected, most of the MSRs that aren't emulated by KVM are inaccessible by KVM.

Yes, TDX will block access to MSR_IA32_XSS anyway because tdx_has_emulated_msr() will return false for MSR_IA32_XSS.

However kvm_load_host_xsave_state() is not TDX-specific code and it relies upon vcpu->arch.ia32_xss, so there is reason to block access to it when vcpu->arch.guest_state_protected is true.

Quoted 16 lines
Quoted 15 lines

          /*
           * KVM supports exposing PT to the guest, but does not support
           * IA32_XSS[bit 8]. Guests have to use RDMSR/WRMSR rather than

arch/x86/kvm/x86.c +0 -0
@@ -4375,6 +4375,10 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)

          if (!msr_info->host_initiated && !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) return 1;

arch/x86/kvm/x86.c +4 -0
+
+        if (vcpu->arch.guest_state_protected)
+            return 1;
+

          msr_info->data = vcpu->arch.ia32_xss;
          break;
      case MSR_K7_CLK_CTL:

#7 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

Quoted 76 lines · Adrian Hunter
Quoted 66 lines · Xiaoyao Li
Quoted 65 lines · Adrian Hunter

From: Sean Christopherson <[email protected]>

Allow the use of kvm_load_host_xsave_state() with vcpu->arch.guest_state_protected == true. This will allow TDX to reuse kvm_load_host_xsave_state() instead of creating its own version.

For consistency, amend kvm_load_guest_xsave_state() also.

Ensure that guest state that kvm_load_host_xsave_state() depends upon, such as MSR_IA32_XSS, cannot be changed by user space, if guest_state_protected.

[Adrian: wrote commit message]

---
TD vcpu enter/exit v2:
  - New patch

---
   arch/x86/kvm/svm/svm.c |  7 +++++--
   arch/x86/kvm/x86.c     | 18 +++++++++++-------
   2 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/svm.c +3 -1
index 7640a84e554a..b4bcfe15ad5e 100644
@@ -4253,7 +4253,9 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
           svm_set_dr6(svm, DR6_ACTIVE_LOW);
         clgi();
-    kvm_load_guest_xsave_state(vcpu);
+
+    if (!vcpu->arch.guest_state_protected)
+        kvm_load_guest_xsave_state(vcpu);
         kvm_wait_lapic_expire(vcpu);
   @@ -4282,7 +4284,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
       if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
           kvm_before_interrupt(vcpu, KVM_HANDLING_NMI);

  -    kvm_load_host_xsave_state(vcpu);
+    if (!vcpu->arch.guest_state_protected)
+        kvm_load_host_xsave_state(vcpu);
      stgi();
        /* Any pending NMI will happen here */

arch/x86/kvm/x86.c +1 -6
index bbb6b7f40b3a..5cf9f023fd4b 100644
@@ -1169,11 +1169,9 @@ EXPORT_SYMBOL_GPL(kvm_lmsw);
     void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
   {
-    if (vcpu->arch.guest_state_protected)
-        return;
+    WARN_ON_ONCE(vcpu->arch.guest_state_protected);
         if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
           if (vcpu->arch.xcr0 != kvm_host.xcr0)
               xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
   @@ -1192,13 +1190,11 @@ EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
     void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
   {
-    if (vcpu->arch.guest_state_protected)
-        return;
-
Quoted 8 lines
Quoted 6 lines

      if (cpu_feature_enabled(X86_FEATURE_PKU) && ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE))) {
-        vcpu->arch.pkru = rdpkru();
+        if (!vcpu->arch.guest_state_protected) +            vcpu->arch.pkru = rdpkru();

this needs justification.

Quoted 5 lines

It was proposed by Sean here:

	https://lore.kernel.org/all/[email protected]/

which is part of the email thread referenced by the "Link:" tag above

IMHO, this change needs to be put in patch 07, which is the better place to justify it.

Quoted 17 lines
Quoted 17 lines
Quoted 7 lines

          if (vcpu->arch.pkru != vcpu->arch.host_pkru) wrpkru(vcpu->arch.host_pkru);
      }

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3918

Quoted 7 lines
arch/x86/kvm/x86.c +4 -0
@@ -3916,6 +3912,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
           if (!msr_info->host_initiated &&
               !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES))
               return 1;
+
+        if (vcpu->arch.guest_state_protected)
+            return 1;
+

this and below change need to be a separate patch. So that we can discuss independently.

Quoted 8 lines
Quoted 1 line

I see no reason to make MSR_IA32_XSS special than other MSRs. When guest_state_protected, most of the MSRs that aren't emulated by KVM are inaccessible by KVM.

Yes, TDX will block access to MSR_IA32_XSS anyway because tdx_has_emulated_msr() will return false for MSR_IA32_XSS.

However kvm_load_host_xsave_state() is not TDX-specific code and it relies upon vcpu->arch.ia32_xss, so there is reason to block access to it when vcpu->arch.guest_state_protected is true.

It is TDX specific logic that TDX requires vcpu->arch.ia32_xss unchanged since TDX is going to utilize kvm_load_host_xsave_state() to restore host xsave state and relies on vcpu->arch.ia32_xss to be always the value of XFAM & XSS_MASK.

So please put this change into the TDX specific patch with the clear justfication.

Quoted 17 lines
Quoted 16 lines
Quoted 15 lines
arch/x86/kvm/x86.c +4 -0
           /*
            * KVM supports exposing PT to the guest, but does not support
            * IA32_XSS[bit 8]. Guests have to use RDMSR/WRMSR rather than
@@ -4375,6 +4375,10 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
           if (!msr_info->host_initiated &&
               !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES))
               return 1;
+
+        if (vcpu->arch.guest_state_protected)
+            return 1;
+
           msr_info->data = vcpu->arch.ia32_xss;
           break;
       case MSR_K7_CLK_CTL:
#8 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

Quoted 107 lines · Xiaoyao Li
Quoted 76 lines · Adrian Hunter
Quoted 66 lines · Xiaoyao Li
Quoted 65 lines · Adrian Hunter

From: Sean Christopherson <[email protected]>

Allow the use of kvm_load_host_xsave_state() with vcpu->arch.guest_state_protected == true. This will allow TDX to reuse kvm_load_host_xsave_state() instead of creating its own version.

For consistency, amend kvm_load_guest_xsave_state() also.

Ensure that guest state that kvm_load_host_xsave_state() depends upon, such as MSR_IA32_XSS, cannot be changed by user space, if guest_state_protected.

[Adrian: wrote commit message]

---
TD vcpu enter/exit v2:
   - New patch
---
   arch/x86/kvm/svm/svm.c |  7 +++++--
   arch/x86/kvm/x86.c     | 18 +++++++++++-------

   2 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/svm.c +3 -1
index 7640a84e554a..b4bcfe15ad5e 100644
@@ -4253,7 +4253,9 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
            svm_set_dr6(svm, DR6_ACTIVE_LOW);
          clgi();
-    kvm_load_guest_xsave_state(vcpu);
+
+    if (!vcpu->arch.guest_state_protected)
+        kvm_load_guest_xsave_state(vcpu);
          kvm_wait_lapic_expire(vcpu);
    @@ -4282,7 +4284,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu,
        if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
            kvm_before_interrupt(vcpu, KVM_HANDLING_NMI);

   -    kvm_load_host_xsave_state(vcpu);
+    if (!vcpu->arch.guest_state_protected)
+        kvm_load_host_xsave_state(vcpu);
       stgi();
         /* Any pending NMI will happen here */

arch/x86/kvm/x86.c +1 -6
index bbb6b7f40b3a..5cf9f023fd4b 100644
@@ -1169,11 +1169,9 @@ EXPORT_SYMBOL_GPL(kvm_lmsw);
      void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
    {
-    if (vcpu->arch.guest_state_protected)
-        return;
+    WARN_ON_ONCE(vcpu->arch.guest_state_protected);
          if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
            if (vcpu->arch.xcr0 != kvm_host.xcr0)
                xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
    @@ -1192,13 +1190,11 @@ EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
      void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
    {
-    if (vcpu->arch.guest_state_protected)
-        return;
-
Quoted 8 lines
Quoted 6 lines

       if (cpu_feature_enabled(X86_FEATURE_PKU) && ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE))) { -        vcpu->arch.pkru = rdpkru();
+        if (!vcpu->arch.guest_state_protected) +            vcpu->arch.pkru = rdpkru();

this needs justification.

Quoted 5 lines

It was proposed by Sean here:

    https://lore.kernel.org/all/[email protected]/

which is part of the email thread referenced by the "Link:" tag above

IMHO, this change needs to be put in patch 07, which is the better place to justify it.

Quoted 22 lines
Quoted 17 lines
Quoted 10 lines

           if (vcpu->arch.pkru != vcpu->arch.host_pkru) wrpkru(vcpu->arch.host_pkru);
       }

arch/x86/kvm/x86.c +0 -0
@@ -3916,6 +3912,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)

           if (!msr_info->host_initiated && !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) return 1;

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3918

Quoted 4 lines
arch/x86/kvm/x86.c +4 -0
+
+        if (vcpu->arch.guest_state_protected)
+            return 1;
+

this and below change need to be a separate patch. So that we can discuss independently.

Quoted 1 line

I see no reason to make MSR_IA32_XSS special than other MSRs. When guest_state_protected, most of the MSRs that aren't emulated by KVM are inaccessible by KVM.

Yes, TDX will block access to MSR_IA32_XSS anyway because tdx_has_emulated_msr() will return false for MSR_IA32_XSS.

Quoted 7 lines
Quoted 3 lines

However kvm_load_host_xsave_state() is not TDX-specific code and it relies upon vcpu->arch.ia32_xss, so there is reason to block access to it when vcpu->arch.guest_state_protected is true.

It is TDX specific logic that TDX requires vcpu->arch.ia32_xss unchanged since TDX is going to utilize kvm_load_host_xsave_state() to restore host xsave state and relies on vcpu->arch.ia32_xss to be always the value of XFAM & XSS_MASK.

So please put this change into the TDX specific patch with the clear justfication.

This patch set is owned by Paolo now, so it is up to him.

Quoted 18 lines
Quoted 17 lines
Quoted 16 lines
Quoted 15 lines

           /*
            * KVM supports exposing PT to the guest, but does not support
            * IA32_XSS[bit 8]. Guests have to use RDMSR/WRMSR rather than

arch/x86/kvm/x86.c +0 -0
@@ -4375,6 +4375,10 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)

           if (!msr_info->host_initiated && !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) return 1;

arch/x86/kvm/x86.c +4 -0
+
+        if (vcpu->arch.guest_state_protected)
+            return 1;
+

           msr_info->data = vcpu->arch.ia32_xss;
           break;
       case MSR_K7_CLK_CTL:

maintainer
#9 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

In reply to Xiaoyao Li
Quoted 94 lines · Xiaoyao Li
Quoted 67 lines · Adrian Hunter

From: Sean Christopherson <[email protected]>

Allow the use of kvm_load_host_xsave_state() with vcpu->arch.guest_state_protected == true. This will allow TDX to reuse kvm_load_host_xsave_state() instead of creating its own version.

For consistency, amend kvm_load_guest_xsave_state() also.

Ensure that guest state that kvm_load_host_xsave_state() depends upon, such as MSR_IA32_XSS, cannot be changed by user space, if guest_state_protected.

[Adrian: wrote commit message]

---
TD vcpu enter/exit v2:
  - New patch
---
  arch/x86/kvm/svm/svm.c |  7 +++++--
  arch/x86/kvm/x86.c     | 18 +++++++++++-------

  2 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/svm.c +5 -2
index 7640a84e554a..b4bcfe15ad5e 100644
@@ -4253,7 +4253,9 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct 
 kvm_vcpu *vcpu,
           svm_set_dr6(svm, DR6_ACTIVE_LOW);
       clgi();
-    kvm_load_guest_xsave_state(vcpu);
+
+    if (!vcpu->arch.guest_state_protected)
+        kvm_load_guest_xsave_state(vcpu);
       kvm_wait_lapic_expire(vcpu);
@@ -4282,7 +4284,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct 
 kvm_vcpu *vcpu,
       if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
           kvm_before_interrupt(vcpu, KVM_HANDLING_NMI);
-    kvm_load_host_xsave_state(vcpu);
+    if (!vcpu->arch.guest_state_protected)
+        kvm_load_host_xsave_state(vcpu);
       stgi();
       /* Any pending NMI will happen here */
arch/x86/kvm/x86.c +1 -6
index bbb6b7f40b3a..5cf9f023fd4b 100644
@@ -1169,11 +1169,9 @@ EXPORT_SYMBOL_GPL(kvm_lmsw);
   void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
   {
-    if (vcpu->arch.guest_state_protected)
-        return;
+    WARN_ON_ONCE(vcpu->arch.guest_state_protected);
       if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
-
           if (vcpu->arch.xcr0 != kvm_host.xcr0)
               xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
@@ -1192,13 +1190,11 @@ EXPORT_SYMBOL_GPL(kvm_load_guest_xsave_state);
   void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
   {
-    if (vcpu->arch.guest_state_protected)
-        return;
-
Quoted 6 lines

      if (cpu_feature_enabled(X86_FEATURE_PKU) && ((vcpu->arch.xcr0 & XFEATURE_MASK_PKRU) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE))) { -        vcpu->arch.pkru = rdpkru();
+        if (!vcpu->arch.guest_state_protected) +            vcpu->arch.pkru = rdpkru();

this needs justification.

Quoted 10 lines

          if (vcpu->arch.pkru != vcpu->arch.host_pkru) wrpkru(vcpu->arch.host_pkru);
      }

arch/x86/kvm/x86.c +0 -0
@@ -3916,6 +3912,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, 
 struct msr_data *msr_info)

          if (!msr_info->host_initiated && !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) return 1;

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3918

Quoted 4 lines
arch/x86/kvm/x86.c +4 -0
+
+        if (vcpu->arch.guest_state_protected)
+            return 1;
+

this and below change need to be a separate patch. So that we can discuss independently.

Quoted 1 line

I see no reason to make MSR_IA32_XSS special than other MSRs. When guest_state_protected, most of the MSRs that aren't emulated by KVM are inaccessible by KVM.

I agree with Xiaoyao that this change is sensible but should be proposed separately for both SNP and TDX.

Allowing the use of kvm_load_host_xsave_state() is really ugly, especially since the corresponding code is so simple:

        if (cpu_feature_enabled(X86_FEATURE_PKU) && vcpu->arch.pkru != 0)
                        wrpkru(vcpu->arch.host_pkru);

	if (kvm_host.xcr0 != kvm_tdx->xfam & kvm_caps.supported_xcr0)
		xsetbv(XCR_XFEATURE_ENABLED_MASK, kvm_host.xcr0);

	/*
	 * All TDX hosts support XSS; but even if they didn't, both
	 * arms of the comparison would be 0 and the wrmsrl would be
	 * skipped.
	 */
	if (kvm_host.xss != kvm_tdx->xfam & kvm_caps.supported_xss)
		wrmsrl(MSR_IA32_XSS, kvm_host.xss);

This is really all that should be in patch 7. I'll test it and decide what to do.

Paolo

Quoted 17 lines
Quoted 15 lines

          /*
           * KVM supports exposing PT to the guest, but does not support
           * IA32_XSS[bit 8]. Guests have to use RDMSR/WRMSR rather than

arch/x86/kvm/x86.c +0 -0
@@ -4375,6 +4375,10 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, 
 struct msr_data *msr_info)

          if (!msr_info->host_initiated && !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) return 1;

arch/x86/kvm/x86.c +4 -0
+
+        if (vcpu->arch.guest_state_protected)
+            return 1;
+

          msr_info->data = vcpu->arch.ia32_xss;
          break;
      case MSR_K7_CLK_CTL:

#10 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

Quoted 8 lines · Paolo Bonzini

I agree with Xiaoyao that this change is sensible but should be proposed separately for both SNP and TDX.

Allowing the use of kvm_load_host_xsave_state() is really ugly, especially since the corresponding code is so simple:

        if (cpu_feature_enabled(X86_FEATURE_PKU) && vcpu->arch.pkru != 0)
                        wrpkru(vcpu->arch.host_pkru);

It's clearly not "so simple", because this code is buggy.

The justification for using kvm_load_host_xsave_state() is that either KVM gets the TDX state model correct and the existing flows Just Work, or we handle all that state as one-offs and at best replicate concepts and flows, and at worst have bugs that are unique to TDX, e.g. because we get the "simple" code wrong, we miss flows that subtly consume state, etc.

maintainer
#11 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

Il gio 6 mar 2025, 21:44 Sean Christopherson <[email protected]> ha scritto:

Quoted 6 lines
Quoted 5 lines

Allowing the use of kvm_load_host_xsave_state() is really ugly, especially since the corresponding code is so simple:

        if (cpu_feature_enabled(X86_FEATURE_PKU) && vcpu->arch.pkru != 0)
                        wrpkru(vcpu->arch.host_pkru);
Quoted 7 lines

It's clearly not "so simple", because this code is buggy.

The justification for using kvm_load_host_xsave_state() is that either KVM gets the TDX state model correct and the existing flows Just Work, or we handle all that state as one-offs and at best replicate concepts and flows, and at worst have bugs that are unique to TDX, e.g. because we get the "simple" code wrong, we miss flows that subtly consume state, etc.

A typo doesn't change the fact that kvm_load_host_xsave_state is optimized with knowledge of the guest CR0 and CR4; faking the values so that the same field means both "exit value" and "guest value", just so that the common code does the right thing for pkru/xcr0/xss, is unmaintainable and conceptually just wrong. And while the change for XSS (and possibly other MSRs) is actually correct, it should be justified for both SEV-ES/SNP and TDX rather than sneaked into the TDX patches.

While there could be other flows that consume guest state, they're just as bound to do the wrong thing if vcpu->arch is only guaranteed to be somehow plausible (think anything that for whatever reason uses cpu_role). There's no way the existing flows for !guest_state_protected should run _at all_ when the register state is not there. If they do, it's a bug and fixing them is the right thing to do (it may feel like whack-a-mole but isn't). See for example the KVM_CAP_SYNC_REGS calls that Adrian mentioned in the reply to 05/12, and for which I'll send a patch too: I haven't checked if it's possible, but I wonder if userspace could misuse sync_regs() to change guest xcr0 and xss, and trick the TDX *host* into running with some bits of xcr0 and xss unexpectedly cleared.

TDX provides well known values for the host for all three registers that are being restored here, so there's no need to reuse code that is meant to do something completely different. I'm not singling out TDX, the same would be true for SEV-ES/SNP swap type C.

Paolo

Paolo

#12 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

Quoted 15 lines · Paolo Bonzini

Il gio 6 mar 2025, 21:44 Sean Christopherson <[email protected]> ha scritto:

Quoted 6 lines
Quoted 5 lines

Allowing the use of kvm_load_host_xsave_state() is really ugly, especially since the corresponding code is so simple:

        if (cpu_feature_enabled(X86_FEATURE_PKU) && vcpu->arch.pkru != 0)
                        wrpkru(vcpu->arch.host_pkru);
Quoted 7 lines

It's clearly not "so simple", because this code is buggy.

The justification for using kvm_load_host_xsave_state() is that either KVM gets the TDX state model correct and the existing flows Just Work, or we handle all that state as one-offs and at best replicate concepts and flows, and at worst have bugs that are unique to TDX, e.g. because we get the "simple" code wrong, we miss flows that subtly consume state, etc.

Quoted 3 lines

A typo doesn't change the fact that kvm_load_host_xsave_state is optimized with knowledge of the guest CR0 and CR4; faking the values so that the same field means both "exit value" and "guest value",

I can't argue against that, but I still absolutely detest carrying dedicated code for SEV and TDX state management. It's bad enough that figuring out WTF actually happens basically requires encyclopedic knowledge of massive specs.

I tried to figure out a way to share code, but everything I can come up with that doesn't fake vCPU state makes the non-TDX code a mess. :-(

Quoted 1 line

just so that the common code does the right thing for pkru/xcr0/xss,

FWIW, it's not just to that KVM does the right thing for those values, it's a defense in depth mechanism so that *when*, not if, KVM screws up, the odds of the bug being fatal to KVM and/or the guest are reduced.

Quoted 1 line

is > unmaintainable and conceptually just wrong.

I don't necessarily disagree, but what we have today isn't maintainable either. Without actual sanity check and safeguards in the low level helpers, we absolutely are playing a game of whack-a-mole.

E.g. see commit 9b42d1e8e4fe ("KVM: x86: Play nice with protected guests in complete_hypercall_exit()").

At a glance, kvm_hv_hypercall() is still broken, because is_protmode() will return false incorrectly.

Quoted 8 lines

And while the change for XSS (and possibly other MSRs) is actually correct, it should be justified for both SEV-ES/SNP and TDX rather than sneaked into the TDX patches.

While there could be other flows that consume guest state, they're just as bound to do the wrong thing if vcpu->arch is only guaranteed to be somehow plausible (think anything that for whatever reason uses cpu_role).

But the MMU code is *already* broken. kvm_init_mmu() => vcpu_to_role_regs(). It "works" because the fubar role is never truly consumed. I'm sure there are more examples.

Quoted 3 lines

There's no way the existing flows for !guest_state_protected should run _at all_ when the register state is not there. If they do, it's a bug and fixing them is the right thing to do (it may feel like whack-a-mole but isn't)

Eh, it's still whack-a-mole, there just happen to be a finite number of moles :-)

maintainer
#13 lore raw

Re: [PATCH V2 02/12] KVM: x86: Allow the use of kvm_load_host_xsave_state() with guest_state_protected

Quoted 8 lines · Sean Christopherson

On Thu, Mar 06, 2025, Paolo Bonzini wrote:
I still absolutely detest carrying dedicated code
for SEV and TDX state management. It's bad enough that figuring out WTF actually happens basically requires encyclopedic knowledge of massive specs.

I tried to figure out a way to share code, but everything I can come up with that doesn't fake vCPU state makes the non-TDX code a mess. :-(

The only thing worse is requiring encyclopedic knowledge of both the specs and KVM. :) And yeah, we do require some knowledge of parts of KVM
that *shouldn't* matter for protected-state guests, but it shouldn't be worse than needed.

There's different microcode/firmware for VMX/SVM/SEV-ES+/TDX, the chance of sharing code is lower and lower as more stuff is added there---as is the case
for SEV-ES/SNP and TDX. Which is why state management code for TDX is anyway doing its own thing most of the time---there's no point in sharing a little bit which is not even the hardest.

Quoted 5 lines
Quoted 1 line

just so that the common code does the right thing for pkru/xcr0/xss,

FWIW, it's not just to that KVM does the right thing for those values, it's a defense in depth mechanism so that *when*, not if, KVM screws up, the odds of the bug being fatal to KVM and/or the guest are reduced.

I would say the other way round is true too. Not relying too much on fake values in vcpu->arch can be more robust.

Quoted 8 lines

Without actual sanity check and safeguards in the low level helpers, we absolutely are playing a game of whack-a-mole.

E.g. see commit 9b42d1e8e4fe ("KVM: x86: Play nice with protected guests in complete_hypercall_exit()").

At a glance, kvm_hv_hypercall() is still broken, because is_protmode() will return false incorrectly.

So the fixes are needed anyway and we're playing the game anyway. :(

Quoted 4 lines
Quoted 4 lines

And while the change for XSS (and possibly other MSRs) is actually correct, it should be justified for both SEV-ES/SNP and TDX rather than sneaked into the TDX patches.

Quoted 8 lines
Quoted 4 lines

While there could be other flows that consume guest state, they're just as bound to do the wrong thing if vcpu->arch is only guaranteed to be somehow plausible (think anything that for whatever reason uses cpu_role).

But the MMU code is *already* broken. kvm_init_mmu() => vcpu_to_role_regs(). It "works" because the fubar role is never truly consumed. I'm sure there are more examples.

Yes, and there should be at least a WARN_ON_ONCE when it is accessed, even if we don't completely cull the initialization of cpu_role... Loading the XSAVE state isn't any different.

I'm okay with placing some values in cr0/cr4 or even xcr0/xss, but do not wish to use them more than the absolute minimum necessary. And I would rather not set more than the bare minimum needed in CR4... why set CR4.PKE for example, if KVM anyway has no business using the guest PKRU.

Paolo

Quoted 5 lines
Quoted 3 lines

There's no way the existing flows for !guest_state_protected should run _at all_ when the register state is not there. If they do, it's a bug and fixing them is the right thing to do (it may feel like whack-a-mole but isn't)

Eh, it's still whack-a-mole, there just happen to be a finite number of moles :-)

#14 lore raw

PATCH v2 3/12 KVM: TDX: Set arch.has_protected_state to true

In reply to Adrian Hunter

TDX VMs have protected state. Accordingly, set arch.has_protected_state to true.

This will cause the following IOCTL functions to return an error:

	kvm_arch_vcpu_ioctl()	case KVM_GET_SREGS2
	kvm_arch_vcpu_ioctl()	case KVM_SET_SREGS2
	kvm_arch_vcpu_ioctl_get_regs()
	kvm_arch_vcpu_ioctl_set_regs()
	kvm_arch_vcpu_ioctl_get_sregs()
	kvm_arch_vcpu_ioctl_set_sregs()
	kvm_vcpu_ioctl_x86_get_debugregs()
	kvm_vcpu_ioctl_x86_set_debugregs
	kvm_vcpu_ioctl_x86_get_xcrs()
	kvm_vcpu_ioctl_x86_set_xcrs()

In addition, the following will error for confidential FPU state:

	kvm_vcpu_ioctl_x86_get_xsave ()
	kvm_vcpu_ioctl_x86_get_xsave2()
	kvm_vcpu_ioctl_x86_set_xsave()
	kvm_arch_vcpu_ioctl_get_fpu()
	kvm_arch_vcpu_ioctl_set_fpu()

And finally, in accordance with commit 66155de93bcf ("KVM: x86: Disallow read-only memslots for SEV-ES and SEV-SNP (and TDX)"), read-only memslots will be disallowed.

---
TD vcpu enter/exit v2:
- New patch

---
 arch/x86/kvm/vmx/tdx.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/kvm/vmx/tdx.c +1 -0
index ea9498028212..a7ebdafdfd82 100644
@@ -553,6 +553,7 @@ int tdx_vm_init(struct kvm *kvm)
 {
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
 
+	kvm->arch.has_protected_state = true;
 	kvm->arch.has_private_mem = true;
 
 	/*
Signature · 1 line
2.43.0
#15 lore raw

Re: [PATCH V2 03/12] KVM: TDX: Set arch.has_protected_state to true

Quoted 41 lines · Adrian Hunter

TDX VMs have protected state. Accordingly, set arch.has_protected_state to true.

This will cause the following IOCTL functions to return an error:

	kvm_arch_vcpu_ioctl()	case KVM_GET_SREGS2
	kvm_arch_vcpu_ioctl()	case KVM_SET_SREGS2
	kvm_arch_vcpu_ioctl_get_regs()
	kvm_arch_vcpu_ioctl_set_regs()
	kvm_arch_vcpu_ioctl_get_sregs()
	kvm_arch_vcpu_ioctl_set_sregs()
	kvm_vcpu_ioctl_x86_get_debugregs()
	kvm_vcpu_ioctl_x86_set_debugregs
	kvm_vcpu_ioctl_x86_get_xcrs()
	kvm_vcpu_ioctl_x86_set_xcrs()

In addition, the following will error for confidential FPU state:

	kvm_vcpu_ioctl_x86_get_xsave ()
	kvm_vcpu_ioctl_x86_get_xsave2()
	kvm_vcpu_ioctl_x86_set_xsave()
	kvm_arch_vcpu_ioctl_get_fpu()
	kvm_arch_vcpu_ioctl_set_fpu()

And finally, in accordance with commit 66155de93bcf ("KVM: x86: Disallow read-only memslots for SEV-ES and SEV-SNP (and TDX)"), read-only memslots will be disallowed.

---
TD vcpu enter/exit v2:
- New patch

---
  arch/x86/kvm/vmx/tdx.c | 1 +
  1 file changed, 1 insertion(+)
arch/x86/kvm/vmx/tdx.c +0 -0
index ea9498028212..a7ebdafdfd82 100644

arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 556

Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -553,6 +553,7 @@ int tdx_vm_init(struct kvm *kvm)
  {
  	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
 
+	kvm->arch.has_protected_state = true;

This can be squashed into the one that implements the tdx_vm_init();

Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +0 -0
  	kvm->arch.has_private_mem = true;
 
  	/*
#16 lore raw

Re: [PATCH V2 03/12] KVM: TDX: Set arch.has_protected_state to true

Quoted 37 lines · Xiaoyao Li
Quoted 36 lines · Adrian Hunter

TDX VMs have protected state. Accordingly, set arch.has_protected_state to true.

This will cause the following IOCTL functions to return an error:

    kvm_arch_vcpu_ioctl()    case KVM_GET_SREGS2
    kvm_arch_vcpu_ioctl()    case KVM_SET_SREGS2
    kvm_arch_vcpu_ioctl_get_regs()
    kvm_arch_vcpu_ioctl_set_regs()
    kvm_arch_vcpu_ioctl_get_sregs()
    kvm_arch_vcpu_ioctl_set_sregs()
    kvm_vcpu_ioctl_x86_get_debugregs()
    kvm_vcpu_ioctl_x86_set_debugregs
    kvm_vcpu_ioctl_x86_get_xcrs()
    kvm_vcpu_ioctl_x86_set_xcrs()

In addition, the following will error for confidential FPU state:

    kvm_vcpu_ioctl_x86_get_xsave ()
    kvm_vcpu_ioctl_x86_get_xsave2()
    kvm_vcpu_ioctl_x86_set_xsave()
    kvm_arch_vcpu_ioctl_get_fpu()
    kvm_arch_vcpu_ioctl_set_fpu()

And finally, in accordance with commit 66155de93bcf ("KVM: x86: Disallow read-only memslots for SEV-ES and SEV-SNP (and TDX)"), read-only memslots will be disallowed.

---
TD vcpu enter/exit v2:
  - New patch
---
  arch/x86/kvm/vmx/tdx.c | 1 +

  1 file changed, 1 insertion(+)
Quoted 10 lines

arch/x86/kvm/vmx/tdx.c int tdx_vm_init(struct kvm *kvm) · line 555

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +0 -0
index ea9498028212..a7ebdafdfd82 100644
@@ -553,6 +553,7 @@ int tdx_vm_init(struct kvm *kvm)
   {
       struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
   +    kvm->arch.has_protected_state = true;

This can be squashed into the one that implements the tdx_vm_init();

This has been done in kvm-coco-queue.
We have re-based on kvm-coco-queue so we in-sync on this.

#17 lore raw

PATCH v2 4/12 KVM: VMX: Move common fields of struct vcpu_{vmx,tdx} to a struct

In reply to Adrian Hunter

From: Binbin Wu <[email protected]>

Move common fields of struct vcpu_vmx and struct vcpu_tdx to struct vcpu_vt, to share the code between VMX/TDX as much as possible and to make TDX exit handling more VMX like.

No functional change intended.

[Adrian: move code that depends on struct vcpu_vmx back to vmx.h]

---
TD vcpu enter/exit v2:
- New patch

---
 arch/x86/kvm/vmx/common.h      |  68 +++++++++++++++++++++
 arch/x86/kvm/vmx/main.c        |   4 ++
 arch/x86/kvm/vmx/nested.c      |  10 ++--
 arch/x86/kvm/vmx/posted_intr.c |  18 +++---
 arch/x86/kvm/vmx/tdx.h         |  16 +----
 arch/x86/kvm/vmx/vmx.c         |  99 +++++++++++++++----------------
 arch/x86/kvm/vmx/vmx.h         | 104 +++++++++++++--------------------
 7 files changed, 178 insertions(+), 141 deletions(-)
arch/x86/kvm/vmx/common.h +68 -0
index 7a592467a044..9d4982694f06 100644
@@ -6,6 +6,74 @@
 
 #include "mmu.h"
 
+union vmx_exit_reason {
+	struct {
+		u32	basic			: 16;
+		u32	reserved16		: 1;
+		u32	reserved17		: 1;
+		u32	reserved18		: 1;
+		u32	reserved19		: 1;
+		u32	reserved20		: 1;
+		u32	reserved21		: 1;
+		u32	reserved22		: 1;
+		u32	reserved23		: 1;
+		u32	reserved24		: 1;
+		u32	reserved25		: 1;
+		u32	bus_lock_detected	: 1;
+		u32	enclave_mode		: 1;
+		u32	smi_pending_mtf		: 1;
+		u32	smi_from_vmx_root	: 1;
+		u32	reserved30		: 1;
+		u32	failed_vmentry		: 1;
+	};
+	u32 full;
+};
+
+struct vcpu_vt {
+	/* Posted interrupt descriptor */
+	struct pi_desc pi_desc;
+
+	/* Used if this vCPU is waiting for PI notification wakeup. */
+	struct list_head pi_wakeup_list;
+
+	union vmx_exit_reason exit_reason;
+
+	unsigned long	exit_qualification;
+	u32		exit_intr_info;
+
+	/*
+	 * If true, guest state has been loaded into hardware, and host state
+	 * saved into vcpu_{vt,vmx,tdx}.  If false, host state is loaded into
+	 * hardware.
+	 */
+	bool		guest_state_loaded;
+
+#ifdef CONFIG_X86_64
+	u64		msr_host_kernel_gs_base;
+#endif
+
+	unsigned long	host_debugctlmsr;
+};
+
+#ifdef CONFIG_INTEL_TDX_HOST
+
+static __always_inline bool is_td(struct kvm *kvm)
+{
+	return kvm->arch.vm_type == KVM_X86_TDX_VM;
+}
+
+static __always_inline bool is_td_vcpu(struct kvm_vcpu *vcpu)
+{
+	return is_td(vcpu->kvm);
+}
+
+#else
+
+static inline bool is_td(struct kvm *kvm) { return false; }
+static inline bool is_td_vcpu(struct kvm_vcpu *vcpu) { return false; }
+
+#endif
+
 static inline bool vt_is_tdx_private_gpa(struct kvm *kvm, gpa_t gpa)
 {
 	/* For TDX the direct mask is the shared mask. */
arch/x86/kvm/vmx/main.c +4 -0
index 0ff7394f8466..1cc1c06461f2 100644
@@ -10,6 +10,10 @@
 #include "tdx.h"
 #include "tdx_arch.h"
 
+#ifdef CONFIG_INTEL_TDX_HOST
+static_assert(offsetof(struct vcpu_vmx, vt) == offsetof(struct vcpu_tdx, vt));
+#endif
+
 static void vt_disable_virtualization_cpu(void)
 {
 	/* Note, TDX *and* VMX need to be disabled if TDX is enabled. */
arch/x86/kvm/vmx/nested.c +5 -5
index 8a7af02d466e..3add9f1073ff 100644
@@ -275,7 +275,7 @@ static void vmx_sync_vmcs_host_state(struct vcpu_vmx *vmx,
 {
 	struct vmcs_host_state *dest, *src;
 
-	if (unlikely(!vmx->guest_state_loaded))
+	if (unlikely(!vmx->vt.guest_state_loaded))
 		return;
 
 	src = &prev->host_state;
@@ -425,7 +425,7 @@ static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu,
 		 * tables also changed, but KVM should not treat EPT Misconfig
 		 * VM-Exits as writes.
 		 */
-		WARN_ON_ONCE(vmx->exit_reason.basic != EXIT_REASON_EPT_VIOLATION);
+		WARN_ON_ONCE(vmx->vt.exit_reason.basic != EXIT_REASON_EPT_VIOLATION);
 
 		/*
 		 * PML Full and EPT Violation VM-Exits both use bit 12 to report
@@ -4622,7 +4622,7 @@ static void prepare_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,
 {
 	/* update exit information fields: */
 	vmcs12->vm_exit_reason = vm_exit_reason;
-	if (to_vmx(vcpu)->exit_reason.enclave_mode)
+	if (vmx_get_exit_reason(vcpu).enclave_mode)
 		vmcs12->vm_exit_reason |= VMX_EXIT_REASONS_SGX_ENCLAVE_MODE;
 	vmcs12->exit_qualification = exit_qualification;
 
@@ -6115,7 +6115,7 @@ static int handle_vmfunc(struct kvm_vcpu *vcpu)
 	 * nested VM-Exit.  Pass the original exit reason, i.e. don't hardcode
 	 * EXIT_REASON_VMFUNC as the exit reason.
 	 */
-	nested_vmx_vmexit(vcpu, vmx->exit_reason.full,
+	nested_vmx_vmexit(vcpu, vmx->vt.exit_reason.full,
 			  vmx_get_intr_info(vcpu),
 			  vmx_get_exit_qual(vcpu));
 	return 1;
@@ -6560,7 +6560,7 @@ static bool nested_vmx_l1_wants_exit(struct kvm_vcpu *vcpu,
 bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
-	union vmx_exit_reason exit_reason = vmx->exit_reason;
+	union vmx_exit_reason exit_reason = vmx->vt.exit_reason;
 	unsigned long exit_qual;
 	u32 exit_intr_info;
 
arch/x86/kvm/vmx/posted_intr.c +9 -9
index ec08fa3caf43..5696e0f9f924 100644
@@ -33,7 +33,7 @@ static DEFINE_PER_CPU(raw_spinlock_t, wakeup_vcpus_on_cpu_lock);
 
 static inline struct pi_desc *vcpu_to_pi_desc(struct kvm_vcpu *vcpu)
 {
-	return &(to_vmx(vcpu)->pi_desc);
+	return &(to_vt(vcpu)->pi_desc);
 }
 
 static int pi_try_set_control(struct pi_desc *pi_desc, u64 *pold, u64 new)
@@ -53,7 +53,7 @@ static int pi_try_set_control(struct pi_desc *pi_desc, u64 *pold, u64 new)
 void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)
 {
 	struct pi_desc *pi_desc = vcpu_to_pi_desc(vcpu);
-	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	struct vcpu_vt *vt = to_vt(vcpu);
 	struct pi_desc old, new;
 	unsigned long flags;
 	unsigned int dest;
@@ -90,7 +90,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)
 	 */
 	if (pi_desc->nv == POSTED_INTR_WAKEUP_VECTOR) {
 		raw_spin_lock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu));
-		list_del(&vmx->pi_wakeup_list);
+		list_del(&vt->pi_wakeup_list);
 		raw_spin_unlock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu));
 	}
 
@@ -146,14 +146,14 @@ static bool vmx_can_use_vtd_pi(struct kvm *kvm)
 static void pi_enable_wakeup_handler(struct kvm_vcpu *vcpu)
 {
 	struct pi_desc *pi_desc = vcpu_to_pi_desc(vcpu);
-	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	struct vcpu_vt *vt = to_vt(vcpu);
 	struct pi_desc old, new;
 	unsigned long flags;
 
 	local_irq_save(flags);
 
 	raw_spin_lock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu));
-	list_add_tail(&vmx->pi_wakeup_list,
+	list_add_tail(&vt->pi_wakeup_list,
 		      &per_cpu(wakeup_vcpus_on_cpu, vcpu->cpu));
 	raw_spin_unlock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu));
 
@@ -220,13 +220,13 @@ void pi_wakeup_handler(void)
 	int cpu = smp_processor_id();
 	struct list_head *wakeup_list = &per_cpu(wakeup_vcpus_on_cpu, cpu);
 	raw_spinlock_t *spinlock = &per_cpu(wakeup_vcpus_on_cpu_lock, cpu);
-	struct vcpu_vmx *vmx;
+	struct vcpu_vt *vt;
 
 	raw_spin_lock(spinlock);
-	list_for_each_entry(vmx, wakeup_list, pi_wakeup_list) {
+	list_for_each_entry(vt, wakeup_list, pi_wakeup_list) {
 
-		if (pi_test_on(&vmx->pi_desc))
-			kvm_vcpu_wake_up(&vmx->vcpu);
+		if (pi_test_on(&vt->pi_desc))
+			kvm_vcpu_wake_up(vt_to_vcpu(vt));
 	}
 	raw_spin_unlock(spinlock);
 }
arch/x86/kvm/vmx/tdx.h +3 -13
index 3904479892f3..ba880dae547f 100644
@@ -6,6 +6,8 @@
 #include "tdx_errno.h"
 
 #ifdef CONFIG_INTEL_TDX_HOST
+#include "common.h"
+
 int tdx_bringup(void);
 void tdx_cleanup(void);
 
@@ -43,6 +45,7 @@ enum vcpu_tdx_state {
 
 struct vcpu_tdx {
 	struct kvm_vcpu	vcpu;
+	struct vcpu_vt vt;
 
 	struct tdx_vp vp;
 
@@ -55,16 +58,6 @@ void tdh_vp_rd_failed(struct vcpu_tdx *tdx, char *uclass, u32 field, u64 err);
 void tdh_vp_wr_failed(struct vcpu_tdx *tdx, char *uclass, char *op, u32 field,
 		      u64 val, u64 err);
 
-static inline bool is_td(struct kvm *kvm)
-{
-	return kvm->arch.vm_type == KVM_X86_TDX_VM;
-}
-
-static inline bool is_td_vcpu(struct kvm_vcpu *vcpu)
-{
-	return is_td(vcpu->kvm);
-}
-
 static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u32 field)
 {
 	u64 err, data;
@@ -174,9 +167,6 @@ struct vcpu_tdx {
 	struct kvm_vcpu	vcpu;
 };
 
-static inline bool is_td(struct kvm *kvm) { return false; }
-static inline bool is_td_vcpu(struct kvm_vcpu *vcpu) { return false; }
-
 #endif
 
 #endif
arch/x86/kvm/vmx/vmx.c +50 -49

316 lines not on this page · raw message

arch/x86/kvm/vmx/vmx.h +39 -65

155 lines not on this page · raw message

Signature · 1 line
2.43.0
#18 lore raw

PATCH v2 5/12 KVM: TDX: Implement TDX vcpu enter/exit path

In reply to Adrian Hunter

From: Isaku Yamahata <[email protected]>

Implement callbacks to enter/exit a TDX VCPU by calling tdh_vp_enter(). Ensure the TDX VCPU is in a correct state to run.

Do not pass arguments from/to vcpu->arch.regs[] unconditionally. Instead, marshall state to/from the appropriate x86 registers only when needed, i.e., to handle some TDVMCALL sub-leaves following KVM's ABI to leverage the existing code.

---
TD vcpu enter/exit v2:
- Move VCPU_TD_STATE_INITIALIZED check to tdx_vcpu_pre_run() (Xiaoyao)
- Check TD_STATE_RUNNABLE also in tdx_vcpu_pre_run() (Yan)
- Add back 'noinstr' for tdx_vcpu_enter_exit() (Sean)
- Add WARN_ON_ONCE if force_immediate_exit (Sean)
- Add vp_enter_args to vcpu_tdx to store the input/output arguments for tdh_vp_enter().
- Don't copy arguments to/from vcpu->arch.regs[] unconditionally. (Sean)

TD vcpu enter/exit v1:
- Make argument of tdx_vcpu_enter_exit() struct kvm_vcpu.
- Update for the wrapper functions for SEAMCALLs. (Sean)
- Remove noinstr (Sean)
- Add a missing comma, clarify sched_in part, and update changelog to match code by dropping the PMU related paragraph (Binbin)
https://lore.kernel.org/lkml/[email protected]/
- Remove the union tdx_exit_reason. (Sean)
https://lore.kernel.org/kvm/[email protected]/
- Remove the code of special handling of vcpu->kvm->vm_bugged (Rick)
https://lore.kernel.org/kvm/[email protected]/
- For !tdx->initialized case, set tdx->vp_enter_ret to TDX_SW_ERROR to avoid collision with EXIT_REASON_EXCEPTION_NMI.

v19:
- Removed export_symbol_gpl(host_xcr0) to the patch that uses it

Changes v15 -> v16:
- use __seamcall_saved_ret()
- As struct tdx_module_args doesn't match with vcpu.arch.regs, copy regs before/after calling __seamcall_saved_ret().

---
 arch/x86/kvm/vmx/main.c    | 20 ++++++++++++++--
 arch/x86/kvm/vmx/tdx.c     | 47 ++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx.h     |  3 +++
 arch/x86/kvm/vmx/x86_ops.h |  7 ++++++
 4 files changed, 75 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +18 -2
index 1cc1c06461f2..301c1a26606f 100644
@@ -133,6 +133,22 @@ static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	vmx_vcpu_load(vcpu, cpu);
 }
 
+static int vt_vcpu_pre_run(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_pre_run(vcpu);
+
+	return vmx_vcpu_pre_run(vcpu);
+}
+
+static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_run(vcpu, force_immediate_exit);
+
+	return vmx_vcpu_run(vcpu, force_immediate_exit);
+}
+
 static void vt_flush_tlb_all(struct kvm_vcpu *vcpu)
 {
 	if (is_td_vcpu(vcpu)) {
@@ -272,8 +288,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.flush_tlb_gva = vt_flush_tlb_gva,
 	.flush_tlb_guest = vt_flush_tlb_guest,
 
-	.vcpu_pre_run = vmx_vcpu_pre_run,
-	.vcpu_run = vmx_vcpu_run,
+	.vcpu_pre_run = vt_vcpu_pre_run,
+	.vcpu_run = vt_vcpu_run,
 	.handle_exit = vmx_handle_exit,
 	.skip_emulated_instruction = vmx_skip_emulated_instruction,
 	.update_emulated_instruction = vmx_update_emulated_instruction,
arch/x86/kvm/vmx/tdx.c +47 -0

59 lines not on this page · raw message

arch/x86/kvm/vmx/tdx.h +3 -0
index ba880dae547f..8339bbf0fdd4 100644
@@ -46,11 +46,14 @@ enum vcpu_tdx_state {
 struct vcpu_tdx {
 	struct kvm_vcpu	vcpu;
 	struct vcpu_vt vt;
+	struct tdx_module_args vp_enter_args;
 
 	struct tdx_vp vp;
 
 	struct list_head cpu_list;
 
+	u64 vp_enter_ret;
+
 	enum vcpu_tdx_state state;
 };
 
arch/x86/kvm/vmx/x86_ops.h +7 -0
index ff6370787926..83aac44b779b 100644
@@ -131,6 +131,8 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);
 int tdx_vcpu_create(struct kvm_vcpu *vcpu);
 void tdx_vcpu_free(struct kvm_vcpu *vcpu);
 void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
+int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu);
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit);
 
 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp);
 
@@ -158,6 +160,11 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOP
 static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; }
 static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {}
 static inline void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {}
+static inline int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; }
+static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
+{
+	return EXIT_FASTPATH_NONE;
+}
 
 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) { return -EOPNOTSUPP; }
 
Signature · 1 line
2.43.0
#19 lore raw

Re: [PATCH V2 05/12] KVM: TDX: Implement TDX vcpu enter/exit path

Quoted 10 lines · Adrian Hunter
arch/x86/kvm/vmx/tdx.c +10 -0
+#define TDX_REGS_UNSUPPORTED_SET	(BIT(VCPU_EXREG_RFLAGS) |	\
+					 BIT(VCPU_EXREG_SEGMENTS))
+
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
+{
+	/*
+	 * force_immediate_exit requires vCPU entering for events injection with
+	 * an immediately exit followed. But The TDX module doesn't guarantee
+	 * entry, it's already possible for KVM to_think_ it completely entry
+	 * to the guest without actually having done so.

arch/x86/kvm/vmx/tdx.c void tdx_vcpu_free(struct kvm_vcpu *vcpu) · line 717

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+	 * Since KVM never needs to force an immediate exit for TDX, and can't
+	 * do direct injection, just warn on force_immediate_exit.
+	 */
+	WARN_ON_ONCE(force_immediate_exit);
+
+	trace_kvm_entry(vcpu, force_immediate_exit);
+
+	tdx_vcpu_enter_exit(vcpu);
+
+	vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;

I don't understand this. Why only clear RFLAGS and SEGMENTS?

When creating the vcpu, vcpu->arch.regs_avail = ~0 in kvm_arch_vcpu_create().

now it only clears RFLAGS and SEGMENTS for TDX vcpu, which leaves other bits set. But I don't see any code that syncs the guest value of into vcpu->arch.regs[reg].

Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+	trace_kvm_exit(vcpu, KVM_ISA_VMX);
+
+	return EXIT_FASTPATH_NONE;
+}
#20 lore raw

Re: [PATCH V2 05/12] KVM: TDX: Implement TDX vcpu enter/exit path

Quoted 24 lines · Xiaoyao Li
Quoted 10 lines · Adrian Hunter
arch/x86/kvm/vmx/tdx.c +10 -0
+#define TDX_REGS_UNSUPPORTED_SET    (BIT(VCPU_EXREG_RFLAGS) |    \
+                     BIT(VCPU_EXREG_SEGMENTS))
+
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
+{
+    /*
+     * force_immediate_exit requires vCPU entering for events injection with
+     * an immediately exit followed. But The TDX module doesn't guarantee
+     * entry, it's already possible for KVM to_think_ it completely entry
+     * to the guest without actually having done so.

arch/x86/kvm/vmx/tdx.c void tdx_vcpu_free(struct kvm_vcpu *vcpu) · line 717

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+     * Since KVM never needs to force an immediate exit for TDX, and can't
+     * do direct injection, just warn on force_immediate_exit.
+     */
+    WARN_ON_ONCE(force_immediate_exit);
+
+    trace_kvm_entry(vcpu, force_immediate_exit);
+
+    tdx_vcpu_enter_exit(vcpu);
+
+    vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;

I don't understand this. Why only clear RFLAGS and SEGMENTS?

Quoted 3 lines

When creating the vcpu, vcpu->arch.regs_avail = ~0 in kvm_arch_vcpu_create().

now it only clears RFLAGS and SEGMENTS for TDX vcpu, which leaves other bits set. But I don't see any code that syncs the guest value of into vcpu->arch.regs[reg].

TDX guest registers are generally not known but values are placed into vcpu->arch.regs when needed to work with common code.

We used to use ~VMX_REGS_LAZY_LOAD_SET and tdx_cache_reg() which has since been removed.

tdx_cache_reg() did not support RFLAGS, SEGMENTS, EXIT_INFO_1/EXIT_INFO_2 but EXIT_INFO_1/EXIT_INFO_2 became needed, so that just left RFLAGS, SEGMENTS.

Quoted 6 lines
Quoted 5 lines

+    trace_kvm_exit(vcpu, KVM_ISA_VMX);

arch/x86/kvm/vmx/tdx.c +3 -0
+
+    return EXIT_FASTPATH_NONE;
+}
#21 lore raw

Re: [PATCH V2 05/12] KVM: TDX: Implement TDX vcpu enter/exit path

Quoted 33 lines · Adrian Hunter
Quoted 24 lines · Xiaoyao Li
Quoted 10 lines · Adrian Hunter
arch/x86/kvm/vmx/tdx.c +10 -0
+#define TDX_REGS_UNSUPPORTED_SET    (BIT(VCPU_EXREG_RFLAGS) |    \
+                     BIT(VCPU_EXREG_SEGMENTS))
+
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
+{
+    /*
+     * force_immediate_exit requires vCPU entering for events injection with
+     * an immediately exit followed. But The TDX module doesn't guarantee
+     * entry, it's already possible for KVM to_think_ it completely entry
+     * to the guest without actually having done so.

arch/x86/kvm/vmx/tdx.c void tdx_vcpu_free(struct kvm_vcpu *vcpu) · line 717

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+     * Since KVM never needs to force an immediate exit for TDX, and can't
+     * do direct injection, just warn on force_immediate_exit.
+     */
+    WARN_ON_ONCE(force_immediate_exit);
+
+    trace_kvm_entry(vcpu, force_immediate_exit);
+
+    tdx_vcpu_enter_exit(vcpu);
+
+    vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;

I don't understand this. Why only clear RFLAGS and SEGMENTS?

Quoted 3 lines

When creating the vcpu, vcpu->arch.regs_avail = ~0 in kvm_arch_vcpu_create().

now it only clears RFLAGS and SEGMENTS for TDX vcpu, which leaves other bits set. But I don't see any code that syncs the guest value of into vcpu->arch.regs[reg].

TDX guest registers are generally not known but values are placed into vcpu->arch.regs when needed to work with common code.

Quoted 6 lines

We used to use ~VMX_REGS_LAZY_LOAD_SET and tdx_cache_reg() which has since been removed.

tdx_cache_reg() did not support RFLAGS, SEGMENTS, EXIT_INFO_1/EXIT_INFO_2 but EXIT_INFO_1/EXIT_INFO_2 became needed, so that just left RFLAGS, SEGMENTS.

Quote what Sean said [1]

“I'm also not convinced letting KVM read garbage for RIP, RSP, CR3, or PDPTRs is at all reasonable. CR3 and PDPTRs should be unreachable, and I gotta imagine the same holds true for RSP. Allow reads/writes to RIP is fine, in that it probably simplifies the overall code.”

We need to justify why to let KVM read "garbage" of VCPU_REGS_RIP, VCPU_EXREG_PDPTR, VCPU_EXREG_CR0, VCPU_EXREG_CR3, VCPU_EXREG_CR4, VCPU_EXREG_EXIT_INFO_1, and VCPU_EXREG_EXIT_INFO_2 are neeed.

The changelog justify nothing for it.

btw, how EXIT_INFO_1/EXIT_INFO_2 became needed? It seems I cannot find any TDX code use them.

[1] https://lore.kernel.org/all/[email protected]/

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines

+    trace_kvm_exit(vcpu, KVM_ISA_VMX);

arch/x86/kvm/vmx/tdx.c +3 -0
+
+    return EXIT_FASTPATH_NONE;
+}
#22 lore raw

Re: [PATCH V2 05/12] KVM: TDX: Implement TDX vcpu enter/exit path

Quoted 48 lines · Xiaoyao Li
Quoted 33 lines · Adrian Hunter
Quoted 24 lines · Xiaoyao Li
Quoted 10 lines · Adrian Hunter
arch/x86/kvm/vmx/tdx.c +10 -0
+#define TDX_REGS_UNSUPPORTED_SET    (BIT(VCPU_EXREG_RFLAGS) |    \
+                     BIT(VCPU_EXREG_SEGMENTS))
+
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
+{
+    /*
+     * force_immediate_exit requires vCPU entering for events injection with
+     * an immediately exit followed. But The TDX module doesn't guarantee
+     * entry, it's already possible for KVM to_think_ it completely entry
+     * to the guest without actually having done so.

arch/x86/kvm/vmx/tdx.c void tdx_vcpu_free(struct kvm_vcpu *vcpu) · line 717

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+     * Since KVM never needs to force an immediate exit for TDX, and can't
+     * do direct injection, just warn on force_immediate_exit.
+     */
+    WARN_ON_ONCE(force_immediate_exit);
+
+    trace_kvm_entry(vcpu, force_immediate_exit);
+
+    tdx_vcpu_enter_exit(vcpu);
+
+    vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;

I don't understand this. Why only clear RFLAGS and SEGMENTS?

Quoted 3 lines

When creating the vcpu, vcpu->arch.regs_avail = ~0 in kvm_arch_vcpu_create().

now it only clears RFLAGS and SEGMENTS for TDX vcpu, which leaves other bits set. But I don't see any code that syncs the guest value of into vcpu->arch.regs[reg].

TDX guest registers are generally not known but values are placed into vcpu->arch.regs when needed to work with common code.

Quoted 6 lines

We used to use ~VMX_REGS_LAZY_LOAD_SET and tdx_cache_reg() which has since been removed.

tdx_cache_reg() did not support RFLAGS, SEGMENTS, EXIT_INFO_1/EXIT_INFO_2 but EXIT_INFO_1/EXIT_INFO_2 became needed, so that just left RFLAGS, SEGMENTS.

Quote what Sean said [1]

  “I'm also not convinced letting KVM read garbage for RIP, RSP, CR3, or PDPTRs is at all reasonable.  CR3 and PDPTRs should be unreachable, and I gotta imagine the same holds true for RSP.  Allow reads/writes to RIP is fine, in that it probably simplifies the overall code.”

Quoted 5 lines

We need to justify why to let KVM read "garbage" of VCPU_REGS_RIP, VCPU_EXREG_PDPTR, VCPU_EXREG_CR0, VCPU_EXREG_CR3, VCPU_EXREG_CR4, VCPU_EXREG_EXIT_INFO_1, and VCPU_EXREG_EXIT_INFO_2 are neeed.

The changelog justify nothing for it.

Could add VCPU_REGS_RIP, VCPU_REGS_RSP, VCPU_EXREG_CR3, VCPU_EXREG_PDPTR. But not VCPU_EXREG_CR0 nor VCPU_EXREG_CR4 since we started using them.

Quoted 2 lines

btw, how EXIT_INFO_1/EXIT_INFO_2 became needed? It seems I cannot find any TDX code use them.

vmx_get_exit_qual() / vmx_get_intr_info() are now used by TDX.

Quoted 11 lines

[1] https://lore.kernel.org/all/[email protected]/

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines

+    trace_kvm_exit(vcpu, KVM_ISA_VMX);

arch/x86/kvm/vmx/tdx.c +3 -0
+
+    return EXIT_FASTPATH_NONE;
+}
maintainer
#23 lore raw

Re: [PATCH V2 05/12] KVM: TDX: Implement TDX vcpu enter/exit path

Quoted 49 lines · Adrian Hunter
Quoted 48 lines · Xiaoyao Li
Quoted 33 lines · Adrian Hunter
Quoted 24 lines · Xiaoyao Li
Quoted 10 lines · Adrian Hunter
arch/x86/kvm/vmx/tdx.c +10 -0
+#define TDX_REGS_UNSUPPORTED_SET    (BIT(VCPU_EXREG_RFLAGS) |    \
+                     BIT(VCPU_EXREG_SEGMENTS))
+
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
+{
+    /*
+     * force_immediate_exit requires vCPU entering for events injection with
+     * an immediately exit followed. But The TDX module doesn't guarantee
+     * entry, it's already possible for KVM to_think_ it completely entry
+     * to the guest without actually having done so.

arch/x86/kvm/vmx/tdx.c void tdx_vcpu_free(struct kvm_vcpu *vcpu) · line 717

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+     * Since KVM never needs to force an immediate exit for TDX, and can't
+     * do direct injection, just warn on force_immediate_exit.
+     */
+    WARN_ON_ONCE(force_immediate_exit);
+
+    trace_kvm_entry(vcpu, force_immediate_exit);
+
+    tdx_vcpu_enter_exit(vcpu);
+
+    vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;

I don't understand this. Why only clear RFLAGS and SEGMENTS?

Quoted 3 lines

When creating the vcpu, vcpu->arch.regs_avail = ~0 in kvm_arch_vcpu_create().

now it only clears RFLAGS and SEGMENTS for TDX vcpu, which leaves other bits set. But I don't see any code that syncs the guest value of into vcpu->arch.regs[reg].

TDX guest registers are generally not known but values are placed into vcpu->arch.regs when needed to work with common code.

Quoted 6 lines

We used to use ~VMX_REGS_LAZY_LOAD_SET and tdx_cache_reg() which has since been removed.

tdx_cache_reg() did not support RFLAGS, SEGMENTS, EXIT_INFO_1/EXIT_INFO_2 but EXIT_INFO_1/EXIT_INFO_2 became needed, so that just left RFLAGS, SEGMENTS.

Quote what Sean said [1]

  “I'm also not convinced letting KVM read garbage for RIP, RSP, CR3, or PDPTRs is at all reasonable.  CR3 and PDPTRs should be unreachable, and I gotta imagine the same holds true for RSP.  Allow reads/writes to RIP is fine, in that it probably simplifies the overall code.”

Quoted 8 lines
Quoted 5 lines

We need to justify why to let KVM read "garbage" of VCPU_REGS_RIP, VCPU_EXREG_PDPTR, VCPU_EXREG_CR0, VCPU_EXREG_CR3, VCPU_EXREG_CR4, VCPU_EXREG_EXIT_INFO_1, and VCPU_EXREG_EXIT_INFO_2 are neeed.

The changelog justify nothing for it.

Could add VCPU_REGS_RIP, VCPU_REGS_RSP, VCPU_EXREG_CR3, VCPU_EXREG_PDPTR. But not VCPU_EXREG_CR0 nor VCPU_EXREG_CR4 since we started using them.

Hi Adrian,

how is CR0 used? And CR4 is only used other than for loading the XSAVE state, I think?

I will change this to a list of specific available registers instead of using "&= ~", and it would be even better if CR0/CR4 are not on the list.

Paolo

Quoted 17 lines
Quoted 1 line

btw, how EXIT_INFO_1/EXIT_INFO_2 became needed? It seems I cannot find any TDX code use them.

vmx_get_exit_qual() / vmx_get_intr_info() are now used by TDX.

Quoted 11 lines

[1] https://lore.kernel.org/all/[email protected]/

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines

+    trace_kvm_exit(vcpu, KVM_ISA_VMX);

arch/x86/kvm/vmx/tdx.c +3 -0
+
+    return EXIT_FASTPATH_NONE;
+}
#24 lore raw

Re: [PATCH V2 05/12] KVM: TDX: Implement TDX vcpu enter/exit path

Quoted 54 lines · Paolo Bonzini
Quoted 53 lines · Adrian Hunter
Quoted 52 lines · Xiaoyao Li
Quoted 33 lines · Adrian Hunter
Quoted 24 lines · Xiaoyao Li
Quoted 10 lines · Adrian Hunter
arch/x86/kvm/vmx/tdx.c +10 -0
+#define TDX_REGS_UNSUPPORTED_SET    (BIT(VCPU_EXREG_RFLAGS) |    \
+                     BIT(VCPU_EXREG_SEGMENTS))
+
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
+{
+    /*
+     * force_immediate_exit requires vCPU entering for events injection with
+     * an immediately exit followed. But The TDX module doesn't guarantee
+     * entry, it's already possible for KVM to_think_ it completely entry
+     * to the guest without actually having done so.

arch/x86/kvm/vmx/tdx.c void tdx_vcpu_free(struct kvm_vcpu *vcpu) · line 717

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+     * Since KVM never needs to force an immediate exit for TDX, and can't
+     * do direct injection, just warn on force_immediate_exit.
+     */
+    WARN_ON_ONCE(force_immediate_exit);
+
+    trace_kvm_entry(vcpu, force_immediate_exit);
+
+    tdx_vcpu_enter_exit(vcpu);
+
+    vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;

I don't understand this. Why only clear RFLAGS and SEGMENTS?

Quoted 3 lines

When creating the vcpu, vcpu->arch.regs_avail = ~0 in kvm_arch_vcpu_create().

now it only clears RFLAGS and SEGMENTS for TDX vcpu, which leaves other bits set. But I don't see any code that syncs the guest value of into vcpu->arch.regs[reg].

TDX guest registers are generally not known but values are placed into vcpu->arch.regs when needed to work with common code.

Quoted 6 lines

We used to use ~VMX_REGS_LAZY_LOAD_SET and tdx_cache_reg() which has since been removed.

tdx_cache_reg() did not support RFLAGS, SEGMENTS, EXIT_INFO_1/EXIT_INFO_2 but EXIT_INFO_1/EXIT_INFO_2 became needed, so that just left RFLAGS, SEGMENTS.

Quote what Sean said [1]

   “I'm also not convinced letting KVM read garbage for RIP, RSP, CR3, or PDPTRs is at all reasonable.  CR3 and PDPTRs should be unreachable, and I gotta imagine the same holds true for RSP.  Allow reads/writes to RIP is fine, in that it probably simplifies the overall code.”

We need to justify why to let KVM read "garbage" of VCPU_REGS_RIP, VCPU_EXREG_PDPTR, VCPU_EXREG_CR0, VCPU_EXREG_CR3, VCPU_EXREG_CR4, VCPU_EXREG_EXIT_INFO_1, and VCPU_EXREG_EXIT_INFO_2 are neeed.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

The changelog justify nothing for it.

Could add VCPU_REGS_RIP, VCPU_REGS_RSP, VCPU_EXREG_CR3, VCPU_EXREG_PDPTR. But not VCPU_EXREG_CR0 nor VCPU_EXREG_CR4 since we started using them.

Hi Adrian,

how is CR0 used? And CR4 is only used other than for loading the XSAVE state, I think?

I meant it is used in the sense that patch "[PATCH V2 07/12] KVM: TDX:
restore host xsave state when exit from the guest TD" provides a value for it.

But it looks like it might be accessible via:

	store_regs()
		__get_sregs()
			__get_sregs_common()

Sean wanted a maximal CR0 value consistent with the CR4.

CR4 is also being used in kvm_update_cpuid_runtime().

Quoted 23 lines

I will change this to a list of specific available registers instead of using "&= ~", and it would be even better if CR0/CR4 are not on the list.

Paolo

Quoted 17 lines
Quoted 1 line

btw, how EXIT_INFO_1/EXIT_INFO_2 became needed? It seems I cannot find any TDX code use them.

vmx_get_exit_qual() / vmx_get_intr_info() are now used by TDX.

Quoted 11 lines

[1] https://lore.kernel.org/all/[email protected]/

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines

+    trace_kvm_exit(vcpu, KVM_ISA_VMX);

arch/x86/kvm/vmx/tdx.c +3 -0
+
+    return EXIT_FASTPATH_NONE;
+}
#25 lore raw

PATCH v2 6/12 KVM: TDX: vcpu_run: save/restore host state(host kernel gs)

In reply to Adrian Hunter

From: Isaku Yamahata <[email protected]>

On entering/exiting TDX vcpu, preserved or clobbered CPU state is different from the VMX case. Add TDX hooks to save/restore host/guest CPU state. Save/restore kernel GS base MSR.

---
TD vcpu enter/exit v2:
- Use 1 variable named 'guest_state_loaded' to track host state save/restore (Sean)
- Rebased due to moving guest_state_loaded/msr_host_kernel_gs_base to struct vcpu_vt.

TD vcpu enter/exit v1:
- Clarify comment (Binbin)
- Use lower case preserved and add the for VMX in log (Tony)
- Fix bisectability issue with includes (Kai)

---
 arch/x86/kvm/vmx/main.c    | 24 +++++++++++++++++++++--
 arch/x86/kvm/vmx/tdx.c     | 40 ++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/x86_ops.h |  4 ++++
 3 files changed, 66 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/main.c +22 -2
index 301c1a26606f..341aa537ca72 100644
@@ -133,6 +133,26 @@ static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	vmx_vcpu_load(vcpu, cpu);
 }
 
+static void vt_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu)) {
+		tdx_prepare_switch_to_guest(vcpu);
+		return;
+	}
+
+	vmx_prepare_switch_to_guest(vcpu);
+}
+
+static void vt_vcpu_put(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu)) {
+		tdx_vcpu_put(vcpu);
+		return;
+	}
+
+	vmx_vcpu_put(vcpu);
+}
+
 static int vt_vcpu_pre_run(struct kvm_vcpu *vcpu)
 {
 	if (is_td_vcpu(vcpu))
@@ -253,9 +273,9 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.vcpu_free = vt_vcpu_free,
 	.vcpu_reset = vt_vcpu_reset,
 
-	.prepare_switch_to_guest = vmx_prepare_switch_to_guest,
+	.prepare_switch_to_guest = vt_prepare_switch_to_guest,
 	.vcpu_load = vt_vcpu_load,
-	.vcpu_put = vmx_vcpu_put,
+	.vcpu_put = vt_vcpu_put,
 
 	.update_exception_bitmap = vmx_update_exception_bitmap,
 	.get_feature_msr = vmx_get_feature_msr,
arch/x86/kvm/vmx/tdx.c +40 -0
index 95420ffd0022..3f3d61935a58 100644
@@ -2,6 +2,7 @@
 #include <linux/cleanup.h>
 #include <linux/cpu.h>
 #include <asm/cpufeature.h>
+#include <linux/mmu_context.h>
 #include <asm/tdx.h>
 #include "capabilities.h"
 #include "mmu.h"
@@ -11,6 +12,7 @@
 #include "vmx.h"
 #include "mmu/spte.h"
 #include "common.h"
+#include "posted_intr.h"
 #include <trace/events/kvm.h>
 #include "trace.h"
 
@@ -642,6 +644,44 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	local_irq_enable();
 }
 
+/*
+ * Compared to vmx_prepare_switch_to_guest(), there is not much to do
+ * as SEAMCALL/SEAMRET calls take care of most of save and restore.
+ */
+void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_vt *vt = to_vt(vcpu);
+
+	if (vt->guest_state_loaded)
+		return;
+
+	if (likely(is_64bit_mm(current->mm)))
+		vt->msr_host_kernel_gs_base = current->thread.gsbase;
+	else
+		vt->msr_host_kernel_gs_base = read_msr(MSR_KERNEL_GS_BASE);
+
+	vt->guest_state_loaded = true;
+}
+
+static void tdx_prepare_switch_to_host(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_vt *vt = to_vt(vcpu);
+
+	if (!vt->guest_state_loaded)
+		return;
+
+	++vcpu->stat.host_state_reload;
+	wrmsrl(MSR_KERNEL_GS_BASE, vt->msr_host_kernel_gs_base);
+
+	vt->guest_state_loaded = false;
+}
+
+void tdx_vcpu_put(struct kvm_vcpu *vcpu)
+{
+	vmx_vcpu_pi_put(vcpu);
+	tdx_prepare_switch_to_host(vcpu);
+}
+
 void tdx_vcpu_free(struct kvm_vcpu *vcpu)
 {
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
arch/x86/kvm/vmx/x86_ops.h +4 -0
index 83aac44b779b..f856eac8f1e8 100644
@@ -133,6 +133,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu);
 void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
 int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu);
 fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit);
+void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);
+void tdx_vcpu_put(struct kvm_vcpu *vcpu);
 
 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp);
 
@@ -165,6 +167,8 @@ static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediat
 {
 	return EXIT_FASTPATH_NONE;
 }
+static inline void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) {}
+static inline void tdx_vcpu_put(struct kvm_vcpu *vcpu) {}
 
 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) { return -EOPNOTSUPP; }
 
Signature · 1 line
2.43.0
#26 lore raw

PATCH v2 7/12 KVM: TDX: restore host xsave state when exit from the guest TD

In reply to Adrian Hunter

From: Isaku Yamahata <[email protected]>

On exiting from the guest TD, xsave state is clobbered. Restore xsave state on TD exit.

Set up guest state so that existing kvm_load_host_xsave_state() can be used. Do not allow VCPU entry if guest state conflicts with the TD's configuration.

---
TD vcpu enter/exit v2:
- Drop PT and CET feature flags (Chao)
- Use cpu_feature_enabled() instead of static_cpu_has() (Chao)
- Restore PKRU only if the host value differs from defined exit value (Chao)
- Use defined masks to separate XFAM bits into XCR0/XSS (Adrian)
- Use existing kvm_load_host_xsave_state() in place of tdx_restore_host_xsave_state() by defining guest CR4, XCR0, XSS and PKRU (Sean)
- Do not enter if vital guest state is invalid (Adrian)

TD vcpu enter/exit v1:
- Remove noinstr on tdx_vcpu_enter_exit() (Sean)
- Switch to kvm_host struct for xcr0 and xss

v19:
- Add EXPORT_SYMBOL_GPL(host_xcr0)

v15 -> v16:
- Added CET flag mask

---
 arch/x86/kvm/vmx/tdx.c | 72 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 66 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +66 -6
index 3f3d61935a58..e4355553569a 100644
@@ -83,16 +83,21 @@ static u64 tdx_get_supported_attrs(const struct tdx_sys_info_td_conf *td_conf)
 	return val;
 }
 
+/*
+ * Before returning from TDH.VP.ENTER, the TDX Module assigns:
+ *   XCR0 to the TD’s user-mode feature bits of XFAM (bits 7:0, 9, 18:17)
+ *   IA32_XSS to the TD's supervisor-mode feature bits of XFAM (bits 8, 16:10)
+ */
+#define TDX_XFAM_XCR0_MASK	(GENMASK(7, 0) | BIT(9) | GENMASK(18, 17))
+#define TDX_XFAM_XSS_MASK	(BIT(8) | GENMASK(16, 10))
+#define TDX_XFAM_MASK		(TDX_XFAM_XCR0_MASK | TDX_XFAM_XSS_MASK)
+
 static u64 tdx_get_supported_xfam(const struct tdx_sys_info_td_conf *td_conf)
 {
 	u64 val = kvm_caps.supported_xcr0 | kvm_caps.supported_xss;
 
-	/*
-	 * PT and CET can be exposed to TD guest regardless of KVM's XSS, PT
-	 * and, CET support.
-	 */
-	val |= XFEATURE_MASK_PT | XFEATURE_MASK_CET_USER |
-	       XFEATURE_MASK_CET_KERNEL;
+	/* Ensure features are in the masks */
+	val &= TDX_XFAM_MASK;
 
 	if ((val & td_conf->xfam_fixed1) != td_conf->xfam_fixed1)
 		return 0;
@@ -724,6 +729,19 @@ int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static bool tdx_guest_state_is_invalid(struct kvm_vcpu *vcpu)
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
+
+	return vcpu->arch.xcr0 != (kvm_tdx->xfam & TDX_XFAM_XCR0_MASK) ||
+	       vcpu->arch.ia32_xss != (kvm_tdx->xfam & TDX_XFAM_XSS_MASK) ||
+	       vcpu->arch.pkru ||
+	       (cpu_feature_enabled(X86_FEATURE_XSAVE) &&
+		!kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) ||
+	       (cpu_feature_enabled(X86_FEATURE_XSAVES) &&
+		!guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES));
+}
+
 static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_tdx *tdx = to_tdx(vcpu);
@@ -740,6 +758,8 @@ static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu)
 
 fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
 {
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+
 	/*
 	 * force_immediate_exit requires vCPU entering for events injection with
 	 * an immediately exit followed. But The TDX module doesn't guarantee
@@ -750,10 +770,22 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
 	 */
 	WARN_ON_ONCE(force_immediate_exit);
 
+	if (WARN_ON_ONCE(tdx_guest_state_is_invalid(vcpu))) {
+		/*
+		 * Invalid exit_reason becomes KVM_EXIT_INTERNAL_ERROR, refer
+		 * tdx_handle_exit().
+		 */
+		tdx->vt.exit_reason.full = -1u;
+		tdx->vp_enter_ret = -1u;
+		return EXIT_FASTPATH_NONE;
+	}
+
 	trace_kvm_entry(vcpu, force_immediate_exit);
 
 	tdx_vcpu_enter_exit(vcpu);
 
+	kvm_load_host_xsave_state(vcpu);
+
 	vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;
 
 	trace_kvm_exit(vcpu, KVM_ISA_VMX);
@@ -1878,9 +1910,23 @@ static int tdx_vcpu_get_cpuid(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
 	return r;
 }
 
+static u64 tdx_guest_cr0(struct kvm_vcpu *vcpu, u64 cr4)
+{
+	u64 cr0 = ~CR0_RESERVED_BITS;
+
+	if (cr4 & X86_CR4_CET)
+		cr0 |= X86_CR0_WP;
+
+	cr0 |= X86_CR0_PE | X86_CR0_NE;
+	cr0 &= ~(X86_CR0_NW | X86_CR0_CD);
+
+	return cr0;
+}
+
 static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
 {
 	u64 apic_base;
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
 	struct vcpu_tdx *tdx = to_tdx(vcpu);
 	int ret;
 
@@ -1903,6 +1949,20 @@ static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
 	if (ret)
 		return ret;
 
+	vcpu->arch.cr4 = ~vcpu->arch.cr4_guest_rsvd_bits;
+	vcpu->arch.cr0 = tdx_guest_cr0(vcpu, vcpu->arch.cr4);
+	/*
+	 * On return from VP.ENTER, the TDX Module sets XCR0 and XSS to the
+	 * maximal values supported by the guest, and zeroes PKRU, so from
+	 * KVM's perspective, those are the guest's values at all times.
+	 */
+	vcpu->arch.ia32_xss = kvm_tdx->xfam & TDX_XFAM_XSS_MASK;
+	vcpu->arch.xcr0 = kvm_tdx->xfam & TDX_XFAM_XCR0_MASK;
+	vcpu->arch.pkru = 0;
+
+	/* TODO: freeze vCPU model before kvm_update_cpuid_runtime() */
+	kvm_update_cpuid_runtime(vcpu);
+
 	tdx->state = VCPU_TD_STATE_INITIALIZED;
 
 	return 0;
Signature · 1 line
2.43.0
#27 lore raw

Re: [PATCH V2 07/12] KVM: TDX: restore host xsave state when exit from the guest TD

Quoted 42 lines · Adrian Hunter

From: Isaku Yamahata <[email protected]>

On exiting from the guest TD, xsave state is clobbered. Restore xsave state on TD exit.

Set up guest state so that existing kvm_load_host_xsave_state() can be used. Do not allow VCPU entry if guest state conflicts with the TD's configuration.

---
TD vcpu enter/exit v2:
- Drop PT and CET feature flags (Chao)
- Use cpu_feature_enabled() instead of static_cpu_has() (Chao)
- Restore PKRU only if the host value differs from defined exit value (Chao)
- Use defined masks to separate XFAM bits into XCR0/XSS (Adrian)
- Use existing kvm_load_host_xsave_state() in place of tdx_restore_host_xsave_state() by defining guest CR4, XCR0, XSS and PKRU (Sean)
- Do not enter if vital guest state is invalid (Adrian)

TD vcpu enter/exit v1:
- Remove noinstr on tdx_vcpu_enter_exit() (Sean)
- Switch to kvm_host struct for xcr0 and xss

v19:
- Add EXPORT_SYMBOL_GPL(host_xcr0)

v15 -> v16:
- Added CET flag mask

---
  arch/x86/kvm/vmx/tdx.c | 72 ++++++++++++++++++++++++++++++++++++++----
  1 file changed, 66 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +0 -0
index 3f3d61935a58..e4355553569a 100644
@@ -83,16 +83,21 @@ static u64 tdx_get_supported_attrs(const struct tdx_sys_info_td_conf *td_conf)
  	return val;

arch/x86/kvm/vmx/tdx.c static u64 tdx_get_supported_attrs(const struct tdx_sys_info_td_conf *td_conf) · line 93

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +8 -0
  }
 
+/*
+ * Before returning from TDH.VP.ENTER, the TDX Module assigns:
+ *   XCR0 to the TD’s user-mode feature bits of XFAM (bits 7:0, 9, 18:17)
+ *   IA32_XSS to the TD's supervisor-mode feature bits of XFAM (bits 8, 16:10)
+ */
+#define TDX_XFAM_XCR0_MASK	(GENMASK(7, 0) | BIT(9) | GENMASK(18, 17))
+#define TDX_XFAM_XSS_MASK	(BIT(8) | GENMASK(16, 10))
+#define TDX_XFAM_MASK		(TDX_XFAM_XCR0_MASK | TDX_XFAM_XSS_MASK)

No need to define TDX-specific mask for XFAM. kernel defines XFEATURE_MASK_* and you can define XFEATURE_XCR0_MASK and XFEATURE_XSS_MASK with XFEATURE_MASK_*.

arch/x86/kvm/vmx/tdx.c static u64 tdx_get_supported_attrs(const struct tdx_sys_info_td_conf *td_conf) · line 100

Quoted 12 lines
arch/x86/kvm/vmx/tdx.c +2 -6
  static u64 tdx_get_supported_xfam(const struct tdx_sys_info_td_conf *td_conf)
  {
  	u64 val = kvm_caps.supported_xcr0 | kvm_caps.supported_xss;
 
-	/*
-	 * PT and CET can be exposed to TD guest regardless of KVM's XSS, PT
-	 * and, CET support.
-	 */
-	val |= XFEATURE_MASK_PT | XFEATURE_MASK_CET_USER |
-	       XFEATURE_MASK_CET_KERNEL;
+	/* Ensure features are in the masks */
+	val &= TDX_XFAM_MASK;

It's unncessary.

kvm_caps.supported_xcr0 | kvm_caps.supported_xss must be the subset of TDX_XFAM_MASK;

Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +1 -0
  	if ((val & td_conf->xfam_fixed1) != td_conf->xfam_fixed1)
  		return 0;
@@ -724,6 +729,19 @@ int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu)
  	return 1;
  }
 
+static bool tdx_guest_state_is_invalid(struct kvm_vcpu *vcpu)

arch/x86/kvm/vmx/tdx.c int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu) · line 742

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+{
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
+
+	return vcpu->arch.xcr0 != (kvm_tdx->xfam & TDX_XFAM_XCR0_MASK) ||
+	       vcpu->arch.ia32_xss != (kvm_tdx->xfam & TDX_XFAM_XSS_MASK) ||
+	       vcpu->arch.pkru ||
+	       (cpu_feature_enabled(X86_FEATURE_XSAVE) &&
+		!kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) ||
+	       (cpu_feature_enabled(X86_FEATURE_XSAVES) &&
+		!guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES));

guest_cpu_cap_has() is better to put into the path when userspace configures the vcpu model. So that KVM can return error to userspace earlier before running the vcpu.

Quoted 61 lines
arch/x86/kvm/vmx/tdx.c +29 -0
+}> +
  static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu)
  {
  	struct vcpu_tdx *tdx = to_tdx(vcpu);
@@ -740,6 +758,8 @@ static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu)
 
  fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
  {
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+
  	/*
  	 * force_immediate_exit requires vCPU entering for events injection with
  	 * an immediately exit followed. But The TDX module doesn't guarantee
@@ -750,10 +770,22 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
  	 */
  	WARN_ON_ONCE(force_immediate_exit);
 
+	if (WARN_ON_ONCE(tdx_guest_state_is_invalid(vcpu))) {
+		/*
+		 * Invalid exit_reason becomes KVM_EXIT_INTERNAL_ERROR, refer
+		 * tdx_handle_exit().
+		 */
+		tdx->vt.exit_reason.full = -1u;
+		tdx->vp_enter_ret = -1u;
+		return EXIT_FASTPATH_NONE;
+	}
+
  	trace_kvm_entry(vcpu, force_immediate_exit);
 
  	tdx_vcpu_enter_exit(vcpu);
 
+	kvm_load_host_xsave_state(vcpu);
+
  	vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;
 
  	trace_kvm_exit(vcpu, KVM_ISA_VMX);
@@ -1878,9 +1910,23 @@ static int tdx_vcpu_get_cpuid(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
  	return r;
  }
 
+static u64 tdx_guest_cr0(struct kvm_vcpu *vcpu, u64 cr4)
+{
+	u64 cr0 = ~CR0_RESERVED_BITS;
+
+	if (cr4 & X86_CR4_CET)
+		cr0 |= X86_CR0_WP;
+
+	cr0 |= X86_CR0_PE | X86_CR0_NE;
+	cr0 &= ~(X86_CR0_NW | X86_CR0_CD);
+
+	return cr0;
+}
+
  static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
  {
  	u64 apic_base;
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
  	struct vcpu_tdx *tdx = to_tdx(vcpu);
  	int ret;
 

arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd) · line 1952

Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -1903,6 +1949,20 @@ static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
  	if (ret)
  		return ret;
 
+	vcpu->arch.cr4 = ~vcpu->arch.cr4_guest_rsvd_bits;

when userspace doesn't configure XFEATURE_MASK_PKRU in XFAM, it seems kvm_load_host_xsave_state() will skip restore host's PKRU value.

Besides, vcpu->arch.cr4_guest_rsvd_bits depends on KVM_SET_CPUID*, we need enfore the dependency that tdx_vcpu_init() needs to be called after vcpu model is configured.

arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd) · line 1958

Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+	vcpu->arch.cr0 = tdx_guest_cr0(vcpu, vcpu->arch.cr4);
+	/*
+	 * On return from VP.ENTER, the TDX Module sets XCR0 and XSS to the
+	 * maximal values supported by the guest, and zeroes PKRU, so from
+	 * KVM's perspective, those are the guest's values at all times.
+	 */

It's better to also call out that this is only to make kvm_load_host_xsave_state() work for TDX. They don't represent the real guest value.

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7 -0
+	vcpu->arch.ia32_xss = kvm_tdx->xfam & TDX_XFAM_XSS_MASK;
+	vcpu->arch.xcr0 = kvm_tdx->xfam & TDX_XFAM_XCR0_MASK;
+	vcpu->arch.pkru = 0;
+
+	/* TODO: freeze vCPU model before kvm_update_cpuid_runtime() */
+	kvm_update_cpuid_runtime(vcpu);
+
  	tdx->state = VCPU_TD_STATE_INITIALIZED;
 
  	return 0;
#28 lore raw

Re: [PATCH V2 07/12] KVM: TDX: restore host xsave state when exit from the guest TD

Quoted 46 lines · Xiaoyao Li
Quoted 45 lines · Adrian Hunter

From: Isaku Yamahata <[email protected]>

On exiting from the guest TD, xsave state is clobbered.  Restore xsave state on TD exit.

Set up guest state so that existing kvm_load_host_xsave_state() can be used. Do not allow VCPU entry if guest state conflicts with the TD's configuration.

---
TD vcpu enter/exit v2:
  - Drop PT and CET feature flags (Chao)
  - Use cpu_feature_enabled() instead of static_cpu_has() (Chao)
  - Restore PKRU only if the host value differs from defined exit value (Chao)
  - Use defined masks to separate XFAM bits into XCR0/XSS (Adrian)
  - Use existing kvm_load_host_xsave_state() in place of tdx_restore_host_xsave_state() by defining guest CR4, XCR0, XSS and PKRU (Sean)
  - Do not enter if vital guest state is invalid (Adrian)

TD vcpu enter/exit v1:
  - Remove noinstr on tdx_vcpu_enter_exit() (Sean)
  - Switch to kvm_host struct for xcr0 and xss

v19:
  - Add EXPORT_SYMBOL_GPL(host_xcr0)

v15 -> v16:
  - Added CET flag mask
---
  arch/x86/kvm/vmx/tdx.c | 72 ++++++++++++++++++++++++++++++++++++++----

  1 file changed, 66 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +0 -0
index 3f3d61935a58..e4355553569a 100644
@@ -83,16 +83,21 @@ static u64 tdx_get_supported_attrs(const struct tdx_sys_info_td_conf *td_conf)
       return val;
   }

  +/*
+ * Before returning from TDH.VP.ENTER, the TDX Module assigns:

Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static u64 tdx_get_supported_attrs(const struct tdx_sys_info_td_conf *td_conf) · line 93

Quoted 6 lines

+ *   XCR0 to the TD’s user-mode feature bits of XFAM (bits 7:0, 9, 18:17)
+ *   IA32_XSS to the TD's supervisor-mode feature bits of XFAM (bits 8, 16:10)
+ */

arch/x86/kvm/vmx/tdx.c +3 -0
+#define TDX_XFAM_XCR0_MASK    (GENMASK(7, 0) | BIT(9) | GENMASK(18, 17))
+#define TDX_XFAM_XSS_MASK    (BIT(8) | GENMASK(16, 10))
+#define TDX_XFAM_MASK        (TDX_XFAM_XCR0_MASK | TDX_XFAM_XSS_MASK)

No need to define TDX-specific mask for XFAM. kernel defines XFEATURE_MASK_* and you can define XFEATURE_XCR0_MASK and XFEATURE_XSS_MASK with XFEATURE_MASK_*.

Curently, those masks are being added only when the (host) kernel uses them.

Quoted 13 lines
Quoted 12 lines

  static u64 tdx_get_supported_xfam(const struct tdx_sys_info_td_conf *td_conf) {
      u64 val = kvm_caps.supported_xcr0 | kvm_caps.supported_xss; -    /*
-     * PT and CET can be exposed to TD guest regardless of KVM's XSS, PT -     * and, CET support.
-     */
-    val |= XFEATURE_MASK_PT | XFEATURE_MASK_CET_USER |
-           XFEATURE_MASK_CET_KERNEL;
+    /* Ensure features are in the masks */
+    val &= TDX_XFAM_MASK;

Quoted 3 lines

It's unncessary.

kvm_caps.supported_xcr0 | kvm_caps.supported_xss must be the subset of TDX_XFAM_MASK;

The code has changed in kvm-coco-queue.

Quoted 11 lines
Quoted 11 lines

      if ((val & td_conf->xfam_fixed1) != td_conf->xfam_fixed1) return 0;

arch/x86/kvm/vmx/tdx.c +4 -0
@@ -724,6 +729,19 @@ int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu)
       return 1;
   }
   +static bool tdx_guest_state_is_invalid(struct kvm_vcpu *vcpu)
+{
+    struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
+
+    return vcpu->arch.xcr0 != (kvm_tdx->xfam & TDX_XFAM_XCR0_MASK) ||
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu) · line 742

Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+           vcpu->arch.ia32_xss != (kvm_tdx->xfam & TDX_XFAM_XSS_MASK) ||
+           vcpu->arch.pkru ||
+           (cpu_feature_enabled(X86_FEATURE_XSAVE) &&
+        !kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) ||
+           (cpu_feature_enabled(X86_FEATURE_XSAVES) &&
+        !guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES));

guest_cpu_cap_has() is better to put into the path when userspace configures the vcpu model. So that KVM can return error to userspace earlier before running the vcpu.

The purpose of the function is to protect host state from being restored incorrectly, which is why it is called before tdx_vcpu_enter_exit(). i.e. it is protecting against unexpected changes to guest state information that do not match TD configuration. That is largely because the KVM code base is quite complex and the consequences of messing up host state are dire. guest_cpu_cap_has() is very quick, so there is not really any reason not to use it here.

Quoted 52 lines
Quoted 52 lines
arch/x86/kvm/vmx/tdx.c +26 -0
+}> +
   static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu)
   {
       struct vcpu_tdx *tdx = to_tdx(vcpu);
@@ -740,6 +758,8 @@ static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu)
     fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
   {
+    struct vcpu_tdx *tdx = to_tdx(vcpu);
+
       /*
        * force_immediate_exit requires vCPU entering for events injection with
        * an immediately exit followed. But The TDX module doesn't guarantee
@@ -750,10 +770,22 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
        */
       WARN_ON_ONCE(force_immediate_exit);
   +    if (WARN_ON_ONCE(tdx_guest_state_is_invalid(vcpu))) {
+        /*
+         * Invalid exit_reason becomes KVM_EXIT_INTERNAL_ERROR, refer
+         * tdx_handle_exit().
+         */
+        tdx->vt.exit_reason.full = -1u;
+        tdx->vp_enter_ret = -1u;
+        return EXIT_FASTPATH_NONE;
+    }
+
       trace_kvm_entry(vcpu, force_immediate_exit);
         tdx_vcpu_enter_exit(vcpu);
   +    kvm_load_host_xsave_state(vcpu);
+
       vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;
         trace_kvm_exit(vcpu, KVM_ISA_VMX);
@@ -1878,9 +1910,23 @@ static int tdx_vcpu_get_cpuid(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
       return r;
   }
   +static u64 tdx_guest_cr0(struct kvm_vcpu *vcpu, u64 cr4)
+{
+    u64 cr0 = ~CR0_RESERVED_BITS;
+
+    if (cr4 & X86_CR4_CET)
+        cr0 |= X86_CR0_WP;
+
+    cr0 |= X86_CR0_PE | X86_CR0_NE;
+    cr0 &= ~(X86_CR0_NW | X86_CR0_CD);
+
+    return cr0;
+}
+
   static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
   {
       u64 apic_base;
+    struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
Quoted 8 lines
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +0 -0
       struct vcpu_tdx *tdx = to_tdx(vcpu);
       int ret;
   @@ -1903,6 +1949,20 @@ static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)

      if (ret)
          return ret;
  +    vcpu->arch.cr4 = ~vcpu->arch.cr4_guest_rsvd_bits;

when userspace doesn't configure XFEATURE_MASK_PKRU in XFAM, it seems kvm_load_host_xsave_state() will skip restore host's PKRU value.

That's correct.

Quoted 2 lines

Besides, vcpu->arch.cr4_guest_rsvd_bits depends on KVM_SET_CPUID*, we need enfore the dependency that tdx_vcpu_init() needs to be called after vcpu model is configured.

Sean had some code to freeze the CPU model, hence the subsequent TODO. refer https://lore.kernel.org/all/[email protected]/
Anything that matters will be caught by tdx_guest_state_is_invalid().

Quoted 9 lines
Quoted 7 lines

+    vcpu->arch.cr0 = tdx_guest_cr0(vcpu, vcpu->arch.cr4);
+    /*
+     * On return from VP.ENTER, the TDX Module sets XCR0 and XSS to the +     * maximal values supported by the guest, and zeroes PKRU, so from +     * KVM's perspective, those are the guest's values at all times. +     */

It's better to also call out that this is only to make kvm_load_host_xsave_state() work for TDX. They don't represent the real guest value.

It is to provide a reasonable value not just for
kvm_load_host_xsave_state().

Note Sean wrote the comment.

This patch set is owned by Paolo now, so it is up to him.

Quoted 11 lines
Quoted 10 lines

+    vcpu->arch.ia32_xss = kvm_tdx->xfam & TDX_XFAM_XSS_MASK; +    vcpu->arch.xcr0 = kvm_tdx->xfam & TDX_XFAM_XCR0_MASK; +    vcpu->arch.pkru = 0;

arch/x86/kvm/vmx/tdx.c +4 -0
+
+    /* TODO: freeze vCPU model before kvm_update_cpuid_runtime() */
+    kvm_update_cpuid_runtime(vcpu);
+

      tdx->state = VCPU_TD_STATE_INITIALIZED;
        return 0;

#29 lore raw

Re: [PATCH V2 07/12] KVM: TDX: restore host xsave state when exit from the guest TD

Quoted 3 lines · Adrian Hunter
Quoted 2 lines
Quoted 1 line

  +    vcpu->arch.cr4 = ~vcpu->arch.cr4_guest_rsvd_bits;

when userspace doesn't configure XFEATURE_MASK_PKRU in XFAM, it seems kvm_load_host_xsave_state() will skip restore host's PKRU value.

That's correct.

FYI, in this case, it's safe and functional though kvm_load_host_xsave_state() skips restoring the host's PKRU value. Because host's PKRU is not clobbered after TD exit when XFAM.PKU is 0. So no need to restore.

#30 lore raw

PATCH v2 8/12 KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr

In reply to Adrian Hunter

From: Chao Gao <[email protected]>

Several MSRs are constant and only used in userspace(ring 3). But VMs may have different values. KVM uses kvm_set_user_return_msr() to switch to guest's values and leverages user return notifier to restore them when the kernel is to return to userspace. To eliminate unnecessary wrmsr, KVM also caches the value it wrote to an MSR last time.

TDX module unconditionally resets some of these MSRs to architectural INIT state on TD exit. It makes the cached values in kvm_user_return_msrs are inconsistent with values in hardware. This inconsistency needs to be fixed. Otherwise, it may mislead kvm_on_user_return() to skip restoring some MSRs to the host's values. kvm_set_user_return_msr() can help correct this case, but it is not optimal as it always does a wrmsr. So, introduce a variation of kvm_set_user_return_msr() to update cached values and skip that wrmsr.

---
TD vcpu enter/exit v2:
- No changes

TD vcpu enter/exit v1:
- Rename functions and remove useless comment (Binbin)

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/x86.c              | 24 +++++++++++++++++++-----
 2 files changed, 20 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 6b686d62c735..e557a441fade 100644
@@ -2322,6 +2322,7 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low,
 int kvm_add_user_return_msr(u32 msr);
 int kvm_find_user_return_msr(u32 msr);
 int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask);
+void kvm_user_return_msr_update_cache(unsigned int index, u64 val);
 
 static inline bool kvm_is_supported_user_return_msr(u32 msr)
 {
arch/x86/kvm/x86.c +19 -5
index 5cf9f023fd4b..15447fe7687c 100644
@@ -636,6 +636,15 @@ static void kvm_user_return_msr_cpu_online(void)
 	}
 }
 
+static void kvm_user_return_register_notifier(struct kvm_user_return_msrs *msrs)
+{
+	if (!msrs->registered) {
+		msrs->urn.on_user_return = kvm_on_user_return;
+		user_return_notifier_register(&msrs->urn);
+		msrs->registered = true;
+	}
+}
+
 int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask)
 {
 	struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
@@ -649,15 +658,20 @@ int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask)
 		return 1;
 
 	msrs->values[slot].curr = value;
-	if (!msrs->registered) {
-		msrs->urn.on_user_return = kvm_on_user_return;
-		user_return_notifier_register(&msrs->urn);
-		msrs->registered = true;
-	}
+	kvm_user_return_register_notifier(msrs);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_set_user_return_msr);
 
+void kvm_user_return_msr_update_cache(unsigned int slot, u64 value)
+{
+	struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
+
+	msrs->values[slot].curr = value;
+	kvm_user_return_register_notifier(msrs);
+}
+EXPORT_SYMBOL_GPL(kvm_user_return_msr_update_cache);
+
 static void drop_user_return_notifiers(void)
 {
 	struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
Signature · 1 line
2.43.0
Reviewed-by on v2 8/12
#31 lore raw

Re: [PATCH V2 08/12] KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr

Quoted 17 lines · Adrian Hunter

From: Chao Gao <[email protected]>

Several MSRs are constant and only used in userspace(ring 3). But VMs may have different values. KVM uses kvm_set_user_return_msr() to switch to guest's values and leverages user return notifier to restore them when the kernel is to return to userspace. To eliminate unnecessary wrmsr, KVM also caches the value it wrote to an MSR last time.

TDX module unconditionally resets some of these MSRs to architectural INIT state on TD exit. It makes the cached values in kvm_user_return_msrs are inconsistent with values in hardware. This inconsistency needs to be fixed. Otherwise, it may mislead kvm_on_user_return() to skip restoring some MSRs to the host's values. kvm_set_user_return_msr() can help correct this case, but it is not optimal as it always does a wrmsr. So, introduce a variation of kvm_set_user_return_msr() to update cached values and skip that wrmsr.

Quoted 4 lines
Quoted 69 lines

---
TD vcpu enter/exit v2:
- No changes

TD vcpu enter/exit v1:
- Rename functions and remove useless comment (Binbin)

---
  arch/x86/include/asm/kvm_host.h |  1 +
  arch/x86/kvm/x86.c              | 24 +++++++++++++++++++-----
  2 files changed, 20 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 6b686d62c735..e557a441fade 100644
@@ -2322,6 +2322,7 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low,
  int kvm_add_user_return_msr(u32 msr);
  int kvm_find_user_return_msr(u32 msr);
  int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask);
+void kvm_user_return_msr_update_cache(unsigned int index, u64 val);
 
  static inline bool kvm_is_supported_user_return_msr(u32 msr)
  {
arch/x86/kvm/x86.c +19 -5
index 5cf9f023fd4b..15447fe7687c 100644
@@ -636,6 +636,15 @@ static void kvm_user_return_msr_cpu_online(void)
  	}
  }
 
+static void kvm_user_return_register_notifier(struct kvm_user_return_msrs *msrs)
+{
+	if (!msrs->registered) {
+		msrs->urn.on_user_return = kvm_on_user_return;
+		user_return_notifier_register(&msrs->urn);
+		msrs->registered = true;
+	}
+}
+
  int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask)
  {
  	struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
@@ -649,15 +658,20 @@ int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask)
  		return 1;
 
  	msrs->values[slot].curr = value;
-	if (!msrs->registered) {
-		msrs->urn.on_user_return = kvm_on_user_return;
-		user_return_notifier_register(&msrs->urn);
-		msrs->registered = true;
-	}
+	kvm_user_return_register_notifier(msrs);
  	return 0;
  }
  EXPORT_SYMBOL_GPL(kvm_set_user_return_msr);
 
+void kvm_user_return_msr_update_cache(unsigned int slot, u64 value)
+{
+	struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
+
+	msrs->values[slot].curr = value;
+	kvm_user_return_register_notifier(msrs);
+}
+EXPORT_SYMBOL_GPL(kvm_user_return_msr_update_cache);
+
  static void drop_user_return_notifiers(void)
  {
  	struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
#32 lore raw

PATCH v2 9/12 KVM: TDX: restore user ret MSRs

In reply to Adrian Hunter

From: Isaku Yamahata <[email protected]>

Several user ret MSRs are clobbered on TD exit. Restore those values on TD exit and before returning to ring 3.

---
TD vcpu enter/exit v2:
- No changes

TD vcpu enter/exit v1:
- Rename tdx_user_return_update_cache() -> tdx_user_return_msr_update_cache() (extrapolated from Binbin)
- Adjust to rename in previous patches (Binbin)
- Simplify comment (Tony)
- Move code change in tdx_hardware_setup() to __tdx_bringup().

---
 arch/x86/kvm/vmx/tdx.c | 44 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/tdx.c +43 -1
index e4355553569a..a0f5cdfd290b 100644
@@ -729,6 +729,28 @@ int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+struct tdx_uret_msr {
+	u32 msr;
+	unsigned int slot;
+	u64 defval;
+};
+
+static struct tdx_uret_msr tdx_uret_msrs[] = {
+	{.msr = MSR_SYSCALL_MASK, .defval = 0x20200 },
+	{.msr = MSR_STAR,},
+	{.msr = MSR_LSTAR,},
+	{.msr = MSR_TSC_AUX,},
+};
+
+static void tdx_user_return_msr_update_cache(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(tdx_uret_msrs); i++)
+		kvm_user_return_msr_update_cache(tdx_uret_msrs[i].slot,
+						 tdx_uret_msrs[i].defval);
+}
+
 static bool tdx_guest_state_is_invalid(struct kvm_vcpu *vcpu)
 {
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
@@ -784,6 +806,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
 
 	tdx_vcpu_enter_exit(vcpu);
 
+	tdx_user_return_msr_update_cache();
+
 	kvm_load_host_xsave_state(vcpu);
 
 	vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;
@@ -2245,7 +2269,25 @@ static bool __init kvm_can_support_tdx(void)
 static int __init __tdx_bringup(void)
 {
 	const struct tdx_sys_info_td_conf *td_conf;
-	int r;
+	int r, i;
+
+	for (i = 0; i < ARRAY_SIZE(tdx_uret_msrs); i++) {
+		/*
+		 * Check if MSRs (tdx_uret_msrs) can be saved/restored
+		 * before returning to user space.
+		 *
+		 * this_cpu_ptr(user_return_msrs)->registered isn't checked
+		 * because the registration is done at vcpu runtime by
+		 * tdx_user_return_msr_update_cache().
+		 */
+		tdx_uret_msrs[i].slot = kvm_find_user_return_msr(tdx_uret_msrs[i].msr);
+		if (tdx_uret_msrs[i].slot == -1) {
+			/* If any MSR isn't supported, it is a KVM bug */
+			pr_err("MSR %x isn't included by kvm_find_user_return_msr\n",
+				tdx_uret_msrs[i].msr);
+			return -EIO;
+		}
+	}
 
 	/*
 	 * Enabling TDX requires enabling hardware virtualization first,
Signature · 1 line
2.43.0
#33 lore raw

Re: [PATCH V2 09/12] KVM: TDX: restore user ret MSRs

Quoted 38 lines · Adrian Hunter

From: Isaku Yamahata <[email protected]>

Several user ret MSRs are clobbered on TD exit. Restore those values on TD exit and before returning to ring 3.

---
TD vcpu enter/exit v2:
- No changes

TD vcpu enter/exit v1:
- Rename tdx_user_return_update_cache() -> tdx_user_return_msr_update_cache() (extrapolated from Binbin)
- Adjust to rename in previous patches (Binbin)
- Simplify comment (Tony)
- Move code change in tdx_hardware_setup() to __tdx_bringup().

---
  arch/x86/kvm/vmx/tdx.c | 44 +++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 43 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/tdx.c +6 -0
index e4355553569a..a0f5cdfd290b 100644
@@ -729,6 +729,28 @@ int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu)
  	return 1;
  }
 
+struct tdx_uret_msr {
+	u32 msr;
+	unsigned int slot;
+	u64 defval;
+};
+

arch/x86/kvm/vmx/tdx.c int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu) · line 747

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+static struct tdx_uret_msr tdx_uret_msrs[] = {
+	{.msr = MSR_SYSCALL_MASK, .defval = 0x20200 },
+	{.msr = MSR_STAR,},
+	{.msr = MSR_LSTAR,},
+	{.msr = MSR_TSC_AUX,},
+};
+
+static void tdx_user_return_msr_update_cache(void)
+{
+	int i;

I think it can be optimized to skip update cache if it the caches are updated already. No need to update the cache after every TD exit.

Quoted 44 lines
arch/x86/kvm/vmx/tdx.c +26 -1
+	for (i = 0; i < ARRAY_SIZE(tdx_uret_msrs); i++)
+		kvm_user_return_msr_update_cache(tdx_uret_msrs[i].slot,
+						 tdx_uret_msrs[i].defval);
+}
+
  static bool tdx_guest_state_is_invalid(struct kvm_vcpu *vcpu)
  {
  	struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
@@ -784,6 +806,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
 
  	tdx_vcpu_enter_exit(vcpu);
 
+	tdx_user_return_msr_update_cache();
+
  	kvm_load_host_xsave_state(vcpu);
 
  	vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;
@@ -2245,7 +2269,25 @@ static bool __init kvm_can_support_tdx(void)
  static int __init __tdx_bringup(void)
  {
  	const struct tdx_sys_info_td_conf *td_conf;
-	int r;
+	int r, i;
+
+	for (i = 0; i < ARRAY_SIZE(tdx_uret_msrs); i++) {
+		/*
+		 * Check if MSRs (tdx_uret_msrs) can be saved/restored
+		 * before returning to user space.
+		 *
+		 * this_cpu_ptr(user_return_msrs)->registered isn't checked
+		 * because the registration is done at vcpu runtime by
+		 * tdx_user_return_msr_update_cache().
+		 */
+		tdx_uret_msrs[i].slot = kvm_find_user_return_msr(tdx_uret_msrs[i].msr);
+		if (tdx_uret_msrs[i].slot == -1) {
+			/* If any MSR isn't supported, it is a KVM bug */
+			pr_err("MSR %x isn't included by kvm_find_user_return_msr\n",
+				tdx_uret_msrs[i].msr);
+			return -EIO;
+		}
+	}
 
  	/*
  	 * Enabling TDX requires enabling hardware virtualization first,
#34 lore raw

Re: [PATCH V2 09/12] KVM: TDX: restore user ret MSRs

Quoted 42 lines · Xiaoyao Li
Quoted 41 lines · Adrian Hunter

From: Isaku Yamahata <[email protected]>

Several user ret MSRs are clobbered on TD exit.  Restore those values on TD exit and before returning to ring 3.

---
TD vcpu enter/exit v2:
  - No changes

TD vcpu enter/exit v1:
  - Rename tdx_user_return_update_cache() -> tdx_user_return_msr_update_cache() (extrapolated from Binbin)
  - Adjust to rename in previous patches (Binbin)
  - Simplify comment (Tony)
  - Move code change in tdx_hardware_setup() to __tdx_bringup().
---
  arch/x86/kvm/vmx/tdx.c | 44 +++++++++++++++++++++++++++++++++++++++++-

  1 file changed, 43 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/tdx.c +9 -0
index e4355553569a..a0f5cdfd290b 100644
@@ -729,6 +729,28 @@ int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu)
       return 1;
   }
   +struct tdx_uret_msr {
+    u32 msr;
+    unsigned int slot;
+    u64 defval;
+};
+
+static struct tdx_uret_msr tdx_uret_msrs[] = {
+    {.msr = MSR_SYSCALL_MASK, .defval = 0x20200 },
+    {.msr = MSR_STAR,},
+    {.msr = MSR_LSTAR,},
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu) · line 747

Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+    {.msr = MSR_TSC_AUX,},
+};
+
+static void tdx_user_return_msr_update_cache(void)
+{
+    int i;

I think it can be optimized to skip update cache if it the caches are updated already. No need to update the cache after every TD exit.

It might be ok to move tdx_user_return_msr_update_cache() to tdx_prepare_switch_to_host() but tdx_user_return_msr_update_cache() should be pretty quick, so probably not worth messing around with it at this stage.

This patch set is owned by Paolo now, so it is up to him.

Quoted 42 lines
Quoted 41 lines
arch/x86/kvm/vmx/tdx.c +25 -1
+    for (i = 0; i < ARRAY_SIZE(tdx_uret_msrs); i++)
+        kvm_user_return_msr_update_cache(tdx_uret_msrs[i].slot,
+                         tdx_uret_msrs[i].defval);
+}
+
   static bool tdx_guest_state_is_invalid(struct kvm_vcpu *vcpu)
   {
       struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
@@ -784,6 +806,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
         tdx_vcpu_enter_exit(vcpu);
   +    tdx_user_return_msr_update_cache();
+
       kvm_load_host_xsave_state(vcpu);
         vcpu->arch.regs_avail &= ~TDX_REGS_UNSUPPORTED_SET;
@@ -2245,7 +2269,25 @@ static bool __init kvm_can_support_tdx(void)
   static int __init __tdx_bringup(void)
   {
       const struct tdx_sys_info_td_conf *td_conf;
-    int r;
+    int r, i;
+
+    for (i = 0; i < ARRAY_SIZE(tdx_uret_msrs); i++) {
+        /*
+         * Check if MSRs (tdx_uret_msrs) can be saved/restored
+         * before returning to user space.
+         *
+         * this_cpu_ptr(user_return_msrs)->registered isn't checked
+         * because the registration is done at vcpu runtime by
+         * tdx_user_return_msr_update_cache().
+         */
+        tdx_uret_msrs[i].slot = kvm_find_user_return_msr(tdx_uret_msrs[i].msr);
+        if (tdx_uret_msrs[i].slot == -1) {
+            /* If any MSR isn't supported, it is a KVM bug */
+            pr_err("MSR %x isn't included by kvm_find_user_return_msr\n",
+                tdx_uret_msrs[i].msr);
+            return -EIO;
+        }
+    }
         /*
        * Enabling TDX requires enabling hardware virtualization first,
#35 lore raw

PATCH v2 10/12 KVM: TDX: Disable support for TSX and WAITPKG

In reply to Adrian Hunter

Support for restoring IA32_TSX_CTRL MSR and IA32_UMWAIT_CONTROL MSR is not yet implemented, so disable support for TSX and WAITPKG for now. Clear the associated CPUID bits returned by KVM_TDX_CAPABILITIES, and return an error if those bits are set in KVM_TDX_INIT_VM.

---
TD vcpu enter/exit v2:
- New patch

---
 arch/x86/kvm/vmx/tdx.c | 43 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
arch/x86/kvm/vmx/tdx.c +43 -0
index a0f5cdfd290b..70996af4be64 100644
@@ -117,6 +117,44 @@ static u32 tdx_set_guest_phys_addr_bits(const u32 eax, int addr_bits)
 	return (eax & ~GENMASK(23, 16)) | (addr_bits & 0xff) << 16;
 }
 
+#define TDX_FEATURE_TSX (__feature_bit(X86_FEATURE_HLE) | __feature_bit(X86_FEATURE_RTM))
+
+static bool has_tsx(const struct kvm_cpuid_entry2 *entry)
+{
+	return entry->function == 7 && entry->index == 0 &&
+	       (entry->ebx & TDX_FEATURE_TSX);
+}
+
+static void clear_tsx(struct kvm_cpuid_entry2 *entry)
+{
+	entry->ebx &= ~TDX_FEATURE_TSX;
+}
+
+static bool has_waitpkg(const struct kvm_cpuid_entry2 *entry)
+{
+	return entry->function == 7 && entry->index == 0 &&
+	       (entry->ecx & __feature_bit(X86_FEATURE_WAITPKG));
+}
+
+static void clear_waitpkg(struct kvm_cpuid_entry2 *entry)
+{
+	entry->ecx &= ~__feature_bit(X86_FEATURE_WAITPKG);
+}
+
+static void tdx_clear_unsupported_cpuid(struct kvm_cpuid_entry2 *entry)
+{
+	if (has_tsx(entry))
+		clear_tsx(entry);
+
+	if (has_waitpkg(entry))
+		clear_waitpkg(entry);
+}
+
+static bool tdx_unsupported_cpuid(const struct kvm_cpuid_entry2 *entry)
+{
+	return has_tsx(entry) || has_waitpkg(entry);
+}
+
 #define KVM_TDX_CPUID_NO_SUBLEAF	((__u32)-1)
 
 static void td_init_cpuid_entry2(struct kvm_cpuid_entry2 *entry, unsigned char idx)
@@ -140,6 +178,8 @@ static void td_init_cpuid_entry2(struct kvm_cpuid_entry2 *entry, unsigned char i
 	 */
 	if (entry->function == 0x80000008)
 		entry->eax = tdx_set_guest_phys_addr_bits(entry->eax, 0xff);
+
+	tdx_clear_unsupported_cpuid(entry);
 }
 
 static int init_kvm_tdx_caps(const struct tdx_sys_info_td_conf *td_conf,
@@ -1214,6 +1254,9 @@ static int setup_tdparams_cpuids(struct kvm_cpuid2 *cpuid,
 		if (!entry)
 			continue;
 
+		if (tdx_unsupported_cpuid(entry))
+			return -EINVAL;
+
 		copy_cnt++;
 
 		value = &td_params->cpuid_values[i];
Signature · 1 line
2.43.0
#36 lore raw

PATCH v2 11/12 KVM: TDX: Save and restore IA32_DEBUGCTL

In reply to Adrian Hunter

Save the IA32_DEBUGCTL MSR before entering a TDX VCPU and restore it afterwards. The TDX Module preserves bits 1, 12, and 14, so if no other bits are set, no restore is done.

---
TD vcpu enter/exit v2:
- New patch
- Rebased due to moving host_debugctlmsr to struct vcpu_vt.

---
 arch/x86/kvm/vmx/tdx.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/kvm/vmx/tdx.c +10 -0
index 70996af4be64..0bce00415f42 100644
@@ -705,6 +705,8 @@ void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
 	else
 		vt->msr_host_kernel_gs_base = read_msr(MSR_KERNEL_GS_BASE);
 
+	vt->host_debugctlmsr = get_debugctlmsr();
+
 	vt->guest_state_loaded = true;
 }
 
@@ -818,9 +820,14 @@ static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu)
 #define TDX_REGS_UNSUPPORTED_SET	(BIT(VCPU_EXREG_RFLAGS) |	\
 					 BIT(VCPU_EXREG_SEGMENTS))
 
+#define TDX_DEBUGCTL_PRESERVED (DEBUGCTLMSR_BTF | \
+				DEBUGCTLMSR_FREEZE_PERFMON_ON_PMI | \
+				DEBUGCTLMSR_FREEZE_IN_SMM)
+
 fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
 {
 	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct vcpu_vt *vt = to_vt(vcpu);
 
 	/*
 	 * force_immediate_exit requires vCPU entering for events injection with
@@ -846,6 +853,9 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)
 
 	tdx_vcpu_enter_exit(vcpu);
 
+	if (vt->host_debugctlmsr & ~TDX_DEBUGCTL_PRESERVED)
+		update_debugctlmsr(vt->host_debugctlmsr);
+
 	tdx_user_return_msr_update_cache();
 
 	kvm_load_host_xsave_state(vcpu);
Signature · 1 line
2.43.0
#37 lore raw

PATCH v2 12/12 KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior

In reply to Adrian Hunter

From: Isaku Yamahata <[email protected]>

Add a flag KVM_DEBUGREG_AUTO_SWITCH to skip saving/restoring guest DRs.

TDX-SEAM unconditionally saves/restores guest DRs on TD exit/enter, and resets DRs to architectural INIT state on TD exit. Use the new flag KVM_DEBUGREG_AUTO_SWITCH to indicate that KVM doesn't need to save/restore guest DRs. KVM still needs to restore host DRs after TD exit if there are active breakpoints in the host, which is covered by the existing code.

MOV-DR exiting is always cleared for TDX guests, so the handler for DR access is never called, and KVM_DEBUGREG_WONT_EXIT is never set. Add a warning if both KVM_DEBUGREG_WONT_EXIT and KVM_DEBUGREG_AUTO_SWITCH are set.

Opportunistically convert the KVM_DEBUGREG_* definitions to use BIT().

Message-ID: <[email protected]>

---
TD vcpu enter/exit v2:
- Moved from TDX "the rest" to "TD vcpu enter/exit"

TDX "the rest" v1:
- Update the comment about KVM_DEBUGREG_AUTO_SWITCH.
- Check explicitly KVM_DEBUGREG_AUTO_SWITCH is not set in switch_db_regs before restoring guest DRs, because KVM_DEBUGREG_BP_ENABLED could be set by userspace. (Paolo)
https://lore.kernel.org/lkml/[email protected]/
- Fix the issue that host DRs are not restored in v19 (Binbin)
https://lore.kernel.org/kvm/[email protected]/
- Update the changelog a bit.

---
 arch/x86/include/asm/kvm_host.h | 11 +++++++++--
 arch/x86/kvm/vmx/tdx.c          |  1 +
 arch/x86/kvm/x86.c              |  4 +++-
 3 files changed, 13 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +9 -2
index e557a441fade..bcfd89c28308 100644
@@ -606,8 +606,15 @@ struct kvm_pmu {
 struct kvm_pmu_ops;
 
 enum {
-	KVM_DEBUGREG_BP_ENABLED = 1,
-	KVM_DEBUGREG_WONT_EXIT = 2,
+	KVM_DEBUGREG_BP_ENABLED		= BIT(0),
+	KVM_DEBUGREG_WONT_EXIT		= BIT(1),
+	/*
+	 * Guest debug registers (DR0-3, DR6 and DR7) are saved/restored by
+	 * hardware on exit from or enter to guest. KVM needn't switch them.
+	 * DR0-3, DR6 and DR7 are set to their architectural INIT value on VM
+	 * exit, host values need to be restored.
+	 */
+	KVM_DEBUGREG_AUTO_SWITCH	= BIT(2),
 };
 
 struct kvm_mtrr {
arch/x86/kvm/vmx/tdx.c +1 -0
index 0bce00415f42..0863bdaf761a 100644
@@ -652,6 +652,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.efer = EFER_SCE | EFER_LME | EFER_LMA | EFER_NX;
 
+	vcpu->arch.switch_db_regs = KVM_DEBUGREG_AUTO_SWITCH;
 	vcpu->arch.cr0_guest_owned_bits = -1ul;
 	vcpu->arch.cr4_guest_owned_bits = -1ul;
 
arch/x86/kvm/x86.c +3 -1
index 15447fe7687c..b023283e7ed4 100644
@@ -10977,7 +10977,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	if (vcpu->arch.guest_fpu.xfd_err)
 		wrmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);
 
-	if (unlikely(vcpu->arch.switch_db_regs)) {
+	if (unlikely(vcpu->arch.switch_db_regs &&
+		     !(vcpu->arch.switch_db_regs & KVM_DEBUGREG_AUTO_SWITCH))) {
 		set_debugreg(0, 7);
 		set_debugreg(vcpu->arch.eff_db[0], 0);
 		set_debugreg(vcpu->arch.eff_db[1], 1);
@@ -11024,6 +11025,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	 */
 	if (unlikely(vcpu->arch.switch_db_regs & KVM_DEBUGREG_WONT_EXIT)) {
 		WARN_ON(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP);
+		WARN_ON(vcpu->arch.switch_db_regs & KVM_DEBUGREG_AUTO_SWITCH);
 		kvm_x86_call(sync_dirty_debug_regs)(vcpu);
 		kvm_update_dr0123(vcpu);
 		kvm_update_dr7(vcpu);
Signature · 1 line
2.43.0