Skip to contentSkip to conversation

KVM: x86: nSVM: Improve PAT virtualization

PATCH Applied Jim Mattson 3 participants 15 messages Apr 7 – May 21

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v8 cover KVM: x86: nSVM: Improve PAT virtualization

Currently, KVM's implementation of nested SVM treats the PAT MSR the same way whether or not nested NPT is enabled: L1 and L2 share a single PAT. However, the AMD APM specifies that when nested NPT is enabled, the host (L1) and the guest (L2) should have independent PATs: hPAT for L1 and gPAT for L2.

This patch series implements independent PATs for L1 and L2 when nested NPT is enabled, but only when a new quirk, KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT, is disabled. By default, the quirk is enabled, preserving KVM's legacy behavior. When the quirk is disabled, KVM correctly virtualizes a separate PAT register for L2, using the g_pat field in the VMCB.

Guest accesses to the IA32_PAT MSR are redirected to either hPAT or gPAT depending on the current mode and whether nested NPT is enabled. All other accesses, including userspace accesses via KVM_{GET,SET}_MSRS, continue to reference hPAT. L2's gPAT is saved and restored via a new 'gpat' field in kvm_svm_nested_state_hdr, which is within the existing padding of the header to maintain ABI compatibility.

v1: https://lore.kernel.org/kvm/[email protected]/
v2: https://lore.kernel.org/kvm/[email protected]/
v3: https://lore.kernel.org/kvm/[email protected]/
v4: https://lore.kernel.org/kvm/[email protected]/
v5: https://lore.kernel.org/kvm/[email protected]/
v6: https://lore.kernel.org/kvm/[email protected]/
v7: https://lore.kernel.org/kvm/[email protected]/

v7 -> v8:
* Indentation changes to conform to Sean's aesthetic [Sean]
* Updated comment in svm_pat_accesses_gpat() [Sean]
* Restored the common behavior for get/set IA32_PAT [Sean]
* Reordered declarations in svm_set_nested_state() for ASCII art [Sean]
* Dropped the selftest [Sean]

Jim Mattson (8):
KVM: x86: Define KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT
KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest
mode
KVM: x86: nSVM: Cache and validate vmcb12 g_pat
KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT
KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT
KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT
KVM: Documentation: document KVM_{GET,SET}_NESTED_STATE for SVM
KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

 Documentation/virt/kvm/api.rst  | 26 ++++++++++++++
 arch/x86/include/asm/kvm_host.h |  3 +-
 arch/x86/include/uapi/asm/kvm.h |  2 ++
 arch/x86/kvm/svm/nested.c       | 64 ++++++++++++++++++++++++---------
 arch/x86/kvm/svm/svm.c          | 41 +++++++++++++++++----
 arch/x86/kvm/svm/svm.h          | 18 +++++++++-
 6 files changed, 130 insertions(+), 24 deletions(-)
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#1 lore raw

PATCH v8 1/8 KVM: x86: Define KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT

Define a quirk to control whether nested SVM shares L1's PAT with L2 (legacy behavior) or gives L2 its own independent gPAT (correct behavior per the APM).

When the quirk is enabled (default), L2 shares L1's PAT, preserving the legacy KVM behavior. When userspace disables the quirk, KVM correctly virtualizes the PAT for nested SVM guests, giving L2 a separate gPAT as specified in the AMD architecture.

---
 Documentation/virt/kvm/api.rst  | 14 ++++++++++++++
 arch/x86/include/asm/kvm_host.h |  3 ++-
 arch/x86/include/uapi/asm/kvm.h |  1 +
 arch/x86/kvm/svm/svm.h          | 10 ++++++++++
 4 files changed, 27 insertions(+), 1 deletion(-)
Documentation/virt/kvm/api.rst +14 -0
index 032516783e96..2d56f17e3760 100644
@@ -8551,6 +8551,20 @@ KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM   By default, KVM relaxes the consisten
                                            bit to be cleared.  Note that the vmcs02
                                            bit is still completely controlled by the
                                            host, regardless of the quirk setting.
+
+KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT        By default, KVM for nested SVM guests
+                                           shares the IA32_PAT MSR between L1 and
+                                           L2. This is legacy behavior and does
+                                           not match the AMD architecture
+                                           specification. When this quirk is
+                                           disabled and nested paging (NPT) is
+                                           enabled for L2, KVM correctly
+                                           virtualizes a separate guest PAT
+                                           register for L2, using the g_pat
+                                           field in the VMCB. When NPT is
+                                           disabled for L2, L1 and L2 continue
+                                           to share the IA32_PAT MSR regardless
+                                           of the quirk setting.
 ========================================   ================================================
 
 7.32 KVM_CAP_MAX_VCPU_ID
arch/x86/include/asm/kvm_host.h +2 -1
index c470e40a00aa..f77d64bbd409 100644
@@ -2526,7 +2526,8 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);
 	 KVM_X86_QUIRK_SLOT_ZAP_ALL |		\
 	 KVM_X86_QUIRK_STUFF_FEATURE_MSRS |	\
 	 KVM_X86_QUIRK_IGNORE_GUEST_PAT |	\
-	 KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM)
+	 KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM |	\
+	 KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT)
 
 #define KVM_X86_CONDITIONAL_QUIRKS		\
 	(KVM_X86_QUIRK_CD_NW_CLEARED |		\
arch/x86/include/uapi/asm/kvm.h +1 -0
index 5f2b30d0405c..3ada2fa9ca86 100644
@@ -477,6 +477,7 @@ struct kvm_sync_regs {
 #define KVM_X86_QUIRK_STUFF_FEATURE_MSRS	(1 << 8)
 #define KVM_X86_QUIRK_IGNORE_GUEST_PAT		(1 << 9)
 #define KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM (1 << 10)
+#define KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT	(1 << 11)
 
 #define KVM_STATE_NESTED_FORMAT_VMX	0
 #define KVM_STATE_NESTED_FORMAT_SVM	1
arch/x86/kvm/svm/svm.h +10 -0
index fd0652b32c81..a91942269f6a 100644
@@ -630,6 +630,16 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)
 	return svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_NP;
 }
 
+static inline bool l2_has_separate_pat(struct kvm_vcpu *vcpu)
+{
+	/*
+	 * If KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled while a vCPU
+	 * is running, the L2 IA32_PAT semantics for that vCPU are undefined.
+	 */
+	return nested_npt_enabled(to_svm(vcpu)) &&
+	       !kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT);
+}
+
 static inline bool nested_vnmi_enabled(struct vcpu_svm *svm)
 {
 	return guest_cpu_cap_has(&svm->vcpu, X86_FEATURE_VNMI) &&
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#2 lore raw

PATCH v8 2/8 KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode

In reply to Jim Mattson

When running an L2 guest and writing to MSR_IA32_CR_PAT, the host PAT value is stored in both vmcb01's g_pat field and vmcb02's g_pat field, but the clean bit was only being cleared for vmcb02.

Introduce the helper vmcb_set_gpat() which sets vmcb->save.g_pat and marks the VMCB dirty for VMCB_NPT. Use this helper in both svm_set_msr() for updating vmcb01 and in nested_vmcb02_compute_g_pat() for updating vmcb02, ensuring both VMCBs' NPT fields are properly marked dirty.

Fixes 4995a3685f1b ("KVM: SVM: Use a separate vmcb for the nested L2 guest")
---
 arch/x86/kvm/svm/nested.c | 2 +-
 arch/x86/kvm/svm/svm.c    | 3 +--
 arch/x86/kvm/svm/svm.h    | 6 ++++++
 3 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/nested.c +1 -1
index 961804df5f45..3575c9386e94 100644
@@ -697,7 +697,7 @@ void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm)
 		return;
 
 	/* FIXME: merge g_pat from vmcb01 and vmcb12.  */
-	svm->nested.vmcb02.ptr->save.g_pat = svm->vmcb01.ptr->save.g_pat;
+	vmcb_set_gpat(svm->nested.vmcb02.ptr, svm->vmcb01.ptr->save.g_pat);
 }
 
 static bool nested_vmcb12_has_lbrv(struct kvm_vcpu *vcpu)
arch/x86/kvm/svm/svm.c +1 -2
index e7fdd7a9c280..56b6bd5dfdca 100644
@@ -2970,10 +2970,9 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 		if (ret)
 			break;
 
-		svm->vmcb01.ptr->save.g_pat = data;
+		vmcb_set_gpat(svm->vmcb01.ptr, data);
 		if (is_guest_mode(vcpu))
 			nested_vmcb02_compute_g_pat(svm);
-		vmcb_mark_dirty(svm->vmcb, VMCB_NPT);
 		break;
 	case MSR_IA32_SPEC_CTRL:
 		if (!msr->host_initiated &&
arch/x86/kvm/svm/svm.h +6 -0
index a91942269f6a..dfb0a73be606 100644
@@ -453,6 +453,12 @@ static inline bool vmcb12_is_dirty(struct vmcb_ctrl_area_cached *control, int bi
 	return !test_bit(bit, (unsigned long *)&control->clean);
 }
 
+static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)
+{
+	vmcb->save.g_pat = data;
+	vmcb_mark_dirty(vmcb, VMCB_NPT);
+}
+
 static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
 {
 	return container_of(vcpu, struct vcpu_svm, vcpu);
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#3 lore raw

PATCH v8 3/8 KVM: x86: nSVM: Cache and validate vmcb12 g_pat

In reply to Jim Mattson

When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and nested paging is enabled in vmcb12, validate g_pat at emulated VMRUN and cause an immediate VMEXIT with exit code VMEXIT_INVALID if it is invalid, as specified in the APM, volume 2: "Nested Paging and VMRUN/VMEXIT."

Fixes 3d6368ef580a ("KVM: SVM: Add VMRUN handler")
---
 arch/x86/kvm/svm/nested.c | 23 +++++++++++++++++++----
 arch/x86/kvm/svm/svm.h    |  1 +
 2 files changed, 20 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/nested.c +19 -4
index 3575c9386e94..515a8545e8e0 100644
@@ -410,7 +410,8 @@ static bool nested_vmcb_check_controls(struct kvm_vcpu *vcpu,
 
 /* Common checks that apply to both L1 and L2 state.  */
 static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
-				   struct vmcb_save_area_cached *save)
+				   struct vmcb_save_area_cached *save,
+				   bool check_gpat)
 {
 	if (CC(!(save->efer & EFER_SVME)))
 		return false;
@@ -445,6 +446,15 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
 	if (CC(!kvm_valid_efer(vcpu, save->efer)))
 		return false;
 
+	/*
+	 * If userspace contrives to get an invalid g_pat into vmcb02 by
+	 * disabling KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT in a race with
+	 * this check, it should be prepared for the KVM_EXIT_FAIL_ENTRY
+	 * that will follow.
+	 */
+	if (check_gpat && CC(!kvm_pat_valid(save->g_pat)))
+		return false;
+
 	return true;
 }
 
@@ -452,7 +462,8 @@ int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 
-	if (!nested_vmcb_check_save(vcpu, &svm->nested.save) ||
+	if (!nested_vmcb_check_save(vcpu, &svm->nested.save,
+				    l2_has_separate_pat(vcpu)) ||
 	    !nested_vmcb_check_controls(vcpu, &svm->nested.ctl))
 		return -EINVAL;
 
@@ -562,6 +573,7 @@ static void __nested_copy_vmcb_save_to_cache(struct vmcb_save_area_cached *to,
 
 	to->rax = from->rax;
 	to->cr2 = from->cr2;
+	to->g_pat = from->g_pat;
 
 	svm_copy_lbrs(to, from);
 }
@@ -1974,13 +1986,16 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
 	/*
 	 * Validate host state saved from before VMRUN (see
-	 * nested_svm_check_permissions).
+	 * nested_svm_check_permissions). Note that the g_pat field is not
+	 * validated, because (a) it may have been clobbered by SMM before
+	 * KVM_GET_NESTED_STATE, and (b) it is not loaded at emulated
+	 * #VMEXIT.
 	 */
 	__nested_copy_vmcb_save_to_cache(&save_cached, save);
 	if (!(save->cr0 & X86_CR0_PG) ||
 	    !(save->cr0 & X86_CR0_PE) ||
 	    (save->rflags & X86_EFLAGS_VM) ||
-	    !nested_vmcb_check_save(vcpu, &save_cached))
+	    !nested_vmcb_check_save(vcpu, &save_cached, false))
 		goto out_free;
 
 
arch/x86/kvm/svm/svm.h +1 -0
index dfb0a73be606..fdd6286d965e 100644
@@ -165,6 +165,7 @@ struct vmcb_save_area_cached {
 	u64 isst_addr;
 	u64 rax;
 	u64 cr2;
+	u64 g_pat;
 	u64 dbgctl;
 	u64 br_from;
 	u64 br_to;
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#4 lore raw

PATCH v8 4/8 KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT

In reply to Jim Mattson

When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and nested NPT is enabled in vmcb12, copy the (cached and validated) vmcb12 g_pat field to vmcb02's g_pat, giving L2 its own independent guest PAT register.

When the quirk is enabled (default), or when NPT is enabled but nested NPT is disabled, copy L1's IA32_PAT MSR to the vmcb02 g_pat field, since L2 shares the IA32_PAT MSR with L1.

When NPT is disabled, the g_pat field is ignored by hardware.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/nested.c +7 -3
index 515a8545e8e0..58574e803812 100644
@@ -727,9 +727,6 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 	struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 
-	nested_vmcb02_compute_g_pat(svm);
-	vmcb_mark_dirty(vmcb02, VMCB_NPT);
-
 	/* Load the nested guest state */
 	if (svm->nested.vmcb12_gpa != svm->nested.last_vmcb12_gpa) {
 		new_vmcb12 = true;
@@ -760,6 +757,13 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 		vmcb_mark_dirty(vmcb02, VMCB_CET);
 	}
 
+	if (l2_has_separate_pat(vcpu)) {
+		if (unlikely(new_vmcb12 || vmcb12_is_dirty(control, VMCB_NPT)))
+			vmcb_set_gpat(vmcb02, svm->nested.save.g_pat);
+	} else if (npt_enabled) {
+		vmcb_set_gpat(vmcb02, vcpu->arch.pat);
+	}
+
 	kvm_set_rflags(vcpu, save->rflags | X86_EFLAGS_FIXED);
 
 	svm_set_efer(vcpu, svm->nested.save.efer);
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#5 lore raw

PATCH 1/1 KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

In reply to Jim Mattson

When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and the vCPU is in guest mode with nested NPT enabled, guest accesses to IA32_PAT are redirected to the gPAT register, which is stored in VMCB02's g_pat field.

Non-guest accesses (e.g. from userspace) to IA32_PAT are always redirected to hPAT, which is stored in vcpu->arch.pat.

Directing host-initiated accesses to hPAT ensures that KVM_GET/SET_MSRS and KVM_GET/SET_NESTED_STATE are independent of each other and can be ordered arbitrarily during save and restore. gPAT is saved and restored separately via KVM_GET/SET_NESTED_STATE.

Use WARN_ON_ONCE to flag any host-initiated accesses originating from KVM itself rather than userspace.

Use pr_warn_once to flag any use of the common MSR-handling code (now shared by VMX and TDX) for IA32_PAT by a vCPU that is SVM-capable.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c |  9 ---------
 arch/x86/kvm/svm/svm.c    | 36 +++++++++++++++++++++++++++++++++---
 arch/x86/kvm/svm/svm.h    |  1 -
 3 files changed, 33 insertions(+), 13 deletions(-)
arch/x86/kvm/svm/nested.c +0 -9
index 58574e803812..8c9dd685b616 100644
@@ -703,15 +703,6 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 	return 0;
 }
 
-void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm)
-{
-	if (!svm->nested.vmcb02.ptr)
-		return;
-
-	/* FIXME: merge g_pat from vmcb01 and vmcb12.  */
-	vmcb_set_gpat(svm->nested.vmcb02.ptr, svm->vmcb01.ptr->save.g_pat);
-}
-
 static bool nested_vmcb12_has_lbrv(struct kvm_vcpu *vcpu)
 {
 	return guest_cpu_cap_has(vcpu, X86_FEATURE_LBRV) &&
arch/x86/kvm/svm/svm.c +33 -3
index 56b6bd5dfdca..8d968ead6f45 100644
@@ -2767,6 +2767,20 @@ static bool sev_es_prevent_msr_access(struct kvm_vcpu *vcpu,
 	       !msr_write_intercepted(vcpu, msr_info->index);
 }
 
+static bool svm_pat_accesses_gpat(struct kvm_vcpu *vcpu, bool from_host)
+{
+	/*
+	 * When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and nested
+	 * NPT is enabled, L2 has a separate PAT from L1.  Guest accesses
+	 * to IA32_PAT while running L2 target L2's gPAT; host-initiated
+	 * accesses always target L1's hPAT so that KVM_GET/SET_MSRS and
+	 * KVM_GET/SET_NESTED_STATE are independent of each other and can
+	 * be ordered arbitrarily during save and restore.
+	 */
+	WARN_ON_ONCE(from_host && vcpu->wants_to_run);
+	return !from_host && is_guest_mode(vcpu) && l2_has_separate_pat(vcpu);
+}
+
 static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -2883,6 +2897,12 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_AMD64_DE_CFG:
 		msr_info->data = svm->msr_decfg;
 		break;
+	case MSR_IA32_CR_PAT:
+		if (svm_pat_accesses_gpat(vcpu, msr_info->host_initiated)) {
+			msr_info->data = svm->vmcb->save.g_pat;
+			break;
+		}
+		return kvm_get_msr_common(vcpu, msr_info);
 	default:
 		return kvm_get_msr_common(vcpu, msr_info);
 	}
@@ -2966,13 +2986,23 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 
 		break;
 	case MSR_IA32_CR_PAT:
+		if (svm_pat_accesses_gpat(vcpu, msr->host_initiated)) {
+			if (!kvm_pat_valid(data))
+				return 1;
+
+			vmcb_set_gpat(svm->vmcb, data);
+			break;
+		}
+
 		ret = kvm_set_msr_common(vcpu, msr);
 		if (ret)
 			break;
 
-		vmcb_set_gpat(svm->vmcb01.ptr, data);
-		if (is_guest_mode(vcpu))
-			nested_vmcb02_compute_g_pat(svm);
+		if (npt_enabled) {
+			vmcb_set_gpat(svm->vmcb01.ptr, data);
+			if (is_guest_mode(vcpu) && !l2_has_separate_pat(vcpu))
+				vmcb_set_gpat(svm->vmcb, data);
+		}
 		break;
 	case MSR_IA32_SPEC_CTRL:
 		if (!msr->host_initiated &&
arch/x86/kvm/svm/svm.h +0 -1
index fdd6286d965e..677e899bda33 100644
@@ -881,7 +881,6 @@ void nested_copy_vmcb_control_to_cache(struct vcpu_svm *svm,
 void nested_copy_vmcb_save_to_cache(struct vcpu_svm *svm,
 				    struct vmcb_save_area *save);
 void nested_sync_control_from_vmcb02(struct vcpu_svm *svm);
-void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm);
 void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#6 lore raw

Re: [PATCH] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 12 lines · Jim Mattson

When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and the vCPU is in guest mode with nested NPT enabled, guest accesses to IA32_PAT are redirected to the gPAT register, which is stored in VMCB02's g_pat field.

Non-guest accesses (e.g. from userspace) to IA32_PAT are always redirected to hPAT, which is stored in vcpu->arch.pat.

Directing host-initiated accesses to hPAT ensures that KVM_GET/SET_MSRS and KVM_GET/SET_NESTED_STATE are independent of each other and can be ordered arbitrarily during save and restore. gPAT is saved and restored separately via KVM_GET/SET_NESTED_STATE.

Quoted 5 lines

Use WARN_ON_ONCE to flag any host-initiated accesses originating from KVM itself rather than userspace.

Use pr_warn_once to flag any use of the common MSR-handling code (now shared by VMX and TDX) for IA32_PAT by a vCPU that is SVM-capable.

Changelog is stale, but otherwise this LGTM. I'll fixup the changelog when applying (in a few weeks).

#7 lore raw

Re: [PATCH] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 14 lines · Sean Christopherson
Quoted 12 lines · Jim Mattson

When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and the vCPU is in guest mode with nested NPT enabled, guest accesses to IA32_PAT are redirected to the gPAT register, which is stored in VMCB02's g_pat field.

Non-guest accesses (e.g. from userspace) to IA32_PAT are always redirected to hPAT, which is stored in vcpu->arch.pat.

Directing host-initiated accesses to hPAT ensures that KVM_GET/SET_MSRS and KVM_GET/SET_NESTED_STATE are independent of each other and can be ordered arbitrarily during save and restore. gPAT is saved and restored separately via KVM_GET/SET_NESTED_STATE.

Quoted 8 lines
Quoted 5 lines

Use WARN_ON_ONCE to flag any host-initiated accesses originating from KVM itself rather than userspace.

Use pr_warn_once to flag any use of the common MSR-handling code (now shared by VMX and TDX) for IA32_PAT by a vCPU that is SVM-capable.

Changelog is stale, but otherwise this LGTM. I'll fixup the changelog when applying (in a few weeks).

Oh, crud. This was supposed to be 5/8, but I made some changes after checkpatch.pl complained and then tried to just regenerate this one, but I totally flubbed it.

#8 lore raw

Re: [PATCH] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 21 lines · Jim Mattson
Quoted 20 lines · Sean Christopherson
Quoted 17 lines · Jim Mattson

When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and the vCPU is in guest mode with nested NPT enabled, guest accesses to IA32_PAT are redirected to the gPAT register, which is stored in VMCB02's g_pat field.

Non-guest accesses (e.g. from userspace) to IA32_PAT are always redirected to hPAT, which is stored in vcpu->arch.pat.

Directing host-initiated accesses to hPAT ensures that KVM_GET/SET_MSRS and KVM_GET/SET_NESTED_STATE are independent of each other and can be ordered arbitrarily during save and restore. gPAT is saved and restored separately via KVM_GET/SET_NESTED_STATE.

Use WARN_ON_ONCE to flag any host-initiated accesses originating from KVM itself rather than userspace.

Use pr_warn_once to flag any use of the common MSR-handling code (now shared by VMX and TDX) for IA32_PAT by a vCPU that is SVM-capable.

Quoted 6 lines
Quoted 2 lines

Changelog is stale, but otherwise this LGTM. I'll fixup the changelog when applying (in a few weeks).

Oh, crud. This was supposed to be 5/8, but I made some changes after checkpatch.pl complained and then tried to just regenerate this one, but I totally flubbed it.

Huh. The patch shows up when I grab the thread via b4 mbox and open it with mutt, but b4 am skips it. I'm guessing there's version-based filtering somewhere in b4.

No need for a v9 on my account, I can splice in 5/8 when applying.

#9 lore raw

PATCH v8 6/8 KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT

In reply to Jim Mattson

According to the APM volume 3 pseudo-code for "VMRUN," when nested paging is enabled in the vmcb, the guest PAT register (gPAT) is saved to the vmcb on emulated VMEXIT.

When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and the vCPU is in guest mode with nested NPT enabled, save the vmcb02 g_pat field to the vmcb12 g_pat field on emulated VMEXIT.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/svm/nested.c +3 -0
index 8c9dd685b616..cf6356c775e6 100644
@@ -1250,6 +1250,9 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu)
 	vmcb12->save.dr6    = svm->vcpu.arch.dr6;
 	vmcb12->save.cpl    = vmcb02->save.cpl;
 
+	if (l2_has_separate_pat(vcpu))
+		vmcb12->save.g_pat = vmcb02->save.g_pat;
+
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) {
 		vmcb12->save.s_cet	= vmcb02->save.s_cet;
 		vmcb12->save.isst_addr	= vmcb02->save.isst_addr;
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#10 lore raw

PATCH v8 7/8 KVM: Documentation: document KVM_{GET,SET}_NESTED_STATE for SVM

In reply to Jim Mattson

Document the nested state constants and structures for SVM that were added by commit cc440cdad5b7 ("KVM: nSVM: implement KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE").

Fixes cc440cdad5b7 ("KVM: nSVM: implement KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE")
---
 Documentation/virt/kvm/api.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)
Documentation/virt/kvm/api.rst +11 -0
index 2d56f17e3760..0a2d873ca5a3 100644
@@ -4942,10 +4942,13 @@ Errors:
   #define KVM_STATE_NESTED_FORMAT_SVM		1
 
   #define KVM_STATE_NESTED_VMX_VMCS_SIZE	0x1000
+  #define KVM_STATE_NESTED_SVM_VMCB_SIZE	0x1000
 
   #define KVM_STATE_NESTED_VMX_SMM_GUEST_MODE	0x00000001
   #define KVM_STATE_NESTED_VMX_SMM_VMXON	0x00000002
 
+  #define KVM_STATE_NESTED_GIF_SET		0x00000100
+
   #define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE 0x00000001
 
   struct kvm_vmx_nested_state_hdr {
@@ -4960,11 +4963,19 @@ Errors:
 	__u64 preemption_timer_deadline;
   };
 
+  struct kvm_svm_nested_state_hdr {
+	__u64 vmcb_pa;
+  };
+
   struct kvm_vmx_nested_state_data {
 	__u8 vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];
 	__u8 shadow_vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];
   };
 
+  struct kvm_svm_nested_state_data {
+	__u8 vmcb12[KVM_STATE_NESTED_SVM_VMCB_SIZE];
+  };
+
 This ioctl copies the vcpu's nested virtualization state from the kernel to
 userspace.
 
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#11 lore raw

PATCH v8 8/8 KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

In reply to Jim Mattson

Add a 'gpat' field to kvm_svm_nested_state_hdr to carry L2's guest PAT value across save and restore.

When KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT is disabled and the vCPU is in guest mode with nested NPT enabled, save vmcb02's g_pat into the header on KVM_GET_NESTED_STATE, and restore it on KVM_SET_NESTED_STATE.

Host-initiated accesses to IA32_PAT (via KVM_GET/SET_MSRS) always target L1's hPAT, so they cannot be used to save or restore gPAT. The separate header field ensures that KVM_GET/SET_MSRS and KVM_GET/SET_NESTED_STATE are independent and can be ordered arbitrarily during save and restore.

Note that struct kvm_svm_nested_state_hdr is included in a union padded to 120 bytes, so there is room to add the gpat field without changing any offsets.

Fixes cc440cdad5b7 ("KVM: nSVM: implement KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE")
---
 Documentation/virt/kvm/api.rst  |  1 +
 arch/x86/include/uapi/asm/kvm.h |  1 +
 arch/x86/kvm/svm/nested.c       | 19 +++++++++++++++++++
 3 files changed, 21 insertions(+)
Documentation/virt/kvm/api.rst +1 -0
index 0a2d873ca5a3..d6bbb7bad173 100644
@@ -4965,6 +4965,7 @@ Errors:
 
   struct kvm_svm_nested_state_hdr {
 	__u64 vmcb_pa;
+	__u64 gpat;
   };
 
   struct kvm_vmx_nested_state_data {
arch/x86/include/uapi/asm/kvm.h +1 -0
index 3ada2fa9ca86..1585ec804066 100644
@@ -533,6 +533,7 @@ struct kvm_svm_nested_state_data {
 
 struct kvm_svm_nested_state_hdr {
 	__u64 vmcb_pa;
+	__u64 gpat;
 };
 
 /* for KVM_CAP_NESTED_STATE */
arch/x86/kvm/svm/nested.c +19 -0
index cf6356c775e6..9682099193d4 100644
@@ -1868,6 +1868,9 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
 	/* First fill in the header and copy it out.  */
 	if (is_guest_mode(vcpu)) {
 		kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
+		kvm_state.hdr.svm.gpat = 0;
+		if (l2_has_separate_pat(vcpu))
+			kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat;
 		kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
 		kvm_state.flags |= KVM_STATE_NESTED_GUEST_MODE;
 
@@ -1920,6 +1923,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	struct vmcb_save_area *save;
 	struct vmcb_save_area_cached save_cached;
 	struct vmcb_ctrl_area_cached ctl_cached;
+	bool use_separate_l2_pat;
 	unsigned long cr0;
 	int ret;
 
@@ -1996,6 +2000,17 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	    !nested_vmcb_check_save(vcpu, &save_cached, false))
 		goto out_free;
 
+	/*
+	 * Validate gPAT when the shared PAT quirk is disabled (i.e. L2
+	 * has its own gPAT). This is done separately from the
+	 * vmcb_save_area_cached validation above, because gPAT is L2
+	 * state, but the vmcb_save_area_cached is populated with L1 state.
+	 */
+	use_separate_l2_pat = (ctl_cached.misc_ctl & SVM_MISC_ENABLE_NP) &&
+			      !kvm_check_has_quirk(vcpu->kvm,
+						   KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT);
+	if (use_separate_l2_pat && !kvm_pat_valid(kvm_state->hdr.svm.gpat))
+		goto out_free;
 
 	/*
 	 * All checks done, we can enter guest mode. Userspace provides
@@ -2022,6 +2037,10 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	nested_copy_vmcb_control_to_cache(svm, ctl);
 
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
+
+	if (use_separate_l2_pat)
+		vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat);
+
 	nested_vmcb02_prepare_control(svm);
 
 	/*
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
maintainer Applied
#12 lore raw
In reply to Jim Mattson
Quoted 6 lines · Jim Mattson

Currently, KVM's implementation of nested SVM treats the PAT MSR the same way whether or not nested NPT is enabled: L1 and L2 share a single PAT. However, the AMD APM specifies that when nested NPT is enabled, the host (L1) and the guest (L2) should have independent PATs: hPAT for L1 and gPAT for L2.

Quoted 7 lines

This patch series implements independent PATs for L1 and L2 when nested NPT is enabled, but only when a new quirk, KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT, is disabled. By default, the quirk is enabled, preserving KVM's legacy behavior. When the quirk is disabled, KVM correctly virtualizes a separate PAT register for L2, using the g_pat field in the VMCB.

[...]

Applied to kvm-x86 svm. Yosry and/or Jim, please double check the result, the goof with patch 5 was slightly more annoying than I was expecting.

Thanks!

[1/8] KVM: x86: Define KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT

      https://github.com/kvm-x86/linux/commit/822790ab0149
[2/8] KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode
      https://github.com/kvm-x86/linux/commit/0a8aeb15848e
[3/8] KVM: x86: nSVM: Cache and validate vmcb12 g_pat
      https://github.com/kvm-x86/linux/commit/4b83e4ba836e
[4/8] KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT
      https://github.com/kvm-x86/linux/commit/02233c73f8ae
[6/8] KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT
      https://github.com/kvm-x86/linux/commit/d65cf222b899
[7/8] KVM: Documentation: document KVM_{GET,SET}_NESTED_STATE for SVM
      https://github.com/kvm-x86/linux/commit/32ebdbce3b23
[8/8] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE
      https://github.com/kvm-x86/linux/commit/4f256d5770fe

--
https://github.com/kvm-x86/linux/tree/next

#13 lore raw
Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Jim Mattson

Currently, KVM's implementation of nested SVM treats the PAT MSR the same way whether or not nested NPT is enabled: L1 and L2 share a single PAT. However, the AMD APM specifies that when nested NPT is enabled, the host (L1) and the guest (L2) should have independent PATs: hPAT for L1 and gPAT for L2.

This patch series implements independent PATs for L1 and L2 when nested NPT is enabled, but only when a new quirk, KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT, is disabled. By default, the quirk is enabled, preserving KVM's legacy behavior. When the quirk is disabled, KVM correctly virtualizes a separate PAT register for L2, using the g_pat field in the VMCB.

Quoted 4 lines
Quoted 1 line

[...]

Applied to kvm-x86 svm. Yosry and/or Jim, please double check the result, the goof with patch 5 was slightly more annoying than I was expecting.

The result looks good to me. I also ran the selftest from v7 and it passes. I couldn't help myself from reworking it and cleaning it up, I will send a patch your way soon.

Quoted 20 lines

Thanks!

[1/8] KVM: x86: Define KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT

      https://github.com/kvm-x86/linux/commit/822790ab0149
[2/8] KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode
      https://github.com/kvm-x86/linux/commit/0a8aeb15848e
[3/8] KVM: x86: nSVM: Cache and validate vmcb12 g_pat
      https://github.com/kvm-x86/linux/commit/4b83e4ba836e
[4/8] KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT
      https://github.com/kvm-x86/linux/commit/02233c73f8ae
[6/8] KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT
      https://github.com/kvm-x86/linux/commit/d65cf222b899
[7/8] KVM: Documentation: document KVM_{GET,SET}_NESTED_STATE for SVM
      https://github.com/kvm-x86/linux/commit/32ebdbce3b23
[8/8] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE
      https://github.com/kvm-x86/linux/commit/4f256d5770fe

--
https://github.com/kvm-x86/linux/tree/next

#14 lore raw
Quoted 14 lines · Yosry Ahmed
Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Jim Mattson

Currently, KVM's implementation of nested SVM treats the PAT MSR the same way whether or not nested NPT is enabled: L1 and L2 share a single PAT. However, the AMD APM specifies that when nested NPT is enabled, the host (L1) and the guest (L2) should have independent PATs: hPAT for L1 and gPAT for L2.

This patch series implements independent PATs for L1 and L2 when nested NPT is enabled, but only when a new quirk, KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT, is disabled. By default, the quirk is enabled, preserving KVM's legacy behavior. When the quirk is disabled, KVM correctly virtualizes a separate PAT register for L2, using the g_pat field in the VMCB.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

[...]

Applied to kvm-x86 svm. Yosry and/or Jim, please double check the result, the goof with patch 5 was slightly more annoying than I was expecting.

The result looks good to me. I also ran the selftest from v7 and it passes. I couldn't help myself from reworking it and cleaning it up, I will send a patch your way soon.

As promised, if you want something you can directly run against the current kvm-x86 svm:
https://lore.kernel.org/kvm/[email protected]/