Skip to contentSkip to conversation

KVM: x86: nSVM: Improve PAT virtualization

PATCH Applied Jim Mattson 3 participants 22 messages Feb 24 – Mar 5

Subsystems (4)

v5 of 8 Applied Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v5 cover KVM: x86: nSVM: Improve PAT virtualization

Currently, KVM's implementation of nested SVM treats the PAT MSR the same way whether or not nested NPT is enabled: L1 and L2 share a single PAT. However, the APM specifies that when nested NPT is enabled, the host (L1) and the guest (L2) should have independent PATs: hPAT for L1 and gPAT for L2. This patch series implements the architectural specification in KVM.

Use the existing PAT MSR (vcpu->arch.pat) for hPAT. Add a new field, svm->nested.gpat, for gPAT. With nested NPT enabled, redirect guest accesses to the IA32_PAT MSR to gPAT. All other accesses, including userspace accesses via KVM_{GET,SET}_MSRS, continue to reference hPAT. The special handling of userspace accesses ensures save/restore forward compatibility (i.e. resuming a new checkpoint on an older kernel). When an old kernel restores a checkpoint from a new kernel, the gPAT will be lost, and L2 will simply use L1's PAT, which is the existing behavior of the old kernel anyway.

v1: https://lore.kernel.org/kvm/[email protected]/
v2: https://lore.kernel.org/kvm/[email protected]/
v3: https://lore.kernel.org/kvm/[email protected]/
v4: https://lore.kernel.org/kvm/[email protected]/

v4 -> v5:
* Separate commit to remove vmcb_is_dirty() from first v4 commit [Yosry, Sean]
* Introduce svm_get_pat and svm_set_pat to keep all hPAT vs gPAT logic together in one place.
* Remove the no longer common logic for get/set IA32_PAT. [Sean]
* Make vmcb02's g_pat authoritative for gPAT.
* Clear legacy_gpat_semantics when forcing the vCPU out of guest mode and when processing a second KVM_SET_NESTED_STATE that doesn't have legacy semantics. [Sean]

Note that this series should be applied after Yosry's v5 "Nested SVM fixes, cleanups, and hardening."

Jim Mattson (10):
KVM: SVM: Remove vmcb_is_dirty()
KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest
mode
KVM: x86: nSVM: Cache and validate vmcb12 g_pat
KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT
KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT
KVM: x86: Remove common handling of MSR_IA32_CR_PAT
KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT
KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE
KVM: x86: nSVM: Handle restore of legacy nested state
KVM: selftests: nSVM: Add svm_nested_pat test

 arch/x86/include/uapi/asm/kvm.h               |   5 +
 arch/x86/kvm/svm/nested.c                     |  63 +++-
 arch/x86/kvm/svm/svm.c                        |  57 +++-
 arch/x86/kvm/svm/svm.h                        |  20 +-
 arch/x86/kvm/vmx/vmx.c                        |   9 +-
 arch/x86/kvm/x86.c                            |   9 -
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/svm_nested_pat_test.c   | 298 ++++++++++++++++++
 8 files changed, 422 insertions(+), 40 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_pat_test.c
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#1 lore raw

PATCH v5 1/10 KVM: x86: SVM: Remove vmcb_is_dirty()

After commit dd26d1b5d6ed ("KVM: nSVM: Cache all used fields from VMCB12"), vmcb_is_dirty() has no callers. Remove the function.

---
 arch/x86/kvm/svm/svm.h | 5 -----
 1 file changed, 5 deletions(-)
arch/x86/kvm/svm/svm.h +0 -5
index 0bb93879abfe..8f9e6a39659c 100644
@@ -434,11 +434,6 @@ static inline void vmcb_mark_dirty(struct vmcb *vmcb, int bit)
 	vmcb->control.clean &= ~(1 << bit);
 }
 
-static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)
-{
-        return !test_bit(bit, (unsigned long *)&vmcb->control.clean);
-}
-
 static inline bool vmcb12_is_dirty(struct vmcb_ctrl_area_cached *control, int bit)
 {
 	return !test_bit(bit, (unsigned long *)&control->clean);
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#2 lore raw

PATCH v5 2/10 KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode

In reply to Jim Mattson

When running an L2 guest and writing to MSR_IA32_CR_PAT, the host PAT value is stored in both vmcb01's g_pat field and vmcb02's g_pat field, but the clean bit was only being cleared for vmcb02.

Introduce the helper vmcb_set_gpat() which sets vmcb->save.g_pat and marks the VMCB dirty for VMCB_NPT. Use this helper in both svm_set_msr() for updating vmcb01 and in nested_vmcb02_compute_g_pat() for updating vmcb02, ensuring both VMCBs' NPT fields are properly marked dirty.

Fixes 4995a3685f1b ("KVM: SVM: Use a separate vmcb for the nested L2 guest")
---
 arch/x86/kvm/svm/nested.c | 2 +-
 arch/x86/kvm/svm/svm.c    | 3 +--
 arch/x86/kvm/svm/svm.h    | 6 ++++++
 3 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/nested.c +1 -1
index d80b1bde6630..b72a1f3c4144 100644
@@ -707,7 +707,7 @@ void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm)
 		return;
 
 	/* FIXME: merge g_pat from vmcb01 and vmcb12.  */
-	svm->nested.vmcb02.ptr->save.g_pat = svm->vmcb01.ptr->save.g_pat;
+	vmcb_set_gpat(svm->nested.vmcb02.ptr, svm->vmcb01.ptr->save.g_pat);
 }
 
 static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
arch/x86/kvm/svm/svm.c +1 -2
index be9d562fabde..6c41f2317777 100644
@@ -2924,10 +2924,9 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 		if (ret)
 			break;
 
-		svm->vmcb01.ptr->save.g_pat = data;
+		vmcb_set_gpat(svm->vmcb01.ptr, data);
 		if (is_guest_mode(vcpu))
 			nested_vmcb02_compute_g_pat(svm);
-		vmcb_mark_dirty(svm->vmcb, VMCB_NPT);
 		break;
 	case MSR_IA32_SPEC_CTRL:
 		if (!msr->host_initiated &&
arch/x86/kvm/svm/svm.h +6 -0
index 8f9e6a39659c..9850ed01e16e 100644
@@ -439,6 +439,12 @@ static inline bool vmcb12_is_dirty(struct vmcb_ctrl_area_cached *control, int bi
 	return !test_bit(bit, (unsigned long *)&control->clean);
 }
 
+static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)
+{
+	vmcb->save.g_pat = data;
+	vmcb_mark_dirty(vmcb, VMCB_NPT);
+}
+
 static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
 {
 	return container_of(vcpu, struct vcpu_svm, vcpu);
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#3 lore raw

PATCH v5 3/10 KVM: x86: nSVM: Cache and validate vmcb12 g_pat

In reply to Jim Mattson

Cache g_pat from vmcb12 in vmcb_save_area_cached to avoid TOCTTOU issues, and add a validity check so that when nested paging is enabled for vmcb12, an invalid g_pat at emulated VMRUN causes an immediate VMEXIT with exit code VMEXIT_INVALID, as specified in the APM, volume 2: "Nested Paging and VMRUN/VMEXIT."

Fixes 3d6368ef580a ("KVM: SVM: Add VMRUN handler")
---
 arch/x86/kvm/svm/nested.c | 17 +++++++++++++----
 arch/x86/kvm/svm/svm.h    |  1 +
 2 files changed, 14 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/nested.c +13 -4
index b72a1f3c4144..91b35adb83f8 100644
@@ -426,7 +426,8 @@ static bool nested_vmcb_check_controls(struct kvm_vcpu *vcpu,
 
 /* Common checks that apply to both L1 and L2 state.  */
 static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
-				   struct vmcb_save_area_cached *save)
+				   struct vmcb_save_area_cached *save,
+				   bool check_gpat)
 {
 	if (CC(!(save->efer & EFER_SVME)))
 		return false;
@@ -462,6 +463,9 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
 	if (CC(!kvm_valid_efer(vcpu, save->efer)))
 		return false;
 
+	if (check_gpat && CC(!kvm_pat_valid(save->g_pat)))
+		return false;
+
 	return true;
 }
 
@@ -573,6 +577,7 @@ static void __nested_copy_vmcb_save_to_cache(struct vmcb_save_area_cached *to,
 
 	to->rax = from->rax;
 	to->cr2 = from->cr2;
+	to->g_pat = from->g_pat;
 
 	svm_copy_lbrs(to, from);
 }
@@ -1036,7 +1041,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
 
 	enter_guest_mode(vcpu);
 
-	if (!nested_vmcb_check_save(vcpu, &svm->nested.save) ||
+	if (!nested_vmcb_check_save(vcpu, &svm->nested.save,
+				    nested_npt_enabled(svm)) ||
 	    !nested_vmcb_check_controls(vcpu, &svm->nested.ctl,
 					svm->vmcb01.ptr->save.cr0))
 		return -EINVAL;
@@ -2006,13 +2012,16 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
 	/*
 	 * Validate host state saved from before VMRUN (see
-	 * nested_svm_check_permissions).
+	 * nested_svm_check_permissions). Note that the g_pat field is not
+	 * validated, because (a) it may have been clobbered by SMM before
+	 * KVM_GET_NESTED_STATE, and (b) it is not loaded at emulated
+	 * #VMEXIT.
 	 */
 	__nested_copy_vmcb_save_to_cache(&save_cached, save);
 	if (!(save->cr0 & X86_CR0_PG) ||
 	    !(save->cr0 & X86_CR0_PE) ||
 	    (save->rflags & X86_EFLAGS_VM) ||
-	    !nested_vmcb_check_save(vcpu, &save_cached))
+	    !nested_vmcb_check_save(vcpu, &save_cached, false))
 		goto out_free;
 
 
arch/x86/kvm/svm/svm.h +1 -0
index 9850ed01e16e..a49c48459e0b 100644
@@ -161,6 +161,7 @@ struct vmcb_save_area_cached {
 	u64 isst_addr;
 	u64 rax;
 	u64 cr2;
+	u64 g_pat;
 	u64 dbgctl;
 	u64 br_from;
 	u64 br_to;
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#4 lore raw

PATCH v5 4/10 KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT

In reply to Jim Mattson

When nested NPT is enabled in vmcb12, copy the (cached and validated) vmcb12 g_pat field to the guest PAT register. Under KVM, the guest PAT register lives in svm->nested.save.g_pat.

When NPT is enabled, but nested NPT is disabled, copy L1's IA32_PAT MSR to the vmcb02 g_pat field, since L2 shares the IA32_PAT MSR with L1.

When NPT is disabled, the g_pat field is ignored by hardware.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/nested.c +7 -3
index 91b35adb83f8..dc8275837120 100644
@@ -724,9 +724,6 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 	struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 
-	nested_vmcb02_compute_g_pat(svm);
-	vmcb_mark_dirty(vmcb02, VMCB_NPT);
-
 	/* Load the nested guest state */
 	if (svm->nested.vmcb12_gpa != svm->nested.last_vmcb12_gpa) {
 		new_vmcb12 = true;
@@ -757,6 +754,13 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 		vmcb_mark_dirty(vmcb02, VMCB_CET);
 	}
 
+	if (nested_npt_enabled(svm)) {
+		if (unlikely(new_vmcb12 || vmcb12_is_dirty(control, VMCB_NPT)))
+			vmcb_set_gpat(vmcb02, svm->nested.save.g_pat);
+	} else if (npt_enabled) {
+		vmcb_set_gpat(vmcb02, vcpu->arch.pat);
+	}
+
 	kvm_set_rflags(vcpu, save->rflags | X86_EFLAGS_FIXED);
 
 	svm_set_efer(vcpu, svm->nested.save.efer);
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#5 lore raw

PATCH v5 5/10 KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

In reply to Jim Mattson

When the vCPU is in guest mode with nested NPT enabled, guest accesses to IA32_PAT are redirected to the gPAT register, which is stored in VMCB02's g_pat field.

Non-guest accesses (e.g. from userspace) to IA32_PAT are always redirected to hPAT, which is stored in vcpu->arch.pat.

This is architected behavior. It also makes it possible to restore a new checkpoint on an old kernel with reasonable semantics. After the restore, gPAT will be lost, and L2 will run on L1's PAT. Note that the old kernel would have always run L2 on L1's PAT.

Add WARN_ON_ONCE to flag any host-initiated accesses originating from KVM itself rather than userspace.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c |  9 -------
 arch/x86/kvm/svm/svm.c    | 52 ++++++++++++++++++++++++++++++++++-----
 arch/x86/kvm/svm/svm.h    |  1 -
 3 files changed, 46 insertions(+), 16 deletions(-)
arch/x86/kvm/svm/nested.c +0 -9
index dc8275837120..69b577a4915c 100644
@@ -706,15 +706,6 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 	return 0;
 }
 
-void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm)
-{
-	if (!svm->nested.vmcb02.ptr)
-		return;
-
-	/* FIXME: merge g_pat from vmcb01 and vmcb12.  */
-	vmcb_set_gpat(svm->nested.vmcb02.ptr, svm->vmcb01.ptr->save.g_pat);
-}
-
 static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 {
 	struct vmcb_ctrl_area_cached *control = &svm->nested.ctl;
arch/x86/kvm/svm/svm.c +46 -6

71 lines not on this page · raw message

arch/x86/kvm/svm/svm.h +0 -1
index a49c48459e0b..58b0b935d049 100644
@@ -840,7 +840,6 @@ void nested_copy_vmcb_control_to_cache(struct vcpu_svm *svm,
 void nested_copy_vmcb_save_to_cache(struct vcpu_svm *svm,
 				    struct vmcb_save_area *save);
 void nested_sync_control_from_vmcb02(struct vcpu_svm *svm);
-void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm);
 void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#6 lore raw

Re: [PATCH v5 05/10] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 5 lines · Jim Mattson
arch/x86/kvm/svm/svm.c +5 -0
+static void svm_set_pat(struct kvm_vcpu *vcpu, bool from_host, u64 data)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+
+	if (svm_pat_accesses_gpat(vcpu, from_host)) {

arch/x86/kvm/svm/svm.c static bool sev_es_prevent_msr_access(struct kvm_vcpu *vcpu, · line 2755

Quoted 10 lines
arch/x86/kvm/svm/svm.c +10 -0
+		vmcb_set_gpat(svm->vmcb, data);
+	} else {
+		svm->vcpu.arch.pat = data;
+		if (npt_enabled) {
+			vmcb_set_gpat(svm->vmcb01.ptr, data);
+			if (is_guest_mode(&svm->vcpu) &&
+			    !nested_npt_enabled(svm))
+				vmcb_set_gpat(svm->vmcb, data);
+		}
+	}

Overall, this LGTM. For this particular code, any objection to using early returns to reduce indentation? The else branch above is a bit gnarly, especially when legacy_gpat_semantics comes along.

I.e. end up with this

static void svm_set_pat(struct kvm_vcpu *vcpu, bool from_host, u64 data)

  {
	struct vcpu_svm *svm = to_svm(vcpu);

	if (svm_pat_accesses_gpat(vcpu, from_host)) {
		vmcb_set_gpat(svm->vmcb, data);
		return;
	}

	svm->vcpu.arch.pat = data;

	if (!npt_enabled)
		return;

	vmcb_set_gpat(svm->vmcb01.ptr, data);
	if (is_guest_mode(&svm->vcpu) &&
	    (svm->nested.legacy_gpat_semantics || !nested_npt_enabled(svm)))
		vmcb_set_gpat(svm->vmcb, data);

}

I can fixup when applying (unless you and/or Yosry object).

#7 lore raw

Re: [PATCH v5 05/10] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 5 lines · Sean Christopherson

Overall, this LGTM. For this particular code, any objection to using early returns to reduce indentation? The else branch above is a bit gnarly, especially when legacy_gpat_semantics comes along.
...
I can fixup when applying (unless you and/or Yosry object).

No objection.

#8 lore raw

Re: [PATCH v5 05/10] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

In reply to Sean Christopherson
Quoted 42 lines · Sean Christopherson
Quoted 5 lines · Jim Mattson
arch/x86/kvm/svm/svm.c +5 -0
+static void svm_set_pat(struct kvm_vcpu *vcpu, bool from_host, u64 data)
+{
+     struct vcpu_svm *svm = to_svm(vcpu);
+
+     if (svm_pat_accesses_gpat(vcpu, from_host)) {

arch/x86/kvm/svm/svm.c static bool sev_es_prevent_msr_access(struct kvm_vcpu *vcpu, · line 2755

Quoted 10 lines
arch/x86/kvm/svm/svm.c +10 -0
+             vmcb_set_gpat(svm->vmcb, data);
+     } else {
+             svm->vcpu.arch.pat = data;
+             if (npt_enabled) {
+                     vmcb_set_gpat(svm->vmcb01.ptr, data);
+                     if (is_guest_mode(&svm->vcpu) &&
+                         !nested_npt_enabled(svm))
+                             vmcb_set_gpat(svm->vmcb, data);
+             }
+     }

Overall, this LGTM. For this particular code, any objection to using early returns to reduce indentation? The else branch above is a bit gnarly, especially when legacy_gpat_semantics comes along.

I.e. end up with this

  static void svm_set_pat(struct kvm_vcpu *vcpu, bool from_host, u64 data)
  {
        struct vcpu_svm *svm = to_svm(vcpu);
        if (svm_pat_accesses_gpat(vcpu, from_host)) {
                vmcb_set_gpat(svm->vmcb, data);
                return;
        }

        svm->vcpu.arch.pat = data;

        if (!npt_enabled)
                return;

        vmcb_set_gpat(svm->vmcb01.ptr, data);
        if (is_guest_mode(&svm->vcpu) &&
            (svm->nested.legacy_gpat_semantics || !nested_npt_enabled(svm)))
                vmcb_set_gpat(svm->vmcb, data);
Quoted 3 lines

}

I can fixup when applying (unless you and/or Yosry object).

LGTM.

#9 lore raw

PATCH v5 6/10 KVM: x86: Remove common handling of MSR_IA32_CR_PAT

In reply to Jim Mattson

SVM now has completely independent handling of MSR_IA32_CR_PAT in svm_get_msr() and svm_set_msr().

To avoid any confusion, move the logic for MSR_IA32_CR_PAT from kvm_get_msr_common() and kvm_set_msr_common() into vmx_get_msr() and vmx_set_msr().

---
 arch/x86/kvm/vmx/vmx.c | 9 ++++++---
 arch/x86/kvm/x86.c     | 9 ---------
 2 files changed, 6 insertions(+), 12 deletions(-)
arch/x86/kvm/vmx/vmx.c +6 -3
index 967b58a8ab9d..f5127dbd9104 100644
@@ -2146,6 +2146,9 @@ int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		    !(vcpu->arch.arch_capabilities & ARCH_CAP_TSX_CTRL_MSR))
 			return 1;
 		goto find_uret_msr;
+	case MSR_IA32_CR_PAT:
+		msr_info->data = vcpu->arch.pat;
+		break;
 	case MSR_IA32_UMWAIT_CONTROL:
 		if (!msr_info->host_initiated && !vmx_has_waitpkg(vmx))
 			return 1;
@@ -2468,10 +2471,10 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 			return 1;
 		goto find_uret_msr;
 	case MSR_IA32_CR_PAT:
-		ret = kvm_set_msr_common(vcpu, msr_info);
-		if (ret)
-			break;
+		if (!kvm_pat_valid(data))
+			return 1;
 
+		vcpu->arch.pat = data;
 		if (is_guest_mode(vcpu) &&
 		    get_vmcs12(vcpu)->vm_exit_controls & VM_EXIT_SAVE_IA32_PAT)
 			get_vmcs12(vcpu)->guest_ia32_pat = data;
arch/x86/kvm/x86.c +0 -9
index 416899b5dbe4..41936f83a17f 100644
@@ -4025,12 +4025,6 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 			return 1;
 		}
 		break;
-	case MSR_IA32_CR_PAT:
-		if (!kvm_pat_valid(data))
-			return 1;
-
-		vcpu->arch.pat = data;
-		break;
 	case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000:
 	case MSR_MTRRdefType:
 		return kvm_mtrr_set_msr(vcpu, msr, data);
@@ -4436,9 +4430,6 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		msr_info->data = kvm_scale_tsc(rdtsc(), ratio) + offset;
 		break;
 	}
-	case MSR_IA32_CR_PAT:
-		msr_info->data = vcpu->arch.pat;
-		break;
 	case MSR_MTRRcap:
 	case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000:
 	case MSR_MTRRdefType:
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#10 lore raw

PATCH v5 7/10 KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT

In reply to Jim Mattson

According to the APM volume 3 pseudo-code for "VMRUN," when nested paging is enabled in the vmcb, the guest PAT register (gPAT) is saved to the vmcb on emulated VMEXIT.

When nested NPT is enabled, save the vmcb02 g_pat field to the vmcb12 g_pat field on emulated VMEXIT.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/svm/nested.c +3 -0
index 69b577a4915c..26f758e294ab 100644
@@ -1312,6 +1312,9 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 	vmcb12->save.dr6    = svm->vcpu.arch.dr6;
 	vmcb12->save.cpl    = vmcb02->save.cpl;
 
+	if (nested_npt_enabled(svm))
+		vmcb12->save.g_pat = vmcb02->save.g_pat;
+
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) {
 		vmcb12->save.s_cet	= vmcb02->save.s_cet;
 		vmcb12->save.isst_addr	= vmcb02->save.isst_addr;
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#11 lore raw

PATCH v5 8/10 KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

In reply to Jim Mattson

Add a 'flags' field to the SVM nested state header, and use bit 0 of the flags to indicate that gPAT is stored in the nested state.

If in guest mode with NPT enabled, store the current vmcb->save.g_pat value into the header of the nested state, and set the flag.

Note that struct kvm_svm_nested_state_hdr is included in a union padded to 120 bytes, so there is room to add the flags field and the gpat field without changing any offsets.

Fixes cc440cdad5b7 ("KVM: nSVM: implement KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE")
---
 arch/x86/include/uapi/asm/kvm.h |  5 +++++
 arch/x86/kvm/svm/nested.c       | 17 +++++++++++++++++
 2 files changed, 22 insertions(+)
arch/x86/include/uapi/asm/kvm.h +5 -0
index 846a63215ce1..664d04d1db3f 100644
@@ -495,6 +495,8 @@ struct kvm_sync_regs {
 
 #define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE	0x00000001
 
+#define KVM_STATE_SVM_VALID_GPAT	0x00000001
+
 /* vendor-independent attributes for system fd (group 0) */
 #define KVM_X86_GRP_SYSTEM		0
 #  define KVM_X86_XCOMP_GUEST_SUPP	0
@@ -531,6 +533,9 @@ struct kvm_svm_nested_state_data {
 
 struct kvm_svm_nested_state_hdr {
 	__u64 vmcb_pa;
+	__u32 flags;
+	__u32 reserved;
+	__u64 gpat;
 };
 
 /* for KVM_CAP_NESTED_STATE */
arch/x86/kvm/svm/nested.c +17 -0
index 26f758e294ab..5a35277f2364 100644
@@ -1893,6 +1893,10 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
 	/* First fill in the header and copy it out.  */
 	if (is_guest_mode(vcpu)) {
 		kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
+		if (nested_npt_enabled(svm)) {
+			kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+			kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat;
+		}
 		kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
 		kvm_state.flags |= KVM_STATE_NESTED_GUEST_MODE;
 
@@ -2022,6 +2026,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	    !nested_vmcb_check_save(vcpu, &save_cached, false))
 		goto out_free;
 
+	/*
+	 * Validate gPAT, if provided. This is done separately from the
+	 * vmcb_save_area_cached validation above, because gPAT is L2
+	 * state, but the vmcb_save_area_cached is populated with L1 state.
+	 */
+	if ((kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT) &&
+	    !kvm_pat_valid(kvm_state->hdr.svm.gpat))
+		goto out_free;
 
 	/*
 	 * All checks done, we can enter guest mode. Userspace provides
@@ -2062,6 +2074,11 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 		goto out_free;
 
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
+
+	if (nested_npt_enabled(svm) &&
+	    (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT))
+		vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat);
+
 	nested_vmcb02_prepare_control(svm, svm->vmcb->save.rip, svm->vmcb->save.cs.base);
 
 	svm->nested.force_msr_bitmap_recalc = true;
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#12 lore raw

Re: [PATCH v5 08/10] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

Quoted 46 lines · Jim Mattson

Add a 'flags' field to the SVM nested state header, and use bit 0 of the flags to indicate that gPAT is stored in the nested state.

If in guest mode with NPT enabled, store the current vmcb->save.g_pat value into the header of the nested state, and set the flag.

Note that struct kvm_svm_nested_state_hdr is included in a union padded to 120 bytes, so there is room to add the flags field and the gpat field without changing any offsets.

Fixes cc440cdad5b7 ("KVM: nSVM: implement KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE")
---
 arch/x86/include/uapi/asm/kvm.h |  5 +++++
 arch/x86/kvm/svm/nested.c       | 17 +++++++++++++++++
 2 files changed, 22 insertions(+)
arch/x86/include/uapi/asm/kvm.h +5 -0
index 846a63215ce1..664d04d1db3f 100644
@@ -495,6 +495,8 @@ struct kvm_sync_regs {
 
 #define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE	0x00000001
 
+#define KVM_STATE_SVM_VALID_GPAT	0x00000001
+
 /* vendor-independent attributes for system fd (group 0) */
 #define KVM_X86_GRP_SYSTEM		0
 #  define KVM_X86_XCOMP_GUEST_SUPP	0
@@ -531,6 +533,9 @@ struct kvm_svm_nested_state_data {
 
 struct kvm_svm_nested_state_hdr {
 	__u64 vmcb_pa;
+	__u32 flags;
+	__u32 reserved;
+	__u64 gpat;
 };
 
 /* for KVM_CAP_NESTED_STATE */
arch/x86/kvm/svm/nested.c +0 -0
index 26f758e294ab..5a35277f2364 100644

arch/x86/kvm/svm/nested.c static int svm_get_nested_state(struct kvm_vcpu *vcpu, · line 1897

Quoted 5 lines
arch/x86/kvm/svm/nested.c +2 -0
@@ -1893,6 +1893,10 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
 	/* First fill in the header and copy it out.  */
 	if (is_guest_mode(vcpu)) {
 		kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
+		if (nested_npt_enabled(svm)) {
+			kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;

Bugger. This isn't going to work. KVM doesn't reserve any bytes in the header for SVM, so there's no guarantee/requirement that old userspace won't provide garbage. The flag needs to go in kvm_nested_state.flags. We only reserved space for 16 flags between VMX and SVM, but that's a future problem as we can always add e.g. KVM_STATE_NESTED_EXT_FLAGS when we've exausted the current space.

Ugh. And vmx_set_nested_state() doesn't check for unsupported flags, at all. But that too is largely a future problem though, i.e. is a non-issue until nVMX wants to add a new flag. *sigh*

Anyways, for gPAT, unless I'm missing something, I'm going to squash this:

arch/x86/include/uapi/asm/kvm.h +1 -4
index 664d04d1db3f..0c1f97c9a2d8 100644
@@ -485,6 +485,7 @@ struct kvm_sync_regs {
 #define KVM_STATE_NESTED_EVMCS         0x00000004
 #define KVM_STATE_NESTED_MTF_PENDING   0x00000008
 #define KVM_STATE_NESTED_GIF_SET       0x00000100
+#define KVM_STATE_NESTED_GPAT_VALID    0x00000200
 
 #define KVM_STATE_NESTED_SMM_GUEST_MODE        0x00000001
 #define KVM_STATE_NESTED_SMM_VMXON     0x00000002
@@ -495,8 +496,6 @@ struct kvm_sync_regs {
 
 #define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE        0x00000001
 
-#define KVM_STATE_SVM_VALID_GPAT       0x00000001
-
 /* vendor-independent attributes for system fd (group 0) */
 #define KVM_X86_GRP_SYSTEM             0
 #  define KVM_X86_XCOMP_GUEST_SUPP     0
@@ -533,8 +532,6 @@ struct kvm_svm_nested_state_data {
 
 struct kvm_svm_nested_state_hdr {
        __u64 vmcb_pa;
-       __u32 flags;
-       __u32 reserved;
        __u64 gpat;
 };
 
arch/x86/kvm/svm/nested.c +5 -5
index 991ee4c03363..099bf8ac10ee 100644
@@ -1848,7 +1848,7 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
        if (is_guest_mode(vcpu)) {
                kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
                if (nested_npt_enabled(svm)) {
-                       kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+                       kvm_state->flags |= KVM_STATE_NESTED_GPAT_VALID;
                        kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat;
                }
                kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
@@ -1914,7 +1914,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
        if (kvm_state->flags & ~(KVM_STATE_NESTED_GUEST_MODE |
                                 KVM_STATE_NESTED_RUN_PENDING |
-                                KVM_STATE_NESTED_GIF_SET))
+                                KVM_STATE_NESTED_GIF_SET |
+                                KVM_STATE_NESTED_GPAT_VALID))
                return -EINVAL;
 
        /*
@@ -1984,7 +1985,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
         * vmcb_save_area_cached validation above, because gPAT is L2
         * state, but the vmcb_save_area_cached is populated with L1 state.
         */
-       if ((kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT) &&
+       if ((kvm_state->flags & KVM_STATE_NESTED_GPAT_VALID) &&
            !kvm_pat_valid(kvm_state->hdr.svm.gpat))
                goto out_free;
 
@@ -2013,7 +2014,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
        svm_switch_vmcb(svm, &svm->nested.vmcb02);
 
        if (nested_npt_enabled(svm)) {
-               if (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT)
+               if (kvm_state->flags & KVM_STATE_NESTED_GPAT_VALID)
                        vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat);
        }
--

One could argue we should even be more paranoid and do this as well:

	if (!(kvm_state->flags & KVM_STATE_NESTED_GUEST_MODE)) {
		if (kvm_state->flags & KVM_STATE_NESTED_GPAT_VALID)
			return -EINVAL:

		svm_leave_nested(vcpu);
		svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
		return 0;
	}

but KVM doesn't enforce GUEST_MODE for KVM_STATE_NESTED_RUN_PENDING, and it's easy to just ignore gPAT, so I'm inclined to not bother.

#13 lore raw

Re: [PATCH v5 08/10] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

Quoted 14 lines · Sean Christopherson
arch/x86/kvm/svm/nested.c +1 -1
index 991ee4c03363..099bf8ac10ee 100644
@@ -1848,7 +1848,7 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
        if (is_guest_mode(vcpu)) {
                kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
                if (nested_npt_enabled(svm)) {
-                       kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+                       kvm_state->flags |= KVM_STATE_NESTED_GPAT_VALID;
                        kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat;
                }
                kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1919

Quoted 7 lines
arch/x86/kvm/svm/nested.c +2 -1
@@ -1914,7 +1914,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
        if (kvm_state->flags & ~(KVM_STATE_NESTED_GUEST_MODE |
                                 KVM_STATE_NESTED_RUN_PENDING |
-                                KVM_STATE_NESTED_GIF_SET))
+                                KVM_STATE_NESTED_GIF_SET |
+                                KVM_STATE_NESTED_GPAT_VALID))
                return -EINVAL;

Unless I'm missing something, this breaks forward compatibility completely. An older kernel will refuse to accept a nested state blob with GPAT_VALID set.

#14 lore raw

Re: [PATCH v5 08/10] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

Quoted 16 lines · Jim Mattson
Quoted 15 lines · Sean Christopherson
arch/x86/kvm/svm/nested.c +1 -1
index 991ee4c03363..099bf8ac10ee 100644
@@ -1848,7 +1848,7 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
        if (is_guest_mode(vcpu)) {
                kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
                if (nested_npt_enabled(svm)) {
-                       kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+                       kvm_state->flags |= KVM_STATE_NESTED_GPAT_VALID;
                        kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat;
                }
                kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
@@ -1914,7 +1914,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
Quoted 10 lines

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1919

Quoted 6 lines
arch/x86/kvm/svm/nested.c +2 -1
        if (kvm_state->flags & ~(KVM_STATE_NESTED_GUEST_MODE |
                                 KVM_STATE_NESTED_RUN_PENDING |
-                                KVM_STATE_NESTED_GIF_SET))
+                                KVM_STATE_NESTED_GIF_SET |
+                                KVM_STATE_NESTED_GPAT_VALID))
                return -EINVAL;

Unless I'm missing something, this breaks forward compatibility completely. An older kernel will refuse to accept a nested state blob with GPAT_VALID set.

Argh, so we've painted ourselves into an impossible situation by restricting the set of valid flags. I.e. VMX's omission of checks on unknown flags is a feature, not a bug.

Chatted with Jim offlist, and he pointed out that KVM's standard way to deal with this is to make setting the flag opt-in, e.g. KVM_CAP_X86_TRIPLE_FAULT_EVENT and KVM_CAP_EXCEPTION_PAYLOAD.

As much as I want to retroactively change KVM's documentation to state doing KVM_SET_NESTED_STATE with data that didn't come from KVM_GET_NESTED_STATE is unsupported, that feels too restrictive and could really bite us in the future. And it doesn't help if there's already userspace that's putting garbage into the header.

So yeah, I don't see a better option than adding yet another capability.

Can you send a new version based on kvm-x86 next? (give me ~hour to drop these and push). This has snowballed beyond what I'm comfortable doing as fixup. :-(

#15 lore raw

Re: [PATCH v5 08/10] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

Quoted 43 lines · Sean Christopherson
Quoted 16 lines · Jim Mattson
Quoted 15 lines · Sean Christopherson
arch/x86/kvm/svm/nested.c +1 -1
index 991ee4c03363..099bf8ac10ee 100644
@@ -1848,7 +1848,7 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
        if (is_guest_mode(vcpu)) {
                kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
                if (nested_npt_enabled(svm)) {
-                       kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+                       kvm_state->flags |= KVM_STATE_NESTED_GPAT_VALID;
                        kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat;
                }
                kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
@@ -1914,7 +1914,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
Quoted 10 lines

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1919

Quoted 6 lines
arch/x86/kvm/svm/nested.c +2 -1
        if (kvm_state->flags & ~(KVM_STATE_NESTED_GUEST_MODE |
                                 KVM_STATE_NESTED_RUN_PENDING |
-                                KVM_STATE_NESTED_GIF_SET))
+                                KVM_STATE_NESTED_GIF_SET |
+                                KVM_STATE_NESTED_GPAT_VALID))
                return -EINVAL;

Unless I'm missing something, this breaks forward compatibility completely. An older kernel will refuse to accept a nested state blob with GPAT_VALID set.

Argh, so we've painted ourselves into an impossible situation by restricting the set of valid flags. I.e. VMX's omission of checks on unknown flags is a feature, not a bug.

Chatted with Jim offlist, and he pointed out that KVM's standard way to deal with this is to make setting the flag opt-in, e.g. KVM_CAP_X86_TRIPLE_FAULT_EVENT and KVM_CAP_EXCEPTION_PAYLOAD.

As much as I want to retroactively change KVM's documentation to state doing KVM_SET_NESTED_STATE with data that didn't come from KVM_GET_NESTED_STATE is unsupported, that feels too restrictive and could really bite us in the future. And it doesn't help if there's already userspace that's putting garbage into the header.

Quoted 4 lines

So yeah, I don't see a better option than adding yet another capability.

Can you send a new version based on kvm-x86 next? (give me ~hour to drop these and push). This has snowballed beyond what I'm comfortable doing as fixup. :-(

Will the current patches still be reachable through kvm-x86-next-2026.03.03? I imagine Jim will want to pull those and change them directly as they have all your fixups (rather than reconstructing them).

#16 lore raw

Re: [PATCH v5 08/10] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

Quoted 46 lines · Yosry Ahmed
Quoted 45 lines · Sean Christopherson
Quoted 16 lines · Jim Mattson
Quoted 15 lines · Sean Christopherson
arch/x86/kvm/svm/nested.c +1 -1
index 991ee4c03363..099bf8ac10ee 100644
@@ -1848,7 +1848,7 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
        if (is_guest_mode(vcpu)) {
                kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
                if (nested_npt_enabled(svm)) {
-                       kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+                       kvm_state->flags |= KVM_STATE_NESTED_GPAT_VALID;
                        kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat;
                }
                kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
@@ -1914,7 +1914,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
Quoted 10 lines

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1919

Quoted 6 lines
arch/x86/kvm/svm/nested.c +2 -1
        if (kvm_state->flags & ~(KVM_STATE_NESTED_GUEST_MODE |
                                 KVM_STATE_NESTED_RUN_PENDING |
-                                KVM_STATE_NESTED_GIF_SET))
+                                KVM_STATE_NESTED_GIF_SET |
+                                KVM_STATE_NESTED_GPAT_VALID))
                return -EINVAL;

Unless I'm missing something, this breaks forward compatibility completely. An older kernel will refuse to accept a nested state blob with GPAT_VALID set.

Argh, so we've painted ourselves into an impossible situation by restricting the set of valid flags. I.e. VMX's omission of checks on unknown flags is a feature, not a bug.

Chatted with Jim offlist, and he pointed out that KVM's standard way to deal with this is to make setting the flag opt-in, e.g. KVM_CAP_X86_TRIPLE_FAULT_EVENT and KVM_CAP_EXCEPTION_PAYLOAD.

As much as I want to retroactively change KVM's documentation to state doing KVM_SET_NESTED_STATE with data that didn't come from KVM_GET_NESTED_STATE is unsupported, that feels too restrictive and could really bite us in the future. And it doesn't help if there's already userspace that's putting garbage into the header.

So yeah, I don't see a better option than adding yet another capability.

Quoted 7 lines
Quoted 2 lines

Can you send a new version based on kvm-x86 next? (give me ~hour to drop these and push). This has snowballed beyond what I'm comfortable doing as fixup. :-(

Will the current patches still be reachable through kvm-x86-next-2026.03.03? I imagine Jim will want to pull those and change them directly as they have all your fixups (rather than reconstructing them).

Oh, yeah! I was actually going to self-respond with that suggestion, then squirrel!

#17 lore raw

Re: [PATCH v5 08/10] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

Quoted 40 lines · Sean Christopherson
Quoted 38 lines · Yosry Ahmed
Quoted 37 lines · Sean Christopherson
Quoted 16 lines · Jim Mattson
Quoted 15 lines · Sean Christopherson
arch/x86/kvm/svm/nested.c +1 -1
index 991ee4c03363..099bf8ac10ee 100644
@@ -1848,7 +1848,7 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
        if (is_guest_mode(vcpu)) {
                kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
                if (nested_npt_enabled(svm)) {
-                       kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+                       kvm_state->flags |= KVM_STATE_NESTED_GPAT_VALID;
                        kvm_state.hdr.svm.gpat = svm->vmcb->save.g_pat;
                }
                kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
@@ -1914,7 +1914,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
Quoted 10 lines

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 1919

Quoted 6 lines
arch/x86/kvm/svm/nested.c +2 -1
        if (kvm_state->flags & ~(KVM_STATE_NESTED_GUEST_MODE |
                                 KVM_STATE_NESTED_RUN_PENDING |
-                                KVM_STATE_NESTED_GIF_SET))
+                                KVM_STATE_NESTED_GIF_SET |
+                                KVM_STATE_NESTED_GPAT_VALID))
                return -EINVAL;

Unless I'm missing something, this breaks forward compatibility completely. An older kernel will refuse to accept a nested state blob with GPAT_VALID set.

Argh, so we've painted ourselves into an impossible situation by restricting the set of valid flags. I.e. VMX's omission of checks on unknown flags is a feature, not a bug.

Chatted with Jim offlist, and he pointed out that KVM's standard way to deal with this is to make setting the flag opt-in, e.g. KVM_CAP_X86_TRIPLE_FAULT_EVENT and KVM_CAP_EXCEPTION_PAYLOAD.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

As much as I want to retroactively change KVM's documentation to state doing KVM_SET_NESTED_STATE with data that didn't come from KVM_GET_NESTED_STATE is unsupported, that feels too restrictive and could really bite us in the future. And it doesn't help if there's already userspace that's putting garbage into the header.

So yeah, I don't see a better option than adding yet another capability.

Capability or quirk?

/me ducks.

#18 lore raw

PATCH v5 9/10 KVM: x86: nSVM: Handle restore of legacy nested state

In reply to Jim Mattson

When nested NPT is enabled and KVM_SET_NESTED_STATE is used to restore an old checkpoint (without a valid gPAT), the current IA32_PAT value must be used as L2's gPAT.

Unfortunately, checkpoint restore is non-atomic, and the order in which state components are restored is not specified. Hence, the current IA32_PAT value may be restored by KVM_SET_MSRS after KVM_SET_NESTED_STATE. To further complicate matters, there may be a KVM_GET_NESTED_STATE before the next KVM_RUN.

Introduce a new boolean, svm->nested.legacy_gpat_semantics. When set, hPAT updates are also applied to gPAT, preserving the old behavior (i.e. L2 shares L1's PAT). Set this boolean when restoring legacy state (i.e. nested NPT is enabled, but no GPAT is provided) in KVM_SET_NESTED_STATE. Clear this boolean in svm_vcpu_pre_run(), to ensure that hPAT and gPAT are decoupled before the vCPU resumes execution. Also clear this boolean when the vCPU is forced out of guest mode by svm_leave_nested().

---
 arch/x86/kvm/svm/nested.c | 13 ++++++++++---
 arch/x86/kvm/svm/svm.c    |  8 ++++++--
 arch/x86/kvm/svm/svm.h    |  9 +++++++++
 3 files changed, 25 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/nested.c +10 -3
index 5a35277f2364..b68eddcbc217 100644
@@ -1537,6 +1537,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
 	if (is_guest_mode(vcpu)) {
 		svm->nested.nested_run_pending = 0;
 		svm->nested.vmcb12_gpa = INVALID_GPA;
+		svm->nested.legacy_gpat_semantics = false;
 
 		leave_guest_mode(vcpu);
 
@@ -2075,9 +2076,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
 
-	if (nested_npt_enabled(svm) &&
-	    (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT))
-		vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat);
+	svm->nested.legacy_gpat_semantics =
+		nested_npt_enabled(svm) &&
+		!(kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT);
+	if (nested_npt_enabled(svm)) {
+		u64 g_pat = svm->nested.legacy_gpat_semantics ?
+			    vcpu->arch.pat : kvm_state->hdr.svm.gpat;
+
+		vmcb_set_gpat(svm->nested.vmcb02.ptr, g_pat);
+	}
 
 	nested_vmcb02_prepare_control(svm, svm->vmcb->save.rip, svm->vmcb->save.cs.base);
 
arch/x86/kvm/svm/svm.c +6 -2
index 00dba10991a5..ac45702f566e 100644
@@ -2727,7 +2727,8 @@ static bool svm_pat_accesses_gpat(struct kvm_vcpu *vcpu, bool from_host)
 	 * with older kernels.
 	 */
 	WARN_ON_ONCE(from_host && vcpu->wants_to_run);
-	return !from_host && is_guest_mode(vcpu) && nested_npt_enabled(svm);
+	return !svm->nested.legacy_gpat_semantics && !from_host &&
+		is_guest_mode(vcpu) && nested_npt_enabled(svm);
 }
 
 static u64 svm_get_pat(struct kvm_vcpu *vcpu, bool from_host)
@@ -2749,7 +2750,8 @@ static void svm_set_pat(struct kvm_vcpu *vcpu, bool from_host, u64 data)
 		if (npt_enabled) {
 			vmcb_set_gpat(svm->vmcb01.ptr, data);
 			if (is_guest_mode(&svm->vcpu) &&
-			    !nested_npt_enabled(svm))
+			    (svm->nested.legacy_gpat_semantics ||
+			     !nested_npt_enabled(svm)))
 				vmcb_set_gpat(svm->vmcb, data);
 		}
 	}
@@ -4262,6 +4264,8 @@ static int svm_vcpu_pre_run(struct kvm_vcpu *vcpu)
 	if (to_kvm_sev_info(vcpu->kvm)->need_init)
 		return -EINVAL;
 
+	to_svm(vcpu)->nested.legacy_gpat_semantics = false;
+
 	return 1;
 }
 
arch/x86/kvm/svm/svm.h +9 -0
index 58b0b935d049..626efef878a5 100644
@@ -238,6 +238,15 @@ struct svm_nested_state {
 	 * on its side.
 	 */
 	bool force_msr_bitmap_recalc;
+
+	/*
+	 * Indicates that a legacy nested state (without a valid gPAT) was
+	 * recently restored. Until the next KVM_RUN, updates to hPAT are
+	 * also applied to gPAT, preserving legacy behavior (i.e. L2 shares
+	 * L1's PAT). Because checkpoint restore is non-atomic, this
+	 * complication is necessary for backward compatibility.
+	 */
+	bool legacy_gpat_semantics;
 };
 
 struct vcpu_sev_es_state {
Signature · 1 line
2.53.0.371.g1d285c8824-goog
#19 lore raw

Re: [PATCH v5 09/10] KVM: x86: nSVM: Handle restore of legacy nested state

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 2084

Quoted 13 lines · Jim Mattson
arch/x86/kvm/svm/nested.c +6 -3
@@ -2075,9 +2076,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
 
-	if (nested_npt_enabled(svm) &&
-	    (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT))
-		vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat);
+	svm->nested.legacy_gpat_semantics =
+		nested_npt_enabled(svm) &&
+		!(kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT);
+	if (nested_npt_enabled(svm)) {
+		u64 g_pat = svm->nested.legacy_gpat_semantics ?
+			    vcpu->arch.pat : kvm_state->hdr.svm.gpat;

This is all a bit gnarly, e.g. the indentation and wrapping, as well as the logic (not that it's wrong, just a bit hard to follow the chain of events).

Rather than set legacy_gpat_semantics directly, what if we clear it by default, and then set it %true in the exact path where KVM uses legacy semantics.

	svm->nested.legacy_gpat_semantics = false;
	if (nested_npt_enabled(svm)) {
		if (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT) {
			vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat);
		} else {
			svm->nested.legacy_gpat_semantics = true;
			vmcb_set_gpat(svm->vmcb, vcpu->arch.pat);
		}
	}

As a bonus, if the previous patch is deliberately "bad" and does:

	if (nested_npt_enabled(svm)) {
		if (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT)
			vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat);
	}

then the diff for this snippet shrinks to:

@@ -2025,9 +2026,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,

        svm_switch_vmcb(svm, &svm->nested.vmcb02);
+       svm->nested.legacy_gpat_semantics = false;
        if (nested_npt_enabled(svm)) {
-               if (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT)
+               if (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT) {
                        vmcb_set_gpat(svm->vmcb, kvm_state->hdr.svm.gpat);

+ } else {
+ svm->nested.legacy_gpat_semantics = true;
+ vmcb_set_gpat(svm->vmcb, vcpu->arch.pat);
+ }

        }
 
        nested_vmcb02_prepare_control(svm);

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 2086

Quoted 2 lines
arch/x86/kvm/svm/nested.c +2 -0
+
+		vmcb_set_gpat(svm->nested.vmcb02.ptr, g_pat);

I don't like the switch from svm->vmcb to svm->nested.vmcb02.ptr. For better or worse, the existing code uses svm->vmcb, so I think it makes sense to use that. If we want to explicitly use vmcb02, then we should capture svm->nested.vmcb02.ptr locally as vmcb02 (in a future cleanup patch).

Quoted 9 lines
arch/x86/kvm/svm/nested.c +1 -0
+	}
 
 	nested_vmcb02_prepare_control(svm, svm->vmcb->save.rip, svm->vmcb->save.cs.base);
 
arch/x86/kvm/svm/svm.c +0 -0
index 00dba10991a5..ac45702f566e 100644

arch/x86/kvm/svm/svm.c static bool svm_pat_accesses_gpat(struct kvm_vcpu *vcpu, bool from_host) · line 2731

Quoted 6 lines
arch/x86/kvm/svm/svm.c +2 -1
@@ -2727,7 +2727,8 @@ static bool svm_pat_accesses_gpat(struct kvm_vcpu *vcpu, bool from_host)
 	 * with older kernels.
 	 */
 	WARN_ON_ONCE(from_host && vcpu->wants_to_run);
-	return !from_host && is_guest_mode(vcpu) && nested_npt_enabled(svm);
+	return !svm->nested.legacy_gpat_semantics && !from_host &&
+		is_guest_mode(vcpu) && nested_npt_enabled(svm);

Align indentation (it's a shame return wasn't spelled retturn, it would save many spaces).

#20 lore raw

PATCH v5 10/10 KVM: selftests: nSVM: Add svm_nested_pat test

In reply to Jim Mattson

Verify that KVM correctly virtualizes the host PAT MSR and the guest PAT register for nested SVM guests.

With nested NPT disabled:
* L1 and L2 share the same PAT
* The vmcb12.g_pat is ignored

With nested NPT enabled:
* An invalid g_pat in vmcb12 causes VMEXIT_INVALID
* RDMSR(IA32_PAT) from L2 returns the value of the guest PAT register
* WRMSR(IA32_PAT) from L2 is reflected in vmcb12's g_pat on VMEXIT
* RDMSR(IA32_PAT) from L1 returns the value of the host PAT MSR
* Save/restore with the vCPU in guest mode preserves both hPAT and gPAT

---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/svm_nested_pat_test.c   | 298 ++++++++++++++++++
 2 files changed, 299 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_pat_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 36b48e766e49..08a7bec34e4b 100644
@@ -110,6 +110,7 @@ TEST_GEN_PROGS_x86 += x86/state_test
 TEST_GEN_PROGS_x86 += x86/vmx_preemption_timer_test
 TEST_GEN_PROGS_x86 += x86/svm_vmcall_test
 TEST_GEN_PROGS_x86 += x86/svm_int_ctl_test
+TEST_GEN_PROGS_x86 += x86/svm_nested_pat_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test
 TEST_GEN_PROGS_x86 += x86/svm_lbr_nested_state
tools/testing/selftests/kvm/x86/svm_nested_pat_test.c new file +298 -0

298 lines not on this page · raw message

Signature · 1 line
2.53.0.371.g1d285c8824-goog
maintainer Applied
#21 lore raw
In reply to Jim Mattson
Quoted 8 lines · Jim Mattson

Currently, KVM's implementation of nested SVM treats the PAT MSR the same way whether or not nested NPT is enabled: L1 and L2 share a single PAT. However, the APM specifies that when nested NPT is enabled, the host (L1) and the guest (L2) should have independent PATs: hPAT for L1 and gPAT for L2. This patch series implements the architectural specification in KVM.

[...]

Applied patch 1 to kvm-x86 nested. I'll wait for a new version for the gPAT stuff.

[01/10] KVM: x86: SVM: Remove vmcb_is_dirty()

        https://github.com/kvm-x86/linux/commit/66b207f175f1

--
https://github.com/kvm-x86/linux/tree/next