Skip to contentSkip to conversation

KVM: x86: nSVM: Improve PAT virtualization

PATCH Superseded Jim Mattson 4 participants 34 messages Feb 12 – Mar 26

Subsystems (4)

v4 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v4 cover KVM: x86: nSVM: Improve PAT virtualization

Currently, KVM's implementation of nested SVM treats the PAT MSR the same way whether or not nested NPT is enabled: L1 and L2 share a single PAT. However, the APM specifies that when nested NPT is enabled, the host (L1) and the guest (L2) should have independent PATs: hPAT for L1 and gPAT for L2. This patch series implements the architectural specification in KVM.

Use the existing PAT MSR (vcpu->arch.pat) for hPAT. Add a new field, svm->nested.gpat, for gPAT. With nested NPT enabled, redirect guest accesses to the IA32_PAT MSR to gPAT. All other accesses, including userspace accesses via KVM_{GET,SET}_MSRS, continue to reference hPAT. The special handling of userspace accesses ensures save/restore forward compatibility (i.e. resuming a new checkpoint on an older kernel). When an old kernel restores a checkpoint from a new kernel, the gPAT will be lost, and L2 will simply use L1's PAT, which is the existing behavior of the old kernel anyway.

v1: https://lore.kernel.org/kvm/[email protected]/
v2: https://lore.kernel.org/kvm/[email protected]/
v3: https://lore.kernel.org/kvm/[email protected]/

v3 -> v4:
* Rebase on top of Yosry's v5 "Nested SVM fixes, cleanups, and hardening"
* Rename the svm_set_vmcb_gpat() helper to vmcb_set_gpat() for consistency with other VMCB helpers [Yosry].
* Cache g_pat within struct vmcb_save_area_cached (as

     svm->nested.save.g_pat) instead of using a standalone field in
     svm->nested [Sean].

* Update nested_vmcb_check_save() to optionally validate the cached g_pat, depending on a new boolean argument [Yosry].
* Reduce indentation in nested_vmcb02_prepare_save() when setting the guest PAT [Sean].

Jim Mattson (8):
KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest
mode
KVM: x86: nSVM: Cache and validate vmcb12 g_pat
KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT
KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT
KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT
KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE
KVM: x86: nSVM: Handle restore of legacy nested state
KVM: selftests: nSVM: Add svm_nested_pat test

 arch/x86/include/uapi/asm/kvm.h               |   5 +
 arch/x86/kvm/svm/nested.c                     |  60 +++-
 arch/x86/kvm/svm/svm.c                        |  40 ++-
 arch/x86/kvm/svm/svm.h                        |  38 ++-
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/svm_nested_pat_test.c   | 298 ++++++++++++++++++
 6 files changed, 413 insertions(+), 29 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_pat_test.c
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#1 lore raw

PATCH v4 1/8 KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode

When running an L2 guest and writing to MSR_IA32_CR_PAT, the host PAT value is stored in both vmcb01's g_pat field and vmcb02's g_pat field, but the clean bit was only being cleared for vmcb02.

Introduce the helper vmcb_set_gpat() which sets vmcb->save.g_pat and marks the VMCB dirty for VMCB_NPT. Use this helper in both svm_set_msr() for updating vmcb01 and in nested_vmcb02_compute_g_pat() for updating vmcb02, ensuring both VMCBs' NPT fields are properly marked dirty.

Fixes 4995a3685f1b ("KVM: SVM: Use a separate vmcb for the nested L2 guest")
---
 arch/x86/kvm/svm/nested.c | 2 +-
 arch/x86/kvm/svm/svm.c    | 3 +--
 arch/x86/kvm/svm/svm.h    | 9 +++++----
 3 files changed, 7 insertions(+), 7 deletions(-)
arch/x86/kvm/svm/nested.c +1 -1
index d80b1bde6630..b72a1f3c4144 100644
@@ -707,7 +707,7 @@ void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm)
 		return;
 
 	/* FIXME: merge g_pat from vmcb01 and vmcb12.  */
-	svm->nested.vmcb02.ptr->save.g_pat = svm->vmcb01.ptr->save.g_pat;
+	vmcb_set_gpat(svm->nested.vmcb02.ptr, svm->vmcb01.ptr->save.g_pat);
 }
 
 static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
arch/x86/kvm/svm/svm.c +1 -2
index 364915f42e13..529cbac57814 100644
@@ -2924,10 +2924,9 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 		if (ret)
 			break;
 
-		svm->vmcb01.ptr->save.g_pat = data;
+		vmcb_set_gpat(svm->vmcb01.ptr, data);
 		if (is_guest_mode(vcpu))
 			nested_vmcb02_compute_g_pat(svm);
-		vmcb_mark_dirty(svm->vmcb, VMCB_NPT);
 		break;
 	case MSR_IA32_SPEC_CTRL:
 		if (!msr->host_initiated &&
arch/x86/kvm/svm/svm.h +5 -4
index 0bb93879abfe..9850ed01e16e 100644
@@ -434,14 +434,15 @@ static inline void vmcb_mark_dirty(struct vmcb *vmcb, int bit)
 	vmcb->control.clean &= ~(1 << bit);
 }
 
-static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)
+static inline bool vmcb12_is_dirty(struct vmcb_ctrl_area_cached *control, int bit)
 {
-        return !test_bit(bit, (unsigned long *)&vmcb->control.clean);
+	return !test_bit(bit, (unsigned long *)&control->clean);
 }
 
-static inline bool vmcb12_is_dirty(struct vmcb_ctrl_area_cached *control, int bit)
+static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)
 {
-	return !test_bit(bit, (unsigned long *)&control->clean);
+	vmcb->save.g_pat = data;
+	vmcb_mark_dirty(vmcb, VMCB_NPT);
 }
 
 static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#2 lore raw

Re: [PATCH v4 1/8] KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode

Quoted 51 lines · Jim Mattson

When running an L2 guest and writing to MSR_IA32_CR_PAT, the host PAT value is stored in both vmcb01's g_pat field and vmcb02's g_pat field, but the clean bit was only being cleared for vmcb02.

Introduce the helper vmcb_set_gpat() which sets vmcb->save.g_pat and marks the VMCB dirty for VMCB_NPT. Use this helper in both svm_set_msr() for updating vmcb01 and in nested_vmcb02_compute_g_pat() for updating vmcb02, ensuring both VMCBs' NPT fields are properly marked dirty.

Fixes 4995a3685f1b ("KVM: SVM: Use a separate vmcb for the nested L2 guest")
---
 arch/x86/kvm/svm/nested.c | 2 +-
 arch/x86/kvm/svm/svm.c    | 3 +--
 arch/x86/kvm/svm/svm.h    | 9 +++++----
 3 files changed, 7 insertions(+), 7 deletions(-)
arch/x86/kvm/svm/nested.c +1 -1
index d80b1bde6630..b72a1f3c4144 100644
@@ -707,7 +707,7 @@ void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm)
 		return;
 
 	/* FIXME: merge g_pat from vmcb01 and vmcb12.  */
-	svm->nested.vmcb02.ptr->save.g_pat = svm->vmcb01.ptr->save.g_pat;
+	vmcb_set_gpat(svm->nested.vmcb02.ptr, svm->vmcb01.ptr->save.g_pat);
 }
 
 static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
arch/x86/kvm/svm/svm.c +1 -2
index 364915f42e13..529cbac57814 100644
@@ -2924,10 +2924,9 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 		if (ret)
 			break;
 
-		svm->vmcb01.ptr->save.g_pat = data;
+		vmcb_set_gpat(svm->vmcb01.ptr, data);
 		if (is_guest_mode(vcpu))
 			nested_vmcb02_compute_g_pat(svm);
-		vmcb_mark_dirty(svm->vmcb, VMCB_NPT);
 		break;
 	case MSR_IA32_SPEC_CTRL:
 		if (!msr->host_initiated &&
arch/x86/kvm/svm/svm.h +0 -0
index 0bb93879abfe..9850ed01e16e 100644

arch/x86/kvm/svm/svm.h static inline void vmcb_mark_dirty(struct vmcb *vmcb, int bit) · line 437

Quoted 4 lines
arch/x86/kvm/svm/svm.h +0 -1
@@ -434,14 +434,15 @@ static inline void vmcb_mark_dirty(struct vmcb *vmcb, int bit)
 	vmcb->control.clean &= ~(1 << bit);
 }
 
-static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)

Huh, I assume the removed of vmcb_is_dirty() was not intentional?

Quoted 18 lines
arch/x86/kvm/svm/svm.h +5 -3
+static inline bool vmcb12_is_dirty(struct vmcb_ctrl_area_cached *control, int bit)
 {
-        return !test_bit(bit, (unsigned long *)&vmcb->control.clean);
+	return !test_bit(bit, (unsigned long *)&control->clean);
 }
 
-static inline bool vmcb12_is_dirty(struct vmcb_ctrl_area_cached *control, int bit)
+static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)
 {
-	return !test_bit(bit, (unsigned long *)&control->clean);
+	vmcb->save.g_pat = data;
+	vmcb_mark_dirty(vmcb, VMCB_NPT);
 }
 
 static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#3 lore raw

Re: [PATCH v4 1/8] KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode

Quoted 9 lines · Yosry Ahmed
Quoted 8 lines · Jim Mattson
arch/x86/kvm/svm/svm.h +0 -0
index 0bb93879abfe..9850ed01e16e 100644
@@ -434,14 +434,15 @@ static inline void vmcb_mark_dirty(struct vmcb *vmcb, int bit)
 	vmcb->control.clean &= ~(1 << bit);
 }
 
Quoted 3 lines

arch/x86/kvm/svm/svm.h static inline void vmcb_mark_dirty(struct vmcb *vmcb, int bit) · line 437

Quoted 1 line
arch/x86/kvm/svm/svm.h +0 -1
-static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)

Huh, I assume the removed of vmcb_is_dirty() was not intentional?

Regardless of whether or not it was intentional, IMO it's a good change. KVM should never check vmcb12 directly, and I can't think of a legitimate case where KVM should condition its behavior on vmcb0{1,2} being clean/dirty.

Unless a v5 is needed, I'll split it to a separate patch when applying.

#4 lore raw

Re: [PATCH v4 1/8] KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode

Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Yosry Ahmed
Quoted 8 lines · Jim Mattson
arch/x86/kvm/svm/svm.h +0 -0
index 0bb93879abfe..9850ed01e16e 100644
@@ -434,14 +434,15 @@ static inline void vmcb_mark_dirty(struct vmcb *vmcb, int bit)
 	vmcb->control.clean &= ~(1 << bit);
 }
 
Quoted 7 lines
Quoted 3 lines

arch/x86/kvm/svm/svm.h static inline void vmcb_mark_dirty(struct vmcb *vmcb, int bit) · line 437

Quoted 1 line
arch/x86/kvm/svm/svm.h +0 -1
-static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)

Huh, I assume the removed of vmcb_is_dirty() was not intentional?

Regardless of whether or not it was intentional, IMO it's a good change. KVM should never check vmcb12 directly, and I can't think of a legitimate case where KVM should condition its behavior on vmcb0{1,2} being clean/dirty.

Funny enough, I removed all usages of vmcb_is_dirty() in my series, I just didn't drop it:

https://lore.kernel.org/kvm/[email protected]/

So Jim was cleaning up after me :)

Quoted 2 lines

Unless a v5 is needed, I'll split it to a separate patch when applying.

#5 lore raw

Re: [PATCH v4 1/8] KVM: x86: nSVM: Clear VMCB_NPT clean bit when updating hPAT from guest mode

Quoted 6 lines · Yosry Ahmed

Funny enough, I removed all usages of vmcb_is_dirty() in my series, I just didn't drop it:

https://lore.kernel.org/kvm/[email protected]/

So Jim was cleaning up after me :)

Sorry; I removed it when handling the merge conflicts, thought about moving it to a separate patch, and then forgot about it.

#6 lore raw

PATCH v4 2/8 KVM: x86: nSVM: Cache and validate vmcb12 g_pat

In reply to Jim Mattson

Cache g_pat from vmcb12 in vmcb_save_area_cached to avoid TOCTTOU issues, and add a validity check so that when nested paging is enabled for vmcb12, an invalid g_pat at emulated VMRUN causes an immediate VMEXIT with exit code VMEXIT_INVALID, as specified in the APM, volume 2: "Nested Paging and VMRUN/VMEXIT."

Fixes 3d6368ef580a ("KVM: SVM: Add VMRUN handler")
---
 arch/x86/kvm/svm/nested.c | 17 +++++++++++++----
 arch/x86/kvm/svm/svm.h    |  1 +
 2 files changed, 14 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/nested.c +13 -4
index b72a1f3c4144..91b35adb83f8 100644
@@ -426,7 +426,8 @@ static bool nested_vmcb_check_controls(struct kvm_vcpu *vcpu,
 
 /* Common checks that apply to both L1 and L2 state.  */
 static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
-				   struct vmcb_save_area_cached *save)
+				   struct vmcb_save_area_cached *save,
+				   bool check_gpat)
 {
 	if (CC(!(save->efer & EFER_SVME)))
 		return false;
@@ -462,6 +463,9 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
 	if (CC(!kvm_valid_efer(vcpu, save->efer)))
 		return false;
 
+	if (check_gpat && CC(!kvm_pat_valid(save->g_pat)))
+		return false;
+
 	return true;
 }
 
@@ -573,6 +577,7 @@ static void __nested_copy_vmcb_save_to_cache(struct vmcb_save_area_cached *to,
 
 	to->rax = from->rax;
 	to->cr2 = from->cr2;
+	to->g_pat = from->g_pat;
 
 	svm_copy_lbrs(to, from);
 }
@@ -1036,7 +1041,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
 
 	enter_guest_mode(vcpu);
 
-	if (!nested_vmcb_check_save(vcpu, &svm->nested.save) ||
+	if (!nested_vmcb_check_save(vcpu, &svm->nested.save,
+				    nested_npt_enabled(svm)) ||
 	    !nested_vmcb_check_controls(vcpu, &svm->nested.ctl,
 					svm->vmcb01.ptr->save.cr0))
 		return -EINVAL;
@@ -2006,13 +2012,16 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
 	/*
 	 * Validate host state saved from before VMRUN (see
-	 * nested_svm_check_permissions).
+	 * nested_svm_check_permissions). Note that the g_pat field is not
+	 * validated, because (a) it may have been clobbered by SMM before
+	 * KVM_GET_NESTED_STATE, and (b) it is not loaded at emulated
+	 * #VMEXIT.
 	 */
 	__nested_copy_vmcb_save_to_cache(&save_cached, save);
 	if (!(save->cr0 & X86_CR0_PG) ||
 	    !(save->cr0 & X86_CR0_PE) ||
 	    (save->rflags & X86_EFLAGS_VM) ||
-	    !nested_vmcb_check_save(vcpu, &save_cached))
+	    !nested_vmcb_check_save(vcpu, &save_cached, false))
 		goto out_free;
 
 
arch/x86/kvm/svm/svm.h +1 -0
index 9850ed01e16e..a49c48459e0b 100644
@@ -161,6 +161,7 @@ struct vmcb_save_area_cached {
 	u64 isst_addr;
 	u64 rax;
 	u64 cr2;
+	u64 g_pat;
 	u64 dbgctl;
 	u64 br_from;
 	u64 br_to;
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
Reviewed-by on v4 2/8
#7 lore raw

Re: [PATCH v4 2/8] KVM: x86: nSVM: Cache and validate vmcb12 g_pat

Quoted 56 lines · Jim Mattson

Cache g_pat from vmcb12 in vmcb_save_area_cached to avoid TOCTTOU issues, and add a validity check so that when nested paging is enabled for vmcb12, an invalid g_pat at emulated VMRUN causes an immediate VMEXIT with exit code VMEXIT_INVALID, as specified in the APM, volume 2: "Nested Paging and VMRUN/VMEXIT."

Fixes 3d6368ef580a ("KVM: SVM: Add VMRUN handler")
---
 arch/x86/kvm/svm/nested.c | 17 +++++++++++++----
 arch/x86/kvm/svm/svm.h    |  1 +
 2 files changed, 14 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/nested.c +8 -2
index b72a1f3c4144..91b35adb83f8 100644
@@ -426,7 +426,8 @@ static bool nested_vmcb_check_controls(struct kvm_vcpu *vcpu,
 
 /* Common checks that apply to both L1 and L2 state.  */
 static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
-				   struct vmcb_save_area_cached *save)
+				   struct vmcb_save_area_cached *save,
+				   bool check_gpat)
 {
 	if (CC(!(save->efer & EFER_SVME)))
 		return false;
@@ -462,6 +463,9 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
 	if (CC(!kvm_valid_efer(vcpu, save->efer)))
 		return false;
 
+	if (check_gpat && CC(!kvm_pat_valid(save->g_pat)))
+		return false;
+
 	return true;
 }
 
@@ -573,6 +577,7 @@ static void __nested_copy_vmcb_save_to_cache(struct vmcb_save_area_cached *to,
 
 	to->rax = from->rax;
 	to->cr2 = from->cr2;
+	to->g_pat = from->g_pat;
 
 	svm_copy_lbrs(to, from);
 }
@@ -1036,7 +1041,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
 
 	enter_guest_mode(vcpu);
 
-	if (!nested_vmcb_check_save(vcpu, &svm->nested.save) ||
+	if (!nested_vmcb_check_save(vcpu, &svm->nested.save,
+				    nested_npt_enabled(svm)) ||
 	    !nested_vmcb_check_controls(vcpu, &svm->nested.ctl,
 					svm->vmcb01.ptr->save.cr0))
 		return -EINVAL;

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 2018

Quoted 8 lines
arch/x86/kvm/svm/nested.c +4 -1
@@ -2006,13 +2012,16 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
 	/*
 	 * Validate host state saved from before VMRUN (see
-	 * nested_svm_check_permissions).
+	 * nested_svm_check_permissions). Note that the g_pat field is not
+	 * validated, because (a) it may have been clobbered by SMM before
+	 * KVM_GET_NESTED_STATE, and (b) it is not loaded at emulated
+	 * #VMEXIT.

(b) here means that svm_copy_vmrun_state() does not copy it to vmcb01, and the value is restored by KVM_SET_MSRS, right?

If my understanding is correct:

Quoted 25 lines
arch/x86/kvm/svm/nested.c +1 -1
 	 */
 	__nested_copy_vmcb_save_to_cache(&save_cached, save);
 	if (!(save->cr0 & X86_CR0_PG) ||
 	    !(save->cr0 & X86_CR0_PE) ||
 	    (save->rflags & X86_EFLAGS_VM) ||
-	    !nested_vmcb_check_save(vcpu, &save_cached))
+	    !nested_vmcb_check_save(vcpu, &save_cached, false))
 		goto out_free;
 
 
arch/x86/kvm/svm/svm.h +1 -0
index 9850ed01e16e..a49c48459e0b 100644
@@ -161,6 +161,7 @@ struct vmcb_save_area_cached {
 	u64 isst_addr;
 	u64 rax;
 	u64 cr2;
+	u64 g_pat;
 	u64 dbgctl;
 	u64 br_from;
 	u64 br_to;
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#8 lore raw

Re: [PATCH v4 2/8] KVM: x86: nSVM: Cache and validate vmcb12 g_pat

Quoted 12 lines · Yosry Ahmed

arch/x86/kvm/svm/nested.c static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 2018

Quoted 9 lines
arch/x86/kvm/svm/nested.c +4 -1
@@ -2006,13 +2012,16 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
      /*
       * Validate host state saved from before VMRUN (see
-      * nested_svm_check_permissions).
+      * nested_svm_check_permissions). Note that the g_pat field is not
+      * validated, because (a) it may have been clobbered by SMM before
+      * KVM_GET_NESTED_STATE, and (b) it is not loaded at emulated
+      * #VMEXIT.

(b) here means that svm_copy_vmrun_state() does not copy it to vmcb01, and the value is restored by KVM_SET_MSRS, right?

Actually, (b) refers to the open-coded block of assignments in nested_svm_vmexit() under the comment:

        /*
         * Restore processor state that had been saved in vmcb01
         */
Quoted 3 lines

If my understanding is correct:

#9 lore raw

Re: [PATCH v4 2/8] KVM: x86: nSVM: Cache and validate vmcb12 g_pat

Quoted 20 lines · Jim Mattson
Quoted 4 lines · Yosry Ahmed
(quoted diff) +0 -0
@@ -2006,13 +2012,16 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
 /*

Review static int svm_set_nested_state(struct kvm_vcpu *vcpu, · line 2023

Quoted 10 lines
(quoted diff) +0 -0
 * Validate host state saved from before VMRUN (see
 - * nested_svm_check_permissions).
 + * nested_svm_check_permissions). Note that the g_pat field is not
 + * validated, because (a) it may have been clobbered by SMM before
 + * KVM_GET_NESTED_STATE, and (b) it is not loaded at emulated
 + * #VMEXIT.
 
 (b) here means that svm_copy_vmrun_state() does not copy it to vmcb01,
 and the value is restored by KVM_SET_MSRS, right?
 

Actually, (b) refers to the open-coded block of assignments in nested_svm_vmexit() under the comment:

Quoted 4 lines

/*
* Restore processor state that had been saved in vmcb01 */

Yeah IIUC it's the same thing, we migrate them and copy them here to vmcb01 so that we can restore them in nested_svm_vmexit().

#10 lore raw

PATCH v4 3/8 KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT

In reply to Jim Mattson

When nested NPT is enabled in vmcb12, copy the (cached and validated) vmcb12 g_pat field to the guest PAT register. Under KVM, the guest PAT register lives in svm->nested.save.g_pat.

When NPT is enabled, but nested NPT is disabled, copy L1's IA32_PAT MSR to the vmcb02 g_pat field, since L2 shares the IA32_PAT MSR with L1.

When NPT is disabled, the g_pat field is ignored by hardware.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/nested.c +7 -3
index 91b35adb83f8..dc8275837120 100644
@@ -724,9 +724,6 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 	struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 
-	nested_vmcb02_compute_g_pat(svm);
-	vmcb_mark_dirty(vmcb02, VMCB_NPT);
-
 	/* Load the nested guest state */
 	if (svm->nested.vmcb12_gpa != svm->nested.last_vmcb12_gpa) {
 		new_vmcb12 = true;
@@ -757,6 +754,13 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 		vmcb_mark_dirty(vmcb02, VMCB_CET);
 	}
 
+	if (nested_npt_enabled(svm)) {
+		if (unlikely(new_vmcb12 || vmcb12_is_dirty(control, VMCB_NPT)))
+			vmcb_set_gpat(vmcb02, svm->nested.save.g_pat);
+	} else if (npt_enabled) {
+		vmcb_set_gpat(vmcb02, vcpu->arch.pat);
+	}
+
 	kvm_set_rflags(vcpu, save->rflags | X86_EFLAGS_FIXED);
 
 	svm_set_efer(vcpu, svm->nested.save.efer);
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
Reviewed-by on v4 3/8
#11 lore raw

Re: [PATCH v4 3/8] KVM: x86: nSVM: Set vmcb02.g_pat correctly for nested NPT

Quoted 11 lines · Jim Mattson

When nested NPT is enabled in vmcb12, copy the (cached and validated) vmcb12 g_pat field to the guest PAT register. Under KVM, the guest PAT register lives in svm->nested.save.g_pat.

When NPT is enabled, but nested NPT is disabled, copy L1's IA32_PAT MSR to the vmcb02 g_pat field, since L2 shares the IA32_PAT MSR with L1.

When NPT is disabled, the g_pat field is ignored by hardware.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
Quoted 35 lines
---
 arch/x86/kvm/svm/nested.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/nested.c +7 -3
index 91b35adb83f8..dc8275837120 100644
@@ -724,9 +724,6 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 	struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 
-	nested_vmcb02_compute_g_pat(svm);
-	vmcb_mark_dirty(vmcb02, VMCB_NPT);
-
 	/* Load the nested guest state */
 	if (svm->nested.vmcb12_gpa != svm->nested.last_vmcb12_gpa) {
 		new_vmcb12 = true;
@@ -757,6 +754,13 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 		vmcb_mark_dirty(vmcb02, VMCB_CET);
 	}
 
+	if (nested_npt_enabled(svm)) {
+		if (unlikely(new_vmcb12 || vmcb12_is_dirty(control, VMCB_NPT)))
+			vmcb_set_gpat(vmcb02, svm->nested.save.g_pat);
+	} else if (npt_enabled) {
+		vmcb_set_gpat(vmcb02, vcpu->arch.pat);
+	}
+
 	kvm_set_rflags(vcpu, save->rflags | X86_EFLAGS_FIXED);
 
 	svm_set_efer(vcpu, svm->nested.save.efer);
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#12 lore raw

PATCH v4 4/8 KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

In reply to Jim Mattson

When the vCPU is in guest mode with nested NPT enabled, guest accesses to IA32_PAT are redirected to the gPAT register, which is stored in svm->nested.save.g_pat.

Non-guest accesses (e.g. from userspace) to IA32_PAT are always redirected to hPAT, which is stored in vcpu->arch.pat.

This is architected behavior. It also makes it possible to restore a new checkpoint on an old kernel with reasonable semantics. After the restore, gPAT will be lost, and L2 will run on L1's PAT. Note that the old kernel would have always run L2 on L1's PAT.

Add WARN_ON_ONCE to both svm_get_msr() and svm_set_msr() to flag any host-initiated accesses originating from KVM itself rather than userspace.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c |  9 ---------
 arch/x86/kvm/svm/svm.c    | 37 ++++++++++++++++++++++++++++++-------
 arch/x86/kvm/svm/svm.h    | 17 ++++++++++++++++-
 3 files changed, 46 insertions(+), 17 deletions(-)
arch/x86/kvm/svm/nested.c +0 -9
index dc8275837120..69b577a4915c 100644
@@ -706,15 +706,6 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 	return 0;
 }
 
-void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm)
-{
-	if (!svm->nested.vmcb02.ptr)
-		return;
-
-	/* FIXME: merge g_pat from vmcb01 and vmcb12.  */
-	vmcb_set_gpat(svm->nested.vmcb02.ptr, svm->vmcb01.ptr->save.g_pat);
-}
-
 static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 {
 	struct vmcb_ctrl_area_cached *control = &svm->nested.ctl;
arch/x86/kvm/svm/svm.c +30 -7
index 529cbac57814..205bf07896ad 100644
@@ -2837,6 +2837,21 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_AMD64_DE_CFG:
 		msr_info->data = svm->msr_decfg;
 		break;
+	case MSR_IA32_CR_PAT:
+		/*
+		 * When nested NPT is enabled, L2 has a separate PAT from
+		 * L1.  Guest accesses to IA32_PAT while running L2 target
+		 * L2's gPAT; host-initiated accesses always target L1's
+		 * hPAT for backward and forward KVM_GET_MSRS compatibility
+		 * with older kernels.
+		 */
+		WARN_ON_ONCE(msr_info->host_initiated && vcpu->wants_to_run);
+		if (!msr_info->host_initiated && is_guest_mode(vcpu) &&
+		    nested_npt_enabled(svm))
+			msr_info->data = svm->nested.save.g_pat;
+		else
+			msr_info->data = vcpu->arch.pat;
+		break;
 	default:
 		return kvm_get_msr_common(vcpu, msr_info);
 	}
@@ -2920,13 +2935,21 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 
 		break;
 	case MSR_IA32_CR_PAT:
-		ret = kvm_set_msr_common(vcpu, msr);
-		if (ret)
-			break;
-
-		vmcb_set_gpat(svm->vmcb01.ptr, data);
-		if (is_guest_mode(vcpu))
-			nested_vmcb02_compute_g_pat(svm);
+		if (!kvm_pat_valid(data))
+			return 1;
+		/*
+		 * When nested NPT is enabled, L2 has a separate PAT from
+		 * L1.  Guest accesses to IA32_PAT while running L2 target
+		 * L2's gPAT; host-initiated accesses always target L1's
+		 * hPAT for backward and forward KVM_SET_MSRS compatibility
+		 * with older kernels.
+		 */
+		WARN_ON_ONCE(msr->host_initiated && vcpu->wants_to_run);
+		if (!msr->host_initiated && is_guest_mode(vcpu) &&
+		    nested_npt_enabled(svm))
+			svm_set_gpat(svm, data);
+		else
+			svm_set_hpat(svm, data);
 		break;
 	case MSR_IA32_SPEC_CTRL:
 		if (!msr->host_initiated &&
arch/x86/kvm/svm/svm.h +16 -1
index a49c48459e0b..88549705133f 100644
@@ -607,6 +607,22 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)
 	return svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_NP;
 }
 
+static inline void svm_set_gpat(struct vcpu_svm *svm, u64 data)
+{
+	svm->nested.save.g_pat = data;
+	vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+}
+
+static inline void svm_set_hpat(struct vcpu_svm *svm, u64 data)
+{
+	svm->vcpu.arch.pat = data;
+	if (npt_enabled) {
+		vmcb_set_gpat(svm->vmcb01.ptr, data);
+		if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
+			vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+	}
+}
+
 static inline bool nested_vnmi_enabled(struct vcpu_svm *svm)
 {
 	return guest_cpu_cap_has(&svm->vcpu, X86_FEATURE_VNMI) &&
@@ -840,7 +856,6 @@ void nested_copy_vmcb_control_to_cache(struct vcpu_svm *svm,
 void nested_copy_vmcb_save_to_cache(struct vcpu_svm *svm,
 				    struct vmcb_save_area *save);
 void nested_sync_control_from_vmcb02(struct vcpu_svm *svm);
-void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm);
 void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#13 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 111 lines · Jim Mattson

When the vCPU is in guest mode with nested NPT enabled, guest accesses to IA32_PAT are redirected to the gPAT register, which is stored in svm->nested.save.g_pat.

Non-guest accesses (e.g. from userspace) to IA32_PAT are always redirected to hPAT, which is stored in vcpu->arch.pat.

This is architected behavior. It also makes it possible to restore a new checkpoint on an old kernel with reasonable semantics. After the restore, gPAT will be lost, and L2 will run on L1's PAT. Note that the old kernel would have always run L2 on L1's PAT.

Add WARN_ON_ONCE to both svm_get_msr() and svm_set_msr() to flag any host-initiated accesses originating from KVM itself rather than userspace.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c |  9 ---------
 arch/x86/kvm/svm/svm.c    | 37 ++++++++++++++++++++++++++++++-------
 arch/x86/kvm/svm/svm.h    | 17 ++++++++++++++++-
 3 files changed, 46 insertions(+), 17 deletions(-)
arch/x86/kvm/svm/nested.c +0 -9
index dc8275837120..69b577a4915c 100644
@@ -706,15 +706,6 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 	return 0;
 }
 
-void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm)
-{
-	if (!svm->nested.vmcb02.ptr)
-		return;
-
-	/* FIXME: merge g_pat from vmcb01 and vmcb12.  */
-	vmcb_set_gpat(svm->nested.vmcb02.ptr, svm->vmcb01.ptr->save.g_pat);
-}
-
 static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
 {
 	struct vmcb_ctrl_area_cached *control = &svm->nested.ctl;
arch/x86/kvm/svm/svm.c +30 -7
index 529cbac57814..205bf07896ad 100644
@@ -2837,6 +2837,21 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_AMD64_DE_CFG:
 		msr_info->data = svm->msr_decfg;
 		break;
+	case MSR_IA32_CR_PAT:
+		/*
+		 * When nested NPT is enabled, L2 has a separate PAT from
+		 * L1.  Guest accesses to IA32_PAT while running L2 target
+		 * L2's gPAT; host-initiated accesses always target L1's
+		 * hPAT for backward and forward KVM_GET_MSRS compatibility
+		 * with older kernels.
+		 */
+		WARN_ON_ONCE(msr_info->host_initiated && vcpu->wants_to_run);
+		if (!msr_info->host_initiated && is_guest_mode(vcpu) &&
+		    nested_npt_enabled(svm))
+			msr_info->data = svm->nested.save.g_pat;
+		else
+			msr_info->data = vcpu->arch.pat;
+		break;
 	default:
 		return kvm_get_msr_common(vcpu, msr_info);
 	}
@@ -2920,13 +2935,21 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 
 		break;
 	case MSR_IA32_CR_PAT:
-		ret = kvm_set_msr_common(vcpu, msr);
-		if (ret)
-			break;
-
-		vmcb_set_gpat(svm->vmcb01.ptr, data);
-		if (is_guest_mode(vcpu))
-			nested_vmcb02_compute_g_pat(svm);
+		if (!kvm_pat_valid(data))
+			return 1;
+		/*
+		 * When nested NPT is enabled, L2 has a separate PAT from
+		 * L1.  Guest accesses to IA32_PAT while running L2 target
+		 * L2's gPAT; host-initiated accesses always target L1's
+		 * hPAT for backward and forward KVM_SET_MSRS compatibility
+		 * with older kernels.
+		 */
+		WARN_ON_ONCE(msr->host_initiated && vcpu->wants_to_run);
+		if (!msr->host_initiated && is_guest_mode(vcpu) &&
+		    nested_npt_enabled(svm))
+			svm_set_gpat(svm, data);
+		else
+			svm_set_hpat(svm, data);
 		break;
 	case MSR_IA32_SPEC_CTRL:
 		if (!msr->host_initiated &&
arch/x86/kvm/svm/svm.h +5 -0
index a49c48459e0b..88549705133f 100644
@@ -607,6 +607,22 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)
 	return svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_NP;
 }
 
+static inline void svm_set_gpat(struct vcpu_svm *svm, u64 data)
+{
+	svm->nested.save.g_pat = data;
+	vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+}

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 624

Quoted 10 lines
arch/x86/kvm/svm/svm.h +10 -0
+
+static inline void svm_set_hpat(struct vcpu_svm *svm, u64 data)
+{
+	svm->vcpu.arch.pat = data;
+	if (npt_enabled) {
+		vmcb_set_gpat(svm->vmcb01.ptr, data);
+		if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
+			vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+	}
+}

Is it me, or is it a bit confusing that svm_set_gpat() sets L2's gPAT not L1's, and svm_set_hpat() calls vmcb_set_gpat()?

"gpat" means different things in the context of the VMCB or otherwise, which kinda makes sense but is also not super clear. Maybe svm_set_l1_gpat() and svm_set_l2_gpat() is more clear?

Will leave it up to Sean to decide if the naming is good enough, but honestly I don't want to stall this series, so hopefully any renames can be done as a follow up or when the series is applied.

Quoted 15 lines
arch/x86/kvm/svm/svm.h +1 -1
+
 static inline bool nested_vnmi_enabled(struct vcpu_svm *svm)
 {
 	return guest_cpu_cap_has(&svm->vcpu, X86_FEATURE_VNMI) &&
@@ -840,7 +856,6 @@ void nested_copy_vmcb_control_to_cache(struct vcpu_svm *svm,
 void nested_copy_vmcb_save_to_cache(struct vcpu_svm *svm,
 				    struct vmcb_save_area *save);
 void nested_sync_control_from_vmcb02(struct vcpu_svm *svm);
-void nested_vmcb02_compute_g_pat(struct vcpu_svm *svm);
 void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#14 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Please trim your replies. Scrolling through 100+ lines of quoted text to find the ~12 lines of context that actually matter is annoying.

Quoted 9 lines · Yosry Ahmed
Quoted 8 lines · Jim Mattson
arch/x86/kvm/svm/svm.h +0 -0
index a49c48459e0b..88549705133f 100644
@@ -607,6 +607,22 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)
 	return svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_NP;
 }
 
Quoted 10 lines

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 619

Quoted 10 lines
arch/x86/kvm/svm/svm.h +10 -0
+static inline void svm_set_gpat(struct vcpu_svm *svm, u64 data)
+{
+	svm->nested.save.g_pat = data;
+	vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+}
+
+static inline void svm_set_hpat(struct vcpu_svm *svm, u64 data)
+{
+	svm->vcpu.arch.pat = data;
+	if (npt_enabled) {

Peeking at the future patches, if we make this:

	if (!npt_enabled)
		return;

then we can end up with this:

	if (npt_enabled)
		return;

	vmcb_set_gpat(svm->vmcb01.ptr, data);
	if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
		vmcb_set_gpat(svm->nested.vmcb02.ptr, data);

	if (svm->nested.legacy_gpat_semantics)
		svm_set_l2_pat(svm, data);

Because legacy_gpat_semantics can only be true if npt_enabled is true. Without that guard, KVM _looks_ buggy because it's setting gpat in the VMCB even when it shouldn't exist.

Actually, calling svm_set_l2_pat() when !is_guest_mode() is wrong too, no? E.g. shouldn't we end up with this?

static inline void svm_set_l1_pat(struct vcpu_svm *svm, u64 data)

  {
	svm->vcpu.arch.pat = data;

	if (npt_enabled)
		return;

	vmcb_set_gpat(svm->vmcb01.ptr, data);

	if (is_guest_mode(&svm->vcpu)) {
		if (svm->nested.legacy_gpat_semantics)
			svm_set_l2_pat(svm, data);
		else if (!nested_npt_enabled(svm))
			vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
	}

}

Quoted 8 lines

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 624

Quoted 5 lines
arch/x86/kvm/svm/svm.h +5 -0
+		vmcb_set_gpat(svm->vmcb01.ptr, data);
+		if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
+			vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+	}
+}

Is it me, or is it a bit confusing that svm_set_gpat() sets L2's gPAT not L1's, and svm_set_hpat() calls vmcb_set_gpat()?

It's not just you. I don't find it confusing per se, more that it's really subtle.

Quoted 3 lines

"gpat" means different things in the context of the VMCB or otherwise, which kinda makes sense but is also not super clear. Maybe svm_set_l1_gpat() and svm_set_l2_gpat() is more clear?

I think just svm_set_l1_pat() and svm_set_l2_pat(), because gpat straight up doesn't exist when NPT is disabled/unsupported.

#15 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 42 lines · Sean Christopherson

Please trim your replies. Scrolling through 100+ lines of quoted text to find the ~12 lines of context that actually matter is annoying.

Quoted 9 lines · Yosry Ahmed
Quoted 8 lines · Jim Mattson
arch/x86/kvm/svm/svm.h +0 -0
index a49c48459e0b..88549705133f 100644
@@ -607,6 +607,22 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)
    return svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_NP;
 }
 
Quoted 10 lines

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 619

Quoted 10 lines
arch/x86/kvm/svm/svm.h +10 -0
+static inline void svm_set_gpat(struct vcpu_svm *svm, u64 data)
+{
+   svm->nested.save.g_pat = data;
+   vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+}
+
+static inline void svm_set_hpat(struct vcpu_svm *svm, u64 data)
+{
+   svm->vcpu.arch.pat = data;
+   if (npt_enabled) {

Peeking at the future patches, if we make this:

        if (!npt_enabled)
                return;

then we can end up with this:

        if (npt_enabled)
                return;

        vmcb_set_gpat(svm->vmcb01.ptr, data);
        if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
                vmcb_set_gpat(svm->nested.vmcb02.ptr, data);

        if (svm->nested.legacy_gpat_semantics)
                svm_set_l2_pat(svm, data);
Quoted 6 lines

Because legacy_gpat_semantics can only be true if npt_enabled is true. Without that guard, KVM _looks_ buggy because it's setting gpat in the VMCB even when it shouldn't exist.

Actually, calling svm_set_l2_pat() when !is_guest_mode() is wrong too, no? E.g. shouldn't we end up with this?

Sigh. legacy_gpat_semantics is supposed to be set only when is_guest_mode() and nested_npt_enabled(). I forgot about back-to-back invocations of KVM_SET_NESTED_STATE. Are there other ways of leaving guest mode or disabling nested NPT before the next KVM_RUN?

Quoted 30 lines
  static inline void svm_set_l1_pat(struct vcpu_svm *svm, u64 data)
  {
        svm->vcpu.arch.pat = data;
        if (npt_enabled)
                return;

        vmcb_set_gpat(svm->vmcb01.ptr, data);

        if (is_guest_mode(&svm->vcpu)) {
                if (svm->nested.legacy_gpat_semantics)
                        svm_set_l2_pat(svm, data);
                else if (!nested_npt_enabled(svm))
                        vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
        }

}

Quoted 8 lines

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 624

Quoted 5 lines
arch/x86/kvm/svm/svm.h +5 -0
+           vmcb_set_gpat(svm->vmcb01.ptr, data);
+           if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
+                   vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+   }
+}

Is it me, or is it a bit confusing that svm_set_gpat() sets L2's gPAT not L1's, and svm_set_hpat() calls vmcb_set_gpat()?

It's not just you. I don't find it confusing per se, more that it's really subtle.

Quoted 6 lines
Quoted 3 lines

"gpat" means different things in the context of the VMCB or otherwise, which kinda makes sense but is also not super clear. Maybe svm_set_l1_gpat() and svm_set_l2_gpat() is more clear?

I think just svm_set_l1_pat() and svm_set_l2_pat(), because gpat straight up doesn't exist when NPT is disabled/unsupported.

My intention was that "gpat" and "hpat" were from the perspective of the vCPU.

I dislike svm_set_l1_pat() and svm_set_l2_pat(). As you point out above, there is no independent L2 PAT when nested NPT is disabled. I think that's less obvious than the fact that there is no gPAT from the vCPU's perspective. My preference is to follow the APM terminology when possible. Making up our own terms just leads to confusion.

#16 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 27 lines · Jim Mattson
Quoted 26 lines · Sean Christopherson
Quoted 4 lines

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 619

Quoted 4 lines
arch/x86/kvm/svm/svm.h +4 -0
+static inline void svm_set_hpat(struct vcpu_svm *svm, u64 data)
+{
+   svm->vcpu.arch.pat = data;
+   if (npt_enabled) {

Peeking at the future patches, if we make this:

        if (!npt_enabled)
                return;

then we can end up with this:

        if (npt_enabled)
                return;

        vmcb_set_gpat(svm->vmcb01.ptr, data);
        if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
                vmcb_set_gpat(svm->nested.vmcb02.ptr, data);

        if (svm->nested.legacy_gpat_semantics)
                svm_set_l2_pat(svm, data);

Because legacy_gpat_semantics can only be true if npt_enabled is true. Without that guard, KVM _looks_ buggy because it's setting gpat in the VMCB even when it shouldn't exist.

Quoted 7 lines
Quoted 2 lines

Actually, calling svm_set_l2_pat() when !is_guest_mode() is wrong too, no? E.g. shouldn't we end up with this?

Sigh. legacy_gpat_semantics is supposed to be set only when is_guest_mode() and nested_npt_enabled(). I forgot about back-to-back invocations of KVM_SET_NESTED_STATE. Are there other ways of leaving guest mode or disabling nested NPT before the next KVM_RUN?

KVM_SET_VCPU_EVENTS will do it if userspace forces a change in SMM state:

		if (!!(vcpu->arch.hflags & HF_SMM_MASK) != events->smi.smm) {
			kvm_leave_nested(vcpu);
			kvm_smm_changed(vcpu, events->smi.smm);
		}

I honestly wasn't even thinking of anything in particular, it just looked weird.

Quoted 19 lines
Quoted 12 lines
Quoted 8 lines

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 624

Quoted 5 lines
arch/x86/kvm/svm/svm.h +5 -0
+           vmcb_set_gpat(svm->vmcb01.ptr, data);
+           if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
+                   vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+   }
+}

Is it me, or is it a bit confusing that svm_set_gpat() sets L2's gPAT not L1's, and svm_set_hpat() calls vmcb_set_gpat()?

It's not just you. I don't find it confusing per se, more that it's really subtle.

Quoted 6 lines
Quoted 3 lines

"gpat" means different things in the context of the VMCB or otherwise, which kinda makes sense but is also not super clear. Maybe svm_set_l1_gpat() and svm_set_l2_gpat() is more clear?

I think just svm_set_l1_pat() and svm_set_l2_pat(), because gpat straight up doesn't exist when NPT is disabled/unsupported.

Quoted 7 lines

My intention was that "gpat" and "hpat" were from the perspective of the vCPU.

I dislike svm_set_l1_pat() and svm_set_l2_pat(). As you point out above, there is no independent L2 PAT when nested NPT is disabled. I think that's less obvious than the fact that there is no gPAT from the vCPU's perspective. My preference is to follow the APM terminology when possible. Making up our own terms just leads to confusion.

How about svm_set_pat() and svm_get_gpat()? Because hPAT doesn't exist when NPT is unsupported/disabled, but KVM still needs to set the vCPU's emulated PAT value.

#17 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 37 lines · Sean Christopherson
Quoted 27 lines · Jim Mattson
Quoted 26 lines · Sean Christopherson
Quoted 4 lines

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 619

Quoted 4 lines
arch/x86/kvm/svm/svm.h +4 -0
+static inline void svm_set_hpat(struct vcpu_svm *svm, u64 data)
+{
+   svm->vcpu.arch.pat = data;
+   if (npt_enabled) {

Peeking at the future patches, if we make this:

        if (!npt_enabled)
                return;

then we can end up with this:

        if (npt_enabled)
                return;

        vmcb_set_gpat(svm->vmcb01.ptr, data);
        if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
                vmcb_set_gpat(svm->nested.vmcb02.ptr, data);

        if (svm->nested.legacy_gpat_semantics)
                svm_set_l2_pat(svm, data);

Because legacy_gpat_semantics can only be true if npt_enabled is true. Without that guard, KVM _looks_ buggy because it's setting gpat in the VMCB even when it shouldn't exist.

Quoted 7 lines
Quoted 2 lines

Actually, calling svm_set_l2_pat() when !is_guest_mode() is wrong too, no? E.g. shouldn't we end up with this?

Sigh. legacy_gpat_semantics is supposed to be set only when is_guest_mode() and nested_npt_enabled(). I forgot about back-to-back invocations of KVM_SET_NESTED_STATE. Are there other ways of leaving guest mode or disabling nested NPT before the next KVM_RUN?

Quoted 8 lines

KVM_SET_VCPU_EVENTS will do it if userspace forces a change in SMM state:

                if (!!(vcpu->arch.hflags & HF_SMM_MASK) != events->smi.smm) {
                        kvm_leave_nested(vcpu);
                        kvm_smm_changed(vcpu, events->smi.smm);
                }

I honestly wasn't even thinking of anything in particular, it just looked weird.

At the very least, then, kvm_leave_nested() has to clear legacy_gpat_semantics. I will look for other paths.

Quoted 21 lines
Quoted 21 lines
Quoted 12 lines
Quoted 8 lines

arch/x86/kvm/svm/svm.h static inline bool nested_npt_enabled(struct vcpu_svm *svm) · line 624

Quoted 5 lines
arch/x86/kvm/svm/svm.h +5 -0
+           vmcb_set_gpat(svm->vmcb01.ptr, data);
+           if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
+                   vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+   }
+}

Is it me, or is it a bit confusing that svm_set_gpat() sets L2's gPAT not L1's, and svm_set_hpat() calls vmcb_set_gpat()?

It's not just you. I don't find it confusing per se, more that it's really subtle.

Quoted 6 lines
Quoted 3 lines

"gpat" means different things in the context of the VMCB or otherwise, which kinda makes sense but is also not super clear. Maybe svm_set_l1_gpat() and svm_set_l2_gpat() is more clear?

I think just svm_set_l1_pat() and svm_set_l2_pat(), because gpat straight up doesn't exist when NPT is disabled/unsupported.

My intention was that "gpat" and "hpat" were from the perspective of the vCPU.

Quoted 8 lines
Quoted 5 lines

I dislike svm_set_l1_pat() and svm_set_l2_pat(). As you point out above, there is no independent L2 PAT when nested NPT is disabled. I think that's less obvious than the fact that there is no gPAT from the vCPU's perspective. My preference is to follow the APM terminology when possible. Making up our own terms just leads to confusion.

How about svm_set_pat() and svm_get_gpat()? Because hPAT doesn't exist when NPT is unsupported/disabled, but KVM still needs to set the vCPU's emulated PAT value.

What if we don't break it up this way at all? Instead of distributing the logic between svm_[gs]set_msr() and a few helper functions, we could just have svm_[gs]et_msr() call svm_[gs]et_pat(), and all of the logic can go in these two functions.

#18 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 28 lines · Jim Mattson
Quoted 27 lines · Sean Christopherson
Quoted 26 lines
Quoted 12 lines
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+           vmcb_set_gpat(svm->vmcb01.ptr, data);
+           if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
+                   vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
+   }
+}

Is it me, or is it a bit confusing that svm_set_gpat() sets L2's gPAT not L1's, and svm_set_hpat() calls vmcb_set_gpat()?

It's not just you. I don't find it confusing per se, more that it's really subtle.

Quoted 6 lines
Quoted 3 lines

"gpat" means different things in the context of the VMCB or otherwise, which kinda makes sense but is also not super clear. Maybe svm_set_l1_gpat() and svm_set_l2_gpat() is more clear?

I think just svm_set_l1_pat() and svm_set_l2_pat(), because gpat straight up doesn't exist when NPT is disabled/unsupported.

My intention was that "gpat" and "hpat" were from the perspective of the vCPU.

I dislike svm_set_l1_pat() and svm_set_l2_pat(). As you point out above, there is no independent L2 PAT when nested NPT is disabled. I think that's less obvious than the fact that there is no gPAT from the vCPU's perspective. My preference is to follow the APM terminology when possible. Making up our own terms just leads to confusion.

Quoted 7 lines
Quoted 2 lines

How about svm_set_pat() and svm_get_gpat()? Because hPAT doesn't exist when NPT is unsupported/disabled, but KVM still needs to set the vCPU's emulated PAT value.

What if we don't break it up this way at all? Instead of distributing the logic between svm_[gs]set_msr() and a few helper functions, we could just have svm_[gs]et_msr() call svm_[gs]et_pat(), and all of the logic can go in these two functions.

I like it. And AFAICT it largely Just Works, because the calls from svm_set_nested_state() will always be routed to gpat since the calls are already guarded with is_guest_mode() + nested_npt_enabled().

Side topic, either as a prep patch (to duplicate code) or as a follow-up patch (to move the PAT handling in x86.c to vmx.c), the "common" handling of PAT should be fully forked between VMX and SVM. As of this patch, it's not just misleading, it's actively dangerous since calling kvm_get_msr_common() for SVM would get the wrong value.

FWIW, this is what I ended up with when hacking on top of your patches to see how this played out.

---
 arch/x86/kvm/svm/nested.c |  4 +--
 arch/x86/kvm/svm/svm.c    | 64 +++++++++++++++++++++++++--------------
 arch/x86/kvm/svm/svm.h    | 19 +-----------
 arch/x86/kvm/vmx/vmx.c    | 10 ++++--
 arch/x86/kvm/x86.c        |  9 ------
 5 files changed, 51 insertions(+), 55 deletions(-)
arch/x86/kvm/svm/nested.c +2 -2
index d854d29b0bd8..361f189d3967 100644
@@ -2075,9 +2075,9 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 
 	if (nested_npt_enabled(svm)) {
 		if (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT) {
-			svm_set_gpat(svm, kvm_state->hdr.svm.gpat);
+			svm_set_pat(vcpu, kvm_state->hdr.svm.gpat, true);
 		} else {
-			svm_set_gpat(svm, vcpu->arch.pat);
+			svm_set_pat(vcpu, vcpu->arch.pat, true);
 			svm->nested.legacy_gpat_semantics = true;
 		}
 	}
arch/x86/kvm/svm/svm.c +41 -23

82 lines not on this page · raw message

arch/x86/kvm/svm/svm.h +1 -18
index 0bb9fdcb489d..71502db3f679 100644
@@ -616,24 +616,6 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)
 	return svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_NP;
 }
 
-static inline void svm_set_gpat(struct vcpu_svm *svm, u64 data)
-{
-	svm->nested.save.g_pat = data;
-	vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
-}
-
-static inline void svm_set_hpat(struct vcpu_svm *svm, u64 data)
-{
-	svm->vcpu.arch.pat = data;
-	if (npt_enabled) {
-		vmcb_set_gpat(svm->vmcb01.ptr, data);
-		if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
-			vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
-	}
-	if (svm->nested.legacy_gpat_semantics)
-		svm_set_gpat(svm, data);
-}
-
 static inline bool nested_vnmi_enabled(struct vcpu_svm *svm)
 {
 	return guest_cpu_cap_has(&svm->vcpu, X86_FEATURE_VNMI) &&
@@ -780,6 +762,7 @@ void svm_enable_lbrv(struct kvm_vcpu *vcpu);
 void svm_update_lbrv(struct kvm_vcpu *vcpu);
 
 int svm_set_efer(struct kvm_vcpu *vcpu, u64 efer);
+void svm_set_pat(struct kvm_vcpu *vcpu, u64 pat, bool host_initiated);
 void svm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0);
 void svm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4);
 void disable_nmi_singlestep(struct vcpu_svm *svm);
arch/x86/kvm/vmx/vmx.c +7 -3
index 967b58a8ab9d..546056e690eb 100644
@@ -2141,6 +2141,9 @@ int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 #endif
 	case MSR_EFER:
 		return kvm_get_msr_common(vcpu, msr_info);
+	case MSR_IA32_CR_PAT:
+		msr_info->data = vcpu->arch.pat;
+		break;
 	case MSR_IA32_TSX_CTRL:
 		if (!msr_info->host_initiated &&
 		    !(vcpu->arch.arch_capabilities & ARCH_CAP_TSX_CTRL_MSR))
@@ -2468,9 +2471,10 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 			return 1;
 		goto find_uret_msr;
 	case MSR_IA32_CR_PAT:
-		ret = kvm_set_msr_common(vcpu, msr_info);
-		if (ret)
-			break;
+		if (!kvm_pat_valid(data))
+			return 1;
+
+		vcpu->arch.pat = data;
 
 		if (is_guest_mode(vcpu) &&
 		    get_vmcs12(vcpu)->vm_exit_controls & VM_EXIT_SAVE_IA32_PAT)
arch/x86/kvm/x86.c +0 -9
index 416899b5dbe4..41936f83a17f 100644
@@ -4025,12 +4025,6 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 			return 1;
 		}
 		break;
-	case MSR_IA32_CR_PAT:
-		if (!kvm_pat_valid(data))
-			return 1;
-
-		vcpu->arch.pat = data;
-		break;
 	case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000:
 	case MSR_MTRRdefType:
 		return kvm_mtrr_set_msr(vcpu, msr, data);
@@ -4436,9 +4430,6 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		msr_info->data = kvm_scale_tsc(rdtsc(), ratio) + offset;
 		break;
 	}
-	case MSR_IA32_CR_PAT:
-		msr_info->data = vcpu->arch.pat;
-		break;
 	case MSR_MTRRcap:
 	case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000:
 	case MSR_MTRRdefType:

base-commit: 7539434a6984ba5accfdd8e296fb834558f95df4 --

#19 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 9 lines

I like it. And AFAICT it largely Just Works, because the calls from svm_set_nested_state() will always be routed to gpat since the calls are already guarded with is_guest_mode() + nested_npt_enabled().

Side topic, either as a prep patch (to duplicate code) or as a follow-up patch (to move the PAT handling in x86.c to vmx.c), the "common" handling of PAT should be fully forked between VMX and SVM. As of this patch, it's not just misleading, it's actively dangerous since calling kvm_get_msr_common() for SVM would get the wrong value.

+1 on both points.

Quoted 12 lines

FWIW, this is what I ended up with when hacking on top of your patches to see how this played out.

---

arch/x86/kvm/svm/svm.c +0 -4
@@ -2838,16 +2876,7 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		msr_info->data = svm->msr_decfg;
 		break;
 	case MSR_IA32_CR_PAT:
-		/*
-		 * When nested NPT is enabled, L2 has a separate PAT from
-		 * L1.  Guest accesses to IA32_PAT while running L2 target
-		 * L2's gPAT; host-initiated accesses always target L1's

arch/x86/kvm/svm/svm.c static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 2882

Quoted 10 lines
arch/x86/kvm/svm/svm.c +1 -6
-		 * hPAT for backward and forward KVM_GET_MSRS compatibility
-		 * with older kernels.
-		 */
-		WARN_ON_ONCE(msr_info->host_initiated && vcpu->wants_to_run);
-		if (!msr_info->host_initiated && is_guest_mode(vcpu) &&
-		    nested_npt_enabled(svm))
+		if (svm_is_access_to_gpat(vcpu, msr_info->host_initiated))
 			msr_info->data = svm->nested.save.g_pat;
 		else
 			msr_info->data = vcpu->arch.pat;

I'd go a step further here and add svm_get_pat(), then this just becomes:

	msr_info->data = svm_get_pat(vcpu, msr_info->host_initiated);

It's more consistent with svm_set_msr(), and completely abstracts the L1 vs. L2 PAT logic with the helpers.

Quoted 22 lines
arch/x86/kvm/svm/svm.c +2 -13
@@ -2937,19 +2966,8 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 	case MSR_IA32_CR_PAT:
 		if (!kvm_pat_valid(data))
 			return 1;
-		/*
-		 * When nested NPT is enabled, L2 has a separate PAT from
-		 * L1.  Guest accesses to IA32_PAT while running L2 target
-		 * L2's gPAT; host-initiated accesses always target L1's
-		 * hPAT for backward and forward KVM_SET_MSRS compatibility
-		 * with older kernels.
-		 */
-		WARN_ON_ONCE(msr->host_initiated && vcpu->wants_to_run);
-		if (!msr->host_initiated && is_guest_mode(vcpu) &&
-		    nested_npt_enabled(svm))
-			svm_set_gpat(svm, data);
-		else
-			svm_set_hpat(svm, data);
+
+		svm_set_pat(vcpu, data, msr->host_initiated);
 		break;
 	case MSR_IA32_SPEC_CTRL:
 		if (!msr->host_initiated &&
#20 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 22 lines · Yosry Ahmed
Quoted 8 lines
arch/x86/kvm/svm/svm.c +0 -4
@@ -2838,16 +2876,7 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		msr_info->data = svm->msr_decfg;
 		break;
 	case MSR_IA32_CR_PAT:
-		/*
-		 * When nested NPT is enabled, L2 has a separate PAT from
-		 * L1.  Guest accesses to IA32_PAT while running L2 target
-		 * L2's gPAT; host-initiated accesses always target L1's

arch/x86/kvm/svm/svm.c static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 2882

Quoted 10 lines
arch/x86/kvm/svm/svm.c +1 -6
-		 * hPAT for backward and forward KVM_GET_MSRS compatibility
-		 * with older kernels.
-		 */
-		WARN_ON_ONCE(msr_info->host_initiated && vcpu->wants_to_run);
-		if (!msr_info->host_initiated && is_guest_mode(vcpu) &&
-		    nested_npt_enabled(svm))
+		if (svm_is_access_to_gpat(vcpu, msr_info->host_initiated))
 			msr_info->data = svm->nested.save.g_pat;
 		else
 			msr_info->data = vcpu->arch.pat;

I'd go a step further here and add svm_get_pat(), then this just becomes:

Quoted 4 lines
	msr_info->data = svm_get_pat(vcpu, msr_info->host_initiated);

It's more consistent with svm_set_msr(), and completely abstracts the L1 vs. L2 PAT logic with the helpers.

Either way works for me.

#21 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

In reply to Sean Christopherson
Quoted 5 lines · Sean Christopherson

Side topic, either as a prep patch (to duplicate code) or as a follow-up patch (to move the PAT handling in x86.c to vmx.c), the "common" handling of PAT should be fully forked between VMX and SVM. As of this patch, it's not just misleading, it's actively dangerous since calling kvm_get_msr_common() for SVM would get the wrong value.

Though I included this change in v5 and v6, TIL that TDX calls kvm_[gs]et_msr_common(MSR_IA32_CR_PAT), so the common handling is not fully forked.

#22 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 9 lines · Jim Mattson
Quoted 5 lines · Sean Christopherson

Side topic, either as a prep patch (to duplicate code) or as a follow-up patch (to move the PAT handling in x86.c to vmx.c), the "common" handling of PAT should be fully forked between VMX and SVM. As of this patch, it's not just misleading, it's actively dangerous since calling kvm_get_msr_common() for SVM would get the wrong value.

Quoted 3 lines

Though I included this change in v5 and v6, TIL that TDX calls kvm_[gs]et_msr_common(MSR_IA32_CR_PAT), so the common handling is not fully forked.

Do you plan to drop this patch or add PAT handling in tdx_{get/set}_msr()? If you'll drop it, maybe add a warning if kvm_[gs]et_msr_common(MSR_IA32_CR_PAT) is called from SVM?

#23 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 11 lines · Yosry Ahmed
Quoted 9 lines · Jim Mattson
Quoted 5 lines · Sean Christopherson

Side topic, either as a prep patch (to duplicate code) or as a follow-up patch (to move the PAT handling in x86.c to vmx.c), the "common" handling of PAT should be fully forked between VMX and SVM. As of this patch, it's not just misleading, it's actively dangerous since calling kvm_get_msr_common() for SVM would get the wrong value.

Quoted 7 lines
Quoted 3 lines

Though I included this change in v5 and v6, TIL that TDX calls kvm_[gs]et_msr_common(MSR_IA32_CR_PAT), so the common handling is not fully forked.

Do you plan to drop this patch or add PAT handling in tdx_{get/set}_msr()? If you'll drop it, maybe add a warning if kvm_[gs]et_msr_common(MSR_IA32_CR_PAT) is called from SVM?

I plan to leave the MSR_IA32_CR_PAT code in kvm_[gs]et_msr_common(). Replicating the code in tdx.c seems like the wrong direction to go.

There's no precedent that I can see for checking the vendor module in common code (though we do have kvm_x86_ops.name). I could add a warning if invoked with (vcpu->arch.efer & EFER_SVME). WDYT?

#24 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 21 lines · Jim Mattson
Quoted 11 lines · Yosry Ahmed
Quoted 9 lines · Jim Mattson
Quoted 5 lines · Sean Christopherson

Side topic, either as a prep patch (to duplicate code) or as a follow-up patch (to move the PAT handling in x86.c to vmx.c), the "common" handling of PAT should be fully forked between VMX and SVM. As of this patch, it's not just misleading, it's actively dangerous since calling kvm_get_msr_common() for SVM would get the wrong value.

Quoted 7 lines
Quoted 3 lines

Though I included this change in v5 and v6, TIL that TDX calls kvm_[gs]et_msr_common(MSR_IA32_CR_PAT), so the common handling is not fully forked.

Do you plan to drop this patch or add PAT handling in tdx_{get/set}_msr()? If you'll drop it, maybe add a warning if kvm_[gs]et_msr_common(MSR_IA32_CR_PAT) is called from SVM?

Quoted 6 lines

I plan to leave the MSR_IA32_CR_PAT code in kvm_[gs]et_msr_common(). Replicating the code in tdx.c seems like the wrong direction to go.

There's no precedent that I can see for checking the vendor module in common code (though we do have kvm_x86_ops.name). I could add a warning if invoked with (vcpu->arch.efer & EFER_SVME). WDYT?

Yeah this should work. I assume Sean can decide to drop it when applying if he dislikes it, but given that he said the code is misleading and it would be dangerous if we end up calling from SVM -- I personally think the warning will act as documentation for the former and safeguard for the latter.

#25 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

In reply to Sean Christopherson
Quoted 2 lines · Sean Christopherson

Please trim your replies. Scrolling through 100+ lines of quoted text to find the ~12 lines of context that actually matter is annoying.

Ack.

Quoted 2 lines

Actually, calling svm_set_l2_pat() when !is_guest_mode() is wrong too, no? E.g. shouldn't we end up with this?

legacy_gpat_semantics should only be set in guest mode, and it's cleared in pre-run, so before we can exit guest mode IIUC. But having the guard in place for both cases is probably simpler anyway.

Quoted 18 lines

static inline void svm_set_l1_pat(struct vcpu_svm *svm, u64 data)

  {
	svm->vcpu.arch.pat = data;

	if (npt_enabled)
		return;

	vmcb_set_gpat(svm->vmcb01.ptr, data);

	if (is_guest_mode(&svm->vcpu)) {
		if (svm->nested.legacy_gpat_semantics)
			svm_set_l2_pat(svm, data);
		else if (!nested_npt_enabled(svm))
			vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
	}

}

#26 lore raw

Re: [PATCH v4 4/8] KVM: x86: nSVM: Redirect IA32_PAT accesses to either hPAT or gPAT

Quoted 11 lines · Yosry Ahmed
Quoted 2 lines · Sean Christopherson

Please trim your replies. Scrolling through 100+ lines of quoted text to find the ~12 lines of context that actually matter is annoying.

Ack.

Quoted 2 lines

Actually, calling svm_set_l2_pat() when !is_guest_mode() is wrong too, no? E.g. shouldn't we end up with this?

legacy_gpat_semantics should only be set in guest mode, and it's cleared in pre-run, so before we can exit guest mode IIUC.

Never mind that, Jim just mentioned the cases of back-to-back KVM_SET_NESTED_STATE.

#27 lore raw

PATCH v4 5/8 KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT

In reply to Jim Mattson

According to the APM volume 3 pseudo-code for "VMRUN," when nested paging is enabled in the vmcb, the guest PAT register (gPAT) is saved to the vmcb on emulated VMEXIT.

When nested NPT is enabled, save the vmcb02 g_pat field to the vmcb12 g_pat field on emulated VMEXIT.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
---
 arch/x86/kvm/svm/nested.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/svm/nested.c +3 -0
index 69b577a4915c..26f758e294ab 100644
@@ -1312,6 +1312,9 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 	vmcb12->save.dr6    = svm->vcpu.arch.dr6;
 	vmcb12->save.cpl    = vmcb02->save.cpl;
 
+	if (nested_npt_enabled(svm))
+		vmcb12->save.g_pat = vmcb02->save.g_pat;
+
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) {
 		vmcb12->save.s_cet	= vmcb02->save.s_cet;
 		vmcb12->save.isst_addr	= vmcb02->save.isst_addr;
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
Reviewed-by on v4 5/8
#28 lore raw

Re: [PATCH v4 5/8] KVM: x86: nSVM: Save gPAT to vmcb12.g_pat on VMEXIT

Quoted 9 lines · Jim Mattson

According to the APM volume 3 pseudo-code for "VMRUN," when nested paging is enabled in the vmcb, the guest PAT register (gPAT) is saved to the vmcb on emulated VMEXIT.

When nested NPT is enabled, save the vmcb02 g_pat field to the vmcb12 g_pat field on emulated VMEXIT.

Fixes 15038e147247 ("KVM: SVM: obey guest PAT")
Quoted 21 lines
---
 arch/x86/kvm/svm/nested.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/svm/nested.c +3 -0
index 69b577a4915c..26f758e294ab 100644
@@ -1312,6 +1312,9 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 	vmcb12->save.dr6    = svm->vcpu.arch.dr6;
 	vmcb12->save.cpl    = vmcb02->save.cpl;
 
+	if (nested_npt_enabled(svm))
+		vmcb12->save.g_pat = vmcb02->save.g_pat;
+
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) {
 		vmcb12->save.s_cet	= vmcb02->save.s_cet;
 		vmcb12->save.isst_addr	= vmcb02->save.isst_addr;
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#29 lore raw

PATCH v4 6/8 KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

In reply to Jim Mattson

Add a 'flags' field to the SVM nested state header, and use bit 0 of the flags to indicate that gPAT is stored in the nested state.

If in guest mode with NPT enabled, store the current vmcb->save.g_pat value into the header of the nested state, and set the flag.

Note that struct kvm_svm_nested_state_hdr is included in a union padded to 120 bytes, so there is room to add the flags field and the gpat field without changing any offsets.

Fixes cc440cdad5b7 ("KVM: nSVM: implement KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE")
---
 arch/x86/include/uapi/asm/kvm.h |  5 +++++
 arch/x86/kvm/svm/nested.c       | 16 ++++++++++++++++
 2 files changed, 21 insertions(+)
arch/x86/include/uapi/asm/kvm.h +5 -0
index 846a63215ce1..664d04d1db3f 100644
@@ -495,6 +495,8 @@ struct kvm_sync_regs {
 
 #define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE	0x00000001
 
+#define KVM_STATE_SVM_VALID_GPAT	0x00000001
+
 /* vendor-independent attributes for system fd (group 0) */
 #define KVM_X86_GRP_SYSTEM		0
 #  define KVM_X86_XCOMP_GUEST_SUPP	0
@@ -531,6 +533,9 @@ struct kvm_svm_nested_state_data {
 
 struct kvm_svm_nested_state_hdr {
 	__u64 vmcb_pa;
+	__u32 flags;
+	__u32 reserved;
+	__u64 gpat;
 };
 
 /* for KVM_CAP_NESTED_STATE */
arch/x86/kvm/svm/nested.c +16 -0
index 26f758e294ab..f73f3e586012 100644
@@ -1893,6 +1893,10 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
 	/* First fill in the header and copy it out.  */
 	if (is_guest_mode(vcpu)) {
 		kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
+		if (nested_npt_enabled(svm)) {
+			kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+			kvm_state.hdr.svm.gpat = svm->nested.save.g_pat;
+		}
 		kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
 		kvm_state.flags |= KVM_STATE_NESTED_GUEST_MODE;
 
@@ -2022,6 +2026,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	    !nested_vmcb_check_save(vcpu, &save_cached, false))
 		goto out_free;
 
+	/*
+	 * Validate gPAT, if provided. This is done separately from the
+	 * vmcb_save_area_cached validation above, because gPAT is L2
+	 * state, but the vmcb_save_area_cached is populated with L1 state.
+	 */
+	if ((kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT) &&
+	    !kvm_pat_valid(kvm_state->hdr.svm.gpat))
+		goto out_free;
 
 	/*
 	 * All checks done, we can enter guest mode. Userspace provides
@@ -2061,6 +2073,10 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	if (ret)
 		goto out_free;
 
+	if (nested_npt_enabled(svm) &&
+	    (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT))
+		svm_set_gpat(svm, kvm_state->hdr.svm.gpat);
+
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
 	nested_vmcb02_prepare_control(svm, svm->vmcb->save.rip, svm->vmcb->save.cs.base);
 
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
Reviewed-by on v4 6/8
#30 lore raw

Re: [PATCH v4 6/8] KVM: x86: nSVM: Save/restore gPAT with KVM_{GET,SET}_NESTED_STATE

Quoted 6 lines · Jim Mattson

Add a 'flags' field to the SVM nested state header, and use bit 0 of the flags to indicate that gPAT is stored in the nested state.

If in guest mode with NPT enabled, store the current vmcb->save.g_pat value into the header of the nested state, and set the flag.

Quoted 6 lines

Note that struct kvm_svm_nested_state_hdr is included in a union padded to 120 bytes, so there is room to add the flags field and the gpat field without changing any offsets.

Fixes cc440cdad5b7 ("KVM: nSVM: implement KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE")
Quoted 72 lines
---
 arch/x86/include/uapi/asm/kvm.h |  5 +++++
 arch/x86/kvm/svm/nested.c       | 16 ++++++++++++++++
 2 files changed, 21 insertions(+)
arch/x86/include/uapi/asm/kvm.h +5 -0
index 846a63215ce1..664d04d1db3f 100644
@@ -495,6 +495,8 @@ struct kvm_sync_regs {
 
 #define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE	0x00000001
 
+#define KVM_STATE_SVM_VALID_GPAT	0x00000001
+
 /* vendor-independent attributes for system fd (group 0) */
 #define KVM_X86_GRP_SYSTEM		0
 #  define KVM_X86_XCOMP_GUEST_SUPP	0
@@ -531,6 +533,9 @@ struct kvm_svm_nested_state_data {
 
 struct kvm_svm_nested_state_hdr {
 	__u64 vmcb_pa;
+	__u32 flags;
+	__u32 reserved;
+	__u64 gpat;
 };
 
 /* for KVM_CAP_NESTED_STATE */
arch/x86/kvm/svm/nested.c +16 -0
index 26f758e294ab..f73f3e586012 100644
@@ -1893,6 +1893,10 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
 	/* First fill in the header and copy it out.  */
 	if (is_guest_mode(vcpu)) {
 		kvm_state.hdr.svm.vmcb_pa = svm->nested.vmcb12_gpa;
+		if (nested_npt_enabled(svm)) {
+			kvm_state.hdr.svm.flags |= KVM_STATE_SVM_VALID_GPAT;
+			kvm_state.hdr.svm.gpat = svm->nested.save.g_pat;
+		}
 		kvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;
 		kvm_state.flags |= KVM_STATE_NESTED_GUEST_MODE;
 
@@ -2022,6 +2026,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	    !nested_vmcb_check_save(vcpu, &save_cached, false))
 		goto out_free;
 
+	/*
+	 * Validate gPAT, if provided. This is done separately from the
+	 * vmcb_save_area_cached validation above, because gPAT is L2
+	 * state, but the vmcb_save_area_cached is populated with L1 state.
+	 */
+	if ((kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT) &&
+	    !kvm_pat_valid(kvm_state->hdr.svm.gpat))
+		goto out_free;
 
 	/*
 	 * All checks done, we can enter guest mode. Userspace provides
@@ -2061,6 +2073,10 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	if (ret)
 		goto out_free;
 
+	if (nested_npt_enabled(svm) &&
+	    (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT))
+		svm_set_gpat(svm, kvm_state->hdr.svm.gpat);
+
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
 	nested_vmcb02_prepare_control(svm, svm->vmcb->save.rip, svm->vmcb->save.cs.base);
 
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
#31 lore raw

PATCH v4 7/8 KVM: x86: nSVM: Handle restore of legacy nested state

In reply to Jim Mattson

When nested NPT is enabled and KVM_SET_NESTED_STATE is used to restore an old checkpoint (without a valid gPAT), the current IA32_PAT value must be used as L2's gPAT.

Unfortunately, checkpoint restore is non-atomic, and the order in which state components are restored is not specified. Hence, the current IA32_PAT value may be restored by KVM_SET_MSRS after KVM_SET_NESTED_STATE. To further complicate matters, there may be a KVM_GET_NESTED_STATE before the next KVM_RUN.

Introduce a new boolean, svm->nested.legacy_gpat_semantics. When set, hPAT updates are also applied to gPAT, preserving the old behavior (i.e. L2 shares L1's PAT). Set this boolean when restoring legacy state (i.e. nested NPT is enabled, but no GPAT is provided) in KVM_SET_NESTED_STATE. Clear this boolean in svm_vcpu_pre_run(), to ensure that hPAT and gPAT are decoupled before the vCPU resumes execution.

---
 arch/x86/kvm/svm/nested.c | 11 ++++++++---
 arch/x86/kvm/svm/svm.c    |  2 ++
 arch/x86/kvm/svm/svm.h    | 11 +++++++++++
 3 files changed, 21 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/nested.c +8 -3
index f73f3e586012..d854d29b0bd8 100644
@@ -2073,9 +2073,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	if (ret)
 		goto out_free;
 
-	if (nested_npt_enabled(svm) &&
-	    (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT))
-		svm_set_gpat(svm, kvm_state->hdr.svm.gpat);
+	if (nested_npt_enabled(svm)) {
+		if (kvm_state->hdr.svm.flags & KVM_STATE_SVM_VALID_GPAT) {
+			svm_set_gpat(svm, kvm_state->hdr.svm.gpat);
+		} else {
+			svm_set_gpat(svm, vcpu->arch.pat);
+			svm->nested.legacy_gpat_semantics = true;
+		}
+	}
 
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
 	nested_vmcb02_prepare_control(svm, svm->vmcb->save.rip, svm->vmcb->save.cs.base);
arch/x86/kvm/svm/svm.c +2 -0
index 205bf07896ad..d951d25f1f91 100644
@@ -4245,6 +4245,8 @@ static int svm_vcpu_pre_run(struct kvm_vcpu *vcpu)
 	if (to_kvm_sev_info(vcpu->kvm)->need_init)
 		return -EINVAL;
 
+	to_svm(vcpu)->nested.legacy_gpat_semantics = false;
+
 	return 1;
 }
 
arch/x86/kvm/svm/svm.h +11 -0
index 88549705133f..0bb9fdcb489d 100644
@@ -238,6 +238,15 @@ struct svm_nested_state {
 	 * on its side.
 	 */
 	bool force_msr_bitmap_recalc;
+
+	/*
+	 * Indicates that a legacy nested state (without a valid gPAT) was
+	 * recently restored. Until the next KVM_RUN, updates to hPAT are
+	 * also applied to gPAT, preserving legacy behavior (i.e. L2 shares
+	 * L1's PAT). Because checkpoint restore is non-atomic, this
+	 * complication is necessary for backward compatibility.
+	 */
+	bool legacy_gpat_semantics;
 };
 
 struct vcpu_sev_es_state {
@@ -621,6 +630,8 @@ static inline void svm_set_hpat(struct vcpu_svm *svm, u64 data)
 		if (is_guest_mode(&svm->vcpu) && !nested_npt_enabled(svm))
 			vmcb_set_gpat(svm->nested.vmcb02.ptr, data);
 	}
+	if (svm->nested.legacy_gpat_semantics)
+		svm_set_gpat(svm, data);
 }
 
 static inline bool nested_vnmi_enabled(struct vcpu_svm *svm)
Signature · 1 line
2.53.0.239.g8d8fc8a987-goog
Reviewed-by on v4 7/8
#32 lore raw

Re: [PATCH v4 7/8] KVM: x86: nSVM: Handle restore of legacy nested state

Quoted 10 lines · Jim Mattson

When nested NPT is enabled and KVM_SET_NESTED_STATE is used to restore an old checkpoint (without a valid gPAT), the current IA32_PAT value must be used as L2's gPAT.

Unfortunately, checkpoint restore is non-atomic, and the order in which state components are restored is not specified. Hence, the current IA32_PAT value may be restored by KVM_SET_MSRS after KVM_SET_NESTED_STATE. To further complicate matters, there may be a KVM_GET_NESTED_STATE before the next KVM_RUN.

Quoted 8 lines

Introduce a new boolean, svm->nested.legacy_gpat_semantics. When set, hPAT updates are also applied to gPAT, preserving the old behavior (i.e. L2 shares L1's PAT). Set this boolean when restoring legacy state (i.e. nested NPT is enabled, but no GPAT is provided) in KVM_SET_NESTED_STATE. Clear this boolean in svm_vcpu_pre_run(), to ensure that hPAT and gPAT are decoupled before the vCPU resumes execution.

#33 lore raw

PATCH v4 8/8 KVM: selftests: nSVM: Add svm_nested_pat test

In reply to Jim Mattson

Verify that KVM correctly virtualizes the host PAT MSR and the guest PAT register for nested SVM guests.

With nested NPT disabled:
* L1 and L2 share the same PAT
* The vmcb12.g_pat is ignored

With nested NPT enabled:
* An invalid g_pat in vmcb12 causes VMEXIT_INVALID
* RDMSR(IA32_PAT) from L2 returns the value of the guest PAT register
* WRMSR(IA32_PAT) from L2 is reflected in vmcb12's g_pat on VMEXIT
* RDMSR(IA32_PAT) from L1 returns the value of the host PAT MSR
* Save/restore with the vCPU in guest mode preserves both hPAT and gPAT

---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/svm_nested_pat_test.c   | 298 ++++++++++++++++++
 2 files changed, 299 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_pat_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 7810f9db5f77..5554e40f73f8 100644
@@ -110,6 +110,7 @@ TEST_GEN_PROGS_x86 += x86/state_test
 TEST_GEN_PROGS_x86 += x86/vmx_preemption_timer_test
 TEST_GEN_PROGS_x86 += x86/svm_vmcall_test
 TEST_GEN_PROGS_x86 += x86/svm_int_ctl_test
+TEST_GEN_PROGS_x86 += x86/svm_nested_pat_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test
 TEST_GEN_PROGS_x86 += x86/svm_lbr_nested_state
tools/testing/selftests/kvm/x86/svm_nested_pat_test.c new file +298 -0

298 lines not on this page · raw message

Signature · 1 line
2.53.0.239.g8d8fc8a987-goog