Skip to contentSkip to conversation

KVM: X86: Fix and clean up for register caches

PATCH In review Lai Jiangshan 5 participants 48 messages Nov 8, 2021 – Dec 15, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover KVM: X86: Fix and clean up for register caches

From: Lai Jiangshan <[email protected]>

The patchset was started when I read the code of nested_svm_load_cr3() and found that it marks CR3 available other than dirty when changing vcpu->arch.cr3. I thought its caller has ensured that vmcs.GUEST_CR3 will be or already be set to @cr3 so that it doesn't need to be marked dirty. And later I found that it is not true and it must be a bug in a rare case before I realized that all the code just (ab)uses vcpu->arch.regs_avail for VCPU_EXREG_CR3 and there is not such bug of using regs_avail here.
(The above finding becomes a low meaning patch_15 rather than a fix)

The unhappyness of the reading code made me do some cleanup for regs_avail and regs_dirty and kvm_register_xxx() functions in the hope that the code become clearer with less misunderstanding.

Major focus was on VCPU_EXREG_CR3 and VCPU_EXREG_PDPTR. They are ensured to be marked the correct tags (available or dirty), and the value is ensured to be synced to architecture before run if it is marked dirty.

When cleaning VCPU_EXREG_PDPTR, I also checked if the corresponding cr0/cr4 pdptr bits are all intercepted when !tdp_enabled, and I think it is not clear enough, so X86_CR4_PDPTR_BITS is added as self-comments in the code.

Lai Jiangshan (15):
KVM: X86: Ensure the dirty PDPTEs to be loaded
KVM: VMX: Mark VCPU_EXREG_PDPTR available in ept_save_pdptrs()
KVM: SVM: Always clear available of VCPU_EXREG_PDPTR in svm_vcpu_run()
KVM: VMX: Add and use X86_CR4_TLB_BITS when !enable_ept
KVM: VMX: Add and use X86_CR4_PDPTR_BITS when !enable_ept
KVM: X86: Move CR0 pdptr_bits into header file as X86_CR0_PDPTR_BITS
KVM: SVM: Remove outdate comment in svm_load_mmu_pgd()
KVM: SVM: Remove useless check in svm_load_mmu_pgd()
KVM: SVM: Remove the unneeded code to mark available for CR3
KVM: X86: Mark CR3 dirty when vcpu->arch.cr3 is changed
KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty
KVM: VMX: Reset the bits that are meaningful to be reset in
vmx_register_cache_reset()
KVM: SVM: Add and use svm_register_cache_reset()
KVM: X86: Remove kvm_register_clear_available()
KVM: nVMX: Always write vmcs.GUEST_CR3 during nested VM-Exit

 arch/x86/kvm/kvm_cache_regs.h | 13 ++++++------
 arch/x86/kvm/svm/nested.c     |  1 -
 arch/x86/kvm/svm/svm.c        | 17 ++++++++--------
 arch/x86/kvm/svm/svm.h        | 26 ++++++++++++++++++++++++
 arch/x86/kvm/vmx/nested.c     | 30 ++++++++++++++++++----------
 arch/x86/kvm/vmx/vmx.c        | 12 +++++++-----
 arch/x86/kvm/vmx/vmx.h        | 37 +++++++++++++++++++++++++----------
 arch/x86/kvm/x86.c            | 13 ++++++------
 8 files changed, 101 insertions(+), 48 deletions(-)
Signature · 1 line
2.19.1.6.gb485710b
#1 lore raw

PATCH 1/15 KVM: X86: Ensure the dirty PDPTEs to be loaded

From: Lai Jiangshan <[email protected]>

For VMX, the dirty PDPTEs needs to be loaded before the coming VMENTER via vmx_load_mmu_pgd() if EPT is enabled.

But not all paths that call load_pdptrs() will cause vmx_load_mmu_pgd() to be invoked. Normally, kvm_mmu_reset_context() and KVM_REQ_LOAD_MMU_PGD are used to launch later vmx_load_mmu_pgd().

The commit d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR0.CD and CR0.NW.

The commit 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush") skips KVM_REQ_LOAD_MMU_PGD after load_pdptrs() when rewriting the CR3 with the same value.

The commit a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR4.PGE.

Fixes d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed")
Fixes 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush")
Fixes a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE")
---
 arch/x86/kvm/x86.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kvm/x86.c +2 -0
index ac83d873d65b..034c233ea5a1 100644
@@ -830,6 +830,8 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
 
 	memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
 	kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
+	/* Ensure the dirty PDPTEs to be loaded. */
+	kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
 	vcpu->arch.pdptrs_from_userspace = false;
 
 	return 1;
Signature · 1 line
2.19.1.6.gb485710b
#2 lore raw

PATCH 2/15 KVM: VMX: Mark VCPU_EXREG_PDPTR available in ept_save_pdptrs()

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

mmu->pdptrs[] and vmcs.GUEST_PDPTR[0-3] are synced, so mmu->pdptrs is available and GUEST_PDPTR[0-3] is not dirty.

---
 arch/x86/kvm/vmx/vmx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +1 -1
index 39db4f56bffd..79e5df5fbb32 100644
@@ -3001,7 +3001,7 @@ void ept_save_pdptrs(struct kvm_vcpu *vcpu)
 	mmu->pdptrs[2] = vmcs_read64(GUEST_PDPTR2);
 	mmu->pdptrs[3] = vmcs_read64(GUEST_PDPTR3);
 
-	kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
+	kvm_register_mark_available(vcpu, VCPU_EXREG_PDPTR);
 }
 
 #define CR3_EXITING_BITS (CPU_BASED_CR3_LOAD_EXITING | \
Signature · 1 line
2.19.1.6.gb485710b
#3 lore raw

PATCH 3/15 KVM: SVM: Always clear available of VCPU_EXREG_PDPTR in svm_vcpu_run()

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

Make it the same logic to handle the availability of VCPU_EXREG_PDPTR as VMX and also remove a branch in svm_vcpu_run().

---
 arch/x86/kvm/svm/svm.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/svm.c +9 -4
index 88a730ad47a1..3e7043173668 100644
@@ -1583,10 +1583,16 @@ static void svm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags)
 
 static void svm_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg)
 {
+	kvm_register_mark_available(vcpu, reg);
+
 	switch (reg) {
 	case VCPU_EXREG_PDPTR:
-		BUG_ON(!npt_enabled);
-		load_pdptrs(vcpu, vcpu->arch.walk_mmu, kvm_read_cr3(vcpu));
+		/*
+		 * When !npt_enabled, mmu->pdptrs[] is already available since
+		 * it is always updated per SDM when moving to CRs.
+		 */
+		if (npt_enabled)
+			load_pdptrs(vcpu, vcpu->arch.walk_mmu, kvm_read_cr3(vcpu));
 		break;
 	default:
 		KVM_BUG_ON(1, vcpu->kvm);
@@ -3964,8 +3970,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
 	svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
 	vmcb_mark_all_clean(svm->vmcb);
 
-	if (npt_enabled)
-		kvm_register_clear_available(vcpu, VCPU_EXREG_PDPTR);
+	kvm_register_clear_available(vcpu, VCPU_EXREG_PDPTR);
 
 	/*
 	 * We need to handle MC intercepts here before the vcpu has a chance to
Signature · 1 line
2.19.1.6.gb485710b
#4 lore raw

PATCH 4/15 KVM: VMX: Add and use X86_CR4_TLB_BITS when !enable_ept

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

In set_cr4_guest_host_mask(), X86_CR4_PGE is set to be intercepted when !enable_ept just because X86_CR4_PGE is the only bit that is responsible for flushing TLB but listed in KVM_POSSIBLE_CR4_GUEST_BITS.

It is clearer and self-documented to use X86_CR4_TLB_BITS instead.

No functionality changed.

---
 arch/x86/kvm/kvm_cache_regs.h | 2 ++
 arch/x86/kvm/vmx/vmx.c        | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/kvm_cache_regs.h +2 -0
index 90e1ffdc05b7..8fe036efa654 100644
@@ -9,6 +9,8 @@
 	(X86_CR4_PVI | X86_CR4_DE | X86_CR4_PCE | X86_CR4_OSFXSR  \
 	 | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE)
 
+#define X86_CR4_TLB_BITS (X86_CR4_PGE | X86_CR4_PCIDE | X86_CR4_PAE | X86_CR4_SMEP)
+
 #define BUILD_KVM_GPR_ACCESSORS(lname, uname)				      \
 static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *vcpu)\
 {									      \
arch/x86/kvm/vmx/vmx.c +1 -1
index 79e5df5fbb32..1795702dc6de 100644
@@ -4107,7 +4107,7 @@ void set_cr4_guest_host_mask(struct vcpu_vmx *vmx)
 	vcpu->arch.cr4_guest_owned_bits = KVM_POSSIBLE_CR4_GUEST_BITS &
 					  ~vcpu->arch.cr4_guest_rsvd_bits;
 	if (!enable_ept)
-		vcpu->arch.cr4_guest_owned_bits &= ~X86_CR4_PGE;
+		vcpu->arch.cr4_guest_owned_bits &= ~X86_CR4_TLB_BITS;
 	if (is_guest_mode(&vmx->vcpu))
 		vcpu->arch.cr4_guest_owned_bits &=
 			~get_vmcs12(vcpu)->cr4_guest_host_mask;
Signature · 1 line
2.19.1.6.gb485710b
maintainer
#5 lore raw

Re: [PATCH 04/15] KVM: VMX: Add and use X86_CR4_TLB_BITS when !enable_ept

Quoted 7 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

In set_cr4_guest_host_mask(), X86_CR4_PGE is set to be intercepted when !enable_ept just because X86_CR4_PGE is the only bit that is responsible for flushing TLB but listed in KVM_POSSIBLE_CR4_GUEST_BITS.

It is clearer and self-documented to use X86_CR4_TLB_BITS instead.

Very good idea, but I'd go with a slightly clearer X86_CR4_TLBFLUSH_BITS.

Paolo

Quoted 35 lines

No functionality changed.

---
  arch/x86/kvm/kvm_cache_regs.h | 2 ++
  arch/x86/kvm/vmx/vmx.c        | 2 +-
  2 files changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/kvm_cache_regs.h +2 -0
index 90e1ffdc05b7..8fe036efa654 100644
@@ -9,6 +9,8 @@
  	(X86_CR4_PVI | X86_CR4_DE | X86_CR4_PCE | X86_CR4_OSFXSR  \
  	 | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE)
 
+#define X86_CR4_TLB_BITS (X86_CR4_PGE | X86_CR4_PCIDE | X86_CR4_PAE | X86_CR4_SMEP)
+
  #define BUILD_KVM_GPR_ACCESSORS(lname, uname)				      \
  static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *vcpu)\
  {									      \
arch/x86/kvm/vmx/vmx.c +1 -1
index 79e5df5fbb32..1795702dc6de 100644
@@ -4107,7 +4107,7 @@ void set_cr4_guest_host_mask(struct vcpu_vmx *vmx)
  	vcpu->arch.cr4_guest_owned_bits = KVM_POSSIBLE_CR4_GUEST_BITS &
  					  ~vcpu->arch.cr4_guest_rsvd_bits;
  	if (!enable_ept)
-		vcpu->arch.cr4_guest_owned_bits &= ~X86_CR4_PGE;
+		vcpu->arch.cr4_guest_owned_bits &= ~X86_CR4_TLB_BITS;
  	if (is_guest_mode(&vmx->vcpu))
  		vcpu->arch.cr4_guest_owned_bits &=
  			~get_vmcs12(vcpu)->cr4_guest_host_mask;
#6 lore raw

PATCH 5/15 KVM: VMX: Add and use X86_CR4_PDPTR_BITS when !enable_ept

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

In set_cr4_guest_host_mask(), all cr4 pdptr bits are already set to be intercepted in an unclear way.

Add X86_CR4_PDPTR_BITS to make it clear and self-documented.

No functionality changed.

---
 arch/x86/kvm/kvm_cache_regs.h | 1 +
 arch/x86/kvm/vmx/vmx.c        | 4 +++-
 arch/x86/kvm/x86.c            | 4 +---
 3 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/kvm/kvm_cache_regs.h +1 -0
index 8fe036efa654..592f9eb9753b 100644
@@ -10,6 +10,7 @@
 	 | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE)
 
 #define X86_CR4_TLB_BITS (X86_CR4_PGE | X86_CR4_PCIDE | X86_CR4_PAE | X86_CR4_SMEP)
+#define X86_CR4_PDPTR_BITS (X86_CR4_PGE | X86_CR4_PSE | X86_CR4_PAE | X86_CR4_SMEP)
 
 #define BUILD_KVM_GPR_ACCESSORS(lname, uname)				      \
 static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *vcpu)\
arch/x86/kvm/vmx/vmx.c +3 -1
index 1795702dc6de..d94e51e9c08f 100644
@@ -4106,8 +4106,10 @@ void set_cr4_guest_host_mask(struct vcpu_vmx *vmx)
 
 	vcpu->arch.cr4_guest_owned_bits = KVM_POSSIBLE_CR4_GUEST_BITS &
 					  ~vcpu->arch.cr4_guest_rsvd_bits;
-	if (!enable_ept)
+	if (!enable_ept) {
 		vcpu->arch.cr4_guest_owned_bits &= ~X86_CR4_TLB_BITS;
+		vcpu->arch.cr4_guest_owned_bits &= ~X86_CR4_PDPTR_BITS;
+	}
 	if (is_guest_mode(&vmx->vcpu))
 		vcpu->arch.cr4_guest_owned_bits &=
 			~get_vmcs12(vcpu)->cr4_guest_host_mask;
arch/x86/kvm/x86.c +1 -3
index 034c233ea5a1..b92d4241b4d9 100644
@@ -1052,8 +1052,6 @@ EXPORT_SYMBOL_GPL(kvm_post_set_cr4);
 int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 {
 	unsigned long old_cr4 = kvm_read_cr4(vcpu);
-	unsigned long pdptr_bits = X86_CR4_PGE | X86_CR4_PSE | X86_CR4_PAE |
-				   X86_CR4_SMEP;
 
 	if (!kvm_is_valid_cr4(vcpu, cr4))
 		return 1;
@@ -1064,7 +1062,7 @@ int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 		if ((cr4 ^ old_cr4) & X86_CR4_LA57)
 			return 1;
 	} else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE)
-		   && ((cr4 ^ old_cr4) & pdptr_bits)
+		   && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS)
 		   && !load_pdptrs(vcpu, vcpu->arch.walk_mmu,
 				   kvm_read_cr3(vcpu)))
 		return 1;
Signature · 1 line
2.19.1.6.gb485710b
#7 lore raw

PATCH 6/15 KVM: X86: Move CR0 pdptr_bits into header file as X86_CR0_PDPTR_BITS

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

Not functionality changed.

---
 arch/x86/kvm/kvm_cache_regs.h | 3 +++
 arch/x86/kvm/x86.c            | 3 +--
 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/kvm/kvm_cache_regs.h +3 -0
index 592f9eb9753b..54a996adb18d 100644
@@ -9,9 +9,12 @@
 	(X86_CR4_PVI | X86_CR4_DE | X86_CR4_PCE | X86_CR4_OSFXSR  \
 	 | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE)
 
+#define X86_CR0_PDPTR_BITS (X86_CR0_CD | X86_CR0_NW | X86_CR0_PG)
 #define X86_CR4_TLB_BITS (X86_CR4_PGE | X86_CR4_PCIDE | X86_CR4_PAE | X86_CR4_SMEP)
 #define X86_CR4_PDPTR_BITS (X86_CR4_PGE | X86_CR4_PSE | X86_CR4_PAE | X86_CR4_SMEP)
 
+static_assert(!(KVM_POSSIBLE_CR0_GUEST_BITS & X86_CR0_PDPTR_BITS));
+
 #define BUILD_KVM_GPR_ACCESSORS(lname, uname)				      \
 static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *vcpu)\
 {									      \
arch/x86/kvm/x86.c +1 -2
index b92d4241b4d9..e5f5042d4842 100644
@@ -858,7 +858,6 @@ EXPORT_SYMBOL_GPL(kvm_post_set_cr0);
 int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
 {
 	unsigned long old_cr0 = kvm_read_cr0(vcpu);
-	unsigned long pdptr_bits = X86_CR0_CD | X86_CR0_NW | X86_CR0_PG;
 
 	cr0 |= X86_CR0_ET;
 
@@ -888,7 +887,7 @@ int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
 	}
 #endif
 	if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) &&
-	    is_pae(vcpu) && ((cr0 ^ old_cr0) & pdptr_bits) &&
+	    is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) &&
 	    !load_pdptrs(vcpu, vcpu->arch.walk_mmu, kvm_read_cr3(vcpu)))
 		return 1;
 
Signature · 1 line
2.19.1.6.gb485710b
#8 lore raw

PATCH 7/15 KVM: SVM: Remove outdated comment in svm_load_mmu_pgd()

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

The comment had been added in the commit 689f3bf21628 ("KVM: x86: unify callbacks to load paging root") and its related code was removed later, and it has nothing to do with the next line of code.

So the comment should be removed too.

---
 arch/x86/kvm/svm/svm.c | 1 -
 1 file changed, 1 deletion(-)
arch/x86/kvm/svm/svm.c +0 -1
index 3e7043173668..e3607fa025d3 100644
@@ -4000,7 +4000,6 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 
 		hv_track_root_tdp(vcpu, root_hpa);
 
-		/* Loading L2's CR3 is handled by enter_svm_guest_mode.  */
 		if (!test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
 			return;
 		cr3 = vcpu->arch.cr3;
Signature · 1 line
2.19.1.6.gb485710b
#9 lore raw

PATCH 8/15 KVM: SVM: Remove useless check in svm_load_mmu_pgd()

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

VCPU_EXREG_CR3 is never cleared from vcpu->arch.regs_avail in SVM so the if-branch is always false and useless, just remove it.

---
 arch/x86/kvm/svm/svm.c | 2 --
 1 file changed, 2 deletions(-)
arch/x86/kvm/svm/svm.c +0 -2
index e3607fa025d3..b7da66935e72 100644
@@ -4000,8 +4000,6 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 
 		hv_track_root_tdp(vcpu, root_hpa);
 
-		if (!test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
-			return;
 		cr3 = vcpu->arch.cr3;
 	} else if (vcpu->arch.mmu->shadow_root_level >= PT64_ROOT_4LEVEL) {
 		cr3 = __sme_set(root_hpa) | kvm_get_active_pcid(vcpu);
Signature · 1 line
2.19.1.6.gb485710b
#10 lore raw

PATCH 9/15 KVM: SVM: Remove the unneeded code to mark available for CR3

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

VCPU_EXREG_CR3 is never cleared from vcpu->arch.regs_avail in SVM so marking available for CR3 is mere an NOP, just remove it.

And it is not required to mark it dirty since VCPU_EXREG_CR3 is neither never cleared from vcpu->arch.regs_dirty and SVM doesn't use the dirty information of VCPU_EXREG_CR3.

---
 arch/x86/kvm/svm/nested.c | 1 -
 1 file changed, 1 deletion(-)
arch/x86/kvm/svm/nested.c +0 -1
index 13a58722e097..2d88ff584d61 100644
@@ -444,7 +444,6 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 		kvm_mmu_new_pgd(vcpu, cr3);
 
 	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
 
 	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
 	kvm_init_mmu(vcpu);
Signature · 1 line
2.19.1.6.gb485710b
maintainer
#11 lore raw

Re: [PATCH 09/15] KVM: SVM: Remove the unneeded code to mark available for CR3

Quoted 20 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

VCPU_EXREG_CR3 is never cleared from vcpu->arch.regs_avail in SVM so marking available for CR3 is mere an NOP, just remove it.

And it is not required to mark it dirty since VCPU_EXREG_CR3 is neither never cleared from vcpu->arch.regs_dirty and SVM doesn't use the dirty information of VCPU_EXREG_CR3.

---
  arch/x86/kvm/svm/nested.c | 1 -
  1 file changed, 1 deletion(-)
arch/x86/kvm/svm/nested.c +0 -0
index 13a58722e097..2d88ff584d61 100644

arch/x86/kvm/svm/nested.c static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, · line 449

Quoted 7 lines
arch/x86/kvm/svm/nested.c +0 -1
@@ -444,7 +444,6 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
  		kvm_mmu_new_pgd(vcpu, cr3);
 
  	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
 
  	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
  	kvm_init_mmu(vcpu);

These two patches can be merged, I think, with a commit message like

    VCPU_EXREG_CR3 is never cleared from vcpu->arch.regs_avail or
    vcpu->arch.regs_dirty in SVM; therefore, marking CR3 as available is
    merely a NOP, and testing it will likewise always succeed.

Paolo

#12 lore raw

PATCH 10/15 KVM: X86: Mark CR3 dirty when vcpu->arch.cr3 is changed

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When vcpu->arch.cr3 is changed, it should be marked dirty unless it is being updated to the value of the architecture guest CR3 (i.e. VMX.GUEST_CR3 or vmcb->save.cr3 when tdp is enabled).

This patch has no functionality changed because kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3) is superset of kvm_register_mark_available(vcpu, VCPU_EXREG_CR3) with additional change to vcpu->arch.regs_dirty, but no code uses regs_dirty for VCPU_EXREG_CR3. (vmx_load_mmu_pgd() uses vcpu->arch.regs_avail instead to test if VCPU_EXREG_CR3 dirty which means current code (ab)uses regs_avail for VCPU_EXREG_CR3 dirty information.)

---
 arch/x86/kvm/vmx/nested.c | 2 +-
 arch/x86/kvm/x86.c        | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/nested.c +1 -1
index dc0e5f80715d..ee5a68c2ea3a 100644
@@ -1134,7 +1134,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 		kvm_mmu_new_pgd(vcpu, cr3);
 
 	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
+	kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
 
 	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
 	kvm_init_mmu(vcpu);
arch/x86/kvm/x86.c +2 -2
index e5f5042d4842..6ca19cac4aff 100644
@@ -1159,7 +1159,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
 		kvm_mmu_new_pgd(vcpu, cr3);
 
 	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
+	kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
 
 handle_tlb_flush:
 	/*
@@ -10591,7 +10591,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
 	vcpu->arch.cr2 = sregs->cr2;
 	*mmu_reset_needed |= kvm_read_cr3(vcpu) != sregs->cr3;
 	vcpu->arch.cr3 = sregs->cr3;
-	kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
+	kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
 
 	kvm_set_cr8(vcpu, sregs->cr8);
 
Signature · 1 line
2.19.1.6.gb485710b
#13 lore raw

PATCH 11/15 KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When vcpu->arch.cr3 is changed, it is marked dirty, so vmcs.GUEST_CR3 can be updated only when kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3).

---
 arch/x86/kvm/vmx/vmx.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +2 -2
index d94e51e9c08f..38b65b97fb7b 100644
@@ -3126,9 +3126,9 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 
 		if (!enable_unrestricted_guest && !is_paging(vcpu))
 			guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr;
-		else if (test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
+		else if (kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3))
 			guest_cr3 = vcpu->arch.cr3;
-		else /* vmcs01.GUEST_CR3 is already up-to-date. */
+		else /* vmcs.GUEST_CR3 is already up-to-date. */
 			update_guest_cr3 = false;
 		vmx_ept_load_pdptrs(vcpu);
 	} else {
Signature · 1 line
2.19.1.6.gb485710b
#14 lore raw

Re: [PATCH 11/15] KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty

Quoted 16 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When vcpu->arch.cr3 is changed, it is marked dirty, so vmcs.GUEST_CR3 can be updated only when kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3).

---
 arch/x86/kvm/vmx/vmx.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index d94e51e9c08f..38b65b97fb7b 100644
@@ -3126,9 +3126,9 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 

arch/x86/kvm/vmx/vmx.c static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · line 3134

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +2 -2
 		if (!enable_unrestricted_guest && !is_paging(vcpu))
 			guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr;
-		else if (test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
+		else if (kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3))
 			guest_cr3 = vcpu->arch.cr3;
-		else /* vmcs01.GUEST_CR3 is already up-to-date. */
+		else /* vmcs.GUEST_CR3 is already up-to-date. */
 			update_guest_cr3 = false;
 		vmx_ept_load_pdptrs(vcpu);
 	} else {

I just bisected this patch to break booting a VM with ept=1 but unrestricted_guest=0 (I needed to re-test unrestricted_guest=0 bug related to SMM, but didn't want to boot without EPT. With ept=0,the VM boots with this patch applied).

Reverting this patch on top of kvm/queue also works.

Best regards,
Maxim levitsky

#15 lore raw

Re: [PATCH 11/15] KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty

Quoted 17 lines · Maxim Levitsky
Quoted 16 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When vcpu->arch.cr3 is changed, it is marked dirty, so vmcs.GUEST_CR3 can be updated only when kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3).

---
  arch/x86/kvm/vmx/vmx.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index d94e51e9c08f..38b65b97fb7b 100644
@@ -3126,9 +3126,9 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 
Quoted 16 lines

arch/x86/kvm/vmx/vmx.c static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · line 3134

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +2 -2
  		if (!enable_unrestricted_guest && !is_paging(vcpu))
  			guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr;
-		else if (test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
+		else if (kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3))
  			guest_cr3 = vcpu->arch.cr3;
-		else /* vmcs01.GUEST_CR3 is already up-to-date. */
+		else /* vmcs.GUEST_CR3 is already up-to-date. */
  			update_guest_cr3 = false;
  		vmx_ept_load_pdptrs(vcpu);
  	} else {

I just bisected this patch to break booting a VM with ept=1 but unrestricted_guest=0 (I needed to re-test unrestricted_guest=0 bug related to SMM, but didn't want to boot without EPT. With ept=0,the VM boots with this patch applied).

Thanks for reporting.

Sorry, I never tested it with unrestricted_guest=0. I can't reproduce it now shortly with unrestricted_guest=0. Maybe it can be reproduced easily if I try more guests or I write a piece of guest code to deliberate hit it if the following analyses is correct.

All the paths changing %cr3 are followed with kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3) and GUEST_CR3 will be expected to be updated.

What I missed is the case of "if (!enable_unrestricted_guest && !is_paging(vcpu))" in vmx_load_mmu_pgd() which doesn't load GUEST_CR3 but clears dirty of VCPU_EXREG_CR3 (when after next run).

So when CR0 !PG -> PG, VCPU_EXREG_CR3 dirty bit should be set.

Maybe adding the following patch on top of the original patch can work.

Thanks
Lai

arch/x86/kvm/x86.c +1 -0
index 85127b3e3690..55b45005ebb9 100644
@@ -858,6 +858,7 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon
 	if ((cr0 ^ old_cr0) & X86_CR0_PG) {
 		kvm_clear_async_pf_completion_queue(vcpu);
 		kvm_async_pf_hash_reset(vcpu);
+		kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
 	}
 
 	if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS)
#16 lore raw

Re: [PATCH 11/15] KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty

Quoted 4 lines · Lai Jiangshan

What I missed is the case of "if (!enable_unrestricted_guest && !is_paging(vcpu))" in vmx_load_mmu_pgd() which doesn't load GUEST_CR3 but clears dirty of VCPU_EXREG_CR3 (when after next run).

Oops.

What I missed is the case of "if (!enable_unrestricted_guest && !is_paging(vcpu))" in vmx_load_mmu_pgd() which doesn't load GUEST_CR3 with the guest cr3 and VCPU_EXREG_CR3 dirty bit is cleared after VMEXIT. When !PG -> PG, GUEST_CR3 is still the ept_identity_map_addr, and VCPU_EXREG_CR3 dirty bit is not set, so vmx_load_mmu_pgd() doesn't update GUEST_CR3.

Quoted 2 lines

So when CR0 !PG -> PG, VCPU_EXREG_CR3 dirty bit should be set.

#17 lore raw

Re: [PATCH 11/15] KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty

In reply to Lai Jiangshan
Quoted 62 lines · Lai Jiangshan
Quoted 23 lines · Maxim Levitsky
Quoted 22 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When vcpu->arch.cr3 is changed, it is marked dirty, so vmcs.GUEST_CR3 can be updated only when kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3).

---
  arch/x86/kvm/vmx/vmx.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +2 -2
index d94e51e9c08f..38b65b97fb7b 100644
@@ -3126,9 +3126,9 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
  		if (!enable_unrestricted_guest && !is_paging(vcpu))
  			guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr;
-		else if (test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
+		else if (kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3))
  			guest_cr3 = vcpu->arch.cr3;
-		else /* vmcs01.GUEST_CR3 is already up-to-date. */
+		else /* vmcs.GUEST_CR3 is already up-to-date. */
Quoted 9 lines

arch/x86/kvm/vmx/vmx.c static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · line 3133

Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +0 -0
  			update_guest_cr3 = false;
  		vmx_ept_load_pdptrs(vcpu);
  	} else {

I just bisected this patch to break booting a VM with ept=1 but unrestricted_guest=0 (I needed to re-test unrestricted_guest=0 bug related to SMM, but didn't want to boot without EPT. With ept=0,the VM boots with this patch applied).

Thanks for reporting.

Sorry, I never tested it with unrestricted_guest=0. I can't reproduce it now shortly with unrestricted_guest=0. Maybe it can be reproduced easily if I try more guests or I write a piece of guest code to deliberate hit it if the following analyses is correct.

All the paths changing %cr3 are followed with kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3) and GUEST_CR3 will be expected to be updated.

What I missed is the case of "if (!enable_unrestricted_guest && !is_paging(vcpu))" in vmx_load_mmu_pgd() which doesn't load GUEST_CR3 but clears dirty of VCPU_EXREG_CR3 (when after next run).

So when CR0 !PG -> PG, VCPU_EXREG_CR3 dirty bit should be set.

Maybe adding the following patch on top of the original patch can work.

Thanks
Lai

arch/x86/kvm/x86.c +0 -0
index 85127b3e3690..55b45005ebb9 100644

arch/x86/kvm/x86.c void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon · line 861

Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
@@ -858,6 +858,7 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon
 	if ((cr0 ^ old_cr0) & X86_CR0_PG) {
 		kvm_clear_async_pf_completion_queue(vcpu);
 		kvm_async_pf_hash_reset(vcpu);
+		kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);

The better place for this is the path in vmx_set_cr0() that handles EPT + !URG. I believe nested_vmx_restore_host_state(), which is used to restore host state if hardware detects a VM-Fail that KVM misses on nested VM-Enter, is also missing a call to mark CR3 dirty.

arch/x86/kvm/vmx/nested.c +1 -1
index 2f6f465e575f..b0c3eb80f5d5 100644
@@ -4426,7 +4426,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)
 
        nested_ept_uninit_mmu_context(vcpu);
        vcpu->arch.cr3 = vmcs_readl(GUEST_CR3);
-       kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
+       kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
 
        /*
         * Use ept_save_pdptrs(vcpu) to load the MMU's cached PDPTRs
arch/x86/kvm/vmx/vmx.c +3 -1
index 640f4719612c..1701cee96678 100644
@@ -3066,8 +3066,10 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
                }
 
                /* Note, vmx_set_cr4() consumes the new vcpu->arch.cr0. */
-               if ((old_cr0_pg ^ cr0) & X86_CR0_PG)
+               if ((old_cr0_pg ^ cr0) & X86_CR0_PG) {
                        vmx_set_cr4(vcpu, kvm_read_cr4(vcpu));
+                       kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
+               }
        }
 
        /* depends on vcpu->arch.cr0 to be set to a new value */
maintainer
#18 lore raw

Re: [PATCH 11/15] KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty

Quoted 10 lines · Sean Christopherson

The better place for this is the path in vmx_set_cr0() that handles EPT + !URG. I believe nested_vmx_restore_host_state(), which is used to restore host state if hardware detects a VM-Fail that KVM misses on nested VM-Enter, is also missing a call to mark CR3 dirty.

arch/x86/kvm/vmx/nested.c +0 -0
index 2f6f465e575f..b0c3eb80f5d5 100644

arch/x86/kvm/vmx/nested.c static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu) · line 4432

Quoted 8 lines
arch/x86/kvm/vmx/nested.c +1 -1
@@ -4426,7 +4426,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)
 
         nested_ept_uninit_mmu_context(vcpu);
         vcpu->arch.cr3 = vmcs_readl(GUEST_CR3);
-       kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
+       kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
 
         /*
          * Use ept_save_pdptrs(vcpu) to load the MMU's cached PDPTRs

Yes, I agree. Jiangshan, please send all your fixes tomorrow, or tell me if you prefer that I push the reverts to kvm/next.

Paolo

#19 lore raw

Re: [PATCH 11/15] KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty

In reply to Sean Christopherson
Quoted 90 lines · Sean Christopherson
Quoted 59 lines · Lai Jiangshan
Quoted 22 lines · Maxim Levitsky
Quoted 21 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When vcpu->arch.cr3 is changed, it is marked dirty, so vmcs.GUEST_CR3 can be updated only when kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3).

---
  arch/x86/kvm/vmx/vmx.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +1 -2
index d94e51e9c08f..38b65b97fb7b 100644
@@ -3126,9 +3126,9 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
  		if (!enable_unrestricted_guest && !is_paging(vcpu))
  			guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr;
-		else if (test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
+		else if (kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3))
  			guest_cr3 = vcpu->arch.cr3;
-		else /* vmcs01.GUEST_CR3 is already up-to-date. */
Quoted 9 lines

arch/x86/kvm/vmx/vmx.c static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · line 3133

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +1 -0
+		else /* vmcs.GUEST_CR3 is already up-to-date. */
  			update_guest_cr3 = false;
  		vmx_ept_load_pdptrs(vcpu);
  	} else {

I just bisected this patch to break booting a VM with ept=1 but unrestricted_guest=0 (I needed to re-test unrestricted_guest=0 bug related to SMM, but didn't want to boot without EPT. With ept=0,the VM boots with this patch applied).

Thanks for reporting.

Sorry, I never tested it with unrestricted_guest=0. I can't reproduce it now shortly with unrestricted_guest=0. Maybe it can be reproduced easily if I try more guests or I write a piece of guest code to deliberate hit it if the following analyses is correct.

All the paths changing %cr3 are followed with kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3) and GUEST_CR3 will be expected to be updated.

What I missed is the case of "if (!enable_unrestricted_guest && !is_paging(vcpu))" in vmx_load_mmu_pgd() which doesn't load GUEST_CR3 but clears dirty of VCPU_EXREG_CR3 (when after next run).

So when CR0 !PG -> PG, VCPU_EXREG_CR3 dirty bit should be set.

Maybe adding the following patch on top of the original patch can work.

Thanks
Lai

arch/x86/kvm/x86.c +0 -0
index 85127b3e3690..55b45005ebb9 100644

arch/x86/kvm/x86.c void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon · line 861

Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
@@ -858,6 +858,7 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon
 	if ((cr0 ^ old_cr0) & X86_CR0_PG) {
 		kvm_clear_async_pf_completion_queue(vcpu);
 		kvm_async_pf_hash_reset(vcpu);
+		kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);

The better place for this is the path in vmx_set_cr0() that handles EPT + !URG. I believe nested_vmx_restore_host_state(), which is used to restore host state if hardware detects a VM-Fail that KVM misses on nested VM-Enter, is also missing a call to mark CR3 dirty.

arch/x86/kvm/vmx/nested.c +1 -1
index 2f6f465e575f..b0c3eb80f5d5 100644
@@ -4426,7 +4426,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)
 
        nested_ept_uninit_mmu_context(vcpu);
        vcpu->arch.cr3 = vmcs_readl(GUEST_CR3);
-       kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
+       kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
 
        /*
         * Use ept_save_pdptrs(vcpu) to load the MMU's cached PDPTRs
arch/x86/kvm/vmx/vmx.c +0 -0
index 640f4719612c..1701cee96678 100644
@@ -3066,8 +3066,10 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
                }

arch/x86/kvm/vmx/vmx.c void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) · line 3075

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +3 -1
 
                /* Note, vmx_set_cr4() consumes the new vcpu->arch.cr0. */
-               if ((old_cr0_pg ^ cr0) & X86_CR0_PG)
+               if ((old_cr0_pg ^ cr0) & X86_CR0_PG) {
                        vmx_set_cr4(vcpu, kvm_read_cr4(vcpu));
+                       kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
+               }
        }
 
        /* depends on vcpu->arch.cr0 to be set to a new value */

I tested this patch and my guests boot as well. I didn't test more stuff like migration or so, will do tomorrow.

Best regards,
Maxim Levitsky

#20 lore raw

Re: [PATCH 11/15] KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty

In reply to Lai Jiangshan
Quoted 61 lines · Lai Jiangshan
Quoted 17 lines · Maxim Levitsky
Quoted 16 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When vcpu->arch.cr3 is changed, it is marked dirty, so vmcs.GUEST_CR3 can be updated only when kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3).

---
  arch/x86/kvm/vmx/vmx.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index d94e51e9c08f..38b65b97fb7b 100644
@@ -3126,9 +3126,9 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 
Quoted 15 lines

arch/x86/kvm/vmx/vmx.c static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · line 3134

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +2 -2
  		if (!enable_unrestricted_guest && !is_paging(vcpu))
  			guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr;
-		else if (test_bit(VCPU_EXREG_CR3, (ulong *)&vcpu->arch.regs_avail))
+		else if (kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3))
  			guest_cr3 = vcpu->arch.cr3;
-		else /* vmcs01.GUEST_CR3 is already up-to-date. */
+		else /* vmcs.GUEST_CR3 is already up-to-date. */
  			update_guest_cr3 = false;
  		vmx_ept_load_pdptrs(vcpu);
  	} else {

I just bisected this patch to break booting a VM with ept=1 but unrestricted_guest=0 (I needed to re-test unrestricted_guest=0 bug related to SMM, but didn't want to boot without EPT. With ept=0,the VM boots with this patch applied).

Thanks for reporting.

Sorry, I never tested it with unrestricted_guest=0. I can't reproduce it now shortly with unrestricted_guest=0. Maybe it can be reproduced easily if I try more guests or I write a piece of guest code to deliberate hit it if the following analyses is correct.

All the paths changing %cr3 are followed with kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3) and GUEST_CR3 will be expected to be updated.

What I missed is the case of "if (!enable_unrestricted_guest && !is_paging(vcpu))" in vmx_load_mmu_pgd() which doesn't load GUEST_CR3 but clears dirty of VCPU_EXREG_CR3 (when after next run).

So when CR0 !PG -> PG, VCPU_EXREG_CR3 dirty bit should be set.

Maybe adding the following patch on top of the original patch can work.

Thanks
Lai

arch/x86/kvm/x86.c +0 -0
index 85127b3e3690..55b45005ebb9 100644

arch/x86/kvm/x86.c void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon · line 864

Quoted 7 lines
arch/x86/kvm/x86.c +1 -0
@@ -858,6 +858,7 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon
  	if ((cr0 ^ old_cr0) & X86_CR0_PG) {
  		kvm_clear_async_pf_completion_queue(vcpu);
  		kvm_async_pf_hash_reset(vcpu);
+		kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
  	}
 
  	if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS)

Tested this patch and my guests boot. I didn't test more stuff like migration or so, will do tomorrow.

Best regards,
Maxim Levitsky

#21 lore raw

PATCH 12/15 KVM: VMX: Reset the bits that are meaningful to be reset in vmx_register_cache_reset()

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

Add meaningful bits as VMX_REGS_AVAIL_SET and VMX_REGS_DIRTY_SET.

---
 arch/x86/kvm/vmx/vmx.h | 37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)
arch/x86/kvm/vmx/vmx.h +27 -10
index e7db42e3b0ce..465aa415c3cb 100644
@@ -437,16 +437,33 @@ BUILD_CONTROLS_SHADOW(secondary_exec, SECONDARY_VM_EXEC_CONTROL)
 
 static inline void vmx_register_cache_reset(struct kvm_vcpu *vcpu)
 {
-	vcpu->arch.regs_avail = ~((1 << VCPU_REGS_RIP) | (1 << VCPU_REGS_RSP)
-				  | (1 << VCPU_EXREG_RFLAGS)
-				  | (1 << VCPU_EXREG_PDPTR)
-				  | (1 << VCPU_EXREG_SEGMENTS)
-				  | (1 << VCPU_EXREG_CR0)
-				  | (1 << VCPU_EXREG_CR3)
-				  | (1 << VCPU_EXREG_CR4)
-				  | (1 << VCPU_EXREG_EXIT_INFO_1)
-				  | (1 << VCPU_EXREG_EXIT_INFO_2));
-	vcpu->arch.regs_dirty = 0;
+/*
+ * VMX_REGS_AVAIL_SET - The set of registers that will be updated in cache on
+ *			demand.  Other registers not listed here are synced to
+ *			the cache immediately after VM-Exit.
+ *
+ * VMX_REGS_DIRTY_SET - The set of registers that might be outdated in
+ *			architecture. Other registers not listed here are synced
+ *			to the architecture immediately when modifying.
+ */
+#define VMX_REGS_AVAIL_SET	((1 << VCPU_REGS_RIP) |\
+				(1 << VCPU_REGS_RSP) |\
+				(1 << VCPU_EXREG_RFLAGS) |\
+				(1 << VCPU_EXREG_PDPTR) |\
+				(1 << VCPU_EXREG_SEGMENTS) |\
+				(1 << VCPU_EXREG_CR0) |\
+				(1 << VCPU_EXREG_CR3) |\
+				(1 << VCPU_EXREG_CR4) |\
+				(1 << VCPU_EXREG_EXIT_INFO_1) |\
+				(1 << VCPU_EXREG_EXIT_INFO_2))
+
+#define VMX_REGS_DIRTY_SET	((1 << VCPU_REGS_RIP) |\
+				(1 << VCPU_REGS_RSP) |\
+				(1 << VCPU_EXREG_PDPTR) |\
+				(1 << VCPU_EXREG_CR3))
+
+	vcpu->arch.regs_avail &= ~VMX_REGS_AVAIL_SET;
+	vcpu->arch.regs_dirty &= ~VMX_REGS_DIRTY_SET;
 }
 
 static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm)
Signature · 1 line
2.19.1.6.gb485710b
maintainer
#22 lore raw

Re: [PATCH 12/15] KVM: VMX: Reset the bits that are meaningful to be reset in vmx_register_cache_reset()

Quoted 8 lines · Lai Jiangshan

+/*
+ * VMX_REGS_AVAIL_SET - The set of registers that will be updated in cache on
+ * demand. Other registers not listed here are synced to
+ * the cache immediately after VM-Exit.
+ *
+ * VMX_REGS_DIRTY_SET - The set of registers that might be outdated in
+ * architecture. Other registers not listed here are synced
+ * to the architecture immediately when modifying.

Slightly more expressive:

/*
* VMX_REGS_LAZY_LOAD_SET - The set of registers that will be updated in the
* cache on demand. Other registers not listed here are synced to
* the cache immediately after VM-Exit.
*/
...

/*
* VMX_REGS_LAZY_UPDATE_SET - The set of registers that might be outdated in
* VMCS. Other registers not listed here are synced to the VMCS
* immediately when modified.
*/
...

	BUILD_BUG_ON(VMX_REGS_LAZY_UPDATE_SET & ~VMX_REGS_LAZY_LOAD_SET);
        vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET;
        vcpu->arch.regs_dirty &= ~VMX_REGS_LAZY_UPDATE_SET;

That is lazily loaded registers become unavailable, and lazily updated registers become unavailable and dirty.

Paolo

#23 lore raw

PATCH 13/15 KVM: SVM: Add and use svm_register_cache_reset()

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

It resets all the appropriate bits like vmx.

---
 arch/x86/kvm/svm/svm.c |  3 +--
 arch/x86/kvm/svm/svm.h | 26 ++++++++++++++++++++++++++
 2 files changed, 27 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/svm.c +1 -2
index b7da66935e72..ba9cfddd2875 100644
@@ -3969,8 +3969,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
 
 	svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
 	vmcb_mark_all_clean(svm->vmcb);
-
-	kvm_register_clear_available(vcpu, VCPU_EXREG_PDPTR);
+	svm_register_cache_reset(vcpu);
 
 	/*
 	 * We need to handle MC intercepts here before the vcpu has a chance to
arch/x86/kvm/svm/svm.h +26 -0
index 0d7bbe548ac3..1cf5d5e2d0cd 100644
@@ -274,6 +274,32 @@ static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)
         return !test_bit(bit, (unsigned long *)&vmcb->control.clean);
 }
 
+static inline void svm_register_cache_reset(struct kvm_vcpu *vcpu)
+{
+/*
+ * SVM_REGS_AVAIL_SET - The set of registers that will be updated in cache on
+ *			demand.  Other registers not listed here are synced to
+ *			the cache immediately after VM-Exit.
+ *
+ * SVM_REGS_DIRTY_SET - The set of registers that might be outdated in
+ *			architecture. Other registers not listed here are synced
+ *			to the architecture immediately when modifying.
+ *
+ *			Special case: VCPU_EXREG_CR3 should be in this set due
+ *			to the fact.  But KVM_REQ_LOAD_MMU_PGD is always
+ *			requested when the cache vcpu->arch.cr3 is changed and
+ *			svm_load_mmu_pgd() always syncs the new CR3 value into
+ *			the architecture.  So the dirty information of
+ *			VCPU_EXREG_CR3 is not used which means VCPU_EXREG_CR3
+ *			isn't required to be put in this set.
+ */
+#define SVM_REGS_AVAIL_SET	(1 << VCPU_EXREG_PDPTR)
+#define SVM_REGS_DIRTY_SET	(0)
+
+	vcpu->arch.regs_avail &= ~SVM_REGS_AVAIL_SET;
+	vcpu->arch.regs_dirty &= ~SVM_REGS_DIRTY_SET;
+}
+
 static inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
 {
 	return container_of(vcpu, struct vcpu_svm, vcpu);
Signature · 1 line
2.19.1.6.gb485710b
maintainer
#24 lore raw

Re: [PATCH 13/15] KVM: SVM: Add and use svm_register_cache_reset()

Quoted 47 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

It resets all the appropriate bits like vmx.

---
  arch/x86/kvm/svm/svm.c |  3 +--
  arch/x86/kvm/svm/svm.h | 26 ++++++++++++++++++++++++++
  2 files changed, 27 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/svm.c +1 -2
index b7da66935e72..ba9cfddd2875 100644
@@ -3969,8 +3969,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
 
  	svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
  	vmcb_mark_all_clean(svm->vmcb);
-
-	kvm_register_clear_available(vcpu, VCPU_EXREG_PDPTR);
+	svm_register_cache_reset(vcpu);
 
  	/*
  	 * We need to handle MC intercepts here before the vcpu has a chance to
arch/x86/kvm/svm/svm.h +15 -0
index 0d7bbe548ac3..1cf5d5e2d0cd 100644
@@ -274,6 +274,32 @@ static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)
          return !test_bit(bit, (unsigned long *)&vmcb->control.clean);
  }
 
+static inline void svm_register_cache_reset(struct kvm_vcpu *vcpu)
+{
+/*
+ * SVM_REGS_AVAIL_SET - The set of registers that will be updated in cache on
+ *			demand.  Other registers not listed here are synced to
+ *			the cache immediately after VM-Exit.
+ *
+ * SVM_REGS_DIRTY_SET - The set of registers that might be outdated in
+ *			architecture. Other registers not listed here are synced
+ *			to the architecture immediately when modifying.
+ *
+ *			Special case: VCPU_EXREG_CR3 should be in this set due
+ *			to the fact.  But KVM_REQ_LOAD_MMU_PGD is always
+ *			requested when the cache vcpu->arch.cr3 is changed and
+ *			svm_load_mmu_pgd() always syncs the new CR3 value into

arch/x86/kvm/svm/svm.h static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit) · line 301

Quoted 10 lines
arch/x86/kvm/svm/svm.h +10 -0
+ *			the architecture.  So the dirty information of
+ *			VCPU_EXREG_CR3 is not used which means VCPU_EXREG_CR3
+ *			isn't required to be put in this set.
+ */
+#define SVM_REGS_AVAIL_SET	(1 << VCPU_EXREG_PDPTR)
+#define SVM_REGS_DIRTY_SET	(0)
+
+	vcpu->arch.regs_avail &= ~SVM_REGS_AVAIL_SET;
+	vcpu->arch.regs_dirty &= ~SVM_REGS_DIRTY_SET;
+}

I think touching regs_dirty is confusing here, so I'd go with this:

vcpu->arch.regs_avail &= ~SVM_REGS_LAZY_LOAD_SET;

/*
* SVM does not use vcpu->arch.regs_dirty. The only register that
* might be out of date in the VMCB is CR3, but KVM_REQ_LOAD_MMU_PGD
* is always requested when the cache vcpu->arch.cr3 is changed and
* svm_load_mmu_pgd() always syncs the new CR3 value into the VMCB. */

(VMX instead needs VCPU_EXREG_CR3 mostly because it does not want to update it unconditionally on exit).

Paolo

#25 lore raw

Re: [PATCH 13/15] KVM: SVM: Add and use svm_register_cache_reset()

Quoted 49 lines · Paolo Bonzini
Quoted 48 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

It resets all the appropriate bits like vmx.

---
  arch/x86/kvm/svm/svm.c |  3 +-- arch/x86/kvm/svm/svm.h | 26 ++++++++++++++++++++++++++

  2 files changed, 27 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/svm.c +1 -2
index b7da66935e72..ba9cfddd2875 100644
@@ -3969,8 +3969,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
       svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
       vmcb_mark_all_clean(svm->vmcb);
-
-    kvm_register_clear_available(vcpu, VCPU_EXREG_PDPTR);
+    svm_register_cache_reset(vcpu);
       /*
        * We need to handle MC intercepts here before the vcpu has a chance to
arch/x86/kvm/svm/svm.h +19 -0
index 0d7bbe548ac3..1cf5d5e2d0cd 100644
@@ -274,6 +274,32 @@ static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)
           return !test_bit(bit, (unsigned long *)&vmcb->control.clean);
   }
+static inline void svm_register_cache_reset(struct kvm_vcpu *vcpu)
+{
+/*
+ * SVM_REGS_AVAIL_SET - The set of registers that will be updated in cache on
+ *            demand.  Other registers not listed here are synced to
+ *            the cache immediately after VM-Exit.
+ *
+ * SVM_REGS_DIRTY_SET - The set of registers that might be outdated in
+ *            architecture. Other registers not listed here are synced
+ *            to the architecture immediately when modifying.
+ *
+ *            Special case: VCPU_EXREG_CR3 should be in this set due
+ *            to the fact.  But KVM_REQ_LOAD_MMU_PGD is always
+ *            requested when the cache vcpu->arch.cr3 is changed and
+ *            svm_load_mmu_pgd() always syncs the new CR3 value into
+ *            the architecture.  So the dirty information of
+ *            VCPU_EXREG_CR3 is not used which means VCPU_EXREG_CR3
+ *            isn't required to be put in this set.
+ */
Quoted 8 lines

arch/x86/kvm/svm/svm.h static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit) · line 301

Quoted 6 lines
arch/x86/kvm/svm/svm.h +6 -0
+#define SVM_REGS_AVAIL_SET    (1 << VCPU_EXREG_PDPTR)
+#define SVM_REGS_DIRTY_SET    (0)
+
+    vcpu->arch.regs_avail &= ~SVM_REGS_AVAIL_SET;
+    vcpu->arch.regs_dirty &= ~SVM_REGS_DIRTY_SET;
+}

I think touching regs_dirty is confusing here, so I'd go with this:

It makes the code the same as vmx by clearing all the SVM_REGS_DIRTY_SET bits. And the compiler will remove this line of code since SVM_REGS_DIRTY_SET is (0), and regs_dirty is not touched.

Using VMX_REGS_DIRTY_SET and SVM_REGS_DIRTY_SET and making the code similar is my intent for patch12,13. If it causes confusing, I would like to make a second thought. SVM_REGS_DIRTY_SET does be special in svm where VCPU_EXREG_CR3 is in it by definition, but it is not added into SVM_REGS_DIRTY_SET in the patch just for optimization to allow the compiler optimizes the line of code out.

I'm Ok to not queue patch12,13,14.

Quoted 14 lines

        vcpu->arch.regs_avail &= ~SVM_REGS_LAZY_LOAD_SET;

        /*
         * SVM does not use vcpu->arch.regs_dirty.  The only register that
         * might be out of date in the VMCB is CR3, but KVM_REQ_LOAD_MMU_PGD
         * is always requested when the cache vcpu->arch.cr3 is changed and
         * svm_load_mmu_pgd() always syncs the new CR3 value into the VMCB. */

(VMX instead needs VCPU_EXREG_CR3 mostly because it does not want to update it unconditionally on exit).

Paolo

maintainer
#26 lore raw

Re: [PATCH 13/15] KVM: SVM: Add and use svm_register_cache_reset()

Quoted 6 lines · Lai Jiangshan

Using VMX_REGS_DIRTY_SET and SVM_REGS_DIRTY_SET and making the code similar is my intent for patch12,13.  If it causes confusing, I would like to make a second thought.  SVM_REGS_DIRTY_SET does be special in svm where VCPU_EXREG_CR3 is in it by definition, but it is not added into SVM_REGS_DIRTY_SET in the patch just for optimization to allow the compiler optimizes the line of code out.

I think this is where we disagree. In my opinion it is enough to document that CR3 _can_ be out of date, but it doesn't have to be marked dirty because its dirty bit is effectively KVM_REQ_LOAD_MMU_PGD.

For VMX, it is important to clear VCPU_EXREG_CR3 because the combination "avail=0, dirty=1" is nonsensical:

	av d
	0  0    in VMCS
	0  1    *INVALID*
	1  0	in vcpu->arch
	1  1	in vcpu->arch, needs store

But on SVM, VCPU_EXREG_CR3 is always available.

Thinking more about it, it makes more sense for VMX to reset _all_ bits of dirty to 0, just like it was before your change, but doing so even earlier in vmx_vcpu_run.

I appreciate that VMX_REGS_LAZY_UPDATE_SET is useful for documentation, but it's also important that the values in avail/dirty make sense as a pair.

So here is what I would do:

arch/x86/kvm/kvm_cache_regs.h +7 -0
index 6e6d0d01f18d..ac3d3bd662f4 100644
@@ -43,6 +43,13 @@ BUILD_KVM_GPR_ACCESSORS(r14, R14)
 BUILD_KVM_GPR_ACCESSORS(r15, R15)
 #endif
 
+/*
+ * avail  dirty
+ * 0	  0	  register in VMCS/VMCB
+ * 0	  1	  *INVALID*
+ * 1	  0	  register in vcpu->arch
+ * 1	  1	  register in vcpu->arch, needs to be stored back
+ */
 static inline bool kvm_register_is_available(struct kvm_vcpu *vcpu,
 					     enum kvm_reg reg)
 {
arch/x86/kvm/vmx/vmx.c +2 -1
index 6fce61fc98e3..72ae67e214b5 100644
@@ -6635,6 +6635,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
 		vmcs_writel(GUEST_RSP, vcpu->arch.regs[VCPU_REGS_RSP]);
 	if (kvm_register_is_dirty(vcpu, VCPU_REGS_RIP))
 		vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]);
+	vcpu->arch.regs_dirty = 0;
 
 	cr3 = __get_current_cr3_fast();
 	if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) {
@@ -6729,7 +6730,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
 	loadsegment(es, __USER_DS);
 #endif
 
-	vmx_register_cache_reset(vcpu);
+	vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET;
 
 	pt_guest_exit(vmx);
 
arch/x86/kvm/vmx/vmx.h +15 -13
index 4df2ac24ffc1..f978699480e3 100644
@@ -473,19 +473,21 @@ BUILD_CONTROLS_SHADOW(pin, PIN_BASED_VM_EXEC_CONTROL)
 BUILD_CONTROLS_SHADOW(exec, CPU_BASED_VM_EXEC_CONTROL)
 BUILD_CONTROLS_SHADOW(secondary_exec, SECONDARY_VM_EXEC_CONTROL)
 
-static inline void vmx_register_cache_reset(struct kvm_vcpu *vcpu)
-{
-	vcpu->arch.regs_avail = ~((1 << VCPU_REGS_RIP) | (1 << VCPU_REGS_RSP)
-				  | (1 << VCPU_EXREG_RFLAGS)
-				  | (1 << VCPU_EXREG_PDPTR)
-				  | (1 << VCPU_EXREG_SEGMENTS)
-				  | (1 << VCPU_EXREG_CR0)
-				  | (1 << VCPU_EXREG_CR3)
-				  | (1 << VCPU_EXREG_CR4)
-				  | (1 << VCPU_EXREG_EXIT_INFO_1)
-				  | (1 << VCPU_EXREG_EXIT_INFO_2));
-	vcpu->arch.regs_dirty = 0;
-}
+/*
+ * VMX_REGS_LAZY_LOAD_SET - The set of registers that will be updated in the
+ * cache on demand.  Other registers not listed here are synced to
+ * the cache immediately after VM-Exit.
+ */
+#define VMX_REGS_LAZY_LOAD_SET	((1 << VCPU_REGS_RIP) |         \
+				(1 << VCPU_REGS_RSP) |          \
+				(1 << VCPU_EXREG_RFLAGS) |      \
+				(1 << VCPU_EXREG_PDPTR) |       \
+				(1 << VCPU_EXREG_SEGMENTS) |    \
+				(1 << VCPU_EXREG_CR0) |         \
+				(1 << VCPU_EXREG_CR3) |         \
+				(1 << VCPU_EXREG_CR4) |         \
+				(1 << VCPU_EXREG_EXIT_INFO_1) | \
+				(1 << VCPU_EXREG_EXIT_INFO_2))
 
 static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm)
 {

and likewise for SVM:

arch/x86/kvm/svm/svm.c +2 -1
index eb2a2609cae8..4b22aa7d55d0 100644
@@ -3944,6 +3944,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
 		vcpu->arch.regs[VCPU_REGS_RSP] = svm->vmcb->save.rsp;
 		vcpu->arch.regs[VCPU_REGS_RIP] = svm->vmcb->save.rip;
 	}
+	vcpu->arch.regs_dirty = 0;
 
 	if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
 		kvm_before_interrupt(vcpu);
@@ -3978,7 +3978,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
 		vcpu->arch.apf.host_apf_flags =
 			kvm_read_and_reset_apf_flags();
 
-	kvm_register_clear_available(vcpu, VCPU_EXREG_PDPTR);
+	vcpu->arch.regs_avail &= ~SVM_REGS_LAZY_LOAD_SET;
 
 	/*
 	 * We need to handle MC intercepts here before the vcpu has a chance to
arch/x86/kvm/svm/svm.h +10 -0
index 32769d227860..b3c3c3098216 100644
@@ -321,6 +321,16 @@ static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)
         return !test_bit(bit, (unsigned long *)&vmcb->control.clean);
 }
 
+/*
+ * Only the PDPTRs are loaded on demand into the shadow MMU.  All other
+ * fields are synchronized in handle_exit, because accessing the VMCB is cheap.
+ *
+ * CR3 might be out of date in the VMCB but it is not marked dirty; instead,
+ * KVM_REQ_LOAD_MMU_PGD is always requested when the cached vcpu->arch.cr3
+ * is changed.  svm_load_mmu_pgd() then syncs the new CR3 value into the VMCB.
+ */
+#define SVM_REGS_LAZY_LOAD_SET	(1 << VCPU_EXREG_PDPTR)
+
 static inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
 {
 	return container_of(vcpu, struct vcpu_svm, vcpu);
Reviewed-by
#27 lore raw

Re: [PATCH 13/15] KVM: SVM: Add and use svm_register_cache_reset()

Quoted 27 lines · Paolo Bonzini
Quoted 6 lines · Lai Jiangshan

Using VMX_REGS_DIRTY_SET and SVM_REGS_DIRTY_SET and making the code similar is my intent for patch12,13.  If it causes confusing, I would like to make a second thought.  SVM_REGS_DIRTY_SET does be special in svm where VCPU_EXREG_CR3 is in it by definition, but it is not added into SVM_REGS_DIRTY_SET in the patch just for optimization to allow the compiler optimizes the line of code out.

I think this is where we disagree.  In my opinion it is enough to document that CR3 _can_ be out of date, but it doesn't have to be marked dirty because its dirty bit is effectively KVM_REQ_LOAD_MMU_PGD.

For VMX, it is important to clear VCPU_EXREG_CR3 because the combination "avail=0, dirty=1" is nonsensical:

    av d
    0  0    in VMCS
    0  1    *INVALID*
    1  0    in vcpu->arch
    1  1    in vcpu->arch, needs store

But on SVM, VCPU_EXREG_CR3 is always available.

Thinking more about it, it makes more sense for VMX to reset _all_ bits of dirty to 0, just like it was before your change, but doing so even earlier in vmx_vcpu_run.

Quoted 5 lines

I appreciate that VMX_REGS_LAZY_UPDATE_SET is useful for documentation, but it's also important that the values in avail/dirty make sense as a pair.

So here is what I would do:

Quoted 125 lines
arch/x86/kvm/kvm_cache_regs.h +7 -0
index 6e6d0d01f18d..ac3d3bd662f4 100644
@@ -43,6 +43,13 @@ BUILD_KVM_GPR_ACCESSORS(r14, R14)
  BUILD_KVM_GPR_ACCESSORS(r15, R15)
  #endif
 
+/*
+ * avail  dirty
+ * 0      0      register in VMCS/VMCB
+ * 0      1      *INVALID*
+ * 1      0      register in vcpu->arch
+ * 1      1      register in vcpu->arch, needs to be stored back
+ */
  static inline bool kvm_register_is_available(struct kvm_vcpu *vcpu,
                           enum kvm_reg reg)
  {
arch/x86/kvm/vmx/vmx.c +2 -1
index 6fce61fc98e3..72ae67e214b5 100644
@@ -6635,6 +6635,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
          vmcs_writel(GUEST_RSP, vcpu->arch.regs[VCPU_REGS_RSP]);
      if (kvm_register_is_dirty(vcpu, VCPU_REGS_RIP))
          vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]);
+    vcpu->arch.regs_dirty = 0;
 
      cr3 = __get_current_cr3_fast();
      if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) {
@@ -6729,7 +6730,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
      loadsegment(es, __USER_DS);
  #endif
 
-    vmx_register_cache_reset(vcpu);
+    vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET;
 
      pt_guest_exit(vmx);
 
arch/x86/kvm/vmx/vmx.h +15 -13
index 4df2ac24ffc1..f978699480e3 100644
@@ -473,19 +473,21 @@ BUILD_CONTROLS_SHADOW(pin, PIN_BASED_VM_EXEC_CONTROL)
  BUILD_CONTROLS_SHADOW(exec, CPU_BASED_VM_EXEC_CONTROL)
  BUILD_CONTROLS_SHADOW(secondary_exec, SECONDARY_VM_EXEC_CONTROL)
 
-static inline void vmx_register_cache_reset(struct kvm_vcpu *vcpu)
-{
-    vcpu->arch.regs_avail = ~((1 << VCPU_REGS_RIP) | (1 << VCPU_REGS_RSP)
-                  | (1 << VCPU_EXREG_RFLAGS)
-                  | (1 << VCPU_EXREG_PDPTR)
-                  | (1 << VCPU_EXREG_SEGMENTS)
-                  | (1 << VCPU_EXREG_CR0)
-                  | (1 << VCPU_EXREG_CR3)
-                  | (1 << VCPU_EXREG_CR4)
-                  | (1 << VCPU_EXREG_EXIT_INFO_1)
-                  | (1 << VCPU_EXREG_EXIT_INFO_2));
-    vcpu->arch.regs_dirty = 0;
-}
+/*
+ * VMX_REGS_LAZY_LOAD_SET - The set of registers that will be updated in the
+ * cache on demand.  Other registers not listed here are synced to
+ * the cache immediately after VM-Exit.
+ */
+#define VMX_REGS_LAZY_LOAD_SET    ((1 << VCPU_REGS_RIP) |         \
+                (1 << VCPU_REGS_RSP) |          \
+                (1 << VCPU_EXREG_RFLAGS) |      \
+                (1 << VCPU_EXREG_PDPTR) |       \
+                (1 << VCPU_EXREG_SEGMENTS) |    \
+                (1 << VCPU_EXREG_CR0) |         \
+                (1 << VCPU_EXREG_CR3) |         \
+                (1 << VCPU_EXREG_CR4) |         \
+                (1 << VCPU_EXREG_EXIT_INFO_1) | \
+                (1 << VCPU_EXREG_EXIT_INFO_2))
 
  static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm)
  {

and likewise for SVM:

arch/x86/kvm/svm/svm.c +2 -1
index eb2a2609cae8..4b22aa7d55d0 100644
@@ -3944,6 +3944,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
          vcpu->arch.regs[VCPU_REGS_RSP] = svm->vmcb->save.rsp;
          vcpu->arch.regs[VCPU_REGS_RIP] = svm->vmcb->save.rip;
      }
+    vcpu->arch.regs_dirty = 0;
 
      if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
          kvm_before_interrupt(vcpu);
@@ -3978,7 +3978,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu)
          vcpu->arch.apf.host_apf_flags =
              kvm_read_and_reset_apf_flags();
 
-    kvm_register_clear_available(vcpu, VCPU_EXREG_PDPTR);
+    vcpu->arch.regs_avail &= ~SVM_REGS_LAZY_LOAD_SET;
 
      /*
       * We need to handle MC intercepts here before the vcpu has a chance to
arch/x86/kvm/svm/svm.h +10 -0
index 32769d227860..b3c3c3098216 100644
@@ -321,6 +321,16 @@ static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit)
          return !test_bit(bit, (unsigned long *)&vmcb->control.clean);
  }
 
+/*
+ * Only the PDPTRs are loaded on demand into the shadow MMU.  All other
+ * fields are synchronized in handle_exit, because accessing the VMCB is cheap.
+ *
+ * CR3 might be out of date in the VMCB but it is not marked dirty; instead,
+ * KVM_REQ_LOAD_MMU_PGD is always requested when the cached vcpu->arch.cr3
+ * is changed.  svm_load_mmu_pgd() then syncs the new CR3 value into the VMCB.
+ */
+#define SVM_REGS_LAZY_LOAD_SET    (1 << VCPU_EXREG_PDPTR)
+
  static inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
  {
      return container_of(vcpu, struct vcpu_svm, vcpu);
#28 lore raw

PATCH 14/15 KVM: X86: Remove kvm_register_clear_available()

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

It has no user.

---
 arch/x86/kvm/kvm_cache_regs.h | 7 -------
 1 file changed, 7 deletions(-)
arch/x86/kvm/kvm_cache_regs.h +0 -7
index 54a996adb18d..0f8847b981e5 100644
@@ -61,13 +61,6 @@ static inline void kvm_register_mark_available(struct kvm_vcpu *vcpu,
 	__set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail);
 }
 
-static inline void kvm_register_clear_available(struct kvm_vcpu *vcpu,
-					       enum kvm_reg reg)
-{
-	__clear_bit(reg, (unsigned long *)&vcpu->arch.regs_avail);
-	__clear_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty);
-}
-
 static inline void kvm_register_mark_dirty(struct kvm_vcpu *vcpu,
 					   enum kvm_reg reg)
 {
Signature · 1 line
2.19.1.6.gb485710b
#29 lore raw

PATCH 15/15 KVM: nVMX: Always write vmcs.GUEST_CR3 during nested VM-Exit

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

For VM-Enter, vmcs.GUEST_CR3 and vcpu->arch.cr3 are synced and it is better to mark VCPU_EXREG_CR3 available rather than dirty to reduce a redundant vmwrite(GUEST_CR3) in vmx_load_mmu_pgd().

But nested_vmx_load_cr3() is also served for VM-Exit which doesn't set vmcs.GUEST_CR3.

This patch moves writing to vmcs.GUEST_CR3 into nested_vmx_load_cr3() for both nested VM-Eneter/Exit and use kvm_register_mark_available().

This patch doesn't cause any extra writing to vmcs.GUEST_CR3 and if userspace is modifying CR3 with KVM_SET_SREGS later, the dirty info for VCPU_EXREG_CR3 would be set for next writing to vmcs.GUEST_CR3 and no update will be lost.

---
 arch/x86/kvm/vmx/nested.c | 32 +++++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 11 deletions(-)
arch/x86/kvm/vmx/nested.c +21 -11
index ee5a68c2ea3a..4ddd4b1b0503 100644
@@ -1133,8 +1133,28 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 	if (!nested_ept)
 		kvm_mmu_new_pgd(vcpu, cr3);
 
+	/*
+	 * Immediately write vmcs.GUEST_CR3 when changing vcpu->arch.cr3.
+	 *
+	 * VCPU_EXREG_CR3 is marked available rather than dirty because
+	 * vcpu->arch.cr3 and vmcs.GUEST_CR3 are synced when enable_ept and
+	 * vmcs.GUEST_CR3 is irrelevant to vcpu->arch.cr3 when !enable_ept.
+	 *
+	 * For VM-Enter case, it will be propagated to vmcs12 on nested
+	 * VM-Exit, which can occur without actually running L2 and thus
+	 * without hitting vmx_load_mmu_pgd(), e.g. if L1 is entering L2 with
+	 * vmcs12.GUEST_ACTIVITYSTATE=HLT, in which case KVM will intercept
+	 * the transition to HLT instead of running L2.
+	 *
+	 * For VM-Exit case, it is likely that vmcs.GUEST_CR3 == cr3 here, but
+	 * L1 may set HOST_CR3 to a value other than its CR3 before VM-Entry,
+	 * so we just update it unconditionally.
+	 */
+	if (enable_ept)
+		vmcs_writel(GUEST_CR3, cr3);
+
 	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
+	kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
 
 	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
 	kvm_init_mmu(vcpu);
@@ -2600,16 +2620,6 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,
 				from_vmentry, entry_failure_code))
 		return -EINVAL;
 
-	/*
-	 * Immediately write vmcs02.GUEST_CR3.  It will be propagated to vmcs12
-	 * on nested VM-Exit, which can occur without actually running L2 and
-	 * thus without hitting vmx_load_mmu_pgd(), e.g. if L1 is entering L2 with
-	 * vmcs12.GUEST_ACTIVITYSTATE=HLT, in which case KVM will intercept the
-	 * transition to HLT instead of running L2.
-	 */
-	if (enable_ept)
-		vmcs_writel(GUEST_CR3, vmcs12->guest_cr3);
-
 	/* Late preparation of GUEST_PDPTRs now that EFER and CRs are set. */
 	if (load_guest_pdptrs_vmcs12 && nested_cpu_has_ept(vmcs12) &&
 	    is_pae_paging(vcpu)) {
Signature · 1 line
2.19.1.6.gb485710b
maintainer
#30 lore raw

Re: [PATCH 15/15] KVM: nVMX: Always write vmcs.GUEST_CR3 during nested VM-Exit

Quoted 64 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

For VM-Enter, vmcs.GUEST_CR3 and vcpu->arch.cr3 are synced and it is better to mark VCPU_EXREG_CR3 available rather than dirty to reduce a redundant vmwrite(GUEST_CR3) in vmx_load_mmu_pgd().

But nested_vmx_load_cr3() is also served for VM-Exit which doesn't set vmcs.GUEST_CR3.

This patch moves writing to vmcs.GUEST_CR3 into nested_vmx_load_cr3() for both nested VM-Eneter/Exit and use kvm_register_mark_available().

This patch doesn't cause any extra writing to vmcs.GUEST_CR3 and if userspace is modifying CR3 with KVM_SET_SREGS later, the dirty info for VCPU_EXREG_CR3 would be set for next writing to vmcs.GUEST_CR3 and no update will be lost.

---
  arch/x86/kvm/vmx/nested.c | 32 +++++++++++++++++++++-----------
  1 file changed, 21 insertions(+), 11 deletions(-)
arch/x86/kvm/vmx/nested.c +21 -4
index ee5a68c2ea3a..4ddd4b1b0503 100644
@@ -1133,8 +1133,28 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
  	if (!nested_ept)
  		kvm_mmu_new_pgd(vcpu, cr3);
 
+	/*
+	 * Immediately write vmcs.GUEST_CR3 when changing vcpu->arch.cr3.
+	 *
+	 * VCPU_EXREG_CR3 is marked available rather than dirty because
+	 * vcpu->arch.cr3 and vmcs.GUEST_CR3 are synced when enable_ept and
+	 * vmcs.GUEST_CR3 is irrelevant to vcpu->arch.cr3 when !enable_ept.
+	 *
+	 * For VM-Enter case, it will be propagated to vmcs12 on nested
+	 * VM-Exit, which can occur without actually running L2 and thus
+	 * without hitting vmx_load_mmu_pgd(), e.g. if L1 is entering L2 with
+	 * vmcs12.GUEST_ACTIVITYSTATE=HLT, in which case KVM will intercept
+	 * the transition to HLT instead of running L2.
+	 *
+	 * For VM-Exit case, it is likely that vmcs.GUEST_CR3 == cr3 here, but
+	 * L1 may set HOST_CR3 to a value other than its CR3 before VM-Entry,
+	 * so we just update it unconditionally.
+	 */
+	if (enable_ept)
+		vmcs_writel(GUEST_CR3, cr3);
+
  	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
+	kvm_register_mark_available(vcpu, VCPU_EXREG_CR3);
 
  	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
  	kvm_init_mmu(vcpu);
@@ -2600,16 +2620,6 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,
  				from_vmentry, entry_failure_code))
  		return -EINVAL;
 
-	/*
-	 * Immediately write vmcs02.GUEST_CR3.  It will be propagated to vmcs12
-	 * on nested VM-Exit, which can occur without actually running L2 and

arch/x86/kvm/vmx/nested.c static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, · line 2625

Quoted 10 lines
arch/x86/kvm/vmx/nested.c +0 -7
-	 * thus without hitting vmx_load_mmu_pgd(), e.g. if L1 is entering L2 with
-	 * vmcs12.GUEST_ACTIVITYSTATE=HLT, in which case KVM will intercept the
-	 * transition to HLT instead of running L2.
-	 */
-	if (enable_ept)
-		vmcs_writel(GUEST_CR3, vmcs12->guest_cr3);
-
  	/* Late preparation of GUEST_PDPTRs now that EFER and CRs are set. */
  	if (load_guest_pdptrs_vmcs12 && nested_cpu_has_ept(vmcs12) &&
  	    is_pae_paging(vcpu)) {

I have to think more about this one and patch 17. I queued the rest.

Paolo

#31 lore raw

[PATCH 16/15] KVM: X86: Update mmu->pdptrs only when it is changed

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

It is unchanged in most cases.

---
 arch/x86/kvm/x86.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
arch/x86/kvm/x86.c +7 -4
index 6ca19cac4aff..0176eaa86a35 100644
@@ -828,10 +828,13 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
 		}
 	}
 
-	memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
-	kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-	/* Ensure the dirty PDPTEs to be loaded. */
-	kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+	kvm_register_mark_available(vcpu, VCPU_EXREG_PDPTR);
+	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
+		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
+		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
+		/* Ensure the dirty PDPTEs to be loaded. */
+		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+	}
 	vcpu->arch.pdptrs_from_userspace = false;
 
 	return 1;
Signature · 1 line
2.19.1.6.gb485710b
#32 lore raw

Re: [PATCH 16/15] KVM: X86: Update mmu->pdptrs only when it is changed

Quoted 18 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

It is unchanged in most cases.

---
 arch/x86/kvm/x86.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
arch/x86/kvm/x86.c +0 -1
index 6ca19cac4aff..0176eaa86a35 100644
@@ -828,10 +828,13 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
 		}
 	}
 
-	memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 837

Quoted 10 lines
arch/x86/kvm/x86.c +7 -3
-	kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-	/* Ensure the dirty PDPTEs to be loaded. */
-	kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+	kvm_register_mark_available(vcpu, VCPU_EXREG_PDPTR);
+	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
+		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
+		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
+		/* Ensure the dirty PDPTEs to be loaded. */
+		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+	}

Can this be unqueued until there's sufficient justification that (a) this is correct and (b) actually provides a meaningful performance optimization? There have been far too many PDPTR caching bugs to make this change without an analysis of why it's safe, e.g. what guarantees the that PDPTRs in the VMCS are sync'd with mmu->pdptrs? I'm not saying they aren't, I just want the changelog to prove that they are.

The next patch does add a fairly heavy unload of the current root for !TDP, but that's a bug fix and should be ordered before any optimizations anyways.

Quoted 6 lines
arch/x86/kvm/x86.c +0 -0
 	vcpu->arch.pdptrs_from_userspace = false;
 
 	return 1;
Signature · 1 line
2.19.1.6.gb485710b
#33 lore raw

Re: [PATCH 16/15] KVM: X86: Update mmu->pdptrs only when it is changed

Quoted 28 lines · Sean Christopherson
Quoted 27 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

It is unchanged in most cases.

---
  arch/x86/kvm/x86.c | 11 +++++++----
  1 file changed, 7 insertions(+), 4 deletions(-)
arch/x86/kvm/x86.c +6 -4
index 6ca19cac4aff..0176eaa86a35 100644
@@ -828,10 +828,13 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
  		}
  	}
 
-	memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
-	kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-	/* Ensure the dirty PDPTEs to be loaded. */
-	kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+	kvm_register_mark_available(vcpu, VCPU_EXREG_PDPTR);
+	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
+		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
+		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
+		/* Ensure the dirty PDPTEs to be loaded. */
+		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
Quoted 8 lines

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 837

Quoted 1 line
arch/x86/kvm/x86.c +1 -0
+	}

Can this be unqueued until there's sufficient justification that (a) this is correct and (b) actually provides a meaningful performance optimization? There have been far too many PDPTR caching bugs to make this change without an analysis of why it's safe, e.g. what guarantees the that PDPTRs in the VMCS are sync'd with mmu->pdptrs? I'm not saying they aren't, I just want the changelog to prove that they are.

I have zero intent to improve performance for 32bit guests.

For correctness, the patch needs to be revaluated, I agree it to be unqueued.

Quoted 3 lines

The next patch does add a fairly heavy unload of the current root for !TDP, but that's a bug fix and should be ordered before any optimizations anyways.

I don't have strong option on the patch ordering and I have a preference which is already applied to other patchset.

This patch is a preparation patch for next patch. It has very limited or even negative value without next patch and it was not in the original 15-patch patchset until I found the defect fixed by the next patch that I appended both as "16/15" and "17/15".

But kvm has many small defects for so many years, I has fixed several in this circle and there are some pending.

And the kvm works for so many years even with these small defects and they only hit when the guest deliberately do something, and even the guest does these things, it only reveals that the kvm doesn't fully conform the SDM, it can't hurt host any degree.

For important fix or for bug that can hurt host, I would definitely make the bug fix first and other patches are ordered later.

For defect like this, I prefer "clean" patches policy: several cleanup or preparation patches first to give the code a better basic and then fix the defects.

It is OK for me if fixes like this (normal guest doesn't hit it and it can't hurt host) go into or doesn't go into the stable tree.

For example, I used my patch ordering policy in "permission_fault() for SMAP" patchset where the fix went in patch3 which fixes implicit supervisor access when the CPL < 3. For next spin, I would be a new preparation patch first to add "implicit access" in pfec. The fix itself can't go as the first patch.

This is a reply not specified to this patch. And for this patch and next patch I will take your suggestion or another possible approach.

Thanks
Lai

maintainer
#34 lore raw

Re: [PATCH 16/15] KVM: X86: Update mmu->pdptrs only when it is changed

In reply to Sean Christopherson
Quoted 3 lines · Sean Christopherson

what guarantees the that PDPTRs in the VMCS are sync'd with mmu->pdptrs? I'm not saying they aren't, I just want the changelog to prove that they are.

If they aren't synced you should *already* have dirty VCPU_EXREG_PDPTR and pending KVM_REQ_LOAD_MMU_PGD, shouldn't you? As long as the caching invariants are respected, this patch is fairly safe, and if they aren't there are plenty of preexisting bugs anyway.

Paolo

Quoted 3 lines

The next patch does add a fairly heavy unload of the current root for !TDP, but that's a bug fix and should be ordered before any optimizations anyways.

#35 lore raw

Re: [PATCH 16/15] KVM: X86: Update mmu->pdptrs only when it is changed

Quoted 7 lines · Paolo Bonzini
Quoted 3 lines · Sean Christopherson

what guarantees the that PDPTRs in the VMCS are sync'd with mmu->pdptrs?  I'm not saying they aren't, I just want the changelog to prove that they are.

If they aren't synced you should *already* have dirty VCPU_EXREG_PDPTR and pending KVM_REQ_LOAD_MMU_PGD, shouldn't you? As long as the caching invariants are respected, this patch is fairly safe, and if they aren't there are plenty of preexisting bugs anyway.

They can be not synced in other side: not available.

If (!kvm_register_is_available(vcpu, VCPU_EXREG_PDPTR)) it will make no sense to compare mmu->pdptrs when EPT is enabled.

Because vmcs might have different copy, it is better to just mark it dirty in load_pdptrs().

(SVM is OK even with NPT enabled, since vmcb doesn't have a copy)

I haven't investigated enough then and today. It is quit complicated.

Thanks
Lai

Quoted 4 lines
Quoted 4 lines

The next patch does add a fairly heavy unload of the current root for !TDP, but that's a bug fix and should be ordered before any optimizations anyways.

#36 lore raw

[PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

For shadow paging, the pae_root needs to be reconstructed before the coming VMENTER if the guest PDPTEs is changed.

But not all paths that call load_pdptrs() will cause the pae_root to be reconstructed. Normally, kvm_mmu_reset_context() and kvm_mmu_free_roots() are used to launch later reconstruction.

The commit d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR0.CD and CR0.NW.

The commit 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush") skips kvm_mmu_free_roots() after load_pdptrs() when rewriting the CR3 with the same value.

The commit a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR4.PGE.

Normally, the guest doesn't change the PDPTEs before doing only the above operation without touching other bits that can force pae_root to be reconstructed. Guests like linux would keep the PDPTEs unchaged for every instance of pagetable.

Fixes d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed")
Fixes 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush")
Fixes a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE")
---
 arch/x86/kvm/x86.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +8 -2
index 0176eaa86a35..cfba337e46ab 100644
@@ -832,8 +832,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
 	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
 		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
 		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-		/* Ensure the dirty PDPTEs to be loaded. */
-		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
+		 * enabled or pae_root to be reconstructed for shadow paging.
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);
 	}
 	vcpu->arch.pdptrs_from_userspace = false;
 
Signature · 1 line
2.19.1.6.gb485710b
#37 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

Hello, Paolo

any thought/concern about this one?

Thanks
Lai

Quoted 56 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

For shadow paging, the pae_root needs to be reconstructed before the coming VMENTER if the guest PDPTEs is changed.

But not all paths that call load_pdptrs() will cause the pae_root to be reconstructed. Normally, kvm_mmu_reset_context() and kvm_mmu_free_roots() are used to launch later reconstruction.

The commit d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR0.CD and CR0.NW.

The commit 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush") skips kvm_mmu_free_roots() after load_pdptrs() when rewriting the CR3 with the same value.

The commit a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR4.PGE.

Normally, the guest doesn't change the PDPTEs before doing only the above operation without touching other bits that can force pae_root to be reconstructed. Guests like linux would keep the PDPTEs unchaged for every instance of pagetable.

Fixes d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed")
Fixes 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush")
Fixes a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE")
---
  arch/x86/kvm/x86.c | 10 ++++++++--
  1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +8 -2
index 0176eaa86a35..cfba337e46ab 100644
@@ -832,8 +832,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
  	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
  		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
  		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-		/* Ensure the dirty PDPTEs to be loaded. */
-		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
+		 * enabled or pae_root to be reconstructed for shadow paging.
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);
  	}
  	vcpu->arch.pdptrs_from_userspace = false;
 
#38 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

In reply to Lai Jiangshan
Quoted 9 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

For shadow paging, the pae_root needs to be reconstructed before the coming VMENTER if the guest PDPTEs is changed.

But not all paths that call load_pdptrs() will cause the pae_root to be reconstructed. Normally, kvm_mmu_reset_context() and kvm_mmu_free_roots() are used to launch later reconstruction.

Quoted 7 lines

The commit d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR0.CD and CR0.NW.

The commit 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush") skips kvm_mmu_free_roots() after load_pdptrs() when rewriting the CR3 with the same value.

This isn't accurate, prior to that commit KVM wasn't guaranteed to do kvm_mmu_free_roots() if it got a hit on the current CR3 or if a previous CR3 in the cache matched the new CR3 (the "cache" has done some odd things in the past).

So I think this particular flavor would be:

Fixes 7c390d350f8b ("kvm: x86: Add fast CR3 switch code path")
Quoted 25 lines

The commit a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR4.PGE.

Normally, the guest doesn't change the PDPTEs before doing only the above operation without touching other bits that can force pae_root to be reconstructed. Guests like linux would keep the PDPTEs unchaged for every instance of pagetable.

Fixes d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed")
Fixes 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush")
Fixes a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE")
---
 arch/x86/kvm/x86.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +0 -0
index 0176eaa86a35..cfba337e46ab 100644
@@ -832,8 +832,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
 	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
 		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
 		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 842

Quoted 10 lines
arch/x86/kvm/x86.c +8 -2
-		/* Ensure the dirty PDPTEs to be loaded. */
-		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
+		 * enabled or pae_root to be reconstructed for shadow paging.
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);

Shouldn't matter since it's legacy shadow paging, but @mmu should be used instead of vcpu->arch.mmuvcpu->arch.mmu.

To avoid a dependency on the previous patch, I think it makes sense to have this be:

	if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)))
		kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOT_CURRENT);

before the memcpy().

Then we can decide independently if skipping the KVM_REQ_LOAD_MMU_PGD if the PDPTRs are unchanged with respect to the MMU is safe.

#39 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

Quoted 5 lines · Sean Christopherson
Quoted 4 lines

The commit 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush") skips kvm_mmu_free_roots() after load_pdptrs() when rewriting the CR3 with the same value.

Quoted 7 lines

This isn't accurate, prior to that commit KVM wasn't guaranteed to do kvm_mmu_free_roots() if it got a hit on the current CR3 or if a previous CR3 in the cache matched the new CR3 (the "cache" has done some odd things in the past).

So I think this particular flavor would be:

Fixes 7c390d350f8b ("kvm: x86: Add fast CR3 switch code path")

If guest is 32bit, fast_cr3_switch() always return false, and kvm_mmu_free_roots() is always called, and no cr3 goes in prev_root.

And from 21823fbda552, fast_cr3_switch() and kvm_mmu_free_roots() are both skipped when cr3 is unchanged.

Quoted 31 lines
Quoted 31 lines

The commit a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE") skips kvm_mmu_reset_context() after load_pdptrs() when changing CR4.PGE.

Normally, the guest doesn't change the PDPTEs before doing only the above operation without touching other bits that can force pae_root to be reconstructed. Guests like linux would keep the PDPTEs unchaged for every instance of pagetable.

Fixes d81135a57aa6("KVM: x86: do not reset mmu if CR0.CD and CR0.NW are changed")
Fixes 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush")
Fixes a91a7c709600("KVM: X86: Don't reset mmu context when toggling X86_CR4_PGE")
---
  arch/x86/kvm/x86.c | 10 ++++++++--
  1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +3 -2
index 0176eaa86a35..cfba337e46ab 100644
@@ -832,8 +832,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
  	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
  		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
  		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-		/* Ensure the dirty PDPTEs to be loaded. */
-		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
+		 * enabled or pae_root to be reconstructed for shadow paging.
Quoted 8 lines

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 842

Quoted 5 lines
arch/x86/kvm/x86.c +5 -0
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);

Shouldn't matter since it's legacy shadow paging, but @mmu should be used instead of vcpu->arch.mmuvcpu->arch.mmu.

@mmu is the "guest mmu" (vcpu->arch.walk_mmu), which is used to walk including loading pdptr.

vcpu->arch.mmu is for host constructing mmu for shadowed or tdp mmu which is used in host side management including kvm_mmu_free_roots().

Even they are the same pointer now for !tdp, the meaning is different. I prefer to distinguish them even before kvm_mmu is split different for guest mmu (vcpu->arch.walk_mmu) and host constructing mmu (vcpu->arch.mmu).

(I once searched all the usage of undistinguished usage of kvm_mmu *mmu, and found a bug, see "Use vcpu->arch.walk_mmu for kvm_mmu_invlpg()")

I think Paolo is doing the splitting, unless I would take the job because I have some patches pending depended them.

Quoted 6 lines

To avoid a dependency on the previous patch, I think it makes sense to have this be:

	if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)))
		kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOT_CURRENT);

Yes, it is a good idea to add this first.

Thanks for review and suggestion.
Lai

#40 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

Quoted 12 lines · Lai Jiangshan
Quoted 9 lines · Sean Christopherson
Quoted 4 lines

The commit 21823fbda552("KVM: x86: Invalidate all PGDs for the current PCID on MOV CR3 w/ flush") skips kvm_mmu_free_roots() after load_pdptrs() when rewriting the CR3 with the same value.

This isn't accurate, prior to that commit KVM wasn't guaranteed to do kvm_mmu_free_roots() if it got a hit on the current CR3 or if a previous CR3 in the cache matched the new CR3 (the "cache" has done some odd things in the past).

Quoted 6 lines
Quoted 3 lines

So I think this particular flavor would be:

Fixes 7c390d350f8b ("kvm: x86: Add fast CR3 switch code path")

If guest is 32bit, fast_cr3_switch() always return false, and kvm_mmu_free_roots() is always called, and no cr3 goes in prev_root.

Oh, duh, PAE paging.

Quoted 15 lines
Quoted 11 lines

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 842

Quoted 8 lines
arch/x86/kvm/x86.c +8 -0
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
+		 * enabled or pae_root to be reconstructed for shadow paging.
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);

Shouldn't matter since it's legacy shadow paging, but @mmu should be used instead of vcpu->arch.mmuvcpu->arch.mmu.

@mmu is the "guest mmu" (vcpu->arch.walk_mmu), which is used to walk including loading pdptr.

Quoted 6 lines

vcpu->arch.mmu is for host constructing mmu for shadowed or tdp mmu which is used in host side management including kvm_mmu_free_roots().

Even they are the same pointer now for !tdp, the meaning is different. I prefer to distinguish them even before kvm_mmu is split different for guest mmu (vcpu->arch.walk_mmu) and host constructing mmu (vcpu->arch.mmu).

I see where you're coming from. I was viewing it from the perspective of, "they're the same thing for shadow paging, why reread mmu?".

I'm ok with explicitly referencing arch.mmu, but maybe add a comment?

maintainer
#41 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

In reply to Sean Christopherson
Quoted 8 lines · Sean Christopherson
Quoted 8 lines
arch/x86/kvm/x86.c +2 -2
@@ -832,8 +832,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
  	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
  		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
  		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-		/* Ensure the dirty PDPTEs to be loaded. */
-		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
Quoted 8 lines

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 842

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+		 * enabled or pae_root to be reconstructed for shadow paging.
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);

Shouldn't matter since it's legacy shadow paging, but @mmu should be used instead of vcpu->arch.mmuvcpu->arch.mmu.

In kvm/next actually there's no mmu parameter to load_pdptrs, so it's okay to keep vcpu->arch.mmu.

Quoted 9 lines

To avoid a dependency on the previous patch, I think it makes sense to have this be:

	if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)))
		kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOT_CURRENT);

before the memcpy().

Then we can decide independently if skipping the KVM_REQ_LOAD_MMU_PGD if the PDPTRs are unchanged with respect to the MMU is safe.

Do you disagree that there's already an invariant that the PDPTRs can only be dirty if KVM_REQ_LOAD_MMU_PGD---and therefore a previous change to the PDPTRs would have triggered KVM_REQ_LOAD_MMU_PGD? This is opposed to the guest TLB flush due to MOV CR3; that one has to be done unconditionally for PAE paging, and it is handled separately within kvm_set_cr3.

Paolo

#42 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

Quoted 26 lines · Paolo Bonzini
Quoted 8 lines · Sean Christopherson
Quoted 8 lines
arch/x86/kvm/x86.c +2 -2
@@ -832,8 +832,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
  	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
  		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
  		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-		/* Ensure the dirty PDPTEs to be loaded. */
-		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
Quoted 8 lines

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 842

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+		 * enabled or pae_root to be reconstructed for shadow paging.
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);

Shouldn't matter since it's legacy shadow paging, but @mmu should be used instead of vcpu->arch.mmuvcpu->arch.mmu.

In kvm/next actually there's no mmu parameter to load_pdptrs, so it's okay to keep vcpu->arch.mmu.

Quoted 5 lines

To avoid a dependency on the previous patch, I think it makes sense to have this be:

	if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)))
		kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOT_CURRENT);
Quoted 8 lines
Quoted 4 lines

before the memcpy().

Then we can decide independently if skipping the KVM_REQ_LOAD_MMU_PGD if the PDPTRs are unchanged with respect to the MMU is safe.

Do you disagree that there's already an invariant that the PDPTRs can only be dirty if KVM_REQ_LOAD_MMU_PGD---and therefore a previous change to the PDPTRs would have triggered KVM_REQ_LOAD_MMU_PGD?

What I think is moot, because commit 24cd19a28cb7 ("KVM: X86: Update mmu->pdptrs only when it is changed") breaks nested VMs with EPT in L0 and PAE shadow paging in L2. Reproducing is trivial, just disable EPT in L1 and run a VM. I haven't investigating how it breaks things, because why it's broken is secondary for me.

My primary concern is that we would even consider optimizing the PDPTR logic without a mountain of evidence that any patch is correct for all scenarios. We had to add an entire ioctl() just to get PDPTRs functional. This apparently wasn't validated against a simple use case, let alone against things like migration with nested VMs, multliple L2s, etc...

#43 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

Quoted 32 lines · Sean Christopherson
Quoted 31 lines · Paolo Bonzini
Quoted 8 lines · Sean Christopherson
Quoted 8 lines
arch/x86/kvm/x86.c +2 -2
@@ -832,8 +832,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
  	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
  		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
  		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-		/* Ensure the dirty PDPTEs to be loaded. */
-		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
Quoted 8 lines

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 842

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+		 * enabled or pae_root to be reconstructed for shadow paging.
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);

Shouldn't matter since it's legacy shadow paging, but @mmu should be used instead of vcpu->arch.mmuvcpu->arch.mmu.

In kvm/next actually there's no mmu parameter to load_pdptrs, so it's okay to keep vcpu->arch.mmu.

Quoted 9 lines

To avoid a dependency on the previous patch, I think it makes sense to have this be:

	if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)))
		kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOT_CURRENT);

before the memcpy().

Then we can decide independently if skipping the KVM_REQ_LOAD_MMU_PGD if the PDPTRs are unchanged with respect to the MMU is safe.

Quoted 7 lines
Quoted 3 lines

Do you disagree that there's already an invariant that the PDPTRs can only be dirty if KVM_REQ_LOAD_MMU_PGD---and therefore a previous change to the PDPTRs would have triggered KVM_REQ_LOAD_MMU_PGD?

What I think is moot, because commit 24cd19a28cb7 ("KVM: X86: Update mmu->pdptrs only when it is changed") breaks nested VMs with EPT in L0 and PAE shadow paging in L2. Reproducing is trivial, just disable EPT in L1 and run a VM. I haven't

Doh, s/L2/L1

Quoted 7 lines

investigating how it breaks things, because why it's broken is secondary for me.

My primary concern is that we would even consider optimizing the PDPTR logic without a mountain of evidence that any patch is correct for all scenarios. We had to add an entire ioctl() just to get PDPTRs functional. This apparently wasn't validated against a simple use case, let alone against things like migration with nested VMs, multliple L2s, etc...

#44 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

In reply to Sean Christopherson
Quoted 41 lines · Sean Christopherson
Quoted 26 lines · Paolo Bonzini
Quoted 8 lines · Sean Christopherson
Quoted 8 lines
arch/x86/kvm/x86.c +2 -2
@@ -832,8 +832,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3)
  	if (memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) {
  		memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
  		kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
-		/* Ensure the dirty PDPTEs to be loaded. */
-		kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		/*
+		 * Ensure the dirty PDPTEs to be loaded for VMX with EPT
Quoted 8 lines

arch/x86/kvm/x86.c int load_pdptrs(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, unsigned long cr3) · line 842

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+		 * enabled or pae_root to be reconstructed for shadow paging.
+		 */
+		if (tdp_enabled)
+			kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+		else
+			kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT);

Shouldn't matter since it's legacy shadow paging, but @mmu should be used instead of vcpu->arch.mmuvcpu->arch.mmu.

In kvm/next actually there's no mmu parameter to load_pdptrs, so it's okay to keep vcpu->arch.mmu.

Quoted 5 lines

To avoid a dependency on the previous patch, I think it makes sense to have this be:

	if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)))
		kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOT_CURRENT);
Quoted 8 lines
Quoted 4 lines

before the memcpy().

Then we can decide independently if skipping the KVM_REQ_LOAD_MMU_PGD if the PDPTRs are unchanged with respect to the MMU is safe.

Do you disagree that there's already an invariant that the PDPTRs can only be dirty if KVM_REQ_LOAD_MMU_PGD---and therefore a previous change to the PDPTRs would have triggered KVM_REQ_LOAD_MMU_PGD?

What I think is moot, because commit 24cd19a28cb7 ("KVM: X86: Update mmu->pdptrs only when it is changed") breaks nested VMs with EPT in L0 and PAE shadow paging in L2. Reproducing is trivial, just disable EPT in L1 and run a VM. I haven't investigating how it breaks things, because why it's broken is secondary for me.

Quoted 5 lines

My primary concern is that we would even consider optimizing the PDPTR logic without a mountain of evidence that any patch is correct for all scenarios. We had to add an entire ioctl() just to get PDPTRs functional. This apparently wasn't validated against a simple use case, let alone against things like migration with nested VMs, multliple L2s, etc...

I did validate the *SREGS2* against all the cases I could (like migration, EPT/NPT disabled/etc. I even started testing SMM to see how it affects PDPTRs, and patched seabios to use PAE paging. I still could have missed something.

But note that qemu still doesn't use that ioctl (patch stuck in review).

Best regards,
Maxim Levitsky

Quoted 1 line
maintainer
#45 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

Quoted 8 lines · Maxim Levitsky
Quoted 3 lines

This apparently wasn't validated against a simple use case, let alone against things like migration with nested VMs, multliple L2s, etc...

I did validate the *SREGS2* against all the cases I could (like migration, EPT/NPT disabled/etc. I even started testing SMM to see how it affects PDPTRs, and patched seabios to use PAE paging. I still could have missed something.

Don't worry, I think Sean was talking about patch 16 and specifically digging at me (who deserved it completely).

Paolo

#46 lore raw

Re: [PATCH 17/15] KVM: X86: Ensure pae_root to be reconstructed for shadow paging if the guest PDPTEs is changed

Quoted 5 lines · Paolo Bonzini
Quoted 4 lines · Maxim Levitsky
Quoted 3 lines

This apparently wasn't validated against a simple use case, let alone against things like migration with nested VMs, multliple L2s, etc...

Quoted 7 lines
Quoted 4 lines

I did validate the *SREGS2* against all the cases I could (like migration, EPT/NPT disabled/etc. I even started testing SMM to see how it affects PDPTRs, and patched seabios to use PAE paging. I still could have missed something.

Don't worry, I think Sean was talking about patch 16 and specifically digging at me (who deserved it completely).

Yes, patch 16. My goal wasn't to dig at anyone, I just wanted to dramatically emphasize how ridiculousy fragile and complex the PDPTR crud is due to the number of edge cases.

#47 lore raw
In reply to Lai Jiangshan

Hello,all

Ping

Thanks
Lai

Quoted 54 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

The patchset was started when I read the code of nested_svm_load_cr3() and found that it marks CR3 available other than dirty when changing vcpu->arch.cr3. I thought its caller has ensured that vmcs.GUEST_CR3 will be or already be set to @cr3 so that it doesn't need to be marked dirty. And later I found that it is not true and it must be a bug in a rare case before I realized that all the code just (ab)uses vcpu->arch.regs_avail for VCPU_EXREG_CR3 and there is not such bug of using regs_avail here.
(The above finding becomes a low meaning patch_15 rather than a fix)

The unhappyness of the reading code made me do some cleanup for regs_avail and regs_dirty and kvm_register_xxx() functions in the hope that the code become clearer with less misunderstanding.

Major focus was on VCPU_EXREG_CR3 and VCPU_EXREG_PDPTR. They are ensured to be marked the correct tags (available or dirty), and the value is ensured to be synced to architecture before run if it is marked dirty.

When cleaning VCPU_EXREG_PDPTR, I also checked if the corresponding cr0/cr4 pdptr bits are all intercepted when !tdp_enabled, and I think it is not clear enough, so X86_CR4_PDPTR_BITS is added as self-comments in the code.

Lai Jiangshan (15):
KVM: X86: Ensure the dirty PDPTEs to be loaded
KVM: VMX: Mark VCPU_EXREG_PDPTR available in ept_save_pdptrs()
KVM: SVM: Always clear available of VCPU_EXREG_PDPTR in svm_vcpu_run()
KVM: VMX: Add and use X86_CR4_TLB_BITS when !enable_ept
KVM: VMX: Add and use X86_CR4_PDPTR_BITS when !enable_ept
KVM: X86: Move CR0 pdptr_bits into header file as X86_CR0_PDPTR_BITS
KVM: SVM: Remove outdate comment in svm_load_mmu_pgd()
KVM: SVM: Remove useless check in svm_load_mmu_pgd()
KVM: SVM: Remove the unneeded code to mark available for CR3
KVM: X86: Mark CR3 dirty when vcpu->arch.cr3 is changed
KVM: VMX: Update vmcs.GUEST_CR3 only when the guest CR3 is dirty
KVM: VMX: Reset the bits that are meaningful to be reset in
vmx_register_cache_reset()
KVM: SVM: Add and use svm_register_cache_reset()
KVM: X86: Remove kvm_register_clear_available()
KVM: nVMX: Always write vmcs.GUEST_CR3 during nested VM-Exit

  arch/x86/kvm/kvm_cache_regs.h | 13 ++++++------
  arch/x86/kvm/svm/nested.c     |  1 -
  arch/x86/kvm/svm/svm.c        | 17 ++++++++--------
  arch/x86/kvm/svm/svm.h        | 26 ++++++++++++++++++++++++
  arch/x86/kvm/vmx/nested.c     | 30 ++++++++++++++++++----------
  arch/x86/kvm/vmx/vmx.c        | 12 +++++++-----
  arch/x86/kvm/vmx/vmx.h        | 37 +++++++++++++++++++++++++----------
  arch/x86/kvm/x86.c            | 13 ++++++------
  8 files changed, 101 insertions(+), 48 deletions(-)