Skip to contentSkip to conversation

More parallel operations for the TDP MMU

PATCH Applied Ben Gardon 3 participants 22 messages Apr 1, 2021 – May 27, 2021

Subsystems (3)
lore
Share & tools
lore

Ben Gardon

author
#0 lore raw

v2 cover More parallel operations for the TDP MMU

Now that the TDP MMU is able to handle page faults in parallel, it's a relatively small change to expand to other operations. This series allows zapping a range of GFNs, reclaiming collapsible SPTEs (when disabling dirty logging), and enabling dirty logging to all happen under the MMU lock in read mode.

This is partly a cleanup + rewrite of the last few patches of the parallel page faults series. I've incorporated feedback from Sean and Paolo, but the patches have changed so much that I'm sending this as a separate series.

Ran kvm-unit-tests + selftests on an SMP kernel + Intel Skylake, with the TDP MMU enabled and disabled. This series introduces no new failures or warnings.

I know this will conflict horribly with the patches from Sean's series which were just queued, and I'll send a v2 to fix those conflicts + address any feedback on this v1.

Changelog
v2:
-- Rebased patches on top of kvm/queue to incorporate Sean's recent
TLB flushing changes
-- Dropped patch 5: "KVM: x86/mmu: comment for_each_tdp_mmu_root
requires MMU write lock" as the following patch to protect the roots list with RCU adds lockdep which makes the comment somewhat redundant.

Ben Gardon (13):
KVM: x86/mmu: Re-add const qualifier in
kvm_tdp_mmu_zap_collapsible_sptes
KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU
KVM: x86/mmu: use tdp_mmu_free_sp to free roots
KVM: x86/mmu: Merge TDP MMU put and free root
KVM: x86/mmu: Refactor yield safe root iterator
KVM: x86/mmu: Make TDP MMU root refcount atomic
KVM: x86/mmu: handle cmpxchg failure in kvm_tdp_mmu_get_root
KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU
KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read
lock
KVM: x86/mmu: Fast invalidation for TDP MMU
KVM: x86/mmu: Tear down roots in fast invalidation thread

 arch/x86/include/asm/kvm_host.h |  21 +-
 arch/x86/kvm/mmu/mmu.c          | 115 +++++++---
 arch/x86/kvm/mmu/mmu_internal.h |  27 +--
 arch/x86/kvm/mmu/tdp_mmu.c      | 375 +++++++++++++++++++++++---------
 arch/x86/kvm/mmu/tdp_mmu.h      |  28 ++-
 include/linux/kvm_host.h        |   2 +-
 6 files changed, 407 insertions(+), 161 deletions(-)
Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#1 lore raw

PATCH v2 1/13 KVM: x86/mmu: Re-add const qualifier in kvm_tdp_mmu_zap_collapsible_sptes

kvm_tdp_mmu_zap_collapsible_sptes unnecessarily removes the const qualifier from its memlsot argument, leading to a compiler warning. Add the const annotation and pass it to subsequent functions.

---
 arch/x86/kvm/mmu/mmu.c          | 10 +++++-----
 arch/x86/kvm/mmu/mmu_internal.h |  5 +++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  5 +++--
 arch/x86/kvm/mmu/tdp_mmu.h      |  3 ++-
 include/linux/kvm_host.h        |  2 +-
 5 files changed, 14 insertions(+), 11 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -5
index efb41f31e80a..617809529987 100644
@@ -715,8 +715,7 @@ static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn)
  * handling slots that are not large page aligned.
  */
 static struct kvm_lpage_info *lpage_info_slot(gfn_t gfn,
-					      struct kvm_memory_slot *slot,
-					      int level)
+		const struct kvm_memory_slot *slot, int level)
 {
 	unsigned long idx;
 
@@ -2735,7 +2734,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
 }
 
 static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-				  struct kvm_memory_slot *slot)
+				  const struct kvm_memory_slot *slot)
 {
 	unsigned long hva;
 	pte_t *pte;
@@ -2761,8 +2760,9 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	return level;
 }
 
-int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_memory_slot *slot,
-			      gfn_t gfn, kvm_pfn_t pfn, int max_level)
+int kvm_mmu_max_mapping_level(struct kvm *kvm,
+			      const struct kvm_memory_slot *slot, gfn_t gfn,
+			      kvm_pfn_t pfn, int max_level)
 {
 	struct kvm_lpage_info *linfo;
 
arch/x86/kvm/mmu/mmu_internal.h +3 -2
index cead1d81e663..d44fe8a43a19 100644
@@ -161,8 +161,9 @@ enum {
 #define SET_SPTE_NEED_REMOTE_TLB_FLUSH	BIT(1)
 #define SET_SPTE_SPURIOUS		BIT(2)
 
-int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_memory_slot *slot,
-			      gfn_t gfn, kvm_pfn_t pfn, int max_level);
+int kvm_mmu_max_mapping_level(struct kvm *kvm,
+			      const struct kvm_memory_slot *slot, gfn_t gfn,
+			      kvm_pfn_t pfn, int max_level);
 int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn,
 			    int max_level, kvm_pfn_t *pfnp,
 			    bool huge_page_disallowed, int *req_level);
arch/x86/kvm/mmu/tdp_mmu.c +3 -2
index ccf0d774a181..d5210a212c59 100644
@@ -1255,7 +1255,7 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm,
  */
 static bool zap_collapsible_spte_range(struct kvm *kvm,
 				       struct kvm_mmu_page *root,
-				       struct kvm_memory_slot *slot,
+				       const struct kvm_memory_slot *slot,
 				       bool flush)
 {
 	gfn_t start = slot->base_gfn;
@@ -1296,7 +1296,8 @@ static bool zap_collapsible_spte_range(struct kvm *kvm,
  * be replaced by large mappings, for GFNs within the slot.
  */
 bool kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
-				       struct kvm_memory_slot *slot, bool flush)
+				       const struct kvm_memory_slot *slot,
+				       bool flush)
 {
 	struct kvm_mmu_page *root;
 
arch/x86/kvm/mmu/tdp_mmu.h +2 -1
index bf3ce169122e..d7007480b3d2 100644
@@ -57,7 +57,8 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm,
 				       gfn_t gfn, unsigned long mask,
 				       bool wrprot);
 bool kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
-				       struct kvm_memory_slot *slot, bool flush);
+				       const struct kvm_memory_slot *slot,
+				       bool flush);
 
 bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm,
 				   struct kvm_memory_slot *slot, gfn_t gfn);
include/linux/kvm_host.h +1 -1
index e6d77353025c..5e0d17b1ac2b 100644
@@ -1124,7 +1124,7 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn)
 }
 
 static inline unsigned long
-__gfn_to_hva_memslot(struct kvm_memory_slot *slot, gfn_t gfn)
+__gfn_to_hva_memslot(const struct kvm_memory_slot *slot, gfn_t gfn)
 {
 	return slot->userspace_addr + (gfn - slot->base_gfn) * PAGE_SIZE;
 }
Signature · 1 line
2.31.0.208.g409f899ff0-goog
#2 lore raw

Re: [PATCH v2 01/13] KVM: x86/mmu: Re-add const qualifier in kvm_tdp_mmu_zap_collapsible_sptes

Quoted 18 lines · Ben Gardon

kvm_tdp_mmu_zap_collapsible_sptes unnecessarily removes the const qualifier from its memlsot argument, leading to a compiler warning. Add the const annotation and pass it to subsequent functions.

---
 arch/x86/kvm/mmu/mmu.c          | 10 +++++-----
 arch/x86/kvm/mmu/mmu_internal.h |  5 +++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  5 +++--
 arch/x86/kvm/mmu/tdp_mmu.h      |  3 ++-
 include/linux/kvm_host.h        |  2 +-
 5 files changed, 14 insertions(+), 11 deletions(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index efb41f31e80a..617809529987 100644

arch/x86/kvm/mmu/mmu.c static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn) · line 718

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +1 -2
@@ -715,8 +715,7 @@ static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn)
  * handling slots that are not large page aligned.
  */
 static struct kvm_lpage_info *lpage_info_slot(gfn_t gfn,
-					      struct kvm_memory_slot *slot,
-					      int level)
+		const struct kvm_memory_slot *slot, int level)

I'd prefer to let this poke out (it's only 2 chars) instead of having an unaligned param list. And if we want to make it pretty, we could always do:

static struct kvm_lpage_info *lpage_info_slot(const struct kvm_memory_slot *slot,
gfn_t gfn, int level)

author
#3 lore raw

PATCH v2 2/13 KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU

In reply to Ben Gardon

The TDP MMU is almost the only user of kvm_mmu_get_root and kvm_mmu_put_root. There is only one use of put_root in mmu.c for the legacy / shadow MMU. Open code that one use and move the get / put functions to the TDP MMU so they can be extended in future commits.

No functional change intended.

---
 arch/x86/kvm/mmu/mmu.c          | 10 ++++------
 arch/x86/kvm/mmu/mmu_internal.h | 16 ----------------
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 +++---
 arch/x86/kvm/mmu/tdp_mmu.h      | 18 ++++++++++++++++++
 4 files changed, 25 insertions(+), 25 deletions(-)
arch/x86/kvm/mmu/mmu.c +4 -6
index 617809529987..9c7ef7ca8bf6 100644
@@ -3153,12 +3153,10 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
 
 	sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
 
-	if (kvm_mmu_put_root(kvm, sp)) {
-		if (is_tdp_mmu_page(sp))
-			kvm_tdp_mmu_free_root(kvm, sp);
-		else if (sp->role.invalid)
-			kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
-	}
+	if (is_tdp_mmu_page(sp) && kvm_tdp_mmu_put_root(kvm, sp))
+		kvm_tdp_mmu_free_root(kvm, sp);
+	else if (!--sp->root_count && sp->role.invalid)
+		kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 
 	*root_hpa = INVALID_PAGE;
 }
arch/x86/kvm/mmu/mmu_internal.h +0 -16
index d44fe8a43a19..9347d73996b5 100644
@@ -113,22 +113,6 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
 void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 					u64 start_gfn, u64 pages);
 
-static inline void kvm_mmu_get_root(struct kvm *kvm, struct kvm_mmu_page *sp)
-{
-	BUG_ON(!sp->root_count);
-	lockdep_assert_held(&kvm->mmu_lock);
-
-	++sp->root_count;
-}
-
-static inline bool kvm_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *sp)
-{
-	lockdep_assert_held(&kvm->mmu_lock);
-	--sp->root_count;
-
-	return !sp->root_count;
-}
-
 static inline int kvm_mmu_role_as_id(union kvm_mmu_page_role role)
 {
 	return role.smm ? 1 : 0;
arch/x86/kvm/mmu/tdp_mmu.c +3 -3
index d5210a212c59..6f612ac755a0 100644
@@ -43,7 +43,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
 
 static void tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
 {
-	if (kvm_mmu_put_root(kvm, root))
+	if (kvm_tdp_mmu_put_root(kvm, root))
 		kvm_tdp_mmu_free_root(kvm, root);
 }
 
@@ -55,7 +55,7 @@ static inline bool tdp_mmu_next_root_valid(struct kvm *kvm,
 	if (list_entry_is_head(root, &kvm->arch.tdp_mmu_roots, link))
 		return false;
 
-	kvm_mmu_get_root(kvm, root);
+	kvm_tdp_mmu_get_root(kvm, root);
 	return true;
 
 }
@@ -154,7 +154,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 	/* Check for an existing root before allocating a new one. */
 	for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) {
 		if (root->role.word == role.word) {
-			kvm_mmu_get_root(kvm, root);
+			kvm_tdp_mmu_get_root(kvm, root);
 			goto out;
 		}
 	}
arch/x86/kvm/mmu/tdp_mmu.h +18 -0
index d7007480b3d2..c9a081c786a5 100644
@@ -8,6 +8,24 @@
 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
 void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root);
 
+static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
+					struct kvm_mmu_page *root)
+{
+	BUG_ON(!root->root_count);
+	lockdep_assert_held(&kvm->mmu_lock);
+
+	++root->root_count;
+}
+
+static inline bool kvm_tdp_mmu_put_root(struct kvm *kvm,
+					struct kvm_mmu_page *root)
+{
+	lockdep_assert_held(&kvm->mmu_lock);
+	--root->root_count;
+
+	return !root->root_count;
+}
+
 bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, gfn_t start,
 				 gfn_t end, bool can_yield, bool flush);
 static inline bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id,
Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#4 lore raw

PATCH v2 3/13 KVM: x86/mmu: use tdp_mmu_free_sp to free roots

In reply to Ben Gardon

Minor cleanup to deduplicate the code used to free a struct kvm_mmu_page in the TDP MMU.

No functional change intended.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +7 -8
index 6f612ac755a0..320cc4454737 100644
@@ -92,6 +92,12 @@ static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
 			  gfn_t start, gfn_t end, bool can_yield, bool flush);
 
+static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
+{
+	free_page((unsigned long)sp->spt);
+	kmem_cache_free(mmu_page_header_cache, sp);
+}
+
 void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root)
 {
 	gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
@@ -105,8 +111,7 @@ void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root)
 
 	zap_gfn_range(kvm, root, 0, max_gfn, false, false);
 
-	free_page((unsigned long)root->spt);
-	kmem_cache_free(mmu_page_header_cache, root);
+	tdp_mmu_free_sp(root);
 }
 
 static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu,
@@ -168,12 +173,6 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 	return __pa(root->spt);
 }
 
-static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
-{
-	free_page((unsigned long)sp->spt);
-	kmem_cache_free(mmu_page_header_cache, sp);
-}
-
 /*
  * This is called through call_rcu in order to free TDP page table memory
  * safely with respect to other kernel threads that may be operating on
Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#5 lore raw

PATCH v2 4/13 KVM: x86/mmu: Merge TDP MMU put and free root

In reply to Ben Gardon

kvm_tdp_mmu_put_root and kvm_tdp_mmu_free_root are always called together, so merge the functions to simplify TDP MMU root refcounting / freeing.

---
 arch/x86/kvm/mmu/mmu.c     |  4 +--
 arch/x86/kvm/mmu/tdp_mmu.c | 54 ++++++++++++++++++--------------------
 arch/x86/kvm/mmu/tdp_mmu.h | 10 +------
 3 files changed, 28 insertions(+), 40 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -2
index 9c7ef7ca8bf6..47d996a8074f 100644
@@ -3153,8 +3153,8 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
 
 	sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
 
-	if (is_tdp_mmu_page(sp) && kvm_tdp_mmu_put_root(kvm, sp))
-		kvm_tdp_mmu_free_root(kvm, sp);
+	if (is_tdp_mmu_page(sp))
+		kvm_tdp_mmu_put_root(kvm, sp);
 	else if (!--sp->root_count && sp->role.invalid)
 		kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 
arch/x86/kvm/mmu/tdp_mmu.c +25 -29

73 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +1 -9
index c9a081c786a5..d4e32ac5f4c9 100644
@@ -6,7 +6,6 @@
 #include <linux/kvm_host.h>
 
 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
-void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root);
 
 static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
 					struct kvm_mmu_page *root)
@@ -17,14 +16,7 @@ static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
 	++root->root_count;
 }
 
-static inline bool kvm_tdp_mmu_put_root(struct kvm *kvm,
-					struct kvm_mmu_page *root)
-{
-	lockdep_assert_held(&kvm->mmu_lock);
-	--root->root_count;
-
-	return !root->root_count;
-}
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
 
 bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, gfn_t start,
 				 gfn_t end, bool can_yield, bool flush);
Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#6 lore raw

PATCH v2 5/13 KVM: x86/mmu: Refactor yield safe root iterator

In reply to Ben Gardon

Refactor the yield safe TDP MMU root iterator to be more amenable to changes in future commits which will allow it to be used under the MMU lock in read mode. Currently the iterator requires a complicated dance between the helper functions and different parts of the for loop which makes it hard to reason about. Moving all the logic into a single function simplifies the iterator substantially.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 45 ++++++++++++++++++++++----------------
 1 file changed, 26 insertions(+), 19 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +26 -19
index 279a725061f7..670c5e3ad80e 100644
@@ -68,26 +68,34 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
 	tdp_mmu_free_sp(root);
 }
 
-static inline bool tdp_mmu_next_root_valid(struct kvm *kvm,
-					   struct kvm_mmu_page *root)
+/*
+ * Finds the next valid root after root (or the first valid root if root
+ * is NULL), takes a reference on it, and returns that next root. If root
+ * is not NULL, this thread should have already taken a reference on it, and
+ * that reference will be dropped. If no valid root is found, this
+ * function will return NULL.
+ */
+static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
+					      struct kvm_mmu_page *prev_root)
 {
-	lockdep_assert_held_write(&kvm->mmu_lock);
+	struct kvm_mmu_page *next_root;
 
-	if (list_entry_is_head(root, &kvm->arch.tdp_mmu_roots, link))
-		return false;
+	lockdep_assert_held_write(&kvm->mmu_lock);
 
-	kvm_tdp_mmu_get_root(kvm, root);
-	return true;
+	if (prev_root)
+		next_root = list_next_entry(prev_root, link);
+	else
+		next_root = list_first_entry(&kvm->arch.tdp_mmu_roots,
+					     typeof(*next_root), link);
 
-}
+	if (list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link))
+		next_root = NULL;
+	else
+		kvm_tdp_mmu_get_root(kvm, next_root);
 
-static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
-						     struct kvm_mmu_page *root)
-{
-	struct kvm_mmu_page *next_root;
+	if (prev_root)
+		kvm_tdp_mmu_put_root(kvm, prev_root);
 
-	next_root = list_next_entry(root, link);
-	kvm_tdp_mmu_put_root(kvm, root);
 	return next_root;
 }
 
@@ -98,11 +106,10 @@ static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
  * recent root. (Unless keeping a live reference is desirable.)
  */
 #define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id)		\
-	for (_root = list_first_entry(&_kvm->arch.tdp_mmu_roots,	\
-				      typeof(*_root), link);		\
-	     tdp_mmu_next_root_valid(_kvm, _root);			\
-	     _root = tdp_mmu_next_root(_kvm, _root))			\
-		if (kvm_mmu_page_as_id(_root) != _as_id) {		\
+	for (_root = tdp_mmu_next_root(_kvm, NULL);		\
+	     _root;						\
+	     _root = tdp_mmu_next_root(_kvm, _root))		\
+		if (kvm_mmu_page_as_id(_root) != _as_id) {	\
 		} else
 
 #define for_each_tdp_mmu_root(_kvm, _root, _as_id)			\
Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#7 lore raw

PATCH v2 6/13 KVM: x86/mmu: Make TDP MMU root refcount atomic

In reply to Ben Gardon

In order to parallelize more operations for the TDP MMU, make the refcount on TDP MMU roots atomic, so that a future patch can allow multiple threads to take a reference on the root concurrently, while holding the MMU lock in read mode.

---

Changelog
v2:
-- Split failure handling for kvm_tdp_mmu_get_root out into a
seperate commit.

 arch/x86/kvm/mmu/mmu_internal.h |  6 +++++-
 arch/x86/kvm/mmu/tdp_mmu.c      |  4 ++--
 arch/x86/kvm/mmu/tdp_mmu.h      | 10 +++++++---
 3 files changed, 14 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/mmu_internal.h +5 -1
index 9347d73996b5..f63d0fdb8567 100644
@@ -50,7 +50,11 @@ struct kvm_mmu_page {
 	u64 *spt;
 	/* hold the gfn of each spte inside spt */
 	gfn_t *gfns;
-	int root_count;          /* Currently serving as active root */
+	/* Currently serving as active root */
+	union {
+		int root_count;
+		refcount_t tdp_mmu_root_count;
+	};
 	unsigned int unsync_children;
 	struct kvm_rmap_head parent_ptes; /* rmap pointers to parent sptes */
 	DECLARE_BITMAP(unsync_child_bitmap, 512);
arch/x86/kvm/mmu/tdp_mmu.c +2 -2
index 670c5e3ad80e..697ea882a3e4 100644
@@ -56,7 +56,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
-	if (--root->root_count)
+	if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
 		return;
 
 	WARN_ON(!root->tdp_mmu_page);
@@ -168,7 +168,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 	}
 
 	root = alloc_tdp_mmu_page(vcpu, 0, vcpu->arch.mmu->shadow_root_level);
-	root->root_count = 1;
+	refcount_set(&root->tdp_mmu_root_count, 1);
 
 	list_add(&root->link, &kvm->arch.tdp_mmu_roots);
 
arch/x86/kvm/mmu/tdp_mmu.h +7 -3
index d4e32ac5f4c9..1ec7914ecff9 100644
@@ -10,10 +10,14 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
 static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
 					struct kvm_mmu_page *root)
 {
-	BUG_ON(!root->root_count);
-	lockdep_assert_held(&kvm->mmu_lock);
+	lockdep_assert_held_write(&kvm->mmu_lock);
 
-	++root->root_count;
+	/*
+	 * This should never fail since roots are removed from the roots
+	 * list under the MMU write lock when their reference count falls
+	 * to zero.
+	 */
+	refcount_inc_not_zero(&root->tdp_mmu_root_count);
 }
 
 void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#8 lore raw

PATCH v2 7/13 KVM: x86/mmu: handle cmpxchg failure in kvm_tdp_mmu_get_root

In reply to Ben Gardon

To reduce dependence on the MMU write lock, don't rely on the assumption that the atomic operation in kvm_tdp_mmu_get_root will always succeed. By not relying on that assumption, threads do not need to hold the MMU lock in write mode in order to take a reference on a TDP MMU root.

In the root iterator, this change means that some roots might have to be skipped if they are found to have a zero refcount. This will still never happen as of this patch, but a future patch will need that flexibility to make the root iterator safe under the MMU read lock.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 11 ++++++-----
 arch/x86/kvm/mmu/tdp_mmu.h | 13 +++----------
 2 files changed, 9 insertions(+), 15 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +6 -5
index 697ea882a3e4..886bc170f2a5 100644
@@ -88,10 +88,12 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 		next_root = list_first_entry(&kvm->arch.tdp_mmu_roots,
 					     typeof(*next_root), link);
 
+	while (!list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link) &&
+	       !kvm_tdp_mmu_get_root(kvm, next_root))
+		next_root = list_next_entry(next_root, link);
+
 	if (list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link))
 		next_root = NULL;
-	else
-		kvm_tdp_mmu_get_root(kvm, next_root);
 
 	if (prev_root)
 		kvm_tdp_mmu_put_root(kvm, prev_root);
@@ -161,10 +163,9 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 
 	/* Check for an existing root before allocating a new one. */
 	for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) {
-		if (root->role.word == role.word) {
-			kvm_tdp_mmu_get_root(kvm, root);
+		if (root->role.word == role.word &&
+		    kvm_tdp_mmu_get_root(kvm, root))
 			goto out;
-		}
 	}
 
 	root = alloc_tdp_mmu_page(vcpu, 0, vcpu->arch.mmu->shadow_root_level);
arch/x86/kvm/mmu/tdp_mmu.h +3 -10
index 1ec7914ecff9..f0a26214e999 100644
@@ -7,17 +7,10 @@
 
 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
 
-static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
-					struct kvm_mmu_page *root)
+__must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,
+						     struct kvm_mmu_page *root)
 {
-	lockdep_assert_held_write(&kvm->mmu_lock);
-
-	/*
-	 * This should never fail since roots are removed from the roots
-	 * list under the MMU write lock when their reference count falls
-	 * to zero.
-	 */
-	refcount_inc_not_zero(&root->tdp_mmu_root_count);
+	return refcount_inc_not_zero(&root->tdp_mmu_root_count);
 }
 
 void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#9 lore raw

PATCH v2 8/13 KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU

In reply to Ben Gardon

Protect the contents of the TDP MMU roots list with RCU in preparation for a future patch which will allow the iterator macro to be used under the MMU lock in read mode.

---

Changelog
v2:
-- add lockdep condition for tdp_mmu_pages_lock to for_each_tdp_mmu_root
-- fix problem with unexported lockdep function
-- updated comments in kvm_host.h

 arch/x86/include/asm/kvm_host.h | 21 +++++++---
 arch/x86/kvm/mmu/tdp_mmu.c      | 69 +++++++++++++++++++--------------
 2 files changed, 55 insertions(+), 35 deletions(-)
arch/x86/include/asm/kvm_host.h +16 -5
index 99778ac51243..e02e8b8a875b 100644
@@ -1050,25 +1050,36 @@ struct kvm_arch {
 	bool tdp_mmu_enabled;
 
 	/*
-	 * List of struct kvmp_mmu_pages being used as roots.
+	 * List of struct kvm_mmu_pages being used as roots.
 	 * All struct kvm_mmu_pages in the list should have
 	 * tdp_mmu_page set.
-	 * All struct kvm_mmu_pages in the list should have a positive
-	 * root_count except when a thread holds the MMU lock and is removing
-	 * an entry from the list.
+	 *
+	 * For reads, this list is protected by:
+	 *	the MMU lock in read mode + RCU or
+	 *	the MMU lock in write mode
+	 *
+	 * For writes, this list is protected by:
+	 *	the MMU lock in read mode + the tdp_mmu_pages_lock or
+	 *	the MMU lock in write mode
+	 *
+	 * Roots will remain in the list until their tdp_mmu_root_count
+	 * drops to zero, at which point the thread that decremented the
+	 * count to zero should removed the root from the list and clean
+	 * it up, freeing the root after an RCU grace period.
 	 */
 	struct list_head tdp_mmu_roots;
 
 	/*
 	 * List of struct kvmp_mmu_pages not being used as roots.
 	 * All struct kvm_mmu_pages in the list should have
-	 * tdp_mmu_page set and a root_count of 0.
+	 * tdp_mmu_page set and a tdp_mmu_root_count of 0.
 	 */
 	struct list_head tdp_mmu_pages;
 
 	/*
 	 * Protects accesses to the following fields when the MMU lock
 	 * is held in read mode:
+	 *  - tdp_mmu_roots (above)
 	 *  - tdp_mmu_pages (above)
 	 *  - the link field of struct kvm_mmu_pages used by the TDP MMU
 	 *  - lpage_disallowed_mmu_pages
arch/x86/kvm/mmu/tdp_mmu.c +39 -30

117 lines not on this page · raw message

Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#10 lore raw

PATCH v2 9/13 KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock

In reply to Ben Gardon

To reduce lock contention and interference with page fault handlers, allow the TDP MMU function to zap a GFN range to operate under the MMU read lock.

---
 arch/x86/kvm/mmu/mmu.c     |  22 +++++---
 arch/x86/kvm/mmu/tdp_mmu.c | 111 ++++++++++++++++++++++++++-----------
 arch/x86/kvm/mmu/tdp_mmu.h |  14 +++--
 3 files changed, 102 insertions(+), 45 deletions(-)
arch/x86/kvm/mmu/mmu.c +15 -7
index 47d996a8074f..d03a7a8b7ea2 100644
@@ -3154,7 +3154,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
 	sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
 
 	if (is_tdp_mmu_page(sp))
-		kvm_tdp_mmu_put_root(kvm, sp);
+		kvm_tdp_mmu_put_root(kvm, sp, false);
 	else if (!--sp->root_count && sp->role.invalid)
 		kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 
@@ -5507,16 +5507,24 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 		}
 	}
 
-	if (is_tdp_mmu_enabled(kvm)) {
-		for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
-			flush = kvm_tdp_mmu_zap_gfn_range(kvm, i, gfn_start,
-							  gfn_end, flush);
-	}
-
 	if (flush)
 		kvm_flush_remote_tlbs_with_address(kvm, gfn_start, gfn_end);
 
 	write_unlock(&kvm->mmu_lock);
+
+	if (is_tdp_mmu_enabled(kvm)) {
+		flush = false;
+
+		read_lock(&kvm->mmu_lock);
+		for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
+			flush = kvm_tdp_mmu_zap_gfn_range(kvm, i, gfn_start,
+							  gfn_end, flush, true);
+		if (flush)
+			kvm_flush_remote_tlbs_with_address(kvm, gfn_start,
+							   gfn_end);
+
+		read_unlock(&kvm->mmu_lock);
+	}
 }
 
 static bool slot_rmap_write_protect(struct kvm *kvm,
arch/x86/kvm/mmu/tdp_mmu.c +78 -33

262 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +9 -5
index f0a26214e999..d703c6d6024a 100644
@@ -13,14 +13,18 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,
 	return refcount_inc_not_zero(&root->tdp_mmu_root_count);
 }
 
-void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
+			  bool shared);
 
 bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, gfn_t start,
-				 gfn_t end, bool can_yield, bool flush);
+				 gfn_t end, bool can_yield, bool flush,
+				 bool shared);
 static inline bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id,
-					     gfn_t start, gfn_t end, bool flush)
+					     gfn_t start, gfn_t end, bool flush,
+					     bool shared)
 {
-	return __kvm_tdp_mmu_zap_gfn_range(kvm, as_id, start, end, true, flush);
+	return __kvm_tdp_mmu_zap_gfn_range(kvm, as_id, start, end, true, flush,
+					   shared);
 }
 static inline bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
@@ -37,7 +41,7 @@ static inline bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 	 */
 	lockdep_assert_held_write(&kvm->mmu_lock);
 	return __kvm_tdp_mmu_zap_gfn_range(kvm, kvm_mmu_page_as_id(sp),
-					   sp->gfn, end, false, false);
+					   sp->gfn, end, false, false, false);
 }
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 
Signature · 1 line
2.31.0.208.g409f899ff0-goog
maintainer
#11 lore raw

Re: [PATCH v2 09/13] KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock

Quoted 11 lines · Ben Gardon
arch/x86/kvm/mmu/tdp_mmu.c +3 -1
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
+			  bool shared)
  {
  	gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
 
-	lockdep_assert_held_write(&kvm->mmu_lock);
+	kvm_lockdep_assert_mmu_lock_held(kvm, shared);
 
  	if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
  		return;

arch/x86/kvm/mmu/tdp_mmu.c void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) · line 97

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
@@ -81,7 +92,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
  	list_del_rcu(&root->link);
  	spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 
-	zap_gfn_range(kvm, root, 0, max_gfn, false, false);
+	zap_gfn_range(kvm, root, 0, max_gfn, false, false, shared);
 
  	call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback);

Instead of patch 13, would it make sense to delay the zap_gfn_range and call_rcu to a work item (either unconditionally, or only if shared==false)? Then the zap_gfn_range would be able to yield and take the mmu_lock for read, similar to kvm_tdp_mmu_zap_invalidated_roots.

If done unconditionally, this would also allow removing the "shared" argument to kvm_tdp_mmu_put_root, tdp_mmu_next_root and for_each_tdp_mmu_root_yield_safe, so I would place that change before this patch.

Paolo

author
#12 lore raw

Re: [PATCH v2 09/13] KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock

Quoted 26 lines · Paolo Bonzini
Quoted 11 lines · Ben Gardon
arch/x86/kvm/mmu/tdp_mmu.c +3 -1
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
+                       bool shared)
  {
      gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
 
-     lockdep_assert_held_write(&kvm->mmu_lock);
+     kvm_lockdep_assert_mmu_lock_held(kvm, shared);
 
      if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
              return;

arch/x86/kvm/mmu/tdp_mmu.c void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) · line 97

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
@@ -81,7 +92,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
      list_del_rcu(&root->link);
      spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 
-     zap_gfn_range(kvm, root, 0, max_gfn, false, false);
+     zap_gfn_range(kvm, root, 0, max_gfn, false, false, shared);
 
      call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback);

Instead of patch 13, would it make sense to delay the zap_gfn_range and call_rcu to a work item (either unconditionally, or only if shared==false)? Then the zap_gfn_range would be able to yield and take the mmu_lock for read, similar to kvm_tdp_mmu_zap_invalidated_roots.

Quoted 7 lines

If done unconditionally, this would also allow removing the "shared" argument to kvm_tdp_mmu_put_root, tdp_mmu_next_root and for_each_tdp_mmu_root_yield_safe, so I would place that change before this patch.

Paolo

I tried that and it created problems. I believe the issue was that on VM teardown memslots would be freed and the memory reallocated before the root was torn down, resulting in a use-after free from mark_pfn_dirty. Perhaps this could be resolved by forcing memslot changes to wait until that work item was processed before returning. I can look into it but I suspect there will be a lot of "gotchas" involved.

author
#13 lore raw

PATCH v2 10/13 KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock

In reply to Ben Gardon

To speed the process of disabling dirty logging, change the TDP MMU function which zaps collapsible SPTEs to run under the MMU read lock.

---
 arch/x86/kvm/mmu/mmu.c     | 14 ++++++++++----
 arch/x86/kvm/mmu/tdp_mmu.c | 17 +++++++++++++----
 2 files changed, 23 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +10 -4
index d03a7a8b7ea2..5939813e3043 100644
@@ -5612,13 +5612,19 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
 	write_lock(&kvm->mmu_lock);
 	flush = slot_handle_leaf(kvm, slot, kvm_mmu_zap_collapsible_spte, true);
 
-	if (is_tdp_mmu_enabled(kvm))
-		flush = kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot, flush);
-
 	if (flush)
 		kvm_arch_flush_remote_tlbs_memslot(kvm, slot);
-
 	write_unlock(&kvm->mmu_lock);
+
+	if (is_tdp_mmu_enabled(kvm)) {
+		flush = false;
+
+		read_lock(&kvm->mmu_lock);
+		flush = kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot, flush);
+		if (flush)
+			kvm_arch_flush_remote_tlbs_memslot(kvm, slot);
+		read_unlock(&kvm->mmu_lock);
+	}
 }
 
 void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
arch/x86/kvm/mmu/tdp_mmu.c +13 -4
index 6917403484ce..0e6ffa04e5e1 100644
@@ -1323,7 +1323,8 @@ static bool zap_collapsible_spte_range(struct kvm *kvm,
 	rcu_read_lock();
 
 	tdp_root_for_each_pte(iter, root, start, end) {
-		if (tdp_mmu_iter_cond_resched(kvm, &iter, flush, false)) {
+retry:
+		if (tdp_mmu_iter_cond_resched(kvm, &iter, flush, true)) {
 			flush = false;
 			continue;
 		}
@@ -1338,8 +1339,14 @@ static bool zap_collapsible_spte_range(struct kvm *kvm,
 							    pfn, PG_LEVEL_NUM))
 			continue;
 
-		tdp_mmu_set_spte(kvm, &iter, 0);
-
+		if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) {
+			/*
+			 * The iter must explicitly re-read the SPTE because
+			 * the atomic cmpxchg failed.
+			 */
+			iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
+			goto retry;
+		}
 		flush = true;
 	}
 
@@ -1358,7 +1365,9 @@ bool kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
 {
 	struct kvm_mmu_page *root;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root, slot->as_id, false)
+	lockdep_assert_held_read(&kvm->mmu_lock);
+
+	for_each_tdp_mmu_root_yield_safe(kvm, root, slot->as_id, true)
 		flush = zap_collapsible_spte_range(kvm, root, slot, flush);
 
 	return flush;
Signature · 1 line
2.31.0.208.g409f899ff0-goog
maintainer
#14 lore raw

Re: [PATCH v2 10/13] KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock

Quoted 2 lines · Ben Gardon

To speed the process of disabling dirty logging, change the TDP MMU function which zaps collapsible SPTEs to run under the MMU read lock.

Technically it only reduces the impact on the running VM; it doesn't speed it up right? Something like:

To reduce the impact of disabling dirty logging, change the TDP MMU function which zaps collapsible SPTEs to run under the MMU read lock. This way, page faults on zapped SPTEs can proceed in parallel with kvm_mmu_zap_collapsible_sptes.

Paolo

author
#15 lore raw

PATCH v2 11/13 KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read lock

In reply to Ben Gardon

To reduce lock contention and interference with page fault handlers, allow the TDP MMU functions which enable and disable dirty logging to operate under the MMU read lock.

---
 arch/x86/kvm/mmu/mmu.c     | 16 +++++++---
 arch/x86/kvm/mmu/tdp_mmu.c | 62 ++++++++++++++++++++++++++++++--------
 2 files changed, 61 insertions(+), 17 deletions(-)
arch/x86/kvm/mmu/mmu.c +12 -4
index 5939813e3043..a3837f8ad4ed 100644
@@ -5543,10 +5543,14 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm,
 	write_lock(&kvm->mmu_lock);
 	flush = slot_handle_level(kvm, memslot, slot_rmap_write_protect,
 				start_level, KVM_MAX_HUGEPAGE_LEVEL, false);
-	if (is_tdp_mmu_enabled(kvm))
-		flush |= kvm_tdp_mmu_wrprot_slot(kvm, memslot, PG_LEVEL_4K);
 	write_unlock(&kvm->mmu_lock);
 
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		flush |= kvm_tdp_mmu_wrprot_slot(kvm, memslot, PG_LEVEL_4K);
+		read_unlock(&kvm->mmu_lock);
+	}
+
 	/*
 	 * We can flush all the TLBs out of the mmu lock without TLB
 	 * corruption since we just change the spte from writable to
@@ -5649,10 +5653,14 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
 
 	write_lock(&kvm->mmu_lock);
 	flush = slot_handle_leaf(kvm, memslot, __rmap_clear_dirty, false);
-	if (is_tdp_mmu_enabled(kvm))
-		flush |= kvm_tdp_mmu_clear_dirty_slot(kvm, memslot);
 	write_unlock(&kvm->mmu_lock);
 
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		flush |= kvm_tdp_mmu_clear_dirty_slot(kvm, memslot);
+		read_unlock(&kvm->mmu_lock);
+	}
+
 	/*
 	 * It's also safe to flush TLBs out of mmu lock here as currently this
 	 * function is only used for dirty logging, in which case flushing TLB
arch/x86/kvm/mmu/tdp_mmu.c +49 -13

120 lines not on this page · raw message

Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#16 lore raw

PATCH v2 12/13 KVM: x86/mmu: Fast invalidation for TDP MMU

In reply to Ben Gardon

Provide a real mechanism for fast invalidation by marking roots as invalid so that their reference count will quickly fall to zero and they will be torn down.

One negative side affect of this approach is that a vCPU thread will likely drop the last reference to a root and be saddled with the work of tearing down an entire paging structure. This issue will be resolved in a later commit.

---

Changelog
v2:
-- open code root invalidation

 arch/x86/kvm/mmu/mmu.c     | 26 +++++++++++++++++++++++---
 arch/x86/kvm/mmu/tdp_mmu.h |  3 +++
 2 files changed, 26 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +23 -3
index a3837f8ad4ed..ba0c65076200 100644
@@ -5418,6 +5418,8 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm)
  */
 static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 {
+	struct kvm_mmu_page *root;
+
 	lockdep_assert_held(&kvm->slots_lock);
 
 	write_lock(&kvm->mmu_lock);
@@ -5432,6 +5434,27 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 	 */
 	kvm->arch.mmu_valid_gen = kvm->arch.mmu_valid_gen ? 0 : 1;
 
+
+	if (is_tdp_mmu_enabled(kvm)) {
+		/*
+		 * Mark each TDP MMU root as invalid so that other threads
+		 * will drop their references and allow the root count to
+		 * go to 0.
+		 *
+		 * This has essentially the same effect for the TDP MMU
+		 * as updating mmu_valid_gen above does for the shadow
+		 * MMU.
+		 *
+		 * In order to ensure all threads see this change when
+		 * handling the MMU reload signal, this must happen in the
+		 * same critical section as kvm_reload_remote_mmus, and
+		 * before kvm_zap_obsolete_pages as kvm_zap_obsolete_pages
+		 * could drop the MMU lock and yield.
+		 */
+		list_for_each_entry(root, &kvm->arch.tdp_mmu_roots, link)
+			root->role.invalid = true;
+	}
+
 	/*
 	 * Notify all vcpus to reload its shadow page table and flush TLB.
 	 * Then all vcpus will switch to new shadow page table with the new
@@ -5444,9 +5467,6 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 
 	kvm_zap_obsolete_pages(kvm);
 
-	if (is_tdp_mmu_enabled(kvm))
-		kvm_tdp_mmu_zap_all(kvm);
-
 	write_unlock(&kvm->mmu_lock);
 }
 
arch/x86/kvm/mmu/tdp_mmu.h +3 -0
index d703c6d6024a..8fa3e7421a93 100644
@@ -10,6 +10,9 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
 __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,
 						     struct kvm_mmu_page *root)
 {
+	if (root->role.invalid)
+		return false;
+
 	return refcount_inc_not_zero(&root->tdp_mmu_root_count);
 }
 
Signature · 1 line
2.31.0.208.g409f899ff0-goog
author
#17 lore raw

PATCH v2 13/13 KVM: x86/mmu: Tear down roots in fast invalidation thread

In reply to Ben Gardon

To avoid saddling a vCPU thread with the work of tearing down an entire paging structure, take a reference on each root before they become obsolete, so that the thread initiating the fast invalidation can tear down the paging structure and (most likely) release the last reference. As a bonus, this teardown can happen under the MMU lock in read mode so as not to block the progress of vCPU threads.

---

Changelog
v2:
-- rename kvm_tdp_mmu_zap_all_fast to
kvm_tdp_mmu_zap_invalidated_roots

 arch/x86/kvm/mmu/mmu.c     | 21 +++++++++++-
 arch/x86/kvm/mmu/tdp_mmu.c | 68 ++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/mmu/tdp_mmu.h |  1 +
 3 files changed, 89 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +20 -1
index ba0c65076200..5f2064ee7220 100644
@@ -5441,6 +5441,18 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 		 * will drop their references and allow the root count to
 		 * go to 0.
 		 *
+		 * Also take a reference on all roots so that this thread
+		 * can do the bulk of the work required to free the roots
+		 * once they are invalidated. Without this reference, a
+		 * vCPU thread might drop the last reference to a root and
+		 * get stuck with tearing down the entire paging structure.
+		 *
+		 * Roots which have a zero refcount should be skipped as
+		 * they're already being torn down.
+		 * Already invalid roots should be referenced again so that
+		 * they aren't freed before kvm_tdp_mmu_zap_all_fast is
+		 * done with them.
+		 *
 		 * This has essentially the same effect for the TDP MMU
 		 * as updating mmu_valid_gen above does for the shadow
 		 * MMU.
@@ -5452,7 +5464,8 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 		 * could drop the MMU lock and yield.
 		 */
 		list_for_each_entry(root, &kvm->arch.tdp_mmu_roots, link)
-			root->role.invalid = true;
+			if (refcount_inc_not_zero(&root->tdp_mmu_root_count))
+				root->role.invalid = true;
 	}
 
 	/*
@@ -5468,6 +5481,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 	kvm_zap_obsolete_pages(kvm);
 
 	write_unlock(&kvm->mmu_lock);
+
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		kvm_tdp_mmu_zap_invalidated_roots(kvm);
+		read_unlock(&kvm->mmu_lock);
+	}
 }
 
 static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
arch/x86/kvm/mmu/tdp_mmu.c +68 -0

74 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +1 -0
index 8fa3e7421a93..f8db381e3059 100644
@@ -47,6 +47,7 @@ static inline bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 					   sp->gfn, end, false, false, false);
 }
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
+void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm);
 
 int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
 		    int map_writable, int max_level, kvm_pfn_t pfn,
Signature · 1 line
2.31.0.208.g409f899ff0-goog
maintainer Applied
#18 lore raw
In reply to Ben Gardon
Quoted 44 lines · Ben Gardon

Now that the TDP MMU is able to handle page faults in parallel, it's a relatively small change to expand to other operations. This series allows zapping a range of GFNs, reclaiming collapsible SPTEs (when disabling dirty logging), and enabling dirty logging to all happen under the MMU lock in read mode.

This is partly a cleanup + rewrite of the last few patches of the parallel page faults series. I've incorporated feedback from Sean and Paolo, but the patches have changed so much that I'm sending this as a separate series.

Ran kvm-unit-tests + selftests on an SMP kernel + Intel Skylake, with the TDP MMU enabled and disabled. This series introduces no new failures or warnings.

I know this will conflict horribly with the patches from Sean's series which were just queued, and I'll send a v2 to fix those conflicts + address any feedback on this v1.

Changelog
v2:
-- Rebased patches on top of kvm/queue to incorporate Sean's recent
TLB flushing changes
-- Dropped patch 5: "KVM: x86/mmu: comment for_each_tdp_mmu_root
requires MMU write lock" as the following patch to protect the roots list with RCU adds lockdep which makes the comment somewhat redundant.

Ben Gardon (13):
KVM: x86/mmu: Re-add const qualifier in
kvm_tdp_mmu_zap_collapsible_sptes
KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU
KVM: x86/mmu: use tdp_mmu_free_sp to free roots
KVM: x86/mmu: Merge TDP MMU put and free root
KVM: x86/mmu: Refactor yield safe root iterator
KVM: x86/mmu: Make TDP MMU root refcount atomic
KVM: x86/mmu: handle cmpxchg failure in kvm_tdp_mmu_get_root
KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU
KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read
lock
KVM: x86/mmu: Fast invalidation for TDP MMU
KVM: x86/mmu: Tear down roots in fast invalidation thread

Quoted 8 lines
  arch/x86/include/asm/kvm_host.h |  21 +-
  arch/x86/kvm/mmu/mmu.c          | 115 +++++++---
  arch/x86/kvm/mmu/mmu_internal.h |  27 +--
  arch/x86/kvm/mmu/tdp_mmu.c      | 375 +++++++++++++++++++++++---------
  arch/x86/kvm/mmu/tdp_mmu.h      |  28 ++-
  include/linux/kvm_host.h        |   2 +-
  6 files changed, 407 insertions(+), 161 deletions(-)

Applied to kvm/mmu-notifier-queue, thanks.

Paolo

#19 lore raw
Quoted 54 lines · Paolo Bonzini
Quoted 44 lines · Ben Gardon

Now that the TDP MMU is able to handle page faults in parallel, it's a relatively small change to expand to other operations. This series allows zapping a range of GFNs, reclaiming collapsible SPTEs (when disabling dirty logging), and enabling dirty logging to all happen under the MMU lock in read mode.

This is partly a cleanup + rewrite of the last few patches of the parallel page faults series. I've incorporated feedback from Sean and Paolo, but the patches have changed so much that I'm sending this as a separate series.

Ran kvm-unit-tests + selftests on an SMP kernel + Intel Skylake, with the TDP MMU enabled and disabled. This series introduces no new failures or warnings.

I know this will conflict horribly with the patches from Sean's series which were just queued, and I'll send a v2 to fix those conflicts + address any feedback on this v1.

Changelog
v2:
-- Rebased patches on top of kvm/queue to incorporate Sean's recent
TLB flushing changes
-- Dropped patch 5: "KVM: x86/mmu: comment for_each_tdp_mmu_root
requires MMU write lock" as the following patch to protect the roots list with RCU adds lockdep which makes the comment somewhat redundant.

Ben Gardon (13):
KVM: x86/mmu: Re-add const qualifier in
kvm_tdp_mmu_zap_collapsible_sptes
KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU
KVM: x86/mmu: use tdp_mmu_free_sp to free roots
KVM: x86/mmu: Merge TDP MMU put and free root
KVM: x86/mmu: Refactor yield safe root iterator
KVM: x86/mmu: Make TDP MMU root refcount atomic
KVM: x86/mmu: handle cmpxchg failure in kvm_tdp_mmu_get_root
KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU
KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read
lock
KVM: x86/mmu: Fast invalidation for TDP MMU
KVM: x86/mmu: Tear down roots in fast invalidation thread

Quoted 8 lines
  arch/x86/include/asm/kvm_host.h |  21 +-
  arch/x86/kvm/mmu/mmu.c          | 115 +++++++---
  arch/x86/kvm/mmu/mmu_internal.h |  27 +--
  arch/x86/kvm/mmu/tdp_mmu.c      | 375 +++++++++++++++++++++++---------
  arch/x86/kvm/mmu/tdp_mmu.h      |  28 ++-
  include/linux/kvm_host.h        |   2 +-
  6 files changed, 407 insertions(+), 161 deletions(-)
Quoted 1 line

Applied to kvm/mmu-notifier-queue, thanks.

What's the plan for kvm/mmu-notifier-queue? More specifically, are the hashes stable, i.e. will non-critical review feedback get squashed? I was finally getting around to reviewing this, but what's sitting in that branch doesn't appear to be exactly what's posted here. If the hashes are stable, I'll probably test and review functionality, but not do a thorough review.

Thanks!

maintainer
#20 lore raw
Quoted 6 lines · Sean Christopherson
Quoted 1 line

Applied to kvm/mmu-notifier-queue, thanks.

What's the plan for kvm/mmu-notifier-queue? More specifically, are the hashes stable, i.e. will non-critical review feedback get squashed? I was finally getting around to reviewing this, but what's sitting in that branch doesn't appear to be exactly what's posted here. If the hashes are stable, I'll probably test and review functionality, but not do a thorough review.

It's all in 5.13 except for the lock elision patch, for which I was waiting for a review. I'll post that patch separately.

Thanks,

Paolo

#21 lore raw
Quoted 10 lines · Paolo Bonzini
Quoted 6 lines · Sean Christopherson
Quoted 1 line

Applied to kvm/mmu-notifier-queue, thanks.

What's the plan for kvm/mmu-notifier-queue? More specifically, are the hashes stable, i.e. will non-critical review feedback get squashed? I was finally getting around to reviewing this, but what's sitting in that branch doesn't appear to be exactly what's posted here. If the hashes are stable, I'll probably test and review functionality, but not do a thorough review.

It's all in 5.13 except for the lock elision patch, for which I was waiting for a review. I'll post that patch separately.

Ha, stable indeed. Thanks!