Skip to contentSkip to conversation

More parallel operations for the TDP MMU

PATCH Superseded Ben Gardon 4 participants 32 messages Mar 31, 2021 – Apr 1, 2021

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Ben Gardon

author
#0 lore raw

cover More parallel operations for the TDP MMU

Now that the TDP MMU is able to handle page faults in parallel, it's a relatively small change to expand to other operations. This series allows zapping a range of GFNs, reclaiming collapsible SPTEs (when disabling dirty logging), and enabling dirty logging to all happen under the MMU lock in read mode.

This is partly a cleanup + rewrite of the last few patches of the parallel page faults series. I've incorporated feedback from Sean and Paolo, but the patches have changed so much that I'm sending this as a separate series.

Ran kvm-unit-tests + selftests on an SMP kernel + Intel Skylake, with the TDP MMU enabled and disabled. This series introduces no new failures or warnings.

I know this will conflict horribly with the patches from Sean's series which were just queued, and I'll send a v2 to fix those conflicts + address any feedback on this v1.

Ben Gardon (13):
KVM: x86/mmu: Re-add const qualifier in
kvm_tdp_mmu_zap_collapsible_sptes
KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU
KVM: x86/mmu: use tdp_mmu_free_sp to free roots
KVM: x86/mmu: Merge TDP MMU put and free root
KVM: x86/mmu: comment for_each_tdp_mmu_root requires MMU write lock
KVM: x86/mmu: Refactor yield safe root iterator
KVM: x86/mmu: Make TDP MMU root refcount atomic
KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU
KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read
lock
KVM: x86/mmu: Fast invalidation for TDP MMU
KVM: x86/mmu: Tear down roots in fast invalidation thread

 arch/x86/kvm/mmu/mmu.c          |  70 +++---
 arch/x86/kvm/mmu/mmu_internal.h |  27 +--
 arch/x86/kvm/mmu/tdp_mmu.c      | 383 ++++++++++++++++++++++++--------
 arch/x86/kvm/mmu/tdp_mmu.h      |  21 +-
 include/linux/kvm_host.h        |   2 +-
 5 files changed, 357 insertions(+), 146 deletions(-)
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
author
#1 lore raw

PATCH 1/13 KVM: x86/mmu: Re-add const qualifier in kvm_tdp_mmu_zap_collapsible_sptes

kvm_tdp_mmu_zap_collapsible_sptes unnecessarily removes the const qualifier from its memlsot argument, leading to a compiler warning. Add the const annotation and pass it to subsequent functions.

---
 arch/x86/kvm/mmu/mmu.c          | 10 +++++-----
 arch/x86/kvm/mmu/mmu_internal.h |  5 +++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  4 ++--
 arch/x86/kvm/mmu/tdp_mmu.h      |  2 +-
 include/linux/kvm_host.h        |  2 +-
 5 files changed, 12 insertions(+), 11 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -5
index c6ed633594a2..f75cbb0fcc9c 100644
@@ -715,8 +715,7 @@ static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn)
  * handling slots that are not large page aligned.
  */
 static struct kvm_lpage_info *lpage_info_slot(gfn_t gfn,
-					      struct kvm_memory_slot *slot,
-					      int level)
+		const struct kvm_memory_slot *slot, int level)
 {
 	unsigned long idx;
 
@@ -2736,7 +2735,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
 }
 
 static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-				  struct kvm_memory_slot *slot)
+				  const struct kvm_memory_slot *slot)
 {
 	unsigned long hva;
 	pte_t *pte;
@@ -2762,8 +2761,9 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	return level;
 }
 
-int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_memory_slot *slot,
-			      gfn_t gfn, kvm_pfn_t pfn, int max_level)
+int kvm_mmu_max_mapping_level(struct kvm *kvm,
+			      const struct kvm_memory_slot *slot, gfn_t gfn,
+			      kvm_pfn_t pfn, int max_level)
 {
 	struct kvm_lpage_info *linfo;
 
arch/x86/kvm/mmu/mmu_internal.h +3 -2
index e03267e93459..fc88f62d7bd9 100644
@@ -156,8 +156,9 @@ enum {
 #define SET_SPTE_NEED_REMOTE_TLB_FLUSH	BIT(1)
 #define SET_SPTE_SPURIOUS		BIT(2)
 
-int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_memory_slot *slot,
-			      gfn_t gfn, kvm_pfn_t pfn, int max_level);
+int kvm_mmu_max_mapping_level(struct kvm *kvm,
+			      const struct kvm_memory_slot *slot, gfn_t gfn,
+			      kvm_pfn_t pfn, int max_level);
 int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn,
 			    int max_level, kvm_pfn_t *pfnp,
 			    bool huge_page_disallowed, int *req_level);
arch/x86/kvm/mmu/tdp_mmu.c +2 -2
index f2c335854afb..6d4f4e305163 100644
@@ -1268,7 +1268,7 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm,
  */
 static void zap_collapsible_spte_range(struct kvm *kvm,
 				       struct kvm_mmu_page *root,
-				       struct kvm_memory_slot *slot)
+				       const struct kvm_memory_slot *slot)
 {
 	gfn_t start = slot->base_gfn;
 	gfn_t end = start + slot->npages;
@@ -1309,7 +1309,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
  * be replaced by large mappings, for GFNs within the slot.
  */
 void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
-				       struct kvm_memory_slot *slot)
+				       const struct kvm_memory_slot *slot)
 {
 	struct kvm_mmu_page *root;
 	int root_as_id;
arch/x86/kvm/mmu/tdp_mmu.h +1 -1
index 3b761c111bff..683d1d69c8c8 100644
@@ -34,7 +34,7 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm,
 				       gfn_t gfn, unsigned long mask,
 				       bool wrprot);
 void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
-				       struct kvm_memory_slot *slot);
+				       const struct kvm_memory_slot *slot);
 
 bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm,
 				   struct kvm_memory_slot *slot, gfn_t gfn);
include/linux/kvm_host.h +1 -1
index 1b65e7204344..74e56e8673a6 100644
@@ -1116,7 +1116,7 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn)
 }
 
 static inline unsigned long
-__gfn_to_hva_memslot(struct kvm_memory_slot *slot, gfn_t gfn)
+__gfn_to_hva_memslot(const struct kvm_memory_slot *slot, gfn_t gfn)
 {
 	return slot->userspace_addr + (gfn - slot->base_gfn) * PAGE_SIZE;
 }
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
author
#2 lore raw

PATCH 2/13 KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU

In reply to Ben Gardon

The TDP MMU is almost the only user of kvm_mmu_get_root and kvm_mmu_put_root. There is only one use of put_root in mmu.c for the legacy / shadow MMU. Open code that one use and move the get / put functions to the TDP MMU so they can be extended in future commits.

No functional change intended.

---
 arch/x86/kvm/mmu/mmu.c          | 10 ++++------
 arch/x86/kvm/mmu/mmu_internal.h | 16 ----------------
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 +++---
 arch/x86/kvm/mmu/tdp_mmu.h      | 18 ++++++++++++++++++
 4 files changed, 25 insertions(+), 25 deletions(-)
arch/x86/kvm/mmu/mmu.c +4 -6
index f75cbb0fcc9c..618cc011f446 100644
@@ -3154,12 +3154,10 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
 
 	sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
 
-	if (kvm_mmu_put_root(kvm, sp)) {
-		if (is_tdp_mmu_page(sp))
-			kvm_tdp_mmu_free_root(kvm, sp);
-		else if (sp->role.invalid)
-			kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
-	}
+	if (is_tdp_mmu_page(sp) && kvm_tdp_mmu_put_root(kvm, sp))
+		kvm_tdp_mmu_free_root(kvm, sp);
+	else if (!--sp->root_count && sp->role.invalid)
+		kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 
 	*root_hpa = INVALID_PAGE;
 }
arch/x86/kvm/mmu/mmu_internal.h +0 -16
index fc88f62d7bd9..788dcf77c957 100644
@@ -118,22 +118,6 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
 void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 					u64 start_gfn, u64 pages);
 
-static inline void kvm_mmu_get_root(struct kvm *kvm, struct kvm_mmu_page *sp)
-{
-	BUG_ON(!sp->root_count);
-	lockdep_assert_held(&kvm->mmu_lock);
-
-	++sp->root_count;
-}
-
-static inline bool kvm_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *sp)
-{
-	lockdep_assert_held(&kvm->mmu_lock);
-	--sp->root_count;
-
-	return !sp->root_count;
-}
-
 /*
  * Return values of handle_mmio_page_fault, mmu.page_fault, and fast_page_fault().
  *
arch/x86/kvm/mmu/tdp_mmu.c +3 -3
index 6d4f4e305163..1929cc7a42ac 100644
@@ -43,7 +43,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
 
 static void tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
 {
-	if (kvm_mmu_put_root(kvm, root))
+	if (kvm_tdp_mmu_put_root(kvm, root))
 		kvm_tdp_mmu_free_root(kvm, root);
 }
 
@@ -55,7 +55,7 @@ static inline bool tdp_mmu_next_root_valid(struct kvm *kvm,
 	if (list_entry_is_head(root, &kvm->arch.tdp_mmu_roots, link))
 		return false;
 
-	kvm_mmu_get_root(kvm, root);
+	kvm_tdp_mmu_get_root(kvm, root);
 	return true;
 
 }
@@ -150,7 +150,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 	/* Check for an existing root before allocating a new one. */
 	for_each_tdp_mmu_root(kvm, root) {
 		if (root->role.word == role.word) {
-			kvm_mmu_get_root(kvm, root);
+			kvm_tdp_mmu_get_root(kvm, root);
 			goto out;
 		}
 	}
arch/x86/kvm/mmu/tdp_mmu.h +18 -0
index 683d1d69c8c8..2dc3b3ba48fb 100644
@@ -8,6 +8,24 @@
 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
 void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root);
 
+static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
+					struct kvm_mmu_page *root)
+{
+	BUG_ON(!root->root_count);
+	lockdep_assert_held(&kvm->mmu_lock);
+
+	++root->root_count;
+}
+
+static inline bool kvm_tdp_mmu_put_root(struct kvm *kvm,
+					struct kvm_mmu_page *root)
+{
+	lockdep_assert_held(&kvm->mmu_lock);
+	--root->root_count;
+
+	return !root->root_count;
+}
+
 bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end);
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
author
#3 lore raw

PATCH 3/13 KVM: x86/mmu: use tdp_mmu_free_sp to free roots

In reply to Ben Gardon

Minor cleanup to deduplicate the code used to free a struct kvm_mmu_page in the TDP MMU.

No functional change intended.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +7 -8
index 1929cc7a42ac..5a2698d64957 100644
@@ -88,6 +88,12 @@ static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
 			  gfn_t start, gfn_t end, bool can_yield);
 
+static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
+{
+	free_page((unsigned long)sp->spt);
+	kmem_cache_free(mmu_page_header_cache, sp);
+}
+
 void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root)
 {
 	gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
@@ -101,8 +107,7 @@ void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root)
 
 	zap_gfn_range(kvm, root, 0, max_gfn, false);
 
-	free_page((unsigned long)root->spt);
-	kmem_cache_free(mmu_page_header_cache, root);
+	tdp_mmu_free_sp(root);
 }
 
 static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu,
@@ -164,12 +169,6 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 	return __pa(root->spt);
 }
 
-static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
-{
-	free_page((unsigned long)sp->spt);
-	kmem_cache_free(mmu_page_header_cache, sp);
-}
-
 /*
  * This is called through call_rcu in order to free TDP page table memory
  * safely with respect to other kernel threads that may be operating on
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
author
#4 lore raw

PATCH 4/13 KVM: x86/mmu: Merge TDP MMU put and free root

In reply to Ben Gardon

kvm_tdp_mmu_put_root and kvm_tdp_mmu_free_root are always called together, so merge the functions to simplify TDP MMU root refcounting / freeing.

---
 arch/x86/kvm/mmu/mmu.c     |  4 +--
 arch/x86/kvm/mmu/tdp_mmu.c | 54 ++++++++++++++++++--------------------
 arch/x86/kvm/mmu/tdp_mmu.h | 10 +------
 3 files changed, 28 insertions(+), 40 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -2
index 618cc011f446..667d64daa82c 100644
@@ -3154,8 +3154,8 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
 
 	sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
 
-	if (is_tdp_mmu_page(sp) && kvm_tdp_mmu_put_root(kvm, sp))
-		kvm_tdp_mmu_free_root(kvm, sp);
+	if (is_tdp_mmu_page(sp))
+		kvm_tdp_mmu_put_root(kvm, sp);
 	else if (!--sp->root_count && sp->role.invalid)
 		kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 
arch/x86/kvm/mmu/tdp_mmu.c +25 -29

73 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +1 -9
index 2dc3b3ba48fb..5d950e987fc7 100644
@@ -6,7 +6,6 @@
 #include <linux/kvm_host.h>
 
 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
-void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root);
 
 static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
 					struct kvm_mmu_page *root)
@@ -17,14 +16,7 @@ static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
 	++root->root_count;
 }
 
-static inline bool kvm_tdp_mmu_put_root(struct kvm *kvm,
-					struct kvm_mmu_page *root)
-{
-	lockdep_assert_held(&kvm->mmu_lock);
-	--root->root_count;
-
-	return !root->root_count;
-}
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
 
 bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end);
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
author
#5 lore raw

PATCH 5/13 KVM: x86/mmu: comment for_each_tdp_mmu_root requires MMU write lock

In reply to Ben Gardon

Currently, iterating over the list of TDP MMU roots can only be done under the MMU write lock, but that will change in future commits. Add a defensive comment to for_each_tdp_mmu_root noting that it must only be used under the MMU lock in write mode. That function will not be modified to work under the lock in read mode.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
index 368091adab09..365fa9f2f856 100644
@@ -103,6 +103,7 @@ static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 	     tdp_mmu_next_root_valid(_kvm, _root);			\
 	     _root = tdp_mmu_next_root(_kvm, _root))
 
+/* Only safe under the MMU lock in write mode, without yielding. */
 #define for_each_tdp_mmu_root(_kvm, _root)				\
 	list_for_each_entry(_root, &_kvm->arch.tdp_mmu_roots, link)
 
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
author
#6 lore raw

PATCH 6/13 KVM: x86/mmu: Refactor yield safe root iterator

In reply to Ben Gardon

Refactor the yield safe TDP MMU root iterator to be more amenable to changes in future commits which will allow it to be used under the MMU lock in read mode. Currently the iterator requires a complicated dance between the helper functions and different parts of the for loop which makes it hard to reason about. Moving all the logic into a single function simplifies the iterator substantially.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 43 ++++++++++++++++++++++----------------
 1 file changed, 25 insertions(+), 18 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +25 -18
index 365fa9f2f856..ab1d26b40164 100644
@@ -68,26 +68,34 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
 	tdp_mmu_free_sp(root);
 }
 
-static inline bool tdp_mmu_next_root_valid(struct kvm *kvm,
-					   struct kvm_mmu_page *root)
+/*
+ * Finds the next valid root after root (or the first valid root if root
+ * is NULL), takes a reference on it, and returns that next root. If root
+ * is not NULL, this thread should have already taken a reference on it, and
+ * that reference will be dropped. If no valid root is found, this
+ * function will return NULL.
+ */
+static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
+					      struct kvm_mmu_page *prev_root)
 {
-	lockdep_assert_held_write(&kvm->mmu_lock);
+	struct kvm_mmu_page *next_root;
 
-	if (list_entry_is_head(root, &kvm->arch.tdp_mmu_roots, link))
-		return false;
+	lockdep_assert_held_write(&kvm->mmu_lock);
 
-	kvm_tdp_mmu_get_root(kvm, root);
-	return true;
+	if (prev_root)
+		next_root = list_next_entry(prev_root, link);
+	else
+		next_root = list_first_entry(&kvm->arch.tdp_mmu_roots,
+					     typeof(*next_root), link);
 
-}
+	if (list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link))
+		next_root = NULL;
+	else
+		kvm_tdp_mmu_get_root(kvm, next_root);
 
-static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
-						     struct kvm_mmu_page *root)
-{
-	struct kvm_mmu_page *next_root;
+	if (prev_root)
+		kvm_tdp_mmu_put_root(kvm, prev_root);
 
-	next_root = list_next_entry(root, link);
-	kvm_tdp_mmu_put_root(kvm, root);
 	return next_root;
 }
 
@@ -97,10 +105,9 @@ static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
  * if exiting the loop early, the caller must drop the reference to the most
  * recent root. (Unless keeping a live reference is desirable.)
  */
-#define for_each_tdp_mmu_root_yield_safe(_kvm, _root)				\
-	for (_root = list_first_entry(&_kvm->arch.tdp_mmu_roots,	\
-				      typeof(*_root), link);		\
-	     tdp_mmu_next_root_valid(_kvm, _root);			\
+#define for_each_tdp_mmu_root_yield_safe(_kvm, _root)	\
+	for (_root = tdp_mmu_next_root(_kvm, NULL);	\
+	     _root;					\
 	     _root = tdp_mmu_next_root(_kvm, _root))
 
 /* Only safe under the MMU lock in write mode, without yielding. */
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
author
#7 lore raw

PATCH 7/13 KVM: x86/mmu: Make TDP MMU root refcount atomic

In reply to Ben Gardon

In order to parallelize more operations for the TDP MMU, make the refcount on TDP MMU roots atomic, so that a future patch can allow multiple threads to take a reference on the root concurrently, while holding the MMU lock in read mode.

---
 arch/x86/kvm/mmu/mmu_internal.h |  6 +++++-
 arch/x86/kvm/mmu/tdp_mmu.c      | 15 ++++++++-------
 arch/x86/kvm/mmu/tdp_mmu.h      |  9 +++------
 3 files changed, 16 insertions(+), 14 deletions(-)
arch/x86/kvm/mmu/mmu_internal.h +5 -1
index 788dcf77c957..0a040d6a4f35 100644
@@ -50,7 +50,11 @@ struct kvm_mmu_page {
 	u64 *spt;
 	/* hold the gfn of each spte inside spt */
 	gfn_t *gfns;
-	int root_count;          /* Currently serving as active root */
+	/* Currently serving as active root */
+	union {
+		int root_count;
+		refcount_t tdp_mmu_root_count;
+	};
 	unsigned int unsync_children;
 	struct kvm_rmap_head parent_ptes; /* rmap pointers to parent sptes */
 	DECLARE_BITMAP(unsync_child_bitmap, 512);
arch/x86/kvm/mmu/tdp_mmu.c +8 -7
index ab1d26b40164..1f0b2d6124a2 100644
@@ -56,7 +56,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
-	if (--root->root_count)
+	if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
 		return;
 
 	WARN_ON(!root->tdp_mmu_page);
@@ -88,10 +88,12 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 		next_root = list_first_entry(&kvm->arch.tdp_mmu_roots,
 					     typeof(*next_root), link);
 
+	while (!list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link) &&
+	       !kvm_tdp_mmu_get_root(kvm, next_root))
+		next_root = list_next_entry(next_root, link);
+
 	if (list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link))
 		next_root = NULL;
-	else
-		kvm_tdp_mmu_get_root(kvm, next_root);
 
 	if (prev_root)
 		kvm_tdp_mmu_put_root(kvm, prev_root);
@@ -158,14 +160,13 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 
 	/* Check for an existing root before allocating a new one. */
 	for_each_tdp_mmu_root(kvm, root) {
-		if (root->role.word == role.word) {
-			kvm_tdp_mmu_get_root(kvm, root);
+		if (root->role.word == role.word &&
+		    kvm_tdp_mmu_get_root(kvm, root))
 			goto out;
-		}
 	}
 
 	root = alloc_tdp_mmu_page(vcpu, 0, vcpu->arch.mmu->shadow_root_level);
-	root->root_count = 1;
+	refcount_set(&root->tdp_mmu_root_count, 1);
 
 	list_add(&root->link, &kvm->arch.tdp_mmu_roots);
 
arch/x86/kvm/mmu/tdp_mmu.h +3 -6
index 5d950e987fc7..9961df505067 100644
@@ -7,13 +7,10 @@
 
 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
 
-static inline void kvm_tdp_mmu_get_root(struct kvm *kvm,
-					struct kvm_mmu_page *root)
+__must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,
+						     struct kvm_mmu_page *root)
 {
-	BUG_ON(!root->root_count);
-	lockdep_assert_held(&kvm->mmu_lock);
-
-	++root->root_count;
+	return refcount_inc_not_zero(&root->tdp_mmu_root_count);
 }
 
 void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
#8 lore raw

Re: [PATCH 07/13] KVM: x86/mmu: Make TDP MMU root refcount atomic

Quoted 7 lines · Ben Gardon

In order to parallelize more operations for the TDP MMU, make the refcount on TDP MMU roots atomic, so that a future patch can allow multiple threads to take a reference on the root concurrently, while holding the MMU lock in read mode.

---

...

Quoted 16 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -2
@@ -88,10 +88,12 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 		next_root = list_first_entry(&kvm->arch.tdp_mmu_roots,
 					     typeof(*next_root), link);
 
+	while (!list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link) &&
+	       !kvm_tdp_mmu_get_root(kvm, next_root))
+		next_root = list_next_entry(next_root, link);
+
 	if (list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link))
 		next_root = NULL;
-	else
-		kvm_tdp_mmu_get_root(kvm, next_root);
 
 	if (prev_root)
 		kvm_tdp_mmu_put_root(kvm, prev_root);

arch/x86/kvm/mmu/tdp_mmu.c hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) · line 164

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +2 -2
@@ -158,14 +160,13 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 
 	/* Check for an existing root before allocating a new one. */
 	for_each_tdp_mmu_root(kvm, root) {
-		if (root->role.word == role.word) {
-			kvm_tdp_mmu_get_root(kvm, root);
+		if (root->role.word == role.word &&
+		    kvm_tdp_mmu_get_root(kvm, root))

I'm not opposed to changing this logic while making the refcount atomic, but it needs to be explained in the changelog. As is, the changelog makes it sound like the patch is a pure refactoring of the type.

author
#9 lore raw

Re: [PATCH 07/13] KVM: x86/mmu: Make TDP MMU root refcount atomic

Quoted 29 lines · Sean Christopherson
Quoted 7 lines · Ben Gardon

In order to parallelize more operations for the TDP MMU, make the refcount on TDP MMU roots atomic, so that a future patch can allow multiple threads to take a reference on the root concurrently, while holding the MMU lock in read mode.

---

...

Quoted 17 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -2
@@ -88,10 +88,12 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
              next_root = list_first_entry(&kvm->arch.tdp_mmu_roots,
                                           typeof(*next_root), link);
 
+     while (!list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link) &&
+            !kvm_tdp_mmu_get_root(kvm, next_root))
+             next_root = list_next_entry(next_root, link);
+
      if (list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link))
              next_root = NULL;
-     else
-             kvm_tdp_mmu_get_root(kvm, next_root);
 
      if (prev_root)
              kvm_tdp_mmu_put_root(kvm, prev_root);
@@ -158,14 +160,13 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 
Quoted 10 lines

arch/x86/kvm/mmu/tdp_mmu.c hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) · line 164

Quoted 6 lines
arch/x86/kvm/mmu/tdp_mmu.c +2 -2
      /* Check for an existing root before allocating a new one. */
      for_each_tdp_mmu_root(kvm, root) {
-             if (root->role.word == role.word) {
-                     kvm_tdp_mmu_get_root(kvm, root);
+             if (root->role.word == role.word &&
+                 kvm_tdp_mmu_get_root(kvm, root))

I'm not opposed to changing this logic while making the refcount atomic, but it needs to be explained in the changelog. As is, the changelog makes it sound like the patch is a pure refactoring of the type.

Thanks for pointing that out. I'll add a note in the description in v2. Those felt like natural changes since the introduction of the atomic requires additional failure handling. I don't think there's any way to add it as a separate commit without just introducing dead code, but that would certainly be preferable.

author
#10 lore raw

PATCH 8/13 KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU

In reply to Ben Gardon

Protect the contents of the TDP MMU roots list with RCU in preparation for a future patch which will allow the iterator macro to be used under the MMU lock in read mode.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 64 +++++++++++++++++++++-----------------
 1 file changed, 36 insertions(+), 28 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +36 -28
index 1f0b2d6124a2..d255125059c4 100644
@@ -50,6 +50,22 @@ static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
 	kmem_cache_free(mmu_page_header_cache, sp);
 }
 
+/*
+ * This is called through call_rcu in order to free TDP page table memory
+ * safely with respect to other kernel threads that may be operating on
+ * the memory.
+ * By only accessing TDP MMU page table memory in an RCU read critical
+ * section, and freeing it after a grace period, lockless access to that
+ * memory won't use it after it is freed.
+ */
+static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head)
+{
+	struct kvm_mmu_page *sp = container_of(head, struct kvm_mmu_page,
+					       rcu_head);
+
+	tdp_mmu_free_sp(sp);
+}
+
 void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
 {
 	gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
@@ -61,11 +77,13 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
 
 	WARN_ON(!root->tdp_mmu_page);
 
-	list_del(&root->link);
+	spin_lock(&kvm->arch.tdp_mmu_pages_lock);
+	list_del_rcu(&root->link);
+	spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 
 	zap_gfn_range(kvm, root, 0, max_gfn, false);
 
-	tdp_mmu_free_sp(root);
+	call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback);
 }
 
 /*
@@ -82,18 +100,21 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
+	rcu_read_lock();
+
 	if (prev_root)
-		next_root = list_next_entry(prev_root, link);
+		next_root = list_next_or_null_rcu(&kvm->arch.tdp_mmu_roots,
+						  &prev_root->link,
+						  typeof(*prev_root), link);
 	else
-		next_root = list_first_entry(&kvm->arch.tdp_mmu_roots,
-					     typeof(*next_root), link);
+		next_root = list_first_or_null_rcu(&kvm->arch.tdp_mmu_roots,
+						   typeof(*next_root), link);
 
-	while (!list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link) &&
-	       !kvm_tdp_mmu_get_root(kvm, next_root))
-		next_root = list_next_entry(next_root, link);
+	while (next_root && !kvm_tdp_mmu_get_root(kvm, next_root))
+		next_root = list_next_or_null_rcu(&kvm->arch.tdp_mmu_roots,
+				&next_root->link, typeof(*next_root), link);
 
-	if (list_entry_is_head(next_root, &kvm->arch.tdp_mmu_roots, link))
-		next_root = NULL;
+	rcu_read_unlock();
 
 	if (prev_root)
 		kvm_tdp_mmu_put_root(kvm, prev_root);
@@ -114,7 +135,8 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 
 /* Only safe under the MMU lock in write mode, without yielding. */
 #define for_each_tdp_mmu_root(_kvm, _root)				\
-	list_for_each_entry(_root, &_kvm->arch.tdp_mmu_roots, link)
+	list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link,	\
+				lockdep_is_held_write(&kvm->mmu_lock))
 
 static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu,
 						   int level)
@@ -168,28 +190,14 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 	root = alloc_tdp_mmu_page(vcpu, 0, vcpu->arch.mmu->shadow_root_level);
 	refcount_set(&root->tdp_mmu_root_count, 1);
 
-	list_add(&root->link, &kvm->arch.tdp_mmu_roots);
+	spin_lock(&kvm->arch.tdp_mmu_pages_lock);
+	list_add_rcu(&root->link, &kvm->arch.tdp_mmu_roots);
+	spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 
 out:
 	return __pa(root->spt);
 }
 
-/*
- * This is called through call_rcu in order to free TDP page table memory
- * safely with respect to other kernel threads that may be operating on
- * the memory.
- * By only accessing TDP MMU page table memory in an RCU read critical
- * section, and freeing it after a grace period, lockless access to that
- * memory won't use it after it is freed.
- */
-static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head)
-{
-	struct kvm_mmu_page *sp = container_of(head, struct kvm_mmu_page,
-					       rcu_head);
-
-	tdp_mmu_free_sp(sp);
-}
-
 static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,
 				u64 old_spte, u64 new_spte, int level,
 				bool shared);
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
maintainer
#11 lore raw

Re: [PATCH 08/13] KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU

arch/x86/kvm/mmu/tdp_mmu.c void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) · line 82

Quoted 13 lines · Ben Gardon

Protect the contents of the TDP MMU roots list with RCU in preparation for a future patch which will allow the iterator macro to be used under the MMU lock in read mode.

---
  arch/x86/kvm/mmu/tdp_mmu.c | 64 +++++++++++++++++++++-----------------
  1 file changed, 36 insertions(+), 28 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +3 -0
+	spin_lock(&kvm->arch.tdp_mmu_pages_lock);
+	list_del_rcu(&root->link);
+	spin_unlock(&kvm->arch.tdp_mmu_pages_lock);

Please update the comment above tdp_mmu_pages_lock in arch/x86/include/asm/kvm_host.h as well.

arch/x86/kvm/mmu/tdp_mmu.c static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, · line 139

Quoted 5 lines

/* Only safe under the MMU lock in write mode, without yielding. */
#define for_each_tdp_mmu_root(_kvm, _root) \

arch/x86/kvm/mmu/tdp_mmu.c +2 -1
-	list_for_each_entry(_root, &_kvm->arch.tdp_mmu_roots, link)
+	list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link,	\
+				lockdep_is_held_write(&kvm->mmu_lock))

This should also add "... || lockdep_is_help(&kvm->arch.tdp_mmu_pages_lock)", if only for documentation purposes.

Paolo

author
#12 lore raw

Re: [PATCH 08/13] KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU

Quoted 11 lines · Paolo Bonzini
Quoted 9 lines · Ben Gardon

Protect the contents of the TDP MMU roots list with RCU in preparation for a future patch which will allow the iterator macro to be used under the MMU lock in read mode.

---
  arch/x86/kvm/mmu/tdp_mmu.c | 64 +++++++++++++++++++++-----------------
  1 file changed, 36 insertions(+), 28 deletions(-)
Quoted 8 lines

arch/x86/kvm/mmu/tdp_mmu.c void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) · line 82

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +3 -0
+     spin_lock(&kvm->arch.tdp_mmu_pages_lock);
+     list_del_rcu(&root->link);
+     spin_unlock(&kvm->arch.tdp_mmu_pages_lock);

Please update the comment above tdp_mmu_pages_lock in arch/x86/include/asm/kvm_host.h as well.

Ah yes, thank you for catching that. Will do.

Quoted 10 lines

arch/x86/kvm/mmu/tdp_mmu.c static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, · line 139

Quoted 6 lines

/* Only safe under the MMU lock in write mode, without yielding. */
#define for_each_tdp_mmu_root(_kvm, _root) \

arch/x86/kvm/mmu/tdp_mmu.c +2 -1
-     list_for_each_entry(_root, &_kvm->arch.tdp_mmu_roots, link)
+     list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link, \
+                             lockdep_is_held_write(&kvm->mmu_lock))

This should also add "... || lockdep_is_help(&kvm->arch.tdp_mmu_pages_lock)", if only for documentation purposes.

Good idea. I hope we never have a function try to protect its loop over the roots with that lock, but it would be correct.

Quoted 3 lines

Paolo

#13 lore raw

Re: [PATCH 08/13] KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU

In reply to Ben Gardon
Thank you for the patch! Yet something to improve:

Hi Ben,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on next-20210331]
[cannot apply to kvm/queue tip/master linux/master linus/master v5.12-rc5 v5.12-rc4 v5.12-rc3 v5.12-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Ben-Gardon/More-parallel-operations-for-the-TDP-MMU/20210401-051137
base:    7a43c78d0573e0bbbb0456b033e2b9a895b89464
config: x86_64-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/2b2c6d3bdc35269df5f9293a02da5b71c74095f3
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Ben-Gardon/More-parallel-operations-for-the-TDP-MMU/20210401-051137
        git checkout 2b2c6d3bdc35269df5f9293a02da5b71c74095f3
        # save the attached .config to linux build tree
        make W=1 ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

In file included from include/linux/rculist.h:11,

                    from include/linux/pid.h:5,
                    from include/linux/sched.h:14,
                    from include/linux/hardirq.h:9,
                    from include/linux/kvm_host.h:7,
                    from arch/x86/kvm/mmu.h:5,
                    from arch/x86/kvm/mmu/tdp_mmu.c:3:
   arch/x86/kvm/mmu/tdp_mmu.c: In function 'kvm_tdp_mmu_get_vcpu_root_hpa':
>> arch/x86/kvm/mmu/tdp_mmu.c:139:5: error: implicit declaration of function 'lockdep_is_held_write'; did you mean 'lockdep_is_held_type'? [-Werror=implicit-function-declaration]
     139 |     lockdep_is_held_write(&kvm->mmu_lock))
         |     ^~~~~~~~~~~~~~~~~~~~~
   include/linux/rcupdate.h:318:52: note: in definition of macro 'RCU_LOCKDEP_WARN'
     318 |   if (debug_lockdep_rcu_enabled() && !__warned && (c)) { \
         |                                                    ^
   include/linux/rculist.h:391:7: note: in expansion of macro '__list_check_rcu'
     391 |  for (__list_check_rcu(dummy, ## cond, 0),   \
         |       ^~~~~~~~~~~~~~~~
   arch/x86/kvm/mmu/tdp_mmu.c:138:2: note: in expansion of macro 'list_for_each_entry_rcu'
     138 |  list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link, \
         |  ^~~~~~~~~~~~~~~~~~~~~~~
   arch/x86/kvm/mmu/tdp_mmu.c:184:2: note: in expansion of macro 'for_each_tdp_mmu_root'
     184 |  for_each_tdp_mmu_root(kvm, root) {
         |  ^~~~~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +139 arch/x86/kvm/mmu/tdp_mmu.c

     2	
   > 3	#include "mmu.h"
     4	#include "mmu_internal.h"
     5	#include "mmutrace.h"
     6	#include "tdp_iter.h"
     7	#include "tdp_mmu.h"
     8	#include "spte.h"
     9	
    10	#include <asm/cmpxchg.h>
    11	#include <trace/events/kvm.h>
    12	
    13	static bool __read_mostly tdp_mmu_enabled = false;
    14	module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0644);
    15	
    16	/* Initializes the TDP MMU for the VM, if enabled. */
    17	void kvm_mmu_init_tdp_mmu(struct kvm *kvm)
    18	{
    19		if (!tdp_enabled || !READ_ONCE(tdp_mmu_enabled))
    20			return;
    21	
    22		/* This should not be changed for the lifetime of the VM. */
    23		kvm->arch.tdp_mmu_enabled = true;
    24	
    25		INIT_LIST_HEAD(&kvm->arch.tdp_mmu_roots);
    26		spin_lock_init(&kvm->arch.tdp_mmu_pages_lock);
    27		INIT_LIST_HEAD(&kvm->arch.tdp_mmu_pages);
    28	}
    29	
    30	void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
    31	{
    32		if (!kvm->arch.tdp_mmu_enabled)
    33			return;
    34	
    35		WARN_ON(!list_empty(&kvm->arch.tdp_mmu_roots));
    36	
    37		/*
    38		 * Ensure that all the outstanding RCU callbacks to free shadow pages
    39		 * can run before the VM is torn down.
    40		 */
    41		rcu_barrier();
    42	}
    43	
    44	static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
    45				  gfn_t start, gfn_t end, bool can_yield);
    46	
    47	static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
    48	{
    49		free_page((unsigned long)sp->spt);
    50		kmem_cache_free(mmu_page_header_cache, sp);
    51	}
    52	
    53	/*
    54	 * This is called through call_rcu in order to free TDP page table memory
    55	 * safely with respect to other kernel threads that may be operating on
    56	 * the memory.
    57	 * By only accessing TDP MMU page table memory in an RCU read critical
    58	 * section, and freeing it after a grace period, lockless access to that
    59	 * memory won't use it after it is freed.
    60	 */
    61	static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head)
    62	{
    63		struct kvm_mmu_page *sp = container_of(head, struct kvm_mmu_page,
    64						       rcu_head);
    65	
    66		tdp_mmu_free_sp(sp);
    67	}
    68	
    69	void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
    70	{
    71		gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
    72	
    73		lockdep_assert_held_write(&kvm->mmu_lock);
    74	
    75		if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
    76			return;
    77	
    78		WARN_ON(!root->tdp_mmu_page);
    79	
    80		spin_lock(&kvm->arch.tdp_mmu_pages_lock);
    81		list_del_rcu(&root->link);
    82		spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
    83	
    84		zap_gfn_range(kvm, root, 0, max_gfn, false);
    85	
    86		call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback);
    87	}
    88	
    89	/*
    90	 * Finds the next valid root after root (or the first valid root if root
    91	 * is NULL), takes a reference on it, and returns that next root. If root
    92	 * is not NULL, this thread should have already taken a reference on it, and
    93	 * that reference will be dropped. If no valid root is found, this
    94	 * function will return NULL.
    95	 */
    96	static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
    97						      struct kvm_mmu_page *prev_root)
    98	{
    99		struct kvm_mmu_page *next_root;
   100	
   101		lockdep_assert_held_write(&kvm->mmu_lock);
   102	
   103		rcu_read_lock();
   104	
   105		if (prev_root)
   106			next_root = list_next_or_null_rcu(&kvm->arch.tdp_mmu_roots,
   107							  &prev_root->link,
   108							  typeof(*prev_root), link);
   109		else
   110			next_root = list_first_or_null_rcu(&kvm->arch.tdp_mmu_roots,
   111							   typeof(*next_root), link);
   112	
   113		while (next_root && !kvm_tdp_mmu_get_root(kvm, next_root))
   114			next_root = list_next_or_null_rcu(&kvm->arch.tdp_mmu_roots,
   115					&next_root->link, typeof(*next_root), link);
   116	
   117		rcu_read_unlock();
   118	
   119		if (prev_root)
   120			kvm_tdp_mmu_put_root(kvm, prev_root);
   121	
   122		return next_root;
   123	}
   124	
   125	/*
   126	 * Note: this iterator gets and puts references to the roots it iterates over.
   127	 * This makes it safe to release the MMU lock and yield within the loop, but
   128	 * if exiting the loop early, the caller must drop the reference to the most
   129	 * recent root. (Unless keeping a live reference is desirable.)
   130	 */
   131	#define for_each_tdp_mmu_root_yield_safe(_kvm, _root)	\
   132		for (_root = tdp_mmu_next_root(_kvm, NULL);	\
   133		     _root;					\
   134		     _root = tdp_mmu_next_root(_kvm, _root))
   135	
   136	/* Only safe under the MMU lock in write mode, without yielding. */
   137	#define for_each_tdp_mmu_root(_kvm, _root)				\
   138		list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link,	\
 > 139					lockdep_is_held_write(&kvm->mmu_lock))

140

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author
#14 lore raw

Re: [PATCH 08/13] KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU

Quoted 35 lines · kernel test robot

Hi Ben,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on next-20210331]
[cannot apply to kvm/queue tip/master linux/master linus/master v5.12-rc5 v5.12-rc4 v5.12-rc3 v5.12-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Ben-Gardon/More-parallel-operations-for-the-TDP-MMU/20210401-051137
base:    7a43c78d0573e0bbbb0456b033e2b9a895b89464
config: x86_64-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/2b2c6d3bdc35269df5f9293a02da5b71c74095f3
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Ben-Gardon/More-parallel-operations-for-the-TDP-MMU/20210401-051137
        git checkout 2b2c6d3bdc35269df5f9293a02da5b71c74095f3
        # save the attached .config to linux build tree
        make W=1 ARCH=x86_64

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

In file included from include/linux/rculist.h:11,

                    from include/linux/pid.h:5,
                    from include/linux/sched.h:14,
                    from include/linux/hardirq.h:9,
                    from include/linux/kvm_host.h:7,
                    from arch/x86/kvm/mmu.h:5,
                    from arch/x86/kvm/mmu/tdp_mmu.c:3:
Quoted 3 lines

arch/x86/kvm/mmu/tdp_mmu.c: In function 'kvm_tdp_mmu_get_vcpu_root_hpa':

Quoted 1 line
Quoted 1 line

arch/x86/kvm/mmu/tdp_mmu.c:139:5: error: implicit declaration of function 'lockdep_is_held_write'; did you mean 'lockdep_is_held_type'? [-Werror=implicit-function-declaration]

     139 |     lockdep_is_held_write(&kvm->mmu_lock))

Huh, I wonder why this isn't exported in some configuration. I'll fix this in v2 as well.

Quoted 161 lines
         |     ^~~~~~~~~~~~~~~~~~~~~
   include/linux/rcupdate.h:318:52: note: in definition of macro 'RCU_LOCKDEP_WARN'
     318 |   if (debug_lockdep_rcu_enabled() && !__warned && (c)) { \
         |                                                    ^
   include/linux/rculist.h:391:7: note: in expansion of macro '__list_check_rcu'
     391 |  for (__list_check_rcu(dummy, ## cond, 0),   \
         |       ^~~~~~~~~~~~~~~~
   arch/x86/kvm/mmu/tdp_mmu.c:138:2: note: in expansion of macro 'list_for_each_entry_rcu'
     138 |  list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link, \
         |  ^~~~~~~~~~~~~~~~~~~~~~~
   arch/x86/kvm/mmu/tdp_mmu.c:184:2: note: in expansion of macro 'for_each_tdp_mmu_root'
     184 |  for_each_tdp_mmu_root(kvm, root) {
         |  ^~~~~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +139 arch/x86/kvm/mmu/tdp_mmu.c

     2
Quoted 1 line

3 #include "mmu.h"

     4  #include "mmu_internal.h"
     5  #include "mmutrace.h"
     6  #include "tdp_iter.h"
     7  #include "tdp_mmu.h"
     8  #include "spte.h"
     9
    10  #include <asm/cmpxchg.h>
    11  #include <trace/events/kvm.h>
    12
    13  static bool __read_mostly tdp_mmu_enabled = false;
    14  module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0644);
    15
    16  /* Initializes the TDP MMU for the VM, if enabled. */
    17  void kvm_mmu_init_tdp_mmu(struct kvm *kvm)
    18  {
    19          if (!tdp_enabled || !READ_ONCE(tdp_mmu_enabled))
    20                  return;
    21
    22          /* This should not be changed for the lifetime of the VM. */
    23          kvm->arch.tdp_mmu_enabled = true;
    24
    25          INIT_LIST_HEAD(&kvm->arch.tdp_mmu_roots);
    26          spin_lock_init(&kvm->arch.tdp_mmu_pages_lock);
    27          INIT_LIST_HEAD(&kvm->arch.tdp_mmu_pages);
    28  }
    29
    30  void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
    31  {
    32          if (!kvm->arch.tdp_mmu_enabled)
    33                  return;
    34
    35          WARN_ON(!list_empty(&kvm->arch.tdp_mmu_roots));
    36
    37          /*
    38           * Ensure that all the outstanding RCU callbacks to free shadow pages
    39           * can run before the VM is torn down.
    40           */
    41          rcu_barrier();
    42  }
    43
    44  static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
    45                            gfn_t start, gfn_t end, bool can_yield);
    46
    47  static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
    48  {
    49          free_page((unsigned long)sp->spt);
    50          kmem_cache_free(mmu_page_header_cache, sp);
    51  }
    52
    53  /*
    54   * This is called through call_rcu in order to free TDP page table memory
    55   * safely with respect to other kernel threads that may be operating on
    56   * the memory.
    57   * By only accessing TDP MMU page table memory in an RCU read critical
    58   * section, and freeing it after a grace period, lockless access to that
    59   * memory won't use it after it is freed.
    60   */
    61  static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head)
    62  {
    63          struct kvm_mmu_page *sp = container_of(head, struct kvm_mmu_page,
    64                                                 rcu_head);
    65
    66          tdp_mmu_free_sp(sp);
    67  }
    68
    69  void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
    70  {
    71          gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
    72
    73          lockdep_assert_held_write(&kvm->mmu_lock);
    74
    75          if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
    76                  return;
    77
    78          WARN_ON(!root->tdp_mmu_page);
    79
    80          spin_lock(&kvm->arch.tdp_mmu_pages_lock);
    81          list_del_rcu(&root->link);
    82          spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
    83
    84          zap_gfn_range(kvm, root, 0, max_gfn, false);
    85
    86          call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback);
    87  }
    88
    89  /*
    90   * Finds the next valid root after root (or the first valid root if root
    91   * is NULL), takes a reference on it, and returns that next root. If root
    92   * is not NULL, this thread should have already taken a reference on it, and
    93   * that reference will be dropped. If no valid root is found, this
    94   * function will return NULL.
    95   */
    96  static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
    97                                                struct kvm_mmu_page *prev_root)
    98  {
    99          struct kvm_mmu_page *next_root;
   100
   101          lockdep_assert_held_write(&kvm->mmu_lock);
   102
   103          rcu_read_lock();
   104
   105          if (prev_root)
   106                  next_root = list_next_or_null_rcu(&kvm->arch.tdp_mmu_roots,
   107                                                    &prev_root->link,
   108                                                    typeof(*prev_root), link);
   109          else
   110                  next_root = list_first_or_null_rcu(&kvm->arch.tdp_mmu_roots,
   111                                                     typeof(*next_root), link);
   112
   113          while (next_root && !kvm_tdp_mmu_get_root(kvm, next_root))
   114                  next_root = list_next_or_null_rcu(&kvm->arch.tdp_mmu_roots,
   115                                  &next_root->link, typeof(*next_root), link);
   116
   117          rcu_read_unlock();
   118
   119          if (prev_root)
   120                  kvm_tdp_mmu_put_root(kvm, prev_root);
   121
   122          return next_root;
   123  }
   124
   125  /*
   126   * Note: this iterator gets and puts references to the roots it iterates over.
   127   * This makes it safe to release the MMU lock and yield within the loop, but
   128   * if exiting the loop early, the caller must drop the reference to the most
   129   * recent root. (Unless keeping a live reference is desirable.)
   130   */
   131  #define for_each_tdp_mmu_root_yield_safe(_kvm, _root)   \
   132          for (_root = tdp_mmu_next_root(_kvm, NULL);     \
   133               _root;                                     \
   134               _root = tdp_mmu_next_root(_kvm, _root))
   135
   136  /* Only safe under the MMU lock in write mode, without yielding. */
   137  #define for_each_tdp_mmu_root(_kvm, _root)                              \
   138          list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link, \
Quoted 1 line

139 lockdep_is_held_write(&kvm->mmu_lock))

140

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author
#15 lore raw

PATCH 9/13 KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock

In reply to Ben Gardon

To reduce lock contention and interference with page fault handlers, allow the TDP MMU function to zap a GFN range to operate under the MMU read lock.

---
 arch/x86/kvm/mmu/mmu.c     |  15 ++++--
 arch/x86/kvm/mmu/tdp_mmu.c | 102 ++++++++++++++++++++++++++-----------
 arch/x86/kvm/mmu/tdp_mmu.h |   6 ++-
 3 files changed, 87 insertions(+), 36 deletions(-)
arch/x86/kvm/mmu/mmu.c +10 -5
index 667d64daa82c..dcbfc784cf2f 100644
@@ -3155,7 +3155,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
 	sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
 
 	if (is_tdp_mmu_page(sp))
-		kvm_tdp_mmu_put_root(kvm, sp);
+		kvm_tdp_mmu_put_root(kvm, sp, false);
 	else if (!--sp->root_count && sp->role.invalid)
 		kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 
@@ -5514,13 +5514,17 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 		}
 	}
 
+	write_unlock(&kvm->mmu_lock);
+
 	if (is_tdp_mmu_enabled(kvm)) {
-		flush = kvm_tdp_mmu_zap_gfn_range(kvm, gfn_start, gfn_end);
+		read_lock(&kvm->mmu_lock);
+		flush = kvm_tdp_mmu_zap_gfn_range(kvm, gfn_start, gfn_end,
+						  true);
 		if (flush)
 			kvm_flush_remote_tlbs(kvm);
-	}
 
-	write_unlock(&kvm->mmu_lock);
+		read_unlock(&kvm->mmu_lock);
+	}
 }
 
 static bool slot_rmap_write_protect(struct kvm *kvm,
@@ -5959,7 +5963,8 @@ static void kvm_recover_nx_lpages(struct kvm *kvm)
 		WARN_ON_ONCE(!sp->lpage_disallowed);
 		if (is_tdp_mmu_page(sp)) {
 			kvm_tdp_mmu_zap_gfn_range(kvm, sp->gfn,
-				sp->gfn + KVM_PAGES_PER_HPAGE(sp->role.level));
+				sp->gfn + KVM_PAGES_PER_HPAGE(sp->role.level),
+				false);
 		} else {
 			kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
 			WARN_ON_ONCE(sp->lpage_disallowed);
arch/x86/kvm/mmu/tdp_mmu.c +73 -29

255 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +4 -2
index 9961df505067..855e58856815 100644
@@ -13,9 +13,11 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,
 	return refcount_inc_not_zero(&root->tdp_mmu_root_count);
 }
 
-void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
+			  bool shared);
 
-bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end);
+bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end,
+			       bool shared);
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 
 int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
maintainer
#16 lore raw

Re: [PATCH 09/13] KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock

Quoted 33 lines · Ben Gardon

To reduce lock contention and interference with page fault handlers, allow the TDP MMU function to zap a GFN range to operate under the MMU read lock.

---
  arch/x86/kvm/mmu/mmu.c     |  15 ++++--
  arch/x86/kvm/mmu/tdp_mmu.c | 102 ++++++++++++++++++++++++++-----------
  arch/x86/kvm/mmu/tdp_mmu.h |   6 ++-
  3 files changed, 87 insertions(+), 36 deletions(-)
arch/x86/kvm/mmu/mmu.c +4 -2
index 667d64daa82c..dcbfc784cf2f 100644
@@ -3155,7 +3155,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
  	sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
 
  	if (is_tdp_mmu_page(sp))
-		kvm_tdp_mmu_put_root(kvm, sp);
+		kvm_tdp_mmu_put_root(kvm, sp, false);
  	else if (!--sp->root_count && sp->role.invalid)
  		kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 
@@ -5514,13 +5514,17 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
  		}
  	}
 
+	write_unlock(&kvm->mmu_lock);
+
  	if (is_tdp_mmu_enabled(kvm)) {
-		flush = kvm_tdp_mmu_zap_gfn_range(kvm, gfn_start, gfn_end);
+		read_lock(&kvm->mmu_lock);

arch/x86/kvm/mmu/mmu.c void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) · line 5528

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +4 -2
+		flush = kvm_tdp_mmu_zap_gfn_range(kvm, gfn_start, gfn_end,
+						  true);
  		if (flush)
  			kvm_flush_remote_tlbs(kvm);
-	}
 
-	write_unlock(&kvm->mmu_lock);
+		read_unlock(&kvm->mmu_lock);
+	}
  }

This will conflict with Sean's MMU notifier series patches:

KVM: x86/mmu: Pass address space ID to __kvm_tdp_mmu_zap_gfn_range()

What I can do for now is change the mmu.c part of that patch to

arch/x86/kvm/mmu/mmu.c +6 -6
index e6e02360ef67..9882bbd9b742 100644
@@ -5510,15 +5510,15 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 		}
 	}
 
-	if (flush)
-		kvm_flush_remote_tlbs_with_address(kvm, gfn_start, gfn_end);
-
 	if (is_tdp_mmu_enabled(kvm)) {
-		flush = kvm_tdp_mmu_zap_gfn_range(kvm, gfn_start, gfn_end);
-		if (flush)
-			kvm_flush_remote_tlbs(kvm);
+		for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
+			flush = kvm_tdp_mmu_zap_gfn_range(kvm, i, gfn_start,
+							  gfn_end, flush);
 	}
 
+	if (flush)
+		kvm_flush_remote_tlbs_with_address(kvm, gfn_start, gfn_end);
+
 	write_unlock(&kvm->mmu_lock);
 }
 

but you will have to add a separate "if (flush)" when moving the write_unlock earlier, since there's no downgrade function for rwlocks. In practice it's not a huge deal since unless running nested there will be only one active MMU.

Paolo

Quoted 310 lines
arch/x86/kvm/mmu/mmu.c +2 -1
  static bool slot_rmap_write_protect(struct kvm *kvm,
@@ -5959,7 +5963,8 @@ static void kvm_recover_nx_lpages(struct kvm *kvm)
  		WARN_ON_ONCE(!sp->lpage_disallowed);
  		if (is_tdp_mmu_page(sp)) {
  			kvm_tdp_mmu_zap_gfn_range(kvm, sp->gfn,
-				sp->gfn + KVM_PAGES_PER_HPAGE(sp->role.level));
+				sp->gfn + KVM_PAGES_PER_HPAGE(sp->role.level),
+				false);
  		} else {
  			kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
  			WARN_ON_ONCE(sp->lpage_disallowed);
arch/x86/kvm/mmu/tdp_mmu.c +73 -29
index d255125059c4..0e99e4675dd4 100644
@@ -27,6 +27,15 @@ void kvm_mmu_init_tdp_mmu(struct kvm *kvm)
  	INIT_LIST_HEAD(&kvm->arch.tdp_mmu_pages);
  }
 
+static __always_inline void kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,
+							     bool shared)
+{
+	if (shared)
+		lockdep_assert_held_read(&kvm->mmu_lock);
+	else
+		lockdep_assert_held_write(&kvm->mmu_lock);
+}
+
  void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
  {
  	if (!kvm->arch.tdp_mmu_enabled)
@@ -42,7 +51,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
  }
 
  static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
-			  gfn_t start, gfn_t end, bool can_yield);
+			  gfn_t start, gfn_t end, bool can_yield, bool shared);
 
  static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
  {
@@ -66,11 +75,12 @@ static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head)
  	tdp_mmu_free_sp(sp);
  }
 
-void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
+			  bool shared)
  {
  	gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
 
-	lockdep_assert_held_write(&kvm->mmu_lock);
+	kvm_lockdep_assert_mmu_lock_held(kvm, shared);
 
  	if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
  		return;
@@ -81,7 +91,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
  	list_del_rcu(&root->link);
  	spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 
-	zap_gfn_range(kvm, root, 0, max_gfn, false);
+	zap_gfn_range(kvm, root, 0, max_gfn, false, shared);
 
  	call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback);
  }
@@ -94,11 +104,11 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
   * function will return NULL.
   */
  static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
-					      struct kvm_mmu_page *prev_root)
+					      struct kvm_mmu_page *prev_root,
+					      bool shared)
  {
  	struct kvm_mmu_page *next_root;
 
-	lockdep_assert_held_write(&kvm->mmu_lock);
 
  	rcu_read_lock();
 
@@ -117,7 +127,7 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
  	rcu_read_unlock();
 
  	if (prev_root)
-		kvm_tdp_mmu_put_root(kvm, prev_root);
+		kvm_tdp_mmu_put_root(kvm, prev_root, shared);
 
  	return next_root;
  }
@@ -127,11 +137,15 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
   * This makes it safe to release the MMU lock and yield within the loop, but
   * if exiting the loop early, the caller must drop the reference to the most
   * recent root. (Unless keeping a live reference is desirable.)
+ *
+ * If shared is set, this function is operating under the MMU lock in read
+ * mode. In the unlikely event that this thread must free a root, the lock
+ * will be temporarily dropped and reacquired in write mode.
   */
-#define for_each_tdp_mmu_root_yield_safe(_kvm, _root)	\
-	for (_root = tdp_mmu_next_root(_kvm, NULL);	\
-	     _root;					\
-	     _root = tdp_mmu_next_root(_kvm, _root))
+#define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _shared)	\
+	for (_root = tdp_mmu_next_root(_kvm, NULL, _shared);	\
+	     _root;						\
+	     _root = tdp_mmu_next_root(_kvm, _root, _shared))
 
  /* Only safe under the MMU lock in write mode, without yielding. */
  #define for_each_tdp_mmu_root(_kvm, _root)				\
@@ -632,7 +646,8 @@ static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm,
   * Return false if a yield was not needed.
   */
  static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,
-					     struct tdp_iter *iter, bool flush)
+					     struct tdp_iter *iter, bool flush,
+					     bool shared)
  {
  	/* Ensure forward progress has been made before yielding. */
  	if (iter->next_last_level_gfn == iter->yielded_gfn)
@@ -644,7 +659,11 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,
  		if (flush)
  			kvm_flush_remote_tlbs(kvm);
 
-		cond_resched_rwlock_write(&kvm->mmu_lock);
+		if (shared)
+			cond_resched_rwlock_read(&kvm->mmu_lock);
+		else
+			cond_resched_rwlock_write(&kvm->mmu_lock);
+
  		rcu_read_lock();
 
  		WARN_ON(iter->gfn > iter->next_last_level_gfn);
@@ -662,23 +681,33 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,
   * non-root pages mapping GFNs strictly within that range. Returns true if
   * SPTEs have been cleared and a TLB flush is needed before releasing the
   * MMU lock.
+ *
   * If can_yield is true, will release the MMU lock and reschedule if the
   * scheduler needs the CPU or there is contention on the MMU lock. If this
   * function cannot yield, it will not release the MMU lock or reschedule and
   * the caller must ensure it does not supply too large a GFN range, or the
   * operation can cause a soft lockup.
+ *
+ * If shared is true, this thread holds the MMU lock in read mode and must
+ * account for the possibility that other threads are modifying the paging
+ * structures concurrently. If shared is false, this thread should hold the
+ * MMU lock in write mode.
   */
  static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
-			  gfn_t start, gfn_t end, bool can_yield)
+			  gfn_t start, gfn_t end, bool can_yield, bool shared)
  {
  	struct tdp_iter iter;
  	bool flush_needed = false;
 
+	kvm_lockdep_assert_mmu_lock_held(kvm, shared);
+
  	rcu_read_lock();
 
  	tdp_root_for_each_pte(iter, root, start, end) {
+retry:
  		if (can_yield &&
-		    tdp_mmu_iter_cond_resched(kvm, &iter, flush_needed)) {
+		    tdp_mmu_iter_cond_resched(kvm, &iter, flush_needed,
+					      shared)) {
  			flush_needed = false;
  			continue;
  		}
@@ -696,8 +725,17 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
  		    !is_last_spte(iter.old_spte, iter.level))
  			continue;
 
-		tdp_mmu_set_spte(kvm, &iter, 0);
-		flush_needed = true;
+		if (!shared) {
+			tdp_mmu_set_spte(kvm, &iter, 0);
+			flush_needed = true;
+		} else if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) {
+			/*
+			 * The iter must explicitly re-read the SPTE because
+			 * the atomic cmpxchg failed.
+			 */
+			iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
+			goto retry;
+		}
  	}
 
  	rcu_read_unlock();
@@ -709,14 +747,20 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
   * non-root pages mapping GFNs strictly within that range. Returns true if
   * SPTEs have been cleared and a TLB flush is needed before releasing the
   * MMU lock.
+ *
+ * If shared is true, this thread holds the MMU lock in read mode and must
+ * account for the possibility that other threads are modifying the paging
+ * structures concurrently. If shared is false, this thread should hold the
+ * MMU in write mode.
   */
-bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end)
+bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end,
+			       bool shared)
  {
  	struct kvm_mmu_page *root;
  	bool flush = false;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root)
-		flush |= zap_gfn_range(kvm, root, start, end, true);
+	for_each_tdp_mmu_root_yield_safe(kvm, root, shared)
+		flush |= zap_gfn_range(kvm, root, start, end, true, shared);
 
  	return flush;
  }
@@ -726,7 +770,7 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm)
  	gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
  	bool flush;
 
-	flush = kvm_tdp_mmu_zap_gfn_range(kvm, 0, max_gfn);
+	flush = kvm_tdp_mmu_zap_gfn_range(kvm, 0, max_gfn, false);
  	if (flush)
  		kvm_flush_remote_tlbs(kvm);
  }
@@ -893,7 +937,7 @@ static __always_inline int kvm_tdp_mmu_handle_hva_range(struct kvm *kvm,
  	int ret = 0;
  	int as_id;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root) {
+	for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
  		as_id = kvm_mmu_page_as_id(root);
  		slots = __kvm_memslots(kvm, as_id);
  		kvm_for_each_memslot(memslot, slots) {
@@ -933,7 +977,7 @@ static int zap_gfn_range_hva_wrapper(struct kvm *kvm,
  				     struct kvm_mmu_page *root, gfn_t start,
  				     gfn_t end, unsigned long unused)
  {
-	return zap_gfn_range(kvm, root, start, end, false);
+	return zap_gfn_range(kvm, root, start, end, false, false);
  }
 
  int kvm_tdp_mmu_zap_hva_range(struct kvm *kvm, unsigned long start,
@@ -1098,7 +1142,7 @@ static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
 
  	for_each_tdp_pte_min_level(iter, root->spt, root->role.level,
  				   min_level, start, end) {
-		if (tdp_mmu_iter_cond_resched(kvm, &iter, false))
+		if (tdp_mmu_iter_cond_resched(kvm, &iter, false, false))
  			continue;
 
  		if (!is_shadow_present_pte(iter.old_spte) ||
@@ -1128,7 +1172,7 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, struct kvm_memory_slot *slot,
  	int root_as_id;
  	bool spte_set = false;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root) {
+	for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
  		root_as_id = kvm_mmu_page_as_id(root);
  		if (root_as_id != slot->as_id)
  			continue;
@@ -1157,7 +1201,7 @@ static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
  	rcu_read_lock();
 
  	tdp_root_for_each_leaf_pte(iter, root, start, end) {
-		if (tdp_mmu_iter_cond_resched(kvm, &iter, false))
+		if (tdp_mmu_iter_cond_resched(kvm, &iter, false, false))
  			continue;
 
  		if (spte_ad_need_write_protect(iter.old_spte)) {
@@ -1193,7 +1237,7 @@ bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
  	int root_as_id;
  	bool spte_set = false;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root) {
+	for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
  		root_as_id = kvm_mmu_page_as_id(root);
  		if (root_as_id != slot->as_id)
  			continue;
@@ -1291,7 +1335,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
  	rcu_read_lock();
 
  	tdp_root_for_each_pte(iter, root, start, end) {
-		if (tdp_mmu_iter_cond_resched(kvm, &iter, spte_set)) {
+		if (tdp_mmu_iter_cond_resched(kvm, &iter, spte_set, false)) {
  			spte_set = false;
  			continue;
  		}
@@ -1326,7 +1370,7 @@ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
  	struct kvm_mmu_page *root;
  	int root_as_id;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root) {
+	for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
  		root_as_id = kvm_mmu_page_as_id(root);
  		if (root_as_id != slot->as_id)
  			continue;
arch/x86/kvm/mmu/tdp_mmu.h +4 -2
index 9961df505067..855e58856815 100644
@@ -13,9 +13,11 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,
  	return refcount_inc_not_zero(&root->tdp_mmu_root_count);
  }
 
-void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
+			  bool shared);
 
-bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end);
+bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end,
+			       bool shared);
  void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 
  int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
author
#17 lore raw

Re: [PATCH 09/13] KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock

Quoted 79 lines · Paolo Bonzini
Quoted 33 lines · Ben Gardon

To reduce lock contention and interference with page fault handlers, allow the TDP MMU function to zap a GFN range to operate under the MMU read lock.

---
  arch/x86/kvm/mmu/mmu.c     |  15 ++++--
  arch/x86/kvm/mmu/tdp_mmu.c | 102 ++++++++++++++++++++++++++-----------
  arch/x86/kvm/mmu/tdp_mmu.h |   6 ++-
  3 files changed, 87 insertions(+), 36 deletions(-)
arch/x86/kvm/mmu/mmu.c +4 -2
index 667d64daa82c..dcbfc784cf2f 100644
@@ -3155,7 +3155,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
      sp = to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK);
 
      if (is_tdp_mmu_page(sp))
-             kvm_tdp_mmu_put_root(kvm, sp);
+             kvm_tdp_mmu_put_root(kvm, sp, false);
      else if (!--sp->root_count && sp->role.invalid)
              kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 
@@ -5514,13 +5514,17 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
              }
      }
 
+     write_unlock(&kvm->mmu_lock);
+
      if (is_tdp_mmu_enabled(kvm)) {
-             flush = kvm_tdp_mmu_zap_gfn_range(kvm, gfn_start, gfn_end);
+             read_lock(&kvm->mmu_lock);

arch/x86/kvm/mmu/mmu.c void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) · line 5528

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +4 -2
+             flush = kvm_tdp_mmu_zap_gfn_range(kvm, gfn_start, gfn_end,
+                                               true);
              if (flush)
                      kvm_flush_remote_tlbs(kvm);
-     }
 
-     write_unlock(&kvm->mmu_lock);
+             read_unlock(&kvm->mmu_lock);
+     }
  }

This will conflict with Sean's MMU notifier series patches:

KVM: x86/mmu: Pass address space ID to __kvm_tdp_mmu_zap_gfn_range()

What I can do for now is change the mmu.c part of that patch to

arch/x86/kvm/mmu/mmu.c +6 -6
index e6e02360ef67..9882bbd9b742 100644
@@ -5510,15 +5510,15 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
                }
        }
 
-       if (flush)
-               kvm_flush_remote_tlbs_with_address(kvm, gfn_start, gfn_end);
-
        if (is_tdp_mmu_enabled(kvm)) {
-               flush = kvm_tdp_mmu_zap_gfn_range(kvm, gfn_start, gfn_end);
-               if (flush)
-                       kvm_flush_remote_tlbs(kvm);
+               for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
+                       flush = kvm_tdp_mmu_zap_gfn_range(kvm, i, gfn_start,
+                                                         gfn_end, flush);
        }
 
+       if (flush)
+               kvm_flush_remote_tlbs_with_address(kvm, gfn_start, gfn_end);
+
        write_unlock(&kvm->mmu_lock);
  }
 
Quoted 5 lines

but you will have to add a separate "if (flush)" when moving the write_unlock earlier, since there's no downgrade function for rwlocks. In practice it's not a huge deal since unless running nested there will be only one active MMU.

Paolo

Thank you for doing that. I also figured that the extra flushes when running nested would probably be worth it to get the parallelism gains. I don't mind working out those conflicts in v2.

Quoted 312 lines
Quoted 311 lines
arch/x86/kvm/mmu/mmu.c +2 -1
  static bool slot_rmap_write_protect(struct kvm *kvm,
@@ -5959,7 +5963,8 @@ static void kvm_recover_nx_lpages(struct kvm *kvm)
              WARN_ON_ONCE(!sp->lpage_disallowed);
              if (is_tdp_mmu_page(sp)) {
                      kvm_tdp_mmu_zap_gfn_range(kvm, sp->gfn,
-                             sp->gfn + KVM_PAGES_PER_HPAGE(sp->role.level));
+                             sp->gfn + KVM_PAGES_PER_HPAGE(sp->role.level),
+                             false);
              } else {
                      kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
                      WARN_ON_ONCE(sp->lpage_disallowed);
arch/x86/kvm/mmu/tdp_mmu.c +73 -29
index d255125059c4..0e99e4675dd4 100644
@@ -27,6 +27,15 @@ void kvm_mmu_init_tdp_mmu(struct kvm *kvm)
      INIT_LIST_HEAD(&kvm->arch.tdp_mmu_pages);
  }
 
+static __always_inline void kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,
+                                                          bool shared)
+{
+     if (shared)
+             lockdep_assert_held_read(&kvm->mmu_lock);
+     else
+             lockdep_assert_held_write(&kvm->mmu_lock);
+}
+
  void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
  {
      if (!kvm->arch.tdp_mmu_enabled)
@@ -42,7 +51,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
  }
 
  static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
-                       gfn_t start, gfn_t end, bool can_yield);
+                       gfn_t start, gfn_t end, bool can_yield, bool shared);
 
  static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
  {
@@ -66,11 +75,12 @@ static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head)
      tdp_mmu_free_sp(sp);
  }
 
-void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
+                       bool shared)
  {
      gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
 
-     lockdep_assert_held_write(&kvm->mmu_lock);
+     kvm_lockdep_assert_mmu_lock_held(kvm, shared);
 
      if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
              return;
@@ -81,7 +91,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
      list_del_rcu(&root->link);
      spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 
-     zap_gfn_range(kvm, root, 0, max_gfn, false);
+     zap_gfn_range(kvm, root, 0, max_gfn, false, shared);
 
      call_rcu(&root->rcu_head, tdp_mmu_free_sp_rcu_callback);
  }
@@ -94,11 +104,11 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
   * function will return NULL.
   */
  static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
-                                           struct kvm_mmu_page *prev_root)
+                                           struct kvm_mmu_page *prev_root,
+                                           bool shared)
  {
      struct kvm_mmu_page *next_root;
 
-     lockdep_assert_held_write(&kvm->mmu_lock);
 
      rcu_read_lock();
 
@@ -117,7 +127,7 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
      rcu_read_unlock();
 
      if (prev_root)
-             kvm_tdp_mmu_put_root(kvm, prev_root);
+             kvm_tdp_mmu_put_root(kvm, prev_root, shared);
 
      return next_root;
  }
@@ -127,11 +137,15 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
   * This makes it safe to release the MMU lock and yield within the loop, but
   * if exiting the loop early, the caller must drop the reference to the most
   * recent root. (Unless keeping a live reference is desirable.)
+ *
+ * If shared is set, this function is operating under the MMU lock in read
+ * mode. In the unlikely event that this thread must free a root, the lock
+ * will be temporarily dropped and reacquired in write mode.
   */
-#define for_each_tdp_mmu_root_yield_safe(_kvm, _root)        \
-     for (_root = tdp_mmu_next_root(_kvm, NULL);     \
-          _root;                                     \
-          _root = tdp_mmu_next_root(_kvm, _root))
+#define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _shared)       \
+     for (_root = tdp_mmu_next_root(_kvm, NULL, _shared);    \
+          _root;                                             \
+          _root = tdp_mmu_next_root(_kvm, _root, _shared))
 
  /* Only safe under the MMU lock in write mode, without yielding. */
  #define for_each_tdp_mmu_root(_kvm, _root)                          \
@@ -632,7 +646,8 @@ static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm,
   * Return false if a yield was not needed.
   */
  static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,
-                                          struct tdp_iter *iter, bool flush)
+                                          struct tdp_iter *iter, bool flush,
+                                          bool shared)
  {
      /* Ensure forward progress has been made before yielding. */
      if (iter->next_last_level_gfn == iter->yielded_gfn)
@@ -644,7 +659,11 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,
              if (flush)
                      kvm_flush_remote_tlbs(kvm);
 
-             cond_resched_rwlock_write(&kvm->mmu_lock);
+             if (shared)
+                     cond_resched_rwlock_read(&kvm->mmu_lock);
+             else
+                     cond_resched_rwlock_write(&kvm->mmu_lock);
+
              rcu_read_lock();
 
              WARN_ON(iter->gfn > iter->next_last_level_gfn);
@@ -662,23 +681,33 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,
   * non-root pages mapping GFNs strictly within that range. Returns true if
   * SPTEs have been cleared and a TLB flush is needed before releasing the
   * MMU lock.
+ *
   * If can_yield is true, will release the MMU lock and reschedule if the
   * scheduler needs the CPU or there is contention on the MMU lock. If this
   * function cannot yield, it will not release the MMU lock or reschedule and
   * the caller must ensure it does not supply too large a GFN range, or the
   * operation can cause a soft lockup.
+ *
+ * If shared is true, this thread holds the MMU lock in read mode and must
+ * account for the possibility that other threads are modifying the paging
+ * structures concurrently. If shared is false, this thread should hold the
+ * MMU lock in write mode.
   */
  static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
-                       gfn_t start, gfn_t end, bool can_yield)
+                       gfn_t start, gfn_t end, bool can_yield, bool shared)
  {
      struct tdp_iter iter;
      bool flush_needed = false;
 
+     kvm_lockdep_assert_mmu_lock_held(kvm, shared);
+
      rcu_read_lock();
 
      tdp_root_for_each_pte(iter, root, start, end) {
+retry:
              if (can_yield &&
-                 tdp_mmu_iter_cond_resched(kvm, &iter, flush_needed)) {
+                 tdp_mmu_iter_cond_resched(kvm, &iter, flush_needed,
+                                           shared)) {
                      flush_needed = false;
                      continue;
              }
@@ -696,8 +725,17 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
                  !is_last_spte(iter.old_spte, iter.level))
                      continue;
 
-             tdp_mmu_set_spte(kvm, &iter, 0);
-             flush_needed = true;
+             if (!shared) {
+                     tdp_mmu_set_spte(kvm, &iter, 0);
+                     flush_needed = true;
+             } else if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) {
+                     /*
+                      * The iter must explicitly re-read the SPTE because
+                      * the atomic cmpxchg failed.
+                      */
+                     iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
+                     goto retry;
+             }
      }
 
      rcu_read_unlock();
@@ -709,14 +747,20 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
   * non-root pages mapping GFNs strictly within that range. Returns true if
   * SPTEs have been cleared and a TLB flush is needed before releasing the
   * MMU lock.
+ *
+ * If shared is true, this thread holds the MMU lock in read mode and must
+ * account for the possibility that other threads are modifying the paging
+ * structures concurrently. If shared is false, this thread should hold the
+ * MMU in write mode.
   */
-bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end)
+bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end,
+                            bool shared)
  {
      struct kvm_mmu_page *root;
      bool flush = false;
 
-     for_each_tdp_mmu_root_yield_safe(kvm, root)
-             flush |= zap_gfn_range(kvm, root, start, end, true);
+     for_each_tdp_mmu_root_yield_safe(kvm, root, shared)
+             flush |= zap_gfn_range(kvm, root, start, end, true, shared);
 
      return flush;
  }
@@ -726,7 +770,7 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm)
      gfn_t max_gfn = 1ULL << (shadow_phys_bits - PAGE_SHIFT);
      bool flush;
 
-     flush = kvm_tdp_mmu_zap_gfn_range(kvm, 0, max_gfn);
+     flush = kvm_tdp_mmu_zap_gfn_range(kvm, 0, max_gfn, false);
      if (flush)
              kvm_flush_remote_tlbs(kvm);
  }
@@ -893,7 +937,7 @@ static __always_inline int kvm_tdp_mmu_handle_hva_range(struct kvm *kvm,
      int ret = 0;
      int as_id;
 
-     for_each_tdp_mmu_root_yield_safe(kvm, root) {
+     for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
              as_id = kvm_mmu_page_as_id(root);
              slots = __kvm_memslots(kvm, as_id);
              kvm_for_each_memslot(memslot, slots) {
@@ -933,7 +977,7 @@ static int zap_gfn_range_hva_wrapper(struct kvm *kvm,
                                   struct kvm_mmu_page *root, gfn_t start,
                                   gfn_t end, unsigned long unused)
  {
-     return zap_gfn_range(kvm, root, start, end, false);
+     return zap_gfn_range(kvm, root, start, end, false, false);
  }
 
  int kvm_tdp_mmu_zap_hva_range(struct kvm *kvm, unsigned long start,
@@ -1098,7 +1142,7 @@ static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
 
      for_each_tdp_pte_min_level(iter, root->spt, root->role.level,
                                 min_level, start, end) {
-             if (tdp_mmu_iter_cond_resched(kvm, &iter, false))
+             if (tdp_mmu_iter_cond_resched(kvm, &iter, false, false))
                      continue;
 
              if (!is_shadow_present_pte(iter.old_spte) ||
@@ -1128,7 +1172,7 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, struct kvm_memory_slot *slot,
      int root_as_id;
      bool spte_set = false;
 
-     for_each_tdp_mmu_root_yield_safe(kvm, root) {
+     for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
              root_as_id = kvm_mmu_page_as_id(root);
              if (root_as_id != slot->as_id)
                      continue;
@@ -1157,7 +1201,7 @@ static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
      rcu_read_lock();
 
      tdp_root_for_each_leaf_pte(iter, root, start, end) {
-             if (tdp_mmu_iter_cond_resched(kvm, &iter, false))
+             if (tdp_mmu_iter_cond_resched(kvm, &iter, false, false))
                      continue;
 
              if (spte_ad_need_write_protect(iter.old_spte)) {
@@ -1193,7 +1237,7 @@ bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, struct kvm_memory_slot *slot)
      int root_as_id;
      bool spte_set = false;
 
-     for_each_tdp_mmu_root_yield_safe(kvm, root) {
+     for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
              root_as_id = kvm_mmu_page_as_id(root);
              if (root_as_id != slot->as_id)
                      continue;
@@ -1291,7 +1335,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
      rcu_read_lock();
 
      tdp_root_for_each_pte(iter, root, start, end) {
-             if (tdp_mmu_iter_cond_resched(kvm, &iter, spte_set)) {
+             if (tdp_mmu_iter_cond_resched(kvm, &iter, spte_set, false)) {
                      spte_set = false;
                      continue;
              }
@@ -1326,7 +1370,7 @@ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
      struct kvm_mmu_page *root;
      int root_as_id;
 
-     for_each_tdp_mmu_root_yield_safe(kvm, root) {
+     for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
              root_as_id = kvm_mmu_page_as_id(root);
              if (root_as_id != slot->as_id)
                      continue;
arch/x86/kvm/mmu/tdp_mmu.h +4 -2
index 9961df505067..855e58856815 100644
@@ -13,9 +13,11 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,
      return refcount_inc_not_zero(&root->tdp_mmu_root_count);
  }
 
-void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root);
+void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
+                       bool shared);
 
-bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end);
+bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end,
+                            bool shared);
  void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 
  int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
author
#18 lore raw

PATCH 10/13 KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock

In reply to Ben Gardon

To speed the process of disabling dirty logging, change the TDP MMU function which zaps collapsible SPTEs to run under the MMU read lock.

---
 arch/x86/kvm/mmu/mmu.c     |  9 ++++++---
 arch/x86/kvm/mmu/tdp_mmu.c | 17 +++++++++++++----
 2 files changed, 19 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +6 -3
index dcbfc784cf2f..81967b4e7d76 100644
@@ -5610,10 +5610,13 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
 
 	write_lock(&kvm->mmu_lock);
 	slot_handle_leaf(kvm, slot, kvm_mmu_zap_collapsible_spte, true);
-
-	if (is_tdp_mmu_enabled(kvm))
-		kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot);
 	write_unlock(&kvm->mmu_lock);
+
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		kvm_tdp_mmu_zap_collapsible_sptes(kvm, memslot);
+		read_unlock(&kvm->mmu_lock);
+	}
 }
 
 void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
arch/x86/kvm/mmu/tdp_mmu.c +13 -4
index 0e99e4675dd4..862acb868abd 100644
@@ -1335,7 +1335,8 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 	rcu_read_lock();
 
 	tdp_root_for_each_pte(iter, root, start, end) {
-		if (tdp_mmu_iter_cond_resched(kvm, &iter, spte_set, false)) {
+retry:
+		if (tdp_mmu_iter_cond_resched(kvm, &iter, spte_set, true)) {
 			spte_set = false;
 			continue;
 		}
@@ -1350,8 +1351,14 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 							    pfn, PG_LEVEL_NUM))
 			continue;
 
-		tdp_mmu_set_spte(kvm, &iter, 0);
-
+		if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) {
+			/*
+			 * The iter must explicitly re-read the SPTE because
+			 * the atomic cmpxchg failed.
+			 */
+			iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
+			goto retry;
+		}
 		spte_set = true;
 	}
 
@@ -1370,7 +1377,9 @@ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
 	struct kvm_mmu_page *root;
 	int root_as_id;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root, false) {
+	lockdep_assert_held_read(&kvm->mmu_lock);
+
+	for_each_tdp_mmu_root_yield_safe(kvm, root, true) {
 		root_as_id = kvm_mmu_page_as_id(root);
 		if (root_as_id != slot->as_id)
 			continue;
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
maintainer
#19 lore raw

Re: [PATCH 10/13] KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock

Quoted 9 lines · Ben Gardon
arch/x86/kvm/mmu/mmu.c +0 -1
index dcbfc784cf2f..81967b4e7d76 100644
@@ -5610,10 +5610,13 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
 
  	write_lock(&kvm->mmu_lock);
  	slot_handle_leaf(kvm, slot, kvm_mmu_zap_collapsible_spte, true);
-

arch/x86/kvm/mmu/mmu.c void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, · line 5620

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +6 -2
-	if (is_tdp_mmu_enabled(kvm))
-		kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot);
  	write_unlock(&kvm->mmu_lock);
+
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		kvm_tdp_mmu_zap_collapsible_sptes(kvm, memslot);
+		read_unlock(&kvm->mmu_lock);
+	}
  }

Same here, this will conflict with

KVM: x86/mmu: Coalesce TLB flushes when zapping collapsible SPTEs

Again, you will have to add back some "if (flush)" before the write_unlock.

Paolo

author
#20 lore raw

PATCH 11/13 KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read lock

In reply to Ben Gardon

To reduce lock contention and interference with page fault handlers, allow the TDP MMU functions which enable and disable dirty logging to operate under the MMU read lock.

---
 arch/x86/kvm/mmu/mmu.c     | 16 +++++++---
 arch/x86/kvm/mmu/tdp_mmu.c | 62 ++++++++++++++++++++++++++++++--------
 2 files changed, 61 insertions(+), 17 deletions(-)
arch/x86/kvm/mmu/mmu.c +12 -4
index 81967b4e7d76..bf535c9f7ff2 100644
@@ -5543,10 +5543,14 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm,
 	write_lock(&kvm->mmu_lock);
 	flush = slot_handle_level(kvm, memslot, slot_rmap_write_protect,
 				start_level, KVM_MAX_HUGEPAGE_LEVEL, false);
-	if (is_tdp_mmu_enabled(kvm))
-		flush |= kvm_tdp_mmu_wrprot_slot(kvm, memslot, PG_LEVEL_4K);
 	write_unlock(&kvm->mmu_lock);
 
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		flush |= kvm_tdp_mmu_wrprot_slot(kvm, memslot, PG_LEVEL_4K);
+		read_unlock(&kvm->mmu_lock);
+	}
+
 	/*
 	 * We can flush all the TLBs out of the mmu lock without TLB
 	 * corruption since we just change the spte from writable to
@@ -5641,10 +5645,14 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
 
 	write_lock(&kvm->mmu_lock);
 	flush = slot_handle_leaf(kvm, memslot, __rmap_clear_dirty, false);
-	if (is_tdp_mmu_enabled(kvm))
-		flush |= kvm_tdp_mmu_clear_dirty_slot(kvm, memslot);
 	write_unlock(&kvm->mmu_lock);
 
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		flush |= kvm_tdp_mmu_clear_dirty_slot(kvm, memslot);
+		read_unlock(&kvm->mmu_lock);
+	}
+
 	/*
 	 * It's also safe to flush TLBs out of mmu lock here as currently this
 	 * function is only used for dirty logging, in which case flushing TLB
arch/x86/kvm/mmu/tdp_mmu.c +49 -13

117 lines not on this page · raw message

Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
author
#21 lore raw

PATCH 12/13 KVM: x86/mmu: Fast invalidation for TDP MMU

In reply to Ben Gardon

Provide a real mechanism for fast invalidation by marking roots as invalid so that their reference count will quickly fall to zero and they will be torn down.

One negative side affect of this approach is that a vCPU thread will likely drop the last reference to a root and be saddled with the work of tearing down an entire paging structure. This issue will be resolved in a later commit.

---
 arch/x86/kvm/mmu/mmu.c     |  6 +++---
 arch/x86/kvm/mmu/tdp_mmu.c | 14 ++++++++++++++
 arch/x86/kvm/mmu/tdp_mmu.h |  5 +++++
 3 files changed, 22 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -3
index bf535c9f7ff2..49b7097fb55b 100644
@@ -5430,6 +5430,9 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 	write_lock(&kvm->mmu_lock);
 	trace_kvm_mmu_zap_all_fast(kvm);
 
+	if (is_tdp_mmu_enabled(kvm))
+		kvm_tdp_mmu_invalidate_roots(kvm);
+
 	/*
 	 * Toggle mmu_valid_gen between '0' and '1'.  Because slots_lock is
 	 * held for the entire duration of zapping obsolete pages, it's
@@ -5451,9 +5454,6 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 
 	kvm_zap_obsolete_pages(kvm);
 
-	if (is_tdp_mmu_enabled(kvm))
-		kvm_tdp_mmu_zap_all(kvm);
-
 	write_unlock(&kvm->mmu_lock);
 }
 
arch/x86/kvm/mmu/tdp_mmu.c +14 -0
index 0c90dc034819..428ff6778426 100644
@@ -789,6 +789,20 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm)
 		kvm_flush_remote_tlbs(kvm);
 }
 
+/*
+ * This function depends on running in the same MMU lock cirical section as
+ * kvm_reload_remote_mmus. Since this is in the same critical section, no new
+ * roots will be created between this function and the MMU reload signals
+ * being sent.
+ */
+void kvm_tdp_mmu_invalidate_roots(struct kvm *kvm)
+{
+	struct kvm_mmu_page *root;
+
+	for_each_tdp_mmu_root(kvm, root)
+		root->role.invalid = true;
+}
+
 /*
  * Installs a last-level SPTE to handle a TDP page fault.
  * (NPT/EPT violation/misconfiguration)
arch/x86/kvm/mmu/tdp_mmu.h +5 -0
index 855e58856815..ff4978817fb8 100644
@@ -10,6 +10,9 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
 __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,
 						     struct kvm_mmu_page *root)
 {
+	if (root->role.invalid)
+		return false;
+
 	return refcount_inc_not_zero(&root->tdp_mmu_root_count);
 }
 
@@ -20,6 +23,8 @@ bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end,
 			       bool shared);
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 
+void kvm_tdp_mmu_invalidate_roots(struct kvm *kvm);
+
 int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
 		    int map_writable, int max_level, kvm_pfn_t pfn,
 		    bool prefault);
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
#22 lore raw

Re: [PATCH 12/13] KVM: x86/mmu: Fast invalidation for TDP MMU

Quoted 11 lines · Ben Gardon

Provide a real mechanism for fast invalidation by marking roots as invalid so that their reference count will quickly fall to zero and they will be torn down.

One negative side affect of this approach is that a vCPU thread will likely drop the last reference to a root and be saddled with the work of tearing down an entire paging structure. This issue will be resolved in a later commit.

---

...

Quoted 5 lines

+/*
+ * This function depends on running in the same MMU lock cirical section as
+ * kvm_reload_remote_mmus. Since this is in the same critical section, no new
+ * roots will be created between this function and the MMU reload signals
+ * being sent.

Eww. That goes beyond just adding a lockdep assertion here. I know you want to isolate the TDP MMU as much as possible, but this really feels like it should be open coded in kvm_mmu_zap_all_fast(). And assuming this lands after as_id is added to for_each_tdp_mmu_root(), it's probably easier to open code anyways, e.g. use list_for_each_entry() directly instead of bouncing through an iterator.

Quoted 8 lines

+ */

arch/x86/kvm/mmu/tdp_mmu.c +7 -0
+void kvm_tdp_mmu_invalidate_roots(struct kvm *kvm)
+{
+	struct kvm_mmu_page *root;
+
+	for_each_tdp_mmu_root(kvm, root)
+		root->role.invalid = true;
+}
author
#23 lore raw

Re: [PATCH 12/13] KVM: x86/mmu: Fast invalidation for TDP MMU

Quoted 22 lines · Sean Christopherson
Quoted 11 lines · Ben Gardon

Provide a real mechanism for fast invalidation by marking roots as invalid so that their reference count will quickly fall to zero and they will be torn down.

One negative side affect of this approach is that a vCPU thread will likely drop the last reference to a root and be saddled with the work of tearing down an entire paging structure. This issue will be resolved in a later commit.

---

...

Quoted 5 lines

+/*
+ * This function depends on running in the same MMU lock cirical section as
+ * kvm_reload_remote_mmus. Since this is in the same critical section, no new
+ * roots will be created between this function and the MMU reload signals
+ * being sent.

Quoted 5 lines

Eww. That goes beyond just adding a lockdep assertion here. I know you want to isolate the TDP MMU as much as possible, but this really feels like it should be open coded in kvm_mmu_zap_all_fast(). And assuming this lands after as_id is added to for_each_tdp_mmu_root(), it's probably easier to open code anyways, e.g. use list_for_each_entry() directly instead of bouncing through an iterator.

Yeah, that's fair. I'll open-code it here. I agree that it will remove confusion from the function, though it would be nice to be able to use for_each_tdp_mmu_root for the lockdep and rcu annotations.

Quoted 9 lines
Quoted 9 lines

+ */

arch/x86/kvm/mmu/tdp_mmu.c +7 -0
+void kvm_tdp_mmu_invalidate_roots(struct kvm *kvm)
+{
+     struct kvm_mmu_page *root;
+
+     for_each_tdp_mmu_root(kvm, root)
+             root->role.invalid = true;
+}
maintainer
#24 lore raw

Re: [PATCH 12/13] KVM: x86/mmu: Fast invalidation for TDP MMU

In reply to Ben Gardon
Quoted 8 lines · Ben Gardon
arch/x86/kvm/mmu/mmu.c +3 -0
+	if (is_tdp_mmu_enabled(kvm))
+		kvm_tdp_mmu_invalidate_roots(kvm);
+
  	/*
  	 * Toggle mmu_valid_gen between '0' and '1'.  Because slots_lock is
  	 * held for the entire duration of zapping obsolete pages, it's

arch/x86/kvm/mmu/mmu.c static void kvm_mmu_zap_all_fast(struct kvm *kvm) · line 5456

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +0 -3
@@ -5451,9 +5454,6 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 
  	kvm_zap_obsolete_pages(kvm);
 
-	if (is_tdp_mmu_enabled(kvm))
-		kvm_tdp_mmu_zap_all(kvm);
-

This is just cosmetic, but I'd prefer to keep the call to kvm_tdp_mmu_invalidate_roots at the original place, so that it's clear in the next patch that it's two separate parts because of the different locking requirements.

Paolo

author
#25 lore raw

Re: [PATCH 12/13] KVM: x86/mmu: Fast invalidation for TDP MMU

Quoted 13 lines · Paolo Bonzini
Quoted 11 lines · Ben Gardon
arch/x86/kvm/mmu/mmu.c +3 -0
+     if (is_tdp_mmu_enabled(kvm))
+             kvm_tdp_mmu_invalidate_roots(kvm);
+
      /*
       * Toggle mmu_valid_gen between '0' and '1'.  Because slots_lock is
       * held for the entire duration of zapping obsolete pages, it's
@@ -5451,9 +5454,6 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 
      kvm_zap_obsolete_pages(kvm);
 
Quoted 8 lines

arch/x86/kvm/mmu/mmu.c static void kvm_mmu_zap_all_fast(struct kvm *kvm) · line 5456

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +0 -3
-     if (is_tdp_mmu_enabled(kvm))
-             kvm_tdp_mmu_zap_all(kvm);
-

This is just cosmetic, but I'd prefer to keep the call to kvm_tdp_mmu_invalidate_roots at the original place, so that it's clear in the next patch that it's two separate parts because of the different locking requirements.

I'm not sure exactly what you mean and I could certainly do a better job explaining in the commit description, but it's actually quite important that kvm_tdp_mmu_invalidate_roots at least precede kvm_zap_obsolete_pages as kvm_zap_obsolete_pages drops the lock and yields. If kvm_tdp_mmu_invalidate_roots doesn't go first then vCPUs could wind up dropping their ref on an old root and then picking it up again before the last root had a chance to drop its ref.

Explaining in the description that kvm_tdp_mmu_zap_all is being dropped because it is no longer necessary (as opposed to being moved) might help make that cleaner.

Alternatively I could just leave kvm_tdp_mmu_zap_all and replace it in the next patch.

Quoted 3 lines

Paolo

maintainer
#26 lore raw

Re: [PATCH 12/13] KVM: x86/mmu: Fast invalidation for TDP MMU

Quoted 4 lines · Ben Gardon
Quoted 4 lines

This is just cosmetic, but I'd prefer to keep the call to kvm_tdp_mmu_invalidate_roots at the original place, so that it's clear in the next patch that it's two separate parts because of the different locking requirements.

Quoted 10 lines

I'm not sure exactly what you mean and I could certainly do a better job explaining in the commit description, but it's actually quite important that kvm_tdp_mmu_invalidate_roots at least precede kvm_zap_obsolete_pages as kvm_zap_obsolete_pages drops the lock and yields. If kvm_tdp_mmu_invalidate_roots doesn't go first then vCPUs could wind up dropping their ref on an old root and then picking it up again before the last root had a chance to drop its ref. Explaining in the description that kvm_tdp_mmu_zap_all is being dropped because it is no longer necessary (as opposed to being moved) might help make that cleaner.

No, what would help is the remark you just made about kvm_zap_obsolete_pages yielding. But that doesn't matter after 13/13 though, does it? Perhaps it's easier to just combine them.

Paolo

author
#27 lore raw

PATCH 13/13 KVM: x86/mmu: Tear down roots in fast invalidation thread

In reply to Ben Gardon

To avoid saddling a vCPU thread with the work of tearing down an entire paging structure, take a reference on each root before they become obsolete, so that the thread initiating the fast invalidation can tear down the paging structure and (most likely) release the last reference. As a bonus, this teardown can happen under the MMU lock in read mode so as not to block the progress of vCPU threads.

---
 arch/x86/kvm/mmu/mmu.c     |  6 ++++
 arch/x86/kvm/mmu/tdp_mmu.c | 74 +++++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/mmu/tdp_mmu.h |  1 +
 3 files changed, 80 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +6 -0
index 49b7097fb55b..22742619698d 100644
@@ -5455,6 +5455,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 	kvm_zap_obsolete_pages(kvm);
 
 	write_unlock(&kvm->mmu_lock);
+
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		kvm_tdp_mmu_zap_all_fast(kvm);
+		read_unlock(&kvm->mmu_lock);
+	}
 }
 
 static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
arch/x86/kvm/mmu/tdp_mmu.c +73 -1

86 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +1 -0
index ff4978817fb8..d6d98f9047cd 100644
@@ -24,6 +24,7 @@ bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end,
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 
 void kvm_tdp_mmu_invalidate_roots(struct kvm *kvm);
+void kvm_tdp_mmu_zap_all_fast(struct kvm *kvm);
 
 int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
 		    int map_writable, int max_level, kvm_pfn_t pfn,
Signature · 1 line
2.31.0.291.g576ba9dcdaf-goog
#28 lore raw

Re: [PATCH 13/13] KVM: x86/mmu: Tear down roots in fast invalidation thread

Quoted 11 lines · Ben Gardon
---
 arch/x86/kvm/mmu/mmu.c     |  6 ++++
 arch/x86/kvm/mmu/tdp_mmu.c | 74 +++++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/mmu/tdp_mmu.h |  1 +
 3 files changed, 80 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index 49b7097fb55b..22742619698d 100644

arch/x86/kvm/mmu/mmu.c static void kvm_mmu_zap_all_fast(struct kvm *kvm) · line 5461

Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +4 -0
@@ -5455,6 +5455,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 	kvm_zap_obsolete_pages(kvm);
 
 	write_unlock(&kvm->mmu_lock);
+
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		kvm_tdp_mmu_zap_all_fast(kvm);

Purely because it exists first, I think we should follow the legacy MMU's terminology, i.e. kvm_tdp_mmu_zap_obsolete_pages().

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +2 -0
+		read_unlock(&kvm->mmu_lock);
+	}
 }
maintainer
#29 lore raw

Re: [PATCH 13/13] KVM: x86/mmu: Tear down roots in fast invalidation thread

Quoted 6 lines · Sean Christopherson

arch/x86/kvm/mmu/mmu.c static void kvm_mmu_zap_all_fast(struct kvm *kvm) · line 5461

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+	if (is_tdp_mmu_enabled(kvm)) {
+		read_lock(&kvm->mmu_lock);
+		kvm_tdp_mmu_zap_all_fast(kvm);

Purely because it exists first, I think we should follow the legacy MMU's terminology, i.e. kvm_tdp_mmu_zap_obsolete_pages().

It's a bit different, obsolete pages in kvm_zap_obsolete_pages have an old mmu_valid_gen. They are not invalid, so it is inaccurate to talk about obsolete pages in the context of the TDP MMU. But I agree that the function should be renamed.

Paolo

maintainer
#30 lore raw

Re: [PATCH 13/13] KVM: x86/mmu: Tear down roots in fast invalidation thread

In reply to Ben Gardon

arch/x86/kvm/mmu/tdp_mmu.c void kvm_tdp_mmu_zap_all(struct kvm *kvm) · line 841

Quoted 9 lines · Ben Gardon
arch/x86/kvm/mmu/tdp_mmu.c +9 -0
+/*
+ * Since kvm_tdp_mmu_invalidate_roots has acquired a reference to each
+ * invalidated root, they will not be freed until this function drops the
+ * reference. Before dropping that reference, tear down the paging
+ * structure so that whichever thread does drop the last reference
+ * only has to do a trivial ammount of work. Since the roots are invalid,
+ * no new SPTEs should be created under them.
+ */
+void kvm_tdp_mmu_zap_all_fast(struct kvm *kvm)

Please rename this to kvm_tdp_mmu_zap_invalidated_roots.

Paolo

maintainer
#31 lore raw
In reply to Ben Gardon
Quoted 11 lines · Ben Gardon

Now that the TDP MMU is able to handle page faults in parallel, it's a relatively small change to expand to other operations. This series allows zapping a range of GFNs, reclaiming collapsible SPTEs (when disabling dirty logging), and enabling dirty logging to all happen under the MMU lock in read mode.

This is partly a cleanup + rewrite of the last few patches of the parallel page faults series. I've incorporated feedback from Sean and Paolo, but the patches have changed so much that I'm sending this as a separate series.

Quoted 7 lines

Ran kvm-unit-tests + selftests on an SMP kernel + Intel Skylake, with the TDP MMU enabled and disabled. This series introduces no new failures or warnings.

I know this will conflict horribly with the patches from Sean's series which were just queued, and I'll send a v2 to fix those conflicts + address any feedback on this v1.

Mostly looks good (the only substantial remark is from Sean's reply to patch 7); I've made a couple adjustments to the patches I had queued to ease the fixing of conflicts. The patches should hit kvm/queue later today, right after I send my 5.11 pull request to Linus.

I have also just sent a completely unrelated remark on the existing page fault function, which was prompted by reviewing these patches. If you agree, I can take care of the change or you can include it in v2, as you prefer (I don't expect conflicts).

Paolo