Skip to contentSkip to conversation

KVM: x86/mmu: Drop async zapping of TDP MMU roots

PATCH Applied Sean Christopherson 2 participants 9 messages Sep 16, 2023 – Sep 22, 2023

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

cover KVM: x86/mmu: Drop async zapping of TDP MMU roots

Yank out the asynchronous zapping of TDP MMU roots. In some setups, using unbounded workqueues can consumes all CPUs for extended durations, and create significant jitter in the system.

Specifically, the behavior causes audio glitches in ChromeOS VMs with virtio-gpu when running games in the guest. Gory details in patch 3.

I tagged all of this for stable so that this gets back to v6.1 (I already did the backport to verify it's not awful). This bug is bad enough that the workaround for the ChromeOS usecase is to simply disable the TDP MMU, which I really do not want to do for the v6.1 kernel (or the v6.6. kernel).

Sean Christopherson (3):
KVM: x86/mmu: Open code walking TDP MMU roots for mmu_notifier's zap
SPTEs
KVM: x86/mmu: Take "shared" instead of "as_id" TDP MMU's yield-safe
iterator
KVM: x86/mmu: Stop zapping invalidated TDP MMU roots asynchronously

 arch/x86/include/asm/kvm_host.h |   3 +-
 arch/x86/kvm/mmu/mmu.c          |  21 ++---
 arch/x86/kvm/mmu/mmu_internal.h |  13 ++-
 arch/x86/kvm/mmu/tdp_mmu.c      | 147 ++++++++++++++------------------
 arch/x86/kvm/mmu/tdp_mmu.h      |   5 +-
 arch/x86/kvm/x86.c              |   5 +-
 6 files changed, 80 insertions(+), 114 deletions(-)

base-commit: 0bb80ecc33a8fb5a682236443c1e740d5c917d1d

Signature · 1 line
2.42.0.459.ge4e396fd5e-goog
author maintainer
#1 lore raw

PATCH 1/3 KVM: x86/mmu: Open code walking TDP MMU roots for mmu_notifier's zap SPTEs

Use the "inner" TDP MMU root walker when zapping SPTEs in response to an mmu_notifier invalidation instead of invoking kvm_tdp_mmu_zap_leafs(). This will allow reworking for_each_tdp_mmu_root_yield_safe() to do more work, and to also make it usable in more places, without increasing the number of params to the point where it adds no value.

The mmu_notifier path is a bit of a special snowflake, e.g. it zaps only a single address space (because it's per-slot), and can't always yield.

Drop the @can_yield param from tdp_mmu_zap_leafs() as its sole remaining caller unconditionally passes "true".

---
 arch/x86/kvm/mmu/mmu.c     |  2 +-
 arch/x86/kvm/mmu/tdp_mmu.c | 13 +++++++++----
 arch/x86/kvm/mmu/tdp_mmu.h |  4 ++--
 3 files changed, 12 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index e1d011c67cc6..59f5e40b8f55 100644
@@ -6260,7 +6260,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 	if (tdp_mmu_enabled) {
 		for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
 			flush = kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start,
-						      gfn_end, true, flush);
+						      gfn_end, flush);
 	}
 
 	if (flush)
arch/x86/kvm/mmu/tdp_mmu.c +9 -4
index 6c63f2d1675f..89aaa2463373 100644
@@ -878,12 +878,12 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,
  * more SPTEs were zapped since the MMU lock was last acquired.
  */
 bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end,
-			   bool can_yield, bool flush)
+			   bool flush)
 {
 	struct kvm_mmu_page *root;
 
 	for_each_tdp_mmu_root_yield_safe(kvm, root, as_id)
-		flush = tdp_mmu_zap_leafs(kvm, root, start, end, can_yield, flush);
+		flush = tdp_mmu_zap_leafs(kvm, root, start, end, true, flush);
 
 	return flush;
 }
@@ -1146,8 +1146,13 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 bool kvm_tdp_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range,
 				 bool flush)
 {
-	return kvm_tdp_mmu_zap_leafs(kvm, range->slot->as_id, range->start,
-				     range->end, range->may_block, flush);
+	struct kvm_mmu_page *root;
+
+	__for_each_tdp_mmu_root_yield_safe(kvm, root, range->slot->as_id, false, false)
+		flush = tdp_mmu_zap_leafs(kvm, root, range->start, range->end,
+					  range->may_block, flush);
+
+	return flush;
 }
 
 typedef bool (*tdp_handler_t)(struct kvm *kvm, struct tdp_iter *iter,
arch/x86/kvm/mmu/tdp_mmu.h +2 -2
index 0a63b1afabd3..eb4fa345d3a4 100644
@@ -20,8 +20,8 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root)
 void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
 			  bool shared);
 
-bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start,
-				 gfn_t end, bool can_yield, bool flush);
+bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end,
+			   bool flush);
 bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp);
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm);
Signature · 1 line
2.42.0.459.ge4e396fd5e-goog
maintainer
#2 lore raw

Re: [PATCH 1/3] KVM: x86/mmu: Open code walking TDP MMU roots for mmu_notifier's zap SPTEs

Quoted 11 lines · Sean Christopherson

Use the "inner" TDP MMU root walker when zapping SPTEs in response to an mmu_notifier invalidation instead of invoking kvm_tdp_mmu_zap_leafs(). This will allow reworking for_each_tdp_mmu_root_yield_safe() to do more work, and to also make it usable in more places, without increasing the number of params to the point where it adds no value.

The mmu_notifier path is a bit of a special snowflake, e.g. it zaps only a single address space (because it's per-slot), and can't always yield.

Drop the @can_yield param from tdp_mmu_zap_leafs() as its sole remaining caller unconditionally passes "true".

Slightly rewritten commit log:

---
The mmu_notifier path is a bit of a special snowflake, e.g. it zaps only a single address space (because it's per-slot), and can't always yield. Because of this, it calls kvm_tdp_mmu_zap_leafs() in ways that no one else does.

Iterate manually over the leafs in response to an mmu_notifier invalidation, instead of invoking kvm_tdp_mmu_zap_leafs(). Drop the @can_yield param from kvm_tdp_mmu_zap_leafs() as its sole remaining caller unconditionally passes "true".
---

and using the "__" macro can be moved to the second patch.

Paolo

author maintainer
#3 lore raw

PATCH 2/3 KVM: x86/mmu: Take "shared" instead of "as_id" TDP MMU's yield-safe iterator

In reply to Sean Christopherson

Replace the address space ID in for_each_tdp_mmu_root_yield_safe() with a shared (vs. exclusive) param, and have the walker iterate over all address spaces as all callers want to process all address spaces. Drop the @as_id param as well as the manual address space iteration in callers.

Add the @shared param even though the two current callers pass "false" unconditionally, as the main reason for refactoring the walker is to simplify using it to zap invalid TDP MMU roots, which is done with mmu_lock held for read.

---
 arch/x86/kvm/mmu/mmu.c     |  8 ++------
 arch/x86/kvm/mmu/tdp_mmu.c | 20 ++++++++++----------
 arch/x86/kvm/mmu/tdp_mmu.h |  3 +--
 3 files changed, 13 insertions(+), 18 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -6
index 59f5e40b8f55..54f94f644b42 100644
@@ -6246,7 +6246,6 @@ static bool kvm_rmap_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_e
 void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 {
 	bool flush;
-	int i;
 
 	if (WARN_ON_ONCE(gfn_end <= gfn_start))
 		return;
@@ -6257,11 +6256,8 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 
 	flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
 
-	if (tdp_mmu_enabled) {
-		for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
-			flush = kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start,
-						      gfn_end, flush);
-	}
+	if (tdp_mmu_enabled)
+		flush = kvm_tdp_mmu_zap_leafs(kvm, gfn_start, gfn_end, flush);
 
 	if (flush)
 		kvm_flush_remote_tlbs_range(kvm, gfn_start, gfn_end - gfn_start);
arch/x86/kvm/mmu/tdp_mmu.c +10 -10
index 89aaa2463373..7cb1902ae032 100644
@@ -211,8 +211,12 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 #define for_each_valid_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared)	\
 	__for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, true)
 
-#define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id)			\
-	__for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, false, false)
+#define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _shared)			\
+	for (_root = tdp_mmu_next_root(_kvm, NULL, _shared, false);		\
+	     _root;								\
+	     _root = tdp_mmu_next_root(_kvm, _root, _shared, false))		\
+		if (!kvm_lockdep_assert_mmu_lock_held(_kvm, _shared)) {		\
+		} else
 
 /*
  * Iterate over all TDP MMU roots.  Requires that mmu_lock be held for write,
@@ -877,12 +881,11 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,
  * true if a TLB flush is needed before releasing the MMU lock, i.e. if one or
  * more SPTEs were zapped since the MMU lock was last acquired.
  */
-bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end,
-			   bool flush)
+bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, gfn_t start, gfn_t end, bool flush)
 {
 	struct kvm_mmu_page *root;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root, as_id)
+	for_each_tdp_mmu_root_yield_safe(kvm, root, false)
 		flush = tdp_mmu_zap_leafs(kvm, root, start, end, true, flush);
 
 	return flush;
@@ -891,7 +894,6 @@ bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end,
 void kvm_tdp_mmu_zap_all(struct kvm *kvm)
 {
 	struct kvm_mmu_page *root;
-	int i;
 
 	/*
 	 * Zap all roots, including invalid roots, as all SPTEs must be dropped
@@ -905,10 +907,8 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm)
 	 * is being destroyed or the userspace VMM has exited.  In both cases,
 	 * KVM_RUN is unreachable, i.e. no vCPUs will ever service the request.
 	 */
-	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
-		for_each_tdp_mmu_root_yield_safe(kvm, root, i)
-			tdp_mmu_zap_root(kvm, root, false);
-	}
+	for_each_tdp_mmu_root_yield_safe(kvm, root, false)
+		tdp_mmu_zap_root(kvm, root, false);
 }
 
 /*
arch/x86/kvm/mmu/tdp_mmu.h +1 -2
index eb4fa345d3a4..bc088953f929 100644
@@ -20,8 +20,7 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root)
 void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
 			  bool shared);
 
-bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end,
-			   bool flush);
+bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, gfn_t start, gfn_t end, bool flush);
 bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp);
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm);
Signature · 1 line
2.42.0.459.ge4e396fd5e-goog
maintainer
#4 lore raw

Re: [PATCH 2/3] KVM: x86/mmu: Take "shared" instead of "as_id" TDP MMU's yield-safe iterator

Quoted 5 lines · Sean Christopherson

Replace the address space ID in for_each_tdp_mmu_root_yield_safe() with a shared (vs. exclusive) param, and have the walker iterate over all address spaces as all callers want to process all address spaces. Drop the @as_id param as well as the manual address space iteration in callers.

Quoted 7 lines

Add the @shared param even though the two current callers pass "false" unconditionally, as the main reason for refactoring the walker is to simplify using it to zap invalid TDP MMU roots, which is done with mmu_lock held for read.

You konw what, I don't really like the "bool shared" arguments anymore.
For example, neither tdp_mmu_next_root nor kvm_tdp_mmu_put_root need to know if the lock is taken for read or write; protection is achieved via RCU and tdp_mmu_pages_lock. It's more self-documenting to remove the argument and assert that the lock is taken.

Likewise, the argument is more or less unnecessary in the for_each_*_tdp_mmu_root_yield_safe() macros. Many users check for the lock before calling it; and all of them either call small functions that do the check, or end up calling tdp_mmu_set_spte_atomic() and tdp_mmu_iter_set_spte(), so the per-iteration checks are also overkill.

It may be useful to a few assertions to make up for the lost check before the first execution of the body of for_each_*_tdp_mmu_root_yield_safe(), but even this is more for documentation reasons than to catch actual bugs.

I'll send a v2.

Paolo

Quoted 101 lines
---
  arch/x86/kvm/mmu/mmu.c     |  8 ++------
  arch/x86/kvm/mmu/tdp_mmu.c | 20 ++++++++++----------
  arch/x86/kvm/mmu/tdp_mmu.h |  3 +--
  3 files changed, 13 insertions(+), 18 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -6
index 59f5e40b8f55..54f94f644b42 100644
@@ -6246,7 +6246,6 @@ static bool kvm_rmap_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_e
  void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
  {
  	bool flush;
-	int i;
 
  	if (WARN_ON_ONCE(gfn_end <= gfn_start))
  		return;
@@ -6257,11 +6256,8 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 
  	flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
 
-	if (tdp_mmu_enabled) {
-		for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
-			flush = kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start,
-						      gfn_end, flush);
-	}
+	if (tdp_mmu_enabled)
+		flush = kvm_tdp_mmu_zap_leafs(kvm, gfn_start, gfn_end, flush);
 
  	if (flush)
  		kvm_flush_remote_tlbs_range(kvm, gfn_start, gfn_end - gfn_start);
arch/x86/kvm/mmu/tdp_mmu.c +10 -10
index 89aaa2463373..7cb1902ae032 100644
@@ -211,8 +211,12 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
  #define for_each_valid_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared)	\
  	__for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, _shared, true)
 
-#define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id)			\
-	__for_each_tdp_mmu_root_yield_safe(_kvm, _root, _as_id, false, false)
+#define for_each_tdp_mmu_root_yield_safe(_kvm, _root, _shared)			\
+	for (_root = tdp_mmu_next_root(_kvm, NULL, _shared, false);		\
+	     _root;								\
+	     _root = tdp_mmu_next_root(_kvm, _root, _shared, false))		\
+		if (!kvm_lockdep_assert_mmu_lock_held(_kvm, _shared)) {		\
+		} else
 
  /*
   * Iterate over all TDP MMU roots.  Requires that mmu_lock be held for write,
@@ -877,12 +881,11 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root,
   * true if a TLB flush is needed before releasing the MMU lock, i.e. if one or
   * more SPTEs were zapped since the MMU lock was last acquired.
   */
-bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end,
-			   bool flush)
+bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, gfn_t start, gfn_t end, bool flush)
  {
  	struct kvm_mmu_page *root;
 
-	for_each_tdp_mmu_root_yield_safe(kvm, root, as_id)
+	for_each_tdp_mmu_root_yield_safe(kvm, root, false)
  		flush = tdp_mmu_zap_leafs(kvm, root, start, end, true, flush);
 
  	return flush;
@@ -891,7 +894,6 @@ bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end,
  void kvm_tdp_mmu_zap_all(struct kvm *kvm)
  {
  	struct kvm_mmu_page *root;
-	int i;
 
  	/*
  	 * Zap all roots, including invalid roots, as all SPTEs must be dropped
@@ -905,10 +907,8 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm)
  	 * is being destroyed or the userspace VMM has exited.  In both cases,
  	 * KVM_RUN is unreachable, i.e. no vCPUs will ever service the request.
  	 */
-	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
-		for_each_tdp_mmu_root_yield_safe(kvm, root, i)
-			tdp_mmu_zap_root(kvm, root, false);
-	}
+	for_each_tdp_mmu_root_yield_safe(kvm, root, false)
+		tdp_mmu_zap_root(kvm, root, false);
  }
 
  /*
arch/x86/kvm/mmu/tdp_mmu.h +1 -2
index eb4fa345d3a4..bc088953f929 100644
@@ -20,8 +20,7 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root)
  void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root,
  			  bool shared);
 
-bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end,
-			   bool flush);
+bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, gfn_t start, gfn_t end, bool flush);
  bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp);
  void kvm_tdp_mmu_zap_all(struct kvm *kvm);
  void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm);
author maintainer
#5 lore raw

Re: [PATCH 2/3] KVM: x86/mmu: Take "shared" instead of "as_id" TDP MMU's yield-safe iterator

Quoted 11 lines · Paolo Bonzini
Quoted 10 lines · Sean Christopherson

Replace the address space ID in for_each_tdp_mmu_root_yield_safe() with a shared (vs. exclusive) param, and have the walker iterate over all address spaces as all callers want to process all address spaces. Drop the @as_id param as well as the manual address space iteration in callers.

Add the @shared param even though the two current callers pass "false" unconditionally, as the main reason for refactoring the walker is to simplify using it to zap invalid TDP MMU roots, which is done with mmu_lock held for read.

Quoted 4 lines
Quoted 2 lines

You konw what, I don't really like the "bool shared" arguments anymore.

Yeah, I don't like the "shared" arguments either. Never did, but they are necessary for some paths, and I don't see an obviously better solution. :-/

Quoted 10 lines

For example, neither tdp_mmu_next_root nor kvm_tdp_mmu_put_root need to know if the lock is taken for read or write; protection is achieved via RCU and tdp_mmu_pages_lock. It's more self-documenting to remove the argument and assert that the lock is taken.

Likewise, the argument is more or less unnecessary in the for_each_*_tdp_mmu_root_yield_safe() macros. Many users check for the lock before calling it; and all of them either call small functions that do the check, or end up calling tdp_mmu_set_spte_atomic() and tdp_mmu_iter_set_spte(), so the per-iteration checks are also overkill.

Agreed.

Quoted 3 lines

It may be useful to a few assertions to make up for the lost check before the first execution of the body of for_each_*_tdp_mmu_root_yield_safe(), but even this is more for documentation reasons than to catch actual bugs.

I think it's more than sufficient, arguably even better, to document which paths *require* mmu_lock be held for read vs. write, and which paths work with either.

Quoted 1 line

I'll send a v2.

Can we do a cleanup of the @shared arguments on top? I would like to keep the diff reasonably small to minimize the v6.1 backport.

author maintainer
#6 lore raw

PATCH 3/3 KVM: x86/mmu: Stop zapping invalidated TDP MMU roots asynchronously

In reply to Sean Christopherson

Stop zapping invalidate TDP MMU roots via work queue now that KVM preserves TDP MMU roots until they are explicitly invalidated. Zapping roots asynchronously was effectively a workaround to avoid stalling a vCPU for an extended during if a vCPU unloaded a root, which at the time happened whenever the guest toggled CR0.WP (a frequent operation for some guest kernels).

While a clever hack, zapping roots via an unbound worker had subtle, unintended consequences on host scheduling, especially when zapping multiple roots, e.g. as part of a memslot. Because the work of zapping a root is no longer bound to the task that initiated the zap, things like the CPU affinity and priority of the original task get lost. Losing the affinity and priority can be especially problematic if unbound workqueues aren't affined to a small number of CPUs, as zapping multiple roots can cause KVM to heavily utilize the majority of CPUs in the system, *beyond* the CPUs KVM is already using to run vCPUs.

When deleting a memslot via KVM_SET_USER_MEMORY_REGION, the async root zap can result in KVM occupying all logical CPUs for ~8ms, and result in high priority tasks not being scheduled in in a timely manner. In v5.15, which doesn't preserve unloaded roots, the issues were even more noticeable as KVM would zap roots more frequently and could occupy all CPUs for 50ms+.

Consuming all CPUs for an extended duration can lead to significant jitter throughout the system, e.g. on ChromeOS with virtio-gpu, deleting memslots is a semi-frequent operation as memslots are deleted and recreated with different host virtual addresses to react to host GPU drivers allocating and freeing GPU blobs. On ChromeOS, the jitter manifests as audio blips during games due to the audio server's tasks not getting scheduled in promptly, despite the tasks having a high realtime priority.

Deleting memslots isn't exactly a fast path and should be avoided when possible, and ChromeOS is working towards utilizing MAP_FIXED to avoid the memslot shenanigans, but KVM is squarely in the wrong. Not to mention that removing the async zapping eliminates a non-trivial amount of complexity.

Note, one of the subtle behaviors hidden behind the async zapping is that KVM would zap invalidated roots only once (ignoring partial zaps from things like mmu_notifier events). Preserve this behavior by adding a flag to identify roots that are scheduled to be zapped versus roots that have already been zapped but not yet freed.

Add a comment calling out why kvm_tdp_mmu_invalidate_all_roots() can encounter invalid roots, as it's not at all obvious why zapping invalidated roots shouldn't simply zap all invalid roots.

---
 arch/x86/include/asm/kvm_host.h |   3 +-
 arch/x86/kvm/mmu/mmu.c          |  13 +---
 arch/x86/kvm/mmu/mmu_internal.h |  13 ++--
 arch/x86/kvm/mmu/tdp_mmu.c      | 116 +++++++++++++-------------------
 arch/x86/kvm/mmu/tdp_mmu.h      |   2 +-
 arch/x86/kvm/x86.c              |   5 +-
 6 files changed, 59 insertions(+), 93 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -2
index 1a4def36d5bb..17715cb8731d 100644
@@ -1419,7 +1419,6 @@ struct kvm_arch {
 	 * the thread holds the MMU lock in write mode.
 	 */
 	spinlock_t tdp_mmu_pages_lock;
-	struct workqueue_struct *tdp_mmu_zap_wq;
 #endif /* CONFIG_X86_64 */
 
 	/*
@@ -1835,7 +1834,7 @@ void kvm_mmu_vendor_module_exit(void);
 
 void kvm_mmu_destroy(struct kvm_vcpu *vcpu);
 int kvm_mmu_create(struct kvm_vcpu *vcpu);
-int kvm_mmu_init_vm(struct kvm *kvm);
+void kvm_mmu_init_vm(struct kvm *kvm);
 void kvm_mmu_uninit_vm(struct kvm *kvm);
 
 void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu/mmu.c +3 -10
index 54f94f644b42..f7901cb4d2fa 100644
@@ -6167,20 +6167,15 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
 	return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages));
 }
 
-int kvm_mmu_init_vm(struct kvm *kvm)
+void kvm_mmu_init_vm(struct kvm *kvm)
 {
-	int r;
-
 	INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
 	INIT_LIST_HEAD(&kvm->arch.zapped_obsolete_pages);
 	INIT_LIST_HEAD(&kvm->arch.possible_nx_huge_pages);
 	spin_lock_init(&kvm->arch.mmu_unsync_pages_lock);
 
-	if (tdp_mmu_enabled) {
-		r = kvm_mmu_init_tdp_mmu(kvm);
-		if (r < 0)
-			return r;
-	}
+	if (tdp_mmu_enabled)
+		kvm_mmu_init_tdp_mmu(kvm);
 
 	kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
 	kvm->arch.split_page_header_cache.gfp_zero = __GFP_ZERO;
@@ -6189,8 +6184,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 
 	kvm->arch.split_desc_cache.kmem_cache = pte_list_desc_cache;
 	kvm->arch.split_desc_cache.gfp_zero = __GFP_ZERO;
-
-	return 0;
 }
 
 static void mmu_free_vm_memory_caches(struct kvm *kvm)
arch/x86/kvm/mmu/mmu_internal.h +5 -8
index b102014e2c60..93b9d50c24ad 100644
@@ -58,7 +58,10 @@ struct kvm_mmu_page {
 
 	bool tdp_mmu_page;
 	bool unsync;
-	u8 mmu_valid_gen;
+	union {
+		u8 mmu_valid_gen;
+		bool tdp_mmu_scheduled_root_to_zap;
+	};
 
 	 /*
 	  * The shadow page can't be replaced by an equivalent huge page
@@ -100,13 +103,7 @@ struct kvm_mmu_page {
 		struct kvm_rmap_head parent_ptes; /* rmap pointers to parent sptes */
 		tdp_ptep_t ptep;
 	};
-	union {
-		DECLARE_BITMAP(unsync_child_bitmap, 512);
-		struct {
-			struct work_struct tdp_mmu_async_work;
-			void *tdp_mmu_async_data;
-		};
-	};
+	DECLARE_BITMAP(unsync_child_bitmap, 512);
 
 	/*
 	 * Tracks shadow pages that, if zapped, would allow KVM to create an NX
arch/x86/kvm/mmu/tdp_mmu.c +48 -68

166 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +1 -1
index bc088953f929..733a3aef3a96 100644
@@ -7,7 +7,7 @@
 
 #include "spte.h"
 
-int kvm_mmu_init_tdp_mmu(struct kvm *kvm);
+void kvm_mmu_init_tdp_mmu(struct kvm *kvm);
 void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm);
 
 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +1 -4
index 6c9c81e82e65..9f18b06bbda6 100644
@@ -12308,9 +12308,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	if (ret)
 		goto out;
 
-	ret = kvm_mmu_init_vm(kvm);
-	if (ret)
-		goto out_page_track;
+	kvm_mmu_init_vm(kvm);
 
 	ret = static_call(kvm_x86_vm_init)(kvm);
 	if (ret)
@@ -12355,7 +12353,6 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
 out_uninit_mmu:
 	kvm_mmu_uninit_vm(kvm);
-out_page_track:
 	kvm_page_track_cleanup(kvm);
 out:
 	return ret;
Signature · 1 line
2.42.0.459.ge4e396fd5e-goog
maintainer
#7 lore raw

Re: [PATCH 3/3] KVM: x86/mmu: Stop zapping invalidated TDP MMU roots asynchronously

Review

Quoted 4 lines · Sean Christopherson
(quoted diff) +4 -0
+		if (!root->tdp_mmu_scheduled_root_to_zap)
+			continue;
+
+		root->tdp_mmu_scheduled_root_to_zap = false;

This is protected by slots_lock... tricky.

Worth squashing in a comment and also a small update to another comment:

arch/x86/kvm/mmu/mmu_internal.h +2 -0
index 93b9d50c24ad..decc1f153669 100644
@@ -60,6 +60,8 @@ struct kvm_mmu_page {
 	bool unsync;
 	union {
 		u8 mmu_valid_gen;
+
+		/* Only accessed under slots_lock.  */
 		bool tdp_mmu_scheduled_root_to_zap;
 	};
 
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index ca3304c2c00c..070ee5b2c271 100644
@@ -246,7 +246,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)
 	 * by a memslot update or by the destruction of the VM.  Initialize the
 	 * refcount to two; one reference for the vCPU, and one reference for
 	 * the TDP MMU itself, which is held until the root is invalidated and
-	 * is ultimately put by tdp_mmu_zap_root_work().
+	 * is ultimately put by kvm_tdp_mmu_zap_invalidated_roots().
 	 */
 	refcount_set(&root->tdp_mmu_root_count, 2);
 

Paolo

Quoted 1 line
(quoted diff) +1 -0
+		KVM_BUG_ON(!root->role.invalid, kvm);
maintainer Applied
#8 lore raw
In reply to Sean Christopherson

Queued, thanks.

I changed a bit the splitting of the patches, to avoid mixing removal of one argument and with the addition of another (splitting into four patches wasn't particularly enlightening for a couple lines change), but checked that the final result is the same.

Paolo