Skip to contentSkip to conversation

KVM: x86/mmu: Optimize TDP MMU huge page recovery during disable-dirty-log

PATCH Applied David Matlack 3 participants 16 messages Aug 23, 2024 – Nov 5, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover KVM: x86/mmu: Optimize TDP MMU huge page recovery during disable-dirty-log

Rework the TDP MMU disable-dirty-log path to batch TLB flushes and recover huge page mappings, rather than zapping and flushing for every potential huge page mapping.

With this series, dirty_log_perf_test shows a decrease in the time it takes to disable dirty logging, as well as a decrease in the number of vCPU faults:

 $ ./dirty_log_perf_test -s anonymous_hugetlb_2mb -v 64 -e -b 4g

Before: Disabling dirty logging time: 14.334453428s (131072 flushes)
After: Disabling dirty logging time: 4.794969689s (76 flushes)

Before: 393,599 kvm:kvm_page_fault
After: 262,575 kvm:kvm_page_fault

v2:
- Use a separate iterator to walk down to child SPTEs during huge page recovery [Sean]
- Return SHADOW_NONPRESENT_VALUE in error conditions in make_huge_spte() [Vipin][off-list]

v1: https://lore.kernel.org/kvm/[email protected]/

David Matlack (6):
KVM: x86/mmu: Drop @max_level from kvm_mmu_max_mapping_level()
KVM: x86/mmu: Batch TLB flushes when zapping collapsible TDP MMU SPTEs
KVM: x86/mmu: Refactor TDP MMU iter need resched check
KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of
zapping
KVM: x86/mmu: Rename make_huge_page_split_spte() to make_small_spte()
KVM: x86/mmu: WARN if huge page recovery triggered during dirty
logging

 arch/x86/include/asm/kvm_host.h |   4 +-
 arch/x86/kvm/mmu/mmu.c          |  16 ++--
 arch/x86/kvm/mmu/mmu_internal.h |   3 +-
 arch/x86/kvm/mmu/spte.c         |  43 +++++++++--
 arch/x86/kvm/mmu/spte.h         |   5 +-
 arch/x86/kvm/mmu/tdp_mmu.c      | 129 +++++++++++++++++---------------
 arch/x86/kvm/mmu/tdp_mmu.h      |   4 +-
 arch/x86/kvm/x86.c              |  18 ++---
 8 files changed, 128 insertions(+), 94 deletions(-)

base-commit: 728d17c2cb8cc5f9ac899173d0e9a67fb8887622

Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
#1 lore raw

PATCH v2 1/6 KVM: x86/mmu: Drop @max_level from kvm_mmu_max_mapping_level()

Drop the @max_level parameter from kvm_mmu_max_mapping_level(). All callers pass in PG_LEVEL_NUM, so @max_level can be replaced with PG_LEVEL_NUM in the function body.

No functional change intended.

---
 arch/x86/kvm/mmu/mmu.c          | 8 +++-----
 arch/x86/kvm/mmu/mmu_internal.h | 3 +--
 arch/x86/kvm/mmu/tdp_mmu.c      | 3 +--
 3 files changed, 5 insertions(+), 9 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -5
index d25c2b395116..2e92d9e9b311 100644
@@ -3109,13 +3109,12 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 }
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
-			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level)
+			      const struct kvm_memory_slot *slot, gfn_t gfn)
 {
 	bool is_private = kvm_slot_can_be_private(slot) &&
 			  kvm_mem_is_private(kvm, gfn);
 
-	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, max_level, is_private);
+	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_private);
 }
 
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
@@ -6877,8 +6876,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 * mapping if the indirect sp has level = 1.
 		 */
 		if (sp->role.direct &&
-		    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn,
-							       PG_LEVEL_NUM)) {
+		    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn)) {
 			kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
 			if (kvm_available_flush_remote_tlbs_range())
arch/x86/kvm/mmu/mmu_internal.h +1 -2
index 1721d97743e9..fee385e75405 100644
@@ -344,8 +344,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 }
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
-			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level);
+			      const struct kvm_memory_slot *slot, gfn_t gfn);
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_level);
 
arch/x86/kvm/mmu/tdp_mmu.c +1 -2
index 3c55955bcaf8..2a843b9c8d81 100644
@@ -1655,8 +1655,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 		if (iter.gfn < start || iter.gfn >= end)
 			continue;
 
-		max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot,
-							      iter.gfn, PG_LEVEL_NUM);
+		max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot, iter.gfn);
 		if (max_mapping_level < iter.level)
 			continue;
 
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
#2 lore raw

PATCH v2 2/6 KVM: x86/mmu: Batch TLB flushes when zapping collapsible TDP MMU SPTEs

In reply to David Matlack

Set SPTEs directly to SHADOW_NONPRESENT_VALUE and batch up TLB flushes when zapping collapsible SPTEs, rather than freezing them first.

Freezing the SPTE first is not required. It is fine for another thread holding mmu_lock for read to immediately install a present entry before TLBs are flushed because the underlying mapping is not changing. vCPUs that translate through the stale 4K mappings or a new huge page mapping will still observe the same GPA->HPA translations.

KVM must only flush TLBs before dropping RCU (to avoid use-after-free of the zapped page tables) and before dropping mmu_lock (to synchronize with mmu_notifiers invalidating mappings).

In VMs backed with 2MiB pages, batching TLB flushes improves the time it takes to zap collapsible SPTEs to disable dirty logging:

 $ ./dirty_log_perf_test -s anonymous_hugetlb_2mb -v 64 -e -b 4g

Before: Disabling dirty logging time: 14.334453428s (131072 flushes)
After: Disabling dirty logging time: 4.794969689s (76 flushes)

Skipping freezing SPTEs also avoids stalling vCPU threads on the frozen SPTE for the time it takes to perform a remote TLB flush. vCPUs faulting on the zapped mapping can now immediately install a new huge mapping and proceed with guest execution.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 55 +++++++-------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +10 -45
index 2a843b9c8d81..27adbb3ecb02 100644
@@ -591,48 +591,6 @@ static inline int __must_check tdp_mmu_set_spte_atomic(struct kvm *kvm,
 	return 0;
 }
 
-static inline int __must_check tdp_mmu_zap_spte_atomic(struct kvm *kvm,
-						       struct tdp_iter *iter)
-{
-	int ret;
-
-	lockdep_assert_held_read(&kvm->mmu_lock);
-
-	/*
-	 * Freeze the SPTE by setting it to a special, non-present value. This
-	 * will stop other threads from immediately installing a present entry
-	 * in its place before the TLBs are flushed.
-	 *
-	 * Delay processing of the zapped SPTE until after TLBs are flushed and
-	 * the FROZEN_SPTE is replaced (see below).
-	 */
-	ret = __tdp_mmu_set_spte_atomic(iter, FROZEN_SPTE);
-	if (ret)
-		return ret;
-
-	kvm_flush_remote_tlbs_gfn(kvm, iter->gfn, iter->level);
-
-	/*
-	 * No other thread can overwrite the frozen SPTE as they must either
-	 * wait on the MMU lock or use tdp_mmu_set_spte_atomic() which will not
-	 * overwrite the special frozen SPTE value. Use the raw write helper to
-	 * avoid an unnecessary check on volatile bits.
-	 */
-	__kvm_tdp_mmu_write_spte(iter->sptep, SHADOW_NONPRESENT_VALUE);
-
-	/*
-	 * Process the zapped SPTE after flushing TLBs, and after replacing
-	 * FROZEN_SPTE with 0. This minimizes the amount of time vCPUs are
-	 * blocked by the FROZEN_SPTE and reduces contention on the child
-	 * SPTEs.
-	 */
-	handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte,
-			    SHADOW_NONPRESENT_VALUE, iter->level, true);
-
-	return 0;
-}
-
-
 /*
  * tdp_mmu_set_spte - Set a TDP MMU SPTE and handle the associated bookkeeping
  * @kvm:	      KVM instance
@@ -1625,13 +1583,16 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 	gfn_t end = start + slot->npages;
 	struct tdp_iter iter;
 	int max_mapping_level;
+	bool flush = false;
 
 	rcu_read_lock();
 
 	for_each_tdp_pte_min_level(iter, root, PG_LEVEL_2M, start, end) {
 retry:
-		if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true))
+		if (tdp_mmu_iter_cond_resched(kvm, &iter, flush, true)) {
+			flush = false;
 			continue;
+		}
 
 		if (iter.level > KVM_MAX_HUGEPAGE_LEVEL ||
 		    !is_shadow_present_pte(iter.old_spte))
@@ -1659,11 +1620,15 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 		if (max_mapping_level < iter.level)
 			continue;
 
-		/* Note, a successful atomic zap also does a remote TLB flush. */
-		if (tdp_mmu_zap_spte_atomic(kvm, &iter))
+		if (tdp_mmu_set_spte_atomic(kvm, &iter, SHADOW_NONPRESENT_VALUE))
 			goto retry;
+
+		flush = true;
 	}
 
+	if (flush)
+		kvm_flush_remote_tlbs_memslot(kvm, slot);
+
 	rcu_read_unlock();
 }
 
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
#3 lore raw

PATCH v2 3/6 KVM: x86/mmu: Refactor TDP MMU iter need resched check

In reply to David Matlack

Refactor the TDP MMU iterator "need resched" checks into a helper function so they can be called from a different code path in a subsequent commit.

No functional change intended.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +11 -5
index 27adbb3ecb02..9b8299ee4abb 100644
@@ -646,6 +646,16 @@ static inline void tdp_mmu_iter_set_spte(struct kvm *kvm, struct tdp_iter *iter,
 #define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end)		\
 	for_each_tdp_pte(_iter, root_to_sp(_mmu->root.hpa), _start, _end)
 
+static inline bool __must_check tdp_mmu_iter_need_resched(struct kvm *kvm,
+							  struct tdp_iter *iter)
+{
+	/* Ensure forward progress has been made before yielding. */
+	if (iter->next_last_level_gfn == iter->yielded_gfn)
+		return false;
+
+	return need_resched() || rwlock_needbreak(&kvm->mmu_lock);
+}
+
 /*
  * Yield if the MMU lock is contended or this thread needs to return control
  * to the scheduler.
@@ -666,11 +676,7 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm,
 {
 	WARN_ON_ONCE(iter->yielded);
 
-	/* Ensure forward progress has been made before yielding. */
-	if (iter->next_last_level_gfn == iter->yielded_gfn)
-		return false;
-
-	if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {
+	if (tdp_mmu_iter_need_resched(kvm, iter)) {
 		if (flush)
 			kvm_flush_remote_tlbs(kvm);
 
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
#4 lore raw

Re: [PATCH v2 3/6] KVM: x86/mmu: Refactor TDP MMU iter need resched check

Quoted 33 lines · David Matlack

Refactor the TDP MMU iterator "need resched" checks into a helper function so they can be called from a different code path in a subsequent commit.

No functional change intended.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +10 -0
index 27adbb3ecb02..9b8299ee4abb 100644
@@ -646,6 +646,16 @@ static inline void tdp_mmu_iter_set_spte(struct kvm *kvm, struct tdp_iter *iter,
 #define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end)		\
 	for_each_tdp_pte(_iter, root_to_sp(_mmu->root.hpa), _start, _end)
 
+static inline bool __must_check tdp_mmu_iter_need_resched(struct kvm *kvm,
+							  struct tdp_iter *iter)
+{
+	/* Ensure forward progress has been made before yielding. */
+	if (iter->next_last_level_gfn == iter->yielded_gfn)
+		return false;
+
+	return need_resched() || rwlock_needbreak(&kvm->mmu_lock);
+}
+
 /*
  * Yield if the MMU lock is contended or this thread needs to return control
  * to the scheduler.

arch/x86/kvm/mmu/tdp_mmu.c static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm, · line 679

Quoted 9 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -5
@@ -666,11 +676,7 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm,
 {
 	WARN_ON_ONCE(iter->yielded);
 
-	/* Ensure forward progress has been made before yielding. */
-	if (iter->next_last_level_gfn == iter->yielded_gfn)
-		return false;
-
-	if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {
+	if (tdp_mmu_iter_need_resched(kvm, iter)) {

Huh. There's a subtle behavioral change here, that I would not have noticed had I not _just_ looked at this code[*]. Falling through means the "don't yield to ensure forward progress case" would return iter->yielded, not %false. Per the WARN above, iter->yielded _should_ be false, but if KVM had a bug that caused it to get stuck, then that bug would escalate into an even worse bug of putting KVM into a potentially unbreakable infinite loop.

Which is extremely unlikely, but it's a good excuse to clean this up :-) I'll test and post the below, and plan on slotting it in before this patch (you might even see it show up in kvm-x86 before it gets posted).

[*] https://lore.kernel.org/all/[email protected]

From: Sean Christopherson <[email protected]>
Date: Wed, 30 Oct 2024 16:28:31 -0700
Subject: [PATCH] KVM: x86/mmu: Check yielded_gfn for forward progress iff
resched is needed

Swap the order of the checks in tdp_mmu_iter_cond_resched() so that KVM checks to see if a resched is needed _before_ checking to see if yielding must be disallowed to guarantee forward progress. Iterating over TDP MMU SPTEs is a hot path, e.g. tearing down a root can touch millions of SPTEs, and not needing to reschedule is by far the common case. On the other handle, disallowing yielding because forward progress has not been made is a very rare case.

Returning early for the common case (no resched), effectively reduces the number of checks from 2 to 1 for the common case, and should make the code slightly more predictable for the CPU.

To resolve a weird conundrum where the forward progress check currently returns false, but the need resched check subtly returns iter->yielded, which _should_ be false (enforced by a WARN), return false unconditionally (which might also help make the sequence more predicatble). If KVM has a bug where iter->yielded is left danging, continuing to yield is neither right nor wrong, it was simply an artifact of how the original code was written.

Unconditionally returning false when yielding is unnecessary or unwanted will also allow extracting the "should resched" logic to a separate helper in a future patch.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +14 -14
index 076343c3c8a7..8170b16b91c3 100644
@@ -658,29 +658,29 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm,
 {
 	WARN_ON_ONCE(iter->yielded);
 
+	if (!need_resched() && !rwlock_needbreak(&kvm->mmu_lock))
+		return false;
+
 	/* Ensure forward progress has been made before yielding. */
 	if (iter->next_last_level_gfn == iter->yielded_gfn)
 		return false;
 
-	if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {
-		if (flush)
-			kvm_flush_remote_tlbs(kvm);
+	if (flush)
+		kvm_flush_remote_tlbs(kvm);
 
-		rcu_read_unlock();
+	rcu_read_unlock();
 
-		if (shared)
-			cond_resched_rwlock_read(&kvm->mmu_lock);
-		else
-			cond_resched_rwlock_write(&kvm->mmu_lock);
+	if (shared)
+		cond_resched_rwlock_read(&kvm->mmu_lock);
+	else
+		cond_resched_rwlock_write(&kvm->mmu_lock);
 
-		rcu_read_lock();
+	rcu_read_lock();
 
-		WARN_ON_ONCE(iter->gfn > iter->next_last_level_gfn);
+	WARN_ON_ONCE(iter->gfn > iter->next_last_level_gfn);
 
-		iter->yielded = true;
-	}
-
-	return iter->yielded;
+	iter->yielded = true;
+	return true;
 }
 
 static inline gfn_t tdp_mmu_max_gfn_exclusive(void)

base-commit: 35ef80eb29ab5f7b7c7264c7f21a64b3aa046921

#5 lore raw

PATCH v2 4/6 KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping

In reply to David Matlack

Recover TDP MMU huge page mappings in-place instead of zapping them when dirty logging is disabled, and rename functions that recover huge page mappings when dirty logging is disabled to move away from the "zap collapsible spte" terminology.

Before KVM flushes TLBs, guest accesses may be translated through either the (stale) small SPTE or the (new) huge SPTE. This is already possible when KVM is doing eager page splitting (where TLB flushes are also batched), and when vCPUs are faulting in huge mappings (where TLBs are flushed after the new huge SPTE is installed).

Recovering huge pages reduces the number of page faults when dirty logging is disabled:

 $ perf stat -e kvm:kvm_page_fault -- ./dirty_log_perf_test -s anonymous_hugetlb_2mb -v 64 -e -b 4g

Before: 393,599 kvm:kvm_page_fault
After: 262,575 kvm:kvm_page_fault

vCPU throughput and the latency of disabling dirty-logging are about equal compared to zapping, but avoiding faults can be beneficial to remove vCPU jitter in extreme scenarios.

---
 arch/x86/include/asm/kvm_host.h |  4 +--
 arch/x86/kvm/mmu/mmu.c          |  6 ++--
 arch/x86/kvm/mmu/spte.c         | 39 +++++++++++++++++++++++--
 arch/x86/kvm/mmu/spte.h         |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      | 52 +++++++++++++++++++++++++++------
 arch/x86/kvm/mmu/tdp_mmu.h      |  4 +--
 arch/x86/kvm/x86.c              | 18 +++++-------
 7 files changed, 94 insertions(+), 30 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -2
index 1811a42fa093..8f9bd7c0e139 100644
@@ -1953,8 +1953,8 @@ void kvm_mmu_try_split_huge_pages(struct kvm *kvm,
 				  const struct kvm_memory_slot *memslot,
 				  u64 start, u64 end,
 				  int target_level);
-void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
-				   const struct kvm_memory_slot *memslot);
+void kvm_mmu_recover_huge_pages(struct kvm *kvm,
+				const struct kvm_memory_slot *memslot);
 void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
 				   const struct kvm_memory_slot *memslot);
 void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen);
arch/x86/kvm/mmu/mmu.c +3 -3
index 2e92d9e9b311..2f8b1ebcbe9c 100644
@@ -6904,8 +6904,8 @@ static void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm,
 		kvm_flush_remote_tlbs_memslot(kvm, slot);
 }
 
-void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
-				   const struct kvm_memory_slot *slot)
+void kvm_mmu_recover_huge_pages(struct kvm *kvm,
+				const struct kvm_memory_slot *slot)
 {
 	if (kvm_memslots_have_rmaps(kvm)) {
 		write_lock(&kvm->mmu_lock);
@@ -6915,7 +6915,7 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
 
 	if (tdp_mmu_enabled) {
 		read_lock(&kvm->mmu_lock);
-		kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot);
+		kvm_tdp_mmu_recover_huge_pages(kvm, slot);
 		read_unlock(&kvm->mmu_lock);
 	}
 }
arch/x86/kvm/mmu/spte.c +36 -3
index 8f7eb3ad88fc..a12437bf6e0c 100644
@@ -268,15 +268,14 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	return wrprot;
 }
 
-static u64 make_spte_executable(u64 spte)
+static u64 modify_spte_protections(u64 spte, u64 set, u64 clear)
 {
 	bool is_access_track = is_access_track_spte(spte);
 
 	if (is_access_track)
 		spte = restore_acc_track_spte(spte);
 
-	spte &= ~shadow_nx_mask;
-	spte |= shadow_x_mask;
+	spte = (spte | set) & ~clear;
 
 	if (is_access_track)
 		spte = mark_spte_for_access_track(spte);
@@ -284,6 +283,16 @@ static u64 make_spte_executable(u64 spte)
 	return spte;
 }
 
+static u64 make_spte_executable(u64 spte)
+{
+	return modify_spte_protections(spte, shadow_x_mask, shadow_nx_mask);
+}
+
+static u64 make_spte_nonexecutable(u64 spte)
+{
+	return modify_spte_protections(spte, shadow_nx_mask, shadow_x_mask);
+}
+
 /*
  * Construct an SPTE that maps a sub-page of the given huge page SPTE where
  * `index` identifies which sub-page.
@@ -320,6 +329,30 @@ u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte,
 	return child_spte;
 }
 
+u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level)
+{
+	u64 huge_spte;
+
+	if (KVM_BUG_ON(!is_shadow_present_pte(small_spte), kvm))
+		return SHADOW_NONPRESENT_VALUE;
+
+	if (KVM_BUG_ON(level == PG_LEVEL_4K, kvm))
+		return SHADOW_NONPRESENT_VALUE;
+
+	huge_spte = small_spte | PT_PAGE_SIZE_MASK;
+
+	/*
+	 * huge_spte already has the address of the sub-page being collapsed
+	 * from small_spte, so just clear the lower address bits to create the
+	 * huge page address.
+	 */
+	huge_spte &= KVM_HPAGE_MASK(level) | ~PAGE_MASK;
+
+	if (is_nx_huge_page_enabled(kvm))
+		huge_spte = make_spte_nonexecutable(huge_spte);
+
+	return huge_spte;
+}
 
 u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled)
 {
arch/x86/kvm/mmu/spte.h +1 -0
index 2cb816ea2430..990d599eb827 100644
@@ -503,6 +503,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	       bool host_writable, u64 *new_spte);
 u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte,
 		      	      union kvm_mmu_page_role role, int index);
+u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level);
 u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled);
 u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access);
 u64 mark_spte_for_access_track(u64 spte);
arch/x86/kvm/mmu/tdp_mmu.c +43 -9

82 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +2 -2
index 1b74e058a81c..ddea2827d1ad 100644
@@ -40,8 +40,8 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm,
 				       struct kvm_memory_slot *slot,
 				       gfn_t gfn, unsigned long mask,
 				       bool wrprot);
-void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,
-				       const struct kvm_memory_slot *slot);
+void kvm_tdp_mmu_recover_huge_pages(struct kvm *kvm,
+				    const struct kvm_memory_slot *slot);
 
 bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm,
 				   struct kvm_memory_slot *slot, gfn_t gfn,
arch/x86/kvm/x86.c +7 -11
index 966fb301d44b..3d09c12847d5 100644
@@ -13053,19 +13053,15 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,
 
 	if (!log_dirty_pages) {
 		/*
-		 * Dirty logging tracks sptes in 4k granularity, meaning that
-		 * large sptes have to be split.  If live migration succeeds,
-		 * the guest in the source machine will be destroyed and large
-		 * sptes will be created in the destination.  However, if the
-		 * guest continues to run in the source machine (for example if
-		 * live migration fails), small sptes will remain around and
-		 * cause bad performance.
+		 * Recover huge page mappings in the slot now that dirty logging
+		 * is disabled, i.e. now that KVM does not have to track guest
+		 * writes at 4KiB granularity.
 		 *
-		 * Scan sptes if dirty logging has been stopped, dropping those
-		 * which can be collapsed into a single large-page spte.  Later
-		 * page faults will create the large-page sptes.
+		 * Dirty logging might be disabled by userspace if an ongoing VM
+		 * live migration is cancelled and the VM must continue running
+		 * on the source.
 		 */
-		kvm_mmu_zap_collapsible_sptes(kvm, new);
+		kvm_mmu_recover_huge_pages(kvm, new);
 	} else {
 		/*
 		 * Initially-all-set does not require write protecting any page,
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
#6 lore raw

Re: [PATCH v2 4/6] KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping

arch/x86/kvm/mmu/spte.c bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, · line 278

Quoted 10 lines · David Matlack
arch/x86/kvm/mmu/spte.c +2 -2
+static u64 modify_spte_protections(u64 spte, u64 set, u64 clear)
 {
        bool is_access_track = is_access_track_spte(spte);
 
        if (is_access_track)
                spte = restore_acc_track_spte(spte);
 
-       spte &= ~shadow_nx_mask;
-       spte |= shadow_x_mask;
+       spte = (spte | set) & ~clear;

We should add a check here WARN_ON_ONCE(set & clear) because if both have a common bit set to 1 then the result will be different between:
1. spte = (spt | set) & ~clear
2. spte = (spt | ~clear) & set

In the current form, 'clear' has more authority in the final value of spte.

arch/x86/kvm/mmu/spte.c u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte, · line 341

Quoted 11 lines
arch/x86/kvm/mmu/spte.c +10 -0
+u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level)
+{
+       u64 huge_spte;
+
+       if (KVM_BUG_ON(!is_shadow_present_pte(small_spte), kvm))
+               return SHADOW_NONPRESENT_VALUE;
+
+       if (KVM_BUG_ON(level == PG_LEVEL_4K, kvm))
+               return SHADOW_NONPRESENT_VALUE;
+

KVM_BUG_ON() is very aggressive. We should replace it with WARN_ON_ONCE()

#7 lore raw

Re: [PATCH v2 4/6] KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping

Quoted 12 lines · Vipin Sharma

arch/x86/kvm/mmu/spte.c bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, · line 278

Quoted 10 lines · David Matlack
arch/x86/kvm/mmu/spte.c +2 -2
+static u64 modify_spte_protections(u64 spte, u64 set, u64 clear)
 {
        bool is_access_track = is_access_track_spte(spte);
 
        if (is_access_track)
                spte = restore_acc_track_spte(spte);
 
-       spte &= ~shadow_nx_mask;
-       spte |= shadow_x_mask;
+       spte = (spte | set) & ~clear;
Quoted 6 lines

We should add a check here WARN_ON_ONCE(set & clear) because if both have a common bit set to 1 then the result will be different between:
1. spte = (spt | set) & ~clear
2. spte = (spt | ~clear) & set

In the current form, 'clear' has more authority in the final value of spte.

KVM_MMU_WARN_ON(), overlapping @set and @clear is definitely something that should be caught during development, i.e. we don't need to carry the WARN_ON_ONCE() in production kernels

Quoted 4 lines
Quoted 4 lines
arch/x86/kvm/mmu/spte.c +4 -0
+u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level)
+{
+       u64 huge_spte;
+
Quoted 8 lines

arch/x86/kvm/mmu/spte.c u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte, · line 341

Quoted 6 lines
arch/x86/kvm/mmu/spte.c +6 -0
+       if (KVM_BUG_ON(!is_shadow_present_pte(small_spte), kvm))
+               return SHADOW_NONPRESENT_VALUE;
+
+       if (KVM_BUG_ON(level == PG_LEVEL_4K, kvm))
+               return SHADOW_NONPRESENT_VALUE;
+

KVM_BUG_ON() is very aggressive. We should replace it with WARN_ON_ONCE()

I'm tempted to say KVM_MMU_WARN_ON() here too.

#8 lore raw

Re: [PATCH v2 4/6] KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping

Quoted 38 lines · Sean Christopherson
Quoted 12 lines · Vipin Sharma

arch/x86/kvm/mmu/spte.c bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, · line 278

Quoted 10 lines · David Matlack
arch/x86/kvm/mmu/spte.c +2 -2
+static u64 modify_spte_protections(u64 spte, u64 set, u64 clear)
 {
        bool is_access_track = is_access_track_spte(spte);
 
        if (is_access_track)
                spte = restore_acc_track_spte(spte);
 
-       spte &= ~shadow_nx_mask;
-       spte |= shadow_x_mask;
+       spte = (spte | set) & ~clear;
Quoted 6 lines

We should add a check here WARN_ON_ONCE(set & clear) because if both have a common bit set to 1 then the result will be different between:
1. spte = (spt | set) & ~clear
2. spte = (spt | ~clear) & set

In the current form, 'clear' has more authority in the final value of spte.

KVM_MMU_WARN_ON(), overlapping @set and @clear is definitely something that should be caught during development, i.e. we don't need to carry the WARN_ON_ONCE() in production kernels

Quoted 4 lines
Quoted 4 lines
arch/x86/kvm/mmu/spte.c +4 -0
+u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level)
+{
+       u64 huge_spte;
+
Quoted 8 lines

arch/x86/kvm/mmu/spte.c u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte, · line 341

Quoted 6 lines
arch/x86/kvm/mmu/spte.c +6 -0
+       if (KVM_BUG_ON(!is_shadow_present_pte(small_spte), kvm))
+               return SHADOW_NONPRESENT_VALUE;
+
+       if (KVM_BUG_ON(level == PG_LEVEL_4K, kvm))
+               return SHADOW_NONPRESENT_VALUE;
+

KVM_BUG_ON() is very aggressive. We should replace it with WARN_ON_ONCE()

Quoted 1 line

I'm tempted to say KVM_MMU_WARN_ON() here too.

I am fine with KVM_MMU_WARN_ON() here. Callers should check for the value they provided and returned from this API and if it's important to them in Production then decide on next steps accordingly.

#9 lore raw

Re: [PATCH v2 4/6] KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping

Quoted 39 lines · Vipin Sharma
Quoted 38 lines · Sean Christopherson
Quoted 12 lines · Vipin Sharma

arch/x86/kvm/mmu/spte.c bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, · line 278

Quoted 10 lines · David Matlack
arch/x86/kvm/mmu/spte.c +2 -2
+static u64 modify_spte_protections(u64 spte, u64 set, u64 clear)
 {
        bool is_access_track = is_access_track_spte(spte);
 
        if (is_access_track)
                spte = restore_acc_track_spte(spte);
 
-       spte &= ~shadow_nx_mask;
-       spte |= shadow_x_mask;
+       spte = (spte | set) & ~clear;
Quoted 6 lines

We should add a check here WARN_ON_ONCE(set & clear) because if both have a common bit set to 1 then the result will be different between:
1. spte = (spt | set) & ~clear
2. spte = (spt | ~clear) & set

In the current form, 'clear' has more authority in the final value of spte.

KVM_MMU_WARN_ON(), overlapping @set and @clear is definitely something that should be caught during development, i.e. we don't need to carry the WARN_ON_ONCE() in production kernels

Quoted 12 lines

arch/x86/kvm/mmu/spte.c u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte, · line 341

Quoted 10 lines
arch/x86/kvm/mmu/spte.c +10 -0
+u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level)
+{
+       u64 huge_spte;
+
+       if (KVM_BUG_ON(!is_shadow_present_pte(small_spte), kvm))
+               return SHADOW_NONPRESENT_VALUE;
+
+       if (KVM_BUG_ON(level == PG_LEVEL_4K, kvm))
+               return SHADOW_NONPRESENT_VALUE;
+

KVM_BUG_ON() is very aggressive. We should replace it with WARN_ON_ONCE()

Quoted 5 lines
Quoted 1 line

I'm tempted to say KVM_MMU_WARN_ON() here too.

I am fine with KVM_MMU_WARN_ON() here. Callers should check for the value they provided and returned from this API and if it's important to them in Production then decide on next steps accordingly.

Coming back to this, I opted to match the behavior of make_small_spte() and do:

	KVM_BUG_ON(!is_shadow_present_pte(small_spte) || level == PG_LEVEL_4K, kvm);

As explained in commit 3d4415ed75a57, the scenario is meant to be impossible. If the check fails in production, odds are good there's SPTE memory corruption and we _want_ to kill the VM.

KVM: x86/mmu: Bug the VM if KVM tries to split a !hugepage SPTE

Bug the VM instead of simply warning if KVM tries to split a SPTE that is non-present or not-huge. KVM is guaranteed to end up in a broken state as the callers fully expect a valid SPTE, e.g. the shadow MMU will add an rmap entry, and all MMUs will account the expected small page. Returning '0' is also technically wrong now that SHADOW_NONPRESENT_VALUE exists, i.e. would cause KVM to create a potential #VE SPTE.

While it would be possible to have the callers gracefully handle failure, doing so would provide no practical value as the scenario really should be impossible, while the error handling would add a non-trivial amount of noise.

There's also no need to return SHADOW_NONPRESENT_VALUE. KVM_BUG_ON() ensures all vCPUs are kicked out of the guest, so while the return SPTE may be a bit nonsensical, it will never be consumed by hardware. Theoretically, KVM could wander down a weird path in the future, but again, the most likely scenario is that there was host memory corruption, so potential weird paths are the least of KVM's worries at that point.

More importantly, in the _current_ code, returning SHADOW_NONPRESENT_VALUE happens to be benign, but that's 100% due to make_huge_spte() only being used by the TDP MMU. If the shaduw MMU ever started using make_huge_spte(), returning a !present SPTE would be all but guaranteed to cause fatal problems.

#10 lore raw

Re: [PATCH v2 4/6] KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping

Quoted 7 lines · Sean Christopherson

On Wed, Oct 09, 2024, Vipin Sharma wrote:

Coming back to this, I opted to match the behavior of make_small_spte() and do:

        KVM_BUG_ON(!is_shadow_present_pte(small_spte) || level == PG_LEVEL_4K, kvm);

Should these be two separate KVM_BUG_ON(), to aid in debugging?

Quoted 25 lines

As explained in commit 3d4415ed75a57, the scenario is meant to be impossible. If the check fails in production, odds are good there's SPTE memory corruption and we _want_ to kill the VM.

KVM: x86/mmu: Bug the VM if KVM tries to split a !hugepage SPTE

Bug the VM instead of simply warning if KVM tries to split a SPTE that is non-present or not-huge. KVM is guaranteed to end up in a broken state as the callers fully expect a valid SPTE, e.g. the shadow MMU will add an rmap entry, and all MMUs will account the expected small page. Returning '0' is also technically wrong now that SHADOW_NONPRESENT_VALUE exists, i.e. would cause KVM to create a potential #VE SPTE.

While it would be possible to have the callers gracefully handle failure, doing so would provide no practical value as the scenario really should be impossible, while the error handling would add a non-trivial amount of noise.

There's also no need to return SHADOW_NONPRESENT_VALUE. KVM_BUG_ON() ensures all vCPUs are kicked out of the guest, so while the return SPTE may be a bit nonsensical, it will never be consumed by hardware. Theoretically, KVM could wander down a weird path in the future, but again, the most likely scenario is that there was host memory corruption, so potential weird paths are the least of KVM's worries at that point.

Quoted 4 lines

More importantly, in the _current_ code, returning SHADOW_NONPRESENT_VALUE happens to be benign, but that's 100% due to make_huge_spte() only being used by the TDP MMU. If the shaduw MMU ever started using make_huge_spte(), returning a !present SPTE would be all but guaranteed to cause fatal problems.

I think the caller should be given the opportunity to handle a failure. In the current code, TDP is able to handle the error condition, so penalizing a VM seems wrong. We have gone from a state of reduced performance to either very good performance or VM being killed.

If shadow MMU starts using make_huge_spte() and doesn't add logic to handle this scenario (killing vm or something else) then that is a coding bug of that feature which should be fixed.

#11 lore raw

Re: [PATCH v2 4/6] KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping

Quoted 10 lines · Vipin Sharma
Quoted 7 lines · Sean Christopherson

On Wed, Oct 09, 2024, Vipin Sharma wrote:

Coming back to this, I opted to match the behavior of make_small_spte() and do:

        KVM_BUG_ON(!is_shadow_present_pte(small_spte) || level == PG_LEVEL_4K, kvm);

Should these be two separate KVM_BUG_ON(), to aid in debugging?

I smushed them together to may make_huge_page_split_spte()'s style, and because practically speaking the assertion will never fail. And if it does fail, figuring out what failed wil be quite easy as the two variables being checked are function parameters, i.e. are all but guaranteed to be in RSI and RDX.

Quoted 30 lines
Quoted 25 lines

As explained in commit 3d4415ed75a57, the scenario is meant to be impossible. If the check fails in production, odds are good there's SPTE memory corruption and we _want_ to kill the VM.

KVM: x86/mmu: Bug the VM if KVM tries to split a !hugepage SPTE

Bug the VM instead of simply warning if KVM tries to split a SPTE that is non-present or not-huge. KVM is guaranteed to end up in a broken state as the callers fully expect a valid SPTE, e.g. the shadow MMU will add an rmap entry, and all MMUs will account the expected small page. Returning '0' is also technically wrong now that SHADOW_NONPRESENT_VALUE exists, i.e. would cause KVM to create a potential #VE SPTE.

While it would be possible to have the callers gracefully handle failure, doing so would provide no practical value as the scenario really should be impossible, while the error handling would add a non-trivial amount of noise.

There's also no need to return SHADOW_NONPRESENT_VALUE. KVM_BUG_ON() ensures all vCPUs are kicked out of the guest, so while the return SPTE may be a bit nonsensical, it will never be consumed by hardware. Theoretically, KVM could wander down a weird path in the future, but again, the most likely scenario is that there was host memory corruption, so potential weird paths are the least of KVM's worries at that point.

Quoted 4 lines

More importantly, in the _current_ code, returning SHADOW_NONPRESENT_VALUE happens to be benign, but that's 100% due to make_huge_spte() only being used by the TDP MMU. If the shaduw MMU ever started using make_huge_spte(), returning a !present SPTE would be all but guaranteed to cause fatal problems.

Quoted 5 lines

I think the caller should be given the opportunity to handle a failure. In the current code, TDP is able to handle the error condition, so penalizing a VM seems wrong. We have gone from a state of reduced performance to either very good performance or VM being killed.

The context of how the failure can happens matters. The only way this helper can fail is if there is a blatant bug in the caller, or if there is data corruption of some form. The use of KVM_BUG_ON() is a very clear signal to developers and readers that the caller _must_ ensure the SPTE is shadow-present SPTE, and that the new SPTE will be a huge SPTE. Critically, treating bad input as fatal to the VM also allow the caller to assume success.

Quoted 3 lines

If shadow MMU starts using make_huge_spte() and doesn't add logic to handle this scenario (killing vm or something else) then that is a coding bug of that feature which should be fixed.

No, because allowing make_huge_spte() to fail, even if there is a WARN, adds non-trivial complexity with zero real-world benefit. At some point, success must be assumed/guaranteed. Forcing a future developer to think about the best way to handle a failure that "can't" happen is a waste of their time.

E.g. as an example where allowing failure is both more absurd and more painful, imagine if kvm_mmu_prepare_zap_page() return an error if mmu_lock weren't held. Trying to gracefully handle error would be madness, and so it simply asserts that mmu_lock is held.

#12 lore raw

PATCH v2 5/6 KVM: x86/mmu: Rename make_huge_page_split_spte() to make_small_spte()

In reply to David Matlack

Rename make_huge_page_split_spte() to make_small_spte(). This ensures that the usage of "small_spte" and "huge_spte" are consistent between make_huge_spte() and make_small_spte().

This should also reduce some confusion as make_huge_page_split_spte() almost reads like it will create a huge SPTE, when in fact it is creating a small SPTE to split the huge SPTE.

No functional change intended.

---
 arch/x86/kvm/mmu/mmu.c     | 2 +-
 arch/x86/kvm/mmu/spte.c    | 4 ++--
 arch/x86/kvm/mmu/spte.h    | 4 ++--
 arch/x86/kvm/mmu/tdp_mmu.c | 2 +-
 4 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index 2f8b1ebcbe9c..8967508b63f9 100644
@@ -6693,7 +6693,7 @@ static void shadow_mmu_split_huge_page(struct kvm *kvm,
 			continue;
 		}
 
-		spte = make_huge_page_split_spte(kvm, huge_spte, sp->role, index);
+		spte = make_small_spte(kvm, huge_spte, sp->role, index);
 		mmu_spte_set(sptep, spte);
 		__rmap_add(kvm, cache, slot, sptep, gfn, sp->role.access);
 	}
arch/x86/kvm/mmu/spte.c +2 -2
index a12437bf6e0c..fe010e3404b1 100644
@@ -300,8 +300,8 @@ static u64 make_spte_nonexecutable(u64 spte)
  * This is used during huge page splitting to build the SPTEs that make up the
  * new page table.
  */
-u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte,
-			      union kvm_mmu_page_role role, int index)
+u64 make_small_spte(struct kvm *kvm, u64 huge_spte,
+		    union kvm_mmu_page_role role, int index)
 {
 	u64 child_spte = huge_spte;
 
arch/x86/kvm/mmu/spte.h +2 -2
index 990d599eb827..3aee16e0a575 100644
@@ -501,8 +501,8 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	       unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn,
 	       u64 old_spte, bool prefetch, bool can_unsync,
 	       bool host_writable, u64 *new_spte);
-u64 make_huge_page_split_spte(struct kvm *kvm, u64 huge_spte,
-		      	      union kvm_mmu_page_role role, int index);
+u64 make_small_spte(struct kvm *kvm, u64 huge_spte,
+		    union kvm_mmu_page_role role, int index);
 u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level);
 u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled);
 u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access);
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index be70f0f22550..4c1cd41750ad 100644
@@ -1334,7 +1334,7 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter,
 	 * not been linked in yet and thus is not reachable from any other CPU.
 	 */
 	for (i = 0; i < SPTE_ENT_PER_PAGE; i++)
-		sp->spt[i] = make_huge_page_split_spte(kvm, huge_spte, sp->role, i);
+		sp->spt[i] = make_small_spte(kvm, huge_spte, sp->role, i);
 
 	/*
 	 * Replace the huge spte with a pointer to the populated lower level
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
#13 lore raw

PATCH v2 6/6 KVM: x86/mmu: WARN if huge page recovery triggered during dirty logging

In reply to David Matlack

WARN and bail out of recover_huge_pages_range() if dirty logging is enabled. KVM shouldn't be recovering huge pages during dirty logging anyway, since KVM needs to track writes at 4KiB. However its not out of the possibility that that changes in the future.

If KVM wants to recover huge pages during dirty logging, make_huge_spte() must be updated to write-protect the new huge page mapping. Otherwise, writes through the newly recovered huge page mapping will not be tracked.

Note that this potential risk did not exist back when KVM zapped to recover huge page mappings, since subsequent accesses would just be faulted in at PG_LEVEL_4K if dirty logging was enabled.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/tdp_mmu.c +3 -0
index 4c1cd41750ad..301a2c19bfe9 100644
@@ -1619,6 +1619,9 @@ static void recover_huge_pages_range(struct kvm *kvm,
 	u64 huge_spte;
 	int r;
 
+	if (WARN_ON_ONCE(kvm_slot_dirty_track_enabled(slot)))
+		return;
+
 	rcu_read_lock();
 
 	for_each_tdp_pte_min_level(iter, root, PG_LEVEL_2M, start, end) {
Signature · 1 line
2.46.0.295.g3b9ea8a38a-goog
#14 lore raw
In reply to David Matlack
Quoted 33 lines · David Matlack

Rework the TDP MMU disable-dirty-log path to batch TLB flushes and recover huge page mappings, rather than zapping and flushing for every potential huge page mapping.

With this series, dirty_log_perf_test shows a decrease in the time it takes to disable dirty logging, as well as a decrease in the number of vCPU faults:

 $ ./dirty_log_perf_test -s anonymous_hugetlb_2mb -v 64 -e -b 4g

Before: Disabling dirty logging time: 14.334453428s (131072 flushes)
After: Disabling dirty logging time: 4.794969689s (76 flushes)

Before: 393,599 kvm:kvm_page_fault
After: 262,575 kvm:kvm_page_fault

v2:
- Use a separate iterator to walk down to child SPTEs during huge page recovery [Sean]
- Return SHADOW_NONPRESENT_VALUE in error conditions in make_huge_spte() [Vipin][off-list]

v1: https://lore.kernel.org/kvm/[email protected]/

David Matlack (6):
KVM: x86/mmu: Drop @max_level from kvm_mmu_max_mapping_level()
KVM: x86/mmu: Batch TLB flushes when zapping collapsible TDP MMU SPTEs
KVM: x86/mmu: Refactor TDP MMU iter need resched check
KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of
zapping
KVM: x86/mmu: Rename make_huge_page_split_spte() to make_small_spte()
KVM: x86/mmu: WARN if huge page recovery triggered during dirty
logging

Quoted 9 lines
 arch/x86/include/asm/kvm_host.h |   4 +-
 arch/x86/kvm/mmu/mmu.c          |  16 ++--
 arch/x86/kvm/mmu/mmu_internal.h |   3 +-
 arch/x86/kvm/mmu/spte.c         |  43 +++++++++--
 arch/x86/kvm/mmu/spte.h         |   5 +-
 arch/x86/kvm/mmu/tdp_mmu.c      | 129 +++++++++++++++++---------------
 arch/x86/kvm/mmu/tdp_mmu.h      |   4 +-
 arch/x86/kvm/x86.c              |  18 ++---
 8 files changed, 128 insertions(+), 94 deletions(-)

FYI, these are sitting in kvm-x86 mmu, but will be rebased next week, at which point I'll send the "official" thank yous.

[1/8] KVM: x86/mmu: Drop @max_level from kvm_mmu_max_mapping_level()

      https://github.com/kvm-x86/linux/commit/8ccd51cb5911
[2/8] KVM: x86/mmu: Batch TLB flushes when zapping collapsible TDP MMU SPTEs
      https://github.com/kvm-x86/linux/commit/35ef80eb29ab
[3/8] KVM: x86/mmu: Check yielded_gfn for forward progress iff resched is needed
      https://github.com/kvm-x86/linux/commit/d400ce271d9c
[4/8] KVM: x86/mmu: Demote the WARN on yielded in xxx_cond_resched() to KVM_MMU_WARN_ON
      https://github.com/kvm-x86/linux/commit/012a5c17cba4
[5/8] KVM: x86/mmu: Refactor TDP MMU iter need resched check
      https://github.com/kvm-x86/linux/commit/cb059b9e2432
[6/8] KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping
      https://github.com/kvm-x86/linux/commit/13237fb40c74
[7/8] KVM: x86/mmu: Rename make_huge_page_split_spte() to make_small_spte()
      https://github.com/kvm-x86/linux/commit/1d2a6d0b6438
[8/8] KVM: x86/mmu: WARN if huge page recovery triggered during dirty logging
      https://github.com/kvm-x86/linux/commit/ed5ca61d995f
maintainer Applied
#15 lore raw
In reply to David Matlack
Quoted 8 lines · David Matlack

Rework the TDP MMU disable-dirty-log path to batch TLB flushes and recover huge page mappings, rather than zapping and flushing for every potential huge page mapping.

With this series, dirty_log_perf_test shows a decrease in the time it takes to disable dirty logging, as well as a decrease in the number of vCPU faults:

[...]

Applied to kvm-x86 mmu, thanks! Note, these won't be contiguous due to the insertion of the other patches, but the hashes should be correct (knock wood).

[1/6] KVM: x86/mmu: Drop @max_level from kvm_mmu_max_mapping_level()

      https://github.com/kvm-x86/linux/commit/8ccd51cb5911
[2/6] KVM: x86/mmu: Batch TLB flushes when zapping collapsible TDP MMU SPTEs
      https://github.com/kvm-x86/linux/commit/35ef80eb29ab
[3/6] KVM: x86/mmu: Refactor TDP MMU iter need resched check
      https://github.com/kvm-x86/linux/commit/dd2e7dbc4ae2
[4/6] KVM: x86/mmu: Recover TDP MMU huge page mappings in-place instead of zapping
      https://github.com/kvm-x86/linux/commit/13e2e4f62a4b
[5/6] KVM: x86/mmu: Rename make_huge_page_split_spte() to make_small_spte()
      https://github.com/kvm-x86/linux/commit/430e264b7653
[6/6] KVM: x86/mmu: WARN if huge page recovery triggered during dirty logging
      https://github.com/kvm-x86/linux/commit/06c4cd957b5c

--
https://github.com/kvm-x86/linux/tree/next