Skip to contentSkip to conversation

KVM: x86/mmu: Zap empty TDP leaf page tables

RFC PATCH New Hao Zhang 1 participant 2 messages Aug 31

Subsystems (3)
lore
Share & tools
lore

Hao Zhang

author
#0 lore raw

v2 cover KVM: x86/mmu: Zap empty TDP leaf page tables

From: Hao Zhang <[email protected]>

Hi,

v1 tried to avoid repeated walks of retained empty TDP leaf page tables by adding a per-shadow-page hint. The main v1 feedback was to avoid adding more metadata to struct kvm_mmu_page and to look for a more generic iterator-based approach.

Previous RFC:
https://lore.kernel.org/all/[email protected]/

This RFC v2 is a metadata-free alternative. It adds an opt-in TDP iterator event that tells the caller when the walk has completed a child page table. tdp_mmu_zap_leafs() uses that event to unlink an empty 4K leaf page table after a fully covered 2MiB range has been walked.

The tradeoff is intentional:

* v1/hint keeps the page-table page linked and remembers that it is empty across later invalidations.
* v2/unlink avoids persistent per-MMU-page metadata by making the parent SPTE non-present, so later invalidations naturally stop descending into that empty 4K leaf page table.

That also means v2 can pay allocation/free/refault cost if the same leaf page table is repeatedly emptied and repopulated. To limit that cost, v2 does not unlink a leaf page table if a leaf SPTE in that page table was zapped in the current pass.

When a shadow page is unlinked, tdp_mmu_zap_leafs() also services pending TLB invalidations with kvm_flush_remote_tlbs() before dropping RCU protection. The incoming @flush state can reflect earlier zaps in the walk, not only the just-completed child range, so this RFC uses the existing full flush primitive instead of trying to narrow the flush to the child range.

The patch also avoids changing TDX/mirror external page-table lifetime:
mirror shadow pages are not unlinked by this optimization. Please confirm whether that is the preferred boundary, or whether unlinking mirror leaf page tables is safe after the TDX maintainers audit the external SPT lifetime.

The main RFC question is whether this metadata-free unlink tradeoff is acceptable, or whether KVM should keep retaining empty 4K TDP leaf page tables unless there is an explicit per-page-table hint.

Test setup
==========

All numbers below are from the same x86 host and the same 4-vCPU, 1GiB guest workload. Guest vCPUs continuously fault/write guest memory while the host repeatedly invalidates guest memory with MADV_DONTNEED. The reported values are medians from five repetitions.

"sp-only" is the shadow-present-only iterator prototype that skips non-present shadow-present SPTEs without unlinking empty page tables.

Measured paths:

* tdp_mmu_zap_leafs() duration via paired perf probes.
* vCPU fault-side mmu_lock wait time via queued_read_lock_slowpath() perf probes.

Scenarios:

* same16/same4: repeatedly invalidate the same 16MiB or 4MiB range.
* c16/c4/c2/c1: rolling 16MiB, 4MiB, 2MiB, and 1MiB chunks across the 1GiB guest.

zap_total_ms_med
================

  scenario   original   sp-only   v1 hint   v2 unlink
  --------   --------   -------   -------   ---------
  same16       330.3     221.8      35.3       61.9
  same4        341.3     234.2      54.4       52.4
  c16         1456.2    1696.0    1386.1     1549.9
  c4          1252.0    1290.6    1243.8     1575.7
  c2          1197.0    1173.9    1246.2     1501.3
  c1          1352.8    1198.9    1138.4     1304.2

Relative to the original kernel, v2 reduces repeated same-range zap time by 81% for same16 and 85% for same4. Rolling workloads are mixed:
c16 is +6%, c4 is +26%, c2 is +25%, and c1 is -4%.
Compared to the shadow-present-only iterator change, v2 is mixed on rolling workloads: better on c16, worse on c4/c2/c1. Same-range zap time remains 3.6x faster for same16 and 4.5x faster for same4.

fwait_p999_us_med
=================

  scenario   original   sp-only   v1 hint   v2 unlink
  --------   --------   -------   -------   ---------
  same16          78       162        61          74
  same4           62       152        38          57
  c16             72       242        77         114
  c4             102       248       105         134
  c2             108       270       107         148
  c1              92       266        93          95

An additional no-unlink control on top of v2's iterator changes brought same16/same4 back to the original kernel's zap time, while also bringing rolling fwait back close to original/v1 levels. That confirms the same-range win and the rolling cost both come from unlinking, not from the iterator event itself.

Changes since v1
================

* Replace the per-kvm_mmu_page empty-child hint with a metadata-free unlink-based approach.
* Add an opt-in TDP iterator event for callers that want notification after completing a child page table.
* Do not unlink leaf page tables that had a leaf SPTE zapped in the current pass, to reduce refault churn in rolling invalidations.
* Skip mirror shadow pages, so this optimization does not alter TDX external SPT lifetime.
* Drop the unused skip-child iterator API from the v1/hint direction.
* Move the event-valid fast gate to the caller.

Hao Zhang (1):
KVM: x86/mmu: Zap empty TDP leaf page tables

 arch/x86/kvm/mmu/tdp_iter.c | 29 +++++++++++++--
 arch/x86/kvm/mmu/tdp_iter.h | 10 ++++++
 arch/x86/kvm/mmu/tdp_mmu.c  | 88 +++++++++++++++++++++++++++++++++++++--------
 3 files changed, 109 insertions(+), 18 deletions(-)

base-commit: 45c13f3f9e3bb15fd89ff2864c6f627a3b4b4229

Signature · 1 line
2.15.0
author
#1 lore raw

RFC PATCH v2 1/1 KVM: x86/mmu: Zap empty TDP leaf page tables

From: Hao Zhang <[email protected]>

KVM's TDP MMU intentionally zaps only leaf SPTEs for some invalidations, leaving the page table hierarchy intact so that subsequent faults can reuse the existing shadow pages.

That behavior can be pathological if a later invalidation covers a range whose retained 4K leaf tables contain only non-present SPTEs. KVM still descends into every retained leaf page table and scans all 512 entries even though there is nothing left to zap.

Add an opt-in TDP iterator event that reports when the walk has finished a child page table, and use that event in tdp_mmu_zap_leafs() to unlink empty 4K leaf page tables that are fully covered by the zap.

Do not unlink a leaf page table that had a leaf SPTE zapped in the current pass. Doing so would add allocation/free churn for rolling invalidation workloads that refault the same page table.

Because unlinking a shadow page changes the old leaf-only RCU assumption, flush any pending remote TLBs before dropping RCU protection if a shadow page was removed.

---
 arch/x86/kvm/mmu/tdp_iter.c | 29 +++++++++++++--
 arch/x86/kvm/mmu/tdp_iter.h | 10 ++++++
 arch/x86/kvm/mmu/tdp_mmu.c  | 88 +++++++++++++++++++++++++++++++++++++--------
 3 files changed, 109 insertions(+), 18 deletions(-)
arch/x86/kvm/mmu/tdp_iter.c +26 -3
index 9e17bfa80901..d9be4298410d 100644
@@ -132,7 +132,8 @@ static bool try_step_side(struct tdp_iter *iter)
  * can continue from the next entry in the parent page table. Returns true on a
  * successful step up, false if already in the root page.
  */
-static bool try_step_up(struct tdp_iter *iter)
+static bool try_step_up(struct tdp_iter *iter,
+			struct tdp_iter_child_event *event)
 {
 	if (iter->level == iter->root_level)
 		return false;
@@ -141,6 +142,14 @@ static bool try_step_up(struct tdp_iter *iter)
 	iter->gfn = gfn_round_for_level(iter->gfn, iter->level);
 	tdp_iter_refresh_sptep(iter);
 
+	if (event && !event->valid) {
+		event->sptep = iter->sptep;
+		event->old_spte = iter->old_spte;
+		event->gfn = iter->gfn;
+		event->level = iter->level;
+		event->valid = true;
+	}
+
 	return true;
 }
 
@@ -160,8 +169,12 @@ static bool try_step_up(struct tdp_iter *iter)
  *    SPTE will have already been visited, and so the iterator must also step
  *    to the side again.
  */
-void tdp_iter_next(struct tdp_iter *iter)
+static void __tdp_iter_next(struct tdp_iter *iter,
+			    struct tdp_iter_child_event *event)
 {
+	if (event)
+		event->valid = false;
+
 	if (iter->yielded) {
 		tdp_iter_restart(iter);
 		return;
@@ -173,7 +186,17 @@ void tdp_iter_next(struct tdp_iter *iter)
 	do {
 		if (try_step_side(iter))
 			return;
-	} while (try_step_up(iter));
+	} while (try_step_up(iter, event));
 	iter->valid = false;
 }
 
+void tdp_iter_next(struct tdp_iter *iter)
+{
+	__tdp_iter_next(iter, NULL);
+}
+
+void tdp_iter_next_post_order(struct tdp_iter *iter,
+			      struct tdp_iter_child_event *event)
+{
+	__tdp_iter_next(iter, event);
+}
arch/x86/kvm/mmu/tdp_iter.h +10 -0
index f898d8d0d93c..b7e2605fbf9e 100644
@@ -123,6 +123,14 @@ struct tdp_iter {
 	bool yielded;
 };
 
+struct tdp_iter_child_event {
+	tdp_ptep_t sptep;
+	u64 old_spte;
+	gfn_t gfn;
+	int level;
+	bool valid;
+};
+
 /*
  * Iterates over every SPTE mapping the GFN range [start, end) in a
  * preorder traversal.
@@ -145,6 +153,8 @@ tdp_ptep_t spte_to_child_pt(u64 pte, int level);
 void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root,
 		    int min_level, gfn_t next_last_level_gfn, gfn_t gfn_bits);
 void tdp_iter_next(struct tdp_iter *iter);
+void tdp_iter_next_post_order(struct tdp_iter *iter,
+			      struct tdp_iter_child_event *event);
 void tdp_iter_restart(struct tdp_iter *iter);
 
 #endif /* __KVM_X86_MMU_TDP_ITER_H */
arch/x86/kvm/mmu/tdp_mmu.c +73 -15

119 lines not on this page · raw message

Signature · 1 line
2.15.0