Skip to contentSkip to conversation

KVM: x86/mmu: Run TDP MMU NX huge page recovery under MMU read lock

PATCH Superseded James Houghton 2 participants 12 messages Jun 13, 2025 – Jun 16, 2025

Subsystems (4)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover KVM: x86/mmu: Run TDP MMU NX huge page recovery under MMU read lock

Hi Sean/Paolo,

I'm finishing off Vipin's NX huge page recovery optimization for the TDP MMU from last year.

NX huge page recovery can cause guest performance jitter, originally noticed with network tests in Windows guests. Please see Vipin's earlier performance results[1]. Below is some new data I have collected with the nx_huge_pages_perf_test that I've included with this series.

The NX huge page recovery for the shadow MMU is still done under the MMU write lock, but with the TDP MMU, we can instead do it under the MMU read lock by:

1. Tracking the possible NX huge pages for the two MMUs separately (patch 1).
2. Updating the NX huge page recovery routine for the TDP MMU to
- zap SPTEs atomically, and
- grab tdp_mmu_pages_lock to iterate over the NX huge page list
(patch 3).

I threw in patch 4 because it seems harmless and closer to the "right" thing to do. Feel free to drop it if you don't agree with me. :)

I'm also grabbing David's execute_perf_test[3] while I'm at it. It was dropped before simply because it didn't apply at the time. David's test works well as a stress test for NX huge page recovery when NX huge page recovery is tuned to be very aggressive.

Changes since v3[2]:
- Dropped the move of the sp->nx_huge_page_disallowed check to outside of the tdp_mmu_pages_lock.
- Implemented Sean's array suggestion for possible_nx_huge_pages.
- Implemented some other cleanup suggestions from Sean.
- Made shadow MMU not take the RCU lock in NX huge page recovery.
- Added a selftest for measuring jitter.
- Added David's execute_perf_test[3].

-- Results

$ cat /sys/module/kvm/parameters/nx_huge_pages_recovery_period_ms
100
$ cat /sys/module/kvm/parameters/nx_huge_pages_recovery_ratio
4
$ ./nx_huge_pages_perf_test -b 16G -s anonymous_hugetlb_1gb
[Unpatched] Max fault latency: 8496724 cycles
[Unpatched] Max fault latency: 8404426 cycles
[ Patched ] Max fault latency: 49418 cycles
[ Patched ] Max fault latency: 51948 cycles
$ ./nx_huge_pages_perf_test -b 16G -s anonymous_hugetlb_2mb
[Unpatched] Max fault latency: 5320740 cycles
[Unpatched] Max fault latency: 5384554 cycles
[ Patched ] Max fault latency: 50052 cycles
[ Patched ] Max fault latency: 103774 cycles
$ ./nx_huge_pages_perf_test -b 16G -s anonymous_thp
[Unpatched] Max fault latency: 7625022 cycles
[Unpatched] Max fault latency: 6339934 cycles
[ Patched ] Max fault latency: 107976 cycles
[ Patched ] Max fault latency: 108386 cycles
$ ./nx_huge_pages_perf_test -b 16G -s anonymous
[Unpatched] Max fault latency: 143036 cycles
[Unpatched] Max fault latency: 287444 cycles
[ Patched ] Max fault latency: 274626 cycles
[ Patched ] Max fault latency: 303984 cycles

We can see about a 100x decrease in maximum fault latency for both 2M pages and 1G pages. This test is only timing writes to unmapped pages that are not themselves currently undergoing NX huge page recovery. The test only produces interesting results when NX huge page recovery is actually occurring, so the parameters are tuned to make it very likely for NX huge page recovery to occur in the middle of the test.

Based on latest kvm/next.

[1]: https://lore.kernel.org/kvm/[email protected]/
[2]: https://lore.kernel.org/kvm/[email protected]/
[3]: https://lore.kernel.org/kvm/[email protected]/

David Matlack (1):
KVM: selftests: Introduce a selftest to measure execution performance

James Houghton (3):
KVM: x86/mmu: Only grab RCU lock for nx hugepage recovery for TDP MMU
KVM: selftests: Provide extra mmap flags in vm_mem_add()
KVM: selftests: Add an NX huge pages jitter test

Vipin Sharma (3):
KVM: x86/mmu: Track TDP MMU NX huge pages separately
KVM: x86/mmu: Rename kvm_tdp_mmu_zap_sp() to better indicate its
purpose
KVM: x86/mmu: Recover TDP MMU NX huge pages using MMU read lock

 arch/x86/include/asm/kvm_host.h               |  39 ++-
 arch/x86/kvm/mmu/mmu.c                        | 175 +++++++++-----
 arch/x86/kvm/mmu/mmu_internal.h               |   7 +-
 arch/x86/kvm/mmu/tdp_mmu.c                    |  49 +++-
 arch/x86/kvm/mmu/tdp_mmu.h                    |   3 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../testing/selftests/kvm/execute_perf_test.c | 199 ++++++++++++++++
 .../testing/selftests/kvm/include/kvm_util.h  |   3 +-
 .../testing/selftests/kvm/include/memstress.h |   4 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  15 +-
 tools/testing/selftests/kvm/lib/memstress.c   |  25 +-
 .../kvm/x86/nx_huge_pages_perf_test.c         | 223 ++++++++++++++++++
 .../kvm/x86/private_mem_conversions_test.c    |   2 +-
 13 files changed, 646 insertions(+), 100 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/execute_perf_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/nx_huge_pages_perf_test.c

base-commit: 8046d29dde17002523f94d3e6e0ebe486ce52166

Signature · 1 line
2.50.0.rc2.692.g299adb8693-goog
#1 lore raw

PATCH v4 1/7 KVM: x86/mmu: Track TDP MMU NX huge pages separately

From: Vipin Sharma <[email protected]>

Introduce struct kvm_possible_nx_huge_pages to track the list of possible NX huge pages and the number of pages on the list.

When calculating how many pages to zap, we use the new counts we have (instead of kvm->stat.nx_lpage_splits, which would be the sum of the two new counts).

---
 arch/x86/include/asm/kvm_host.h | 39 ++++++++++++++--------
 arch/x86/kvm/mmu/mmu.c          | 58 +++++++++++++++++++++------------
 arch/x86/kvm/mmu/mmu_internal.h |  7 ++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  4 +--
 4 files changed, 71 insertions(+), 37 deletions(-)
arch/x86/include/asm/kvm_host.h +26 -13
index b4a391929cdba..9df15c9717771 100644
@@ -1334,6 +1334,30 @@ enum kvm_apicv_inhibit {
 	__APICV_INHIBIT_REASON(SEV),			\
 	__APICV_INHIBIT_REASON(LOGICAL_ID_ALIASED)
 
+struct kvm_possible_nx_huge_pages {
+	/*
+	 * A list of kvm_mmu_page structs that, if zapped, could possibly be
+	 * replaced by an NX huge page.  A shadow page is on this list if its
+	 * existence disallows an NX huge page (nx_huge_page_disallowed is set)
+	 * and there are no other conditions that prevent a huge page, e.g.
+	 * the backing host page is huge, dirtly logging is not enabled for its
+	 * memslot, etc...  Note, zapping shadow pages on this list doesn't
+	 * guarantee an NX huge page will be created in its stead, e.g. if the
+	 * guest attempts to execute from the region then KVM obviously can't
+	 * create an NX huge page (without hanging the guest).
+	 */
+	struct list_head pages;
+	u64 nr_pages;
+};
+
+enum kvm_mmu_type {
+	KVM_SHADOW_MMU,
+#ifdef CONFIG_X86_64
+	KVM_TDP_MMU,
+#endif
+	KVM_NR_MMU_TYPES,
+};
+
 struct kvm_arch {
 	unsigned long n_used_mmu_pages;
 	unsigned long n_requested_mmu_pages;
@@ -1346,18 +1370,7 @@ struct kvm_arch {
 	bool pre_fault_allowed;
 	struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES];
 	struct list_head active_mmu_pages;
-	/*
-	 * A list of kvm_mmu_page structs that, if zapped, could possibly be
-	 * replaced by an NX huge page.  A shadow page is on this list if its
-	 * existence disallows an NX huge page (nx_huge_page_disallowed is set)
-	 * and there are no other conditions that prevent a huge page, e.g.
-	 * the backing host page is huge, dirtly logging is not enabled for its
-	 * memslot, etc...  Note, zapping shadow pages on this list doesn't
-	 * guarantee an NX huge page will be created in its stead, e.g. if the
-	 * guest attempts to execute from the region then KVM obviously can't
-	 * create an NX huge page (without hanging the guest).
-	 */
-	struct list_head possible_nx_huge_pages;
+	struct kvm_possible_nx_huge_pages possible_nx_huge_pages[KVM_NR_MMU_TYPES];
 #ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
 	struct kvm_page_track_notifier_head track_notifier_head;
 #endif
@@ -1516,7 +1529,7 @@ struct kvm_arch {
 	 * is held in read mode:
 	 *  - tdp_mmu_roots (above)
 	 *  - the link field of kvm_mmu_page structs used by the TDP MMU
-	 *  - possible_nx_huge_pages;
+	 *  - possible_nx_huge_pages[KVM_TDP_MMU];
 	 *  - the possible_nx_huge_page_link field of kvm_mmu_page structs used
 	 *    by the TDP MMU
 	 * Because the lock is only taken within the MMU lock, strictly
arch/x86/kvm/mmu/mmu.c +38 -20

153 lines not on this page · raw message

arch/x86/kvm/mmu/mmu_internal.h +5 -2
index db8f33e4de624..a8fd2de13f707 100644
@@ -413,7 +413,10 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_level);
 
-void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
-void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
+void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp,
+				 enum kvm_mmu_type mmu_type);
+void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp,
+				   enum kvm_mmu_type mmu_type);
 
+extern unsigned int nx_huge_pages_recovery_ratio;
 #endif /* __KVM_X86_MMU_INTERNAL_H */
arch/x86/kvm/mmu/tdp_mmu.c +2 -2
index 7f3d7229b2c1f..48b070f9f4e13 100644
@@ -355,7 +355,7 @@ static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 
 	spin_lock(&kvm->arch.tdp_mmu_pages_lock);
 	sp->nx_huge_page_disallowed = false;
-	untrack_possible_nx_huge_page(kvm, sp);
+	untrack_possible_nx_huge_page(kvm, sp, KVM_TDP_MMU);
 	spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 }
 
@@ -1303,7 +1303,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 		    fault->req_level >= iter.level) {
 			spin_lock(&kvm->arch.tdp_mmu_pages_lock);
 			if (sp->nx_huge_page_disallowed)
-				track_possible_nx_huge_page(kvm, sp);
+				track_possible_nx_huge_page(kvm, sp, KVM_TDP_MMU);
 			spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 		}
 	}
Signature · 1 line
2.50.0.rc2.692.g299adb8693-goog
#2 lore raw

Re: [PATCH v4 1/7] KVM: x86/mmu: Track TDP MMU NX huge pages separately

kernel test robot noticed the following build errors:

Hi James,

kernel test robot noticed the following build errors:

[auto build test ERROR on 8046d29dde17002523f94d3e6e0ebe486ce52166]

url:    https://github.com/intel-lab-lkp/linux/commits/James-Houghton/KVM-x86-mmu-Track-TDP-MMU-NX-huge-pages-separately/20250614-042620
base:   8046d29dde17002523f94d3e6e0ebe486ce52166
patch link:    https://lore.kernel.org/r/20250613202315.2790592-2-jthoughton%40google.com
patch subject: [PATCH v4 1/7] KVM: x86/mmu: Track TDP MMU NX huge pages separately
config: i386-randconfig-003-20250614 (https://download.01.org/0day-ci/archive/20250614/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250614/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

arch/x86/kvm/mmu/mmu.c: In function 'kvm_recover_nx_huge_pages':

>> arch/x86/kvm/mmu/mmu.c:7609:38: error: 'KVM_TDP_MMU' undeclared (first use in this function)
    7609 |                 else if (mmu_type == KVM_TDP_MMU)
         |                                      ^~~~~~~~~~~

arch/x86/kvm/mmu/mmu.c:7609:38: note: each undeclared identifier is reported only once for each function it appears in

vim +/KVM_TDP_MMU +7609 arch/x86/kvm/mmu/mmu.c

7537
7538 static void kvm_recover_nx_huge_pages(struct kvm *kvm,
7539 enum kvm_mmu_type mmu_type)
7540 {
7541 unsigned long to_zap = nx_huge_pages_to_zap(kvm, mmu_type);
7542 struct list_head *nx_huge_pages;
7543 struct kvm_memory_slot *slot;
7544 struct kvm_mmu_page *sp;
7545 LIST_HEAD(invalid_list);
7546 bool flush = false;
7547 int rcu_idx;
7548
7549 nx_huge_pages = &kvm->arch.possible_nx_huge_pages[mmu_type].pages;
7550
7551 rcu_idx = srcu_read_lock(&kvm->srcu);
7552 write_lock(&kvm->mmu_lock);
7553
7554 /*
7555 * Zapping TDP MMU shadow pages, including the remote TLB flush, must
7556 * be done under RCU protection, because the pages are freed via RCU
7557 * callback.
7558 */
7559 rcu_read_lock();
7560
7561 for ( ; to_zap; --to_zap) {
7562 if (list_empty(nx_huge_pages))
7563 break;
7564
7565 /*
7566 * We use a separate list instead of just using active_mmu_pages
7567 * because the number of shadow pages that be replaced with an
7568 * NX huge page is expected to be relatively small compared to
7569 * the total number of shadow pages. And because the TDP MMU
7570 * doesn't use active_mmu_pages.
7571 */
7572 sp = list_first_entry(nx_huge_pages,
7573 struct kvm_mmu_page,
7574 possible_nx_huge_page_link);
7575 WARN_ON_ONCE(!sp->nx_huge_page_disallowed);
7576 WARN_ON_ONCE(!sp->role.direct);
7577
7578 /*
7579 * Unaccount and do not attempt to recover any NX Huge Pages
7580 * that are being dirty tracked, as they would just be faulted
7581 * back in as 4KiB pages. The NX Huge Pages in this slot will be
7582 * recovered, along with all the other huge pages in the slot,
7583 * when dirty logging is disabled.
7584 *
7585 * Since gfn_to_memslot() is relatively expensive, it helps to
7586 * skip it if it the test cannot possibly return true. On the
7587 * other hand, if any memslot has logging enabled, chances are
7588 * good that all of them do, in which case unaccount_nx_huge_page()
7589 * is much cheaper than zapping the page.
7590 *
7591 * If a memslot update is in progress, reading an incorrect value
7592 * of kvm->nr_memslots_dirty_logging is not a problem: if it is
7593 * becoming zero, gfn_to_memslot() will be done unnecessarily; if
7594 * it is becoming nonzero, the page will be zapped unnecessarily.
7595 * Either way, this only affects efficiency in racy situations,
7596 * and not correctness.
7597 */
7598 slot = NULL;
7599 if (atomic_read(&kvm->nr_memslots_dirty_logging)) {
7600 struct kvm_memslots *slots;
7601
7602 slots = kvm_memslots_for_spte_role(kvm, sp->role);
7603 slot = __gfn_to_memslot(slots, sp->gfn);
7604 WARN_ON_ONCE(!slot);
7605 }
7606
7607 if (slot && kvm_slot_dirty_track_enabled(slot))
7608 unaccount_nx_huge_page(kvm, sp);

> 7609			else if (mmu_type == KVM_TDP_MMU)

7610 flush |= kvm_tdp_mmu_zap_sp(kvm, sp);
7611 else
7612 kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
7613 WARN_ON_ONCE(sp->nx_huge_page_disallowed);
7614
7615 if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {
7616 kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
7617 rcu_read_unlock();
7618
7619 cond_resched_rwlock_write(&kvm->mmu_lock);
7620 flush = false;
7621
7622 rcu_read_lock();
7623 }
7624 }
7625 kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
7626
7627 rcu_read_unlock();
7628
7629 write_unlock(&kvm->mmu_lock);
7630 srcu_read_unlock(&kvm->srcu, rcu_idx);
7631 }
7632

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#3 lore raw

Re: [PATCH v4 1/7] KVM: x86/mmu: Track TDP MMU NX huge pages separately

Quoted 7 lines

All errors (new ones prefixed by >>):

arch/x86/kvm/mmu/mmu.c: In function 'kvm_recover_nx_huge_pages':

Quoted 1 line
Quoted 1 line

arch/x86/kvm/mmu/mmu.c:7609:38: error: 'KVM_TDP_MMU' undeclared (first use in this function)

    7609 |                 else if (mmu_type == KVM_TDP_MMU)
         |                                      ^~~~~~~~~~~

arch/x86/kvm/mmu/mmu.c:7609:38: note: each undeclared identifier is reported only once for each function it appears in

Sorry for not trying to build on i386. :(

Fixup for this, as Sean originally had[1]:

[1]: https://lore.kernel.org/kvm/[email protected]/

arch/x86/include/asm/kvm_host.h +4 -0
index 9df15c9717771..d544a269c1920 100644
@@ -1358,6 +1358,10 @@ enum kvm_mmu_type {
 	KVM_NR_MMU_TYPES,
 };
 
+#ifndef CONFIG_X86_64
+#define KVM_TDP_MMU -1
+#endif
+
 struct kvm_arch {
 	unsigned long n_used_mmu_pages;
 	unsigned long n_requested_mmu_pages;
#4 lore raw

PATCH v4 2/7 KVM: x86/mmu: Rename kvm_tdp_mmu_zap_sp() to better indicate its purpose

In reply to James Houghton

From: Vipin Sharma <[email protected]>

kvm_tdp_mmu_zap_sp() is only used for NX huge page recovery, so rename it to kvm_tdp_mmu_zap_possible_nx_huge_page(). In a future commit, this function will be changed to include logic specific to NX huge page recovery.

---
 arch/x86/kvm/mmu/mmu.c     | 2 +-
 arch/x86/kvm/mmu/tdp_mmu.c | 3 ++-
 arch/x86/kvm/mmu/tdp_mmu.h | 3 ++-
 3 files changed, 5 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index f44d7f3acc179..b074f7bb5cc58 100644
@@ -7607,7 +7607,7 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 		if (slot && kvm_slot_dirty_track_enabled(slot))
 			unaccount_nx_huge_page(kvm, sp);
 		else if (mmu_type == KVM_TDP_MMU)
-			flush |= kvm_tdp_mmu_zap_sp(kvm, sp);
+			flush |= kvm_tdp_mmu_zap_possible_nx_huge_page(kvm, sp);
 		else
 			kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
 		WARN_ON_ONCE(sp->nx_huge_page_disallowed);
arch/x86/kvm/mmu/tdp_mmu.c +2 -1
index 48b070f9f4e13..19907eb04a9c4 100644
@@ -925,7 +925,8 @@ static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root,
 	rcu_read_unlock();
 }
 
-bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
+bool kvm_tdp_mmu_zap_possible_nx_huge_page(struct kvm *kvm,
+					   struct kvm_mmu_page *sp)
 {
 	u64 old_spte;
 
arch/x86/kvm/mmu/tdp_mmu.h +2 -1
index 52acf99d40a00..bd62977c9199e 100644
@@ -64,7 +64,8 @@ static inline struct kvm_mmu_page *tdp_mmu_get_root(struct kvm_vcpu *vcpu,
 }
 
 bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, gfn_t start, gfn_t end, bool flush);
-bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp);
+bool kvm_tdp_mmu_zap_possible_nx_huge_page(struct kvm *kvm,
+					   struct kvm_mmu_page *sp);
 void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 void kvm_tdp_mmu_invalidate_roots(struct kvm *kvm,
 				  enum kvm_tdp_mmu_root_types root_types);
Signature · 1 line
2.50.0.rc2.692.g299adb8693-goog
#5 lore raw

PATCH v4 3/7 KVM: x86/mmu: Recover TDP MMU NX huge pages using MMU read lock

In reply to James Houghton

From: Vipin Sharma <[email protected]>

Use MMU read lock to recover TDP MMU NX huge pages. Iterate over the huge pages list under tdp_mmu_pages_lock protection and unaccount the page before dropping the lock.

We must not zap an SPTE if:
- The SPTE is a root page.
- The SPTE does not point at the SP's page table.

If the SPTE does not point at the SP's page table, then something else has changed the SPTE, so we cannot safely zap it.

Warn if zapping SPTE fails and current SPTE is still pointing to same page table. This should never happen.

There is always a race between dirty logging, vCPU faults, and NX huge page recovery for backing a gfn by an NX huge page or an executable small page. Unaccounting sooner during the list traversal is increasing the window of that race. Functionally, it is okay, because accounting doesn't protect against iTLB multi-hit bug, it is there purely to prevent KVM from bouncing a gfn between two page sizes. The only downside is that a vCPU will end up doing more work in tearing down all the child SPTEs. This should be a very rare race.

Zapping under MMU read lock unblocks vCPUs which are waiting for MMU read lock. This optimizaion is done to solve a guest jitter issue on Windows VM which was observing an increase in network latency.

---
 arch/x86/kvm/mmu/mmu.c     | 99 ++++++++++++++++++++++++--------------
 arch/x86/kvm/mmu/tdp_mmu.c | 42 +++++++++++++---
 2 files changed, 98 insertions(+), 43 deletions(-)
arch/x86/kvm/mmu/mmu.c +63 -36
index b074f7bb5cc58..10ba328b664d7 100644
@@ -7535,12 +7535,40 @@ static unsigned long nx_huge_pages_to_zap(struct kvm *kvm,
 	return ratio ? DIV_ROUND_UP(pages, ratio) : 0;
 }
 
+static bool kvm_mmu_sp_dirty_logging_enabled(struct kvm *kvm,
+					     struct kvm_mmu_page *sp)
+{
+	struct kvm_memory_slot *slot = NULL;
+
+	/*
+	 * Since gfn_to_memslot() is relatively expensive, it helps to skip it if
+	 * it the test cannot possibly return true.  On the other hand, if any
+	 * memslot has logging enabled, chances are good that all of them do, in
+	 * which case unaccount_nx_huge_page() is much cheaper than zapping the
+	 * page.
+	 *
+	 * If a memslot update is in progress, reading an incorrect value of
+	 * kvm->nr_memslots_dirty_logging is not a problem: if it is becoming
+	 * zero, gfn_to_memslot() will be done unnecessarily; if it is becoming
+	 * nonzero, the page will be zapped unnecessarily.  Either way, this only
+	 * affects efficiency in racy situations, and not correctness.
+	 */
+	if (atomic_read(&kvm->nr_memslots_dirty_logging)) {
+		struct kvm_memslots *slots;
+
+		slots = kvm_memslots_for_spte_role(kvm, sp->role);
+		slot = __gfn_to_memslot(slots, sp->gfn);
+		WARN_ON_ONCE(!slot);
+	}
+	return slot && kvm_slot_dirty_track_enabled(slot);
+}
+
 static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 				      enum kvm_mmu_type mmu_type)
 {
 	unsigned long to_zap = nx_huge_pages_to_zap(kvm, mmu_type);
+	bool is_tdp = mmu_type == KVM_TDP_MMU;
 	struct list_head *nx_huge_pages;
-	struct kvm_memory_slot *slot;
 	struct kvm_mmu_page *sp;
 	LIST_HEAD(invalid_list);
 	bool flush = false;
@@ -7549,7 +7577,10 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 	nx_huge_pages = &kvm->arch.possible_nx_huge_pages[mmu_type].pages;
 
 	rcu_idx = srcu_read_lock(&kvm->srcu);
-	write_lock(&kvm->mmu_lock);
+	if (is_tdp)
+		read_lock(&kvm->mmu_lock);
+	else
+		write_lock(&kvm->mmu_lock);
 
 	/*
 	 * Zapping TDP MMU shadow pages, including the remote TLB flush, must
@@ -7559,8 +7590,13 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 	rcu_read_lock();
 
 	for ( ; to_zap; --to_zap) {
-		if (list_empty(nx_huge_pages))
+		if (is_tdp)
+			spin_lock(&kvm->arch.tdp_mmu_pages_lock);
+		if (list_empty(nx_huge_pages)) {
+			if (is_tdp)
+				spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
 			break;
+		}
 
 		/*
 		 * We use a separate list instead of just using active_mmu_pages
@@ -7575,50 +7611,38 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 		WARN_ON_ONCE(!sp->nx_huge_page_disallowed);
 		WARN_ON_ONCE(!sp->role.direct);
 
+		unaccount_nx_huge_page(kvm, sp);
+
+		if (is_tdp)
+			spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
+
 		/*
-		 * Unaccount and do not attempt to recover any NX Huge Pages
-		 * that are being dirty tracked, as they would just be faulted
-		 * back in as 4KiB pages. The NX Huge Pages in this slot will be
-		 * recovered, along with all the other huge pages in the slot,
-		 * when dirty logging is disabled.
-		 *
-		 * Since gfn_to_memslot() is relatively expensive, it helps to
-		 * skip it if it the test cannot possibly return true.  On the
-		 * other hand, if any memslot has logging enabled, chances are
-		 * good that all of them do, in which case unaccount_nx_huge_page()
-		 * is much cheaper than zapping the page.
-		 *
-		 * If a memslot update is in progress, reading an incorrect value
-		 * of kvm->nr_memslots_dirty_logging is not a problem: if it is
-		 * becoming zero, gfn_to_memslot() will be done unnecessarily; if
-		 * it is becoming nonzero, the page will be zapped unnecessarily.
-		 * Either way, this only affects efficiency in racy situations,
-		 * and not correctness.
+		 * Do not attempt to recover any NX Huge Pages that are being
+		 * dirty tracked, as they would just be faulted back in as 4KiB
+		 * pages. The NX Huge Pages in this slot will be recovered,
+		 * along with all the other huge pages in the slot, when dirty
+		 * logging is disabled.
 		 */
-		slot = NULL;
-		if (atomic_read(&kvm->nr_memslots_dirty_logging)) {
-			struct kvm_memslots *slots;
+		if (!kvm_mmu_sp_dirty_logging_enabled(kvm, sp)) {
+			if (is_tdp)
+				flush |= kvm_tdp_mmu_zap_possible_nx_huge_page(kvm, sp);
+			else
+				kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
 
-			slots = kvm_memslots_for_spte_role(kvm, sp->role);
-			slot = __gfn_to_memslot(slots, sp->gfn);
-			WARN_ON_ONCE(!slot);
 		}
 
-		if (slot && kvm_slot_dirty_track_enabled(slot))
-			unaccount_nx_huge_page(kvm, sp);
-		else if (mmu_type == KVM_TDP_MMU)
-			flush |= kvm_tdp_mmu_zap_possible_nx_huge_page(kvm, sp);
-		else
-			kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
 		WARN_ON_ONCE(sp->nx_huge_page_disallowed);
 
 		if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {
 			kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
 			rcu_read_unlock();
 
-			cond_resched_rwlock_write(&kvm->mmu_lock);
-			flush = false;
+			if (is_tdp)
+				cond_resched_rwlock_read(&kvm->mmu_lock);
+			else
+				cond_resched_rwlock_write(&kvm->mmu_lock);
 
+			flush = false;
 			rcu_read_lock();
 		}
 	}
@@ -7626,7 +7650,10 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 
 	rcu_read_unlock();
 
-	write_unlock(&kvm->mmu_lock);
+	if (is_tdp)
+		read_unlock(&kvm->mmu_lock);
+	else
+		write_unlock(&kvm->mmu_lock);
 	srcu_read_unlock(&kvm->srcu, rcu_idx);
 }
 
arch/x86/kvm/mmu/tdp_mmu.c +35 -7

56 lines not on this page · raw message

Signature · 1 line
2.50.0.rc2.692.g299adb8693-goog
#6 lore raw

Re: [PATCH v4 3/7] KVM: x86/mmu: Recover TDP MMU NX huge pages using MMU read lock

kernel test robot noticed the following build errors:

Hi James,

kernel test robot noticed the following build errors:

[auto build test ERROR on 8046d29dde17002523f94d3e6e0ebe486ce52166]

url:    https://github.com/intel-lab-lkp/linux/commits/James-Houghton/KVM-x86-mmu-Track-TDP-MMU-NX-huge-pages-separately/20250614-042620
base:   8046d29dde17002523f94d3e6e0ebe486ce52166
patch link:    https://lore.kernel.org/r/20250613202315.2790592-4-jthoughton%40google.com
patch subject: [PATCH v4 3/7] KVM: x86/mmu: Recover TDP MMU NX huge pages using MMU read lock
config: i386-buildonly-randconfig-003-20250614 (https://download.01.org/0day-ci/archive/20250614/[email protected]/config)
compiler: clang version 20.1.2 (https://github.com/llvm/llvm-project 58df0ef89dd64126512e4ee27b4ac3fd8ddf6247)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250614/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

arch/x86/kvm/mmu/mmu.c:7570:28: error: use of undeclared identifier 'KVM_TDP_MMU'

    7570 |         bool is_tdp = mmu_type == KVM_TDP_MMU;
         |                                   ^
>> arch/x86/kvm/mmu/mmu.c:7594:25: error: no member named 'tdp_mmu_pages_lock' in 'struct kvm_arch'

7594 | spin_lock(&kvm->arch.tdp_mmu_pages_lock);
| ~~~~~~~~~ ^
arch/x86/kvm/mmu/mmu.c:7597:28: error: no member named 'tdp_mmu_pages_lock' in 'struct kvm_arch'
7597 | spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
| ~~~~~~~~~ ^
arch/x86/kvm/mmu/mmu.c:7617:27: error: no member named 'tdp_mmu_pages_lock' in 'struct kvm_arch'
7617 | spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
| ~~~~~~~~~ ^
4 errors generated.

vim +7594 arch/x86/kvm/mmu/mmu.c

7565
7566 static void kvm_recover_nx_huge_pages(struct kvm *kvm,
7567 enum kvm_mmu_type mmu_type)
7568 {
7569 unsigned long to_zap = nx_huge_pages_to_zap(kvm, mmu_type);

> 7570		bool is_tdp = mmu_type == KVM_TDP_MMU;
  7571		struct list_head *nx_huge_pages;
  7572		struct kvm_mmu_page *sp;
  7573		LIST_HEAD(invalid_list);
  7574		bool flush = false;
  7575		int rcu_idx;
  7576	
  7577		nx_huge_pages = &kvm->arch.possible_nx_huge_pages[mmu_type].pages;
  7578	
  7579		rcu_idx = srcu_read_lock(&kvm->srcu);
  7580		if (is_tdp)
  7581			read_lock(&kvm->mmu_lock);
  7582		else
  7583			write_lock(&kvm->mmu_lock);
  7584	
  7585		/*
  7586		 * Zapping TDP MMU shadow pages, including the remote TLB flush, must
  7587		 * be done under RCU protection, because the pages are freed via RCU
  7588		 * callback.
  7589		 */
  7590		rcu_read_lock();
  7591	
  7592		for ( ; to_zap; --to_zap) {
  7593			if (is_tdp)
> 7594				spin_lock(&kvm->arch.tdp_mmu_pages_lock);

7595 if (list_empty(nx_huge_pages)) {
7596 if (is_tdp)
7597 spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
7598 break;
7599 }
7600
7601 /*
7602 * We use a separate list instead of just using active_mmu_pages
7603 * because the number of shadow pages that be replaced with an
7604 * NX huge page is expected to be relatively small compared to
7605 * the total number of shadow pages. And because the TDP MMU
7606 * doesn't use active_mmu_pages.
7607 */
7608 sp = list_first_entry(nx_huge_pages,
7609 struct kvm_mmu_page,
7610 possible_nx_huge_page_link);
7611 WARN_ON_ONCE(!sp->nx_huge_page_disallowed);
7612 WARN_ON_ONCE(!sp->role.direct);
7613
7614 unaccount_nx_huge_page(kvm, sp);
7615
7616 if (is_tdp)
7617 spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
7618
7619 /*
7620 * Do not attempt to recover any NX Huge Pages that are being
7621 * dirty tracked, as they would just be faulted back in as 4KiB
7622 * pages. The NX Huge Pages in this slot will be recovered,
7623 * along with all the other huge pages in the slot, when dirty
7624 * logging is disabled.
7625 */
7626 if (!kvm_mmu_sp_dirty_logging_enabled(kvm, sp)) {
7627 if (is_tdp)
7628 flush |= kvm_tdp_mmu_zap_possible_nx_huge_page(kvm, sp);
7629 else
7630 kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
7631
7632 }
7633
7634 WARN_ON_ONCE(sp->nx_huge_page_disallowed);
7635
7636 if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {
7637 kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
7638 rcu_read_unlock();
7639
7640 if (is_tdp)
7641 cond_resched_rwlock_read(&kvm->mmu_lock);
7642 else
7643 cond_resched_rwlock_write(&kvm->mmu_lock);
7644
7645 flush = false;
7646 rcu_read_lock();
7647 }
7648 }
7649 kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
7650
7651 rcu_read_unlock();
7652
7653 if (is_tdp)
7654 read_unlock(&kvm->mmu_lock);
7655 else
7656 write_unlock(&kvm->mmu_lock);
7657 srcu_read_unlock(&kvm->srcu, rcu_idx);
7658 }
7659

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#7 lore raw

Re: [PATCH v4 3/7] KVM: x86/mmu: Recover TDP MMU NX huge pages using MMU read lock

Quoted 6 lines

All errors (new ones prefixed by >>):

arch/x86/kvm/mmu/mmu.c:7570:28: error: use of undeclared identifier 'KVM_TDP_MMU'

    7570 |         bool is_tdp = mmu_type == KVM_TDP_MMU;
         |                                   ^
Quoted 1 line
Quoted 1 line

arch/x86/kvm/mmu/mmu.c:7594:25: error: no member named 'tdp_mmu_pages_lock' in 'struct kvm_arch'

Quoted 9 lines

7594 | spin_lock(&kvm->arch.tdp_mmu_pages_lock);
| ~~~~~~~~~ ^
arch/x86/kvm/mmu/mmu.c:7597:28: error: no member named 'tdp_mmu_pages_lock' in 'struct kvm_arch'
7597 | spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
| ~~~~~~~~~ ^
arch/x86/kvm/mmu/mmu.c:7617:27: error: no member named 'tdp_mmu_pages_lock' in 'struct kvm_arch'
7617 | spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
| ~~~~~~~~~ ^
4 errors generated.

Fixup for this below.

I also realized that the variable name is_tdp is bad/misleading, so I've changed it to is_tdp_mmu as part of this fixup too. Sean/Paolo, let me know if I should just go ahead and post the fixed series, given the size of this fixup.

I don't really like having to #ifdef all the places where we take tdp_mmu_pages_lock, but I couldn't find a way to avoid that. Even doing

  #ifndef CONFIG_X86_64
  #define is_tdp_mmu false
  #endif

didn't work. :(

Anyway, here's the fixup:

arch/x86/kvm/mmu/mmu.c +15 -9
index 10ba328b664d7..7df1b4ead705b 100644
@@ -7564,10 +7564,10 @@ static bool kvm_mmu_sp_dirty_logging_enabled(struct kvm *kvm,
 }
 
 static void kvm_recover_nx_huge_pages(struct kvm *kvm,
-				      enum kvm_mmu_type mmu_type)
+				      const enum kvm_mmu_type mmu_type)
 {
 	unsigned long to_zap = nx_huge_pages_to_zap(kvm, mmu_type);
-	bool is_tdp = mmu_type == KVM_TDP_MMU;
+	bool is_tdp_mmu = mmu_type == KVM_TDP_MMU;
 	struct list_head *nx_huge_pages;
 	struct kvm_mmu_page *sp;
 	LIST_HEAD(invalid_list);
@@ -7577,7 +7577,7 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 	nx_huge_pages = &kvm->arch.possible_nx_huge_pages[mmu_type].pages;
 
 	rcu_idx = srcu_read_lock(&kvm->srcu);
-	if (is_tdp)
+	if (is_tdp_mmu)
 		read_lock(&kvm->mmu_lock);
 	else
 		write_lock(&kvm->mmu_lock);
@@ -7590,11 +7590,15 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 	rcu_read_lock();
 
 	for ( ; to_zap; --to_zap) {
-		if (is_tdp)
+#ifdef CONFIG_X86_64
+		if (is_tdp_mmu)
 			spin_lock(&kvm->arch.tdp_mmu_pages_lock);
+#endif
 		if (list_empty(nx_huge_pages)) {
-			if (is_tdp)
+#ifdef CONFIG_X86_64
+			if (is_tdp_mmu)
 				spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
+#endif
 			break;
 		}
 
@@ -7613,8 +7617,10 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 
 		unaccount_nx_huge_page(kvm, sp);
 
-		if (is_tdp)
+#ifdef CONFIG_X86_64
+		if (is_tdp_mmu)
 			spin_unlock(&kvm->arch.tdp_mmu_pages_lock);
+#endif
 
 		/*
 		 * Do not attempt to recover any NX Huge Pages that are being
@@ -7624,7 +7630,7 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 		 * logging is disabled.
 		 */
 		if (!kvm_mmu_sp_dirty_logging_enabled(kvm, sp)) {
-			if (is_tdp)
+			if (is_tdp_mmu)
 				flush |= kvm_tdp_mmu_zap_possible_nx_huge_page(kvm, sp);
 			else
 				kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
@@ -7637,7 +7643,7 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 			kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
 			rcu_read_unlock();
 
-			if (is_tdp)
+			if (is_tdp_mmu)
 				cond_resched_rwlock_read(&kvm->mmu_lock);
 			else
 				cond_resched_rwlock_write(&kvm->mmu_lock);
@@ -7650,7 +7656,7 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 
 	rcu_read_unlock();
 
-	if (is_tdp)
+	if (is_tdp_mmu)
 		read_unlock(&kvm->mmu_lock);
 	else
 		write_unlock(&kvm->mmu_lock);
#8 lore raw

PATCH v4 4/7 KVM: x86/mmu: Only grab RCU lock for nx hugepage recovery for TDP MMU

In reply to James Houghton

Now that we have separate paths for the TDP MMU, it is trivial to only grab rcu_read_lock() for the TDP MMU case. We do not need to grab it for the shadow MMU, as pages are not RCU-freed in that case.

---
 arch/x86/kvm/mmu/mmu.c | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)
arch/x86/kvm/mmu/mmu.c +18 -16
index 10ba328b664d7..51df92973d574 100644
@@ -7577,17 +7577,18 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 	nx_huge_pages = &kvm->arch.possible_nx_huge_pages[mmu_type].pages;
 
 	rcu_idx = srcu_read_lock(&kvm->srcu);
-	if (is_tdp)
+	if (is_tdp) {
 		read_lock(&kvm->mmu_lock);
-	else
+		/*
+		 * Zapping TDP MMU shadow pages, including the remote TLB flush,
+		 * must be done under RCU protection, because the pages are
+		 * freed via RCU callback.
+		 */
+		rcu_read_lock();
+	} else {
 		write_lock(&kvm->mmu_lock);
+	}
 
-	/*
-	 * Zapping TDP MMU shadow pages, including the remote TLB flush, must
-	 * be done under RCU protection, because the pages are freed via RCU
-	 * callback.
-	 */
-	rcu_read_lock();
 
 	for ( ; to_zap; --to_zap) {
 		if (is_tdp)
@@ -7635,25 +7636,26 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm,
 
 		if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {
 			kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
-			rcu_read_unlock();
 
-			if (is_tdp)
+			if (is_tdp) {
+				rcu_read_unlock();
 				cond_resched_rwlock_read(&kvm->mmu_lock);
-			else
+				rcu_read_lock();
+			} else {
 				cond_resched_rwlock_write(&kvm->mmu_lock);
+			}
 
 			flush = false;
-			rcu_read_lock();
 		}
 	}
 	kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
 
-	rcu_read_unlock();
-
-	if (is_tdp)
+	if (is_tdp) {
+		rcu_read_unlock();
 		read_unlock(&kvm->mmu_lock);
-	else
+	} else {
 		write_unlock(&kvm->mmu_lock);
+	}
 	srcu_read_unlock(&kvm->srcu, rcu_idx);
 }
 
Signature · 1 line
2.50.0.rc2.692.g299adb8693-goog
#9 lore raw

PATCH v4 5/7 KVM: selftests: Introduce a selftest to measure execution performance

In reply to James Houghton

From: David Matlack <[email protected]>

Introduce a new selftest, execute_perf_test, that uses the perf_test_util framework to measure the performance of executing code within a VM. This test is similar to the other perf_test_util-based tests in that it spins up a variable number of vCPUs and runs them concurrently, accessing memory.

In order to support execution, extend perf_test_util to populate guest memory with return instructions rather than random garbage. This way memory can be execute simply by calling it.

Currently only x86_64 supports execution, but other architectures can be easily added by providing their return code instruction.

---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../testing/selftests/kvm/execute_perf_test.c | 199 ++++++++++++++++++
 .../testing/selftests/kvm/include/memstress.h |   4 +
 tools/testing/selftests/kvm/lib/memstress.c   |  25 ++-
 4 files changed, 227 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/execute_perf_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 38b95998e1e6b..0dc435e944632 100644
@@ -137,6 +137,7 @@ TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test
 TEST_GEN_PROGS_x86 += access_tracking_perf_test
 TEST_GEN_PROGS_x86 += coalesced_io_test
 TEST_GEN_PROGS_x86 += dirty_log_perf_test
+TEST_GEN_PROGS_x86 += execute_perf_test
 TEST_GEN_PROGS_x86 += guest_memfd_test
 TEST_GEN_PROGS_x86 += hardware_disable_test
 TEST_GEN_PROGS_x86 += memslot_modification_stress_test
tools/testing/selftests/kvm/execute_perf_test.c new file +199 -0

199 lines not on this page · raw message

tools/testing/selftests/kvm/include/memstress.h +4 -0
index 9071eb6dea60a..ab2a0c05e3fd2 100644
@@ -50,6 +50,9 @@ struct memstress_args {
  	/* Test is done, stop running vCPUs. */
  	bool stop_vcpus;
 
+	/* If vCPUs should execute from memory. */
+	bool execute;
+
 	struct memstress_vcpu_args vcpu_args[KVM_MAX_VCPUS];
 };
 
@@ -63,6 +66,7 @@ void memstress_destroy_vm(struct kvm_vm *vm);
 
 void memstress_set_write_percent(struct kvm_vm *vm, uint32_t write_percent);
 void memstress_set_random_access(struct kvm_vm *vm, bool random_access);
+void memstress_set_execute(struct kvm_vm *vm, bool execute);
 
 void memstress_start_vcpu_threads(int vcpus, void (*vcpu_fn)(struct memstress_vcpu_args *));
 void memstress_join_vcpu_threads(int vcpus);
tools/testing/selftests/kvm/lib/memstress.c +23 -2
index 313277486a1de..49677742ec92d 100644
@@ -40,6 +40,16 @@ static bool all_vcpu_threads_running;
 
 static struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
 
+/*
+ * When writing to guest memory, write the opcode for the `ret` instruction so
+ * that subsequent iteractions can exercise instruction fetch by calling the
+ * memory.
+ *
+ * NOTE: Non-x86 architectures would to use different values here to support
+ * execute.
+ */
+#define RETURN_OPCODE 0xC3
+
 /*
  * Continuously write to the first 8 bytes of each page in the
  * specified region.
@@ -75,8 +85,10 @@ void memstress_guest_code(uint32_t vcpu_idx)
 
 			addr = gva + (page * args->guest_page_size);
 
-			if (__guest_random_bool(&rand_state, args->write_percent))
-				*(uint64_t *)addr = 0x0123456789ABCDEF;
+			if (args->execute)
+				((void (*)(void)) addr)();
+			else if (__guest_random_bool(&rand_state, args->write_percent))
+				*(uint64_t *)addr = RETURN_OPCODE;
 			else
 				READ_ONCE(*(uint64_t *)addr);
 		}
@@ -259,6 +271,15 @@ void __weak memstress_setup_nested(struct kvm_vm *vm, int nr_vcpus, struct kvm_v
 	exit(KSFT_SKIP);
 }
 
+void memstress_set_execute(struct kvm_vm *vm, bool execute)
+{
+#ifndef __x86_64__
+	TEST_FAIL("Execute not supported on thhis architecture; see RETURN_OPCODE.");
+#endif
+	memstress_args.execute = execute;
+	sync_global_to_guest(vm, memstress_args);
+}
+
 static void *vcpu_thread_main(void *data)
 {
 	struct vcpu_thread *vcpu = data;
Signature · 1 line
2.50.0.rc2.692.g299adb8693-goog
#10 lore raw

PATCH v4 6/7 KVM: selftests: Provide extra mmap flags in vm_mem_add()

In reply to James Houghton

The immediate application here is to allow selftests to pass MAP_POPULATE (to time fault time without having to allocate guest memory).

---
 tools/testing/selftests/kvm/include/kvm_util.h    |  3 ++-
 tools/testing/selftests/kvm/lib/kvm_util.c        | 15 +++++++++------
 .../kvm/x86/private_mem_conversions_test.c        |  2 +-
 3 files changed, 12 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +2 -1
index bee65ca087217..4aafd5bf786e2 100644
@@ -629,7 +629,8 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm,
 	uint32_t flags);
 void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 		uint64_t guest_paddr, uint32_t slot, uint64_t npages,
-		uint32_t flags, int guest_memfd_fd, uint64_t guest_memfd_offset);
+		uint32_t flags, int guest_memfd_fd, uint64_t guest_memfd_offset,
+		int extra_mmap_flags);
 
 #ifndef vm_arch_has_protected_memory
 static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm)
tools/testing/selftests/kvm/lib/kvm_util.c +9 -6
index a055343a7bf75..8157a0fd7f8b3 100644
@@ -977,13 +977,15 @@ void vm_set_user_memory_region2(struct kvm_vm *vm, uint32_t slot, uint32_t flags
 /* FIXME: This thing needs to be ripped apart and rewritten. */
 void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 		uint64_t guest_paddr, uint32_t slot, uint64_t npages,
-		uint32_t flags, int guest_memfd, uint64_t guest_memfd_offset)
+		uint32_t flags, int guest_memfd, uint64_t guest_memfd_offset,
+		int extra_mmap_flags)
 {
 	int ret;
 	struct userspace_mem_region *region;
 	size_t backing_src_pagesz = get_backing_src_pagesz(src_type);
 	size_t mem_size = npages * vm->page_size;
 	size_t alignment;
+	int mmap_flags;
 
 	TEST_REQUIRE_SET_USER_MEMORY_REGION2();
 
@@ -1066,9 +1068,11 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 		region->fd = kvm_memfd_alloc(region->mmap_size,
 					     src_type == VM_MEM_SRC_SHARED_HUGETLB);
 
+	mmap_flags = vm_mem_backing_src_alias(src_type)->flag |
+		     extra_mmap_flags;
+
 	region->mmap_start = mmap(NULL, region->mmap_size,
-				  PROT_READ | PROT_WRITE,
-				  vm_mem_backing_src_alias(src_type)->flag,
+				  PROT_READ | PROT_WRITE, mmap_flags,
 				  region->fd, 0);
 	TEST_ASSERT(region->mmap_start != MAP_FAILED,
 		    __KVM_SYSCALL_ERROR("mmap()", (int)(unsigned long)MAP_FAILED));
@@ -1143,8 +1147,7 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
 	/* If shared memory, create an alias. */
 	if (region->fd >= 0) {
 		region->mmap_alias = mmap(NULL, region->mmap_size,
-					  PROT_READ | PROT_WRITE,
-					  vm_mem_backing_src_alias(src_type)->flag,
+					  PROT_READ | PROT_WRITE, mmap_flags,
 					  region->fd, 0);
 		TEST_ASSERT(region->mmap_alias != MAP_FAILED,
 			    __KVM_SYSCALL_ERROR("mmap()",  (int)(unsigned long)MAP_FAILED));
@@ -1159,7 +1162,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm,
 				 uint64_t guest_paddr, uint32_t slot,
 				 uint64_t npages, uint32_t flags)
 {
-	vm_mem_add(vm, src_type, guest_paddr, slot, npages, flags, -1, 0);
+	vm_mem_add(vm, src_type, guest_paddr, slot, npages, flags, -1, 0, 0);
 }
 
 /*
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +1 -1
index 82a8d88b5338e..637e9e57fce46 100644
@@ -399,7 +399,7 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, uint32_t
 	for (i = 0; i < nr_memslots; i++)
 		vm_mem_add(vm, src_type, BASE_DATA_GPA + slot_size * i,
 			   BASE_DATA_SLOT + i, slot_size / vm->page_size,
-			   KVM_MEM_GUEST_MEMFD, memfd, slot_size * i);
+			   KVM_MEM_GUEST_MEMFD, memfd, slot_size * i, 0);
 
 	for (i = 0; i < nr_vcpus; i++) {
 		uint64_t gpa =  BASE_DATA_GPA + i * per_cpu_size;
Signature · 1 line
2.50.0.rc2.692.g299adb8693-goog
#11 lore raw

PATCH v4 7/7 KVM: selftests: Add an NX huge pages jitter test

In reply to James Houghton

Add a test that checks how much NX huge page recovery affects vCPUs that are faulting on pages not undergoing NX huge page recovery. To do this, this test uses a single vCPU to touch all of guest memory. After every 1G of guest memory, it will switch from writing to executing. Only the writes are timed.

With this setup, while the guest is in the middle of reading a 1G region, NX huge page recovery (provided it is set aggressive enough) will start to recover huge pages in the previous 1G region.

---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../kvm/x86/nx_huge_pages_perf_test.c         | 223 ++++++++++++++++++
 2 files changed, 224 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/nx_huge_pages_perf_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 0dc435e944632..4b5be9f0bac5b 100644
@@ -88,6 +88,7 @@ TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
 TEST_GEN_PROGS_x86 += x86/monitor_mwait_test
 TEST_GEN_PROGS_x86 += x86/nested_emulation_test
 TEST_GEN_PROGS_x86 += x86/nested_exceptions_test
+TEST_GEN_PROGS_x86 += x86/nx_huge_pages_perf_test
 TEST_GEN_PROGS_x86 += x86/platform_info_test
 TEST_GEN_PROGS_x86 += x86/pmu_counters_test
 TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test
tools/testing/selftests/kvm/x86/nx_huge_pages_perf_test.c new file +223 -0

223 lines not on this page · raw message

Signature · 1 line
2.50.0.rc2.692.g299adb8693-goog