Skip to contentSkip to conversation

mm: multi-gen LRU: Walk secondary MMU page tables while aging

PATCH Superseded James Houghton 7 participants 44 messages Jul 24, 2024 – Aug 30, 2024

Subsystems (15)

v6 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v6 cover mm: multi-gen LRU: Walk secondary MMU page tables while aging

This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim.

This series does the following:
1. Improve locking for the existing test/clear_young notifiers for x86

    and arm64.

2. Add a fast_only parameter into test_young() and clear_young(), and add helper functions for using the new parameter (e.g. mmu_notifier_clear_young_fast_only(). Non-trivially implement the fast-only test_young() and clear_young() for x86_64.
3. Incorporate mmu_notifier_clear_young_fast_only() into MGLRU aging.
4. Add an MGLRU mode (-l) to access_tracking_perf_test to show that aging is working properly.

Please note that mmu_notifier_test_young_fast_only() is added but not used in this series. I am happy to remove it if that would be appropriate.

The fast-only notifiers serve a particular purpose: for aging, we neither want to delay other operations (e.g. unmapping for eviction) nor do we want to be delayed by these other operations ourselves. By default, the implementations of test_young() and clear_young() are meant to be *accurate*, not fast. The fast-only notifiers will only give age information that can be gathered fast.

The fast-only notifiers are non-trivially implemented for only x86_64 right now (as the KVM/x86 TDP MMU is the only secondary MMU that supports lockless Accessed bit harvesting).

To make aging work for more than just x86, the fast-only clear_young() notifier must be non-trivially implemented by those other architectures and HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY needs to be set.

access_tracking_perf_test now has a mode (-p) to check performance of MGLRU aging while the VM is faulting memory in. See the v4 cover letter[2] for performance data collected with this test.

Previous versions of this series included logic in MGLRU and KVM to support batching the updates to secondary page tables. This version removes this logic, as it was complex and not necessary to enable proactive reclaim. This optimization, as well as enabling aging for arm64 and powerpc, can be done in a later series.

=== Previous Versions ===

Since v5[1]:
- Reworked test_clear_young_fast_only() into a new parameter for the existing notifiers (thanks Sean).
- Added mmu_notifier.has_fast_aging to tell mm if calling fast-only notifiers should be done.
- Added mm_has_fast_young_notifiers() to inform users if calling fast-only notifier helpers is worthwhile (for look-around to use).
- Changed MGLRU to invoke a single notifier instead of two when aging and doing look-around (thanks Yu).
- For KVM/x86, check indirect_shadow_pages > 0 instead of kvm_memslots_have_rmaps() when collecting age information (thanks Sean).
- For KVM/arm, some fixes from Oliver.
- Small fixes to access_tracking_perf_test.
- Added missing !MMU_NOTIFIER version of mmu_notifier_clear_young().

Since v4[2]:
- Removed Kconfig that controlled when aging was enabled. Aging will be done whenever the architecture supports it (thanks Yu).
- Added a new MMU notifier, test_clear_young_fast_only(), specifically for MGLRU to use.
- Add kvm_fast_{test_,}age_gfn, implemented by x86.
- Fix locking for clear_flush_young().
- Added KVM_MMU_NOTIFIER_YOUNG_LOCKLESS to clean up locking changes (thanks Sean).
- Fix WARN_ON and other cleanup for the arm64 locking changes (thanks Oliver).

Since v3[3]:
- Vastly simplified the series (thanks David). Removed mmu notifier batching logic entirely.
- Cleaned up how locking is done for mmu_notifier_test/clear_young (thanks David).
- Look-around is now only done when there are no secondary MMUs subscribed to MMU notifiers.
- CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added.
- Fixed the lockless implementation of kvm_{test,}age_gfn for x86 (thanks David).
- Added MGLRU functional and performance tests to access_tracking_perf_test (thanks Axel).
- In v3, an mm would be completely ignored (for aging) if there was a secondary MMU but support for secondary MMU walking was missing. Now, missing secondary MMU walking support simply skips the notifier calls (except for eviction).
- Added a sanity check for that range->lockless and range->on_lock are never both provided for the memslot walk.

For the changes since v2[4], see v3.

Based on latest kvm/next.

[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/linux-mm/[email protected]/
[4]: https://lore.kernel.org/kvmarm/[email protected]/

James Houghton (11):
KVM: Add lockless memslot walk to KVM
KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER
mm: Add fast_only bool to test_young and clear_young MMU notifiers
mm: Add has_fast_aging to struct mmu_notifier
KVM: Pass fast_only to kvm_{test_,}age_gfn
KVM: x86: Optimize kvm_{test_,}age_gfn a little bit
KVM: x86: Implement fast_only versions of kvm_{test_,}age_gfn
mm: multi-gen LRU: Have secondary MMUs participate in aging
KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test

 Documentation/admin-guide/mm/multigen_lru.rst |   6 +-
 arch/arm64/kvm/Kconfig                        |   1 +
 arch/arm64/kvm/hyp/pgtable.c                  |  15 +-
 arch/arm64/kvm/mmu.c                          |  30 +-
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/kvm/Kconfig                          |   2 +
 arch/x86/kvm/mmu/mmu.c                        |  23 +-
 arch/x86/kvm/mmu/tdp_iter.h                   |  27 +-
 arch/x86/kvm/mmu/tdp_mmu.c                    |  67 ++-
 include/linux/kvm_host.h                      |   2 +
 include/linux/mmu_notifier.h                  |  67 ++-
 include/linux/mmzone.h                        |   6 +-
 include/trace/events/kvm.h                    |  19 +-
 mm/damon/vaddr.c                              |   2 -
 mm/mmu_notifier.c                             |  38 +-
 mm/rmap.c                                     |   9 +-
 mm/vmscan.c                                   | 148 +++++--
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/access_tracking_perf_test.c | 369 +++++++++++++++--
 .../selftests/kvm/include/lru_gen_util.h      |  55 +++
 .../testing/selftests/kvm/lib/lru_gen_util.c  | 391 ++++++++++++++++++
 virt/kvm/Kconfig                              |   7 +
 virt/kvm/kvm_main.c                           |  73 ++--
 23 files changed, 1194 insertions(+), 165 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h
 create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c

base-commit: 332d2c1d713e232e163386c35a3ba0c1b90df83f

Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#1 lore raw

PATCH v6 1/11 KVM: Add lockless memslot walk to KVM

Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing.

Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS.

The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply.

---
 include/linux/kvm_host.h |  1 +
 virt/kvm/Kconfig         |  3 +++
 virt/kvm/kvm_main.c      | 26 +++++++++++++++++++-------
 3 files changed, 23 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +1 -0
index 689e8be873a7..8cd80f969cff 100644
@@ -266,6 +266,7 @@ struct kvm_gfn_range {
 	gfn_t end;
 	union kvm_mmu_notifier_arg arg;
 	bool may_block;
+	bool lockless;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
virt/kvm/Kconfig +3 -0
index b14e14cdbfb9..632334861001 100644
@@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER
        select MMU_NOTIFIER
        bool
 
+config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
+       bool
+
 config KVM_GENERIC_MEMORY_ATTRIBUTES
        depends on KVM_GENERIC_MMU_NOTIFIER
        bool
virt/kvm/kvm_main.c +19 -7
index d0788d0a72cc..33f8997a5c29 100644
@@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range {
 	on_lock_fn_t on_lock;
 	bool flush_on_ret;
 	bool may_block;
+	bool lockless;
 };
 
 /*
@@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
 			 IS_KVM_NULL_FN(range->handler)))
 		return r;
 
+	/* on_lock will never be called for lockless walks */
+	if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock)))
+		return r;
+
 	idx = srcu_read_lock(&kvm->srcu);
 
 	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
@@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
 			gfn_range.start = hva_to_gfn_memslot(hva_start, slot);
 			gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot);
 			gfn_range.slot = slot;
+			gfn_range.lockless = range->lockless;
 
 			if (!r.found_memslot) {
 				r.found_memslot = true;
-				KVM_MMU_LOCK(kvm);
-				if (!IS_KVM_NULL_FN(range->on_lock))
-					range->on_lock(kvm);
-
-				if (IS_KVM_NULL_FN(range->handler))
-					goto mmu_unlock;
+				if (!range->lockless) {
+					KVM_MMU_LOCK(kvm);
+					if (!IS_KVM_NULL_FN(range->on_lock))
+						range->on_lock(kvm);
+
+					if (IS_KVM_NULL_FN(range->handler))
+						goto mmu_unlock;
+				}
 			}
 			r.ret |= range->handler(kvm, &gfn_range);
 		}
@@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
 		kvm_flush_remote_tlbs(kvm);
 
 mmu_unlock:
-	if (r.found_memslot)
+	if (r.found_memslot && !range->lockless)
 		KVM_MMU_UNLOCK(kvm);
 
 	srcu_read_unlock(&kvm->srcu, idx);
@@ -679,6 +687,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,
 		.on_lock	= (void *)kvm_null_fn,
 		.flush_on_ret	= true,
 		.may_block	= false,
+		.lockless	=
+			IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
 	};
 
 	return __kvm_handle_hva_range(kvm, &range).ret;
@@ -697,6 +707,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn
 		.on_lock	= (void *)kvm_null_fn,
 		.flush_on_ret	= false,
 		.may_block	= false,
+		.lockless	=
+			IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
 	};
 
 	return __kvm_handle_hva_range(kvm, &range).ret;
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Reviewed-by on v6 1/11
#2 lore raw

Re: [PATCH v6 01/11] KVM: Add lockless memslot walk to KVM

Quoted 11 lines · James Houghton

Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing.

Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS.

The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply.

Aside from the cleanup suggestion (which should be in separate patches anyway):

Quoted 99 lines
---
 include/linux/kvm_host.h |  1 +
 virt/kvm/Kconfig         |  3 +++
 virt/kvm/kvm_main.c      | 26 +++++++++++++++++++-------
 3 files changed, 23 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +1 -0
index 689e8be873a7..8cd80f969cff 100644
@@ -266,6 +266,7 @@ struct kvm_gfn_range {
 	gfn_t end;
 	union kvm_mmu_notifier_arg arg;
 	bool may_block;
+	bool lockless;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
virt/kvm/Kconfig +3 -0
index b14e14cdbfb9..632334861001 100644
@@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER
        select MMU_NOTIFIER
        bool
 
+config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
+       bool
+
 config KVM_GENERIC_MEMORY_ATTRIBUTES
        depends on KVM_GENERIC_MMU_NOTIFIER
        bool
virt/kvm/kvm_main.c +17 -7
index d0788d0a72cc..33f8997a5c29 100644
@@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range {
 	on_lock_fn_t on_lock;
 	bool flush_on_ret;
 	bool may_block;
+	bool lockless;
 };
 
 /*
@@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
 			 IS_KVM_NULL_FN(range->handler)))
 		return r;
 
+	/* on_lock will never be called for lockless walks */
+	if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock)))
+		return r;
+
 	idx = srcu_read_lock(&kvm->srcu);
 
 	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
@@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
 			gfn_range.start = hva_to_gfn_memslot(hva_start, slot);
 			gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot);
 			gfn_range.slot = slot;
+			gfn_range.lockless = range->lockless;
 
 			if (!r.found_memslot) {
 				r.found_memslot = true;
-				KVM_MMU_LOCK(kvm);
-				if (!IS_KVM_NULL_FN(range->on_lock))
-					range->on_lock(kvm);
-
-				if (IS_KVM_NULL_FN(range->handler))
-					goto mmu_unlock;
+				if (!range->lockless) {
+					KVM_MMU_LOCK(kvm);
+					if (!IS_KVM_NULL_FN(range->on_lock))
+						range->on_lock(kvm);
+
+					if (IS_KVM_NULL_FN(range->handler))
+						goto mmu_unlock;
+				}
 			}
 			r.ret |= range->handler(kvm, &gfn_range);
 		}
@@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
 		kvm_flush_remote_tlbs(kvm);
 
 mmu_unlock:
-	if (r.found_memslot)
+	if (r.found_memslot && !range->lockless)
 		KVM_MMU_UNLOCK(kvm);
 
 	srcu_read_unlock(&kvm->srcu, idx);
@@ -679,6 +687,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,
 		.on_lock	= (void *)kvm_null_fn,
 		.flush_on_ret	= true,
 		.may_block	= false,
+		.lockless	=
+			IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
 	};
 
 	return __kvm_handle_hva_range(kvm, &range).ret;

virt/kvm/kvm_main.c static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn · line 711

Quoted 5 lines
virt/kvm/kvm_main.c +2 -0
@@ -697,6 +707,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn
 		.on_lock	= (void *)kvm_null_fn,
 		.flush_on_ret	= false,
 		.may_block	= false,
+		.lockless	=
+			IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),

kvm_handle_hva_range{,_no_flush}() have very generic names but they're intimately tied to the "young" notifiers. Whereas __kvm_handle_hva_range() is the truly generic handler function.

This is arguably a pre-existing issue, but adding CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS makes these functions even more intamtely tied to the "young" notifiers.

We could rename kvm_handle_hva_range{,_no_flush}() but I think the cleanest thing to do might be to just drop them entirely and move their contents into their callers (there are only 2 callers of these 3 functions). That will create a little duplication but IMO will make the code easier to read.

And then we can also rename __kvm_handle_hva_range() to kvm_handle_hva_range().

e.g. Something like this as the end result:

virt/kvm/kvm_main.c +40 -48
index 86fb2b560d98..0146c83e24bd 100644
@@ -590,8 +590,8 @@ static void kvm_null_fn(void)
 	     node;							     \
 	     node = interval_tree_iter_next(node, start, last))	     \
 
-static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
-							   const struct kvm_mmu_notifier_range *range)
+static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
+							 const struct kvm_mmu_notifier_range *range)
 {
 	struct kvm_mmu_notifier_return r = {
 		.ret = false,
@@ -674,48 +674,6 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
 	return r;
 }
 
-static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,
-						unsigned long start,
-						unsigned long end,
-						gfn_handler_t handler)
-{
-	struct kvm *kvm = mmu_notifier_to_kvm(mn);
-	const struct kvm_mmu_notifier_range range = {
-		.start		= start,
-		.end		= end,
-		.handler	= handler,
-		.on_lock	= (void *)kvm_null_fn,
-		.flush_on_ret	= true,
-		.may_block	= false,
-		.lockless	=
-			IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
-	};
-
-	return __kvm_handle_hva_range(kvm, &range).ret;
-}
-
-static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn,
-							 unsigned long start,
-							 unsigned long end,
-							 gfn_handler_t handler,
-							 bool fast_only)
-{
-	struct kvm *kvm = mmu_notifier_to_kvm(mn);
-	const struct kvm_mmu_notifier_range range = {
-		.start			= start,
-		.end			= end,
-		.handler		= handler,
-		.on_lock		= (void *)kvm_null_fn,
-		.flush_on_ret		= false,
-		.may_block		= false,
-		.lockless		=
-			IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
-		.arg.fast_only		= fast_only,
-	};
-
-	return __kvm_handle_hva_range(kvm, &range).ret;
-}
-
 void kvm_mmu_invalidate_begin(struct kvm *kvm)
 {
 	lockdep_assert_held_write(&kvm->mmu_lock);
@@ -808,7 +766,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 	 * that guest memory has been reclaimed.  This needs to be done *after*
 	 * dropping mmu_lock, as x86's reclaim path is slooooow.
 	 */
-	if (__kvm_handle_hva_range(kvm, &hva_range).found_memslot)
+	if (kvm_handle_hva_range(kvm, &hva_range).found_memslot)
 		kvm_arch_guest_memory_reclaimed(kvm);
 
 	return 0;
@@ -854,7 +812,7 @@ static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
 	};
 	bool wake;
 
-	__kvm_handle_hva_range(kvm, &hva_range);
+	kvm_handle_hva_range(kvm, &hva_range);
 
 	/* Pairs with the increment in range_start(). */
 	spin_lock(&kvm->mn_invalidate_lock);
@@ -876,6 +834,17 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,
 					      unsigned long start,
 					      unsigned long end)
 {
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
+	const struct kvm_mmu_notifier_range range = {
+		.start		= start,
+		.end		= end,
+		.handler	= kvm_age_gfn,
+		.on_lock	= (void *)kvm_null_fn,
+		.flush_on_ret	= true,
+		.may_block	= false,
+		.lockless	= IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
+	};
+
 	trace_kvm_age_hva(start, end, false);
 
 	return kvm_handle_hva_range(mn, start, end, kvm_age_gfn);
@@ -887,6 +856,18 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
 					unsigned long end,
 					bool fast_only)
 {
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
+	const struct kvm_mmu_notifier_range range = {
+		.start		= start,
+		.end		= end,
+		.handler	= kvm_age_gfn,
+		.on_lock	= (void *)kvm_null_fn,
+		.flush_on_ret	= false,
+		.may_block	= false,
+		.lockless	= IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
+		.arg.fast_only	= fast_only,
+	};
+
 	trace_kvm_age_hva(start, end, fast_only);
 
 	/*
@@ -902,8 +883,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
 	 * cadence. If we find this inaccurate, we might come up with a
 	 * more sophisticated heuristic later.
 	 */
-	return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn,
-					     fast_only);
+	return kvm_handle_hva_range(kvm, &range).ret;
 }
 
 static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
@@ -911,6 +891,18 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
 				       unsigned long address,
 				       bool fast_only)
 {
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
+	const struct kvm_mmu_notifier_range range = {
+		.start		= address,
+		.end		= address + 1,
+		.handler	= kvm_test_age_gfn,
+		.on_lock	= (void *)kvm_null_fn,
+		.flush_on_ret	= false,
+		.may_block	= false,
+		.lockless	= IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
+		.arg.fast_only	= fast_only,
+	};
+
 	trace_kvm_test_age_hva(address, fast_only);
 
 	return kvm_handle_hva_range_no_flush(mn, address, address + 1,
Quoted 6 lines
virt/kvm/kvm_main.c +0 -0
 	};
 
 	return __kvm_handle_hva_range(kvm, &range).ret;
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#3 lore raw

Re: [PATCH v6 01/11] KVM: Add lockless memslot walk to KVM

Quoted 11 lines · David Matlack
Quoted 9 lines · James Houghton

Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing.

Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS.

The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply.

Quoted 7 lines
Quoted 2 lines

Aside from the cleanup suggestion (which should be in separate patches anyway):

Thanks David!

Quoted 114 lines
Quoted 100 lines
---
 include/linux/kvm_host.h |  1 +
 virt/kvm/Kconfig         |  3 +++
 virt/kvm/kvm_main.c      | 26 +++++++++++++++++++-------
 3 files changed, 23 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +1 -0
index 689e8be873a7..8cd80f969cff 100644
@@ -266,6 +266,7 @@ struct kvm_gfn_range {
      gfn_t end;
      union kvm_mmu_notifier_arg arg;
      bool may_block;
+     bool lockless;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
virt/kvm/Kconfig +3 -0
index b14e14cdbfb9..632334861001 100644
@@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER
        select MMU_NOTIFIER
        bool
 
+config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
+       bool
+
 config KVM_GENERIC_MEMORY_ATTRIBUTES
        depends on KVM_GENERIC_MMU_NOTIFIER
        bool
virt/kvm/kvm_main.c +17 -7
index d0788d0a72cc..33f8997a5c29 100644
@@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range {
      on_lock_fn_t on_lock;
      bool flush_on_ret;
      bool may_block;
+     bool lockless;
 };
 
 /*
@@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
                       IS_KVM_NULL_FN(range->handler)))
              return r;
 
+     /* on_lock will never be called for lockless walks */
+     if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock)))
+             return r;
+
      idx = srcu_read_lock(&kvm->srcu);
 
      for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
@@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
                      gfn_range.start = hva_to_gfn_memslot(hva_start, slot);
                      gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot);
                      gfn_range.slot = slot;
+                     gfn_range.lockless = range->lockless;
 
                      if (!r.found_memslot) {
                              r.found_memslot = true;
-                             KVM_MMU_LOCK(kvm);
-                             if (!IS_KVM_NULL_FN(range->on_lock))
-                                     range->on_lock(kvm);
-
-                             if (IS_KVM_NULL_FN(range->handler))
-                                     goto mmu_unlock;
+                             if (!range->lockless) {
+                                     KVM_MMU_LOCK(kvm);
+                                     if (!IS_KVM_NULL_FN(range->on_lock))
+                                             range->on_lock(kvm);
+
+                                     if (IS_KVM_NULL_FN(range->handler))
+                                             goto mmu_unlock;
+                             }
                      }
                      r.ret |= range->handler(kvm, &gfn_range);
              }
@@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,
              kvm_flush_remote_tlbs(kvm);
 
 mmu_unlock:
-     if (r.found_memslot)
+     if (r.found_memslot && !range->lockless)
              KVM_MMU_UNLOCK(kvm);
 
      srcu_read_unlock(&kvm->srcu, idx);
@@ -679,6 +687,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,
              .on_lock        = (void *)kvm_null_fn,
              .flush_on_ret   = true,
              .may_block      = false,
+             .lockless       =
+                     IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
      };
 
      return __kvm_handle_hva_range(kvm, &range).ret;

virt/kvm/kvm_main.c static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn · line 711

Quoted 5 lines
virt/kvm/kvm_main.c +2 -0
@@ -697,6 +707,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn
              .on_lock        = (void *)kvm_null_fn,
              .flush_on_ret   = false,
              .may_block      = false,
+             .lockless       =
+                     IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),

kvm_handle_hva_range{,_no_flush}() have very generic names but they're intimately tied to the "young" notifiers. Whereas __kvm_handle_hva_range() is the truly generic handler function.

This is arguably a pre-existing issue, but adding CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS makes these functions even more intamtely tied to the "young" notifiers.

Quoted 8 lines

We could rename kvm_handle_hva_range{,_no_flush}() but I think the cleanest thing to do might be to just drop them entirely and move their contents into their callers (there are only 2 callers of these 3 functions). That will create a little duplication but IMO will make the code easier to read.

And then we can also rename __kvm_handle_hva_range() to kvm_handle_hva_range().

Thanks for the suggestion, I think this is a good idea. I'm curious how others feel, as this indeed does duplicate the code some. Perhaps it is better just to rename kvm_handle_hva_range() to kvm_handle_hva_range_age() or something like that, and something similar for _no_flush(). :/

But yeah I think it's fine to just do the manipulation you're suggesting. I'll include it in v7 unless others say not to.

#4 lore raw

PATCH v6 2/11 KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

In reply to James Houghton

Walk the TDP MMU in an RCU read-side critical section. This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the R/X bits).

If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.

Harvesting age information from the shadow MMU is still done while holding the MMU write lock.

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/Kconfig            |  1 +
 arch/x86/kvm/mmu/mmu.c          | 10 ++++-
 arch/x86/kvm/mmu/tdp_iter.h     | 27 +++++++------
 arch/x86/kvm/mmu/tdp_mmu.c      | 67 +++++++++++++++++++++++++--------
 5 files changed, 77 insertions(+), 29 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 950a03e0181e..096988262005 100644
@@ -1456,6 +1456,7 @@ struct kvm_arch {
 	 * tdp_mmu_page set.
 	 *
 	 * For reads, this list is protected by:
+	 *	RCU alone or
 	 *	the MMU lock in read mode + RCU or
 	 *	the MMU lock in write mode
 	 *
arch/x86/kvm/Kconfig +1 -0
index 4287a8071a3a..6ac43074c5e9 100644
@@ -23,6 +23,7 @@ config KVM
 	depends on X86_LOCAL_APIC
 	select KVM_COMMON
 	select KVM_GENERIC_MMU_NOTIFIER
+	select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_PFNCACHE
 	select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +8 -2
index 901be9e420a4..7b93ce8f0680 100644
@@ -1633,8 +1633,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
-	if (kvm_memslots_have_rmaps(kvm))
+	if (kvm_memslots_have_rmaps(kvm)) {
+		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
+		write_unlock(&kvm->mmu_lock);
+	}
 
 	if (tdp_mmu_enabled)
 		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
@@ -1646,8 +1649,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
-	if (kvm_memslots_have_rmaps(kvm))
+	if (kvm_memslots_have_rmaps(kvm)) {
+		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
+		write_unlock(&kvm->mmu_lock);
+	}
 
 	if (tdp_mmu_enabled)
 		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
arch/x86/kvm/mmu/tdp_iter.h +15 -12
index 2880fd392e0c..510936a8455a 100644
@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)
 	return xchg(rcu_dereference(sptep), new_spte);
 }
 
+static inline u64 tdp_mmu_clear_spte_bits_atomic(tdp_ptep_t sptep, u64 mask)
+{
+	atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
+
+	return (u64)atomic64_fetch_and(~mask, sptep_atomic);
+}
+
 static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 {
 	KVM_MMU_WARN_ON(is_ept_ve_possible(new_spte));
@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 }
 
 /*
- * SPTEs must be modified atomically if they are shadow-present, leaf
- * SPTEs, and have volatile bits, i.e. has bits that can be set outside
- * of mmu_lock.  The Writable bit can be set by KVM's fast page fault
- * handler, and Accessed and Dirty bits can be set by the CPU.
+ * SPTEs must be modified atomically if they have bits that can be set outside
+ * of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the
+ * Writable bit can be set by KVM's fast page fault handler, the Accessed and
+ * Dirty bits can be set by the CPU, and the Accessed and R/X bits can be
+ * cleared by age_gfn_range.
  *
  * Note, non-leaf SPTEs do have Accessed bits and those bits are
  * technically volatile, but KVM doesn't consume the Accessed bit of
@@ -46,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 static inline bool kvm_tdp_mmu_spte_need_atomic_write(u64 old_spte, int level)
 {
 	return is_shadow_present_pte(old_spte) &&
-	       is_last_spte(old_spte, level) &&
-	       spte_has_volatile_bits(old_spte);
+	       is_last_spte(old_spte, level);
 }
 
 static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
@@ -63,12 +70,8 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
 static inline u64 tdp_mmu_clear_spte_bits(tdp_ptep_t sptep, u64 old_spte,
 					  u64 mask, int level)
 {
-	atomic64_t *sptep_atomic;
-
-	if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) {
-		sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
-		return (u64)atomic64_fetch_and(~mask, sptep_atomic);
-	}
+	if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level))
+		return tdp_mmu_clear_spte_bits_atomic(sptep, mask);
 
 	__kvm_tdp_mmu_write_spte(sptep, old_spte & ~mask);
 	return old_spte;
arch/x86/kvm/mmu/tdp_mmu.c +52 -15

112 lines not on this page · raw message

Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Reviewed-by on v6 2/11
#5 lore raw

Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 11 lines · James Houghton

Walk the TDP MMU in an RCU read-side critical section. This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the R/X bits).

If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.

Quoted 5 lines

Harvesting age information from the shadow MMU is still done while holding the MMU write lock.

Aside from the comment fixes below,

Quoted 8 lines
---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/Kconfig            |  1 +
 arch/x86/kvm/mmu/mmu.c          | 10 ++++-
 arch/x86/kvm/mmu/tdp_iter.h     | 27 +++++++------
 arch/x86/kvm/mmu/tdp_mmu.c      | 67 +++++++++++++++++++++++++--------
 5 files changed, 77 insertions(+), 29 deletions(-)

[...]

Quoted 18 lines
arch/x86/kvm/mmu/tdp_iter.h +7 -0
@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)
 	return xchg(rcu_dereference(sptep), new_spte);
 }
 
+static inline u64 tdp_mmu_clear_spte_bits_atomic(tdp_ptep_t sptep, u64 mask)
+{
+	atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
+
+	return (u64)atomic64_fetch_and(~mask, sptep_atomic);
+}
+
 static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 {
 	KVM_MMU_WARN_ON(is_ept_ve_possible(new_spte));
@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 }

arch/x86/kvm/mmu/tdp_iter.h static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) · line 45

Quoted 10 lines
arch/x86/kvm/mmu/tdp_iter.h +4 -4
 
 /*
- * SPTEs must be modified atomically if they are shadow-present, leaf
- * SPTEs, and have volatile bits, i.e. has bits that can be set outside
- * of mmu_lock.  The Writable bit can be set by KVM's fast page fault
- * handler, and Accessed and Dirty bits can be set by the CPU.
+ * SPTEs must be modified atomically if they have bits that can be set outside
+ * of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the
+ * Writable bit can be set by KVM's fast page fault handler, the Accessed and
+ * Dirty bits can be set by the CPU, and the Accessed and R/X bits can be

"R/X bits" should be "W/R/X bits".

arch/x86/kvm/mmu/tdp_iter.h static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) · line 46

Quoted 1 line
arch/x86/kvm/mmu/tdp_iter.h +1 -0
+ * cleared by age_gfn_range.

nit: "age_gfn_range()"

#6 lore raw

Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 16 lines · David Matlack
Quoted 14 lines · James Houghton

Walk the TDP MMU in an RCU read-side critical section. This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the R/X bits).

If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.

Harvesting age information from the shadow MMU is still done while holding the MMU write lock.

Quoted 6 lines
Quoted 2 lines

Aside from the comment fixes below,

Thank you!

Quoted 29 lines
Quoted 9 lines
---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/Kconfig            |  1 +
 arch/x86/kvm/mmu/mmu.c          | 10 ++++-
 arch/x86/kvm/mmu/tdp_iter.h     | 27 +++++++------
 arch/x86/kvm/mmu/tdp_mmu.c      | 67 +++++++++++++++++++++++++--------
 5 files changed, 77 insertions(+), 29 deletions(-)

[...]

Quoted 19 lines
arch/x86/kvm/mmu/tdp_iter.h +7 -0
@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)
      return xchg(rcu_dereference(sptep), new_spte);
 }
 
+static inline u64 tdp_mmu_clear_spte_bits_atomic(tdp_ptep_t sptep, u64 mask)
+{
+     atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
+
+     return (u64)atomic64_fetch_and(~mask, sptep_atomic);
+}
+
 static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 {
      KVM_MMU_WARN_ON(is_ept_ve_possible(new_spte));
@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 }
 
Quoted 11 lines

arch/x86/kvm/mmu/tdp_iter.h static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) · line 45

Quoted 9 lines
arch/x86/kvm/mmu/tdp_iter.h +4 -4
 /*
- * SPTEs must be modified atomically if they are shadow-present, leaf
- * SPTEs, and have volatile bits, i.e. has bits that can be set outside
- * of mmu_lock.  The Writable bit can be set by KVM's fast page fault
- * handler, and Accessed and Dirty bits can be set by the CPU.
+ * SPTEs must be modified atomically if they have bits that can be set outside
+ * of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the
+ * Writable bit can be set by KVM's fast page fault handler, the Accessed and
+ * Dirty bits can be set by the CPU, and the Accessed and R/X bits can be

"R/X bits" should be "W/R/X bits".

Thanks. Right, we are clearing all of VMX_EPT_RWX_MASK.

Quoted 4 lines

arch/x86/kvm/mmu/tdp_iter.h static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) · line 46

Quoted 2 lines
arch/x86/kvm/mmu/tdp_iter.h +1 -0
+ * cleared by age_gfn_range.

nit: "age_gfn_range()"

Thanks, will fix this and all the other places where I've left off the ().

#7 lore raw

Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

In reply to James Houghton
Quoted 1 line · James Houghton

Walk the TDP MMU in an RCU read-side critical section.

...without holding mmu_lock, while doing xxx. There are a lot of TDP MMU walks, pand they all need RCU protection.

Quoted 8 lines

This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the R/X bits).

If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.

Please avoid pronouns. E.g. s/we/KVM (and adjust grammar as needed), so that it's clear what actor in particular is doing the retry.

Quoted 131 lines

Harvesting age information from the shadow MMU is still done while holding the MMU write lock.

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/Kconfig            |  1 +
 arch/x86/kvm/mmu/mmu.c          | 10 ++++-
 arch/x86/kvm/mmu/tdp_iter.h     | 27 +++++++------
 arch/x86/kvm/mmu/tdp_mmu.c      | 67 +++++++++++++++++++++++++--------
 5 files changed, 77 insertions(+), 29 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 950a03e0181e..096988262005 100644
@@ -1456,6 +1456,7 @@ struct kvm_arch {
 	 * tdp_mmu_page set.
 	 *
 	 * For reads, this list is protected by:
+	 *	RCU alone or
 	 *	the MMU lock in read mode + RCU or
 	 *	the MMU lock in write mode
 	 *
arch/x86/kvm/Kconfig +1 -0
index 4287a8071a3a..6ac43074c5e9 100644
@@ -23,6 +23,7 @@ config KVM
 	depends on X86_LOCAL_APIC
 	select KVM_COMMON
 	select KVM_GENERIC_MMU_NOTIFIER
+	select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_PFNCACHE
 	select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +8 -2
index 901be9e420a4..7b93ce8f0680 100644
@@ -1633,8 +1633,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
-	if (kvm_memslots_have_rmaps(kvm))
+	if (kvm_memslots_have_rmaps(kvm)) {
+		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
+		write_unlock(&kvm->mmu_lock);
+	}
 
 	if (tdp_mmu_enabled)
 		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
@@ -1646,8 +1649,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
-	if (kvm_memslots_have_rmaps(kvm))
+	if (kvm_memslots_have_rmaps(kvm)) {
+		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
+		write_unlock(&kvm->mmu_lock);
+	}
 
 	if (tdp_mmu_enabled)
 		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
arch/x86/kvm/mmu/tdp_iter.h +15 -12
index 2880fd392e0c..510936a8455a 100644
@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)
 	return xchg(rcu_dereference(sptep), new_spte);
 }
 
+static inline u64 tdp_mmu_clear_spte_bits_atomic(tdp_ptep_t sptep, u64 mask)
+{
+	atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
+
+	return (u64)atomic64_fetch_and(~mask, sptep_atomic);
+}
+
 static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 {
 	KVM_MMU_WARN_ON(is_ept_ve_possible(new_spte));
@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 }
 
 /*
- * SPTEs must be modified atomically if they are shadow-present, leaf
- * SPTEs, and have volatile bits, i.e. has bits that can be set outside
- * of mmu_lock.  The Writable bit can be set by KVM's fast page fault
- * handler, and Accessed and Dirty bits can be set by the CPU.
+ * SPTEs must be modified atomically if they have bits that can be set outside
+ * of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the
+ * Writable bit can be set by KVM's fast page fault handler, the Accessed and
+ * Dirty bits can be set by the CPU, and the Accessed and R/X bits can be
+ * cleared by age_gfn_range.
  *
  * Note, non-leaf SPTEs do have Accessed bits and those bits are
  * technically volatile, but KVM doesn't consume the Accessed bit of
@@ -46,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 static inline bool kvm_tdp_mmu_spte_need_atomic_write(u64 old_spte, int level)
 {
 	return is_shadow_present_pte(old_spte) &&
-	       is_last_spte(old_spte, level) &&
-	       spte_has_volatile_bits(old_spte);
+	       is_last_spte(old_spte, level);
 }
 
 static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
@@ -63,12 +70,8 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
 static inline u64 tdp_mmu_clear_spte_bits(tdp_ptep_t sptep, u64 old_spte,
 					  u64 mask, int level)
 {
-	atomic64_t *sptep_atomic;
-
-	if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) {
-		sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
-		return (u64)atomic64_fetch_and(~mask, sptep_atomic);
-	}
+	if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level))
+		return tdp_mmu_clear_spte_bits_atomic(sptep, mask);
 
 	__kvm_tdp_mmu_write_spte(sptep, old_spte & ~mask);
 	return old_spte;
arch/x86/kvm/mmu/tdp_mmu.c +0 -0
index c7dc49ee7388..3f13b2db53de 100644

arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm, · line 36

Quoted 8 lines
arch/x86/kvm/mmu/tdp_mmu.c +5 -0
@@ -29,6 +29,11 @@ static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,
 
 	return true;
 }
+static __always_inline bool kvm_lockdep_assert_rcu_read_lock_held(void)
+{
+	WARN_ON_ONCE(!rcu_read_lock_held());
+	return true;
+}

I doubt KVM needs a manual WARN, the RCU deference stuff should yell loudly if something is missing an rcu_read_lock().

arch/x86/kvm/mmu/tdp_mmu.c static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, · line 190

Quoted 11 lines
arch/x86/kvm/mmu/tdp_mmu.c +5 -0
 void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
 {
@@ -178,6 +183,15 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
 		     ((_only_valid) && (_root)->role.invalid))) {		\
 		} else
 
+/*
+ * Iterate over all TDP MMU roots in an RCU read-side critical section.
+ */
+#define for_each_tdp_mmu_root_rcu(_kvm, _root, _as_id)				\
+	list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link)		\

This should just process valid roots:

https://lore.kernel.org/all/[email protected]

Quoted 8 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+		if (kvm_lockdep_assert_rcu_read_lock_held() &&			\
+		    (_as_id >= 0 && kvm_mmu_page_as_id(_root) != _as_id)) {	\
+		} else
+
 #define for_each_tdp_mmu_root(_kvm, _root, _as_id)			\
 	__for_each_tdp_mmu_root(_kvm, _root, _as_id, false)
 

arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1244

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
@@ -1224,6 +1238,27 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm,
 	return ret;
 }
 
+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless(
+		struct kvm *kvm,
+		struct kvm_gfn_range *range,
+		tdp_handler_t handler)

Please burn all the Google3 from your brain, and code ;-)

Quoted 40 lines
arch/x86/kvm/mmu/tdp_mmu.c +19 -7
+	struct kvm_mmu_page *root;
+	struct tdp_iter iter;
+	bool ret = false;
+
+	rcu_read_lock();
+
+	for_each_tdp_mmu_root_rcu(kvm, root, range->slot->as_id) {
+		tdp_root_for_each_leaf_pte(iter, root, range->start, range->end)
+			ret |= handler(kvm, &iter, range);
+	}
+
+	rcu_read_unlock();
+
+	return ret;
+}
+
 /*
  * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero
  * if any of the GFNs in the range have been accessed.
@@ -1237,28 +1272,30 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter,
 {
 	u64 new_spte;
 
+retry:
 	/* If we have a non-accessed entry we don't need to change the pte. */
 	if (!is_accessed_spte(iter->old_spte))
 		return false;
 
 	if (spte_ad_enabled(iter->old_spte)) {
-		iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep,
-							 iter->old_spte,
-							 shadow_accessed_mask,
-							 iter->level);
+		iter->old_spte = tdp_mmu_clear_spte_bits_atomic(iter->sptep,
+						shadow_accessed_mask);
 		new_spte = iter->old_spte & ~shadow_accessed_mask;
 	} else {
-		/*
-		 * Capture the dirty status of the page, so that it doesn't get
-		 * lost when the SPTE is marked for access tracking.

arch/x86/kvm/mmu/tdp_mmu.c static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, · line 1293

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +9 -1
-		 */
+		new_spte = mark_spte_for_access_track(iter->old_spte);
+		if (__tdp_mmu_set_spte_atomic(iter, new_spte)) {
+			/*
+			 * The cmpxchg failed. If the spte is still a
+			 * last-level spte, we can safely retry.
+			 */
+			if (is_shadow_present_pte(iter->old_spte) &&
+			    is_last_spte(iter->old_spte, iter->level))
+				goto retry;

Do we have a feel for how often conflicts actually happen? I.e. is it worth retrying and having to worry about infinite loops, however improbable they may be?

#8 lore raw

Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 6 lines · Sean Christopherson
Quoted 1 line · James Houghton

Walk the TDP MMU in an RCU read-side critical section.

...without holding mmu_lock, while doing xxx. There are a lot of TDP MMU walks, pand they all need RCU protection.

Added "without holding mmu_lock when harvesting and potentially updating age information on sptes".

Quoted 11 lines
Quoted 8 lines

This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the R/X bits).

If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.

Please avoid pronouns. E.g. s/we/KVM (and adjust grammar as needed), so that it's clear what actor in particular is doing the retry.

Fixed. Though, I have also changed this to reflect the change in the retry logic I've made, given your other comment.

Quoted 132 lines
Quoted 132 lines

Harvesting age information from the shadow MMU is still done while holding the MMU write lock.

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/Kconfig            |  1 +
 arch/x86/kvm/mmu/mmu.c          | 10 ++++-
 arch/x86/kvm/mmu/tdp_iter.h     | 27 +++++++------
 arch/x86/kvm/mmu/tdp_mmu.c      | 67 +++++++++++++++++++++++++--------
 5 files changed, 77 insertions(+), 29 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 950a03e0181e..096988262005 100644
@@ -1456,6 +1456,7 @@ struct kvm_arch {
       * tdp_mmu_page set.
       *
       * For reads, this list is protected by:
+      *      RCU alone or
       *      the MMU lock in read mode + RCU or
       *      the MMU lock in write mode
       *
arch/x86/kvm/Kconfig +1 -0
index 4287a8071a3a..6ac43074c5e9 100644
@@ -23,6 +23,7 @@ config KVM
      depends on X86_LOCAL_APIC
      select KVM_COMMON
      select KVM_GENERIC_MMU_NOTIFIER
+     select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
      select HAVE_KVM_IRQCHIP
      select HAVE_KVM_PFNCACHE
      select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +8 -2
index 901be9e420a4..7b93ce8f0680 100644
@@ -1633,8 +1633,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
      bool young = false;
 
-     if (kvm_memslots_have_rmaps(kvm))
+     if (kvm_memslots_have_rmaps(kvm)) {
+             write_lock(&kvm->mmu_lock);
              young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
+             write_unlock(&kvm->mmu_lock);
+     }
 
      if (tdp_mmu_enabled)
              young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
@@ -1646,8 +1649,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
      bool young = false;
 
-     if (kvm_memslots_have_rmaps(kvm))
+     if (kvm_memslots_have_rmaps(kvm)) {
+             write_lock(&kvm->mmu_lock);
              young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
+             write_unlock(&kvm->mmu_lock);
+     }
 
      if (tdp_mmu_enabled)
              young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
arch/x86/kvm/mmu/tdp_iter.h +15 -12
index 2880fd392e0c..510936a8455a 100644
@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)
      return xchg(rcu_dereference(sptep), new_spte);
 }
 
+static inline u64 tdp_mmu_clear_spte_bits_atomic(tdp_ptep_t sptep, u64 mask)
+{
+     atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
+
+     return (u64)atomic64_fetch_and(~mask, sptep_atomic);
+}
+
 static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 {
      KVM_MMU_WARN_ON(is_ept_ve_possible(new_spte));
@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 }
 
 /*
- * SPTEs must be modified atomically if they are shadow-present, leaf
- * SPTEs, and have volatile bits, i.e. has bits that can be set outside
- * of mmu_lock.  The Writable bit can be set by KVM's fast page fault
- * handler, and Accessed and Dirty bits can be set by the CPU.
+ * SPTEs must be modified atomically if they have bits that can be set outside
+ * of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the
+ * Writable bit can be set by KVM's fast page fault handler, the Accessed and
+ * Dirty bits can be set by the CPU, and the Accessed and R/X bits can be
+ * cleared by age_gfn_range.
  *
  * Note, non-leaf SPTEs do have Accessed bits and those bits are
  * technically volatile, but KVM doesn't consume the Accessed bit of
@@ -46,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)
 static inline bool kvm_tdp_mmu_spte_need_atomic_write(u64 old_spte, int level)
 {
      return is_shadow_present_pte(old_spte) &&
-            is_last_spte(old_spte, level) &&
-            spte_has_volatile_bits(old_spte);
+            is_last_spte(old_spte, level);
 }
 
 static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
@@ -63,12 +70,8 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
 static inline u64 tdp_mmu_clear_spte_bits(tdp_ptep_t sptep, u64 old_spte,
                                        u64 mask, int level)
 {
-     atomic64_t *sptep_atomic;
-
-     if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) {
-             sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
-             return (u64)atomic64_fetch_and(~mask, sptep_atomic);
-     }
+     if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level))
+             return tdp_mmu_clear_spte_bits_atomic(sptep, mask);
 
      __kvm_tdp_mmu_write_spte(sptep, old_spte & ~mask);
      return old_spte;
arch/x86/kvm/mmu/tdp_mmu.c +0 -0
index c7dc49ee7388..3f13b2db53de 100644
@@ -29,6 +29,11 @@ static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,
 
Quoted 10 lines

arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm, · line 36

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +5 -0
      return true;
 }
+static __always_inline bool kvm_lockdep_assert_rcu_read_lock_held(void)
+{
+     WARN_ON_ONCE(!rcu_read_lock_held());
+     return true;
+}

I doubt KVM needs a manual WARN, the RCU deference stuff should yell loudly if something is missing an rcu_read_lock().

You're right -- removed.

Quoted 14 lines

arch/x86/kvm/mmu/tdp_mmu.c static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, · line 190

Quoted 11 lines
arch/x86/kvm/mmu/tdp_mmu.c +5 -0
 void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
 {
@@ -178,6 +183,15 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,
                   ((_only_valid) && (_root)->role.invalid))) {               \
              } else
 
+/*
+ * Iterate over all TDP MMU roots in an RCU read-side critical section.
+ */
+#define for_each_tdp_mmu_root_rcu(_kvm, _root, _as_id)                               \
+     list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link)         \

This should just process valid roots:

Quoted 1 line

https://lore.kernel.org/all/[email protected]

Thanks! I've added || (_root)->role.invalid) to the below conditional expression, and I've renamed the macro to for_each_valid_tdp_mmu_root_rcu.

Quoted 11 lines
Quoted 11 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+             if (kvm_lockdep_assert_rcu_read_lock_held() &&                  \
+                 (_as_id >= 0 && kvm_mmu_page_as_id(_root) != _as_id)) {     \
+             } else
+
 #define for_each_tdp_mmu_root(_kvm, _root, _as_id)                   \
      __for_each_tdp_mmu_root(_kvm, _root, _as_id, false)
 
@@ -1224,6 +1238,27 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm,
      return ret;
 }
 
Quoted 6 lines

arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1244

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless(
+             struct kvm *kvm,
+             struct kvm_gfn_range *range,
+             tdp_handler_t handler)

Please burn all the Google3 from your brain, and code ;-)

I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.

Perhaps I should put static __always_inline bool on its own line?

Quoted 47 lines
Quoted 47 lines
arch/x86/kvm/mmu/tdp_mmu.c +24 -8
+     struct kvm_mmu_page *root;
+     struct tdp_iter iter;
+     bool ret = false;
+
+     rcu_read_lock();
+
+     for_each_tdp_mmu_root_rcu(kvm, root, range->slot->as_id) {
+             tdp_root_for_each_leaf_pte(iter, root, range->start, range->end)
+                     ret |= handler(kvm, &iter, range);
+     }
+
+     rcu_read_unlock();
+
+     return ret;
+}
+
 /*
  * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero
  * if any of the GFNs in the range have been accessed.
@@ -1237,28 +1272,30 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter,
 {
      u64 new_spte;
 
+retry:
      /* If we have a non-accessed entry we don't need to change the pte. */
      if (!is_accessed_spte(iter->old_spte))
              return false;
 
      if (spte_ad_enabled(iter->old_spte)) {
-             iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep,
-                                                      iter->old_spte,
-                                                      shadow_accessed_mask,
-                                                      iter->level);
+             iter->old_spte = tdp_mmu_clear_spte_bits_atomic(iter->sptep,
+                                             shadow_accessed_mask);
              new_spte = iter->old_spte & ~shadow_accessed_mask;
      } else {
-             /*
-              * Capture the dirty status of the page, so that it doesn't get
-              * lost when the SPTE is marked for access tracking.
-              */
+             new_spte = mark_spte_for_access_track(iter->old_spte);
+             if (__tdp_mmu_set_spte_atomic(iter, new_spte)) {
+                     /*
+                      * The cmpxchg failed. If the spte is still a
+                      * last-level spte, we can safely retry.
Quoted 8 lines

arch/x86/kvm/mmu/tdp_mmu.c static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, · line 1293

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+                      */
+                     if (is_shadow_present_pte(iter->old_spte) &&
+                         is_last_spte(iter->old_spte, iter->level))
+                             goto retry;

Do we have a feel for how often conflicts actually happen? I.e. is it worth retrying and having to worry about infinite loops, however improbable they may be?

I'm not sure how common this is. I think it's probably better not to retry actually. If the cmpxchg fails, this spte is probably young anyway, so I can just return true instead of potentially retrying. This is all best-effort anyway.

#9 lore raw

Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 6 lines · James Houghton
Quoted 5 lines · Sean Christopherson
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless(
+             struct kvm *kvm,
+             struct kvm_gfn_range *range,
+             tdp_handler_t handler)
Quoted 6 lines
Quoted 1 line

Please burn all the Google3 from your brain, and code ;-)

I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.

Perhaps I should put static __always_inline bool on its own line?

Noooo. Do not wrap before the function name. Linus has a nice explanation/rant on this[1].

In this case, I'm pretty sure you can avoid the helper and simply handle all aging paths in a single API, e.g. similar to what I proposed for the shadow MMU[2].

[1] https://lore.kernel.org/all/CAHk-=wjoLAYG446ZNHfg=GhjSY6nFmuB_wA8fYd5iLBNXjo9Bw@mail.gmail.com
[2] https://lore.kernel.org/all/[email protected]

Quoted 39 lines
Quoted 30 lines
Quoted 30 lines

/*
* Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero
* if any of the GFNs in the range have been accessed.

arch/x86/kvm/mmu/tdp_mmu.c +8 -8
@@ -1237,28 +1272,30 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter,
 {
      u64 new_spte;
 
+retry:
      /* If we have a non-accessed entry we don't need to change the pte. */
      if (!is_accessed_spte(iter->old_spte))
              return false;
 
      if (spte_ad_enabled(iter->old_spte)) {
-             iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep,
-                                                      iter->old_spte,
-                                                      shadow_accessed_mask,
-                                                      iter->level);
+             iter->old_spte = tdp_mmu_clear_spte_bits_atomic(iter->sptep,
+                                             shadow_accessed_mask);
              new_spte = iter->old_spte & ~shadow_accessed_mask;
      } else {
-             /*
-              * Capture the dirty status of the page, so that it doesn't get
-              * lost when the SPTE is marked for access tracking.
-              */
+             new_spte = mark_spte_for_access_track(iter->old_spte);
+             if (__tdp_mmu_set_spte_atomic(iter, new_spte)) {
+                     /*
+                      * The cmpxchg failed. If the spte is still a
+                      * last-level spte, we can safely retry.
Quoted 8 lines

arch/x86/kvm/mmu/tdp_mmu.c static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, · line 1293

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+                      */
+                     if (is_shadow_present_pte(iter->old_spte) &&
+                         is_last_spte(iter->old_spte, iter->level))
+                             goto retry;

Do we have a feel for how often conflicts actually happen? I.e. is it worth retrying and having to worry about infinite loops, however improbable they may be?

Quoted 4 lines

I'm not sure how common this is. I think it's probably better not to retry actually. If the cmpxchg fails, this spte is probably young anyway, so I can just return true instead of potentially retrying. This is all best-effort anyway.

+1

#10 lore raw

Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 9 lines · Sean Christopherson
Quoted 8 lines · James Houghton
Quoted 6 lines · Sean Christopherson

arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1244

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless(
+             struct kvm *kvm,
+             struct kvm_gfn_range *range,
+             tdp_handler_t handler)

Please burn all the Google3 from your brain, and code ;-)

Quoted 7 lines
Quoted 4 lines

I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.

Perhaps I should put static __always_inline bool on its own line?

Noooo. Do not wrap before the function name. Linus has a nice explanation/rant on this[1].

IMHO, run clang-format on your stuff and just be happy with 99% of what it spits out. Saves *so much time* and usually arguing..

clang-format will occasionally decide to wrap in the GNU way, if it can put the arguments all on one line. People will never agree on small details of style, but it would be really nice if we can at least agree not to nitpick clang-format's decisions :) :)

Jason

#11 lore raw

Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Sean Christopherson
Quoted 11 lines · James Houghton
Quoted 6 lines · Sean Christopherson

arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1244

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless(
+             struct kvm *kvm,
+             struct kvm_gfn_range *range,
+             tdp_handler_t handler)

Please burn all the Google3 from your brain, and code ;-)

I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Perhaps I should put static __always_inline bool on its own line?

Noooo. Do not wrap before the function name. Linus has a nice explanation/rant on this[1].

IMHO, run clang-format on your stuff and just be happy with 99% of what it spits out. Saves *so much time* and usually arguing..

Heh, nope, not bending on this one. The time I spend far hunting for implementations because of wraps before the function name far exceeds the time it takes me to push back on these warts in review.

#12 lore raw

Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 19 lines · Sean Christopherson
Quoted 18 lines · Jason Gunthorpe
Quoted 16 lines · Sean Christopherson
Quoted 6 lines · James Houghton
Quoted 5 lines · Sean Christopherson
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless(
+             struct kvm *kvm,
+             struct kvm_gfn_range *range,
+             tdp_handler_t handler)
Quoted 6 lines
Quoted 1 line

Please burn all the Google3 from your brain, and code ;-)

I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.

Perhaps I should put static __always_inline bool on its own line?

Noooo. Do not wrap before the function name. Linus has a nice explanation/rant on this[1].

Quoted 6 lines
Quoted 2 lines

IMHO, run clang-format on your stuff and just be happy with 99% of what it spits out. Saves *so much time* and usually arguing..

Heh, nope, not bending on this one. The time I spend far hunting for implementations because of wraps before the function name far exceeds the time it takes me to push back on these warts in review.

clangd solved that problem for me :)

Jason

#13 lore raw

PATCH v6 3/11 KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

In reply to James Houghton

Replace the MMU write locks (taken in the memslot iteration loop) for read locks.

Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change.

---
 arch/arm64/kvm/Kconfig       |  1 +
 arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++------
 arch/arm64/kvm/mmu.c         | 30 ++++++++++++++++++++++--------
 3 files changed, 32 insertions(+), 14 deletions(-)
arch/arm64/kvm/Kconfig +1 -0
index 58f09370d17e..7a1af8141c0e 100644
@@ -22,6 +22,7 @@ menuconfig KVM
 	select KVM_COMMON
 	select KVM_GENERIC_HARDWARE_ENABLING
 	select KVM_GENERIC_MMU_NOTIFIER
+	select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
arch/arm64/kvm/hyp/pgtable.c +9 -6
index 9e2bbee77491..a24a2a857456 100644
@@ -1319,10 +1319,10 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	data->young = true;
 
 	/*
-	 * stage2_age_walker() is always called while holding the MMU lock for
-	 * write, so this will always succeed. Nonetheless, this deliberately
-	 * follows the race detection pattern of the other stage-2 walkers in
-	 * case the locking mechanics of the MMU notifiers is ever changed.
+	 * This walk is not exclusive; the PTE is permitted to change from
+	 * under us. If there is a race to update this PTE, then the GFN is
+	 * most likely young, so failing to clear the AF is likely to be
+	 * inconsequential.
 	 */
 	if (data->mkold && !stage2_try_set_pte(ctx, new))
 		return -EAGAIN;
@@ -1345,10 +1345,13 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr,
 	struct kvm_pgtable_walker walker = {
 		.cb		= stage2_age_walker,
 		.arg		= &data,
-		.flags		= KVM_PGTABLE_WALK_LEAF,
+		.flags		= KVM_PGTABLE_WALK_LEAF |
+				  KVM_PGTABLE_WALK_SHARED,
 	};
+	int r;
 
-	WARN_ON(kvm_pgtable_walk(pgt, addr, size, &walker));
+	r = kvm_pgtable_walk(pgt, addr, size, &walker);
+	WARN_ON_ONCE(r && r != -EAGAIN);
 	return data.young;
 }
 
arch/arm64/kvm/mmu.c +22 -8
index 6981b1bc0946..e37765f6f2a1 100644
@@ -1912,29 +1912,43 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	u64 size = (range->end - range->start) << PAGE_SHIFT;
+	bool young = false;
+
+	read_lock(&kvm->mmu_lock);
 
 	if (!kvm->arch.mmu.pgt)
-		return false;
+		goto out;
 
-	return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt,
-						   range->start << PAGE_SHIFT,
-						   size, true);
+	young = kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt,
+						    range->start << PAGE_SHIFT,
+						    size, true);
 	/*
 	 * TODO: Handle nested_mmu structures here using the reverse mapping in
 	 * a later version of patch series.
 	 */
+
+out:
+	read_unlock(&kvm->mmu_lock);
+	return young;
 }
 
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	u64 size = (range->end - range->start) << PAGE_SHIFT;
+	bool young = false;
+
+	read_lock(&kvm->mmu_lock);
 
 	if (!kvm->arch.mmu.pgt)
-		return false;
+		goto out;
 
-	return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt,
-						   range->start << PAGE_SHIFT,
-						   size, false);
+	young = kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt,
+						    range->start << PAGE_SHIFT,
+						    size, false);
+
+out:
+	read_unlock(&kvm->mmu_lock);
+	return young;
 }
 
 phys_addr_t kvm_mmu_get_httbr(void)
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#14 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 4 lines · James Houghton

Replace the MMU write locks (taken in the memslot iteration loop) for read locks.

Quoted 8 lines

Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change.

---

Here is some data to show that this patch at least *can* be helpful:

# arm64 patched to do aging (i.e., set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY)
# The test is faulting memory in while doing aging as fast as possible.
# taskset -c 0-32 ./access_tracking_perf_test -l -r /dev/cgroup/memory
-p -v 32 -m 3

# Write lock
vcpu wall time : 3.039207157s
lru_gen avg pass duration : 1.660541541s, (passes:2, total:3.321083083s)

# Read lock
vcpu wall time : 3.010848445s
lru_gen avg pass duration : 0.306623698s, (passes:11, total:3.372860688s)

Aging is able to run significantly faster, but vCPU runtime isn't affected much (in this test).

It would be really nice to motivate this patch with a test that didn't require patching the kernel... Oliver and Marc, please let me know if you'd like to see more data. I'm also happy to simply drop this patch.

#15 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 25 lines · James Houghton
Quoted 4 lines · James Houghton

Replace the MMU write locks (taken in the memslot iteration loop) for read locks.

Quoted 8 lines

Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change.

---

Here is some data to show that this patch at least *can* be helpful:

# arm64 patched to do aging (i.e., set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY)
# The test is faulting memory in while doing aging as fast as possible.
# taskset -c 0-32 ./access_tracking_perf_test -l -r /dev/cgroup/memory
-p -v 32 -m 3

# Write lock
vcpu wall time : 3.039207157s
lru_gen avg pass duration : 1.660541541s, (passes:2, total:3.321083083s)

Quoted 6 lines

# Read lock
vcpu wall time : 3.010848445s
lru_gen avg pass duration : 0.306623698s, (passes:11, total:3.372860688s)

Aging is able to run significantly faster, but vCPU runtime isn't affected much (in this test).

Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking mmap_lock for read when handling faults:

https://lore.kernel.org/all/[email protected]

#16 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 31 lines · Sean Christopherson
Quoted 29 lines · James Houghton
Quoted 4 lines · James Houghton

Replace the MMU write locks (taken in the memslot iteration loop) for read locks.

Quoted 8 lines

Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change.

---

Here is some data to show that this patch at least *can* be helpful:

# arm64 patched to do aging (i.e., set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY)
# The test is faulting memory in while doing aging as fast as possible.
# taskset -c 0-32 ./access_tracking_perf_test -l -r /dev/cgroup/memory
-p -v 32 -m 3

# Write lock
vcpu wall time : 3.039207157s
lru_gen avg pass duration : 1.660541541s, (passes:2, total:3.321083083s)

# Read lock
vcpu wall time : 3.010848445s
lru_gen avg pass duration : 0.306623698s, (passes:11, total:3.372860688s)

Quoted 7 lines
Quoted 2 lines

Aging is able to run significantly faster, but vCPU runtime isn't affected much (in this test).

Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking mmap_lock for read when handling faults:

https://lore.kernel.org/all/[email protected]

For the above test, I don't think it's mmap_lock -- the reclaim path, e.g., when zswapping guest memory, has two stages: aging (scanning PTEs) and eviction (unmapping PTEs). Only testing the former isn't realistic at all. IOW, for a r/w lock use case, only testing the read lock path would be bad coverage.

#17 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 1 line · Yu Zhao

On Fri, Aug 16, 2024 at 6:46 PM Sean Christopherson <[email protected]> wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking mmap_lock for read when handling faults:

https://lore.kernel.org/all/[email protected]

For the above test, I don't think it's mmap_lock

Yeah, I don't think this is related to the mmap_lock.

James is likely using hardware that has FEAT_HAFDBS, so vCPUs won't fault for an Access flag update. Even if he's on a machine w/o it, Access flag faults are handled outside the mmap_lock.

Forcing SW management of the AF at stage-2 would be the best case for demonstrating the locking improvement:

arch/arm64/kvm/hyp/pgtable.c +0 -2
index a24a2a857456..a640e8a8c6ea 100644
@@ -669,8 +669,6 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift)
 	 * happen to be running on a design that has unadvertised support for
 	 * HAFDBS. Here be dragons.
 	 */
-	if (!cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
-		vtcr |= VTCR_EL2_HA;
 #endif /* CONFIG_ARM64_HW_AFDBM */
 
 	if (kvm_lpa2_is_enabled())

Changing the config option would work too, but I wasn't sure if FEAT_HAFDBS on the primary MMU influenced MGLRU heuristics.

Quoted 4 lines

-- the reclaim path,
e.g., when zswapping guest memory, has two stages: aging (scanning PTEs) and eviction (unmapping PTEs). Only testing the former isn't realistic at all.

AIUI, the intention of this test data is to provide some justification for why Marc + I should consider the locking change *outside* of any MMU notifier changes. So from that POV, this is meant as a hacked up microbenchmark and not meant to be realistic.

And really, the arm64 change has nothing to do with this series at this point, which is disappointing. In the interest of moving this feature along for both architectures, would you be able help James with:

- Identifying a benchmark that you believe is realistic

- Suggestions on how to run that benchmark on Google infrastructure

Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY as well.

Signature · 2 lines
Thanks,
Oliver
#18 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 8 lines · Oliver Upton
Quoted 1 line · Yu Zhao

On Fri, Aug 16, 2024 at 6:46 PM Sean Christopherson <[email protected]> wrote:

[...]

Quoted 3 lines
Quoted 3 lines

Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking mmap_lock for read when handling faults:

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

https://lore.kernel.org/all/[email protected]

For the above test, I don't think it's mmap_lock

Yeah, I don't think this is related to the mmap_lock.

James is likely using hardware that has FEAT_HAFDBS, so vCPUs won't fault for an Access flag update.

Huh, didn't know that was a thing on ARM. Ooh, that lends even more credence to my assertion that marking folios accessed in handle_access_fault() can go away[*]. I assume hardware-assisted updates means this code in handle_access_fault() will no longer be hit, as KVM simply won't ever get access faults? If so, I'll add that info to the changelog.

	if (kvm_pte_valid(pte))
		kvm_set_pfn_accessed(kvm_pte_to_pfn(pte));

[*] https://lore.kernel.org/all/[email protected]

Quoted 2 lines

Even if he's on a machine w/o it, Access flag faults are handled outside the mmap_lock.

Oh, right, they go down handle_access_fault(), not user_mem_abort().

Reviewing late Friday afternoon, never a good idea ;-)

#19 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

In reply to Oliver Upton
Quoted 8 lines · Oliver Upton
Quoted 1 line · Yu Zhao

On Fri, Aug 16, 2024 at 6:46 PM Sean Christopherson <[email protected]> wrote:

[...]

Quoted 2 lines
Quoted 2 lines

Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking mmap_lock for read when handling faults:

I had no real expectation. I was hoping that maybe there could be a vCPU runtime improvement, given that user_mem_abort() (being called because we're faulting memory in continuously in this test) has to take the KVM MMU lock for reading, and aging is taking it for reading vs. writing. I think that's why aging is a lot slower when using the write lock: it is waiting for the readers to drop the lock, but I guess the delay on the *readers* due to the pending writer seems to be pretty minimal.

Quoted 10 lines
Quoted 4 lines
Quoted 2 lines

https://lore.kernel.org/all/[email protected]

For the above test, I don't think it's mmap_lock

Yeah, I don't think this is related to the mmap_lock.

James is likely using hardware that has FEAT_HAFDBS, so vCPUs won't fault for an Access flag update. Even if he's on a machine w/o it, Access flag faults are handled outside the mmap_lock.

Yeah I was running on Ampere Altra CPUs.

Quoted 8 lines

Forcing SW management of the AF at stage-2 would be the best case for demonstrating the locking improvement:

arch/arm64/kvm/hyp/pgtable.c +0 -0
index a24a2a857456..a640e8a8c6ea 100644

arch/arm64/kvm/hyp/pgtable.c u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) · line 674

Quoted 8 lines
arch/arm64/kvm/hyp/pgtable.c +0 -2
@@ -669,8 +669,6 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift)
         * happen to be running on a design that has unadvertised support for
         * HAFDBS. Here be dragons.
         */
-       if (!cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
-               vtcr |= VTCR_EL2_HA;
 #endif /* CONFIG_ARM64_HW_AFDBM */
 
        if (kvm_lpa2_is_enabled())

Thanks!

Quoted 2 lines

Changing the config option would work too, but I wasn't sure if FEAT_HAFDBS on the primary MMU influenced MGLRU heuristics.

Indeed, disabling CONFIG_ARM64_HW_AFDBM will cause MGLRU not to do aging.

Quoted 15 lines
Quoted 4 lines

-- the reclaim path,
e.g., when zswapping guest memory, has two stages: aging (scanning PTEs) and eviction (unmapping PTEs). Only testing the former isn't realistic at all.

AIUI, the intention of this test data is to provide some justification for why Marc + I should consider the locking change *outside* of any MMU notifier changes. So from that POV, this is meant as a hacked up microbenchmark and not meant to be realistic.

And really, the arm64 change has nothing to do with this series at this point, which is disappointing. In the interest of moving this feature along for both architectures, would you be able help James with:

Quoted 7 lines

- Identifying a benchmark that you believe is realistic

- Suggestions on how to run that benchmark on Google infrastructure

Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY as well.

I'll keep trying some other approaches I can take for getting similar testing that Yu had; it is somewhat difficult for me to reproduce those tests (and it really shouldn't be.... sorry).

I think it makes most sense for me to drop the arm64 patch for now and re-propose it (or something stronger) alongside enabling aging. Does that sound ok?

#20 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 8 lines · James Houghton
Quoted 3 lines · Oliver Upton

Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY as well.

I'll keep trying some other approaches I can take for getting similar testing that Yu had; it is somewhat difficult for me to reproduce those tests (and it really shouldn't be.... sorry).

No need to apologize. Getting good test hardware for arm64 is a complete chore. Sure would love a functional workstation with cores from this decade...

Quoted 3 lines

I think it makes most sense for me to drop the arm64 patch for now and re-propose it (or something stronger) alongside enabling aging. Does that sound ok?

I'm a bit disappointed that we haven't gotten forward progress on the arm64 patches, but I also recognize this is the direction of travel as the x86 patches are shaping up.

So yeah, I'm OK with it, but I'd love to get the arm64 side sorted out soon while the context is still fresh.

Signature · 2 lines
Thanks,
Oliver
#21 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Quoted 19 lines · Oliver Upton
Quoted 8 lines · James Houghton
Quoted 3 lines · Oliver Upton

Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY as well.

I'll keep trying some other approaches I can take for getting similar testing that Yu had; it is somewhat difficult for me to reproduce those tests (and it really shouldn't be.... sorry).

No need to apologize. Getting good test hardware for arm64 is a complete chore. Sure would love a functional workstation with cores from this decade...

Quoted 3 lines

I think it makes most sense for me to drop the arm64 patch for now and re-propose it (or something stronger) alongside enabling aging. Does that sound ok?

Quoted 6 lines

I'm a bit disappointed that we haven't gotten forward progress on the arm64 patches, but I also recognize this is the direction of travel as the x86 patches are shaping up.

So yeah, I'm OK with it, but I'd love to get the arm64 side sorted out soon while the context is still fresh.

Converting the aging notifiers to holding mmu_lock for read seems like a pure win and minimal churn. Can we keep that patch in v7 (which depends on the lockless notifier refactor, i.e. is not completely stand-alone)? We can revisit enabling MGLRU on arm64 in a subsequent series.

Quoted 4 lines

--
Thanks,
Oliver

#22 lore raw

Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn

Hey David,

Quoted 24 lines · David Matlack
Quoted 23 lines · Oliver Upton
Quoted 8 lines · James Houghton
Quoted 3 lines · Oliver Upton

Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY as well.

I'll keep trying some other approaches I can take for getting similar testing that Yu had; it is somewhat difficult for me to reproduce those tests (and it really shouldn't be.... sorry).

No need to apologize. Getting good test hardware for arm64 is a complete chore. Sure would love a functional workstation with cores from this decade...

Quoted 3 lines

I think it makes most sense for me to drop the arm64 patch for now and re-propose it (or something stronger) alongside enabling aging. Does that sound ok?

I'm a bit disappointed that we haven't gotten forward progress on the arm64 patches, but I also recognize this is the direction of travel as the x86 patches are shaping up.

Quoted 8 lines
Quoted 2 lines

So yeah, I'm OK with it, but I'd love to get the arm64 side sorted out soon while the context is still fresh.

Converting the aging notifiers to holding mmu_lock for read seems like a pure win and minimal churn. Can we keep that patch in v7 (which depends on the lockless notifier refactor, i.e. is not completely stand-alone)? We can revisit enabling MGLRU on arm64 in a subsequent series.

Even though the churn is minimal in LOC, locking changes are significant. If one thing has become clear, there are some strong opinions about arm64 participating in MGLRU w/ the read lock. So it is almost guaranteed that these read lock changes will eventually get thrown out in favor of an RCU-protected walker.

Then we're stuck with potentially 3 flavors of locking in kernels that people actually use, and dealing with breakage that only affects that intermediate step is gonna be annoying.

Signature · 2 lines
Thanks,
Oliver
#23 lore raw

PATCH v6 4/11 mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER

In reply to James Houghton

Remove the now unnecessary ifdef in mm/damon/vaddr.c as well.

---
 include/linux/mmu_notifier.h | 7 +++++++
 mm/damon/vaddr.c             | 2 --
 2 files changed, 7 insertions(+), 2 deletions(-)
include/linux/mmu_notifier.h +7 -0
index d39ebb10caeb..e2dd57ca368b 100644
@@ -606,6 +606,13 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm,
 	return 0;
 }
 
+static inline int mmu_notifier_clear_young(struct mm_struct *mm,
+					   unsigned long start,
+					   unsigned long end)
+{
+	return 0;
+}
+
 static inline int mmu_notifier_test_young(struct mm_struct *mm,
 					  unsigned long address)
 {
mm/damon/vaddr.c +0 -2
index 381559e4a1fa..a453d77565e6 100644
@@ -351,11 +351,9 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
 		set_huge_pte_at(mm, addr, pte, entry, psize);
 	}
 
-#ifdef CONFIG_MMU_NOTIFIER
 	if (mmu_notifier_clear_young(mm, addr,
 				     addr + huge_page_size(hstate_vma(vma))))
 		referenced = true;
-#endif /* CONFIG_MMU_NOTIFIER */
 
 	if (referenced)
 		folio_set_young(folio);
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Acked-by on v6 4/11
#24 lore raw

Re: [PATCH v6 04/11] mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER

Quoted 32 lines · James Houghton

Remove the now unnecessary ifdef in mm/damon/vaddr.c as well.

---
  include/linux/mmu_notifier.h | 7 +++++++
  mm/damon/vaddr.c             | 2 --
  2 files changed, 7 insertions(+), 2 deletions(-)
include/linux/mmu_notifier.h +7 -0
index d39ebb10caeb..e2dd57ca368b 100644
@@ -606,6 +606,13 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm,
  	return 0;
  }
 
+static inline int mmu_notifier_clear_young(struct mm_struct *mm,
+					   unsigned long start,
+					   unsigned long end)
+{
+	return 0;
+}
+
  static inline int mmu_notifier_test_young(struct mm_struct *mm,
  					  unsigned long address)
  {
mm/damon/vaddr.c +0 -0
index 381559e4a1fa..a453d77565e6 100644
@@ -351,11 +351,9 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
  		set_huge_pte_at(mm, addr, pte, entry, psize);

mm/damon/vaddr.c static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, · line 359

Quoted 10 lines
mm/damon/vaddr.c +0 -2
  	}
 
-#ifdef CONFIG_MMU_NOTIFIER
  	if (mmu_notifier_clear_young(mm, addr,
  				     addr + huge_page_size(hstate_vma(vma))))
  		referenced = true;
-#endif /* CONFIG_MMU_NOTIFIER */
 
  	if (referenced)
  		folio_set_young(folio);
Signature · 3 lines
Cheers,

David / dhildenb
#26 lore raw

PATCH v6 5/11 mm: Add fast_only bool to test_young and clear_young MMU notifiers

In reply to James Houghton

For implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.

Also add mmu_notifier_test_young_fast_only() and mmu_notifier_clear_young_fast_only() helpers to set fast_only for these notifiers.

---
 include/linux/mmu_notifier.h | 46 +++++++++++++++++++++++++++++++-----
 include/trace/events/kvm.h   | 19 +++++++++------
 mm/mmu_notifier.c            | 12 ++++++----
 virt/kvm/kvm_main.c          | 12 ++++++----
 4 files changed, 67 insertions(+), 22 deletions(-)
include/linux/mmu_notifier.h +40 -6
index e2dd57ca368b..45c5995ebd84 100644
@@ -110,7 +110,8 @@ struct mmu_notifier_ops {
 	int (*clear_young)(struct mmu_notifier *subscription,
 			   struct mm_struct *mm,
 			   unsigned long start,
-			   unsigned long end);
+			   unsigned long end,
+			   bool fast_only);
 
 	/*
 	 * test_young is called to check the young/accessed bitflag in
@@ -120,7 +121,8 @@ struct mmu_notifier_ops {
 	 */
 	int (*test_young)(struct mmu_notifier *subscription,
 			  struct mm_struct *mm,
-			  unsigned long address);
+			  unsigned long address,
+			  bool fast_only);
 
 	/*
 	 * invalidate_range_start() and invalidate_range_end() must be
@@ -380,9 +382,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm,
 					  unsigned long end);
 extern int __mmu_notifier_clear_young(struct mm_struct *mm,
 				      unsigned long start,
-				      unsigned long end);
+				      unsigned long end,
+				      bool fast_only);
 extern int __mmu_notifier_test_young(struct mm_struct *mm,
-				     unsigned long address);
+				     unsigned long address,
+				     bool fast_only);
 extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r);
 extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r);
 extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm,
@@ -416,7 +420,16 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm,
 					   unsigned long end)
 {
 	if (mm_has_notifiers(mm))
-		return __mmu_notifier_clear_young(mm, start, end);
+		return __mmu_notifier_clear_young(mm, start, end, false);
+	return 0;
+}
+
+static inline int mmu_notifier_clear_young_fast_only(struct mm_struct *mm,
+						     unsigned long start,
+						     unsigned long end)
+{
+	if (mm_has_notifiers(mm))
+		return __mmu_notifier_clear_young(mm, start, end, true);
 	return 0;
 }
 
@@ -424,7 +437,15 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm,
 					  unsigned long address)
 {
 	if (mm_has_notifiers(mm))
-		return __mmu_notifier_test_young(mm, address);
+		return __mmu_notifier_test_young(mm, address, false);
+	return 0;
+}
+
+static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm,
+						    unsigned long address)
+{
+	if (mm_has_notifiers(mm))
+		return __mmu_notifier_test_young(mm, address, true);
 	return 0;
 }
 
@@ -613,12 +634,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm,
 	return 0;
 }
 
+static inline int mmu_notifier_clear_young_fast_only(struct mm_struct *mm,
+						     unsigned long start,
+						     unsigned long end)
+{
+	return 0;
+}
+
 static inline int mmu_notifier_test_young(struct mm_struct *mm,
 					  unsigned long address)
 {
 	return 0;
 }
 
+static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm,
+						    unsigned long address)
+{
+	return 0;
+}
+
 static inline void
 mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range)
 {
include/trace/events/kvm.h +12 -7
index 74e40d5d4af4..6d9485cf3e51 100644
@@ -457,36 +457,41 @@ TRACE_EVENT(kvm_unmap_hva_range,
 );
 
 TRACE_EVENT(kvm_age_hva,
-	TP_PROTO(unsigned long start, unsigned long end),
-	TP_ARGS(start, end),
+	TP_PROTO(unsigned long start, unsigned long end, bool fast_only),
+	TP_ARGS(start, end, fast_only),
 
 	TP_STRUCT__entry(
 		__field(	unsigned long,	start		)
 		__field(	unsigned long,	end		)
+		__field(	bool,		fast_only	)
 	),
 
 	TP_fast_assign(
 		__entry->start		= start;
 		__entry->end		= end;
+		__entry->fast_only	= fast_only;
 	),
 
-	TP_printk("mmu notifier age hva: %#016lx -- %#016lx",
-		  __entry->start, __entry->end)
+	TP_printk("mmu notifier age hva: %#016lx -- %#016lx fast_only: %d",
+		  __entry->start, __entry->end, __entry->fast_only)
 );
 
 TRACE_EVENT(kvm_test_age_hva,
-	TP_PROTO(unsigned long hva),
-	TP_ARGS(hva),
+	TP_PROTO(unsigned long hva, bool fast_only),
+	TP_ARGS(hva, fast_only),
 
 	TP_STRUCT__entry(
 		__field(	unsigned long,	hva		)
+		__field(	bool,		fast_only	)
 	),
 
 	TP_fast_assign(
 		__entry->hva		= hva;
+		__entry->fast_only	= fast_only;
 	),
 
-	TP_printk("mmu notifier test age hva: %#016lx", __entry->hva)
+	TP_printk("mmu notifier test age hva: %#016lx fast_only: %d",
+		  __entry->hva, __entry->fast_only)
 );
 
 #endif /* _TRACE_KVM_MAIN_H */
mm/mmu_notifier.c +8 -4
index 8982e6139d07..f9a0ca6ffe65 100644
@@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,
 
 int __mmu_notifier_clear_young(struct mm_struct *mm,
 			       unsigned long start,
-			       unsigned long end)
+			       unsigned long end,
+			       bool fast_only)
 {
 	struct mmu_notifier *subscription;
 	int young = 0, id;
@@ -395,7 +396,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,
 				 srcu_read_lock_held(&srcu)) {
 		if (subscription->ops->clear_young)
 			young |= subscription->ops->clear_young(subscription,
-								mm, start, end);
+								mm, start, end,
+								fast_only);
 	}
 	srcu_read_unlock(&srcu, id);
 
@@ -403,7 +405,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,
 }
 
 int __mmu_notifier_test_young(struct mm_struct *mm,
-			      unsigned long address)
+			      unsigned long address,
+			      bool fast_only)
 {
 	struct mmu_notifier *subscription;
 	int young = 0, id;
@@ -414,7 +417,8 @@ int __mmu_notifier_test_young(struct mm_struct *mm,
 				 srcu_read_lock_held(&srcu)) {
 		if (subscription->ops->test_young) {
 			young = subscription->ops->test_young(subscription, mm,
-							      address);
+							      address,
+							      fast_only);
 			if (young)
 				break;
 		}
virt/kvm/kvm_main.c +7 -5
index 33f8997a5c29..959b6d5d8ce4 100644
@@ -874,7 +874,7 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,
 					      unsigned long start,
 					      unsigned long end)
 {
-	trace_kvm_age_hva(start, end);
+	trace_kvm_age_hva(start, end, false);
 
 	return kvm_handle_hva_range(mn, start, end, kvm_age_gfn);
 }
@@ -882,9 +882,10 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,
 static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
 					struct mm_struct *mm,
 					unsigned long start,
-					unsigned long end)
+					unsigned long end,
+					bool fast_only)
 {
-	trace_kvm_age_hva(start, end);
+	trace_kvm_age_hva(start, end, fast_only);
 
 	/*
 	 * Even though we do not flush TLB, this will still adversely
@@ -904,9 +905,10 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
 
 static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
 				       struct mm_struct *mm,
-				       unsigned long address)
+				       unsigned long address,
+				       bool fast_only)
 {
-	trace_kvm_test_age_hva(address);
+	trace_kvm_test_age_hva(address, fast_only);
 
 	return kvm_handle_hva_range_no_flush(mn, address, address + 1,
 					     kvm_test_age_gfn);
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#27 lore raw

Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers

Quoted 5 lines · James Houghton

For implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.

But what are the semantics if "fast_only" cannot be achieved by the implementer?

Can we add some documentation to the new functions that explain what this mysterious "fast_only" is and what the expected semantics are? Please? :)

Signature · 3 lines
Cheers,

David / dhildenb
#28 lore raw

Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers

Quoted 8 lines · David Hildenbrand
Quoted 5 lines · James Houghton

For implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.

Quoted 6 lines

But what are the semantics if "fast_only" cannot be achieved by the implementer?

Can we add some documentation to the new functions that explain what this mysterious "fast_only" is and what the expected semantics are? Please? :)

Thanks for pointing out the missing documentation. How's this?

include/linux/mmu_notifier.h +14 -0
index 45c5995ebd84..c21992036dd3 100644
@@ -106,6 +106,18 @@ struct mmu_notifier_ops {
         * clear_young is a lightweight version of clear_flush_young. Like the
         * latter, it is supposed to test-and-clear the young/accessed bitflag
         * in the secondary pte, but it may omit flushing the secondary tlb.
+        *
+        * The fast_only parameter indicates that this call should not block,
+        * and this function should not cause other MMU notifier calls to
+        * block. Usually this means that the implementation should be
+        * lockless.
+        *
+        * When called with fast_only, this notifier will be a no-op unless
+        * has_fast_aging is set on the struct mmu_notifier.
+        *
+        * When fast_only is true, if the implementer cannot determine that a
+        * range is young without blocking, it should return 0 (i.e.,
+        * that the range is NOT young).
         */
        int (*clear_young)(struct mmu_notifier *subscription,
                           struct mm_struct *mm,
@@ -118,6 +130,8 @@ struct mmu_notifier_ops {
         * the secondary pte. This is used to know if the page is
         * frequently used without actually clearing the flag or tearing
         * down the secondary mapping on the page.
+        *
+        * The fast_only parameter has the same meaning as with clear_young.
         */
        int (*test_young)(struct mmu_notifier *subscription,
                          struct mm_struct *mm,

I've also moved the commit that follows this one (the one that adds has_fast_aging) to be before this one so that the comment makes sense.

#29 lore raw

Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers

Quoted 23 lines · James Houghton
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · James Houghton

For implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.

Quoted 6 lines

But what are the semantics if "fast_only" cannot be achieved by the implementer?

Can we add some documentation to the new functions that explain what this mysterious "fast_only" is and what the expected semantics are? Please? :)

Thanks for pointing out the missing documentation. How's this?

include/linux/mmu_notifier.h +0 -0
index 45c5995ebd84..c21992036dd3 100644

include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 109

Quoted 4 lines
include/linux/mmu_notifier.h +1 -0
@@ -106,6 +106,18 @@ struct mmu_notifier_ops {
          * clear_young is a lightweight version of clear_flush_young. Like the
          * latter, it is supposed to test-and-clear the young/accessed bitflag
          * in the secondary pte, but it may omit flushing the secondary tlb.
+        *

Probably makes sense to highlight the parameters like @fast_only

include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116

Quoted 7 lines
include/linux/mmu_notifier.h +7 -0
+        * The fast_only parameter indicates that this call should not block,
+        * and this function should not cause other MMU notifier calls to
+        * block. Usually this means that the implementation should be
+        * lockless.
+        *
+        * When called with fast_only, this notifier will be a no-op unless
+        * has_fast_aging is set on the struct mmu_notifier.

"... and will return 0 (NOT young)." ?

Quoted 17 lines
include/linux/mmu_notifier.h +6 -0
+        *
+        * When fast_only is true, if the implementer cannot determine that a
+        * range is young without blocking, it should return 0 (i.e.,
+        * that the range is NOT young).
          */
         int (*clear_young)(struct mmu_notifier *subscription,
                            struct mm_struct *mm,
@@ -118,6 +130,8 @@ struct mmu_notifier_ops {
          * the secondary pte. This is used to know if the page is
          * frequently used without actually clearing the flag or tearing
          * down the secondary mapping on the page.
+        *
+        * The fast_only parameter has the same meaning as with clear_young.
          */
         int (*test_young)(struct mmu_notifier *subscription,
                           struct mm_struct *mm,
Quoted 2 lines

I've also moved the commit that follows this one (the one that adds has_fast_aging) to be before this one so that the comment makes sense.

Makes sense, thanks!

Signature · 3 lines
Cheers,

David / dhildenb
#30 lore raw

Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers

Quoted 20 lines · David Hildenbrand
Quoted 18 lines · James Houghton
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · James Houghton

For implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.

Quoted 6 lines

But what are the semantics if "fast_only" cannot be achieved by the implementer?

Can we add some documentation to the new functions that explain what this mysterious "fast_only" is and what the expected semantics are? Please? :)

Thanks for pointing out the missing documentation. How's this?

Quoted 11 lines

include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 109

Quoted 9 lines
include/linux/mmu_notifier.h +1 -0
index 45c5995ebd84..c21992036dd3 100644
@@ -106,6 +106,18 @@ struct mmu_notifier_ops {
          * clear_young is a lightweight version of clear_flush_young. Like the
          * latter, it is supposed to test-and-clear the young/accessed bitflag
          * in the secondary pte, but it may omit flushing the secondary tlb.
+        *

Probably makes sense to highlight the parameters like @fast_only

Will do.

Quoted 10 lines

include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116

Quoted 8 lines
include/linux/mmu_notifier.h +7 -0
+        * The fast_only parameter indicates that this call should not block,
+        * and this function should not cause other MMU notifier calls to
+        * block. Usually this means that the implementation should be
+        * lockless.
+        *
+        * When called with fast_only, this notifier will be a no-op unless
+        * has_fast_aging is set on the struct mmu_notifier.

"... and will return 0 (NOT young)." ?

Thanks, I'll add this.

Quoted 18 lines
Quoted 18 lines
include/linux/mmu_notifier.h +6 -0
+        *
+        * When fast_only is true, if the implementer cannot determine that a
+        * range is young without blocking, it should return 0 (i.e.,
+        * that the range is NOT young).
          */
         int (*clear_young)(struct mmu_notifier *subscription,
                            struct mm_struct *mm,
@@ -118,6 +130,8 @@ struct mmu_notifier_ops {
          * the secondary pte. This is used to know if the page is
          * frequently used without actually clearing the flag or tearing
          * down the secondary mapping on the page.
+        *
+        * The fast_only parameter has the same meaning as with clear_young.
          */
         int (*test_young)(struct mmu_notifier *subscription,
                           struct mm_struct *mm,
Quoted 5 lines
Quoted 2 lines

I've also moved the commit that follows this one (the one that adds has_fast_aging) to be before this one so that the comment makes sense.

Makes sense, thanks!

Thanks David!

#31 lore raw

Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers

In reply to James Houghton
Quoted 4 lines · James Houghton
include/linux/mmu_notifier.h +0 -0
@@ -106,6 +106,18 @@ struct mmu_notifier_ops {
         * clear_young is a lightweight version of clear_flush_young. Like the

include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116

Quoted 10 lines
include/linux/mmu_notifier.h +8 -0
         * latter, it is supposed to test-and-clear the young/accessed bitflag
         * in the secondary pte, but it may omit flushing the secondary tlb.
+        *
+        * The fast_only parameter indicates that this call should not block,
+        * and this function should not cause other MMU notifier calls to
+        * block. Usually this means that the implementation should be
+        * lockless.
+        *
+        * When called with fast_only, this notifier will be a no-op unless
+        * has_fast_aging is set on the struct mmu_notifier.

If you add a has_fast_aging I wonder if it is better to introduce new ops instead? The semantics are a bit easier to explain that way

Jason

#32 lore raw

Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers

Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · James Houghton
include/linux/mmu_notifier.h +3 -0
@@ -106,6 +106,18 @@ struct mmu_notifier_ops {
         * clear_young is a lightweight version of clear_flush_young. Like the
         * latter, it is supposed to test-and-clear the young/accessed bitflag
         * in the secondary pte, but it may omit flushing the secondary tlb.
+        *
+        * The fast_only parameter indicates that this call should not block,
+        * and this function should not cause other MMU notifier calls to
Quoted 8 lines

include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116

Quoted 5 lines
include/linux/mmu_notifier.h +5 -0
+        * block. Usually this means that the implementation should be
+        * lockless.
+        *
+        * When called with fast_only, this notifier will be a no-op unless
+        * has_fast_aging is set on the struct mmu_notifier.

If you add a has_fast_aging I wonder if it is better to introduce new ops instead? The semantics are a bit easier to explain that way

v5 implemented these with a new op[1]. *Just* having the new op is kind of problematic -- we have yet another op to do something very similar to what already exists. We are left with two options:
consolidate everything into a single notifier[2] or add a new parameter to test/clear_young()[3]. The latter, implemented in this v6, is somewhat simpler to implement (fewer LoC, reduces some duplication in KVM), though it does indeed make the explanation for test/clear_young() slightly more complex. I don't feel very strongly, but unless you do, I think I just ought to stick with how the v6 does it. :)

Thanks Jason!

[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/CADrL8HVHcKSW3hiHzKTit07gzo36jtCZCnM9ZpueyifgNdGggw@mail.gmail.com/
[3]: https://lore.kernel.org/linux-mm/CADrL8HXhGFWwHt728Bg15x1YxJmS=WD8z=KJc_Koaah=OvHDwg@mail.gmail.com/

#33 lore raw

Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers

Quoted 21 lines · James Houghton
Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · James Houghton
include/linux/mmu_notifier.h +3 -0
@@ -106,6 +106,18 @@ struct mmu_notifier_ops {
         * clear_young is a lightweight version of clear_flush_young. Like the
         * latter, it is supposed to test-and-clear the young/accessed bitflag
         * in the secondary pte, but it may omit flushing the secondary tlb.
+        *
+        * The fast_only parameter indicates that this call should not block,
+        * and this function should not cause other MMU notifier calls to
Quoted 8 lines

include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116

Quoted 5 lines
include/linux/mmu_notifier.h +5 -0
+        * block. Usually this means that the implementation should be
+        * lockless.
+        *
+        * When called with fast_only, this notifier will be a no-op unless
+        * has_fast_aging is set on the struct mmu_notifier.

If you add a has_fast_aging I wonder if it is better to introduce new ops instead? The semantics are a bit easier to explain that way

Quoted 10 lines

v5 implemented these with a new op[1]. *Just* having the new op is kind of problematic -- we have yet another op to do something very similar to what already exists. We are left with two options:
consolidate everything into a single notifier[2] or add a new parameter to test/clear_young()[3]. The latter, implemented in this v6, is somewhat simpler to implement (fewer LoC, reduces some duplication in KVM), though it does indeed make the explanation for test/clear_young() slightly more complex. I don't feel very strongly, but unless you do, I think I just ought to stick with how the v6 does it. :)

If it does makes the code simpler then it is probably the better choice

Jason

#34 lore raw

PATCH v6 6/11 mm: Add has_fast_aging to struct mmu_notifier

In reply to James Houghton

has_fast_aging should be set by subscribers that non-trivially implement fast_only versions of both test_young() and clear_young().

Fast aging must be opt-in. For a subscriber that has not been enlightened with "fast aging", the test/clear_young() will behave identically whether or not fast_only is given.

Given that KVM is the only test/clear_young() implementer, we could instead add an equivalent check in KVM, but doing so would incur an indirect function call every time, even if the notifier ends up being a no-op.

Add mm_has_fast_young_notifiers() in case a caller wants to know if it should skip many calls to the mmu notifiers that may not be necessary (like MGLRU look-around).

---
 include/linux/mmu_notifier.h | 14 ++++++++++++++
 mm/mmu_notifier.c            | 26 ++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
include/linux/mmu_notifier.h +14 -0
index 45c5995ebd84..e23fc10f864b 100644
@@ -233,6 +233,7 @@ struct mmu_notifier {
 	struct mm_struct *mm;
 	struct rcu_head rcu;
 	unsigned int users;
+	bool has_fast_aging;
 };
 
 /**
@@ -387,6 +388,7 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm,
 extern int __mmu_notifier_test_young(struct mm_struct *mm,
 				     unsigned long address,
 				     bool fast_only);
+extern bool __mm_has_fast_young_notifiers(struct mm_struct *mm);
 extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r);
 extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r);
 extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm,
@@ -449,6 +451,13 @@ static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm,
 	return 0;
 }
 
+static inline bool mm_has_fast_young_notifiers(struct mm_struct *mm)
+{
+	if (mm_has_notifiers(mm))
+		return __mm_has_fast_young_notifiers(mm);
+	return 0;
+}
+
 static inline void
 mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range)
 {
@@ -653,6 +662,11 @@ static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm,
 	return 0;
 }
 
+static inline bool mm_has_fast_young_notifiers(struct mm_struct *mm)
+{
+	return 0;
+}
+
 static inline void
 mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range)
 {
mm/mmu_notifier.c +26 -0
index f9a0ca6ffe65..f9ec810c8a1b 100644
@@ -382,6 +382,26 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,
 	return young;
 }
 
+bool __mm_has_fast_young_notifiers(struct mm_struct *mm)
+{
+	struct mmu_notifier *subscription;
+	bool has_fast_aging = false;
+	int id;
+
+	id = srcu_read_lock(&srcu);
+	hlist_for_each_entry_rcu(subscription,
+				 &mm->notifier_subscriptions->list, hlist,
+				 srcu_read_lock_held(&srcu)) {
+		if (subscription->has_fast_aging) {
+			has_fast_aging = true;
+			break;
+		}
+	}
+	srcu_read_unlock(&srcu, id);
+
+	return has_fast_aging;
+}
+
 int __mmu_notifier_clear_young(struct mm_struct *mm,
 			       unsigned long start,
 			       unsigned long end,
@@ -394,6 +414,9 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,
 	hlist_for_each_entry_rcu(subscription,
 				 &mm->notifier_subscriptions->list, hlist,
 				 srcu_read_lock_held(&srcu)) {
+		if (fast_only && !subscription->has_fast_aging)
+			continue;
+
 		if (subscription->ops->clear_young)
 			young |= subscription->ops->clear_young(subscription,
 								mm, start, end,
@@ -415,6 +438,9 @@ int __mmu_notifier_test_young(struct mm_struct *mm,
 	hlist_for_each_entry_rcu(subscription,
 				 &mm->notifier_subscriptions->list, hlist,
 				 srcu_read_lock_held(&srcu)) {
+		if (fast_only && !subscription->has_fast_aging)
+			continue;
+
 		if (subscription->ops->test_young) {
 			young = subscription->ops->test_young(subscription, mm,
 							      address,
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#35 lore raw

PATCH v6 7/11 KVM: Pass fast_only to kvm_{test_,}age_gfn

In reply to James Houghton

Provide the basics for architectures to implement a fast-only version of kvm_{test_,}age_gfn.

Add CONFIG_HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY that architectures will set if they non-trivially implement test_young() and clear_young() when called with fast_only.

---
 include/linux/kvm_host.h |  1 +
 virt/kvm/Kconfig         |  4 ++++
 virt/kvm/kvm_main.c      | 37 +++++++++++++++++++++----------------
 3 files changed, 26 insertions(+), 16 deletions(-)
include/linux/kvm_host.h +1 -0
index 8cd80f969cff..944c5fba2344 100644
@@ -258,6 +258,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER
 union kvm_mmu_notifier_arg {
 	unsigned long attributes;
+	bool fast_only;
 };
 
 struct kvm_gfn_range {
virt/kvm/Kconfig +4 -0
index 632334861001..cb4d5384c2f2 100644
@@ -103,6 +103,10 @@ config KVM_GENERIC_MMU_NOTIFIER
 config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
        bool
 
+config HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY
+       select KVM_GENERIC_MMU_NOTIFIER
+       bool
+
 config KVM_GENERIC_MEMORY_ATTRIBUTES
        depends on KVM_GENERIC_MMU_NOTIFIER
        bool
virt/kvm/kvm_main.c +21 -16
index 959b6d5d8ce4..86fb2b560d98 100644
@@ -697,18 +697,20 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,
 static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn,
 							 unsigned long start,
 							 unsigned long end,
-							 gfn_handler_t handler)
+							 gfn_handler_t handler,
+							 bool fast_only)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
 	const struct kvm_mmu_notifier_range range = {
-		.start		= start,
-		.end		= end,
-		.handler	= handler,
-		.on_lock	= (void *)kvm_null_fn,
-		.flush_on_ret	= false,
-		.may_block	= false,
-		.lockless	=
+		.start			= start,
+		.end			= end,
+		.handler		= handler,
+		.on_lock		= (void *)kvm_null_fn,
+		.flush_on_ret		= false,
+		.may_block		= false,
+		.lockless		=
 			IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
+		.arg.fast_only		= fast_only,
 	};
 
 	return __kvm_handle_hva_range(kvm, &range).ret;
@@ -900,7 +902,8 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
 	 * cadence. If we find this inaccurate, we might come up with a
 	 * more sophisticated heuristic later.
 	 */
-	return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn);
+	return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn,
+					     fast_only);
 }
 
 static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
@@ -911,7 +914,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
 	trace_kvm_test_age_hva(address, fast_only);
 
 	return kvm_handle_hva_range_no_flush(mn, address, address + 1,
-					     kvm_test_age_gfn);
+					     kvm_test_age_gfn, fast_only);
 }
 
 static void kvm_mmu_notifier_release(struct mmu_notifier *mn,
@@ -926,17 +929,19 @@ static void kvm_mmu_notifier_release(struct mmu_notifier *mn,
 }
 
 static const struct mmu_notifier_ops kvm_mmu_notifier_ops = {
-	.invalidate_range_start	= kvm_mmu_notifier_invalidate_range_start,
-	.invalidate_range_end	= kvm_mmu_notifier_invalidate_range_end,
-	.clear_flush_young	= kvm_mmu_notifier_clear_flush_young,
-	.clear_young		= kvm_mmu_notifier_clear_young,
-	.test_young		= kvm_mmu_notifier_test_young,
-	.release		= kvm_mmu_notifier_release,
+	.invalidate_range_start		= kvm_mmu_notifier_invalidate_range_start,
+	.invalidate_range_end		= kvm_mmu_notifier_invalidate_range_end,
+	.clear_flush_young		= kvm_mmu_notifier_clear_flush_young,
+	.clear_young			= kvm_mmu_notifier_clear_young,
+	.test_young			= kvm_mmu_notifier_test_young,
+	.release			= kvm_mmu_notifier_release,
 };
 
 static int kvm_init_mmu_notifier(struct kvm *kvm)
 {
 	kvm->mmu_notifier.ops = &kvm_mmu_notifier_ops;
+	kvm->mmu_notifier.has_fast_aging =
+		IS_ENABLED(CONFIG_HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY);
 	return mmu_notifier_register(&kvm->mmu_notifier, current->mm);
 }
 
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#36 lore raw

PATCH v6 8/11 KVM: x86: Optimize kvm_{test_,}age_gfn a little bit

In reply to James Houghton

Optimize both kvm_age_gfn and kvm_test_age_gfn's interaction with the shadow MMU by, rather than checking if our memslot has rmaps, check if there are any indirect_shadow_pages at all.

Also, for kvm_test_age_gfn, reorder the TDP MMU check to be first. If we find that the range is young, we do not need to check the shadow MMU.

---
 arch/x86/kvm/mmu/mmu.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +13 -8
index 7b93ce8f0680..919d59385f89 100644
@@ -1629,19 +1629,24 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot,
 	__rmap_add(vcpu->kvm, cache, slot, spte, gfn, access);
 }
 
+static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm)
+{
+	return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages);
+}
+
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
-	if (kvm_memslots_have_rmaps(kvm)) {
+	if (tdp_mmu_enabled)
+		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
+
+	if (kvm_has_shadow_mmu_sptes(kvm)) {
 		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
 		write_unlock(&kvm->mmu_lock);
 	}
 
-	if (tdp_mmu_enabled)
-		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
-
 	return young;
 }
 
@@ -1649,15 +1654,15 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
-	if (kvm_memslots_have_rmaps(kvm)) {
+	if (tdp_mmu_enabled)
+		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
+
+	if (!young && kvm_has_shadow_mmu_sptes(kvm)) {
 		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
 		write_unlock(&kvm->mmu_lock);
 	}
 
-	if (tdp_mmu_enabled)
-		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
-
 	return young;
 }
 
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#37 lore raw

Re: [PATCH v6 08/11] KVM: x86: Optimize kvm_{test_,}age_gfn a little bit

Quoted 1 line · James Houghton

Optimize both kvm_age_gfn and kvm_test_age_gfn's interaction with the

nit: Use () when referring to functions.

Quoted 2 lines

shadow MMU by, rather than checking if our memslot has rmaps, check if there are any indirect_shadow_pages at all.

What is optimized by checking indirect_shadow_pages instead of have_rmaps and what's the benefit? Smells like a premature optimization.

Quoted 3 lines

Also, for kvm_test_age_gfn, reorder the TDP MMU check to be first. If we find that the range is young, we do not need to check the shadow MMU.

This should be a separate commit since it's a logically distinct change and no dependency on the other change in this commit (other than both touch the same function).

Splitting the commits up will also make it easier to write more specific short logs (instead of "optimize a little bit" :)

Also, the commit re-orders kvm_age_gfn() as well but the commit message only mentions kvm_test_age_gfn(). No objection to keeping the two functions consistent but it should be called out in the commit message.

Quoted 40 lines
---
 arch/x86/kvm/mmu/mmu.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -4
index 7b93ce8f0680..919d59385f89 100644
@@ -1629,19 +1629,24 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot,
 	__rmap_add(vcpu->kvm, cache, slot, spte, gfn, access);
 }
 
+static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm)
+{
+	return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages);
+}
+
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
-	if (kvm_memslots_have_rmaps(kvm)) {
+	if (tdp_mmu_enabled)
+		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
+
+	if (kvm_has_shadow_mmu_sptes(kvm)) {
 		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
 		write_unlock(&kvm->mmu_lock);
 	}
 
-	if (tdp_mmu_enabled)
-		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
-
 	return young;
 }
 

arch/x86/kvm/mmu/mmu.c bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · line 1660

Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +4 -1
@@ -1649,15 +1654,15 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
-	if (kvm_memslots_have_rmaps(kvm)) {
+	if (tdp_mmu_enabled)
+		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
+
+	if (!young && kvm_has_shadow_mmu_sptes(kvm)) {

nit: A short comment here might be helpful to explain why young is checked.

Quoted 14 lines
arch/x86/kvm/mmu/mmu.c +0 -3
 		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
 		write_unlock(&kvm->mmu_lock);
 	}
 
-	if (tdp_mmu_enabled)
-		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
-
 	return young;
 }
 
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#38 lore raw

Re: [PATCH v6 08/11] KVM: x86: Optimize kvm_{test_,}age_gfn a little bit

Quoted 10 lines · David Matlack
Quoted 1 line · James Houghton

Optimize both kvm_age_gfn and kvm_test_age_gfn's interaction with the

nit: Use () when referring to functions.

Quoted 2 lines

shadow MMU by, rather than checking if our memslot has rmaps, check if there are any indirect_shadow_pages at all.

What is optimized by checking indirect_shadow_pages instead of have_rmaps and what's the benefit? Smells like a premature optimization.

Checking indirect_shadow_pages avoids taking mmu_lock for write when KVM doesn't currently have shadow MMU pages, but did at some point in the past, whereas kvm_memslots_have_rmaps() is sticky and will return true forever.

Quoted 9 lines
Quoted 2 lines

Also, for kvm_test_age_gfn, reorder the TDP MMU check to be first. If we find that the range is young, we do not need to check the shadow MMU.

This should be a separate commit since it's a logically distinct change and no dependency on the other change in this commit (other than both touch the same function).

Splitting the commits up will also make it easier to write more specific short logs (instead of "optimize a little bit" :)

+1. Especially code movement and refactoring, e.g. factoring out tdp_mmu_clear_spte_bits_atomic() would ideally be in a standalone patch that's dead simple to review.

#39 lore raw

Re: [PATCH v6 08/11] KVM: x86: Optimize kvm_{test_,}age_gfn a little bit

Quoted 10 lines · Sean Christopherson
Quoted 8 lines · David Matlack
Quoted 1 line · James Houghton

Optimize both kvm_age_gfn and kvm_test_age_gfn's interaction with the

nit: Use () when referring to functions.

Quoted 2 lines

shadow MMU by, rather than checking if our memslot has rmaps, check if there are any indirect_shadow_pages at all.

Quoted 6 lines
Quoted 2 lines

What is optimized by checking indirect_shadow_pages instead of have_rmaps and what's the benefit? Smells like a premature optimization.

Checking indirect_shadow_pages avoids taking mmu_lock for write when KVM doesn't currently have shadow MMU pages, but did at some point in the past, whereas kvm_memslots_have_rmaps() is sticky and will return true forever.

Thanks for the clear explanation.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Also, for kvm_test_age_gfn, reorder the TDP MMU check to be first. If we find that the range is young, we do not need to check the shadow MMU.

This should be a separate commit since it's a logically distinct change and no dependency on the other change in this commit (other than both touch the same function).

Done.

Quoted 6 lines
Quoted 2 lines

Splitting the commits up will also make it easier to write more specific short logs (instead of "optimize a little bit" :)

+1. Especially code movement and refactoring, e.g. factoring out tdp_mmu_clear_spte_bits_atomic() would ideally be in a standalone patch that's dead simple to review.

I have now split out the creation of tdp_mmu_clear_spte_bits_atomic() into its own patch. Though I'm not entirely convinced splitting out every refactor like that is always a good thing.

#40 lore raw

PATCH v6 9/11 KVM: x86: Implement fast_only versions of kvm_{test_,}age_gfn

In reply to James Houghton

These fast-only versions simply ignore the shadow MMU. We can locklessly handle the shadow MMU later.

Set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY for X86_64 only, as that is the only case where the TDP MMU might be used. Without the TDP MMU, the fast-only notifiers will always be no-ops. It would be ideal not to report has_fast_only if !tdp_mmu_enabled, but tdp_mmu_enabled can be changed at any time.

---
 arch/x86/kvm/Kconfig   | 1 +
 arch/x86/kvm/mmu/mmu.c | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kvm/Kconfig +1 -0
index 6ac43074c5e9..ed9049cf1255 100644
@@ -24,6 +24,7 @@ config KVM
 	select KVM_COMMON
 	select KVM_GENERIC_MMU_NOTIFIER
 	select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
+	select HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY if X86_64
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_PFNCACHE
 	select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +2 -2
index 919d59385f89..3c6c9442434a 100644
@@ -1641,7 +1641,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 	if (tdp_mmu_enabled)
 		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
 
-	if (kvm_has_shadow_mmu_sptes(kvm)) {
+	if (!range->arg.fast_only && kvm_has_shadow_mmu_sptes(kvm)) {
 		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
 		write_unlock(&kvm->mmu_lock);
@@ -1657,7 +1657,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 	if (tdp_mmu_enabled)
 		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
 
-	if (!young && kvm_has_shadow_mmu_sptes(kvm)) {
+	if (!young && !range->arg.fast_only && kvm_has_shadow_mmu_sptes(kvm)) {
 		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
 		write_unlock(&kvm->mmu_lock);
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#41 lore raw

Re: [PATCH v6 09/11] KVM: x86: Implement fast_only versions of kvm_{test_,}age_gfn

Quoted 8 lines · James Houghton

These fast-only versions simply ignore the shadow MMU. We can locklessly handle the shadow MMU later.

Set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY for X86_64 only, as that is the only case where the TDP MMU might be used. Without the TDP MMU, the fast-only notifiers will always be no-ops. It would be ideal not to report has_fast_only if !tdp_mmu_enabled, but tdp_mmu_enabled can be changed at any time.

tdp_mmu_enabled is a read-only KVM parameter. And even when it was writable, it was still fixed for a given VM at VM creation time.

Would it make more sense to have kvm_arch_post_init_vm() set has_fast_aging if the architecture supports it. And for x86 that means iff tdp_mmu_enabled.

Quoted 44 lines
---
 arch/x86/kvm/Kconfig   | 1 +
 arch/x86/kvm/mmu/mmu.c | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kvm/Kconfig +1 -0
index 6ac43074c5e9..ed9049cf1255 100644
@@ -24,6 +24,7 @@ config KVM
 	select KVM_COMMON
 	select KVM_GENERIC_MMU_NOTIFIER
 	select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS
+	select HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY if X86_64
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_PFNCACHE
 	select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +2 -2
index 919d59385f89..3c6c9442434a 100644
@@ -1641,7 +1641,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 	if (tdp_mmu_enabled)
 		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
 
-	if (kvm_has_shadow_mmu_sptes(kvm)) {
+	if (!range->arg.fast_only && kvm_has_shadow_mmu_sptes(kvm)) {
 		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
 		write_unlock(&kvm->mmu_lock);
@@ -1657,7 +1657,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 	if (tdp_mmu_enabled)
 		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
 
-	if (!young && kvm_has_shadow_mmu_sptes(kvm)) {
+	if (!young && !range->arg.fast_only && kvm_has_shadow_mmu_sptes(kvm)) {
 		write_lock(&kvm->mmu_lock);
 		young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
 		write_unlock(&kvm->mmu_lock);
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#42 lore raw

PATCH v6 10/11 mm: multi-gen LRU: Have secondary MMUs participate in aging

In reply to James Houghton

Secondary MMUs are currently consulted for access/age information at eviction time, but before then, we don't get accurate age information. That is, pages that are mostly accessed through a secondary MMU (like guest memory, used by KVM) will always just proceed down to the oldest generation, and then at eviction time, if KVM reports the page to be young, the page will be activated/promoted back to the youngest generation.

The added feature bit (0x8), if disabled, will make MGLRU behave as if there are no secondary MMUs subscribed to MMU notifiers except at eviction time.

Implement aging with the new mmu_notifier_clear_young_fast_only() notifier. For architectures that do not support this notifier, this becomes a no-op. For architectures that do implement it, it should be fast enough to make aging worth it (usually the case if the notifier is implemented locklessly).

---
 Documentation/admin-guide/mm/multigen_lru.rst |   6 +-
 include/linux/mmzone.h                        |   6 +-
 mm/rmap.c                                     |   9 +-
 mm/vmscan.c                                   | 148 ++++++++++++++----
 4 files changed, 127 insertions(+), 42 deletions(-)
Documentation/admin-guide/mm/multigen_lru.rst +5 -1
index 33e068830497..e1862407652c 100644
@@ -48,6 +48,10 @@ Values Components
        verified on x86 varieties other than Intel and AMD. If it is
        disabled, the multi-gen LRU will suffer a negligible
        performance degradation.
+0x0008 Clear the accessed bit in secondary MMU page tables when aging
+       instead of waiting until eviction time. This results in accurate
+       page age information for pages that are mainly used by a
+       secondary MMU.
 [yYnN] Apply to all the components above.
 ====== ===============================================================
 
@@ -56,7 +60,7 @@ E.g.,
 
     echo y >/sys/kernel/mm/lru_gen/enabled
     cat /sys/kernel/mm/lru_gen/enabled
-    0x0007
+    0x000f
     echo 5 >/sys/kernel/mm/lru_gen/enabled
     cat /sys/kernel/mm/lru_gen/enabled
     0x0005
include/linux/mmzone.h +4 -2
index 586a8f0104d7..ee82e635e75b 100644
@@ -400,6 +400,7 @@ enum {
 	LRU_GEN_CORE,
 	LRU_GEN_MM_WALK,
 	LRU_GEN_NONLEAF_YOUNG,
+	LRU_GEN_SECONDARY_MMU_WALK,
 	NR_LRU_GEN_CAPS
 };
 
@@ -557,7 +558,7 @@ struct lru_gen_memcg {
 
 void lru_gen_init_pgdat(struct pglist_data *pgdat);
 void lru_gen_init_lruvec(struct lruvec *lruvec);
-void lru_gen_look_around(struct page_vma_mapped_walk *pvmw);
+bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw);
 
 void lru_gen_init_memcg(struct mem_cgroup *memcg);
 void lru_gen_exit_memcg(struct mem_cgroup *memcg);
@@ -576,8 +577,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec)
 {
 }
 
-static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
+static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 {
+	return false;
 }
 
 static inline void lru_gen_init_memcg(struct mem_cgroup *memcg)
mm/rmap.c +3 -6
index e8fc5ecb59b2..24a3ff639919 100644
@@ -870,13 +870,10 @@ static bool folio_referenced_one(struct folio *folio,
 			continue;
 		}
 
-		if (pvmw.pte) {
-			if (lru_gen_enabled() &&
-			    pte_young(ptep_get(pvmw.pte))) {
-				lru_gen_look_around(&pvmw);
+		if (lru_gen_enabled() && pvmw.pte) {
+			if (lru_gen_look_around(&pvmw))
 				referenced++;
-			}
-
+		} else if (pvmw.pte) {
 			if (ptep_clear_flush_young_notify(vma, address,
 						pvmw.pte))
 				referenced++;
mm/vmscan.c +115 -33

314 lines not on this page · raw message

Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
#43 lore raw

PATCH v6 11/11 KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test

In reply to James Houghton

This test now has two modes of operation:
1. (default) To check how much vCPU performance was affected by access tracking (previously existed, now supports MGLRU aging).
2. (-p) To also benchmark how fast MGLRU can do aging while vCPUs are faulting in memory.

Mode (1) also serves as a way to verify that aging is working properly for pages only accessed by KVM. It will fail if one does not have the 0x8 lru_gen feature bit.

To support MGLRU, the test creates a memory cgroup, moves itself into it, then uses the lru_gen debugfs output to track memory in that cgroup. The logic to parse the lru_gen debugfs output has been put into selftests/kvm/lib/lru_gen_util.c.

---
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/access_tracking_perf_test.c | 369 +++++++++++++++--
 .../selftests/kvm/include/lru_gen_util.h      |  55 +++
 .../testing/selftests/kvm/lib/lru_gen_util.c  | 391 ++++++++++++++++++
 4 files changed, 786 insertions(+), 30 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h
 create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c
tools/testing/selftests/kvm/Makefile +1 -0
index b084ba2262a0..0ab8d3f4628c 100644
@@ -22,6 +22,7 @@ LIBKVM += lib/elf.c
 LIBKVM += lib/guest_modes.c
 LIBKVM += lib/io.c
 LIBKVM += lib/kvm_util.c
+LIBKVM += lib/lru_gen_util.c
 LIBKVM += lib/memstress.c
 LIBKVM += lib/guest_sprintf.c
 LIBKVM += lib/rbtree.c
tools/testing/selftests/kvm/access_tracking_perf_test.c +339 -30

488 lines not on this page · raw message

tools/testing/selftests/kvm/include/lru_gen_util.h new file +55 -0

55 lines not on this page · raw message

tools/testing/selftests/kvm/lib/lru_gen_util.c new file +391 -0

391 lines not on this page · raw message

Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog