mm: multi-gen LRU: Walk secondary MMU page tables while aging
Subsystems (15)
v6 cover mm: multi-gen LRU: Walk secondary MMU page tables while aging
This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim.
This series does the following:
1. Improve locking for the existing test/clear_young notifiers for x86
and arm64.
2. Add a fast_only parameter into test_young() and clear_young(), and add helper functions for using the new parameter (e.g. mmu_notifier_clear_young_fast_only(). Non-trivially implement the fast-only test_young() and clear_young() for x86_64.
3. Incorporate mmu_notifier_clear_young_fast_only() into MGLRU aging.
4. Add an MGLRU mode (-l) to access_tracking_perf_test to show that aging is working properly.
Please note that mmu_notifier_test_young_fast_only() is added but not used in this series. I am happy to remove it if that would be appropriate.
The fast-only notifiers serve a particular purpose: for aging, we neither want to delay other operations (e.g. unmapping for eviction) nor do we want to be delayed by these other operations ourselves. By default, the implementations of test_young() and clear_young() are meant to be *accurate*, not fast. The fast-only notifiers will only give age information that can be gathered fast.
The fast-only notifiers are non-trivially implemented for only x86_64 right now (as the KVM/x86 TDP MMU is the only secondary MMU that supports lockless Accessed bit harvesting).
To make aging work for more than just x86, the fast-only clear_young() notifier must be non-trivially implemented by those other architectures and HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY needs to be set.
access_tracking_perf_test now has a mode (-p) to check performance of MGLRU aging while the VM is faulting memory in. See the v4 cover letter[2] for performance data collected with this test.
Previous versions of this series included logic in MGLRU and KVM to support batching the updates to secondary page tables. This version removes this logic, as it was complex and not necessary to enable proactive reclaim. This optimization, as well as enabling aging for arm64 and powerpc, can be done in a later series.
=== Previous Versions ===
Since v5[1]:
- Reworked test_clear_young_fast_only() into a new parameter for the existing notifiers (thanks Sean).
- Added mmu_notifier.has_fast_aging to tell mm if calling fast-only notifiers should be done.
- Added mm_has_fast_young_notifiers() to inform users if calling fast-only notifier helpers is worthwhile (for look-around to use).
- Changed MGLRU to invoke a single notifier instead of two when aging and doing look-around (thanks Yu).
- For KVM/x86, check indirect_shadow_pages > 0 instead of kvm_memslots_have_rmaps() when collecting age information (thanks Sean).
- For KVM/arm, some fixes from Oliver.
- Small fixes to access_tracking_perf_test.
- Added missing !MMU_NOTIFIER version of mmu_notifier_clear_young().
Since v4[2]:
- Removed Kconfig that controlled when aging was enabled. Aging will be done whenever the architecture supports it (thanks Yu).
- Added a new MMU notifier, test_clear_young_fast_only(), specifically for MGLRU to use.
- Add kvm_fast_{test_,}age_gfn, implemented by x86.
- Fix locking for clear_flush_young().
- Added KVM_MMU_NOTIFIER_YOUNG_LOCKLESS to clean up locking changes (thanks Sean).
- Fix WARN_ON and other cleanup for the arm64 locking changes (thanks Oliver).
Since v3[3]:
- Vastly simplified the series (thanks David). Removed mmu notifier batching logic entirely.
- Cleaned up how locking is done for mmu_notifier_test/clear_young (thanks David).
- Look-around is now only done when there are no secondary MMUs subscribed to MMU notifiers.
- CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added.
- Fixed the lockless implementation of kvm_{test,}age_gfn for x86 (thanks David).
- Added MGLRU functional and performance tests to access_tracking_perf_test (thanks Axel).
- In v3, an mm would be completely ignored (for aging) if there was a secondary MMU but support for secondary MMU walking was missing. Now, missing secondary MMU walking support simply skips the notifier calls (except for eviction).
- Added a sanity check for that range->lockless and range->on_lock are never both provided for the memslot walk.
For the changes since v2[4], see v3.
Based on latest kvm/next.
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/linux-mm/[email protected]/
[4]: https://lore.kernel.org/kvmarm/[email protected]/
James Houghton (11):
KVM: Add lockless memslot walk to KVM
KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER
mm: Add fast_only bool to test_young and clear_young MMU notifiers
mm: Add has_fast_aging to struct mmu_notifier
KVM: Pass fast_only to kvm_{test_,}age_gfn
KVM: x86: Optimize kvm_{test_,}age_gfn a little bit
KVM: x86: Implement fast_only versions of kvm_{test_,}age_gfn
mm: multi-gen LRU: Have secondary MMUs participate in aging
KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test
Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +- arch/arm64/kvm/mmu.c | 30 +- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 2 + arch/x86/kvm/mmu/mmu.c | 23 +- arch/x86/kvm/mmu/tdp_iter.h | 27 +- arch/x86/kvm/mmu/tdp_mmu.c | 67 ++- include/linux/kvm_host.h | 2 + include/linux/mmu_notifier.h | 67 ++- include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 19 +- mm/damon/vaddr.c | 2 - mm/mmu_notifier.c | 38 +- mm/rmap.c | 9 +- mm/vmscan.c | 148 +++++-- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 369 +++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ virt/kvm/Kconfig | 7 + virt/kvm/kvm_main.c | 73 ++-- 23 files changed, 1194 insertions(+), 165 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c
base-commit: 332d2c1d713e232e163386c35a3ba0c1b90df83f
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
PATCH v6 1/11 KVM: Add lockless memslot walk to KVM
Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing.
Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS.
The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply.
--- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 26 +++++++++++++++++++------- 3 files changed, 23 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +1 -0
@@ -266,6 +266,7 @@ struct kvm_gfn_range {gfn_t end;union kvm_mmu_notifier_arg arg;bool may_block; +bool lockless; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);
virt/kvm/Kconfig +3 -0
@@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool
virt/kvm/kvm_main.c +19 -7
@@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range {on_lock_fn_t on_lock;bool flush_on_ret;bool may_block; +bool lockless; };/*
@@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler)))return r; +/* on_lock will never be called for lockless walks */ +if (WARN_ON_ONCE (range->lockless && !IS_KVM_NULL_FN (range->on_lock))) +return r; + idx =srcu_read_lock (&kvm->srcu);for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) {
@@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start =hva_to_gfn_memslot (hva_start, slot); gfn_range.end =hva_to_gfn_memslot (hva_end +PAGE_SIZE -1 , slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless;if (!r.found_memslot) { r.found_memslot = true; -KVM_MMU_LOCK (kvm); -if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm); - -if (IS_KVM_NULL_FN (range->handler)) - goto mmu_unlock; +if (!range->lockless) { +KVM_MMU_LOCK (kvm); +if (!IS_KVM_NULL_FN (range->on_lock)) + range->on_lock (kvm); + +if (IS_KVM_NULL_FN (range->handler)) + goto mmu_unlock; + } } r.ret |= range->handler (kvm, &gfn_range); }
@@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,kvm_flush_remote_tlbs (kvm); mmu_unlock: -if (r.found_memslot) +if (r.found_memslot && !range->lockless)KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);
@@ -679,6 +687,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };return __kvm_handle_hva_range (kvm, &range).ret;
@@ -697,6 +707,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };return __kvm_handle_hva_range (kvm, &range).ret;
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Re: [PATCH v6 01/11] KVM: Add lockless memslot walk to KVM
Quoted 11 lines · James Houghton
Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing.
Architectures that do their own locking must select
CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS.The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply.
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>
Aside from the cleanup suggestion (which should be in separate patches anyway):
Quoted 99 lines
--- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 26 +++++++++++++++++++------- 3 files changed, 23 insertions(+), 7 deletions(-)include/linux/kvm_host.h +1
-0@@ -266,6 +266,7 @@ struct kvm_gfn_range {gfn_t end;union kvm_mmu_notifier_arg arg;bool may_block; +bool lockless; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);virt/kvm/Kconfig +3
-0@@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER boolvirt/kvm/kvm_main.c +17
-7@@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range {on_lock_fn_t on_lock;bool flush_on_ret;bool may_block; +bool lockless; };/* @@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler)))return r; +/* on_lock will never be called for lockless walks */ +if (WARN_ON_ONCE (range->lockless && !IS_KVM_NULL_FN (range->on_lock))) +return r; + idx =srcu_read_lock (&kvm->srcu);for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) {@@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start =hva_to_gfn_memslot (hva_start, slot); gfn_range.end =hva_to_gfn_memslot (hva_end +PAGE_SIZE -1 , slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless;if (!r.found_memslot) { r.found_memslot = true; -KVM_MMU_LOCK (kvm); -if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm); - -if (IS_KVM_NULL_FN (range->handler)) - goto mmu_unlock; +if (!range->lockless) { +KVM_MMU_LOCK (kvm); +if (!IS_KVM_NULL_FN (range->on_lock)) + range->on_lock (kvm); + +if (IS_KVM_NULL_FN (range->handler)) + goto mmu_unlock; + } } r.ret |= range->handler (kvm, &gfn_range); }@@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,kvm_flush_remote_tlbs (kvm); mmu_unlock: -if (r.found_memslot) +if (r.found_memslot && !range->lockless)KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);@@ -679,6 +687,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };return __kvm_handle_hva_range (kvm, &range).ret;
virt/kvm/kvm_main.c static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn · line 711
Quoted 5 lines
virt/kvm/kvm_main.c +2
-0@@ -697,6 +707,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),
kvm_handle_hva_range{,_no_flush}() have very generic names but they're intimately tied to the "young" notifiers. Whereas __kvm_handle_hva_range() is the truly generic handler function.
This is arguably a pre-existing issue, but adding CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS makes these functions even more intamtely tied to the "young" notifiers.
We could rename kvm_handle_hva_range{,_no_flush}() but I think the cleanest thing to do might be to just drop them entirely and move their contents into their callers (there are only 2 callers of these 3 functions). That will create a little duplication but IMO will make the code easier to read.
And then we can also rename __kvm_handle_hva_range() to kvm_handle_hva_range().
e.g. Something like this as the end result:
virt/kvm/kvm_main.c +40 -48
@@ -590,8 +590,8 @@ static void kvm_null_fn(void)node ; \ node = interval_tree_iter_next(node, start, last)) \ -static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, -const struct kvm_mmu_notifier_range *range) +static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, +const struct kvm_mmu_notifier_range *range) {struct kvm_mmu_notifier_return r = { .ret = false,
@@ -674,48 +674,6 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,return r; } -static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, -unsigned long start, -unsigned long end, -gfn_handler_t handler) -{ -struct kvm *kvm =mmu_notifier_to_kvm (mn); -const struct kvm_mmu_notifier_range range = { - .start = start, - .end = end, - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .flush_on_ret = true, - .may_block = false, - .lockless = -IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), - }; - -return __kvm_handle_hva_range (kvm, &range).ret; -} - -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, -unsigned long start, -unsigned long end, -gfn_handler_t handler, -bool fast_only) -{ -struct kvm *kvm =mmu_notifier_to_kvm (mn); -const struct kvm_mmu_notifier_range range = { - .start = start, - .end = end, - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .flush_on_ret = false, - .may_block = false, - .lockless = -IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), - .arg.fast_only = fast_only, - }; - -return __kvm_handle_hva_range (kvm, &range).ret; -} -void kvm_mmu_invalidate_begin (struct kvm *kvm) {lockdep_assert_held_write (&kvm->mmu_lock);
@@ -808,7 +766,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,* that guest memory has been reclaimed. This needs to be done *after* * dropping mmu_lock, as x86's reclaim path is slooooow. */ -if (__kvm_handle_hva_range (kvm, &hva_range).found_memslot) +if (kvm_handle_hva_range (kvm, &hva_range).found_memslot)kvm_arch_guest_memory_reclaimed (kvm);return 0 ;
@@ -854,7 +812,7 @@ static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn, };bool wake; -__kvm_handle_hva_range (kvm, &hva_range); +kvm_handle_hva_range (kvm, &hva_range);/* Pairs with the increment in range_start(). */ spin_lock (&kvm->mn_invalidate_lock);
@@ -876,6 +834,17 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, unsigned long start, unsigned long end) { +struct kvm *kvm =mmu_notifier_to_kvm (mn); +const struct kvm_mmu_notifier_range range = { + .start = start, + .end = end, + .handler = kvm_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = true, + .may_block = false, + .lockless =IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), + }; +trace_kvm_age_hva (start, end, false);return kvm_handle_hva_range (mn, start, end, kvm_age_gfn);
@@ -887,6 +856,18 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, unsigned long end, bool fast_only) { +struct kvm *kvm =mmu_notifier_to_kvm (mn); +const struct kvm_mmu_notifier_range range = { + .start = start, + .end = end, + .handler = kvm_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = false, + .may_block = false, + .lockless =IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), + .arg.fast_only = fast_only, + }; +trace_kvm_age_hva (start, end, fast_only);/*
@@ -902,8 +883,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,* cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ -return kvm_handle_hva_range_no_flush (mn, start, end, kvm_age_gfn, - fast_only); +return kvm_handle_hva_range (kvm, &range).ret; } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
@@ -911,6 +891,18 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, unsigned long address, bool fast_only) { +struct kvm *kvm =mmu_notifier_to_kvm (mn); +const struct kvm_mmu_notifier_range range = { + .start = address, + .end = address +1 , + .handler = kvm_test_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = false, + .may_block = false, + .lockless =IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), + .arg.fast_only = fast_only, + }; +trace_kvm_test_age_hva (address, fast_only); return kvm_handle_hva_range_no_flush(mn, address, address + 1,
Quoted 6 lines
virt/kvm/kvm_main.c +0
-0};return __kvm_handle_hva_range (kvm, &range).ret;Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Re: [PATCH v6 01/11] KVM: Add lockless memslot walk to KVM
Quoted 11 lines · David Matlack
Quoted 9 lines · James Houghton
Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing.
Architectures that do their own locking must select
CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS.The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply.
Quoted 7 lines
Quoted 2 lines
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>Aside from the cleanup suggestion (which should be in separate patches anyway):
Reviewed-by David Matlack <[email protected]>
Thanks David!
Quoted 114 lines
Quoted 100 lines
--- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 26 +++++++++++++++++++------- 3 files changed, 23 insertions(+), 7 deletions(-)include/linux/kvm_host.h +1
-0@@ -266,6 +266,7 @@ struct kvm_gfn_range {gfn_t end;union kvm_mmu_notifier_arg arg;bool may_block; +bool lockless; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);virt/kvm/Kconfig +3
-0@@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER boolvirt/kvm/kvm_main.c +17
-7@@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range {on_lock_fn_t on_lock;bool flush_on_ret;bool may_block; +bool lockless; };/* @@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler)))return r; +/* on_lock will never be called for lockless walks */ +if (WARN_ON_ONCE (range->lockless && !IS_KVM_NULL_FN (range->on_lock))) +return r; + idx =srcu_read_lock (&kvm->srcu);for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) {@@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start =hva_to_gfn_memslot (hva_start, slot); gfn_range.end =hva_to_gfn_memslot (hva_end +PAGE_SIZE -1 , slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless;if (!r.found_memslot) { r.found_memslot = true; -KVM_MMU_LOCK (kvm); -if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm); - -if (IS_KVM_NULL_FN (range->handler)) - goto mmu_unlock; +if (!range->lockless) { +KVM_MMU_LOCK (kvm); +if (!IS_KVM_NULL_FN (range->on_lock)) + range->on_lock (kvm); + +if (IS_KVM_NULL_FN (range->handler)) + goto mmu_unlock; + } } r.ret |= range->handler (kvm, &gfn_range); }@@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,kvm_flush_remote_tlbs (kvm); mmu_unlock: -if (r.found_memslot) +if (r.found_memslot && !range->lockless)KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);@@ -679,6 +687,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };return __kvm_handle_hva_range (kvm, &range).ret;virt/kvm/kvm_main.c static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn · line 711
Quoted 5 lines
virt/kvm/kvm_main.c +2
-0@@ -697,6 +707,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS),kvm_handle_hva_range{,_no_flush}() have very generic names but they're intimately tied to the "young" notifiers. Whereas
__kvm_handle_hva_range()is the truly generic handler function.This is arguably a pre-existing issue, but adding
CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESSmakes these functions even more intamtely tied to the "young" notifiers.
Quoted 8 lines
We could rename kvm_handle_hva_range{,_no_flush}() but I think the cleanest thing to do might be to just drop them entirely and move their contents into their callers (there are only 2 callers of these 3 functions). That will create a little duplication but IMO will make the code easier to read.
And then we can also rename
__kvm_handle_hva_range()tokvm_handle_hva_range().
Thanks for the suggestion, I think this is a good idea. I'm curious how others feel, as this indeed does duplicate the code some. Perhaps it is better just to rename kvm_handle_hva_range() to kvm_handle_hva_range_age() or something like that, and something similar for _no_flush(). :/
But yeah I think it's fine to just do the manipulation you're suggesting. I'll include it in v7 unless others say not to.
PATCH v6 2/11 KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
In reply to James HoughtonWalk the TDP MMU in an RCU read-side critical section. This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the R/X bits).
If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.
Harvesting age information from the shadow MMU is still done while holding the MMU write lock.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 10 ++++- arch/x86/kvm/mmu/tdp_iter.h | 27 +++++++------ arch/x86/kvm/mmu/tdp_mmu.c | 67 +++++++++++++++++++++++++-------- 5 files changed, 77 insertions(+), 29 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1456,6 +1456,7 @@ struct kvm_arch {* tdp_mmu_page set. * * For reads, this list is protected by: +* RCU alone or * the MMU lock in read mode + RCU or * the MMU lock in write mode *
arch/x86/kvm/Kconfig +1 -0
@@ -23,6 +23,7 @@ config KVM depends on X86_LOCAL_APIC select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +8 -2
@@ -1633,8 +1633,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range);
@@ -1646,8 +1649,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range);
arch/x86/kvm/mmu/tdp_iter.h +15 -12
@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)return xchg (rcu_dereference (sptep), new_spte); } +static inline u64 tdp_mmu_clear_spte_bits_atomic (tdp_ptep_t sptep,u64 mask) +{ +atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference (sptep); + +return (u64 )atomic64_fetch_and (~mask, sptep_atomic); +} +static inline void __kvm_tdp_mmu_write_spte (tdp_ptep_t sptep,u64 new_spte) {KVM_MMU_WARN_ON (is_ept_ve_possible (new_spte));
@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) }/* -* SPTEs must be modified atomically if they are shadow-present, leaf -* SPTEs, and have volatile bits, i.e. has bits that can be set outside -* of mmu_lock. The Writable bit can be set by KVM's fast page fault -* handler, and Accessed and Dirty bits can be set by the CPU. +* SPTEs must be modified atomically if they have bits that can be set outside +* of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the +* Writable bit can be set by KVM's fast page fault handler, the Accessed and +* Dirty bits can be set by the CPU, and the Accessed and R/X bits can be +* cleared by age_gfn_range. * * Note, non-leaf SPTEs do have Accessed bits and those bits are * technically volatile, but KVM doesn't consume the Accessed bit of
@@ -46,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)static inline bool kvm_tdp_mmu_spte_need_atomic_write (u64 old_spte,int level) {return is_shadow_present_pte (old_spte) && -is_last_spte (old_spte, level) && -spte_has_volatile_bits (old_spte); +is_last_spte (old_spte, level); } static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
@@ -63,12 +70,8 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,static inline u64 tdp_mmu_clear_spte_bits (tdp_ptep_t sptep,u64 old_spte,u64 mask,int level) { -atomic64_t *sptep_atomic; - -if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) { - sptep_atomic = (atomic64_t *)rcu_dereference (sptep); -return (u64 )atomic64_fetch_and (~mask, sptep_atomic); - } +if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) +return tdp_mmu_clear_spte_bits_atomic (sptep, mask);__kvm_tdp_mmu_write_spte (sptep, old_spte & ~mask);return old_spte;
arch/x86/kvm/mmu/tdp_mmu.c +52 -15
112 lines not on this page · raw message
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 11 lines · James Houghton
Walk the TDP MMU in an RCU read-side critical section. This requires a way to do RCU-safe walking of the
tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, andkvm_tdp_mmu_spte_need_atomic_write()has been updated to account for the fact thatkvm_age_gfncan now lockless update the accessed bit and the R/X bits).If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.
Quoted 5 lines
Harvesting age information from the shadow MMU is still done while holding the MMU write lock.
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>
Aside from the comment fixes below,
Quoted 8 lines
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 10 ++++- arch/x86/kvm/mmu/tdp_iter.h | 27 +++++++------ arch/x86/kvm/mmu/tdp_mmu.c | 67 +++++++++++++++++++++++++-------- 5 files changed, 77 insertions(+), 29 deletions(-)
[...]
Quoted 18 lines
arch/x86/kvm/mmu/tdp_iter.h +7
-0@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)return xchg (rcu_dereference (sptep), new_spte); } +static inline u64 tdp_mmu_clear_spte_bits_atomic (tdp_ptep_t sptep,u64 mask) +{ +atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference (sptep); + +return (u64 )atomic64_fetch_and (~mask, sptep_atomic); +} +static inline void __kvm_tdp_mmu_write_spte (tdp_ptep_t sptep,u64 new_spte) {KVM_MMU_WARN_ON (is_ept_ve_possible (new_spte));@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) }
arch/x86/kvm/mmu/tdp_iter.h static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) · line 45
Quoted 10 lines
arch/x86/kvm/mmu/tdp_iter.h +4
-4/* -* SPTEs must be modified atomically if they are shadow-present, leaf -* SPTEs, and have volatile bits, i.e. has bits that can be set outside -* of mmu_lock. The Writable bit can be set by KVM's fast page fault -* handler, and Accessed and Dirty bits can be set by the CPU. +* SPTEs must be modified atomically if they have bits that can be set outside +* of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the +* Writable bit can be set by KVM's fast page fault handler, the Accessed and +* Dirty bits can be set by the CPU, and the Accessed and R/X bits can be
"R/X bits" should be "W/R/X bits".
arch/x86/kvm/mmu/tdp_iter.h static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) · line 46
Quoted 1 line
arch/x86/kvm/mmu/tdp_iter.h +1
-0+* cleared by age_gfn_range.
nit: "age_gfn_range()"
Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 16 lines · David Matlack
Quoted 14 lines · James Houghton
Walk the TDP MMU in an RCU read-side critical section. This requires a way to do RCU-safe walking of the
tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, andkvm_tdp_mmu_spte_need_atomic_write()has been updated to account for the fact thatkvm_age_gfncan now lockless update the accessed bit and the R/X bits).If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.
Harvesting age information from the shadow MMU is still done while holding the MMU write lock.
Quoted 6 lines
Quoted 2 lines
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>Aside from the comment fixes below,
Reviewed-by David Matlack <[email protected]>
Thank you!
Quoted 29 lines
Quoted 9 lines
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 10 ++++- arch/x86/kvm/mmu/tdp_iter.h | 27 +++++++------ arch/x86/kvm/mmu/tdp_mmu.c | 67 +++++++++++++++++++++++++-------- 5 files changed, 77 insertions(+), 29 deletions(-)[...]
Quoted 19 lines
arch/x86/kvm/mmu/tdp_iter.h +7
-0@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)return xchg (rcu_dereference (sptep), new_spte); } +static inline u64 tdp_mmu_clear_spte_bits_atomic (tdp_ptep_t sptep,u64 mask) +{ +atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference (sptep); + +return (u64 )atomic64_fetch_and (~mask, sptep_atomic); +} +static inline void __kvm_tdp_mmu_write_spte (tdp_ptep_t sptep,u64 new_spte) {KVM_MMU_WARN_ON (is_ept_ve_possible (new_spte));@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) }
Quoted 11 lines
arch/x86/kvm/mmu/tdp_iter.h static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) · line 45
Quoted 9 lines
arch/x86/kvm/mmu/tdp_iter.h +4
-4/* -* SPTEs must be modified atomically if they are shadow-present, leaf -* SPTEs, and have volatile bits, i.e. has bits that can be set outside -* of mmu_lock. The Writable bit can be set by KVM's fast page fault -* handler, and Accessed and Dirty bits can be set by the CPU. +* SPTEs must be modified atomically if they have bits that can be set outside +* of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the +* Writable bit can be set by KVM's fast page fault handler, the Accessed and +* Dirty bits can be set by the CPU, and the Accessed and R/X bits can be "R/X bits" should be "W/R/X bits".
Thanks. Right, we are clearing all of VMX_EPT_RWX_MASK.
Quoted 4 lines
arch/x86/kvm/mmu/tdp_iter.h static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) · line 46
Quoted 2 lines
arch/x86/kvm/mmu/tdp_iter.h +1
-0+* cleared by age_gfn_range. nit: "
age_gfn_range()"
Thanks, will fix this and all the other places where I've left off the ().
Sean Christopherson
maintainerRe: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
In reply to James HoughtonQuoted 1 line · James Houghton
Walk the TDP MMU in an RCU read-side critical section.
...without holding mmu_lock, while doing xxx. There are a lot of TDP MMU walks, pand they all need RCU protection.
Quoted 8 lines
This requires a way to do RCU-safe walking of the
tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, andkvm_tdp_mmu_spte_need_atomic_write()has been updated to account for the fact thatkvm_age_gfncan now lockless update the accessed bit and the R/X bits).If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.
Please avoid pronouns. E.g. s/we/KVM (and adjust grammar as needed), so that it's clear what actor in particular is doing the retry.
Quoted 131 lines
Harvesting age information from the shadow MMU is still done while holding the MMU write lock.
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 10 ++++- arch/x86/kvm/mmu/tdp_iter.h | 27 +++++++------ arch/x86/kvm/mmu/tdp_mmu.c | 67 +++++++++++++++++++++++++-------- 5 files changed, 77 insertions(+), 29 deletions(-)arch/x86/include/asm/kvm_host.h +1
-0@@ -1456,6 +1456,7 @@ struct kvm_arch {* tdp_mmu_page set. * * For reads, this list is protected by: +* RCU alone or * the MMU lock in read mode + RCU or * the MMU lock in write mode * arch/x86/kvm/Kconfig +1
-0@@ -23,6 +23,7 @@ config KVM depends on X86_LOCAL_APIC select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSOarch/x86/kvm/mmu/mmu.c +8
-2@@ -1633,8 +1633,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range);@@ -1646,8 +1649,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range);arch/x86/kvm/mmu/tdp_iter.h +15
-12@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)return xchg (rcu_dereference (sptep), new_spte); } +static inline u64 tdp_mmu_clear_spte_bits_atomic (tdp_ptep_t sptep,u64 mask) +{ +atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference (sptep); + +return (u64 )atomic64_fetch_and (~mask, sptep_atomic); +} +static inline void __kvm_tdp_mmu_write_spte (tdp_ptep_t sptep,u64 new_spte) {KVM_MMU_WARN_ON (is_ept_ve_possible (new_spte));@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) }/* -* SPTEs must be modified atomically if they are shadow-present, leaf -* SPTEs, and have volatile bits, i.e. has bits that can be set outside -* of mmu_lock. The Writable bit can be set by KVM's fast page fault -* handler, and Accessed and Dirty bits can be set by the CPU. +* SPTEs must be modified atomically if they have bits that can be set outside +* of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the +* Writable bit can be set by KVM's fast page fault handler, the Accessed and +* Dirty bits can be set by the CPU, and the Accessed and R/X bits can be +* cleared by age_gfn_range. * * Note, non-leaf SPTEs do have Accessed bits and those bits are * technically volatile, but KVM doesn't consume the Accessed bit of @@ -46,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)static inline bool kvm_tdp_mmu_spte_need_atomic_write (u64 old_spte,int level) {return is_shadow_present_pte (old_spte) && -is_last_spte (old_spte, level) && -spte_has_volatile_bits (old_spte); +is_last_spte (old_spte, level); } static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,@@ -63,12 +70,8 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,static inline u64 tdp_mmu_clear_spte_bits (tdp_ptep_t sptep,u64 old_spte,u64 mask,int level) { -atomic64_t *sptep_atomic; - -if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) { - sptep_atomic = (atomic64_t *)rcu_dereference (sptep); -return (u64 )atomic64_fetch_and (~mask, sptep_atomic); - } +if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) +return tdp_mmu_clear_spte_bits_atomic (sptep, mask);__kvm_tdp_mmu_write_spte (sptep, old_spte & ~mask);return old_spte;arch/x86/kvm/mmu/tdp_mmu.c +0
-0
arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm, · line 36
Quoted 8 lines
arch/x86/kvm/mmu/tdp_mmu.c +5
-0@@ -29,6 +29,11 @@ static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,return true; } +static __always_inline bool kvm_lockdep_assert_rcu_read_lock_held(void) +{ +WARN_ON_ONCE (!rcu_read_lock_held ()); +return true; +}
I doubt KVM needs a manual WARN, the RCU deference stuff should yell loudly if something is missing an rcu_read_lock().
arch/x86/kvm/mmu/tdp_mmu.c static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, · line 190
Quoted 11 lines
arch/x86/kvm/mmu/tdp_mmu.c +5
-0void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm) {@@ -178,6 +183,15 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, ((_only_valid) && (_root)->role.invalid))) { \ } else +/* +* Iterate over all TDP MMU roots in an RCU read-side critical section. +*/ +#define for_each_tdp_mmu_root_rcu (_kvm, _root, _as_id) \ + list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link) \
This should just process valid roots:
Quoted 8 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+if (kvm_lockdep_assert_rcu_read_lock_held () && \ + (_as_id >=0 &&kvm_mmu_page_as_id (_root) != _as_id)) { \ + } else +#define for_each_tdp_mmu_root (_kvm, _root, _as_id) \ __for_each_tdp_mmu_root(_kvm, _root, _as_id, false)
arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1244
Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0@@ -1224,6 +1238,27 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm,return ret; } +static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless( +struct kvm *kvm, +struct kvm_gfn_range *range, +tdp_handler_t handler)
Please burn all the Google3 from your brain, and code ;-)
Quoted 40 lines
arch/x86/kvm/mmu/tdp_mmu.c +19
-7+struct kvm_mmu_page *root; +struct tdp_iter iter; +bool ret = false; + +rcu_read_lock (); + +for_each_tdp_mmu_root_rcu (kvm, root, range->slot->as_id) { +tdp_root_for_each_leaf_pte (iter, root, range->start, range->end) + ret |=handler (kvm, &iter, range); + } + +rcu_read_unlock (); + +return ret; +} +/* * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero * if any of the GFNs in the range have been accessed. @@ -1237,28 +1272,30 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, {u64 new_spte; +retry:/* If we have a non-accessed entry we don't need to change the pte. */ if (!is_accessed_spte (iter->old_spte))return false;if (spte_ad_enabled (iter->old_spte)) { - iter->old_spte =tdp_mmu_clear_spte_bits (iter->sptep, - iter->old_spte, - shadow_accessed_mask, - iter->level); + iter->old_spte =tdp_mmu_clear_spte_bits_atomic (iter->sptep, + shadow_accessed_mask); new_spte = iter->old_spte & ~shadow_accessed_mask; }else { -/* -* Capture the dirty status of the page, so that it doesn't get -* lost when the SPTE is marked for access tracking.
arch/x86/kvm/mmu/tdp_mmu.c static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, · line 1293
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +9
-1-*/ + new_spte =mark_spte_for_access_track (iter->old_spte); +if (__tdp_mmu_set_spte_atomic (iter, new_spte)) { +/* +* The cmpxchg failed. If the spte is still a +* last-level spte, we can safely retry. +*/ +if (is_shadow_present_pte (iter->old_spte) && +is_last_spte (iter->old_spte, iter->level)) + goto retry;
Do we have a feel for how often conflicts actually happen? I.e. is it worth retrying and having to worry about infinite loops, however improbable they may be?
Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 6 lines · Sean Christopherson
Quoted 1 line · James Houghton
Walk the TDP MMU in an RCU read-side critical section.
...without holding
mmu_lock, while doing xxx. There are a lot of TDP MMU walks, pand they all need RCU protection.
Added "without holding mmu_lock when harvesting and potentially updating age information on sptes".
Quoted 11 lines
Quoted 8 lines
This requires a way to do RCU-safe walking of the
tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, andkvm_tdp_mmu_spte_need_atomic_write()has been updated to account for the fact thatkvm_age_gfncan now lockless update the accessed bit and the R/X bits).If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.
Please avoid pronouns. E.g. s/we/KVM (and adjust grammar as needed), so that it's clear what actor in particular is doing the retry.
Fixed. Though, I have also changed this to reflect the change in the retry logic I've made, given your other comment.
Quoted 132 lines
Quoted 132 lines
Harvesting age information from the shadow MMU is still done while holding the MMU write lock.
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 10 ++++- arch/x86/kvm/mmu/tdp_iter.h | 27 +++++++------ arch/x86/kvm/mmu/tdp_mmu.c | 67 +++++++++++++++++++++++++-------- 5 files changed, 77 insertions(+), 29 deletions(-)arch/x86/include/asm/kvm_host.h +1
-0@@ -1456,6 +1456,7 @@ struct kvm_arch {* tdp_mmu_page set. * * For reads, this list is protected by: +* RCU alone or * the MMU lock in read mode + RCU or * the MMU lock in write mode * arch/x86/kvm/Kconfig +1
-0@@ -23,6 +23,7 @@ config KVM depends on X86_LOCAL_APIC select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSOarch/x86/kvm/mmu/mmu.c +8
-2@@ -1633,8 +1633,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range);@@ -1646,8 +1649,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range);arch/x86/kvm/mmu/tdp_iter.h +15
-12@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)return xchg (rcu_dereference (sptep), new_spte); } +static inline u64 tdp_mmu_clear_spte_bits_atomic (tdp_ptep_t sptep,u64 mask) +{ +atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference (sptep); + +return (u64 )atomic64_fetch_and (~mask, sptep_atomic); +} +static inline void __kvm_tdp_mmu_write_spte (tdp_ptep_t sptep,u64 new_spte) {KVM_MMU_WARN_ON (is_ept_ve_possible (new_spte));@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) }/* -* SPTEs must be modified atomically if they are shadow-present, leaf -* SPTEs, and have volatile bits, i.e. has bits that can be set outside -* of mmu_lock. The Writable bit can be set by KVM's fast page fault -* handler, and Accessed and Dirty bits can be set by the CPU. +* SPTEs must be modified atomically if they have bits that can be set outside +* of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the +* Writable bit can be set by KVM's fast page fault handler, the Accessed and +* Dirty bits can be set by the CPU, and the Accessed and R/X bits can be +* cleared by age_gfn_range. * * Note, non-leaf SPTEs do have Accessed bits and those bits are * technically volatile, but KVM doesn't consume the Accessed bit of @@ -46,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)static inline bool kvm_tdp_mmu_spte_need_atomic_write (u64 old_spte,int level) {return is_shadow_present_pte (old_spte) && -is_last_spte (old_spte, level) && -spte_has_volatile_bits (old_spte); +is_last_spte (old_spte, level); } static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,@@ -63,12 +70,8 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,static inline u64 tdp_mmu_clear_spte_bits (tdp_ptep_t sptep,u64 old_spte,u64 mask,int level) { -atomic64_t *sptep_atomic; - -if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) { - sptep_atomic = (atomic64_t *)rcu_dereference (sptep); -return (u64 )atomic64_fetch_and (~mask, sptep_atomic); - } +if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) +return tdp_mmu_clear_spte_bits_atomic (sptep, mask);__kvm_tdp_mmu_write_spte (sptep, old_spte & ~mask);return old_spte;arch/x86/kvm/mmu/tdp_mmu.c +0
-0@@ -29,6 +29,11 @@ static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm, · line 36
Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +5
-0return true; } +static __always_inline bool kvm_lockdep_assert_rcu_read_lock_held(void) +{ +WARN_ON_ONCE (!rcu_read_lock_held ()); +return true; +}I doubt KVM needs a manual WARN, the RCU deference stuff should yell loudly if something is missing an
rcu_read_lock().
You're right -- removed.
Quoted 14 lines
arch/x86/kvm/mmu/tdp_mmu.c static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, · line 190
Quoted 11 lines
arch/x86/kvm/mmu/tdp_mmu.c +5
-0void kvm_mmu_uninit_tdp_mmu (struct kvm *kvm) {@@ -178,6 +183,15 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, ((_only_valid) && (_root)->role.invalid))) { \ } else +/* +* Iterate over all TDP MMU roots in an RCU read-side critical section. +*/ +#define for_each_tdp_mmu_root_rcu (_kvm, _root, _as_id) \ + list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link) \This should just process valid roots:
Quoted 1 line
Thanks! I've added || (_root)->role.invalid) to the below conditional expression, and I've renamed the macro to for_each_valid_tdp_mmu_root_rcu.
Quoted 11 lines
Quoted 11 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+if (kvm_lockdep_assert_rcu_read_lock_held () && \ + (_as_id >=0 &&kvm_mmu_page_as_id (_root) != _as_id)) { \ + } else +#define for_each_tdp_mmu_root (_kvm, _root, _as_id) \ __for_each_tdp_mmu_root(_kvm, _root, _as_id, false)@@ -1224,6 +1238,27 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm,return ret; }
Quoted 6 lines
arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1244
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless( +struct kvm *kvm, +struct kvm_gfn_range *range, +tdp_handler_t handler)Please burn all the Google3 from your brain, and code ;-)
I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.
Perhaps I should put static __always_inline bool on its own line?
Quoted 47 lines
Quoted 47 lines
arch/x86/kvm/mmu/tdp_mmu.c +24
-8+struct kvm_mmu_page *root; +struct tdp_iter iter; +bool ret = false; + +rcu_read_lock (); + +for_each_tdp_mmu_root_rcu (kvm, root, range->slot->as_id) { +tdp_root_for_each_leaf_pte (iter, root, range->start, range->end) + ret |=handler (kvm, &iter, range); + } + +rcu_read_unlock (); + +return ret; +} +/* * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero * if any of the GFNs in the range have been accessed. @@ -1237,28 +1272,30 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, {u64 new_spte; +retry:/* If we have a non-accessed entry we don't need to change the pte. */ if (!is_accessed_spte (iter->old_spte))return false;if (spte_ad_enabled (iter->old_spte)) { - iter->old_spte =tdp_mmu_clear_spte_bits (iter->sptep, - iter->old_spte, - shadow_accessed_mask, - iter->level); + iter->old_spte =tdp_mmu_clear_spte_bits_atomic (iter->sptep, + shadow_accessed_mask); new_spte = iter->old_spte & ~shadow_accessed_mask; }else { -/* -* Capture the dirty status of the page, so that it doesn't get -* lost when the SPTE is marked for access tracking. -*/ + new_spte =mark_spte_for_access_track (iter->old_spte); +if (__tdp_mmu_set_spte_atomic (iter, new_spte)) { +/* +* The cmpxchg failed. If the spte is still a +* last-level spte, we can safely retry.
Quoted 8 lines
arch/x86/kvm/mmu/tdp_mmu.c static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, · line 1293
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+*/ +if (is_shadow_present_pte (iter->old_spte) && +is_last_spte (iter->old_spte, iter->level)) + goto retry;Do we have a feel for how often conflicts actually happen? I.e. is it worth retrying and having to worry about infinite loops, however improbable they may be?
I'm not sure how common this is. I think it's probably better not to retry actually. If the cmpxchg fails, this spte is probably young anyway, so I can just return true instead of potentially retrying. This is all best-effort anyway.
Sean Christopherson
maintainerRe: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 6 lines · James Houghton
Quoted 5 lines · Sean Christopherson
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless( +struct kvm *kvm, +struct kvm_gfn_range *range, +tdp_handler_t handler)
Quoted 6 lines
Quoted 1 line
Please burn all the Google3 from your brain, and code ;-)
I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.
Perhaps I should put
static __always_inline boolon its own line?
Noooo. Do not wrap before the function name. Linus has a nice explanation/rant on this[1].
In this case, I'm pretty sure you can avoid the helper and simply handle all aging paths in a single API, e.g. similar to what I proposed for the shadow MMU[2].
[1] https://lore.kernel.org/all/CAHk-=wjoLAYG446ZNHfg=GhjSY6nFmuB_wA8fYd5iLBNXjo9Bw@mail.gmail.com
[2] https://lore.kernel.org/all/[email protected]
Quoted 39 lines
Quoted 30 lines
Quoted 30 lines
/*
* Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero
* if any of the GFNs in the range have been accessed.arch/x86/kvm/mmu/tdp_mmu.c +8
-8@@ -1237,28 +1272,30 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, {u64 new_spte; +retry:/* If we have a non-accessed entry we don't need to change the pte. */ if (!is_accessed_spte (iter->old_spte))return false;if (spte_ad_enabled (iter->old_spte)) { - iter->old_spte =tdp_mmu_clear_spte_bits (iter->sptep, - iter->old_spte, - shadow_accessed_mask, - iter->level); + iter->old_spte =tdp_mmu_clear_spte_bits_atomic (iter->sptep, + shadow_accessed_mask); new_spte = iter->old_spte & ~shadow_accessed_mask; }else { -/* -* Capture the dirty status of the page, so that it doesn't get -* lost when the SPTE is marked for access tracking. -*/ + new_spte =mark_spte_for_access_track (iter->old_spte); +if (__tdp_mmu_set_spte_atomic (iter, new_spte)) { +/* +* The cmpxchg failed. If the spte is still a +* last-level spte, we can safely retry. Quoted 8 lines
arch/x86/kvm/mmu/tdp_mmu.c static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, · line 1293
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+*/ +if (is_shadow_present_pte (iter->old_spte) && +is_last_spte (iter->old_spte, iter->level)) + goto retry;Do we have a feel for how often conflicts actually happen? I.e. is it worth retrying and having to worry about infinite loops, however improbable they may be?
Quoted 4 lines
I'm not sure how common this is. I think it's probably better not to retry actually. If the cmpxchg fails, this spte is probably young anyway, so I can just
return trueinstead of potentially retrying. This is all best-effort anyway.
+1
Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 9 lines · Sean Christopherson
Quoted 8 lines · James Houghton
Quoted 6 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1244
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless( +struct kvm *kvm, +struct kvm_gfn_range *range, +tdp_handler_t handler)Please burn all the Google3 from your brain, and code ;-)
Quoted 7 lines
Quoted 4 lines
I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.
Perhaps I should put
static __always_inline boolon its own line?Noooo. Do not wrap before the function name. Linus has a nice explanation/rant on this[1].
IMHO, run clang-format on your stuff and just be happy with 99% of what it spits out. Saves *so much time* and usually arguing..
clang-format will occasionally decide to wrap in the GNU way, if it can put the arguments all on one line. People will never agree on small details of style, but it would be really nice if we can at least agree not to nitpick clang-format's decisions :) :)
Jason
Sean Christopherson
maintainerRe: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Sean Christopherson
Quoted 11 lines · James Houghton
Quoted 6 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, · line 1244
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless( +struct kvm *kvm, +struct kvm_gfn_range *range, +tdp_handler_t handler)Please burn all the Google3 from your brain, and code ;-)
I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Perhaps I should put
static __always_inline boolon its own line?Noooo. Do not wrap before the function name. Linus has a nice explanation/rant on this[1].
IMHO, run clang-format on your stuff and just be happy with 99% of what it spits out. Saves *so much time* and usually arguing..
Heh, nope, not bending on this one. The time I spend far hunting for implementations because of wraps before the function name far exceeds the time it takes me to push back on these warts in review.
Re: [PATCH v6 02/11] KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 19 lines · Sean Christopherson
Quoted 18 lines · Jason Gunthorpe
Quoted 16 lines · Sean Christopherson
Quoted 6 lines · James Houghton
Quoted 5 lines · Sean Christopherson
Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0+static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless( +struct kvm *kvm, +struct kvm_gfn_range *range, +tdp_handler_t handler)Quoted 6 lines
Quoted 1 line
Please burn all the Google3 from your brain, and code ;-)
I indented this way to avoid going past the 80 character limit. I've adjusted it to be more like the other functions in this file.
Perhaps I should put
static __always_inline boolon its own line?Noooo. Do not wrap before the function name. Linus has a nice explanation/rant on this[1].
Quoted 6 lines
Quoted 2 lines
IMHO, run clang-format on your stuff and just be happy with 99% of what it spits out. Saves *so much time* and usually arguing..
Heh, nope, not bending on this one. The time I spend far hunting for implementations because of wraps before the function name far exceeds the time it takes me to push back on these warts in review.
clangd solved that problem for me :)
Jason
PATCH v6 3/11 KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
In reply to James HoughtonReplace the MMU write locks (taken in the memslot iteration loop) for read locks.
Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change.
--- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++------ arch/arm64/kvm/mmu.c | 30 ++++++++++++++++++++++-------- 3 files changed, 32 insertions(+), 14 deletions(-)
arch/arm64/kvm/Kconfig +1 -0
@@ -22,6 +22,7 @@ menuconfig KVM select KVM_COMMON select KVM_GENERIC_HARDWARE_ENABLING select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT
arch/arm64/kvm/hyp/pgtable.c +9 -6
@@ -1319,10 +1319,10 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, data->young = true;/* -* stage2_age_walker() is always called while holding the MMU lock for -* write, so this will always succeed. Nonetheless, this deliberately -* follows the race detection pattern of the other stage-2 walkers in -* case the locking mechanics of the MMU notifiers is ever changed. +* This walk is not exclusive; the PTE is permitted to change from +* under us. If there is a race to update this PTE, then the GFN is +* most likely young, so failing to clear the AF is likely to be +* inconsequential. */ if (data->mkold && !stage2_try_set_pte (ctx, new))return -EAGAIN ;
@@ -1345,10 +1345,13 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr,struct kvm_pgtable_walker walker = { .cb = stage2_age_walker, .arg = &data, - .flags =KVM_PGTABLE_WALK_LEAF , + .flags =KVM_PGTABLE_WALK_LEAF | +KVM_PGTABLE_WALK_SHARED , }; +int r; -WARN_ON (kvm_pgtable_walk (pgt, addr, size, &walker)); + r =kvm_pgtable_walk (pgt, addr, size, &walker); +WARN_ON_ONCE (r && r != -EAGAIN );return data.young; }
arch/arm64/kvm/mmu.c +22 -8
@@ -1912,29 +1912,43 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {u64 size = (range->end - range->start) <<PAGE_SHIFT ; +bool young = false; + +read_lock (&kvm->mmu_lock);if (!kvm->arch.mmu.pgt) -return false; + goto out; -return kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, - range->start <<PAGE_SHIFT , - size, true); + young =kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, + range->start <<PAGE_SHIFT , + size, true);/* * TODO: Handle nested_mmu structures here using the reverse mapping in * a later version of patch series. */ + +out: +read_unlock (&kvm->mmu_lock); +return young; }bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {u64 size = (range->end - range->start) <<PAGE_SHIFT ; +bool young = false; + +read_lock (&kvm->mmu_lock);if (!kvm->arch.mmu.pgt) -return false; + goto out; -return kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, - range->start <<PAGE_SHIFT , - size, false); + young =kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, + range->start <<PAGE_SHIFT , + size, false); + +out: +read_unlock (&kvm->mmu_lock); +return young; }phys_addr_t kvm_mmu_get_httbr (void )
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 4 lines · James Houghton
Replace the MMU write locks (taken in the memslot iteration loop) for read locks.
Quoted 8 lines
Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The
stage2_age_walker()callback is safe to race with itself; update the comment to reflect the synchronization change.Signed-off-by James Houghton <[email protected]>---
Here is some data to show that this patch at least *can* be helpful:
# arm64 patched to do aging (i.e., set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY) # The test is faulting memory in while doing aging as fast as possible. # taskset -c 0-32 ./access_tracking_perf_test -l -r /dev/cgroup/memory -p -v 32 -m 3
# Write lock
vcpu wall time : 3.039207157slru_gen avg pass duration : 1.660541541s, (passes:2, total:3.321083083s)
# Read lock
vcpu wall time : 3.010848445slru_gen avg pass duration : 0.306623698s, (passes:11, total:3.372860688s)
Aging is able to run significantly faster, but vCPU runtime isn't affected much (in this test).
It would be really nice to motivate this patch with a test that didn't require patching the kernel... Oliver and Marc, please let me know if you'd like to see more data. I'm also happy to simply drop this patch.
Sean Christopherson
maintainerRe: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 25 lines · James Houghton
Quoted 4 lines · James Houghton
Replace the MMU write locks (taken in the memslot iteration loop) for read locks.
Quoted 8 lines
Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The
stage2_age_walker()callback is safe to race with itself; update the comment to reflect the synchronization change.Signed-off-by James Houghton <[email protected]>---
Here is some data to show that this patch at least *can* be helpful:
# arm64 patched to do aging (i.e., set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY) # The test is faulting memory in while doing aging as fast as possible. # taskset -c 0-32 ./access_tracking_perf_test -l -r /dev/cgroup/memory -p -v 32 -m 3# Write lock
vcpu wall time : 3.039207157slru_genavg pass duration : 1.660541541s, (passes:2, total:3.321083083s)
Quoted 6 lines
# Read lock
vcpu wall time : 3.010848445slru_genavg pass duration : 0.306623698s, (passes:11, total:3.372860688s)Aging is able to run significantly faster, but vCPU runtime isn't affected much (in this test).
Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking mmap_lock for read when handling faults:
Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 31 lines · Sean Christopherson
Quoted 29 lines · James Houghton
Quoted 4 lines · James Houghton
Replace the MMU write locks (taken in the memslot iteration loop) for read locks.
Quoted 8 lines
Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The
stage2_age_walker()callback is safe to race with itself; update the comment to reflect the synchronization change.Signed-off-by James Houghton <[email protected]>---
Here is some data to show that this patch at least *can* be helpful:
# arm64 patched to do aging (i.e., set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY) # The test is faulting memory in while doing aging as fast as possible. # taskset -c 0-32 ./access_tracking_perf_test -l -r /dev/cgroup/memory -p -v 32 -m 3# Write lock
vcpu wall time : 3.039207157slru_genavg pass duration : 1.660541541s, (passes:2, total:3.321083083s)# Read lock
vcpu wall time : 3.010848445slru_genavg pass duration : 0.306623698s, (passes:11, total:3.372860688s)
Quoted 7 lines
Quoted 2 lines
Aging is able to run significantly faster, but vCPU runtime isn't affected much (in this test).
Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking
mmap_lockfor read when handling faults:
For the above test, I don't think it's mmap_lock -- the reclaim path, e.g., when zswapping guest memory, has two stages: aging (scanning PTEs) and eviction (unmapping PTEs). Only testing the former isn't realistic at all. IOW, for a r/w lock use case, only testing the read lock path would be bad coverage.
Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 1 line · Yu Zhao
On Fri, Aug 16, 2024 at 6:46 PM Sean Christopherson <[email protected]> wrote:
[...]
Quoted 6 lines
Quoted 4 lines
Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking
mmap_lockfor read when handling faults:For the above test, I don't think it's
mmap_lock
Yeah, I don't think this is related to the mmap_lock.
James is likely using hardware that has FEAT_HAFDBS, so vCPUs won't fault for an Access flag update. Even if he's on a machine w/o it, Access flag faults are handled outside the mmap_lock.
Forcing SW management of the AF at stage-2 would be the best case for demonstrating the locking improvement:
arch/arm64/kvm/hyp/pgtable.c +0 -2
@@ -669,8 +669,6 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift)* happen to be running on a design that has unadvertised support for * HAFDBS. Here be dragons. */ -if (!cpus_have_final_cap (ARM64_WORKAROUND_AMPERE_AC03_CPU_38 )) - vtcr |=VTCR_EL2_HA ;#endif /* CONFIG_ARM64_HW_AFDBM */ if (kvm_lpa2_is_enabled ())
Changing the config option would work too, but I wasn't sure if FEAT_HAFDBS on the primary MMU influenced MGLRU heuristics.
Quoted 4 lines
-- the reclaim path,
e.g., when zswapping guest memory, has two stages: aging (scanning PTEs) and eviction (unmapping PTEs). Only testing the former isn't realistic at all.
AIUI, the intention of this test data is to provide some justification for why Marc + I should consider the locking change *outside* of any MMU notifier changes. So from that POV, this is meant as a hacked up microbenchmark and not meant to be realistic.
And really, the arm64 change has nothing to do with this series at this point, which is disappointing. In the interest of moving this feature along for both architectures, would you be able help James with:
- Identifying a benchmark that you believe is realistic
- Suggestions on how to run that benchmark on Google infrastructure
Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY as well.
Signature · 2 lines
Thanks, Oliver
Sean Christopherson
maintainerRe: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 8 lines · Oliver Upton
Quoted 1 line · Yu Zhao
On Fri, Aug 16, 2024 at 6:46 PM Sean Christopherson <[email protected]> wrote:
[...]
Quoted 3 lines
Quoted 3 lines
Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking
mmap_lockfor read when handling faults:
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
For the above test, I don't think it's
mmap_lockYeah, I don't think this is related to the
mmap_lock.James is likely using hardware that has
FEAT_HAFDBS, so vCPUs won't fault for an Access flag update.
Huh, didn't know that was a thing on ARM. Ooh, that lends even more credence to my assertion that marking folios accessed in handle_access_fault() can go away[*]. I assume hardware-assisted updates means this code in handle_access_fault() will no longer be hit, as KVM simply won't ever get access faults? If so, I'll add that info to the changelog.
if (kvm_pte_valid(pte)) kvm_set_pfn_accessed(kvm_pte_to_pfn(pte));
[*] https://lore.kernel.org/all/[email protected]
Quoted 2 lines
Even if he's on a machine w/o it, Access flag faults are handled outside the
mmap_lock.
Oh, right, they go down handle_access_fault(), not user_mem_abort().
Reviewing late Friday afternoon, never a good idea ;-)
Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
In reply to Oliver UptonQuoted 8 lines · Oliver Upton
Quoted 1 line · Yu Zhao
On Fri, Aug 16, 2024 at 6:46 PM Sean Christopherson <[email protected]> wrote:
[...]
Quoted 2 lines
Quoted 2 lines
Were you expecting vCPU runtime to improve (more)? If so, lack of movement could be due to KVM arm64 taking
mmap_lockfor read when handling faults:
I had no real expectation. I was hoping that maybe there could be a vCPU runtime improvement, given that user_mem_abort() (being called because we're faulting memory in continuously in this test) has to take the KVM MMU lock for reading, and aging is taking it for reading vs. writing. I think that's why aging is a lot slower when using the write lock: it is waiting for the readers to drop the lock, but I guess the delay on the *readers* due to the pending writer seems to be pretty minimal.
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
For the above test, I don't think it's
mmap_lockYeah, I don't think this is related to the
mmap_lock.James is likely using hardware that has
FEAT_HAFDBS, so vCPUs won't fault for an Access flag update. Even if he's on a machine w/o it, Access flag faults are handled outside themmap_lock.
Yeah I was running on Ampere Altra CPUs.
Quoted 8 lines
Forcing SW management of the AF at stage-2 would be the best case for demonstrating the locking improvement:
arch/arm64/kvm/hyp/pgtable.c +0
-0
arch/arm64/kvm/hyp/pgtable.c u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) · line 674
Quoted 8 lines
arch/arm64/kvm/hyp/pgtable.c +0
-2@@ -669,8 +669,6 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift)* happen to be running on a design that has unadvertised support for * HAFDBS. Here be dragons. */ -if (!cpus_have_final_cap (ARM64_WORKAROUND_AMPERE_AC03_CPU_38 )) - vtcr |=VTCR_EL2_HA ;#endif /* CONFIG_ARM64_HW_AFDBM */ if (kvm_lpa2_is_enabled ())
Thanks!
Quoted 2 lines
Changing the config option would work too, but I wasn't sure if
FEAT_HAFDBSon the primary MMU influenced MGLRU heuristics.
Indeed, disabling CONFIG_ARM64_HW_AFDBM will cause MGLRU not to do aging.
Quoted 15 lines
Quoted 4 lines
-- the reclaim path,
e.g., when zswapping guest memory, has two stages: aging (scanning PTEs) and eviction (unmapping PTEs). Only testing the former isn't realistic at all.AIUI, the intention of this test data is to provide some justification for why Marc + I should consider the locking change *outside* of any MMU notifier changes. So from that POV, this is meant as a hacked up microbenchmark and not meant to be realistic.
And really, the arm64 change has nothing to do with this series at this point, which is disappointing. In the interest of moving this feature along for both architectures, would you be able help James with:
Quoted 7 lines
- Identifying a benchmark that you believe is realistic
- Suggestions on how to run that benchmark on Google infrastructure
Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to
HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLYas well.
I'll keep trying some other approaches I can take for getting similar testing that Yu had; it is somewhat difficult for me to reproduce those tests (and it really shouldn't be.... sorry).
I think it makes most sense for me to drop the arm64 patch for now and re-propose it (or something stronger) alongside enabling aging. Does that sound ok?
Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 8 lines · James Houghton
Quoted 3 lines · Oliver Upton
Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to
HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLYas well.I'll keep trying some other approaches I can take for getting similar testing that Yu had; it is somewhat difficult for me to reproduce those tests (and it really shouldn't be.... sorry).
No need to apologize. Getting good test hardware for arm64 is a complete chore. Sure would love a functional workstation with cores from this decade...
Quoted 3 lines
I think it makes most sense for me to drop the arm64 patch for now and re-propose it (or something stronger) alongside enabling aging. Does that sound ok?
I'm a bit disappointed that we haven't gotten forward progress on the arm64 patches, but I also recognize this is the direction of travel as the x86 patches are shaping up.
So yeah, I'm OK with it, but I'd love to get the arm64 side sorted out soon while the context is still fresh.
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 19 lines · Oliver Upton
Quoted 8 lines · James Houghton
Quoted 3 lines · Oliver Upton
Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to
HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLYas well.I'll keep trying some other approaches I can take for getting similar testing that Yu had; it is somewhat difficult for me to reproduce those tests (and it really shouldn't be.... sorry).
No need to apologize. Getting good test hardware for arm64 is a complete chore. Sure would love a functional workstation with cores from this decade...
Quoted 3 lines
I think it makes most sense for me to drop the arm64 patch for now and re-propose it (or something stronger) alongside enabling aging. Does that sound ok?
Quoted 6 lines
I'm a bit disappointed that we haven't gotten forward progress on the arm64 patches, but I also recognize this is the direction of travel as the x86 patches are shaping up.
So yeah, I'm OK with it, but I'd love to get the arm64 side sorted out soon while the context is still fresh.
Converting the aging notifiers to holding mmu_lock for read seems like a pure win and minimal churn. Can we keep that patch in v7 (which depends on the lockless notifier refactor, i.e. is not completely stand-alone)? We can revisit enabling MGLRU on arm64 in a subsequent series.
Quoted 4 lines
--
Thanks,
Oliver
Re: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Hey David,
Quoted 24 lines · David Matlack
Quoted 23 lines · Oliver Upton
Quoted 8 lines · James Houghton
Quoted 3 lines · Oliver Upton
Asking since you had a setup / data earlier on when you were carrying the series. Hopefully with supportive data we can get arm64 to opt-in to
HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLYas well.I'll keep trying some other approaches I can take for getting similar testing that Yu had; it is somewhat difficult for me to reproduce those tests (and it really shouldn't be.... sorry).
No need to apologize. Getting good test hardware for arm64 is a complete chore. Sure would love a functional workstation with cores from this decade...
Quoted 3 lines
I think it makes most sense for me to drop the arm64 patch for now and re-propose it (or something stronger) alongside enabling aging. Does that sound ok?
I'm a bit disappointed that we haven't gotten forward progress on the arm64 patches, but I also recognize this is the direction of travel as the x86 patches are shaping up.
Quoted 8 lines
Quoted 2 lines
So yeah, I'm OK with it, but I'd love to get the arm64 side sorted out soon while the context is still fresh.
Converting the aging notifiers to holding
mmu_lockfor read seems like a pure win and minimal churn. Can we keep that patch in v7 (which depends on the lockless notifier refactor, i.e. is not completely stand-alone)? We can revisit enabling MGLRU on arm64 in a subsequent series.
Even though the churn is minimal in LOC, locking changes are significant. If one thing has become clear, there are some strong opinions about arm64 participating in MGLRU w/ the read lock. So it is almost guaranteed that these read lock changes will eventually get thrown out in favor of an RCU-protected walker.
Then we're stuck with potentially 3 flavors of locking in kernels that people actually use, and dealing with breakage that only affects that intermediate step is gonna be annoying.
Signature · 2 lines
Thanks, Oliver
PATCH v6 4/11 mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER
In reply to James HoughtonRemove the now unnecessary ifdef in mm/damon/vaddr.c as well.
--- include/linux/mmu_notifier.h | 7 +++++++ mm/damon/vaddr.c | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-)
include/linux/mmu_notifier.h +7 -0
@@ -606,6 +606,13 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm,return 0 ; } +static inline int mmu_notifier_clear_young (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +return 0 ; +} +static inline int mmu_notifier_test_young (struct mm_struct *mm,unsigned long address) {
mm/damon/vaddr.c +0 -2
@@ -351,11 +351,9 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,set_huge_pte_at (mm, addr, pte, entry, psize); } -#ifdef CONFIG_MMU_NOTIFIER if (mmu_notifier_clear_young (mm, addr, addr +huge_page_size (hstate_vma (vma)))) referenced = true; -#endif /* CONFIG_MMU_NOTIFIER */ if (referenced)folio_set_young (folio);
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Re: [PATCH v6 04/11] mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER
Quoted 32 lines · James Houghton
Remove the now unnecessary ifdef in
mm/damon/vaddr.cas well.Signed-off-by James Houghton <[email protected]>--- include/linux/mmu_notifier.h | 7 +++++++ mm/damon/vaddr.c | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-)include/linux/mmu_notifier.h +7
-0@@ -606,6 +606,13 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm,return 0 ; } +static inline int mmu_notifier_clear_young (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +return 0 ; +} +static inline int mmu_notifier_test_young (struct mm_struct *mm,unsigned long address) {mm/damon/vaddr.c +0
-0@@ -351,11 +351,9 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,set_huge_pte_at (mm, addr, pte, entry, psize);
mm/damon/vaddr.c static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, · line 359
Quoted 10 lines
mm/damon/vaddr.c +0
-2} -#ifdef CONFIG_MMU_NOTIFIER if (mmu_notifier_clear_young (mm, addr, addr +huge_page_size (hstate_vma (vma)))) referenced = true; -#endif /* CONFIG_MMU_NOTIFIER */ if (referenced)folio_set_young (folio);
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v6 04/11] mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER
In reply to James HoughtonQuoted 7 lines · James Houghton
Remove the now unnecessary ifdef in
mm/damon/vaddr.cas well.Signed-off-by James Houghton <[email protected]>--- include/linux/mmu_notifier.h | 7 +++++++ mm/damon/vaddr.c | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-)
Jason
PATCH v6 5/11 mm: Add fast_only bool to test_young and clear_young MMU notifiers
In reply to James HoughtonFor implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.
Also add mmu_notifier_test_young_fast_only() and mmu_notifier_clear_young_fast_only() helpers to set fast_only for these notifiers.
--- include/linux/mmu_notifier.h | 46 +++++++++++++++++++++++++++++++----- include/trace/events/kvm.h | 19 +++++++++------ mm/mmu_notifier.c | 12 ++++++---- virt/kvm/kvm_main.c | 12 ++++++---- 4 files changed, 67 insertions(+), 22 deletions(-)
include/linux/mmu_notifier.h +40 -6
@@ -110,7 +110,8 @@ struct mmu_notifier_ops {int (*clear_young)(struct mmu_notifier *subscription,struct mm_struct *mm,unsigned long start, -unsigned long end); +unsigned long end, +bool fast_only);/* * test_young is called to check the young/accessed bitflag in
@@ -120,7 +121,8 @@ struct mmu_notifier_ops {*/ int (*test_young)(struct mmu_notifier *subscription,struct mm_struct *mm, -unsigned long address); +unsigned long address, +bool fast_only);/* * invalidate_range_start() and invalidate_range_end() must be
@@ -380,9 +382,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long end);extern int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end); +unsigned long end, +bool fast_only);extern int __mmu_notifier_test_young (struct mm_struct *mm, -unsigned long address); +unsigned long address, +bool fast_only);extern int __mmu_notifier_invalidate_range_start (struct mmu_notifier_range *r);extern void __mmu_notifier_invalidate_range_end (struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm,
@@ -416,7 +420,16 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) {if (mm_has_notifiers (mm)) -return __mmu_notifier_clear_young (mm, start, end); +return __mmu_notifier_clear_young (mm, start, end, false); +return 0 ; +} + +static inline int mmu_notifier_clear_young_fast_only (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +if (mm_has_notifiers (mm)) +return __mmu_notifier_clear_young (mm, start, end, true);return 0 ; }
@@ -424,7 +437,15 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) {if (mm_has_notifiers (mm)) -return __mmu_notifier_test_young (mm, address); +return __mmu_notifier_test_young (mm, address, false); +return 0 ; +} + +static inline int mmu_notifier_test_young_fast_only (struct mm_struct *mm, +unsigned long address) +{ +if (mm_has_notifiers (mm)) +return __mmu_notifier_test_young (mm, address, true);return 0 ; }
@@ -613,12 +634,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm,return 0 ; } +static inline int mmu_notifier_clear_young_fast_only (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +return 0 ; +} +static inline int mmu_notifier_test_young (struct mm_struct *mm,unsigned long address) {return 0 ; } +static inline int mmu_notifier_test_young_fast_only (struct mm_struct *mm, +unsigned long address) +{ +return 0 ; +} +static inline void mmu_notifier_invalidate_range_start (struct mmu_notifier_range *range) {
include/trace/events/kvm.h +12 -7
@@ -457,36 +457,41 @@ TRACE_EVENT(kvm_unmap_hva_range, );TRACE_EVENT (kvm_age_hva, - TP_PROTO(unsigned long start, unsigned long end), -TP_ARGS (start, end), + TP_PROTO(unsigned long start, unsigned long end, bool fast_only), +TP_ARGS (start, end, fast_only),TP_STRUCT__entry ( __field( unsigned long, start ) __field( unsigned long, end ) +__field ( bool, fast_only ) ),TP_fast_assign ( __entry->start = start; __entry->end = end; + __entry->fast_only = fast_only; ), -TP_printk ("mmu notifier age hva: %#016lx -- %#016lx" , - __entry->start, __entry->end) +TP_printk ("mmu notifier age hva: %#016lx -- %#016lx fast_only: %d" , + __entry->start, __entry->end, __entry->fast_only) );TRACE_EVENT (kvm_test_age_hva, - TP_PROTO(unsigned long hva), -TP_ARGS (hva), + TP_PROTO(unsigned long hva, bool fast_only), +TP_ARGS (hva, fast_only),TP_STRUCT__entry ( __field( unsigned long, hva ) +__field ( bool, fast_only ) ),TP_fast_assign ( __entry->hva = hva; + __entry->fast_only = fast_only; ), -TP_printk ("mmu notifier test age hva: %#016lx" , __entry->hva) +TP_printk ("mmu notifier test age hva: %#016lx fast_only: %d" , + __entry->hva, __entry->fast_only) );#endif /* _TRACE_KVM_MAIN_H */
mm/mmu_notifier.c +8 -4
@@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;
@@ -395,7 +396,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, srcu_read_lock_held(&srcu)) {if (subscription->ops->clear_young) young |= subscription->ops->clear_young (subscription, - mm, start, end); + mm, start, end, + fast_only); }srcu_read_unlock (&srcu, id);
@@ -403,7 +405,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, }int __mmu_notifier_test_young (struct mm_struct *mm, -unsigned long address) +unsigned long address, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;
@@ -414,7 +417,8 @@ int __mmu_notifier_test_young(struct mm_struct *mm, srcu_read_lock_held(&srcu)) {if (subscription->ops->test_young) { young = subscription->ops->test_young (subscription, mm, - address); + address, + fast_only);if (young)break ; }
virt/kvm/kvm_main.c +7 -5
@@ -874,7 +874,7 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,unsigned long start, unsigned long end) { -trace_kvm_age_hva (start ,end ); +trace_kvm_age_hva (start ,end ,false );return kvm_handle_hva_range (mn, start, end, kvm_age_gfn); }
@@ -882,9 +882,10 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,static int kvm_mmu_notifier_clear_young (struct mmu_notifier *mn,struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +bool fast_only) { -trace_kvm_age_hva (start, end); +trace_kvm_age_hva (start, end, fast_only);/* * Even though we do not flush TLB, this will still adversely
@@ -904,9 +905,10 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,static int kvm_mmu_notifier_test_young (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long address) +unsigned long address, +bool fast_only) { -trace_kvm_test_age_hva (address); +trace_kvm_test_age_hva (address, fast_only);return kvm_handle_hva_range_no_flush (mn, address, address +1 , kvm_test_age_gfn);
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers
Quoted 5 lines · James Houghton
For implementers, the
fast_onlybool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.
But what are the semantics if "fast_only" cannot be achieved by the implementer?
Can we add some documentation to the new functions that explain what this mysterious "fast_only" is and what the expected semantics are? Please? :)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · James Houghton
For implementers, the
fast_onlybool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.
Quoted 6 lines
But what are the semantics if "
fast_only" cannot be achieved by the implementer?Can we add some documentation to the new functions that explain what this mysterious "
fast_only" is and what the expected semantics are? Please? :)
Thanks for pointing out the missing documentation. How's this?
include/linux/mmu_notifier.h +14 -0
@@ -106,6 +106,18 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* The fast_only parameter indicates that this call should not block, +* and this function should not cause other MMU notifier calls to +* block. Usually this means that the implementation should be +* lockless. +* +* When called with fast_only, this notifier will be a no-op unless +* has_fast_aging is set on the struct mmu_notifier. +* +* When fast_only is true, if the implementer cannot determine that a +* range is young without blocking, it should return 0 (i.e., +* that the range is NOT young). */ int (*clear_young)(struct mmu_notifier *subscription, struct mm_struct *mm,
@@ -118,6 +130,8 @@ struct mmu_notifier_ops {* the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. +* +* The fast_only parameter has the same meaning as with clear_young. */ int (*test_young)(struct mmu_notifier *subscription, struct mm_struct *mm,
I've also moved the commit that follows this one (the one that adds has_fast_aging) to be before this one so that the comment makes sense.
Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers
Quoted 23 lines · James Houghton
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · James Houghton
For implementers, the
fast_onlybool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.Quoted 6 lines
But what are the semantics if "
fast_only" cannot be achieved by the implementer?Can we add some documentation to the new functions that explain what this mysterious "
fast_only" is and what the expected semantics are? Please? :)Thanks for pointing out the missing documentation. How's this?
include/linux/mmu_notifier.h +0
-0
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 109
Quoted 4 lines
include/linux/mmu_notifier.h +1
-0@@ -106,6 +106,18 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +*
Probably makes sense to highlight the parameters like @fast_only
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116
Quoted 7 lines
include/linux/mmu_notifier.h +7
-0+* The fast_only parameter indicates that this call should not block, +* and this function should not cause other MMU notifier calls to +* block. Usually this means that the implementation should be +* lockless. +* +* When called with fast_only, this notifier will be a no-op unless +* has_fast_aging is set on the struct mmu_notifier.
"... and will return 0 (NOT young)." ?
Quoted 17 lines
include/linux/mmu_notifier.h +6
-0+* +* When fast_only is true, if the implementer cannot determine that a +* range is young without blocking, it should return 0 (i.e., +* that the range is NOT young). */ int (*clear_young)(struct mmu_notifier *subscription, struct mm_struct *mm,@@ -118,6 +130,8 @@ struct mmu_notifier_ops {* the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. +* +* The fast_only parameter has the same meaning as with clear_young. */ int (*test_young)(struct mmu_notifier *subscription, struct mm_struct *mm,
Quoted 2 lines
I've also moved the commit that follows this one (the one that adds
has_fast_aging) to be before this one so that the comment makes sense.
Makes sense, thanks!
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers
Quoted 20 lines · David Hildenbrand
Quoted 18 lines · James Houghton
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · James Houghton
For implementers, the
fast_onlybool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.Quoted 6 lines
But what are the semantics if "
fast_only" cannot be achieved by the implementer?Can we add some documentation to the new functions that explain what this mysterious "
fast_only" is and what the expected semantics are? Please? :)Thanks for pointing out the missing documentation. How's this?
Quoted 11 lines
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 109
Quoted 9 lines
include/linux/mmu_notifier.h +1
-0@@ -106,6 +106,18 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* Probably makes sense to highlight the parameters like @fast_only
Will do.
Quoted 10 lines
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116
Quoted 8 lines
include/linux/mmu_notifier.h +7
-0+* The fast_only parameter indicates that this call should not block, +* and this function should not cause other MMU notifier calls to +* block. Usually this means that the implementation should be +* lockless. +* +* When called with fast_only, this notifier will be a no-op unless +* has_fast_aging is set on the struct mmu_notifier. "... and will return 0 (NOT young)." ?
Thanks, I'll add this.
Quoted 18 lines
Quoted 18 lines
include/linux/mmu_notifier.h +6
-0+* +* When fast_only is true, if the implementer cannot determine that a +* range is young without blocking, it should return 0 (i.e., +* that the range is NOT young). */ int (*clear_young)(struct mmu_notifier *subscription, struct mm_struct *mm,@@ -118,6 +130,8 @@ struct mmu_notifier_ops {* the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. +* +* The fast_only parameter has the same meaning as with clear_young. */ int (*test_young)(struct mmu_notifier *subscription, struct mm_struct *mm,
Quoted 5 lines
Quoted 2 lines
I've also moved the commit that follows this one (the one that adds
has_fast_aging) to be before this one so that the comment makes sense.Makes sense, thanks!
Thanks David!
Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers
In reply to James HoughtonQuoted 4 lines · James Houghton
include/linux/mmu_notifier.h +0
-0@@ -106,6 +106,18 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116
Quoted 10 lines
include/linux/mmu_notifier.h +8
-0* latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* The fast_only parameter indicates that this call should not block, +* and this function should not cause other MMU notifier calls to +* block. Usually this means that the implementation should be +* lockless. +* +* When called with fast_only, this notifier will be a no-op unless +* has_fast_aging is set on the struct mmu_notifier.
If you add a has_fast_aging I wonder if it is better to introduce new ops instead? The semantics are a bit easier to explain that way
Jason
Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers
Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · James Houghton
include/linux/mmu_notifier.h +3
-0@@ -106,6 +106,18 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* The fast_only parameter indicates that this call should not block, +* and this function should not cause other MMU notifier calls to
Quoted 8 lines
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116
Quoted 5 lines
include/linux/mmu_notifier.h +5
-0+* block. Usually this means that the implementation should be +* lockless. +* +* When called with fast_only, this notifier will be a no-op unless +* has_fast_aging is set on the struct mmu_notifier. If you add a
has_fast_agingI wonder if it is better to introduce new ops instead? The semantics are a bit easier to explain that way
v5 implemented these with a new op[1]. *Just* having the new op is kind of problematic -- we have yet another op to do something very similar to what already exists. We are left with two options:
consolidate everything into a single notifier[2] or add a new parameter to test/clear_young()[3]. The latter, implemented in this v6, is somewhat simpler to implement (fewer LoC, reduces some duplication in KVM), though it does indeed make the explanation for test/clear_young() slightly more complex. I don't feel very strongly, but unless you do, I think I just ought to stick with how the v6 does it. :)
Thanks Jason!
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/CADrL8HVHcKSW3hiHzKTit07gzo36jtCZCnM9ZpueyifgNdGggw@mail.gmail.com/
[3]: https://lore.kernel.org/linux-mm/CADrL8HXhGFWwHt728Bg15x1YxJmS=WD8z=KJc_Koaah=OvHDwg@mail.gmail.com/
Re: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers
Quoted 21 lines · James Houghton
Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · James Houghton
include/linux/mmu_notifier.h +3
-0@@ -106,6 +106,18 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* The fast_only parameter indicates that this call should not block, +* and this function should not cause other MMU notifier calls to Quoted 8 lines
include/linux/mmu_notifier.h struct mmu_notifier_ops { · line 116
Quoted 5 lines
include/linux/mmu_notifier.h +5
-0+* block. Usually this means that the implementation should be +* lockless. +* +* When called with fast_only, this notifier will be a no-op unless +* has_fast_aging is set on the struct mmu_notifier. If you add a
has_fast_agingI wonder if it is better to introduce new ops instead? The semantics are a bit easier to explain that way
Quoted 10 lines
v5 implemented these with a new op[1]. *Just* having the new op is kind of problematic -- we have yet another op to do something very similar to what already exists. We are left with two options:
consolidate everything into a single notifier[2] or add a new parameter to test/clear_young()[3]. The latter, implemented in this v6, is somewhat simpler to implement (fewer LoC, reduces some duplication in KVM), though it does indeed make the explanation for test/clear_young() slightly more complex. I don't feel very strongly, but unless you do, I think I just ought to stick with how the v6 does it. :)
If it does makes the code simpler then it is probably the better choice
Jason
PATCH v6 6/11 mm: Add has_fast_aging to struct mmu_notifier
In reply to James Houghtonhas_fast_aging should be set by subscribers that non-trivially implement fast_only versions of both test_young() and clear_young().
Fast aging must be opt-in. For a subscriber that has not been enlightened with "fast aging", the test/clear_young() will behave identically whether or not fast_only is given.
Given that KVM is the only test/clear_young() implementer, we could instead add an equivalent check in KVM, but doing so would incur an indirect function call every time, even if the notifier ends up being a no-op.
Add mm_has_fast_young_notifiers() in case a caller wants to know if it should skip many calls to the mmu notifiers that may not be necessary (like MGLRU look-around).
--- include/linux/mmu_notifier.h | 14 ++++++++++++++ mm/mmu_notifier.c | 26 ++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
include/linux/mmu_notifier.h +14 -0
@@ -233,6 +233,7 @@ struct mmu_notifier {struct mm_struct *mm;struct rcu_head rcu;unsigned int users; +bool has_fast_aging; };/**
@@ -387,6 +388,7 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm,extern int __mmu_notifier_test_young (struct mm_struct *mm,unsigned long address,bool fast_only); +extern bool __mm_has_fast_young_notifiers (struct mm_struct *mm);extern int __mmu_notifier_invalidate_range_start (struct mmu_notifier_range *r);extern void __mmu_notifier_invalidate_range_end (struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm,
@@ -449,6 +451,13 @@ static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm,return 0 ; } +static inline bool mm_has_fast_young_notifiers (struct mm_struct *mm) +{ +if (mm_has_notifiers (mm)) +return __mm_has_fast_young_notifiers (mm); +return 0 ; +} +static inline void mmu_notifier_invalidate_range_start (struct mmu_notifier_range *range) {
@@ -653,6 +662,11 @@ static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm,return 0 ; } +static inline bool mm_has_fast_young_notifiers (struct mm_struct *mm) +{ +return 0 ; +} +static inline void mmu_notifier_invalidate_range_start (struct mmu_notifier_range *range) {
mm/mmu_notifier.c +26 -0
@@ -382,6 +382,26 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,return young; } +bool __mm_has_fast_young_notifiers (struct mm_struct *mm) +{ +struct mmu_notifier *subscription; +bool has_fast_aging = false; +int id; + + id =srcu_read_lock (&srcu); +hlist_for_each_entry_rcu (subscription, + &mm->notifier_subscriptions->list, hlist, +srcu_read_lock_held (&srcu)) { +if (subscription->has_fast_aging) { + has_fast_aging = true; +break ; + } + } +srcu_read_unlock (&srcu, id); + +return has_fast_aging; +} + int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, unsigned long end,
@@ -394,6 +414,9 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { +if (fast_only && !subscription->has_fast_aging) +continue ; +if (subscription->ops->clear_young) young |= subscription->ops->clear_young(subscription, mm, start, end,
@@ -415,6 +438,9 @@ int __mmu_notifier_test_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { +if (fast_only && !subscription->has_fast_aging) +continue ; +if (subscription->ops->test_young) { young = subscription->ops->test_young(subscription, mm, address,
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
PATCH v6 7/11 KVM: Pass fast_only to kvm_{test_,}age_gfn
In reply to James HoughtonProvide the basics for architectures to implement a fast-only version of kvm_{test_,}age_gfn.
Add CONFIG_HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY that architectures will set if they non-trivially implement test_young() and clear_young() when called with fast_only.
--- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 4 ++++ virt/kvm/kvm_main.c | 37 +++++++++++++++++++++---------------- 3 files changed, 26 insertions(+), 16 deletions(-)
include/linux/kvm_host.h +1 -0
@@ -258,6 +258,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg {unsigned long attributes; +bool fast_only; }; struct kvm_gfn_range {
virt/kvm/Kconfig +4 -0
@@ -103,6 +103,10 @@ config KVM_GENERIC_MMU_NOTIFIER config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS bool +config HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY + select KVM_GENERIC_MMU_NOTIFIER + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool
virt/kvm/kvm_main.c +21 -16
@@ -697,18 +697,20 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn,unsigned long start,unsigned long end, -gfn_handler_t handler) +gfn_handler_t handler, +bool fast_only) {struct kvm *kvm =mmu_notifier_to_kvm (mn);const struct kvm_mmu_notifier_range range = { - .start = start, - .end = end, - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .flush_on_ret = false, - .may_block = false, - .lockless = + .start = start, + .end = end, + .handler = handler, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = false, + .may_block = false, + .lockless =IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), + .arg.fast_only = fast_only, };return __kvm_handle_hva_range (kvm, &range).ret;
@@ -900,7 +902,8 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,* cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ -return kvm_handle_hva_range_no_flush (mn, start, end, kvm_age_gfn); +return kvm_handle_hva_range_no_flush (mn, start, end, kvm_age_gfn, + fast_only); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
@@ -911,7 +914,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,trace_kvm_test_age_hva (address, fast_only);return kvm_handle_hva_range_no_flush (mn, address, address +1 , - kvm_test_age_gfn); + kvm_test_age_gfn, fast_only); } static void kvm_mmu_notifier_release(struct mmu_notifier *mn,
@@ -926,17 +929,19 @@ static void kvm_mmu_notifier_release(struct mmu_notifier *mn, }static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { - .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start, - .invalidate_range_end = kvm_mmu_notifier_invalidate_range_end, - .clear_flush_young = kvm_mmu_notifier_clear_flush_young, - .clear_young = kvm_mmu_notifier_clear_young, - .test_young = kvm_mmu_notifier_test_young, - .release = kvm_mmu_notifier_release, + .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start, + .invalidate_range_end = kvm_mmu_notifier_invalidate_range_end, + .clear_flush_young = kvm_mmu_notifier_clear_flush_young, + .clear_young = kvm_mmu_notifier_clear_young, + .test_young = kvm_mmu_notifier_test_young, + .release = kvm_mmu_notifier_release, };static int kvm_init_mmu_notifier (struct kvm *kvm) { kvm->mmu_notifier.ops = &kvm_mmu_notifier_ops; + kvm->mmu_notifier.has_fast_aging = +IS_ENABLED (CONFIG_HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY );return mmu_notifier_register (&kvm->mmu_notifier, current->mm); }
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
PATCH v6 8/11 KVM: x86: Optimize kvm_{test_,}age_gfn a little bit
In reply to James HoughtonOptimize both kvm_age_gfn and kvm_test_age_gfn's interaction with the shadow MMU by, rather than checking if our memslot has rmaps, check if there are any indirect_shadow_pages at all.
Also, for kvm_test_age_gfn, reorder the TDP MMU check to be first. If we find that the range is young, we do not need to check the shadow MMU.
--- arch/x86/kvm/mmu/mmu.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +13 -8
@@ -1629,19 +1629,24 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot,__rmap_add (vcpu->kvm, cache, slot, spte, gfn, access); } +static bool kvm_has_shadow_mmu_sptes (struct kvm *kvm) +{ +return !tdp_mmu_enabled ||READ_ONCE (kvm->arch.indirect_shadow_pages); +} +bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) { +if (tdp_mmu_enabled) + young |=kvm_tdp_mmu_age_gfn_range (kvm, range); + +if (kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap);write_unlock (&kvm->mmu_lock); } -if (tdp_mmu_enabled) - young |=kvm_tdp_mmu_age_gfn_range (kvm, range); -return young; }
@@ -1649,15 +1654,15 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) { +if (tdp_mmu_enabled) + young |=kvm_tdp_mmu_test_age_gfn (kvm, range); + +if (!young &&kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap);write_unlock (&kvm->mmu_lock); } -if (tdp_mmu_enabled) - young |=kvm_tdp_mmu_test_age_gfn (kvm, range); -return young; }
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Re: [PATCH v6 08/11] KVM: x86: Optimize kvm_{test_,}age_gfn a little bit
Quoted 1 line · James Houghton
Optimize both
kvm_age_gfnand kvm_test_age_gfn's interaction with the
nit: Use () when referring to functions.
Quoted 2 lines
shadow MMU by, rather than checking if our memslot has rmaps, check if there are any
indirect_shadow_pagesat all.
What is optimized by checking indirect_shadow_pages instead of have_rmaps and what's the benefit? Smells like a premature optimization.
Quoted 3 lines
Also, for
kvm_test_age_gfn, reorder the TDP MMU check to be first. If we find that the range is young, we do not need to check the shadow MMU.
This should be a separate commit since it's a logically distinct change and no dependency on the other change in this commit (other than both touch the same function).
Splitting the commits up will also make it easier to write more specific short logs (instead of "optimize a little bit" :)
Also, the commit re-orders kvm_age_gfn() as well but the commit message only mentions kvm_test_age_gfn(). No objection to keeping the two functions consistent but it should be called out in the commit message.
Quoted 40 lines
Signed-off-by James Houghton <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)arch/x86/kvm/mmu/mmu.c +9
-4@@ -1629,19 +1629,24 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot,__rmap_add (vcpu->kvm, cache, slot, spte, gfn, access); } +static bool kvm_has_shadow_mmu_sptes (struct kvm *kvm) +{ +return !tdp_mmu_enabled ||READ_ONCE (kvm->arch.indirect_shadow_pages); +} +bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) { +if (tdp_mmu_enabled) + young |=kvm_tdp_mmu_age_gfn_range (kvm, range); + +if (kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap);write_unlock (&kvm->mmu_lock); } -if (tdp_mmu_enabled) - young |=kvm_tdp_mmu_age_gfn_range (kvm, range); -return young; }
arch/x86/kvm/mmu/mmu.c bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · line 1660
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +4
-1@@ -1649,15 +1654,15 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) { +if (tdp_mmu_enabled) + young |=kvm_tdp_mmu_test_age_gfn (kvm, range); + +if (!young &&kvm_has_shadow_mmu_sptes (kvm)) {
nit: A short comment here might be helpful to explain why young is checked.
Quoted 14 lines
arch/x86/kvm/mmu/mmu.c +0
-3write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap);write_unlock (&kvm->mmu_lock); } -if (tdp_mmu_enabled) - young |=kvm_tdp_mmu_test_age_gfn (kvm, range); -return young; }Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Sean Christopherson
maintainerRe: [PATCH v6 08/11] KVM: x86: Optimize kvm_{test_,}age_gfn a little bit
Quoted 10 lines · David Matlack
Quoted 1 line · James Houghton
Optimize both
kvm_age_gfnand kvm_test_age_gfn's interaction with thenit: Use () when referring to functions.
Quoted 2 lines
shadow MMU by, rather than checking if our memslot has rmaps, check if there are any
indirect_shadow_pagesat all.What is optimized by checking
indirect_shadow_pagesinstead ofhave_rmapsand what's the benefit? Smells like a premature optimization.
Checking indirect_shadow_pages avoids taking mmu_lock for write when KVM doesn't currently have shadow MMU pages, but did at some point in the past, whereas kvm_memslots_have_rmaps() is sticky and will return true forever.
Quoted 9 lines
Quoted 2 lines
Also, for
kvm_test_age_gfn, reorder the TDP MMU check to be first. If we find that the range is young, we do not need to check the shadow MMU.This should be a separate commit since it's a logically distinct change and no dependency on the other change in this commit (other than both touch the same function).
Splitting the commits up will also make it easier to write more specific short logs (instead of "optimize a little bit" :)
+1. Especially code movement and refactoring, e.g. factoring out tdp_mmu_clear_spte_bits_atomic() would ideally be in a standalone patch that's dead simple to review.
Re: [PATCH v6 08/11] KVM: x86: Optimize kvm_{test_,}age_gfn a little bit
Quoted 10 lines · Sean Christopherson
Quoted 8 lines · David Matlack
Quoted 1 line · James Houghton
Optimize both
kvm_age_gfnand kvm_test_age_gfn's interaction with thenit: Use () when referring to functions.
Quoted 2 lines
shadow MMU by, rather than checking if our memslot has rmaps, check if there are any
indirect_shadow_pagesat all.
Quoted 6 lines
Quoted 2 lines
What is optimized by checking
indirect_shadow_pagesinstead ofhave_rmapsand what's the benefit? Smells like a premature optimization.Checking
indirect_shadow_pagesavoids takingmmu_lockfor write when KVM doesn't currently have shadow MMU pages, but did at some point in the past, whereaskvm_memslots_have_rmaps()is sticky and will return true forever.
Thanks for the clear explanation.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Also, for
kvm_test_age_gfn, reorder the TDP MMU check to be first. If we find that the range is young, we do not need to check the shadow MMU.This should be a separate commit since it's a logically distinct change and no dependency on the other change in this commit (other than both touch the same function).
Done.
Quoted 6 lines
Quoted 2 lines
Splitting the commits up will also make it easier to write more specific short logs (instead of "optimize a little bit" :)
+1. Especially code movement and refactoring, e.g. factoring out
tdp_mmu_clear_spte_bits_atomic()would ideally be in a standalone patch that's dead simple to review.
I have now split out the creation of tdp_mmu_clear_spte_bits_atomic() into its own patch. Though I'm not entirely convinced splitting out every refactor like that is always a good thing.
PATCH v6 9/11 KVM: x86: Implement fast_only versions of kvm_{test_,}age_gfn
In reply to James HoughtonThese fast-only versions simply ignore the shadow MMU. We can locklessly handle the shadow MMU later.
Set HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY for X86_64 only, as that is the only case where the TDP MMU might be used. Without the TDP MMU, the fast-only notifiers will always be no-ops. It would be ideal not to report has_fast_only if !tdp_mmu_enabled, but tdp_mmu_enabled can be changed at any time.
--- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kvm/Kconfig +1 -0
@@ -24,6 +24,7 @@ config KVM select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + select HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY if X86_64 select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +2 -2
@@ -1641,7 +1641,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range); -if (kvm_has_shadow_mmu_sptes (kvm)) { +if (!range->arg.fast_only &&kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap);write_unlock (&kvm->mmu_lock);
@@ -1657,7 +1657,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range); -if (!young &&kvm_has_shadow_mmu_sptes (kvm)) { +if (!young && !range->arg.fast_only &&kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap);write_unlock (&kvm->mmu_lock);
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Re: [PATCH v6 09/11] KVM: x86: Implement fast_only versions of kvm_{test_,}age_gfn
Quoted 8 lines · James Houghton
These fast-only versions simply ignore the shadow MMU. We can locklessly handle the shadow MMU later.
Set
HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLYforX86_64only, as that is the only case where the TDP MMU might be used. Without the TDP MMU, the fast-only notifiers will always be no-ops. It would be ideal not to reporthas_fast_onlyif !tdp_mmu_enabled, buttdp_mmu_enabledcan be changed at any time.
tdp_mmu_enabled is a read-only KVM parameter. And even when it was writable, it was still fixed for a given VM at VM creation time.
Would it make more sense to have kvm_arch_post_init_vm() set has_fast_aging if the architecture supports it. And for x86 that means iff tdp_mmu_enabled.
Quoted 44 lines
Signed-off-by James Houghton <[email protected]>--- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)arch/x86/kvm/Kconfig +1
-0@@ -24,6 +24,7 @@ config KVM select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + select HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY if X86_64 select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSOarch/x86/kvm/mmu/mmu.c +2
-2@@ -1641,7 +1641,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range); -if (kvm_has_shadow_mmu_sptes (kvm)) { +if (!range->arg.fast_only &&kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap);write_unlock (&kvm->mmu_lock);@@ -1657,7 +1657,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range); -if (!young &&kvm_has_shadow_mmu_sptes (kvm)) { +if (!young && !range->arg.fast_only &&kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap);write_unlock (&kvm->mmu_lock);Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
PATCH v6 10/11 mm: multi-gen LRU: Have secondary MMUs participate in aging
In reply to James HoughtonSecondary MMUs are currently consulted for access/age information at eviction time, but before then, we don't get accurate age information. That is, pages that are mostly accessed through a secondary MMU (like guest memory, used by KVM) will always just proceed down to the oldest generation, and then at eviction time, if KVM reports the page to be young, the page will be activated/promoted back to the youngest generation.
The added feature bit (0x8), if disabled, will make MGLRU behave as if there are no secondary MMUs subscribed to MMU notifiers except at eviction time.
Implement aging with the new mmu_notifier_clear_young_fast_only() notifier. For architectures that do not support this notifier, this becomes a no-op. For architectures that do implement it, it should be fast enough to make aging worth it (usually the case if the notifier is implemented locklessly).
--- Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 148 ++++++++++++++---- 4 files changed, 127 insertions(+), 42 deletions(-)
Documentation/admin-guide/mm/multigen_lru.rst +5 -1
@@ -48,6 +48,10 @@ Values Components verified on x86 varieties other than Intel and AMD. If it is disabled, the multi-gen LRU will suffer a negligible performance degradation. +0x0008 Clear the accessed bit in secondary MMU page tables when aging + instead of waiting until eviction time. This results in accurate + page age information for pages that are mainly used by a + secondary MMU. [yYnN] Apply to all the components above. ====== ===============================================================
@@ -56,7 +60,7 @@ E.g., echo y >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled - 0x0007 + 0x000f echo 5 >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0005
include/linux/mmzone.h +4 -2
@@ -400,6 +400,7 @@ enum {LRU_GEN_CORE ,LRU_GEN_MM_WALK ,LRU_GEN_NONLEAF_YOUNG , +LRU_GEN_SECONDARY_MMU_WALK , NR_LRU_GEN_CAPS };
@@ -557,7 +558,7 @@ struct lru_gen_memcg {void lru_gen_init_pgdat (struct pglist_data *pgdat);void lru_gen_init_lruvec (struct lruvec *lruvec); -void lru_gen_look_around (struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around (struct page_vma_mapped_walk *pvmw);void lru_gen_init_memcg (struct mem_cgroup *memcg);void lru_gen_exit_memcg (struct mem_cgroup *memcg);
@@ -576,8 +577,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } -static inline void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around (struct page_vma_mapped_walk *pvmw) { +return false; }static inline void lru_gen_init_memcg (struct mem_cgroup *memcg)
mm/rmap.c +3 -6
@@ -870,13 +870,10 @@ static bool folio_referenced_one(struct folio *folio,continue ; } -if (pvmw.pte) { -if (lru_gen_enabled () && -pte_young (ptep_get (pvmw.pte))) { -lru_gen_look_around (&pvmw); +if (lru_gen_enabled () && pvmw.pte) { +if (lru_gen_look_around (&pvmw)) referenced++; - } - + }else if (pvmw.pte) {if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) referenced++;
mm/vmscan.c +115 -33
314 lines not on this page · raw message
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
PATCH v6 11/11 KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test
In reply to James HoughtonThis test now has two modes of operation:
1. (default) To check how much vCPU performance was affected by access tracking (previously existed, now supports MGLRU aging).
2. (-p) To also benchmark how fast MGLRU can do aging while vCPUs are faulting in memory.
Mode (1) also serves as a way to verify that aging is working properly for pages only accessed by KVM. It will fail if one does not have the 0x8 lru_gen feature bit.
To support MGLRU, the test creates a memory cgroup, moves itself into it, then uses the lru_gen debugfs output to track memory in that cgroup. The logic to parse the lru_gen debugfs output has been put into selftests/kvm/lib/lru_gen_util.c.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 369 +++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ 4 files changed, 786 insertions(+), 30 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -22,6 +22,7 @@ LIBKVM += lib/elf.c LIBKVM += lib/guest_modes.c LIBKVM += lib/io.c LIBKVM += lib/kvm_util.c +LIBKVM += lib/lru_gen_util.c LIBKVM += lib/memstress.c LIBKVM += lib/guest_sprintf.c LIBKVM += lib/rbtree.c
tools/testing/selftests/kvm/access_tracking_perf_test.c +339 -30
488 lines not on this page · raw message
tools/testing/selftests/kvm/include/lru_gen_util.h new file +55 -0
55 lines not on this page · raw message
tools/testing/selftests/kvm/lib/lru_gen_util.c new file +391 -0
391 lines not on this page · raw message
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog