mm: multi-gen LRU: Walk secondary MMU page tables while aging
Subsystems (14)
v5 cover mm: multi-gen LRU: Walk secondary MMU page tables while aging
This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim.
This series does the following:
1. Improve locking for the existing test/clear_young notifiers for x86
and arm64.
2. Add a new notifier test_clear_young_fast_only(), implemented only by
KVM/x86. 3. Incorporate test_clear_young_fast_only() into MGLRU aging.
To make aging work for more than just x86, the test_clear_young_fast_only() notifier must be implemented by those other architectures.
access_tracking_perf_test now has a mode (-p) to check performance of MGLRU aging while the VM is faulting memory in. See the v4 cover letter[1] for performance data collected with this test.
Previous versions of this series included logic in MGLRU and KVM to support batching the updates to secondary page tables. This version removes this logic, as it was complex and not necessary to enable proactive reclaim. This optimization, as well as the additional optimizations for arm64 and powerpc, can be done in a later series.
=== Previous Versions ===
This v5 re-adds a lot of logic that was present in v3 and earlier versions of the series. There is an important difference I want to call out:
- should_look_around() can sometimes require two notifiers instead of one. This is necessary if I forbid myself from modifying mmu_notifier_clear_young().
It may simply be better to do what v2/v3 did have and not have a fast-only notifier, and merge them all. This makes the API slightly more complex. I'm not sure which is better.
Change log:
Since v4[1]:
- Removed Kconfig that controlled when aging was enabled. Aging will be done whenever the architecture supports it (thanks Yu).
- Added a new MMU notifier, test_clear_young_fast_only(), specifically for MGLRU to use.
- Add kvm_fast_{test_,}age_gfn, implemented by x86.
- Fix locking for clear_flush_young().
- Added KVM_MMU_NOTIFIER_YOUNG_LOCKLESS to clean up locking changes (thanks Sean).
- Fix WARN_ON and other cleanup for the arm64 locking changes (thanks Oliver).
Since v3[2]:
- Vastly simplified the series (thanks David). Removed mmu notifier batching logic entirely.
- Cleaned up how locking is done for mmu_notifier_test/clear_young (thanks David).
- Look-around is now only done when there are no secondary MMUs subscribed to MMU notifiers.
- CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added.
- Fixed the lockless implementation of kvm_{test,}age_gfn for x86 (thanks David).
- Added MGLRU functional and performance tests to access_tracking_perf_test (thanks Axel).
- In v3, an mm would be completely ignored (for aging) if there was a secondary MMU but support for secondary MMU walking was missing. Now, missing secondary MMU walking support simply skips the notifier calls (except for eviction).
- Added a sanity check for that range->lockless and range->on_lock are never both provided for the memslot walk.
For the changes since v2[3], see v3.
Based on 6.10-rc3.
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/kvmarm/[email protected]/
James Houghton (8):
KVM: Add lockless memslot walk to KVM
KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
mm: Add test_clear_young_fast_only MMU notifier
KVM: Add kvm_fast_age_gfn and kvm_fast_test_age_gfn
KVM: x86: Implement kvm_fast_test_age_gfn and kvm_fast_age_gfn
mm: multi-gen LRU: Have secondary MMUs participate in aging
KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test
Yu Zhao (1):
KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask
Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +- arch/arm64/kvm/mmu.c | 26 +- arch/x86/include/asm/kvm_host.h | 14 + arch/x86/kvm/Kconfig | 2 + arch/x86/kvm/mmu.h | 6 - arch/x86/kvm/mmu/mmu.c | 60 ++- arch/x86/kvm/mmu/spte.h | 1 - arch/x86/kvm/mmu/tdp_iter.h | 27 +- arch/x86/kvm/mmu/tdp_mmu.c | 67 ++- include/linux/kvm_host.h | 8 + include/linux/mmu_notifier.h | 50 +++ include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 22 + mm/mmu_notifier.c | 26 ++ mm/rmap.c | 9 +- mm/vmscan.c | 185 +++++++-- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 365 ++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ virt/kvm/Kconfig | 7 + virt/kvm/kvm_main.c | 73 +++- 24 files changed, 1283 insertions(+), 140 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c
base-commit: 83a7eefedc9b56fe7bfeff13b6c7356688ffa670
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
PATCH v5 1/9 KVM: Add lockless memslot walk to KVM
Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing.
Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS.
The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply.
--- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 26 +++++++++++++++++++------- 3 files changed, 23 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +1 -0
@@ -266,6 +266,7 @@ struct kvm_gfn_range {gfn_t end;union kvm_mmu_notifier_arg arg;bool may_block; +bool lockless; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);
virt/kvm/Kconfig +3 -0
@@ -97,6 +97,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool
virt/kvm/kvm_main.c +19 -7
@@ -558,6 +558,7 @@ struct kvm_mmu_notifier_range {on_lock_fn_t on_lock;bool flush_on_ret;bool may_block; +bool lockless; };/*
@@ -612,6 +613,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler)))return r; +/* on_lock will never be called for lockless walks */ +if (WARN_ON_ONCE (range->lockless && !IS_KVM_NULL_FN (range->on_lock))) +return r; + idx =srcu_read_lock (&kvm->srcu);for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) {
@@ -643,15 +648,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start =hva_to_gfn_memslot (hva_start, slot); gfn_range.end =hva_to_gfn_memslot (hva_end +PAGE_SIZE -1 , slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless;if (!r.found_memslot) { r.found_memslot = true; -KVM_MMU_LOCK (kvm); -if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm); - -if (IS_KVM_NULL_FN (range->handler)) -break ; +if (!range->lockless) { +KVM_MMU_LOCK (kvm); +if (!IS_KVM_NULL_FN (range->on_lock)) + range->on_lock (kvm); + +if (IS_KVM_NULL_FN (range->handler)) +break ; + } } r.ret |= range->handler (kvm, &gfn_range); }
@@ -660,7 +668,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,if (range->flush_on_ret && r.ret)kvm_flush_remote_tlbs (kvm); -if (r.found_memslot) +if (r.found_memslot && !range->lockless)KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);
@@ -681,6 +689,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };return __kvm_handle_hva_range (kvm, &range).ret;
@@ -699,6 +709,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };return __kvm_handle_hva_range (kvm, &range).ret;
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
PATCH v5 2/9 KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
In reply to James HoughtonWalk the TDP MMU in an RCU read-side critical section. This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the R/X bits).
If the cmpxchg for marking the spte for access tracking fails, we simply retry if the spte is still a leaf PTE. If it isn't, we return false to continue the walk.
Harvesting age information from the shadow MMU is still done while holding the MMU write lock.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 10 ++++- arch/x86/kvm/mmu/tdp_iter.h | 27 +++++++------ arch/x86/kvm/mmu/tdp_mmu.c | 67 +++++++++++++++++++++++++-------- 5 files changed, 77 insertions(+), 29 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1438,6 +1438,7 @@ struct kvm_arch {* tdp_mmu_page set. * * For reads, this list is protected by: +* RCU alone or * the MMU lock in read mode + RCU or * the MMU lock in write mode *
arch/x86/kvm/Kconfig +1 -0
@@ -23,6 +23,7 @@ config KVM depends on X86_LOCAL_APIC select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +8 -2
@@ -1633,8 +1633,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range);
@@ -1646,8 +1649,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_test_age_rmap); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range);
arch/x86/kvm/mmu/tdp_iter.h +15 -12
@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)return xchg (rcu_dereference (sptep), new_spte); } +static inline u64 tdp_mmu_clear_spte_bits_atomic (tdp_ptep_t sptep,u64 mask) +{ +atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference (sptep); + +return (u64 )atomic64_fetch_and (~mask, sptep_atomic); +} +static inline void __kvm_tdp_mmu_write_spte (tdp_ptep_t sptep,u64 new_spte) {KVM_MMU_WARN_ON (is_ept_ve_possible (new_spte));
@@ -32,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) }/* -* SPTEs must be modified atomically if they are shadow-present, leaf -* SPTEs, and have volatile bits, i.e. has bits that can be set outside -* of mmu_lock. The Writable bit can be set by KVM's fast page fault -* handler, and Accessed and Dirty bits can be set by the CPU. +* SPTEs must be modified atomically if they have bits that can be set outside +* of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the +* Writable bit can be set by KVM's fast page fault handler, the Accessed and +* Dirty bits can be set by the CPU, and the Accessed and R/X bits can be +* cleared by age_gfn_range. * * Note, non-leaf SPTEs do have Accessed bits and those bits are * technically volatile, but KVM doesn't consume the Accessed bit of
@@ -46,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)static inline bool kvm_tdp_mmu_spte_need_atomic_write (u64 old_spte,int level) {return is_shadow_present_pte (old_spte) && -is_last_spte (old_spte, level) && -spte_has_volatile_bits (old_spte); +is_last_spte (old_spte, level); } static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
@@ -63,12 +70,8 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,static inline u64 tdp_mmu_clear_spte_bits (tdp_ptep_t sptep,u64 old_spte,u64 mask,int level) { -atomic64_t *sptep_atomic; - -if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) { - sptep_atomic = (atomic64_t *)rcu_dereference (sptep); -return (u64 )atomic64_fetch_and (~mask, sptep_atomic); - } +if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) +return tdp_mmu_clear_spte_bits_atomic (sptep, mask);__kvm_tdp_mmu_write_spte (sptep, old_spte & ~mask);return old_spte;
arch/x86/kvm/mmu/tdp_mmu.c +52 -15
112 lines not on this page · raw message
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
PATCH v5 3/9 KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
In reply to James HoughtonReplace the MMU write locks (taken in the memslot iteration loop) for read locks.
Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change.
--- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++------ arch/arm64/kvm/mmu.c | 26 ++++++++++++++++++++------ 3 files changed, 30 insertions(+), 12 deletions(-)
arch/arm64/kvm/Kconfig +1 -0
@@ -22,6 +22,7 @@ menuconfig KVM select KVM_COMMON select KVM_GENERIC_HARDWARE_ENABLING select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT
arch/arm64/kvm/hyp/pgtable.c +9 -6
@@ -1319,10 +1319,10 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, data->young = true;/* -* stage2_age_walker() is always called while holding the MMU lock for -* write, so this will always succeed. Nonetheless, this deliberately -* follows the race detection pattern of the other stage-2 walkers in -* case the locking mechanics of the MMU notifiers is ever changed. +* This walk may not be exclusive; the PTE is permitted to change +* from under us. If there is a race to update this PTE, then the +* GFN is most likely young, so failing to clear the AF is likely +* to be inconsequential. */ if (data->mkold && !stage2_try_set_pte (ctx, new))return -EAGAIN ;
@@ -1345,10 +1345,13 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr,struct kvm_pgtable_walker walker = { .cb = stage2_age_walker, .arg = &data, - .flags =KVM_PGTABLE_WALK_LEAF , + .flags =KVM_PGTABLE_WALK_LEAF | +KVM_PGTABLE_WALK_SHARED , }; +int r; -WARN_ON (kvm_pgtable_walk (pgt, addr, size, &walker)); + r =kvm_pgtable_walk (pgt, addr, size, &walker); +WARN_ON (r && r != -EAGAIN );return data.young; }
arch/arm64/kvm/mmu.c +20 -6
@@ -1773,25 +1773,39 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {u64 size = (range->end - range->start) <<PAGE_SHIFT ; +bool young = false; + +read_lock (&kvm->mmu_lock);if (!kvm->arch.mmu.pgt)return false; -return kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, - range->start <<PAGE_SHIFT , - size, true); + young =kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, + range->start <<PAGE_SHIFT , + size, true); + +out: +read_unlock (&kvm->mmu_lock); +return young; }bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {u64 size = (range->end - range->start) <<PAGE_SHIFT ; +bool young = false; + +read_lock (&kvm->mmu_lock);if (!kvm->arch.mmu.pgt)return false; -return kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, - range->start <<PAGE_SHIFT , - size, false); + young =kvm_pgtable_stage2_test_clear_young (kvm->arch.mmu.pgt, + range->start <<PAGE_SHIFT , + size, false); + +out: +read_unlock (&kvm->mmu_lock); +return young; }phys_addr_t kvm_mmu_get_httbr (void )
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
Re: [PATCH v5 3/9] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 33 lines · James Houghton
Replace the MMU write locks (taken in the memslot iteration loop) for read locks.
Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The
stage2_age_walker()callback is safe to race with itself; update the comment to reflect the synchronization change.Signed-off-by James Houghton <[email protected]>--- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++------ arch/arm64/kvm/mmu.c | 26 ++++++++++++++++++++------ 3 files changed, 30 insertions(+), 12 deletions(-)arch/arm64/kvm/Kconfig +1
-0@@ -22,6 +22,7 @@ menuconfig KVM select KVM_COMMON select KVM_GENERIC_HARDWARE_ENABLING select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECTarch/arm64/kvm/hyp/pgtable.c +0
-0
arch/arm64/kvm/hyp/pgtable.c static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, · line 1322
Quoted 8 lines
arch/arm64/kvm/hyp/pgtable.c +1
-4@@ -1319,10 +1319,10 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, data->young = true;/* -* stage2_age_walker() is always called while holding the MMU lock for -* write, so this will always succeed. Nonetheless, this deliberately -* follows the race detection pattern of the other stage-2 walkers in -* case the locking mechanics of the MMU notifiers is ever changed. +* This walk may not be exclusive; the PTE is permitted to change
s/may not/is not/
Quoted 9 lines
arch/arm64/kvm/hyp/pgtable.c +3
-0+* from under us. If there is a race to update this PTE, then the +* GFN is most likely young, so failing to clear the AF is likely +* to be inconsequential. */ if (data->mkold && !stage2_try_set_pte (ctx, new))return -EAGAIN ;@@ -1345,10 +1345,13 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr,struct kvm_pgtable_walker walker = { .cb = stage2_age_walker,
arch/arm64/kvm/hyp/pgtable.c bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, · line 1354
Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +5
-2.arg = &data, - .flags = KVM_PGTABLE_WALK_LEAF, + .flags = KVM_PGTABLE_WALK_LEAF | + KVM_PGTABLE_WALK_SHARED, }; +int r; -WARN_ON (kvm_pgtable_walk (pgt, addr, size, &walker)); + r =kvm_pgtable_walk (pgt, addr, size, &walker); +WARN_ON (r && r != -EAGAIN );
I could've been more explicit last time around, could you please tone this down to WARN_ON_ONCE() as well?
Quoted 8 lines
arch/arm64/kvm/hyp/pgtable.c +0
-0return data.young; }arch/arm64/kvm/mmu.c +0
-0
arch/arm64/kvm/mmu.c bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) · line 1781
Quoted 9 lines
arch/arm64/kvm/mmu.c +3
-0@@ -1773,25 +1773,39 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {u64 size = (range->end - range->start) <<PAGE_SHIFT ; +bool young = false; + +read_lock (&kvm->mmu_lock);if (!kvm->arch.mmu.pgt)return false;
I'm guessing you meant to have 'goto out' here, since this early return fails to drop the mmu_lock.
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v5 3/9] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 37 lines · Oliver Upton
Quoted 35 lines · James Houghton
Replace the MMU write locks (taken in the memslot iteration loop) for read locks.
Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The
stage2_age_walker()callback is safe to race with itself; update the comment to reflect the synchronization change.Signed-off-by James Houghton <[email protected]>--- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++------ arch/arm64/kvm/mmu.c | 26 ++++++++++++++++++++------ 3 files changed, 30 insertions(+), 12 deletions(-)arch/arm64/kvm/Kconfig +1
-0@@ -22,6 +22,7 @@ menuconfig KVM select KVM_COMMON select KVM_GENERIC_HARDWARE_ENABLING select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECTarch/arm64/kvm/hyp/pgtable.c +0
-0@@ -1319,10 +1319,10 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, data->young = true;
Quoted 8 lines
arch/arm64/kvm/hyp/pgtable.c static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, · line 1322
Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +1
-4/* -* stage2_age_walker() is always called while holding the MMU lock for -* write, so this will always succeed. Nonetheless, this deliberately -* follows the race detection pattern of the other stage-2 walkers in -* case the locking mechanics of the MMU notifiers is ever changed. +* This walk may not be exclusive; the PTE is permitted to change s/may not/is not/
Will fix.
Quoted 17 lines
Quoted 17 lines
arch/arm64/kvm/hyp/pgtable.c +6
-1+* from under us. If there is a race to update this PTE, then the +* GFN is most likely young, so failing to clear the AF is likely +* to be inconsequential. */ if (data->mkold && !stage2_try_set_pte (ctx, new))return -EAGAIN ;@@ -1345,10 +1345,13 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr,struct kvm_pgtable_walker walker = { .cb = stage2_age_walker, .arg = &data, - .flags =KVM_PGTABLE_WALK_LEAF , + .flags =KVM_PGTABLE_WALK_LEAF | +KVM_PGTABLE_WALK_SHARED , }; +int r;
Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, · line 1354
Quoted 3 lines
arch/arm64/kvm/hyp/pgtable.c +2
-1-WARN_ON (kvm_pgtable_walk (pgt, addr, size, &walker)); + r =kvm_pgtable_walk (pgt, addr, size, &walker); +WARN_ON (r && r != -EAGAIN );I could've been more explicit last time around, could you please tone this down to
WARN_ON_ONCE()as well?
Will do, thanks.
Quoted 16 lines
Quoted 16 lines
arch/arm64/kvm/hyp/pgtable.c +0
-0return data.young; }arch/arm64/kvm/mmu.c +3
-0@@ -1773,25 +1773,39 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {u64 size = (range->end - range->start) <<PAGE_SHIFT ; +bool young = false; + +read_lock (&kvm->mmu_lock);
Quoted 5 lines
arch/arm64/kvm/mmu.c bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) · line 1781
Quoted 2 lines
arch/arm64/kvm/mmu.c +0
-0if (!kvm->arch.mmu.pgt)return false;I'm guessing you meant to have 'goto out' here, since this early return fails to drop the
mmu_lock.
Ah sorry! Thanks for catching this.
PATCH v5 4/9 mm: Add test_clear_young_fast_only MMU notifier
In reply to James HoughtonThis new notifier is for multi-gen LRU specifically, as it wants to be able to get and clear age information from secondary MMUs only if it can be done "fast".
By having this notifier specifically created for MGLRU, what "fast" means comes down to what is "fast" enough to improve MGLRU's ability to reclaim most of the time.
--- include/linux/mmu_notifier.h | 50 ++++++++++++++++++++++++++++++++++++ mm/mmu_notifier.c | 26 +++++++++++++++++++ 2 files changed, 76 insertions(+)
include/linux/mmu_notifier.h +50 -0
@@ -61,6 +61,15 @@ enum mmu_notifier_event {#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +/* +* Bits in the return value for test_clear_young_fast_only. +* +* MMU_NOTIFIER_FAST_YOUNG: notifier succeeded, secondary MMU reports young. +* MMU_NOTIFIER_FAST_FAILED: notifier failed. +*/ +#define MMU_NOTIFIER_FAST_YOUNG (1 << 0) +#define MMU_NOTIFIER_FAST_FAILED (1 << 1) +struct mmu_notifier_ops {/* * Called either by mmu_notifier_unregister or when the mm is
@@ -122,6 +131,24 @@ struct mmu_notifier_ops {struct mm_struct *mm, unsigned long address); +/* +* test_clear_young_fast_only is called to check (and optionally clear) +* the young/accessed bitflag in the secondary pte such that the +* secondary MMU must implement it in a way that will not significantly +* disrupt other MMU operations. In other words, speed is more +* important than accuracy. +* +* Returns MMU_NOTIFIER_FAST_YOUNG if the secondary pte(s) were young. +* Returns MMU_NOTIFIER_FAST_FAILED if the secondary MMU could not do +* an accurate fast-only test and/or clear of the young/accessed +* flag. +*/ +int (*test_clear_young_fast_only)(struct mmu_notifier *subscription, +struct mm_struct *mm, +unsigned long start, +unsigned long end, +bool clear); +/* * invalidate_range_start() and invalidate_range_end() must be * paired and are called only when the mmap_lock and/or the
@@ -383,6 +410,10 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end);extern int __mmu_notifier_test_young (struct mm_struct *mm,unsigned long address); +extern int __mmu_notifier_test_clear_young_fast_only (struct mm_struct *mm, +unsigned long start, +unsigned long end, +bool clear);extern int __mmu_notifier_invalidate_range_start (struct mmu_notifier_range *r);extern void __mmu_notifier_invalidate_range_end (struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm,
@@ -428,6 +459,17 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm,return 0 ; } +static inline int mmu_notifier_test_clear_young_fast_only (struct mm_struct *mm, +unsigned long start, +unsigned long end, +bool clear) +{ +if (mm_has_notifiers (mm)) +return __mmu_notifier_test_clear_young_fast_only (mm, start, end, + clear); +return 0 ; +} +static inline void mmu_notifier_invalidate_range_start (struct mmu_notifier_range *range) {
@@ -612,6 +654,14 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm,return 0 ; } +static inline int mmu_notifier_test_clear_young_fast_only (struct mm_struct *mm, +unsigned long start, +unsigned long end, +bool clear) +{ +return 0 ; +} +static inline void mmu_notifier_invalidate_range_start (struct mmu_notifier_range *range) {
mm/mmu_notifier.c +26 -0
@@ -424,6 +424,32 @@ int __mmu_notifier_test_young(struct mm_struct *mm,return young; } +int __mmu_notifier_test_clear_young_fast_only (struct mm_struct *mm, +unsigned long start, +unsigned long end, +bool clear) +{ +struct mmu_notifier *subscription; +int ret =0 , id; + + id =srcu_read_lock (&srcu); +hlist_for_each_entry_rcu (subscription, + &mm->notifier_subscriptions->list, hlist, +srcu_read_lock_held (&srcu)) { +if (subscription->ops->test_clear_young_fast_only) { + ret = subscription->ops->test_clear_young_fast_only ( + subscription, mm, start, end, clear); +if (ret &MMU_NOTIFIER_FAST_FAILED ) +break ; +if (!clear && (ret &MMU_NOTIFIER_FAST_YOUNG )) +break ; + } + } +srcu_read_unlock (&srcu, id); + +return ret; +} +static int mn_itree_invalidate (struct mmu_notifier_subscriptions *subscriptions,const struct mmu_notifier_range *range) {
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 2 lines · James Houghton
This new notifier is for multi-gen LRU specifically
Let me call it out before others do: we can't be this self-serving.
Quoted 9 lines
as it wants to be
able to get and clear age information from secondary MMUs only if it can be done "fast".By having this notifier specifically created for MGLRU, what "fast" means comes down to what is "fast" enough to improve MGLRU's ability to reclaim most of the time.
Signed-off-by James Houghton <[email protected]>
If we'd like this to pass other MM reviewers, especially the MMU notifier maintainers, we'd need to design a generic API that can benefit all the *existing* users: idle page tracking [1], DAMON [2] and MGLRU.
Also I personally prefer to extend the existing callbacks by adding new parameters, and on top of that, I'd try to consolidate the existing callbacks -- it'd be less of a hard sell if my changes result in less code, not more.
(v2 did all these, btw.)
[1] https://docs.kernel.org/admin-guide/mm/idle_page_tracking.html
[2] https://www.kernel.org/doc/html/latest/mm/damon/index.html
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 22 lines · Yu Zhao
Quoted 2 lines · James Houghton
This new notifier is for multi-gen LRU specifically
Let me call it out before others do: we can't be this self-serving.
Quoted 9 lines
as it wants to be
able to get and clear age information from secondary MMUs only if it can be done "fast".By having this notifier specifically created for MGLRU, what "fast" means comes down to what is "fast" enough to improve MGLRU's ability to reclaim most of the time.
Signed-off-by James Houghton <[email protected]>If we'd like this to pass other MM reviewers, especially the MMU notifier maintainers, we'd need to design a generic API that can benefit all the *existing* users: idle page tracking [1], DAMON [2] and MGLRU.
Quoted 6 lines
Also I personally prefer to extend the existing callbacks by adding new parameters, and on top of that, I'd try to consolidate the existing callbacks -- it'd be less of a hard sell if my changes result in less code, not more.
(v2 did all these, btw.)
I think consolidating the callbacks is cleanest, like you had it in v2. I really wasn't sure about this change honestly, but it was my attempt to incorporate feedback like this[3] from v4. I'll consolidate the callbacks like you had in v2.
Instead of the bitmap like you had, I imagine we'll have some kind of flags argument that has bits like MMU_NOTIFIER_YOUNG_CLEAR, MMU_NOTIFIER_YOUNG_FAST_ONLY, and other ones as they come up. Does that sound ok?
Do idle page tracking and DAMON need this new "fast-only" notifier? Or do they benefit from a generic API in other ways? Sorry if I missed this from some other mail.
I've got feedback saying that tying the definition of "fast" to MGLRU specifically is helpful. So instead of MMU_NOTIFIER_YOUNG_FAST_ONLY, maybe MMU_NOTIFIER_YOUNG_LRU_GEN_FAST to mean "do fast-for-MGLRU notifier". It sounds like you'd prefer the more generic one.
Thanks for the feedback -- I don't want to keep this series lingering on the list, so I'll try and get newer versions out sooner rather than later.
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 7 lines · James Houghton
Quoted 6 lines · Yu Zhao
Quoted 2 lines · James Houghton
This new notifier is for multi-gen LRU specifically
Let me call it out before others do: we can't be this self-serving.
Establishing motivation for a change is always a good idea. The wording could be a bit crisper, but the connection between the new MMU notifier and MGLRU is valuable. I do not view the wording of the changeset as excluding other users of the 'fast' notifier.
Quoted 4 lines
I think consolidating the callbacks is cleanest, like you had it in v2. I really wasn't sure about this change honestly, but it was my attempt to incorporate feedback like this[3] from v4. I'll consolidate the callbacks like you had in v2.
My strong preference is to have the callers expectations of the secondary MMU be explicit. Having ->${BLAH}_fast_only() makes this abundantly clear both at the callsite and in the implementation.
Quoted 8 lines
Instead of the bitmap like you had, I imagine we'll have some kind of flags argument that has bits like
MMU_NOTIFIER_YOUNG_CLEAR,MMU_NOTIFIER_YOUNG_FAST_ONLY, and other ones as they come up. Does that sound ok?Do idle page tracking and DAMON need this new "fast-only" notifier? Or do they benefit from a generic API in other ways? Sorry if I missed this from some other mail.
Let's also keep in mind we aren't establishing an ABI here. If we have direct line of sight (i.e. patches) on how to leverage the new MMU notifier for DAMON and idle page tracking then great, let's try and build something that satisfies all users. Otherwise, it isn't that big of a deal if the interface needs to change slightly when someone decides to leverage the MMU notifier for something else.
Quoted 8 lines
I've got feedback saying that tying the definition of "fast" to MGLRU specifically is helpful. So instead of
MMU_NOTIFIER_YOUNG_FAST_ONLY, maybeMMU_NOTIFIER_YOUNG_LRU_GEN_FASTto mean "do fast-for-MGLRU notifier". It sounds like you'd prefer the more generic one.Thanks for the feedback -- I don't want to keep this series lingering on the list, so I'll try and get newer versions out sooner rather than later.
Let's make sure we get alignment on this before you proceed, I don't get the sense that we're getting to a common understanding of where to go with this.
Signature · 2 lines
Thanks, Oliver
Sean Christopherson
maintainerRe: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 9 lines · Oliver Upton
Quoted 4 lines · James Houghton
I think consolidating the callbacks is cleanest, like you had it in v2. I really wasn't sure about this change honestly, but it was my attempt to incorporate feedback like this[3] from v4. I'll consolidate the callbacks like you had in v2.
My strong preference is to have the callers expectations of the secondary MMU be explicit. Having ->${BLAH}_fast_only() makes this abundantly clear both at the callsite and in the implementation.
Partially agreed. We don't need a dedicated mmu_notifier API to achieve that for the callsites, e.g. ptep_clear_young_notify() passes fast_only=false, and a new ptep_clear_young_notify_fast_only() does the obvious.
On the back end, odds are very good KVM is going to squish the "fast" and "slow" paths back into a common helper, so IMO having dedicated fast_only() APIs for the mmu_notifier hooks doesn't add much value in the end.
I'm not opposed to dedicated hooks, but I after poking around a bit, I suspect that passing a fast_only flag will end up being less cleaner for all parties.
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 15 lines · Sean Christopherson
Quoted 9 lines · Oliver Upton
Quoted 4 lines · James Houghton
I think consolidating the callbacks is cleanest, like you had it in v2. I really wasn't sure about this change honestly, but it was my attempt to incorporate feedback like this[3] from v4. I'll consolidate the callbacks like you had in v2.
My strong preference is to have the callers expectations of the secondary MMU be explicit. Having ->${BLAH}_fast_only() makes this abundantly clear both at the callsite and in the implementation.
Partially agreed. We don't need a dedicated
mmu_notifierAPI to achieve that for the callsites, e.g.ptep_clear_young_notify()passes fast_only=false, and a newptep_clear_young_notify_fast_only()does the obvious.
Quoted 6 lines
On the back end, odds are very good KVM is going to squish the "fast" and "slow" paths back into a common helper, so IMO having dedicated
fast_only()APIs for themmu_notifierhooks doesn't add much value in the end.I'm not opposed to dedicated hooks, but I after poking around a bit, I suspect that passing a
fast_onlyflag will end up being less cleaner for all parties.
Yeah, I think I'm headed in the same direction after actually reading the MM side of this, heh.
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 21 lines · Oliver Upton
Quoted 19 lines · Sean Christopherson
Quoted 9 lines · Oliver Upton
Quoted 4 lines · James Houghton
I think consolidating the callbacks is cleanest, like you had it in v2. I really wasn't sure about this change honestly, but it was my attempt to incorporate feedback like this[3] from v4. I'll consolidate the callbacks like you had in v2.
My strong preference is to have the callers expectations of the secondary MMU be explicit. Having ->${BLAH}_fast_only() makes this abundantly clear both at the callsite and in the implementation.
Partially agreed. We don't need a dedicated
mmu_notifierAPI to achieve that for the callsites, e.g.ptep_clear_young_notify()passes fast_only=false, and a newptep_clear_young_notify_fast_only()does the obvious.On the back end, odds are very good KVM is going to squish the "fast" and "slow" paths back into a common helper, so IMO having dedicated
fast_only()APIs for themmu_notifierhooks doesn't add much value in the end.
Quoted 5 lines
Quoted 2 lines
I'm not opposed to dedicated hooks, but I after poking around a bit, I suspect that passing a
fast_onlyflag will end up being less cleaner for all parties.Yeah, I think I'm headed in the same direction after actually reading the MM side of this, heh.
Yeah, I agree. What I have now for v6 is that the test_young() and clear_young() notifiers themselves now take a bool fast_only. When called with the existing helpers (e.g. mmu_notifier_test_young(), fast_only is false, and I have new helpers (e.g. mmu_notifier_test_young_fast_only()) that will set fast_only to true. Seems clean to me. Thanks!
Sean Christopherson
maintainerRe: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
In reply to James HoughtonQuoted 28 lines · James Houghton
Quoted 27 lines · Yu Zhao
Quoted 2 lines · James Houghton
This new notifier is for multi-gen LRU specifically
Let me call it out before others do: we can't be this self-serving.
Quoted 9 lines
as it wants to be
able to get and clear age information from secondary MMUs only if it can be done "fast".By having this notifier specifically created for MGLRU, what "fast" means comes down to what is "fast" enough to improve MGLRU's ability to reclaim most of the time.
Signed-off-by James Houghton <[email protected]>If we'd like this to pass other MM reviewers, especially the MMU notifier maintainers, we'd need to design a generic API that can benefit all the *existing* users: idle page tracking [1], DAMON [2] and MGLRU.
Also I personally prefer to extend the existing callbacks by adding new parameters, and on top of that, I'd try to consolidate the existing callbacks -- it'd be less of a hard sell if my changes result in less code, not more.
Quoted 6 lines
Quoted 1 line
(v2 did all these, btw.)
I think consolidating the callbacks is cleanest, like you had it in v2. I really wasn't sure about this change honestly, but it was my attempt to incorporate feedback like this[3] from v4. I'll consolidate the callbacks like you had in v2.
James, wait for others to chime in before committing yourself to a course of action, otherwise you're going to get ping-ponged to hell and back.
Quoted 4 lines
Instead of the bitmap like you had, I imagine we'll have some kind of flags argument that has bits like
MMU_NOTIFIER_YOUNG_CLEAR,MMU_NOTIFIER_YOUNG_FAST_ONLY, and other ones as they come up. Does that sound ok?
Why do we need a bundle of flags? If we extend .clear_young() and .test_young() as Yu suggests, then we only need a single "bool fast_only".
As for adding a fast_only versus dedicated APIs, I don't have a strong preference. Extending will require a small amount of additional churn, e.g. to pass in false, but that doesn't seem problematic on its own. On the plus side, there would be less copy+paste in include/linux/mmu_notifier.h (though that could be solved with macros :-) ).
E.g.
--
mm/mmu_notifier.c +19 -12
@@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;
@@ -393,9 +394,12 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { -if (subscription->ops->clear_young) - young |= subscription->ops->clear_young (subscription, - mm, start, end); +if (!subscription->ops->clear_young || + fast_only && !subscription->ops->has_fast_aging) +continue ; + + young |= subscription->ops->clear_young (subscription, + mm, start, end); }srcu_read_unlock (&srcu, id);
@@ -403,7 +407,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, }int __mmu_notifier_test_young (struct mm_struct *mm, -unsigned long address) +unsigned long address, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;
@@ -412,12 +417,15 @@ int __mmu_notifier_test_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { -if (subscription->ops->test_young) { - young = subscription->ops->test_young (subscription, mm, - address); -if (young) -break ; - } +if (!subscription->ops->test_young) +continue ; + +if (fast_only && !subscription->ops->has_fast_aging) +continue ; + + young = subscription->ops->test_young (subscription, mm, address); +if (young) +break ; }srcu_read_unlock (&srcu, id); --
It might also require multiplexing the return value to differentiate between "young" and "failed". Ugh, but the code already does that, just in a bespoke way.
Double ugh. Peeking ahead at the "failure" code, NAK to adding kvm_arch_young_notifier_likely_fast for all the same reasons I objected to kvm_arch_has_test_clear_young() in v1. Please stop trying to do anything like that, I will NAK each every attempt to have core mm/ code call directly into KVM.
Anyways, back to this code, before we spin another version, we need to agree on exactly what behavior we want out of secondary MMUs. Because to me, the behavior proposed in this version doesn't make any sense.
Signalling failure because KVM _might_ have relevant aging information in SPTEs that require taking kvm->mmu_lock is a terrible tradeoff. And for the test_young case, it's flat out wrong, e.g. if a page is marked Accessed in the TDP MMU, then KVM should return "young", not "failed".
If KVM is using the TDP MMU, i.e. has_fast_aging=true, then there will be rmaps if and only if L1 ran a nested VM at some point. But as proposed, KVM doesn't actually check if there are any shadow TDP entries to process. That could be fixed by looking at kvm->arch.indirect_shadow_pages, but even then it's not clear that bailing if kvm->arch.indirect_shadow_pages > 0 makes sense.
E.g. if L1 happens to be running an L2, but <10% of the VM's memory is exposed to L2, then "failure" is pretty much guaranteed to a false positive. And even for the pages that are exposed to L2, "failure" will occur if and only if the pages are being accessed _only_ by L2.
There most definitely are use cases where the majority of a VM's memory is accessed only by L2. But if those use cases are performing poorly under MGLRU, then IMO we should figure out a way to enhance KVM to do a fast harvest of nested TDP Accessed information, not make MGRLU+KVM suck for a VMs that run nested VMs.
Oh, and calling into mmu_notifiers to do the "slow" version if the fast version fails is suboptimal.
So rather than failing the fast aging, I think what we want is to know if an mmu_notifier found a young SPTE during a fast lookup. E.g. something like this in KVM, where using kvm_has_shadow_mmu_sptes() instead of kvm_memslots_have_rmaps() is an optional optimization to avoid taking mmu_lock for write in paths where a (very rare) false negative is acceptable.
static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm)
{
return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages);}
static int __kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range,
bool fast_only)
{
int young = 0;
if (!fast_only && kvm_has_shadow_mmu_sptes(kvm)) {
write_lock(&kvm->mmu_lock);
young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
write_unlock(&kvm->mmu_lock);
}
if (tdp_mmu_enabled && kvm_tdp_mmu_age_gfn_range(kvm, range))
young = 1 | MMU_NOTIFY_WAS_FAST;
return (int)young;}
and then in lru_gen_look_around():
if (spin_is_contended(pvmw->ptl)) return false; /* exclude special VMAs containing anon pages from COW */ if (vma->vm_flags & VM_SPECIAL) return false; young = ptep_clear_young_notify(vma, addr, pte); if (!young) return false; if (!(young & MMU_NOTIFY_WAS_FAST)) return true; young = 1;
with the lookaround done using ptep_clear_young_notify_fast().
The MMU_NOTIFY_WAS_FAST flag is gross, but AFAICT it would Just Work without needing to update all users of ptep_clear_young_notify() and friends.
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 32 lines · Sean Christopherson
Quoted 30 lines · James Houghton
Quoted 28 lines · Yu Zhao
Quoted 2 lines · James Houghton
This new notifier is for multi-gen LRU specifically
Let me call it out before others do: we can't be this self-serving.
Quoted 9 lines
as it wants to be
able to get and clear age information from secondary MMUs only if it can be done "fast".By having this notifier specifically created for MGLRU, what "fast" means comes down to what is "fast" enough to improve MGLRU's ability to reclaim most of the time.
Signed-off-by James Houghton <[email protected]>If we'd like this to pass other MM reviewers, especially the MMU notifier maintainers, we'd need to design a generic API that can benefit all the *existing* users: idle page tracking [1], DAMON [2] and MGLRU.
Also I personally prefer to extend the existing callbacks by adding new parameters, and on top of that, I'd try to consolidate the existing callbacks -- it'd be less of a hard sell if my changes result in less code, not more.
(v2 did all these, btw.)
Quoted 7 lines
Quoted 4 lines
I think consolidating the callbacks is cleanest, like you had it in v2. I really wasn't sure about this change honestly, but it was my attempt to incorporate feedback like this[3] from v4. I'll consolidate the callbacks like you had in v2.
James, wait for others to chime in before committing yourself to a course of action, otherwise you're going to get ping-ponged to hell and back.
Ah yeah. I really mean "I'll do it, provided the other feedback is in line with this".
Quoted 8 lines
Quoted 5 lines
Instead of the bitmap like you had, I imagine we'll have some kind of flags argument that has bits like
MMU_NOTIFIER_YOUNG_CLEAR,MMU_NOTIFIER_YOUNG_FAST_ONLY, and other ones as they come up. Does that sound ok?Why do we need a bundle of flags? If we extend .clear_young() and .test_young() as Yu suggests, then we only need a single "bool
fast_only".
We don't need to. In my head it's a little easier to collapse them (slightly less code, and at the callsite you have a flag with a name instead of a true/false). Making it a bool SGTM.
Quoted 5 lines
As for adding a
fast_onlyversus dedicated APIs, I don't have a strong preference. Extending will require a small amount of additional churn, e.g. to pass in false, but that doesn't seem problematic on its own. On the plus side, there would be less copy+paste ininclude/linux/mmu_notifier.h(though that could be solved with macros :-) ).
I think having the extra bool is cleaner than the new fast_only notifier, definitely.
Quoted 21 lines
E.g.
--
mm/mmu_notifier.c +2
-1@@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;@@ -393,9 +394,12 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,
mm/mmu_notifier.c int __mmu_notifier_clear_young(struct mm_struct *mm, · line 402
Quoted 10 lines
mm/mmu_notifier.c +6
-3srcu_read_lock_held(&srcu)) { -if (subscription->ops->clear_young) - young |= subscription->ops->clear_young (subscription, - mm, start, end); +if (!subscription->ops->clear_young || + fast_only && !subscription->ops->has_fast_aging) +continue ; + + young |= subscription->ops->clear_young (subscription, + mm, start, end);
KVM changing has_fast_aging dynamically would be slow, wouldn't it? I feel like it's simpler to just pass in fast_only into clear_young itself (and this is how I interpreted what you wrote above anyway).
Quoted 36 lines
mm/mmu_notifier.c +11
-8}srcu_read_unlock (&srcu, id);@@ -403,7 +407,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, }int __mmu_notifier_test_young (struct mm_struct *mm, -unsigned long address) +unsigned long address, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;@@ -412,12 +417,15 @@ int __mmu_notifier_test_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { -if (subscription->ops->test_young) { - young = subscription->ops->test_young (subscription, mm, - address); -if (young) -break ; - } +if (!subscription->ops->test_young) +continue ; + +if (fast_only && !subscription->ops->has_fast_aging) +continue ; + + young = subscription->ops->test_young (subscription, mm, address); +if (young) +break ; }srcu_read_unlock (&srcu, id); --
Quoted 2 lines
It might also require multiplexing the return value to differentiate between "young" and "failed". Ugh, but the code already does that, just in a bespoke way.
Yeah, that is necessary.
Quoted 4 lines
Double ugh. Peeking ahead at the "failure" code, NAK to adding
kvm_arch_young_notifier_likely_fastfor all the same reasons I objected tokvm_arch_has_test_clear_young()in v1. Please stop trying to do anything like that, I will NAK each every attempt to have coremm/code call directly into KVM.
Sorry to make you repeat yourself; I'll leave it out of v6. I don't like it either, but I wasn't sure how important it was to avoid calling into unnecessary notifiers if the TDP MMU were completely disabled.
Quoted 8 lines
Anyways, back to this code, before we spin another version, we need to agree on exactly what behavior we want out of secondary MMUs. Because to me, the behavior proposed in this version doesn't make any sense.
Signalling failure because KVM _might_ have relevant aging information in SPTEs that require taking
kvm->mmu_lockis a terrible tradeoff. And for thetest_youngcase, it's flat out wrong, e.g. if a page is marked Accessed in the TDP MMU, then KVM should return "young", not "failed".
Sorry for this oversight. What about something like:
1. test (and maybe clear) A bits on TDP MMU
2. If accessed && !should_clear: return (fast)
3. if (fast_only): return (fast)
4. If !(must check shadow MMU): return (fast)
5. test (and maybe clear) A bits in shadow MMU
6. return (slow)
Some of this reordering (and maybe a change from kvm_shadow_root_allocated() to checking indirect_shadow_pages or something else) can be done in its own patch.
Quoted 11 lines
If KVM is using the TDP MMU, i.e. has_fast_aging=true, then there will be rmaps if and only if L1 ran a nested VM at some point. But as proposed, KVM doesn't actually check if there are any shadow TDP entries to process. That could be fixed by looking at
kvm->arch.indirect_shadow_pages, but even then it's not clear that bailing ifkvm->arch.indirect_shadow_pages> 0 makes sense.E.g. if L1 happens to be running an L2, but <10% of the VM's memory is exposed to L2, then "failure" is pretty much guaranteed to a false positive. And even for the pages that are exposed to L2, "failure" will occur if and only if the pages are being accessed _only_ by L2.
Quoted 4 lines
There most definitely are use cases where the majority of a VM's memory is accessed only by L2. But if those use cases are performing poorly under MGLRU, then IMO we should figure out a way to enhance KVM to do a fast harvest of nested TDP Accessed information, not make MGRLU+KVM suck for a VMs that run nested VMs.
This makes sense. I don't have data today to say that we would get a huge win from speeding up harvesting Accessed information from the shadow MMU would be helpful. Getting this information for the TDP MMU is at least better than no information at all.
Quoted 3 lines
Oh, and calling into
mmu_notifiersto do the "slow" version if the fast version fails is suboptimal.
Agreed. I didn't like this when I wrote it. This can be easily fixed by making mmu_notifier_clear_young() return "fast" and "young or not", which I will do.
Quoted 16 lines
So rather than failing the fast aging, I think what we want is to know if an
mmu_notifierfound a young SPTE during a fast lookup. E.g. something like this in KVM, where usingkvm_has_shadow_mmu_sptes()instead ofkvm_memslots_have_rmaps()is an optional optimization to avoid takingmmu_lockfor write in paths where a (very rare) false negative is acceptable.static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) { return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages); } static int __kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range, bool fast_only) { int young = 0;
Quoted 8 lines
if (!fast_only && kvm_has_shadow_mmu_sptes(kvm)) {
write_lock(&kvm->mmu_lock);
young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
write_unlock(&kvm->mmu_lock);
}
if (tdp_mmu_enabled && kvm_tdp_mmu_age_gfn_range(kvm, range))
young = 1 | MMU_NOTIFY_WAS_FAST;I don't think this line is quite right. We might set MMU_NOTIFY_WAS_FAST even when we took the mmu_lock. I understand what you mean though, thanks.
Quoted 22 lines
return (int)young;}
and then in
lru_gen_look_around():if (spin_is_contended(pvmw->ptl)) return false; /* exclude special VMAs containing anon pages from COW */ if (vma->vm_flags & VM_SPECIAL) return false; young = ptep_clear_young_notify(vma, addr, pte); if (!young) return false; if (!(young & MMU_NOTIFY_WAS_FAST)) return true; young = 1;
Quoted 4 lines
with the lookaround done using
ptep_clear_young_notify_fast().The
MMU_NOTIFY_WAS_FASTflag is gross, but AFAICT it would Just Work without needing to update all users ofptep_clear_young_notify()and friends.
Sounds good to me.
Thanks for all the feedback!
Sean Christopherson
maintainerRe: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 23 lines · James Houghton
Quoted 22 lines · Sean Christopherson
--
mm/mmu_notifier.c +2
-4@@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;@@ -393,9 +394,12 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { -if (subscription->ops->clear_young) - young |= subscription->ops->clear_young (subscription, - mm, start, end);
Quoted 8 lines
mm/mmu_notifier.c int __mmu_notifier_clear_young(struct mm_struct *mm, · line 402
Quoted 6 lines
mm/mmu_notifier.c +6
-0+if (!subscription->ops->clear_young || + fast_only && !subscription->ops->has_fast_aging) +continue ; + + young |= subscription->ops->clear_young (subscription, + mm, start, end);KVM changing
has_fast_agingdynamically would be slow, wouldn't it?
No, it could/would be done quite quickly. But, I'm not suggesting has_fast_aging be dynamic, i.e. it's not an "all aging is guaranteed to be fast", it's a "this MMU _can_ do fast aging". It's a bit fuzzy/weird mostly because KVM can essentially have multiple secondary MMUs wired up to the same mmu_notifier.
Quoted 2 lines
I feel like it's simpler to just pass in
fast_onlyintoclear_youngitself (and this is how I interpreted what you wrote above anyway).
Eh, maybe? A "has_fast_aging" flag is more robust in the sense that it requires secondary MMUs to opt-in, i.e. all secondary MMUs will be considered "slow" by default.
It's somewhat of a moot point because KVM is the only secondary MMU that implements .clear_young() and .test_young() (which I keep forgetting), and that seems unlikely to change.
A flag would also avoid an indirect call and thus a RETPOLINE when CONFIG_RETPOLINE=y, i.e. would be a minor optimization when KVM doesn't suppport fast aging. But that's probably a pretty unlikely combination, so it's probably not a valid argument.
So, I guess I don't have a strong opinion?
Quoted 9 lines
Quoted 4 lines
Double ugh. Peeking ahead at the "failure" code, NAK to adding
kvm_arch_young_notifier_likely_fastfor all the same reasons I objected tokvm_arch_has_test_clear_young()in v1. Please stop trying to do anything like that, I will NAK each every attempt to have coremm/code call directly into KVM.Sorry to make you repeat yourself; I'll leave it out of v6. I don't like it either, but I wasn't sure how important it was to avoid calling into unnecessary notifiers if the TDP MMU were completely disabled.
If it's important, e.g. for performance, then the mmu_notifier should have a flag
so that the behavior doesn't assume a KVM backend. Hence my has_fast_aging
suggestion.
Quoted 9 lines
Quoted 8 lines
Anyways, back to this code, before we spin another version, we need to agree on exactly what behavior we want out of secondary MMUs. Because to me, the behavior proposed in this version doesn't make any sense.
Signalling failure because KVM _might_ have relevant aging information in SPTEs that require taking
kvm->mmu_lockis a terrible tradeoff. And for thetest_youngcase, it's flat out wrong, e.g. if a page is marked Accessed in the TDP MMU, then KVM should return "young", not "failed".
Quoted 8 lines
Sorry for this oversight. What about something like:
1. test (and maybe clear) A bits on TDP MMU
2. If accessed && !should_clear: return (fast)
3. if (fast_only): return (fast)
4. If !(must check shadow MMU): return (fast)
5. test (and maybe clear) A bits in shadow MMU
6. return (slow)
I don't understand where the "must check shadow MMU" in #4 comes from. I also don't think it's necessary; see below.
Quoted 26 lines
Some of this reordering (and maybe a change from
kvm_shadow_root_allocated()to checkingindirect_shadow_pagesor something else) can be done in its own patch.Quoted 22 lines
So rather than failing the fast aging, I think what we want is to know if an
mmu_notifierfound a young SPTE during a fast lookup. E.g. something like this in KVM, where usingkvm_has_shadow_mmu_sptes()instead ofkvm_memslots_have_rmaps()is an optional optimization to avoid takingmmu_lockfor write in paths where a (very rare) false negative is acceptable.static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) { return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages); } static int __kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range, bool fast_only) { int young = 0;if (!fast_only && kvm_has_shadow_mmu_sptes(kvm)) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); }
Quoted 6 lines
Quoted 2 lines
if (tdp_mmu_enabled && kvm_tdp_mmu_age_gfn_range(kvm, range)) young = 1 | MMU_NOTIFY_WAS_FAST;I don't think this line is quite right. We might set
MMU_NOTIFY_WAS_FASTeven when we took themmu_lock. I understand what you mean though, thanks.
The name sucks, but I believe the logic is correct. As posted here in v5, the MGRLU code wants to age both fast _and_ slow MMUs. AIUI, the intent is to always get aging information, but only look around at other PTEs if it can be done fast.
if (should_walk_secondary_mmu()) {
notifier_result =
mmu_notifier_test_clear_young_fast_only(
vma->vm_mm, addr, addr + PAGE_SIZE,
/*clear=*/true);
}
if (notifier_result & MMU_NOTIFIER_FAST_FAILED)
secondary_young = mmu_notifier_clear_young(vma->vm_mm, addr,
addr + PAGE_SIZE);
else {
secondary_young = notifier_result & MMU_NOTIFIER_FAST_YOUNG;
notifier_was_fast = true;
}The change, relative to v5, that I am proposing is that MGLRU looks around if the page was young in _a_ "fast" secondary MMU, whereas v5 looks around if and only if _all_ secondary MMUs are fast.
In other words, if a fast MMU had a young SPTE, look around _that_ MMU, via the fast_only flag.
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 50 lines · Sean Christopherson
Quoted 23 lines · James Houghton
Quoted 22 lines · Sean Christopherson
--
mm/mmu_notifier.c +2
-4@@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;@@ -393,9 +394,12 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { -if (subscription->ops->clear_young) - young |= subscription->ops->clear_young (subscription, - mm, start, end);Quoted 8 lines
mm/mmu_notifier.c int __mmu_notifier_clear_young(struct mm_struct *mm, · line 402
Quoted 6 lines
mm/mmu_notifier.c +6
-0+if (!subscription->ops->clear_young || + fast_only && !subscription->ops->has_fast_aging) +continue ; + + young |= subscription->ops->clear_young (subscription, + mm, start, end);KVM changing
has_fast_agingdynamically would be slow, wouldn't it?No, it could/would be done quite quickly. But, I'm not suggesting
has_fast_agingbe dynamic, i.e. it's not an "all aging is guaranteed to be fast", it's a "this MMU _can_ do fast aging". It's a bit fuzzy/weird mostly because KVM can essentially have multiple secondary MMUs wired up to the samemmu_notifier.Quoted 2 lines
I feel like it's simpler to just pass in
fast_onlyintoclear_youngitself (and this is how I interpreted what you wrote above anyway).Eh, maybe? A "
has_fast_aging" flag is more robust in the sense that it requires secondary MMUs to opt-in, i.e. all secondary MMUs will be considered "slow" by default.It's somewhat of a moot point because KVM is the only secondary MMU that implements .clear_young() and .test_young() (which I keep forgetting), and that seems unlikely to change.
Quoted 5 lines
A flag would also avoid an indirect call and thus a RETPOLINE when
CONFIG_RETPOLINE=y, i.e. would be a minor optimization when KVM doesn't suppport fast aging. But that's probably a pretty unlikely combination, so it's probably not a valid argument.So, I guess I don't have a strong opinion?
(Sorry for the somewhat delayed response... spent some time actually writing what this would look like.)
I see what you mean, thanks! So has_fast_aging might be set by KVM if the architecture sets a Kconfig saying that it understands the concept of fast aging, basically what the presence of this v5's test_clear_young_fast_only() indicates.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Double ugh. Peeking ahead at the "failure" code, NAK to adding
kvm_arch_young_notifier_likely_fastfor all the same reasons I objected tokvm_arch_has_test_clear_young()in v1. Please stop trying to do anything like that, I will NAK each every attempt to have coremm/code call directly into KVM.
Quoted 8 lines
Quoted 4 lines
Sorry to make you repeat yourself; I'll leave it out of v6. I don't like it either, but I wasn't sure how important it was to avoid calling into unnecessary notifiers if the TDP MMU were completely disabled.
If it's important, e.g. for performance, then the
mmu_notifiershould have a flag
so that the behavior doesn't assume a KVM backend. Hence myhas_fast_aging
suggestion.
Thanks! That makes sense.
Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Anyways, back to this code, before we spin another version, we need to agree on exactly what behavior we want out of secondary MMUs. Because to me, the behavior proposed in this version doesn't make any sense.
Signalling failure because KVM _might_ have relevant aging information in SPTEs that require taking
kvm->mmu_lockis a terrible tradeoff. And for thetest_youngcase, it's flat out wrong, e.g. if a page is marked Accessed in the TDP MMU, then KVM should return "young", not "failed".Quoted 8 lines
Sorry for this oversight. What about something like:
1. test (and maybe clear) A bits on TDP MMU
2. If accessed && !should_clear: return (fast)
3. if (fast_only): return (fast)
4. If !(must check shadow MMU): return (fast)
5. test (and maybe clear) A bits in shadow MMU
6. return (slow)
Quoted 2 lines
I don't understand where the "must check shadow MMU" in #4 comes from. I also don't think it's necessary; see below.
I just meant kvm_has_shadow_mmu_sptes() or kvm_memslots_have_rmaps(). I like the logic you suggest below. :)
Quoted 3 lines
Quoted 3 lines
Some of this reordering (and maybe a change from
kvm_shadow_root_allocated()to checkingindirect_shadow_pagesor something else) can be done in its own patch.
So just to be clear, for test_young(), I intend to have a patch in v6 to elide the shadow MMU check if the TDP MMU indicates Accessed. Seems like a pure win; no reason not to include it if we're making logic changes here anyway.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
So rather than failing the fast aging, I think what we want is to know if an
mmu_notifierfound a young SPTE during a fast lookup. E.g. something like this in KVM, where usingkvm_has_shadow_mmu_sptes()instead ofkvm_memslots_have_rmaps()is an optional optimization to avoid takingmmu_lockfor write in paths where a (very rare) false negative is acceptable.static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) { return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages); } static int __kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range, bool fast_only) { int young = 0;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
if (!fast_only && kvm_has_shadow_mmu_sptes(kvm)) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); } if (tdp_mmu_enabled && kvm_tdp_mmu_age_gfn_range(kvm, range)) young = 1 | MMU_NOTIFY_WAS_FAST;
The most straightforward way (IMHO) to return something like 1 | MMU_NOTIFY_WAS_FAST up to the MMU notifier itself is to make gfn_handler_t return int instead of bool.
In this v5, I worked around this need by using bool *failed in patch 5[1]. I think the way this is going to look now in v6 would be cleaner by actually changing gfn_handler_t to return int, and then we can write something like what you wrote here. What do you think?
[1]: https://lore.kernel.org/linux-mm/[email protected]/
Quoted 23 lines
Quoted 3 lines
I don't think this line is quite right. We might set
MMU_NOTIFY_WAS_FASTeven when we took themmu_lock. I understand what you mean though, thanks.The name sucks, but I believe the logic is correct. As posted here in v5, the MGRLU code wants to age both fast _and_ slow MMUs. AIUI, the intent is to always get aging information, but only look around at other PTEs if it can be done fast.
if (should_walk_secondary_mmu()) { notifier_result = mmu_notifier_test_clear_young_fast_only( vma->vm_mm, addr, addr + PAGE_SIZE, /*clear=*/true); } if (notifier_result & MMU_NOTIFIER_FAST_FAILED) secondary_young = mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); else { secondary_young = notifier_result & MMU_NOTIFIER_FAST_YOUNG; notifier_was_fast = true; }
Quoted 6 lines
The change, relative to v5, that I am proposing is that MGLRU looks around if the page was young in _a_ "fast" secondary MMU, whereas v5 looks around if and only if _all_ secondary MMUs are fast.
In other words, if a fast MMU had a young SPTE, look around _that_ MMU, via the
fast_onlyflag.
Oh, yeah, that's a lot more intelligent than what I had. I think I fully understand your suggestion; I guess we'll see in v6. :)
I wonder if this still makes sense if whether or not an MMU is "fast" is determined by how contended some lock(s) are at the time. I think it does, but I guess we can discuss more if it turns out that having an architecture participate like this is actually something we want to do (i.e., that performance results say it's a good idea).
Thanks Sean!
Sean Christopherson
maintainerRe: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 7 lines · James Houghton
Quoted 5 lines · Sean Christopherson
A flag would also avoid an indirect call and thus a RETPOLINE when
CONFIG_RETPOLINE=y, i.e. would be a minor optimization when KVM doesn't suppport fast aging. But that's probably a pretty unlikely combination, so it's probably not a valid argument.So, I guess I don't have a strong opinion?
Quoted 7 lines
(Sorry for the somewhat delayed response... spent some time actually writing what this would look like.)
I see what you mean, thanks! So
has_fast_agingmight be set by KVM if the architecture sets aKconfigsaying that it understands the concept of fast aging, basically what the presence of this v5'stest_clear_young_fast_only()indicates.
It would need to be a runtime setting, because KVM x86-64 with tdp_mmu_enabled=false doesn't support fast aging (uses the shadow MMU even for TDP).
Quoted 6 lines
Quoted 2 lines
I don't understand where the "must check shadow MMU" in #4 comes from. I also don't think it's necessary; see below.
I just meant
kvm_has_shadow_mmu_sptes()orkvm_memslots_have_rmaps(). I like the logic you suggest below. :)
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Some of this reordering (and maybe a change from
kvm_shadow_root_allocated()to checkingindirect_shadow_pagesor something else) can be done in its own patch.So just to be clear, for
test_young(), I intend to have a patch in v6 to elide the shadow MMU check if the TDP MMU indicates Accessed. Seems like a pure win; no reason not to include it if we're making logic changes here anyway.
I don't think that's correct. The initial fast_only=false aging should process shadow MMUs (nested TDP) and TDP MMUs, otherwise a future fast_only=false would get a false positive on young due to failing to clear the Accessed bit in the shadow MMU. E.g. if page X is accessed by both L1 and L2, then aged, and never accessed again, the Accessed bit would still be set in the page tables for L2.
My thought for MMU_NOTIFY_WAS_FAST below (which again is a bad name) is to communicate to MGLRU that the page was found to be young in an MMU that supports fast aging, i.e. that looking around at other SPTEs is worth doing.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
So rather than failing the fast aging, I think what we want is to know if an
mmu_notifierfound a young SPTE during a fast lookup. E.g. something like this in KVM, where usingkvm_has_shadow_mmu_sptes()instead ofkvm_memslots_have_rmaps()is an optional optimization to avoid takingmmu_lockfor write in paths where a (very rare) false negative is acceptable.static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) { return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages); } static int __kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range, bool fast_only) { int young = 0;if (!fast_only && kvm_has_shadow_mmu_sptes(kvm)) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); }
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
if (tdp_mmu_enabled && kvm_tdp_mmu_age_gfn_range(kvm, range)) young = 1 | MMU_NOTIFY_WAS_FAST;The most straightforward way (IMHO) to return something like
1 | MMU_NOTIFY_WAS_FASTup to the MMU notifier itself is to makegfn_handler_treturn int instead of bool.
Hrm, all the options are unpleasant. Modifying gfn_handler_t to return an int will require an absurd amount of churn (all implementations in all archictures), and I don't love that the APIs that return true/false to indicate "flush" would lose their boolean-ness.
One idea would be to add kvm_mmu_notifier_arg.aging_was_fast or so, and then refactor kvm_handle_hva_range_no_flush() into a dedicated aging helper, and have it morph the KVM-internal flag into an MMU_NOTIFIER flag. It's not perect either, but it requires far less churn and keeps some of the KVM<=>mmu_notifer details in common KVM code.
include/linux/kvm_host.h +1 -0
@@ -258,6 +258,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg {unsigned long attributes; +bool aging_was_fast; }; struct kvm_gfn_range {
virt/kvm/kvm_main.c +12 -23
@@ -685,10 +685,10 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,return __kvm_handle_hva_range (kvm, &range).ret; } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, -unsigned long start, -unsigned long end, -gfn_handler_t handler) +static __always_inline int kvm_age_hva_range(struct mmu_notifier *mn, +unsigned long start, +unsigned long end, +bool flush_if_young) {struct kvm *kvm =mmu_notifier_to_kvm (mn);const struct kvm_mmu_notifier_range range = {
@@ -696,11 +696,14 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .end = end, .handler = handler, .on_lock = (void *)kvm_null_fn, - .flush_on_ret = false, + .flush_on_ret = flush_if_young, .may_block = false, + .aging_was_fast = false, }; -return __kvm_handle_hva_range (kvm, &range).ret; +bool young =__kvm_handle_hva_range (kvm, &range).ret; + +return (int )young | (range.aging_was_fast ?MMU_NOTIFIER_FAST_AGING :0 ); }void kvm_mmu_invalidate_begin (struct kvm *kvm)
@@ -865,7 +868,7 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, {trace_kvm_age_hva (start, end); -return kvm_handle_hva_range (mn, start, end, kvm_age_gfn); +return kvm_age_hva_range (mn, start, end, true); } static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
@@ -875,20 +878,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, {trace_kvm_age_hva (start, end); -/* -* Even though we do not flush TLB, this will still adversely -* affect performance on pre-Haswell Intel EPT, where there is -* no EPT Access Bit to clear so that we have to tear down EPT -* tables instead. If we find this unacceptable, we can always -* add a parameter to kvm_age_hva so that it effectively doesn't -* do anything on clear_young. -* -* Also note that currently we never issue secondary TLB flushes -* from clear_young, leaving this job up to the regular system -* cadence. If we find this inaccurate, we might come up with a -* more sophisticated heuristic later. -*/ -return kvm_handle_hva_range_no_flush (mn, start, end, kvm_age_gfn); +return kvm_age_hva_range (mn, start, end, false); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,
@@ -897,8 +887,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, {trace_kvm_test_age_hva (address); -return kvm_handle_hva_range_no_flush (mn, address, address +1 , - kvm_test_age_gfn); +return kvm_age_hva_range (mn, address, address +1 , false); } static void kvm_mmu_notifier_release(struct mmu_notifier *mn,
Quoted 4 lines
Quoted 4 lines
The change, relative to v5, that I am proposing is that MGLRU looks around if the page was young in _a_ "fast" secondary MMU, whereas v5 looks around if and only if _all_ secondary MMUs are fast.
Quoted 8 lines
Quoted 2 lines
In other words, if a fast MMU had a young SPTE, look around _that_ MMU, via the
fast_onlyflag.Oh, yeah, that's a lot more intelligent than what I had. I think I fully understand your suggestion; I guess we'll see in v6. :)
I wonder if this still makes sense if whether or not an MMU is "fast" is determined by how contended some lock(s) are at the time.
No. Just because a lock wasn't contended on the initial aging doesn't mean it won't be contended on the next round. E.g. when using KVM x86's shadow MMU, which takes mmu_lock for write for all operations, an aging operation could get lucky and sneak in while mmu_lock happened to be free, but then get stuck behind a large queue of operations.
The fast-ness needs to be predictable and all but guaranteed, i.e. lockless or in an MMU that takes mmu_lock for read in all but the most rare paths.
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · James Houghton
Quoted 5 lines · Sean Christopherson
A flag would also avoid an indirect call and thus a RETPOLINE when
CONFIG_RETPOLINE=y, i.e. would be a minor optimization when KVM doesn't suppport fast aging. But that's probably a pretty unlikely combination, so it's probably not a valid argument.So, I guess I don't have a strong opinion?
(Sorry for the somewhat delayed response... spent some time actually writing what this would look like.)
Quoted 7 lines
Quoted 4 lines
I see what you mean, thanks! So
has_fast_agingmight be set by KVM if the architecture sets aKconfigsaying that it understands the concept of fast aging, basically what the presence of this v5'stest_clear_young_fast_only()indicates.It would need to be a runtime setting, because KVM x86-64 with tdp_mmu_enabled=false doesn't support fast aging (uses the shadow MMU even for TDP).
I see. I'm not sure if it makes sense to put this in ops as you originally had it then (it seems like a bit of a pain anyway). I could just make it a member of struct mmu_notifier itself.
Quoted 10 lines
Quoted 4 lines
So just to be clear, for
test_young(), I intend to have a patch in v6 to elide the shadow MMU check if the TDP MMU indicates Accessed. Seems like a pure win; no reason not to include it if we're making logic changes here anyway.I don't think that's correct. The initial fast_only=false aging should process shadow MMUs (nested TDP) and TDP MMUs, otherwise a future fast_only=false would get a false positive on young due to failing to clear the Accessed bit in the shadow MMU. E.g. if page X is accessed by both L1 and L2, then aged, and never accessed again, the Accessed bit would still be set in the page tables for L2.
For clear_young(fast_only=false), yeah we need to check and clear Accessed for both MMUs. But for test_young(fast_only=false), I don't see why we couldn't just return early if the TDP MMU reports young.
Quoted 3 lines
My thought for
MMU_NOTIFY_WAS_FASTbelow (which again is a bad name) is to communicate to MGLRU that the page was found to be young in an MMU that supports fast aging, i.e. that looking around at other SPTEs is worth doing.
That makes sense; I don't think this little test_young() optimization affects that.
Quoted 34 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
So rather than failing the fast aging, I think what we want is to know if an
mmu_notifierfound a young SPTE during a fast lookup. E.g. something like this in KVM, where usingkvm_has_shadow_mmu_sptes()instead ofkvm_memslots_have_rmaps()is an optional optimization to avoid takingmmu_lockfor write in paths where a (very rare) false negative is acceptable.static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) { return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages); } static int __kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range, bool fast_only) { int young = 0;if (!fast_only && kvm_has_shadow_mmu_sptes(kvm)) { write_lock(&kvm->mmu_lock); young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); write_unlock(&kvm->mmu_lock); }Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
if (tdp_mmu_enabled && kvm_tdp_mmu_age_gfn_range(kvm, range)) young = 1 | MMU_NOTIFY_WAS_FAST;The most straightforward way (IMHO) to return something like
1 | MMU_NOTIFY_WAS_FASTup to the MMU notifier itself is to makegfn_handler_treturn int instead of bool.Hrm, all the options are unpleasant. Modifying
gfn_handler_tto return an int will require an absurd amount of churn (all implementations in all archictures), and I don't love that the APIs that return true/false to indicate "flush" would lose their boolean-ness.
Quoted 5 lines
One idea would be to add kvm_mmu_notifier_arg.aging_was_fast or so, and then refactor
kvm_handle_hva_range_no_flush()into a dedicated aging helper, and have it morph the KVM-internal flag into anMMU_NOTIFIERflag. It's not perect either, but it requires far less churn and keeps some of the KVM<=>mmu_notifer details in common KVM code.
SGTM. I think this will work. Thanks!
Quoted 104 lines
include/linux/kvm_host.h +1
-0@@ -258,6 +258,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg {unsigned long attributes; +bool aging_was_fast; }; struct kvm_gfn_range {virt/kvm/kvm_main.c +12
-23@@ -685,10 +685,10 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,return __kvm_handle_hva_range (kvm, &range).ret; } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, -unsigned long start, -unsigned long end, -gfn_handler_t handler) +static __always_inline int kvm_age_hva_range(struct mmu_notifier *mn, +unsigned long start, +unsigned long end, +bool flush_if_young) {struct kvm *kvm =mmu_notifier_to_kvm (mn);const struct kvm_mmu_notifier_range range = {@@ -696,11 +696,14 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .end = end, .handler = handler, .on_lock = (void *)kvm_null_fn, - .flush_on_ret = false, + .flush_on_ret = flush_if_young, .may_block = false, + .aging_was_fast = false, }; -return __kvm_handle_hva_range (kvm, &range).ret; +bool young =__kvm_handle_hva_range (kvm, &range).ret; + +return (int )young | (range.aging_was_fast ?MMU_NOTIFIER_FAST_AGING :0 ); }void kvm_mmu_invalidate_begin (struct kvm *kvm)@@ -865,7 +868,7 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, {trace_kvm_age_hva (start, end); -return kvm_handle_hva_range (mn, start, end, kvm_age_gfn); +return kvm_age_hva_range (mn, start, end, true); } static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,@@ -875,20 +878,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, {trace_kvm_age_hva (start, end); -/* -* Even though we do not flush TLB, this will still adversely -* affect performance on pre-Haswell Intel EPT, where there is -* no EPT Access Bit to clear so that we have to tear down EPT -* tables instead. If we find this unacceptable, we can always -* add a parameter to kvm_age_hva so that it effectively doesn't -* do anything on clear_young. -* -* Also note that currently we never issue secondary TLB flushes -* from clear_young, leaving this job up to the regular system -* cadence. If we find this inaccurate, we might come up with a -* more sophisticated heuristic later. -*/ -return kvm_handle_hva_range_no_flush (mn, start, end, kvm_age_gfn); +return kvm_age_hva_range (mn, start, end, false); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,@@ -897,8 +887,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, {trace_kvm_test_age_hva (address); -return kvm_handle_hva_range_no_flush (mn, address, address +1 , - kvm_test_age_gfn); +return kvm_age_hva_range (mn, address, address +1 , false); } static void kvm_mmu_notifier_release(struct mmu_notifier *mn,Quoted 6 lines
Quoted 6 lines
The change, relative to v5, that I am proposing is that MGLRU looks around if the page was young in _a_ "fast" secondary MMU, whereas v5 looks around if and only if _all_ secondary MMUs are fast.
Quoted 8 lines
Quoted 2 lines
In other words, if a fast MMU had a young SPTE, look around _that_ MMU, via the
fast_onlyflag.Oh, yeah, that's a lot more intelligent than what I had. I think I fully understand your suggestion; I guess we'll see in v6. :)
I wonder if this still makes sense if whether or not an MMU is "fast" is determined by how contended some lock(s) are at the time.
Quoted 8 lines
No. Just because a lock wasn't contended on the initial aging doesn't mean it won't be contended on the next round. E.g. when using KVM x86's shadow MMU, which takes
mmu_lockfor write for all operations, an aging operation could get lucky and sneak in whilemmu_lockhappened to be free, but then get stuck behind a large queue of operations.The fast-ness needs to be predictable and all but guaranteed, i.e. lockless or in an MMU that takes
mmu_lockfor read in all but the most rare paths.
Aging and look-around themselves only use the fast-only notifiers, so they won't ever wait on a lock (well... provided KVM is written like that, which I think is a given). should_look_around() will use the slow notifier because it (despite its name) is responsible for accurately determining if a page is young lest we evict a young page.
So in this case where "fast" means "lock not contended for now", I don't think it's necessarily wrong for MGLRU to attempt to find young pages, even if sometimes it will bail out because a lock is contended/held for a few or even a majority of the pages. Not doing look-around is the same as doing look-around and finding that no pages are young.
Anyway, I don't think this bit is really all that important unless we can demonstrate that KVM participating like this actually results in a measurable win.
Sean Christopherson
maintainerRe: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 18 lines · James Houghton
Quoted 17 lines · Sean Christopherson
Quoted 14 lines · James Houghton
Quoted 5 lines · Sean Christopherson
A flag would also avoid an indirect call and thus a RETPOLINE when
CONFIG_RETPOLINE=y, i.e. would be a minor optimization when KVM doesn't suppport fast aging. But that's probably a pretty unlikely combination, so it's probably not a valid argument.So, I guess I don't have a strong opinion?
(Sorry for the somewhat delayed response... spent some time actually writing what this would look like.)
I see what you mean, thanks! So
has_fast_agingmight be set by KVM if the architecture sets aKconfigsaying that it understands the concept of fast aging, basically what the presence of this v5'stest_clear_young_fast_only()indicates.
Quoted 6 lines
Quoted 2 lines
It would need to be a runtime setting, because KVM x86-64 with tdp_mmu_enabled=false doesn't support fast aging (uses the shadow MMU even for TDP).
I see. I'm not sure if it makes sense to put this in
opsas you originally had it then (it seems like a bit of a pain anyway). I could just make it a member ofstruct mmu_notifieritself.
Ah, right, because the ops are const. Yeah, losing the const just to set a flag would be unfortunate.
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
So just to be clear, for
test_young(), I intend to have a patch in v6 to elide the shadow MMU check if the TDP MMU indicates Accessed. Seems like a pure win; no reason not to include it if we're making logic changes here anyway.I don't think that's correct. The initial fast_only=false aging should process shadow MMUs (nested TDP) and TDP MMUs, otherwise a future fast_only=false would get a false positive on young due to failing to clear the Accessed bit in the shadow MMU. E.g. if page X is accessed by both L1 and L2, then aged, and never accessed again, the Accessed bit would still be set in the page tables for L2.
Quoted 3 lines
For
clear_young(fast_only=false), yeah we need to check and clear Accessed for both MMUs. But fortest_young(fast_only=false), I don't see why we couldn't just return early if the TDP MMU reports young.
Ooh, good point.
Quoted 12 lines
Quoted 12 lines
Quoted 5 lines
Oh, yeah, that's a lot more intelligent than what I had. I think I fully understand your suggestion; I guess we'll see in v6. :)
I wonder if this still makes sense if whether or not an MMU is "fast" is determined by how contended some lock(s) are at the time.
No. Just because a lock wasn't contended on the initial aging doesn't mean it won't be contended on the next round. E.g. when using KVM x86's shadow MMU, which takes
mmu_lockfor write for all operations, an aging operation could get lucky and sneak in whilemmu_lockhappened to be free, but then get stuck behind a large queue of operations.
Quoted 6 lines
Quoted 2 lines
The fast-ness needs to be predictable and all but guaranteed, i.e. lockless or in an MMU that takes
mmu_lockfor read in all but the most rare paths.Aging and look-around themselves only use the fast-only notifiers, so they won't ever wait on a lock (well... provided KVM is written like that, which I think is a given).
Regarding aging, is that actually the behavior that we want? I thought the plan is to have the initial test look at all MMUs, i.e. be potentially slow, but only do the lookaround if it can be fast. IIUC, that was Yu's intent (and peeking back at v2, that is indeed the case, unless I'm misreading the code).
If KVM _never_ consults shadow (nested TDP) MMUs, then a VM running an L2 will end up with hot pages (used by L2) swapped out.
Or are you saying that the "test" could be slow, but not "clear"? That's also suboptimal, because any pages accessed by L2 will always appear hot.
Quoted 5 lines
should_look_around()will use the slow notifier because it (despite its name) is responsible for accurately determining if a page is young lest we evict a young page.So in this case where "fast" means "lock not contended for now",
No. In KVM, "fast" needs to be a property of the MMU, not a reflection of system state at some random snapshot in time.
Quoted 2 lines
I don't think it's necessarily wrong for MGLRU to attempt to find young pages, even if sometimes it will bail out because a lock is contended/held
lru_gen_look_around() skips lookaround if something else is waiting on the page table lock, but that is a far cry from what KVM would be doing. (a) the PTL is already held, and (b) it is scoped precisely to the range being processed. Not looking around makes sense because there's a high probability of the PTEs in question being modified by a different task, i.e. of the look around being a waste of time.
In KVM, mmu_lock is not yet held, so KVM would need to use try-lock to avoid waiting, and would need to bail from the middle of the aging walk if a different task contends mmu_lock.
I agree that's not "wrong", but in part because mmu_lock is scoped to the entire VM, it risks ending up with semi-random, hard to debug behavior. E.g. a user could see intermittent "failures" that come and go based on seemingly unrelated behavior in KVM. And implementing the "bail" behavior in the shadow MMU would require non-trivial changes.
In other words, I would very strongly prefer that the shadow MMU be all or nothing, i.e. is either part of look-around or isn't. And if nested TDP doesn't fair well with MGLRU, then we (or whoever cares) can spend the time+effort to make it work with fast-aging.
Ooh! Actually, after fiddling a bit to see how feasible fast-aging in the shadow MMU would be, I'm pretty sure we can do straight there for nested TDP. Or rather, I suspect/hope we can get close enough for an initial merge, which would allow aging_is_fast to be a property of the mmu_notifier, i.e. would simplify things because KVM wouldn't need to communicate MMU_NOTIFY_WAS_FAST for each notification.
Walking KVM's rmaps requires mmu_lock because adding/removing rmap entries is done in such a way that a lockless walk would be painfully complex. But if there is exactly _one_ rmap entry for a gfn, then slot->arch.rmap[...] points directly at that one SPTE. And with nested TDP, unless L1 is doing something uncommon, e.g. mapping the same page into multiple L2s, that overwhelming vast majority of rmaps have only one entry. That's not the case for legacy shadow paging because kernels almost always map a pfn using multiple virtual addresses, e.g. Linux's direct map along with any userspace mappings.
E.g. with a QEMU+KVM setup running Linux as L2, in my setup, only one gfn has multiple rmap entries (IIRC, it's from QEMU remapping BIOS into low memory during boot).
So, if we bifurcate aging behavior based on whether or not the TDP MMU is enabled, then whether or not aging is fast is constant (after KVM loads). Rougly, the KVM side of things would be the below, plus a bunch of conversions to WRITE_ONCE() to ensure a stable rmap value (KVM already plays nice with lockless accesses to SPTEs, thanks to the fast page fault path).
If KVM adds an rmap entry after the READ_ONCE(), then functionally all is still well because the original SPTE pointer is still valid. If the rmap entry is removed, then KVM just needs to ensure the owning page table isn't freed. That could be done either with a cmpxchg() (KVM zaps leafs SPTE before freeing page tables, and the rmap stuff doesn't actually walk upper level entries), or by enhancing the shadow MMU's lockless walk logic to allow lockless walks from non-vCPU tasks.
And in the (hopefully) unlikely scenario someone has a use case where L1 maps a gfn into multiple L2s (or aliases in bizarre ways), then we can tackle making the nested TDP shadow MMU rmap walks always lockless.
E.g. again very roughly, if we went with the latter:
@@ -1629,22 +1629,45 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot,
__rmap_add(vcpu->kvm, cache, slot, spte, gfn, access);
}
+static __always_inline bool kvm_handle_gfn_range_lockless(struct kvm *kvm,
+ struct kvm_gfn_range *range,
+ typedefme handler)
+{
+ gfn_t gfn; + int level; + u64 *spte; + bool ret;
+
+ walk_shadow_page_lockless_begin(???);
+
+ for (gfn = range->start; gfn < range->end; gfn++) {
+ for (level = PG_LEVEL_4K; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
+ spte = (void *)READ_ONCE(gfn_to_rmap(gfn, level, range->slot)->val);
+
+ /* Skip the gfn if there are multiple SPTEs. */
+ if ((unsigned long)spte & 1)
+ continue;
+
+ ret |= handler(spte);
+ }
+ }
+
+ walk_shadow_page_lockless_end(???);
+}
+
static int __kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range,
bool fast_only)
{
bool young = false;- if (kvm_memslots_have_rmaps(kvm)) {
- if (fast_only)
- return -1;
-
- write_lock(&kvm->mmu_lock);
- young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
- write_unlock(&kvm->mmu_lock);
- }
-
- if (tdp_mmu_enabled)
+ if (tdp_mmu_enabled) {
young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
+ young |= kvm_handle_gfn_range_lockless(kvm, range, kvm_age_rmap_fast);
+ } else if (!fast_only) {
+ write_lock(&kvm->mmu_lock);
+ young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
+ write_unlock(&kvm->mmu_lock);+ }
return (int)young;
}
Quoted 2 lines
for a few or even a majority of the pages. Not doing look-around is the same as doing look-around and finding that no pages are young.
No, because the former is deterministic and predictable, the latter is not.
Quoted 3 lines
Anyway, I don't think this bit is really all that important unless we can demonstrate that KVM participating like this actually results in a measurable win.
Participating like what? You've lost me a bit. Are we talking past each other?
What I am saying is that we do this (note that this is slightly different than an earlier sketch; I botched the ordering of spin_is_contend() in that one, and didn't account for the page being young in the primary MMU).
if (pte_young(ptep_get(pte)))
young = 1 | MMU_NOTIFY_WAS_FAST;
young |= ptep_clear_young_notify(vma, addr, pte);
if (!young)
return false;
if (!(young & MMU_NOTIFY_WAS_FAST))
return true;
if (spin_is_contended(pvmw->ptl))
return false;
/* exclude special VMAs containing anon pages from COW */
if (vma->vm_flags & VM_SPECIAL)
return false;Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 17 lines · Sean Christopherson
Quoted 15 lines · James Houghton
Quoted 13 lines · Sean Christopherson
Quoted 2 lines · James Houghton
I wonder if this still makes sense if whether or not an MMU is "fast" is determined by how contended some lock(s) are at the time.
No. Just because a lock wasn't contended on the initial aging doesn't mean it won't be contended on the next round. E.g. when using KVM x86's shadow MMU, which takes
mmu_lockfor write for all operations, an aging operation could get lucky and sneak in whilemmu_lockhappened to be free, but then get stuck behind a large queue of operations.The fast-ness needs to be predictable and all but guaranteed, i.e. lockless or in an MMU that takes
mmu_lockfor read in all but the most rare paths.
Quoted 8 lines
Quoted 3 lines
Aging and look-around themselves only use the fast-only notifiers, so they won't ever wait on a lock (well... provided KVM is written like that, which I think is a given).
Regarding aging, is that actually the behavior that we want? I thought the plan is to have the initial test look at all MMUs, i.e. be potentially slow, but only do the lookaround if it can be fast. IIUC, that was Yu's intent (and peeking back at v2, that is indeed the case, unless I'm misreading the code).
I believe what I said is correct. There are three separate things going on here:
1. Aging (when we hit the low watermark, scan PTEs to find young pages)
2. Eviction (pick a page to evict; if it is definitely not young, evict it)
3. Look-around (upon finding a page is young upon attempted eviction, check adjacent pages if they are young too)
(1) and (3) both use the fast-only notifier, (2) does not. In v2[1], this is true, as in the (1) and (3) paths, the notifier being called is the test_clear_young() notifier with the bitmap present. If the bitmap is present, the shadow MMU is not consulted.
For (2), there is an mmu_notifier_clear_young() with no bitmap called in should_look_around(), just like in this v5. (2) is the only one that needs to use the slow notifier; the (1) and (3) are best-effort attempts to improve the decision for which pages to evict.
[1]: https://lore.kernel.org/linux-mm/[email protected]/
Quoted 3 lines
If KVM _never_ consults shadow (nested TDP) MMUs, then a VM running an L2 will end up with hot pages (used by L2) swapped out.
The shadow MMU is consulted at eviction time -- only at eviction time. So pages used by L2 won't be swapped out unless they're still cold at eviction time.
In my (and Yu's) head, not being able to do aging for nested TDP is ok because running nested VMs is much more rare than running non-nested VMs. And in the non-nested case, being able to do aging is a strict improvement over what we have now.
We could look into being able to do aging with the shadow MMU, but I don't think that should necessarily block this series.
Quoted 2 lines
Or are you saying that the "test" could be slow, but not "clear"? That's also suboptimal, because any pages accessed by L2 will always appear hot.
No. I hope what I said above makes sense.
Quoted 8 lines
Quoted 5 lines
should_look_around()will use the slow notifier because it (despite its name) is responsible for accurately determining if a page is young lest we evict a young page.So in this case where "fast" means "lock not contended for now",
No. In KVM, "fast" needs to be a property of the MMU, not a reflection of system state at some random snapshot in time.
Ok, that's fine with me.
Quoted 14 lines
Quoted 2 lines
I don't think it's necessarily wrong for MGLRU to attempt to find young pages, even if sometimes it will bail out because a lock is contended/held
lru_gen_look_around()skips lookaround if something else is waiting on the page table lock, but that is a far cry from what KVM would be doing. (a) the PTL is already held, and (b) it is scoped precisely to the range being processed. Not looking around makes sense because there's a high probability of the PTEs in question being modified by a different task, i.e. of the look around being a waste of time.In KVM,
mmu_lockis not yet held, so KVM would need to use try-lock to avoid waiting, and would need to bail from the middle of the aging walk if a different task contendsmmu_lock.
Quoted 5 lines
I agree that's not "wrong", but in part because
mmu_lockis scoped to the entire VM, it risks ending up with semi-random, hard to debug behavior. E.g. a user could see intermittent "failures" that come and go based on seemingly unrelated behavior in KVM. And implementing the "bail" behavior in the shadow MMU would require non-trivial changes.
This is a great point, thanks Sean. I won't send any patches that make other architectures trylock() for the fast notifier.
Quoted 91 lines
In other words, I would very strongly prefer that the shadow MMU be all or nothing, i.e. is either part of look-around or isn't. And if nested TDP doesn't fair well with MGLRU, then we (or whoever cares) can spend the time+effort to make it work with fast-aging.
Ooh! Actually, after fiddling a bit to see how feasible fast-aging in the shadow MMU would be, I'm pretty sure we can do straight there for nested TDP. Or rather, I suspect/hope we can get close enough for an initial merge, which would allow
aging_is_fastto be a property of themmu_notifier, i.e. would simplify things because KVM wouldn't need to communicateMMU_NOTIFY_WAS_FASTfor each notification.Walking KVM's rmaps requires
mmu_lockbecause adding/removing rmap entries is done in such a way that a lockless walk would be painfully complex. But if there is exactly _one_ rmap entry for a gfn, then slot->arch.rmap[...] points directly at that one SPTE. And with nested TDP, unless L1 is doing something uncommon, e.g. mapping the same page into multiple L2s, that overwhelming vast majority of rmaps have only one entry. That's not the case for legacy shadow paging because kernels almost always map a pfn using multiple virtual addresses, e.g. Linux's direct map along with any userspace mappings.E.g. with a QEMU+KVM setup running Linux as L2, in my setup, only one gfn has multiple rmap entries (IIRC, it's from QEMU remapping BIOS into low memory during boot).
So, if we bifurcate aging behavior based on whether or not the TDP MMU is enabled, then whether or not aging is fast is constant (after KVM loads). Rougly, the KVM side of things would be the below, plus a bunch of conversions to
WRITE_ONCE()to ensure a stable rmap value (KVM already plays nice with lockless accesses to SPTEs, thanks to the fast page fault path).If KVM adds an rmap entry after the
READ_ONCE(), then functionally all is still well because the original SPTE pointer is still valid. If the rmap entry is removed, then KVM just needs to ensure the owning page table isn't freed. That could be done either with acmpxchg()(KVM zaps leafs SPTE before freeing page tables, and the rmap stuff doesn't actually walk upper level entries), or by enhancing the shadow MMU's lockless walk logic to allow lockless walks from non-vCPU tasks.And in the (hopefully) unlikely scenario someone has a use case where L1 maps a gfn into multiple L2s (or aliases in bizarre ways), then we can tackle making the nested TDP shadow MMU rmap walks always lockless.
E.g. again very roughly, if we went with the latter:
(quoted diff) +27
-10@@ -1629,22 +1629,45 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot, __rmap_add(vcpu->kvm, cache, slot, spte, gfn, access); } +static __always_inline bool kvm_handle_gfn_range_lockless(struct kvm *kvm, + struct kvm_gfn_range *range, + typedefme handler) +{ + gfn_t gfn; + int level; + u64 *spte; + bool ret; + + walk_shadow_page_lockless_begin(???); + + for (gfn = range->start; gfn < range->end; gfn++) { + for (level = PG_LEVEL_4K; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) { + spte = (void *)READ_ONCE(gfn_to_rmap(gfn, level, range->slot)->val); + + /* Skip the gfn if there are multiple SPTEs. */ + if ((unsigned long)spte & 1) + continue; + + ret |= handler(spte); + } + } + + walk_shadow_page_lockless_end(???); +} + static int __kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range, bool fast_only) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) { - if (fast_only) - return -1; - - write_lock(&kvm->mmu_lock); - young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); - write_unlock(&kvm->mmu_lock); - } - - if (tdp_mmu_enabled) + if (tdp_mmu_enabled) {
Review static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot, · line 1673
Quoted 10 lines
(quoted diff) +6
-0young |= kvm_tdp_mmu_age_gfn_range(kvm, range); + young |= kvm_handle_gfn_range_lockless(kvm, range, kvm_age_rmap_fast); + } else if (!fast_only) { + write_lock(&kvm->mmu_lock); + young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + write_unlock(&kvm->mmu_lock); + } return (int)young; }
Hmm, interesting. I need to spend a little bit more time digesting this.
Would you like to see this included in v6? (It'd be nice to avoid the WAS_FAST stuff....) Should we leave it for a later series? I haven't formed my own opinion yet.
Quoted 4 lines
Quoted 2 lines
for a few or even a majority of the pages. Not doing look-around is the same as doing look-around and finding that no pages are young.
No, because the former is deterministic and predictable, the latter is not.
Fair enough. I just meant in terms of the end result.
Quoted 5 lines
Quoted 3 lines
Anyway, I don't think this bit is really all that important unless we can demonstrate that KVM participating like this actually results in a measurable win.
Participating like what? You've lost me a bit. Are we talking past each other?
Participating in aging and look-around by trylock()ing instead of being lockless.
All I meant was: it's not important to try to figure out how to get this non-deterministic trylock()-based "fast" notifier implementation to work, because I'm not actually suggesting we actually do this. There was some suggestion previously to make arm64 work like this (in lieu of a lockless implementation), but I'd need to actually verify that this performs/behaves well before actually sending a patch.
Quoted 4 lines
What I am saying is that we do this (note that this is slightly different than an earlier sketch; I botched the ordering of
spin_is_contend()in that one, and didn't account for the page being young in the primary MMU).
Quoted 16 lines
if (pte_young(ptep_get(pte)))
young = 1 | MMU_NOTIFY_WAS_FAST;
young |= ptep_clear_young_notify(vma, addr, pte);
if (!young)
return false;
if (!(young & MMU_NOTIFY_WAS_FAST))
return true;
if (spin_is_contended(pvmw->ptl))
return false;
/* exclude special VMAs containing anon pages from COW */
if (vma->vm_flags & VM_SPECIAL)
return false;SGTM. I think we're on the same page here.
Sean Christopherson
maintainerRe: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 27 lines · James Houghton
Quoted 17 lines · Sean Christopherson
Quoted 15 lines · James Houghton
Quoted 13 lines · Sean Christopherson
Quoted 2 lines · James Houghton
I wonder if this still makes sense if whether or not an MMU is "fast" is determined by how contended some lock(s) are at the time.
No. Just because a lock wasn't contended on the initial aging doesn't mean it won't be contended on the next round. E.g. when using KVM x86's shadow MMU, which takes
mmu_lockfor write for all operations, an aging operation could get lucky and sneak in whilemmu_lockhappened to be free, but then get stuck behind a large queue of operations.The fast-ness needs to be predictable and all but guaranteed, i.e. lockless or in an MMU that takes
mmu_lockfor read in all but the most rare paths.Quoted 8 lines
Quoted 3 lines
Aging and look-around themselves only use the fast-only notifiers, so they won't ever wait on a lock (well... provided KVM is written like that, which I think is a given).
Regarding aging, is that actually the behavior that we want? I thought the plan is to have the initial test look at all MMUs, i.e. be potentially slow, but only do the lookaround if it can be fast. IIUC, that was Yu's intent (and peeking back at v2, that is indeed the case, unless I'm misreading the code).
Quoted 6 lines
I believe what I said is correct. There are three separate things going on here:
1. Aging (when we hit the low watermark, scan PTEs to find young pages)
2. Eviction (pick a page to evict; if it is definitely not young, evict it)
3. Look-around (upon finding a page is young upon attempted eviction, check adjacent pages if they are young too)
Ah, I now see the difference between #1 and #2, and your responses make a lot more sense. Thanks!
Quoted 11 lines
Quoted 2 lines
If KVM _never_ consults shadow (nested TDP) MMUs, then a VM running an L2 will end up with hot pages (used by L2) swapped out.
The shadow MMU is consulted at eviction time -- only at eviction time. So pages used by L2 won't be swapped out unless they're still cold at eviction time.
In my (and Yu's) head, not being able to do aging for nested TDP is ok because running nested VMs is much more rare than running non-nested VMs. And in the non-nested case, being able to do aging is a strict improvement over what we have now.
Yes and no. Running nested VMs is indeed rare when viewing them as a percentage of all VMs in the fleet, but for many use cases, the primary workload of a VM is to run nested VMs. E.g. say x% of VMs in the fleet run nested VMs, where 'x' is likely very small, but for those x% VMs, they run nested VMs 99% of the time (completely made up number).
So yes, I completely agree that aging for non-nested VMs is a strict improvement, but I also think don't think we should completely dismiss nested VMs as a problem not worth solving.
Quoted 2 lines
We could look into being able to do aging with the shadow MMU, but I don't think that should necessarily block this series.
...
Quoted 14 lines
Quoted 14 lines
Ooh! Actually, after fiddling a bit to see how feasible fast-aging in the shadow MMU would be, I'm pretty sure we can do straight there for nested TDP. Or rather, I suspect/hope we can get close enough for an initial merge, which would allow
aging_is_fastto be a property of themmu_notifier, i.e. would simplify things because KVM wouldn't need to communicateMMU_NOTIFY_WAS_FASTfor each notification.Walking KVM's rmaps requires
mmu_lockbecause adding/removing rmap entries is done in such a way that a lockless walk would be painfully complex. But if there is exactly _one_ rmap entry for a gfn, then slot->arch.rmap[...] points directly at that one SPTE. And with nested TDP, unless L1 is doing something uncommon, e.g. mapping the same page into multiple L2s, that overwhelming vast majority of rmaps have only one entry. That's not the case for legacy shadow paging because kernels almost always map a pfn using multiple virtual addresses, e.g. Linux's direct map along with any userspace mappings.
...
Quoted 5 lines
Hmm, interesting. I need to spend a little bit more time digesting this.
Would you like to see this included in v6? (It'd be nice to avoid the
WAS_FASTstuff....) Should we leave it for a later series? I haven't formed my own opinion yet.
I would say it depends on the viability and complexity of my idea. E.g. if it pans out more or less like my rough sketch, then it's probably worth taking on the extra code+complexity in KVM to avoid the whole WAS_FAST goo.
Note, if we do go this route, the implementation would need to be tweaked to handle the difference in behavior between aging and last-minute checks for eviction, which I obviously didn't understand when I threw together that hack-a-patch.
I need to think more about how best to handle that though, e.g. skipping GFNs with multiple mappings is probably the worst possible behavior, as we'd risk evicting hot pages. But falling back to taking mmu_lock for write isn't all that desirable either.
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 9 lines · Sean Christopherson
Quoted 7 lines · James Houghton
Quoted 6 lines · Sean Christopherson
Ooh! Actually, after fiddling a bit to see how feasible fast-aging in the shadow MMU would be, I'm pretty sure we can do straight there for nested TDP. Or rather, I suspect/hope we can get close enough for an initial merge, which would allow
aging_is_fastto be a property of themmu_notifier, i.e. would simplify things because KVM wouldn't need to communicateMMU_NOTIFY_WAS_FASTfor each notification.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Walking KVM's rmaps requires
mmu_lockbecause adding/removing rmap entries is done in such a way that a lockless walk would be painfully complex. But if there is exactly _one_ rmap entry for a gfn, then slot->arch.rmap[...] points directly at that one SPTE. And with nested TDP, unless L1 is doing something uncommon, e.g. mapping the same page into multiple L2s, that overwhelming vast majority of rmaps have only one entry. That's not the case for legacy shadow paging because kernels almost always map a pfn using multiple virtual addresses, e.g. Linux's direct map along with any userspace mappings.
Hi Sean, sorry for taking so long to get back to you.
So just to make sure I have this right: if L1 is using TDP, the gfns in L0 will usually only be mapped by a single spte. If L1 is not using TDP, then all bets are off. Is that true?
If that is true, given that we don't really have control over whether or not L1 decides to use TDP, the lockless shadow MMU walk will work, but, if L1 is not using TDP, it will often return false negatives (says "old" for an actually-young gfn). So then I don't really understand conditioning the lockless shadow MMU walk on us (L0) using the TDP MMU[1]. We care about L1, right?
(Maybe you're saying that, when the TDP MMU is enabled, the only cases where the shadow MMU is used are cases where gfns are practically always mapped by a single shadow PTE. This isn't how I understood your mail, but this is what your hack-a-patch[1] makes me think.)
[1] https://lore.kernel.org/linux-mm/[email protected]/
Quoted 13 lines
...
Quoted 5 lines
Hmm, interesting. I need to spend a little bit more time digesting this.
Would you like to see this included in v6? (It'd be nice to avoid the
WAS_FASTstuff....) Should we leave it for a later series? I haven't formed my own opinion yet.I would say it depends on the viability and complexity of my idea. E.g. if it pans out more or less like my rough sketch, then it's probably worth taking on the extra code+complexity in KVM to avoid the whole
WAS_FASTgoo.
Quoted 8 lines
Note, if we do go this route, the implementation would need to be tweaked to handle the difference in behavior between aging and last-minute checks for eviction, which I obviously didn't understand when I threw together that hack-a-patch.
I need to think more about how best to handle that though, e.g. skipping GFNs with multiple mappings is probably the worst possible behavior, as we'd risk evicting hot pages. But falling back to taking
mmu_lockfor write isn't all that desirable either.
I think falling back to the write lock is more desirable than evicting a young page.
I've attached what I think could work, a diff on top of this series. It builds at least. It uses rcu_read_lock/unlock() for walk_shadow_page_lockless_begin/end(NULL), and it puts a synchronize_rcu() in kvm_mmu_commit_zap_page().
It doesn't get rid of the WAS_FAST things because it doesn't do exactly what [1] does. It basically makes three calls now: lockless TDP MMU, lockless shadow MMU, locked shadow MMU. It only calls the locked shadow MMU bits if the lockless bits say !young (instead of being conditioned on tdp_mmu_enabled). My choice is definitely questionable for the clear path.
Thanks!
arch/x86/kvm/mmu/mmu.c +116 -17
@@ -640,6 +640,37 @@ static bool mmu_spte_age(u64 *sptep)return true; } +/* +* Similar to mmu_spte_age(), but this one should be used for lockless shadow +* page table walks. +*/ +static bool mmu_spte_age_lockless (u64 *sptep) +{ +u64 old_spte =mmu_spte_get_lockless (sptep); +u64 new_spte; + +if (!is_accessed_spte (old_spte)) +return false; + +if (spte_ad_enabled (old_spte)) +clear_bit ((ffs (shadow_accessed_mask) -1 ), + (unsigned long *)sptep); +else { + new_spte =mark_spte_for_access_track (old_spte); +if (!try_cmpxchg64 (sptep, &old_spte, new_spte)) +/* +* If the spte changed, it's likely that the gfn +* is young. +*/ +return true; + +if (is_writable_pte (old_spte)) +kvm_set_pfn_dirty (spte_to_pfn (old_spte)); + } + +return true; +} +static inline bool is_tdp_mmu_active (struct kvm_vcpu *vcpu) {return tdp_mmu_enabled && vcpu->arch.mmu->root_role.direct;
@@ -647,6 +678,11 @@ static inline bool is_tdp_mmu_active(struct kvm_vcpu *vcpu)static void walk_shadow_page_lockless_begin (struct kvm_vcpu *vcpu) { +if (!vcpu) { +rcu_read_lock (); +return ; + } +if (is_tdp_mmu_active (vcpu)) {kvm_tdp_mmu_walk_lockless_begin (); }else {
@@ -666,6 +702,11 @@ static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu)static void walk_shadow_page_lockless_end (struct kvm_vcpu *vcpu) { +if (!vcpu) { +rcu_read_unlock (); +return ; + } +if (is_tdp_mmu_active (vcpu)) {kvm_tdp_mmu_walk_lockless_end (); }else {
@@ -949,14 +990,14 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte,int count =0 ;if (!rmap_head->val) { - rmap_head->val = (unsigned long )spte; +WRITE_ONCE (&rmap_head->val, (unsigned long )spte); }else if (!(rmap_head->val &1 )) { desc =kvm_mmu_memory_cache_alloc (cache); desc->sptes[0 ] = (u64 *)rmap_head->val; desc->sptes[1 ] = spte; desc->spte_count =2 ; desc->tail_count =0 ; - rmap_head->val = (unsigned long )desc |1 ; +WRITE_ONCE (&rmap_head->val, (unsigned long )desc |1 ); ++count; }else { desc = (struct pte_list_desc *)(rmap_head->val & ~1ul );
@@ -971,7 +1012,7 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, desc->more = (struct pte_list_desc *)(rmap_head->val & ~1ul ); desc->spte_count =0 ; desc->tail_count = count; - rmap_head->val = (unsigned long )desc |1 ; +WRITE_ONCE (&rmap_head->val, (unsigned long )desc |1 ); } desc->sptes[desc->spte_count++] = spte; }
@@ -1009,9 +1050,10 @@ static void pte_list_desc_remove_entry(struct kvm *kvm,* head at the next descriptor, i.e. the new head. */ if (!head_desc->more) - rmap_head->val =0 ; +WRITE_ONCE (&rmap_head->val,0 );else - rmap_head->val = (unsigned long )head_desc->more |1 ; +WRITE_ONCE (&rmap_head->val, + (unsigned long )head_desc->more |1 );mmu_free_pte_list_desc (head_desc); }
@@ -1028,7 +1070,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte,if (KVM_BUG_ON_DATA_CORRUPTION ((u64 *)rmap_head->val != spte, kvm))return ; - rmap_head->val =0 ; +WRITE_ONCE (&rmap_head->val,0 ); }else { desc = (struct pte_list_desc *)(rmap_head->val & ~1ul );while (desc) {
@@ -1078,7 +1120,7 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, } out:/* rmap_head is meaningless now, remember to reset it */ - rmap_head->val =0 ; +WRITE_ONCE (&rmap_head->val,0 );return true; }
@@ -1634,17 +1676,64 @@ static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm)return !tdp_mmu_enabled ||READ_ONCE (kvm->arch.indirect_shadow_pages); } +static bool kvm_age_rmap_fast (u64 *sptep) +{ +return mmu_spte_age_lockless (sptep); +} + +static bool kvm_test_age_rmap_fast (u64 *sptep) +{ +return is_accessed_spte (READ_ONCE (*sptep)); +} + +typedef bool (*rmap_lockless_handler_t )(u64 *sptep); + +static __always_inline bool kvm_handle_gfn_range_lockless( +struct kvm *kvm,struct kvm_gfn_range *range, +rmap_lockless_handler_t handler) +{ +struct kvm_rmap_head *rmap; +u64 *sptep; +gfn_t gfn; +int level; +bool ret = false; + +walk_shadow_page_lockless_begin (NULL ); + +for (gfn = range->start; gfn < range->end; gfn++) { +for (level =PG_LEVEL_4K ; level <=KVM_MAX_HUGEPAGE_LEVEL ; + level++) { + rmap =gfn_to_rmap (gfn, level, range->slot); + sptep = (void *)READ_ONCE (rmap->val); + +/* Skip this gfn if multiple SPTEs mapping it */ +if ((unsigned long )sptep &1 ) +continue ; + + ret |=handler (sptep); + } + } + +walk_shadow_page_lockless_end (NULL ); + +return ret; +} +bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) { -bool young = false; +bool young = false, shadow_young = false; -if (tdp_mmu_enabled) { +if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range); -if (young) - range->arg.report_fast = true; - } -if (!range->arg.fast_only &&kvm_has_shadow_mmu_sptes (kvm)) { + shadow_young =kvm_handle_gfn_range_lockless (kvm, range, + kvm_age_rmap_fast); + young |= shadow_young; +if (young) + range->arg.report_fast = true; + +else if (!shadow_young && !range->arg.fast_only && +kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young =kvm_handle_gfn_range (kvm, range, kvm_age_rmap);write_unlock (&kvm->mmu_lock);
@@ -1657,11 +1746,15 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (tdp_mmu_enabled) { +if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range); -if (young) - range->arg.report_fast = true; - } + +if (!young) + young |=kvm_handle_gfn_range_lockless (kvm, range, + kvm_test_age_rmap_fast); + +if (young) + range->arg.report_fast = true;if (!young && !range->arg.fast_only &&kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock);
@@ -2636,6 +2729,12 @@ static void kvm_mmu_commit_zap_page(struct kvm *kvm,*/ kvm_flush_remote_tlbs (kvm); +/* +* Wait for any non-vCPU lockless shadow page table walkers to stop +* using the shadow pages we're about to free. +*/ +synchronize_rcu (); +list_for_each_entry_safe (sp, nsp, invalid_list, link) {WARN_ON_ONCE (!sp->role.invalid || sp->root_count);kvm_mmu_free_shadow_page (sp);
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 26 lines · James Houghton
Quoted 9 lines · Sean Christopherson
Quoted 7 lines · James Houghton
Quoted 6 lines · Sean Christopherson
Ooh! Actually, after fiddling a bit to see how feasible fast-aging in the shadow MMU would be, I'm pretty sure we can do straight there for nested TDP. Or rather, I suspect/hope we can get close enough for an initial merge, which would allow
aging_is_fastto be a property of themmu_notifier, i.e. would simplify things because KVM wouldn't need to communicateMMU_NOTIFY_WAS_FASTfor each notification.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Walking KVM's rmaps requires
mmu_lockbecause adding/removing rmap entries is done in such a way that a lockless walk would be painfully complex. But if there is exactly _one_ rmap entry for a gfn, then slot->arch.rmap[...] points directly at that one SPTE. And with nested TDP, unless L1 is doing something uncommon, e.g. mapping the same page into multiple L2s, that overwhelming vast majority of rmaps have only one entry. That's not the case for legacy shadow paging because kernels almost always map a pfn using multiple virtual addresses, e.g. Linux's direct map along with any userspace mappings.Hi Sean, sorry for taking so long to get back to you.
So just to make sure I have this right: if L1 is using TDP, the gfns in L0 will usually only be mapped by a single spte. If L1 is not using TDP, then all bets are off. Is that true?
Quoted 6 lines
If that is true, given that we don't really have control over whether or not L1 decides to use TDP, the lockless shadow MMU walk will work, but, if L1 is not using TDP, it will often return false negatives (says "old" for an actually-young gfn). So then I don't really understand conditioning the lockless shadow MMU walk on us (L0) using the TDP MMU[1]. We care about L1, right?
Ok I think I understand now. If L1 is using shadow paging, L2 is accessing memory the same way L1 would, so we use the TDP MMU at L0 for this case (if tdp_mmu_enabled). If L1 is using TDP, then we must use the shadow MMU, so that's the interesting case.
Quoted 4 lines
(Maybe you're saying that, when the TDP MMU is enabled, the only cases where the shadow MMU is used are cases where gfns are practically always mapped by a single shadow PTE. This isn't how I understood your mail, but this is what your hack-a-patch[1] makes me think.)
So it appears that this interpretation is actually what you meant.
Quoted 33 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
Quoted 13 lines
...
Quoted 5 lines
Hmm, interesting. I need to spend a little bit more time digesting this.
Would you like to see this included in v6? (It'd be nice to avoid the
WAS_FASTstuff....) Should we leave it for a later series? I haven't formed my own opinion yet.I would say it depends on the viability and complexity of my idea. E.g. if it pans out more or less like my rough sketch, then it's probably worth taking on the extra code+complexity in KVM to avoid the whole
WAS_FASTgoo.Quoted 8 lines
Note, if we do go this route, the implementation would need to be tweaked to handle the difference in behavior between aging and last-minute checks for eviction, which I obviously didn't understand when I threw together that hack-a-patch.
I need to think more about how best to handle that though, e.g. skipping GFNs with multiple mappings is probably the worst possible behavior, as we'd risk evicting hot pages. But falling back to taking
mmu_lockfor write isn't all that desirable either.I think falling back to the write lock is more desirable than evicting a young page.
I've attached what I think could work, a diff on top of this series. It builds at least. It uses rcu_read_lock/unlock() for walk_shadow_page_lockless_begin/end(NULL), and it puts a
synchronize_rcu()inkvm_mmu_commit_zap_page().
Quoted 6 lines
It doesn't get rid of the
WAS_FASTthings because it doesn't do exactly what [1] does. It basically makes three calls now: lockless TDP MMU, lockless shadow MMU, locked shadow MMU. It only calls the locked shadow MMU bits if the lockless bits say !young (instead of being conditioned ontdp_mmu_enabled). My choice is definitely questionable for the clear path.
I still don't think we should get rid of the WAS_FAST stuff.
The assumption that the L1 VM will almost never share pages between L2 VMs is questionable. The real question becomes: do we care to have accurate age information for this case? I think so.
It's not completely trivial to get the lockless walking of the shadow MMU rmaps correct either (please see the patch I attached here[1]). And the WAS_FAST functionality isn't even that complex to begin with.
Thanks for your patience.
[1]: https://lore.kernel.org/linux-mm/CADrL8HW=kCLoWBwoiSOCd8WHFvBdWaguZ2ureo4eFy9D67+owg@mail.gmail.com/
Sean Christopherson
maintainerRe: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 34 lines · James Houghton
Quoted 26 lines · James Houghton
Quoted 9 lines · Sean Christopherson
Quoted 7 lines · James Houghton
Quoted 6 lines · Sean Christopherson
Ooh! Actually, after fiddling a bit to see how feasible fast-aging in the shadow MMU would be, I'm pretty sure we can do straight there for nested TDP. Or rather, I suspect/hope we can get close enough for an initial merge, which would allow
aging_is_fastto be a property of themmu_notifier, i.e. would simplify things because KVM wouldn't need to communicateMMU_NOTIFY_WAS_FASTfor each notification.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Walking KVM's rmaps requires
mmu_lockbecause adding/removing rmap entries is done in such a way that a lockless walk would be painfully complex. But if there is exactly _one_ rmap entry for a gfn, then slot->arch.rmap[...] points directly at that one SPTE. And with nested TDP, unless L1 is doing something uncommon, e.g. mapping the same page into multiple L2s, that overwhelming vast majority of rmaps have only one entry. That's not the case for legacy shadow paging because kernels almost always map a pfn using multiple virtual addresses, e.g. Linux's direct map along with any userspace mappings.Hi Sean, sorry for taking so long to get back to you.
So just to make sure I have this right: if L1 is using TDP, the gfns in L0 will usually only be mapped by a single spte. If L1 is not using TDP, then all bets are off. Is that true?
Quoted 6 lines
If that is true, given that we don't really have control over whether or not L1 decides to use TDP, the lockless shadow MMU walk will work, but, if L1 is not using TDP, it will often return false negatives (says "old" for an actually-young gfn). So then I don't really understand conditioning the lockless shadow MMU walk on us (L0) using the TDP MMU[1]. We care about L1, right?
Quoted 4 lines
Ok I think I understand now. If L1 is using shadow paging, L2 is accessing memory the same way L1 would, so we use the TDP MMU at L0 for this case (if
tdp_mmu_enabled). If L1 is using TDP, then we must use the shadow MMU, so that's the interesting case.
Yep.
Quoted 6 lines
Quoted 4 lines
(Maybe you're saying that, when the TDP MMU is enabled, the only cases where the shadow MMU is used are cases where gfns are practically always mapped by a single shadow PTE. This isn't how I understood your mail, but this is what your hack-a-patch[1] makes me think.)
So it appears that this interpretation is actually what you meant.
Yep.
Quoted 32 lines
Quoted 32 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
Quoted 13 lines
...
Quoted 5 lines
Hmm, interesting. I need to spend a little bit more time digesting this.
Would you like to see this included in v6? (It'd be nice to avoid the
WAS_FASTstuff....) Should we leave it for a later series? I haven't formed my own opinion yet.I would say it depends on the viability and complexity of my idea. E.g. if it pans out more or less like my rough sketch, then it's probably worth taking on the extra code+complexity in KVM to avoid the whole
WAS_FASTgoo.Quoted 8 lines
Note, if we do go this route, the implementation would need to be tweaked to handle the difference in behavior between aging and last-minute checks for eviction, which I obviously didn't understand when I threw together that hack-a-patch.
I need to think more about how best to handle that though, e.g. skipping GFNs with multiple mappings is probably the worst possible behavior, as we'd risk evicting hot pages. But falling back to taking
mmu_lockfor write isn't all that desirable either.I think falling back to the write lock is more desirable than evicting a young page.
I've attached what I think could work, a diff on top of this series. It builds at least. It uses rcu_read_lock/unlock() for walk_shadow_page_lockless_begin/end(NULL), and it puts a
synchronize_rcu()inkvm_mmu_commit_zap_page().
Quoted 8 lines
Quoted 6 lines
It doesn't get rid of the
WAS_FASTthings because it doesn't do exactly what [1] does. It basically makes three calls now: lockless TDP MMU, lockless shadow MMU, locked shadow MMU. It only calls the locked shadow MMU bits if the lockless bits say !young (instead of being conditioned ontdp_mmu_enabled). My choice is definitely questionable for the clear path.I still don't think we should get rid of the
WAS_FASTstuff.
I do :-)
Quoted 3 lines
The assumption that the L1 VM will almost never share pages between L2 VMs is questionable. The real question becomes: do we care to have accurate age information for this case? I think so.
I think you're conflating two different things. WAS_FAST isn't about accuracy, it's about supporting lookaround in conditionally fast secondary MMUs.
Accuracy only comes into play when we're talking about the last-minute check, which, IIUC, has nothing to do with WAS_FAST because any potential lookaround has already been performed.
Quoted 2 lines
It's not completely trivial to get the lockless walking of the shadow MMU rmaps correct either (please see the patch I attached here[1]).
Heh, it's not correct. Invoking synchronize_rcu() in kvm_mmu_commit_zap_page() is illegal, as mmu_lock (rwlock) is held and synchronize_rcu() might_sleep().
For kvm_test_age_rmap_fast(), KVM can blindly read READ_ONCE(*sptep). KVM might read garbage, but that would be an _extremely_ rare scenario, and reporting a zapped page as being young is acceptable in that 1 in a billion situation.
For kvm_age_rmap_fast(), i.e. where KVM needs to write, I'm pretty sure KVM can handle that by rechecking the rmap and using CMPXCHG to write the SPTE. If the rmap is unchanged, then the old SPTE value is guaranteed to be valid, in the sense that its value most definitely came from a KVM shadow page table. Ah, drat, that won't work, because very theoretically, the page table could be freed, reallocated, and rewritten with the exact same value by something other than KVM. Hrm.
Looking more closely, I think we can go straight to supporting rmap walks outside of mmu_lock. There will still be a "lock", but it will be a *very* rudimentary lock, akin to the TDP MMU's REMOVED_SPTE approach. Bit 0 of rmap_head->val is used to indicate "many", while bits 63:3/31:2 on 64-bit/32-bit KVM hold the pointer (to a SPTE or a list). That means bit 1 is available for shenanigans.
If we use bit 1 to lock the rmap, then the fast mmu_notifier can safely walk the entire rmap chain. And with a reader/write scheme, the rmap walks that are performed under mmu_lock don't need to lock the rmap, which means flows like kvm_mmu_zap_collapsible_spte() don't need to be modified to avoid recursive self-deadlock. Lastly, the locking can be conditioned on the rmap being valid, i.e. having at least one SPTE. That way the common case of a gfn not having any rmaps is a glorified nop.
Adding the locking isn't actually all that difficult, with the *huge* caveat that the below patch is compile-tested only. The vast majority of the churn is to make it so existing code ignores the new KVM_RMAP_LOCKED bit.
I don't know that we should pursue such an approach in this series unless we have to. E.g. if we can avoid WAS_FAST or don't have to carry too much intermediate complexity, then it'd probably be better to land the TDP MMU support first and then add nested TDP support later.
At the very least, it does make me more confident that a fast walk of the rmaps is very doable (at least for nested TDP), i.e. makes me even more steadfast against adding WAS_FAST.
Quoted 1 line
And the
WAS_FASTfunctionality isn't even that complex to begin with.
I agree the raw code isn't terribly complex, but it's not trivial either. And the concept and *behavior* is complex, which is just as much of a maintenance burden as the code itself. E.g. it requires knowing that KVM has multiple MMUs buried behind a single mmu_notifier, and that a "hit" on the fast MMU will trigger lookaround on the fast MMU, but not the slow MMU. Understanding and describing the implications of that behavior isn't easy. E.g. if GFN=X is young in the TDP MMU, but X+1..X+N are young only in the shadow MMU, is doing lookaround and making decisions based purely on the TDP MMU state the "right" behavior?
I also really don't like bleeding KVM details into the mmu_nofitier APIs. The need for WAS_FAST is 100% a KVM limitation. AFAIK, no other secondary MMU has multiple MMU implementations active behind a single notifier, and other than lack of support, nothing fundamentally prevents a fast query in the shadow MMU.
--- arch/x86/kvm/mmu/mmu.c | 163 ++++++++++++++++++++++++++++++++--------- 1 file changed, 128 insertions(+), 35 deletions(-)
arch/x86/kvm/mmu/mmu.c +128 -35
@@ -935,9 +935,59 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu* About rmap_head encoding: * * If the bit zero of rmap_head->val is clear, then it points to the only spte -* in this rmap chain. Otherwise, (rmap_head->val & ~1) points to a struct +* in this rmap chain. Otherwise, (rmap_head->val & ~3) points to a struct * pte_list_desc containing more mappings. */ +#define KVM_RMAP_MANY BIT(0) +#define KVM_RMAP_LOCKED BIT(1) + +static unsigned long kvm_rmap_lock (struct kvm_rmap_head *rmap_head) +{ +unsigned long old_val, new_val; + + old_val =READ_ONCE (rmap_head->val); +if (!old_val) +return 0 ; + +do { +while (old_val &KVM_RMAP_LOCKED ) { + old_val =READ_ONCE (rmap_head->val); +cpu_relax (); + } +if (!old_val) +return 0 ; + + new_val = old_val |KVM_RMAP_LOCKED ; + }while (!try_cmpxchg (&rmap_head->val, &old_val, new_val)); + +return old_val; +} + +static unsigned long kvm_rmap_write_lock (struct kvm_rmap_head *rmap_head) +{ +return kvm_rmap_lock (rmap_head); +} + +static void kvm_rmap_write_ulock (struct kvm_rmap_head *rmap_head, +unsigned long new_val) +{ +WARN_ON_ONCE (new_val &KVM_RMAP_LOCKED ); +WRITE_ONCE (rmap_head->val, new_val); +} + +static unsigned long kvm_rmap_read_lock (struct kvm_rmap_head *rmap_head) +{ +return kvm_rmap_lock (rmap_head); +} + +static void kvm_rmap_read_unlock (struct kvm_rmap_head *rmap_head, +unsigned long old_val) +{ +if (!old_val) +return ; + +WRITE_ONCE (rmap_head->val, old_val & ~KVM_RMAP_LOCKED ); +}/* * Returns the number of pointers in the rmap chain, not counting the new one.
@@ -945,21 +995,24 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpustatic int pte_list_add (struct kvm_mmu_memory_cache *cache,u64 *spte,struct kvm_rmap_head *rmap_head) { +unsigned long old_val, new_val;struct pte_list_desc *desc;int count =0 ; -if (!rmap_head->val) { - rmap_head->val = (unsigned long )spte; - }else if (!(rmap_head->val &1 )) { + old_val =kvm_rmap_write_lock (rmap_head); + +if (!old_val) { + new_val = (unsigned long )spte; + }else if (!(old_val &KVM_RMAP_MANY )) { desc =kvm_mmu_memory_cache_alloc (cache); - desc->sptes[0 ] = (u64 *)rmap_head->val; + desc->sptes[0 ] = (u64 *)old_val; desc->sptes[1 ] = spte; desc->spte_count =2 ; desc->tail_count =0 ; - rmap_head->val = (unsigned long )desc |1 ; + new_val = (unsigned long )desc |KVM_RMAP_MANY ; ++count; }else { - desc = (struct pte_list_desc *)(rmap_head->val & ~1ul ); + desc = (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY ); count = desc->tail_count + desc->spte_count;/*
@@ -968,21 +1021,25 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte,*/ if (desc->spte_count ==PTE_LIST_EXT ) { desc =kvm_mmu_memory_cache_alloc (cache); - desc->more = (struct pte_list_desc *)(rmap_head->val & ~1ul ); + desc->more = (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY ); desc->spte_count =0 ; desc->tail_count = count; - rmap_head->val = (unsigned long )desc |1 ; + new_val = (unsigned long )desc |KVM_RMAP_MANY ; + }else { + new_val = old_val; } desc->sptes[desc->spte_count++] = spte; } + +kvm_rmap_write_ulock (rmap_head, new_val); +return count; } -static void pte_list_desc_remove_entry (struct kvm *kvm, -struct kvm_rmap_head *rmap_head, +static void pte_list_desc_remove_entry (struct kvm *kvm,unsigned long *rmap_val,struct pte_list_desc *desc,int i) { -struct pte_list_desc *head_desc = (struct pte_list_desc *)(rmap_head->val & ~1ul ); +struct pte_list_desc *head_desc = (struct pte_list_desc *)(*rmap_val & ~KVM_RMAP_MANY );int j = head_desc->spte_count -1 ;/*
@@ -1009,9 +1066,9 @@ static void pte_list_desc_remove_entry(struct kvm *kvm,* head at the next descriptor, i.e. the new head. */ if (!head_desc->more) - rmap_head->val =0 ; + *rmap_val =0 ;else - rmap_head->val = (unsigned long )head_desc->more |1 ; + *rmap_val = (unsigned long )head_desc->more |KVM_RMAP_MANY ;mmu_free_pte_list_desc (head_desc); }
@@ -1019,24 +1076,26 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, struct kvm_rmap_head *rmap_head) {struct pte_list_desc *desc; +unsigned long rmap_val;int i; -if (KVM_BUG_ON_DATA_CORRUPTION (!rmap_head->val, kvm)) -return ; + rmap_val =kvm_rmap_write_lock (rmap_head); +if (KVM_BUG_ON_DATA_CORRUPTION (!rmap_val, kvm)) + goto out; -if (!(rmap_head->val &1 )) { -if (KVM_BUG_ON_DATA_CORRUPTION ((u64 *)rmap_head->val != spte, kvm)) -return ; +if (!(rmap_val &KVM_RMAP_MANY )) { +if (KVM_BUG_ON_DATA_CORRUPTION ((u64 *)rmap_val != spte, kvm)) + goto out; - rmap_head->val =0 ; + rmap_val =0 ; }else { - desc = (struct pte_list_desc *)(rmap_head->val & ~1ul ); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY );while (desc) {for (i =0 ; i < desc->spte_count; ++i) {if (desc->sptes[i] == spte) { -pte_list_desc_remove_entry (kvm, rmap_head, +pte_list_desc_remove_entry (kvm, &rmap_val, desc, i); -return ; + goto out; } } desc = desc->more;
@@ -1044,6 +1103,9 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte,KVM_BUG_ON_DATA_CORRUPTION (true, kvm); } + +out: +kvm_rmap_write_ulock (rmap_head, rmap_val); } static void kvm_zap_one_rmap_spte(struct kvm *kvm,
@@ -1058,17 +1120,19 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, struct kvm_rmap_head *rmap_head) {struct pte_list_desc *desc, *next; +unsigned long rmap_val;int i; -if (!rmap_head->val) + rmap_val =kvm_rmap_write_lock (rmap_head); +if (!rmap_val)return false; -if (!(rmap_head->val &1 )) { -mmu_spte_clear_track_bits (kvm, (u64 *)rmap_head->val); +if (!(rmap_val &KVM_RMAP_MANY )) { +mmu_spte_clear_track_bits (kvm, (u64 *)rmap_val); goto out; } - desc = (struct pte_list_desc *)(rmap_head->val & ~1ul ); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY );for (; desc; desc = next) {for (i =0 ; i < desc->spte_count; i++)
@@ -1078,20 +1142,21 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, } out:/* rmap_head is meaningless now, remember to reset it */ - rmap_head->val =0 ; +kvm_rmap_write_ulock (rmap_head,0 );return true; }unsigned int pte_list_count (struct kvm_rmap_head *rmap_head) { +unsigned long rmap_val =READ_ONCE (rmap_head->val) & ~KVM_RMAP_LOCKED ;struct pte_list_desc *desc; -if (!rmap_head->val) +if (!rmap_val)return 0 ; -else if (!(rmap_head->val &1 )) +else if (!(rmap_val &KVM_RMAP_MANY ))return 1 ; - desc = (struct pte_list_desc *)(rmap_head->val & ~1ul ); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY );return desc->tail_count + desc->spte_count; }
@@ -1134,6 +1199,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte)*/ struct rmap_iterator {/* private fields */ +struct rmap_head *head;struct pte_list_desc *desc;/* holds the sptep if not NULL */ int pos;/* index of the sptep */ };
@@ -1148,18 +1214,19 @@ struct rmap_iterator {static u64 *rmap_get_first (struct kvm_rmap_head *rmap_head,struct rmap_iterator *iter) { +unsigned long rmap_val =READ_ONCE (rmap_head->val) & ~KVM_RMAP_LOCKED ;u64 *sptep; -if (!rmap_head->val) +if (!rmap_val)return NULL ; -if (!(rmap_head->val &1 )) { +if (!(rmap_val &KVM_RMAP_MANY )) { iter->desc =NULL ; - sptep = (u64 *)rmap_head->val; + sptep = (u64 *)rmap_val; goto out; } - iter->desc = (struct pte_list_desc *)(rmap_head->val & ~1ul ); + iter->desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY ); iter->pos =0 ; sptep = iter->desc->sptes[iter->pos]; out:
@@ -1553,6 +1620,32 @@ static __always_inline bool kvm_handle_gfn_range(struct kvm *kvm,return ret; } +static __always_inline bool kvm_handle_gfn_range_lockless(struct kvm *kvm, +struct kvm_gfn_range *range, +rmap_handler_t handler) +{ +struct kvm_rmap_head *rmap_head; +unsigned long rmap_val; +bool ret = false; +gfn_t gfn; +int level; + +for (gfn = range->start; gfn < range->end; gfn++) { +for (level =PG_LEVEL_4K ; level <=KVM_MAX_HUGEPAGE_LEVEL ; level++) { + rmap_head =gfn_to_rmap (gfn, level, range->slot); + rmap_val =kvm_rmap_read_lock (rmap_head); + +if (rmap_val) + ret |=handler (kvm, rmap_head, range->slot, gfn, level); + +kvm_rmap_read_unlock (rmap_head, rmap_val); + } + } + +return ret; +} + +bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) {bool flush = false;
base-commit: 771df9ffadb8204e61d3e98f36c5067102aab78f
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 13 lines · Sean Christopherson
Quoted 4 lines · James Houghton
Quoted 2 lines · James Houghton
On Mon, Jun 17, 2024 at 11:37 AM Sean Christopherson <[email protected]> wrote:
I still don't think we should get rid of the
WAS_FASTstuff.I do :-)
Quoted 3 lines
The assumption that the L1 VM will almost never share pages between L2 VMs is questionable. The real question becomes: do we care to have accurate age information for this case? I think so.
Quoted 6 lines
I think you're conflating two different things.
WAS_FASTisn't about accuracy, it's about supporting lookaround in conditionally fast secondary MMUs.Accuracy only comes into play when we're talking about the last-minute check, which, IIUC, has nothing to do with
WAS_FASTbecause any potential lookaround has already been performed.
Sorry, I thought you meant: have the MMU notifier only ever be lockless (when tdp_mmu_enabled), and just return a potentially wrong result in the unlikely case that L1 is sharing pages between L2s.
I think it's totally fine to just drop WAS_FAST. So then we can either do look-around (1) always, or (2) only when there is a secondary MMU with has_fast_aging. (2) is pretty simple, I'll just do that.
We can add some shadow MMU lockless support later to make the look-around not as useless for the nested TDP case.
Quoted 31 lines
Quoted 2 lines
It's not completely trivial to get the lockless walking of the shadow MMU rmaps correct either (please see the patch I attached here[1]).
Heh, it's not correct. Invoking
synchronize_rcu()inkvm_mmu_commit_zap_page()is illegal, asmmu_lock(rwlock) is held andsynchronize_rcu()might_sleep().For
kvm_test_age_rmap_fast(), KVM can blindly readREAD_ONCE(*sptep). KVM might read garbage, but that would be an _extremely_ rare scenario, and reporting a zapped page as being young is acceptable in that 1 in a billion situation.For
kvm_age_rmap_fast(), i.e. where KVM needs to write, I'm pretty sure KVM can handle that by rechecking the rmap and using CMPXCHG to write the SPTE. If the rmap is unchanged, then the old SPTE value is guaranteed to be valid, in the sense that its value most definitely came from a KVM shadow page table. Ah, drat, that won't work, because very theoretically, the page table could be freed, reallocated, and rewritten with the exact same value by something other than KVM. Hrm.Looking more closely, I think we can go straight to supporting rmap walks outside of
mmu_lock. There will still be a "lock", but it will be a *very* rudimentary lock, akin to the TDP MMU'sREMOVED_SPTEapproach. Bit 0 ofrmap_head->valis used to indicate "many", while bits 63:3/31:2 on 64-bit/32-bit KVM hold the pointer (to a SPTE or a list). That means bit 1 is available for shenanigans.If we use bit 1 to lock the rmap, then the fast
mmu_notifiercan safely walk the entire rmap chain. And with a reader/write scheme, the rmap walks that are performed undermmu_lockdon't need to lock the rmap, which means flows likekvm_mmu_zap_collapsible_spte()don't need to be modified to avoid recursive self-deadlock. Lastly, the locking can be conditioned on the rmap being valid, i.e. having at least one SPTE. That way the common case of a gfn not having any rmaps is a glorified nop.
Quoted 3 lines
Adding the locking isn't actually all that difficult, with the *huge* caveat that the below patch is compile-tested only. The vast majority of the churn is to make it so existing code ignores the new
KVM_RMAP_LOCKEDbit.
This is very interesting, thanks for laying out how this could be done. I don't want to hold this series up on getting the details of the shadow MMU lockless walk exactly right. :)
Quoted 4 lines
I don't know that we should pursue such an approach in this series unless we have to. E.g. if we can avoid
WAS_FASTor don't have to carry too much intermediate complexity, then it'd probably be better to land the TDP MMU support first and then add nested TDP support later.
Agreed!
Quoted 15 lines
At the very least, it does make me more confident that a fast walk of the rmaps is very doable (at least for nested TDP), i.e. makes me even more steadfast against adding
WAS_FAST.Quoted 1 line
And the
WAS_FASTfunctionality isn't even that complex to begin with.I agree the raw code isn't terribly complex, but it's not trivial either. And the concept and *behavior* is complex, which is just as much of a maintenance burden as the code itself. E.g. it requires knowing that KVM has multiple MMUs buried behind a single
mmu_notifier, and that a "hit" on the fast MMU will trigger lookaround on the fast MMU, but not the slow MMU. Understanding and describing the implications of that behavior isn't easy. E.g. if GFN=X is young in the TDP MMU, but X+1..X+N are young only in the shadow MMU, is doing lookaround and making decisions based purely on the TDP MMU state the "right" behavior?
Quoted 4 lines
I also really don't like bleeding KVM details into the
mmu_nofitierAPIs. The need forWAS_FASTis 100% a KVM limitation. AFAIK, no other secondary MMU has multiple MMU implementations active behind a single notifier, and other than lack of support, nothing fundamentally prevents a fast query in the shadow MMU.
Makes sense.
So in v6, I will make the following changes:
1. Drop the WAS_FAST complexity.
2. Add a function like mm_has_fast_aging_notifiers(), use that to determine if we should be doing look-around.
3. Maybe change the notifier calls slightly[1], still need to check performance.
Does that sound good to you?
Thanks!
[1]: https://lore.kernel.org/linux-mm/CAOUHufb2f_EwHY5LQ59k7Nh7aS1-ZbOKtkoysb8BtxRNRFMypQ@mail.gmail.com/
Sean Christopherson
maintainerRe: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 31 lines · James Houghton
Quoted 13 lines · Sean Christopherson
Quoted 4 lines · James Houghton
Quoted 2 lines · James Houghton
On Mon, Jun 17, 2024 at 11:37 AM Sean Christopherson <[email protected]> wrote:
I still don't think we should get rid of the
WAS_FASTstuff.I do :-)
Quoted 3 lines
The assumption that the L1 VM will almost never share pages between L2 VMs is questionable. The real question becomes: do we care to have accurate age information for this case? I think so.
Quoted 6 lines
I think you're conflating two different things.
WAS_FASTisn't about accuracy, it's about supporting lookaround in conditionally fast secondary MMUs.Accuracy only comes into play when we're talking about the last-minute check, which, IIUC, has nothing to do with
WAS_FASTbecause any potential lookaround has already been performed.Sorry, I thought you meant: have the MMU notifier only ever be lockless (when
tdp_mmu_enabled), and just return a potentially wrong result in the unlikely case that L1 is sharing pages between L2s.I think it's totally fine to just drop
WAS_FAST. So then we can either do look-around (1) always, or (2) only when there is a secondary MMU withhas_fast_aging. (2) is pretty simple, I'll just do that.We can add some shadow MMU lockless support later to make the look-around not as useless for the nested TDP case.
...
Quoted 7 lines
Quoted 3 lines
Adding the locking isn't actually all that difficult, with the *huge* caveat that the below patch is compile-tested only. The vast majority of the churn is to make it so existing code ignores the new
KVM_RMAP_LOCKEDbit.This is very interesting, thanks for laying out how this could be done. I don't want to hold this series up on getting the details of the shadow MMU lockless walk exactly right. :)
...
Quoted 3 lines
1. Drop the
WAS_FASTcomplexity.
2. Add a function likemm_has_fast_aging_notifiers(), use that to determine if we should be doing look-around.
I would prefer a flag over a function. Long-term, if my pseudo-lockless rmap idea pans out, KVM can set the flag during VM creation. Until then, KVM can set the flag during creation and then toggle it in (un)account_shadowed(). Races will be possible, but they should be extremely rare and quite benign, all things considered.
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
Quoted 11 lines · Sean Christopherson
Quoted 3 lines · James Houghton
1. Drop the
WAS_FASTcomplexity.
2. Add a function likemm_has_fast_aging_notifiers(), use that to determine if we should be doing look-around.I would prefer a flag over a function. Long-term, if my pseudo-lockless rmap idea pans out, KVM can set the flag during VM creation. Until then, KVM can set the flag during creation and then toggle it in (un)account_shadowed(). Races will be possible, but they should be extremely rare and quite benign, all things considered.
So I think you're talking about .has_fast_aging bool on struct mmu_notifier, and we would do look-around if that is set, right? So we'd need to have something like mm_has_fast_aging_notifiers() to check if any of the subscribed mmu notifiers have .has_fast_aging.
We could toggle it in (un)account_shadowed(), but I think I'd need to demonstrate some performance win to justify it.
Re: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier
In reply to James HoughtonQuoted 29 lines · James Houghton
Quoted 27 lines · Yu Zhao
Quoted 2 lines · James Houghton
This new notifier is for multi-gen LRU specifically
Let me call it out before others do: we can't be this self-serving.
Quoted 9 lines
as it wants to be
able to get and clear age information from secondary MMUs only if it can be done "fast".By having this notifier specifically created for MGLRU, what "fast" means comes down to what is "fast" enough to improve MGLRU's ability to reclaim most of the time.
Signed-off-by James Houghton <[email protected]>If we'd like this to pass other MM reviewers, especially the MMU notifier maintainers, we'd need to design a generic API that can benefit all the *existing* users: idle page tracking [1], DAMON [2] and MGLRU.
Also I personally prefer to extend the existing callbacks by adding new parameters, and on top of that, I'd try to consolidate the existing callbacks -- it'd be less of a hard sell if my changes result in less code, not more.
Quoted 6 lines
Quoted 1 line
(v2 did all these, btw.)
I think consolidating the callbacks is cleanest, like you had it in v2. I really wasn't sure about this change honestly, but it was my attempt to incorporate feedback like this[3] from v4. I'll consolidate the callbacks like you had in v2.
Only if you think updating the MMU notifier API is overall the best option.
As I said earlier, I don't mind not doing the extra work, i.e., the bitmap/_fast_only parameters and the _FAST_YOUNG return value, as long as we know where we win and lose, i.e., simplicity and regressions. I would call all the potential regressions (the nested VM case, the arm64 case and the memcached benchmark) minor, compared with what we got from v2. But my opinion doesn't really matter much because I'm not your customer :)
PATCH v5 5/9 KVM: Add kvm_fast_age_gfn and kvm_fast_test_age_gfn
In reply to James HoughtonProvide the basics for allowing architectures to implement mmu_notifier_test_clear_young_fast_only().
Add CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER that architectures will set if they implement the fast-only notifier.
kvm_fast_age_gfn and kvm_fast_test_age_gfn both need to support returning a tri-state state of:
1. fast && young,
2. fast && !young,
3. !fast
This could be done by making gfn_handler_t return int, but that would mean a lot of churn. Instead, include a new kvm_mmu_notifier_arg 'bool *failed' for kvm_fast_{test,}_age_gfn to optionally use.
--- include/linux/kvm_host.h | 7 ++++++ include/trace/events/kvm.h | 22 ++++++++++++++++++ virt/kvm/Kconfig | 4 ++++ virt/kvm/kvm_main.c | 47 ++++++++++++++++++++++++++++++-------- 4 files changed, 71 insertions(+), 9 deletions(-)
include/linux/kvm_host.h +7 -0
@@ -258,6 +258,9 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg {unsigned long attributes; +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER +bool *failed; +#endif }; struct kvm_gfn_range {
@@ -271,7 +274,11 @@ struct kvm_gfn_range {bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range); +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER +bool kvm_fast_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range); +bool kvm_fast_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);#endif +#endif /* CONFIG_KVM_GENERIC_MMU_NOTIFIER */ enum {OUTSIDE_GUEST_MODE ,
include/trace/events/kvm.h +22 -0
@@ -489,6 +489,28 @@ TRACE_EVENT(kvm_test_age_hva,TP_printk ("mmu notifier test age hva: %#016lx" , __entry->hva) ); +TRACE_EVENT (kvm_fast_test_age_hva, + TP_PROTO(unsigned long start, unsigned long end, bool clear), +TP_ARGS (start, end, clear), + +TP_STRUCT__entry ( + __field( unsigned long, start ) + __field( unsigned long, end ) +__field ( bool, clear ) + ), + +TP_fast_assign ( + __entry->start = start; + __entry->end = end; + __entry->clear = clear; + ), + +TP_printk ("mmu notifier fast test age: hva: %#016lx -- %#016lx " +"clear: %d" , + __entry->start, __entry->end, + __entry->clear) +); +#endif /* _TRACE_KVM_MAIN_H */ /* This part must be outside protection */
virt/kvm/Kconfig +4 -0
@@ -100,6 +100,10 @@ config KVM_GENERIC_MMU_NOTIFIER config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS bool +config HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER + select KVM_GENERIC_MMU_NOTIFIER + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool
virt/kvm/kvm_main.c +38 -9
79 lines not on this page · raw message
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
PATCH v5 6/9 KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask
In reply to James HoughtonFrom: Yu Zhao <[email protected]>
tdp_mmu_enabled and shadow_accessed_mask are needed to implement kvm_arch_young_notifier_likely_fast().
--- arch/x86/include/asm/kvm_host.h | 6 ++++++ arch/x86/kvm/mmu.h | 6 ------ arch/x86/kvm/mmu/spte.h | 1 - 3 files changed, 6 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +6 -0
@@ -1855,6 +1855,7 @@ struct kvm_arch_async_pf {extern u32 __read_mostly kvm_nr_uret_msrs;extern u64 __read_mostly host_efer; +extern u64 __read_mostly shadow_accessed_mask;extern bool __read_mostly allow_smaller_maxphyaddr;extern bool __read_mostly enable_apicv;extern struct kvm_x86_ops kvm_x86_ops;
@@ -1960,6 +1961,11 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin, bool mask);extern bool tdp_enabled; +#ifdef CONFIG_X86_64 +extern bool tdp_mmu_enabled; +#else +#define tdp_mmu_enabled false +#endif u64 vcpu_tsc_khz (struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu.h +0 -6
@@ -271,12 +271,6 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm)return smp_load_acquire (&kvm->arch.shadow_root_allocated); } -#ifdef CONFIG_X86_64 -extern bool tdp_mmu_enabled; -#else -#define tdp_mmu_enabled false -#endif -static inline bool kvm_memslots_have_rmaps (struct kvm *kvm) {return !tdp_mmu_enabled ||kvm_shadow_root_allocated (kvm);
arch/x86/kvm/mmu/spte.h +0 -1
@@ -172,7 +172,6 @@ extern u64 __read_mostly shadow_mmu_writable_mask;extern u64 __read_mostly shadow_nx_mask;extern u64 __read_mostly shadow_x_mask;/* mutual exclusive with nx_mask */ extern u64 __read_mostly shadow_user_mask; -extern u64 __read_mostly shadow_accessed_mask;extern u64 __read_mostly shadow_dirty_mask;extern u64 __read_mostly shadow_mmio_value;extern u64 __read_mostly shadow_mmio_mask;
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
PATCH v5 7/9 KVM: x86: Implement kvm_fast_test_age_gfn and kvm_fast_age_gfn
In reply to James HoughtonThe fast-only notifier will only report an accurate result when the shadow MMU is not in use.
Implement kvm_arch_young_notifier_likely_fast(), as MGLRU will check this function to see if it should even be attempting the fast-only notifier. We only want to attempt the notifier if there is a chance that it will succeed (i.e., that we're using the TDP MMU).
--- arch/x86/include/asm/kvm_host.h | 7 +++++ arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 50 ++++++++++++++++++++++++++++++--- 3 files changed, 54 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +7 -0
@@ -2338,4 +2338,11 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);*/ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_young_notifier_likely_fast kvm_arch_young_notifier_likely_fast +static inline bool kvm_arch_young_notifier_likely_fast (void ) +{ +return IS_ENABLED (CONFIG_X86_64 ) && tdp_mmu_enabled && + shadow_accessed_mask; +} +#endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/Kconfig +1 -0
@@ -24,6 +24,7 @@ config KVM select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + select HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +46 -4
78 lines not on this page · raw message
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
PATCH v5 8/9 mm: multi-gen LRU: Have secondary MMUs participate in aging
In reply to James HoughtonSecondary MMUs are currently consulted for access/age information at eviction time, but before then, we don't get accurate age information. That is, pages that are mostly accessed through a secondary MMU (like guest memory, used by KVM) will always just proceed down to the oldest generation, and then at eviction time, if KVM reports the page to be young, the page will be activated/promoted back to the youngest generation.
The added feature bit (0x8), if disabled, will make MGLRU behave as if there are no secondary MMUs subscribed to MMU notifiers except at eviction time.
Implement aging with the new mmu_notifier_test_clear_young_fast_only() notifier. For architectures that do not support this notifier, this becomes a no-op. For architectures that do implement it, it should be fast enough to make aging worth it.
---
Notes:
should_look_around() can sometimes use two notifiers now instead of one.
This simply comes from restricting myself from not changing mmu_notifier_clear_young() to return more than just "young or not".
I could change mmu_notifier_clear_young() (and mmu_notifier_test_young()) to return if it was fast or not. At that point, I could just as well combine all the notifiers into one notifier, like what was in v2 and v3.
Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 185 ++++++++++++++---- 4 files changed, 164 insertions(+), 42 deletions(-)
Documentation/admin-guide/mm/multigen_lru.rst +5 -1
@@ -48,6 +48,10 @@ Values Components verified on x86 varieties other than Intel and AMD. If it is disabled, the multi-gen LRU will suffer a negligible performance degradation. +0x0008 Continuously clear the accessed bit in secondary MMU page + tables instead of waiting until eviction time. This results in + accurate page age information for pages that are mainly used by + a secondary MMU. [yYnN] Apply to all the components above. ====== ===============================================================
@@ -56,7 +60,7 @@ E.g., echo y >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled - 0x0007 + 0x000f echo 5 >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0005
include/linux/mmzone.h +4 -2
@@ -400,6 +400,7 @@ enum {LRU_GEN_CORE ,LRU_GEN_MM_WALK ,LRU_GEN_NONLEAF_YOUNG , +LRU_GEN_SECONDARY_MMU_WALK , NR_LRU_GEN_CAPS };
@@ -557,7 +558,7 @@ struct lru_gen_memcg {void lru_gen_init_pgdat (struct pglist_data *pgdat);void lru_gen_init_lruvec (struct lruvec *lruvec); -void lru_gen_look_around (struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around (struct page_vma_mapped_walk *pvmw);void lru_gen_init_memcg (struct mem_cgroup *memcg);void lru_gen_exit_memcg (struct mem_cgroup *memcg);
@@ -576,8 +577,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } -static inline void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around (struct page_vma_mapped_walk *pvmw) { +return false; }static inline void lru_gen_init_memcg (struct mem_cgroup *memcg)
mm/rmap.c +3 -6
@@ -870,13 +870,10 @@ static bool folio_referenced_one(struct folio *folio,continue ; } -if (pvmw.pte) { -if (lru_gen_enabled () && -pte_young (ptep_get (pvmw.pte))) { -lru_gen_look_around (&pvmw); +if (lru_gen_enabled () && pvmw.pte) { +if (lru_gen_look_around (&pvmw)) referenced++; - } - + }else if (pvmw.pte) {if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) referenced++;
mm/vmscan.c +152 -33
351 lines not on this page · raw message
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog
Sean Christopherson
maintainerRe: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
Quoted 11 lines · James Houghton
mm/rmap.c +0
-3@@ -870,13 +870,10 @@ static bool folio_referenced_one(struct folio *folio,continue ; } -if (pvmw.pte) { -if (lru_gen_enabled () && -pte_young (ptep_get (pvmw.pte))) {
mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 879
Quoted 10 lines
mm/rmap.c +3
-3-lru_gen_look_around (&pvmw); +if (lru_gen_enabled () && pvmw.pte) { +if (lru_gen_look_around (&pvmw)) referenced++; - } - + }else if (pvmw.pte) {if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) referenced++;
Random question not really related to KVM/secondary MMU participation. AFAICT, the MGLRU approach doesn't flush TLBs after aging pages. How does MGLRU mitigate false negatives on pxx_young() due to the CPU not setting Accessed bits because of stale TLB entries?
Re: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
Quoted 20 lines · Sean Christopherson
Quoted 18 lines · James Houghton
mm/rmap.c +3
-6@@ -870,13 +870,10 @@ static bool folio_referenced_one(struct folio *folio,continue ; } -if (pvmw.pte) { -if (lru_gen_enabled () && -pte_young (ptep_get (pvmw.pte))) { -lru_gen_look_around (&pvmw); +if (lru_gen_enabled () && pvmw.pte) { +if (lru_gen_look_around (&pvmw)) referenced++; - } - + }else if (pvmw.pte) {
Quoted 8 lines
mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 879
Quoted 3 lines
mm/rmap.c +0
-0if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) referenced++;Random question not really related to KVM/secondary MMU participation. AFAICT, the MGLRU approach doesn't flush TLBs after aging pages. How does MGLRU mitigate false negatives on
pxx_young()due to the CPU not setting Accessed bits because of stale TLB entries?
I do think there can be false negatives but we have not been able to measure their practical impacts since we disabled the flush on some host MMUs long ago (NOT by MGLRU), e.g., on x86 and ppc, ptep_clear_flush_young() is just ptep_test_andclear_young(). The theoretical basis is that, given the TLB coverage trend (Figure 1 in [1]), when a system is running out of memory, it's unlikely to have many long-lived entries in its TLB. IOW, if that system had a stable working set (hot memory) that can fit into its TLB, it wouldn't hit page reclaim. Again, this is based on the theory (proposition) that for most systems, their TLB coverages are much smaller than their memory sizes.
If/when the above proposition doesn't hold, the next step in the page reclaim path, which is to unmap the PTE, will cause a page fault. The fault can be minor or major (requires IO), depending on the race between the reclaiming and accessing threads. In this case, the tradeoff, in a steady state, is between the PF cost of pages we shouldn't reclaim and the flush cost of pages we scan. The PF cost is higher than the flush cost per page. But we scan many pages and only reclaim a few of them; pages we shouldn't reclaim are a (small) portion of the latter.
[1] https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdf
Sean Christopherson
maintainerRe: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
Quoted 30 lines · Yu Zhao
Quoted 20 lines · Sean Christopherson
Quoted 18 lines · James Houghton
mm/rmap.c +3
-6@@ -870,13 +870,10 @@ static bool folio_referenced_one(struct folio *folio,continue ; } -if (pvmw.pte) { -if (lru_gen_enabled () && -pte_young (ptep_get (pvmw.pte))) { -lru_gen_look_around (&pvmw); +if (lru_gen_enabled () && pvmw.pte) { +if (lru_gen_look_around (&pvmw)) referenced++; - } - + }else if (pvmw.pte) {Quoted 8 lines
mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 879
Quoted 3 lines
mm/rmap.c +0
-0if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) referenced++;Random question not really related to KVM/secondary MMU participation. AFAICT, the MGLRU approach doesn't flush TLBs after aging pages. How does MGLRU mitigate false negatives on
pxx_young()due to the CPU not setting Accessed bits because of stale TLB entries?
Quoted 4 lines
I do think there can be false negatives but we have not been able to measure their practical impacts since we disabled the flush on some host MMUs long ago (NOT by MGLRU), e.g., on x86 and ppc,
ptep_clear_flush_young()is justptep_test_andclear_young().
Aha! That's what I was missing, I somehow didn't see x86's ptep_clear_flush_young().
That begs the question, why does KVM flush TLBs on architectures that don't need to? And since kvm_mmu_notifier_clear_young() explicitly doesn't flush, are there even any KVM-supported architectures for which the flush is mandatory?
Skipping the flush on KVM x86 seems like a complete no-brainer.
Will, Marc and/or Oliver, what are arm64's requirements in this area? E.g. I see that arm64's version of __ptep_clear_flush_young() does TLBI but not DSB. Should KVM be doing something similar? Can KVM safely skip even the TBLI?
Quoted 19 lines
theoretical basis is that, given the TLB coverage trend (Figure 1 in [1]), when a system is running out of memory, it's unlikely to have many long-lived entries in its TLB. IOW, if that system had a stable working set (hot memory) that can fit into its TLB, it wouldn't hit page reclaim. Again, this is based on the theory (proposition) that for most systems, their TLB coverages are much smaller than their memory sizes.
If/when the above proposition doesn't hold, the next step in the page reclaim path, which is to unmap the PTE, will cause a page fault. The fault can be minor or major (requires IO), depending on the race between the reclaiming and accessing threads. In this case, the tradeoff, in a steady state, is between the PF cost of pages we shouldn't reclaim and the flush cost of pages we scan. The PF cost is higher than the flush cost per page. But we scan many pages and only reclaim a few of them; pages we shouldn't reclaim are a (small) portion of the latter.
[1] https://www.usenix.org/legacy/events/osdi02/tech/full_papers/navarro/navarro.pdf
Re: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · Yu Zhao
I do think there can be false negatives but we have not been able to measure their practical impacts since we disabled the flush on some host MMUs long ago (NOT by MGLRU), e.g., on x86 and ppc,
ptep_clear_flush_young()is justptep_test_andclear_young().Aha! That's what I was missing, I somehow didn't see x86's
ptep_clear_flush_young().
Heh, well the helper name isn't exactly giving any hints...
Quoted 9 lines
That begs the question, why does KVM flush TLBs on architectures that don't need to? And since
kvm_mmu_notifier_clear_young()explicitly doesn't flush, are there even any KVM-supported architectures for which the flush is mandatory?Skipping the flush on KVM x86 seems like a complete no-brainer.
Will, Marc and/or Oliver, what are arm64's requirements in this area? E.g. I see that arm64's version of
__ptep_clear_flush_young()does TLBI but not DSB. Should KVM be doing something similar? Can KVM safely skip even the TBLI?
Short answer, yes, KVM can elide TLBIs when clearing AF.
Long answer: Software needs to be extremely careful to ensure that TLBI elision doesn't lead to a failure to uphold break-before-make requirements, if we're only concerned with architecture-specific requirements. IOW, the AF cannot be used as a hint for the presence of TLB entries for a given PTE.
There's the obvious failure of skipping TLBIs for old pages when unmapping, but that isn't an architecture-specific issue.
So, since KVM/arm64 doesn't play any games with the AF at stage-2, leaving out a TLBI when aging ought to be fine.
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
In reply to James HoughtonQuoted 38 lines · James Houghton
Secondary MMUs are currently consulted for access/age information at eviction time, but before then, we don't get accurate age information. That is, pages that are mostly accessed through a secondary MMU (like guest memory, used by KVM) will always just proceed down to the oldest generation, and then at eviction time, if KVM reports the page to be young, the page will be activated/promoted back to the youngest generation.
The added feature bit (
0x8), if disabled, will make MGLRU behave as if there are no secondary MMUs subscribed to MMU notifiers except at eviction time.Implement aging with the new
mmu_notifier_test_clear_young_fast_only()notifier. For architectures that do not support this notifier, this becomes a no-op. For architectures that do implement it, it should be fast enough to make aging worth it.Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>---
Notes:
should_look_around()can sometimes use two notifiers now instead of one.This simply comes from restricting myself from not changing
mmu_notifier_clear_young()to return more than just "young or not".I could change
mmu_notifier_clear_young()(andmmu_notifier_test_young()) to return if it was fast or not. At that point, I could just as well combine all the notifiers into one notifier, like what was in v2 and v3.Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 185 ++++++++++++++---- 4 files changed, 164 insertions(+), 42 deletions(-)
...
Quoted 31 lines
static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
structmm_walk*args)
{mm/vmscan.c +5
-3@@ -3357,8 +3416,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,struct pglist_data *pgdat =lruvec_pgdat (walk->lruvec);DEFINE_MAX_SEQ (walk->lruvec);int old_gen, new_gen =lru_gen_from_seq (max_seq); +struct mm_struct *mm = args->mm; - pte =pte_offset_map_nolock (args->mm, pmd, start &PMD_MASK , &ptl); + pte =pte_offset_map_nolock (mm, pmd, start &PMD_MASK , &ptl);if (!pte)return false;if (!spin_trylock (ptl)) {@@ -3376,11 +3436,12 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, total++; walk->mm_stats[MM_LEAF_TOTAL ]++; - pfn =get_pte_pfn (ptent, args->vma, addr); + pfn =get_pte_pfn (ptent, args->vma, addr, pgdat);if (pfn ==-1 )continue ; -if (!pte_young (ptent)) { +if (!pte_young (ptent) && + !lru_gen_notifier_test_young (mm, addr)) { walk->mm_stats[MM_LEAF_OLD ]++;continue ; }@@ -3389,8 +3450,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)
mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3458
Quoted 10 lines
mm/vmscan.c +3
-2continue ; -if (!ptep_test_and_clear_young (args->vma, addr, pte + i)) -VM_WARN_ON_ONCE (true); +lru_gen_notifier_clear_young (mm, addr, addr +PAGE_SIZE ); +if (pte_young (ptent)) +ptep_test_and_clear_young (args->vma, addr, pte + i); young++; walk->mm_stats[MM_LEAF_YOUNG ]++;
There are two ways to structure the test conditions in walk_pte_range():
1. a single pass into the MMU notifier (combine test/clear) which causes a cache miss from get_pfn_page() if the page is NOT young.
2. two passes into the MMU notifier (separate test/clear) if the page is young, which does NOT cause a cache miss if the page is NOT young.
v2 can batch up to 64 PTEs, i.e., it only goes into the MMU notifier twice every 64 PTEs, and therefore the second option is a clear win.
But you are doing twice per PTE. So what's the rationale behind going with the second option? Was the first option considered?
In addition, what about the non-lockless cases? Would this change make them worse by grabbing the MMU lock twice per PTE?
Re: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
Quoted 92 lines · Yu Zhao
Quoted 38 lines · James Houghton
Secondary MMUs are currently consulted for access/age information at eviction time, but before then, we don't get accurate age information. That is, pages that are mostly accessed through a secondary MMU (like guest memory, used by KVM) will always just proceed down to the oldest generation, and then at eviction time, if KVM reports the page to be young, the page will be activated/promoted back to the youngest generation.
The added feature bit (
0x8), if disabled, will make MGLRU behave as if there are no secondary MMUs subscribed to MMU notifiers except at eviction time.Implement aging with the new
mmu_notifier_test_clear_young_fast_only()notifier. For architectures that do not support this notifier, this becomes a no-op. For architectures that do implement it, it should be fast enough to make aging worth it.Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>---
Notes:
should_look_around()can sometimes use two notifiers now instead of one.This simply comes from restricting myself from not changing
mmu_notifier_clear_young()to return more than just "young or not".I could change
mmu_notifier_clear_young()(andmmu_notifier_test_young()) to return if it was fast or not. At that point, I could just as well combine all the notifiers into one notifier, like what was in v2 and v3.Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 185 ++++++++++++++---- 4 files changed, 164 insertions(+), 42 deletions(-)...
Quoted 31 lines
static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
structmm_walk*args)
{mm/vmscan.c +5
-3@@ -3357,8 +3416,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,struct pglist_data *pgdat =lruvec_pgdat (walk->lruvec);DEFINE_MAX_SEQ (walk->lruvec);int old_gen, new_gen =lru_gen_from_seq (max_seq); +struct mm_struct *mm = args->mm; - pte =pte_offset_map_nolock (args->mm, pmd, start &PMD_MASK , &ptl); + pte =pte_offset_map_nolock (mm, pmd, start &PMD_MASK , &ptl);if (!pte)return false;if (!spin_trylock (ptl)) {@@ -3376,11 +3436,12 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, total++; walk->mm_stats[MM_LEAF_TOTAL ]++; - pfn =get_pte_pfn (ptent, args->vma, addr); + pfn =get_pte_pfn (ptent, args->vma, addr, pgdat);if (pfn ==-1 )continue ; -if (!pte_young (ptent)) { +if (!pte_young (ptent) && + !lru_gen_notifier_test_young (mm, addr)) { walk->mm_stats[MM_LEAF_OLD ]++;continue ; }@@ -3389,8 +3450,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3458
Quoted 10 lines
mm/vmscan.c +3
-2continue ; -if (!ptep_test_and_clear_young (args->vma, addr, pte + i)) -VM_WARN_ON_ONCE (true); +lru_gen_notifier_clear_young (mm, addr, addr +PAGE_SIZE ); +if (pte_young (ptent)) +ptep_test_and_clear_young (args->vma, addr, pte + i); young++; walk->mm_stats[MM_LEAF_YOUNG ]++;There are two ways to structure the test conditions in
walk_pte_range():
1. a single pass into the MMU notifier (combine test/clear) which causes a cache miss fromget_pfn_page()if the page is NOT young.
2. two passes into the MMU notifier (separate test/clear) if the page is young, which does NOT cause a cache miss if the page is NOT young.
Quoted 5 lines
v2 can batch up to 64 PTEs, i.e., it only goes into the MMU notifier twice every 64 PTEs, and therefore the second option is a clear win.
But you are doing twice per PTE. So what's the rationale behind going with the second option? Was the first option considered?
Hi Yu,
I didn't consider changing this from your v2[1]. Thanks for bringing it up.
The only real change I have made is that I reordered the (!test_spte_young() && !pte_young()) to what it is now (!pte_young() && !lru_gen_notifier_test_young()) because pte_young() can be evaluated much faster.
I am happy to change the initial test_young() notifier to a clear_young() (and drop the later clear_young(). In fact, I think I should. Making the condition (!pte_young() && !lru_gen_notifier_clear_young()) makes sense to me. This returns the same result as if it were !lru_gen_notifier_test_young() instead, there is no need for a second clear_young(), and we don't call get_pfn_folio() on pages that are not young.
WDYT? Have I misunderstood your comment?
Also, I take it your comment was not just about walk_pte_range() but about the similar bits in lru_gen_look_around() as well, so I'll make whatever changes we agree on there too (or maybe factor out the common bits).
[1]: https://lore.kernel.org/kvmarm/[email protected]/
Quoted 2 lines
In addition, what about the non-lockless cases? Would this change make them worse by grabbing the MMU lock twice per PTE?
That's a good point. Yes I think calling the notifier twice here would indeed exacerbate problems with a non-lockless notifier.
Thanks!
Re: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
Quoted 109 lines · James Houghton
Quoted 92 lines · Yu Zhao
Quoted 38 lines · James Houghton
Secondary MMUs are currently consulted for access/age information at eviction time, but before then, we don't get accurate age information. That is, pages that are mostly accessed through a secondary MMU (like guest memory, used by KVM) will always just proceed down to the oldest generation, and then at eviction time, if KVM reports the page to be young, the page will be activated/promoted back to the youngest generation.
The added feature bit (
0x8), if disabled, will make MGLRU behave as if there are no secondary MMUs subscribed to MMU notifiers except at eviction time.Implement aging with the new
mmu_notifier_test_clear_young_fast_only()notifier. For architectures that do not support this notifier, this becomes a no-op. For architectures that do implement it, it should be fast enough to make aging worth it.Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>---
Notes:
should_look_around()can sometimes use two notifiers now instead of one.This simply comes from restricting myself from not changing
mmu_notifier_clear_young()to return more than just "young or not".I could change
mmu_notifier_clear_young()(andmmu_notifier_test_young()) to return if it was fast or not. At that point, I could just as well combine all the notifiers into one notifier, like what was in v2 and v3.Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 185 ++++++++++++++---- 4 files changed, 164 insertions(+), 42 deletions(-)...
Quoted 31 lines
static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
structmm_walk*args)
{mm/vmscan.c +5
-3@@ -3357,8 +3416,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,struct pglist_data *pgdat =lruvec_pgdat (walk->lruvec);DEFINE_MAX_SEQ (walk->lruvec);int old_gen, new_gen =lru_gen_from_seq (max_seq); +struct mm_struct *mm = args->mm; - pte =pte_offset_map_nolock (args->mm, pmd, start &PMD_MASK , &ptl); + pte =pte_offset_map_nolock (mm, pmd, start &PMD_MASK , &ptl);if (!pte)return false;if (!spin_trylock (ptl)) {@@ -3376,11 +3436,12 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, total++; walk->mm_stats[MM_LEAF_TOTAL ]++; - pfn =get_pte_pfn (ptent, args->vma, addr); + pfn =get_pte_pfn (ptent, args->vma, addr, pgdat);if (pfn ==-1 )continue ; -if (!pte_young (ptent)) { +if (!pte_young (ptent) && + !lru_gen_notifier_test_young (mm, addr)) { walk->mm_stats[MM_LEAF_OLD ]++;continue ; }@@ -3389,8 +3450,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3458
Quoted 10 lines
mm/vmscan.c +3
-2continue ; -if (!ptep_test_and_clear_young (args->vma, addr, pte + i)) -VM_WARN_ON_ONCE (true); +lru_gen_notifier_clear_young (mm, addr, addr +PAGE_SIZE ); +if (pte_young (ptent)) +ptep_test_and_clear_young (args->vma, addr, pte + i); young++; walk->mm_stats[MM_LEAF_YOUNG ]++;There are two ways to structure the test conditions in
walk_pte_range():
1. a single pass into the MMU notifier (combine test/clear) which causes a cache miss fromget_pfn_page()if the page is NOT young.
2. two passes into the MMU notifier (separate test/clear) if the page is young, which does NOT cause a cache miss if the page is NOT young.Quoted 5 lines
v2 can batch up to 64 PTEs, i.e., it only goes into the MMU notifier twice every 64 PTEs, and therefore the second option is a clear win.
But you are doing twice per PTE. So what's the rationale behind going with the second option? Was the first option considered?
Hi Yu,
I didn't consider changing this from your v2[1]. Thanks for bringing it up.
The only real change I have made is that I reordered the (!test_spte_young() && !pte_young()) to what it is now (!pte_young() && !lru_gen_notifier_test_young()) because
pte_young()can be evaluated much faster.
Quoted 7 lines
I am happy to change the initial
test_young()notifier to aclear_young()(and drop the laterclear_young(). In fact, I think I should. Making the condition (!pte_young() && !lru_gen_notifier_clear_young()) makes sense to me. This returns the same result as if it were !lru_gen_notifier_test_young() instead, there is no need for a secondclear_young(), and we don't callget_pfn_folio()on pages that are not young.
We don't want to do that because we would lose the A-bit for a folio that's beyond the current reclaim scope, i.e., the cases where get_pfn_folio() returns NULL (a folio from another memcg, e.g.).
Quoted 1 line
WDYT? Have I misunderstood your comment?
I hope this is clear enough:
@@ -3395,7 +3395,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
if (pfn == -1)
continue;- if (!pte_young(ptent)) {
+ if (!pte_young(ptent) && !mm_has_notifiers(args->mm)) {
walk->mm_stats[MM_LEAF_OLD]++;
continue;
}
@@ -3404,8 +3404,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned
long start, unsigned long end,
if (!folio)
continue;- if (!ptep_test_and_clear_young(args->vma, addr, pte + i))
- VM_WARN_ON_ONCE(true);
+ if (!ptep_clear_young_notify(args->vma, addr, pte + i))
+ continue;
young++;
walk->mm_stats[MM_LEAF_YOUNG]++;Quoted 5 lines
Also, I take it your comment was not just about
walk_pte_range()but about the similar bits inlru_gen_look_around()as well, so I'll make whatever changes we agree on there too (or maybe factor out the common bits).
Quoted 7 lines
[1]: https://lore.kernel.org/kvmarm/[email protected]/
Quoted 2 lines
In addition, what about the non-lockless cases? Would this change make them worse by grabbing the MMU lock twice per PTE?
That's a good point. Yes I think calling the notifier twice here would indeed exacerbate problems with a non-lockless notifier.
I think so too, but I haven't verified it. Please do?
Re: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
Quoted 68 lines · Yu Zhao
Quoted 39 lines · James Houghton
Quoted 22 lines · Yu Zhao
mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3458
Quoted 12 lines · James Houghton
mm/vmscan.c +3
-2@@ -3389,8 +3450,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)continue ; -if (!ptep_test_and_clear_young (args->vma, addr, pte + i)) -VM_WARN_ON_ONCE (true); +lru_gen_notifier_clear_young (mm, addr, addr +PAGE_SIZE ); +if (pte_young (ptent)) +ptep_test_and_clear_young (args->vma, addr, pte + i); young++; walk->mm_stats[MM_LEAF_YOUNG ]++;There are two ways to structure the test conditions in
walk_pte_range():
1. a single pass into the MMU notifier (combine test/clear) which causes a cache miss fromget_pfn_page()if the page is NOT young.
2. two passes into the MMU notifier (separate test/clear) if the page is young, which does NOT cause a cache miss if the page is NOT young.Quoted 5 lines
v2 can batch up to 64 PTEs, i.e., it only goes into the MMU notifier twice every 64 PTEs, and therefore the second option is a clear win.
But you are doing twice per PTE. So what's the rationale behind going with the second option? Was the first option considered?
Hi Yu,
I didn't consider changing this from your v2[1]. Thanks for bringing it up.
The only real change I have made is that I reordered the (!test_spte_young() && !pte_young()) to what it is now (!pte_young() && !lru_gen_notifier_test_young()) because
pte_young()can be evaluated much faster.Quoted 7 lines
I am happy to change the initial
test_young()notifier to aclear_young()(and drop the laterclear_young(). In fact, I think I should. Making the condition (!pte_young() && !lru_gen_notifier_clear_young()) makes sense to me. This returns the same result as if it were !lru_gen_notifier_test_young() instead, there is no need for a secondclear_young(), and we don't callget_pfn_folio()on pages that are not young.We don't want to do that because we would lose the A-bit for a folio that's beyond the current reclaim scope, i.e., the cases where
get_pfn_folio()returns NULL (a folio from another memcg, e.g.).Quoted 1 line
WDYT? Have I misunderstood your comment?
I hope this is clear enough:
mm/vmscan.c +1
-1@@ -3395,7 +3395,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (pfn == -1) continue; -if (!pte_young (ptent)) { +if (!pte_young (ptent) && !mm_has_notifiers (args->mm)) { walk->mm_stats[MM_LEAF_OLD ]++;continue ;}
mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned · line 3408
Quoted 7 lines
mm/vmscan.c +1
-2@@ -3404,8 +3404,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (!folio) continue; -if (!ptep_test_and_clear_young (args->vma, addr, pte + i)) -VM_WARN_ON_ONCE (true); +if (!ptep_clear_young_notify (args->vma, addr, pte + i))
walk->mm_stats[MM_LEAF_OLD]++ should be here, I take it.
Quoted 12 lines
mm/vmscan.c +1
-0+continue ; young++;walk->mm_stats[MM_LEAF_YOUNG]++;Quoted 7 lines
Also, I take it your comment was not just about
walk_pte_range()but about the similar bits inlru_gen_look_around()as well, so I'll make whatever changes we agree on there too (or maybe factor out the common bits).
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
In addition, what about the non-lockless cases? Would this change make them worse by grabbing the MMU lock twice per PTE?
That's a good point. Yes I think calling the notifier twice here would indeed exacerbate problems with a non-lockless notifier.
I think so too, but I haven't verified it. Please do?
I have some results now, sorry for the wait.
It seems like one notifier is definitely better. It doesn't look like the read lock actually made anything worse with what I was testing (faulting memory in while doing aging). This is kind of surprising, but either way, I'll change it to the single notifier in v6. Thanks Yu!
Here are the results I'm basing this conclusion on, using the selftest added at the end of this series.
# Use taskset to minimize NUMA concern.
# Give an extra core for the aging thread.
# THPs disabled (echo never > /sys/kernel/mm/transparent_hugepage/enabled)
x86:
# taskset -c 0-32 ./access_tracking_perf_test -l -v 32 # # One notifier Populating memory : 1.933017284s Writing to populated memory : 0.017323539s Reading from populated memory : 0.013113260s lru_gen: Aging : 0.894133259s lru_gen: Aging : 0.738950525s Writing to idle memory : 0.059661329s lru_gen: Aging : 0.922719935s lru_gen: Aging : 0.829129877s Reading from idle memory : 0.059095098s lru_gen: Aging : 0.922689975s
# # Two notifiers
Populating memory : 1.842645795s
Writing to populated memory : 0.017277075s
Reading from populated memory : 0.013047457slru_gen: Aging : 0.900751764slru_gen: Aging : 0.707203167s
Writing to idle memory : 0.060663733slru_gen: Aging : 1.539957250s <------ got longerlru_gen: Aging : 0.797475887s
Reading from idle memory : 0.084415591slru_gen: Aging : 1.539417121s <------ got longer
arm64*:
(*Patched to do aging; not done in v5 or v6. Doing this to see if the read lock is made substantially worse by using two notifiers vs. one.)
# taskset -c 0-16 ./access_tracking_perf_test -l -v 16 -m 3 # # One notifier Populating memory : 1.439261355s Writing to populated memory : 0.009755279s Reading from populated memory : 0.007714120s lru_gen: Aging : 0.540183328s lru_gen: Aging : 0.455427973s Writing to idle memory : 0.010130399s lru_gen: Aging : 0.563424247s lru_gen: Aging : 0.500419850s Reading from idle memory : 0.008519640s lru_gen: Aging : 0.563178643s
# # Two notifiers
Populating memory : 1.526805625s
Writing to populated memory : 0.009836118s
Reading from populated memory : 0.007757280slru_gen: Aging : 0.537770978slru_gen: Aging : 0.421915391s
Writing to idle memory : 0.010281959slru_gen: Aging : 0.971448688s <------ got longerlru_gen: Aging : 0.466956547s
Reading from idle memory : 0.008588559slru_gen: Aging : 0.971030648s <------ got longer
arm64, faulting memory in while aging:
# perf record -g -- taskset -c 0-16 ./access_tracking_perf_test -l -v 16 -m 3 -p # # One notifier vcpu wall time : 1.433908058s lru_gen avg pass duration : 0.172128073s, (passes:11, total:1.893408807s)
# # Two notifiers
vcpu wall time : 1.450387765slru_gen avg pass duration : 0.175652974s, (passes:10, total:1.756529744s)
# perf report
# # One notifier
- 6.25% 0.00% access_tracking [kernel.kallsyms] [k] try_to_inc_max_seq
- try_to_inc_max_seq
- 6.06% walk_page_range
__walk_page_range
- walk_pgd_range
- 6.04% walk_pud_range
- 4.73% __mmu_notifier_clear_young
+ 4.29% kvm_mmu_notifier_clear_young# # Two notifiers
- 6.43% 0.00% access_tracking [kernel.kallsyms] [k] try_to_inc_max_seq
- try_to_inc_max_seq
- 6.25% walk_page_range __walk_page_range
- walk_pgd_range
- 6.23% walk_pud_range
- 2.75% __mmu_notifier_test_young
+ 2.48% kvm_mmu_notifier_test_young
- 2.39% __mmu_notifier_clear_young
+ 2.19% kvm_mmu_notifier_clear_young
Re: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging
Quoted 98 lines · James Houghton
Quoted 68 lines · Yu Zhao
Quoted 39 lines · James Houghton
Quoted 22 lines · Yu Zhao
mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3458
Quoted 12 lines · James Houghton
mm/vmscan.c +3
-2@@ -3389,8 +3450,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)continue ; -if (!ptep_test_and_clear_young (args->vma, addr, pte + i)) -VM_WARN_ON_ONCE (true); +lru_gen_notifier_clear_young (mm, addr, addr +PAGE_SIZE ); +if (pte_young (ptent)) +ptep_test_and_clear_young (args->vma, addr, pte + i); young++; walk->mm_stats[MM_LEAF_YOUNG ]++;There are two ways to structure the test conditions in
walk_pte_range():
1. a single pass into the MMU notifier (combine test/clear) which causes a cache miss fromget_pfn_page()if the page is NOT young.
2. two passes into the MMU notifier (separate test/clear) if the page is young, which does NOT cause a cache miss if the page is NOT young.Quoted 5 lines
v2 can batch up to 64 PTEs, i.e., it only goes into the MMU notifier twice every 64 PTEs, and therefore the second option is a clear win.
But you are doing twice per PTE. So what's the rationale behind going with the second option? Was the first option considered?
Hi Yu,
I didn't consider changing this from your v2[1]. Thanks for bringing it up.
The only real change I have made is that I reordered the (!test_spte_young() && !pte_young()) to what it is now (!pte_young() && !lru_gen_notifier_test_young()) because
pte_young()can be evaluated much faster.Quoted 7 lines
I am happy to change the initial
test_young()notifier to aclear_young()(and drop the laterclear_young(). In fact, I think I should. Making the condition (!pte_young() && !lru_gen_notifier_clear_young()) makes sense to me. This returns the same result as if it were !lru_gen_notifier_test_young() instead, there is no need for a secondclear_young(), and we don't callget_pfn_folio()on pages that are not young.We don't want to do that because we would lose the A-bit for a folio that's beyond the current reclaim scope, i.e., the cases where
get_pfn_folio()returns NULL (a folio from another memcg, e.g.).Quoted 1 line
WDYT? Have I misunderstood your comment?
I hope this is clear enough:
mm/vmscan.c +1
-1@@ -3395,7 +3395,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (pfn == -1) continue; -if (!pte_young (ptent)) { +if (!pte_young (ptent) && !mm_has_notifiers (args->mm)) { walk->mm_stats[MM_LEAF_OLD ]++;continue ;}mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned · line 3408
Quoted 7 lines
mm/vmscan.c +1
-2@@ -3404,8 +3404,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (!folio) continue; -if (!ptep_test_and_clear_young (args->vma, addr, pte + i)) -VM_WARN_ON_ONCE (true); +if (!ptep_clear_young_notify (args->vma, addr, pte + i))walk->mm_stats[MM_LEAF_OLD]++ should be here, I take it.
Quoted 18 lines
mm/vmscan.c +1
-0+continue ; young++;walk->mm_stats[MM_LEAF_YOUNG]++;Quoted 12 lines
Also, I take it your comment was not just about
walk_pte_range()but about the similar bits inlru_gen_look_around()as well, so I'll make whatever changes we agree on there too (or maybe factor out the common bits).[1]: https://lore.kernel.org/kvmarm/[email protected]/
Quoted 2 lines
In addition, what about the non-lockless cases? Would this change make them worse by grabbing the MMU lock twice per PTE?
That's a good point. Yes I think calling the notifier twice here would indeed exacerbate problems with a non-lockless notifier.
Quoted 7 lines
Quoted 1 line
I think so too, but I haven't verified it. Please do?
I have some results now, sorry for the wait.
It seems like one notifier is definitely better. It doesn't look like the read lock actually made anything worse with what I was testing (faulting memory in while doing aging). This is kind of surprising,
Not at all if you were only doing the aging path, which only takes the lock for read.
Under memory pressure, we need to both the aging and eviction, and the latter has to take the lock for write (to unmap). And that's when the real contention happens, because the search space is too big -- the entire system memory for global reclaim -- unmapping can easily collide with clearing the A-bit.
Quoted 100 lines
but either way, I'll change it to the single notifier in v6. Thanks Yu!
Here are the results I'm basing this conclusion on, using the selftest added at the end of this series.
# Use taskset to minimize NUMA concern.
# Give an extra core for the aging thread.
# THPs disabled (echo never >/sys/kernel/mm/transparent_hugepage/enabled)x86:
# taskset -c 0-32 ./access_tracking_perf_test -l -v 32 # # One notifier Populating memory : 1.933017284s Writing to populated memory : 0.017323539s Reading from populated memory : 0.013113260s lru_gen: Aging : 0.894133259s lru_gen: Aging : 0.738950525s Writing to idle memory : 0.059661329s lru_gen: Aging : 0.922719935s lru_gen: Aging : 0.829129877s Reading from idle memory : 0.059095098s lru_gen: Aging : 0.922689975s# # Two notifiers
Populating memory : 1.842645795s
Writing to populated memory : 0.017277075s
Reading from populated memory : 0.013047457slru_gen: Aging : 0.900751764slru_gen: Aging : 0.707203167s
Writing to idle memory : 0.060663733slru_gen: Aging : 1.539957250s <------ got longerlru_gen: Aging : 0.797475887s
Reading from idle memory : 0.084415591slru_gen: Aging : 1.539417121s <------ got longerarm64*:
(*Patched to do aging; not done in v5 or v6. Doing this to see if the read lock is made substantially worse by using two notifiers vs. one.)# taskset -c 0-16 ./access_tracking_perf_test -l -v 16 -m 3 # # One notifier Populating memory : 1.439261355s Writing to populated memory : 0.009755279s Reading from populated memory : 0.007714120s lru_gen: Aging : 0.540183328s lru_gen: Aging : 0.455427973s Writing to idle memory : 0.010130399s lru_gen: Aging : 0.563424247s lru_gen: Aging : 0.500419850s Reading from idle memory : 0.008519640s lru_gen: Aging : 0.563178643s# # Two notifiers
Populating memory : 1.526805625s
Writing to populated memory : 0.009836118s
Reading from populated memory : 0.007757280slru_gen: Aging : 0.537770978slru_gen: Aging : 0.421915391s
Writing to idle memory : 0.010281959slru_gen: Aging : 0.971448688s <------ got longerlru_gen: Aging : 0.466956547s
Reading from idle memory : 0.008588559slru_gen: Aging : 0.971030648s <------ got longerarm64, faulting memory in while aging:
# perf record -g -- taskset -c 0-16 ./access_tracking_perf_test -l -v 16 -m 3 -p # # One notifier vcpu wall time : 1.433908058s lru_gen avg pass duration : 0.172128073s, (passes:11, total:1.893408807s)# # Two notifiers
vcpu wall time : 1.450387765slru_genavg pass duration : 0.175652974s, (passes:10, total:1.756529744s)# perf report # # One notifier - 6.25% 0.00% access_tracking [kernel.kallsyms] [k] try_to_inc_max_seq - try_to_inc_max_seq - 6.06% walk_page_range __walk_page_range - walk_pgd_range - 6.04% walk_pud_range - 4.73% __mmu_notifier_clear_young + 4.29% kvm_mmu_notifier_clear_young# # Two notifiers
- 6.43% 0.00%access_tracking[kernel.kallsyms] [k]try_to_inc_max_seq
-try_to_inc_max_seq
- 6.25%walk_page_range__walk_page_range
-walk_pgd_range
- 6.23%walk_pud_range
- 2.75%__mmu_notifier_test_young
+ 2.48%kvm_mmu_notifier_test_young
- 2.39%__mmu_notifier_clear_young
+ 2.19%kvm_mmu_notifier_clear_young
PATCH v5 9/9 KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test
In reply to James HoughtonThis test now has two modes of operation:
1. (default) To check how much vCPU performance was affected by access tracking (previously existed, now supports MGLRU aging).
2. (-p) To also benchmark how fast MGLRU can do aging while vCPUs are faulting in memory.
Mode (1) also serves as a way to verify that aging is working properly for pages only accessed by KVM. It will fail if one does not have the 0x8 lru_gen feature bit.
To support MGLRU, the test creates a memory cgroup, moves itself into it, then uses the lru_gen debugfs output to track memory in that cgroup. The logic to parse the lru_gen debugfs output has been put into selftests/kvm/lib/lru_gen_util.c.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 365 ++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ 4 files changed, 782 insertions(+), 30 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -22,6 +22,7 @@ LIBKVM += lib/elf.c LIBKVM += lib/guest_modes.c LIBKVM += lib/io.c LIBKVM += lib/kvm_util.c +LIBKVM += lib/lru_gen_util.c LIBKVM += lib/memstress.c LIBKVM += lib/guest_sprintf.c LIBKVM += lib/rbtree.c
tools/testing/selftests/kvm/access_tracking_perf_test.c +335 -30
484 lines not on this page · raw message
tools/testing/selftests/kvm/include/lru_gen_util.h new file +55 -0
55 lines not on this page · raw message
tools/testing/selftests/kvm/lib/lru_gen_util.c new file +391 -0
391 lines not on this page · raw message
Signature · 1 line
2.45.2.505.gda0bf45e8d-goog