mm: multi-gen LRU: Walk secondary MMU page tables while aging
Subsystems (13)
v7 cover mm: multi-gen LRU: Walk secondary MMU page tables while aging
This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim.
This series includes:
1. Cleanup, add support for locklessly memslot walks in KVM (patches 1-2).
2. Support for lockless aging for x86 TDP MMU (patches 3-4).
3. Further small optimizations (patches 5-6).
4. Support for lockless harvesting of access information for the x86 shadow MMU (patches 7-10).
5. Some mm cleanup (patch 11).
6. Add fast-only aging MMU notifiers (patches 12-13).
7. Support fast-only aging in KVM/x86 (patches 14-16).
8. Have KVM participate in MGLRU aging (patch 17).
9. Updates to the access_tracking_perf_test to verify MGLRU functionality (patch 18).
Patches 1-10 are pure optimizations and could be applied without the rest of the series, though the lockless shadow MMU lockless patches become more useful in the context of MGLRU aging.
Please note that mmu_notifier_test_young_fast_only() is added but not used in this series. I am happy to remove it if that would be appropriate.
The fast-only notifiers serve a particular purpose: for aging, we neither want to delay other operations (e.g. unmapping for eviction) nor do we want to be delayed by these other operations ourselves. By default, the implementations of test_young() and clear_young() are meant to be *accurate*, not fast. The fast-only notifiers will only give age information that can be gathered fast.
The fast-only notifiers are non-trivially implemented for only x86. The TDP MMU and the shadow MMU are both supported, but the shadow MMU will not actually age sptes locklessly if A/D bits in the spte have been disabled (i.e., if L1 disables them).
access_tracking_perf_test now has a mode (-p) to check performance of MGLRU aging while the VM is faulting memory in.
This series has been tested with access_tracking_perf_test and Sean's mmu_stress_test[6], both with tdp_mmu=0 and tdp_mmu=1.
=== Previous Versions ===
Since v6[1]:
- Rebased on top of kvm-x86/next and Sean's lockless rmap walking changes[6].
- Removed HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY (thanks DavidM).
- Split up kvm_age_gfn() / kvm_test_age_gfn() optimizations (thanks DavidM and Sean).
- Improved new MMU notifier documentation (thanks DavidH).
- Dropped arm64 locking change.
- No longer retry for CAS failure in TDP MMU non-A/D case (thanks Sean).
- Added some R-bys and A-bys.
Since v5[2]:
- Reworked test_clear_young_fast_only() into a new parameter for the existing notifiers (thanks Sean).
- Added mmu_notifier.has_fast_aging to tell mm if calling fast-only notifiers should be done.
- Added mm_has_fast_young_notifiers() to inform users if calling fast-only notifier helpers is worthwhile (for look-around to use).
- Changed MGLRU to invoke a single notifier instead of two when aging and doing look-around (thanks Yu).
- For KVM/x86, check indirect_shadow_pages > 0 instead of kvm_memslots_have_rmaps() when collecting age information (thanks Sean).
- For KVM/arm, some fixes from Oliver.
- Small fixes to access_tracking_perf_test.
- Added missing !MMU_NOTIFIER version of mmu_notifier_clear_young().
Since v4[3]:
- Removed Kconfig that controlled when aging was enabled. Aging will be done whenever the architecture supports it (thanks Yu).
- Added a new MMU notifier, test_clear_young_fast_only(), specifically for MGLRU to use.
- Add kvm_fast_{test_,}age_gfn, implemented by x86.
- Fix locking for clear_flush_young().
- Added KVM_MMU_NOTIFIER_YOUNG_LOCKLESS to clean up locking changes (thanks Sean).
- Fix WARN_ON and other cleanup for the arm64 locking changes (thanks Oliver).
Since v3[4]:
- Vastly simplified the series (thanks David). Removed mmu notifier batching logic entirely.
- Cleaned up how locking is done for mmu_notifier_test/clear_young (thanks David).
- Look-around is now only done when there are no secondary MMUs subscribed to MMU notifiers.
- CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added.
- Fixed the lockless implementation of kvm_{test,}age_gfn for x86 (thanks David).
- Added MGLRU functional and performance tests to access_tracking_perf_test (thanks Axel).
- In v3, an mm would be completely ignored (for aging) if there was a secondary MMU but support for secondary MMU walking was missing. Now, missing secondary MMU walking support simply skips the notifier calls (except for eviction).
- Added a sanity check for that range->lockless and range->on_lock are never both provided for the memslot walk.
For the changes since v2[5], see v3.
Based on latest kvm-x86/next.
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/linux-mm/[email protected]/
[4]: https://lore.kernel.org/linux-mm/[email protected]/
[5]: https://lore.kernel.org/kvmarm/[email protected]/
[6]: https://lore.kernel.org/kvm/[email protected]/
James Houghton (14):
KVM: Remove kvm_handle_hva_range helper functions
KVM: Add lockless memslot walk to KVM
KVM: x86/mmu: Factor out spte atomic bit clearing routine
KVM: x86/mmu: Relax locking for kvm_test_age_gfn and kvm_age_gfn
KVM: x86/mmu: Rearrange kvm_{test_,}age_gfn
KVM: x86/mmu: Only check gfn age in shadow MMU if
indirect_shadow_pages > 0
mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER
mm: Add has_fast_aging to struct mmu_notifier
mm: Add fast_only bool to test_young and clear_young MMU notifiers
KVM: Pass fast_only to kvm_{test_,}age_gfn
KVM: x86/mmu: Locklessly harvest access information from shadow MMU
KVM: x86/mmu: Enable has_fast_aging
mm: multi-gen LRU: Have secondary MMUs participate in aging
KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test
Sean Christopherson (4):
KVM: x86/mmu: Refactor low level rmap helpers to prep for walking w/o
mmu_lock
KVM: x86/mmu: Add infrastructure to allow walking rmaps outside of
mmu_lock
KVM: x86/mmu: Add support for lockless walks of rmap SPTEs
KVM: x86/mmu: Support rmap walks without holding mmu_lock when aging
gfns
Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/x86/include/asm/kvm_host.h | 4 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 355 ++++++++++++---- arch/x86/kvm/mmu/tdp_iter.h | 27 +- arch/x86/kvm/mmu/tdp_mmu.c | 57 ++- include/linux/kvm_host.h | 2 + include/linux/mmu_notifier.h | 82 +++- include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 19 +- mm/damon/vaddr.c | 2 - mm/mmu_notifier.c | 38 +- mm/rmap.c | 9 +- mm/vmscan.c | 148 +++++-- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 369 +++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ virt/kvm/Kconfig | 3 + virt/kvm/kvm_main.c | 124 +++--- 20 files changed, 1451 insertions(+), 248 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c
base-commit: 3cc25d5adcfd2a2c33baa0b2a1979c2dbc9b990b
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 1/18 KVM: Remove kvm_handle_hva_range helper functions
kvm_handle_hva_range is only used by the young notifiers. In a later patch, it will be even further tied to the young notifiers. Instead of renaming kvm_handle_hva_range to something like kvm_handle_hva_range_young, simply remove kvm_handle_hva_range. This seems slightly more readable, though there is slightly more code duplication.
Finally, rename __kvm_handle_hva_range to kvm_handle_hva_range, now that the name is available.
--- virt/kvm/kvm_main.c | 81 +++++++++++++++++++++------------------------ 1 file changed, 37 insertions(+), 44 deletions(-)
virt/kvm/kvm_main.c +37 -44
@@ -589,8 +589,8 @@ static void kvm_null_fn(void)node ; \ node = interval_tree_iter_next(node, start, last)) \ -static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, -const struct kvm_mmu_notifier_range *range) +static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, +const struct kvm_mmu_notifier_range *range) {struct kvm_mmu_notifier_return r = { .ret = false,
@@ -666,42 +666,6 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm,return r; } -static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, -unsigned long start, -unsigned long end, -gfn_handler_t handler) -{ -struct kvm *kvm =mmu_notifier_to_kvm (mn); -const struct kvm_mmu_notifier_range range = { - .start = start, - .end = end, - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .flush_on_ret = true, - .may_block = false, - }; - -return __kvm_handle_hva_range (kvm, &range).ret; -} - -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, -unsigned long start, -unsigned long end, -gfn_handler_t handler) -{ -struct kvm *kvm =mmu_notifier_to_kvm (mn); -const struct kvm_mmu_notifier_range range = { - .start = start, - .end = end, - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .flush_on_ret = false, - .may_block = false, - }; - -return __kvm_handle_hva_range (kvm, &range).ret; -} -void kvm_mmu_invalidate_begin (struct kvm *kvm) {lockdep_assert_held_write (&kvm->mmu_lock);
@@ -794,7 +758,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,* that guest memory has been reclaimed. This needs to be done *after* * dropping mmu_lock, as x86's reclaim path is slooooow. */ -if (__kvm_handle_hva_range (kvm, &hva_range).found_memslot) +if (kvm_handle_hva_range (kvm, &hva_range).found_memslot)kvm_arch_guest_memory_reclaimed (kvm);return 0 ;
@@ -840,7 +804,7 @@ static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn, };bool wake; -__kvm_handle_hva_range (kvm, &hva_range); +kvm_handle_hva_range (kvm, &hva_range);/* Pairs with the increment in range_start(). */ spin_lock (&kvm->mn_invalidate_lock);
@@ -862,9 +826,19 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, unsigned long start, unsigned long end) { +struct kvm *kvm =mmu_notifier_to_kvm (mn); +const struct kvm_mmu_notifier_range range = { + .start = start, + .end = end, + .handler = kvm_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = true, + .may_block = false, + }; +trace_kvm_age_hva (start, end); -return kvm_handle_hva_range (mn, start, end, kvm_age_gfn); +return kvm_handle_hva_range (kvm, &range).ret; } static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,
@@ -872,6 +846,16 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, unsigned long start, unsigned long end) { +struct kvm *kvm =mmu_notifier_to_kvm (mn); +const struct kvm_mmu_notifier_range range = { + .start = start, + .end = end, + .handler = kvm_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = false, + .may_block = false, + }; +trace_kvm_age_hva (start, end);/*
@@ -887,17 +871,26 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,* cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ -return kvm_handle_hva_range_no_flush (mn, start, end, kvm_age_gfn); +return kvm_handle_hva_range (kvm, &range).ret; }static int kvm_mmu_notifier_test_young (struct mmu_notifier *mn,struct mm_struct *mm,unsigned long address) { +struct kvm *kvm =mmu_notifier_to_kvm (mn); +const struct kvm_mmu_notifier_range range = { + .start = address, + .end = address +1 , + .handler = kvm_test_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = false, + .may_block = false, + }; +trace_kvm_test_age_hva (address); -return kvm_handle_hva_range_no_flush (mn, address, address +1 , - kvm_test_age_gfn); +return kvm_handle_hva_range (kvm, &range).ret; } static void kvm_mmu_notifier_release(struct mmu_notifier *mn,
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 2/18 KVM: Add lockless memslot walk to KVM
In reply to James HoughtonProvide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing.
Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS.
The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply.
--- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 28 +++++++++++++++++++++------- 3 files changed, 25 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +1 -0
@@ -266,6 +266,7 @@ struct kvm_gfn_range {gfn_t end;union kvm_mmu_notifier_arg arg;bool may_block; +bool lockless; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);
virt/kvm/Kconfig +3 -0
@@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool
virt/kvm/kvm_main.c +21 -7
@@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range {on_lock_fn_t on_lock;bool flush_on_ret;bool may_block; +bool lockless; };/*
@@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler)))return r; +/* on_lock will never be called for lockless walks */ +if (WARN_ON_ONCE (range->lockless && !IS_KVM_NULL_FN (range->on_lock))) +return r; + idx =srcu_read_lock (&kvm->srcu);for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) {
@@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, gfn_range.start =hva_to_gfn_memslot (hva_start, slot); gfn_range.end =hva_to_gfn_memslot (hva_end +PAGE_SIZE -1 , slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless;if (!r.found_memslot) { r.found_memslot = true; -KVM_MMU_LOCK (kvm); -if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm); - -if (IS_KVM_NULL_FN (range->handler)) - goto mmu_unlock; +if (!range->lockless) { +KVM_MMU_LOCK (kvm); +if (!IS_KVM_NULL_FN (range->on_lock)) + range->on_lock (kvm); + +if (IS_KVM_NULL_FN (range->handler)) + goto mmu_unlock; + } } r.ret |= range->handler (kvm, &gfn_range); }
@@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,kvm_flush_remote_tlbs (kvm); mmu_unlock: -if (r.found_memslot) +if (r.found_memslot && !range->lockless)KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);
@@ -834,6 +842,8 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };trace_kvm_age_hva (start, end);
@@ -854,6 +864,8 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };trace_kvm_age_hva (start, end);
@@ -886,6 +898,8 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), };trace_kvm_test_age_hva (address);
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 3/18 KVM: x86/mmu: Factor out spte atomic bit clearing routine
In reply to James HoughtonThis new function, tdp_mmu_clear_spte_bits_atomic(), will be used in a follow-up patch to enable lockless Accessed and R/W/X bit clearing.
--- arch/x86/kvm/mmu/tdp_iter.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu/tdp_iter.h +9 -4
@@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte)return xchg (rcu_dereference (sptep), new_spte); } +static inline u64 tdp_mmu_clear_spte_bits_atomic (tdp_ptep_t sptep,u64 mask) +{ +atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference (sptep); + +return (u64 )atomic64_fetch_and (~mask, sptep_atomic); +} +static inline void __kvm_tdp_mmu_write_spte (tdp_ptep_t sptep,u64 new_spte) {KVM_MMU_WARN_ON (is_ept_ve_possible (new_spte));
@@ -65,10 +72,8 @@ static inline u64 tdp_mmu_clear_spte_bits(tdp_ptep_t sptep, u64 old_spte, {atomic64_t *sptep_atomic; -if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) { - sptep_atomic = (atomic64_t *)rcu_dereference (sptep); -return (u64 )atomic64_fetch_and (~mask, sptep_atomic); - } +if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level)) +return tdp_mmu_clear_spte_bits_atomic (sptep, mask);__kvm_tdp_mmu_write_spte (sptep, old_spte & ~mask);return old_spte;
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 4/18 KVM: x86/mmu: Relax locking for kvm_test_age_gfn and kvm_age_gfn
In reply to James HoughtonWalk the TDP MMU in an RCU read-side critical section without holding mmu_lock when harvesting and potentially updating age information on sptes. This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the W/R/X bits).
If the cmpxchg for marking the spte for access tracking fails, leave it as is and treat it as if it were young, as if the spte is being actively modified, it is most likely young.
Harvesting age information from the shadow MMU is still done while holding the MMU write lock.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 10 ++++-- arch/x86/kvm/mmu/tdp_iter.h | 14 ++++---- arch/x86/kvm/mmu/tdp_mmu.c | 57 ++++++++++++++++++++++++--------- 5 files changed, 58 insertions(+), 25 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1454,6 +1454,7 @@ struct kvm_arch {* tdp_mmu_page set. * * For reads, this list is protected by: +* RCU alone or * the MMU lock in read mode + RCU or * the MMU lock in write mode *
arch/x86/kvm/Kconfig +1 -0
@@ -23,6 +23,7 @@ config KVM depends on X86_LOCAL_APIC select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSO
arch/x86/kvm/mmu/mmu.c +8 -2
@@ -1649,8 +1649,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_rmap_age_gfn_range (kvm, range, false); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_age_gfn_range (kvm, range);
@@ -1662,8 +1665,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) +if (kvm_memslots_have_rmaps (kvm)) { +write_lock (&kvm->mmu_lock); young =kvm_rmap_age_gfn_range (kvm, range, true); +write_unlock (&kvm->mmu_lock); + }if (tdp_mmu_enabled) young |=kvm_tdp_mmu_test_age_gfn (kvm, range);
arch/x86/kvm/mmu/tdp_iter.h +6 -8
@@ -39,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) }/* -* SPTEs must be modified atomically if they are shadow-present, leaf -* SPTEs, and have volatile bits, i.e. has bits that can be set outside -* of mmu_lock. The Writable bit can be set by KVM's fast page fault -* handler, and Accessed and Dirty bits can be set by the CPU. +* SPTEs must be modified atomically if they have bits that can be set outside +* of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the +* Writable bit can be set by KVM's fast page fault handler, the Accessed and +* Dirty bits can be set by the CPU, and the Accessed and R/X bits can be +* cleared by age_gfn_range. * * Note, non-leaf SPTEs do have Accessed bits and those bits are * technically volatile, but KVM doesn't consume the Accessed bit of
@@ -53,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte)static inline bool kvm_tdp_mmu_spte_need_atomic_write (u64 old_spte,int level) {return is_shadow_present_pte (old_spte) && -is_last_spte (old_spte, level) && -spte_has_volatile_bits (old_spte); +is_last_spte (old_spte, level); } static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
@@ -70,8 +70,6 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,static inline u64 tdp_mmu_clear_spte_bits (tdp_ptep_t sptep,u64 old_spte,u64 mask,int level) { -atomic64_t *sptep_atomic; -if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level))return tdp_mmu_clear_spte_bits_atomic (sptep, mask);
arch/x86/kvm/mmu/tdp_mmu.c +42 -15
91 lines not on this page · raw message
Signature · 1 line
2.46.0.792.g87dc391469-goog
Re: [PATCH v7 04/18] KVM: x86/mmu: Relax locking for kvm_test_age_gfn and kvm_age_gfn
Quoted 13 lines · James Houghton
Walk the TDP MMU in an RCU read-side critical section without holding
mmu_lockwhen harvesting and potentially updating age information on sptes. This requires a way to do RCU-safe walking of thetdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, andkvm_tdp_mmu_spte_need_atomic_write()has been updated to account for the fact thatkvm_age_gfncan now lockless update the accessed bit and the W/R/X bits).If the cmpxchg for marking the spte for access tracking fails, leave it as is and treat it as if it were young, as if the spte is being actively modified, it is most likely young.
Quoted 5 lines
Harvesting age information from the shadow MMU is still done while holding the MMU write lock.
Suggested-by Yu Zhao <[email protected]>Signed-off-by James Houghton <[email protected]>
Oh no! I have left off David Matlack's Reviewed-bys[1, 2] from this patch and from patch 2[3], and I failed to apply his comment suggestion on this patch like I said I would. Sorry David, I have fixed up my tree now.
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/kvm/[email protected]/
Re: [PATCH v7 04/18] KVM: x86/mmu: Relax locking for kvm_test_age_gfn and kvm_age_gfn
In reply to James Houghtonarch/x86/kvm/mmu/tdp_iter.h static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte, · line 75
Quoted 9 lines · James Houghton
arch/x86/kvm/mmu/tdp_iter.h +0
-2@@ -70,8 +70,6 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,static inline u64 tdp_mmu_clear_spte_bits (tdp_ptep_t sptep,u64 old_spte,u64 mask,int level) { -atomic64_t *sptep_atomic; -if (kvm_tdp_mmu_spte_need_atomic_write (old_spte, level))return tdp_mmu_clear_spte_bits_atomic (sptep, mask);
This delete should have gone in the previous patch. Will be fixed in v8.
PATCH v7 5/18 KVM: x86/mmu: Rearrange kvm_{test_,}age_gfn
In reply to James HoughtonReorder the TDP MMU check to be first for both kvm_test_age_gfn and kvm_age_gfn. For kvm_test_age_gfn, this allows us to completely avoid needing to grab the MMU lock when the TDP MMU reports that the page is young. Do the same for kvm_age_gfn merely for consistency.
--- arch/x86/kvm/mmu/mmu.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -9
@@ -1649,15 +1649,15 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; +if (tdp_mmu_enabled) + young =kvm_tdp_mmu_age_gfn_range (kvm, range); +if (kvm_memslots_have_rmaps (kvm)) {write_lock (&kvm->mmu_lock); - young =kvm_rmap_age_gfn_range (kvm, range, false); + young |=kvm_rmap_age_gfn_range (kvm, range, false);write_unlock (&kvm->mmu_lock); } -if (tdp_mmu_enabled) - young |=kvm_tdp_mmu_age_gfn_range (kvm, range); -return young; }
@@ -1665,15 +1665,15 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) {bool young = false; -if (kvm_memslots_have_rmaps (kvm)) { +if (tdp_mmu_enabled) + young =kvm_tdp_mmu_test_age_gfn (kvm, range); + +if (!young &&kvm_memslots_have_rmaps (kvm)) {write_lock (&kvm->mmu_lock); - young =kvm_rmap_age_gfn_range (kvm, range, true); + young |=kvm_rmap_age_gfn_range (kvm, range, true);write_unlock (&kvm->mmu_lock); } -if (tdp_mmu_enabled) - young |=kvm_tdp_mmu_test_age_gfn (kvm, range); -return young; }
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 6/18 KVM: x86/mmu: Only check gfn age in shadow MMU if indirect_shadow_pages > 0
In reply to James HoughtonOptimize both kvm_age_gfn and kvm_test_age_gfn's interaction with the shadow MMU by, rather than checking if our memslot has rmaps, check if there are any indirect_shadow_pages at all.
--- arch/x86/kvm/mmu/mmu.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -2
@@ -1645,6 +1645,11 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm,return young; } +static bool kvm_has_shadow_mmu_sptes (struct kvm *kvm) +{ +return !tdp_mmu_enabled ||READ_ONCE (kvm->arch.indirect_shadow_pages); +} +bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range) {bool young = false;
@@ -1652,7 +1657,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)if (tdp_mmu_enabled) young =kvm_tdp_mmu_age_gfn_range (kvm, range); -if (kvm_memslots_have_rmaps (kvm)) { +if (kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young |=kvm_rmap_age_gfn_range (kvm, range, false);write_unlock (&kvm->mmu_lock);
@@ -1668,7 +1673,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)if (tdp_mmu_enabled) young =kvm_tdp_mmu_test_age_gfn (kvm, range); -if (!young &&kvm_memslots_have_rmaps (kvm)) { +if (!young &&kvm_has_shadow_mmu_sptes (kvm)) {write_lock (&kvm->mmu_lock); young |=kvm_rmap_age_gfn_range (kvm, range, true);write_unlock (&kvm->mmu_lock);
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 7/18 KVM: x86/mmu: Refactor low level rmap helpers to prep for walking w/o mmu_lock
In reply to James HoughtonFrom: Sean Christopherson <[email protected]>
Refactor the pte_list and rmap code to always read and write rmap_head->val exactly once, e.g. by collecting changes in a local variable and then propagating those changes back to rmap_head->val as appropriate. This will allow implementing a per-rmap rwlock (of sorts) by adding a LOCKED bit into the rmap value alongside the MANY bit.
--- arch/x86/kvm/mmu/mmu.c | 83 +++++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 33 deletions(-)
arch/x86/kvm/mmu/mmu.c +50 -33
@@ -920,21 +920,24 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpustatic int pte_list_add (struct kvm_mmu_memory_cache *cache,u64 *spte,struct kvm_rmap_head *rmap_head) { +unsigned long old_val, new_val;struct pte_list_desc *desc;int count =0 ; -if (!rmap_head->val) { - rmap_head->val = (unsigned long )spte; - }else if (!(rmap_head->val &KVM_RMAP_MANY )) { + old_val = rmap_head->val; + +if (!old_val) { + new_val = (unsigned long )spte; + }else if (!(old_val &KVM_RMAP_MANY )) { desc =kvm_mmu_memory_cache_alloc (cache); - desc->sptes[0 ] = (u64 *)rmap_head->val; + desc->sptes[0 ] = (u64 *)old_val; desc->sptes[1 ] = spte; desc->spte_count =2 ; desc->tail_count =0 ; - rmap_head->val = (unsigned long )desc |KVM_RMAP_MANY ; + new_val = (unsigned long )desc |KVM_RMAP_MANY ; ++count; }else { - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY ); + desc = (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY ); count = desc->tail_count + desc->spte_count;/*
@@ -943,21 +946,25 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte,*/ if (desc->spte_count ==PTE_LIST_EXT ) { desc =kvm_mmu_memory_cache_alloc (cache); - desc->more = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY ); + desc->more = (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY ); desc->spte_count =0 ; desc->tail_count = count; - rmap_head->val = (unsigned long )desc |KVM_RMAP_MANY ; + new_val = (unsigned long )desc |KVM_RMAP_MANY ; + }else { + new_val = old_val; } desc->sptes[desc->spte_count++] = spte; } + + rmap_head->val = new_val; +return count; } -static void pte_list_desc_remove_entry (struct kvm *kvm, -struct kvm_rmap_head *rmap_head, +static void pte_list_desc_remove_entry (struct kvm *kvm,unsigned long *rmap_val,struct pte_list_desc *desc,int i) { -struct pte_list_desc *head_desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY ); +struct pte_list_desc *head_desc = (struct pte_list_desc *)(*rmap_val & ~KVM_RMAP_MANY );int j = head_desc->spte_count -1 ;/*
@@ -984,9 +991,9 @@ static void pte_list_desc_remove_entry(struct kvm *kvm,* head at the next descriptor, i.e. the new head. */ if (!head_desc->more) - rmap_head->val =0 ; + *rmap_val =0 ;else - rmap_head->val = (unsigned long )head_desc->more |KVM_RMAP_MANY ; + *rmap_val = (unsigned long )head_desc->more |KVM_RMAP_MANY ;mmu_free_pte_list_desc (head_desc); }
@@ -994,24 +1001,26 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, struct kvm_rmap_head *rmap_head) {struct pte_list_desc *desc; +unsigned long rmap_val;int i; -if (KVM_BUG_ON_DATA_CORRUPTION (!rmap_head->val, kvm)) -return ; + rmap_val = rmap_head->val; +if (KVM_BUG_ON_DATA_CORRUPTION (!rmap_val, kvm)) + goto out; -if (!(rmap_head->val &KVM_RMAP_MANY )) { -if (KVM_BUG_ON_DATA_CORRUPTION ((u64 *)rmap_head->val != spte, kvm)) -return ; +if (!(rmap_val &KVM_RMAP_MANY )) { +if (KVM_BUG_ON_DATA_CORRUPTION ((u64 *)rmap_val != spte, kvm)) + goto out; - rmap_head->val =0 ; + rmap_val =0 ; }else { - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY ); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY );while (desc) {for (i =0 ; i < desc->spte_count; ++i) {if (desc->sptes[i] == spte) { -pte_list_desc_remove_entry (kvm, rmap_head, +pte_list_desc_remove_entry (kvm, &rmap_val, desc, i); -return ; + goto out; } } desc = desc->more;
@@ -1019,6 +1028,9 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte,KVM_BUG_ON_DATA_CORRUPTION (true, kvm); } + +out: + rmap_head->val = rmap_val; } static void kvm_zap_one_rmap_spte(struct kvm *kvm,
@@ -1033,17 +1045,19 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, struct kvm_rmap_head *rmap_head) {struct pte_list_desc *desc, *next; +unsigned long rmap_val;int i; -if (!rmap_head->val) + rmap_val = rmap_head->val; +if (!rmap_val)return false; -if (!(rmap_head->val &KVM_RMAP_MANY )) { -mmu_spte_clear_track_bits (kvm, (u64 *)rmap_head->val); +if (!(rmap_val &KVM_RMAP_MANY )) { +mmu_spte_clear_track_bits (kvm, (u64 *)rmap_val); goto out; } - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY ); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY );for (; desc; desc = next) {for (i =0 ; i < desc->spte_count; i++)
@@ -1059,14 +1073,15 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm,unsigned int pte_list_count (struct kvm_rmap_head *rmap_head) { +unsigned long rmap_val = rmap_head->val;struct pte_list_desc *desc; -if (!rmap_head->val) +if (!rmap_val)return 0 ; -else if (!(rmap_head->val &KVM_RMAP_MANY )) +else if (!(rmap_val &KVM_RMAP_MANY ))return 1 ; - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY ); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY );return desc->tail_count + desc->spte_count; }
@@ -1109,6 +1124,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte)*/ struct rmap_iterator {/* private fields */ +struct rmap_head *head;struct pte_list_desc *desc;/* holds the sptep if not NULL */ int pos;/* index of the sptep */ };
@@ -1123,18 +1139,19 @@ struct rmap_iterator {static u64 *rmap_get_first (struct kvm_rmap_head *rmap_head,struct rmap_iterator *iter) { +unsigned long rmap_val = rmap_head->val;u64 *sptep; -if (!rmap_head->val) +if (!rmap_val)return NULL ; -if (!(rmap_head->val &KVM_RMAP_MANY )) { +if (!(rmap_val &KVM_RMAP_MANY )) { iter->desc =NULL ; - sptep = (u64 *)rmap_head->val; + sptep = (u64 *)rmap_val; goto out; } - iter->desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY ); + iter->desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY ); iter->pos =0 ; sptep = iter->desc->sptes[iter->pos]; out:
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 8/18 KVM: x86/mmu: Add infrastructure to allow walking rmaps outside of mmu_lock
In reply to James HoughtonFrom: Sean Christopherson <[email protected]>
Steal another bit from rmap entries (which are word aligned pointers, i.e. have 2 free bits on 32-bit KVM, and 3 free bits on 64-bit KVM), and use the bit to implement a *very* rudimentary per-rmap spinlock. The only anticipated usage of the lock outside of mmu_lock is for aging gfns, and collisions between aging and other MMU rmap operations are quite rare, e.g. unless userspace is being silly and aging a tiny range over and over in a tight loop, time between contention when aging an actively running VM is O(seconds). In short, a more sophisticated locking scheme shouldn't be necessary.
Note, the lock only protects the rmap structure itself, SPTEs that are pointed at by a locked rmap can still be modified and zapped by another task (KVM drops/zaps SPTEs before deleting the rmap entries)
--- arch/x86/include/asm/kvm_host.h | 3 +- arch/x86/kvm/mmu/mmu.c | 129 +++++++++++++++++++++++++++++--- 2 files changed, 120 insertions(+), 12 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -1
@@ -26,6 +26,7 @@#include <linux/irqbypass.h> #include <linux/hyperv.h> #include <linux/kfifo.h> +#include <linux/atomic/atomic-long.h> #include <asm/apic.h> #include <asm/pvclock-abi.h>
@@ -401,7 +402,7 @@ union kvm_cpu_role { };struct kvm_rmap_head { -unsigned long val; +atomic_long_t val; };struct kvm_pio_request {
arch/x86/kvm/mmu/mmu.c +118 -11
198 lines not on this page · raw message
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 9/18 KVM: x86/mmu: Add support for lockless walks of rmap SPTEs
In reply to James HoughtonFrom: Sean Christopherson <[email protected]>
Add a lockless version of for_each_rmap_spte(), which is pretty much the same as the normal version, except that it doesn't BUG() the host if a non-present SPTE is encountered. When mmu_lock is held, it should be impossible for a different task to zap a SPTE, _and_ zapped SPTEs must be removed from their rmap chain prior to dropping mmu_lock. Thus, the normal walker BUG()s if a non-present SPTE is encountered as something is wildly broken.
When walking rmaps without holding mmu_lock, the SPTEs pointed at by the rmap chain can be zapped/dropped, and so a lockless walk can observe a non-present SPTE if it runs concurrently with a different operation that is zapping SPTEs.
--- arch/x86/kvm/mmu/mmu.c | 75 +++++++++++++++++++++++------------------- 1 file changed, 42 insertions(+), 33 deletions(-)
arch/x86/kvm/mmu/mmu.c +42 -33
@@ -932,7 +932,7 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu*/ #define KVM_RMAP_LOCKED BIT(1) -static unsigned long kvm_rmap_lock (struct kvm_rmap_head *rmap_head) +static unsigned long __kvm_rmap_lock (struct kvm_rmap_head *rmap_head) {unsigned long old_val, new_val;
@@ -976,14 +976,25 @@ static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head)*/ }while (!atomic_long_try_cmpxchg_acquire (&rmap_head->val, &old_val, new_val)); -/* Return the old value, i.e. _without_ the LOCKED bit set. */ +/* +* Return the old value, i.e. _without_ the LOCKED bit set. It's +* impossible for the return value to be 0 (see above), i.e. the read- +* only unlock flow can't get a false positive and fail to unlock. +*/ return old_val; } +static unsigned long kvm_rmap_lock (struct kvm *kvm, +struct kvm_rmap_head *rmap_head) +{ +lockdep_assert_held_write (&kvm->mmu_lock); +return __kvm_rmap_lock (rmap_head); +} +static void kvm_rmap_unlock (struct kvm_rmap_head *rmap_head,unsigned long new_val) { -WARN_ON_ONCE (new_val &KVM_RMAP_LOCKED ); +KVM_MMU_WARN_ON (new_val &KVM_RMAP_LOCKED );/* * Ensure that all accesses to the rmap have completed * before we actually unlock the rmap.
@@ -1023,14 +1034,14 @@ static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head,/* * Returns the number of pointers in the rmap chain, not counting the new one. */ -static int pte_list_add (struct kvm_mmu_memory_cache *cache,u64 *spte, -struct kvm_rmap_head *rmap_head) +static int pte_list_add (struct kvm *kvm,struct kvm_mmu_memory_cache *cache, +u64 *spte,struct kvm_rmap_head *rmap_head) {unsigned long old_val, new_val;struct pte_list_desc *desc;int count =0 ; - old_val =kvm_rmap_lock (rmap_head); + old_val =kvm_rmap_lock (kvm, rmap_head);if (!old_val) { new_val = (unsigned long )spte;
@@ -1110,7 +1121,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte,unsigned long rmap_val;int i; - rmap_val =kvm_rmap_lock (rmap_head); + rmap_val =kvm_rmap_lock (kvm, rmap_head);if (KVM_BUG_ON_DATA_CORRUPTION (!rmap_val, kvm)) goto out;
@@ -1154,7 +1165,7 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm,unsigned long rmap_val;int i; - rmap_val =kvm_rmap_lock (rmap_head); + rmap_val =kvm_rmap_lock (kvm, rmap_head);if (!rmap_val)return false;
@@ -1246,23 +1257,18 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) {unsigned long rmap_val =kvm_rmap_get (rmap_head); -u64 *sptep;if (!rmap_val)return NULL ;if (!(rmap_val &KVM_RMAP_MANY )) { iter->desc =NULL ; - sptep = (u64 *)rmap_val; - goto out; +return (u64 *)rmap_val; } iter->desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY ); iter->pos =0 ; - sptep = iter->desc->sptes[iter->pos]; -out: -BUG_ON (!is_shadow_present_pte (*sptep)); -return sptep; +return iter->desc->sptes[iter->pos]; }/*
@@ -1272,14 +1278,11 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head,*/ static u64 *rmap_get_next (struct rmap_iterator *iter) { -u64 *sptep; -if (iter->desc) {if (iter->pos <PTE_LIST_EXT -1 ) { ++iter->pos; - sptep = iter->desc->sptes[iter->pos]; -if (sptep) - goto out; +if (iter->desc->sptes[iter->pos]) +return iter->desc->sptes[iter->pos]; } iter->desc = iter->desc->more;
@@ -1287,20 +1290,24 @@ static u64 *rmap_get_next(struct rmap_iterator *iter)if (iter->desc) { iter->pos =0 ;/* desc->sptes[0] cannot be NULL */ - sptep = iter->desc->sptes[iter->pos]; - goto out; +return iter->desc->sptes[iter->pos]; } }return NULL ; -out: -BUG_ON (!is_shadow_present_pte (*sptep)); -return sptep; } -#define for_each_rmap_spte (_rmap_head_, _iter_, _spte_) \ - for (_spte_ = rmap_get_first(_rmap_head_, _iter_); \ - _spte_; _spte_ = rmap_get_next(_iter_)) +#define __for_each_rmap_spte (_rmap_head_, _iter_, _sptep_) \ + for (_sptep_ = rmap_get_first(_rmap_head_, _iter_); \ + _sptep_; _sptep_ = rmap_get_next(_iter_)) + +#define for_each_rmap_spte (_rmap_head_, _iter_, _sptep_) \ + __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + if (!WARN_ON_ONCE(!is_shadow_present_pte(*(_sptep_)))) \ + +#define for_each_rmap_spte_lockless (_rmap_head_, _iter_, _sptep_, _spte_) \ + __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + if (is_shadow_present_pte(_spte_ = mmu_spte_get_lockless(sptep)))static void drop_spte (struct kvm *kvm,u64 *sptep) {
@@ -1396,11 +1403,12 @@ static bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head,struct rmap_iterator iter;bool flush = false; -for_each_rmap_spte (rmap_head, &iter, sptep) +for_each_rmap_spte (rmap_head, &iter, sptep) {if (spte_ad_need_write_protect (*sptep)) flush |=spte_wrprot_for_clear_dirty (sptep);else flush |=spte_clear_dirty (sptep); + }return flush; }
@@ -1710,7 +1718,7 @@ static void __rmap_add(struct kvm *kvm,kvm_update_page_stats (kvm, sp->role.level,1 ); rmap_head =gfn_to_rmap (gfn, sp->role.level, slot); - rmap_count =pte_list_add (cache, spte, rmap_head); + rmap_count =pte_list_add (kvm, cache, spte, rmap_head);if (rmap_count > kvm->stat.max_mmu_rmap_size) kvm->stat.max_mmu_rmap_size = rmap_count;
@@ -1859,13 +1867,14 @@ static unsigned kvm_page_table_hashfn(gfn_t gfn)return hash_64 (gfn,KVM_MMU_HASH_SHIFT ); } -static void mmu_page_add_parent_pte (struct kvm_mmu_memory_cache *cache, +static void mmu_page_add_parent_pte (struct kvm *kvm, +struct kvm_mmu_memory_cache *cache,struct kvm_mmu_page *sp,u64 *parent_pte) {if (!parent_pte)return ; -pte_list_add (cache, parent_pte, &sp->parent_ptes); +pte_list_add (kvm, cache, parent_pte, &sp->parent_ptes); } static void mmu_page_remove_parent_pte(struct kvm *kvm, struct kvm_mmu_page *sp,
@@ -2555,7 +2564,7 @@ static void __link_shadow_page(struct kvm *kvm,mmu_spte_set (sptep, spte); -mmu_page_add_parent_pte (cache, sp, sptep); +mmu_page_add_parent_pte (kvm, cache, sp, sptep);/* * The non-direct sub-pagetable must be updated before linking. For
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 10/18 KVM: x86/mmu: Support rmap walks without holding mmu_lock when aging gfns
In reply to James HoughtonFrom: Sean Christopherson <[email protected]>
Because an L1 KVM can disable A/D bits for its L2, even if kvm_ad_enabled() in L0, we cannot always locklessly age, as aging requires marking non-A/D sptes for access tracking, which is not supported locklessly yet.
We can always gather age information locklessly though.
--- arch/x86/kvm/mmu/mmu.c | 66 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 62 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu/mmu.c +62 -4
@@ -1014,13 +1014,11 @@ static unsigned long kvm_rmap_get(struct kvm_rmap_head *rmap_head)* locking is the same, but the caller is disallowed from modifying the rmap, * and so the unlock flow is a nop if the rmap is/was empty. */ -__maybe_unused static unsigned long kvm_rmap_lock_readonly (struct kvm_rmap_head *rmap_head) {return __kvm_rmap_lock (rmap_head); } -__maybe_unused static void kvm_rmap_unlock_readonly (struct kvm_rmap_head *rmap_head,unsigned long old_val) {
@@ -1736,8 +1734,53 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot,__rmap_add (vcpu->kvm, cache, slot, spte, gfn, access); } -static bool kvm_rmap_age_gfn_range (struct kvm *kvm, -struct kvm_gfn_range *range,bool test_only) +static bool kvm_rmap_age_gfn_range_lockless (struct kvm *kvm, +struct kvm_gfn_range *range, +bool test_only) +{ +struct kvm_rmap_head *rmap_head; +struct rmap_iterator iter; +unsigned long rmap_val; +bool young = false; +u64 *sptep; +gfn_t gfn; +int level; +u64 spte; + +for (level =PG_LEVEL_4K ; level <=KVM_MAX_HUGEPAGE_LEVEL ; level++) { +for (gfn = range->start; gfn < range->end; + gfn +=KVM_PAGES_PER_HPAGE (level)) { + rmap_head =gfn_to_rmap (gfn, level, range->slot); + rmap_val =kvm_rmap_lock_readonly (rmap_head); + +for_each_rmap_spte_lockless (rmap_head, &iter, sptep, spte) { +if (!is_accessed_spte (spte)) +continue ; + +if (test_only) { +kvm_rmap_unlock_readonly (rmap_head, rmap_val); +return true; + } + +/* +* Marking SPTEs for access tracking outside of +* mmu_lock is unsupported. Report the page as +* young, but otherwise leave it as-is. +*/ +if (spte_ad_enabled (spte)) +clear_bit ((ffs (shadow_accessed_mask) -1 ), + (unsigned long *)sptep); + young = true; + } + +kvm_rmap_unlock_readonly (rmap_head, rmap_val); + } + } +return young; +} + +static bool __kvm_rmap_age_gfn_range (struct kvm *kvm, +struct kvm_gfn_range *range,bool test_only) {struct slot_rmap_walk_iterator iterator;struct rmap_iterator iter;
@@ -1776,6 +1819,21 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm,return young; } +static bool kvm_rmap_age_gfn_range (struct kvm *kvm, +struct kvm_gfn_range *range,bool test_only) +{ +/* +* We can always locklessly test if an spte is young. Because marking +* non-A/D sptes for access tracking without holding the mmu_lock is +* not currently supported, we cannot always locklessly clear. +*/ +if (test_only) +return kvm_rmap_age_gfn_range_lockless (kvm, range, test_only); + +lockdep_assert_held_write (&kvm->mmu_lock); +return __kvm_rmap_age_gfn_range (kvm, range, test_only); +} +static bool kvm_has_shadow_mmu_sptes (struct kvm *kvm) {return !tdp_mmu_enabled ||READ_ONCE (kvm->arch.indirect_shadow_pages);
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 11/18 mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER
In reply to James HoughtonRemove the now unnecessary ifdef in mm/damon/vaddr.c as well.
--- include/linux/mmu_notifier.h | 7 +++++++ mm/damon/vaddr.c | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-)
include/linux/mmu_notifier.h +7 -0
@@ -606,6 +606,13 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm,return 0 ; } +static inline int mmu_notifier_clear_young (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +return 0 ; +} +static inline int mmu_notifier_test_young (struct mm_struct *mm,unsigned long address) {
mm/damon/vaddr.c +0 -2
@@ -351,11 +351,9 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,set_huge_pte_at (mm, addr, pte, entry, psize); } -#ifdef CONFIG_MMU_NOTIFIER if (mmu_notifier_clear_young (mm, addr, addr +huge_page_size (hstate_vma (vma)))) referenced = true; -#endif /* CONFIG_MMU_NOTIFIER */ if (referenced)folio_set_young (folio);
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 12/18 mm: Add has_fast_aging to struct mmu_notifier
In reply to James Houghtonhas_fast_aging should be set by subscribers that non-trivially implement fast-only versions of both test_young() and clear_young().
Fast aging must be opt-in. For a subscriber that has not been enlightened with "fast aging", the test/clear_young() will behave identically whether or not fast_only is given. Have KVM always opt out for now; specific architectures can opt-in later.
Given that KVM is the only test/clear_young() implementer, we could instead add an equivalent check in KVM, but doing so would incur an indirect function call every time, even if the notifier ends up being a no-op.
Add mm_has_fast_young_notifiers() in case a caller wants to know if it should skip many calls to the mmu notifiers that may not be necessary (like MGLRU look-around).
--- include/linux/mmu_notifier.h | 14 ++++++++++++++ mm/mmu_notifier.c | 20 ++++++++++++++++++++ virt/kvm/kvm_main.c | 1 + 3 files changed, 35 insertions(+)
include/linux/mmu_notifier.h +14 -0
@@ -231,6 +231,7 @@ struct mmu_notifier {struct mm_struct *mm;struct rcu_head rcu;unsigned int users; +bool has_fast_aging; };/**
@@ -383,6 +384,7 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end);extern int __mmu_notifier_test_young (struct mm_struct *mm,unsigned long address); +extern bool __mm_has_fast_young_notifiers (struct mm_struct *mm);extern int __mmu_notifier_invalidate_range_start (struct mmu_notifier_range *r);extern void __mmu_notifier_invalidate_range_end (struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm,
@@ -428,6 +430,13 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm,return 0 ; } +static inline bool mm_has_fast_young_notifiers (struct mm_struct *mm) +{ +if (mm_has_notifiers (mm)) +return __mm_has_fast_young_notifiers (mm); +return 0 ; +} +static inline void mmu_notifier_invalidate_range_start (struct mmu_notifier_range *range) {
@@ -619,6 +628,11 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm,return 0 ; } +static inline bool mm_has_fast_young_notifiers (struct mm_struct *mm) +{ +return 0 ; +} +static inline void mmu_notifier_invalidate_range_start (struct mmu_notifier_range *range) {
mm/mmu_notifier.c +20 -0
@@ -382,6 +382,26 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,return young; } +bool __mm_has_fast_young_notifiers (struct mm_struct *mm) +{ +struct mmu_notifier *subscription; +bool has_fast_aging = false; +int id; + + id =srcu_read_lock (&srcu); +hlist_for_each_entry_rcu (subscription, + &mm->notifier_subscriptions->list, hlist, +srcu_read_lock_held (&srcu)) { +if (subscription->has_fast_aging) { + has_fast_aging = true; +break ; + } + } +srcu_read_unlock (&srcu, id); + +return has_fast_aging; +} +int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start,unsigned long end)
virt/kvm/kvm_main.c +1 -0
@@ -930,6 +930,7 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = {static int kvm_init_mmu_notifier (struct kvm *kvm) { kvm->mmu_notifier.ops = &kvm_mmu_notifier_ops; + kvm->mmu_notifier.has_fast_aging = false;return mmu_notifier_register (&kvm->mmu_notifier, current->mm); }
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 13/18 mm: Add fast_only bool to test_young and clear_young MMU notifiers
In reply to James HoughtonFor implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless.
Also add mmu_notifier_test_young_fast_only() and mmu_notifier_clear_young_fast_only() helpers to set fast_only for these notifiers.
--- include/linux/mmu_notifier.h | 61 ++++++++++++++++++++++++++++++++---- include/trace/events/kvm.h | 19 ++++++----- mm/mmu_notifier.c | 18 ++++++++--- virt/kvm/kvm_main.c | 12 ++++--- 4 files changed, 88 insertions(+), 22 deletions(-)
include/linux/mmu_notifier.h +55 -6
@@ -106,21 +106,38 @@ struct mmu_notifier_ops {* clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. +* +* The fast_only parameter indicates that this call should not block, +* and this function should not cause other MMU notifier calls to +* block. Usually this means that the implementation should be +* lockless. +* +* When called with fast_only, this notifier will be a no-op (and +* return that the range is NOT young), unless has_fast_aging is set +* on the struct mmu_notifier. +* +* When fast_only is true, if the implementer cannot determine that a +* range is young without blocking, it should return 0 (i.e., that +* the range is NOT young). */ int (*clear_young)(struct mmu_notifier *subscription,struct mm_struct *mm,unsigned long start, -unsigned long end); +unsigned long end, +bool fast_only);/* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. +* +* The fast_only parameter has the same meaning as with clear_young. */ int (*test_young)(struct mmu_notifier *subscription,struct mm_struct *mm, -unsigned long address); +unsigned long address, +bool fast_only);/* * invalidate_range_start() and invalidate_range_end() must be
@@ -381,9 +398,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long end);extern int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end); +unsigned long end, +bool fast_only);extern int __mmu_notifier_test_young (struct mm_struct *mm, -unsigned long address); +unsigned long address, +bool fast_only);extern bool __mm_has_fast_young_notifiers (struct mm_struct *mm);extern int __mmu_notifier_invalidate_range_start (struct mmu_notifier_range *r);extern void __mmu_notifier_invalidate_range_end (struct mmu_notifier_range *r);
@@ -418,7 +437,16 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) {if (mm_has_notifiers (mm)) -return __mmu_notifier_clear_young (mm, start, end); +return __mmu_notifier_clear_young (mm, start, end, false); +return 0 ; +} + +static inline int mmu_notifier_clear_young_fast_only (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +if (mm_has_notifiers (mm)) +return __mmu_notifier_clear_young (mm, start, end, true);return 0 ; }
@@ -426,7 +454,15 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) {if (mm_has_notifiers (mm)) -return __mmu_notifier_test_young (mm, address); +return __mmu_notifier_test_young (mm, address, false); +return 0 ; +} + +static inline int mmu_notifier_test_young_fast_only (struct mm_struct *mm, +unsigned long address) +{ +if (mm_has_notifiers (mm)) +return __mmu_notifier_test_young (mm, address, true);return 0 ; }
@@ -622,12 +658,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm,return 0 ; } +static inline int mmu_notifier_clear_young_fast_only (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +return 0 ; +} +static inline int mmu_notifier_test_young (struct mm_struct *mm,unsigned long address) {return 0 ; } +static inline int mmu_notifier_test_young_fast_only (struct mm_struct *mm, +unsigned long address) +{ +return 0 ; +} +static inline bool mm_has_fast_young_notifiers (struct mm_struct *mm) {return 0 ;
include/trace/events/kvm.h +12 -7
@@ -457,36 +457,41 @@ TRACE_EVENT(kvm_unmap_hva_range, );TRACE_EVENT (kvm_age_hva, - TP_PROTO(unsigned long start, unsigned long end), -TP_ARGS (start, end), + TP_PROTO(unsigned long start, unsigned long end, bool fast_only), +TP_ARGS (start, end, fast_only),TP_STRUCT__entry ( __field( unsigned long, start ) __field( unsigned long, end ) +__field ( bool, fast_only ) ),TP_fast_assign ( __entry->start = start; __entry->end = end; + __entry->fast_only = fast_only; ), -TP_printk ("mmu notifier age hva: %#016lx -- %#016lx" , - __entry->start, __entry->end) +TP_printk ("mmu notifier age hva: %#016lx -- %#016lx fast_only: %d" , + __entry->start, __entry->end, __entry->fast_only) );TRACE_EVENT (kvm_test_age_hva, - TP_PROTO(unsigned long hva), -TP_ARGS (hva), + TP_PROTO(unsigned long hva, bool fast_only), +TP_ARGS (hva, fast_only),TP_STRUCT__entry ( __field( unsigned long, hva ) +__field ( bool, fast_only ) ),TP_fast_assign ( __entry->hva = hva; + __entry->fast_only = fast_only; ), -TP_printk ("mmu notifier test age hva: %#016lx" , __entry->hva) +TP_printk ("mmu notifier test age hva: %#016lx fast_only: %d" , + __entry->hva, __entry->fast_only) );#endif /* _TRACE_KVM_MAIN_H */
mm/mmu_notifier.c +14 -4
@@ -404,7 +404,8 @@ bool __mm_has_fast_young_notifiers(struct mm_struct *mm)int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;
@@ -413,9 +414,13 @@ int __mmu_notifier_clear_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { +if (fast_only && !subscription->has_fast_aging) +continue ; +if (subscription->ops->clear_young) young |= subscription->ops->clear_young (subscription, - mm, start, end); + mm, start, end, + fast_only); }srcu_read_unlock (&srcu, id);
@@ -423,7 +428,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, }int __mmu_notifier_test_young (struct mm_struct *mm, -unsigned long address) +unsigned long address, +bool fast_only) {struct mmu_notifier *subscription;int young =0 , id;
@@ -432,9 +438,13 @@ int __mmu_notifier_test_young(struct mm_struct *mm,hlist_for_each_entry_rcu (subscription, &mm->notifier_subscriptions->list, hlist,srcu_read_lock_held (&srcu)) { +if (fast_only && !subscription->has_fast_aging) +continue ; +if (subscription->ops->test_young) { young = subscription->ops->test_young (subscription, mm, - address); + address, + fast_only);if (young)break ; }
virt/kvm/kvm_main.c +7 -5
@@ -846,7 +846,7 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), }; -trace_kvm_age_hva (start, end); +trace_kvm_age_hva (start, end, false);return kvm_handle_hva_range (kvm, &range).ret; }
@@ -854,7 +854,8 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,static int kvm_mmu_notifier_clear_young (struct mmu_notifier *mn,struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +bool fast_only) {struct kvm *kvm =mmu_notifier_to_kvm (mn);const struct kvm_mmu_notifier_range range = {
@@ -868,7 +869,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), }; -trace_kvm_age_hva (start, end); +trace_kvm_age_hva (start, end, fast_only);/* * Even though we do not flush TLB, this will still adversely
@@ -888,7 +889,8 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn,static int kvm_mmu_notifier_test_young (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long address) +unsigned long address, +bool fast_only) {struct kvm *kvm =mmu_notifier_to_kvm (mn);const struct kvm_mmu_notifier_range range = {
@@ -902,7 +904,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn,IS_ENABLED (CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS ), }; -trace_kvm_test_age_hva (address); +trace_kvm_test_age_hva (address, fast_only);return kvm_handle_hva_range (kvm, &range).ret; }
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 14/18 KVM: Pass fast_only to kvm_{test_,}age_gfn
In reply to James HoughtonProvide the basics for architectures to implement a fast-only version of kvm_{test_,}age_gfn.
--- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 2 ++ 2 files changed, 3 insertions(+)
include/linux/kvm_host.h +1 -0
@@ -258,6 +258,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg {unsigned long attributes; +bool fast_only; }; struct kvm_gfn_range {
virt/kvm/kvm_main.c +2 -0
@@ -867,6 +867,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, .may_block = false, .lockless = IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), + .arg.fast_only = fast_only, };trace_kvm_age_hva (start, end, fast_only);
@@ -902,6 +903,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, .may_block = false, .lockless = IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), + .arg.fast_only = fast_only, };trace_kvm_test_age_hva (address, fast_only);
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 15/18 KVM: x86/mmu: Locklessly harvest access information from shadow MMU
In reply to James HoughtonMove where the lock is taken for the shadow MMU case to only take the lock when !range->arg.fast_only (i.e., for the non-fast_only aging MMU notifiers).
--- arch/x86/kvm/mmu/mmu.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)
arch/x86/kvm/mmu/mmu.c +16 -11
@@ -1822,16 +1822,24 @@ static bool __kvm_rmap_age_gfn_range(struct kvm *kvm,static bool kvm_rmap_age_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range,bool test_only) { +bool young; +/* * We can always locklessly test if an spte is young. Because marking * non-A/D sptes for access tracking without holding the mmu_lock is * not currently supported, we cannot always locklessly clear. +* +* For fast_only, we must not take the mmu_lock, so locklessly age in +* that case even though we will not be able to clear the age for +* non-A/D sptes. */ -if (test_only) +if (test_only || range->arg.fast_only)return kvm_rmap_age_gfn_range_lockless (kvm, range, test_only); -lockdep_assert_held_write (&kvm->mmu_lock); -return __kvm_rmap_age_gfn_range (kvm, range, test_only); +write_lock (&kvm->mmu_lock); + young =__kvm_rmap_age_gfn_range (kvm, range, test_only); +write_unlock (&kvm->mmu_lock); +return young; }static bool kvm_has_shadow_mmu_sptes (struct kvm *kvm)
@@ -1846,11 +1854,8 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)if (tdp_mmu_enabled) young =kvm_tdp_mmu_age_gfn_range (kvm, range); -if (kvm_has_shadow_mmu_sptes (kvm)) { -write_lock (&kvm->mmu_lock); +if (kvm_has_shadow_mmu_sptes (kvm)) young |=kvm_rmap_age_gfn_range (kvm, range, false); -write_unlock (&kvm->mmu_lock); - }return young; }
@@ -1862,11 +1867,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)if (tdp_mmu_enabled) young =kvm_tdp_mmu_test_age_gfn (kvm, range); -if (!young &&kvm_has_shadow_mmu_sptes (kvm)) { -write_lock (&kvm->mmu_lock); +if (young) +return young; + +if (kvm_has_shadow_mmu_sptes (kvm)) young |=kvm_rmap_age_gfn_range (kvm, range, true); -write_unlock (&kvm->mmu_lock); - }return young; }
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 16/18 KVM: x86/mmu: Enable has_fast_aging
In reply to James HoughtonBecause the x86 MMU locklessly implements fast_only versions of kvm_test_age_gfn and kvm_age_gfn, we can advertise support for has_fast_aging to allow MGLRU to quickly find better eviction candidates.
There is one case where the MMU is not 100% accurate: for the shadow MMU, when A/D bits are not in use, young sptes will never be aged with the fast_only kvm_age_gfn. In this case, such pages will consistently appear young, so they will be the least likely eviction candidates.
--- arch/x86/kvm/mmu/mmu.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kvm/mmu/mmu.c +2 -0
@@ -7708,6 +7708,8 @@ int kvm_mmu_post_init_vm(struct kvm *kvm) {int err; + kvm->mmu_notifier.has_fast_aging = true; +if (nx_hugepage_mitigation_hard_disabled)return 0 ;
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 17/18 mm: multi-gen LRU: Have secondary MMUs participate in aging
In reply to James HoughtonSecondary MMUs are currently consulted for access/age information at eviction time, but before then, we don't get accurate age information. That is, pages that are mostly accessed through a secondary MMU (like guest memory, used by KVM) will always just proceed down to the oldest generation, and then at eviction time, if KVM reports the page to be young, the page will be activated/promoted back to the youngest generation.
The added feature bit (0x8), if disabled, will make MGLRU behave as if there are no secondary MMUs subscribed to MMU notifiers except at eviction time.
Implement aging with the new mmu_notifier_clear_young_fast_only() notifier. For architectures that do not support this notifier, this becomes a no-op. For architectures that do implement it, it should be fast enough to make aging worth it (usually the case if the notifier is implemented locklessly).
--- Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 148 ++++++++++++++---- 4 files changed, 127 insertions(+), 42 deletions(-)
Documentation/admin-guide/mm/multigen_lru.rst +5 -1
@@ -48,6 +48,10 @@ Values Components verified on x86 varieties other than Intel and AMD. If it is disabled, the multi-gen LRU will suffer a negligible performance degradation. +0x0008 Clear the accessed bit in secondary MMU page tables when aging + instead of waiting until eviction time. This results in accurate + page age information for pages that are mainly used by a + secondary MMU. [yYnN] Apply to all the components above. ====== ===============================================================
@@ -56,7 +60,7 @@ E.g., echo y >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled - 0x0007 + 0x000f echo 5 >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0005
include/linux/mmzone.h +4 -2
@@ -400,6 +400,7 @@ enum {LRU_GEN_CORE ,LRU_GEN_MM_WALK ,LRU_GEN_NONLEAF_YOUNG , +LRU_GEN_SECONDARY_MMU_WALK , NR_LRU_GEN_CAPS };
@@ -557,7 +558,7 @@ struct lru_gen_memcg {void lru_gen_init_pgdat (struct pglist_data *pgdat);void lru_gen_init_lruvec (struct lruvec *lruvec); -void lru_gen_look_around (struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around (struct page_vma_mapped_walk *pvmw);void lru_gen_init_memcg (struct mem_cgroup *memcg);void lru_gen_exit_memcg (struct mem_cgroup *memcg);
@@ -576,8 +577,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } -static inline void lru_gen_look_around (struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around (struct page_vma_mapped_walk *pvmw) { +return false; }static inline void lru_gen_init_memcg (struct mem_cgroup *memcg)
mm/rmap.c +3 -6
@@ -870,13 +870,10 @@ static bool folio_referenced_one(struct folio *folio,continue ; } -if (pvmw.pte) { -if (lru_gen_enabled () && -pte_young (ptep_get (pvmw.pte))) { -lru_gen_look_around (&pvmw); +if (lru_gen_enabled () && pvmw.pte) { +if (lru_gen_look_around (&pvmw)) referenced++; - } - + }else if (pvmw.pte) {if (ptep_clear_flush_young_notify (vma, address, pvmw.pte)) referenced++;
mm/vmscan.c +115 -33
314 lines not on this page · raw message
Signature · 1 line
2.46.0.792.g87dc391469-goog
PATCH v7 18/18 KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test
In reply to James HoughtonThis test now has two modes of operation:
1. (default) To check how much vCPU performance was affected by access tracking (previously existed, now supports MGLRU aging).
2. (-p) To also benchmark how fast MGLRU can do aging while vCPUs are faulting in memory.
Mode (1) also serves as a way to verify that aging is working properly for pages only accessed by KVM. It will fail if one does not have the 0x8 lru_gen feature bit.
To support MGLRU, the test creates a memory cgroup, moves itself into it, then uses the lru_gen debugfs output to track memory in that cgroup. The logic to parse the lru_gen debugfs output has been put into selftests/kvm/lib/lru_gen_util.c.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 369 +++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ 4 files changed, 786 insertions(+), 30 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -22,6 +22,7 @@ LIBKVM += lib/elf.c LIBKVM += lib/guest_modes.c LIBKVM += lib/io.c LIBKVM += lib/kvm_util.c +LIBKVM += lib/lru_gen_util.c LIBKVM += lib/memstress.c LIBKVM += lib/guest_sprintf.c LIBKVM += lib/rbtree.c
tools/testing/selftests/kvm/access_tracking_perf_test.c +339 -30
488 lines not on this page · raw message
tools/testing/selftests/kvm/include/lru_gen_util.h new file +55 -0
55 lines not on this page · raw message
tools/testing/selftests/kvm/lib/lru_gen_util.c new file +391 -0
391 lines not on this page · raw message
Signature · 1 line
2.46.0.792.g87dc391469-goog
Sean Christopherson
maintainerQuoted 4 lines · James Houghton
This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim.
...
Quoted 11 lines
James Houghton (14):
KVM: Removekvm_handle_hva_rangehelper functions
KVM: Add lockless memslot walk to KVM
KVM: x86/mmu: Factor out spte atomic bit clearing routine
KVM: x86/mmu: Relax locking forkvm_test_age_gfnandkvm_age_gfn
KVM: x86/mmu: Rearrange kvm_{test_,}age_gfn
KVM: x86/mmu: Only check gfn age in shadow MMU if
indirect_shadow_pages> 0
mm: Add missingmmu_notifier_clear_youngfor !MMU_NOTIFIER
mm: Addhas_fast_agingto structmmu_notifier
mm: Addfast_onlybool totest_youngandclear_youngMMU notifiers
Per offline discussions, there's a non-zero chance that fast_only won't be needed, because it may be preferable to incorporate secondary MMUs into MGLRU, even if they don't support "fast" aging.
What's the status on that front? Even if the status is "TBD", it'd be very helpful to let others know, so that they don't spend time reviewing code that might be completely thrown away.
Quoted 14 lines
KVM: Pass
fast_onlyto kvm_{test_,}age_gfn
KVM: x86/mmu: Locklessly harvest access information from shadow MMU
KVM: x86/mmu: Enablehas_fast_aging
mm: multi-gen LRU: Have secondary MMUs participate in aging
KVM: selftests: Add multi-gen LRU aging toaccess_tracking_perf_testSean Christopherson (4):
KVM: x86/mmu: Refactor low level rmap helpers to prep for walking w/o
mmu_lock
KVM: x86/mmu: Add infrastructure to allow walking rmaps outside of
mmu_lock
KVM: x86/mmu: Add support for lockless walks of rmap SPTEs
KVM: x86/mmu: Support rmap walks without holdingmmu_lockwhen aging
gfns
Quoted 21 lines · Sean Christopherson
Quoted 4 lines · James Houghton
This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim.
...
Quoted 11 lines
James Houghton (14):
KVM: Removekvm_handle_hva_rangehelper functions
KVM: Add lockless memslot walk to KVM
KVM: x86/mmu: Factor out spte atomic bit clearing routine
KVM: x86/mmu: Relax locking forkvm_test_age_gfnandkvm_age_gfn
KVM: x86/mmu: Rearrange kvm_{test_,}age_gfn
KVM: x86/mmu: Only check gfn age in shadow MMU if
indirect_shadow_pages> 0
mm: Add missingmmu_notifier_clear_youngfor !MMU_NOTIFIER
mm: Addhas_fast_agingto structmmu_notifier
mm: Addfast_onlybool totest_youngandclear_youngMMU notifiers
Quoted 7 lines
Per offline discussions, there's a non-zero chance that
fast_onlywon't be needed, because it may be preferable to incorporate secondary MMUs into MGLRU, even if they don't support "fast" aging.What's the status on that front? Even if the status is "TBD", it'd be very helpful to let others know, so that they don't spend time reviewing code that might be completely thrown away.
The fast_only MMU notifier changes will probably be removed in v8.
ChromeOS folks found that the way MGLRU *currently* interacts with KVM is problematic. That is, today, with the MM_WALK MGLRU capability enabled, normal PTEs have their Accessed bits cleared via a page table scan and then during an rmap walk upon attempted eviction, whereas, KVM SPTEs only have their Accessed bits cleared via the rmap walk at eviction time. So KVM SPTEs have their Accessed bits cleared less frequently than normal PTEs, and therefore they appear younger than they should.
It turns out that this causes tab open latency regressions on ChromeOS where a significant amount of memory is being used by a VM. IIUC, the fix for this is to have MGLRU age SPTEs as often as it ages normal PTEs; i.e., it should call the correct MMU notifiers each time it clears A bits on PTEs. The final patch in this series sort of does this, but instead of calling the new fast_only notifier, we need to call the normal test/clear_young() notifiers regardless of how fast they are.
This also means that the MGLRU changes no longer depend on the KVM optimizations, as they can motivated independently.
Yu, have I gotten anything wrong here? Do you have any more details to share?
Quoted 51 lines · James Houghton
Quoted 21 lines · Sean Christopherson
Quoted 4 lines · James Houghton
This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim.
...
Quoted 11 lines
James Houghton (14):
KVM: Removekvm_handle_hva_rangehelper functions
KVM: Add lockless memslot walk to KVM
KVM: x86/mmu: Factor out spte atomic bit clearing routine
KVM: x86/mmu: Relax locking forkvm_test_age_gfnandkvm_age_gfn
KVM: x86/mmu: Rearrange kvm_{test_,}age_gfn
KVM: x86/mmu: Only check gfn age in shadow MMU if
indirect_shadow_pages> 0
mm: Add missingmmu_notifier_clear_youngfor !MMU_NOTIFIER
mm: Addhas_fast_agingto structmmu_notifier
mm: Addfast_onlybool totest_youngandclear_youngMMU notifiersQuoted 7 lines
Per offline discussions, there's a non-zero chance that
fast_onlywon't be needed, because it may be preferable to incorporate secondary MMUs into MGLRU, even if they don't support "fast" aging.What's the status on that front? Even if the status is "TBD", it'd be very helpful to let others know, so that they don't spend time reviewing code that might be completely thrown away.
The
fast_onlyMMU notifier changes will probably be removed in v8.ChromeOS folks found that the way MGLRU *currently* interacts with KVM is problematic. That is, today, with the
MM_WALKMGLRU capability enabled, normal PTEs have their Accessed bits cleared via a page table scan and then during an rmap walk upon attempted eviction, whereas, KVM SPTEs only have their Accessed bits cleared via the rmap walk at eviction time. So KVM SPTEs have their Accessed bits cleared less frequently than normal PTEs, and therefore they appear younger than they should.It turns out that this causes tab open latency regressions on ChromeOS where a significant amount of memory is being used by a VM. IIUC, the fix for this is to have MGLRU age SPTEs as often as it ages normal PTEs; i.e., it should call the correct MMU notifiers each time it clears A bits on PTEs. The final patch in this series sort of does this, but instead of calling the new
fast_onlynotifier, we need to call the normal test/clear_young() notifiers regardless of how fast they are.
Quoted 4 lines
This also means that the MGLRU changes no longer depend on the KVM optimizations, as they can motivated independently.
Yu, have I gotten anything wrong here? Do you have any more details to share?
Yes, that's precisely the problem. My original justification [1] for not scanning KVM MMU when lockless is not supported turned out to be harmful to some workloads too.
On one hand, scanning KVM MMU when not lockless can cause the KVM MMU lock contention; on the other hand, not scanning KVM MMU can skew anon/file LRU aging and thrash page cache. Given the lock contention is being tackled, the latter seems to be the lesser of two evils.
[1] https://lore.kernel.org/linux-mm/CAOUHufYFHKLwt1PWp2uS6g174GZYRZURWJAmdUWs5eaKmhEeyQ@mail.gmail.com/