KVM: x86/mmu: Age sptes locklessly
Status from this message: Applied to kvm-x86 mmu, thanks!
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v9 Latest11 patches 15 repliesApplied
- v811 patches 28 repliesSuperseded
- v718 patches 5 repliesSuperseded
Earlier versions · 3
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v6 | 11 patches | 32 replies | Superseded | |
| v5 | 9 patches | 33 replies | Superseded | |
| v4 | 7 patches | 27 replies | Superseded |
Patches in v9 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/11 | KVM: Rename kvm_handle_hva_range()virt/ | +16 | 0 replies | |
| 2/11 | KVM: Add lockless memslot walk to KVMinclude/ virt/ virt/ | +20 | 2 replies | |
| 3/11 | KVM: x86/mmu: Factor out spte atomic bit clearing routinearch/ | +9 | 0 replies | |
| 4/11 | KVM: x86/mmu: Relax locking for kvm_test_age_gfn() and kvm_age_gfn()arch/ arch/ arch/ +3 more | +48 | 2 replies | |
| 5/11 | KVM: x86/mmu: Rename spte_has_volatile_bits() to spte_needs_atomic_write()Documentation/ arch/ arch/ +2 more | +11 | 2 replies | |
| 6/11 | KVM: x86/mmu: Skip shadow MMU test_young if TDP MMU reports page as youngarch/ | +9 | 0 replies | |
| 7/11 | KVM: x86/mmu: Only check gfn age in shadow MMU if indirect_shadow_pages > 0arch/ | +7 | 0 replies | |
| 8/11 | KVM: x86/mmu: Refactor low level rmap helpers to prep for walking w/o mmu_lockarch/ | +50 | 0 replies | |
| 9/11 | KVM: x86/mmu: Add infrastructure to allow walking rmaps outside of mmu_lockarch/ arch/ | +98 | 0 replies | |
| 10/11 | KVM: x86/mmu: Add support for lockless walks of rmap SPTEsarch/ | +94 | 0 replies | |
| 11/11 | KVM: x86/mmu: Support rmap walks without holding mmu_lock when aging gfnsarch/ | +39 | 0 replies |
Reviews trailers per patch
5 of 11 patches have a Reviewed-by or Acked-by.
| Patch | David Matlack | Yu Zhao | James Houghton |
|---|---|---|---|
| 1/11 | |||
| 2/11 | R-b | ||
| 3/11 | A-b | ||
| 4/11 | R-b | R-b | |
| 5/11 | |||
| 6/11 | A-b | ||
| 7/11 | A-b | ||
| 8/11 | |||
| 9/11 | |||
| 10/11 | |||
| 11/11 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.
Cover letter [PATCH v9 00/11] KVM: x86/mmu: Age sptes locklessly
By aging sptes locklessly with the TDP MMU and the shadow MMU, neither vCPUs nor reclaim (mmu_notifier_invalidate_range*) will get stuck waiting for aging. This contention reduction improves guest performance and saves a significant amount of Google Cloud's CPU usage, and it has valuable improvements for ChromeOS, as Yu has mentioned previously[1].
Please see v8[8] for some performance results using access_tracking_perf_test patched to use MGLRU.
Neither access_tracking_perf_test nor mmu_stress_test trigger any splats (with CONFIG_LOCKDEP=y) with the TDP MMU and with the shadow MMU.
=== Previous Versions ===
Since v8[8]:
- Re-added the kvm_handle_hva_range helpers and applied Sean's kvm_{handle -> age}_hva_range rename.
- Renamed spte_has_volatile_bits() to spte_needs_atomic_write() and removed its Accessed bit check. Undid change to tdp_mmu_spte_need_atomic_write().
- Renamed KVM_MMU_NOTIFIER_{YOUNG -> AGING}_LOCKLESS.
- cpu_relax(), lockdep, preempt_disable(), and locking fixups for per-rmap lock (thanks Lai and Sean).
- Renamed kvm_{has -> may_have}_shadow_mmu_sptes().
- Rebased onto latest kvm/next, including changing for_each_tdp_mmu_root_rcu to use types.
- Dropped MGLRU changes from access_tracking_perf_test.
- Picked up Acked-bys from Yu. (thank you!)
Since v7[7]:
- Dropped MGLRU changes.
- Dropped DAMON cleanup.
- Dropped MMU notifier changes completely.
- Made shadow MMU aging *always* lockless, not just lockless when the now-removed "fast_only" clear notifier was used.
- Given that the MGLRU changes no longer introduce a new MGLRU capability, drop the new capability check from the selftest.
- Rebased on top of latest kvm-x86/next, including the x86 mmu changes for marking pages as dirty.
Since v6[6]:
- Rebased on top of kvm-x86/next and Sean's lockless rmap walking changes.
- Removed HAVE_KVM_MMU_NOTIFIER_YOUNG_FAST_ONLY (thanks DavidM).
- Split up kvm_age_gfn() / kvm_test_age_gfn() optimizations (thanks DavidM and Sean).
- Improved new MMU notifier documentation (thanks DavidH).
- Dropped arm64 locking change.
- No longer retry for CAS failure in TDP MMU non-A/D case (thanks Sean).
- Added some R-bys and A-bys.
Since v5[5]:
- Reworked test_clear_young_fast_only() into a new parameter for the existing notifiers (thanks Sean).
- Added mmu_notifier.has_fast_aging to tell mm if calling fast-only notifiers should be done.
- Added mm_has_fast_young_notifiers() to inform users if calling fast-only notifier helpers is worthwhile (for look-around to use).
- Changed MGLRU to invoke a single notifier instead of two when aging and doing look-around (thanks Yu).
- For KVM/x86, check indirect_shadow_pages > 0 instead of kvm_memslots_have_rmaps() when collecting age information (thanks Sean).
- For KVM/arm, some fixes from Oliver.
- Small fixes to access_tracking_perf_test.
- Added missing !MMU_NOTIFIER version of mmu_notifier_clear_young().
Since v4[4]:
- Removed Kconfig that controlled when aging was enabled. Aging will be done whenever the architecture supports it (thanks Yu).
- Added a new MMU notifier, test_clear_young_fast_only(), specifically for MGLRU to use.
- Add kvm_fast_{test_,}age_gfn, implemented by x86.
- Fix locking for clear_flush_young().
- Added KVM_MMU_NOTIFIER_YOUNG_LOCKLESS to clean up locking changes (thanks Sean).
- Fix WARN_ON and other cleanup for the arm64 locking changes (thanks Oliver).
Since v3[3]:
- Vastly simplified the series (thanks David). Removed mmu notifier batching logic entirely.
- Cleaned up how locking is done for mmu_notifier_test/clear_young (thanks David).
- Look-around is now only done when there are no secondary MMUs subscribed to MMU notifiers.
- CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added.
- Fixed the lockless implementation of kvm_{test,}age_gfn for x86 (thanks David).
- Added MGLRU functional and performance tests to access_tracking_perf_test (thanks Axel).
- In v3, an mm would be completely ignored (for aging) if there was a secondary MMU but support for secondary MMU walking was missing. Now, missing secondary MMU walking support simply skips the notifier calls (except for eviction).
- Added a sanity check for that range->lockless and range->on_lock are never both provided for the memslot walk.
For the changes since v2[2], see v3.
Based on latest kvm/next.
[1]: https://lore.kernel.org/kvm/CAOUHufYS0XyLEf_V+q5SCW54Zy2aW5nL8CnSWreM8d1rX5NKYg@mail.gmail.com/
[2]: https://lore.kernel.org/kvmarm/[email protected]/
[3]: https://lore.kernel.org/linux-mm/[email protected]/
[4]: https://lore.kernel.org/linux-mm/[email protected]/
[5]: https://lore.kernel.org/linux-mm/[email protected]/
[6]: https://lore.kernel.org/linux-mm/[email protected]/
[7]: https://lore.kernel.org/kvm/[email protected]/
[8]: https://lore.kernel.org/kvm/[email protected]/
James Houghton (7):
KVM: Rename kvm_handle_hva_range()
KVM: Add lockless memslot walk to KVM
KVM: x86/mmu: Factor out spte atomic bit clearing routine
KVM: x86/mmu: Relax locking for kvm_test_age_gfn() and kvm_age_gfn()
KVM: x86/mmu: Rename spte_has_volatile_bits() to
spte_needs_atomic_write()
KVM: x86/mmu: Skip shadow MMU test_young if TDP MMU reports page as
young
KVM: x86/mmu: Only check gfn age in shadow MMU if
indirect_shadow_pages > 0
Sean Christopherson (4):
KVM: x86/mmu: Refactor low level rmap helpers to prep for walking w/o
mmu_lock
KVM: x86/mmu: Add infrastructure to allow walking rmaps outside of
mmu_lock
KVM: x86/mmu: Add support for lockless walks of rmap SPTEs
KVM: x86/mmu: Support rmap walks without holding mmu_lock when aging
gfns
Documentation/virt/kvm/locking.rst | 4 +- arch/x86/include/asm/kvm_host.h | 4 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 364 +++++++++++++++++++++-------- arch/x86/kvm/mmu/spte.c | 19 +- arch/x86/kvm/mmu/spte.h | 2 +- arch/x86/kvm/mmu/tdp_iter.h | 26 ++- arch/x86/kvm/mmu/tdp_mmu.c | 36 ++- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 2 + virt/kvm/kvm_main.c | 56 +++-- 11 files changed, 364 insertions(+), 151 deletions(-)
base-commit: f7bafceba76e9ab475b413578c1757ee18c3e44b
Signature · 1 line
2.48.1.362.g079036d154-goog