Skip to content

Allow parallel MMU operations with TDP MMU

Ben Gardon kvm lkml oe-kbuild-all 2 versions – Applied

Status from this message: Queued for 5.11-rc, thanks.

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest28 patches 36 repliesApplied
  2. v124 patches 45 repliesApplied

Changes in v2 from the cover letter

- Removed the MMU lock union + using a spinlock when the TDP MMU is disabled
- Merged RCU commits
- Extended additional MMU operations to operate in parallel
- Ammended dirty log perf test to cover newly parallelized code paths
- Misc refactorings (see changelogs for individual commits)
- Big thanks to Sean and Paolo for their thorough review of v1

Patches in v2 Applied

#Subject and filesSizeRepliesApplied
1/28KVM: x86/mmu: change TDP MMU yield function returns to match cond_reschedarch/x86/kvm/mmu/tdp_mmu.c+29 -100 replies
2/28KVM: x86/mmu: Add comment on __tdp_mmu_set_sptearch/x86/kvm/mmu/tdp_mmu.c+16 -00 replies
3/28KVM: x86/mmu: Add lockdep when setting a TDP MMU SPTEarch/x86/kvm/mmu/tdp_mmu.c+2 -00 replies
4/28KVM: x86/mmu: Don't redundantly clear TDP MMU pt memoryarch/x86/kvm/mmu/tdp_mmu.c+0 -10 replies
5/28KVM: x86/mmu: Factor out handling of removed page tablesarch/x86/kvm/mmu/tdp_mmu.c+42 -290 replies
6/28locking/rwlocks: Add contention detection for rwlocksinclude/asm-generic/qrwlock.h include/linux/rwlock.h+25 -69 replies
7/28sched: Add needbreak for rwlocksinclude/linux/sched.h+17 -00 replies
8/28sched: Add cond_resched_rwlockinclude/linux/sched.h kernel/sched/core.c+52 -00 replies
9/28KVM: x86/mmu: Fix braces in kvm_recover_nx_lpagesarch/x86/kvm/mmu/mmu.c+2 -20 replies
10/28KVM: x86/mmu: Fix TDP MMU zap collapsible SPTEsarch/x86/kvm/mmu/tdp_mmu.c+3 -31 reply
11/28KVM: x86/mmu: Merge flush and non-flush tdp_mmu_iter_cond_reschedarch/x86/kvm/mmu/tdp_mmu.c+13 -290 replies
12/28KVM: x86/mmu: Rename goal_gfn to next_last_level_gfnarch/x86/kvm/mmu/tdp_iter.c arch/x86/kvm/mmu/tdp_iter.h+12 -120 replies
13/28KVM: x86/mmu: Ensure forward progress when yielding in TDP MMU iterarch/x86/kvm/mmu/tdp_iter.c arch/x86/kvm/mmu/tdp_iter.h arch/x86/kvm/mmu/tdp_mmu.c+23 -231 reply
14/28KVM: x86/mmu: Yield in TDU MMU iter even if no SPTES changedarch/x86/kvm/mmu/tdp_mmu.c+22 -100 replies
15/28KVM: x86/mmu: Skip no-op changes in TDP MMU functionsarch/x86/kvm/mmu/tdp_mmu.c+4 -20 replies
16/28KVM: x86/mmu: Clear dirtied pages mask bit before early breakarch/x86/kvm/mmu/tdp_mmu.c+2 -20 replies
17/28KVM: x86/mmu: Protect TDP MMU page table memory with RCUarch/x86/kvm/mmu/mmu_internal.h arch/x86/kvm/mmu/tdp_iter.c arch/x86/kvm/mmu/tdp_iter.h +1 more+103 -210 replies
18/28KVM: x86/mmu: Use an rwlock for the x86 MMUarch/x86/include/asm/kvm_host.h arch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/page_track.c +6 more+112 -811 reply
19/28KVM: x86/mmu: Factor out functions to add/remove TDP MMU pagesarch/x86/kvm/mmu/tdp_mmu.c+39 -80 replies
20/28KVM: x86/mmu: Use atomic ops to set SPTEs in TDP MMU maparch/x86/include/asm/kvm_host.h arch/x86/kvm/mmu/tdp_mmu.c+122 -337 replies
21/28KVM: x86/mmu: Flush TLBs after zap in TDP MMU PF handlerarch/x86/kvm/mmu/spte.h arch/x86/kvm/mmu/tdp_mmu.c+74 -101 reply
22/28KVM: x86/mmu: Mark SPTEs in disconnected pages as removedarch/x86/kvm/mmu/tdp_mmu.c+30 -61 reply
23/28KVM: x86/mmu: Allow parallel page faults for the TDP MMUarch/x86/kvm/mmu/mmu.c+10 -24 replies
24/28KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lockarch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/mmu_internal.h arch/x86/kvm/mmu/tdp_mmu.c +1 more+145 -424 replies
25/28KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lockarch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_mmu.c+17 -102 replies
26/28KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read lockarch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_mmu.c arch/x86/kvm/mmu/tdp_mmu.h+84 -251 reply
27/28KVM: selftests: Add backing src parameter to dirty_log_perf_testtools/testing/selftests/kvm/demand_paging_test.c tools/testing/selftests/kvm/dirty_log_perf_test.c tools/testing/selftests/kvm/include/kvm_util.h +4 more+62 -140 replies
28/28KVM: selftests: Disable dirty logging with vCPUs runningtools/testing/selftests/kvm/dirty_log_perf_test.c+5 -51 reply

Reviews trailers per patch

17 of 28 patches have a Reviewed-by or Acked-by.

PatchPeter FeinerPaolo BonziniSean ChristophersonPeter ZijlstraDavidlohr BuesoWaiman Long
1/28R-b A-b
2/28R-b A-b
3/28R-b A-b R-b
4/28R-b A-b R-b
5/28R-b A-b
6/28A-b A-b A-b A-b
7/28A-b A-b A-b A-b
8/28A-b A-b A-b A-b
9/28
10/28
11/28
12/28
13/28R-b
14/28R-b
15/28R-b
16/28
17/28R-b
18/28R-b
19/28
20/28R-b
21/28R-b
22/28R-b
23/28R-b
24/28
25/28
26/28
27/28
28/28

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 6 reviewers.

Read the v2 thread

Cover letter [PATCH v2 00/28] Allow parallel MMU operations with TDP MMU

The TDP MMU was implemented to simplify and improve the performance of KVM's memory management on modern hardware with TDP (EPT / NPT). To build on the existing performance improvements of the TDP MMU, add the ability to handle vCPU page faults, enabling and disabling dirty logging, and removing mappings, in parallel. In the current implementation, vCPU page faults (actually EPT/NPT violations/misconfigurations) are the largest source of MMU lock contention on VMs with many vCPUs. This contention, and the resulting page fault latency, can soft-lock guests and degrade performance. Handling page faults in parallel is especially useful when booting VMs, enabling dirty logging, and handling demand paging. In all these cases vCPUs are constantly incurring page faults on each new page accessed.

Broadly, the following changes were required to allow parallel page faults (and other MMU operations):
-- Contention detection and yielding added to rwlocks to bring them up to
feature parity with spin locks, at least as far as the use of the MMU lock is concerned.
-- TDP MMU page table memory is protected with RCU and freed in RCU
callbacks to allow multiple threads to operate on that memory concurrently.
-- The MMU lock was changed to an rwlock on x86. This allows the page
fault handlers to acquire the MMU lock in read mode and handle page faults in parallel, and other operations to maintain exclusive use of the lock by acquiring it in write mode.
-- An additional lock is added to protect some data structures needed by
the page fault handlers, for relatively infrequent operations.
-- The page fault handler is modified to use atomic cmpxchgs to set SPTEs
and some page fault handler operations are modified slightly to work concurrently with other threads.

This series also contains a few bug fixes and optimizations, related to the above, but not strictly part of enabling parallel page fault handling.

Correctness testing:
The following tests were performed with an SMP kernel and DBX kernel on an Intel Skylake machine. The tests were run both with and without the TDP MMU enabled.
-- This series introduces no new failures in kvm-unit-tests SMP + no TDP MMU no new failures
SMP + TDP MMU no new failures
DBX + no TDP MMU no new failures
DBX + TDP MMU no new failures
-- All KVM selftests behave as expected
SMP + no TDP MMU all pass except ./x86_64/vmx_preemption_timer_test SMP + TDP MMU all pass except ./x86_64/vmx_preemption_timer_test (./x86_64/vmx_preemption_timer_test also fails without this patch set, both with the TDP MMU on and off.)
DBX + no TDP MMU all pass
DBX + TDP MMU all pass
-- A VM can be booted running a Debian 9 and all memory accessed SMP + no TDP MMU works
SMP + TDP MMU works
DBX + no TDP MMU works
DBX + TDP MMU works

This series can be viewed in Gerrit at:
https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/7172

Changelog v1 -> v2:
- Removed the MMU lock union + using a spinlock when the TDP MMU is disabled
- Merged RCU commits
- Extended additional MMU operations to operate in parallel
- Ammended dirty log perf test to cover newly parallelized code paths
- Misc refactorings (see changelogs for individual commits)
- Big thanks to Sean and Paolo for their thorough review of v1

Ben Gardon (28):
KVM: x86/mmu: change TDP MMU yield function returns to match
cond_resched
KVM: x86/mmu: Add comment on __tdp_mmu_set_spte
KVM: x86/mmu: Add lockdep when setting a TDP MMU SPTE
KVM: x86/mmu: Don't redundantly clear TDP MMU pt memory
KVM: x86/mmu: Factor out handling of removed page tables locking/rwlocks: Add contention detection for rwlocks
sched: Add needbreak for rwlocks
sched: Add cond_resched_rwlock
KVM: x86/mmu: Fix braces in kvm_recover_nx_lpages
KVM: x86/mmu: Fix TDP MMU zap collapsible SPTEs
KVM: x86/mmu: Merge flush and non-flush tdp_mmu_iter_cond_resched
KVM: x86/mmu: Rename goal_gfn to next_last_level_gfn
KVM: x86/mmu: Ensure forward progress when yielding in TDP MMU iter
KVM: x86/mmu: Yield in TDU MMU iter even if no SPTES changed
KVM: x86/mmu: Skip no-op changes in TDP MMU functions
KVM: x86/mmu: Clear dirtied pages mask bit before early break
KVM: x86/mmu: Protect TDP MMU page table memory with RCU
KVM: x86/mmu: Use an rwlock for the x86 MMU
KVM: x86/mmu: Factor out functions to add/remove TDP MMU pages
KVM: x86/mmu: Use atomic ops to set SPTEs in TDP MMU map
KVM: x86/mmu: Flush TLBs after zap in TDP MMU PF handler
KVM: x86/mmu: Mark SPTEs in disconnected pages as removed
KVM: x86/mmu: Allow parallel page faults for the TDP MMU
KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read
lock
KVM: selftests: Add backing src parameter to dirty_log_perf_test
KVM: selftests: Disable dirty logging with vCPUs running

 arch/x86/include/asm/kvm_host.h               |  15 +
 arch/x86/kvm/mmu/mmu.c                        | 120 +--
 arch/x86/kvm/mmu/mmu_internal.h               |   9 +-
 arch/x86/kvm/mmu/page_track.c                 |   8 +-
 arch/x86/kvm/mmu/paging_tmpl.h                |   8 +-
 arch/x86/kvm/mmu/spte.h                       |  21 +-
 arch/x86/kvm/mmu/tdp_iter.c                   |  46 +-
 arch/x86/kvm/mmu/tdp_iter.h                   |  21 +-
 arch/x86/kvm/mmu/tdp_mmu.c                    | 741 ++++++++++++++----
 arch/x86/kvm/mmu/tdp_mmu.h                    |   5 +-
 arch/x86/kvm/x86.c                            |   4 +-
 include/asm-generic/qrwlock.h                 |  24 +-
 include/linux/kvm_host.h                      |   5 +
 include/linux/rwlock.h                        |   7 +
 include/linux/sched.h                         |  29 +
 kernel/sched/core.c                           |  40 +
 .../selftests/kvm/demand_paging_test.c        |   3 +-
 .../selftests/kvm/dirty_log_perf_test.c       |  25 +-
 .../testing/selftests/kvm/include/kvm_util.h  |   6 -
 .../selftests/kvm/include/perf_test_util.h    |   3 +-
 .../testing/selftests/kvm/include/test_util.h |  14 +
 .../selftests/kvm/lib/perf_test_util.c        |   6 +-
 tools/testing/selftests/kvm/lib/test_util.c   |  29 +
 virt/kvm/dirty_ring.c                         |  10 +
 virt/kvm/kvm_main.c                           |  46 +-
 25 files changed, 963 insertions(+), 282 deletions(-)
Signature · 1 line
2.30.0.365.g02bc693789-goog