Skip to content

KVM: x86/mmu: Overhaul TDP MMU zapping and flushing

Sean Christopherson kvm lkml 1 version Applied

Status from this message: Queued, thanks.

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest28 patches 44 repliesApplied

Patches in v1 Applied

#Subject and filesSizeRepliesApplied
1/28KVM: x86/mmu: Use yield-safe TDP MMU root iter in MMU notifier unmappingarch/x86/kvm/mmu/tdp_mmu.c+1 -12 replies
2/28KVM: x86/mmu: Skip tlb flush if it has been done in zap_gfn_range()arch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_mmu.c+3 -30 replies
3/28KVM: x86/mmu: Remove spurious TLB flushes in TDP MMU zap collapsible patharch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_mmu.c arch/x86/kvm/mmu/tdp_mmu.h+11 -250 replies
4/28KVM: x86/mmu: Retry page fault if root is invalidated by memslot updatearch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/paging_tmpl.h+23 -33 replies
5/28KVM: x86/mmu: Check for present SPTE when clearing dirty bit in TDP MMUarch/x86/kvm/mmu/tdp_mmu.c+3 -01 reply
6/28KVM: x86/mmu: Formalize TDP MMU's (unintended?) deferred TLB flush logicarch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_mmu.c+17 -10 replies
7/28KVM: x86/mmu: Document that zapping invalidated roots doesn't need to flusharch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_mmu.c+17 -80 replies
8/28KVM: x86/mmu: Drop unused @kvm param from kvm_tdp_mmu_get_root()arch/x86/kvm/mmu/tdp_mmu.c arch/x86/kvm/mmu/tdp_mmu.h+5 -51 reply
9/28KVM: x86/mmu: Require mmu_lock be held for write in unyielding root iterarch/x86/kvm/mmu/tdp_mmu.c+12 -42 replies
10/28KVM: x86/mmu: Allow yielding when zapping GFNs for defunct TDP MMU rootarch/x86/kvm/mmu/tdp_mmu.c+6 -15 replies
11/28KVM: x86/mmu: Check for !leaf=>leaf, not PFN change, in TDP MMU SP removalarch/x86/kvm/mmu/tdp_mmu.c+5 -20 replies
12/28KVM: x86/mmu: Batch TLB flushes from TDP MMU for MMU notifier change_sptearch/x86/kvm/mmu/tdp_mmu.c+6 -71 reply
13/28KVM: x86/mmu: Drop RCU after processing each root in MMU notifier hooksarch/x86/kvm/mmu/tdp_mmu.c+4 -31 reply
14/28KVM: x86/mmu: Add helpers to read/write TDP MMU SPTEs and document RCUarch/x86/kvm/mmu/tdp_iter.c arch/x86/kvm/mmu/tdp_iter.h arch/x86/kvm/mmu/tdp_mmu.c+26 -101 reply
15/28KVM: x86/mmu: Take TDP MMU roots off list when invalidating all rootsarch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_mmu.c arch/x86/kvm/mmu/tdp_mmu.h+108 -834 replies
16/28KVM: x86/mmu: WARN if old _or_ new SPTE is REMOVED in non-atomic patharch/x86/kvm/mmu/tdp_mmu.c+2 -21 reply
17/28KVM: x86/mmu: Terminate yield-friendly walk if invalid root observedarch/x86/kvm/mmu/tdp_mmu.c+12 -71 reply
18/28KVM: x86/mmu: Refactor low-level TDP MMU set SPTE helper to take raw valsarch/x86/kvm/mmu/tdp_mmu.c+29 -181 reply
19/28KVM: x86/mmu: Zap only the target TDP MMU shadow page in NX recoveryarch/x86/kvm/mmu/mmu_internal.h arch/x86/kvm/mmu/tdp_iter.h arch/x86/kvm/mmu/tdp_mmu.c +1 more+33 -223 replies
20/28KVM: x86/mmu: Use common TDP MMU zap helper for MMU notifier unmap hookarch/x86/kvm/mmu/tdp_mmu.c+2 -71 reply
21/28KVM: x86/mmu: Add TDP MMU helper to zap a rootarch/x86/kvm/mmu/tdp_mmu.c+9 -23 replies
22/28KVM: x86/mmu: Skip remote TLB flush when zapping all of TDP MMUarch/x86/kvm/mmu/tdp_mmu.c+6 -51 reply
23/28KVM: x86/mmu: Use "zap root" path for "slow" zap of all TDP MMU SPTEsarch/x86/kvm/mmu/tdp_mmu.c+5 -20 replies
24/28KVM: x86/mmu: Add dedicated helper to zap TDP MMU root shadow pagearch/x86/kvm/mmu/tdp_mmu.c+66 -251 reply
25/28KVM: x86/mmu: Require mmu_lock be held for write to zap TDP MMU rangearch/x86/kvm/mmu/tdp_mmu.c+6 -231 reply
26/28KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()arch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_mmu.c arch/x86/kvm/mmu/tdp_mmu.h+14 -391 reply
27/28KVM: x86/mmu: Do remote TLB flush before dropping RCU in TDP MMU reschedarch/x86/kvm/mmu/tdp_mmu.c+2 -25 replies
28/28KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow pagesarch/x86/kvm/mmu/mmu.c arch/x86/kvm/mmu/tdp_iter.h arch/x86/kvm/mmu/tdp_mmu.c+26 -161 reply

Reviews trailers per patch

15 of 28 patches have a Reviewed-by or Acked-by.

PatchBen Gardon
1/28R-b
2/28
3/28
4/28R-b
5/28R-b
6/28
7/28
8/28R-b
9/28
10/28R-b
11/28
12/28R-b
13/28R-b
14/28R-b
15/28
16/28R-b
17/28
18/28R-b
19/28
20/28R-b
21/28
22/28
23/28
24/28
25/28R-b
26/28R-b
27/28R-b
28/28R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 00/28] KVM: x86/mmu: Overhaul TDP MMU zapping and flushing

Overhaul TDP MMU's handling of zapping and TLB flushing to reduce the number of TLB flushes, and to clean up the zapping code. The final patch realizes the biggest change, which is to use RCU to defer any TLB flush due to zapping a SP to the caller. The largest cleanup is to separate the flows for zapping roots (zap _everything_), zapping leaf SPTEs (zap guest mappings for whatever reason), and zapping a specific SP (NX recovery). They're currently smushed into a single zap_gfn_range(), which was a good idea at the time, but became a mess when trying to handle the different rules, e.g. TLB flushes aren't needed when zapping a root because KVM can safely zap a root if and only if it's unreachable.

For booting an 8 vCPU, remote_tlb_flush (requests) goes from roughly 180 (600) to 130 (215).

Please don't apply patches 02 and 03, they've been posted elsehwere and by other people. I included them here because some of the patches have pseudo-dependencies on their changes. Patch 01 is also posted separately. I had a brain fart and sent it out realizing that doing so would lead to oddities.

Hou Wenlong (1):
KVM: x86/mmu: Skip tlb flush if it has been done in zap_gfn_range()

Sean Christopherson (27):
KVM: x86/mmu: Use yield-safe TDP MMU root iter in MMU notifier
unmapping
KVM: x86/mmu: Remove spurious TLB flushes in TDP MMU zap collapsible
path
KVM: x86/mmu: Retry page fault if root is invalidated by memslot
update
KVM: x86/mmu: Check for present SPTE when clearing dirty bit in TDP
MMU
KVM: x86/mmu: Formalize TDP MMU's (unintended?) deferred TLB flush
logic
KVM: x86/mmu: Document that zapping invalidated roots doesn't need to
flush
KVM: x86/mmu: Drop unused @kvm param from kvm_tdp_mmu_get_root()
KVM: x86/mmu: Require mmu_lock be held for write in unyielding root
iter
KVM: x86/mmu: Allow yielding when zapping GFNs for defunct TDP MMU
root
KVM: x86/mmu: Check for !leaf=>leaf, not PFN change, in TDP MMU SP
removal
KVM: x86/mmu: Batch TLB flushes from TDP MMU for MMU notifier
change_spte
KVM: x86/mmu: Drop RCU after processing each root in MMU notifier
hooks
KVM: x86/mmu: Add helpers to read/write TDP MMU SPTEs and document RCU
KVM: x86/mmu: Take TDP MMU roots off list when invalidating all roots
KVM: x86/mmu: WARN if old _or_ new SPTE is REMOVED in non-atomic path
KVM: x86/mmu: Terminate yield-friendly walk if invalid root observed
KVM: x86/mmu: Refactor low-level TDP MMU set SPTE helper to take raw
vals
KVM: x86/mmu: Zap only the target TDP MMU shadow page in NX recovery
KVM: x86/mmu: Use common TDP MMU zap helper for MMU notifier unmap
hook
KVM: x86/mmu: Add TDP MMU helper to zap a root
KVM: x86/mmu: Skip remote TLB flush when zapping all of TDP MMU
KVM: x86/mmu: Use "zap root" path for "slow" zap of all TDP MMU SPTEs
KVM: x86/mmu: Add dedicated helper to zap TDP MMU root shadow page
KVM: x86/mmu: Require mmu_lock be held for write to zap TDP MMU range
KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()
KVM: x86/mmu: Do remote TLB flush before dropping RCU in TDP MMU
resched
KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow
pages

 arch/x86/kvm/mmu/mmu.c          |  74 +++--
 arch/x86/kvm/mmu/mmu_internal.h |   7 +-
 arch/x86/kvm/mmu/paging_tmpl.h  |   3 +-
 arch/x86/kvm/mmu/tdp_iter.c     |   6 +-
 arch/x86/kvm/mmu/tdp_iter.h     |  15 +-
 arch/x86/kvm/mmu/tdp_mmu.c      | 526 +++++++++++++++++++-------------
 arch/x86/kvm/mmu/tdp_mmu.h      |  48 +--
 7 files changed, 406 insertions(+), 273 deletions(-)
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog