Skip to content

RFC KVM: x86/mmu: Allocate memslot rmaps on demand

Fred Griffoul kvm lkml 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest5 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5KVM: x86/mmu: Walk memslot rmaps by indexarch/x86/kvm/mmu/mmu.c+24 -100 replies
2/5KVM: x86: Count rmaps per level in a debugfs helperarch/x86/kvm/debugfs.c+19 -140 replies
3/5KVM: x86/mmu: Allocate memslot rmaps in on-demand chunksarch/x86/include/asm/kvm_host.h arch/x86/kvm/debugfs.c arch/x86/kvm/mmu/mmu.c +1 more+153 -220 replies
4/5KVM: selftests: Declare __tdp_map() in the x86 processor headertools/testing/selftests/kvm/include/x86/processor.h+1 -00 replies
5/5KVM: selftests: Test dirty logging of sparse nested mappingstools/testing/selftests/kvm/Makefile.kvm tools/testing/selftests/kvm/x86/nested_dirty_log_sparse_test.c+276 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/5] KVM: x86/mmu: Allocate memslot rmaps on demand

From: Fred Griffoul <[email protected]>

When an L1 guest runs a nested guest, KVM allocates rmaps for all of L1's memory the first time it creates a shadow root, one 8-byte head per 4 KiB page. For an L1 with 4 TiB of memory that is 8 GiB, allocated at once on the first nested VM-Enter, whether L2 maps 1% or all of L1's memory. Sean pointed at this cost on Mushahid's lpage_info patch [1].

This RFC allocates rmaps in page-sized chunks instead, when the first SPTE in a chunk's range is added. Each memslot keeps a per-level directory of chunks, 8 bytes per 2 MiB at the 4 KiB level, so the 4 TiB L1 above allocates 16 MiB on the first nested VM-Enter. A missing chunk reads as empty rmaps and slot walks skip it. When L2 maps every gfn, rmaps cost what they did before plus the directory.

Patches 1 and 2 prepare the slot walk and the debugfs rmap statistics, with no functional change. Patch 3 does the chunking. Patches 4 and 5 add a selftest in which L2 writes pages around chunk boundaries and checks the dirty log after the memslot is write-protected, including eager splitting of nested huge pages.

I would like feedback on:
- The data structure. A directory of page-sized chunks keeps lookups to one extra load. An xarray would be simpler to grow but adds a tree walk to every rmap lookup.
- Freeing chunks once they are empty. That would let the shadow MMU page limit bound rmap memory too, but lockless aging reads chunks without mmu_lock, so freeing needs an RCU grace period. I left it out of this RFC.
- Doing the same for gfn_write_track, 2 bytes per 4 KiB page, which is still allocated for every memslot on the first shadow root.

Limitations:
- Chunks are only freed with the memslot.
- Eager page splitting on dirty-log enable splits L2's huge mappings, so every 2 MiB that L2 mapped with a huge page then gets a chunk.

The series is based on kvm.git next.

[1] https://lore.kernel.org/all/[email protected]/

Fred

Fred Griffoul (5):
KVM: x86/mmu: Walk memslot rmaps by index
KVM: x86: Count rmaps per level in a debugfs helper
KVM: x86/mmu: Allocate memslot rmaps in on-demand chunks
KVM: selftests: Declare __tdp_map() in the x86 processor header
KVM: selftests: Test dirty logging of sparse nested mappings

 arch/x86/include/asm/kvm_host.h               |  22 +-
 arch/x86/kvm/debugfs.c                        |  46 ++-
 arch/x86/kvm/mmu/mmu.c                        | 126 +++++++-
 arch/x86/kvm/x86.c                            |  22 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../kvm/x86/nested_dirty_log_sparse_test.c    | 275 ++++++++++++++++++
 7 files changed, 460 insertions(+), 33 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/nested_dirty_log_sparse_test.c

base-commit: d21eef2084a9af87ac2a783bc7c2b175ba114e4e

Signature · 1 line
2.47.3