Skip to content

KVM: riscv: Age G-stage PTEs locklessly

SeungJu Cheon kvm kvm-riscv linux-riscv lkml 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest5 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5KVM: riscv: Age all G-stage PTEs in a GFN rangearch/riscv/include/asm/kvm_gstage.h arch/riscv/kvm/gstage.c arch/riscv/kvm/mmu.c+40 -180 replies
2/5KVM: riscv: Read G-stage PTEs once when walking page tablesarch/riscv/kvm/gstage.c+27 -210 replies
3/5KVM: riscv: Write-protect G-stage PTEs atomicallyarch/riscv/kvm/gstage.c+9 -60 replies
4/5KVM: riscv: Free G-stage page tables after an RCU grace periodarch/riscv/include/asm/kvm_gstage.h arch/riscv/kvm/gstage.c arch/riscv/kvm/main.c +1 more+32 -50 replies
5/5KVM: riscv: Age G-stage PTEs locklesslyarch/riscv/kvm/Kconfig arch/riscv/kvm/mmu.c+14 -130 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly

Aging G-stage PTEs currently runs with mmu_lock held for write, taken by the common MMU notifier code. When MGLRU or kswapd ages a large range, every vCPU taking a G-stage fault blocks on the lock, and the dirty-logging read-side fast path blocks behind aging as well.

This series lets RISC-V select KVM_MMU_LOCKLESS_AGING, so G-stage aging can run without mmu_lock while protecting page-table walks with RCU.

1. Age all G-stage PTEs in a GFN range
Bug fix, posted separately [1] and included unchanged so the series applies on riscv_kvm_queue. Drop it if already applied.

2. Read G-stage PTEs once when walking page tables
A lockless walker must derive presence, leaf-ness and the child address from a single snapshot of each entry.

3. Write-protect G-stage PTEs atomically
Aging will clear the Accessed bit without mmu_lock; the plain read-modify-write in GSTAGE_OP_WP could clobber that update.

4. Free G-stage page tables after an RCU grace period Table pages and the root outlive any RCU reader. Root teardown keeps pgd_levels intact so a walker holding the old root sees consistent metadata.

5. Age G-stage PTEs locklessly
Select KVM_MMU_LOCKLESS_AGING and walk under rcu_read_lock().

This builds on Jinyu Tang's rwlock and cmpxchg helpers already in riscv_kvm_queue. Wang Yechao's huge page recovery series [2] adds a second page-table free site in kvm_riscv_gstage_recover_huge(); whichever lands second will need to switch that put_page() to gstage_free_page_table(). I can rebase the series as needed.

A possible follow-up is to run clear-dirty-log and the general fault path under the read side of mmu_lock, as arm64 and x86 do. Patch 3 provides the atomic write-protection needed for the former.

Testing
-------
QEMU virt, -cpu rv64,h=true (Svadu), 4 vCPUs, TCG. The host kernel was built with KASAN, PROVE_LOCKING, PROVE_RCU, LOCK_STAT and LRU_GEN.

After each patch:
- dirty_log_perf_test -v 2 -b 256M -i 3
- access_tracking_perf_test -v 2 -b 256M -s anonymous_thp (2M THP disabled and 64K mTHP enabled)
- 20 VM create/destroy iterations

With MGLRU aging forced on the VM's cgroup every 50-100ms:
- dirty_log_perf_test -v 3 -b 256M -i 3
- 30 VM create/destroy iterations

With CONFIG_KVM=m, 5 module load/unload cycles, each with a VM created and destroyed immediately before module unload, to exercise module teardown after queuing RCU frees.

No WARN, KASAN, lockdep or RCU reports were observed. The kvm_age_hva tracepoint recorded approximately 65k events per aging pass.

Performance results from dirty_log_perf_test -v 3 -b 256M -i 3, with lock_stat enabled for mmu_lock and 86 aging passes:

no aging aging
before after before after

  guest dirty time       12.5s   12.4s     18.6s    16.7s
  mmu_lock W wait total  0.15s   0.17s     44.4s    0.15s
  mmu_lock W contentions  35k     35k      1.0M     30k
  mmu_lock R wait total   0       0        7.5s     0

Before the series, 93% of write-lock waiters were from kvm_mmu_notifier_clear_young(), and the dirty-log read-side fast path waited for mmu_lock 352k times. After the series, aging does not acquire mmu_lock.

Absolute numbers are inflated by TCG; only the relative change is meaningful. Under TCG, the atomic write-protect change in patch 3 increased the clear-dirty-log time from approximately 16ms to 30ms, as TCG serialises AMO instructions; hardware numbers would be welcome.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/kvm-riscv/[email protected]/

SeungJu Cheon (5):
KVM: riscv: Age all G-stage PTEs in a GFN range
KVM: riscv: Read G-stage PTEs once when walking page tables
KVM: riscv: Write-protect G-stage PTEs atomically
KVM: riscv: Free G-stage page tables after an RCU grace period
KVM: riscv: Age G-stage PTEs locklessly

 arch/riscv/include/asm/kvm_gstage.h |   4 +-
 arch/riscv/kvm/Kconfig              |   1 +
 arch/riscv/kvm/gstage.c             | 111 +++++++++++++++++++++-------
 arch/riscv/kvm/main.c               |   3 +
 arch/riscv/kvm/mmu.c                |  56 +++++++-------
 5 files changed, 117 insertions(+), 58 deletions(-)

base-commit: 41e81f7e3ef96594fb840445343c0ee7723aa550

Signature · 1 line
2.52.0