KVM: don't zap secondary MMU on permission upgrade
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v3 Latest9 patches 18 repliesIn review
- v29 patches 3 repliesSuperseded
- v1 RFC4 patches 8 repliesSuperseded
Changes in v3 from the cover letter
=============
- Patch 4 (was 5): check the old entry in the existing write-bit helpers, for any MM_CP_TRY_CHANGE_WRITABLE change (David).
- Patch 6: add mm_cp_is_perm_upgrade() here instead of in the old patch 5.
- Trim some commit messages.
Patches in v3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/9 | mm: fix eager write-bit restore for write-notified shared mappingsmm/ | +14 | 2 replies | |
| 2/9 | riscv: mm: add pud_soft_dirty()arch/ | +5 | 0 replies | |
| 3/9 | mm/huge_memory: honor MM_CP_TRY_CHANGE_WRITABLE in change_huge_pud()include/ mm/ | +32 | 4 replies | |
| 4/9 | mm: keep writable entries writable on protection changesmm/ mm/ | +27 | 2 replies | |
| 5/9 | mm/mprotect: add MM_CP_PERM_UPGRADEinclude/ mm/ | +15 | 5 replies | |
| 6/9 | mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADEinclude/ mm/ mm/ +1 more | +19 | 0 replies | |
| 7/9 | KVM: pass the mmu_notifier event and flags to kvm_gfn_rangeinclude/ virt/ | +11 | 0 replies | |
| 8/9 | KVM: x86/mmu: don't zap secondary MMU on permission upgradearch/ | +7 | 0 replies | |
| 9/9 | KVM: s390: don't unmap the gmap on permission upgradearch/ arch/ | +14 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.
Cover letter [PATCH v3 0/9] KVM: don't zap secondary MMU on permission upgrade
Stop KVM from zapping its mappings on permission upgrades on x86 and s390, as these keep the PFNs unchanged and revoke no access. KVM faults in the new permissions on demand, so dropping mappings early gains nothing.
To do so, add an MMU notifier flag that marks an MMU_NOTIFY_PROTECTION_VMA invalidation as a strict permission upgrade. For the flag to be accurate, change_protection() keeps writable entries writable on such changes. A few cases where it needlessly makes or leaves entries read-only are fixed along the way.
Background
==========
change_protection() pairs every protection change with an MMU_NOTIFY_PROTECTION_VMA invalidation, and KVM zaps its SPTEs in response, as it cannot tell whether permissions widened or narrowed. This is costly, e.g. during a QEMU background snapshot, which uses uffd-wp to write-protect guest RAM and unprotects each page once it is saved. Every unprotect then makes the guest refault memory it could already access.
Secondary MMUs that share the CPU page tables (e.g. ARM SMMU, IOMMU/ATS) can keep read-only entries in their TLBs that a fault does not flush, so they are already notified of permission upgrades on the CPU TLB flush path [1]. KVM keeps its own page tables and faults in permissions on demand, so it needs no notification, as Sean noted when that signalling was first proposed [2].
Series overview
===============
Patches 1-4 fix cases where change_protection() needlessly makes or leaves entries read-only, so that the next write to them faults. They only affect performance and can be picked up independently.
Patches 5-6 add MM_CP_PERM_UPGRADE for mprotect() calls that only add permissions, and MMU_NOTIFIER_RANGE_PERM_UPGRADE, which is set for these and for userfaultfd unprotects.
Patches 7-9 pass the MMU notifier event and flags to KVM and skip the zap on x86 and s390.
Considerations
==============
1. This series covers only x86 and s390. Claudio confirmed that skipping the zap should be safe on s390 and offered to test it [3], so patch 9 is only compile-tested. Other architectures should be able to skip it as well, but I have not tested or benchmarked them. Moving the code to arch-independent KVM would also avoid the SEV cache flush in kvm_arch_guest_memory_reclaimed().
2. struct kvm_gfn_range carries the MMU notifier event and flags, as agreed at LPC [4] and proposed in the past for other purposes
[5] [6]. The fields are only set for MMU notifier invalidations.
3. Writable migration and device-private entries are still made read-only on a permission upgrade. This does not affect secondary MMUs (see patch 6), but the downgrade could be skipped as well if preferred.
4. hugetlb does not benefit, as hugetlb_change_protection() does not restore the write bit and may unshare page tables (see patch 6).
5. The notification cannot simply be skipped in MM code (see [7]). Secondary MMUs that keep their own page tables but cannot fault rely on it to pick up new permissions. A flag lets each user opt in instead.
Benchmarks
==========
These were measured on v1.
AMD EPYC 9654 96-Core Processor host running a recent Linux kernel (6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while the host takes a QEMU background snapshot. Results are aggregated over 5 runs on a stock kernel and on one with this series, rebooting the guest between runs, with NUMA balancing disabled.
pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush debugfs stats):
guest pattern baseline patched
-----------------------------------------------------------------------
read64 (read) pf/s=950,015 tlb/s=1,440 pf/s=2,503 tlb/s=0.1 rand-set (read+write) pf/s=1,004,353 tlb/s=862 pf/s=4,073 tlb/s=0.3 write64 (write) pf/s=968,018 tlb/s=94 pf/s=2,292 tlb/s=0.3
Guest read latency was sampled with a paced probe thread in the guest, reading one byte per page with an idle gap between reads, while stress-ng ran with read64. Each figure is the median across 5 runs, with the run-to-run min and max, in ns:
metric baseline patched
------------------------------------------------------
p50 (min max) 340 (270 380) 260 (250 260)
p99 (min max) 19789 (16885 20200) 5207 (4076 5638) p99.9 (min max) 23445 (22845 24036) 6850 (6259 7110)
The median improves modestly, while p99 drops ~3.8x.
A separate stress-ng run with write64 and --verify showed no data corruption and no livelocks.
Changes in v3
=============
- Patch 4 (was 5): check the old entry in the existing write-bit helpers, for any MM_CP_TRY_CHANGE_WRITABLE change (David).
- Patch 6: add mm_cp_is_perm_upgrade() here instead of in the old patch 5.
- Trim some commit messages.
Changes in v2
=============
- New patch 1: fix setting MM_CP_TRY_CHANGE_WRITABLE for shmem with uffd-wp or soft-dirty (Sashiko).
- New patches 2-3: honor MM_CP_TRY_CHANGE_WRITABLE in change_huge_pud(), adding pud_soft_dirty() on riscv (Sashiko).
- Patch 4 (was 1): only treat changes to the read, write and execute flags as an upgrade, plus VM_ACCOUNT (Sashiko).
- New patch 5: keep writable entries writable on permission upgrades (Sashiko).
- Patch 6 (was 2): don't set the flag for hugetlb (Sashiko).
- Patch 7 (was 3): pass the MMU notifier event and flags instead of a bool (Claudio).
- Patch 8 (was 4): check the flag in range->flags. The APIC-access page reload issue that Sashiko pointed out is pre-existing.
- New patch 9: skip the unmap on s390 as well (Claudio).
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
References
==========
[1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/all/20261030133939.630e29af@p-imbrenda/
[4] https://lpc.events/event/20/contributions/2552/
[5] https://lore.kernel.org/all/[email protected]
[6] https://lore.kernel.org/all/[email protected]
[7] https://lore.kernel.org/all/[email protected]/
Carlos López (9):
mm: fix eager write-bit restore for write-notified shared mappings
riscv: mm: add pud_soft_dirty()
mm/huge_memory: honor MM_CP_TRY_CHANGE_WRITABLE in change_huge_pud()
mm: keep writable entries writable on protection changes mm/mprotect: add MM_CP_PERM_UPGRADE
mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
KVM: pass the mmu_notifier event and flags to kvm_gfn_range
KVM: x86/mmu: don't zap secondary MMU on permission upgrade
KVM: s390: don't unmap the gmap on permission upgrade
arch/riscv/include/asm/pgtable.h | 5 ++++ arch/s390/kvm/arm64/arm.c | 7 +++++ arch/s390/kvm/s390/s390.c | 7 +++++ arch/x86/kvm/mmu/mmu.c | 7 +++++ include/linux/kvm_host.h | 3 +++ include/linux/mm.h | 2 ++ include/linux/mmu_notifier.h | 5 ++++ include/linux/pgtable.h | 5 ++++ mm/huge_memory.c | 46 ++++++++++++++++++++++++++++---- mm/hugetlb.c | 4 +++ mm/internal.h | 7 +++++ mm/mprotect.c | 29 ++++++++++++++++++-- mm/vma.h | 23 +++++++++------- virt/kvm/kvm_main.c | 8 ++++++ 14 files changed, 142 insertions(+), 16 deletions(-)
Signature · 1 line
2.51.0