KVM: x86/mmu: Prefetch forward run of pages on TDP page faults
RFC 1/1 KVM: x86/mmu: Prefetch forward run of pages on TDP page faults
The TDP MMU installs one SPTE per fault, so faulting in guest memory via userfaultfd (e.g. snapshot restore) without hugetlbfs costs a VM-exit per 4KiB page. Mirror the shadow MMU's prefetch: after a 4KiB fault, resolve the forward run of host-present pages in the faulting leaf table (one guest 2MiB region) with one non-blocking GUP and fill the empty SPTEs. When userfaultfd populates in large chunks this maps up to 511 neighbours per fault, cutting EPT violations up to 512x.
Touching 128MiB backed by userfaultfd with 2MiB UFFD_COPY chunks:
c8i.metal-48xl 129.1 -> 100.3 ms (-22%), nested 495.8 -> 121.5 ms (-75%). With no batching (one copy per fault) there is a ~2-4% regression.
Best-effort and minimal: skips AD-disabled SPs, mirror (TDX) roots and guest_memfd; installs atomically into empty entries only.
Add pf_prefetch_{called,pages,mapped,unused} stats.
---
Hello,
I'm Marco Marangoni, from AWS Firecracker.
This is the first time I (try to) submit a patch upstream. I'll do my best to respect your time, but I apologize in advance, since I'll get something wrong for sure :)
I'm looking for opportunities to improve VM snapshot restore time in case the guest memory is registered with UFFD, _without_ using hugetlbfs.
I found a pretty big opportunity in the x86 TDP MMU, borrowing an approach already used in the shadow MMU. By prefetching SPTEs around a faulting GFN, the number of EPT violations can be significantly reduced. For example, if the UFFD handler copies memory in 2MiB chunks, it's theoretically possible to prefetch all 512 4KiB pages in a single VM exit, reducing the number of EPT violations by 512x. In my tests, when using UFFD_COPY with 2MiB chunks, touching 128MiB of guest memory is 22% faster on a c8i.metal-48xl (from 129.1 ms to 100.3 ms), and 75% faster on nested virtualization (from 495.8 ms to 121.5 ms). With UFFD using 512KiB chunks, it's 15.3% faster on metal, and 70% faster on nested with respect to baseline.
The attached patch is a proof of concept, and it's not ready to be merged.
It works by doing a fast GUP forward walk of up to 511 pages, limited to the faulting leaf page table (one guest 2 MiB region), stopping at the first already-mapped SPTE or the first host hole. By design, this PoC is minimal; I'd be happy to extend the approach to ARM, guest_memfd, etc.
Before I polish it, I'd like your feedback on the approach chosen, specifically:
- Since I need to store up to 512 struct page pointers returned by GUP, I added a pointer in the kvm_vcpu_arch struct to an auxiliary 4KiB page per vCPU. Any concerns?
- Prefetching works forward only: in case the guest is accessing memory in reverse order, this patch won't help
- There's a slight 2-4% performance regression when UFFD does not batch memory copy (i.e. prefetching does nothing). Do you think it's worth adding a way to enable/disable this prefetching mechanism?
- SPTE install is open coded rather than adapting existing TDP helpers like tdp_mmu_set_spte_atomic. Is it worth adding a new helper or adapting existing ones?
- When dirty logging is active, prefetch installs writable SPTEs and make_spte() eagerly marks each prefetched page dirty, adding pages to the dirty set that the guest never wrote. Should prefetch be suppressed on slots with dirty tracking enabled?
Some downsides can be fixed by some alternative patches I explored (let me know if you want me to share them):
- "hierarchical prefetch": in this alternative approach, the prefetching happens in a "hierarchical" way, prefetching more and more pages in power of two increments, in batches of 8.
- "direct prefetch": in this proof of concept, I experimented with skipping GUP entirely, and doing a lockless page walk directly in the KVM module, similarly to what host_pfn_mapping_level does
Performance-wise, they're similar to the attached patch on metal (within margin of error), and neither mitigates the regression in case UFFD doesn't batch memory.
On the positive side, those can be implemented without extra allocations, and work both "forward and backward";
On the negative side, "hierarchical prefetch" is 10% slower on nested, and more complex. "direct prefetch" is relatively simple and fast, but feels like a hack, since it bypasses GUP.
It's worth mentioning that I also evaluated using the existing KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as it requires the vCPU to be paused.
I look forward to your reply,
Marco
arch/x86/include/asm/kvm_host.h | 11 ++++ arch/x86/kvm/mmu/mmu.c | 17 +++++- arch/x86/kvm/mmu/tdp_mmu.c | 97 +++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 4 ++ 4 files changed, 128 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +11 -0
@@ -897,6 +897,13 @@ struct kvm_vcpu_arch {*/ struct kvm_mmu_memory_cache mmu_external_spt_cache; +/* +* Per-vCPU scratch (one page, 512 page ptrs) for the wide GUP done +* during TDP fault-time prefetch. Owning vCPU thread only; not for +* zap/mmu-notifier contexts. +*/ +struct page **mmu_prefetch_pages; +/* * QEMU userspace and the guest each have their own FPU state. * In vcpu_run, we switch between the user and guest FPU contexts.
@@ -1722,6 +1729,10 @@ struct kvm_vcpu_stat {u64 pf_fast;u64 pf_mmio_spte_created;u64 pf_guest; +u64 pf_prefetch_called; +u64 pf_prefetch_pages; +u64 pf_prefetch_mapped; +u64 pf_prefetch_unused;u64 tlb_flush;u64 invlpg;
arch/x86/kvm/mmu/mmu.c +16 -1
@@ -636,6 +636,11 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_shadowed_info_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_external_spt_cache);kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_header_cache); + +if (vcpu->arch.mmu_prefetch_pages) { +free_page ((unsigned long )vcpu->arch.mmu_prefetch_pages); + vcpu->arch.mmu_prefetch_pages =NULL ; + } }static void mmu_free_pte_list_desc (struct pte_list_desc *pte_list_desc)
@@ -6808,6 +6813,13 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu) {int ret; +/* Prefetch scratch: one page of 512 page ptrs. Freed via kvm_mmu_destroy(). */ +BUILD_BUG_ON (SPTE_ENT_PER_PAGE *sizeof (struct page *) >PAGE_SIZE ); + vcpu->arch.mmu_prefetch_pages = + (struct page **)__get_free_page (GFP_KERNEL_ACCOUNT ); +if (!vcpu->arch.mmu_prefetch_pages) +return -ENOMEM ; + vcpu->arch.mmu_pte_list_desc_cache.kmem_cache = pte_list_desc_cache; vcpu->arch.mmu_pte_list_desc_cache.gfp_zero = __GFP_ZERO;
@@ -6824,7 +6836,7 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu) ret =__kvm_mmu_create (vcpu, &vcpu->arch.guest_mmu);if (ret) -return ret; + goto fail_prefetch; ret =__kvm_mmu_create (vcpu, &vcpu->arch.root_mmu);if (ret)
@@ -6833,6 +6845,9 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)return ret; fail_allocate_root:free_mmu_pages (&vcpu->arch.guest_mmu); + fail_prefetch: +free_page ((unsigned long )vcpu->arch.mmu_prefetch_pages); + vcpu->arch.mmu_prefetch_pages =NULL ;return ret; }
arch/x86/kvm/mmu/tdp_mmu.c +97 -0
109 lines not on this page · raw message
arch/x86/kvm/x86.c +4 -0
@@ -271,6 +271,10 @@ const struct kvm_stats_desc kvm_vcpu_stats_desc[] = {STATS_DESC_COUNTER (VCPU , pf_fast),STATS_DESC_COUNTER (VCPU , pf_mmio_spte_created),STATS_DESC_COUNTER (VCPU , pf_guest), +STATS_DESC_COUNTER (VCPU , pf_prefetch_called), +STATS_DESC_COUNTER (VCPU , pf_prefetch_pages), +STATS_DESC_COUNTER (VCPU , pf_prefetch_mapped), +STATS_DESC_COUNTER (VCPU , pf_prefetch_unused),STATS_DESC_COUNTER (VCPU , tlb_flush),STATS_DESC_COUNTER (VCPU , invlpg), STATS_DESC_COUNTER(VCPU, exits),
Signature · 1 line
2.47.3
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] Bypassing the mandatory MMU notifier retry protocol during prefetch leads to a Use-After-Free…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Bypassing the mandatory MMU notifier retry protocol during prefetch leads to a Use-After-Free of host pages.
- [High] Unconditional write privileges in prefetch cause unnecessary host Copy-On-Write and flood the KVM dirty ring. --
--- Patch [1]: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults --- commit e7ef47a2b5ba2806fbad91e89e8ac0b8f9cf602d
Author: Marco Marangoni <[email protected]>
KVM: x86/mmu: Prefetch forward run of pages on TDP page faults
This patch implements a fast GUP forward walk to prefetch up to 512 SPTEs (one guest 2MiB region) during a 4KiB page fault in the TDP MMU. It aims to reduce EPT violations by resolving adjacent pages without blocking, stopping at the first already-mapped SPTE or host hole.
Quoted 3 lines
arch/x86/kvm/mmu/tdp_mmu.c +0
-0
[ ... ]
Quoted 8 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-0@@ -1209,6 +1209,99 @@ static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter,static int tdp_mmu_split_huge_page (struct kvm *kvm,struct tdp_iter *iter,struct kvm_mmu_page *sp,bool shared); +/* +* Prefetch the forward run of host-present pages after the fault, within the +* faulting leaf table (512 pages). One non-blocking GUP fills the empty SPTEs. +* Forward only; capped at the first present SPTE and the first host hole.
arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, · line 1225
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +10
-0+*/ +static void tdp_mmu_pte_prefetch (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault, +struct tdp_iter *iter) +{ +struct kvm_mmu_page *sp =sptep_to_sp (rcu_dereference (iter->sptep)); +struct page **pages = vcpu->arch.mmu_prefetch_pages; +struct kvm_memory_slot *slot = fault->slot; +unsigned int access = sp->role.access; +bool host_writable = !(slot->flags &KVM_MEM_READONLY );
[Severity: High]
Does this code unconditionally request write privileges? Since host_writable is based only on the slot flags, it seems to bypass the original fault intent and whether dirty logging is active. When dirty logging is enabled, this could result in make_spte() marking up to 511 unmodified pages as dirty, which might flood the KVM dirty ring. Additionally, because the existing kvm_prefetch_pages() helper unconditionally uses FOLL_WRITE, could this force unnecessary host Copy-On-Write for pages the guest only intended to read?
arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, · line 1238
Quoted 13 lines
arch/x86/kvm/mmu/tdp_mmu.c +13
-0+gfn_t start_gfn, slot_end; +int start, count, nr, i; + +if (sp_ad_disabled (sp)) +return ; + +/* Mirror (TDX) needs set_external_spte(); gmem pfns aren't in GUP's tables. */ +if (is_mirror_sp (sp) ||kvm_slot_has_gmem (slot)) +return ; + +/* Racing invalidation may be stale. No mmu_seq recheck: GUP is under the lock. */ +if (unlikely (vcpu->kvm->mmu_invalidate_in_progress)) +return ;
[Severity: Critical]
Is it safe to bypass the mmu_invalidate_seq retry protocol here?
The TDP MMU operates with mmu_lock held in read mode, which allows concurrent MMU notifiers like kvm_unmap_gfn_range() to run in parallel. Could a concurrent host memory unmap free the physical page after kvm_prefetch_pages() resolves the PFN?
[ ... ]
Quoted 9 lines
arch/x86/kvm/mmu/tdp_mmu.c +9
-0+/* Non-blocking GUP; stops at the first host hole. */ + nr =kvm_prefetch_pages (slot, start_gfn, pages, count); +if (nr <=0 ) +return ; + + vcpu->stat.pf_prefetch_pages += nr; + +for (i =0 ; i < nr; i++) { +u64 *sptep = sp->spt + start + i;
arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, · line 1286
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +10
-0+u64 old_spte =SHADOW_NONPRESENT_VALUE ; +gfn_t gfn = start_gfn + i; +u64 new_spte; + +make_spte (vcpu, sp, slot, access, gfn, +page_to_pfn (pages[i]), old_spte, + true/* prefetch */ , false, host_writable, &new_spte); + +/* cmpxchg from empty is the race check; present/MMIO/frozen fails it. */ +if (try_cmpxchg64 (sptep, &old_spte, new_spte)) {
[Severity: Critical]
If a concurrent host memory unmap happened during kvm_prefetch_pages(), could this cmpxchg install a valid, writable SPTE to a freed page since there is no re-check of the invalidation sequence?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Sean Christopherson
maintainerQuoted 3 lines · Marco Marangoni
It's worth mentioning that I also evaluated using the existing
KVM_PRE_FAULT_MEMORYioctl, but this doesn't work well for our use-case, as it requires the vCPU to be paused.
What about if/when KVM Userfault[*] comes along? I.e. pre-fault memory when the vCPU exits to userspace.
If KVM Userfault + KVM_PRE_FAULT_MEMORY is viable for Firecracker and provides the desired performance, I would strongly prefer that approach over KVM doing its own prefetching. That way other architectures don't need to implement prefetching as well (arm64 already supports KVM_PRE_FAULT_MEMORY, and RISC-V has patches posted).
I'm aiming to get back to that series "soon", this would simply be more motivation to get KVM Userfault landed.
Quoted 8 lines · Sean Christopherson
Quoted 3 lines · Marco Marangoni
It's worth mentioning that I also evaluated using the existing
KVM_PRE_FAULT_MEMORYioctl, but this doesn't work well for our use-case, as it requires the vCPU to be paused.What about if/when KVM Userfault[*] comes along? I.e. pre-fault memory when the vCPU exits to userspace.
My understanding is that Firecracker performs its userfaultfd operations in a separate process. KVM Userfault doesn't break this, it just means Firecracker would need a way to signal the other process when we get KVM_EXIT_USERFAULTs. Firecracker might be using async PF; using KVM Userfault basically disables async PF.
Quoted 4 lines
If KVM Userfault +
KVM_PRE_FAULT_MEMORYis viable for Firecracker and provides the desired performance, I would strongly prefer that approach over KVM doing its own prefetching. That way other architectures don't need to implement prefetching as well (arm64 already supportsKVM_PRE_FAULT_MEMORY, and RISC-V has patches posted).
I also really like this approach if it is workable for Firecracker.
Quoted 2 lines
I'm aiming to get back to that series "soon", this would simply be more motivation to get KVM Userfault landed.
Yay!
Quoted 2 lines
Quoted 2 lines · James Houghton
Firecracker might be using async PF; using KVM Userfault basically disables async PF.
I've been working on exactly this series [1] on top of your KVM Userfault series that turns userfault exits into async PFs, with userspace signalling completion either via ioctl or this idea of an exitless variant which skips the KVM_RUN round-trip per fault which has some exciting numbers!
It's running in our CI at the moment, and hoping to publish this as an RFC soon :)
Signature · 2 lines
Thanks, Jack
Quoted 9 lines · Marco Marangoni
The TDP MMU installs one SPTE per fault, so faulting in guest memory via userfaultfd (e.g. snapshot restore) without hugetlbfs costs a VM-exit per 4KiB page. Mirror the shadow MMU's prefetch: after a 4KiB fault, resolve the forward run of host-present pages in the faulting leaf table (one guest 2MiB region) with one non-blocking GUP and fill the empty SPTEs. When userfaultfd populates in large chunks this maps up to 511 neighbours per fault, cutting EPT violations up to 512x.
Quoted 3 lines
Touching 128MiB backed by userfaultfd with 2MiB
UFFD_COPYchunks:
c8i.metal-48xl 129.1 -> 100.3 ms (-22%), nested 495.8 -> 121.5 ms (-75%). With no batching (one copy per fault) there is a ~2-4% regression.
Hi Marco!
I think part of the problem in this case is that UFFDIO_COPY will install 4K pages (IIRC), I think a more natural way to fix this problem is to:
1. MADV_COLLAPSE after doing UFFDIO_COPY.
2. Make UFFDIO_COPY install PMDs when it is able to do so.
These don't solve the exact same problem, but really userfaultfd should already try to install PMDs when it can (#2). If we have #2, #1 is mostly a no-op.
What do you think?
Thanks for the replies!
Quoted 8 lines · Sean Christopherson
Quoted 3 lines · Marco Marangoni
It's worth mentioning that I also evaluated using the existing
KVM_PRE_FAULT_MEMORYioctl, but this doesn't work well for our use-case, as it requires the vCPU to be paused.What about if/when KVM Userfault[*] comes along? I.e. pre-fault memory when the vCPU exits to userspace.
KVM Userfault + KVM_PRE_FAULT_MEMORY is a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching. I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.
Quoted 4 lines · James Houghton
I think part of the problem in this case is that
UFFDIO_COPYwill install 4K pages (IIRC), I think a more natural way to fix this problem is to:
Quoted 8 lines
1.
MADV_COLLAPSEafter doingUFFDIO_COPY.
2. MakeUFFDIO_COPYinstall PMDs when it is able to do so.These don't solve the exact same problem, but really userfaultfd should already try to install PMDs when it can (#2). If we have #2, #1 is mostly a no-op.
What do you think?
Directly installing PMDs after an UFFD_COPY is something I already investigated. I didn't mention it originally, since it touches exclusively the MM module.
For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my proposal (which "only" improves by 75% and 22% respectively). However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks). I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.
Quoted 10 lines · Marco
Thanks for the replies!
Quoted 6 lines · Sean Christopherson
Quoted 3 lines · Marco Marangoni
It's worth mentioning that I also evaluated using the existing
KVM_PRE_FAULT_MEMORYioctl, but this doesn't work well for our use-case, as it requires the vCPU to be paused.
Quoted 5 lines
Quoted 2 lines
What about if/when KVM Userfault[*] comes along? I.e. pre-fault memory when the vCPU exits to userspace.
KVM Userfault +
KVM_PRE_FAULT_MEMORYis a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching. I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.
What if you used KVM_PRE_FAULT_MEMORY without KVM Userfault?
If you want to avoid pausing a vCPU, what if you made another vCPU (KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I haven't really looked into that before... I'm guessing there's something fundamentally wrong with this approach.
If this doesn't work (and a VM-scoped KVM_PRE_FAULT_MEMORY doesn't make sense either), then perhaps the TDP MMU prefetching logic makes sense.
Quoted 8 lines · James Houghton
Quoted 7 lines
I think part of the problem in this case is that
UFFDIO_COPYwill install 4K pages (IIRC), I think a more natural way to fix this problem is to:1.
MADV_COLLAPSEafter doingUFFDIO_COPY.
2. MakeUFFDIO_COPYinstall PMDs when it is able to do so.
Quoted 8 lines
Quoted 5 lines
These don't solve the exact same problem, but really userfaultfd should already try to install PMDs when it can (#2). If we have #2, #1 is mostly a no-op.
What do you think?
Directly installing PMDs after an
UFFD_COPYis something I already investigated. I didn't mention it originally, since it touches exclusively the MM module.
For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my proposal (which "only" improves by 75% and 22% respectively).
Did you or one of your colleagues ever post it on list? I'm curious to see it. :)
Quoted 2 lines
However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks). I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.
I see. So we really are dealing with 4K mappings.
In which case, prefetching at 2M does seem kind of arbitrary, which makes me even more in favor of this being mostly userspace-driven.
Quoted 20 lines · James Houghton
Quoted 10 lines · Marco
Thanks for the replies!
Quoted 6 lines · Sean Christopherson
Quoted 3 lines · Marco Marangoni
It's worth mentioning that I also evaluated using the existing
KVM_PRE_FAULT_MEMORYioctl, but this doesn't work well for our use-case, as it requires the vCPU to be paused.Quoted 5 lines
Quoted 2 lines
What about if/when KVM Userfault[*] comes along? I.e. pre-fault memory when the vCPU exits to userspace.
KVM Userfault +
KVM_PRE_FAULT_MEMORYis a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching. I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.What if you used
KVM_PRE_FAULT_MEMORYwithout KVM Userfault?
Quoted 8 lines
If you want to avoid pausing a vCPU, what if you made another vCPU (
KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I haven't really looked into that before... I'm guessing there's something fundamentally wrong with this approach.If this doesn't work (and a VM-scoped
KVM_PRE_FAULT_MEMORYdoesn't make sense either), then perhaps the TDP MMU prefetching logic makes sense.
~~Unfortunately~~ Fortunately I'll be on Holidays for the next few weeks; I'll try the permanently paused vCPU approach as soon as I can. In principle it could solve the problem without requiring any kernel changes.
Quoted 13 lines · James Houghton
Quoted 13 lines · James Houghton
Quoted 11 lines
I think part of the problem in this case is that
UFFDIO_COPYwill install 4K pages (IIRC), I think a more natural way to fix this problem is to:1.
MADV_COLLAPSEafter doingUFFDIO_COPY.
2. MakeUFFDIO_COPYinstall PMDs when it is able to do so.These don't solve the exact same problem, but really userfaultfd should already try to install PMDs when it can (#2). If we have #2, #1 is mostly a no-op.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
What do you think?
Directly installing PMDs after an
UFFD_COPYis something I already investigated. I didn't mention it originally, since it touches exclusively the MM module.
For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my > proposal (which "only" improves by 75% and 22% respectively).Did you or one of your colleagues ever post it on list? I'm curious to see it. :)
I wanted to tackle the use-case of arbitrary UFFD chunks first, so no, not yet at least.
If you're curious, the proof-of-concept is on https://github.com/marco-marangoni/firecracker/blob/thp-uffd/resources/patches/thp_uffd.patch; it lacks mTHP support and only handles UFFD_COPY.
Quoted 8 lines · James Houghton
Quoted 3 lines
However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks). I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.
I see. So we really are dealing with 4K mappings.
In which case, prefetching at 2M does seem kind of arbitrary, which makes me even more in favor of this being mostly userspace-driven.
Yes, 2MiB is arbitrary.
One of the possible implementation for this prefetching patch is to skip the GUP and create the EPT PT directly from the page table, in which case 2MiB is natural.
Thanks again for the interesting replies, I'll try the KVM_PRE_FAULT_MEMORY with a paused vCPU when I'm back from PTO.
Marco
Sean Christopherson
maintainerQuoted 11 lines · James Houghton
Quoted 10 lines · Marco
Thanks for the replies!
Quoted 6 lines · Sean Christopherson
Quoted 3 lines · Marco Marangoni
It's worth mentioning that I also evaluated using the existing
KVM_PRE_FAULT_MEMORYioctl, but this doesn't work well for our use-case, as it requires the vCPU to be paused.
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
What about if/when KVM Userfault[*] comes along? I.e. pre-fault memory when the vCPU exits to userspace.
KVM Userfault +
KVM_PRE_FAULT_MEMORYis a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching. I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.
Hrm, right. Although _if_ we can figure out a clever way to allow prefaulting without a vCPU (or at least, a real vCPU), that would allow both to coexist. But at that point I'm probably just being extremely stubborn. :-)
Quoted 6 lines
What if you used
KVM_PRE_FAULT_MEMORYwithout KVM Userfault?If you want to avoid pausing a vCPU, what if you made another vCPU (
KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I haven't really looked into that before... I'm guessing there's something fundamentally wrong with this approach.
Yeah, shoving a vCPU into the VM that shouldn't exist probably won't end well. E.g. on x86, the vCPU would kinda sorta be visible/reachable by the guest as the fake vCPU would respond to IRQs and whatnot.
Quoted 35 lines
If this doesn't work (and a VM-scoped
KVM_PRE_FAULT_MEMORYdoesn't make sense either), then perhaps the TDP MMU prefetching logic makes sense.Quoted 12 lines · James Houghton
Quoted 11 lines
I think part of the problem in this case is that
UFFDIO_COPYwill install 4K pages (IIRC), I think a more natural way to fix this problem is to:1.
MADV_COLLAPSEafter doingUFFDIO_COPY.
2. MakeUFFDIO_COPYinstall PMDs when it is able to do so.These don't solve the exact same problem, but really userfaultfd should already try to install PMDs when it can (#2). If we have #2, #1 is mostly a no-op.
Quoted 8 lines
Quoted 1 line
What do you think?
Directly installing PMDs after an
UFFD_COPYis something I already investigated. I didn't mention it originally, since it touches exclusively the MM module. For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my proposal (which "only" improves by 75% and 22% respectively).Did you or one of your colleagues ever post it on list? I'm curious to see it. :)
Quoted 6 lines
However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks). I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.
Quoted 4 lines
I see. So we really are dealing with 4K mappings.
In which case, prefetching at 2M does seem kind of arbitrary, which makes me even more in favor of this being mostly userspace-driven.
Letting userspace control the prefetch size is easy enough though, e.g. via a module param or CAP.
I'm not opposed to the idea of KVM driving prefetching/prefaulting, I just don't want to add a fourth version: indirect MMU, direct MMU, KVM_PRE_FAULT_MEMORY, and now the TDP MMU. Now that KVM_PRE_FAULT_MEMORY is a thing, I don't see any reason why we can't use the core logic for KVM's own prefaulting. At that point, using the prefault flow would let us drop prefetching for direct shadow MMUs, i.e. would be a net reduction in code and complexity.
Somewhat off the cuff and *very* lightly tested, but this seems to do what I want. If it provides comparable performance, I'll write a changelog (or two? e.g. to have direct MMUs switch in a separate patch), and let Sashiko and other bots rip apart my idea.
Note! This has a hard dependency on in-flight prefaulting fixes[*]. Without those, prefaulting will hang the vCPU if the root is invalidated. [*] https://lore.kernel.org/all/[email protected]
Note #2! The below deliberately ignores A/D-disabled MMUs. I can't think of any reason why it matters whether or not KVM can precisely detect accessed SPTEs, all of the aging stuff is already extremely fuzzy.
arch/x86/kvm/mmu/mmu.c +35 -64
@@ -118,6 +118,9 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(tdp_mmu_enabled);bool __read_mostly eager_page_split = true;module_param (eager_page_split, bool,0644 ); +unsigned int __read_mostly auto_prefault_nr_pages =KVM_PAGES_PER_HPAGE (PG_LEVEL_2M ); +module_param (auto_prefault_nr_pages, uint,0644 ); +static int max_huge_page_level__read_mostly ;static int tdp_root_level__read_mostly ;static int max_tdp_level__read_mostly ;
@@ -3205,69 +3208,6 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,return true; } -static bool direct_pte_prefetch_many (struct kvm_vcpu *vcpu, -struct kvm_mmu_page *sp, -u64 *start,u64 *end) -{ -gfn_t gfn =kvm_mmu_page_get_gfn (sp,spte_index (start)); -unsigned int access = sp->role.access; - -return kvm_mmu_prefetch_sptes (vcpu, gfn, start, end - start, access); -} - -static void __direct_pte_prefetch (struct kvm_vcpu *vcpu, -struct kvm_mmu_page *sp,u64 *sptep) -{ -u64 *spte, *start =NULL ; -int i; - -WARN_ON_ONCE (!sp->role.direct); - - i =spte_index (sptep) & ~(PTE_PREFETCH_NUM -1 ); - spte = sp->spt + i; - -for (i =0 ; i <PTE_PREFETCH_NUM ; i++, spte++) { -if (is_shadow_present_pte (*spte) || spte == sptep) { -if (!start) -continue ; -if (!direct_pte_prefetch_many (vcpu, sp, start, spte)) -return ; - - start =NULL ; - }else if (!start) - start = spte; - } -if (start) -direct_pte_prefetch_many (vcpu, sp, start, spte); -} - -static void direct_pte_prefetch (struct kvm_vcpu *vcpu,u64 *sptep) -{ -struct kvm_mmu_page *sp; - - sp =sptep_to_sp (sptep); - -/* -* Without accessed bits, there's no way to distinguish between -* actually accessed translations and prefetched, so disable pte -* prefetch if accessed bits aren't available. -*/ -if (sp_ad_disabled (sp)) -return ; - -if (sp->role.level >PG_LEVEL_4K ) -return ; - -/* -* If addresses are being invalidated, skip prefetching to avoid -* accidentally prefetching those addresses. -*/ -if (unlikely (vcpu->kvm->mmu_invalidate_in_progress)) -return ; - -__direct_pte_prefetch (vcpu, sp, sptep); -} -/* * Lookup the mapping level for @gfn in the current mm. *
@@ -6580,11 +6520,40 @@ static int kvm_mmu_write_protect_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,return RET_PF_EMULATE ; } +static void kvm_mmu_auto_prefault (struct kvm_vcpu *vcpu,gpa_t start, +u64 error_code,u8 level) +{ +gfn_t nr_pages =READ_ONCE (auto_prefault_nr_pages); +gfn_t i, o =KVM_PAGES_PER_HPAGE (level); +int nr_pages_msb; + +if (unlikely (error_code &PFERR_RSVD_MASK )) +return ; + + nr_pages =min (nr_pages,KVM_PAGES_PER_HPAGE (PG_LEVEL_1G )); + nr_pages_msb =find_last_bit ((unsigned long *)&nr_pages,sizeof (nr_pages)); + + start =ALIGN_DOWN (start,gfn_to_gpa (BIT_ULL (nr_pages_msb))); + +for (i =KVM_PAGES_PER_HPAGE (level); i < nr_pages; i +=KVM_PAGES_PER_HPAGE (level)) { +gpa_t gpa = start +gfn_to_gpa (i); + +if (gpa < start ||gpa_to_gfn (gpa) >kvm_mmu_max_gfn ()) +return ; + +if (kvm_tdp_page_prefault (vcpu, gpa, error_code, &level)) +return ; + } + +pr_warn_ratelimited ("Prefaulted ~%llu pages at %llx\n" , i - o, start); +} + int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,void *insn,int insn_len) {int r, emulation_type =EMULTYPE_PF ;bool direct = vcpu->arch.mmu->root_role.direct; +u8 level;if (WARN_ON_ONCE (!VALID_PAGE (vcpu->arch.mmu->root.hpa)))return RET_PF_RETRY ;
@@ -6617,7 +6586,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err vcpu->stat.pf_taken++; r =kvm_mmu_do_page_fault (vcpu, cr2_or_gpa, error_code, false, - &emulation_type,NULL ); + &emulation_type, &level);if (KVM_BUG_ON (r ==RET_PF_INVALID , vcpu->kvm))return -EIO ; }
@@ -6628,6 +6597,8 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errif (r ==RET_PF_WRITE_PROTECTED ) r =kvm_mmu_write_protect_fault (vcpu, cr2_or_gpa, error_code, &emulation_type); +else if (r ==RET_PF_FIXED ) +kvm_mmu_auto_prefault (vcpu, cr2_or_gpa, error_code, level);if (r ==RET_PF_FIXED ) vcpu->stat.pf_fixed++;
arch/x86/kvm/mmu/paging_tmpl.h +1 -4
@@ -617,7 +617,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw, sp =sptep_to_sp (sptep); -if (sp->role.level >PG_LEVEL_4K ) +if (sp->role.level >PG_LEVEL_4K || sp->role.direct)return ;/*
@@ -627,9 +627,6 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,if (unlikely (vcpu->kvm->mmu_invalidate_in_progress))return ; -if (sp->role.direct) -return __direct_pte_prefetch (vcpu, sp, sptep); - i =spte_index (sptep) & ~(PTE_PREFETCH_NUM -1 ); spte = sp->spt + i;
Sean Christopherson
maintainerQuoted 5 lines · Sean Christopherson
Somewhat off the cuff and *very* lightly tested, but this seems to do what I want. If it provides comparable performance, I'll write a changelog (or two? e.g. to have direct MMUs switch in a separate patch), and let Sashiko and other bots rip apart my idea.
Quoted 7 lines
Note! This has a hard dependency on in-flight prefaulting fixes[*]. Without those, prefaulting will hang the vCPU if the root is invalidated. [*] https://lore.kernel.org/all/[email protected]
Note #2! The below deliberately ignores A/D-disabled MMUs. I can't think of any reason why it matters whether or not KVM can precisely detect accessed SPTEs, all of the aging stuff is already extremely fuzzy.
And of course I posted an untested version (I ripped out the direct MMU prefetching as an afterthough, and dropped a printk). This version should actually compile.
arch/x86/kvm/mmu/mmu.c +36 -72
@@ -118,6 +118,9 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(tdp_mmu_enabled);bool __read_mostly eager_page_split = true;module_param (eager_page_split, bool,0644 ); +unsigned int __read_mostly auto_prefault_nr_pages =KVM_PAGES_PER_HPAGE (PG_LEVEL_2M ); +module_param (auto_prefault_nr_pages, uint,0644 ); +static int max_huge_page_level__read_mostly ;static int tdp_root_level__read_mostly ;static int max_tdp_level__read_mostly ;
@@ -3205,69 +3208,6 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,return true; } -static bool direct_pte_prefetch_many (struct kvm_vcpu *vcpu, -struct kvm_mmu_page *sp, -u64 *start,u64 *end) -{ -gfn_t gfn =kvm_mmu_page_get_gfn (sp,spte_index (start)); -unsigned int access = sp->role.access; - -return kvm_mmu_prefetch_sptes (vcpu, gfn, start, end - start, access); -} - -static void __direct_pte_prefetch (struct kvm_vcpu *vcpu, -struct kvm_mmu_page *sp,u64 *sptep) -{ -u64 *spte, *start =NULL ; -int i; - -WARN_ON_ONCE (!sp->role.direct); - - i =spte_index (sptep) & ~(PTE_PREFETCH_NUM -1 ); - spte = sp->spt + i; - -for (i =0 ; i <PTE_PREFETCH_NUM ; i++, spte++) { -if (is_shadow_present_pte (*spte) || spte == sptep) { -if (!start) -continue ; -if (!direct_pte_prefetch_many (vcpu, sp, start, spte)) -return ; - - start =NULL ; - }else if (!start) - start = spte; - } -if (start) -direct_pte_prefetch_many (vcpu, sp, start, spte); -} - -static void direct_pte_prefetch (struct kvm_vcpu *vcpu,u64 *sptep) -{ -struct kvm_mmu_page *sp; - - sp =sptep_to_sp (sptep); - -/* -* Without accessed bits, there's no way to distinguish between -* actually accessed translations and prefetched, so disable pte -* prefetch if accessed bits aren't available. -*/ -if (sp_ad_disabled (sp)) -return ; - -if (sp->role.level >PG_LEVEL_4K ) -return ; - -/* -* If addresses are being invalidated, skip prefetching to avoid -* accidentally prefetching those addresses. -*/ -if (unlikely (vcpu->kvm->mmu_invalidate_in_progress)) -return ; - -__direct_pte_prefetch (vcpu, sp, sptep); -} -/* * Lookup the mapping level for @gfn in the current mm. *
@@ -3502,8 +3442,8 @@ static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) {struct kvm_shadow_walk_iterator it;struct kvm_mmu_page *sp; -int ret, access;gfn_t base_gfn = fault->gfn; +int access;kvm_mmu_hugepage_adjust (vcpu, fault);
@@ -3534,13 +3474,8 @@ static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)if (WARN_ON_ONCE (it.level != fault->goal_level))return -EFAULT ; - ret =mmu_set_spte (vcpu, fault->slot, it.sptep, access, - base_gfn, fault->pfn, fault); -if (ret ==RET_PF_SPURIOUS ) -return ret; - -direct_pte_prefetch (vcpu, it.sptep); -return ret; +return mmu_set_spte (vcpu, fault->slot, it.sptep, access, base_gfn, + fault->pfn, fault); }static void kvm_send_hwpoison_signal (struct kvm_memory_slot *slot,gfn_t gfn)
@@ -6580,11 +6515,38 @@ static int kvm_mmu_write_protect_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,return RET_PF_EMULATE ; } +static void kvm_mmu_auto_prefault (struct kvm_vcpu *vcpu,gpa_t start, +u64 error_code,u8 level) +{ +gfn_t nr_pages =READ_ONCE (auto_prefault_nr_pages); +int nr_pages_msb; +gfn_t i; + +if (unlikely (error_code &PFERR_RSVD_MASK )) +return ; + + nr_pages =min (nr_pages,KVM_PAGES_PER_HPAGE (PG_LEVEL_1G )); + nr_pages_msb =find_last_bit ((unsigned long *)&nr_pages,sizeof (nr_pages)); + + start =ALIGN_DOWN (start,gfn_to_gpa (BIT_ULL (nr_pages_msb))); + +for (i =KVM_PAGES_PER_HPAGE (level); i < nr_pages; i +=KVM_PAGES_PER_HPAGE (level)) { +gpa_t gpa = start +gfn_to_gpa (i); + +if (gpa < start ||gpa_to_gfn (gpa) >kvm_mmu_max_gfn ()) +return ; + +if (kvm_tdp_page_prefault (vcpu, gpa, error_code, &level)) +return ; + } +} + int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,void *insn,int insn_len) {int r, emulation_type =EMULTYPE_PF ;bool direct = vcpu->arch.mmu->root_role.direct; +u8 level;if (WARN_ON_ONCE (!VALID_PAGE (vcpu->arch.mmu->root.hpa)))return RET_PF_RETRY ;
@@ -6617,7 +6579,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err vcpu->stat.pf_taken++; r =kvm_mmu_do_page_fault (vcpu, cr2_or_gpa, error_code, false, - &emulation_type,NULL ); + &emulation_type, &level);if (KVM_BUG_ON (r ==RET_PF_INVALID , vcpu->kvm))return -EIO ; }
@@ -6628,6 +6590,8 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errif (r ==RET_PF_WRITE_PROTECTED ) r =kvm_mmu_write_protect_fault (vcpu, cr2_or_gpa, error_code, &emulation_type); +else if (r ==RET_PF_FIXED ) +kvm_mmu_auto_prefault (vcpu, cr2_or_gpa, error_code, level);if (r ==RET_PF_FIXED ) vcpu->stat.pf_fixed++;
arch/x86/kvm/mmu/paging_tmpl.h +1 -4
@@ -617,7 +617,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw, sp =sptep_to_sp (sptep); -if (sp->role.level >PG_LEVEL_4K ) +if (sp->role.level >PG_LEVEL_4K || sp->role.direct)return ;/*
@@ -627,9 +627,6 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,if (unlikely (vcpu->kvm->mmu_invalidate_in_progress))return ; -if (sp->role.direct) -return __direct_pte_prefetch (vcpu, sp, sptep); - i =spte_index (sptep) & ~(PTE_PREFETCH_NUM -1 ); spte = sp->spt + i;
Sean Christopherson
maintainerQuoted 10 lines · Sean Christopherson
Quoted 9 lines · Sean Christopherson
Somewhat off the cuff and *very* lightly tested, but this seems to do what I want. If it provides comparable performance, I'll write a changelog (or two? e.g. to have direct MMUs switch in a separate patch), and let Sashiko and other bots rip apart my idea.
Note! This has a hard dependency on in-flight prefaulting fixes[*]. Without those, prefaulting will hang the vCPU if the root is invalidated. [*] https://lore.kernel.org/all/[email protected]
Quoted 6 lines
Quoted 3 lines
Note #2! The below deliberately ignores A/D-disabled MMUs. I can't think of any reason why it matters whether or not KVM can precisely detect accessed SPTEs, all of the aging stuff is already extremely fuzzy.
And of course I posted an untested version (I ripped out the direct MMU prefetching as an afterthough, and dropped a printk). This version should actually compile.
This breaks dirty_log_test and dirty_log_page_splitting_test, because KVM creates writable SPTEs in direct MMUs whenever possible. Because nothing can be simple, the below in turn breaks pre_fault_memory_test, but I suspect that's a test flaw.
Note, this would also short-circuit async #PF completion when dirty logging is enabled. I think that's a good thing? If not, we could teach kvm_mmu_do_page_fault() to differentiate between async #PF and unprompted prefetching.
arch/x86/kvm/mmu/mmu.c +8 -14
@@ -874,19 +874,11 @@ static void unaccount_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp)untrack_possible_nx_huge_page (kvm, sp,KVM_SHADOW_MMU ); } -static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap (struct kvm_vcpu *vcpu, -gfn_t gfn, -bool no_dirty_log) +static bool kvm_is_memslot_usable_for_prefetch (struct kvm_memory_slot *slot, +unsigned int access) { -struct kvm_memory_slot *slot; - - slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); -if (!slot || slot->flags &KVM_MEMSLOT_INVALID ) -return NULL ; -if (no_dirty_log &&kvm_slot_dirty_track_enabled (slot)) -return NULL ; - -return slot; +return slot && !(slot->flags &KVM_MEMSLOT_INVALID ) && + (!(access &ACC_WRITE_MASK ) || !kvm_slot_dirty_track_enabled (slot)); }/*
@@ -3181,8 +3173,8 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,if (WARN_ON_ONCE (nr_pages >PTE_PREFETCH_NUM ))return false; - slot =gfn_to_memslot_dirty_bitmap (vcpu, gfn, access &ACC_WRITE_MASK ); -if (!slot) + slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); +if (!kvm_is_memslot_usable_for_prefetch (slot, access))return false; nr_pages =kvm_prefetch_pages (slot, gfn, pages, nr_pages);
@@ -4946,6 +4938,8 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,*/ fault.gfn =gpa_to_gfn (fault.addr) & ~kvm_gfn_direct_bits (vcpu->kvm); fault.slot =kvm_vcpu_gfn_to_memslot (vcpu, fault.gfn); +if (prefetch && !kvm_is_memslot_usable_for_prefetch (fault.slot,ACC_ALL )) +return RET_PF_WRITE_PROTECTED ; }/*
Quoted 4 lines · Sean Christopherson
If it provides comparable performance, I'll write a changelog (or two? e.g. to have direct MMUs switch in a separate patch), and let Sashiko and other bots rip apart my idea.
Wow, thanks so much for such a quick patch.
I gave it a shot in my test setup. While it's a bit slower than my patch, it's still ~69% faster on nested (rather than ~75%), which is amazing considering how much simpler it is than my proposal.