KVM: pfncache: Add guest_memfd support to pfncache
v4 cover KVM: pfncache: Add guest_memfd support to pfncache
[ based on 6.18 with [1] ]
This patch series adds guest_memfd support to gfn_to_pfn_cache (a.k.a. pfncache). (This is still labelled RFC since its dependency [1] has not yet been merged.)
=== Problem Statement ===
pfncache does not work with guest_memfd. pfncaches resolve PFNs via hva_to_pfn(), which requires a userspace mapping and relies on GUP. This does not work for guest_memfd in the following two ways:
* guest_memfd created without MMAP flag does not have a userspace mapping due to the nature of private memory.
* guest_memfd created with NO_DIRECT_MAP flag uses an AS_NO_DIRECT_MAP mapping, which is rejected by GUP.
In addition, pfncaches map RAM pages via kmap(), which typically returns an address derived from the direct map. So kmap() cannot be used for NO_DIRECT_MAP guest_memfd. pfncaches require fault-free KHVAs since they can be used from atomic context. Thus, it cannot fall back to access via a userspace mapping like KVM does for other accesses to NO_DIRECT_MAP guest_memfd.
The introduction of guest_memfd support necessitates additional invalidation paths in addition to the existing MMU notifier path: one from guest_memfd invalidation and another from memory attribute updates.
=== Core Approach ===
* Resolve PFNs for guest_memfd-backed GPAs via kvm_gmem_get_pfn().
* Obtain a fault-free KHVA for NO_DIRECT_MAP pages via vmap().
* Hook pfncache invalidation into guest_memfd invalidation (punch hole / release / error handling) as well as into memory attribute updates (switch between shared and private memories).
* Reuse mn_active_invalidate_count to synchronize the new invalidation paths with the existing pfncache retry logic.
=== Design Considerations (Feedback Appreciated) ===
* Reusing mn_active_invalidate_count allows reusing the existing pfncache retry logic as-is and enables invalidating pfncaches without holding mmu_lock from guest_memfd invalidation context. As a side effect, active memslots swap is blocked while mn_active_invalidate_count > 0. To avoid this block, it would be possible to introduce a dedicated counter instead.
* Although both guest_memfd invalidation and memory attribute update are driven by GFN ranges, pfncache invalidation is performed using HVA ranges. GPA-based pfncaches have memslot/GPA context, whereas HVA-based pfncaches do not. Using GFN-based invalidation would miss HVA-based pfncaches.
* The current implementation does not support HVA-based pfncaches for NO_DIRECT_MAP guest_memfd. HVA-based pfncaches do not store memslot/GPA context, so they cannot determine whether the target is gmem-backed and always fall back to GUP (hva_to_pfn()), which fails for NO_DIRECT_MAP pages. Adding a memslot/GPA lookup is possible but would add overhead to all HVA-based pfncache activations and refreshes. At the time of writing, only Xen uses HVA-based pfncaches.
=== Changelog ===
Changes since RFC v3:
- Drop the rename of mn_* invalidate-related fields to generic ones, as suggested by Sean. Keep the mn_ prefix.
- Fix incorrect HVA range computation in pfncache invalidation for guest_memfd and memory attribute update paths. gfn_to_hva_memslot() with gfn_end == slot->base_gfn + slot->npages triggers array_index_nospec() clamping, resulting in an empty range. Use hva_start + (gfn_end - gfn_start) * PAGE_SIZE instead.
- Add selftests that exercise pfncache with guest_memfd-backed memory (NO_DIRECT_MAP and SW_PROTECTED_VM) and verify invalidation paths (punch_hole, private-to-shared conversion, file release).
Changes since RFC v2:
- Drop avoidance of silent kvm-clock activation failure.
- Fix a compile error for kvm_for_each_memslot().
Changes since RFC v1:
- Prevent kvm-clock activation from failing silently.
- Generalize serialization mechanism for invalidation.
- Hook pfncache invalidation into guest_memfd invalidation and memory attribute updates.
RFC v3: https://lore.kernel.org/all/[email protected]/
RFC v2: https://lore.kernel.org/all/[email protected]/
RFC v1: https://lore.kernel.org/all/[email protected]/
[1]: https://lore.kernel.org/all/[email protected]/
Takahiro Itazuri (7):
KVM: pfncache: Resolve PFNs via kvm_gmem_get_pfn() for gmem-backed GPAs
KVM: pfncache: Obtain KHVA via vmap() for gmem with NO_DIRECT_MAP
KVM: Rename invalidate_begin to invalidate_start for consistency
KVM: pfncache: Rename invalidate_start() helper
KVM: pfncache: Invalidate on gmem invalidation and memattr updates
KVM: selftests: Test pfncache with gmem-backed memory
KVM: selftests: Test pfncache invalidation for gmem-backed memory
arch/x86/kvm/mmu/mmu.c | 2 +- include/linux/kvm_host.h | 2 +- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/pfncache_gmem_test.c | 222 ++++++++++++++++++ virt/kvm/guest_memfd.c | 64 ++++- virt/kvm/kvm_main.c | 55 ++++- virt/kvm/kvm_mm.h | 12 +- virt/kvm/pfncache.c | 110 +++++++-- 8 files changed, 427 insertions(+), 41 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86/pfncache_gmem_test.c
Signature · 1 line
2.50.1
RFC PATCH v4 1/7 KVM: pfncache: Resolve PFNs via kvm_gmem_get_pfn() for gmem-backed GPAs
Currently, pfncaches always resolve PFNs via hva_to_pfn(), which requires a userspace mapping and relies on GUP. This does not work for guest_memfd in the following two ways:
* guest_memfd created without MMAP flag does not have a userspace mapping.
* guest_memfd created with NO_DIRECT_MAP flag uses an AS_NO_DIRECT_MAP mapping, which is rejected by GUP.
Resolve PFNs via kvm_gmem_get_pfn() for guest_memfd-backed and GPA-based pfncaches. Otherwise, fall back to the existing hva_to_pfn().
The current implementation does not support HVA-based pfncaches for NO_DIRECT_MAP guest_memfd. HVA-based pfncaches do not store memslot/GPA context, so they cannot determine whether the target is guest_memfd-backed and always fall back to hva_to_pfn(). Adding a memslot/GPA lookup is possibile but would add overhead to all HVA-based pfncache activations and refreshes. At the time of writing, only Xen uses HVA-based pfncaches.
--- virt/kvm/pfncache.c | 66 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 54 insertions(+), 12 deletions(-)
virt/kvm/pfncache.c +54 -12
@@ -152,7 +152,53 @@ static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long mmu_sreturn kvm->mmu_invalidate_seq != mmu_seq; } -static kvm_pfn_t hva_to_pfn_retry (struct gfn_to_pfn_cache *gpc) +/* +* Determine whether a GPA-based pfncache is backed by guest_memfd, i.e. needs +* to be resolved via kvm_gmem_get_pfn() rather than GUP. +* +* The caller holds gpc->refresh_lock, but does not hold gpc->lock nor +* kvm->slots_lock. Reading slot->flags (via kvm_slot_has_gmem() and +* kvm_memslot_is_gmem_only()) is safe because memslot changes bump +* slots->generation, which is detected in kvm_gpc_check(), forcing callers +* to invoke kvm_gpc_refresh(). +* +* Looking up memory attributes (via kvm_mem_is_private()) can race with +* KVM_SET_MEMORY_ATTRIBUTES, which takes kvm->slots_lock to serialize +* writers but doesn't exclude lockless readers. Handling that race is deferred +* to a subsequent commit that wires up pfncache invalidation for gmem events. +*/ +static inline bool gpc_is_gmem_backed (struct gfn_to_pfn_cache *gpc) +{ +lockdep_assert_held (&gpc->refresh_lock); + +/* For HVA-based pfncaches, memslot is NULL */ +return gpc->memslot &&kvm_slot_has_gmem (gpc->memslot) && + (kvm_memslot_is_gmem_only (gpc->memslot) || +kvm_mem_is_private (gpc->kvm,gpa_to_gfn (gpc->gpa))); +} + +static kvm_pfn_t gpc_to_pfn (struct gfn_to_pfn_cache *gpc,struct page **page) +{ +if (gpc_is_gmem_backed (gpc)) { +kvm_pfn_t pfn; + +if (kvm_gmem_get_pfn (gpc->kvm, gpc->memslot, +gpa_to_gfn (gpc->gpa), &pfn, page,NULL )) +return KVM_PFN_ERR_FAULT ; + +return pfn; + } + +return hva_to_pfn (&(struct kvm_follow_pfn ) { + .slot = gpc->memslot, + .gfn =gpa_to_gfn (gpc->gpa), + .flags =FOLL_WRITE , + .hva = gpc->uhva, + .refcounted_page = page, + }); +} + +static kvm_pfn_t gpc_to_pfn_retry (struct gfn_to_pfn_cache *gpc) {/* Note, the new page offset may be different than the old! */ void *old_khva = (void *)PAGE_ALIGN_DOWN ((uintptr_t )gpc->khva);
@@ -161,14 +207,6 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)unsigned long mmu_seq;struct page *page; -struct kvm_follow_pfn kfp = { - .slot = gpc->memslot, - .gfn =gpa_to_gfn (gpc->gpa), - .flags =FOLL_WRITE , - .hva = gpc->uhva, - .refcounted_page = &page, - }; -lockdep_assert_held (&gpc->refresh_lock);lockdep_assert_held_write (&gpc->lock);
@@ -206,7 +244,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)cond_resched (); } - new_pfn =hva_to_pfn (&kfp); + new_pfn =gpc_to_pfn (gpc, &page);if (is_error_noslot_pfn (new_pfn)) goto out_error;
@@ -319,7 +357,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l } } -/* Note: the offset must be correct before calling hva_to_pfn_retry() */ +/* Note: the offset must be correct before calling gpc_to_pfn_retry() */ gpc->uhva += page_offset;/*
@@ -327,7 +365,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l* drop the lock and do the HVA to PFN lookup again. */ if (!gpc->valid || hva_change) { - ret =hva_to_pfn_retry (gpc); + ret =gpc_to_pfn_retry (gpc); }else {/* * If the HVA→PFN mapping was already valid, don't unmap it.
@@ -441,6 +479,10 @@ int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long uhva, unsigif (!access_ok ((void __user *)uhva, len))return -EINVAL ; +/* +* HVA-based caches always resolve PFNs via GUP (hva_to_pfn()), which +* does not work for NO_DIRECT_MAP guest_memfd. +*/ return __kvm_gpc_activate (gpc,INVALID_GPA , uhva, len); }
Signature · 1 line
2.50.1
RFC PATCH v4 2/7 KVM: pfncache: Obtain KHVA via vmap() for gmem with NO_DIRECT_MAP
In reply to Takahiro ItazuriCurrently, pfncaches map RAM pages via kmap(), which typically returns a kernel address derived from the direct map. However, guest_memfd created with GUEST_MEMFD_FLAG_NO_DIRECT_MAP has their direct map removed and uses an AS_NO_DIRECT_MAP mapping. So kmap() cannot be used in this case.
pfncaches can be used from atomic context where page faults cannot be tolerated. Therefore, it cannot fall back to access via a userspace mapping like KVM does for other accesses to NO_DIRECT_MAP guest_memfd.
To obtain a fault-free kernel host virtual address (KHVA), use vmap() for NO_DIRECT_MAP pages. Since gpc_map() is the sole producer of KHVA for pfncaches and only vmap() returns a vmalloc address, gpc_unmap() can reliably pair vunmap() using is_vmalloc_addr().
Although vm_map_ram() could be faster than vmap(), mixing short-lived and long-lived vm_map_ram() can lead to fragmentation. For this reason, vm_map_ram() is recommended only for short-lived ones. Since pfncaches typically have a lifetime comparable to that of the VM, vm_map_ram() is deliberately not used here.
pfncaches are not dynamically allocated but are statically allocated on a per-VM and per-vCPU basis. For a normal VM (i.e. non-Xen), there is one pfncache per vCPU. For a Xen VM, there is one per-VM pfncache and five per-vCPU pfncaches. Given the maximum of 1024 vCPUs, a normal VM can have up to 1024 pfncaches, consuming 4 MB of virtual address space. A Xen VM can have up to 5121 pfncaches, consuming approximately 20 MB of virtual address space. Although the vmalloc area is limited on 32-bit systems, it should be large enough and typically tens of TB on 64-bit systems (e.g. 32 TB for 4-level paging and 12800 TB for 5-level paging on x86_64). If virtual address space exhaustion becomes a concern, migration to an mm-local region could be considered in the future. Note that vmap() and vm_map_ram() only create virtual mappings to existing pages; they do not allocate new physical pages.
--- virt/kvm/pfncache.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-)
virt/kvm/pfncache.c +28 -5
@@ -16,6 +16,7 @@#include <linux/highmem.h> #include <linux/module.h> #include <linux/errno.h> +#include <linux/pagemap.h> #include "kvm_mm.h"
@@ -98,8 +99,19 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)static void *gpc_map (kvm_pfn_t pfn) { -if (pfn_valid (pfn)) -return kmap (pfn_to_page (pfn)); +if (pfn_valid (pfn)) { +struct page *page =pfn_to_page (pfn); +struct page *head =compound_head (page); +struct address_space *mapping =READ_ONCE (head->mapping); + +if (mapping &&mapping_no_direct_map (mapping)) { +struct page *pages[] = { page }; + +return vmap (pages,1 ,VM_MAP ,PAGE_KERNEL ); + } + +return kmap (page); + }#ifdef CONFIG_HAS_IOMEM return memremap (pfn_to_hpa (pfn),PAGE_SIZE ,MEMREMAP_WB );
@@ -115,7 +127,15 @@ static void gpc_unmap(kvm_pfn_t pfn, void *khva)return ;if (pfn_valid (pfn)) { -kunmap (pfn_to_page (pfn)); +/* +* For valid PFNs, gpc_map() returns either a kmap() address +* (non-vmalloc) or a vmap() address (vmalloc). +*/ +if (is_vmalloc_addr (khva)) +vunmap (khva); +else +kunmap (pfn_to_page (pfn)); +return ; }
@@ -250,8 +270,11 @@ static kvm_pfn_t gpc_to_pfn_retry(struct gfn_to_pfn_cache *gpc)/* * Obtain a new kernel mapping if KVM itself will access the -* pfn. Note, kmap() and memremap() can both sleep, so this -* too must be done outside of gpc->lock! +* pfn. Note, kmap(), vmap() and memremap() can all sleep, so +* this too must be done outside of gpc->lock! +* Note that even though gpc->lock is dropped, it's still fine +* to read gpc->pfn and other fields because gpc->refresh_lock +* mutex prevents them from being updated. */ if (new_pfn == gpc->pfn) new_khva = old_khva;
Signature · 1 line
2.50.1
RFC PATCH v4 3/7 KVM: Rename invalidate_begin to invalidate_start for consistency
In reply to Takahiro ItazuriRename kvm_mmu_invalidate_begin() to kvm_mmu_invalidate_start() to align with mmu_notifier_ops.invalidate_range_start(), which is the callback that ultimately drives KVM's MMU invalidation.
While the naming within KVM itself is a close split between "_begin" and "_start", conforming to the mmu_notifier_ops naming is the right call since invalidate_range_start() is the external API that KVM hooks into.
$ git grep -E "invalidate(_range)?_begin" **/kvm | wc -l 11 $ git grep -E "invalidate(_range)?_start" **/kvm | wc -l 16
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 2 +- include/linux/kvm_host.h | 2 +- virt/kvm/guest_memfd.c | 14 +++++++------- virt/kvm/kvm_main.c | 6 +++--- 4 files changed, 12 insertions(+), 12 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -6859,7 +6859,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)write_lock (&kvm->mmu_lock); -kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_start (kvm);kvm_mmu_invalidate_range_add (kvm, gfn_start, gfn_end);
include/linux/kvm_host.h +1 -1
@@ -1566,7 +1566,7 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);void *kvm_mmu_memory_cache_alloc (struct kvm_mmu_memory_cache *mc);#endif -void kvm_mmu_invalidate_begin (struct kvm *kvm); +void kvm_mmu_invalidate_start (struct kvm *kvm);void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end);void kvm_mmu_invalidate_end (struct kvm *kvm);bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);
virt/kvm/guest_memfd.c +7 -7
@@ -206,7 +206,7 @@ static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inreturn KVM_FILTER_PRIVATE ; } -static void __kvm_gmem_invalidate_begin (struct gmem_file *f,pgoff_t start, +static void __kvm_gmem_invalidate_start (struct gmem_file *f,pgoff_t start,pgoff_t end,enum kvm_gfn_range_filter attr_filter) {
@@ -230,7 +230,7 @@ static void __kvm_gmem_invalidate_begin(struct gmem_file *f, pgoff_t start, found_memslot = true;KVM_MMU_LOCK (kvm); -kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_start (kvm); } flush |=kvm_mmu_unmap_gfn_range (kvm, &gfn_range);
@@ -243,7 +243,7 @@ static void __kvm_gmem_invalidate_begin(struct gmem_file *f, pgoff_t start,KVM_MMU_UNLOCK (kvm); } -static void kvm_gmem_invalidate_begin (struct inode *inode,pgoff_t start, +static void kvm_gmem_invalidate_start (struct inode *inode,pgoff_t start,pgoff_t end) {enum kvm_gfn_range_filter attr_filter;
@@ -252,7 +252,7 @@ static void kvm_gmem_invalidate_begin(struct inode *inode, pgoff_t start, attr_filter =kvm_gmem_get_invalidate_filter (inode);kvm_gmem_for_each_file (f, inode->i_mapping) -__kvm_gmem_invalidate_begin (f, start, end, attr_filter); +__kvm_gmem_invalidate_start (f, start, end, attr_filter); } static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,
@@ -287,7 +287,7 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)*/ filemap_invalidate_lock (inode->i_mapping); -kvm_gmem_invalidate_begin (inode, start, end); +kvm_gmem_invalidate_start (inode, start, end);truncate_inode_pages_range (inode->i_mapping, offset, offset + len -1 );
@@ -401,7 +401,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)* Zap all SPTEs pointed at by this file. Do not free the backing * memory, as its lifetime is associated with the inode, not the file. */ -__kvm_gmem_invalidate_begin (f,0 ,-1ul , +__kvm_gmem_invalidate_start (f,0 ,-1ul ,kvm_gmem_get_invalidate_filter (inode));__kvm_gmem_invalidate_end (f,0 ,-1ul );
@@ -582,7 +582,7 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol start = folio->index; end = start +folio_nr_pages (folio); -kvm_gmem_invalidate_begin (mapping->host, start, end); +kvm_gmem_invalidate_start (mapping->host, start, end);/* * Do not truncate the range, what action is taken in response to the
virt/kvm/kvm_main.c +3 -3
@@ -678,7 +678,7 @@ static __always_inline int kvm_age_hva_range_no_flush(struct mmu_notifier *mn,return kvm_age_hva_range (mn, start, end, handler, false); } -void kvm_mmu_invalidate_begin (struct kvm *kvm) +void kvm_mmu_invalidate_start (struct kvm *kvm) {lockdep_assert_held_write (&kvm->mmu_lock);/*
@@ -734,7 +734,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, .start = range->start, .end = range->end, .handler = kvm_mmu_unmap_gfn_range, - .on_lock = kvm_mmu_invalidate_begin, + .on_lock = kvm_mmu_invalidate_start, .flush_on_ret = true, .may_block = mmu_notifier_range_blockable(range), };
@@ -2571,7 +2571,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, .end = end, .arg.attributes = attributes, .handler = kvm_pre_set_memory_attributes, - .on_lock = kvm_mmu_invalidate_begin, + .on_lock = kvm_mmu_invalidate_start, .flush_on_ret = true, .may_block = true, };
Signature · 1 line
2.50.1
RFC PATCH v4 4/7 KVM: pfncache: Rename invalidate_start() helper
In reply to Takahiro ItazuriRename gfn_to_pfn_cache_invalidate_start() to gpc_invalidate_hva_range_start() to explicitly indicate that it takes a range of HVA range.
No functional changes intended.
--- virt/kvm/kvm_main.c | 2 +- virt/kvm/kvm_mm.h | 12 ++++++------ virt/kvm/pfncache.c | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-)
virt/kvm/kvm_main.c +1 -1
@@ -763,7 +763,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,* mn_active_invalidate_count (see above) instead of * mmu_invalidate_in_progress. */ -gfn_to_pfn_cache_invalidate_start (kvm, range->start, range->end); +gpc_invalidate_hva_range_start (kvm, range->start, range->end);/* * If one or more memslots were found and thus zapped, notify arch code
virt/kvm/kvm_mm.h +6 -6
@@ -56,13 +56,13 @@ struct kvm_follow_pfn {kvm_pfn_t hva_to_pfn (struct kvm_follow_pfn *kfp);#ifdef CONFIG_HAVE_KVM_PFNCACHE -void gfn_to_pfn_cache_invalidate_start (struct kvm *kvm, -unsigned long start, -unsigned long end); +void gpc_invalidate_hva_range_start (struct kvm *kvm, +unsigned long start, +unsigned long end);#else -static inline void gfn_to_pfn_cache_invalidate_start (struct kvm *kvm, -unsigned long start, -unsigned long end) +static inline void gpc_invalidate_hva_range_start (struct kvm *kvm, +unsigned long start, +unsigned long end) { }#endif /* HAVE_KVM_PFNCACHE */
virt/kvm/pfncache.c +2 -2
@@ -23,8 +23,8 @@/* * MMU notifier 'invalidate_range_start' hook. */ -void gfn_to_pfn_cache_invalidate_start (struct kvm *kvm,unsigned long start, -unsigned long end) +void gpc_invalidate_hva_range_start (struct kvm *kvm,unsigned long start, +unsigned long end) {struct gfn_to_pfn_cache *gpc;
Signature · 1 line
2.50.1
Re: [RFC PATCH v4 4/7] KVM: pfncache: Rename invalidate_start() helper
Quoted 18 lines · Takahiro Itazuri
Rename
gfn_to_pfn_cache_invalidate_start()togpc_invalidate_hva_range_start()to explicitly indicate that it takes a range of HVA range.No functional changes intended.
Signed-off-by Takahiro Itazuri <[email protected]>--- virt/kvm/kvm_main.c | 2 +- virt/kvm/kvm_mm.h | 12 ++++++------ virt/kvm/pfncache.c | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-)virt/kvm/kvm_main.c +0
-0
virt/kvm/kvm_main.c static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, · line 766
Quoted 5 lines
virt/kvm/kvm_main.c +1
-1@@ -763,7 +763,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,* mn_active_invalidate_count (see above) instead of * mmu_invalidate_in_progress. */ -gfn_to_pfn_cache_invalidate_start (kvm, range->start, range->end); +gpc_invalidate_hva_range_start (kvm, range->start, range->end);
Is this actually not specific to the start? It may be called from the MMU notifier's .invalidate_range_start hook, but since we're renaming it, perhaps just gpc_invalidate_hva_range() would be more suitable.
Quoted 6 lines
virt/kvm/kvm_main.c +0
-0/* * If one or more memslots were found and thus zapped, notify arch code
Sean Christopherson
maintainerRe: [RFC PATCH v4 4/7] KVM: pfncache: Rename invalidate_start() helper
Quoted 15 lines · Ackerley Tng
Quoted 13 lines · Takahiro Itazuri
Rename
gfn_to_pfn_cache_invalidate_start()togpc_invalidate_hva_range_start()to explicitly indicate that it takes a range of HVA range.No functional changes intended.
Signed-off-by Takahiro Itazuri <[email protected]>--- virt/kvm/kvm_main.c | 2 +- virt/kvm/kvm_mm.h | 12 ++++++------ virt/kvm/pfncache.c | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-)
Quoted 14 lines
virt/kvm/kvm_main.c static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, · line 766
Quoted 10 lines
virt/kvm/kvm_main.c +1
-1@@ -763,7 +763,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,* mn_active_invalidate_count (see above) instead of * mmu_invalidate_in_progress. */ -gfn_to_pfn_cache_invalidate_start (kvm, range->start, range->end); +gpc_invalidate_hva_range_start (kvm, range->start, range->end);Is this actually not specific to the start? It may be called from the MMU notifier's .invalidate_range_start hook, but since we're renaming it, perhaps just
gpc_invalidate_hva_range()would be more suitable.
Before we touch this name, we need to bottom out on optimizing gpc invalidation, because odds are good the behavior and thus the "best" name could end up being quite different.
RFC PATCH v4 5/7 KVM: pfncache: Invalidate on gmem invalidation and memattr updates
In reply to Takahiro ItazuriInvalidate pfncaches when guest_memfd invalidation or memory attribute updates render cached PFN resolutions stale.
Reuse mn_active_invalidate_count to synchronize with the existing retry logic and preserve ordering against mmu_invalidate_seq.
Invalidation needs to be performed using HVA ranges so that both GPA-based and HVA-based pfncaches are covered. Internally GPA-based ones translate GPA to memslot/UHVA first and then resolve PFN, while HVA-based ones only resolve PFN and do not store memslot/GPA context. Technically, it is possible to make HVA-based pfncaches search the corresponding memslot/GPA when activated/refreshed, but it would add overhead to a greater or lesser extent, regardless of guest_memfd-backed or not. At the time of writing, only Xen uses HVA-based pfncaches.
--- virt/kvm/guest_memfd.c | 50 ++++++++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 47 ++++++++++++++++++++++++++++++++++++++- virt/kvm/pfncache.c | 21 ++++++++++-------- 3 files changed, 108 insertions(+), 10 deletions(-)
virt/kvm/guest_memfd.c +50 -0
@@ -215,6 +215,33 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,struct kvm *kvm = f->kvm;unsigned long index; +/* +* Prevent pfncaches from being activated / refreshed using stale PFN +* resolutions. To invalidate pfncaches _before_ invalidating the +* secondary MMUs (i.e. without acquiring mmu_lock), pfncaches must use +* mn_active_invalidate_count instead of mmu_invalidate_in_progress. +*/ +spin_lock (&kvm->mn_invalidate_lock); + kvm->mn_active_invalidate_count++; +spin_unlock (&kvm->mn_invalidate_lock); + +/* +* Invalidation of pfncaches must be done using a HVA range. pfncaches +* can be either GPA-based or HVA-based, and all pfncaches store uhva +* while HVA-based pfncaches do not have gpa/memslot context. Thus, +* using GFN ranges would miss invalidating HVA-based ones. +*/ +xa_for_each_range (&f->bindings, index, slot, start, end -1 ) { +pgoff_t pgoff = slot->gmem.pgoff; +gfn_t gfn_start = slot->base_gfn +max (pgoff, start) - pgoff; +gfn_t gfn_end = slot->base_gfn +min (pgoff + slot->npages, end) - pgoff; + +unsigned long hva_start =gfn_to_hva_memslot (slot, gfn_start); +unsigned long hva_end = hva_start + (gfn_end - gfn_start) *PAGE_SIZE ; + +gpc_invalidate_hva_range_start (kvm, hva_start, hva_end); + } +xa_for_each_range (&f->bindings, index, slot, start, end -1 ) {pgoff_t pgoff = slot->gmem.pgoff;
@@ -259,12 +286,35 @@ static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start, pgoff_t end) {struct kvm *kvm = f->kvm; +bool wake;if (xa_find (&f->bindings, &start, end -1 ,XA_PRESENT )) {KVM_MMU_LOCK (kvm);kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm); } + +/* +* This must be done after the increment of mmu_invalidate_seq and +* smp_wmb() in kvm_mmu_invalidate_end() to guarantee that +* gpc_invalidate_retry() observes either the old (non-zero) +* mn_active_invalidate_count or the new (incremented) mmu_invalidate_seq. +*/ +spin_lock (&kvm->mn_invalidate_lock); +if (!WARN_ON_ONCE (!kvm->mn_active_invalidate_count)) + kvm->mn_active_invalidate_count--; + wake = !kvm->mn_active_invalidate_count; +spin_unlock (&kvm->mn_invalidate_lock); + +/* +* guest_memfd invalidation itself doesn't need to block active memslots +* swap as bindings updates are serialized by filemap_invalidate_lock(). +* However, mn_active_invalidate_count is shared with the MMU notifier +* path, so the waiter must be woken when mn_active_invalidate_count +* drops to zero. +*/ +if (wake) +rcuwait_wake_up (&kvm->mn_memslots_update_rcuwait); } static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,
virt/kvm/kvm_main.c +46 -1
67 lines not on this page · raw message
virt/kvm/pfncache.c +12 -9
@@ -144,7 +144,7 @@ static void gpc_unmap(kvm_pfn_t pfn, void *khva)#endif } -static inline bool mmu_notifier_retry_cache (struct kvm *kvm,unsigned long mmu_seq) +static inline bool gpc_invalidate_retry (struct kvm *kvm,unsigned long mmu_seq) {/* * mn_active_invalidate_count acts for all intents and purposes
@@ -178,14 +178,17 @@ static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long mmu_s* * The caller holds gpc->refresh_lock, but does not hold gpc->lock nor * kvm->slots_lock. Reading slot->flags (via kvm_slot_has_gmem() and -* kvm_memslot_is_gmem_only()) is safe because memslot changes bump -* slots->generation, which is detected in kvm_gpc_check(), forcing callers -* to invoke kvm_gpc_refresh(). +* kvm_memslot_is_gmem_only()) and looking up memory attributes (via +* kvm_mem_is_private()) without those locks is safe because: * -* Looking up memory attributes (via kvm_mem_is_private()) can race with -* KVM_SET_MEMORY_ATTRIBUTES, which takes kvm->slots_lock to serialize -* writers but doesn't exclude lockless readers. Handling that race is deferred -* to a subsequent commit that wires up pfncache invalidation for gmem events. +* - memslot changes bump slots->generation, which is detected in +* kvm_gpc_check(), forcing callers to invoke kvm_gpc_refresh(). +* +* - Memory attribute changes and gmem invalidations elevate +* mn_active_invalidate_count and bump mmu_invalidate_seq, bracketing the +* pfncache invalidation. gpc_invalidate_retry() observes either of these +* changes and forces a retry of the refresh loop in gpc_to_pfn_retry(), so +* any stale value read here will be re-evaluated. */ static inline bool gpc_is_gmem_backed (struct gfn_to_pfn_cache *gpc) {
@@ -293,7 +296,7 @@ static kvm_pfn_t gpc_to_pfn_retry(struct gfn_to_pfn_cache *gpc)* attempting to refresh. */ WARN_ON_ONCE (gpc->valid); - }while (mmu_notifier_retry_cache (gpc->kvm, mmu_seq)); + }while (gpc_invalidate_retry (gpc->kvm, mmu_seq)); gpc->valid = true; gpc->pfn = new_pfn;
Signature · 1 line
2.50.1
Re: [RFC PATCH v4 5/7] KVM: pfncache: Invalidate on gmem invalidation and memattr updates
Quoted 17 lines · Takahiro Itazuri
[...snip...]
--- virt/kvm/guest_memfd.c | 50 ++++++++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 47 ++++++++++++++++++++++++++++++++++++++- virt/kvm/pfncache.c | 21 ++++++++++-------- 3 files changed, 108 insertions(+), 10 deletions(-)virt/kvm/guest_memfd.c +0
-0@@ -215,6 +215,33 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,struct kvm *kvm = f->kvm;unsigned long index;
virt/kvm/guest_memfd.c static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, · line 227
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+/* +* Prevent pfncaches from being activated / refreshed using stale PFN +* resolutions. To invalidate pfncaches _before_ invalidating the +* secondary MMUs (i.e. without acquiring mmu_lock), pfncaches must use +* mn_active_invalidate_count instead of mmu_invalidate_in_progress. +*/ +spin_lock (&kvm->mn_invalidate_lock); + kvm->mn_active_invalidate_count++; +spin_unlock (&kvm->mn_invalidate_lock); +
I think we should refactor this out into another function and reuse it in kvm_mmu_notifier_invalidate_range_start().
Quoted 7 lines
virt/kvm/guest_memfd.c +7
-0+/* +* Invalidation of pfncaches must be done using a HVA range. pfncaches +* can be either GPA-based or HVA-based, and all pfncaches store uhva +* while HVA-based pfncaches do not have gpa/memslot context. Thus, +* using GFN ranges would miss invalidating HVA-based ones. +*/ +xa_for_each_range (&f->bindings, index, slot, start, end -1 ) {
virt/kvm/guest_memfd.c static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, · line 244
Quoted 10 lines
virt/kvm/guest_memfd.c +10
-0+pgoff_t pgoff = slot->gmem.pgoff; +gfn_t gfn_start = slot->base_gfn +max (pgoff, start) - pgoff; +gfn_t gfn_end = slot->base_gfn +min (pgoff + slot->npages, end) - pgoff; + +unsigned long hva_start =gfn_to_hva_memslot (slot, gfn_start); +unsigned long hva_end = hva_start + (gfn_end - gfn_start) *PAGE_SIZE ; + +gpc_invalidate_hva_range_start (kvm, hva_start, hva_end); + } +
Perhaps refactor this loop into another function? Or perhaps it's time to refactor the xa_for_each_range()...
Quoted 27 lines
virt/kvm/guest_memfd.c +14
-0xa_for_each_range (&f->bindings, index, slot, start, end -1 ) {pgoff_t pgoff = slot->gmem.pgoff;@@ -259,12 +286,35 @@ static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start, pgoff_t end) {struct kvm *kvm = f->kvm; +bool wake;if (xa_find (&f->bindings, &start, end -1 ,XA_PRESENT )) {KVM_MMU_LOCK (kvm);kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm); } + +/* +* This must be done after the increment of mmu_invalidate_seq and +* smp_wmb() in kvm_mmu_invalidate_end() to guarantee that +* gpc_invalidate_retry() observes either the old (non-zero) +* mn_active_invalidate_count or the new (incremented) mmu_invalidate_seq. +*/ +spin_lock (&kvm->mn_invalidate_lock); +if (!WARN_ON_ONCE (!kvm->mn_active_invalidate_count)) + kvm->mn_active_invalidate_count--; + wake = !kvm->mn_active_invalidate_count; +spin_unlock (&kvm->mn_invalidate_lock); +
virt/kvm/guest_memfd.c static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start, · line 318
Quoted 10 lines
virt/kvm/guest_memfd.c +9
-0+/* +* guest_memfd invalidation itself doesn't need to block active memslots +* swap as bindings updates are serialized by filemap_invalidate_lock(). +* However, mn_active_invalidate_count is shared with the MMU notifier +* path, so the waiter must be woken when mn_active_invalidate_count +* drops to zero. +*/ +if (wake) +rcuwait_wake_up (&kvm->mn_memslots_update_rcuwait); }
There could be some code reuse here too.
Quoted 41 lines
virt/kvm/guest_memfd.c +0
-0static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,virt/kvm/kvm_main.c +21
-1@@ -2583,9 +2583,11 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, .on_lock = kvm_mmu_invalidate_end, .may_block = true, }; +struct kvm_memslots *slots =kvm_memslots (kvm); +struct kvm_memory_slot *slot;unsigned long i;void *entry; -int r =0 ; +int r =0 , bkt; entry = attributes ?xa_mk_value (attributes) :NULL ;@@ -2609,6 +2611,34 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,cond_resched (); } +/* +* Prevent pfncaches from being activated / refreshed using stale PFN +* resolutions. To invalidate pfncaches _before_ invalidating the +* secondary MMUs (i.e. without acquiring mmu_lock), pfncaches must use +* mn_active_invalidate_count instead of mmu_invalidate_in_progress. +*/ +spin_lock (&kvm->mn_invalidate_lock); + kvm->mn_active_invalidate_count++; +spin_unlock (&kvm->mn_invalidate_lock); + +/* +* Invalidation of pfncaches must be done using a HVA range. pfncaches +* can be either GPA-based or HVA-based, and all pfncaches store uhva +* while HVA-based pfncaches do not have gpa/memslot info. Thus, +* using GFN ranges would miss invalidating HVA-based ones. +*/ +kvm_for_each_memslot (slot, bkt, slots) { +gfn_t gfn_start =max (start, slot->base_gfn);
virt/kvm/kvm_main.c static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, · line 2641
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+gfn_t gfn_end =min (end, slot->base_gfn + slot->npages); + +if (gfn_start < gfn_end) { +unsigned long hva_start =gfn_to_hva_memslot (slot, gfn_start); +unsigned long hva_end = hva_start + (gfn_end - gfn_start) *PAGE_SIZE ; + +gpc_invalidate_hva_range_start (kvm, hva_start, hva_end); + } + } +
Can we reuse kvm_handle_gfn_range()?
Quoted 6 lines
virt/kvm/kvm_main.c +0
-0kvm_handle_gfn_range (kvm, &pre_set_range);for (i = start; i < end; i++) {@@ -2620,6 +2650,21 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,kvm_handle_gfn_range (kvm, &post_set_range);
virt/kvm/kvm_main.c static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, · line 2661
Quoted 10 lines
virt/kvm/kvm_main.c +9
-0+/* +* This must be done after the increment of mmu_invalidate_seq and +* smp_wmb() in kvm_mmu_invalidate_end() to guarantee that +* gpc_invalidate_retry() observes either the old (non-zero) +* mn_active_invalidate_count or the new (incremented) mmu_invalidate_seq. +* +* mn_memslots_update_rcuwait does not need to be waked when +* mn_active_invalidate_count drops to zero because active memslots swap +* is also done while holding slots_lock.
I see that this invalidate count and waking stuff is done in kvm_mmu_notifier_invalidate_range_end(), can that function be reused, or maybe some refactoring will help here
Quoted 14 lines
virt/kvm/kvm_main.c +6
-0+*/ +spin_lock (&kvm->mn_invalidate_lock); +if (!WARN_ON_ONCE (!kvm->mn_active_invalidate_count)) + kvm->mn_active_invalidate_count--; +spin_unlock (&kvm->mn_invalidate_lock); + out_unlock:mutex_unlock (&kvm->slots_lock);virt/kvm/pfncache.c +0
-0
virt/kvm/pfncache.c static void gpc_unmap(kvm_pfn_t pfn, void *khva) · line 147
Quoted 5 lines
virt/kvm/pfncache.c +1
-1@@ -144,7 +144,7 @@ static void gpc_unmap(kvm_pfn_t pfn, void *khva)#endif } -static inline bool mmu_notifier_retry_cache (struct kvm *kvm,unsigned long mmu_seq) +static inline bool gpc_invalidate_retry (struct kvm *kvm,unsigned long mmu_seq)
Perhaps this rename can be a separate patch. I think this could be set up as some renaming/refactoring patches, then a separate patch for calling those functions from gmem and kvm_vm_set_mem_attributes().
Quoted 3 lines
[...snip...]
RFC PATCH v4 6/7 KVM: selftests: Test pfncache with gmem-backed memory
In reply to Takahiro ItazuriAdd a selftest that exercises pfncache (gfn_to_pfn_cache) with guest_memfd-backed memory by using kvm-clock as the test vehicle.
The test creates two VM configurations:
- NO_DIRECT_MAP VM: All memory is gmem-backed (MMAP | INIT_SHARED | NO_DIRECT_MAP). KVM_MEMSLOT_GMEM_ONLY is set, so pfncache resolves PFNs via kvm_gmem_get_pfn() and maps KHVAs via vmap().
- SW_PROTECTED_VM: Memory starts private. pfncache resolves PFNs via kvm_gmem_get_pfn() for private pages. This validates the private memory pfncache path for future extensibility (e.g. pKVM-like VMs).
The guest writes MSR_KVM_SYSTEM_TIME_NEW (triggering kvm_gpc_activate() internally), reads the pvclock structure to verify KVM wrote through the pfncache KHVA correctly, and reports the kvm-clock value to the host for bounds checking against KVM_GET_CLOCK.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/pfncache_gmem_test.c | 193 ++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/pfncache_gmem_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -92,6 +92,7 @@ TEST_GEN_PROGS_x86 += x86/nested_emulation_test TEST_GEN_PROGS_x86 += x86/nested_exceptions_test TEST_GEN_PROGS_x86 += x86/platform_info_test TEST_GEN_PROGS_x86 += x86/pmu_counters_test +TEST_GEN_PROGS_x86 += x86/pfncache_gmem_test TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test TEST_GEN_PROGS_x86 += x86/private_mem_conversions_test TEST_GEN_PROGS_x86 += x86/private_mem_kvm_exits_test
tools/testing/selftests/kvm/x86/pfncache_gmem_test.c new file +193 -0
193 lines not on this page · raw message
Signature · 1 line
2.50.1
Re: [RFC PATCH v4 6/7] KVM: selftests: Test pfncache with gmem-backed memory
Quoted 5 lines · Takahiro Itazuri
Add a selftest that exercises pfncache (
gfn_to_pfn_cache) with guest_memfd-backed memory by using kvm-clock as the test vehicle.The test creates two VM configurations:
Quoted 8 lines
-
NO_DIRECT_MAPVM: All memory is gmem-backed (MMAP |INIT_SHARED|NO_DIRECT_MAP).KVM_MEMSLOT_GMEM_ONLYis set, so pfncache resolves PFNs viakvm_gmem_get_pfn()and maps KHVAs viavmap().-
SW_PROTECTED_VM: Memory starts private. pfncache resolves PFNs viakvm_gmem_get_pfn()for private pages. This validates the private memory pfncache path for future extensibility (e.g. pKVM-like VMs).
I think to make the test very explicit about what it's testing, perhaps always allocate a guest_memfd memslot for the clock, and then make setup_vm() like setup_vm(vm_type, gmem_flags), where gmem_flags is
+ MMAP | INIT_SHARED | NO_DIRECT_MAP
+ 0 (for init private, hence no HVA)
setup_vm() can then check if the vm_type and gmem_flags being requested is supported and skip if not supported.
Perhaps then have a test_gpc(vm_type, gmem_flags) that calls setup_vm() and then runs the test, then main can be like
test_gpc(DEFAULT_VM, MMAP | INIT_SHARED | NO_DIRECT_MAP);
test_gpc(SW_PROTECTED_VM, MMAP | INIT_SHARED | NO_DIRECT_MAP); <== new case
test_gpc(SW_PROTECTED_VM, 0);At first I thought nothing about the VM type being tested here but I guess your goal to test that non-CoCo VMs using guest_memfd with NO_DIRECT_MAP can also use the updated gpc? I guess adding the middle case is optional but perhaps shows that the vm type doesn't really matter.
The path where the gpc gets invalidated is not tested, but I guess you're probably waiting for my conversions series :P (sorry!)
Quoted 6 lines
The guest writes
MSR_KVM_SYSTEM_TIME_NEW(triggeringkvm_gpc_activate()internally), reads the pvclock structure to verify KVM wrote through the pfncache KHVA correctly, and reports the kvm-clock value to the host for bounds checking againstKVM_GET_CLOCK.Signed-off-by Takahiro Itazuri <[email protected]>
RFC PATCH v4 7/7 KVM: selftests: Test pfncache invalidation for gmem-backed memory
In reply to Takahiro ItazuriExtend pfncache_gmem_test to verify pfncache invalidation paths:
- punch_hole: fallocate(PUNCH_HOLE) on the pvclock page's guest_memfd frees the backing page and invalidates the pfncache. The next vcpu_run re-resolves the PFN with a freshly allocated page.
- file release: kvm_vm_free() closes the guest_memfd fd, triggering a full-range pfncache invalidation.
- private-to-shared conversion: KVM_SET_MEMORY_ATTRIBUTES changes the pvclock page from private to shared, invalidating the pfncache. The PFN is re-resolved via GUP instead of kvm_gmem_get_pfn().
--- .../selftests/kvm/x86/pfncache_gmem_test.c | 39 ++++++++++++++++--- 1 file changed, 34 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/x86/pfncache_gmem_test.c +34 -5
@@ -36,11 +36,10 @@ static void guest_main(vm_paddr_t pvti_pa, struct pvclock_vcpu_time_info *pvti) {int stage =0 ; -wrmsr (MSR_KVM_SYSTEM_TIME_NEW , pvti_pa |KVM_MSR_ENABLED ); -for (;;) {uint64_t clock; +wrmsr (MSR_KVM_SYSTEM_TIME_NEW , pvti_pa |KVM_MSR_ENABLED );GUEST_ASSERT (pvti->system_time !=0 ); clock =__pvclock_read_cycles (pvti,rdtsc ());GUEST_SYNC_CLOCK (stage++, clock);
@@ -97,7 +96,8 @@ static uint64_t run_and_verify_kvm_clock(struct kvm_vcpu *vcpu,#define PVCLOCK_GPA (1ULL << 32)static struct kvm_vm *setup_vm (struct vm_shape shape, -struct kvm_vcpu **vcpu_out) +struct kvm_vcpu **vcpu_out, +vm_paddr_t *pvti_gpa_out) {struct kvm_vm *vm;
@@ -118,6 +118,9 @@ static struct kvm_vm *setup_vm(struct vm_shape shape,vcpu_args_set (*vcpu_out,2 , (vm_paddr_t )PVCLOCK_GPA , (struct pvclock_vcpu_time_info *)PVCLOCK_GPA );vm_mem_set_private (vm,PVCLOCK_GPA ,getpagesize ()); + +if (pvti_gpa_out) + *pvti_gpa_out =PVCLOCK_GPA ; }else {vm_vaddr_t pvti_gva;vm_paddr_t pvti_gpa;
@@ -125,6 +128,9 @@ static struct kvm_vm *setup_vm(struct vm_shape shape, pvti_gva =vm_vaddr_alloc (vm,getpagesize (),0x10000 ); pvti_gpa =addr_gva2gpa (vm, pvti_gva);vcpu_args_set (*vcpu_out,2 , pvti_gpa, pvti_gva); + +if (pvti_gpa_out) + *pvti_gpa_out = pvti_gpa; }return vm;
@@ -139,16 +145,27 @@ static void test_no_direct_map(void) };struct kvm_vcpu *vcpu;struct kvm_vm *vm; +vm_paddr_t pvti_gpa;uint64_t clock =0 ;pr_info ("Testing pfncache with NO_DIRECT_MAP guest_memfd\n" ); - vm =setup_vm (shape, &vcpu); + vm =setup_vm (shape, &vcpu, &pvti_gpa);/* Verify kvm-clock works with gmem-backed pfncache (vmap KHVA) */ clock =run_and_verify_kvm_clock (vcpu, clock); clock =run_and_verify_kvm_clock (vcpu, clock); +/* +* Punch a hole in the pvclock page's guest_memfd backing. This +* invalidates the pfncache; the next vcpu_run re-resolves the PFN +* with a freshly allocated page. +*/ +pr_info (" punch_hole on pvclock page\n" ); +vm_guest_mem_punch_hole (vm, pvti_gpa,getpagesize ()); + clock =run_and_verify_kvm_clock (vcpu, clock); + +/* Smoke test: VM teardown (closing guest_memfd) doesn't crash. */ kvm_vm_free (vm); }
@@ -160,16 +177,28 @@ static void test_sw_protected_vm(void) };struct kvm_vcpu *vcpu;struct kvm_vm *vm; +vm_paddr_t pvti_gpa;uint64_t clock =0 ;pr_info ("Testing pfncache with SW_PROTECTED_VM (guest_memfd-backed private memory)\n" ); - vm =setup_vm (shape, &vcpu); + vm =setup_vm (shape, &vcpu, &pvti_gpa);/* Verify kvm-clock works with gmem-backed private memory */ clock =run_and_verify_kvm_clock (vcpu, clock); clock =run_and_verify_kvm_clock (vcpu, clock); +/* Convert pvclock page from private to shared */ +pr_info (" converting pvclock page: private -> shared\n" ); +vm_mem_set_shared (vm, pvti_gpa,getpagesize ()); + clock =run_and_verify_kvm_clock (vcpu,0 ); + +/* Convert back to private */ +pr_info (" converting pvclock page: shared -> private\n" ); +vm_mem_set_private (vm, pvti_gpa,getpagesize ()); + clock =run_and_verify_kvm_clock (vcpu,0 ); + +/* Smoke test: VM teardown (closing guest_memfd) doesn't crash. */ kvm_vm_free (vm); }
Signature · 1 line
2.50.1
Sean Christopherson
maintainerQuoted 6 lines · Takahiro Itazuri
[ based on 6.18 with [1] ]
This patch series adds
guest_memfdsupport togfn_to_pfn_cache(a.k.a. pfncache). (This is still labelled RFC since its dependency [1] has not yet been merged.)
Quoted 8 lines
=== Problem Statement ===
pfncache does not work with
guest_memfd. pfncaches resolve PFNs viahva_to_pfn(), which requires a userspace mapping and relies on GUP. This does not work forguest_memfdin the following two ways:*
guest_memfdcreated without MMAP flag does not have a userspace mapping due to the nature of private memory.
Sorry, but NAK. At least until we have a *very* clear and generally agreed-upon plan for how KVM is going to handle accessing guest memory NO_DIRECT_MAP (my brain finally caught up to what this series wants to do).
Simply allowing gpcs to tap directly into guest_memfd doesn't allow KVM x86 to use guest_memfd for non-CoCo VMs. There might be a special class of VMs that can squeak by, but any VM that finds itself in __kvm_{read,write}_guest_page() or in FNAME(walk_addr_generic) needs to have guest memory mapped into host userspace. Predicting what memory a "normal" VM will use for TDP or the source/destination of emulated MMIO is simply not feasible.
At that point, the "guest_memfd created without MMAP flag" use case is meaningless, because creating such a guest_memfd instance is *only* useful for backing PRIVATE memory, which KVM obviously shouldn't be mapping.
This topic was discussed heavily at LPC 2024 (and/or LPC 2023?), and the consensus was that mapping guest_memfd into host userspace is the least awful choice, the only complete alternative being a *massive* rework of KVM x86 internals (other architectures don't suffer as much due to limited emulation).
Quoted 1 line
KVM: Rename
invalidate_begintoinvalidate_startfor consistencyo
I'll grab this one for 7.2.
Quoted 4 lines
KVM: pfncache: Rename
invalidate_start()helper
KVM: pfncache: Invalidate on gmem invalidation and memattr updates
KVM: selftests: Test pfncache with gmem-backed memory
KVM: selftests: Test pfncache invalidation for gmem-backed memory
Quoted 9 lines · Takahiro Itazuri
[ based on 6.18 with [1] ]
This patch series adds
guest_memfdsupport togfn_to_pfn_cache(a.k.a. pfncache). (This is still labelled RFC since its dependency [1] has not yet been merged.)=== Problem Statement ===
[...]
Applied patch 3 to kvm-x86 generic, thanks!
[3/7] KVM: Rename invalidate_begin to invalidate_start for consistency
https://github.com/kvm-x86/linux/commit/190cc5370a8b