KVM: allow mapping non-refcounted pages
v10 cover KVM: allow mapping non-refcounted pages
From: David Stevens <[email protected]>
This patch series adds support for mapping VM_IO and VM_PFNMAP memory that is backed by struct pages that aren't currently being refcounted (e.g. tail pages of non-compound higher order allocations) into the guest.
Our use case is virtio-gpu blob resources [1], which directly map host graphics buffers into the guest as "vram" for the virtio-gpu device. This feature currently does not work on systems using the amdgpu driver, as that driver allocates non-compound higher order pages via ttm_pool_alloc_page().
First, this series replaces the gfn_to_pfn_memslot() API with a more extensible kvm_follow_pfn() API. The updated API rearranges gfn_to_pfn_memslot()'s args into a struct and where possible packs the bool arguments into a FOLL_ flags argument. The refactoring changes do not change any behavior.
From there, this series extends the kvm_follow_pfn() API so that non-refconuted pages can be safely handled. This invloves adding an input parameter to indicate whether the caller can safely use non-refcounted pfns and an output parameter to tell the caller whether or not the returned page is refcounted. This change includes a breaking change, by disallowing non-refcounted pfn mappings by default, as such mappings are unsafe. To allow such systems to continue to function, an opt-in module parameter is added to allow the unsafe behavior.
This series only adds support for non-refcounted pages to x86. Other MMUs can likely be updated without too much difficulty, but it is not needed at this point. Updating other parts of KVM (e.g. pfncache) is not straightforward [2].
[1]
https://patchwork.kernel.org/project/dri-devel/cover/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
v9 -> v10:
- Re-add FOLL_GET changes.
- Split x86/mmu spte+non-refcount-page patch into two patches.
- Rename 'foll' variables to 'kfp'.
- Properly gate usage of refcount spte bit when it's not available.
- Replace kfm_follow_pfn's is_refcounted_page output parameter with a struct page *refcounted_page pointing to the page in question.
- Add patch downgrading BUG_ON to WARN_ON_ONCE.
v8 -> v9:
- Make paying attention to is_refcounted_page mandatory. This means that FOLL_GET is no longer necessary. For compatibility with un-migrated callers, add a temporary parameter to sidestep ref-counting issues.
- Add allow_unsafe_mappings, which is a breaking change.
- Migrate kvm_vcpu_map and other callsites used by x86 to the new API.
- Drop arm and ppc changes.
v7 -> v8:
- Set access bits before releasing mmu_lock.
- Pass FOLL_GET on 32-bit x86 or !tdp_enabled.
- Refactor FOLL_GET handling, add kvm_follow_refcounted_pfn helper.
- Set refcounted bit on >4k pages.
- Add comments and apply formatting suggestions.
- rebase on kvm next branch.
v6 -> v7:
- Replace __gfn_to_pfn_memslot with a more flexible __kvm_faultin_pfn, and extend that API to support non-refcounted pages (complete rewrite).
David Stevens (7):
KVM: Relax BUG_ON argument validation
KVM: mmu: Introduce kvm_follow_pfn()
KVM: mmu: Improve handling of non-refcounted pfns
KVM: Migrate kvm_vcpu_map() to kvm_follow_pfn()
KVM: x86: Migrate to kvm_follow_pfn()
KVM: x86/mmu: Track if sptes refer to refcounted pages
KVM: x86/mmu: Handle non-refcounted pages
Sean Christopherson (1):
KVM: Assert that a page's refcount is elevated when marking
accessed/dirty
arch/x86/kvm/mmu/mmu.c | 104 +++++++--- arch/x86/kvm/mmu/mmu_internal.h | 2 + arch/x86/kvm/mmu/paging_tmpl.h | 7 +- arch/x86/kvm/mmu/spte.c | 4 +- arch/x86/kvm/mmu/spte.h | 22 +- arch/x86/kvm/mmu/tdp_mmu.c | 22 +- arch/x86/kvm/x86.c | 11 +- include/linux/kvm_host.h | 53 ++++- virt/kvm/guest_memfd.c | 8 +- virt/kvm/kvm_main.c | 349 +++++++++++++++++++------------- virt/kvm/kvm_mm.h | 3 +- virt/kvm/pfncache.c | 11 +- 12 files changed, 399 insertions(+), 197 deletions(-)
base-commit: 54be6c6c5ae8e0d93a6c4641cb7528eb0b6ba478
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
PATCH v10 1/8 KVM: Assert that a page's refcount is elevated when marking accessed/dirty
From: Sean Christopherson <[email protected]>
Assert that a page's refcount is elevated, i.e. that _something_ holds a reference to the page, when KVM marks a page as accessed and/or dirty. KVM typically doesn't hold a reference to pages that are mapped into the guest, e.g. to allow page migration, compaction, swap, etc., and instead relies on mmu_notifiers to react to changes in the primary MMU.
Incorrect handling of mmu_notifier events (or similar mechanisms) can result in KVM keeping a mapping beyond the lifetime of the backing page, i.e. can (and often does) result in use-after-free. Yelling if KVM marks a freed page as accessed/dirty doesn't prevent badness as KVM usually only does A/D updates when unmapping memory from the guest, i.e. the assertion fires well after an underlying bug has occurred, but yelling does help detect, triage, and debug use-after-free bugs.
Note, the assertion must use page_count(), NOT page_ref_count()! For hugepages, the returned struct page may be a tailpage and thus not have its own refcount.
--- virt/kvm/kvm_main.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
virt/kvm/kvm_main.c +13 -0
@@ -3204,6 +3204,19 @@ EXPORT_SYMBOL_GPL(kvm_vcpu_unmap);static bool kvm_is_ad_tracked_page (struct page *page) { +/* +* Assert that KVM isn't attempting to mark a freed page as Accessed or +* Dirty, i.e. that KVM's MMU doesn't have a use-after-free bug. KVM +* (typically) doesn't pin pages that are mapped in KVM's MMU, and +* instead relies on mmu_notifiers to know when a mapping needs to be +* zapped/invalidated. Unmapping from KVM's MMU must happen _before_ +* KVM returns from its mmu_notifier, i.e. the page should have an +* elevated refcount at this point even though KVM doesn't hold a +* reference of its own. +*/ +if (WARN_ON_ONCE (!page_count (page))) +return false; +/* * Per page-flags.h, pages tagged PG_reserved "should in general not be * touched (e.g. set dirty) except by its owner".
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
PATCH v10 2/8 KVM: Relax BUG_ON argument validation
In reply to David StevensFrom: David Stevens <[email protected]>
hva_to_pfn() includes a check that KVM isn't trying to do an async page fault in a situation where it can't sleep. Downgrade this check from a BUG_ON() to a WARN_ON_ONCE(), since DoS'ing the guest (at worst) is better than bringing down the host.
--- virt/kvm/kvm_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
virt/kvm/kvm_main.c +1 -1
@@ -2979,7 +2979,7 @@ kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool interruptible,int npages, r;/* we can do it either atomically or asynchronously, not both */ -BUG_ON (atomic && async); +WARN_ON_ONCE (atomic && async);if (hva_to_pfn_fast (addr, write_fault, writable, &pfn))return pfn;
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
PATCH v10 3/8 KVM: mmu: Introduce kvm_follow_pfn()
In reply to David StevensFrom: David Stevens <[email protected]>
Introduce kvm_follow_pfn(), which will replace __gfn_to_pfn_memslot(). This initial implementation is just a refactor of the existing API which uses a single structure for passing the arguments. The arguments are further refactored as follows:
- The write_fault and interruptible boolean flags and the in parameter part of async are replaced by setting FOLL_WRITE, FOLL_INTERRUPTIBLE, and FOLL_NOWAIT respectively in a new flags argument.
- The out parameter portion of the async parameter is now a return value.
- The writable in/out parameter is split into a separate. try_map_writable in parameter and writable out parameter.
- All other parameter are the same.
Upcoming changes will add the ability to get a pfn without needing to take a ref to the underlying page.
--- include/linux/kvm_host.h | 18 ++++ virt/kvm/kvm_main.c | 191 +++++++++++++++++++++------------------ virt/kvm/kvm_mm.h | 3 +- virt/kvm/pfncache.c | 10 +- 4 files changed, 131 insertions(+), 91 deletions(-)
include/linux/kvm_host.h +18 -0
@@ -97,6 +97,7 @@#define KVM_PFN_ERR_HWPOISON (KVM_PFN_ERR_MASK + 1)#define KVM_PFN_ERR_RO_FAULT (KVM_PFN_ERR_MASK + 2)#define KVM_PFN_ERR_SIGPENDING (KVM_PFN_ERR_MASK + 3) +#define KVM_PFN_ERR_NEEDS_IO (KVM_PFN_ERR_MASK + 4)/* * error pfns indicate that the gfn is in slot but faild to
@@ -1209,6 +1210,23 @@ unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot, gfn_t gfn,void kvm_release_page_clean (struct page *page);void kvm_release_page_dirty (struct page *page); +struct kvm_follow_pfn { +const struct kvm_memory_slot *slot; +gfn_t gfn; +/* FOLL_* flags modifying lookup behavior. */ +unsigned int flags; +/* Whether this function can sleep. */ +bool atomic; +/* Try to create a writable mapping even for a read fault. */ +bool try_map_writable; + +/* Outputs of kvm_follow_pfn */ +hva_t hva; +bool writable; +}; + +kvm_pfn_t kvm_follow_pfn (struct kvm_follow_pfn *kfp); +kvm_pfn_t gfn_to_pfn (struct kvm *kvm,gfn_t gfn);kvm_pfn_t gfn_to_pfn_prot (struct kvm *kvm,gfn_t gfn,bool write_fault,bool *writable);
virt/kvm/kvm_main.c +104 -87
325 lines not on this page · raw message
virt/kvm/kvm_mm.h +1 -2
@@ -20,8 +20,7 @@#define KVM_MMU_UNLOCK (kvm) spin_unlock(&(kvm)->mmu_lock)#endif /* KVM_HAVE_MMU_RWLOCK */ -kvm_pfn_t hva_to_pfn (unsigned long addr,bool atomic,bool interruptible, -bool *async,bool write_fault,bool *writable); +kvm_pfn_t hva_to_pfn (struct kvm_follow_pfn *foll);#ifdef CONFIG_HAVE_KVM_PFNCACHE void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm,
virt/kvm/pfncache.c +8 -2
@@ -144,6 +144,12 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)kvm_pfn_t new_pfn =KVM_PFN_ERR_FAULT ;void *new_khva =NULL ;unsigned long mmu_seq; +struct kvm_follow_pfn kfp = { + .slot = gpc->memslot, + .gfn =gpa_to_gfn (gpc->gpa), + .flags =FOLL_WRITE , + .hva = gpc->uhva, + };lockdep_assert_held (&gpc->refresh_lock);
@@ -182,8 +188,8 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)cond_resched (); } -/* We always request a writeable mapping */ - new_pfn =hva_to_pfn (gpc->uhva, false, false,NULL , true,NULL ); +/* We always request a writable mapping */ + new_pfn =hva_to_pfn (&kfp);if (is_error_noslot_pfn (new_pfn)) goto out_error;
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
PATCH v10 4/8 KVM: mmu: Improve handling of non-refcounted pfns
In reply to David StevensFrom: David Stevens <[email protected]>
KVM's handling of non-refcounted pfns has two problems:
- pfns without struct pages can be accessed without the protection of a mmu notifier. This is unsafe because KVM cannot monitor or control the lifespan of such pfns, so it may continue to access the pfns after they are freed.
- struct pages without refcounting (e.g. tail pages of non-compound higher order pages) cannot be used at all, as gfn_to_pfn does not provide enough information for callers to be able to avoid underflowing the refcount.
This patch extends the kvm_follow_pfn() API to properly handle these cases:
- First, it adds FOLL_GET to the list of supported flags, to indicate whether or not the caller actually wants to take a refcount.
- Second, it adds a guarded_by_mmu_notifier parameter that is used to avoid returning non-refcounted pages when the caller cannot safely use them.
- Third, it adds an is_refcounted_page output parameter so that callers can tell whether or not a pfn has a struct page that needs to be passed to put_page.
Since callers need to be updated on a case-by-case basis to pay attention to is_refcounted_page, the new behavior of returning non-refcounted pages is opt-in via the allow_non_refcounted_struct_page parameter. Once all callers have been updated, this parameter should be removed.
The fact that non-refcounted pfns can no longer be accessed without mmu notifier protection by default is a breaking change. This patch provides a module parameter that system admins can use to re-enable the previous unsafe behavior when userspace is trusted not to migrate/free/etc non-refcounted pfns that are mapped into the guest. There is no timeline for updating everything in KVM to use mmu notifiers to alleviate the need for this module parameter.
--- include/linux/kvm_host.h | 24 +++++++++ virt/kvm/kvm_main.c | 108 ++++++++++++++++++++++++++------------- virt/kvm/pfncache.c | 3 +- 3 files changed, 98 insertions(+), 37 deletions(-)
include/linux/kvm_host.h +24 -0
@@ -1219,10 +1219,34 @@ struct kvm_follow_pfn {bool atomic;/* Try to create a writable mapping even for a read fault. */ bool try_map_writable; +/* +* Usage of the returned pfn will be guared by a mmu notifier. If +* FOLL_GET is not set, this must be true. +*/ +bool guarded_by_mmu_notifier; +/* +* When false, do not return pfns for non-refcounted struct pages. +* +* TODO: This allows callers to use kvm_release_pfn on the pfns +* returned by gfn_to_pfn without worrying about corrupting the +* refcount of non-refcounted pages. Once all callers respect +* refcounted_page, this flag should be removed. +*/ +bool allow_non_refcounted_struct_page;/* Outputs of kvm_follow_pfn */ hva_t hva;bool writable; +/* +* Non-NULL if the returned pfn is for a page with a valid refcount, +* NULL if the returned pfn has no struct page or if the struct page is +* not being refcounted (e.g. tail pages of non-compound higher order +* allocations from IO/PFNMAP mappings). +* +* NOTE: This will still be set if FOLL_GET is not specified, but the +* returned page will not have an elevated refcount. +*/ +struct page *refcounted_page; };kvm_pfn_t kvm_follow_pfn (struct kvm_follow_pfn *kfp);
virt/kvm/kvm_main.c +72 -36
195 lines not on this page · raw message
virt/kvm/pfncache.c +2 -1
@@ -147,8 +147,9 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)struct kvm_follow_pfn kfp = { .slot = gpc->memslot, .gfn =gpa_to_gfn (gpc->gpa), - .flags =FOLL_WRITE , + .flags =FOLL_GET |FOLL_WRITE , .hva = gpc->uhva, + .allow_non_refcounted_struct_page = false, };lockdep_assert_held (&gpc->refresh_lock);
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Paolo Bonzini
maintainerRe: [PATCH v10 4/8] KVM: mmu: Improve handling of non-refcounted pfns
virt/kvm/kvm_main.c static int hva_to_pfn_remapped(struct vm_area_struct *vma, · line 2957
Quoted 7 lines · David Stevens
virt/kvm/kvm_main.c +7
-0+/* +* TODO: Remove the first branch once all callers have been +* taught to play nice with non-refcounted struct pages. +*/ +if (page && !kfp->refcounted_page && + !kfp->allow_non_refcounted_struct_page) { + r = -EFAULT ;
Is the TODO practical, considering that 32-bit AMD as well as all non-TDP x86 do not support non-refcounted pages?
If the field is not going to go away, it's better to point out (in the definition of the struct) that some architectures may not have enough free space in the PTEs for the required tracking; and then drop the TODO.
virt/kvm/kvm_main.c static int hva_to_pfn_remapped(struct vm_area_struct *vma, · line 2961
Quoted 4 lines
virt/kvm/kvm_main.c +4
-0+ }else if (!kfp->refcounted_page && + !kfp->guarded_by_mmu_notifier && + !allow_unsafe_mappings) { + r = -EFAULT ;
Why is allow_unsafe_mappings desirable at all?
None of this is worth a respin, it can be fixed when applying.
Paolo
Sean Christopherson
maintainerRe: [PATCH v10 4/8] KVM: mmu: Improve handling of non-refcounted pfns
Quoted 7 lines · Paolo Bonzini
virt/kvm/kvm_main.c static int hva_to_pfn_remapped(struct vm_area_struct *vma, · line 2961
Quoted 4 lines · David Stevens
virt/kvm/kvm_main.c +4
-0+ }else if (!kfp->refcounted_page && + !kfp->guarded_by_mmu_notifier && + !allow_unsafe_mappings) { + r = -EFAULT ;Why is
allow_unsafe_mappingsdesirable at all?
It's for use cases where memory is hidden from the kernel and managed by userspace, e.g. where AWS uses /dev/mem (I think) to map guest memory. From a kernel perspective, that is unsafe because KVM won't do the right thing if userspace unmaps memory while it is exposed to L2 via a pfn in vmcs02.
I suggested allow_unsafe_mappings as a way to make upstream KVM safe by default, without completely breaking support for AWS and friends.
PATCH v10 5/8 KVM: Migrate kvm_vcpu_map to __kvm_follow_pfn
In reply to David StevensFrom: David Stevens <[email protected]>
Migrate kvm_vcpu_map to __kvm_follow_pfn. Track is_refcounted_page so that kvm_vcpu_unmap know whether or not it needs to release the page.
--- include/linux/kvm_host.h | 2 +- virt/kvm/kvm_main.c | 24 ++++++++++++++---------- 2 files changed, 15 insertions(+), 11 deletions(-)
include/linux/kvm_host.h +1 -1
@@ -295,6 +295,7 @@ struct kvm_host_map {void *hva;kvm_pfn_t pfn;kvm_pfn_t gfn; +bool is_refcounted_page; };/*
@@ -1265,7 +1266,6 @@ void kvm_release_pfn_dirty(kvm_pfn_t pfn);void kvm_set_pfn_dirty (kvm_pfn_t pfn);void kvm_set_pfn_accessed (kvm_pfn_t pfn); -void kvm_release_pfn (kvm_pfn_t pfn,bool dirty);int kvm_read_guest_page (struct kvm *kvm,gfn_t gfn,void *data,int offset,int len);int kvm_read_guest (struct kvm *kvm,gpa_t gpa,void *data,unsigned long len);
virt/kvm/kvm_main.c +14 -10
@@ -3188,24 +3188,22 @@ struct page *gfn_to_page(struct kvm *kvm, gfn_t gfn) }EXPORT_SYMBOL_GPL (gfn_to_page); -void kvm_release_pfn (kvm_pfn_t pfn,bool dirty) -{ -if (dirty) -kvm_release_pfn_dirty (pfn); -else -kvm_release_pfn_clean (pfn); -} -int kvm_vcpu_map (struct kvm_vcpu *vcpu,gfn_t gfn,struct kvm_host_map *map) {kvm_pfn_t pfn;void *hva =NULL ;struct page *page =KVM_UNMAPPED_PAGE ; +struct kvm_follow_pfn kfp = { + .slot =gfn_to_memslot (vcpu->kvm, gfn), + .gfn = gfn, + .flags =FOLL_GET |FOLL_WRITE , + .allow_non_refcounted_struct_page = true, + };if (!map)return -EINVAL ; - pfn =gfn_to_pfn (vcpu->kvm, gfn); + pfn =kvm_follow_pfn (&kfp);if (is_error_noslot_pfn (pfn))return -EINVAL ;
@@ -3225,6 +3223,7 @@ int kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map) map->hva = hva; map->pfn = pfn; map->gfn = gfn; + map->is_refcounted_page = !!kfp.refcounted_page;return 0 ; }
@@ -3248,7 +3247,12 @@ void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map, bool dirty)if (dirty)kvm_vcpu_mark_page_dirty (vcpu, map->gfn); -kvm_release_pfn (map->pfn, dirty); +if (map->is_refcounted_page) { +if (dirty) +kvm_release_page_dirty (map->page); +else +kvm_release_page_clean (map->page); + } map->hva =NULL ; map->page =NULL ;
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
PATCH v10 5/8 KVM: Migrate kvm_vcpu_map() to kvm_follow_pfn()
In reply to David StevensFrom: David Stevens <[email protected]>
Migrate kvm_vcpu_map() to kvm_follow_pfn(). Track is_refcounted_page so that kvm_vcpu_unmap() know whether or not it needs to release the page.
--- include/linux/kvm_host.h | 2 +- virt/kvm/kvm_main.c | 24 ++++++++++++++---------- 2 files changed, 15 insertions(+), 11 deletions(-)
include/linux/kvm_host.h +1 -1
@@ -295,6 +295,7 @@ struct kvm_host_map {void *hva;kvm_pfn_t pfn;kvm_pfn_t gfn; +bool is_refcounted_page; };/*
@@ -1265,7 +1266,6 @@ void kvm_release_pfn_dirty(kvm_pfn_t pfn);void kvm_set_pfn_dirty (kvm_pfn_t pfn);void kvm_set_pfn_accessed (kvm_pfn_t pfn); -void kvm_release_pfn (kvm_pfn_t pfn,bool dirty);int kvm_read_guest_page (struct kvm *kvm,gfn_t gfn,void *data,int offset,int len);int kvm_read_guest (struct kvm *kvm,gpa_t gpa,void *data,unsigned long len);
virt/kvm/kvm_main.c +14 -10
@@ -3188,24 +3188,22 @@ struct page *gfn_to_page(struct kvm *kvm, gfn_t gfn) }EXPORT_SYMBOL_GPL (gfn_to_page); -void kvm_release_pfn (kvm_pfn_t pfn,bool dirty) -{ -if (dirty) -kvm_release_pfn_dirty (pfn); -else -kvm_release_pfn_clean (pfn); -} -int kvm_vcpu_map (struct kvm_vcpu *vcpu,gfn_t gfn,struct kvm_host_map *map) {kvm_pfn_t pfn;void *hva =NULL ;struct page *page =KVM_UNMAPPED_PAGE ; +struct kvm_follow_pfn kfp = { + .slot =gfn_to_memslot (vcpu->kvm, gfn), + .gfn = gfn, + .flags =FOLL_GET |FOLL_WRITE , + .allow_non_refcounted_struct_page = true, + };if (!map)return -EINVAL ; - pfn =gfn_to_pfn (vcpu->kvm, gfn); + pfn =kvm_follow_pfn (&kfp);if (is_error_noslot_pfn (pfn))return -EINVAL ;
@@ -3225,6 +3223,7 @@ int kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map) map->hva = hva; map->pfn = pfn; map->gfn = gfn; + map->is_refcounted_page = !!kfp.refcounted_page;return 0 ; }
@@ -3248,7 +3247,12 @@ void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map, bool dirty)if (dirty)kvm_vcpu_mark_page_dirty (vcpu, map->gfn); -kvm_release_pfn (map->pfn, dirty); +if (map->is_refcounted_page) { +if (dirty) +kvm_release_page_dirty (map->page); +else +kvm_release_page_clean (map->page); + } map->hva =NULL ; map->page =NULL ;
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
PATCH v10 6/8 KVM: x86: Migrate to __kvm_follow_pfn
In reply to David StevensFrom: David Stevens <[email protected]>
Migrate functions which need access to is_refcounted_page to __kvm_follow_pfn. The functions which need this are __kvm_faultin_pfn and reexecute_instruction. The former requires replacing the async in/out parameter with FOLL_NOWAIT parameter and the KVM_PFN_ERR_NEEDS_IO return value. Handling non-refcounted pages is complicated, so it will be done in a followup. The latter is a straightforward refactor.
APIC related callers do not need to migrate because KVM controls the memslot, so it will always be regular memory. Prefetch related callers do not need to be migrated because atomic gfn_to_pfn calls can never make it to hva_to_pfn_remapped.
--- arch/x86/kvm/mmu/mmu.c | 43 ++++++++++++++++++++++++++++++++---------- arch/x86/kvm/x86.c | 11 +++++++++-- virt/kvm/kvm_main.c | 11 ++++------- 3 files changed, 46 insertions(+), 19 deletions(-)
arch/x86/kvm/mmu/mmu.c +33 -10
@@ -4331,7 +4331,14 @@ static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,static int __kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot; -bool async; +struct kvm_follow_pfn kfp = { + .slot = slot, + .gfn = fault->gfn, + .flags =FOLL_GET | (fault->write ?FOLL_WRITE :0 ), + .try_map_writable = true, + .guarded_by_mmu_notifier = true, + .allow_non_refcounted_struct_page = false, + };/* * Retry the page fault if the gfn hit a memslot that is being deleted
@@ -4368,12 +4375,20 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultif (fault->is_private)return kvm_faultin_pfn_private (vcpu, fault); - async = false; - fault->pfn =__gfn_to_pfn_memslot (slot, fault->gfn, false, false, &async, - fault->write, &fault->map_writable, - &fault->hva); -if (!async) -return RET_PF_CONTINUE ;/* *pfn has correct page already */ + kfp.flags |=FOLL_NOWAIT ; + fault->pfn =kvm_follow_pfn (&kfp); + +if (!is_error_noslot_pfn (fault->pfn)) + goto success; + +/* +* If kvm_follow_pfn() failed because I/O is needed to fault in the +* page, then either set up an asynchronous #PF to do the I/O, or if +* doing an async #PF isn't possible, retry kvm_follow_pfn() with +* I/O allowed. All other failures are fatal, i.e. retrying won't help. +*/ +if (fault->pfn !=KVM_PFN_ERR_NEEDS_IO ) +return RET_PF_CONTINUE ;if (!fault->prefetch &&kvm_can_do_async_pf (vcpu)) {trace_kvm_try_async_get_page (fault->addr, fault->gfn);
@@ -4391,9 +4406,17 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* to wait for IO. Note, gup always bails if it is unable to quickly * get a page and a fatal signal, i.e. SIGKILL, is pending. */ - fault->pfn =__gfn_to_pfn_memslot (slot, fault->gfn, false, true,NULL , - fault->write, &fault->map_writable, - &fault->hva); + kfp.flags |=FOLL_INTERRUPTIBLE ; + kfp.flags &= ~FOLL_NOWAIT ; + fault->pfn =kvm_follow_pfn (&kfp); + +if (!is_error_noslot_pfn (fault->pfn)) + goto success; + +return RET_PF_CONTINUE ; +success: + fault->hva = kfp.hva; + fault->map_writable = kfp.writable;return RET_PF_CONTINUE ; }
arch/x86/kvm/x86.c +9 -2
@@ -8747,6 +8747,7 @@ static bool reexecute_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, {gpa_t gpa = cr2_or_gpa;kvm_pfn_t pfn; +struct kvm_follow_pfn kfp;if (!(emulation_type &EMULTYPE_ALLOW_RETRY_PF ))return false;
@@ -8776,7 +8777,13 @@ static bool reexecute_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,* retry instruction -> write #PF -> emulation fail -> retry * instruction -> ... */ - pfn =gfn_to_pfn (vcpu->kvm,gpa_to_gfn (gpa)); + kfp = (struct kvm_follow_pfn ) { + .slot =gfn_to_memslot (vcpu->kvm,gpa_to_gfn (gpa)), + .gfn =gpa_to_gfn (gpa), + .flags =FOLL_GET |FOLL_WRITE , + .allow_non_refcounted_struct_page = true, + }; + pfn =kvm_follow_pfn (&kfp);/* * If the instruction failed on the error pfn, it can not be fixed,
@@ -8785,7 +8792,7 @@ static bool reexecute_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,if (is_error_noslot_pfn (pfn))return false; -kvm_release_pfn_clean (pfn); +kvm_release_page_clean (kfp.refcounted_page);/* The instructions are well-emulated on direct mmu. */ if (vcpu->arch.mmu->root_role.direct) {
virt/kvm/kvm_main.c +4 -7
@@ -3297,6 +3297,9 @@ void kvm_release_page_clean(struct page *page) {WARN_ON (is_error_page (page)); +if (!page) +return ; +kvm_set_page_accessed (page);put_page (page); }
@@ -3304,16 +3307,10 @@ EXPORT_SYMBOL_GPL(kvm_release_page_clean);void kvm_release_pfn_clean (kvm_pfn_t pfn) { -struct page *page; -if (is_error_noslot_pfn (pfn))return ; - page =kvm_pfn_to_refcounted_page (pfn); -if (!page) -return ; - -kvm_release_page_clean (page); +kvm_release_page_clean (kvm_pfn_to_refcounted_page (pfn)); }EXPORT_SYMBOL_GPL (kvm_release_pfn_clean);
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
PATCH v10 6/8 KVM: x86: Migrate to kvm_follow_pfn()
In reply to David StevensFrom: David Stevens <[email protected]>
Migrate functions which need to be able to map non-refcounted struct pages to kvm_follow_pfn(). These functions are kvm_faultin_pfn() and reexecute_instruction(). The former requires replacing the async in/out parameter with FOLL_NOWAIT parameter and the KVM_PFN_ERR_NEEDS_IO return value (actually handling non-refcounted pages is complicated, so it will be done in a followup). The latter is a straightforward refactor.
APIC related callers do not need to migrate because KVM controls the memslot, so it will always be regular memory. Prefetch related callers do not need to be migrated because atomic gfn_to_pfn() calls can never make it to hva_to_pfn_remapped().
--- arch/x86/kvm/mmu/mmu.c | 43 ++++++++++++++++++++++++++++++++---------- arch/x86/kvm/x86.c | 11 +++++++++-- virt/kvm/kvm_main.c | 11 ++++------- 3 files changed, 46 insertions(+), 19 deletions(-)
arch/x86/kvm/mmu/mmu.c +33 -10
@@ -4331,7 +4331,14 @@ static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,static int __kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot; -bool async; +struct kvm_follow_pfn kfp = { + .slot = slot, + .gfn = fault->gfn, + .flags =FOLL_GET | (fault->write ?FOLL_WRITE :0 ), + .try_map_writable = true, + .guarded_by_mmu_notifier = true, + .allow_non_refcounted_struct_page = false, + };/* * Retry the page fault if the gfn hit a memslot that is being deleted
@@ -4368,12 +4375,20 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultif (fault->is_private)return kvm_faultin_pfn_private (vcpu, fault); - async = false; - fault->pfn =__gfn_to_pfn_memslot (slot, fault->gfn, false, false, &async, - fault->write, &fault->map_writable, - &fault->hva); -if (!async) -return RET_PF_CONTINUE ;/* *pfn has correct page already */ + kfp.flags |=FOLL_NOWAIT ; + fault->pfn =kvm_follow_pfn (&kfp); + +if (!is_error_noslot_pfn (fault->pfn)) + goto success; + +/* +* If kvm_follow_pfn() failed because I/O is needed to fault in the +* page, then either set up an asynchronous #PF to do the I/O, or if +* doing an async #PF isn't possible, retry kvm_follow_pfn() with +* I/O allowed. All other failures are fatal, i.e. retrying won't help. +*/ +if (fault->pfn !=KVM_PFN_ERR_NEEDS_IO ) +return RET_PF_CONTINUE ;if (!fault->prefetch &&kvm_can_do_async_pf (vcpu)) {trace_kvm_try_async_get_page (fault->addr, fault->gfn);
@@ -4391,9 +4406,17 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* to wait for IO. Note, gup always bails if it is unable to quickly * get a page and a fatal signal, i.e. SIGKILL, is pending. */ - fault->pfn =__gfn_to_pfn_memslot (slot, fault->gfn, false, true,NULL , - fault->write, &fault->map_writable, - &fault->hva); + kfp.flags |=FOLL_INTERRUPTIBLE ; + kfp.flags &= ~FOLL_NOWAIT ; + fault->pfn =kvm_follow_pfn (&kfp); + +if (!is_error_noslot_pfn (fault->pfn)) + goto success; + +return RET_PF_CONTINUE ; +success: + fault->hva = kfp.hva; + fault->map_writable = kfp.writable;return RET_PF_CONTINUE ; }
arch/x86/kvm/x86.c +9 -2
@@ -8747,6 +8747,7 @@ static bool reexecute_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, {gpa_t gpa = cr2_or_gpa;kvm_pfn_t pfn; +struct kvm_follow_pfn kfp;if (!(emulation_type &EMULTYPE_ALLOW_RETRY_PF ))return false;
@@ -8776,7 +8777,13 @@ static bool reexecute_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,* retry instruction -> write #PF -> emulation fail -> retry * instruction -> ... */ - pfn =gfn_to_pfn (vcpu->kvm,gpa_to_gfn (gpa)); + kfp = (struct kvm_follow_pfn ) { + .slot =gfn_to_memslot (vcpu->kvm,gpa_to_gfn (gpa)), + .gfn =gpa_to_gfn (gpa), + .flags =FOLL_GET |FOLL_WRITE , + .allow_non_refcounted_struct_page = true, + }; + pfn =kvm_follow_pfn (&kfp);/* * If the instruction failed on the error pfn, it can not be fixed,
@@ -8785,7 +8792,7 @@ static bool reexecute_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,if (is_error_noslot_pfn (pfn))return false; -kvm_release_pfn_clean (pfn); +kvm_release_page_clean (kfp.refcounted_page);/* The instructions are well-emulated on direct mmu. */ if (vcpu->arch.mmu->root_role.direct) {
virt/kvm/kvm_main.c +4 -7
@@ -3297,6 +3297,9 @@ void kvm_release_page_clean(struct page *page) {WARN_ON (is_error_page (page)); +if (!page) +return ; +kvm_set_page_accessed (page);put_page (page); }
@@ -3304,16 +3307,10 @@ EXPORT_SYMBOL_GPL(kvm_release_page_clean);void kvm_release_pfn_clean (kvm_pfn_t pfn) { -struct page *page; -if (is_error_noslot_pfn (pfn))return ; - page =kvm_pfn_to_refcounted_page (pfn); -if (!page) -return ; - -kvm_release_page_clean (page); +kvm_release_page_clean (kvm_pfn_to_refcounted_page (pfn)); }EXPORT_SYMBOL_GPL (kvm_release_pfn_clean);
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
PATCH v10 7/8 KVM: x86/mmu: Track if sptes refer to refcounted pages
In reply to David StevensFrom: David Stevens <[email protected]>
Use one of the unused bits in EPT sptes to track whether or not an spte refers to a struct page that has a valid refcount, in preparation for adding support for mapping such pages into guests. The new bit is used to avoid triggering a page_count() == 0 warning and to avoid touching A/D bits of unknown usage.
Non-EPT sptes don't have any free bits to use, so this tracking is not possible when TDP is disabled or on 32-bit x86.
--- arch/x86/kvm/mmu/mmu.c | 43 +++++++++++++++++++--------------- arch/x86/kvm/mmu/paging_tmpl.h | 5 ++-- arch/x86/kvm/mmu/spte.c | 4 +++- arch/x86/kvm/mmu/spte.h | 22 ++++++++++++++++- arch/x86/kvm/mmu/tdp_mmu.c | 21 ++++++++++------- include/linux/kvm_host.h | 3 +++ virt/kvm/kvm_main.c | 6 +++-- 7 files changed, 70 insertions(+), 34 deletions(-)
arch/x86/kvm/mmu/mmu.c +24 -19
@@ -541,12 +541,14 @@ static bool mmu_spte_update(u64 *sptep, u64 new_spte)if (is_accessed_spte (old_spte) && !is_accessed_spte (new_spte)) { flush = true; -kvm_set_pfn_accessed (spte_to_pfn (old_spte)); +if (is_refcounted_page_spte (old_spte)) +kvm_set_page_accessed (pfn_to_page (spte_to_pfn (old_spte))); }if (is_dirty_spte (old_spte) && !is_dirty_spte (new_spte)) { flush = true; -kvm_set_pfn_dirty (spte_to_pfn (old_spte)); +if (is_refcounted_page_spte (old_spte)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (old_spte))); }return flush;
@@ -578,20 +580,23 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep) pfn =spte_to_pfn (old_spte); -/* -* KVM doesn't hold a reference to any pages mapped into the guest, and -* instead uses the mmu_notifier to ensure that KVM unmaps any pages -* before they are reclaimed. Sanity check that, if the pfn is backed -* by a refcounted page, the refcount is elevated. -*/ - page =kvm_pfn_to_refcounted_page (pfn); -WARN_ON_ONCE (page && !page_count (page)); +if (is_refcounted_page_spte (old_spte)) { +/* +* KVM doesn't hold a reference to any pages mapped into the +* guest, and instead uses the mmu_notifier to ensure that KVM +* unmaps any pages before they are reclaimed. Sanity check +* that, if the pfn is backed by a refcounted page, the +* refcount is elevated. +*/ + page =kvm_pfn_to_refcounted_page (pfn); +WARN_ON_ONCE (!page || !page_count (page)); -if (is_accessed_spte (old_spte)) -kvm_set_pfn_accessed (pfn); +if (is_accessed_spte (old_spte)) +kvm_set_page_accessed (pfn_to_page (pfn)); -if (is_dirty_spte (old_spte)) -kvm_set_pfn_dirty (pfn); +if (is_dirty_spte (old_spte)) +kvm_set_page_dirty (pfn_to_page (pfn)); + }return old_spte; }
@@ -627,8 +632,8 @@ static bool mmu_spte_age(u64 *sptep)* Capture the dirty status of the page, so that it doesn't get * lost when the SPTE is marked for access tracking. */ -if (is_writable_pte (spte)) -kvm_set_pfn_dirty (spte_to_pfn (spte)); +if (is_writable_pte (spte) &&is_refcounted_page_spte (spte)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (spte))); spte =mark_spte_for_access_track (spte);mmu_spte_update_no_track (sptep, spte);
@@ -1267,8 +1272,8 @@ static bool spte_wrprot_for_clear_dirty(u64 *sptep) {bool was_writable =test_and_clear_bit (PT_WRITABLE_SHIFT , (unsigned long *)sptep); -if (was_writable && !spte_ad_enabled (*sptep)) -kvm_set_pfn_dirty (spte_to_pfn (*sptep)); +if (was_writable && !spte_ad_enabled (*sptep) &&is_refcounted_page_spte (*sptep)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (*sptep)));return was_writable; }
@@ -2946,7 +2951,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, } wrprot =make_spte (vcpu, sp, slot, pte_access, gfn, pfn, *sptep, prefetch, - true, host_writable, &spte); + true, host_writable, true, &spte);if (*sptep == spte) { ret =RET_PF_SPURIOUS ;
arch/x86/kvm/mmu/paging_tmpl.h +3 -2
@@ -902,7 +902,7 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,*/ static int FNAME (sync_spte )(struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp,int i) { -bool host_writable; +bool host_writable, is_refcounted;gpa_t first_pte_gpa;u64 *sptep, spte;struct kvm_memory_slot *slot;
@@ -959,10 +959,11 @@ static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int sptep = &sp->spt[i]; spte = *sptep; host_writable = spte & shadow_host_writable_mask; + is_refcounted =is_refcounted_page_spte (spte); slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn);make_spte (vcpu, sp, slot, pte_access, gfn,spte_to_pfn (spte), spte, true, false, - host_writable, &spte); + host_writable, is_refcounted, &spte);return mmu_spte_update (sptep, spte); }
arch/x86/kvm/mmu/spte.c +3 -1
@@ -138,7 +138,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,const struct kvm_memory_slot *slot, unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn, u64 old_spte, bool prefetch, bool can_unsync, - bool host_writable, u64 *new_spte) + bool host_writable, bool is_refcounted, u64 *new_spte) { int level = sp->role.level;u64 spte =SPTE_MMU_PRESENT_MASK ;
@@ -188,6 +188,8 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,if (level >PG_LEVEL_4K ) spte |=PT_PAGE_SIZE_MASK ; +if (spte_has_refcount_bit () && is_refcounted) + spte |=SPTE_MMU_PAGE_REFCOUNTED ;if (shadow_memtype_mask) spte |= static_call(kvm_x86_get_mt_mask)(vcpu, gfn,
arch/x86/kvm/mmu/spte.h +21 -1
@@ -96,6 +96,11 @@ static_assert(!(EPT_SPTE_MMU_WRITABLE & SHADOW_ACC_TRACK_SAVED_MASK));/* Defined only to keep the above static asserts readable. */ #undef SHADOW_ACC_TRACK_SAVED_MASK +/* +* Indicates that the SPTE refers to a page with a valid refcount. +*/ +#define SPTE_MMU_PAGE_REFCOUNTED BIT_ULL(59) +/* * Due to limited space in PTEs, the MMIO generation is a 19 bit subset of * the memslots generation and is derived as follows:
@@ -345,6 +350,21 @@ static inline bool is_dirty_spte(u64 spte)return dirty_mask ? spte & dirty_mask : spte &PT_WRITABLE_MASK ; } +/* +* Extra bits are only available for TDP SPTEs, since bits 62:52 are reserved +* for PAE paging, including NPT PAE. When a tracking bit isn't available, we +* will reject mapping non-refcounted struct pages. +*/ +static inline bool spte_has_refcount_bit (void ) +{ +return tdp_enabled &&IS_ENABLED (CONFIG_X86_64 ); +} + +static inline bool is_refcounted_page_spte (u64 spte) +{ +return !spte_has_refcount_bit () || (spte &SPTE_MMU_PAGE_REFCOUNTED ); +} +static inline u64 get_rsvd_bits (struct rsvd_bits_validate *rsvd_check,u64 pte,int level) {
@@ -475,7 +495,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,const struct kvm_memory_slot *slot, unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn, u64 old_spte, bool prefetch, bool can_unsync, - bool host_writable, u64 *new_spte); + bool host_writable, bool is_refcounted, u64 *new_spte);u64 make_huge_page_split_spte (struct kvm *kvm,u64 huge_spte,union kvm_mmu_page_role role,int index);u64 make_nonleaf_spte (u64 *child_pt,bool ad_disabled);
arch/x86/kvm/mmu/tdp_mmu.c +12 -9
@@ -414,6 +414,7 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,bool was_leaf = was_present &&is_last_spte (old_spte, level);bool is_leaf = is_present &&is_last_spte (new_spte, level);bool pfn_changed =spte_to_pfn (old_spte) !=spte_to_pfn (new_spte); +bool is_refcounted =is_refcounted_page_spte (old_spte);WARN_ON_ONCE (level >PT64_ROOT_MAX_LEVEL );WARN_ON_ONCE (level <PG_LEVEL_4K );
@@ -478,9 +479,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,if (is_leaf != was_leaf)kvm_update_page_stats (kvm, level, is_leaf ?1 :-1 ); -if (was_leaf &&is_dirty_spte (old_spte) && +if (was_leaf &&is_dirty_spte (old_spte) && is_refcounted && (!is_present || !is_dirty_spte (new_spte) || pfn_changed)) -kvm_set_pfn_dirty (spte_to_pfn (old_spte)); +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (old_spte)));/* * Recursively handle child PTs if the change removed a subtree from
@@ -492,9 +493,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, (is_leaf || !is_present || WARN_ON_ONCE(pfn_changed)))handle_removed_pt (kvm,spte_to_child_pt (old_spte, level), shared); -if (was_leaf &&is_accessed_spte (old_spte) && +if (was_leaf &&is_accessed_spte (old_spte) && is_refcounted && (!is_present || !is_accessed_spte (new_spte) || pfn_changed)) -kvm_set_pfn_accessed (spte_to_pfn (old_spte)); +kvm_set_page_accessed (pfn_to_page (spte_to_pfn (old_spte))); }/*
@@ -956,8 +957,8 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, new_spte =make_mmio_spte (vcpu, iter->gfn,ACC_ALL ); else wrprot =make_spte (vcpu, sp, fault->slot,ACC_ALL , iter->gfn, - fault->pfn, iter->old_spte, fault->prefetch, true, - fault->map_writable, &new_spte); + fault->pfn, iter->old_spte, fault->prefetch, true, + fault->map_writable, true, &new_spte);if (new_spte == iter->old_spte) ret =RET_PF_SPURIOUS ;
@@ -1178,8 +1179,9 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter,* Capture the dirty status of the page, so that it doesn't get * lost when the SPTE is marked for access tracking. */ -if (is_writable_pte (iter->old_spte)) -kvm_set_pfn_dirty (spte_to_pfn (iter->old_spte)); +if (is_writable_pte (iter->old_spte) && +is_refcounted_page_spte (iter->old_spte)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (iter->old_spte))); new_spte =mark_spte_for_access_track (iter->old_spte); iter->old_spte = kvm_tdp_mmu_write_spte(iter->sptep,
@@ -1602,7 +1604,8 @@ static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root,trace_kvm_tdp_mmu_spte_changed (iter.as_id, iter.gfn, iter.level, iter.old_spte, iter.old_spte & ~dbit); -kvm_set_pfn_dirty (spte_to_pfn (iter.old_spte)); +if (is_refcounted_page_spte (iter.old_spte)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (iter.old_spte))); }rcu_read_unlock ();
include/linux/kvm_host.h +3 -0
@@ -1211,6 +1211,9 @@ unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot, gfn_t gfn,void kvm_release_page_clean (struct page *page);void kvm_release_page_dirty (struct page *page); +void kvm_set_page_accessed (struct page *page); +void kvm_set_page_dirty (struct page *page); +struct kvm_follow_pfn {const struct kvm_memory_slot *slot;gfn_t gfn;
virt/kvm/kvm_main.c +4 -2
@@ -3281,17 +3281,19 @@ static bool kvm_is_ad_tracked_page(struct page *page)return !PageReserved (page); } -static void kvm_set_page_dirty (struct page *page) +void kvm_set_page_dirty (struct page *page) {if (kvm_is_ad_tracked_page (page))SetPageDirty (page); } +EXPORT_SYMBOL_GPL (kvm_set_page_dirty); -static void kvm_set_page_accessed (struct page *page) +void kvm_set_page_accessed (struct page *page) {if (kvm_is_ad_tracked_page (page))mark_page_accessed (page); } +EXPORT_SYMBOL_GPL (kvm_set_page_accessed);void kvm_release_page_clean (struct page *page) {
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Paolo Bonzini
maintainerRe: [PATCH v10 7/8] KVM: x86/mmu: Track if sptes refer to refcounted pages
Quoted 10 lines · David Stevens
From: David Stevens <[email protected]>
Use one of the unused bits in EPT sptes to track whether or not an spte refers to a struct page that has a valid refcount, in preparation for adding support for mapping such pages into guests. The new bit is used to avoid triggering a
page_count()== 0 warning and to avoid touching A/D bits of unknown usage.Non-EPT sptes don't have any free bits to use, so this tracking is not possible when TDP is disabled or on 32-bit x86.
TDX will add support for non-zero non-present PTEs. We could use this to use inverted bit 8 to mark present PTEs (bit 8 set for non-present, bit 8 clear for present) for both shadow paging and AMD NPT. This would free bit 11 for SPTE_MMU_PAGE_REFCOUNTED.
No need to do it now, though.
Paolo
Quoted 303 lines
Signed-off-by David Stevens <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 43 +++++++++++++++++++--------------- arch/x86/kvm/mmu/paging_tmpl.h | 5 ++-- arch/x86/kvm/mmu/spte.c | 4 +++- arch/x86/kvm/mmu/spte.h | 22 ++++++++++++++++- arch/x86/kvm/mmu/tdp_mmu.c | 21 ++++++++++------- include/linux/kvm_host.h | 3 +++ virt/kvm/kvm_main.c | 6 +++-- 7 files changed, 70 insertions(+), 34 deletions(-)arch/x86/kvm/mmu/mmu.c +24
-19@@ -541,12 +541,14 @@ static bool mmu_spte_update(u64 *sptep, u64 new_spte)if (is_accessed_spte (old_spte) && !is_accessed_spte (new_spte)) { flush = true; -kvm_set_pfn_accessed (spte_to_pfn (old_spte)); +if (is_refcounted_page_spte (old_spte)) +kvm_set_page_accessed (pfn_to_page (spte_to_pfn (old_spte))); }if (is_dirty_spte (old_spte) && !is_dirty_spte (new_spte)) { flush = true; -kvm_set_pfn_dirty (spte_to_pfn (old_spte)); +if (is_refcounted_page_spte (old_spte)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (old_spte))); }return flush;@@ -578,20 +580,23 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep) pfn =spte_to_pfn (old_spte); -/* -* KVM doesn't hold a reference to any pages mapped into the guest, and -* instead uses the mmu_notifier to ensure that KVM unmaps any pages -* before they are reclaimed. Sanity check that, if the pfn is backed -* by a refcounted page, the refcount is elevated. -*/ - page =kvm_pfn_to_refcounted_page (pfn); -WARN_ON_ONCE (page && !page_count (page)); +if (is_refcounted_page_spte (old_spte)) { +/* +* KVM doesn't hold a reference to any pages mapped into the +* guest, and instead uses the mmu_notifier to ensure that KVM +* unmaps any pages before they are reclaimed. Sanity check +* that, if the pfn is backed by a refcounted page, the +* refcount is elevated. +*/ + page =kvm_pfn_to_refcounted_page (pfn); +WARN_ON_ONCE (!page || !page_count (page)); -if (is_accessed_spte (old_spte)) -kvm_set_pfn_accessed (pfn); +if (is_accessed_spte (old_spte)) +kvm_set_page_accessed (pfn_to_page (pfn)); -if (is_dirty_spte (old_spte)) -kvm_set_pfn_dirty (pfn); +if (is_dirty_spte (old_spte)) +kvm_set_page_dirty (pfn_to_page (pfn)); + }return old_spte; }@@ -627,8 +632,8 @@ static bool mmu_spte_age(u64 *sptep)* Capture the dirty status of the page, so that it doesn't get * lost when the SPTE is marked for access tracking. */ -if (is_writable_pte (spte)) -kvm_set_pfn_dirty (spte_to_pfn (spte)); +if (is_writable_pte (spte) &&is_refcounted_page_spte (spte)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (spte))); spte =mark_spte_for_access_track (spte);mmu_spte_update_no_track (sptep, spte);@@ -1267,8 +1272,8 @@ static bool spte_wrprot_for_clear_dirty(u64 *sptep) {bool was_writable =test_and_clear_bit (PT_WRITABLE_SHIFT , (unsigned long *)sptep); -if (was_writable && !spte_ad_enabled (*sptep)) -kvm_set_pfn_dirty (spte_to_pfn (*sptep)); +if (was_writable && !spte_ad_enabled (*sptep) &&is_refcounted_page_spte (*sptep)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (*sptep)));return was_writable; }@@ -2946,7 +2951,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, } wrprot =make_spte (vcpu, sp, slot, pte_access, gfn, pfn, *sptep, prefetch, - true, host_writable, &spte); + true, host_writable, true, &spte);if (*sptep == spte) { ret =RET_PF_SPURIOUS ;arch/x86/kvm/mmu/paging_tmpl.h +3
-2@@ -902,7 +902,7 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,*/ static int FNAME (sync_spte )(struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp,int i) { -bool host_writable; +bool host_writable, is_refcounted;gpa_t first_pte_gpa;u64 *sptep, spte;struct kvm_memory_slot *slot;@@ -959,10 +959,11 @@ static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int sptep = &sp->spt[i]; spte = *sptep; host_writable = spte & shadow_host_writable_mask; + is_refcounted =is_refcounted_page_spte (spte); slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn);make_spte (vcpu, sp, slot, pte_access, gfn,spte_to_pfn (spte), spte, true, false, - host_writable, &spte); + host_writable, is_refcounted, &spte);return mmu_spte_update (sptep, spte); }arch/x86/kvm/mmu/spte.c +3
-1@@ -138,7 +138,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,const struct kvm_memory_slot *slot, unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn, u64 old_spte, bool prefetch, bool can_unsync, - bool host_writable, u64 *new_spte) + bool host_writable, bool is_refcounted, u64 *new_spte) { int level = sp->role.level;u64 spte =SPTE_MMU_PRESENT_MASK ;@@ -188,6 +188,8 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,if (level >PG_LEVEL_4K ) spte |=PT_PAGE_SIZE_MASK ; +if (spte_has_refcount_bit () && is_refcounted) + spte |=SPTE_MMU_PAGE_REFCOUNTED ;if (shadow_memtype_mask) spte |= static_call(kvm_x86_get_mt_mask)(vcpu, gfn,arch/x86/kvm/mmu/spte.h +21
-1@@ -96,6 +96,11 @@ static_assert(!(EPT_SPTE_MMU_WRITABLE & SHADOW_ACC_TRACK_SAVED_MASK));/* Defined only to keep the above static asserts readable. */ #undef SHADOW_ACC_TRACK_SAVED_MASK +/* +* Indicates that the SPTE refers to a page with a valid refcount. +*/ +#define SPTE_MMU_PAGE_REFCOUNTED BIT_ULL(59) +/* * Due to limited space in PTEs, the MMIO generation is a 19 bit subset of * the memslots generation and is derived as follows: @@ -345,6 +350,21 @@ static inline bool is_dirty_spte(u64 spte)return dirty_mask ? spte & dirty_mask : spte &PT_WRITABLE_MASK ; } +/* +* Extra bits are only available for TDP SPTEs, since bits 62:52 are reserved +* for PAE paging, including NPT PAE. When a tracking bit isn't available, we +* will reject mapping non-refcounted struct pages. +*/ +static inline bool spte_has_refcount_bit (void ) +{ +return tdp_enabled &&IS_ENABLED (CONFIG_X86_64 ); +} + +static inline bool is_refcounted_page_spte (u64 spte) +{ +return !spte_has_refcount_bit () || (spte &SPTE_MMU_PAGE_REFCOUNTED ); +} +static inline u64 get_rsvd_bits (struct rsvd_bits_validate *rsvd_check,u64 pte,int level) {@@ -475,7 +495,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,const struct kvm_memory_slot *slot, unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn, u64 old_spte, bool prefetch, bool can_unsync, - bool host_writable, u64 *new_spte); + bool host_writable, bool is_refcounted, u64 *new_spte);u64 make_huge_page_split_spte (struct kvm *kvm,u64 huge_spte,union kvm_mmu_page_role role,int index);u64 make_nonleaf_spte (u64 *child_pt,bool ad_disabled);arch/x86/kvm/mmu/tdp_mmu.c +12
-9@@ -414,6 +414,7 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,bool was_leaf = was_present &&is_last_spte (old_spte, level);bool is_leaf = is_present &&is_last_spte (new_spte, level);bool pfn_changed =spte_to_pfn (old_spte) !=spte_to_pfn (new_spte); +bool is_refcounted =is_refcounted_page_spte (old_spte);WARN_ON_ONCE (level >PT64_ROOT_MAX_LEVEL );WARN_ON_ONCE (level <PG_LEVEL_4K );@@ -478,9 +479,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,if (is_leaf != was_leaf)kvm_update_page_stats (kvm, level, is_leaf ?1 :-1 ); -if (was_leaf &&is_dirty_spte (old_spte) && +if (was_leaf &&is_dirty_spte (old_spte) && is_refcounted && (!is_present || !is_dirty_spte (new_spte) || pfn_changed)) -kvm_set_pfn_dirty (spte_to_pfn (old_spte)); +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (old_spte)));/* * Recursively handle child PTs if the change removed a subtree from @@ -492,9 +493,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, (is_leaf || !is_present || WARN_ON_ONCE(pfn_changed)))handle_removed_pt (kvm,spte_to_child_pt (old_spte, level), shared); -if (was_leaf &&is_accessed_spte (old_spte) && +if (was_leaf &&is_accessed_spte (old_spte) && is_refcounted && (!is_present || !is_accessed_spte (new_spte) || pfn_changed)) -kvm_set_pfn_accessed (spte_to_pfn (old_spte)); +kvm_set_page_accessed (pfn_to_page (spte_to_pfn (old_spte))); }/* @@ -956,8 +957,8 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, new_spte =make_mmio_spte (vcpu, iter->gfn,ACC_ALL ); else wrprot =make_spte (vcpu, sp, fault->slot,ACC_ALL , iter->gfn, - fault->pfn, iter->old_spte, fault->prefetch, true, - fault->map_writable, &new_spte); + fault->pfn, iter->old_spte, fault->prefetch, true, + fault->map_writable, true, &new_spte);if (new_spte == iter->old_spte) ret =RET_PF_SPURIOUS ;@@ -1178,8 +1179,9 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter,* Capture the dirty status of the page, so that it doesn't get * lost when the SPTE is marked for access tracking. */ -if (is_writable_pte (iter->old_spte)) -kvm_set_pfn_dirty (spte_to_pfn (iter->old_spte)); +if (is_writable_pte (iter->old_spte) && +is_refcounted_page_spte (iter->old_spte)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (iter->old_spte))); new_spte =mark_spte_for_access_track (iter->old_spte); iter->old_spte = kvm_tdp_mmu_write_spte(iter->sptep,@@ -1602,7 +1604,8 @@ static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root,trace_kvm_tdp_mmu_spte_changed (iter.as_id, iter.gfn, iter.level, iter.old_spte, iter.old_spte & ~dbit); -kvm_set_pfn_dirty (spte_to_pfn (iter.old_spte)); +if (is_refcounted_page_spte (iter.old_spte)) +kvm_set_page_dirty (pfn_to_page (spte_to_pfn (iter.old_spte))); }rcu_read_unlock ();include/linux/kvm_host.h +3
-0@@ -1211,6 +1211,9 @@ unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot, gfn_t gfn,void kvm_release_page_clean (struct page *page);void kvm_release_page_dirty (struct page *page); +void kvm_set_page_accessed (struct page *page); +void kvm_set_page_dirty (struct page *page); +struct kvm_follow_pfn {const struct kvm_memory_slot *slot;gfn_t gfn;virt/kvm/kvm_main.c +4
-2@@ -3281,17 +3281,19 @@ static bool kvm_is_ad_tracked_page(struct page *page)return !PageReserved (page); } -static void kvm_set_page_dirty (struct page *page) +void kvm_set_page_dirty (struct page *page) {if (kvm_is_ad_tracked_page (page))SetPageDirty (page); } +EXPORT_SYMBOL_GPL (kvm_set_page_dirty); -static void kvm_set_page_accessed (struct page *page) +void kvm_set_page_accessed (struct page *page) {if (kvm_is_ad_tracked_page (page))mark_page_accessed (page); } +EXPORT_SYMBOL_GPL (kvm_set_page_accessed);void kvm_release_page_clean (struct page *page) {
Sean Christopherson
maintainerRe: [PATCH v10 7/8] KVM: x86/mmu: Track if sptes refer to refcounted pages
Quoted 9 lines · Paolo Bonzini
Quoted 8 lines · David Stevens
From: David Stevens <[email protected]>
Use one of the unused bits in EPT sptes to track whether or not an spte refers to a struct page that has a valid refcount, in preparation for adding support for mapping such pages into guests. The new bit is used to avoid triggering a
page_count()== 0 warning and to avoid touching A/D bits of unknown usage.
Quoted 7 lines
Quoted 2 lines
Non-EPT sptes don't have any free bits to use, so this tracking is not possible when TDP is disabled or on 32-bit x86.
TDX will add support for non-zero non-present PTEs. We could use this to use inverted bit 8 to mark present PTEs (bit 8 set for non-present, bit 8 clear for present) for both shadow paging and AMD NPT. This would free bit 11 for
SPTE_MMU_PAGE_REFCOUNTED.
Ooh, that's much more clever than where I was headed, which was to abuse the PAT bit (kernel configures 0400 to be WB), which would actually be quite elegant if
(a) the PAT bit didn't move around or (b) the kernel used a non-zero PCD or PWT bit for WB :-/
Sean Christopherson
maintainerRe: [PATCH v10 7/8] KVM: x86/mmu: Track if sptes refer to refcounted pages
In reply to David Stevensarch/x86/kvm/mmu/spte.h static inline bool is_dirty_spte(u64 spte) · line 365
Quoted 12 lines · David Stevens
+ * Extra bits are only available for TDP SPTEs, since bits 62:52 are reserved
+ * for PAE paging, including NPT PAE. When a tracking bit isn't available, we
+ * will reject mapping non-refcounted struct pages.
+ */arch/x86/kvm/mmu/spte.h +8
-0+static inline bool spte_has_refcount_bit (void ) +{ +return tdp_enabled &&IS_ENABLED (CONFIG_X86_64 ); +} + +static inline bool is_refcounted_page_spte (u64 spte) +{ +return !spte_has_refcount_bit () || (spte &SPTE_MMU_PAGE_REFCOUNTED );
The preferred way to handle this in KVM's MMU is to use a global variable, e.g.
extern u64 __read_mostly shadow_refcounted_mask;
that avoids conditional branches when creating SPTEs, and arguably yields slightly more readable code, e.g.
return !shadow_refcounted_mask || (spte & shadow_refcounted_mask);
PATCH v10 8/8 KVM: x86/mmu: Handle non-refcounted pages
In reply to David StevensFrom: David Stevens <[email protected]>
Handle non-refcounted pages in kvm_faultin_pfn(). This allows the host to map memory into the guest that is backed by non-refcounted struct pages - for example, the tail pages of higher order non-compound pages allocated by the amdgpu driver via ttm_pool_alloc_page.
--- arch/x86/kvm/mmu/mmu.c | 24 +++++++++++++++++------- arch/x86/kvm/mmu/mmu_internal.h | 2 ++ arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 3 ++- include/linux/kvm_host.h | 6 ++++-- virt/kvm/guest_memfd.c | 8 ++++---- virt/kvm/kvm_main.c | 10 ++++++++-- 7 files changed, 38 insertions(+), 17 deletions(-)
arch/x86/kvm/mmu/mmu.c +17 -7
@@ -2924,6 +2924,11 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot,bool host_writable = !fault || fault->map_writable;bool prefetch = !fault || fault->prefetch;bool write_fault = fault && fault->write; +/* +* Prefetching uses gfn_to_page_many_atomic, which never gets +* non-refcounted pages. +*/ +bool is_refcounted = !fault || !!fault->accessed_page;if (unlikely (is_noslot_pfn (pfn))) { vcpu->stat.pf_mmio_spte_created++;
@@ -2951,7 +2956,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, } wrprot =make_spte (vcpu, sp, slot, pte_access, gfn, pfn, *sptep, prefetch, - true, host_writable, true, &spte); + true, host_writable, is_refcounted, &spte);if (*sptep == spte) { ret =RET_PF_SPURIOUS ;
@@ -4319,8 +4324,8 @@ static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,return -EFAULT ; } - r =kvm_gmem_get_pfn (vcpu->kvm, fault->slot, fault->gfn, &fault->pfn, - &max_order); + r =kvm_gmem_get_pfn (vcpu->kvm, fault->slot, fault->gfn, + &fault->pfn, &fault->accessed_page, &max_order);if (r) {kvm_mmu_prepare_memory_fault_exit (vcpu, fault);return r;
@@ -4330,6 +4335,9 @@ static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu, fault->max_level); fault->map_writable = !(fault->slot->flags &KVM_MEM_READONLY ); +/* kvm_gmem_get_pfn takes a refcount, but accessed_page doesn't need it. */ +put_page (fault->accessed_page); +return RET_PF_CONTINUE ; }
@@ -4339,10 +4347,10 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultstruct kvm_follow_pfn kfp = { .slot = slot, .gfn = fault->gfn, - .flags =FOLL_GET | (fault->write ?FOLL_WRITE :0 ), + .flags = fault->write ?FOLL_WRITE :0 , .try_map_writable = true, .guarded_by_mmu_notifier = true, - .allow_non_refcounted_struct_page = false, + .allow_non_refcounted_struct_page =spte_has_refcount_bit (), };/*
@@ -4359,6 +4367,7 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault fault->slot =NULL ; fault->pfn =KVM_PFN_NOSLOT ; fault->map_writable = false; + fault->accessed_page =NULL ;return RET_PF_CONTINUE ; }/*
@@ -4422,6 +4431,7 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault success: fault->hva = kfp.hva; fault->map_writable = kfp.writable; + fault->accessed_page = kfp.refcounted_page;return RET_PF_CONTINUE ; }
@@ -4510,8 +4520,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault r =direct_map (vcpu, fault); out_unlock: +kvm_set_page_accessed (fault->accessed_page);write_unlock (&vcpu->kvm->mmu_lock); -kvm_release_pfn_clean (fault->pfn);return r; }
@@ -4586,8 +4596,8 @@ static int kvm_tdp_mmu_page_fault(struct kvm_vcpu *vcpu, r =kvm_tdp_mmu_map (vcpu, fault); out_unlock: +kvm_set_page_accessed (fault->accessed_page);read_unlock (&vcpu->kvm->mmu_lock); -kvm_release_pfn_clean (fault->pfn); return r; }#endif
arch/x86/kvm/mmu/mmu_internal.h +2 -0
@@ -240,6 +240,8 @@ struct kvm_page_fault {kvm_pfn_t pfn;hva_t hva;bool map_writable; +/* Does NOT have an elevated refcount */ +struct page *accessed_page;/* * Indicates the guest is trying to write a gfn that contains one or
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
@@ -847,8 +847,8 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault r =FNAME (fetch)(vcpu, fault, &walker); out_unlock: +kvm_set_page_accessed (fault->accessed_page);write_unlock (&vcpu->kvm->mmu_lock); -kvm_release_pfn_clean (fault->pfn);return r; }
arch/x86/kvm/mmu/tdp_mmu.c +2 -1
@@ -958,7 +958,8 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,else wrprot =make_spte (vcpu, sp, fault->slot,ACC_ALL , iter->gfn, fault->pfn, iter->old_spte, fault->prefetch, true, - fault->map_writable, true, &new_spte); + fault->map_writable, !!fault->accessed_page, + &new_spte);if (new_spte == iter->old_spte) ret =RET_PF_SPURIOUS ;
include/linux/kvm_host.h +4 -2
@@ -2421,11 +2421,13 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)#ifdef CONFIG_KVM_PRIVATE_MEM int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot, -gfn_t gfn,kvm_pfn_t *pfn,int *max_order); +gfn_t gfn,kvm_pfn_t *pfn,struct page **page, +int *max_order);#else static inline int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn, -kvm_pfn_t *pfn,int *max_order) +kvm_pfn_t *pfn,struct page **page, +int *max_order) {KVM_BUG_ON (1 , kvm);return -EIO ;
virt/kvm/guest_memfd.c +4 -4
@@ -483,12 +483,12 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot) }int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot, -gfn_t gfn,kvm_pfn_t *pfn,int *max_order) +gfn_t gfn,kvm_pfn_t *pfn,struct page **page, +int *max_order) {pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff;struct kvm_gmem *gmem;struct folio *folio; -struct page *page;struct file *file;int r;
@@ -514,9 +514,9 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, goto out_unlock; } - page =folio_file_page (folio, index); + *page =folio_file_page (folio, index); - *pfn =page_to_pfn (page); + *pfn =page_to_pfn (*page);if (max_order) *max_order =0 ;
virt/kvm/kvm_main.c +8 -2
@@ -3288,11 +3288,17 @@ void kvm_set_page_dirty(struct page *page) }EXPORT_SYMBOL_GPL (kvm_set_page_dirty); -void kvm_set_page_accessed (struct page *page) +static void __kvm_set_page_accessed (struct page *page) {if (kvm_is_ad_tracked_page (page))mark_page_accessed (page); } + +void kvm_set_page_accessed (struct page *page) +{ +if (page) +__kvm_set_page_accessed (page); +}EXPORT_SYMBOL_GPL (kvm_set_page_accessed);void kvm_release_page_clean (struct page *page)
@@ -3302,7 +3308,7 @@ void kvm_release_page_clean(struct page *page)if (!page)return ; -kvm_set_page_accessed (page); +__kvm_set_page_accessed (page);put_page (page); }EXPORT_SYMBOL_GPL (kvm_release_page_clean);
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
Paolo Bonzini
maintainerQuoted 28 lines · David Stevens
From: David Stevens <[email protected]>
This patch series adds support for mapping
VM_IOandVM_PFNMAPmemory that is backed by struct pages that aren't currently being refcounted (e.g. tail pages of non-compound higher order allocations) into the guest.Our use case is virtio-gpu blob resources [1], which directly map host graphics buffers into the guest as "vram" for the virtio-gpu device. This feature currently does not work on systems using the amdgpu driver, as that driver allocates non-compound higher order pages via
ttm_pool_alloc_page().First, this series replaces the
gfn_to_pfn_memslot()API with a more extensiblekvm_follow_pfn()API. The updated API rearranges gfn_to_pfn_memslot()'s args into a struct and where possible packs the bool arguments into aFOLL_flags argument. The refactoring changes do not change any behavior.From there, this series extends the
kvm_follow_pfn()API so that
non-refconuted pages can be safely handled. This invloves adding an input parameter to indicate whether the caller can safely use non-refcounted pfns and an output parameter to tell the caller whether or not the returned page is refcounted. This change includes a breaking change, by disallowing non-refcounted pfn mappings by default, as such mappings are unsafe. To allow such systems to continue to function, an opt-in module parameter is added to allow the unsafe behavior.
Quoted 4 lines
This series only adds support for non-refcounted pages to x86. Other MMUs can likely be updated without too much difficulty, but it is not needed at this point. Updating other parts of KVM (e.g. pfncache) is not straightforward [2].
Looks good to me, apart that two patches were sent twice. I only have a small comment on patch 4, to which I'll reply separately.
Paolo
Quoted 61 lines
[1]
https://patchwork.kernel.org/project/dri-devel/cover/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/v9 -> v10:
- Re-addFOLL_GETchanges.
- Split x86/mmu spte+non-refcount-page patch into two patches.
- Rename 'foll' variables to 'kfp'.
- Properly gate usage of refcount spte bit when it's not available.
- Replace kfm_follow_pfn'sis_refcounted_pageoutput parameter with a struct page*refcounted_pagepointing to the page in question.
- Add patch downgradingBUG_ONtoWARN_ON_ONCE.
v8 -> v9:
- Make paying attention tois_refcounted_pagemandatory. This means thatFOLL_GETis no longer necessary. For compatibility with un-migrated callers, add a temporary parameter to sidestep ref-counting issues.
- Addallow_unsafe_mappings, which is a breaking change.
- Migratekvm_vcpu_mapand other callsites used by x86 to the new API.
- Drop arm and ppc changes.
v7 -> v8:
- Set access bits before releasingmmu_lock.
- PassFOLL_GETon 32-bit x86 or !tdp_enabled.
- RefactorFOLL_GEThandling, addkvm_follow_refcounted_pfnhelper.
- Set refcounted bit on >4k pages.
- Add comments and apply formatting suggestions.
- rebase on kvm next branch.
v6 -> v7:
- Replace__gfn_to_pfn_memslotwith a more flexible__kvm_faultin_pfn, and extend that API to support non-refcounted pages (complete rewrite).David Stevens (7):
KVM: RelaxBUG_ONargument validation
KVM: mmu: Introducekvm_follow_pfn()
KVM: mmu: Improve handling of non-refcounted pfns
KVM: Migratekvm_vcpu_map()tokvm_follow_pfn()
KVM: x86: Migrate tokvm_follow_pfn()
KVM: x86/mmu: Track if sptes refer to refcounted pages
KVM: x86/mmu: Handle non-refcounted pagesSean Christopherson (1):
KVM: Assert that a page's refcount is elevated when marking
accessed/dirtyarch/x86/kvm/mmu/mmu.c | 104 +++++++--- arch/x86/kvm/mmu/mmu_internal.h | 2 + arch/x86/kvm/mmu/paging_tmpl.h | 7 +- arch/x86/kvm/mmu/spte.c | 4 +- arch/x86/kvm/mmu/spte.h | 22 +- arch/x86/kvm/mmu/tdp_mmu.c | 22 +- arch/x86/kvm/x86.c | 11 +- include/linux/kvm_host.h | 53 ++++- virt/kvm/guest_memfd.c | 8 +- virt/kvm/kvm_main.c | 349 +++++++++++++++++++------------- virt/kvm/kvm_mm.h | 3 +- virt/kvm/pfncache.c | 11 +- 12 files changed, 399 insertions(+), 197 deletions(-)base-commit:
54be6c6c5ae8e0d93a6c4641cb7528eb0b6ba478