KVM: Scalable memslots implementation
v6 cover KVM: Scalable memslots implementation
From: "Maciej S. Szmigiero" <[email protected]>
This series contains the sixth iteration of the scalable memslots patch set. It is based on Sean's version "5.5", but with integrated patches for issues that arose during its review round.
In addition to that, the kvm_for_each_memslot_in_gfn_range() implementation was reworked to return only strictly overlapping memslots and to use iterators.
However, I've dropped a similar kvm_for_each_memslot_in_hva_range() rework since the existing implementation was already returning only strictly overlapping memslots and it was already based on interval tree iterators, so wrapping them in another layer of iterators would only add unnecessary complexity.
The code in this "for"-like macro is also self-contained and very simple, so let's keep it this way.
The series was tested on x86 with KASAN on and booted various guests successfully (including nested ones; with TDP MMU both enabled and disabled).
Sean's previous version (5.5) is available here:
[1]: https://lore.kernel.org/kvm/[email protected]/
Changes from v5.5:
* Note open-coding of kvm_copy_memslots() in the commit message of patch 3,
* When changing kvm_arch_prepare_memory_region() signature change it also in the RISC-V implementation of this function in patch 5,
* Avoid overflowing "nr_mmu_pages" in patch 18,
* Add a comment to the "slot id to memslot" hash table declaration explaining its bucket count in patch 21,
* Don't check twice for "old" being != NULL in kvm_replace_memslot() in patch 21,
* Remove unnecessary "new" NULL check in kvm_replace_memslot() in patch 22,
* Split out changing kvm_invalidate_memslot() to call kvm_arch_flush_shadow_memslot() on an old slot from patch 24 to a separate patch 25,
* Make kvm_for_each_memslot_in_gfn_range() return only strictly overlapping memslots and rewrite its implementation to use iterators in patch 26.
Maciej S. Szmigiero (12):
KVM: Resync only arch fields when slots_arch_lock gets reacquired
KVM: x86: Don't call kvm_mmu_change_mmu_pages() if the count hasn't
changed
KVM: x86: Use nr_memslot_pages to avoid traversing the memslots array
KVM: Integrate gfn_to_memslot_approx() into search_memslots()
KVM: Move WARN on invalid memslot index to update_memslots()
KVM: Resolve memslot ID via a hash table instead of via a static array
KVM: Use interval tree to do fast hva lookup in memslots
KVM: s390: Introduce kvm_s390_get_gfn_end()
KVM: Keep memslots in tree-based structures instead of array-based
ones
KVM: Call kvm_arch_flush_shadow_memslot() on the old slot in
kvm_invalidate_memslot()
KVM: Optimize gfn lookup in kvm_zap_gfn_range()
KVM: Optimize overlapping memslots check
Sean Christopherson (17):
KVM: Require total number of memslot pages to fit in an unsigned long
KVM: Open code kvm_delete_memslot() into its only caller
KVM: Use "new" memslot's address space ID instead of dedicated param
KVM: Let/force architectures to deal with arch specific memslot data
KVM: arm64: Use "new" memslot instead of userspace memory region
KVM: MIPS: Drop pr_debug from memslot commit to avoid using "mem"
KVM: PPC: Avoid referencing userspace memory region in memslot updates
KVM: s390: Use "new" memslot instead of userspace memory region
KVM: x86: Use "new" memslot instead of userspace memory region
KVM: RISC-V: Use "new" memslot instead of userspace memory region
KVM: Stop passing kvm_userspace_memory_region to arch memslot hooks
KVM: Use prepare/commit hooks to handle generic memslot metadata
updates
KVM: x86: Don't assume old/new memslots are non-NULL at memslot commit
KVM: s390: Skip gfn/size sanity checks on memslot DELETE or FLAGS_ONLY
KVM: Don't make a full copy of the old memslot in
__kvm_set_memory_region()
KVM: Wait 'til the bitter end to initialize the "new" memslot
KVM: Dynamically allocate "new" memslots from the get-go
arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/mmu.c | 27 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 9 +- arch/powerpc/include/asm/kvm_ppc.h | 14 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s.c | 14 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 28 +- arch/powerpc/kvm/book3s_hv_nested.c | 4 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 14 +- arch/powerpc/kvm/book3s_pr.c | 9 +- arch/powerpc/kvm/booke.c | 7 +- arch/powerpc/kvm/powerpc.c | 9 +- arch/riscv/kvm/mmu.c | 31 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 98 ++-- arch/s390/kvm/kvm-s390.h | 14 + arch/s390/kvm/pv.c | 4 +- arch/x86/include/asm/kvm_host.h | 3 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/debugfs.c | 6 +- arch/x86/kvm/mmu/mmu.c | 38 +- arch/x86/kvm/x86.c | 41 +- include/linux/kvm_host.h | 277 ++++++--- virt/kvm/kvm_main.c | 835 ++++++++++++++++------------ 26 files changed, 854 insertions(+), 637 deletions(-)
PATCH v6 1/29 KVM: Require total number of memslot pages to fit in an unsigned long
From: Sean Christopherson <[email protected]>
Explicitly disallow creating more memslot pages than can fit in an unsigned long, KVM doesn't correctly handle a total number of memslot pages that doesn't fit in an unsigned long and remedying that would be a waste of time.
For a 64-bit kernel, this is a nop as memslots are not allowed to overlap in the gfn address space.
With a 32-bit kernel, userspace can at most address 3gb of virtual memory, whereas wrapping the total number of pages would require 4tb+ of guest physical memory. Even with x86's second address space for SMM, userspace would need to alias all of guest memory more than one _thousand_ times. And on older x86 hardware with MAXPHYADDR < 43, the guest couldn't actually access any of those aliases even if userspace lied about guest.MAXPHYADDR.
On 390 and arm64, this is a nop as they don't support 32-bit hosts.
On x86, practically speaking this is simply acknowledging reality as the existing kvm_mmu_calculate_default_mmu_pages() assumes the total number of pages fits in an "unsigned long".
On PPC, this is likely a nop as every flavor of PPC KVM assumes gfns (and gpas!) fit in unsigned long. arch/powerpc/kvm/book3s_32_mmu_host.c goes a step further and fails the build if CONFIG_PTE_64BIT=y, which presumably means that it does't support 64-bit physical addresses.
On MIPS, this is also likely a nop as the core MMU helpers assume gpas fit in unsigned long, e.g. see kvm_mips_##name##_pte.
And finally, RISC-V is a "don't care" as it doesn't exist in any release, i.e. there is no established ABI to break.
--- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+)
include/linux/kvm_host.h +1 -0
@@ -554,6 +554,7 @@ struct kvm {*/ struct mutex slots_arch_lock;struct mm_struct *mm;/* userspace tied to this vm */ +unsigned long nr_memslot_pages;struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM ];struct xarray vcpu_array;
virt/kvm/kvm_main.c +19 -0
@@ -1640,6 +1640,15 @@ static int kvm_set_memslot(struct kvm *kvm,update_memslots (slots, new, change); slots =install_new_memslots (kvm, as_id, slots); +/* +* Update the total number of memslot pages before calling the arch +* hook so that architectures can consume the result directly. +*/ +if (change ==KVM_MR_DELETE ) + kvm->nr_memslot_pages -= old.npages; +else if (change ==KVM_MR_CREATE ) + kvm->nr_memslot_pages += new->npages; +kvm_arch_commit_memory_region (kvm, mem, &old, new, change);/* Free the old memslot's metadata. Note, this is the full copy!!! */
@@ -1670,6 +1679,9 @@ static int kvm_delete_memslot(struct kvm *kvm,if (!old->npages)return -EINVAL ; +if (WARN_ON_ONCE (kvm->nr_memslot_pages < old->npages)) +return -EIO ; +memset (&new,0 ,sizeof (new)); new.id = old->id;/*
@@ -1753,6 +1765,13 @@ int __kvm_set_memory_region(struct kvm *kvm,if (!old.npages) { change =KVM_MR_CREATE ; new.dirty_bitmap =NULL ; + +/* +* To simplify KVM internals, the total number of pages across +* all memslots must fit in an unsigned long. +*/ +if ((kvm->nr_memslot_pages + new.npages) < kvm->nr_memslot_pages) +return -EINVAL ; }else {/* Modify an existing slot. */ if ((new.userspace_addr != old.userspace_addr) || (new.npages != old.npages) ||
PATCH v6 2/29 KVM: Open code kvm_delete_memslot() into its only caller
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Fold kvm_delete_memslot() into __kvm_set_memory_region() to free up the "kvm_delete_memslot()" name for use in a future helper. The delete logic isn't so complex/long that it truly needs a helper, and it will be simplified a wee bit further in upcoming commits.
No functional change intended.
--- virt/kvm/kvm_main.c | 42 +++++++++++++++++------------------------- 1 file changed, 17 insertions(+), 25 deletions(-)
virt/kvm/kvm_main.c +17 -25
@@ -1670,29 +1670,6 @@ static int kvm_set_memslot(struct kvm *kvm,return r; } -static int kvm_delete_memslot (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem, -struct kvm_memory_slot *old,int as_id) -{ -struct kvm_memory_slot new; - -if (!old->npages) -return -EINVAL ; - -if (WARN_ON_ONCE (kvm->nr_memslot_pages < old->npages)) -return -EIO ; - -memset (&new,0 ,sizeof (new)); - new.id = old->id; -/* -* This is only for debugging purpose; it should never be referenced -* for a removed memslot. -*/ - new.as_id = as_id; - -return kvm_set_memslot (kvm, mem, &new, as_id,KVM_MR_DELETE ); -} -/* * Allocate some memory and give it an address in the guest physical address * space.
@@ -1749,8 +1726,23 @@ int __kvm_set_memory_region(struct kvm *kvm, old.id = id; } -if (!mem->memory_size) -return kvm_delete_memslot (kvm, mem, &old, as_id); +if (!mem->memory_size) { +if (!old.npages) +return -EINVAL ; + +if (WARN_ON_ONCE (kvm->nr_memslot_pages < old.npages)) +return -EIO ; + +memset (&new,0 ,sizeof (new)); + new.id = id; +/* +* This is only for debugging purpose; it should never be +* referenced for a removed memslot. +*/ + new.as_id = as_id; + +return kvm_set_memslot (kvm, mem, &new, as_id,KVM_MR_DELETE ); + } new.as_id = as_id; new.id = id;
PATCH v6 3/29 KVM: Resync only arch fields when slots_arch_lock gets reacquired
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
There is no need to copy the whole memslot data after releasing slots_arch_lock for a moment to install temporary memslots copy in kvm_set_memslot() since this lock only protects the arch field of each memslot.
Just resync this particular field after reacquiring slots_arch_lock.
Note, this also eliminates the need to manually clear the INVALID flag when restoring memslots; the "setting" of the INVALID flag was an unwanted side effect of copying the entire memslots.
Since kvm_copy_memslots() has just one caller remaining now open-code it instead.
--- virt/kvm/kvm_main.c | 45 +++++++++++++++++++++++++-------------------- 1 file changed, 25 insertions(+), 20 deletions(-)
virt/kvm/kvm_main.c +25 -20
@@ -1517,12 +1517,6 @@ static size_t kvm_memslots_size(int slots) (sizeof (struct kvm_memory_slot ) * slots); } -static void kvm_copy_memslots (struct kvm_memslots *to, -struct kvm_memslots *from) -{ -memcpy (to, from,kvm_memslots_size (from->used_slots)); -} -/* * Note, at a minimum, the current number of used slots must be allocated, even * when deleting a memslot, as we need a complete duplicate of the memslots for
@@ -1541,11 +1535,22 @@ static struct kvm_memslots *kvm_dup_memslots(struct kvm_memslots *old, slots =kvzalloc (new_size,GFP_KERNEL_ACCOUNT );if (likely (slots)) -kvm_copy_memslots (slots, old); +memcpy (slots, old,kvm_memslots_size (old->used_slots));return slots; } +static void kvm_copy_memslots_arch (struct kvm_memslots *to, +struct kvm_memslots *from) +{ +int i; + +WARN_ON_ONCE (to->used_slots != from->used_slots); + +for (i =0 ; i < from->used_slots; i++) + to->memslots[i].arch = from->memslots[i].arch; +} + static int kvm_set_memslot(struct kvm *kvm, const struct kvm_userspace_memory_region *mem, struct kvm_memory_slot *new, int as_id,
@@ -1586,9 +1591,10 @@ static int kvm_set_memslot(struct kvm *kvm, slot->flags |=KVM_MEMSLOT_INVALID ;/* -* We can re-use the memory from the old memslots. -* It will be overwritten with a copy of the new memslots -* after reacquiring the slots_arch_lock below. +* We can re-use the old memslots, the only difference from the +* newly installed memslots is the invalid flag, which will get +* dropped by update_memslots anyway. We'll also revert to the +* old memslots if preparing the new memory region fails. */ slots =install_new_memslots (kvm, as_id, slots);
@@ -1605,12 +1611,14 @@ static int kvm_set_memslot(struct kvm *kvm,mutex_lock (&kvm->slots_arch_lock);/* -* The arch-specific fields of the memslots could have changed -* between releasing the slots_arch_lock in -* install_new_memslots and here, so get a fresh copy of the -* slots. +* The arch-specific fields of the now-active memslots could +* have been modified between releasing slots_arch_lock in +* install_new_memslots and re-acquiring slots_arch_lock above. +* Copy them to the inactive memslots. Arch code is required +* to retrieve memslots *after* acquiring slots_arch_lock, thus +* the active memslots are guaranteed to be fresh. */ -kvm_copy_memslots (slots,__kvm_memslots (kvm, as_id)); +kvm_copy_memslots_arch (slots,__kvm_memslots (kvm, as_id)); }/*
@@ -1659,13 +1667,10 @@ static int kvm_set_memslot(struct kvm *kvm,return 0 ; out_slots: -if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) { - slot =id_to_memslot (slots, new->id); - slot->flags &= ~KVM_MEMSLOT_INVALID ; +if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) slots =install_new_memslots (kvm, as_id, slots); - }else { +else mutex_unlock (&kvm->slots_arch_lock); - }kvfree (slots);return r; }
Re: [PATCH v6 03/29] KVM: Resync only arch fields when slots_arch_lock gets reacquired
Quoted 13 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
There is no need to copy the whole memslot data after releasing
slots_arch_lockfor a moment to install temporary memslots copy inkvm_set_memslot()since this lock only protects the arch field of each memslot.Just resync this particular field after reacquiring
slots_arch_lock.Note, this also eliminates the need to manually clear the INVALID flag when restoring memslots; the "setting" of the INVALID flag was an unwanted side effect of copying the entire memslots.
Quoted 6 lines
Since
kvm_copy_memslots()has just one caller remaining now open-code it instead.Signed-off-by Maciej S. Szmigiero <[email protected]>[sean: tweak shortlog, note INVALID flag in changelog, revert comment]Signed-off-by Sean Christopherson <[email protected]>
Heh, I think you can drop my SoB? This is new territory for me, I don't know the rules for this particular situation.
Re: [PATCH v6 03/29] KVM: Resync only arch fields when slots_arch_lock gets reacquired
Quoted 17 lines · Sean Christopherson
Quoted 16 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
There is no need to copy the whole memslot data after releasing
slots_arch_lockfor a moment to install temporary memslots copy inkvm_set_memslot()since this lock only protects the arch field of each memslot.Just resync this particular field after reacquiring
slots_arch_lock.Note, this also eliminates the need to manually clear the INVALID flag when restoring memslots; the "setting" of the INVALID flag was an unwanted side effect of copying the entire memslots.
Since
kvm_copy_memslots()has just one caller remaining now open-code it instead.
Quoted 9 lines
Quoted 3 lines
Signed-off-by Maciej S. Szmigiero <[email protected]>[sean: tweak shortlog, note INVALID flag in changelog, revert comment]Signed-off-by Sean Christopherson <[email protected]>Heh, I think you can drop my SoB? This is new territory for me, I don't know the rules for this particular situation.
Reviewed-by Sean Christopherson <[email protected]>
Will replace your SoB with your R-b on this patch then.
Thanks,
Maciej
Paolo Bonzini
maintainerRe: [PATCH v6 03/29] KVM: Resync only arch fields when slots_arch_lock gets reacquired
In reply to Sean ChristophersonQuoted 5 lines · Sean Christopherson
Quoted 3 lines
Signed-off-by Maciej S. Szmigiero<[email protected]>[sean: tweak shortlog, note INVALID flag in changelog, revert comment]Signed-off-by Sean Christopherson<[email protected]>Heh, I think you can drop my SoB? This is new territory for me, I don't know the rules for this particular situation.
I'll add a co-developed-by for all patches that you two worked on together.
Paolo
Quoted 2 lines
Reviewed-by Sean Christopherson<[email protected]>
PATCH v6 4/29 KVM: Use "new" memslot's address space ID instead of dedicated param
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Now that the address space ID is stored in every slot, including fake slots used for deletion, use the slot's as_id instead of passing in the redundant information as a param to kvm_set_memslot(). This will greatly simplify future memslot work by avoiding passing a large number of variables around purely to honor @as_id.
Drop a comment in the DELETE path about new->as_id being provided purely for debug, as that's now a lie.
No functional change intended.
--- virt/kvm/kvm_main.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)
virt/kvm/kvm_main.c +9 -13
@@ -1553,7 +1553,7 @@ static void kvm_copy_memslots_arch(struct kvm_memslots *to,static int kvm_set_memslot (struct kvm *kvm,const struct kvm_userspace_memory_region *mem, -struct kvm_memory_slot *new,int as_id, +struct kvm_memory_slot *new,enum kvm_mr_change change) {struct kvm_memory_slot *slot, old;
@@ -1576,7 +1576,7 @@ static int kvm_set_memslot(struct kvm *kvm,*/ mutex_lock (&kvm->slots_arch_lock); - slots =kvm_dup_memslots (__kvm_memslots (kvm, as_id), change); + slots =kvm_dup_memslots (__kvm_memslots (kvm, new->as_id), change);if (!slots) {mutex_unlock (&kvm->slots_arch_lock);return -ENOMEM ;
@@ -1596,7 +1596,7 @@ static int kvm_set_memslot(struct kvm *kvm,* dropped by update_memslots anyway. We'll also revert to the * old memslots if preparing the new memory region fails. */ - slots =install_new_memslots (kvm, as_id, slots); + slots =install_new_memslots (kvm, new->as_id, slots);/* From this point no new shadow pages pointing to a deleted, * or moved, memslot will be created.
@@ -1618,7 +1618,7 @@ static int kvm_set_memslot(struct kvm *kvm,* to retrieve memslots *after* acquiring slots_arch_lock, thus * the active memslots are guaranteed to be fresh. */ -kvm_copy_memslots_arch (slots,__kvm_memslots (kvm, as_id)); +kvm_copy_memslots_arch (slots,__kvm_memslots (kvm, new->as_id)); }/*
@@ -1635,7 +1635,7 @@ static int kvm_set_memslot(struct kvm *kvm,WARN_ON_ONCE (change !=KVM_MR_CREATE );memset (&old,0 ,sizeof (old)); old.id = new->id; - old.as_id = as_id; + old.as_id = new->as_id; }/* Copy the arch-specific data, again after (re)acquiring slots_arch_lock. */
@@ -1646,7 +1646,7 @@ static int kvm_set_memslot(struct kvm *kvm, goto out_slots;update_memslots (slots, new, change); - slots =install_new_memslots (kvm, as_id, slots); + slots =install_new_memslots (kvm, new->as_id, slots);/* * Update the total number of memslot pages before calling the arch
@@ -1668,7 +1668,7 @@ static int kvm_set_memslot(struct kvm *kvm, out_slots:if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) - slots =install_new_memslots (kvm, as_id, slots); + slots =install_new_memslots (kvm, new->as_id, slots);else mutex_unlock (&kvm->slots_arch_lock);kvfree (slots);
@@ -1740,13 +1740,9 @@ int __kvm_set_memory_region(struct kvm *kvm,memset (&new,0 ,sizeof (new)); new.id = id; -/* -* This is only for debugging purpose; it should never be -* referenced for a removed memslot. -*/ new.as_id = as_id; -return kvm_set_memslot (kvm, mem, &new, as_id,KVM_MR_DELETE ); +return kvm_set_memslot (kvm, mem, &new,KVM_MR_DELETE ); } new.as_id = as_id;
@@ -1809,7 +1805,7 @@ int __kvm_set_memory_region(struct kvm *kvm,bitmap_set (new.dirty_bitmap,0 , new.npages); } - r =kvm_set_memslot (kvm, mem, &new, as_id, change); + r =kvm_set_memslot (kvm, mem, &new, change);if (r) goto out_bitmap;
PATCH v6 5/29 KVM: Let/force architectures to deal with arch specific memslot data
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Pass the "old" slot to kvm_arch_prepare_memory_region() and force arch code to handle propagating arch specific data from "new" to "old" when necessary. This is a baby step towards dynamically allocating "new" from the get go, and is a (very) minor performance boost on x86 due to not unnecessarily copying arch data.
For PPC HV, copy the rmap in the !CREATE and !DELETE paths, i.e. for MOVE and FLAGS_ONLY. This is functionally a nop as the previous behavior would overwrite the pointer for CREATE, and eventually discard/ignore it for DELETE.
For x86, copy the arch data only for FLAGS_ONLY changes. Unlike PPC HV, x86 needs to reallocate arch data in the MOVE case as the size of x86's allocations depend on the alignment of the memslot's gfn.
Opportunistically tweak kvm_arch_prepare_memory_region()'s param order to match the "commit" prototype.
--- arch/arm64/kvm/mmu.c | 7 ++++--- arch/mips/kvm/mips.c | 3 ++- arch/powerpc/include/asm/kvm_ppc.h | 10 ++++++---- arch/powerpc/kvm/book3s.c | 12 ++++++------ arch/powerpc/kvm/book3s_hv.c | 17 ++++++++++------- arch/powerpc/kvm/book3s_pr.c | 9 +++++---- arch/powerpc/kvm/booke.c | 5 +++-- arch/powerpc/kvm/powerpc.c | 5 +++-- arch/riscv/kvm/mmu.c | 7 ++++--- arch/s390/kvm/kvm-s390.c | 3 ++- arch/x86/kvm/x86.c | 15 +++++++++++---- include/linux/kvm_host.h | 3 ++- virt/kvm/kvm_main.c | 5 +---- 13 files changed, 59 insertions(+), 42 deletions(-)
arch/arm64/kvm/mmu.c +4 -3
@@ -1486,8 +1486,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change) {hva_t hva = mem->userspace_addr;
@@ -1502,7 +1503,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,* Prevent userspace from creating a memory region outside of the IPA * space addressable by the KVM guest IPA space. */ -if ((memslot->base_gfn + memslot->npages) > (kvm_phys_size (kvm) >>PAGE_SHIFT )) +if ((new->base_gfn + new->npages) > (kvm_phys_size (kvm) >>PAGE_SHIFT ))return -EFAULT ;mmap_read_lock (current->mm);
@@ -1536,7 +1537,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,if (vma->vm_flags &VM_PFNMAP ) {/* IO region dirty page logging not allowed */ -if (memslot->flags &KVM_MEM_LOG_DIRTY_PAGES ) { +if (new->flags &KVM_MEM_LOG_DIRTY_PAGES ) { ret = -EINVAL ;break ; }
arch/mips/kvm/mips.c +2 -1
@@ -214,8 +214,9 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change) {return 0 ;
arch/powerpc/include/asm/kvm_ppc.h +6 -4
@@ -200,12 +200,13 @@ extern void kvmppc_core_destroy_vm(struct kvm *kvm);extern void kvmppc_core_free_memslot (struct kvm *kvm,struct kvm_memory_slot *slot);extern int kvmppc_core_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change);extern void kvmppc_core_commit_memory_region (struct kvm *kvm,const struct kvm_userspace_memory_region *mem, -const struct kvm_memory_slot *old, +struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change); extern int kvm_vm_ioctl_get_smmu_info(struct kvm *kvm,
@@ -274,12 +275,13 @@ struct kvmppc_ops {int (*get_dirty_log)(struct kvm *kvm,struct kvm_dirty_log *log);void (*flush_memslot)(struct kvm *kvm,struct kvm_memory_slot *memslot);int (*prepare_memory_region)(struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change);void (*commit_memory_region)(struct kvm *kvm,const struct kvm_userspace_memory_region *mem, -const struct kvm_memory_slot *old, +struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change);bool (*unmap_gfn_range)(struct kvm *kvm,struct kvm_gfn_range *range);
arch/powerpc/kvm/book3s.c +6 -6
@@ -847,17 +847,17 @@ void kvmppc_core_flush_memslot(struct kvm *kvm, struct kvm_memory_slot *memslot) }int kvmppc_core_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot, -const struct kvm_userspace_memory_region *mem, -enum kvm_mr_change change) +const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new, +enum kvm_mr_change change) { -return kvm->arch.kvm_ops->prepare_memory_region (kvm, memslot, mem, - change); +return kvm->arch.kvm_ops->prepare_memory_region (kvm, mem, old, new, change); }void kvmppc_core_commit_memory_region (struct kvm *kvm,const struct kvm_userspace_memory_region *mem, -const struct kvm_memory_slot *old, +struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change) {
arch/powerpc/kvm/book3s_hv.c +10 -7
@@ -4854,17 +4854,20 @@ static void kvmppc_core_free_memslot_hv(struct kvm_memory_slot *slot) }static int kvmppc_core_prepare_memory_region_hv (struct kvm *kvm, -struct kvm_memory_slot *slot, -const struct kvm_userspace_memory_region *mem, -enum kvm_mr_change change) +const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new, +enum kvm_mr_change change) {unsigned long npages = mem->memory_size >>PAGE_SHIFT ;if (change ==KVM_MR_CREATE ) { - slot->arch.rmap =vzalloc (array_size (npages, -sizeof (*slot->arch.rmap))); -if (!slot->arch.rmap) + new->arch.rmap =vzalloc (array_size (npages, +sizeof (*new->arch.rmap))); +if (!new->arch.rmap)return -ENOMEM ; + }else if (change !=KVM_MR_DELETE ) { + new->arch.rmap = old->arch.rmap; }return 0 ;
@@ -4872,7 +4875,7 @@ static int kvmppc_core_prepare_memory_region_hv(struct kvm *kvm,static void kvmppc_core_commit_memory_region_hv (struct kvm *kvm,const struct kvm_userspace_memory_region *mem, -const struct kvm_memory_slot *old, +struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change) {
arch/powerpc/kvm/book3s_pr.c +5 -4
@@ -1899,16 +1899,17 @@ static void kvmppc_core_flush_memslot_pr(struct kvm *kvm, }static int kvmppc_core_prepare_memory_region_pr (struct kvm *kvm, -struct kvm_memory_slot *memslot, -const struct kvm_userspace_memory_region *mem, -enum kvm_mr_change change) +const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new, +enum kvm_mr_change change) {return 0 ; }static void kvmppc_core_commit_memory_region_pr (struct kvm *kvm,const struct kvm_userspace_memory_region *mem, -const struct kvm_memory_slot *old, +struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change) {
arch/powerpc/kvm/booke.c +3 -2
@@ -1821,8 +1821,9 @@ void kvmppc_core_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) }int kvmppc_core_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change) {return 0 ;
@@ -1830,7 +1831,7 @@ int kvmppc_core_prepare_memory_region(struct kvm *kvm,void kvmppc_core_commit_memory_region (struct kvm *kvm,const struct kvm_userspace_memory_region *mem, -const struct kvm_memory_slot *old, +struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change) {
arch/powerpc/kvm/powerpc.c +3 -2
@@ -698,11 +698,12 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change) { -return kvmppc_core_prepare_memory_region (kvm, memslot, mem, change); +return kvmppc_core_prepare_memory_region (kvm, mem, old, new, change); } void kvm_arch_commit_memory_region(struct kvm *kvm,
arch/riscv/kvm/mmu.c +4 -3
@@ -477,8 +477,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change) {hva_t hva = mem->userspace_addr;
@@ -494,7 +495,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,* Prevent userspace from creating a memory region outside of the GPA * space addressable by the KVM guest GPA space. */ -if ((memslot->base_gfn + memslot->npages) >= +if ((new->base_gfn + new->npages) >= (stage2_gpa_size >>PAGE_SHIFT ))return -EFAULT ;
@@ -541,7 +542,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, pa += vm_start - vma->vm_start;/* IO region dirty page logging not allowed */ -if (memslot->flags &KVM_MEM_LOG_DIRTY_PAGES ) { +if (new->flags &KVM_MEM_LOG_DIRTY_PAGES ) { ret = -EINVAL ; goto out; }
arch/s390/kvm/kvm-s390.c +2 -1
@@ -5007,8 +5007,9 @@ vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf)/* Section: memory related */ int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change) {/* A few sanity checks. We can have memory slots which have to be
arch/x86/kvm/x86.c +11 -4
@@ -11723,13 +11723,20 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot, -const struct kvm_userspace_memory_region *mem, -enum kvm_mr_change change) +const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new, +enum kvm_mr_change change) {if (change ==KVM_MR_CREATE || change ==KVM_MR_MOVE ) -return kvm_alloc_memslot_metadata (kvm, memslot, +return kvm_alloc_memslot_metadata (kvm, new, mem->memory_size >>PAGE_SHIFT ); + +if (change ==KVM_MR_FLAGS_ONLY ) +memcpy (&new->arch, &old->arch,sizeof (old->arch)); +else if (WARN_ON_ONCE (change !=KVM_MR_DELETE )) +return -EIO ; +return 0 ; }
include/linux/kvm_host.h +2 -1
@@ -835,8 +835,9 @@ int __kvm_set_memory_region(struct kvm *kvm,void kvm_arch_free_memslot (struct kvm *kvm,struct kvm_memory_slot *slot);void kvm_arch_memslots_updated (struct kvm *kvm,u64 gen);int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot,const struct kvm_userspace_memory_region *mem, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,enum kvm_mr_change change); void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_userspace_memory_region *mem,
virt/kvm/kvm_main.c +1 -4
@@ -1638,10 +1638,7 @@ static int kvm_set_memslot(struct kvm *kvm, old.as_id = new->as_id; } -/* Copy the arch-specific data, again after (re)acquiring slots_arch_lock. */ -memcpy (&new->arch, &old.arch,sizeof (old.arch)); - - r =kvm_arch_prepare_memory_region (kvm, new, mem, change); + r =kvm_arch_prepare_memory_region (kvm, mem, &old, new, change);if (r) goto out_slots;
PATCH v6 6/29 KVM: arm64: Use "new" memslot instead of userspace memory region
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Get the slot ID, hva, etc... from the "new" memslot instead of the userspace memory region when preparing/committing a memory region. This will allow a future commit to drop @mem from the prepare/commit hooks once all architectures convert to using "new".
Opportunistically wait to get the hva begin+end until after filtering out the DELETE case in anticipation of a future commit passing NULL for @new when deleting a memslot.
--- arch/arm64/kvm/mmu.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
arch/arm64/kvm/mmu.c +6 -4
@@ -1473,14 +1473,14 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,* allocated dirty_bitmap[], dirty pages will be tracked while the * memory slot is write protected. */ -if (change !=KVM_MR_DELETE && mem->flags &KVM_MEM_LOG_DIRTY_PAGES ) { +if (change !=KVM_MR_DELETE && new->flags &KVM_MEM_LOG_DIRTY_PAGES ) {/* * If we're with initial-all-set, we don't need to write * protect any pages because they're all reported as dirty. * Huge pages and normal pages will be write protect gradually. */ if (!kvm_dirty_log_manual_protect_and_init_set (kvm)) { -kvm_mmu_wp_memory_region (kvm, mem->slot); +kvm_mmu_wp_memory_region (kvm, new->id); } } }
@@ -1491,8 +1491,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,struct kvm_memory_slot *new, enum kvm_mr_change change) { - hva_t hva = mem->userspace_addr; -hva_t reg_end = hva + mem->memory_size; + hva_t hva, reg_end;int ret =0 ; if (change != KVM_MR_CREATE && change != KVM_MR_MOVE &&
@@ -1506,6 +1505,9 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,if ((new->base_gfn + new->npages) > (kvm_phys_size (kvm) >>PAGE_SHIFT ))return -EFAULT ; + hva = new->userspace_addr; + reg_end = hva + (new->npages <<PAGE_SHIFT ); +mmap_read_lock (current->mm);/* * A memory region could potentially cover multiple VMAs, and any holes
PATCH v6 7/29 KVM: MIPS: Drop pr_debug from memslot commit to avoid using "mem"
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Remove an old (circa 2012) kvm_debug from kvm_arch_commit_memory_region() to print basic information when committing a memslot change. The primary motivation for removing the kvm_debug is to avoid using @mem, the user memory region, so that said param can be removed.
Alternatively, the debug message could be converted to use @new, but that would require synthesizing select state to play nice with the DELETED case, which will pass NULL for @new in the future. And there's no argument to be had for dumping generic information in an arch callback, i.e. if there's a good reason for the debug message, then it belongs in common KVM code where all architectures can benefit.
--- arch/mips/kvm/mips.c | 4 ---- 1 file changed, 4 deletions(-)
arch/mips/kvm/mips.c +0 -4
@@ -230,10 +230,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, {int needs_flush; -kvm_debug ("%s: kvm: %p slot: %d, GPA: %llx, size: %llx, QVA: %llx\n" , - __func__, kvm, mem->slot, mem->guest_phys_addr, - mem->memory_size, mem->userspace_addr); -/* * If dirty page logging is enabled, write protect all pages in the slot * ready for dirty logging.
PATCH v6 8/29 KVM: PPC: Avoid referencing userspace memory region in memslot updates
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
For PPC HV, get the number of pages directly from the new memslot instead of computing the same from the userspace memory region, and explicitly check for !DELETE instead of inferring the same when toggling mmio_update. The motivation for these changes is to avoid referencing the @mem param so that it can be dropped in a future commit.
No functional change intended.
--- arch/powerpc/include/asm/kvm_ppc.h | 4 ---- arch/powerpc/kvm/book3s.c | 6 ++---- arch/powerpc/kvm/book3s_hv.c | 12 +++--------- arch/powerpc/kvm/book3s_pr.c | 2 -- arch/powerpc/kvm/booke.c | 2 -- arch/powerpc/kvm/powerpc.c | 4 ++-- 6 files changed, 7 insertions(+), 23 deletions(-)
arch/powerpc/include/asm/kvm_ppc.h +0 -4
@@ -200,12 +200,10 @@ extern void kvmppc_core_destroy_vm(struct kvm *kvm);extern void kvmppc_core_free_memslot (struct kvm *kvm,struct kvm_memory_slot *slot);extern int kvmppc_core_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change);extern void kvmppc_core_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change);
@@ -275,12 +273,10 @@ struct kvmppc_ops {int (*get_dirty_log)(struct kvm *kvm,struct kvm_dirty_log *log);void (*flush_memslot)(struct kvm *kvm,struct kvm_memory_slot *memslot);int (*prepare_memory_region)(struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change);void (*commit_memory_region)(struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change);
arch/powerpc/kvm/book3s.c +2 -4
@@ -847,21 +847,19 @@ void kvmppc_core_flush_memslot(struct kvm *kvm, struct kvm_memory_slot *memslot) }int kvmppc_core_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change) { -return kvm->arch.kvm_ops->prepare_memory_region (kvm, mem, old, new, change); +return kvm->arch.kvm_ops->prepare_memory_region (kvm, old, new, change); }void kvmppc_core_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change) { - kvm->arch.kvm_ops->commit_memory_region (kvm, mem, old, new, change); + kvm->arch.kvm_ops->commit_memory_region (kvm, old, new, change); }bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range)
arch/powerpc/kvm/book3s_hv.c +3 -9
@@ -4854,15 +4854,12 @@ static void kvmppc_core_free_memslot_hv(struct kvm_memory_slot *slot) }static int kvmppc_core_prepare_memory_region_hv (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change) { -unsigned long npages = mem->memory_size >>PAGE_SHIFT ; -if (change ==KVM_MR_CREATE ) { - new->arch.rmap =vzalloc (array_size (npages, + new->arch.rmap =vzalloc (array_size (new->npages,sizeof (*new->arch.rmap)));if (!new->arch.rmap)return -ENOMEM ;
@@ -4874,20 +4871,17 @@ static int kvmppc_core_prepare_memory_region_hv(struct kvm *kvm, }static void kvmppc_core_commit_memory_region_hv (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change) { -unsigned long npages = mem->memory_size >>PAGE_SHIFT ; -/* -* If we are making a new memslot, it might make +* If we are creating or modifying a memslot, it might make * some address that was previously cached as emulated * MMIO be no longer emulated MMIO, so invalidate * all the caches of emulated MMIO translations. */ -if (npages) +if (change !=KVM_MR_DELETE )atomic64_inc (&kvm->arch.mmio_update);/*
arch/powerpc/kvm/book3s_pr.c +0 -2
@@ -1899,7 +1899,6 @@ static void kvmppc_core_flush_memslot_pr(struct kvm *kvm, }static int kvmppc_core_prepare_memory_region_pr (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -1908,7 +1907,6 @@ static int kvmppc_core_prepare_memory_region_pr(struct kvm *kvm, }static void kvmppc_core_commit_memory_region_pr (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change)
arch/powerpc/kvm/booke.c +0 -2
@@ -1821,7 +1821,6 @@ void kvmppc_core_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) }int kvmppc_core_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -1830,7 +1829,6 @@ int kvmppc_core_prepare_memory_region(struct kvm *kvm, }void kvmppc_core_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change)
arch/powerpc/kvm/powerpc.c +2 -2
@@ -703,7 +703,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,struct kvm_memory_slot *new, enum kvm_mr_change change) { - return kvmppc_core_prepare_memory_region(kvm, mem, old, new, change); + return kvmppc_core_prepare_memory_region(kvm, old, new, change); } void kvm_arch_commit_memory_region(struct kvm *kvm,
@@ -712,7 +712,7 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,const struct kvm_memory_slot *new, enum kvm_mr_change change) { -kvmppc_core_commit_memory_region (kvm ,mem ,old ,new ,change ); +kvmppc_core_commit_memory_region (kvm ,old ,new ,change ); } void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
PATCH v6 9/29 KVM: s390: Use "new" memslot instead of userspace memory region
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Get the gfn, size, and hva from the new memslot instead of the userspace memory region when preparing/committing memory region changes. This will allow a future commit to drop the @mem param.
Note, this has a subtle functional change as KVM would previously reject DELETE if userspace provided a garbage userspace_addr or guest_phys_addr, whereas KVM zeros those fields in the "new" memslot when deleting an existing memslot. Arguably the old behavior is more correct, but there's zero benefit into requiring userspace to provide sane values for hva and gfn.
--- arch/s390/kvm/kvm-s390.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
arch/s390/kvm/kvm-s390.c +8 -5
@@ -5012,18 +5012,20 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,struct kvm_memory_slot *new, enum kvm_mr_change change) { + gpa_t size = new->npages * PAGE_SIZE; +/* A few sanity checks. We can have memory slots which have to be located/ended at a segment boundary (1MB). The memory in userland is ok to be fragmented into various different vmas. It is okay to mmap() and munmap() stuff in this slot after doing this call at any time */ -if (mem->userspace_addr &0xffffful ) +if (new->userspace_addr &0xffffful )return -EINVAL ; -if (mem->memory_size &0xffffful ) +if (size &0xffffful )return -EINVAL ; -if (mem->guest_phys_addr + mem->memory_size > kvm->arch.mem_limit) +if ((new->base_gfn *PAGE_SIZE ) + size > kvm->arch.mem_limit)return -EINVAL ;/* When we are protected, we should not change the memory slots */
@@ -5052,8 +5054,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,break ; fallthrough;case KVM_MR_CREATE : - rc =gmap_map_segment (kvm->arch.gmap, mem->userspace_addr, - mem->guest_phys_addr, mem->memory_size); + rc =gmap_map_segment (kvm->arch.gmap, new->userspace_addr, + new->base_gfn *PAGE_SIZE , + new->npages *PAGE_SIZE );break ;case KVM_MR_FLAGS_ONLY :break ;
PATCH v6 10/29 KVM: x86: Use "new" memslot instead of userspace memory region
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Get the number of pages directly from the new memslot instead of computing the same from the userspace memory region when allocating memslot metadata. This will allow a future patch to drop @mem.
No functional change intended.
--- arch/x86/kvm/x86.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
arch/x86/kvm/x86.c +3 -4
@@ -11642,9 +11642,9 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages) }static int kvm_alloc_memslot_metadata (struct kvm *kvm, -struct kvm_memory_slot *slot, -unsigned long npages) +struct kvm_memory_slot *slot) { +unsigned long npages = slot->npages;int i, r;/*
@@ -11729,8 +11729,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, enum kvm_mr_change change) {if (change ==KVM_MR_CREATE || change ==KVM_MR_MOVE ) -return kvm_alloc_memslot_metadata (kvm, new, - mem->memory_size >>PAGE_SHIFT ); +return kvm_alloc_memslot_metadata (kvm, new);if (change ==KVM_MR_FLAGS_ONLY )memcpy (&new->arch, &old->arch,sizeof (old->arch));
PATCH v6 11/29 KVM: RISC-V: Use "new" memslot instead of userspace memory region
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Get the slot ID, hva, etc... from the "new" memslot instead of the userspace memory region when preparing/committing a memory region. This will allow a future commit to drop @mem from the prepare/commit hooks once all architectures convert to using "new".
Opportunistically wait to get the various "new" values until after filtering out the DELETE case in anticipation of a future commit passing NULL for @new when deleting a memslot.
--- arch/riscv/kvm/mmu.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
arch/riscv/kvm/mmu.c +13 -9
@@ -472,8 +472,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,* allocated dirty_bitmap[], dirty pages will be tracked while * the memory slot is write protected. */ -if (change !=KVM_MR_DELETE && mem->flags &KVM_MEM_LOG_DIRTY_PAGES ) -stage2_wp_memory_region (kvm, mem->slot); +if (change !=KVM_MR_DELETE && new->flags &KVM_MEM_LOG_DIRTY_PAGES ) +stage2_wp_memory_region (kvm, new->id); } int kvm_arch_prepare_memory_region(struct kvm *kvm,
@@ -482,9 +482,9 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,struct kvm_memory_slot *new, enum kvm_mr_change change) { - hva_t hva = mem->userspace_addr; -hva_t reg_end = hva + mem->memory_size; -bool writable = !(mem->flags &KVM_MEM_READONLY ); + hva_t hva, reg_end, size; +gpa_t base_gpa; +bool writable;int ret =0 ; if (change != KVM_MR_CREATE && change != KVM_MR_MOVE &&
@@ -499,6 +499,12 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, (stage2_gpa_size >> PAGE_SHIFT))return -EFAULT ; + hva = new->userspace_addr; + size = new->npages <<PAGE_SHIFT ; + reg_end = hva + size; + base_gpa = new->base_gfn <<PAGE_SHIFT ; + writable = !(new->flags &KVM_MEM_READONLY ); +mmap_read_lock (current->mm);/*
@@ -534,8 +540,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, vm_end =min (reg_end, vma->vm_end);if (vma->vm_flags &VM_PFNMAP ) { -gpa_t gpa = mem->guest_phys_addr + - (vm_start - mem->userspace_addr); +gpa_t gpa = base_gpa + (vm_start - hva);phys_addr_t pa; pa = (phys_addr_t )vma->vm_pgoff <<PAGE_SHIFT ;
@@ -560,8 +565,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,spin_lock (&kvm->mmu_lock);if (ret) -stage2_unmap_range (kvm, mem->guest_phys_addr, - mem->memory_size, false); +stage2_unmap_range (kvm, base_gpa, size, false);spin_unlock (&kvm->mmu_lock); out:
PATCH v6 12/29 KVM: Stop passing kvm_userspace_memory_region to arch memslot hooks
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Drop the @mem param from kvm_arch_{prepare,commit}_memory_region() now that its use has been removed in all architectures.
No functional change intended.
--- arch/arm64/kvm/mmu.c | 2 -- arch/mips/kvm/mips.c | 2 -- arch/powerpc/kvm/powerpc.c | 2 -- arch/riscv/kvm/mmu.c | 2 -- arch/s390/kvm/kvm-s390.c | 2 -- arch/x86/kvm/x86.c | 2 -- include/linux/kvm_host.h | 2 -- virt/kvm/kvm_main.c | 9 ++++----- 8 files changed, 4 insertions(+), 19 deletions(-)
arch/arm64/kvm/mmu.c +0 -2
@@ -1463,7 +1463,6 @@ int kvm_mmu_init(u32 *hyp_va_bits) }void kvm_arch_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -1486,7 +1485,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, }int kvm_arch_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change)
arch/mips/kvm/mips.c +0 -2
@@ -214,7 +214,6 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, }int kvm_arch_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -223,7 +222,6 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, }void kvm_arch_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change)
arch/powerpc/kvm/powerpc.c +0 -2
@@ -698,7 +698,6 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) }int kvm_arch_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -707,7 +706,6 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, }void kvm_arch_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change)
arch/riscv/kvm/mmu.c +0 -2
@@ -462,7 +462,6 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, }void kvm_arch_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -477,7 +476,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, }int kvm_arch_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change)
arch/s390/kvm/kvm-s390.c +0 -2
@@ -5007,7 +5007,6 @@ vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf)/* Section: memory related */ int kvm_arch_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -5035,7 +5034,6 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, }void kvm_arch_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change)
arch/x86/kvm/x86.c +0 -2
@@ -11723,7 +11723,6 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) }int kvm_arch_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -11827,7 +11826,6 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, }void kvm_arch_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change)
include/linux/kvm_host.h +0 -2
@@ -835,12 +835,10 @@ int __kvm_set_memory_region(struct kvm *kvm,void kvm_arch_free_memslot (struct kvm *kvm,struct kvm_memory_slot *slot);void kvm_arch_memslots_updated (struct kvm *kvm,u64 gen);int kvm_arch_prepare_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,const struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change);void kvm_arch_commit_memory_region (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *old,const struct kvm_memory_slot *new,enum kvm_mr_change change);
virt/kvm/kvm_main.c +4 -5
@@ -1552,7 +1552,6 @@ static void kvm_copy_memslots_arch(struct kvm_memslots *to, }static int kvm_set_memslot (struct kvm *kvm, -const struct kvm_userspace_memory_region *mem,struct kvm_memory_slot *new,enum kvm_mr_change change) {
@@ -1638,7 +1637,7 @@ static int kvm_set_memslot(struct kvm *kvm, old.as_id = new->as_id; } - r =kvm_arch_prepare_memory_region (kvm, mem, &old, new, change); + r =kvm_arch_prepare_memory_region (kvm, &old, new, change);if (r) goto out_slots;
@@ -1654,7 +1653,7 @@ static int kvm_set_memslot(struct kvm *kvm,else if (change ==KVM_MR_CREATE ) kvm->nr_memslot_pages += new->npages; -kvm_arch_commit_memory_region (kvm, mem, &old, new, change); +kvm_arch_commit_memory_region (kvm, &old, new, change);/* Free the old memslot's metadata. Note, this is the full copy!!! */ if (change ==KVM_MR_DELETE )
@@ -1739,7 +1738,7 @@ int __kvm_set_memory_region(struct kvm *kvm, new.id = id; new.as_id = as_id; -return kvm_set_memslot (kvm, mem, &new,KVM_MR_DELETE ); +return kvm_set_memslot (kvm, &new,KVM_MR_DELETE ); } new.as_id = as_id;
@@ -1802,7 +1801,7 @@ int __kvm_set_memory_region(struct kvm *kvm,bitmap_set (new.dirty_bitmap,0 , new.npages); } - r =kvm_set_memslot (kvm, mem, &new, change); + r =kvm_set_memslot (kvm, &new, change);if (r) goto out_bitmap;
PATCH v6 13/29 KVM: Use prepare/commit hooks to handle generic memslot metadata updates
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Handle the generic memslot metadata, a.k.a. dirty bitmap, updates at the same time that arch handles it's own metadata updates, i.e. at memslot prepare and commit. This will simplify converting @new to a dynamically allocated object, and more closely aligns common KVM with architecture code.
No functional change intended.
--- virt/kvm/kvm_main.c | 109 +++++++++++++++++++++++++++----------------- 1 file changed, 66 insertions(+), 43 deletions(-)
virt/kvm/kvm_main.c +66 -43
@@ -1551,6 +1551,69 @@ static void kvm_copy_memslots_arch(struct kvm_memslots *to, to->memslots[i].arch = from->memslots[i].arch; } +static int kvm_prepare_memory_region (struct kvm *kvm, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new, +enum kvm_mr_change change) +{ +int r; + +/* +* If dirty logging is disabled, nullify the bitmap; the old bitmap +* will be freed on "commit". If logging is enabled in both old and +* new, reuse the existing bitmap. If logging is enabled only in the +* new and KVM isn't using a ring buffer, allocate and initialize a +* new bitmap. +*/ +if (!(new->flags &KVM_MEM_LOG_DIRTY_PAGES )) + new->dirty_bitmap =NULL ; +else if (old->dirty_bitmap) + new->dirty_bitmap = old->dirty_bitmap; +else if (!kvm->dirty_ring_size) { + r =kvm_alloc_dirty_bitmap (new); +if (r) +return r; + +if (kvm_dirty_log_manual_protect_and_init_set (kvm)) +bitmap_set (new->dirty_bitmap,0 , new->npages); + } + + r =kvm_arch_prepare_memory_region (kvm, old, new, change); + +/* Free the bitmap on failure if it was allocated above. */ +if (r && new->dirty_bitmap && !old->dirty_bitmap) +kvm_destroy_dirty_bitmap (new); + +return r; +} + +static void kvm_commit_memory_region (struct kvm *kvm, +struct kvm_memory_slot *old, +const struct kvm_memory_slot *new, +enum kvm_mr_change change) +{ +/* +* Update the total number of memslot pages before calling the arch +* hook so that architectures can consume the result directly. +*/ +if (change ==KVM_MR_DELETE ) + kvm->nr_memslot_pages -= old->npages; +else if (change ==KVM_MR_CREATE ) + kvm->nr_memslot_pages += new->npages; + +kvm_arch_commit_memory_region (kvm, old, new, change); + +/* +* Free the old memslot's metadata. On DELETE, free the whole thing, +* otherwise free the dirty bitmap as needed (the below effectively +* checks both the flags and whether a ring buffer is being used). +*/ +if (change ==KVM_MR_DELETE ) +kvm_free_memslot (kvm, old); +else if (old->dirty_bitmap && !new->dirty_bitmap) +kvm_destroy_dirty_bitmap (old); +} +static int kvm_set_memslot (struct kvm *kvm,struct kvm_memory_slot *new,enum kvm_mr_change change)
@@ -1637,27 +1700,14 @@ static int kvm_set_memslot(struct kvm *kvm, old.as_id = new->as_id; } - r =kvm_arch_prepare_memory_region (kvm, &old, new, change); + r =kvm_prepare_memory_region (kvm, &old, new, change);if (r) goto out_slots;update_memslots (slots, new, change); slots =install_new_memslots (kvm, new->as_id, slots); -/* -* Update the total number of memslot pages before calling the arch -* hook so that architectures can consume the result directly. -*/ -if (change ==KVM_MR_DELETE ) - kvm->nr_memslot_pages -= old.npages; -else if (change ==KVM_MR_CREATE ) - kvm->nr_memslot_pages += new->npages; - -kvm_arch_commit_memory_region (kvm, &old, new, change); - -/* Free the old memslot's metadata. Note, this is the full copy!!! */ -if (change ==KVM_MR_DELETE ) -kvm_free_memslot (kvm, &old); +kvm_commit_memory_region (kvm, &old, new, change);kvfree (slots);return 0 ;
@@ -1753,7 +1803,6 @@ int __kvm_set_memory_region(struct kvm *kvm,if (!old.npages) { change =KVM_MR_CREATE ; - new.dirty_bitmap =NULL ;/* * To simplify KVM internals, the total number of pages across
@@ -1773,9 +1822,6 @@ int __kvm_set_memory_region(struct kvm *kvm, change =KVM_MR_FLAGS_ONLY ; else /* Nothing to change. */ return 0; - -/* Copy dirty_bitmap from the current memslot. */ - new.dirty_bitmap = old.dirty_bitmap; }if ((change ==KVM_MR_CREATE ) || (change ==KVM_MR_MOVE )) {
@@ -1789,30 +1835,7 @@ int __kvm_set_memory_region(struct kvm *kvm, } } -/* Allocate/free page dirty bitmap as needed */ -if (!(new.flags &KVM_MEM_LOG_DIRTY_PAGES )) - new.dirty_bitmap =NULL ; -else if (!new.dirty_bitmap && !kvm->dirty_ring_size) { - r =kvm_alloc_dirty_bitmap (&new); -if (r) -return r; - -if (kvm_dirty_log_manual_protect_and_init_set (kvm)) -bitmap_set (new.dirty_bitmap,0 , new.npages); - } - - r =kvm_set_memslot (kvm, &new, change); -if (r) - goto out_bitmap; - -if (old.dirty_bitmap && !new.dirty_bitmap) -kvm_destroy_dirty_bitmap (&old); -return 0 ; - -out_bitmap: -if (new.dirty_bitmap && !old.dirty_bitmap) -kvm_destroy_dirty_bitmap (&new); -return r; +return kvm_set_memslot (kvm, &new, change); }EXPORT_SYMBOL_GPL (__kvm_set_memory_region);
PATCH v6 14/29 KVM: x86: Don't assume old/new memslots are non-NULL at memslot commit
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Play nice with a NULL @old or @new when handling memslot updates so that common KVM can pass NULL for one or the other in CREATE and DELETE cases instead of having to synthesize a dummy memslot.
No functional change intended.
--- arch/x86/kvm/x86.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
arch/x86/kvm/x86.c +6 -4
@@ -11758,13 +11758,15 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,const struct kvm_memory_slot *new, enum kvm_mr_change change) { - bool log_dirty_pages = new->flags & KVM_MEM_LOG_DIRTY_PAGES; + u32 old_flags = old ? old->flags : 0; +u32 new_flags = new ? new->flags :0 ; +bool log_dirty_pages = new_flags &KVM_MEM_LOG_DIRTY_PAGES ;/* * Update CPU dirty logging if dirty logging is being toggled. This * applies to all operations. */ -if ((old->flags ^ new->flags) &KVM_MEM_LOG_DIRTY_PAGES ) +if ((old_flags ^ new_flags) &KVM_MEM_LOG_DIRTY_PAGES )kvm_mmu_update_cpu_dirty_logging (kvm, log_dirty_pages);/*
@@ -11782,7 +11784,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,* MOVE/DELETE: The old mappings will already have been cleaned up by * kvm_arch_flush_shadow_memslot(). */ -if ((change !=KVM_MR_FLAGS_ONLY ) || (new->flags &KVM_MEM_READONLY )) +if ((change !=KVM_MR_FLAGS_ONLY ) || (new_flags &KVM_MEM_READONLY ))return ;/*
@@ -11790,7 +11792,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,* other flag is LOG_DIRTY_PAGES, i.e. something is wrong if dirty * logging isn't being toggled on or off. */ -if (WARN_ON_ONCE (!((old->flags ^ new->flags) &KVM_MEM_LOG_DIRTY_PAGES ))) +if (WARN_ON_ONCE (!((old_flags ^ new_flags) &KVM_MEM_LOG_DIRTY_PAGES )))return ;if (!log_dirty_pages) {
PATCH v6 15/29 KVM: s390: Skip gfn/size sanity checks on memslot DELETE or FLAGS_ONLY
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Sanity check the hva, gfn, and size of a userspace memory region only if any of those properties can change, i.e. skip the checks for DELETE and FLAGS_ONLY. KVM doesn't allow moving the hva or changing the size, a gfn change shows up as a MOVE even if flags are being modified, and the checks are pointless for the DELETE case as userspace_addr and gfn_base are zeroed by common KVM.
No functional change intended.
--- arch/s390/kvm/kvm-s390.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
arch/s390/kvm/kvm-s390.c +9 -4
@@ -5011,7 +5011,14 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,struct kvm_memory_slot *new, enum kvm_mr_change change) { - gpa_t size = new->npages * PAGE_SIZE; + gpa_t size; + +/* When we are protected, we should not change the memory slots */ +if (kvm_s390_pv_get_handle (kvm)) +return -EINVAL ; + +if (change ==KVM_MR_DELETE || change ==KVM_MR_FLAGS_ONLY ) +return 0 ;/* A few sanity checks. We can have memory slots which have to be located/ended at a segment boundary (1MB). The memory in userland is
@@ -5021,15 +5028,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,if (new->userspace_addr &0xffffful )return -EINVAL ; + size = new->npages *PAGE_SIZE ;if (size &0xffffful )return -EINVAL ;if ((new->base_gfn *PAGE_SIZE ) + size > kvm->arch.mem_limit)return -EINVAL ; -/* When we are protected, we should not change the memory slots */ -if (kvm_s390_pv_get_handle (kvm)) -return -EINVAL ;return 0 ; }
PATCH v6 16/29 KVM: Don't make a full copy of the old memslot in __kvm_set_memory_region()
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Stop making a full copy of the old memslot in __kvm_set_memory_region() now that metadata updates are handled by kvm_set_memslot(), i.e. now that the old memslot's dirty bitmap doesn't need to be referenced after the memslot and its pointer is modified/invalidated by kvm_set_memslot().
No functional change intended.
--- virt/kvm/kvm_main.c | 35 +++++++++++++---------------------- 1 file changed, 13 insertions(+), 22 deletions(-)
virt/kvm/kvm_main.c +13 -22
@@ -1732,8 +1732,8 @@ static int kvm_set_memslot(struct kvm *kvm,int __kvm_set_memory_region (struct kvm *kvm,const struct kvm_userspace_memory_region *mem) { -struct kvm_memory_slot old, new; -struct kvm_memory_slot *tmp; +struct kvm_memory_slot *old, *tmp; +struct kvm_memory_slot new;enum kvm_mr_change change;int as_id, id;int r;
@@ -1763,25 +1763,16 @@ int __kvm_set_memory_region(struct kvm *kvm,return -EINVAL ;/* -* Make a full copy of the old memslot, the pointer will become stale -* when the memslots are re-sorted by update_memslots(), and the old -* memslot needs to be referenced after calling update_memslots(), e.g. -* to free its resources and for arch specific behavior. +* Note, the old memslot (and the pointer itself!) may be invalidated +* and/or destroyed by kvm_set_memslot(). */ - tmp =id_to_memslot (__kvm_memslots (kvm, as_id), id); -if (tmp) { - old = *tmp; - tmp =NULL ; - }else { -memset (&old,0 ,sizeof (old)); - old.id = id; - } + old =id_to_memslot (__kvm_memslots (kvm, as_id), id);if (!mem->memory_size) { -if (!old.npages) +if (!old || !old->npages)return -EINVAL ; -if (WARN_ON_ONCE (kvm->nr_memslot_pages < old.npages)) +if (WARN_ON_ONCE (kvm->nr_memslot_pages < old->npages))return -EIO ;memset (&new,0 ,sizeof (new));
@@ -1801,7 +1792,7 @@ int __kvm_set_memory_region(struct kvm *kvm,if (new.npages >KVM_MEM_MAX_NR_PAGES )return -EINVAL ; -if (!old.npages) { +if (!old || !old->npages) { change =KVM_MR_CREATE ;/*
@@ -1811,14 +1802,14 @@ int __kvm_set_memory_region(struct kvm *kvm,if ((kvm->nr_memslot_pages + new.npages) < kvm->nr_memslot_pages)return -EINVAL ; }else {/* Modify an existing slot. */ -if ((new.userspace_addr != old.userspace_addr) || - (new.npages != old.npages) || - ((new.flags ^ old.flags) &KVM_MEM_READONLY )) +if ((new.userspace_addr != old->userspace_addr) || + (new.npages != old->npages) || + ((new.flags ^ old->flags) &KVM_MEM_READONLY ))return -EINVAL ; -if (new.base_gfn != old.base_gfn) +if (new.base_gfn != old->base_gfn) change =KVM_MR_MOVE ; -else if (new.flags != old.flags) +else if (new.flags != old->flags) change =KVM_MR_FLAGS_ONLY ;else /* Nothing to change. */ return 0 ;
PATCH v6 17/29 KVM: x86: Don't call kvm_mmu_change_mmu_pages() if the count hasn't changed
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
There is no point in calling kvm_mmu_change_mmu_pages() for memslot operations that don't change the total page count, so do it just for KVM_MR_CREATE and KVM_MR_DELETE.
--- arch/x86/kvm/x86.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +2 -1
@@ -11832,7 +11832,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_memory_slot *new, enum kvm_mr_change change) { - if (!kvm->arch.n_requested_mmu_pages) + if (!kvm->arch.n_requested_mmu_pages && + (change == KVM_MR_CREATE || change == KVM_MR_DELETE)) kvm_mmu_change_mmu_pages(kvm, kvm_mmu_calculate_default_mmu_pages(kvm));
PATCH v6 18/29 KVM: x86: Use nr_memslot_pages to avoid traversing the memslots array
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
There is no point in recalculating from scratch the total number of pages in all memslots each time a memslot is created or deleted. Use KVM's cached nr_memslot_pages to compute the default max number of MMU pages.
Note that even with nr_memslot_pages capped at ULONG_MAX we can't safely multiply it by KVM_PERMILLE_MMU_PAGES (20) since this operation can possibly overflow an unsigned long variable.
Write this "* 20 / 1000" operation as "/ 50" instead to avoid such overflow.
--- arch/x86/include/asm/kvm_host.h | 3 +-- arch/x86/kvm/mmu/mmu.c | 24 ------------------------ arch/x86/kvm/x86.c | 10 +++++++--- 3 files changed, 8 insertions(+), 29 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -2
@@ -135,7 +135,7 @@#define KVM_HPAGE_MASK (x) (~(KVM_HPAGE_SIZE(x) - 1))#define KVM_PAGES_PER_HPAGE (x) (KVM_HPAGE_SIZE(x) / PAGE_SIZE) -#define KVM_PERMILLE_MMU_PAGES 20 +#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50#define KVM_MIN_ALLOC_MMU_PAGES 64UL#define KVM_MMU_HASH_SHIFT 12#define KVM_NUM_MMU_PAGES (1 << KVM_MMU_HASH_SHIFT)
@@ -1590,7 +1590,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, const struct kvm_memory_slot *memslot);void kvm_mmu_zap_all (struct kvm *kvm);void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen); -unsigned long kvm_mmu_calculate_default_mmu_pages (struct kvm *kvm);void kvm_mmu_change_mmu_pages (struct kvm *kvm,unsigned long kvm_nr_mmu_pages);int load_pdptrs (struct kvm_vcpu *vcpu,unsigned long cr3);
arch/x86/kvm/mmu/mmu.c +0 -24
@@ -6131,30 +6131,6 @@ int kvm_mmu_module_init(void)return ret; } -/* -* Calculate mmu pages needed for kvm. -*/ -unsigned long kvm_mmu_calculate_default_mmu_pages (struct kvm *kvm) -{ -unsigned long nr_mmu_pages; -unsigned long nr_pages =0 ; -struct kvm_memslots *slots; -struct kvm_memory_slot *memslot; -int i; - -for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { - slots =__kvm_memslots (kvm, i); - -kvm_for_each_memslot (memslot, slots) - nr_pages += memslot->npages; - } - - nr_mmu_pages = nr_pages *KVM_PERMILLE_MMU_PAGES /1000 ; - nr_mmu_pages =max (nr_mmu_pages,KVM_MIN_ALLOC_MMU_PAGES ); - -return nr_mmu_pages; -} -void kvm_mmu_destroy (struct kvm_vcpu *vcpu) {kvm_mmu_unload (vcpu);
arch/x86/kvm/x86.c +7 -3
@@ -11833,9 +11833,13 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, enum kvm_mr_change change) {if (!kvm->arch.n_requested_mmu_pages && - (change ==KVM_MR_CREATE || change ==KVM_MR_DELETE )) -kvm_mmu_change_mmu_pages (kvm, -kvm_mmu_calculate_default_mmu_pages (kvm)); + (change ==KVM_MR_CREATE || change ==KVM_MR_DELETE )) { +unsigned long nr_mmu_pages; + + nr_mmu_pages = kvm->nr_memslot_pages /KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO ; + nr_mmu_pages =max (nr_mmu_pages,KVM_MIN_ALLOC_MMU_PAGES ); +kvm_mmu_change_mmu_pages (kvm, nr_mmu_pages); + }kvm_mmu_slot_apply_flags (kvm, old, new, change);
Re: [PATCH v6 18/29] KVM: x86: Use nr_memslot_pages to avoid traversing the memslots array
Quoted 10 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
There is no point in recalculating from scratch the total number of pages in all memslots each time a memslot is created or deleted. Use KVM's cached
nr_memslot_pagesto compute the default max number of MMU pages.Note that even with
nr_memslot_pagescapped atULONG_MAXwe can't safely multiply it byKVM_PERMILLE_MMU_PAGES(20) since this operation can possibly overflow an unsigned long variable.
Quoted 6 lines
Write this "* 20 / 1000" operation as "/ 50" instead to avoid such overflow.
Signed-off-by Maciej S. Szmigiero <[email protected]>[sean: use common KVM field and rework changelog accordingly]Signed-off-by Sean Christopherson <[email protected]>
My SoB can definitely be dropped for this one, just consider it review feedback that happened to have an SoB attached.
Re: [PATCH v6 18/29] KVM: x86: Use nr_memslot_pages to avoid traversing the memslots array
Quoted 14 lines · Sean Christopherson
Quoted 13 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
There is no point in recalculating from scratch the total number of pages in all memslots each time a memslot is created or deleted. Use KVM's cached
nr_memslot_pagesto compute the default max number of MMU pages.Note that even with
nr_memslot_pagescapped atULONG_MAXwe can't safely multiply it byKVM_PERMILLE_MMU_PAGES(20) since this operation can possibly overflow an unsigned long variable.Write this "* 20 / 1000" operation as "/ 50" instead to avoid such overflow.
Quoted 9 lines
Quoted 3 lines
Signed-off-by Maciej S. Szmigiero <[email protected]>[sean: use common KVM field and rework changelog accordingly]Signed-off-by Sean Christopherson <[email protected]>My SoB can definitely be dropped for this one, just consider it review feedback that happened to have an SoB attached.
Reviewed-by Sean Christopherson <[email protected]>
...and on this one, too.
Thanks,
Maciej
PATCH v6 19/29 KVM: Integrate gfn_to_memslot_approx() into search_memslots()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
s390 arch has gfn_to_memslot_approx() which is almost identical to search_memslots(), differing only in that in case the gfn falls in a hole one of the memslots bordering the hole is returned.
Add this lookup mode as an option to search_memslots() so we don't have two almost identical functions for looking up a memslot by its gfn.
--- arch/s390/kvm/kvm-s390.c | 45 +++++++--------------------------------- include/linux/kvm_host.h | 35 ++++++++++++++++++++++++------- virt/kvm/kvm_main.c | 2 +- 3 files changed, 36 insertions(+), 46 deletions(-)
arch/s390/kvm/kvm-s390.c +8 -37
@@ -1943,41 +1943,6 @@ static long kvm_s390_set_skeys(struct kvm *kvm, struct kvm_s390_skeys *args)/* for consistency */ #define KVM_S390_CMMA_SIZE_MAX ((u32)KVM_S390_SKEYS_MAX) -/* -* Similar to gfn_to_memslot, but returns the index of a memslot also when the -* address falls in a hole. In that case the index of one of the memslots -* bordering the hole is returned. -*/ -static int gfn_to_memslot_approx (struct kvm_memslots *slots,gfn_t gfn) -{ -int start =0 , end = slots->used_slots; -int slot =atomic_read (&slots->last_used_slot); -struct kvm_memory_slot *memslots = slots->memslots; - -if (gfn >= memslots[slot].base_gfn && - gfn < memslots[slot].base_gfn + memslots[slot].npages) -return slot; - -while (start < end) { - slot = start + (end - start) /2 ; - -if (gfn >= memslots[slot].base_gfn) - end = slot; -else - start = slot +1 ; - } - -if (start >= slots->used_slots) -return slots->used_slots -1 ; - -if (gfn >= memslots[start].base_gfn && - gfn < memslots[start].base_gfn + memslots[start].npages) { -atomic_set (&slots->last_used_slot, start); - } - -return start; -} -static int kvm_s390_peek_cmma (struct kvm *kvm,struct kvm_s390_cmma_log *args,u8 *res,unsigned long bufsize) {
@@ -2001,11 +1966,17 @@ static int kvm_s390_peek_cmma(struct kvm *kvm, struct kvm_s390_cmma_log *args,return 0 ; } +static struct kvm_memory_slot *gfn_to_memslot_approx (struct kvm_memslots *slots, +gfn_t gfn) +{ +return ____gfn_to_memslot (slots, gfn, true); +} +static unsigned long kvm_s390_next_dirty_cmma (struct kvm_memslots *slots,unsigned long cur_gfn) { -int slotidx =gfn_to_memslot_approx (slots, cur_gfn); -struct kvm_memory_slot *ms = slots->memslots + slotidx; +struct kvm_memory_slot *ms =gfn_to_memslot_approx (slots, cur_gfn); +int slotidx = ms - slots->memslots;unsigned long ofs = cur_gfn - ms->base_gfn;if (ms->base_gfn + ms->npages <= cur_gfn) {
include/linux/kvm_host.h +27 -8
@@ -1250,10 +1250,14 @@ try_get_memslot(struct kvm_memslots *slots, int slot_index, gfn_t gfn)* Returns a pointer to the memslot that contains gfn and records the index of * the slot in index. Otherwise returns NULL. * +* With "approx" set returns the memslot also when the address falls +* in a hole. In that case one of the memslots bordering the hole is +* returned. +* * IMPORTANT: Slots are sorted from highest GFN to lowest GFN! */ static inline struct kvm_memory_slot * -search_memslots (struct kvm_memslots *slots,gfn_t gfn,int *index) +search_memslots (struct kvm_memslots *slots,gfn_t gfn,int *index,bool approx) {int start =0 , end = slots->used_slots;struct kvm_memory_slot *memslots = slots->memslots;
@@ -1271,22 +1275,26 @@ search_memslots(struct kvm_memslots *slots, gfn_t gfn, int *index) start = slot +1 ; } +if (approx && start >= slots->used_slots) { + *index = slots->used_slots -1 ; +return &memslots[slots->used_slots -1 ]; + } + slot =try_get_memslot (slots, start, gfn);if (slot) { *index = start;return slot; } +if (approx) { + *index = start; +return &memslots[start]; + }return NULL ; } -/* -* __gfn_to_memslot() and its descendants are here because it is called from -* non-modular code in arch/powerpc/kvm/book3s_64_vio{,_hv}.c. gfn_to_memslot() -* itself isn't here as an inline because that would bloat other code too much. -*/ static inline struct kvm_memory_slot * -__gfn_to_memslot (struct kvm_memslots *slots,gfn_t gfn) +____gfn_to_memslot (struct kvm_memslots *slots,gfn_t gfn,bool approx) {struct kvm_memory_slot *slot;int slot_index =atomic_read (&slots->last_used_slot);
@@ -1295,7 +1303,7 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn)if (slot)return slot; - slot =search_memslots (slots, gfn, &slot_index); + slot =search_memslots (slots, gfn, &slot_index, approx);if (slot) {atomic_set (&slots->last_used_slot, slot_index);return slot;
@@ -1304,6 +1312,17 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn)return NULL ; } +/* +* __gfn_to_memslot() and its descendants are here to allow arch code to inline +* the lookups in hot paths. gfn_to_memslot() itself isn't here as an inline +* because that would bloat other code too much. +*/ +static inline struct kvm_memory_slot * +__gfn_to_memslot (struct kvm_memslots *slots,gfn_t gfn) +{ +return ____gfn_to_memslot (slots, gfn, false); +} +static inline unsigned long __gfn_to_hva_memslot (const struct kvm_memory_slot *slot,gfn_t gfn) {
virt/kvm/kvm_main.c +1 -1
@@ -2143,7 +2143,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfn* search_memslots() instead of __gfn_to_memslot() to avoid * thrashing the VM-wide last_used_index in kvm_memslots. */ - slot =search_memslots (slots, gfn, &slot_index); + slot =search_memslots (slots, gfn, &slot_index, false);if (slot) { vcpu->last_used_slot = slot_index;return slot;
PATCH v6 20/29 KVM: Move WARN on invalid memslot index to update_memslots()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
Since kvm_memslot_move_forward() can theoretically return a negative memslot index even when kvm_memslot_move_backward() returned a positive one (and so did not WARN) let's just move the warning to the common code.
--- virt/kvm/kvm_main.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
virt/kvm/kvm_main.c +4 -2
@@ -1324,8 +1324,7 @@ static inline int kvm_memslot_move_backward(struct kvm_memslots *slots,struct kvm_memory_slot *mslots = slots->memslots;int i; -if (WARN_ON_ONCE (slots->id_to_index[memslot->id] ==-1 ) || -WARN_ON_ONCE (!slots->used_slots)) +if (slots->id_to_index[memslot->id] ==-1 || !slots->used_slots)return -1 ;/*
@@ -1429,6 +1428,9 @@ static void update_memslots(struct kvm_memslots *slots, i =kvm_memslot_move_backward (slots, memslot); i =kvm_memslot_move_forward (slots, memslot, i); +if (WARN_ON_ONCE (i <0 )) +return ; +/* * Copy the memslot to its new position in memslots and update * its index accordingly.
PATCH v6 21/29 KVM: Resolve memslot ID via a hash table instead of via a static array
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
Memslot ID to the corresponding memslot mappings are currently kept as indices in static id_to_index array.
The size of this array depends on the maximum allowed memslot count (regardless of the number of memslots actually in use).
This has become especially problematic recently, when memslot count cap was removed, so the maximum count is now full 32k memslots - the maximum allowed by the current KVM API.
Keeping these IDs in a hash table (instead of an array) avoids this problem.
Resolving a memslot ID to the actual memslot (instead of its index) will also enable transitioning away from an array-based implementation of the whole memslots structure in a later commit.
--- include/linux/kvm_host.h | 25 +++++++---- virt/kvm/kvm_main.c | 95 +++++++++++++++++++++++++++++++--------- 2 files changed, 91 insertions(+), 29 deletions(-)
include/linux/kvm_host.h +16 -9
@@ -29,6 +29,7 @@#include <linux/refcount.h> #include <linux/nospec.h> #include <linux/notifier.h> +#include <linux/hashtable.h> #include <linux/xarray.h> #include <asm/signal.h>
@@ -428,6 +429,7 @@ static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu)#define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)struct kvm_memory_slot { +struct hlist_node id_node;gfn_t base_gfn;unsigned long npages;unsigned long *dirty_bitmap;
@@ -529,8 +531,15 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)*/ struct kvm_memslots {u64 generation; -/* The mapping table from slot id to the index in memslots[]. */ -short id_to_index[KVM_MEM_SLOTS_NUM ]; +/* +* The mapping table from slot id to the index in memslots[]. +* +* 7-bit bucket count matches the size of the old id to index array for +* 512 slots, while giving good performance with this slot count. +* Higher bucket counts bring only small performance improvements but +* always result in higher memory usage (even for lower memslot counts). +*/ + DECLARE_HASHTABLE(id_hash, 7);atomic_t last_used_slot;int used_slots;struct kvm_memory_slot memslots[];
@@ -798,16 +807,14 @@ static inline struct kvm_memslots *kvm_vcpu_memslots(struct kvm_vcpu *vcpu)static inline struct kvm_memory_slot *id_to_memslot (struct kvm_memslots *slots,int id) { -int index = slots->id_to_index[id];struct kvm_memory_slot *slot; -if (index <0 ) -return NULL ; - - slot = &slots->memslots[index]; +hash_for_each_possible (slots->id_hash, slot, id_node, id) { +if (slot->id == id) +return slot; + } -WARN_ON (slot->id != id); -return slot; +return NULL ; }/*
virt/kvm/kvm_main.c +75 -20
191 lines not on this page · raw message
Sean Christopherson
maintainerRe: [PATCH v6 21/29] KVM: Resolve memslot ID via a hash table instead of via a static array
Quoted 14 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
Memslot ID to the corresponding memslot mappings are currently kept as indices in static
id_to_indexarray.
The size of this array depends on the maximum allowed memslot count (regardless of the number of memslots actually in use).This has become especially problematic recently, when memslot count cap was removed, so the maximum count is now full 32k memslots - the maximum allowed by the current KVM API.
Keeping these IDs in a hash table (instead of an array) avoids this problem.
Quoted 7 lines
Resolving a memslot ID to the actual memslot (instead of its index) will also enable transitioning away from an array-based implementation of the whole memslots structure in a later commit.
Signed-off-by Maciej S. Szmigiero <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
Nit, your SoB should come last since you were the last person to handle the patch.
Re: [PATCH v6 21/29] KVM: Resolve memslot ID via a hash table instead of via a static array
Quoted 19 lines · Sean Christopherson
Quoted 18 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
Memslot ID to the corresponding memslot mappings are currently kept as indices in static
id_to_indexarray.
The size of this array depends on the maximum allowed memslot count (regardless of the number of memslots actually in use).This has become especially problematic recently, when memslot count cap was removed, so the maximum count is now full 32k memslots - the maximum allowed by the current KVM API.
Keeping these IDs in a hash table (instead of an array) avoids this problem.
Resolving a memslot ID to the actual memslot (instead of its index) will also enable transitioning away from an array-based implementation of the whole memslots structure in a later commit.
Quoted 6 lines
Quoted 3 lines
Signed-off-by Maciej S. Szmigiero <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Nit, your SoB should come last since you were the last person to handle the patch.
Thought that my SoB should come first as coming from the author of this patch.
Documentation/process/submitting-patches.rst says that:
Quoted 5 lines
Any further SoBs (Signed-off-by:'s) following the author's SoB are from people handling and transporting the patch, but were not involved in its development. SoB chains should reflect the **real** route a patch took as it was propagated to the maintainers and ultimately to Linus, with the first SoB entry signalling primary authorship of a single author.
So "further SoBs follow[] the author's SoB" and "the first SoB entry signal[s] primary authorship".
But at the same time "SoB chains should reflect the **real** route a patch took" - these rules contradict each other in our case.
Thanks,
Maciej
Sean Christopherson
maintainerRe: [PATCH v6 21/29] KVM: Resolve memslot ID via a hash table instead of via a static array
Quoted 37 lines · Maciej S. Szmigiero
Quoted 19 lines · Sean Christopherson
Quoted 18 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
Memslot ID to the corresponding memslot mappings are currently kept as indices in static
id_to_indexarray.
The size of this array depends on the maximum allowed memslot count (regardless of the number of memslots actually in use).This has become especially problematic recently, when memslot count cap was removed, so the maximum count is now full 32k memslots - the maximum allowed by the current KVM API.
Keeping these IDs in a hash table (instead of an array) avoids this problem.
Resolving a memslot ID to the actual memslot (instead of its index) will also enable transitioning away from an array-based implementation of the whole memslots structure in a later commit.
Quoted 6 lines
Quoted 3 lines
Signed-off-by Maciej S. Szmigiero <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Nit, your SoB should come last since you were the last person to handle the patch.
Thought that my SoB should come first as coming from the author of this patch.
Documentation/process/submitting-patches.rstsays that:Quoted 5 lines
Any further SoBs (Signed-off-by:'s) following the author's SoB are from people handling and transporting the patch, but were not involved in its development. SoB chains should reflect the **real** route a patch took as it was propagated to the maintainers and ultimately to Linus, with the first SoB entry signalling primary authorship of a single author.
Quoted 4 lines
So "further SoBs follow[] the author's SoB" and "the first SoB entry signal[s] primary authorship".
But at the same time "SoB chains should reflect the **real** route a patch took" - these rules contradict each other in our case.
Yeah, this is a unusual case. If we wanted to be super strict, for patches written by you without a Co-developed-by, the SoB chain should be:
but that's a bit ridiculous and probably unnecessary since my changes were little more than code review feedback, which is why I think it's ok to just drop my SoB for patches authored solely by you.
Co-developed-by is a slightly different case. Because patches with multiple authors are likely handed back and forth multiple times, as was the case here, and because each author needs a SoB anyways, the normal rules are tweaked slightly to require that the person submitting the patch is always last to capture that they were the person that did the actual submission.
There's another "When to use Acked-by:, Cc:, and Co-developed-by:" section in submitting-patches.rst that covers this:
Co-developed-by: states that the patch was co-created by multiple developers; it is used to give attribution to co-authors (in addition to the author attributed by the From: tag) when several people work on a single patch. Since
Co-developed-by: denotes authorship, every Co-developed-by: must be immediately followed by a Signed-off-by: of the associated co-author. Standard sign-off procedure applies, i.e. the ordering of Signed-off-by: tags should reflect the chronological history of the patch insofar as possible, regardless of whether the author is attributed via From: or Co-developed-by:. Notably, the last
Signed-off-by: must always be that of the developer submitting the patch.
Note, the From: tag is optional when the From: author is also the person (and email) listed in the From: line of the email header.
Example of a patch submitted by the From: author::
<changelog>
Example of a patch submitted by a Co-developed-by: author::
From: From Author <[email protected]>
<changelog>
PATCH v6 22/29 KVM: Use interval tree to do fast hva lookup in memslots
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
The current memslots implementation only allows quick binary search by gfn, quick lookup by hva is not possible - the implementation has to do a linear scan of the whole memslots array, even though the operation being performed might apply just to a single memslot.
This significantly hurts performance of per-hva operations with higher memslot counts.
Since hva ranges can overlap between memslots an interval tree is needed for tracking them.
--- arch/arm64/kvm/Kconfig | 1 + arch/mips/kvm/Kconfig | 1 + arch/powerpc/kvm/Kconfig | 1 + arch/s390/kvm/Kconfig | 1 + arch/x86/kvm/Kconfig | 1 + include/linux/kvm_host.h | 3 +++ virt/kvm/kvm_main.c | 53 +++++++++++++++++++++++++++++----------- 7 files changed, 47 insertions(+), 14 deletions(-)
arch/arm64/kvm/Kconfig +1 -0
@@ -39,6 +39,7 @@ menuconfig KVM select HAVE_KVM_IRQ_BYPASS select HAVE_KVM_VCPU_RUN_PID_CHANGE select SCHED_INFO + select INTERVAL_TREE help Support hosting virtualized guest machines.
arch/mips/kvm/Kconfig +1 -0
@@ -27,6 +27,7 @@ config KVM select KVM_MMIO select MMU_NOTIFIER select SRCU + select INTERVAL_TREE help Support for hosting Guest kernels.
arch/powerpc/kvm/Kconfig +1 -0
@@ -26,6 +26,7 @@ config KVM select KVM_VFIO select IRQ_BYPASS_MANAGER select HAVE_KVM_IRQ_BYPASS + select INTERVAL_TREE config KVM_BOOK3S_HANDLER bool
arch/s390/kvm/Kconfig +1 -0
@@ -33,6 +33,7 @@ config KVM select HAVE_KVM_NO_POLL select SRCU select KVM_VFIO + select INTERVAL_TREE help Support hosting paravirtualized guest machines using the SIE virtualization capability on the mainframe. This should work
arch/x86/kvm/Kconfig +1 -0
@@ -43,6 +43,7 @@ config KVM select KVM_GENERIC_DIRTYLOG_READ_PROTECT select KVM_VFIO select SRCU + select INTERVAL_TREE select HAVE_KVM_PM_NOTIFIER if PM help Support hosting fully virtualized guest machines using hardware
include/linux/kvm_host.h +3 -0
@@ -30,6 +30,7 @@#include <linux/nospec.h> #include <linux/notifier.h> #include <linux/hashtable.h> +#include <linux/interval_tree.h> #include <linux/xarray.h> #include <asm/signal.h>
@@ -430,6 +431,7 @@ static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu)struct kvm_memory_slot {struct hlist_node id_node; +struct interval_tree_node hva_node;gfn_t base_gfn;unsigned long npages;unsigned long *dirty_bitmap;
@@ -531,6 +533,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)*/ struct kvm_memslots {u64 generation; +struct rb_root_cached hva_tree;/* * The mapping table from slot id to the index in memslots[]. *
virt/kvm/kvm_main.c +39 -14
136 lines not on this page · raw message
PATCH v6 23/29 KVM: s390: Introduce kvm_s390_get_gfn_end()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
And use it where s390 code would just access the memslot with the highest gfn directly.
No functional change intended.
--- arch/s390/kvm/kvm-s390.c | 2 +- arch/s390/kvm/kvm-s390.h | 12 ++++++++++++ arch/s390/kvm/pv.c | 4 +--- 3 files changed, 14 insertions(+), 4 deletions(-)
arch/s390/kvm/kvm-s390.c +1 -1
@@ -2014,7 +2014,7 @@ static int kvm_s390_get_cmma(struct kvm *kvm, struct kvm_s390_cmma_log *args,if (!ms)return 0 ; next_gfn =kvm_s390_next_dirty_cmma (slots, cur_gfn +1 ); - mem_end = slots->memslots[0 ].base_gfn + slots->memslots[0 ].npages; + mem_end =kvm_s390_get_gfn_end (slots);while (args->count < bufsize) { hva =gfn_to_hva (kvm, cur_gfn);
arch/s390/kvm/kvm-s390.h +12 -0
@@ -217,6 +217,18 @@ static inline void kvm_s390_set_user_cpu_state_ctrl(struct kvm *kvm) kvm->arch.user_cpu_state_ctrl =1 ; } +/* get the end gfn of the last (highest gfn) memslot */ +static inline unsigned long kvm_s390_get_gfn_end (struct kvm_memslots *slots) +{ +struct kvm_memory_slot *ms; + +if (WARN_ON (!slots->used_slots)) +return 0 ; + + ms = slots->memslots; +return ms->base_gfn + ms->npages; +} +/* implemented in pv.c */ int kvm_s390_pv_destroy_cpu (struct kvm_vcpu *vcpu,u16 *rc,u16 *rrc);int kvm_s390_pv_create_cpu (struct kvm_vcpu *vcpu,u16 *rc,u16 *rrc);
arch/s390/kvm/pv.c +1 -3
@@ -116,7 +116,6 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)unsigned long base = uv_info.guest_base_stor_len;unsigned long virt = uv_info.guest_virt_var_stor_len;unsigned long npages =0 , vlen =0 ; -struct kvm_memory_slot *memslot; kvm->arch.pv.stor_var =NULL ; kvm->arch.pv.stor_base =__get_free_pages (GFP_KERNEL_ACCOUNT ,get_order (base));
@@ -130,8 +129,7 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)* Slots are sorted by GFN */ mutex_lock (&kvm->slots_lock); - memslot =kvm_memslots (kvm)->memslots; - npages = memslot->base_gfn + memslot->npages; + npages =kvm_s390_get_gfn_end (kvm_memslots (kvm));mutex_unlock (&kvm->slots_lock); kvm->arch.pv.guest_len = npages *PAGE_SIZE ;
PATCH v6 24/29 KVM: Keep memslots in tree-based structures instead of array-based ones
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
The current memslot code uses a (reverse gfn-ordered) memslot array for keeping track of them.
Because the memslot array that is currently in use cannot be modified every memslot management operation (create, delete, move, change flags) has to make a copy of the whole array so it has a scratch copy to work on.
Strictly speaking, however, it is only necessary to make copy of the memslot that is being modified, copying all the memslots currently present is just a limitation of the array-based memslot implementation.
Two memslot sets, however, are still needed so the VM continues to run on the currently active set while the requested operation is being performed on the second, currently inactive one.
In order to have two memslot sets, but only one copy of actual memslots it is necessary to split out the memslot data from the memslot sets.
The memslots themselves should be also kept independent of each other so they can be individually added or deleted.
These two memslot sets should normally point to the same set of memslots. They can, however, be desynchronized when performing a memslot management operation by replacing the memslot to be modified by its copy. After the operation is complete, both memslot sets once again point to the same, common set of memslot data.
This commit implements the aforementioned idea.
For tracking of gfns an ordinary rbtree is used since memslots cannot overlap in the guest address space and so this data structure is sufficient for ensuring that lookups are done quickly.
The "last used slot" mini-caches (both per-slot set one and per-vCPU one), that keep track of the last found-by-gfn memslot, are still present in the new code.
--- arch/arm64/kvm/mmu.c | 8 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 3 +- arch/powerpc/kvm/book3s_hv_nested.c | 4 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 14 +- arch/s390/kvm/kvm-s390.c | 24 +- arch/s390/kvm/kvm-s390.h | 6 +- arch/x86/kvm/debugfs.c | 6 +- arch/x86/kvm/mmu/mmu.c | 8 +- include/linux/kvm_host.h | 143 +++--- virt/kvm/kvm_main.c | 761 ++++++++++++++-------------- 11 files changed, 503 insertions(+), 478 deletions(-)
arch/arm64/kvm/mmu.c +4 -4
@@ -210,13 +210,13 @@ static void stage2_flush_vm(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *memslot; -int idx; +int idx, bkt; idx =srcu_read_lock (&kvm->srcu);spin_lock (&kvm->mmu_lock); slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) +kvm_for_each_memslot (memslot, bkt, slots)stage2_flush_memslot (kvm, memslot);spin_unlock (&kvm->mmu_lock);
@@ -595,14 +595,14 @@ void stage2_unmap_vm(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *memslot; -int idx; +int idx, bkt; idx =srcu_read_lock (&kvm->srcu);mmap_read_lock (current->mm);spin_lock (&kvm->mmu_lock); slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) +kvm_for_each_memslot (memslot, bkt, slots)stage2_unmap_memslot (kvm, memslot);spin_unlock (&kvm->mmu_lock);
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -2
@@ -734,11 +734,11 @@ void kvmppc_rmap_reset(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *memslot; -int srcu_idx; +int srcu_idx, bkt; srcu_idx =srcu_read_lock (&kvm->srcu); slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) {/* Mutual exclusion with kvm_unmap_hva_range etc. */ spin_lock (&kvm->mmu_lock);/*
arch/powerpc/kvm/book3s_hv.c +2 -1
@@ -5880,11 +5880,12 @@ static int kvmhv_svm_off(struct kvm *kvm)for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) {struct kvm_memory_slot *memslot;struct kvm_memslots *slots =__kvm_memslots (kvm, i); +int bkt;if (!slots)continue ; -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) {kvmppc_uvmem_drop_pages (memslot, kvm, true);uv_unregister_mem_slot (kvm->arch.lpid, memslot->id); }
arch/powerpc/kvm/book3s_hv_nested.c +2 -2
@@ -749,7 +749,7 @@ void kvmhv_release_all_nested(struct kvm *kvm)struct kvm_nested_guest *gp;struct kvm_nested_guest *freelist =NULL ;struct kvm_memory_slot *memslot; -int srcu_idx; +int srcu_idx, bkt;spin_lock (&kvm->mmu_lock);for (i =0 ; i <= kvm->arch.max_nested_lpid; i++) {
@@ -770,7 +770,7 @@ void kvmhv_release_all_nested(struct kvm *kvm) } srcu_idx =srcu_read_lock (&kvm->srcu); -kvm_for_each_memslot (memslot,kvm_memslots (kvm)) +kvm_for_each_memslot (memslot, bkt,kvm_memslots (kvm))kvmhv_free_memslot_nest_rmap (memslot);srcu_read_unlock (&kvm->srcu, srcu_idx); }
arch/powerpc/kvm/book3s_hv_uvmem.c +7 -7
@@ -459,7 +459,7 @@ unsigned long kvmppc_h_svm_init_start(struct kvm *kvm)struct kvm_memslots *slots;struct kvm_memory_slot *memslot, *m;int ret =H_SUCCESS ; -int srcu_idx; +int srcu_idx, bkt; kvm->arch.secure_guest =KVMPPC_SECURE_INIT_START ;
@@ -478,7 +478,7 @@ unsigned long kvmppc_h_svm_init_start(struct kvm *kvm)/* register the memslot */ slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) { ret =__kvmppc_uvmem_memslot_create (kvm, memslot);if (ret)break ;
@@ -486,7 +486,7 @@ unsigned long kvmppc_h_svm_init_start(struct kvm *kvm)if (ret) { slots =kvm_memslots (kvm); -kvm_for_each_memslot (m, slots) { +kvm_for_each_memslot (m, bkt, slots) {if (m == memslot)break ;__kvmppc_uvmem_memslot_delete (kvm, memslot);
@@ -647,7 +647,7 @@ void kvmppc_uvmem_drop_pages(const struct kvm_memory_slot *slot,unsigned long kvmppc_h_svm_init_abort (struct kvm *kvm) { -int srcu_idx; +int srcu_idx, bkt;struct kvm_memory_slot *memslot;/*
@@ -662,7 +662,7 @@ unsigned long kvmppc_h_svm_init_abort(struct kvm *kvm) srcu_idx =srcu_read_lock (&kvm->srcu); -kvm_for_each_memslot (memslot,kvm_memslots (kvm)) +kvm_for_each_memslot (memslot, bkt,kvm_memslots (kvm))kvmppc_uvmem_drop_pages (memslot, kvm, false);srcu_read_unlock (&kvm->srcu, srcu_idx);
@@ -821,7 +821,7 @@ unsigned long kvmppc_h_svm_init_done(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *memslot; -int srcu_idx; +int srcu_idx, bkt;long ret =H_SUCCESS ;if (!(kvm->arch.secure_guest &KVMPPC_SECURE_INIT_START ))
@@ -830,7 +830,7 @@ unsigned long kvmppc_h_svm_init_done(struct kvm *kvm)/* migrate any unmoved normal pfn to device pfns*/ srcu_idx =srcu_read_lock (&kvm->srcu); slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) { ret =kvmppc_uv_migrate_mem_slot (kvm, memslot);if (ret) {/*
arch/s390/kvm/kvm-s390.c +11 -13
@@ -1037,13 +1037,13 @@ static int kvm_s390_vm_start_migration(struct kvm *kvm)struct kvm_memory_slot *ms;struct kvm_memslots *slots;unsigned long ram_pages =0 ; -int slotnr; +int bkt;/* migration mode already enabled */ if (kvm->arch.migration_mode)return 0 ; slots =kvm_memslots (kvm); -if (!slots || !slots->used_slots) +if (!slots ||kvm_memslots_empty (slots))return -EINVAL ;if (!kvm->arch.use_cmma) {
@@ -1051,8 +1051,7 @@ static int kvm_s390_vm_start_migration(struct kvm *kvm)return 0 ; }/* mark all the pages in active slots as dirty */ -for (slotnr =0 ; slotnr < slots->used_slots; slotnr++) { - ms = slots->memslots + slotnr; +kvm_for_each_memslot (ms, bkt, slots) {if (!ms->dirty_bitmap)return -EINVAL ;/*
@@ -1976,22 +1975,21 @@ static unsigned long kvm_s390_next_dirty_cmma(struct kvm_memslots *slots, unsigned long cur_gfn) {struct kvm_memory_slot *ms =gfn_to_memslot_approx (slots, cur_gfn); -int slotidx = ms - slots->memslots;unsigned long ofs = cur_gfn - ms->base_gfn; +struct rb_node *mnode = &ms->gfn_node[slots->node_idx];if (ms->base_gfn + ms->npages <= cur_gfn) { - slotidx--; + mnode =rb_next (mnode);/* If we are above the highest slot, wrap around */ -if (slotidx <0 ) - slotidx = slots->used_slots -1 ; +if (!mnode) + mnode =rb_first (&slots->gfn_tree); - ms = slots->memslots + slotidx; + ms = container_of(mnode, struct kvm_memory_slot, gfn_node[slots->node_idx]); ofs =0 ; } ofs =find_next_bit (kvm_second_dirty_bitmap (ms), ms->npages, ofs); -while ((slotidx >0 ) && (ofs >= ms->npages)) { - slotidx--; - ms = slots->memslots + slotidx; +while (ofs >= ms->npages && (mnode =rb_next (mnode))) { + ms = container_of(mnode, struct kvm_memory_slot, gfn_node[slots->node_idx]); ofs =find_next_bit (kvm_second_dirty_bitmap (ms), ms->npages,0 ); }return ms->base_gfn + ofs;
@@ -2004,7 +2002,7 @@ static int kvm_s390_get_cmma(struct kvm *kvm, struct kvm_s390_cmma_log *args,struct kvm_memslots *slots =kvm_memslots (kvm);struct kvm_memory_slot *ms; -if (unlikely (!slots->used_slots)) +if (unlikely (kvm_memslots_empty (slots)))return 0 ; cur_gfn =kvm_s390_next_dirty_cmma (slots, args->start_gfn);
arch/s390/kvm/kvm-s390.h +4 -2
@@ -220,12 +220,14 @@ static inline void kvm_s390_set_user_cpu_state_ctrl(struct kvm *kvm)/* get the end gfn of the last (highest gfn) memslot */ static inline unsigned long kvm_s390_get_gfn_end (struct kvm_memslots *slots) { +struct rb_node *node;struct kvm_memory_slot *ms; -if (WARN_ON (!slots->used_slots)) +if (WARN_ON (kvm_memslots_empty (slots)))return 0 ; - ms = slots->memslots; + node =rb_last (&slots->gfn_tree); + ms = container_of(node, struct kvm_memory_slot, gfn_node[slots->node_idx]);return ms->base_gfn + ms->npages; }
arch/x86/kvm/debugfs.c +3 -3
@@ -107,9 +107,10 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v)write_lock (&kvm->mmu_lock);for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { +int bkt; + slots =__kvm_memslots (kvm, i); -for (j =0 ; j < slots->used_slots; j++) { - slot = &slots->memslots[j]; +kvm_for_each_memslot (slot, bkt, slots)for (k =0 ; k <KVM_NR_PAGE_SIZES ; k++) { rmap = slot->arch.rmap[k]; lpage_size =kvm_mmu_slot_lpages (slot, k +1 );
@@ -121,7 +122,6 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v) cur[index]++; } } - } }write_unlock (&kvm->mmu_lock);
arch/x86/kvm/mmu/mmu.c +4 -4
@@ -3397,7 +3397,7 @@ static int mmu_first_shadow_root_alloc(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *slot; -int r =0 , i; +int r =0 , i, bkt;/* * Check if this is the first shadow root being allocated before
@@ -3422,7 +3422,7 @@ static int mmu_first_shadow_root_alloc(struct kvm *kvm)for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { slots =__kvm_memslots (kvm, i); -kvm_for_each_memslot (slot, slots) { +kvm_for_each_memslot (slot, bkt, slots) {/* * Both of these functions are no-ops if the target is * already allocated, so unconditionally calling both
@@ -5706,14 +5706,14 @@ static bool __kvm_zap_rmaps(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)struct kvm_memslots *slots;bool flush = false;gfn_t start, end; -int i; +int i, bkt;if (!kvm_memslots_have_rmaps (kvm))return flush;for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { slots =__kvm_memslots (kvm, i); -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) { start =max (gfn_start, memslot->base_gfn); end =min (gfn_end, memslot->base_gfn + memslot->npages);if (start >= end)
include/linux/kvm_host.h +71 -72
241 lines not on this page · raw message
virt/kvm/kvm_main.c +393 -368
948 lines not on this page · raw message
PATCH v6 25/29 KVM: Call kvm_arch_flush_shadow_memslot() on the old slot in kvm_invalidate_memslot()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
kvm_invalidate_memslot() calls kvm_arch_flush_shadow_memslot() on the active, but KVM_MEMSLOT_INVALID slot.
Do it on the inactive (but valid) old slot instead since arch code really should not get passed such invalid slot.
--- virt/kvm/kvm_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
virt/kvm/kvm_main.c +1 -1
@@ -1632,7 +1632,7 @@ static void kvm_invalidate_memslot(struct kvm *kvm,* - gfn_to_hva (kvm_read_guest, gfn_to_pfn) * - kvm_is_visible_gfn (mmu_check_root) */ -kvm_arch_flush_shadow_memslot (kvm, working_slot); +kvm_arch_flush_shadow_memslot (kvm, old);/* Was released by kvm_swap_active_memslots, reacquire. */ mutex_lock (&kvm->slots_arch_lock);
Re: [PATCH v6 25/29] KVM: Call kvm_arch_flush_shadow_memslot() on the old slot in kvm_invalidate_memslot()
Quoted 6 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
kvm_invalidate_memslot()callskvm_arch_flush_shadow_memslot()on the active, butKVM_MEMSLOT_INVALIDslot.
Do it on the inactive (but valid) old slot instead since arch code really should not get passed such invalid slot.
One other thing that's worth noting in the changelog is that "old->arch" may have stale data. IMO that's perfectly ok, but it's definitely a quirk. Ideally KVM would disallow touching "arch" for an INVALID slot, but that would require another arch hook if kvm_prepare_memory_region() failed to refresh old->arch if necessary before restoring it. :-/
Paolo, thoughts on this goofy case? I don't love it, but I dislike having
kvm_arch_flush_shadow_memslot(kvm, invalid_slot);
in the final code even more.
Quoted 19 lines
Suggested-by Sean Christopherson <[email protected]>Signed-off-by Maciej S. Szmigiero <[email protected]>--- virt/kvm/kvm_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)virt/kvm/kvm_main.c +1
-1@@ -1632,7 +1632,7 @@ static void kvm_invalidate_memslot(struct kvm *kvm,* - gfn_to_hva (kvm_read_guest, gfn_to_pfn) * - kvm_is_visible_gfn (mmu_check_root) */ -kvm_arch_flush_shadow_memslot (kvm, working_slot); +kvm_arch_flush_shadow_memslot (kvm, old);/* Was released by kvm_swap_active_memslots, reacquire. */ mutex_lock (&kvm->slots_arch_lock);
Re: [PATCH v6 25/29] KVM: Call kvm_arch_flush_shadow_memslot() on the old slot in kvm_invalidate_memslot()
Quoted 10 lines · Sean Christopherson
Quoted 6 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
kvm_invalidate_memslot()callskvm_arch_flush_shadow_memslot()on the active, butKVM_MEMSLOT_INVALIDslot.
Do it on the inactive (but valid) old slot instead since arch code really should not get passed such invalid slot.One other thing that's worth noting in the changelog is that "
old->arch" may have stale data. IMO that's perfectly ok, but it's definitely a quirk.
Will add this caveat to this commit message.
Quoted 4 lines
Ideally KVM
would disallow touching "arch" for an INVALID slot, but that would require another arch hook ifkvm_prepare_memory_region()failed to refreshold->archif necessary before restoring it. :-/
It looks to me that only PPC book3s_hv actually uses the "arch" field in its kvm_arch_flush_shadow_memslot() implementation.
But at the same time it does not seem to do lazy allocation for rmaps or similar stuff.
However, the code there is complex enough for this change to be of higher-than-usual risk.
That's why I have split it out from the main memslots sets rework.
Thanks,
Maciej
PATCH v6 26/29 KVM: Optimize gfn lookup in kvm_zap_gfn_range()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
Introduce a memslots gfn upper bound operation and use it to optimize kvm_zap_gfn_range().
This way this handler can do a quick lookup for intersecting gfns and won't have to do a linear scan of the whole memslot set.
--- arch/x86/kvm/mmu/mmu.c | 12 +++-- include/linux/kvm_host.h | 99 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -3
@@ -5704,19 +5704,22 @@ static bool __kvm_zap_rmaps(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) {const struct kvm_memory_slot *memslot;struct kvm_memslots *slots; +struct kvm_memslot_iter iter;bool flush = false;gfn_t start, end; -int i, bkt; +int i;if (!kvm_memslots_have_rmaps (kvm))return flush;for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { slots =__kvm_memslots (kvm, i); -kvm_for_each_memslot (memslot, bkt, slots) { + +kvm_for_each_memslot_in_gfn_range (&iter, slots, gfn_start, gfn_end) { + memslot =kvm_memslot_iter_slot (&iter); start =max (gfn_start, memslot->base_gfn); end =min (gfn_end, memslot->base_gfn + memslot->npages); -if (start >= end) +if (WARN_ON_ONCE (start >= end))continue ; flush = slot_handle_level_range(kvm, memslot, kvm_zap_rmapp,
@@ -5737,6 +5740,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)bool flush;int i; +if (WARN_ON_ONCE (gfn_end <= gfn_start)) +return ; +write_lock (&kvm->mmu_lock);kvm_inc_notifier_count (kvm, gfn_start, gfn_end);
include/linux/kvm_host.h +99 -0
105 lines not on this page · raw message
Sean Christopherson
maintainerRe: [PATCH v6 26/29] KVM: Optimize gfn lookup in kvm_zap_gfn_range()
Quoted 14 lines · Maciej S. Szmigiero
include/linux/kvm_host.h +6
-0@@ -848,6 +848,105 @@ struct kvm_memory_slot *id_to_memslot(struct kvm_memslots *slots, int id)return NULL ; } +/* Iterator used for walking memslots that overlap a gfn range. */ +struct kvm_memslot_iter { +struct kvm_memslots *slots; +gfn_t end; +struct rb_node *node; +};
...
include/linux/kvm_host.h struct kvm_memory_slot *id_to_memslot(struct kvm_memslots *slots, int id) · line 920
Quoted 3 lines
include/linux/kvm_host.h +3
-0+static inline struct kvm_memory_slot *kvm_memslot_iter_slot (struct kvm_memslot_iter *iter) +{ + return container_of(iter->node, struct kvm_memory_slot, gfn_node[iter->slots->node_idx]);
Having to use a helper in callers of kvm_for_each_memslot_in_gfn_range() is a bit ugly, any reason not to grab @slot as well? Then the callers just do iter.slot, which IMO is much more readable.
And if we do that, I'd also vote to omit slots and end from the iterator. It would mean passing in slots and end to kvm_memslot_iter_is_valid() and kvm_memslot_iter_next(), but that's more idiomatic in a for-loop if iter is considered to be _just_ the iterator part. "slots" is arguable, but "end" really shouldn't be part of the iterator.
Quoted 32 lines
include/linux/kvm_host.h +29
-0+} + +static inline bool kvm_memslot_iter_is_valid (struct kvm_memslot_iter *iter) +{ +struct kvm_memory_slot *memslot; + +if (!iter->node) +return false; + + memslot =kvm_memslot_iter_slot (iter); + +/* +* If this slot starts beyond or at the end of the range so does +* every next one +*/ +return memslot->base_gfn < iter->end; +} + +static inline void kvm_memslot_iter_next (struct kvm_memslot_iter *iter) +{ + iter->node =rb_next (iter->node); +} + +/* Iterate over each memslot at least partially intersecting [start, end) range */ +#define kvm_for_each_memslot_in_gfn_range (iter, slots, start, end) \ + for (kvm_memslot_iter_start(iter, slots, start, end); \ + kvm_memslot_iter_is_valid(iter); \ + kvm_memslot_iter_next(iter)) +/* * KVM_SET_USER_MEMORY_REGION ioctl allows the following operations: * - create a new memory slot
Re: [PATCH v6 26/29] KVM: Optimize gfn lookup in kvm_zap_gfn_range()
Quoted 18 lines · Sean Christopherson
Quoted 14 lines · Maciej S. Szmigiero
include/linux/kvm_host.h +6
-0@@ -848,6 +848,105 @@ struct kvm_memory_slot *id_to_memslot(struct kvm_memslots *slots, int id)return NULL ; } +/* Iterator used for walking memslots that overlap a gfn range. */ +struct kvm_memslot_iter { +struct kvm_memslots *slots; +gfn_t end; +struct rb_node *node; +};...
Quoted 7 lines
include/linux/kvm_host.h struct kvm_memory_slot *id_to_memslot(struct kvm_memslots *slots, int id) · line 920
Quoted 3 lines
include/linux/kvm_host.h +3
-0+static inline struct kvm_memory_slot *kvm_memslot_iter_slot (struct kvm_memslot_iter *iter) +{ + return container_of(iter->node, struct kvm_memory_slot, gfn_node[iter->slots->node_idx]);Having to use a helper in callers of
kvm_for_each_memslot_in_gfn_range()is a bit ugly, any reason not to grab @slot as well? Then the callers just do iter.slot, which IMO is much more readable.
"slot" can be easily calculated from "node" together with either "slots" or "node_idx" (the code above just adjusts a pointer) so storing it in the iterator makes little sense if the later are already stored there.
Quoted 4 lines
And if we do that, I'd also vote to omit slots and end from the iterator. It would mean passing in slots and end to
kvm_memslot_iter_is_valid()andkvm_memslot_iter_next(), but that's more idiomatic in a for-loop if iter is considered to be _just_ the iterator part. "slots" is arguable, but "end" really shouldn't be part of the iterator.
You're right that we can get away with not storing "end", will remove it.
Thanks,
Maciej
Sean Christopherson
maintainerRe: [PATCH v6 26/29] KVM: Optimize gfn lookup in kvm_zap_gfn_range()
Quoted 26 lines · Maciej S. Szmigiero
Quoted 17 lines · Sean Christopherson
Quoted 13 lines · Maciej S. Szmigiero
include/linux/kvm_host.h +6
-0@@ -848,6 +848,105 @@ struct kvm_memory_slot *id_to_memslot(struct kvm_memslots *slots, int id)return NULL ; } +/* Iterator used for walking memslots that overlap a gfn range. */ +struct kvm_memslot_iter { +struct kvm_memslots *slots; +gfn_t end; +struct rb_node *node; +};...
Quoted 7 lines
include/linux/kvm_host.h struct kvm_memory_slot *id_to_memslot(struct kvm_memslots *slots, int id) · line 920
Quoted 3 lines
include/linux/kvm_host.h +3
-0+static inline struct kvm_memory_slot *kvm_memslot_iter_slot (struct kvm_memslot_iter *iter) +{ + return container_of(iter->node, struct kvm_memory_slot, gfn_node[iter->slots->node_idx]);Having to use a helper in callers of
kvm_for_each_memslot_in_gfn_range()is a bit ugly, any reason not to grab @slot as well? Then the callers just do iter.slot, which IMO is much more readable.
Quoted 3 lines
"slot" can be easily calculated from "node" together with either "slots" or "
node_idx" (the code above just adjusts a pointer) so storing it in the iterator makes little sense if the later are already stored there.
I don't want the callers to have to calculate the slot. It's mostly syntatic sugar, but I really do think it improves readability. And since the first thing every caller will do is retrieve the slot, I see no benefit in forcing the caller to do the work.
E.g. in the simple kvm_check_memslot_overlap() usage, iter.slot->id is intuitive and easy to parse, whereas kvm_memslot_iter_slot(&iter)->id is slightly more difficult to parse and raises questions about why a function call is necessary and what the function might be doing.
static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,
gfn_t start, gfn_t end)
{
struct kvm_memslot_iter iter;
kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
if (iter.slot->id != id)
return true;
}
return false;
}vs.
static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,
gfn_t start, gfn_t end)
{
struct kvm_memslot_iter iter;
kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
if (kvm_memslot_iter_slot(&iter)->id != id)
return true;
}
return false;
}Re: [PATCH v6 26/29] KVM: Optimize gfn lookup in kvm_zap_gfn_range()
Quoted 36 lines · Sean Christopherson
Quoted 26 lines · Maciej S. Szmigiero
Quoted 17 lines · Sean Christopherson
Quoted 13 lines · Maciej S. Szmigiero
include/linux/kvm_host.h +6
-0@@ -848,6 +848,105 @@ struct kvm_memory_slot *id_to_memslot(struct kvm_memslots *slots, int id)return NULL ; } +/* Iterator used for walking memslots that overlap a gfn range. */ +struct kvm_memslot_iter { +struct kvm_memslots *slots; +gfn_t end; +struct rb_node *node; +};...
Quoted 7 lines
include/linux/kvm_host.h struct kvm_memory_slot *id_to_memslot(struct kvm_memslots *slots, int id) · line 920
Quoted 3 lines
include/linux/kvm_host.h +3
-0+static inline struct kvm_memory_slot *kvm_memslot_iter_slot (struct kvm_memslot_iter *iter) +{ + return container_of(iter->node, struct kvm_memory_slot, gfn_node[iter->slots->node_idx]);Having to use a helper in callers of
kvm_for_each_memslot_in_gfn_range()is a bit ugly, any reason not to grab @slot as well? Then the callers just do iter.slot, which IMO is much more readable.Quoted 3 lines
"slot" can be easily calculated from "node" together with either "slots" or "
node_idx" (the code above just adjusts a pointer) so storing it in the iterator makes little sense if the later are already stored there.I don't want the callers to have to calculate the slot. It's mostly syntatic sugar, but I really do think it improves readability. And since the first thing every caller will do is retrieve the slot, I see no benefit in forcing the caller to do the work.
Quoted 4 lines
E.g. in the simple
kvm_check_memslot_overlap()usage,iter.slot->idis intuitive and easy to parse, whereas kvm_memslot_iter_slot(&iter)->id is slightly more difficult to parse and raises questions about why a function call is necessary and what the function might be doing.
Personally, I don't think it's that much less readable, but I will change the code to store "slots" instead (as you wish) since it's the last remaining change - other than Paolo's call whether we should keep or drop the kvm_arch_flush_shadow_memslot()-related patch 25.
Thanks,
Maciej
PATCH v6 27/29 KVM: Optimize overlapping memslots check
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
Do a quick lookup for possibly overlapping gfns when creating or moving a memslot instead of performing a linear scan of the whole memslot set.
--- virt/kvm/kvm_main.c | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-)
virt/kvm/kvm_main.c +21 -14
@@ -1817,6 +1817,18 @@ static int kvm_set_memslot(struct kvm *kvm,return 0 ; } +static bool kvm_check_memslot_overlap (struct kvm_memslots *slots,int id, +gfn_t start,gfn_t end) +{ +struct kvm_memslot_iter iter; + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) +if (kvm_memslot_iter_slot (&iter)->id != id) +return true; + +return false; +} +/* * Allocate some memory and give it an address in the guest physical address * space.
@@ -1828,8 +1840,9 @@ static int kvm_set_memslot(struct kvm *kvm,int __kvm_set_memory_region (struct kvm *kvm,const struct kvm_userspace_memory_region *mem) { -struct kvm_memory_slot *old, *tmp; +struct kvm_memory_slot *old;struct kvm_memory_slot new; +struct kvm_memslots *slots;enum kvm_mr_change change;int as_id, id;int r;
@@ -1858,11 +1871,13 @@ int __kvm_set_memory_region(struct kvm *kvm,if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)return -EINVAL ; + slots =__kvm_memslots (kvm, as_id); +/* * Note, the old memslot (and the pointer itself!) may be invalidated * and/or destroyed by kvm_set_memslot(). */ - old =id_to_memslot (__kvm_memslots (kvm, as_id), id); + old =id_to_memslot (slots, id);if (!mem->memory_size) {if (!old || !old->npages)
@@ -1911,18 +1926,10 @@ int __kvm_set_memory_region(struct kvm *kvm,return 0 ; } -if ((change ==KVM_MR_CREATE ) || (change ==KVM_MR_MOVE )) { -int bkt; - -/* Check for overlaps */ -kvm_for_each_memslot (tmp, bkt,__kvm_memslots (kvm, as_id)) { -if (tmp->id == id) -continue ; -if (!((new.base_gfn + new.npages <= tmp->base_gfn) || - (new.base_gfn >= tmp->base_gfn + tmp->npages))) -return -EEXIST ; - } - } +if ((change ==KVM_MR_CREATE || change ==KVM_MR_MOVE ) && +kvm_check_memslot_overlap (slots, id, new.base_gfn, + new.base_gfn + new.npages)) +return -EEXIST ;return kvm_set_memslot (kvm, old, &new, change); }
Re: [PATCH v6 27/29] KVM: Optimize overlapping memslots check
Quoted 17 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
Do a quick lookup for possibly overlapping gfns when creating or moving a memslot instead of performing a linear scan of the whole memslot set.
Signed-off-by Maciej S. Szmigiero <[email protected]>[sean: tweaked params to avoid churn in future cleanup]Signed-off-by Sean Christopherson <[email protected]>--- virt/kvm/kvm_main.c | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-)virt/kvm/kvm_main.c +0
-0
virt/kvm/kvm_main.c static int kvm_set_memslot(struct kvm *kvm, · line 1825
Quoted 9 lines
virt/kvm/kvm_main.c +6
-0@@ -1817,6 +1817,18 @@ static int kvm_set_memslot(struct kvm *kvm,return 0 ; } +static bool kvm_check_memslot_overlap (struct kvm_memslots *slots,int id, +gfn_t start,gfn_t end) +{ +struct kvm_memslot_iter iter; + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end)
The for loop needs curly braces, per coding-style.rst:
Also, use braces when a loop contains more than a single simple statement:
.. code-block:: c
while (condition) {
if (test)
do_something();
}With that,
Re: [PATCH v6 27/29] KVM: Optimize overlapping memslots check
Quoted 34 lines · Sean Christopherson
Quoted 17 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
Do a quick lookup for possibly overlapping gfns when creating or moving a memslot instead of performing a linear scan of the whole memslot set.
Signed-off-by Maciej S. Szmigiero <[email protected]>[sean: tweaked params to avoid churn in future cleanup]Signed-off-by Sean Christopherson <[email protected]>--- virt/kvm/kvm_main.c | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-)virt/kvm/kvm_main.c +0
-0virt/kvm/kvm_main.c static int kvm_set_memslot(struct kvm *kvm, · line 1825
Quoted 9 lines
virt/kvm/kvm_main.c +6
-0@@ -1817,6 +1817,18 @@ static int kvm_set_memslot(struct kvm *kvm,return 0 ; } +static bool kvm_check_memslot_overlap (struct kvm_memslots *slots,int id, +gfn_t start,gfn_t end) +{ +struct kvm_memslot_iter iter; + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end)The for loop needs curly braces, per
coding-style.rst:Also, use braces when a loop contains more than a single simple statement:
.. code-block:: c
Quoted 9 lines
while (condition) { if (test) do_something(); }With that,
Reviewed-by Sean Christopherson <[email protected]>
Will add these braces (and your R-b, too).
Thanks,
Maciej
PATCH v6 28/29 KVM: Wait 'til the bitter end to initialize the "new" memslot
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Initialize the "new" memslot in the !DELETE path only after the various sanity checks have passed. This will allow a future commit to allocate @new dynamically without having to copy a memslot, and without having to deal with freeing @new in error paths and in the "nothing to change" path that's hiding in the sanity checks.
No functional change intended.
--- virt/kvm/kvm_main.c | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-)
virt/kvm/kvm_main.c +20 -17
@@ -1844,6 +1844,8 @@ int __kvm_set_memory_region(struct kvm *kvm,struct kvm_memory_slot new;struct kvm_memslots *slots;enum kvm_mr_change change; +unsigned long npages; +gfn_t base_gfn;int as_id, id;int r;
@@ -1870,6 +1872,8 @@ int __kvm_set_memory_region(struct kvm *kvm,return -EINVAL ;if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)return -EINVAL ; +if ((mem->memory_size >>PAGE_SHIFT ) >KVM_MEM_MAX_NR_PAGES ) +return -EINVAL ; slots =__kvm_memslots (kvm, as_id);
@@ -1893,15 +1897,8 @@ int __kvm_set_memory_region(struct kvm *kvm,return kvm_set_memslot (kvm, old, &new,KVM_MR_DELETE ); } - new.as_id = as_id; - new.id = id; - new.base_gfn = mem->guest_phys_addr >>PAGE_SHIFT ; - new.npages = mem->memory_size >>PAGE_SHIFT ; - new.flags = mem->flags; - new.userspace_addr = mem->userspace_addr; - -if (new.npages >KVM_MEM_MAX_NR_PAGES ) -return -EINVAL ; + base_gfn = (mem->guest_phys_addr >>PAGE_SHIFT ); + npages = (mem->memory_size >>PAGE_SHIFT );if (!old || !old->npages) { change =KVM_MR_CREATE ;
@@ -1910,27 +1907,33 @@ int __kvm_set_memory_region(struct kvm *kvm,* To simplify KVM internals, the total number of pages across * all memslots must fit in an unsigned long. */ -if ((kvm->nr_memslot_pages + new.npages) < kvm->nr_memslot_pages) +if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)return -EINVAL ; }else {/* Modify an existing slot. */ -if ((new.userspace_addr != old->userspace_addr) || - (new.npages != old->npages) || - ((new.flags ^ old->flags) &KVM_MEM_READONLY )) +if ((mem->userspace_addr != old->userspace_addr) || + (npages != old->npages) || + ((mem->flags ^ old->flags) &KVM_MEM_READONLY ))return -EINVAL ; -if (new.base_gfn != old->base_gfn) +if (base_gfn != old->base_gfn) change =KVM_MR_MOVE ; -else if (new.flags != old->flags) +else if (mem->flags != old->flags) change =KVM_MR_FLAGS_ONLY ;else /* Nothing to change. */ return 0 ; }if ((change ==KVM_MR_CREATE || change ==KVM_MR_MOVE ) && -kvm_check_memslot_overlap (slots, id, new.base_gfn, - new.base_gfn + new.npages)) +kvm_check_memslot_overlap (slots, id, base_gfn, base_gfn + npages))return -EEXIST ; + new.as_id = as_id; + new.id = id; + new.base_gfn = base_gfn; + new.npages = npages; + new.flags = mem->flags; + new.userspace_addr = mem->userspace_addr; +return kvm_set_memslot (kvm, old, &new, change); }EXPORT_SYMBOL_GPL (__kvm_set_memory_region);
PATCH v6 29/29 KVM: Dynamically allocate "new" memslots from the get-go
In reply to Maciej S. SzmigieroFrom: Sean Christopherson <[email protected]>
Allocate the "new" memslot for !DELETE memslot updates straight away instead of filling an intermediate on-stack object and forcing kvm_set_memslot() to juggle the allocation and do weird things like reuse the old memslot object in MOVE.
In the MOVE case, this results in an "extra" memslot allocation due to allocating both the "new" slot and the "invalid" slot, but that's a temporary and not-huge allocation, and MOVE is a relatively rare memslot operation.
Regarding MOVE, drop the open-coded management of the gfn tree with a call to kvm_replace_memslot(), which already handles the case where new->base_gfn != old->base_gfn. This is made possible by virtue of not having to copy the "new" memslot data after erasing the old memslot from the gfn tree. Using kvm_replace_memslot(), and more specifically not reusing the old memslot, means the MOVE case now does hva tree and hash list updates, but that's a small price to pay for simplifying the code and making MOVE align with all the other flavors of updates. The "extra" updates are firmly in the noise from a performance perspective, e.g. the "move (in)active area" selfttests show a (very, very) slight improvement.
--- virt/kvm/kvm_main.c | 178 +++++++++++++++++++------------------------- 1 file changed, 77 insertions(+), 101 deletions(-)
virt/kvm/kvm_main.c +77 -101
@@ -1505,23 +1505,25 @@ static int kvm_prepare_memory_region(struct kvm *kvm,* new and KVM isn't using a ring buffer, allocate and initialize a * new bitmap. */ -if (!(new->flags &KVM_MEM_LOG_DIRTY_PAGES )) - new->dirty_bitmap =NULL ; -else if (old->dirty_bitmap) - new->dirty_bitmap = old->dirty_bitmap; -else if (!kvm->dirty_ring_size) { - r =kvm_alloc_dirty_bitmap (new); -if (r) -return r; +if (change !=KVM_MR_DELETE ) { +if (!(new->flags &KVM_MEM_LOG_DIRTY_PAGES )) + new->dirty_bitmap =NULL ; +else if (old && old->dirty_bitmap) + new->dirty_bitmap = old->dirty_bitmap; +else if (!kvm->dirty_ring_size) { + r =kvm_alloc_dirty_bitmap (new); +if (r) +return r; -if (kvm_dirty_log_manual_protect_and_init_set (kvm)) -bitmap_set (new->dirty_bitmap,0 , new->npages); +if (kvm_dirty_log_manual_protect_and_init_set (kvm)) +bitmap_set (new->dirty_bitmap,0 , new->npages); + } } r =kvm_arch_prepare_memory_region (kvm, old, new, change);/* Free the bitmap on failure if it was allocated above. */ -if (r && new->dirty_bitmap && !old->dirty_bitmap) +if (r && new && new->dirty_bitmap && old && !old->dirty_bitmap)kvm_destroy_dirty_bitmap (new);return r;
@@ -1608,16 +1610,16 @@ static void kvm_copy_memslot(struct kvm_memory_slot *dest,static void kvm_invalidate_memslot (struct kvm *kvm,struct kvm_memory_slot *old, -struct kvm_memory_slot *working_slot) +struct kvm_memory_slot *invalid_slot) {/* * Mark the current slot INVALID. As with all memslot modifications, * this must be done on an unreachable slot to avoid modifying the * current slot in the active tree. */ -kvm_copy_memslot (working_slot, old); - working_slot->flags |=KVM_MEMSLOT_INVALID ; -kvm_replace_memslot (kvm, old, working_slot); +kvm_copy_memslot (invalid_slot, old); + invalid_slot->flags |=KVM_MEMSLOT_INVALID ; +kvm_replace_memslot (kvm, old, invalid_slot);/* * Activate the slot that is now marked INVALID, but don't propagate
@@ -1644,20 +1646,15 @@ static void kvm_invalidate_memslot(struct kvm *kvm,* above. Writers are required to retrieve memslots *after* acquiring * slots_arch_lock, thus the active slot's data is guaranteed to be fresh. */ - old->arch = working_slot->arch; + old->arch = invalid_slot->arch; }static void kvm_create_memslot (struct kvm *kvm, -const struct kvm_memory_slot *new, -struct kvm_memory_slot *working) +struct kvm_memory_slot *new) { -/* -* Add the new memslot to the inactive set as a copy of the -* new memslot data provided by userspace. -*/ -kvm_copy_memslot (working, new); -kvm_replace_memslot (kvm,NULL , working); -kvm_activate_memslot (kvm,NULL , working); +/* Add the new memslot to the inactive set and activate. */ +kvm_replace_memslot (kvm,NULL , new); +kvm_activate_memslot (kvm,NULL , new); } static void kvm_delete_memslot(struct kvm *kvm,
@@ -1666,65 +1663,36 @@ static void kvm_delete_memslot(struct kvm *kvm, {/* * Remove the old memslot (in the inactive memslots) by passing NULL as -* the "new" slot. +* the "new" slot, and for the invalid version in the active slots. */ kvm_replace_memslot (kvm, old,NULL ); - -/* And do the same for the invalid version in the active slot. */ kvm_activate_memslot (kvm, invalid_slot,NULL ); - -/* Free the invalid slot, the caller will clean up the old slot. */ -kfree (invalid_slot); } -static struct kvm_memory_slot *kvm_move_memslot (struct kvm *kvm, -struct kvm_memory_slot *old, -const struct kvm_memory_slot *new, -struct kvm_memory_slot *invalid_slot) +static void kvm_move_memslot (struct kvm *kvm, +struct kvm_memory_slot *old, +struct kvm_memory_slot *new, +struct kvm_memory_slot *invalid_slot) { -struct kvm_memslots *slots =kvm_get_inactive_memslots (kvm, old->as_id); - -/* -* The memslot's gfn is changing, remove it from the inactive tree, it -* will be re-added with its updated gfn. Because its range is -* changing, an in-place replace is not possible. -*/ -kvm_erase_gfn_node (slots, old); - -/* -* The old slot is now fully disconnected, reuse its memory for the -* persistent copy of "new". -*/ -kvm_copy_memslot (old, new); - -/* Re-add to the gfn tree with the updated gfn */ -kvm_insert_gfn_node (slots, old); - -/* Replace the current INVALID slot with the updated memslot. */ -kvm_activate_memslot (kvm, invalid_slot, old); -/* -* Clear the INVALID flag so that the invalid_slot is now a perfect -* copy of the old slot. Return it for cleanup in the caller. +* Replace the old memslot in the inactive slots, and then swap slots +* and replace the current INVALID with the new as well. */ -WARN_ON_ONCE (!(invalid_slot->flags &KVM_MEMSLOT_INVALID )); - invalid_slot->flags &= ~KVM_MEMSLOT_INVALID ; -return invalid_slot; +kvm_replace_memslot (kvm, old, new); +kvm_activate_memslot (kvm, invalid_slot, new); }static void kvm_update_flags_memslot (struct kvm *kvm,struct kvm_memory_slot *old, -const struct kvm_memory_slot *new, -struct kvm_memory_slot *working_slot) +struct kvm_memory_slot *new) {/* * Similar to the MOVE case, but the slot doesn't need to be zapped as * an intermediate step. Instead, the old memslot is simply replaced * with a new, updated copy in both memslot sets. */ -kvm_copy_memslot (working_slot, new); -kvm_replace_memslot (kvm, old, working_slot); -kvm_activate_memslot (kvm, old, working_slot); +kvm_replace_memslot (kvm, old, new); +kvm_activate_memslot (kvm, old, new); } static int kvm_set_memslot(struct kvm *kvm,
@@ -1732,19 +1700,9 @@ static int kvm_set_memslot(struct kvm *kvm,struct kvm_memory_slot *new, enum kvm_mr_change change) { - struct kvm_memory_slot *working; + struct kvm_memory_slot *invalid_slot;int r; -/* -* Modifications are done on an unreachable slot. Any changes are then -* (eventually) propagated to both the active and inactive slots. This -* allocation would ideally be on-demand (in helpers), but is done here -* to avoid having to handle failure after kvm_prepare_memory_region(). -*/ - working =kzalloc (sizeof (*working),GFP_KERNEL_ACCOUNT ); -if (!working) -return -ENOMEM ; -/* * Released in kvm_swap_active_memslots. *
@@ -1769,9 +1727,19 @@ static int kvm_set_memslot(struct kvm *kvm,* (and without a lock), a window would exist between effecting the * delete/move and committing the changes in arch code where KVM or a * guest could access a non-existent memslot. +* +* Modifications are done on a temporary, unreachable slot. The old +* slot needs to be preserved in case a later step fails and the +* invalidation needs to be reverted. */ -if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) -kvm_invalidate_memslot (kvm, old, working); +if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) { + invalid_slot =kzalloc (sizeof (*invalid_slot),GFP_KERNEL_ACCOUNT ); +if (!invalid_slot) { +mutex_unlock (&kvm->slots_arch_lock); +return -ENOMEM ; + } +kvm_invalidate_memslot (kvm, old, invalid_slot); + } r =kvm_prepare_memory_region (kvm, old, new, change);if (r) {
@@ -1781,11 +1749,12 @@ static int kvm_set_memslot(struct kvm *kvm,* in the inactive slots. Changing the active memslots also * release slots_arch_lock. */ -if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) -kvm_activate_memslot (kvm, working, old); -else +if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) { +kvm_activate_memslot (kvm, invalid_slot, old); +kfree (invalid_slot); + }else {mutex_unlock (&kvm->slots_arch_lock); -kfree (working); + }return r; }
@@ -1797,16 +1766,20 @@ static int kvm_set_memslot(struct kvm *kvm,* old slot is detached but otherwise preserved. */ if (change ==KVM_MR_CREATE ) -kvm_create_memslot (kvm, new, working); +kvm_create_memslot (kvm, new);else if (change ==KVM_MR_DELETE ) -kvm_delete_memslot (kvm, old, working); +kvm_delete_memslot (kvm, old, invalid_slot);else if (change ==KVM_MR_MOVE ) - old =kvm_move_memslot (kvm, old, new, working); +kvm_move_memslot (kvm, old, new, invalid_slot);else if (change ==KVM_MR_FLAGS_ONLY ) -kvm_update_flags_memslot (kvm, old, new, working); +kvm_update_flags_memslot (kvm, old, new);else BUG (); +/* Free the temporary INVALID slot used for DELETE and MOVE. */ +if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) +kfree (invalid_slot); +/* * No need to refresh new->arch, changes after dropping slots_arch_lock * will directly hit the final, active memsot. Architectures are
@@ -1840,8 +1813,7 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,int __kvm_set_memory_region (struct kvm *kvm,const struct kvm_userspace_memory_region *mem) { -struct kvm_memory_slot *old; -struct kvm_memory_slot new; +struct kvm_memory_slot *old, *new;struct kvm_memslots *slots;enum kvm_mr_change change;unsigned long npages;
@@ -1890,11 +1862,7 @@ int __kvm_set_memory_region(struct kvm *kvm,if (WARN_ON_ONCE (kvm->nr_memslot_pages < old->npages))return -EIO ; -memset (&new,0 ,sizeof (new)); - new.id = id; - new.as_id = as_id; - -return kvm_set_memslot (kvm, old, &new,KVM_MR_DELETE ); +return kvm_set_memslot (kvm, old,NULL ,KVM_MR_DELETE ); } base_gfn = (mem->guest_phys_addr >>PAGE_SHIFT );
@@ -1927,14 +1895,22 @@ int __kvm_set_memory_region(struct kvm *kvm, kvm_check_memslot_overlap(slots, id, base_gfn, base_gfn + npages)) return -EEXIST; - new.as_id = as_id; - new.id = id; - new.base_gfn = base_gfn; - new.npages = npages; - new.flags = mem->flags; - new.userspace_addr = mem->userspace_addr; +/* Allocate a slot that will persist in the memslot. */ + new =kzalloc (sizeof (*new),GFP_KERNEL_ACCOUNT ); +if (!new) +return -ENOMEM ; + + new->as_id = as_id; + new->id = id; + new->base_gfn = base_gfn; + new->npages = npages; + new->flags = mem->flags; + new->userspace_addr = mem->userspace_addr; -return kvm_set_memslot (kvm, old, &new, change); + r =kvm_set_memslot (kvm, old, new, change); +if (r) +kfree (new); +return r; }EXPORT_SYMBOL_GPL (__kvm_set_memory_region);
Sean Christopherson
maintainerQuoted 10 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
This series contains the sixth iteration of the scalable memslots patch set. It is based on Sean's version "5.5", but with integrated patches for issues that arose during its review round.
In addition to that, the
kvm_for_each_memslot_in_gfn_range()implementation was reworked to return only strictly overlapping memslots and to use iterators.
Quoted 7 lines
However, I've dropped a similar
kvm_for_each_memslot_in_hva_range()rework since the existing implementation was already returning only strictly overlapping memslots and it was already based on interval tree iterators, so wrapping them in another layer of iterators would only add unnecessary complexity.
The code in this "for"-like macro is also self-contained and very simple, so let's keep it this way.
If kvm_for_each_memslot_in_hva_range() ever gains a user outside of kvm_main.c it should definitely get an iterator container so that callers don't need to do the container_of() stuff. I'd still prefer a container even now, but it's not a sticking point.