KVM: Scalable memslots implementation
v4 cover KVM: Scalable memslots implementation
From: "Maciej S. Szmigiero" <[email protected]>
The current memslot code uses a (reverse) gfn-ordered memslot array for keeping track of them.
This only allows quick binary search by gfn, quick lookup by hva is not possible (the implementation has to do a linear scan of the whole memslot array).
Because the memslot array that is currently in use cannot be modified every memslot management operation (create, delete, move, change flags) has to make a copy of the whole array so it has a scratch copy to work on.
Strictly speaking, however, it is only necessary to make copy of the memslot that is being modified, copying all the memslots currently present is just a limitation of the array-based memslot implementation.
Two memslot sets, however, are still needed so the VM continues to run on the currently active set while the requested operation is being performed on the second, currently inactive one.
In order to have two memslot sets, but only one copy of the actual memslots it is necessary to split out the memslot data from the memslot sets.
The memslots themselves should be also kept independent of each other so they can be individually added or deleted.
These two memslot sets should normally point to the same set of memslots. They can, however, be desynchronized when performing a memslot management operation by replacing the memslot to be modified by its copy. After the operation is complete, both memslot sets once again point to the same, common set of memslot data.
This series implements the aforementioned idea.
The new implementation uses two trees to perform quick lookups:
For tracking of gfn an ordinary rbtree is used since memslots cannot overlap in the guest address space and so this data structure is sufficient for ensuring that lookups are done quickly.
For tracking of hva, however, an interval tree is needed since they can overlap between memslots.
ID to memslot mappings are kept in a hash table instead of using a statically allocated "id_to_index" array.
The "last used slot" mini-caches (both per-slot set one and per-vCPU one), that keep track of the last found-by-gfn memslot, are still present in the new code.
There was also a desire to make the new structure operate on "pay as you go" basis, that is, that the user only pays the price of the memslot count that is actually used, not of the maximum count allowed.
The operation semantics were carefully matched to the original implementation, the outside-visible behavior should not change. Only the timing will be different.
Making lookup and memslot management operations O(log(n)) brings some performance benefits (tested on a Xeon 8167M machine):
509 slots in use:
Test Before After Improvement Map 0.0232s 0.0223s 4% Unmap 0.0724s 0.0315s 56% Unmap 2M 0.00155s 0.000869s 44% Move active 0.000101s 0.0000792s 22% Move inactive 0.000108s 0.0000847s 21% Slot setup 0.0135s 0.00803s 41%
100 slots in use:
Test Before After Improvement Map 0.0195s 0.0191s None Unmap 0.0374s 0.0312s 17% Unmap 2M 0.000470s 0.000447s 5% Move active 0.0000956s 0.0000800s 16% Move inactive 0.000101s 0.0000840s 17% Slot setup 0.00260s 0.00174s 33%
50 slots in use:
Test Before After Improvement Map 0.0192s 0.0190s None Unmap 0.0339s 0.0311s 8% Unmap 2M 0.000399s 0.000395s None Move active 0.0000999s 0.0000854s 15% Move inactive 0.0000992s 0.0000826s 17% Slot setup 0.00141s 0.000990s 30%
30 slots in use:
Test Before After Improvement Map 0.0192s 0.0190s None Unmap 0.0325s 0.0310s 5% Unmap 2M 0.000373s 0.000373s None Move active 0.000100s 0.0000865s 14% Move inactive 0.000106s 0.0000918s 13% Slot setup 0.000989s 0.000775s 22%
10 slots in use:
Test Before After Improvement Map 0.0192s 0.0186s 3% Unmap 0.0313s 0.0310s None Unmap 2M 0.000348s 0.000351s None Move active 0.000110s 0.0000948s 14% Move inactive 0.000111s 0.0000933s 16% Slot setup 0.000342s 0.000283s 17%
32k slots in use:
Test Before After Improvement Map (8194) 0.200s 0.0541s 73% Unmap 3.88s 0.0351s 99% Unmap 2M 3.88s 0.0348s 99% Move active 0.00142s 0.0000786s 94% Move inactive 0.00148s 0.0000880s 94% Slot setup 16.1s 0.59s 96%
Since the map test can be done with up to 8194 slots, the result above for this test was obtained running it with that maximum number of slots.
In both the old and new memslot code case the measurements were done against the new KVM selftest framework, with TDP MMU disabled.
On x86-64 the code was well tested, passed KVM unit tests and KVM selftests with KASAN on.
And, of course, booted various guests successfully (including nested ones with TDP MMU enabled).
On other KVM platforms the code was compile-tested only.
Changes since v1:
* Drop the already merged HVA handler retpoline-friendliness patch,
* Split the scalable memslots patch into 8 smaller ones,
* Rebase onto the current kvm/queue,
* Make sure that ranges at both memslot's hva_nodes are always initialized,
* Remove kvm_mmu_calculate_default_mmu_pages() prototype, too, when removing this function,
* Redo benchmarks, measure 32k memslots on the old implementation, too.
Changes since v2:
* Rebase onto the current kvm/queue, taking into account the now-merged MMU notifiers rewrite.
This reduces the diffstat by ~50 lines.
Changes since v3:
* Rebase onto the current (Aug 12) kvm/queue, taking into account the introduction of slots_arch_lock (and lazy rmaps allocation) and per-vCPU "last used slot" mini-cache,
* Change n_memslots_pages to u64 to avoid overflowing it on 32-bit KVM,
* Skip the kvm_mmu_change_mmu_pages() call for memslot operations that don't change the total page count anyway,
* Move n_memslots_pages recalc to kvm_arch_prepare_memory_region() so a proper error code can be returned in case we spot an underflow,
* Integrate the open-coded s390 gfn_to_memslot_approx() into the main KVM code by adding "approx" parameter to the existing __gfn_to_memslot() while renaming it to __gfn_to_memslot_approx() and introducing __gfn_to_memslot() wrapper so existing call sites won't be affected, since __gfn_to_memslot() now offers an extra functionality over search_memslots().
This last fact wasn't the case at the time the previous version of this series was posted,
* Split out the move of WARN that's triggered on invalid memslot index to a separate patch,
* Rename the old slot variable in kvm_memslot_delete() to "oldslot", add an additional comment why we delete a memslot from the hash table in kvm_memslot_move_backward() in "KVM: Resolve memslot ID via a hash table instead of via a static array" patch,
* Rename a patch from "Introduce memslots hva tree" to "Use interval tree to do fast hva lookup in memslots",
* Document that the kvm_for_each_hva_range_memslot() range is inclusive,
* Rename kvm_for_each_hva_range_memslot to kvm_for_each_memslot_in_hva_range, move this macro to kvm_main.c,
* Update the WARN in __kvm_handle_hva_range() and kvm_zap_gfn_range() to also trigger on empty ranges,
* Use "bkt" instead of "ctr" for hash bucket index variables,
* Add a comment describing the idea behind the memslots dual set system above struct kvm_memory_slot,
* Rename "memslots_all" to "__memslots" in struct kvm and add comments there describing its two memslot members,
* Remove kvm_memslots_idx() helper and store the set index explicitly in the particular memslots set instead,
* Open code kvm_init_memslots(),
* Rename swap_memslots() into kvm_swap_active_memslots() and make it also swap pointers themselves to the provided two sets,
* Initialize "parent" outside of the for loop in kvm_memslot_gfn_insert(),
* Add kvm_memslot_gfn_erase() and kvm_memslot_gfn_replace() helpers,
* Fold kvm_init_memslot_hva_ranges() into kvm_copy_memslot(),
* Remove kvm_copy_replace_memslot() and introduce kvm_activate_memslot() instead,
* Rename "slotsact" / "slotsina" into "active" / "inactive" in kvm_set_memslot(), add a big comment what are the semantics of "slotact" / "slotina" variables to this function,
* Move WARN about a missing old slot to kvm_set_memslot() and return -EIO in this case,
* Set KVM_MEMSLOT_INVALID flag on a temporary slot before (rather than after) replacing it in the inactive set in kvm_set_memslot() as it is a bit more intuitive way to do it,
* Move handling of an error from kvm_arch_prepare_memory_region() directly below a call to this function in kvm_set_memslot() from the end of this function,
* Rework some of the comments in kvm_set_memslot(),
* Rename "oldslot" / "nslot" into "old" / "new" here and there in "Keep memslots in tree-based structures instead of array-based ones" patch,
* Add Sean's "Co-developed-by:" to the aforementioned patch since many of its above changes are coming from his proposed patch (and he did a lot of good work reviewing both this patch and the whole series),
* Introduce kvm_for_each_memslot_in_gfn_range() and use it in the last two patches,
* Unfold long lines here and there,
* Retest the patch series.
arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/mmu.c | 15 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 3 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 3 +- arch/powerpc/kvm/book3s_hv_nested.c | 4 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 14 +- arch/powerpc/kvm/powerpc.c | 5 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 66 +-- arch/s390/kvm/kvm-s390.h | 14 + arch/s390/kvm/pv.c | 4 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/debugfs.c | 6 +- arch/x86/kvm/mmu/mmu.c | 35 +- arch/x86/kvm/x86.c | 40 +- include/linux/kvm_host.h | 224 +++++++--- virt/kvm/kvm_main.c | 651 +++++++++++++++------------- 21 files changed, 622 insertions(+), 473 deletions(-)
PATCH v4 1/13 KVM: x86: Cache total page count to avoid traversing the memslot array
From: "Maciej S. Szmigiero" <[email protected]>
There is no point in recalculating from scratch the total number of pages in all memslots each time a memslot is created or deleted.
Just cache the value and update it accordingly on each such operation so the code doesn't need to traverse the whole memslot array each time.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/mmu/mmu.c | 24 ------------------------ arch/x86/kvm/x86.c | 18 +++++++++++++++--- 3 files changed, 16 insertions(+), 28 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1025,6 +1025,7 @@ struct kvm_x86_msr_filter {#define APICV_INHIBIT_REASON_X2APIC 5struct kvm_arch { +u64 n_memslots_pages;unsigned long n_used_mmu_pages;unsigned long n_requested_mmu_pages;unsigned long n_max_mmu_pages;
@@ -1552,7 +1553,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, const struct kvm_memory_slot *memslot);void kvm_mmu_zap_all (struct kvm *kvm);void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen); -unsigned long kvm_mmu_calculate_default_mmu_pages (struct kvm *kvm);void kvm_mmu_change_mmu_pages (struct kvm *kvm,unsigned long kvm_nr_mmu_pages);int load_pdptrs (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu,unsigned long cr3);
arch/x86/kvm/mmu/mmu.c +0 -24
@@ -6046,30 +6046,6 @@ int kvm_mmu_module_init(void)return ret; } -/* -* Calculate mmu pages needed for kvm. -*/ -unsigned long kvm_mmu_calculate_default_mmu_pages (struct kvm *kvm) -{ -unsigned long nr_mmu_pages; -unsigned long nr_pages =0 ; -struct kvm_memslots *slots; -struct kvm_memory_slot *memslot; -int i; - -for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { - slots =__kvm_memslots (kvm, i); - -kvm_for_each_memslot (memslot, slots) - nr_pages += memslot->npages; - } - - nr_mmu_pages = nr_pages *KVM_PERMILLE_MMU_PAGES /1000 ; - nr_mmu_pages =max (nr_mmu_pages,KVM_MIN_ALLOC_MMU_PAGES ); - -return nr_mmu_pages; -} -void kvm_mmu_destroy (struct kvm_vcpu *vcpu) {kvm_mmu_unload (vcpu);
arch/x86/kvm/x86.c +15 -3
@@ -11588,9 +11588,21 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,const struct kvm_memory_slot *new, enum kvm_mr_change change) { - if (!kvm->arch.n_requested_mmu_pages) - kvm_mmu_change_mmu_pages(kvm, - kvm_mmu_calculate_default_mmu_pages(kvm)); + if (change == KVM_MR_CREATE) + kvm->arch.n_memslots_pages += new->npages; + else if (change == KVM_MR_DELETE) { + WARN_ON(kvm->arch.n_memslots_pages < old->npages); + kvm->arch.n_memslots_pages -= old->npages; + } + +if (!kvm->arch.n_requested_mmu_pages) { +unsigned long nr_mmu_pages; + + nr_mmu_pages = kvm->arch.n_memslots_pages * +KVM_PERMILLE_MMU_PAGES /1000 ; + nr_mmu_pages =max (nr_mmu_pages,KVM_MIN_ALLOC_MMU_PAGES ); +kvm_mmu_change_mmu_pages (kvm, nr_mmu_pages); + }kvm_mmu_slot_apply_flags (kvm, old, new, change);
kernel test robot
build botRe: [PATCH v4 01/13] KVM: x86: Cache total page count to avoid traversing the memslot array
Thank you for the patch! Yet something to improve:
Hi "Maciej,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on kvm/queue]
[also build test ERROR on kvms390/next v5.14-rc5 next-20210813] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Maciej-S-Szmigiero/KVM-Scalable-memslots-implementation/20210814-040318 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue config: i386-randconfig-a005-20210814 (attached as .config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/172883fe88fb9dc0533b916b2f4177008d35e05a git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Maciej-S-Szmigiero/KVM-Scalable-memslots-implementation/20210814-040318 git checkout 172883fe88fb9dc0533b916b2f4177008d35e05a # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>, old ones prefixed by <<):
>> ERROR: modpost: "__udivdi3" [arch/x86/kvm/kvm.ko] undefined!
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [PATCH v4 01/13] KVM: x86: Cache total page count to avoid traversing the memslot array
In reply to Maciej S. SzmigieroThank you for the patch! Yet something to improve:
Hi "Maciej,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on kvm/queue]
[also build test ERROR on kvms390/next v5.14-rc5 next-20210813] [cannot apply to kvm-ppc/kvm-ppc-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Maciej-S-Szmigiero/KVM-Scalable-memslots-implementation/20210814-040318 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue config: i386-randconfig-s001-20210813 (attached as .config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.3-348-gf0e6938b-dirty # https://github.com/0day-ci/linux/commit/172883fe88fb9dc0533b916b2f4177008d35e05a git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Maciej-S-Szmigiero/KVM-Scalable-memslots-implementation/20210814-040318 git checkout 172883fe88fb9dc0533b916b2f4177008d35e05a # save the attached .config to linux build tree make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' O=build_dir ARCH=i386 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
ld: arch/x86/kvm/x86.o: in function `kvm_arch_commit_memory_region':
>> arch/x86/kvm/x86.c:11602: undefined reference to `__udivdi3'
vim +11602 arch/x86/kvm/x86.c
11584
11585 void kvm_arch_commit_memory_region(struct kvm *kvm,
11586 const struct kvm_userspace_memory_region *mem,
11587 struct kvm_memory_slot *old,
11588 const struct kvm_memory_slot *new,
11589 enum kvm_mr_change change)
11590 {
11591 if (change == KVM_MR_CREATE)
11592 kvm->arch.n_memslots_pages += new->npages;
11593 else if (change == KVM_MR_DELETE) {
11594 WARN_ON(kvm->arch.n_memslots_pages < old->npages);
11595 kvm->arch.n_memslots_pages -= old->npages;
11596 }
11597
11598 if (!kvm->arch.n_requested_mmu_pages) {
11599 unsigned long nr_mmu_pages;
11600
11601 nr_mmu_pages = kvm->arch.n_memslots_pages *
11602 KVM_PERMILLE_MMU_PAGES / 1000;
11603 nr_mmu_pages = max(nr_mmu_pages, KVM_MIN_ALLOC_MMU_PAGES);
11604 kvm_mmu_change_mmu_pages(kvm, nr_mmu_pages);
11605 }
11606
11607 kvm_mmu_slot_apply_flags(kvm, old, new, change);
11608
11609 /* Free the arrays associated with the old memslot. */
11610 if (change == KVM_MR_MOVE)
11611 kvm_arch_free_memslot(kvm, old);
11612 }
11613
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [PATCH v4 01/13] KVM: x86: Cache total page count to avoid traversing the memslot array
Quoted 4 lines · kernel test robot
Hi "Maciej,
Thank you for the patch! Yet something to improve:
(..)
Quoted 2 lines
ld:
arch/x86/kvm/x86.o: in function `kvm_arch_commit_memory_region':Quoted 1 line
Quoted 1 line
arch/x86/kvm/x86.c:11602: undefined reference to `__udivdi3'
You are right, Mr Robot, this needs to be wrapped in div64_u64. Will fix this when I do a respin of the series.
Thanks,
Maciej
PATCH v4 2/13 KVM: x86: Don't call kvm_mmu_change_mmu_pages() if the count hasn't changed
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
There is no point in calling kvm_mmu_change_mmu_pages() for memslot operations that don't change the total page count, so do it just for KVM_MR_CREATE and KVM_MR_DELETE.
--- arch/x86/kvm/x86.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-)
arch/x86/kvm/x86.c +14 -12
@@ -11588,20 +11588,22 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,const struct kvm_memory_slot *new, enum kvm_mr_change change) { - if (change == KVM_MR_CREATE) - kvm->arch.n_memslots_pages += new->npages; - else if (change == KVM_MR_DELETE) { - WARN_ON(kvm->arch.n_memslots_pages < old->npages); - kvm->arch.n_memslots_pages -= old->npages; - } + if (change == KVM_MR_CREATE || change == KVM_MR_DELETE) { + if (change == KVM_MR_CREATE) + kvm->arch.n_memslots_pages += new->npages; + else { + WARN_ON(kvm->arch.n_memslots_pages < old->npages); + kvm->arch.n_memslots_pages -= old->npages; + } -if (!kvm->arch.n_requested_mmu_pages) { -unsigned long nr_mmu_pages; +if (!kvm->arch.n_requested_mmu_pages) { +unsigned long nr_mmu_pages; - nr_mmu_pages = kvm->arch.n_memslots_pages * -KVM_PERMILLE_MMU_PAGES /1000 ; - nr_mmu_pages =max (nr_mmu_pages,KVM_MIN_ALLOC_MMU_PAGES ); -kvm_mmu_change_mmu_pages (kvm, nr_mmu_pages); + nr_mmu_pages = kvm->arch.n_memslots_pages * +KVM_PERMILLE_MMU_PAGES /1000 ; + nr_mmu_pages =max (nr_mmu_pages,KVM_MIN_ALLOC_MMU_PAGES ); +kvm_mmu_change_mmu_pages (kvm, nr_mmu_pages); + } }kvm_mmu_slot_apply_flags (kvm, old, new, change);
PATCH v4 3/13 KVM: Add "old" memslot parameter to kvm_arch_prepare_memory_region()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
This is needed for the next commit, which moves n_memslots_pages recalculation from kvm_arch_commit_memory_region() to the aforementioned function to allow for returning an error code.
While we are at it let's also rename the "memslot" parameter to "new" for consistency with kvm_arch_commit_memory_region(), which uses the same argument set now.
No functional change intended.
--- arch/arm64/kvm/mmu.c | 7 ++++--- arch/mips/kvm/mips.c | 3 ++- arch/powerpc/kvm/powerpc.c | 5 +++-- arch/s390/kvm/kvm-s390.c | 3 ++- arch/x86/kvm/x86.c | 5 +++-- include/linux/kvm_host.h | 3 ++- virt/kvm/kvm_main.c | 2 +- 7 files changed, 17 insertions(+), 11 deletions(-)
arch/arm64/kvm/mmu.c +4 -3
@@ -1434,7 +1434,8 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,const struct kvm_userspace_memory_region *mem,enum kvm_mr_change change) {
@@ -1450,7 +1451,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,* Prevent userspace from creating a memory region outside of the IPA * space addressable by the KVM guest IPA space. */ -if ((memslot->base_gfn + memslot->npages) > (kvm_phys_size (kvm) >>PAGE_SHIFT )) +if ((new->base_gfn + new->npages) > (kvm_phys_size (kvm) >>PAGE_SHIFT ))return -EFAULT ;mmap_read_lock (current->mm);
@@ -1482,7 +1483,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,if (vma->vm_flags &VM_PFNMAP ) {/* IO region dirty page logging not allowed */ -if (memslot->flags &KVM_MEM_LOG_DIRTY_PAGES ) { +if (new->flags &KVM_MEM_LOG_DIRTY_PAGES ) { ret = -EINVAL ;break ; }
arch/mips/kvm/mips.c +2 -1
@@ -233,7 +233,8 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,const struct kvm_userspace_memory_region *mem,enum kvm_mr_change change) {
arch/powerpc/kvm/powerpc.c +3 -2
@@ -706,11 +706,12 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,const struct kvm_userspace_memory_region *mem,enum kvm_mr_change change) { -return kvmppc_core_prepare_memory_region (kvm, memslot, mem, change); +return kvmppc_core_prepare_memory_region (kvm, new, mem, change); } void kvm_arch_commit_memory_region(struct kvm *kvm,
arch/s390/kvm/kvm-s390.c +2 -1
@@ -4990,7 +4990,8 @@ vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf)/* Section: memory related */ int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,const struct kvm_userspace_memory_region *mem,enum kvm_mr_change change) {
arch/x86/kvm/x86.c +3 -2
@@ -11485,12 +11485,13 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) }int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,const struct kvm_userspace_memory_region *mem,enum kvm_mr_change change) {if (change ==KVM_MR_CREATE || change ==KVM_MR_MOVE ) -return kvm_alloc_memslot_metadata (kvm, memslot, +return kvm_alloc_memslot_metadata (kvm, new, mem->memory_size >>PAGE_SHIFT );return 0 ; }
include/linux/kvm_host.h +2 -1
@@ -832,7 +832,8 @@ int __kvm_set_memory_region(struct kvm *kvm,void kvm_arch_free_memslot (struct kvm *kvm,struct kvm_memory_slot *slot);void kvm_arch_memslots_updated (struct kvm *kvm,u64 gen);int kvm_arch_prepare_memory_region (struct kvm *kvm, -struct kvm_memory_slot *memslot, +const struct kvm_memory_slot *old, +struct kvm_memory_slot *new,const struct kvm_userspace_memory_region *mem,enum kvm_mr_change change); void kvm_arch_commit_memory_region(struct kvm *kvm,
virt/kvm/kvm_main.c +1 -1
@@ -1583,7 +1583,7 @@ static int kvm_set_memslot(struct kvm *kvm,kvm_copy_memslots (slots,__kvm_memslots (kvm, as_id)); } - r =kvm_arch_prepare_memory_region (kvm, new, mem, change); + r =kvm_arch_prepare_memory_region (kvm, old, new, mem, change);if (r) goto out_slots;
PATCH v4 4/13 KVM: x86: Move n_memslots_pages recalc to kvm_arch_prepare_memory_region()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
This allows us to return a proper error code in case we spot an underflow.
--- arch/x86/kvm/x86.c | 43 +++++++++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 18 deletions(-)
arch/x86/kvm/x86.c +25 -18
@@ -11490,9 +11490,23 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,const struct kvm_userspace_memory_region *mem, enum kvm_mr_change change) { - if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) - return kvm_alloc_memslot_metadata(kvm, new, - mem->memory_size >> PAGE_SHIFT); + if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) { + int ret; + + ret =kvm_alloc_memslot_metadata (kvm, new, + mem->memory_size >>PAGE_SHIFT ); +if (ret) +return ret; + +if (change ==KVM_MR_CREATE ) + kvm->arch.n_memslots_pages += new->npages; + } else if (change == KVM_MR_DELETE) { +if (WARN_ON (kvm->arch.n_memslots_pages < old->npages)) +return -EIO ; + + kvm->arch.n_memslots_pages -= old->npages; + } +return 0 ; }
@@ -11589,22 +11603,15 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_memory_slot *new, enum kvm_mr_change change) { - if (change == KVM_MR_CREATE || change == KVM_MR_DELETE) { - if (change == KVM_MR_CREATE) - kvm->arch.n_memslots_pages += new->npages; - else { - WARN_ON(kvm->arch.n_memslots_pages < old->npages); - kvm->arch.n_memslots_pages -= old->npages; - } - -if (!kvm->arch.n_requested_mmu_pages) { -unsigned long nr_mmu_pages; +/* Only CREATE or DELETE affects n_memslots_pages */ +if ((change ==KVM_MR_CREATE || change ==KVM_MR_DELETE ) && + !kvm->arch.n_requested_mmu_pages) { +unsigned long nr_mmu_pages; - nr_mmu_pages = kvm->arch.n_memslots_pages * -KVM_PERMILLE_MMU_PAGES /1000 ; - nr_mmu_pages =max (nr_mmu_pages,KVM_MIN_ALLOC_MMU_PAGES ); -kvm_mmu_change_mmu_pages (kvm, nr_mmu_pages); - } + nr_mmu_pages = kvm->arch.n_memslots_pages * +KVM_PERMILLE_MMU_PAGES /1000 ; + nr_mmu_pages =max (nr_mmu_pages,KVM_MIN_ALLOC_MMU_PAGES ); +kvm_mmu_change_mmu_pages (kvm, nr_mmu_pages); }kvm_mmu_slot_apply_flags (kvm, old, new, change);
PATCH v4 5/13 KVM: Integrate gfn_to_memslot_approx() into search_memslots()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
s390 arch has gfn_to_memslot_approx() which is almost identical to search_memslots(), differing only in that in case the gfn falls in a hole one of the memslots bordering the hole is returned.
Add this lookup mode as an option to search_memslots() so we don't have two almost identical functions for looking up a memslot by its gfn.
--- arch/s390/kvm/kvm-s390.c | 39 ++------------------------------------- include/linux/kvm_host.h | 25 ++++++++++++++++++++++--- virt/kvm/kvm_main.c | 2 +- 3 files changed, 25 insertions(+), 41 deletions(-)
arch/s390/kvm/kvm-s390.c +2 -37
@@ -1941,41 +1941,6 @@ static long kvm_s390_set_skeys(struct kvm *kvm, struct kvm_s390_skeys *args)/* for consistency */ #define KVM_S390_CMMA_SIZE_MAX ((u32)KVM_S390_SKEYS_MAX) -/* -* Similar to gfn_to_memslot, but returns the index of a memslot also when the -* address falls in a hole. In that case the index of one of the memslots -* bordering the hole is returned. -*/ -static int gfn_to_memslot_approx (struct kvm_memslots *slots,gfn_t gfn) -{ -int start =0 , end = slots->used_slots; -int slot =atomic_read (&slots->last_used_slot); -struct kvm_memory_slot *memslots = slots->memslots; - -if (gfn >= memslots[slot].base_gfn && - gfn < memslots[slot].base_gfn + memslots[slot].npages) -return slot; - -while (start < end) { - slot = start + (end - start) /2 ; - -if (gfn >= memslots[slot].base_gfn) - end = slot; -else - start = slot +1 ; - } - -if (start >= slots->used_slots) -return slots->used_slots -1 ; - -if (gfn >= memslots[start].base_gfn && - gfn < memslots[start].base_gfn + memslots[start].npages) { -atomic_set (&slots->last_used_slot, start); - } - -return start; -} -static int kvm_s390_peek_cmma (struct kvm *kvm,struct kvm_s390_cmma_log *args,u8 *res,unsigned long bufsize) {
@@ -2002,8 +1967,8 @@ static int kvm_s390_peek_cmma(struct kvm *kvm, struct kvm_s390_cmma_log *args,static unsigned long kvm_s390_next_dirty_cmma (struct kvm_memslots *slots,unsigned long cur_gfn) { -int slotidx =gfn_to_memslot_approx (slots, cur_gfn); -struct kvm_memory_slot *ms = slots->memslots + slotidx; +struct kvm_memory_slot *ms =__gfn_to_memslot_approx (slots, cur_gfn, true); +int slotidx = ms - slots->memslots;unsigned long ofs = cur_gfn - ms->base_gfn;if (ms->base_gfn + ms->npages <= cur_gfn) {
include/linux/kvm_host.h +22 -3
@@ -1232,10 +1232,14 @@ try_get_memslot(struct kvm_memslots *slots, int slot_index, gfn_t gfn)* Returns a pointer to the memslot that contains gfn and records the index of * the slot in index. Otherwise returns NULL. * +* With "approx" set returns the memslot also when the address falls +* in a hole. In that case one of the memslots bordering the hole is +* returned. +* * IMPORTANT: Slots are sorted from highest GFN to lowest GFN! */ static inline struct kvm_memory_slot * -search_memslots (struct kvm_memslots *slots,gfn_t gfn,int *index) +search_memslots (struct kvm_memslots *slots,gfn_t gfn,int *index,bool approx) {int start =0 , end = slots->used_slots;struct kvm_memory_slot *memslots = slots->memslots;
@@ -1253,11 +1257,20 @@ search_memslots(struct kvm_memslots *slots, gfn_t gfn, int *index) start = slot +1 ; } +if (approx && start >= slots->used_slots) { + *index = slots->used_slots -1 ; +return &memslots[slots->used_slots -1 ]; + } + slot =try_get_memslot (slots, start, gfn);if (slot) { *index = start;return slot; } +if (approx) { + *index = start; +return &memslots[start]; + }return NULL ; }
@@ -1268,7 +1281,7 @@ search_memslots(struct kvm_memslots *slots, gfn_t gfn, int *index)* itself isn't here as an inline because that would bloat other code too much. */ static inline struct kvm_memory_slot * -__gfn_to_memslot (struct kvm_memslots *slots,gfn_t gfn) +__gfn_to_memslot_approx (struct kvm_memslots *slots,gfn_t gfn,bool approx) {struct kvm_memory_slot *slot;int slot_index =atomic_read (&slots->last_used_slot);
@@ -1277,7 +1290,7 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn)if (slot)return slot; - slot =search_memslots (slots, gfn, &slot_index); + slot =search_memslots (slots, gfn, &slot_index, approx);if (slot) {atomic_set (&slots->last_used_slot, slot_index);return slot;
@@ -1286,6 +1299,12 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn)return NULL ; } +static inline struct kvm_memory_slot * +__gfn_to_memslot (struct kvm_memslots *slots,gfn_t gfn) +{ +return __gfn_to_memslot_approx (slots, gfn, false); +} +static inline unsigned long __gfn_to_hva_memslot (const struct kvm_memory_slot *slot,gfn_t gfn) {
virt/kvm/kvm_main.c +1 -1
@@ -2074,7 +2074,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfn* search_memslots() instead of __gfn_to_memslot() to avoid * thrashing the VM-wide last_used_index in kvm_memslots. */ - slot =search_memslots (slots, gfn, &slot_index); + slot =search_memslots (slots, gfn, &slot_index, false);if (slot) { vcpu->last_used_slot = slot_index;return slot;
Re: [PATCH v4 05/13] KVM: Integrate gfn_to_memslot_approx() into search_memslots()
Quoted 10 lines · On Fri, 13 Aug 2021 21:33:18 +0200 "Maciej S. Szmigiero"…
From: "Maciej S. Szmigiero" <[email protected]>
s390 arch has
gfn_to_memslot_approx()which is almost identical tosearch_memslots(), differing only in that in case the gfn falls in a hole one of the memslots bordering the hole is returned.Add this lookup mode as an option to
search_memslots()so we don't have two almost identical functions for looking up a memslot by its gfn.Signed-off-by Maciej S. Szmigiero <[email protected]>
Quoted 148 lines
--- arch/s390/kvm/kvm-s390.c | 39 ++------------------------------------- include/linux/kvm_host.h | 25 ++++++++++++++++++++++--- virt/kvm/kvm_main.c | 2 +- 3 files changed, 25 insertions(+), 41 deletions(-)arch/s390/kvm/kvm-s390.c +2
-37@@ -1941,41 +1941,6 @@ static long kvm_s390_set_skeys(struct kvm *kvm, struct kvm_s390_skeys *args)/* for consistency */ #define KVM_S390_CMMA_SIZE_MAX ((u32)KVM_S390_SKEYS_MAX) -/* -* Similar to gfn_to_memslot, but returns the index of a memslot also when the -* address falls in a hole. In that case the index of one of the memslots -* bordering the hole is returned. -*/ -static int gfn_to_memslot_approx (struct kvm_memslots *slots,gfn_t gfn) -{ -int start =0 , end = slots->used_slots; -int slot =atomic_read (&slots->last_used_slot); -struct kvm_memory_slot *memslots = slots->memslots; - -if (gfn >= memslots[slot].base_gfn && - gfn < memslots[slot].base_gfn + memslots[slot].npages) -return slot; - -while (start < end) { - slot = start + (end - start) /2 ; - -if (gfn >= memslots[slot].base_gfn) - end = slot; -else - start = slot +1 ; - } - -if (start >= slots->used_slots) -return slots->used_slots -1 ; - -if (gfn >= memslots[start].base_gfn && - gfn < memslots[start].base_gfn + memslots[start].npages) { -atomic_set (&slots->last_used_slot, start); - } - -return start; -} -static int kvm_s390_peek_cmma (struct kvm *kvm,struct kvm_s390_cmma_log *args,u8 *res,unsigned long bufsize) {@@ -2002,8 +1967,8 @@ static int kvm_s390_peek_cmma(struct kvm *kvm, struct kvm_s390_cmma_log *args,static unsigned long kvm_s390_next_dirty_cmma (struct kvm_memslots *slots,unsigned long cur_gfn) { -int slotidx =gfn_to_memslot_approx (slots, cur_gfn); -struct kvm_memory_slot *ms = slots->memslots + slotidx; +struct kvm_memory_slot *ms =__gfn_to_memslot_approx (slots, cur_gfn, true); +int slotidx = ms - slots->memslots;unsigned long ofs = cur_gfn - ms->base_gfn;if (ms->base_gfn + ms->npages <= cur_gfn) {include/linux/kvm_host.h +22
-3@@ -1232,10 +1232,14 @@ try_get_memslot(struct kvm_memslots *slots, int slot_index, gfn_t gfn)* Returns a pointer to the memslot that contains gfn and records the index of * the slot in index. Otherwise returns NULL. * +* With "approx" set returns the memslot also when the address falls +* in a hole. In that case one of the memslots bordering the hole is +* returned. +* * IMPORTANT: Slots are sorted from highest GFN to lowest GFN! */ static inline struct kvm_memory_slot * -search_memslots (struct kvm_memslots *slots,gfn_t gfn,int *index) +search_memslots (struct kvm_memslots *slots,gfn_t gfn,int *index,bool approx) {int start =0 , end = slots->used_slots;struct kvm_memory_slot *memslots = slots->memslots;@@ -1253,11 +1257,20 @@ search_memslots(struct kvm_memslots *slots, gfn_t gfn, int *index) start = slot +1 ; } +if (approx && start >= slots->used_slots) { + *index = slots->used_slots -1 ; +return &memslots[slots->used_slots -1 ]; + } + slot =try_get_memslot (slots, start, gfn);if (slot) { *index = start;return slot; } +if (approx) { + *index = start; +return &memslots[start]; + }return NULL ; }@@ -1268,7 +1281,7 @@ search_memslots(struct kvm_memslots *slots, gfn_t gfn, int *index)* itself isn't here as an inline because that would bloat other code too much. */ static inline struct kvm_memory_slot * -__gfn_to_memslot (struct kvm_memslots *slots,gfn_t gfn) +__gfn_to_memslot_approx (struct kvm_memslots *slots,gfn_t gfn,bool approx) {struct kvm_memory_slot *slot;int slot_index =atomic_read (&slots->last_used_slot);@@ -1277,7 +1290,7 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn)if (slot)return slot; - slot =search_memslots (slots, gfn, &slot_index); + slot =search_memslots (slots, gfn, &slot_index, approx);if (slot) {atomic_set (&slots->last_used_slot, slot_index);return slot;@@ -1286,6 +1299,12 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn)return NULL ; } +static inline struct kvm_memory_slot * +__gfn_to_memslot (struct kvm_memslots *slots,gfn_t gfn) +{ +return __gfn_to_memslot_approx (slots, gfn, false); +} +static inline unsigned long __gfn_to_hva_memslot (const struct kvm_memory_slot *slot,gfn_t gfn) {virt/kvm/kvm_main.c +1
-1@@ -2074,7 +2074,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfn* search_memslots() instead of __gfn_to_memslot() to avoid * thrashing the VM-wide last_used_index in kvm_memslots. */ - slot =search_memslots (slots, gfn, &slot_index); + slot =search_memslots (slots, gfn, &slot_index, false);if (slot) { vcpu->last_used_slot = slot_index;return slot;
PATCH v4 6/13 KVM: Move WARN on invalid memslot index to update_memslots()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
Since kvm_memslot_move_forward() can theoretically return a negative memslot index even when kvm_memslot_move_backward() returned a positive one (and so did not WARN) let's just move the warning to the common code.
--- virt/kvm/kvm_main.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
virt/kvm/kvm_main.c +4 -2
@@ -1293,8 +1293,7 @@ static inline int kvm_memslot_move_backward(struct kvm_memslots *slots,struct kvm_memory_slot *mslots = slots->memslots;int i; -if (WARN_ON_ONCE (slots->id_to_index[memslot->id] ==-1 ) || -WARN_ON_ONCE (!slots->used_slots)) +if (slots->id_to_index[memslot->id] ==-1 || !slots->used_slots)return -1 ;/*
@@ -1398,6 +1397,9 @@ static void update_memslots(struct kvm_memslots *slots, i =kvm_memslot_move_backward (slots, memslot); i =kvm_memslot_move_forward (slots, memslot, i); +if (WARN_ON_ONCE (i <0 )) +return ; +/* * Copy the memslot to its new position in memslots and update * its index accordingly.
Re: [PATCH v4 06/13] KVM: Move WARN on invalid memslot index to update_memslots()
Quoted 7 lines · On Fri, 13 Aug 2021 21:33:19 +0200 "Maciej S. Szmigiero"…
From: "Maciej S. Szmigiero" <[email protected]>
Since
kvm_memslot_move_forward()can theoretically return a negative memslot index even whenkvm_memslot_move_backward()returned a positive one (and so did not WARN) let's just move the warning to the common code.Signed-off-by Maciej S. Szmigiero <[email protected]>
Quoted 28 lines
--- virt/kvm/kvm_main.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +4
-2@@ -1293,8 +1293,7 @@ static inline int kvm_memslot_move_backward(struct kvm_memslots *slots,struct kvm_memory_slot *mslots = slots->memslots;int i; -if (WARN_ON_ONCE (slots->id_to_index[memslot->id] ==-1 ) || -WARN_ON_ONCE (!slots->used_slots)) +if (slots->id_to_index[memslot->id] ==-1 || !slots->used_slots)return -1 ;/* @@ -1398,6 +1397,9 @@ static void update_memslots(struct kvm_memslots *slots, i =kvm_memslot_move_backward (slots, memslot); i =kvm_memslot_move_forward (slots, memslot, i); +if (WARN_ON_ONCE (i <0 )) +return ; +/* * Copy the memslot to its new position in memslots and update * its index accordingly.
Re: [PATCH v4 06/13] KVM: Move WARN on invalid memslot index to update_memslots()
In reply to Maciej S. SzmigieroQuoted 27 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
Since
kvm_memslot_move_forward()can theoretically return a negative memslot index even whenkvm_memslot_move_backward()returned a positive one (and so did not WARN) let's just move the warning to the common code.Signed-off-by Maciej S. Szmigiero <[email protected]>--- virt/kvm/kvm_main.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +1
-2@@ -1293,8 +1293,7 @@ static inline int kvm_memslot_move_backward(struct kvm_memslots *slots,struct kvm_memory_slot *mslots = slots->memslots;int i; -if (WARN_ON_ONCE (slots->id_to_index[memslot->id] ==-1 ) || -WARN_ON_ONCE (!slots->used_slots)) +if (slots->id_to_index[memslot->id] ==-1 || !slots->used_slots)return -1 ;/*
virt/kvm/kvm_main.c static void update_memslots(struct kvm_memslots *slots, · line 1405
Quoted 9 lines
virt/kvm/kvm_main.c +3
-0@@ -1398,6 +1397,9 @@ static void update_memslots(struct kvm_memslots *slots, i =kvm_memslot_move_backward (slots, memslot); i =kvm_memslot_move_forward (slots, memslot, i); +if (WARN_ON_ONCE (i <0 )) +return ; +/* * Copy the memslot to its new position in memslots and update * its index accordingly.
Note that WARN_ON_* is frowned upon, because it can result in crashes with panic_on_warn enabled, which is what some distributions do enable.
We tend to work around that by using pr_warn()/pr_warn_once(), avoiding eventually crashing the system when there is a way to continue.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v4 06/13] KVM: Move WARN on invalid memslot index to update_memslots()
Quoted 38 lines · David Hildenbrand
Quoted 24 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
Since
kvm_memslot_move_forward()can theoretically return a negative memslot index even whenkvm_memslot_move_backward()returned a positive one (and so did not WARN) let's just move the warning to the common code.Signed-off-by Maciej S. Szmigiero <[email protected]>---
virt/kvm/kvm_main.c| 6 ++++--1 file changed, 4 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +1
-2@@ -1293,8 +1293,7 @@ static inline int kvm_memslot_move_backward(struct kvm_memslots *slots, struct kvm_memory_slot *mslots = slots->memslots; int i; - if (WARN_ON_ONCE(slots->id_to_index[memslot->id] == -1) || - WARN_ON_ONCE(!slots->used_slots)) + if (slots->id_to_index[memslot->id] == -1 || !slots->used_slots) return -1; /*virt/kvm/kvm_main.c static void update_memslots(struct kvm_memslots *slots, · line 1405
Quoted 9 lines
virt/kvm/kvm_main.c +3
-0@@ -1398,6 +1397,9 @@ static void update_memslots(struct kvm_memslots *slots, i = kvm_memslot_move_backward(slots, memslot); i = kvm_memslot_move_forward(slots, memslot, i); + if (WARN_ON_ONCE(i < 0)) + return; + /** Copy the memslot to its new position in memslots and update * its index accordingly. Note that WARN_ON_* is frowned upon, because it can result in crashes with
panic_on_warnenabled, which is what some distributions do enable.
Quoted 2 lines
We tend to work around that by using pr_warn()/pr_warn_once(), avoiding eventually crashing the system when there is a way to continue.
This patch uses WARN_ON_ONCE because:
1) It was used in the old code and the patch merely moves the check from kvm_memslot_move_backward() to its caller,
2) This chunk of code is wholly replaced by patch 11 from this series anyway ("Keep memslots in tree-based structures instead of array-based ones").
Thanks,
Maciej
Re: [PATCH v4 06/13] KVM: Move WARN on invalid memslot index to update_memslots()
Quoted 42 lines · Maciej S. Szmigiero
Quoted 38 lines · David Hildenbrand
Quoted 24 lines · Maciej S. Szmigiero
From: "Maciej S. Szmigiero" <[email protected]>
Since
kvm_memslot_move_forward()can theoretically return a negative memslot index even whenkvm_memslot_move_backward()returned a positive one (and so did not WARN) let's just move the warning to the common code.Signed-off-by Maciej S. Szmigiero <[email protected]>--- virt/kvm/kvm_main.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +1
-2@@ -1293,8 +1293,7 @@ static inline int kvm_memslot_move_backward(struct kvm_memslots *slots, struct kvm_memory_slot *mslots = slots->memslots; int i; - if (WARN_ON_ONCE(slots->id_to_index[memslot->id] == -1) || - WARN_ON_ONCE(!slots->used_slots)) + if (slots->id_to_index[memslot->id] == -1 || !slots->used_slots) return -1; /*virt/kvm/kvm_main.c static void update_memslots(struct kvm_memslots *slots, · line 1405
Quoted 9 lines
virt/kvm/kvm_main.c +3
-0@@ -1398,6 +1397,9 @@ static void update_memslots(struct kvm_memslots *slots, i = kvm_memslot_move_backward(slots, memslot); i = kvm_memslot_move_forward(slots, memslot, i); + if (WARN_ON_ONCE(i < 0)) + return; + /** Copy the memslot to its new position in memslots and update * its index accordingly. Note that WARN_ON_* is frowned upon, because it can result in crashes with
panic_on_warnenabled, which is what some distributions do enable.Quoted 2 lines
We tend to work around that by using pr_warn()/pr_warn_once(), avoiding eventually crashing the system when there is a way to continue.
Quoted 6 lines
This patch uses
WARN_ON_ONCEbecause:
1) It was used in the old code and the patch merely moves the check fromkvm_memslot_move_backward()to its caller,2) This chunk of code is wholly replaced by patch 11 from this series anyway ("Keep memslots in tree-based structures instead of array-based ones").
Okay, that makes sense then, thanks!
Signature · 3 lines
Thanks, David / dhildenb
PATCH v4 7/13 KVM: Just resync arch fields when slots_arch_lock gets reacquired
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
There is no need to copy the whole memslot data after releasing slots_arch_lock for a moment to install temporary memslots copy in kvm_set_memslot() since this lock only protects the arch field of each memslot.
Just resync this particular field after reacquiring slots_arch_lock.
--- virt/kvm/kvm_main.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
virt/kvm/kvm_main.c +12 -5
@@ -1494,6 +1494,15 @@ static void kvm_copy_memslots(struct kvm_memslots *to,memcpy (to, from,kvm_memslots_size (from->used_slots)); } +static void kvm_copy_memslots_arch (struct kvm_memslots *to, +struct kvm_memslots *from) +{ +int i; + +for (i =0 ; i < from->used_slots; i++) + to->memslots[i].arch = from->memslots[i].arch; +} +/* * Note, at a minimum, the current number of used slots must be allocated, even * when deleting a memslot, as we need a complete duplicate of the memslots for
@@ -1579,10 +1588,10 @@ static int kvm_set_memslot(struct kvm *kvm,/* * The arch-specific fields of the memslots could have changed * between releasing the slots_arch_lock in -* install_new_memslots and here, so get a fresh copy of the -* slots. +* install_new_memslots and here, so get a fresh copy of these +* fields. */ -kvm_copy_memslots (slots,__kvm_memslots (kvm, as_id)); +kvm_copy_memslots_arch (slots,__kvm_memslots (kvm, as_id)); } r =kvm_arch_prepare_memory_region (kvm, old, new, mem, change);
@@ -1599,8 +1608,6 @@ static int kvm_set_memslot(struct kvm *kvm, out_slots:if (change ==KVM_MR_DELETE || change ==KVM_MR_MOVE ) { - slot =id_to_memslot (slots, old->id); - slot->flags &= ~KVM_MEMSLOT_INVALID ; slots =install_new_memslots (kvm, as_id, slots); }else {mutex_unlock (&kvm->slots_arch_lock);
PATCH v4 8/13 KVM: Resolve memslot ID via a hash table instead of via a static array
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
Memslot ID to the corresponding memslot mappings are currently kept as indices in static id_to_index array.
The size of this array depends on the maximum allowed memslot count (regardless of the number of memslots actually in use).
This has become especially problematic recently, when memslot count cap was removed, so the maximum count is now full 32k memslots - the maximum allowed by the current KVM API.
Keeping these IDs in a hash table (instead of an array) avoids this problem.
Resolving a memslot ID to the actual memslot (instead of its index) will also enable transitioning away from an array-based implementation of the whole memslots structure in a later commit.
--- include/linux/kvm_host.h | 16 +++++------ virt/kvm/kvm_main.c | 61 +++++++++++++++++++++++++++++++--------- 2 files changed, 55 insertions(+), 22 deletions(-)
include/linux/kvm_host.h +8 -8
@@ -29,6 +29,7 @@#include <linux/refcount.h> #include <linux/nospec.h> #include <linux/notifier.h> +#include <linux/hashtable.h> #include <asm/signal.h> #include <linux/kvm.h>
@@ -426,6 +427,7 @@ static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu)#define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)struct kvm_memory_slot { +struct hlist_node id_node;gfn_t base_gfn;unsigned long npages;unsigned long *dirty_bitmap;
@@ -528,7 +530,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)struct kvm_memslots {u64 generation;/* The mapping table from slot id to the index in memslots[]. */ -short id_to_index[KVM_MEM_SLOTS_NUM ]; + DECLARE_HASHTABLE(id_hash, 7);atomic_t last_used_slot;int used_slots;struct kvm_memory_slot memslots[];
@@ -795,16 +797,14 @@ static inline struct kvm_memslots *kvm_vcpu_memslots(struct kvm_vcpu *vcpu)static inline struct kvm_memory_slot *id_to_memslot (struct kvm_memslots *slots,int id) { -int index = slots->id_to_index[id];struct kvm_memory_slot *slot; -if (index <0 ) -return NULL ; - - slot = &slots->memslots[index]; +hash_for_each_possible (slots->id_hash, slot, id_node, id) { +if (slot->id == id) +return slot; + } -WARN_ON (slot->id != id); -return slot; +return NULL ; }/*
virt/kvm/kvm_main.c +47 -14
159 lines not on this page · raw message
PATCH v4 9/13 KVM: Use interval tree to do fast hva lookup in memslots
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
The current memslots implementation only allows quick binary search by gfn, quick lookup by hva is not possible - the implementation has to do a linear scan of the whole memslots array, even though the operation being performed might apply just to a single memslot.
This significantly hurts performance of per-hva operations with higher memslot counts.
Since hva ranges can overlap between memslots an interval tree is needed for tracking them.
--- arch/arm64/kvm/Kconfig | 1 + arch/mips/kvm/Kconfig | 1 + arch/powerpc/kvm/Kconfig | 1 + arch/s390/kvm/Kconfig | 1 + arch/x86/kvm/Kconfig | 1 + include/linux/kvm_host.h | 3 +++ virt/kvm/kvm_main.c | 48 ++++++++++++++++++++++++++++++++++------ 7 files changed, 49 insertions(+), 7 deletions(-)
arch/arm64/kvm/Kconfig +1 -0
@@ -37,6 +37,7 @@ menuconfig KVM select HAVE_KVM_IRQ_BYPASS select HAVE_KVM_VCPU_RUN_PID_CHANGE select SCHED_INFO + select INTERVAL_TREE help Support hosting virtualized guest machines.
arch/mips/kvm/Kconfig +1 -0
@@ -27,6 +27,7 @@ config KVM select KVM_MMIO select MMU_NOTIFIER select SRCU + select INTERVAL_TREE help Support for hosting Guest kernels.
arch/powerpc/kvm/Kconfig +1 -0
@@ -26,6 +26,7 @@ config KVM select KVM_VFIO select IRQ_BYPASS_MANAGER select HAVE_KVM_IRQ_BYPASS + select INTERVAL_TREE config KVM_BOOK3S_HANDLER bool
arch/s390/kvm/Kconfig +1 -0
@@ -33,6 +33,7 @@ config KVM select HAVE_KVM_NO_POLL select SRCU select KVM_VFIO + select INTERVAL_TREE help Support hosting paravirtualized guest machines using the SIE virtualization capability on the mainframe. This should work
arch/x86/kvm/Kconfig +1 -0
@@ -43,6 +43,7 @@ config KVM select KVM_GENERIC_DIRTYLOG_READ_PROTECT select KVM_VFIO select SRCU + select INTERVAL_TREE select HAVE_KVM_PM_NOTIFIER if PM help Support hosting fully virtualized guest machines using hardware
include/linux/kvm_host.h +3 -0
@@ -30,6 +30,7 @@#include <linux/nospec.h> #include <linux/notifier.h> #include <linux/hashtable.h> +#include <linux/interval_tree.h> #include <asm/signal.h> #include <linux/kvm.h>
@@ -428,6 +429,7 @@ static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu)struct kvm_memory_slot {struct hlist_node id_node; +struct interval_tree_node hva_node;gfn_t base_gfn;unsigned long npages;unsigned long *dirty_bitmap;
@@ -529,6 +531,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)*/ struct kvm_memslots {u64 generation; +struct rb_root_cached hva_tree;/* The mapping table from slot id to the index in memslots[]. */ DECLARE_HASHTABLE(id_hash, 7);atomic_t last_used_slot;
virt/kvm/kvm_main.c +41 -7
138 lines not on this page · raw message
PATCH v4 10/13 KVM: s390: Introduce kvm_s390_get_gfn_end()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
And use it where s390 code would just access the memslot with the highest gfn directly.
No functional change intended.
--- arch/s390/kvm/kvm-s390.c | 2 +- arch/s390/kvm/kvm-s390.h | 12 ++++++++++++ arch/s390/kvm/pv.c | 4 +--- 3 files changed, 14 insertions(+), 4 deletions(-)
arch/s390/kvm/kvm-s390.c +1 -1
@@ -2006,7 +2006,7 @@ static int kvm_s390_get_cmma(struct kvm *kvm, struct kvm_s390_cmma_log *args,if (!ms)return 0 ; next_gfn =kvm_s390_next_dirty_cmma (slots, cur_gfn +1 ); - mem_end = slots->memslots[0 ].base_gfn + slots->memslots[0 ].npages; + mem_end =kvm_s390_get_gfn_end (slots);while (args->count < bufsize) { hva =gfn_to_hva (kvm, cur_gfn);
arch/s390/kvm/kvm-s390.h +12 -0
@@ -208,6 +208,18 @@ static inline int kvm_s390_user_cpu_state_ctrl(struct kvm *kvm)return kvm->arch.user_cpu_state_ctrl !=0 ; } +/* get the end gfn of the last (highest gfn) memslot */ +static inline unsigned long kvm_s390_get_gfn_end (struct kvm_memslots *slots) +{ +struct kvm_memory_slot *ms; + +if (WARN_ON (!slots->used_slots)) +return 0 ; + + ms = slots->memslots; +return ms->base_gfn + ms->npages; +} +/* implemented in pv.c */ int kvm_s390_pv_destroy_cpu (struct kvm_vcpu *vcpu,u16 *rc,u16 *rrc);int kvm_s390_pv_create_cpu (struct kvm_vcpu *vcpu,u16 *rc,u16 *rrc);
arch/s390/kvm/pv.c +1 -3
@@ -117,7 +117,6 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)unsigned long base = uv_info.guest_base_stor_len;unsigned long virt = uv_info.guest_virt_var_stor_len;unsigned long npages =0 , vlen =0 ; -struct kvm_memory_slot *memslot; kvm->arch.pv.stor_var =NULL ; kvm->arch.pv.stor_base =__get_free_pages (GFP_KERNEL_ACCOUNT ,get_order (base));
@@ -131,8 +130,7 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)* Slots are sorted by GFN */ mutex_lock (&kvm->slots_lock); - memslot =kvm_memslots (kvm)->memslots; - npages = memslot->base_gfn + memslot->npages; + npages =kvm_s390_get_gfn_end (kvm_memslots (kvm));mutex_unlock (&kvm->slots_lock); kvm->arch.pv.guest_len = npages *PAGE_SIZE ;
Re: [PATCH v4 10/13] KVM: s390: Introduce kvm_s390_get_gfn_end()
Quoted 8 lines · On Fri, 13 Aug 2021 21:33:23 +0200 "Maciej S. Szmigiero"…
From: "Maciej S. Szmigiero" <[email protected]>
And use it where s390 code would just access the memslot with the highest gfn directly.
No functional change intended.
Signed-off-by Maciej S. Szmigiero <[email protected]>
Quoted 64 lines
--- arch/s390/kvm/kvm-s390.c | 2 +- arch/s390/kvm/kvm-s390.h | 12 ++++++++++++ arch/s390/kvm/pv.c | 4 +--- 3 files changed, 14 insertions(+), 4 deletions(-)arch/s390/kvm/kvm-s390.c +1
-1@@ -2006,7 +2006,7 @@ static int kvm_s390_get_cmma(struct kvm *kvm, struct kvm_s390_cmma_log *args,if (!ms)return 0 ; next_gfn =kvm_s390_next_dirty_cmma (slots, cur_gfn +1 ); - mem_end = slots->memslots[0 ].base_gfn + slots->memslots[0 ].npages; + mem_end =kvm_s390_get_gfn_end (slots);while (args->count < bufsize) { hva =gfn_to_hva (kvm, cur_gfn);arch/s390/kvm/kvm-s390.h +12
-0@@ -208,6 +208,18 @@ static inline int kvm_s390_user_cpu_state_ctrl(struct kvm *kvm)return kvm->arch.user_cpu_state_ctrl !=0 ; } +/* get the end gfn of the last (highest gfn) memslot */ +static inline unsigned long kvm_s390_get_gfn_end (struct kvm_memslots *slots) +{ +struct kvm_memory_slot *ms; + +if (WARN_ON (!slots->used_slots)) +return 0 ; + + ms = slots->memslots; +return ms->base_gfn + ms->npages; +} +/* implemented in pv.c */ int kvm_s390_pv_destroy_cpu (struct kvm_vcpu *vcpu,u16 *rc,u16 *rrc);int kvm_s390_pv_create_cpu (struct kvm_vcpu *vcpu,u16 *rc,u16 *rrc);arch/s390/kvm/pv.c +1
-3@@ -117,7 +117,6 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)unsigned long base = uv_info.guest_base_stor_len;unsigned long virt = uv_info.guest_virt_var_stor_len;unsigned long npages =0 , vlen =0 ; -struct kvm_memory_slot *memslot; kvm->arch.pv.stor_var =NULL ; kvm->arch.pv.stor_base =__get_free_pages (GFP_KERNEL_ACCOUNT ,get_order (base));@@ -131,8 +130,7 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)* Slots are sorted by GFN */ mutex_lock (&kvm->slots_lock); - memslot =kvm_memslots (kvm)->memslots; - npages = memslot->base_gfn + memslot->npages; + npages =kvm_s390_get_gfn_end (kvm_memslots (kvm));mutex_unlock (&kvm->slots_lock); kvm->arch.pv.guest_len = npages *PAGE_SIZE ;
PATCH v4 11/13 KVM: Keep memslots in tree-based structures instead of array-based ones
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
The current memslot code uses a (reverse gfn-ordered) memslot array for keeping track of them.
Because the memslot array that is currently in use cannot be modified every memslot management operation (create, delete, move, change flags) has to make a copy of the whole array so it has a scratch copy to work on.
Strictly speaking, however, it is only necessary to make copy of the memslot that is being modified, copying all the memslots currently present is just a limitation of the array-based memslot implementation.
Two memslot sets, however, are still needed so the VM continues to run on the currently active set while the requested operation is being performed on the second, currently inactive one.
In order to have two memslot sets, but only one copy of actual memslots it is necessary to split out the memslot data from the memslot sets.
The memslots themselves should be also kept independent of each other so they can be individually added or deleted.
These two memslot sets should normally point to the same set of memslots. They can, however, be desynchronized when performing a memslot management operation by replacing the memslot to be modified by its copy. After the operation is complete, both memslot sets once again point to the same, common set of memslot data.
This commit implements the aforementioned idea.
For tracking of gfns an ordinary rbtree is used since memslots cannot overlap in the guest address space and so this data structure is sufficient for ensuring that lookups are done quickly.
The "last used slot" mini-caches (both per-slot set one and per-vCPU one), that keep track of the last found-by-gfn memslot, are still present in the new code.
--- arch/arm64/kvm/mmu.c | 8 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 3 +- arch/powerpc/kvm/book3s_hv_nested.c | 4 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 14 +- arch/s390/kvm/kvm-s390.c | 24 +- arch/s390/kvm/kvm-s390.h | 6 +- arch/x86/kvm/debugfs.c | 6 +- arch/x86/kvm/mmu/mmu.c | 4 +- arch/x86/kvm/x86.c | 4 +- include/linux/kvm_host.h | 140 +++--- virt/kvm/kvm_main.c | 659 +++++++++++++--------------- 12 files changed, 425 insertions(+), 451 deletions(-)
arch/arm64/kvm/mmu.c +4 -4
@@ -209,13 +209,13 @@ static void stage2_flush_vm(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *memslot; -int idx; +int idx, bkt; idx =srcu_read_lock (&kvm->srcu);spin_lock (&kvm->mmu_lock); slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) +kvm_for_each_memslot (memslot, bkt, slots)stage2_flush_memslot (kvm, memslot);spin_unlock (&kvm->mmu_lock);
@@ -548,14 +548,14 @@ void stage2_unmap_vm(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *memslot; -int idx; +int idx, bkt; idx =srcu_read_lock (&kvm->srcu);mmap_read_lock (current->mm);spin_lock (&kvm->mmu_lock); slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) +kvm_for_each_memslot (memslot, bkt, slots)stage2_unmap_memslot (kvm, memslot);spin_unlock (&kvm->mmu_lock);
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -2
@@ -734,11 +734,11 @@ void kvmppc_rmap_reset(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *memslot; -int srcu_idx; +int srcu_idx, bkt; srcu_idx =srcu_read_lock (&kvm->srcu); slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) {/* Mutual exclusion with kvm_unmap_hva_range etc. */ spin_lock (&kvm->mmu_lock);/*
arch/powerpc/kvm/book3s_hv.c +2 -1
@@ -5791,11 +5791,12 @@ static int kvmhv_svm_off(struct kvm *kvm)for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) {struct kvm_memory_slot *memslot;struct kvm_memslots *slots =__kvm_memslots (kvm, i); +int bkt;if (!slots)continue ; -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) {kvmppc_uvmem_drop_pages (memslot, kvm, true);uv_unregister_mem_slot (kvm->arch.lpid, memslot->id); }
arch/powerpc/kvm/book3s_hv_nested.c +2 -2
@@ -730,7 +730,7 @@ void kvmhv_release_all_nested(struct kvm *kvm)struct kvm_nested_guest *gp;struct kvm_nested_guest *freelist =NULL ;struct kvm_memory_slot *memslot; -int srcu_idx; +int srcu_idx, bkt;spin_lock (&kvm->mmu_lock);for (i =0 ; i <= kvm->arch.max_nested_lpid; i++) {
@@ -751,7 +751,7 @@ void kvmhv_release_all_nested(struct kvm *kvm) } srcu_idx =srcu_read_lock (&kvm->srcu); -kvm_for_each_memslot (memslot,kvm_memslots (kvm)) +kvm_for_each_memslot (memslot, bkt,kvm_memslots (kvm))kvmhv_free_memslot_nest_rmap (memslot);srcu_read_unlock (&kvm->srcu, srcu_idx); }
arch/powerpc/kvm/book3s_hv_uvmem.c +7 -7
@@ -459,7 +459,7 @@ unsigned long kvmppc_h_svm_init_start(struct kvm *kvm)struct kvm_memslots *slots;struct kvm_memory_slot *memslot, *m;int ret =H_SUCCESS ; -int srcu_idx; +int srcu_idx, bkt; kvm->arch.secure_guest =KVMPPC_SECURE_INIT_START ;
@@ -478,7 +478,7 @@ unsigned long kvmppc_h_svm_init_start(struct kvm *kvm)/* register the memslot */ slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) { ret =__kvmppc_uvmem_memslot_create (kvm, memslot);if (ret)break ;
@@ -486,7 +486,7 @@ unsigned long kvmppc_h_svm_init_start(struct kvm *kvm)if (ret) { slots =kvm_memslots (kvm); -kvm_for_each_memslot (m, slots) { +kvm_for_each_memslot (m, bkt, slots) {if (m == memslot)break ;__kvmppc_uvmem_memslot_delete (kvm, memslot);
@@ -647,7 +647,7 @@ void kvmppc_uvmem_drop_pages(const struct kvm_memory_slot *slot,unsigned long kvmppc_h_svm_init_abort (struct kvm *kvm) { -int srcu_idx; +int srcu_idx, bkt;struct kvm_memory_slot *memslot;/*
@@ -662,7 +662,7 @@ unsigned long kvmppc_h_svm_init_abort(struct kvm *kvm) srcu_idx =srcu_read_lock (&kvm->srcu); -kvm_for_each_memslot (memslot,kvm_memslots (kvm)) +kvm_for_each_memslot (memslot, bkt,kvm_memslots (kvm))kvmppc_uvmem_drop_pages (memslot, kvm, false);srcu_read_unlock (&kvm->srcu, srcu_idx);
@@ -821,7 +821,7 @@ unsigned long kvmppc_h_svm_init_done(struct kvm *kvm) {struct kvm_memslots *slots;struct kvm_memory_slot *memslot; -int srcu_idx; +int srcu_idx, bkt;long ret =H_SUCCESS ;if (!(kvm->arch.secure_guest &KVMPPC_SECURE_INIT_START ))
@@ -830,7 +830,7 @@ unsigned long kvmppc_h_svm_init_done(struct kvm *kvm)/* migrate any unmoved normal pfn to device pfns*/ srcu_idx =srcu_read_lock (&kvm->srcu); slots =kvm_memslots (kvm); -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) { ret =kvmppc_uv_migrate_mem_slot (kvm, memslot);if (ret) {/*
arch/s390/kvm/kvm-s390.c +11 -13
@@ -1035,13 +1035,13 @@ static int kvm_s390_vm_start_migration(struct kvm *kvm)struct kvm_memory_slot *ms;struct kvm_memslots *slots;unsigned long ram_pages =0 ; -int slotnr; +int bkt;/* migration mode already enabled */ if (kvm->arch.migration_mode)return 0 ; slots =kvm_memslots (kvm); -if (!slots || !slots->used_slots) +if (!slots ||kvm_memslots_empty (slots))return -EINVAL ;if (!kvm->arch.use_cmma) {
@@ -1049,8 +1049,7 @@ static int kvm_s390_vm_start_migration(struct kvm *kvm)return 0 ; }/* mark all the pages in active slots as dirty */ -for (slotnr =0 ; slotnr < slots->used_slots; slotnr++) { - ms = slots->memslots + slotnr; +kvm_for_each_memslot (ms, bkt, slots) {if (!ms->dirty_bitmap)return -EINVAL ;/*
@@ -1968,22 +1967,21 @@ static unsigned long kvm_s390_next_dirty_cmma(struct kvm_memslots *slots, unsigned long cur_gfn) {struct kvm_memory_slot *ms =__gfn_to_memslot_approx (slots, cur_gfn, true); -int slotidx = ms - slots->memslots;unsigned long ofs = cur_gfn - ms->base_gfn; +struct rb_node *mnode = &ms->gfn_node[slots->node_idx];if (ms->base_gfn + ms->npages <= cur_gfn) { - slotidx--; + mnode =rb_next (mnode);/* If we are above the highest slot, wrap around */ -if (slotidx <0 ) - slotidx = slots->used_slots -1 ; +if (!mnode) + mnode =rb_first (&slots->gfn_tree); - ms = slots->memslots + slotidx; + ms = container_of(mnode, struct kvm_memory_slot, gfn_node[slots->node_idx]); ofs =0 ; } ofs =find_next_bit (kvm_second_dirty_bitmap (ms), ms->npages, ofs); -while ((slotidx >0 ) && (ofs >= ms->npages)) { - slotidx--; - ms = slots->memslots + slotidx; +while (ofs >= ms->npages && (mnode =rb_next (mnode))) { + ms = container_of(mnode, struct kvm_memory_slot, gfn_node[slots->node_idx]); ofs =find_next_bit (kvm_second_dirty_bitmap (ms), ms->npages,0 ); }return ms->base_gfn + ofs;
@@ -1996,7 +1994,7 @@ static int kvm_s390_get_cmma(struct kvm *kvm, struct kvm_s390_cmma_log *args,struct kvm_memslots *slots =kvm_memslots (kvm);struct kvm_memory_slot *ms; -if (unlikely (!slots->used_slots)) +if (unlikely (kvm_memslots_empty (slots)))return 0 ; cur_gfn =kvm_s390_next_dirty_cmma (slots, args->start_gfn);
arch/s390/kvm/kvm-s390.h +4 -2
@@ -211,12 +211,14 @@ static inline int kvm_s390_user_cpu_state_ctrl(struct kvm *kvm)/* get the end gfn of the last (highest gfn) memslot */ static inline unsigned long kvm_s390_get_gfn_end (struct kvm_memslots *slots) { +struct rb_node *node;struct kvm_memory_slot *ms; -if (WARN_ON (!slots->used_slots)) +if (WARN_ON (kvm_memslots_empty (slots)))return 0 ; - ms = slots->memslots; + node =rb_last (&slots->gfn_tree); + ms = container_of(node, struct kvm_memory_slot, gfn_node[slots->node_idx]);return ms->base_gfn + ms->npages; }
arch/x86/kvm/debugfs.c +3 -3
@@ -108,9 +108,10 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v)write_lock (&kvm->mmu_lock);for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { +int bkt; + slots =__kvm_memslots (kvm, i); -for (j =0 ; j < slots->used_slots; j++) { - slot = &slots->memslots[j]; +kvm_for_each_memslot (slot, bkt, slots)for (k =0 ; k <KVM_NR_PAGE_SIZES ; k++) { rmap = slot->arch.rmap[k]; lpage_size =kvm_mmu_slot_lpages (slot, k +1 );
@@ -122,7 +123,6 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v) cur[index]++; } } - } }write_unlock (&kvm->mmu_lock);
arch/x86/kvm/mmu/mmu.c +3 -1
@@ -5643,8 +5643,10 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)if (kvm_memslots_have_rmaps (kvm)) {for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { +int bkt; + slots =__kvm_memslots (kvm, i); -kvm_for_each_memslot (memslot, slots) { +kvm_for_each_memslot (memslot, bkt, slots) {gfn_t start, end; start =max (gfn_start, memslot->base_gfn);
arch/x86/kvm/x86.c +3 -1
@@ -11384,8 +11384,10 @@ int alloc_all_memslots_rmaps(struct kvm *kvm) }for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { +int bkt; + slots =__kvm_memslots (kvm, i); -kvm_for_each_memslot (slot, slots) { +kvm_for_each_memslot (slot, bkt, slots) { r =memslot_rmap_alloc (slot, slot->npages);if (r) {mutex_unlock (&kvm->slots_arch_lock);
include/linux/kvm_host.h +67 -73
232 lines not on this page · raw message
virt/kvm/kvm_main.c +317 -342
848 lines not on this page · raw message
PATCH v4 12/13 KVM: Optimize gfn lookup in kvm_zap_gfn_range()
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
Introduce a memslots gfn upper bound operation and use it to optimize kvm_zap_gfn_range().
This way this handler can do a quick lookup for intersecting gfns and won't have to do a linear scan of the whole memslot set.
--- arch/x86/kvm/mmu/mmu.c | 11 +++++-- include/linux/kvm_host.h | 69 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -2
@@ -5637,18 +5637,25 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)int i;bool flush = false; +if (WARN_ON_ONCE (gfn_end <= gfn_start)) +return ; +write_lock (&kvm->mmu_lock);kvm_inc_notifier_count (kvm, gfn_start, gfn_end);if (kvm_memslots_have_rmaps (kvm)) {for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { -int bkt; +int idx; +struct rb_node *node; slots =__kvm_memslots (kvm, i); -kvm_for_each_memslot (memslot, bkt, slots) { + idx = slots->node_idx; + +kvm_for_each_memslot_in_gfn_range (node, slots, gfn_start, gfn_end) {gfn_t start, end; + memslot = container_of(node, struct kvm_memory_slot, gfn_node[idx]); start =max (gfn_start, memslot->base_gfn); end =min (gfn_end, memslot->base_gfn + memslot->npages);if (start >= end)
include/linux/kvm_host.h +69 -0
75 lines not on this page · raw message
PATCH v4 13/13 KVM: Optimize overlapping memslots check
In reply to Maciej S. SzmigieroFrom: "Maciej S. Szmigiero" <[email protected]>
Do a quick lookup for possibly overlapping gfns when creating or moving a memslot instead of performing a linear scan of the whole memslot set.
--- virt/kvm/kvm_main.c | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-)
virt/kvm/kvm_main.c +27 -9
@@ -1679,6 +1679,30 @@ static int kvm_delete_memslot(struct kvm *kvm,return kvm_set_memslot (kvm, mem, old, &new, as_id,KVM_MR_DELETE ); } +static bool kvm_check_memslot_overlap (struct kvm_memslots *slots, +struct kvm_memory_slot *nslot) +{ +int idx = slots->node_idx; +gfn_t nend = nslot->base_gfn + nslot->npages; +struct rb_node *node; + +kvm_for_each_memslot_in_gfn_range (node, slots, nslot->base_gfn, nend) { +struct kvm_memory_slot *cslot; +gfn_t cend; + + cslot = container_of(node, struct kvm_memory_slot, gfn_node[idx]); + cend = cslot->base_gfn + cslot->npages; +if (cslot->id == nslot->id) +continue ; + +/* kvm_for_each_in_gfn_no_more() guarantees that cslot->base_gfn < nend */ +if (cend > nslot->base_gfn) +return true; + } + +return false; +} +/* * Allocate some memory and give it an address in the guest physical address * space.
@@ -1764,16 +1788,10 @@ int __kvm_set_memory_region(struct kvm *kvm, }if ((change ==KVM_MR_CREATE ) || (change ==KVM_MR_MOVE )) { -int bkt; -/* Check for overlaps */ -kvm_for_each_memslot (tmp, bkt,__kvm_memslots (kvm, as_id)) { -if (tmp->id == id) -continue ; -if (!((new.base_gfn + new.npages <= tmp->base_gfn) || - (new.base_gfn >= tmp->base_gfn + tmp->npages))) -return -EEXIST ; - } +if (kvm_check_memslot_overlap (__kvm_memslots (kvm, as_id), + &new)) +return -EEXIST ; }/* Allocate/free page dirty bitmap as needed */