KVM: Use kvcalloc() to allocate lpage_info arrays and dirty bitmaps
PATCH 1/1 KVM: Use kvcalloc() to allocate lpage_info arrays and dirty bitmaps
Use kvcalloc() instead of __vcalloc() for the lpage_info arrays and memslot dirty bitmaps, and switch their frees from vfree() back to kvfree(). kvcalloc() serves sub-page requests from the slab and falls back to vmalloc for anything larger; kvfree() handles either allocator. The vfree() pairing came from commit a952d608f0be ("KVM:
Use vfree for memory allocated by vcalloc()/__vcalloc()").
Commit 37b2a6510a48 ("KVM: use __vcalloc for very large allocations") moved these sites onto __vcalloc() to escape kvmalloc()'s INT_MAX size cap, since memslot metadata sizes scale with the memslot size and can exceed 2 GiB. Neither site can reach the cap: lpage_info holds one 4 byte entry per hugepage granule (per 2 MiB at level 2, per 1 GiB at level 3), so reaching INT_MAX would take a 1 PiB memslot, while userspace slots are capped at 8 TiB by KVM_MEM_MAX_NR_PAGES and x86's internal slots at 4 GiB by __x86_set_memory_region()'s u32 size. The dirty bitmap is two bitmaps at one bit per page, npages/4 bytes, at most 512 MiB, and internal slots never have one because kvm_set_internal_memslot() rejects any flags.
__vcalloc() makes every allocation at least a page, so a single page memslot consumes 8 KiB of vmalloc for 8 bytes of lpage_info and another 4 KiB for a 16 byte dirty bitmap when dirty logging is enabled. This overhead scales with the number of slots and VMs on a host, adding up to memory pressure when guest address spaces are fragmented into small slots.
The rmap and gfn_write_track arrays keep __vcalloc() and vfree():
the 4K rmap and gfn_write_track are per-page arrays, 8 and 2 bytes per 4 KiB page, which legitimately cross INT_MAX below the 8 TiB slot ceiling; the smaller higher-level rmaps share the 4K rmap's allocation loop; and none of them allocate under the TDP MMU, where the waste above was observed.
--- arch/x86/kvm/x86.c | 6 +++--- virt/kvm/kvm_main.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/x86.c +3 -3
@@ -13488,7 +13488,7 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)memslot_rmap_free (slot);for (i =1 ; i <KVM_NR_PAGE_SIZES ; ++i) { -vfree (slot->arch.lpage_info[i -1 ]); +kvfree (slot->arch.lpage_info[i -1 ]); slot->arch.lpage_info[i -1 ] =NULL ; }
@@ -13544,7 +13544,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm, lpages =__kvm_mmu_slot_lpages (slot, npages, level); - linfo =__vcalloc (lpages,sizeof (*linfo),GFP_KERNEL_ACCOUNT ); + linfo =kvcalloc (lpages,sizeof (*linfo),GFP_KERNEL_ACCOUNT );if (!linfo) goto out_free;
@@ -13580,7 +13580,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,memslot_rmap_free (slot);for (i =1 ; i <KVM_NR_PAGE_SIZES ; ++i) { -vfree (slot->arch.lpage_info[i -1 ]); +kvfree (slot->arch.lpage_info[i -1 ]); slot->arch.lpage_info[i -1 ] =NULL ; }return -ENOMEM ;
virt/kvm/kvm_main.c +2 -2
@@ -927,7 +927,7 @@ static void kvm_destroy_dirty_bitmap(struct kvm_memory_slot *memslot)if (!memslot->dirty_bitmap)return ; -vfree (memslot->dirty_bitmap); +kvfree (memslot->dirty_bitmap); memslot->dirty_bitmap =NULL ; }
@@ -1422,7 +1422,7 @@ static int kvm_alloc_dirty_bitmap(struct kvm_memory_slot *memslot) {unsigned long dirty_bytes =kvm_dirty_bitmap_bytes (memslot); - memslot->dirty_bitmap =__vcalloc (2 , dirty_bytes,GFP_KERNEL_ACCOUNT ); + memslot->dirty_bitmap =kvcalloc (2 , dirty_bytes,GFP_KERNEL_ACCOUNT );if (!memslot->dirty_bitmap)return -ENOMEM ;
base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda
Signature · 1 line
2.47.3
Sean Christopherson
maintainerQuoted 6 lines · Mushahid Hussain
__vcalloc()makes every allocation at least a page, so a single page memslot consumes 8 KiB of vmalloc for 8 bytes oflpage_infoand another 4 KiB for a 16 byte dirty bitmap when dirty logging is enabled. This overhead scales with the number of slots and VMs on a host, adding up to memory pressure when guest address spaces are fragmented into small slots.
If memslots are fragmented that badly, then the rmaps are also going to be extremely wasteful.
Quoted 5 lines
The rmap and
gfn_write_trackarrays keep__vcalloc()andvfree():
the 4K rmap andgfn_write_trackare per-page arrays, 8 and 2 bytes per 4 KiB page, which legitimately crossINT_MAXbelow the 8 TiB slot ceiling; the smaller higher-level rmaps share the 4K rmap's allocation loop; and none of them allocate under the TDP MMU,
Until nested virtualization gets used, and then KVM pays the overhead cost for every memslot.
Rather than flip-flop because of a semi-arbitrary limit that has nothing to do with KVM, I think we should provide dedicated KVM APIs for allocating memslot metadata, and pick a pivot that makes sense for KVM. Or just pivot on INT_MAX to route to kv() vs. v() to play nice with the "not crazy" rule.
Quoted 1 line
where the waste above was observed.