Skip to contentSkip to conversation

KVM: x86/mmu: Repurpose MMU shrinker into page cache shrinker

PATCH Superseded Vipin Sharma 2 participants 9 messages Sep 13, 2024 – Oct 2, 2024

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover KVM: x86/mmu: Repurpose MMU shrinker into page cache shrinker

This series is extracted out from the NUMA aware page table series[1]. MMU shrinker changes were in patches 1 to 9 in the old series.

This series is changing KVM MMU shrinker behaviour by emptying MMU page caches which are used during page fault and MMU load operations. It also incorporates feedback from the NUMA aware page table series[1] regarding MMU shrinker.

KVM MMU shrinker has not been very effective in alleviating pain under memory pressure. It frees up the pages actively being used which results in VM degradation. VM will take fault and bring them again in page tables. More discussions happened at [2]. Overall, consensus was to reprupose it into the code which frees pages from KVM MMU page caches.

Recently [3], there was a discussion to disable shrinker for TDP MMU. Revival of this series is result of that discussion.

There are two major differences from the old series.
1. There is no global accounting of cache pages. It is dynamically calculated in mmu_shrink_count(). This has two effects; i) counting will be inaccurate but code is much simpler, and ii) kvm_lock being used here, this should be fine as mmu_shrink_scan() also holds the lock for its operation.
2. Only empty mmu_shadow_page_cache and mmu_shadowed_info_cache. This version doesn't empty split_shadow_page_cache as it is used only during dirty logging operation and is one per VM unlike other two which are per vCPU. I am not fully convinced that adding it is needed as it will add the cost of adding one more mutex and synchronizing it in shrinker. Also, if a VM is being dirty tracked most likely it will be migrated (memory pressure might be the reason in the first place) so better to not hinder migration effort and let vCPUs free up their caches. If someone convinces me to add split cache as well then I can send a separate patch to add that as well.

[1] https://lore.kernel.org/kvm/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/kvm/[email protected]/#t

v1:
- No global counting of pages in cache. As this number might not remain same between calls of mmu_shrink_count() and mmu_shrink_scan().
- Count cache pages in mmu_shrink_count(). KVM can tolerate inaccuracy here.
- Empty mmu_shadow_page_cache and mmu_shadowed_info_cache only. Don't empty split_shadow_page_cache.

v0: Patches 1-9 from NUMA aware page table series.
https://lore.kernel.org/kvm/[email protected]/

Vipin Sharma (2):
KVM: x86/mmu: Change KVM mmu shrinker to no-op
KVM: x86/mmu: Use MMU shrinker to shrink KVM MMU memory caches

 arch/x86/include/asm/kvm_host.h |   7 +-
 arch/x86/kvm/mmu/mmu.c          | 139 +++++++++++++-------------------
 arch/x86/kvm/mmu/paging_tmpl.h  |  14 ++--
 include/linux/kvm_host.h        |   1 +
 virt/kvm/kvm_main.c             |   8 +-
 5 files changed, 78 insertions(+), 91 deletions(-)

base-commit: 12680d7b8ac4db2eba6237a21a93d2b0e78a52a6

Signature · 1 line
2.46.0.662.g92d0881bb0-goog
#1 lore raw

PATCH 1/2 KVM: x86/mmu: Change KVM mmu shrinker to no-op

Remove global kvm_total_used_mmu_pages and page zapping flow from MMU shrinker. Keep shrinker infrastructure in place to reuse in future commits for freeing KVM page caches. Remove zapped_obsolete_pages list from struct kvm_arch{} and use local list in kvm_zap_obsolete_pages() since MMU shrinker is not using it anymore.

mmu_shrink_scan() is very disruptive to VMs. It picks the first VM in the vm_list, zaps the oldest page which is most likely an upper level SPTEs and most like to be reused. Prior to TDP MMU, this is even more disruptive in nested VMs case, considering L1 SPTEs will be the oldest even though most of the entries are for L2 SPTEs.

As discussed in
https://lore.kernel.org/lkml/[email protected]/ shrinker logic has not be very useful in actually keeping VMs performant and reducing memory usage.

---
 arch/x86/include/asm/kvm_host.h |  1 -
 arch/x86/kvm/mmu/mmu.c          | 92 +++------------------------------
 2 files changed, 8 insertions(+), 85 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
index b0c0bc0ed813..cbfe31bac6cf 100644
@@ -1309,7 +1309,6 @@ struct kvm_arch {
 	bool pre_fault_allowed;
 	struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES];
 	struct list_head active_mmu_pages;
-	struct list_head zapped_obsolete_pages;
 	/*
 	 * A list of kvm_mmu_page structs that, if zapped, could possibly be
 	 * replaced by an NX huge page.  A shadow page is on this list if its
arch/x86/kvm/mmu/mmu.c +8 -84

170 lines not on this page · raw message

Signature · 1 line
2.46.0.662.g92d0881bb0-goog
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] KVM: x86/mmu: Change KVM mmu shrinker to no-op

Quoted 17 lines · Vipin Sharma

Remove global kvm_total_used_mmu_pages and page zapping flow from MMU shrinker. Keep shrinker infrastructure in place to reuse in future commits for freeing KVM page caches. Remove zapped_obsolete_pages list from struct kvm_arch{} and use local list in kvm_zap_obsolete_pages() since MMU shrinker is not using it anymore.

mmu_shrink_scan() is very disruptive to VMs. It picks the first VM in the vm_list, zaps the oldest page which is most likely an upper level SPTEs and most like to be reused. Prior to TDP MMU, this is even more disruptive in nested VMs case, considering L1 SPTEs will be the oldest even though most of the entries are for L2 SPTEs.

As discussed in
https://lore.kernel.org/lkml/[email protected]/ shrinker logic has not be very useful in actually keeping VMs performant and reducing memory usage.

Quoted 4 lines

---

#3 lore raw

PATCH 2/2 KVM: x86/mmu: Use MMU shrinker to shrink KVM MMU memory caches

In reply to Vipin Sharma

Use MMU shrinker to iterate through all the vCPUs of all the VMs and free pages allocated in MMU memory caches. Protect cache allocation in page fault and MMU load path from MMU shrinker by using a per vCPU mutex. In MMU shrinker, move the iterated VM to the end of the VMs list so that the pain of emptying cache spread among other VMs too.

The specific caches to empty are mmu_shadow_page_cache and mmu_shadowed_info_cache as these caches store whole pages. Emptying them will give more impact to shrinker compared to other caches like mmu_pte_list_desc_cache{} and mmu_page_header_cache{}

Holding per vCPU mutex lock ensures that a vCPU doesn't get surprised by finding its cache emptied after filling them up for page table allocations during page fault handling and MMU load operation. Per vCPU mutex also makes sure there is only race between MMU shrinker and all other vCPUs. This should result in very less contention.

---
 arch/x86/include/asm/kvm_host.h |  6 +++
 arch/x86/kvm/mmu/mmu.c          | 69 +++++++++++++++++++++++++++------
 arch/x86/kvm/mmu/paging_tmpl.h  | 14 ++++---
 include/linux/kvm_host.h        |  1 +
 virt/kvm/kvm_main.c             |  8 +++-
 5 files changed, 81 insertions(+), 17 deletions(-)
arch/x86/include/asm/kvm_host.h +6 -0
index cbfe31bac6cf..63eaf03111eb 100644
@@ -811,6 +811,12 @@ struct kvm_vcpu_arch {
 	 */
 	struct kvm_mmu *walk_mmu;
 
+	/*
+	 * Protect cache from getting emptied in MMU shrinker while vCPU might
+	 * use cache for fault handling or loading MMU.  As this is a per vCPU
+	 * lock, only contention might happen when MMU shrinker runs.
+	 */
+	struct mutex mmu_memory_cache_lock;
 	struct kvm_mmu_memory_cache mmu_pte_list_desc_cache;
 	struct kvm_mmu_memory_cache mmu_shadow_page_cache;
 	struct kvm_mmu_memory_cache mmu_shadowed_info_cache;
arch/x86/kvm/mmu/mmu.c +58 -11

143 lines not on this page · raw message

arch/x86/kvm/mmu/paging_tmpl.h +9 -5
index 405bd7ceee2a..084a5c532078 100644
@@ -809,13 +809,14 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 		return RET_PF_EMULATE;
 	}
 
+	mutex_lock(&vcpu->arch.mmu_memory_cache_lock);
 	r = mmu_topup_memory_caches(vcpu, true);
 	if (r)
-		return r;
+		goto out_mmu_memory_cache_unlock;
 
 	r = kvm_faultin_pfn(vcpu, fault, walker.pte_access);
 	if (r != RET_PF_CONTINUE)
-		return r;
+		goto out_mmu_memory_cache_unlock;
 
 	/*
 	 * Do not change pte_access if the pfn is a mmio page, otherwise
@@ -840,16 +841,19 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	write_lock(&vcpu->kvm->mmu_lock);
 
 	if (is_page_fault_stale(vcpu, fault))
-		goto out_unlock;
+		goto out_mmu_unlock;
 
 	r = make_mmu_pages_available(vcpu);
 	if (r)
-		goto out_unlock;
+		goto out_mmu_unlock;
 	r = FNAME(fetch)(vcpu, fault, &walker);
 
-out_unlock:
+out_mmu_unlock:
 	write_unlock(&vcpu->kvm->mmu_lock);
 	kvm_release_pfn_clean(fault->pfn);
+out_mmu_memory_cache_unlock:
+	mutex_unlock(&vcpu->arch.mmu_memory_cache_lock);
+
 	return r;
 }
 
include/linux/kvm_host.h +1 -0
index b23c6d48392f..288e503f14a0 100644
@@ -1446,6 +1446,7 @@ void kvm_flush_remote_tlbs_memslot(struct kvm *kvm,
 int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min);
 int __kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int capacity, int min);
 int kvm_mmu_memory_cache_nr_free_objects(struct kvm_mmu_memory_cache *mc);
+int kvm_mmu_empty_memory_cache(struct kvm_mmu_memory_cache *mc);
 void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);
 void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 #endif
virt/kvm/kvm_main.c +7 -1
index cb2b78e92910..5d89ca218791 100644
@@ -451,15 +451,21 @@ int kvm_mmu_memory_cache_nr_free_objects(struct kvm_mmu_memory_cache *mc)
 	return mc->nobjs;
 }
 
-void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc)
+int kvm_mmu_empty_memory_cache(struct kvm_mmu_memory_cache *mc)
 {
+	int freed = mc->nobjs;
 	while (mc->nobjs) {
 		if (mc->kmem_cache)
 			kmem_cache_free(mc->kmem_cache, mc->objects[--mc->nobjs]);
 		else
 			free_page((unsigned long)mc->objects[--mc->nobjs]);
 	}
+	return freed;
+}
 
+void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc)
+{
+	kvm_mmu_empty_memory_cache(mc);
 	kvfree(mc->objects);
 
 	mc->objects = NULL;
Signature · 1 line
2.46.0.662.g92d0881bb0-goog
#4 lore raw

Re: [PATCH 2/2] KVM: x86/mmu: Use MMU shrinker to shrink KVM MMU memory caches

Quoted 161 lines · Vipin Sharma

Use MMU shrinker to iterate through all the vCPUs of all the VMs and free pages allocated in MMU memory caches. Protect cache allocation in page fault and MMU load path from MMU shrinker by using a per vCPU mutex. In MMU shrinker, move the iterated VM to the end of the VMs list so that the pain of emptying cache spread among other VMs too.

The specific caches to empty are mmu_shadow_page_cache and mmu_shadowed_info_cache as these caches store whole pages. Emptying them will give more impact to shrinker compared to other caches like mmu_pte_list_desc_cache{} and mmu_page_header_cache{}

Holding per vCPU mutex lock ensures that a vCPU doesn't get surprised by finding its cache emptied after filling them up for page table allocations during page fault handling and MMU load operation. Per vCPU mutex also makes sure there is only race between MMU shrinker and all other vCPUs. This should result in very less contention.

---
 arch/x86/include/asm/kvm_host.h |  6 +++
 arch/x86/kvm/mmu/mmu.c          | 69 +++++++++++++++++++++++++++------
 arch/x86/kvm/mmu/paging_tmpl.h  | 14 ++++---
 include/linux/kvm_host.h        |  1 +
 virt/kvm/kvm_main.c             |  8 +++-
 5 files changed, 81 insertions(+), 17 deletions(-)
arch/x86/include/asm/kvm_host.h +6 -0
index cbfe31bac6cf..63eaf03111eb 100644
@@ -811,6 +811,12 @@ struct kvm_vcpu_arch {
 	 */
 	struct kvm_mmu *walk_mmu;
 
+	/*
+	 * Protect cache from getting emptied in MMU shrinker while vCPU might
+	 * use cache for fault handling or loading MMU.  As this is a per vCPU
+	 * lock, only contention might happen when MMU shrinker runs.
+	 */
+	struct mutex mmu_memory_cache_lock;
 	struct kvm_mmu_memory_cache mmu_pte_list_desc_cache;
 	struct kvm_mmu_memory_cache mmu_shadow_page_cache;
 	struct kvm_mmu_memory_cache mmu_shadowed_info_cache;
arch/x86/kvm/mmu/mmu.c +30 -10
index 213e46b55dda..8e2935347615 100644
@@ -4524,29 +4524,33 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	if (r != RET_PF_INVALID)
 		return r;
 
+	mutex_lock(&vcpu->arch.mmu_memory_cache_lock);
 	r = mmu_topup_memory_caches(vcpu, false);
 	if (r)
-		return r;
+		goto out_mmu_memory_cache_unlock;
 
 	r = kvm_faultin_pfn(vcpu, fault, ACC_ALL);
 	if (r != RET_PF_CONTINUE)
-		return r;
+		goto out_mmu_memory_cache_unlock;
 
 	r = RET_PF_RETRY;
 	write_lock(&vcpu->kvm->mmu_lock);
 
 	if (is_page_fault_stale(vcpu, fault))
-		goto out_unlock;
+		goto out_mmu_unlock;
 
 	r = make_mmu_pages_available(vcpu);
 	if (r)
-		goto out_unlock;
+		goto out_mmu_unlock;
 
 	r = direct_map(vcpu, fault);
 
-out_unlock:
+out_mmu_unlock:
 	write_unlock(&vcpu->kvm->mmu_lock);
 	kvm_release_pfn_clean(fault->pfn);
+out_mmu_memory_cache_unlock:
+	mutex_unlock(&vcpu->arch.mmu_memory_cache_lock);
+
 	return r;
 }
 
@@ -4617,25 +4621,28 @@ static int kvm_tdp_mmu_page_fault(struct kvm_vcpu *vcpu,
 	if (r != RET_PF_INVALID)
 		return r;
 
+	mutex_lock(&vcpu->arch.mmu_memory_cache_lock);
 	r = mmu_topup_memory_caches(vcpu, false);
 	if (r)
-		return r;
+		goto out_mmu_memory_cache_unlock;
 
 	r = kvm_faultin_pfn(vcpu, fault, ACC_ALL);
 	if (r != RET_PF_CONTINUE)
-		return r;
+		goto out_mmu_memory_cache_unlock;
 
 	r = RET_PF_RETRY;
 	read_lock(&vcpu->kvm->mmu_lock);
 
 	if (is_page_fault_stale(vcpu, fault))
-		goto out_unlock;
+		goto out_mmu_unlock;
 
 	r = kvm_tdp_mmu_map(vcpu, fault);
 
-out_unlock:
+out_mmu_unlock:
 	read_unlock(&vcpu->kvm->mmu_lock);
 	kvm_release_pfn_clean(fault->pfn);
+out_mmu_memory_cache_unlock:
+	mutex_unlock(&vcpu->arch.mmu_memory_cache_lock);
 	return r;
 }
 #endif
@@ -5691,6 +5698,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)
 {
 	int r;
 
+	mutex_lock(&vcpu->arch.mmu_memory_cache_lock);
 	r = mmu_topup_memory_caches(vcpu, !vcpu->arch.mmu->root_role.direct);
 	if (r)
 		goto out;
@@ -5717,6 +5725,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)
 	 */
 	kvm_x86_call(flush_tlb_current)(vcpu);
 out:
+	mutex_unlock(&vcpu->arch.mmu_memory_cache_lock);
 	return r;
 }
 
@@ -6303,6 +6312,7 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
 	if (!vcpu->arch.mmu_shadow_page_cache.init_value)
 		vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
 
+	mutex_init(&vcpu->arch.mmu_memory_cache_lock);
 	vcpu->arch.mmu = &vcpu->arch.root_mmu;
 	vcpu->arch.walk_mmu = &vcpu->arch.root_mmu;
 
@@ -6997,13 +7007,50 @@ void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
 static unsigned long mmu_shrink_scan(struct shrinker *shrink,
 				     struct shrink_control *sc)
 {
-	return SHRINK_STOP;
+	struct kvm *kvm, *next_kvm, *first_kvm = NULL;
+	unsigned long i, freed = 0;
+	struct kvm_vcpu *vcpu;
+
+	mutex_lock(&kvm_lock);
+	list_for_each_entry_safe(kvm, next_kvm, &vm_list, vm_list) {
+		if (!first_kvm)
+			first_kvm = kvm;
+		else if (first_kvm == kvm)
+			break;
+

arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen) · line 7030

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+		list_move_tail(&kvm->vm_list, &vm_list);
+
+		kvm_for_each_vcpu(i, vcpu, kvm) {
+			if (!mutex_trylock(&vcpu->arch.mmu_memory_cache_lock))
+				continue;
+			freed += kvm_mmu_empty_memory_cache(&vcpu->arch.mmu_shadow_page_cache);
+			freed += kvm_mmu_empty_memory_cache(&vcpu->arch.mmu_shadowed_info_cache);
+			mutex_unlock(&vcpu->arch.mmu_memory_cache_lock);
+			if (freed >= sc->nr_to_scan)
+				goto out;

Looking at the caller in mm/shrinker.c, sc->nr_to_scan will be <= 128 (SHRINK_BATCH), which is only enough for 2 vCPUs. So I think the shrinker will only ever free 2 vCPU caches of each VM (probably the first 2 vCPUs) before reordering the list and moving onto the next VM on the next call.

Does that match the behavior you observe?

Quoted 113 lines
arch/x86/kvm/mmu/mmu.c +18 -1
+		}
+	}
+out:
+	mutex_unlock(&kvm_lock);
+	return freed;
 }
 
 static unsigned long mmu_shrink_count(struct shrinker *shrink,
 				      struct shrink_control *sc)
 {
-	return SHRINK_EMPTY;
+	unsigned long i, count = 0;
+	struct kvm_vcpu *vcpu;
+	struct kvm *kvm;
+
+	mutex_lock(&kvm_lock);
+	list_for_each_entry(kvm, &vm_list, vm_list) {
+		kvm_for_each_vcpu(i, vcpu, kvm) {
+			count += READ_ONCE(vcpu->arch.mmu_shadow_page_cache.nobjs);
+			count += READ_ONCE(vcpu->arch.mmu_shadowed_info_cache.nobjs);
+		}
+	}
+	mutex_unlock(&kvm_lock);
+	return !count ? SHRINK_EMPTY : count;
 }
 
 static struct shrinker *mmu_shrinker;
arch/x86/kvm/mmu/paging_tmpl.h +9 -5
index 405bd7ceee2a..084a5c532078 100644
@@ -809,13 +809,14 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 		return RET_PF_EMULATE;
 	}
 
+	mutex_lock(&vcpu->arch.mmu_memory_cache_lock);
 	r = mmu_topup_memory_caches(vcpu, true);
 	if (r)
-		return r;
+		goto out_mmu_memory_cache_unlock;
 
 	r = kvm_faultin_pfn(vcpu, fault, walker.pte_access);
 	if (r != RET_PF_CONTINUE)
-		return r;
+		goto out_mmu_memory_cache_unlock;
 
 	/*
 	 * Do not change pte_access if the pfn is a mmio page, otherwise
@@ -840,16 +841,19 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	write_lock(&vcpu->kvm->mmu_lock);
 
 	if (is_page_fault_stale(vcpu, fault))
-		goto out_unlock;
+		goto out_mmu_unlock;
 
 	r = make_mmu_pages_available(vcpu);
 	if (r)
-		goto out_unlock;
+		goto out_mmu_unlock;
 	r = FNAME(fetch)(vcpu, fault, &walker);
 
-out_unlock:
+out_mmu_unlock:
 	write_unlock(&vcpu->kvm->mmu_lock);
 	kvm_release_pfn_clean(fault->pfn);
+out_mmu_memory_cache_unlock:
+	mutex_unlock(&vcpu->arch.mmu_memory_cache_lock);
+
 	return r;
 }
 
include/linux/kvm_host.h +1 -0
index b23c6d48392f..288e503f14a0 100644
@@ -1446,6 +1446,7 @@ void kvm_flush_remote_tlbs_memslot(struct kvm *kvm,
 int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min);
 int __kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int capacity, int min);
 int kvm_mmu_memory_cache_nr_free_objects(struct kvm_mmu_memory_cache *mc);
+int kvm_mmu_empty_memory_cache(struct kvm_mmu_memory_cache *mc);
 void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);
 void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 #endif
virt/kvm/kvm_main.c +7 -1
index cb2b78e92910..5d89ca218791 100644
@@ -451,15 +451,21 @@ int kvm_mmu_memory_cache_nr_free_objects(struct kvm_mmu_memory_cache *mc)
 	return mc->nobjs;
 }
 
-void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc)
+int kvm_mmu_empty_memory_cache(struct kvm_mmu_memory_cache *mc)
 {
+	int freed = mc->nobjs;
 	while (mc->nobjs) {
 		if (mc->kmem_cache)
 			kmem_cache_free(mc->kmem_cache, mc->objects[--mc->nobjs]);
 		else
 			free_page((unsigned long)mc->objects[--mc->nobjs]);
 	}
+	return freed;
+}
 
+void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc)
+{
+	kvm_mmu_empty_memory_cache(mc);
 	kvfree(mc->objects);
 
 	mc->objects = NULL;
Signature · 1 line
2.46.0.662.g92d0881bb0-goog
#5 lore raw

Re: [PATCH 2/2] KVM: x86/mmu: Use MMU shrinker to shrink KVM MMU memory caches

Quoted 35 lines · David Matlack
Quoted 16 lines · Vipin Sharma
arch/x86/kvm/mmu/mmu.c +11 -1
@@ -6997,13 +7007,50 @@ void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
 static unsigned long mmu_shrink_scan(struct shrinker *shrink,
                                   struct shrink_control *sc)
 {
-     return SHRINK_STOP;
+     struct kvm *kvm, *next_kvm, *first_kvm = NULL;
+     unsigned long i, freed = 0;
+     struct kvm_vcpu *vcpu;
+
+     mutex_lock(&kvm_lock);
+     list_for_each_entry_safe(kvm, next_kvm, &vm_list, vm_list) {
+             if (!first_kvm)
+                     first_kvm = kvm;
+             else if (first_kvm == kvm)
+                     break;
+

arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen) · line 7030

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+             list_move_tail(&kvm->vm_list, &vm_list);
+
+             kvm_for_each_vcpu(i, vcpu, kvm) {
+                     if (!mutex_trylock(&vcpu->arch.mmu_memory_cache_lock))
+                             continue;
+                     freed += kvm_mmu_empty_memory_cache(&vcpu->arch.mmu_shadow_page_cache);
+                     freed += kvm_mmu_empty_memory_cache(&vcpu->arch.mmu_shadowed_info_cache);
+                     mutex_unlock(&vcpu->arch.mmu_memory_cache_lock);
+                     if (freed >= sc->nr_to_scan)
+                             goto out;

Looking at the caller in mm/shrinker.c, sc->nr_to_scan will be <= 128 (SHRINK_BATCH), which is only enough for 2 vCPUs. So I think the shrinker will only ever free 2 vCPU caches of each VM (probably the first 2 vCPUs) before reordering the list and moving onto the next VM on the next call.

Quoted 2 lines

Does that match the behavior you observe?

Yes, for dropping cache one time on a big VM, I get multiple calls of mmu_shrink_scan() where sc->nr_to_scan is at max 128 in each call.

mmu_memory_cache_lock availability will play a role in selecting the two vCPUs. On a VM where not much faults are happening it will probably be the first two vCPUs.

#6 lore raw
In reply to Vipin Sharma
Quoted 2 lines · Vipin Sharma

This series is extracted out from the NUMA aware page table series[1]. MMU shrinker changes were in patches 1 to 9 in the old series.

I'm curious how you tested this series. Would it be posisble to write a selftest to exercise KVM's shrinker interactions? I don't think it needs to be anything fancy to be useful (e.g. just run a VM, trigger lots of shrinking, and make sure nothing blows up).

There appears to be a debugfs interface which could be used to trigger shrinking from a selftest.

https://docs.kernel.org/admin-guide/mm/shrinker_debugfs.html

#7 lore raw
Quoted 9 lines · David Matlack
Quoted 2 lines · Vipin Sharma

This series is extracted out from the NUMA aware page table series[1]. MMU shrinker changes were in patches 1 to 9 in the old series.

I'm curious how you tested this series. Would it be posisble to write a selftest to exercise KVM's shrinker interactions? I don't think it needs to be anything fancy to be useful (e.g. just run a VM, trigger lots of shrinking, and make sure nothing blows up).

My testing was dropping caches (echo 2 > /proc/sys/vm/drop_caches) in background while running dirty_log_perf_test selftest multiple times. I added printk in shrink_count() and shrink_scan() to make sure pages are being reported and released.

I can write a test which can spawn a thread to drop caches and a VM which touches all of its pages to generate page faults. Only downside is it will not detect if KVM MMU shrinker is being invoked, counting and freeing pages.

Quoted 5 lines

There appears to be a debugfs interface which could be used to trigger shrinking from a selftest.

https://docs.kernel.org/admin-guide/mm/shrinker_debugfs.html

This is interesting and it does what is needed to test KVM MMU shrinker. However, this needs CONFIG_DEBUG_FS and CONFIG_SHRINKER_DEBUG. I think using shrinker_debugfs will be better, selftest can just skip if it cannot find shrinker_debugfs files. One downside is that this test will not run if these configs are not enabled.

Which one do you prefer? I am preferring shrinker_debugfs but concerned about its dependency on those two configs, not sure if it is okay to have this kind of dependency in a selftests.

#8 lore raw
Quoted 35 lines · Vipin Sharma
Quoted 9 lines · David Matlack
Quoted 2 lines · Vipin Sharma

This series is extracted out from the NUMA aware page table series[1]. MMU shrinker changes were in patches 1 to 9 in the old series.

I'm curious how you tested this series. Would it be posisble to write a selftest to exercise KVM's shrinker interactions? I don't think it needs to be anything fancy to be useful (e.g. just run a VM, trigger lots of shrinking, and make sure nothing blows up).

My testing was dropping caches (echo 2 > /proc/sys/vm/drop_caches) in background while running dirty_log_perf_test selftest multiple times. I added printk in shrink_count() and shrink_scan() to make sure pages are being reported and released.

I can write a test which can spawn a thread to drop caches and a VM which touches all of its pages to generate page faults. Only downside is it will not detect if KVM MMU shrinker is being invoked, counting and freeing pages.

Quoted 5 lines

There appears to be a debugfs interface which could be used to trigger shrinking from a selftest.

https://docs.kernel.org/admin-guide/mm/shrinker_debugfs.html

This is interesting and it does what is needed to test KVM MMU shrinker. However, this needs CONFIG_DEBUG_FS and CONFIG_SHRINKER_DEBUG. I think using shrinker_debugfs will be better, selftest can just skip if it cannot find shrinker_debugfs files. One downside is that this test will not run if these configs are not enabled.

Quoted 3 lines

Which one do you prefer? I am preferring shrinker_debugfs but concerned about its dependency on those two configs, not sure if it is okay to have this kind of dependency in a selftests.

Configs are there to be enabled. If shrinker_debugfs is the right way to test this, I don't see any reason to shy away from using it.