Skip to contentSkip to conversation

Lazily allocate memslot rmaps

PATCH Superseded Ben Gardon 4 participants 19 messages May 11, 2021

Subsystems (3)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Ben Gardon

author
#0 lore raw

v4 cover Lazily allocate memslot rmaps

This series enables KVM to save memory when using the TDP MMU by waiting to allocate memslot rmaps until they are needed. To do this, KVM tracks whether or not a shadow root has been allocated. In order to get away with not allocating the rmaps, KVM must also be sure to skip operations which iterate over the rmaps. If the TDP MMU is in use and we have not allocated a shadow root, these operations would essentially be op-ops anyway. Skipping the rmap operations has a secondary benefit of avoiding acquiring the MMU lock in write mode in many cases, substantially reducing MMU lock contention.

This series was tested on an Intel Skylake machine. With the TDP MMU off and on, this introduced no new failures on kvm-unit-tests or KVM selftests.

Changelog:
v2:
Incorporated feedback from Paolo and Sean
Replaced the memslot_assignment_lock with slots_arch_lock, which has a larger critical section.

v3:
Removed shadow_mmu_active as suggested by Sean
Removed everything except adding a return value to kvm_mmu_init_tdp_mmu from patch 1 of v2
Added RCU protection and better memory ordering for installing the memslot rmaps as suggested by Paolo
Reordered most of the patches

v4:
Renamed functions to allocate and free memslots based on feedback from David.
Eliminated the goto in memslot_rmap_alloc, as David suggested. Eliminated kvm_memslots_have_rmaps and updated comments on uses of memslots_have_rmaps. Suggested by Paolo.
Changed the description on patch 7 to one Paolo suggested. Collected Reviewed-by tags from David.
Dropped the patch to add RCU notations to rmap accesses.

Ben Gardon (7):
KVM: x86/mmu: Deduplicate rmap freeing
KVM: x86/mmu: Factor out allocating memslot rmap
KVM: mmu: Refactor memslot copy
KVM: mmu: Add slots_arch_lock for memslot arch fields
KVM: x86/mmu: Add a field to control memslot rmap allocation
KVM: x86/mmu: Skip rmap operations if rmaps not allocated
KVM: x86/mmu: Lazily allocate memslot rmaps

 arch/x86/include/asm/kvm_host.h |   8 ++
 arch/x86/kvm/mmu/mmu.c          | 153 +++++++++++++++++++++-----------
 arch/x86/kvm/mmu/tdp_mmu.c      |   6 +-
 arch/x86/kvm/mmu/tdp_mmu.h      |   4 +-
 arch/x86/kvm/x86.c              | 108 ++++++++++++++++++----
 include/linux/kvm_host.h        |   9 ++
 virt/kvm/kvm_main.c             |  54 ++++++++---
 7 files changed, 255 insertions(+), 87 deletions(-)
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
author
#1 lore raw

PATCH v4 1/7 KVM: x86/mmu: Deduplicate rmap freeing

Small code deduplication. No functional change expected.

---
 arch/x86/kvm/x86.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)
arch/x86/kvm/x86.c +11 -8
index 5bd550eaf683..1e1f4f31e586 100644
@@ -10887,17 +10887,23 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 	kvm_hv_destroy_vm(kvm);
 }
 
-void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
+static void memslot_rmap_free(struct kvm_memory_slot *slot)
 {
 	int i;
 
 	for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
 		kvfree(slot->arch.rmap[i]);
 		slot->arch.rmap[i] = NULL;
+	}
+}
 
-		if (i == 0)
-			continue;
+void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
+{
+	int i;
+
+	memslot_rmap_free(slot);
 
+	for (i = 1; i < KVM_NR_PAGE_SIZES; ++i) {
 		kvfree(slot->arch.lpage_info[i - 1]);
 		slot->arch.lpage_info[i - 1] = NULL;
 	}
@@ -10963,12 +10969,9 @@ static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
 	return 0;
 
 out_free:
-	for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
-		kvfree(slot->arch.rmap[i]);
-		slot->arch.rmap[i] = NULL;
-		if (i == 0)
-			continue;
+	memslot_rmap_free(slot);
 
+	for (i = 1; i < KVM_NR_PAGE_SIZES; ++i) {
 		kvfree(slot->arch.lpage_info[i - 1]);
 		slot->arch.lpage_info[i - 1] = NULL;
 	}
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
author
#2 lore raw

PATCH v4 2/7 KVM: x86/mmu: Factor out allocating memslot rmap

In reply to Ben Gardon

Small refactor to facilitate allocating rmaps for all memslots at once.

No functional change expected.

---
 arch/x86/kvm/x86.c | 39 ++++++++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 9 deletions(-)
arch/x86/kvm/x86.c +30 -9
index 1e1f4f31e586..cc0440b5b35d 100644
@@ -10911,10 +10911,35 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
 	kvm_page_track_free_memslot(slot);
 }
 
+static int memslot_rmap_alloc(struct kvm_memory_slot *slot,
+			      unsigned long npages)
+{
+	int i;
+
+	for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
+		int lpages;
+		int level = i + 1;
+
+		lpages = gfn_to_index(slot->base_gfn + npages - 1,
+				      slot->base_gfn, level) + 1;
+
+		slot->arch.rmap[i] =
+			kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
+				 GFP_KERNEL_ACCOUNT);
+		if (!slot->arch.rmap[i]) {
+			memslot_rmap_free(slot);
+			return -ENOMEM;
+		}
+	}
+
+	return 0;
+}
+
 static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
 				      unsigned long npages)
 {
 	int i;
+	int r;
 
 	/*
 	 * Clear out the previous array pointers for the KVM_MR_MOVE case.  The
@@ -10923,7 +10948,11 @@ static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
 	 */
 	memset(&slot->arch, 0, sizeof(slot->arch));
 
-	for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
+	r = memslot_rmap_alloc(slot, npages);
+	if (r)
+		return r;
+
+	for (i = 1; i < KVM_NR_PAGE_SIZES; ++i) {
 		struct kvm_lpage_info *linfo;
 		unsigned long ugfn;
 		int lpages;
@@ -10932,14 +10961,6 @@ static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
 		lpages = gfn_to_index(slot->base_gfn + npages - 1,
 				      slot->base_gfn, level) + 1;
 
-		slot->arch.rmap[i] =
-			kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
-				 GFP_KERNEL_ACCOUNT);
-		if (!slot->arch.rmap[i])
-			goto out_free;
-		if (i == 0)
-			continue;
-
 		linfo = kvcalloc(lpages, sizeof(*linfo), GFP_KERNEL_ACCOUNT);
 		if (!linfo)
 			goto out_free;
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
#3 lore raw

Re: [PATCH v4 2/7] KVM: x86/mmu: Factor out allocating memslot rmap

Quoted 18 lines · Ben Gardon

Small refactor to facilitate allocating rmaps for all memslots at once.

No functional change expected.

---
 arch/x86/kvm/x86.c | 39 ++++++++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 9 deletions(-)
arch/x86/kvm/x86.c +1 -0
index 1e1f4f31e586..cc0440b5b35d 100644
@@ -10911,10 +10911,35 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
 	kvm_page_track_free_memslot(slot);
 }
 
+static int memslot_rmap_alloc(struct kvm_memory_slot *slot,

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10924

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+			      unsigned long npages)
+{
+	int i;
+
+	for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
+		int lpages;
+		int level = i + 1;
+
+		lpages = gfn_to_index(slot->base_gfn + npages - 1,
+				      slot->base_gfn, level) + 1;

Might as well assign lpages at its declaration, i.e.

		int lpages = gfn_to_index(slot->base_gfn + npages - 1,
					  slot->base_gfn, level) + 1;

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10928

Quoted 4 lines
arch/x86/kvm/x86.c +4 -0
+
+		slot->arch.rmap[i] =
+			kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
+				 GFP_KERNEL_ACCOUNT);

Eh, I don't think avoiding a 3 char overrun is worth splitting across three lines. E.g. this is perfectly readable

		slot->arch.rmap[i] = kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
					      GFP_KERNEL_ACCOUNT);

Alternatively, the rmap size could be captured in a local var, e.g.

	const int sz = sizeof(*slot->arch.rmap[0]);

	...

		slot->arch.rmap[i] = kvcalloc(lpages, sz, GFP_KERNEL_ACCOUNT);
		if (!slot->arch.rmap[i]) {
			memslot_rmap_free(slot);
			return -ENOMEM;
		}

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10931

Quoted 3 lines
arch/x86/kvm/x86.c +3 -0
+		if (!slot->arch.rmap[i]) {
+			memslot_rmap_free(slot);
+			return -ENOMEM;

Reaaaally getting into nitpicks, what do you think about changing this to a goto with the error handling at the bottom? Obviously not necessary by any means, but for me it makes it easier to see that all rmaps are freed on failure. My eyes skipped over that on the first read through. E.g.

		if (!slot_arch.rmap[i])
			goto err;
	}

	return 0;

err:

	memslot_rmap_free(slot);
	return -ENOMEM;

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10942

Quoted 11 lines
arch/x86/kvm/x86.c +7 -0
+		}
+	}
+
+	return 0;
+}
+
 static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
 				      unsigned long npages)
 {
 	int i;
+	int r;

Personal preference, for short declarations like this I like putting 'em on a single line.

Quoted 33 lines
arch/x86/kvm/x86.c +5 -9
 	/*
 	 * Clear out the previous array pointers for the KVM_MR_MOVE case.  The
@@ -10923,7 +10948,11 @@ static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
 	 */
 	memset(&slot->arch, 0, sizeof(slot->arch));
 
-	for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
+	r = memslot_rmap_alloc(slot, npages);
+	if (r)
+		return r;
+
+	for (i = 1; i < KVM_NR_PAGE_SIZES; ++i) {
 		struct kvm_lpage_info *linfo;
 		unsigned long ugfn;
 		int lpages;
@@ -10932,14 +10961,6 @@ static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
 		lpages = gfn_to_index(slot->base_gfn + npages - 1,
 				      slot->base_gfn, level) + 1;
 
-		slot->arch.rmap[i] =
-			kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
-				 GFP_KERNEL_ACCOUNT);
-		if (!slot->arch.rmap[i])
-			goto out_free;
-		if (i == 0)
-			continue;
-
 		linfo = kvcalloc(lpages, sizeof(*linfo), GFP_KERNEL_ACCOUNT);
 		if (!linfo)
 			goto out_free;
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
author
#4 lore raw

Re: [PATCH v4 2/7] KVM: x86/mmu: Factor out allocating memslot rmap

Quoted 33 lines · Sean Christopherson
Quoted 18 lines · Ben Gardon

Small refactor to facilitate allocating rmaps for all memslots at once.

No functional change expected.

---
 arch/x86/kvm/x86.c | 39 ++++++++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 9 deletions(-)
arch/x86/kvm/x86.c +1 -0
index 1e1f4f31e586..cc0440b5b35d 100644
@@ -10911,10 +10911,35 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
      kvm_page_track_free_memslot(slot);
 }
 
+static int memslot_rmap_alloc(struct kvm_memory_slot *slot,

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10924

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+                           unsigned long npages)
+{
+     int i;
+
+     for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
+             int lpages;
+             int level = i + 1;
+
+             lpages = gfn_to_index(slot->base_gfn + npages - 1,
+                                   slot->base_gfn, level) + 1;

Might as well assign lpages at its declaration, i.e.

Quoted 2 lines
                int lpages = gfn_to_index(slot->base_gfn + npages - 1,
                                          slot->base_gfn, level) + 1;

I'll do this if I end up sending out a v5.

Quoted 11 lines

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10928

Quoted 4 lines
arch/x86/kvm/x86.c +4 -0
+
+             slot->arch.rmap[i] =
+                     kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
+                              GFP_KERNEL_ACCOUNT);

Eh, I don't think avoiding a 3 char overrun is worth splitting across three lines. E.g. this is perfectly readable

                slot->arch.rmap[i] = kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
                                              GFP_KERNEL_ACCOUNT);
Quoted 7 lines

Alternatively, the rmap size could be captured in a local var, e.g.

        const int sz = sizeof(*slot->arch.rmap[0]);

        ...

                slot->arch.rmap[i] = kvcalloc(lpages, sz, GFP_KERNEL_ACCOUNT);

I like this suggestion. Much nicer. Will incorporate if I send a v5.

Quoted 20 lines
                if (!slot->arch.rmap[i]) {
                        memslot_rmap_free(slot);
                        return -ENOMEM;
                }

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10931

Quoted 3 lines
arch/x86/kvm/x86.c +3 -0
+             if (!slot->arch.rmap[i]) {
+                     memslot_rmap_free(slot);
+                     return -ENOMEM;

Reaaaally getting into nitpicks, what do you think about changing this to a goto with the error handling at the bottom? Obviously not necessary by any means, but for me it makes it easier to see that all rmaps are freed on failure. My eyes skipped over that on the first read through. E.g.

                if (!slot_arch.rmap[i])
                        goto err;
        }

        return 0;
Quoted 4 lines

err:

        memslot_rmap_free(slot);
        return -ENOMEM;

Lol, I had a goto in v3, but David Hildenbrand suggested removing it and putting the free in the loop. I think I like it more this way too.

Quoted 49 lines

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10942

Quoted 12 lines
arch/x86/kvm/x86.c +7 -0
+             }
+     }
+
+     return 0;
+}
+
 static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
                                    unsigned long npages)
 {
      int i;
+     int r;

Personal preference, for short declarations like this I like putting 'em on a single line.

Quoted 33 lines
arch/x86/kvm/x86.c +5 -10
      /*
       * Clear out the previous array pointers for the KVM_MR_MOVE case.  The
@@ -10923,7 +10948,11 @@ static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
       */
      memset(&slot->arch, 0, sizeof(slot->arch));
 
-     for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
+     r = memslot_rmap_alloc(slot, npages);
+     if (r)
+             return r;
+
+     for (i = 1; i < KVM_NR_PAGE_SIZES; ++i) {
              struct kvm_lpage_info *linfo;
              unsigned long ugfn;
              int lpages;
@@ -10932,14 +10961,6 @@ static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
              lpages = gfn_to_index(slot->base_gfn + npages - 1,
                                    slot->base_gfn, level) + 1;
 
-             slot->arch.rmap[i] =
-                     kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
-                              GFP_KERNEL_ACCOUNT);
-             if (!slot->arch.rmap[i])
-                     goto out_free;
-             if (i == 0)
-                     continue;
-
              linfo = kvcalloc(lpages, sizeof(*linfo), GFP_KERNEL_ACCOUNT);
              if (!linfo)
                      goto out_free;
--

2.31.1.607.g51e8a6a459-goog

#5 lore raw

Re: [PATCH v4 2/7] KVM: x86/mmu: Factor out allocating memslot rmap

Quoted 80 lines · Ben Gardon
Quoted 33 lines · Sean Christopherson
Quoted 18 lines · Ben Gardon

Small refactor to facilitate allocating rmaps for all memslots at once.

No functional change expected.

---
  arch/x86/kvm/x86.c | 39 ++++++++++++++++++++++++++++++---------
  1 file changed, 30 insertions(+), 9 deletions(-)
arch/x86/kvm/x86.c +1 -0
index 1e1f4f31e586..cc0440b5b35d 100644
@@ -10911,10 +10911,35 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
       kvm_page_track_free_memslot(slot);
  }
 
+static int memslot_rmap_alloc(struct kvm_memory_slot *slot,

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10924

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+                           unsigned long npages)
+{
+     int i;
+
+     for (i = 0; i < KVM_NR_PAGE_SIZES; ++i) {
+             int lpages;
+             int level = i + 1;
+
+             lpages = gfn_to_index(slot->base_gfn + npages - 1,
+                                   slot->base_gfn, level) + 1;

Might as well assign lpages at its declaration, i.e.

Quoted 2 lines
                 int lpages = gfn_to_index(slot->base_gfn + npages - 1,
                                           slot->base_gfn, level) + 1;

I'll do this if I end up sending out a v5.

Quoted 11 lines

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10928

Quoted 4 lines
arch/x86/kvm/x86.c +4 -0
+
+             slot->arch.rmap[i] =
+                     kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
+                              GFP_KERNEL_ACCOUNT);

Eh, I don't think avoiding a 3 char overrun is worth splitting across three lines. E.g. this is perfectly readable

                 slot->arch.rmap[i] = kvcalloc(lpages, sizeof(*slot->arch.rmap[i]),
                                               GFP_KERNEL_ACCOUNT);
Quoted 7 lines

Alternatively, the rmap size could be captured in a local var, e.g.

         const int sz = sizeof(*slot->arch.rmap[0]);

         ...

                 slot->arch.rmap[i] = kvcalloc(lpages, sz, GFP_KERNEL_ACCOUNT);

I like this suggestion. Much nicer. Will incorporate if I send a v5.

Quoted 20 lines
                 if (!slot->arch.rmap[i]) {
                         memslot_rmap_free(slot);
                         return -ENOMEM;
                 }

arch/x86/kvm/x86.c void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) · line 10931

Quoted 3 lines
arch/x86/kvm/x86.c +3 -0
+             if (!slot->arch.rmap[i]) {
+                     memslot_rmap_free(slot);
+                     return -ENOMEM;

Reaaaally getting into nitpicks, what do you think about changing this to a goto with the error handling at the bottom? Obviously not necessary by any means, but for me it makes it easier to see that all rmaps are freed on failure. My eyes skipped over that on the first read through. E.g.

                 if (!slot_arch.rmap[i])
                         goto err;
         }

         return 0;
Quoted 7 lines
Quoted 4 lines

err:

         memslot_rmap_free(slot);
         return -ENOMEM;

Lol, I had a goto in v3, but David Hildenbrand suggested removing it and putting the free in the loop. I think I like it more this way too.

No strong opinion, I tend to stick to Documentation/process/coding-style.rst which states

"The goto statement comes in handy when a function exits from multiple locations and some common work such as cleanup has to be done."

As we only have a single error exit and no complicated locking, at least for me the "goto" makes it unnecessary hard to read.

Signature · 3 lines
Thanks,

David / dhildenb
author
#6 lore raw

PATCH v4 3/7 KVM: mmu: Refactor memslot copy

In reply to Ben Gardon

Factor out copying kvm_memslots from allocating the memory for new ones in preparation for adding a new lock to protect the arch-specific fields of the memslots.

No functional change intended.

---
 virt/kvm/kvm_main.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
virt/kvm/kvm_main.c +16 -7
index 6b4feb92dc79..9e106742b388 100644
@@ -1306,6 +1306,18 @@ static struct kvm_memslots *install_new_memslots(struct kvm *kvm,
 	return old_memslots;
 }
 
+static size_t kvm_memslots_size(int slots)
+{
+	return sizeof(struct kvm_memslots) +
+	       (sizeof(struct kvm_memory_slot) * slots);
+}
+
+static void kvm_copy_memslots(struct kvm_memslots *from,
+			      struct kvm_memslots *to)
+{
+	memcpy(to, from, kvm_memslots_size(from->used_slots));
+}
+
 /*
  * Note, at a minimum, the current number of used slots must be allocated, even
  * when deleting a memslot, as we need a complete duplicate of the memslots for
@@ -1315,19 +1327,16 @@ static struct kvm_memslots *kvm_dup_memslots(struct kvm_memslots *old,
 					     enum kvm_mr_change change)
 {
 	struct kvm_memslots *slots;
-	size_t old_size, new_size;
-
-	old_size = sizeof(struct kvm_memslots) +
-		   (sizeof(struct kvm_memory_slot) * old->used_slots);
+	size_t new_size;
 
 	if (change == KVM_MR_CREATE)
-		new_size = old_size + sizeof(struct kvm_memory_slot);
+		new_size = kvm_memslots_size(old->used_slots + 1);
 	else
-		new_size = old_size;
+		new_size = kvm_memslots_size(old->used_slots);
 
 	slots = kvzalloc(new_size, GFP_KERNEL_ACCOUNT);
 	if (likely(slots))
-		memcpy(slots, old, old_size);
+		kvm_copy_memslots(old, slots);
 
 	return slots;
 }
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
#7 lore raw

Re: [PATCH v4 3/7] KVM: mmu: Refactor memslot copy

Quoted 17 lines · Ben Gardon

Factor out copying kvm_memslots from allocating the memory for new ones in preparation for adding a new lock to protect the arch-specific fields of the memslots.

No functional change intended.

---
 virt/kvm/kvm_main.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
virt/kvm/kvm_main.c +0 -0
index 6b4feb92dc79..9e106742b388 100644

virt/kvm/kvm_main.c static struct kvm_memslots *install_new_memslots(struct kvm *kvm, · line 1309

Quoted 4 lines
virt/kvm/kvm_main.c +1 -0
@@ -1306,6 +1306,18 @@ static struct kvm_memslots *install_new_memslots(struct kvm *kvm,
 	return old_memslots;
 }
 
+static size_t kvm_memslots_size(int slots)

Can we call this kvm_calc_memslots_size()? This doesn't actually return the true size of a given memslots instance since the allocated size may be greater than the size computed by looking at used_slots.

Quoted 41 lines
virt/kvm/kvm_main.c +15 -7
+{
+	return sizeof(struct kvm_memslots) +
+	       (sizeof(struct kvm_memory_slot) * slots);
+}
+
+static void kvm_copy_memslots(struct kvm_memslots *from,
+			      struct kvm_memslots *to)
+{
+	memcpy(to, from, kvm_memslots_size(from->used_slots));
+}
+
 /*
  * Note, at a minimum, the current number of used slots must be allocated, even
  * when deleting a memslot, as we need a complete duplicate of the memslots for
@@ -1315,19 +1327,16 @@ static struct kvm_memslots *kvm_dup_memslots(struct kvm_memslots *old,
 					     enum kvm_mr_change change)
 {
 	struct kvm_memslots *slots;
-	size_t old_size, new_size;
-
-	old_size = sizeof(struct kvm_memslots) +
-		   (sizeof(struct kvm_memory_slot) * old->used_slots);
+	size_t new_size;
 
 	if (change == KVM_MR_CREATE)
-		new_size = old_size + sizeof(struct kvm_memory_slot);
+		new_size = kvm_memslots_size(old->used_slots + 1);
 	else
-		new_size = old_size;
+		new_size = kvm_memslots_size(old->used_slots);
 
 	slots = kvzalloc(new_size, GFP_KERNEL_ACCOUNT);
 	if (likely(slots))
-		memcpy(slots, old, old_size);
+		kvm_copy_memslots(old, slots);
 
 	return slots;
 }
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
author
#8 lore raw

PATCH v4 4/7 KVM: mmu: Add slots_arch_lock for memslot arch fields

In reply to Ben Gardon

Add a new lock to protect the arch-specific fields of memslots if they need to be modified in a kvm->srcu read critical section. A future commit will use this lock to lazily allocate memslot rmaps for x86.

---
 include/linux/kvm_host.h |  9 +++++++++
 virt/kvm/kvm_main.c      | 31 ++++++++++++++++++++++++++-----
 2 files changed, 35 insertions(+), 5 deletions(-)
include/linux/kvm_host.h +9 -0
index 8895b95b6a22..2d5e797fbb08 100644
@@ -472,6 +472,15 @@ struct kvm {
 #endif /* KVM_HAVE_MMU_RWLOCK */
 
 	struct mutex slots_lock;
+
+	/*
+	 * Protects the arch-specific fields of struct kvm_memory_slots in
+	 * use by the VM. To be used under the slots_lock (above) or in a
+	 * kvm->srcu read cirtical section where acquiring the slots_lock
+	 * would lead to deadlock with the synchronize_srcu in
+	 * install_new_memslots.
+	 */
+	struct mutex slots_arch_lock;
 	struct mm_struct *mm; /* userspace tied to this vm */
 	struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];
 	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
virt/kvm/kvm_main.c +26 -5
index 9e106742b388..5c40d83754b1 100644
@@ -908,6 +908,7 @@ static struct kvm *kvm_create_vm(unsigned long type)
 	mutex_init(&kvm->lock);
 	mutex_init(&kvm->irq_lock);
 	mutex_init(&kvm->slots_lock);
+	mutex_init(&kvm->slots_arch_lock);
 	INIT_LIST_HEAD(&kvm->devices);
 
 	BUILD_BUG_ON(KVM_MEM_SLOTS_NUM > SHRT_MAX);
@@ -1280,6 +1281,10 @@ static struct kvm_memslots *install_new_memslots(struct kvm *kvm,
 	slots->generation = gen | KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS;
 
 	rcu_assign_pointer(kvm->memslots[as_id], slots);
+
+	/* Acquired in kvm_set_memslot. */
+	mutex_unlock(&kvm->slots_arch_lock);
+
 	synchronize_srcu_expedited(&kvm->srcu);
 
 	/*
@@ -1351,6 +1356,9 @@ static int kvm_set_memslot(struct kvm *kvm,
 	struct kvm_memslots *slots;
 	int r;
 
+	/* Released in install_new_memslots. */
+	mutex_lock(&kvm->slots_arch_lock);
+
 	slots = kvm_dup_memslots(__kvm_memslots(kvm, as_id), change);
 	if (!slots)
 		return -ENOMEM;
@@ -1364,10 +1372,9 @@ static int kvm_set_memslot(struct kvm *kvm,
 		slot->flags |= KVM_MEMSLOT_INVALID;
 
 		/*
-		 * We can re-use the old memslots, the only difference from the
-		 * newly installed memslots is the invalid flag, which will get
-		 * dropped by update_memslots anyway.  We'll also revert to the
-		 * old memslots if preparing the new memory region fails.
+		 * We can re-use the memory from the old memslots.
+		 * It will be overwritten with a copy of the new memslots
+		 * after reacquiring the slots_arch_lock below.
 		 */
 		slots = install_new_memslots(kvm, as_id, slots);
 
@@ -1379,6 +1386,17 @@ static int kvm_set_memslot(struct kvm *kvm,
 		 *	- kvm_is_visible_gfn (mmu_check_root)
 		 */
 		kvm_arch_flush_shadow_memslot(kvm, slot);
+
+		/* Released in install_new_memslots. */
+		mutex_lock(&kvm->slots_arch_lock);
+
+		/*
+		 * The arch-specific fields of the memslots could have changed
+		 * between releasing the slots_arch_lock in
+		 * install_new_memslots and here, so get a fresh copy of the
+		 * slots.
+		 */
+		kvm_copy_memslots(__kvm_memslots(kvm, as_id), slots);
 	}
 
 	r = kvm_arch_prepare_memory_region(kvm, new, mem, change);
@@ -1394,8 +1412,11 @@ static int kvm_set_memslot(struct kvm *kvm,
 	return 0;
 
 out_slots:
-	if (change == KVM_MR_DELETE || change == KVM_MR_MOVE)
+	if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) {
+		slot = id_to_memslot(slots, old->id);
+		slot->flags &= ~KVM_MEMSLOT_INVALID;
 		slots = install_new_memslots(kvm, as_id, slots);
+	}
 	kvfree(slots);
 	return r;
 }
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
#9 lore raw

Re: [PATCH v4 4/7] KVM: mmu: Add slots_arch_lock for memslot arch fields

Quoted 15 lines · Ben Gardon

Add a new lock to protect the arch-specific fields of memslots if they need to be modified in a kvm->srcu read critical section. A future commit will use this lock to lazily allocate memslot rmaps for x86.

---
 include/linux/kvm_host.h |  9 +++++++++
 virt/kvm/kvm_main.c      | 31 ++++++++++++++++++++++++++-----
 2 files changed, 35 insertions(+), 5 deletions(-)
include/linux/kvm_host.h +0 -0
index 8895b95b6a22..2d5e797fbb08 100644

include/linux/kvm_host.h struct kvm { · line 479

Quoted 8 lines
include/linux/kvm_host.h +5 -0
@@ -472,6 +472,15 @@ struct kvm {
 #endif /* KVM_HAVE_MMU_RWLOCK */
 
 	struct mutex slots_lock;
+
+	/*
+	 * Protects the arch-specific fields of struct kvm_memory_slots in
+	 * use by the VM. To be used under the slots_lock (above) or in a
+	 * kvm->srcu read cirtical section where acquiring the slots_lock
                          ^^^^^^^^
			  critical
Quoted 7 lines
include/linux/kvm_host.h +4 -0
+	 * would lead to deadlock with the synchronize_srcu in
+	 * install_new_memslots.
+	 */
+	struct mutex slots_arch_lock;
 	struct mm_struct *mm; /* userspace tied to this vm */
 	struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];
 	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
#10 lore raw

Re: [PATCH v4 4/7] KVM: mmu: Add slots_arch_lock for memslot arch fields

In reply to Ben Gardon
Quoted 54 lines · Ben Gardon

Add a new lock to protect the arch-specific fields of memslots if they need to be modified in a kvm->srcu read critical section. A future commit will use this lock to lazily allocate memslot rmaps for x86.

---
 include/linux/kvm_host.h |  9 +++++++++
 virt/kvm/kvm_main.c      | 31 ++++++++++++++++++++++++++-----
 2 files changed, 35 insertions(+), 5 deletions(-)
include/linux/kvm_host.h +9 -0
index 8895b95b6a22..2d5e797fbb08 100644
@@ -472,6 +472,15 @@ struct kvm {
 #endif /* KVM_HAVE_MMU_RWLOCK */
 
 	struct mutex slots_lock;
+
+	/*
+	 * Protects the arch-specific fields of struct kvm_memory_slots in
+	 * use by the VM. To be used under the slots_lock (above) or in a
+	 * kvm->srcu read cirtical section where acquiring the slots_lock
+	 * would lead to deadlock with the synchronize_srcu in
+	 * install_new_memslots.
+	 */
+	struct mutex slots_arch_lock;
 	struct mm_struct *mm; /* userspace tied to this vm */
 	struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];
 	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
virt/kvm/kvm_main.c +5 -0
index 9e106742b388..5c40d83754b1 100644
@@ -908,6 +908,7 @@ static struct kvm *kvm_create_vm(unsigned long type)
 	mutex_init(&kvm->lock);
 	mutex_init(&kvm->irq_lock);
 	mutex_init(&kvm->slots_lock);
+	mutex_init(&kvm->slots_arch_lock);
 	INIT_LIST_HEAD(&kvm->devices);
 
 	BUILD_BUG_ON(KVM_MEM_SLOTS_NUM > SHRT_MAX);
@@ -1280,6 +1281,10 @@ static struct kvm_memslots *install_new_memslots(struct kvm *kvm,
 	slots->generation = gen | KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS;
 
 	rcu_assign_pointer(kvm->memslots[as_id], slots);
+
+	/* Acquired in kvm_set_memslot. */
+	mutex_unlock(&kvm->slots_arch_lock);
+
 	synchronize_srcu_expedited(&kvm->srcu);
 
 	/*

virt/kvm/kvm_main.c static int kvm_set_memslot(struct kvm *kvm, · line 1359

Quoted 4 lines
virt/kvm/kvm_main.c +1 -0
@@ -1351,6 +1356,9 @@ static int kvm_set_memslot(struct kvm *kvm,
 	struct kvm_memslots *slots;
 	int r;
 
+	/* Released in install_new_memslots. */

This needs a much more comprehensive comment, either here or above the declaration of slots_arch_lock. I can't find anything that explicitly states the the lock must be held from the time the previous memslots are duplicated/copied until the new memslots are set. Without that information, the rules/expecations are not clear.

virt/kvm/kvm_main.c static int kvm_set_memslot(struct kvm *kvm, · line 1364

Quoted 5 lines
virt/kvm/kvm_main.c +2 -0
+	mutex_lock(&kvm->slots_arch_lock);
+
 	slots = kvm_dup_memslots(__kvm_memslots(kvm, as_id), change);
 	if (!slots)
 		return -ENOMEM;

Fails to drop slots_arch_lock.

Quoted 19 lines
virt/kvm/kvm_main.c +4 -4
@@ -1364,10 +1372,9 @@ static int kvm_set_memslot(struct kvm *kvm,
 		slot->flags |= KVM_MEMSLOT_INVALID;
 
 		/*
-		 * We can re-use the old memslots, the only difference from the
-		 * newly installed memslots is the invalid flag, which will get
-		 * dropped by update_memslots anyway.  We'll also revert to the
-		 * old memslots if preparing the new memory region fails.
+		 * We can re-use the memory from the old memslots.
+		 * It will be overwritten with a copy of the new memslots
+		 * after reacquiring the slots_arch_lock below.
 		 */
 		slots = install_new_memslots(kvm, as_id, slots);
 
@@ -1379,6 +1386,17 @@ static int kvm_set_memslot(struct kvm *kvm,
 		 *	- kvm_is_visible_gfn (mmu_check_root)
 		 */
 		kvm_arch_flush_shadow_memslot(kvm, slot);
+

virt/kvm/kvm_main.c static int kvm_set_memslot(struct kvm *kvm, · line 1399

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+		/* Released in install_new_memslots. */
+		mutex_lock(&kvm->slots_arch_lock);
+
+		/*
+		 * The arch-specific fields of the memslots could have changed
+		 * between releasing the slots_arch_lock in
+		 * install_new_memslots and here, so get a fresh copy of the
+		 * slots.
+		 */
+		kvm_copy_memslots(__kvm_memslots(kvm, as_id), slots);

Ow. This is feedback for a previous patch, but kvm_copy_memslots() absolutely needs to swap the order of params to match memcpy(), i.e. @to is first, @from is second.

virt/kvm/kvm_main.c static int kvm_set_memslot(struct kvm *kvm, · line 1419

Quoted 13 lines
virt/kvm/kvm_main.c +4 -1
 	}
 
 	r = kvm_arch_prepare_memory_region(kvm, new, mem, change);
@@ -1394,8 +1412,11 @@ static int kvm_set_memslot(struct kvm *kvm,
 	return 0;
 
 out_slots:
-	if (change == KVM_MR_DELETE || change == KVM_MR_MOVE)
+	if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) {
+		slot = id_to_memslot(slots, old->id);
+		slot->flags &= ~KVM_MEMSLOT_INVALID;
 		slots = install_new_memslots(kvm, as_id, slots);
+	}

Fails to drop slots_arch_lock if kvm_arch_prepare_memory_region() fails for anything other than DELETE and MOVE.

I really, really don't like dropping the lock in install_new_memslots(). Unlocking bugs aside, IMO it makes it quite difficult to understand exactly what slots_arch_lock protects. Unfortunately I'm just whining at this point since I don't have a better idea :-(

Quoted 6 lines
virt/kvm/kvm_main.c +0 -0
 	kvfree(slots);
 	return r;
 }
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
maintainer
#11 lore raw

Re: [PATCH v4 4/7] KVM: mmu: Add slots_arch_lock for memslot arch fields

Quoted 7 lines · Sean Christopherson

Review

Quoted 1 line
(quoted diff) +1 -0
+	/* Released in install_new_memslots. */

This needs a much more comprehensive comment, either here or above the declaration of slots_arch_lock. I can't find anything that explicitly states the the lock must be held from the time the previous memslots are duplicated/copied until the new memslots are set. Without that information, the rules/expecations are not clear.

Indeed, this needs basically a description of the races that can happen, as you explained them in the v1 review.

Quoted 2 lines

Unfortunately I'm just whining at this point since I don't have a better idea

Yeah, the synchronize_srcu call in install_new_memslots throws a wrench in most alternatives.

Paolo

author
#12 lore raw

PATCH v4 5/7 KVM: x86/mmu: Add a field to control memslot rmap allocation

In reply to Ben Gardon

Add a field to control whether new memslots should have rmaps allocated for them. As of this change, it's not safe to skip allocating rmaps, so the field is always set to allocate rmaps. Future changes will make it safe to operate without rmaps, using the TDP MMU. Then further changes will allow the rmaps to be allocated lazily when needed for nested oprtation.

No functional change expected.

---
 arch/x86/include/asm/kvm_host.h |  6 ++++++
 arch/x86/kvm/mmu/mmu.c          |  2 ++
 arch/x86/kvm/x86.c              | 13 ++++++++-----
 3 files changed, 16 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +6 -0
index 55efbacfc244..fc75ed49bfee 100644
@@ -1124,6 +1124,12 @@ struct kvm_arch {
 	 */
 	spinlock_t tdp_mmu_pages_lock;
 #endif /* CONFIG_X86_64 */
+
+	/*
+	 * If set, rmaps have been allocated for all memslots and should be
+	 * allocated for any newly created or modified memslots.
+	 */
+	bool memslots_have_rmaps;
 };
 
 struct kvm_vm_stat {
arch/x86/kvm/mmu/mmu.c +2 -0
index 0144c40d09c7..f059f2e8c6fe 100644
@@ -5469,6 +5469,8 @@ void kvm_mmu_init_vm(struct kvm *kvm)
 
 	kvm_mmu_init_tdp_mmu(kvm);
 
+	kvm->arch.memslots_have_rmaps = true;
+
 	node->track_write = kvm_mmu_pte_write;
 	node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
 	kvm_page_track_register_notifier(kvm, node);
arch/x86/kvm/x86.c +8 -5
index cc0440b5b35d..03b6bcff2a53 100644
@@ -10935,7 +10935,8 @@ static int memslot_rmap_alloc(struct kvm_memory_slot *slot,
 	return 0;
 }
 
-static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
+static int kvm_alloc_memslot_metadata(struct kvm *kvm,
+				      struct kvm_memory_slot *slot,
 				      unsigned long npages)
 {
 	int i;
@@ -10948,9 +10949,11 @@ static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
 	 */
 	memset(&slot->arch, 0, sizeof(slot->arch));
 
-	r = memslot_rmap_alloc(slot, npages);
-	if (r)
-		return r;
+	if (kvm->arch.memslots_have_rmaps) {
+		r = memslot_rmap_alloc(slot, npages);
+		if (r)
+			return r;
+	}
 
 	for (i = 1; i < KVM_NR_PAGE_SIZES; ++i) {
 		struct kvm_lpage_info *linfo;
@@ -11021,7 +11024,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
 				enum kvm_mr_change change)
 {
 	if (change == KVM_MR_CREATE || change == KVM_MR_MOVE)
-		return kvm_alloc_memslot_metadata(memslot,
+		return kvm_alloc_memslot_metadata(kvm, memslot,
 						  mem->memory_size >> PAGE_SHIFT);
 	return 0;
 }
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
author
#13 lore raw

PATCH v4 6/7 KVM: x86/mmu: Skip rmap operations if rmaps not allocated

In reply to Ben Gardon

If only the TDP MMU is being used to manage the memory mappings for a VM, then many rmap operations can be skipped as they are guaranteed to be no-ops. This saves some time which would be spent on the rmap operation. It also avoids acquiring the MMU lock in write mode for many operations.

This makes it safe to run the VM without rmaps allocated, when only using the TDP MMU and sets the stage for waiting to allocate the rmaps until they're needed.

---
 arch/x86/kvm/mmu/mmu.c | 128 +++++++++++++++++++++++++----------------
 1 file changed, 77 insertions(+), 51 deletions(-)
arch/x86/kvm/mmu/mmu.c +77 -51
index f059f2e8c6fe..b0bdb924d519 100644
@@ -1189,6 +1189,10 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
 	if (is_tdp_mmu_enabled(kvm))
 		kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,
 				slot->base_gfn + gfn_offset, mask, true);
+
+	if (!kvm->arch.memslots_have_rmaps)
+		return;
+
 	while (mask) {
 		rmap_head = __gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask),
 					  PG_LEVEL_4K, slot);
@@ -1218,6 +1222,10 @@ static void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm,
 	if (is_tdp_mmu_enabled(kvm))
 		kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,
 				slot->base_gfn + gfn_offset, mask, false);
+
+	if (!kvm->arch.memslots_have_rmaps)
+		return;
+
 	while (mask) {
 		rmap_head = __gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask),
 					  PG_LEVEL_4K, slot);
@@ -1260,9 +1268,12 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
 	int i;
 	bool write_protected = false;
 
-	for (i = PG_LEVEL_4K; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
-		rmap_head = __gfn_to_rmap(gfn, i, slot);
-		write_protected |= __rmap_write_protect(kvm, rmap_head, true);
+	if (kvm->arch.memslots_have_rmaps) {
+		for (i = PG_LEVEL_4K; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
+			rmap_head = __gfn_to_rmap(gfn, i, slot);
+			write_protected |= __rmap_write_protect(kvm, rmap_head,
+								true);
+		}
 	}
 
 	if (is_tdp_mmu_enabled(kvm))
@@ -1433,9 +1444,10 @@ static __always_inline bool kvm_handle_gfn_range(struct kvm *kvm,
 
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
-	bool flush;
+	bool flush = false;
 
-	flush = kvm_handle_gfn_range(kvm, range, kvm_unmap_rmapp);
+	if (kvm->arch.memslots_have_rmaps)
+		flush = kvm_handle_gfn_range(kvm, range, kvm_unmap_rmapp);
 
 	if (is_tdp_mmu_enabled(kvm))
 		flush |= kvm_tdp_mmu_unmap_gfn_range(kvm, range, flush);
@@ -1445,9 +1457,10 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
-	bool flush;
+	bool flush = false;
 
-	flush = kvm_handle_gfn_range(kvm, range, kvm_set_pte_rmapp);
+	if (kvm->arch.memslots_have_rmaps)
+		flush = kvm_handle_gfn_range(kvm, range, kvm_set_pte_rmapp);
 
 	if (is_tdp_mmu_enabled(kvm))
 		flush |= kvm_tdp_mmu_set_spte_gfn(kvm, range);
@@ -1500,9 +1513,10 @@ static void rmap_recycle(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)
 
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
-	bool young;
+	bool young = false;
 
-	young = kvm_handle_gfn_range(kvm, range, kvm_age_rmapp);
+	if (kvm->arch.memslots_have_rmaps)
+		young = kvm_handle_gfn_range(kvm, range, kvm_age_rmapp);
 
 	if (is_tdp_mmu_enabled(kvm))
 		young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
@@ -1512,9 +1526,10 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
-	bool young;
+	bool young = false;
 
-	young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmapp);
+	if (kvm->arch.memslots_have_rmaps)
+		young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmapp);
 
 	if (is_tdp_mmu_enabled(kvm))
 		young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
@@ -5440,7 +5455,8 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 	 */
 	kvm_reload_remote_mmus(kvm);
 
-	kvm_zap_obsolete_pages(kvm);
+	if (kvm->arch.memslots_have_rmaps)
+		kvm_zap_obsolete_pages(kvm);
 
 	write_unlock(&kvm->mmu_lock);
 
@@ -5492,29 +5508,29 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 	int i;
 	bool flush = false;
 
-	write_lock(&kvm->mmu_lock);
-	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
-		slots = __kvm_memslots(kvm, i);
-		kvm_for_each_memslot(memslot, slots) {
-			gfn_t start, end;
-
-			start = max(gfn_start, memslot->base_gfn);
-			end = min(gfn_end, memslot->base_gfn + memslot->npages);
-			if (start >= end)
-				continue;
+	if (kvm->arch.memslots_have_rmaps) {
+		write_lock(&kvm->mmu_lock);
+		for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+			slots = __kvm_memslots(kvm, i);
+			kvm_for_each_memslot(memslot, slots) {
+				gfn_t start, end;
+
+				start = max(gfn_start, memslot->base_gfn);
+				end = min(gfn_end, memslot->base_gfn + memslot->npages);
+				if (start >= end)
+					continue;
 
-			flush = slot_handle_level_range(kvm, memslot, kvm_zap_rmapp,
-							PG_LEVEL_4K,
-							KVM_MAX_HUGEPAGE_LEVEL,
-							start, end - 1, true, flush);
+				flush = slot_handle_level_range(kvm, memslot,
+						kvm_zap_rmapp, PG_LEVEL_4K,
+						KVM_MAX_HUGEPAGE_LEVEL, start,
+						end - 1, true, flush);
+			}
 		}
+		if (flush)
+			kvm_flush_remote_tlbs_with_address(kvm, gfn_start, gfn_end);
+		write_unlock(&kvm->mmu_lock);
 	}
 
-	if (flush)
-		kvm_flush_remote_tlbs_with_address(kvm, gfn_start, gfn_end);
-
-	write_unlock(&kvm->mmu_lock);
-
 	if (is_tdp_mmu_enabled(kvm)) {
 		flush = false;
 
@@ -5541,12 +5557,15 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm,
 				      struct kvm_memory_slot *memslot,
 				      int start_level)
 {
-	bool flush;
+	bool flush = false;
 
-	write_lock(&kvm->mmu_lock);
-	flush = slot_handle_level(kvm, memslot, slot_rmap_write_protect,
-				start_level, KVM_MAX_HUGEPAGE_LEVEL, false);
-	write_unlock(&kvm->mmu_lock);
+	if (kvm->arch.memslots_have_rmaps) {
+		write_lock(&kvm->mmu_lock);
+		flush = slot_handle_level(kvm, memslot, slot_rmap_write_protect,
+					  start_level, KVM_MAX_HUGEPAGE_LEVEL,
+					  false);
+		write_unlock(&kvm->mmu_lock);
+	}
 
 	if (is_tdp_mmu_enabled(kvm)) {
 		read_lock(&kvm->mmu_lock);
@@ -5616,16 +5635,15 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
 	struct kvm_memory_slot *slot = (struct kvm_memory_slot *)memslot;
 	bool flush;
 
-	write_lock(&kvm->mmu_lock);
-	flush = slot_handle_leaf(kvm, slot, kvm_mmu_zap_collapsible_spte, true);
-
-	if (flush)
-		kvm_arch_flush_remote_tlbs_memslot(kvm, slot);
-	write_unlock(&kvm->mmu_lock);
+	if (kvm->arch.memslots_have_rmaps) {
+		write_lock(&kvm->mmu_lock);
+		flush = slot_handle_leaf(kvm, slot, kvm_mmu_zap_collapsible_spte, true);
+		if (flush)
+			kvm_arch_flush_remote_tlbs_memslot(kvm, slot);
+		write_unlock(&kvm->mmu_lock);
+	}
 
 	if (is_tdp_mmu_enabled(kvm)) {
-		flush = false;
-
 		read_lock(&kvm->mmu_lock);
 		flush = kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot, flush);
 		if (flush)
@@ -5652,11 +5670,14 @@ void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
 void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
 				   struct kvm_memory_slot *memslot)
 {
-	bool flush;
+	bool flush = false;
 
-	write_lock(&kvm->mmu_lock);
-	flush = slot_handle_leaf(kvm, memslot, __rmap_clear_dirty, false);
-	write_unlock(&kvm->mmu_lock);
+	if (kvm->arch.memslots_have_rmaps) {
+		write_lock(&kvm->mmu_lock);
+		flush = slot_handle_leaf(kvm, memslot, __rmap_clear_dirty,
+					 false);
+		write_unlock(&kvm->mmu_lock);
+	}
 
 	if (is_tdp_mmu_enabled(kvm)) {
 		read_lock(&kvm->mmu_lock);
@@ -5681,6 +5702,14 @@ void kvm_mmu_zap_all(struct kvm *kvm)
 	int ign;
 
 	write_lock(&kvm->mmu_lock);
+	if (is_tdp_mmu_enabled(kvm))
+		kvm_tdp_mmu_zap_all(kvm);
+
+	if (!kvm->arch.memslots_have_rmaps) {
+		write_unlock(&kvm->mmu_lock);
+		return;
+	}
+
 restart:
 	list_for_each_entry_safe(sp, node, &kvm->arch.active_mmu_pages, link) {
 		if (WARN_ON(sp->role.invalid))
@@ -5693,9 +5722,6 @@ void kvm_mmu_zap_all(struct kvm *kvm)
 
 	kvm_mmu_commit_zap_page(kvm, &invalid_list);
 
-	if (is_tdp_mmu_enabled(kvm))
-		kvm_tdp_mmu_zap_all(kvm);
-
 	write_unlock(&kvm->mmu_lock);
 }
 
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
#14 lore raw

Re: [PATCH v4 6/7] KVM: x86/mmu: Skip rmap operations if rmaps not allocated

arch/x86/kvm/mmu/mmu.c bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, · line 1275

Quoted 12 lines · Ben Gardon
arch/x86/kvm/mmu/mmu.c +5 -3
@@ -1260,9 +1268,12 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
 	int i;
 	bool write_protected = false;
 
-	for (i = PG_LEVEL_4K; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
-		rmap_head = __gfn_to_rmap(gfn, i, slot);
-		write_protected |= __rmap_write_protect(kvm, rmap_head, true);
+	if (kvm->arch.memslots_have_rmaps) {
+		for (i = PG_LEVEL_4K; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
+			rmap_head = __gfn_to_rmap(gfn, i, slot);
+			write_protected |= __rmap_write_protect(kvm, rmap_head,
+								true);

I vote to let "true" poke out.

Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +1 -0
+		}
 	}
 
 	if (is_tdp_mmu_enabled(kvm))

...

arch/x86/kvm/mmu/mmu.c static void kvm_mmu_zap_all_fast(struct kvm *kvm) · line 5459

Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +2 -1
@@ -5440,7 +5455,8 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 	 */
 	kvm_reload_remote_mmus(kvm);
 
-	kvm_zap_obsolete_pages(kvm);
+	if (kvm->arch.memslots_have_rmaps)
+		kvm_zap_obsolete_pages(kvm);

Hmm, for cases where we're iterating over the list of active_mmu_pages, I would prefer to either leave the code as-is or short-circuit the helpers with a more explicit:

	if (list_empty(&kvm->arch.active_mmu_pages))
		return ...;

I'd probably vote for leaving the code as it is; the loop iteration and list_empty check in kvm_mmu_commit_zap_page() add a single compare-and-jump in the worst case scenario.

In other words, restrict use of memslots_have_rmaps to flows that directly walk the rmaps, as opposed to partially overloading memslots_have_rmaps to mean "is using legacy MMU".

Quoted 2 lines
(quoted diff) +0 -0
 	write_unlock(&kvm->mmu_lock);
 

...

arch/x86/kvm/mmu/mmu.c void kvm_mmu_zap_all(struct kvm *kvm) · line 5710

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +6 -0
@@ -5681,6 +5702,14 @@ void kvm_mmu_zap_all(struct kvm *kvm)
 	int ign;
 
 	write_lock(&kvm->mmu_lock);
+	if (is_tdp_mmu_enabled(kvm))
+		kvm_tdp_mmu_zap_all(kvm);
+
+	if (!kvm->arch.memslots_have_rmaps) {
+		write_unlock(&kvm->mmu_lock);
+		return;

Another case where falling through to walking active_mmu_pages is perfectly ok.

Quoted 18 lines
arch/x86/kvm/mmu/mmu.c +2 -3
+	}
+
 restart:
 	list_for_each_entry_safe(sp, node, &kvm->arch.active_mmu_pages, link) {
 		if (WARN_ON(sp->role.invalid))
@@ -5693,9 +5722,6 @@ void kvm_mmu_zap_all(struct kvm *kvm)
 
 	kvm_mmu_commit_zap_page(kvm, &invalid_list);
 
-	if (is_tdp_mmu_enabled(kvm))
-		kvm_tdp_mmu_zap_all(kvm);
-
 	write_unlock(&kvm->mmu_lock);
 }
 
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
author
#15 lore raw

PATCH v4 7/7 KVM: x86/mmu: Lazily allocate memslot rmaps

In reply to Ben Gardon

If the TDP MMU is in use, wait to allocate the rmaps until the shadow MMU is actually used. (i.e. a nested VM is launched.) This saves memory equal to 0.2% of guest memory in cases where the TDP MMU is used and there are no nested guests involved.

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu/mmu.c          | 53 +++++++++++++++++++++++----------
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 ++--
 arch/x86/kvm/mmu/tdp_mmu.h      |  4 +--
 arch/x86/kvm/x86.c              | 45 +++++++++++++++++++++++++++-
 5 files changed, 89 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
index fc75ed49bfee..7b65f82ade1c 100644
@@ -1868,4 +1868,6 @@ static inline int kvm_cpu_get_apicid(int mps_cpu)
 
 int kvm_cpu_dirty_log_size(void);
 
+int alloc_all_memslots_rmaps(struct kvm *kvm);
+
 #endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/mmu/mmu.c +37 -16

143 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.c +4 -2
index 95eeb5ac6a8a..ea00c9502ba1 100644
@@ -14,10 +14,10 @@ static bool __read_mostly tdp_mmu_enabled = false;
 module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0644);
 
 /* Initializes the TDP MMU for the VM, if enabled. */
-void kvm_mmu_init_tdp_mmu(struct kvm *kvm)
+bool kvm_mmu_init_tdp_mmu(struct kvm *kvm)
 {
 	if (!tdp_enabled || !READ_ONCE(tdp_mmu_enabled))
-		return;
+		return false;
 
 	/* This should not be changed for the lifetime of the VM. */
 	kvm->arch.tdp_mmu_enabled = true;
@@ -25,6 +25,8 @@ void kvm_mmu_init_tdp_mmu(struct kvm *kvm)
 	INIT_LIST_HEAD(&kvm->arch.tdp_mmu_roots);
 	spin_lock_init(&kvm->arch.tdp_mmu_pages_lock);
 	INIT_LIST_HEAD(&kvm->arch.tdp_mmu_pages);
+
+	return true;
 }
 
 static __always_inline void kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm,
arch/x86/kvm/mmu/tdp_mmu.h +2 -2
index 5fdf63090451..b046ab5137a1 100644
@@ -80,12 +80,12 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,
 			 int *root_level);
 
 #ifdef CONFIG_X86_64
-void kvm_mmu_init_tdp_mmu(struct kvm *kvm);
+bool kvm_mmu_init_tdp_mmu(struct kvm *kvm);
 void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm);
 static inline bool is_tdp_mmu_enabled(struct kvm *kvm) { return kvm->arch.tdp_mmu_enabled; }
 static inline bool is_tdp_mmu_page(struct kvm_mmu_page *sp) { return sp->tdp_mmu_page; }
 #else
-static inline void kvm_mmu_init_tdp_mmu(struct kvm *kvm) {}
+static inline bool kvm_mmu_init_tdp_mmu(struct kvm *kvm) { return false; }
 static inline void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) {}
 static inline bool is_tdp_mmu_enabled(struct kvm *kvm) { return false; }
 static inline bool is_tdp_mmu_page(struct kvm_mmu_page *sp) { return false; }
arch/x86/kvm/x86.c +44 -1

63 lines not on this page · raw message

Signature · 1 line
2.31.1.607.g51e8a6a459-goog
#16 lore raw

Re: [PATCH v4 7/7] KVM: x86/mmu: Lazily allocate memslot rmaps

arch/x86/kvm/x86.c static int memslot_rmap_alloc(struct kvm_memory_slot *slot, · line 10944

Quoted 10 lines · Ben Gardon
arch/x86/kvm/x86.c +5 -0
@@ -10935,6 +10937,46 @@ static int memslot_rmap_alloc(struct kvm_memory_slot *slot,
 	return 0;
 }
 
+int alloc_all_memslots_rmaps(struct kvm *kvm)
+{
+	struct kvm_memslots *slots;
+	struct kvm_memory_slot *slot;
+	int r = 0;

No need to initialize r. And then it makes sense to put i and r on the same line.

Quoted 7 lines
arch/x86/kvm/x86.c +7 -0
+	int i;
+
+	/*
+	 * Check memslots_have_rmaps early before acquiring the
+	 * slots_arch_lock below.
+	 */
+	if (smp_load_acquire(&kvm->arch.memslots_have_rmaps))

arch/x86/kvm/x86.c static int memslot_rmap_alloc(struct kvm_memory_slot *slot, · line 10961

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+		return 0;
+
+	mutex_lock(&kvm->slots_arch_lock);
+
+	/*
+	 * Read memslots_have_rmaps again, under the slots arch lock,
+	 * before allocating the rmaps
+	 */
+	if (smp_load_acquire(&kvm->arch.memslots_have_rmaps))
+		return 0;

This fails to drop slots_arch_lock.

Quoted 34 lines
arch/x86/kvm/x86.c +20 -1
+
+	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+		slots = __kvm_memslots(kvm, i);
+		kvm_for_each_memslot(slot, slots) {
+			r = memslot_rmap_alloc(slot, slot->npages);
+			if (r) {
+				mutex_unlock(&kvm->slots_arch_lock);
+				return r;
+			}
+		}
+	}
+
+	/* Write rmap pointers before memslots_have_rmaps */
+	smp_store_release(&kvm->arch.memslots_have_rmaps, true);
+	mutex_unlock(&kvm->slots_arch_lock);
+	return 0;
+}
+
 static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 				      struct kvm_memory_slot *slot,
 				      unsigned long npages)
@@ -10949,7 +10991,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 	 */
 	memset(&slot->arch, 0, sizeof(slot->arch));
 
-	if (kvm->arch.memslots_have_rmaps) {
+	/* Read memslots_have_rmaps before allocating the rmaps */
+	if (smp_load_acquire(&kvm->arch.memslots_have_rmaps)) {
 		r = memslot_rmap_alloc(slot, npages);
 		if (r)
 			return r;
Signature · 1 line
2.31.1.607.g51e8a6a459-goog
#17 lore raw

Re: [PATCH v4 7/7] KVM: x86/mmu: Lazily allocate memslot rmaps

In reply to Ben Gardon
Quoted 30 lines · Ben Gardon

If the TDP MMU is in use, wait to allocate the rmaps until the shadow MMU is actually used. (i.e. a nested VM is launched.) This saves memory equal to 0.2% of guest memory in cases where the TDP MMU is used and there are no nested guests involved.

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu/mmu.c          | 53 +++++++++++++++++++++++----------
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 ++--
 arch/x86/kvm/mmu/tdp_mmu.h      |  4 +--
 arch/x86/kvm/x86.c              | 45 +++++++++++++++++++++++++++-
 5 files changed, 89 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
index fc75ed49bfee..7b65f82ade1c 100644
@@ -1868,4 +1868,6 @@ static inline int kvm_cpu_get_apicid(int mps_cpu)
 
 int kvm_cpu_dirty_log_size(void);
 
+int alloc_all_memslots_rmaps(struct kvm *kvm);
+
 #endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/mmu/mmu.c +0 -0
index b0bdb924d519..183afccd2944 100644

arch/x86/kvm/mmu/mmu.c static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, · line 1193

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -1190,7 +1190,8 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
 		kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,
 				slot->base_gfn + gfn_offset, mask, true);
 
-	if (!kvm->arch.memslots_have_rmaps)
+	/* Read memslots_have_rmaps before the rmaps themselves */

IIRC, you open coded reading memslots_have_rmaps because of a circular dependency, but you can solve that simply by defining the helper in mmu.h instead of kvm_host.h.

And I think you could even make it static in mmu.c and omit the smp_load_acuquire from the three users in x86.c, though that's probably not worth it.

Either way, reading the same comment over and over and over, just to make checkpatch happy, gets more than a bit tedious.

That would also allow you to elaborate on why the smp_load_acquire() is necessary, and preferably what it pairs with.

Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +1 -0
+	if (!smp_load_acquire(&kvm->arch.memslots_have_rmaps))
 		return;
 
 	while (mask) {
#18 lore raw

Re: [PATCH v4 7/7] KVM: x86/mmu: Lazily allocate memslot rmaps

Quoted 41 lines · Sean Christopherson
Quoted 30 lines · Ben Gardon

If the TDP MMU is in use, wait to allocate the rmaps until the shadow MMU is actually used. (i.e. a nested VM is launched.) This saves memory equal to 0.2% of guest memory in cases where the TDP MMU is used and there are no nested guests involved.

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu/mmu.c          | 53 +++++++++++++++++++++++----------
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 ++--
 arch/x86/kvm/mmu/tdp_mmu.h      |  4 +--
 arch/x86/kvm/x86.c              | 45 +++++++++++++++++++++++++++-
 5 files changed, 89 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
index fc75ed49bfee..7b65f82ade1c 100644
@@ -1868,4 +1868,6 @@ static inline int kvm_cpu_get_apicid(int mps_cpu)
 
 int kvm_cpu_dirty_log_size(void);
 
+int alloc_all_memslots_rmaps(struct kvm *kvm);
+
 #endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/mmu/mmu.c +0 -0
index b0bdb924d519..183afccd2944 100644

arch/x86/kvm/mmu/mmu.c static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, · line 1193

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -1190,7 +1190,8 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
 		kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,
 				slot->base_gfn + gfn_offset, mask, true);
 
-	if (!kvm->arch.memslots_have_rmaps)
+	/* Read memslots_have_rmaps before the rmaps themselves */

IIRC, you open coded reading memslots_have_rmaps because of a circular dependency, but you can solve that simply by defining the helper in mmu.h instead of kvm_host.h.

Quoted 8 lines

And I think you could even make it static in mmu.c and omit the smp_load_acuquire from the three users in x86.c, though that's probably not worth it.

Either way, reading the same comment over and over and over, just to make checkpatch happy, gets more than a bit tedious.

That would also allow you to elaborate on why the smp_load_acquire() is necessary, and preferably what it pairs with.

Belated thought: you could also introduce the helper in patch 06 in order to miminize thrash in this patch.