Skip to contentSkip to conversation

KVM: x86/mmu: Use page-track only for... page tracking

PATCH In review Sean Christopherson 2 participants 7 messages Nov 10, 2022 – Nov 11, 2022

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

cover KVM: x86/mmu: Use page-track only for... page tracking

Don't bounce through the page-track notifier when zapping+flushing SPTEs in response to memslot changes as the need to zap+flush isn't strictly limited to page-tracking. With that done, register KVM's notifier on the first allocation of a shadow root, as KVM's ->track_write() hook is used only to react to writes to gPTEs.

Aside from avoiding a RETPOLINE on emulated writes, dropping KVM's internal use will allow removing ->track_flush_slot() altogether once KVM-GT moves to a different hook[*]. Tracking "flushes" of slots is a poor fit for KVM-GT's needs as KVM-GT needs to drop its write-protection only when a memslot change is guaranteed to be committed, whereas the "flush" call is speculative in the sense that KVM may abort a memslot update after flushing the original memslot.

https://lore.kernel.org/all/[email protected]

Sean Christopherson (2):
KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot
change
KVM: x86/mmu: Register page-tracker on first shadow root allocation

 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/mmu/mmu.c          | 24 ++++++++----------------
 arch/x86/kvm/x86.c              |  2 ++
 3 files changed, 11 insertions(+), 16 deletions(-)

base-commit: d663b8a285986072428a6a145e5994bc275df994

Signature · 1 line
2.38.1.431.g37b22c650d-goog
author maintainer
#1 lore raw

PATCH 1/2 KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change

Call kvm_mmu_zap_all_fast() directly when flushing a memslot instead of bounding through the page-track mechanism. KVM (unfortunately) needs to zap and flush all page tables on memslot DELETE/MOVE irrespective of whether KVM is shadowing guest page tables.

This will allow changing KVM to register a page-track notifier on the first shadow root allocation, and will also allow deleting the misguided kvm_page_track_flush_slot() hook itself once KVM-GT also moves to a different method for reacting to memslot changes.

No functional change intended.

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/mmu/mmu.c          | 10 +---------
 arch/x86/kvm/x86.c              |  2 ++
 3 files changed, 4 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 81114a376c4e..382cfffb7e6c 100644
@@ -1765,6 +1765,7 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
 void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
 				   const struct kvm_memory_slot *memslot);
 void kvm_mmu_zap_all(struct kvm *kvm);
+void kvm_mmu_zap_all_fast(struct kvm *kvm);
 void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen);
 void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages);
 
arch/x86/kvm/mmu/mmu.c +1 -9
index 93c389eaf471..0a5ae07a190e 100644
@@ -5943,7 +5943,7 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm)
  * not use any resource of the being-deleted slot or all slots
  * after calling the function.
  */
-static void kvm_mmu_zap_all_fast(struct kvm *kvm)
+void kvm_mmu_zap_all_fast(struct kvm *kvm)
 {
 	lockdep_assert_held(&kvm->slots_lock);
 
@@ -5999,13 +5999,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
 	return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages));
 }
 
-static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm,
-			struct kvm_memory_slot *slot,
-			struct kvm_page_track_notifier_node *node)
-{
-	kvm_mmu_zap_all_fast(kvm);
-}
-
 int kvm_mmu_init_vm(struct kvm *kvm)
 {
 	struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
@@ -6021,7 +6014,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 		return r;
 
 	node->track_write = kvm_mmu_pte_write;
-	node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
 	kvm_page_track_register_notifier(kvm, node);
 
 	kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
arch/x86/kvm/x86.c +2 -0
index e46e458c5b08..5da86fe3c113 100644
@@ -12550,6 +12550,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
 void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 				   struct kvm_memory_slot *slot)
 {
+	kvm_mmu_zap_all_fast(kvm);
+
 	kvm_page_track_flush_slot(kvm, slot);
 }
 
Signature · 1 line
2.38.1.431.g37b22c650d-goog
#2 lore raw

Re: [PATCH 1/2] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change

Quoted 10 lines · Sean Christopherson

Call kvm_mmu_zap_all_fast() directly when flushing a memslot instead of bounding through the page-track mechanism. KVM (unfortunately) needs to zap and flush all page tables on memslot DELETE/MOVE irrespective of whether KVM is shadowing guest page tables.

This will allow changing KVM to register a page-track notifier on the first shadow root allocation, and will also allow deleting the misguided kvm_page_track_flush_slot() hook itself once KVM-GT also moves to a different method for reacting to memslot changes.

<...>

Quoted 13 lines
arch/x86/kvm/mmu/mmu.c +0 -1
@@ -6021,7 +6014,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 		return r;
 
 	node->track_write = kvm_mmu_pte_write;
-	node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
 	kvm_page_track_register_notifier(kvm, node);
 
 	kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
arch/x86/kvm/x86.c +0 -0
index e46e458c5b08..5da86fe3c113 100644

arch/x86/kvm/x86.c void kvm_arch_flush_shadow_all(struct kvm *kvm) · line 12555

Quoted 6 lines
arch/x86/kvm/x86.c +2 -0
@@ -12550,6 +12550,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
 void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 				   struct kvm_memory_slot *slot)
 {
+	kvm_mmu_zap_all_fast(kvm);
+
 	kvm_page_track_flush_slot(kvm, slot);

Could we move this kvm_page_track_flush_slot() to right before kvm_commit_memory_region()?
As KVM now does not need track_flush_slot any more and kvmgt is the only user to track_flush_slot, we can rename it to track_slot_changed to notify the new/deleted/moved slot.
Do you think it's good?

Thanks
Yan

Quoted 5 lines
arch/x86/kvm/x86.c +0 -0
 }
 
Signature · 1 line
2.38.1.431.g37b22c650d-goog
author maintainer
#3 lore raw

Re: [PATCH 1/2] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change

Quoted 19 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson

Call kvm_mmu_zap_all_fast() directly when flushing a memslot instead of bounding through the page-track mechanism. KVM (unfortunately) needs to zap and flush all page tables on memslot DELETE/MOVE irrespective of whether KVM is shadowing guest page tables.

This will allow changing KVM to register a page-track notifier on the first shadow root allocation, and will also allow deleting the misguided kvm_page_track_flush_slot() hook itself once KVM-GT also moves to a different method for reacting to memslot changes.

<...>

Quoted 7 lines
(quoted diff) +0 -1
@@ -6021,7 +6014,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 		return r;
 
 	node->track_write = kvm_mmu_pte_write;
-	node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
 	kvm_page_track_register_notifier(kvm, node);
 
Quoted 14 lines

arch/x86/kvm/x86.c void kvm_arch_flush_shadow_all(struct kvm *kvm) · line 12555

Quoted 12 lines
(quoted diff) +0 -0
 	kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
arch/x86/kvm/x86.c +2 -0
index e46e458c5b08..5da86fe3c113 100644
@@ -12550,6 +12550,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
 void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 				   struct kvm_memory_slot *slot)
 {
+	kvm_mmu_zap_all_fast(kvm);
+
 	kvm_page_track_flush_slot(kvm, slot);

Could we move this kvm_page_track_flush_slot() to right before kvm_commit_memory_region()?

More or less. The page-track stuff is x86-specific, just move it into x86's kvm_arch_commit_memory_region().

Quoted 4 lines

As KVM now does not need track_flush_slot any more and kvmgt is the only user to track_flush_slot, we can rename it to track_slot_changed to notify the new/deleted/moved slot.
Do you think it's good?

Given that KVM/KVM-GT have never propery supported the MOVE case, and (IIUC) that there's no danger to the kernel if KVM-GT fails to write-protect a moved memslot, I would say just change the hook to ->remove_memslot(). I.e. even if the memslot is being moved, simply notify KVM-GT that the old memslot is being removed.

E.g.

arch/x86/kvm/x86.c +3 -0
index 5a2821ca03b8..437e3832e377 100644
@@ -12566,6 +12566,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
                                const struct kvm_memory_slot *new,
                                enum kvm_mr_change change)
 {
+       if (change == KVM_MR_DELETE || change == KVM_MR_MOVE)
+               kvm_page_track_remove_memslot(kvm, old);
+
        if (!kvm->arch.n_requested_mmu_pages &&
            (change == KVM_MR_CREATE || change == KVM_MR_DELETE)) {
                unsigned long nr_mmu_pages;
#4 lore raw

Re: [PATCH 1/2] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change

Quoted 43 lines · Sean Christopherson
Quoted 19 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson

Call kvm_mmu_zap_all_fast() directly when flushing a memslot instead of bounding through the page-track mechanism. KVM (unfortunately) needs to zap and flush all page tables on memslot DELETE/MOVE irrespective of whether KVM is shadowing guest page tables.

This will allow changing KVM to register a page-track notifier on the first shadow root allocation, and will also allow deleting the misguided kvm_page_track_flush_slot() hook itself once KVM-GT also moves to a different method for reacting to memslot changes.

<...>

Quoted 7 lines
(quoted diff) +0 -1
@@ -6021,7 +6014,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 		return r;
 
 	node->track_write = kvm_mmu_pte_write;
-	node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
 	kvm_page_track_register_notifier(kvm, node);
 
Quoted 14 lines

arch/x86/kvm/x86.c void kvm_arch_flush_shadow_all(struct kvm *kvm) · line 12555

Quoted 12 lines
(quoted diff) +0 -0
 	kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
arch/x86/kvm/x86.c +2 -0
index e46e458c5b08..5da86fe3c113 100644
@@ -12550,6 +12550,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
 void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 				   struct kvm_memory_slot *slot)
 {
+	kvm_mmu_zap_all_fast(kvm);
+
 	kvm_page_track_flush_slot(kvm, slot);

Could we move this kvm_page_track_flush_slot() to right before kvm_commit_memory_region()?

More or less. The page-track stuff is x86-specific, just move it into x86's kvm_arch_commit_memory_region().

Quoted 4 lines

As KVM now does not need track_flush_slot any more and kvmgt is the only user to track_flush_slot, we can rename it to track_slot_changed to notify the new/deleted/moved slot.
Do you think it's good?

Quoted 5 lines

Given that KVM/KVM-GT have never propery supported the MOVE case, and (IIUC) that there's no danger to the kernel if KVM-GT fails to write-protect a moved memslot, I would say just change the hook to ->remove_memslot(). I.e. even if the memslot is being moved, simply notify KVM-GT that the old memslot is being removed.

I think it should be good.
We can refine the support of MOVE later after it really happens. Will do it by following your suggestions and based on this series :)

Thanks
Yan

Quoted 16 lines

E.g.

arch/x86/kvm/x86.c +3 -0
index 5a2821ca03b8..437e3832e377 100644
@@ -12566,6 +12566,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
                                const struct kvm_memory_slot *new,
                                enum kvm_mr_change change)
 {
+       if (change == KVM_MR_DELETE || change == KVM_MR_MOVE)
+               kvm_page_track_remove_memslot(kvm, old);
+
        if (!kvm->arch.n_requested_mmu_pages &&
            (change == KVM_MR_CREATE || change == KVM_MR_DELETE)) {
                unsigned long nr_mmu_pages;
author maintainer
#5 lore raw

PATCH 2/2 KVM: x86/mmu: Register page-tracker on first shadow root allocation

In reply to Sean Christopherson

Defer registering KVM's shadow page tracker until the first shadow root allocation now that KVM doesn't rely on the tracker to zap+flush SPTEs when a memslot is moved or deleted.

---
 arch/x86/kvm/mmu/mmu.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -7
index 0a5ae07a190e..d35a86a60d4f 100644
@@ -3678,11 +3678,14 @@ static int mmu_first_shadow_root_alloc(struct kvm *kvm)
 		}
 	}
 
+out_success:
+	/* Register KVM's page-tracker to react to guest writes to gPTEs. */
+	kvm_page_track_register_notifier(kvm, &kvm->arch.mmu_sp_tracker);
+
 	/*
 	 * Ensure that shadow_root_allocated becomes true strictly after
 	 * all the related pointers are set.
 	 */
-out_success:
 	smp_store_release(&kvm->arch.shadow_root_allocated, true);
 
 out_unlock:
@@ -6001,7 +6004,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
 
 int kvm_mmu_init_vm(struct kvm *kvm)
 {
-	struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
 	int r;
 
 	INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
@@ -6013,8 +6015,7 @@ int kvm_mmu_init_vm(struct kvm *kvm)
 	if (r < 0)
 		return r;
 
-	node->track_write = kvm_mmu_pte_write;
-	kvm_page_track_register_notifier(kvm, node);
+	kvm->arch.mmu_sp_tracker.track_write = kvm_mmu_pte_write;
 
 	kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
 	kvm->arch.split_page_header_cache.gfp_zero = __GFP_ZERO;
@@ -6036,9 +6037,8 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
 
 void kvm_mmu_uninit_vm(struct kvm *kvm)
 {
-	struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
-
-	kvm_page_track_unregister_notifier(kvm, node);
+	if (kvm_shadow_root_allocated(kvm))
+		kvm_page_track_unregister_notifier(kvm, &kvm->arch.mmu_sp_tracker);
 
 	kvm_mmu_uninit_tdp_mmu(kvm);
 
Signature · 1 line
2.38.1.431.g37b22c650d-goog
author maintainer
#6 lore raw
In reply to Sean Christopherson
Quoted 14 lines · Sean Christopherson

Don't bounce through the page-track notifier when zapping+flushing SPTEs in response to memslot changes as the need to zap+flush isn't strictly limited to page-tracking. With that done, register KVM's notifier on the first allocation of a shadow root, as KVM's ->track_write() hook is used only to react to writes to gPTEs.

Aside from avoiding a RETPOLINE on emulated writes, dropping KVM's internal use will allow removing ->track_flush_slot() altogether once KVM-GT moves to a different hook[*]. Tracking "flushes" of slots is a poor fit for KVM-GT's needs as KVM-GT needs to drop its write-protection only when a memslot change is guaranteed to be committed, whereas the "flush" call is speculative in the sense that KVM may abort a memslot update after flushing the original memslot.

Quoted 6 lines

https://lore.kernel.org/all/[email protected]

Sean Christopherson (2):
KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot
change
KVM: x86/mmu: Register page-tracker on first shadow root allocation

Don't merge this series, I'm going to (hopefully) send a (much larger) v2 that more aggressively cleans up the page tracker APIs, and will replace patch 2 with a completely different patch.