KVM: x86/mmu: Use page-track only for... page tracking
Sean Christopherson
maintainercover KVM: x86/mmu: Use page-track only for... page tracking
Don't bounce through the page-track notifier when zapping+flushing SPTEs in response to memslot changes as the need to zap+flush isn't strictly limited to page-tracking. With that done, register KVM's notifier on the first allocation of a shadow root, as KVM's ->track_write() hook is used only to react to writes to gPTEs.
Aside from avoiding a RETPOLINE on emulated writes, dropping KVM's internal use will allow removing ->track_flush_slot() altogether once KVM-GT moves to a different hook[*]. Tracking "flushes" of slots is a poor fit for KVM-GT's needs as KVM-GT needs to drop its write-protection only when a memslot change is guaranteed to be committed, whereas the "flush" call is speculative in the sense that KVM may abort a memslot update after flushing the original memslot.
https://lore.kernel.org/all/[email protected]
Sean Christopherson (2):
KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot
change
KVM: x86/mmu: Register page-tracker on first shadow root allocation
arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu/mmu.c | 24 ++++++++---------------- arch/x86/kvm/x86.c | 2 ++ 3 files changed, 11 insertions(+), 16 deletions(-)
base-commit: d663b8a285986072428a6a145e5994bc275df994
Signature · 1 line
2.38.1.431.g37b22c650d-goog
Sean Christopherson
maintainerPATCH 1/2 KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change
Call kvm_mmu_zap_all_fast() directly when flushing a memslot instead of bounding through the page-track mechanism. KVM (unfortunately) needs to zap and flush all page tables on memslot DELETE/MOVE irrespective of whether KVM is shadowing guest page tables.
This will allow changing KVM to register a page-track notifier on the first shadow root allocation, and will also allow deleting the misguided kvm_page_track_flush_slot() hook itself once KVM-GT also moves to a different method for reacting to memslot changes.
No functional change intended.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu/mmu.c | 10 +--------- arch/x86/kvm/x86.c | 2 ++ 3 files changed, 4 insertions(+), 9 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1765,6 +1765,7 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,void kvm_mmu_slot_leaf_clear_dirty (struct kvm *kvm,const struct kvm_memory_slot *memslot);void kvm_mmu_zap_all (struct kvm *kvm); +void kvm_mmu_zap_all_fast (struct kvm *kvm);void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen);void kvm_mmu_change_mmu_pages (struct kvm *kvm,unsigned long kvm_nr_mmu_pages);
arch/x86/kvm/mmu/mmu.c +1 -9
@@ -5943,7 +5943,7 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm)* not use any resource of the being-deleted slot or all slots * after calling the function. */ -static void kvm_mmu_zap_all_fast (struct kvm *kvm) +void kvm_mmu_zap_all_fast (struct kvm *kvm) {lockdep_assert_held (&kvm->slots_lock);
@@ -5999,13 +5999,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)return unlikely (!list_empty_careful (&kvm->arch.zapped_obsolete_pages)); } -static void kvm_mmu_invalidate_zap_pages_in_memslot (struct kvm *kvm, -struct kvm_memory_slot *slot, -struct kvm_page_track_notifier_node *node) -{ -kvm_mmu_zap_all_fast (kvm); -} -int kvm_mmu_init_vm (struct kvm *kvm) {struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;
@@ -6021,7 +6014,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)return r; node->track_write = kvm_mmu_pte_write; - node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;kvm_page_track_register_notifier (kvm, node); kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;
arch/x86/kvm/x86.c +2 -0
@@ -12550,6 +12550,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) { +kvm_mmu_zap_all_fast (kvm); +kvm_page_track_flush_slot (kvm, slot); }
Signature · 1 line
2.38.1.431.g37b22c650d-goog
Re: [PATCH 1/2] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change
Quoted 10 lines · Sean Christopherson
Call
kvm_mmu_zap_all_fast()directly when flushing a memslot instead of bounding through the page-track mechanism. KVM (unfortunately) needs to zap and flush all page tables on memslot DELETE/MOVE irrespective of whether KVM is shadowing guest page tables.This will allow changing KVM to register a page-track notifier on the first shadow root allocation, and will also allow deleting the misguided
kvm_page_track_flush_slot()hook itself once KVM-GT also moves to a different method for reacting to memslot changes.
<...>
Quoted 13 lines
arch/x86/kvm/mmu/mmu.c +0
-1@@ -6021,7 +6014,6 @@ int kvm_mmu_init_vm(struct kvm *kvm)return r; node->track_write = kvm_mmu_pte_write; - node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;kvm_page_track_register_notifier (kvm, node); kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c void kvm_arch_flush_shadow_all(struct kvm *kvm) · line 12555
Quoted 6 lines
arch/x86/kvm/x86.c +2
-0@@ -12550,6 +12550,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) { +kvm_mmu_zap_all_fast (kvm); +kvm_page_track_flush_slot (kvm, slot);
Could we move this kvm_page_track_flush_slot() to right before kvm_commit_memory_region()?
As KVM now does not need track_flush_slot any more and kvmgt is the only user to track_flush_slot, we can rename it to track_slot_changed to notify the new/deleted/moved slot.
Do you think it's good?
Thanks
Yan
Quoted 5 lines
arch/x86/kvm/x86.c +0
-0}Signature · 1 line
2.38.1.431.g37b22c650d-goog
Sean Christopherson
maintainerRe: [PATCH 1/2] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change
Quoted 19 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson
Call
kvm_mmu_zap_all_fast()directly when flushing a memslot instead of bounding through the page-track mechanism. KVM (unfortunately) needs to zap and flush all page tables on memslot DELETE/MOVE irrespective of whether KVM is shadowing guest page tables.This will allow changing KVM to register a page-track notifier on the first shadow root allocation, and will also allow deleting the misguided
kvm_page_track_flush_slot()hook itself once KVM-GT also moves to a different method for reacting to memslot changes.<...>
Quoted 7 lines
(quoted diff) +0
-1@@ -6021,7 +6014,6 @@ int kvm_mmu_init_vm(struct kvm *kvm) return r; node->track_write = kvm_mmu_pte_write; - node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot; kvm_page_track_register_notifier(kvm, node);
Quoted 14 lines
arch/x86/kvm/x86.c void kvm_arch_flush_shadow_all(struct kvm *kvm) · line 12555
Quoted 12 lines
(quoted diff) +0
-0kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;arch/x86/kvm/x86.c +2
-0@@ -12550,6 +12550,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) { +kvm_mmu_zap_all_fast (kvm); +kvm_page_track_flush_slot (kvm, slot);Could we move this
kvm_page_track_flush_slot()to right beforekvm_commit_memory_region()?
More or less. The page-track stuff is x86-specific, just move it into x86's kvm_arch_commit_memory_region().
Quoted 4 lines
As KVM now does not need
track_flush_slotany more and kvmgt is the only user totrack_flush_slot, we can rename it totrack_slot_changedto notify the new/deleted/moved slot.
Do you think it's good?
Given that KVM/KVM-GT have never propery supported the MOVE case, and (IIUC) that there's no danger to the kernel if KVM-GT fails to write-protect a moved memslot, I would say just change the hook to ->remove_memslot(). I.e. even if the memslot is being moved, simply notify KVM-GT that the old memslot is being removed.
E.g.
arch/x86/kvm/x86.c +3 -0
@@ -12566,6 +12566,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_memory_slot *new, enum kvm_mr_change change) { + if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) + kvm_page_track_remove_memslot(kvm, old); +if (!kvm->arch.n_requested_mmu_pages && (change ==KVM_MR_CREATE || change ==KVM_MR_DELETE )) {unsigned long nr_mmu_pages;
Re: [PATCH 1/2] KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot change
Quoted 43 lines · Sean Christopherson
Quoted 19 lines · Yan Zhao
Quoted 10 lines · Sean Christopherson
Call
kvm_mmu_zap_all_fast()directly when flushing a memslot instead of bounding through the page-track mechanism. KVM (unfortunately) needs to zap and flush all page tables on memslot DELETE/MOVE irrespective of whether KVM is shadowing guest page tables.This will allow changing KVM to register a page-track notifier on the first shadow root allocation, and will also allow deleting the misguided
kvm_page_track_flush_slot()hook itself once KVM-GT also moves to a different method for reacting to memslot changes.<...>
Quoted 7 lines
(quoted diff) +0
-1@@ -6021,7 +6014,6 @@ int kvm_mmu_init_vm(struct kvm *kvm) return r; node->track_write = kvm_mmu_pte_write; - node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot; kvm_page_track_register_notifier(kvm, node);Quoted 14 lines
arch/x86/kvm/x86.c void kvm_arch_flush_shadow_all(struct kvm *kvm) · line 12555
Quoted 12 lines
(quoted diff) +0
-0kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache;arch/x86/kvm/x86.c +2
-0@@ -12550,6 +12550,8 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) { +kvm_mmu_zap_all_fast (kvm); +kvm_page_track_flush_slot (kvm, slot);Could we move this
kvm_page_track_flush_slot()to right beforekvm_commit_memory_region()?More or less. The page-track stuff is x86-specific, just move it into x86's
kvm_arch_commit_memory_region().Quoted 4 lines
As KVM now does not need
track_flush_slotany more and kvmgt is the only user totrack_flush_slot, we can rename it totrack_slot_changedto notify the new/deleted/moved slot.
Do you think it's good?
Quoted 5 lines
Given that KVM/KVM-GT have never propery supported the MOVE case, and (IIUC) that there's no danger to the kernel if KVM-GT fails to write-protect a moved memslot, I would say just change the hook to ->remove_memslot(). I.e. even if the memslot is being moved, simply notify KVM-GT that the old memslot is being removed.
I think it should be good.
We can refine the support of MOVE later after it really happens. Will do it by following your suggestions and based on this series :)
Thanks
Yan
Quoted 16 lines
E.g.
arch/x86/kvm/x86.c +3
-0@@ -12566,6 +12566,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_memory_slot *new, enum kvm_mr_change change) { + if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) + kvm_page_track_remove_memslot(kvm, old); +if (!kvm->arch.n_requested_mmu_pages && (change ==KVM_MR_CREATE || change ==KVM_MR_DELETE )) {unsigned long nr_mmu_pages;
Sean Christopherson
maintainerPATCH 2/2 KVM: x86/mmu: Register page-tracker on first shadow root allocation
In reply to Sean ChristophersonDefer registering KVM's shadow page tracker until the first shadow root allocation now that KVM doesn't rely on the tracker to zap+flush SPTEs when a memslot is moved or deleted.
--- arch/x86/kvm/mmu/mmu.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +7 -7
@@ -3678,11 +3678,14 @@ static int mmu_first_shadow_root_alloc(struct kvm *kvm) } } +out_success: +/* Register KVM's page-tracker to react to guest writes to gPTEs. */ +kvm_page_track_register_notifier (kvm, &kvm->arch.mmu_sp_tracker); +/* * Ensure that shadow_root_allocated becomes true strictly after * all the related pointers are set. */ -out_success:smp_store_release (&kvm->arch.shadow_root_allocated, true); out_unlock:
@@ -6001,7 +6004,6 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)int kvm_mmu_init_vm (struct kvm *kvm) { -struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker;int r;INIT_LIST_HEAD (&kvm->arch.active_mmu_pages);
@@ -6013,8 +6015,7 @@ int kvm_mmu_init_vm(struct kvm *kvm)if (r <0 )return r; - node->track_write = kvm_mmu_pte_write; -kvm_page_track_register_notifier (kvm, node); + kvm->arch.mmu_sp_tracker.track_write = kvm_mmu_pte_write; kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache; kvm->arch.split_page_header_cache.gfp_zero = __GFP_ZERO;
@@ -6036,9 +6037,8 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)void kvm_mmu_uninit_vm (struct kvm *kvm) { -struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker; - -kvm_page_track_unregister_notifier (kvm, node); +if (kvm_shadow_root_allocated (kvm)) +kvm_page_track_unregister_notifier (kvm, &kvm->arch.mmu_sp_tracker);kvm_mmu_uninit_tdp_mmu (kvm);
Signature · 1 line
2.38.1.431.g37b22c650d-goog
Sean Christopherson
maintainerQuoted 14 lines · Sean Christopherson
Don't bounce through the page-track notifier when zapping+flushing SPTEs in response to memslot changes as the need to zap+flush isn't strictly limited to page-tracking. With that done, register KVM's notifier on the first allocation of a shadow root, as KVM's ->track_write() hook is used only to react to writes to gPTEs.
Aside from avoiding a RETPOLINE on emulated writes, dropping KVM's internal use will allow removing ->track_flush_slot() altogether once KVM-GT moves to a different hook[*]. Tracking "flushes" of slots is a poor fit for KVM-GT's needs as KVM-GT needs to drop its write-protection only when a memslot change is guaranteed to be committed, whereas the "flush" call is speculative in the sense that KVM may abort a memslot update after flushing the original memslot.
Quoted 6 lines
https://lore.kernel.org/all/[email protected]
Sean Christopherson (2):
KVM: x86/mmu: Don't rely on page-track mechanism to flush on memslot
change
KVM: x86/mmu: Register page-tracker on first shadow root allocation
Don't merge this series, I'm going to (hopefully) send a (much larger) v2 that more aggressively cleans up the page tracker APIs, and will replace patch 2 with a completely different patch.