More parallel operations for the TDP MMU
v2 cover More parallel operations for the TDP MMU
Now that the TDP MMU is able to handle page faults in parallel, it's a relatively small change to expand to other operations. This series allows zapping a range of GFNs, reclaiming collapsible SPTEs (when disabling dirty logging), and enabling dirty logging to all happen under the MMU lock in read mode.
This is partly a cleanup + rewrite of the last few patches of the parallel page faults series. I've incorporated feedback from Sean and Paolo, but the patches have changed so much that I'm sending this as a separate series.
Ran kvm-unit-tests + selftests on an SMP kernel + Intel Skylake, with the TDP MMU enabled and disabled. This series introduces no new failures or warnings.
I know this will conflict horribly with the patches from Sean's series which were just queued, and I'll send a v2 to fix those conflicts + address any feedback on this v1.
Changelog
v2:
-- Rebased patches on top of kvm/queue to incorporate Sean's recent
TLB flushing changes
-- Dropped patch 5: "KVM: x86/mmu: comment for_each_tdp_mmu_root
requires MMU write lock" as the following patch to protect the roots list with RCU adds lockdep which makes the comment somewhat redundant.
Ben Gardon (13):
KVM: x86/mmu: Re-add const qualifier in
kvm_tdp_mmu_zap_collapsible_sptes
KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU
KVM: x86/mmu: use tdp_mmu_free_sp to free roots
KVM: x86/mmu: Merge TDP MMU put and free root
KVM: x86/mmu: Refactor yield safe root iterator
KVM: x86/mmu: Make TDP MMU root refcount atomic
KVM: x86/mmu: handle cmpxchg failure in kvm_tdp_mmu_get_root
KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU
KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read
lock
KVM: x86/mmu: Fast invalidation for TDP MMU
KVM: x86/mmu: Tear down roots in fast invalidation thread
arch/x86/include/asm/kvm_host.h | 21 +- arch/x86/kvm/mmu/mmu.c | 115 +++++++--- arch/x86/kvm/mmu/mmu_internal.h | 27 +-- arch/x86/kvm/mmu/tdp_mmu.c | 375 +++++++++++++++++++++++--------- arch/x86/kvm/mmu/tdp_mmu.h | 28 ++- include/linux/kvm_host.h | 2 +- 6 files changed, 407 insertions(+), 161 deletions(-)
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 1/13 KVM: x86/mmu: Re-add const qualifier in kvm_tdp_mmu_zap_collapsible_sptes
kvm_tdp_mmu_zap_collapsible_sptes unnecessarily removes the const qualifier from its memlsot argument, leading to a compiler warning. Add the const annotation and pass it to subsequent functions.
--- arch/x86/kvm/mmu/mmu.c | 10 +++++----- arch/x86/kvm/mmu/mmu_internal.h | 5 +++-- arch/x86/kvm/mmu/tdp_mmu.c | 5 +++-- arch/x86/kvm/mmu/tdp_mmu.h | 3 ++- include/linux/kvm_host.h | 2 +- 5 files changed, 14 insertions(+), 11 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -5
@@ -715,8 +715,7 @@ static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn)* handling slots that are not large page aligned. */ static struct kvm_lpage_info *lpage_info_slot (gfn_t gfn, -struct kvm_memory_slot *slot, -int level) +const struct kvm_memory_slot *slot,int level) {unsigned long idx;
@@ -2735,7 +2734,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) }static int host_pfn_mapping_level (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, -struct kvm_memory_slot *slot) +const struct kvm_memory_slot *slot) {unsigned long hva;pte_t *pte;
@@ -2761,8 +2760,9 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,return level; } -int kvm_mmu_max_mapping_level (struct kvm *kvm,struct kvm_memory_slot *slot, -gfn_t gfn,kvm_pfn_t pfn,int max_level) +int kvm_mmu_max_mapping_level (struct kvm *kvm, +const struct kvm_memory_slot *slot,gfn_t gfn, +kvm_pfn_t pfn,int max_level) {struct kvm_lpage_info *linfo;
arch/x86/kvm/mmu/mmu_internal.h +3 -2
@@ -161,8 +161,9 @@ enum {#define SET_SPTE_NEED_REMOTE_TLB_FLUSH BIT(1)#define SET_SPTE_SPURIOUS BIT(2) -int kvm_mmu_max_mapping_level (struct kvm *kvm,struct kvm_memory_slot *slot, -gfn_t gfn,kvm_pfn_t pfn,int max_level); +int kvm_mmu_max_mapping_level (struct kvm *kvm, +const struct kvm_memory_slot *slot,gfn_t gfn, +kvm_pfn_t pfn,int max_level);int kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,gfn_t gfn,int max_level,kvm_pfn_t *pfnp,bool huge_page_disallowed,int *req_level);
arch/x86/kvm/mmu/tdp_mmu.c +3 -2
@@ -1255,7 +1255,7 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm,*/ static bool zap_collapsible_spte_range (struct kvm *kvm,struct kvm_mmu_page *root, -struct kvm_memory_slot *slot, +const struct kvm_memory_slot *slot,bool flush) {gfn_t start = slot->base_gfn;
@@ -1296,7 +1296,8 @@ static bool zap_collapsible_spte_range(struct kvm *kvm,* be replaced by large mappings, for GFNs within the slot. */ bool kvm_tdp_mmu_zap_collapsible_sptes (struct kvm *kvm, -struct kvm_memory_slot *slot,bool flush) +const struct kvm_memory_slot *slot, +bool flush) {struct kvm_mmu_page *root;
arch/x86/kvm/mmu/tdp_mmu.h +2 -1
@@ -57,7 +57,8 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm, gfn_t gfn, unsigned long mask, bool wrprot);bool kvm_tdp_mmu_zap_collapsible_sptes (struct kvm *kvm, -struct kvm_memory_slot *slot,bool flush); +const struct kvm_memory_slot *slot, +bool flush);bool kvm_tdp_mmu_write_protect_gfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn);
include/linux/kvm_host.h +1 -1
@@ -1124,7 +1124,7 @@ __gfn_to_memslot(struct kvm_memslots *slots, gfn_t gfn) }static inline unsigned long -__gfn_to_hva_memslot (struct kvm_memory_slot *slot,gfn_t gfn) +__gfn_to_hva_memslot (const struct kvm_memory_slot *slot,gfn_t gfn) {return slot->userspace_addr + (gfn - slot->base_gfn) *PAGE_SIZE ; }
Signature · 1 line
2.31.0.208.g409f899ff0-goog
Sean Christopherson
maintainerRe: [PATCH v2 01/13] KVM: x86/mmu: Re-add const qualifier in kvm_tdp_mmu_zap_collapsible_sptes
Quoted 18 lines · Ben Gardon
kvm_tdp_mmu_zap_collapsible_sptesunnecessarily removes the const qualifier from its memlsot argument, leading to a compiler warning. Add the const annotation and pass it to subsequent functions.Signed-off-by Ben Gardon <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 10 +++++----- arch/x86/kvm/mmu/mmu_internal.h | 5 +++-- arch/x86/kvm/mmu/tdp_mmu.c | 5 +++-- arch/x86/kvm/mmu/tdp_mmu.h | 3 ++- include/linux/kvm_host.h | 2 +- 5 files changed, 14 insertions(+), 11 deletions(-)arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn) · line 718
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +1
-2@@ -715,8 +715,7 @@ static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn)* handling slots that are not large page aligned. */ static struct kvm_lpage_info *lpage_info_slot (gfn_t gfn, -struct kvm_memory_slot *slot, -int level) +const struct kvm_memory_slot *slot,int level)
I'd prefer to let this poke out (it's only 2 chars) instead of having an unaligned param list. And if we want to make it pretty, we could always do:
static struct kvm_lpage_info *lpage_info_slot(const struct kvm_memory_slot *slot,
gfn_t gfn, int level)
PATCH v2 2/13 KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU
In reply to Ben GardonThe TDP MMU is almost the only user of kvm_mmu_get_root and kvm_mmu_put_root. There is only one use of put_root in mmu.c for the legacy / shadow MMU. Open code that one use and move the get / put functions to the TDP MMU so they can be extended in future commits.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 10 ++++------ arch/x86/kvm/mmu/mmu_internal.h | 16 ---------------- arch/x86/kvm/mmu/tdp_mmu.c | 6 +++--- arch/x86/kvm/mmu/tdp_mmu.h | 18 ++++++++++++++++++ 4 files changed, 25 insertions(+), 25 deletions(-)
arch/x86/kvm/mmu/mmu.c +4 -6
@@ -3153,12 +3153,10 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, sp =to_shadow_page (*root_hpa &PT64_BASE_ADDR_MASK ); -if (kvm_mmu_put_root (kvm, sp)) { -if (is_tdp_mmu_page (sp)) -kvm_tdp_mmu_free_root (kvm, sp); -else if (sp->role.invalid) -kvm_mmu_prepare_zap_page (kvm, sp, invalid_list); - } +if (is_tdp_mmu_page (sp) &&kvm_tdp_mmu_put_root (kvm, sp)) +kvm_tdp_mmu_free_root (kvm, sp); +else if (!--sp->root_count && sp->role.invalid) +kvm_mmu_prepare_zap_page (kvm, sp, invalid_list); *root_hpa =INVALID_PAGE ; }
arch/x86/kvm/mmu/mmu_internal.h +0 -16
@@ -113,22 +113,6 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,void kvm_flush_remote_tlbs_with_address (struct kvm *kvm,u64 start_gfn,u64 pages); -static inline void kvm_mmu_get_root (struct kvm *kvm,struct kvm_mmu_page *sp) -{ -BUG_ON (!sp->root_count); -lockdep_assert_held (&kvm->mmu_lock); - - ++sp->root_count; -} - -static inline bool kvm_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *sp) -{ -lockdep_assert_held (&kvm->mmu_lock); - --sp->root_count; - -return !sp->root_count; -} -static inline int kvm_mmu_role_as_id (union kvm_mmu_page_role role) {return role.smm ?1 :0 ;
arch/x86/kvm/mmu/tdp_mmu.c +3 -3
@@ -43,7 +43,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)static void tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root) { -if (kvm_mmu_put_root (kvm, root)) +if (kvm_tdp_mmu_put_root (kvm, root))kvm_tdp_mmu_free_root (kvm, root); }
@@ -55,7 +55,7 @@ static inline bool tdp_mmu_next_root_valid(struct kvm *kvm,if (list_entry_is_head (root, &kvm->arch.tdp_mmu_roots, link))return false; -kvm_mmu_get_root (kvm, root); +kvm_tdp_mmu_get_root (kvm, root);return true; }
@@ -154,7 +154,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)/* Check for an existing root before allocating a new one. */ for_each_tdp_mmu_root (kvm, root,kvm_mmu_role_as_id (role)) {if (root->role.word == role.word) { -kvm_mmu_get_root (kvm, root); +kvm_tdp_mmu_get_root (kvm, root); goto out; } }
arch/x86/kvm/mmu/tdp_mmu.h +18 -0
@@ -8,6 +8,24 @@hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu);void kvm_tdp_mmu_free_root (struct kvm *kvm,struct kvm_mmu_page *root); +static inline void kvm_tdp_mmu_get_root (struct kvm *kvm, +struct kvm_mmu_page *root) +{ +BUG_ON (!root->root_count); +lockdep_assert_held (&kvm->mmu_lock); + + ++root->root_count; +} + +static inline bool kvm_tdp_mmu_put_root (struct kvm *kvm, +struct kvm_mmu_page *root) +{ +lockdep_assert_held (&kvm->mmu_lock); + --root->root_count; + +return !root->root_count; +} +bool __kvm_tdp_mmu_zap_gfn_range (struct kvm *kvm,int as_id,gfn_t start,gfn_t end,bool can_yield,bool flush); static inline bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id,
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 3/13 KVM: x86/mmu: use tdp_mmu_free_sp to free roots
In reply to Ben GardonMinor cleanup to deduplicate the code used to free a struct kvm_mmu_page in the TDP MMU.
No functional change intended.
--- arch/x86/kvm/mmu/tdp_mmu.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +7 -8
@@ -92,6 +92,12 @@ static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,static bool zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root,gfn_t start,gfn_t end,bool can_yield,bool flush); +static void tdp_mmu_free_sp (struct kvm_mmu_page *sp) +{ +free_page ((unsigned long )sp->spt); +kmem_cache_free (mmu_page_header_cache, sp); +} +void kvm_tdp_mmu_free_root (struct kvm *kvm,struct kvm_mmu_page *root) {gfn_t max_gfn =1ULL << (shadow_phys_bits -PAGE_SHIFT );
@@ -105,8 +111,7 @@ void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root)zap_gfn_range (kvm, root,0 , max_gfn, false, false); -free_page ((unsigned long )root->spt); -kmem_cache_free (mmu_page_header_cache, root); +tdp_mmu_free_sp (root); } static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu,
@@ -168,12 +173,6 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)return __pa (root->spt); } -static void tdp_mmu_free_sp (struct kvm_mmu_page *sp) -{ -free_page ((unsigned long )sp->spt); -kmem_cache_free (mmu_page_header_cache, sp); -} -/* * This is called through call_rcu in order to free TDP page table memory * safely with respect to other kernel threads that may be operating on
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 4/13 KVM: x86/mmu: Merge TDP MMU put and free root
In reply to Ben Gardonkvm_tdp_mmu_put_root and kvm_tdp_mmu_free_root are always called together, so merge the functions to simplify TDP MMU root refcounting / freeing.
--- arch/x86/kvm/mmu/mmu.c | 4 +-- arch/x86/kvm/mmu/tdp_mmu.c | 54 ++++++++++++++++++-------------------- arch/x86/kvm/mmu/tdp_mmu.h | 10 +------ 3 files changed, 28 insertions(+), 40 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -2
@@ -3153,8 +3153,8 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, sp =to_shadow_page (*root_hpa &PT64_BASE_ADDR_MASK ); -if (is_tdp_mmu_page (sp) &&kvm_tdp_mmu_put_root (kvm, sp)) -kvm_tdp_mmu_free_root (kvm, sp); +if (is_tdp_mmu_page (sp)) +kvm_tdp_mmu_put_root (kvm, sp);else if (!--sp->root_count && sp->role.invalid)kvm_mmu_prepare_zap_page (kvm, sp, invalid_list);
arch/x86/kvm/mmu/tdp_mmu.c +25 -29
73 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.h +1 -9
@@ -6,7 +6,6 @@#include <linux/kvm_host.h> hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu); -void kvm_tdp_mmu_free_root (struct kvm *kvm,struct kvm_mmu_page *root);static inline void kvm_tdp_mmu_get_root (struct kvm *kvm,struct kvm_mmu_page *root)
@@ -17,14 +16,7 @@ static inline void kvm_tdp_mmu_get_root(struct kvm *kvm, ++root->root_count; } -static inline bool kvm_tdp_mmu_put_root (struct kvm *kvm, -struct kvm_mmu_page *root) -{ -lockdep_assert_held (&kvm->mmu_lock); - --root->root_count; - -return !root->root_count; -} +void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root);bool __kvm_tdp_mmu_zap_gfn_range (struct kvm *kvm,int as_id,gfn_t start,gfn_t end,bool can_yield,bool flush);
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 5/13 KVM: x86/mmu: Refactor yield safe root iterator
In reply to Ben GardonRefactor the yield safe TDP MMU root iterator to be more amenable to changes in future commits which will allow it to be used under the MMU lock in read mode. Currently the iterator requires a complicated dance between the helper functions and different parts of the for loop which makes it hard to reason about. Moving all the logic into a single function simplifies the iterator substantially.
--- arch/x86/kvm/mmu/tdp_mmu.c | 45 ++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 19 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +26 -19
@@ -68,26 +68,34 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)tdp_mmu_free_sp (root); } -static inline bool tdp_mmu_next_root_valid (struct kvm *kvm, -struct kvm_mmu_page *root) +/* +* Finds the next valid root after root (or the first valid root if root +* is NULL), takes a reference on it, and returns that next root. If root +* is not NULL, this thread should have already taken a reference on it, and +* that reference will be dropped. If no valid root is found, this +* function will return NULL. +*/ +static struct kvm_mmu_page *tdp_mmu_next_root (struct kvm *kvm, +struct kvm_mmu_page *prev_root) { -lockdep_assert_held_write (&kvm->mmu_lock); +struct kvm_mmu_page *next_root; -if (list_entry_is_head (root, &kvm->arch.tdp_mmu_roots, link)) -return false; +lockdep_assert_held_write (&kvm->mmu_lock); -kvm_tdp_mmu_get_root (kvm, root); -return true; +if (prev_root) + next_root =list_next_entry (prev_root, link); +else + next_root =list_first_entry (&kvm->arch.tdp_mmu_roots, +typeof (*next_root), link); -} +if (list_entry_is_head (next_root, &kvm->arch.tdp_mmu_roots, link)) + next_root =NULL ; +else +kvm_tdp_mmu_get_root (kvm, next_root); -static inline struct kvm_mmu_page *tdp_mmu_next_root (struct kvm *kvm, -struct kvm_mmu_page *root) -{ -struct kvm_mmu_page *next_root; +if (prev_root) +kvm_tdp_mmu_put_root (kvm, prev_root); - next_root =list_next_entry (root, link); -kvm_tdp_mmu_put_root (kvm, root);return next_root; }
@@ -98,11 +106,10 @@ static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm,* recent root. (Unless keeping a live reference is desirable.) */ #define for_each_tdp_mmu_root_yield_safe (_kvm, _root, _as_id) \ - for (_root = list_first_entry(&_kvm->arch.tdp_mmu_roots, \ - typeof(*_root), link); \ - tdp_mmu_next_root_valid(_kvm, _root); \ - _root = tdp_mmu_next_root(_kvm, _root)) \ - if (kvm_mmu_page_as_id(_root) != _as_id) { \ + for (_root = tdp_mmu_next_root(_kvm, NULL); \ + _root; \ + _root = tdp_mmu_next_root(_kvm, _root)) \ + if (kvm_mmu_page_as_id(_root) != _as_id) { \ } else#define for_each_tdp_mmu_root (_kvm, _root, _as_id) \
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 6/13 KVM: x86/mmu: Make TDP MMU root refcount atomic
In reply to Ben GardonIn order to parallelize more operations for the TDP MMU, make the refcount on TDP MMU roots atomic, so that a future patch can allow multiple threads to take a reference on the root concurrently, while holding the MMU lock in read mode.
---
Changelog
v2:
-- Split failure handling for kvm_tdp_mmu_get_root out into a
seperate commit.
arch/x86/kvm/mmu/mmu_internal.h | 6 +++++- arch/x86/kvm/mmu/tdp_mmu.c | 4 ++-- arch/x86/kvm/mmu/tdp_mmu.h | 10 +++++++--- 3 files changed, 14 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/mmu_internal.h +5 -1
@@ -50,7 +50,11 @@ struct kvm_mmu_page {u64 *spt;/* hold the gfn of each spte inside spt */ gfn_t *gfns; -int root_count;/* Currently serving as active root */ +/* Currently serving as active root */ +union { +int root_count; +refcount_t tdp_mmu_root_count; + };unsigned int unsync_children;struct kvm_rmap_head parent_ptes;/* rmap pointers to parent sptes */ DECLARE_BITMAP (unsync_child_bitmap,512 );
arch/x86/kvm/mmu/tdp_mmu.c +2 -2
@@ -56,7 +56,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)lockdep_assert_held_write (&kvm->mmu_lock); -if (--root->root_count) +if (!refcount_dec_and_test (&root->tdp_mmu_root_count))return ;WARN_ON (!root->tdp_mmu_page);
@@ -168,7 +168,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) } root =alloc_tdp_mmu_page (vcpu,0 , vcpu->arch.mmu->shadow_root_level); - root->root_count =1 ; +refcount_set (&root->tdp_mmu_root_count,1 );list_add (&root->link, &kvm->arch.tdp_mmu_roots);
arch/x86/kvm/mmu/tdp_mmu.h +7 -3
@@ -10,10 +10,14 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);static inline void kvm_tdp_mmu_get_root (struct kvm *kvm,struct kvm_mmu_page *root) { -BUG_ON (!root->root_count); -lockdep_assert_held (&kvm->mmu_lock); +lockdep_assert_held_write (&kvm->mmu_lock); - ++root->root_count; +/* +* This should never fail since roots are removed from the roots +* list under the MMU write lock when their reference count falls +* to zero. +*/ +refcount_inc_not_zero (&root->tdp_mmu_root_count); }void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root);
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 7/13 KVM: x86/mmu: handle cmpxchg failure in kvm_tdp_mmu_get_root
In reply to Ben GardonTo reduce dependence on the MMU write lock, don't rely on the assumption that the atomic operation in kvm_tdp_mmu_get_root will always succeed. By not relying on that assumption, threads do not need to hold the MMU lock in write mode in order to take a reference on a TDP MMU root.
In the root iterator, this change means that some roots might have to be skipped if they are found to have a zero refcount. This will still never happen as of this patch, but a future patch will need that flexibility to make the root iterator safe under the MMU read lock.
--- arch/x86/kvm/mmu/tdp_mmu.c | 11 ++++++----- arch/x86/kvm/mmu/tdp_mmu.h | 13 +++---------- 2 files changed, 9 insertions(+), 15 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +6 -5
@@ -88,10 +88,12 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, next_root =list_first_entry (&kvm->arch.tdp_mmu_roots,typeof (*next_root), link); +while (!list_entry_is_head (next_root, &kvm->arch.tdp_mmu_roots, link) && + !kvm_tdp_mmu_get_root (kvm, next_root)) + next_root =list_next_entry (next_root, link); +if (list_entry_is_head (next_root, &kvm->arch.tdp_mmu_roots, link)) next_root =NULL ; -else -kvm_tdp_mmu_get_root (kvm, next_root);if (prev_root)kvm_tdp_mmu_put_root (kvm, prev_root);
@@ -161,10 +163,9 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu)/* Check for an existing root before allocating a new one. */ for_each_tdp_mmu_root (kvm, root,kvm_mmu_role_as_id (role)) { -if (root->role.word == role.word) { -kvm_tdp_mmu_get_root (kvm, root); +if (root->role.word == role.word && +kvm_tdp_mmu_get_root (kvm, root)) goto out; - } } root =alloc_tdp_mmu_page (vcpu,0 , vcpu->arch.mmu->shadow_root_level);
arch/x86/kvm/mmu/tdp_mmu.h +3 -10
@@ -7,17 +7,10 @@hpa_t kvm_tdp_mmu_get_vcpu_root_hpa (struct kvm_vcpu *vcpu); -static inline void kvm_tdp_mmu_get_root (struct kvm *kvm, -struct kvm_mmu_page *root) +__must_check static inline bool kvm_tdp_mmu_get_root (struct kvm *kvm, +struct kvm_mmu_page *root) { -lockdep_assert_held_write (&kvm->mmu_lock); - -/* -* This should never fail since roots are removed from the roots -* list under the MMU write lock when their reference count falls -* to zero. -*/ -refcount_inc_not_zero (&root->tdp_mmu_root_count); +return refcount_inc_not_zero (&root->tdp_mmu_root_count); }void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root);
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 8/13 KVM: x86/mmu: Protect the tdp_mmu_roots list with RCU
In reply to Ben GardonProtect the contents of the TDP MMU roots list with RCU in preparation for a future patch which will allow the iterator macro to be used under the MMU lock in read mode.
---
Changelog
v2:
-- add lockdep condition for tdp_mmu_pages_lock to for_each_tdp_mmu_root
-- fix problem with unexported lockdep function
-- updated comments in kvm_host.h
arch/x86/include/asm/kvm_host.h | 21 +++++++--- arch/x86/kvm/mmu/tdp_mmu.c | 69 +++++++++++++++++++-------------- 2 files changed, 55 insertions(+), 35 deletions(-)
arch/x86/include/asm/kvm_host.h +16 -5
@@ -1050,25 +1050,36 @@ struct kvm_arch {bool tdp_mmu_enabled;/* -* List of struct kvmp_mmu_pages being used as roots. +* List of struct kvm_mmu_pages being used as roots. * All struct kvm_mmu_pages in the list should have * tdp_mmu_page set. -* All struct kvm_mmu_pages in the list should have a positive -* root_count except when a thread holds the MMU lock and is removing -* an entry from the list. +* +* For reads, this list is protected by: +* the MMU lock in read mode + RCU or +* the MMU lock in write mode +* +* For writes, this list is protected by: +* the MMU lock in read mode + the tdp_mmu_pages_lock or +* the MMU lock in write mode +* +* Roots will remain in the list until their tdp_mmu_root_count +* drops to zero, at which point the thread that decremented the +* count to zero should removed the root from the list and clean +* it up, freeing the root after an RCU grace period. */ struct list_head tdp_mmu_roots;/* * List of struct kvmp_mmu_pages not being used as roots. * All struct kvm_mmu_pages in the list should have -* tdp_mmu_page set and a root_count of 0. +* tdp_mmu_page set and a tdp_mmu_root_count of 0. */ struct list_head tdp_mmu_pages;/* * Protects accesses to the following fields when the MMU lock * is held in read mode: +* - tdp_mmu_roots (above) * - tdp_mmu_pages (above) * - the link field of struct kvm_mmu_pages used by the TDP MMU * - lpage_disallowed_mmu_pages
arch/x86/kvm/mmu/tdp_mmu.c +39 -30
117 lines not on this page · raw message
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 9/13 KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
In reply to Ben GardonTo reduce lock contention and interference with page fault handlers, allow the TDP MMU function to zap a GFN range to operate under the MMU read lock.
--- arch/x86/kvm/mmu/mmu.c | 22 +++++--- arch/x86/kvm/mmu/tdp_mmu.c | 111 ++++++++++++++++++++++++++----------- arch/x86/kvm/mmu/tdp_mmu.h | 14 +++-- 3 files changed, 102 insertions(+), 45 deletions(-)
arch/x86/kvm/mmu/mmu.c +15 -7
@@ -3154,7 +3154,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, sp =to_shadow_page (*root_hpa &PT64_BASE_ADDR_MASK );if (is_tdp_mmu_page (sp)) -kvm_tdp_mmu_put_root (kvm, sp); +kvm_tdp_mmu_put_root (kvm, sp, false);else if (!--sp->root_count && sp->role.invalid)kvm_mmu_prepare_zap_page (kvm, sp, invalid_list);
@@ -5507,16 +5507,24 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) } } -if (is_tdp_mmu_enabled (kvm)) { -for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) - flush =kvm_tdp_mmu_zap_gfn_range (kvm, i, gfn_start, - gfn_end, flush); - } -if (flush)kvm_flush_remote_tlbs_with_address (kvm, gfn_start, gfn_end);write_unlock (&kvm->mmu_lock); + +if (is_tdp_mmu_enabled (kvm)) { + flush = false; + +read_lock (&kvm->mmu_lock); +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) + flush =kvm_tdp_mmu_zap_gfn_range (kvm, i, gfn_start, + gfn_end, flush, true); +if (flush) +kvm_flush_remote_tlbs_with_address (kvm, gfn_start, + gfn_end); + +read_unlock (&kvm->mmu_lock); + } } static bool slot_rmap_write_protect(struct kvm *kvm,
arch/x86/kvm/mmu/tdp_mmu.c +78 -33
262 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.h +9 -5
@@ -13,14 +13,18 @@ __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm,return refcount_inc_not_zero (&root->tdp_mmu_root_count); } -void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root); +void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root, +bool shared);bool __kvm_tdp_mmu_zap_gfn_range (struct kvm *kvm,int as_id,gfn_t start, -gfn_t end,bool can_yield,bool flush); +gfn_t end,bool can_yield,bool flush, +bool shared);static inline bool kvm_tdp_mmu_zap_gfn_range (struct kvm *kvm,int as_id, -gfn_t start,gfn_t end,bool flush) +gfn_t start,gfn_t end,bool flush, +bool shared) { -return __kvm_tdp_mmu_zap_gfn_range (kvm, as_id, start, end, true, flush); +return __kvm_tdp_mmu_zap_gfn_range (kvm, as_id, start, end, true, flush, + shared); }static inline bool kvm_tdp_mmu_zap_sp (struct kvm *kvm,struct kvm_mmu_page *sp) {
@@ -37,7 +41,7 @@ static inline bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp)*/ lockdep_assert_held_write (&kvm->mmu_lock);return __kvm_tdp_mmu_zap_gfn_range (kvm,kvm_mmu_page_as_id (sp), - sp->gfn, end, false, false); + sp->gfn, end, false, false, false); }void kvm_tdp_mmu_zap_all (struct kvm *kvm);
Signature · 1 line
2.31.0.208.g409f899ff0-goog
Paolo Bonzini
maintainerRe: [PATCH v2 09/13] KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
Quoted 11 lines · Ben Gardon
arch/x86/kvm/mmu/tdp_mmu.c +3
-1+void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root, +bool shared) {gfn_t max_gfn =1ULL << (shadow_phys_bits -PAGE_SHIFT ); -lockdep_assert_held_write (&kvm->mmu_lock); +kvm_lockdep_assert_mmu_lock_held (kvm, shared);if (!refcount_dec_and_test (&root->tdp_mmu_root_count))return ;
arch/x86/kvm/mmu/tdp_mmu.c void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) · line 97
Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1
-1@@ -81,7 +92,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)list_del_rcu (&root->link);spin_unlock (&kvm->arch.tdp_mmu_pages_lock); -zap_gfn_range (kvm, root,0 , max_gfn, false, false); +zap_gfn_range (kvm, root,0 , max_gfn, false, false, shared);call_rcu (&root->rcu_head, tdp_mmu_free_sp_rcu_callback);
Instead of patch 13, would it make sense to delay the zap_gfn_range and call_rcu to a work item (either unconditionally, or only if shared==false)? Then the zap_gfn_range would be able to yield and take the mmu_lock for read, similar to kvm_tdp_mmu_zap_invalidated_roots.
If done unconditionally, this would also allow removing the "shared" argument to kvm_tdp_mmu_put_root, tdp_mmu_next_root and for_each_tdp_mmu_root_yield_safe, so I would place that change before this patch.
Paolo
Re: [PATCH v2 09/13] KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
Quoted 26 lines · Paolo Bonzini
Quoted 11 lines · Ben Gardon
arch/x86/kvm/mmu/tdp_mmu.c +3
-1+void kvm_tdp_mmu_put_root (struct kvm *kvm,struct kvm_mmu_page *root, +bool shared) {gfn_t max_gfn =1ULL << (shadow_phys_bits -PAGE_SHIFT ); -lockdep_assert_held_write (&kvm->mmu_lock); +kvm_lockdep_assert_mmu_lock_held (kvm, shared);if (!refcount_dec_and_test (&root->tdp_mmu_root_count))return ;arch/x86/kvm/mmu/tdp_mmu.c void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) · line 97
Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1
-1@@ -81,7 +92,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)list_del_rcu (&root->link);spin_unlock (&kvm->arch.tdp_mmu_pages_lock); -zap_gfn_range (kvm, root,0 , max_gfn, false, false); +zap_gfn_range (kvm, root,0 , max_gfn, false, false, shared);call_rcu (&root->rcu_head, tdp_mmu_free_sp_rcu_callback);Instead of patch 13, would it make sense to delay the
zap_gfn_rangeandcall_rcuto a work item (either unconditionally, or only if shared==false)? Then thezap_gfn_rangewould be able to yield and take themmu_lockfor read, similar tokvm_tdp_mmu_zap_invalidated_roots.
Quoted 7 lines
If done unconditionally, this would also allow removing the "shared" argument to
kvm_tdp_mmu_put_root,tdp_mmu_next_rootandfor_each_tdp_mmu_root_yield_safe, so I would place that change before this patch.Paolo
I tried that and it created problems. I believe the issue was that on VM teardown memslots would be freed and the memory reallocated before the root was torn down, resulting in a use-after free from mark_pfn_dirty. Perhaps this could be resolved by forcing memslot changes to wait until that work item was processed before returning. I can look into it but I suspect there will be a lot of "gotchas" involved.
PATCH v2 10/13 KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
In reply to Ben GardonTo speed the process of disabling dirty logging, change the TDP MMU function which zaps collapsible SPTEs to run under the MMU read lock.
--- arch/x86/kvm/mmu/mmu.c | 14 ++++++++++---- arch/x86/kvm/mmu/tdp_mmu.c | 17 +++++++++++++---- 2 files changed, 23 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +10 -4
@@ -5612,13 +5612,19 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,write_lock (&kvm->mmu_lock); flush =slot_handle_leaf (kvm, slot, kvm_mmu_zap_collapsible_spte, true); -if (is_tdp_mmu_enabled (kvm)) - flush =kvm_tdp_mmu_zap_collapsible_sptes (kvm, slot, flush); -if (flush)kvm_arch_flush_remote_tlbs_memslot (kvm, slot); -write_unlock (&kvm->mmu_lock); + +if (is_tdp_mmu_enabled (kvm)) { + flush = false; + +read_lock (&kvm->mmu_lock); + flush =kvm_tdp_mmu_zap_collapsible_sptes (kvm, slot, flush); +if (flush) +kvm_arch_flush_remote_tlbs_memslot (kvm, slot); +read_unlock (&kvm->mmu_lock); + } } void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
arch/x86/kvm/mmu/tdp_mmu.c +13 -4
@@ -1323,7 +1323,8 @@ static bool zap_collapsible_spte_range(struct kvm *kvm,rcu_read_lock ();tdp_root_for_each_pte (iter, root, start, end) { -if (tdp_mmu_iter_cond_resched (kvm, &iter, flush, false)) { +retry: +if (tdp_mmu_iter_cond_resched (kvm, &iter, flush, true)) { flush = false;continue ; }
@@ -1338,8 +1339,14 @@ static bool zap_collapsible_spte_range(struct kvm *kvm, pfn, PG_LEVEL_NUM))continue ; -tdp_mmu_set_spte (kvm, &iter,0 ); - +if (!tdp_mmu_zap_spte_atomic (kvm, &iter)) { +/* +* The iter must explicitly re-read the SPTE because +* the atomic cmpxchg failed. +*/ + iter.old_spte =READ_ONCE (*rcu_dereference (iter.sptep)); + goto retry; + } flush = true; }
@@ -1358,7 +1365,9 @@ bool kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm, {struct kvm_mmu_page *root; -for_each_tdp_mmu_root_yield_safe (kvm, root, slot->as_id, false) +lockdep_assert_held_read (&kvm->mmu_lock); + +for_each_tdp_mmu_root_yield_safe (kvm, root, slot->as_id, true) flush =zap_collapsible_spte_range (kvm, root, slot, flush);return flush;
Signature · 1 line
2.31.0.208.g409f899ff0-goog
Paolo Bonzini
maintainerRe: [PATCH v2 10/13] KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
Quoted 2 lines · Ben Gardon
To speed the process of disabling dirty logging, change the TDP MMU function which zaps collapsible SPTEs to run under the MMU read lock.
Technically it only reduces the impact on the running VM; it doesn't speed it up right? Something like:
To reduce the impact of disabling dirty logging, change the TDP MMU function which zaps collapsible SPTEs to run under the MMU read lock. This way, page faults on zapped SPTEs can proceed in parallel with kvm_mmu_zap_collapsible_sptes.
Paolo
PATCH v2 11/13 KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read lock
In reply to Ben GardonTo reduce lock contention and interference with page fault handlers, allow the TDP MMU functions which enable and disable dirty logging to operate under the MMU read lock.
--- arch/x86/kvm/mmu/mmu.c | 16 +++++++--- arch/x86/kvm/mmu/tdp_mmu.c | 62 ++++++++++++++++++++++++++++++-------- 2 files changed, 61 insertions(+), 17 deletions(-)
arch/x86/kvm/mmu/mmu.c +12 -4
@@ -5543,10 +5543,14 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm,write_lock (&kvm->mmu_lock); flush =slot_handle_level (kvm, memslot, slot_rmap_write_protect, start_level,KVM_MAX_HUGEPAGE_LEVEL , false); -if (is_tdp_mmu_enabled (kvm)) - flush |=kvm_tdp_mmu_wrprot_slot (kvm, memslot,PG_LEVEL_4K );write_unlock (&kvm->mmu_lock); +if (is_tdp_mmu_enabled (kvm)) { +read_lock (&kvm->mmu_lock); + flush |=kvm_tdp_mmu_wrprot_slot (kvm, memslot,PG_LEVEL_4K ); +read_unlock (&kvm->mmu_lock); + } +/* * We can flush all the TLBs out of the mmu lock without TLB * corruption since we just change the spte from writable to
@@ -5649,10 +5653,14 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,write_lock (&kvm->mmu_lock); flush =slot_handle_leaf (kvm, memslot, __rmap_clear_dirty, false); -if (is_tdp_mmu_enabled (kvm)) - flush |=kvm_tdp_mmu_clear_dirty_slot (kvm, memslot);write_unlock (&kvm->mmu_lock); +if (is_tdp_mmu_enabled (kvm)) { +read_lock (&kvm->mmu_lock); + flush |=kvm_tdp_mmu_clear_dirty_slot (kvm, memslot); +read_unlock (&kvm->mmu_lock); + } +/* * It's also safe to flush TLBs out of mmu lock here as currently this * function is only used for dirty logging, in which case flushing TLB
arch/x86/kvm/mmu/tdp_mmu.c +49 -13
120 lines not on this page · raw message
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 12/13 KVM: x86/mmu: Fast invalidation for TDP MMU
In reply to Ben GardonProvide a real mechanism for fast invalidation by marking roots as invalid so that their reference count will quickly fall to zero and they will be torn down.
One negative side affect of this approach is that a vCPU thread will likely drop the last reference to a root and be saddled with the work of tearing down an entire paging structure. This issue will be resolved in a later commit.
---
Changelog
v2:
-- open code root invalidation
arch/x86/kvm/mmu/mmu.c | 26 +++++++++++++++++++++++--- arch/x86/kvm/mmu/tdp_mmu.h | 3 +++ 2 files changed, 26 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +23 -3
@@ -5418,6 +5418,8 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm)*/ static void kvm_mmu_zap_all_fast (struct kvm *kvm) { +struct kvm_mmu_page *root; +lockdep_assert_held (&kvm->slots_lock);write_lock (&kvm->mmu_lock);
@@ -5432,6 +5434,27 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)*/ kvm->arch.mmu_valid_gen = kvm->arch.mmu_valid_gen ?0 :1 ; + +if (is_tdp_mmu_enabled (kvm)) { +/* +* Mark each TDP MMU root as invalid so that other threads +* will drop their references and allow the root count to +* go to 0. +* +* This has essentially the same effect for the TDP MMU +* as updating mmu_valid_gen above does for the shadow +* MMU. +* +* In order to ensure all threads see this change when +* handling the MMU reload signal, this must happen in the +* same critical section as kvm_reload_remote_mmus, and +* before kvm_zap_obsolete_pages as kvm_zap_obsolete_pages +* could drop the MMU lock and yield. +*/ +list_for_each_entry (root, &kvm->arch.tdp_mmu_roots, link) + root->role.invalid = true; + } +/* * Notify all vcpus to reload its shadow page table and flush TLB. * Then all vcpus will switch to new shadow page table with the new
@@ -5444,9 +5467,6 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)kvm_zap_obsolete_pages (kvm); -if (is_tdp_mmu_enabled (kvm)) -kvm_tdp_mmu_zap_all (kvm); -write_unlock (&kvm->mmu_lock); }
arch/x86/kvm/mmu/tdp_mmu.h +3 -0
@@ -10,6 +10,9 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);__must_check static inline bool kvm_tdp_mmu_get_root (struct kvm *kvm,struct kvm_mmu_page *root) { +if (root->role.invalid) +return false; +return refcount_inc_not_zero (&root->tdp_mmu_root_count); }
Signature · 1 line
2.31.0.208.g409f899ff0-goog
PATCH v2 13/13 KVM: x86/mmu: Tear down roots in fast invalidation thread
In reply to Ben GardonTo avoid saddling a vCPU thread with the work of tearing down an entire paging structure, take a reference on each root before they become obsolete, so that the thread initiating the fast invalidation can tear down the paging structure and (most likely) release the last reference. As a bonus, this teardown can happen under the MMU lock in read mode so as not to block the progress of vCPU threads.
---
Changelog
v2:
-- rename kvm_tdp_mmu_zap_all_fast to
kvm_tdp_mmu_zap_invalidated_roots
arch/x86/kvm/mmu/mmu.c | 21 +++++++++++- arch/x86/kvm/mmu/tdp_mmu.c | 68 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.h | 1 + 3 files changed, 89 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +20 -1
@@ -5441,6 +5441,18 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)* will drop their references and allow the root count to * go to 0. * +* Also take a reference on all roots so that this thread +* can do the bulk of the work required to free the roots +* once they are invalidated. Without this reference, a +* vCPU thread might drop the last reference to a root and +* get stuck with tearing down the entire paging structure. +* +* Roots which have a zero refcount should be skipped as +* they're already being torn down. +* Already invalid roots should be referenced again so that +* they aren't freed before kvm_tdp_mmu_zap_all_fast is +* done with them. +* * This has essentially the same effect for the TDP MMU * as updating mmu_valid_gen above does for the shadow * MMU.
@@ -5452,7 +5464,8 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)* could drop the MMU lock and yield. */ list_for_each_entry (root, &kvm->arch.tdp_mmu_roots, link) - root->role.invalid = true; +if (refcount_inc_not_zero (&root->tdp_mmu_root_count)) + root->role.invalid = true; }/*
@@ -5468,6 +5481,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)kvm_zap_obsolete_pages (kvm);write_unlock (&kvm->mmu_lock); + +if (is_tdp_mmu_enabled (kvm)) { +read_lock (&kvm->mmu_lock); +kvm_tdp_mmu_zap_invalidated_roots (kvm); +read_unlock (&kvm->mmu_lock); + } }static bool kvm_has_zapped_obsolete_pages (struct kvm *kvm)
arch/x86/kvm/mmu/tdp_mmu.c +68 -0
74 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.h +1 -0
@@ -47,6 +47,7 @@ static inline bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp) sp->gfn, end, false, false, false); }void kvm_tdp_mmu_zap_all (struct kvm *kvm); +void kvm_tdp_mmu_zap_invalidated_roots (struct kvm *kvm); int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, int map_writable, int max_level, kvm_pfn_t pfn,
Signature · 1 line
2.31.0.208.g409f899ff0-goog
Quoted 44 lines · Ben Gardon
Now that the TDP MMU is able to handle page faults in parallel, it's a relatively small change to expand to other operations. This series allows zapping a range of GFNs, reclaiming collapsible SPTEs (when disabling dirty logging), and enabling dirty logging to all happen under the MMU lock in read mode.
This is partly a cleanup + rewrite of the last few patches of the parallel page faults series. I've incorporated feedback from Sean and Paolo, but the patches have changed so much that I'm sending this as a separate series.
Ran kvm-unit-tests + selftests on an SMP kernel + Intel Skylake, with the TDP MMU enabled and disabled. This series introduces no new failures or warnings.
I know this will conflict horribly with the patches from Sean's series which were just queued, and I'll send a v2 to fix those conflicts + address any feedback on this v1.
Changelog
v2:
-- Rebased patches on top of kvm/queue to incorporate Sean's recent
TLB flushing changes
-- Dropped patch 5: "KVM: x86/mmu: commentfor_each_tdp_mmu_root
requires MMU write lock" as the following patch to protect the roots list with RCU adds lockdep which makes the comment somewhat redundant.Ben Gardon (13):
KVM: x86/mmu: Re-add const qualifier in
kvm_tdp_mmu_zap_collapsible_sptes
KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU
KVM: x86/mmu: usetdp_mmu_free_spto free roots
KVM: x86/mmu: Merge TDP MMU put and free root
KVM: x86/mmu: Refactor yield safe root iterator
KVM: x86/mmu: Make TDP MMU root refcount atomic
KVM: x86/mmu: handle cmpxchg failure inkvm_tdp_mmu_get_root
KVM: x86/mmu: Protect thetdp_mmu_rootslist with RCU
KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read
lock
KVM: x86/mmu: Fast invalidation for TDP MMU
KVM: x86/mmu: Tear down roots in fast invalidation thread
Quoted 8 lines
arch/x86/include/asm/kvm_host.h | 21 +- arch/x86/kvm/mmu/mmu.c | 115 +++++++--- arch/x86/kvm/mmu/mmu_internal.h | 27 +-- arch/x86/kvm/mmu/tdp_mmu.c | 375 +++++++++++++++++++++++--------- arch/x86/kvm/mmu/tdp_mmu.h | 28 ++- include/linux/kvm_host.h | 2 +- 6 files changed, 407 insertions(+), 161 deletions(-)
Applied to kvm/mmu-notifier-queue, thanks.
Paolo
Sean Christopherson
maintainerQuoted 54 lines · Paolo Bonzini
Quoted 44 lines · Ben Gardon
Now that the TDP MMU is able to handle page faults in parallel, it's a relatively small change to expand to other operations. This series allows zapping a range of GFNs, reclaiming collapsible SPTEs (when disabling dirty logging), and enabling dirty logging to all happen under the MMU lock in read mode.
This is partly a cleanup + rewrite of the last few patches of the parallel page faults series. I've incorporated feedback from Sean and Paolo, but the patches have changed so much that I'm sending this as a separate series.
Ran kvm-unit-tests + selftests on an SMP kernel + Intel Skylake, with the TDP MMU enabled and disabled. This series introduces no new failures or warnings.
I know this will conflict horribly with the patches from Sean's series which were just queued, and I'll send a v2 to fix those conflicts + address any feedback on this v1.
Changelog
v2:
-- Rebased patches on top of kvm/queue to incorporate Sean's recent
TLB flushing changes
-- Dropped patch 5: "KVM: x86/mmu: commentfor_each_tdp_mmu_root
requires MMU write lock" as the following patch to protect the roots list with RCU adds lockdep which makes the comment somewhat redundant.Ben Gardon (13):
KVM: x86/mmu: Re-add const qualifier in
kvm_tdp_mmu_zap_collapsible_sptes
KVM: x86/mmu: Move kvm_mmu_(get|put)_root to TDP MMU
KVM: x86/mmu: usetdp_mmu_free_spto free roots
KVM: x86/mmu: Merge TDP MMU put and free root
KVM: x86/mmu: Refactor yield safe root iterator
KVM: x86/mmu: Make TDP MMU root refcount atomic
KVM: x86/mmu: handle cmpxchg failure inkvm_tdp_mmu_get_root
KVM: x86/mmu: Protect thetdp_mmu_rootslist with RCU
KVM: x86/mmu: Allow zap gfn range to operate under the mmu read lock
KVM: x86/mmu: Allow zapping collapsible SPTEs to use MMU read lock
KVM: x86/mmu: Allow enabling / disabling dirty logging under MMU read
lock
KVM: x86/mmu: Fast invalidation for TDP MMU
KVM: x86/mmu: Tear down roots in fast invalidation threadQuoted 8 lines
arch/x86/include/asm/kvm_host.h | 21 +- arch/x86/kvm/mmu/mmu.c | 115 +++++++--- arch/x86/kvm/mmu/mmu_internal.h | 27 +-- arch/x86/kvm/mmu/tdp_mmu.c | 375 +++++++++++++++++++++++--------- arch/x86/kvm/mmu/tdp_mmu.h | 28 ++- include/linux/kvm_host.h | 2 +- 6 files changed, 407 insertions(+), 161 deletions(-)
Quoted 1 line
Applied to kvm/mmu-notifier-queue, thanks.
What's the plan for kvm/mmu-notifier-queue? More specifically, are the hashes stable, i.e. will non-critical review feedback get squashed? I was finally getting around to reviewing this, but what's sitting in that branch doesn't appear to be exactly what's posted here. If the hashes are stable, I'll probably test and review functionality, but not do a thorough review.
Thanks!
Paolo Bonzini
maintainerQuoted 6 lines · Sean Christopherson
Quoted 1 line
Applied to kvm/mmu-notifier-queue, thanks.
What's the plan for kvm/mmu-notifier-queue? More specifically, are the hashes stable, i.e. will non-critical review feedback get squashed? I was finally getting around to reviewing this, but what's sitting in that branch doesn't appear to be exactly what's posted here. If the hashes are stable, I'll probably test and review functionality, but not do a thorough review.
It's all in 5.13 except for the lock elision patch, for which I was waiting for a review. I'll post that patch separately.
Thanks,
Paolo
Sean Christopherson
maintainerQuoted 10 lines · Paolo Bonzini
Quoted 6 lines · Sean Christopherson
Quoted 1 line
Applied to kvm/mmu-notifier-queue, thanks.
What's the plan for kvm/mmu-notifier-queue? More specifically, are the hashes stable, i.e. will non-critical review feedback get squashed? I was finally getting around to reviewing this, but what's sitting in that branch doesn't appear to be exactly what's posted here. If the hashes are stable, I'll probably test and review functionality, but not do a thorough review.
It's all in 5.13 except for the lock elision patch, for which I was waiting for a review. I'll post that patch separately.
Ha, stable indeed. Thanks!