Fix RCU warnings in TDP MMU
v3 cover Fix RCU warnings in TDP MMU
The Linux Test Robot found a few RCU warnings in the TDP MMU:
https://www.spinics.net/lists/kernel/msg3845500.html
https://www.spinics.net/lists/kernel/msg3845521.html
Fix these warnings and cleanup a hack in tdp_mmu_iter_cond_resched.
Tested by compiling as suggested in the test robot report and confirmed that the warnings go away with this series applied. Also ran kvm-unit-tests on an Intel Skylake machine with the TDP MMU enabled and confirmed that the series introduced no new failures.
Ben Gardon (3):
KVM: x86/mmu: Fix RCU usage in handle_removed_tdp_mmu_page
KVM: x86/mmu: Fix RCU usage when atomically zapping SPTEs
KVM: x86/mmu: Factor out tdp_iter_return_to_root
Sean Christopherson (1):
KVM: x86/mmu: Store the address space ID in the TDP iterator
arch/x86/kvm/mmu/mmu_internal.h | 5 +++++ arch/x86/kvm/mmu/tdp_iter.c | 30 +++++++++++++++---------- arch/x86/kvm/mmu/tdp_iter.h | 4 +++- arch/x86/kvm/mmu/tdp_mmu.c | 40 +++++++++++++-------------------- 4 files changed, 41 insertions(+), 38 deletions(-)
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
PATCH v3 1/4 KVM: x86/mmu: Fix RCU usage in handle_removed_tdp_mmu_page
The pt passed into handle_removed_tdp_mmu_page does not need RCU protection, as it is not at any risk of being freed by another thread at that point. However, the implicit cast from tdp_sptep_t to u64 * dropped the __rcu annotation without a proper rcu_derefrence. Fix this by passing the pt as a tdp_ptep_t and then rcu_dereferencing it in the function.
--- arch/x86/kvm/mmu/tdp_mmu.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +8 -3
@@ -301,11 +301,16 @@ static void tdp_mmu_unlink_page(struct kvm *kvm, struct kvm_mmu_page *sp,* * Given a page table that has been removed from the TDP paging structure, * iterates through the page table to clear SPTEs and free child page tables. +* +* Note that pt is passed in as a tdp_ptep_t, but it does not need RCU +* protection. Since this thread removed it from the paging structure, +* this thread will be responsible for ensuring the page is freed. Hence the +* early rcu_dereferences in the function. */ -static void handle_removed_tdp_mmu_page (struct kvm *kvm,u64 *pt, +static void handle_removed_tdp_mmu_page (struct kvm *kvm,tdp_ptep_t pt,bool shared) { -struct kvm_mmu_page *sp =sptep_to_sp (pt); +struct kvm_mmu_page *sp =sptep_to_sp (rcu_dereference (pt));int level = sp->role.level;gfn_t base_gfn = sp->gfn;u64 old_child_spte;
@@ -318,7 +323,7 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt,tdp_mmu_unlink_page (kvm, sp, shared);for (i =0 ; i <PT64_ENT_PER_PAGE ; i++) { - sptep = pt + i; + sptep =rcu_dereference (pt) + i; gfn = base_gfn + (i *KVM_PAGES_PER_HPAGE (level -1 ));if (shared) {
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
PATCH v3 2/4 KVM: x86/mmu: Fix RCU usage when atomically zapping SPTEs
In reply to Ben GardonFix a missing rcu_dereference in tdp_mmu_zap_spte_atomic.
--- arch/x86/kvm/mmu/tdp_mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
@@ -543,7 +543,7 @@ static inline bool tdp_mmu_zap_spte_atomic(struct kvm *kvm,* here since the SPTE is going from non-present * to non-present. */ -WRITE_ONCE (*iter->sptep,0 ); +WRITE_ONCE (*rcu_dereference (iter->sptep),0 );return true; }
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Re: [PATCH v3 2/4] KVM: x86/mmu: Fix RCU usage when atomically zapping SPTEs
Quoted 5 lines · Ben Gardon
Fix a missing
rcu_dereferenceintdp_mmu_zap_spte_atomic.Reported-by kernel test robot <[email protected]>Signed-off-by Ben Gardon <[email protected]>---
PATCH v3 3/4 KVM: x86/mmu: Factor out tdp_iter_return_to_root
In reply to Ben GardonIn tdp_mmu_iter_cond_resched there is a call to tdp_iter_start which causes the iterator to continue its walk over the paging structure from the root. This is needed after a yield as paging structure could have been freed in the interim.
The tdp_iter_start call is not very clear and something of a hack. It requires exposing tdp_iter fields not used elsewhere in tdp_mmu.c and the effect is not obvious from the function name. Factor a more aptly named function out of tdp_iter_start and call it from tdp_mmu_iter_cond_resched and tdp_iter_start.
No functional change intended.
--- arch/x86/kvm/mmu/tdp_iter.c | 24 +++++++++++++++++------- arch/x86/kvm/mmu/tdp_iter.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 4 +--- 3 files changed, 19 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/tdp_iter.c +17 -7
@@ -20,6 +20,21 @@ static gfn_t round_gfn_for_level(gfn_t gfn, int level)return gfn & -KVM_PAGES_PER_HPAGE (level); } +/* +* Return the TDP iterator to the root PT and allow it to continue its +* traversal over the paging structure from there. +*/ +void tdp_iter_restart (struct tdp_iter *iter) +{ + iter->yielded_gfn = iter->next_last_level_gfn; + iter->level = iter->root_level; + + iter->gfn =round_gfn_for_level (iter->next_last_level_gfn, iter->level); +tdp_iter_refresh_sptep (iter); + + iter->valid = true; +} +/* * Sets a TDP iterator to walk a pre-order traversal of the paging structure * rooted at root_pt, starting with the walk to translate next_last_level_gfn.
@@ -31,16 +46,11 @@ void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level,WARN_ON (root_level >PT64_ROOT_MAX_LEVEL ); iter->next_last_level_gfn = next_last_level_gfn; - iter->yielded_gfn = iter->next_last_level_gfn; iter->root_level = root_level; iter->min_level = min_level; - iter->level = root_level; - iter->pt_path[iter->level -1 ] = (tdp_ptep_t )root_pt; + iter->pt_path[iter->root_level -1 ] = (tdp_ptep_t )root_pt; - iter->gfn =round_gfn_for_level (iter->next_last_level_gfn, iter->level); -tdp_iter_refresh_sptep (iter); - - iter->valid = true; +tdp_iter_restart (iter); }/*
arch/x86/kvm/mmu/tdp_iter.h +1 -0
@@ -63,5 +63,6 @@ void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level, int min_level, gfn_t next_last_level_gfn);void tdp_iter_next (struct tdp_iter *iter);tdp_ptep_t tdp_iter_root_pt (struct tdp_iter *iter); +void tdp_iter_restart (struct tdp_iter *iter);#endif /* __KVM_X86_MMU_TDP_ITER_H */
arch/x86/kvm/mmu/tdp_mmu.c +1 -3
@@ -664,9 +664,7 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,WARN_ON (iter->gfn > iter->next_last_level_gfn); -tdp_iter_start (iter, iter->pt_path[iter->root_level -1 ], - iter->root_level, iter->min_level, - iter->next_last_level_gfn); +tdp_iter_restart (iter);return true; }
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Re: [PATCH v3 3/4] KVM: x86/mmu: Factor out tdp_iter_return_to_root
Quoted 1 line · Ben Gardon
No functional change intended.
I want royalties.
Quoted 2 lines
Signed-off-by Ben Gardon <[email protected]>---
PATCH v3 4/4 KVM: x86/mmu: Store the address space ID in the TDP iterator
In reply to Ben GardonFrom: Sean Christopherson <[email protected]>
Store the address space ID in the TDP iterator so that it can be retrieved without having to bounce through the root shadow page. This streamlines the code and fixes a Sparse warning about not properly using rcu_dereference() when grabbing the ID from the root on the fly.
--- arch/x86/kvm/mmu/mmu_internal.h | 5 +++++ arch/x86/kvm/mmu/tdp_iter.c | 6 +----- arch/x86/kvm/mmu/tdp_iter.h | 3 ++- arch/x86/kvm/mmu/tdp_mmu.c | 23 +++++------------------ 4 files changed, 13 insertions(+), 24 deletions(-)
arch/x86/kvm/mmu/mmu_internal.h +5 -0
@@ -78,6 +78,11 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep)return to_shadow_page (__pa (sptep)); } +static inline int kvm_mmu_page_as_id (struct kvm_mmu_page *sp) +{ +return sp->role.smm ?1 :0 ; +} +static inline bool kvm_vcpu_ad_need_write_protect (struct kvm_vcpu *vcpu) {/*
arch/x86/kvm/mmu/tdp_iter.c +1 -5
@@ -49,6 +49,7 @@ void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level, iter->root_level = root_level; iter->min_level = min_level; iter->pt_path[iter->root_level -1 ] = (tdp_ptep_t )root_pt; + iter->as_id =kvm_mmu_page_as_id (sptep_to_sp (root_pt));tdp_iter_restart (iter); }
@@ -169,8 +170,3 @@ void tdp_iter_next(struct tdp_iter *iter) iter->valid = false; } -tdp_ptep_t tdp_iter_root_pt (struct tdp_iter *iter) -{ -return iter->pt_path[iter->root_level -1 ]; -} -
arch/x86/kvm/mmu/tdp_iter.h +2 -1
@@ -36,6 +36,8 @@ struct tdp_iter {int min_level;/* The iterator's current level within the paging structure */ int level; +/* The address space ID, i.e. SMM vs. regular. */ +int as_id;/* A snapshot of the value at sptep */ u64 old_spte;/*
@@ -62,7 +64,6 @@ tdp_ptep_t spte_to_child_pt(u64 pte, int level);void tdp_iter_start (struct tdp_iter *iter,u64 *root_pt,int root_level,int min_level,gfn_t next_last_level_gfn);void tdp_iter_next (struct tdp_iter *iter); -tdp_ptep_t tdp_iter_root_pt (struct tdp_iter *iter);void tdp_iter_restart (struct tdp_iter *iter);#endif /* __KVM_X86_MMU_TDP_ITER_H */
arch/x86/kvm/mmu/tdp_mmu.c +5 -18
@@ -203,11 +203,6 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, u64 old_spte, u64 new_spte, int level, bool shared); -static int kvm_mmu_page_as_id (struct kvm_mmu_page *sp) -{ -return sp->role.smm ?1 :0 ; -} -static void handle_changed_spte_acc_track (u64 old_spte,u64 new_spte,int level) {bool pfn_changed =spte_to_pfn (old_spte) !=spte_to_pfn (new_spte);
@@ -497,10 +492,6 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { -u64 *root_pt =tdp_iter_root_pt (iter); -struct kvm_mmu_page *root =sptep_to_sp (root_pt); -int as_id =kvm_mmu_page_as_id (root); -lockdep_assert_held_read (&kvm->mmu_lock);/*
@@ -514,8 +505,8 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, new_spte) != iter->old_spte)return false; -handle_changed_spte (kvm, as_id, iter->gfn, iter->old_spte, new_spte, - iter->level, true); +handle_changed_spte (kvm, iter->as_id, iter->gfn, iter->old_spte, + new_spte, iter->level, true);return true; }
@@ -569,10 +560,6 @@ static inline void __tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte, bool record_acc_track, bool record_dirty_log) { - tdp_ptep_t root_pt = tdp_iter_root_pt(iter); -struct kvm_mmu_page *root =sptep_to_sp (root_pt); -int as_id =kvm_mmu_page_as_id (root); - lockdep_assert_held_write(&kvm->mmu_lock);/*
@@ -586,13 +573,13 @@ static inline void __tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter,WRITE_ONCE (*rcu_dereference (iter->sptep), new_spte); -__handle_changed_spte (kvm, as_id, iter->gfn, iter->old_spte, new_spte, - iter->level, false); +__handle_changed_spte (kvm, iter->as_id, iter->gfn, iter->old_spte, + new_spte, iter->level, false);if (record_acc_track)handle_changed_spte_acc_track (iter->old_spte, new_spte, iter->level);if (record_dirty_log) -handle_changed_spte_dirty_log (kvm, as_id, iter->gfn, +handle_changed_spte_dirty_log (kvm, iter->as_id, iter->gfn, iter->old_spte, new_spte, iter->level); }
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Quoted 16 lines · Ben Gardon
The Linux Test Robot found a few RCU warnings in the TDP MMU:
https://www.spinics.net/lists/kernel/msg3845500.html
https://www.spinics.net/lists/kernel/msg3845521.htmlFix these warnings and cleanup a hack in
tdp_mmu_iter_cond_resched.Tested by compiling as suggested in the test robot report and confirmed that the warnings go away with this series applied. Also ran kvm-unit-tests on an Intel Skylake machine with the TDP MMU enabled and confirmed that the series introduced no new failures.
Ben Gardon (3):
KVM: x86/mmu: Fix RCU usage inhandle_removed_tdp_mmu_page
KVM: x86/mmu: Fix RCU usage when atomically zapping SPTEs
KVM: x86/mmu: Factor outtdp_iter_return_to_root
Quoted 9 lines
Sean Christopherson (1):
KVM: x86/mmu: Store the address space ID in the TDP iteratorarch/x86/kvm/mmu/mmu_internal.h | 5 +++++ arch/x86/kvm/mmu/tdp_iter.c | 30 +++++++++++++++---------- arch/x86/kvm/mmu/tdp_iter.h | 4 +++- arch/x86/kvm/mmu/tdp_mmu.c | 40 +++++++++++++-------------------- 4 files changed, 41 insertions(+), 38 deletions(-)
Queued, thanks.
Paolo