Fix RCU warnings in TDP MMU
v2 cover Fix RCU warnings in TDP MMU
The Linux Test Robot found a few RCU warnings in the TDP MMU:
https://www.spinics.net/lists/kernel/msg3845500.html
https://www.spinics.net/lists/kernel/msg3845521.html
Fix these warnings and cleanup a hack in tdp_mmu_iter_cond_resched.
Tested by compiling as suggested in the test robot report and confirmed that the warnings go away with this series applied. Also ran kvm-unit-tests on an Intel Skylake machine with the TDP MMU enabled and confirmed that the series introduced no new failures.
Ben Gardon (4):
KVM: x86/mmu: Fix RCU usage in handle_removed_tdp_mmu_page
KVM: x86/mmu: Fix RCU usage when atomically zapping SPTEs
KVM: x86/mmu: Factor out tdp_iter_return_to_root
KVM: x86/mmu: Store the address space ID in the TDP iterator
arch/x86/kvm/mmu/mmu_internal.h | 5 +++++ arch/x86/kvm/mmu/tdp_iter.c | 30 +++++++++++++++---------- arch/x86/kvm/mmu/tdp_iter.h | 4 +++- arch/x86/kvm/mmu/tdp_mmu.c | 40 +++++++++++++-------------------- 4 files changed, 41 insertions(+), 38 deletions(-)
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
PATCH v2 1/4 KVM: x86/mmu: Fix RCU usage in handle_removed_tdp_mmu_page
The pt passed into handle_removed_tdp_mmu_page does not need RCU protection, as it is not at any risk of being freed by another thread at that point. However, the implicit cast from tdp_sptep_t to u64 * dropped the __rcu annotation without a proper rcu_derefrence. Fix this by passing the pt as a tdp_ptep_t and then rcu_dereferencing it in the function.
--- arch/x86/kvm/mmu/tdp_mmu.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +8 -3
@@ -301,11 +301,16 @@ static void tdp_mmu_unlink_page(struct kvm *kvm, struct kvm_mmu_page *sp,* * Given a page table that has been removed from the TDP paging structure, * iterates through the page table to clear SPTEs and free child page tables. +* +* Note that pt is passed in as a tdp_ptep_t, but it does not need RCU +* protection. Since this thread removed it from the paging structure, +* this thread will be responsible for ensuring the page is freed. Hence the +* early rcu_dereferences in the function. */ -static void handle_removed_tdp_mmu_page (struct kvm *kvm,u64 *pt, +static void handle_removed_tdp_mmu_page (struct kvm *kvm,tdp_ptep_t pt,bool shared) { -struct kvm_mmu_page *sp =sptep_to_sp (pt); +struct kvm_mmu_page *sp =sptep_to_sp (rcu_dereference (pt));int level = sp->role.level;gfn_t base_gfn = sp->gfn;u64 old_child_spte;
@@ -318,7 +323,7 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt,tdp_mmu_unlink_page (kvm, sp, shared);for (i =0 ; i <PT64_ENT_PER_PAGE ; i++) { - sptep = pt + i; + sptep =rcu_dereference (pt) + i; gfn = base_gfn + (i *KVM_PAGES_PER_HPAGE (level -1 ));if (shared) {
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
PATCH v2 2/4 KVM: x86/mmu: Fix RCU usage when atomically zapping SPTEs
In reply to Ben GardonFix a missing rcu_dereference in tdp_mmu_zap_spte_atomic.
--- arch/x86/kvm/mmu/tdp_mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
@@ -543,7 +543,7 @@ static inline bool tdp_mmu_zap_spte_atomic(struct kvm *kvm,* here since the SPTE is going from non-present * to non-present. */ -WRITE_ONCE (*iter->sptep,0 ); +WRITE_ONCE (*rcu_dereference (iter->sptep),0 );return true; }
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Re: [PATCH v2 2/4] KVM: x86/mmu: Fix RCU usage when atomically zapping SPTEs
Quoted 5 lines · Ben Gardon
Fix a missing
rcu_dereferenceintdp_mmu_zap_spte_atomic.Reported-by kernel test robot <[email protected]>Signed-off-by Ben Gardon <[email protected]>---
PATCH v2 3/4 KVM: x86/mmu: Factor out tdp_iter_return_to_root
In reply to Ben GardonIn tdp_mmu_iter_cond_resched there is a call to tdp_iter_start which causes the iterator to continue its walk over the paging structure from the root. This is needed after a yield as paging structure could have been freed in the interim.
The tdp_iter_start call is not very clear and something of a hack. It requires exposing tdp_iter fields not used elsewhere in tdp_mmu.c and the effect is not obvious from the function name. Factor a more aptly named function out of tdp_iter_start and call it from tdp_mmu_iter_cond_resched and tdp_iter_start.
No functional change intended.
--- arch/x86/kvm/mmu/tdp_iter.c | 24 +++++++++++++++++------- arch/x86/kvm/mmu/tdp_iter.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 4 +--- 3 files changed, 19 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/tdp_iter.c +17 -7
@@ -20,6 +20,21 @@ static gfn_t round_gfn_for_level(gfn_t gfn, int level)return gfn & -KVM_PAGES_PER_HPAGE (level); } +/* +* Return the TDP iterator to the root PT and allow it to continue its +* traversal over the paging structure from there. +*/ +void tdp_iter_restart (struct tdp_iter *iter) +{ + iter->yielded_gfn = iter->next_last_level_gfn; + iter->level = iter->root_level; + + iter->gfn =round_gfn_for_level (iter->next_last_level_gfn, iter->level); +tdp_iter_refresh_sptep (iter); + + iter->valid = true; +} +/* * Sets a TDP iterator to walk a pre-order traversal of the paging structure * rooted at root_pt, starting with the walk to translate next_last_level_gfn.
@@ -31,16 +46,11 @@ void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level,WARN_ON (root_level >PT64_ROOT_MAX_LEVEL ); iter->next_last_level_gfn = next_last_level_gfn; - iter->yielded_gfn = iter->next_last_level_gfn; iter->root_level = root_level; iter->min_level = min_level; - iter->level = root_level; - iter->pt_path[iter->level -1 ] = (tdp_ptep_t )root_pt; + iter->pt_path[iter->root_level -1 ] = (tdp_ptep_t )root_pt; - iter->gfn =round_gfn_for_level (iter->next_last_level_gfn, iter->level); -tdp_iter_refresh_sptep (iter); - - iter->valid = true; +tdp_iter_restart (iter); }/*
arch/x86/kvm/mmu/tdp_iter.h +1 -0
@@ -63,5 +63,6 @@ void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level, int min_level, gfn_t next_last_level_gfn);void tdp_iter_next (struct tdp_iter *iter);tdp_ptep_t tdp_iter_root_pt (struct tdp_iter *iter); +void tdp_iter_restart (struct tdp_iter *iter);#endif /* __KVM_X86_MMU_TDP_ITER_H */
arch/x86/kvm/mmu/tdp_mmu.c +1 -3
@@ -664,9 +664,7 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,WARN_ON (iter->gfn > iter->next_last_level_gfn); -tdp_iter_start (iter, iter->pt_path[iter->root_level -1 ], - iter->root_level, iter->min_level, - iter->next_last_level_gfn); +tdp_iter_restart (iter);return true; }
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Re: [PATCH v2 3/4] KVM: x86/mmu: Factor out tdp_iter_return_to_root
Quoted 11 lines · Ben Gardon
In
tdp_mmu_iter_cond_reschedthere is a call totdp_iter_startwhich causes the iterator to continue its walk over the paging structure from the root. This is needed after a yield as paging structure could have been freed in the interim.The
tdp_iter_startcall is not very clear and something of a hack. It requires exposingtdp_iterfields not used elsewhere intdp_mmu.cand the effect is not obvious from the function name. Factor a more aptly named function out oftdp_iter_startand call it fromtdp_mmu_iter_cond_reschedandtdp_iter_start.
Quoted 4 lines
No functional change intended.
Signed-off-by Ben Gardon <[email protected]>---
Very nice, sooo much easier to read.
PATCH v2 4/4 KVM: x86/mmu: Store the address space ID in the TDP iterator
In reply to Ben GardonStore the address space ID in the TDP iterator so that it can be retrieved without having to bounce through the root shadow page. This streamlines the code and fixes a Sparse warning about not properly using rcu_dereference() when grabbing the ID from the root on the fly.
--- arch/x86/kvm/mmu/mmu_internal.h | 5 +++++ arch/x86/kvm/mmu/tdp_iter.c | 6 +----- arch/x86/kvm/mmu/tdp_iter.h | 3 ++- arch/x86/kvm/mmu/tdp_mmu.c | 23 +++++------------------ 4 files changed, 13 insertions(+), 24 deletions(-)
arch/x86/kvm/mmu/mmu_internal.h +5 -0
@@ -78,6 +78,11 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep)return to_shadow_page (__pa (sptep)); } +static inline int kvm_mmu_page_as_id (struct kvm_mmu_page *sp) +{ +return sp->role.smm ?1 :0 ; +} +static inline bool kvm_vcpu_ad_need_write_protect (struct kvm_vcpu *vcpu) {/*
arch/x86/kvm/mmu/tdp_iter.c +1 -5
@@ -49,6 +49,7 @@ void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level, iter->root_level = root_level; iter->min_level = min_level; iter->pt_path[iter->root_level -1 ] = (tdp_ptep_t )root_pt; + iter->as_id =kvm_mmu_page_as_id (sptep_to_sp (root_pt));tdp_iter_restart (iter); }
@@ -169,8 +170,3 @@ void tdp_iter_next(struct tdp_iter *iter) iter->valid = false; } -tdp_ptep_t tdp_iter_root_pt (struct tdp_iter *iter) -{ -return iter->pt_path[iter->root_level -1 ]; -} -
arch/x86/kvm/mmu/tdp_iter.h +2 -1
@@ -36,6 +36,8 @@ struct tdp_iter {int min_level;/* The iterator's current level within the paging structure */ int level; +/* The address space ID, i.e. SMM vs. regular. */ +int as_id;/* A snapshot of the value at sptep */ u64 old_spte;/*
@@ -62,7 +64,6 @@ tdp_ptep_t spte_to_child_pt(u64 pte, int level);void tdp_iter_start (struct tdp_iter *iter,u64 *root_pt,int root_level,int min_level,gfn_t next_last_level_gfn);void tdp_iter_next (struct tdp_iter *iter); -tdp_ptep_t tdp_iter_root_pt (struct tdp_iter *iter);void tdp_iter_restart (struct tdp_iter *iter);#endif /* __KVM_X86_MMU_TDP_ITER_H */
arch/x86/kvm/mmu/tdp_mmu.c +5 -18
@@ -203,11 +203,6 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, u64 old_spte, u64 new_spte, int level, bool shared); -static int kvm_mmu_page_as_id (struct kvm_mmu_page *sp) -{ -return sp->role.smm ?1 :0 ; -} -static void handle_changed_spte_acc_track (u64 old_spte,u64 new_spte,int level) {bool pfn_changed =spte_to_pfn (old_spte) !=spte_to_pfn (new_spte);
@@ -497,10 +492,6 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { -u64 *root_pt =tdp_iter_root_pt (iter); -struct kvm_mmu_page *root =sptep_to_sp (root_pt); -int as_id =kvm_mmu_page_as_id (root); -lockdep_assert_held_read (&kvm->mmu_lock);/*
@@ -514,8 +505,8 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, new_spte) != iter->old_spte)return false; -handle_changed_spte (kvm, as_id, iter->gfn, iter->old_spte, new_spte, - iter->level, true); +handle_changed_spte (kvm, iter->as_id, iter->gfn, iter->old_spte, + new_spte, iter->level, true);return true; }
@@ -569,10 +560,6 @@ static inline void __tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte, bool record_acc_track, bool record_dirty_log) { - tdp_ptep_t root_pt = tdp_iter_root_pt(iter); -struct kvm_mmu_page *root =sptep_to_sp (root_pt); -int as_id =kvm_mmu_page_as_id (root); - lockdep_assert_held_write(&kvm->mmu_lock);/*
@@ -586,13 +573,13 @@ static inline void __tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter,WRITE_ONCE (*rcu_dereference (iter->sptep), new_spte); -__handle_changed_spte (kvm, as_id, iter->gfn, iter->old_spte, new_spte, - iter->level, false); +__handle_changed_spte (kvm, iter->as_id, iter->gfn, iter->old_spte, + new_spte, iter->level, false);if (record_acc_track)handle_changed_spte_acc_track (iter->old_spte, new_spte, iter->level);if (record_dirty_log) -handle_changed_spte_dirty_log (kvm, as_id, iter->gfn, +handle_changed_spte_dirty_log (kvm, iter->as_id, iter->gfn, iter->old_spte, new_spte, iter->level); }
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Sean Christopherson
maintainerRe: [PATCH v2 4/4] KVM: x86/mmu: Store the address space ID in the TDP iterator
On Mon, Mar 15, 2021, Ben Gardon wrote:
Missing "From: Sean Christopherson <[email protected]>", i.e. the commit in your local tree needs "git commit --amend --author="Sean Christopherson <[email protected]>". Alternatively, you could just erase my SOB ;-)
Quoted 9 lines
Store the address space ID in the TDP iterator so that it can be retrieved without having to bounce through the root shadow page. This streamlines the code and fixes a Sparse warning about not properly using
rcu_dereference()when grabbing the ID from the root on the fly.Reported-by kernel test robot <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Ben Gardon <[email protected]>---