optimize spte zapping in zap_gfn_range()
cover optimize spte zapping in zap_gfn_range()
TDP MMU SPTE zapping process currently uses two levels of iterations. The first level iteration happens at the for loop within the zap_gfn_range() with the purpose of calibrating the accurate range for zapping. The second level itreration start at tdp_mmu_set_spte{,_atomic}() that tears down the whole paging structures (leaf and non-leaf SPTEs) within the range. The former iteration is yield safe, while the second one is not.
In many cases, zapping SPTE process could be optimized since the non-leaf SPTEs could most likely be retained for the next allocation. On the other hand, for large scale SPTE zapping scenarios, we may end up zapping too many SPTEs and use excessive CPU time that causes the RCU stall warning.
The follow selftest reproduces the warning:
(env: kvm.tdp_mmu=Y)
./dirty_log_perf_test -v 64 -b 8GThis patch set revert a previous optimization and create a helper __zap_gfn_range() to help optimize the zapping process.
In particular, it does the following two things:
- optimize the zapping by retaining some non-leaf SPTEs.
- avoid RCU stall warning when zapping too many SPTEs.
Mingwei Zhang (2):
Revert "KVM: x86/mmu: Don't step down in the TDP iterator when zapping
all SPTEs"
KVM: mmu/x86: optimize zapping by retaining non-leaf SPTEs and avoid
rcu stall
arch/x86/kvm/mmu/tdp_mmu.c | 66 +++++++++++++++++++++++--------------- 1 file changed, 41 insertions(+), 25 deletions(-)
--
2.34.0.rc2.393.gf8c9666880-goog
PATCH 1/2 Revert "KVM: x86/mmu: Don't step down in the TDP iterator when zapping all SPTEs"
Not stepping down in TDP iterator in zap_all case avoids re-reading the non-leaf SPTEs, thus accelerates the zapping process . But when the number of SPTEs is too large, we may run out of CPU time and causes a RCU stall warnings in __handle_changed_pte() in the context of zap_gfn_range().
Revert this patch to allow eliminating RCU stall warning using a two-phase zapping for zap_all case.
This reverts commit 0103098fb4f13b447b26ed514bcd3140f6791047.
--- arch/x86/kvm/mmu/tdp_mmu.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +1 -8
@@ -706,12 +706,6 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,bool zap_all = (start ==0 && end >= max_gfn_host);struct tdp_iter iter; -/* -* No need to try to step down in the iterator when zapping all SPTEs, -* zapping the top-level non-leaf SPTEs will recurse on their children. -*/ -int min_level = zap_all ? root->role.level :PG_LEVEL_4K ; -/* * Bound the walk at host.MAXPHYADDR, guest accesses beyond that will * hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF,
@@ -723,8 +717,7 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,rcu_read_lock (); -for_each_tdp_pte_min_level (iter, root->spt, root->role.level, - min_level, start, end) { +tdp_root_for_each_pte (iter, root, start, end) { retry:if (can_yield &&tdp_mmu_iter_cond_resched (kvm, &iter, flush, shared)) {
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Re: [PATCH 1/2] Revert "KVM: x86/mmu: Don't step down in the TDP iterator when zapping all SPTEs"
Quoted 8 lines · Mingwei Zhang
Not stepping down in TDP iterator in
zap_allcase avoids re-reading the non-leaf SPTEs, thus accelerates the zapping process . But when the number of SPTEs is too large, we may run out of CPU time and causes a RCU stall warnings in__handle_changed_pte()in the context ofzap_gfn_range().Revert this patch to allow eliminating RCU stall warning using a two-phase zapping for
zap_allcase.
Quoted 7 lines
This reverts commit
0103098fb4f13b447b26ed514bcd3140f6791047.Cc Sean Christopherson <[email protected]>Cc Ben Gardon <[email protected]>Cc David Matlack <[email protected]>Signed-off-by Mingwei Zhang <[email protected]>
Quoted 34 lines
--- arch/x86/kvm/mmu/tdp_mmu.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)arch/x86/kvm/mmu/tdp_mmu.c +1
-8@@ -706,12 +706,6 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,bool zap_all = (start ==0 && end >= max_gfn_host);struct tdp_iter iter; -/* -* No need to try to step down in the iterator when zapping all SPTEs, -* zapping the top-level non-leaf SPTEs will recurse on their children. -*/ -int min_level = zap_all ? root->role.level :PG_LEVEL_4K ; -/* * Bound the walk at host.MAXPHYADDR, guest accesses beyond that will * hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF, @@ -723,8 +717,7 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,rcu_read_lock (); -for_each_tdp_pte_min_level (iter, root->spt, root->role.level, - min_level, start, end) { +tdp_root_for_each_pte (iter, root, start, end) { retry:if (can_yield &&tdp_mmu_iter_cond_resched (kvm, &iter, flush, shared)) {Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
PATCH 2/2 KVM: mmu/x86: optimize zapping by retaining non-leaf SPTEs and avoid rcu stall
In reply to Mingwei ZhangTDP MMU SPTE zapping process currently uses two levels of iterations. The first level iteration happens at the for loop within the zap_gfn_range() with the purpose of calibrating the accurate range for zapping. The second level itreration start at tdp_mmu_set_spte{,_atomic}() that tears down the whole paging structures (leaf and non-leaf SPTEs) within the range. The former iteration is yield safe, while the second one is not.
In many cases, zapping SPTE process could be optimized since the non-leaf SPTEs could most likely be retained for the next allocation. On the other hand, for large scale SPTE zapping scenarios, we may end up zapping too many SPTEs and use excessive CPU time that causes the RCU stall warning.
The follow selftest reproduces the warning:
(env: kvm.tdp_mmu=Y) ./dirty_log_perf_test -v 64 -b 8G
Optimize the zapping process by skipping all SPTEs above a certain level in the first iteration. This allows us to control the granularity of the actual zapping and invoke tdp_mmu_iter_cond_resched() on time. In addition, we would retain some of the non-leaf SPTEs to accelerate next allocation.
For the selection of the certain level, we choose the PG_LEVEL_1G because it is currently the largest page size supported and it natually fits the scenario of splitting large pages.
For zap_all case (usually) at VM teardown time, we use a two-phase
mechanism: the 1st phase zaps all SPTEs at PG_LEVEL_1G level and 2nd phase zaps everything else. This is achieved by the helper function __zap_gfn_range().
--- arch/x86/kvm/mmu/tdp_mmu.c | 57 ++++++++++++++++++++++++++------------ 1 file changed, 40 insertions(+), 17 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +40 -17
@@ -697,24 +697,16 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,* account for the possibility that other threads are modifying the paging * structures concurrently. If shared is false, this thread should hold the * MMU lock in write mode. +* +* If zap_all is true, eliminate all the paging structures that contains the +* SPTEs. */ -static bool zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, -gfn_t start,gfn_t end,bool can_yield,bool flush, -bool shared) +static bool __zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool shared,bool zap_all) { -gfn_t max_gfn_host =1ULL << (shadow_phys_bits -PAGE_SHIFT ); -bool zap_all = (start ==0 && end >= max_gfn_host);struct tdp_iter iter; -/* -* Bound the walk at host.MAXPHYADDR, guest accesses beyond that will -* hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF, -* and so KVM will never install a SPTE for such addresses. -*/ - end =min (end, max_gfn_host); - -kvm_lockdep_assert_mmu_lock_held (kvm, shared); -rcu_read_lock ();tdp_root_for_each_pte (iter, root, start, end) {
@@ -725,17 +717,24 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,continue ; } -if (!is_shadow_present_pte (iter.old_spte)) +/* +* In zap_all case, ignore the checking of present since we have +* to zap everything. +*/ +if (!zap_all && !is_shadow_present_pte (iter.old_spte))continue ;/* * If this is a non-last-level SPTE that covers a larger range * than should be zapped, continue, and zap the mappings at a -* lower level, except when zapping all SPTEs. +* lower level. Actual zapping started at proper granularity +* that is not so large as to cause a soft lockup when handling +* the changed pte (which does not yield). */ if (!zap_all && (iter.gfn < start || - iter.gfn +KVM_PAGES_PER_HPAGE (iter.level) > end) && + iter.gfn +KVM_PAGES_PER_HPAGE (iter.level) > end || + iter.level >PG_LEVEL_1G ) && !is_last_spte (iter.old_spte, iter.level))continue ;
@@ -756,6 +755,30 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,return flush; } +static bool zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool shared) +{ +gfn_t max_gfn_host =1ULL << (shadow_phys_bits -PAGE_SHIFT ); +bool zap_all = (start ==0 && end >= max_gfn_host); + +/* +* Bound the walk at host.MAXPHYADDR, guest accesses beyond that will +* hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF, +* and so KVM will never install a SPTE for such addresses. +*/ + end =min (end, max_gfn_host); + +kvm_lockdep_assert_mmu_lock_held (kvm, shared); + + flush =__zap_gfn_range (kvm, root, start, end, can_yield, flush, shared, + false); +if (zap_all) + flush =__zap_gfn_range (kvm, root, start, end, can_yield, flush, + shared, true); +return flush; +} +/* * Tears down the mappings for the range of gfns, [start, end), and frees the * non-root pages mapping GFNs strictly within that range. Returns true if
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Re: [PATCH 2/2] KVM: mmu/x86: optimize zapping by retaining non-leaf SPTEs and avoid rcu stall
Quoted 4 lines · Mingwei Zhang
TDP MMU SPTE zapping process currently uses two levels of iterations. The first level iteration happens at the for loop within the
zap_gfn_range()with the purpose of calibrating the accurate range for zapping. The second level itreration start at tdp_mmu_set_spte{,_atomic}() that tears down the
iteration
Quoted 2 lines
whole paging structures (leaf and non-leaf SPTEs) within the range. The former iteration is yield safe, while the second one is not.
I know what you mean but I'd suggest being more specific than "yield safe". For example:
Unlike the outer loop, the recursive zapping done under tdp_mmu_set_spte{,_atomic} does not yield. Since zapping is done with a pre-order traversal, zapping sufficiently large ranges can lead to RCU stall warnings.
I'd also clarify here that the TDP MMU iterator uses a pre-order traversal which causes us KVM to end up doing the maximum amount of zapping under tdp_mmu_set_spte{,_atomic} and not the outer for loop.
Quoted 70 lines
In many cases, zapping SPTE process could be optimized since the non-leaf SPTEs could most likely be retained for the next allocation. On the other hand, for large scale SPTE zapping scenarios, we may end up zapping too many SPTEs and use excessive CPU time that causes the RCU stall warning.
The follow selftest reproduces the warning:
(env: kvm.tdp_mmu=Y) ./dirty_log_perf_test -v 64 -b 8GOptimize the zapping process by skipping all SPTEs above a certain level in the first iteration. This allows us to control the granularity of the actual zapping and invoke
tdp_mmu_iter_cond_resched()on time. In addition, we would retain some of the non-leaf SPTEs to accelerate next allocation.For the selection of the
certain level, we choose thePG_LEVEL_1Gbecause it is currently the largest page size supported and it natually fits the scenario of splitting large pages.For
zap_allcase (usually) at VM teardown time, we use a two-phase
mechanism: the 1st phase zaps all SPTEs atPG_LEVEL_1Glevel and 2nd phase zaps everything else. This is achieved by the helper function__zap_gfn_range().Cc Sean Christopherson <[email protected]>Cc Ben Gardon <[email protected]>Cc David Matlack <[email protected]>Signed-off-by Mingwei Zhang <[email protected]>--- arch/x86/kvm/mmu/tdp_mmu.c | 57 ++++++++++++++++++++++++++------------ 1 file changed, 40 insertions(+), 17 deletions(-)arch/x86/kvm/mmu/tdp_mmu.c +6
-14@@ -697,24 +697,16 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,* account for the possibility that other threads are modifying the paging * structures concurrently. If shared is false, this thread should hold the * MMU lock in write mode. +* +* If zap_all is true, eliminate all the paging structures that contains the +* SPTEs. */ -static bool zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, -gfn_t start,gfn_t end,bool can_yield,bool flush, -bool shared) +static bool __zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool shared,bool zap_all) { -gfn_t max_gfn_host =1ULL << (shadow_phys_bits -PAGE_SHIFT ); -bool zap_all = (start ==0 && end >= max_gfn_host);struct tdp_iter iter; -/* -* Bound the walk at host.MAXPHYADDR, guest accesses beyond that will -* hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF, -* and so KVM will never install a SPTE for such addresses. -*/ - end =min (end, max_gfn_host); - -kvm_lockdep_assert_mmu_lock_held (kvm, shared); -rcu_read_lock ();tdp_root_for_each_pte (iter, root, start, end) {
arch/x86/kvm/mmu/tdp_mmu.c static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, · line 725
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +5
-1@@ -725,17 +717,24 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,continue ; } -if (!is_shadow_present_pte (iter.old_spte)) +/* +* In zap_all case, ignore the checking of present since we have +* to zap everything. +*/ +if (!zap_all && !is_shadow_present_pte (iter.old_spte))continue ;
I don't believe there's any reason to attempt to zap a non-present spte, even in the zap_all case. In any case, this change deserves its own patch and a commit message that describes why the old logic is incorrect and how this fixes it.
Quoted 6 lines
arch/x86/kvm/mmu/tdp_mmu.c +1
-1/* * If this is a non-last-level SPTE that covers a larger range * than should be zapped, continue, and zap the mappings at a -* lower level, except when zapping all SPTEs. +* lower level. Actual zapping started at proper granularity
arch/x86/kvm/mmu/tdp_mmu.c static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, · line 739
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-1+* that is not so large as to cause a soft lockup when handling +* the changed pte (which does not yield). */ if (!zap_all && (iter.gfn < start || - iter.gfn +KVM_PAGES_PER_HPAGE (iter.level) > end) && + iter.gfn +KVM_PAGES_PER_HPAGE (iter.level) > end || + iter.level >PG_LEVEL_1G ) && !is_last_spte (iter.old_spte, iter.level))continue ;
This if statement is getting a bit long. I'd suggest breaking out the level check and also using KVM_MAX_HUGEPAGE_LEVEL.
e.g.
/*
* If not doing zap_all, only zap up to the huge page level to
* avoid doing too much work in the recursive tdp_mmu_set_spte*
* call below, since it does not yield.
*
* This will potentially leave behind some childless page tables
* but that's ok because ...
*/
if (!zap_all && iter.level > KVM_MAX_HUGEPAGE_LEVEL)
continue;
And on that note, what is the reasoning for why it's ok to leave behind childless page tables? I assume it's because most of the time we'll use that page table again in the future, and at worst we leave the page table allocated until the VM is cleaned up?
Quoted 35 lines
arch/x86/kvm/mmu/tdp_mmu.c +24
-0@@ -756,6 +755,30 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,return flush; } +static bool zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool shared) +{ +gfn_t max_gfn_host =1ULL << (shadow_phys_bits -PAGE_SHIFT ); +bool zap_all = (start ==0 && end >= max_gfn_host); + +/* +* Bound the walk at host.MAXPHYADDR, guest accesses beyond that will +* hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF, +* and so KVM will never install a SPTE for such addresses. +*/ + end =min (end, max_gfn_host); + +kvm_lockdep_assert_mmu_lock_held (kvm, shared); + + flush =__zap_gfn_range (kvm, root, start, end, can_yield, flush, shared, + false); +if (zap_all) + flush =__zap_gfn_range (kvm, root, start, end, can_yield, flush, + shared, true); +return flush; +} +/* * Tears down the mappings for the range of gfns, [start, end), and frees the * non-root pages mapping GFNs strictly within that range. Returns true if Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Re: [PATCH 2/2] KVM: mmu/x86: optimize zapping by retaining non-leaf SPTEs and avoid rcu stall
Quoted 14 lines · David Matlack
Quoted 4 lines · Mingwei Zhang
TDP MMU SPTE zapping process currently uses two levels of iterations. The first level iteration happens at the for loop within the
zap_gfn_range()with the purpose of calibrating the accurate range for zapping. The second level itreration start at tdp_mmu_set_spte{,_atomic}() that tears down theiteration
Quoted 2 lines
whole paging structures (leaf and non-leaf SPTEs) within the range. The former iteration is yield safe, while the second one is not.
I know what you mean but I'd suggest being more specific than "yield safe". For example:
Quoted 8 lines
Unlike the outer loop, the recursive zapping done under tdp_mmu_set_spte{,_atomic} does not yield. Since zapping is done with a pre-order traversal, zapping sufficiently large ranges can lead to RCU stall warnings.
I'd also clarify here that the TDP MMU iterator uses a pre-order traversal which causes us KVM to end up doing the maximum amount of zapping under tdp_mmu_set_spte{,_atomic} and not the outer for loop.
(Ah sorry for the redundant suggestion. I wrote this paragraph and then reworded my suggested wording to mention the pre-order traversal.)
Quoted 160 lines
Quoted 71 lines
In many cases, zapping SPTE process could be optimized since the non-leaf SPTEs could most likely be retained for the next allocation. On the other hand, for large scale SPTE zapping scenarios, we may end up zapping too many SPTEs and use excessive CPU time that causes the RCU stall warning.
The follow selftest reproduces the warning:
(env: kvm.tdp_mmu=Y) ./dirty_log_perf_test -v 64 -b 8GOptimize the zapping process by skipping all SPTEs above a certain level in the first iteration. This allows us to control the granularity of the actual zapping and invoke
tdp_mmu_iter_cond_resched()on time. In addition, we would retain some of the non-leaf SPTEs to accelerate next allocation.For the selection of the
certain level, we choose thePG_LEVEL_1Gbecause it is currently the largest page size supported and it natually fits the scenario of splitting large pages.For
zap_allcase (usually) at VM teardown time, we use a two-phase
mechanism: the 1st phase zaps all SPTEs atPG_LEVEL_1Glevel and 2nd phase zaps everything else. This is achieved by the helper function__zap_gfn_range().Cc Sean Christopherson <[email protected]>Cc Ben Gardon <[email protected]>Cc David Matlack <[email protected]>Signed-off-by Mingwei Zhang <[email protected]>--- arch/x86/kvm/mmu/tdp_mmu.c | 57 ++++++++++++++++++++++++++------------ 1 file changed, 40 insertions(+), 17 deletions(-)arch/x86/kvm/mmu/tdp_mmu.c +6
-14@@ -697,24 +697,16 @@ static inline bool tdp_mmu_iter_cond_resched(struct kvm *kvm,* account for the possibility that other threads are modifying the paging * structures concurrently. If shared is false, this thread should hold the * MMU lock in write mode. +* +* If zap_all is true, eliminate all the paging structures that contains the +* SPTEs. */ -static bool zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, -gfn_t start,gfn_t end,bool can_yield,bool flush, -bool shared) +static bool __zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool shared,bool zap_all) { -gfn_t max_gfn_host =1ULL << (shadow_phys_bits -PAGE_SHIFT ); -bool zap_all = (start ==0 && end >= max_gfn_host);struct tdp_iter iter; -/* -* Bound the walk at host.MAXPHYADDR, guest accesses beyond that will -* hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF, -* and so KVM will never install a SPTE for such addresses. -*/ - end =min (end, max_gfn_host); - -kvm_lockdep_assert_mmu_lock_held (kvm, shared); -rcu_read_lock ();tdp_root_for_each_pte (iter, root, start, end) {arch/x86/kvm/mmu/tdp_mmu.c static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, · line 725
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +5
-1@@ -725,17 +717,24 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,continue ; } -if (!is_shadow_present_pte (iter.old_spte)) +/* +* In zap_all case, ignore the checking of present since we have +* to zap everything. +*/ +if (!zap_all && !is_shadow_present_pte (iter.old_spte))continue ;I don't believe there's any reason to attempt to zap a non-present spte, even in the
zap_allcase. In any case, this change deserves its own patch and a commit message that describes why the old logic is incorrect and how this fixes it.Quoted 6 lines
arch/x86/kvm/mmu/tdp_mmu.c +1
-1/* * If this is a non-last-level SPTE that covers a larger range * than should be zapped, continue, and zap the mappings at a -* lower level, except when zapping all SPTEs. +* lower level. Actual zapping started at proper granularity arch/x86/kvm/mmu/tdp_mmu.c static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, · line 739
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +4
-1+* that is not so large as to cause a soft lockup when handling +* the changed pte (which does not yield). */ if (!zap_all && (iter.gfn < start || - iter.gfn +KVM_PAGES_PER_HPAGE (iter.level) > end) && + iter.gfn +KVM_PAGES_PER_HPAGE (iter.level) > end || + iter.level >PG_LEVEL_1G ) && !is_last_spte (iter.old_spte, iter.level))continue ;This if statement is getting a bit long. I'd suggest breaking out the level check and also using
KVM_MAX_HUGEPAGE_LEVEL.e.g.
/*
* If not doingzap_all, only zap up to the huge page level to
* avoid doing too much work in the recursive tdp_mmu_set_spte*
* call below, since it does not yield.
*
* This will potentially leave behind some childless page tables
* but that's ok because ...
*/
if (!zap_all && iter.level >KVM_MAX_HUGEPAGE_LEVEL)
continue;And on that note, what is the reasoning for why it's ok to leave behind childless page tables? I assume it's because most of the time we'll use that page table again in the future, and at worst we leave the page table allocated until the VM is cleaned up?
Quoted 35 lines
arch/x86/kvm/mmu/tdp_mmu.c +24
-0@@ -756,6 +755,30 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,return flush; } +static bool zap_gfn_range (struct kvm *kvm,struct kvm_mmu_page *root, +gfn_t start,gfn_t end,bool can_yield,bool flush, +bool shared) +{ +gfn_t max_gfn_host =1ULL << (shadow_phys_bits -PAGE_SHIFT ); +bool zap_all = (start ==0 && end >= max_gfn_host); + +/* +* Bound the walk at host.MAXPHYADDR, guest accesses beyond that will +* hit a #PF(RSVD) and never get to an EPT Violation/Misconfig / #NPF, +* and so KVM will never install a SPTE for such addresses. +*/ + end =min (end, max_gfn_host); + +kvm_lockdep_assert_mmu_lock_held (kvm, shared); + + flush =__zap_gfn_range (kvm, root, start, end, can_yield, flush, shared, + false); +if (zap_all) + flush =__zap_gfn_range (kvm, root, start, end, can_yield, flush, + shared, true); +return flush; +} +/* * Tears down the mappings for the range of gfns, [start, end), and frees the * non-root pages mapping GFNs strictly within that range. Returns true if Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Sean Christopherson
maintainerRe: [PATCH 2/2] KVM: mmu/x86: optimize zapping by retaining non-leaf SPTEs and avoid rcu stall
In reply to David MatlackQuoted 11 lines · David Matlack
Review
Quoted 6 lines
(quoted diff) +5
-0+ /* + * In zap_all case, ignore the checking of present since we have + * to zap everything. + */ + if (!zap_all && !is_shadow_present_pte(iter.old_spte)) continue;I don't believe there's any reason to attempt to zap a non-present spte, even in the
zap_allcase. In any case, this change deserves its own patch and a commit message that describes why the old logic is incorrect and how this fixes it.
Yep, at best it's wasted cycles, at worst it will trigger WARN/BUG due to using accessors that require the caller to check for a shadow present SPTE.
Quoted 31 lines
Quoted 4 lines
(quoted diff) +1
-1* If this is a non-last-level SPTE that covers a larger range * than should be zapped, continue, and zap the mappings at a - * lower level, except when zapping all SPTEs. + * lower level. Actual zapping started at proper granularityReview
Quoted 10 lines
(quoted diff) +4
-1+ * that is not so large as to cause a soft lockup when handling + * the changed pte (which does not yield). */ if (!zap_all && (iter.gfn < start || - iter.gfn + KVM_PAGES_PER_HPAGE(iter.level) > end) && + iter.gfn + KVM_PAGES_PER_HPAGE(iter.level) > end || + iter.level > PG_LEVEL_1G) && !is_last_spte(iter.old_spte, iter.level)) continue;This if statement is getting a bit long. I'd suggest breaking out the level check and also using
KVM_MAX_HUGEPAGE_LEVEL.e.g.
/*
* If not doingzap_all, only zap up to the huge page level to
* avoid doing too much work in the recursive tdp_mmu_set_spte*
* call below, since it does not yield.
*
* This will potentially leave behind some childless page tables
* but that's ok because ...
*/
if (!zap_all && iter.level >KVM_MAX_HUGEPAGE_LEVEL)
continue;
Quoted 4 lines
And on that note, what is the reasoning for why it's ok to leave behind childless page tables? I assume it's because most of the time we'll use that page table again in the future, and at worst we leave the page table allocated until the VM is cleaned up?
Yes. If zap_all=false, KVM is zapping a gfn range but not dropping or invalidating the root. The non-leaf paging structures are perfectly valid and can be reused. There's certainly no guarantees that a structure will be reused, but keeping the pages is ok because the memory consumed scales with the size of the VM, and the number of shadow pages used for TDP is quite small, e.g. an order of magnitude less than what's needed for shadow paging.
If we're ok waiting until my zap n' flush rework[*] lands, this is much easier to handle, e.g. I think this will do it.
arch/x86/kvm/mmu/tdp_mmu.c +13 -6
@@ -765,7 +765,7 @@ static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, bool shared, bool root_is_unreachable) {struct tdp_iter iter; - +bool leafs_only;gfn_t end =tdp_mmu_max_gfn_host ();gfn_t start =0 ;
@@ -773,12 +773,11 @@ static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root,rcu_read_lock (); -/* -* No need to try to step down in the iterator when zapping an entire -* root, zapping an upper-level SPTE will recurse on its children. -*/ +again: +/* Add comment here. */ for_each_tdp_pte_min_level (iter, root->spt, root->role.level, - root->role.level, start, end) { + leafs_only ?PG_LEVEL_4K : root->role.level, + start, end) { retry:if (tdp_mmu_iter_cond_resched (kvm, &iter, false, shared))continue ;
@@ -786,6 +785,9 @@ static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root,if (!is_shadow_present_pte (iter.old_spte))continue ; +if (leafs_only && !is_last_spte (iter.old_spte, iter.level)) +continue ; +if (!shared) {tdp_mmu_set_spte (kvm, &iter,0 ); }else if (!tdp_mmu_set_spte_atomic (kvm, &iter,0 )) {
@@ -807,6 +809,11 @@ static void tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root,WARN_ON_ONCE (root_is_unreachable && root->role.invalid); } +if (leafs_only) { + leafs_only = false; + goto again; + } +rcu_read_unlock (); }