Skip to contentSkip to conversation

KVM: x86/mmu: Micro-optimize TDP MMU cond_resched()

PATCH In review Sean Christopherson 2 participants 5 messages Oct 31, 2024 – Nov 5, 2024

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

cover KVM: x86/mmu: Micro-optimize TDP MMU cond_resched()

Invert the order of checks in tdp_mmu_iter_cond_resched() so that the common case (no resched needed) is checked first, and opportunsitically clean up a wart where the helper would return "yielded" in an error path, even if that invocation didn't actually yield. The latter cleanup allows extracting the base "need resched" logic to a separate helper.

Note, I "speculatively" pushed these patches to kvm-x86 mmu, as they are effectively needed for David's series to optimize hugepage recovery when disabling dirty logging. I'll rework the commits sometime next weeks, e.g. fix a few changelog typos (I found two, so far), to capture reviews, and add Links. And if there are problems, I'll yank you the entire series (David's and mine).

012a5c17cba4d165961a4643c5e10a151f003733 KVM: x86/mmu: Demote the WARN on yielded in xxx_cond_resched() to KVM_MMU_WARN_ON d400ce271d9c555a93e38fb36ecb2d3bd60b234c KVM: x86/mmu: Check yielded_gfn for forward progress iff resched is needed

Sean Christopherson (2):
KVM: x86/mmu: Check yielded_gfn for forward progress iff resched is
needed
KVM: x86/mmu: Demote the WARN on yielded in xxx_cond_resched() to
KVM_MMU_WARN_ON

 arch/x86/kvm/mmu/tdp_mmu.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

base-commit: 5cb1659f412041e4780f2e8ee49b2e03728a2ba6

Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
author maintainer
#1 lore raw

PATCH 1/2 KVM: x86/mmu: Check yielded_gfn for forward progress iff resched is needed

Swap the order of the checks in tdp_mmu_iter_cond_resched() so that KVM checks to see if a resched is needed _before_ checking to see if yielding must be disallowed to guarantee forward progress. Iterating over TDP MMU SPTEs is a hot path, e.g. tearing down a root can touch millions of SPTEs, and not needing to reschedule is by far the common case. On the other hand, disallowing yielding because forward progress has not been made is a very rare case.

Returning early for the common case (no resched), effectively reduces the number of checks from 2 to 1 for the common case, and should make the code slightly more predictable for the CPU.

To resolve a weird conundrum where the forward progress check currently returns false, but the need resched check subtly returns iter->yielded, which _should_ be false (enforced by a WARN), return false unconditionally (which might also help make the sequence more predictable). If KVM has a bug where iter->yielded is left danging, continuing to yield is neither right nor wrong, it was simply an artifact of how the original code was written.

Unconditionally returning false when yielding is unnecessary or unwanted will also allow extracting the "should resched" logic to a separate helper in a future patch.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +14 -14
index 91caa73a905b..a06f3d5cb651 100644
@@ -700,29 +700,29 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm,
 {
 	WARN_ON_ONCE(iter->yielded);
 
+	if (!need_resched() && !rwlock_needbreak(&kvm->mmu_lock))
+		return false;
+
 	/* Ensure forward progress has been made before yielding. */
 	if (iter->next_last_level_gfn == iter->yielded_gfn)
 		return false;
 
-	if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {
-		if (flush)
-			kvm_flush_remote_tlbs(kvm);
+	if (flush)
+		kvm_flush_remote_tlbs(kvm);
 
-		rcu_read_unlock();
+	rcu_read_unlock();
 
-		if (shared)
-			cond_resched_rwlock_read(&kvm->mmu_lock);
-		else
-			cond_resched_rwlock_write(&kvm->mmu_lock);
+	if (shared)
+		cond_resched_rwlock_read(&kvm->mmu_lock);
+	else
+		cond_resched_rwlock_write(&kvm->mmu_lock);
 
-		rcu_read_lock();
+	rcu_read_lock();
 
-		WARN_ON_ONCE(iter->gfn > iter->next_last_level_gfn);
+	WARN_ON_ONCE(iter->gfn > iter->next_last_level_gfn);
 
-		iter->yielded = true;
-	}
-
-	return iter->yielded;
+	iter->yielded = true;
+	return true;
 }
 
 static inline gfn_t tdp_mmu_max_gfn_exclusive(void)
Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] KVM: x86/mmu: Check yielded_gfn for forward progress iff resched is needed

Quoted 21 lines · Sean Christopherson

Swap the order of the checks in tdp_mmu_iter_cond_resched() so that KVM checks to see if a resched is needed _before_ checking to see if yielding must be disallowed to guarantee forward progress. Iterating over TDP MMU SPTEs is a hot path, e.g. tearing down a root can touch millions of SPTEs, and not needing to reschedule is by far the common case. On the other hand, disallowing yielding because forward progress has not been made is a very rare case.

Returning early for the common case (no resched), effectively reduces the number of checks from 2 to 1 for the common case, and should make the code slightly more predictable for the CPU.

To resolve a weird conundrum where the forward progress check currently returns false, but the need resched check subtly returns iter->yielded, which _should_ be false (enforced by a WARN), return false unconditionally (which might also help make the sequence more predictable). If KVM has a bug where iter->yielded is left danging, continuing to yield is neither right nor wrong, it was simply an artifact of how the original code was written.

Quoted 6 lines

Unconditionally returning false when yielding is unnecessary or unwanted will also allow extracting the "should resched" logic to a separate helper in a future patch.

Feel free to add:

author maintainer
#3 lore raw

PATCH 2/2 KVM: x86/mmu: Demote the WARN on yielded in xxx_cond_resched() to KVM_MMU_WARN_ON

In reply to Sean Christopherson

Convert the WARN in tdp_mmu_iter_cond_resched() that the iterator hasn't already yielded to a KVM_MMU_WARN_ON() so the code is compiled out for production kernels (assuming production kernels disable KVM_PROVE_MMU).

Checking for a needed reschedule is a hot path, and KVM sanity checks iter->yielded in several other less-hot paths, i.e. the odds of KVM not flagging that something went sideways are quite low. Furthermore, the odds of KVM not noticing *and* the WARN detecting something worth investigating are even lower.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index a06f3d5cb651..c158ef8c1a36 100644
@@ -698,7 +698,7 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm,
 							  struct tdp_iter *iter,
 							  bool flush, bool shared)
 {
-	WARN_ON_ONCE(iter->yielded);
+	KVM_MMU_WARN_ON(iter->yielded);
 
 	if (!need_resched() && !rwlock_needbreak(&kvm->mmu_lock))
 		return false;
Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
author maintainer
#4 lore raw
In reply to Sean Christopherson
Quoted 6 lines · Sean Christopherson

Invert the order of checks in tdp_mmu_iter_cond_resched() so that the common case (no resched needed) is checked first, and opportunsitically clean up a wart where the helper would return "yielded" in an error path, even if that invocation didn't actually yield. The latter cleanup allows extracting the base "need resched" logic to a separate helper.

Quoted 8 lines

Note, I "speculatively" pushed these patches to kvm-x86 mmu, as they are effectively needed for David's series to optimize hugepage recovery when disabling dirty logging. I'll rework the commits sometime next weeks, e.g. fix a few changelog typos (I found two, so far), to capture reviews, and add Links. And if there are problems, I'll yank you the entire series (David's and mine).

[...]

"Officially" applied to kvm-x86 mmu.

[1/2] KVM: x86/mmu: Check yielded_gfn for forward progress iff resched is needed

      https://github.com/kvm-x86/linux/commit/e287e4316713
[2/2] KVM: x86/mmu: Demote the WARN on yielded in xxx_cond_resched() to KVM_MMU_WARN_ON
      https://github.com/kvm-x86/linux/commit/38b0ac47169b

--
https://github.com/kvm-x86/linux/tree/next