Skip to contentSkip to conversation

KVM: x86/mmu: Skip !MMU-present SPTEs when removing SP in exclusive mode

PATCH In review Sean Christopherson 3 participants 5 messages Mar 10, 2021 – Mar 12, 2021

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

PATCH 1/1 KVM: x86/mmu: Skip !MMU-present SPTEs when removing SP in exclusive mode

If mmu_lock is held for write, don't bother setting !PRESENT SPTEs to REMOVED_SPTE when recursively zapping SPTEs as part of shadow page removal. The concurrent write protections provided by REMOVED_SPTE are not needed, there are no backing page side effects to record, and MMIO SPTEs can be left as is since they are protected by the memslot generation, not by ensuring that the MMIO SPTE is unreachable (which is racy with respect to lockless walks regardless of zapping behavior).

Skipping !PRESENT drastically reduces the number of updates needed to tear down sparsely populated MMUs, e.g. when tearing down a 6gb VM that didn't touch much memory, 6929/7168 (~96.6%) of SPTEs were '0' and could be skipped.

Avoiding the write itself is likely close to a wash, but avoiding __handle_changed_spte() is a clear-cut win as that involves saving and restoring all non-volatile GPRs (it's a subtly big function), as well as several conditional branches before bailing out.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/x86/kvm/mmu/tdp_mmu.c +11 -0
index 50ef757c5586..f0c99fa04ef2 100644
@@ -323,7 +323,18 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt,
 				cpu_relax();
 			}
 		} else {
+			/*
+			 * If the SPTE is not MMU-present, there is no backing
+			 * page associated with the SPTE and so no side effects
+			 * that need to be recorded, and exclusive ownership of
+			 * mmu_lock ensures the SPTE can't be made present.
+			 * Note, zapping MMIO SPTEs is also unnecessary as they
+			 * are guarded by the memslots generation, not by being
+			 * unreachable.
+			 */
 			old_child_spte = READ_ONCE(*sptep);
+			if (!is_shadow_present_pte(old_child_spte))
+				continue;
 
 			/*
 			 * Marking the SPTE as a removed SPTE is not
Signature · 1 line
2.30.1.766.gb4fecdf3b7-goog
maintainer
#1 lore raw
Quoted 13 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c +5 -0
index 50ef757c5586..f0c99fa04ef2 100644
@@ -323,7 +323,18 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt,
  				cpu_relax();
  			}
  		} else {
+			/*
+			 * If the SPTE is not MMU-present, there is no backing
+			 * page associated with the SPTE and so no side effects
+			 * that need to be recorded, and exclusive ownership of
+			 * mmu_lock ensures the SPTE can't be made present.

arch/x86/kvm/mmu/tdp_mmu.c static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt, · line 340

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +6 -0
+			 * Note, zapping MMIO SPTEs is also unnecessary as they
+			 * are guarded by the memslots generation, not by being
+			 * unreachable.
+			 */
  			old_child_spte = READ_ONCE(*sptep);
+			if (!is_shadow_present_pte(old_child_spte))
+				continue;
 
  			/*
  			 * Marking the SPTE as a removed SPTE is not

Ben, do you plan to make this path take mmu_lock for read? If so, this wouldn't be too useful IIUC.

Paolo

author maintainer
#2 lore raw
Quoted 18 lines · Paolo Bonzini
Quoted 17 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c +9 -0
index 50ef757c5586..f0c99fa04ef2 100644
@@ -323,7 +323,18 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt,
  				cpu_relax();
  			}
  		} else {
+			/*
+			 * If the SPTE is not MMU-present, there is no backing
+			 * page associated with the SPTE and so no side effects
+			 * that need to be recorded, and exclusive ownership of
+			 * mmu_lock ensures the SPTE can't be made present.
+			 * Note, zapping MMIO SPTEs is also unnecessary as they
+			 * are guarded by the memslots generation, not by being
+			 * unreachable.
+			 */
Quoted 8 lines

arch/x86/kvm/mmu/tdp_mmu.c static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt, · line 340

Quoted 5 lines
arch/x86/kvm/mmu/tdp_mmu.c +2 -0
  			old_child_spte = READ_ONCE(*sptep);
+			if (!is_shadow_present_pte(old_child_spte))
+				continue;
  			/*
  			 * Marking the SPTE as a removed SPTE is not

Ben, do you plan to make this path take mmu_lock for read? If so, this wouldn't be too useful IIUC.

I can see kvm_mmu_zap_all_fast()->kvm_tdp_mmu_zap_all() moving to a shared-mode flow, but I don't think we'll ever want to move away from exclusive-mode zapping for kvm_arch_flush_shadow_all()->kvm_mmu_zap_all()->kvm_tdp_mmu_zap_all(). In that case, the VM is dead or dying; freeing memory should be done as quickly as possible.

#3 lore raw
Quoted 29 lines · Sean Christopherson
Quoted 18 lines · Paolo Bonzini
Quoted 17 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c +9 -0
index 50ef757c5586..f0c99fa04ef2 100644
@@ -323,7 +323,18 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt,
                            cpu_relax();
                    }
            } else {
+                   /*
+                    * If the SPTE is not MMU-present, there is no backing
+                    * page associated with the SPTE and so no side effects
+                    * that need to be recorded, and exclusive ownership of
+                    * mmu_lock ensures the SPTE can't be made present.
+                    * Note, zapping MMIO SPTEs is also unnecessary as they
+                    * are guarded by the memslots generation, not by being
+                    * unreachable.
+                    */
Quoted 8 lines

arch/x86/kvm/mmu/tdp_mmu.c static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt, · line 340

Quoted 5 lines
arch/x86/kvm/mmu/tdp_mmu.c +2 -0
                    old_child_spte = READ_ONCE(*sptep);
+                   if (!is_shadow_present_pte(old_child_spte))
+                           continue;
                    /*
                     * Marking the SPTE as a removed SPTE is not

Ben, do you plan to make this path take mmu_lock for read? If so, this wouldn't be too useful IIUC.

Quoted 5 lines

I can see kvm_mmu_zap_all_fast()->kvm_tdp_mmu_zap_all() moving to a shared-mode flow, but I don't think we'll ever want to move away from exclusive-mode zapping for kvm_arch_flush_shadow_all()->kvm_mmu_zap_all()->kvm_tdp_mmu_zap_all(). In that case, the VM is dead or dying; freeing memory should be done as quickly as possible.

Yeah, as Sean said, zapping under the MMU lock in write mode probably shouldn't go away, even if we find we're able to do it in read mode in some flows.

This optimization also makes me think we could also skip the __handle_changed_spte call in the read mode case if the SPTE change was !PRESENT -> REMOVED.

maintainer
#4 lore raw
Quoted 22 lines · Ben Gardon
Quoted 21 lines · Sean Christopherson
Quoted 19 lines · Paolo Bonzini
Quoted 18 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c +9 -0
index 50ef757c5586..f0c99fa04ef2 100644
@@ -323,7 +323,18 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt,
                             cpu_relax();
                     }
             } else {
+                   /*
+                    * If the SPTE is not MMU-present, there is no backing
+                    * page associated with the SPTE and so no side effects
+                    * that need to be recorded, and exclusive ownership of
+                    * mmu_lock ensures the SPTE can't be made present.
+                    * Note, zapping MMIO SPTEs is also unnecessary as they
+                    * are guarded by the memslots generation, not by being
+                    * unreachable.
+                    */
                     old_child_spte = READ_ONCE(*sptep);
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

arch/x86/kvm/mmu/tdp_mmu.c static void handle_removed_tdp_mmu_page(struct kvm *kvm, u64 *pt, · line 340

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +2 -0
+                   if (!is_shadow_present_pte(old_child_spte))
+                           continue;
                     /*
                      * Marking the SPTE as a removed SPTE is not

This optimization also makes me think we could also skip the __handle_changed_spte call in the read mode case if the SPTE change was !PRESENT -> REMOVED.

Yes, I think so. It should be a separate patch anyway, so I've queued this one.

Paolo