Skip to contentSkip to conversation

KVM: x86/mmu: Complete prefetch for trailing SPTEs for direct, legacy MMU

PATCH Applied Sean Christopherson 5 participants 5 messages Aug 18, 2021 – Sep 23, 2021

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

PATCH 1/1 KVM: x86/mmu: Complete prefetch for trailing SPTEs for direct, legacy MMU

Make a final call to direct_pte_prefetch_many() if there are "trailing" SPTEs to prefetch, i.e. SPTEs for GFNs following the faulting GFN. The call to direct_pte_prefetch_many() in the loop only handles the case where there are !PRESENT SPTEs preceding a PRESENT SPTE.

E.g. if the faulting GFN is a multiple of 8 (the prefetch size) and all SPTEs for the following GFNs are !PRESENT, the loop will terminate with "start = sptep+1" and not prefetch any SPTEs.

Prefetching trailing SPTEs as intended can drastically reduce the number of guest page faults, e.g. accessing the first byte of every 4kb page in a 6gb chunk of virtual memory, in a VM with 8gb of preallocated memory, the number of pf_fixed events observed in L0 drops from ~1.75M to <0.27M.

Note, this only affects memory that is backed by 4kb pages as KVM doesn't prefetch when installing hugepages. Shadow paging prefetching is not affected as it does not batch the prefetches due to the need to process the corresponding guest PTE. The TDP MMU is not affected because it doesn't have prefetching, yet...

Fixes 957ed9effd80 ("KVM: MMU: prefetch ptes when intercepted guest #PF")
Cc Sergey Senozhatsky <[email protected]>
Cc Ben Gardon <[email protected]>

---

Cc'd Ben as this highlights a potential gap with the TDP MMU, which lacks prefetching of any sort. For large VMs, which are likely backed by hugepages anyways, this is a non-issue as the benefits of holding mmu_lock for read likely masks the cost of taking more VM-Exits. But VMs with a small number of vCPUs won't benefit as much from parallel page faults, e.g. there's no benefit at all if there's a single vCPU.

 arch/x86/kvm/mmu/mmu.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +3 -1
index a272ccbddfa1..daf7df35f788 100644
@@ -2818,11 +2818,13 @@ static void __direct_pte_prefetch(struct kvm_vcpu *vcpu,
 			if (!start)
 				continue;
 			if (direct_pte_prefetch_many(vcpu, sp, start, spte) < 0)
-				break;
+				return;
 			start = NULL;
 		} else if (!start)
 			start = spte;
 	}
+	if (start)
+		direct_pte_prefetch_many(vcpu, sp, start, spte);
 }
 
 static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
Signature · 1 line
2.33.0.rc1.237.g0d66db33f3-goog
Tested-by on v1 1/1
#1 lore raw

[..]

Quoted 14 lines

Make a final call to direct_pte_prefetch_many() if there are "trailing" SPTEs to prefetch, i.e. SPTEs for GFNs following the faulting GFN. The call to direct_pte_prefetch_many() in the loop only handles the case where there are !PRESENT SPTEs preceding a PRESENT SPTE.

E.g. if the faulting GFN is a multiple of 8 (the prefetch size) and all SPTEs for the following GFNs are !PRESENT, the loop will terminate with "start = sptep+1" and not prefetch any SPTEs.

Prefetching trailing SPTEs as intended can drastically reduce the number of guest page faults, e.g. accessing the first byte of every 4kb page in a 6gb chunk of virtual memory, in a VM with 8gb of preallocated memory, the number of pf_fixed events observed in L0 drops from ~1.75M to <0.27M.

Quoted 5 lines

Note, this only affects memory that is backed by 4kb pages as KVM doesn't prefetch when installing hugepages. Shadow paging prefetching is not affected as it does not batch the prefetches due to the need to process the corresponding guest PTE. The TDP MMU is not affected because it doesn't have prefetching, yet...

I ran some tests.

- VM Boot up

From

EPT_VIOLATION 1192184 75.18% 4.40% 0.77us 18020.01us 4.32us ( +- 1.71% )

to

EPT_VIOLATION 947460 69.92% 4.64% 0.69us 34902.15us 5.06us ( +- 1.64% )

- Running test app (in VM)

From

EPT_VIOLATION 6550167 71.05% 11.76% 0.77us 32562.18us 3.51us ( +- 0.36% )

to

EPT_VIOLATION 5489904 68.32% 11.29% 0.71us 16564.19us 3.92us ( +- 0.29% )

Reviewed-by on v1 1/1
#2 lore raw
In reply to Sean Christopherson
Quoted 44 lines · Sean Christopherson

Make a final call to direct_pte_prefetch_many() if there are "trailing" SPTEs to prefetch, i.e. SPTEs for GFNs following the faulting GFN. The call to direct_pte_prefetch_many() in the loop only handles the case where there are !PRESENT SPTEs preceding a PRESENT SPTE.

E.g. if the faulting GFN is a multiple of 8 (the prefetch size) and all SPTEs for the following GFNs are !PRESENT, the loop will terminate with "start = sptep+1" and not prefetch any SPTEs.

Prefetching trailing SPTEs as intended can drastically reduce the number of guest page faults, e.g. accessing the first byte of every 4kb page in a 6gb chunk of virtual memory, in a VM with 8gb of preallocated memory, the number of pf_fixed events observed in L0 drops from ~1.75M to <0.27M.

Note, this only affects memory that is backed by 4kb pages as KVM doesn't prefetch when installing hugepages. Shadow paging prefetching is not affected as it does not batch the prefetches due to the need to process the corresponding guest PTE. The TDP MMU is not affected because it doesn't have prefetching, yet...

Fixes 957ed9effd80 ("KVM: MMU: prefetch ptes when intercepted guest #PF")
Cc Sergey Senozhatsky <[email protected]>
Cc Ben Gardon <[email protected]>

---

Cc'd Ben as this highlights a potential gap with the TDP MMU, which lacks prefetching of any sort. For large VMs, which are likely backed by hugepages anyways, this is a non-issue as the benefits of holding mmu_lock for read likely masks the cost of taking more VM-Exits. But VMs with a small number of vCPUs won't benefit as much from parallel page faults, e.g. there's no benefit at all if there's a single vCPU.

 arch/x86/kvm/mmu/mmu.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index a272ccbddfa1..daf7df35f788 100644
@@ -2818,11 +2818,13 @@ static void __direct_pte_prefetch(struct kvm_vcpu *vcpu,
                        if (!start)
                                continue;

arch/x86/kvm/mmu/mmu.c static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, · line 2828

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +3 -1
                        if (direct_pte_prefetch_many(vcpu, sp, start, spte) < 0)
-                               break;
+                               return;
                        start = NULL;
                } else if (!start)
                        start = spte;
        }
+       if (start)
+               direct_pte_prefetch_many(vcpu, sp, start, spte);
 }
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +0 -1
 static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
--

2.33.0.rc1.237.g0d66db33f3-goog

Reviewed-by
#3 lore raw
Quoted 23 lines · Lai Jiangshan
Quoted 21 lines · Sean Christopherson

Make a final call to direct_pte_prefetch_many() if there are "trailing" SPTEs to prefetch, i.e. SPTEs for GFNs following the faulting GFN. The call to direct_pte_prefetch_many() in the loop only handles the case where there are !PRESENT SPTEs preceding a PRESENT SPTE.

E.g. if the faulting GFN is a multiple of 8 (the prefetch size) and all SPTEs for the following GFNs are !PRESENT, the loop will terminate with "start = sptep+1" and not prefetch any SPTEs.

Prefetching trailing SPTEs as intended can drastically reduce the number of guest page faults, e.g. accessing the first byte of every 4kb page in a 6gb chunk of virtual memory, in a VM with 8gb of preallocated memory, the number of pf_fixed events observed in L0 drops from ~1.75M to <0.27M.

Note, this only affects memory that is backed by 4kb pages as KVM doesn't prefetch when installing hugepages. Shadow paging prefetching is not affected as it does not batch the prefetches due to the need to process the corresponding guest PTE. The TDP MMU is not affected because it doesn't have prefetching, yet...

Quoted 4 lines
Quoted 4 lines
Fixes 957ed9effd80 ("KVM: MMU: prefetch ptes when intercepted guest #PF")
Cc Sergey Senozhatsky <[email protected]>
Cc Ben Gardon <[email protected]>
Quoted 8 lines
Quoted 8 lines

---

Cc'd Ben as this highlights a potential gap with the TDP MMU, which lacks prefetching of any sort. For large VMs, which are likely backed by hugepages anyways, this is a non-issue as the benefits of holding mmu_lock for read likely masks the cost of taking more VM-Exits. But VMs with a small number of vCPUs won't benefit as much from parallel page faults, e.g. there's no benefit at all if there's a single vCPU.

Yeah, that probably does represent a reduction in performance for very small VMs. Besides keeping read critical sections small, there's no reason not to do prefetching with the TDP MMU, it just needs to be implemented.

Quoted 4 lines
Quoted 4 lines
 arch/x86/kvm/mmu/mmu.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 16 lines
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +0 -0
index a272ccbddfa1..daf7df35f788 100644
@@ -2818,11 +2818,13 @@ static void __direct_pte_prefetch(struct kvm_vcpu *vcpu,
                        if (!start)

arch/x86/kvm/mmu/mmu.c static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, · line 2827

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +3 -1
                                continue;
                        if (direct_pte_prefetch_many(vcpu, sp, start, spte) < 0)
-                               break;
+                               return;
                        start = NULL;
                } else if (!start)
                        start = spte;
        }
+       if (start)
+               direct_pte_prefetch_many(vcpu, sp, start, spte);

It might be worth explaining some of what you laid out in the commit description here. This function's implementation is not the easiest to read.

Quoted 10 lines

arch/x86/kvm/mmu/mmu.c static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, · line 2828

Quoted 1 line
arch/x86/kvm/mmu/mmu.c +0 -0
 }
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +0 -1
 static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)
--

2.33.0.rc1.237.g0d66db33f3-goog

maintainer Applied
#4 lore raw
In reply to Sean Christopherson
Quoted 46 lines · Sean Christopherson

Make a final call to direct_pte_prefetch_many() if there are "trailing" SPTEs to prefetch, i.e. SPTEs for GFNs following the faulting GFN. The call to direct_pte_prefetch_many() in the loop only handles the case where there are !PRESENT SPTEs preceding a PRESENT SPTE.

E.g. if the faulting GFN is a multiple of 8 (the prefetch size) and all SPTEs for the following GFNs are !PRESENT, the loop will terminate with "start = sptep+1" and not prefetch any SPTEs.

Prefetching trailing SPTEs as intended can drastically reduce the number of guest page faults, e.g. accessing the first byte of every 4kb page in a 6gb chunk of virtual memory, in a VM with 8gb of preallocated memory, the number of pf_fixed events observed in L0 drops from ~1.75M to <0.27M.

Note, this only affects memory that is backed by 4kb pages as KVM doesn't prefetch when installing hugepages. Shadow paging prefetching is not affected as it does not batch the prefetches due to the need to process the corresponding guest PTE. The TDP MMU is not affected because it doesn't have prefetching, yet...

Fixes 957ed9effd80 ("KVM: MMU: prefetch ptes when intercepted guest #PF")
Cc Sergey Senozhatsky <[email protected]>
Cc Ben Gardon <[email protected]>

---

Cc'd Ben as this highlights a potential gap with the TDP MMU, which lacks prefetching of any sort. For large VMs, which are likely backed by hugepages anyways, this is a non-issue as the benefits of holding mmu_lock for read likely masks the cost of taking more VM-Exits. But VMs with a small number of vCPUs won't benefit as much from parallel page faults, e.g. there's no benefit at all if there's a single vCPU.

  arch/x86/kvm/mmu/mmu.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -1
index a272ccbddfa1..daf7df35f788 100644
@@ -2818,11 +2818,13 @@ static void __direct_pte_prefetch(struct kvm_vcpu *vcpu,
  			if (!start)
  				continue;
  			if (direct_pte_prefetch_many(vcpu, sp, start, spte) < 0)
-				break;

arch/x86/kvm/mmu/mmu.c static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, · line 2830

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+				return;
  			start = NULL;
  		} else if (!start)
  			start = spte;
  	}
+	if (start)
+		direct_pte_prefetch_many(vcpu, sp, start, spte);
  }
 
  static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)

Queued, thanks.

Paolo