Skip to contentSkip to conversation

ARM64: Guest performance improvement during dirty

PATCH Applied Jing Zhang 3 participants 10 messages Jan 18, 2022 – Feb 8, 2022

Subsystems (4)
lore
Share & tools
lore

Jing Zhang

author
#0 lore raw

v2 cover ARM64: Guest performance improvement during dirty

This patch is to reduce the performance degradation of guest workload during dirty logging on ARM64. A fast path is added to handle permission relaxation during dirty logging. The MMU lock is replaced with rwlock, by which all permision relaxations on leaf pte can be performed under the read lock. This greatly reduces the MMU lock contention during dirty logging. With this solution, the source guest workload performance degradation can be improved by more than 60%.

Problem:
* A Google internal live migration test shows that the source guest workload performance has >99% degradation for about 105 seconds, >50% degradation for about 112 seconds, >10% degradation for about 112 seconds on ARM64. This shows that most of the time, the guest workload degradtion is above 99%, which obviously needs some improvement compared to the test result on x86 (>99% for 6s, >50% for 9s, >10% for 27s).
* Tested H/W: Ampere Altra 3GHz, #CPU: 64, #Mem: 256GB, PageSize: 4K
* VM spec: #vCPU: 48, #Mem/vCPU: 4GB, PageSize: 4K, 2M hugepage backed

Analysis:
* We enabled CONFIG_LOCK_STAT in kernel and used dirty_log_perf_test to get

    the number of contentions of MMU lock and the "dirty memory time" on
    various VM spec. The "dirty memory time" is the time vCPU threads spent
    in KVM after fault. Higher "dirty memory time" means higher degradation
    to guest workload.
    '-m 2' specifies the mode "PA-bits:48,  VA-bits:48,  4K pages".
    By using test command
    ./dirty_log_perf_test -b 2G -m 2 -i 2 -s anonymous_hugetlb_2mb -v [#vCPU]
    Below are the results:
    +-------+------------------------+-----------------------+
    | #vCPU | dirty memory time (ms) | number of contentions |
    +-------+------------------------+-----------------------+
    | 1     | 926                    | 0                     |
    +-------+------------------------+-----------------------+
    | 2     | 1189                   | 4732558               |
    +-------+------------------------+-----------------------+
    | 4     | 2503                   | 11527185              |
    +-------+------------------------+-----------------------+
    | 8     | 5069                   | 24881677              |
    +-------+------------------------+-----------------------+
    | 16    | 10340                  | 50347956              |
    +-------+------------------------+-----------------------+
    | 32    | 20351                  | 100605720             |
    +-------+------------------------+-----------------------+
    | 64    | 40994                  | 201442478             |
    +-------+------------------------+-----------------------+

* From the test results above, the "dirty memory time" and the number of MMU lock contention scale with the number of vCPUs. That means all the dirty memory operations from all vCPU threads have been serialized by the MMU lock. Further analysis also shows that the permission relaxation during dirty logging is where vCPU threads get serialized.

Solution:
* On ARM64, there is no mechanism as PML (Page Modification Logging) and the dirty-bit solution for dirty logging is much complicated compared to the write-protection solution. The straight way to reduce the guest performance degradation is to enhance the concurrency for the permission fault path during dirty logging.
* In this patch, we only put leaf PTE permission relaxation for dirty

    logging under read lock, all others would go under write lock.
    Below are the results based on the fast path solution:
    +-------+------------------------+
    | #vCPU | dirty memory time (ms) |
    +-------+------------------------+
    | 1     | 965                    |
    +-------+------------------------+
    | 2     | 1006                   |
    +-------+------------------------+
    | 4     | 1128                   |
    +-------+------------------------+
    | 8     | 2005                   |
    +-------+------------------------+
    | 16    | 3903                   |
    +-------+------------------------+
    | 32    | 7595                   |
    +-------+------------------------+
    | 64    | 15783                  |
    +-------+------------------------+

* Furtuer analysis shows that there is another bottleneck caused by the

    setup of the test code itself. The 3rd commit is meant to fix that by
    setting up vgic in the test code. With the test code fix, below are
    the results which show better improvement.
    +-------+------------------------+
    | #vCPU | dirty memory time (ms) |
    +-------+------------------------+
    | 1     | 803                    |
    +-------+------------------------+
    | 2     | 843                    |
    +-------+------------------------+
    | 4     | 942                    |
    +-------+------------------------+
    | 8     | 1458                   |
    +-------+------------------------+
    | 16    | 2853                   |
    +-------+------------------------+
    | 32    | 5886                   |
    +-------+------------------------+
    | 64    | 12190                  |
    +-------+------------------------+
    All "dirty memory time" has been reduced by more than 60% when the
    number of vCPU grows.

* Based on the solution, the test results from the Google internal live migration test also shows more than 60% improvement with >99% for 30s, >50% for 58s and >10% for 76s.

---

* v1 -> v2
- Renamed flag name from use_mmu_readlock to logging_perm_fault.
- Removed unnecessary check for fault_granule to use readlock.
* RFC -> v1
- Rebase to kvm/queue, commit fea31d169094
(KVM: x86/pmu: Fix available_event_types check for REF_CPU_CYCLES event)
- Moved the fast path in user_mem_abort, as suggested by Marc.
- Addressed other comments from Marc.

[v1] https://lore.kernel.org/all/[email protected]
[RFC] https://lore.kernel.org/all/[email protected]

---

Jing Zhang (3):
KVM: arm64: Use read/write spin lock for MMU protection
KVM: arm64: Add fast path to handle permission relaxation during dirty
logging
KVM: selftests: Add vgic initialization for dirty log perf test for
ARM

 arch/arm64/include/asm/kvm_host.h             |  2 +
 arch/arm64/kvm/mmu.c                          | 49 ++++++++++++-------
 .../selftests/kvm/dirty_log_perf_test.c       | 10 ++++
 3 files changed, 43 insertions(+), 18 deletions(-)

base-commit: fea31d1690945e6dd6c3e89ec5591490857bc3d4

Signature · 1 line
2.34.1.703.g22d0c6ccf7-goog
author
#1 lore raw

PATCH v2 1/3 KVM: arm64: Use read/write spin lock for MMU protection

Replace MMU spinlock with rwlock and update all instances of the lock being acquired with a write lock acquisition.
Future commit will add a fast path for permission relaxation during dirty logging under a read lock.

---
 arch/arm64/include/asm/kvm_host.h |  2 ++
 arch/arm64/kvm/mmu.c              | 36 +++++++++++++++----------------
 2 files changed, 20 insertions(+), 18 deletions(-)
arch/arm64/include/asm/kvm_host.h +2 -0
index 3b44ea17af88..6c99c0335bae 100644
@@ -50,6 +50,8 @@
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
 				     KVM_DIRTY_LOG_INITIALLY_SET)
 
+#define KVM_HAVE_MMU_RWLOCK
+
 /*
  * Mode of operation configurable with kvm-arm.mode early param.
  * See Documentation/admin-guide/kernel-parameters.txt for more information.
arch/arm64/kvm/mmu.c +18 -18
index bc2aba953299..cafd5813c949 100644
@@ -58,7 +58,7 @@ static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr,
 			break;
 
 		if (resched && next != end)
-			cond_resched_lock(&kvm->mmu_lock);
+			cond_resched_rwlock_write(&kvm->mmu_lock);
 	} while (addr = next, addr != end);
 
 	return ret;
@@ -179,7 +179,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64
 	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
 	phys_addr_t end = start + size;
 
-	assert_spin_locked(&kvm->mmu_lock);
+	lockdep_assert_held_write(&kvm->mmu_lock);
 	WARN_ON(size & ~PAGE_MASK);
 	WARN_ON(stage2_apply_range(kvm, start, end, kvm_pgtable_stage2_unmap,
 				   may_block));
@@ -213,13 +213,13 @@ static void stage2_flush_vm(struct kvm *kvm)
 	int idx, bkt;
 
 	idx = srcu_read_lock(&kvm->srcu);
-	spin_lock(&kvm->mmu_lock);
+	write_lock(&kvm->mmu_lock);
 
 	slots = kvm_memslots(kvm);
 	kvm_for_each_memslot(memslot, bkt, slots)
 		stage2_flush_memslot(kvm, memslot);
 
-	spin_unlock(&kvm->mmu_lock);
+	write_unlock(&kvm->mmu_lock);
 	srcu_read_unlock(&kvm->srcu, idx);
 }
 
@@ -720,13 +720,13 @@ void stage2_unmap_vm(struct kvm *kvm)
 
 	idx = srcu_read_lock(&kvm->srcu);
 	mmap_read_lock(current->mm);
-	spin_lock(&kvm->mmu_lock);
+	write_lock(&kvm->mmu_lock);
 
 	slots = kvm_memslots(kvm);
 	kvm_for_each_memslot(memslot, bkt, slots)
 		stage2_unmap_memslot(kvm, memslot);
 
-	spin_unlock(&kvm->mmu_lock);
+	write_unlock(&kvm->mmu_lock);
 	mmap_read_unlock(current->mm);
 	srcu_read_unlock(&kvm->srcu, idx);
 }
@@ -736,14 +736,14 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
 	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
 	struct kvm_pgtable *pgt = NULL;
 
-	spin_lock(&kvm->mmu_lock);
+	write_lock(&kvm->mmu_lock);
 	pgt = mmu->pgt;
 	if (pgt) {
 		mmu->pgd_phys = 0;
 		mmu->pgt = NULL;
 		free_percpu(mmu->last_vcpu_ran);
 	}
-	spin_unlock(&kvm->mmu_lock);
+	write_unlock(&kvm->mmu_lock);
 
 	if (pgt) {
 		kvm_pgtable_stage2_destroy(pgt);
@@ -783,10 +783,10 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
 		if (ret)
 			break;
 
-		spin_lock(&kvm->mmu_lock);
+		write_lock(&kvm->mmu_lock);
 		ret = kvm_pgtable_stage2_map(pgt, addr, PAGE_SIZE, pa, prot,
 					     &cache);
-		spin_unlock(&kvm->mmu_lock);
+		write_unlock(&kvm->mmu_lock);
 		if (ret)
 			break;
 
@@ -834,9 +834,9 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot)
 	start = memslot->base_gfn << PAGE_SHIFT;
 	end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
 
-	spin_lock(&kvm->mmu_lock);
+	write_lock(&kvm->mmu_lock);
 	stage2_wp_range(&kvm->arch.mmu, start, end);
-	spin_unlock(&kvm->mmu_lock);
+	write_unlock(&kvm->mmu_lock);
 	kvm_flush_remote_tlbs(kvm);
 }
 
@@ -1212,7 +1212,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (exec_fault && device)
 		return -ENOEXEC;
 
-	spin_lock(&kvm->mmu_lock);
+	write_lock(&kvm->mmu_lock);
 	pgt = vcpu->arch.hw_mmu->pgt;
 	if (mmu_notifier_retry(kvm, mmu_seq))
 		goto out_unlock;
@@ -1271,7 +1271,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	}
 
 out_unlock:
-	spin_unlock(&kvm->mmu_lock);
+	write_unlock(&kvm->mmu_lock);
 	kvm_set_pfn_accessed(pfn);
 	kvm_release_pfn_clean(pfn);
 	return ret != -EAGAIN ? ret : 0;
@@ -1286,10 +1286,10 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
 
 	trace_kvm_access_fault(fault_ipa);
 
-	spin_lock(&vcpu->kvm->mmu_lock);
+	write_lock(&vcpu->kvm->mmu_lock);
 	mmu = vcpu->arch.hw_mmu;
 	kpte = kvm_pgtable_stage2_mkyoung(mmu->pgt, fault_ipa);
-	spin_unlock(&vcpu->kvm->mmu_lock);
+	write_unlock(&vcpu->kvm->mmu_lock);
 
 	pte = __pte(kpte);
 	if (pte_valid(pte))
@@ -1692,9 +1692,9 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 	gpa_t gpa = slot->base_gfn << PAGE_SHIFT;
 	phys_addr_t size = slot->npages << PAGE_SHIFT;
 
-	spin_lock(&kvm->mmu_lock);
+	write_lock(&kvm->mmu_lock);
 	unmap_stage2_range(&kvm->arch.mmu, gpa, size);
-	spin_unlock(&kvm->mmu_lock);
+	write_unlock(&kvm->mmu_lock);
 }
 
 /*
Signature · 1 line
2.34.1.703.g22d0c6ccf7-goog
Tested-by Reviewed-by on v2 1/3
#2 lore raw

Re: [PATCH v2 1/3] KVM: arm64: Use read/write spin lock for MMU protection

Hi Jing,

Quoted 5 lines · Jing Zhang

Replace MMU spinlock with rwlock and update all instances of the lock being acquired with a write lock acquisition.
Future commit will add a fast path for permission relaxation during dirty logging under a read lock.

Looking at the code, building it and running it, it seems that all instances of the lock are covered.

Thanks,
/fuad

Quoted 161 lines
---
 arch/arm64/include/asm/kvm_host.h |  2 ++
 arch/arm64/kvm/mmu.c              | 36 +++++++++++++++----------------
 2 files changed, 20 insertions(+), 18 deletions(-)
arch/arm64/include/asm/kvm_host.h +2 -0
index 3b44ea17af88..6c99c0335bae 100644
@@ -50,6 +50,8 @@
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
                                     KVM_DIRTY_LOG_INITIALLY_SET)
 
+#define KVM_HAVE_MMU_RWLOCK
+
 /*
  * Mode of operation configurable with kvm-arm.mode early param.
  * See Documentation/admin-guide/kernel-parameters.txt for more information.
arch/arm64/kvm/mmu.c +18 -19
index bc2aba953299..cafd5813c949 100644
@@ -58,7 +58,7 @@ static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr,
                        break;
 
                if (resched && next != end)
-                       cond_resched_lock(&kvm->mmu_lock);
+                       cond_resched_rwlock_write(&kvm->mmu_lock);
        } while (addr = next, addr != end);
 
        return ret;
@@ -179,7 +179,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64
        struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
        phys_addr_t end = start + size;
 
-       assert_spin_locked(&kvm->mmu_lock);
+       lockdep_assert_held_write(&kvm->mmu_lock);
        WARN_ON(size & ~PAGE_MASK);
        WARN_ON(stage2_apply_range(kvm, start, end, kvm_pgtable_stage2_unmap,
                                   may_block));
@@ -213,13 +213,13 @@ static void stage2_flush_vm(struct kvm *kvm)
        int idx, bkt;
 
        idx = srcu_read_lock(&kvm->srcu);
-       spin_lock(&kvm->mmu_lock);
+       write_lock(&kvm->mmu_lock);
 
        slots = kvm_memslots(kvm);
        kvm_for_each_memslot(memslot, bkt, slots)
                stage2_flush_memslot(kvm, memslot);
 
-       spin_unlock(&kvm->mmu_lock);
+       write_unlock(&kvm->mmu_lock);
        srcu_read_unlock(&kvm->srcu, idx);
 }
 
@@ -720,13 +720,13 @@ void stage2_unmap_vm(struct kvm *kvm)
 
        idx = srcu_read_lock(&kvm->srcu);
        mmap_read_lock(current->mm);
-       spin_lock(&kvm->mmu_lock);
+       write_lock(&kvm->mmu_lock);
 
        slots = kvm_memslots(kvm);
        kvm_for_each_memslot(memslot, bkt, slots)
                stage2_unmap_memslot(kvm, memslot);
 
-       spin_unlock(&kvm->mmu_lock);
+       write_unlock(&kvm->mmu_lock);
        mmap_read_unlock(current->mm);
        srcu_read_unlock(&kvm->srcu, idx);
 }
@@ -736,14 +736,14 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
        struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
        struct kvm_pgtable *pgt = NULL;
 
-       spin_lock(&kvm->mmu_lock);
+       write_lock(&kvm->mmu_lock);
        pgt = mmu->pgt;
        if (pgt) {
                mmu->pgd_phys = 0;
                mmu->pgt = NULL;
                free_percpu(mmu->last_vcpu_ran);
        }
-       spin_unlock(&kvm->mmu_lock);
+       write_unlock(&kvm->mmu_lock);
 
        if (pgt) {
                kvm_pgtable_stage2_destroy(pgt);
@@ -783,10 +783,10 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
                if (ret)
                        break;
 
-               spin_lock(&kvm->mmu_lock);
+               write_lock(&kvm->mmu_lock);
                ret = kvm_pgtable_stage2_map(pgt, addr, PAGE_SIZE, pa, prot,
                                             &cache);
-               spin_unlock(&kvm->mmu_lock);
+               write_unlock(&kvm->mmu_lock);
                if (ret)
                        break;
 
@@ -834,9 +834,9 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot)
        start = memslot->base_gfn << PAGE_SHIFT;
        end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
 
-       spin_lock(&kvm->mmu_lock);
+       write_lock(&kvm->mmu_lock);
        stage2_wp_range(&kvm->arch.mmu, start, end);
-       spin_unlock(&kvm->mmu_lock);
+       write_unlock(&kvm->mmu_lock);
        kvm_flush_remote_tlbs(kvm);
 }
 
@@ -1212,7 +1212,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        if (exec_fault && device)
                return -ENOEXEC;
 
-       spin_lock(&kvm->mmu_lock);
+       write_lock(&kvm->mmu_lock);
        pgt = vcpu->arch.hw_mmu->pgt;
        if (mmu_notifier_retry(kvm, mmu_seq))
                goto out_unlock;
@@ -1271,7 +1271,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        }
 
 out_unlock:
-       spin_unlock(&kvm->mmu_lock);
+       write_unlock(&kvm->mmu_lock);
        kvm_set_pfn_accessed(pfn);
        kvm_release_pfn_clean(pfn);
        return ret != -EAGAIN ? ret : 0;
@@ -1286,10 +1286,10 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
 
        trace_kvm_access_fault(fault_ipa);
 
-       spin_lock(&vcpu->kvm->mmu_lock);
+       write_lock(&vcpu->kvm->mmu_lock);
        mmu = vcpu->arch.hw_mmu;
        kpte = kvm_pgtable_stage2_mkyoung(mmu->pgt, fault_ipa);
-       spin_unlock(&vcpu->kvm->mmu_lock);
+       write_unlock(&vcpu->kvm->mmu_lock);
 
        pte = __pte(kpte);
        if (pte_valid(pte))
@@ -1692,9 +1692,9 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
        gpa_t gpa = slot->base_gfn << PAGE_SHIFT;
        phys_addr_t size = slot->npages << PAGE_SHIFT;
 
-       spin_lock(&kvm->mmu_lock);
+       write_lock(&kvm->mmu_lock);
        unmap_stage2_range(&kvm->arch.mmu, gpa, size);
-       spin_unlock(&kvm->mmu_lock);
+       write_unlock(&kvm->mmu_lock);
 }
 
 /*
--

2.34.1.703.g22d0c6ccf7-goog

Quoted 4 lines

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#3 lore raw

PATCH v2 2/3 KVM: arm64: Add fast path to handle permission relaxation during dirty logging

In reply to Jing Zhang

To reduce MMU lock contention during dirty logging, all permission relaxation operations would be performed under read lock.

---
 arch/arm64/kvm/mmu.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
arch/arm64/kvm/mmu.c +15 -2
index cafd5813c949..10df5d855d54 100644
@@ -1080,6 +1080,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	gfn_t gfn;
 	kvm_pfn_t pfn;
 	bool logging_active = memslot_is_logging(memslot);
+	bool logging_perm_fault = false;
 	unsigned long fault_level = kvm_vcpu_trap_get_fault_level(vcpu);
 	unsigned long vma_pagesize, fault_granule;
 	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
@@ -1114,6 +1115,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (logging_active) {
 		force_pte = true;
 		vma_shift = PAGE_SHIFT;
+		logging_perm_fault = (fault_status == FSC_PERM && write_fault);
 	} else {
 		vma_shift = get_vma_page_shift(vma, hva);
 	}
@@ -1212,7 +1214,15 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (exec_fault && device)
 		return -ENOEXEC;
 
-	write_lock(&kvm->mmu_lock);
+	/*
+	 * To reduce MMU contentions and enhance concurrency during dirty
+	 * logging dirty logging, only acquire read lock for permission
+	 * relaxation.
+	 */
+	if (logging_perm_fault)
+		read_lock(&kvm->mmu_lock);
+	else
+		write_lock(&kvm->mmu_lock);
 	pgt = vcpu->arch.hw_mmu->pgt;
 	if (mmu_notifier_retry(kvm, mmu_seq))
 		goto out_unlock;
@@ -1271,7 +1281,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	}
 
 out_unlock:
-	write_unlock(&kvm->mmu_lock);
+	if (logging_perm_fault)
+		read_unlock(&kvm->mmu_lock);
+	else
+		write_unlock(&kvm->mmu_lock);
 	kvm_set_pfn_accessed(pfn);
 	kvm_release_pfn_clean(pfn);
 	return ret != -EAGAIN ? ret : 0;
Signature · 1 line
2.34.1.703.g22d0c6ccf7-goog
Tested-by Reviewed-by on v2 2/3
#4 lore raw

Re: [PATCH v2 2/3] KVM: arm64: Add fast path to handle permission relaxation during dirty logging

Hi Jing,

Quoted 30 lines · Jing Zhang

To reduce MMU lock contention during dirty logging, all permission relaxation operations would be performed under read lock.

---
 arch/arm64/kvm/mmu.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
arch/arm64/kvm/mmu.c +2 -0
index cafd5813c949..10df5d855d54 100644
@@ -1080,6 +1080,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        gfn_t gfn;
        kvm_pfn_t pfn;
        bool logging_active = memslot_is_logging(memslot);
+       bool logging_perm_fault = false;
        unsigned long fault_level = kvm_vcpu_trap_get_fault_level(vcpu);
        unsigned long vma_pagesize, fault_granule;
        enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
@@ -1114,6 +1115,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        if (logging_active) {
                force_pte = true;
                vma_shift = PAGE_SHIFT;
+               logging_perm_fault = (fault_status == FSC_PERM && write_fault);
        } else {
                vma_shift = get_vma_page_shift(vma, hva);
        }

arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1221

Quoted 9 lines
arch/arm64/kvm/mmu.c +5 -1
@@ -1212,7 +1214,15 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        if (exec_fault && device)
                return -ENOEXEC;
 
-       write_lock(&kvm->mmu_lock);
+       /*
+        * To reduce MMU contentions and enhance concurrency during dirty
+        * logging dirty logging, only acquire read lock for permission
+        * relaxation.
+        */

A couple of nits:
"dirty logging" is repeated twice
s/contentions/contention

Other than that,

Thanks,
/fuad

Quoted 22 lines
arch/arm64/kvm/mmu.c +8 -2
+       if (logging_perm_fault)
+               read_lock(&kvm->mmu_lock);
+       else
+               write_lock(&kvm->mmu_lock);
        pgt = vcpu->arch.hw_mmu->pgt;
        if (mmu_notifier_retry(kvm, mmu_seq))
                goto out_unlock;
@@ -1271,7 +1281,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        }
 
 out_unlock:
-       write_unlock(&kvm->mmu_lock);
+       if (logging_perm_fault)
+               read_unlock(&kvm->mmu_lock);
+       else
+               write_unlock(&kvm->mmu_lock);
        kvm_set_pfn_accessed(pfn);
        kvm_release_pfn_clean(pfn);
        return ret != -EAGAIN ? ret : 0;
--

2.34.1.703.g22d0c6ccf7-goog

Quoted 4 lines

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#5 lore raw

PATCH v2 3/3 KVM: selftests: Add vgic initialization for dirty log perf test for ARM

In reply to Jing Zhang

For ARM64, if no vgic is setup before the dirty log perf test, the userspace irqchip would be used, which would affect the dirty log perf test result.

---
 tools/testing/selftests/kvm/dirty_log_perf_test.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
tools/testing/selftests/kvm/dirty_log_perf_test.c +10 -0
index 1954b964d1cf..b501338d9430 100644
@@ -18,6 +18,12 @@
 #include "test_util.h"
 #include "perf_test_util.h"
 #include "guest_modes.h"
+#ifdef __aarch64__
+#include "aarch64/vgic.h"
+
+#define GICD_BASE_GPA			0x8000000ULL
+#define GICR_BASE_GPA			0x80A0000ULL
+#endif
 
 /* How many host loops to run by default (one KVM_GET_DIRTY_LOG for each loop)*/
 #define TEST_HOST_LOOP_N		2UL
@@ -200,6 +206,10 @@ static void run_test(enum vm_guest_mode mode, void *arg)
 		vm_enable_cap(vm, &cap);
 	}
 
+#ifdef __aarch64__
+	vgic_v3_setup(vm, nr_vcpus, 64, GICD_BASE_GPA, GICR_BASE_GPA);
+#endif
+
 	/* Start the iterations */
 	iteration = 0;
 	host_quit = false;
Signature · 1 line
2.34.1.703.g22d0c6ccf7-goog
Tested-by Reviewed-by on v2 3/3
#6 lore raw

Re: [PATCH v2 3/3] KVM: selftests: Add vgic initialization for dirty log perf test for ARM

Hi Jing,

Quoted 15 lines · Jing Zhang

For ARM64, if no vgic is setup before the dirty log perf test, the userspace irqchip would be used, which would affect the dirty log perf test result.

---
 tools/testing/selftests/kvm/dirty_log_perf_test.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
tools/testing/selftests/kvm/dirty_log_perf_test.c +0 -0
index 1954b964d1cf..b501338d9430 100644

tools/testing/selftests/kvm/dirty_log_perf_test.c line 26

Quoted 9 lines
tools/testing/selftests/kvm/dirty_log_perf_test.c +6 -0
@@ -18,6 +18,12 @@
 #include "test_util.h"
 #include "perf_test_util.h"
 #include "guest_modes.h"
+#ifdef __aarch64__
+#include "aarch64/vgic.h"
+
+#define GICD_BASE_GPA                  0x8000000ULL
+#define GICR_BASE_GPA                  0x80A0000ULL
+#endif

As you'd mentioned in a previous email, these values are used by other tests as well as QEMU.

Thanks,
/fuad

Quoted 16 lines
tools/testing/selftests/kvm/dirty_log_perf_test.c +4 -1
 /* How many host loops to run by default (one KVM_GET_DIRTY_LOG for each loop)*/
 #define TEST_HOST_LOOP_N               2UL
@@ -200,6 +206,10 @@ static void run_test(enum vm_guest_mode mode, void *arg)
                vm_enable_cap(vm, &cap);
        }
 
+#ifdef __aarch64__
+       vgic_v3_setup(vm, nr_vcpus, 64, GICD_BASE_GPA, GICR_BASE_GPA);
+#endif
+
        /* Start the iterations */
        iteration = 0;
        host_quit = false;
--

2.34.1.703.g22d0c6ccf7-goog

Quoted 4 lines

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#7 lore raw
In reply to Jing Zhang

Hi Jing,

Quoted 47 lines · Jing Zhang

This patch is to reduce the performance degradation of guest workload during dirty logging on ARM64. A fast path is added to handle permission relaxation during dirty logging. The MMU lock is replaced with rwlock, by which all permision relaxations on leaf pte can be performed under the read lock. This greatly reduces the MMU lock contention during dirty logging. With this solution, the source guest workload performance degradation can be improved by more than 60%.

Problem:
* A Google internal live migration test shows that the source guest workload performance has >99% degradation for about 105 seconds, >50% degradation for about 112 seconds, >10% degradation for about 112 seconds on ARM64. This shows that most of the time, the guest workload degradtion is above 99%, which obviously needs some improvement compared to the test result on x86 (>99% for 6s, >50% for 9s, >10% for 27s).
* Tested H/W: Ampere Altra 3GHz, #CPU: 64, #Mem: 256GB, PageSize: 4K
* VM spec: #vCPU: 48, #Mem/vCPU: 4GB, PageSize: 4K, 2M hugepage backed

Analysis:
* We enabled CONFIG_LOCK_STAT in kernel and used dirty_log_perf_test to get

    the number of contentions of MMU lock and the "dirty memory time" on
    various VM spec. The "dirty memory time" is the time vCPU threads spent
    in KVM after fault. Higher "dirty memory time" means higher degradation
    to guest workload.
    '-m 2' specifies the mode "PA-bits:48,  VA-bits:48,  4K pages".
    By using test command
    ./dirty_log_perf_test -b 2G -m 2 -i 2 -s anonymous_hugetlb_2mb -v [#vCPU]
    Below are the results:
    +-------+------------------------+-----------------------+
    | #vCPU | dirty memory time (ms) | number of contentions |
    +-------+------------------------+-----------------------+
    | 1     | 926                    | 0                     |
    +-------+------------------------+-----------------------+
    | 2     | 1189                   | 4732558               |
    +-------+------------------------+-----------------------+
    | 4     | 2503                   | 11527185              |
    +-------+------------------------+-----------------------+
    | 8     | 5069                   | 24881677              |
    +-------+------------------------+-----------------------+
    | 16    | 10340                  | 50347956              |
    +-------+------------------------+-----------------------+
    | 32    | 20351                  | 100605720             |
    +-------+------------------------+-----------------------+
    | 64    | 40994                  | 201442478             |
    +-------+------------------------+-----------------------+
Quoted 5 lines

* From the test results above, the "dirty memory time" and the number of MMU lock contention scale with the number of vCPUs. That means all the dirty memory operations from all vCPU threads have been serialized by the MMU lock. Further analysis also shows that the permission relaxation during dirty logging is where vCPU threads get serialized.

I am curious about any changes to performance for this case (the base case) with the changes in patch 3.

Thanks,
/fuad

Quoted 84 lines

Solution:
* On ARM64, there is no mechanism as PML (Page Modification Logging) and the dirty-bit solution for dirty logging is much complicated compared to the write-protection solution. The straight way to reduce the guest performance degradation is to enhance the concurrency for the permission fault path during dirty logging.
* In this patch, we only put leaf PTE permission relaxation for dirty

    logging under read lock, all others would go under write lock.
    Below are the results based on the fast path solution:
    +-------+------------------------+
    | #vCPU | dirty memory time (ms) |
    +-------+------------------------+
    | 1     | 965                    |
    +-------+------------------------+
    | 2     | 1006                   |
    +-------+------------------------+
    | 4     | 1128                   |
    +-------+------------------------+
    | 8     | 2005                   |
    +-------+------------------------+
    | 16    | 3903                   |
    +-------+------------------------+
    | 32    | 7595                   |
    +-------+------------------------+
    | 64    | 15783                  |
    +-------+------------------------+

* Furtuer analysis shows that there is another bottleneck caused by the

    setup of the test code itself. The 3rd commit is meant to fix that by
    setting up vgic in the test code. With the test code fix, below are
    the results which show better improvement.
    +-------+------------------------+
    | #vCPU | dirty memory time (ms) |
    +-------+------------------------+
    | 1     | 803                    |
    +-------+------------------------+
    | 2     | 843                    |
    +-------+------------------------+
    | 4     | 942                    |
    +-------+------------------------+
    | 8     | 1458                   |
    +-------+------------------------+
    | 16    | 2853                   |
    +-------+------------------------+
    | 32    | 5886                   |
    +-------+------------------------+
    | 64    | 12190                  |
    +-------+------------------------+
    All "dirty memory time" has been reduced by more than 60% when the
    number of vCPU grows.

* Based on the solution, the test results from the Google internal live migration test also shows more than 60% improvement with >99% for 30s, >50% for 58s and >10% for 76s.

---

* v1 -> v2
- Renamed flag name from use_mmu_readlock to logging_perm_fault.
- Removed unnecessary check for fault_granule to use readlock.
* RFC -> v1
- Rebase to kvm/queue, commit fea31d169094
(KVM: x86/pmu: Fix available_event_types check for REF_CPU_CYCLES event)
- Moved the fast path in user_mem_abort, as suggested by Marc.
- Addressed other comments from Marc.

[v1] https://lore.kernel.org/all/[email protected]
[RFC] https://lore.kernel.org/all/[email protected]

---

Jing Zhang (3):
KVM: arm64: Use read/write spin lock for MMU protection
KVM: arm64: Add fast path to handle permission relaxation during dirty
logging
KVM: selftests: Add vgic initialization for dirty log perf test for
ARM

 arch/arm64/include/asm/kvm_host.h             |  2 +
 arch/arm64/kvm/mmu.c                          | 49 ++++++++++++-------
 .../selftests/kvm/dirty_log_perf_test.c       | 10 ++++
 3 files changed, 43 insertions(+), 18 deletions(-)
Quoted 8 lines

base-commit: fea31d1690945e6dd6c3e89ec5591490857bc3d4 --
2.34.1.703.g22d0c6ccf7-goog

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#8 lore raw

Hi Fuad,

Quoted 57 lines · Fuad Tabba

Hi Jing,

Quoted 47 lines · Jing Zhang

This patch is to reduce the performance degradation of guest workload during dirty logging on ARM64. A fast path is added to handle permission relaxation during dirty logging. The MMU lock is replaced with rwlock, by which all permision relaxations on leaf pte can be performed under the read lock. This greatly reduces the MMU lock contention during dirty logging. With this solution, the source guest workload performance degradation can be improved by more than 60%.

Problem:
* A Google internal live migration test shows that the source guest workload performance has >99% degradation for about 105 seconds, >50% degradation for about 112 seconds, >10% degradation for about 112 seconds on ARM64. This shows that most of the time, the guest workload degradtion is above 99%, which obviously needs some improvement compared to the test result on x86 (>99% for 6s, >50% for 9s, >10% for 27s).
* Tested H/W: Ampere Altra 3GHz, #CPU: 64, #Mem: 256GB, PageSize: 4K
* VM spec: #vCPU: 48, #Mem/vCPU: 4GB, PageSize: 4K, 2M hugepage backed

Analysis:
* We enabled CONFIG_LOCK_STAT in kernel and used dirty_log_perf_test to get

    the number of contentions of MMU lock and the "dirty memory time" on
    various VM spec. The "dirty memory time" is the time vCPU threads spent
    in KVM after fault. Higher "dirty memory time" means higher degradation
    to guest workload.
    '-m 2' specifies the mode "PA-bits:48,  VA-bits:48,  4K pages".
    By using test command
    ./dirty_log_perf_test -b 2G -m 2 -i 2 -s anonymous_hugetlb_2mb -v [#vCPU]
    Below are the results:
    +-------+------------------------+-----------------------+
    | #vCPU | dirty memory time (ms) | number of contentions |
    +-------+------------------------+-----------------------+
    | 1     | 926                    | 0                     |
    +-------+------------------------+-----------------------+
    | 2     | 1189                   | 4732558               |
    +-------+------------------------+-----------------------+
    | 4     | 2503                   | 11527185              |
    +-------+------------------------+-----------------------+
    | 8     | 5069                   | 24881677              |
    +-------+------------------------+-----------------------+
    | 16    | 10340                  | 50347956              |
    +-------+------------------------+-----------------------+
    | 32    | 20351                  | 100605720             |
    +-------+------------------------+-----------------------+
    | 64    | 40994                  | 201442478             |
    +-------+------------------------+-----------------------+
Quoted 5 lines

* From the test results above, the "dirty memory time" and the number of MMU lock contention scale with the number of vCPUs. That means all the dirty memory operations from all vCPU threads have been serialized by the MMU lock. Further analysis also shows that the permission relaxation during dirty logging is where vCPU threads get serialized.

Quoted 100 lines

I am curious about any changes to performance for this case (the base case) with the changes in patch 3.

Thanks,
/fuad

Quoted 92 lines

Solution:
* On ARM64, there is no mechanism as PML (Page Modification Logging) and the dirty-bit solution for dirty logging is much complicated compared to the write-protection solution. The straight way to reduce the guest performance degradation is to enhance the concurrency for the permission fault path during dirty logging.
* In this patch, we only put leaf PTE permission relaxation for dirty

    logging under read lock, all others would go under write lock.
    Below are the results based on the fast path solution:
    +-------+------------------------+
    | #vCPU | dirty memory time (ms) |
    +-------+------------------------+
    | 1     | 965                    |
    +-------+------------------------+
    | 2     | 1006                   |
    +-------+------------------------+
    | 4     | 1128                   |
    +-------+------------------------+
    | 8     | 2005                   |
    +-------+------------------------+
    | 16    | 3903                   |
    +-------+------------------------+
    | 32    | 7595                   |
    +-------+------------------------+
    | 64    | 15783                  |
    +-------+------------------------+

* Furtuer analysis shows that there is another bottleneck caused by the

    setup of the test code itself. The 3rd commit is meant to fix that by
    setting up vgic in the test code. With the test code fix, below are
    the results which show better improvement.
    +-------+------------------------+
    | #vCPU | dirty memory time (ms) |
    +-------+------------------------+
    | 1     | 803                    |
    +-------+------------------------+
    | 2     | 843                    |
    +-------+------------------------+
    | 4     | 942                    |
    +-------+------------------------+
    | 8     | 1458                   |
    +-------+------------------------+
    | 16    | 2853                   |
    +-------+------------------------+
    | 32    | 5886                   |
    +-------+------------------------+
    | 64    | 12190                  |
    +-------+------------------------+
    All "dirty memory time" has been reduced by more than 60% when the
    number of vCPU grows.

* Based on the solution, the test results from the Google internal live migration test also shows more than 60% improvement with >99% for 30s, >50% for 58s and >10% for 76s.

---

* v1 -> v2
- Renamed flag name from use_mmu_readlock to logging_perm_fault.
- Removed unnecessary check for fault_granule to use readlock.
* RFC -> v1
- Rebase to kvm/queue, commit fea31d169094
(KVM: x86/pmu: Fix available_event_types check for REF_CPU_CYCLES event)
- Moved the fast path in user_mem_abort, as suggested by Marc.
- Addressed other comments from Marc.

[v1] https://lore.kernel.org/all/[email protected]
[RFC] https://lore.kernel.org/all/[email protected]

---

Jing Zhang (3):
KVM: arm64: Use read/write spin lock for MMU protection
KVM: arm64: Add fast path to handle permission relaxation during dirty
logging
KVM: selftests: Add vgic initialization for dirty log perf test for
ARM

 arch/arm64/include/asm/kvm_host.h             |  2 +
 arch/arm64/kvm/mmu.c                          | 49 ++++++++++++-------
 .../selftests/kvm/dirty_log_perf_test.c       | 10 ++++
 3 files changed, 43 insertions(+), 18 deletions(-)

base-commit: fea31d1690945e6dd6c3e89ec5591490857bc3d4 --
2.34.1.703.g22d0c6ccf7-goog

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Thanks for all your reviews and testing.

Jing
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

maintainer Applied
#9 lore raw
In reply to Jing Zhang
Quoted 9 lines · Jing Zhang

This patch is to reduce the performance degradation of guest workload during dirty logging on ARM64. A fast path is added to handle permission relaxation during dirty logging. The MMU lock is replaced with rwlock, by which all permision relaxations on leaf pte can be performed under the read lock. This greatly reduces the MMU lock contention during dirty logging. With this solution, the source guest workload performance degradation can be improved by more than 60%.

[...]

Applied to next, thanks!

[1/3] KVM: arm64: Use read/write spin lock for MMU protection

      commit: fcc5bf89635a05e627cdd2e9ec52c989c8dfe2ab
[2/3] KVM: arm64: Add fast path to handle permission relaxation during dirty logging
      commit: f783ef1c0e82e4fc311a972472ff61f6d1d0e22d
[3/3] KVM: selftests: Add vgic initialization for dirty log perf test for ARM
      commit: c340f7899af6f83bd937f8838949bb32da54c8a4

Cheers,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.