arm64: tlb: call kvm_call_hyp once during kvm_tlb_flush_vmid_range

RFC PATCH In review yezhenyu (A) 2 participants 4 messages Feb 9 – Feb 16

Subsystems (2)

arm64 KVM arm64

lore

Share & tools

lore

yezhenyu (A)

author

2026-02-09 13:14 UTC #0 lore raw

RFC PATCH 1/1 arm64: tlb: call kvm_call_hyp once during kvm_tlb_flush_vmid_range

From 9982be89f55bd99b3683337223284f0011ed248e Mon Sep 17 00:00:00 2001
From: eillon <[email protected]>
Date: Mon, 9 Feb 2026 19:48:46 +0800
Subject: [RFC][PATCH v1] arm64: tlb: call kvm_call_hyp once during
kvm_tlb_flush_vmid_range

The kvm_tlb_flush_vmid_range() function is performance-critical during live migration, but there is a while loop when the system support flush tlb by range when the size is larger than MAX_TLBI_RANGE_PAGES.

This results in frequent entry to kvm_call_hyp() and then a large amount of time is spent in kvm_clear_dirty_log_protect() during migration(more than 50%). So, when the address range is large than MAX_TLBI_RANGE_PAGES, directly call __kvm_tlb_flush_vmid to optimize performance.

---
 arch/arm64/kvm/hyp/pgtable.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

arch/arm64/kvm/hyp/pgtable.c +8 ~~-10~~

index 874244df7..9da22b882 100644
@@ -675,21 +675,19 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt)
 void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
 				phys_addr_t addr, size_t size)
 {
-	unsigned long pages, inval_pages;
+	unsigned long pages = size >> PAGE_SHIFT;
 
-	if (!system_supports_tlb_range()) {
+	/*
+	 * This function is performance-critical during live migration;
+	 * thus, when the address range is large than MAX_TLBI_RANGE_PAGES,
+	 * directly call __kvm_tlb_flush_vmid to optimize performance.
+	 */
+	if (!system_supports_tlb_range() || pages > MAX_TLBI_RANGE_PAGES) {
 		kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
 		return;
 	}
 
-	pages = size >> PAGE_SHIFT;
-	while (pages > 0) {
-		inval_pages = min(pages, MAX_TLBI_RANGE_PAGES);
-		kvm_call_hyp(__kvm_tlb_flush_vmid_range, mmu, addr, inval_pages);
-
-		addr += inval_pages << PAGE_SHIFT;
-		pages -= inval_pages;
-	}
+	kvm_call_hyp(__kvm_tlb_flush_vmid_range, mmu, addr, pages);
 }
 
 #define KVM_S2_MEMATTR(pgt, attr) PAGE_S2_MEMATTR(attr, stage2_has_fwb(pgt))

--
2.43.0

Marc Zyngier

maintainer

2026-02-09 14:35 UTC #1 lore raw

Quoted 7 lines · yezhenyu (A)

From 9982be89f55bd99b3683337223284f0011ed248e Mon Sep 17 00:00:00 2001
From: eillon <[email protected]>
Date: Mon, 9 Feb 2026 19:48:46 +0800
Subject: [RFC][PATCH v1] arm64: tlb: call kvm_call_hyp once during
kvm_tlb_flush_vmid_range

Quoted 5 lines

The kvm_tlb_flush_vmid_range() function is performance-critical during live migration, but there is a while loop when the system support flush tlb by range when the size is larger than MAX_TLBI_RANGE_PAGES.
This results in frequent entry to kvm_call_hyp() and then a large

What is the cost of kvm_call_hyp()?

Quoted 2 lines

amount of time is spent in kvm_clear_dirty_log_protect() during migration(more than 50%).

50% of what time? The guest's run-time? The time spent doing TLBIs compared to the time spent in kvm_clear_dirty_log_protect()?

Quoted 3 lines

So, when the address range is large than MAX_TLBI_RANGE_PAGES, directly call __kvm_tlb_flush_vmid to optimize performance.

Multiple things here:

- there is no SoB, which means that patch cannot be considered for merging

- there is no data showing how this change improves the situation for a large enough set of workloads

- there is no description of a test that could be run on multiple implementations to check whether this change has a positive or negative impact

If you want to progress this sort of things, you will need to address these points.

Thanks,

M.

Signature · 1 line

Without deviation from the norm, progress is not possible.

yezhenyu (A)

author

2026-02-12 12:02 UTC #2 lore raw

Thanks for your review.

Quoted 9 lines · Marc Zyngier

Quoted 7 lines · yezhenyu (A)
From 9982be89f55bd99b3683337223284f0011ed248e Mon Sep 17 00:00:00 2001
From: eillon <[email protected]>
Date: Mon, 9 Feb 2026 19:48:46 +0800
Subject: [RFC][PATCH v1] arm64: tlb: call kvm_call_hyp once during
kvm_tlb_flush_vmid_range

Quoted 8 lines

Quoted 5 lines
The kvm_tlb_flush_vmid_range() function is performance-critical during live migration, but there is a while loop when the system support flush tlb by range when the size is larger than MAX_TLBI_RANGE_PAGES.
This results in frequent entry to kvm_call_hyp() and then a large
What is the cost of kvm_call_hyp()?

Most cost of kvm_tlb_flush_vmid_range() is __tlb_switch_to_host(), which is called in every __kvm_tlb_flush_vmid/__kvm_tlb_flush_vmid_range.

Quoted 6 lines

Quoted 2 lines
amount of time is spent in kvm_clear_dirty_log_protect() during migration(more than 50%).
50% of what time? The guest's run-time? The time spent doing TLBIs compared to the time spent in kvm_clear_dirty_log_protect()?

kvm_clear_dirty_log_protect() cost more than 50% time during ram_find_and_save_block(), but not every time.
I captured the flame graph during the live migration, and the distribution of several key functions is as follows(sorry I cannot transfer the SVG files outside my company):

    ram_find_and_save_block(): 84.01%
        memory_region_clear_dirty_bitmap(): 33.40%
            kvm_clear_dirty_log_protect(): 26.74%
                kvm_arch_flush_remote_tlbs_range(): 9.67%
                    __tlb_switch_to_host(): 9.51%
                kvm_arch_mmu_enable_log_dirty_pt_masked(): 9.38%
        ram_save_target_page_legacy(): 43.41%

The memory_region_clear_dirty_bitmap() cost about 40% of ram_find_and_save_block(), and the kvm_arch_flush_remote_tlbs_range() cost about 29% of memory_region_clear_dirty_bitmap().

And after the patch apply, the distribution of several key functions is as follows:

    ram_find_and_save_block(): 53.84%
        memory_region_clear_dirty_bitmap(): 2.28%
            kvm_clear_dirty_log_protect(): 1.75%
                kvm_arch_flush_remote_tlbs_range(): 0.03%
                    __tlb_switch_to_host(): 0.03%
                kvm_arch_mmu_enable_log_dirty_pt_masked(): 0.96%
        ram_save_target_page_legacy(): 38.97%

The memory_region_clear_dirty_bitmap() cost about 4% of ram_find_and_save_block(), and the kvm_arch_flush_remote_tlbs_range() cost about 1% of memory_region_clear_dirty_bitmap().

Quoted 9 lines

Quoted 3 lines
So, when the address range is large than MAX_TLBI_RANGE_PAGES, directly call __kvm_tlb_flush_vmid to optimize performance.
Multiple things here:
- there is no SoB, which means that patch cannot be considered for merging

If there are no other issues with this patch, I can resend it with the SoB (Signed-off-by) tag.

Quoted 6 lines

- there is no data showing how this change improves the situation for a large enough set of workloads
- there is no description of a test that could be run on multiple implementations to check whether this change has a positive or negative impact

This patch affected the migration bandwidth during the live migration. With the same physical bandwidth, the optimization effect of this patch can be observed by monitoring the real live migration bandwidth.

I have test this in an RDMA-like environment, the physical bandwidth is about 100GBps; without this patch, the migration bandwidth is below 10 GBps, and after this patch apply, the migration bandwidth can reach 50 GBps.

Quoted 8 lines

If you want to progress this sort of things, you will need to address these points.
Thanks,
	M.

Marc Zyngier

maintainer

2026-02-16 13:05 UTC #3 lore raw

Quoted 17 lines · yezhenyu (A)

Thanks for your review.
Quoted 13 lines · Marc Zyngier
Quoted 11 lines · yezhenyu (A)
From 9982be89f55bd99b3683337223284f0011ed248e Mon Sep 17 00:00:00 2001
From: eillon <[email protected]>
Date: Mon, 9 Feb 2026 19:48:46 +0800
Subject: [RFC][PATCH v1] arm64: tlb: call kvm_call_hyp once during
kvm_tlb_flush_vmid_range
The kvm_tlb_flush_vmid_range() function is performance-critical during live migration, but there is a while loop when the system support flush tlb by range when the size is larger than MAX_TLBI_RANGE_PAGES.

Quoted 7 lines

Quoted 4 lines
Quoted 1 line
This results in frequent entry to kvm_call_hyp() and then a large
What is the cost of kvm_call_hyp()?
Most cost of kvm_tlb_flush_vmid_range() is __tlb_switch_to_host(), which is called in every __kvm_tlb_flush_vmid/__kvm_tlb_flush_vmid_range.

That was not my question: you indicate that frequent calls to kvm_call_hyp() are making things costly. I find this assertion surprising, given that on a VHE system, this is exactly *nothing*.

Quoted 38 lines

Quoted 7 lines
Quoted 3 lines
amount of time is spent in kvm_clear_dirty_log_protect() during migration(more than 50%).
50% of what time? The guest's run-time? The time spent doing TLBIs compared to the time spent in kvm_clear_dirty_log_protect()?
kvm_clear_dirty_log_protect() cost more than 50% time during ram_find_and_save_block(), but not every time.
I captured the flame graph during the live migration, and the distribution of several key functions is as follows(sorry I cannot transfer the SVG files outside my company):
    ram_find_and_save_block(): 84.01%
        memory_region_clear_dirty_bitmap(): 33.40%
            kvm_clear_dirty_log_protect(): 26.74%
                kvm_arch_flush_remote_tlbs_range(): 9.67%
                    __tlb_switch_to_host(): 9.51%
                kvm_arch_mmu_enable_log_dirty_pt_masked(): 9.38%
        ram_save_target_page_legacy(): 43.41%
The memory_region_clear_dirty_bitmap() cost about 40% of ram_find_and_save_block(), and the kvm_arch_flush_remote_tlbs_range() cost about 29% of memory_region_clear_dirty_bitmap().
And after the patch apply, the distribution of several key functions is as follows:
    ram_find_and_save_block(): 53.84%
        memory_region_clear_dirty_bitmap(): 2.28%
            kvm_clear_dirty_log_protect(): 1.75%
                kvm_arch_flush_remote_tlbs_range(): 0.03%
                    __tlb_switch_to_host(): 0.03%
                kvm_arch_mmu_enable_log_dirty_pt_masked(): 0.96%
        ram_save_target_page_legacy(): 38.97%

Quoted 3 lines

The memory_region_clear_dirty_bitmap() cost about 4% of ram_find_and_save_block(), and the kvm_arch_flush_remote_tlbs_range() cost about 1% of memory_region_clear_dirty_bitmap().

What is ram_find_and_save_block()? userspace code?

Quoted 21 lines

Quoted 10 lines
Quoted 4 lines
So, when the address range is large than MAX_TLBI_RANGE_PAGES, directly call __kvm_tlb_flush_vmid to optimize performance.
Multiple things here:
- there is no SoB, which means that patch cannot be considered for merging
If there are no other issues with this patch, I can resend it with the SoB (Signed-off-by) tag.
Quoted 6 lines
- there is no data showing how this change improves the situation for a large enough set of workloads
- there is no description of a test that could be run on multiple implementations to check whether this change has a positive or negative impact

Quoted 8 lines

This patch affected the migration bandwidth during the live migration. With the same physical bandwidth, the optimization effect of this patch can be observed by monitoring the real live migration bandwidth.
I have test this in an RDMA-like environment, the physical bandwidth is about 100GBps; without this patch, the migration bandwidth is below 10 GBps, and after this patch apply, the migration bandwidth can reach 50 GBps.

Again: how can other people reproduce your findings? Please provide a test, and its exact configuration. If this truly results in a 5x improvement, it shouldn't be hard to reproduce.

Thanks,

M.

Signature · 1 line

Without deviation from the norm, progress is not possible.