Skip to contentSkip to conversation

KVM: arm64: Add support for FEAT_TLBIRANGE

PATCH Superseded Raghavendra Rao Ananta 3 participants 18 messages May 19, 2023 – Jun 2, 2023

Subsystems (3)

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v4 cover KVM: arm64: Add support for FEAT_TLBIRANGE

In certain code paths, KVM/ARM currently invalidates the entire VM's page-tables instead of just invalidating a necessary range. For example, when collapsing a table PTE to a block PTE, instead of iterating over each PTE and flushing them, KVM uses 'vmalls12e1is' TLBI operation to flush all the entries. This is inefficient since the guest would have to refill the TLBs again, even for the addresses that aren't covered by the table entry. The performance impact would scale poorly if many addresses in the VM is going through this remapping.

For architectures that implement FEAT_TLBIRANGE, KVM can replace such inefficient paths by performing the invalidations only on the range of addresses that are in scope. This series tries to achieve the same in the areas of stage-2 map, unmap and write-protecting the pages.

Patch-1 refactors the core arm64's __flush_tlb_range() to be used by other entities.

Patch-2 adds a range-based TLBI mechanism for KVM (VHE and nVHE).

Patch-3 implements the kvm_arch_flush_remote_tlbs_range() for arm64.

Patch-4 aims to flush only the memslot that undergoes a write-protect, instead of the entire VM.

Patch-5 operates on stage2_try_break_pte() to use the range based TLBI instructions when collapsing a table entry. The map path is the immediate consumer of this when KVM remaps a table entry into a block.

Patch-6 modifies the stage-2 unmap path in which, if the system supports FEAT_TLBIRANGE, the TLB invalidations are skipped during the page-table. walk. Instead it's done in one go after the entire walk is finished.

The series is based off of upstream v6.4-rc2, and applied David Matlack's common API for TLB invalidations[1] on top.

The performance evaluation was done on a hardware that supports FEAT_TLBIRANGE, on a VHE configuration, using a modified kvm_page_table_test.
The modified version updates the guest code in the ADJUST_MAPPINGS case to not only access this page but also to access up to 512 pages backwards
for every new page it iterates through. This is done to test the effect of TLBI misses after KVM has handled a fault.

The series captures the impact in the map and unmap paths as described above.

$ kvm_page_table_test -m 2 -v 128 -s anonymous_hugetlb_2mb -b $i
+--------+------------------------------+------------------------------+
| mem_sz |    ADJUST_MAPPINGS (s)       |      Unmap VM (s)            |
|  (GB)  | Baseline | Baseline + series | Baseline | Baseline + series |
+--------+----------|-------------------+------------------------------+
|   1    |   3.44   |   2.97            | 0.007     | 0.005            |
|   2    |   5.56   |   5.63            | 0.010     | 0.006            |
|   4    |  11.03   |  10.44            | 0.015     | 0.008            |
|   8    |  24.54   |  19.00            | 0.024     | 0.011            |
|  16    |  40.16   |  36.83            | 0.041     | 0.018            |
|  32    |  75.76   |  73.84            | 0.074     | 0.029            |
|  64    | 151.58   | 152.62            | 0.148     | 0.050            |
| 128    | 330.42   | 306.86            | 0.280     | 0.090            |
+--------+----------+-------------------+----------+-------------------+
$ kvm_page_table_test -m 2 -b 128G -s anonymous_hugetlb_2mb -v $i
+--------+------------------------------+
| vCPUs  |    ADJUST_MAPPINGS (s)       |
|        | Baseline | Baseline + series |
+--------+----------|-------------------+
|   1    | 138.69   | 135.58            |
|   2    | 138.77   | 137.54            |
|   4    | 162.57   | 135.82            |
|   8    | 154.92   | 143.67            |
|  16    | 122.02   | 118.86            |
|  32    | 119.99   | 118.81            |
|  64    | 190.70   | 169.36            |
| 128    | 330.42   | 306.86            |   
+--------+----------+-------------------+

For the ADJUST_MAPPINGS cases, which maps back the 4K table entries to 2M hugepages, the series sees an average improvement of ~7%. For unmapping 2M hugepages, we see at least a 3x improvement.

$ kvm_page_table_test -m 2 -b $i
+--------+------------------------------+
| mem_sz |      Unmap VM (s)            |
|  (GB)  | Baseline | Baseline + series |
+--------+------------------------------+
|   1    |  0.52    |  0.13             |
|   2    |  1.03    |  0.25             |
|   4    |  2.04    |  0.47             |
|   8    |  4.05    |  0.94             |
|  16    |  8.11    |  1.82             |
|  32    | 16.11    |  3.69             |
|  64    | 32.35    |  7.22             |
| 128    | 64.66    | 14.69             |   
+--------+----------+-------------------+

The series sees an average gain of 4x when the guest backed by PAGE_SIZE (4K) pages.

v4:
Thanks again, Oliver for all the comments
- Updated the __kvm_tlb_flush_vmid_range() implementation for nVHE to adjust with the modfied __tlb_switch_to_guest() that accepts a new 'bool nsh' arg.
- Renamed stage2_put_pte() to stage2_unmap_put_pte() and removed the 'skip_flush' argument.
- Defined stage2_unmap_defer_tlb_flush() to check if the PTE flushes can be deferred during the unmap table walk. It's being called from stage2_unmap_put_pte() and kvm_pgtable_stage2_unmap().
- Got rid of the 'struct stage2_unmap_data'.

v3:
https://lore.kernel.org/all/[email protected]/
Thanks, Oliver for all the suggestions.
- The core flush API (__kvm_tlb_flush_vmid_range()) now checks if the system support FEAT_TLBIRANGE or not, thus elimiating the redundancy in the upper layers.
- If FEAT_TLBIRANGE is not supported, the implementation falls back to invalidating all the TLB entries with the VMID, instead of doing an iterative flush for the range.
- The kvm_arch_flush_remote_tlbs_range() doesn't return -EOPNOTSUPP if the system doesn't implement FEAT_TLBIRANGE. It depends on __kvm_tlb_flush_vmid_range() to do take care of the decisions and return 0 regardless of the underlying feature support.
- __kvm_tlb_flush_vmid_range() doesn't take 'level' as input to calculate the 'stride'. Instead, it always assumes PAGE_SIZE.
- Fast unmap path is eliminated. Instead, the existing unmap walker is modified to skip the TLBIs during the walk, and do it all at once after the walk, using the range-based instructions.

v2:
https://lore.kernel.org/all/[email protected]/
- Rebased the series on top of David Matlack's series for common TLB invalidation API[1].
- Implement kvm_arch_flush_remote_tlbs_range() for arm64, by extending the support introduced by [1].
- Use kvm_flush_remote_tlbs_memslot() introduced by [1] to flush only the current memslot after write-protect.
- Modified the __kvm_tlb_flush_range() macro to accepts 'level' as an argument to calculate the 'stride' instead of just using PAGE_SIZE.
- Split the patch that introduces the range-based TLBI to KVM and the implementation of IPA-based invalidation into its own patches.
- Dropped the patch that tries to optimize the mmu notifiers paths.
- Rename the function kvm_table_pte_flush() to kvm_pgtable_stage2_flush_range(), and accept the range of addresses to flush. [Oliver]
- Drop the 'tlb_level' argument for stage2_try_break_pte() and directly pass '0' as 'tlb_level' to kvm_pgtable_stage2_flush_range(). [Oliver]

v1:
https://lore.kernel.org/all/[email protected]/

Thank you.
Raghavendra

[1]:
https://lore.kernel.org/linux-arm-kernel/[email protected]/

Raghavendra Rao Ananta (6):
arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range
KVM: arm64: Implement __kvm_tlb_flush_vmid_range()
KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()
KVM: arm64: Flush only the memslot after write-protect
KVM: arm64: Invalidate the table entries upon a range
KVM: arm64: Use TLBI range-based intructions for unmap

 arch/arm64/include/asm/kvm_asm.h   |   3 +
 arch/arm64/include/asm/kvm_host.h  |   3 +
 arch/arm64/include/asm/tlbflush.h  | 108 +++++++++++++++--------------
 arch/arm64/kvm/hyp/nvhe/hyp-main.c |  11 +++
 arch/arm64/kvm/hyp/nvhe/tlb.c      |  37 ++++++++++
 arch/arm64/kvm/hyp/pgtable.c       |  44 +++++++++---
 arch/arm64/kvm/hyp/vhe/tlb.c       |  35 ++++++++++
 arch/arm64/kvm/mmu.c               |  13 +++-
 8 files changed, 192 insertions(+), 62 deletions(-)
Signature · 1 line
2.40.1.698.g37aff9b760-goog
#1 lore raw

PATCH v4 1/6 arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range

Currently, the core TLB flush functionality of __flush_tlb_range() hardcodes vae1is (and variants) for the flush operation. In the upcoming patches, the KVM code reuses this core algorithm with ipas2e1is for range based TLB invalidations based on the IPA. Hence, extract the core flush functionality of __flush_tlb_range() into its own macro that accepts an 'op' argument to pass any TLBI operation, such that other callers (KVM) can benefit.

No functional changes intended.

---
 arch/arm64/include/asm/tlbflush.h | 108 +++++++++++++++---------------
 1 file changed, 55 insertions(+), 53 deletions(-)
arch/arm64/include/asm/tlbflush.h +55 -53
index 412a3b9a3c25d..4775378b6da1b 100644
@@ -278,14 +278,61 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
  */
 #define MAX_TLBI_OPS	PTRS_PER_PTE
 
+/* When the CPU does not support TLB range operations, flush the TLB
+ * entries one by one at the granularity of 'stride'. If the TLB
+ * range ops are supported, then:
+ *
+ * 1. If 'pages' is odd, flush the first page through non-range
+ *    operations;
+ *
+ * 2. For remaining pages: the minimum range granularity is decided
+ *    by 'scale', so multiple range TLBI operations may be required.
+ *    Start from scale = 0, flush the corresponding number of pages
+ *    ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it
+ *    until no pages left.
+ *
+ * Note that certain ranges can be represented by either num = 31 and
+ * scale or num = 0 and scale + 1. The loop below favours the latter
+ * since num is limited to 30 by the __TLBI_RANGE_NUM() macro.
+ */
+#define __flush_tlb_range_op(op, start, pages, stride,			\
+				asid, tlb_level, tlbi_user) do {	\
+	int num = 0;							\
+	int scale = 0;							\
+	unsigned long addr;						\
+									\
+	while (pages > 0) {						\
+		if (!system_supports_tlb_range() ||			\
+		    pages % 2 == 1) {					\
+			addr = __TLBI_VADDR(start, asid);		\
+			__tlbi_level(op, addr, tlb_level);		\
+			if (tlbi_user)					\
+				__tlbi_user_level(op, addr, tlb_level);	\
+			start += stride;				\
+			pages -= stride >> PAGE_SHIFT;			\
+			continue;					\
+		}							\
+									\
+		num = __TLBI_RANGE_NUM(pages, scale);			\
+		if (num >= 0) {						\
+			addr = __TLBI_VADDR_RANGE(start, asid, scale,	\
+						  num, tlb_level);	\
+			__tlbi(r##op, addr);				\
+			if (tlbi_user)					\
+				__tlbi_user(r##op, addr);		\
+			start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; \
+			pages -= __TLBI_RANGE_PAGES(num, scale);	\
+		}							\
+		scale++;						\
+	}								\
+} while (0)
+
 static inline void __flush_tlb_range(struct vm_area_struct *vma,
 				     unsigned long start, unsigned long end,
 				     unsigned long stride, bool last_level,
 				     int tlb_level)
 {
-	int num = 0;
-	int scale = 0;
-	unsigned long asid, addr, pages;
+	unsigned long asid, pages;
 
 	start = round_down(start, stride);
 	end = round_up(end, stride);
@@ -307,56 +354,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,
 	dsb(ishst);
 	asid = ASID(vma->vm_mm);
 
-	/*
-	 * When the CPU does not support TLB range operations, flush the TLB
-	 * entries one by one at the granularity of 'stride'. If the TLB
-	 * range ops are supported, then:
-	 *
-	 * 1. If 'pages' is odd, flush the first page through non-range
-	 *    operations;
-	 *
-	 * 2. For remaining pages: the minimum range granularity is decided
-	 *    by 'scale', so multiple range TLBI operations may be required.
-	 *    Start from scale = 0, flush the corresponding number of pages
-	 *    ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it
-	 *    until no pages left.
-	 *
-	 * Note that certain ranges can be represented by either num = 31 and
-	 * scale or num = 0 and scale + 1. The loop below favours the latter
-	 * since num is limited to 30 by the __TLBI_RANGE_NUM() macro.
-	 */
-	while (pages > 0) {
-		if (!system_supports_tlb_range() ||
-		    pages % 2 == 1) {
-			addr = __TLBI_VADDR(start, asid);
-			if (last_level) {
-				__tlbi_level(vale1is, addr, tlb_level);
-				__tlbi_user_level(vale1is, addr, tlb_level);
-			} else {
-				__tlbi_level(vae1is, addr, tlb_level);
-				__tlbi_user_level(vae1is, addr, tlb_level);
-			}
-			start += stride;
-			pages -= stride >> PAGE_SHIFT;
-			continue;
-		}
-
-		num = __TLBI_RANGE_NUM(pages, scale);
-		if (num >= 0) {
-			addr = __TLBI_VADDR_RANGE(start, asid, scale,
-						  num, tlb_level);
-			if (last_level) {
-				__tlbi(rvale1is, addr);
-				__tlbi_user(rvale1is, addr);
-			} else {
-				__tlbi(rvae1is, addr);
-				__tlbi_user(rvae1is, addr);
-			}
-			start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT;
-			pages -= __TLBI_RANGE_PAGES(num, scale);
-		}
-		scale++;
-	}
+	if (last_level)
+		__flush_tlb_range_op(vale1is, start, pages, stride, asid, tlb_level, true);
+	else
+		__flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true);
+
 	dsb(ish);
 }
 
Signature · 1 line
2.40.1.698.g37aff9b760-goog
#2 lore raw

PATCH v4 2/6 KVM: arm64: Implement __kvm_tlb_flush_vmid_range()

In reply to Raghavendra Rao Ananta

Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is in the map, unmap, and dirty-logging paths with ripas2e1is instead.

---
 arch/arm64/include/asm/kvm_asm.h   |  3 +++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++
 arch/arm64/kvm/hyp/nvhe/tlb.c      | 39 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/vhe/tlb.c       | 35 +++++++++++++++++++++++++++
 4 files changed, 88 insertions(+)
arch/arm64/include/asm/kvm_asm.h +3 -0
index 43c3bc0f9544d..33352d9399e32 100644
@@ -79,6 +79,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_init_vm,
 	__KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu,
 	__KVM_HOST_SMCCC_FUNC___pkvm_teardown_vm,
+	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range,
 };
 
 #define DECLARE_KVM_VHE_SYM(sym)	extern char sym[]
@@ -225,6 +226,8 @@ extern void __kvm_flush_vm_context(void);
 extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu);
 extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
 				     int level);
+extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+					phys_addr_t start, phys_addr_t end);
 extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 
 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +11 -0
index 728e01d4536b0..81d30737dc7c9 100644
@@ -125,6 +125,16 @@ static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt)
 	__kvm_tlb_flush_vmid_ipa(kern_hyp_va(mmu), ipa, level);
 }
 
+static void
+handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
+	DECLARE_REG(phys_addr_t, start, host_ctxt, 2);
+	DECLARE_REG(phys_addr_t, end, host_ctxt, 3);
+
+	__kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, end);
+}
+
 static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
@@ -315,6 +325,7 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__kvm_vcpu_run),
 	HANDLE_FUNC(__kvm_flush_vm_context),
 	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa),
+	HANDLE_FUNC(__kvm_tlb_flush_vmid_range),
 	HANDLE_FUNC(__kvm_tlb_flush_vmid),
 	HANDLE_FUNC(__kvm_flush_cpu_context),
 	HANDLE_FUNC(__kvm_timer_set_cntvoff),
arch/arm64/kvm/hyp/nvhe/tlb.c +39 -0
index 978179133f4b9..d4ea549c4b5c4 100644
@@ -130,6 +130,45 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
 	__tlb_switch_to_host(&cxt);
 }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, phys_addr_t end)
+{
+	struct tlb_inv_context cxt;
+	unsigned long pages, stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+	end = round_up(end, stride);
+	pages = (end - start) >> PAGE_SHIFT;
+
+	if (!system_supports_tlb_range() || pages >= MAX_TLBI_RANGE_PAGES) {
+		__kvm_tlb_flush_vmid(mmu);
+		return;
+	}
+
+	dsb(ishst);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt);
+
+	__flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false);
+
+	dsb(ish);
+	__tlbi(vmalle1is);
+	dsb(ish);
+	isb();
+
+	/* See the comment below in __kvm_tlb_flush_vmid_ipa() */
+	if (icache_is_vpipt())
+		icache_inval_all_pou();
+
+	__tlb_switch_to_host(&cxt);
+}
+
 void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
 {
 	struct tlb_inv_context cxt;
arch/arm64/kvm/hyp/vhe/tlb.c +35 -0
index 24cef9b87f9e9..f34d6dd9e4674 100644
@@ -111,6 +111,41 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
 	__tlb_switch_to_host(&cxt);
 }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, phys_addr_t end)
+{
+	struct tlb_inv_context cxt;
+	unsigned long pages, stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+	end = round_up(end, stride);
+	pages = (end - start) >> PAGE_SHIFT;
+
+	if (!system_supports_tlb_range() || pages >= MAX_TLBI_RANGE_PAGES) {
+		__kvm_tlb_flush_vmid(mmu);
+		return;
+	}
+
+	dsb(ishst);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt);
+
+	__flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false);
+
+	dsb(ish);
+	__tlbi(vmalle1is);
+	dsb(ish);
+	isb();
+
+	__tlb_switch_to_host(&cxt);
+}
+
 void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
 {
 	struct tlb_inv_context cxt;
Signature · 1 line
2.40.1.698.g37aff9b760-goog
maintainer
#3 lore raw

Re: [PATCH v4 2/6] KVM: arm64: Implement __kvm_tlb_flush_vmid_range()

Quoted 21 lines · Raghavendra Rao Ananta

Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is in the map, unmap, and dirty-logging paths with ripas2e1is instead.

---
 arch/arm64/include/asm/kvm_asm.h   |  3 +++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++
 arch/arm64/kvm/hyp/nvhe/tlb.c      | 39 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/vhe/tlb.c       | 35 +++++++++++++++++++++++++++
 4 files changed, 88 insertions(+)
arch/arm64/include/asm/kvm_asm.h +0 -0
index 43c3bc0f9544d..33352d9399e32 100644

arch/arm64/include/asm/kvm_asm.h enum __kvm_host_smccc_func { · line 82

Quoted 4 lines
arch/arm64/include/asm/kvm_asm.h +1 -0
@@ -79,6 +79,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_init_vm,
 	__KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu,
 	__KVM_HOST_SMCCC_FUNC___pkvm_teardown_vm,
+	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range,

nit: please keep this close to the other TLB operations.

Quoted 57 lines
arch/arm64/include/asm/kvm_asm.h +2 -0
 };
 
 #define DECLARE_KVM_VHE_SYM(sym)	extern char sym[]
@@ -225,6 +226,8 @@ extern void __kvm_flush_vm_context(void);
 extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu);
 extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
 				     int level);
+extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+					phys_addr_t start, phys_addr_t end);
 extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 
 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +11 -0
index 728e01d4536b0..81d30737dc7c9 100644
@@ -125,6 +125,16 @@ static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt)
 	__kvm_tlb_flush_vmid_ipa(kern_hyp_va(mmu), ipa, level);
 }
 
+static void
+handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
+	DECLARE_REG(phys_addr_t, start, host_ctxt, 2);
+	DECLARE_REG(phys_addr_t, end, host_ctxt, 3);
+
+	__kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, end);
+}
+
 static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
@@ -315,6 +325,7 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__kvm_vcpu_run),
 	HANDLE_FUNC(__kvm_flush_vm_context),
 	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa),
+	HANDLE_FUNC(__kvm_tlb_flush_vmid_range),
 	HANDLE_FUNC(__kvm_tlb_flush_vmid),
 	HANDLE_FUNC(__kvm_flush_cpu_context),
 	HANDLE_FUNC(__kvm_timer_set_cntvoff),
arch/arm64/kvm/hyp/nvhe/tlb.c +8 -0
index 978179133f4b9..d4ea549c4b5c4 100644
@@ -130,6 +130,45 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
 	__tlb_switch_to_host(&cxt);
 }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, phys_addr_t end)
+{
+	struct tlb_inv_context cxt;
+	unsigned long pages, stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at

arch/arm64/kvm/hyp/nvhe/tlb.c void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, · line 150

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +10 -0
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+	end = round_up(end, stride);
+	pages = (end - start) >> PAGE_SHIFT;
+
+	if (!system_supports_tlb_range() || pages >= MAX_TLBI_RANGE_PAGES) {
+		__kvm_tlb_flush_vmid(mmu);
+		return;

Why do we give up on "pages >= MAX_TLBI_RANGE_PAGES"? I see no rationale for it in the patch. My understanding is that this is the maximum representable as a range, in which case this is a programming error.

Or are you *on purpose* making the two equivalent?

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#4 lore raw

Re: [PATCH v4 2/6] KVM: arm64: Implement __kvm_tlb_flush_vmid_range()

Quoted 19 lines · Marc Zyngier
Quoted 16 lines · Raghavendra Rao Ananta

Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is in the map, unmap, and dirty-logging paths with ripas2e1is instead.

---
 arch/arm64/include/asm/kvm_asm.h   |  3 +++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++
 arch/arm64/kvm/hyp/nvhe/tlb.c      | 39 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/vhe/tlb.c       | 35 +++++++++++++++++++++++++++
 4 files changed, 88 insertions(+)
Quoted 12 lines

arch/arm64/include/asm/kvm_asm.h enum __kvm_host_smccc_func { · line 82

Quoted 9 lines
arch/arm64/include/asm/kvm_asm.h +1 -0
index 43c3bc0f9544d..33352d9399e32 100644
@@ -79,6 +79,7 @@ enum __kvm_host_smccc_func {
      __KVM_HOST_SMCCC_FUNC___pkvm_init_vm,
      __KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu,
      __KVM_HOST_SMCCC_FUNC___pkvm_teardown_vm,
+     __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range,

nit: please keep this close to the other TLB operations.

Sure, I'll reorder this.

Quoted 73 lines
Quoted 57 lines
arch/arm64/include/asm/kvm_asm.h +2 -0
 };
 
 #define DECLARE_KVM_VHE_SYM(sym)     extern char sym[]
@@ -225,6 +226,8 @@ extern void __kvm_flush_vm_context(void);
 extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu);
 extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
                                   int level);
+extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+                                     phys_addr_t start, phys_addr_t end);
 extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 
 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +11 -0
index 728e01d4536b0..81d30737dc7c9 100644
@@ -125,6 +125,16 @@ static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt)
      __kvm_tlb_flush_vmid_ipa(kern_hyp_va(mmu), ipa, level);
 }
 
+static void
+handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt)
+{
+     DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
+     DECLARE_REG(phys_addr_t, start, host_ctxt, 2);
+     DECLARE_REG(phys_addr_t, end, host_ctxt, 3);
+
+     __kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, end);
+}
+
 static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt)
 {
      DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
@@ -315,6 +325,7 @@ static const hcall_t host_hcall[] = {
      HANDLE_FUNC(__kvm_vcpu_run),
      HANDLE_FUNC(__kvm_flush_vm_context),
      HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa),
+     HANDLE_FUNC(__kvm_tlb_flush_vmid_range),
      HANDLE_FUNC(__kvm_tlb_flush_vmid),
      HANDLE_FUNC(__kvm_flush_cpu_context),
      HANDLE_FUNC(__kvm_timer_set_cntvoff),
arch/arm64/kvm/hyp/nvhe/tlb.c +8 -0
index 978179133f4b9..d4ea549c4b5c4 100644
@@ -130,6 +130,45 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
      __tlb_switch_to_host(&cxt);
 }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+                             phys_addr_t start, phys_addr_t end)
+{
+     struct tlb_inv_context cxt;
+     unsigned long pages, stride;
+
+     /*
+      * Since the range of addresses may not be mapped at

arch/arm64/kvm/hyp/nvhe/tlb.c void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, · line 150

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +10 -0
+      * the same level, assume the worst case as PAGE_SIZE
+      */
+     stride = PAGE_SIZE;
+     start = round_down(start, stride);
+     end = round_up(end, stride);
+     pages = (end - start) >> PAGE_SHIFT;
+
+     if (!system_supports_tlb_range() || pages >= MAX_TLBI_RANGE_PAGES) {
+             __kvm_tlb_flush_vmid(mmu);
+             return;

Why do we give up on "pages >= MAX_TLBI_RANGE_PAGES"? I see no rationale for it in the patch. My understanding is that this is the maximum representable as a range, in which case this is a programming error.

Quoted 2 lines

Or are you *on purpose* making the two equivalent?

Yes basically, I was trying to align the logic with what we have for __flush_tlb_range(). But, if you feel that it's mostly caused by a programming error, do we want to not do any flush at all?

Thank you.

Raghavendra

Quoted 6 lines

Thanks,

        M.

--
Without deviation from the norm, progress is not possible.

#5 lore raw

PATCH v4 3/6 KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

In reply to Raghavendra Rao Ananta

Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB.

---
 arch/arm64/include/asm/kvm_host.h |  3 +++
 arch/arm64/kvm/hyp/nvhe/tlb.c     |  4 +---
 arch/arm64/kvm/mmu.c              | 11 +++++++++++
 3 files changed, 15 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
index 81ab41b84f436..343fb530eea9c 100644
@@ -1081,6 +1081,9 @@ struct kvm *kvm_arch_alloc_vm(void);
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
 int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages);
+
 static inline bool kvm_vm_is_protected(struct kvm *kvm)
 {
 	return false;
arch/arm64/kvm/hyp/nvhe/tlb.c +1 -3
index d4ea549c4b5c4..d2c7c1bc6d441 100644
@@ -150,10 +150,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
 		return;
 	}
 
-	dsb(ishst);
-
 	/* Switch to requested VMID */
-	__tlb_switch_to_guest(mmu, &cxt);
+	__tlb_switch_to_guest(mmu, &cxt, false);
 
 	__flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false);
 
arch/arm64/kvm/mmu.c +11 -0
index d0a0d3dca9316..e3673b4c10292 100644
@@ -92,6 +92,17 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 	return 0;
 }
 
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages)
+{
+	phys_addr_t start, end;
+
+	start = start_gfn << PAGE_SHIFT;
+	end = (start_gfn + pages) << PAGE_SHIFT;
+
+	kvm_call_hyp(__kvm_tlb_flush_vmid_range, &kvm->arch.mmu, start, end);
+	return 0;
+}
+
 static bool kvm_is_device_pfn(unsigned long pfn)
 {
 	return !pfn_is_map_memory(pfn);
Signature · 1 line
2.40.1.698.g37aff9b760-goog
maintainer
#6 lore raw

Re: [PATCH v4 3/6] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

Quoted 30 lines · Raghavendra Rao Ananta

Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB.

---
 arch/arm64/include/asm/kvm_host.h |  3 +++
 arch/arm64/kvm/hyp/nvhe/tlb.c     |  4 +---
 arch/arm64/kvm/mmu.c              | 11 +++++++++++
 3 files changed, 15 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
index 81ab41b84f436..343fb530eea9c 100644
@@ -1081,6 +1081,9 @@ struct kvm *kvm_arch_alloc_vm(void);
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
 int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages);
+
 static inline bool kvm_vm_is_protected(struct kvm *kvm)
 {
 	return false;
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
index d4ea549c4b5c4..d2c7c1bc6d441 100644

arch/arm64/kvm/hyp/nvhe/tlb.c void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, · line 154

Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +1 -3
@@ -150,10 +150,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
 		return;
 	}
 
-	dsb(ishst);
-
 	/* Switch to requested VMID */
-	__tlb_switch_to_guest(mmu, &cxt);
+	__tlb_switch_to_guest(mmu, &cxt, false);

This hunk is in the wrong patch, isn't it?

Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
 	__flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false);
 
arch/arm64/kvm/mmu.c +0 -0
index d0a0d3dca9316..e3673b4c10292 100644
@@ -92,6 +92,17 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 	return 0;

arch/arm64/kvm/mmu.c int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 102

Quoted 10 lines
arch/arm64/kvm/mmu.c +8 -0
 }
 
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages)
+{
+	phys_addr_t start, end;
+
+	start = start_gfn << PAGE_SHIFT;
+	end = (start_gfn + pages) << PAGE_SHIFT;
+
+	kvm_call_hyp(__kvm_tlb_flush_vmid_range, &kvm->arch.mmu, start, end);

So that's the point that I think is not right. It is the MMU code that should drive the invalidation method, and not the HYP code. The HYP code should be as dumb as possible, and the logic should be kept in the MMU code.

So when a range invalidation is forwarded to HYP, it's a *valid* range invalidation. not something that can fallback to VMID-wide invalidation.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#7 lore raw

Re: [PATCH v4 3/6] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

Quoted 36 lines · Marc Zyngier
Quoted 33 lines · Raghavendra Rao Ananta

Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB.

---
 arch/arm64/include/asm/kvm_host.h |  3 +++
 arch/arm64/kvm/hyp/nvhe/tlb.c     |  4 +---
 arch/arm64/kvm/mmu.c              | 11 +++++++++++
 3 files changed, 15 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
index 81ab41b84f436..343fb530eea9c 100644
@@ -1081,6 +1081,9 @@ struct kvm *kvm_arch_alloc_vm(void);
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
 int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages);
+
 static inline bool kvm_vm_is_protected(struct kvm *kvm)
 {
      return false;
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
index d4ea549c4b5c4..d2c7c1bc6d441 100644
@@ -150,10 +150,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
              return;
      }
 
Quoted 8 lines

arch/arm64/kvm/hyp/nvhe/tlb.c void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, · line 154

Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +1 -3
-     dsb(ishst);
-
      /* Switch to requested VMID */
-     __tlb_switch_to_guest(mmu, &cxt);
+     __tlb_switch_to_guest(mmu, &cxt, false);

This hunk is in the wrong patch, isn't it?

Ah, you are right. It should be part of the previous patch. I think I introduced it accidentally when I rebased the series. I'll remove it in the next spin.

Quoted 25 lines
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
      __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false);
 
arch/arm64/kvm/mmu.c +0 -0
index d0a0d3dca9316..e3673b4c10292 100644
@@ -92,6 +92,17 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
      return 0;

arch/arm64/kvm/mmu.c int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 102

Quoted 10 lines
arch/arm64/kvm/mmu.c +8 -0
 }
 
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages)
+{
+     phys_addr_t start, end;
+
+     start = start_gfn << PAGE_SHIFT;
+     end = (start_gfn + pages) << PAGE_SHIFT;
+
+     kvm_call_hyp(__kvm_tlb_flush_vmid_range, &kvm->arch.mmu, start, end);

So that's the point that I think is not right. It is the MMU code that should drive the invalidation method, and not the HYP code. The HYP code should be as dumb as possible, and the logic should be kept in the MMU code.

Quoted 3 lines

So when a range invalidation is forwarded to HYP, it's a *valid* range invalidation. not something that can fallback to VMID-wide invalidation.

I'm guessing that you are referring to patch-2. Do you recommend moving the 'pages >= MAX_TLBI_RANGE_PAGES' logic here and simply return an error? How about for the other check:
system_supports_tlb_range()?
The idea was for __kvm_tlb_flush_vmid_range() to also implement a fallback mechanism in case the system doesn't support the range-based instructions. But if we end up calling __kvm_tlb_flush_vmid_range() from multiple cases, we'd end up duplicating the checks. WDYT?

Quoted 6 lines

Thanks,

        M.

--
Without deviation from the norm, progress is not possible.

maintainer
#8 lore raw

Re: [PATCH v4 3/6] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

Quoted 79 lines · Raghavendra Rao Ananta
Quoted 36 lines · Marc Zyngier
Quoted 33 lines · Raghavendra Rao Ananta

Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB.

---
 arch/arm64/include/asm/kvm_host.h |  3 +++
 arch/arm64/kvm/hyp/nvhe/tlb.c     |  4 +---
 arch/arm64/kvm/mmu.c              | 11 +++++++++++
 3 files changed, 15 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
index 81ab41b84f436..343fb530eea9c 100644
@@ -1081,6 +1081,9 @@ struct kvm *kvm_arch_alloc_vm(void);
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
 int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages);
+
 static inline bool kvm_vm_is_protected(struct kvm *kvm)
 {
      return false;
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
index d4ea549c4b5c4..d2c7c1bc6d441 100644
@@ -150,10 +150,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
              return;
      }
 
Quoted 8 lines

arch/arm64/kvm/hyp/nvhe/tlb.c void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, · line 154

Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +1 -3
-     dsb(ishst);
-
      /* Switch to requested VMID */
-     __tlb_switch_to_guest(mmu, &cxt);
+     __tlb_switch_to_guest(mmu, &cxt, false);

This hunk is in the wrong patch, isn't it?

Ah, you are right. It should be part of the previous patch. I think I introduced it accidentally when I rebased the series. I'll remove it in the next spin.

Quoted 25 lines
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
      __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false);
 
arch/arm64/kvm/mmu.c +0 -0
index d0a0d3dca9316..e3673b4c10292 100644
@@ -92,6 +92,17 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
      return 0;

arch/arm64/kvm/mmu.c int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 102

Quoted 10 lines
arch/arm64/kvm/mmu.c +8 -0
 }
 
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages)
+{
+     phys_addr_t start, end;
+
+     start = start_gfn << PAGE_SHIFT;
+     end = (start_gfn + pages) << PAGE_SHIFT;
+
+     kvm_call_hyp(__kvm_tlb_flush_vmid_range, &kvm->arch.mmu, start, end);

So that's the point that I think is not right. It is the MMU code that should drive the invalidation method, and not the HYP code. The HYP code should be as dumb as possible, and the logic should be kept in the MMU code.

Quoted 3 lines

So when a range invalidation is forwarded to HYP, it's a *valid* range invalidation. not something that can fallback to VMID-wide invalidation.

Quoted 8 lines

I'm guessing that you are referring to patch-2. Do you recommend moving the 'pages >= MAX_TLBI_RANGE_PAGES' logic here and simply return an error? How about for the other check:
system_supports_tlb_range()?
The idea was for __kvm_tlb_flush_vmid_range() to also implement a fallback mechanism in case the system doesn't support the range-based instructions. But if we end up calling __kvm_tlb_flush_vmid_range() from multiple cases, we'd end up duplicating the checks. WDYT?

My take is that there should be a single helper deciding to issue either a number of range-based TLBIs depending on start/end, or a single VMID-based TLBI. Having multiple calling sites is not a problem, and even if that code gets duplicated, big deal.

But a hypercall that falls back to global invalidation based on a range evaluation error (more than MAX_TLBI_RANGE_PAGES) is papering over a latent bug.

There should be no logic whatsoever in any of the two tlb.c files. Only a switch to the correct context, and the requested invalidation, which *must* be architecturally correct.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#9 lore raw

Re: [PATCH v4 3/6] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

Quoted 91 lines · Marc Zyngier
Quoted 79 lines · Raghavendra Rao Ananta
Quoted 36 lines · Marc Zyngier
Quoted 33 lines · Raghavendra Rao Ananta

Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB.

---
 arch/arm64/include/asm/kvm_host.h |  3 +++
 arch/arm64/kvm/hyp/nvhe/tlb.c     |  4 +---
 arch/arm64/kvm/mmu.c              | 11 +++++++++++
 3 files changed, 15 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
index 81ab41b84f436..343fb530eea9c 100644
@@ -1081,6 +1081,9 @@ struct kvm *kvm_arch_alloc_vm(void);
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
 int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages);
+
 static inline bool kvm_vm_is_protected(struct kvm *kvm)
 {
      return false;
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
index d4ea549c4b5c4..d2c7c1bc6d441 100644
@@ -150,10 +150,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
              return;
      }
 
Quoted 8 lines

arch/arm64/kvm/hyp/nvhe/tlb.c void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, · line 154

Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +1 -3
-     dsb(ishst);
-
      /* Switch to requested VMID */
-     __tlb_switch_to_guest(mmu, &cxt);
+     __tlb_switch_to_guest(mmu, &cxt, false);

This hunk is in the wrong patch, isn't it?

Ah, you are right. It should be part of the previous patch. I think I introduced it accidentally when I rebased the series. I'll remove it in the next spin.

Quoted 25 lines
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
      __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false);
 
arch/arm64/kvm/mmu.c +0 -0
index d0a0d3dca9316..e3673b4c10292 100644
@@ -92,6 +92,17 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
      return 0;

arch/arm64/kvm/mmu.c int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 102

Quoted 10 lines
arch/arm64/kvm/mmu.c +8 -0
 }
 
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages)
+{
+     phys_addr_t start, end;
+
+     start = start_gfn << PAGE_SHIFT;
+     end = (start_gfn + pages) << PAGE_SHIFT;
+
+     kvm_call_hyp(__kvm_tlb_flush_vmid_range, &kvm->arch.mmu, start, end);

So that's the point that I think is not right. It is the MMU code that should drive the invalidation method, and not the HYP code. The HYP code should be as dumb as possible, and the logic should be kept in the MMU code.

Quoted 3 lines

So when a range invalidation is forwarded to HYP, it's a *valid* range invalidation. not something that can fallback to VMID-wide invalidation.

Quoted 8 lines

I'm guessing that you are referring to patch-2. Do you recommend moving the 'pages >= MAX_TLBI_RANGE_PAGES' logic here and simply return an error? How about for the other check:
system_supports_tlb_range()?
The idea was for __kvm_tlb_flush_vmid_range() to also implement a fallback mechanism in case the system doesn't support the range-based instructions. But if we end up calling __kvm_tlb_flush_vmid_range() from multiple cases, we'd end up duplicating the checks. WDYT?

Quoted 5 lines

My take is that there should be a single helper deciding to issue either a number of range-based TLBIs depending on start/end, or a single VMID-based TLBI. Having multiple calling sites is not a problem, and even if that code gets duplicated, big deal.

Hypothetically, if I move the check to this patch and return an error if this situation occurs, since I'm dependending on David's common MMU code [1], kvm_main.c would end of calling kvm_flush_remote_tlbs() and we'd be doing a VMID-based TLBI.
One idea would be to issue a WARN_ON() and return 0 so that we don't issue any TLBIs. Thoughts?

Quoted 4 lines

But a hypercall that falls back to global invalidation based on a range evaluation error (more than MAX_TLBI_RANGE_PAGES) is papering over a latent bug.

If I understand this correctly, MAX_TLBI_RANGE_PAGES is specifically the capacity of the range-based instructions itself, isn't it? Is it incorrect for the caller to request a higher range be invalidated even on systems that do not support these instructions? Probably that's why __flush_tlb_range() falls back to a global flush when the range request is exceeded?

Thank you.
Raghavendra

[1]: https://lore.kernel.org/linux-arm-kernel/[email protected]/

Quoted 10 lines

There should be no logic whatsoever in any of the two tlb.c files. Only a switch to the correct context, and the requested invalidation, which *must* be architecturally correct.

Thanks,

        M.

--
Without deviation from the norm, progress is not possible.

maintainer
#10 lore raw

Re: [PATCH v4 3/6] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

Quoted 98 lines · Raghavendra Rao Ananta
Quoted 91 lines · Marc Zyngier
Quoted 79 lines · Raghavendra Rao Ananta
Quoted 36 lines · Marc Zyngier
Quoted 33 lines · Raghavendra Rao Ananta

Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB.

---
 arch/arm64/include/asm/kvm_host.h |  3 +++
 arch/arm64/kvm/hyp/nvhe/tlb.c     |  4 +---
 arch/arm64/kvm/mmu.c              | 11 +++++++++++
 3 files changed, 15 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
index 81ab41b84f436..343fb530eea9c 100644
@@ -1081,6 +1081,9 @@ struct kvm *kvm_arch_alloc_vm(void);
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
 int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages);
+
 static inline bool kvm_vm_is_protected(struct kvm *kvm)
 {
      return false;
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
index d4ea549c4b5c4..d2c7c1bc6d441 100644
@@ -150,10 +150,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
              return;
      }
 
Quoted 8 lines

arch/arm64/kvm/hyp/nvhe/tlb.c void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, · line 154

Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +1 -3
-     dsb(ishst);
-
      /* Switch to requested VMID */
-     __tlb_switch_to_guest(mmu, &cxt);
+     __tlb_switch_to_guest(mmu, &cxt, false);

This hunk is in the wrong patch, isn't it?

Ah, you are right. It should be part of the previous patch. I think I introduced it accidentally when I rebased the series. I'll remove it in the next spin.

Quoted 25 lines
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/tlb.c +0 -0
      __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false);
 
arch/arm64/kvm/mmu.c +0 -0
index d0a0d3dca9316..e3673b4c10292 100644
@@ -92,6 +92,17 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
      return 0;

arch/arm64/kvm/mmu.c int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 102

Quoted 10 lines
arch/arm64/kvm/mmu.c +8 -0
 }
 
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages)
+{
+     phys_addr_t start, end;
+
+     start = start_gfn << PAGE_SHIFT;
+     end = (start_gfn + pages) << PAGE_SHIFT;
+
+     kvm_call_hyp(__kvm_tlb_flush_vmid_range, &kvm->arch.mmu, start, end);

So that's the point that I think is not right. It is the MMU code that should drive the invalidation method, and not the HYP code. The HYP code should be as dumb as possible, and the logic should be kept in the MMU code.

Quoted 3 lines

So when a range invalidation is forwarded to HYP, it's a *valid* range invalidation. not something that can fallback to VMID-wide invalidation.

Quoted 8 lines

I'm guessing that you are referring to patch-2. Do you recommend moving the 'pages >= MAX_TLBI_RANGE_PAGES' logic here and simply return an error? How about for the other check:
system_supports_tlb_range()?
The idea was for __kvm_tlb_flush_vmid_range() to also implement a fallback mechanism in case the system doesn't support the range-based instructions. But if we end up calling __kvm_tlb_flush_vmid_range() from multiple cases, we'd end up duplicating the checks. WDYT?

Quoted 5 lines

My take is that there should be a single helper deciding to issue either a number of range-based TLBIs depending on start/end, or a single VMID-based TLBI. Having multiple calling sites is not a problem, and even if that code gets duplicated, big deal.

Quoted 6 lines

Hypothetically, if I move the check to this patch and return an error if this situation occurs, since I'm dependending on David's common MMU code [1], kvm_main.c would end of calling kvm_flush_remote_tlbs() and we'd be doing a VMID-based TLBI.
One idea would be to issue a WARN_ON() and return 0 so that we don't issue any TLBIs. Thoughts?

Then we should fix the infrastructure. And no, not issuing TLBs is not an acceptable outcome. Might as well panic the kernel, because silent memory corruption is not something I'm willing to entertain.

My take is as follows:

- either the core code doesn't specify a range, and we blow the whole thing as we do today

- or it specifies a range, and we apply range invalidation as permitted by the architecture and the implementation (either a *series* of range invalidation, or a full VMID invalidation).

As for the "common MMU" stuff, something such as "flush_remote_tlbs" really shows that this code isn't common at all. It is just x86 creep, and I have zero problem ignoring it.

Quoted 11 lines
Quoted 5 lines

But a hypercall that falls back to global invalidation based on a range evaluation error (more than MAX_TLBI_RANGE_PAGES) is papering over a latent bug.

If I understand this correctly, MAX_TLBI_RANGE_PAGES is specifically the capacity of the range-based instructions itself, isn't it? Is it incorrect for the caller to request a higher range be invalidated even on systems that do not support these instructions? Probably that's why __flush_tlb_range() falls back to a global flush when the range request is exceeded?

This is indeed the architectural limit. But invalidating the whole VMID isn't necessarily the right thing either. If you can preserve some of the TLBs by only issuing a couple of range invalidation that span *in total* more than MAX_TLBI_RANGE_PAGES.

Again, what I'm objecting to is the silent upgrading to VMID-wide invalidation being hidden away in the HYP code. That's just wrong. The HYP code is not the place for abstraction.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#11 lore raw

PATCH v4 4/6 KVM: arm64: Flush only the memslot after write-protect

In reply to Raghavendra Rao Ananta

After write-protecting the region, currently KVM invalidates the entire TLB entries using kvm_flush_remote_tlbs(). Instead, scope the invalidation only to the targeted memslot. If supported, the architecture would use the range-based TLBI instructions to flush the memslot or else fallback to flushing all of the TLBs.

---
 arch/arm64/kvm/mmu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/mmu.c +1 -1
index e3673b4c10292..2ea6eb4ea763e 100644
@@ -996,7 +996,7 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot)
 	write_lock(&kvm->mmu_lock);
 	stage2_wp_range(&kvm->arch.mmu, start, end);
 	write_unlock(&kvm->mmu_lock);
-	kvm_flush_remote_tlbs(kvm);
+	kvm_flush_remote_tlbs_memslot(kvm, memslot);
 }
 
 /**
Signature · 1 line
2.40.1.698.g37aff9b760-goog
#12 lore raw

PATCH v4 5/6 KVM: arm64: Invalidate the table entries upon a range

In reply to Raghavendra Rao Ananta

Currently, during the operations such as a hugepage collapse, KVM would flush the entire VM's context using 'vmalls12e1is' TLBI operation. Specifically, if the VM is faulting on many hugepages (say after dirty-logging), it creates a performance penalty for the guest whose pages have already been faulted earlier as they would have to refill their TLBs again.

Instead, call __kvm_tlb_flush_vmid_range() for table entries. If the system supports it, only the required range will be flushed. Else, it'll fallback to the previous mechanism.

---
 arch/arm64/kvm/hyp/pgtable.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +6 -3
index 3d61bd3e591d2..b8f0dbd12f773 100644
@@ -745,10 +745,13 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
 	 * Perform the appropriate TLB invalidation based on the evicted pte
 	 * value (if any).
 	 */
-	if (kvm_pte_table(ctx->old, ctx->level))
-		kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
-	else if (kvm_pte_valid(ctx->old))
+	if (kvm_pte_table(ctx->old, ctx->level)) {
+		u64 end = ctx->addr + kvm_granule_size(ctx->level);
+
+		kvm_call_hyp(__kvm_tlb_flush_vmid_range, mmu, ctx->addr, end);
+	} else if (kvm_pte_valid(ctx->old)) {
 		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+	}
 
 	if (stage2_pte_is_counted(ctx->old))
 		mm_ops->put_page(ctx->ptep);
Signature · 1 line
2.40.1.698.g37aff9b760-goog
#13 lore raw

PATCH v4 6/6 KVM: arm64: Use TLBI range-based intructions for unmap

In reply to Raghavendra Rao Ananta

The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck.

Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition this upon S2FWB in order to avoid walking the page-table again to perform the CMOs after issuing the TLBI.

Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic.

---
 arch/arm64/kvm/hyp/pgtable.c | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +30 -5
index b8f0dbd12f773..5832ee3418fb0 100644
@@ -771,16 +771,34 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n
 	smp_store_release(ctx->ptep, new);
 }
 
-static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu,
-			   struct kvm_pgtable_mm_ops *mm_ops)
+static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
 {
+	/*
+	 * If FEAT_TLBIRANGE is implemented, defer the individial PTE
+	 * TLB invalidations until the entire walk is finished, and
+	 * then use the range-based TLBI instructions to do the
+	 * invalidations. Condition this upon S2FWB in order to avoid
+	 * a page-table walk again to perform the CMOs after TLBI.
+	 */
+	return system_supports_tlb_range() && stage2_has_fwb(pgt);
+}
+
+static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
+				struct kvm_s2_mmu *mmu,
+				struct kvm_pgtable_mm_ops *mm_ops)
+{
+	struct kvm_pgtable *pgt = ctx->arg;
+
 	/*
 	 * Clear the existing PTE, and perform break-before-make with
 	 * TLB maintenance if it was valid.
 	 */
 	if (kvm_pte_valid(ctx->old)) {
 		kvm_clear_pte(ctx->ptep);
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+
+		if (!stage2_unmap_defer_tlb_flush(pgt))
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+					ctx->addr, ctx->level);
 	}
 
 	mm_ops->put_page(ctx->ptep);
@@ -1015,7 +1033,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	 * block entry and rely on the remaining portions being faulted
 	 * back lazily.
 	 */
-	stage2_put_pte(ctx, mmu, mm_ops);
+	stage2_unmap_put_pte(ctx, mmu, mm_ops);
 
 	if (need_flush && mm_ops->dcache_clean_inval_poc)
 		mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops),
@@ -1029,13 +1047,20 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 
 int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
 {
+	int ret;
 	struct kvm_pgtable_walker walker = {
 		.cb	= stage2_unmap_walker,
 		.arg	= pgt,
 		.flags	= KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
 	};
 
-	return kvm_pgtable_walk(pgt, addr, size, &walker);
+	ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+	if (stage2_unmap_defer_tlb_flush(pgt))
+		/* Perform the deferred TLB invalidations */
+		kvm_call_hyp(__kvm_tlb_flush_vmid_range, pgt->mmu,
+				addr, addr + size);
+
+	return ret;
 }
 
 struct stage2_attr_data {
Signature · 1 line
2.40.1.698.g37aff9b760-goog
#14 lore raw

Re: [PATCH v4 6/6] KVM: arm64: Use TLBI range-based intructions for unmap

Quoted 10 lines · Raghavendra Rao Ananta

The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck.

Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition this upon S2FWB in order to avoid walking the page-table again to perform the CMOs after issuing the TLBI.

nit: Rather than discussing a theoretical CMO walker, I think this is more readable if you mention the existing behavior of the walker.

Condition deferred TLB invalidation on the system supporting FWB, as the optimization is entirely pointless when the unmap walker needs to perform CMOs.

Quoted 14 lines

Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic.

---
 arch/arm64/kvm/hyp/pgtable.c | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +0 -0
index b8f0dbd12f773..5832ee3418fb0 100644

arch/arm64/kvm/hyp/pgtable.c static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n · line 777

Quoted 9 lines
arch/arm64/kvm/hyp/pgtable.c +3 -2
@@ -771,16 +771,34 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n
 	smp_store_release(ctx->ptep, new);
 }
 
-static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu,
-			   struct kvm_pgtable_mm_ops *mm_ops)
+static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
 {
+	/*
+	 * If FEAT_TLBIRANGE is implemented, defer the individial PTE

typo: individual

Also, 'PTE' isn't significant here.

arch/arm64/kvm/hyp/pgtable.c static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n · line 782

Quoted 5 lines
arch/arm64/kvm/hyp/pgtable.c +5 -0
+	 * TLB invalidations until the entire walk is finished, and
+	 * then use the range-based TLBI instructions to do the
+	 * invalidations. Condition this upon S2FWB in order to avoid
+	 * a page-table walk again to perform the CMOs after TLBI.
+	 */

Apply the wording suggestion from the changelog here as well.

Quoted 57 lines
arch/arm64/kvm/hyp/pgtable.c +22 -3
+	return system_supports_tlb_range() && stage2_has_fwb(pgt);
+}
+
+static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
+				struct kvm_s2_mmu *mmu,
+				struct kvm_pgtable_mm_ops *mm_ops)
+{
+	struct kvm_pgtable *pgt = ctx->arg;
+
 	/*
 	 * Clear the existing PTE, and perform break-before-make with
 	 * TLB maintenance if it was valid.
 	 */
 	if (kvm_pte_valid(ctx->old)) {
 		kvm_clear_pte(ctx->ptep);
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+
+		if (!stage2_unmap_defer_tlb_flush(pgt))
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+					ctx->addr, ctx->level);
 	}
 
 	mm_ops->put_page(ctx->ptep);
@@ -1015,7 +1033,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	 * block entry and rely on the remaining portions being faulted
 	 * back lazily.
 	 */
-	stage2_put_pte(ctx, mmu, mm_ops);
+	stage2_unmap_put_pte(ctx, mmu, mm_ops);
 
 	if (need_flush && mm_ops->dcache_clean_inval_poc)
 		mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops),
@@ -1029,13 +1047,20 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 
 int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
 {
+	int ret;
 	struct kvm_pgtable_walker walker = {
 		.cb	= stage2_unmap_walker,
 		.arg	= pgt,
 		.flags	= KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
 	};
 
-	return kvm_pgtable_walk(pgt, addr, size, &walker);
+	ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+	if (stage2_unmap_defer_tlb_flush(pgt))
+		/* Perform the deferred TLB invalidations */
+		kvm_call_hyp(__kvm_tlb_flush_vmid_range, pgt->mmu,
+				addr, addr + size);
+
+	return ret;
 }
 
 struct stage2_attr_data {
Signature · 1 line
2.40.1.698.g37aff9b760-goog
Signature · 2 lines
Thanks,
Oliver
maintainer
#15 lore raw

Re: [PATCH v4 6/6] KVM: arm64: Use TLBI range-based intructions for unmap

In reply to Raghavendra Rao Ananta
Quoted 11 lines · Raghavendra Rao Ananta

The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck.

Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition this upon S2FWB in order to avoid walking the page-table again to perform the CMOs after issuing the TLBI.

But that's the real bottleneck. TLBIs are cheap compared to CMOs, even on remarkably bad implementations. What is your plan to fix this?

Quoted 39 lines

Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic.

---
 arch/arm64/kvm/hyp/pgtable.c | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +17 -2
index b8f0dbd12f773..5832ee3418fb0 100644
@@ -771,16 +771,34 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n
 	smp_store_release(ctx->ptep, new);
 }
 
-static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu,
-			   struct kvm_pgtable_mm_ops *mm_ops)
+static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
 {
+	/*
+	 * If FEAT_TLBIRANGE is implemented, defer the individial PTE
+	 * TLB invalidations until the entire walk is finished, and
+	 * then use the range-based TLBI instructions to do the
+	 * invalidations. Condition this upon S2FWB in order to avoid
+	 * a page-table walk again to perform the CMOs after TLBI.
+	 */
+	return system_supports_tlb_range() && stage2_has_fwb(pgt);
+}
+
+static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
+				struct kvm_s2_mmu *mmu,
+				struct kvm_pgtable_mm_ops *mm_ops)
+{
+	struct kvm_pgtable *pgt = ctx->arg;
+
 	/*

arch/arm64/kvm/hyp/pgtable.c static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n · line 801

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +4 -1
 	 * Clear the existing PTE, and perform break-before-make with
 	 * TLB maintenance if it was valid.
 	 */
 	if (kvm_pte_valid(ctx->old)) {
 		kvm_clear_pte(ctx->ptep);
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+
+		if (!stage2_unmap_defer_tlb_flush(pgt))
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+					ctx->addr, ctx->level);

This really doesn't match the comment anymore.

Overall, I'm very concerned that we lose the consistency property that the current code has: once called, the TLBs and the page tables are synchronised.

Yes, this patch looks correct. But it is also really fragile.

Quoted 19 lines
arch/arm64/kvm/hyp/pgtable.c +2 -1
 	}
 
 	mm_ops->put_page(ctx->ptep);
@@ -1015,7 +1033,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	 * block entry and rely on the remaining portions being faulted
 	 * back lazily.
 	 */
-	stage2_put_pte(ctx, mmu, mm_ops);
+	stage2_unmap_put_pte(ctx, mmu, mm_ops);
 
 	if (need_flush && mm_ops->dcache_clean_inval_poc)
 		mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops),
@@ -1029,13 +1047,20 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 
 int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
 {
+	int ret;
 	struct kvm_pgtable_walker walker = {
 		.cb	= stage2_unmap_walker,

arch/arm64/kvm/hyp/pgtable.c static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, · line 1061

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +5 -1
 		.arg	= pgt,
 		.flags	= KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
 	};
 
-	return kvm_pgtable_walk(pgt, addr, size, &walker);
+	ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+	if (stage2_unmap_defer_tlb_flush(pgt))
+		/* Perform the deferred TLB invalidations */
+		kvm_call_hyp(__kvm_tlb_flush_vmid_range, pgt->mmu,
+				addr, addr + size);

This "kvm_call_hyp(__kvm_tlb_flush_vmid_range,...)" could do with a wrapper from the point where you introduce it.

arch/arm64/kvm/hyp/pgtable.c static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, · line 1065

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +2 -0
+
+	return ret;
 }
 

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#16 lore raw

Re: [PATCH v4 6/6] KVM: arm64: Use TLBI range-based intructions for unmap

Quoted 9 lines · Marc Zyngier
Quoted 6 lines · Raghavendra Rao Ananta

The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck.

Quoted 9 lines
Quoted 5 lines

Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition this upon S2FWB in order to avoid walking the page-table again to perform the CMOs after issuing the TLBI.

But that's the real bottleneck. TLBIs are cheap compared to CMOs, even on remarkably bad implementations. What is your plan to fix this?

Correct me if I'm wrong, but my understanding was that a multiple issuance of TLBI + DSB was the bottleneck, and this patch tries to avoid this by issuing only one TLBI + DSB at the end.

Quoted 43 lines
Quoted 43 lines

Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic.

---
 arch/arm64/kvm/hyp/pgtable.c | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +17 -2
index b8f0dbd12f773..5832ee3418fb0 100644
@@ -771,16 +771,34 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n
      smp_store_release(ctx->ptep, new);
 }
 
-static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu,
-                        struct kvm_pgtable_mm_ops *mm_ops)
+static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
 {
+     /*
+      * If FEAT_TLBIRANGE is implemented, defer the individial PTE
+      * TLB invalidations until the entire walk is finished, and
+      * then use the range-based TLBI instructions to do the
+      * invalidations. Condition this upon S2FWB in order to avoid
+      * a page-table walk again to perform the CMOs after TLBI.
+      */
+     return system_supports_tlb_range() && stage2_has_fwb(pgt);
+}
+
+static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
+                             struct kvm_s2_mmu *mmu,
+                             struct kvm_pgtable_mm_ops *mm_ops)
+{
+     struct kvm_pgtable *pgt = ctx->arg;
+
      /*
       * Clear the existing PTE, and perform break-before-make with
       * TLB maintenance if it was valid.
       */
      if (kvm_pte_valid(ctx->old)) {
Quoted 9 lines

arch/arm64/kvm/hyp/pgtable.c static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n · line 801

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +4 -1
              kvm_clear_pte(ctx->ptep);
-             kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+
+             if (!stage2_unmap_defer_tlb_flush(pgt))
+                     kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+                                     ctx->addr, ctx->level);

This really doesn't match the comment anymore.

Right, I can re-write this in the next spin.

Quoted 6 lines

Overall, I'm very concerned that we lose the consistency property that the current code has: once called, the TLBs and the page tables are synchronised.

Yes, this patch looks correct. But it is also really fragile.

Yeah, we were a little skeptical about this too. Till v2, we had a different implementation in which we had an independent fast unmap path that disconnects the PTE hierarchy if the unmap range was exactly KVM_PGTABLE_MIN_BLOCK_LEVEL [1]. But this had some problems, and we pivoted to the current implementation.

Quoted 23 lines
Quoted 23 lines
arch/arm64/kvm/hyp/pgtable.c +2 -1
      }
 
      mm_ops->put_page(ctx->ptep);
@@ -1015,7 +1033,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
       * block entry and rely on the remaining portions being faulted
       * back lazily.
       */
-     stage2_put_pte(ctx, mmu, mm_ops);
+     stage2_unmap_put_pte(ctx, mmu, mm_ops);
 
      if (need_flush && mm_ops->dcache_clean_inval_poc)
              mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops),
@@ -1029,13 +1047,20 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 
 int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
 {
+     int ret;
      struct kvm_pgtable_walker walker = {
              .cb     = stage2_unmap_walker,
              .arg    = pgt,
              .flags  = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
      };
 
Quoted 10 lines

arch/arm64/kvm/hyp/pgtable.c static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, · line 1061

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +5 -1
-     return kvm_pgtable_walk(pgt, addr, size, &walker);
+     ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+     if (stage2_unmap_defer_tlb_flush(pgt))
+             /* Perform the deferred TLB invalidations */
+             kvm_call_hyp(__kvm_tlb_flush_vmid_range, pgt->mmu,
+                             addr, addr + size);

This "kvm_call_hyp(__kvm_tlb_flush_vmid_range,...)" could do with a wrapper from the point where you introduce it.

Sorry, I didn't get this comment. Do you mind elaborating on it?

Thank you.
Raghavendra

[1]: https://lore.kernel.org/all/[email protected]/

Quoted 11 lines

arch/arm64/kvm/hyp/pgtable.c static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, · line 1065

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +2 -0
+
+     return ret;
 }
 

Thanks,

        M.

--
Without deviation from the norm, progress is not possible.

maintainer
#17 lore raw

Re: [PATCH v4 6/6] KVM: arm64: Use TLBI range-based intructions for unmap

Quoted 17 lines · Raghavendra Rao Ananta
Quoted 15 lines · Marc Zyngier
Quoted 11 lines · Raghavendra Rao Ananta

The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck.

Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition this upon S2FWB in order to avoid walking the page-table again to perform the CMOs after issuing the TLBI.

Quoted 6 lines
Quoted 3 lines

But that's the real bottleneck. TLBIs are cheap compared to CMOs, even on remarkably bad implementations. What is your plan to fix this?

Correct me if I'm wrong, but my understanding was that a multiple issuance of TLBI + DSB was the bottleneck, and this patch tries to avoid this by issuing only one TLBI + DSB at the end.

At least on some of the machines I have access to, CMOs are fare more expensive than TLBIs, and they are the ones causing slowdowns. Your system shows a different behaviour, and that's fine, but you can't draw a general conclusion from it.

Quoted 58 lines
Quoted 43 lines
Quoted 43 lines

Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic.

---
 arch/arm64/kvm/hyp/pgtable.c | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +17 -2
index b8f0dbd12f773..5832ee3418fb0 100644
@@ -771,16 +771,34 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n
      smp_store_release(ctx->ptep, new);
 }
 
-static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu,
-                        struct kvm_pgtable_mm_ops *mm_ops)
+static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
 {
+     /*
+      * If FEAT_TLBIRANGE is implemented, defer the individial PTE
+      * TLB invalidations until the entire walk is finished, and
+      * then use the range-based TLBI instructions to do the
+      * invalidations. Condition this upon S2FWB in order to avoid
+      * a page-table walk again to perform the CMOs after TLBI.
+      */
+     return system_supports_tlb_range() && stage2_has_fwb(pgt);
+}
+
+static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
+                             struct kvm_s2_mmu *mmu,
+                             struct kvm_pgtable_mm_ops *mm_ops)
+{
+     struct kvm_pgtable *pgt = ctx->arg;
+
      /*
       * Clear the existing PTE, and perform break-before-make with
       * TLB maintenance if it was valid.
       */
      if (kvm_pte_valid(ctx->old)) {
Quoted 9 lines

arch/arm64/kvm/hyp/pgtable.c static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n · line 801

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +4 -1
              kvm_clear_pte(ctx->ptep);
-             kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+
+             if (!stage2_unmap_defer_tlb_flush(pgt))
+                     kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+                                     ctx->addr, ctx->level);

This really doesn't match the comment anymore.

Right, I can re-write this in the next spin.

Quoted 4 lines

Overall, I'm very concerned that we lose the consistency property that the current code has: once called, the TLBs and the page tables are synchronised.

Quoted 7 lines
Quoted 2 lines

Yes, this patch looks correct. But it is also really fragile.

Yeah, we were a little skeptical about this too. Till v2, we had a different implementation in which we had an independent fast unmap path that disconnects the PTE hierarchy if the unmap range was exactly KVM_PGTABLE_MIN_BLOCK_LEVEL [1]. But this had some problems, and we pivoted to the current implementation.

Can we at least have some sort of runtime assertions that at the point we release the write lock, the TLBs have been invalidated? Even if that's tied to some debug config.

Quoted 34 lines
Quoted 24 lines
Quoted 24 lines
arch/arm64/kvm/hyp/pgtable.c +2 -1
      }
 
      mm_ops->put_page(ctx->ptep);
@@ -1015,7 +1033,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
       * block entry and rely on the remaining portions being faulted
       * back lazily.
       */
-     stage2_put_pte(ctx, mmu, mm_ops);
+     stage2_unmap_put_pte(ctx, mmu, mm_ops);
 
      if (need_flush && mm_ops->dcache_clean_inval_poc)
              mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops),
@@ -1029,13 +1047,20 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 
 int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
 {
+     int ret;
      struct kvm_pgtable_walker walker = {
              .cb     = stage2_unmap_walker,
              .arg    = pgt,
              .flags  = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
      };
 
Quoted 10 lines

arch/arm64/kvm/hyp/pgtable.c static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, · line 1061

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +5 -1
-     return kvm_pgtable_walk(pgt, addr, size, &walker);
+     ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+     if (stage2_unmap_defer_tlb_flush(pgt))
+             /* Perform the deferred TLB invalidations */
+             kvm_call_hyp(__kvm_tlb_flush_vmid_range, pgt->mmu,
+                             addr, addr + size);

This "kvm_call_hyp(__kvm_tlb_flush_vmid_range,...)" could do with a wrapper from the point where you introduce it.

Quoted 1 line

Sorry, I didn't get this comment. Do you mind elaborating on it?

All I'm saying is that you should have a wrapper like:

void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
phys_addr_t base, size_t size)

{
	kvm_call_hyp(__kvm_tlb_flush_vmid_range,
		     mmu, base, base + size);
}

and use it throughout the code.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.