Skip to contentSkip to conversation

Implement Eager Page Splitting for ARM

PATCH Superseded Ricardo Koller 4 participants 35 messages Mar 7, 2023 – Apr 10, 2023

Subsystems (4)

v6 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v6 cover Implement Eager Page Splitting for ARM

Eager Page Splitting improves the performance of dirty-logging (used in live migrations) when guest memory is backed by huge-pages. It's an optimization used in Google Cloud since 2016 on x86, and for the last couple of months on ARM.

Background and motivation
=========================
Dirty logging is typically used for live-migration iterative copying. KVM implements dirty-logging at the PAGE_SIZE granularity (will refer to 4K pages from now on). It does it by faulting on write-protected 4K pages. Therefore, enabling dirty-logging on a huge-page requires breaking it into 4K pages in the first place. KVM does this breaking on fault, and because it's in the critical path it only maps the 4K page that faulted; every other 4K page is left unmapped. This is not great for performance on ARM for a couple of reasons:

- Splitting on fault can halt vcpus for milliseconds in some implementations. Splitting a block PTE requires using a broadcasted TLB invalidation (TLBI) for every huge-page (due to the break-before-make requirement). Note that x86 doesn't need this. We observed some implementations that take millliseconds to complete broadcasted TLBIs when done in parallel from multiple vcpus. And that's exactly what happens when doing it on fault: multiple vcpus fault at the same time triggering TLBIs in parallel.

- Read intensive guest workloads end up paying for dirty-logging. Only mapping the faulting 4K page means that all the other pages that were part of the huge-page will now be unmapped. The effect is that any access, including reads, now has to fault.

Eager Page Splitting (on ARM)
=============================
Eager Page Splitting fixes the above two issues by eagerly splitting huge-pages when enabling dirty logging. The goal is to avoid doing it while faulting on write-protected pages. This is what the TDP MMU does for x86 [0], except that x86 does it for different reasons: to avoid grabbing the MMU lock on fault. Note that taking care of write-protection faults still requires grabbing the MMU lock on ARM, but not on x86 (with the fast_page_fault path).

An additional benefit of eagerly splitting huge-pages is that it can be done in a controlled way (e.g., via an IOCTL). This series provides two knobs for doing it, just like its x86 counterpart: when enabling dirty logging, and when using the KVM_CLEAR_DIRTY_LOG ioctl. The benefit of doing it on KVM_CLEAR_DIRTY_LOG is that this ioctl takes ranges, and not complete memslots like when enabling dirty logging. This means that the cost of splitting (mainly broadcasted TLBIs) can be throttled: split a range, wait for a bit, split another range, etc. The benefits of this approach were presented by Oliver Upton at KVM Forum 2022 [1].

Implementation
==============
Patches 3-4 add a pgtable utility function for splitting huge block
PTEs: kvm_pgtable_stage2_split(). Patches 5-9 add support for eagerly splitting huge-pages when enabling dirty-logging and when using the KVM_CLEAR_DIRTY_LOG ioctl. Note that this is just like what x86 does, and the code is actually based on it. And finally, patch 9:

KVM: arm64: Use local TLBI on permission relaxation

adds support for using local TLBIs instead of broadcasts when doing permission relaxation. This last patch is key to achieving good performance during dirty-logging, as eagerly breaking huge-pages replaces mapping new pages with permission relaxation. Got this patch (indirectly) from Marc Z. and took the liberty of adding a commit message.

Note: this applies on top of 6.3-rc1.

Performance evaluation
======================
The performance benefits were tested using the dirty_log_perf_test selftest with 2M huge-pages.

The first test uses a write-only sequential workload where the stride is 2M instead of 4K [2]. The idea with this experiment is to emulate a random access pattern writing a different huge-page at every access. Observe that the benefit increases with the number of vcpus: up to 5.76x for 152 vcpus. This table shows the guest dirtying time when using the CLEAR ioctl (and KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2):

/dirty_log_perf_test_sparse -s anonymous_hugetlb_2mb -b 1G -v $i -i 3 -m 2

	+-------+----------+------------------+
	| vCPUs | 6.2-rc3  | 6.2-rc3 + series |
	|       |    (ms)  |             (ms) |
	+-------+----------+------------------+
	|    1  |    2.63  |          1.66    |
	|    2  |    2.95  |          1.70    |
	|    4  |    3.21  |          1.71    |
	|    8  |    4.97  |          1.78    |
	|   16  |    9.51  |          1.82    |
	|   32  |   20.15  |          3.03    |
	|   64  |   40.09  |          5.80    |
	|  128  |   80.08  |         12.24    |
	|  152  |  109.81  |         15.14    |
	+-------+----------+------------------+

This secondv test measures the benefit of eager page splitting on read intensive workloads (1 write for every 10 reads). As in the other test, the benefit increases with the number of vcpus, up to 8.82x for 152 vcpus. This table shows the guest dirtying time when using the CLEAR ioctl (and KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2):

./dirty_log_perf_test -s anonymous_hugetlb_2mb -b 1G -v $i -i 3 -m 2 -w 10

	+-------+----------+------------------+
	| vCPUs | 6.2-rc3  | 6.2-rc3 + series |
	|       |   (sec)  |            (sec) |
	+-------+----------+------------------+
	|    1  |    0.65  |          0.07    |
	|    2  |    0.70  |          0.08    |
	|    4  |    0.71  |          0.08    |
	|    8  |    0.72  |          0.08    |
	|   16  |    0.76  |          0.08    |
	|   32  |    1.61  |          0.14    |
	|   64  |    3.46  |          0.30    |
	|  128  |    5.49  |          0.64    |
	|  152  |    6.44  |          0.63    |
	+-------+----------+------------------+

Changes from v5:
https://lore.kernel.org/kvmarm/[email protected]/
- fixed message in "Use local TLBI on permission relaxation". (Vladimir)
- s/removed/unlinked in first commit message. (Shaoqin)
- rebased series
- collected r-b's from Shaoqin

Changes from v4:
https://lore.kernel.org/kvmarm/[email protected]/
- nits on some comments (s/removed/unlinked and remove @new). (Shaoqin)

Changes from v3:
https://lore.kernel.org/kvmarm/[email protected]/
- KVM_PGTABLE_WALK_SKIP_CMO to use BIT(5). (Shaoqin)
- Rewritten commit message for "Rename free_unlinked to free_removed" using Oliver's suggestion. (Oliver)
- "un" -> "an" typo. (Shaoqin)
- kvm_pgtable_stage2_create_unlinked() to return a "kvm_pte_t *". (Oliver)
- refactored stage2_block_get_nr_page_tables(). (Oliver)
- /s/bock/block. (Shaoqin)

Changes from v2:
https://lore.kernel.org/kvmarm/[email protected]/
- removed redundant kvm_pte_table() check from split walker function. (Gavin)
- fix compilation of patch 8 by moving some definitions from path 9. (Gavin)
- add comment for kvm_mmu_split_nr_page_tables(). (Gavin)

Changes from v1:
https://lore.kernel.org/kvmarm/[email protected]/
- added a capability to set the eager splitting chunk size. This indirectly sets the number of pages in the cache. It also allows for opting out of this feature. (Oliver, Marc)
- changed kvm_pgtable_stage2_split() to split 1g huge-pages using either 513 or 1 at a time (with a cache of 1). (Oliver, Marc)
- added force_pte arg to kvm_pgtable_stage2_create_removed().
- renamed free_removed to free_unlinked. (Ben and Oliver)
- added KVM_PGTABLE_WALK ctx->flags for skipping BBM and CMO, instead of KVM_PGTABLE_WALK_REMOVED. (Oliver)

Changes from the RFC:
https://lore.kernel.org/kvmarm/[email protected]/
- dropped the changes to split on POST visits. No visible perf benefit.
- changed the kvm_pgtable_stage2_free_removed() implementation to reuse the stage2 mapper.
- dropped the FEAT_BBM changes and optimization. Will send this on a different series.

Thanks,
Ricardo

Marc Zyngier (1):
KVM: arm64: Use local TLBI on permission relaxation

Ricardo Koller (11):
KVM: arm64: Rename free_removed to free_unlinked
KVM: arm64: Add KVM_PGTABLE_WALK ctx->flags for skipping BBM and CMO
KVM: arm64: Add helper for creating unlinked stage2 subtrees
KVM: arm64: Add kvm_pgtable_stage2_split()
KVM: arm64: Refactor kvm_arch_commit_memory_region()
KVM: arm64: Add kvm_uninit_stage2_mmu()
KVM: arm64: Export kvm_are_all_memslots_empty()
KVM: arm64: Add KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
KVM: arm64: Split huge pages when dirty logging is enabled
KVM: arm64: Open-code kvm_mmu_write_protect_pt_masked()
KVM: arm64: Split huge pages during KVM_CLEAR_DIRTY_LOG

 Documentation/virt/kvm/api.rst        |  26 ++++
 arch/arm64/include/asm/kvm_asm.h      |   4 +
 arch/arm64/include/asm/kvm_host.h     |  19 +++
 arch/arm64/include/asm/kvm_mmu.h      |   1 +
 arch/arm64/include/asm/kvm_pgtable.h  |  84 ++++++++++-
 arch/arm64/kvm/arm.c                  |  22 +++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c    |  10 ++
 arch/arm64/kvm/hyp/nvhe/mem_protect.c |   6 +-
 arch/arm64/kvm/hyp/nvhe/tlb.c         |  54 +++++++
 arch/arm64/kvm/hyp/pgtable.c          | 194 ++++++++++++++++++++++++--
 arch/arm64/kvm/hyp/vhe/tlb.c          |  32 +++++
 arch/arm64/kvm/mmu.c                  | 188 +++++++++++++++++++++----
 include/linux/kvm_host.h              |   2 +
 include/uapi/linux/kvm.h              |   1 +
 virt/kvm/kvm_main.c                   |   2 +-
 15 files changed, 591 insertions(+), 54 deletions(-)
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
#1 lore raw

PATCH v6 1/12 KVM: arm64: Rename free_removed to free_unlinked

Normalize on referring to tables outside of an active paging structure as 'unlinked'.

A subsequent change to KVM will add support for building page tables that are not part of an active paging structure. The existing 'removed_table' terminology is quite clunky when applied in this context.

No functional change intended.

---
 arch/arm64/include/asm/kvm_pgtable.h  |  8 ++++----
 arch/arm64/kvm/hyp/nvhe/mem_protect.c |  6 +++---
 arch/arm64/kvm/hyp/pgtable.c          |  6 +++---
 arch/arm64/kvm/mmu.c                  | 10 +++++-----
 4 files changed, 15 insertions(+), 15 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +4 -4
index 4cd6762bda80..26a4293726c1 100644
@@ -104,7 +104,7 @@ static inline bool kvm_level_supports_block_mapping(u32 level)
  *				allocation is physically contiguous.
  * @free_pages_exact:		Free an exact number of memory pages previously
  *				allocated by zalloc_pages_exact.
- * @free_removed_table:		Free a removed paging structure by unlinking and
+ * @free_unlinked_table:	Free an unlinked paging structure by unlinking and
  *				dropping references.
  * @get_page:			Increment the refcount on a page.
  * @put_page:			Decrement the refcount on a page. When the
@@ -124,7 +124,7 @@ struct kvm_pgtable_mm_ops {
 	void*		(*zalloc_page)(void *arg);
 	void*		(*zalloc_pages_exact)(size_t size);
 	void		(*free_pages_exact)(void *addr, size_t size);
-	void		(*free_removed_table)(void *addr, u32 level);
+	void		(*free_unlinked_table)(void *addr, u32 level);
 	void		(*get_page)(void *addr);
 	void		(*put_page)(void *addr);
 	int		(*page_count)(void *addr);
@@ -440,7 +440,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
 
 /**
- * kvm_pgtable_stage2_free_removed() - Free a removed stage-2 paging structure.
+ * kvm_pgtable_stage2_free_unlinked() - Free an unlinked stage-2 paging structure.
  * @mm_ops:	Memory management callbacks.
  * @pgtable:	Unlinked stage-2 paging structure to be freed.
  * @level:	Level of the stage-2 paging structure to be freed.
@@ -448,7 +448,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
  * The page-table is assumed to be unreachable by any hardware walkers prior to
  * freeing and therefore no TLB invalidation is performed.
  */
-void kvm_pgtable_stage2_free_removed(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level);
+void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level);
 
 /**
  * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table.
arch/arm64/kvm/hyp/nvhe/mem_protect.c +3 -3
index 552653fa18be..b030170d803b 100644
@@ -91,9 +91,9 @@ static void host_s2_put_page(void *addr)
 	hyp_put_page(&host_s2_pool, addr);
 }
 
-static void host_s2_free_removed_table(void *addr, u32 level)
+static void host_s2_free_unlinked_table(void *addr, u32 level)
 {
-	kvm_pgtable_stage2_free_removed(&host_mmu.mm_ops, addr, level);
+	kvm_pgtable_stage2_free_unlinked(&host_mmu.mm_ops, addr, level);
 }
 
 static int prepare_s2_pool(void *pgt_pool_base)
@@ -110,7 +110,7 @@ static int prepare_s2_pool(void *pgt_pool_base)
 	host_mmu.mm_ops = (struct kvm_pgtable_mm_ops) {
 		.zalloc_pages_exact = host_s2_zalloc_pages_exact,
 		.zalloc_page = host_s2_zalloc_page,
-		.free_removed_table = host_s2_free_removed_table,
+		.free_unlinked_table = host_s2_free_unlinked_table,
 		.phys_to_virt = hyp_phys_to_virt,
 		.virt_to_phys = hyp_virt_to_phys,
 		.page_count = hyp_page_count,
arch/arm64/kvm/hyp/pgtable.c +3 -3
index 3d61bd3e591d..a3246d6cddec 100644
@@ -860,7 +860,7 @@ static int stage2_map_walk_table_pre(const struct kvm_pgtable_visit_ctx *ctx,
 	if (ret)
 		return ret;
 
-	mm_ops->free_removed_table(childp, ctx->level);
+	mm_ops->free_unlinked_table(childp, ctx->level);
 	return 0;
 }
 
@@ -905,7 +905,7 @@ static int stage2_map_walk_leaf(const struct kvm_pgtable_visit_ctx *ctx,
  * The TABLE_PRE callback runs for table entries on the way down, looking
  * for table entries which we could conceivably replace with a block entry
  * for this mapping. If it finds one it replaces the entry and calls
- * kvm_pgtable_mm_ops::free_removed_table() to tear down the detached table.
+ * kvm_pgtable_mm_ops::free_unlinked_table() to tear down the detached table.
  *
  * Otherwise, the LEAF callback performs the mapping at the existing leaves
  * instead.
@@ -1276,7 +1276,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt)
 	pgt->pgd = NULL;
 }
 
-void kvm_pgtable_stage2_free_removed(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level)
+void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level)
 {
 	kvm_pteref_t ptep = (kvm_pteref_t)pgtable;
 	struct kvm_pgtable_walker walker = {
arch/arm64/kvm/mmu.c +5 -5
index 7113587222ff..efdaab3f154d 100644
@@ -131,21 +131,21 @@ static void kvm_s2_free_pages_exact(void *virt, size_t size)
 
 static struct kvm_pgtable_mm_ops kvm_s2_mm_ops;
 
-static void stage2_free_removed_table_rcu_cb(struct rcu_head *head)
+static void stage2_free_unlinked_table_rcu_cb(struct rcu_head *head)
 {
 	struct page *page = container_of(head, struct page, rcu_head);
 	void *pgtable = page_to_virt(page);
 	u32 level = page_private(page);
 
-	kvm_pgtable_stage2_free_removed(&kvm_s2_mm_ops, pgtable, level);
+	kvm_pgtable_stage2_free_unlinked(&kvm_s2_mm_ops, pgtable, level);
 }
 
-static void stage2_free_removed_table(void *addr, u32 level)
+static void stage2_free_unlinked_table(void *addr, u32 level)
 {
 	struct page *page = virt_to_page(addr);
 
 	set_page_private(page, (unsigned long)level);
-	call_rcu(&page->rcu_head, stage2_free_removed_table_rcu_cb);
+	call_rcu(&page->rcu_head, stage2_free_unlinked_table_rcu_cb);
 }
 
 static void kvm_host_get_page(void *addr)
@@ -682,7 +682,7 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = {
 	.zalloc_page		= stage2_memcache_zalloc_page,
 	.zalloc_pages_exact	= kvm_s2_zalloc_pages_exact,
 	.free_pages_exact	= kvm_s2_free_pages_exact,
-	.free_removed_table	= stage2_free_removed_table,
+	.free_unlinked_table	= stage2_free_unlinked_table,
 	.get_page		= kvm_host_get_page,
 	.put_page		= kvm_s2_put_page,
 	.page_count		= kvm_host_page_count,
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
#2 lore raw

PATCH v6 2/12 KVM: arm64: Add KVM_PGTABLE_WALK ctx->flags for skipping BBM and CMO

In reply to Ricardo Koller

Add two flags to kvm_pgtable_visit_ctx, KVM_PGTABLE_WALK_SKIP_BBM and KVM_PGTABLE_WALK_SKIP_CMO, to indicate that the walk should not perform break-before-make (BBM) nor cache maintenance operations (CMO). This will by a future commit to create unlinked tables not accessible to the HW page-table walker. This is safe as these unlinked tables are not visible to the HW page-table walker.

---
 arch/arm64/include/asm/kvm_pgtable.h | 18 ++++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c         | 27 ++++++++++++++++-----------
 2 files changed, 34 insertions(+), 11 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +18 -0
index 26a4293726c1..c7a269cad053 100644
@@ -195,6 +195,12 @@ typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end,
  *					with other software walkers.
  * @KVM_PGTABLE_WALK_HANDLE_FAULT:	Indicates the page-table walk was
  *					invoked from a fault handler.
+ * @KVM_PGTABLE_WALK_SKIP_BBM:		Visit and update table entries
+ *					without Break-before-make
+ *					requirements.
+ * @KVM_PGTABLE_WALK_SKIP_CMO:		Visit and update table entries
+ *					without Cache maintenance
+ *					operations required.
  */
 enum kvm_pgtable_walk_flags {
 	KVM_PGTABLE_WALK_LEAF			= BIT(0),
@@ -202,6 +208,8 @@ enum kvm_pgtable_walk_flags {
 	KVM_PGTABLE_WALK_TABLE_POST		= BIT(2),
 	KVM_PGTABLE_WALK_SHARED			= BIT(3),
 	KVM_PGTABLE_WALK_HANDLE_FAULT		= BIT(4),
+	KVM_PGTABLE_WALK_SKIP_BBM		= BIT(5),
+	KVM_PGTABLE_WALK_SKIP_CMO		= BIT(6),
 };
 
 struct kvm_pgtable_visit_ctx {
@@ -223,6 +231,16 @@ static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c
 	return ctx->flags & KVM_PGTABLE_WALK_SHARED;
 }
 
+static inline bool kvm_pgtable_walk_skip_bbm(const struct kvm_pgtable_visit_ctx *ctx)
+{
+	return ctx->flags & KVM_PGTABLE_WALK_SKIP_BBM;
+}
+
+static inline bool kvm_pgtable_walk_skip_cmo(const struct kvm_pgtable_visit_ctx *ctx)
+{
+	return ctx->flags & KVM_PGTABLE_WALK_SKIP_CMO;
+}
+
 /**
  * struct kvm_pgtable_walker - Hook into a page-table walk.
  * @cb:		Callback function to invoke during the walk.
arch/arm64/kvm/hyp/pgtable.c +16 -11
index a3246d6cddec..4f703cc4cb03 100644
@@ -741,14 +741,17 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
 	if (!stage2_try_set_pte(ctx, KVM_INVALID_PTE_LOCKED))
 		return false;
 
-	/*
-	 * Perform the appropriate TLB invalidation based on the evicted pte
-	 * value (if any).
-	 */
-	if (kvm_pte_table(ctx->old, ctx->level))
-		kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
-	else if (kvm_pte_valid(ctx->old))
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+	if (!kvm_pgtable_walk_skip_bbm(ctx)) {
+		/*
+		 * Perform the appropriate TLB invalidation based on the
+		 * evicted pte value (if any).
+		 */
+		if (kvm_pte_table(ctx->old, ctx->level))
+			kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
+		else if (kvm_pte_valid(ctx->old))
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+				     ctx->addr, ctx->level);
+	}
 
 	if (stage2_pte_is_counted(ctx->old))
 		mm_ops->put_page(ctx->ptep);
@@ -832,11 +835,13 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx,
 		return -EAGAIN;
 
 	/* Perform CMOs before installation of the guest stage-2 PTE */
-	if (mm_ops->dcache_clean_inval_poc && stage2_pte_cacheable(pgt, new))
+	if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->dcache_clean_inval_poc &&
+	    stage2_pte_cacheable(pgt, new))
 		mm_ops->dcache_clean_inval_poc(kvm_pte_follow(new, mm_ops),
-						granule);
+					       granule);
 
-	if (mm_ops->icache_inval_pou && stage2_pte_executable(new))
+	if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->icache_inval_pou &&
+	    stage2_pte_executable(new))
 		mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule);
 
 	stage2_make_pte(ctx, new);
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
maintainer
#3 lore raw

Re: [PATCH v6 02/12] KVM: arm64: Add KVM_PGTABLE_WALK ctx->flags for skipping BBM and CMO

Quoted 5 lines · Ricardo Koller

Add two flags to kvm_pgtable_visit_ctx, KVM_PGTABLE_WALK_SKIP_BBM and KVM_PGTABLE_WALK_SKIP_CMO, to indicate that the walk should not perform break-before-make (BBM) nor cache maintenance operations (CMO). This will by a future commit to create unlinked tables not

This will *be used*?

Quoted 2 lines

accessible to the HW page-table walker. This is safe as these unlinked tables are not visible to the HW page-table walker.

I don't think this last sentence makes much sense. The PTW is always coherent with the CPU caches and doesn't require cache maintenance (CMOs are solely for the pages the PTs point to).

But this makes me question this patch further.

The key observation here is that if you are creating new PTs that shadow an existing structure and still points to the same data pages, the cache state is independent of the intermediate PT walk, and thus CMOs are pointless anyway. So skipping CMOs makes sense.

I agree with the assertion that there is little point in doing BBM when *creating* page tables, as all PTs start in an invalid state. But then, why do you need to skip it? The invalidation calls are already gated on the previous pointer being valid, which I presume won't be the case for what you describe here.

Quoted 13 lines
---
 arch/arm64/include/asm/kvm_pgtable.h | 18 ++++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c         | 27 ++++++++++++++++-----------
 2 files changed, 34 insertions(+), 11 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +0 -0
index 26a4293726c1..c7a269cad053 100644

arch/arm64/include/asm/kvm_pgtable.h typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end, · line 203

Quoted 9 lines
arch/arm64/include/asm/kvm_pgtable.h +6 -0
@@ -195,6 +195,12 @@ typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end,
  *					with other software walkers.
  * @KVM_PGTABLE_WALK_HANDLE_FAULT:	Indicates the page-table walk was
  *					invoked from a fault handler.
+ * @KVM_PGTABLE_WALK_SKIP_BBM:		Visit and update table entries
+ *					without Break-before-make
+ *					requirements.
+ * @KVM_PGTABLE_WALK_SKIP_CMO:		Visit and update table entries
+ *					without Cache maintenance
+ *					operations required.

We have both I and D side CMOs. Is it reasonable to always treat them identically?

Quoted 13 lines
arch/arm64/include/asm/kvm_pgtable.h +2 -0
  */
 enum kvm_pgtable_walk_flags {
 	KVM_PGTABLE_WALK_LEAF			= BIT(0),
@@ -202,6 +208,8 @@ enum kvm_pgtable_walk_flags {
 	KVM_PGTABLE_WALK_TABLE_POST		= BIT(2),
 	KVM_PGTABLE_WALK_SHARED			= BIT(3),
 	KVM_PGTABLE_WALK_HANDLE_FAULT		= BIT(4),
+	KVM_PGTABLE_WALK_SKIP_BBM		= BIT(5),
+	KVM_PGTABLE_WALK_SKIP_CMO		= BIT(6),
 };
 
 struct kvm_pgtable_visit_ctx {

arch/arm64/include/asm/kvm_pgtable.h static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c · line 236

Quoted 6 lines
arch/arm64/include/asm/kvm_pgtable.h +3 -0
@@ -223,6 +231,16 @@ static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c
 	return ctx->flags & KVM_PGTABLE_WALK_SHARED;
 }
 
+static inline bool kvm_pgtable_walk_skip_bbm(const struct kvm_pgtable_visit_ctx *ctx)
+{
+	return ctx->flags & KVM_PGTABLE_WALK_SKIP_BBM;

Probably worth wrapping this with an 'unlikely'.

arch/arm64/include/asm/kvm_pgtable.h static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c · line 241

Quoted 5 lines
arch/arm64/include/asm/kvm_pgtable.h +5 -0
+}
+
+static inline bool kvm_pgtable_walk_skip_cmo(const struct kvm_pgtable_visit_ctx *ctx)
+{
+	return ctx->flags & KVM_PGTABLE_WALK_SKIP_CMO;

Same here.

Also, why are these in kvm_pgtable.h? Can't they be moved inside pgtable.c and thus have the "inline" attribute dropped?

Quoted 17 lines
arch/arm64/include/asm/kvm_pgtable.h +2 -0
+}
+
 /**
  * struct kvm_pgtable_walker - Hook into a page-table walk.
  * @cb:		Callback function to invoke during the walk.
arch/arm64/kvm/hyp/pgtable.c +0 -4
index a3246d6cddec..4f703cc4cb03 100644
@@ -741,14 +741,17 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
 	if (!stage2_try_set_pte(ctx, KVM_INVALID_PTE_LOCKED))
 		return false;
 
-	/*
-	 * Perform the appropriate TLB invalidation based on the evicted pte
-	 * value (if any).
-	 */

arch/arm64/kvm/hyp/pgtable.c static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, · line 749

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +6 -4
-	if (kvm_pte_table(ctx->old, ctx->level))
-		kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
-	else if (kvm_pte_valid(ctx->old))
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+	if (!kvm_pgtable_walk_skip_bbm(ctx)) {
+		/*
+		 * Perform the appropriate TLB invalidation based on the
+		 * evicted pte value (if any).
+		 */
+		if (kvm_pte_table(ctx->old, ctx->level))

You're not skipping BBM here. You're skipping the TLB invalidation. Not quite the same thing.

Quoted 15 lines
arch/arm64/kvm/hyp/pgtable.c +7 -1
+			kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
+		else if (kvm_pte_valid(ctx->old))
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+				     ctx->addr, ctx->level);
+	}
 
 	if (stage2_pte_is_counted(ctx->old))
 		mm_ops->put_page(ctx->ptep);
@@ -832,11 +835,13 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx,
 		return -EAGAIN;
 
 	/* Perform CMOs before installation of the guest stage-2 PTE */
-	if (mm_ops->dcache_clean_inval_poc && stage2_pte_cacheable(pgt, new))
+	if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->dcache_clean_inval_poc &&
+	    stage2_pte_cacheable(pgt, new))

arch/arm64/kvm/hyp/pgtable.c static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx, · line 847

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +3 -2
 		mm_ops->dcache_clean_inval_poc(kvm_pte_follow(new, mm_ops),
-						granule);
+					       granule);
 
-	if (mm_ops->icache_inval_pou && stage2_pte_executable(new))
+	if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->icache_inval_pou &&
+	    stage2_pte_executable(new))
 		mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule);
 
 	stage2_make_pte(ctx, new);

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#4 lore raw

Re: [PATCH v6 02/12] KVM: arm64: Add KVM_PGTABLE_WALK ctx->flags for skipping BBM and CMO

Quoted 24 lines · Marc Zyngier
Quoted 5 lines · Ricardo Koller

Add two flags to kvm_pgtable_visit_ctx, KVM_PGTABLE_WALK_SKIP_BBM and KVM_PGTABLE_WALK_SKIP_CMO, to indicate that the walk should not perform break-before-make (BBM) nor cache maintenance operations (CMO). This will by a future commit to create unlinked tables not

This will *be used*?

Quoted 2 lines

accessible to the HW page-table walker. This is safe as these unlinked tables are not visible to the HW page-table walker.

I don't think this last sentence makes much sense. The PTW is always coherent with the CPU caches and doesn't require cache maintenance (CMOs are solely for the pages the PTs point to).

But this makes me question this patch further.

The key observation here is that if you are creating new PTs that shadow an existing structure and still points to the same data pages, the cache state is independent of the intermediate PT walk, and thus CMOs are pointless anyway. So skipping CMOs makes sense.

Quoted 6 lines

I agree with the assertion that there is little point in doing BBM when *creating* page tables, as all PTs start in an invalid state. But then, why do you need to skip it? The invalidation calls are already gated on the previous pointer being valid, which I presume won't be the case for what you describe here.

I need to change the SKIP_BBM name; it's confusing, sorry for that. As you noticed below, SKIP_BBM just skips the TLB invalidation step in the BBM, so the invalidation still occurs with SKIP_BBM=true.

Thanks for the reviews Marc.

Quoted 122 lines
Quoted 13 lines
---
 arch/arm64/include/asm/kvm_pgtable.h | 18 ++++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c         | 27 ++++++++++++++++-----------
 2 files changed, 34 insertions(+), 11 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +0 -0
index 26a4293726c1..c7a269cad053 100644

arch/arm64/include/asm/kvm_pgtable.h typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end, · line 203

Quoted 9 lines
arch/arm64/include/asm/kvm_pgtable.h +6 -0
@@ -195,6 +195,12 @@ typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end,
  *					with other software walkers.
  * @KVM_PGTABLE_WALK_HANDLE_FAULT:	Indicates the page-table walk was
  *					invoked from a fault handler.
+ * @KVM_PGTABLE_WALK_SKIP_BBM:		Visit and update table entries
+ *					without Break-before-make
+ *					requirements.
+ * @KVM_PGTABLE_WALK_SKIP_CMO:		Visit and update table entries
+ *					without Cache maintenance
+ *					operations required.

We have both I and D side CMOs. Is it reasonable to always treat them identically?

Quoted 13 lines
arch/arm64/include/asm/kvm_pgtable.h +2 -0
  */
 enum kvm_pgtable_walk_flags {
 	KVM_PGTABLE_WALK_LEAF			= BIT(0),
@@ -202,6 +208,8 @@ enum kvm_pgtable_walk_flags {
 	KVM_PGTABLE_WALK_TABLE_POST		= BIT(2),
 	KVM_PGTABLE_WALK_SHARED			= BIT(3),
 	KVM_PGTABLE_WALK_HANDLE_FAULT		= BIT(4),
+	KVM_PGTABLE_WALK_SKIP_BBM		= BIT(5),
+	KVM_PGTABLE_WALK_SKIP_CMO		= BIT(6),
 };
 
 struct kvm_pgtable_visit_ctx {

arch/arm64/include/asm/kvm_pgtable.h static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c · line 236

Quoted 6 lines
arch/arm64/include/asm/kvm_pgtable.h +3 -0
@@ -223,6 +231,16 @@ static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c
 	return ctx->flags & KVM_PGTABLE_WALK_SHARED;
 }
 
+static inline bool kvm_pgtable_walk_skip_bbm(const struct kvm_pgtable_visit_ctx *ctx)
+{
+	return ctx->flags & KVM_PGTABLE_WALK_SKIP_BBM;

Probably worth wrapping this with an 'unlikely'.

arch/arm64/include/asm/kvm_pgtable.h static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c · line 241

Quoted 5 lines
arch/arm64/include/asm/kvm_pgtable.h +5 -0
+}
+
+static inline bool kvm_pgtable_walk_skip_cmo(const struct kvm_pgtable_visit_ctx *ctx)
+{
+	return ctx->flags & KVM_PGTABLE_WALK_SKIP_CMO;

Same here.

Also, why are these in kvm_pgtable.h? Can't they be moved inside pgtable.c and thus have the "inline" attribute dropped?

Quoted 17 lines
arch/arm64/include/asm/kvm_pgtable.h +2 -0
+}
+
 /**
  * struct kvm_pgtable_walker - Hook into a page-table walk.
  * @cb:		Callback function to invoke during the walk.
arch/arm64/kvm/hyp/pgtable.c +0 -4
index a3246d6cddec..4f703cc4cb03 100644
@@ -741,14 +741,17 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
 	if (!stage2_try_set_pte(ctx, KVM_INVALID_PTE_LOCKED))
 		return false;
 
-	/*
-	 * Perform the appropriate TLB invalidation based on the evicted pte
-	 * value (if any).
-	 */

arch/arm64/kvm/hyp/pgtable.c static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, · line 749

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +6 -4
-	if (kvm_pte_table(ctx->old, ctx->level))
-		kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
-	else if (kvm_pte_valid(ctx->old))
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+	if (!kvm_pgtable_walk_skip_bbm(ctx)) {
+		/*
+		 * Perform the appropriate TLB invalidation based on the
+		 * evicted pte value (if any).
+		 */
+		if (kvm_pte_table(ctx->old, ctx->level))

You're not skipping BBM here. You're skipping the TLB invalidation. Not quite the same thing.

Quoted 15 lines
arch/arm64/kvm/hyp/pgtable.c +7 -1
+			kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
+		else if (kvm_pte_valid(ctx->old))
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+				     ctx->addr, ctx->level);
+	}
 
 	if (stage2_pte_is_counted(ctx->old))
 		mm_ops->put_page(ctx->ptep);
@@ -832,11 +835,13 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx,
 		return -EAGAIN;
 
 	/* Perform CMOs before installation of the guest stage-2 PTE */
-	if (mm_ops->dcache_clean_inval_poc && stage2_pte_cacheable(pgt, new))
+	if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->dcache_clean_inval_poc &&
+	    stage2_pte_cacheable(pgt, new))

arch/arm64/kvm/hyp/pgtable.c static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx, · line 847

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +3 -2
 		mm_ops->dcache_clean_inval_poc(kvm_pte_follow(new, mm_ops),
-						granule);
+					       granule);
 
-	if (mm_ops->icache_inval_pou && stage2_pte_executable(new))
+	if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->icache_inval_pou &&
+	    stage2_pte_executable(new))
 		mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule);
 
 	stage2_make_pte(ctx, new);

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#5 lore raw

PATCH v6 3/12 KVM: arm64: Add helper for creating unlinked stage2 subtrees

In reply to Ricardo Koller

Add a stage2 helper, kvm_pgtable_stage2_create_unlinked(), for creating unlinked tables (which is the opposite of kvm_pgtable_stage2_free_unlinked()). Creating an unlinked table is useful for splitting PMD and PUD blocks into subtrees of PAGE_SIZE PTEs. For example, a PUD can be split into PAGE_SIZE PTEs by first creating a fully populated tree, and then use it to replace the PUD in a single step. This will be used in a subsequent commit for eager huge-page splitting (a dirty-logging optimization).

No functional change intended. This new function will be used in a subsequent commit.

---
 arch/arm64/include/asm/kvm_pgtable.h | 28 +++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c         | 46 ++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +28 -0
index c7a269cad053..b7b3fc0fa7a5 100644
@@ -468,6 +468,34 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
  */
 void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level);
 
+/**
+ * kvm_pgtable_stage2_create_unlinked() - Create an unlinked stage-2 paging structure.
+ * @pgt:	Page-table structure initialised by kvm_pgtable_stage2_init*().
+ * @phys:	Physical address of the memory to map.
+ * @level:	Starting level of the stage-2 paging structure to be created.
+ * @prot:	Permissions and attributes for the mapping.
+ * @mc:		Cache of pre-allocated and zeroed memory from which to allocate
+ *		page-table pages.
+ * @force_pte:  Force mappings to PAGE_SIZE granularity.
+ *
+ * Returns an unlinked page-table tree. If @force_pte is true or
+ * @level is 2 (the PMD level), then the tree is mapped up to the
+ * PAGE_SIZE leaf PTE; the tree is mapped up one level otherwise.
+ * This new page-table tree is not reachable (i.e., it is unlinked)
+ * from the root pgd and it's therefore unreachableby the hardware
+ * page-table walker. No TLB invalidation or CMOs are performed.
+ *
+ * If device attributes are not explicitly requested in @prot, then the
+ * mapping will be normal, cacheable.
+ *
+ * Return: The fully populated (unlinked) stage-2 paging structure, or
+ * an ERR_PTR(error) on failure.
+ */
+kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
+					      u64 phys, u32 level,
+					      enum kvm_pgtable_prot prot,
+					      void *mc, bool force_pte);
+
 /**
  * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table.
  * @pgt:	Page-table structure initialised by kvm_pgtable_stage2_init*().
arch/arm64/kvm/hyp/pgtable.c +46 -0

52 lines not on this page · raw message

Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
maintainer
#6 lore raw

Re: [PATCH v6 03/12] KVM: arm64: Add helper for creating unlinked stage2 subtrees

Quoted 5 lines · Ricardo Koller

Add a stage2 helper, kvm_pgtable_stage2_create_unlinked(), for creating unlinked tables (which is the opposite of kvm_pgtable_stage2_free_unlinked()). Creating an unlinked table is useful for splitting PMD and PUD blocks into subtrees of PAGE_SIZE

Please drop the PMD/PUD verbiage. That's specially confusing when everything is described in terms of 'level'

Quoted 1 line

PTEs. For example, a PUD can be split into PAGE_SIZE PTEs by first

for example: s/a PUD/a level 1 mapping/

Quoted 6 lines

creating a fully populated tree, and then use it to replace the PUD in a single step. This will be used in a subsequent commit for eager huge-page splitting (a dirty-logging optimization).

No functional change intended. This new function will be used in a subsequent commit.

Drop this last sentence, it doesn't say anything that you haven't already said.

Quoted 19 lines
---
 arch/arm64/include/asm/kvm_pgtable.h | 28 +++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c         | 46 ++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +3 -0
index c7a269cad053..b7b3fc0fa7a5 100644
@@ -468,6 +468,34 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
  */
 void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level);
 
+/**
+ * kvm_pgtable_stage2_create_unlinked() - Create an unlinked stage-2 paging structure.
+ * @pgt:	Page-table structure initialised by kvm_pgtable_stage2_init*().

arch/arm64/include/asm/kvm_pgtable.h void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); · line 483

Quoted 10 lines
arch/arm64/include/asm/kvm_pgtable.h +10 -0
+ * @phys:	Physical address of the memory to map.
+ * @level:	Starting level of the stage-2 paging structure to be created.
+ * @prot:	Permissions and attributes for the mapping.
+ * @mc:		Cache of pre-allocated and zeroed memory from which to allocate
+ *		page-table pages.
+ * @force_pte:  Force mappings to PAGE_SIZE granularity.
+ *
+ * Returns an unlinked page-table tree. If @force_pte is true or
+ * @level is 2 (the PMD level), then the tree is mapped up to the
+ * PAGE_SIZE leaf PTE; the tree is mapped up one level otherwise.

I wouldn't make this "one level" assumption, as this really depends on the size of what gets mapped (and future evolution of this code).

arch/arm64/include/asm/kvm_pgtable.h void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); · line 492

Quoted 9 lines
arch/arm64/include/asm/kvm_pgtable.h +9 -0
+ * This new page-table tree is not reachable (i.e., it is unlinked)
+ * from the root pgd and it's therefore unreachableby the hardware
+ * page-table walker. No TLB invalidation or CMOs are performed.
+ *
+ * If device attributes are not explicitly requested in @prot, then the
+ * mapping will be normal, cacheable.
+ *
+ * Return: The fully populated (unlinked) stage-2 paging structure, or
+ * an ERR_PTR(error) on failure.

What guarantees that this new unlinked structure is kept in sync with the original one? AFAICT, nothing does.

Quoted 29 lines
arch/arm64/include/asm/kvm_pgtable.h +6 -0
+ */
+kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
+					      u64 phys, u32 level,
+					      enum kvm_pgtable_prot prot,
+					      void *mc, bool force_pte);
+
 /**
  * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table.
  * @pgt:	Page-table structure initialised by kvm_pgtable_stage2_init*().
arch/arm64/kvm/hyp/pgtable.c +12 -0
index 4f703cc4cb03..6bdfcb671b32 100644
@@ -1212,6 +1212,52 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size)
 	return kvm_pgtable_walk(pgt, addr, size, &walker);
 }
 
+kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
+					      u64 phys, u32 level,
+					      enum kvm_pgtable_prot prot,
+					      void *mc, bool force_pte)
+{
+	struct stage2_map_data map_data = {
+		.phys		= phys,
+		.mmu		= pgt->mmu,
+		.memcache	= mc,
+		.force_pte	= force_pte,
+	};
+	struct kvm_pgtable_walker walker = {

arch/arm64/kvm/hyp/pgtable.c int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) · line 1236

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +10 -0
+		.cb		= stage2_map_walker,
+		.flags		= KVM_PGTABLE_WALK_LEAF |
+				  KVM_PGTABLE_WALK_SKIP_BBM |
+				  KVM_PGTABLE_WALK_SKIP_CMO,
+		.arg		= &map_data,
+	};
+	/* .addr (the IPA) is irrelevant for an unlinked table */
+	struct kvm_pgtable_walk_data data = {
+		.walker	= &walker,
+		.addr	= 0,

Is that always true? What if the caller expect a non-block-aligned mapping? You should at least check that phys is aligned to the granule size of 'level', or bad stuff may happen.

Quoted 17 lines
arch/arm64/kvm/hyp/pgtable.c +17 -0
+		.end	= kvm_granule_size(level),
+	};
+	struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops;
+	kvm_pte_t *pgtable;
+	int ret;
+
+	ret = stage2_set_prot_attr(pgt, prot, &map_data.attr);
+	if (ret)
+		return ERR_PTR(ret);
+
+	pgtable = mm_ops->zalloc_page(mc);
+	if (!pgtable)
+		return ERR_PTR(-ENOMEM);
+
+	ret = __kvm_pgtable_walk(&data, mm_ops, (kvm_pteref_t)pgtable,
+				 level + 1);
+	if (ret) {

arch/arm64/kvm/hyp/pgtable.c int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) · line 1263

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +7 -0
+		kvm_pgtable_stage2_free_unlinked(mm_ops, pgtable, level);
+		mm_ops->put_page(pgtable);
+		return ERR_PTR(ret);
+	}
+
+	return pgtable;
+}
 
 int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			      struct kvm_pgtable_mm_ops *mm_ops,
	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#7 lore raw

Re: [PATCH v6 03/12] KVM: arm64: Add helper for creating unlinked stage2 subtrees

Quoted 68 lines · Marc Zyngier
Quoted 5 lines · Ricardo Koller

Add a stage2 helper, kvm_pgtable_stage2_create_unlinked(), for creating unlinked tables (which is the opposite of kvm_pgtable_stage2_free_unlinked()). Creating an unlinked table is useful for splitting PMD and PUD blocks into subtrees of PAGE_SIZE

Please drop the PMD/PUD verbiage. That's specially confusing when everything is described in terms of 'level'

Quoted 1 line

PTEs. For example, a PUD can be split into PAGE_SIZE PTEs by first

for example: s/a PUD/a level 1 mapping/

Quoted 6 lines

creating a fully populated tree, and then use it to replace the PUD in a single step. This will be used in a subsequent commit for eager huge-page splitting (a dirty-logging optimization).

No functional change intended. This new function will be used in a subsequent commit.

Drop this last sentence, it doesn't say anything that you haven't already said.

Quoted 19 lines
---
 arch/arm64/include/asm/kvm_pgtable.h | 28 +++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c         | 46 ++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +3 -0
index c7a269cad053..b7b3fc0fa7a5 100644
@@ -468,6 +468,34 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
  */
 void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level);
 
+/**
+ * kvm_pgtable_stage2_create_unlinked() - Create an unlinked stage-2 paging structure.
+ * @pgt:	Page-table structure initialised by kvm_pgtable_stage2_init*().

arch/arm64/include/asm/kvm_pgtable.h void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); · line 483

Quoted 10 lines
arch/arm64/include/asm/kvm_pgtable.h +10 -0
+ * @phys:	Physical address of the memory to map.
+ * @level:	Starting level of the stage-2 paging structure to be created.
+ * @prot:	Permissions and attributes for the mapping.
+ * @mc:		Cache of pre-allocated and zeroed memory from which to allocate
+ *		page-table pages.
+ * @force_pte:  Force mappings to PAGE_SIZE granularity.
+ *
+ * Returns an unlinked page-table tree. If @force_pte is true or
+ * @level is 2 (the PMD level), then the tree is mapped up to the
+ * PAGE_SIZE leaf PTE; the tree is mapped up one level otherwise.

I wouldn't make this "one level" assumption, as this really depends on the size of what gets mapped (and future evolution of this code).

arch/arm64/include/asm/kvm_pgtable.h void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); · line 492

Quoted 9 lines
arch/arm64/include/asm/kvm_pgtable.h +9 -0
+ * This new page-table tree is not reachable (i.e., it is unlinked)
+ * from the root pgd and it's therefore unreachableby the hardware
+ * page-table walker. No TLB invalidation or CMOs are performed.
+ *
+ * If device attributes are not explicitly requested in @prot, then the
+ * mapping will be normal, cacheable.
+ *
+ * Return: The fully populated (unlinked) stage-2 paging structure, or
+ * an ERR_PTR(error) on failure.
Quoted 3 lines

What guarantees that this new unlinked structure is kept in sync with the original one? AFAICT, nothing does.

That should be the job of the caller: kvm_pgtable_stage2_split() in this series.

Quoted 35 lines
Quoted 35 lines
arch/arm64/include/asm/kvm_pgtable.h +6 -0
+ */
+kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
+					      u64 phys, u32 level,
+					      enum kvm_pgtable_prot prot,
+					      void *mc, bool force_pte);
+
 /**
  * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table.
  * @pgt:	Page-table structure initialised by kvm_pgtable_stage2_init*().
arch/arm64/kvm/hyp/pgtable.c +18 -0
index 4f703cc4cb03..6bdfcb671b32 100644
@@ -1212,6 +1212,52 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size)
 	return kvm_pgtable_walk(pgt, addr, size, &walker);
 }
 
+kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
+					      u64 phys, u32 level,
+					      enum kvm_pgtable_prot prot,
+					      void *mc, bool force_pte)
+{
+	struct stage2_map_data map_data = {
+		.phys		= phys,
+		.mmu		= pgt->mmu,
+		.memcache	= mc,
+		.force_pte	= force_pte,
+	};
+	struct kvm_pgtable_walker walker = {
+		.cb		= stage2_map_walker,
+		.flags		= KVM_PGTABLE_WALK_LEAF |
+				  KVM_PGTABLE_WALK_SKIP_BBM |
+				  KVM_PGTABLE_WALK_SKIP_CMO,
+		.arg		= &map_data,
+	};
Quoted 9 lines

arch/arm64/kvm/hyp/pgtable.c int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) · line 1236

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
+	/* .addr (the IPA) is irrelevant for an unlinked table */
+	struct kvm_pgtable_walk_data data = {
+		.walker	= &walker,
+		.addr	= 0,

Is that always true? What if the caller expect a non-block-aligned mapping? You should at least check that phys is aligned to the granule size of 'level', or bad stuff may happen.

Good point, it should be true, but I will add a check to fail with EINVAL on that case. Will also double check the caller's code to be sure mappings are always block-aligned.

Quoted 25 lines
Quoted 25 lines
arch/arm64/kvm/hyp/pgtable.c +24 -0
+		.end	= kvm_granule_size(level),
+	};
+	struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops;
+	kvm_pte_t *pgtable;
+	int ret;
+
+	ret = stage2_set_prot_attr(pgt, prot, &map_data.attr);
+	if (ret)
+		return ERR_PTR(ret);
+
+	pgtable = mm_ops->zalloc_page(mc);
+	if (!pgtable)
+		return ERR_PTR(-ENOMEM);
+
+	ret = __kvm_pgtable_walk(&data, mm_ops, (kvm_pteref_t)pgtable,
+				 level + 1);
+	if (ret) {
+		kvm_pgtable_stage2_free_unlinked(mm_ops, pgtable, level);
+		mm_ops->put_page(pgtable);
+		return ERR_PTR(ret);
+	}
+
+	return pgtable;
+}
 
Quoted 6 lines

arch/arm64/kvm/hyp/pgtable.c int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) · line 1263

Quoted 2 lines
arch/arm64/kvm/hyp/pgtable.c +0 -0
 int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			      struct kvm_pgtable_mm_ops *mm_ops,
	M.

ACK on all the other comments.

Thanks,
Ricardo

Quoted 1 line

Without deviation from the norm, progress is not possible.

#8 lore raw

PATCH v6 4/12 KVM: arm64: Add kvm_pgtable_stage2_split()

In reply to Ricardo Koller

Add a new stage2 function, kvm_pgtable_stage2_split(), for splitting a range of huge pages. This will be used for eager-splitting huge pages into PAGE_SIZE pages. The goal is to avoid having to split huge pages on write-protection faults, and instead use this function to do it ahead of time for large ranges (e.g., all guest memory in 1G chunks at a time).

No functional change intended. This new function will be used in a subsequent commit.

---
 arch/arm64/include/asm/kvm_pgtable.h |  30 +++++++
 arch/arm64/kvm/hyp/pgtable.c         | 113 +++++++++++++++++++++++++++
 2 files changed, 143 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +30 -0
index b7b3fc0fa7a5..40e323a718fc 100644
@@ -665,6 +665,36 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr);
  */
 int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);
 
+/**
+ * kvm_pgtable_stage2_split() - Split a range of huge pages into leaf PTEs pointing
+ *				to PAGE_SIZE guest pages.
+ * @pgt:	 Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr:	 Intermediate physical address from which to split.
+ * @size:	 Size of the range.
+ * @mc:		 Cache of pre-allocated and zeroed memory from which to allocate
+ *		 page-table pages.
+ * @mc_capacity: Number of pages in @mc.
+ *
+ * @addr and the end (@addr + @size) are effectively aligned down and up to
+ * the top level huge-page block size. This is an example using 1GB
+ * huge-pages and 4KB granules.
+ *
+ *                          [---input range---]
+ *                          :                 :
+ * [--1G block pte--][--1G block pte--][--1G block pte--][--1G block pte--]
+ *                          :                 :
+ *                   [--2MB--][--2MB--][--2MB--][--2MB--]
+ *                          :                 :
+ *                   [ ][ ][:][ ][ ][ ][ ][ ][:][ ][ ][ ]
+ *                          :                 :
+ *
+ * Return: 0 on success, negative error code on failure. Note that
+ * kvm_pgtable_stage2_split() is best effort: it tries to break as many
+ * blocks in the input range as allowed by @mc_capacity.
+ */
+int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,
+			     void *mc, u64 mc_capacity);
+
 /**
  * kvm_pgtable_walk() - Walk a page-table.
  * @pgt:	Page-table structure initialised by kvm_pgtable_*_init().
arch/arm64/kvm/hyp/pgtable.c +113 -0

119 lines not on this page · raw message

Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
maintainer
#9 lore raw

Re: [PATCH v6 04/12] KVM: arm64: Add kvm_pgtable_stage2_split()

Quoted 10 lines · Ricardo Koller

Add a new stage2 function, kvm_pgtable_stage2_split(), for splitting a range of huge pages. This will be used for eager-splitting huge pages into PAGE_SIZE pages. The goal is to avoid having to split huge pages on write-protection faults, and instead use this function to do it ahead of time for large ranges (e.g., all guest memory in 1G chunks at a time).

No functional change intended. This new function will be used in a subsequent commit.

Same comment as before about the usefulness of this last sentence.

Quoted 28 lines
---
 arch/arm64/include/asm/kvm_pgtable.h |  30 +++++++
 arch/arm64/kvm/hyp/pgtable.c         | 113 +++++++++++++++++++++++++++
 2 files changed, 143 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +12 -0
index b7b3fc0fa7a5..40e323a718fc 100644
@@ -665,6 +665,36 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr);
  */
 int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);
 
+/**
+ * kvm_pgtable_stage2_split() - Split a range of huge pages into leaf PTEs pointing
+ *				to PAGE_SIZE guest pages.
+ * @pgt:	 Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr:	 Intermediate physical address from which to split.
+ * @size:	 Size of the range.
+ * @mc:		 Cache of pre-allocated and zeroed memory from which to allocate
+ *		 page-table pages.
+ * @mc_capacity: Number of pages in @mc.
+ *
+ * @addr and the end (@addr + @size) are effectively aligned down and up to
+ * the top level huge-page block size. This is an example using 1GB

arch/arm64/include/asm/kvm_pgtable.h bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr); · line 689

Quoted 10 lines
arch/arm64/include/asm/kvm_pgtable.h +10 -0
+ * huge-pages and 4KB granules.
+ *
+ *                          [---input range---]
+ *                          :                 :
+ * [--1G block pte--][--1G block pte--][--1G block pte--][--1G block pte--]
+ *                          :                 :
+ *                   [--2MB--][--2MB--][--2MB--][--2MB--]
+ *                          :                 :
+ *                   [ ][ ][:][ ][ ][ ][ ][ ][:][ ][ ][ ]
+ *                          :                 :

So here, what alignment do we effectively get?

Quoted 16 lines
arch/arm64/include/asm/kvm_pgtable.h +8 -0
+ *
+ * Return: 0 on success, negative error code on failure. Note that
+ * kvm_pgtable_stage2_split() is best effort: it tries to break as many
+ * blocks in the input range as allowed by @mc_capacity.
+ */
+int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,
+			     void *mc, u64 mc_capacity);
+
 /**
  * kvm_pgtable_walk() - Walk a page-table.
  * @pgt:	Page-table structure initialised by kvm_pgtable_*_init().
arch/arm64/kvm/hyp/pgtable.c +0 -0
index 6bdfcb671b32..3149b98d1701 100644

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1265

Quoted 7 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
@@ -1259,6 +1259,119 @@ kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
 	return pgtable;
 }
 
+struct stage2_split_data {
+	struct kvm_s2_mmu		*mmu;
+	void				*memcache;
+	u64				mc_capacity;

Why isn't this a pointer to a *real* memcache structure?

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1272

Quoted 7 lines
arch/arm64/kvm/hyp/pgtable.c +7 -0
+};
+
+/*
+ * Get the number of page-tables needed to replace a block with a
+ * fully populated tree, up to the PTE level, at particular level.
+ */
+static inline int stage2_block_get_nr_page_tables(u32 level)

Please drop the inline. The compiler will figure it out.

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1276

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
+{
+	if (WARN_ON_ONCE(level < KVM_PGTABLE_MIN_BLOCK_LEVEL ||
+			 level >= KVM_PGTABLE_MAX_LEVELS))
+		return -EINVAL;

Move this check to the 'default' case below.

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1282

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +6 -0
+
+	switch (level) {
+	case 1:
+		return PTRS_PER_PTE + 1;
+	case 2:
+		return 1;

This is odd. Replacing a block by a table always requires 'PTRS_PER_PTE + 1' pages. Why 1? If this is some special treatment for level-2 mappings, please spell it out.

Quoted 13 lines
arch/arm64/kvm/hyp/pgtable.c +13 -0
+	case 3:
+		return 0;
+	default:
+		return -EINVAL;
+	};
+}
+
+static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx,
+			       enum kvm_pgtable_walk_flags visit)
+{
+	struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
+	struct stage2_split_data *data = ctx->arg;
+	kvm_pte_t pte = ctx->old, new, *childp;

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1305

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +10 -0
+	enum kvm_pgtable_prot prot;
+	void *mc = data->memcache;
+	u32 level = ctx->level;
+	bool force_pte;
+	int nr_pages;
+	u64 phys;
+
+	/* No huge-pages exist at the last level */
+	if (level == KVM_PGTABLE_MAX_LEVELS - 1)
+		return 0;

Why the check for level 3 in the previous function if never get there?

Quoted 8 lines
arch/arm64/kvm/hyp/pgtable.c +8 -0
+
+	/* We only split valid block mappings */
+	if (!kvm_pte_valid(pte))
+		return 0;
+
+	nr_pages = stage2_block_get_nr_page_tables(level);
+	if (nr_pages < 0)
+		return nr_pages;

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1323

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +10 -0
+
+	if (data->mc_capacity >= nr_pages) {
+		/* Build a tree mapped down to the PTE granularity. */
+		force_pte = true;
+	} else {
+		/*
+		 * Don't force PTEs. This requires a single page of PMDs at the
+		 * PUD level, or a single page of PTEs at the PMD level. If we
+		 * are at the PUD level, the PTEs will be created recursively.
+		 */

I don't understand how you reach this 'single page' conclusion. You need to explain why you get there.

Quoted 44 lines
arch/arm64/kvm/hyp/pgtable.c +44 -0
+		force_pte = false;
+		nr_pages = 1;
+	}
+
+	if (data->mc_capacity < nr_pages)
+		return -ENOMEM;
+
+	phys = kvm_pte_to_phys(pte);
+	prot = kvm_pgtable_stage2_pte_prot(pte);
+
+	childp = kvm_pgtable_stage2_create_unlinked(data->mmu->pgt, phys,
+						    level, prot, mc, force_pte);
+	if (IS_ERR(childp))
+		return PTR_ERR(childp);
+
+	if (!stage2_try_break_pte(ctx, data->mmu)) {
+		kvm_pgtable_stage2_free_unlinked(mm_ops, childp, level);
+		mm_ops->put_page(childp);
+		return -EAGAIN;
+	}
+
+	/*
+	 * Note, the contents of the page table are guaranteed to be made
+	 * visible before the new PTE is assigned because stage2_make_pte()
+	 * writes the PTE using smp_store_release().
+	 */
+	new = kvm_init_table_pte(childp, mm_ops);
+	stage2_make_pte(ctx, new);
+	dsb(ishst);
+	data->mc_capacity -= nr_pages;
+	return 0;
+}
+
+int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,
+			     void *mc, u64 mc_capacity)
+{
+	struct stage2_split_data split_data = {
+		.mmu		= pgt->mmu,
+		.memcache	= mc,
+		.mc_capacity	= mc_capacity,
+	};
+
+	struct kvm_pgtable_walker walker = {
+		.cb	= stage2_split_walker,

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1377

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +7 -0
+		.flags	= KVM_PGTABLE_WALK_LEAF,
+		.arg	= &split_data,
+	};
+
+	return kvm_pgtable_walk(pgt, addr, size, &walker);
+}
+
 int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			      struct kvm_pgtable_mm_ops *mm_ops,
 			      enum kvm_pgtable_stage2_flags flags,

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#10 lore raw

Re: [PATCH v6 04/12] KVM: arm64: Add kvm_pgtable_stage2_split()

Quoted 47 lines · Marc Zyngier
Quoted 10 lines · Ricardo Koller

Add a new stage2 function, kvm_pgtable_stage2_split(), for splitting a range of huge pages. This will be used for eager-splitting huge pages into PAGE_SIZE pages. The goal is to avoid having to split huge pages on write-protection faults, and instead use this function to do it ahead of time for large ranges (e.g., all guest memory in 1G chunks at a time).

No functional change intended. This new function will be used in a subsequent commit.

Same comment as before about the usefulness of this last sentence.

Quoted 32 lines
---
 arch/arm64/include/asm/kvm_pgtable.h |  30 +++++++
 arch/arm64/kvm/hyp/pgtable.c         | 113 +++++++++++++++++++++++++++
 2 files changed, 143 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +16 -0
index b7b3fc0fa7a5..40e323a718fc 100644
@@ -665,6 +665,36 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr);
  */
 int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);
 
+/**
+ * kvm_pgtable_stage2_split() - Split a range of huge pages into leaf PTEs pointing
+ *				to PAGE_SIZE guest pages.
+ * @pgt:	 Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr:	 Intermediate physical address from which to split.
+ * @size:	 Size of the range.
+ * @mc:		 Cache of pre-allocated and zeroed memory from which to allocate
+ *		 page-table pages.
+ * @mc_capacity: Number of pages in @mc.
+ *
+ * @addr and the end (@addr + @size) are effectively aligned down and up to
+ * the top level huge-page block size. This is an example using 1GB
+ * huge-pages and 4KB granules.
+ *
+ *                          [---input range---]
+ *                          :                 :
Quoted 9 lines

arch/arm64/include/asm/kvm_pgtable.h bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr); · line 689

Quoted 6 lines
arch/arm64/include/asm/kvm_pgtable.h +6 -0
+ * [--1G block pte--][--1G block pte--][--1G block pte--][--1G block pte--]
+ *                          :                 :
+ *                   [--2MB--][--2MB--][--2MB--][--2MB--]
+ *                          :                 :
+ *                   [ ][ ][:][ ][ ][ ][ ][ ][:][ ][ ][ ]
+ *                          :                 :

So here, what alignment do we effectively get?

The function tries to split any block that overlaps with the input range. Here's another example that might be more helpful:

* [---input range---]
* : :
* [--1G block pte--][--2MB--][--2MB--][--1G block pte--][--1G block pte--]

is split like this:

* [--1G block pte--][--2MB--][ ][ ][ ][ ][ ][ ][ ][ ][ ][--1G block pte--]

                              <-------split range------->

I think I will just use this new description instead.

Quoted 18 lines
Quoted 18 lines
arch/arm64/include/asm/kvm_pgtable.h +7 -0
+ * Return: 0 on success, negative error code on failure. Note that
+ * kvm_pgtable_stage2_split() is best effort: it tries to break as many
+ * blocks in the input range as allowed by @mc_capacity.
+ */
+int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,
+			     void *mc, u64 mc_capacity);
+
 /**
  * kvm_pgtable_walk() - Walk a page-table.
  * @pgt:	Page-table structure initialised by kvm_pgtable_*_init().
arch/arm64/kvm/hyp/pgtable.c +0 -0
index 6bdfcb671b32..3149b98d1701 100644
@@ -1259,6 +1259,119 @@ kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
 	return pgtable;
 }
 
Quoted 7 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1265

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
+struct stage2_split_data {
+	struct kvm_s2_mmu		*mmu;
+	void				*memcache;
+	u64				mc_capacity;

Why isn't this a pointer to a *real* memcache structure?

Mainly because I wanted this function to be like the other pgtable.c funtions that use opaque pointers to handle the vhe and nvhe cases. vhe uses "struct kvm_mmu_memory_cache" while nvhe uses "struct hyp_pool". This series only implements the vhe case but I didn't want to restrict kvm_pgtable_stage2_split() to vhe just because of this. Just in case, I have not tried it in nvhe.

Quoted 17 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1272

Quoted 7 lines
arch/arm64/kvm/hyp/pgtable.c +7 -0
+};
+
+/*
+ * Get the number of page-tables needed to replace a block with a
+ * fully populated tree, up to the PTE level, at particular level.
+ */
+static inline int stage2_block_get_nr_page_tables(u32 level)

Please drop the inline. The compiler will figure it out.

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1276

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
+{
+	if (WARN_ON_ONCE(level < KVM_PGTABLE_MIN_BLOCK_LEVEL ||
+			 level >= KVM_PGTABLE_MAX_LEVELS))
+		return -EINVAL;

Move this check to the 'default' case below.

Quoted 10 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1282

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +6 -0
+
+	switch (level) {
+	case 1:
+		return PTRS_PER_PTE + 1;
+	case 2:
+		return 1;

This is odd. Replacing a block by a table always requires 'PTRS_PER_PTE + 1' pages. Why 1? If this is some special treatment for level-2 mappings, please spell it out.

I'm not sure I understand. I'm interpreting "level=X" as in "level X entry". More specifically, using PAGE_SIZE=4096 as an example:

a level 3 entry (a PTE): a 4096 block needs 0 page-table pages a level 2 entry: a 2M block needs 1 page-table pages a level 1 entry: a 1G block needs 512+1 page-table pages

Quoted 18 lines
Quoted 18 lines
arch/arm64/kvm/hyp/pgtable.c +17 -0
+	case 3:
+		return 0;
+	default:
+		return -EINVAL;
+	};
+}
+
+static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx,
+			       enum kvm_pgtable_walk_flags visit)
+{
+	struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
+	struct stage2_split_data *data = ctx->arg;
+	kvm_pte_t pte = ctx->old, new, *childp;
+	enum kvm_pgtable_prot prot;
+	void *mc = data->memcache;
+	u32 level = ctx->level;
+	bool force_pte;
Quoted 9 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1305

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +6 -0
+	int nr_pages;
+	u64 phys;
+
+	/* No huge-pages exist at the last level */
+	if (level == KVM_PGTABLE_MAX_LEVELS - 1)
+		return 0;

Why the check for level 3 in the previous function if never get there?

Was trying to make stage2_block_get_nr_page_tables() useful for other cases. It's still correct for other cases to ask how many page-table pages are needed for a PTE (stage2_block_get_nr_page_tables(3) -> 0).

Quoted 13 lines
Quoted 13 lines
arch/arm64/kvm/hyp/pgtable.c +13 -0
+
+	/* We only split valid block mappings */
+	if (!kvm_pte_valid(pte))
+		return 0;
+
+	nr_pages = stage2_block_get_nr_page_tables(level);
+	if (nr_pages < 0)
+		return nr_pages;
+
+	if (data->mc_capacity >= nr_pages) {
+		/* Build a tree mapped down to the PTE granularity. */
+		force_pte = true;
+	} else {
Quoted 8 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1323

Quoted 5 lines
arch/arm64/kvm/hyp/pgtable.c +5 -0
+		/*
+		 * Don't force PTEs. This requires a single page of PMDs at the
+		 * PUD level, or a single page of PTEs at the PMD level. If we
+		 * are at the PUD level, the PTEs will be created recursively.
+		 */

I don't understand how you reach this 'single page' conclusion. You need to explain why you get there.

Ack, will expand it.

Quoted 62 lines
Quoted 45 lines
arch/arm64/kvm/hyp/pgtable.c +44 -0
+		force_pte = false;
+		nr_pages = 1;
+	}
+
+	if (data->mc_capacity < nr_pages)
+		return -ENOMEM;
+
+	phys = kvm_pte_to_phys(pte);
+	prot = kvm_pgtable_stage2_pte_prot(pte);
+
+	childp = kvm_pgtable_stage2_create_unlinked(data->mmu->pgt, phys,
+						    level, prot, mc, force_pte);
+	if (IS_ERR(childp))
+		return PTR_ERR(childp);
+
+	if (!stage2_try_break_pte(ctx, data->mmu)) {
+		kvm_pgtable_stage2_free_unlinked(mm_ops, childp, level);
+		mm_ops->put_page(childp);
+		return -EAGAIN;
+	}
+
+	/*
+	 * Note, the contents of the page table are guaranteed to be made
+	 * visible before the new PTE is assigned because stage2_make_pte()
+	 * writes the PTE using smp_store_release().
+	 */
+	new = kvm_init_table_pte(childp, mm_ops);
+	stage2_make_pte(ctx, new);
+	dsb(ishst);
+	data->mc_capacity -= nr_pages;
+	return 0;
+}
+
+int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,
+			     void *mc, u64 mc_capacity)
+{
+	struct stage2_split_data split_data = {
+		.mmu		= pgt->mmu,
+		.memcache	= mc,
+		.mc_capacity	= mc_capacity,
+	};
+
+	struct kvm_pgtable_walker walker = {
+		.cb	= stage2_split_walker,

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1377

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +7 -0
+		.flags	= KVM_PGTABLE_WALK_LEAF,
+		.arg	= &split_data,
+	};
+
+	return kvm_pgtable_walk(pgt, addr, size, &walker);
+}
+
 int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			      struct kvm_pgtable_mm_ops *mm_ops,
 			      enum kvm_pgtable_stage2_flags flags,

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
maintainer
#11 lore raw

Re: [PATCH v6 04/12] KVM: arm64: Add kvm_pgtable_stage2_split()

Quoted 99 lines · Ricardo Koller
Quoted 47 lines · Marc Zyngier
Quoted 10 lines · Ricardo Koller

Add a new stage2 function, kvm_pgtable_stage2_split(), for splitting a range of huge pages. This will be used for eager-splitting huge pages into PAGE_SIZE pages. The goal is to avoid having to split huge pages on write-protection faults, and instead use this function to do it ahead of time for large ranges (e.g., all guest memory in 1G chunks at a time).

No functional change intended. This new function will be used in a subsequent commit.

Same comment as before about the usefulness of this last sentence.

Quoted 32 lines
---
 arch/arm64/include/asm/kvm_pgtable.h |  30 +++++++
 arch/arm64/kvm/hyp/pgtable.c         | 113 +++++++++++++++++++++++++++
 2 files changed, 143 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +16 -0
index b7b3fc0fa7a5..40e323a718fc 100644
@@ -665,6 +665,36 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr);
  */
 int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);
 
+/**
+ * kvm_pgtable_stage2_split() - Split a range of huge pages into leaf PTEs pointing
+ *				to PAGE_SIZE guest pages.
+ * @pgt:	 Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr:	 Intermediate physical address from which to split.
+ * @size:	 Size of the range.
+ * @mc:		 Cache of pre-allocated and zeroed memory from which to allocate
+ *		 page-table pages.
+ * @mc_capacity: Number of pages in @mc.
+ *
+ * @addr and the end (@addr + @size) are effectively aligned down and up to
+ * the top level huge-page block size. This is an example using 1GB
+ * huge-pages and 4KB granules.
+ *
+ *                          [---input range---]
+ *                          :                 :
Quoted 9 lines

arch/arm64/include/asm/kvm_pgtable.h bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr); · line 689

Quoted 6 lines
arch/arm64/include/asm/kvm_pgtable.h +6 -0
+ * [--1G block pte--][--1G block pte--][--1G block pte--][--1G block pte--]
+ *                          :                 :
+ *                   [--2MB--][--2MB--][--2MB--][--2MB--]
+ *                          :                 :
+ *                   [ ][ ][:][ ][ ][ ][ ][ ][:][ ][ ][ ]
+ *                          :                 :

So here, what alignment do we effectively get?

The function tries to split any block that overlaps with the input range. Here's another example that might be more helpful:

* [---input range---]
* : :
* [--1G block pte--][--2MB--][--2MB--][--1G block pte--][--1G block pte--]

is split like this:

* [--1G block pte--][--2MB--][ ][ ][ ][ ][ ][ ][ ][ ][ ][--1G block pte--]

                              <-------split range------->

I think I will just use this new description instead.

Quoted 18 lines
Quoted 18 lines
arch/arm64/include/asm/kvm_pgtable.h +7 -0
+ * Return: 0 on success, negative error code on failure. Note that
+ * kvm_pgtable_stage2_split() is best effort: it tries to break as many
+ * blocks in the input range as allowed by @mc_capacity.
+ */
+int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,
+			     void *mc, u64 mc_capacity);
+
 /**
  * kvm_pgtable_walk() - Walk a page-table.
  * @pgt:	Page-table structure initialised by kvm_pgtable_*_init().
arch/arm64/kvm/hyp/pgtable.c +0 -0
index 6bdfcb671b32..3149b98d1701 100644
@@ -1259,6 +1259,119 @@ kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
 	return pgtable;
 }
 
Quoted 7 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1265

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
+struct stage2_split_data {
+	struct kvm_s2_mmu		*mmu;
+	void				*memcache;
+	u64				mc_capacity;

Why isn't this a pointer to a *real* memcache structure?

Quoted 6 lines

Mainly because I wanted this function to be like the other pgtable.c funtions that use opaque pointers to handle the vhe and nvhe cases. vhe uses "struct kvm_mmu_memory_cache" while nvhe uses "struct hyp_pool". This series only implements the vhe case but I didn't want to restrict kvm_pgtable_stage2_split() to vhe just because of this. Just in case, I have not tried it in nvhe.

Do you really mean nVHE here? or do you actually mean pKVM? The former shouldn't be any different from VHE (the PT code runs in the same context, give or take), and it is only the latter that is using something else altogether.

And since pKVM cannot really do page splitting in the normal KVM sense, this code shouldn't even be there!

Quoted 29 lines
Quoted 18 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1272

Quoted 8 lines
arch/arm64/kvm/hyp/pgtable.c +7 -0
+};
+
+/*
+ * Get the number of page-tables needed to replace a block with a
+ * fully populated tree, up to the PTE level, at particular level.
+ */
+static inline int stage2_block_get_nr_page_tables(u32 level)

Please drop the inline. The compiler will figure it out.

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1276

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
+{
+	if (WARN_ON_ONCE(level < KVM_PGTABLE_MIN_BLOCK_LEVEL ||
+			 level >= KVM_PGTABLE_MAX_LEVELS))
+		return -EINVAL;

Move this check to the 'default' case below.

Quoted 10 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1282

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +6 -0
+
+	switch (level) {
+	case 1:
+		return PTRS_PER_PTE + 1;
+	case 2:
+		return 1;

This is odd. Replacing a block by a table always requires 'PTRS_PER_PTE + 1' pages. Why 1? If this is some special treatment for level-2 mappings, please spell it out.

Quoted 6 lines

I'm not sure I understand. I'm interpreting "level=X" as in "level X entry". More specifically, using PAGE_SIZE=4096 as an example:

a level 3 entry (a PTE): a 4096 block needs 0 page-table pages a level 2 entry: a 2M block needs 1 page-table pages a level 1 entry: a 1G block needs 512+1 page-table pages

Ah, gotcha. I was reasoning at the block level, not at the entry level. Maybe some extra idiot-proof explanation would help here for the next time I look at this after having paged it out.

Quoted 29 lines
Quoted 19 lines
Quoted 19 lines
arch/arm64/kvm/hyp/pgtable.c +17 -0
+	case 3:
+		return 0;
+	default:
+		return -EINVAL;
+	};
+}
+
+static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx,
+			       enum kvm_pgtable_walk_flags visit)
+{
+	struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
+	struct stage2_split_data *data = ctx->arg;
+	kvm_pte_t pte = ctx->old, new, *childp;
+	enum kvm_pgtable_prot prot;
+	void *mc = data->memcache;
+	u32 level = ctx->level;
+	bool force_pte;
Quoted 9 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1305

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +6 -0
+	int nr_pages;
+	u64 phys;
+
+	/* No huge-pages exist at the last level */
+	if (level == KVM_PGTABLE_MAX_LEVELS - 1)
+		return 0;

Why the check for level 3 in the previous function if never get there?

Quoted 3 lines

Was trying to make stage2_block_get_nr_page_tables() useful for other cases. It's still correct for other cases to ask how many page-table pages are needed for a PTE (stage2_block_get_nr_page_tables(3) -> 0).

Right. I don't mind either way, but the double check somehow tickled me.

Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
arch/arm64/kvm/hyp/pgtable.c +13 -0
+
+	/* We only split valid block mappings */
+	if (!kvm_pte_valid(pte))
+		return 0;
+
+	nr_pages = stage2_block_get_nr_page_tables(level);
+	if (nr_pages < 0)
+		return nr_pages;
+
+	if (data->mc_capacity >= nr_pages) {
+		/* Build a tree mapped down to the PTE granularity. */
+		force_pte = true;
+	} else {
Quoted 8 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1323

Quoted 5 lines
arch/arm64/kvm/hyp/pgtable.c +5 -0
+		/*
+		 * Don't force PTEs. This requires a single page of PMDs at the
+		 * PUD level, or a single page of PTEs at the PMD level. If we
+		 * are at the PUD level, the PTEs will be created recursively.
+		 */

I don't understand how you reach this 'single page' conclusion. You need to explain why you get there.

Quoted 1 line

Ack, will expand it.

Thanks. The above explanation you gave helped, so something long these lines would be good.

Cheers,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#12 lore raw

Re: [PATCH v6 04/12] KVM: arm64: Add kvm_pgtable_stage2_split()

Quoted 108 lines · Marc Zyngier
Quoted 99 lines · Ricardo Koller
Quoted 47 lines · Marc Zyngier
Quoted 10 lines · Ricardo Koller

Add a new stage2 function, kvm_pgtable_stage2_split(), for splitting a range of huge pages. This will be used for eager-splitting huge pages into PAGE_SIZE pages. The goal is to avoid having to split huge pages on write-protection faults, and instead use this function to do it ahead of time for large ranges (e.g., all guest memory in 1G chunks at a time).

No functional change intended. This new function will be used in a subsequent commit.

Same comment as before about the usefulness of this last sentence.

Quoted 32 lines
---
 arch/arm64/include/asm/kvm_pgtable.h |  30 +++++++
 arch/arm64/kvm/hyp/pgtable.c         | 113 +++++++++++++++++++++++++++
 2 files changed, 143 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +16 -0
index b7b3fc0fa7a5..40e323a718fc 100644
@@ -665,6 +665,36 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr);
  */
 int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);
 
+/**
+ * kvm_pgtable_stage2_split() - Split a range of huge pages into leaf PTEs pointing
+ *				to PAGE_SIZE guest pages.
+ * @pgt:	 Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr:	 Intermediate physical address from which to split.
+ * @size:	 Size of the range.
+ * @mc:		 Cache of pre-allocated and zeroed memory from which to allocate
+ *		 page-table pages.
+ * @mc_capacity: Number of pages in @mc.
+ *
+ * @addr and the end (@addr + @size) are effectively aligned down and up to
+ * the top level huge-page block size. This is an example using 1GB
+ * huge-pages and 4KB granules.
+ *
+ *                          [---input range---]
+ *                          :                 :
Quoted 9 lines

arch/arm64/include/asm/kvm_pgtable.h bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr); · line 689

Quoted 6 lines
arch/arm64/include/asm/kvm_pgtable.h +6 -0
+ * [--1G block pte--][--1G block pte--][--1G block pte--][--1G block pte--]
+ *                          :                 :
+ *                   [--2MB--][--2MB--][--2MB--][--2MB--]
+ *                          :                 :
+ *                   [ ][ ][:][ ][ ][ ][ ][ ][:][ ][ ][ ]
+ *                          :                 :

So here, what alignment do we effectively get?

The function tries to split any block that overlaps with the input range. Here's another example that might be more helpful:

* [---input range---]
* : :
* [--1G block pte--][--2MB--][--2MB--][--1G block pte--][--1G block pte--]

is split like this:

* [--1G block pte--][--2MB--][ ][ ][ ][ ][ ][ ][ ][ ][ ][--1G block pte--]

                              <-------split range------->

I think I will just use this new description instead.

Quoted 18 lines
Quoted 18 lines
arch/arm64/include/asm/kvm_pgtable.h +7 -0
+ * Return: 0 on success, negative error code on failure. Note that
+ * kvm_pgtable_stage2_split() is best effort: it tries to break as many
+ * blocks in the input range as allowed by @mc_capacity.
+ */
+int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,
+			     void *mc, u64 mc_capacity);
+
 /**
  * kvm_pgtable_walk() - Walk a page-table.
  * @pgt:	Page-table structure initialised by kvm_pgtable_*_init().
arch/arm64/kvm/hyp/pgtable.c +0 -0
index 6bdfcb671b32..3149b98d1701 100644
@@ -1259,6 +1259,119 @@ kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt,
 	return pgtable;
 }
 
Quoted 7 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1265

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
+struct stage2_split_data {
+	struct kvm_s2_mmu		*mmu;
+	void				*memcache;
+	u64				mc_capacity;

Why isn't this a pointer to a *real* memcache structure?

Quoted 6 lines

Mainly because I wanted this function to be like the other pgtable.c funtions that use opaque pointers to handle the vhe and nvhe cases. vhe uses "struct kvm_mmu_memory_cache" while nvhe uses "struct hyp_pool". This series only implements the vhe case but I didn't want to restrict kvm_pgtable_stage2_split() to vhe just because of this. Just in case, I have not tried it in nvhe.

Quoted 8 lines

Do you really mean nVHE here? or do you actually mean pKVM? The former shouldn't be any different from VHE (the PT code runs in the same context, give or take), and it is only the latter that is using something else altogether.

And since pKVM cannot really do page splitting in the normal KVM sense, this code shouldn't even be there!

I see, then this should definitely be using "struct kvm_mmu_memory_cache". Thanks for the info.

Quoted 109 lines
Quoted 29 lines
Quoted 18 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1272

Quoted 8 lines
arch/arm64/kvm/hyp/pgtable.c +7 -0
+};
+
+/*
+ * Get the number of page-tables needed to replace a block with a
+ * fully populated tree, up to the PTE level, at particular level.
+ */
+static inline int stage2_block_get_nr_page_tables(u32 level)

Please drop the inline. The compiler will figure it out.

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1276

Quoted 4 lines
arch/arm64/kvm/hyp/pgtable.c +4 -0
+{
+	if (WARN_ON_ONCE(level < KVM_PGTABLE_MIN_BLOCK_LEVEL ||
+			 level >= KVM_PGTABLE_MAX_LEVELS))
+		return -EINVAL;

Move this check to the 'default' case below.

Quoted 10 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1282

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +6 -0
+
+	switch (level) {
+	case 1:
+		return PTRS_PER_PTE + 1;
+	case 2:
+		return 1;

This is odd. Replacing a block by a table always requires 'PTRS_PER_PTE + 1' pages. Why 1? If this is some special treatment for level-2 mappings, please spell it out.

Quoted 6 lines

I'm not sure I understand. I'm interpreting "level=X" as in "level X entry". More specifically, using PAGE_SIZE=4096 as an example:

a level 3 entry (a PTE): a 4096 block needs 0 page-table pages a level 2 entry: a 2M block needs 1 page-table pages a level 1 entry: a 1G block needs 512+1 page-table pages

Ah, gotcha. I was reasoning at the block level, not at the entry level. Maybe some extra idiot-proof explanation would help here for the next time I look at this after having paged it out.

Quoted 29 lines
Quoted 19 lines
Quoted 19 lines
arch/arm64/kvm/hyp/pgtable.c +17 -0
+	case 3:
+		return 0;
+	default:
+		return -EINVAL;
+	};
+}
+
+static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx,
+			       enum kvm_pgtable_walk_flags visit)
+{
+	struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
+	struct stage2_split_data *data = ctx->arg;
+	kvm_pte_t pte = ctx->old, new, *childp;
+	enum kvm_pgtable_prot prot;
+	void *mc = data->memcache;
+	u32 level = ctx->level;
+	bool force_pte;
Quoted 9 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1305

Quoted 6 lines
arch/arm64/kvm/hyp/pgtable.c +6 -0
+	int nr_pages;
+	u64 phys;
+
+	/* No huge-pages exist at the last level */
+	if (level == KVM_PGTABLE_MAX_LEVELS - 1)
+		return 0;

Why the check for level 3 in the previous function if never get there?

Quoted 3 lines

Was trying to make stage2_block_get_nr_page_tables() useful for other cases. It's still correct for other cases to ask how many page-table pages are needed for a PTE (stage2_block_get_nr_page_tables(3) -> 0).

Right. I don't mind either way, but the double check somehow tickled me.

Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
arch/arm64/kvm/hyp/pgtable.c +13 -0
+
+	/* We only split valid block mappings */
+	if (!kvm_pte_valid(pte))
+		return 0;
+
+	nr_pages = stage2_block_get_nr_page_tables(level);
+	if (nr_pages < 0)
+		return nr_pages;
+
+	if (data->mc_capacity >= nr_pages) {
+		/* Build a tree mapped down to the PTE granularity. */
+		force_pte = true;
+	} else {
Quoted 8 lines

arch/arm64/kvm/hyp/pgtable.c kvm_pte_t *kvm_pgtable_stage2_create_unlinked(struct kvm_pgtable *pgt, · line 1323

Quoted 5 lines
arch/arm64/kvm/hyp/pgtable.c +5 -0
+		/*
+		 * Don't force PTEs. This requires a single page of PMDs at the
+		 * PUD level, or a single page of PTEs at the PMD level. If we
+		 * are at the PUD level, the PTEs will be created recursively.
+		 */

I don't understand how you reach this 'single page' conclusion. You need to explain why you get there.

Quoted 1 line

Ack, will expand it.

Thanks. The above explanation you gave helped, so something long these lines would be good.

Cheers,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#13 lore raw

PATCH v6 5/12 KVM: arm64: Refactor kvm_arch_commit_memory_region()

In reply to Ricardo Koller

Refactor kvm_arch_commit_memory_region() as a preparation for a future commit to look cleaner and more understandable. Also, it looks more like its x86 counterpart (in kvm_mmu_slot_apply_flags()).

No functional change intended.

---
 arch/arm64/kvm/mmu.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
arch/arm64/kvm/mmu.c +11 -4
index efdaab3f154d..37d7d2aa472a 100644
@@ -1761,20 +1761,27 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 				   const struct kvm_memory_slot *new,
 				   enum kvm_mr_change change)
 {
+	bool log_dirty_pages = new && new->flags & KVM_MEM_LOG_DIRTY_PAGES;
+
 	/*
 	 * At this point memslot has been committed and there is an
 	 * allocated dirty_bitmap[], dirty pages will be tracked while the
 	 * memory slot is write protected.
 	 */
-	if (change != KVM_MR_DELETE && new->flags & KVM_MEM_LOG_DIRTY_PAGES) {
+	if (log_dirty_pages) {
+
+		if (change == KVM_MR_DELETE)
+			return;
+
 		/*
 		 * If we're with initial-all-set, we don't need to write
 		 * protect any pages because they're all reported as dirty.
 		 * Huge pages and normal pages will be write protect gradually.
 		 */
-		if (!kvm_dirty_log_manual_protect_and_init_set(kvm)) {
-			kvm_mmu_wp_memory_region(kvm, new->id);
-		}
+		if (kvm_dirty_log_manual_protect_and_init_set(kvm))
+			return;
+
+		kvm_mmu_wp_memory_region(kvm, new->id);
 	}
 }
 
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
#14 lore raw

PATCH v6 6/12 KVM: arm64: Add kvm_uninit_stage2_mmu()

In reply to Ricardo Koller

Add kvm_uninit_stage2_mmu() and move kvm_free_stage2_pgd() into it. A future commit will add some more things to do inside of kvm_uninit_stage2_mmu().

No functional change intended.

---
 arch/arm64/include/asm/kvm_mmu.h | 1 +
 arch/arm64/kvm/mmu.c             | 7 ++++++-
 2 files changed, 7 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_mmu.h +1 -0
index 083cc47dca08..7d173da5bd51 100644
@@ -168,6 +168,7 @@ void __init free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
 int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type);
+void kvm_uninit_stage2_mmu(struct kvm *kvm);
 void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu);
 int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
 			  phys_addr_t pa, unsigned long size, bool writable);
arch/arm64/kvm/mmu.c +6 -1
index 37d7d2aa472a..a2800e5c4271 100644
@@ -767,6 +767,11 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 	return err;
 }
 
+void kvm_uninit_stage2_mmu(struct kvm *kvm)
+{
+	kvm_free_stage2_pgd(&kvm->arch.mmu);
+}
+
 static void stage2_unmap_memslot(struct kvm *kvm,
 				 struct kvm_memory_slot *memslot)
 {
@@ -1855,7 +1860,7 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen)
 
 void kvm_arch_flush_shadow_all(struct kvm *kvm)
 {
-	kvm_free_stage2_pgd(&kvm->arch.mmu);
+	kvm_uninit_stage2_mmu(kvm);
 }
 
 void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
#15 lore raw

PATCH v6 7/12 KVM: arm64: Export kvm_are_all_memslots_empty()

In reply to Ricardo Koller

Export kvm_are_all_memslots_empty(). This will be used by a future commit when checking before setting a capability.

No functional change intended.

---
 include/linux/kvm_host.h | 2 ++
 virt/kvm/kvm_main.c      | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +2 -0
index 8ada23756b0e..c6fa634f236d 100644
@@ -990,6 +990,8 @@ static inline bool kvm_memslots_empty(struct kvm_memslots *slots)
 	return RB_EMPTY_ROOT(&slots->gfn_tree);
 }
 
+bool kvm_are_all_memslots_empty(struct kvm *kvm);
+
 #define kvm_for_each_memslot(memslot, bkt, slots)			      \
 	hash_for_each(slots->id_hash, bkt, memslot, id_node[slots->node_idx]) \
 		if (WARN_ON_ONCE(!memslot->npages)) {			      \
virt/kvm/kvm_main.c +1 -1
index d255964ec331..897b000787be 100644
@@ -4596,7 +4596,7 @@ int __attribute__((weak)) kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 	return -EINVAL;
 }
 
-static bool kvm_are_all_memslots_empty(struct kvm *kvm)
+bool kvm_are_all_memslots_empty(struct kvm *kvm)
 {
 	int i;
 
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
maintainer
#16 lore raw

Re: [PATCH v6 07/12] KVM: arm64: Export kvm_are_all_memslots_empty()

Quoted 3 lines · Ricardo Koller

Export kvm_are_all_memslots_empty(). This will be used by a future commit when checking before setting a capability.

You're not exporting this function, you are making it globally visible, which is a very different things. Please keep "export" description for symbols that are actually exported to modules.

Quoted 2 lines

No functional change intended.

I wish people stopped adding this pointless sentence to commit messages. All changes have a functional change one way or another, unless you are only changing a comment. Yes, I've been guilty of writing that too, and I wish I didn't.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#17 lore raw

Re: [PATCH v6 07/12] KVM: arm64: Export kvm_are_all_memslots_empty()

Quoted 7 lines · Marc Zyngier
Quoted 1 line · Ricardo Koller

No functional change intended.

I wish people stopped adding this pointless sentence to commit messages. All changes have a functional change one way or another, unless you are only changing a comment.

The implied context is that there is no change in runtime functionality, which does hold true for many changes. I personally find the annotation helpful, both for code review and when doing git archaeology. If a changelog states that the author doesn't/didn't intend a functional change, then _any_ change in (runtime) functionality becomes a red flag, and for me, prompts a closer look regardless of whether or not I have other concerns with the patch/commit.

maintainer
#18 lore raw

Re: [PATCH v6 07/12] KVM: arm64: Export kvm_are_all_memslots_empty()

Quoted 10 lines · Sean Christopherson
Quoted 7 lines · Marc Zyngier
Quoted 1 line · Ricardo Koller

No functional change intended.

I wish people stopped adding this pointless sentence to commit messages. All changes have a functional change one way or another, unless you are only changing a comment.

Quoted 6 lines

The implied context is that there is no change in runtime functionality, which does hold true for many changes. I personally find the annotation helpful, both for code review and when doing git archaeology. If a changelog states that the author doesn't/didn't intend a functional change, then _any_ change in (runtime) functionality becomes a red flag, and for me, prompts a closer look regardless of whether or not I have other concerns with the patch/commit.

And I think it lures the reviewer into a false sense of security. No intended change, must be fine. Except when it is not. More often than not, we end-up with seemingly innocent changes that break things.

It is even worse when things get (for good or bad reasons) backported to -stable or an internal tree of some description. "No functional change" can become something very different in another context. How do you communicate this?

Maybe I'll add a standard disclaimer to my own patches ("Here be dragons!").

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#19 lore raw

Re: [PATCH v6 07/12] KVM: arm64: Export kvm_are_all_memslots_empty()

Quoted 19 lines · Marc Zyngier
Quoted 10 lines · Sean Christopherson
Quoted 7 lines · Marc Zyngier
Quoted 1 line · Ricardo Koller

No functional change intended.

I wish people stopped adding this pointless sentence to commit messages. All changes have a functional change one way or another, unless you are only changing a comment.

Quoted 6 lines

The implied context is that there is no change in runtime functionality, which does hold true for many changes. I personally find the annotation helpful, both for code review and when doing git archaeology. If a changelog states that the author doesn't/didn't intend a functional change, then _any_ change in (runtime) functionality becomes a red flag, and for me, prompts a closer look regardless of whether or not I have other concerns with the patch/commit.

Quoted 8 lines

And I think it lures the reviewer into a false sense of security. No intended change, must be fine. Except when it is not. More often than not, we end-up with seemingly innocent changes that break things.

It is even worse when things get (for good or bad reasons) backported to -stable or an internal tree of some description. "No functional change" can become something very different in another context. How do you communicate this?

For KVM x86, we opt out of AUTOSEL, so barring errors elsewhere in the process, a maintainer needs to review such patches at some point. And again, for me, sending a patch to stable that was intended to be a nop is a flag that the backport warrants a closer look, e.g. extra justification for why a patch that's (allegedly) a nop needs to be backported to stable kernels.

I agree it's imperfect, e.g could lead downstream maintainers astray if they view the disclaimer as a statement of truth as opposed to be a statement of intent. But IMO the good things that come from being able to know the author's intent far outweigh the probability of bad things happening because a reviewer and/or downstream consumer put too much weight on the statement.

My opinion is certainly influenced by having spent far too much time digging through historical KVM x86 commits, where it's all too often unclear if a buggy/flawed commit was simply a coding goof, versus the commit doing exactly what the author intended and being broken because of bad assumptions, incorrect interpretation of a spec, etc. But even with that bias in mind, I still think explicitly declaring an author's intent for these types of changes is overall a net positive.

Anywho, I don't mean to step on toes and force my preferences down everyone's throats, just wanted to provide my reasoning for including the disclaimer and encouraging other KVM x86 contributors to do the same.

#20 lore raw

PATCH v6 8/12 KVM: arm64: Add KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE

In reply to Ricardo Koller

Add a capability for userspace to specify the eager split chunk size. The chunk size specifies how many pages to break at a time, using a single allocation. Bigger the chunk size, more pages need to be allocated ahead of time.

---
 Documentation/virt/kvm/api.rst    | 26 ++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h | 19 +++++++++++++++++++
 arch/arm64/kvm/arm.c              | 22 ++++++++++++++++++++++
 arch/arm64/kvm/mmu.c              |  3 +++
 include/uapi/linux/kvm.h          |  1 +
 5 files changed, 71 insertions(+)
Documentation/virt/kvm/api.rst +26 -0
index 62de0768d6aa..872dae7cfbe0 100644
@@ -8380,6 +8380,32 @@ structure.
 When getting the Modified Change Topology Report value, the attr->addr
 must point to a byte where the value will be stored or retrieved from.
 
+8.40 KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+---------------------------------------
+
+:Capability: KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+:Architectures: arm64
+:Type: vm
+:Parameters: arg[0] is the new chunk size.
+:Returns: 0 on success, -EINVAL if any memslot has been created.
+
+This capability sets the chunk size used in Eager Page Splitting.
+
+Eager Page Splitting improves the performance of dirty-logging (used
+in live migrations) when guest memory is backed by huge-pages.  This
+optimization is enabled by default on arm64. It avoids splitting
+huge-pages (into PAGE_SIZE pages) on fault, by doing it eagerly when
+enabling dirty logging (with the KVM_MEM_LOG_DIRTY_PAGES flag for a
+memory region), or when using KVM_CLEAR_DIRTY_LOG.
+
+The chunk size specifies how many pages to break at a time, using a
+single allocation for each chunk. Bigger the chunk size, more pages
+need to be allocated ahead of time. A good heuristic is to pick the
+size of the huge-pages as the chunk size.
+
+If the chunk size (arg[0]) is zero, then no eager page splitting is
+performed. The default value PMD size (e.g., 2M when PAGE_SIZE is 4K).
+
 9. Known KVM API problems
 =========================
 
arch/arm64/include/asm/kvm_host.h +19 -0
index a1892a8f6032..b7755d0cbd4d 100644
@@ -158,6 +158,25 @@ struct kvm_s2_mmu {
 	/* The last vcpu id that ran on each physical CPU */
 	int __percpu *last_vcpu_ran;
 
+#define KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT	PMD_SIZE
+	/*
+	 * Memory cache used to split
+	 * KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE worth of huge pages. It
+	 * is used to allocate stage2 page tables while splitting huge
+	 * pages. Note that the choice of EAGER_PAGE_SPLIT_CHUNK_SIZE
+	 * influences both the capacity of the split page cache, and
+	 * how often KVM reschedules. Be wary of raising CHUNK_SIZE
+	 * too high.
+	 *
+	 * A good heuristic to pick CHUNK_SIZE is that it should be
+	 * the size of the huge-pages backing guest memory. If not
+	 * known, the PMD size (usually 2M) is a good guess.
+	 *
+	 * Protected by kvm->slots_lock.
+	 */
+	struct kvm_mmu_memory_cache split_page_cache;
+	uint64_t split_page_chunk_size;
+
 	struct kvm_arch *arch;
 };
 
arch/arm64/kvm/arm.c +22 -0
index 3bd732eaf087..3468fee223ae 100644
@@ -91,6 +91,22 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		r = 0;
 		set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags);
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		mutex_lock(&kvm->lock);
+		mutex_lock(&kvm->slots_lock);
+		/*
+		 * To keep things simple, allow changing the chunk
+		 * size only if there are no memslots created.
+		 */
+		if (!kvm_are_all_memslots_empty(kvm)) {
+			r = -EINVAL;
+		} else {
+			r = 0;
+			kvm->arch.mmu.split_page_chunk_size = cap->args[0];
+		}
+		mutex_unlock(&kvm->slots_lock);
+		mutex_unlock(&kvm->lock);
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -288,6 +304,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PTRAUTH_GENERIC:
 		r = system_has_full_ptr_auth();
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		if (kvm)
+			r = kvm->arch.mmu.split_page_chunk_size;
+		else
+			r = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+		break;
 	default:
 		r = 0;
 	}
arch/arm64/kvm/mmu.c +3 -0
index a2800e5c4271..898985b09321 100644
@@ -756,6 +756,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 	for_each_possible_cpu(cpu)
 		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
 
+	mmu->split_page_cache.gfp_zero = __GFP_ZERO;
+	mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+
 	mmu->pgt = pgt;
 	mmu->pgd_phys = __pa(pgt->pgd);
 	return 0;
include/uapi/linux/kvm.h +1 -0
index d77aef872a0a..af43acdc7901 100644
@@ -1184,6 +1184,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225
 #define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226
+#define KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE 227
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
maintainer
#21 lore raw

Re: [PATCH v6 08/12] KVM: arm64: Add KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE

Quoted 21 lines · Ricardo Koller

Add a capability for userspace to specify the eager split chunk size. The chunk size specifies how many pages to break at a time, using a single allocation. Bigger the chunk size, more pages need to be allocated ahead of time.

---
 Documentation/virt/kvm/api.rst    | 26 ++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h | 19 +++++++++++++++++++
 arch/arm64/kvm/arm.c              | 22 ++++++++++++++++++++++
 arch/arm64/kvm/mmu.c              |  3 +++
 include/uapi/linux/kvm.h          |  1 +
 5 files changed, 71 insertions(+)
Documentation/virt/kvm/api.rst +0 -0
index 62de0768d6aa..872dae7cfbe0 100644

Documentation/virt/kvm/api.rst structure. · line 8389

Quoted 10 lines
Documentation/virt/kvm/api.rst +7 -0
@@ -8380,6 +8380,32 @@ structure.
 When getting the Modified Change Topology Report value, the attr->addr
 must point to a byte where the value will be stored or retrieved from.
 
+8.40 KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+---------------------------------------
+
+:Capability: KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+:Architectures: arm64
+:Type: vm
+:Parameters: arg[0] is the new chunk size.

split chunk size?

Documentation/virt/kvm/api.rst structure. · line 8390

Quoted 1 line
Documentation/virt/kvm/api.rst +1 -0
+:Returns: 0 on success, -EINVAL if any memslot has been created.

nit: if any memslot was *already* created.

Documentation/virt/kvm/api.rst structure.

Quoted 6 lines
Documentation/virt/kvm/api.rst +6 -0
+
+This capability sets the chunk size used in Eager Page Splitting.
+
+Eager Page Splitting improves the performance of dirty-logging (used
+in live migrations) when guest memory is backed by huge-pages.  This
+optimization is enabled by default on arm64.

Why enabled by default? It means that systems that do not want to pay the extra memory for this have to do an explicit call to disable it. I'd rather see this as a buy-in option.

Documentation/virt/kvm/api.rst structure. · line 8404

Quoted 9 lines

It avoids splitting

Documentation/virt/kvm/api.rst +8 -0
+huge-pages (into PAGE_SIZE pages) on fault, by doing it eagerly when
+enabling dirty logging (with the KVM_MEM_LOG_DIRTY_PAGES flag for a
+memory region), or when using KVM_CLEAR_DIRTY_LOG.
+
+The chunk size specifies how many pages to break at a time, using a
+single allocation for each chunk. Bigger the chunk size, more pages
+need to be allocated ahead of time. A good heuristic is to pick the
+size of the huge-pages as the chunk size.

How about making this a requirement rather than a heuristic? You could also tell userspace what are the block sizes that are acceptable (1G, 512M, 2M, 64K...) by exposing a 64bit bitmap (each bit describing a block size).

Documentation/virt/kvm/api.rst structure. · line 8407

Quoted 3 lines
Documentation/virt/kvm/api.rst +3 -0
+
+If the chunk size (arg[0]) is zero, then no eager page splitting is
+performed. The default value PMD size (e.g., 2M when PAGE_SIZE is 4K).

I really dislike exposing the notion of PMD to userspace. Not only this is a concept that is mostly foreign to the arm64 architecture, this isn't a userspace concept at all. Another reason to talk about block sizes (but I really want this to default to 0 and keep the current behaviour as the default).

Quoted 15 lines
Documentation/virt/kvm/api.rst +1 -0
+
 9. Known KVM API problems
 =========================
arch/arm64/include/asm/kvm_host.h +3 -0
index a1892a8f6032..b7755d0cbd4d 100644
@@ -158,6 +158,25 @@ struct kvm_s2_mmu {
 	/* The last vcpu id that ran on each physical CPU */
 	int __percpu *last_vcpu_ran;
 
+#define KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT	PMD_SIZE
+	/*
+	 * Memory cache used to split

arch/arm64/include/asm/kvm_host.h struct kvm_s2_mmu { · line 173

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +10 -0
+	 * KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE worth of huge pages. It
+	 * is used to allocate stage2 page tables while splitting huge
+	 * pages. Note that the choice of EAGER_PAGE_SPLIT_CHUNK_SIZE
+	 * influences both the capacity of the split page cache, and
+	 * how often KVM reschedules. Be wary of raising CHUNK_SIZE
+	 * too high.
+	 *
+	 * A good heuristic to pick CHUNK_SIZE is that it should be
+	 * the size of the huge-pages backing guest memory. If not
+	 * known, the PMD size (usually 2M) is a good guess.

This is a 4kB-ness. Nothing "usual" about it (and my 16kB hosts definitely object!).

arch/arm64/include/asm/kvm_host.h struct kvm_s2_mmu { · line 177

Quoted 4 lines
arch/arm64/include/asm/kvm_host.h +4 -0
+	 *
+	 * Protected by kvm->slots_lock.
+	 */
+	struct kvm_mmu_memory_cache split_page_cache;

If this is living in kvm_s2_mmu, and that this is a proper memcache, why does patch #4 have this horrible 'struct stage2_split_data' that uses a 'void *' and carries a kvm_s2_mmu pointer?

Surely, passing the memcache around should be enough (and container_of is your forever friend).

Quoted 64 lines
arch/arm64/include/asm/kvm_host.h +2 -0
+	uint64_t split_page_chunk_size;
+
 	struct kvm_arch *arch;
 };
 
arch/arm64/kvm/arm.c +22 -0
index 3bd732eaf087..3468fee223ae 100644
@@ -91,6 +91,22 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		r = 0;
 		set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags);
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		mutex_lock(&kvm->lock);
+		mutex_lock(&kvm->slots_lock);
+		/*
+		 * To keep things simple, allow changing the chunk
+		 * size only if there are no memslots created.
+		 */
+		if (!kvm_are_all_memslots_empty(kvm)) {
+			r = -EINVAL;
+		} else {
+			r = 0;
+			kvm->arch.mmu.split_page_chunk_size = cap->args[0];
+		}
+		mutex_unlock(&kvm->slots_lock);
+		mutex_unlock(&kvm->lock);
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -288,6 +304,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PTRAUTH_GENERIC:
 		r = system_has_full_ptr_auth();
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		if (kvm)
+			r = kvm->arch.mmu.split_page_chunk_size;
+		else
+			r = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+		break;
 	default:
 		r = 0;
 	}
arch/arm64/kvm/mmu.c +3 -0
index a2800e5c4271..898985b09321 100644
@@ -756,6 +756,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 	for_each_possible_cpu(cpu)
 		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
 
+	mmu->split_page_cache.gfp_zero = __GFP_ZERO;
+	mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+
 	mmu->pgt = pgt;
 	mmu->pgd_phys = __pa(pgt->pgd);
 	return 0;
include/uapi/linux/kvm.h +0 -0
index d77aef872a0a..af43acdc7901 100644

include/uapi/linux/kvm.h struct kvm_ppc_resize_hpt { · line 1190

Quoted 7 lines
include/uapi/linux/kvm.h +1 -0
@@ -1184,6 +1184,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225
 #define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226
+#define KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE 227
 
 #ifdef KVM_CAP_IRQ_ROUTING
 

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#22 lore raw

Re: [PATCH v6 08/12] KVM: arm64: Add KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE

Quoted 40 lines · Marc Zyngier
Quoted 21 lines · Ricardo Koller

Add a capability for userspace to specify the eager split chunk size. The chunk size specifies how many pages to break at a time, using a single allocation. Bigger the chunk size, more pages need to be allocated ahead of time.

---
 Documentation/virt/kvm/api.rst    | 26 ++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h | 19 +++++++++++++++++++
 arch/arm64/kvm/arm.c              | 22 ++++++++++++++++++++++
 arch/arm64/kvm/mmu.c              |  3 +++
 include/uapi/linux/kvm.h          |  1 +
 5 files changed, 71 insertions(+)
Documentation/virt/kvm/api.rst +0 -0
index 62de0768d6aa..872dae7cfbe0 100644

Documentation/virt/kvm/api.rst structure. · line 8389

Quoted 10 lines
Documentation/virt/kvm/api.rst +7 -0
@@ -8380,6 +8380,32 @@ structure.
 When getting the Modified Change Topology Report value, the attr->addr
 must point to a byte where the value will be stored or retrieved from.
 
+8.40 KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+---------------------------------------
+
+:Capability: KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+:Architectures: arm64
+:Type: vm
+:Parameters: arg[0] is the new chunk size.

split chunk size?

Documentation/virt/kvm/api.rst structure. · line 8390

Quoted 1 line
Documentation/virt/kvm/api.rst +1 -0
+:Returns: 0 on success, -EINVAL if any memslot has been created.

nit: if any memslot was *already* created.

Quoted 11 lines

Documentation/virt/kvm/api.rst structure.

Quoted 6 lines
Documentation/virt/kvm/api.rst +6 -0
+
+This capability sets the chunk size used in Eager Page Splitting.
+
+Eager Page Splitting improves the performance of dirty-logging (used
+in live migrations) when guest memory is backed by huge-pages.  This
+optimization is enabled by default on arm64.

Why enabled by default? It means that systems that do not want to pay the extra memory for this have to do an explicit call to disable it. I'd rather see this as a buy-in option.

Will disable by default.

Quoted 11 lines

Documentation/virt/kvm/api.rst structure. · line 8404

Quoted 9 lines

It avoids splitting

Documentation/virt/kvm/api.rst +8 -0
+huge-pages (into PAGE_SIZE pages) on fault, by doing it eagerly when
+enabling dirty logging (with the KVM_MEM_LOG_DIRTY_PAGES flag for a
+memory region), or when using KVM_CLEAR_DIRTY_LOG.
+
+The chunk size specifies how many pages to break at a time, using a
+single allocation for each chunk. Bigger the chunk size, more pages
+need to be allocated ahead of time. A good heuristic is to pick the
+size of the huge-pages as the chunk size.

How about making this a requirement rather than a heuristic?

Sounds good. Planning to return EINVAL for anything that's not a supported block size.

Quoted 4 lines

You could
also tell userspace what are the block sizes that are acceptable (1G, 512M, 2M, 64K...) by exposing a 64bit bitmap (each bit describing a block size).

Good idea, I'm thinking of using a new KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES capability to return this bitmap.

Quoted 49 lines

Documentation/virt/kvm/api.rst structure. · line 8407

Quoted 4 lines
Documentation/virt/kvm/api.rst +3 -0
+
+If the chunk size (arg[0]) is zero, then no eager page splitting is
+performed. The default value PMD size (e.g., 2M when PAGE_SIZE is 4K).

I really dislike exposing the notion of PMD to userspace. Not only this is a concept that is mostly foreign to the arm64 architecture, this isn't a userspace concept at all. Another reason to talk about block sizes (but I really want this to default to 0 and keep the current behaviour as the default).

Quoted 15 lines
Documentation/virt/kvm/api.rst +1 -0
+
 9. Known KVM API problems
 =========================
arch/arm64/include/asm/kvm_host.h +3 -0
index a1892a8f6032..b7755d0cbd4d 100644
@@ -158,6 +158,25 @@ struct kvm_s2_mmu {
 	/* The last vcpu id that ran on each physical CPU */
 	int __percpu *last_vcpu_ran;
 
+#define KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT	PMD_SIZE
+	/*
+	 * Memory cache used to split

arch/arm64/include/asm/kvm_host.h struct kvm_s2_mmu { · line 173

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +10 -0
+	 * KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE worth of huge pages. It
+	 * is used to allocate stage2 page tables while splitting huge
+	 * pages. Note that the choice of EAGER_PAGE_SPLIT_CHUNK_SIZE
+	 * influences both the capacity of the split page cache, and
+	 * how often KVM reschedules. Be wary of raising CHUNK_SIZE
+	 * too high.
+	 *
+	 * A good heuristic to pick CHUNK_SIZE is that it should be
+	 * the size of the huge-pages backing guest memory. If not
+	 * known, the PMD size (usually 2M) is a good guess.

This is a 4kB-ness. Nothing "usual" about it (and my 16kB hosts definitely object!).

arch/arm64/include/asm/kvm_host.h struct kvm_s2_mmu { · line 177

Quoted 4 lines
arch/arm64/include/asm/kvm_host.h +4 -0
+	 *
+	 * Protected by kvm->slots_lock.
+	 */
+	struct kvm_mmu_memory_cache split_page_cache;

If this is living in kvm_s2_mmu, and that this is a proper memcache, why does patch #4 have this horrible 'struct stage2_split_data' that uses a 'void *' and carries a kvm_s2_mmu pointer?

Quoted 3 lines

Surely, passing the memcache around should be enough (and container_of is your forever friend).

This simplified things quite a bit.

Quoted 69 lines
Quoted 69 lines
arch/arm64/include/asm/kvm_host.h +2 -0
+	uint64_t split_page_chunk_size;
+
 	struct kvm_arch *arch;
 };
 
arch/arm64/kvm/arm.c +22 -0
index 3bd732eaf087..3468fee223ae 100644
@@ -91,6 +91,22 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		r = 0;
 		set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags);
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		mutex_lock(&kvm->lock);
+		mutex_lock(&kvm->slots_lock);
+		/*
+		 * To keep things simple, allow changing the chunk
+		 * size only if there are no memslots created.
+		 */
+		if (!kvm_are_all_memslots_empty(kvm)) {
+			r = -EINVAL;
+		} else {
+			r = 0;
+			kvm->arch.mmu.split_page_chunk_size = cap->args[0];
+		}
+		mutex_unlock(&kvm->slots_lock);
+		mutex_unlock(&kvm->lock);
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -288,6 +304,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PTRAUTH_GENERIC:
 		r = system_has_full_ptr_auth();
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		if (kvm)
+			r = kvm->arch.mmu.split_page_chunk_size;
+		else
+			r = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+		break;
 	default:
 		r = 0;
 	}
arch/arm64/kvm/mmu.c +3 -0
index a2800e5c4271..898985b09321 100644
@@ -756,6 +756,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 	for_each_possible_cpu(cpu)
 		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
 
+	mmu->split_page_cache.gfp_zero = __GFP_ZERO;
+	mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+
 	mmu->pgt = pgt;
 	mmu->pgd_phys = __pa(pgt->pgd);
 	return 0;
include/uapi/linux/kvm.h +1 -0
index d77aef872a0a..af43acdc7901 100644
@@ -1184,6 +1184,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225
 #define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226
+#define KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE 227
 
Quoted 9 lines

include/uapi/linux/kvm.h struct kvm_ppc_resize_hpt { · line 1190

Quoted 2 lines
include/uapi/linux/kvm.h +0 -0
 #ifdef KVM_CAP_IRQ_ROUTING
 

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.

Thanks,
Ricardo

#23 lore raw

Re: [PATCH v6 08/12] KVM: arm64: Add KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE

In reply to Ricardo Koller

Hi Ricardo,

Quoted 57 lines · Ricardo Koller

Add a capability for userspace to specify the eager split chunk size. The chunk size specifies how many pages to break at a time, using a single allocation. Bigger the chunk size, more pages need to be allocated ahead of time.

---
 Documentation/virt/kvm/api.rst    | 26 ++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h | 19 +++++++++++++++++++
 arch/arm64/kvm/arm.c              | 22 ++++++++++++++++++++++
 arch/arm64/kvm/mmu.c              |  3 +++
 include/uapi/linux/kvm.h          |  1 +
 5 files changed, 71 insertions(+)
Documentation/virt/kvm/api.rst +26 -0
index 62de0768d6aa..872dae7cfbe0 100644
@@ -8380,6 +8380,32 @@ structure.
 When getting the Modified Change Topology Report value, the attr->addr
 must point to a byte where the value will be stored or retrieved from.
 
+8.40 KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+---------------------------------------
+
+:Capability: KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+:Architectures: arm64
+:Type: vm
+:Parameters: arg[0] is the new chunk size.
+:Returns: 0 on success, -EINVAL if any memslot has been created.
+
+This capability sets the chunk size used in Eager Page Splitting.
+
+Eager Page Splitting improves the performance of dirty-logging (used
+in live migrations) when guest memory is backed by huge-pages.  This
+optimization is enabled by default on arm64. It avoids splitting
+huge-pages (into PAGE_SIZE pages) on fault, by doing it eagerly when
+enabling dirty logging (with the KVM_MEM_LOG_DIRTY_PAGES flag for a
+memory region), or when using KVM_CLEAR_DIRTY_LOG.
+
+The chunk size specifies how many pages to break at a time, using a
+single allocation for each chunk. Bigger the chunk size, more pages
+need to be allocated ahead of time. A good heuristic is to pick the
+size of the huge-pages as the chunk size.
+
+If the chunk size (arg[0]) is zero, then no eager page splitting is
+performed. The default value PMD size (e.g., 2M when PAGE_SIZE is 4K).
+
 9. Known KVM API problems
 =========================
 
arch/arm64/include/asm/kvm_host.h +0 -0
index a1892a8f6032..b7755d0cbd4d 100644

arch/arm64/include/asm/kvm_host.h struct kvm_s2_mmu { · line 166

Quoted 9 lines
arch/arm64/include/asm/kvm_host.h +6 -0
@@ -158,6 +158,25 @@ struct kvm_s2_mmu {
 	/* The last vcpu id that ran on each physical CPU */
 	int __percpu *last_vcpu_ran;
 
+#define KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT	PMD_SIZE
+	/*
+	 * Memory cache used to split
+	 * KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE worth of huge pages. It
+	 * is used to allocate stage2 page tables while splitting huge
+	 * pages. Note that the choice of EAGER_PAGE_SPLIT_CHUNK_SIZE

Nit: s/EAGER_PAGE_SPLIT_CHUNK_SIZE/EAGER_SPLIT_CHUNK_SIZE/ ? (or 'split_page_chunk_size' to make the comment consistent with the field name?)

Quoted 21 lines
arch/arm64/include/asm/kvm_host.h +13 -0
+	 * influences both the capacity of the split page cache, and
+	 * how often KVM reschedules. Be wary of raising CHUNK_SIZE
+	 * too high.
+	 *
+	 * A good heuristic to pick CHUNK_SIZE is that it should be
+	 * the size of the huge-pages backing guest memory. If not
+	 * known, the PMD size (usually 2M) is a good guess.
+	 *
+	 * Protected by kvm->slots_lock.
+	 */
+	struct kvm_mmu_memory_cache split_page_cache;
+	uint64_t split_page_chunk_size;
+
 	struct kvm_arch *arch;
 };
 
arch/arm64/kvm/arm.c +0 -0
index 3bd732eaf087..3468fee223ae 100644

arch/arm64/kvm/arm.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 95

Quoted 5 lines
arch/arm64/kvm/arm.c +2 -0
@@ -91,6 +91,22 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		r = 0;
 		set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags);
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		mutex_lock(&kvm->lock);

Do we need to hold kvm->lock here ?

Thanks,
Reiji

Quoted 59 lines
arch/arm64/kvm/arm.c +20 -0
+		mutex_lock(&kvm->slots_lock);
+		/*
+		 * To keep things simple, allow changing the chunk
+		 * size only if there are no memslots created.
+		 */
+		if (!kvm_are_all_memslots_empty(kvm)) {
+			r = -EINVAL;
+		} else {
+			r = 0;
+			kvm->arch.mmu.split_page_chunk_size = cap->args[0];
+		}
+		mutex_unlock(&kvm->slots_lock);
+		mutex_unlock(&kvm->lock);
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -288,6 +304,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PTRAUTH_GENERIC:
 		r = system_has_full_ptr_auth();
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		if (kvm)
+			r = kvm->arch.mmu.split_page_chunk_size;
+		else
+			r = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+		break;
 	default:
 		r = 0;
 	}
arch/arm64/kvm/mmu.c +3 -0
index a2800e5c4271..898985b09321 100644
@@ -756,6 +756,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 	for_each_possible_cpu(cpu)
 		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
 
+	mmu->split_page_cache.gfp_zero = __GFP_ZERO;
+	mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+
 	mmu->pgt = pgt;
 	mmu->pgd_phys = __pa(pgt->pgd);
 	return 0;
include/uapi/linux/kvm.h +1 -0
index d77aef872a0a..af43acdc7901 100644
@@ -1184,6 +1184,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225
 #define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226
+#define KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE 227
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
#24 lore raw

Re: [PATCH v6 08/12] KVM: arm64: Add KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE

Quoted 63 lines · Reiji Watanabe

Hi Ricardo,

Quoted 60 lines · Ricardo Koller

Add a capability for userspace to specify the eager split chunk size. The chunk size specifies how many pages to break at a time, using a single allocation. Bigger the chunk size, more pages need to be allocated ahead of time.

---
 Documentation/virt/kvm/api.rst    | 26 ++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h | 19 +++++++++++++++++++
 arch/arm64/kvm/arm.c              | 22 ++++++++++++++++++++++
 arch/arm64/kvm/mmu.c              |  3 +++
 include/uapi/linux/kvm.h          |  1 +
 5 files changed, 71 insertions(+)
Documentation/virt/kvm/api.rst +26 -0
index 62de0768d6aa..872dae7cfbe0 100644
@@ -8380,6 +8380,32 @@ structure.
 When getting the Modified Change Topology Report value, the attr->addr
 must point to a byte where the value will be stored or retrieved from.
 
+8.40 KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+---------------------------------------
+
+:Capability: KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
+:Architectures: arm64
+:Type: vm
+:Parameters: arg[0] is the new chunk size.
+:Returns: 0 on success, -EINVAL if any memslot has been created.
+
+This capability sets the chunk size used in Eager Page Splitting.
+
+Eager Page Splitting improves the performance of dirty-logging (used
+in live migrations) when guest memory is backed by huge-pages.  This
+optimization is enabled by default on arm64. It avoids splitting
+huge-pages (into PAGE_SIZE pages) on fault, by doing it eagerly when
+enabling dirty logging (with the KVM_MEM_LOG_DIRTY_PAGES flag for a
+memory region), or when using KVM_CLEAR_DIRTY_LOG.
+
+The chunk size specifies how many pages to break at a time, using a
+single allocation for each chunk. Bigger the chunk size, more pages
+need to be allocated ahead of time. A good heuristic is to pick the
+size of the huge-pages as the chunk size.
+
+If the chunk size (arg[0]) is zero, then no eager page splitting is
+performed. The default value PMD size (e.g., 2M when PAGE_SIZE is 4K).
+
 9. Known KVM API problems
 =========================
 
arch/arm64/include/asm/kvm_host.h +0 -0
index a1892a8f6032..b7755d0cbd4d 100644
@@ -158,6 +158,25 @@ struct kvm_s2_mmu {
 	/* The last vcpu id that ran on each physical CPU */
 	int __percpu *last_vcpu_ran;
 
Quoted 12 lines

arch/arm64/include/asm/kvm_host.h struct kvm_s2_mmu { · line 166

Quoted 6 lines
arch/arm64/include/asm/kvm_host.h +6 -0
+#define KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT	PMD_SIZE
+	/*
+	 * Memory cache used to split
+	 * KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE worth of huge pages. It
+	 * is used to allocate stage2 page tables while splitting huge
+	 * pages. Note that the choice of EAGER_PAGE_SPLIT_CHUNK_SIZE

Nit: s/EAGER_PAGE_SPLIT_CHUNK_SIZE/EAGER_SPLIT_CHUNK_SIZE/ ? (or 'split_page_chunk_size' to make the comment consistent with the field name?)

I missed this on v7! sorry Reiji. Will fix this on the next version.

Quoted 16 lines
Quoted 16 lines
arch/arm64/include/asm/kvm_host.h +13 -0
+	 * influences both the capacity of the split page cache, and
+	 * how often KVM reschedules. Be wary of raising CHUNK_SIZE
+	 * too high.
+	 *
+	 * A good heuristic to pick CHUNK_SIZE is that it should be
+	 * the size of the huge-pages backing guest memory. If not
+	 * known, the PMD size (usually 2M) is a good guess.
+	 *
+	 * Protected by kvm->slots_lock.
+	 */
+	struct kvm_mmu_memory_cache split_page_cache;
+	uint64_t split_page_chunk_size;
+
 	struct kvm_arch *arch;
 };
 
Quoted 12 lines

arch/arm64/kvm/arm.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 95

Quoted 10 lines
arch/arm64/kvm/arm.c +2 -0
index 3bd732eaf087..3468fee223ae 100644
@@ -91,6 +91,22 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		r = 0;
 		set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags);
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		mutex_lock(&kvm->lock);

Do we need to hold kvm->lock here ?

We don't need it. It's safe to have it here, but it's not required.

The ordering of locks is:

	kvm->lock --> kvm->slots_lock

and kvm->lock is not held at this point, so it's safe to grab it. I found instances where kvm->lock is held before grabbing kvm->slots_lock, and others where it's not. So it's not required.

I will remove it for the next version.

Thanks,
Ricardo

Quoted 65 lines

Thanks,
Reiji

Quoted 60 lines
arch/arm64/kvm/arm.c +14 -0
+		mutex_lock(&kvm->slots_lock);
+		/*
+		 * To keep things simple, allow changing the chunk
+		 * size only if there are no memslots created.
+		 */
+		if (!kvm_are_all_memslots_empty(kvm)) {
+			r = -EINVAL;
 
+		} else {
+			r = 0;
+			kvm->arch.mmu.split_page_chunk_size = cap->args[0];
+		}
+		mutex_unlock(&kvm->slots_lock);
+		mutex_unlock(&kvm->lock);
+		break;
 	default:
 		r = -EINVAL;
arch/arm64/kvm/arm.c +6 -0
 		break;
@@ -288,6 +304,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PTRAUTH_GENERIC:
 		r = system_has_full_ptr_auth();
 		break;
+	case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
+		if (kvm)
+			r = kvm->arch.mmu.split_page_chunk_size;
+		else
+			r = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+		break;
 	default:
 		r = 0;
 	}
arch/arm64/kvm/mmu.c +3 -0
index a2800e5c4271..898985b09321 100644
@@ -756,6 +756,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 	for_each_possible_cpu(cpu)
 		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
 
+	mmu->split_page_cache.gfp_zero = __GFP_ZERO;
+	mmu->split_page_chunk_size = KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT;
+
 	mmu->pgt = pgt;
 	mmu->pgd_phys = __pa(pgt->pgd);
 	return 0;
include/uapi/linux/kvm.h +1 -0
index d77aef872a0a..af43acdc7901 100644
@@ -1184,6 +1184,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225
 #define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226
+#define KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE 227
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
#25 lore raw

PATCH v6 9/12 KVM: arm64: Split huge pages when dirty logging is enabled

In reply to Ricardo Koller

Split huge pages eagerly when enabling dirty logging. The goal is to avoid doing it while faulting on write-protected pages, which negatively impacts guest performance.

A memslot marked for dirty logging is split in 1GB pieces at a time. This is in order to release the mmu_lock and give other kernel threads the opportunity to run, and also in order to allocate enough pages to split a 1GB range worth of huge pages (or a single 1GB huge page). Note that these page allocations can fail, so eager page splitting is best-effort. This is not a correctness issue though, as huge pages can still be split on write-faults.

The benefits of eager page splitting are the same as in x86, added with commit a3fe5dbda0a4 ("KVM: x86/mmu: Split huge pages mapped by the TDP MMU when dirty logging is enabled"). For example, when running dirty_log_perf_test with 64 virtual CPUs (Ampere Altra), 1GB per vCPU, 50% reads, and 2MB HugeTLB memory, the time it takes vCPUs to access all of their memory after dirty logging is enabled decreased by 44% from 2.58s to 1.42s.

---
 arch/arm64/kvm/mmu.c | 118 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 116 insertions(+), 2 deletions(-)
arch/arm64/kvm/mmu.c +116 -2
index 898985b09321..b1b8da5f8b6c 100644
@@ -31,14 +31,21 @@ static phys_addr_t __ro_after_init hyp_idmap_vector;
 
 static unsigned long __ro_after_init io_map_base;
 
-static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
+static phys_addr_t __stage2_range_addr_end(phys_addr_t addr, phys_addr_t end,
+					   phys_addr_t size)
 {
-	phys_addr_t size = kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL);
 	phys_addr_t boundary = ALIGN_DOWN(addr + size, size);
 
 	return (boundary - 1 < end - 1) ? boundary : end;
 }
 
+static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
+{
+	phys_addr_t size = kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL);
+
+	return __stage2_range_addr_end(addr, end, size);
+}
+
 /*
  * Release kvm_mmu_lock periodically if the memory region is large. Otherwise,
  * we may see kernel panics with CONFIG_DETECT_HUNG_TASK,
@@ -75,6 +82,77 @@ static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr,
 #define stage2_apply_range_resched(mmu, addr, end, fn)			\
 	stage2_apply_range(mmu, addr, end, fn, true)
 
+static bool need_topup_split_page_cache_or_resched(struct kvm *kvm, uint64_t min)
+{
+	struct kvm_mmu_memory_cache *cache;
+
+	if (need_resched() || rwlock_needbreak(&kvm->mmu_lock))
+		return true;
+
+	cache = &kvm->arch.mmu.split_page_cache;
+	return kvm_mmu_memory_cache_nr_free_objects(cache) < min;
+}
+
+/*
+ * Get the maximum number of page-tables needed to split a range of
+ * blocks into PAGE_SIZE PTEs. It assumes the range is already mapped
+ * at the PMD level, or at the PUD level if allowed.
+ */
+static int kvm_mmu_split_nr_page_tables(u64 range)
+{
+	int n = 0;
+
+	if (KVM_PGTABLE_MIN_BLOCK_LEVEL < 2)
+		n += DIV_ROUND_UP_ULL(range, PUD_SIZE);
+	n += DIV_ROUND_UP_ULL(range, PMD_SIZE);
+	return n;
+}
+
+static int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr,
+				    phys_addr_t end)
+{
+	struct kvm_mmu_memory_cache *cache;
+	struct kvm_pgtable *pgt;
+	int ret;
+	u64 next;
+	u64 chunk_size = kvm->arch.mmu.split_page_chunk_size;
+	int cache_capacity = kvm_mmu_split_nr_page_tables(chunk_size);
+
+	if (chunk_size == 0)
+		return 0;
+
+	lockdep_assert_held_write(&kvm->mmu_lock);
+
+	cache = &kvm->arch.mmu.split_page_cache;
+
+	do {
+		if (need_topup_split_page_cache_or_resched(kvm,
+							   cache_capacity)) {
+			write_unlock(&kvm->mmu_lock);
+			cond_resched();
+			/* Eager page splitting is best-effort. */
+			ret = __kvm_mmu_topup_memory_cache(cache,
+							   cache_capacity,
+							   cache_capacity);
+			write_lock(&kvm->mmu_lock);
+			if (ret)
+				break;
+		}
+
+		pgt = kvm->arch.mmu.pgt;
+		if (!pgt)
+			return -EINVAL;
+
+		next = __stage2_range_addr_end(addr, end, chunk_size);
+		ret = kvm_pgtable_stage2_split(pgt, addr, next - addr,
+					       cache, cache_capacity);
+		if (ret)
+			break;
+	} while (addr = next, addr != end);
+
+	return ret;
+}
+
 static bool memslot_is_logging(struct kvm_memory_slot *memslot)
 {
 	return memslot->dirty_bitmap && !(memslot->flags & KVM_MEM_READONLY);
@@ -773,6 +851,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 void kvm_uninit_stage2_mmu(struct kvm *kvm)
 {
 	kvm_free_stage2_pgd(&kvm->arch.mmu);
+	kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 }
 
 static void stage2_unmap_memslot(struct kvm *kvm,
@@ -999,6 +1078,31 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
 	stage2_wp_range(&kvm->arch.mmu, start, end);
 }
 
+/**
+ * kvm_mmu_split_memory_region() - split the stage 2 blocks into PAGE_SIZE
+ *				   pages for memory slot
+ * @kvm:	The KVM pointer
+ * @slot:	The memory slot to split
+ *
+ * Acquires kvm->mmu_lock. Called with kvm->slots_lock mutex acquired,
+ * serializing operations for VM memory regions.
+ */
+static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
+{
+	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memory_slot *memslot = id_to_memslot(slots, slot);
+	phys_addr_t start, end;
+
+	lockdep_assert_held(&kvm->slots_lock);
+
+	start = memslot->base_gfn << PAGE_SHIFT;
+	end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
+
+	write_lock(&kvm->mmu_lock);
+	kvm_mmu_split_huge_pages(kvm, start, end);
+	write_unlock(&kvm->mmu_lock);
+}
+
 /*
  * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected
  * dirty pages.
@@ -1790,6 +1894,16 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 			return;
 
 		kvm_mmu_wp_memory_region(kvm, new->id);
+		kvm_mmu_split_memory_region(kvm, new->id);
+	} else {
+		/*
+		 * Free any leftovers from the eager page splitting cache. Do
+		 * this when deleting, moving, disabling dirty logging, or
+		 * creating the memslot (a nop). Doing it for deletes makes
+		 * sure we don't leak memory, and there's no need to keep the
+		 * cache around for any of the other cases.
+		 */
+		kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 	}
 }
 
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
maintainer
#26 lore raw

Re: [PATCH v6 09/12] KVM: arm64: Split huge pages when dirty logging is enabled

Quoted 56 lines · Ricardo Koller

Split huge pages eagerly when enabling dirty logging. The goal is to avoid doing it while faulting on write-protected pages, which negatively impacts guest performance.

A memslot marked for dirty logging is split in 1GB pieces at a time. This is in order to release the mmu_lock and give other kernel threads the opportunity to run, and also in order to allocate enough pages to split a 1GB range worth of huge pages (or a single 1GB huge page). Note that these page allocations can fail, so eager page splitting is best-effort. This is not a correctness issue though, as huge pages can still be split on write-faults.

The benefits of eager page splitting are the same as in x86, added with commit a3fe5dbda0a4 ("KVM: x86/mmu: Split huge pages mapped by the TDP MMU when dirty logging is enabled"). For example, when running dirty_log_perf_test with 64 virtual CPUs (Ampere Altra), 1GB per vCPU, 50% reads, and 2MB HugeTLB memory, the time it takes vCPUs to access all of their memory after dirty logging is enabled decreased by 44% from 2.58s to 1.42s.

---
 arch/arm64/kvm/mmu.c | 118 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 116 insertions(+), 2 deletions(-)
arch/arm64/kvm/mmu.c +9 -2
index 898985b09321..b1b8da5f8b6c 100644
@@ -31,14 +31,21 @@ static phys_addr_t __ro_after_init hyp_idmap_vector;
 
 static unsigned long __ro_after_init io_map_base;
 
-static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
+static phys_addr_t __stage2_range_addr_end(phys_addr_t addr, phys_addr_t end,
+					   phys_addr_t size)
 {
-	phys_addr_t size = kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL);
 	phys_addr_t boundary = ALIGN_DOWN(addr + size, size);
 
 	return (boundary - 1 < end - 1) ? boundary : end;
 }
 
+static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
+{
+	phys_addr_t size = kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL);
+
+	return __stage2_range_addr_end(addr, end, size);
+}
+
 /*
  * Release kvm_mmu_lock periodically if the memory region is large. Otherwise,
  * we may see kernel panics with CONFIG_DETECT_HUNG_TASK,

arch/arm64/kvm/mmu.c static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, · line 85

Quoted 4 lines
arch/arm64/kvm/mmu.c +1 -0
@@ -75,6 +82,77 @@ static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr,
 #define stage2_apply_range_resched(mmu, addr, end, fn)			\
 	stage2_apply_range(mmu, addr, end, fn, true)
 
+static bool need_topup_split_page_cache_or_resched(struct kvm *kvm, uint64_t min)

Please don't use the words "page cache", it triggers a painful Pavlovian reflex. Something like "need_split_memcache_topup_or_reched" makes me feel less anxious.

arch/arm64/kvm/mmu.c static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, · line 97

Quoted 12 lines
arch/arm64/kvm/mmu.c +12 -0
+{
+	struct kvm_mmu_memory_cache *cache;
+
+	if (need_resched() || rwlock_needbreak(&kvm->mmu_lock))
+		return true;
+
+	cache = &kvm->arch.mmu.split_page_cache;
+	return kvm_mmu_memory_cache_nr_free_objects(cache) < min;
+}
+
+/*
+ * Get the maximum number of page-tables needed to split a range of

nit: page-table pages.

Quoted 17 lines
arch/arm64/kvm/mmu.c +17 -0
+ * blocks into PAGE_SIZE PTEs. It assumes the range is already mapped
+ * at the PMD level, or at the PUD level if allowed.
+ */
+static int kvm_mmu_split_nr_page_tables(u64 range)
+{
+	int n = 0;
+
+	if (KVM_PGTABLE_MIN_BLOCK_LEVEL < 2)
+		n += DIV_ROUND_UP_ULL(range, PUD_SIZE);
+	n += DIV_ROUND_UP_ULL(range, PMD_SIZE);
+	return n;
+}
+
+static int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr,
+				    phys_addr_t end)
+{
+	struct kvm_mmu_memory_cache *cache;

arch/arm64/kvm/mmu.c static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, · line 124

Quoted 10 lines
arch/arm64/kvm/mmu.c +10 -0
+	struct kvm_pgtable *pgt;
+	int ret;
+	u64 next;
+	u64 chunk_size = kvm->arch.mmu.split_page_chunk_size;
+	int cache_capacity = kvm_mmu_split_nr_page_tables(chunk_size);
+
+	if (chunk_size == 0)
+		return 0;
+
+	lockdep_assert_held_write(&kvm->mmu_lock);

Please check for the lock being held early, even in the 0-sized chunk condition.

arch/arm64/kvm/mmu.c static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, · line 130

Quoted 6 lines
arch/arm64/kvm/mmu.c +6 -0
+
+	cache = &kvm->arch.mmu.split_page_cache;
+
+	do {
+		if (need_topup_split_page_cache_or_resched(kvm,
+							   cache_capacity)) {

Since cache_capacity is stored in the kvm struct, why not just passing it to the helper function and let it deal with it?

Quoted 46 lines
arch/arm64/kvm/mmu.c +32 -0
+			write_unlock(&kvm->mmu_lock);
+			cond_resched();
+			/* Eager page splitting is best-effort. */
+			ret = __kvm_mmu_topup_memory_cache(cache,
+							   cache_capacity,
+							   cache_capacity);
+			write_lock(&kvm->mmu_lock);
+			if (ret)
+				break;
+		}
+
+		pgt = kvm->arch.mmu.pgt;
+		if (!pgt)
+			return -EINVAL;
+
+		next = __stage2_range_addr_end(addr, end, chunk_size);
+		ret = kvm_pgtable_stage2_split(pgt, addr, next - addr,
+					       cache, cache_capacity);
+		if (ret)
+			break;
+	} while (addr = next, addr != end);
+
+	return ret;
+}
+
 static bool memslot_is_logging(struct kvm_memory_slot *memslot)
 {
 	return memslot->dirty_bitmap && !(memslot->flags & KVM_MEM_READONLY);
@@ -773,6 +851,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 void kvm_uninit_stage2_mmu(struct kvm *kvm)
 {
 	kvm_free_stage2_pgd(&kvm->arch.mmu);
+	kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 }
 
 static void stage2_unmap_memslot(struct kvm *kvm,
@@ -999,6 +1078,31 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
 	stage2_wp_range(&kvm->arch.mmu, start, end);
 }
 
+/**
+ * kvm_mmu_split_memory_region() - split the stage 2 blocks into PAGE_SIZE
+ *				   pages for memory slot
+ * @kvm:	The KVM pointer
+ * @slot:	The memory slot to split
+ *

arch/arm64/kvm/mmu.c static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, · line 1096

Quoted 10 lines
arch/arm64/kvm/mmu.c +10 -0
+ * Acquires kvm->mmu_lock. Called with kvm->slots_lock mutex acquired,
+ * serializing operations for VM memory regions.
+ */
+static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
+{
+	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memory_slot *memslot = id_to_memslot(slots, slot);
+	phys_addr_t start, end;
+
+	lockdep_assert_held(&kvm->slots_lock);

You have already accessed the memslots by the time you check for the lock. Not great.

Quoted 13 lines
arch/arm64/kvm/mmu.c +9 -0
+
+	start = memslot->base_gfn << PAGE_SHIFT;
+	end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
+
+	write_lock(&kvm->mmu_lock);
+	kvm_mmu_split_huge_pages(kvm, start, end);
+	write_unlock(&kvm->mmu_lock);
+}
+
 /*
  * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected
  * dirty pages.

arch/arm64/kvm/mmu.c void kvm_arch_commit_memory_region(struct kvm *kvm, · line 1897

Quoted 4 lines
arch/arm64/kvm/mmu.c +1 -0
@@ -1790,6 +1894,16 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 			return;
 
 		kvm_mmu_wp_memory_region(kvm, new->id);
+		kvm_mmu_split_memory_region(kvm, new->id);

Would there be an advantage in merging these two operations somehow?

arch/arm64/kvm/mmu.c void kvm_arch_commit_memory_region(struct kvm *kvm, · line 1909

Quoted 12 lines
arch/arm64/kvm/mmu.c +9 -0
+	} else {
+		/*
+		 * Free any leftovers from the eager page splitting cache. Do
+		 * this when deleting, moving, disabling dirty logging, or
+		 * creating the memslot (a nop). Doing it for deletes makes
+		 * sure we don't leak memory, and there's no need to keep the
+		 * cache around for any of the other cases.
+		 */
+		kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 	}
 }
 

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#27 lore raw

Re: [PATCH v6 09/12] KVM: arm64: Split huge pages when dirty logging is enabled

Quoted 61 lines · Marc Zyngier
Quoted 59 lines · Ricardo Koller

Split huge pages eagerly when enabling dirty logging. The goal is to avoid doing it while faulting on write-protected pages, which negatively impacts guest performance.

A memslot marked for dirty logging is split in 1GB pieces at a time. This is in order to release the mmu_lock and give other kernel threads the opportunity to run, and also in order to allocate enough pages to split a 1GB range worth of huge pages (or a single 1GB huge page). Note that these page allocations can fail, so eager page splitting is best-effort. This is not a correctness issue though, as huge pages can still be split on write-faults.

The benefits of eager page splitting are the same as in x86, added with commit a3fe5dbda0a4 ("KVM: x86/mmu: Split huge pages mapped by the TDP MMU when dirty logging is enabled"). For example, when running dirty_log_perf_test with 64 virtual CPUs (Ampere Altra), 1GB per vCPU, 50% reads, and 2MB HugeTLB memory, the time it takes vCPUs to access all of their memory after dirty logging is enabled decreased by 44% from 2.58s to 1.42s.

---
 arch/arm64/kvm/mmu.c | 118 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 116 insertions(+), 2 deletions(-)
arch/arm64/kvm/mmu.c +9 -2
index 898985b09321..b1b8da5f8b6c 100644
@@ -31,14 +31,21 @@ static phys_addr_t __ro_after_init hyp_idmap_vector;
 
 static unsigned long __ro_after_init io_map_base;
 
-static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
+static phys_addr_t __stage2_range_addr_end(phys_addr_t addr, phys_addr_t end,
+					   phys_addr_t size)
 {
-	phys_addr_t size = kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL);
 	phys_addr_t boundary = ALIGN_DOWN(addr + size, size);
 
 	return (boundary - 1 < end - 1) ? boundary : end;
 }
 
+static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
+{
+	phys_addr_t size = kvm_granule_size(KVM_PGTABLE_MIN_BLOCK_LEVEL);
+
+	return __stage2_range_addr_end(addr, end, size);
+}
+
 /*
  * Release kvm_mmu_lock periodically if the memory region is large. Otherwise,
  * we may see kernel panics with CONFIG_DETECT_HUNG_TASK,
@@ -75,6 +82,77 @@ static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr,
 #define stage2_apply_range_resched(mmu, addr, end, fn)			\
 	stage2_apply_range(mmu, addr, end, fn, true)
 
Quoted 6 lines

arch/arm64/kvm/mmu.c static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, · line 85

Quoted 1 line
arch/arm64/kvm/mmu.c +1 -0
+static bool need_topup_split_page_cache_or_resched(struct kvm *kvm, uint64_t min)

Please don't use the words "page cache", it triggers a painful Pavlovian reflex. Something like "need_split_memcache_topup_or_reched" makes me feel less anxious.

fixed

Quoted 6 lines
Quoted 6 lines
arch/arm64/kvm/mmu.c +6 -0
+{
+	struct kvm_mmu_memory_cache *cache;
+
+	if (need_resched() || rwlock_needbreak(&kvm->mmu_lock))
+		return true;
+
Quoted 9 lines

arch/arm64/kvm/mmu.c static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, · line 97

Quoted 6 lines
arch/arm64/kvm/mmu.c +6 -0
+	cache = &kvm->arch.mmu.split_page_cache;
+	return kvm_mmu_memory_cache_nr_free_objects(cache) < min;
+}
+
+/*
+ * Get the maximum number of page-tables needed to split a range of

nit: page-table pages.

fixed

Quoted 22 lines
Quoted 22 lines
arch/arm64/kvm/mmu.c +22 -0
+ * blocks into PAGE_SIZE PTEs. It assumes the range is already mapped
+ * at the PMD level, or at the PUD level if allowed.
+ */
+static int kvm_mmu_split_nr_page_tables(u64 range)
+{
+	int n = 0;
+
+	if (KVM_PGTABLE_MIN_BLOCK_LEVEL < 2)
+		n += DIV_ROUND_UP_ULL(range, PUD_SIZE);
+	n += DIV_ROUND_UP_ULL(range, PMD_SIZE);
+	return n;
+}
+
+static int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr,
+				    phys_addr_t end)
+{
+	struct kvm_mmu_memory_cache *cache;
+	struct kvm_pgtable *pgt;
+	int ret;
+	u64 next;
+	u64 chunk_size = kvm->arch.mmu.split_page_chunk_size;
+	int cache_capacity = kvm_mmu_split_nr_page_tables(chunk_size);
Quoted 9 lines

arch/arm64/kvm/mmu.c static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, · line 124

Quoted 5 lines
arch/arm64/kvm/mmu.c +5 -0
+
+	if (chunk_size == 0)
+		return 0;
+
+	lockdep_assert_held_write(&kvm->mmu_lock);

Please check for the lock being held early, even in the 0-sized chunk condition.

fixed

Quoted 10 lines

arch/arm64/kvm/mmu.c static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, · line 130

Quoted 6 lines
arch/arm64/kvm/mmu.c +6 -0
+
+	cache = &kvm->arch.mmu.split_page_cache;
+
+	do {
+		if (need_topup_split_page_cache_or_resched(kvm,
+							   cache_capacity)) {

Since cache_capacity is stored in the kvm struct, why not just passing it to the helper function and let it deal with it?

removed the cache_capacity arg.

Quoted 51 lines
Quoted 51 lines
arch/arm64/kvm/mmu.c +37 -0
+			write_unlock(&kvm->mmu_lock);
+			cond_resched();
+			/* Eager page splitting is best-effort. */
+			ret = __kvm_mmu_topup_memory_cache(cache,
+							   cache_capacity,
+							   cache_capacity);
+			write_lock(&kvm->mmu_lock);
+			if (ret)
+				break;
+		}
+
+		pgt = kvm->arch.mmu.pgt;
+		if (!pgt)
+			return -EINVAL;
+
+		next = __stage2_range_addr_end(addr, end, chunk_size);
+		ret = kvm_pgtable_stage2_split(pgt, addr, next - addr,
+					       cache, cache_capacity);
+		if (ret)
+			break;
+	} while (addr = next, addr != end);
+
+	return ret;
+}
+
 static bool memslot_is_logging(struct kvm_memory_slot *memslot)
 {
 	return memslot->dirty_bitmap && !(memslot->flags & KVM_MEM_READONLY);
@@ -773,6 +851,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 void kvm_uninit_stage2_mmu(struct kvm *kvm)
 {
 	kvm_free_stage2_pgd(&kvm->arch.mmu);
+	kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 }
 
 static void stage2_unmap_memslot(struct kvm *kvm,
@@ -999,6 +1078,31 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
 	stage2_wp_range(&kvm->arch.mmu, start, end);
 }
 
+/**
+ * kvm_mmu_split_memory_region() - split the stage 2 blocks into PAGE_SIZE
+ *				   pages for memory slot
+ * @kvm:	The KVM pointer
+ * @slot:	The memory slot to split
+ *
+ * Acquires kvm->mmu_lock. Called with kvm->slots_lock mutex acquired,
+ * serializing operations for VM memory regions.
+ */
+static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
+{
Quoted 9 lines

arch/arm64/kvm/mmu.c static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, · line 1096

Quoted 5 lines
arch/arm64/kvm/mmu.c +5 -0
+	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memory_slot *memslot = id_to_memslot(slots, slot);
+	phys_addr_t start, end;
+
+	lockdep_assert_held(&kvm->slots_lock);

You have already accessed the memslots by the time you check for the lock. Not great.

fixed

Quoted 15 lines
Quoted 15 lines
arch/arm64/kvm/mmu.c +9 -0
+
+	start = memslot->base_gfn << PAGE_SHIFT;
+	end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
+
+	write_lock(&kvm->mmu_lock);
+	kvm_mmu_split_huge_pages(kvm, start, end);
+	write_unlock(&kvm->mmu_lock);
+}
+
 /*
  * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected
  * dirty pages.
@@ -1790,6 +1894,16 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 			return;
 
Quoted 5 lines

arch/arm64/kvm/mmu.c void kvm_arch_commit_memory_region(struct kvm *kvm, · line 1897

Quoted 2 lines
arch/arm64/kvm/mmu.c +1 -0
 		kvm_mmu_wp_memory_region(kvm, new->id);
+		kvm_mmu_split_memory_region(kvm, new->id);

Would there be an advantage in merging these two operations somehow?

I guess we could. The only issue is that it could be useful to write-protect a memslot without splitting huge pages.

Quoted 19 lines

arch/arm64/kvm/mmu.c void kvm_arch_commit_memory_region(struct kvm *kvm, · line 1909

Quoted 12 lines
arch/arm64/kvm/mmu.c +9 -0
+	} else {
+		/*
+		 * Free any leftovers from the eager page splitting cache. Do
+		 * this when deleting, moving, disabling dirty logging, or
+		 * creating the memslot (a nop). Doing it for deletes makes
+		 * sure we don't leak memory, and there's no need to keep the
+		 * cache around for any of the other cases.
+		 */
+		kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 	}
 }
 

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#28 lore raw

PATCH v6 10/12 KVM: arm64: Open-code kvm_mmu_write_protect_pt_masked()

In reply to Ricardo Koller

Move the functionality of kvm_mmu_write_protect_pt_masked() into its caller, kvm_arch_mmu_enable_log_dirty_pt_masked(). This will be used in a subsequent commit in order to share some of the code in kvm_arch_mmu_enable_log_dirty_pt_masked().

No functional change intended.

---
 arch/arm64/kvm/mmu.c | 42 +++++++++++++++---------------------------
 1 file changed, 15 insertions(+), 27 deletions(-)
arch/arm64/kvm/mmu.c +15 -27
index b1b8da5f8b6c..910aea6bbd1e 100644
@@ -1056,28 +1056,6 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot)
 	kvm_flush_remote_tlbs(kvm);
 }
 
-/**
- * kvm_mmu_write_protect_pt_masked() - write protect dirty pages
- * @kvm:	The KVM pointer
- * @slot:	The memory slot associated with mask
- * @gfn_offset:	The gfn offset in memory slot
- * @mask:	The mask of dirty pages at offset 'gfn_offset' in this memory
- *		slot to be write protected
- *
- * Walks bits set in mask write protects the associated pte's. Caller must
- * acquire kvm_mmu_lock.
- */
-static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
-		struct kvm_memory_slot *slot,
-		gfn_t gfn_offset, unsigned long mask)
-{
-	phys_addr_t base_gfn = slot->base_gfn + gfn_offset;
-	phys_addr_t start = (base_gfn +  __ffs(mask)) << PAGE_SHIFT;
-	phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT;
-
-	stage2_wp_range(&kvm->arch.mmu, start, end);
-}
-
 /**
  * kvm_mmu_split_memory_region() - split the stage 2 blocks into PAGE_SIZE
  *				   pages for memory slot
@@ -1104,17 +1082,27 @@ static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
 }
 
 /*
- * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected
- * dirty pages.
+ * kvm_arch_mmu_enable_log_dirty_pt_masked() - enable dirty logging for selected pages.
+ * @kvm:	The KVM pointer
+ * @slot:	The memory slot associated with mask
+ * @gfn_offset:	The gfn offset in memory slot
+ * @mask:	The mask of pages at offset 'gfn_offset' in this memory
+ *		slot to enable dirty logging on
  *
- * It calls kvm_mmu_write_protect_pt_masked to write protect selected pages to
- * enable dirty logging for them.
+ * Writes protect selected pages to enable dirty logging for them. Caller must
+ * acquire kvm->mmu_lock.
  */
 void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 		struct kvm_memory_slot *slot,
 		gfn_t gfn_offset, unsigned long mask)
 {
-	kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask);
+	phys_addr_t base_gfn = slot->base_gfn + gfn_offset;
+	phys_addr_t start = (base_gfn +  __ffs(mask)) << PAGE_SHIFT;
+	phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT;
+
+	lockdep_assert_held_write(&kvm->mmu_lock);
+
+	stage2_wp_range(&kvm->arch.mmu, start, end);
 }
 
 static void kvm_send_hwpoison_signal(unsigned long address, short lsb)
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
#29 lore raw

PATCH v6 11/12 KVM: arm64: Split huge pages during KVM_CLEAR_DIRTY_LOG

In reply to Ricardo Koller

This is the arm64 counterpart of commit cb00a70bd4b7 ("KVM: x86/mmu:
Split huge pages mapped by the TDP MMU during KVM_CLEAR_DIRTY_LOG"), which has the benefit of splitting the cost of splitting a memslot across multiple ioctls.

Split huge pages on the range specified using KVM_CLEAR_DIRTY_LOG. And do not split when enabling dirty logging if KVM_DIRTY_LOG_INITIALLY_SET is set.

---
 arch/arm64/kvm/mmu.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
arch/arm64/kvm/mmu.c +12 -3
index 910aea6bbd1e..d54223b5db97 100644
@@ -1089,8 +1089,8 @@ static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
  * @mask:	The mask of pages at offset 'gfn_offset' in this memory
  *		slot to enable dirty logging on
  *
- * Writes protect selected pages to enable dirty logging for them. Caller must
- * acquire kvm->mmu_lock.
+ * Splits selected pages to PAGE_SIZE and then writes protect them to enable
+ * dirty logging for them. Caller must acquire kvm->mmu_lock.
  */
 void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 		struct kvm_memory_slot *slot,
@@ -1103,6 +1103,13 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
 	stage2_wp_range(&kvm->arch.mmu, start, end);
+
+	/*
+	 * If initially-all-set mode is not set, then huge-pages were already
+	 * split when enabling dirty logging: no need to do it again.
+	 */
+	if (kvm_dirty_log_manual_protect_and_init_set(kvm))
+		kvm_mmu_split_huge_pages(kvm, start, end);
 }
 
 static void kvm_send_hwpoison_signal(unsigned long address, short lsb)
@@ -1889,7 +1896,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 		 * this when deleting, moving, disabling dirty logging, or
 		 * creating the memslot (a nop). Doing it for deletes makes
 		 * sure we don't leak memory, and there's no need to keep the
-		 * cache around for any of the other cases.
+		 * cache around for any of the other cases. Keeping the cache
+		 * is useful for successive KVM_CLEAR_DIRTY_LOG calls, which is
+		 * not handled in this function.
 		 */
 		kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 	}
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
maintainer
#30 lore raw

Re: [PATCH v6 11/12] KVM: arm64: Split huge pages during KVM_CLEAR_DIRTY_LOG

Quoted 20 lines · Ricardo Koller

This is the arm64 counterpart of commit cb00a70bd4b7 ("KVM: x86/mmu:
Split huge pages mapped by the TDP MMU during KVM_CLEAR_DIRTY_LOG"), which has the benefit of splitting the cost of splitting a memslot across multiple ioctls.

Split huge pages on the range specified using KVM_CLEAR_DIRTY_LOG. And do not split when enabling dirty logging if KVM_DIRTY_LOG_INITIALLY_SET is set.

---
 arch/arm64/kvm/mmu.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
arch/arm64/kvm/mmu.c +0 -0
index 910aea6bbd1e..d54223b5db97 100644

arch/arm64/kvm/mmu.c static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot) · line 1093

Quoted 7 lines
arch/arm64/kvm/mmu.c +2 -2
@@ -1089,8 +1089,8 @@ static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
  * @mask:	The mask of pages at offset 'gfn_offset' in this memory
  *		slot to enable dirty logging on
  *
- * Writes protect selected pages to enable dirty logging for them. Caller must
- * acquire kvm->mmu_lock.
+ * Splits selected pages to PAGE_SIZE and then writes protect them to enable
+ * dirty logging for them. Caller must acquire kvm->mmu_lock.

The code does things in the opposite order...

arch/arm64/kvm/mmu.c void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, · line 1111

Quoted 13 lines
arch/arm64/kvm/mmu.c +6 -0
  */
 void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 		struct kvm_memory_slot *slot,
@@ -1103,6 +1103,13 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
 	stage2_wp_range(&kvm->arch.mmu, start, end);
+
+	/*
+	 * If initially-all-set mode is not set, then huge-pages were already
+	 * split when enabling dirty logging: no need to do it again.
+	 */
+	if (kvm_dirty_log_manual_protect_and_init_set(kvm))

This contradicts the comment. Which one is correct?

arch/arm64/kvm/mmu.c void kvm_arch_commit_memory_region(struct kvm *kvm, · line 1901

Quoted 12 lines
arch/arm64/kvm/mmu.c +4 -1
+		kvm_mmu_split_huge_pages(kvm, start, end);
 }
 
 static void kvm_send_hwpoison_signal(unsigned long address, short lsb)
@@ -1889,7 +1896,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 		 * this when deleting, moving, disabling dirty logging, or
 		 * creating the memslot (a nop). Doing it for deletes makes
 		 * sure we don't leak memory, and there's no need to keep the
-		 * cache around for any of the other cases.
+		 * cache around for any of the other cases. Keeping the cache
+		 * is useful for successive KVM_CLEAR_DIRTY_LOG calls, which is
+		 * not handled in this function.

Where is it handled then?

arch/arm64/kvm/mmu.c void kvm_arch_commit_memory_region(struct kvm *kvm, · line 1904

Quoted 3 lines
arch/arm64/kvm/mmu.c +0 -0
 		 */
 		kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 	}

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#31 lore raw

Re: [PATCH v6 11/12] KVM: arm64: Split huge pages during KVM_CLEAR_DIRTY_LOG

Quoted 17 lines · Marc Zyngier
Quoted 15 lines · Ricardo Koller

This is the arm64 counterpart of commit cb00a70bd4b7 ("KVM: x86/mmu:
Split huge pages mapped by the TDP MMU during KVM_CLEAR_DIRTY_LOG"), which has the benefit of splitting the cost of splitting a memslot across multiple ioctls.

Split huge pages on the range specified using KVM_CLEAR_DIRTY_LOG. And do not split when enabling dirty logging if KVM_DIRTY_LOG_INITIALLY_SET is set.

---
 arch/arm64/kvm/mmu.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
Quoted 14 lines

arch/arm64/kvm/mmu.c static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot) · line 1093

Quoted 12 lines
arch/arm64/kvm/mmu.c +2 -2
index 910aea6bbd1e..d54223b5db97 100644
@@ -1089,8 +1089,8 @@ static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
  * @mask:	The mask of pages at offset 'gfn_offset' in this memory
  *		slot to enable dirty logging on
  *
- * Writes protect selected pages to enable dirty logging for them. Caller must
- * acquire kvm->mmu_lock.
+ * Splits selected pages to PAGE_SIZE and then writes protect them to enable
+ * dirty logging for them. Caller must acquire kvm->mmu_lock.

The code does things in the opposite order...

Fixed the comment.

Quoted 7 lines
Quoted 7 lines
arch/arm64/kvm/mmu.c +0 -0
  */
 void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 		struct kvm_memory_slot *slot,
@@ -1103,6 +1103,13 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
Quoted 9 lines

arch/arm64/kvm/mmu.c void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, · line 1111

Quoted 7 lines
arch/arm64/kvm/mmu.c +6 -0
 	stage2_wp_range(&kvm->arch.mmu, start, end);
+
+	/*
+	 * If initially-all-set mode is not set, then huge-pages were already
+	 * split when enabling dirty logging: no need to do it again.
+	 */
+	if (kvm_dirty_log_manual_protect_and_init_set(kvm))

This contradicts the comment. Which one is correct?a

Changed the comment.

Quoted 4 lines
Quoted 4 lines
arch/arm64/kvm/mmu.c +1 -0
+		kvm_mmu_split_huge_pages(kvm, start, end);
 }
 
Quoted 11 lines

arch/arm64/kvm/mmu.c void kvm_arch_commit_memory_region(struct kvm *kvm, · line 1901

Quoted 9 lines
arch/arm64/kvm/mmu.c +3 -1
 static void kvm_send_hwpoison_signal(unsigned long address, short lsb)
@@ -1889,7 +1896,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 		 * this when deleting, moving, disabling dirty logging, or
 		 * creating the memslot (a nop). Doing it for deletes makes
 		 * sure we don't leak memory, and there's no need to keep the
-		 * cache around for any of the other cases.
+		 * cache around for any of the other cases. Keeping the cache
+		 * is useful for successive KVM_CLEAR_DIRTY_LOG calls, which is
+		 * not handled in this function.

Where is it handled then?

This last sentence doesn't make much sense, so I removed it. CLEAR calls don't even go through this function.

Quoted 11 lines

arch/arm64/kvm/mmu.c void kvm_arch_commit_memory_region(struct kvm *kvm, · line 1904

Quoted 4 lines
arch/arm64/kvm/mmu.c +0 -0
 		 */
 		kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 	}

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#32 lore raw

PATCH v6 12/12 KVM: arm64: Use local TLBI on permission relaxation

In reply to Ricardo Koller

From: Marc Zyngier <[email protected]>

Broadcasted TLB invalidations (TLBI) are usually less performant than their local variant. In particular, we observed some implementations that take millliseconds to complete parallel broadcasted TLBIs.

It's safe to use local, non-shareable, TLBIs when relaxing permissions on a PTE in the KVM case for a couple of reasons. First, according to the ARM Arm (DDI 0487H.a D5-4913), permission relaxation does not need break-before-make. Second, the VTTBR_EL2.CnP==0 case, where each PE has its own TLB entry for the same page, is tolerated correctly by KVM when doing permission relaxation. Not having changes broadcasted to all PEs is correct for this case, as it's safe to have other PEs fault on permission on the same page.

---
 arch/arm64/include/asm/kvm_asm.h   |  4 +++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 10 ++++++
 arch/arm64/kvm/hyp/nvhe/tlb.c      | 54 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/pgtable.c       |  2 +-
 arch/arm64/kvm/hyp/vhe/tlb.c       | 32 ++++++++++++++++++
 5 files changed, 101 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_asm.h +4 -0
index 43c3bc0f9544..bb17b2ead4c7 100644
@@ -68,6 +68,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___kvm_vcpu_run,
 	__KVM_HOST_SMCCC_FUNC___kvm_flush_vm_context,
 	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa,
+	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa_nsh,
 	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid,
 	__KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context,
 	__KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff,
@@ -225,6 +226,9 @@ extern void __kvm_flush_vm_context(void);
 extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu);
 extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
 				     int level);
+extern void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
+					 phys_addr_t ipa,
+					 int level);
 extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 
 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
index 728e01d4536b..c6bf1e49ca93 100644
@@ -125,6 +125,15 @@ static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt)
 	__kvm_tlb_flush_vmid_ipa(kern_hyp_va(mmu), ipa, level);
 }
 
+static void handle___kvm_tlb_flush_vmid_ipa_nsh(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
+	DECLARE_REG(phys_addr_t, ipa, host_ctxt, 2);
+	DECLARE_REG(int, level, host_ctxt, 3);
+
+	__kvm_tlb_flush_vmid_ipa_nsh(kern_hyp_va(mmu), ipa, level);
+}
+
 static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
@@ -315,6 +324,7 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__kvm_vcpu_run),
 	HANDLE_FUNC(__kvm_flush_vm_context),
 	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa),
+	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa_nsh),
 	HANDLE_FUNC(__kvm_tlb_flush_vmid),
 	HANDLE_FUNC(__kvm_flush_cpu_context),
 	HANDLE_FUNC(__kvm_timer_set_cntvoff),
arch/arm64/kvm/hyp/nvhe/tlb.c +54 -0

60 lines not on this page · raw message

arch/arm64/kvm/hyp/pgtable.c +1 -1
index 3149b98d1701..dcf7ec1810c7 100644
@@ -1179,7 +1179,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 				       KVM_PGTABLE_WALK_HANDLE_FAULT |
 				       KVM_PGTABLE_WALK_SHARED);
 	if (!ret)
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level);
+		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level);
 	return ret;
 }
 
arch/arm64/kvm/hyp/vhe/tlb.c +32 -0
index 24cef9b87f9e..e69da550cdc5 100644
@@ -111,6 +111,38 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
 	__tlb_switch_to_host(&cxt);
 }
 
+void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
+				  phys_addr_t ipa, int level)
+{
+	struct tlb_inv_context cxt;
+
+	dsb(nshst);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt);
+
+	/*
+	 * We could do so much better if we had the VA as well.
+	 * Instead, we invalidate Stage-2 for this IPA, and the
+	 * whole of Stage-1. Weep...
+	 */
+	ipa >>= 12;
+	__tlbi_level(ipas2e1, ipa, level);
+
+	/*
+	 * We have to ensure completion of the invalidation at Stage-2,
+	 * since a table walk on another CPU could refill a TLB with a
+	 * complete (S1 + S2) walk based on the old Stage-2 mapping if
+	 * the Stage-1 invalidation happened first.
+	 */
+	dsb(nsh);
+	__tlbi(vmalle1);
+	dsb(nsh);
+	isb();
+
+	__tlb_switch_to_host(&cxt);
+}
+
 void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
 {
 	struct tlb_inv_context cxt;
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
maintainer
#33 lore raw

Re: [PATCH v6 12/12] KVM: arm64: Use local TLBI on permission relaxation

Quoted 2 lines · Ricardo Koller

From: Marc Zyngier <[email protected]>

Thanks for writing a commit message for my hacks!

Quoted 2 lines

Broadcasted TLB invalidations (TLBI) are usually less performant than

More precisely, TLBIs targeting the Inner Shareable domain. Also, 's/broadcasted/broadcast/', as this is an adjective and not a verb indicative of the past tense..

Quoted 1 line

their local variant. In particular, we observed some implementations

non-shareable rather than local. 'Local' has all sort of odd implementation specific meanings (local to *what* is the usual question that follows...).

Quoted 3 lines

that take millliseconds to complete parallel broadcasted TLBIs.

It's safe to use local, non-shareable, TLBIs when relaxing permissions

s/local//

Quoted 3 lines

on a PTE in the KVM case for a couple of reasons. First, according to the ARM Arm (DDI 0487H.a D5-4913), permission relaxation does not need break-before-make.

This requires some more details, and references to the latest revision of the ARM ARM (0487I.a). In that particular revision, the relevant information is contained in D8.13.1 "Using break-before-make when updating translation table entries", and more importantly in the rule R_WHZWS, which states that only a change of output address or block size require a BBM.

Quoted 5 lines

Second, the VTTBR_EL2.CnP==0 case, where each PE
has its own TLB entry for the same page, is tolerated correctly by KVM when doing permission relaxation. Not having changes broadcasted to all PEs is correct for this case, as it's safe to have other PEs fault on permission on the same page.

I'm not sure mentioning CnP is relevant here. If CnP==1, the TLBI will nuke the TLB visible by the sibling PE, but not any other. So this is always a partial TLB invalidation, irrespective of CnP.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#34 lore raw

Re: [PATCH v6 12/12] KVM: arm64: Use local TLBI on permission relaxation

Quoted 43 lines · Marc Zyngier
Quoted 2 lines · Ricardo Koller

From: Marc Zyngier <[email protected]>

Thanks for writing a commit message for my hacks!

Quoted 2 lines

Broadcasted TLB invalidations (TLBI) are usually less performant than

More precisely, TLBIs targeting the Inner Shareable domain. Also, 's/broadcasted/broadcast/', as this is an adjective and not a verb indicative of the past tense..

Quoted 1 line

their local variant. In particular, we observed some implementations

non-shareable rather than local. 'Local' has all sort of odd implementation specific meanings (local to *what* is the usual question that follows...).

Quoted 3 lines

that take millliseconds to complete parallel broadcasted TLBIs.

It's safe to use local, non-shareable, TLBIs when relaxing permissions

s/local//

Quoted 3 lines

on a PTE in the KVM case for a couple of reasons. First, according to the ARM Arm (DDI 0487H.a D5-4913), permission relaxation does not need break-before-make.

This requires some more details, and references to the latest revision of the ARM ARM (0487I.a). In that particular revision, the relevant information is contained in D8.13.1 "Using break-before-make when updating translation table entries", and more importantly in the rule R_WHZWS, which states that only a change of output address or block size require a BBM.

Quoted 5 lines

Second, the VTTBR_EL2.CnP==0 case, where each PE
has its own TLB entry for the same page, is tolerated correctly by KVM when doing permission relaxation. Not having changes broadcasted to all PEs is correct for this case, as it's safe to have other PEs fault on permission on the same page.

Quoted 8 lines

I'm not sure mentioning CnP is relevant here. If CnP==1, the TLBI will nuke the TLB visible by the sibling PE, but not any other. So this is always a partial TLB invalidation, irrespective of CnP.

Thanks,

	M.

Thanks Marc. Sent a new version incorporating all the above.

Thanks,
Ricardo

Quoted 2 lines
Signature · 1 line
Without deviation from the norm, progress is not possible.