Skip to contentSkip to conversation

KVM: arm64: Add KVM_PRE_FAULT_MEMORY support

PATCH In review Jack Thomson 5 participants 19 messages Jun 12 – Aug 25

Subsystems (5)
Documentation KVM Kselftest arm64 +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v5 cover KVM: arm64: Add KVM_PRE_FAULT_MEMORY support

From: Jack Thomson <[email protected]>

Hi,

This series adds arm64 support for KVM_PRE_FAULT_MEMORY, which was added for x86 in [1]. The ioctl allows userspace to populate stage-2 mappings before running a vCPU, reducing the number of stage-2 faults taken in the run path. This is useful for post-copy migration, where stage-2 fault latency shows up directly in memory-intensive workloads.

On arm64, the GPA supplied to the ioctl is treated as an IPA in the userspace-owned VM's memslot address space. If the vCPU most recently ran a nested guest, KVM still targets the VM's canonical stage-2. It does not interpret the GPA as an L2 IPA, and does not try to populate the nested/shadow stage-2 selected by the vCPU's last run state.

The patches are:

- Allow callers of kvm_pgtable_get_leaf() to pass walk flags, so the prefault path can walk stage-2 under the MMU read lock.

- Add arm64 support for KVM_PRE_FAULT_MEMORY.

- Enable pre_fault_memory_test on arm64.

- Add a backing-source option to pre_fault_memory_test.

- Add a nested (NV) selftest that prefaults on a vCPU whose last-run context is backed by a shadow stage-2 MMU with an empty nested stage-2 root.

The prefault flag and page_size output in the stage-2 fault descriptor remain in this series so the arm64 implementation can advance by the mapping granule installed by the fault path and report poison without queueing a SIGBUS.

Tested with pre_fault_memory_test under an arm64 QEMU setup with anonymous, shmem, anonymous_thp, anonymous_hugetlb and shared_hugetlb backings, including 64K, 2M and 32M hugetlb pools, and with the new nv_pre_fault_memory_test on an NV-capable setup.

=== Changes since v4 [2] ===

- Reworked nested virt semantics: arm64 now treats the ioctl GPA as the VM/memslot IPA and always targets the canonical stage-2. It no longer translates an L2 IPA through L1's stage-2.

- Documented the arm64 nested behavior in the KVM API text.

- Switch to the canonical stage-2 with the vCPU put/load helpers when the vCPU last ran with a nested/shadow MMU, keeping VMID, VNCR and shadow-MMU refcount state consistent.

- Split the kvm_pgtable_get_leaf() walk-flag plumbing into a prep patch and walk existing mappings with KVM_PGTABLE_WALK_SHARED under the MMU read lock.

- Tightened prefault fault handling: preserve fault info, set IL in the synthetic ESR, handle existing mappings, return -EAGAIN for invalid memslot races, and report -EHWPOISON without queueing SIGBUS.

- Avoid directly walking stage-2 page tables when pKVM is enabled. Protected VMs remain unsupported via -EOPNOTSUPP.

- Preserve the selected selftest memory backing when recreating the racing memslot.

- Add the nested (NV) prefault selftest, including an empty nested stage-2 root to catch accidental L2-IPA interpretation.

=== Changes since v3 [3] ===

- Return -EOPNOTSUPP for protected VMs.

- Reworked nested-vCPU handling to translate an L2 IPA through L1's stage-2. This has been superseded by the canonical VM-IPA semantics described above.

- Make page_size unsigned and keep local declarations ordered at the top of kvm_arch_vcpu_pre_fault_memory().

=== Changes since v2 [4] ===

- Update the synthetic fault info. Thanks Suzuki.

- Remove the selftest change for unaligned mmap allocations. Thanks Sean.

[1]: https://lore.kernel.org/kvm/[email protected]/
[2]: https://lore.kernel.org/linux-arm-kernel/[email protected]/
[3]: https://lore.kernel.org/linux-arm-kernel/[email protected]/
[4]: https://lore.kernel.org/linux-arm-kernel/[email protected]/

Jack Thomson (5):
KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()
KVM: arm64: Add pre_fault_memory implementation
KVM: selftests: Enable pre_fault_memory_test for arm64
KVM: selftests: Add option for different backing in pre-fault tests
KVM: selftests: Add nested pre-fault test for arm64

 Documentation/virt/kvm/api.rst                |  18 +-
 arch/arm64/include/asm/kvm_pgtable.h          |   5 +-
 arch/arm64/kvm/Kconfig                        |   1 +
 arch/arm64/kvm/arm.c                          |   1 +
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         |  10 +-
 arch/arm64/kvm/hyp/pgtable.c                  |   5 +-
 arch/arm64/kvm/mmu.c                          | 164 +++++++++++++-
 arch/arm64/kvm/nested.c                       |   2 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../kvm/arm64/nv_pre_fault_memory_test.c      | 200 ++++++++++++++++++
 .../selftests/kvm/pre_fault_memory_test.c     | 150 ++++++++++---
 11 files changed, 513 insertions(+), 45 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c

base-commit: 98f826f3c500fda08d51fca434b7aefa6a2f7076

Signature · 1 line
2.43.0
#1 lore raw

PATCH v5 1/5 KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()

From: Jack Thomson <[email protected]>

Allow callers of kvm_pgtable_get_leaf() to specify the page-table walk flags, in preparation for performing walks under the MMU read lock.

Reading a stage-2 leaf while only holding the read lock requires
KVM_PGTABLE_WALK_SHARED: parallel faults (which also only hold the read lock) can unlink table pages and free them via RCU, so the walker must be inside an RCU read-side critical section, which the shared walk flag provides via kvm_pgtable_walk_begin().

All existing callers either hold the write lock, walk with interrupts disabled, or run at hyp where shared walks are rejected; they keep the current behaviour by passing no flags.

No functional change intended.

---
 arch/arm64/include/asm/kvm_pgtable.h  |  5 ++++-
 arch/arm64/kvm/hyp/nvhe/mem_protect.c | 10 +++++-----
 arch/arm64/kvm/hyp/pgtable.c          |  5 +++--
 arch/arm64/kvm/mmu.c                  |  2 +-
 arch/arm64/kvm/nested.c               |  2 +-
 5 files changed, 14 insertions(+), 10 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +4 -1
index 41a8687938eb..d0167f7dfbee 100644
@@ -859,6 +859,8 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
  * @addr:	Input address for the start of the walk.
  * @ptep:	Pointer to storage for the retrieved PTE.
  * @level:	Pointer to storage for the level of the retrieved PTE.
+ * @flags:	Flags to control the page-table walk
+ *		(see struct kvm_pgtable_visit_ctx).
  *
  * The offset of @addr within a page is ignored.
  *
@@ -869,7 +871,8 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
  * Return: 0 on success, negative error code on failure.
  */
 int kvm_pgtable_get_leaf(struct kvm_pgtable *pgt, u64 addr,
-			 kvm_pte_t *ptep, s8 *level);
+			 kvm_pte_t *ptep, s8 *level,
+			 enum kvm_pgtable_walk_flags flags);
 
 /**
  * kvm_pgtable_stage2_pte_prot() - Retrieve the protection attributes of a
arch/arm64/kvm/hyp/nvhe/mem_protect.c +5 -5
index 25f04629014e..3b765c9ff7e8 100644
@@ -522,7 +522,7 @@ static int host_stage2_adjust_range(u64 addr, struct kvm_mem_range *range)
 	int ret;
 
 	hyp_assert_lock_held(&host_mmu.lock);
-	ret = kvm_pgtable_get_leaf(&host_mmu.pgt, addr, &pte, &level);
+	ret = kvm_pgtable_get_leaf(&host_mmu.pgt, addr, &pte, &level, 0);
 	if (ret)
 		return ret;
 
@@ -890,7 +890,7 @@ static int get_valid_guest_pte(struct pkvm_hyp_vm *vm, u64 ipa, kvm_pte_t *ptep,
 	s8 level;
 	int ret;
 
-	ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level);
+	ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level, 0);
 	if (ret)
 		return ret;
 	if (guest_pte_is_poisoned(pte))
@@ -939,7 +939,7 @@ int __pkvm_vcpu_in_poison_fault(struct pkvm_hyp_vcpu *hyp_vcpu)
 	ipa |= FAR_TO_FIPA_OFFSET(kvm_vcpu_get_hfar(&hyp_vcpu->vcpu));
 
 	guest_lock_component(vm);
-	ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level);
+	ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level, 0);
 	if (ret)
 		goto unlock;
 
@@ -1293,7 +1293,7 @@ static int host_stage2_get_guest_info(phys_addr_t phys, struct pkvm_hyp_vm **vm,
 		return -EPERM;
 	}
 
-	ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, &level);
+	ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, &level, 0);
 	if (ret)
 		return ret;
 
@@ -1522,7 +1522,7 @@ static int __check_host_shared_guest(struct pkvm_hyp_vm *vm, u64 *__phys, u64 ip
 	s8 level;
 	int ret;
 
-	ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level);
+	ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level, 0);
 	if (ret)
 		return ret;
 	if (!kvm_pte_valid(pte))
arch/arm64/kvm/hyp/pgtable.c +3 -2
index 0c1defa5fb0f..6a839a32e246 100644
@@ -298,12 +298,13 @@ static int leaf_walker(const struct kvm_pgtable_visit_ctx *ctx,
 }
 
 int kvm_pgtable_get_leaf(struct kvm_pgtable *pgt, u64 addr,
-			 kvm_pte_t *ptep, s8 *level)
+			 kvm_pte_t *ptep, s8 *level,
+			 enum kvm_pgtable_walk_flags flags)
 {
 	struct leaf_walk_data data;
 	struct kvm_pgtable_walker walker = {
 		.cb	= leaf_walker,
-		.flags	= KVM_PGTABLE_WALK_LEAF,
+		.flags	= flags | KVM_PGTABLE_WALK_LEAF,
 		.arg	= &data,
 	};
 	int ret;
arch/arm64/kvm/mmu.c +1 -1
index 4da9281312eb..c720f07cb82e 100644
@@ -839,7 +839,7 @@ static int get_user_mapping_size(struct kvm *kvm, u64 addr)
 	 * IPI-ing threads).
 	 */
 	local_irq_save(flags);
-	ret = kvm_pgtable_get_leaf(&pgt, addr, &pte, &level);
+	ret = kvm_pgtable_get_leaf(&pgt, addr, &pte, &level, 0);
 	local_irq_restore(flags);
 
 	if (ret)
arch/arm64/kvm/nested.c +1 -1
index 38f672e94087..e45aed6d9e65 100644
@@ -559,7 +559,7 @@ static u8 get_guest_mapping_ttl(struct kvm_s2_mmu *mmu, u64 addr)
 		return 0;
 
 	tmp &= ~(sz - 1);
-	if (kvm_pgtable_get_leaf(mmu->pgt, tmp, &pte, NULL))
+	if (kvm_pgtable_get_leaf(mmu->pgt, tmp, &pte, NULL, 0))
 		goto again;
 	if (!(pte & PTE_VALID))
 		goto again;
Signature · 1 line
2.43.0
#2 lore raw

PATCH v5 2/5 KVM: arm64: Add pre_fault_memory implementation

In reply to Jack Thomson

From: Jack Thomson <[email protected]>

Add arm64 support for KVM_PRE_FAULT_MEMORY by synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.

If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl; vcpu_put() at ioctl exit drops the hw_mmu and the next vcpu_load() reselects the correct MMU from vCPU state.

Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with -EHWPOISON instead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.

Advertise KVM_CAP_PRE_FAULT_MEMORY on arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns -EOPNOTSUPP if invoked anyway.

---
 Documentation/virt/kvm/api.rst |  18 +++-
 arch/arm64/kvm/Kconfig         |   1 +
 arch/arm64/kvm/arm.c           |   1 +
 arch/arm64/kvm/mmu.c           | 162 +++++++++++++++++++++++++++++++++
 4 files changed, 178 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +14 -4
index 52bbbb553ce1..657e05656fa6 100644
@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 ---------------------------
 
 :Capability: KVM_CAP_PRE_FAULT_MEMORY
-:Architectures: none
+:Architectures: x86, arm64
 :Type: vcpu ioctl
 :Parameters: struct kvm_pre_fault_memory (in/out)
 :Returns: 0 if at least one page is processed, < 0 on error
@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 Errors:
 
   ========== ===============================================================
+  EAGAIN     A memslot update raced with the ioctl before any page was
+             processed.
   EINVAL     The specified `gpa` and `size` were invalid (e.g. not
              page aligned, causes an overflow, or size is zero).
   ENOENT     The specified `gpa` is outside defined memslots.
   EINTR      An unmasked signal is pending and no page was processed.
   EFAULT     The parameter address was invalid.
+  EHWPOISON  A poisoned host page was encountered.
   EOPNOTSUPP Mapping memory for a GPA is unsupported by the
              hypervisor, and/or for the current vCPU state/mode.
   EIO        unexpected error conditions (also causes a WARN)
@@ -6494,7 +6497,14 @@ Errors:
 KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory
 for the current vCPU state.  KVM maps memory as if the vCPU generated a
 stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
-CoW.  However, KVM does not mark any newly created stage-2 PTE as Accessed.
+CoW.  However, on x86, KVM does not mark any newly created stage-2 PTE as
+Accessed.  On arm64, newly created stage-2 PTEs are marked Accessed.
+
+On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's
+memslot address space.  If the vCPU most recently ran a nested guest, KVM
+still targets the VM's canonical stage-2, and does not interpret `gpa` as
+a nested guest IPA or target the nested/shadow stage-2 selected by the
+vCPU's last run state.
 
 In the case of confidential VM types where there is an initial set up of
 private guest memory before the guest is 'finalized'/measured, this ioctl
@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel.
 
 When the ioctl returns, the input values are updated to point to the
 remaining range.  If `size` > 0 on return, the caller can just issue
-the ioctl again with the same `struct kvm_map_memory` argument.
+the ioctl again with the same `struct kvm_pre_fault_memory` argument.
 
-Shadow page tables cannot support this ioctl because they
+On x86, shadow page tables cannot support this ioctl because they
 are indexed by virtual address or nested guest physical address.
 Calling this ioctl when the guest is using shadow page tables (for
 example because it is running a nested guest with nested page tables)
arch/arm64/kvm/Kconfig +1 -0
index 449154f9a485..6b89262e8ba7 100644
@@ -24,6 +24,7 @@ menuconfig KVM
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
+	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select VIRT_XFER_TO_GUEST_WORK
 	select KVM_VFIO
 	select HAVE_KVM_DIRTY_RING_ACQ_REL
arch/arm64/kvm/arm.c +1 -0
index 9453321ef8c6..dcb92bee13af 100644
@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_COUNTER_OFFSET:
 	case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS:
 	case KVM_CAP_ARM_SEA_TO_USER:
+	case KVM_CAP_PRE_FAULT_MEMORY:
 		r = 1;
 		break;
 	case KVM_CAP_SET_GUEST_DEBUG2:
arch/arm64/kvm/mmu.c +162 -0

183 lines not on this page · raw message

Signature · 1 line
2.43.0
#3 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Hi Jack,

I've been trying to review the patch, until I realised that I don't understand what KVM_PRE_FAULT_MEMORY is supposed to do.

Does it guarantee that, after it completes and returns to userspace, the address range [gpa, gpa + size] is mapped in the VM's stage 2 tables? Or does it guarantee that the address range was mapped at stage 2 at some point during the ioctl's execution, but parts, or all of it, might not be mapped after the ioctl completes and returns to userspace?

Thanks,
Alex

Quoted 307 lines · Jack Thomson

From: Jack Thomson <[email protected]>

Add arm64 support for KVM_PRE_FAULT_MEMORY by synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.

If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl; vcpu_put() at ioctl exit drops the hw_mmu and the next vcpu_load() reselects the correct MMU from vCPU state.

Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with -EHWPOISON instead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.

Advertise KVM_CAP_PRE_FAULT_MEMORY on arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns -EOPNOTSUPP if invoked anyway.

---
 Documentation/virt/kvm/api.rst |  18 +++-
 arch/arm64/kvm/Kconfig         |   1 +
 arch/arm64/kvm/arm.c           |   1 +
 arch/arm64/kvm/mmu.c           | 162 +++++++++++++++++++++++++++++++++
 4 files changed, 178 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +14 -4
index 52bbbb553ce1..657e05656fa6 100644
@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 ---------------------------
 
 :Capability: KVM_CAP_PRE_FAULT_MEMORY
-:Architectures: none
+:Architectures: x86, arm64
 :Type: vcpu ioctl
 :Parameters: struct kvm_pre_fault_memory (in/out)
 :Returns: 0 if at least one page is processed, < 0 on error
@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 Errors:
 
   ========== ===============================================================
+  EAGAIN     A memslot update raced with the ioctl before any page was
+             processed.
   EINVAL     The specified `gpa` and `size` were invalid (e.g. not
              page aligned, causes an overflow, or size is zero).
   ENOENT     The specified `gpa` is outside defined memslots.
   EINTR      An unmasked signal is pending and no page was processed.
   EFAULT     The parameter address was invalid.
+  EHWPOISON  A poisoned host page was encountered.
   EOPNOTSUPP Mapping memory for a GPA is unsupported by the
              hypervisor, and/or for the current vCPU state/mode.
   EIO        unexpected error conditions (also causes a WARN)
@@ -6494,7 +6497,14 @@ Errors:
 KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory
 for the current vCPU state.  KVM maps memory as if the vCPU generated a
 stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
-CoW.  However, KVM does not mark any newly created stage-2 PTE as Accessed.
+CoW.  However, on x86, KVM does not mark any newly created stage-2 PTE as
+Accessed.  On arm64, newly created stage-2 PTEs are marked Accessed.
+
+On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's
+memslot address space.  If the vCPU most recently ran a nested guest, KVM
+still targets the VM's canonical stage-2, and does not interpret `gpa` as
+a nested guest IPA or target the nested/shadow stage-2 selected by the
+vCPU's last run state.
 
 In the case of confidential VM types where there is an initial set up of
 private guest memory before the guest is 'finalized'/measured, this ioctl
@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel.
 
 When the ioctl returns, the input values are updated to point to the
 remaining range.  If `size` > 0 on return, the caller can just issue
-the ioctl again with the same `struct kvm_map_memory` argument.
+the ioctl again with the same `struct kvm_pre_fault_memory` argument.
 
-Shadow page tables cannot support this ioctl because they
+On x86, shadow page tables cannot support this ioctl because they
 are indexed by virtual address or nested guest physical address.
 Calling this ioctl when the guest is using shadow page tables (for
 example because it is running a nested guest with nested page tables)
arch/arm64/kvm/Kconfig +1 -0
index 449154f9a485..6b89262e8ba7 100644
@@ -24,6 +24,7 @@ menuconfig KVM
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
+	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select VIRT_XFER_TO_GUEST_WORK
 	select KVM_VFIO
 	select HAVE_KVM_DIRTY_RING_ACQ_REL
arch/arm64/kvm/arm.c +1 -0
index 9453321ef8c6..dcb92bee13af 100644
@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_COUNTER_OFFSET:
 	case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS:
 	case KVM_CAP_ARM_SEA_TO_USER:
+	case KVM_CAP_PRE_FAULT_MEMORY:
 		r = 1;
 		break;
 	case KVM_CAP_SET_GUEST_DEBUG2:
arch/arm64/kvm/mmu.c +162 -0
index c720f07cb82e..4bf048bbcf8b 100644
@@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {
 	struct kvm_s2_trans	*nested;
 	struct kvm_memory_slot	*memslot;
 	unsigned long		hva;
+	unsigned long		*page_size;
+	bool			prefault;
 };
 
 static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
@@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd,
 				      &s2vi->map_writable, &s2vi->page);
 	if (unlikely(is_error_noslot_pfn(s2vi->pfn))) {
 		if (s2vi->pfn == KVM_PFN_ERR_HWPOISON) {
+			/*
+			 * When prefaulting, report the poison via -EHWPOISON
+			 * only; don't also queue a SIGBUS as the run path
+			 * does for the faulting vCPU thread.
+			 */
+			if (s2fd->prefault)
+				return -EHWPOISON;
 			kvm_send_hwpoison_signal(s2fd->hva, __ffs(s2vi->vma_pagesize));
 			return 0;
 		}
@@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable);
 	kvm_fault_unlock(kvm);
 
+	if (s2fd->page_size && !ret)
+		*s2fd->page_size = mapping_size;
+
 	/*
 	 * Mark the page dirty only if the fault is handled successfully,
 	 * making sure we adjust the canonical IPA if the mapping size has
@@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)
 
 	trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled);
 }
+
+/*
+ * Prefaulting always targets the canonical stage-2.  If the vCPU last ran
+ * in a nested context, swap in the canonical MMU via the vCPU put/load
+ * helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount
+ * state stay consistent.
+ *
+ * The swap is deliberately not undone: nothing runs in between the
+ * per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the
+ * generic prefault loop, and the vcpu_put() at ioctl exit discards
+ * vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next
+ * vcpu_load() re-derives the correct MMU from the vCPU's context.  If the
+ * prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu()
+ * keeps the canonical MMU in place for the reload.  Leaving the swap in
+ * place also bounds the cost to at most one put/load pair per ioctl,
+ * rather than two pairs per prefaulted page.
+ */
+static void kvm_pre_fault_load_canonical_mmu(struct kvm_vcpu *vcpu)
+{
+	if (!vcpu_has_nv(vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu)
+		return;
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+}
+
+long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
+				    struct kvm_pre_fault_memory *range)
+{
+	struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault;
+	struct kvm_vcpu_fault_info fault_backup = *fault_info;
+	s8 walk_level = KVM_PGTABLE_LAST_LEVEL;
+	unsigned long page_size = PAGE_SIZE;
+	struct kvm_memory_slot *memslot;
+	phys_addr_t gpa = range->gpa;
+	struct kvm_pgtable *pgt;
+	phys_addr_t end;
+	kvm_pte_t pte;
+	hva_t hva;
+	gfn_t gfn;
+	long ret;
+
+	if (vcpu_is_protected(vcpu))
+		return -EOPNOTSUPP;
+
+	/*
+	 * Interpret range->gpa in the userspace-owned VM's IPA space, not in
+	 * any nested guest IPA space that may have been active on the vCPU's
+	 * last run.  Always target the canonical stage-2.
+	 */
+	kvm_pre_fault_load_canonical_mmu(vcpu);
+
+	if (gpa >= kvm_phys_size(vcpu->arch.hw_mmu)) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	gfn = gpa_to_gfn(gpa);
+	memslot = gfn_to_memslot(vcpu->kvm, gfn);
+	if (!memslot) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	/*
+	 * A racing memslot deletion or move installs an invalid slot before
+	 * zapping stage-2.  Ask userspace to retry once the update settles.
+	 */
+	if (memslot->flags & KVM_MEMSLOT_INVALID) {
+		ret = -EAGAIN;
+		goto out;
+	}
+
+	/*
+	 * pKVM stage-2 mappings aren't directly walkable from the host; let
+	 * the fault path handle both new and existing mappings.
+	 */
+	if (!is_protected_kvm_enabled()) {
+		pgt = vcpu->arch.hw_mmu->pgt;
+		scoped_guard(read_lock, &vcpu->kvm->mmu_lock) {
+			ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, &walk_level,
+						   KVM_PGTABLE_WALK_SHARED);
+		}
+		if (ret)
+			goto out;
+
+		if (kvm_pte_valid(pte)) {
+			page_size = kvm_granule_size(walk_level);
+			if (!(pte & KVM_PTE_LEAF_ATTR_LO_S2_AF))
+				handle_access_fault(vcpu, gpa);
+			goto out_success;
+		}
+	}
+
+	/*
+	 * Synthesize a read translation fault for the canonical IPA, at the
+	 * level where the stage-2 walk currently ends (the last level under
+	 * pKVM, where stage-2 isn't walkable from the host).
+	 */
+	fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) |
+		ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(walk_level);
+	fault_info->hpfar_el2 = HPFAR_EL2_NS |
+		FIELD_PREP(HPFAR_EL2_FIPA, gpa >> 12);
+
+	struct kvm_s2_fault_desc s2fd = {
+		.vcpu		= vcpu,
+		.fault_ipa	= gpa,
+		.nested		= NULL,
+		.memslot	= memslot,
+		.page_size	= &page_size,
+		.prefault	= true,
+	};
+
+	/*
+	 * As in the run path, -EAGAIN from the abort handlers is treated as
+	 * progress: either a parallel fault installed the mapping, or a racing
+	 * invalidation is in flight and the next access will refault.
+	 */
+	if (kvm_slot_has_gmem(memslot)) {
+		ret = gmem_abort(&s2fd);
+	} else {
+		hva = gfn_to_hva_memslot_prot(memslot, gfn, NULL);
+		if (kvm_is_error_hva(hva)) {
+			ret = -EFAULT;
+			goto out;
+		}
+
+		s2fd.hva = hva;
+		ret = user_mem_abort(&s2fd);
+	}
+
+	if (ret < 0)
+		goto out;
+
+out_success:
+	end = ALIGN_DOWN(gpa, page_size) + page_size;
+	ret = min_t(u64, range->size, end - gpa);
+out:
+	/*
+	 * Restore the synthetic fault state so a subsequent KVM_RUN does not
+	 * observe it. kvm_handle_mmio_return() runs before guest entry can
+	 * refresh fault.esr_el2 from hardware, so leaving the synthetic ESR
+	 * in place would corrupt the completion of a pending MMIO exit.
+	 */
+	*fault_info = fault_backup;
+	return ret;
+}
Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Hi,

Quoted 10 lines · Alexandru Elisei

Hi Jack,

I've been trying to review the patch, until I realised that I don't understand what KVM_PRE_FAULT_MEMORY is supposed to do.

Does it guarantee that, after it completes and returns to userspace, the address range [gpa, gpa + size] is mapped in the VM's stage 2 tables? Or does it guarantee that the address range was mapped at stage 2 at some point during the ioctl's execution, but parts, or all of it, might not be mapped after the ioctl completes and returns to userspace?

Nevermind, spend some time thinking about it and it turns out it's impossible for KVM to guarantee that the memory is still mapped after the ioctls returns. So I answered my own question, thanks.

Alex

Quoted 313 lines

Thanks,
Alex

Quoted 307 lines · Jack Thomson

From: Jack Thomson <[email protected]>

Add arm64 support for KVM_PRE_FAULT_MEMORY by synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.

If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl; vcpu_put() at ioctl exit drops the hw_mmu and the next vcpu_load() reselects the correct MMU from vCPU state.

Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with -EHWPOISON instead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.

Advertise KVM_CAP_PRE_FAULT_MEMORY on arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns -EOPNOTSUPP if invoked anyway.

---
 Documentation/virt/kvm/api.rst |  18 +++-
 arch/arm64/kvm/Kconfig         |   1 +
 arch/arm64/kvm/arm.c           |   1 +
 arch/arm64/kvm/mmu.c           | 162 +++++++++++++++++++++++++++++++++
 4 files changed, 178 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +14 -4
index 52bbbb553ce1..657e05656fa6 100644
@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 ---------------------------
 
 :Capability: KVM_CAP_PRE_FAULT_MEMORY
-:Architectures: none
+:Architectures: x86, arm64
 :Type: vcpu ioctl
 :Parameters: struct kvm_pre_fault_memory (in/out)
 :Returns: 0 if at least one page is processed, < 0 on error
@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 Errors:
 
   ========== ===============================================================
+  EAGAIN     A memslot update raced with the ioctl before any page was
+             processed.
   EINVAL     The specified `gpa` and `size` were invalid (e.g. not
              page aligned, causes an overflow, or size is zero).
   ENOENT     The specified `gpa` is outside defined memslots.
   EINTR      An unmasked signal is pending and no page was processed.
   EFAULT     The parameter address was invalid.
+  EHWPOISON  A poisoned host page was encountered.
   EOPNOTSUPP Mapping memory for a GPA is unsupported by the
              hypervisor, and/or for the current vCPU state/mode.
   EIO        unexpected error conditions (also causes a WARN)
@@ -6494,7 +6497,14 @@ Errors:
 KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory
 for the current vCPU state.  KVM maps memory as if the vCPU generated a
 stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
-CoW.  However, KVM does not mark any newly created stage-2 PTE as Accessed.
+CoW.  However, on x86, KVM does not mark any newly created stage-2 PTE as
+Accessed.  On arm64, newly created stage-2 PTEs are marked Accessed.
+
+On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's
+memslot address space.  If the vCPU most recently ran a nested guest, KVM
+still targets the VM's canonical stage-2, and does not interpret `gpa` as
+a nested guest IPA or target the nested/shadow stage-2 selected by the
+vCPU's last run state.
 
 In the case of confidential VM types where there is an initial set up of
 private guest memory before the guest is 'finalized'/measured, this ioctl
@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel.
 
 When the ioctl returns, the input values are updated to point to the
 remaining range.  If `size` > 0 on return, the caller can just issue
-the ioctl again with the same `struct kvm_map_memory` argument.
+the ioctl again with the same `struct kvm_pre_fault_memory` argument.
 
-Shadow page tables cannot support this ioctl because they
+On x86, shadow page tables cannot support this ioctl because they
 are indexed by virtual address or nested guest physical address.
 Calling this ioctl when the guest is using shadow page tables (for
 example because it is running a nested guest with nested page tables)
arch/arm64/kvm/Kconfig +1 -0
index 449154f9a485..6b89262e8ba7 100644
@@ -24,6 +24,7 @@ menuconfig KVM
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
+	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select VIRT_XFER_TO_GUEST_WORK
 	select KVM_VFIO
 	select HAVE_KVM_DIRTY_RING_ACQ_REL
arch/arm64/kvm/arm.c +1 -0
index 9453321ef8c6..dcb92bee13af 100644
@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_COUNTER_OFFSET:
 	case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS:
 	case KVM_CAP_ARM_SEA_TO_USER:
+	case KVM_CAP_PRE_FAULT_MEMORY:
 		r = 1;
 		break;
 	case KVM_CAP_SET_GUEST_DEBUG2:
arch/arm64/kvm/mmu.c +162 -0
index c720f07cb82e..4bf048bbcf8b 100644
@@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {
 	struct kvm_s2_trans	*nested;
 	struct kvm_memory_slot	*memslot;
 	unsigned long		hva;
+	unsigned long		*page_size;
+	bool			prefault;
 };
 
 static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
@@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd,
 				      &s2vi->map_writable, &s2vi->page);
 	if (unlikely(is_error_noslot_pfn(s2vi->pfn))) {
 		if (s2vi->pfn == KVM_PFN_ERR_HWPOISON) {
+			/*
+			 * When prefaulting, report the poison via -EHWPOISON
+			 * only; don't also queue a SIGBUS as the run path
+			 * does for the faulting vCPU thread.
+			 */
+			if (s2fd->prefault)
+				return -EHWPOISON;
 			kvm_send_hwpoison_signal(s2fd->hva, __ffs(s2vi->vma_pagesize));
 			return 0;
 		}
@@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable);
 	kvm_fault_unlock(kvm);
 
+	if (s2fd->page_size && !ret)
+		*s2fd->page_size = mapping_size;
+
 	/*
 	 * Mark the page dirty only if the fault is handled successfully,
 	 * making sure we adjust the canonical IPA if the mapping size has
@@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)
 
 	trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled);
 }
+
+/*
+ * Prefaulting always targets the canonical stage-2.  If the vCPU last ran
+ * in a nested context, swap in the canonical MMU via the vCPU put/load
+ * helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount
+ * state stay consistent.
+ *
+ * The swap is deliberately not undone: nothing runs in between the
+ * per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the
+ * generic prefault loop, and the vcpu_put() at ioctl exit discards
+ * vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next
+ * vcpu_load() re-derives the correct MMU from the vCPU's context.  If the
+ * prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu()
+ * keeps the canonical MMU in place for the reload.  Leaving the swap in
+ * place also bounds the cost to at most one put/load pair per ioctl,
+ * rather than two pairs per prefaulted page.
+ */
+static void kvm_pre_fault_load_canonical_mmu(struct kvm_vcpu *vcpu)
+{
+	if (!vcpu_has_nv(vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu)
+		return;
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+}
+
+long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
+				    struct kvm_pre_fault_memory *range)
+{
+	struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault;
+	struct kvm_vcpu_fault_info fault_backup = *fault_info;
+	s8 walk_level = KVM_PGTABLE_LAST_LEVEL;
+	unsigned long page_size = PAGE_SIZE;
+	struct kvm_memory_slot *memslot;
+	phys_addr_t gpa = range->gpa;
+	struct kvm_pgtable *pgt;
+	phys_addr_t end;
+	kvm_pte_t pte;
+	hva_t hva;
+	gfn_t gfn;
+	long ret;
+
+	if (vcpu_is_protected(vcpu))
+		return -EOPNOTSUPP;
+
+	/*
+	 * Interpret range->gpa in the userspace-owned VM's IPA space, not in
+	 * any nested guest IPA space that may have been active on the vCPU's
+	 * last run.  Always target the canonical stage-2.
+	 */
+	kvm_pre_fault_load_canonical_mmu(vcpu);
+
+	if (gpa >= kvm_phys_size(vcpu->arch.hw_mmu)) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	gfn = gpa_to_gfn(gpa);
+	memslot = gfn_to_memslot(vcpu->kvm, gfn);
+	if (!memslot) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	/*
+	 * A racing memslot deletion or move installs an invalid slot before
+	 * zapping stage-2.  Ask userspace to retry once the update settles.
+	 */
+	if (memslot->flags & KVM_MEMSLOT_INVALID) {
+		ret = -EAGAIN;
+		goto out;
+	}
+
+	/*
+	 * pKVM stage-2 mappings aren't directly walkable from the host; let
+	 * the fault path handle both new and existing mappings.
+	 */
+	if (!is_protected_kvm_enabled()) {
+		pgt = vcpu->arch.hw_mmu->pgt;
+		scoped_guard(read_lock, &vcpu->kvm->mmu_lock) {
+			ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, &walk_level,
+						   KVM_PGTABLE_WALK_SHARED);
+		}
+		if (ret)
+			goto out;
+
+		if (kvm_pte_valid(pte)) {
+			page_size = kvm_granule_size(walk_level);
+			if (!(pte & KVM_PTE_LEAF_ATTR_LO_S2_AF))
+				handle_access_fault(vcpu, gpa);
+			goto out_success;
+		}
+	}
+
+	/*
+	 * Synthesize a read translation fault for the canonical IPA, at the
+	 * level where the stage-2 walk currently ends (the last level under
+	 * pKVM, where stage-2 isn't walkable from the host).
+	 */
+	fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) |
+		ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(walk_level);
+	fault_info->hpfar_el2 = HPFAR_EL2_NS |
+		FIELD_PREP(HPFAR_EL2_FIPA, gpa >> 12);
+
+	struct kvm_s2_fault_desc s2fd = {
+		.vcpu		= vcpu,
+		.fault_ipa	= gpa,
+		.nested		= NULL,
+		.memslot	= memslot,
+		.page_size	= &page_size,
+		.prefault	= true,
+	};
+
+	/*
+	 * As in the run path, -EAGAIN from the abort handlers is treated as
+	 * progress: either a parallel fault installed the mapping, or a racing
+	 * invalidation is in flight and the next access will refault.
+	 */
+	if (kvm_slot_has_gmem(memslot)) {
+		ret = gmem_abort(&s2fd);
+	} else {
+		hva = gfn_to_hva_memslot_prot(memslot, gfn, NULL);
+		if (kvm_is_error_hva(hva)) {
+			ret = -EFAULT;
+			goto out;
+		}
+
+		s2fd.hva = hva;
+		ret = user_mem_abort(&s2fd);
+	}
+
+	if (ret < 0)
+		goto out;
+
+out_success:
+	end = ALIGN_DOWN(gpa, page_size) + page_size;
+	ret = min_t(u64, range->size, end - gpa);
+out:
+	/*
+	 * Restore the synthetic fault state so a subsequent KVM_RUN does not
+	 * observe it. kvm_handle_mmio_return() runs before guest entry can
+	 * refresh fault.esr_el2 from hardware, so leaving the synthetic ESR
+	 * in place would corrupt the completion of a pending MMIO exit.
+	 */
+	*fault_info = fault_backup;
+	return ret;
+}
Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Hi Alex,

Quoted 6 lines · Alexandru Elisei
Quoted 5 lines

Does it guarantee that, after it completes and returns to userspace, the address range [gpa, gpa + size] is mapped in the VM's stage 2 tables? Or does it guarantee that the address range was mapped at stage 2 at some point during the ioctl's execution, but parts, or all of it, might not be mapped after the ioctl completes and returns to userspace?

Quoted 6 lines

Nevermind, spend some time thinking about it and it turns out it's impossible for KVM to guarantee that the memory is still mapped after the ioctls returns. So I answered my own question, thanks.

Alex

Yeah it's best effort, as if a vCPU had faulted the range itself. This could be undone by an invalidation or memslot update. The value is in the common cast (for us, post-copy migration) the guest doesn't pay the stage-2 fault fault cost at run time.

I think it would be worth me adding a sentence to the documentation noting there is no persistance guarentee.

Signature · 2 lines
Thanks,
Jack
#6 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

In reply to Jack Thomson

Hi Jack,

Quoted 117 lines · Jack Thomson

From: Jack Thomson <[email protected]>

Add arm64 support for KVM_PRE_FAULT_MEMORY by synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.

If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl; vcpu_put() at ioctl exit drops the hw_mmu and the next vcpu_load() reselects the correct MMU from vCPU state.

Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with -EHWPOISON instead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.

Advertise KVM_CAP_PRE_FAULT_MEMORY on arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns -EOPNOTSUPP if invoked anyway.

---
 Documentation/virt/kvm/api.rst |  18 +++-
 arch/arm64/kvm/Kconfig         |   1 +
 arch/arm64/kvm/arm.c           |   1 +
 arch/arm64/kvm/mmu.c           | 162 +++++++++++++++++++++++++++++++++
 4 files changed, 178 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +14 -4
index 52bbbb553ce1..657e05656fa6 100644
@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 ---------------------------
 
 :Capability: KVM_CAP_PRE_FAULT_MEMORY
-:Architectures: none
+:Architectures: x86, arm64
 :Type: vcpu ioctl
 :Parameters: struct kvm_pre_fault_memory (in/out)
 :Returns: 0 if at least one page is processed, < 0 on error
@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 Errors:
 
   ========== ===============================================================
+  EAGAIN     A memslot update raced with the ioctl before any page was
+             processed.
   EINVAL     The specified `gpa` and `size` were invalid (e.g. not
              page aligned, causes an overflow, or size is zero).
   ENOENT     The specified `gpa` is outside defined memslots.
   EINTR      An unmasked signal is pending and no page was processed.
   EFAULT     The parameter address was invalid.
+  EHWPOISON  A poisoned host page was encountered.
   EOPNOTSUPP Mapping memory for a GPA is unsupported by the
              hypervisor, and/or for the current vCPU state/mode.
   EIO        unexpected error conditions (also causes a WARN)
@@ -6494,7 +6497,14 @@ Errors:
 KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory
 for the current vCPU state.  KVM maps memory as if the vCPU generated a
 stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
-CoW.  However, KVM does not mark any newly created stage-2 PTE as Accessed.
+CoW.  However, on x86, KVM does not mark any newly created stage-2 PTE as
+Accessed.  On arm64, newly created stage-2 PTEs are marked Accessed.
+
+On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's
+memslot address space.  If the vCPU most recently ran a nested guest, KVM
+still targets the VM's canonical stage-2, and does not interpret `gpa` as
+a nested guest IPA or target the nested/shadow stage-2 selected by the
+vCPU's last run state.
 
 In the case of confidential VM types where there is an initial set up of
 private guest memory before the guest is 'finalized'/measured, this ioctl
@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel.
 
 When the ioctl returns, the input values are updated to point to the
 remaining range.  If `size` > 0 on return, the caller can just issue
-the ioctl again with the same `struct kvm_map_memory` argument.
+the ioctl again with the same `struct kvm_pre_fault_memory` argument.
 
-Shadow page tables cannot support this ioctl because they
+On x86, shadow page tables cannot support this ioctl because they
 are indexed by virtual address or nested guest physical address.
 Calling this ioctl when the guest is using shadow page tables (for
 example because it is running a nested guest with nested page tables)
arch/arm64/kvm/Kconfig +1 -0
index 449154f9a485..6b89262e8ba7 100644
@@ -24,6 +24,7 @@ menuconfig KVM
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
+	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select VIRT_XFER_TO_GUEST_WORK
 	select KVM_VFIO
 	select HAVE_KVM_DIRTY_RING_ACQ_REL
arch/arm64/kvm/arm.c +1 -0
index 9453321ef8c6..dcb92bee13af 100644
@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_COUNTER_OFFSET:
 	case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS:
 	case KVM_CAP_ARM_SEA_TO_USER:
+	case KVM_CAP_PRE_FAULT_MEMORY:
 		r = 1;
 		break;
 	case KVM_CAP_SET_GUEST_DEBUG2:
arch/arm64/kvm/mmu.c +0 -0
index c720f07cb82e..4bf048bbcf8b 100644

arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574

Quoted 4 lines
arch/arm64/kvm/mmu.c +1 -0
@@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {
 	struct kvm_s2_trans	*nested;
 	struct kvm_memory_slot	*memslot;
 	unsigned long		hva;
+	unsigned long		*page_size;

As far as I know, fault handling was reworked to use struct kvm_s2_fault_desc to store the fault information that user_mem_abort() needs to handle the fault. Adding a 'page_size' field, that represents the result of the gpa mapping process, might not be desirable.

Quoted 56 lines
arch/arm64/kvm/mmu.c +35 -0
+	bool			prefault;
 };
 
 static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
@@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd,
 				      &s2vi->map_writable, &s2vi->page);
 	if (unlikely(is_error_noslot_pfn(s2vi->pfn))) {
 		if (s2vi->pfn == KVM_PFN_ERR_HWPOISON) {
+			/*
+			 * When prefaulting, report the poison via -EHWPOISON
+			 * only; don't also queue a SIGBUS as the run path
+			 * does for the faulting vCPU thread.
+			 */
+			if (s2fd->prefault)
+				return -EHWPOISON;
 			kvm_send_hwpoison_signal(s2fd->hva, __ffs(s2vi->vma_pagesize));
 			return 0;
 		}
@@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable);
 	kvm_fault_unlock(kvm);
 
+	if (s2fd->page_size && !ret)
+		*s2fd->page_size = mapping_size;
+
 	/*
 	 * Mark the page dirty only if the fault is handled successfully,
 	 * making sure we adjust the canonical IPA if the mapping size has
@@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)
 
 	trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled);
 }
+
+/*
+ * Prefaulting always targets the canonical stage-2.  If the vCPU last ran
+ * in a nested context, swap in the canonical MMU via the vCPU put/load
+ * helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount
+ * state stay consistent.
+ *
+ * The swap is deliberately not undone: nothing runs in between the
+ * per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the
+ * generic prefault loop, and the vcpu_put() at ioctl exit discards
+ * vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next
+ * vcpu_load() re-derives the correct MMU from the vCPU's context.  If the
+ * prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu()
+ * keeps the canonical MMU in place for the reload.  Leaving the swap in
+ * place also bounds the cost to at most one put/load pair per ioctl,
+ * rather than two pairs per prefaulted page.
+ */
+static void kvm_pre_fault_load_canonical_mmu(struct kvm_vcpu *vcpu)
+{
+	if (!vcpu_has_nv(vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu)
+		return;
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2805

Quoted 10 lines
arch/arm64/kvm/mmu.c +10 -0
+	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+}
+
+long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
+				    struct kvm_pre_fault_memory *range)
+{
+	struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault;
+	struct kvm_vcpu_fault_info fault_backup = *fault_info;

I'm not sure you need to make a backup here. vcpu->arch.fault is populated each time the CPU takes a fault.

Quoted 28 lines
arch/arm64/kvm/mmu.c +28 -0
+	s8 walk_level = KVM_PGTABLE_LAST_LEVEL;
+	unsigned long page_size = PAGE_SIZE;
+	struct kvm_memory_slot *memslot;
+	phys_addr_t gpa = range->gpa;
+	struct kvm_pgtable *pgt;
+	phys_addr_t end;
+	kvm_pte_t pte;
+	hva_t hva;
+	gfn_t gfn;
+	long ret;
+
+	if (vcpu_is_protected(vcpu))
+		return -EOPNOTSUPP;
+
+	/*
+	 * Interpret range->gpa in the userspace-owned VM's IPA space, not in
+	 * any nested guest IPA space that may have been active on the vCPU's
+	 * last run.  Always target the canonical stage-2.
+	 */
+	kvm_pre_fault_load_canonical_mmu(vcpu);
+
+	if (gpa >= kvm_phys_size(vcpu->arch.hw_mmu)) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	gfn = gpa_to_gfn(gpa);
+	memslot = gfn_to_memslot(vcpu->kvm, gfn);

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2843

Quoted 10 lines
arch/arm64/kvm/mmu.c +10 -0
+	if (!memslot) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	/*
+	 * A racing memslot deletion or move installs an invalid slot before
+	 * zapping stage-2.  Ask userspace to retry once the update settles.
+	 */
+	if (memslot->flags & KVM_MEMSLOT_INVALID) {

I don't think that's something we should care about, the flag can be set immediately after the check as the function doesn't take kvm->slots_lock. kvm_vcpu_prefault_memory() takes the srcu lock in read mode, so the function is safe to run even if userspace does something silly like deleting a memslot at the same time that it's prefaulting the guest memory it represents.

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2856

Quoted 13 lines
arch/arm64/kvm/mmu.c +13 -0
+		ret = -EAGAIN;
+		goto out;
+	}
+
+	/*
+	 * pKVM stage-2 mappings aren't directly walkable from the host; let
+	 * the fault path handle both new and existing mappings.
+	 */
+	if (!is_protected_kvm_enabled()) {
+		pgt = vcpu->arch.hw_mmu->pgt;
+		scoped_guard(read_lock, &vcpu->kvm->mmu_lock) {
+			ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, &walk_level,
+						   KVM_PGTABLE_WALK_SHARED);

It might not be obvious, but taking kvm->mmu_lock in read mode does not guarantee that gpa will be mapped when kvm_pgtable_get_leaf() returns. That's because a concurrent kvm_pgtable_stage2_map() for a different gpa can destroy the mapping of the current gpa.

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2857

Quoted 1 line
arch/arm64/kvm/mmu.c +1 -0
+		}

The kvm->mmu_lock is dropped here, which means that it is possible for a MMU notifier callback to have just unmapped the entire stage 2 for the VM.

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2864

Quoted 7 lines
arch/arm64/kvm/mmu.c +7 -0
+		if (ret)
+			goto out;
+
+		if (kvm_pte_valid(pte)) {
+			page_size = kvm_granule_size(walk_level);
+			if (!(pte & KVM_PTE_LEAF_ATTR_LO_S2_AF))
+				handle_access_fault(vcpu, gpa);

handle_access_fault() will fail if the mapping is gone. But I guess that's fine if the ioctl does not guarantee that memory is still mapped after it completes.

Also, the documentation that this patch adds says: 'On arm64, newly created stage-2 PTEs are marked Accessed'. Does not say anything about marking **existing** ptes as accessed. Would be useful to explain the code does it.

Quoted 22 lines
arch/arm64/kvm/mmu.c +22 -0
+			goto out_success;
+		}
+	}
+
+	/*
+	 * Synthesize a read translation fault for the canonical IPA, at the
+	 * level where the stage-2 walk currently ends (the last level under
+	 * pKVM, where stage-2 isn't walkable from the host).
+	 */
+	fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) |
+		ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(walk_level);
+	fault_info->hpfar_el2 = HPFAR_EL2_NS |
+		FIELD_PREP(HPFAR_EL2_FIPA, gpa >> 12);
+
+	struct kvm_s2_fault_desc s2fd = {
+		.vcpu		= vcpu,
+		.fault_ipa	= gpa,
+		.nested		= NULL,
+		.memslot	= memslot,
+		.page_size	= &page_size,
+		.prefault	= true,
+	};

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2896

Quoted 10 lines
arch/arm64/kvm/mmu.c +10 -0
+
+	/*
+	 * As in the run path, -EAGAIN from the abort handlers is treated as
+	 * progress: either a parallel fault installed the mapping, or a racing
+	 * invalidation is in flight and the next access will refault.
+	 */
+	if (kvm_slot_has_gmem(memslot)) {
+		ret = gmem_abort(&s2fd);
+	} else {
+		hva = gfn_to_hva_memslot_prot(memslot, gfn, NULL);

There's gfn_to_hva_memslot(memslot, gfn), is that what you are looking for?

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2903

Quoted 7 lines
arch/arm64/kvm/mmu.c +7 -0
+		if (kvm_is_error_hva(hva)) {
+			ret = -EFAULT;
+			goto out;
+		}
+
+		s2fd.hva = hva;
+		ret = user_mem_abort(&s2fd);

If is_protected_kvm_enabled(), this function doesn't check if the mapping already exists. user_mem_abort() will call **p**kvm_pgtable_stage2_map() (emphasis is mine) to create the mapping, and pkvm_pgtable_stage2_map() will return -EAGAIN, and keep returning -EAGAIN until the mapping is destroyed, which might be some time away.

Same thing happens **without** pKVM, kvm_pgtable_stage2_map() will return -EAGAIN if the mapping exists. But that's not such a big issue, when the user calls KVM_PRE_FAULT_MEMORY on the remaining range kvm_pgtable_get_leaf() will detect the existing mapping. Which you have already taken into consideration judging by the comment, so all good here.

Thanks,
Alex

Quoted 22 lines
arch/arm64/kvm/mmu.c +18 -0
+	}
+
+	if (ret < 0)
+		goto out;
+
+out_success:
+	end = ALIGN_DOWN(gpa, page_size) + page_size;
+	ret = min_t(u64, range->size, end - gpa);
+out:
+	/*
+	 * Restore the synthetic fault state so a subsequent KVM_RUN does not
+	 * observe it. kvm_handle_mmio_return() runs before guest entry can
+	 * refresh fault.esr_el2 from hardware, so leaving the synthetic ESR
+	 * in place would corrupt the completion of a pending MMIO exit.
+	 */
+	*fault_info = fault_backup;
+	return ret;
+}
Signature · 1 line
2.43.0
#7 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Quoted 4 lines · Alexandru Elisei

Hi Jack,

Quoted 1 line · Jack Thomson

From: Jack Thomson <[email protected]>

....

Quoted 14 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2903

Quoted 7 lines
arch/arm64/kvm/mmu.c +7 -0
+		if (kvm_is_error_hva(hva)) {
+			ret = -EFAULT;
+			goto out;
+		}
+
+		s2fd.hva = hva;
+		ret = user_mem_abort(&s2fd);

If is_protected_kvm_enabled(), this function doesn't check if the mapping already exists. user_mem_abort() will call **p**kvm_pgtable_stage2_map() (emphasis is mine) to create the mapping, and pkvm_pgtable_stage2_map() will return -EAGAIN, and keep returning -EAGAIN until the mapping is destroyed, which might be some time away.

Quoted 6 lines

Same thing happens **without** pKVM, kvm_pgtable_stage2_map() will return -EAGAIN if the mapping exists. But that's not such a big issue, when the user calls KVM_PRE_FAULT_MEMORY on the remaining range kvm_pgtable_get_leaf() will detect the existing mapping. Which you have already taken into consideration judging by the comment, so all good here.

But both gmem_abort() and user_mem_abort() handle -EAGAIN internally, don't they?

For gmem_abort(), we have:

1692: return ret != -EAGAIN ? ret : 0;

For user_mem_abort(), the call path is:

user_mem_abort()
-> kvm_s2_fault_map()

    -> KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, ...)

and kvm_s2_fault_map() does:

2111: if (ret != -EAGAIN)

		return ret;
	return 0;

-aneesh

#8 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Hi Aneesh,

Quoted 23 lines · Aneesh Kumar K.V
Quoted 4 lines · Alexandru Elisei

Hi Jack,

Quoted 1 line · Jack Thomson

From: Jack Thomson <[email protected]>

....

Quoted 14 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2903

Quoted 7 lines
arch/arm64/kvm/mmu.c +7 -0
+		if (kvm_is_error_hva(hva)) {
+			ret = -EFAULT;
+			goto out;
+		}
+
+		s2fd.hva = hva;
+		ret = user_mem_abort(&s2fd);

If is_protected_kvm_enabled(), this function doesn't check if the mapping already exists. user_mem_abort() will call **p**kvm_pgtable_stage2_map() (emphasis is mine) to create the mapping, and pkvm_pgtable_stage2_map() will return -EAGAIN, and keep returning -EAGAIN until the mapping is destroyed, which might be some time away.

Quoted 8 lines
Quoted 6 lines

Same thing happens **without** pKVM, kvm_pgtable_stage2_map() will return -EAGAIN if the mapping exists. But that's not such a big issue, when the user calls KVM_PRE_FAULT_MEMORY on the remaining range kvm_pgtable_get_leaf() will detect the existing mapping. Which you have already taken into consideration judging by the comment, so all good here.

But both gmem_abort() and user_mem_abort() handle -EAGAIN internally, don't they?

I think you're right, please ignore my comment about an existing mapping with is_protected_kvm_enabled().

Thanks,
Alex

#9 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Hi Aneesh and Alex

Quoted 5 lines · Alexandru Elisei
Quoted 1 line · Aneesh Kumar K.V

But both gmem_abort() and user_mem_abort() handle -EAGAIN internally, don't they?

I think you're right, please ignore my comment about an existing mapping with is_protected_kvm_enabled().

Thanks Aneesh, yeah that's the intent. Vincent pointed me to the pgt->pkvm_mappings interval tree that can be used to detect the existing mappings in pKVM, so I'll be updating to use that in v6.

Signature · 2 lines
Thanks,
Jack
#10 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

In reply to Alexandru Elisei

Hi Alex,

Quoted 7 lines · Alexandru Elisei

arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574

Quoted 1 line
arch/arm64/kvm/mmu.c +1 -0
+	unsigned long		*page_size;

As far as I know, fault handling was reworked to use struct kvm_s2_fault_desc to store the fault information that user_mem_abort() needs to handle the fault. Adding a 'page_size' field, that represents the result of the gpa mapping process, might not be desirable.

Yeah agreed, Vincent suggested a separate output struct that kvm_s2_fault_map() fills with what was actually mapped. I'll do that in v6 thanks!

Quoted 6 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2805

Quoted 2 lines
arch/arm64/kvm/mmu.c +2 -0
+	struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault;
+	struct kvm_vcpu_fault_info fault_backup = *fault_info;

I'm not sure you need to make a backup here. vcpu->arch.fault is populated each time the CPU takes a fault.

So this was actually flagged up by sashiko when I ran it before submission, it suggested handling this for the case in which the vCPU exited for MMIO, the next KVM_RUN calls kvm_handle_mmio_return which uses the vcu->arch.fault.esr_el2. I'll add some comments to this maybe to explain why this would be needed.

Quoted 8 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2843

Quoted 1 line
arch/arm64/kvm/mmu.c +1 -0
+	if (memslot->flags & KVM_MEMSLOT_INVALID) {

I don't think that's something we should care about, the flag can be set immediately after the check as the function doesn't take kvm->slots_lock. kvm_vcpu_prefault_memory() takes the srcu lock in read mode, so the function is safe to run even if userspace does something silly like deleting a memslot at the same time that it's prefaulting the guest memory it represents.

I was there not as much for safety, but rather to pick up that error. This way rather than userspace getting an -EFAULT it can just retry. This is the same way it is handled in x86 as well.

Quoted 5 lines

It might not be obvious, but taking kvm->mmu_lock in read mode does not guarantee that gpa will be mapped when kvm_pgtable_get_leaf() returns. That's because a concurrent kvm_pgtable_stage2_map() for a different gpa can destroy the mapping of the current gpa.

[ ... ]

Quoted 3 lines

The kvm->mmu_lock is dropped here, which means that it is possible for a MMU notifier callback to have just unmapped the entire stage 2 for the VM.

Yeah as we mentioned in the other thread, this is best effort and we can't guarantee that is survives the return. We do this for forward progression, reporting the correct advance size for ranges which are already mapped, instead of repeating the full dance for every page of them.

Quoted 4 lines

handle_access_fault() will fail if the mapping is gone. But I guess that's fine if the ioctl does not guarantee that memory is still mapped after it completes.

Yeah I think it's fine and will degrade to a no-op in this case.

Quoted 5 lines

Also, the documentation that this patch adds says: 'On arm64, newly created stage-2 PTEs are marked Accessed'. Does not say anything about marking **existing** ptes as accessed. Would be useful to explain the code does it.

Ack, will do thanks!

Quoted 4 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2896

Quoted 1 line
arch/arm64/kvm/mmu.c +1 -0
+		hva = gfn_to_hva_memslot_prot(memslot, gfn, NULL);

There's gfn_to_hva_memslot(memslot, gfn), is that what you are looking for?

I think gfn_to_hva_memslot() resolves the hva for write, so would fail on KVM_MEM_READONLY slots, and prefualting is a read. Also this mirrors the guest abort path so matches the way the run path resolves the HVA.

Signature · 2 lines
Thanks,
Jack
#11 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Hi Jack,

Quoted 22 lines · Jack

Hi Alex,

Quoted 7 lines · Alexandru Elisei

arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574

Quoted 1 line
arch/arm64/kvm/mmu.c +1 -0
+	unsigned long		*page_size;

As far as I know, fault handling was reworked to use struct kvm_s2_fault_desc to store the fault information that user_mem_abort() needs to handle the fault. Adding a 'page_size' field, that represents the result of the gpa mapping process, might not be desirable.

Yeah agreed, Vincent suggested a separate output struct that kvm_s2_fault_map() fills with what was actually mapped. I'll do that in v6 thanks!

Quoted 6 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2805

Quoted 2 lines
arch/arm64/kvm/mmu.c +2 -0
+	struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault;
+	struct kvm_vcpu_fault_info fault_backup = *fault_info;

I'm not sure you need to make a backup here. vcpu->arch.fault is populated each time the CPU takes a fault.

Quoted 5 lines

So this was actually flagged up by sashiko when I ran it before submission, it suggested handling this for the case in which the vCPU exited for MMIO, the next KVM_RUN calls kvm_handle_mmio_return which uses the vcu->arch.fault.esr_el2. I'll add some comments to this maybe to explain why this would be needed.

Ok, I see, VCPU exits to userspace due to MMIO, userspace calls KVM_PRE_FAULT_MEMORY, userspace resumes VCPU and uses fault_info from KVM_PRE_FAULT_MEMORY. Nicely spotted.

Quoted 10 lines
Quoted 9 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2843

Quoted 2 lines
arch/arm64/kvm/mmu.c +1 -0
+	if (memslot->flags & KVM_MEMSLOT_INVALID) {

I don't think that's something we should care about, the flag can be set immediately after the check as the function doesn't take kvm->slots_lock. kvm_vcpu_prefault_memory() takes the srcu lock in read mode, so the function is safe to run even if userspace does something silly like deleting a memslot at the same time that it's prefaulting the guest memory it represents.

Quoted 3 lines

I was there not as much for safety, but rather to pick up that error. This way rather than userspace getting an -EFAULT it can just retry. This is the same way it is handled in x86 as well.

My point was that since there's no serialization between memslot changes and KVM_PRE_FAULT_MEMORY, the condition can become true immediately after the if statement. Doesn't really matter though anyway.

Quoted 25 lines
Quoted 6 lines

It might not be obvious, but taking kvm->mmu_lock in read mode does not guarantee that gpa will be mapped when kvm_pgtable_get_leaf() returns. That's because a concurrent kvm_pgtable_stage2_map() for a different gpa can destroy the mapping of the current gpa.

[ ... ]

Quoted 3 lines

The kvm->mmu_lock is dropped here, which means that it is possible for a MMU notifier callback to have just unmapped the entire stage 2 for the VM.

Yeah as we mentioned in the other thread, this is best effort and we can't guarantee that is survives the return. We do this for forward progression, reporting the correct advance size for ranges which are already mapped, instead of repeating the full dance for every page of them.

Quoted 4 lines

handle_access_fault() will fail if the mapping is gone. But I guess that's fine if the ioctl does not guarantee that memory is still mapped after it completes.

Yeah I think it's fine and will degrade to a no-op in this case.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

Also, the documentation that this patch adds says: 'On arm64, newly created stage-2 PTEs are marked Accessed'. Does not say anything about marking **existing** ptes as accessed. Would be useful to explain the code does it.

Ack, will do thanks!

I think there's a word missing there, I meant to say that it would be useful to explain *why* the x86 implementation does not mark new PTEs as accessed, while arm64 marks them as accessed, and arm64 also does that for existing PTEs.

Quoted 9 lines
Quoted 5 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2896

Quoted 2 lines
arch/arm64/kvm/mmu.c +1 -0
+		hva = gfn_to_hva_memslot_prot(memslot, gfn, NULL);

There's gfn_to_hva_memslot(memslot, gfn), is that what you are looking for?

I think gfn_to_hva_memslot() resolves the hva for write, so would fail on KVM_MEM_READONLY slots, and prefualting is a read. Also this mirrors the guest abort path so matches the way the run path resolves the HVA.

Ah, I missed that, thanks for explaining!

I was wondering if you have any plans about posting an updated series.

Thanks,
Alex

#12 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Hi Alex,

On 25/08/2026 1:29 pm, Alexandru Elisei wrote:

[...]

Quoted 5 lines

I think there's a word missing there, I meant to say that it would be useful to explain *why* the x86 implementation does not mark new PTEs as accessed, while arm64 marks them as accessed, and arm64 also does that for existing PTEs.

Ah got you, I'll update the docs in v6 to explain that. Thanks!

Quoted 1 line

I was wondering if you have any plans about posting an updated series.

Yeah I have v6 almost ready, I've addressed the comments. While testing it found an issue on pKVM which would case a hyp panic if we prefaulted before it's first run, the EL2 context load blows up. So I've also go a fix for that bundled in.

I'm just tidying a few bits up and hopefully have it out within the next couple of days.

Signature · 2 lines
Thanks,
Jack
#13 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

In reply to Jack Thomson

Hi,

Quoted 117 lines · Jack Thomson

From: Jack Thomson <[email protected]>

Add arm64 support for KVM_PRE_FAULT_MEMORY by synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.

If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl; vcpu_put() at ioctl exit drops the hw_mmu and the next vcpu_load() reselects the correct MMU from vCPU state.

Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with -EHWPOISON instead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.

Advertise KVM_CAP_PRE_FAULT_MEMORY on arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns -EOPNOTSUPP if invoked anyway.

---
 Documentation/virt/kvm/api.rst |  18 +++-
 arch/arm64/kvm/Kconfig         |   1 +
 arch/arm64/kvm/arm.c           |   1 +
 arch/arm64/kvm/mmu.c           | 162 +++++++++++++++++++++++++++++++++
 4 files changed, 178 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +14 -4
index 52bbbb553ce1..657e05656fa6 100644
@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 ---------------------------
 
 :Capability: KVM_CAP_PRE_FAULT_MEMORY
-:Architectures: none
+:Architectures: x86, arm64
 :Type: vcpu ioctl
 :Parameters: struct kvm_pre_fault_memory (in/out)
 :Returns: 0 if at least one page is processed, < 0 on error
@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
 Errors:
 
   ========== ===============================================================
+  EAGAIN     A memslot update raced with the ioctl before any page was
+             processed.
   EINVAL     The specified `gpa` and `size` were invalid (e.g. not
              page aligned, causes an overflow, or size is zero).
   ENOENT     The specified `gpa` is outside defined memslots.
   EINTR      An unmasked signal is pending and no page was processed.
   EFAULT     The parameter address was invalid.
+  EHWPOISON  A poisoned host page was encountered.
   EOPNOTSUPP Mapping memory for a GPA is unsupported by the
              hypervisor, and/or for the current vCPU state/mode.
   EIO        unexpected error conditions (also causes a WARN)
@@ -6494,7 +6497,14 @@ Errors:
 KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory
 for the current vCPU state.  KVM maps memory as if the vCPU generated a
 stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
-CoW.  However, KVM does not mark any newly created stage-2 PTE as Accessed.
+CoW.  However, on x86, KVM does not mark any newly created stage-2 PTE as
+Accessed.  On arm64, newly created stage-2 PTEs are marked Accessed.
+
+On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's
+memslot address space.  If the vCPU most recently ran a nested guest, KVM
+still targets the VM's canonical stage-2, and does not interpret `gpa` as
+a nested guest IPA or target the nested/shadow stage-2 selected by the
+vCPU's last run state.
 
 In the case of confidential VM types where there is an initial set up of
 private guest memory before the guest is 'finalized'/measured, this ioctl
@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel.
 
 When the ioctl returns, the input values are updated to point to the
 remaining range.  If `size` > 0 on return, the caller can just issue
-the ioctl again with the same `struct kvm_map_memory` argument.
+the ioctl again with the same `struct kvm_pre_fault_memory` argument.
 
-Shadow page tables cannot support this ioctl because they
+On x86, shadow page tables cannot support this ioctl because they
 are indexed by virtual address or nested guest physical address.
 Calling this ioctl when the guest is using shadow page tables (for
 example because it is running a nested guest with nested page tables)
arch/arm64/kvm/Kconfig +1 -0
index 449154f9a485..6b89262e8ba7 100644
@@ -24,6 +24,7 @@ menuconfig KVM
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
 	select KVM_MMIO
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
+	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select VIRT_XFER_TO_GUEST_WORK
 	select KVM_VFIO
 	select HAVE_KVM_DIRTY_RING_ACQ_REL
arch/arm64/kvm/arm.c +1 -0
index 9453321ef8c6..dcb92bee13af 100644
@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_COUNTER_OFFSET:
 	case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS:
 	case KVM_CAP_ARM_SEA_TO_USER:
+	case KVM_CAP_PRE_FAULT_MEMORY:
 		r = 1;
 		break;
 	case KVM_CAP_SET_GUEST_DEBUG2:
arch/arm64/kvm/mmu.c +0 -0
index c720f07cb82e..4bf048bbcf8b 100644

arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574

Quoted 4 lines
arch/arm64/kvm/mmu.c +1 -0
@@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {
 	struct kvm_s2_trans	*nested;
 	struct kvm_memory_slot	*memslot;
 	unsigned long		hva;
+	unsigned long		*page_size;

It looks a bit odd in that struct.

Also, transparent_hugepage_adjust() modifies pfn and gfn. So perhaps it'd be better to introduce a struct kvm_s2_fault_mapping that can be used to get what has actually been mapped in kvm_s2_fault_map()?

Quoted 69 lines
arch/arm64/kvm/mmu.c +48 -0
+	bool			prefault;
 };
 
 static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
@@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd,
 				      &s2vi->map_writable, &s2vi->page);
 	if (unlikely(is_error_noslot_pfn(s2vi->pfn))) {
 		if (s2vi->pfn == KVM_PFN_ERR_HWPOISON) {
+			/*
+			 * When prefaulting, report the poison via -EHWPOISON
+			 * only; don't also queue a SIGBUS as the run path
+			 * does for the faulting vCPU thread.
+			 */
+			if (s2fd->prefault)
+				return -EHWPOISON;
 			kvm_send_hwpoison_signal(s2fd->hva, __ffs(s2vi->vma_pagesize));
 			return 0;
 		}
@@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable);
 	kvm_fault_unlock(kvm);
 
+	if (s2fd->page_size && !ret)
+		*s2fd->page_size = mapping_size;
+
 	/*
 	 * Mark the page dirty only if the fault is handled successfully,
 	 * making sure we adjust the canonical IPA if the mapping size has
@@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)
 
 	trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled);
 }
+
+/*
+ * Prefaulting always targets the canonical stage-2.  If the vCPU last ran
+ * in a nested context, swap in the canonical MMU via the vCPU put/load
+ * helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount
+ * state stay consistent.
+ *
+ * The swap is deliberately not undone: nothing runs in between the
+ * per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the
+ * generic prefault loop, and the vcpu_put() at ioctl exit discards
+ * vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next
+ * vcpu_load() re-derives the correct MMU from the vCPU's context.  If the
+ * prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu()
+ * keeps the canonical MMU in place for the reload.  Leaving the swap in
+ * place also bounds the cost to at most one put/load pair per ioctl,
+ * rather than two pairs per prefaulted page.
+ */
+static void kvm_pre_fault_load_canonical_mmu(struct kvm_vcpu *vcpu)
+{
+	if (!vcpu_has_nv(vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu)
+		return;
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+}
+
+long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
+				    struct kvm_pre_fault_memory *range)
+{
+	struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault;
+	struct kvm_vcpu_fault_info fault_backup = *fault_info;
+	s8 walk_level = KVM_PGTABLE_LAST_LEVEL;
+	unsigned long page_size = PAGE_SIZE;
+	struct kvm_memory_slot *memslot;

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2818

Quoted 10 lines
arch/arm64/kvm/mmu.c +10 -0
+	phys_addr_t gpa = range->gpa;
+	struct kvm_pgtable *pgt;
+	phys_addr_t end;
+	kvm_pte_t pte;
+	hva_t hva;
+	gfn_t gfn;
+	long ret;
+
+	if (vcpu_is_protected(vcpu))
+		return -EOPNOTSUPP;

It could just call pkvm_mem_abort() when the VM is protected?

Quoted 23 lines
arch/arm64/kvm/mmu.c +23 -0
+
+	/*
+	 * Interpret range->gpa in the userspace-owned VM's IPA space, not in
+	 * any nested guest IPA space that may have been active on the vCPU's
+	 * last run.  Always target the canonical stage-2.
+	 */
+	kvm_pre_fault_load_canonical_mmu(vcpu);
+
+	if (gpa >= kvm_phys_size(vcpu->arch.hw_mmu)) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	gfn = gpa_to_gfn(gpa);
+	memslot = gfn_to_memslot(vcpu->kvm, gfn);
+	if (!memslot) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	/*
+	 * A racing memslot deletion or move installs an invalid slot before
+	 * zapping stage-2.  Ask userspace to retry once the update settles.

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2851

Quoted 10 lines
arch/arm64/kvm/mmu.c +10 -0
+	 */
+	if (memslot->flags & KVM_MEMSLOT_INVALID) {
+		ret = -EAGAIN;
+		goto out;
+	}
+
+	/*
+	 * pKVM stage-2 mappings aren't directly walkable from the host; let
+	 * the fault path handle both new and existing mappings.
+	 */

pKVM has a host interval-tree where we keep track of what has been mapped and at what level. This tree is also protected with the mmu_lock.

Quoted 74 lines
arch/arm64/kvm/mmu.c +70 -0
+	if (!is_protected_kvm_enabled()) {
+		pgt = vcpu->arch.hw_mmu->pgt;
+		scoped_guard(read_lock, &vcpu->kvm->mmu_lock) {
+			ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, &walk_level,
+						   KVM_PGTABLE_WALK_SHARED);
+		}
+		if (ret)
+			goto out;
+
+		if (kvm_pte_valid(pte)) {
+			page_size = kvm_granule_size(walk_level);
+			if (!(pte & KVM_PTE_LEAF_ATTR_LO_S2_AF))
+				handle_access_fault(vcpu, gpa);
+			goto out_success;
+		}
+	}
+
+	/*
+	 * Synthesize a read translation fault for the canonical IPA, at the
+	 * level where the stage-2 walk currently ends (the last level under
+	 * pKVM, where stage-2 isn't walkable from the host).
+	 */
+	fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) |
+		ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(walk_level);
+	fault_info->hpfar_el2 = HPFAR_EL2_NS |
+		FIELD_PREP(HPFAR_EL2_FIPA, gpa >> 12);
+
+	struct kvm_s2_fault_desc s2fd = {
+		.vcpu		= vcpu,
+		.fault_ipa	= gpa,
+		.nested		= NULL,
+		.memslot	= memslot,
+		.page_size	= &page_size,
+		.prefault	= true,
+	};
+
+	/*
+	 * As in the run path, -EAGAIN from the abort handlers is treated as
+	 * progress: either a parallel fault installed the mapping, or a racing
+	 * invalidation is in flight and the next access will refault.
+	 */
+	if (kvm_slot_has_gmem(memslot)) {
+		ret = gmem_abort(&s2fd);
+	} else {
+		hva = gfn_to_hva_memslot_prot(memslot, gfn, NULL);
+		if (kvm_is_error_hva(hva)) {
+			ret = -EFAULT;
+			goto out;
+		}
+
+		s2fd.hva = hva;
+		ret = user_mem_abort(&s2fd);
+	}
+
+	if (ret < 0)
+		goto out;
+
+out_success:
+	end = ALIGN_DOWN(gpa, page_size) + page_size;
+	ret = min_t(u64, range->size, end - gpa);
+out:
+	/*
+	 * Restore the synthetic fault state so a subsequent KVM_RUN does not
+	 * observe it. kvm_handle_mmio_return() runs before guest entry can
+	 * refresh fault.esr_el2 from hardware, so leaving the synthetic ESR
+	 * in place would corrupt the completion of a pending MMIO exit.
+	 */
+	*fault_info = fault_backup;
+	return ret;
+}
Signature · 1 line
2.43.0
#14 lore raw

Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation

Hi Vincent,

Quoted 8 lines · Vincent Donnefort

arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574

Quoted 1 line
arch/arm64/kvm/mmu.c +1 -0
+	unsigned long		*page_size;

It looks a bit odd in that struct.

Also, transparent_hugepage_adjust() modifies pfn and gfn. So perhaps it'd be better to introduce a struct kvm_s2_fault_mapping that can be used to get what has actually been mapped in kvm_s2_fault_map()?

Yeah that's much nicer I'll do that thanks for the suggestion!

Quoted 5 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2818

Quoted 2 lines
arch/arm64/kvm/mmu.c +2 -0
+	if (vcpu_is_protected(vcpu))
+		return -EOPNOTSUPP;

It could just call pkvm_mem_abort() when the VM is protected?

I see, I don't have a setup to test this properly myself, so at the moment I left the protected vm's out and don't advertise the capability, if there's a usecase happy to look at it as a follow-up.

Quoted 8 lines

arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2851

Quoted 4 lines
arch/arm64/kvm/mmu.c +4 -0
+	/*
+	 * pKVM stage-2 mappings aren't directly walkable from the host; let
+	 * the fault path handle both new and existing mappings.
+	 */

pKVM has a host interval-tree where we keep track of what has been mapped and at what level. This tree is also protected with the mmu_lock.

Thanks, that sounds better. I'll have a look at moving to that for v6 then.

Signature · 2 lines
Thanks,
Jack
#15 lore raw

PATCH v5 3/5 KVM: selftests: Enable pre_fault_memory_test for arm64

In reply to Jack Thomson

From: Jack Thomson <[email protected]>

Enable the pre_fault_memory_test to run on arm64 by making it work with different guest page sizes and testing multiple guest configurations.

Update the test_assert to compare against the UCALL_EXIT_REASON, for portability, as arm64 exits with KVM_EXIT_MMIO while x86 uses KVM_EXIT_IO.

---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/pre_fault_memory_test.c     | 115 ++++++++++++++----
 2 files changed, 92 insertions(+), 24 deletions(-)
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 9118a5a51b89..4609d8f23e38 100644
@@ -194,6 +194,7 @@ TEST_GEN_PROGS_arm64 += guest_memfd_test
 TEST_GEN_PROGS_arm64 += mmu_stress_test
 TEST_GEN_PROGS_arm64 += rseq_test
 TEST_GEN_PROGS_arm64 += steal_time
+TEST_GEN_PROGS_arm64 += pre_fault_memory_test
 
 TEST_GEN_PROGS_s390 = $(TEST_GEN_PROGS_COMMON)
 TEST_GEN_PROGS_s390 += s390/memop
tools/testing/selftests/kvm/pre_fault_memory_test.c +91 -24

184 lines not on this page · raw message

Signature · 1 line
2.43.0
#16 lore raw

PATCH v5 4/5 KVM: selftests: Add option for different backing in pre-fault tests

In reply to Jack Thomson

From: Jack Thomson <[email protected]>

Add a -s option to specify different memory backing types for the pre-fault tests (e.g. anonymous, hugetlb), allowing testing of the pre-fault functionality across different memory configurations.

---
 .../selftests/kvm/pre_fault_memory_test.c     | 51 +++++++++++++------
 1 file changed, 36 insertions(+), 15 deletions(-)
tools/testing/selftests/kvm/pre_fault_memory_test.c +36 -15
index 9f5f0d1a5db1..c850cf28e86a 100644
@@ -45,6 +45,7 @@ struct slot_worker_data {
 	struct kvm_vm *vm;
 	gpa_t gpa;
 	u32 flags;
+	enum vm_mem_backing_src_type mem_backing_src;
 	bool worker_ready;
 	bool prefault_ready;
 	bool recreate_slot;
@@ -65,14 +66,16 @@ static void *delete_slot_worker(void *__data)
 	while (!READ_ONCE(data->recreate_slot))
 		cpu_relax();
 
-	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, data->gpa,
+	vm_userspace_mem_region_add(vm, data->mem_backing_src, data->gpa,
 				    TEST_SLOT, test_config.test_num_pages, data->flags);
 
 	return NULL;
 }
 
 static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset,
-			     u64 size, u64 expected_left, bool private)
+			     u64 size, u64 expected_left,
+			     enum vm_mem_backing_src_type mem_backing_src,
+			     bool private)
 {
 	struct kvm_pre_fault_memory range = {
 		.gpa = base_gpa + offset,
@@ -83,6 +86,7 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset,
 		.vm = vcpu->vm,
 		.gpa = base_gpa,
 		.flags = private ? KVM_MEM_GUEST_MEMFD : 0,
+		.mem_backing_src = mem_backing_src,
 	};
 	bool slot_recreated = false;
 	pthread_t slot_worker;
@@ -172,11 +176,13 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset,
 struct test_params {
 	unsigned long vm_type;
 	bool private;
+	enum vm_mem_backing_src_type mem_backing_src;
 };
 
 static void __test_pre_fault_memory(enum vm_guest_mode guest_mode, void *arg)
 {
 	gpa_t gpa, gva, alignment, guest_page_size, host_page_size;
+	gpa_t backing_src_pagesz, mem_page_size;
 	struct test_params *p = arg;
 	const struct vm_shape shape = {
 		.mode = guest_mode,
@@ -188,24 +194,28 @@ static void __test_pre_fault_memory(enum vm_guest_mode guest_mode, void *arg)
 	struct ucall uc;
 
 	pr_info("Testing guest mode: %s\n", vm_guest_mode_string(guest_mode));
+	pr_info("Testing memory backing src type: %s\n",
+		vm_mem_backing_src_alias(p->mem_backing_src)->name);
 
 	vm = vm_create_shape_with_one_vcpu(shape, &vcpu, guest_code);
 
 	guest_page_size = vm_guest_mode_params[guest_mode].page_size;
 	host_page_size = getpagesize();
+	backing_src_pagesz = get_backing_src_pagesz(p->mem_backing_src);
+	mem_page_size = max(host_page_size, backing_src_pagesz);
 
 	test_config.page_size = guest_page_size;
 	test_config.test_size = align_up(TEST_BASE_SIZE + test_config.page_size,
-					 host_page_size);
+					 mem_page_size);
 	test_config.test_num_pages = vm_calc_num_guest_pages(vm->mode, test_config.test_size);
 
 	gpa = (vm->max_gfn - test_config.test_num_pages) * test_config.page_size;
 	alignment = SZ_2M;
-	alignment = max(alignment, host_page_size);
+	alignment = max(alignment, mem_page_size);
 	gpa = align_down(gpa, alignment);
 	gva = gpa & ((1ULL << (vm->va_bits - 1)) - 1);
 
-	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+	vm_userspace_mem_region_add(vm, p->mem_backing_src,
 				    gpa, TEST_SLOT, test_config.test_num_pages,
 				    p->private ? KVM_MEM_GUEST_MEMFD : 0);
 	virt_map(vm, gva, gpa, test_config.test_num_pages);
@@ -213,14 +223,18 @@ static void __test_pre_fault_memory(enum vm_guest_mode guest_mode, void *arg)
 	if (p->private)
 		vm_mem_set_private(vm, gpa, test_config.test_size);
 
-	pre_fault_memory(vcpu, gpa, 0, test_config.test_size, 0, p->private);
+	pre_fault_memory(vcpu, gpa, 0, test_config.test_size, 0,
+			 p->mem_backing_src, p->private);
 	/* Retry the same range after the first prefault attempt. */
-	pre_fault_memory(vcpu, gpa, 0, test_config.test_size, 0, p->private);
+	pre_fault_memory(vcpu, gpa, 0, test_config.test_size, 0,
+			 p->mem_backing_src, p->private);
 	pre_fault_memory(vcpu, gpa,
 			 test_config.test_size - host_page_size,
-			 host_page_size * 2, host_page_size, p->private);
+			 host_page_size * 2, host_page_size,
+			 p->mem_backing_src, p->private);
 	pre_fault_memory(vcpu, gpa, test_config.test_size,
-			 host_page_size, host_page_size, p->private);
+			 host_page_size, host_page_size,
+			 p->mem_backing_src, p->private);
 
 	vcpu_args_set(vcpu, 1, gva);
 
@@ -249,11 +263,13 @@ static void __test_pre_fault_memory(enum vm_guest_mode guest_mode, void *arg)
 	kvm_vm_free(vm);
 }
 
-static void test_pre_fault_memory(unsigned long vm_type, bool private)
+static void test_pre_fault_memory(unsigned long vm_type, enum vm_mem_backing_src_type backing_src,
+				  bool private)
 {
 	struct test_params p = {
 		.vm_type = vm_type,
 		.private = private,
+		.mem_backing_src = backing_src,
 	};
 
 	if (vm_type && !(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(vm_type))) {
@@ -267,23 +283,28 @@ static void test_pre_fault_memory(unsigned long vm_type, bool private)
 static void help(char *name)
 {
 	puts("");
-	printf("usage: %s [-h] [-m mode]\n", name);
+	printf("usage: %s [-h] [-m mode] [-s mem-type]\n", name);
 	puts("");
 	guest_modes_help();
+	backing_src_help("-s");
 	puts("");
 }
 
 int main(int argc, char *argv[])
 {
+	enum vm_mem_backing_src_type backing = DEFAULT_VM_MEM_SRC;
 	int opt;
 
 	guest_modes_append_default();
 
-	while ((opt = getopt(argc, argv, "hm:")) != -1) {
+	while ((opt = getopt(argc, argv, "hm:s:")) != -1) {
 		switch (opt) {
 		case 'm':
 			guest_modes_cmdline(optarg);
 			break;
+		case 's':
+			backing = parse_backing_src_type(optarg);
+			break;
 		case 'h':
 		default:
 			help(argv[0]);
@@ -293,10 +314,10 @@ int main(int argc, char *argv[])
 
 	TEST_REQUIRE(kvm_check_cap(KVM_CAP_PRE_FAULT_MEMORY));
 
-	test_pre_fault_memory(0, false);
+	test_pre_fault_memory(0, backing, false);
 #ifdef __x86_64__
-	test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, false);
-	test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, true);
+	test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, backing, false);
+	test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, backing, true);
 #endif
 	return 0;
 }
Signature · 1 line
2.43.0
#17 lore raw

PATCH v5 5/5 KVM: selftests: Add nested pre-fault test for arm64

In reply to Jack Thomson

From: Jack Thomson <[email protected]>

Add an arm64 nested-virt selftest for KVM_PRE_FAULT_MEMORY. The guest enters vEL1 and exits to userspace with a nested/shadow stage-2 MMU as the vCPU's last-run context.

Before prefaulting, userspace enables HCR_EL2.VM and points VTTBR_EL2 at an empty nested stage-2 root. A prefault implementation that incorrectly treats the userspace GPA as an L2 IPA will fail the ioctl; the correct path swaps to the canonical stage-2 and succeeds.

Restore the original nested state before resuming the guest, then touch the prefaulted range to check that vEL1 still runs correctly.

---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../kvm/arm64/nv_pre_fault_memory_test.c      | 200 ++++++++++++++++++
 2 files changed, 201 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 4609d8f23e38..63d79245b47d 100644
@@ -170,6 +170,7 @@ TEST_GEN_PROGS_arm64 += arm64/debug-exceptions
 TEST_GEN_PROGS_arm64 += arm64/hello_el2
 TEST_GEN_PROGS_arm64 += arm64/host_sve
 TEST_GEN_PROGS_arm64 += arm64/hypercalls
+TEST_GEN_PROGS_arm64 += arm64/nv_pre_fault_memory_test
 TEST_GEN_PROGS_arm64 += arm64/external_aborts
 TEST_GEN_PROGS_arm64 += arm64/page_fault_test
 TEST_GEN_PROGS_arm64 += arm64/psci_test
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file +200 -0

200 lines not on this page · raw message

Signature · 1 line
2.43.0
#18 lore raw

Re: [PATCH v5 5/5] KVM: selftests: Add nested pre-fault test for arm64

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The `KVM_PRE_FAULT_MEMORY` ioctl loop in `pre_fault()` fails to handle partial success, causing…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The KVM_PRE_FAULT_MEMORY ioctl loop in pre_fault() fails to handle partial success, causing spurious test failures if the ioctl yields or is interrupted. --

commit ee1e827d8efbd103bf9ab1322ee50ba8c3895869
Author: Jack Thomson <[email protected]>

KVM: selftests: Add nested pre-fault test for arm64

This commit adds an arm64 nested-virt selftest for KVM_PRE_FAULT_MEMORY to verify that prefaulting correctly targets the canonical stage-2 when the guest's last-run context has a nested/shadow stage-2 MMU.

Quoted 5 lines
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file +0 -0
new file mode 100644
index 0000000000000..2bbd5540599c0

[ ... ]

tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c line 99

Quoted 11 lines
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file +11 -0
+static void pre_fault(struct kvm_vcpu *vcpu, u64 gpa, u64 size)
+{
+	struct kvm_pre_fault_memory range = {
+		.gpa = gpa,
+		.size = size,
+	};
+	int ret;
+
+	do {
+		ret = __vcpu_ioctl(vcpu, KVM_PRE_FAULT_MEMORY, &range);
+	} while (ret < 0 && errno == EINTR);

[Severity: Medium]
Does this loop handle partial successes correctly? If the ioctl is interrupted by a signal or yields after successfully mapping at least one page, it returns 0 and updates range.size with the remaining bytes to process.

If that happens, this loop will exit immediately and the test will spuriously fail the TEST_ASSERT_EQ(range.size, 0) assertion below. Should this loop continue as long as ret == 0 and range.size > 0?

Quoted 5 lines
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file +5 -0
+
+	TEST_ASSERT(!ret, "KVM_PRE_FAULT_MEMORY failed, ret: %d errno: %d",
+		    ret, errno);
+	TEST_ASSERT_EQ(range.size, 0);
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5