KVM: arm64: Add KVM_PRE_FAULT_MEMORY support
Subsystems (5)
v5 cover KVM: arm64: Add KVM_PRE_FAULT_MEMORY support
From: Jack Thomson <[email protected]>
Hi,
This series adds arm64 support for KVM_PRE_FAULT_MEMORY, which was added for x86 in [1]. The ioctl allows userspace to populate stage-2 mappings before running a vCPU, reducing the number of stage-2 faults taken in the run path. This is useful for post-copy migration, where stage-2 fault latency shows up directly in memory-intensive workloads.
On arm64, the GPA supplied to the ioctl is treated as an IPA in the userspace-owned VM's memslot address space. If the vCPU most recently ran a nested guest, KVM still targets the VM's canonical stage-2. It does not interpret the GPA as an L2 IPA, and does not try to populate the nested/shadow stage-2 selected by the vCPU's last run state.
The patches are:
- Allow callers of kvm_pgtable_get_leaf() to pass walk flags, so the prefault path can walk stage-2 under the MMU read lock.
- Add arm64 support for KVM_PRE_FAULT_MEMORY.
- Enable pre_fault_memory_test on arm64.
- Add a backing-source option to pre_fault_memory_test.
- Add a nested (NV) selftest that prefaults on a vCPU whose last-run context is backed by a shadow stage-2 MMU with an empty nested stage-2 root.
The prefault flag and page_size output in the stage-2 fault descriptor remain in this series so the arm64 implementation can advance by the mapping granule installed by the fault path and report poison without queueing a SIGBUS.
Tested with pre_fault_memory_test under an arm64 QEMU setup with anonymous, shmem, anonymous_thp, anonymous_hugetlb and shared_hugetlb backings, including 64K, 2M and 32M hugetlb pools, and with the new nv_pre_fault_memory_test on an NV-capable setup.
=== Changes since v4 [2] ===
- Reworked nested virt semantics: arm64 now treats the ioctl GPA as the VM/memslot IPA and always targets the canonical stage-2. It no longer translates an L2 IPA through L1's stage-2.
- Documented the arm64 nested behavior in the KVM API text.
- Switch to the canonical stage-2 with the vCPU put/load helpers when the vCPU last ran with a nested/shadow MMU, keeping VMID, VNCR and shadow-MMU refcount state consistent.
- Split the kvm_pgtable_get_leaf() walk-flag plumbing into a prep patch and walk existing mappings with KVM_PGTABLE_WALK_SHARED under the MMU read lock.
- Tightened prefault fault handling: preserve fault info, set IL in the synthetic ESR, handle existing mappings, return -EAGAIN for invalid memslot races, and report -EHWPOISON without queueing SIGBUS.
- Avoid directly walking stage-2 page tables when pKVM is enabled. Protected VMs remain unsupported via -EOPNOTSUPP.
- Preserve the selected selftest memory backing when recreating the racing memslot.
- Add the nested (NV) prefault selftest, including an empty nested stage-2 root to catch accidental L2-IPA interpretation.
=== Changes since v3 [3] ===
- Return -EOPNOTSUPP for protected VMs.
- Reworked nested-vCPU handling to translate an L2 IPA through L1's stage-2. This has been superseded by the canonical VM-IPA semantics described above.
- Make page_size unsigned and keep local declarations ordered at the top of kvm_arch_vcpu_pre_fault_memory().
=== Changes since v2 [4] ===
- Update the synthetic fault info. Thanks Suzuki.
- Remove the selftest change for unaligned mmap allocations. Thanks Sean.
[1]: https://lore.kernel.org/kvm/[email protected]/
[2]: https://lore.kernel.org/linux-arm-kernel/[email protected]/
[3]: https://lore.kernel.org/linux-arm-kernel/[email protected]/
[4]: https://lore.kernel.org/linux-arm-kernel/[email protected]/
Jack Thomson (5):
KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()
KVM: arm64: Add pre_fault_memory implementation
KVM: selftests: Enable pre_fault_memory_test for arm64
KVM: selftests: Add option for different backing in pre-fault tests
KVM: selftests: Add nested pre-fault test for arm64
Documentation/virt/kvm/api.rst | 18 +- arch/arm64/include/asm/kvm_pgtable.h | 5 +- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 10 +- arch/arm64/kvm/hyp/pgtable.c | 5 +- arch/arm64/kvm/mmu.c | 164 +++++++++++++- arch/arm64/kvm/nested.c | 2 +- tools/testing/selftests/kvm/Makefile.kvm | 2 + .../kvm/arm64/nv_pre_fault_memory_test.c | 200 ++++++++++++++++++ .../selftests/kvm/pre_fault_memory_test.c | 150 ++++++++++--- 11 files changed, 513 insertions(+), 45 deletions(-) create mode 100644 tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c
base-commit: 98f826f3c500fda08d51fca434b7aefa6a2f7076
Signature · 1 line
2.43.0
PATCH v5 1/5 KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()
From: Jack Thomson <[email protected]>
Allow callers of kvm_pgtable_get_leaf() to specify the page-table walk flags, in preparation for performing walks under the MMU read lock.
Reading a stage-2 leaf while only holding the read lock requiresKVM_PGTABLE_WALK_SHARED: parallel faults (which also only hold the read lock) can unlink table pages and free them via RCU, so the walker must be inside an RCU read-side critical section, which the shared walk flag provides via kvm_pgtable_walk_begin().
All existing callers either hold the write lock, walk with interrupts disabled, or run at hyp where shared walks are rejected; they keep the current behaviour by passing no flags.
No functional change intended.
--- arch/arm64/include/asm/kvm_pgtable.h | 5 ++++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 10 +++++----- arch/arm64/kvm/hyp/pgtable.c | 5 +++-- arch/arm64/kvm/mmu.c | 2 +- arch/arm64/kvm/nested.c | 2 +- 5 files changed, 14 insertions(+), 10 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +4 -1
@@ -859,6 +859,8 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,* @addr: Input address for the start of the walk. * @ptep: Pointer to storage for the retrieved PTE. * @level: Pointer to storage for the level of the retrieved PTE. +* @flags: Flags to control the page-table walk +* (see struct kvm_pgtable_visit_ctx). * * The offset of @addr within a page is ignored. *
@@ -869,7 +871,8 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,* Return: 0 on success, negative error code on failure. */ int kvm_pgtable_get_leaf (struct kvm_pgtable *pgt,u64 addr, -kvm_pte_t *ptep,s8 *level); +kvm_pte_t *ptep,s8 *level, +enum kvm_pgtable_walk_flags flags);/** * kvm_pgtable_stage2_pte_prot() - Retrieve the protection attributes of a
arch/arm64/kvm/hyp/nvhe/mem_protect.c +5 -5
@@ -522,7 +522,7 @@ static int host_stage2_adjust_range(u64 addr, struct kvm_mem_range *range)int ret;hyp_assert_lock_held (&host_mmu.lock); - ret =kvm_pgtable_get_leaf (&host_mmu.pgt, addr, &pte, &level); + ret =kvm_pgtable_get_leaf (&host_mmu.pgt, addr, &pte, &level,0 );if (ret)return ret;
@@ -890,7 +890,7 @@ static int get_valid_guest_pte(struct pkvm_hyp_vm *vm, u64 ipa, kvm_pte_t *ptep,s8 level;int ret; - ret =kvm_pgtable_get_leaf (&vm->pgt, ipa, &pte, &level); + ret =kvm_pgtable_get_leaf (&vm->pgt, ipa, &pte, &level,0 );if (ret)return ret;if (guest_pte_is_poisoned (pte))
@@ -939,7 +939,7 @@ int __pkvm_vcpu_in_poison_fault(struct pkvm_hyp_vcpu *hyp_vcpu) ipa |=FAR_TO_FIPA_OFFSET (kvm_vcpu_get_hfar (&hyp_vcpu->vcpu));guest_lock_component (vm); - ret =kvm_pgtable_get_leaf (&vm->pgt, ipa, &pte, &level); + ret =kvm_pgtable_get_leaf (&vm->pgt, ipa, &pte, &level,0 );if (ret) goto unlock;
@@ -1293,7 +1293,7 @@ static int host_stage2_get_guest_info(phys_addr_t phys, struct pkvm_hyp_vm **vm,return -EPERM ; } - ret =kvm_pgtable_get_leaf (&host_mmu.pgt, phys, &pte, &level); + ret =kvm_pgtable_get_leaf (&host_mmu.pgt, phys, &pte, &level,0 );if (ret)return ret;
@@ -1522,7 +1522,7 @@ static int __check_host_shared_guest(struct pkvm_hyp_vm *vm, u64 *__phys, u64 ips8 level;int ret; - ret =kvm_pgtable_get_leaf (&vm->pgt, ipa, &pte, &level); + ret =kvm_pgtable_get_leaf (&vm->pgt, ipa, &pte, &level,0 );if (ret)return ret;if (!kvm_pte_valid (pte))
arch/arm64/kvm/hyp/pgtable.c +3 -2
@@ -298,12 +298,13 @@ static int leaf_walker(const struct kvm_pgtable_visit_ctx *ctx, }int kvm_pgtable_get_leaf (struct kvm_pgtable *pgt,u64 addr, -kvm_pte_t *ptep,s8 *level) +kvm_pte_t *ptep,s8 *level, +enum kvm_pgtable_walk_flags flags) {struct leaf_walk_data data;struct kvm_pgtable_walker walker = { .cb = leaf_walker, - .flags =KVM_PGTABLE_WALK_LEAF , + .flags = flags |KVM_PGTABLE_WALK_LEAF , .arg = &data, };int ret;
arch/arm64/kvm/mmu.c +1 -1
@@ -839,7 +839,7 @@ static int get_user_mapping_size(struct kvm *kvm, u64 addr)* IPI-ing threads). */ local_irq_save (flags); - ret =kvm_pgtable_get_leaf (&pgt, addr, &pte, &level); + ret =kvm_pgtable_get_leaf (&pgt, addr, &pte, &level,0 );local_irq_restore (flags);if (ret)
arch/arm64/kvm/nested.c +1 -1
@@ -559,7 +559,7 @@ static u8 get_guest_mapping_ttl(struct kvm_s2_mmu *mmu, u64 addr)return 0 ; tmp &= ~(sz -1 ); -if (kvm_pgtable_get_leaf (mmu->pgt, tmp, &pte,NULL )) +if (kvm_pgtable_get_leaf (mmu->pgt, tmp, &pte,NULL ,0 )) goto again;if (!(pte &PTE_VALID )) goto again;
Signature · 1 line
2.43.0
PATCH v5 2/5 KVM: arm64: Add pre_fault_memory implementation
In reply to Jack ThomsonFrom: Jack Thomson <[email protected]>
Add arm64 support for KVM_PRE_FAULT_MEMORY by synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.
If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl; vcpu_put() at ioctl exit drops the hw_mmu and the next vcpu_load() reselects the correct MMU from vCPU state.
Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with -EHWPOISON instead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.
Advertise KVM_CAP_PRE_FAULT_MEMORY on arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns -EOPNOTSUPP if invoked anyway.
--- Documentation/virt/kvm/api.rst | 18 +++- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 162 +++++++++++++++++++++++++++++++++ 4 files changed, 178 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +14 -4
@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. --------------------------- :Capability: KVM_CAP_PRE_FAULT_MEMORY -:Architectures: none +:Architectures: x86, arm64 :Type: vcpu ioctl :Parameters: struct kvm_pre_fault_memory (in/out) :Returns: 0 if at least one page is processed, < 0 on error
@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. Errors: ========== =============================================================== + EAGAIN A memslot update raced with the ioctl before any page was + processed. EINVAL The specified `gpa` and `size` were invalid (e.g. not page aligned, causes an overflow, or size is zero). ENOENT The specified `gpa` is outside defined memslots. EINTR An unmasked signal is pending and no page was processed. EFAULT The parameter address was invalid. + EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the hypervisor, and/or for the current vCPU state/mode. EIO unexpected error conditions (also causes a WARN)
@@ -6494,7 +6497,14 @@ Errors: KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory for the current vCPU state. KVM maps memory as if the vCPU generated a stage-2 read page fault, e.g. faults in memory as needed, but doesn't break -CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed. +CoW. However, on x86, KVM does not mark any newly created stage-2 PTE as +Accessed. On arm64, newly created stage-2 PTEs are marked Accessed. + +On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's +memslot address space. If the vCPU most recently ran a nested guest, KVM +still targets the VM's canonical stage-2, and does not interpret `gpa` as +a nested guest IPA or target the nested/shadow stage-2 selected by the +vCPU's last run state. In the case of confidential VM types where there is an initial set up of private guest memory before the guest is 'finalized'/measured, this ioctl
@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel. When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue -the ioctl again with the same `struct kvm_map_memory` argument. +the ioctl again with the same `struct kvm_pre_fault_memory` argument. -Shadow page tables cannot support this ioctl because they +On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address. Calling this ioctl when the guest is using shadow page tables (for example because it is running a nested guest with nested page tables)
arch/arm64/kvm/Kconfig +1 -0
@@ -24,6 +24,7 @@ menuconfig KVM select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT + select KVM_GENERIC_PRE_FAULT_MEMORY select VIRT_XFER_TO_GUEST_WORK select KVM_VFIO select HAVE_KVM_DIRTY_RING_ACQ_REL
arch/arm64/kvm/arm.c +1 -0
@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_COUNTER_OFFSET :case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS :case KVM_CAP_ARM_SEA_TO_USER : +case KVM_CAP_PRE_FAULT_MEMORY : r =1 ;break ;case KVM_CAP_SET_GUEST_DEBUG2 :
arch/arm64/kvm/mmu.c +162 -0
183 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v5 3/5 KVM: selftests: Enable pre_fault_memory_test for arm64
In reply to Jack ThomsonFrom: Jack Thomson <[email protected]>
Enable the pre_fault_memory_test to run on arm64 by making it work with different guest page sizes and testing multiple guest configurations.
Update the test_assert to compare against the UCALL_EXIT_REASON, for portability, as arm64 exits with KVM_EXIT_MMIO while x86 uses KVM_EXIT_IO.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/pre_fault_memory_test.c | 115 ++++++++++++++---- 2 files changed, 92 insertions(+), 24 deletions(-)
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -194,6 +194,7 @@ TEST_GEN_PROGS_arm64 += guest_memfd_test TEST_GEN_PROGS_arm64 += mmu_stress_test TEST_GEN_PROGS_arm64 += rseq_test TEST_GEN_PROGS_arm64 += steal_time +TEST_GEN_PROGS_arm64 += pre_fault_memory_test TEST_GEN_PROGS_s390 = $(TEST_GEN_PROGS_COMMON) TEST_GEN_PROGS_s390 += s390/memop
tools/testing/selftests/kvm/pre_fault_memory_test.c +91 -24
184 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v5 4/5 KVM: selftests: Add option for different backing in pre-fault tests
In reply to Jack ThomsonFrom: Jack Thomson <[email protected]>
Add a -s option to specify different memory backing types for the pre-fault tests (e.g. anonymous, hugetlb), allowing testing of the pre-fault functionality across different memory configurations.
--- .../selftests/kvm/pre_fault_memory_test.c | 51 +++++++++++++------ 1 file changed, 36 insertions(+), 15 deletions(-)
tools/testing/selftests/kvm/pre_fault_memory_test.c +36 -15
@@ -45,6 +45,7 @@ struct slot_worker_data {struct kvm_vm *vm;gpa_t gpa;u32 flags; +enum vm_mem_backing_src_type mem_backing_src;bool worker_ready;bool prefault_ready;bool recreate_slot;
@@ -65,14 +66,16 @@ static void *delete_slot_worker(void *__data)while (!READ_ONCE (data->recreate_slot))cpu_relax (); -vm_userspace_mem_region_add (vm,VM_MEM_SRC_ANONYMOUS , data->gpa, +vm_userspace_mem_region_add (vm, data->mem_backing_src, data->gpa,TEST_SLOT , test_config.test_num_pages, data->flags);return NULL ; }static void pre_fault_memory (struct kvm_vcpu *vcpu,u64 base_gpa,u64 offset, -u64 size,u64 expected_left,bool private) +u64 size,u64 expected_left, +enum vm_mem_backing_src_type mem_backing_src, +bool private) {struct kvm_pre_fault_memory range = { .gpa = base_gpa + offset,
@@ -83,6 +86,7 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset, .vm = vcpu->vm, .gpa = base_gpa, .flags = private ? KVM_MEM_GUEST_MEMFD : 0, + .mem_backing_src = mem_backing_src, };bool slot_recreated = false;pthread_t slot_worker;
@@ -172,11 +176,13 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset,struct test_params {unsigned long vm_type;bool private; +enum vm_mem_backing_src_type mem_backing_src; };static void __test_pre_fault_memory (enum vm_guest_mode guest_mode,void *arg) {gpa_t gpa, gva, alignment, guest_page_size, host_page_size; +gpa_t backing_src_pagesz, mem_page_size;struct test_params *p = arg;const struct vm_shape shape = { .mode = guest_mode,
@@ -188,24 +194,28 @@ static void __test_pre_fault_memory(enum vm_guest_mode guest_mode, void *arg)struct ucall uc;pr_info ("Testing guest mode: %s\n" ,vm_guest_mode_string (guest_mode)); +pr_info ("Testing memory backing src type: %s\n" , +vm_mem_backing_src_alias (p->mem_backing_src)->name); vm =vm_create_shape_with_one_vcpu (shape, &vcpu, guest_code); guest_page_size = vm_guest_mode_params[guest_mode].page_size; host_page_size =getpagesize (); + backing_src_pagesz =get_backing_src_pagesz (p->mem_backing_src); + mem_page_size =max (host_page_size, backing_src_pagesz); test_config.page_size = guest_page_size; test_config.test_size =align_up (TEST_BASE_SIZE + test_config.page_size, - host_page_size); + mem_page_size); test_config.test_num_pages =vm_calc_num_guest_pages (vm->mode, test_config.test_size); gpa = (vm->max_gfn - test_config.test_num_pages) * test_config.page_size; alignment =SZ_2M ; - alignment =max (alignment, host_page_size); + alignment =max (alignment, mem_page_size); gpa =align_down (gpa, alignment); gva = gpa & ((1ULL << (vm->va_bits -1 )) -1 ); -vm_userspace_mem_region_add (vm,VM_MEM_SRC_ANONYMOUS , +vm_userspace_mem_region_add (vm, p->mem_backing_src, gpa,TEST_SLOT , test_config.test_num_pages, p->private ?KVM_MEM_GUEST_MEMFD :0 );virt_map (vm, gva, gpa, test_config.test_num_pages);
@@ -213,14 +223,18 @@ static void __test_pre_fault_memory(enum vm_guest_mode guest_mode, void *arg)if (p->private)vm_mem_set_private (vm, gpa, test_config.test_size); -pre_fault_memory (vcpu, gpa,0 , test_config.test_size,0 , p->private); +pre_fault_memory (vcpu, gpa,0 , test_config.test_size,0 , + p->mem_backing_src, p->private);/* Retry the same range after the first prefault attempt. */ -pre_fault_memory (vcpu, gpa,0 , test_config.test_size,0 , p->private); +pre_fault_memory (vcpu, gpa,0 , test_config.test_size,0 , + p->mem_backing_src, p->private);pre_fault_memory (vcpu, gpa, test_config.test_size - host_page_size, - host_page_size *2 , host_page_size, p->private); + host_page_size *2 , host_page_size, + p->mem_backing_src, p->private);pre_fault_memory (vcpu, gpa, test_config.test_size, - host_page_size, host_page_size, p->private); + host_page_size, host_page_size, + p->mem_backing_src, p->private);vcpu_args_set (vcpu,1 , gva);
@@ -249,11 +263,13 @@ static void __test_pre_fault_memory(enum vm_guest_mode guest_mode, void *arg)kvm_vm_free (vm); } -static void test_pre_fault_memory (unsigned long vm_type,bool private) +static void test_pre_fault_memory (unsigned long vm_type,enum vm_mem_backing_src_type backing_src, +bool private) {struct test_params p = { .vm_type = vm_type, .private = private, + .mem_backing_src = backing_src, };if (vm_type && !(kvm_check_cap (KVM_CAP_VM_TYPES ) &BIT (vm_type))) {
@@ -267,23 +283,28 @@ static void test_pre_fault_memory(unsigned long vm_type, bool private)static void help (char *name) {puts ("" ); -printf ("usage: %s [-h] [-m mode]\n" , name); +printf ("usage: %s [-h] [-m mode] [-s mem-type]\n" , name);puts ("" );guest_modes_help (); +backing_src_help ("-s" );puts ("" ); }int main (int argc,char *argv[]) { +enum vm_mem_backing_src_type backing =DEFAULT_VM_MEM_SRC ;int opt;guest_modes_append_default (); -while ((opt =getopt (argc, argv,"hm:" )) !=-1 ) { +while ((opt =getopt (argc, argv,"hm:s:" )) !=-1 ) {switch (opt) {case 'm' :guest_modes_cmdline (optarg);break ; +case 's' : + backing =parse_backing_src_type (optarg); +break ;case 'h' :default :help (argv[0 ]);
@@ -293,10 +314,10 @@ int main(int argc, char *argv[])TEST_REQUIRE (kvm_check_cap (KVM_CAP_PRE_FAULT_MEMORY )); -test_pre_fault_memory (0 , false); +test_pre_fault_memory (0 , backing, false);#ifdef __x86_64__ -test_pre_fault_memory (KVM_X86_SW_PROTECTED_VM , false); -test_pre_fault_memory (KVM_X86_SW_PROTECTED_VM , true); +test_pre_fault_memory (KVM_X86_SW_PROTECTED_VM , backing, false); +test_pre_fault_memory (KVM_X86_SW_PROTECTED_VM , backing, true);#endif return 0 ; }
Signature · 1 line
2.43.0
PATCH v5 5/5 KVM: selftests: Add nested pre-fault test for arm64
In reply to Jack ThomsonFrom: Jack Thomson <[email protected]>
Add an arm64 nested-virt selftest for KVM_PRE_FAULT_MEMORY. The guest enters vEL1 and exits to userspace with a nested/shadow stage-2 MMU as the vCPU's last-run context.
Before prefaulting, userspace enables HCR_EL2.VM and points VTTBR_EL2 at an empty nested stage-2 root. A prefault implementation that incorrectly treats the userspace GPA as an L2 IPA will fail the ioctl; the correct path swaps to the canonical stage-2 and succeeds.
Restore the original nested state before resuming the guest, then touch the prefaulted range to check that vEL1 still runs correctly.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../kvm/arm64/nv_pre_fault_memory_test.c | 200 ++++++++++++++++++ 2 files changed, 201 insertions(+) create mode 100644 tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -170,6 +170,7 @@ TEST_GEN_PROGS_arm64 += arm64/debug-exceptions TEST_GEN_PROGS_arm64 += arm64/hello_el2 TEST_GEN_PROGS_arm64 += arm64/host_sve TEST_GEN_PROGS_arm64 += arm64/hypercalls +TEST_GEN_PROGS_arm64 += arm64/nv_pre_fault_memory_test TEST_GEN_PROGS_arm64 += arm64/external_aborts TEST_GEN_PROGS_arm64 += arm64/page_fault_test TEST_GEN_PROGS_arm64 += arm64/psci_test
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file +200 -0
200 lines not on this page · raw message
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v5 5/5] KVM: selftests: Add nested pre-fault test for arm64
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The `KVM_PRE_FAULT_MEMORY` ioctl loop in `pre_fault()` fails to handle partial success, causing…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The KVM_PRE_FAULT_MEMORY ioctl loop in pre_fault() fails to handle partial success, causing spurious test failures if the ioctl yields or is interrupted. --
commit ee1e827d8efbd103bf9ab1322ee50ba8c3895869
Author: Jack Thomson <[email protected]>
KVM: selftests: Add nested pre-fault test for arm64
This commit adds an arm64 nested-virt selftest for KVM_PRE_FAULT_MEMORY to verify that prefaulting correctly targets the canonical stage-2 when the guest's last-run context has a nested/shadow stage-2 MMU.
Quoted 5 lines
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file +0
-0
[ ... ]
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c line 99
Quoted 11 lines
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file +11
-0+static void pre_fault (struct kvm_vcpu *vcpu,u64 gpa,u64 size) +{ +struct kvm_pre_fault_memory range = { + .gpa = gpa, + .size = size, + }; +int ret; + +do { + ret =__vcpu_ioctl (vcpu,KVM_PRE_FAULT_MEMORY , &range); + }while (ret <0 && errno ==EINTR );
[Severity: Medium]
Does this loop handle partial successes correctly? If the ioctl is interrupted by a signal or yields after successfully mapping at least one page, it returns 0 and updates range.size with the remaining bytes to process.
If that happens, this loop will exit immediately and the test will spuriously fail the TEST_ASSERT_EQ(range.size, 0) assertion below. Should this loop continue as long as ret == 0 and range.size > 0?
Quoted 5 lines
tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file +5
-0+ +TEST_ASSERT (!ret,"KVM_PRE_FAULT_MEMORY failed, ret: %d errno: %d" , + ret, errno); +TEST_ASSERT_EQ (range.size,0 ); +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Jack ThomsonHi Jack,
I've been trying to review the patch, until I realised that I don't understand what KVM_PRE_FAULT_MEMORY is supposed to do.
Does it guarantee that, after it completes and returns to userspace, the address range [gpa, gpa + size] is mapped in the VM's stage 2 tables? Or does it guarantee that the address range was mapped at stage 2 at some point during the ioctl's execution, but parts, or all of it, might not be mapped after the ioctl completes and returns to userspace?
Thanks,
Alex
Quoted 307 lines · Jack Thomson
From: Jack Thomson <[email protected]>
Add arm64 support for
KVM_PRE_FAULT_MEMORYby synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl;
vcpu_put()at ioctl exit drops thehw_mmuand the nextvcpu_load()reselects the correct MMU from vCPU state.Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with
-EHWPOISONinstead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.Advertise
KVM_CAP_PRE_FAULT_MEMORYon arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns-EOPNOTSUPPif invoked anyway.Signed-off-by Jack Thomson <[email protected]>--- Documentation/virt/kvm/api.rst | 18 +++- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 162 +++++++++++++++++++++++++++++++++ 4 files changed, 178 insertions(+), 4 deletions(-)Documentation/virt/kvm/api.rst +14
-4@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. --------------------------- :Capability: KVM_CAP_PRE_FAULT_MEMORY -:Architectures: none +:Architectures: x86, arm64 :Type: vcpu ioctl :Parameters: struct kvm_pre_fault_memory (in/out) :Returns: 0 if at least one page is processed, < 0 on error@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. Errors: ========== =============================================================== + EAGAIN A memslot update raced with the ioctl before any page was + processed. EINVAL The specified `gpa` and `size` were invalid (e.g. not page aligned, causes an overflow, or size is zero). ENOENT The specified `gpa` is outside defined memslots. EINTR An unmasked signal is pending and no page was processed. EFAULT The parameter address was invalid. + EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the hypervisor, and/or for the current vCPU state/mode. EIO unexpected error conditions (also causes a WARN)@@ -6494,7 +6497,14 @@ Errors: KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory for the current vCPU state. KVM maps memory as if the vCPU generated a stage-2 read page fault, e.g. faults in memory as needed, but doesn't break -CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed. +CoW. However, on x86, KVM does not mark any newly created stage-2 PTE as +Accessed. On arm64, newly created stage-2 PTEs are marked Accessed. + +On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's +memslot address space. If the vCPU most recently ran a nested guest, KVM +still targets the VM's canonical stage-2, and does not interpret `gpa` as +a nested guest IPA or target the nested/shadow stage-2 selected by the +vCPU's last run state. In the case of confidential VM types where there is an initial set up of private guest memory before the guest is 'finalized'/measured, this ioctl@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel. When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue -the ioctl again with the same `struct kvm_map_memory` argument. +the ioctl again with the same `struct kvm_pre_fault_memory` argument. -Shadow page tables cannot support this ioctl because they +On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address. Calling this ioctl when the guest is using shadow page tables (for example because it is running a nested guest with nested page tables)arch/arm64/kvm/Kconfig +1
-0@@ -24,6 +24,7 @@ menuconfig KVM select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT + select KVM_GENERIC_PRE_FAULT_MEMORY select VIRT_XFER_TO_GUEST_WORK select KVM_VFIO select HAVE_KVM_DIRTY_RING_ACQ_RELarch/arm64/kvm/arm.c +1
-0@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_COUNTER_OFFSET :case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS :case KVM_CAP_ARM_SEA_TO_USER : +case KVM_CAP_PRE_FAULT_MEMORY : r =1 ;break ;case KVM_CAP_SET_GUEST_DEBUG2 :arch/arm64/kvm/mmu.c +162
-0@@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {struct kvm_s2_trans *nested;struct kvm_memory_slot *memslot;unsigned long hva; +unsigned long *page_size; +bool prefault; };static int gmem_abort (const struct kvm_s2_fault_desc *s2fd)@@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd, &s2vi->map_writable, &s2vi->page);if (unlikely (is_error_noslot_pfn (s2vi->pfn))) {if (s2vi->pfn ==KVM_PFN_ERR_HWPOISON ) { +/* +* When prefaulting, report the poison via -EHWPOISON +* only; don't also queue a SIGBUS as the run path +* does for the faulting vCPU thread. +*/ +if (s2fd->prefault) +return -EHWPOISON ;kvm_send_hwpoison_signal (s2fd->hva,__ffs (s2vi->vma_pagesize));return 0 ; }@@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,kvm_release_faultin_page (kvm, s2vi->page, !!ret, writable);kvm_fault_unlock (kvm); +if (s2fd->page_size && !ret) + *s2fd->page_size = mapping_size; +/* * Mark the page dirty only if the fault is handled successfully, * making sure we adjust the canonical IPA if the mapping size has @@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)trace_kvm_toggle_cache (*vcpu_pc (vcpu), was_enabled, now_enabled); } + +/* +* Prefaulting always targets the canonical stage-2. If the vCPU last ran +* in a nested context, swap in the canonical MMU via the vCPU put/load +* helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount +* state stay consistent. +* +* The swap is deliberately not undone: nothing runs in between the +* per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the +* generic prefault loop, and the vcpu_put() at ioctl exit discards +* vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next +* vcpu_load() re-derives the correct MMU from the vCPU's context. If the +* prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu() +* keeps the canonical MMU in place for the reload. Leaving the swap in +* place also bounds the cost to at most one put/load pair per ioctl, +* rather than two pairs per prefaulted page. +*/ +static void kvm_pre_fault_load_canonical_mmu (struct kvm_vcpu *vcpu) +{ +if (!vcpu_has_nv (vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu) +return ; + +preempt_disable (); +kvm_arch_vcpu_put (vcpu); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; +kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +preempt_enable (); +} + +long kvm_arch_vcpu_pre_fault_memory (struct kvm_vcpu *vcpu, +struct kvm_pre_fault_memory *range) +{ +struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault; +struct kvm_vcpu_fault_info fault_backup = *fault_info; +s8 walk_level =KVM_PGTABLE_LAST_LEVEL ; +unsigned long page_size =PAGE_SIZE ; +struct kvm_memory_slot *memslot; +phys_addr_t gpa = range->gpa; +struct kvm_pgtable *pgt; +phys_addr_t end; +kvm_pte_t pte; +hva_t hva; +gfn_t gfn; +long ret; + +if (vcpu_is_protected (vcpu)) +return -EOPNOTSUPP ; + +/* +* Interpret range->gpa in the userspace-owned VM's IPA space, not in +* any nested guest IPA space that may have been active on the vCPU's +* last run. Always target the canonical stage-2. +*/ +kvm_pre_fault_load_canonical_mmu (vcpu); + +if (gpa >=kvm_phys_size (vcpu->arch.hw_mmu)) { + ret = -ENOENT ; + goto out; + } + + gfn =gpa_to_gfn (gpa); + memslot =gfn_to_memslot (vcpu->kvm, gfn); +if (!memslot) { + ret = -ENOENT ; + goto out; + } + +/* +* A racing memslot deletion or move installs an invalid slot before +* zapping stage-2. Ask userspace to retry once the update settles. +*/ +if (memslot->flags &KVM_MEMSLOT_INVALID ) { + ret = -EAGAIN ; + goto out; + } + +/* +* pKVM stage-2 mappings aren't directly walkable from the host; let +* the fault path handle both new and existing mappings. +*/ +if (!is_protected_kvm_enabled ()) { + pgt = vcpu->arch.hw_mmu->pgt; + scoped_guard(read_lock, &vcpu->kvm->mmu_lock) { + ret =kvm_pgtable_get_leaf (pgt, gpa, &pte, &walk_level, +KVM_PGTABLE_WALK_SHARED ); + } +if (ret) + goto out; + +if (kvm_pte_valid (pte)) { + page_size =kvm_granule_size (walk_level); +if (!(pte &KVM_PTE_LEAF_ATTR_LO_S2_AF )) +handle_access_fault (vcpu, gpa); + goto out_success; + } + } + +/* +* Synthesize a read translation fault for the canonical IPA, at the +* level where the stage-2 walk currently ends (the last level under +* pKVM, where stage-2 isn't walkable from the host). +*/ + fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) | + ESR_ELx_IL |ESR_ELx_FSC_FAULT_L (walk_level); + fault_info->hpfar_el2 =HPFAR_EL2_NS | +FIELD_PREP (HPFAR_EL2_FIPA , gpa >>12 ); + +struct kvm_s2_fault_desc s2fd = { + .vcpu = vcpu, + .fault_ipa = gpa, + .nested =NULL , + .memslot = memslot, + .page_size = &page_size, + .prefault = true, + }; + +/* +* As in the run path, -EAGAIN from the abort handlers is treated as +* progress: either a parallel fault installed the mapping, or a racing +* invalidation is in flight and the next access will refault. +*/ +if (kvm_slot_has_gmem (memslot)) { + ret =gmem_abort (&s2fd); + }else { + hva =gfn_to_hva_memslot_prot (memslot, gfn,NULL ); +if (kvm_is_error_hva (hva)) { + ret = -EFAULT ; + goto out; + } + + s2fd.hva = hva; + ret =user_mem_abort (&s2fd); + } + +if (ret <0 ) + goto out; + +out_success: + end =ALIGN_DOWN (gpa, page_size) + page_size; + ret =min_t (u64, range->size, end - gpa); +out: +/* +* Restore the synthetic fault state so a subsequent KVM_RUN does not +* observe it. kvm_handle_mmio_return() runs before guest entry can +* refresh fault.esr_el2 from hardware, so leaving the synthetic ESR +* in place would corrupt the completion of a pending MMIO exit. +*/ + *fault_info = fault_backup; +return ret; +}Signature · 1 line
2.43.0
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
Hi,
Quoted 10 lines · Alexandru Elisei
Hi Jack,
I've been trying to review the patch, until I realised that I don't understand what
KVM_PRE_FAULT_MEMORYis supposed to do.Does it guarantee that, after it completes and returns to userspace, the address range [gpa, gpa + size] is mapped in the VM's stage 2 tables? Or does it guarantee that the address range was mapped at stage 2 at some point during the ioctl's execution, but parts, or all of it, might not be mapped after the ioctl completes and returns to userspace?
Nevermind, spend some time thinking about it and it turns out it's impossible for KVM to guarantee that the memory is still mapped after the ioctls returns. So I answered my own question, thanks.
Alex
Quoted 313 lines
Thanks,
AlexQuoted 307 lines · Jack Thomson
From: Jack Thomson <[email protected]>
Add arm64 support for
KVM_PRE_FAULT_MEMORYby synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl;
vcpu_put()at ioctl exit drops thehw_mmuand the nextvcpu_load()reselects the correct MMU from vCPU state.Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with
-EHWPOISONinstead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.Advertise
KVM_CAP_PRE_FAULT_MEMORYon arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns-EOPNOTSUPPif invoked anyway.Signed-off-by Jack Thomson <[email protected]>--- Documentation/virt/kvm/api.rst | 18 +++- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 162 +++++++++++++++++++++++++++++++++ 4 files changed, 178 insertions(+), 4 deletions(-)Documentation/virt/kvm/api.rst +14
-4@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. --------------------------- :Capability: KVM_CAP_PRE_FAULT_MEMORY -:Architectures: none +:Architectures: x86, arm64 :Type: vcpu ioctl :Parameters: struct kvm_pre_fault_memory (in/out) :Returns: 0 if at least one page is processed, < 0 on error@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. Errors: ========== =============================================================== + EAGAIN A memslot update raced with the ioctl before any page was + processed. EINVAL The specified `gpa` and `size` were invalid (e.g. not page aligned, causes an overflow, or size is zero). ENOENT The specified `gpa` is outside defined memslots. EINTR An unmasked signal is pending and no page was processed. EFAULT The parameter address was invalid. + EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the hypervisor, and/or for the current vCPU state/mode. EIO unexpected error conditions (also causes a WARN)@@ -6494,7 +6497,14 @@ Errors: KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory for the current vCPU state. KVM maps memory as if the vCPU generated a stage-2 read page fault, e.g. faults in memory as needed, but doesn't break -CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed. +CoW. However, on x86, KVM does not mark any newly created stage-2 PTE as +Accessed. On arm64, newly created stage-2 PTEs are marked Accessed. + +On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's +memslot address space. If the vCPU most recently ran a nested guest, KVM +still targets the VM's canonical stage-2, and does not interpret `gpa` as +a nested guest IPA or target the nested/shadow stage-2 selected by the +vCPU's last run state. In the case of confidential VM types where there is an initial set up of private guest memory before the guest is 'finalized'/measured, this ioctl@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel. When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue -the ioctl again with the same `struct kvm_map_memory` argument. +the ioctl again with the same `struct kvm_pre_fault_memory` argument. -Shadow page tables cannot support this ioctl because they +On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address. Calling this ioctl when the guest is using shadow page tables (for example because it is running a nested guest with nested page tables)arch/arm64/kvm/Kconfig +1
-0@@ -24,6 +24,7 @@ menuconfig KVM select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT + select KVM_GENERIC_PRE_FAULT_MEMORY select VIRT_XFER_TO_GUEST_WORK select KVM_VFIO select HAVE_KVM_DIRTY_RING_ACQ_RELarch/arm64/kvm/arm.c +1
-0@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_COUNTER_OFFSET :case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS :case KVM_CAP_ARM_SEA_TO_USER : +case KVM_CAP_PRE_FAULT_MEMORY : r =1 ;break ;case KVM_CAP_SET_GUEST_DEBUG2 :arch/arm64/kvm/mmu.c +162
-0@@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {struct kvm_s2_trans *nested;struct kvm_memory_slot *memslot;unsigned long hva; +unsigned long *page_size; +bool prefault; };static int gmem_abort (const struct kvm_s2_fault_desc *s2fd)@@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd, &s2vi->map_writable, &s2vi->page);if (unlikely (is_error_noslot_pfn (s2vi->pfn))) {if (s2vi->pfn ==KVM_PFN_ERR_HWPOISON ) { +/* +* When prefaulting, report the poison via -EHWPOISON +* only; don't also queue a SIGBUS as the run path +* does for the faulting vCPU thread. +*/ +if (s2fd->prefault) +return -EHWPOISON ;kvm_send_hwpoison_signal (s2fd->hva,__ffs (s2vi->vma_pagesize));return 0 ; }@@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,kvm_release_faultin_page (kvm, s2vi->page, !!ret, writable);kvm_fault_unlock (kvm); +if (s2fd->page_size && !ret) + *s2fd->page_size = mapping_size; +/* * Mark the page dirty only if the fault is handled successfully, * making sure we adjust the canonical IPA if the mapping size has @@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)trace_kvm_toggle_cache (*vcpu_pc (vcpu), was_enabled, now_enabled); } + +/* +* Prefaulting always targets the canonical stage-2. If the vCPU last ran +* in a nested context, swap in the canonical MMU via the vCPU put/load +* helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount +* state stay consistent. +* +* The swap is deliberately not undone: nothing runs in between the +* per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the +* generic prefault loop, and the vcpu_put() at ioctl exit discards +* vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next +* vcpu_load() re-derives the correct MMU from the vCPU's context. If the +* prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu() +* keeps the canonical MMU in place for the reload. Leaving the swap in +* place also bounds the cost to at most one put/load pair per ioctl, +* rather than two pairs per prefaulted page. +*/ +static void kvm_pre_fault_load_canonical_mmu (struct kvm_vcpu *vcpu) +{ +if (!vcpu_has_nv (vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu) +return ; + +preempt_disable (); +kvm_arch_vcpu_put (vcpu); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; +kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +preempt_enable (); +} + +long kvm_arch_vcpu_pre_fault_memory (struct kvm_vcpu *vcpu, +struct kvm_pre_fault_memory *range) +{ +struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault; +struct kvm_vcpu_fault_info fault_backup = *fault_info; +s8 walk_level =KVM_PGTABLE_LAST_LEVEL ; +unsigned long page_size =PAGE_SIZE ; +struct kvm_memory_slot *memslot; +phys_addr_t gpa = range->gpa; +struct kvm_pgtable *pgt; +phys_addr_t end; +kvm_pte_t pte; +hva_t hva; +gfn_t gfn; +long ret; + +if (vcpu_is_protected (vcpu)) +return -EOPNOTSUPP ; + +/* +* Interpret range->gpa in the userspace-owned VM's IPA space, not in +* any nested guest IPA space that may have been active on the vCPU's +* last run. Always target the canonical stage-2. +*/ +kvm_pre_fault_load_canonical_mmu (vcpu); + +if (gpa >=kvm_phys_size (vcpu->arch.hw_mmu)) { + ret = -ENOENT ; + goto out; + } + + gfn =gpa_to_gfn (gpa); + memslot =gfn_to_memslot (vcpu->kvm, gfn); +if (!memslot) { + ret = -ENOENT ; + goto out; + } + +/* +* A racing memslot deletion or move installs an invalid slot before +* zapping stage-2. Ask userspace to retry once the update settles. +*/ +if (memslot->flags &KVM_MEMSLOT_INVALID ) { + ret = -EAGAIN ; + goto out; + } + +/* +* pKVM stage-2 mappings aren't directly walkable from the host; let +* the fault path handle both new and existing mappings. +*/ +if (!is_protected_kvm_enabled ()) { + pgt = vcpu->arch.hw_mmu->pgt; + scoped_guard(read_lock, &vcpu->kvm->mmu_lock) { + ret =kvm_pgtable_get_leaf (pgt, gpa, &pte, &walk_level, +KVM_PGTABLE_WALK_SHARED ); + } +if (ret) + goto out; + +if (kvm_pte_valid (pte)) { + page_size =kvm_granule_size (walk_level); +if (!(pte &KVM_PTE_LEAF_ATTR_LO_S2_AF )) +handle_access_fault (vcpu, gpa); + goto out_success; + } + } + +/* +* Synthesize a read translation fault for the canonical IPA, at the +* level where the stage-2 walk currently ends (the last level under +* pKVM, where stage-2 isn't walkable from the host). +*/ + fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) | + ESR_ELx_IL |ESR_ELx_FSC_FAULT_L (walk_level); + fault_info->hpfar_el2 =HPFAR_EL2_NS | +FIELD_PREP (HPFAR_EL2_FIPA , gpa >>12 ); + +struct kvm_s2_fault_desc s2fd = { + .vcpu = vcpu, + .fault_ipa = gpa, + .nested =NULL , + .memslot = memslot, + .page_size = &page_size, + .prefault = true, + }; + +/* +* As in the run path, -EAGAIN from the abort handlers is treated as +* progress: either a parallel fault installed the mapping, or a racing +* invalidation is in flight and the next access will refault. +*/ +if (kvm_slot_has_gmem (memslot)) { + ret =gmem_abort (&s2fd); + }else { + hva =gfn_to_hva_memslot_prot (memslot, gfn,NULL ); +if (kvm_is_error_hva (hva)) { + ret = -EFAULT ; + goto out; + } + + s2fd.hva = hva; + ret =user_mem_abort (&s2fd); + } + +if (ret <0 ) + goto out; + +out_success: + end =ALIGN_DOWN (gpa, page_size) + page_size; + ret =min_t (u64, range->size, end - gpa); +out: +/* +* Restore the synthetic fault state so a subsequent KVM_RUN does not +* observe it. kvm_handle_mmio_return() runs before guest entry can +* refresh fault.esr_el2 from hardware, so leaving the synthetic ESR +* in place would corrupt the completion of a pending MMIO exit. +*/ + *fault_info = fault_backup; +return ret; +}Signature · 1 line
2.43.0
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Jack ThomsonHi Jack,
Quoted 117 lines · Jack Thomson
From: Jack Thomson <[email protected]>
Add arm64 support for
KVM_PRE_FAULT_MEMORYby synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl;
vcpu_put()at ioctl exit drops thehw_mmuand the nextvcpu_load()reselects the correct MMU from vCPU state.Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with
-EHWPOISONinstead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.Advertise
KVM_CAP_PRE_FAULT_MEMORYon arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns-EOPNOTSUPPif invoked anyway.Signed-off-by Jack Thomson <[email protected]>--- Documentation/virt/kvm/api.rst | 18 +++- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 162 +++++++++++++++++++++++++++++++++ 4 files changed, 178 insertions(+), 4 deletions(-)Documentation/virt/kvm/api.rst +14
-4@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. --------------------------- :Capability: KVM_CAP_PRE_FAULT_MEMORY -:Architectures: none +:Architectures: x86, arm64 :Type: vcpu ioctl :Parameters: struct kvm_pre_fault_memory (in/out) :Returns: 0 if at least one page is processed, < 0 on error@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. Errors: ========== =============================================================== + EAGAIN A memslot update raced with the ioctl before any page was + processed. EINVAL The specified `gpa` and `size` were invalid (e.g. not page aligned, causes an overflow, or size is zero). ENOENT The specified `gpa` is outside defined memslots. EINTR An unmasked signal is pending and no page was processed. EFAULT The parameter address was invalid. + EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the hypervisor, and/or for the current vCPU state/mode. EIO unexpected error conditions (also causes a WARN)@@ -6494,7 +6497,14 @@ Errors: KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory for the current vCPU state. KVM maps memory as if the vCPU generated a stage-2 read page fault, e.g. faults in memory as needed, but doesn't break -CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed. +CoW. However, on x86, KVM does not mark any newly created stage-2 PTE as +Accessed. On arm64, newly created stage-2 PTEs are marked Accessed. + +On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's +memslot address space. If the vCPU most recently ran a nested guest, KVM +still targets the VM's canonical stage-2, and does not interpret `gpa` as +a nested guest IPA or target the nested/shadow stage-2 selected by the +vCPU's last run state. In the case of confidential VM types where there is an initial set up of private guest memory before the guest is 'finalized'/measured, this ioctl@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel. When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue -the ioctl again with the same `struct kvm_map_memory` argument. +the ioctl again with the same `struct kvm_pre_fault_memory` argument. -Shadow page tables cannot support this ioctl because they +On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address. Calling this ioctl when the guest is using shadow page tables (for example because it is running a nested guest with nested page tables)arch/arm64/kvm/Kconfig +1
-0@@ -24,6 +24,7 @@ menuconfig KVM select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT + select KVM_GENERIC_PRE_FAULT_MEMORY select VIRT_XFER_TO_GUEST_WORK select KVM_VFIO select HAVE_KVM_DIRTY_RING_ACQ_RELarch/arm64/kvm/arm.c +1
-0@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_COUNTER_OFFSET :case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS :case KVM_CAP_ARM_SEA_TO_USER : +case KVM_CAP_PRE_FAULT_MEMORY : r =1 ;break ;case KVM_CAP_SET_GUEST_DEBUG2 :arch/arm64/kvm/mmu.c +0
-0
arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574
Quoted 4 lines
arch/arm64/kvm/mmu.c +1
-0@@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {struct kvm_s2_trans *nested;struct kvm_memory_slot *memslot;unsigned long hva; +unsigned long *page_size;
As far as I know, fault handling was reworked to use struct kvm_s2_fault_desc to store the fault information that user_mem_abort() needs to handle the fault. Adding a 'page_size' field, that represents the result of the gpa mapping process, might not be desirable.
Quoted 56 lines
arch/arm64/kvm/mmu.c +35
-0+bool prefault; };static int gmem_abort (const struct kvm_s2_fault_desc *s2fd)@@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd, &s2vi->map_writable, &s2vi->page);if (unlikely (is_error_noslot_pfn (s2vi->pfn))) {if (s2vi->pfn ==KVM_PFN_ERR_HWPOISON ) { +/* +* When prefaulting, report the poison via -EHWPOISON +* only; don't also queue a SIGBUS as the run path +* does for the faulting vCPU thread. +*/ +if (s2fd->prefault) +return -EHWPOISON ;kvm_send_hwpoison_signal (s2fd->hva,__ffs (s2vi->vma_pagesize));return 0 ; }@@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,kvm_release_faultin_page (kvm, s2vi->page, !!ret, writable);kvm_fault_unlock (kvm); +if (s2fd->page_size && !ret) + *s2fd->page_size = mapping_size; +/* * Mark the page dirty only if the fault is handled successfully, * making sure we adjust the canonical IPA if the mapping size has @@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)trace_kvm_toggle_cache (*vcpu_pc (vcpu), was_enabled, now_enabled); } + +/* +* Prefaulting always targets the canonical stage-2. If the vCPU last ran +* in a nested context, swap in the canonical MMU via the vCPU put/load +* helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount +* state stay consistent. +* +* The swap is deliberately not undone: nothing runs in between the +* per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the +* generic prefault loop, and the vcpu_put() at ioctl exit discards +* vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next +* vcpu_load() re-derives the correct MMU from the vCPU's context. If the +* prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu() +* keeps the canonical MMU in place for the reload. Leaving the swap in +* place also bounds the cost to at most one put/load pair per ioctl, +* rather than two pairs per prefaulted page. +*/ +static void kvm_pre_fault_load_canonical_mmu (struct kvm_vcpu *vcpu) +{ +if (!vcpu_has_nv (vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu) +return ; + +preempt_disable (); +kvm_arch_vcpu_put (vcpu);
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2805
Quoted 10 lines
arch/arm64/kvm/mmu.c +10
-0+ vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; +kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +preempt_enable (); +} + +long kvm_arch_vcpu_pre_fault_memory (struct kvm_vcpu *vcpu, +struct kvm_pre_fault_memory *range) +{ +struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault; +struct kvm_vcpu_fault_info fault_backup = *fault_info;
I'm not sure you need to make a backup here. vcpu->arch.fault is populated each time the CPU takes a fault.
Quoted 28 lines
arch/arm64/kvm/mmu.c +28
-0+s8 walk_level =KVM_PGTABLE_LAST_LEVEL ; +unsigned long page_size =PAGE_SIZE ; +struct kvm_memory_slot *memslot; +phys_addr_t gpa = range->gpa; +struct kvm_pgtable *pgt; +phys_addr_t end; +kvm_pte_t pte; +hva_t hva; +gfn_t gfn; +long ret; + +if (vcpu_is_protected (vcpu)) +return -EOPNOTSUPP ; + +/* +* Interpret range->gpa in the userspace-owned VM's IPA space, not in +* any nested guest IPA space that may have been active on the vCPU's +* last run. Always target the canonical stage-2. +*/ +kvm_pre_fault_load_canonical_mmu (vcpu); + +if (gpa >=kvm_phys_size (vcpu->arch.hw_mmu)) { + ret = -ENOENT ; + goto out; + } + + gfn =gpa_to_gfn (gpa); + memslot =gfn_to_memslot (vcpu->kvm, gfn);
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2843
Quoted 10 lines
arch/arm64/kvm/mmu.c +10
-0+if (!memslot) { + ret = -ENOENT ; + goto out; + } + +/* +* A racing memslot deletion or move installs an invalid slot before +* zapping stage-2. Ask userspace to retry once the update settles. +*/ +if (memslot->flags &KVM_MEMSLOT_INVALID ) {
I don't think that's something we should care about, the flag can be set immediately after the check as the function doesn't take kvm->slots_lock. kvm_vcpu_prefault_memory() takes the srcu lock in read mode, so the function is safe to run even if userspace does something silly like deleting a memslot at the same time that it's prefaulting the guest memory it represents.
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2856
Quoted 13 lines
arch/arm64/kvm/mmu.c +13
-0+ ret = -EAGAIN ; + goto out; + } + +/* +* pKVM stage-2 mappings aren't directly walkable from the host; let +* the fault path handle both new and existing mappings. +*/ +if (!is_protected_kvm_enabled ()) { + pgt = vcpu->arch.hw_mmu->pgt; + scoped_guard(read_lock, &vcpu->kvm->mmu_lock) { + ret =kvm_pgtable_get_leaf (pgt, gpa, &pte, &walk_level, +KVM_PGTABLE_WALK_SHARED );
It might not be obvious, but taking kvm->mmu_lock in read mode does not guarantee that gpa will be mapped when kvm_pgtable_get_leaf() returns. That's because a concurrent kvm_pgtable_stage2_map() for a different gpa can destroy the mapping of the current gpa.
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2857
Quoted 1 line
arch/arm64/kvm/mmu.c +1
-0+ }
The kvm->mmu_lock is dropped here, which means that it is possible for a MMU notifier callback to have just unmapped the entire stage 2 for the VM.
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2864
Quoted 7 lines
arch/arm64/kvm/mmu.c +7
-0+if (ret) + goto out; + +if (kvm_pte_valid (pte)) { + page_size =kvm_granule_size (walk_level); +if (!(pte &KVM_PTE_LEAF_ATTR_LO_S2_AF )) +handle_access_fault (vcpu, gpa);
handle_access_fault() will fail if the mapping is gone. But I guess that's fine if the ioctl does not guarantee that memory is still mapped after it completes.
Also, the documentation that this patch adds says: 'On arm64, newly created stage-2 PTEs are marked Accessed'. Does not say anything about marking **existing** ptes as accessed. Would be useful to explain the code does it.
Quoted 22 lines
arch/arm64/kvm/mmu.c +22
-0+ goto out_success; + } + } + +/* +* Synthesize a read translation fault for the canonical IPA, at the +* level where the stage-2 walk currently ends (the last level under +* pKVM, where stage-2 isn't walkable from the host). +*/ + fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) | + ESR_ELx_IL |ESR_ELx_FSC_FAULT_L (walk_level); + fault_info->hpfar_el2 =HPFAR_EL2_NS | +FIELD_PREP (HPFAR_EL2_FIPA , gpa >>12 ); + +struct kvm_s2_fault_desc s2fd = { + .vcpu = vcpu, + .fault_ipa = gpa, + .nested =NULL , + .memslot = memslot, + .page_size = &page_size, + .prefault = true, + };
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2896
Quoted 10 lines
arch/arm64/kvm/mmu.c +10
-0+ +/* +* As in the run path, -EAGAIN from the abort handlers is treated as +* progress: either a parallel fault installed the mapping, or a racing +* invalidation is in flight and the next access will refault. +*/ +if (kvm_slot_has_gmem (memslot)) { + ret =gmem_abort (&s2fd); + }else { + hva =gfn_to_hva_memslot_prot (memslot, gfn,NULL );
There's gfn_to_hva_memslot(memslot, gfn), is that what you are looking for?
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2903
Quoted 7 lines
arch/arm64/kvm/mmu.c +7
-0+if (kvm_is_error_hva (hva)) { + ret = -EFAULT ; + goto out; + } + + s2fd.hva = hva; + ret =user_mem_abort (&s2fd);
If is_protected_kvm_enabled(), this function doesn't check if the mapping already exists. user_mem_abort() will call **p**kvm_pgtable_stage2_map() (emphasis is mine) to create the mapping, and pkvm_pgtable_stage2_map() will return -EAGAIN, and keep returning -EAGAIN until the mapping is destroyed, which might be some time away.
Same thing happens **without** pKVM, kvm_pgtable_stage2_map() will return -EAGAIN if the mapping exists. But that's not such a big issue, when the user calls KVM_PRE_FAULT_MEMORY on the remaining range kvm_pgtable_get_leaf() will detect the existing mapping. Which you have already taken into consideration judging by the comment, so all good here.
Thanks,
Alex
Quoted 22 lines
arch/arm64/kvm/mmu.c +18
-0+ } + +if (ret <0 ) + goto out; + +out_success: + end =ALIGN_DOWN (gpa, page_size) + page_size; + ret =min_t (u64, range->size, end - gpa); +out: +/* +* Restore the synthetic fault state so a subsequent KVM_RUN does not +* observe it. kvm_handle_mmio_return() runs before guest entry can +* refresh fault.esr_el2 from hardware, so leaving the synthetic ESR +* in place would corrupt the completion of a pending MMIO exit. +*/ + *fault_info = fault_backup; +return ret; +}Signature · 1 line
2.43.0
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Jack ThomsonHi,
Quoted 117 lines · Jack Thomson
From: Jack Thomson <[email protected]>
Add arm64 support for
KVM_PRE_FAULT_MEMORYby synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected.If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl;
vcpu_put()at ioctl exit drops thehw_mmuand the nextvcpu_load()reselects the correct MMU from vCPU state.Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with
-EHWPOISONinstead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range.Advertise
KVM_CAP_PRE_FAULT_MEMORYon arm64. Protected VMs remain
unsupported: pKVM filters the capability, and the ioctl returns-EOPNOTSUPPif invoked anyway.Signed-off-by Jack Thomson <[email protected]>--- Documentation/virt/kvm/api.rst | 18 +++- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 162 +++++++++++++++++++++++++++++++++ 4 files changed, 178 insertions(+), 4 deletions(-)Documentation/virt/kvm/api.rst +14
-4@@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. --------------------------- :Capability: KVM_CAP_PRE_FAULT_MEMORY -:Architectures: none +:Architectures: x86, arm64 :Type: vcpu ioctl :Parameters: struct kvm_pre_fault_memory (in/out) :Returns: 0 if at least one page is processed, < 0 on error@@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. Errors: ========== =============================================================== + EAGAIN A memslot update raced with the ioctl before any page was + processed. EINVAL The specified `gpa` and `size` were invalid (e.g. not page aligned, causes an overflow, or size is zero). ENOENT The specified `gpa` is outside defined memslots. EINTR An unmasked signal is pending and no page was processed. EFAULT The parameter address was invalid. + EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the hypervisor, and/or for the current vCPU state/mode. EIO unexpected error conditions (also causes a WARN)@@ -6494,7 +6497,14 @@ Errors: KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory for the current vCPU state. KVM maps memory as if the vCPU generated a stage-2 read page fault, e.g. faults in memory as needed, but doesn't break -CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed. +CoW. However, on x86, KVM does not mark any newly created stage-2 PTE as +Accessed. On arm64, newly created stage-2 PTEs are marked Accessed. + +On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's +memslot address space. If the vCPU most recently ran a nested guest, KVM +still targets the VM's canonical stage-2, and does not interpret `gpa` as +a nested guest IPA or target the nested/shadow stage-2 selected by the +vCPU's last run state. In the case of confidential VM types where there is an initial set up of private guest memory before the guest is 'finalized'/measured, this ioctl@@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel. When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue -the ioctl again with the same `struct kvm_map_memory` argument. +the ioctl again with the same `struct kvm_pre_fault_memory` argument. -Shadow page tables cannot support this ioctl because they +On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address. Calling this ioctl when the guest is using shadow page tables (for example because it is running a nested guest with nested page tables)arch/arm64/kvm/Kconfig +1
-0@@ -24,6 +24,7 @@ menuconfig KVM select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT + select KVM_GENERIC_PRE_FAULT_MEMORY select VIRT_XFER_TO_GUEST_WORK select KVM_VFIO select HAVE_KVM_DIRTY_RING_ACQ_RELarch/arm64/kvm/arm.c +1
-0@@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_COUNTER_OFFSET :case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS :case KVM_CAP_ARM_SEA_TO_USER : +case KVM_CAP_PRE_FAULT_MEMORY : r =1 ;break ;case KVM_CAP_SET_GUEST_DEBUG2 :arch/arm64/kvm/mmu.c +0
-0
arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574
Quoted 4 lines
arch/arm64/kvm/mmu.c +1
-0@@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {struct kvm_s2_trans *nested;struct kvm_memory_slot *memslot;unsigned long hva; +unsigned long *page_size;
It looks a bit odd in that struct.
Also, transparent_hugepage_adjust() modifies pfn and gfn. So perhaps it'd be better to introduce a struct kvm_s2_fault_mapping that can be used to get what has actually been mapped in kvm_s2_fault_map()?
Quoted 69 lines
arch/arm64/kvm/mmu.c +48
-0+bool prefault; };static int gmem_abort (const struct kvm_s2_fault_desc *s2fd)@@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd, &s2vi->map_writable, &s2vi->page);if (unlikely (is_error_noslot_pfn (s2vi->pfn))) {if (s2vi->pfn ==KVM_PFN_ERR_HWPOISON ) { +/* +* When prefaulting, report the poison via -EHWPOISON +* only; don't also queue a SIGBUS as the run path +* does for the faulting vCPU thread. +*/ +if (s2fd->prefault) +return -EHWPOISON ;kvm_send_hwpoison_signal (s2fd->hva,__ffs (s2vi->vma_pagesize));return 0 ; }@@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,kvm_release_faultin_page (kvm, s2vi->page, !!ret, writable);kvm_fault_unlock (kvm); +if (s2fd->page_size && !ret) + *s2fd->page_size = mapping_size; +/* * Mark the page dirty only if the fault is handled successfully, * making sure we adjust the canonical IPA if the mapping size has @@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)trace_kvm_toggle_cache (*vcpu_pc (vcpu), was_enabled, now_enabled); } + +/* +* Prefaulting always targets the canonical stage-2. If the vCPU last ran +* in a nested context, swap in the canonical MMU via the vCPU put/load +* helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount +* state stay consistent. +* +* The swap is deliberately not undone: nothing runs in between the +* per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the +* generic prefault loop, and the vcpu_put() at ioctl exit discards +* vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next +* vcpu_load() re-derives the correct MMU from the vCPU's context. If the +* prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu() +* keeps the canonical MMU in place for the reload. Leaving the swap in +* place also bounds the cost to at most one put/load pair per ioctl, +* rather than two pairs per prefaulted page. +*/ +static void kvm_pre_fault_load_canonical_mmu (struct kvm_vcpu *vcpu) +{ +if (!vcpu_has_nv (vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu) +return ; + +preempt_disable (); +kvm_arch_vcpu_put (vcpu); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; +kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +preempt_enable (); +} + +long kvm_arch_vcpu_pre_fault_memory (struct kvm_vcpu *vcpu, +struct kvm_pre_fault_memory *range) +{ +struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault; +struct kvm_vcpu_fault_info fault_backup = *fault_info; +s8 walk_level =KVM_PGTABLE_LAST_LEVEL ; +unsigned long page_size =PAGE_SIZE ; +struct kvm_memory_slot *memslot;
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2818
Quoted 10 lines
arch/arm64/kvm/mmu.c +10
-0+phys_addr_t gpa = range->gpa; +struct kvm_pgtable *pgt; +phys_addr_t end; +kvm_pte_t pte; +hva_t hva; +gfn_t gfn; +long ret; + +if (vcpu_is_protected (vcpu)) +return -EOPNOTSUPP ;
It could just call pkvm_mem_abort() when the VM is protected?
Quoted 23 lines
arch/arm64/kvm/mmu.c +23
-0+ +/* +* Interpret range->gpa in the userspace-owned VM's IPA space, not in +* any nested guest IPA space that may have been active on the vCPU's +* last run. Always target the canonical stage-2. +*/ +kvm_pre_fault_load_canonical_mmu (vcpu); + +if (gpa >=kvm_phys_size (vcpu->arch.hw_mmu)) { + ret = -ENOENT ; + goto out; + } + + gfn =gpa_to_gfn (gpa); + memslot =gfn_to_memslot (vcpu->kvm, gfn); +if (!memslot) { + ret = -ENOENT ; + goto out; + } + +/* +* A racing memslot deletion or move installs an invalid slot before +* zapping stage-2. Ask userspace to retry once the update settles.
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2851
Quoted 10 lines
arch/arm64/kvm/mmu.c +10
-0+*/ +if (memslot->flags &KVM_MEMSLOT_INVALID ) { + ret = -EAGAIN ; + goto out; + } + +/* +* pKVM stage-2 mappings aren't directly walkable from the host; let +* the fault path handle both new and existing mappings. +*/
pKVM has a host interval-tree where we keep track of what has been mapped and at what level. This tree is also protected with the mmu_lock.
Quoted 74 lines
arch/arm64/kvm/mmu.c +70
-0+if (!is_protected_kvm_enabled ()) { + pgt = vcpu->arch.hw_mmu->pgt; + scoped_guard(read_lock, &vcpu->kvm->mmu_lock) { + ret =kvm_pgtable_get_leaf (pgt, gpa, &pte, &walk_level, +KVM_PGTABLE_WALK_SHARED ); + } +if (ret) + goto out; + +if (kvm_pte_valid (pte)) { + page_size =kvm_granule_size (walk_level); +if (!(pte &KVM_PTE_LEAF_ATTR_LO_S2_AF )) +handle_access_fault (vcpu, gpa); + goto out_success; + } + } + +/* +* Synthesize a read translation fault for the canonical IPA, at the +* level where the stage-2 walk currently ends (the last level under +* pKVM, where stage-2 isn't walkable from the host). +*/ + fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) | + ESR_ELx_IL |ESR_ELx_FSC_FAULT_L (walk_level); + fault_info->hpfar_el2 =HPFAR_EL2_NS | +FIELD_PREP (HPFAR_EL2_FIPA , gpa >>12 ); + +struct kvm_s2_fault_desc s2fd = { + .vcpu = vcpu, + .fault_ipa = gpa, + .nested =NULL , + .memslot = memslot, + .page_size = &page_size, + .prefault = true, + }; + +/* +* As in the run path, -EAGAIN from the abort handlers is treated as +* progress: either a parallel fault installed the mapping, or a racing +* invalidation is in flight and the next access will refault. +*/ +if (kvm_slot_has_gmem (memslot)) { + ret =gmem_abort (&s2fd); + }else { + hva =gfn_to_hva_memslot_prot (memslot, gfn,NULL ); +if (kvm_is_error_hva (hva)) { + ret = -EFAULT ; + goto out; + } + + s2fd.hva = hva; + ret =user_mem_abort (&s2fd); + } + +if (ret <0 ) + goto out; + +out_success: + end =ALIGN_DOWN (gpa, page_size) + page_size; + ret =min_t (u64, range->size, end - gpa); +out: +/* +* Restore the synthetic fault state so a subsequent KVM_RUN does not +* observe it. kvm_handle_mmio_return() runs before guest entry can +* refresh fault.esr_el2 from hardware, so leaving the synthetic ESR +* in place would corrupt the completion of a pending MMIO exit. +*/ + *fault_info = fault_backup; +return ret; +}Signature · 1 line
2.43.0
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Alexandru EliseiQuoted 4 lines · Alexandru Elisei
Hi Jack,
Quoted 1 line · Jack Thomson
From: Jack Thomson <[email protected]>
....
Quoted 14 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2903
Quoted 7 lines
arch/arm64/kvm/mmu.c +7
-0+if (kvm_is_error_hva (hva)) { + ret = -EFAULT ; + goto out; + } + + s2fd.hva = hva; + ret =user_mem_abort (&s2fd);If
is_protected_kvm_enabled(), this function doesn't check if the mapping already exists.user_mem_abort()will call **p**kvm_pgtable_stage2_map() (emphasis is mine) to create the mapping, andpkvm_pgtable_stage2_map()will return-EAGAIN, and keep returning-EAGAINuntil the mapping is destroyed, which might be some time away.
Quoted 6 lines
Same thing happens **without** pKVM,
kvm_pgtable_stage2_map()will return-EAGAINif the mapping exists. But that's not such a big issue, when the user callsKVM_PRE_FAULT_MEMORYon the remaining rangekvm_pgtable_get_leaf()will detect the existing mapping. Which you have already taken into consideration judging by the comment, so all good here.
But both gmem_abort() and user_mem_abort() handle -EAGAIN internally, don't they?
For gmem_abort(), we have:
1692: return ret != -EAGAIN ? ret : 0;
For user_mem_abort(), the call path is:
user_mem_abort()
-> kvm_s2_fault_map()
-> KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, ...)
and kvm_s2_fault_map() does:
2111: if (ret != -EAGAIN)
return ret; return 0;
-aneesh
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
Hi Aneesh,
Quoted 23 lines · Aneesh Kumar K.V
Quoted 4 lines · Alexandru Elisei
Hi Jack,
Quoted 1 line · Jack Thomson
From: Jack Thomson <[email protected]>
....
Quoted 14 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2903
Quoted 7 lines
arch/arm64/kvm/mmu.c +7
-0+if (kvm_is_error_hva (hva)) { + ret = -EFAULT ; + goto out; + } + + s2fd.hva = hva; + ret =user_mem_abort (&s2fd);If
is_protected_kvm_enabled(), this function doesn't check if the mapping already exists.user_mem_abort()will call **p**kvm_pgtable_stage2_map() (emphasis is mine) to create the mapping, andpkvm_pgtable_stage2_map()will return-EAGAIN, and keep returning-EAGAINuntil the mapping is destroyed, which might be some time away.
Quoted 8 lines
Quoted 6 lines
Same thing happens **without** pKVM,
kvm_pgtable_stage2_map()will return-EAGAINif the mapping exists. But that's not such a big issue, when the user callsKVM_PRE_FAULT_MEMORYon the remaining rangekvm_pgtable_get_leaf()will detect the existing mapping. Which you have already taken into consideration judging by the comment, so all good here.But both
gmem_abort()anduser_mem_abort()handle-EAGAINinternally, don't they?
I think you're right, please ignore my comment about an existing mapping with is_protected_kvm_enabled().
Thanks,
Alex
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Alexandru EliseiHi Alex,
Quoted 6 lines · Alexandru Elisei
Quoted 5 lines
Does it guarantee that, after it completes and returns to userspace, the address range [gpa, gpa + size] is mapped in the VM's stage 2 tables? Or does it guarantee that the address range was mapped at stage 2 at some point during the ioctl's execution, but parts, or all of it, might not be mapped after the ioctl completes and returns to userspace?
Quoted 6 lines
Nevermind, spend some time thinking about it and it turns out it's impossible for KVM to guarantee that the memory is still mapped after the ioctls returns. So I answered my own question, thanks.
Alex
Yeah it's best effort, as if a vCPU had faulted the range itself. This could be undone by an invalidation or memslot update. The value is in the common cast (for us, post-copy migration) the guest doesn't pay the stage-2 fault fault cost at run time.
I think it would be worth me adding a sentence to the documentation noting there is no persistance guarentee.
Signature · 2 lines
Thanks, Jack
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Alexandru EliseiHi Alex,
Quoted 7 lines · Alexandru Elisei
arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574
Quoted 1 line
arch/arm64/kvm/mmu.c +1
-0+unsigned long *page_size;As far as I know, fault handling was reworked to use struct
kvm_s2_fault_descto store the fault information thatuser_mem_abort()needs to handle the fault. Adding a 'page_size' field, that represents the result of the gpa mapping process, might not be desirable.
Yeah agreed, Vincent suggested a separate output struct that kvm_s2_fault_map() fills with what was actually mapped. I'll do that in v6 thanks!
Quoted 6 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2805
Quoted 2 lines
arch/arm64/kvm/mmu.c +2
-0+struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault; +struct kvm_vcpu_fault_info fault_backup = *fault_info;I'm not sure you need to make a backup here.
vcpu->arch.faultis populated each time the CPU takes a fault.
So this was actually flagged up by sashiko when I ran it before submission, it suggested handling this for the case in which the vCPU exited for MMIO, the next KVM_RUN calls kvm_handle_mmio_return which uses the vcu->arch.fault.esr_el2. I'll add some comments to this maybe to explain why this would be needed.
Quoted 8 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2843
Quoted 1 line
arch/arm64/kvm/mmu.c +1
-0+if (memslot->flags &KVM_MEMSLOT_INVALID ) {I don't think that's something we should care about, the flag can be set immediately after the check as the function doesn't take
kvm->slots_lock.kvm_vcpu_prefault_memory()takes the srcu lock in read mode, so the function is safe to run even if userspace does something silly like deleting a memslot at the same time that it's prefaulting the guest memory it represents.
I was there not as much for safety, but rather to pick up that error. This way rather than userspace getting an -EFAULT it can just retry. This is the same way it is handled in x86 as well.
Quoted 5 lines
It might not be obvious, but taking
kvm->mmu_lockin read mode does not guarantee that gpa will be mapped whenkvm_pgtable_get_leaf()returns. That's because a concurrentkvm_pgtable_stage2_map()for a different gpa can destroy the mapping of the current gpa.
[ ... ]
Quoted 3 lines
The
kvm->mmu_lockis dropped here, which means that it is possible for a MMU notifier callback to have just unmapped the entire stage 2 for the VM.
Yeah as we mentioned in the other thread, this is best effort and we can't guarantee that is survives the return. We do this for forward progression, reporting the correct advance size for ranges which are already mapped, instead of repeating the full dance for every page of them.
Quoted 4 lines
handle_access_fault()will fail if the mapping is gone. But I guess that's fine if the ioctl does not guarantee that memory is still mapped after it completes.
Yeah I think it's fine and will degrade to a no-op in this case.
Quoted 5 lines
Also, the documentation that this patch adds says: 'On arm64, newly created stage-2 PTEs are marked Accessed'. Does not say anything about marking **existing** ptes as accessed. Would be useful to explain the code does it.
Ack, will do thanks!
Quoted 4 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2896
Quoted 1 line
arch/arm64/kvm/mmu.c +1
-0+ hva =gfn_to_hva_memslot_prot (memslot, gfn,NULL );There's
gfn_to_hva_memslot(memslot, gfn), is that what you are looking for?
I think gfn_to_hva_memslot() resolves the hva for write, so would fail on KVM_MEM_READONLY slots, and prefualting is a read. Also this mirrors the guest abort path so matches the way the run path resolves the HVA.
Signature · 2 lines
Thanks, Jack
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Vincent DonnefortHi Vincent,
Quoted 8 lines · Vincent Donnefort
arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574
Quoted 1 line
arch/arm64/kvm/mmu.c +1
-0+unsigned long *page_size;It looks a bit odd in that struct.
Also,
transparent_hugepage_adjust()modifies pfn and gfn. So perhaps it'd be better to introduce a structkvm_s2_fault_mappingthat can be used to get what has actually been mapped inkvm_s2_fault_map()?
Yeah that's much nicer I'll do that thanks for the suggestion!
Quoted 5 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2818
Quoted 2 lines
arch/arm64/kvm/mmu.c +2
-0+if (vcpu_is_protected (vcpu)) +return -EOPNOTSUPP ;It could just call
pkvm_mem_abort()when the VM is protected?
I see, I don't have a setup to test this properly myself, so at the moment I left the protected vm's out and don't advertise the capability, if there's a usecase happy to look at it as a follow-up.
Quoted 8 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2851
Quoted 4 lines
arch/arm64/kvm/mmu.c +4
-0+/* +* pKVM stage-2 mappings aren't directly walkable from the host; let +* the fault path handle both new and existing mappings. +*/ pKVM has a host interval-tree where we keep track of what has been mapped and at what level. This tree is also protected with the
mmu_lock.
Thanks, that sounds better. I'll have a look at moving to that for v6 then.
Signature · 2 lines
Thanks, Jack
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Alexandru EliseiHi Aneesh and Alex
Quoted 5 lines · Alexandru Elisei
Quoted 1 line · Aneesh Kumar K.V
But both
gmem_abort()anduser_mem_abort()handle-EAGAINinternally, don't they?I think you're right, please ignore my comment about an existing mapping with
is_protected_kvm_enabled().
Thanks Aneesh, yeah that's the intent. Vincent pointed me to the pgt->pkvm_mappings interval tree that can be used to detect the existing mappings in pKVM, so I'll be updating to use that in v6.
Signature · 2 lines
Thanks, Jack
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
In reply to Thomson, JackHi Jack,
Quoted 22 lines · Jack
Hi Alex,
Quoted 7 lines · Alexandru Elisei
arch/arm64/kvm/mmu.c struct kvm_s2_fault_desc { · line 1574
Quoted 1 line
arch/arm64/kvm/mmu.c +1
-0+unsigned long *page_size;As far as I know, fault handling was reworked to use struct
kvm_s2_fault_descto store the fault information thatuser_mem_abort()needs to handle the fault. Adding a 'page_size' field, that represents the result of the gpa mapping process, might not be desirable.Yeah agreed, Vincent suggested a separate output struct that
kvm_s2_fault_map()fills with what was actually mapped. I'll do that in v6 thanks!Quoted 6 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2805
Quoted 2 lines
arch/arm64/kvm/mmu.c +2
-0+struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault; +struct kvm_vcpu_fault_info fault_backup = *fault_info;I'm not sure you need to make a backup here.
vcpu->arch.faultis populated each time the CPU takes a fault.
Quoted 5 lines
So this was actually flagged up by sashiko when I ran it before submission, it suggested handling this for the case in which the vCPU exited for MMIO, the next
KVM_RUNcallskvm_handle_mmio_returnwhich uses thevcu->arch.fault.esr_el2. I'll add some comments to this maybe to explain why this would be needed.
Ok, I see, VCPU exits to userspace due to MMIO, userspace calls KVM_PRE_FAULT_MEMORY, userspace resumes VCPU and uses fault_info from KVM_PRE_FAULT_MEMORY. Nicely spotted.
Quoted 10 lines
Quoted 9 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2843
Quoted 2 lines
arch/arm64/kvm/mmu.c +1
-0+if (memslot->flags &KVM_MEMSLOT_INVALID ) {I don't think that's something we should care about, the flag can be set immediately after the check as the function doesn't take
kvm->slots_lock.kvm_vcpu_prefault_memory()takes the srcu lock in read mode, so the function is safe to run even if userspace does something silly like deleting a memslot at the same time that it's prefaulting the guest memory it represents.
Quoted 3 lines
I was there not as much for safety, but rather to pick up that error. This way rather than userspace getting an
-EFAULTit can just retry. This is the same way it is handled in x86 as well.
My point was that since there's no serialization between memslot changes and KVM_PRE_FAULT_MEMORY, the condition can become true immediately after the if statement. Doesn't really matter though anyway.
Quoted 25 lines
Quoted 6 lines
It might not be obvious, but taking
kvm->mmu_lockin read mode does not guarantee that gpa will be mapped whenkvm_pgtable_get_leaf()returns. That's because a concurrentkvm_pgtable_stage2_map()for a different gpa can destroy the mapping of the current gpa.[ ... ]
Quoted 3 lines
The
kvm->mmu_lockis dropped here, which means that it is possible for a MMU notifier callback to have just unmapped the entire stage 2 for the VM.Yeah as we mentioned in the other thread, this is best effort and we can't guarantee that is survives the return. We do this for forward progression, reporting the correct advance size for ranges which are already mapped, instead of repeating the full dance for every page of them.
Quoted 4 lines
handle_access_fault()will fail if the mapping is gone. But I guess that's fine if the ioctl does not guarantee that memory is still mapped after it completes.Yeah I think it's fine and will degrade to a no-op in this case.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Also, the documentation that this patch adds says: 'On arm64, newly created stage-2 PTEs are marked Accessed'. Does not say anything about marking **existing** ptes as accessed. Would be useful to explain the code does it.
Ack, will do thanks!
I think there's a word missing there, I meant to say that it would be useful to explain *why* the x86 implementation does not mark new PTEs as accessed, while arm64 marks them as accessed, and arm64 also does that for existing PTEs.
Quoted 9 lines
Quoted 5 lines
arch/arm64/kvm/mmu.c void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) · line 2896
Quoted 2 lines
arch/arm64/kvm/mmu.c +1
-0+ hva =gfn_to_hva_memslot_prot (memslot, gfn,NULL );There's
gfn_to_hva_memslot(memslot, gfn), is that what you are looking for?I think
gfn_to_hva_memslot()resolves the hva for write, so would fail onKVM_MEM_READONLYslots, and prefualting is a read. Also this mirrors the guest abort path so matches the way the run path resolves the HVA.
Ah, I missed that, thanks for explaining!
I was wondering if you have any plans about posting an updated series.
Thanks,
Alex
Re: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation
Hi Alex,
On 25/08/2026 1:29 pm, Alexandru Elisei wrote:
[...]
Quoted 5 lines
I think there's a word missing there, I meant to say that it would be useful to explain *why* the x86 implementation does not mark new PTEs as accessed, while arm64 marks them as accessed, and arm64 also does that for existing PTEs.
Ah got you, I'll update the docs in v6 to explain that. Thanks!
Quoted 1 line
I was wondering if you have any plans about posting an updated series.
Yeah I have v6 almost ready, I've addressed the comments. While testing it found an issue on pKVM which would case a hyp panic if we prefaulted before it's first run, the EL2 context load blows up. So I've also go a fix for that bundled in.
I'm just tidying a few bits up and hopefully have it out within the next couple of days.
Signature · 2 lines
Thanks, Jack