Avoiding slow get-user-pages via memory fault exit
Subsystems (7)
v2 cover Avoiding slow get-user-pages via memory fault exit
Hi Sean, here's what I'm planing to send up as v2 of the scalable userfaultfd series.
Don't worry, I'm not asking you to review this all :) I just have a few remaining questions regarding KVM_CAP_MEMORY_FAULT_EXIT which seem important enough to mention before I ask for more attention from others, and they'll be clearer with the patches in hand. Anything else I'm happy to find out about when I send the actual v2.
I want your opinion on
1. The general API I've set up for KVM_CAP_MEMORY_FAULT_EXIT (described in the api.rst file)
2. Whether the UNKNOWN exit reason cases (everywhere but handle_error_pfn atm) would need to be given "real" reasons before this could be merged.
3. If you think I've missed sites that currently -EFAULT to userspace
About (3): after we agreed to only tackle cases where -EFAULT currently makes it to userspace, I went though our list and tried to trace which EFAULTS actually bubble up to KVM_RUN. That set ended being suspiciously small, so I wanted to sanity-check my findings with you. Lmk if you see obvious errors in my list below.
--- EFAULTs under KVM_RUN ---
Confident that needs conversion (already converted)
---------------------------------------------------
* direct_map
* handle_error_pfn
* setup_vmgexit_scratch
* kvm_handle_page_fault
* FNAME(fetch)
EFAULT does not propagate to userspace (do not convert)
-------------------------------------------------------
* record_steal_time (arch/x86/kvm/x86.c:3463)
* hva_to_pfn_retry
* kvm_vcpu_map
* FNAME(update_accessed_dirty_bits)
* __kvm_gfn_to_hva_cache_init
Might actually make it to userspace, but only through kvm_read|write_guest_offset_cached- would be covered by those conversions
* kvm_gfn_to_hva_cache_init
* __kvm_read_guest_page
* hva_to_pfn_remapped
handle_error_pfn will handle this for the scalable uffd case. Don't think other callers -EFAULT to userspace.
Still unsure if needs conversion
--------------------------------
* __kvm_read_guest_atomic
The EFAULT might be propagated though FNAME(sync_page)?
* kvm_write_guest_offset_cached (virt/kvm/kvm_main.c:3226)
* __kvm_write_guest_page
Called from kvm_write_guest_offset_cached: if that needs change, this does too
* kvm_write_guest_page
Two interesting paths:
- kvm_pv_clock_pairing returns a custom KVM_EFAULT error here (arch/x86/kvm/x86.c:9578)
- kvm_write_guest_offset_cached returns this directly (so if that needs change, this does too)
* kvm_read_guest_offset_cached
I actually do see a path to userspace, but it's through hyper-v, which we've said is out of scope for round 1.
--- Actual Cover Letter ---
Omitted: hasn't changed much since v1 anyways
--- Changelog ---
WIP v2
- Introduce KVM_CAP_X86_MEMORY_FAULT_EXIT.
- API changes:
- Gate KVM_CAP_MEMORY_FAULT_NOWAIT behind KVM_CAP_x86_MEMORY_FAULT_EXIT (on x86 only: arm has no such requirement).
- Switched to memslot flag
- Take Oliver's simplification to the "allow fast gup for readable faults" logic.
- Slightly redefine the return code of user_mem_abort.
- Fix documentation errors brought up by Marc
- Reword commit messages in imperative mood
v1: https://lore.kernel.org/kvm/[email protected]/
Anish Moorthy (14):
KVM: selftests: Allow many vCPUs and reader threads per UFFD in demand
paging test
KVM: selftests: Use EPOLL in userfaultfd_util reader threads and
signal errors via TEST_ASSERT
KVM: Allow hva_pfn_fast to resolve read-only faults.
KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run
field
KVM: x86: Implement memory fault exit for direct_map
KVM: x86: Implement memory fault exit for kvm_handle_page_fault
KVM: x86: Implement memory fault exit for setup_vmgexit_scratch
KVM: x86: Implement memory fault exit for FNAME(fetch)
KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
KVM: x86: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
KVM: arm64: Allow user_mem_abort to return 0 to signal a 'normal' exit
KVM: arm64: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
KVM: selftests: Add memslot_flags parameter to memstress_create_vm
KVM: selftests: Handle memory fault exits in demand_paging_test
Documentation/virt/kvm/api.rst | 74 ++++- arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 29 +- arch/x86/kvm/mmu/mmu.c | 42 ++- arch/x86/kvm/mmu/paging_tmpl.h | 4 +- arch/x86/kvm/svm/sev.c | 4 +- arch/x86/kvm/x86.c | 2 + include/linux/kvm_host.h | 22 ++ include/uapi/linux/kvm.h | 19 ++ tools/include/uapi/linux/kvm.h | 17 ++ .../selftests/kvm/aarch64/page_fault_test.c | 4 +- .../selftests/kvm/access_tracking_perf_test.c | 2 +- .../selftests/kvm/demand_paging_test.c | 253 ++++++++++++++---- .../selftests/kvm/dirty_log_perf_test.c | 2 +- .../testing/selftests/kvm/include/memstress.h | 2 +- .../selftests/kvm/include/userfaultfd_util.h | 18 +- tools/testing/selftests/kvm/lib/memstress.c | 4 +- .../selftests/kvm/lib/userfaultfd_util.c | 160 ++++++----- .../kvm/memslot_modification_stress_test.c | 2 +- virt/kvm/kvm_main.c | 41 ++- 20 files changed, 544 insertions(+), 158 deletions(-)
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 1/14 KVM: selftests: Allow many vCPUs and reader threads per UFFD in demand paging test
At the moment, demand_paging_test does not support profiling/testing multiple vCPU threads concurrently faulting on a single uffd because
(a) "-u" (run test in userfaultfd mode) creates a uffd for each vCPU's region, so that each uffd services a single vCPU thread.
(b) "-u -o" (userfaultfd mode + overlapped vCPU memory accesses) simply doesn't work: the test tries to register the same memory to multiple uffds, causing an error.
Add support for many vcpus per uffd by
(1) Keeping "-u" behavior unchanged.
(2) Making "-u -a" create a single uffd for all of guest memory.
(3) Making "-u -o" implicitly pass "-a", solving the problem in (b).
In cases (2) and (3) all vCPU threads fault on a single uffd.
With multiple potentially multiple vCPU per UFFD, it makes sense to allow configuring the number reader threads per UFFD as well: add the "-r" flag to do so.
--- .../selftests/kvm/aarch64/page_fault_test.c | 4 +- .../selftests/kvm/demand_paging_test.c | 62 +++++++++---- .../selftests/kvm/include/userfaultfd_util.h | 18 +++- .../selftests/kvm/lib/userfaultfd_util.c | 86 +++++++++++++------ 4 files changed, 125 insertions(+), 45 deletions(-)
tools/testing/selftests/kvm/aarch64/page_fault_test.c +2 -2
@@ -376,14 +376,14 @@ static void setup_uffd(struct kvm_vm *vm, struct test_params *p, *pt_uffd =uffd_setup_demand_paging (uffd_mode,0 , pt_args.hva, pt_args.paging_size, - test->uffd_pt_handler); +1 , test->uffd_pt_handler); *data_uffd =NULL ;if (test->uffd_data_handler) *data_uffd =uffd_setup_demand_paging (uffd_mode,0 , data_args.hva, data_args.paging_size, - test->uffd_data_handler); +1 , test->uffd_data_handler); } static void free_uffd(struct test_desc *test, struct uffd_desc *pt_uffd,
tools/testing/selftests/kvm/demand_paging_test.c +47 -15
154 lines not on this page · raw message
tools/testing/selftests/kvm/include/userfaultfd_util.h +15 -3
@@ -17,18 +17,30 @@typedef int (*uffd_handler_t )(int uffd_mode,int uffd,struct uffd_msg *msg); +struct uffd_reader_args { +int uffd_mode; +int uffd; +useconds_t delay; +uffd_handler_t handler; +/* Holds the read end of the pipe for killing the reader. */ +int pipe; +}; +struct uffd_desc {int uffd_mode;int uffd; -int pipefds[2 ];useconds_t delay;uffd_handler_t handler; -pthread_t thread; +uint64_t num_readers; +/* Holds the write ends of the pipes for killing the readers. */ +int *pipefds; +pthread_t *readers; +struct uffd_reader_args *reader_args; };struct uffd_desc *uffd_setup_demand_paging (int uffd_mode,useconds_t delay,void *hva,uint64_t len, -uffd_handler_t handler); +uint64_t num_readers,uffd_handler_t handler);void uffd_stop_demand_paging (struct uffd_desc *uffd);
tools/testing/selftests/kvm/lib/userfaultfd_util.c +61 -25
145 lines not on this page · raw message
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 2/14 KVM: selftests: Use EPOLL in userfaultfd_util reader threads and signal errors via TEST_ASSERT
In reply to Anish MoorthyWith multiple reader threads POLLing a single UFFD, the test suffers from the thundering herd problem: performance degrades as the number of reader threads is increased. Solve this issue [1] by switching the the polling mechanism to EPOLL + EPOLLEXCLUSIVE.
Also, change the error-handling convention of uffd_handler_thread_fn. Instead of just printing errors and returning early from the polling loop, check for them via TEST_ASSERT. "return NULL" is reserved for a successful exit from uffd_handler_thread_fn, ie one triggered by a write to the exit pipe.
Performance samples generated by the command in [2] are given below.
Num Reader Threads, Paging Rate (POLL), Paging Rate (EPOLL)
1 249k 185k 2 201k 235k 4 186k 155k 16 150k 217k 32 89k 198k
[1] Single-vCPU performance does suffer somewhat.
[2] ./demand_paging_test -u MINOR -s shmem -v 4 -o -r <num readers>
--- .../selftests/kvm/demand_paging_test.c | 1 - .../selftests/kvm/lib/userfaultfd_util.c | 76 +++++++++---------- 2 files changed, 37 insertions(+), 40 deletions(-)
tools/testing/selftests/kvm/demand_paging_test.c +0 -1
@@ -13,7 +13,6 @@#include <stdio.h> #include <stdlib.h> #include <time.h> -#include <poll.h> #include <pthread.h> #include <linux/userfaultfd.h> #include <sys/syscall.h>
tools/testing/selftests/kvm/lib/userfaultfd_util.c +37 -39
111 lines not on this page · raw message
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 3/14 KVM: Allow hva_pfn_fast to resolve read-only faults.
In reply to Anish Moorthyhva_to_pfn_fast currently just fails for read-only faults, which is unnecessary. Instead, try pinning the page without passing FOLL_WRITE. This allows read-only faults to (potentially) be resolved without falling back to slow GUP.
--- virt/kvm/kvm_main.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
virt/kvm/kvm_main.c +3 -5
@@ -2479,7 +2479,7 @@ static inline int check_user_page_hwpoison(unsigned long addr) }/* -* The fast path to get the writable pfn which will be stored in @pfn, +* The fast path to get the pfn which will be stored in @pfn, * true indicates success, otherwise false is returned. It's also the * only part that runs if we can in atomic context. */
@@ -2487,16 +2487,14 @@ static bool hva_to_pfn_fast(unsigned long addr, bool write_fault, bool *writable, kvm_pfn_t *pfn) { struct page *page[1]; -/* * Fast pin a writable pfn only if it is a write fault request * or the caller allows to map a writable pfn for a read fault * request. */ -if (!(write_fault || writable)) -return false; +unsigned int gup_flags = (write_fault || writable) ?FOLL_WRITE :0 ; -if (get_user_page_fast_only (addr,FOLL_WRITE , page)) { +if (get_user_page_fast_only (addr, gup_flags, page)) { *pfn =page_to_pfn (page[0 ]);if (writable)
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 4/14 KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
In reply to Anish MoorthyMemory fault exits allow KVM to return useful information from KVM_RUN instead of having to -EFAULT when a guest memory access goes wrong. Document the intent and API of the new capability, and introduce helper functions which will be useful in places where it needs to be implemented.
Also allow the capability to be enabled, even though that won't currently *do* anything: implementations at the relevant -EFAULT sites will performed in subsequent commits.
--- Documentation/virt/kvm/api.rst | 37 ++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 1 + include/linux/kvm_host.h | 16 +++++++++++++++ include/uapi/linux/kvm.h | 16 +++++++++++++++ tools/include/uapi/linux/kvm.h | 15 ++++++++++++++ virt/kvm/kvm_main.c | 28 +++++++++++++++++++++++++ 6 files changed, 113 insertions(+)
Documentation/virt/kvm/api.rst +37 -0
@@ -6636,6 +6636,19 @@ array field represents return values. The userspace should update the return values of SBI call before resuming the VCPU. For more details on RISC-V SBI spec refer, https://github.com/riscv/riscv-sbi-doc. +:: + + /* KVM_EXIT_MEMORY_FAULT */ + struct { + __u64 flags; + __u64 gpa; + __u64 len; /* in bytes */ + } memory_fault; + +Indicates a memory fault on the guest physical address range [gpa, gpa + len). +flags is a bitfield describing the reasons(s) for the fault. See +KVM_CAP_X86_MEMORY_FAULT_EXIT for more details. + :: /* KVM_EXIT_NOTIFY */
@@ -7669,6 +7682,30 @@ This capability is aimed to mitigate the threat that malicious VMs can cause CPU stuck (due to event windows don't open up) and make the CPU unavailable to host or other VMs. +7.34 KVM_CAP_X86_MEMORY_FAULT_EXIT +---------------------------------- + +:Architectures: x86 +:Parameters: args[0] is a bitfield specifying what reasons to exit upon. +:Returns: 0 on success, -EINVAL if unsupported or if unrecognized exit reason + specified. + +This capability transforms -EFAULTs returned by KVM_RUN in response to guest +memory accesses into VM exits (KVM_EXIT_MEMORY_FAULT), with 'gpa' and 'len' +describing the problematic range of memory and 'flags' describing the reason(s) +for the fault. + +The implementation is currently incomplete. Please notify the maintainers if you +come across a case where it needs to be implemented. + +Through args[0], the capability can be set on a per-exit-reason basis. +Currently, the only exit reasons supported are + +1. KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) + +Memory fault exits with a reason of UNKNOWN should not be depended upon: they +may be added, removed, or reclassified under a stable reason. + 8. Other capabilities. ======================
arch/x86/kvm/x86.c +1 -0
@@ -4425,6 +4425,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_VAPIC :case KVM_CAP_ENABLE_CAP :case KVM_CAP_VM_DISABLE_NX_HUGE_PAGES : +case KVM_CAP_X86_MEMORY_FAULT_EXIT : r =1 ;break ;case KVM_CAP_EXIT_HYPERCALL :
include/linux/kvm_host.h +16 -0
@@ -805,6 +805,7 @@ struct kvm {struct notifier_block pm_notifier;#endif char stats_id[KVM_STATS_NAME_SIZE ]; +uint64_t memfault_exit_reasons; };#define kvm_err (fmt, ...) \
@@ -2278,4 +2279,19 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)/* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +/* +* If memory fault exits are enabled for any of the reasons given in exit_flags +* then sets up a KVM_EXIT_MEMORY_FAULT for the given guest physical address, +* length, and flags and returns -1. +* Otherwise, returns -EFAULT +*/ +inline int kvm_memfault_exit_or_efault ( +struct kvm_vcpu *vcpu,uint64_t gpa,uint64_t len,uint64_t exit_flags); + +/* +* Checks that all of the bits specified in 'reasons' correspond to known +* memory fault exit reasons. +*/ +bool kvm_memfault_exit_flags_valid (uint64_t reasons); +#endif
include/uapi/linux/kvm.h +16 -0
@@ -264,6 +264,7 @@ struct kvm_xen_exit {#define KVM_EXIT_RISCV_SBI 35#define KVM_EXIT_RISCV_CSR 36#define KVM_EXIT_NOTIFY 37 +#define KVM_EXIT_MEMORY_FAULT 38/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -505,6 +506,17 @@ struct kvm_run {#define KVM_NOTIFY_CONTEXT_INVALID (1 << 0)__u32 flags; } notify; +/* KVM_EXIT_MEMORY_FAULT */ +struct { +/* +* Indicates a memory fault on the guest physical address range +* [gpa, gpa + len). flags is a bitfield describing the reasons(s) +* for the fault. +*/ +__u64 flags; +__u64 gpa; +__u64 len;/* in bytes */ + } memory_fault;/* Fix the size of the union. */ char padding[256 ]; };
@@ -1184,6 +1196,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224#define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225#define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226 +#define KVM_CAP_X86_MEMORY_FAULT_EXIT 227#ifdef KVM_CAP_IRQ_ROUTING
@@ -2237,4 +2250,7 @@ struct kvm_s390_zpci_op {/* flags for kvm_s390_zpci_op->u.reg_aen.flags */ #define KVM_S390_ZPCIOP_REGAEN_HOST (1 << 0) +/* Exit reasons for KVM_EXIT_MEMORY_FAULT */ +#define KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) +#endif /* __LINUX_KVM_H */
tools/include/uapi/linux/kvm.h +15 -0
@@ -264,6 +264,7 @@ struct kvm_xen_exit {#define KVM_EXIT_RISCV_SBI 35#define KVM_EXIT_RISCV_CSR 36#define KVM_EXIT_NOTIFY 37 +#define KVM_EXIT_MEMORY_FAULT 38/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -505,6 +506,17 @@ struct kvm_run {#define KVM_NOTIFY_CONTEXT_INVALID (1 << 0)__u32 flags; } notify; +/* KVM_EXIT_MEMORY_FAULT */ +struct { +/* +* Indicates a memory fault on the guest physical address range +* [gpa, gpa + len). flags is a bitfield describing the reasons(s) +* for the fault. +*/ +__u64 flags; +__u64 gpa; +__u64 len;/* in bytes */ + } memory_fault;/* Fix the size of the union. */ char padding[256 ]; };
@@ -2228,4 +2240,7 @@ struct kvm_s390_zpci_op {/* flags for kvm_s390_zpci_op->u.reg_aen.flags */ #define KVM_S390_ZPCIOP_REGAEN_HOST (1 << 0) +/* Exit reasons for KVM_EXIT_MEMORY_FAULT */ +#define KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) +#endif /* __LINUX_KVM_H */
virt/kvm/kvm_main.c +28 -0
@@ -1142,6 +1142,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)spin_lock_init (&kvm->mn_invalidate_lock);rcuwait_init (&kvm->mn_memslots_update_rcuwait);xa_init (&kvm->vcpu_array); + kvm->memfault_exit_reasons =0 ;INIT_LIST_HEAD (&kvm->gpc_list);spin_lock_init (&kvm->gpc_lock);
@@ -4671,6 +4672,14 @@ static int kvm_vm_ioctl_enable_cap_generic(struct kvm *kvm,return r; } +case KVM_CAP_X86_MEMORY_FAULT_EXIT : { +if (!kvm_vm_ioctl_check_extension (kvm,KVM_CAP_X86_MEMORY_FAULT_EXIT )) +return -EINVAL ; +else if (!kvm_memfault_exit_flags_valid (cap->args[0 ])) +return -EINVAL ; + kvm->memfault_exit_reasons = cap->args[0 ]; +return 0 ; + }default :return kvm_vm_ioctl_enable_cap (kvm, cap); }
@@ -6172,3 +6181,22 @@ int kvm_vm_create_worker_thread(struct kvm *kvm, kvm_vm_thread_fn_t thread_fn,return init_context.err; } + +inline int kvm_memfault_exit_or_efault ( +struct kvm_vcpu *vcpu,uint64_t gpa,uint64_t len,uint64_t exit_flags) +{ +if (!(vcpu->kvm->memfault_exit_reasons & exit_flags)) +return -EFAULT ; + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; + vcpu->run->memory_fault.gpa = gpa; + vcpu->run->memory_fault.len = len; + vcpu->run->memory_fault.flags = exit_flags; +return -1 ; +} + +bool kvm_memfault_exit_flags_valid (uint64_t reasons) +{ +uint64_t valid_flags =KVM_MEMFAULT_REASON_UNKNOWN ; + +return !(reasons & !valid_flags); +}
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 43 lines · Anish Moorthy
Memory fault exits allow KVM to return useful information from
KVM_RUNinstead of having to-EFAULTwhen a guest memory access goes wrong. Document the intent and API of the new capability, and introduce helper functions which will be useful in places where it needs to be implemented.Also allow the capability to be enabled, even though that won't currently *do* anything: implementations at the relevant
-EFAULTsites will performed in subsequent commits.--- Documentation/virt/kvm/api.rst | 37 ++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 1 + include/linux/kvm_host.h | 16 +++++++++++++++ include/uapi/linux/kvm.h | 16 +++++++++++++++ tools/include/uapi/linux/kvm.h | 15 ++++++++++++++ virt/kvm/kvm_main.c | 28 +++++++++++++++++++++++++ 6 files changed, 113 insertions(+)Documentation/virt/kvm/api.rst +13
-0@@ -6636,6 +6636,19 @@ array field represents return values. The userspace should update the return values of SBI call before resuming the VCPU. For more details on RISC-V SBI spec refer, https://github.com/riscv/riscv-sbi-doc. +:: + + /* KVM_EXIT_MEMORY_FAULT */ + struct { + __u64 flags; + __u64 gpa; + __u64 len; /* in bytes */ + } memory_fault; + +Indicates a memory fault on the guest physical address range [gpa, gpa + len). +flags is a bitfield describing the reasons(s) for the fault. See +KVM_CAP_X86_MEMORY_FAULT_EXIT for more details. + :: /* KVM_EXIT_NOTIFY */
Documentation/virt/kvm/api.rst This capability is aimed to mitigate the threat that malicious VMs can · line 7688
Quoted 7 lines
Documentation/virt/kvm/api.rst +4
-0@@ -7669,6 +7682,30 @@ This capability is aimed to mitigate the threat that malicious VMs can cause CPU stuck (due to event windows don't open up) and make the CPU unavailable to host or other VMs. +7.34 KVM_CAP_X86_MEMORY_FAULT_EXIT +---------------------------------- + +:Architectures: x86
Why x86 specific?
Quoted 165 lines
Documentation/virt/kvm/api.rst +20
-0+:Parameters: args[0] is a bitfield specifying what reasons to exit upon. +:Returns: 0 on success, -EINVAL if unsupported or if unrecognized exit reason + specified. + +This capability transforms -EFAULTs returned by KVM_RUN in response to guest +memory accesses into VM exits (KVM_EXIT_MEMORY_FAULT), with 'gpa' and 'len' +describing the problematic range of memory and 'flags' describing the reason(s) +for the fault. + +The implementation is currently incomplete. Please notify the maintainers if you +come across a case where it needs to be implemented. + +Through args[0], the capability can be set on a per-exit-reason basis. +Currently, the only exit reasons supported are + +1. KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) + +Memory fault exits with a reason of UNKNOWN should not be depended upon: they +may be added, removed, or reclassified under a stable reason. + 8. Other capabilities. ======================arch/x86/kvm/x86.c +1
-0@@ -4425,6 +4425,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_VAPIC :case KVM_CAP_ENABLE_CAP :case KVM_CAP_VM_DISABLE_NX_HUGE_PAGES : +case KVM_CAP_X86_MEMORY_FAULT_EXIT : r =1 ;break ;case KVM_CAP_EXIT_HYPERCALL :include/linux/kvm_host.h +16
-0@@ -805,6 +805,7 @@ struct kvm {struct notifier_block pm_notifier;#endif char stats_id[KVM_STATS_NAME_SIZE ]; +uint64_t memfault_exit_reasons; };#define kvm_err (fmt, ...) \@@ -2278,4 +2279,19 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)/* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +/* +* If memory fault exits are enabled for any of the reasons given in exit_flags +* then sets up a KVM_EXIT_MEMORY_FAULT for the given guest physical address, +* length, and flags and returns -1. +* Otherwise, returns -EFAULT +*/ +inline int kvm_memfault_exit_or_efault ( +struct kvm_vcpu *vcpu,uint64_t gpa,uint64_t len,uint64_t exit_flags); + +/* +* Checks that all of the bits specified in 'reasons' correspond to known +* memory fault exit reasons. +*/ +bool kvm_memfault_exit_flags_valid (uint64_t reasons); +#endif include/uapi/linux/kvm.h +16
-0@@ -264,6 +264,7 @@ struct kvm_xen_exit {#define KVM_EXIT_RISCV_SBI 35#define KVM_EXIT_RISCV_CSR 36#define KVM_EXIT_NOTIFY 37 +#define KVM_EXIT_MEMORY_FAULT 38/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */ @@ -505,6 +506,17 @@ struct kvm_run {#define KVM_NOTIFY_CONTEXT_INVALID (1 << 0)__u32 flags; } notify; +/* KVM_EXIT_MEMORY_FAULT */ +struct { +/* +* Indicates a memory fault on the guest physical address range +* [gpa, gpa + len). flags is a bitfield describing the reasons(s) +* for the fault. +*/ +__u64 flags; +__u64 gpa; +__u64 len;/* in bytes */ + } memory_fault;/* Fix the size of the union. */ char padding[256 ]; };@@ -1184,6 +1196,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224#define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225#define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226 +#define KVM_CAP_X86_MEMORY_FAULT_EXIT 227#ifdef KVM_CAP_IRQ_ROUTING @@ -2237,4 +2250,7 @@ struct kvm_s390_zpci_op {/* flags for kvm_s390_zpci_op->u.reg_aen.flags */ #define KVM_S390_ZPCIOP_REGAEN_HOST (1 << 0) +/* Exit reasons for KVM_EXIT_MEMORY_FAULT */ +#define KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) +#endif /* __LINUX_KVM_H */ tools/include/uapi/linux/kvm.h +15
-0@@ -264,6 +264,7 @@ struct kvm_xen_exit {#define KVM_EXIT_RISCV_SBI 35#define KVM_EXIT_RISCV_CSR 36#define KVM_EXIT_NOTIFY 37 +#define KVM_EXIT_MEMORY_FAULT 38/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */ @@ -505,6 +506,17 @@ struct kvm_run {#define KVM_NOTIFY_CONTEXT_INVALID (1 << 0)__u32 flags; } notify; +/* KVM_EXIT_MEMORY_FAULT */ +struct { +/* +* Indicates a memory fault on the guest physical address range +* [gpa, gpa + len). flags is a bitfield describing the reasons(s) +* for the fault. +*/ +__u64 flags; +__u64 gpa; +__u64 len;/* in bytes */ + } memory_fault;/* Fix the size of the union. */ char padding[256 ]; };@@ -2228,4 +2240,7 @@ struct kvm_s390_zpci_op {/* flags for kvm_s390_zpci_op->u.reg_aen.flags */ #define KVM_S390_ZPCIOP_REGAEN_HOST (1 << 0) +/* Exit reasons for KVM_EXIT_MEMORY_FAULT */ +#define KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) +#endif /* __LINUX_KVM_H */ virt/kvm/kvm_main.c +1
-0@@ -1142,6 +1142,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)spin_lock_init (&kvm->mn_invalidate_lock);rcuwait_init (&kvm->mn_memslots_update_rcuwait);xa_init (&kvm->vcpu_array); + kvm->memfault_exit_reasons =0 ;INIT_LIST_HEAD (&kvm->gpc_list);spin_lock_init (&kvm->gpc_lock);@@ -4671,6 +4672,14 @@ static int kvm_vm_ioctl_enable_cap_generic(struct kvm *kvm,
virt/kvm/kvm_main.c static int kvm_vm_ioctl_enable_cap_generic(struct kvm *kvm, · line 4682
Quoted 10 lines
virt/kvm/kvm_main.c +8
-0return r; } +case KVM_CAP_X86_MEMORY_FAULT_EXIT : { +if (!kvm_vm_ioctl_check_extension (kvm,KVM_CAP_X86_MEMORY_FAULT_EXIT )) +return -EINVAL ; +else if (!kvm_memfault_exit_flags_valid (cap->args[0 ])) +return -EINVAL ; + kvm->memfault_exit_reasons = cap->args[0 ]; +return 0 ; + }
Is KVM_CAP_X86_MEMORY_FAULT_EXIT really specific to x86?
If so, this should go to kvm_vm_ioctl_enable_cap() in arch/x86/kvm/x86.c. (Or make it non-arch specific.)
Quoted 8 lines
virt/kvm/kvm_main.c +1
-0default :return kvm_vm_ioctl_enable_cap (kvm, cap); }@@ -6172,3 +6181,22 @@ int kvm_vm_create_worker_thread(struct kvm *kvm, kvm_vm_thread_fn_t thread_fn,return init_context.err; } +
virt/kvm/kvm_main.c int kvm_vm_create_worker_thread(struct kvm *kvm, kvm_vm_thread_fn_t thread_fn, · line 6194
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+inline int kvm_memfault_exit_or_efault ( +struct kvm_vcpu *vcpu,uint64_t gpa,uint64_t len,uint64_t exit_flags) +{ +if (!(vcpu->kvm->memfault_exit_reasons & exit_flags)) +return -EFAULT ; + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; + vcpu->run->memory_fault.gpa = gpa; + vcpu->run->memory_fault.len = len; + vcpu->run->memory_fault.flags = exit_flags; +return -1 ;
Why -1? 0? Anyway enum exit_fastpath_completion is x86 kvm mmu internal convention. As WIP, it's okay for now, though.
Quoted 11 lines
virt/kvm/kvm_main.c +8
-0+} + +bool kvm_memfault_exit_flags_valid (uint64_t reasons) +{ +uint64_t valid_flags =KVM_MEMFAULT_REASON_UNKNOWN ; + +return !(reasons & !valid_flags); +}Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Signature · 1 line
Isaku Yamahata <[email protected]>
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 6 lines · Isaku Yamahata
Quoted 4 lines
+7.34
KVM_CAP_X86_MEMORY_FAULT_EXIT
+----------------------------------
+
+:Architectures: x86Why x86 specific?
Sean was the only one to bring this functionality up and originally did so in the context of some x86-specific functions, so I assumed that x86 was the only ask and that maybe the other architectures had alternative solutions. Admittedly I also wanted to avoid wading through another big set of -EFAULT references :/
Those are the only reasons though. Marc, Oliver, should I bring this capability to Arm as well?
Quoted 14 lines
Quoted 14 lines
virt/kvm/kvm_main.c +1
-0@@ -1142,6 +1142,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)spin_lock_init (&kvm->mn_invalidate_lock);rcuwait_init (&kvm->mn_memslots_update_rcuwait);xa_init (&kvm->vcpu_array); + kvm->memfault_exit_reasons =0 ;INIT_LIST_HEAD (&kvm->gpc_list);spin_lock_init (&kvm->gpc_lock);@@ -4671,6 +4672,14 @@ static int kvm_vm_ioctl_enable_cap_generic(struct kvm *kvm,
Quoted 14 lines
virt/kvm/kvm_main.c static int kvm_vm_ioctl_enable_cap_generic(struct kvm *kvm, · line 4682
Quoted 10 lines
virt/kvm/kvm_main.c +8
-0return r; } +case KVM_CAP_X86_MEMORY_FAULT_EXIT : { +if (!kvm_vm_ioctl_check_extension (kvm,KVM_CAP_X86_MEMORY_FAULT_EXIT )) +return -EINVAL ; +else if (!kvm_memfault_exit_flags_valid (cap->args[0 ])) +return -EINVAL ; + kvm->memfault_exit_reasons = cap->args[0 ]; +return 0 ; + }Is
KVM_CAP_X86_MEMORY_FAULT_EXITreally specific to x86?
If so, this should go tokvm_vm_ioctl_enable_cap()inarch/x86/kvm/x86.c. (Or make it non-arch specific.)
Ah, thanks for the catch: I renamed my old non-x86 specific capability, and forgot to move this block.
Quoted 5 lines
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+inline int kvm_memfault_exit_or_efault ( +struct kvm_vcpu *vcpu,uint64_t gpa,uint64_t len,uint64_t exit_flags) +{ +if (!(vcpu->kvm->memfault_exit_reasons & exit_flags)) +return -EFAULT ;
Quoted 8 lines
virt/kvm/kvm_main.c int kvm_vm_create_worker_thread(struct kvm *kvm, kvm_vm_thread_fn_t thread_fn, · line 6194
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+ vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; + vcpu->run->memory_fault.gpa = gpa; + vcpu->run->memory_fault.len = len; + vcpu->run->memory_fault.flags = exit_flags; +return -1 ;Why -1? 0? Anyway enum
exit_fastpath_completionis x86 kvm mmu internal convention. As WIP, it's okay for now, though.
The -1 isn't to indicate a failure in this function itself, but to allow callers to substitute this for "return -EFAULT." A return code of zero would mask errors and cause KVM to proceed in ways that it shouldn't. For instance, "setup_vmgexit_scratch" uses it like this
if (kvm_read_guest(svm->vcpu.kvm, scratch_gpa_beg, scratch_va, len)) {
...- return -EFAULT;
+ return kvm_memfault_exit_or_efault(...);
}
and looking at one of its callers (sev_handle_vmgexit) shows how a return code of zero would cause a different control flow
case SVM_VMGEXIT_MMIO_READ:
ret = setup_vmgexit_scratch(svm, true, control->exit_info_2); if (ret)
break;
ret = ret = kvm_sev_es_mmio_read(vcpu,
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 7 lines · Anish Moorthy
Quoted 5 lines · Isaku Yamahata
Quoted 4 lines
+7.34
KVM_CAP_X86_MEMORY_FAULT_EXIT
+----------------------------------
+
+:Architectures: x86
Quoted 7 lines
Quoted 1 line
Why x86 specific?
Sean was the only one to bring this functionality up and originally did so in the context of some x86-specific functions, so I assumed that x86 was the only ask and that maybe the other architectures had alternative solutions. Admittedly I also wanted to avoid wading through another big set of
-EFAULTreferences :/
There isn't much :) Sanity checks in mmu.c and some currently unhandled failures to write guest memory in pvtime.c
Quoted 2 lines
Those are the only reasons though. Marc, Oliver, should I bring this capability to Arm as well?
The x86 implementation shouldn't preclude UAPI reuse, but I'm not strongly motivated in either direction on this. A clear use case where the exit information is actionable rather than just informational would make the change more desirable.
Signature · 2 lines
Thanks, Oliver
Sean Christopherson
maintainerRe: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
In reply to Anish MoorthyQuoted 29 lines · Anish Moorthy
Quoted 5 lines · Isaku Yamahata
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+inline int kvm_memfault_exit_or_efault ( +struct kvm_vcpu *vcpu,uint64_t gpa,uint64_t len,uint64_t exit_flags) +{ +if (!(vcpu->kvm->memfault_exit_reasons & exit_flags)) +return -EFAULT ;Quoted 8 lines
virt/kvm/kvm_main.c int kvm_vm_create_worker_thread(struct kvm *kvm, kvm_vm_thread_fn_t thread_fn, · line 6194
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+ vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; + vcpu->run->memory_fault.gpa = gpa; + vcpu->run->memory_fault.len = len; + vcpu->run->memory_fault.flags = exit_flags; +return -1 ;Why -1? 0? Anyway enum
exit_fastpath_completionis x86 kvm mmu internal convention. As WIP, it's okay for now, though.The -1 isn't to indicate a failure in this function itself, but to allow callers to substitute this for "return
-EFAULT." A return code of zero would mask errors and cause KVM to proceed in ways that it shouldn't. For instance, "setup_vmgexit_scratch" uses it like thisif (kvm_read_guest(svm->vcpu.kvm, scratch_gpa_beg, scratch_va, len)) { ...- return
-EFAULT;
+ returnkvm_memfault_exit_or_efault(...);
}and looking at one of its callers (
sev_handle_vmgexit) shows how a return code of zero would cause a different control flow
Quoted 6 lines
case
SVM_VMGEXIT_MMIO_READ:
ret =setup_vmgexit_scratch(svm, true, control->exit_info_2); if (ret)break;ret = ret = kvm_sev_es_mmio_read(vcpu,
Hmm, I generally agree with Isaku, the helper should really return 0. Returning -1 might work, but it'll likely confuse userspace, and will definitely confuse KVM developers.
The "0 means exit to userspace" behavior is definitely a pain though, and is likely going to make this all extremely fragile.
I wonder if we can get away with returning -EFAULT, but still filling vcpu->run with KVM_EXIT_MEMORY_FAULT and all the other metadata. That would likely simplify the implementation greatly, and would let KVM fill vcpu->run unconditonally. KVM would still need a capability to advertise support to userspace, but userspace wouldn't need to opt in. I think this may have been my very original though, and I just never actually wrote it down...
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 6 lines · Sean Christopherson
I wonder if we can get away with returning
-EFAULT, but still fillingvcpu->runwithKVM_EXIT_MEMORY_FAULTand all the other metadata. That would likely simplify the implementation greatly, and would let KVM fillvcpu->rununconditonally. KVM would still need a capability to advertise support to userspace, but userspace wouldn't need to opt in. I think this may have been my very original though, and I just never actually wrote it down...
Oh, good to know that's actually an option. I thought of that too, but assumed that returning a negative error code was a no-go for a proper vCPU exit. But if that's not true then I think it's the obvious solution because it precludes any uncaught behavior-change bugs.
A couple of notes
1. Since we'll likely miss some -EFAULT returns, we'll need to make sure that the user can check for / doesn't see a stale kvm_run::memory_fault field when a missed -EFAULT makes it to userspace. It's a small and easy-to-fix detail, but I thought I'd point it out.
2. I don't think this would simplify the series that much, since we still need to find the call sites returning -EFAULT to userspace and populate memory_fault only in those spots to avoid populating it for -EFAULTs which don't make it to userspace. We *could* relax that condition and just document that memory_fault should be ignored when KVM_RUN does not return -EFAULT... but I don't think that's a good solution from a coder/maintainer perspective.
Sean Christopherson
maintainerRe: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 13 lines · Anish Moorthy
Quoted 6 lines · Sean Christopherson
I wonder if we can get away with returning
-EFAULT, but still fillingvcpu->runwithKVM_EXIT_MEMORY_FAULTand all the other metadata. That would likely simplify the implementation greatly, and would let KVM fillvcpu->rununconditonally. KVM would still need a capability to advertise support to userspace, but userspace wouldn't need to opt in. I think this may have been my very original though, and I just never actually wrote it down...Oh, good to know that's actually an option. I thought of that too, but assumed that returning a negative error code was a no-go for a proper vCPU exit. But if that's not true then I think it's the obvious solution because it precludes any uncaught behavior-change bugs.
Quoted 6 lines
A couple of notes
1. Since we'll likely miss some-EFAULTreturns, we'll need to make sure that the user can check for / doesn't see a stalekvm_run::memory_faultfield when a missed-EFAULTmakes it to userspace. It's a small and easy-to-fix detail, but I thought I'd point it out.
Ya, this is the main concern for me as well. I'm not as confident that it's easy-to-fix/avoid though.
Quoted 4 lines
2. I don't think this would simplify the series that much, since we still need to find the call sites returning
-EFAULTto userspace and populatememory_faultonly in those spots to avoid populating it for -EFAULTs which don't make it to userspace.
Filling kvm_run::memory_fault even if KVM never exits to userspace is perfectly ok. It's not ideal, but it's ok.
Quoted 3 lines
We *could* relax that condition and just document that
memory_faultshould be ignored whenKVM_RUNdoes not return-EFAULT... but I don't think that's a good solution from a coder/maintainer perspective.
You've got things backward. memory_fault _must_ be ignored if KVM doesn't return the associated "magic combo", where the magic value is either "0+KVM_EXIT_MEMORY_FAULT" or "-EFAULT+KVM_EXIT_MEMORY_FAULT".
Filling kvm_run::memory_fault but not exiting to userspace is ok because userspace never sees the data, i.e. userspace is completely unaware. This behavior is not ideal from a KVM perspective as allowing KVM to fill the kvm_run union without exiting to userspace can lead to other bugs, e.g. effective corruption of the kvm_run union, but at least from a uABI perspective, the behavior is acceptable.
The reverse, userspace consuming kvm_run::memory_fault without being explicitly told the data is valid, is not ok/safe. KVM's contract is that fields contained in kvm_run's big union are valid if and only if KVM returns '0' and the associated exit reason is set in kvm_run::exit_reason.
From an ABI perspective, I don't see anything fundamentally wrong with bending that rule slightly by saying that kvm_run::memory_fault is valid if KVM returns -EFAULT+KVM_EXIT_MEMORY_FAULT. It won't break existing userspace that is unaware of KVM_EXIT_MEMORY_FAULT, and userspace can precisely check for the combination.
My big concern with piggybacking -EFAULT is that userspace will be fed stale if KVM exits with -EFAULT in a patch that _doesn't_ fill kvm_run::memory_fault. Returning a negative error code isn't hazardous in and of itself, e.g. KVM has had bugs in the past where KVM returns '0' but doesn't fill kvm_run::exit_reason. The big danger is that KVM has existing paths that return -EFAULT, i.e. we can introduce bugs simply by doing nothing, whereas returning '0' would largely be limited to new code.
The counter-argument is that propagating '0' correctly up the stack carries its own risk due to plenty of code correctly treating '0' as "success" and not "exit to userspace".
And we can mitigate the risk of using -EFAULT. E.g. fill in kvm_run::memory_fault even if we are 99.9999% confident the -EFAULT can't get out to userspace in the context of KVM_RUN, and set kvm_run::exit_reason to some arbitrary value at the start of KVM_RUN to prevent reusing memory_fault from a previous userspace exit.
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 25 lines · Sean Christopherson
Quoted 13 lines · Anish Moorthy
Quoted 6 lines · Sean Christopherson
I wonder if we can get away with returning
-EFAULT, but still fillingvcpu->runwithKVM_EXIT_MEMORY_FAULTand all the other metadata. That would likely simplify the implementation greatly, and would let KVM fillvcpu->rununconditonally. KVM would still need a capability to advertise support to userspace, but userspace wouldn't need to opt in. I think this may have been my very original though, and I just never actually wrote it down...Oh, good to know that's actually an option. I thought of that too, but assumed that returning a negative error code was a no-go for a proper vCPU exit. But if that's not true then I think it's the obvious solution because it precludes any uncaught behavior-change bugs.
Quoted 6 lines
A couple of notes
1. Since we'll likely miss some-EFAULTreturns, we'll need to make sure that the user can check for / doesn't see a stalekvm_run::memory_faultfield when a missed-EFAULTmakes it to userspace. It's a small and easy-to-fix detail, but I thought I'd point it out.Ya, this is the main concern for me as well. I'm not as confident that it's easy-to-fix/avoid though.
Quoted 7 lines
Quoted 4 lines
2. I don't think this would simplify the series that much, since we still need to find the call sites returning
-EFAULTto userspace and populatememory_faultonly in those spots to avoid populating it for -EFAULTs which don't make it to userspace.Filling
kvm_run::memory_faulteven if KVM never exits to userspace is perfectly ok. It's not ideal, but it's ok.
Right- I was just pointing out that doing so could mislead readers of the code if they assume that kvm_run::memory_fault is populated iff it was going to be associated w/ an exit to userspace," which I know I would.
Quoted 7 lines
Quoted 3 lines
We *could* relax that condition and just document that
memory_faultshould be ignored whenKVM_RUNdoes not return-EFAULT... but I don't think that's a good solution from a coder/maintainer perspective.You've got things backward.
memory_fault_must_ be ignored if KVM doesn't return the associated "magic combo", where the magic value is either "0+KVM_EXIT_MEMORY_FAULT" or "-EFAULT+KVM_EXIT_MEMORY_FAULT".
I think we're saying the same thing- I was using "should" to mean "must."
Quoted 5 lines
Filling
kvm_run::memory_faultbut not exiting to userspace is ok because userspace never sees the data, i.e. userspace is completely unaware. This behavior is not ideal from a KVM perspective as allowing KVM to fill thekvm_rununion without exiting to userspace can lead to other bugs, e.g. effective corruption of thekvm_rununion
Ooh, I didn't think of the corruption issue here: thanks for pointing it out.
Quoted 1 line
but at least from a uABI perspective, the behavior is acceptable.
This does complicate things for KVM implementation though, right? In particular, we'd have to make sure that KVM_RUN never conditionally modifies its return value/exit reason based on reads from kvm_run:
that seems like a slightly weird thing to do, but I don't want to assume anything here.
Anyways, unless that's not (and never will be) a problem, allowing corruption of kvm_run seems very risky.
Quoted 18 lines
The reverse, userspace consuming
kvm_run::memory_faultwithout being explicitly told the data is valid, is not ok/safe. KVM's contract is that fields contained in kvm_run's big union are valid if and only if KVM returns '0' and the associated exit reason is set inkvm_run::exit_reason.From an ABI perspective, I don't see anything fundamentally wrong with bending that rule slightly by saying that
kvm_run::memory_faultis valid if KVM returns -EFAULT+KVM_EXIT_MEMORY_FAULT. It won't break existing userspace that is unaware ofKVM_EXIT_MEMORY_FAULT, and userspace can precisely check for the combination.My big concern with piggybacking
-EFAULTis that userspace will be fed stale if KVM exits with-EFAULTin a patch that _doesn't_ fillkvm_run::memory_fault. Returning a negative error code isn't hazardous in and of itself, e.g. KVM has had bugs in the past where KVM returns '0' but doesn't fillkvm_run::exit_reason. The big danger is that KVM has existing paths that return-EFAULT, i.e. we can introduce bugs simply by doing nothing, whereas returning '0' would largely be limited to new code.
Quoted 8 lines
The counter-argument is that propagating '0' correctly up the stack carries its own risk due to plenty of code correctly treating '0' as "success" and not "exit to userspace".
And we can mitigate the risk of using
-EFAULT. E.g. fill inkvm_run::memory_faulteven if we are 99.9999% confident the-EFAULTcan't get out to userspace in the context ofKVM_RUN, and setkvm_run::exit_reasonto some arbitrary value at the start ofKVM_RUNto prevent reusingmemory_faultfrom a previous userspace exit.
Right, this is what I had in mind when I called this "small and easy-to-fix." Piggybacking -EFAULT seems like the right thing to do to me, but I'm still uneasy about possibly corrupting kvm_run for masked -EFAULTS.
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
In reply to Sean ChristophersonQuoted 41 lines · Sean Christopherson
Quoted 13 lines · Anish Moorthy
Quoted 6 lines · Sean Christopherson
I wonder if we can get away with returning
-EFAULT, but still fillingvcpu->runwithKVM_EXIT_MEMORY_FAULTand all the other metadata. That would likely simplify the implementation greatly, and would let KVM fillvcpu->rununconditonally. KVM would still need a capability to advertise support to userspace, but userspace wouldn't need to opt in. I think this may have been my very original though, and I just never actually wrote it down...Oh, good to know that's actually an option. I thought of that too, but assumed that returning a negative error code was a no-go for a proper vCPU exit. But if that's not true then I think it's the obvious solution because it precludes any uncaught behavior-change bugs.
Quoted 6 lines
A couple of notes
1. Since we'll likely miss some-EFAULTreturns, we'll need to make sure that the user can check for / doesn't see a stalekvm_run::memory_faultfield when a missed-EFAULTmakes it to userspace. It's a small and easy-to-fix detail, but I thought I'd point it out.Ya, this is the main concern for me as well. I'm not as confident that it's easy-to-fix/avoid though.
Quoted 4 lines
2. I don't think this would simplify the series that much, since we still need to find the call sites returning
-EFAULTto userspace and populatememory_faultonly in those spots to avoid populating it for -EFAULTs which don't make it to userspace.Filling
kvm_run::memory_faulteven if KVM never exits to userspace is perfectly ok. It's not ideal, but it's ok.Quoted 3 lines
We *could* relax that condition and just document that
memory_faultshould be ignored whenKVM_RUNdoes not return-EFAULT... but I don't think that's a good solution from a coder/maintainer perspective.You've got things backward.
memory_fault_must_ be ignored if KVM doesn't return the associated "magic combo", where the magic value is either "0+KVM_EXIT_MEMORY_FAULT" or "-EFAULT+KVM_EXIT_MEMORY_FAULT".
Quoted 5 lines
Filling
kvm_run::memory_faultbut not exiting to userspace is ok because userspace never sees the data, i.e. userspace is completely unaware. This behavior is not ideal from a KVM perspective as allowing KVM to fill thekvm_rununion without exiting to userspace can lead to other bugs, e.g. effective corruption of thekvm_rununion, but at least from a uABI perspective, the behavior is acceptable.
Actually, I don't think the idea of filling in kvm_run.memory_fault for -EFAULTs which don't make it to userspace works at all. Consider the direct_map function, which bubbles its -EFAULT to kvm_mmu_do_page_fault. kvm_mmu_do_page_fault is called from both kvm_arch_async_page_ready (which ignores the return value), and by kvm_mmu_page_fault (where the return value does make it to userspace). Populating kvm_run.memory_fault anywhere in or under kvm_mmu_do_page_fault seems an immediate no-go, because a wayward kvm_arch_async_page_ready could (presumably) overwrite/corrupt an already-set kvm_run.memory_fault / other kvm_run field.
That in turn looks problematic for the memory-fault-exit-on-fast-gup-failure part of this series, because there are at least a couple of cases for which kvm_mmu_do_page_fault will -EFAULT. One is the early-efault-on-fast-gup-failure case which was the original purpose of this series. Another is a -EFAULT from FNAME(fetch) (passed up through FNAME(page_fault)). There might be other cases as well. But unless userspace can/should resolve *all* such -EFAULTs in the same manner, a kvm_run.memory_fault populated in "kvm_mmu_page_fault" wouldn't be actionable. At least, not without a whole lot of plumbing code to make it so.
Sean, am I missing anything here?
Sean Christopherson
maintainerRe: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 48 lines · Anish Moorthy
Quoted 41 lines · Sean Christopherson
Quoted 13 lines · Anish Moorthy
Quoted 6 lines · Sean Christopherson
I wonder if we can get away with returning
-EFAULT, but still fillingvcpu->runwithKVM_EXIT_MEMORY_FAULTand all the other metadata. That would likely simplify the implementation greatly, and would let KVM fillvcpu->rununconditonally. KVM would still need a capability to advertise support to userspace, but userspace wouldn't need to opt in. I think this may have been my very original though, and I just never actually wrote it down...Oh, good to know that's actually an option. I thought of that too, but assumed that returning a negative error code was a no-go for a proper vCPU exit. But if that's not true then I think it's the obvious solution because it precludes any uncaught behavior-change bugs.
Quoted 6 lines
A couple of notes
1. Since we'll likely miss some-EFAULTreturns, we'll need to make sure that the user can check for / doesn't see a stalekvm_run::memory_faultfield when a missed-EFAULTmakes it to userspace. It's a small and easy-to-fix detail, but I thought I'd point it out.Ya, this is the main concern for me as well. I'm not as confident that it's easy-to-fix/avoid though.
Quoted 4 lines
2. I don't think this would simplify the series that much, since we still need to find the call sites returning
-EFAULTto userspace and populatememory_faultonly in those spots to avoid populating it for -EFAULTs which don't make it to userspace.Filling
kvm_run::memory_faulteven if KVM never exits to userspace is perfectly ok. It's not ideal, but it's ok.Quoted 3 lines
We *could* relax that condition and just document that
memory_faultshould be ignored whenKVM_RUNdoes not return-EFAULT... but I don't think that's a good solution from a coder/maintainer perspective.You've got things backward.
memory_fault_must_ be ignored if KVM doesn't return the associated "magic combo", where the magic value is either "0+KVM_EXIT_MEMORY_FAULT" or "-EFAULT+KVM_EXIT_MEMORY_FAULT".Quoted 5 lines
Filling
kvm_run::memory_faultbut not exiting to userspace is ok because userspace never sees the data, i.e. userspace is completely unaware. This behavior is not ideal from a KVM perspective as allowing KVM to fill thekvm_rununion without exiting to userspace can lead to other bugs, e.g. effective corruption of thekvm_rununion, but at least from a uABI perspective, the behavior is acceptable.
Quoted 10 lines
Actually, I don't think the idea of filling in kvm_run.memory_fault for -EFAULTs which don't make it to userspace works at all. Consider the
direct_mapfunction, which bubbles its-EFAULTtokvm_mmu_do_page_fault.kvm_mmu_do_page_faultis called from bothkvm_arch_async_page_ready(which ignores the return value), and bykvm_mmu_page_fault(where the return value does make it to userspace). Populating kvm_run.memory_fault anywhere in or underkvm_mmu_do_page_faultseems an immediate no-go, because a waywardkvm_arch_async_page_readycould (presumably) overwrite/corrupt an already-set kvm_run.memory_fault / otherkvm_runfield.
This particular case is a non-issue. kvm_check_async_pf_completion() is called only when the current task has control of the vCPU, i.e. is the current "running" vCPU. That's not a coincidence either, invoking kvm_mmu_do_page_fault() without having control of the vCPU would be fraught with races, e.g. the entire KVM MMU context would be unstable.
That will hold true for all cases. Using a vCPU that is not loaded (not the current "running" vCPU in KVM's misleading terminology) to access guest memory is simply not safe, as the vCPU state is non-deterministic. There are paths where KVM accesses, and even modifies, vCPU state asynchronously, e.g. for IRQ delivery and making requests, but those are very controlled flows with dedicated machinery to make them SMP safe.
That said, I agree that there's a risk that KVM could clobber vcpu->run_run by hitting an -EFAULT without the vCPU loaded, but that's a solvable problem, e.g. the helper to fill KVM_EXIT_MEMORY_FAULT could be hardened to yell if called without the target vCPU being loaded:
int kvm_handle_efault(struct kvm_vcpu *vcpu, ...)
{
preempt_disable();
if (WARN_ON_ONCE(vcpu != __this_cpu_read(kvm_running_vcpu)))
goto out;
vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT;
...
out:
preempt_enable();
return -EFAULT;
}FWIW, I completely agree that filling KVM_EXIT_MEMORY_FAULT without guaranteeing that KVM "immediately" exits to userspace isn't ideal, but given the amount of historical code that we need to deal with, it seems like the lesser of all evils. Unless I'm misunderstanding the use cases, unnecessarily filling kvm_run is a far better failure mode than KVM not filling kvm_run when it should, i.e. false positives are ok, false negatives are fatal.
Quoted 9 lines
That in turn looks problematic for the memory-fault-exit-on-fast-gup-failure part of this series, because there are at least a couple of cases for which
kvm_mmu_do_page_faultwill-EFAULT. One is the early-efault-on-fast-gup-failure case which was the original purpose of this series. Another is a-EFAULTfrom FNAME(fetch) (passed up throughFNAME(page_fault)). There might be other cases as well. But unless userspace can/should resolve *all* such -EFAULTs in the same manner, a kvm_run.memory_fault populated in "kvm_mmu_page_fault" wouldn't be actionable.
Killing the VM, which is what all VMMs do today in response to -EFAULT, is an action. As I've pointed out elsewhere in this thread, userspace needs to be able to identify "faults" that it (userspace) can resolve without a hint from KVM.
In other words, KVM is still returning -EFAULT (or a variant thereof), the _only_ difference, for all intents and purposes, is that userspace is given a bit more information about the source of the -EFAULT.
Quoted 1 line
At least, not without a whole lot of plumbing code to make it so.
Plumbing where?
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 92 lines · Sean Christopherson
Quoted 48 lines · Anish Moorthy
Quoted 41 lines · Sean Christopherson
Quoted 13 lines · Anish Moorthy
Quoted 6 lines · Sean Christopherson
I wonder if we can get away with returning
-EFAULT, but still fillingvcpu->runwithKVM_EXIT_MEMORY_FAULTand all the other metadata. That would likely simplify the implementation greatly, and would let KVM fillvcpu->rununconditonally. KVM would still need a capability to advertise support to userspace, but userspace wouldn't need to opt in. I think this may have been my very original though, and I just never actually wrote it down...Oh, good to know that's actually an option. I thought of that too, but assumed that returning a negative error code was a no-go for a proper vCPU exit. But if that's not true then I think it's the obvious solution because it precludes any uncaught behavior-change bugs.
Quoted 6 lines
A couple of notes
1. Since we'll likely miss some-EFAULTreturns, we'll need to make sure that the user can check for / doesn't see a stalekvm_run::memory_faultfield when a missed-EFAULTmakes it to userspace. It's a small and easy-to-fix detail, but I thought I'd point it out.Ya, this is the main concern for me as well. I'm not as confident that it's easy-to-fix/avoid though.
Quoted 4 lines
2. I don't think this would simplify the series that much, since we still need to find the call sites returning
-EFAULTto userspace and populatememory_faultonly in those spots to avoid populating it for -EFAULTs which don't make it to userspace.Filling
kvm_run::memory_faulteven if KVM never exits to userspace is perfectly ok. It's not ideal, but it's ok.Quoted 3 lines
We *could* relax that condition and just document that
memory_faultshould be ignored whenKVM_RUNdoes not return-EFAULT... but I don't think that's a good solution from a coder/maintainer perspective.You've got things backward.
memory_fault_must_ be ignored if KVM doesn't return the associated "magic combo", where the magic value is either "0+KVM_EXIT_MEMORY_FAULT" or "-EFAULT+KVM_EXIT_MEMORY_FAULT".Quoted 5 lines
Filling
kvm_run::memory_faultbut not exiting to userspace is ok because userspace never sees the data, i.e. userspace is completely unaware. This behavior is not ideal from a KVM perspective as allowing KVM to fill thekvm_rununion without exiting to userspace can lead to other bugs, e.g. effective corruption of thekvm_rununion, but at least from a uABI perspective, the behavior is acceptable.Quoted 10 lines
Actually, I don't think the idea of filling in kvm_run.memory_fault for -EFAULTs which don't make it to userspace works at all. Consider the
direct_mapfunction, which bubbles its-EFAULTtokvm_mmu_do_page_fault.kvm_mmu_do_page_faultis called from bothkvm_arch_async_page_ready(which ignores the return value), and bykvm_mmu_page_fault(where the return value does make it to userspace). Populating kvm_run.memory_fault anywhere in or underkvm_mmu_do_page_faultseems an immediate no-go, because a waywardkvm_arch_async_page_readycould (presumably) overwrite/corrupt an already-set kvm_run.memory_fault / otherkvm_runfield.This particular case is a non-issue.
kvm_check_async_pf_completion()is called only when the current task has control of the vCPU, i.e. is the current "running" vCPU. That's not a coincidence either, invokingkvm_mmu_do_page_fault()without having control of the vCPU would be fraught with races, e.g. the entire KVM MMU context would be unstable.That will hold true for all cases. Using a vCPU that is not loaded (not the current "running" vCPU in KVM's misleading terminology) to access guest memory is simply not safe, as the vCPU state is non-deterministic. There are paths where KVM accesses, and even modifies, vCPU state asynchronously, e.g. for IRQ delivery and making requests, but those are very controlled flows with dedicated machinery to make them SMP safe.
That said, I agree that there's a risk that KVM could clobber
vcpu->run_runby hitting an-EFAULTwithout the vCPU loaded, but that's a solvable problem, e.g. the helper to fillKVM_EXIT_MEMORY_FAULTcould be hardened to yell if called without the target vCPU being loaded:int kvm_handle_efault(struct kvm_vcpu *vcpu, ...) { preempt_disable(); if (WARN_ON_ONCE(vcpu != __this_cpu_read(kvm_running_vcpu))) goto out; vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT; ... out: preempt_enable(); return -EFAULT; }
Quoted 6 lines
FWIW, I completely agree that filling
KVM_EXIT_MEMORY_FAULTwithout guaranteeing that KVM "immediately" exits to userspace isn't ideal, but given the amount of historical code that we need to deal with, it seems like the lesser of all evils. Unless I'm misunderstanding the use cases, unnecessarily fillingkvm_runis a far better failure mode than KVM not fillingkvm_runwhen it should, i.e. false positives are ok, false negatives are fatal.
Don't you have this in reverse? False negatives will just result in userspace not having useful extra information for the -EFAULT it receives from KVM_RUN, in which case userspace can do what you mentioned all VMMs do today and just terminate the VM. Whereas a false positive might cause a double-write to the KVM_RUN struct, either putting incorrect information in kvm_run.memory_fault or corrupting another member of the union.
Quoted 14 lines
Quoted 9 lines
That in turn looks problematic for the memory-fault-exit-on-fast-gup-failure part of this series, because there are at least a couple of cases for which
kvm_mmu_do_page_faultwill-EFAULT. One is the early-efault-on-fast-gup-failure case which was the original purpose of this series. Another is a-EFAULTfrom FNAME(fetch) (passed up throughFNAME(page_fault)). There might be other cases as well. But unless userspace can/should resolve *all* such -EFAULTs in the same manner, a kvm_run.memory_fault populated in "kvm_mmu_page_fault" wouldn't be actionable.Killing the VM, which is what all VMMs do today in response to
-EFAULT, is an action. As I've pointed out elsewhere in this thread, userspace needs to be able to identify "faults" that it (userspace) can resolve without a hint from KVM.
Quoted 7 lines
In other words, KVM is still returning
-EFAULT(or a variant thereof), the _only_ difference, for all intents and purposes, is that userspace is given a bit more information about the source of the-EFAULT.Quoted 1 line
At least, not without a whole lot of plumbing code to make it so.
Plumbing where?
In this example, I meant plumbing code to get a kvm_run.memory_fault.flags which is more specific than (eg) MEMFAULT_REASON_PAGE_FAULT_FAILURE from the -EFAULT paths under kvm_mmu_page_fault. My idea for how userspace would distinguish fast-gup failures was that kvm_faultin_pfn would set a special bit in kvm_run.memory_fault.flags to indicate its failure. But (still assuming that we shouldn't have false-positive kvm_run.memory_fault fills) if the memory_fault can only be populated from kvm_mmu_page_fault then either failures from FNAME(page_fault) and kvm_faultin_pfn will be indistinguishable to userspace, or those functions will need to plumb more specific exit reasons all the way up to kvm_mmu_page_fault.
But, since you've made this point elsewhere, my guess is that your answer is that it's actually userspace's job to detect the "specific" reason for the fault and resolve it.
Sean Christopherson
maintainerRe: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 52 lines · Anish Moorthy
Quoted 51 lines · Sean Christopherson
Quoted 7 lines · Anish Moorthy
Quoted 5 lines · Sean Christopherson
Filling
kvm_run::memory_faultbut not exiting to userspace is ok because userspace never sees the data, i.e. userspace is completely unaware. This behavior is not ideal from a KVM perspective as allowing KVM to fill thekvm_rununion without exiting to userspace can lead to other bugs, e.g. effective corruption of thekvm_rununion, but at least from a uABI perspective, the behavior is acceptable.Quoted 10 lines
Actually, I don't think the idea of filling in kvm_run.memory_fault for -EFAULTs which don't make it to userspace works at all. Consider the
direct_mapfunction, which bubbles its-EFAULTtokvm_mmu_do_page_fault.kvm_mmu_do_page_faultis called from bothkvm_arch_async_page_ready(which ignores the return value), and bykvm_mmu_page_fault(where the return value does make it to userspace). Populating kvm_run.memory_fault anywhere in or underkvm_mmu_do_page_faultseems an immediate no-go, because a waywardkvm_arch_async_page_readycould (presumably) overwrite/corrupt an already-set kvm_run.memory_fault / otherkvm_runfield.This particular case is a non-issue.
kvm_check_async_pf_completion()is called only when the current task has control of the vCPU, i.e. is the current "running" vCPU. That's not a coincidence either, invokingkvm_mmu_do_page_fault()without having control of the vCPU would be fraught with races, e.g. the entire KVM MMU context would be unstable.That will hold true for all cases. Using a vCPU that is not loaded (not the current "running" vCPU in KVM's misleading terminology) to access guest memory is simply not safe, as the vCPU state is non-deterministic. There are paths where KVM accesses, and even modifies, vCPU state asynchronously, e.g. for IRQ delivery and making requests, but those are very controlled flows with dedicated machinery to make them SMP safe.
That said, I agree that there's a risk that KVM could clobber
vcpu->run_runby hitting an-EFAULTwithout the vCPU loaded, but that's a solvable problem, e.g. the helper to fillKVM_EXIT_MEMORY_FAULTcould be hardened to yell if called without the target vCPU being loaded:int kvm_handle_efault(struct kvm_vcpu *vcpu, ...) { preempt_disable(); if (WARN_ON_ONCE(vcpu != __this_cpu_read(kvm_running_vcpu))) goto out; vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT; ... out: preempt_enable(); return -EFAULT; }
Quoted 8 lines
Quoted 6 lines
FWIW, I completely agree that filling
KVM_EXIT_MEMORY_FAULTwithout guaranteeing that KVM "immediately" exits to userspace isn't ideal, but given the amount of historical code that we need to deal with, it seems like the lesser of all evils. Unless I'm misunderstanding the use cases, unnecessarily fillingkvm_runis a far better failure mode than KVM not fillingkvm_runwhen it should, i.e. false positives are ok, false negatives are fatal.Don't you have this in reverse?
No, I don't think so.
Quoted 3 lines
False negatives will just result in userspace not having useful extra information for the
-EFAULTit receives fromKVM_RUN, in which case userspace can do what you mentioned all VMMs do today and just terminate the VM.
And that is _really_ bad behavior if we have any hope of userspace actually being able to rely on this functionality. E.g. any false negative when userspace is trying to do postcopy demand paging will be fatal to the VM.
Quoted 2 lines
Whereas a false positive might cause a double-write to the
KVM_RUNstruct, either putting incorrect information in kvm_run.memory_fault or
Recording unused information on -EFAULT in kvm_run doesn't make the information incorrect.
Quoted 1 line
corrupting another member of the union.
Only if KVM accesses guest memory after initiating an exit to userspace, which would be a KVM irrespective of kvm_run.memory_fault. We actually have exactly this type of bug today in the trainwreck that is KVM's MMIO emulation[*], but KVM gets away with the shoddy behavior by virtue of the scenario simply not triggered by any real-world code.
And if we're really concerned about clobbering state, we could add hardening/auditing code to ensure that KVM actually exits when kvm_run.exit_reason is set (though there are a non-zero number of exceptions, e.g. the aformentioned MMIO mess, nested SVM/VMX pages, and probably a few others).
Prior to cleanups a few years back[2], emulation failures had issues similar to what we are discussing, where KVM would fail to exit to userspace, not fill kvm_run, etc. Those are the types of bugs I want to avoid here.
[1] https://lkml.kernel.org/r/ZBNrWZQhMX8AHzWM%40google.com
[2] https://lore.kernel.org/kvm/[email protected]
Quoted 22 lines
Quoted 14 lines
Quoted 9 lines
That in turn looks problematic for the memory-fault-exit-on-fast-gup-failure part of this series, because there are at least a couple of cases for which
kvm_mmu_do_page_faultwill-EFAULT. One is the early-efault-on-fast-gup-failure case which was the original purpose of this series. Another is a-EFAULTfrom FNAME(fetch) (passed up throughFNAME(page_fault)). There might be other cases as well. But unless userspace can/should resolve *all* such -EFAULTs in the same manner, a kvm_run.memory_fault populated in "kvm_mmu_page_fault" wouldn't be actionable.Killing the VM, which is what all VMMs do today in response to
-EFAULT, is an action. As I've pointed out elsewhere in this thread, userspace needs to be able to identify "faults" that it (userspace) can resolve without a hint from KVM.Quoted 7 lines
In other words, KVM is still returning
-EFAULT(or a variant thereof), the _only_ difference, for all intents and purposes, is that userspace is given a bit more information about the source of the-EFAULT.Quoted 1 line
At least, not without a whole lot of plumbing code to make it so.
Plumbing where?
Quoted 10 lines
In this example, I meant plumbing code to get a kvm_run.memory_fault.flags which is more specific than (eg)
MEMFAULT_REASON_PAGE_FAULT_FAILUREfrom the-EFAULTpaths underkvm_mmu_page_fault. My idea for how userspace would distinguish fast-gup failures was thatkvm_faultin_pfnwould set a special bit in kvm_run.memory_fault.flags to indicate its failure. But (still assuming that we shouldn't have false-positive kvm_run.memory_fault fills) if thememory_faultcan only be populated fromkvm_mmu_page_faultthen either failures fromFNAME(page_fault)andkvm_faultin_pfnwill be indistinguishable to userspace, or those functions will need to plumb more specific exit reasons all the way up tokvm_mmu_page_fault.
Setting a flag that essentially says "failure when handling a guest page fault" is problematic on multiple fronts. Tying the ABI to KVM's internal implementation is not an option, i.e. the ABI would need to be defined as "on page faults from the guest". And then the resulting behavior would be non-deterministic, e.g. userspace would see different behavior if KVM accessed a "bad" gfn via emulation instead of in response to a guest page fault. And because of hardware TLBs, it would even be possible for the behavior to be non-deterministic on the same platform running the same guest code (though this would be exteremly unliklely in practice).
And even if userspace is ok with only handling guest page faults_today_, I highly doubt that will hold forever. I.e. at some point there will be a use case that wants to react to uaccess failures on fast-only memslots.
Ignoring all of those issues, simplify flagging "this -EFAULT occurred when handling a guest page fault" isn't precise enough for userspace to blindly resolve the failure. Even if KVM went through the trouble of setting information if and only if get_user_page_fast_only() failed while handling a guest page fault, userspace would still need/want a way to verify that the failure was expected and can be resolved, e.g. to guard against userspace bugs due to wrongly unmapping or mprotecting a page.
Quoted 3 lines
But, since you've made this point elsewhere, my guess is that your answer is that it's actually userspace's job to detect the "specific" reason for the fault and resolve it.
Yes, it's userspace's responsibity. I simply don't see how KVM can provide information that userspace doesn't already have without creating an unmaintainable uABI, at least not without some deep, deep plumbing into gup(). I.e. unless gup() were changed to explicitly communicate that it failed because of a uffd equivalent, at best a flag in kvm_run would be a hint that userspace _might_ be able to resolve the fault. And even if we modified gup(), we'd still have all the open questions about what to do when KVM encounters a fault on a uaccess.
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 65 lines · Sean Christopherson
Quoted 52 lines · Anish Moorthy
Quoted 51 lines · Sean Christopherson
Quoted 7 lines · Anish Moorthy
Quoted 5 lines · Sean Christopherson
Filling
kvm_run::memory_faultbut not exiting to userspace is ok because userspace never sees the data, i.e. userspace is completely unaware. This behavior is not ideal from a KVM perspective as allowing KVM to fill thekvm_rununion without exiting to userspace can lead to other bugs, e.g. effective corruption of thekvm_rununion, but at least from a uABI perspective, the behavior is acceptable.Quoted 10 lines
Actually, I don't think the idea of filling in kvm_run.memory_fault for -EFAULTs which don't make it to userspace works at all. Consider the
direct_mapfunction, which bubbles its-EFAULTtokvm_mmu_do_page_fault.kvm_mmu_do_page_faultis called from bothkvm_arch_async_page_ready(which ignores the return value), and bykvm_mmu_page_fault(where the return value does make it to userspace). Populating kvm_run.memory_fault anywhere in or underkvm_mmu_do_page_faultseems an immediate no-go, because a waywardkvm_arch_async_page_readycould (presumably) overwrite/corrupt an already-set kvm_run.memory_fault / otherkvm_runfield.This particular case is a non-issue.
kvm_check_async_pf_completion()is called only when the current task has control of the vCPU, i.e. is the current "running" vCPU. That's not a coincidence either, invokingkvm_mmu_do_page_fault()without having control of the vCPU would be fraught with races, e.g. the entire KVM MMU context would be unstable.That will hold true for all cases. Using a vCPU that is not loaded (not the current "running" vCPU in KVM's misleading terminology) to access guest memory is simply not safe, as the vCPU state is non-deterministic. There are paths where KVM accesses, and even modifies, vCPU state asynchronously, e.g. for IRQ delivery and making requests, but those are very controlled flows with dedicated machinery to make them SMP safe.
That said, I agree that there's a risk that KVM could clobber
vcpu->run_runby hitting an-EFAULTwithout the vCPU loaded, but that's a solvable problem, e.g. the helper to fillKVM_EXIT_MEMORY_FAULTcould be hardened to yell if called without the target vCPU being loaded:int kvm_handle_efault(struct kvm_vcpu *vcpu, ...) { preempt_disable(); if (WARN_ON_ONCE(vcpu != __this_cpu_read(kvm_running_vcpu))) goto out; vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT; ... out: preempt_enable(); return -EFAULT; }Quoted 8 lines
Quoted 6 lines
FWIW, I completely agree that filling
KVM_EXIT_MEMORY_FAULTwithout guaranteeing that KVM "immediately" exits to userspace isn't ideal, but given the amount of historical code that we need to deal with, it seems like the lesser of all evils. Unless I'm misunderstanding the use cases, unnecessarily fillingkvm_runis a far better failure mode than KVM not fillingkvm_runwhen it should, i.e. false positives are ok, false negatives are fatal.Don't you have this in reverse?
No, I don't think so.
Quoted 7 lines
Quoted 3 lines
False negatives will just result in userspace not having useful extra information for the
-EFAULTit receives fromKVM_RUN, in which case userspace can do what you mentioned all VMMs do today and just terminate the VM.And that is _really_ bad behavior if we have any hope of userspace actually being able to rely on this functionality. E.g. any false negative when userspace is trying to do postcopy demand paging will be fatal to the VM.
But since -EFAULTs from KVM_RUN today are already fatal, so there's no new failure introduced by an -EFAULT w/o a populated memory_fault field right? Obviously that's of no real use to userspace, but that seems like part of the point of starting with a partial conversion: to allow for filling holes in the implementation in the future.
It seems like what you're really concerned about here is the interaction with the memslot fast-gup-only flag. Obviously, failing to populate kvm_run.memory_fault for new userspace-visible -EFAULTs caused by that flag would cause new fatal failures for the guest, which would make the feature actually harmful. But as far as I know (and please lmk if I'm wrong), the memslot flag only needs to be used by the kvm_handle_error_pfn (x86) and user_mem_abort (arm64) functions, meaning that those are the only places where we need to check/populate kvm_run.memory_fault for new userspace-visible -EFAULTs.
Quoted 10 lines
Quoted 2 lines
Whereas a false positive might cause a double-write to the
KVM_RUNstruct, either putting incorrect information in kvm_run.memory_fault orRecording unused information on
-EFAULTinkvm_rundoesn't make the information incorrect.Quoted 1 line
corrupting another member of the union.
Only if KVM accesses guest memory after initiating an exit to userspace, which would be a KVM irrespective of kvm_run.memory_fault.
Ah good: I was concerned that this was a valid set of code paths in KVM. Although I'm assuming that "initiating an exit to userspace" includes the "returning -EFAULT from KVM_RUN" cases, because we wouldn't want EFAULTs to stomp on each other as well (the kvm_mmu_do_page_fault usages were supposed to be one such example, though I'm glad to know that they're not a problem).
Quoted 78 lines
And if we're really concerned about clobbering state, we could add hardening/auditing code to ensure that KVM actually exits when kvm_run.exit_reason is set (though there are a non-zero number of exceptions, e.g. the aformentioned MMIO mess, nested SVM/VMX pages, and probably a few others).
Prior to cleanups a few years back[2], emulation failures had issues similar to what we are discussing, where KVM would fail to exit to userspace, not fill
kvm_run, etc. Those are the types of bugs I want to avoid here.[1] https://lkml.kernel.org/r/ZBNrWZQhMX8AHzWM%40google.com
[2] https://lore.kernel.org/kvm/[email protected]Quoted 22 lines
Quoted 14 lines
Quoted 9 lines
That in turn looks problematic for the memory-fault-exit-on-fast-gup-failure part of this series, because there are at least a couple of cases for which
kvm_mmu_do_page_faultwill-EFAULT. One is the early-efault-on-fast-gup-failure case which was the original purpose of this series. Another is a-EFAULTfrom FNAME(fetch) (passed up throughFNAME(page_fault)). There might be other cases as well. But unless userspace can/should resolve *all* such -EFAULTs in the same manner, a kvm_run.memory_fault populated in "kvm_mmu_page_fault" wouldn't be actionable.Killing the VM, which is what all VMMs do today in response to
-EFAULT, is an action. As I've pointed out elsewhere in this thread, userspace needs to be able to identify "faults" that it (userspace) can resolve without a hint from KVM.Quoted 7 lines
In other words, KVM is still returning
-EFAULT(or a variant thereof), the _only_ difference, for all intents and purposes, is that userspace is given a bit more information about the source of the-EFAULT.Quoted 1 line
At least, not without a whole lot of plumbing code to make it so.
Plumbing where?
Quoted 10 lines
In this example, I meant plumbing code to get a kvm_run.memory_fault.flags which is more specific than (eg)
MEMFAULT_REASON_PAGE_FAULT_FAILUREfrom the-EFAULTpaths underkvm_mmu_page_fault. My idea for how userspace would distinguish fast-gup failures was thatkvm_faultin_pfnwould set a special bit in kvm_run.memory_fault.flags to indicate its failure. But (still assuming that we shouldn't have false-positive kvm_run.memory_fault fills) if thememory_faultcan only be populated fromkvm_mmu_page_faultthen either failures fromFNAME(page_fault)andkvm_faultin_pfnwill be indistinguishable to userspace, or those functions will need to plumb more specific exit reasons all the way up tokvm_mmu_page_fault.Setting a flag that essentially says "failure when handling a guest page fault" is problematic on multiple fronts. Tying the ABI to KVM's internal implementation is not an option, i.e. the ABI would need to be defined as "on page faults from the guest". And then the resulting behavior would be non-deterministic, e.g. userspace would see different behavior if KVM accessed a "bad" gfn via emulation instead of in response to a guest page fault. And because of hardware TLBs, it would even be possible for the behavior to be non-deterministic on the same platform running the same guest code (though this would be exteremly unliklely in practice).
And even if userspace is ok with only handling guest page
faults_today_, I highly doubt that will hold forever. I.e. at some point there will be a use case that wants to react to uaccess failures on fast-only memslots.Ignoring all of those issues, simplify flagging "this
-EFAULToccurred when handling a guest page fault" isn't precise enough for userspace to blindly resolve the failure. Even if KVM went through the trouble of setting information if and only ifget_user_page_fast_only()failed while handling a guest page fault, userspace would still need/want a way to verify that the failure was expected and can be resolved, e.g. to guard against userspace bugs due to wrongly unmapping or mprotecting a page.Quoted 3 lines
But, since you've made this point elsewhere, my guess is that your answer is that it's actually userspace's job to detect the "specific" reason for the fault and resolve it.
Yes, it's userspace's responsibity. I simply don't see how KVM can provide information that userspace doesn't already have without creating an unmaintainable uABI, at least not without some deep, deep plumbing into
gup(). I.e. unlessgup()were changed to explicitly communicate that it failed because of a uffd equivalent, at best a flag inkvm_runwould be a hint that userspace _might_ be able to resolve the fault. And even if we modifiedgup(), we'd still have all the open questions about what to do when KVM encounters a fault on a uaccess.
Sean Christopherson
maintainerRe: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 18 lines · Anish Moorthy
Quoted 17 lines · Sean Christopherson
Quoted 8 lines · Anish Moorthy
Quoted 6 lines
FWIW, I completely agree that filling
KVM_EXIT_MEMORY_FAULTwithout guaranteeing that KVM "immediately" exits to userspace isn't ideal, but given the amount of historical code that we need to deal with, it seems like the lesser of all evils. Unless I'm misunderstanding the use cases, unnecessarily fillingkvm_runis a far better failure mode than KVM not fillingkvm_runwhen it should, i.e. false positives are ok, false negatives are fatal.Don't you have this in reverse?
No, I don't think so.
Quoted 3 lines
False negatives will just result in userspace not having useful extra information for the
-EFAULTit receives fromKVM_RUN, in which case userspace can do what you mentioned all VMMs do today and just terminate the VM.
Quoted 7 lines
Quoted 3 lines
And that is _really_ bad behavior if we have any hope of userspace actually being able to rely on this functionality. E.g. any false negative when userspace is trying to do postcopy demand paging will be fatal to the VM.
But since -EFAULTs from
KVM_RUNtoday are already fatal, so there's no new failure introduced by an-EFAULTw/o a populatedmemory_faultfield right?
Yes, but it's a bit of a moot piont since the goal of the feature is to avoid killing the VM.
Quoted 3 lines
Obviously that's of no real use to userspace, but that seems like part of the point of starting with a partial conversion: to allow for filling holes in the implementation in the future.
Yes, but I want a forcing function to reveal any holes we missed sooner than later, otherwise the feature will languish since it won't be useful beyond the fast-gup-only use case.
Quoted 9 lines
It seems like what you're really concerned about here is the interaction with the memslot fast-gup-only flag. Obviously, failing to populate kvm_run.memory_fault for new userspace-visible -EFAULTs caused by that flag would cause new fatal failures for the guest, which would make the feature actually harmful. But as far as I know (and please lmk if I'm wrong), the memslot flag only needs to be used by the
kvm_handle_error_pfn(x86) anduser_mem_abort(arm64) functions, meaning that those are the only places where we need to check/populate kvm_run.memory_fault for new userspace-visible -EFAULTs.
No. As you point out, the fast-gup-only case should be pretty easy to get correct, i.e. this should all work just fine for _GCE's current_ use case. I'm more concerned with setting KVM up for success when future use cases come along that might not be ok with unhandled faults in random guest accesses killing the VM.
To be clear, I do not expect us to get this 100% correct on the first attempt, but I do want to have mechanisms in place that will detect any bugs/misses so that we can fix the issues _before_ a use case comes along that needs 100% accuracy.
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
Whereas a false positive might cause a double-write to the
KVM_RUNstruct, either putting incorrect information in kvm_run.memory_fault orRecording unused information on
-EFAULTinkvm_rundoesn't make the information incorrect.Quoted 1 line
corrupting another member of the union.
Only if KVM accesses guest memory after initiating an exit to userspace, which would be a KVM irrespective of kvm_run.memory_fault.
Quoted 6 lines
Ah good: I was concerned that this was a valid set of code paths in KVM. Although I'm assuming that "initiating an exit to userspace" includes the "returning
-EFAULTfromKVM_RUN" cases, because we wouldn't want EFAULTs to stomp on each other as well (thekvm_mmu_do_page_faultusages were supposed to be one such example, though I'm glad to know that they're not a problem).
This one gets into a bit of a grey area. The "rule" is really about the intent, i.e. once KVM intends to exit to userspace, it's a bug if KVM encounters something else and runs into the weeds.
In no small part because of the myriad paths where KVM ignores what be fatal errors in most flows, e.g. record_steal_time(), simply returning -EFAULT from some low level helper doesn't necessarily signal an intent to exit all the way to userspace.
To be honest, I don't have a clear idea of how difficult it will be to detect bugs. In most cases, failure to exit to userspace leads to a fatal error fairly quickly. With userspace faults, it's entirely possible that an exit could be missed and nothing bad would happen.
Hmm, one idea would be to have the initial -EFAULT detection fill kvm_run.memory_fault, but set kvm_run.exit_reason to some magic number, e.g. zero it out. Then KVM could WARN if something tries to overwrite kvm_run.exit_reason. The WARN would need to be buried by a Kconfig or something since kvm_run can be modified by userspace, but other than that I think it would work.
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
In reply to Sean ChristophersonQuoted 73 lines · Sean Christopherson
Quoted 52 lines · Anish Moorthy
Quoted 51 lines · Sean Christopherson
Quoted 7 lines · Anish Moorthy
Quoted 5 lines · Sean Christopherson
Filling
kvm_run::memory_faultbut not exiting to userspace is ok because userspace never sees the data, i.e. userspace is completely unaware. This behavior is not ideal from a KVM perspective as allowing KVM to fill thekvm_rununion without exiting to userspace can lead to other bugs, e.g. effective corruption of thekvm_rununion, but at least from a uABI perspective, the behavior is acceptable.Quoted 10 lines
Actually, I don't think the idea of filling in kvm_run.memory_fault for -EFAULTs which don't make it to userspace works at all. Consider the
direct_mapfunction, which bubbles its-EFAULTtokvm_mmu_do_page_fault.kvm_mmu_do_page_faultis called from bothkvm_arch_async_page_ready(which ignores the return value), and bykvm_mmu_page_fault(where the return value does make it to userspace). Populating kvm_run.memory_fault anywhere in or underkvm_mmu_do_page_faultseems an immediate no-go, because a waywardkvm_arch_async_page_readycould (presumably) overwrite/corrupt an already-set kvm_run.memory_fault / otherkvm_runfield.This particular case is a non-issue.
kvm_check_async_pf_completion()is called only when the current task has control of the vCPU, i.e. is the current "running" vCPU. That's not a coincidence either, invokingkvm_mmu_do_page_fault()without having control of the vCPU would be fraught with races, e.g. the entire KVM MMU context would be unstable.That will hold true for all cases. Using a vCPU that is not loaded (not the current "running" vCPU in KVM's misleading terminology) to access guest memory is simply not safe, as the vCPU state is non-deterministic. There are paths where KVM accesses, and even modifies, vCPU state asynchronously, e.g. for IRQ delivery and making requests, but those are very controlled flows with dedicated machinery to make them SMP safe.
That said, I agree that there's a risk that KVM could clobber
vcpu->run_runby hitting an-EFAULTwithout the vCPU loaded, but that's a solvable problem, e.g. the helper to fillKVM_EXIT_MEMORY_FAULTcould be hardened to yell if called without the target vCPU being loaded:int kvm_handle_efault(struct kvm_vcpu *vcpu, ...) { preempt_disable(); if (WARN_ON_ONCE(vcpu != __this_cpu_read(kvm_running_vcpu))) goto out; vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT; ... out: preempt_enable(); return -EFAULT; }Quoted 8 lines
Quoted 6 lines
FWIW, I completely agree that filling
KVM_EXIT_MEMORY_FAULTwithout guaranteeing that KVM "immediately" exits to userspace isn't ideal, but given the amount of historical code that we need to deal with, it seems like the lesser of all evils. Unless I'm misunderstanding the use cases, unnecessarily fillingkvm_runis a far better failure mode than KVM not fillingkvm_runwhen it should, i.e. false positives are ok, false negatives are fatal.Don't you have this in reverse?
No, I don't think so.
Quoted 3 lines
False negatives will just result in userspace not having useful extra information for the
-EFAULTit receives fromKVM_RUN, in which case userspace can do what you mentioned all VMMs do today and just terminate the VM.And that is _really_ bad behavior if we have any hope of userspace actually being able to rely on this functionality. E.g. any false negative when userspace is trying to do postcopy demand paging will be fatal to the VM.
Quoted 5 lines
Quoted 2 lines
Whereas a false positive might cause a double-write to the
KVM_RUNstruct, either putting incorrect information in kvm_run.memory_fault orRecording unused information on
-EFAULTinkvm_rundoesn't make the information incorrect.
Let's say that some function (converted to annotate its EFAULTs) fills in kvm_run.memory_fault, but the EFAULT is suppressed from being returned from kvm_run. What if, later within the same kvm_run call, some other function (which we've completely overlooked) EFAULTs and that return value actually does make it out to kvm_run? Userspace would get stale information, which could be catastrophic.
Actually even performing the annotations only in functions that currently always bubble EFAULTs to userspace still seems brittle: if new callers are ever added which don't bubble the EFAULTs, then we end up in the same situation.
Sean Christopherson
maintainerRe: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 29 lines · Anish Moorthy
Quoted 22 lines · Sean Christopherson
Quoted 9 lines · Anish Moorthy
Quoted 6 lines · Sean Christopherson
FWIW, I completely agree that filling
KVM_EXIT_MEMORY_FAULTwithout guaranteeing that KVM "immediately" exits to userspace isn't ideal, but given the amount of historical code that we need to deal with, it seems like the lesser of all evils. Unless I'm misunderstanding the use cases, unnecessarily fillingkvm_runis a far better failure mode than KVM not fillingkvm_runwhen it should, i.e. false positives are ok, false negatives are fatal.Don't you have this in reverse?
No, I don't think so.
Quoted 3 lines
False negatives will just result in userspace not having useful extra information for the
-EFAULTit receives fromKVM_RUN, in which case userspace can do what you mentioned all VMMs do today and just terminate the VM.And that is _really_ bad behavior if we have any hope of userspace actually being able to rely on this functionality. E.g. any false negative when userspace is trying to do postcopy demand paging will be fatal to the VM.
Quoted 5 lines
Quoted 2 lines
Whereas a false positive might cause a double-write to the
KVM_RUNstruct, either putting incorrect information in kvm_run.memory_fault orRecording unused information on
-EFAULTinkvm_rundoesn't make the information incorrect.
Quoted 6 lines
Let's say that some function (converted to annotate its EFAULTs) fills in kvm_run.memory_fault, but the EFAULT is suppressed from being returned from
kvm_run. What if, later within the samekvm_runcall, some other function (which we've completely overlooked) EFAULTs and that return value actually does make it out tokvm_run? Userspace would get stale information, which could be catastrophic.
"catastrophic" is a bit hyperbolic. Yes, it would be bad, but at _worst_ userspace will kill the VM, which is the status quo today.
Quoted 4 lines
Actually even performing the annotations only in functions that currently always bubble EFAULTs to userspace still seems brittle: if new callers are ever added which don't bubble the EFAULTs, then we end up in the same situation.
Because of KVM's semi-magical '1 == resume, -errno/0 == exit' "design", that's true for literally every exit to userspace in KVM and every VM-Exit handler. E.g. see commit 2368048bf5c2 ("KVM: x86: Signal #GP, not -EPERM, on bad WRMSR(MCi_CTL/STATUS)"), where KVM returned '-1' instead of '1' when rejecting MSR accesses and inadvertantly killed the VM. A similar bug would be if KVM returned EFAULT instead of -EFAULT, in which case vcpu_run() would resume the guest instead of exiting to userspace and likely put the vCPU into an infinite loop.
Do I want to harden KVM to make things like this less brittle? Absolutely. Do I think we should hold up this functionality just because it doesn't solve all of pre-existing flaws in the related KVM code? No.
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 9 lines · Sean Christopherson
Quoted 6 lines
Let's say that some function (converted to annotate its EFAULTs) fills in kvm_run.memory_fault, but the EFAULT is suppressed from being returned from
kvm_run. What if, later within the samekvm_runcall, some other function (which we've completely overlooked) EFAULTs and that return value actually does make it out tokvm_run? Userspace would get stale information, which could be catastrophic."catastrophic" is a bit hyperbolic. Yes, it would be bad, but at _worst_ userspace will kill the VM, which is the status quo today.
Well what I'm saying is that in these cases userspace *wouldn't know* that kvm_run.memory_fault contains incorrect information for the -EFAULT it actually got (do you disagree?), which could presumably cause it to do bad things like "resolve" faults on incorrect pages and/or infinite-loop on KVM_RUN, etc.
Annotating the efault information as valid only from the call sites which return directly to userspace prevents this class of problem, at the cost of allowing un-annotated EFAULTs to make it to userspace. But to me, paying that cost to make sure the EFAULT information is always correct seems by far preferable to not paying it and allowing userspace to get silently incorrect information.
Quoted 5 lines
Quoted 4 lines
Actually even performing the annotations only in functions that currently always bubble EFAULTs to userspace still seems brittle: if new callers are ever added which don't bubble the EFAULTs, then we end up in the same situation.
Quoted 8 lines
Because of KVM's semi-magical '1 == resume, -errno/0 == exit' "design", that's true for literally every exit to userspace in KVM and every VM-Exit handler. E.g. see commit
2368048bf5c2("KVM: x86: Signal #GP, not-EPERM, on badWRMSR(MCi_CTL/STATUS)"), where KVM returned '-1' instead of '1' when rejecting MSR accesses and inadvertantly killed the VM. A similar bug would be if KVM returned EFAULT instead of-EFAULT, in which casevcpu_run()would resume the guest instead of exiting to userspace and likely put the vCPU into an infinite loop.
Right, good point.
Sean Christopherson
maintainerRe: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Quoted 8 lines · Anish Moorthy
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Let's say that some function (converted to annotate its EFAULTs) fills in kvm_run.memory_fault, but the EFAULT is suppressed from being returned from
kvm_run. What if, later within the samekvm_runcall, some other function (which we've completely overlooked) EFAULTs and that return value actually does make it out tokvm_run? Userspace would get stale information, which could be catastrophic.
Quoted 6 lines
Quoted 2 lines
"catastrophic" is a bit hyperbolic. Yes, it would be bad, but at _worst_ userspace will kill the VM, which is the status quo today.
Well what I'm saying is that in these cases userspace *wouldn't know* that kvm_run.memory_fault contains incorrect information for the
-EFAULTit actually got (do you disagree?),
I disagree in the sense that if the stale information causes a problem, then by definition userspace has to know. It's the whole "if a tree falls in a forest" thing. If KVM reports stale information and literally nothing bad happens, ever, then is the superfluous exit really a problem? Not saying it wouldn't be treated as a bug, just that it might not even warrant a stable backport if the worst case scenario is a spurious exit to userspace (for example).
Quoted 2 lines
which could presumably cause it to do bad things like "resolve" faults on incorrect pages and/or infinite-loop on
KVM_RUN, etc.
Putting the vCPU into an infinite loop is _very_ visible, e.g. see the entire mess surrounding commit 31c25585695a ("Revert "KVM: SVM: avoid infinite loop on NPF from bad address"").
As above, fixing pages that don't need to be fixed isn't itself a major problem. If the extra exits lead to a performance issue, then _that_ is a problem, but again _something_ has to detect the problem and thus it becomes a known thing.
Quoted 6 lines
Annotating the efault information as valid only from the call sites which return directly to userspace prevents this class of problem, at the cost of allowing un-annotated EFAULTs to make it to userspace. But to me, paying that cost to make sure the EFAULT information is always correct seems by far preferable to not paying it and allowing userspace to get silently incorrect information.
I don't think that's a maintainable approach. Filling kvm_run if and only if the -EFAULT has a direct path to userspace is (a) going to require a signficant amount of code churn and (b) falls apart the instant code further up the stack changes. E.g. the relatively straightforward page fault case requires bouncing through 7+ functions to get from kvm_handle_error_pfn() to kvm_arch_vcpu_ioctl_run(), and not all of those are obviously "direct"
if (IS_ENABLED(CONFIG_RETPOLINE) && fault.is_tdp) r = kvm_tdp_page_fault(vcpu, &fault); else r = vcpu->arch.mmu->page_fault(vcpu, &fault); if (fault.write_fault_to_shadow_pgtable && emulation_type) *emulation_type |= EMULTYPE_WRITE_PF_TO_SP; /* * Similar to above, prefetch faults aren't truly spurious, and the * async #PF path doesn't do emulation. Do count faults that are fixed * by the async #PF handler though, otherwise they'll never be counted. */ if (r == RET_PF_FIXED) vcpu->stat.pf_fixed++; else if (prefetch) ; else if (r == RET_PF_EMULATE) vcpu->stat.pf_emulate++; else if (r == RET_PF_SPURIOUS) vcpu->stat.pf_spurious++; return r;
...
if (r == RET_PF_INVALID) {
r = kvm_mmu_do_page_fault(vcpu, cr2_or_gpa,
lower_32_bits(error_code), false,
&emulation_type);
if (KVM_BUG_ON(r == RET_PF_INVALID, vcpu->kvm))
return -EIO;
}
if (r < 0)
return r;
if (r != RET_PF_EMULATE)
return 1;In other words, the "only if it's direct" rule requires visually auditing changes, i.e. catching "violations" via code review, not only to code that adds a new -EFAULT return, but to all code throughout rather large swaths of KVM. The odds of us (or whoever the future maintainers/reviewers are) remembering to enforce the "rule", let alone actually having 100% accuracy, are basically nil.
On the flip side, if we add a helper to fill kvm_run and return -EFAULT, then we can add rule that only time KVM is allowed to return a bare -EFAULT is immediately after a uaccess, i.e. after copy_to/from_user() and the many variants. And _that_ can be enforced through static checkers, e.g. someone with more (read: any) awk/sed skills than me could bang something out in a matter of minutes. Such a static checker won't catch everything, but there would be very, very few bare non-uaccess -EFAULTS left, and those could be filtered out with an allowlist, e.g. similar to how the folks that run smatch and whatnot deal with false positives.
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
Ok. I'm still concerned about the implications of the "annotate everywhere" approach, but I spoke with James and he shares your opinion on the severity of the potential issues. I'll put the patches together and send up a proper v3.
Re: [WIP Patch v2 04/14] KVM: x86: Add KVM_CAP_X86_MEMORY_FAULT_EXIT and associated kvm_run field
In reply to Anish MoorthyOn Wed, Mar 15, 2023 at 02:17:28AM +0000, Anish Moorthy wrote:
[...]
Quoted 10 lines
virt/kvm/kvm_main.c +6
-0@@ -6172,3 +6181,22 @@ int kvm_vm_create_worker_thread(struct kvm *kvm, kvm_vm_thread_fn_t thread_fn,return init_context.err; } + +inline int kvm_memfault_exit_or_efault ( +struct kvm_vcpu *vcpu,uint64_t gpa,uint64_t len,uint64_t exit_flags) +{ +if (!(vcpu->kvm->memfault_exit_reasons & exit_flags)) +return -EFAULT ;
<snip>
virt/kvm/kvm_main.c int kvm_vm_create_worker_thread(struct kvm *kvm, kvm_vm_thread_fn_t thread_fn, · line 6193
Quoted 4 lines
virt/kvm/kvm_main.c +4
-0+ vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; + vcpu->run->memory_fault.gpa = gpa; + vcpu->run->memory_fault.len = len; + vcpu->run->memory_fault.flags = exit_flags;
</snip>
Please spin this off into a helper and make use of it on the arm64 side.
Quoted 13 lines
virt/kvm/kvm_main.c +9
-0+return -1 ; +} + +bool kvm_memfault_exit_flags_valid (uint64_t reasons) +{ +uint64_t valid_flags =KVM_MEMFAULT_REASON_UNKNOWN ; + +return !(reasons & !valid_flags); +}Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Signature · 2 lines
Thanks, Oliver
PATCH v2 5/14 KVM: x86: Implement memory fault exit for direct_map
In reply to Anish MoorthyTODO: The return value of this function is ignored in kvm_arch_async_page_ready. Make sure that the side effects of memory_fault_exit_or_efault are acceptable there.
--- arch/x86/kvm/mmu/mmu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +4 -1
@@ -3193,7 +3193,10 @@ static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) }if (WARN_ON_ONCE (it.level != fault->goal_level)) -return -EFAULT ; +return kvm_memfault_exit_or_efault ( + vcpu, fault->gfn *PAGE_SIZE , +KVM_PAGES_PER_HPAGE (fault->goal_level), +KVM_MEMFAULT_REASON_UNKNOWN ); ret =mmu_set_spte (vcpu, fault->slot, it.sptep,ACC_ALL , base_gfn, fault->pfn, fault);
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 6/14 KVM: x86: Implement memory fault exit for kvm_handle_page_fault
In reply to Anish Moorthy--- arch/x86/kvm/mmu/mmu.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +3 -1
@@ -4375,7 +4375,9 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code,#ifndef CONFIG_X86_64 /* A 64-bit CR2 should be impossible on 32-bit KVM. */ if (WARN_ON_ONCE (fault_address >>32 )) -return -EFAULT ; +return kvm_mefault_exit_or_efault ( + vcpu, fault_address,PAGE_SIZE , +KVM_MEMFAULT_REASON_UNKNOWN );#endif vcpu->arch.l1tf_flush_l1d = true;
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 7/14 KVM: x86: Implement memory fault exit for setup_vmgexit_scratch
In reply to Anish Moorthy--- arch/x86/kvm/svm/sev.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/svm/sev.c +3 -1
@@ -2683,7 +2683,9 @@ static int setup_vmgexit_scratch(struct vcpu_svm *svm, bool sync, u64 len)pr_err ("vmgexit: kvm_read_guest for scratch area failed\n" );kvfree (scratch_va); -return -EFAULT ; +return kvm_memfault_exit_or_efault ( + &svm->vcpu, scratch_gpa_beg, len, +KVM_MEMFAULT_REASON_UNKNOWN ); }/*
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 8/14 KVM: x86: Implement memory fault exit for FNAME(fetch)
In reply to Anish Moorthy--- arch/x86/kvm/mmu/paging_tmpl.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/paging_tmpl.h +3 -1
@@ -717,7 +717,9 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, }if (WARN_ON_ONCE (it.level != fault->goal_level)) -return -EFAULT ; +return kvm_memfault_exit_or_efault ( + vcpu, fault->gfn *PAGE_SIZE ,KVM_PAGES_PER_HPAGE (fault->goal_level), +KVM_MEMFAULT_REASON_UNKNOWN ); ret =mmu_set_spte (vcpu, fault->slot, it.sptep, gw->pte_access, base_gfn, fault->pfn, fault);
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 9/14 KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
In reply to Anish MoorthyAdd documentation, memslot flags, useful helper functions, and the actual new capability itself.
Memory fault exits on absent mappings are particularly useful for userfaultfd-based live migration postcopy. When many vCPUs fault upon a single userfaultfd the faults can take a while to surface to userspace due to having to contend for uffd wait queue locks. Bypassing the uffd entirely by triggering a vCPU exit avoids this contention and can improve the fault rate by as much as 10x.
--- Documentation/virt/kvm/api.rst | 37 +++++++++++++++++++++++++++++++--- include/linux/kvm_host.h | 6 ++++++ include/uapi/linux/kvm.h | 3 +++ tools/include/uapi/linux/kvm.h | 2 ++ virt/kvm/kvm_main.c | 7 ++++++- 5 files changed, 51 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +34 -3
@@ -1312,6 +1312,7 @@ yet and must be cleared on entry. /* for kvm_userspace_memory_region::flags */ #define KVM_MEM_LOG_DIRTY_PAGES (1UL << 0) #define KVM_MEM_READONLY (1UL << 1) + #define KVM_MEM_ABSENT_MAPPING_FAULT (1UL << 2) This ioctl allows the user to create, modify or delete a guest physical memory slot. Bits 0-15 of "slot" specify the slot id and this value
@@ -1342,12 +1343,15 @@ It is recommended that the lower 21 bits of guest_phys_addr and userspace_addr be identical. This allows large pages in the guest to be backed by large pages in the host. -The flags field supports two flags: KVM_MEM_LOG_DIRTY_PAGES and -KVM_MEM_READONLY. The former can be set to instruct KVM to keep track of +The flags field supports three flags + +1. KVM_MEM_LOG_DIRTY_PAGES: can be set to instruct KVM to keep track of writes to memory within the slot. See KVM_GET_DIRTY_LOG ioctl to know how to -use it. The latter can be set, if KVM_CAP_READONLY_MEM capability allows it, +use it. +2. KVM_MEM_READONLY: can be set, if KVM_CAP_READONLY_MEM capability allows it, to make a new slot read-only. In this case, writes to this memory will be posted to userspace as KVM_EXIT_MMIO exits. +3. KVM_MEM_ABSENT_MAPPING_FAULT: see KVM_CAP_MEMORY_FAULT_NOWAIT for details. When the KVM_CAP_SYNC_MMU capability is available, changes in the backing of the memory region are automatically reflected into the guest. For example, an
@@ -7702,10 +7706,37 @@ Through args[0], the capability can be set on a per-exit-reason basis. Currently, the only exit reasons supported are 1. KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) +2. KVM_MEMFAULT_REASON_ABSENT_MAPPING (1 << 1) Memory fault exits with a reason of UNKNOWN should not be depended upon: they may be added, removed, or reclassified under a stable reason. +7.35 KVM_CAP_MEMORY_FAULT_NOWAIT +-------------------------------- + +:Architectures: x86, arm64 +:Returns: -EINVAL. + +The presence of this capability indicates that userspace may pass the +KVM_MEM_ABSENT_MAPPING_FAULT flag to KVM_SET_USER_MEMORY_REGION to cause KVM_RUN +to exit to populate 'kvm_run.memory_fault' and exit to userspace (*) in response +to page faults for which the userspace page tables do not contain present +mappings. Attempting to enable the capability directly will fail. + +The 'gpa' and 'len' fields of kvm_run.memory_fault will be set to the starting +address and length (in bytes) of the faulting page. 'flags' will be set to +KVM_MEMFAULT_REASON_ABSENT_MAPPING. + +Userspace should determine how best to make the mapping present, then take +appropriate action. For instance, in the case of absent mappings this might +involve establishing the mapping for the first time via UFFDIO_COPY/CONTINUE or +faulting the mapping in using MADV_POPULATE_READ/WRITE. After establishing the +mapping, userspace can return to KVM to retry the previous memory access. + +(*) NOTE: On x86, KVM_CAP_X86_MEMORY_FAULT_EXIT must be enabled for the +KVM_MEMFAULT_REASON_ABSENT_MAPPING_reason: otherwise userspace will only receive +a -EFAULT from KVM_RUN without any useful information. + 8. Other capabilities. ======================
include/linux/kvm_host.h +6 -0
@@ -593,6 +593,12 @@ static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *slreturn slot->flags &KVM_MEM_LOG_DIRTY_PAGES ; } +static inline bool kvm_slot_fault_on_absent_mapping ( +const struct kvm_memory_slot *slot) +{ +return slot->flags &KVM_MEM_ABSENT_MAPPING_FAULT ; +} +static inline unsigned long kvm_dirty_bitmap_bytes (struct kvm_memory_slot *memslot) {return ALIGN (memslot->npages,BITS_PER_LONG ) /8 ;
include/uapi/linux/kvm.h +3 -0
@@ -102,6 +102,7 @@ struct kvm_userspace_memory_region {*/ #define KVM_MEM_LOG_DIRTY_PAGES (1UL << 0)#define KVM_MEM_READONLY (1UL << 1) +#define KVM_MEM_ABSENT_MAPPING_FAULT (1UL << 2)/* for KVM_IRQ_LINE */ struct kvm_irq_level {
@@ -1197,6 +1198,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225#define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226#define KVM_CAP_X86_MEMORY_FAULT_EXIT 227 +#define KVM_CAP_MEMORY_FAULT_NOWAIT 228#ifdef KVM_CAP_IRQ_ROUTING
@@ -2252,5 +2254,6 @@ struct kvm_s390_zpci_op {/* Exit reasons for KVM_EXIT_MEMORY_FAULT */ #define KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) +#define KVM_MEMFAULT_REASON_ABSENT_MAPPING (1 << 1)#endif /* __LINUX_KVM_H */
tools/include/uapi/linux/kvm.h +2 -0
@@ -102,6 +102,7 @@ struct kvm_userspace_memory_region {*/ #define KVM_MEM_LOG_DIRTY_PAGES (1UL << 0)#define KVM_MEM_READONLY (1UL << 1) +#define KVM_MEM_ABSENT_MAPPING_FAULT (1UL << 2)/* for KVM_IRQ_LINE */ struct kvm_irq_level {
@@ -2242,5 +2243,6 @@ struct kvm_s390_zpci_op {/* Exit reasons for KVM_EXIT_MEMORY_FAULT */ #define KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) +#define KVM_MEMFAULT_REASON_ABSENT_MAPPING (1 << 1)#endif /* __LINUX_KVM_H */
virt/kvm/kvm_main.c +6 -1
@@ -1525,6 +1525,9 @@ static int check_memory_region_flags(const struct kvm_userspace_memory_region *m valid_flags |=KVM_MEM_READONLY ;#endif +if (kvm_vm_ioctl_check_extension (NULL ,KVM_CAP_MEMORY_FAULT_NOWAIT )) + valid_flags |=KVM_MEM_ABSENT_MAPPING_FAULT ; +if (mem->flags & ~valid_flags)return -EINVAL ;
@@ -6196,7 +6199,9 @@ inline int kvm_memfault_exit_or_efault(bool kvm_memfault_exit_flags_valid (uint64_t reasons) { -uint64_t valid_flags =KVM_MEMFAULT_REASON_UNKNOWN ; +uint64_t valid_flags + =KVM_MEMFAULT_REASON_UNKNOWN + |KVM_MEMFAULT_REASON_ABSENT_MAPPING ;return !(reasons & !valid_flags); }
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Re: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 22 lines · Anish Moorthy
Add documentation, memslot flags, useful helper functions, and the actual new capability itself.
Memory fault exits on absent mappings are particularly useful for userfaultfd-based live migration postcopy. When many vCPUs fault upon a single userfaultfd the faults can take a while to surface to userspace due to having to contend for uffd wait queue locks. Bypassing the uffd entirely by triggering a vCPU exit avoids this contention and can improve the fault rate by as much as 10x.
--- Documentation/virt/kvm/api.rst | 37 +++++++++++++++++++++++++++++++--- include/linux/kvm_host.h | 6 ++++++ include/uapi/linux/kvm.h | 3 +++ tools/include/uapi/linux/kvm.h | 2 ++ virt/kvm/kvm_main.c | 7 ++++++- 5 files changed, 51 insertions(+), 4 deletions(-)Documentation/virt/kvm/api.rst +0
-0
Documentation/virt/kvm/api.rst yet and must be cleared on entry. · line 1315
Quoted 4 lines
Documentation/virt/kvm/api.rst +1
-0@@ -1312,6 +1312,7 @@ yet and must be cleared on entry. /* for kvm_userspace_memory_region::flags */ #define KVM_MEM_LOG_DIRTY_PAGES (1UL << 0) #define KVM_MEM_READONLY (1UL << 1) + #define KVM_MEM_ABSENT_MAPPING_FAULT (1UL << 2)
call it KVM_MEM_EXIT_ABSENT_MAPPING
Quoted 46 lines
Documentation/virt/kvm/api.rst +22
-3This ioctl allows the user to create, modify or delete a guest physical memory slot. Bits 0-15 of "slot" specify the slot id and this value@@ -1342,12 +1343,15 @@ It is recommended that the lower 21 bits of guest_phys_addr and userspace_addr be identical. This allows large pages in the guest to be backed by large pages in the host. -The flags field supports two flags: KVM_MEM_LOG_DIRTY_PAGES and -KVM_MEM_READONLY. The former can be set to instruct KVM to keep track of +The flags field supports three flags + +1. KVM_MEM_LOG_DIRTY_PAGES: can be set to instruct KVM to keep track of writes to memory within the slot. See KVM_GET_DIRTY_LOG ioctl to know how to -use it. The latter can be set, if KVM_CAP_READONLY_MEM capability allows it, +use it. +2. KVM_MEM_READONLY: can be set, if KVM_CAP_READONLY_MEM capability allows it, to make a new slot read-only. In this case, writes to this memory will be posted to userspace as KVM_EXIT_MMIO exits. +3. KVM_MEM_ABSENT_MAPPING_FAULT: see KVM_CAP_MEMORY_FAULT_NOWAIT for details. When the KVM_CAP_SYNC_MMU capability is available, changes in the backing of the memory region are automatically reflected into the guest. For example, an@@ -7702,10 +7706,37 @@ Through args[0], the capability can be set on a per-exit-reason basis. Currently, the only exit reasons supported are 1. KVM_MEMFAULT_REASON_UNKNOWN (1 << 0) +2. KVM_MEMFAULT_REASON_ABSENT_MAPPING (1 << 1) Memory fault exits with a reason of UNKNOWN should not be depended upon: they may be added, removed, or reclassified under a stable reason. +7.35 KVM_CAP_MEMORY_FAULT_NOWAIT +-------------------------------- + +:Architectures: x86, arm64 +:Returns: -EINVAL. + +The presence of this capability indicates that userspace may pass the +KVM_MEM_ABSENT_MAPPING_FAULT flag to KVM_SET_USER_MEMORY_REGION to cause KVM_RUN +to exit to populate 'kvm_run.memory_fault' and exit to userspace (*) in response +to page faults for which the userspace page tables do not contain present +mappings. Attempting to enable the capability directly will fail. + +The 'gpa' and 'len' fields of kvm_run.memory_fault will be set to the starting +address and length (in bytes) of the faulting page. 'flags' will be set to +KVM_MEMFAULT_REASON_ABSENT_MAPPING.
Documentation/virt/kvm/api.rst Through args[0], the capability can be set on a per-exit-reason basis. · line 7738
Quoted 10 lines
Documentation/virt/kvm/api.rst +10
-0+ +Userspace should determine how best to make the mapping present, then take +appropriate action. For instance, in the case of absent mappings this might +involve establishing the mapping for the first time via UFFDIO_COPY/CONTINUE or +faulting the mapping in using MADV_POPULATE_READ/WRITE. After establishing the +mapping, userspace can return to KVM to retry the previous memory access. + +(*) NOTE: On x86, KVM_CAP_X86_MEMORY_FAULT_EXIT must be enabled for the +KVM_MEMFAULT_REASON_ABSENT_MAPPING_reason: otherwise userspace will only receive +a -EFAULT from KVM_RUN without any useful information.
I'm not a fan of this architecture-specific dependency. Userspace is already explicitly opting in to this behavior by way of the memslot flag. These sort of exits are entirely orthogonal to the -EFAULT conversion earlier in the series.
include/linux/kvm_host.h static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *sl · line 597
Quoted 13 lines
Documentation/virt/kvm/api.rst +0
-08. Other capabilities. ======================include/linux/kvm_host.h +2
-0@@ -593,6 +593,12 @@ static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *slreturn slot->flags &KVM_MEM_LOG_DIRTY_PAGES ; } +static inline bool kvm_slot_fault_on_absent_mapping ( +const struct kvm_memory_slot *slot)
Style again...
I'd strongly recommend using 'exit' instead of 'fault' in the verbiage of the KVM implementation. I understand we're giving userspace the illusion of a page fault mechanism, but the term is then overloaded in KVM since we handle literal faults from hardware.
Signature · 2 lines
Thanks, Oliver
Re: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 8 lines · Oliver Upton
Quoted 1 line
+ #define
KVM_MEM_ABSENT_MAPPING_FAULT(1UL << 2)call it
KVM_MEM_EXIT_ABSENT_MAPPING
...
I'm not a fan of this architecture-specific dependency. Userspace is already explicitly opting in to this behavior by way of the memslot flag. These sort of exits are entirely orthogonal to the-EFAULTconversion earlier in the series.
I'm not a fan of the semantics varying between architectures either:
but the reason I have it like that (and that the EFAULT conversions exist in this series in the first place) is (a) not having KVM_CAP_MEMORY_FAULT_EXIT implemented for arm and (b) Sean's following statement from https://lore.kernel.org/kvm/Y%[email protected]/
Quoted 6 lines · Sean Christopherson
The new memslot flag should depend on
KVM_CAP_MEMORY_FAULT_EXIT, butKVM_CAP_MEMORY_FAULT_EXITshould be a standalone thing, i.e. should convert "all" guest-memory-EFAULTStoKVM_CAP_MEMORY_FAULT_EXIT. All in quotes because I would likely be ok with a partial conversion for the initial implementation if there are paths that would require an absurd amount of work to convert.
The best way that I thought of how to do that was to have one cap (KVM_CAP_MEMORY_FAULT_NOWAIT) to make KVM -EFAULT without calling slow GUP, and KVM_CAP_MEMORY_FAULT_EXIT to transform efaults to useful vm exits. But if you think the two are really orthogonal, then we need to resolve the apparent disagreement.
Sean Christopherson
maintainerRe: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 11 lines · Anish Moorthy
Quoted 8 lines · Oliver Upton
Quoted 1 line
+ #define
KVM_MEM_ABSENT_MAPPING_FAULT(1UL << 2)call it
KVM_MEM_EXIT_ABSENT_MAPPING
...
I'm not a fan of this architecture-specific dependency. Userspace is already explicitly opting in to this behavior by way of the memslot flag. These sort of exits are entirely orthogonal to the-EFAULTconversion earlier in the series.
Quoted 5 lines
I'm not a fan of the semantics varying between architectures either:
but the reason I have it like that (and that the EFAULT conversions exist in this series in the first place) is (a) not havingKVM_CAP_MEMORY_FAULT_EXITimplemented for arm and (b) Sean's following statement from https://lore.kernel.org/kvm/Y%[email protected]/
Strictly speaking, if y'all buy my argument that the flag shouldn't control the gup behavior, there won't be semantic differences for the memslot flag. KVM will (obviously) behavior differently if KVM_CAP_MEMORY_FAULT_EXIT is not set, but that will hold true for x86 as well. The only difference is that x86 will also support an orthogonal flag that makes the fast-only memslot flag useful in practice.
So yeah, there will be an arch dependency, but only because arch code needs to actually handle perform the exit, and that's true no matter what.
That said, there's zero reason to put X86 in the name. Just add the capability as KVM_CAP_MEMORY_FAULT_EXIT or whatever and mark it as x86 in the documentation.
Re: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 6 lines · Sean Christopherson
And as I argued in the last version[*], I am _strongly_ opposed to KVM speculating on why KVM is exiting to userspace. I.e. KVM should not set a special flag if the memslot has "fast only" behavior. The only thing the flag should do is control whether or not KVM tries slow paths, what KVM does in response to an unresolved fault should be an orthogonal thing.
I'm guessing you would want changes to patch 10 of this series [*] then, right? Setting a bit/exit reason in kvm_run::memory_fault.flags depending on whether the failure originated from a "fast only" fault is... exactly what I'm doing :/ I'm not totally clear on your usages of the word "flag" above though, the "KVM should not set a special flag... the only thing *the* flag should do" part is throwing me off a bit. What I think you're saying is
"KVM should not set a special bit in kvm_run::memory_fault.flags if the memslot has fast-only behavior. The only thing KVM_MEM_ABSENT_MAPPING_FAULT should do is..."
[1] https://lore.kernel.org/all/[email protected]/
Quoted 7 lines · Sean Christopherson
Strictly speaking, if y'all buy my argument that the flag shouldn't control the gup behavior, there won't be semantic differences for the memslot flag. KVM will (obviously) behavior differently if
KVM_CAP_MEMORY_FAULT_EXITis not set, but that will hold true for x86 as well. The only difference is that x86 will also support an orthogonal flag that makes the fast-only memslot flag useful in practice.
Quoted 8 lines
So yeah, there will be an arch dependency, but only because arch code needs to actually handle perform the exit, and that's true no matter what.
That said, there's zero reason to put X86 in the name. Just add the capability as
KVM_CAP_MEMORY_FAULT_EXITor whatever and mark it as x86 in the documentation.That said, there's zero reason to put X86 in the name. Just add the capability as
KVM_CAP_MEMORY_FAULT_EXITor whatever and mark it as x86 in the documentation.
Again, a little confused on your first "flag" usage here. I figure you can't mean the memslot flag because the whole point of that is to control the GUP behavior, but I'm not sure what else you'd be referring to.
Anyways the idea of having orthogonal features, one to -EFAULTing early before a slow path and another to transform/augment -EFAULTs into/with useful information does make sense to me. But I think the issue here is that we want the fast-only memslot flag to be useful on Arm as well, and with KVM_CAP_MEMORY_FAULT_NOWAIT written as it is now there is a semantic differences between x86 and Arm.
I don't see a way to keep the two features here orthogonal on x86 and linked on arm without keeping that semantic difference. Perhaps the solution here is a bare-bones implementation of KVM_CAP_MEMORY_FAULT_EXIT for Arm? All that actually *needs* to be covered to resolve this difference is the one call site in user_mem_abort. since KVM_CAP_MEMORY_FAULT_EXIT will be allowed to have holes anyways.
Sean Christopherson
maintainerRe: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 8 lines · Anish Moorthy
Quoted 6 lines · Sean Christopherson
And as I argued in the last version[*], I am _strongly_ opposed to KVM speculating on why KVM is exiting to userspace. I.e. KVM should not set a special flag if the memslot has "fast only" behavior. The only thing the flag should do is control whether or not KVM tries slow paths, what KVM does in response to an unresolved fault should be an orthogonal thing.
Quoted 7 lines
I'm guessing you would want changes to patch 10 of this series [*] then, right? Setting a bit/exit reason in
kvm_run::memory_fault.flagsdepending on whether the failure originated from a "fast only" fault is... exactly what I'm doing :/ I'm not totally clear on your usages of the word "flag" above though, the "KVM should not set a special flag... the only thing *the* flag should do" part is throwing me off a bit. What I think you're saying is
Heh, the second "the flag" is referring to the memslot flag. Rewriting the above:
KVM should not set a special flag in kvm_run::memory_fault.flags ... the only thing KVM_MEM_FAST_FAULT_ONLY should do is ..."
Quoted 28 lines
"KVM should not set a special bit in
kvm_run::memory_fault.flagsif the memslot has fast-only behavior. The only thingKVM_MEM_ABSENT_MAPPING_FAULTshould do is..."[1] https://lore.kernel.org/all/[email protected]/
Quoted 7 lines · Sean Christopherson
Strictly speaking, if y'all buy my argument that the flag shouldn't control the gup behavior, there won't be semantic differences for the memslot flag. KVM will (obviously) behavior differently if
KVM_CAP_MEMORY_FAULT_EXITis not set, but that will hold true for x86 as well. The only difference is that x86 will also support an orthogonal flag that makes the fast-only memslot flag useful in practice.Quoted 8 lines
So yeah, there will be an arch dependency, but only because arch code needs to actually handle perform the exit, and that's true no matter what.
That said, there's zero reason to put X86 in the name. Just add the capability as
KVM_CAP_MEMORY_FAULT_EXITor whatever and mark it as x86 in the documentation.That said, there's zero reason to put X86 in the name. Just add the capability as
KVM_CAP_MEMORY_FAULT_EXITor whatever and mark it as x86 in the documentation.Again, a little confused on your first "flag" usage here. I figure you can't mean the memslot flag because the whole point of that is to control the GUP behavior, but I'm not sure what else you'd be referring to.
Quoted 6 lines
Anyways the idea of having orthogonal features, one to -EFAULTing early before a slow path and another to transform/augment -EFAULTs into/with useful information does make sense to me. But I think the issue here is that we want the fast-only memslot flag to be useful on Arm as well, and with
KVM_CAP_MEMORY_FAULT_NOWAITwritten as it is now there is a semantic differences between x86 and Arm.
If and only if userspace enables the capability that transforms -EFAULT.
Quoted 7 lines
I don't see a way to keep the two features here orthogonal on x86 and linked on arm without keeping that semantic difference. Perhaps the solution here is a bare-bones implementation of
KVM_CAP_MEMORY_FAULT_EXITfor Arm? All that actually *needs* to be covered to resolve this difference is the one call site inuser_mem_abort. sinceKVM_CAP_MEMORY_FAULT_EXITwill be allowed to have holes anyways.
As above, so long as userspace must opt into transforming -EFAULT, and can do so independent of KVM_MEM_FAST_FAULT_ONLY (or whatever we call it), the behavior of KVM_MEM_FAST_FAULT_ONLY itself is semantically identical across all architectures.
KVM_MEM_FAST_FAULT_ONLY is obviously not very useful without precise information about the failing address, but IMO that's not reason enough to tie the two together.
Re: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 18 lines · Sean Christopherson
Quoted 8 lines · Anish Moorthy
Quoted 6 lines · Sean Christopherson
And as I argued in the last version[*], I am _strongly_ opposed to KVM speculating on why KVM is exiting to userspace. I.e. KVM should not set a special flag if the memslot has "fast only" behavior. The only thing the flag should do is control whether or not KVM tries slow paths, what KVM does in response to an unresolved fault should be an orthogonal thing.
Quoted 7 lines
I'm guessing you would want changes to patch 10 of this series [*] then, right? Setting a bit/exit reason in
kvm_run::memory_fault.flagsdepending on whether the failure originated from a "fast only" fault is... exactly what I'm doing :/ I'm not totally clear on your usages of the word "flag" above though, the "KVM should not set a special flag... the only thing *the* flag should do" part is throwing me off a bit. What I think you're saying is
Quoted 10 lines
Heh, the second "the flag" is referring to the memslot flag. Rewriting the above:
KVM should not set a special flag in
kvm_run::memory_fault.flags... the only thingKVM_MEM_FAST_FAULT_ONLYshould do is ..."Quoted 5 lines
"KVM should not set a special bit in
kvm_run::memory_fault.flagsif the memslot has fast-only behavior. The only thingKVM_MEM_ABSENT_MAPPING_FAULTshould do is..."
Ok so, just to be clear, you are not opposed to
(a) all -EFAULTs from kvm_faultin_pfn populating the kvm_run.memory_fault and setting kvm_run.memory_fault.flags to, say, FAULTIN_FAILURE if/when kvm_cap_memory_fault_exit is enabled
but *are* opposed to
(b) the combination of the memslot flag and kvm_cap_memory_fault_exit providing any additional information on top of: for instance, a kvm_run.memory_fault.flags of FAULTIN_FAILURE & FAST_FAULT_ONLY.
Is that right?
Quoted 47 lines
Quoted 22 lines · Sean Christopherson
Quoted 7 lines
Strictly speaking, if y'all buy my argument that the flag shouldn't control the gup behavior, there won't be semantic differences for the memslot flag. KVM will (obviously) behavior differently if
KVM_CAP_MEMORY_FAULT_EXITis not set, but that will hold true for x86 as well. The only difference is that x86 will also support an orthogonal flag that makes the fast-only memslot flag useful in practice.Quoted 8 lines
So yeah, there will be an arch dependency, but only because arch code needs to actually handle perform the exit, and that's true no matter what.
That said, there's zero reason to put X86 in the name. Just add the capability as
KVM_CAP_MEMORY_FAULT_EXITor whatever and mark it as x86 in the documentation.That said, there's zero reason to put X86 in the name. Just add the capability as
KVM_CAP_MEMORY_FAULT_EXITor whatever and mark it as x86 in the documentation.Again, a little confused on your first "flag" usage here. I figure you can't mean the memslot flag because the whole point of that is to control the GUP behavior, but I'm not sure what else you'd be referring to.
Quoted 6 lines
Anyways the idea of having orthogonal features, one to -EFAULTing early before a slow path and another to transform/augment -EFAULTs into/with useful information does make sense to me. But I think the issue here is that we want the fast-only memslot flag to be useful on Arm as well, and with
KVM_CAP_MEMORY_FAULT_NOWAITwritten as it is now there is a semantic differences between x86 and Arm.If and only if userspace enables the capability that transforms
-EFAULT.Quoted 7 lines
I don't see a way to keep the two features here orthogonal on x86 and linked on arm without keeping that semantic difference. Perhaps the solution here is a bare-bones implementation of
KVM_CAP_MEMORY_FAULT_EXITfor Arm? All that actually *needs* to be covered to resolve this difference is the one call site inuser_mem_abort. sinceKVM_CAP_MEMORY_FAULT_EXITwill be allowed to have holes anyways.As above, so long as userspace must opt into transforming
-EFAULT, and can do so independent ofKVM_MEM_FAST_FAULT_ONLY(or whatever we call it), the behavior ofKVM_MEM_FAST_FAULT_ONLYitself is semantically identical across all architectures.
KVM_MEM_FAST_FAULT_ONLYis obviously not very useful without precise information about the failing address, but IMO that's not reason enough to tie the two together.
Sean Christopherson
maintainerRe: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
In reply to Oliver UptonQuoted 18 lines · Oliver Upton
Quoted 17 lines · Anish Moorthy
Add documentation, memslot flags, useful helper functions, and the actual new capability itself.
Memory fault exits on absent mappings are particularly useful for userfaultfd-based live migration postcopy. When many vCPUs fault upon a single userfaultfd the faults can take a while to surface to userspace due to having to contend for uffd wait queue locks. Bypassing the uffd entirely by triggering a vCPU exit avoids this contention and can improve the fault rate by as much as 10x.
--- Documentation/virt/kvm/api.rst | 37 +++++++++++++++++++++++++++++++--- include/linux/kvm_host.h | 6 ++++++ include/uapi/linux/kvm.h | 3 +++ tools/include/uapi/linux/kvm.h | 2 ++ virt/kvm/kvm_main.c | 7 ++++++- 5 files changed, 51 insertions(+), 4 deletions(-)
Quoted 11 lines
Documentation/virt/kvm/api.rst yet and must be cleared on entry. · line 1315
Quoted 9 lines
Documentation/virt/kvm/api.rst +1
-0@@ -1312,6 +1312,7 @@ yet and must be cleared on entry. /* for kvm_userspace_memory_region::flags */ #define KVM_MEM_LOG_DIRTY_PAGES (1UL << 0) #define KVM_MEM_READONLY (1UL << 1) + #define KVM_MEM_ABSENT_MAPPING_FAULT (1UL << 2)call it
KVM_MEM_EXIT_ABSENT_MAPPING
Ooh, look, a bikeshed! :-)
I don't think it should have "EXIT" in the name. The exit to userspace is a side effect, e.g. KVM already exits to userspace on unresolved userfaults. The only thing this knob _directly_ controls is whether or not KVM attempts the slow path. If we give the flag a name like "exit on absent userspace mappings", then KVM will appear to do the wrong thing when KVM exits on a truly absent userspace mapping.
And as I argued in the last version[*], I am _strongly_ opposed to KVM speculating on why KVM is exiting to userspace. I.e. KVM should not set a special flag if the memslot has "fast only" behavior. The only thing the flag should do is control whether or not KVM tries slow paths, what KVM does in response to an unresolved fault should be an orthogonal thing.
E.g. If KVM encounters an unmapped page while prefetching SPTEs, KVM will (correctly) not exit to userspace and instead simply terminate the prefetch. Obviously we could solve that through documentation, but I don't see any benefit in making this more complex than it needs to be.
[*] https://lkml.kernel.org/r/Y%2B0RYMfw6pHrSLX4%40google.com
Quoted 22 lines
Quoted 22 lines
Documentation/virt/kvm/api.rst +22
-0+7.35 KVM_CAP_MEMORY_FAULT_NOWAIT +-------------------------------- + +:Architectures: x86, arm64 +:Returns: -EINVAL. + +The presence of this capability indicates that userspace may pass the +KVM_MEM_ABSENT_MAPPING_FAULT flag to KVM_SET_USER_MEMORY_REGION to cause KVM_RUN +to exit to populate 'kvm_run.memory_fault' and exit to userspace (*) in response +to page faults for which the userspace page tables do not contain present +mappings. Attempting to enable the capability directly will fail. + +The 'gpa' and 'len' fields of kvm_run.memory_fault will be set to the starting +address and length (in bytes) of the faulting page. 'flags' will be set to +KVM_MEMFAULT_REASON_ABSENT_MAPPING. + +Userspace should determine how best to make the mapping present, then take +appropriate action. For instance, in the case of absent mappings this might +involve establishing the mapping for the first time via UFFDIO_COPY/CONTINUE or +faulting the mapping in using MADV_POPULATE_READ/WRITE. After establishing the +mapping, userspace can return to KVM to retry the previous memory access. +
Quoted 8 lines
Documentation/virt/kvm/api.rst Through args[0], the capability can be set on a per-exit-reason basis. · line 7738
Quoted 3 lines
Documentation/virt/kvm/api.rst +3
-0+(*) NOTE: On x86, KVM_CAP_X86_MEMORY_FAULT_EXIT must be enabled for the +KVM_MEMFAULT_REASON_ABSENT_MAPPING_reason: otherwise userspace will only receive +a -EFAULT from KVM_RUN without any useful information.I'm not a fan of this architecture-specific dependency. Userspace is already explicitly opting in to this behavior by way of the memslot flag. These sort of exits are entirely orthogonal to the
-EFAULTconversion earlier in the series.
Ya, yet another reason not to speculate on why KVM wasn't able to resolve a fault.
Re: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Sean,
Quoted 31 lines · Sean Christopherson
Quoted 18 lines · Oliver Upton
Quoted 17 lines · Anish Moorthy
Add documentation, memslot flags, useful helper functions, and the actual new capability itself.
Memory fault exits on absent mappings are particularly useful for userfaultfd-based live migration postcopy. When many vCPUs fault upon a single userfaultfd the faults can take a while to surface to userspace due to having to contend for uffd wait queue locks. Bypassing the uffd entirely by triggering a vCPU exit avoids this contention and can improve the fault rate by as much as 10x.
--- Documentation/virt/kvm/api.rst | 37 +++++++++++++++++++++++++++++++--- include/linux/kvm_host.h | 6 ++++++ include/uapi/linux/kvm.h | 3 +++ tools/include/uapi/linux/kvm.h | 2 ++ virt/kvm/kvm_main.c | 7 ++++++- 5 files changed, 51 insertions(+), 4 deletions(-)Quoted 11 lines
Documentation/virt/kvm/api.rst yet and must be cleared on entry. · line 1315
Quoted 9 lines
Documentation/virt/kvm/api.rst +1
-0@@ -1312,6 +1312,7 @@ yet and must be cleared on entry. /* for kvm_userspace_memory_region::flags */ #define KVM_MEM_LOG_DIRTY_PAGES (1UL << 0) #define KVM_MEM_READONLY (1UL << 1) + #define KVM_MEM_ABSENT_MAPPING_FAULT (1UL << 2)call it
KVM_MEM_EXIT_ABSENT_MAPPING
Quoted 1 line
Ooh, look, a bikeshed! :-)
Couldn't help myself :)
Quoted 12 lines
I don't think it should have "EXIT" in the name. The exit to userspace is a side effect, e.g. KVM already exits to userspace on unresolved userfaults. The only thing this knob _directly_ controls is whether or not KVM attempts the slow path. If we give the flag a name like "exit on absent userspace mappings", then KVM will appear to do the wrong thing when KVM exits on a truly absent userspace mapping.
And as I argued in the last version[*], I am _strongly_ opposed to KVM speculating on why KVM is exiting to userspace. I.e. KVM should not set a special flag if the memslot has "fast only" behavior. The only thing the flag should do is control whether or not KVM tries slow paths, what KVM does in response to an unresolved fault should be an orthogonal thing.
Quoted 4 lines
E.g. If KVM encounters an unmapped page while prefetching SPTEs, KVM will (correctly) not exit to userspace and instead simply terminate the prefetch. Obviously we could solve that through documentation, but I don't see any benefit in making this more complex than it needs to be.
I couldn't care less about what the user-facing portion of this thing is called, TBH. We could just refer to it as KVM_MEM_BIT_2 /s
The only bit I wanted to avoid is having a collision in the kernel between literal faults arising from hardware and exits to userspace that we are also calling 'faults'.
Quoted 33 lines
[*] https://lkml.kernel.org/r/Y%2B0RYMfw6pHrSLX4%40google.com
Quoted 22 lines
Quoted 22 lines
Documentation/virt/kvm/api.rst +22
-0+7.35 KVM_CAP_MEMORY_FAULT_NOWAIT +-------------------------------- + +:Architectures: x86, arm64 +:Returns: -EINVAL. + +The presence of this capability indicates that userspace may pass the +KVM_MEM_ABSENT_MAPPING_FAULT flag to KVM_SET_USER_MEMORY_REGION to cause KVM_RUN +to exit to populate 'kvm_run.memory_fault' and exit to userspace (*) in response +to page faults for which the userspace page tables do not contain present +mappings. Attempting to enable the capability directly will fail. + +The 'gpa' and 'len' fields of kvm_run.memory_fault will be set to the starting +address and length (in bytes) of the faulting page. 'flags' will be set to +KVM_MEMFAULT_REASON_ABSENT_MAPPING. + +Userspace should determine how best to make the mapping present, then take +appropriate action. For instance, in the case of absent mappings this might +involve establishing the mapping for the first time via UFFDIO_COPY/CONTINUE or +faulting the mapping in using MADV_POPULATE_READ/WRITE. After establishing the +mapping, userspace can return to KVM to retry the previous memory access. +Quoted 8 lines
Documentation/virt/kvm/api.rst Through args[0], the capability can be set on a per-exit-reason basis. · line 7738
Quoted 3 lines
Documentation/virt/kvm/api.rst +3
-0+(*) NOTE: On x86, KVM_CAP_X86_MEMORY_FAULT_EXIT must be enabled for the +KVM_MEMFAULT_REASON_ABSENT_MAPPING_reason: otherwise userspace will only receive +a -EFAULT from KVM_RUN without any useful information.I'm not a fan of this architecture-specific dependency. Userspace is already explicitly opting in to this behavior by way of the memslot flag. These sort of exits are entirely orthogonal to the
-EFAULTconversion earlier in the series.
Quoted 1 line
Ya, yet another reason not to speculate on why KVM wasn't able to resolve a fault.
Regardless of what we name this memslot flag, we're already getting explicit opt-in from userspace for new behavior. There seems to be zero value in supporting memslot_flag && !MEMORY_FAULT_EXIT (i.e. returning EFAULT), so why even bother?
Requiring two levels of opt-in to have the intended outcome for a single architecture seems nauseating from a userspace perspective.
Signature · 2 lines
Thanks, Oliver
Sean Christopherson
maintainerRe: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 7 lines · Oliver Upton
Quoted 6 lines · Sean Christopherson
Quoted 4 lines · Oliver Upton
I'm not a fan of this architecture-specific dependency. Userspace is already explicitly opting in to this behavior by way of the memslot flag. These sort of exits are entirely orthogonal to the
-EFAULTconversion earlier in the series.
Quoted 6 lines
Quoted 1 line
Ya, yet another reason not to speculate on why KVM wasn't able to resolve a fault.
Regardless of what we name this memslot flag, we're already getting explicit opt-in from userspace for new behavior. There seems to be zero value in supporting
memslot_flag&& !MEMORY_FAULT_EXIT (i.e. returning EFAULT), so why even bother?
Because there are use cases for MEMORY_FAULT_EXIT beyond fast-only gup. We could have the memslot feature depend on the MEMORY_FAULT_EXIT capability, but I don't see how that adds value for either KVM or userspace.
Filling MEMORY_FAULT_EXIT iff the memslot flag is set would also lead to a weird ABI and/or funky KVM code. E.g. if MEMORY_FAULT_EXIT is tied to the fast-only memslot flag, what's the defined behavior if the gfn=>hva translation fails? KVM hasn't actually tried to gup() anything. Obviously not the end of the world, but I'd prefer not to avoid introduce more odditites into KVM, however minor.
Quoted 2 lines
Requiring two levels of opt-in to have the intended outcome for a single architecture seems nauseating from a userspace perspective.
If we usurp -EFAULT, I don't think we'll actually need an opt-in for MEMORY_FAULT_EXIT. KVM will need to add a capability so that userspace can query KVM support, but the actual filling of of kvm_run could be done unconditionally.
Even if we do end up making the behavior opt-in, I would expect them to be largely orthogonal in userspace. E.g. userspace would always enable MEMORY_FAULT_EXIT during startup, and then toggle the memslot flag during postcopy.
Re: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 8 lines · Sean Christopherson
Quoted 7 lines · Oliver Upton
Quoted 6 lines · Sean Christopherson
Quoted 4 lines · Oliver Upton
I'm not a fan of this architecture-specific dependency. Userspace is already explicitly opting in to this behavior by way of the memslot flag. These sort of exits are entirely orthogonal to the
-EFAULTconversion earlier in the series.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Ya, yet another reason not to speculate on why KVM wasn't able to resolve a fault.
Regardless of what we name this memslot flag, we're already getting explicit opt-in from userspace for new behavior. There seems to be zero value in supporting
memslot_flag&& !MEMORY_FAULT_EXIT (i.e. returning EFAULT), so why even bother?Because there are use cases for
MEMORY_FAULT_EXITbeyond fast-only gup.
To be abundantly clear -- I have no issue with (nor care about) the other MEMORY_FAULT_EXIT changes. If we go the route of explicit user opt-in then that deserves its own distinct bit of UAPI. None of my objection pertains to the conversion of existing -EFAULT exits.
Quoted 2 lines
We could have the memslot feature depend on the
MEMORY_FAULT_EXITcapability, but I don't see how that adds value for either KVM or userspace.
That is exactly what I want to avoid! My issue was the language here:
+(*) NOTE: On x86, KVM_CAP_X86_MEMORY_FAULT_EXIT must be enabled for the +KVM_MEMFAULT_REASON_ABSENT_MAPPING_reason: otherwise userspace will only receive +a -EFAULT from KVM_RUN without any useful information.
Which sounds to me as though there are *two* UAPI bits for the whole fast-gup failed interaction (flip a bit in the CAP and set a bit on the memslot, but only for x86).
What I'm asking for is this:
1) A capability advertising MEMORY_FAULT_EXIT to userspace. Either usurp EFAULT or require userspace to enable this capability to convert _existing_ EFAULT exits to the new way of the world.
2) A capability and a single memslot flag to enable the fast-gup-only behavior (naming TBD). This does not depend on (1) in any way, i.e. only setting (2) should still result in MEMORY_FAULT_EXITs when fast gup fails. IOW, enabling (2) should always yield precise fault information to userspace.
Signature · 2 lines
Thanks, Oliver
Sean Christopherson
maintainerRe: [WIP Patch v2 09/14] KVM: Introduce KVM_CAP_MEMORY_FAULT_NOWAIT without implementation
Quoted 28 lines · Oliver Upton
Quoted 8 lines · Sean Christopherson
Quoted 7 lines · Oliver Upton
Quoted 6 lines · Sean Christopherson
Quoted 4 lines · Oliver Upton
I'm not a fan of this architecture-specific dependency. Userspace is already explicitly opting in to this behavior by way of the memslot flag. These sort of exits are entirely orthogonal to the
-EFAULTconversion earlier in the series.Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Ya, yet another reason not to speculate on why KVM wasn't able to resolve a fault.
Regardless of what we name this memslot flag, we're already getting explicit opt-in from userspace for new behavior. There seems to be zero value in supporting
memslot_flag&& !MEMORY_FAULT_EXIT (i.e. returning EFAULT), so why even bother?Because there are use cases for
MEMORY_FAULT_EXITbeyond fast-only gup.To be abundantly clear -- I have no issue with (nor care about) the other
MEMORY_FAULT_EXITchanges. If we go the route of explicit user opt-in then that deserves its own distinct bit of UAPI. None of my objection pertains to the conversion of existing-EFAULTexits.Quoted 2 lines
We could have the memslot feature depend on the
MEMORY_FAULT_EXITcapability, but I don't see how that adds value for either KVM or userspace.That is exactly what I want to avoid! My issue was the language here:
Quoted 7 lines
+(*) NOTE: On x86,
KVM_CAP_X86_MEMORY_FAULT_EXITmust be enabled for the +KVM_MEMFAULT_REASON_ABSENT_MAPPING_reason: otherwise userspace will only receive +a-EFAULTfromKVM_RUNwithout any useful information.Which sounds to me as though there are *two* UAPI bits for the whole fast-gup failed interaction (flip a bit in the CAP and set a bit on the memslot, but only for x86).
It won't be x86 only. Anish's proposed patch has it as x86 specific, but I think we're all in agreement that that is undesirable. There will inevitably be per-arch enabling and enumeration, e.g. to actually fill information and kick out to userspace, but I don't see a sane way to avoid that since the common paths don't have the vCPU (largely by design).
Quoted 11 lines
What I'm asking for is this:
1) A capability advertising
MEMORY_FAULT_EXITto userspace. Either usurp EFAULT or require userspace to enable this capability to convert _existing_ EFAULT exits to the new way of the world.2) A capability and a single memslot flag to enable the fast-gup-only behavior (naming TBD). This does not depend on (1) in any way, i.e. only setting (2) should still result in
MEMORY_FAULT_EXITswhen fast gup fails. IOW, enabling (2) should always yield precise fault information to userspace.
Ah, so 2.2, providing precise fault information on fast-gup-only failures, is the biggest (only?) point of contention.
My objection to that behavior is that it's either going to annoyingly difficult to get right in KVM, and even more annoying to maintain, or we'll end up with "fuzzy" behavior that userspace will inevitably come to rely on, and then we'll be in a real pickle. E.g. if KVM sets the information without checking if gup() itself actually failed, then KVM _might_ fill the info, depending on when KVM detects a problem.
Conversly, if KVM's contract is that it provides precise information if and only if gup() fails, then KVM needs to precisely propagate back up the stack that gup() failed.
To avoid spending more time going in circles, I propose we try to usurp -EFAULT and convert all userspace-exits-from-KVM_RUN -EFAULT paths on x86 (as a guinea pig) without requiring userspace to opt-in. If that approach pans out, then this point of contention goes away because 2.2 Just Works.
PATCH v2 10/14 KVM: x86: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
In reply to Anish MoorthyWhen a memslot has the KVM_MEM_MEMORY_FAULT_EXIT flag set, exit to userspace upon encountering a page fault for which the userspace page tables do not contain a present mapping.
--- arch/x86/kvm/mmu/mmu.c | 33 +++++++++++++++++++++++++-------- arch/x86/kvm/x86.c | 1 + 2 files changed, 26 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +25 -8
@@ -3214,7 +3214,9 @@ static void kvm_send_hwpoison_signal(struct kvm_memory_slot *slot, gfn_t gfn)send_sig_mceerr (BUS_MCEERR_AR , (void __user *)hva,PAGE_SHIFT , current); } -static int kvm_handle_error_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) +static int kvm_handle_error_pfn ( +struct kvm_vcpu *vcpu,struct kvm_page_fault *fault, +bool faulted_on_absent_mapping) {if (is_sigpending_pfn (fault->pfn)) {kvm_handle_signal_exit (vcpu);
@@ -3234,7 +3236,11 @@ static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fareturn RET_PF_RETRY ; } -return -EFAULT ; +return kvm_memfault_exit_or_efault ( + vcpu, fault->gfn *PAGE_SIZE ,PAGE_SIZE , + faulted_on_absent_mapping + ?KVM_MEMFAULT_REASON_ABSENT_MAPPING + :KVM_MEMFAULT_REASON_UNKNOWN ); } static int kvm_handle_noslot_fault(struct kvm_vcpu *vcpu,
@@ -4209,7 +4215,9 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } -static int __kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) +static int __kvm_faultin_pfn ( +struct kvm_vcpu *vcpu,struct kvm_page_fault *fault, +bool fault_on_absent_mapping) {struct kvm_memory_slot *slot = fault->slot;bool async;
@@ -4242,9 +4250,15 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault } async = false; - fault->pfn =__gfn_to_pfn_memslot (slot, fault->gfn, false, false, &async, - fault->write, &fault->map_writable, - &fault->hva); + + fault->pfn =__gfn_to_pfn_memslot ( + slot, fault->gfn, + fault_on_absent_mapping, + false, + fault_on_absent_mapping ?NULL : &async, + fault->write, &fault->map_writable, + &fault->hva); +if (!async)return RET_PF_CONTINUE ;/* *pfn has correct page already */
@@ -4274,16 +4288,19 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, unsigned int access) {int ret; +bool fault_on_absent_mapping + =likely (fault->slot) &&kvm_slot_fault_on_absent_mapping (fault->slot); fault->mmu_seq = vcpu->kvm->mmu_invalidate_seq;smp_rmb (); - ret =__kvm_faultin_pfn (vcpu, fault); + ret =__kvm_faultin_pfn ( + vcpu, fault, fault_on_absent_mapping);if (ret !=RET_PF_CONTINUE )return ret;if (unlikely (is_error_pfn (fault->pfn))) -return kvm_handle_error_pfn (vcpu, fault); +return kvm_handle_error_pfn (vcpu, fault, fault_on_absent_mapping);if (unlikely (!fault->slot))return kvm_handle_noslot_fault (vcpu, fault, access);
arch/x86/kvm/x86.c +1 -0
@@ -4426,6 +4426,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_ENABLE_CAP :case KVM_CAP_VM_DISABLE_NX_HUGE_PAGES :case KVM_CAP_X86_MEMORY_FAULT_EXIT : +case KVM_CAP_MEMORY_FAULT_NOWAIT : r =1 ;break ;case KVM_CAP_EXIT_HYPERCALL :
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Re: [WIP Patch v2 10/14] KVM: x86: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
Quoted 67 lines · Anish Moorthy
When a memslot has the
KVM_MEM_MEMORY_FAULT_EXITflag set, exit to userspace upon encountering a page fault for which the userspace page tables do not contain a present mapping.--- arch/x86/kvm/mmu/mmu.c | 33 +++++++++++++++++++++++++-------- arch/x86/kvm/x86.c | 1 + 2 files changed, 26 insertions(+), 8 deletions(-)arch/x86/kvm/mmu/mmu.c +20
-6@@ -3214,7 +3214,9 @@ static void kvm_send_hwpoison_signal(struct kvm_memory_slot *slot, gfn_t gfn)send_sig_mceerr (BUS_MCEERR_AR , (void __user *)hva,PAGE_SHIFT , current); } -static int kvm_handle_error_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) +static int kvm_handle_error_pfn ( +struct kvm_vcpu *vcpu,struct kvm_page_fault *fault, +bool faulted_on_absent_mapping) {if (is_sigpending_pfn (fault->pfn)) {kvm_handle_signal_exit (vcpu);@@ -3234,7 +3236,11 @@ static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fareturn RET_PF_RETRY ; } -return -EFAULT ; +return kvm_memfault_exit_or_efault ( + vcpu, fault->gfn *PAGE_SIZE ,PAGE_SIZE , + faulted_on_absent_mapping + ?KVM_MEMFAULT_REASON_ABSENT_MAPPING + :KVM_MEMFAULT_REASON_UNKNOWN ); } static int kvm_handle_noslot_fault(struct kvm_vcpu *vcpu,@@ -4209,7 +4215,9 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } -static int __kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) +static int __kvm_faultin_pfn ( +struct kvm_vcpu *vcpu,struct kvm_page_fault *fault, +bool fault_on_absent_mapping) {struct kvm_memory_slot *slot = fault->slot;bool async;@@ -4242,9 +4250,15 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault } async = false; - fault->pfn =__gfn_to_pfn_memslot (slot, fault->gfn, false, false, &async, - fault->write, &fault->map_writable, - &fault->hva); + + fault->pfn =__gfn_to_pfn_memslot ( + slot, fault->gfn, + fault_on_absent_mapping, + false, + fault_on_absent_mapping ?NULL : &async, + fault->write, &fault->map_writable, + &fault->hva); +if (!async)return RET_PF_CONTINUE ;/* *pfn has correct page already */
arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4292
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +2
-0@@ -4274,16 +4288,19 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, unsigned int access) {int ret; +bool fault_on_absent_mapping + =likely (fault->slot) &&kvm_slot_fault_on_absent_mapping (fault->slot);
nit: Instead of passing around the value, we can add a new member to struct kvm_page_fault::fault_on_absent_mapping.
fault->fault_on_absent_mapping = likely(fault->slot) && kvm_slot_fault_on_absent_mapping(fault->slot);
Thanks,
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v2 11/14 KVM: arm64: Allow user_mem_abort to return 0 to signal a 'normal' exit
In reply to Anish Moorthykvm_handle_guest_abort currently just returns 1 if user_mem_abort returns 0. Since 1 is the "resume the guest" code, user_mem_abort is essentially incapable of triggering a "normal" exit: it can only trigger exits by returning a negative value, which indicates an error.
Remove the "if (ret == 0) ret = 1;" statement from kvm_handle_guest_abort and refactor user_mem_abort slightly to allow it to trigger 'normal' exits by returning 0.
--- arch/arm64/kvm/mmu.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/arm64/kvm/mmu.c +9 -6
@@ -1190,7 +1190,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, struct kvm_memory_slot *memslot, unsigned long hva, unsigned long fault_status) { - int ret = 0; + int ret = 1;bool write_fault, writable, force_pte = false;bool exec_fault;bool device = false;
@@ -1281,8 +1281,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, (logging_active && write_fault)) { ret =kvm_mmu_topup_memory_cache (memcache,kvm_mmu_cache_min_pages (kvm)); -if (ret) +if (ret <0 )return ret; +else + ret =1 ; } mmu_seq = vcpu->kvm->mmu_invalidate_seq;
@@ -1305,7 +1307,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, write_fault, &writable, NULL);if (pfn ==KVM_PFN_ERR_HWPOISON ) {kvm_send_hwpoison_signal (hva, vma_shift); -return 0 ; +return 1 ; }if (is_error_noslot_pfn (pfn))return -EFAULT ;
@@ -1387,6 +1389,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,KVM_PGTABLE_WALK_HANDLE_FAULT | KVM_PGTABLE_WALK_SHARED); +if (ret ==0 ) + ret =1 ; +/* Mark the page dirty only if the fault is handled successfully */ if (writable && !ret) {kvm_set_pfn_dirty (pfn);
@@ -1397,7 +1402,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,read_unlock (&kvm->mmu_lock);kvm_set_pfn_accessed (pfn);kvm_release_pfn_clean (pfn); -return ret != -EAGAIN ? ret :0 ; +return ret != -EAGAIN ? ret :1 ; }/* Resolve the access fault by making the page young again. */
@@ -1549,8 +1554,6 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) } ret =user_mem_abort (vcpu, fault_ipa, memslot, hva, fault_status); -if (ret ==0 ) - ret =1 ; out:if (ret == -ENOEXEC ) {kvm_inject_pabt (vcpu,kvm_vcpu_get_hfar (vcpu));
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Re: [WIP Patch v2 11/14] KVM: arm64: Allow user_mem_abort to return 0 to signal a 'normal' exit
Quoted 8 lines · Anish Moorthy
kvm_handle_guest_abortcurrently just returns 1 ifuser_mem_abortreturns 0. Since 1 is the "resume the guest" code,user_mem_abortis essentially incapable of triggering a "normal" exit: it can only trigger exits by returning a negative value, which indicates an error.Remove the "if (ret == 0) ret = 1;" statement from
kvm_handle_guest_abortand refactoruser_mem_abortslightly to allow it to trigger 'normal' exits by returning 0.
You should append '()' to function names, as it makes it abundantly obvious to the reader that the symbols you describe are indeed functions.
I find the changelog a bit too mechanical and doesn't capture the nuance.
Generally, in the context of a vCPU exit, a return value of 1 is used to indicate KVM should return to the guest and 0 is used to complete a 'normal' exit to userspace. user_mem_abort() deviates from this slightly, using 0 to return to the guest.
Just return 1 from user_mem_abort() to return to the guest and drop the return code conversion from kvm_handle_guest_abort(). It is now possible to do a 'normal' exit to userspace from user_mem_abort(), which will be used in a later change.
Quoted 19 lines
Signed-off-by Anish Moorthy <[email protected]>--- arch/arm64/kvm/mmu.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)arch/arm64/kvm/mmu.c +1
-1@@ -1190,7 +1190,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, struct kvm_memory_slot *memslot, unsigned long hva, unsigned long fault_status) { - int ret = 0; + int ret = 1;bool write_fault, writable, force_pte = false;bool exec_fault;bool device = false;
arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1284
Quoted 5 lines
arch/arm64/kvm/mmu.c +1
-1@@ -1281,8 +1281,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, (logging_active && write_fault)) { ret =kvm_mmu_topup_memory_cache (memcache,kvm_mmu_cache_min_pages (kvm)); -if (ret) +if (ret <0 )
There's no need to change this condition.
arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1287
Quoted 3 lines
arch/arm64/kvm/mmu.c +2
-0return ret; + else + ret =1 ;
I'd prefer if you set 'ret' close to where it is actually used, which I believe is only if mmu_invalidate_retry():
if (mmu_invalidate_retry(kvm, mmu_seq)) {
ret = 1;
goto out_unlock;
}Otherwise ret gets written to before exiting.
Signature · 2 lines
Thanks, Oliver
PATCH v2 12/14 KVM: arm64: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
In reply to Anish MoorthyWhen a memslot has the KVM_MEM_MEMORY_FAULT_EXIT flag set, exit to userspace upon encountering a page fault for which the userspace page tables do not contain a present mapping.
--- arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)
arch/arm64/kvm/arm.c +1 -0
@@ -220,6 +220,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_VCPU_ATTRIBUTES :case KVM_CAP_PTP_KVM :case KVM_CAP_ARM_SYSTEM_SUSPEND : +case KVM_CAP_MEMORY_FAULT_NOWAIT : r =1 ;break ;case KVM_CAP_SET_GUEST_DEBUG2 :
arch/arm64/kvm/mmu.c +12 -2
@@ -1206,6 +1206,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,unsigned long vma_pagesize, fault_granule;enum kvm_pgtable_prot prot =KVM_PGTABLE_PROT_R ;struct kvm_pgtable *pgt; +bool exit_on_memory_fault =kvm_slot_fault_on_absent_mapping (memslot); fault_granule =1UL <<ARM64_HW_PGTABLE_LEVEL_SHIFT (fault_level); write_fault =kvm_is_write_fault (vcpu);
@@ -1303,8 +1304,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,*/ smp_rmb (); - pfn =__gfn_to_pfn_memslot (memslot, gfn, false, false,NULL , - write_fault, &writable,NULL ); + pfn =__gfn_to_pfn_memslot ( + memslot, gfn, exit_on_memory_fault, false,NULL , + write_fault, &writable,NULL ); + +if (exit_on_memory_fault && pfn ==KVM_PFN_ERR_FAULT ) { + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; + vcpu->run->memory_fault.flags =0 ; + vcpu->run->memory_fault.gpa = gfn <<PAGE_SHIFT ; + vcpu->run->memory_fault.len = vma_pagesize; +return 0 ; + }if (pfn ==KVM_PFN_ERR_HWPOISON ) {kvm_send_hwpoison_signal (hva, vma_shift);return 1 ;
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Re: [WIP Patch v2 12/14] KVM: arm64: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
Quoted 36 lines · Anish Moorthy
When a memslot has the
KVM_MEM_MEMORY_FAULT_EXITflag set, exit to userspace upon encountering a page fault for which the userspace page tables do not contain a present mapping.Signed-off-by Anish Moorthy <[email protected]>Acked-by James Houghton <[email protected]>--- arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)arch/arm64/kvm/arm.c +1
-0@@ -220,6 +220,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_VCPU_ATTRIBUTES :case KVM_CAP_PTP_KVM :case KVM_CAP_ARM_SYSTEM_SUSPEND : +case KVM_CAP_MEMORY_FAULT_NOWAIT : r =1 ;break ;case KVM_CAP_SET_GUEST_DEBUG2 :arch/arm64/kvm/mmu.c +1
-0@@ -1206,6 +1206,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,unsigned long vma_pagesize, fault_granule;enum kvm_pgtable_prot prot =KVM_PGTABLE_PROT_R ;struct kvm_pgtable *pgt; +bool exit_on_memory_fault =kvm_slot_fault_on_absent_mapping (memslot); fault_granule =1UL <<ARM64_HW_PGTABLE_LEVEL_SHIFT (fault_level); write_fault =kvm_is_write_fault (vcpu);
arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1309
Quoted 8 lines
arch/arm64/kvm/mmu.c +3
-2@@ -1303,8 +1304,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,*/ smp_rmb (); - pfn =__gfn_to_pfn_memslot (memslot, gfn, false, false,NULL , - write_fault, &writable,NULL ); + pfn =__gfn_to_pfn_memslot ( + memslot, gfn, exit_on_memory_fault, false,NULL , + write_fault, &writable,NULL );
As stated before [*], this google3-esque style does not match the kernel style guide. You may want to check if your work machine is setting up a G3-specific editor configuration behind your back.
arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1311
Quoted 1 line
arch/arm64/kvm/mmu.c +1
-0+if (exit_on_memory_fault && pfn ==KVM_PFN_ERR_FAULT ) {
nit: I don't think the local is explicitly necessary. I still find this readable:
if (pfn == KVM_PFN_ERR_FAULT && kvm_slot_fault_on_absent_mapping(memslot))
Quoted 13 lines
arch/arm64/kvm/mmu.c +6
-0+ vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; + vcpu->run->memory_fault.flags =0 ; + vcpu->run->memory_fault.gpa = gfn <<PAGE_SHIFT ; + vcpu->run->memory_fault.len = vma_pagesize; +return 0 ; + }if (pfn ==KVM_PFN_ERR_HWPOISON ) {kvm_send_hwpoison_signal (hva, vma_shift);return 1 ;Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Signature · 2 lines
Thanks, Oliver
Re: [WIP Patch v2 12/14] KVM: arm64: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
Quoted 9 lines · Oliver Upton
arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1309
Quoted 3 lines
arch/arm64/kvm/mmu.c +3
-0+ pfn =__gfn_to_pfn_memslot ( + memslot, gfn, exit_on_memory_fault, false,NULL , + write_fault, &writable,NULL );As stated before [*], this google3-esque style does not match the kernel style guide. You may want to check if your work machine is setting up a G3-specific editor configuration behind your back.
If you're referring to the indentation, then that was definitely me. I'll give the style guide another readthrough before I submit the next version then, since checkpatch.pl doesn't seem to complain here.
Quoted 4 lines
Quoted 1 line
+ if (
exit_on_memory_fault&& pfn ==KVM_PFN_ERR_FAULT) {nit: I don't think the local is explicitly necessary. I still find this readable:
The local was for keeping a consistent value between the two blocks of code here
pfn = __gfn_to_pfn_memslot(
memslot, gfn, exit_on_memory_fault, false, NULL,
write_fault, &writable, NULL);
if (exit_on_memory_fault && pfn == KVM_PFN_ERR_FAULT) {
// Set up vCPU exit and return 0
}I wanted to avoid the possibility of causing an early __gfn_to_pfn_memslot exit but then not populating the vCPU exit.
Re: [WIP Patch v2 12/14] KVM: arm64: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
Quoted 31 lines · Anish Moorthy
Quoted 9 lines · Oliver Upton
arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1309
Quoted 3 lines
arch/arm64/kvm/mmu.c +3
-0+ pfn =__gfn_to_pfn_memslot ( + memslot, gfn, exit_on_memory_fault, false,NULL , + write_fault, &writable,NULL );As stated before [*], this google3-esque style does not match the kernel style guide. You may want to check if your work machine is setting up a G3-specific editor configuration behind your back.
If you're referring to the indentation, then that was definitely me. I'll give the style guide another readthrough before I submit the next version then, since checkpatch.pl doesn't seem to complain here.
Quoted 4 lines
Quoted 1 line
+ if (
exit_on_memory_fault&& pfn ==KVM_PFN_ERR_FAULT) {nit: I don't think the local is explicitly necessary. I still find this readable:
The local was for keeping a consistent value between the two blocks of code here
pfn = __gfn_to_pfn_memslot( memslot, gfn, exit_on_memory_fault, false, NULL, write_fault, &writable, NULL); if (exit_on_memory_fault && pfn == KVM_PFN_ERR_FAULT) { // Set up vCPU exit and return 0 }
Quoted 2 lines
I wanted to avoid the possibility of causing an early
__gfn_to_pfn_memslotexit but then not populating the vCPU exit.
Ignore me, I didn't see the other use of the local.
Signature · 2 lines
Thanks, Oliver
Sean Christopherson
maintainerRe: [WIP Patch v2 12/14] KVM: arm64: Implement KVM_CAP_MEMORY_FAULT_NOWAIT
In reply to Anish MoorthyQuoted 6 lines · Anish Moorthy
Quoted 4 lines · Oliver Upton
Quoted 3 lines
arch/arm64/kvm/mmu.c +3
-0+ pfn =__gfn_to_pfn_memslot ( + memslot, gfn, exit_on_memory_fault, false,NULL , + write_fault, &writable,NULL );
Quoted 7 lines
Quoted 5 lines
As stated before [*], this google3-esque style does not match the kernel style guide. You may want to check if your work machine is setting up a G3-specific editor configuration behind your back.
If you're referring to the indentation, then that was definitely me.
The two issues are (1) don't put newlines immediately after an opening '(', and
(2) align indentation relative to the direct parent '(' that encapsulates the code.
Concretely, the above should be:
pfn = __gfn_to_pfn_memslot(memslot, gfn, exit_on_memory_fault, false, NULL, write_fault, &writable, NULL);
Quoted 2 lines
I'll give the style guide another readthrough before I submit the next version then, since checkpatch.pl doesn't seem to complain here.
I don't think checkpatch looks for these particular style issues. FWIW, you really shouldn't need to read through the formal documentation for these "basic" rules, just spend time poking around the code base. If your code looks different than everything else, then you're likely doing it wrong.
PATCH v2 13/14 KVM: selftests: Add memslot_flags parameter to memstress_create_vm
In reply to Anish MoorthyMemslot flags aren't currently exposed to the tests, and are just always set to 0. Add a parameter to allow tests to manually set those flags.
--- tools/testing/selftests/kvm/access_tracking_perf_test.c | 2 +- tools/testing/selftests/kvm/demand_paging_test.c | 6 ++++-- tools/testing/selftests/kvm/dirty_log_perf_test.c | 2 +- tools/testing/selftests/kvm/include/memstress.h | 2 +- tools/testing/selftests/kvm/lib/memstress.c | 4 ++-- .../selftests/kvm/memslot_modification_stress_test.c | 2 +- 6 files changed, 10 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/access_tracking_perf_test.c +1 -1
@@ -306,7 +306,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)struct kvm_vm *vm;int nr_vcpus = params->nr_vcpus; - vm =memstress_create_vm (mode, nr_vcpus, params->vcpu_memory_bytes,1 , + vm =memstress_create_vm (mode, nr_vcpus, params->vcpu_memory_bytes,1 ,0 , params->backing_src, !overlap_memory_access);memstress_start_vcpu_threads (nr_vcpus, vcpu_thread_main);
tools/testing/selftests/kvm/demand_paging_test.c +4 -2
@@ -146,8 +146,10 @@ static void run_test(enum vm_guest_mode mode, void *arg)int i, num_uffds =0 ;uint64_t uffd_region_size; - vm =memstress_create_vm (mode, nr_vcpus, guest_percpu_mem_size,1 , - p->src_type, p->partition_vcpu_memory_access); + vm =memstress_create_vm ( + mode, nr_vcpus, guest_percpu_mem_size, +1 ,0 , + p->src_type, p->partition_vcpu_memory_access); demand_paging_size =get_backing_src_pagesz (p->src_type);
tools/testing/selftests/kvm/dirty_log_perf_test.c +1 -1
@@ -224,7 +224,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)int i; vm =memstress_create_vm (mode, nr_vcpus, guest_percpu_mem_size, - p->slots, p->backing_src, + p->slots,0 , p->backing_src, p->partition_vcpu_memory_access);pr_info ("Random seed: %u\n" , p->random_seed);
tools/testing/selftests/kvm/include/memstress.h +1 -1
@@ -56,7 +56,7 @@ struct memstress_args {extern struct memstress_args memstress_args;struct kvm_vm *memstress_create_vm (enum vm_guest_mode mode,int nr_vcpus, -uint64_t vcpu_memory_bytes,int slots, +uint64_t vcpu_memory_bytes,int slots,uint32_t slot_flags,enum vm_mem_backing_src_type backing_src,bool partition_vcpu_memory_access);void memstress_destroy_vm (struct kvm_vm *vm);
tools/testing/selftests/kvm/lib/memstress.c +2 -2
@@ -119,7 +119,7 @@ void memstress_setup_vcpus(struct kvm_vm *vm, int nr_vcpus, }struct kvm_vm *memstress_create_vm (enum vm_guest_mode mode,int nr_vcpus, -uint64_t vcpu_memory_bytes,int slots, +uint64_t vcpu_memory_bytes,int slots,uint32_t slot_flags,enum vm_mem_backing_src_type backing_src,bool partition_vcpu_memory_access) {
@@ -207,7 +207,7 @@ struct kvm_vm *memstress_create_vm(enum vm_guest_mode mode, int nr_vcpus,vm_userspace_mem_region_add (vm, backing_src, region_start,MEMSTRESS_MEM_SLOT_INDEX + i, - region_pages,0 ); + region_pages, slot_flags); }/* Do mapping for the demand paging memory slot */
tools/testing/selftests/kvm/memslot_modification_stress_test.c +1 -1
@@ -95,7 +95,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)struct test_params *p = arg;struct kvm_vm *vm; - vm =memstress_create_vm (mode, nr_vcpus, guest_percpu_mem_size,1 , + vm =memstress_create_vm (mode, nr_vcpus, guest_percpu_mem_size,1 ,0 ,VM_MEM_SRC_ANONYMOUS , p->partition_vcpu_memory_access);
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
PATCH v2 14/14 KVM: selftests: Handle memory fault exits in demand_paging_test
In reply to Anish MoorthyDemonstrate a (very basic) scheme for supporting memory fault exits.
From the vCPU threads:
1. Simply issue UFFDIO_COPY/CONTINUEs in response to memory fault exits, with the purpose of establishing the absent mappings. Do so with wake_waiters=false to avoid serializing on the userfaultfd wait queue locks.
2. When the UFFDIO_COPY/CONTINUE in (1) fails with EEXIST, assume that the mapping was already established but is currently absent [A] and attempt to populate it using MADV_POPULATE_WRITE.
Issue UFFDIO_COPY/CONTINUEs from the reader threads as well, but with wake_waiters=true to ensure that any threads sleeping on the uffd are eventually woken up.
A real VMM would track whether it had already COPY/CONTINUEd pages (eg, via a bitmap) to avoid calls destined to EEXIST. However, even the naive approach is enough to demonstrate the performance advantages of KVM_EXIT_MEMORY_FAULT.
[A] In reality it is much likelier that the vCPU thread simply lost a race to establish the mapping for the page.
--- .../selftests/kvm/demand_paging_test.c | 220 +++++++++++++----- 1 file changed, 164 insertions(+), 56 deletions(-)
tools/testing/selftests/kvm/demand_paging_test.c +164 -56
@@ -15,6 +15,7 @@#include <time.h> #include <pthread.h> #include <linux/userfaultfd.h> +#include <sys/mman.h> #include <sys/syscall.h> #include "kvm_util.h"
@@ -31,6 +32,60 @@ static uint64_t guest_percpu_mem_size = DEFAULT_PER_VCPU_MEM_SIZE;static size_t demand_paging_size;static char *guest_data_prototype; +static int num_uffds; +static size_t uffd_region_size; +static struct uffd_desc **uffd_descs; +/* +* Delay when demand paging is performed through userfaultfd or directly by +* vcpu_worker in the case of a KVM_EXIT_MEMORY_FAULT. +*/ +static useconds_t uffd_delay; +static int uffd_mode; + + +static int handle_uffd_page_request ( +int uffd_mode,int uffd,uint64_t hva,bool is_vcpu +); + +static void madv_write_or_err (uint64_t gpa) +{ +int r; +void *hva =addr_gpa2hva (memstress_args.vm, gpa); + + r =madvise (hva, demand_paging_size,MADV_POPULATE_WRITE ); +TEST_ASSERT ( + r ==0 , +"MADV_POPULATE_WRITE on hva 0x%lx (gpa 0x%lx) failed with errno %i\n" , + (uintptr_t ) hva, gpa, errno); +} + +static void ready_page (uint64_t gpa) +{ +int r, uffd; + +/* +* This test only registers memslot 1 w/ userfaultfd. Any accesses outside +* the registered ranges should fault in the physical pages through +* MADV_POPULATE_WRITE. +*/ +if ((gpa < memstress_args.gpa) + || (gpa >= memstress_args.gpa + memstress_args.size)) { +madv_write_or_err (gpa); + }else { +if (uffd_delay) +usleep (uffd_delay); + + uffd = uffd_descs[(gpa - memstress_args.gpa) / uffd_region_size]->uffd; + + r =handle_uffd_page_request ( + uffd_mode, uffd, + (uint64_t )addr_gpa2hva (memstress_args.vm, gpa), true); + +if (r ==EEXIST ) +madv_write_or_err (gpa); + } +} +static void vcpu_worker (struct memstress_vcpu_args *vcpu_args) {struct kvm_vcpu *vcpu = vcpu_args->vcpu;
@@ -42,25 +97,37 @@ static void vcpu_worker(struct memstress_vcpu_args *vcpu_args)clock_gettime (CLOCK_MONOTONIC , &start); -/* Let the guest access its memory */ - ret =_vcpu_run (vcpu); -TEST_ASSERT (ret ==0 ,"vcpu_run failed: %d\n" , ret); -if (get_ucall (vcpu,NULL ) !=UCALL_SYNC ) { -TEST_ASSERT (false, -"Invalid guest sync status: exit_reason=%s\n" , -exit_reason_str (run->exit_reason)); - } +while (true) { +/* Let the guest access its memory */ + ret =_vcpu_run (vcpu); +TEST_ASSERT (ret ==0 || (run->exit_reason ==KVM_EXIT_MEMORY_FAULT ), +"vcpu_run failed: %d\n" , ret); +if (get_ucall (vcpu,NULL ) !=UCALL_SYNC ) { + +if (run->exit_reason ==KVM_EXIT_MEMORY_FAULT ) { +TEST_ASSERT (run->memory_fault.flags ==0 , +"Unrecognized flags 0x%llx on memory fault exit" , + run->memory_fault.flags); +ready_page (run->memory_fault.gpa); +continue ; + } + +TEST_ASSERT (false, +"Invalid guest sync status: exit_reason=%s\n" , +exit_reason_str (run->exit_reason)); + } - ts_diff =timespec_elapsed (start); -PER_VCPU_DEBUG ("vCPU %d execution time: %ld.%.9lds\n" , vcpu_idx, - ts_diff.tv_sec, ts_diff.tv_nsec); + ts_diff =timespec_elapsed (start); +PER_VCPU_DEBUG ("vCPU %d execution time: %ld.%.9lds\n" , vcpu_idx, + ts_diff.tv_sec, ts_diff.tv_nsec); +break ; + } } -static int handle_uffd_page_request (int uffd_mode,int uffd, -struct uffd_msg *msg) +static int handle_uffd_page_request ( +int uffd_mode,int uffd,uint64_t hva,bool is_vcpu) {pid_t tid =syscall (__NR_gettid); -uint64_t addr = msg->arg.pagefault.address;struct timespec start;struct timespec ts_diff;int r;
@@ -71,58 +138,81 @@ static int handle_uffd_page_request(int uffd_mode, int uffd,struct uffdio_copy copy; copy.src = (uint64_t )guest_data_prototype; - copy.dst = addr; + copy.dst = hva; copy.len = demand_paging_size; - copy.mode =0 ; + copy.mode =UFFDIO_COPY_MODE_DONTWAKE ; - r =ioctl (uffd,UFFDIO_COPY , ©);/* -* With multiple vCPU threads fault on a single page and there are -* multiple readers for the UFFD, at least one of the UFFDIO_COPYs -* will fail with EEXIST: handle that case without signaling an -* error. +* With multiple vCPU threads and at least one of multiple reader threads +* or vCPU memory faults, multiple vCPUs accessing an absent page will +* almost certainly cause some thread doing the UFFDIO_COPY here to get +* EEXIST: make sure to allow that case. */ -if (r ==-1 && errno !=EEXIST ) { -pr_info ( -"Failed UFFDIO_COPY in 0x%lx from thread %d, errno = %d\n" , - addr, tid, errno); -return r; - } + r =ioctl (uffd,UFFDIO_COPY , ©); +TEST_ASSERT ( + r ==0 || errno ==EEXIST , +"Thread 0x%x failed UFFDIO_COPY on hva 0x%lx, errno = %d" , +gettid (), hva, errno); }else if (uffd_mode ==UFFDIO_REGISTER_MODE_MINOR ) { +/* The comments in the UFFDIO_COPY branch also apply here. */ struct uffdio_continue cont = {0 }; - cont.range.start = addr; + cont.range.start = hva; cont.range.len = demand_paging_size; + cont.mode =UFFDIO_CONTINUE_MODE_DONTWAKE ; r =ioctl (uffd,UFFDIO_CONTINUE , &cont); -/* See the note about EEXISTs in the UFFDIO_COPY branch. */ -if (r ==-1 && errno !=EEXIST ) { -pr_info ( -"Failed UFFDIO_CONTINUE in 0x%lx from thread %d, errno = %d\n" , - addr, tid, errno); -return r; - } +TEST_ASSERT ( + r ==0 || errno ==EEXIST , +"Thread 0x%x failed UFFDIO_CONTINUE on hva 0x%lx, errno = %d" , +gettid (), hva, errno); }else {TEST_FAIL ("Invalid uffd mode %d" , uffd_mode); } +/* +* If the above UFFDIO_COPY/CONTINUE fails with EEXIST, it will do so without +* waking threads waiting on the UFFD: make sure that happens here. +*/ +if (!is_vcpu) { +struct uffdio_range range = { + .start = hva, + .len = demand_paging_size + }; + r =ioctl (uffd,UFFDIO_WAKE , &range); +TEST_ASSERT ( + r ==0 , +"Thread 0x%x failed UFFDIO_WAKE on hva 0x%lx, errno = %d" , +gettid (), hva, errno); + } + ts_diff =timespec_elapsed (start);PER_PAGE_DEBUG ("UFFD page-in %d \t%ld ns\n" , tid,timespec_to_ns (ts_diff));PER_PAGE_DEBUG ("Paged in %ld bytes at 0x%lx from thread %d\n" , - demand_paging_size, addr, tid); + demand_paging_size, hva, tid);return 0 ; } +static int handle_uffd_page_request_from_uffd ( +int uffd_mode,int uffd,struct uffd_msg *msg) +{ +TEST_ASSERT ( + msg->event ==UFFD_EVENT_PAGEFAULT , +"Received uffd message with event %d != UFFD_EVENT_PAGEFAULT" , + msg->event); +return handle_uffd_page_request ( + uffd_mode, uffd, msg->arg.pagefault.address, false); +} +struct test_params { -int uffd_mode;bool single_uffd; -useconds_t uffd_delay;int readers_per_uffd;enum vm_mem_backing_src_type src_type;bool partition_vcpu_memory_access; +bool memfault_exits; };static void prefault_mem (void *alias,uint64_t len)
@@ -139,18 +229,31 @@ static void prefault_mem(void *alias, uint64_t len)static void run_test (enum vm_guest_mode mode,void *arg) {struct test_params *p = arg; -struct uffd_desc **uffd_descs =NULL ;struct timespec start;struct timespec ts_diff;struct kvm_vm *vm; -int i, num_uffds =0 ; -uint64_t uffd_region_size; +int i; +uint32_t slot_flags =0 ; +bool uffd_memfault_exits = uffd_mode && p->memfault_exits; + +if (uffd_memfault_exits) { +TEST_ASSERT (kvm_has_cap (KVM_CAP_MEMORY_FAULT_NOWAIT ) >0 , +"KVM does not have KVM_CAP_MEMORY_FAULT_NOWAIT" ); + slot_flags =KVM_MEM_ABSENT_MAPPING_FAULT ; + } vm =memstress_create_vm ( mode, nr_vcpus, guest_percpu_mem_size, -1 ,0 , +1 , slot_flags, p->src_type, p->partition_vcpu_memory_access); +if (uffd_memfault_exits) { +if (kvm_has_cap (KVM_CAP_X86_MEMORY_FAULT_EXIT )) +vm_enable_cap ( + vm,KVM_CAP_X86_MEMORY_FAULT_EXIT , +KVM_MEMFAULT_REASON_ABSENT_MAPPING ); + } + demand_paging_size =get_backing_src_pagesz (p->src_type); guest_data_prototype =malloc (demand_paging_size);
@@ -158,12 +261,12 @@ static void run_test(enum vm_guest_mode mode, void *arg) "Failed to allocate buffer for guest data pattern");memset (guest_data_prototype,0xAB , demand_paging_size); -if (p->uffd_mode) { +if (uffd_mode) { num_uffds = p->single_uffd ?1 : nr_vcpus; uffd_region_size = nr_vcpus * guest_percpu_mem_size / num_uffds; uffd_descs =malloc (num_uffds *sizeof (struct uffd_desc *)); -TEST_ASSERT (uffd_descs,"Memory allocation failed" ); +TEST_ASSERT (uffd_descs,"Failed to allocate memory of uffd descriptors" );for (i =0 ; i < num_uffds; i++) {struct memstress_vcpu_args *vcpu_args;
@@ -183,10 +286,10 @@ static void run_test(enum vm_guest_mode mode, void *arg)* requests. */ uffd_descs[i] =uffd_setup_demand_paging ( - p->uffd_mode, p->uffd_delay, vcpu_hva, + uffd_mode, uffd_delay, vcpu_hva, uffd_region_size, p->readers_per_uffd, - &handle_uffd_page_request); + &handle_uffd_page_request_from_uffd); } }
@@ -200,7 +303,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) ts_diff =timespec_elapsed (start);pr_info ("All vCPU threads joined\n" ); -if (p->uffd_mode) { +if (uffd_mode) {/* Tell the user fault fd handler threads to quit */ for (i =0 ; i < num_uffds; i++)uffd_stop_demand_paging (uffd_descs[i]);
@@ -215,7 +318,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)memstress_destroy_vm (vm);free (guest_data_prototype); -if (p->uffd_mode) +if (uffd_mode)free (uffd_descs); }
@@ -224,7 +327,7 @@ static void help(char *name)puts ("" );printf ("usage: %s [-h] [-m vm_mode] [-u uffd_mode] [-a]\n" " [-d uffd_delay_usec] [-r readers_per_uffd] [-b memory]\n" -" [-s type] [-v vcpus] [-o]\n" , name); +" [-w] [-s type] [-v vcpus] [-o]\n" , name);guest_modes_help ();printf (" -u: use userfaultfd to handle vCPU page faults. Mode is a\n" " UFFD registration mode: 'MISSING' or 'MINOR'.\n" );
@@ -235,6 +338,7 @@ static void help(char *name)" FD handler to simulate demand paging\n" " overheads. Ignored without -u.\n");printf (" -r: Set the number of reader threads per uffd.\n" ); +printf (" -w: Enable kvm cap for memory fault exits.\n" );printf (" -b: specify the size of the memory region which should be\n" " demand paged by each vCPU. e.g. 10M or 3G.\n" " Default: 1G\n" );
@@ -254,29 +358,30 @@ int main(int argc, char *argv[]) .partition_vcpu_memory_access = true, .readers_per_uffd = 1, .single_uffd = false, + .memfault_exits = false, };int opt;guest_modes_append_default (); -while ((opt =getopt (argc, argv,"ahom:u:d:b:s:v:r:" )) !=-1 ) { +while ((opt =getopt (argc, argv,"ahowm:u:d:b:s:v:r:" )) !=-1 ) {switch (opt) {case 'm' :guest_modes_cmdline (optarg);break ;case 'u' :if (!strcmp ("MISSING" , optarg)) - p.uffd_mode =UFFDIO_REGISTER_MODE_MISSING ; + uffd_mode =UFFDIO_REGISTER_MODE_MISSING ;else if (!strcmp ("MINOR" , optarg)) - p.uffd_mode =UFFDIO_REGISTER_MODE_MINOR ; -TEST_ASSERT (p.uffd_mode,"UFFD mode must be 'MISSING' or 'MINOR'." ); + uffd_mode =UFFDIO_REGISTER_MODE_MINOR ; +TEST_ASSERT (uffd_mode,"UFFD mode must be 'MISSING' or 'MINOR'." );break ;case 'a' : p.single_uffd = true;break ;case 'd' : - p.uffd_delay =strtoul (optarg,NULL ,0 ); -TEST_ASSERT (p.uffd_delay >=0 ,"A negative UFFD delay is not supported." ); + uffd_delay =strtoul (optarg,NULL ,0 ); +TEST_ASSERT (uffd_delay >=0 ,"A negative UFFD delay is not supported." );break ;case 'b' : guest_percpu_mem_size =parse_size (optarg);
@@ -299,6 +404,9 @@ int main(int argc, char *argv[])"Invalid number of readers per uffd %d: must be >=1" , p.readers_per_uffd);break ; +case 'w' : + p.memfault_exits = true; +break ;case 'h' :default :help (argv[0 ]);
@@ -306,7 +414,7 @@ int main(int argc, char *argv[]) } } -if (p.uffd_mode ==UFFDIO_REGISTER_MODE_MINOR && +if (uffd_mode ==UFFDIO_REGISTER_MODE_MINOR && !backing_src_is_shared (p.src_type)) {TEST_FAIL ("userfaultfd MINOR mode requires shared memory; pick a different -s" ); }
Signature · 1 line
2.40.0.rc1.284.g88254d51c5-goog
Anish,
Generally the 'RFC PATCH' prefix is used for patches that are for feedback only (i.e. not to be considered for inclusion).
Quoted 2 lines · Anish Moorthy
Hi Sean, here's what I'm planing to send up as v2 of the scalable userfaultfd series.
I don't see a ton of value in sending a targeted posting of a series to the list. IOW, just CC all of the appropriate reviewers+maintainers. I promise, we won't bite.
Quoted 46 lines
Don't worry, I'm not asking you to review this all :) I just have a few remaining questions regarding
KVM_CAP_MEMORY_FAULT_EXITwhich seem important enough to mention before I ask for more attention from others, and they'll be clearer with the patches in hand. Anything else I'm happy to find out about when I send the actual v2.I want your opinion on
1. The general API I've set up for
KVM_CAP_MEMORY_FAULT_EXIT(described in theapi.rstfile)
2. Whether the UNKNOWN exit reason cases (everywhere buthandle_error_pfnatm) would need to be given "real" reasons before this could be merged.
3. If you think I've missed sites that currently-EFAULTto userspaceAbout (3): after we agreed to only tackle cases where
-EFAULTcurrently makes it to userspace, I went though our list and tried to trace which EFAULTS actually bubble up toKVM_RUN. That set ended being suspiciously small, so I wanted to sanity-check my findings with you. Lmk if you see obvious errors in my list below.--- EFAULTs under
KVM_RUN---Confident that needs conversion (already converted)
---------------------------------------------------
*direct_map
*handle_error_pfn
*setup_vmgexit_scratch
*kvm_handle_page_fault
* FNAME(fetch)EFAULT does not propagate to userspace (do not convert)
-------------------------------------------------------
*record_steal_time(arch/x86/kvm/x86.c:3463)
*hva_to_pfn_retry
*kvm_vcpu_map
*FNAME(update_accessed_dirty_bits)
*__kvm_gfn_to_hva_cache_init
Might actually make it to userspace, but only through kvm_read|write_guest_offset_cached- would be covered by those conversions
*kvm_gfn_to_hva_cache_init
*__kvm_read_guest_page
*hva_to_pfn_remapped
handle_error_pfnwill handle this for the scalable uffd case. Don't think other callers-EFAULTto userspace.
Quoted 7 lines
Still unsure if needs conversion
--------------------------------
*__kvm_read_guest_atomic
The EFAULT might be propagated thoughFNAME(sync_page)?
*kvm_write_guest_offset_cached(virt/kvm/kvm_main.c:3226)
*__kvm_write_guest_page
Called fromkvm_write_guest_offset_cached: if that needs change, this does too
The low-level accessors are common across architectures and can be called from other contexts besides a vCPU. Is it possible for the caller to catch -EFAULT and convert that into an exit?
Quoted 4 lines
*
kvm_write_guest_page
Two interesting paths:
-kvm_pv_clock_pairingreturns a customKVM_EFAULTerror here (arch/x86/kvm/x86.c:9578)
This is a hypercall handler, so the return code is ABI with the guest. So it shouldn't be converted to an exit to userspace.
Signature · 2 lines
Thanks, Oliver
Sean Christopherson
maintainerQuoted 7 lines · Oliver Upton
Quoted 2 lines · Anish Moorthy
Hi Sean, here's what I'm planing to send up as v2 of the scalable userfaultfd series.
I don't see a ton of value in sending a targeted posting of a series to the list. IOW, just CC all of the appropriate reviewers+maintainers. I promise, we won't bite.
+1. And though I discourage off-list review, if something is really truly not ready for public review, e.g. will do more harm than good by causing confusing, then just send the patches off-list. Half measures like this will just make folks grumpy.
Quoted 54 lines
Quoted 46 lines
Don't worry, I'm not asking you to review this all :) I just have a few remaining questions regarding
KVM_CAP_MEMORY_FAULT_EXITwhich seem important enough to mention before I ask for more attention from others, and they'll be clearer with the patches in hand. Anything else I'm happy to find out about when I send the actual v2.I want your opinion on
1. The general API I've set up for
KVM_CAP_MEMORY_FAULT_EXIT(described in theapi.rstfile)
2. Whether the UNKNOWN exit reason cases (everywhere buthandle_error_pfnatm) would need to be given "real" reasons before this could be merged.
3. If you think I've missed sites that currently-EFAULTto userspaceAbout (3): after we agreed to only tackle cases where
-EFAULTcurrently makes it to userspace, I went though our list and tried to trace which EFAULTS actually bubble up toKVM_RUN. That set ended being suspiciously small, so I wanted to sanity-check my findings with you. Lmk if you see obvious errors in my list below.--- EFAULTs under
KVM_RUN---Confident that needs conversion (already converted)
---------------------------------------------------
*direct_map
*handle_error_pfn
*setup_vmgexit_scratch
*kvm_handle_page_fault
* FNAME(fetch)EFAULT does not propagate to userspace (do not convert)
-------------------------------------------------------
*record_steal_time(arch/x86/kvm/x86.c:3463)
*hva_to_pfn_retry
*kvm_vcpu_map
*FNAME(update_accessed_dirty_bits)
*__kvm_gfn_to_hva_cache_init
Might actually make it to userspace, but only through kvm_read|write_guest_offset_cached- would be covered by those conversions
*kvm_gfn_to_hva_cache_init
*__kvm_read_guest_page
*hva_to_pfn_remapped
handle_error_pfnwill handle this for the scalable uffd case. Don't think other callers-EFAULTto userspace.Quoted 7 lines
Still unsure if needs conversion
--------------------------------
*__kvm_read_guest_atomic
The EFAULT might be propagated thoughFNAME(sync_page)?
*kvm_write_guest_offset_cached(virt/kvm/kvm_main.c:3226)
*__kvm_write_guest_page
Called fromkvm_write_guest_offset_cached: if that needs change, this does too
Quoted 3 lines
The low-level accessors are common across architectures and can be called from other contexts besides a vCPU. Is it possible for the caller to catch
-EFAULTand convert that into an exit?
Ya, as things stand today, the conversions _must_ be performed at the caller, as there are (sadly) far too many flows where KVM squashes the error. E.g. almost all of x86's paravirt code just suppresses user memory faults :-(
Anish, when we discussed this off-list, what I meant by limiting the intial support to existing -EFAULT cases was limiting support to existing cases where KVM directly returns -EFAULT to userspace, not to all existing cases where -EFAULT is ever returned _within KVM_ while handling KVM_RUN. My apologies if I didn't make that clear.
Quoted 8 lines · Sean Christopherson
Quoted 6 lines · Oliver Upton
Quoted 2 lines · Anish Moorthy
Hi Sean, here's what I'm planing to send up as v2 of the scalable userfaultfd series.
I don't see a ton of value in sending a targeted posting of a series to the list.
But isn't it already generating value as you were able to weigh in and provide feedback on technical aspects that you would not have been otherwise able to if Anish had just messaged Sean?
Quoted 2 lines
Quoted 2 lines
IOW, just CC all of the appropriate reviewers+maintainers. I promise, we won't bite.
I disagree. While I think it's fine to reach out to someone off-list to discuss a specific question, if you're going to message all reviewers and maintainers, you should also CC the mailing list. That allows more people to follow along and weigh in if necessary.
Quoted 5 lines
+1. And though I discourage off-list review, if something is really truly not ready for public review, e.g. will do more harm than good by causing confusing, then just send the patches off-list. Half measures like this will just make folks grumpy.
In this specific case, Anish very clearly laid out the reason for sending the patches and asked very specific directed questions in the cover letter and called it out as WIP. Yes "WIP" should have been "RFC" but other than that should anything have been different?
David,
Quoted 7 lines · David Matlack
Quoted 6 lines · Sean Christopherson
Quoted 4 lines · Oliver Upton
Quoted 2 lines · Anish Moorthy
Hi Sean, here's what I'm planing to send up as v2 of the scalable userfaultfd series.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
I don't see a ton of value in sending a targeted posting of a series to the list.
But isn't it already generating value as you were able to weigh in and provide feedback on technical aspects that you would not have been otherwise able to if Anish had just messaged Sean?
No, I only happened upon this series looking at lore. My problem is that none of the affected maintainers or reviewers were cc'ed on the series.
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
IOW, just CC all of the appropriate reviewers+maintainers. I promise, we won't bite.
I disagree. While I think it's fine to reach out to someone off-list to discuss a specific question, if you're going to message all reviewers and maintainers, you should also CC the mailing list. That allows more people to follow along and weigh in if necessary.
I think there may be a slight disconnect here :) I'm in no way encouraging off-list discussion and instead asking that mail on the list arrives in the right folks' inboxes.
Posting an RFC on the list was absolutely the right thing to do.
Quoted 10 lines
Quoted 5 lines
+1. And though I discourage off-list review, if something is really truly not ready for public review, e.g. will do more harm than good by causing confusing, then just send the patches off-list. Half measures like this will just make folks grumpy.
In this specific case, Anish very clearly laid out the reason for sending the patches and asked very specific directed questions in the cover letter and called it out as WIP. Yes "WIP" should have been "RFC" but other than that should anything have been different?
See above
Signature · 2 lines
Thanks, Oliver
Quoted 29 lines · Oliver Upton
David,
Quoted 7 lines · David Matlack
Quoted 6 lines · Sean Christopherson
Quoted 4 lines · Oliver Upton
Quoted 2 lines · Anish Moorthy
Hi Sean, here's what I'm planing to send up as v2 of the scalable userfaultfd series.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
I don't see a ton of value in sending a targeted posting of a series to the list.
But isn't it already generating value as you were able to weigh in and provide feedback on technical aspects that you would not have been otherwise able to if Anish had just messaged Sean?
No, I only happened upon this series looking at lore. My problem is that none of the affected maintainers or reviewers were cc'ed on the series.
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
IOW, just CC all of the appropriate reviewers+maintainers. I promise, we won't bite.
I disagree. While I think it's fine to reach out to someone off-list to discuss a specific question, if you're going to message all reviewers and maintainers, you should also CC the mailing list. That allows more people to follow along and weigh in if necessary.
Quoted 5 lines
I think there may be a slight disconnect here :) I'm in no way encouraging off-list discussion and instead asking that mail on the list arrives in the right folks' inboxes.
Posting an RFC on the list was absolutely the right thing to do.
Doh. I misunderstood what you meant. We are in violent agreement!
Quoted 31 lines · David Matlack
Quoted 29 lines · Oliver Upton
David,
Quoted 7 lines · David Matlack
Quoted 6 lines · Sean Christopherson
Quoted 4 lines · Oliver Upton
Quoted 2 lines · Anish Moorthy
Hi Sean, here's what I'm planing to send up as v2 of the scalable userfaultfd series.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
I don't see a ton of value in sending a targeted posting of a series to the list.
But isn't it already generating value as you were able to weigh in and provide feedback on technical aspects that you would not have been otherwise able to if Anish had just messaged Sean?
No, I only happened upon this series looking at lore. My problem is that none of the affected maintainers or reviewers were cc'ed on the series.
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
IOW, just CC all of the appropriate reviewers+maintainers. I promise, we won't bite.
I disagree. While I think it's fine to reach out to someone off-list to discuss a specific question, if you're going to message all reviewers and maintainers, you should also CC the mailing list. That allows more people to follow along and weigh in if necessary.
Quoted 7 lines
Quoted 5 lines
I think there may be a slight disconnect here :) I'm in no way encouraging off-list discussion and instead asking that mail on the list arrives in the right folks' inboxes.
Posting an RFC on the list was absolutely the right thing to do.
Doh. I misunderstood what you meant. We are in violent agreement!
Noted. Also, thanks Oliver and Isaku for paying attention to the series despite it being obscure.
Quoted 14 lines · Sean Christopherson
Quoted 11 lines · Oliver Upton
Quoted 7 lines
Still unsure if needs conversion
--------------------------------
*__kvm_read_guest_atomic
The EFAULT might be propagated thoughFNAME(sync_page)?
*kvm_write_guest_offset_cached(virt/kvm/kvm_main.c:3226)
*__kvm_write_guest_page
Called fromkvm_write_guest_offset_cached: if that needs change, this does tooThe low-level accessors are common across architectures and can be called from other contexts besides a vCPU. Is it possible for the caller to catch
-EFAULTand convert that into an exit?
Quoted 8 lines
Ya, as things stand today, the conversions _must_ be performed at the caller, as there are (sadly) far too many flows where KVM squashes the error. E.g. almost all of x86's paravirt code just suppresses user memory faults :-(
Anish, when we discussed this off-list, what I meant by limiting the intial support to existing
-EFAULTcases was limiting support to existing cases where KVM directly returns-EFAULTto userspace, not to all existing cases where-EFAULTis ever returned_withinKVM_while handlingKVM_RUN. My apologies if I didn't make that clear.
Don't worry, we eventually got there off-list :)
This brings us back to my original set of questions. As has already been pointed out, I'll have to revisit my "Confident that needs conversion" changes and tweak them so that the vCPU exit is populated only for the call sites where the -EFAULT makes it to userspace. I still want feedback on if I've mis-identified any of the functions in my "EFAULT does not propagate to userspace" list and whether there are functions/callers in the "Still unsure if needs conversion" which do have return paths to KVM_RUN.
Sean Christopherson
maintainerQuoted 16 lines · Anish Moorthy
Quoted 4 lines · David Matlack
Quoted 3 lines
The low-level accessors are common across architectures and can be called from other contexts besides a vCPU. Is it possible for the caller to catch
-EFAULTand convert that into an exit?Quoted 8 lines
Ya, as things stand today, the conversions _must_ be performed at the caller, as there are (sadly) far too many flows where KVM squashes the error. E.g. almost all of x86's paravirt code just suppresses user memory faults :-(
Anish, when we discussed this off-list, what I meant by limiting the intial support to existing
-EFAULTcases was limiting support to existing cases where KVM directly returns-EFAULTto userspace, not to all existing cases where-EFAULTis ever returned_withinKVM_while handlingKVM_RUN. My apologies if I didn't make that clear.Don't worry, we eventually got there off-list :)
Quoted 8 lines
This brings us back to my original set of questions. As has already been pointed out, I'll have to revisit my "Confident that needs conversion" changes and tweak them so that the vCPU exit is populated only for the call sites where the
-EFAULTmakes it to userspace. I still want feedback on if I've mis-identified any of the functions in my "EFAULT does not propagate to userspace" list and whether there are functions/callers in the "Still unsure if needs conversion" which do have return paths toKVM_RUN.
As you've probably gathered from the type of feedback you're receiving, identifying the conversion touchpoints isn't going to be the long pole of this series. Correctly identifying all of the touchpoints may not be easy, but fixing any cases we get wrong will likely be straightforward. And realistically, no matter how many eyeballs look at the code, odds are good we'll miss at least one case. In other words, don't worry too much about getting all the touchpoints correct on the first version. Getting the uAPI right is much more important.
And rather than rely on code review to get things right, we should be able to detect issues programmatically. E.g. use fault injection to make gup() and/or uaccess fail (might even be wired up already?), and hack in a WARN in the KVM_RUN path to assert that KVM_EXIT_MEMORY_FAULT is filled if the return code is -EFAULT (assuming we go don't try to get KVM to return 0 everywhere), e.g. something like the below would at least flag the "misses", although debug could still prove to be annoying.
virt/kvm/kvm_main.c +2 -0
@@ -4100,6 +4100,8 @@ static long kvm_vcpu_ioctl(struct file *filp, } r =kvm_arch_vcpu_ioctl_run (vcpu);trace_kvm_userspace_exit (vcpu->run->exit_reason, r); +WARN_ON (r == -EFAULT && + vcpu->run->exit_reason ==KVM_EXIT_MEMORY_FAULT );break ; }case KVM_GET_REGS : {
Sean Christopherson
maintainerQuoted 47 lines · Sean Christopherson
Quoted 16 lines · Anish Moorthy
Quoted 4 lines · David Matlack
Quoted 3 lines
The low-level accessors are common across architectures and can be called from other contexts besides a vCPU. Is it possible for the caller to catch
-EFAULTand convert that into an exit?Quoted 8 lines
Ya, as things stand today, the conversions _must_ be performed at the caller, as there are (sadly) far too many flows where KVM squashes the error. E.g. almost all of x86's paravirt code just suppresses user memory faults :-(
Anish, when we discussed this off-list, what I meant by limiting the intial support to existing
-EFAULTcases was limiting support to existing cases where KVM directly returns-EFAULTto userspace, not to all existing cases where-EFAULTis ever returned_withinKVM_while handlingKVM_RUN. My apologies if I didn't make that clear.Don't worry, we eventually got there off-list :)
Quoted 8 lines
This brings us back to my original set of questions. As has already been pointed out, I'll have to revisit my "Confident that needs conversion" changes and tweak them so that the vCPU exit is populated only for the call sites where the
-EFAULTmakes it to userspace. I still want feedback on if I've mis-identified any of the functions in my "EFAULT does not propagate to userspace" list and whether there are functions/callers in the "Still unsure if needs conversion" which do have return paths toKVM_RUN.As you've probably gathered from the type of feedback you're receiving, identifying the conversion touchpoints isn't going to be the long pole of this series. Correctly identifying all of the touchpoints may not be easy, but fixing any cases we get wrong will likely be straightforward. And realistically, no matter how many eyeballs look at the code, odds are good we'll miss at least one case. In other words, don't worry too much about getting all the touchpoints correct on the first version. Getting the uAPI right is much more important.
And rather than rely on code review to get things right, we should be able to detect issues programmatically. E.g. use fault injection to make
gup()and/or uaccess fail (might even be wired up already?), and hack in a WARN in theKVM_RUNpath to assert thatKVM_EXIT_MEMORY_FAULTis filled if the return code is-EFAULT(assuming we go don't try to get KVM to return 0 everywhere), e.g. something like the below would at least flag the "misses", although debug could still prove to be annoying.virt/kvm/kvm_main.c +0
-0
virt/kvm/kvm_main.c static long kvm_vcpu_ioctl(struct file *filp, · line 4104
Quoted 5 lines
virt/kvm/kvm_main.c +2
-0@@ -4100,6 +4100,8 @@ static long kvm_vcpu_ioctl(struct file *filp, } r =kvm_arch_vcpu_ioctl_run (vcpu);trace_kvm_userspace_exit (vcpu->run->exit_reason, r); +WARN_ON (r == -EFAULT && + vcpu->run->exit_reason ==KVM_EXIT_MEMORY_FAULT );
Gah, I inverted the second check, this should be
WARN_ON(r == -EFAULT && vcpu->run->exit_reason != KVM_EXIT_MEMORY_FAULT);
Quoted 4 lines
virt/kvm/kvm_main.c +0
-0break ; }case KVM_GET_REGS : {
Sean Christopherson
maintainerQuoted 8 lines · Anish Moorthy
Still unsure if needs conversion
--------------------------------
*__kvm_read_guest_atomic
The EFAULT might be propagated thoughFNAME(sync_page)?
*kvm_write_guest_offset_cached(virt/kvm/kvm_main.c:3226)
*__kvm_write_guest_page
Called fromkvm_write_guest_offset_cached: if that needs change, this does too
*kvm_write_guest_page
Quoted 8 lines
Two interesting paths:
-kvm_pv_clock_pairingreturns a customKVM_EFAULTerror here (arch/x86/kvm/x86.c:9578)
-kvm_write_guest_offset_cachedreturns this directly (so if that needs change, this does too)
*kvm_read_guest_offset_cached
I actually do see a path to userspace, but it's through hyper-v, which we've said is out of scope for round 1.
To clarify: I didn't intend to make Hyper-V explicitly out-of-scope, rather Hyper-V happened to be out-of-scope because the existing code suppresses -EFAULT. I don't think we should make any particular feature/area out-of-scope, as that will lead to even more arbitrary behavior than we already have.
What I intended, and what I still think we should do, is limit the scope of the capability to existing paths that return -EFAULT to userspace. Trying to fix all of the paths that suppress -EFAULT is going to be ridiculously difficult as so much of the behavior is arguaby ABI, and there's no authoritative documentation on what's supposed to happen. I definitely would love to fix those paths in the long term, but for the initial implementation/conversion, I think it makes sense to punt on them, otherwise it'll take months/years to merge this code.
Back to the Hyper-V case, assuming you're referring to the use of kvm_hv_verify_vp_assist() in nested_svm_vmrun(), that code is a mess. KVM shouldn't inject a #GP and then exit to userspace, e.g. the guest might see a spurious #GP if userspace fixes the fault and resume the instruction. And just a few lines below, KVM skips the instruction if kvm_vcpu_map() returns -EFAULT.
As above, ideally that code would be converted to gracefully report the error, but it's such a snafu that the easiest thing might be to change the "return ret;" to "return 1;" until we fix all such KVM-on-HyperV code.