Support Asynchronous Page Fault
Subsystems (7)
v2 cover Support Asynchronous Page Fault
There are two stages of page fault. The guest kernel is responsible for handling stage-1 page fault, while the host kernel is to take care of the stage-2 page fault. When the guest is trapped to host because of stage-2 page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. Similarly, the guest has to wait until the requested memory is ready in the scenario of post-copy live migration.
This series introduces the feature (Asynchronous Page Fault) to improve situation, so that the guest needn't have to wait in the scenarios. With it, the overall performance is improved on the guest. This series depends on the feature "SDEI virtualization" and QEMU changes. All code changes can be found from github:
https://github.com/gwshan/linux ("sdei") # SDEI virtualization
https://github.com/gwshan/linux ("apf") # This series + "sdei"
https://github.com/gwshan/qemu ("apf") # QEMU code changes
About the design, the details can be found from last patch. Generally, it's driven by two notifications: page-not-present and page-ready. They are delivered from the host to guest via SDEI event and PPI separately. In the mean while, each notification is always associated with a token, used to identify the notification. The token is passed by the shared memory between host/guest. Besides, the SMCCC and ioctl interface are mitigated by VMM and guest to configure, enable, disable, even migrate the functionality.
When the guest is trapped to host because of stage-2 page fault, a page-not-present notification is raised by the host, and sent to the guest through dedicated SDEI event (0x40400001) if the requested page can't be populated immediately. In the mean while, a (background) worker is also started to populate the requested page. On receiving the SDEI event, the guest marks the current running process with special flag (TIF_ASYNC_PF) and associates it with a pre-allocated waitqueue. At same time, a (reschedule) IPI is sent to current CPU. After the SDEI event is acknowledged by the guest, the (reschedule) IPI is delivered and it causes context switch from that process tagged with TIF_ASYNC_PF to another process.
Later on, a page-ready notification is sent to guest after the requested page is populated by the (background) worker. On receiving the interrupt, the guest uses the associated token to locate the process, which was previously suspended because of page-not-present. The flag (TIF_ASYNC_PF) is cleared for the suspended process and it's waken up.
The series is organized as below:
PATCH[01-04] makes the GFN hash table management generic so that it
can be shared by x86/arm64.
PATCH[05-06] support KVM hypervisor SMCCC services from Will Deacon. PATCH[07-08] preparatory work to support asynchronous page fault. PATCH[09-10] support asynchronous page fault.
PATCH[11-13] support ioctl and SMCCC interfaces for the functionality. PATHC[14-16] supoort asynchronous page fault for guest
PATCH[17] adds document to explain the design and internals
Testing
=======
The tests are taken using program "testsuite mem", which is written by myself. The program basically does two things: (a) Starts a thread to allocate the specified percentage of available memory, write random values to them by the specified times, and then release them; (b) Thread is started to do calculation if needed.
In the mean while, the guest is always assigned with only one vCPU and 4096MB memory in all test cases. The memory cgroup where qemu process is associated could have different memory limitation settings.
[1] Allocate/free memory without calculation thread
Index -APF +APF Output
---------------------------------------------------------------------
1 39477ms 38367ms +0.28%
2 51272ms 49760ms +0.29%
The consumed time is decreased a bit and it should be benefited from the (background) workers, which runs in parallel.
[2] Allocate/free memory with calculation thread
Index -APF +APF Output
---------------------------------------------------------------------
1 81442ms 7155198892 171335ms 22391255613 -110% +213%
2 122002ms 11438214429 191126ms 24984499197 -57% +119%
The increased calculation amount is almost 2 times of the increased
time.[3] 5 Times of allocate/free memory with calculation thread and post-copy live migration
Index -APF +APF Output
---------------------------------------------------------------------
1 240635ms 19722999876 658955ms 89242030748 +174% +352%
Results retrieved from "info migrate":
Param -APF +APF Output
--------------------------------------------------------------------
store-global-state: on on
only-migratable: off off
send-configuration: on on
send-section-footer: on on
decompress-error-check: on on
clear-bitmap-shift: 18 18
Migration status: completed completed
total time: 9576 ms 10461 ms +9.2%
downtime: 78 ms 44 ms -43.5%
setup: 62 ms 47ms -24.1%
transferred ram: 889007 kbytes 1206436 kbytes +35.7%
throughput: 765.53 mbps 949.08 mbps +24%
remaining ram: 0 kbytes 0 kbytes
total ram: 4325952 kbytes 4325952 kbytes
duplicate: 861559 pages 823954 pages
skipped: 0 pages 0 pages
normal: 219929 pages 299214 pages
normal bytes: 879716 kbytes 1196856 kbytes
dirty sync count: 2 2
page size: 4 kbytes 4 kbytes
multifd bytes: 0 kbytes 0 kbytes
pages-per-second: 33684 72400 +115%
postcopy request count: 121 75 -38%
The asynchronous page fault is beneficial to throughput and speed in the scenario of post-copy live migration.
Chnagelog
=========
v2:
* Rebase to v5.11.rc6 (Gavin)
* Split the patches (James)
* Allocate "struct kvm_arch_async_control" dymaicall and use it to check if the feature has been enabled. The kernel
option (CONFIG_KVM_ASYNC_PF) isn't used. (James)
* Add document to explain the design (James) * Make GFN hash table management generic (James) * Add ioctl commands to support migration (Gavin)
Gavin Shan (15):
KVM: async_pf: Move struct kvm_async_pf around
KVM: async_pf: Add helper function to check completion queue
KVM: async_pf: Make GFN slot management generic
KVM: x86: Use generic async PF slot management
KVM: arm64: Export kvm_handle_user_mem_abort()
KVM: arm64: Add paravirtualization header files
KVM: arm64: Support page-not-present notification
KVM: arm64: Support page-ready notification
KVM: arm64: Support async PF hypercalls
KVM: arm64: Support async PF ioctl commands
KVM: arm64: Export async PF capability
arm64: Detect async PF para-virtualization feature
arm64: Reschedule process on aync PF
arm64: Enable async PF
KVM: arm64: Add async PF document
Will Deacon (2):
arm64: Probe for the presence of KVM hypervisor services during boot
KVM: arm64: Advertise KVM UID to guests via SMCCC
Documentation/virt/kvm/arm/apf.rst | 143 +++++++ Documentation/virt/kvm/arm/index.rst | 1 + arch/arm64/Kconfig | 11 + arch/arm64/include/asm/esr.h | 6 + arch/arm64/include/asm/hypervisor.h | 11 + arch/arm64/include/asm/kvm_emulate.h | 27 +- arch/arm64/include/asm/kvm_host.h | 85 ++++ arch/arm64/include/asm/kvm_para.h | 37 ++ arch/arm64/include/asm/processor.h | 1 + arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/include/uapi/asm/Kbuild | 2 - arch/arm64/include/uapi/asm/kvm.h | 19 + arch/arm64/include/uapi/asm/kvm_para.h | 23 ++ arch/arm64/include/uapi/asm/kvm_sdei.h | 1 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/kvm.c | 452 +++++++++++++++++++++ arch/arm64/kernel/setup.c | 32 ++ arch/arm64/kernel/signal.c | 17 + arch/arm64/kvm/Kconfig | 2 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/arm.c | 37 +- arch/arm64/kvm/async_pf.c | 535 +++++++++++++++++++++++++ arch/arm64/kvm/hypercalls.c | 33 +- arch/arm64/kvm/mmu.c | 75 +++- arch/arm64/kvm/sdei.c | 5 + arch/x86/include/asm/kvm_host.h | 1 - arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/x86.c | 88 +--- include/linux/arm-smccc.h | 40 ++ include/linux/kvm_host.h | 73 +++- include/uapi/linux/kvm.h | 3 + virt/kvm/Kconfig | 3 + virt/kvm/async_pf.c | 91 ++++- virt/kvm/kvm_main.c | 4 +- 35 files changed, 1706 insertions(+), 161 deletions(-) create mode 100644 Documentation/virt/kvm/arm/apf.rst create mode 100644 arch/arm64/include/asm/kvm_para.h create mode 100644 arch/arm64/include/uapi/asm/kvm_para.h create mode 100644 arch/arm64/kernel/kvm.c create mode 100644 arch/arm64/kvm/async_pf.c
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 1/17 KVM: async_pf: Move struct kvm_async_pf around
This moves the definitions of "struct kvm_async_pf" and the related functions after "struct kvm_vcpu" so that newly added inline function can dereference "struct kvm_vcpu" properly. Otherwise, the unexpected build error will be raised:
error: dereferencing pointer to incomplete type ‘struct kvm_vcpu’
return !list_empty_careful(&vcpu->async_pf.done);
^~The sepator between type and field is replaced by tab for "struct kvm_async_pf" since we're here. This is preparatory work for adding new inline function for next patch. This shouldn't cause logical changes.
--- include/linux/kvm_host.h | 43 ++++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 21 deletions(-)
include/linux/kvm_host.h +22 -21
@@ -196,27 +196,6 @@ void kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx,struct kvm_io_device *kvm_io_bus_get_dev (struct kvm *kvm,enum kvm_bus bus_idx,gpa_t addr); -#ifdef CONFIG_KVM_ASYNC_PF -struct kvm_async_pf { -struct work_struct work; -struct list_head link; -struct list_head queue; -struct kvm_vcpu *vcpu; -struct mm_struct *mm; -gpa_t cr2_or_gpa; -unsigned long addr; -struct kvm_arch_async_pf arch; -bool wakeup_all; -bool notpresent_injected; -}; - -void kvm_clear_async_pf_completion_queue (struct kvm_vcpu *vcpu); -void kvm_check_async_pf_completion (struct kvm_vcpu *vcpu); -bool kvm_setup_async_pf (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa, -unsigned long hva,struct kvm_arch_async_pf *arch); -int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu); -#endif -enum {OUTSIDE_GUEST_MODE ,IN_GUEST_MODE ,
@@ -323,6 +302,28 @@ struct kvm_vcpu {struct kvm_dirty_ring dirty_ring; }; +#ifdef CONFIG_KVM_ASYNC_PF +struct kvm_async_pf { +struct work_struct work; +struct list_head link; +struct list_head queue; +struct kvm_vcpu *vcpu; +struct mm_struct *mm; +gpa_t cr2_or_gpa; +unsigned long addr; +struct kvm_arch_async_pf arch; +bool wakeup_all; +bool notpresent_injected; +}; + +void kvm_clear_async_pf_completion_queue (struct kvm_vcpu *vcpu); +void kvm_check_async_pf_completion (struct kvm_vcpu *vcpu); +bool kvm_setup_async_pf (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa, +unsigned long hva,struct kvm_arch_async_pf *arch); +int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu); +#endif + +static inline int kvm_vcpu_exiting_guest_mode (struct kvm_vcpu *vcpu) {/*
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 2/17 KVM: async_pf: Add helper function to check completion queue
In reply to Gavin ShanThis adds inline function kvm_check_async_pf_completion_queue() and stub on !CONFIG_KVM_ASYNC_PF so that the source code won't have to care about CONFIG_KVM_ASYNC_PF. The kernel option is used for once in kvm_main.c and it can be removed then. Besides, the checks on the completion queue are all replaced by the newly introduced helper as list_empty() and list_empty_careful() are interchangeable.
The stub kvm_check_async_pf_completion() on !CONFIG_KVM_ASYNC_PF is also introduced. It will be used by subsequent patch.
--- arch/x86/kvm/x86.c | 2 +- include/linux/kvm_host.h | 12 ++++++++++++ virt/kvm/async_pf.c | 12 ++++++------ virt/kvm/kvm_main.c | 4 +--- 4 files changed, 20 insertions(+), 10 deletions(-)
arch/x86/kvm/x86.c +1 -1
@@ -10794,7 +10794,7 @@ static inline bool kvm_guest_apic_has_interrupt(struct kvm_vcpu *vcpu)static inline bool kvm_vcpu_has_events (struct kvm_vcpu *vcpu) { -if (!list_empty_careful (&vcpu->async_pf.done)) +if (kvm_check_async_pf_completion_queue (vcpu))return true;if (kvm_apic_has_events (vcpu))
include/linux/kvm_host.h +12 -0
@@ -316,11 +316,23 @@ struct kvm_async_pf { bool notpresent_injected; }; +static inline bool kvm_check_async_pf_completion_queue (struct kvm_vcpu *vcpu) +{ +return !list_empty_careful (&vcpu->async_pf.done); +} +void kvm_clear_async_pf_completion_queue (struct kvm_vcpu *vcpu);void kvm_check_async_pf_completion (struct kvm_vcpu *vcpu);bool kvm_setup_async_pf (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa,unsigned long hva,struct kvm_arch_async_pf *arch);int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu); +#else +static inline bool kvm_check_async_pf_completion_queue (struct kvm_vcpu *vcpu) +{ +return false; +} + +static inline void kvm_check_async_pf_completion (struct kvm_vcpu *vcpu) { }#endif
virt/kvm/async_pf.c +6 -6
@@ -70,7 +70,7 @@ static void async_pf_execute(struct work_struct *work)kvm_arch_async_page_present (vcpu, apf);spin_lock (&vcpu->async_pf.lock); - first =list_empty (&vcpu->async_pf.done); + first = !kvm_check_async_pf_completion_queue (vcpu);list_add_tail (&apf->link, &vcpu->async_pf.done); apf->vcpu =NULL ;spin_unlock (&vcpu->async_pf.lock);
@@ -122,7 +122,7 @@ void kvm_clear_async_pf_completion_queue(struct kvm_vcpu *vcpu)spin_lock (&vcpu->async_pf.lock); } -while (!list_empty (&vcpu->async_pf.done)) { +while (kvm_check_async_pf_completion_queue (vcpu)) {struct kvm_async_pf *work =list_first_entry (&vcpu->async_pf.done,typeof (*work), link);
@@ -138,8 +138,8 @@ void kvm_check_async_pf_completion(struct kvm_vcpu *vcpu) {struct kvm_async_pf *work; -while (!list_empty_careful (&vcpu->async_pf.done) && -kvm_arch_can_dequeue_async_page_present (vcpu)) { +while (kvm_check_async_pf_completion_queue (vcpu) && +kvm_arch_can_dequeue_async_page_present (vcpu)) {spin_lock (&vcpu->async_pf.lock); work =list_first_entry (&vcpu->async_pf.done,typeof (*work), link);
@@ -205,7 +205,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu)struct kvm_async_pf *work;bool first; -if (!list_empty_careful (&vcpu->async_pf.done)) +if (kvm_check_async_pf_completion_queue (vcpu))return 0 ; work =kmem_cache_zalloc (async_pf_cache,GFP_ATOMIC );
@@ -216,7 +216,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu)INIT_LIST_HEAD (&work->queue);/* for list_del to work */ spin_lock (&vcpu->async_pf.lock); - first =list_empty (&vcpu->async_pf.done); + first = !kvm_check_async_pf_completion_queue (vcpu);list_add_tail (&work->link, &vcpu->async_pf.done);spin_unlock (&vcpu->async_pf.lock);
virt/kvm/kvm_main.c +1 -3
@@ -2961,10 +2961,8 @@ static bool vcpu_dy_runnable(struct kvm_vcpu *vcpu)if (kvm_arch_dy_runnable (vcpu))return true; -#ifdef CONFIG_KVM_ASYNC_PF -if (!list_empty_careful (&vcpu->async_pf.done)) +if (kvm_check_async_pf_completion_queue (vcpu))return true; -#endif return false; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 3/17 KVM: async_pf: Make GFN slot management generic
In reply to Gavin ShanIt's not allowed to fire duplicate notification for same GFN on x86 platform, with help of a hash table. This mechanism is going to be used by arm64 and this makes the code generic and shareable by multiple platforms.
* As this mechanism isn't needed by all platforms, a new kernel config option (CONFIG_ASYNC_PF_SLOT) is introduced so that it can be disabled at compiling time.
* The code is basically copied from x86 platform and the functions are renamed to reflect the fact: (a) the input parameters are vCPU and GFN. (b) The operations are resetting, searching, adding and removing.
* Helper stub is also added on !CONFIG_KVM_ASYNC_PF because we're going to use IS_ENABLED() instead of #ifdef on arm64 when the asynchronous page fault is supported.
This is preparatory work to use the newly introduced functions on x86 platform and arm64 in subsequent patches.
--- include/linux/kvm_host.h | 18 +++++++++ virt/kvm/Kconfig | 3 ++ virt/kvm/async_pf.c | 79 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 100 insertions(+)
include/linux/kvm_host.h +18 -0
@@ -277,6 +277,9 @@ struct kvm_vcpu { #ifdef CONFIG_KVM_ASYNC_PF struct { +#ifdef CONFIG_KVM_ASYNC_PF_SLOT + gfn_t gfns[ASYNC_PF_PER_VCPU]; +#endif u32 queued; struct list_head queue; struct list_head done;
@@ -321,12 +324,27 @@ static inline bool kvm_check_async_pf_completion_queue(struct kvm_vcpu *vcpu)return !list_empty_careful (&vcpu->async_pf.done); } +#ifdef CONFIG_KVM_ASYNC_PF_SLOT +void kvm_async_pf_reset_slot (struct kvm_vcpu *vcpu); +void kvm_async_pf_add_slot (struct kvm_vcpu *vcpu,gfn_t gfn); +void kvm_async_pf_remove_slot (struct kvm_vcpu *vcpu,gfn_t gfn); +bool kvm_async_pf_find_slot (struct kvm_vcpu *vcpu,gfn_t gfn); +#endif +void kvm_clear_async_pf_completion_queue (struct kvm_vcpu *vcpu);void kvm_check_async_pf_completion (struct kvm_vcpu *vcpu);bool kvm_setup_async_pf (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa,unsigned long hva,struct kvm_arch_async_pf *arch);int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#else +static inline void kvm_async_pf_reset_slot (struct kvm_vcpu *vcpu) { } +static inline void kvm_async_pf_add_slot (struct kvm_vcpu *vcpu,gfn_t gfn) { } +static inline void kvm_async_pf_remove_slot (struct kvm_vcpu *vcpu,gfn_t gfn) { } +static inline bool kvm_async_pf_find_slot (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +return false; +} +static inline bool kvm_check_async_pf_completion_queue (struct kvm_vcpu *vcpu) {return false;
virt/kvm/Kconfig +3 -0
@@ -23,6 +23,9 @@ config KVM_MMIO config KVM_ASYNC_PF bool +config KVM_ASYNC_PF_SLOT + bool + # Toggle to switch between direct notification and batch job config KVM_ASYNC_PF_SYNC bool
virt/kvm/async_pf.c +79 -0
85 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 4/17 KVM: x86: Use generic async PF slot management
In reply to Gavin ShanThis uses the generic slot management mechanism for asynchronous page fault by enabling CONFIG_KVM_ASYNC_PF_SLOT because the private implementation is totally duplicate to the generic one.
The changes introduced by this is pretty mechanical and shouldn't cause any logical changes.
--- arch/x86/include/asm/kvm_host.h | 1 - arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/x86.c | 86 +++------------------------------ 4 files changed, 8 insertions(+), 82 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
@@ -1714,7 +1714,6 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work);void kvm_arch_async_page_present_queued (struct kvm_vcpu *vcpu);bool kvm_arch_can_dequeue_async_page_present (struct kvm_vcpu *vcpu); -extern bool kvm_find_async_pf_gfn (struct kvm_vcpu *vcpu,gfn_t gfn);int kvm_skip_emulated_instruction (struct kvm_vcpu *vcpu);int kvm_complete_insn_gp (struct kvm_vcpu *vcpu,int err);
arch/x86/kvm/Kconfig +1 -0
@@ -34,6 +34,7 @@ config KVM select HAVE_KVM_IRQ_ROUTING select HAVE_KVM_EVENTFD select KVM_ASYNC_PF + select KVM_ASYNC_PF_SLOT select USER_RETURN_NOTIFIER select KVM_MMIO select TASKSTATS
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -3678,7 +3678,7 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,if (!prefault &&kvm_can_do_async_pf (vcpu)) {trace_kvm_try_async_get_page (cr2_or_gpa, gfn); -if (kvm_find_async_pf_gfn (vcpu, gfn)) { +if (kvm_async_pf_find_slot (vcpu, gfn)) {trace_kvm_async_pf_doublefault (cr2_or_gpa, gfn);kvm_make_request (KVM_REQ_APF_HALT , vcpu);return true;
arch/x86/kvm/x86.c +6 -80
134 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 5/17 arm64: Probe for the presence of KVM hypervisor services during boot
In reply to Gavin ShanFrom: Will Deacon <[email protected]>
Although the SMCCC specification provides some limited functionality for describing the presence of hypervisor and firmware services, this is generally applicable only to functions designated as "Arm Architecture Service Functions" and no portable discovery mechanism is provided for standard hypervisor services, despite having a designated range of function identifiers reserved by the specification.
In an attempt to avoid the need for additional firmware changes every time a new function is added, introduce a UID to identify the service provider as being compatible with KVM. Once this has been established, additional services can be discovered via a feature bitmap.
--- arch/arm64/include/asm/hypervisor.h | 11 ++++++++++ arch/arm64/kernel/setup.c | 32 +++++++++++++++++++++++++++++ include/linux/arm-smccc.h | 25 ++++++++++++++++++++++ 3 files changed, 68 insertions(+)
arch/arm64/include/asm/hypervisor.h +11 -0
@@ -2,6 +2,17 @@#ifndef _ASM_ARM64_HYPERVISOR_H#define _ASM_ARM64_HYPERVISOR_H +#include <linux/arm-smccc.h> #include <asm/xen/hypervisor.h> +static inline bool kvm_arm_hyp_service_available (u32 func_id) +{ + extern DECLARE_BITMAP(__kvm_arm_hyp_services, ARM_SMCCC_KVM_NUM_FUNCS); + +if (func_id >=ARM_SMCCC_KVM_NUM_FUNCS ) +return -EINVAL ; + +return test_bit (func_id, __kvm_arm_hyp_services); +} +#endif
arch/arm64/kernel/setup.c +32 -0
@@ -7,6 +7,7 @@*/ #include <linux/acpi.h> +#include <linux/arm-smccc.h> #include <linux/export.h> #include <linux/kernel.h> #include <linux/stddef.h>
@@ -275,12 +276,42 @@ static int __init reserve_memblock_reserved_regions(void)arch_initcall (reserve_memblock_reserved_regions);u64 __cpu_logical_map[NR_CPUS ] = { [0 ...NR_CPUS -1 ] =INVALID_HWID }; +DECLARE_BITMAP(__kvm_arm_hyp_services, ARM_SMCCC_KVM_NUM_FUNCS) = { };u64 cpu_logical_map (unsigned int cpu) {return __cpu_logical_map[cpu]; } +static void __init kvm_init_hyp_services (void ) +{ +struct arm_smccc_res res; +int i; + +arm_smccc_1_1_invoke (ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID , &res); +if (res.a0 !=ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0 || + res.a1 !=ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1 || + res.a2 !=ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 || + res.a3 !=ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3 ) +return ; + +memset (&res,0 ,sizeof (res)); +arm_smccc_1_1_invoke (ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID , &res); +for (i =0 ; i <32 ; ++i) { +if (res.a0 & (i)) +set_bit (i + (32 *0 ), __kvm_arm_hyp_services); +if (res.a1 & (i)) +set_bit (i + (32 *1 ), __kvm_arm_hyp_services); +if (res.a2 & (i)) +set_bit (i + (32 *2 ), __kvm_arm_hyp_services); +if (res.a3 & (i)) +set_bit (i + (32 *3 ), __kvm_arm_hyp_services); + } + +pr_info ("KVM hypervisor services detected (0x%08lx 0x%08lx 0x%08lx 0x%08lx)\n" , + res.a3, res.a2, res.a1, res.a0); +} + void __init __no_sanitize_address setup_arch(char **cmdline_p) { init_mm.start_code = (unsigned long ) _stext;
@@ -353,6 +384,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)else psci_acpi_init (); +kvm_init_hyp_services ();init_bootcpu_ops ();smp_init_cpus ();smp_build_mpidr_hash ();
include/linux/arm-smccc.h +25 -0
@@ -55,6 +55,8 @@#define ARM_SMCCC_OWNER_TRUSTED_OS 50#define ARM_SMCCC_OWNER_TRUSTED_OS_END 63 +#define ARM_SMCCC_FUNC_QUERY_CALL_UID 0xff01 +#define ARM_SMCCC_QUIRK_NONE 0#define ARM_SMCCC_QUIRK_QCOM_A6 1/* Save/restore register a6 */
@@ -102,6 +104,29 @@ ARM_SMCCC_OWNER_STANDARD_HYP, \ 0x21) +#define ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + ARM_SMCCC_OWNER_VENDOR_HYP, \ + ARM_SMCCC_FUNC_QUERY_CALL_UID) + +/* KVM UID value: 28b46fb6-2ec5-11e9-a9ca-4b564d003a74 */ +#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0 0xb66fb428U +#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1 0xe911c52eU +#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 0x564bcaa9U +#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3 0x743a004dU + +/* KVM "vendor specific" services */ +#define ARM_SMCCC_KVM_FUNC_FEATURES 0 +#define ARM_SMCCC_KVM_FUNC_FEATURES_2 127 +#define ARM_SMCCC_KVM_NUM_FUNCS 128 + +#define ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + ARM_SMCCC_OWNER_VENDOR_HYP, \ + ARM_SMCCC_KVM_FUNC_FEATURES) +/* * Return codes defined in ARM DEN 0070A * ARM DEN 0070A is now merged/consolidated into ARM DEN 0028 C
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 6/17 KVM: arm64: Advertise KVM UID to guests via SMCCC
In reply to Gavin ShanFrom: Will Deacon <[email protected]>
We can advertise ourselves to guests as KVM and provide a basic features bitmap for discoverability of future hypervisor services.
--- arch/arm64/kvm/hypercalls.c | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-)
arch/arm64/kvm/hypercalls.c +18 -9
@@ -12,13 +12,13 @@int kvm_hvc_call_handler (struct kvm_vcpu *vcpu) {u32 func_id =smccc_get_function (vcpu); -long val =SMCCC_RET_NOT_SUPPORTED ; +long val[4 ] = {SMCCC_RET_NOT_SUPPORTED };u32 feature;gpa_t gpa;switch (func_id) {case ARM_SMCCC_VERSION_FUNC_ID : - val =ARM_SMCCC_VERSION_1_1 ; + val[0 ] =ARM_SMCCC_VERSION_1_1 ;break ;case ARM_SMCCC_ARCH_FEATURES_FUNC_ID : feature =smccc_get_arg1 (vcpu);
@@ -28,10 +28,10 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case SPECTRE_VULNERABLE :break ;case SPECTRE_MITIGATED : - val =SMCCC_RET_SUCCESS ; + val[0 ] =SMCCC_RET_SUCCESS ;break ;case SPECTRE_UNAFFECTED : - val =SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED ; + val[0 ] =SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED ;break ; }break ;
@@ -54,22 +54,31 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)break ; fallthrough;case SPECTRE_UNAFFECTED : - val =SMCCC_RET_NOT_REQUIRED ; + val[0 ] =SMCCC_RET_NOT_REQUIRED ;break ; }break ;case ARM_SMCCC_HV_PV_TIME_FEATURES : - val =SMCCC_RET_SUCCESS ; + val[0 ] =SMCCC_RET_SUCCESS ;break ; }break ;case ARM_SMCCC_HV_PV_TIME_FEATURES : - val =kvm_hypercall_pv_features (vcpu); + val[0 ] =kvm_hypercall_pv_features (vcpu);break ;case ARM_SMCCC_HV_PV_TIME_ST : gpa =kvm_init_stolen_time (vcpu);if (gpa !=GPA_INVALID ) - val = gpa; + val[0 ] = gpa; +break ; +case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID : + val[0 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0 ; + val[1 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1 ; + val[2 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 ; + val[3 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3 ; +break ; +case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID : + val[0 ] =BIT (ARM_SMCCC_KVM_FUNC_FEATURES );break ;case SDEI_1_0_FN_SDEI_VERSION :case SDEI_1_0_FN_SDEI_EVENT_REGISTER :
@@ -93,6 +102,6 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)return kvm_psci_call (vcpu); } -smccc_set_retval (vcpu, val,0 ,0 ,0 ); +smccc_set_retval (vcpu, val[0 ], val[1 ], val[2 ], val[3 ]);return 1 ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 7/17 KVM: arm64: Export kvm_handle_user_mem_abort()
In reply to Gavin ShanThe main work is handled by user_mem_abort(). After asynchronous page fault is supported, one page fault need to be handled with two calls to this function. It means the page fault needs to be replayed asynchronously in that case. This renames the function to kvm_handle_user_mem_abort() can exports it. Besides, there are more changes introduced in order to accommodate asynchronous page fault:
* Add arguments @esr and @prefault to user_mem_abort(). @esr is the cached value of ESR_EL2 instead of fetching from the current vCPU when the page fault is replayed in scenario of asynchronous page fault. @prefault is used to indicate the page fault is replayed one or not.
* Define helper functions esr_dbat_*() in asm/esr.h to extract or check various fields of the passed ESR_EL2 value because those helper functions defined in asm/kvm_emulate.h assumes the ESR_EL2 value has been cached in vCPU struct. It won't be true on handling the replayed page fault in scenario of asynchronous page fault.
* Some helper functions defined in asm/kvm_emulate.h are used by mmu.c only and seem not to be used by other source file in near future. They are moved to mmu.c and renamed accordingly.
kvm_vcpu_trap_is_exec_fault()
is_exec_fault()
kvm_is_write_fault()
is_write_fault()
kvm_vcpu_trap_get_fault_level()
Replaced by esr_dabt_get_fault_level()
--- arch/arm64/include/asm/esr.h | 6 ++++ arch/arm64/include/asm/kvm_emulate.h | 27 ++--------------- arch/arm64/include/asm/kvm_host.h | 4 +++ arch/arm64/kvm/mmu.c | 43 ++++++++++++++++++++++------ 4 files changed, 48 insertions(+), 32 deletions(-)
arch/arm64/include/asm/esr.h +6 -0
@@ -321,8 +321,14 @@ ESR_ELx_CP15_32_ISS_DIR_READ)#ifndef __ASSEMBLY__ +#include <linux/bitfield.h> #include <asm/types.h> +#define esr_dabt_get_fault_type (esr) (esr & ESR_ELx_FSC_TYPE) +#define esr_dabt_get_fault_level (esr) (FIELD_GET(ESR_ELx_FSC_LEVEL, esr)) +#define esr_dabt_is_wnr (esr) (!!(FIELD_GET(ESR_ELx_WNR, esr))) +#define esr_dabt_is_s1ptw (esr) (!!(FIELD_GET(ESR_ELx_S1PTW, esr))) +static inline bool esr_is_data_abort (u32 esr) {const u32 ec =ESR_ELx_EC (esr);
arch/arm64/include/asm/kvm_emulate.h +3 -24
@@ -282,13 +282,13 @@ static __always_inline int kvm_vcpu_dabt_get_rd(const struct kvm_vcpu *vcpu) static __always_inline bool kvm_vcpu_abt_iss1tw(const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_S1PTW); +return esr_dabt_is_s1ptw (kvm_vcpu_get_esr (vcpu)); }/* Always check for S1PTW *before* using this. */ static __always_inline bool kvm_vcpu_dabt_iswrite(const struct kvm_vcpu *vcpu) { -return kvm_vcpu_get_esr (vcpu) & ESR_ELx_WNR; +return esr_dabt_is_wnr (kvm_vcpu_get_esr (vcpu)); }static inline bool kvm_vcpu_dabt_is_cm (const struct kvm_vcpu *vcpu)
@@ -317,11 +317,6 @@ static inline bool kvm_vcpu_trap_is_iabt(const struct kvm_vcpu *vcpu)return kvm_vcpu_trap_get_class (vcpu) == ESR_ELx_EC_IABT_LOW; } -static inline bool kvm_vcpu_trap_is_exec_fault (const struct kvm_vcpu *vcpu) -{ -return kvm_vcpu_trap_is_iabt (vcpu) && !kvm_vcpu_abt_iss1tw (vcpu); -} - static __always_inline u8 kvm_vcpu_trap_get_fault(const struct kvm_vcpu *vcpu) {return kvm_vcpu_get_esr (vcpu) & ESR_ELx_FSC;
@@ -329,12 +324,7 @@ static __always_inline u8 kvm_vcpu_trap_get_fault(const struct kvm_vcpu *vcpu) static __always_inline u8 kvm_vcpu_trap_get_fault_type(const struct kvm_vcpu *vcpu) { -return kvm_vcpu_get_esr (vcpu) & ESR_ELx_FSC_TYPE; -} - -static __always_inline u8 kvm_vcpu_trap_get_fault_level(const struct kvm_vcpu *vcpu) -{ -return kvm_vcpu_get_esr (vcpu) & ESR_ELx_FSC_LEVEL; +return esr_dabt_get_fault_type (kvm_vcpu_get_esr (vcpu)); } static __always_inline bool kvm_vcpu_abt_issea(const struct kvm_vcpu *vcpu)
@@ -362,17 +352,6 @@ static __always_inline int kvm_vcpu_sys_get_rt(struct kvm_vcpu *vcpu)return ESR_ELx_SYS64_ISS_RT (esr); } -static inline bool kvm_is_write_fault (struct kvm_vcpu *vcpu) -{ -if (kvm_vcpu_abt_iss1tw (vcpu)) -return true; - -if (kvm_vcpu_trap_is_iabt (vcpu)) -return false; - -return kvm_vcpu_dabt_iswrite (vcpu); -} -static inline unsigned long kvm_vcpu_get_mpidr_aff (struct kvm_vcpu *vcpu) {return vcpu_read_sys_reg (vcpu,MPIDR_EL1 ) &MPIDR_HWID_BITMASK ;
arch/arm64/include/asm/kvm_host.h +4 -0
@@ -589,6 +589,10 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu, struct kvm_vcpu_events *events);#define KVM_ARCH_WANT_MMU_NOTIFIER +int kvm_handle_user_mem_abort (struct kvm_vcpu *vcpu, +struct kvm_memory_slot *memslot, +phys_addr_t fault_ipa,unsigned long hva, +unsigned int esr,bool prefault);int kvm_unmap_hva_range (struct kvm *kvm,unsigned long start,unsigned long end,unsigned flags);int kvm_set_spte_hva (struct kvm *kvm,unsigned long hva,pte_t pte);
arch/arm64/kvm/mmu.c +35 -8
72 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 8/17 KVM: arm64: Add paravirtualization header files
In reply to Gavin ShanWe need put more stuff in the paravirtualization header files when the asynchronous page fault is supported. The generic header files can't meet the goal. This duplicate the generic header files to be our platform specific header files. It's the preparatory work to support the asynchronous page fault in subsequent patches:
include/uapi/asm-generic/kvm_para.h
include/asm-generic/kvm_para.h
arch/arm64/include/uapi/asm/kvm_para.h
arch/arm64/include/asm/kvm_para.h
--- arch/arm64/include/asm/kvm_para.h | 27 ++++++++++++++++++++++++++ arch/arm64/include/uapi/asm/Kbuild | 2 -- arch/arm64/include/uapi/asm/kvm_para.h | 5 +++++ 3 files changed, 32 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_para.h create mode 100644 arch/arm64/include/uapi/asm/kvm_para.h
arch/arm64/include/asm/kvm_para.h new file +27 -0
@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM_KVM_PARA_H +#define _ASM_ARM_KVM_PARA_H + +#include <uapi/asm/kvm_para.h> + +static inline bool kvm_check_and_clear_guest_paused (void ) +{ +return false; +} + +static inline unsigned int kvm_arch_para_features (void ) +{ +return 0 ; +} + +static inline unsigned int kvm_arch_para_hints (void ) +{ +return 0 ; +} + +static inline bool kvm_para_available (void ) +{ +return false; +} + +#endif /* _ASM_ARM_KVM_PARA_H */
arch/arm64/include/uapi/asm/Kbuild +0 -2
@@ -1,3 +1 @@ # SPDX-License-Identifier: GPL-2.0 - -generic-y += kvm_para.h
arch/arm64/include/uapi/asm/kvm_para.h new file +5 -0
@@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _UAPI_ASM_ARM_KVM_PARA_H +#define _UAPI_ASM_ARM_KVM_PARA_H + +#endif /* _UAPI_ASM_ARM_KVM_PARA_H */
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 9/17 KVM: arm64: Support page-not-present notification
In reply to Gavin ShanThe requested page might be not resident in memory during the stage-2 page fault. For example, the requested page could be resident in swap device (file). In this case, disk I/O is issued in order to fetch the requested page and it could take tens of milliseconds, even hundreds of milliseconds in extreme situation. During the period, the guest's vCPU is suspended until the requested page becomes ready. Actually, the something else on the guest's vCPU could be rescheduled during the period, so that the time slice isn't wasted as the guest's vCPU can see. This is the primary goal of the feature (Asynchronous Page Fault).
This supports delivery of page-not-present notification through SDEI event when the requested page isn't present. When the notification is received on the guest's vCPU, something else (another process) can be scheduled. The design is highlighted as below:
* There is dedicated memory region shared by host and guest. It's represented by "struct kvm_vcpu_pv_apf_data". The field @reason indicates the reason why the SDEI event is triggered, while the unique @token is used by guest to associate the event with the suspended process.
* One control block is associated with each guest's vCPU and it's represented by "struct kvm_arch_async_pf_control". It allows the guest to configure the functionality to indicate the situations where the host can deliver the page-not-present notification to kick off asyncrhonous page fault. Besides, runtime states are also maintained in this struct.
* Before the page-not-present notification is sent to the guest's vCPU, a worker is started and executed asynchronously on host, to fetch the requested page. "struct kvm{_,_arch}async_pf" is associated with the worker, to track the work.
The feature isn't enabled by CONFIG_KVM_ASYNC_PF yet. Also, the page-ready notification delivery and control path isn't implemented and will be done in the subsequent patches.
--- arch/arm64/include/asm/kvm_host.h | 50 +++++++++ arch/arm64/include/uapi/asm/kvm_para.h | 15 +++ arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/arm.c | 3 + arch/arm64/kvm/async_pf.c | 145 +++++++++++++++++++++++++ arch/arm64/kvm/mmu.c | 32 +++++- 6 files changed, 245 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/async_pf.c
arch/arm64/include/asm/kvm_host.h +50 -0
@@ -273,6 +273,31 @@ struct vcpu_reset_state {bool reset; }; +/* Should be a power of two number */ +#define ASYNC_PF_PER_VCPU 64 + +/* +* The association of gfn and token. The token will be sent to guest as +* page fault address. Also, the guest could be in aarch32 mode. So its +* length should be 32-bits. +*/ +struct kvm_arch_async_pf { +u32 token; +gfn_t gfn; +u32 esr; +}; + +struct kvm_arch_async_pf_control { +struct gfn_to_hva_cache cache; +u64 control_block; +bool send_user_only; +u64 sdei_event_num; + +u16 id; +bool notpresent_pending; +u32 notpresent_token; +}; +struct kvm_vcpu_arch {struct kvm_cpu_context ctxt;void *sve_state;
@@ -375,6 +400,7 @@ struct kvm_vcpu_arch { } steal;struct kvm_sdei_vcpu *sdei; +struct kvm_arch_async_pf_control *apf; };/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
@@ -734,6 +760,30 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,int kvm_arm_vcpu_arch_has_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); +#ifdef CONFIG_KVM_ASYNC_PF +void kvm_arch_async_pf_create_vcpu (struct kvm_vcpu *vcpu); +bool kvm_arch_async_not_present_allowed (struct kvm_vcpu *vcpu); +bool kvm_arch_setup_async_pf (struct kvm_vcpu *vcpu, +u32 esr,gpa_t gpa,gfn_t gfn); +bool kvm_arch_async_page_not_present (struct kvm_vcpu *vcpu, +struct kvm_async_pf *work); +void kvm_arch_async_pf_destroy_vcpu (struct kvm_vcpu *vcpu); +#else +static inline void kvm_arch_async_pf_create_vcpu (struct kvm_vcpu *vcpu) { } +static inline void kvm_arch_async_pf_destroy_vcpu (struct kvm_vcpu *vcpu) { } + +static inline bool kvm_arch_async_not_present_allowed (struct kvm_vcpu *vcpu) +{ +return false; +} + +static inline bool kvm_arch_setup_async_pf (struct kvm_vcpu *vcpu, +u32 esr,gpa_t gpa,gfn_t gfn) +{ +return false; +} +#endif +/* Guest/host FPSIMD coordination helpers */ int kvm_arch_vcpu_run_map_fp (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_load_fp (struct kvm_vcpu *vcpu);
arch/arm64/include/uapi/asm/kvm_para.h +15 -0
@@ -2,4 +2,19 @@#ifndef _UAPI_ASM_ARM_KVM_PARA_H#define _UAPI_ASM_ARM_KVM_PARA_H +#include <linux/types.h> + +/* Async PF */ +#define KVM_ASYNC_PF_ENABLED (1 << 0) +#define KVM_ASYNC_PF_SEND_ALWAYS (1 << 1) + +#define KVM_PV_REASON_PAGE_NOT_PRESENT 1 + +struct kvm_vcpu_pv_apf_data { +__u32 reason; +__u32 token; +__u8 pad[56 ]; +__u32 enabled; +}; +#endif /* _UAPI_ASM_ARM_KVM_PARA_H */
arch/arm64/kvm/Makefile +1 -0
@@ -25,3 +25,4 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ vgic/vgic-its.o vgic/vgic-debug.o kvm-$(CONFIG_HW_PERF_EVENTS) += pmu-emul.o +kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o async_pf.o
arch/arm64/kvm/arm.c +3 -0
@@ -330,6 +330,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)kvm_sdei_create_vcpu (vcpu); +kvm_arch_async_pf_create_vcpu (vcpu); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; err =kvm_vgic_vcpu_init (vcpu);
@@ -351,6 +353,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_cache);kvm_timer_vcpu_terminate (vcpu);kvm_pmu_vcpu_destroy (vcpu); +kvm_arch_async_pf_destroy_vcpu (vcpu);kvm_sdei_destroy_vcpu (vcpu);kvm_arm_vcpu_destroy (vcpu);
arch/arm64/kvm/async_pf.c new file +145 -0
145 lines not on this page · raw message
arch/arm64/kvm/mmu.c +31 -1
@@ -760,6 +760,33 @@ static inline bool is_write_fault(unsigned int esr)return esr_dabt_is_wnr (esr); } +static bool try_async_pf (struct kvm_vcpu *vcpu,u32 esr,gpa_t gpa, +gfn_t gfn,kvm_pfn_t *pfn,bool write, +bool *writable,bool prefault) +{ +struct kvm_arch_async_pf_control *apf = vcpu->arch.apf; +struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); +bool async = false; + +if (apf) { +/* Bail if *pfn has correct page */ + *pfn =__gfn_to_pfn_memslot (slot, gfn, false, + &async, write, writable); +if (!async) +return false; + +if (!prefault &&kvm_arch_async_not_present_allowed (vcpu)) { +if (kvm_async_pf_find_slot (vcpu, gfn) || +kvm_arch_setup_async_pf (vcpu, esr, gpa, gfn)) +return true; + } + } + + *pfn =__gfn_to_pfn_memslot (slot, gfn, false, +NULL , write, writable); +return false; +} + int kvm_handle_user_mem_abort(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, phys_addr_t fault_ipa,
@@ -871,7 +898,10 @@ int kvm_handle_user_mem_abort(struct kvm_vcpu *vcpu,*/ smp_rmb (); - pfn =gfn_to_pfn_prot (kvm, gfn, write_fault, &writable); +if (try_async_pf (vcpu, esr, fault_ipa, gfn, &pfn, + write_fault, &writable, prefault)) +return 1 ; +if (pfn ==KVM_PFN_ERR_HWPOISON ) {kvm_send_hwpoison_signal (hva, vma_shift);return 0 ;
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 10/17 KVM: arm64: Support page-ready notification
In reply to Gavin ShanThe asynchronous page fault starts with a worker when the requested page isn't present. The worker makes the requested page present in the background and the worker, together with the associated information, is queued to the completion queue after that. The worker and the completion queue are checked as below.
* A request (KVM_REQ_ASYNC_PF) is raised if the worker is the first one enqueued to the completion queue. With the request, the completion queue is checked and the worker is dequeued. A PPI is sent to guest as the page-ready notification and the guest should acknowledge the interrupt by SMCCC interface.
* When the notification (PPI) is acknowledged by guest, the completion queue is checked again and next worker is dequeued if we have one. For this particular worker, another notification (PPI) is sent to the guest without raising the request. Once the notification (PPI) is acknowledged by the guest, the completion queue is checked to process next worker, which has been queued to it.
Similar to page-not-present notification, the shared memory region is used to convey the reason and token associated with the page-ready notification. The region is represented by "struct kvm_vcpu_pv_apf_data".
The feature isn't enabled by CONFIG_KVM_ASYNC_PF yet. Also, the control path isn't implemented and will be done in the subsequent patches.
--- arch/arm64/include/asm/kvm_host.h | 17 ++ arch/arm64/include/uapi/asm/kvm_para.h | 1 + arch/arm64/kvm/arm.c | 24 ++- arch/arm64/kvm/async_pf.c | 207 +++++++++++++++++++++++++ arch/arm64/kvm/hypercalls.c | 6 + include/linux/arm-smccc.h | 10 ++ 6 files changed, 262 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +17 -0
@@ -48,6 +48,7 @@#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4)#define KVM_REQ_SDEI KVM_ARCH_REQ(5) +#define KVM_REQ_ASYNC_PF KVM_ARCH_REQ(6)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)
@@ -292,10 +293,12 @@ struct kvm_arch_async_pf_control {u64 control_block;bool send_user_only;u64 sdei_event_num; +u32 irq;u16 id;bool notpresent_pending;u32 notpresent_token; +bool pageready_pending; };struct kvm_vcpu_arch {
@@ -767,6 +770,14 @@ bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, u32 esr, gpa_t gpa, gfn_t gfn);bool kvm_arch_async_page_not_present (struct kvm_vcpu *vcpu,struct kvm_async_pf *work); +void kvm_arch_async_page_present_queued (struct kvm_vcpu *vcpu); +bool kvm_arch_can_dequeue_async_page_present (struct kvm_vcpu *vcpu); +void kvm_arch_async_page_ready (struct kvm_vcpu *vcpu, +struct kvm_async_pf *work); +void kvm_arch_async_page_present (struct kvm_vcpu *vcpu, +struct kvm_async_pf *work); +long kvm_arch_async_pf_hypercall (struct kvm_vcpu *vcpu, +long *r1,long *r2,long *r3);void kvm_arch_async_pf_destroy_vcpu (struct kvm_vcpu *vcpu);#else static inline void kvm_arch_async_pf_create_vcpu (struct kvm_vcpu *vcpu) { }
@@ -782,6 +793,12 @@ static inline bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, {return false; } + +static inline long kvm_arch_async_pf_hypercall (struct kvm_vcpu *vcpu, +long *r1,long *r2,long *r3) +{ +return SMCCC_RET_NOT_SUPPORTED ; +}#endif /* Guest/host FPSIMD coordination helpers */
arch/arm64/include/uapi/asm/kvm_para.h +1 -0
@@ -9,6 +9,7 @@#define KVM_ASYNC_PF_SEND_ALWAYS (1 << 1)#define KVM_PV_REASON_PAGE_NOT_PRESENT 1 +#define KVM_PV_REASON_PAGE_READY 2struct kvm_vcpu_pv_apf_data {__u32 reason;
arch/arm64/kvm/arm.c +21 -3
@@ -484,9 +484,23 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,*/ int kvm_arch_vcpu_runnable (struct kvm_vcpu *v) { +struct kvm_arch_async_pf_control *apf = v->arch.apf;bool irq_lines = *vcpu_hcr (v) & (HCR_VI |HCR_VF ); -return ((irq_lines ||kvm_vgic_vcpu_pending_irq (v)) - && !v->arch.power_off && !v->arch.pause); + +if ((irq_lines ||kvm_vgic_vcpu_pending_irq (v)) && + !v->arch.power_off && !v->arch.pause) +return true; + +if (apf && (apf->control_block &KVM_ASYNC_PF_ENABLED )) { +if (kvm_check_async_pf_completion_queue (v)) +return true; + +if (apf->notpresent_pending || + apf->pageready_pending) +return true; + } + +return false; }bool kvm_arch_vcpu_in_kernel (struct kvm_vcpu *vcpu)
@@ -674,6 +688,9 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_VCPU_RESET , vcpu))kvm_reset_vcpu (vcpu); +if (kvm_check_request (KVM_REQ_ASYNC_PF , vcpu)) +kvm_check_async_pf_completion (vcpu); +if (kvm_check_request (KVM_REQ_SDEI , vcpu))kvm_sdei_deliver (vcpu);
@@ -789,7 +806,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)smp_store_mb (vcpu->mode,IN_GUEST_MODE );if (ret <=0 ||need_new_vmid_gen (&vcpu->arch.hw_mmu->vmid) || -kvm_request_pending (vcpu)) { + (kvm_request_pending (vcpu) && +READ_ONCE (vcpu->requests) != (1UL <<KVM_REQ_ASYNC_PF ))) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ kvm_pmu_sync_hwstate (vcpu);
arch/arm64/kvm/async_pf.c +207 -0
213 lines not on this page · raw message
arch/arm64/kvm/hypercalls.c +6 -0
@@ -79,6 +79,12 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)break ;case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID : val[0 ] =BIT (ARM_SMCCC_KVM_FUNC_FEATURES ); +if (vcpu->arch.apf) + val[0 ] |=BIT (ARM_SMCCC_KVM_FUNC_ASYNC_PF ); +break ; +case ARM_SMCCC_VENDOR_HYP_KVM_ASYNC_PF_FUNC_ID : + val[0 ] =kvm_arch_async_pf_hypercall (vcpu, &val[1 ], + &val[2 ], &val[3 ]);break ;case SDEI_1_0_FN_SDEI_VERSION :case SDEI_1_0_FN_SDEI_EVENT_REGISTER :
include/linux/arm-smccc.h +10 -0
@@ -118,6 +118,7 @@/* KVM "vendor specific" services */ #define ARM_SMCCC_KVM_FUNC_FEATURES 0 +#define ARM_SMCCC_KVM_FUNC_ASYNC_PF 1#define ARM_SMCCC_KVM_FUNC_FEATURES_2 127#define ARM_SMCCC_KVM_NUM_FUNCS 128
@@ -127,6 +128,15 @@ ARM_SMCCC_OWNER_VENDOR_HYP, \ ARM_SMCCC_KVM_FUNC_FEATURES) +/* Asynchronous page fault service */ +#define ARM_SMCCC_KVM_FUNC_ASYNC_PF_IRQ_ACK 5 + +#define ARM_SMCCC_VENDOR_HYP_KVM_ASYNC_PF_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + ARM_SMCCC_OWNER_VENDOR_HYP, \ + ARM_SMCCC_KVM_FUNC_ASYNC_PF) +/* * Return codes defined in ARM DEN 0070A * ARM DEN 0070A is now merged/consolidated into ARM DEN 0028 C
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 11/17 KVM: arm64: Support async PF hypercalls
In reply to Gavin ShanThis introduces (SMCCC) KVM vendor specific services to configure the asynchronous page fault functionality. The following services are introduced:
* ARM_SMCCC_KVM_FUNC_ASYNC_PF_VERSION
Returns the version, which can be used to identify ABI changes in the future.
* ARM_SMCCC_KVM_FUNC_ASYNC_PF_SLOTS
Return maximal number of tokens that current vCPU can have. It's used by guest to allocate the required resources.
* ARM_SMCCC_KVM_FUNC_ASYNC_PF_{SDEI, IRQ}
Return the associated SDEI or (PPI) IRQ number, configured by vCPU ioctl command.
* ARM_SMCCC_KVM_FUNC_ASYNC_PF_ENABLE
Enable or disable asynchronous page fault on current vCPU.
The corresponding SDEI event and (PPI) IRQ are owned by VMM. So they are configured by vCPU ioctl interface and it will be implemented when the asynchronous page fault capability is exported in the subsequent patches.
--- arch/arm64/kvm/async_pf.c | 119 ++++++++++++++++++++++++++++++++++++++ include/linux/arm-smccc.h | 5 ++ 2 files changed, 124 insertions(+)
arch/arm64/kvm/async_pf.c +119 -0
@@ -313,12 +313,115 @@ void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,write_cache (vcpu, offsetof(struct kvm_vcpu_pv_apf_data , token),0 ); } +static void kvm_arch_async_sdei_notifier (struct kvm_vcpu *vcpu, +unsigned long num, +unsigned int state) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_arch_async_pf_control *apf = vcpu->arch.apf; + +if (!apf) +return ; + +if (num != apf->sdei_event_num) { +kvm_err ("%s: Invalid event number (%d-%d %lx-%llx)\n" , + __func__, kvm->userspace_pid, vcpu->vcpu_idx, + num, apf->sdei_event_num); +return ; + } + +switch (state) { +case KVM_SDEI_NOTIFY_DELIVERED : +if (!apf->notpresent_pending) +break ; + + apf->notpresent_token =0 ; + apf->notpresent_pending = false; +break ; +case KVM_SDEI_NOTIFY_COMPLETED : +break ; +default : +kvm_err ("%s: Invalid state (%d-%d %lx-%d)\n" , + __func__, kvm->userspace_pid, vcpu->vcpu_idx, + num, state); + } +} + +static long kvm_arch_async_enable (struct kvm_vcpu *vcpu,u64 data) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_arch_async_pf_control *apf = vcpu->arch.apf; +gpa_t gpa = (data & ~0x3FUL ); +bool enabled, enable; +int ret; + +if (!apf || !irqchip_in_kernel (kvm)) +return SMCCC_RET_NOT_SUPPORTED ; + +/* Bail if the state transition isn't allowed */ + enabled = !!(apf->control_block &KVM_ASYNC_PF_ENABLED ); + enable = !!(data &KVM_ASYNC_PF_ENABLED ); +if (enable == enabled) { +kvm_debug ("%s: Async PF has been %s on (%d-%d %llx-%llx)\n" , + __func__, enabled ?"enabled" :"disabled" , + kvm->userspace_pid, vcpu->vcpu_idx, + apf->control_block, data); +return SMCCC_RET_NOT_REQUIRED ; + } + +/* To disable the functinality */ +if (!enable) { +kvm_clear_async_pf_completion_queue (vcpu); + apf->control_block = data; +return SMCCC_RET_SUCCESS ; + } + +/* +* The SDEI event and IRQ number should have been given +* prior to enablement. +*/ +if (!apf->sdei_event_num || !apf->irq) { +kvm_err ("%s: Invalid SDEI event or IRQ (%d-%d %llx-%d)\n" , + __func__, kvm->userspace_pid, vcpu->vcpu_idx, + apf->sdei_event_num, apf->irq); +return SMCCC_RET_INVALID_PARAMETER ; + } + +/* Register SDEI event notifier */ + ret =kvm_sdei_register_notifier (kvm, apf->sdei_event_num, + kvm_arch_async_sdei_notifier); +if (ret) { +kvm_err ("%s: Error %d registering SDEI notifier (%d-%d %llx)\n" , + __func__, ret, kvm->userspace_pid, vcpu->vcpu_idx, + apf->sdei_event_num); +return SMCCC_RET_NOT_SUPPORTED ; + } + +/* Initialize cache shared by host and guest */ + ret =kvm_gfn_to_hva_cache_init (kvm, &apf->cache, gpa, + offsetofend(struct kvm_vcpu_pv_apf_data, token)); +if (ret) { +kvm_err ("%s: Error %d initializing cache (%d-%d)\n" , + __func__, ret, kvm->userspace_pid, vcpu->vcpu_idx); +return SMCCC_RET_NOT_SUPPORTED ; + } + +/* Flush the token table */ +kvm_async_pf_reset_slot (vcpu); + apf->send_user_only = !(data &KVM_ASYNC_PF_SEND_ALWAYS ); +kvm_async_pf_wakeup_all (vcpu); + apf->control_block = data; + +return SMCCC_RET_SUCCESS ; +} +long kvm_arch_async_pf_hypercall (struct kvm_vcpu *vcpu,long *r1,long *r2,long *r3) {struct kvm *kvm = vcpu->kvm;struct kvm_arch_async_pf_control *apf = vcpu->arch.apf;u32 func; +u64 data;long ret =SMCCC_RET_SUCCESS ;if (!apf) {
@@ -328,6 +431,22 @@ long kvm_arch_async_pf_hypercall(struct kvm_vcpu *vcpu, func =smccc_get_arg1 (vcpu);switch (func) { +case ARM_SMCCC_KVM_FUNC_ASYNC_PF_VERSION : + *r1 =0x010000 ;/* v1.0.0 */ +break ; +case ARM_SMCCC_KVM_FUNC_ASYNC_PF_SLOTS : + *r1 =ASYNC_PF_PER_VCPU ; +break ; +case ARM_SMCCC_KVM_FUNC_ASYNC_PF_SDEI : + *r1 = apf->sdei_event_num; +break ; +case ARM_SMCCC_KVM_FUNC_ASYNC_PF_IRQ : + *r1 = apf->irq; +break ; +case ARM_SMCCC_KVM_FUNC_ASYNC_PF_ENABLE : + data = (smccc_get_arg3 (vcpu) <<32 ) |smccc_get_arg2 (vcpu); + ret =kvm_arch_async_enable (vcpu, data); +break ;case ARM_SMCCC_KVM_FUNC_ASYNC_PF_IRQ_ACK :if (!apf->pageready_pending)break ;
include/linux/arm-smccc.h +5 -0
@@ -129,6 +129,11 @@ ARM_SMCCC_KVM_FUNC_FEATURES)/* Asynchronous page fault service */ +#define ARM_SMCCC_KVM_FUNC_ASYNC_PF_VERSION 0 +#define ARM_SMCCC_KVM_FUNC_ASYNC_PF_SLOTS 1 +#define ARM_SMCCC_KVM_FUNC_ASYNC_PF_SDEI 2 +#define ARM_SMCCC_KVM_FUNC_ASYNC_PF_IRQ 3 +#define ARM_SMCCC_KVM_FUNC_ASYNC_PF_ENABLE 4#define ARM_SMCCC_KVM_FUNC_ASYNC_PF_IRQ_ACK 5#define ARM_SMCCC_VENDOR_HYP_KVM_ASYNC_PF_FUNC_ID \
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 12/17 KVM: arm64: Support async PF ioctl commands
In reply to Gavin ShanThis supports ioctl commands for configuration and migration:
KVM_ARM_ASYNC_PF_CMD_GET_VERSION
Return implementation version
KVM_ARM_ASYNC_PF_CMD_GET_SDEI
Return SDEI event number used for page-not-present notification
KVM_ARM_ASYNC_PF_CMD_GET_IRQ
Return IRQ number used for page-ready notification
KVM_ARM_ASYNC_PF_CMD_GET_CONTROL
Get control block when VM is migrated
KVM_ARM_ASYNC_PF_CMD_SET_SDEI
Set SDEI event number when VM is started or migrated
KVM_ARM_ASYNC_PF_CMD_SET_IRQ
Set IRQ number during when VM is started or migrated
KVM_ARM_ASYNC_PF_CMD_SET_CONTROL
Set control block when VM is migrated
--- arch/arm64/include/asm/kvm_host.h | 14 +++++++ arch/arm64/include/uapi/asm/kvm.h | 19 +++++++++ arch/arm64/kvm/arm.c | 6 +++ arch/arm64/kvm/async_pf.c | 64 +++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 3 ++ 5 files changed, 106 insertions(+)
arch/arm64/include/asm/kvm_host.h +14 -0
@@ -778,6 +778,8 @@ void kvm_arch_async_page_present(struct kvm_vcpu *vcpu, struct kvm_async_pf *work);long kvm_arch_async_pf_hypercall (struct kvm_vcpu *vcpu,long *r1,long *r2,long *r3); +long kvm_arch_async_pf_vm_ioctl (struct kvm *kvm,unsigned long arg); +long kvm_arch_async_pf_vcpu_ioctl (struct kvm_vcpu *vcpu,unsigned long arg);void kvm_arch_async_pf_destroy_vcpu (struct kvm_vcpu *vcpu);#else static inline void kvm_arch_async_pf_create_vcpu (struct kvm_vcpu *vcpu) { }
@@ -799,6 +801,18 @@ static inline long kvm_arch_async_pf_hypercall(struct kvm_vcpu *vcpu, {return SMCCC_RET_NOT_SUPPORTED ; } + +static inline long kvm_arch_async_pf_vm_ioctl (struct kvm *kvm, +unsigned long arg) +{ +return -EPERM ; +} + +static inline long kvm_arch_async_pf_vcpu_ioctl (struct kvm_vcpu *vcpu, +unsigned long arg) +{ +return -EPERM ; +}#endif /* Guest/host FPSIMD coordination helpers */
arch/arm64/include/uapi/asm/kvm.h +19 -0
@@ -403,6 +403,25 @@ struct kvm_vcpu_events {#define KVM_PSCI_RET_INVAL PSCI_RET_INVALID_PARAMS#define KVM_PSCI_RET_DENIED PSCI_RET_DENIED +/* Asynchronous page fault */ +#define KVM_ARM_ASYNC_PF_CMD_GET_VERSION 0 +#define KVM_ARM_ASYNC_PF_CMD_GET_SDEI 1 +#define KVM_ARM_ASYNC_PF_CMD_GET_IRQ 2 +#define KVM_ARM_ASYNC_PF_CMD_GET_CONTROL 3 +#define KVM_ARM_ASYNC_PF_CMD_SET_SDEI 4 +#define KVM_ARM_ASYNC_PF_CMD_SET_IRQ 5 +#define KVM_ARM_ASYNC_PF_CMD_SET_CONTROL 6 + +struct kvm_arm_async_pf_cmd { +__u32 cmd; +union { +__u32 version; +__u64 sdei; +__u32 irq; +__u64 control; + }; +}; +#endif #endif /* __ARM_KVM_H__ */
arch/arm64/kvm/arm.c +6 -0
@@ -1287,6 +1287,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,case KVM_ARM_SDEI_COMMAND : {return kvm_sdei_vcpu_ioctl (vcpu, arg); } +case KVM_ARM_ASYNC_PF_COMMAND : { +return kvm_arch_async_pf_vcpu_ioctl (vcpu, arg); + }default : r = -EINVAL ; }
@@ -1364,6 +1367,9 @@ long kvm_arch_vm_ioctl(struct file *filp,case KVM_ARM_SDEI_COMMAND : {return kvm_sdei_vm_ioctl (kvm, arg); } +case KVM_ARM_ASYNC_PF_COMMAND : { +return kvm_arch_async_pf_vm_ioctl (kvm, arg); + }default :return -EINVAL ; }
arch/arm64/kvm/async_pf.c +64 -0
70 lines not on this page · raw message
include/uapi/linux/kvm.h +3 -0
@@ -1569,6 +1569,9 @@ struct kvm_pv_cmd {/* Available with KVM_CAP_ARM_SDEI */ #define KVM_ARM_SDEI_COMMAND _IOWR(KVMIO, 0xc8, struct kvm_sdei_cmd) +/* Available with KVM_CAP_ASYNC_PF or KVM_CAP_ASYNC_PF_INT */ +#define KVM_ARM_ASYNC_PF_COMMAND _IOWR(KVMIO, 0xc9, struct kvm_arm_async_pf_cmd) +/* Secure Encrypted Virtualization command */ enum sev_cmd_id {/* Guest initialization commands */
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 13/17 KVM: arm64: Export async PF capability
In reply to Gavin ShanThis exports the asynchronous page fault capability:
* Identify capability KVM_CAP_ASYNC_{PF, PF_INT}.
* Standardize SDEI event for asynchronous page fault.
* Enable kernel config CONFIG_KVM_ASYNC_{PF, PF_SLOT}.
--- arch/arm64/include/uapi/asm/kvm_sdei.h | 1 + arch/arm64/kvm/Kconfig | 2 ++ arch/arm64/kvm/arm.c | 4 ++++ arch/arm64/kvm/sdei.c | 5 +++++ 4 files changed, 12 insertions(+)
arch/arm64/include/uapi/asm/kvm_sdei.h +1 -0
@@ -13,6 +13,7 @@#define KVM_SDEI_MAX_VCPUS 512#define KVM_SDEI_INVALID_NUM 0#define KVM_SDEI_DEFAULT_NUM 0x40400000 +#define KVM_SDEI_ASYNC_PF_NUM 0x40400001struct kvm_sdei_event_state {uint64_t num;
arch/arm64/kvm/Kconfig +2 -0
@@ -31,6 +31,8 @@ menuconfig KVM select SRCU select KVM_VFIO select HAVE_KVM_EVENTFD + select KVM_ASYNC_PF + select KVM_ASYNC_PF_SLOT select HAVE_KVM_IRQFD select HAVE_KVM_MSI select HAVE_KVM_IRQCHIP
arch/arm64/kvm/arm.c +4 -0
@@ -269,6 +269,10 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_ARM_SDEI : r =1 ;break ; +case KVM_CAP_ASYNC_PF : +case KVM_CAP_ASYNC_PF_INT : + r =IS_ENABLED (CONFIG_KVM_ASYNC_PF ) ?1 :0 ; +break ;default : r =0 ; }
arch/arm64/kvm/sdei.c +5 -0
@@ -19,6 +19,11 @@ static struct kvm_sdei_event_state defined_kse[] = { 1, SDEI_EVENT_PRIORITY_CRITICAL }, + {KVM_SDEI_ASYNC_PF_NUM , +SDEI_EVENT_TYPE_PRIVATE , +1 , +SDEI_EVENT_PRIORITY_CRITICAL + }, }; static struct kvm_sdei_event *kvm_sdei_find_event(struct kvm *kvm,
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 14/17 arm64: Detect async PF para-virtualization feature
In reply to Gavin ShanThis implements kvm_para_available() to check if para-virtualization features are available or not. Besides, kvm_para_has_feature() is enhanced to detect the asynchronous page fault para-virtualization feature. These two functions are going to be used by guest kernel to enable the asynchronous page fault.
This also adds kernel option (CONFIG_KVM_GUEST), which is the umbrella for the optimizations related to KVM para-virtualization.
--- arch/arm64/Kconfig | 11 +++++++++++ arch/arm64/include/asm/kvm_para.h | 12 +++++++++++- arch/arm64/include/uapi/asm/kvm_para.h | 2 ++ 3 files changed, 24 insertions(+), 1 deletion(-)
arch/arm64/Kconfig +11 -0
@@ -1078,6 +1078,17 @@ config PARAVIRT_TIME_ACCOUNTING If in doubt, say N here. +config KVM_GUEST + bool "KVM Guest Support" + depends on PARAVIRT + default y + help + This option enables various optimizations for running under the KVM + hypervisor. Overhead for the kernel when not running inside KVM should + be minimal. + + In case of doubt, say Y + config KEXEC depends on PM_SLEEP_SMP select KEXEC_CORE
arch/arm64/include/asm/kvm_para.h +11 -1
@@ -3,6 +3,8 @@#define _ASM_ARM_KVM_PARA_H#include <uapi/asm/kvm_para.h> +#include <linux/of.h> +#include <asm/hypervisor.h> static inline bool kvm_check_and_clear_guest_paused (void ) {
@@ -11,7 +13,12 @@ static inline bool kvm_check_and_clear_guest_paused(void)static inline unsigned int kvm_arch_para_features (void ) { -return 0 ; +unsigned int features =0 ; + +if (kvm_arm_hyp_service_available (ARM_SMCCC_KVM_FUNC_ASYNC_PF )) + features |= (1 <<KVM_FEATURE_ASYNC_PF ); + +return features; }static inline unsigned int kvm_arch_para_hints (void )
@@ -21,6 +28,9 @@ static inline unsigned int kvm_arch_para_hints(void)static inline bool kvm_para_available (void ) { +if (IS_ENABLED (CONFIG_KVM_GUEST )) +return true; +return false; }
arch/arm64/include/uapi/asm/kvm_para.h +2 -0
@@ -4,6 +4,8 @@#include <linux/types.h> +#define KVM_FEATURE_ASYNC_PF 0 +/* Async PF */ #define KVM_ASYNC_PF_ENABLED (1 << 0)#define KVM_ASYNC_PF_SEND_ALWAYS (1 << 1)
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 15/17 arm64: Reschedule process on aync PF
In reply to Gavin ShanThe page-not-present notification is delivered by SDEI event. The guest reschedules current process to another one when the SDEI event is received. It's not safe to do so in the SDEI event handler because the SDEI event should be acknowledged as soon as possible.
So the rescheduling is postponed until the current process switches from kernel to user mode. In order to trigger the switch, the SDEI event handler sends (reschedule) IPI to current CPU and it's delivered in time after the SDEI event is acknowledged.
A new thread flag (TIF_ASYNC_PF) is introduced in order to track the state for the process, to be rescheduled. With the flag is set, there is a head of wait-queue is associated with the process. The process keeps rescheduling itself until the flag is cleared when page-ready notification is received through (PPI) interrupt.
--- arch/arm64/include/asm/processor.h | 1 + arch/arm64/include/asm/thread_info.h | 4 +++- arch/arm64/kernel/signal.c | 17 +++++++++++++++++ 3 files changed, 21 insertions(+), 1 deletion(-)
arch/arm64/include/asm/processor.h +1 -0
@@ -154,6 +154,7 @@ struct thread_struct {u64 sctlr_tcf0;u64 gcr_user_excl;#endif +void *data; }; static inline void arch_thread_struct_whitelist(unsigned long *offset,
arch/arm64/include/asm/thread_info.h +3 -1
@@ -65,6 +65,7 @@ void arch_release_task_struct(struct task_struct *tsk);#define TIF_UPROBE 4/* uprobe breakpoint or singlestep */ #define TIF_MTE_ASYNC_FAULT 5/* MTE Asynchronous Tag Check Fault */ #define TIF_NOTIFY_SIGNAL 6/* signal notifications exist */ +#define TIF_ASYNC_PF 7/* Asynchronous page fault */ #define TIF_SYSCALL_TRACE 8/* syscall trace active */ #define TIF_SYSCALL_AUDIT 9/* syscall auditing */ #define TIF_SYSCALL_TRACEPOINT 10/* syscall tracepoint for ftrace */
@@ -95,11 +96,12 @@ void arch_release_task_struct(struct task_struct *tsk);#define _TIF_SVE (1 << TIF_SVE)#define _TIF_MTE_ASYNC_FAULT (1 << TIF_MTE_ASYNC_FAULT)#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) +#define _TIF_ASYNC_PF (1 << TIF_ASYNC_PF)#define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ _TIF_UPROBE | _TIF_MTE_ASYNC_FAULT | \ - _TIF_NOTIFY_SIGNAL) + _TIF_NOTIFY_SIGNAL | _TIF_ASYNC_PF)#define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
arch/arm64/kernel/signal.c +17 -0
@@ -915,6 +915,23 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags) { do { + if (thread_flags & _TIF_ASYNC_PF) { + struct swait_queue_head *wq = + READ_ONCE(current->thread.data); + DECLARE_SWAITQUEUE(wait); + + local_daif_restore(DAIF_PROCCTX_NOIRQ); + + do { + prepare_to_swait_exclusive(wq, + &wait, TASK_UNINTERRUPTIBLE); + if (!test_thread_flag(TIF_ASYNC_PF)) + break; + + schedule(); + } while (test_thread_flag(TIF_ASYNC_PF)); + } + if (thread_flags & _TIF_NEED_RESCHED) { /* Unmask Debug and SError for the next task */ local_daif_restore(DAIF_PROCCTX_NOIRQ);
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 16/17 arm64: Enable async PF
In reply to Gavin ShanThis enables asynchronous page fault from guest side. The design is highlighted as below:
* The per-vCPU shared memory region, which is represented by "struct kvm_vcpu_pv_apf_data", is allocated. The reason and token associated with the received notifications of asynchronous page fault are delivered through it.
* A per-vCPU table, which is represented by "struct kvm_apf_table", is allocated. The process, on which the page-not-present notification is received, is added into the table so that it can reschedule itself on switching from kernel to user mode. Afterwards, the process, identified by token, is removed from the table and put into runnable state when page-ready notification is received.
* During CPU hotplug, the (private) SDEI event is expected to be enabled or disabled on the affected CPU by SDEI client driver. The (PPI) interrupt is enabled or disabled on the affected CPU by ourself. When the system is going to reboot, the SDEI event is disabled and unregistered and the (PPI) interrupt is disabled.
* The SDEI event and (PPI) interrupt number are retrieved from host through SMCCC interface. Besides, the version of the asynchronous page fault is validated when the feature is enabled on the guest.
* The feature is disabled on guest when boot parameter "no-kvmapf" is specified.
--- arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/kvm.c | 452 +++++++++++++++++++++++++++++++++++++ 2 files changed, 453 insertions(+) create mode 100644 arch/arm64/kernel/kvm.c
arch/arm64/kernel/Makefile +1 -0
@@ -47,6 +47,7 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o +obj-$(CONFIG_KVM_GUEST) += kvm.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \
arch/arm64/kernel/kvm.c new file +452 -0
452 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v2 17/17 KVM: arm64: Add async PF document
In reply to Gavin ShanThis adds document to explain the interface for asynchronous page fault and how it works in general.
--- Documentation/virt/kvm/arm/apf.rst | 143 +++++++++++++++++++++++++++ Documentation/virt/kvm/arm/index.rst | 1 + 2 files changed, 144 insertions(+) create mode 100644 Documentation/virt/kvm/arm/apf.rst
Documentation/virt/kvm/arm/apf.rst new file +143 -0
@@ -0,0 +1,143 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Asynchronous Page Fault Support for arm64 +========================================= + +There are two stages of page faults when KVM module is enabled as accelerator +to the guest. The guest is responsible for handling the stage-1 page faults, +while the host handles the stage-2 page faults. During the period of handling +the stage-2 page faults, the guest is suspended until the requested page is +ready. It could take several milliseconds, even hundreds of milliseconds in +extreme situations because I/O might be required to move the requested page +from disk to DRAM. The guest does not do any work when it is suspended. The +feature (Asynchronous Page Fault) is introduced to take advantage of the +suspending period and to improve the overall performance. + +There are two paths in order to fulfil the asynchronous page fault, called +as control path and data path. The control path allows the VMM or guest to +configure the functionality, while the notifications are delivered in data +path. The notifications are classified into page-not-present and page-ready +notifications. + +Data Path +--------- + +There are two types of notifications delivered from host to guest in the +data path: page-not-present and page-ready notification. They are delivered +through SDEI event and (PPI) interrupt separately. Besides, there is a shared +buffer between host and guest to indicate the reason and sequential token, +which is used to identify the asynchronous page fault. The reason and token +resident in the shared buffer is written by host, read and cleared by guest. +An asynchronous page fault is delivered and completed as below. + +(1) When an asynchronous page fault starts, a (workqueue) worker is created + and queued to the vCPU's pending queue. The worker makes the requested + page ready and resident to DRAM in the background. The shared buffer is + updated with reason and sequential token. After that, SDEI event is sent + to guest as page-not-present notification. + +(2) When the SDEI event is received on guest, the current process is tagged + with TIF_ASYNC_PF and associated with a wait queue. The process is ready + to keep rescheduling itself on switching from kernel to user mode. After + that, a reschedule IPI is sent to current CPU and the received SDEI event + is acknowledged. Note that the IPI is delivered when the acknowledgment + on the SDEI event is received on host. + +(3) On the host, the worker is dequeued from the vCPU's pending queue and + enqueued to its completion queue when the requested page becomes ready. + In the mean while, KVM_REQ_ASYNC_PF request is sent the vCPU if the + worker is the first element enqueued to the completion queue. + +(4) With pending KVM_REQ_ASYNC_PF request, the first worker in the completion + queue is dequeued and destroyed. In the mean while, a (PPI) interrupt is + sent to guest with updated reason and token in the shared buffer. + +(5) When the (PPI) interrupt is received on guest, the affected process is + located using the token and waken up after its TIF_ASYNC_PF tag is cleared. + After that, the interrupt is acknowledged through SMCCC interface. The + workers in the completion queue is dequeued and destroyed if any workers + exist, and another (PPI) interrupt is sent to the guest. + +Control Path +------------ + +The configurations are passed through SMCCC or ioctl interface. The SDEI +event and (PPI) interrupt are owned by VMM, so the SDEI event and interrupt +numbers are configured through ioctl command on per-vCPU basis. Besides, +the functionality might be enabled and configured through ioctl interface +by VMM during migration: + + * KVM_ARM_ASYNC_PF_CMD_GET_VERSION + + Returns the current version of the feature, supported by the host. It is + made up of major, minor and revision fields. Each field is one byte in + length. + + * KVM_ARM_ASYNC_PF_CMD_GET_SDEI: + + Retrieve the SDEI event number, used for page-not-present notification, + so that it can be configured on destination VM in the scenario of + migration. + + * KVM_ARM_ASYNC_PF_GET_IRQ: + + Retrieve the IRQ (PPI) number, used for page-ready notification, so that + it can be configured on destination VM in the scenario of migration. + + * KVM_ARM_ASYNC_PF_CMD_GET_CONTROL + + Retrieve the address of control block, so that it can be configured on + destination VM in the scenario of migration. + + * KVM_ARM_ASYNC_PF_CMD_SET_SDEI: + + Used by VMM to configure number of SDEI event, which is used to deliver + page-not-present notification by host. This is used when VM is started + or migrated. + + * KVM_ARM_ASYNC_PF_CMD_SET_IRQ + + Used by VMM to configure number of (PPI) interrupt, which is used to + deliver page-ready notification by host. This is used when VM is started + or migrated. + + * KVM_ARM_ASYNC_PF_CMD_SET_CONTROL + + Set the control block on the destination VM in the scenario of migration. + +The other configurations are passed through SMCCC interface. The host exports +the capability through KVM vendor specific service, which is identified by +ARM_SMCCC_KVM_FUNC_ASYNC_PF_FUNC_ID. There are several functions defined for +this: + + * ARM_SMCCC_KVM_FUNC_ASYNC_PF_VERSION + + Returns the current version of the feature, supported by the host. It is + made up of major, minor and revision fields. Each field is one byte in + length. + + * ARM_SMCCC_KVM_FUNC_ASYNC_PF_SLOTS + + Returns the size of the hashed GFN table. It is used by guest to set up + the capacity of waiting process table. + + * ARM_SMCCC_KVM_FUNC_ASYNC_PF_SDEI + * ARM_SMCCC_KVM_FUNC_ASYNC_PF_IRQ + + Used by the guest to retrieve the SDEI event and (PPI) interrupt number + that are configured by VMM. + + * ARM_SMCCC_KVM_FUNC_ASYNC_PF_ENABLE + + Used by the guest to enable or disable the feature on the specific vCPU. + The argument is made up of shared buffer and flags. The shared buffer + is written by host to indicate the reason about the delivered asynchronous + page fault and token (sequence number) to identify that. There are two + flags are supported: KVM_ASYNC_PF_ENABLED is used to enable or disable + the feature. KVM_ASYNC_PF_SEND_ALWAYS allows to deliver page-not-present + notification regardless of the guest's state. Otherwise, the notification + is delivered only when the guest is in user mode. + + * ARM_SMCCC_KVM_FUNC_ASYNC_PF_IRQ_ACK + + Used by the guest to acknowledge the completion of page-ready notification.
Documentation/virt/kvm/arm/index.rst +1 -0
@@ -7,6 +7,7 @@ ARM .. toctree:: :maxdepth: 2 + apf hyp-abi psci pvtime
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm