Support Asynchronous Page Fault
Subsystems (5)
cover Support Asynchronous Page Fault
There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.
This series introduces the feature (asynchronous page fault) to resolve the issue and improve the guest's performance. It depends on the series to support SDEI virtualization and refactoring SDEI client driver. This also depends on QEMU changes to export SDEI/APFT tables. All the code including this series can be found from github:
https://github.com/gwshan/linux ("sdei_client")
https://github.com/gwshan/linux ("sdei")
https://github.com/gwshan/linux ("apf")
https://github.com/gwshan/qemu ("apf")
The functionality is driven by two notifications: page-not-present and page-ready. They're delivered from the host to guest via SDEI event and PPI separately. In the mean while, each notification is always associated with a token, used to identify the notification. The token is passed by the shared memory between host/guest. Besides, the SMCCC interface is mitigated by the guest to configure, enable or disable the functionality. It's traditional control path.
When the guest is trapped to host because of stage two page fault, a page-not-present notification is raised by the host, and sent to the guest through (KVM private) SDEI event (0x40200001) if the requested page can't be populated immediately. In the mean while, a (background) worker is also started to populate the requested page. On receiving the SDEI event, the guest marks the current running process with special flag (TIF_ASYNC_PF) and associates the process with a pre-defined waitqueue. At same time, a (reschedule) IPI is sent to the CPU where the process was running. After the SDEI event is acknoledged by the guest, the (reschedule) IPI is delivered and it causes context switch from kernel to user space. During the context switch, the process with TIF_ASYNC_PF flag is suspended on the associated waitqueue.
Later on, a page-ready notification is sent to guest after the requested page is populated by the (background) worker. On receiving the interrupt, the guest uses the associated token to locate the process, which was previously suspended because of page-not-present, and wakes it up.
The series is organized as below:
PATCH[1-2]: support KVM hypervisor SMCCC services, which are developed by
Will Deacon.
PATCH[3]: export kvm_handle_user_mem_abort() with @prefault parameter
supported, which is prepatory work to support the feature.
PATCH[4]: support asynchronous page fault in host side.
PATCH[5]: exposes APFT (Asynchronous Page Fault Table) ACPI table, which
will be used by guest kernel to support the feature
PATCH[6]: support asynchronous page fault in guest side.
=======
Testing
=======
In the test case [1] and [2], "testsuite mem" is executed to allocate the specified percentage of free memory (90%) and then release them. In the mean while, the calculation thread is started or not. When the calculation thread isn't started, there isn't obvious performance degradtion. When the calculation thread is started, the performance is improved by 27.7% and 28.6% separately, depending on THP enablement sttus on the host side.
In test case [3] and [4], the kernel image is built and check the used time. The performance is improved by 9.7% and 9.9% separately, depending on THP enablement status on the host side.
[1] Two threads to allocate/free memory and do calculation
vCPU: 1
Memory: 8GB
memory.limit_in_bytes: 2GB
memory.swappiness: 100
host: THP disabled
command: "testsuite mem 90 1 [thread]"
"-": Disabled asynchronous page fault
"+": Enabled asynchronous page fault
"T" With the calculation thread
Idx - + Output T- T+ Output
==========================================================================
1 93.1s 93.6s - 223.8s 21117147961 391.9s 49845637101 -
2 93.3s 94.2s - 237.9s 23394567744 397.0s 50506074773 -
3 93.5s 94.3s - 244.2s 24305177553 405.8s 51853498870 -
4 94.1s 95.0s - 262.8s 27113310073 421.7s 54338181069 -
5 94.3s 95.2s - 272.7s 28565479414 434.3s 56171922019 -
==========================================================================
93.6s 94.4s -0.8% 248.2s 24899136549 410.1s 52543062766
100318841/s 128122562/s +27.7%[2] Two threads to allocate/free memory and do calculation
vCPU: 1
Memory: 8GB
memory.limit_in_bytes: 2GB
memory.swappiness: 100
host: THP enabled
command: "testsuite mem 90 1 [thread]"
"-": Disabled asynchronous page fault
"+": Enabled asynchronous page fault
"T" With the calculation thread
Idx - + Output T- T+ Output
==========================================================================
1 91.3s 91.2s - 218.8s 20319612017 389.6s 49016175698 -
2 91.7s 91.6s - 233.9s 22619566161 402.0s 50901616319 -
3 91.8s 91.9s - 251.1s 25066180266 405.3s 51247353704 -
4 92.7s 92.2s - 251.1s 25262121229 406.9s 51692420054 -
5 93.1s 92.2s - 260.7s 26532616925 425.4s 54412348724 -
==========================================================================
92.1s 91.8s +3.0% 243.1s 23960019319 405.8 51453982899
98560342/s 126796409/s +28.6%[3] Clear kernel image and rebuild it.
vCPU: 24 Memory: 8GB
memory.limit_in_bytes: 2GB
memory.swapiness: 100
Host: THP disabled
command: "make -j 24 clean > /dev/null 2>&1 &&
make -j 24 > /dev/null 2>&1"
Idx Disabled Enabled Output
==================================
1 2211s 2000s +9.5%
2 2333s 2060s +11.7%
3 2568s 2192s +14.6%
4 2631s 2423s +7.9%
5 2756s 2605s +5.4%
==================================
2499s 2256s +9.7%[4] Clear kernel image and rebuild it.
vCPU: 24 Memory: 8GB
memory.limit_in_bytes: 2GB
memory.swapiness: 100
Host: THP enabled
command: "make -j 24 clean > /dev/null 2>&1 &&
make -j 24 > /dev/null 2>&1"
Idx Disabled Enabled Output
==================================
1 2049s 1850s +9.7%
2 2144s 1947s +9.1%
3 2164s 1997s +7.7%
4 2192s 2031s +7.3%
5 2515s 2141s +14.8%
==================================
2214s 1993s +9.9%Gavin Shan (4):
kvm/arm64: Export kvm_handle_user_mem_abort() with prefault mode arm64/kvm: Support async page fault
drivers/acpi: Import ACPI APF table
arm64/kernel: Support async page fault
Will Deacon (2):
arm64: Probe for the presence of KVM hypervisor services during boot arm/arm64: KVM: Advertise KVM UID to guests via SMCCC
arch/arm64/Kconfig | 11 + arch/arm64/include/asm/esr.h | 5 + arch/arm64/include/asm/hypervisor.h | 11 + arch/arm64/include/asm/kvm_emulate.h | 8 +- arch/arm64/include/asm/kvm_host.h | 54 +++ arch/arm64/include/asm/kvm_para.h | 41 +++ arch/arm64/include/asm/processor.h | 1 + arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/include/uapi/asm/Kbuild | 2 - arch/arm64/include/uapi/asm/kvm_para.h | 23 ++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/kvm.c | 478 +++++++++++++++++++++++++ arch/arm64/kernel/setup.c | 32 ++ arch/arm64/kernel/signal.c | 17 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/arm.c | 45 ++- arch/arm64/kvm/async_pf.c | 462 ++++++++++++++++++++++++ arch/arm64/kvm/hypercalls.c | 37 +- arch/arm64/kvm/mmu.c | 47 ++- arch/arm64/kvm/sdei.c | 8 + include/acpi/actbl2.h | 18 + include/linux/arm-smccc.h | 41 +++ 23 files changed, 1321 insertions(+), 27 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_para.h create mode 100644 arch/arm64/include/uapi/asm/kvm_para.h create mode 100644 arch/arm64/kernel/kvm.c create mode 100644 arch/arm64/kvm/async_pf.c
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH 1/6 arm64: Probe for the presence of KVM hypervisor services during boot
From: Will Deacon <[email protected]>
Although the SMCCC specification provides some limited functionality for describing the presence of hypervisor and firmware services, this is generally applicable only to functions designated as "Arm Architecture Service Functions" and no portable discovery mechanism is provided for standard hypervisor services, despite having a designated range of function identifiers reserved by the specification.
In an attempt to avoid the need for additional firmware changes every time a new function is added, introduce a UID to identify the service provider as being compatible with KVM. Once this has been established, additional services can be discovered via a feature bitmap.
--- arch/arm64/include/asm/hypervisor.h | 11 ++++++++++ arch/arm64/kernel/setup.c | 32 +++++++++++++++++++++++++++++ include/linux/arm-smccc.h | 26 +++++++++++++++++++++++ 3 files changed, 69 insertions(+)
arch/arm64/include/asm/hypervisor.h +11 -0
@@ -2,6 +2,17 @@#ifndef _ASM_ARM64_HYPERVISOR_H#define _ASM_ARM64_HYPERVISOR_H +#include <linux/arm-smccc.h> #include <asm/xen/hypervisor.h> +static inline bool kvm_arm_hyp_service_available (u32 func_id) +{ + extern DECLARE_BITMAP(__kvm_arm_hyp_services, ARM_SMCCC_KVM_NUM_FUNCS); + +if (func_id >=ARM_SMCCC_KVM_NUM_FUNCS ) +return -EINVAL ; + +return test_bit (func_id, __kvm_arm_hyp_services); +} +#endif
arch/arm64/kernel/setup.c +32 -0
@@ -7,6 +7,7 @@*/ #include <linux/acpi.h> +#include <linux/arm-smccc.h> #include <linux/export.h> #include <linux/kernel.h> #include <linux/stddef.h>
@@ -275,6 +276,7 @@ static int __init reserve_memblock_reserved_regions(void)arch_initcall (reserve_memblock_reserved_regions);u64 __cpu_logical_map[NR_CPUS ] = { [0 ...NR_CPUS -1 ] =INVALID_HWID }; +DECLARE_BITMAP(__kvm_arm_hyp_services, ARM_SMCCC_KVM_NUM_FUNCS) = { };u64 cpu_logical_map (int cpu) {
@@ -282,6 +284,35 @@ u64 cpu_logical_map(int cpu) }EXPORT_SYMBOL_GPL (cpu_logical_map); +static void __init kvm_init_hyp_services (void ) +{ +struct arm_smccc_res res; +int i; + +arm_smccc_1_1_invoke (ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID , &res); +if (res.a0 !=ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0 || + res.a1 !=ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1 || + res.a2 !=ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 || + res.a3 !=ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3 ) +return ; + +memset (&res,0 ,sizeof (res)); +arm_smccc_1_1_invoke (ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID , &res); +for (i =0 ; i <32 ; ++i) { +if (res.a0 & (i)) +set_bit (i + (32 *0 ), __kvm_arm_hyp_services); +if (res.a1 & (i)) +set_bit (i + (32 *1 ), __kvm_arm_hyp_services); +if (res.a2 & (i)) +set_bit (i + (32 *2 ), __kvm_arm_hyp_services); +if (res.a3 & (i)) +set_bit (i + (32 *3 ), __kvm_arm_hyp_services); + } + +pr_info ("KVM hypervisor services detected (0x%08lx 0x%08lx 0x%08lx 0x%08lx)\n" , + res.a3, res.a2, res.a1, res.a0); +} + void __init __no_sanitize_address setup_arch(char **cmdline_p) { init_mm.start_code = (unsigned long ) _text;
@@ -354,6 +385,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)else psci_acpi_init (); +kvm_init_hyp_services ();init_bootcpu_ops ();smp_init_cpus ();smp_build_mpidr_hash ();
include/linux/arm-smccc.h +26 -0
@@ -49,11 +49,14 @@#define ARM_SMCCC_OWNER_OEM 3#define ARM_SMCCC_OWNER_STANDARD 4#define ARM_SMCCC_OWNER_STANDARD_HYP 5 +#define ARM_SMCCC_OWNER_VENDOR_HYP 6#define ARM_SMCCC_OWNER_TRUSTED_APP 48#define ARM_SMCCC_OWNER_TRUSTED_APP_END 49#define ARM_SMCCC_OWNER_TRUSTED_OS 50#define ARM_SMCCC_OWNER_TRUSTED_OS_END 63 +#define ARM_SMCCC_FUNC_QUERY_CALL_UID 0xff01 +#define ARM_SMCCC_QUIRK_NONE 0#define ARM_SMCCC_QUIRK_QCOM_A6 1/* Save/restore register a6 */
@@ -99,6 +102,29 @@ ARM_SMCCC_OWNER_STANDARD_HYP, \ 0x21) +#define ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + ARM_SMCCC_OWNER_VENDOR_HYP, \ + ARM_SMCCC_FUNC_QUERY_CALL_UID) + +/* KVM UID value: 28b46fb6-2ec5-11e9-a9ca-4b564d003a74 */ +#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0 0xb66fb428U +#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1 0xe911c52eU +#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 0x564bcaa9U +#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3 0x743a004dU + +/* KVM "vendor specific" services */ +#define ARM_SMCCC_KVM_FUNC_FEATURES 0 +#define ARM_SMCCC_KVM_FUNC_FEATURES_2 127 +#define ARM_SMCCC_KVM_NUM_FUNCS 128 + +#define ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + ARM_SMCCC_OWNER_VENDOR_HYP, \ + ARM_SMCCC_KVM_FUNC_FEATURES) +/* * Return codes defined in ARM DEN 0070A * ARM DEN 0070A is now merged/consolidated into ARM DEN 0028 C
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH 2/6 arm/arm64: KVM: Advertise KVM UID to guests via SMCCC
In reply to Gavin ShanFrom: Will Deacon <[email protected]>
We can advertise ourselves to guests as KVM and provide a basic features bitmap for discoverability of future hypervisor services.
--- arch/arm64/kvm/hypercalls.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
arch/arm64/kvm/hypercalls.c +19 -10
@@ -13,13 +13,13 @@int kvm_hvc_call_handler (struct kvm_vcpu *vcpu) {u32 func_id =smccc_get_function (vcpu); -long val =SMCCC_RET_NOT_SUPPORTED ; +long val[4 ] = {SMCCC_RET_NOT_SUPPORTED };u32 feature;gpa_t gpa;switch (func_id) {case ARM_SMCCC_VERSION_FUNC_ID : - val =ARM_SMCCC_VERSION_1_1 ; + val[0 ] =ARM_SMCCC_VERSION_1_1 ;break ;case ARM_SMCCC_ARCH_FEATURES_FUNC_ID : feature =smccc_get_arg1 (vcpu);
@@ -29,10 +29,10 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case KVM_BP_HARDEN_UNKNOWN :break ;case KVM_BP_HARDEN_WA_NEEDED : - val =SMCCC_RET_SUCCESS ; + val[0 ] =SMCCC_RET_SUCCESS ;break ;case KVM_BP_HARDEN_NOT_REQUIRED : - val =SMCCC_RET_NOT_REQUIRED ; + val[0 ] =SMCCC_RET_NOT_REQUIRED ;break ; }break ;
@@ -42,26 +42,35 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case KVM_SSBD_UNKNOWN :break ;case KVM_SSBD_KERNEL : - val =SMCCC_RET_SUCCESS ; + val[0 ] =SMCCC_RET_SUCCESS ;break ;case KVM_SSBD_FORCE_ENABLE :case KVM_SSBD_MITIGATED : - val =SMCCC_RET_NOT_REQUIRED ; + val[0 ] =SMCCC_RET_NOT_REQUIRED ;break ; }break ;case ARM_SMCCC_HV_PV_TIME_FEATURES : - val =SMCCC_RET_SUCCESS ; + val[0 ] =SMCCC_RET_SUCCESS ;break ; }break ;case ARM_SMCCC_HV_PV_TIME_FEATURES : - val =kvm_hypercall_pv_features (vcpu); + val[0 ] =kvm_hypercall_pv_features (vcpu);break ;case ARM_SMCCC_HV_PV_TIME_ST : gpa =kvm_init_stolen_time (vcpu);if (gpa !=GPA_INVALID ) - val = gpa; + val[0 ] = gpa; +break ; +case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID : + val[0 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0 ; + val[1 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1 ; + val[2 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 ; + val[3 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3 ; +break ; +case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID : + val[0 ] =BIT (ARM_SMCCC_KVM_FUNC_FEATURES );break ;case SDEI_1_0_FN_SDEI_VERSION :case SDEI_1_0_FN_SDEI_EVENT_REGISTER :
@@ -85,6 +94,6 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)return kvm_psci_call (vcpu); } -smccc_set_retval (vcpu, val,0 ,0 ,0 ); +smccc_set_retval (vcpu, val[0 ], val[1 ], val[2 ], val[3 ]);return 1 ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH 3/6 kvm/arm64: Export kvm_handle_user_mem_abort() with prefault mode
In reply to Gavin ShanThis renames user_mem_abort() to kvm_handle_user_mem_abort(), and then exports it. The function will be used in asynchronous page fault to populate a page table entry once the corresponding page is populated from the backup device (e.g. swap partition):
* Parameter @fault_status is replace by @esr.
* Parameter @prefault is added
As the @esr is passed as parameter, not fetched from vCPU struct. This also introduces the necessasry helpers in esr.h, to manupulate the @esr. The helpers defined in kvm_emulate.h reuses the newly added helper. This shouldn't cause functional changes.
--- arch/arm64/include/asm/esr.h | 5 +++++ arch/arm64/include/asm/kvm_emulate.h | 8 ++++---- arch/arm64/include/asm/kvm_host.h | 4 ++++ arch/arm64/kvm/mmu.c | 18 ++++++++++++------ 4 files changed, 25 insertions(+), 10 deletions(-)
arch/arm64/include/asm/esr.h +5 -0
@@ -317,8 +317,13 @@ ESR_ELx_CP15_32_ISS_DIR_READ)#ifndef __ASSEMBLY__ +#include <linux/bitfield.h> #include <asm/types.h> +#define esr_get_trap_class (esr) FIELD_GET(ESR_ELx_EC_MASK, esr) +#define esr_is_dabt_wnr (esr) !!(FIELD_GET(ESR_ELx_WNR, esr)) +#define esr_is_dabt_iss1tw (esr) !!(FIELD_GET(ESR_ELx_S1PTW, esr)) +static inline bool esr_is_data_abort (u32 esr) {const u32 ec =ESR_ELx_EC (esr);
arch/arm64/include/asm/kvm_emulate.h +4 -4
@@ -323,13 +323,13 @@ static __always_inline int kvm_vcpu_dabt_get_rd(const struct kvm_vcpu *vcpu) static __always_inline bool kvm_vcpu_dabt_iss1tw(const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_S1PTW); +return esr_is_dabt_iss1tw (kvm_vcpu_get_hsr (vcpu)); } static __always_inline bool kvm_vcpu_dabt_iswrite(const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_WNR) || -kvm_vcpu_dabt_iss1tw (vcpu);/* AF/DBM update */ +return (esr_is_dabt_wnr (kvm_vcpu_get_hsr (vcpu)) || +kvm_vcpu_dabt_iss1tw (vcpu));/* AF/DBM update */ }static inline bool kvm_vcpu_dabt_is_cm (const struct kvm_vcpu *vcpu)
@@ -350,7 +350,7 @@ static __always_inline bool kvm_vcpu_trap_il_is32bit(const struct kvm_vcpu *vcpu static __always_inline u8 kvm_vcpu_trap_get_class(const struct kvm_vcpu *vcpu) { -return ESR_ELx_EC (kvm_vcpu_get_hsr (vcpu)); +return esr_get_trap_class (kvm_vcpu_get_hsr (vcpu)); }static inline bool kvm_vcpu_trap_is_iabt (const struct kvm_vcpu *vcpu)
arch/arm64/include/asm/kvm_host.h +4 -0
@@ -441,6 +441,10 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu, struct kvm_vcpu_events *events);#define KVM_ARCH_WANT_MMU_NOTIFIER +int kvm_handle_user_mem_abort (struct kvm_vcpu *vcpu,phys_addr_t fault_ipa, +struct kvm_memory_slot *memslot, +unsigned long hva,unsigned int esr, +bool prefault);int kvm_unmap_hva_range (struct kvm *kvm,unsigned long start,unsigned long end);int kvm_set_spte_hva (struct kvm *kvm,unsigned long hva,pte_t pte);
arch/arm64/kvm/mmu.c +12 -6
@@ -1787,11 +1787,15 @@ transparent_hugepage_adjust(struct kvm_memory_slot *memslot,return PAGE_SIZE ; } -static int user_mem_abort (struct kvm_vcpu *vcpu,phys_addr_t fault_ipa, -struct kvm_memory_slot *memslot,unsigned long hva, -unsigned long fault_status) +int kvm_handle_user_mem_abort (struct kvm_vcpu *vcpu, +phys_addr_t fault_ipa, +struct kvm_memory_slot *memslot, +unsigned long hva, +unsigned int esr, +bool prefault) {int ret; +unsigned int fault_status = (esr & ESR_ELx_FSC_TYPE);bool write_fault, writable, force_pte = false;bool exec_fault, needs_exec;unsigned long mmu_seq;
@@ -1805,8 +1809,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,bool logging_active =memslot_is_logging (memslot);unsigned long vma_pagesize, flags =0 ; - write_fault =kvm_is_write_fault (vcpu); - exec_fault =kvm_vcpu_trap_is_iabt (vcpu); + exec_fault = (esr_get_trap_class (esr) == ESR_ELx_EC_IABT_LOW); + write_fault = (!exec_fault && + (esr_is_dabt_wnr (esr) ||esr_is_dabt_iss1tw (esr)));VM_BUG_ON (write_fault && exec_fault);if (fault_status ==FSC_PERM && !write_fault && !exec_fault) {
@@ -2116,7 +2121,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) goto out_unlock; } - ret =user_mem_abort (vcpu, fault_ipa, memslot, hva, fault_status); + ret =kvm_handle_user_mem_abort (vcpu, fault_ipa, memslot, hva, +kvm_vcpu_get_hsr (vcpu), false);if (ret ==0 ) ret =1 ; out:
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 3/6] kvm/arm64: Export kvm_handle_user_mem_abort() with prefault mode
Hi Gavin,
Quoted 10 lines · Gavin Shan
This renames
user_mem_abort()tokvm_handle_user_mem_abort(), and then exports it. The function will be used in asynchronous page fault to populate a page table entry once the corresponding page is populated from the backup device (e.g. swap partition):* Parameter @fault_status is replace by @esr.
* Parameter @prefault is addedAs the @esr is passed as parameter, not fetched from vCPU struct. This also introduces the necessasry helpers in
esr.h, to manupulate the @esr.
(Nit: necessary, manipulate)
Quoted 2 lines
The helpers defined in
kvm_emulate.hreuses the newly added helper. This shouldn't cause functional changes.
user_mem_abort() is deep in the the guts of KVM's arch code. I don't think this should be exported. It must be called on the vcpu thread. It must be called under the VMs srcu lock. There are also tricky interactions with whether the vcpu is loaded on this cpu or not...
I think it would be much simpler to always let the guest take the stage2-fault a second time. This keeps the property that pages are only populate in response to a stage2 fault. If the guest behaves, it will only schedule a task that accesses the page once its available.
Thanks,
James
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 3/6] kvm/arm64: Export kvm_handle_user_mem_abort() with prefault mode
Hi James,
Quoted 6 lines · James Morse
Quoted 5 lines · Gavin Shan
This renames
user_mem_abort()tokvm_handle_user_mem_abort(), and then exports it. The function will be used in asynchronous page fault to populate a page table entry once the corresponding page is populated from the backup device (e.g. swap partition):
Quoted 8 lines
Quoted 5 lines
* Parameter @fault_status is replace by @esr.
* Parameter @prefault is addedAs the @esr is passed as parameter, not fetched from vCPU struct. This also introduces the necessasry helpers in
esr.h, to manupulate the @esr.(Nit: necessary, manipulate)
Thanks for your comments. It will be fixed in next revision :)
Quoted 4 lines
Quoted 3 lines
The helpers defined in
kvm_emulate.hreuses the newly added helper. This shouldn't cause functional changes.
Quoted 9 lines
user_mem_abort()is deep in the the guts of KVM's arch code. I don't think this should be exported. It must be called on the vcpu thread. It must be called under the VMs srcu lock. There are also tricky interactions with whether the vcpu is loaded on this cpu or not...I think it would be much simpler to always let the guest take the stage2-fault a second time. This keeps the property that pages are only populate in response to a stage2 fault. If the guest behaves, it will only schedule a task that accesses the page once its available.
The function is called with SRCU hold in this patchset. However, it seems more reasonable to be called in vCPU thread. On the other hand, I'm worrying about the performance degradation. I will evaluate it in next revision and leave the function is called in vCPU thread if performance is acceptable. Otherwise, I would like to leave it as of being :)
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH 4/6 arm64/kvm: Support async page fault
In reply to Gavin ShanThere are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.
In order to resolve the issue and improve the guest's performance, this introduces the feature (asychronous page fault). A page-not-present notification is fired to guest if the requested page isn't ready so that the guest can reschedule current process and pick another process to run. Another page-ready notification is sent to guest so that the process, which was rescheduled previously, can be picked up to run. With this, the guest needn't necessarily suspend itself on stage two page fault. These two notificatons are associated by so-called vCPU scoped tokens, which is the combination of incremental sequence number and vCPU index (@vcpu->vcpu_idx). Besides, the notifications are conveyed by SDEI and PPI, whose numbers are specified by guest through SMCCC interface.
The asynchronous page fault always starts with a background woker if current vCPU has enough token space and no pending notifications. After the worker, which populates the requested page background, is started, a page-not-present notification, together with an unique token, are sent to guest through the specified private SDEI event. The notification is acknowledged by clearing the shared cache (@vcpu->apf.cache). The current process is marked for waiting completion of asynchronous page fault. A subsequent (reschedule) IPI is sent to current vCPU so that the current process can be reschedule and suspended until the asynchronous page is completed. In the meanwhile, the woker populates the requested page and it's queued to the completion queue once the task is completed. At this point or the acknoledgement (by SMCCC) on pending PPI, a PPI is sent to guest for page-ready notification. The guest uses the associated token to locate the suspended process and wakes it up.
In order to fulfil the functions, there are several SMCCC function IDs introduced and outlined as below:
ARM_SMCCC_KVM_FUNC_APF_VERSION
Returns the version, which can be used to identify ABI changes in the future.
ARM_SMCCC_KVM_FUNC_APF_ENABLE
Enable or disable asynchronous page fault on current vCPU.
ARM_SMCCC_KVM_FUNC_APF_TOKEN_NUM
Returns maximal number of tokens that current vCPU can have. It's used by guest to allocate the required resources.
ARM_SMCCC_KVM_FUNC_APF_SDEI
ARM_SMCCC_KVM_FUNC_APF_IRQ
Used by guest to confiugre the SDEI event and PPI numbers.
ARM_SMCCC_KVM_FUNC_APF_IRQ_ACK
Acknowledge the page-ready notificaton in the PPI handler.
--- arch/arm64/include/asm/kvm_host.h | 50 +++ arch/arm64/include/asm/kvm_para.h | 27 ++ arch/arm64/include/uapi/asm/Kbuild | 2 - arch/arm64/include/uapi/asm/kvm_para.h | 23 ++ arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/arm.c | 45 ++- arch/arm64/kvm/async_pf.c | 462 +++++++++++++++++++++++++ arch/arm64/kvm/hypercalls.c | 8 + arch/arm64/kvm/mmu.c | 29 +- arch/arm64/kvm/sdei.c | 8 + include/linux/arm-smccc.h | 15 + 12 files changed, 665 insertions(+), 6 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_para.h create mode 100644 arch/arm64/include/uapi/asm/kvm_para.h create mode 100644 arch/arm64/kvm/async_pf.c
arch/arm64/include/asm/kvm_host.h +50 -0
@@ -46,6 +46,7 @@#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4)#define KVM_REQ_SDEI KVM_ARCH_REQ(5) +#define KVM_REQ_APF_READY KVM_ARCH_REQ(6)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)
@@ -239,6 +240,23 @@ struct vcpu_reset_state {bool reset; }; +#ifdef CONFIG_KVM_ASYNC_PF + +/* Should be a power of two number */ +#define ASYNC_PF_PER_VCPU 64 + +/* +* The association of gfn and token. The token will be sent to guest as +* page fault address. Also, the guest could be in aarch32 mode. So its +* length should be 32-bits. +*/ +struct kvm_arch_async_pf { +u32 token; +gfn_t gfn; +u32 esr; +}; +#endif /* CONFIG_KVM_ASYNC_PF */ +struct kvm_vcpu_arch {struct kvm_cpu_context ctxt;void *sve_state;
@@ -346,6 +364,22 @@ struct kvm_vcpu_arch {struct kvm_sdei_vcpu_event *sdei_normal_event;struct kvm_sdei_vcpu_event *sdei_critical_event;struct list_head sdei_events; + +#ifdef CONFIG_KVM_ASYNC_PF +struct { +struct gfn_to_hva_cache cache; +gfn_t gfns[ASYNC_PF_PER_VCPU ]; +u64 control_block; +bool send_user_only; +u64 sdei_event_num; +u32 irq; + +u16 id; +bool notpresent_pending; +u32 notpresent_token; +bool pageready_pending; + } apf; +#endif };/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
@@ -566,6 +600,22 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,int kvm_arm_vcpu_arch_has_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); +#ifdef CONFIG_KVM_ASYNC_PF +bool kvm_async_pf_hash_find (struct kvm_vcpu *vcpu,gfn_t gfn); +bool kvm_arch_async_not_present_allowed (struct kvm_vcpu *vcpu); +int kvm_arch_setup_async_pf (struct kvm_vcpu *vcpu,u32 esr,gpa_t gpa, +gfn_t gfn); +bool kvm_arch_async_page_not_present (struct kvm_vcpu *vcpu, +struct kvm_async_pf *work); +void kvm_arch_async_page_present_queued (struct kvm_vcpu *vcpu); +bool kvm_arch_can_dequeue_async_page_present (struct kvm_vcpu *vcpu); +void kvm_arch_async_page_ready (struct kvm_vcpu *vcpu, +struct kvm_async_pf *work); +void kvm_arch_async_page_present (struct kvm_vcpu *vcpu, +struct kvm_async_pf *work); +long kvm_async_pf_hypercall (struct kvm_vcpu *vcpu); +#endif /* CONFIG_KVM_ASYNC_PF */ +/* Guest/host FPSIMD coordination helpers */ int kvm_arch_vcpu_run_map_fp (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_load_fp (struct kvm_vcpu *vcpu);
arch/arm64/include/asm/kvm_para.h new file +27 -0
@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM_KVM_PARA_H +#define _ASM_ARM_KVM_PARA_H + +#include <uapi/asm/kvm_para.h> + +static inline bool kvm_check_and_clear_guest_paused (void ) +{ +return false; +} + +static inline unsigned int kvm_arch_para_features (void ) +{ +return 0 ; +} + +static inline unsigned int kvm_arch_para_hints (void ) +{ +return 0 ; +} + +static inline bool kvm_para_available (void ) +{ +return false; +} + +#endif /* _ASM_ARM_KVM_PARA_H */
arch/arm64/include/uapi/asm/Kbuild +0 -2
@@ -1,3 +1 @@ # SPDX-License-Identifier: GPL-2.0 - -generic-y += kvm_para.h
arch/arm64/include/uapi/asm/kvm_para.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _UAPI_ASM_ARM_KVM_PARA_H +#define _UAPI_ASM_ARM_KVM_PARA_H + +#include <linux/types.h> + +#define KVM_FEATURE_ASYNC_PF 0 + +/* Async PF */ +#define KVM_ASYNC_PF_ENABLED (1 << 0) +#define KVM_ASYNC_PF_SEND_ALWAYS (1 << 1) + +#define KVM_PV_REASON_PAGE_NOT_PRESENT 1 +#define KVM_PV_REASON_PAGE_READY 2 + +struct kvm_vcpu_pv_apf_data { +__u32 reason; +__u32 token; +__u8 pad[56 ]; +__u32 enabled; +}; + +#endif /* _UAPI_ASM_ARM_KVM_PARA_H */
arch/arm64/kvm/Kconfig +1 -0
@@ -32,6 +32,7 @@ menuconfig KVM select KVM_VFIO select HAVE_KVM_EVENTFD select HAVE_KVM_IRQFD + select KVM_ASYNC_PF select HAVE_KVM_MSI select HAVE_KVM_IRQCHIP select HAVE_KVM_IRQ_ROUTING
arch/arm64/kvm/Makefile +1 -0
@@ -25,3 +25,4 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ vgic/vgic-its.o vgic/vgic-debug.o kvm-$(CONFIG_KVM_ARM_PMU) += pmu-emul.o +kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o async_pf.o
arch/arm64/kvm/arm.c +42 -3
75 lines not on this page · raw message
arch/arm64/kvm/async_pf.c new file +462 -0
462 lines not on this page · raw message
arch/arm64/kvm/hypercalls.c +8 -0
@@ -71,7 +71,15 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)break ;case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID : val[0 ] =BIT (ARM_SMCCC_KVM_FUNC_FEATURES ); +#ifdef CONFIG_KVM_ASYNC_PF + val[0 ] |=BIT (ARM_SMCCC_KVM_FUNC_APF ); +#endif break ; +#ifdef CONFIG_KVM_ASYNC_PF +case ARM_SMCCC_VENDOR_HYP_KVM_APF_FUNC_ID : + val[0 ] =kvm_async_pf_hypercall (vcpu); +break ; +#endif case SDEI_1_0_FN_SDEI_VERSION :case SDEI_1_0_FN_SDEI_EVENT_REGISTER :case SDEI_1_0_FN_SDEI_EVENT_ENABLE :
arch/arm64/kvm/mmu.c +28 -1
@@ -1787,6 +1787,30 @@ transparent_hugepage_adjust(struct kvm_memory_slot *memslot,return PAGE_SIZE ; } +static bool try_async_pf (struct kvm_vcpu *vcpu,u32 esr,gpa_t gpa, +gfn_t gfn,kvm_pfn_t *pfn,bool write, +bool *writable,bool prefault) +{ +struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); +#ifdef CONFIG_KVM_ASYNC_PF +bool async = false; + +/* Bail if *pfn has correct page */ + *pfn =__gfn_to_pfn_memslot (slot, gfn, false, &async, write, writable); +if (!async) +return false; + +if (!prefault &&kvm_arch_async_not_present_allowed (vcpu)) { +if (kvm_async_pf_hash_find (vcpu, gfn) || +kvm_arch_setup_async_pf (vcpu, esr, gpa, gfn)) +return true; + } +#endif + + *pfn =__gfn_to_pfn_memslot (slot, gfn, false,NULL , write, writable); +return false; +} + int kvm_handle_user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, struct kvm_memory_slot *memslot,
@@ -1870,7 +1894,10 @@ int kvm_handle_user_mem_abort(struct kvm_vcpu *vcpu,*/ smp_rmb (); - pfn =gfn_to_pfn_prot (kvm, gfn, write_fault, &writable); +if (try_async_pf (vcpu, esr, fault_ipa, gfn, &pfn, + write_fault, &writable, prefault)) +return 1 ; +if (pfn ==KVM_PFN_ERR_HWPOISON ) {kvm_send_hwpoison_signal (hva, vma_shift);return 0 ;
arch/arm64/kvm/sdei.c +8 -0
@@ -22,6 +22,14 @@ static struct kvm_sdei_priv kvm_sdei_privs[] = { 0, NULL }, + {0x40200001 ,/* Asynchronous page fault */ +SDEI_EVENT_TYPE_PRIVATE , +1 , +SDEI_EVENT_PRIORITY_CRITICAL , +SDEI_EVENT_REGISTER_RM_ANY , +0 , +NULL + }, };#ifdef CONFIG_ARM_SDE_INTERFACE
include/linux/arm-smccc.h +15 -0
@@ -116,6 +116,7 @@/* KVM "vendor specific" services */ #define ARM_SMCCC_KVM_FUNC_FEATURES 0 +#define ARM_SMCCC_KVM_FUNC_APF 1#define ARM_SMCCC_KVM_FUNC_FEATURES_2 127#define ARM_SMCCC_KVM_NUM_FUNCS 128
@@ -125,6 +126,20 @@ ARM_SMCCC_OWNER_VENDOR_HYP, \ ARM_SMCCC_KVM_FUNC_FEATURES) +/* Async page fault service */ +#define ARM_SMCCC_KVM_FUNC_APF_VERSION 0 +#define ARM_SMCCC_KVM_FUNC_APF_ENABLE 1 +#define ARM_SMCCC_KVM_FUNC_APF_TOKEN_NUM 2 +#define ARM_SMCCC_KVM_FUNC_APF_SDEI 3 +#define ARM_SMCCC_KVM_FUNC_APF_IRQ 4 +#define ARM_SMCCC_KVM_FUNC_APF_IRQ_ACK 5 + +#define ARM_SMCCC_VENDOR_HYP_KVM_APF_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + ARM_SMCCC_OWNER_VENDOR_HYP, \ + ARM_SMCCC_KVM_FUNC_APF) +/* * Return codes defined in ARM DEN 0070A * ARM DEN 0070A is now merged/consolidated into ARM DEN 0028 C
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 4/6] arm64/kvm: Support async page fault
Hi Gavin,
[my mail client went a bit nuts - it thinks this got sent already, sorry if you received it twice!]
I only got so far through this, so may have focussed on the wrong things. This patch has too many things going on. Please split it up.
I think the page-fault and page-present should be done separately, with the gubbins the core-code needs added first. The cap and corresponding documentation should be added last so the whole thing bisects.
It looks like the guest and hypervisor parts are already split up, this is helpful, thanks. Please ensure that the descriptions are clear on this distinction too. Its not at all clear where where the 'reschedule ipi' comes from in this commit message.
I'd really like it if as much of the guest parts as possible lived in common code. There is nothing arch-specific about an interrupt and reading from shared memory.
Quoted 7 lines · Gavin Shan
There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.
Yes, its the cost of abstracting the machine's resources and invisibly carving them up.
Quoted 1 line
In order to resolve the issue and improve the guest's performance,
Why is it an issue? You've missed the important part of the commit message: why we should make this change.
This needs to mention postcopy live migration, which is the reason for doing any of this.
I don't think "I overcommited and swapped the guest to disk" is at all compelling. (as the doctor says: don't do that then)
Quoted 1 line
this introduces the feature (asychronous page fault). A page-not-present
(asynchronous)
Quoted 6 lines
notification is fired to guest if the requested page isn't ready so that the guest can reschedule current process and pick another process to run. Another page-ready notification is sent to guest so that the process, which was rescheduled previously, can be picked up to run. With this,
the guest needn't necessarily suspend itself on stage two page fault.
Be suspended by the hypervisor until the resources are available. Normally the guest has no idea this is happening.
Quoted 1 line
These two notificatons are associated by so-called vCPU scoped tokens,
(notifications)
Quoted 3 lines
which is the combination of incremental sequence number and vCPU index (@vcpu->vcpu_idx). Besides, the notifications are conveyed by SDEI and PPI, whose numbers are specified by guest through SMCCC interface.
... so you don't need the ACPI table?
The 'asynchrononous page fault' notification obviously needs to be (er,) synchronous, so it has to be per-cpu.
Why is the page-ready IRQ a PPI? We have a limited number of these, so shouldn't use one unless it _needs_ to be per-cpu.
'vcpu index' should probably be the MPIDR_EL1, which is something both the host and guest already know that identifies the vcpu. Inventing something new isn't necessary.
Quoted 1 line
The asynchronous page fault always starts with a background woker if
(worker. Please run your commit-messages through a spell checker)
Quoted 2 lines
current vCPU has enough token space and no pending notifications. After the worker, which populates the requested page background, is started,
what does 'requested page background' mean? (missing the words 'in the'?)
Quoted 9 lines
a page-not-present notification, together with an unique token, are sent to guest through the specified private SDEI event. The notification is acknowledged by clearing the shared cache (@vcpu->apf.cache). The current process is marked for waiting completion of asynchronous page fault. A subsequent (reschedule) IPI is sent to current vCPU so that the current process can be reschedule and suspended until the asynchronous page is completed. In the meanwhile, the woker populates the requested page and it's queued to the completion queue once the task is completed.
Quoted 3 lines
At this
point or the acknoledgement (by SMCCC) on pending PPI, a PPI is sent to guest for page-ready notification.
A PPI is made pending, when the PPI is pending?
Quoted 2 lines
The guest uses the associated token
to locate the suspended process and wakes it up.
I hope the 'token' is nothing more than the IPA.
Quoted 2 lines
In order to fulfil the functions, there are several SMCCC function IDs introduced and outlined as below:
Please add a documentation file for these for the benefit of other guests (e.g. kvm-unit-test) describing these in more detail, their parameters, the expected lifecycle etc.
Quoted 37 lines
ARM_SMCCC_KVM_FUNC_APF_VERSION
Returns the version, which can be used to identify ABI changes in the future.
ARM_SMCCC_KVM_FUNC_APF_ENABLE
Enable or disable asynchronous page fault on current vCPU.
ARM_SMCCC_KVM_FUNC_APF_TOKEN_NUM
Returns maximal number of tokens that current vCPU can have. It's used by guest to allocate the required resources.
ARM_SMCCC_KVM_FUNC_APF_SDEI
ARM_SMCCC_KVM_FUNC_APF_IRQ
Used by guest to confiugre the SDEI event and PPI numbers.
ARM_SMCCC_KVM_FUNC_APF_IRQ_ACK
Acknowledge the page-ready notificaton in the PPI handler.arch/arm64/include/asm/kvm_host.h +7
-0@@ -46,6 +46,7 @@#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4)#define KVM_REQ_SDEI KVM_ARCH_REQ(5) +#define KVM_REQ_APF_READY KVM_ARCH_REQ(6)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)@@ -239,6 +240,23 @@ struct vcpu_reset_state {bool reset; }; +#ifdef CONFIG_KVM_ASYNC_PF + +/* Should be a power of two number */ +#define ASYNC_PF_PER_VCPU 64 + +/*
arch/arm64/include/asm/kvm_host.h struct vcpu_reset_state { · line 258
Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +10
-0+* The association of gfn and token. The token will be sent to guest as +* page fault address. Also, the guest could be in aarch32 mode. So its +* length should be 32-bits. +*/ +struct kvm_arch_async_pf { +u32 token; +gfn_t gfn; +u32 esr; +}; +#endif /* CONFIG_KVM_ASYNC_PF */
#ifdef-ing a struct makes it harder to use IS_ENABLED() in the C code, which in turn makes it harder to get complete coverage from the CI builders without aligning the stars to hit the right randconfig combination.
Please avoid doing this unless its absolutely necessary. (its not like a struct definition takes up any space in the binary)
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 368
Quoted 9 lines
arch/arm64/include/asm/kvm_host.h +2
-0struct kvm_vcpu_arch {struct kvm_cpu_context ctxt;void *sve_state;@@ -346,6 +364,22 @@ struct kvm_vcpu_arch {struct kvm_sdei_vcpu_event *sdei_normal_event;struct kvm_sdei_vcpu_event *sdei_critical_event;struct list_head sdei_events; + +#ifdef CONFIG_KVM_ASYNC_PF
See above. Any reason these can't be dynamically allocated if the feature is enabled by the VMM? (You're using a PPI, this has to be allocated by the VMM).
Not all guests would use this, so its a waste of space if its enabled but not in use.
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch {
Quoted 7 lines
arch/arm64/include/asm/kvm_host.h +5
-0+struct { +struct gfn_to_hva_cache cache; +gfn_t gfns[ASYNC_PF_PER_VCPU ]; +u64 control_block; +bool send_user_only;
Hmmm...
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch {
Quoted 4 lines
arch/arm64/include/asm/kvm_host.h +3
-0+u64 sdei_event_num; +u32 irq; +u16 id;
a per-vcpu id? KVM already has one, and the architecture has one that the guest can see too.
Quoted 16 lines
arch/arm64/include/asm/kvm_host.h +5
-0+bool notpresent_pending; +u32 notpresent_token; + bool pageready_pending; + } apf; +#endif };/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ arch/arm64/include/asm/kvm_para.h new file +0
-0
arch/arm64/include/asm/kvm_para.h line 10
Quoted 10 lines
arch/arm64/include/asm/kvm_para.h new file +8
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM_KVM_PARA_H +#define _ASM_ARM_KVM_PARA_H +#include <uapi/asm/kvm_para.h> +static inline bool kvm_check_and_clear_guest_paused (void ) +{ +return false; +}
This is part of kvmclock? aarch64 has an architecture specific way of doing that stuff. Why is this here?!
(if you've turned something on that needs these stubs, its probably worth doing that in an preparation patch so you can explain why, instead of hiding it in here)
Quoted 24 lines
arch/arm64/include/asm/kvm_para.h new file +16
-0+static inline unsigned int kvm_arch_para_features (void ) +{ +return 0 ; +} + +static inline unsigned int kvm_arch_para_hints (void ) +{ +return 0 ; +} + +static inline bool kvm_para_available (void ) +{ +return false; +} + +#endif /* _ASM_ARM_KVM_PARA_H */ arch/arm64/include/uapi/asm/Kbuild +0
-0
arch/arm64/include/uapi/asm/Kbuild line 3
Quoted 3 lines
arch/arm64/include/uapi/asm/Kbuild +0
-2@@ -1,3 +1 @@ # SPDX-License-Identifier: GPL-2.0 - -generic-y += kvm_para.h
The arch specific version is practically the same as the asm-generic one.
Please explain why you need to do this in a preparatory patch.
Quoted 19 lines
arch/arm64/include/uapi/asm/kvm_para.h new file +13
-0@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _UAPI_ASM_ARM_KVM_PARA_H +#define _UAPI_ASM_ARM_KVM_PARA_H + +#include <linux/types.h> + +#define KVM_FEATURE_ASYNC_PF 0 + +/* Async PF */ +#define KVM_ASYNC_PF_ENABLED (1 << 0) +#define KVM_ASYNC_PF_SEND_ALWAYS (1 << 1) + +#define KVM_PV_REASON_PAGE_NOT_PRESENT 1
arch/arm64/include/uapi/asm/kvm_para.h line 23
Quoted 10 lines
arch/arm64/include/uapi/asm/kvm_para.h new file +10
-0+#define KVM_PV_REASON_PAGE_READY 2 + +struct kvm_vcpu_pv_apf_data { +__u32 reason; +__u32 token; +__u8 pad[56 ]; +__u32 enabled; +}; + +#endif /* _UAPI_ASM_ARM_KVM_PARA_H */
Presumably this is the reason to carry around an extra copy of kvm_para.h.
Wouldn't it be better to unpick that dependency instead?
Quoted 18 lines
arch/arm64/kvm/arm.c +6
-0@@ -225,6 +225,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)*/ r =1 ;break ; +#ifdef CONFIG_KVM_ASYNC_PF +case KVM_CAP_ASYNC_PF : +case KVM_CAP_ASYNC_PF_INT : + r =1 ; +break ; +#endif default : r =kvm_arch_vm_ioctl_check_extension (kvm, ext);break ;
arch/arm64/kvm/arm.c int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · line 285
Quoted 6 lines
arch/arm64/kvm/arm.c +3
-0@@ -274,6 +280,9 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)bitmap_zero (vcpu->arch.features,KVM_VCPU_MAX_FEATURES ); vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; +#ifdef CONFIG_KVM_ASYNC_PF + vcpu->arch.apf.control_block =0UL ; +#endif
kvm_vm_ioctl_create_vcpu() allocates vcpu with kmem_cache_zalloc(), so this field will already be zero.
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 436
Quoted 6 lines
arch/arm64/kvm/arm.c +0
-2@@ -432,8 +441,32 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,int kvm_arch_vcpu_runnable (struct kvm_vcpu *v) {bool irq_lines = *vcpu_hcr (v) & (HCR_VI |HCR_VF ); -return ((irq_lines ||kvm_vgic_vcpu_pending_irq (v)) - && !v->arch.power_off && !v->arch.pause);
As you are adding a conditional __copy_from_user() in here, please add an unconditional might_sleep() at the top of this function. This lets us catch locations that will sometimes break the first time they happen.
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 449
Quoted 5 lines
arch/arm64/kvm/arm.c +5
-0+if ((irq_lines ||kvm_vgic_vcpu_pending_irq (v)) && + !v->arch.power_off && !v->arch.pause) +return true; + +#ifdef CONFIG_KVM_ASYNC_PF
Please use IS_ENABLED() here, #ifdeffery make this stuff harder to read and harder to build-test.
(if you can come up with a readable IS_ENABLED() way of returning the _check_extension() result - even better!)
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 458
Quoted 9 lines
arch/arm64/kvm/arm.c +8
-0+if (v->arch.apf.control_block &KVM_ASYNC_PF_ENABLED ) { +u32 val; +int ret; + +if (!list_empty_careful (&v->async_pf.done)) +return true; + ret =kvm_read_guest_offset_cached (v->kvm, &v->arch.apf.cache, + &val,0 ,sizeof (val));
You added a helper for this: kvm_async_pf_read_cache()...
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 465
Quoted 7 lines
arch/arm64/kvm/arm.c +7
-0+if (ret || val) +return true; + + ret =kvm_read_guest_offset_cached (v->kvm, &v->arch.apf.cache, + &val,4 ,sizeof (val)); +if (ret || val) +return true;
These 0 and 4 really need some kind of name to give a hint as to what they are for.
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 465
Quoted 1 line
arch/arm64/kvm/arm.c +1
-0+ }
This really ought to wrapped up in a helper that gives a hint as to what it is doing.
There are two magic locations in guest memory that means the guest controls whether it is runnable or not... this sounds like a source of hard-to-debug issues.
I'd much prefer it if the guest only ever got blocked in the stage2 fault handler, as it does today. This mechanism just provides information about what is going on at stage2, which allows the guest to avoid getting blocked if it can do something about it.
Giving the guest access to the controls doesn't look right.
Quoted 7 lines
arch/arm64/kvm/arm.c +3
-0+#endif + +return false; }bool kvm_arch_vcpu_in_kernel (struct kvm_vcpu *vcpu)
arch/arm64/kvm/arm.c static void check_vcpu_requests(struct kvm_vcpu *vcpu) · line 663
Quoted 7 lines
arch/arm64/kvm/arm.c +4
-0@@ -624,6 +657,11 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_VCPU_RESET , vcpu))kvm_reset_vcpu (vcpu); +#ifdef CONFIG_KVM_ASYNC_PF +if (kvm_check_request (KVM_REQ_APF_READY , vcpu)) +kvm_check_async_pf_completion (vcpu); +#endif
(IS_ENABLED() would save you the link error ... providing a stub in core-code for kvm_check_async_pf_completion() would let you rely on this request never being made instead)
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · line 782
Quoted 11 lines
arch/arm64/kvm/arm.c +2
-1if (kvm_check_request (KVM_REQ_SDEI , vcpu))kvm_sdei_deliver (vcpu);@@ -739,7 +777,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)smp_store_mb (vcpu->mode,IN_GUEST_MODE );if (ret <=0 ||need_new_vmid_gen (&vcpu->kvm->arch.vmid) || -kvm_request_pending (vcpu)) { + (kvm_request_pending (vcpu) && +READ_ONCE (vcpu->requests) != (1UL <<KVM_REQ_APF_READY ))) {
A double READ_ONCE() looks wrong. If you have to do this, presumably you didn't want a vcpu->request to begin with.
It looks like KVM_REQ_APF_READY only exists to move the page-ready work onto the vcpu thread because the vcpu got blocked further out by kvm_arch_vcpu_runnable(), and the second half of the stage2-fault now needs doing.
I think it would be simpler to only ever block in the stage2 fault handler. The guest can only ever be blocked on one stage2 fault at a time, so this abstraction to let it sleep for any number is pointless. If I'm waiting for ipa-4, and get restarted because ipa-6 is now available ... I'm going to fault on ipa-4 for a second time. There is no point restarting the guest until it can make progress again.
From the first stage2 fault you can kick off the page-fault notification (which must be synchronous, so can be notified without taking this fault again). If the vcpu takes a fault that was already notified, it clearly can't handle the page-fault notification and must be blocked until that page is available.
We should be able to depend on the regular wakeup from mm when the vcpu is blocked. The extra information from page-present lets the guest know it can now touch these pages without getting blocked ... they shouldn't need a wakeup.
Quoted 111 lines
arch/arm64/kvm/arm.c +0
-0vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */ arch/arm64/kvm/arm.c +0
-0kvm_pmu_sync_hwstate (vcpu);arch/arm64/kvm/async_pf.c new file +98
-0@@ -0,0 +1,462 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright Gavin Shan, Redhat Inc 2020. +* +* Asynchronous page fault support +*/ + +#include <linux/arm-smccc.h> +#include <linux/kvm_host.h> +#include <asm/kvm_emulate.h> +#include <kvm/arm_hypercalls.h> +#include <kvm/arm_vgic.h> +#include <asm/kvm_sdei.h> +static inline u32 kvm_async_pf_hash_fn (gfn_t gfn) +{ +return hash_32 (gfn &0xffffffff ,order_base_2 (ASYNC_PF_PER_VCPU )); +} + +static inline u32 kvm_async_pf_hash_next (u32 key) +{ +return (key +1 ) & (ASYNC_PF_PER_VCPU -1 ); +} + +static inline void kvm_async_pf_hash_reset (struct kvm_vcpu *vcpu) +{ +int i; + +for (i =0 ; i <ASYNC_PF_PER_VCPU ; i++) + vcpu->arch.apf.gfns[i] = ~0 ; +} +/* +* Add gfn to the hash table. It's ensured there is a free entry +* when this function is called. +*/ +static void kvm_async_pf_hash_add (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +u32 key =kvm_async_pf_hash_fn (gfn); + +while (vcpu->arch.apf.gfns[key] != ~0 ) + key =kvm_async_pf_hash_next (key); + + vcpu->arch.apf.gfns[key] = gfn; +} + +static u32 kvm_async_pf_hash_slot (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +u32 key =kvm_async_pf_hash_fn (gfn); +int i; + +for (i =0 ; i <ASYNC_PF_PER_VCPU ; i++) { +if (vcpu->arch.apf.gfns[key] == gfn || + vcpu->arch.apf.gfns[key] == ~0 ) +break ; + + key =kvm_async_pf_hash_next (key); + } + +return key; +} + +static void kvm_async_pf_hash_remove (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +u32 i, j, k; + + i = j =kvm_async_pf_hash_slot (vcpu, gfn); +while (true) { + vcpu->arch.apf.gfns[i] = ~0 ; + +do { + j =kvm_async_pf_hash_next (j); +if (vcpu->arch.apf.gfns[j] == ~0 ) +return ; + + k =kvm_async_pf_hash_fn (vcpu->arch.apf.gfns[j]); +/* +* k lies cyclically in ]i,j] +* | i.k.j | +* |....j i.k.| or |.k..j i...| +*/ + }while ((i <= j) ? (i < k && k <= j) : (i < k || k <= j)); + + vcpu->arch.apf.gfns[i] = vcpu->arch.apf.gfns[j]; + i = j; + } +} + +bool kvm_async_pf_hash_find (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +u32 key =kvm_async_pf_hash_slot (vcpu, gfn); + +return vcpu->arch.apf.gfns[key] == gfn; +} + +static inline int kvm_async_pf_read_cache (struct kvm_vcpu *vcpu, +u32 offset,u32 *val) +{ + return kvm_read_guest_offset_cached(vcpu->kvm, + &vcpu->arch.apf.cache,
arch/arm64/kvm/async_pf.c line 110
Quoted 10 lines
arch/arm64/kvm/async_pf.c new file +10
-0+ val, offset, sizeof(*val)); +} + +static inline int kvm_async_pf_write_cache (struct kvm_vcpu *vcpu, +u32 offset,u32 val) +{ +return kvm_write_guest_offset_cached (vcpu->kvm, + &vcpu->arch.apf.cache, + &val, offset,sizeof (val)); +}
This hash stuff doesn't look at all specific to arm64. Could it be moved to core code or a shared header file so multiple architectures only need one copy of the code. If they need to be different, it would be good to know why.
Thanks,
James
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 4/6] arm64/kvm: Support async page fault
Hi James,
Quoted 3 lines · James Morse
[my mail client went a bit nuts - it thinks this got sent already, sorry if you received it twice!]
Thanks for your time on this. No problem at all and sorry for late response :)
Quoted 7 lines
I only got so far through this, so may have focussed on the wrong things. This patch has too many things going on. Please split it up.
I think the page-fault and page-present should be done separately, with the gubbins the core-code needs added first. The cap and corresponding documentation should be added last so the whole thing bisects.
Quoted 7 lines
It looks like the guest and hypervisor parts are already split up, this is helpful, thanks. Please ensure that the descriptions are clear on this distinction too. Its not at all clear where where the 'reschedule ipi' comes from in this commit message.
I'd really like it if as much of the guest parts as possible lived in common code. There is nothing arch-specific about an interrupt and reading from shared memory.
Yes, I will split this patch into 3 patches in next revision at least:
[1] support page-not-present notification
[2] support page-ready notification
[3] control/configuration path support
Sure, I will improve the change log to make things more understandable.
I need to evaluate the possiblity of make guest parts common. I guess it guess it might be hard. Currently, there are 3 architectures support or will support the feature: x86/s390/arm64. s390 has unique implementation. Even x86/arm64 share common implementations, but still different to some extents. For example, x86 uses NMI/IRQ for the notifications, but arm64 uses SDEI/IRQ for them. The resources for SDEI should be pre-allocated on arm64, but x86 doesn't have the requirement.
Quoted 14 lines · Gavin Shan
Quoted 7 lines
There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.
Yes, its the cost of abstracting the machine's resources and invisibly carving them up.
Quoted 1 line
In order to resolve the issue and improve the guest's performance,
Quoted 8 lines
Why is it an issue? You've missed the important part of the commit message: why we should make this change.
This needs to mention postcopy live migration, which is the reason for doing any of this.
I don't think "I overcommited and swapped the guest to disk" is at all compelling. (as the doctor says: don't do that then)
Yes, live-migration should be the justification in next revision. Thanks for the reminder :)
Quoted 25 lines
Quoted 2 lines
this introduces the feature (asychronous page fault). A page-not-present
(asynchronous)
Quoted 6 lines
notification is fired to guest if the requested page isn't ready so that the guest can reschedule current process and pick another process to run. Another page-ready notification is sent to guest so that the process, which was rescheduled previously, can be picked up to run. With this,
the guest needn't necessarily suspend itself on stage two page fault.
Be suspended by the hypervisor until the resources are available. Normally the guest has no idea this is happening.
Quoted 1 line
These two notificatons are associated by so-called vCPU scoped tokens,
(notifications)
Quoted 3 lines
Quoted 1 line
which is the combination of incremental sequence number and vCPU index
(@vcpu->vcpu_idx). Besides, the notifications are conveyed by SDEI and PPI, whose numbers are specified by guest through SMCCC interface.
... so you don't need the ACPI table?
Quoted 9 lines
The 'asynchrononous page fault' notification obviously needs to be (er,) synchronous, so it has to be per-cpu.
Why is the page-ready IRQ a PPI? We have a limited number of these, so shouldn't use one unless it _needs_ to be per-cpu.
'vcpu index' should probably be the
MPIDR_EL1, which is something both the host and guest already know that identifies the vcpu. Inventing something new isn't necessary.
ACPI table is needed to pass the PPI number from qemu to guest. The PPI number is determined by qemu. The reason why we need PPI is the asynchronous page fault notification (page-ready) is delivered on per-vCPU basis.
The 'vCPU index' was inherited from x86 implementation. Actually, the information isn't used because the SDEI event and PPI are always sent to the target vCPU. It means the incremental sequence number is enough to identify the associated asynchronous page fault events. So we needn't the vCPU index conveyed as part of the token and I will drop it in next revision.
Quoted 5 lines
Quoted 2 lines
The asynchronous page fault always starts with a background woker if
(worker. Please run your commit-messages through a spell checker)
Sure, Thanks.
Quoted 6 lines
Quoted 3 lines
current vCPU has enough token space and no pending notifications. After the worker, which populates the requested page background, is started,
what does 'requested page background' mean? (missing the words 'in the'?)
Yeah, "in the" is missed and will be fixed if needed in next revision.
Quoted 10 lines
Quoted 9 lines
a page-not-present notification, together with an unique token, are sent to guest through the specified private SDEI event. The notification is acknowledged by clearing the shared cache (@vcpu->apf.cache). The current process is marked for waiting completion of asynchronous page fault. A subsequent (reschedule) IPI is sent to current vCPU so that the current process can be reschedule and suspended until the asynchronous page is completed. In the meanwhile, the woker populates the requested page and it's queued to the completion queue once the task is completed.
Quoted 6 lines
Quoted 3 lines
At this
point or the acknoledgement (by SMCCC) on pending PPI, a PPI is sent to guest for page-ready notification.A PPI is made pending, when the PPI is pending?
Another PPI can't be fired from host to guest before the previous PPI is delivered and handled.
Quoted 6 lines
Quoted 3 lines
The guest uses the associated token
to locate the suspended process and wakes it up.I hope the 'token' is nothing more than the IPA.
Correct. IPA can be used as token either except IPA is 64-bits in length, but token is 32-bits long.
Quoted 61 lines
Quoted 3 lines
In order to fulfil the functions, there are several SMCCC function IDs introduced and outlined as below:
Please add a documentation file for these for the benefit of other guests (e.g. kvm-unit-test) describing these in more detail, their parameters, the expected lifecycle etc.
Quoted 37 lines
ARM_SMCCC_KVM_FUNC_APF_VERSION
Returns the version, which can be used to identify ABI changes in the future.
ARM_SMCCC_KVM_FUNC_APF_ENABLE
Enable or disable asynchronous page fault on current vCPU.
ARM_SMCCC_KVM_FUNC_APF_TOKEN_NUM
Returns maximal number of tokens that current vCPU can have. It's used by guest to allocate the required resources.
ARM_SMCCC_KVM_FUNC_APF_SDEI
ARM_SMCCC_KVM_FUNC_APF_IRQ
Used by guest to confiugre the SDEI event and PPI numbers.
ARM_SMCCC_KVM_FUNC_APF_IRQ_ACK
Acknowledge the page-ready notificaton in the PPI handler.arch/arm64/include/asm/kvm_host.h +7
-0@@ -46,6 +46,7 @@#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4)#define KVM_REQ_SDEI KVM_ARCH_REQ(5) +#define KVM_REQ_APF_READY KVM_ARCH_REQ(6)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)@@ -239,6 +240,23 @@ struct vcpu_reset_state {bool reset; }; +#ifdef CONFIG_KVM_ASYNC_PF + +/* Should be a power of two number */ +#define ASYNC_PF_PER_VCPU 64 + +/* arch/arm64/include/asm/kvm_host.h struct vcpu_reset_state { · line 258
Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +10
-0+* The association of gfn and token. The token will be sent to guest as +* page fault address. Also, the guest could be in aarch32 mode. So its +* length should be 32-bits. +*/ +struct kvm_arch_async_pf { +u32 token; +gfn_t gfn; +u32 esr; +}; +#endif /* CONFIG_KVM_ASYNC_PF */ #ifdef-ing a struct makes it harder to use
IS_ENABLED()in the C code, which in turn makes it harder to get complete coverage from the CI builders without aligning the stars to hit the right randconfig combination.
Quoted 3 lines
Please avoid doing this unless its absolutely necessary. (its not like a struct definition takes up any space in the binary)
Ok. The #ifdef will be dropped in next revision.
Quoted 14 lines
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 368
Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +2
-0struct kvm_vcpu_arch {struct kvm_cpu_context ctxt;void *sve_state;@@ -346,6 +364,22 @@ struct kvm_vcpu_arch {struct kvm_sdei_vcpu_event *sdei_normal_event;struct kvm_sdei_vcpu_event *sdei_critical_event;struct list_head sdei_events; + +#ifdef CONFIG_KVM_ASYNC_PF See above. Any reason these can't be dynamically allocated if the feature is enabled by the VMM? (You're using a PPI, this has to be allocated by the VMM).
Quoted 2 lines
Not all guests would use this, so its a waste of space if its enabled but not in use.
Good idea. It will be dynamically allocated in next revision :)
Quoted 12 lines
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch {
Quoted 8 lines
arch/arm64/include/asm/kvm_host.h +5
-0+struct { +struct gfn_to_hva_cache cache; +gfn_t gfns[ASYNC_PF_PER_VCPU ]; +u64 control_block; +bool send_user_only;Hmmm...
Quoted 7 lines
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch {
Quoted 4 lines
arch/arm64/include/asm/kvm_host.h +3
-0+u64 sdei_event_num; +u32 irq; +u16 id;a per-vcpu id? KVM already has one, and the architecture has one that the guest can see too.
It's different from that one. It's the incremental sequence number associated with asynchronous page fault events.
Quoted 31 lines
Quoted 17 lines
arch/arm64/include/asm/kvm_host.h +5
-0+bool notpresent_pending; +u32 notpresent_token; + bool pageready_pending; + } apf; +#endif };/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ arch/arm64/include/asm/kvm_para.h new file +0
-0arch/arm64/include/asm/kvm_para.h line 10
Quoted 10 lines
arch/arm64/include/asm/kvm_para.h new file +8
-0@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM_KVM_PARA_H +#define _ASM_ARM_KVM_PARA_H +#include <uapi/asm/kvm_para.h> +static inline bool kvm_check_and_clear_guest_paused (void ) +{ +return false; +}This is part of kvmclock? aarch64 has an architecture specific way of doing that stuff. Why is this here?!
Quoted 3 lines
(if you've turned something on that needs these stubs, its probably worth doing that in an preparation patch so you can explain why, instead of hiding it in here)
It's used by the guest to retrieve the supported features through SMCCC (KVM specific feature call). Yep, it should be included in a separate patch, as part of the prepatory work to support asynchronous page fault for the guest.
Quoted 31 lines
Quoted 25 lines
arch/arm64/include/asm/kvm_para.h new file +16
-0+static inline unsigned int kvm_arch_para_features (void ) +{ +return 0 ; +} + +static inline unsigned int kvm_arch_para_hints (void ) +{ +return 0 ; +} + +static inline bool kvm_para_available (void ) +{ +return false; +} + +#endif /* _ASM_ARM_KVM_PARA_H */ arch/arm64/include/uapi/asm/Kbuild +0
-0arch/arm64/include/uapi/asm/Kbuild line 3
Quoted 3 lines
arch/arm64/include/uapi/asm/Kbuild +0
-2@@ -1,3 +1 @@ # SPDX-License-Identifier: GPL-2.0 - -generic-y += kvm_para.hThe arch specific version is practically the same as the asm-generic one.
Quoted 2 lines
Please explain why you need to do this in a preparatory patch.
Sure, will do.
Quoted 33 lines
Quoted 20 lines
arch/arm64/include/uapi/asm/kvm_para.h new file +13
-0@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _UAPI_ASM_ARM_KVM_PARA_H +#define _UAPI_ASM_ARM_KVM_PARA_H + +#include <linux/types.h> + +#define KVM_FEATURE_ASYNC_PF 0 + +/* Async PF */ +#define KVM_ASYNC_PF_ENABLED (1 << 0) +#define KVM_ASYNC_PF_SEND_ALWAYS (1 << 1) + +#define KVM_PV_REASON_PAGE_NOT_PRESENT 1arch/arm64/include/uapi/asm/kvm_para.h line 23
Quoted 10 lines
arch/arm64/include/uapi/asm/kvm_para.h new file +10
-0+#define KVM_PV_REASON_PAGE_READY 2 + +struct kvm_vcpu_pv_apf_data { +__u32 reason; +__u32 token; +__u8 pad[56 ]; +__u32 enabled; +}; + +#endif /* _UAPI_ASM_ARM_KVM_PARA_H */ Presumably this is the reason to carry around an extra copy of
kvm_para.h.
Quoted 2 lines
Wouldn't it be better to unpick that dependency instead?
Yes, it's the interface (or shared data struct) between host and guest. Sorry that I don't understand your comments here. Do you mean to make this common across multiple architectures?
Quoted 21 lines
Quoted 21 lines
arch/arm64/kvm/arm.c +6
-0@@ -225,6 +225,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)*/ r =1 ;break ; +#ifdef CONFIG_KVM_ASYNC_PF +case KVM_CAP_ASYNC_PF : +case KVM_CAP_ASYNC_PF_INT : + r =1 ; +break ; +#endif default : r =kvm_arch_vm_ioctl_check_extension (kvm, ext);break ;@@ -274,6 +280,9 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)bitmap_zero (vcpu->arch.features,KVM_VCPU_MAX_FEATURES );
Quoted 8 lines
arch/arm64/kvm/arm.c int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · line 285
Quoted 4 lines
arch/arm64/kvm/arm.c +3
-0vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; +#ifdef CONFIG_KVM_ASYNC_PF + vcpu->arch.apf.control_block =0UL ; +#endif
kvm_vm_ioctl_create_vcpu()allocates vcpu withkmem_cache_zalloc(), so this field will already be zero.
Yes. The @apf will be dynamically allocated in next revision as you suggested above :)
Quoted 12 lines
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 436
Quoted 7 lines
arch/arm64/kvm/arm.c +0
-2@@ -432,8 +441,32 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,int kvm_arch_vcpu_runnable (struct kvm_vcpu *v) {bool irq_lines = *vcpu_hcr (v) & (HCR_VI |HCR_VF ); -return ((irq_lines ||kvm_vgic_vcpu_pending_irq (v)) - && !v->arch.power_off && !v->arch.pause);As you are adding a conditional
__copy_from_user()in here, please add an unconditionalmight_sleep()at the top of this function. This lets us catch locations that will sometimes break the first time they happen.
Ok.
Quoted 10 lines
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 449
Quoted 6 lines
arch/arm64/kvm/arm.c +5
-0+if ((irq_lines ||kvm_vgic_vcpu_pending_irq (v)) && + !v->arch.power_off && !v->arch.pause) +return true; + +#ifdef CONFIG_KVM_ASYNC_PF Please use
IS_ENABLED()here, #ifdeffery make this stuff harder to read and harder to build-test.
Quoted 3 lines
(if you can come up with a readable
IS_ENABLED()way of returning the_check_extension()result - even better!)
Ok.
Quoted 8 lines
Quoted 7 lines
arch/arm64/kvm/arm.c +6
-0+if (v->arch.apf.control_block &KVM_ASYNC_PF_ENABLED ) { +u32 val; +int ret; + +if (!list_empty_careful (&v->async_pf.done)) +return true;
Quoted 5 lines
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 458
Quoted 2 lines
arch/arm64/kvm/arm.c +2
-0+ ret =kvm_read_guest_offset_cached (v->kvm, &v->arch.apf.cache, + &val,0 ,sizeof (val));You added a helper for this:
kvm_async_pf_read_cache()...
Yep, the helper should be used here.
Quoted 11 lines
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 465
Quoted 8 lines
arch/arm64/kvm/arm.c +7
-0+if (ret || val) +return true; + + ret =kvm_read_guest_offset_cached (v->kvm, &v->arch.apf.cache, + &val,4 ,sizeof (val)); +if (ret || val) +return true;These 0 and 4 really need some kind of name to give a hint as to what they are for.
Yep, that would be better. I probably use offsetof() in next revision.
Quoted 5 lines
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 464
Quoted 2 lines
arch/arm64/kvm/arm.c +1
-0+ }This really ought to wrapped up in a helper that gives a hint as to what it is doing.
Quoted 9 lines
There are two magic locations in guest memory that means the guest controls whether it is runnable or not... this sounds like a source of hard-to-debug issues.
I'd much prefer it if the guest only ever got blocked in the stage2 fault handler, as it does today. This mechanism just provides information about what is going on at stage2, which allows the guest to avoid getting blocked if it can do something about it.
Giving the guest access to the controls doesn't look right.
Yeah, It's a nice comment and I think @notpresent_pending and @pageready_pending in struct kvm_vcpu_arch::apf can alternatively serve the purpose.
Quoted 11 lines
Quoted 11 lines
arch/arm64/kvm/arm.c +3
-0+#endif + +return false; }bool kvm_arch_vcpu_in_kernel (struct kvm_vcpu *vcpu)@@ -624,6 +657,11 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_VCPU_RESET , vcpu))kvm_reset_vcpu (vcpu);
Quoted 8 lines
arch/arm64/kvm/arm.c static void check_vcpu_requests(struct kvm_vcpu *vcpu) · line 663
Quoted 4 lines
arch/arm64/kvm/arm.c +4
-0+#ifdef CONFIG_KVM_ASYNC_PF +if (kvm_check_request (KVM_REQ_APF_READY , vcpu)) +kvm_check_async_pf_completion (vcpu); +#endif (
IS_ENABLED()would save you the link error ... providing a stub in core-code forkvm_check_async_pf_completion()would let you rely on this request never being made instead)
Ok.
Quoted 28 lines
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · line 782
Quoted 12 lines
arch/arm64/kvm/arm.c +2
-1if (kvm_check_request (KVM_REQ_SDEI , vcpu))kvm_sdei_deliver (vcpu);@@ -739,7 +777,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)smp_store_mb (vcpu->mode,IN_GUEST_MODE );if (ret <=0 ||need_new_vmid_gen (&vcpu->kvm->arch.vmid) || -kvm_request_pending (vcpu)) { + (kvm_request_pending (vcpu) && +READ_ONCE (vcpu->requests) != (1UL <<KVM_REQ_APF_READY ))) {A double
READ_ONCE()looks wrong. If you have to do this, presumably you didn't want avcpu->requestto begin with.It looks like
KVM_REQ_APF_READYonly exists to move the page-ready work onto the vcpu thread because the vcpu got blocked further out bykvm_arch_vcpu_runnable(), and the second half of the stage2-fault now needs doing.I think it would be simpler to only ever block in the stage2 fault handler. The guest can only ever be blocked on one stage2 fault at a time, so this abstraction to let it sleep for any number is pointless. If I'm waiting for ipa-4, and get restarted because ipa-6 is now available ... I'm going to fault on ipa-4 for a second time. There is no point restarting the guest until it can make progress again.
Quoted 9 lines
Quoted 1 line
From the first stage2 fault you can kick off the page-fault notification (which must be
synchronous, so can be notified without taking this fault again). If the vcpu takes a fault that was already notified, it clearly can't handle the page-fault notification and must be blocked until that page is available.
We should be able to depend on the regular wakeup from mm when the vcpu is blocked. The extra information from page-present lets the guest know it can now touch these pages without getting blocked ... they shouldn't need a wakeup.
I agree it will become simpler, but it's going to affect the performance greatly if I'm understanding your comments completely. In current design, multiple works associated with asynchronous page fault and one specific vCPU can work in the background at same time. It seems you're suggesting to have one work is available at once on one specific vCPU. If so, the performance will be affected greatly.
Quoted 119 lines
Quoted 119 lines
vcpu->mode = OUTSIDE_GUEST_MODE; isb(); /* Ensure work in x_flush_hwstate is committed */ kvm_pmu_sync_hwstate(vcpu);arch/arm64/kvm/async_pf.c new file +105
-0@@ -0,0 +1,462 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright Gavin Shan, Redhat Inc 2020. +* +* Asynchronous page fault support +*/ + +#include <linux/arm-smccc.h> +#include <linux/kvm_host.h> +#include <asm/kvm_emulate.h> +#include <kvm/arm_hypercalls.h> +#include <kvm/arm_vgic.h> +#include <asm/kvm_sdei.h> +static inline u32 kvm_async_pf_hash_fn (gfn_t gfn) +{ +return hash_32 (gfn &0xffffffff ,order_base_2 (ASYNC_PF_PER_VCPU )); +} + +static inline u32 kvm_async_pf_hash_next (u32 key) +{ +return (key +1 ) & (ASYNC_PF_PER_VCPU -1 ); +} + +static inline void kvm_async_pf_hash_reset (struct kvm_vcpu *vcpu) +{ +int i; + +for (i =0 ; i <ASYNC_PF_PER_VCPU ; i++) + vcpu->arch.apf.gfns[i] = ~0 ; +} +/* +* Add gfn to the hash table. It's ensured there is a free entry +* when this function is called. +*/ +static void kvm_async_pf_hash_add (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +u32 key =kvm_async_pf_hash_fn (gfn); + +while (vcpu->arch.apf.gfns[key] != ~0 ) + key =kvm_async_pf_hash_next (key); + + vcpu->arch.apf.gfns[key] = gfn; +} + +static u32 kvm_async_pf_hash_slot (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +u32 key =kvm_async_pf_hash_fn (gfn); +int i; + +for (i =0 ; i <ASYNC_PF_PER_VCPU ; i++) { +if (vcpu->arch.apf.gfns[key] == gfn || + vcpu->arch.apf.gfns[key] == ~0 ) +break ; + + key =kvm_async_pf_hash_next (key); + } + +return key; +} + +static void kvm_async_pf_hash_remove (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +u32 i, j, k; + + i = j =kvm_async_pf_hash_slot (vcpu, gfn); +while (true) { + vcpu->arch.apf.gfns[i] = ~0 ; + +do { + j =kvm_async_pf_hash_next (j); +if (vcpu->arch.apf.gfns[j] == ~0 ) +return ; + + k =kvm_async_pf_hash_fn (vcpu->arch.apf.gfns[j]); +/* +* k lies cyclically in ]i,j] +* | i.k.j | +* |....j i.k.| or |.k..j i...| +*/ + }while ((i <= j) ? (i < k && k <= j) : (i < k || k <= j)); + + vcpu->arch.apf.gfns[i] = vcpu->arch.apf.gfns[j]; + i = j; + } +} + +bool kvm_async_pf_hash_find (struct kvm_vcpu *vcpu,gfn_t gfn) +{ +u32 key =kvm_async_pf_hash_slot (vcpu, gfn); + +return vcpu->arch.apf.gfns[key] == gfn; +} + +static inline int kvm_async_pf_read_cache (struct kvm_vcpu *vcpu, +u32 offset,u32 *val) +{ +return kvm_read_guest_offset_cached (vcpu->kvm, + &vcpu->arch.apf.cache, + val, offset,sizeof (*val)); +} + +static inline int kvm_async_pf_write_cache (struct kvm_vcpu *vcpu, +u32 offset,u32 val) +{ + return kvm_write_guest_offset_cached(vcpu->kvm,
Quoted 9 lines
arch/arm64/kvm/async_pf.c line 110
Quoted 3 lines
arch/arm64/kvm/async_pf.c new file +3
-0+ &vcpu->arch.apf.cache, + &val, offset, sizeof(val)); +}This hash stuff doesn't look at all specific to arm64. Could it be moved to core code or a shared header file so multiple architectures only need one copy of the code. If they need to be different, it would be good to know why.
Ok. It will be moved to common part shared by x86 and arm64 in next revision.
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH 5/6 drivers/acpi: Import ACPI APF table
In reply to Gavin ShanThis defines the struct for ACPI APF table. The information included in this table will be used by guest kernel to retrieve SDEI event number, PPI number and its triggering properties:
* @sdei_event: number of SDEI event used for page-not-present
* @interrupt: PPI used for page-ready
* @interrupt_flags: PPI's mode and polarity
--- include/acpi/actbl2.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
include/acpi/actbl2.h +18 -0
@@ -44,6 +44,7 @@#define ACPI_SIG_SDEI "SDEI"/* Software Delegated Exception Interface Table */ #define ACPI_SIG_SDEV "SDEV"/* Secure Devices table */ #define ACPI_SIG_NHLT "NHLT"/* Non-HDAudio Link Table */ +#define ACPI_SIG_APFT "APFT"/* Asynchronous Page Fault Table */ /* * All tables must be byte-packed to match the ACPI specification, since
@@ -1737,6 +1738,23 @@ struct acpi_sdev_pcie_path {u8 function; }; +/******************************************************************************* +* +* APFT - Asynchronous Page Fault Table +* +* Customized table used for asynchronous page fault on ARM +* +******************************************************************************/ +struct acpi_table_apft { +struct acpi_table_header header; +u32 sdei_event; +u32 interrupt; +u32 interrupt_flags; +}; + +#define ACPI_APFT_INTERRUPT_MODE (1) +#define ACPI_APFT_INTERRUPT_POLARITY (1<<1) +/* Reset to default packing */ #pragma pack()
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 5/6] drivers/acpi: Import ACPI APF table
Hi Gavin,
Quoted 4 lines · Gavin Shan
This defines the struct for ACPI APF table. The information included in this table will be used by guest kernel to retrieve SDEI event number, PPI number and its triggering properties:
Quoted 8 lines
* @sdei_event: number of SDEI event used for page-not-present
* @interrupt: PPI used for page-ready
* @interrupt_flags: PPI's mode and polaritySigned-off-by Gavin Shan <[email protected]>--- include/acpi/actbl2.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
Whoa! This is not how changes to ACPI work!
We have to work out what is needed, then the standard has to be updated, then the upstream acpica project, then these files get synced back across from the 'upstream' acpica project...
If you need an ACPI table, we need something for DT too. I don't think a well designed paravirt interface shouldn't need this. Wasn't that the whole point of the KVM "vendor specific" services?!
The cover-letter message talks of shared memory, which this doesn't describe.
Ideally this stuff would be discovered via SMCCC, or a (readonly?) page of that shared memory. That way its the same regardless of ACPI or DT.
Thanks,
James
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH 6/6 arm64/kernel: Support async page fault
In reply to Gavin ShanThis supports the asynchronous page fault, basing on the services provided by host. The asynchronous page fault starts with private SDEI event, whose number is specified by the guest. The guest marks the current running process as suspended state by the newly added flag (TIF_ASYNC_PF) and associating it with a wait queue. In the mean while, the process is put into the pre-allocated table, whose index is indeitified by the associated token. In the SDEI event handler, a followup (reschedule) IPI is sent to current CPU to force context switching on current running process. During the context switching, the flag (TIF_ASYNC_PF) is checked and the process waits on the associated wait queue.
The guest receives a PPI, whose number is specified by it, after the page becomes ready on host side. The pre-allocated table is searched using the received token to locate the associated process. The process is waken up after it's found. The previously suspended process resumes for execution and the asynchronous page fault is completed so far.
The implementation is guarded by CONFIG_KVM_GUEST. Also, the boot parameter "no-kvmapf" can be used to disable this feature. Besides, this feature has to be enabled after SDEI service is settled down because this depends on SDEI service.
--- arch/arm64/Kconfig | 11 + arch/arm64/include/asm/kvm_para.h | 26 +- arch/arm64/include/asm/processor.h | 1 + arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/kvm.c | 478 +++++++++++++++++++++++++++ arch/arm64/kernel/signal.c | 17 + 7 files changed, 531 insertions(+), 7 deletions(-) create mode 100644 arch/arm64/kernel/kvm.c
arch/arm64/Kconfig +11 -0
@@ -1053,6 +1053,17 @@ config PARAVIRT under a hypervisor, potentially improving performance significantly over full virtualization. +config KVM_GUEST + bool "KVM Guest Support" + depends on PARAVIRT + default y + help + This option enables various optimizations for running under the KVM + hypervisor. Overhead for the kernel when not running inside KVM should + be minimal. + + In case of doubt, say Y + config PARAVIRT_TIME_ACCOUNTING bool "Paravirtual steal time accounting" select PARAVIRT
arch/arm64/include/asm/kvm_para.h +20 -6
@@ -3,6 +3,20 @@#define _ASM_ARM_KVM_PARA_H#include <uapi/asm/kvm_para.h> +#include <linux/of.h> +#include <asm/hypervisor.h> + +#ifdef CONFIG_KVM_GUEST +static inline int kvm_para_available (void ) +{ +return 1 ; +} +#else +static inline int kvm_para_available (void ) +{ +return 0 ; +} +#endif /* CONFIG_KVM_GUEST */ static inline bool kvm_check_and_clear_guest_paused (void ) {
@@ -11,7 +25,12 @@ static inline bool kvm_check_and_clear_guest_paused(void)static inline unsigned int kvm_arch_para_features (void ) { -return 0 ; +unsigned int features =0 ; + +if (kvm_arm_hyp_service_available (ARM_SMCCC_KVM_FUNC_APF )) + features |= (1 <<KVM_FEATURE_ASYNC_PF ); + +return features; }static inline unsigned int kvm_arch_para_hints (void )
@@ -19,9 +38,4 @@ static inline unsigned int kvm_arch_para_hints(void) return 0; } -static inline bool kvm_para_available (void ) -{ -return false; -} -#endif /* _ASM_ARM_KVM_PARA_H */
arch/arm64/include/asm/processor.h +1 -0
@@ -151,6 +151,7 @@ struct thread_struct {struct ptrauth_keys_user keys_user;struct ptrauth_keys_kernel keys_kernel;#endif +void *data; }; static inline void arch_thread_struct_whitelist(unsigned long *offset,
arch/arm64/include/asm/thread_info.h +3 -1
@@ -67,6 +67,7 @@ void arch_release_task_struct(struct task_struct *tsk);#define TIF_FOREIGN_FPSTATE 3/* CPU's FP state is not current's */ #define TIF_UPROBE 4/* uprobe breakpoint or singlestep */ #define TIF_FSCHECK 5/* Check FS is USER_DS on return */ +#define TIF_ASYNC_PF 6/* Asynchronous page fault */ #define TIF_SYSCALL_TRACE 8/* syscall trace active */ #define TIF_SYSCALL_AUDIT 9/* syscall auditing */ #define TIF_SYSCALL_TRACEPOINT 10/* syscall tracepoint for ftrace */
@@ -86,6 +87,7 @@ void arch_release_task_struct(struct task_struct *tsk);#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)#define _TIF_FOREIGN_FPSTATE (1 << TIF_FOREIGN_FPSTATE) +#define _TIF_ASYNC_PF (1 << TIF_ASYNC_PF)#define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE)#define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT)#define _TIF_SYSCALL_TRACEPOINT (1 << TIF_SYSCALL_TRACEPOINT)
@@ -99,7 +101,7 @@ void arch_release_task_struct(struct task_struct *tsk);#define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ - _TIF_UPROBE | _TIF_FSCHECK) + _TIF_UPROBE | _TIF_FSCHECK | _TIF_ASYNC_PF)#define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
arch/arm64/kernel/Makefile +1 -0
@@ -49,6 +49,7 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o +obj-$(CONFIG_KVM_GUEST) += kvm.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \
arch/arm64/kernel/kvm.c new file +478 -0
478 lines not on this page · raw message
arch/arm64/kernel/signal.c +17 -0
@@ -921,6 +921,23 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,/* Check valid user FS if needed */ addr_limit_user_check (); +if (thread_flags & _TIF_ASYNC_PF) { +struct swait_queue_head *wq = +READ_ONCE (current->thread.data); +DECLARE_SWAITQUEUE (wait); + +local_daif_restore (DAIF_PROCCTX_NOIRQ ); + +do { +prepare_to_swait_exclusive (wq, + &wait,TASK_UNINTERRUPTIBLE ); +if (!test_thread_flag (TIF_ASYNC_PF )) +break ; + +schedule (); + }while (test_thread_flag (TIF_ASYNC_PF )); + } +if (thread_flags & _TIF_NEED_RESCHED) {/* Unmask Debug and SError for the next task */ local_daif_restore (DAIF_PROCCTX_NOIRQ );
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi Gavin,
I think this series would benefit from being in smaller pieces. I got lost in patch 4 for quite a while. Suggestion on where to do that in patch 4.
Quoted 7 lines · Gavin Shan
There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.
You really need to use postcopy live migration to justify these changes. Otherwise the story here is "over-commited hosts suck", which I don't think anyone cares about.
Quoted 3 lines
This series introduces the feature (asynchronous page fault) to resolve the issue and improve the guest's performance. It depends on the series to support SDEI virtualization and refactoring SDEI client driver.
SDEI gives you an NMI ... which you use to set a TIF flag. This can only work reliably for user-space. So much so that you have code in the hypervisor to only deliver the NMI ... when in user-space.
The only reason you would need an NMI is to interrupt interrupts-masked code. Linux can't reschedule when this is the case.
I can only conclude, you really don't need an NMI here.
Why couldn't we use an IRQ here, it would be a lot simpler? ... the reason is the arm architecture can't guarantee us that we take the irq when there is also a stage2 fault for the first instruction.
I reckon we can work around this in the hypervisor:
https://lore.kernel.org/r/[email protected]
My problem with SDEI is, you don't really need an NMI, and it creates extra in-kernel state that has to be migrated. I think having this state in the kernel complicates the user-space handling of SIGBUS_MCEERR_AO signals that don't get delivered to a vCPU thread.
Thanks,
James
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 7 lines · James Morse
SDEI gives you an NMI ... which you use to set a TIF flag. This can only work reliably for user-space. So much so that you have code in the hypervisor to only deliver the NMI ... when in user-space.
The only reason you would need an NMI is to interrupt interrupts-masked code. Linux can't reschedule when this is the case.I can only conclude, you really don't need an NMI here.
I don't think the issue is that you want an NMI. It is just that the synchronous interruption that we want is exactly the same as a SDEI, and so is the notification reply from the guest to the host (e.g. accept the async pagefault or process it synchronously).
Yes, it's more code, but at least in x86 world we tried hard not to invent new paravirtualized mechanisms if we could avoid it, especially in the host->guest direction, and it's almost always paid off. This is because in case we don't get things right, it's much much harder to fix them on both the hypervisor and the guest side; by relying on existing code that work on bare metal hardware, the guest side exists already and you can develop against it.
Paolo
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi James,
Quoted 3 lines · James Morse
I think this series would benefit from being in smaller pieces. I got lost in patch 4 for quite a while. Suggestion on where to do that in patch 4.
Yes, I will split the patches into small ones for easier review in next revision. Thanks for your comments :)
Quoted 10 lines · Gavin Shan
Quoted 7 lines
There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.
Quoted 3 lines
You really need to use postcopy live migration to justify these changes. Otherwise the story here is "over-commited hosts suck", which I don't think anyone cares about.
Yes, I will use live migration as the justification in next revision :)
Quoted 21 lines
Quoted 4 lines
This series introduces the feature (asynchronous page fault) to resolve the issue and improve the guest's performance. It depends on the series to support SDEI virtualization and refactoring SDEI client driver.
SDEI gives you an NMI ... which you use to set a TIF flag. This can only work reliably for user-space. So much so that you have code in the hypervisor to only deliver the NMI ... when in user-space.
The only reason you would need an NMI is to interrupt interrupts-masked code. Linux can't reschedule when this is the case.I can only conclude, you really don't need an NMI here.
Why couldn't we use an IRQ here, it would be a lot simpler? ... the reason is the arm architecture can't guarantee us that we take the irq when there is also a stage2 fault for the first instruction.
I reckon we can work around this in the hypervisor:
https://lore.kernel.org/r/[email protected]
Quoted 5 lines
My problem with SDEI is, you don't really need an NMI, and it creates extra in-kernel state that has to be migrated. I think having this state in the kernel complicates the user-space handling of
SIGBUS_MCEERR_AOsignals that don't get delivered to a vCPU thread.
Currently, the asynchronous page fault is only supported for memory access in guest's userspace, but we needn't to be sticky to the use model in future. It means the asynchornous page fault could be supported for memory access in guest's kernel space where the interrupt can be disabled or masked. So NMI is needed and SDEI fits the use model very well as Paolo replied in another thread.
About the feature to support SDEI virtualization, I thought there might be some use cases where the emulated devices need inject SDEI event to guest. However, I'm not too much familiar with the architecture yet. If it's required by the emulated devices, there are more more justifications to merge the code. However, the implementation itself isn't simple and I would say it's complicated.
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm