Skip to contentSkip to conversation

Support Asynchronous Page Fault

PATCH Superseded Gavin Shan 3 participants 15 messages Aug 18, 2020 – Nov 2, 2020

Subsystems (5)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Gavin Shan

author
#0 lore raw

cover Support Asynchronous Page Fault

There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.

This series introduces the feature (asynchronous page fault) to resolve the issue and improve the guest's performance. It depends on the series to support SDEI virtualization and refactoring SDEI client driver. This also depends on QEMU changes to export SDEI/APFT tables. All the code including this series can be found from github:

https://github.com/gwshan/linux ("sdei_client")
https://github.com/gwshan/linux ("sdei")
https://github.com/gwshan/linux ("apf")
https://github.com/gwshan/qemu ("apf")

The functionality is driven by two notifications: page-not-present and page-ready. They're delivered from the host to guest via SDEI event and PPI separately. In the mean while, each notification is always associated with a token, used to identify the notification. The token is passed by the shared memory between host/guest. Besides, the SMCCC interface is mitigated by the guest to configure, enable or disable the functionality. It's traditional control path.

When the guest is trapped to host because of stage two page fault, a page-not-present notification is raised by the host, and sent to the guest through (KVM private) SDEI event (0x40200001) if the requested page can't be populated immediately. In the mean while, a (background) worker is also started to populate the requested page. On receiving the SDEI event, the guest marks the current running process with special flag (TIF_ASYNC_PF) and associates the process with a pre-defined waitqueue. At same time, a (reschedule) IPI is sent to the CPU where the process was running. After the SDEI event is acknoledged by the guest, the (reschedule) IPI is delivered and it causes context switch from kernel to user space. During the context switch, the process with TIF_ASYNC_PF flag is suspended on the associated waitqueue.

Later on, a page-ready notification is sent to guest after the requested page is populated by the (background) worker. On receiving the interrupt, the guest uses the associated token to locate the process, which was previously suspended because of page-not-present, and wakes it up.

The series is organized as below:

PATCH[1-2]: support KVM hypervisor SMCCC services, which are developed by
Will Deacon.
PATCH[3]: export kvm_handle_user_mem_abort() with @prefault parameter
supported, which is prepatory work to support the feature.
PATCH[4]: support asynchronous page fault in host side.
PATCH[5]: exposes APFT (Asynchronous Page Fault Table) ACPI table, which
will be used by guest kernel to support the feature
PATCH[6]: support asynchronous page fault in guest side.

=======
Testing
=======

In the test case [1] and [2], "testsuite mem" is executed to allocate the specified percentage of free memory (90%) and then release them. In the mean while, the calculation thread is started or not. When the calculation thread isn't started, there isn't obvious performance degradtion. When the calculation thread is started, the performance is improved by 27.7% and 28.6% separately, depending on THP enablement sttus on the host side.

In test case [3] and [4], the kernel image is built and check the used time. The performance is improved by 9.7% and 9.9% separately, depending on THP enablement status on the host side.

[1] Two threads to allocate/free memory and do calculation

    vCPU:                  1
    Memory:                8GB
    memory.limit_in_bytes: 2GB
    memory.swappiness:     100
    host:                  THP disabled
    command:               "testsuite mem 90 1 [thread]"
    "-": Disabled asynchronous page fault
    "+": Enabled asynchronous page fault
    "T"  With the calculation thread

    Idx  -      +     Output  T-                   T+                   Output
    ==========================================================================
    1   93.1s  93.6s    -     223.8s 21117147961   391.9s 49845637101     -
    2   93.3s  94.2s    -     237.9s 23394567744   397.0s 50506074773     -
    3   93.5s  94.3s    -     244.2s 24305177553   405.8s 51853498870     -
    4   94.1s  95.0s    -     262.8s 27113310073   421.7s 54338181069     -
    5   94.3s  95.2s    -     272.7s 28565479414   434.3s 56171922019     -
    ==========================================================================
        93.6s  94.4s   -0.8%  248.2s 24899136549   410.1s 52543062766
                                       100318841/s          128122562/s  +27.7%

[2] Two threads to allocate/free memory and do calculation

    vCPU:                  1
    Memory:                8GB
    memory.limit_in_bytes: 2GB
    memory.swappiness:     100
    host:                  THP enabled
    command:               "testsuite mem 90 1 [thread]"
    "-": Disabled asynchronous page fault
    "+": Enabled asynchronous page fault
    "T"  With the calculation thread

    Idx  -      +     Output  T-                   T+                   Output
    ==========================================================================
    1   91.3s  91.2s    -     218.8s 20319612017   389.6s 49016175698     -
    2   91.7s  91.6s    -     233.9s 22619566161   402.0s 50901616319     -
    3   91.8s  91.9s    -     251.1s 25066180266   405.3s 51247353704     -
    4   92.7s  92.2s    -     251.1s 25262121229   406.9s 51692420054     -
    5   93.1s  92.2s    -     260.7s 26532616925   425.4s 54412348724     -
    ==========================================================================
        92.1s  91.8s   +3.0%  243.1s 23960019319   405.8  51453982899
                                       98560342/s          126796409/s  +28.6%

[3] Clear kernel image and rebuild it.

    vCPU: 24  Memory:      8GB
    memory.limit_in_bytes: 2GB
    memory.swapiness:      100
    Host:                  THP disabled
    command:               "make -j 24 clean > /dev/null 2>&1 &&
                            make -j 24 > /dev/null 2>&1"

    Idx   Disabled   Enabled   Output
    ==================================
    1     2211s      2000s     +9.5%
    2     2333s      2060s     +11.7%
    3     2568s      2192s     +14.6%
    4     2631s      2423s     +7.9%
    5     2756s      2605s     +5.4%
    ==================================
          2499s      2256s     +9.7%

[4] Clear kernel image and rebuild it.

    vCPU: 24  Memory:      8GB
    memory.limit_in_bytes: 2GB
    memory.swapiness:      100
    Host:                  THP enabled
    command:               "make -j 24 clean > /dev/null 2>&1 &&
                            make -j 24 > /dev/null 2>&1"

    Idx   Disabled   Enabled   Output
    ==================================
    1     2049s      1850s     +9.7%
    2     2144s      1947s     +9.1%
    3     2164s      1997s     +7.7%
    4     2192s      2031s     +7.3%
    5     2515s      2141s     +14.8%
    ==================================
          2214s      1993s     +9.9%

Gavin Shan (4):
kvm/arm64: Export kvm_handle_user_mem_abort() with prefault mode arm64/kvm: Support async page fault
drivers/acpi: Import ACPI APF table
arm64/kernel: Support async page fault

Will Deacon (2):
arm64: Probe for the presence of KVM hypervisor services during boot arm/arm64: KVM: Advertise KVM UID to guests via SMCCC

 arch/arm64/Kconfig                     |  11 +
 arch/arm64/include/asm/esr.h           |   5 +
 arch/arm64/include/asm/hypervisor.h    |  11 +
 arch/arm64/include/asm/kvm_emulate.h   |   8 +-
 arch/arm64/include/asm/kvm_host.h      |  54 +++
 arch/arm64/include/asm/kvm_para.h      |  41 +++
 arch/arm64/include/asm/processor.h     |   1 +
 arch/arm64/include/asm/thread_info.h   |   4 +-
 arch/arm64/include/uapi/asm/Kbuild     |   2 -
 arch/arm64/include/uapi/asm/kvm_para.h |  23 ++
 arch/arm64/kernel/Makefile             |   1 +
 arch/arm64/kernel/kvm.c                | 478 +++++++++++++++++++++++++
 arch/arm64/kernel/setup.c              |  32 ++
 arch/arm64/kernel/signal.c             |  17 +
 arch/arm64/kvm/Kconfig                 |   1 +
 arch/arm64/kvm/Makefile                |   1 +
 arch/arm64/kvm/arm.c                   |  45 ++-
 arch/arm64/kvm/async_pf.c              | 462 ++++++++++++++++++++++++
 arch/arm64/kvm/hypercalls.c            |  37 +-
 arch/arm64/kvm/mmu.c                   |  47 ++-
 arch/arm64/kvm/sdei.c                  |   8 +
 include/acpi/actbl2.h                  |  18 +
 include/linux/arm-smccc.h              |  41 +++
 23 files changed, 1321 insertions(+), 27 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_para.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_para.h
 create mode 100644 arch/arm64/kernel/kvm.c
 create mode 100644 arch/arm64/kvm/async_pf.c
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#1 lore raw

PATCH 1/6 arm64: Probe for the presence of KVM hypervisor services during boot

From: Will Deacon <[email protected]>

Although the SMCCC specification provides some limited functionality for describing the presence of hypervisor and firmware services, this is generally applicable only to functions designated as "Arm Architecture Service Functions" and no portable discovery mechanism is provided for standard hypervisor services, despite having a designated range of function identifiers reserved by the specification.

In an attempt to avoid the need for additional firmware changes every time a new function is added, introduce a UID to identify the service provider as being compatible with KVM. Once this has been established, additional services can be discovered via a feature bitmap.

---
 arch/arm64/include/asm/hypervisor.h | 11 ++++++++++
 arch/arm64/kernel/setup.c           | 32 +++++++++++++++++++++++++++++
 include/linux/arm-smccc.h           | 26 +++++++++++++++++++++++
 3 files changed, 69 insertions(+)
arch/arm64/include/asm/hypervisor.h +11 -0
index f9cc1d021791..91e4bd890819 100644
@@ -2,6 +2,17 @@
 #ifndef _ASM_ARM64_HYPERVISOR_H
 #define _ASM_ARM64_HYPERVISOR_H
 
+#include <linux/arm-smccc.h>
 #include <asm/xen/hypervisor.h>
 
+static inline bool kvm_arm_hyp_service_available(u32 func_id)
+{
+	extern DECLARE_BITMAP(__kvm_arm_hyp_services, ARM_SMCCC_KVM_NUM_FUNCS);
+
+	if (func_id >= ARM_SMCCC_KVM_NUM_FUNCS)
+		return -EINVAL;
+
+	return test_bit(func_id, __kvm_arm_hyp_services);
+}
+
 #endif
arch/arm64/kernel/setup.c +32 -0
index 77c4c9bad1b8..6a0360793317 100644
@@ -7,6 +7,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/arm-smccc.h>
 #include <linux/export.h>
 #include <linux/kernel.h>
 #include <linux/stddef.h>
@@ -275,6 +276,7 @@ static int __init reserve_memblock_reserved_regions(void)
 arch_initcall(reserve_memblock_reserved_regions);
 
 u64 __cpu_logical_map[NR_CPUS] = { [0 ... NR_CPUS-1] = INVALID_HWID };
+DECLARE_BITMAP(__kvm_arm_hyp_services, ARM_SMCCC_KVM_NUM_FUNCS) = { };
 
 u64 cpu_logical_map(int cpu)
 {
@@ -282,6 +284,35 @@ u64 cpu_logical_map(int cpu)
 }
 EXPORT_SYMBOL_GPL(cpu_logical_map);
 
+static void __init kvm_init_hyp_services(void)
+{
+	struct arm_smccc_res res;
+	int i;
+
+	arm_smccc_1_1_invoke(ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID, &res);
+	if (res.a0 != ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0 ||
+	    res.a1 != ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1 ||
+	    res.a2 != ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 ||
+	    res.a3 != ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3)
+		return;
+
+	memset(&res, 0, sizeof(res));
+	arm_smccc_1_1_invoke(ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID, &res);
+	for (i = 0; i < 32; ++i) {
+		if (res.a0 & (i))
+			set_bit(i + (32 * 0), __kvm_arm_hyp_services);
+		if (res.a1 & (i))
+			set_bit(i + (32 * 1), __kvm_arm_hyp_services);
+		if (res.a2 & (i))
+			set_bit(i + (32 * 2), __kvm_arm_hyp_services);
+		if (res.a3 & (i))
+			set_bit(i + (32 * 3), __kvm_arm_hyp_services);
+	}
+
+	pr_info("KVM hypervisor services detected (0x%08lx 0x%08lx 0x%08lx 0x%08lx)\n",
+		res.a3, res.a2, res.a1, res.a0);
+}
+
 void __init __no_sanitize_address setup_arch(char **cmdline_p)
 {
 	init_mm.start_code = (unsigned long) _text;
@@ -354,6 +385,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	else
 		psci_acpi_init();
 
+	kvm_init_hyp_services();
 	init_bootcpu_ops();
 	smp_init_cpus();
 	smp_build_mpidr_hash();
include/linux/arm-smccc.h +26 -0
index 15c706fb0a37..1c699fc6dc12 100644
@@ -49,11 +49,14 @@
 #define ARM_SMCCC_OWNER_OEM		3
 #define ARM_SMCCC_OWNER_STANDARD	4
 #define ARM_SMCCC_OWNER_STANDARD_HYP	5
+#define ARM_SMCCC_OWNER_VENDOR_HYP	6
 #define ARM_SMCCC_OWNER_TRUSTED_APP	48
 #define ARM_SMCCC_OWNER_TRUSTED_APP_END	49
 #define ARM_SMCCC_OWNER_TRUSTED_OS	50
 #define ARM_SMCCC_OWNER_TRUSTED_OS_END	63
 
+#define ARM_SMCCC_FUNC_QUERY_CALL_UID	0xff01
+
 #define ARM_SMCCC_QUIRK_NONE		0
 #define ARM_SMCCC_QUIRK_QCOM_A6		1 /* Save/restore register a6 */
 
@@ -99,6 +102,29 @@
 			   ARM_SMCCC_OWNER_STANDARD_HYP,	\
 			   0x21)
 
+#define ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID			\
+	ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,			\
+			   ARM_SMCCC_SMC_32,			\
+			   ARM_SMCCC_OWNER_VENDOR_HYP,		\
+			   ARM_SMCCC_FUNC_QUERY_CALL_UID)
+
+/* KVM UID value: 28b46fb6-2ec5-11e9-a9ca-4b564d003a74 */
+#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0	0xb66fb428U
+#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1	0xe911c52eU
+#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2	0x564bcaa9U
+#define ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3	0x743a004dU
+
+/* KVM "vendor specific" services */
+#define ARM_SMCCC_KVM_FUNC_FEATURES		0
+#define ARM_SMCCC_KVM_FUNC_FEATURES_2		127
+#define ARM_SMCCC_KVM_NUM_FUNCS			128
+
+#define ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID		\
+	ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,			\
+			   ARM_SMCCC_SMC_32,			\
+			   ARM_SMCCC_OWNER_VENDOR_HYP,		\
+			   ARM_SMCCC_KVM_FUNC_FEATURES)
+
 /*
  * Return codes defined in ARM DEN 0070A
  * ARM DEN 0070A is now merged/consolidated into ARM DEN 0028 C
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#2 lore raw

PATCH 2/6 arm/arm64: KVM: Advertise KVM UID to guests via SMCCC

In reply to Gavin Shan

From: Will Deacon <[email protected]>

We can advertise ourselves to guests as KVM and provide a basic features bitmap for discoverability of future hypervisor services.

---
 arch/arm64/kvm/hypercalls.c | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)
arch/arm64/kvm/hypercalls.c +19 -10
index 1268465efa64..cd48fd724a52 100644
@@ -13,13 +13,13 @@
 int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 {
 	u32 func_id = smccc_get_function(vcpu);
-	long val = SMCCC_RET_NOT_SUPPORTED;
+	long val[4] = { SMCCC_RET_NOT_SUPPORTED };
 	u32 feature;
 	gpa_t gpa;
 
 	switch (func_id) {
 	case ARM_SMCCC_VERSION_FUNC_ID:
-		val = ARM_SMCCC_VERSION_1_1;
+		val[0] = ARM_SMCCC_VERSION_1_1;
 		break;
 	case ARM_SMCCC_ARCH_FEATURES_FUNC_ID:
 		feature = smccc_get_arg1(vcpu);
@@ -29,10 +29,10 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 			case KVM_BP_HARDEN_UNKNOWN:
 				break;
 			case KVM_BP_HARDEN_WA_NEEDED:
-				val = SMCCC_RET_SUCCESS;
+				val[0] = SMCCC_RET_SUCCESS;
 				break;
 			case KVM_BP_HARDEN_NOT_REQUIRED:
-				val = SMCCC_RET_NOT_REQUIRED;
+				val[0] = SMCCC_RET_NOT_REQUIRED;
 				break;
 			}
 			break;
@@ -42,26 +42,35 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 			case KVM_SSBD_UNKNOWN:
 				break;
 			case KVM_SSBD_KERNEL:
-				val = SMCCC_RET_SUCCESS;
+				val[0] = SMCCC_RET_SUCCESS;
 				break;
 			case KVM_SSBD_FORCE_ENABLE:
 			case KVM_SSBD_MITIGATED:
-				val = SMCCC_RET_NOT_REQUIRED;
+				val[0] = SMCCC_RET_NOT_REQUIRED;
 				break;
 			}
 			break;
 		case ARM_SMCCC_HV_PV_TIME_FEATURES:
-			val = SMCCC_RET_SUCCESS;
+			val[0] = SMCCC_RET_SUCCESS;
 			break;
 		}
 		break;
 	case ARM_SMCCC_HV_PV_TIME_FEATURES:
-		val = kvm_hypercall_pv_features(vcpu);
+		val[0] = kvm_hypercall_pv_features(vcpu);
 		break;
 	case ARM_SMCCC_HV_PV_TIME_ST:
 		gpa = kvm_init_stolen_time(vcpu);
 		if (gpa != GPA_INVALID)
-			val = gpa;
+			val[0] = gpa;
+		break;
+	case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID:
+		val[0] = ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0;
+		val[1] = ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1;
+		val[2] = ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2;
+		val[3] = ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_3;
+		break;
+	case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID:
+		val[0] = BIT(ARM_SMCCC_KVM_FUNC_FEATURES);
 		break;
 	case SDEI_1_0_FN_SDEI_VERSION:
 	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
@@ -85,6 +94,6 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 		return kvm_psci_call(vcpu);
 	}
 
-	smccc_set_retval(vcpu, val, 0, 0, 0);
+	smccc_set_retval(vcpu, val[0], val[1], val[2], val[3]);
 	return 1;
 }
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#3 lore raw

PATCH 3/6 kvm/arm64: Export kvm_handle_user_mem_abort() with prefault mode

In reply to Gavin Shan

This renames user_mem_abort() to kvm_handle_user_mem_abort(), and then exports it. The function will be used in asynchronous page fault to populate a page table entry once the corresponding page is populated from the backup device (e.g. swap partition):

* Parameter @fault_status is replace by @esr.
* Parameter @prefault is added

As the @esr is passed as parameter, not fetched from vCPU struct. This also introduces the necessasry helpers in esr.h, to manupulate the @esr. The helpers defined in kvm_emulate.h reuses the newly added helper. This shouldn't cause functional changes.

---
 arch/arm64/include/asm/esr.h         |  5 +++++
 arch/arm64/include/asm/kvm_emulate.h |  8 ++++----
 arch/arm64/include/asm/kvm_host.h    |  4 ++++
 arch/arm64/kvm/mmu.c                 | 18 ++++++++++++------
 4 files changed, 25 insertions(+), 10 deletions(-)
arch/arm64/include/asm/esr.h +5 -0
index 035003acfa87..ec0b5d81183c 100644
@@ -317,8 +317,13 @@
 					 ESR_ELx_CP15_32_ISS_DIR_READ)
 
 #ifndef __ASSEMBLY__
+#include <linux/bitfield.h>
 #include <asm/types.h>
 
+#define esr_get_trap_class(esr)		FIELD_GET(ESR_ELx_EC_MASK, esr)
+#define esr_is_dabt_wnr(esr)		!!(FIELD_GET(ESR_ELx_WNR, esr))
+#define esr_is_dabt_iss1tw(esr)		!!(FIELD_GET(ESR_ELx_S1PTW, esr))
+
 static inline bool esr_is_data_abort(u32 esr)
 {
 	const u32 ec = ESR_ELx_EC(esr);
arch/arm64/include/asm/kvm_emulate.h +4 -4
index bb7aee5927a5..2681d1fe4003 100644
@@ -323,13 +323,13 @@ static __always_inline int kvm_vcpu_dabt_get_rd(const struct kvm_vcpu *vcpu)
 
 static __always_inline bool kvm_vcpu_dabt_iss1tw(const struct kvm_vcpu *vcpu)
 {
-	return !!(kvm_vcpu_get_hsr(vcpu) & ESR_ELx_S1PTW);
+	return esr_is_dabt_iss1tw(kvm_vcpu_get_hsr(vcpu));
 }
 
 static __always_inline bool kvm_vcpu_dabt_iswrite(const struct kvm_vcpu *vcpu)
 {
-	return !!(kvm_vcpu_get_hsr(vcpu) & ESR_ELx_WNR) ||
-		kvm_vcpu_dabt_iss1tw(vcpu); /* AF/DBM update */
+	return (esr_is_dabt_wnr(kvm_vcpu_get_hsr(vcpu)) ||
+		kvm_vcpu_dabt_iss1tw(vcpu)); /* AF/DBM update */
 }
 
 static inline bool kvm_vcpu_dabt_is_cm(const struct kvm_vcpu *vcpu)
@@ -350,7 +350,7 @@ static __always_inline bool kvm_vcpu_trap_il_is32bit(const struct kvm_vcpu *vcpu
 
 static __always_inline u8 kvm_vcpu_trap_get_class(const struct kvm_vcpu *vcpu)
 {
-	return ESR_ELx_EC(kvm_vcpu_get_hsr(vcpu));
+	return esr_get_trap_class(kvm_vcpu_get_hsr(vcpu));
 }
 
 static inline bool kvm_vcpu_trap_is_iabt(const struct kvm_vcpu *vcpu)
arch/arm64/include/asm/kvm_host.h +4 -0
index ba8cdc304b81..b6c9851b2a65 100644
@@ -441,6 +441,10 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
 			      struct kvm_vcpu_events *events);
 
 #define KVM_ARCH_WANT_MMU_NOTIFIER
+int kvm_handle_user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
+			      struct kvm_memory_slot *memslot,
+			      unsigned long hva, unsigned int esr,
+			      bool prefault);
 int kvm_unmap_hva_range(struct kvm *kvm,
 			unsigned long start, unsigned long end);
 int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte);
arch/arm64/kvm/mmu.c +12 -6
index 7a7ddc4558a7..b23778392aa1 100644
@@ -1787,11 +1787,15 @@ transparent_hugepage_adjust(struct kvm_memory_slot *memslot,
 	return PAGE_SIZE;
 }
 
-static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
-			  struct kvm_memory_slot *memslot, unsigned long hva,
-			  unsigned long fault_status)
+int kvm_handle_user_mem_abort(struct kvm_vcpu *vcpu,
+			      phys_addr_t fault_ipa,
+			      struct kvm_memory_slot *memslot,
+			      unsigned long hva,
+			      unsigned int esr,
+			      bool prefault)
 {
 	int ret;
+	unsigned int fault_status = (esr & ESR_ELx_FSC_TYPE);
 	bool write_fault, writable, force_pte = false;
 	bool exec_fault, needs_exec;
 	unsigned long mmu_seq;
@@ -1805,8 +1809,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	bool logging_active = memslot_is_logging(memslot);
 	unsigned long vma_pagesize, flags = 0;
 
-	write_fault = kvm_is_write_fault(vcpu);
-	exec_fault = kvm_vcpu_trap_is_iabt(vcpu);
+	exec_fault = (esr_get_trap_class(esr) == ESR_ELx_EC_IABT_LOW);
+	write_fault = (!exec_fault &&
+		       (esr_is_dabt_wnr(esr) || esr_is_dabt_iss1tw(esr)));
 	VM_BUG_ON(write_fault && exec_fault);
 
 	if (fault_status == FSC_PERM && !write_fault && !exec_fault) {
@@ -2116,7 +2121,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 		goto out_unlock;
 	}
 
-	ret = user_mem_abort(vcpu, fault_ipa, memslot, hva, fault_status);
+	ret = kvm_handle_user_mem_abort(vcpu, fault_ipa, memslot, hva,
+					kvm_vcpu_get_hsr(vcpu), false);
 	if (ret == 0)
 		ret = 1;
 out:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#4 lore raw

Re: [PATCH 3/6] kvm/arm64: Export kvm_handle_user_mem_abort() with prefault mode

Hi Gavin,

Quoted 10 lines · Gavin Shan

This renames user_mem_abort() to kvm_handle_user_mem_abort(), and then exports it. The function will be used in asynchronous page fault to populate a page table entry once the corresponding page is populated from the backup device (e.g. swap partition):

* Parameter @fault_status is replace by @esr.
* Parameter @prefault is added

As the @esr is passed as parameter, not fetched from vCPU struct. This also introduces the necessasry helpers in esr.h, to manupulate the @esr.

(Nit: necessary, manipulate)

Quoted 2 lines

The helpers defined in kvm_emulate.h reuses the newly added helper. This shouldn't cause functional changes.

user_mem_abort() is deep in the the guts of KVM's arch code. I don't think this should be exported. It must be called on the vcpu thread. It must be called under the VMs srcu lock. There are also tricky interactions with whether the vcpu is loaded on this cpu or not...

I think it would be much simpler to always let the guest take the stage2-fault a second time. This keeps the property that pages are only populate in response to a stage2 fault. If the guest behaves, it will only schedule a task that accesses the page once its available.

Thanks,

James
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#5 lore raw

Re: [PATCH 3/6] kvm/arm64: Export kvm_handle_user_mem_abort() with prefault mode

Hi James,

Quoted 6 lines · James Morse
Quoted 5 lines · Gavin Shan

This renames user_mem_abort() to kvm_handle_user_mem_abort(), and then exports it. The function will be used in asynchronous page fault to populate a page table entry once the corresponding page is populated from the backup device (e.g. swap partition):

Quoted 8 lines
Quoted 5 lines

* Parameter @fault_status is replace by @esr.
* Parameter @prefault is added

As the @esr is passed as parameter, not fetched from vCPU struct. This also introduces the necessasry helpers in esr.h, to manupulate the @esr.

(Nit: necessary, manipulate)

Thanks for your comments. It will be fixed in next revision :)

Quoted 4 lines
Quoted 3 lines

The helpers defined in kvm_emulate.h reuses the newly added helper. This shouldn't cause functional changes.

Quoted 9 lines

user_mem_abort() is deep in the the guts of KVM's arch code. I don't think this should be exported. It must be called on the vcpu thread. It must be called under the VMs srcu lock. There are also tricky interactions with whether the vcpu is loaded on this cpu or not...

I think it would be much simpler to always let the guest take the stage2-fault a second time. This keeps the property that pages are only populate in response to a stage2 fault. If the guest behaves, it will only schedule a task that accesses the page once its available.

The function is called with SRCU hold in this patchset. However, it seems more reasonable to be called in vCPU thread. On the other hand, I'm worrying about the performance degradation. I will evaluate it in next revision and leave the function is called in vCPU thread if performance is acceptable. Otherwise, I would like to leave it as of being :)

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#6 lore raw

PATCH 4/6 arm64/kvm: Support async page fault

In reply to Gavin Shan

There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.

In order to resolve the issue and improve the guest's performance, this introduces the feature (asychronous page fault). A page-not-present notification is fired to guest if the requested page isn't ready so that the guest can reschedule current process and pick another process to run. Another page-ready notification is sent to guest so that the process, which was rescheduled previously, can be picked up to run. With this, the guest needn't necessarily suspend itself on stage two page fault. These two notificatons are associated by so-called vCPU scoped tokens, which is the combination of incremental sequence number and vCPU index (@vcpu->vcpu_idx). Besides, the notifications are conveyed by SDEI and PPI, whose numbers are specified by guest through SMCCC interface.

The asynchronous page fault always starts with a background woker if current vCPU has enough token space and no pending notifications. After the worker, which populates the requested page background, is started, a page-not-present notification, together with an unique token, are sent to guest through the specified private SDEI event. The notification is acknowledged by clearing the shared cache (@vcpu->apf.cache). The current process is marked for waiting completion of asynchronous page fault. A subsequent (reschedule) IPI is sent to current vCPU so that the current process can be reschedule and suspended until the asynchronous page is completed. In the meanwhile, the woker populates the requested page and it's queued to the completion queue once the task is completed. At this point or the acknoledgement (by SMCCC) on pending PPI, a PPI is sent to guest for page-ready notification. The guest uses the associated token to locate the suspended process and wakes it up.

In order to fulfil the functions, there are several SMCCC function IDs introduced and outlined as below:

ARM_SMCCC_KVM_FUNC_APF_VERSION
Returns the version, which can be used to identify ABI changes in the future.
ARM_SMCCC_KVM_FUNC_APF_ENABLE
Enable or disable asynchronous page fault on current vCPU.
ARM_SMCCC_KVM_FUNC_APF_TOKEN_NUM
Returns maximal number of tokens that current vCPU can have. It's used by guest to allocate the required resources.
ARM_SMCCC_KVM_FUNC_APF_SDEI
ARM_SMCCC_KVM_FUNC_APF_IRQ
Used by guest to confiugre the SDEI event and PPI numbers.
ARM_SMCCC_KVM_FUNC_APF_IRQ_ACK
Acknowledge the page-ready notificaton in the PPI handler.

---
 arch/arm64/include/asm/kvm_host.h      |  50 +++
 arch/arm64/include/asm/kvm_para.h      |  27 ++
 arch/arm64/include/uapi/asm/Kbuild     |   2 -
 arch/arm64/include/uapi/asm/kvm_para.h |  23 ++
 arch/arm64/kvm/Kconfig                 |   1 +
 arch/arm64/kvm/Makefile                |   1 +
 arch/arm64/kvm/arm.c                   |  45 ++-
 arch/arm64/kvm/async_pf.c              | 462 +++++++++++++++++++++++++
 arch/arm64/kvm/hypercalls.c            |   8 +
 arch/arm64/kvm/mmu.c                   |  29 +-
 arch/arm64/kvm/sdei.c                  |   8 +
 include/linux/arm-smccc.h              |  15 +
 12 files changed, 665 insertions(+), 6 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_para.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_para.h
 create mode 100644 arch/arm64/kvm/async_pf.c
arch/arm64/include/asm/kvm_host.h +50 -0
index b6c9851b2a65..2e3bba6a99c3 100644
@@ -46,6 +46,7 @@
 #define KVM_REQ_RECORD_STEAL	KVM_ARCH_REQ(3)
 #define KVM_REQ_RELOAD_GICv4	KVM_ARCH_REQ(4)
 #define KVM_REQ_SDEI		KVM_ARCH_REQ(5)
+#define KVM_REQ_APF_READY	KVM_ARCH_REQ(6)
 
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
 				     KVM_DIRTY_LOG_INITIALLY_SET)
@@ -239,6 +240,23 @@ struct vcpu_reset_state {
 	bool		reset;
 };
 
+#ifdef CONFIG_KVM_ASYNC_PF
+
+/* Should be a power of two number */
+#define ASYNC_PF_PER_VCPU	64
+
+/*
+ * The association of gfn and token. The token will be sent to guest as
+ * page fault address. Also, the guest could be in aarch32 mode. So its
+ * length should be 32-bits.
+ */
+struct kvm_arch_async_pf {
+	u32     token;
+	gfn_t   gfn;
+	u32     esr;
+};
+#endif /* CONFIG_KVM_ASYNC_PF */
+
 struct kvm_vcpu_arch {
 	struct kvm_cpu_context ctxt;
 	void *sve_state;
@@ -346,6 +364,22 @@ struct kvm_vcpu_arch {
 	struct kvm_sdei_vcpu_event	*sdei_normal_event;
 	struct kvm_sdei_vcpu_event	*sdei_critical_event;
 	struct list_head		sdei_events;
+
+#ifdef CONFIG_KVM_ASYNC_PF
+	struct {
+		struct gfn_to_hva_cache	cache;
+		gfn_t			gfns[ASYNC_PF_PER_VCPU];
+		u64			control_block;
+		bool			send_user_only;
+		u64			sdei_event_num;
+		u32			irq;
+
+		u16			id;
+		bool			notpresent_pending;
+		u32			notpresent_token;
+		bool			pageready_pending;
+	} apf;
+#endif
 };
 
 /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
@@ -566,6 +600,22 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,
 int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,
 			       struct kvm_device_attr *attr);
 
+#ifdef CONFIG_KVM_ASYNC_PF
+bool kvm_async_pf_hash_find(struct kvm_vcpu *vcpu, gfn_t gfn);
+bool kvm_arch_async_not_present_allowed(struct kvm_vcpu *vcpu);
+int kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, u32 esr, gpa_t gpa,
+			    gfn_t gfn);
+bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
+				     struct kvm_async_pf *work);
+void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu);
+bool kvm_arch_can_dequeue_async_page_present(struct kvm_vcpu *vcpu);
+void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu,
+			       struct kvm_async_pf *work);
+void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
+				 struct kvm_async_pf *work);
+long kvm_async_pf_hypercall(struct kvm_vcpu *vcpu);
+#endif /* CONFIG_KVM_ASYNC_PF */
+
 /* Guest/host FPSIMD coordination helpers */
 int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu);
 void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu);
arch/arm64/include/asm/kvm_para.h new file +27 -0
new file mode 100644
index 000000000000..0ea481dd1c7a
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM_KVM_PARA_H
+#define _ASM_ARM_KVM_PARA_H
+
+#include <uapi/asm/kvm_para.h>
+
+static inline bool kvm_check_and_clear_guest_paused(void)
+{
+	return false;
+}
+
+static inline unsigned int kvm_arch_para_features(void)
+{
+	return 0;
+}
+
+static inline unsigned int kvm_arch_para_hints(void)
+{
+	return 0;
+}
+
+static inline bool kvm_para_available(void)
+{
+	return false;
+}
+
+#endif /* _ASM_ARM_KVM_PARA_H */
arch/arm64/include/uapi/asm/Kbuild +0 -2
index 602d137932dc..f66554cd5c45 100644
@@ -1,3 +1 @@
 # SPDX-License-Identifier: GPL-2.0
-
-generic-y += kvm_para.h
arch/arm64/include/uapi/asm/kvm_para.h new file +23 -0
new file mode 100644
index 000000000000..f0dbe86c2374
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _UAPI_ASM_ARM_KVM_PARA_H
+#define _UAPI_ASM_ARM_KVM_PARA_H
+
+#include <linux/types.h>
+
+#define KVM_FEATURE_ASYNC_PF	0
+
+/* Async PF */
+#define KVM_ASYNC_PF_ENABLED		(1 << 0)
+#define KVM_ASYNC_PF_SEND_ALWAYS	(1 << 1)
+
+#define KVM_PV_REASON_PAGE_NOT_PRESENT	1
+#define KVM_PV_REASON_PAGE_READY	2
+
+struct kvm_vcpu_pv_apf_data {
+	__u32	reason;
+	__u32	token;
+	__u8	pad[56];
+	__u32	enabled;
+};
+
+#endif /* _UAPI_ASM_ARM_KVM_PARA_H */
arch/arm64/kvm/Kconfig +1 -0
index 13489aff4440..6dd843be8e10 100644
@@ -32,6 +32,7 @@ menuconfig KVM
 	select KVM_VFIO
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
+	select KVM_ASYNC_PF
 	select HAVE_KVM_MSI
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_IRQ_ROUTING
arch/arm64/kvm/Makefile +1 -0
index 5ebd8abd81c8..f99fcccb8cc6 100644
@@ -25,3 +25,4 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \
 	 vgic/vgic-its.o vgic/vgic-debug.o
 
 kvm-$(CONFIG_KVM_ARM_PMU)  += pmu-emul.o
+kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o async_pf.o
arch/arm64/kvm/arm.c +42 -3

75 lines not on this page · raw message

arch/arm64/kvm/async_pf.c new file +462 -0

462 lines not on this page · raw message

arch/arm64/kvm/hypercalls.c +8 -0
index cd48fd724a52..ae7e07aaa521 100644
@@ -71,7 +71,15 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 		break;
 	case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID:
 		val[0] = BIT(ARM_SMCCC_KVM_FUNC_FEATURES);
+#ifdef CONFIG_KVM_ASYNC_PF
+		val[0] |= BIT(ARM_SMCCC_KVM_FUNC_APF);
+#endif
 		break;
+#ifdef CONFIG_KVM_ASYNC_PF
+	case ARM_SMCCC_VENDOR_HYP_KVM_APF_FUNC_ID:
+		val[0] = kvm_async_pf_hypercall(vcpu);
+		break;
+#endif
 	case SDEI_1_0_FN_SDEI_VERSION:
 	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
arch/arm64/kvm/mmu.c +28 -1
index b23778392aa1..81579bdab4a8 100644
@@ -1787,6 +1787,30 @@ transparent_hugepage_adjust(struct kvm_memory_slot *memslot,
 	return PAGE_SIZE;
 }
 
+static bool try_async_pf(struct kvm_vcpu *vcpu, u32 esr, gpa_t gpa,
+			 gfn_t gfn, kvm_pfn_t *pfn, bool write,
+			 bool *writable, bool prefault)
+{
+	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
+#ifdef CONFIG_KVM_ASYNC_PF
+	bool async = false;
+
+	/* Bail if *pfn has correct page */
+	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, &async, write, writable);
+	if (!async)
+		return false;
+
+	if (!prefault && kvm_arch_async_not_present_allowed(vcpu)) {
+		if (kvm_async_pf_hash_find(vcpu, gfn) ||
+		    kvm_arch_setup_async_pf(vcpu, esr, gpa, gfn))
+			return true;
+	}
+#endif
+
+	*pfn = __gfn_to_pfn_memslot(slot, gfn, false, NULL, write, writable);
+	return false;
+}
+
 int kvm_handle_user_mem_abort(struct kvm_vcpu *vcpu,
 			      phys_addr_t fault_ipa,
 			      struct kvm_memory_slot *memslot,
@@ -1870,7 +1894,10 @@ int kvm_handle_user_mem_abort(struct kvm_vcpu *vcpu,
 	 */
 	smp_rmb();
 
-	pfn = gfn_to_pfn_prot(kvm, gfn, write_fault, &writable);
+	if (try_async_pf(vcpu, esr, fault_ipa, gfn, &pfn,
+			 write_fault, &writable, prefault))
+		return 1;
+
 	if (pfn == KVM_PFN_ERR_HWPOISON) {
 		kvm_send_hwpoison_signal(hva, vma_shift);
 		return 0;
arch/arm64/kvm/sdei.c +8 -0
index 2c05edcb3fbb..8a9df119fa73 100644
@@ -22,6 +22,14 @@ static struct kvm_sdei_priv kvm_sdei_privs[] = {
 	  0,
 	  NULL
 	},
+	{ 0x40200001,   /* Asynchronous page fault */
+	  SDEI_EVENT_TYPE_PRIVATE,
+	  1,
+	  SDEI_EVENT_PRIORITY_CRITICAL,
+	  SDEI_EVENT_REGISTER_RM_ANY,
+	  0,
+	  NULL
+	},
 };
 
 #ifdef CONFIG_ARM_SDE_INTERFACE
include/linux/arm-smccc.h +15 -0
index 1c699fc6dc12..31a061a3a093 100644
@@ -116,6 +116,7 @@
 
 /* KVM "vendor specific" services */
 #define ARM_SMCCC_KVM_FUNC_FEATURES		0
+#define ARM_SMCCC_KVM_FUNC_APF			1
 #define ARM_SMCCC_KVM_FUNC_FEATURES_2		127
 #define ARM_SMCCC_KVM_NUM_FUNCS			128
 
@@ -125,6 +126,20 @@
 			   ARM_SMCCC_OWNER_VENDOR_HYP,		\
 			   ARM_SMCCC_KVM_FUNC_FEATURES)
 
+/* Async page fault service */
+#define ARM_SMCCC_KVM_FUNC_APF_VERSION		0
+#define ARM_SMCCC_KVM_FUNC_APF_ENABLE		1
+#define ARM_SMCCC_KVM_FUNC_APF_TOKEN_NUM	2
+#define ARM_SMCCC_KVM_FUNC_APF_SDEI		3
+#define ARM_SMCCC_KVM_FUNC_APF_IRQ		4
+#define ARM_SMCCC_KVM_FUNC_APF_IRQ_ACK		5
+
+#define ARM_SMCCC_VENDOR_HYP_KVM_APF_FUNC_ID			\
+	ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,			\
+			   ARM_SMCCC_SMC_32,			\
+			   ARM_SMCCC_OWNER_VENDOR_HYP,		\
+			   ARM_SMCCC_KVM_FUNC_APF)
+
 /*
  * Return codes defined in ARM DEN 0070A
  * ARM DEN 0070A is now merged/consolidated into ARM DEN 0028 C
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#7 lore raw

Re: [PATCH 4/6] arm64/kvm: Support async page fault

Hi Gavin,

[my mail client went a bit nuts - it thinks this got sent already, sorry if you received it twice!]

I only got so far through this, so may have focussed on the wrong things. This patch has too many things going on. Please split it up.

I think the page-fault and page-present should be done separately, with the gubbins the core-code needs added first. The cap and corresponding documentation should be added last so the whole thing bisects.

It looks like the guest and hypervisor parts are already split up, this is helpful, thanks. Please ensure that the descriptions are clear on this distinction too. Its not at all clear where where the 'reschedule ipi' comes from in this commit message.

I'd really like it if as much of the guest parts as possible lived in common code. There is nothing arch-specific about an interrupt and reading from shared memory.

Quoted 7 lines · Gavin Shan

There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.

Yes, its the cost of abstracting the machine's resources and invisibly carving them up.

Quoted 1 line

In order to resolve the issue and improve the guest's performance,

Why is it an issue? You've missed the important part of the commit message: why we should make this change.

This needs to mention postcopy live migration, which is the reason for doing any of this.

I don't think "I overcommited and swapped the guest to disk" is at all compelling. (as the doctor says: don't do that then)

Quoted 1 line

this introduces the feature (asychronous page fault). A page-not-present

(asynchronous)

Quoted 6 lines

notification is fired to guest if the requested page isn't ready so that the guest can reschedule current process and pick another process to run. Another page-ready notification is sent to guest so that the process, which was rescheduled previously, can be picked up to run. With this,

the guest needn't necessarily suspend itself on stage two page fault.

Be suspended by the hypervisor until the resources are available. Normally the guest has no idea this is happening.

Quoted 1 line

These two notificatons are associated by so-called vCPU scoped tokens,

(notifications)

Quoted 3 lines

which is the combination of incremental sequence number and vCPU index (@vcpu->vcpu_idx). Besides, the notifications are conveyed by SDEI and PPI, whose numbers are specified by guest through SMCCC interface.

... so you don't need the ACPI table?

The 'asynchrononous page fault' notification obviously needs to be (er,) synchronous, so it has to be per-cpu.

Why is the page-ready IRQ a PPI? We have a limited number of these, so shouldn't use one unless it _needs_ to be per-cpu.

'vcpu index' should probably be the MPIDR_EL1, which is something both the host and guest already know that identifies the vcpu. Inventing something new isn't necessary.

Quoted 1 line

The asynchronous page fault always starts with a background woker if

(worker. Please run your commit-messages through a spell checker)

Quoted 2 lines

current vCPU has enough token space and no pending notifications. After the worker, which populates the requested page background, is started,

what does 'requested page background' mean? (missing the words 'in the'?)

Quoted 9 lines

a page-not-present notification, together with an unique token, are sent to guest through the specified private SDEI event. The notification is acknowledged by clearing the shared cache (@vcpu->apf.cache). The current process is marked for waiting completion of asynchronous page fault. A subsequent (reschedule) IPI is sent to current vCPU so that the current process can be reschedule and suspended until the asynchronous page is completed. In the meanwhile, the woker populates the requested page and it's queued to the completion queue once the task is completed.

Quoted 3 lines

At this
point or the acknoledgement (by SMCCC) on pending PPI, a PPI is sent to guest for page-ready notification.

A PPI is made pending, when the PPI is pending?

Quoted 2 lines

The guest uses the associated token
to locate the suspended process and wakes it up.

I hope the 'token' is nothing more than the IPA.

Quoted 2 lines

In order to fulfil the functions, there are several SMCCC function IDs introduced and outlined as below:

Please add a documentation file for these for the benefit of other guests (e.g. kvm-unit-test) describing these in more detail, their parameters, the expected lifecycle etc.

Quoted 37 lines

ARM_SMCCC_KVM_FUNC_APF_VERSION
Returns the version, which can be used to identify ABI changes in the future.
ARM_SMCCC_KVM_FUNC_APF_ENABLE
Enable or disable asynchronous page fault on current vCPU.
ARM_SMCCC_KVM_FUNC_APF_TOKEN_NUM
Returns maximal number of tokens that current vCPU can have. It's used by guest to allocate the required resources.
ARM_SMCCC_KVM_FUNC_APF_SDEI
ARM_SMCCC_KVM_FUNC_APF_IRQ
Used by guest to confiugre the SDEI event and PPI numbers.
ARM_SMCCC_KVM_FUNC_APF_IRQ_ACK
Acknowledge the page-ready notificaton in the PPI handler.

arch/arm64/include/asm/kvm_host.h +7 -0
index b6c9851b2a65..2e3bba6a99c3 100644
@@ -46,6 +46,7 @@
 #define KVM_REQ_RECORD_STEAL	KVM_ARCH_REQ(3)
 #define KVM_REQ_RELOAD_GICv4	KVM_ARCH_REQ(4)
 #define KVM_REQ_SDEI		KVM_ARCH_REQ(5)
+#define KVM_REQ_APF_READY	KVM_ARCH_REQ(6)
 
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
 				     KVM_DIRTY_LOG_INITIALLY_SET)
@@ -239,6 +240,23 @@ struct vcpu_reset_state {
 	bool		reset;
 };
 
+#ifdef CONFIG_KVM_ASYNC_PF
+
+/* Should be a power of two number */
+#define ASYNC_PF_PER_VCPU	64
+
+/*

arch/arm64/include/asm/kvm_host.h struct vcpu_reset_state { · line 258

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +10 -0
+ * The association of gfn and token. The token will be sent to guest as
+ * page fault address. Also, the guest could be in aarch32 mode. So its
+ * length should be 32-bits.
+ */
+struct kvm_arch_async_pf {
+	u32     token;
+	gfn_t   gfn;
+	u32     esr;
+};
+#endif /* CONFIG_KVM_ASYNC_PF */

#ifdef-ing a struct makes it harder to use IS_ENABLED() in the C code, which in turn makes it harder to get complete coverage from the CI builders without aligning the stars to hit the right randconfig combination.

Please avoid doing this unless its absolutely necessary. (its not like a struct definition takes up any space in the binary)

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 368

Quoted 9 lines
arch/arm64/include/asm/kvm_host.h +2 -0
 struct kvm_vcpu_arch {
 	struct kvm_cpu_context ctxt;
 	void *sve_state;
@@ -346,6 +364,22 @@ struct kvm_vcpu_arch {
 	struct kvm_sdei_vcpu_event	*sdei_normal_event;
 	struct kvm_sdei_vcpu_event	*sdei_critical_event;
 	struct list_head		sdei_events;
+
+#ifdef CONFIG_KVM_ASYNC_PF

See above. Any reason these can't be dynamically allocated if the feature is enabled by the VMM? (You're using a PPI, this has to be allocated by the VMM).

Not all guests would use this, so its a waste of space if its enabled but not in use.

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch {

Quoted 7 lines
arch/arm64/include/asm/kvm_host.h +5 -0
+	struct {
+		struct gfn_to_hva_cache	cache;
 
+		gfn_t			gfns[ASYNC_PF_PER_VCPU];
+		u64			control_block;
 
+		bool			send_user_only;

Hmmm...

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch {

Quoted 4 lines
arch/arm64/include/asm/kvm_host.h +3 -0
+		u64			sdei_event_num;
+		u32			irq;
 
+		u16			id;

a per-vcpu id? KVM already has one, and the architecture has one that the guest can see too.

Quoted 16 lines
arch/arm64/include/asm/kvm_host.h +5 -0
+		bool			notpresent_pending;
+		u32			notpresent_token;
+		bool			pageready_pending;
+	} apf;
+#endif
 };
 
 /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
arch/arm64/include/asm/kvm_para.h new file +0 -0
new file mode 100644
index 000000000000..0ea481dd1c7a

arch/arm64/include/asm/kvm_para.h line 10

Quoted 10 lines
arch/arm64/include/asm/kvm_para.h new file +8 -0
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM_KVM_PARA_H
+#define _ASM_ARM_KVM_PARA_H
 
+#include <uapi/asm/kvm_para.h>
 
+static inline bool kvm_check_and_clear_guest_paused(void)
+{
+	return false;
+}

This is part of kvmclock? aarch64 has an architecture specific way of doing that stuff. Why is this here?!

(if you've turned something on that needs these stubs, its probably worth doing that in an preparation patch so you can explain why, instead of hiding it in here)

Quoted 24 lines
arch/arm64/include/asm/kvm_para.h new file +16 -0
+static inline unsigned int kvm_arch_para_features(void)
+{
+	return 0;
+}
+
+static inline unsigned int kvm_arch_para_hints(void)
+{
+	return 0;
+}
+
+static inline bool kvm_para_available(void)
+{
+	return false;
+}
+
+#endif /* _ASM_ARM_KVM_PARA_H */
arch/arm64/include/uapi/asm/Kbuild +0 -0
index 602d137932dc..f66554cd5c45 100644

arch/arm64/include/uapi/asm/Kbuild line 3

Quoted 3 lines
arch/arm64/include/uapi/asm/Kbuild +0 -2
@@ -1,3 +1 @@
 # SPDX-License-Identifier: GPL-2.0
-
-generic-y += kvm_para.h

The arch specific version is practically the same as the asm-generic one.

Please explain why you need to do this in a preparatory patch.

Quoted 19 lines
arch/arm64/include/uapi/asm/kvm_para.h new file +13 -0
new file mode 100644
index 000000000000..f0dbe86c2374
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _UAPI_ASM_ARM_KVM_PARA_H
+#define _UAPI_ASM_ARM_KVM_PARA_H
+
+#include <linux/types.h>
+
+#define KVM_FEATURE_ASYNC_PF	0
+
+/* Async PF */
+#define KVM_ASYNC_PF_ENABLED		(1 << 0)
+#define KVM_ASYNC_PF_SEND_ALWAYS	(1 << 1)
+
+#define KVM_PV_REASON_PAGE_NOT_PRESENT	1

arch/arm64/include/uapi/asm/kvm_para.h line 23

Quoted 10 lines
arch/arm64/include/uapi/asm/kvm_para.h new file +10 -0
+#define KVM_PV_REASON_PAGE_READY	2
+
+struct kvm_vcpu_pv_apf_data {
+	__u32	reason;
+	__u32	token;
+	__u8	pad[56];
+	__u32	enabled;
+};
+
+#endif /* _UAPI_ASM_ARM_KVM_PARA_H */

Presumably this is the reason to carry around an extra copy of kvm_para.h.

Wouldn't it be better to unpick that dependency instead?

Quoted 18 lines
arch/arm64/kvm/arm.c +6 -0
index 4bec6c9ece18..1a57dc5dbf14 100644
@@ -225,6 +225,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		 */
 		r = 1;
 		break;
+#ifdef CONFIG_KVM_ASYNC_PF
+	case KVM_CAP_ASYNC_PF:
+	case KVM_CAP_ASYNC_PF_INT:
+		r = 1;
+		break;
+#endif
 	default:
 		r = kvm_arch_vm_ioctl_check_extension(kvm, ext);
 		break;

arch/arm64/kvm/arm.c int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · line 285

Quoted 6 lines
arch/arm64/kvm/arm.c +3 -0
@@ -274,6 +280,9 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 	bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
 
 	vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO;
+#ifdef CONFIG_KVM_ASYNC_PF
+	vcpu->arch.apf.control_block = 0UL;
+#endif

kvm_vm_ioctl_create_vcpu() allocates vcpu with kmem_cache_zalloc(), so this field will already be zero.

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 436

Quoted 6 lines
arch/arm64/kvm/arm.c +0 -2
@@ -432,8 +441,32 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *v)
 {
 	bool irq_lines = *vcpu_hcr(v) & (HCR_VI | HCR_VF);
-	return ((irq_lines || kvm_vgic_vcpu_pending_irq(v))
-		&& !v->arch.power_off && !v->arch.pause);

As you are adding a conditional __copy_from_user() in here, please add an unconditional might_sleep() at the top of this function. This lets us catch locations that will sometimes break the first time they happen.

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 449

Quoted 5 lines
arch/arm64/kvm/arm.c +5 -0
+	if ((irq_lines || kvm_vgic_vcpu_pending_irq(v)) &&
+	    !v->arch.power_off && !v->arch.pause)
+		return true;
+
+#ifdef CONFIG_KVM_ASYNC_PF

Please use IS_ENABLED() here, #ifdeffery make this stuff harder to read and harder to build-test.

(if you can come up with a readable IS_ENABLED() way of returning the _check_extension() result - even better!)

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 458

Quoted 9 lines
arch/arm64/kvm/arm.c +8 -0
+	if (v->arch.apf.control_block & KVM_ASYNC_PF_ENABLED) {
+		u32 val;
+		int ret;
+
+		if (!list_empty_careful(&v->async_pf.done))
+			return true;
 
+		ret = kvm_read_guest_offset_cached(v->kvm, &v->arch.apf.cache,
+						   &val, 0, sizeof(val));

You added a helper for this: kvm_async_pf_read_cache()...

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 465

Quoted 7 lines
arch/arm64/kvm/arm.c +7 -0
+		if (ret || val)
+			return true;
+
+		ret = kvm_read_guest_offset_cached(v->kvm, &v->arch.apf.cache,
+						   &val, 4, sizeof(val));
+		if (ret || val)
+			return true;

These 0 and 4 really need some kind of name to give a hint as to what they are for.

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 465

Quoted 1 line
arch/arm64/kvm/arm.c +1 -0
+	}

This really ought to wrapped up in a helper that gives a hint as to what it is doing.

There are two magic locations in guest memory that means the guest controls whether it is runnable or not... this sounds like a source of hard-to-debug issues.

I'd much prefer it if the guest only ever got blocked in the stage2 fault handler, as it does today. This mechanism just provides information about what is going on at stage2, which allows the guest to avoid getting blocked if it can do something about it.

Giving the guest access to the controls doesn't look right.

Quoted 7 lines
arch/arm64/kvm/arm.c +3 -0
+#endif
+
+	return false;
 }
 
 bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu)

arch/arm64/kvm/arm.c static void check_vcpu_requests(struct kvm_vcpu *vcpu) · line 663

Quoted 7 lines
arch/arm64/kvm/arm.c +4 -0
@@ -624,6 +657,11 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
 		if (kvm_check_request(KVM_REQ_VCPU_RESET, vcpu))
 			kvm_reset_vcpu(vcpu);
 
+#ifdef CONFIG_KVM_ASYNC_PF
+		if (kvm_check_request(KVM_REQ_APF_READY, vcpu))
+			kvm_check_async_pf_completion(vcpu);
+#endif

(IS_ENABLED() would save you the link error ... providing a stub in core-code for kvm_check_async_pf_completion() would let you rely on this request never being made instead)

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · line 782

Quoted 11 lines
arch/arm64/kvm/arm.c +2 -1
 		if (kvm_check_request(KVM_REQ_SDEI, vcpu))
 			kvm_sdei_deliver(vcpu);
 
@@ -739,7 +777,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		smp_store_mb(vcpu->mode, IN_GUEST_MODE);
 
 		if (ret <= 0 || need_new_vmid_gen(&vcpu->kvm->arch.vmid) ||
-		    kvm_request_pending(vcpu)) {
 
+		    (kvm_request_pending(vcpu) &&
+		     READ_ONCE(vcpu->requests) != (1UL << KVM_REQ_APF_READY))) {

A double READ_ONCE() looks wrong. If you have to do this, presumably you didn't want a vcpu->request to begin with.

It looks like KVM_REQ_APF_READY only exists to move the page-ready work onto the vcpu thread because the vcpu got blocked further out by kvm_arch_vcpu_runnable(), and the second half of the stage2-fault now needs doing.

I think it would be simpler to only ever block in the stage2 fault handler. The guest can only ever be blocked on one stage2 fault at a time, so this abstraction to let it sleep for any number is pointless. If I'm waiting for ipa-4, and get restarted because ipa-6 is now available ... I'm going to fault on ipa-4 for a second time. There is no point restarting the guest until it can make progress again.

From the first stage2 fault you can kick off the page-fault notification (which must be synchronous, so can be notified without taking this fault again). If the vcpu takes a fault that was already notified, it clearly can't handle the page-fault notification and must be blocked until that page is available.

We should be able to depend on the regular wakeup from mm when the vcpu is blocked. The extra information from page-present lets the guest know it can now touch these pages without getting blocked ... they shouldn't need a wakeup.

Quoted 111 lines
arch/arm64/kvm/arm.c +0 -0
 			vcpu->mode = OUTSIDE_GUEST_MODE;
 			isb(); /* Ensure work in x_flush_hwstate is committed */
arch/arm64/kvm/arm.c +0 -0
 			kvm_pmu_sync_hwstate(vcpu);
arch/arm64/kvm/async_pf.c new file +98 -0
new file mode 100644
index 000000000000..910cb3a1bdc2
@@ -0,0 +1,462 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright Gavin Shan, Redhat Inc 2020.
+ *
+ * Asynchronous page fault support
+ */
+
+#include <linux/arm-smccc.h>
+#include <linux/kvm_host.h>
+#include <asm/kvm_emulate.h>
+#include <kvm/arm_hypercalls.h>
+#include <kvm/arm_vgic.h>
+#include <asm/kvm_sdei.h>
 
+static inline u32 kvm_async_pf_hash_fn(gfn_t gfn)
+{
+	return hash_32(gfn & 0xffffffff, order_base_2(ASYNC_PF_PER_VCPU));
+}
+
+static inline u32 kvm_async_pf_hash_next(u32 key)
+{
+	return (key + 1) & (ASYNC_PF_PER_VCPU - 1);
+}
+
+static inline void kvm_async_pf_hash_reset(struct kvm_vcpu *vcpu)
+{
+	int i;
+
+	for (i = 0; i < ASYNC_PF_PER_VCPU; i++)
+		vcpu->arch.apf.gfns[i] = ~0;
+}
 
+/*
+ * Add gfn to the hash table. It's ensured there is a free entry
+ * when this function is called.
+ */
+static void kvm_async_pf_hash_add(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	u32 key = kvm_async_pf_hash_fn(gfn);
+
+	while (vcpu->arch.apf.gfns[key] != ~0)
+		key = kvm_async_pf_hash_next(key);
+
+	vcpu->arch.apf.gfns[key] = gfn;
+}
+
+static u32 kvm_async_pf_hash_slot(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	u32 key = kvm_async_pf_hash_fn(gfn);
+	int i;
+
+	for (i = 0; i < ASYNC_PF_PER_VCPU; i++) {
+		if (vcpu->arch.apf.gfns[key] == gfn ||
+		    vcpu->arch.apf.gfns[key] == ~0)
+			break;
+
+		key = kvm_async_pf_hash_next(key);
+	}
+
+	return key;
+}
+
+static void kvm_async_pf_hash_remove(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	u32 i, j, k;
+
+	i = j = kvm_async_pf_hash_slot(vcpu, gfn);
+	while (true) {
+		vcpu->arch.apf.gfns[i] = ~0;
+
+		do {
+			j = kvm_async_pf_hash_next(j);
+			if (vcpu->arch.apf.gfns[j] == ~0)
+				return;
+
+			k = kvm_async_pf_hash_fn(vcpu->arch.apf.gfns[j]);
+			/*
+			 * k lies cyclically in ]i,j]
+			 * |    i.k.j |
+			 * |....j i.k.| or  |.k..j i...|
+			 */
+		} while ((i <= j) ? (i < k && k <= j) : (i < k || k <= j));
+
+		vcpu->arch.apf.gfns[i] = vcpu->arch.apf.gfns[j];
+		i = j;
+	}
+}
+
+bool kvm_async_pf_hash_find(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	u32 key = kvm_async_pf_hash_slot(vcpu, gfn);
+
+	return vcpu->arch.apf.gfns[key] == gfn;
+}
+
+static inline int kvm_async_pf_read_cache(struct kvm_vcpu *vcpu,
+					  u32 offset, u32 *val)
+{
+	return kvm_read_guest_offset_cached(vcpu->kvm,
+					    &vcpu->arch.apf.cache,

arch/arm64/kvm/async_pf.c line 110

Quoted 10 lines
arch/arm64/kvm/async_pf.c new file +10 -0
+					    val, offset, sizeof(*val));
+}
+
+static inline int kvm_async_pf_write_cache(struct kvm_vcpu *vcpu,
+					   u32 offset, u32 val)
+{
+	return kvm_write_guest_offset_cached(vcpu->kvm,
+					     &vcpu->arch.apf.cache,
+					     &val, offset, sizeof(val));
+}

This hash stuff doesn't look at all specific to arm64. Could it be moved to core code or a shared header file so multiple architectures only need one copy of the code. If they need to be different, it would be good to know why.

Thanks,

James

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#8 lore raw

Re: [PATCH 4/6] arm64/kvm: Support async page fault

Hi James,

Quoted 3 lines · James Morse

[my mail client went a bit nuts - it thinks this got sent already, sorry if you received it twice!]

Thanks for your time on this. No problem at all and sorry for late response :)

Quoted 7 lines

I only got so far through this, so may have focussed on the wrong things. This patch has too many things going on. Please split it up.

I think the page-fault and page-present should be done separately, with the gubbins the core-code needs added first. The cap and corresponding documentation should be added last so the whole thing bisects.

Quoted 7 lines

It looks like the guest and hypervisor parts are already split up, this is helpful, thanks. Please ensure that the descriptions are clear on this distinction too. Its not at all clear where where the 'reschedule ipi' comes from in this commit message.

I'd really like it if as much of the guest parts as possible lived in common code. There is nothing arch-specific about an interrupt and reading from shared memory.

Yes, I will split this patch into 3 patches in next revision at least:
[1] support page-not-present notification
[2] support page-ready notification
[3] control/configuration path support

Sure, I will improve the change log to make things more understandable.

I need to evaluate the possiblity of make guest parts common. I guess it guess it might be hard. Currently, there are 3 architectures support or will support the feature: x86/s390/arm64. s390 has unique implementation. Even x86/arm64 share common implementations, but still different to some extents. For example, x86 uses NMI/IRQ for the notifications, but arm64 uses SDEI/IRQ for them. The resources for SDEI should be pre-allocated on arm64, but x86 doesn't have the requirement.

Quoted 14 lines · Gavin Shan
Quoted 7 lines

There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.

Yes, its the cost of abstracting the machine's resources and invisibly carving them up.

Quoted 1 line

In order to resolve the issue and improve the guest's performance,

Quoted 8 lines

Why is it an issue? You've missed the important part of the commit message: why we should make this change.

This needs to mention postcopy live migration, which is the reason for doing any of this.

I don't think "I overcommited and swapped the guest to disk" is at all compelling. (as the doctor says: don't do that then)

Yes, live-migration should be the justification in next revision. Thanks for the reminder :)

Quoted 25 lines
Quoted 2 lines

this introduces the feature (asychronous page fault). A page-not-present

(asynchronous)

Quoted 6 lines

notification is fired to guest if the requested page isn't ready so that the guest can reschedule current process and pick another process to run. Another page-ready notification is sent to guest so that the process, which was rescheduled previously, can be picked up to run. With this,

the guest needn't necessarily suspend itself on stage two page fault.

Be suspended by the hypervisor until the resources are available. Normally the guest has no idea this is happening.

Quoted 1 line

These two notificatons are associated by so-called vCPU scoped tokens,

(notifications)

Quoted 3 lines
Quoted 1 line

which is the combination of incremental sequence number and vCPU index

(@vcpu->vcpu_idx). Besides, the notifications are conveyed by SDEI and PPI, whose numbers are specified by guest through SMCCC interface.

... so you don't need the ACPI table?

Quoted 9 lines

The 'asynchrononous page fault' notification obviously needs to be (er,) synchronous, so it has to be per-cpu.

Why is the page-ready IRQ a PPI? We have a limited number of these, so shouldn't use one unless it _needs_ to be per-cpu.

'vcpu index' should probably be the MPIDR_EL1, which is something both the host and guest already know that identifies the vcpu. Inventing something new isn't necessary.

ACPI table is needed to pass the PPI number from qemu to guest. The PPI number is determined by qemu. The reason why we need PPI is the asynchronous page fault notification (page-ready) is delivered on per-vCPU basis.

The 'vCPU index' was inherited from x86 implementation. Actually, the information isn't used because the SDEI event and PPI are always sent to the target vCPU. It means the incremental sequence number is enough to identify the associated asynchronous page fault events. So we needn't the vCPU index conveyed as part of the token and I will drop it in next revision.

Quoted 5 lines
Quoted 2 lines

The asynchronous page fault always starts with a background woker if

(worker. Please run your commit-messages through a spell checker)

Sure, Thanks.

Quoted 6 lines
Quoted 3 lines

current vCPU has enough token space and no pending notifications. After the worker, which populates the requested page background, is started,

what does 'requested page background' mean? (missing the words 'in the'?)

Yeah, "in the" is missed and will be fixed if needed in next revision.

Quoted 10 lines
Quoted 9 lines

a page-not-present notification, together with an unique token, are sent to guest through the specified private SDEI event. The notification is acknowledged by clearing the shared cache (@vcpu->apf.cache). The current process is marked for waiting completion of asynchronous page fault. A subsequent (reschedule) IPI is sent to current vCPU so that the current process can be reschedule and suspended until the asynchronous page is completed. In the meanwhile, the woker populates the requested page and it's queued to the completion queue once the task is completed.

Quoted 6 lines
Quoted 3 lines

At this
point or the acknoledgement (by SMCCC) on pending PPI, a PPI is sent to guest for page-ready notification.

A PPI is made pending, when the PPI is pending?

Another PPI can't be fired from host to guest before the previous PPI is delivered and handled.

Quoted 6 lines
Quoted 3 lines

The guest uses the associated token
to locate the suspended process and wakes it up.

I hope the 'token' is nothing more than the IPA.

Correct. IPA can be used as token either except IPA is 64-bits in length, but token is 32-bits long.

Quoted 61 lines
Quoted 3 lines

In order to fulfil the functions, there are several SMCCC function IDs introduced and outlined as below:

Please add a documentation file for these for the benefit of other guests (e.g. kvm-unit-test) describing these in more detail, their parameters, the expected lifecycle etc.

Quoted 37 lines

ARM_SMCCC_KVM_FUNC_APF_VERSION
Returns the version, which can be used to identify ABI changes in the future.
ARM_SMCCC_KVM_FUNC_APF_ENABLE
Enable or disable asynchronous page fault on current vCPU.
ARM_SMCCC_KVM_FUNC_APF_TOKEN_NUM
Returns maximal number of tokens that current vCPU can have. It's used by guest to allocate the required resources.
ARM_SMCCC_KVM_FUNC_APF_SDEI
ARM_SMCCC_KVM_FUNC_APF_IRQ
Used by guest to confiugre the SDEI event and PPI numbers.
ARM_SMCCC_KVM_FUNC_APF_IRQ_ACK
Acknowledge the page-ready notificaton in the PPI handler.

arch/arm64/include/asm/kvm_host.h +7 -0
index b6c9851b2a65..2e3bba6a99c3 100644
@@ -46,6 +46,7 @@
  #define KVM_REQ_RECORD_STEAL	KVM_ARCH_REQ(3)
  #define KVM_REQ_RELOAD_GICv4	KVM_ARCH_REQ(4)
  #define KVM_REQ_SDEI		KVM_ARCH_REQ(5)
+#define KVM_REQ_APF_READY	KVM_ARCH_REQ(6)
 
  #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
  				     KVM_DIRTY_LOG_INITIALLY_SET)
@@ -239,6 +240,23 @@ struct vcpu_reset_state {
  	bool		reset;
  };
 
+#ifdef CONFIG_KVM_ASYNC_PF
+
+/* Should be a power of two number */
+#define ASYNC_PF_PER_VCPU	64
+
+/*

arch/arm64/include/asm/kvm_host.h struct vcpu_reset_state { · line 258

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +10 -0
+ * The association of gfn and token. The token will be sent to guest as
+ * page fault address. Also, the guest could be in aarch32 mode. So its
+ * length should be 32-bits.
+ */
+struct kvm_arch_async_pf {
+	u32     token;
+	gfn_t   gfn;
+	u32     esr;
+};
+#endif /* CONFIG_KVM_ASYNC_PF */

#ifdef-ing a struct makes it harder to use IS_ENABLED() in the C code, which in turn makes it harder to get complete coverage from the CI builders without aligning the stars to hit the right randconfig combination.

Quoted 3 lines

Please avoid doing this unless its absolutely necessary. (its not like a struct definition takes up any space in the binary)

Ok. The #ifdef will be dropped in next revision.

Quoted 14 lines

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 368

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +2 -0
  struct kvm_vcpu_arch {
  	struct kvm_cpu_context ctxt;
  	void *sve_state;
@@ -346,6 +364,22 @@ struct kvm_vcpu_arch {
  	struct kvm_sdei_vcpu_event	*sdei_normal_event;
  	struct kvm_sdei_vcpu_event	*sdei_critical_event;
  	struct list_head		sdei_events;
+
+#ifdef CONFIG_KVM_ASYNC_PF

See above. Any reason these can't be dynamically allocated if the feature is enabled by the VMM? (You're using a PPI, this has to be allocated by the VMM).

Quoted 2 lines

Not all guests would use this, so its a waste of space if its enabled but not in use.

Good idea. It will be dynamically allocated in next revision :)

Quoted 12 lines

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch {

Quoted 8 lines
arch/arm64/include/asm/kvm_host.h +5 -0
+	struct {
+		struct gfn_to_hva_cache	cache;
 
+		gfn_t			gfns[ASYNC_PF_PER_VCPU];
+		u64			control_block;
 
+		bool			send_user_only;

Hmmm...

Quoted 7 lines

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch {

Quoted 4 lines
arch/arm64/include/asm/kvm_host.h +3 -0
+		u64			sdei_event_num;
+		u32			irq;
 
+		u16			id;

a per-vcpu id? KVM already has one, and the architecture has one that the guest can see too.

It's different from that one. It's the incremental sequence number associated with asynchronous page fault events.

Quoted 31 lines
Quoted 17 lines
arch/arm64/include/asm/kvm_host.h +5 -0
+		bool			notpresent_pending;
+		u32			notpresent_token;
+		bool			pageready_pending;
+	} apf;
+#endif
  };
 
  /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
arch/arm64/include/asm/kvm_para.h new file +0 -0
new file mode 100644
index 000000000000..0ea481dd1c7a

arch/arm64/include/asm/kvm_para.h line 10

Quoted 10 lines
arch/arm64/include/asm/kvm_para.h new file +8 -0
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM_KVM_PARA_H
+#define _ASM_ARM_KVM_PARA_H
 
+#include <uapi/asm/kvm_para.h>
 
+static inline bool kvm_check_and_clear_guest_paused(void)
+{
+	return false;
+}

This is part of kvmclock? aarch64 has an architecture specific way of doing that stuff. Why is this here?!

Quoted 3 lines

(if you've turned something on that needs these stubs, its probably worth doing that in an preparation patch so you can explain why, instead of hiding it in here)

It's used by the guest to retrieve the supported features through SMCCC (KVM specific feature call). Yep, it should be included in a separate patch, as part of the prepatory work to support asynchronous page fault for the guest.

Quoted 31 lines
Quoted 25 lines
arch/arm64/include/asm/kvm_para.h new file +16 -0
+static inline unsigned int kvm_arch_para_features(void)
+{
+	return 0;
+}
+
+static inline unsigned int kvm_arch_para_hints(void)
+{
+	return 0;
+}
+
+static inline bool kvm_para_available(void)
+{
+	return false;
+}
+
+#endif /* _ASM_ARM_KVM_PARA_H */
arch/arm64/include/uapi/asm/Kbuild +0 -0
index 602d137932dc..f66554cd5c45 100644

arch/arm64/include/uapi/asm/Kbuild line 3

Quoted 3 lines
arch/arm64/include/uapi/asm/Kbuild +0 -2
@@ -1,3 +1 @@
  # SPDX-License-Identifier: GPL-2.0
-
-generic-y += kvm_para.h

The arch specific version is practically the same as the asm-generic one.

Quoted 2 lines

Please explain why you need to do this in a preparatory patch.

Sure, will do.

Quoted 33 lines
Quoted 20 lines
arch/arm64/include/uapi/asm/kvm_para.h new file +13 -0
new file mode 100644
index 000000000000..f0dbe86c2374
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _UAPI_ASM_ARM_KVM_PARA_H
+#define _UAPI_ASM_ARM_KVM_PARA_H
+
+#include <linux/types.h>
+
+#define KVM_FEATURE_ASYNC_PF	0
+
+/* Async PF */
+#define KVM_ASYNC_PF_ENABLED		(1 << 0)
+#define KVM_ASYNC_PF_SEND_ALWAYS	(1 << 1)
+
+#define KVM_PV_REASON_PAGE_NOT_PRESENT	1

arch/arm64/include/uapi/asm/kvm_para.h line 23

Quoted 10 lines
arch/arm64/include/uapi/asm/kvm_para.h new file +10 -0
+#define KVM_PV_REASON_PAGE_READY	2
+
+struct kvm_vcpu_pv_apf_data {
+	__u32	reason;
+	__u32	token;
+	__u8	pad[56];
+	__u32	enabled;
+};
+
+#endif /* _UAPI_ASM_ARM_KVM_PARA_H */

Presumably this is the reason to carry around an extra copy of kvm_para.h.

Quoted 2 lines

Wouldn't it be better to unpick that dependency instead?

Yes, it's the interface (or shared data struct) between host and guest. Sorry that I don't understand your comments here. Do you mean to make this common across multiple architectures?

Quoted 21 lines
Quoted 21 lines
arch/arm64/kvm/arm.c +6 -0
index 4bec6c9ece18..1a57dc5dbf14 100644
@@ -225,6 +225,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
  		 */
  		r = 1;
  		break;
+#ifdef CONFIG_KVM_ASYNC_PF
+	case KVM_CAP_ASYNC_PF:
+	case KVM_CAP_ASYNC_PF_INT:
+		r = 1;
+		break;
+#endif
  	default:
  		r = kvm_arch_vm_ioctl_check_extension(kvm, ext);
  		break;
@@ -274,6 +280,9 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
  	bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
 
Quoted 8 lines

arch/arm64/kvm/arm.c int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · line 285

Quoted 4 lines
arch/arm64/kvm/arm.c +3 -0
  	vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO;
+#ifdef CONFIG_KVM_ASYNC_PF
+	vcpu->arch.apf.control_block = 0UL;
+#endif

kvm_vm_ioctl_create_vcpu() allocates vcpu with kmem_cache_zalloc(), so this field will already be zero.

Yes. The @apf will be dynamically allocated in next revision as you suggested above :)

Quoted 12 lines

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 436

Quoted 7 lines
arch/arm64/kvm/arm.c +0 -2
@@ -432,8 +441,32 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
  int kvm_arch_vcpu_runnable(struct kvm_vcpu *v)
  {
  	bool irq_lines = *vcpu_hcr(v) & (HCR_VI | HCR_VF);
-	return ((irq_lines || kvm_vgic_vcpu_pending_irq(v))
-		&& !v->arch.power_off && !v->arch.pause);

As you are adding a conditional __copy_from_user() in here, please add an unconditional might_sleep() at the top of this function. This lets us catch locations that will sometimes break the first time they happen.

Ok.

Quoted 10 lines

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 449

Quoted 6 lines
arch/arm64/kvm/arm.c +5 -0
+	if ((irq_lines || kvm_vgic_vcpu_pending_irq(v)) &&
+	    !v->arch.power_off && !v->arch.pause)
+		return true;
+
+#ifdef CONFIG_KVM_ASYNC_PF

Please use IS_ENABLED() here, #ifdeffery make this stuff harder to read and harder to build-test.

Quoted 3 lines

(if you can come up with a readable IS_ENABLED() way of returning the _check_extension() result - even better!)

Ok.

Quoted 8 lines
Quoted 7 lines
arch/arm64/kvm/arm.c +6 -0
+	if (v->arch.apf.control_block & KVM_ASYNC_PF_ENABLED) {
+		u32 val;
+		int ret;
+
+		if (!list_empty_careful(&v->async_pf.done))
+			return true;
Quoted 5 lines

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 458

Quoted 2 lines
arch/arm64/kvm/arm.c +2 -0
+		ret = kvm_read_guest_offset_cached(v->kvm, &v->arch.apf.cache,
+						   &val, 0, sizeof(val));

You added a helper for this: kvm_async_pf_read_cache()...

Yep, the helper should be used here.

Quoted 11 lines

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 465

Quoted 8 lines
arch/arm64/kvm/arm.c +7 -0
+		if (ret || val)
+			return true;
+
+		ret = kvm_read_guest_offset_cached(v->kvm, &v->arch.apf.cache,
+						   &val, 4, sizeof(val));
+		if (ret || val)
+			return true;

These 0 and 4 really need some kind of name to give a hint as to what they are for.

Yep, that would be better. I probably use offsetof() in next revision.

Quoted 5 lines

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 464

Quoted 2 lines
arch/arm64/kvm/arm.c +1 -0
+	}

This really ought to wrapped up in a helper that gives a hint as to what it is doing.

Quoted 9 lines

There are two magic locations in guest memory that means the guest controls whether it is runnable or not... this sounds like a source of hard-to-debug issues.

I'd much prefer it if the guest only ever got blocked in the stage2 fault handler, as it does today. This mechanism just provides information about what is going on at stage2, which allows the guest to avoid getting blocked if it can do something about it.

Giving the guest access to the controls doesn't look right.

Yeah, It's a nice comment and I think @notpresent_pending and @pageready_pending in struct kvm_vcpu_arch::apf can alternatively serve the purpose.

Quoted 11 lines
Quoted 11 lines
arch/arm64/kvm/arm.c +3 -0
+#endif
+
+	return false;
  }
 
  bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu)
@@ -624,6 +657,11 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
  		if (kvm_check_request(KVM_REQ_VCPU_RESET, vcpu))
  			kvm_reset_vcpu(vcpu);
 
Quoted 8 lines

arch/arm64/kvm/arm.c static void check_vcpu_requests(struct kvm_vcpu *vcpu) · line 663

Quoted 4 lines
arch/arm64/kvm/arm.c +4 -0
+#ifdef CONFIG_KVM_ASYNC_PF
+		if (kvm_check_request(KVM_REQ_APF_READY, vcpu))
+			kvm_check_async_pf_completion(vcpu);
+#endif

(IS_ENABLED() would save you the link error ... providing a stub in core-code for kvm_check_async_pf_completion() would let you rely on this request never being made instead)

Ok.

Quoted 28 lines

arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · line 782

Quoted 12 lines
arch/arm64/kvm/arm.c +2 -1
  		if (kvm_check_request(KVM_REQ_SDEI, vcpu))
  			kvm_sdei_deliver(vcpu);
 
@@ -739,7 +777,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
  		smp_store_mb(vcpu->mode, IN_GUEST_MODE);
 
  		if (ret <= 0 || need_new_vmid_gen(&vcpu->kvm->arch.vmid) ||
-		    kvm_request_pending(vcpu)) {
 
+		    (kvm_request_pending(vcpu) &&
+		     READ_ONCE(vcpu->requests) != (1UL << KVM_REQ_APF_READY))) {

A double READ_ONCE() looks wrong. If you have to do this, presumably you didn't want a vcpu->request to begin with.

It looks like KVM_REQ_APF_READY only exists to move the page-ready work onto the vcpu thread because the vcpu got blocked further out by kvm_arch_vcpu_runnable(), and the second half of the stage2-fault now needs doing.

I think it would be simpler to only ever block in the stage2 fault handler. The guest can only ever be blocked on one stage2 fault at a time, so this abstraction to let it sleep for any number is pointless. If I'm waiting for ipa-4, and get restarted because ipa-6 is now available ... I'm going to fault on ipa-4 for a second time. There is no point restarting the guest until it can make progress again.

Quoted 9 lines
Quoted 1 line

From the first stage2 fault you can kick off the page-fault notification (which must be

synchronous, so can be notified without taking this fault again). If the vcpu takes a fault that was already notified, it clearly can't handle the page-fault notification and must be blocked until that page is available.

We should be able to depend on the regular wakeup from mm when the vcpu is blocked. The extra information from page-present lets the guest know it can now touch these pages without getting blocked ... they shouldn't need a wakeup.

I agree it will become simpler, but it's going to affect the performance greatly if I'm understanding your comments completely. In current design, multiple works associated with asynchronous page fault and one specific vCPU can work in the background at same time. It seems you're suggesting to have one work is available at once on one specific vCPU. If so, the performance will be affected greatly.

Quoted 119 lines
Quoted 119 lines
  			vcpu->mode = OUTSIDE_GUEST_MODE;
  			isb(); /* Ensure work in x_flush_hwstate is committed */
  			kvm_pmu_sync_hwstate(vcpu);
arch/arm64/kvm/async_pf.c new file +105 -0
new file mode 100644
index 000000000000..910cb3a1bdc2
@@ -0,0 +1,462 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright Gavin Shan, Redhat Inc 2020.
+ *
+ * Asynchronous page fault support
+ */
+
+#include <linux/arm-smccc.h>
+#include <linux/kvm_host.h>
+#include <asm/kvm_emulate.h>
+#include <kvm/arm_hypercalls.h>
+#include <kvm/arm_vgic.h>
+#include <asm/kvm_sdei.h>
 
+static inline u32 kvm_async_pf_hash_fn(gfn_t gfn)
+{
+	return hash_32(gfn & 0xffffffff, order_base_2(ASYNC_PF_PER_VCPU));
+}
+
+static inline u32 kvm_async_pf_hash_next(u32 key)
+{
+	return (key + 1) & (ASYNC_PF_PER_VCPU - 1);
+}
+
+static inline void kvm_async_pf_hash_reset(struct kvm_vcpu *vcpu)
+{
+	int i;
+
+	for (i = 0; i < ASYNC_PF_PER_VCPU; i++)
+		vcpu->arch.apf.gfns[i] = ~0;
+}
 
+/*
+ * Add gfn to the hash table. It's ensured there is a free entry
+ * when this function is called.
+ */
+static void kvm_async_pf_hash_add(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	u32 key = kvm_async_pf_hash_fn(gfn);
+
+	while (vcpu->arch.apf.gfns[key] != ~0)
+		key = kvm_async_pf_hash_next(key);
+
+	vcpu->arch.apf.gfns[key] = gfn;
+}
+
+static u32 kvm_async_pf_hash_slot(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	u32 key = kvm_async_pf_hash_fn(gfn);
+	int i;
+
+	for (i = 0; i < ASYNC_PF_PER_VCPU; i++) {
+		if (vcpu->arch.apf.gfns[key] == gfn ||
+		    vcpu->arch.apf.gfns[key] == ~0)
+			break;
+
+		key = kvm_async_pf_hash_next(key);
+	}
+
+	return key;
+}
+
+static void kvm_async_pf_hash_remove(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	u32 i, j, k;
+
+	i = j = kvm_async_pf_hash_slot(vcpu, gfn);
+	while (true) {
+		vcpu->arch.apf.gfns[i] = ~0;
+
+		do {
+			j = kvm_async_pf_hash_next(j);
+			if (vcpu->arch.apf.gfns[j] == ~0)
+				return;
+
+			k = kvm_async_pf_hash_fn(vcpu->arch.apf.gfns[j]);
+			/*
+			 * k lies cyclically in ]i,j]
+			 * |    i.k.j |
+			 * |....j i.k.| or  |.k..j i...|
+			 */
+		} while ((i <= j) ? (i < k && k <= j) : (i < k || k <= j));
+
+		vcpu->arch.apf.gfns[i] = vcpu->arch.apf.gfns[j];
+		i = j;
+	}
+}
+
+bool kvm_async_pf_hash_find(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	u32 key = kvm_async_pf_hash_slot(vcpu, gfn);
+
+	return vcpu->arch.apf.gfns[key] == gfn;
+}
+
+static inline int kvm_async_pf_read_cache(struct kvm_vcpu *vcpu,
+					  u32 offset, u32 *val)
+{
+	return kvm_read_guest_offset_cached(vcpu->kvm,
+					    &vcpu->arch.apf.cache,
+					    val, offset, sizeof(*val));
+}
+
+static inline int kvm_async_pf_write_cache(struct kvm_vcpu *vcpu,
+					   u32 offset, u32 val)
+{
+	return kvm_write_guest_offset_cached(vcpu->kvm,
Quoted 9 lines

arch/arm64/kvm/async_pf.c line 110

Quoted 3 lines
arch/arm64/kvm/async_pf.c new file +3 -0
+					     &vcpu->arch.apf.cache,
+					     &val, offset, sizeof(val));
+}

This hash stuff doesn't look at all specific to arm64. Could it be moved to core code or a shared header file so multiple architectures only need one copy of the code. If they need to be different, it would be good to know why.

Ok. It will be moved to common part shared by x86 and arm64 in next revision.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#9 lore raw

PATCH 5/6 drivers/acpi: Import ACPI APF table

In reply to Gavin Shan

This defines the struct for ACPI APF table. The information included in this table will be used by guest kernel to retrieve SDEI event number, PPI number and its triggering properties:

* @sdei_event: number of SDEI event used for page-not-present
* @interrupt: PPI used for page-ready
* @interrupt_flags: PPI's mode and polarity

---
 include/acpi/actbl2.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
include/acpi/actbl2.h +18 -0
index ec66779cb193..2eb715f4463b 100644
@@ -44,6 +44,7 @@
 #define ACPI_SIG_SDEI           "SDEI"	/* Software Delegated Exception Interface Table */
 #define ACPI_SIG_SDEV           "SDEV"	/* Secure Devices table */
 #define ACPI_SIG_NHLT           "NHLT"	/* Non-HDAudio Link Table */
+#define ACPI_SIG_APFT           "APFT"  /* Asynchronous Page Fault Table */
 
 /*
  * All tables must be byte-packed to match the ACPI specification, since
@@ -1737,6 +1738,23 @@ struct acpi_sdev_pcie_path {
 	u8 function;
 };
 
+/*******************************************************************************
+ *
+ * APFT - Asynchronous Page Fault Table
+ *
+ * Customized table used for asynchronous page fault on ARM
+ *
+ ******************************************************************************/
+struct acpi_table_apft {
+	struct acpi_table_header header;
+	u32 sdei_event;
+	u32 interrupt;
+	u32 interrupt_flags;
+};
+
+#define ACPI_APFT_INTERRUPT_MODE	(1)
+#define ACPI_APFT_INTERRUPT_POLARITY	(1<<1)
+
 /* Reset to default packing */
 
 #pragma pack()
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#10 lore raw

Re: [PATCH 5/6] drivers/acpi: Import ACPI APF table

Hi Gavin,

Quoted 4 lines · Gavin Shan

This defines the struct for ACPI APF table. The information included in this table will be used by guest kernel to retrieve SDEI event number, PPI number and its triggering properties:

Quoted 8 lines

* @sdei_event: number of SDEI event used for page-not-present
* @interrupt: PPI used for page-ready
* @interrupt_flags: PPI's mode and polarity

---
 include/acpi/actbl2.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

Whoa! This is not how changes to ACPI work!

We have to work out what is needed, then the standard has to be updated, then the upstream acpica project, then these files get synced back across from the 'upstream' acpica project...

If you need an ACPI table, we need something for DT too. I don't think a well designed paravirt interface shouldn't need this. Wasn't that the whole point of the KVM "vendor specific" services?!

The cover-letter message talks of shared memory, which this doesn't describe.

Ideally this stuff would be discovered via SMCCC, or a (readonly?) page of that shared memory. That way its the same regardless of ACPI or DT.

Thanks,

James
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#11 lore raw

PATCH 6/6 arm64/kernel: Support async page fault

In reply to Gavin Shan

This supports the asynchronous page fault, basing on the services provided by host. The asynchronous page fault starts with private SDEI event, whose number is specified by the guest. The guest marks the current running process as suspended state by the newly added flag (TIF_ASYNC_PF) and associating it with a wait queue. In the mean while, the process is put into the pre-allocated table, whose index is indeitified by the associated token. In the SDEI event handler, a followup (reschedule) IPI is sent to current CPU to force context switching on current running process. During the context switching, the flag (TIF_ASYNC_PF) is checked and the process waits on the associated wait queue.

The guest receives a PPI, whose number is specified by it, after the page becomes ready on host side. The pre-allocated table is searched using the received token to locate the associated process. The process is waken up after it's found. The previously suspended process resumes for execution and the asynchronous page fault is completed so far.

The implementation is guarded by CONFIG_KVM_GUEST. Also, the boot parameter "no-kvmapf" can be used to disable this feature. Besides, this feature has to be enabled after SDEI service is settled down because this depends on SDEI service.

---
 arch/arm64/Kconfig                   |  11 +
 arch/arm64/include/asm/kvm_para.h    |  26 +-
 arch/arm64/include/asm/processor.h   |   1 +
 arch/arm64/include/asm/thread_info.h |   4 +-
 arch/arm64/kernel/Makefile           |   1 +
 arch/arm64/kernel/kvm.c              | 478 +++++++++++++++++++++++++++
 arch/arm64/kernel/signal.c           |  17 +
 7 files changed, 531 insertions(+), 7 deletions(-)
 create mode 100644 arch/arm64/kernel/kvm.c
arch/arm64/Kconfig +11 -0
index e11b4ea06127..5543c0d8159e 100644
@@ -1053,6 +1053,17 @@ config PARAVIRT
 	  under a hypervisor, potentially improving performance significantly
 	  over full virtualization.
 
+config KVM_GUEST
+	bool "KVM Guest Support"
+	depends on PARAVIRT
+	default y
+	help
+	  This option enables various optimizations for running under the KVM
+	  hypervisor. Overhead for the kernel when not running inside KVM should
+	  be minimal.
+
+	  In case of doubt, say Y
+
 config PARAVIRT_TIME_ACCOUNTING
 	bool "Paravirtual steal time accounting"
 	select PARAVIRT
arch/arm64/include/asm/kvm_para.h +20 -6
index 0ea481dd1c7a..1f038652cdcd 100644
@@ -3,6 +3,20 @@
 #define _ASM_ARM_KVM_PARA_H
 
 #include <uapi/asm/kvm_para.h>
+#include <linux/of.h>
+#include <asm/hypervisor.h>
+
+#ifdef CONFIG_KVM_GUEST
+static inline int kvm_para_available(void)
+{
+	return 1;
+}
+#else
+static inline int kvm_para_available(void)
+{
+	return 0;
+}
+#endif /* CONFIG_KVM_GUEST */
 
 static inline bool kvm_check_and_clear_guest_paused(void)
 {
@@ -11,7 +25,12 @@ static inline bool kvm_check_and_clear_guest_paused(void)
 
 static inline unsigned int kvm_arch_para_features(void)
 {
-	return 0;
+	unsigned int features = 0;
+
+	if (kvm_arm_hyp_service_available(ARM_SMCCC_KVM_FUNC_APF))
+		features |= (1 << KVM_FEATURE_ASYNC_PF);
+
+	return features;
 }
 
 static inline unsigned int kvm_arch_para_hints(void)
@@ -19,9 +38,4 @@ static inline unsigned int kvm_arch_para_hints(void)
 	return 0;
 }
 
-static inline bool kvm_para_available(void)
-{
-	return false;
-}
-
 #endif /* _ASM_ARM_KVM_PARA_H */
arch/arm64/include/asm/processor.h +1 -0
index 240fe5e5b720..afd2e04cac3d 100644
@@ -151,6 +151,7 @@ struct thread_struct {
 	struct ptrauth_keys_user	keys_user;
 	struct ptrauth_keys_kernel	keys_kernel;
 #endif
+	void				*data;
 };
 
 static inline void arch_thread_struct_whitelist(unsigned long *offset,
arch/arm64/include/asm/thread_info.h +3 -1
index 5e784e16ee89..5605dc9d2bd3 100644
@@ -67,6 +67,7 @@ void arch_release_task_struct(struct task_struct *tsk);
 #define TIF_FOREIGN_FPSTATE	3	/* CPU's FP state is not current's */
 #define TIF_UPROBE		4	/* uprobe breakpoint or singlestep */
 #define TIF_FSCHECK		5	/* Check FS is USER_DS on return */
+#define TIF_ASYNC_PF		6	/* Asynchronous page fault */
 #define TIF_SYSCALL_TRACE	8	/* syscall trace active */
 #define TIF_SYSCALL_AUDIT	9	/* syscall auditing */
 #define TIF_SYSCALL_TRACEPOINT	10	/* syscall tracepoint for ftrace */
@@ -86,6 +87,7 @@ void arch_release_task_struct(struct task_struct *tsk);
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
 #define _TIF_FOREIGN_FPSTATE	(1 << TIF_FOREIGN_FPSTATE)
+#define _TIF_ASYNC_PF		(1 << TIF_ASYNC_PF)
 #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
 #define _TIF_SYSCALL_AUDIT	(1 << TIF_SYSCALL_AUDIT)
 #define _TIF_SYSCALL_TRACEPOINT	(1 << TIF_SYSCALL_TRACEPOINT)
@@ -99,7 +101,7 @@ void arch_release_task_struct(struct task_struct *tsk);
 
 #define _TIF_WORK_MASK		(_TIF_NEED_RESCHED | _TIF_SIGPENDING | \
 				 _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \
-				 _TIF_UPROBE | _TIF_FSCHECK)
+				 _TIF_UPROBE | _TIF_FSCHECK | _TIF_ASYNC_PF)
 
 #define _TIF_SYSCALL_WORK	(_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \
 				 _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
arch/arm64/kernel/Makefile +1 -0
index a561cbb91d4d..76472934689b 100644
@@ -49,6 +49,7 @@ obj-$(CONFIG_ACPI)			+= acpi.o
 obj-$(CONFIG_ACPI_NUMA)			+= acpi_numa.o
 obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL)	+= acpi_parking_protocol.o
 obj-$(CONFIG_PARAVIRT)			+= paravirt.o
+obj-$(CONFIG_KVM_GUEST)			+= kvm.o
 obj-$(CONFIG_RANDOMIZE_BASE)		+= kaslr.o
 obj-$(CONFIG_HIBERNATION)		+= hibernate.o hibernate-asm.o
 obj-$(CONFIG_KEXEC_CORE)		+= machine_kexec.o relocate_kernel.o	\
arch/arm64/kernel/kvm.c new file +478 -0

478 lines not on this page · raw message

arch/arm64/kernel/signal.c +17 -0
index 3b4f31f35e45..42328efbc946 100644
@@ -921,6 +921,23 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,
 		/* Check valid user FS if needed */
 		addr_limit_user_check();
 
+		if (thread_flags & _TIF_ASYNC_PF) {
+			struct swait_queue_head *wq =
+				READ_ONCE(current->thread.data);
+			DECLARE_SWAITQUEUE(wait);
+
+			local_daif_restore(DAIF_PROCCTX_NOIRQ);
+
+			do {
+				prepare_to_swait_exclusive(wq,
+					&wait, TASK_UNINTERRUPTIBLE);
+				if (!test_thread_flag(TIF_ASYNC_PF))
+					break;
+
+				schedule();
+			} while (test_thread_flag(TIF_ASYNC_PF));
+		}
+
 		if (thread_flags & _TIF_NEED_RESCHED) {
 			/* Unmask Debug and SError for the next task */
 			local_daif_restore(DAIF_PROCCTX_NOIRQ);
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#12 lore raw
In reply to Gavin Shan

Hi Gavin,

I think this series would benefit from being in smaller pieces. I got lost in patch 4 for quite a while. Suggestion on where to do that in patch 4.

Quoted 7 lines · Gavin Shan

There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.

You really need to use postcopy live migration to justify these changes. Otherwise the story here is "over-commited hosts suck", which I don't think anyone cares about.

Quoted 3 lines

This series introduces the feature (asynchronous page fault) to resolve the issue and improve the guest's performance. It depends on the series to support SDEI virtualization and refactoring SDEI client driver.

SDEI gives you an NMI ... which you use to set a TIF flag. This can only work reliably for user-space. So much so that you have code in the hypervisor to only deliver the NMI ... when in user-space.
The only reason you would need an NMI is to interrupt interrupts-masked code. Linux can't reschedule when this is the case.

I can only conclude, you really don't need an NMI here.

Why couldn't we use an IRQ here, it would be a lot simpler? ... the reason is the arm architecture can't guarantee us that we take the irq when there is also a stage2 fault for the first instruction.
I reckon we can work around this in the hypervisor:
https://lore.kernel.org/r/[email protected]

My problem with SDEI is, you don't really need an NMI, and it creates extra in-kernel state that has to be migrated. I think having this state in the kernel complicates the user-space handling of SIGBUS_MCEERR_AO signals that don't get delivered to a vCPU thread.

Thanks,

James
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#13 lore raw
Quoted 7 lines · James Morse

SDEI gives you an NMI ... which you use to set a TIF flag. This can only work reliably for user-space. So much so that you have code in the hypervisor to only deliver the NMI ... when in user-space.
The only reason you would need an NMI is to interrupt interrupts-masked code. Linux can't reschedule when this is the case.

I can only conclude, you really don't need an NMI here.

I don't think the issue is that you want an NMI. It is just that the synchronous interruption that we want is exactly the same as a SDEI, and so is the notification reply from the guest to the host (e.g. accept the async pagefault or process it synchronously).

Yes, it's more code, but at least in x86 world we tried hard not to invent new paravirtualized mechanisms if we could avoid it, especially in the host->guest direction, and it's almost always paid off. This is because in case we don't get things right, it's much much harder to fix them on both the hypervisor and the guest side; by relying on existing code that work on bare metal hardware, the guest side exists already and you can develop against it.

Paolo

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#14 lore raw
In reply to James Morse

Hi James,

Quoted 3 lines · James Morse

I think this series would benefit from being in smaller pieces. I got lost in patch 4 for quite a while. Suggestion on where to do that in patch 4.

Yes, I will split the patches into small ones for easier review in next revision. Thanks for your comments :)

Quoted 10 lines · Gavin Shan
Quoted 7 lines

There are two stages of page fault. The guest kernel is responsible for handling stage one page fault, while the host kernel is to take care of the stage two page fault. When page fault is triggered because of stage two page fault, the guest is suspended until the requested memory (page) is populated. Sometimes, the cost to populate the requested page isn't cheap and can take hundreds of milliseconds in extreme cases. This impacts the overall guest's performance.

Quoted 3 lines

You really need to use postcopy live migration to justify these changes. Otherwise the story here is "over-commited hosts suck", which I don't think anyone cares about.

Yes, I will use live migration as the justification in next revision :)

Quoted 21 lines
Quoted 4 lines

This series introduces the feature (asynchronous page fault) to resolve the issue and improve the guest's performance. It depends on the series to support SDEI virtualization and refactoring SDEI client driver.

SDEI gives you an NMI ... which you use to set a TIF flag. This can only work reliably for user-space. So much so that you have code in the hypervisor to only deliver the NMI ... when in user-space.
The only reason you would need an NMI is to interrupt interrupts-masked code. Linux can't reschedule when this is the case.

I can only conclude, you really don't need an NMI here.

Why couldn't we use an IRQ here, it would be a lot simpler? ... the reason is the arm architecture can't guarantee us that we take the irq when there is also a stage2 fault for the first instruction.
I reckon we can work around this in the hypervisor:
https://lore.kernel.org/r/[email protected]

Quoted 5 lines

My problem with SDEI is, you don't really need an NMI, and it creates extra in-kernel state that has to be migrated. I think having this state in the kernel complicates the user-space handling of SIGBUS_MCEERR_AO signals that don't get delivered to a vCPU thread.

Currently, the asynchronous page fault is only supported for memory access in guest's userspace, but we needn't to be sticky to the use model in future. It means the asynchornous page fault could be supported for memory access in guest's kernel space where the interrupt can be disabled or masked. So NMI is needed and SDEI fits the use model very well as Paolo replied in another thread.

About the feature to support SDEI virtualization, I thought there might be some use cases where the emulated devices need inject SDEI event to guest. However, I'm not too much familiar with the architecture yet. If it's required by the emulated devices, there are more more justifications to merge the code. However, the implementation itself isn't simple and I would say it's complicated.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm