Support SDEI Virtualization
Subsystems (7)
v7 cover Support SDEI Virtualization
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")
The design is quite strightforward by following the specification. The (SDEI) events are classified into the shared and private ones according to their scope. The shared event is system or VM scoped, but the private event is vcpu scoped. This implementation doesn't support the shared event because all the needed events are private. Besides, the critial events aren't supported by the implementation either. It means all events are normal in terms of priority.
There are several objects (data structures) introduced to help on the event registration, enablement, disablement, unregistration, reset, delivery and handling.
* kvm_sdei_event_handler
SDEI event handler, which is provided through EVENT_REGISTER hypercall, is called when the SDEI event is delivered from host to guest.
* kvm_sdei_event_context
The saved (preempted) context when SDEI event is delivered for handling.
* kvm_sdei_vcpu
SDEI events and their states.
The patches are organized as below:
PATCH[01-02] Preparatory work to extend smccc_get_argx() and refactor
hypercall routing mechanism
PATCH[03] Adds SDEI virtualization infrastructure
PATCH[04-16] Supports various SDEI hypercalls and event handling
PATCH[17] Exposes SDEI capability
PATCH[18-19] Support SDEI migration
PATCH[20] Adds document about SDEI
PATCH[21-22] SDEI related selftest cases
The previous revisions can be found:
v6: https://lore.kernel.org/lkml/[email protected]/T/
v5: https://lore.kernel.org/kvmarm/[email protected]/
v4: https://lore.kernel.org/kvmarm/[email protected]/
v3: https://lore.kernel.org/kvmarm/[email protected]/
v2: https://lore.kernel.org/kvmarm/[email protected]/
v1: https://lore.kernel.org/kvmarm/[email protected]/
Testing
=======
[1] The selftest case included in this series works fine. The default SDEI
event, whose number is zero, can be registered, enabled, raised. The
SDEI event handler can be invoked.
[host]# pwd
/home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
[root@virtlab-arm01 kvm]# ./aarch64/sdei
NR_VCPUS: 2 SDEI Event: 0x00000000
--- VERSION
Version: 1.1 (vendor: 0x4b564d)
--- FEATURES
Shared event slots: 0
Private event slots: 0
Relative mode: No
--- PRIVATE_RESET
--- SHARED_RESET
--- PE_UNMASK
--- EVENT_GET_INFO
Type: Private
Priority: Normal
Signaled: Yes
--- EVENT_REGISTER
--- EVENT_ENABLE
--- EVENT_SIGNAL
Handled: Yes
IRQ: No
Status: Registered-Enabled-Running
PC/PSTATE: 000000000040232c 00000000600003c5
Regs: 0000000000000000 0000000000000000
0000000000000000 0000000000000000
--- PE_MASK
--- EVENT_DISABLE
--- EVENT_UNREGISTER
Result: OK[2] There are additional patches in the following repositories to create
procfs entries, allowing to inject SDEI event from host side. The
SDEI client in the guest side registers the SDEI default event, whose
number is zero. Also, the QEMU exports SDEI ACPI table and supports
migration for SDEI.
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")
[2.1] Start the guests and migrate the source VM to the destination
VM.
[host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,cores=3,threads=1 \
-m 1024M,slots=16,maxmem=64G \
: \
-kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x9000000 \
:
[host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,cores=3,threads=1 \
-m 1024M,slots=16,maxmem=64G \
: \
-kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x9000000 \
-incoming tcp:0:4444 \
:
[2.2] Check kernel log on the source VM. The SDEI service is enabled
and the default SDEI event (0x0) is enabled.
[guest-src]# dmesg | grep -i sdei
ACPI: SDEI 0x000000005BC80000 000024 \
(v00 BOCHS BXPC 00000001 BXPC 00000001)
sdei: SDEIv1.1 (0x4b564d) detected in firmware.
SDEI TEST: Version 1.1, Vendor 0x4b564d
sdei_init: SDEI event (0x0) registered
sdei_init: SDEI event (0x0) enabled
(qemu) migrate -d tcp:localhost:4444
[2.3] Migrate the source VM to the destination VM. Inject SDEI event
to the destination VM. The event is raised and handled.
(qemu) migrate -d tcp:localhost:4444
[host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1
[guest-dst]#
=========== SDEI Event (CPU#1) ===========
Event: 0000000000000000 Parameter: 00000000dabfdabf
PC: ffff800008cbb554 PSTATE: 00000000604000c5 SP: ffff800009c7bde0
Regs: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
ffff800016c28000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff800009399008
ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18
0000000000000000 0000000000000000 ffff000000339d00 0000000000000000
0000000000000000 ffff800009c7bde0 ffff800008cbb5c4
Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190
ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000 Changelog
=========
v7:
* Rebased to v5.19.rc1 (Gavin)
* Add hypercall ranges for routing (Oliver)
* Remove support to the critical event and redesigned data structures. Function names are also modified
as Oliver suggested (Oliver)
* Deliver event when it's enabled or the specific PE
is unmasked (Oliver)
* Improve EVENT_COMPLETE_AND_RESUME hypercall to resume
from the specified address (Oliver) * Add patches for SDEI migration and documentation (Gavin) * Misc comments from Oliver Upon (Oliver)
v6:
* Rebased to v5.18.rc1 (Gavin)
* Pass additional argument to smccc_get_arg() (Oliver)
* Add preparatory patch to route hypercalls based on their
owners (Oliver)
* Remove the support for shared event. (Oliver/Gavin)
* Remove the support for migration and add-on patches to
support it in future (Oliver)
* The events are exposed by KVM instead of VMM (Oliver)
* kvm_sdei_state.h is dropped and all the structures are
folded into the corresponding ones in kvm_sdei.h (Oliver)
* Rename 'struct kvm_sdei_registered_event' to
'struct kvm_sdei_event' (Oliver)
* Misc comments from Oliver Upon (Oliver)
v5/v4/v3/v2/v1:
* Skipped here and please visit the history by
https://lore.kernel.org/lkml/[email protected]/T/
Gavin Shan (22):
KVM: arm64: Extend smccc_get_argx()
KVM: arm64: Route hypercalls based on their owner
KVM: arm64: Add SDEI virtualization infrastructure
KVM: arm64: Support EVENT_REGISTER hypercall
KVM: arm64: Support EVENT_{ENABLE, DISABLE} hypercall
KVM: arm64: Support EVENT_CONTEXT hypercall
KVM: arm64: Support EVENT_UNREGISTER hypercall
KVM: arm64: Support EVENT_STATUS hypercall
KVM: arm64: Support EVENT_GET_INFO hypercall
KVM: arm64: Support PE_{MASK, UNMASK} hypercall
KVM: arm64: Support {PRIVATE, SHARED}_RESET hypercall
KVM: arm64: Support event injection and delivery
KVM: arm64: Support EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall
KVM: arm64: Support EVENT_SIGNAL hypercall
KVM: arm64: Support SDEI_FEATURES hypercall
KVM: arm64: Support SDEI_VERSION hypercall
KVM: arm64: Expose SDEI capbility and service
KVM: arm64: Allow large sized pseudo firmware registers
KVM: arm64: Support SDEI event migration
KVM: arm64: Add SDEI document
selftests: KVM: aarch64: Add SDEI case in hypercall tests
selftests: KVM: aarch64: Add SDEI test case
Documentation/virt/kvm/api.rst | 11 + Documentation/virt/kvm/arm/hypercalls.rst | 4 + Documentation/virt/kvm/arm/sdei.rst | 64 ++ arch/arm64/include/asm/kvm_host.h | 3 + arch/arm64/include/asm/kvm_sdei.h | 81 +++ arch/arm64/include/uapi/asm/kvm.h | 18 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 8 + arch/arm64/kvm/hypercalls.c | 182 +++-- arch/arm64/kvm/psci.c | 14 +- arch/arm64/kvm/pvtime.c | 2 +- arch/arm64/kvm/sdei.c | 676 ++++++++++++++++++ arch/arm64/kvm/trng.c | 4 +- include/kvm/arm_hypercalls.h | 19 +- include/linux/arm-smccc.h | 7 + include/uapi/linux/arm_sdei.h | 8 + include/uapi/linux/kvm.h | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/aarch64/hypercalls.c | 11 +- tools/testing/selftests/kvm/aarch64/sdei.c | 450 ++++++++++++ 20 files changed, 1499 insertions(+), 67 deletions(-) create mode 100644 Documentation/virt/kvm/arm/sdei.rst create mode 100644 arch/arm64/include/asm/kvm_sdei.h create mode 100644 arch/arm64/kvm/sdei.c create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 1/22 KVM: arm64: Extend smccc_get_argx()
Currently, there are 3 inline functions to retrieve SMCCC arguments, but the number of arguments is limited to 3. We need to retrieve more SMCCC arguments when SDEI virtualization is supported.
This introduces smccc_get_arg(), which accepts @index to indicate the SMCCC argument to be retrieved. Besides, smccc_get_function() also calls into this newly introduced helper. Further more, we also mechanically replace smccc_get_{arg1, arg2, arg3}() using the newly introduced helper.
--- arch/arm64/kvm/hypercalls.c | 4 ++-- arch/arm64/kvm/psci.c | 14 +++++++------- arch/arm64/kvm/pvtime.c | 2 +- arch/arm64/kvm/trng.c | 4 ++-- include/kvm/arm_hypercalls.h | 19 +++++-------------- 5 files changed, 17 insertions(+), 26 deletions(-)
arch/arm64/kvm/hypercalls.c +2 -2
@@ -41,7 +41,7 @@ static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val)* (virtual or physical) with the first argument of the SMCCC * call. In case the identifier is not supported, error out. */ - feature =smccc_get_arg1 (vcpu); + feature =smccc_get_arg (vcpu,1 );switch (feature) {case KVM_PTP_VIRT_COUNTER : cycles = systime_snapshot.cycles -vcpu_read_sys_reg (vcpu,CNTVOFF_EL2 );
@@ -137,7 +137,7 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) val[0 ] =ARM_SMCCC_VERSION_1_1 ;break ;case ARM_SMCCC_ARCH_FEATURES_FUNC_ID : - feature =smccc_get_arg1 (vcpu); + feature =smccc_get_arg (vcpu,1 );switch (feature) {case ARM_SMCCC_ARCH_WORKAROUND_1 :switch (arm64_get_spectre_v2_state ()) {
arch/arm64/kvm/psci.c +7 -7
@@ -64,7 +64,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)struct kvm_vcpu *vcpu =NULL ;unsigned long cpu_id; - cpu_id =smccc_get_arg1 (source_vcpu); + cpu_id =smccc_get_arg (source_vcpu,1 );if (!kvm_psci_valid_affinity (source_vcpu, cpu_id))return PSCI_RET_INVALID_PARAMS ;
@@ -85,7 +85,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu) reset_state = &vcpu->arch.reset_state; - reset_state->pc =smccc_get_arg2 (source_vcpu); + reset_state->pc =smccc_get_arg (source_vcpu,2 );/* Propagate caller endianness */ reset_state->be =kvm_vcpu_is_be (source_vcpu);
@@ -94,7 +94,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)* NOTE: We always update r0 (or x0) because for PSCI v0.1 * the general purpose registers are undefined upon CPU_ON. */ - reset_state->r0 =smccc_get_arg3 (source_vcpu); + reset_state->r0 =smccc_get_arg (source_vcpu,3 );WRITE_ONCE (reset_state->reset, true);kvm_make_request (KVM_REQ_VCPU_RESET , vcpu);
@@ -121,8 +121,8 @@ static unsigned long kvm_psci_vcpu_affinity_info(struct kvm_vcpu *vcpu)struct kvm *kvm = vcpu->kvm;struct kvm_vcpu *tmp; - target_affinity =smccc_get_arg1 (vcpu); - lowest_affinity_level =smccc_get_arg2 (vcpu); + target_affinity =smccc_get_arg (vcpu,1 ); + lowest_affinity_level =smccc_get_arg (vcpu,2 );if (!kvm_psci_valid_affinity (vcpu, target_affinity))return PSCI_RET_INVALID_PARAMS ;
@@ -318,7 +318,7 @@ static int kvm_psci_1_x_call(struct kvm_vcpu *vcpu, u32 minor) val = minor ==0 ?KVM_ARM_PSCI_1_0 :KVM_ARM_PSCI_1_1 ;break ;case PSCI_1_0_FN_PSCI_FEATURES : - arg =smccc_get_arg1 (vcpu); + arg =smccc_get_arg (vcpu,1 ); val =kvm_psci_check_allowed_function (vcpu, arg);if (val)break ;
@@ -372,7 +372,7 @@ static int kvm_psci_1_x_call(struct kvm_vcpu *vcpu, u32 minor) fallthrough;case PSCI_1_1_FN64_SYSTEM_RESET2 :if (minor >=1 ) { - arg =smccc_get_arg1 (vcpu); + arg =smccc_get_arg (vcpu,1 );if (arg <=PSCI_1_1_RESET_TYPE_SYSTEM_WARM_RESET || arg >=PSCI_1_1_RESET_TYPE_VENDOR_START ) {
arch/arm64/kvm/pvtime.c +1 -1
@@ -34,7 +34,7 @@ void kvm_update_stolen_time(struct kvm_vcpu *vcpu)long kvm_hypercall_pv_features (struct kvm_vcpu *vcpu) { -u32 feature =smccc_get_arg1 (vcpu); +u32 feature =smccc_get_arg (vcpu,1 );long val =SMCCC_RET_NOT_SUPPORTED ;switch (feature) {
arch/arm64/kvm/trng.c +2 -2
@@ -24,7 +24,7 @@ static const uuid_t arm_smc_trng_uuid __aligned(4) = UUID_INIT(static int kvm_trng_do_rnd (struct kvm_vcpu *vcpu,int size) {DECLARE_BITMAP (bits,TRNG_MAX_BITS64 ); -u32 num_bits =smccc_get_arg1 (vcpu); +u32 num_bits =smccc_get_arg (vcpu,1 );int i;if (num_bits >3 * size) {
@@ -60,7 +60,7 @@ int kvm_trng_call(struct kvm_vcpu *vcpu) val =ARM_SMCCC_TRNG_VERSION_1_0 ;break ;case ARM_SMCCC_TRNG_FEATURES : -switch (smccc_get_arg1 (vcpu)) { +switch (smccc_get_arg (vcpu,1 )) {case ARM_SMCCC_TRNG_VERSION :case ARM_SMCCC_TRNG_FEATURES :case ARM_SMCCC_TRNG_GET_UUID :
include/kvm/arm_hypercalls.h +5 -14
@@ -8,24 +8,15 @@int kvm_hvc_call_handler (struct kvm_vcpu *vcpu); -static inline u32 smccc_get_function (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,0 ); -} - -static inline unsigned long smccc_get_arg1 (struct kvm_vcpu *vcpu) +static inline unsigned long smccc_get_arg (struct kvm_vcpu *vcpu, +unsigned char index) { -return vcpu_get_reg (vcpu,1 ); +return vcpu_get_reg (vcpu, index); } -static inline unsigned long smccc_get_arg2 (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,2 ); -} - -static inline unsigned long smccc_get_arg3 (struct kvm_vcpu *vcpu) +static inline u32 smccc_get_function (struct kvm_vcpu *vcpu) { -return vcpu_get_reg (vcpu,3 ); +return smccc_get_arg (vcpu,0 ); } static inline void smccc_set_retval(struct kvm_vcpu *vcpu,
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 2/22 KVM: arm64: Route hypercalls based on their owner
In reply to Gavin Shankvm_hvc_call_handler() directly handles the incoming hypercall, or routes it based on its (function) ID. kvm_psci_call() becomes the gate keeper to handle the hypercall that can't be handled by any one else. It makes kvm_hvc_call_handler() a bit messy.
This reorgnizes the code to route hypercalls to the corresponding handler based on their owners. A hypercall may be handled directly inside the owner's handler or routed to the associated functionality for further handling. By the way, @func_id is renamed to @func, consistent with smccc_get_function().
The only exception is 0.1 PSCI hypercalls. Their owners aren't Standard Secure Service and can't be routed by their owners. So those 0.1 PSCI hypercalls are filtered and routed to kvm_psci_call() in kvm_hvc_call_handler().
--- arch/arm64/kvm/hypercalls.c | 116 ++++++++++++++++++++++++++---------- include/linux/arm-smccc.h | 5 ++ 2 files changed, 88 insertions(+), 33 deletions(-)
arch/arm64/kvm/hypercalls.c +83 -33
@@ -65,9 +65,9 @@ static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val) val[3 ] =lower_32_bits (cycles); } -static bool kvm_hvc_call_default_allowed (u32 func_id) +static bool kvm_hvc_call_default_allowed (u32 func) { -switch (func_id) { +switch (func) {/* * List of function-ids that are not gated with the bitmapped * feature firmware registers, and are to be allowed for
@@ -78,26 +78,26 @@ static bool kvm_hvc_call_default_allowed(u32 func_id)return true;default :/* PSCI 0.2 and up is in the 0:0x1f range */ -if (ARM_SMCCC_OWNER_NUM (func_id) ==ARM_SMCCC_OWNER_STANDARD && -ARM_SMCCC_FUNC_NUM (func_id) <=0x1f ) +if (ARM_SMCCC_OWNER_NUM (func) ==ARM_SMCCC_OWNER_STANDARD && +ARM_SMCCC_FUNC_NUM (func) <=0x1f )return true;/* * KVM's PSCI 0.1 doesn't comply with SMCCC, and has * its own function-id base and range */ -if (func_id >=KVM_PSCI_FN (0 ) && func_id <=KVM_PSCI_FN (3 )) +if (func >=KVM_PSCI_FN (0 ) && func <=KVM_PSCI_FN (3 ))return true;return false; } } -static bool kvm_hvc_call_allowed (struct kvm_vcpu *vcpu,u32 func_id) +static bool kvm_hvc_call_allowed (struct kvm_vcpu *vcpu,u32 func) {struct kvm_smccc_features *smccc_feat = &vcpu->kvm->arch.smccc_feat; -switch (func_id) { +switch (func) {case ARM_SMCCC_TRNG_VERSION :case ARM_SMCCC_TRNG_FEATURES :case ARM_SMCCC_TRNG_GET_UUID :
@@ -117,24 +117,19 @@ static bool kvm_hvc_call_allowed(struct kvm_vcpu *vcpu, u32 func_id)return test_bit (KVM_REG_ARM_VENDOR_HYP_BIT_PTP , &smccc_feat->vendor_hyp_bmap);default : -return kvm_hvc_call_default_allowed (func_id); +return kvm_hvc_call_default_allowed (func); } } -int kvm_hvc_call_handler (struct kvm_vcpu *vcpu) +static int kvm_hvc_arch (struct kvm_vcpu *vcpu,u32 func) {struct kvm_smccc_features *smccc_feat = &vcpu->kvm->arch.smccc_feat; -u32 func_id =smccc_get_function (vcpu); -u64 val[4 ] = {SMCCC_RET_NOT_SUPPORTED }; +u64 val =SMCCC_RET_NOT_SUPPORTED ;u32 feature; -gpa_t gpa; - -if (!kvm_hvc_call_allowed (vcpu, func_id)) - goto out; -switch (func_id) { +switch (func) {case ARM_SMCCC_VERSION_FUNC_ID : - val[0 ] =ARM_SMCCC_VERSION_1_1 ; + val =ARM_SMCCC_VERSION_1_1 ;break ;case ARM_SMCCC_ARCH_FEATURES_FUNC_ID : feature =smccc_get_arg (vcpu,1 );
@@ -144,10 +139,10 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case SPECTRE_VULNERABLE :break ;case SPECTRE_MITIGATED : - val[0 ] =SMCCC_RET_SUCCESS ; + val =SMCCC_RET_SUCCESS ;break ;case SPECTRE_UNAFFECTED : - val[0 ] =SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED ; + val =SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED ;break ; }break ;
@@ -170,7 +165,7 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)break ; fallthrough;case SPECTRE_UNAFFECTED : - val[0 ] =SMCCC_RET_NOT_REQUIRED ; + val =SMCCC_RET_NOT_REQUIRED ;break ; }break ;
@@ -179,28 +174,65 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case SPECTRE_VULNERABLE :break ;case SPECTRE_MITIGATED : - val[0 ] =SMCCC_RET_SUCCESS ; + val =SMCCC_RET_SUCCESS ;break ;case SPECTRE_UNAFFECTED : - val[0 ] =SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED ; + val =SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED ;break ; }break ;case ARM_SMCCC_HV_PV_TIME_FEATURES :if (test_bit (KVM_REG_ARM_STD_HYP_BIT_PV_TIME , &smccc_feat->std_hyp_bmap)) - val[0 ] =SMCCC_RET_SUCCESS ; + val =SMCCC_RET_SUCCESS ;break ; }break ; + } + +smccc_set_retval (vcpu, val,0 ,0 ,0 ); +return 1 ; +} + +static int kvm_hvc_standard (struct kvm_vcpu *vcpu,u32 func) +{ +switch (ARM_SMCCC_FUNC_NUM (func)) { + case ARM_SMCCC_STANDARD_PSCI_START ... ARM_SMCCC_STANDARD_PSCI_END: +return kvm_psci_call (vcpu); + case ARM_SMCCC_STANDARD_TRNG_START ... ARM_SMCCC_STANDARD_TRNG_END: +return kvm_trng_call (vcpu); + } + +smccc_set_retval (vcpu,SMCCC_RET_NOT_SUPPORTED ,0 ,0 ,0 ); +return 1 ; +} + +static int kvm_hvc_standard_hyp (struct kvm_vcpu *vcpu,u32 func) +{ +u64 val =SMCCC_RET_NOT_SUPPORTED ; +gpa_t gpa; + +switch (func) {case ARM_SMCCC_HV_PV_TIME_FEATURES : - val[0 ] =kvm_hypercall_pv_features (vcpu); + val =kvm_hypercall_pv_features (vcpu);break ;case ARM_SMCCC_HV_PV_TIME_ST : gpa =kvm_init_stolen_time (vcpu);if (gpa !=GPA_INVALID ) - val[0 ] = gpa; + val = gpa;break ; + } + +smccc_set_retval (vcpu, val,0 ,0 ,0 ); +return 1 ; +} + +static int kvm_hvc_vendor_hyp (struct kvm_vcpu *vcpu,u32 func) +{ +struct kvm_smccc_features *smccc_feat = &vcpu->kvm->arch.smccc_feat; +u64 val[4 ] = {SMCCC_RET_NOT_SUPPORTED }; + +switch (func) {case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID : val[0 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_0 ; val[1 ] =ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_1 ;
@@ -213,18 +245,36 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID :kvm_ptp_get_time (vcpu, val);break ; -case ARM_SMCCC_TRNG_VERSION : -case ARM_SMCCC_TRNG_FEATURES : -case ARM_SMCCC_TRNG_GET_UUID : -case ARM_SMCCC_TRNG_RND32 : -case ARM_SMCCC_TRNG_RND64 : -return kvm_trng_call (vcpu); -default : + } + +smccc_set_retval (vcpu, val[0 ], val[1 ], val[2 ], val[3 ]); +return 1 ; +} + +int kvm_hvc_call_handler (struct kvm_vcpu *vcpu) +{ +u32 func =smccc_get_function (vcpu); + +if (!kvm_hvc_call_allowed (vcpu, func)) + goto out; + +/* Filter these calls that aren't documented in the specification */ +if (func >=KVM_PSCI_FN_CPU_SUSPEND && func <=KVM_PSCI_FN_MIGRATE )return kvm_psci_call (vcpu); + +switch (ARM_SMCCC_OWNER_NUM (func)) { +case ARM_SMCCC_OWNER_ARCH : +return kvm_hvc_arch (vcpu, func); +case ARM_SMCCC_OWNER_STANDARD : +return kvm_hvc_standard (vcpu, func); +case ARM_SMCCC_OWNER_STANDARD_HYP : +return kvm_hvc_standard_hyp (vcpu, func); +case ARM_SMCCC_OWNER_VENDOR_HYP : +return kvm_hvc_vendor_hyp (vcpu, func); } out: -smccc_set_retval (vcpu, val[0 ], val[1 ], val[2 ], val[3 ]); +smccc_set_retval (vcpu,SMCCC_RET_NOT_SUPPORTED ,0 ,0 ,0 );return 1 ; }
include/linux/arm-smccc.h +5 -0
@@ -55,6 +55,11 @@#define ARM_SMCCC_OWNER_TRUSTED_OS 50#define ARM_SMCCC_OWNER_TRUSTED_OS_END 63 +#define ARM_SMCCC_STANDARD_PSCI_START 0x0000 +#define ARM_SMCCC_STANDARD_PSCI_END 0x001f +#define ARM_SMCCC_STANDARD_TRNG_START 0x0050 +#define ARM_SMCCC_STANDARD_TRNG_END 0x005f +#define ARM_SMCCC_FUNC_QUERY_CALL_UID 0xff01#define ARM_SMCCC_QUIRK_NONE 0
Signature · 1 line
2.23.0
PATCH v7 3/22 KVM: arm64: Add SDEI virtualization infrastructure
In reply to Gavin ShanSoftware Delegated Exception Interface (SDEI) provides a mechanism for registering and servicing system events, as defined by ARM DEN0054C specification. One of these events will be used by Asynchronous Page Fault (Async PF) to deliver notifications from host to guest.
The events are classified into shared and private ones according to their scopes. The shared events are system or VM scoped, but the private events are CPU or VCPU scoped. The shared events can be registered, enabled, unregistered and reset through hypercalls issued from any VCPU. However, the private events are registered, enabled, unregistered and reset on the calling VCPU through hypercalls. Besides, the events are also classified into critical and normal events according their priority. During event delivery and handling, the normal event can be preempted by another critical event, but not in reverse way. The critical event can't be preempted by another normal event.
In the implementation, both shared event and critical event aren't supported. It means all supported events are private and normal. There are several objects introduced to manage events and their states.
* kvm_sdei_event_handler
SDEI event handler, which is provided through EVENT_REGISTER hypercall, is called when the SDEI event is delivered from host to guest.
* kvm_sdei_event_context
The saved (preempted) context when SDEI event is delivered for handling.
* kvm_sdei_vcpu
SDEI events and their states.
The flag KVM_ARM64_SDEI_MASKED is set to mask vCPU from SDEI events by default. Besides, SDEI_NOT_SUPPORTED is returned for all hypercalls at present. The hypercalls will be supported one by one in subsequent patches.
--- arch/arm64/include/asm/kvm_host.h | 2 + arch/arm64/include/asm/kvm_sdei.h | 70 +++++++++++++++++++++++++++++++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 4 ++ arch/arm64/kvm/hypercalls.c | 3 ++ arch/arm64/kvm/sdei.c | 59 ++++++++++++++++++++++++++ include/linux/arm-smccc.h | 2 + 7 files changed, 141 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/kvm_sdei.h create mode 100644 arch/arm64/kvm/sdei.c
arch/arm64/include/asm/kvm_host.h +2 -0
@@ -366,6 +366,7 @@ struct kvm_vcpu_arch {* Anything that is not used directly from assembly code goes * here. */ +struct kvm_sdei_vcpu *sdei;/* * Guest registers we preserve during guest debugging.
@@ -466,6 +467,7 @@ struct kvm_vcpu_arch {#define KVM_ARM64_ON_UNSUPPORTED_CPU (1 << 15)/* Physical CPU not in supported_cpus */ #define KVM_ARM64_HOST_SME_ENABLED (1 << 16)/* SME enabled for EL0 */ #define KVM_ARM64_WFIT (1 << 17)/* WFIT instruction trapped */ +#define KVM_ARM64_SDEI_MASKED (1 << 18)/* SDEI is masked */ #define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \ KVM_GUESTDBG_USE_SW_BP | \
arch/arm64/include/asm/kvm_sdei.h new file +70 -0
70 lines not on this page · raw message
arch/arm64/kvm/Makefile +1 -1
@@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pkvm.o \ - arch_timer.o trng.o vmid.o \ + arch_timer.o trng.o vmid.o sdei.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/arm.c +4 -0
@@ -38,6 +38,7 @@#include <asm/kvm_asm.h> #include <asm/kvm_mmu.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_sdei.h> #include <asm/sections.h> #include <kvm/arm_hypercalls.h>
@@ -337,6 +338,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)kvm_arm_pvtime_vcpu_init (&vcpu->arch); +kvm_sdei_create_vcpu (vcpu); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; err =kvm_vgic_vcpu_init (vcpu);
@@ -358,6 +361,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_cache);kvm_timer_vcpu_terminate (vcpu);kvm_pmu_vcpu_destroy (vcpu); +kvm_sdei_destroy_vcpu (vcpu);kvm_arm_vcpu_destroy (vcpu); }
arch/arm64/kvm/hypercalls.c +3 -0
@@ -5,6 +5,7 @@#include <linux/kvm_host.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_sdei.h> #include <kvm/arm_hypercalls.h> #include <kvm/arm_psci.h>
@@ -199,6 +200,8 @@ static int kvm_hvc_standard(struct kvm_vcpu *vcpu, u32 func)switch (ARM_SMCCC_FUNC_NUM (func)) { case ARM_SMCCC_STANDARD_PSCI_START ... ARM_SMCCC_STANDARD_PSCI_END:return kvm_psci_call (vcpu); + case ARM_SMCCC_STANDARD_SDEI_START ... ARM_SMCCC_STANDARD_SDEI_END: +return kvm_sdei_call (vcpu); case ARM_SMCCC_STANDARD_TRNG_START ... ARM_SMCCC_STANDARD_TRNG_END:return kvm_trng_call (vcpu); }
arch/arm64/kvm/sdei.c new file +59 -0
59 lines not on this page · raw message
include/linux/arm-smccc.h +2 -0
@@ -57,6 +57,8 @@#define ARM_SMCCC_STANDARD_PSCI_START 0x0000#define ARM_SMCCC_STANDARD_PSCI_END 0x001f +#define ARM_SMCCC_STANDARD_SDEI_START 0x0020 +#define ARM_SMCCC_STANDARD_SDEI_END 0x003f#define ARM_SMCCC_STANDARD_TRNG_START 0x0050#define ARM_SMCCC_STANDARD_TRNG_END 0x005f
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 4/22 KVM: arm64: Support EVENT_REGISTER hypercall
In reply to Gavin ShanThis supports EVENT_REGISTER hypercall, which is used by guest to register event. The event won't be raised until it's registered and enabled. There are several cases where the request is injected:
* The event isn't supported by KVM
* The event has been registered or pending for unregistration
* The reserved bits or relative mode are set in @flags argument
--- arch/arm64/kvm/sdei.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/arm64/kvm/sdei.c +31 -0
@@ -13,6 +13,34 @@#include <kvm/arm_hypercalls.h> #include <asm/kvm_sdei.h> +static unsigned long event_register (struct kvm_vcpu *vcpu) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +unsigned int num =smccc_get_arg (vcpu,1 ); +unsigned long flags =smccc_get_arg (vcpu,4 ); + +if (num >=KVM_NR_SDEI_EVENTS ) +return SDEI_INVALID_PARAMETERS ; + +/* Reject if the reserved bits or relative mode are set */ +if (flags & ~0x1UL ) +return SDEI_INVALID_PARAMETERS ; + +/* +* Reject if the event has been registered or pending for +* unregistration. +*/ +if (test_bit (num, &vsdei->registered) || +test_bit (num, &vsdei->running)) +return SDEI_DENIED ; + + vsdei->handlers[num].ep_addr =smccc_get_arg (vcpu,2 ); + vsdei->handlers[num].ep_arg =smccc_get_arg (vcpu,3 ); +set_bit (num, &vsdei->registered); + +return SDEI_SUCCESS ; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -31,6 +59,9 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu) }switch (func) { +case SDEI_1_0_FN_SDEI_EVENT_REGISTER : + ret =event_register (vcpu); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 5/22 KVM: arm64: Support EVENT_{ENABLE, DISABLE} hypercall
In reply to Gavin ShanThis supports EVENT_{ENABLE, DISABLE} hypercall. The event can be pending, but never be delivered to guest until it's enabled.
* The request is rejected if the event isn't registered or pending for unregistration.
* When the event is enabled, KVM_REQ_SDEI is set to so that the event can be delivered to guest, if it has been pending. The delivery logic will be implemented in subsequent patches. For now, it's nothing other than a flag.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/sdei.c | 33 +++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -47,6 +47,7 @@#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4)#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5)#define KVM_REQ_SUSPEND KVM_ARCH_REQ(6) +#define KVM_REQ_SDEI KVM_ARCH_REQ(7)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)
arch/arm64/kvm/sdei.c +33 -0
@@ -41,6 +41,33 @@ static unsigned long event_register(struct kvm_vcpu *vcpu)return SDEI_SUCCESS ; } +static unsigned long event_enable (struct kvm_vcpu *vcpu,bool enable) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +unsigned int num =smccc_get_arg (vcpu,1 ); + +if (num >=KVM_NR_SDEI_EVENTS ) +return SDEI_INVALID_PARAMETERS ; + +/* +* Reject if the event hasn't been registered or pending +* for unregistration. +*/ +if (!test_bit (num, &vsdei->registered)) +return SDEI_DENIED ; + +if (enable) { +set_bit (num, &vsdei->enabled); +if (!(vcpu->arch.flags &KVM_ARM64_SDEI_MASKED ) && +test_bit (num, &vsdei->pending)) +kvm_make_request (KVM_REQ_SDEI , vcpu); + }else { +clear_bit (num, &vsdei->enabled); + } + +return SDEI_SUCCESS ; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -62,6 +89,12 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_EVENT_REGISTER : ret =event_register (vcpu);break ; +case SDEI_1_0_FN_SDEI_EVENT_ENABLE : + ret =event_enable (vcpu, true); +break ; +case SDEI_1_0_FN_SDEI_EVENT_DISABLE : + ret =event_enable (vcpu, false); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 6/22 KVM: arm64: Support EVENT_CONTEXT hypercall
In reply to Gavin ShanThis supports EVENT_CONTEXT hypercall. It's called inside the event handler to retrieve the specified register in the saved or preempted context.
* The request is rejected if no running event handler exists or the parameter ID is out of range.
--- arch/arm64/kvm/sdei.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
arch/arm64/kvm/sdei.c +21 -1
@@ -44,7 +44,7 @@ static unsigned long event_register(struct kvm_vcpu *vcpu)static unsigned long event_enable (struct kvm_vcpu *vcpu,bool enable) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; -unsigned int num =smccc_get_arg (vcpu,1 ); +int num =smccc_get_arg (vcpu,1 );if (num >=KVM_NR_SDEI_EVENTS )return SDEI_INVALID_PARAMETERS ;
@@ -68,6 +68,23 @@ static unsigned long event_enable(struct kvm_vcpu *vcpu, bool enable)return SDEI_SUCCESS ; } +static unsigned long event_context (struct kvm_vcpu *vcpu) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +struct kvm_sdei_event_context *ctxt = &vsdei->ctxt; +unsigned int param_id =smccc_get_arg (vcpu,1 ); + +/* Reject if event handler isn't running */ +if (!vsdei->running) +return SDEI_DENIED ; + +/* Reject if the parameter ID is out of range */ +if (param_id >=ARRAY_SIZE (ctxt->regs)) +return SDEI_INVALID_PARAMETERS ; + +return ctxt->regs[param_id]; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -95,6 +112,9 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_EVENT_DISABLE : ret =event_enable (vcpu, false);break ; +case SDEI_1_0_FN_SDEI_EVENT_CONTEXT : + ret =event_context (vcpu); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 7/22 KVM: arm64: Support EVENT_UNREGISTER hypercall
In reply to Gavin ShanThis supports EVENT_UNREGISTER hypercall. No event will be delivered to guest after it's unregistered.
* Reject if the event hasn't been registered.
* Return SDEI_PENDING if the event handler is running.
* The event is disabled automatically on unregistration.
--- arch/arm64/kvm/sdei.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
arch/arm64/kvm/sdei.c +36 -0
@@ -85,6 +85,39 @@ static unsigned long event_context(struct kvm_vcpu *vcpu)return ctxt->regs[param_id]; } +static unsigned long event_unregister (struct kvm_vcpu *vcpu) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +unsigned int num =smccc_get_arg (vcpu,1 ); + +if (num >=KVM_NR_SDEI_EVENTS ) +return SDEI_INVALID_PARAMETERS ; + +/* +* Reject if the event isn't registered. It's allowed to +* unregister event which has been pending for that. +*/ +if (!test_bit (num, &vsdei->registered)) { +if (test_bit (num, &vsdei->running)) +return SDEI_PENDING ; +else +return SDEI_DENIED ; + } + +/* +* The event is disabled automatically on unregistration, even +* pending for that. +*/ +clear_bit (num, &vsdei->enabled); +clear_bit (num, &vsdei->registered); + +/* Pending for unreigstration if the event handler is running */ +if (test_bit (num, &vsdei->running)) +return SDEI_PENDING ; + +return SDEI_SUCCESS ; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -115,6 +148,9 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_EVENT_CONTEXT : ret =event_context (vcpu);break ; +case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER : + ret =event_unregister (vcpu); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 8/22 KVM: arm64: Support EVENT_STATUS hypercall
In reply to Gavin ShanThis supports EVENT_STATUS hypercall. It's used to retrieve the status about the specified event. A bitmap is returned to represent the event status, which includes state of registration, enablement, and running.
--- arch/arm64/kvm/sdei.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/arm64/kvm/sdei.c +22 -0
@@ -118,6 +118,25 @@ static unsigned long event_unregister(struct kvm_vcpu *vcpu)return SDEI_SUCCESS ; } +static unsigned long event_status (struct kvm_vcpu *vcpu) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +unsigned int num =smccc_get_arg (vcpu,1 ); +unsigned long ret =0 ; + +if (num >=KVM_NR_SDEI_EVENTS ) +return SDEI_INVALID_PARAMETERS ; + +if (test_bit (num, &vsdei->registered)) + ret |= (1UL <<SDEI_EVENT_STATUS_REGISTERED ); +if (test_bit (num, &vsdei->enabled)) + ret |= (1UL <<SDEI_EVENT_STATUS_ENABLED ); +if (test_bit (num, &vsdei->running)) + ret |= (1UL <<SDEI_EVENT_STATUS_RUNNING ); + +return ret; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -151,6 +170,9 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER : ret =event_unregister (vcpu);break ; +case SDEI_1_0_FN_SDEI_EVENT_STATUS : + ret =event_status (vcpu); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 9/22 KVM: arm64: Support EVENT_GET_INFO hypercall
In reply to Gavin ShanThis supports EVENT_GET_INFO hypercall. It's used to retrieve information about the event: type, signaled, priority, routing mode and affinity.
All supported events are private and have normal priority. Besides, all supported events can be signaled from software. It means we just return fixed value for type/signaled/priority.
As the routing mode and affinity are only available for the shared event, which is unsupported. So SDEI_INVALID_PARAMETERS is returned when they're requested.
--- arch/arm64/kvm/sdei.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
arch/arm64/kvm/sdei.c +35 -0
@@ -137,6 +137,38 @@ static unsigned long event_status(struct kvm_vcpu *vcpu)return ret; } +static unsigned long event_info (struct kvm_vcpu *vcpu) +{ +unsigned int num =smccc_get_arg (vcpu,1 ); +unsigned int info =smccc_get_arg (vcpu,2 ); +unsigned long ret =0 ; + +if (num >=KVM_NR_SDEI_EVENTS ) +return SDEI_INVALID_PARAMETERS ; + +/* +* All supported events are private and have normal priority. +* Besides, all supported events can be signaled by software +*/ +switch (info) { +case SDEI_EVENT_INFO_EV_TYPE : + ret =SDEI_EVENT_TYPE_PRIVATE ; +break ; +case SDEI_EVENT_INFO_EV_SIGNALED : + ret =1 ; +break ; +case SDEI_EVENT_INFO_EV_PRIORITY : + ret =SDEI_EVENT_PRIORITY_NORMAL ; +break ; +case SDEI_EVENT_INFO_EV_ROUTING_MODE : +case SDEI_EVENT_INFO_EV_ROUTING_AFF : +default : + ret =SDEI_INVALID_PARAMETERS ; + } + +return ret; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -173,6 +205,9 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_EVENT_STATUS : ret =event_status (vcpu);break ; +case SDEI_1_0_FN_SDEI_EVENT_GET_INFO : + ret =event_info (vcpu); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 10/22 KVM: arm64: Support PE_{MASK, UNMASK} hypercall
In reply to Gavin ShanThis supports PE_{MASK, UNMASK} hypercall. When PE_UNMASK is called, KVM_REQ_SDEI request becomes pending if there are any events need to be delivered.
--- arch/arm64/kvm/sdei.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
arch/arm64/kvm/sdei.c +21 -0
@@ -169,6 +169,21 @@ static unsigned long event_info(struct kvm_vcpu *vcpu)return ret; } +static unsigned long pe_mask (struct kvm_vcpu *vcpu,bool mask) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; + +if (mask) { + vcpu->arch.flags |=KVM_ARM64_SDEI_MASKED ; + }else { + vcpu->arch.flags &= ~KVM_ARM64_SDEI_MASKED ; +if (vsdei->pending) +kvm_make_request (KVM_REQ_SDEI , vcpu); + } + +return SDEI_SUCCESS ; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -208,6 +223,12 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_EVENT_GET_INFO : ret =event_info (vcpu);break ; +case SDEI_1_0_FN_SDEI_PE_MASK : + ret =pe_mask (vcpu, true); +break ; +case SDEI_1_0_FN_SDEI_PE_UNMASK : + ret =pe_mask (vcpu, false); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 11/22 KVM: arm64: Support {PRIVATE, SHARED}_RESET hypercall
In reply to Gavin ShanThis supports {PRIVATE, SHARED}_RESET hypercall. There is nothing to do if we're going to reset the shared events, which are not supported. For the private events, their registered and enabled state are cleared.
--- arch/arm64/kvm/sdei.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
arch/arm64/kvm/sdei.c +26 -0
@@ -184,6 +184,26 @@ static unsigned long pe_mask(struct kvm_vcpu *vcpu, bool mask)return SDEI_SUCCESS ; } +static unsigned long event_reset (struct kvm_vcpu *vcpu,bool private) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +unsigned int num; + +/* +* Nothing to do if we're going to reset the shared events, +* which are unsupported. +*/ +if (!private) +return SDEI_SUCCESS ; + +for (num =0 ; num <KVM_NR_SDEI_EVENTS ; num++) { +clear_bit (num, &vsdei->registered); +clear_bit (num, &vsdei->enabled); + } + +return SDEI_SUCCESS ; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -229,6 +249,12 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_PE_UNMASK : ret =pe_mask (vcpu, false);break ; +case SDEI_1_0_FN_SDEI_PRIVATE_RESET : + ret =event_reset (vcpu, true); +break ; +case SDEI_1_0_FN_SDEI_SHARED_RESET : + ret =event_reset (vcpu, false); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 12/22 KVM: arm64: Support event injection and delivery
In reply to Gavin ShanThis supports event injection, delivery and cancellation. The event is injected and cancelled by kvm_sdei_{inject, cancel}_event(). For event delivery, kvm_sdei_deliver_event() is added to accommodate KVM_REQ_SDEI request.
The KVM_REQ_SDEI request can be raised in several situation:
* PE is unmasked
* Event is enabled
* Completion of currently running event or handler on receiving EVENT_COMPLETE or EVENT_COMPLETE_AND_RESUME hypercall, which will be supported in the subsequent patch.
--- arch/arm64/include/asm/kvm_sdei.h | 4 + arch/arm64/kvm/arm.c | 3 + arch/arm64/kvm/sdei.c | 123 ++++++++++++++++++++++++++++++ 3 files changed, 130 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +4 -0
@@ -64,6 +64,10 @@ struct kvm_sdei_vcpu {/* APIs */ int kvm_sdei_call (struct kvm_vcpu *vcpu); +int kvm_sdei_inject_event (struct kvm_vcpu *vcpu, +unsigned int num,bool immediate); +int kvm_sdei_cancel_event (struct kvm_vcpu *vcpu,unsigned int num); +void kvm_sdei_deliver_event (struct kvm_vcpu *vcpu);void kvm_sdei_create_vcpu (struct kvm_vcpu *vcpu);void kvm_sdei_destroy_vcpu (struct kvm_vcpu *vcpu);
arch/arm64/kvm/arm.c +3 -0
@@ -720,6 +720,9 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_VCPU_RESET , vcpu))kvm_reset_vcpu (vcpu); +if (kvm_check_request (KVM_REQ_SDEI , vcpu)) +kvm_sdei_deliver_event (vcpu); +/* * Clear IRQ_PENDING requests that were made to guarantee * that a VCPU sees new virtual interrupts.
arch/arm64/kvm/sdei.c +123 -0
129 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 13/22 KVM: arm64: Support EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall
In reply to Gavin ShanThis supports EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall. The execution resumes from previously interrupted context when EVENT_COMPLETE hypercall is received.
However, the execution resumes from the specified address when EVENT_COMPLETE_AND_RESUME is received. In this case, context switches like below.
* x0 to x17 are restored from the interrupted context.
* SPSR_EL1 is set to PSTATE of the interrupted context.
* ELR_EL1 is set to PC of the interrupted context.
* PSTATE has nRW cleared, but D/A/I/F set.
* PC is set to the resume address, specified in the first argument of EVENT_COMPLETE_AND_RESUME hypercall.
--- arch/arm64/kvm/sdei.c | 54 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
arch/arm64/kvm/sdei.c +54 -0
@@ -85,6 +85,54 @@ static unsigned long event_context(struct kvm_vcpu *vcpu)return ctxt->regs[param_id]; } +static void event_complete (struct kvm_vcpu *vcpu,bool resume) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +struct kvm_sdei_event_context *ctxt = &vsdei->ctxt; +unsigned long pstate, resume_addr =smccc_get_arg (vcpu,1 ); +unsigned int num, i; + + num =find_next_bit (&vsdei->running,KVM_NR_SDEI_EVENTS ,0 ); +if (num >=KVM_NR_SDEI_EVENTS ) +return ; + +/* Restore registers: x0 -> x17 */ +for (i =0 ; i <ARRAY_SIZE (ctxt->regs); i++) +vcpu_set_reg (vcpu, i, ctxt->regs[i]); + +/* +* The registers are modified accordingly if the execution resumes +* from the specified address. +* +* SPSR_EL1: PSTATE of the interrupted context +* ELR_EL1: PC of the interrupted context +* PSTATE: cleared nRW bit, but D/A/I/F bits are set +* PC: the resume address +*/ +if (resume) { +if (has_vhe ()) { +write_sysreg_el1 (ctxt->pstate,SYS_SPSR ); +write_sysreg_s (ctxt->pc,SYS_ELR_EL12 ); + }else { +__vcpu_sys_reg (vcpu,SPSR_EL1 ) = ctxt->pstate; +__vcpu_sys_reg (vcpu,ELR_EL1 ) = ctxt->pc; + } + + pstate = ctxt->pstate; + pstate &= ~(PSR_MODE32_BIT |PSR_MODE_MASK ); + pstate |= (PSR_D_BIT |PSR_A_BIT |PSR_I_BIT | +PSR_F_BIT | PSR_MODE_EL1h); + *vcpu_cpsr (vcpu) = pstate; + *vcpu_pc (vcpu) = resume_addr; + }else { + *vcpu_cpsr (vcpu) = ctxt->pstate; + *vcpu_pc (vcpu) = ctxt->pc; + } + +/* Update event state */ +clear_bit (num, &vsdei->running); +} +static unsigned long event_unregister (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -234,6 +282,12 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_EVENT_CONTEXT : ret =event_context (vcpu);break ; +case SDEI_1_0_FN_SDEI_EVENT_COMPLETE : +event_complete (vcpu, false); +break ; +case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME : +event_complete (vcpu, true); +break ;case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER : ret =event_unregister (vcpu);break ;
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 14/22 KVM: arm64: Support EVENT_SIGNAL hypercall
In reply to Gavin ShanThis supports EVENT_SIGNAL hypercall. It's used to inject the software signaled event, whose number is zero. The KVM_REQ_SDEI request is raised if the PE is unmasked and the event has been enabled.
--- arch/arm64/kvm/sdei.c | 28 ++++++++++++++++++++++++++++ include/uapi/linux/arm_sdei.h | 3 +++ 2 files changed, 31 insertions(+)
arch/arm64/kvm/sdei.c +28 -0
@@ -232,6 +232,31 @@ static unsigned long pe_mask(struct kvm_vcpu *vcpu, bool mask)return SDEI_SUCCESS ; } +static unsigned long event_signal (struct kvm_vcpu *vcpu) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +unsigned int num =smccc_get_arg (vcpu,1 ); + +if (num >=KVM_NR_SDEI_EVENTS ) +return SDEI_INVALID_PARAMETERS ; + +/* +* The event must be software signaled event, whose +* number is zero. +*/ +if (num !=SDEI_SW_SIGNALED_EVENT || + !test_bit (num, &vsdei->registered)) +return SDEI_INVALID_PARAMETERS ; + +if (!test_and_set_bit (num, &vsdei->pending)) { +if (!(vcpu->arch.flags &KVM_ARM64_SDEI_MASKED ) && +test_bit (num, &vsdei->enabled)) +kvm_make_request (KVM_REQ_SDEI , vcpu); + } + +return SDEI_SUCCESS ; +} +static unsigned long event_reset (struct kvm_vcpu *vcpu,bool private) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -303,6 +328,9 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_PE_UNMASK : ret =pe_mask (vcpu, false);break ; +case SDEI_1_1_FN_SDEI_EVENT_SIGNAL : + ret =event_signal (vcpu); +break ;case SDEI_1_0_FN_SDEI_PRIVATE_RESET : ret =event_reset (vcpu, true);break ;
include/uapi/linux/arm_sdei.h +3 -0
@@ -22,9 +22,12 @@#define SDEI_1_0_FN_SDEI_PE_UNMASK SDEI_1_0_FN(0x0C)#define SDEI_1_0_FN_SDEI_INTERRUPT_BIND SDEI_1_0_FN(0x0D)#define SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE SDEI_1_0_FN(0x0E) +#define SDEI_1_1_FN_SDEI_EVENT_SIGNAL SDEI_1_0_FN(0x0F)#define SDEI_1_0_FN_SDEI_PRIVATE_RESET SDEI_1_0_FN(0x11)#define SDEI_1_0_FN_SDEI_SHARED_RESET SDEI_1_0_FN(0x12) +#define SDEI_SW_SIGNALED_EVENT 0 +#define SDEI_VERSION_MAJOR_SHIFT 48#define SDEI_VERSION_MAJOR_MASK 0x7fff#define SDEI_VERSION_MINOR_SHIFT 32
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 15/22 KVM: arm64: Support SDEI_FEATURES hypercall
In reply to Gavin ShanThis supports SDEI_FEATURES hypercall. It's used to retrieve the supported features, including number of slots for the interrupt binding events and relative mode for event handler. Currently, we simply return zero, meaning none of them is supported.
--- arch/arm64/kvm/sdei.c | 16 ++++++++++++++++ include/uapi/linux/arm_sdei.h | 5 +++++ 2 files changed, 21 insertions(+)
arch/arm64/kvm/sdei.c +16 -0
@@ -277,6 +277,19 @@ static unsigned long event_reset(struct kvm_vcpu *vcpu, bool private)return SDEI_SUCCESS ; } +static unsigned long sdei_features (struct kvm_vcpu *vcpu) +{ +unsigned int feature =smccc_get_arg (vcpu,1 ); + +switch (feature) { +case SDEI_FEATURE_BIND_SLOTS : +case SDEI_FEATURE_RELATIVE_MODE : +return 0 ; + } + +return SDEI_INVALID_PARAMETERS ; +} +int kvm_sdei_call (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -337,6 +350,9 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_SHARED_RESET : ret =event_reset (vcpu, false);break ; +case SDEI_1_1_FN_SDEI_FEATURES : + ret =sdei_features (vcpu); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
include/uapi/linux/arm_sdei.h +5 -0
@@ -23,6 +23,7 @@#define SDEI_1_0_FN_SDEI_INTERRUPT_BIND SDEI_1_0_FN(0x0D)#define SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE SDEI_1_0_FN(0x0E)#define SDEI_1_1_FN_SDEI_EVENT_SIGNAL SDEI_1_0_FN(0x0F) +#define SDEI_1_1_FN_SDEI_FEATURES SDEI_1_0_FN(0x10)#define SDEI_1_0_FN_SDEI_PRIVATE_RESET SDEI_1_0_FN(0x11)#define SDEI_1_0_FN_SDEI_SHARED_RESET SDEI_1_0_FN(0x12)
@@ -73,4 +74,8 @@#define SDEI_EVENT_PRIORITY_NORMAL 0#define SDEI_EVENT_PRIORITY_CRITICAL 1 +/* SDEI_FEATURES */ +#define SDEI_FEATURE_BIND_SLOTS 0 +#define SDEI_FEATURE_RELATIVE_MODE 1 +#endif /* _UAPI_LINUX_ARM_SDEI_H */
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 16/22 KVM: arm64: Support SDEI_VERSION hypercall
In reply to Gavin ShanThis supports SDEI_VERSION hypercall by returning v1.1, which is the specification version we're following. The vendor is set to one of the values returned from ARM_SMCCC_VENDOR_HYP_CALL_UID hypercall.
--- arch/arm64/include/asm/kvm_sdei.h | 3 +++ arch/arm64/kvm/sdei.c | 11 +++++++++++ 2 files changed, 14 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +3 -0
@@ -62,6 +62,9 @@ struct kvm_sdei_vcpu {struct kvm_sdei_event_context ctxt; }; +/* Returned as vendor through SDEI_VERSION hypercall */ +#define KVM_SDEI_VENDOR ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 +/* APIs */ int kvm_sdei_call (struct kvm_vcpu *vcpu); int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
arch/arm64/kvm/sdei.c +11 -0
@@ -13,6 +13,14 @@#include <kvm/arm_hypercalls.h> #include <asm/kvm_sdei.h> +static unsigned long sdei_version (struct kvm_vcpu *vcpu) +{ +/* v1.1 and vendor ID */ +return (1UL <<SDEI_VERSION_MAJOR_SHIFT ) | + (1UL <<SDEI_VERSION_MINOR_SHIFT ) | +KVM_SDEI_VENDOR ; +} +static unsigned long event_register (struct kvm_vcpu *vcpu) {struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
@@ -308,6 +316,9 @@ int kvm_sdei_call(struct kvm_vcpu *vcpu) }switch (func) { +case SDEI_1_0_FN_SDEI_VERSION : + ret =sdei_version (vcpu); +break ;case SDEI_1_0_FN_SDEI_EVENT_REGISTER : ret =event_register (vcpu);break ;
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 17/22 KVM: arm64: Expose SDEI capbility and service
In reply to Gavin ShanThe SDEI functionality is ready and lets exposes the capability and service.
* KVM_CAP_ARM_SDEI
* KVM_REG_ARM_STD_BIT_SDEI in pseudo-firmware bitmap register KVM_REG_ARM_STD_BMAP
* The events for software signaled one (0x0) and Async PF (0x1) are exposed.
--- Documentation/virt/kvm/api.rst | 11 +++++++++++ Documentation/virt/kvm/arm/hypercalls.rst | 4 ++++ arch/arm64/include/asm/kvm_sdei.h | 2 ++ arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/hypercalls.c | 4 ++++ include/uapi/linux/kvm.h | 1 + 7 files changed, 24 insertions(+)
Documentation/virt/kvm/api.rst +11 -0
@@ -7956,6 +7956,17 @@ should adjust CPUID leaf 0xA to reflect that the PMU is disabled. When enabled, KVM will exit to userspace with KVM_EXIT_SYSTEM_EVENT of type KVM_SYSTEM_EVENT_SUSPEND to process the guest suspend request. +8.37 KVM_CAP_ARM_SDEI +--------------------- + +:Capability: KVM_CAP_ARM_SDEI +:Architectures: arm64 +:Type: vm + +This capability indicates that the SDEI virtual service is supported +in the host. A VMM can check whether the service is available to enable +it. + 9. Known KVM API problems =========================
Documentation/virt/kvm/arm/hypercalls.rst +4 -0
@@ -106,6 +106,10 @@ The pseudo-firmware bitmap register are as follows: The bit represents the services offered under v1.0 of ARM True Random Number Generator (TRNG) specification, ARM DEN0098. + Bit-1: KVM_REG_ARM_STD_BIT_SDEI: + The bit represents the services offered under v1.1 of ARM Software + Delegated Exception Interface (SDEI) specification, ARM DEN0054C. + * KVM_REG_ARM_STD_HYP_BMAP: Controls the bitmap of the ARM Standard Hypervisor Service Calls.
arch/arm64/include/asm/kvm_sdei.h +2 -0
@@ -15,6 +15,8 @@#include <linux/bits.h> enum { +KVM_SDEI_EVENT_SW_SIGNALED =0 , +KVM_SDEI_EVENT_ASYNC_PF ,KVM_NR_SDEI_EVENTS , };
arch/arm64/include/uapi/asm/kvm.h +1 -0
@@ -344,6 +344,7 @@ struct kvm_arm_copy_mte_tags {enum {KVM_REG_ARM_STD_BIT_TRNG_V1_0 =0 , +KVM_REG_ARM_STD_BIT_SDEI ,#ifdef __KERNEL__KVM_REG_ARM_STD_BMAP_BIT_COUNT ,#endif
arch/arm64/kvm/arm.c +1 -0
@@ -217,6 +217,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_VCPU_ATTRIBUTES :case KVM_CAP_PTP_KVM :case KVM_CAP_ARM_SYSTEM_SUSPEND : +case KVM_CAP_ARM_SDEI : r =1 ;break ;case KVM_CAP_SET_GUEST_DEBUG2 :
arch/arm64/kvm/hypercalls.c +4 -0
@@ -99,6 +99,10 @@ static bool kvm_hvc_call_allowed(struct kvm_vcpu *vcpu, u32 func)struct kvm_smccc_features *smccc_feat = &vcpu->kvm->arch.smccc_feat;switch (func) { + case SDEI_1_0_FN(ARM_SMCCC_STANDARD_SDEI_START - 0x20) ... +SDEI_1_0_FN (ARM_SMCCC_STANDARD_SDEI_END -0x20 ): +return test_bit (KVM_REG_ARM_STD_BIT_SDEI , + &smccc_feat->std_bmap);case ARM_SMCCC_TRNG_VERSION :case ARM_SMCCC_TRNG_FEATURES :case ARM_SMCCC_TRNG_GET_UUID :
include/uapi/linux/kvm.h +1 -0
@@ -1157,6 +1157,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_VM_TSC_CONTROL 214#define KVM_CAP_SYSTEM_EVENT_DATA 215#define KVM_CAP_ARM_SYSTEM_SUSPEND 216 +#define KVM_CAP_ARM_SDEI 217#ifdef KVM_CAP_IRQ_ROUTING
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 18/22 KVM: arm64: Allow large sized pseudo firmware registers
In reply to Gavin ShanCurrently, the sizes of all pseudo firmware registers are 64 bits. The value from user space is copied to a local 64-bits variable, which is passed on in kvm_arm_set_fw_reg(). It works perfectly until large sized pseudo firmware registers are added for SDEI event handlers, states and context.
This copies the user space's value to local 64-bits variable and passes the local variable on, only if the pseudo firmware register is 64-bits in size. No functional change intended.
--- arch/arm64/kvm/hypercalls.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
arch/arm64/kvm/hypercalls.c +17 -6
@@ -406,14 +406,20 @@ int kvm_arm_get_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)return 0 ; } -static int kvm_arm_set_fw_reg_bmap (struct kvm_vcpu *vcpu,u64 reg_id,u64 val) +static int kvm_arm_set_fw_reg_bmap (struct kvm_vcpu *vcpu, +const struct kvm_one_reg *reg) {int ret =0 ;struct kvm *kvm = vcpu->kvm;struct kvm_smccc_features *smccc_feat = &kvm->arch.smccc_feat;unsigned long *fw_reg_bmap, fw_reg_features; +void __user *uaddr = (void __user *)(long )reg->addr; +u64 val; -switch (reg_id) { +if (copy_from_user (&val, uaddr,KVM_REG_SIZE (reg->id))) +return -EFAULT ; + +switch (reg->id) {case KVM_REG_ARM_STD_BMAP : fw_reg_bmap = &smccc_feat->std_bmap; fw_reg_features =KVM_ARM_SMCCC_STD_FEATURES ;
@@ -454,15 +460,14 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)u64 val;int wa_level; -if (copy_from_user (&val, uaddr,KVM_REG_SIZE (reg->id))) -return -EFAULT ; -switch (reg->id) {case KVM_REG_ARM_PSCI_VERSION : {bool wants_02; wants_02 =test_bit (KVM_ARM_VCPU_PSCI_0_2 , vcpu->arch.features); +if (copy_from_user (&val, uaddr,KVM_REG_SIZE (reg->id))) +return -EFAULT ;switch (val) {case KVM_ARM_PSCI_0_1 :
@@ -483,6 +488,9 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)case KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1 :case KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3 : +if (copy_from_user (&val, uaddr,KVM_REG_SIZE (reg->id))) +return -EFAULT ; +if (val & ~KVM_REG_FEATURE_LEVEL_MASK )return -EINVAL ;
@@ -492,6 +500,9 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)return 0 ;case KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2 : +if (copy_from_user (&val, uaddr,KVM_REG_SIZE (reg->id))) +return -EFAULT ; +if (val & ~(KVM_REG_FEATURE_LEVEL_MASK |KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_ENABLED ))return -EINVAL ;
@@ -529,7 +540,7 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)case KVM_REG_ARM_STD_BMAP :case KVM_REG_ARM_STD_HYP_BMAP :case KVM_REG_ARM_VENDOR_HYP_BMAP : -return kvm_arm_set_fw_reg_bmap (vcpu, reg->id, val); +return kvm_arm_set_fw_reg_bmap (vcpu, reg);default :return -ENOENT ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 19/22 KVM: arm64: Support SDEI event migration
In reply to Gavin ShanThis supports migration for SDEI event handlers, states and context. Several pseudo firmware registers are added to assist the migration work.
* KVM_REG_ARM_SDEI_EVENT_HANDLER_0
KVM_REG_ARM_SDEI_EVENT_HANDLER_1
KVM_REG_ARM_SDEI_EVENT_HANDLER_2
KVM_REG_ARM_SDEI_EVENT_HANDLER_3
128-bits in length. They're mapped to the handler's address and argument for Software Signaled event and Async PF event. Additinal two events are reserved for future needs without too much considerations to the compatible issue. Not too much bandwidth needed to migrate those two additional registers.
* KVM_REG_ARM_SDEI_EVENT_REGISTERED
KVM_REG_ARM_SDEI_EVENT_ENABLED
KVM_REG_ARM_SDEI_EVENT_RUNNING
KVM_REG_ARM_SDEI_EVENT_PENDING
64-bits in length. They're mapped to registered, enabled, running and pending bitmap.
* KVM_REG_ARM_SDEI_EVENT_CONTEXT
2048-bits in length. It's mapped to saved or interrupted context.
* KVM_REG_ARM_SDEI_PE_STATE
64-bits in length. It's mapped to PE's states, which is masked or unmasked.
--- arch/arm64/include/asm/kvm_sdei.h | 2 + arch/arm64/include/uapi/asm/kvm.h | 17 ++++ arch/arm64/kvm/hypercalls.c | 32 ++++++ arch/arm64/kvm/sdei.c | 161 ++++++++++++++++++++++++++++++ 4 files changed, 212 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +2 -0
@@ -73,6 +73,8 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu, unsigned int num, bool immediate);int kvm_sdei_cancel_event (struct kvm_vcpu *vcpu,unsigned int num);void kvm_sdei_deliver_event (struct kvm_vcpu *vcpu); +int kvm_sdei_get_reg (struct kvm_vcpu *vcpu,const struct kvm_one_reg *reg); +int kvm_sdei_set_reg (struct kvm_vcpu *vcpu,const struct kvm_one_reg *reg);void kvm_sdei_create_vcpu (struct kvm_vcpu *vcpu);void kvm_sdei_destroy_vcpu (struct kvm_vcpu *vcpu);
arch/arm64/include/uapi/asm/kvm.h +17 -0
@@ -262,6 +262,11 @@ struct kvm_arm_copy_mte_tags {#define KVM_REG_ARM_FW (0x0014 << KVM_REG_ARM_COPROC_SHIFT)#define KVM_REG_ARM_FW_REG (r) (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \ KVM_REG_ARM_FW | ((r) & 0xffff)) +#define KVM_REG_ARM_FW_REG_128 (r) (KVM_REG_ARM64 | KVM_REG_SIZE_U128 | \ + KVM_REG_ARM_FW | ((r) & 0xffff)) +#define KVM_REG_ARM_FW_REG_2048 (r) (KVM_REG_ARM64 | KVM_REG_SIZE_U2048 | \ + KVM_REG_ARM_FW | ((r) & 0xffff)) +#define KVM_REG_ARM_PSCI_VERSION KVM_REG_ARM_FW_REG(0)#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1 KVM_REG_ARM_FW_REG(1)#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_AVAIL 0
@@ -288,6 +293,18 @@ struct kvm_arm_copy_mte_tags {#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3_AVAIL 1#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3_NOT_REQUIRED 2 +/* SDEI registers */ +#define KVM_REG_ARM_SDEI_EVENT_HANDLER_0 KVM_REG_ARM_FW_REG_128(4) +#define KVM_REG_ARM_SDEI_EVENT_HANDLER_1 KVM_REG_ARM_FW_REG_128(5) +#define KVM_REG_ARM_SDEI_EVENT_HANDLER_2 KVM_REG_ARM_FW_REG_128(6) +#define KVM_REG_ARM_SDEI_EVENT_HANDLER_3 KVM_REG_ARM_FW_REG_128(7) +#define KVM_REG_ARM_SDEI_EVENT_REGISTERED KVM_REG_ARM_FW_REG(8) +#define KVM_REG_ARM_SDEI_EVENT_ENABLED KVM_REG_ARM_FW_REG(9) +#define KVM_REG_ARM_SDEI_EVENT_RUNNING KVM_REG_ARM_FW_REG(10) +#define KVM_REG_ARM_SDEI_EVENT_PENDING KVM_REG_ARM_FW_REG(11) +#define KVM_REG_ARM_SDEI_EVENT_CONTEXT KVM_REG_ARM_FW_REG_2048(12) +#define KVM_REG_ARM_SDEI_PE_STATE KVM_REG_ARM_FW_REG(13) +/* SVE registers */ #define KVM_REG_ARM64_SVE (0x15 << KVM_REG_ARM_COPROC_SHIFT)
arch/arm64/kvm/hypercalls.c +32 -0
@@ -290,6 +290,16 @@ static const u64 kvm_arm_fw_reg_ids[] = {KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1 ,KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2 ,KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3 , +KVM_REG_ARM_SDEI_EVENT_HANDLER_0 , +KVM_REG_ARM_SDEI_EVENT_HANDLER_1 , +KVM_REG_ARM_SDEI_EVENT_HANDLER_2 , +KVM_REG_ARM_SDEI_EVENT_HANDLER_3 , +KVM_REG_ARM_SDEI_EVENT_REGISTERED , +KVM_REG_ARM_SDEI_EVENT_ENABLED , +KVM_REG_ARM_SDEI_EVENT_RUNNING , +KVM_REG_ARM_SDEI_EVENT_PENDING , +KVM_REG_ARM_SDEI_EVENT_CONTEXT , +KVM_REG_ARM_SDEI_PE_STATE ,KVM_REG_ARM_STD_BMAP ,KVM_REG_ARM_STD_HYP_BMAP , KVM_REG_ARM_VENDOR_HYP_BMAP,
@@ -387,6 +397,17 @@ int kvm_arm_get_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)case KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3 : val =get_kernel_wa_level (reg->id) &KVM_REG_FEATURE_LEVEL_MASK ;break ; +case KVM_REG_ARM_SDEI_EVENT_HANDLER_0 : +case KVM_REG_ARM_SDEI_EVENT_HANDLER_1 : +case KVM_REG_ARM_SDEI_EVENT_HANDLER_2 : +case KVM_REG_ARM_SDEI_EVENT_HANDLER_3 : +case KVM_REG_ARM_SDEI_EVENT_REGISTERED : +case KVM_REG_ARM_SDEI_EVENT_ENABLED : +case KVM_REG_ARM_SDEI_EVENT_RUNNING : +case KVM_REG_ARM_SDEI_EVENT_PENDING : +case KVM_REG_ARM_SDEI_EVENT_CONTEXT : +case KVM_REG_ARM_SDEI_PE_STATE : +return kvm_sdei_get_reg (vcpu, reg);case KVM_REG_ARM_STD_BMAP : val =READ_ONCE (smccc_feat->std_bmap);break ;
@@ -537,6 +558,17 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)return -EINVAL ;return 0 ; +case KVM_REG_ARM_SDEI_EVENT_HANDLER_0 : +case KVM_REG_ARM_SDEI_EVENT_HANDLER_1 : +case KVM_REG_ARM_SDEI_EVENT_HANDLER_2 : +case KVM_REG_ARM_SDEI_EVENT_HANDLER_3 : +case KVM_REG_ARM_SDEI_EVENT_REGISTERED : +case KVM_REG_ARM_SDEI_EVENT_ENABLED : +case KVM_REG_ARM_SDEI_EVENT_RUNNING : +case KVM_REG_ARM_SDEI_EVENT_PENDING : +case KVM_REG_ARM_SDEI_EVENT_CONTEXT : +case KVM_REG_ARM_SDEI_PE_STATE : +return kvm_sdei_set_reg (vcpu, reg);case KVM_REG_ARM_STD_BMAP :case KVM_REG_ARM_STD_HYP_BMAP :case KVM_REG_ARM_VENDOR_HYP_BMAP :
arch/arm64/kvm/sdei.c +161 -0
167 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 20/22 KVM: arm64: Add SDEI document
In reply to Gavin ShanThis adds SDEI document, to explain what has been missed from the specification and the pesudo firmware registers used for migration.
--- Documentation/virt/kvm/arm/sdei.rst | 64 +++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 Documentation/virt/kvm/arm/sdei.rst
Documentation/virt/kvm/arm/sdei.rst new file +64 -0
@@ -0,0 +1,64 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================================== +SDEI Virtualization Support for ARM64 +===================================== + +ARM specification DEN0054/C defines Software Delegated Exception Interface +(SDEI). It provides a mechanism for registering and servicing system events +from system firmware. The interface is offered by a higher exception level +to a lower exception level, in other words by a secure platform firmware +to hypervisor or hypervisor to OS or both. + +https://developer.arm.com/documentation/den0054/c + +KVM/arm64 implementation follows the specification to support the defined +hypercalls so that the system events can be registered and serviced from +KVM hypervisor to the guest OS. However, some of specified functionalities +are missed by the implementation. + + * The shared event is not supported. It means all supported events are + private. They need to be registered, enabled, disabled, unregistered + and reset from one particular PE (vCPU). + + * The critical priority is not supported. It means all supported events + have normal priority. So there is no preemption between two events + in the critical and normal priorities. One event can be running at + once on one particular PE (vCPU). + + * Interrupt binding event is not supported. It means all supported + events triggered by software. + + * Relative mode for the event handler's address is not supported. + The event handler address is always an absolute address. + +The event handlers, states and context need to be migrated. Several pseudo +firmware registers are added for this. + + * KVM_REG_ARM_SDEI_EVENT_HANDLER_0 + KVM_REG_ARM_SDEI_EVENT_HANDLER_1 + KVM_REG_ARM_SDEI_EVENT_HANDLER_2 + KVM_REG_ARM_SDEI_EVENT_HANDLER_3 + + The event handler's address and argument for events, whose number + are 0, 1, 2 and 3 respectively. Currently, there are only two + supported events, which are Software Signaled Event (0) and Async + PF Event (1). So the first two registers are only used. zeroes are + returned on reading KVM_REG_ARM_SDEI_EVENT_HANDLER_{2, 3}, and the + values written to them are ignored. + + * KVM_REG_ARM_SDEI_EVENT_REGISTERED + KVM_REG_ARM_SDEI_EVENT_ENABLED + KVM_REG_ARM_SDEI_EVENT_RUNNING + KVM_REG_ARM_SDEI_EVENT_PENDING + + They are mapped to the registered, enabled, running and pending + bitmap respectively. + + * KVM_REG_ARM_SDEI_EVENT_CONTEXT + + The interrupted context. + + * KVM_REG_ARM_SDEI_PE_STATE + + Relect the SDEI masked state on the PE (vCPU).
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 21/22 selftests: KVM: aarch64: Add SDEI case in hypercall tests
In reply to Gavin ShanThis adds SDEI case in hypercall tests. The added hypercalls are issued to mimic what guest kernel does: reset the shared and private events, unmask PE, register/enable/disable/unregister event, and then mask PE.
--- tools/testing/selftests/kvm/aarch64/hypercalls.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/aarch64/hypercalls.c +10 -1
@@ -11,6 +11,7 @@#include <errno.h> #include <linux/arm-smccc.h> +#include <linux/arm_sdei.h> #include <asm/kvm.h> #include <kvm_util.h>
@@ -19,7 +20,7 @@#define FW_REG_ULIMIT_VAL (max_feat_bit) (GENMASK(max_feat_bit, 0))/* Last valid bits of the bitmapped firmware registers */ -#define KVM_REG_ARM_STD_BMAP_BIT_MAX 0 +#define KVM_REG_ARM_STD_BMAP_BIT_MAX 1#define KVM_REG_ARM_STD_HYP_BMAP_BIT_MAX 0#define KVM_REG_ARM_VENDOR_HYP_BMAP_BIT_MAX 1
@@ -68,6 +69,14 @@ static const struct test_hvc_info hvc_info[] = {TEST_HVC_INFO (ARM_SMCCC_TRNG_GET_UUID ,0 ),TEST_HVC_INFO (ARM_SMCCC_TRNG_RND32 ,0 ),TEST_HVC_INFO (ARM_SMCCC_TRNG_RND64 ,0 ), +TEST_HVC_INFO (SDEI_1_0_FN_SDEI_SHARED_RESET ,0 ), +TEST_HVC_INFO (SDEI_1_0_FN_SDEI_PRIVATE_RESET ,0 ), +TEST_HVC_INFO (SDEI_1_0_FN_SDEI_PE_UNMASK ,0 ), +TEST_HVC_INFO (SDEI_1_0_FN_SDEI_EVENT_REGISTER ,SDEI_SW_SIGNALED_EVENT ), +TEST_HVC_INFO (SDEI_1_0_FN_SDEI_EVENT_ENABLE ,SDEI_SW_SIGNALED_EVENT ), +TEST_HVC_INFO (SDEI_1_0_FN_SDEI_EVENT_DISABLE ,SDEI_SW_SIGNALED_EVENT ), +TEST_HVC_INFO (SDEI_1_0_FN_SDEI_EVENT_UNREGISTER ,SDEI_SW_SIGNALED_EVENT ), +TEST_HVC_INFO (SDEI_1_0_FN_SDEI_PE_MASK ,0 ),/* KVM_REG_ARM_STD_HYP_BMAP */ TEST_HVC_INFO(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, ARM_SMCCC_HV_PV_TIME_FEATURES),
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v7 22/22 selftests: KVM: aarch64: Add SDEI test case
In reply to Gavin ShanThis adds SDEI selftest case where the various hypercalls are issued to the software signaled event, whose number is zero. The event is private, signaled and in normal priority.
By default, two vCPUs are started and the following ioctl commands or hypercalls are sent to them in sequence, to simulate how they are used in VMM and the linux guest:
kvm_check_cap(KVM_CAP_ARM_SDEI) (Available functionality)
SDEI_1_0_FN_SDEI_VERSION
SDEI_1_1_FN_SDEI_FEATURES (SDEI capability probing) SDEI_1_0_FN_SDEI_SHARED_RESET (restart SDEI) SDEI_1_0_FN_SDEI_PE_UNMASK (CPU online)
SDEI_1_0_FN_SDEI_EVENT_GET_INFO
SDEI_1_0_FN_SDEI_EVENT_REGISTER (register event) SDEI_1_0_FN_SDEI_EVENT_ENABLE (enable event) SDEI_1_1_FN_SDEI_EVENT_SIGNAL (event injection)
SDEI_1_0_FN_SDEI_EVENT_DISABLE (disable event) SDEI_1_0_FN_SDEI_EVENT_UNREGISTER (unregister event) SDEI_1_0_FN_SDEI_PE_MASK (CPU offline)
--- tools/testing/selftests/kvm/Makefile | 1 + tools/testing/selftests/kvm/aarch64/sdei.c | 450 +++++++++++++++++++++ 2 files changed, 451 insertions(+) create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -109,6 +109,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/debug-exceptions TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list TEST_GEN_PROGS_aarch64 += aarch64/hypercalls TEST_GEN_PROGS_aarch64 += aarch64/psci_test +TEST_GEN_PROGS_aarch64 += aarch64/sdei TEST_GEN_PROGS_aarch64 += aarch64/vcpu_width_config TEST_GEN_PROGS_aarch64 += aarch64/vgic_init TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq
tools/testing/selftests/kvm/aarch64/sdei.c new file +450 -0
450 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi Gavin,
Quoted 158 lines · Gavin Shan
[EXTERNAL EMAIL NOTICE: This email originated from an external sender. Please be mindful of safe email handling and proprietary information protection practices.]
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei")The design is quite strightforward by following the specification. The (SDEI) events are classified into the shared and private ones according to their scope. The shared event is system or VM scoped, but the private event is vcpu scoped. This implementation doesn't support the shared event because all the needed events are private. Besides, the critial events aren't supported by the implementation either. It means all events are normal in terms of priority.
There are several objects (data structures) introduced to help on the event registration, enablement, disablement, unregistration, reset, delivery and handling.
*
kvm_sdei_event_handler
SDEI event handler, which is provided throughEVENT_REGISTERhypercall, is called when the SDEI event is delivered from host to guest.*
kvm_sdei_event_context
The saved (preempted) context when SDEI event is delivered for handling.*
kvm_sdei_vcpu
SDEI events and their states.The patches are organized as below:
PATCH[01-02] Preparatory work to extend
smccc_get_argx()and refactor
hypercall routing mechanism
PATCH[03] Adds SDEI virtualization infrastructure
PATCH[04-16] Supports various SDEI hypercalls and event handling
PATCH[17] Exposes SDEI capability
PATCH[18-19] Support SDEI migration
PATCH[20] Adds document about SDEI
PATCH[21-22] SDEI related selftest casesThe previous revisions can be found:
v6: https://lore.kernel.org/lkml/[email protected]/T/ v5: https://lore.kernel.org/kvmarm/[email protected]/ v4: https://lore.kernel.org/kvmarm/[email protected]/ v3: https://lore.kernel.org/kvmarm/[email protected]/ v2: https://lore.kernel.org/kvmarm/[email protected]/ v1: https://lore.kernel.org/kvmarm/[email protected]/Testing
=======
[1] The selftest case included in this series works fine. The default SDEIevent, whose number is zero, can be registered, enabled, raised. The SDEI event handler can be invoked. [host]# pwd /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm [root@virtlab-arm01 kvm]# ./aarch64/sdei NR_VCPUS: 2 SDEI Event: 0x00000000 --- VERSION Version: 1.1 (vendor: 0x4b564d) --- FEATURES Shared event slots: 0 Private event slots: 0 Relative mode: No --- PRIVATE_RESET --- SHARED_RESET --- PE_UNMASK --- EVENT_GET_INFO Type: Private Priority: Normal Signaled: Yes --- EVENT_REGISTER --- EVENT_ENABLE --- EVENT_SIGNAL Handled: Yes IRQ: No Status: Registered-Enabled-Running PC/PSTATE: 000000000040232c 00000000600003c5 Regs: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 --- PE_MASK --- EVENT_DISABLE --- EVENT_UNREGISTER Result: OK[2] There are additional patches in the following repositories to create
procfs entries, allowing to inject SDEI event from host side. The SDEI client in the guest side registers the SDEI default event, whose number is zero. Also, the QEMU exports SDEI ACPI table and supports migration for SDEI. https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei") [2.1] Start the guests and migrate the source VM to the destination VM. [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 6,sockets=2,cores=3,threads=1 \ -m 1024M,slots=16,maxmem=64G \ : \ -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \ -initrd /home/gavin/sandbox/images/rootfs.cpio.xz \ -append earlycon=pl011,mmio,0x9000000 \ : [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 6,sockets=2,cores=3,threads=1 \ -m 1024M,slots=16,maxmem=64G \ : \ -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \ -initrd /home/gavin/sandbox/images/rootfs.cpio.xz \ -append earlycon=pl011,mmio,0x9000000 \ -incoming tcp:0:4444 \ : [2.2] Check kernel log on the source VM. The SDEI service is enabled and the default SDEI event (0x0) is enabled. [guest-src]# dmesg | grep -i sdei ACPI: SDEI 0x000000005BC80000 000024 \ (v00 BOCHS BXPC 00000001 BXPC 00000001) sdei: SDEIv1.1 (0x4b564d) detected in firmware. SDEI TEST: Version 1.1, Vendor 0x4b564d sdei_init: SDEI event (0x0) registered sdei_init: SDEI event (0x0) enabled (qemu) migrate -d tcp:localhost:4444 [2.3] Migrate the source VM to the destination VM. Inject SDEI event to the destination VM. The event is raised and handled. (qemu) migrate -d tcp:localhost:4444 [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1 [guest-dst]# =========== SDEI Event (CPU#1) =========== Event: 0000000000000000 Parameter: 00000000dabfdabf PC: ffff800008cbb554 PSTATE: 00000000604000c5 SP: ffff800009c7bde0 Regs: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Quoted 8 lines
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff800009399008
ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18
0000000000000000 0000000000000000 ffff000000339d00 0000000000000000
0000000000000000 ffff800009c7bde0 ffff800008cbb5c4
Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190
ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000I tested this patch set. It's okay.
Thanks
Huang Shijie
Quoted 78 lines
Changelog
=========
v7:
* Rebased to v5.19.rc1 (Gavin)
* Add hypercall ranges for routing (Oliver)
* Remove support to the critical event and redesigned data structures. Function names are also modified
as Oliver suggested (Oliver)
* Deliver event when it's enabled or the specific PE
is unmasked (Oliver)
* ImproveEVENT_COMPLETE_AND_RESUMEhypercall to resume
from the specified address (Oliver)
* Add patches for SDEI migration and documentation (Gavin)
* Misc comments from Oliver Upon (Oliver)
v6:
* Rebased to v5.18.rc1 (Gavin)
* Pass additional argument tosmccc_get_arg()(Oliver)
* Add preparatory patch to route hypercalls based on their
owners (Oliver)
* Remove the support for shared event. (Oliver/Gavin)
* Remove the support for migration and add-on patches to
support it in future (Oliver)
* The events are exposed by KVM instead of VMM (Oliver)
*kvm_sdei_state.his dropped and all the structures are
folded into the corresponding ones inkvm_sdei.h(Oliver)
* Rename 'structkvm_sdei_registered_event' to
'structkvm_sdei_event' (Oliver)
* Misc comments from Oliver Upon (Oliver)
v5/v4/v3/v2/v1:
* Skipped here and please visit the history by
https://lore.kernel.org/lkml/[email protected]/T/Gavin Shan (22):
KVM: arm64: Extendsmccc_get_argx()
KVM: arm64: Route hypercalls based on their owner
KVM: arm64: Add SDEI virtualization infrastructure
KVM: arm64: SupportEVENT_REGISTERhypercall
KVM: arm64: Support EVENT_{ENABLE, DISABLE} hypercall
KVM: arm64: SupportEVENT_CONTEXThypercall
KVM: arm64: SupportEVENT_UNREGISTERhypercall
KVM: arm64: SupportEVENT_STATUShypercall
KVM: arm64: SupportEVENT_GET_INFOhypercall
KVM: arm64: Support PE_{MASK, UNMASK} hypercall
KVM: arm64: Support {PRIVATE, SHARED}_RESET hypercall
KVM: arm64: Support event injection and delivery
KVM: arm64: Support EVENT_{COMPLETE,COMPLETE_AND_RESUME} hypercall
KVM: arm64: SupportEVENT_SIGNALhypercall
KVM: arm64: SupportSDEI_FEATUREShypercall
KVM: arm64: SupportSDEI_VERSIONhypercall
KVM: arm64: Expose SDEI capbility and service
KVM: arm64: Allow large sized pseudo firmware registers
KVM: arm64: Support SDEI event migration
KVM: arm64: Add SDEI document
selftests: KVM: aarch64: Add SDEI case in hypercall tests
selftests: KVM: aarch64: Add SDEI test caseDocumentation/virt/kvm/api.rst | 11 + Documentation/virt/kvm/arm/hypercalls.rst | 4 + Documentation/virt/kvm/arm/sdei.rst | 64 ++ arch/arm64/include/asm/kvm_host.h | 3 + arch/arm64/include/asm/kvm_sdei.h | 81 +++ arch/arm64/include/uapi/asm/kvm.h | 18 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 8 + arch/arm64/kvm/hypercalls.c | 182 +++-- arch/arm64/kvm/psci.c | 14 +- arch/arm64/kvm/pvtime.c | 2 +- arch/arm64/kvm/sdei.c | 676 ++++++++++++++++++ arch/arm64/kvm/trng.c | 4 +- include/kvm/arm_hypercalls.h | 19 +- include/linux/arm-smccc.h | 7 + include/uapi/linux/arm_sdei.h | 8 + include/uapi/linux/kvm.h | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/aarch64/hypercalls.c | 11 +- tools/testing/selftests/kvm/aarch64/sdei.c | 450 ++++++++++++ 20 files changed, 1499 insertions(+), 67 deletions(-)
Quoted 8 lines
create mode 100644
Documentation/virt/kvm/arm/sdei.rstcreate mode 100644arch/arm64/include/asm/kvm_sdei.hcreate mode 100644arch/arm64/kvm/sdei.c
create mode 100644tools/testing/selftests/kvm/aarch64/sdei.c--
2.23.0
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi Shijie,
Quoted 166 lines · Shijie Huang
Quoted 149 lines · Gavin Shan
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")The design is quite strightforward by following the specification. The (SDEI) events are classified into the shared and private ones according to their scope. The shared event is system or VM scoped, but the private event is vcpu scoped. This implementation doesn't support the shared event because all the needed events are private. Besides, the critial events aren't supported by the implementation either. It means all events are normal in terms of priority.
There are several objects (data structures) introduced to help on the event registration, enablement, disablement, unregistration, reset, delivery and handling.
*
kvm_sdei_event_handler
SDEI event handler, which is provided throughEVENT_REGISTERhypercall, is called when the SDEI event is delivered from host to guest.*
kvm_sdei_event_context
The saved (preempted) context when SDEI event is delivered for handling.*
kvm_sdei_vcpu
SDEI events and their states.The patches are organized as below:
PATCH[01-02] Preparatory work to extend
smccc_get_argx()and refactor hypercall routing mechanism
PATCH[03] Adds SDEI virtualization infrastructure PATCH[04-16] Supports various SDEI hypercalls and event handling PATCH[17] Exposes SDEI capability
PATCH[18-19] Support SDEI migration
PATCH[20] Adds document about SDEI
PATCH[21-22] SDEI related selftest casesThe previous revisions can be found:
v6: https://lore.kernel.org/lkml/[email protected]/T/
v5: https://lore.kernel.org/kvmarm/[email protected]/
v4: https://lore.kernel.org/kvmarm/[email protected]/
v3: https://lore.kernel.org/kvmarm/[email protected]/
v2: https://lore.kernel.org/kvmarm/[email protected]/
v1: https://lore.kernel.org/kvmarm/[email protected]/Testing
=======
[1] The selftest case included in this series works fine. The default SDEI event, whose number is zero, can be registered, enabled, raised. The SDEI event handler can be invoked.[host]# pwd
/home/gavin/sandbox/linux.main/tools/testing/selftests/kvm[root@virtlab-arm01 kvm]# ./aarch64/sdei
NR_VCPUS: 2 SDEI Event:0x00000000--- VERSION
Version: 1.1 (vendor:0x4b564d) --- FEATURES
Shared event slots: 0
Private event slots: 0
Relative mode: No
---PRIVATE_RESET
---SHARED_RESET
---PE_UNMASK
---EVENT_GET_INFO
Type: Private
Priority: Normal
Signaled: Yes
---EVENT_REGISTER
---EVENT_ENABLE
---EVENT_SIGNAL
Handled: Yes
IRQ: No
Status: Registered-Enabled-Running PC/PSTATE:000000000040232c00000000600003c5Regs: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ---PE_MASK
---EVENT_DISABLE
---EVENT_UNREGISTERResult: OK
[2] There are additional patches in the following repositories to create procfs entries, allowing to inject SDEI event from host side. The SDEI client in the guest side registers the SDEI default event, whose number is zero. Also, the QEMU exports SDEI ACPI table and supports migration for SDEI.
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")[2.1] Start the guests and migrate the source VM to the destination VM.
[host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,cores=3,threads=1 \
-m 1024M,slots=16,maxmem=64G \
: \
-kernel/home/gavin/sandbox/linux.guest/arch/arm64/boot/Image\
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x9000000 \
:[host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,cores=3,threads=1 \
-m 1024M,slots=16,maxmem=64G \
: \
-kernel/home/gavin/sandbox/linux.guest/arch/arm64/boot/Image\
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x9000000 \
-incoming tcp:0:4444 \
:[2.2] Check kernel log on the source VM. The SDEI service is enabled and the default SDEI event (
0x0) is enabled.[guest-src]# dmesg | grep -i sdei
ACPI: SDEI0x000000005BC80000000024 \
(v00 BOCHS BXPC 00000001 BXPC 00000001)
sdei: SDEIv1.1 (0x4b564d) detected in firmware. SDEI TEST: Version 1.1, Vendor0x4b564d
sdei_init: SDEI event (0x0) registered
sdei_init: SDEI event (0x0) enabled(qemu) migrate -d tcp:localhost:4444
[2.3] Migrate the source VM to the destination VM. Inject SDEI event to the destination VM. The event is raised and handled.
(qemu) migrate -d tcp:localhost:4444
[host]# echo 0 >
/proc/kvm/kvm-5360/vcpu-1Quoted 15 lines
[guest-dst]#
=========== SDEI Event (CPU#1) ===========
Event: 0000000000000000 Parameter:00000000dabfdabf
PC: ffff800008cbb554 PSTATE: 00000000604000c5 SP:ffff800009c7bde0Regs:00000000000016eeffff00001ffd2e2800000000000016ed0000000000000001ffff800016c280000000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000ffff800009399008ffff8000097d9af0ffff8000097d99f8ffff8000093a8db8ffff8000097d9b180000000000000000 0000000000000000ffff000000339d000000000000000000 0000000000000000ffff800009c7bde0ffff800008cbb5c4
Context:00000000000016eeffff00001ffd2e2800000000000016ed0000000000000001ffff800016c2800003ffffffffffffff000000024325db59ffff8000097de190ffff00000033a790ffff800008cbb8140000000000000a300000000000000000
Quoted 4 lines
I tested this patch set. It's okay.
Tested-by Huang Shijie <[email protected]>
[...]
Appreciate your efforts to test it through. I will have your tested-by if respin is needed. Thank you for your time on this.
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Vishnu,
Quoted 188 lines · Vishnu Pajjuri
Quoted 169 lines · Gavin Shan
Quoted 168 lines · Shijie Huang
Quoted 149 lines · Gavin Shan
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")The design is quite strightforward by following the specification. The (SDEI) events are classified into the shared and private ones according to their scope. The shared event is system or VM scoped, but the private event is vcpu scoped. This implementation doesn't support the shared event because all the needed events are private. Besides, the critial events aren't supported by the implementation either. It means all events are normal in terms of priority.
There are several objects (data structures) introduced to help on the event registration, enablement, disablement, unregistration, reset, delivery and handling.
*
kvm_sdei_event_handler
SDEI event handler, which is provided throughEVENT_REGISTERhypercall, is called when the SDEI event is delivered from host to guest.*
kvm_sdei_event_context
The saved (preempted) context when SDEI event is delivered for handling.*
kvm_sdei_vcpu
SDEI events and their states.The patches are organized as below:
PATCH[01-02] Preparatory work to extend
smccc_get_argx()and refactor hypercall routing mechanism
PATCH[03] Adds SDEI virtualization infrastructure PATCH[04-16] Supports various SDEI hypercalls and event handling PATCH[17] Exposes SDEI capability
PATCH[18-19] Support SDEI migration
PATCH[20] Adds document about SDEI
PATCH[21-22] SDEI related selftest casesThe previous revisions can be found:
v6: https://lore.kernel.org/lkml/[email protected]/T/
v5: https://lore.kernel.org/kvmarm/[email protected]/
v4: https://lore.kernel.org/kvmarm/[email protected]/
v3: https://lore.kernel.org/kvmarm/[email protected]/
v2: https://lore.kernel.org/kvmarm/[email protected]/
v1: https://lore.kernel.org/kvmarm/[email protected]/Testing
=======
[1] The selftest case included in this series works fine. The default SDEI event, whose number is zero, can be registered, enabled, raised. The SDEI event handler can be invoked.[host]# pwd
/home/gavin/sandbox/linux.main/tools/testing/selftests/kvm[root@virtlab-arm01 kvm]# ./aarch64/sdei
NR_VCPUS: 2 SDEI Event:0x00000000--- VERSION
Version: 1.1 (vendor:0x4b564d) --- FEATURES
Shared event slots: 0
Private event slots: 0
Relative mode: No
---PRIVATE_RESET
---SHARED_RESET
---PE_UNMASK
---EVENT_GET_INFO
Type: Private
Priority: Normal
Signaled: Yes
---EVENT_REGISTER
---EVENT_ENABLE
---EVENT_SIGNAL
Handled: Yes
IRQ: No
Status: Registered-Enabled-Running PC/PSTATE:000000000040232c00000000600003c5Regs: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ---PE_MASK
---EVENT_DISABLE
---EVENT_UNREGISTERResult: OK
[2] There are additional patches in the following repositories to create procfs entries, allowing to inject SDEI event from host side. The SDEI client in the guest side registers the SDEI default event, whose number is zero. Also, the QEMU exports SDEI ACPI table and supports migration for SDEI.
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")[2.1] Start the guests and migrate the source VM to the destination VM.
[host]#
/home/gavin/sandbox/qemu.main/build/qemu-system-aarch64\
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,cores=3,threads=1 \
-m 1024M,slots=16,maxmem=64G \
: \
-kernel/home/gavin/sandbox/linux.guest/arch/arm64/boot/Image\
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x9000000 \
:[host]#
/home/gavin/sandbox/qemu.main/build/qemu-system-aarch64\
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,cores=3,threads=1 \
-m 1024M,slots=16,maxmem=64G \
: \
-kernel/home/gavin/sandbox/linux.guest/arch/arm64/boot/Image\
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x9000000 \
-incoming tcp:0:4444 \
:[2.2] Check kernel log on the source VM. The SDEI service is enabled and the default SDEI event (
0x0) is enabled.[guest-src]# dmesg | grep -i sdei
ACPI: SDEI0x000000005BC80000000024 \
(v00 BOCHS BXPC 00000001 BXPC 00000001)
sdei: SDEIv1.1 (0x4b564d) detected in firmware. SDEI TEST: Version 1.1, Vendor0x4b564d
sdei_init: SDEI event (0x0) registered
sdei_init: SDEI event (0x0) enabled(qemu) migrate -d tcp:localhost:4444
[2.3] Migrate the source VM to the destination VM. Inject SDEI event to the destination VM. The event is raised and handled.
(qemu) migrate -d tcp:localhost:4444
[host]# echo 0 >
/proc/kvm/kvm-5360/vcpu-1Quoted 15 lines
[guest-dst]#
=========== SDEI Event (CPU#1) ===========
Event: 0000000000000000 Parameter:00000000dabfdabf
PC: ffff800008cbb554 PSTATE: 00000000604000c5 SP:ffff800009c7bde0Regs:00000000000016eeffff00001ffd2e2800000000000016ed0000000000000001ffff800016c280000000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000ffff800009399008ffff8000097d9af0ffff8000097d99f8ffff8000093a8db8ffff8000097d9b180000000000000000 0000000000000000ffff000000339d000000000000000000 0000000000000000ffff800009c7bde0ffff800008cbb5c4
Context:00000000000016eeffff00001ffd2e2800000000000016ed0000000000000001ffff800016c2800003ffffffffffffff000000024325db59ffff8000097de190ffff00000033a790ffff800008cbb8140000000000000a300000000000000000I tested this patch set. It's okay.
Quoted 8 lines
Quoted 2 lines
Tested-by Huang Shijie <[email protected]>[...]
Appreciate your efforts to test it through. I will have your tested-by if respin is needed. Thank you for your time on this.
I would like to know the current latest status of this patch series,
Since I didn't find any latest spin for this patch series.
Also, I didn't find any active development branch at
https://github.com/gwshan/linux.
And I observed that the kernel development branch
Quoted 8 lines
"https://github.com/gwshan/linuxkvm/arm64_sdei" moved to
"https://github.com/gwshan/linuxbackup/kvm/arm64_sdei"
I'm curious that what is required to spin this patch series with latest kernel versions
Appreciate any other insights on this...
Thanks for raising concerns. The SDEI events were used to deliver signals, required by the asynchronous page fault (Aync PF). I had several discussions with Marc and Paolo, and we reached to the temporary conclusion that Async PF isn't used in production enviornments like google cloud. I suspended the efforts since then. The SDEI virtualization support won't be needed if we needn't Async PF, unless there are other signals needed to be delivered by SDEI events.
Thanks,
Gavin
Hi Oliver,
Quoted 45 lines · Gavin Shan
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei")The design is quite strightforward by following the specification. The (SDEI) events are classified into the shared and private ones according to their scope. The shared event is system or VM scoped, but the private event is vcpu scoped. This implementation doesn't support the shared event because all the needed events are private. Besides, the critial events aren't supported by the implementation either. It means all events are normal in terms of priority.
There are several objects (data structures) introduced to help on the event registration, enablement, disablement, unregistration, reset, delivery and handling.
*
kvm_sdei_event_handler
SDEI event handler, which is provided throughEVENT_REGISTERhypercall, is called when the SDEI event is delivered from host to guest.*
kvm_sdei_event_context
The saved (preempted) context when SDEI event is delivered for handling.*
kvm_sdei_vcpu
SDEI events and their states.The patches are organized as below:
PATCH[01-02] Preparatory work to extend
smccc_get_argx()and refactor
hypercall routing mechanism
PATCH[03] Adds SDEI virtualization infrastructure
PATCH[04-16] Supports various SDEI hypercalls and event handling
PATCH[17] Exposes SDEI capability
PATCH[18-19] Support SDEI migration
PATCH[20] Adds document about SDEI
PATCH[21-22] SDEI related selftest cases
Quoted 9 lines
The previous revisions can be found:
v6: https://lore.kernel.org/lkml/[email protected]/T/ v5: https://lore.kernel.org/kvmarm/[email protected]/ v4: https://lore.kernel.org/kvmarm/[email protected]/ v3: https://lore.kernel.org/kvmarm/[email protected]/ v2: https://lore.kernel.org/kvmarm/[email protected]/ v1: https://lore.kernel.org/kvmarm/[email protected]/
Copying Oliver's new email address ([email protected]).
Please let me know if I need to rebase and repost the series.
Thanks,
Gavin
Quoted 187 lines
Testing
=======
[1] The selftest case included in this series works fine. The default SDEIevent, whose number is zero, can be registered, enabled, raised. The SDEI event handler can be invoked. [host]# pwd /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm [root@virtlab-arm01 kvm]# ./aarch64/sdei NR_VCPUS: 2 SDEI Event: 0x00000000 --- VERSION Version: 1.1 (vendor: 0x4b564d) --- FEATURES Shared event slots: 0 Private event slots: 0 Relative mode: No --- PRIVATE_RESET --- SHARED_RESET --- PE_UNMASK --- EVENT_GET_INFO Type: Private Priority: Normal Signaled: Yes --- EVENT_REGISTER --- EVENT_ENABLE --- EVENT_SIGNAL Handled: Yes IRQ: No Status: Registered-Enabled-Running PC/PSTATE: 000000000040232c 00000000600003c5 Regs: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 --- PE_MASK --- EVENT_DISABLE --- EVENT_UNREGISTER Result: OK[2] There are additional patches in the following repositories to create
procfs entries, allowing to inject SDEI event from host side. The SDEI client in the guest side registers the SDEI default event, whose number is zero. Also, the QEMU exports SDEI ACPI table and supports migration for SDEI. https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei") [2.1] Start the guests and migrate the source VM to the destination VM. [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 6,sockets=2,cores=3,threads=1 \ -m 1024M,slots=16,maxmem=64G \ : \ -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \ -initrd /home/gavin/sandbox/images/rootfs.cpio.xz \ -append earlycon=pl011,mmio,0x9000000 \ : [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 6,sockets=2,cores=3,threads=1 \ -m 1024M,slots=16,maxmem=64G \ : \ -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \ -initrd /home/gavin/sandbox/images/rootfs.cpio.xz \ -append earlycon=pl011,mmio,0x9000000 \ -incoming tcp:0:4444 \ : [2.2] Check kernel log on the source VM. The SDEI service is enabled and the default SDEI event (0x0) is enabled. [guest-src]# dmesg | grep -i sdei ACPI: SDEI 0x000000005BC80000 000024 \ (v00 BOCHS BXPC 00000001 BXPC 00000001) sdei: SDEIv1.1 (0x4b564d) detected in firmware. SDEI TEST: Version 1.1, Vendor 0x4b564d sdei_init: SDEI event (0x0) registered sdei_init: SDEI event (0x0) enabled (qemu) migrate -d tcp:localhost:4444 [2.3] Migrate the source VM to the destination VM. Inject SDEI event to the destination VM. The event is raised and handled. (qemu) migrate -d tcp:localhost:4444 [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1 [guest-dst]# =========== SDEI Event (CPU#1) =========== Event: 0000000000000000 Parameter: 00000000dabfdabf PC: ffff800008cbb554 PSTATE: 00000000604000c5 SP: ffff800009c7bde0 Regs: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff800009399008 ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000Changelog
=========
v7:
* Rebased to v5.19.rc1 (Gavin)
* Add hypercall ranges for routing (Oliver)
* Remove support to the critical event and redesigned data structures. Function names are also modified
as Oliver suggested (Oliver)
* Deliver event when it's enabled or the specific PE
is unmasked (Oliver)
* ImproveEVENT_COMPLETE_AND_RESUMEhypercall to resume
from the specified address (Oliver)
* Add patches for SDEI migration and documentation (Gavin)
* Misc comments from Oliver Upon (Oliver)
v6:
* Rebased to v5.18.rc1 (Gavin)
* Pass additional argument tosmccc_get_arg()(Oliver)
* Add preparatory patch to route hypercalls based on their
owners (Oliver)
* Remove the support for shared event. (Oliver/Gavin)
* Remove the support for migration and add-on patches to
support it in future (Oliver)
* The events are exposed by KVM instead of VMM (Oliver)
*kvm_sdei_state.his dropped and all the structures are
folded into the corresponding ones inkvm_sdei.h(Oliver)
* Rename 'structkvm_sdei_registered_event' to
'structkvm_sdei_event' (Oliver)
* Misc comments from Oliver Upon (Oliver)
v5/v4/v3/v2/v1:
* Skipped here and please visit the history by
https://lore.kernel.org/lkml/[email protected]/T/Gavin Shan (22):
KVM: arm64: Extendsmccc_get_argx()
KVM: arm64: Route hypercalls based on their owner
KVM: arm64: Add SDEI virtualization infrastructure
KVM: arm64: SupportEVENT_REGISTERhypercall
KVM: arm64: Support EVENT_{ENABLE, DISABLE} hypercall
KVM: arm64: SupportEVENT_CONTEXThypercall
KVM: arm64: SupportEVENT_UNREGISTERhypercall
KVM: arm64: SupportEVENT_STATUShypercall
KVM: arm64: SupportEVENT_GET_INFOhypercall
KVM: arm64: Support PE_{MASK, UNMASK} hypercall
KVM: arm64: Support {PRIVATE, SHARED}_RESET hypercall
KVM: arm64: Support event injection and delivery
KVM: arm64: Support EVENT_{COMPLETE,COMPLETE_AND_RESUME} hypercall
KVM: arm64: SupportEVENT_SIGNALhypercall
KVM: arm64: SupportSDEI_FEATUREShypercall
KVM: arm64: SupportSDEI_VERSIONhypercall
KVM: arm64: Expose SDEI capbility and service
KVM: arm64: Allow large sized pseudo firmware registers
KVM: arm64: Support SDEI event migration
KVM: arm64: Add SDEI document
selftests: KVM: aarch64: Add SDEI case in hypercall tests
selftests: KVM: aarch64: Add SDEI test caseDocumentation/virt/kvm/api.rst | 11 + Documentation/virt/kvm/arm/hypercalls.rst | 4 + Documentation/virt/kvm/arm/sdei.rst | 64 ++ arch/arm64/include/asm/kvm_host.h | 3 + arch/arm64/include/asm/kvm_sdei.h | 81 +++ arch/arm64/include/uapi/asm/kvm.h | 18 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 8 + arch/arm64/kvm/hypercalls.c | 182 +++-- arch/arm64/kvm/psci.c | 14 +- arch/arm64/kvm/pvtime.c | 2 +- arch/arm64/kvm/sdei.c | 676 ++++++++++++++++++ arch/arm64/kvm/trng.c | 4 +- include/kvm/arm_hypercalls.h | 19 +- include/linux/arm-smccc.h | 7 + include/uapi/linux/arm_sdei.h | 8 + include/uapi/linux/kvm.h | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/aarch64/hypercalls.c | 11 +- tools/testing/selftests/kvm/aarch64/sdei.c | 450 ++++++++++++ 20 files changed, 1499 insertions(+), 67 deletions(-)
Quoted 5 lines
create mode 100644
Documentation/virt/kvm/arm/sdei.rstcreate mode 100644arch/arm64/include/asm/kvm_sdei.hcreate mode 100644arch/arm64/kvm/sdei.c
create mode 100644tools/testing/selftests/kvm/aarch64/sdei.c
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerHi Gavin,
Quoted 57 lines · Gavin Shan
Hi Oliver,
Quoted 43 lines · Gavin Shan
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei")The design is quite strightforward by following the specification. The (SDEI) events are classified into the shared and private ones according to their scope. The shared event is system or VM scoped, but the private event is vcpu scoped. This implementation doesn't support the shared event because all the needed events are private. Besides, the critial events aren't supported by the implementation either. It means all events are normal in terms of priority.
There are several objects (data structures) introduced to help on the event registration, enablement, disablement, unregistration, reset, delivery and handling.
*
kvm_sdei_event_handler
SDEI event handler, which is provided throughEVENT_REGISTERhypercall, is called when the SDEI event is delivered from host to guest.
*kvm_sdei_event_context
The saved (preempted) context when SDEI event is delivered for handling.
*kvm_sdei_vcpu
SDEI events and their states.The patches are organized as below:
PATCH[01-02] Preparatory work to extend
smccc_get_argx()and refactor
hypercall routing mechanism
PATCH[03] Adds SDEI virtualization infrastructure
PATCH[04-16] Supports various SDEI hypercalls and event handling
PATCH[17] Exposes SDEI capability
PATCH[18-19] Support SDEI migration
PATCH[20] Adds document about SDEI
PATCH[21-22] SDEI related selftest casesQuoted 9 lines
The previous revisions can be found:
v6: https://lore.kernel.org/lkml/[email protected]/T/ v5: https://lore.kernel.org/kvmarm/[email protected]/ v4: https://lore.kernel.org/kvmarm/[email protected]/ v3: https://lore.kernel.org/kvmarm/[email protected]/ v2: https://lore.kernel.org/kvmarm/[email protected]/ v1: https://lore.kernel.org/kvmarm/[email protected]/
Quoted 3 lines
Copying Oliver's new email address ([email protected]).
Please let me know if I need to rebase and repost the series.
My main issue with this series is that it is a solution in search of a problem. It is only an enabler for Asynchronous Page Fault support, and:
- as far as I know, the core Linux/arm64 maintainers have no plan to support APF. Without it, this is a pointless exercise. And even with it, this introduces a Linux specific behaviour in an otherwise architectural hypervisor (something I'm quite keen on avoiding)
- It gives an incentive to other hypervisor vendors to add random crap to the Linux mm subsystem, which is even worse. At this stage, we might as well go back to the Xen PV days altogether.
- I haven't seen any of the KVM/arm64 users actually asking for the APF horror, and the cloud vendors I directly asked had no plan to use it, and not using it on their x86 systems either
- no performance data nor workloads that could help making an informed decision have been disclosed, and the only argument in its favour seems to be "but x86 has it" (hardly a compelling one)
Given the above, I don't see how to justify this series, as it has no purpose on its own, no matter how well written it is.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Hi Marc,
Quoted 77 lines · Marc Zyngier
Quoted 54 lines · Gavin Shan
Quoted 43 lines · Gavin Shan
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei")The design is quite strightforward by following the specification. The (SDEI) events are classified into the shared and private ones according to their scope. The shared event is system or VM scoped, but the private event is vcpu scoped. This implementation doesn't support the shared event because all the needed events are private. Besides, the critial events aren't supported by the implementation either. It means all events are normal in terms of priority.
There are several objects (data structures) introduced to help on the event registration, enablement, disablement, unregistration, reset, delivery and handling.
*
kvm_sdei_event_handler
SDEI event handler, which is provided throughEVENT_REGISTERhypercall, is called when the SDEI event is delivered from host to guest.
*kvm_sdei_event_context
The saved (preempted) context when SDEI event is delivered for handling.
*kvm_sdei_vcpu
SDEI events and their states.The patches are organized as below:
PATCH[01-02] Preparatory work to extend
smccc_get_argx()and refactor
hypercall routing mechanism
PATCH[03] Adds SDEI virtualization infrastructure
PATCH[04-16] Supports various SDEI hypercalls and event handling
PATCH[17] Exposes SDEI capability
PATCH[18-19] Support SDEI migration
PATCH[20] Adds document about SDEI
PATCH[21-22] SDEI related selftest casesQuoted 9 lines
The previous revisions can be found:
v6: https://lore.kernel.org/lkml/[email protected]/T/ v5: https://lore.kernel.org/kvmarm/[email protected]/ v4: https://lore.kernel.org/kvmarm/[email protected]/ v3: https://lore.kernel.org/kvmarm/[email protected]/ v2: https://lore.kernel.org/kvmarm/[email protected]/ v1: https://lore.kernel.org/kvmarm/[email protected]/Quoted 3 lines
Copying Oliver's new email address ([email protected]).
Please let me know if I need to rebase and repost the series.
My main issue with this series is that it is a solution in search of a problem. It is only an enabler for Asynchronous Page Fault support, and:
- as far as I know, the core Linux/arm64 maintainers have no plan to support APF. Without it, this is a pointless exercise. And even with it, this introduces a Linux specific behaviour in an otherwise architectural hypervisor (something I'm quite keen on avoiding)
- It gives an incentive to other hypervisor vendors to add random crap to the Linux mm subsystem, which is even worse. At this stage, we might as well go back to the Xen PV days altogether.
- I haven't seen any of the KVM/arm64 users actually asking for the APF horror, and the cloud vendors I directly asked had no plan to use it, and not using it on their x86 systems either
Quoted 7 lines
- no performance data nor workloads that could help making an informed decision have been disclosed, and the only argument in its favour seems to be "but x86 has it" (hardly a compelling one)
Given the above, I don't see how to justify this series, as it has no purpose on its own, no matter how well written it is.
Thank you for your time to review the series and provide comments. Long time ago, I compare the features supported on x86 and arm64, to sort out the gaps. Async page fault is one of the missed features. From that on, I started to investigate x86's implementation and work on arm64's implementation. It's the history why I continue to work on Async page fault for arm64. It means there is no customer request, asking to support Async page fault on arm64, on my side.
In order to support Async PF on arm64, there are two parts of changes, which are related to kvm/arm64 and guest kernel. The service of Async page fault won't be enabled if either kvm/arm64 or guest kernel doesn't support it. The service is negotiated between host and guest. So I don't think it would be a problem. It's true that Async page fault is only beneficial to Linux host and Linux guest, until it gets supported on other guest kernels.
SDEI implementation is following the specification. It's true that Async PF isn't specified by arm64 architecture. However, it's also not a architectural feature to x86 either. I guess the benefits count here. The reason we need Async PF (and SDEI virtualization) is the benefit.
If I'm correct, Async PF has been used broadly on x86 because of
'post-copy live migration', which relies on userfaultfd. 'Async page fault' is explicitly mentioned in its document (linux/Documentation/admin-guide/mm/userfaultfd.rst) like below. It's the most important motivation to support Async PF.
Yeah, performance data is definitely helpful to measure the benefit, especially for Async page fault on arm64. I used to revise both serieses (SDEI virtualization and Async page fault) together, meaning 'Async page fault' series is revised if there are any code changes to the series of 'SDEI virtualization', until I found it would be practical to finialize 'SDEI virtualization' before working on 'Async page fault'. It's why I don't post revised series of 'Async page fault' recently. However, I think the performance data released in last year's KVM forum is still relative. I certainly need to regain the performance data when I continue to work on 'Async page fault' series after 'SDEI virutalization' is finalized.
https://static.sched.com/hosted_files/kvmforum2021/cb/sdei_apf_for_arm64_gavin.pdf
(In page 14 and 15, 41% to 68% improvement in live post-copy migration)
Extracted from linux/documentation/admin-guide/mm/userfaultfd.rst
------------------------------------------------------------------
QEMU/KVM
========
QEMU/KVM is using the ``userfaultfd`` syscall to implement postcopy live migration. Postcopy live migration is one form of memory externalization consisting of a virtual machine running with part or all of its memory residing on a different node in the cloud. The ``userfaultfd`` abstraction is generic enough that not a single line of KVM kernel code had to be modified in order to add postcopy live migration to QEMU.
Guest async page faults, ``FOLL_NOWAIT`` and all other ``GUP*`` features work just fine in combination with userfaults. Userfaults trigger async page faults in the guest scheduler so those guest processes that aren't waiting for userfaults (i.e. network bound) can keep running in the guest vcpus.
Thanks,
Gavin
Paolo Bonzini
maintainerQuoted 4 lines · Marc Zyngier
- as far as I know, the core Linux/arm64 maintainers have no plan to support APF. Without it, this is a pointless exercise. And even with it, this introduces a Linux specific behaviour in an otherwise architectural hypervisor (something I'm quite keen on avoiding)
Regarding non-architectural behavior, isn't that the same already for PTP? I understand that the PTP hypercall is a much smaller implementation than SDEI+APF, but it goes to show that KVM is already not "architectural".
There are other cases where paravirtualized solutions can be useful. PTP is one but there are more where KVM/ARM does not have a solution yet, for example lock holder preemption. Unless ARM (the company) has a way to receive input from developers and standardize the interface, similar to the RISC-V SIGs, vendor-specific hypercalls are a sad fact of life. It just happened that until now KVM/ARM hasn't seen much use in some cases (such as desktop virtualization) where overcommitted hosts are more common.
Async page faults per se are not KVM specific, in fact Linux supported them for the IBM s390 hypervisor long before KVM added support. They didn't exist on x86 and ARM, so the developers came up with a new hypercall API and for x86 honestly it wasn't great. For ARM we learnt from the mistakes and it seems to me that SDEI is a good match for the feature. If ARM wants to produce a standard interface for APF, whether based on SDEI or something else, we're all ears.
Regarding plans of core arm64 maintainers to support async page fault, can you provide a pointer to the discussion? I agree that if there's a hard NACK for APF for whatever reason, the whole host-side code is pointless (including SDEI virtualization); but I would like to read more about it.
Quoted 3 lines
- It gives an incentive to other hypervisor vendors to add random crap to the Linux mm subsystem, which is even worse. At this stage, we might as well go back to the Xen PV days altogether.
return -EGREGIOUS;
Since you mention hypervisor vendors and there's only one hypervisor in Linux, I guess you're not talking about the host mm/ subsystem (otherwise yeah, FOLL_NOWAIT is only used by KVM async page faults).
So I suppose you're talking about the guest, and then yeah, it sucks to have multiple hypervisors providing the same functionality in different ways (or multiple hypervisors providing different subsets of PV functionality). It happens on x86 with Hyper-V and KVM, and to a lesser extent Xen and VMware.
But again, KVM/ARM has already crossed that bridge with PTP support, and the guest needs exactly zero code in the Linux mm subsystem (both generic and arch-specific) to support asynchronous page faults. There are 20 lines of code in do_notify_resume(), and the rest is just SDEI gunk. Again, I would be happy to get a pointer to concrete objections from the Linux ARM64 maintainers. Maybe a different implementation is possible, I don't know.
In any case it's absolutely not comparable to Xen PV, and you know it.
Quoted 3 lines
- I haven't seen any of the KVM/arm64 users actually asking for the APF horror, and the cloud vendors I directly asked had no plan to use it, and not using it on their x86 systems either
Please define "horror" in more technical terms. And since this is the second time I'm calling you out on this, I'm also asking you to avoid hyperboles and similar rhetorical gimmicks in the future.
That said: Peter, Sean, Google uses or used postcopy extensively on GCE (https://dl.acm.org/doi/pdf/10.1145/3296975.3186415). If it doesn't use it on x86, do you have any insights on why?
Quoted 3 lines
- no performance data nor workloads that could help making an informed decision have been disclosed, and the only argument in its favour seems to be "but x86 has it" (hardly a compelling one)
Again this is just false, numbers have been posted (https://lwn.net/ml/linux-kernel/[email protected]/ was the first result that came up from a quick mailing list search). If they are not enough, please be more specific.
Thanks,
Paolo
Sean Christopherson
reviewerQuoted 3 lines · Paolo Bonzini
That said: Peter, Sean, Google uses or used postcopy extensively on GCE
(https://dl.acm.org/doi/pdf/10.1145/3296975.3186415). If it doesn't use it on x86, do you have any insights on why?
We still use postcopy, but we don't use async #PF. Async #PF is disabled (mostly?) because the x86 implementation was such a mess prior to switching to IRQ-based delivery and AFAIK we haven't re-evaluated it since that update.