Support SDEI Virtualization
Subsystems (6)
v5 cover Support SDEI Virtualization
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")
For the design and migration needs, please refer to the document in PATCH[21/22] in this series. The series is organized as below:
PATCH[01] Introduces template for smccc_get_argx()
PATCH[02] Adds SDEI virtualization infrastructure
PATCH[03-17] Supports various SDEI hypercalls and event handling PATCH[18-20] Adds ioctl commands to support migration and configuration
and exports SDEI capability
PATCH[21] Adds SDEI document
PATCH[22] Adds SDEI selftest case
Testing
=======
[1] The selftest case included in this series works fine. The default SDEI
event, whose number is zero, can be registered, enabled, raised. The
SDEI event handler can be invoked.
[host]# pwd
/home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
[root@virtlab-arm01 kvm]# ./aarch64/sdei
NR_VCPUS: 2 SDEI Event: 0x00000000
--- VERSION
Version: 1.1 (vendor: 0x4b564d)
--- FEATURES
Shared event slots: 0
Private event slots: 0
Relative mode: No
--- PRIVATE_RESET
--- SHARED_RESET
--- PE_UNMASK
--- EVENT_GET_INFO
Type: Private
Priority: Normal
Signaled: Yes
--- EVENT_REGISTER
--- EVENT_ENABLE
--- EVENT_SIGNAL
Handled: Yes
IRQ: No
Status: Registered-Enabled-Running
PC/PSTATE: 000000000040232c 00000000600003c5
Regs: 0000000000000000 0000000000000000
0000000000000000 0000000000000000
--- PE_MASK
--- EVENT_DISABLE
--- EVENT_UNREGISTER
Result: OK[2] There are additional patches in the following repositories to create
procfs entries, allowing to inject SDEI event from host side. The
SDEI client in the guest side registers the SDEI default event, whose
number is zero. Also, the QEMU exports SDEI ACPI table and supports
migration for SDEI.
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")
[2.1] Start the guests and migrate the source VM to the destination
VM.
[host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,cores=3,threads=1 \
-m 1024M,slots=16,maxmem=64G \
: \
-kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x9000000 \
:
[host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
-accel kvm -machine virt,gic-version=host \
-cpu host -smp 6,sockets=2,cores=3,threads=1 \
-m 1024M,slots=16,maxmem=64G \
: \
-kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append earlycon=pl011,mmio,0x9000000 \
-incoming tcp:0:4444 \
:
[2.2] Check kernel log on the source VM. The SDEI service is enabled
and the default SDEI event (0x0) is enabled.
[guest-src]# dmesg | grep -i sdei
ACPI: SDEI 0x000000005BC80000 000024 \
(v00 BOCHS BXPC 00000001 BXPC 00000001)
sdei: SDEIv1.1 (0x4b564d) detected in firmware.
SDEI TEST: Version 1.1, Vendor 0x4b564d
sdei_init: SDEI event (0x0) registered
sdei_init: SDEI event (0x0) enabled
(qemu) migrate -d tcp:localhost:4444
[2.3] Migrate the source VM to the destination VM. Inject SDEI event
to the destination VM. The event is raised and handled.
(qemu) migrate -d tcp:localhost:4444
[host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1
[guest-dst]#
=========== SDEI Event (CPU#1) ===========
Event: 0000000000000000 Parameter: 00000000dabfdabf
PC: ffff800008cbb554 PSTATE: 00000000604000c5 SP: ffff800009c7bde0
Regs: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
ffff800016c28000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff800009399008
ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18
0000000000000000 0000000000000000 ffff000000339d00 0000000000000000
0000000000000000 ffff800009c7bde0 ffff800008cbb5c4
Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190
ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000 Changelog
=========
v5:
* Rebased to v5.17.rc7 (Gavin)
* Unified names for the objects, data structures, variables and functions. The events have been named as exposed, registered and vcpu event. The staes needs to be migrated
is put into kvm_sdei_state.h (Eric)
* More inline functions to visit SDEI event's properties (Eric)
* Support unregistration pending state (Eric)
* Support v1.1 SDEI specification (Eric)
* Fold the code to inject, deliver and handle SDEI event
from PATCH[v4 13/18/19] into PATCH[v5 13] (Eric)
* Simplified ioctl interface to visit all events at once (Eric/Gavin)
* Improved reference count and avoid its migration. Also,
the limit to memory allocation is added based on it. (Eric)
* Change the return values from hypercall functions (Eric)
* Validate @ksdei and @vsdi in kvm_sdei_hypercall() (Shannon)
* Add document to explain how SDEI virutalization and the
migration are supported (Eric)
* Improved selftest case to inject and handle SDEI event (Gavin)
* Improved comments and commit logs (Eric) * Address misc comments from Eric. Hopefully, all of them are covered in v5 because Eric provided lots of comments
in the last round of review (Eric)
v4:
* Rebased to v5.14.rc5 (Gavin)
v3:
* Rebased to v5.13.rc1 (Gavin)
* Use linux data types in kvm_sdei.h (Gavin)
v2:
* Rebased to v5.11.rc6 (Gavin) * Dropped changes related to SDEI client driver (Gavin) * Removed support for passthrou SDEI events (Gavin) * Redesigned data structures (Gavin)
* Implementation is almost rewritten as the data structures
are totally changed (Gavin)
* Added ioctl commands to support migration (Gavin)
Gavin Shan (22):
KVM: arm64: Introduce template for inline functions
KVM: arm64: Add SDEI virtualization infrastructure
KVM: arm64: Support SDEI_VERSION hypercall
KVM: arm64: Support SDEI_EVENT_REGISTER hypercall
KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall
KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall
KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall
KVM: arm64: Support SDEI_EVENT_STATUS hypercall
KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall
KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall
KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall
KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET
KVM: arm64: Support SDEI_FEATURES hypercall
KVM: arm64: Support SDEI event injection, delivery and cancellation
KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
KVM: arm64: Support SDEI_EVENT_{COMPLETE,COMPLETE_AND_RESUME}
hypercall
KVM: arm64: Support SDEI event notifier
KVM: arm64: Support SDEI ioctl commands on VM
KVM: arm64: Support SDEI ioctl commands on vCPU
KVM: arm64: Export SDEI capability
KVM: arm64: Add SDEI document
KVM: selftests: Add SDEI test case
Documentation/virt/kvm/api.rst | 10 + Documentation/virt/kvm/arm/sdei.rst | 325 +++ arch/arm64/include/asm/kvm_emulate.h | 1 + arch/arm64/include/asm/kvm_host.h | 5 + arch/arm64/include/asm/kvm_sdei.h | 187 ++ arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/include/uapi/asm/kvm_sdei_state.h | 101 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 20 + arch/arm64/kvm/hypercalls.c | 21 + arch/arm64/kvm/inject_fault.c | 29 + arch/arm64/kvm/sdei.c | 1900 ++++++++++++++++++ include/kvm/arm_hypercalls.h | 24 +- include/uapi/linux/arm_sdei.h | 2 + include/uapi/linux/kvm.h | 4 + tools/testing/selftests/kvm/Makefile | 1 + tools/testing/selftests/kvm/aarch64/sdei.c | 525 +++++ 17 files changed, 3145 insertions(+), 13 deletions(-) create mode 100644 Documentation/virt/kvm/arm/sdei.rst create mode 100644 arch/arm64/include/asm/kvm_sdei.h create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h create mode 100644 arch/arm64/kvm/sdei.c create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c
Signature · 1 line
2.23.0
PATCH v5 1/22 KVM: arm64: Introduce template for inline functions
The inline functions used to get the SMCCC parameters have same layout. It means these functions can be presented by an unified template, to make the code simplified. Besides, this adds more similar inline functions like smccc_get_arg{4,5,6,7,8}() to get more SMCCC arguments, which are needed by SDEI virtualization support.
--- include/kvm/arm_hypercalls.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
include/kvm/arm_hypercalls.h +12 -12
@@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)return vcpu_get_reg (vcpu,0 ); } -static inline unsigned long smccc_get_arg1 (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,1 ); -} - -static inline unsigned long smccc_get_arg2 (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,2 ); +#define SMCCC_DECLARE_GET_ARG (reg) \ +static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu_get_reg(vcpu, reg); \ } -static inline unsigned long smccc_get_arg3 (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,3 ); -} +SMCCC_DECLARE_GET_ARG(1) +SMCCC_DECLARE_GET_ARG(2) +SMCCC_DECLARE_GET_ARG(3) +SMCCC_DECLARE_GET_ARG(4) +SMCCC_DECLARE_GET_ARG(5) +SMCCC_DECLARE_GET_ARG(6) +SMCCC_DECLARE_GET_ARG(7) +SMCCC_DECLARE_GET_ARG(8) static inline void smccc_set_retval(struct kvm_vcpu *vcpu, unsigned long a0,
Signature · 1 line
2.23.0
Re: [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions
Hi Gavin,
Quoted 36 lines · Gavin Shan
The inline functions used to get the SMCCC parameters have same layout. It means these functions can be presented by an unified template, to make the code simplified. Besides, this adds more similar inline functions like smccc_get_arg{4,5,6,7,8}() to get more SMCCC arguments, which are needed by SDEI virtualization support.
Signed-off-by Gavin Shan <[email protected]>--- include/kvm/arm_hypercalls.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)include/kvm/arm_hypercalls.h +4
-10@@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)return vcpu_get_reg (vcpu,0 ); } -static inline unsigned long smccc_get_arg1 (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,1 ); -} - -static inline unsigned long smccc_get_arg2 (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,2 ); +#define SMCCC_DECLARE_GET_ARG (reg) \ +static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu_get_reg(vcpu, reg); \ } -static inline unsigned long smccc_get_arg3 (struct kvm_vcpu *vcpu) -{
include/kvm/arm_hypercalls.h static inline u32 smccc_get_function(struct kvm_vcpu *vcpu) · line 29
Quoted 10 lines
include/kvm/arm_hypercalls.h +8
-2-return vcpu_get_reg (vcpu,3 ); -} +SMCCC_DECLARE_GET_ARG(1) +SMCCC_DECLARE_GET_ARG(2) +SMCCC_DECLARE_GET_ARG(3) +SMCCC_DECLARE_GET_ARG(4) +SMCCC_DECLARE_GET_ARG(5) +SMCCC_DECLARE_GET_ARG(6) +SMCCC_DECLARE_GET_ARG(7) +SMCCC_DECLARE_GET_ARG (8 )
Hmm. What if we specify a single inline function where the caller passes the arg # as a parameter? We really just want to abstract away the off-by-one difference between GP registers and SMCCC arguments.
Macros generally make me uneasy for template functions, but I may be in the vocal minority on this topic :)
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions
Hi Oliver,
Quoted 52 lines · Oliver Upton
Quoted 36 lines · Gavin Shan
The inline functions used to get the SMCCC parameters have same layout. It means these functions can be presented by an unified template, to make the code simplified. Besides, this adds more similar inline functions like smccc_get_arg{4,5,6,7,8}() to get more SMCCC arguments, which are needed by SDEI virtualization support.
Signed-off-by Gavin Shan <[email protected]>--- include/kvm/arm_hypercalls.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)include/kvm/arm_hypercalls.h +4
-10@@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)return vcpu_get_reg (vcpu,0 ); } -static inline unsigned long smccc_get_arg1 (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,1 ); -} - -static inline unsigned long smccc_get_arg2 (struct kvm_vcpu *vcpu) -{ -return vcpu_get_reg (vcpu,2 ); +#define SMCCC_DECLARE_GET_ARG (reg) \ +static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu_get_reg(vcpu, reg); \ } -static inline unsigned long smccc_get_arg3 (struct kvm_vcpu *vcpu) -{include/kvm/arm_hypercalls.h static inline u32 smccc_get_function(struct kvm_vcpu *vcpu) · line 29
Quoted 10 lines
include/kvm/arm_hypercalls.h +8
-2-return vcpu_get_reg (vcpu,3 ); -} +SMCCC_DECLARE_GET_ARG(1) +SMCCC_DECLARE_GET_ARG(2) +SMCCC_DECLARE_GET_ARG(3) +SMCCC_DECLARE_GET_ARG(4) +SMCCC_DECLARE_GET_ARG(5) +SMCCC_DECLARE_GET_ARG(6) +SMCCC_DECLARE_GET_ARG(7) +SMCCC_DECLARE_GET_ARG (8 )Hmm. What if we specify a single inline function where the caller passes the arg # as a parameter? We really just want to abstract away the off-by-one difference between GP registers and SMCCC arguments.
Quoted 3 lines
Macros generally make me uneasy for template functions, but I may be in the vocal minority on this topic :)
I think it's a good idea to have smccc_get_arg(unsigned char index). However, it will cause more code changes because the following functions have been used. Anyway, I think it's still worthy to pass @index to differentiate the argument index. I will change it accordingly in next respin.
smccc_get_arg1()
smccc_get_arg2()
smccc_get_arg3()
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 2/22 KVM: arm64: Add SDEI virtualization infrastructure
In reply to Gavin ShanSoftware Delegated Exception Interface (SDEI) provides a mechanism for registering and servicing system events. Those system events are high priority events, which must be serviced immediately. It's going to be used by Asynchronous Page Fault (APF) to deliver notification from KVM to guest. It's noted that SDEI is defined by ARM DEN0054C specification.
This introduces SDEI virtualization infrastructure where the SDEI events are registered and manipulated by the guest through hypercall. The SDEI event is delivered to one specific vCPU by KVM once it's raised. This introduces data structures to represent the needed objects to support the feature, which is highlighted as below. As those objects could be migrated between VMs, these data structures are partially exposed to user space.
* kvm_sdei_exposed_event
The exposed events are determined and added by VMM through ioctl interface. Only the exposed events can be registered from the guest.
* kvm_sdei_registered_event
The events that have been registered from the guest through the SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.
* kvm_sdei_vcpu_event
The events that have been delivered to the target vCPU.
* kvm_sdei_vcpu
Used to save the preempted context when the SDEI event is serviced and delivered. After the SDEI event handling is completed, the execution is resumed from the preempted context.
* kvm_sdei_kvm
Place holder for the exposed and registered events.
The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for now. They will be supported in the subsequent patches.
--- arch/arm64/include/asm/kvm_host.h | 3 + arch/arm64/include/asm/kvm_sdei.h | 171 +++++++++++++ arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/include/uapi/asm/kvm_sdei_state.h | 72 ++++++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 8 + arch/arm64/kvm/hypercalls.c | 21 ++ arch/arm64/kvm/sdei.c | 244 +++++++++++++++++++ include/uapi/linux/arm_sdei.h | 2 + 9 files changed, 523 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/kvm_sdei.h create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h create mode 100644 arch/arm64/kvm/sdei.c
arch/arm64/include/asm/kvm_host.h +3 -0
@@ -113,6 +113,8 @@ struct kvm_arch {/* Interrupt controller */ struct vgic_dist vgic; +struct kvm_sdei_kvm *sdei; +/* Mandated version of PSCI */ u32 psci_version;
@@ -338,6 +340,7 @@ struct kvm_vcpu_arch {* Anything that is not used directly from assembly code goes * here. */ +struct kvm_sdei_vcpu *sdei;/* * Guest registers we preserve during guest debugging.
arch/arm64/include/asm/kvm_sdei.h new file +171 -0
171 lines not on this page · raw message
arch/arm64/include/uapi/asm/kvm.h +1 -0
@@ -36,6 +36,7 @@#include <linux/types.h> #include <asm/ptrace.h> #include <asm/sve_context.h> +#include <asm/kvm_sdei_state.h> #define __KVM_HAVE_GUEST_DEBUG#define __KVM_HAVE_IRQ_LINE
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +72 -0
72 lines not on this page · raw message
arch/arm64/kvm/Makefile +1 -1
@@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o pkvm.o \ - arch_timer.o trng.o\ + arch_timer.o trng.o sdei.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/arm.c +8 -0
@@ -38,6 +38,7 @@#include <asm/kvm_asm.h> #include <asm/kvm_mmu.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_sdei.h> #include <asm/sections.h> #include <kvm/arm_hypercalls.h>
@@ -152,6 +153,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)kvm_vgic_early_init (kvm); +kvm_sdei_init_vm (kvm); +/* The maximum number of VCPUs is limited by the host's GIC model */ kvm->arch.max_vcpus =kvm_arm_default_max_vcpus ();
@@ -179,6 +182,8 @@ void kvm_arch_destroy_vm(struct kvm *kvm)kvm_vgic_destroy (kvm); +kvm_sdei_destroy_vm (kvm); +kvm_destroy_vcpus (kvm);kvm_unshare_hyp (kvm, kvm +1 );
@@ -330,6 +335,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)kvm_arm_pvtime_vcpu_init (&vcpu->arch); +kvm_sdei_create_vcpu (vcpu); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; err =kvm_vgic_vcpu_init (vcpu);
@@ -351,6 +358,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)kvm_mmu_free_memory_cache (&vcpu->arch.mmu_page_cache);kvm_timer_vcpu_terminate (vcpu);kvm_pmu_vcpu_destroy (vcpu); +kvm_sdei_destroy_vcpu (vcpu);kvm_arm_vcpu_destroy (vcpu); }
arch/arm64/kvm/hypercalls.c +21 -0
@@ -5,6 +5,7 @@#include <linux/kvm_host.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_sdei.h> #include <kvm/arm_hypercalls.h> #include <kvm/arm_psci.h>
@@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case ARM_SMCCC_TRNG_RND32 :case ARM_SMCCC_TRNG_RND64 :return kvm_trng_call (vcpu); +case SDEI_1_0_FN_SDEI_VERSION : +case SDEI_1_0_FN_SDEI_EVENT_REGISTER : +case SDEI_1_0_FN_SDEI_EVENT_ENABLE : +case SDEI_1_0_FN_SDEI_EVENT_DISABLE : +case SDEI_1_0_FN_SDEI_EVENT_CONTEXT : +case SDEI_1_0_FN_SDEI_EVENT_COMPLETE : +case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME : +case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER : +case SDEI_1_0_FN_SDEI_EVENT_STATUS : +case SDEI_1_0_FN_SDEI_EVENT_GET_INFO : +case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET : +case SDEI_1_0_FN_SDEI_PE_MASK : +case SDEI_1_0_FN_SDEI_PE_UNMASK : +case SDEI_1_0_FN_SDEI_INTERRUPT_BIND : +case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE : +case SDEI_1_1_FN_SDEI_EVENT_SIGNAL : +case SDEI_1_0_FN_SDEI_PRIVATE_RESET : +case SDEI_1_0_FN_SDEI_SHARED_RESET : +case SDEI_1_1_FN_SDEI_FEATURES : +return kvm_sdei_hypercall (vcpu);default :return kvm_psci_call (vcpu); }
arch/arm64/kvm/sdei.c new file +244 -0
244 lines not on this page · raw message
include/uapi/linux/arm_sdei.h +2 -0
@@ -22,8 +22,10 @@#define SDEI_1_0_FN_SDEI_PE_UNMASK SDEI_1_0_FN(0x0C)#define SDEI_1_0_FN_SDEI_INTERRUPT_BIND SDEI_1_0_FN(0x0D)#define SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE SDEI_1_0_FN(0x0E) +#define SDEI_1_1_FN_SDEI_EVENT_SIGNAL SDEI_1_0_FN(0x0F)#define SDEI_1_0_FN_SDEI_PRIVATE_RESET SDEI_1_0_FN(0x11)#define SDEI_1_0_FN_SDEI_SHARED_RESET SDEI_1_0_FN(0x12) +#define SDEI_1_1_FN_SDEI_FEATURES SDEI_1_0_FN(0x30)#define SDEI_VERSION_MAJOR_SHIFT 48#define SDEI_VERSION_MAJOR_MASK 0x7fff
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
Hi Gavin,
Quoted 5 lines · Gavin Shan
Software Delegated Exception Interface (SDEI) provides a mechanism for registering and servicing system events. Those system events are high priority events, which must be serviced immediately. It's going to be used by Asynchronous Page Fault (APF) to deliver notification from KVM to guest. It's noted that SDEI is defined by ARM DEN0054C specification.
I'm guessing that you're using linked lists for stitching all of this together because the specification provides for 24 bits of event encoding. However, it seems that there will be a finite number of events in KVM. So the APF stuff and a software signaled event.
Given that the number of events in KVM is rather small, would it make more sense to do away with the overhead of linked lists and having the state just represented in a static or allocated array? I think you can cram all of the VM scoped event state into a single structure and hang the implementation off of that.
Quoted 20 lines
This introduces SDEI virtualization infrastructure where the SDEI events are registered and manipulated by the guest through hypercall. The SDEI event is delivered to one specific vCPU by KVM once it's raised. This introduces data structures to represent the needed objects to support the feature, which is highlighted as below. As those objects could be migrated between VMs, these data structures are partially exposed to user space.
*
kvm_sdei_exposed_event
The exposed events are determined and added by VMM through ioctl interface. Only the exposed events can be registered from the guest.*
kvm_sdei_registered_event
The events that have been registered from the guest through theSDEI_1_0_FN_SDEI_EVENT_REGISTERhypercall.*
kvm_sdei_vcpu_event
The events that have been delivered to the target vCPU.
Quoted 7 lines
*
kvm_sdei_vcpu
Used to save the preempted context when the SDEI event is serviced and delivered. After the SDEI event handling is completed, the execution is resumed from the preempted context.*
kvm_sdei_kvm
Place holder for the exposed and registered events.
It might be a good idea to expand these a bit and move them into comments on each of the structures.
Quoted 24 lines
The error of
SDEI_NOT_SUPPORTEDis returned for all SDEI hypercalls for now. They will be supported in the subsequent patches.Signed-off-by Gavin Shan <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 3 + arch/arm64/include/asm/kvm_sdei.h | 171 +++++++++++++ arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/include/uapi/asm/kvm_sdei_state.h | 72 ++++++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 8 + arch/arm64/kvm/hypercalls.c | 21 ++ arch/arm64/kvm/sdei.c | 244 +++++++++++++++++++ include/uapi/linux/arm_sdei.h | 2 + 9 files changed, 523 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/kvm_sdei.h create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h create mode 100644 arch/arm64/kvm/sdei.carch/arm64/include/asm/kvm_host.h +0
-0
arch/arm64/include/asm/kvm_host.h struct kvm_arch { · line 117
Quoted 5 lines
arch/arm64/include/asm/kvm_host.h +2
-0@@ -113,6 +113,8 @@ struct kvm_arch {/* Interrupt controller */ struct vgic_dist vgic; +struct kvm_sdei_kvm *sdei; +
nit: avoid repeating 'kvm'. struct kvm_sdei should be descriptive enough :)
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 344
Quoted 9 lines
arch/arm64/include/asm/kvm_host.h +1
-0/* Mandated version of PSCI */ u32 psci_version;@@ -338,6 +340,7 @@ struct kvm_vcpu_arch {* Anything that is not used directly from assembly code goes * here. */ +struct kvm_sdei_vcpu *sdei;
nit: put your scoping tokens at the beginning of a symbol name, so 'struct kvm_vcpu_sdei'.
[...]
Quoted 25 lines
arch/arm64/kvm/hypercalls.c +10
-0@@ -5,6 +5,7 @@#include <linux/kvm_host.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_sdei.h> #include <kvm/arm_hypercalls.h> #include <kvm/arm_psci.h> @@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case ARM_SMCCC_TRNG_RND32 :case ARM_SMCCC_TRNG_RND64 :return kvm_trng_call (vcpu); +case SDEI_1_0_FN_SDEI_VERSION : +case SDEI_1_0_FN_SDEI_EVENT_REGISTER : +case SDEI_1_0_FN_SDEI_EVENT_ENABLE : +case SDEI_1_0_FN_SDEI_EVENT_DISABLE : +case SDEI_1_0_FN_SDEI_EVENT_CONTEXT : +case SDEI_1_0_FN_SDEI_EVENT_COMPLETE : +case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME : +case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER : +case SDEI_1_0_FN_SDEI_EVENT_STATUS :
arch/arm64/kvm/hypercalls.c int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) · line 173
Quoted 10 lines
arch/arm64/kvm/hypercalls.c +10
-0+case SDEI_1_0_FN_SDEI_EVENT_GET_INFO : +case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET : +case SDEI_1_0_FN_SDEI_PE_MASK : +case SDEI_1_0_FN_SDEI_PE_UNMASK : +case SDEI_1_0_FN_SDEI_INTERRUPT_BIND : +case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE : +case SDEI_1_1_FN_SDEI_EVENT_SIGNAL : +case SDEI_1_0_FN_SDEI_PRIVATE_RESET : +case SDEI_1_0_FN_SDEI_SHARED_RESET : +case SDEI_1_1_FN_SDEI_FEATURES :
Consider only adding switch statements for hypercalls when they're actually implemented.
Additionally, while this isn't directly related to your patch, I do have a gripe about kvm_hvc_call_handler(). It is really ugly that we enumerate the specific hypercalls we support, and otherwise fall through to PSCI.
IMO, a cleaner approach would be to have kvm_hvc_call_handler() simply route a particular service range/service owner to the appropriate handler. We can then terminate individual hypercalls in those handlers, avoiding a catch-all switch such as this one is currently.
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
Hi Oliver,
Quoted 12 lines · Oliver Upton
Quoted 5 lines · Gavin Shan
Software Delegated Exception Interface (SDEI) provides a mechanism for registering and servicing system events. Those system events are high priority events, which must be serviced immediately. It's going to be used by Asynchronous Page Fault (APF) to deliver notification from KVM to guest. It's noted that SDEI is defined by ARM DEN0054C specification.
I'm guessing that you're using linked lists for stitching all of this together because the specification provides for 24 bits of event encoding. However, it seems that there will be a finite number of events in KVM. So the APF stuff and a software signaled event.
Quoted 6 lines
Given that the number of events in KVM is rather small, would it make more sense to do away with the overhead of linked lists and having the state just represented in a static or allocated array? I think you can cram all of the VM scoped event state into a single structure and hang the implementation off of that.
Yes, the number of events in KVM is small. Including the events for Async PF and the software signaled event, 8 events would be enough currently. In the meanwhile, there are several types of objects for various events. Some of them can be put into static array, while the left might need static array of pointers to avoid the linked list:
struct kvm_sdei_exposed_event/state on struct kvm_arch
size: 24 bytes
static array, 8 entries
struct kvm_sdei_registered_event/state on struct kvm_arch
size: 9KB
static array of pointers, still need allocate them dynamically, 8 entries
struct kvm_sdei_vcpu_event/state on struct kvm_vcpu_arch
size: 16 bytes
static array, 8 entries
Quoted 25 lines
Quoted 25 lines
This introduces SDEI virtualization infrastructure where the SDEI events are registered and manipulated by the guest through hypercall. The SDEI event is delivered to one specific vCPU by KVM once it's raised. This introduces data structures to represent the needed objects to support the feature, which is highlighted as below. As those objects could be migrated between VMs, these data structures are partially exposed to user space.
*
kvm_sdei_exposed_event
The exposed events are determined and added by VMM through ioctl interface. Only the exposed events can be registered from the guest.*
kvm_sdei_registered_event
The events that have been registered from the guest through theSDEI_1_0_FN_SDEI_EVENT_REGISTERhypercall.*
kvm_sdei_vcpu_event
The events that have been delivered to the target vCPU.*
kvm_sdei_vcpu
Used to save the preempted context when the SDEI event is serviced and delivered. After the SDEI event handling is completed, the execution is resumed from the preempted context.
Quoted 6 lines
Quoted 2 lines
*
kvm_sdei_kvm
Place holder for the exposed and registered events.It might be a good idea to expand these a bit and move them into comments on each of the structures.
Sure, I will do in next respin.
Quoted 27 lines
Quoted 27 lines
The error of
SDEI_NOT_SUPPORTEDis returned for all SDEI hypercalls for now. They will be supported in the subsequent patches.Signed-off-by Gavin Shan <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 3 + arch/arm64/include/asm/kvm_sdei.h | 171 +++++++++++++ arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/include/uapi/asm/kvm_sdei_state.h | 72 ++++++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 8 + arch/arm64/kvm/hypercalls.c | 21 ++ arch/arm64/kvm/sdei.c | 244 +++++++++++++++++++ include/uapi/linux/arm_sdei.h | 2 + 9 files changed, 523 insertions(+), 1 deletion(-)create mode 100644
arch/arm64/include/asm/kvm_sdei.hcreate mode 100644arch/arm64/include/uapi/asm/kvm_sdei_state.hcreate mode 100644arch/arm64/kvm/sdei.carch/arm64/include/asm/kvm_host.h +0
-0@@ -113,6 +113,8 @@ struct kvm_arch {/* Interrupt controller */ struct vgic_dist vgic;
Quoted 6 lines
arch/arm64/include/asm/kvm_host.h struct kvm_arch { · line 117
Quoted 2 lines
arch/arm64/include/asm/kvm_host.h +2
-0+struct kvm_sdei_kvm *sdei; +nit: avoid repeating 'kvm'. struct
kvm_sdeishould be descriptive enough :)
Indeed, "struct kvm_sdei" is better :)
Quoted 13 lines
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 344
Quoted 9 lines
arch/arm64/include/asm/kvm_host.h +1
-0/* Mandated version of PSCI */ u32 psci_version;@@ -338,6 +340,7 @@ struct kvm_vcpu_arch {* Anything that is not used directly from assembly code goes * here. */ +struct kvm_sdei_vcpu *sdei;nit: put your scoping tokens at the beginning of a symbol name, so 'struct
kvm_vcpu_sdei'.
Quoted 2 lines
[...]
Yep, "struct kvm_vcpu_sdei" is the one I will have in next respin :)
Quoted 44 lines
Quoted 25 lines
arch/arm64/kvm/hypercalls.c +10
-0@@ -5,6 +5,7 @@#include <linux/kvm_host.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_sdei.h> #include <kvm/arm_hypercalls.h> #include <kvm/arm_psci.h> @@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)case ARM_SMCCC_TRNG_RND32 :case ARM_SMCCC_TRNG_RND64 :return kvm_trng_call (vcpu); +case SDEI_1_0_FN_SDEI_VERSION : +case SDEI_1_0_FN_SDEI_EVENT_REGISTER : +case SDEI_1_0_FN_SDEI_EVENT_ENABLE : +case SDEI_1_0_FN_SDEI_EVENT_DISABLE : +case SDEI_1_0_FN_SDEI_EVENT_CONTEXT : +case SDEI_1_0_FN_SDEI_EVENT_COMPLETE : +case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME : +case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER : +case SDEI_1_0_FN_SDEI_EVENT_STATUS :arch/arm64/kvm/hypercalls.c int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) · line 173
Quoted 10 lines
arch/arm64/kvm/hypercalls.c +10
-0+case SDEI_1_0_FN_SDEI_EVENT_GET_INFO : +case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET : +case SDEI_1_0_FN_SDEI_PE_MASK : +case SDEI_1_0_FN_SDEI_PE_UNMASK : +case SDEI_1_0_FN_SDEI_INTERRUPT_BIND : +case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE : +case SDEI_1_1_FN_SDEI_EVENT_SIGNAL : +case SDEI_1_0_FN_SDEI_PRIVATE_RESET : +case SDEI_1_0_FN_SDEI_SHARED_RESET : +case SDEI_1_1_FN_SDEI_FEATURES :Consider only adding switch statements for hypercalls when they're actually implemented.
Additionally, while this isn't directly related to your patch, I do have a gripe about
kvm_hvc_call_handler(). It is really ugly that we enumerate the specific hypercalls we support, and otherwise fall through to PSCI.
Quoted 5 lines
IMO, a cleaner approach would be to have
kvm_hvc_call_handler()simply route a particular service range/service owner to the appropriate handler. We can then terminate individual hypercalls in those handlers, avoiding a catch-all switch such as this one is currently.
Yes, I agree. I can have a separate patch as preparatory work to route the range of hypercalls to their owner for further handling. In this way, we can route the range of SDEI hypercalls to its own handler. I will figure it out in next respin.
Thanks,
Gavin
Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
In reply to Gavin ShanHi Gavin,
More comments, didn't see exactly how all of these structures are getting used.
On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:
[...]
Quoted 16 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +10
-0@@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Definitions of various KVM SDEI event states. +* +* Copyright (C) 2022 Red Hat, Inc. +* +* Author(s): Gavin Shan <[email protected]> +*/ + +#ifndef _UAPI__ASM_KVM_SDEI_STATE_H
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 20
Quoted 10 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +10
-0+#define _UAPI__ASM_KVM_SDEI_STATE_H + +#ifndef __ASSEMBLY__ +#include <linux/types.h> + +/* +* The software signaled event is the default one, which is +* defined in v1.1 specification. +*/ +#define KVM_SDEI_INVALID_EVENT 0xFFFFFFFF
Isn't the constraint that bit 31 must be zero? (DEN 0054C 4.4 "Event number allocation")
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 24
Quoted 4 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +4
-0+#define KVM_SDEI_DEFAULT_EVENT 0 + +#define KVM_SDEI_MAX_VCPUS 512/* Aligned to 64 */ +#define KVM_SDEI_MAX_EVENTS 128
I would *strongly* recommend against having these limits. I find the vCPU limit especially concerning, because we're making KVM_MAX_VCPUS ABI, which it definitely is not. Anything that deals with a vCPU should be accessed through a vCPU FD (and thus agnostic to the maximum number of vCPUs) to avoid such a complication.
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 33
Quoted 8 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +8
-0+struct kvm_sdei_exposed_event_state { +__u64 num; + +__u8 type; +__u8 signaled; +__u8 priority; +__u8 padding[5 ]; +__u64 notifier;
Wait, isn't this a kernel function pointer!?
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 36
Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3
-0+}; + +struct kvm_sdei_registered_event_state {
You should fold these fields together with kvm_sdei_exposed_event_state into a single 'kvm_sdei_event' structure:
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 41
Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5
-0+__u64 num; + +__u8 route_mode; +__u8 padding[3 ]; +__u64 route_affinity;
And these shouldn't be UAPI at the VM scope. Each of these properties could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
Quoted 11 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +11
-0+__u64 ep_address[KVM_SDEI_MAX_VCPUS ]; +__u64 ep_arg[KVM_SDEI_MAX_VCPUS ]; +__u64 registered[KVM_SDEI_MAX_VCPUS /64 ]; +__u64 enabled[KVM_SDEI_MAX_VCPUS /64 ]; +__u64 unregister_pending[KVM_SDEI_MAX_VCPUS /64 ]; +}; + +struct kvm_sdei_vcpu_event_state { +__u64 num; + +__u32 event_count;
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 62
Quoted 10 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +10
-0+__u32 padding; +}; + +struct kvm_sdei_vcpu_regs_state { +__u64 regs[18 ]; +__u64 pc; +__u64 pstate; +}; + +struct kvm_sdei_vcpu_state {
Same goes here, I strongly recommend you try to expose this through the KVM_{GET,SET}_ONE_REG interface if at all possible since it significantly reduces the UAPI burden, both on KVM to maintain it and VMMs to actually use it.
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
Hi Oliver,
Quoted 3 lines · Oliver Upton
More comments, didn't see exactly how all of these structures are getting used.
Ok, thanks for your review and comments.
Quoted 25 lines · Gavin Shan
[...]
Quoted 21 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +15
-0@@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Definitions of various KVM SDEI event states. +* +* Copyright (C) 2022 Red Hat, Inc. +* +* Author(s): Gavin Shan <[email protected]> +*/ + +#ifndef _UAPI__ASM_KVM_SDEI_STATE_H +#define _UAPI__ASM_KVM_SDEI_STATE_H + +#ifndef __ASSEMBLY__ +#include <linux/types.h> +
Quoted 9 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 20
Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5
-0+/* +* The software signaled event is the default one, which is +* defined in v1.1 specification. +*/ +#define KVM_SDEI_INVALID_EVENT 0xFFFFFFFFIsn't the constraint that bit 31 must be zero? (DEN 0054C 4.4 "Event number allocation")
Yes, bit 31 of the event number should be zero. So this is invalid event number, used by struct kvm_sdei_vcpu_state::critical_num and normal_num to indicate if there is event being handled on the corresponding vcpu. When those fields are set to KVM_SDEI_INVALID_EVENT, no event is being handled on the vcpu.
Quoted 11 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 24
Quoted 4 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +4
-0+#define KVM_SDEI_DEFAULT_EVENT 0 + +#define KVM_SDEI_MAX_VCPUS 512/* Aligned to 64 */ +#define KVM_SDEI_MAX_EVENTS 128I would *strongly* recommend against having these limits. I find the vCPU limit especially concerning, because we're making
KVM_MAX_VCPUSABI, which it definitely is not. Anything that deals with a vCPU should be accessed through a vCPU FD (and thus agnostic to the maximum number of vCPUs) to avoid such a complication.
For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled event. As you suggested on PATCH[15/22], we can't assume its usage. I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h
For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to kvm_sdei.h after static arrays to hold the data structures or their pointers are used, as you suggested early for this patch (PATCH[02/22]).
There are two types of (SDEI) events: shared and private. For the private event, it can be registered independently from the vcpus. It also means the address and argument for the entry points, corresponding to @ep_address and @ep_arg in struct kvm_sdei_registered_event, can be different on the individual vcpus. However, all the registered/enabled states and the entry point address and argument are same on all vcpus for the shared event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to represent both shared and private event.
If the data belongs to particular vcpu should be accessed through the vcpu fd, then we need to split or reorganize the data struct as below.
/*
* The events are exposed through ioctl interface or similar
* mechanism (synthetic system registers?) before they can be
* registered. struct kvm_sdei_exposed_event instance is reserved
* from the kvm's static array on receiving the ioctl command
* from VMM.
*/
struct kvm_sdei_exposed_event {
__u32 num;
__u8 type;
__u8 signaled;
__u8 priority;
__u8 padding;
};
/*
* The struct kvm_sdei_registered_event instance is allocated or
* reserved from the static array. For the shared event, the instance
* is linked to kvm, but it will be allocated or reserved from vcpu's
* static array and linked to the vcpu if it's a private event.
*
* The instance is only allocated and reserved upon SDEI_EVENT_REGISTER
* hypercall.
*/
struct kvm_sdei_registered_event {
__u32 num#define KVM_SDEI_EVENT_STATE_REGISTERED (1 << 0)
#define KVM_SDEI_EVENT_STATE_ENABLE (1 << 1)
#define KVM_SDEI_EVENT_STATE_UNREGISTER_PENDING (1 << 2)
__u8 state;
__u8 route_mode;
__u8 padding[2];
__u64 route_affinity;
__u64 ep_address;
__u64 ep_arg;
__u64 notifier;
}Quoted 11 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 33
Quoted 8 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +8
-0+struct kvm_sdei_exposed_event_state { +__u64 num; + +__u8 type; +__u8 signaled; +__u8 priority; +__u8 padding[5 ]; +__u64 notifier;Wait, isn't this a kernel function pointer!?
Yeah, it is a kernel function pointer, used by Async PF to know if the corresponding event has been handled or not. Async PF can cancel the previously injected event for performance concerns. Either Async PF or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF, SDEI is responsible for its migration.
Quoted 7 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 36
Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3
-0+}; + +struct kvm_sdei_registered_event_state {You should fold these fields together with
kvm_sdei_exposed_event_stateinto a single 'kvm_sdei_event' structure:
@route_mode and @route_affinity can't be configured or modified until the event is registered. Besides, they're only valid to the shared events. For private events, they don't have the routing needs. It means those two fields would be part of struct kvm_sdei_registered_event instead of kvm_sdei_exposed_event.
Quoted 9 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 41
Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5
-0+__u64 num; + +__u8 route_mode; +__u8 padding[3 ]; +__u64 route_affinity;And these shouldn't be UAPI at the VM scope. Each of these properties could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
They're accessed through vcpu or kvm fd depending on what type the event is. For the VM-owned shared event, they're accessed through KVM fd. For the vcpu-owned private event, they're accessed through vcpu fd.
I'm not sure if I catch the idea to have a synthetic register and I'm to confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED" system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation defined registers can be adopted, I don't think we need to expose anything through ABI. All the operations and the needed data can be passed through the system registers.
SYS_REG_SDEI_COMMAND
Receives commands like to expose event, register event and change vcpu state etc.
SYS_REG_SDEI_DATA
The needed data corresponding to the received command.
However, I'm not positive that synthetic register can be used here. When Mark Rutland review "PATCH[RFC v1] Async PF support", the implementation defined registers can't be used in a very limited way. That time, a set of implementation defined registers are defined to identify the asynchronous page faults and access to the control data block. However, the idea was rejected. Later on, Marc recommended SDEI for Async PF.
https://www.spinics.net/lists/kvm-arm/msg40315.html
Quoted 18 lines
Quoted 18 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +18
-0+__u64 ep_address[KVM_SDEI_MAX_VCPUS ]; +__u64 ep_arg[KVM_SDEI_MAX_VCPUS ]; +__u64 registered[KVM_SDEI_MAX_VCPUS /64 ]; +__u64 enabled[KVM_SDEI_MAX_VCPUS /64 ]; +__u64 unregister_pending[KVM_SDEI_MAX_VCPUS /64 ]; +}; + +struct kvm_sdei_vcpu_event_state { +__u64 num; + +__u32 event_count; +__u32 padding; +}; + +struct kvm_sdei_vcpu_regs_state { +__u64 regs[18 ]; +__u64 pc; +__u64 pstate;
Quoted 9 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 62
Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3
-0+}; + +struct kvm_sdei_vcpu_state {Same goes here, I strongly recommend you try to expose this through the KVM_{GET,SET}_ONE_REG interface if at all possible since it significantly reduces the UAPI burden, both on KVM to maintain it and VMMs to actually use it.
Yeah, it's much convenient to use the implementation defined register here. However, I'm not positive if we can do this. Please see the details I provided above :)
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
Quoted 20 lines · Gavin Shan
Quoted 11 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 24
Quoted 4 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +4
-0+#define KVM_SDEI_DEFAULT_EVENT 0 + +#define KVM_SDEI_MAX_VCPUS 512/* Aligned to 64 */ +#define KVM_SDEI_MAX_EVENTS 128I would *strongly* recommend against having these limits. I find the vCPU limit especially concerning, because we're making
KVM_MAX_VCPUSABI, which it definitely is not. Anything that deals with a vCPU should be accessed through a vCPU FD (and thus agnostic to the maximum number of vCPUs) to avoid such a complication.For
KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled event. As you suggested on PATCH[15/22], we can't assume its usage. I will define it withSDEI_SW_SIGNALED_EVENTinuapi/linux/arm_sdei.hFor
KVM_SDEI_MAX_EVENTS, it will be moved from this header file tokvm_sdei.hafter static arrays to hold the data structures or their pointers are used, as you suggested early for this patch (PATCH[02/22]).
Quoted 8 lines
There are two types of (SDEI) events: shared and private. For the private event, it can be registered independently from the vcpus. It also means the address and argument for the entry points, corresponding to @ep_address and @ep_arg in struct
kvm_sdei_registered_event, can be different on the individual vcpus. However, all the registered/enabled states and the entry point address and argument are same on all vcpus for the shared event.KVM_SDEI_MAX_VCPUSwas introduced to use same data structure to represent both shared and private event.
You're providing a great deal of abstraction around the SDEI specification, but I really am unconvinced that KVM needs that. This series needs to add support for a single SDEI event (event 0) and async PF to follow. Since we are going to support a static set of events under KVM I believe a lot of the complexity in this design should fade away.
Quoted 12 lines
Quoted 11 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 33
Quoted 8 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +8
-0+struct kvm_sdei_exposed_event_state { +__u64 num; + +__u8 type; +__u8 signaled; +__u8 priority; +__u8 padding[5 ]; +__u64 notifier;Wait, isn't this a kernel function pointer!?
Quoted 5 lines
Yeah, it is a kernel function pointer, used by Async PF to know if the corresponding event has been handled or not. Async PF can cancel the previously injected event for performance concerns. Either Async PF or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF, SDEI is responsible for its migration.
But this is a UAPI header, why would we even consider giving userspace a window into the kernel like this? Couldn't userspace crash the kernel by writing whatever it wants to this field, knowing that it will call it as a function pointer?
Security aside, there's no guarantee that a function winds up at the same address between compiler versions or kernel versions.
Overall, I don't think that userspace should have the ability to add arbitrary SDEI events. KVM takes ownership of its own vendor-specific ABI in patch 3/22 by declaring its vendor identifier, so any new events we support must remain within KVM. There is going to be some state that will need to be migrated for KVM's SDEI events, that ought to be surfaced to userspace through the KVM_{GET,SET}_ONE_REG ioctls.
Given that there isn't much userspace involvement to make SDEI work, do you think it would be possible to drop the proposed UAPI from your series and work on enabling software-signaled SDEI events within KVM first? By this I mean a VM running under KVM shouldn't require any ioctls to make it work.
In so doing, we can discover exactly what the mechanics look like in KVM and only then talk about the necessary UAPI to migrate state. One piece of the mechanics that is top of mind which I'd like to see addressed is the use of linked lists and the preallocations that have been made in structures. KVM will know how many events exist at compile time, so we can represent these statically.
Quoted 25 lines
Quoted 7 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 36
Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3
-0+}; + +struct kvm_sdei_registered_event_state {You should fold these fields together with
kvm_sdei_exposed_event_stateinto a single 'kvm_sdei_event' structure:@route_mode and @route_affinity can't be configured or modified until the event is registered. Besides, they're only valid to the shared events. For private events, they don't have the routing needs. It means those two fields would be part of struct
kvm_sdei_registered_eventinstead ofkvm_sdei_exposed_event.Quoted 9 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 41
Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5
-0+__u64 num; + +__u8 route_mode; +__u8 padding[3 ]; +__u64 route_affinity;And these shouldn't be UAPI at the VM scope. Each of these properties could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
Quoted 3 lines
They're accessed through vcpu or kvm fd depending on what type the event is. For the VM-owned shared event, they're accessed through KVM fd. For the vcpu-owned private event, they're accessed through vcpu fd.
Some of the state that you represent in struct kvm_sdei_registered_event_state is really per-vCPU state. Any time that there's data available at a vCPU granularity userspace should access it with a vCPU FD.
Quoted 6 lines
I'm not sure if I catch the idea to have a synthetic register and I'm to confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED" system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation defined registers can be adopted, I don't think we need to expose anything through ABI. All the operations and the needed data can be passed through the system registers.
No, I'm not talking about the guest-facing interface. I'm talking about what gets exposed to userspace to migrate the VM's state. For parts of the guest that do not map to an architectural construct, we've defined the concept of a firmware pseudo-register. What that really means is we expose a register to userspace that is inaccessible from the guest which migrates KVM's nonarchitected state. We are abusing the fact that VMMs will save/restore whatever registers are reported on KVM_GET_REG_LIST to trick it into migrating KVM state, and it has worked quite nicely to avoid adding new ioctls for new features. It also means that older VMMs are capable of utilizing new KVM features, so long as they obey the prescribed rules.
--
Thanks,
Oliver
Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
Hi Oliver,
Quoted 30 lines · Oliver Upton
Quoted 20 lines · Gavin Shan
Quoted 11 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 24
Quoted 4 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +4
-0+#define KVM_SDEI_DEFAULT_EVENT 0 + +#define KVM_SDEI_MAX_VCPUS 512/* Aligned to 64 */ +#define KVM_SDEI_MAX_EVENTS 128I would *strongly* recommend against having these limits. I find the vCPU limit especially concerning, because we're making
KVM_MAX_VCPUSABI, which it definitely is not. Anything that deals with a vCPU should be accessed through a vCPU FD (and thus agnostic to the maximum number of vCPUs) to avoid such a complication.For
KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled event. As you suggested on PATCH[15/22], we can't assume its usage. I will define it withSDEI_SW_SIGNALED_EVENTinuapi/linux/arm_sdei.hFor
KVM_SDEI_MAX_EVENTS, it will be moved from this header file tokvm_sdei.hafter static arrays to hold the data structures or their pointers are used, as you suggested early for this patch (PATCH[02/22]).Quoted 8 lines
There are two types of (SDEI) events: shared and private. For the private event, it can be registered independently from the vcpus. It also means the address and argument for the entry points, corresponding to @ep_address and @ep_arg in struct
kvm_sdei_registered_event, can be different on the individual vcpus. However, all the registered/enabled states and the entry point address and argument are same on all vcpus for the shared event.KVM_SDEI_MAX_VCPUSwas introduced to use same data structure to represent both shared and private event.
Quoted 6 lines
You're providing a great deal of abstraction around the SDEI specification, but I really am unconvinced that KVM needs that. This series needs to add support for a single SDEI event (event 0) and async PF to follow. Since we are going to support a static set of events under KVM I believe a lot of the complexity in this design should fade away.
Yeah, I think we can drop the functionality to support the shared event since both events are exposed by KVM as private events. Please refer below reply for more details.
Quoted 39 lines
Quoted 12 lines
Quoted 11 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 33
Quoted 8 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +8
-0+struct kvm_sdei_exposed_event_state { +__u64 num; + +__u8 type; +__u8 signaled; +__u8 priority; +__u8 padding[5 ]; +__u64 notifier;Wait, isn't this a kernel function pointer!?
Quoted 5 lines
Yeah, it is a kernel function pointer, used by Async PF to know if the corresponding event has been handled or not. Async PF can cancel the previously injected event for performance concerns. Either Async PF or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF, SDEI is responsible for its migration.
But this is a UAPI header, why would we even consider giving userspace a window into the kernel like this? Couldn't userspace crash the kernel by writing whatever it wants to this field, knowing that it will call it as a function pointer?
Security aside, there's no guarantee that a function winds up at the same address between compiler versions or kernel versions.
Overall, I don't think that userspace should have the ability to add arbitrary SDEI events. KVM takes ownership of its own vendor-specific ABI in patch 3/22 by declaring its vendor identifier, so any new events we support must remain within KVM. There is going to be some state that will need to be migrated for KVM's SDEI events, that ought to be surfaced to userspace through the KVM_{GET,SET}_ONE_REG ioctls.
Given that there isn't much userspace involvement to make SDEI work, do you think it would be possible to drop the proposed UAPI from your series and work on enabling software-signaled SDEI events within KVM first? By this I mean a VM running under KVM shouldn't require any ioctls to make it work.
Quoted 7 lines
In so doing, we can discover exactly what the mechanics look like in KVM and only then talk about the necessary UAPI to migrate state. One piece of the mechanics that is top of mind which I'd like to see addressed is the use of linked lists and the preallocations that have been made in structures. KVM will know how many events exist at compile time, so we can represent these statically.
For @notifier, it is set on the kernel of source VM and migrated to the kernel of destination VM. It's really concerned with security, I will drop this in next respin.
I plan to make the following changes for next revision (v6) as below. Please let me know if there are more things to be covered:
(1) Drop the code to support the shared (SDEI) events since the needed events, either software signaled or Async PF event, are all private. After that, the private event is the only type to be supported. When the shared event becomes unsupported, we can simply return SDEI_NOT_SUPPORTED for hypercall SDEI_EVENT_ROUTING_SET and SDEI_SHARED_RESET.
(2) Drop the code to support migration for now. It means the data structures defined in arch/arm64/include/uapi/asm/kvm_sdei_state.h will be combined to arch/arm64/include/asm/kvm_sdei.h. Besides, all the ioctl commands, including KVM_CAP_ARM_SDEI will be dropped either.
(3) As we discussed before, the linked lists will be replaced by the
static arrays or preallocation. The data structures will be amended
like below.
(3.1) In arch/arm64/include/asm/kvm_sdei.h
struct kvm_sdei_exposed_event {
unsigned int num;
unsigned char type; /* reserved, should be 'private' */
unsigned char signaled;
unsigned char priority;
};
struct kvm_sdei_registered_event {
struct kvm_sdei_exposed_event *exposed_event;
unsigned long ep_address;
unsigned long ep_arg;
#define KVM_SDEI_EVENT_STATE_REGISTERED (1UL << 0)
#define KVM_SDEI_EVENT_STATE_ENABLED (1UL << 1)
#define KVM_SDEI_EVENT_STATE_UNREGISTER_PENDING (1UL << 2)
unsigned long state;
unsigned long event_count; /* number of events pending for hanlding */
};
struct kvm_sdei_context {
struct kvm_sdei_registered_event *registered_event;
unsigned long regs[18];
unsigned long pc;
unsigned long pstate;
};
/* We even needn't a lock */
struct kvm_sdei_vcpu {
unsigned char masked;
struct kvm_sdei_registered_event *registered_event;
struct kvm_sdei_context context[SDEI_EVENT_PRIORITY_CRITICAL + 1];
};
/* struct kvm_sdei won't be existing any more */
(3.2) In arch/arm64/kvm/sdei.c
static struct kvm_sdei_exposed_event exposed_events[] = {
{ .num = SDEI_EVENT_SW_SIGNALED, /* defined in include/uapi/linux/arm_sdei.h */
.type = SDEI_EVENT_TYPE_PRIVATE,
.signaled = 1,
.priority = SDEI_EVENT_PRIORITY_NORMAL,
},
{
.num = SDEI_EVENT_ASYNC_PF, /* defined in include/asm/kvm_sdei.h */
.type = SDEI_EVENT_TYPE_PRIVATE,
.signaled = 1,
.priority = SDEI_EVENT_PRIORITY_CRITICAL,
},
};
In kvm_sdei_create_vcpu(), the events are pre-allocated. They will be destroy
in kvm_sdei_destroy_vcpu().
struct kvm_vcpu_arch::sdei =
kzalloc(sizeof(struct kvm_sdei_vcpu), GFP_KERNEL_ACCOUNT);
struct kvm_sdei_vcpu::registered_event =
kcalloc(sizeof(struct kvm_sdei_exposed_event), ARRAY_SIZE(exposed_events), GFP_KERNEL_ACCOUNT);
In kvm_sdei_hypercall(), SDEI_NOT_SUPPORTED will be returned on hypercall
SDEI_EVENT_ROUTING_SET and SDEI_SHARED_RESET.(4) Stuff to be considered or discussed in future: migration and mechanism. We probably don't want to support the shared event. The function of a event is to notify from host to guest kernel. In this regard, the shared event can be replaced by the private one. With above changes in (1)/(2)/(3), we don't have any VM-scoped properties or states. It means all the properties and states are associated with VCPU. It will help to adopt 'firmware-pseudo' registers and KVM_{GET,SET}_ONE_REG during the migration in the future.
Quoted 25 lines
Quoted 25 lines
Quoted 7 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 36
Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3
-0+}; + +struct kvm_sdei_registered_event_state {You should fold these fields together with
kvm_sdei_exposed_event_stateinto a single 'kvm_sdei_event' structure:@route_mode and @route_affinity can't be configured or modified until the event is registered. Besides, they're only valid to the shared events. For private events, they don't have the routing needs. It means those two fields would be part of struct
kvm_sdei_registered_eventinstead ofkvm_sdei_exposed_event.Quoted 9 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h line 41
Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5
-0+__u64 num; + +__u8 route_mode; +__u8 padding[3 ]; +__u64 route_affinity;And these shouldn't be UAPI at the VM scope. Each of these properties could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
Quoted 8 lines
Quoted 3 lines
They're accessed through vcpu or kvm fd depending on what type the event is. For the VM-owned shared event, they're accessed through KVM fd. For the vcpu-owned private event, they're accessed through vcpu fd.
Some of the state that you represent in struct
kvm_sdei_registered_event_stateis really per-vCPU state. Any time that there's data available at a vCPU granularity userspace should access it with a vCPU FD.
Yeah, I used "struct kvm_sdei_registered_event" to represent the shared and private event. For the shared event, they are VM scoped. However, they're VCPU scoped for the private event. As I suggested above, all the states are VCPU scoped when the private event becomes the only supported one.
Quoted 7 lines
Quoted 6 lines
I'm not sure if I catch the idea to have a synthetic register and I'm to confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED" system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation defined registers can be adopted, I don't think we need to expose anything through ABI. All the operations and the needed data can be passed through the system registers.
Quoted 12 lines
No, I'm not talking about the guest-facing interface. I'm talking about what gets exposed to userspace to migrate the VM's state. For parts of the guest that do not map to an architectural construct, we've defined the concept of a firmware pseudo-register. What that really means is we expose a register to userspace that is inaccessible from the guest which migrates KVM's nonarchitected state. We are abusing the fact that VMMs will save/restore whatever registers are reported on
KVM_GET_REG_LISTto trick it into migrating KVM state, and it has worked quite nicely to avoid adding new ioctls for new features. It also means that older VMMs are capable of utilizing new KVM features, so long as they obey the prescribed rules.
Thanks for the details about the 'firmware pseudo register'. Oliver, is there exiting one in current KVM implementation? I would like to see how it's being used. It's definitely a good idea. Those non-architectural CPU properties can be mapped and migrated in a natural way. I'm not sure if we had similar mechanism or 'firmware pseudo registers' for the VM scoped properties?
Thanks,
Gavin
PATCH v5 3/22 KVM: arm64: Support SDEI_VERSION hypercall
In reply to Gavin ShanThis supports SDEI_VERSION hypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.
--- arch/arm64/kvm/sdei.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/arm64/kvm/sdei.c +10 -0
@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,return pending; } +static unsigned long hypercall_version (struct kvm_vcpu *vcpu) +{ +/* v1.1 and the vendor is KVM */ +return (1UL <<SDEI_VERSION_MAJOR_SHIFT ) | + (1UL <<SDEI_VERSION_MINOR_SHIFT ) | +0x4b564d ; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -142,6 +150,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)switch (func) {case SDEI_1_0_FN_SDEI_VERSION : + ret =hypercall_version (vcpu); +break ;case SDEI_1_0_FN_SDEI_EVENT_REGISTER :case SDEI_1_0_FN_SDEI_EVENT_ENABLE :case SDEI_1_0_FN_SDEI_EVENT_DISABLE :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
Quoted 14 lines · Gavin Shan
This supports
SDEI_VERSIONhypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.Signed-off-by Gavin Shan <[email protected]>--- arch/arm64/kvm/sdei.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/arm64/kvm/sdei.c +0
-0
arch/arm64/kvm/sdei.c static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu, · line 126
Quoted 9 lines
arch/arm64/kvm/sdei.c +6
-0@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,return pending; } +static unsigned long hypercall_version (struct kvm_vcpu *vcpu) +{ +/* v1.1 and the vendor is KVM */ +return (1UL <<SDEI_VERSION_MAJOR_SHIFT ) | + (1UL <<SDEI_VERSION_MINOR_SHIFT ) | +0x4b564d ;
It looks like the SDEI specification states that the vendor-defined version number is 32 bits. Could we just use one of the ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
ASCII 'KVM' is neat, but in reality guest software will just throw it in a macro regardless. Might as well use one of the values we've already trained it to use :-)
Also, it would appear that guest discovery of SDEI relies upon KVM reporting a valid SDEI version. IMO, this patch should come at the very end when KVM actually implements SDEI.
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
Hi Oliver,
Quoted 29 lines · Oliver Upton
Quoted 14 lines · Gavin Shan
This supports
SDEI_VERSIONhypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.Signed-off-by Gavin Shan <[email protected]>--- arch/arm64/kvm/sdei.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/arm64/kvm/sdei.c +0
-0arch/arm64/kvm/sdei.c static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu, · line 126
Quoted 9 lines
arch/arm64/kvm/sdei.c +6
-0@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,return pending; } +static unsigned long hypercall_version (struct kvm_vcpu *vcpu) +{ +/* v1.1 and the vendor is KVM */ +return (1UL <<SDEI_VERSION_MAJOR_SHIFT ) | + (1UL <<SDEI_VERSION_MINOR_SHIFT ) | +0x4b564d ;It looks like the SDEI specification states that the vendor-defined version number is 32 bits. Could we just use one of the ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
Quoted 8 lines
ASCII 'KVM' is neat, but in reality guest software will just throw it in a macro regardless. Might as well use one of the values we've already trained it to use :-)
Also, it would appear that guest discovery of SDEI relies upon KVM reporting a valid SDEI version. IMO, this patch should come at the very end when KVM actually implements SDEI.
Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 if you agree. Its first two characters are "VM" at least.
It's fine to return the version here because the SDEI capability isn't exposed yet. It means no events have been exposed and able to be registered. However, It's also fine to move this patch after the following one:
[PATCH v5 16/22] KVM: arm64: Support SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall
Thanks,
Gavin
Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
Quoted 35 lines · Gavin Shan
Hi Oliver,
Quoted 32 lines · Oliver Upton
Quoted 14 lines · Gavin Shan
This supports
SDEI_VERSIONhypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.Signed-off-by Gavin Shan <[email protected]>--- arch/arm64/kvm/sdei.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/arm64/kvm/sdei.c +0
-0arch/arm64/kvm/sdei.c static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu, · line 125
Quoted 8 lines
arch/arm64/kvm/sdei.c +6
-0@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,return pending; } +static unsigned long hypercall_version (struct kvm_vcpu *vcpu) +{ +/* v1.1 and the vendor is KVM */ +return (1UL <<SDEI_VERSION_MAJOR_SHIFT ) | + (1UL <<SDEI_VERSION_MINOR_SHIFT ) | +0x4b564d ;It looks like the SDEI specification states that the vendor-defined version number is 32 bits. Could we just use one of the ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
ASCII 'KVM' is neat, but in reality guest software will just throw it in a macro regardless. Might as well use one of the values we've already trained it to use :-)
Quoted 8 lines
Quoted 4 lines
Also, it would appear that guest discovery of SDEI relies upon KVM reporting a valid SDEI version. IMO, this patch should come at the very end when KVM actually implements SDEI.
Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use
ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2if you agree. Its first two characters are "VM" at least.
Sounds fine to me. The only other nit I'd say is we should define a macro for it too, something like:
#define KVM_SDEI_VENDOR ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
Hi Oliver,
Quoted 39 lines · Oliver Upton
Quoted 38 lines · Gavin Shan
Quoted 36 lines · Oliver Upton
Quoted 14 lines · Gavin Shan
This supports
SDEI_VERSIONhypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.Signed-off-by Gavin Shan <[email protected]>--- arch/arm64/kvm/sdei.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/arm64/kvm/sdei.c +0
-0arch/arm64/kvm/sdei.c static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu, · line 125
Quoted 8 lines
arch/arm64/kvm/sdei.c +6
-0@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,return pending; } +static unsigned long hypercall_version (struct kvm_vcpu *vcpu) +{ +/* v1.1 and the vendor is KVM */ +return (1UL <<SDEI_VERSION_MAJOR_SHIFT ) | + (1UL <<SDEI_VERSION_MINOR_SHIFT ) | +0x4b564d ;It looks like the SDEI specification states that the vendor-defined version number is 32 bits. Could we just use one of the ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
ASCII 'KVM' is neat, but in reality guest software will just throw it in a macro regardless. Might as well use one of the values we've already trained it to use :-)
Also, it would appear that guest discovery of SDEI relies upon KVM reporting a valid SDEI version. IMO, this patch should come at the very end when KVM actually implements SDEI.
Quoted 9 lines
Quoted 3 lines
Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use
ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2if you agree. Its first two characters are "VM" at least.Sounds fine to me. The only other nit I'd say is we should define a macro for it too, something like:
#define KVM_SDEI_VENDOR ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h. arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by ABI.
Thanks,
Gavin
Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
On Thu, Mar 24, 2022 at 12:07:34PM +0800, Gavin Shan wrote:
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use
ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2if you agree. Its first two characters are "VM" at least.Sounds fine to me. The only other nit I'd say is we should define a macro for it too, something like:
Quoted 7 lines
Quoted 2 lines
#define KVM_SDEI_VENDOR ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2Agreed, and the macro will be put into
arch/arm64/include/asm/kvm_sdei.h.arch/arm64/include/uapi/asm/kvm_sdei_state.hisn't the right place because the dependent macroARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2isn't exposed by ABI.
The argument could definitely be made that our vendor ID should be promoted to UAPI. Even though linux is the only known user of our vendor-specific hypercalls, nothing is stopping other software from using them. Beyond that, these values should really never change anyway.
It isn't a big deal if you add it to internal headers, either, as the only known consumer will be the kernel.
--
Thanks,
Oliver
Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
Hi Oliver,
Quoted 19 lines · Oliver Upton
On Thu, Mar 24, 2022 at 12:07:34PM +0800, Gavin Shan wrote:
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use
ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2if you agree. Its first two characters are "VM" at least.Sounds fine to me. The only other nit I'd say is we should define a macro for it too, something like:
Quoted 7 lines
Quoted 2 lines
#define KVM_SDEI_VENDOR ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2Agreed, and the macro will be put into
arch/arm64/include/asm/kvm_sdei.h.arch/arm64/include/uapi/asm/kvm_sdei_state.hisn't the right place because the dependent macroARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2isn't exposed by ABI.
Quoted 8 lines
The argument could definitely be made that our vendor ID should be promoted to UAPI. Even though linux is the only known user of our vendor-specific hypercalls, nothing is stopping other software from using them. Beyond that, these values should really never change anyway.
It isn't a big deal if you add it to internal headers, either, as the only known consumer will be the kernel.
Sure. Lets add it into include/asm/kvm_sdei.h in next respin. We can expose it when it is needed. For now, I do think Linux is the only user.
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 4/22 KVM: arm64: Support SDEI_EVENT_REGISTER hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_REGISTER hypercall, which is used by guest to register SDEI events. The SDEI event won't be raised until it's registered and enabled explicitly.
Only the exposed events can be registered. For shared event, the registered event instance is created. However, the instance may be not created for the private events.
--- arch/arm64/kvm/sdei.c | 128 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+)
arch/arm64/kvm/sdei.c +128 -0
@@ -14,6 +14,35 @@#include <kvm/arm_hypercalls.h> #include <asm/kvm_sdei.h> +static struct kvm_sdei_exposed_event * +find_exposed_event (struct kvm *kvm,unsigned long num) +{ +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; + +list_for_each_entry (exposed_event, &ksdei->exposed_events, link) { +if (exposed_event->state.num == num) +return exposed_event; + } + +return NULL ; +} + +static struct kvm_sdei_registered_event * +find_registered_event (struct kvm *kvm,unsigned long num) +{ +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_registered_event *registered_event; + +list_for_each_entry (registered_event, + &ksdei->registered_events, link) { +if (registered_event->state.num == num) +return registered_event; + } + +return NULL ; +} +static void remove_all_exposed_events (struct kvm *kvm) {struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
@@ -126,6 +155,103 @@ static unsigned long hypercall_version(struct kvm_vcpu *vcpu)0x4b564d ; } +static unsigned long hypercall_register (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event; +unsigned long event_num =smccc_get_arg1 (vcpu); +unsigned long event_ep_address =smccc_get_arg2 (vcpu); +unsigned long event_ep_arg =smccc_get_arg3 (vcpu); +unsigned long route_mode =smccc_get_arg4 (vcpu); +unsigned long route_affinity =smccc_get_arg5 (vcpu); +int index; +unsigned long ret =SDEI_SUCCESS ; + +if (!kvm_sdei_is_supported (event_num)) { + ret =SDEI_INVALID_PARAMETERS ; + goto out; + } + +if (route_mode !=SDEI_EVENT_REGISTER_RM_ANY && + route_mode !=SDEI_EVENT_REGISTER_RM_PE ) { + ret =SDEI_INVALID_PARAMETERS ; + goto out; + } + +spin_lock (&ksdei->lock); + +/* +* The registered event could have been existing if it's a private +* one. We needn't to create another registered event instance +* in this case. +*/ + registered_event =find_registered_event (kvm, event_num); +if (registered_event) { + exposed_event = registered_event->exposed_event; + index =kvm_sdei_vcpu_index (vcpu, exposed_event); +if (kvm_sdei_is_registered (registered_event, index) || +kvm_sdei_is_unregister_pending (registered_event, index)) { + ret =SDEI_DENIED ; + goto unlock; + } + + registered_event->state.route_mode = route_mode; + registered_event->state.route_affinity = route_affinity; + registered_event->state.ep_address[index] = event_ep_address; + registered_event->state.ep_arg[index] = event_ep_arg; +kvm_sdei_set_registered (registered_event, index); + goto unlock; + } + +/* Check if the exposed event exists */ + exposed_event =find_exposed_event (kvm, event_num); +if (!exposed_event) { + ret =SDEI_INVALID_PARAMETERS ; + goto unlock; + } + +/* +* Check if the count of registered event instances exceeds +* the limit. +*/ +if (ksdei->registered_event_count >=KVM_SDEI_MAX_EVENTS ) { + ret =SDEI_OUT_OF_RESOURCE ; + goto unlock; + } + +/* Allocate the registered event instance */ + registered_event =kzalloc (sizeof (*registered_event), +GFP_KERNEL_ACCOUNT ); +if (!registered_event) { + ret =SDEI_OUT_OF_RESOURCE ; + goto unlock; + } + +/* Initialize the registered event state */ + index =kvm_sdei_vcpu_index (vcpu, exposed_event); + registered_event->state.num = event_num; + registered_event->state.route_mode = route_affinity; + registered_event->state.route_affinity = route_affinity; + registered_event->state.ep_address[index] = event_ep_address; + registered_event->state.ep_arg[index] = event_ep_arg; + registered_event->kvm = kvm; + registered_event->exposed_event = exposed_event; + registered_event->vcpu_event_count =0 ; +kvm_sdei_set_registered (registered_event, index); + +/* Add the registered event instance */ + ksdei->registered_event_count++; + exposed_event->registered_event_count++; +list_add_tail (®istered_event->link, &ksdei->registered_events); + +unlock: +spin_unlock (&ksdei->lock); +out: +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -153,6 +279,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu) ret =hypercall_version (vcpu);break ;case SDEI_1_0_FN_SDEI_EVENT_REGISTER : + ret =hypercall_register (vcpu); +break ;case SDEI_1_0_FN_SDEI_EVENT_ENABLE :case SDEI_1_0_FN_SDEI_EVENT_DISABLE :case SDEI_1_0_FN_SDEI_EVENT_CONTEXT :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 5/22 KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_{ENABLE, DISABLE} hypercall. After SDEI event is registered by guest, it won't be delivered to the guest until it's enabled. For unregistration pending event, we can't enable or disable it as the registered event is going to be destroyed after current event is handled.
--- arch/arm64/kvm/sdei.c | 49 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
arch/arm64/kvm/sdei.c +49 -0
@@ -252,6 +252,51 @@ static unsigned long hypercall_register(struct kvm_vcpu *vcpu)return ret; } +static unsigned long hypercall_enable (struct kvm_vcpu *vcpu,bool enable) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event; +unsigned long event_num =smccc_get_arg1 (vcpu); +int index; +unsigned long ret =SDEI_SUCCESS ; + +if (!kvm_sdei_is_supported (event_num)) { + ret =SDEI_INVALID_PARAMETERS ; + goto out; + } + +spin_lock (&ksdei->lock); + +/* Check if the registered event exists */ + registered_event =find_registered_event (kvm, event_num); +if (!registered_event) { + ret =SDEI_DENIED ; + goto unlock; + } + +/* Check if the event is registered and pending for unregistration */ + exposed_event = registered_event->exposed_event; + index =kvm_sdei_vcpu_index (vcpu, exposed_event); +if (!kvm_sdei_is_registered (registered_event, index) || +kvm_sdei_is_unregister_pending (registered_event, index)) { + ret =SDEI_DENIED ; + goto unlock; + } + +/* Update the enablement state */ +if (enable) +kvm_sdei_set_enabled (registered_event, index); +else +kvm_sdei_clear_enabled (registered_event, index); + +unlock: +spin_unlock (&ksdei->lock); +out: +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -282,7 +327,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu) ret =hypercall_register (vcpu);break ;case SDEI_1_0_FN_SDEI_EVENT_ENABLE : + ret =hypercall_enable (vcpu, true); +break ;case SDEI_1_0_FN_SDEI_EVENT_DISABLE : + ret =hypercall_enable (vcpu, false); +break ;case SDEI_1_0_FN_SDEI_EVENT_CONTEXT :case SDEI_1_0_FN_SDEI_EVENT_COMPLETE :case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 6/22 KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_CONTEXT hypercall. It's used by the guest to retrieve the registers (R0 - R17) in the preempted context in the SDEI event handler. The preempted context is saved prior to servicing or handling the SDEI event and restored after that.
--- arch/arm64/kvm/sdei.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
arch/arm64/kvm/sdei.c +30 -0
@@ -297,6 +297,34 @@ static unsigned long hypercall_enable(struct kvm_vcpu *vcpu, bool enable)return ret; } +static unsigned long hypercall_context (struct kvm_vcpu *vcpu) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +struct kvm_sdei_vcpu_regs_state *regs; +unsigned long param_id =smccc_get_arg1 (vcpu); +unsigned long ret =SDEI_SUCCESS ; + +spin_lock (&vsdei->lock); + +/* Check if the pending event exists */ +if (!vsdei->critical_event && !vsdei->normal_event) { + ret =SDEI_DENIED ; + goto unlock; + } + +/* Fetch the requested register */ + regs = vsdei->critical_event ? &vsdei->state.critical_regs : + &vsdei->state.normal_regs; +if (param_id <ARRAY_SIZE (regs->regs)) + ret = regs->regs[param_id]; +else + ret =SDEI_INVALID_PARAMETERS ; + +unlock: +spin_unlock (&vsdei->lock); +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -333,6 +361,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu) ret =hypercall_enable (vcpu, false);break ;case SDEI_1_0_FN_SDEI_EVENT_CONTEXT : + ret =hypercall_context (vcpu); +break ;case SDEI_1_0_FN_SDEI_EVENT_COMPLETE :case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME :case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 7/22 KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_UNREGISTER hypercall. It's used by the guest to unregister SDEI event. The SDEI event won't be raised to the guest after it's unregistered. The SDEI event is disabled automatically on unregistration.
The currently handled events can't be unregistered. We set the unregistration pending state for the event so that it can be unregistered when the event handler is completed by receiving SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall.
--- arch/arm64/kvm/sdei.c | 133 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+)
arch/arm64/kvm/sdei.c +133 -0
@@ -325,6 +325,135 @@ static unsigned long hypercall_context(struct kvm_vcpu *vcpu)return ret; } +static unsigned long +unregister_one_event (struct kvm *kvm,struct kvm_vcpu *vcpu, +struct kvm_sdei_registered_event *registered_event) +{ +struct kvm_vcpu *vcpup; +struct kvm_sdei_vcpu *vsdei; +struct kvm_sdei_exposed_event *exposed_event; +int index; +bool pending; +unsigned long i, ret =SDEI_SUCCESS ; + +/* +* Cancel all vcpu events that have been queued, but not dispatched +* yet. If the vcpu event has been dispatched, we should mark it +* as pending for unregistration. The unregistration will be executed +* when the event handler is to be completed. +*/ + exposed_event = registered_event->exposed_event; +kvm_for_each_vcpu (i, vcpup, kvm) { + vsdei = vcpup->arch.sdei; +if (!vsdei) +continue ; + +/* +* The private vcpu events are requested to be unregistered +* on the specific vcpu or all vcpus. @vcpu is used to +* identify the cases. For the shared vcpu events, we need +* to unregister them on all vcpus. +*/ +if (kvm_sdei_is_private (exposed_event->state.type) && + vcpu && vcpu != vcpup) +continue ; + +if (registered_event->vcpu_event_count >0 ) { +spin_lock (&vsdei->lock); + pending =remove_all_vcpu_events (vcpup, + registered_event->state.num); +spin_unlock (&vsdei->lock); + }else { + pending = false; + } + +/* +* For the private event, the pending state for unregistration +* is scattered and we need to update them individually. +* However, that same state for the shared event has to be +* updated at once according to @ret after the iteration is +* done. +*/ + ret = pending ?SDEI_PENDING : ret; +if (!kvm_sdei_is_private (exposed_event->state.type)) +continue ; + + index =kvm_sdei_vcpu_index (vcpup, exposed_event); +if (pending) { +kvm_sdei_set_unregister_pending (registered_event, + index); + }else { +kvm_sdei_clear_enabled (registered_event, index); +kvm_sdei_clear_registered (registered_event, index); + } + } + +/* +* Update the pending state for unregistration for the shared event +* at once. +*/ +if (kvm_sdei_is_shared (exposed_event->state.type)) { + index =kvm_sdei_vcpu_index (vcpu, exposed_event); +if (ret ==SDEI_PENDING ) { +kvm_sdei_set_unregister_pending (registered_event, + index); + }else { +kvm_sdei_clear_enabled (registered_event, index); +kvm_sdei_clear_registered (registered_event, index); + } + } + +/* Destroy the registered event instance if needed */ +if (kvm_sdei_none_registered (registered_event)) +remove_one_registered_event (kvm, registered_event); + +return ret; +} + +static unsigned long hypercall_unregister (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event; +unsigned long event_num =smccc_get_arg1 (vcpu); +int index; +unsigned long ret =SDEI_SUCCESS ; + +if (!kvm_sdei_is_supported (event_num)) +return SDEI_INVALID_PARAMETERS ; + +spin_lock (&ksdei->lock); + +/* Check if the registered event exists */ + registered_event =find_registered_event (kvm, event_num); +if (!registered_event) { + ret =SDEI_INVALID_PARAMETERS ; + goto unlock; + } + +/* Check if the event has been registered */ + exposed_event = registered_event->exposed_event; + index =kvm_sdei_vcpu_index (vcpu, exposed_event); +if (!kvm_sdei_is_registered (registered_event, index)) { + ret =SDEI_DENIED ; + goto unlock; + } + +/* Check if the event has been pending for unregistration */ +if (kvm_sdei_is_unregister_pending (registered_event, index)) { + ret =SDEI_PENDING ; + goto unlock; + } + + ret =unregister_one_event (kvm, vcpu, registered_event); + +unlock: +spin_unlock (&ksdei->lock); + +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -365,7 +494,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)break ;case SDEI_1_0_FN_SDEI_EVENT_COMPLETE :case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME : + ret =SDEI_NOT_SUPPORTED ; +break ;case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER : + ret =hypercall_unregister (vcpu); +break ;case SDEI_1_0_FN_SDEI_EVENT_STATUS :case SDEI_1_0_FN_SDEI_EVENT_GET_INFO :case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 8/22 KVM: arm64: Support SDEI_EVENT_STATUS hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_STATUS hypercall. It's used by the guest to retrieve the status about the specified SDEI event. A bitmap is returned to indicate the corresponding status, including registration, enablement and delivery state.
--- arch/arm64/kvm/sdei.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
arch/arm64/kvm/sdei.c +42 -0
@@ -454,6 +454,46 @@ static unsigned long hypercall_unregister(struct kvm_vcpu *vcpu)return ret; } +static unsigned long hypercall_status (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event; +unsigned long event_num =smccc_get_arg1 (vcpu); +int index; +unsigned long ret =0 ; + +if (!kvm_sdei_is_supported (event_num)) { + ret =SDEI_INVALID_PARAMETERS ; + goto out; + } + +spin_lock (&ksdei->lock); + +/* +* Check if the registered event exists. None of the flags +* will be set if it doesn't exist. +*/ + registered_event =find_registered_event (kvm, event_num); +if (!registered_event) + goto unlock; + + exposed_event = registered_event->exposed_event; + index =kvm_sdei_vcpu_index (vcpu, exposed_event); +if (kvm_sdei_is_registered (registered_event, index)) + ret |= (1UL <<SDEI_EVENT_STATUS_REGISTERED ); +if (kvm_sdei_is_enabled (registered_event, index)) + ret |= (1UL <<SDEI_EVENT_STATUS_ENABLED ); +if (registered_event->vcpu_event_count >0 ) + ret |= (1UL <<SDEI_EVENT_STATUS_RUNNING ); + +unlock: +spin_unlock (&ksdei->lock); +out: +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -500,6 +540,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu) ret =hypercall_unregister (vcpu);break ;case SDEI_1_0_FN_SDEI_EVENT_STATUS : + ret =hypercall_status (vcpu); +break ;case SDEI_1_0_FN_SDEI_EVENT_GET_INFO :case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET :case SDEI_1_0_FN_SDEI_PE_MASK :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 9/22 KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_GET_INFO hypercall. It's used by the guest to retrieve various information about the exposed or registered event, including type, signaled, routing mode and affinity. The routing mode and affinity information is only valid to the shared and registered event.
--- arch/arm64/kvm/sdei.c | 73 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+)
arch/arm64/kvm/sdei.c +73 -0
@@ -494,6 +494,77 @@ static unsigned long hypercall_status(struct kvm_vcpu *vcpu)return ret; } +static unsigned long hypercall_info (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event =NULL ; +struct kvm_sdei_registered_event *registered_event =NULL ; +unsigned long event_num =smccc_get_arg1 (vcpu); +unsigned long event_info =smccc_get_arg2 (vcpu); +int index; +unsigned long ret =SDEI_SUCCESS ; + +if (!kvm_sdei_is_supported (event_num)) { + ret =SDEI_INVALID_PARAMETERS ; + goto out; + } + +spin_lock (&ksdei->lock); + +/* +* Retrieve the information from the registered event if it exists. +* Otherwise, we turn into the exposed event if needed. +*/ + registered_event =find_registered_event (kvm, event_num); + exposed_event = registered_event ? registered_event->exposed_event : +find_exposed_event (kvm, event_num); +if (!exposed_event) { + ret =SDEI_INVALID_PARAMETERS ; + goto unlock; + } + +/* Retrieve the requested information */ +switch (event_info) { +case SDEI_EVENT_INFO_EV_TYPE : + ret = exposed_event->state.type; +break ; +case SDEI_EVENT_INFO_EV_SIGNALED : + ret = exposed_event->state.signaled; +break ; +case SDEI_EVENT_INFO_EV_PRIORITY : + ret = exposed_event->state.priority; +break ; +case SDEI_EVENT_INFO_EV_ROUTING_MODE : +case SDEI_EVENT_INFO_EV_ROUTING_AFF : +if (!kvm_sdei_is_shared (exposed_event->state.type)) { + ret =SDEI_INVALID_PARAMETERS ; +break ; + } + + index =kvm_sdei_vcpu_index (vcpu, exposed_event); +if (!registered_event || + !kvm_sdei_is_registered (registered_event, index)) { + ret =SDEI_DENIED ; +break ; + } + +if (event_info ==SDEI_EVENT_INFO_EV_ROUTING_MODE ) + ret = registered_event->state.route_mode; +else + ret = registered_event->state.route_affinity; + +break ; +default : + ret =SDEI_INVALID_PARAMETERS ; + } + +unlock: +spin_unlock (&ksdei->lock); +out: +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -543,6 +614,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu) ret =hypercall_status (vcpu);break ;case SDEI_1_0_FN_SDEI_EVENT_GET_INFO : + ret =hypercall_info (vcpu); +break ;case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET :case SDEI_1_0_FN_SDEI_PE_MASK :case SDEI_1_0_FN_SDEI_PE_UNMASK :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 10/22 KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_ROUTING_SET hypercall. It's used by the guest to set route mode and affinity for the shared and registered events. The request to configure the routing mode and affinity for the private events are disallowed. Besides, It's not allowed to do when the corresponding vCPU events are existing.
--- arch/arm64/kvm/sdei.c | 62 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+)
arch/arm64/kvm/sdei.c +62 -0
@@ -565,6 +565,66 @@ static unsigned long hypercall_info(struct kvm_vcpu *vcpu)return ret; } +static unsigned long hypercall_route (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event; +unsigned long event_num =smccc_get_arg1 (vcpu); +unsigned long route_mode =smccc_get_arg2 (vcpu); +unsigned long route_affinity =smccc_get_arg3 (vcpu); +int index =0 ; +unsigned long ret =SDEI_SUCCESS ; + +if (!kvm_sdei_is_supported (event_num)) { + ret =SDEI_INVALID_PARAMETERS ; + goto out; + } + +/* +* FIXME: The affinity should be verified when it's supported. We +* accept anything for now. +*/ +if (route_mode !=SDEI_EVENT_REGISTER_RM_ANY && + route_mode !=SDEI_EVENT_REGISTER_RM_PE ) { + ret =SDEI_INVALID_PARAMETERS ; + goto out; + } + +spin_lock (&ksdei->lock); + +/* Check if the registered event exists */ + registered_event =find_registered_event (kvm, event_num); +if (!registered_event) { + ret =SDEI_INVALID_PARAMETERS ; + goto unlock; + } + +/* Check the registered event is a shared one */ + exposed_event = registered_event->exposed_event; +if (!kvm_sdei_is_shared (exposed_event->state.type)) { + ret =SDEI_DENIED ; + goto unlock; + } + +if (!kvm_sdei_is_registered (registered_event, index) || +kvm_sdei_is_enabled (registered_event, index) || + registered_event->vcpu_event_count >0 ) { + ret =SDEI_DENIED ; + goto unlock; + } + +/* Update the registered event state */ + registered_event->state.route_mode = route_mode; + registered_event->state.route_affinity = route_affinity; + +unlock: +spin_unlock (&ksdei->lock); +out: +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -617,6 +677,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu) ret =hypercall_info (vcpu);break ;case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET : + ret =hypercall_route (vcpu); +break ;case SDEI_1_0_FN_SDEI_PE_MASK :case SDEI_1_0_FN_SDEI_PE_UNMASK :case SDEI_1_0_FN_SDEI_INTERRUPT_BIND :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 11/22 KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall
In reply to Gavin ShanThis supports SDEI_PE_{MASK, UNMASK} hypercall. They are used by the guest to stop or start receiving SDEI event on the specified vCPU.
--- arch/arm64/kvm/sdei.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
arch/arm64/kvm/sdei.c +16 -0
@@ -625,6 +625,18 @@ static unsigned long hypercall_route(struct kvm_vcpu *vcpu)return ret; } +static unsigned long hypercall_mask (struct kvm_vcpu *vcpu,bool mask) +{ +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +unsigned long ret =SDEI_SUCCESS ; + +spin_lock (&vsdei->lock); + vsdei->state.masked = mask ?1 :0 ; +spin_unlock (&vsdei->lock); + +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -680,7 +692,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu) ret =hypercall_route (vcpu);break ;case SDEI_1_0_FN_SDEI_PE_MASK : + ret =hypercall_mask (vcpu, true); +break ;case SDEI_1_0_FN_SDEI_PE_UNMASK : + ret =hypercall_mask (vcpu, false); +break ;case SDEI_1_0_FN_SDEI_INTERRUPT_BIND :case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE :case SDEI_1_1_FN_SDEI_EVENT_SIGNAL :
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 12/22 KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET
In reply to Gavin ShanThis supports SDEI_{PRIVATE, SHARED}_RESET. They are used by the guest to reset the private events on the calling vCPU or the shared events on all vCPUs.
--- arch/arm64/kvm/sdei.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/arm64/kvm/sdei.c +31 -0
@@ -637,6 +637,31 @@ static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)return ret; } +static unsigned long hypercall_reset (struct kvm_vcpu *vcpu,bool private) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event, *tmp; +unsigned long r, ret =SDEI_SUCCESS ; + +spin_lock (&ksdei->lock); + +list_for_each_entry_safe (registered_event, tmp, + &ksdei->registered_events, link) { + exposed_event = registered_event->exposed_event; +if (private ^kvm_sdei_is_shared (exposed_event->state.type)) +continue ; + + r =unregister_one_event (kvm,NULL , registered_event); + ret = (r ==SDEI_SUCCESS ) ? ret : r; + } + +spin_unlock (&ksdei->lock); + +return ret; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -700,8 +725,14 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)case SDEI_1_0_FN_SDEI_INTERRUPT_BIND :case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE :case SDEI_1_1_FN_SDEI_EVENT_SIGNAL : + ret =SDEI_NOT_SUPPORTED ; +break ;case SDEI_1_0_FN_SDEI_PRIVATE_RESET : + ret =hypercall_reset (vcpu, true); +break ;case SDEI_1_0_FN_SDEI_SHARED_RESET : + ret =hypercall_reset (vcpu, false); +break ;case SDEI_1_1_FN_SDEI_FEATURES :default : ret =SDEI_NOT_SUPPORTED ;
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 13/22 KVM: arm64: Support SDEI_FEATURES hypercall
In reply to Gavin ShanThis supports SDEI_FEATURES hypercall. It's used by the guest to retrieve the supported features, which are number of binding slots and relative mode for the event handler. Currently, none of them is supported.
--- arch/arm64/kvm/sdei.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
arch/arm64/kvm/sdei.c +16 -0
@@ -662,6 +662,20 @@ static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)return ret; } +static unsigned long hypercall_features (struct kvm_vcpu *vcpu) +{ +unsigned long feature =smccc_get_arg1 (vcpu); + +switch (feature) { +case 0 :/* BIND_SLOTS */ +return 0 ; +case 1 :/* RELATIVE_MODE */ +return 0 ; + } + +return SDEI_INVALID_PARAMETERS ; +} +int kvm_sdei_hypercall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm;
@@ -734,6 +748,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu) ret =hypercall_reset (vcpu, false);break ;case SDEI_1_1_FN_SDEI_FEATURES : + ret =hypercall_features (vcpu); +break ;default : ret =SDEI_NOT_SUPPORTED ; }
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 14/22 KVM: arm64: Support SDEI event injection, delivery and cancellation
In reply to Gavin ShanThis supports SDEI event injection, delivery and cancellation. The SDEI event is injected by kvm_sdei_inject_event(). The injected event can be cancelled by kvm_sdei_cancel_event() before it's delivered and handled.
KVM_REQ_SDEI request becomes pending once the SDEI event is injected and kvm_sdei_deliver_event() is called to accommodate the request. The injected SDEI event is delivered and handled in this way. The context for execution is switched like below:
* x0 - x17 are saved. All of them are cleared except the following registers:
x0: SDEI event number
x1: user argument associated with the SDEI event
x2: PC of the interrupted or preempted context
x3: PSTATE of the interrupted or preempted context
* PC is set to the handler of the SDEI event, which was provided during its registration. PSTATE is modified according to the SDEI specification.
* The SDEI event with normal priority can be preempted by that with critical priority. However, no one can preempt the SDEI event with critical event.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/kvm_sdei.h | 4 + arch/arm64/kvm/arm.c | 3 + arch/arm64/kvm/sdei.c | 284 ++++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -46,6 +46,7 @@#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4)#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5) +#define KVM_REQ_SDEI KVM_ARCH_REQ(6)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)
arch/arm64/include/asm/kvm_sdei.h +4 -0
@@ -165,6 +165,10 @@ KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)void kvm_sdei_init_vm (struct kvm *kvm);void kvm_sdei_create_vcpu (struct kvm_vcpu *vcpu);int kvm_sdei_hypercall (struct kvm_vcpu *vcpu); +int kvm_sdei_inject_event (struct kvm_vcpu *vcpu, +unsigned long num,bool immediate); +int kvm_sdei_cancel_event (struct kvm_vcpu *vcpu,unsigned long num); +void kvm_sdei_deliver_event (struct kvm_vcpu *vcpu);void kvm_sdei_destroy_vcpu (struct kvm_vcpu *vcpu);void kvm_sdei_destroy_vm (struct kvm *kvm);
arch/arm64/kvm/arm.c +3 -0
@@ -734,6 +734,9 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_VCPU_RESET , vcpu))kvm_reset_vcpu (vcpu); +if (kvm_check_request (KVM_REQ_SDEI , vcpu)) +kvm_sdei_deliver_event (vcpu); +/* * Clear IRQ_PENDING requests that were made to guarantee * that a VCPU sees new virtual interrupts.
arch/arm64/kvm/sdei.c +284 -0
302 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 15/22 KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest to inject SDEI event, whose number must be zero to the specified vCPU. As the routing mode and affinity isn't supported yet, the calling vCPU is assumed to be the target.
The SDEI event 0x0 is a private one, with normal priority. It's usually used for testing.
--- arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-)
arch/arm64/kvm/sdei.c +63 -1
@@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,return 0 ; } +static unsigned long hypercall_signal (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event; +unsigned long event_num =smccc_get_arg1 (vcpu); +int index; +unsigned long ret =SDEI_SUCCESS ; + +/* @event_num must be zero */ +if (!kvm_sdei_is_default (event_num)) { + ret =SDEI_INVALID_PARAMETERS ; + goto out; + } + +spin_lock (&ksdei->lock); + +/* Check if the registered event exists */ + registered_event =find_registered_event (kvm, event_num); +if (!registered_event) { + ret =SDEI_INVALID_PARAMETERS ; + goto unlock_kvm; + } + +/* +* Check if the event has been registered and enabled. The +* @target_pe parameter isn't checked for now and the event +* is assumed to injected the current vcpu. We should fix it +* when the routing mode and affinity are supported. +*/ + exposed_event = registered_event->exposed_event; + index =kvm_sdei_vcpu_index (vcpu, exposed_event); +if (!kvm_sdei_is_registered (registered_event, index) || + !kvm_sdei_is_enabled (registered_event, index) || +kvm_sdei_is_unregister_pending (registered_event, index)) { + ret =SDEI_INVALID_PARAMETERS ; + goto unlock_kvm; + } + +/* Check if the vcpu has been masked off */ +spin_lock (&vsdei->lock); +if (vsdei->state.masked) { + ret =SDEI_INVALID_PARAMETERS ; + goto unlock_vcpu; + } + +/* Inject the event */ +if (do_inject_event (vcpu, registered_event, false)) + ret =SDEI_INVALID_PARAMETERS ; + +unlock_vcpu: +spin_unlock (&vsdei->lock); +unlock_kvm: +spin_unlock (&ksdei->lock); +out: +return ret; +} +static unsigned long hypercall_reset (struct kvm_vcpu *vcpu,bool private) {struct kvm *kvm = vcpu->kvm;
@@ -827,9 +887,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)break ;case SDEI_1_0_FN_SDEI_INTERRUPT_BIND :case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE : -case SDEI_1_1_FN_SDEI_EVENT_SIGNAL : ret =SDEI_NOT_SUPPORTED ;break ; +case SDEI_1_1_FN_SDEI_EVENT_SIGNAL : + ret =hypercall_signal (vcpu); +break ;case SDEI_1_0_FN_SDEI_PRIVATE_RESET : ret =hypercall_reset (vcpu, true);break ;
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
Hi Gavin,
Quoted 7 lines · Gavin Shan
This supports
SDEI_EVENT_SIGNALhypercall. It's used by the guest to inject SDEI event, whose number must be zero to the specified vCPU. As the routing mode and affinity isn't supported yet, the calling vCPU is assumed to be the target.The SDEI event
0x0is a private one, with normal priority. It's usually used for testing.
I don't know if that is actually the case. One real use that immediately comes to mind is doing an NMI on a wedged CPU. KVM probably shouldn't glean at how the guest may use a particular call, so at most we should just point at the spec and state that event 0 is for software signaled events.
Quoted 16 lines
Signed-off-by Gavin Shan <[email protected]>--- arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-)arch/arm64/kvm/sdei.c +3
-0@@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,return 0 ; } +static unsigned long hypercall_signal (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm;
arch/arm64/kvm/sdei.c static int do_inject_event(struct kvm_vcpu *vcpu, · line 741
Quoted 10 lines
arch/arm64/kvm/sdei.c +10
-0+struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event; +unsigned long event_num =smccc_get_arg1 (vcpu); +int index; +unsigned long ret =SDEI_SUCCESS ; + +/* @event_num must be zero */ +if (!kvm_sdei_is_default (event_num)) {
0 isn't KVM's default event. I'd argue KVM doesn't have a default event to begin with. This has a precise definition coming from the spec. In fact, 'KVM_SDEI_DEFAULT_EVENT' should probably be eliminated, and any missing SDEI definitions should be added to include/uapi/linux/arm_sdei.h.
That goes for any values coming from the specification. KVM's implementation details belong in a KVM header :)
--
Thanks,
Oliver
Re: [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
Hi Oliver,
Quoted 6 lines · Oliver Upton
Quoted 5 lines · Gavin Shan
This supports
SDEI_EVENT_SIGNALhypercall. It's used by the guest to inject SDEI event, whose number must be zero to the specified vCPU. As the routing mode and affinity isn't supported yet, the calling vCPU is assumed to be the target.
Quoted 9 lines
Quoted 2 lines
The SDEI event
0x0is a private one, with normal priority. It's usually used for testing.I don't know if that is actually the case. One real use that immediately comes to mind is doing an NMI on a wedged CPU. KVM probably shouldn't glean at how the guest may use a particular call, so at most we should just point at the spec and state that event 0 is for software signaled events.
Yes, I agree. I will amend the change log in next respin, to make it clear. As you said, it really depend on how the event is used by the guest. We can't assume its usage.
Quoted 32 lines
Quoted 16 lines
Signed-off-by Gavin Shan <[email protected]>--- arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-)arch/arm64/kvm/sdei.c +3
-0@@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,return 0 ; } +static unsigned long hypercall_signal (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm;arch/arm64/kvm/sdei.c static int do_inject_event(struct kvm_vcpu *vcpu, · line 741
Quoted 10 lines
arch/arm64/kvm/sdei.c +10
-0+struct kvm_sdei_kvm *ksdei = kvm->arch.sdei; +struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei; +struct kvm_sdei_exposed_event *exposed_event; +struct kvm_sdei_registered_event *registered_event; +unsigned long event_num =smccc_get_arg1 (vcpu); +int index; +unsigned long ret =SDEI_SUCCESS ; + +/* @event_num must be zero */ +if (!kvm_sdei_is_default (event_num)) {0 isn't KVM's default event. I'd argue KVM doesn't have a default event to begin with. This has a precise definition coming from the spec. In fact, '
KVM_SDEI_DEFAULT_EVENT' should probably be eliminated, and any missing SDEI definitions should be added toinclude/uapi/linux/arm_sdei.h.
Quoted 3 lines
That goes for any values coming from the specification. KVM's implementation details belong in a KVM header :)
Indeed. include/uapi/linux/arm_sdei.h is the right place to define the software signaled event number. The function to validate/check on it is still part of kvm_sdei.h, but its name would be something like kvm_sdei_is_sw_signaled() or kvm_sdei_is_signaled_event().
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 16/22 KVM: arm64: Support SDEI_EVENT_{COMPLETE,COMPLETE_AND_RESUME} hypercall
In reply to Gavin ShanThis supports SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall. They are used by the guest to notify the completion of the SDEI event in the handler. The executing context or registers are modified according to the SDEI specification like below:
* x0 - x17, PC and PState are restored to what values we had in the interrupted or preempted context.
* If it's SDEI_EVENT_COMPLETE_AND_RESUME hypercall, IRQ exception is injected.
--- arch/arm64/include/asm/kvm_emulate.h | 1 + arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/inject_fault.c | 29 +++++++++++ arch/arm64/kvm/sdei.c | 76 +++++++++++++++++++++++++++- 4 files changed, 106 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_emulate.h +1 -0
@@ -37,6 +37,7 @@ bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);void kvm_skip_instr32 (struct kvm_vcpu *vcpu);void kvm_inject_undefined (struct kvm_vcpu *vcpu); +void kvm_inject_irq (struct kvm_vcpu *vcpu);void kvm_inject_vabt (struct kvm_vcpu *vcpu);void kvm_inject_dabt (struct kvm_vcpu *vcpu,unsigned long addr);void kvm_inject_pabt (struct kvm_vcpu *vcpu,unsigned long addr);
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -428,6 +428,7 @@ struct kvm_vcpu_arch {#define KVM_ARM64_EXCEPT_AA32_UND (0 << 9)#define KVM_ARM64_EXCEPT_AA32_IABT (1 << 9)#define KVM_ARM64_EXCEPT_AA32_DABT (2 << 9) +#define KVM_ARM64_EXCEPT_AA32_IRQ (3 << 9)/* For AArch64: */ #define KVM_ARM64_EXCEPT_AA64_ELx_SYNC (0 << 9)#define KVM_ARM64_EXCEPT_AA64_ELx_IRQ (1 << 9)
arch/arm64/kvm/inject_fault.c +29 -0
@@ -66,6 +66,13 @@ static void inject_undef64(struct kvm_vcpu *vcpu)vcpu_write_sys_reg (vcpu, esr,ESR_EL1 ); } +static void inject_irq64 (struct kvm_vcpu *vcpu) +{ + vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1 | + KVM_ARM64_EXCEPT_AA64_ELx_IRQ | +KVM_ARM64_PENDING_EXCEPTION ); +} +#define DFSR_FSC_EXTABT_LPAE 0x10#define DFSR_FSC_EXTABT_nLPAE 0x08#define DFSR_LPAE BIT(9)
@@ -77,6 +84,12 @@ static void inject_undef32(struct kvm_vcpu *vcpu) KVM_ARM64_PENDING_EXCEPTION); } +static void inject_irq32 (struct kvm_vcpu *vcpu) +{ + vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA32_IRQ | +KVM_ARM64_PENDING_EXCEPTION ); +} +/* * Modelled after TakeDataAbortException() and TakePrefetchAbortException * pseudocode.
@@ -160,6 +173,22 @@ void kvm_inject_undefined(struct kvm_vcpu *vcpu)inject_undef64 (vcpu); } +/** +* kvm_inject_irq - inject an IRQ into the guest +* @vcpu: The vCPU in which to inject IRQ +* +* Inject IRQs to the target vCPU. It is assumed that this code is +* called from the VCPU thread and that the VCPU therefore is not +* currently executing guest code. +*/ +void kvm_inject_irq (struct kvm_vcpu *vcpu) +{ +if (vcpu_el1_is_32bit (vcpu)) +inject_irq32 (vcpu); +else +inject_irq64 (vcpu); +} +void kvm_set_sei_esr (struct kvm_vcpu *vcpu,u64 esr) {vcpu_set_vsesr (vcpu, esr & ESR_ELx_ISS_MASK);
arch/arm64/kvm/sdei.c +75 -1
89 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 17/22 KVM: arm64: Support SDEI event notifier
In reply to Gavin ShanThe owner of the SDEI event may want to know if the injected event has been delivered and handled. For example, Async PF cancels the injected SDEI event at early stage of the work, driven by the event has been completed asynchronously.
This supports SDEI event state updating by introducing notifier mechanism. It's notable the notifier (handler) should be capable of migration.
--- arch/arm64/include/asm/kvm_sdei.h | 10 +++++++ arch/arm64/kvm/sdei.c | 45 +++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +10 -0
@@ -18,6 +18,14 @@struct kvm_vcpu ; +typedef void (*kvm_sdei_notifier )(struct kvm_vcpu *vcpu, +unsigned long num, +unsigned int state); +enum { +KVM_SDEI_NOTIFY_DELIVERED , +KVM_SDEI_NOTIFY_COMPLETED , +}; +struct kvm_sdei_exposed_event {struct kvm_sdei_exposed_event_state state;struct kvm *kvm;
@@ -165,6 +173,8 @@ KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)void kvm_sdei_init_vm (struct kvm *kvm);void kvm_sdei_create_vcpu (struct kvm_vcpu *vcpu);int kvm_sdei_hypercall (struct kvm_vcpu *vcpu); +int kvm_sdei_register_event_notifier (struct kvm *kvm,unsigned long num, +kvm_sdei_notifier notifier);int kvm_sdei_inject_event (struct kvm_vcpu *vcpu,unsigned long num,bool immediate);int kvm_sdei_cancel_event (struct kvm_vcpu *vcpu,unsigned long num);
arch/arm64/kvm/sdei.c +45 -0
87 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 18/22 KVM: arm64: Support SDEI ioctl commands on VM
In reply to Gavin ShanThis supports ioctl commands on VM to manage the various objects. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:
* KVM_SDEI_CMD_GET_VERSION
Retrieve the version of current implementation. It's different from the version of the followed SDEI specification. This version is used to indicates what functionalities documented in the SDEI specification have been supported or not supported.
* KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
Return the total count of exposed events.
* KVM_SDEI_CMD_GET_EXPOSED_EVENT
* KVM_SDEI_CMD_SET_EXPOSED_EVENT
Get or set exposed event
* KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
Return the total count of registered events.
* KVM_SDEI_CMD_GET_REGISTERED_EVENT
* KVM_SDEI_CMD_SET_REGISTERED_EVENT
Get or set registered event.
--- arch/arm64/include/asm/kvm_sdei.h | 1 + arch/arm64/include/uapi/asm/kvm_sdei_state.h | 20 ++ arch/arm64/kvm/arm.c | 3 + arch/arm64/kvm/sdei.c | 302 +++++++++++++++++++ include/uapi/linux/kvm.h | 3 + 5 files changed, 329 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +1 -0
@@ -179,6 +179,7 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu, unsigned long num, bool immediate);int kvm_sdei_cancel_event (struct kvm_vcpu *vcpu,unsigned long num);void kvm_sdei_deliver_event (struct kvm_vcpu *vcpu); +long kvm_sdei_vm_ioctl (struct kvm *kvm,unsigned long arg);void kvm_sdei_destroy_vcpu (struct kvm_vcpu *vcpu);void kvm_sdei_destroy_vm (struct kvm *kvm);
arch/arm64/include/uapi/asm/kvm_sdei_state.h +20 -0
@@ -68,5 +68,25 @@ struct kvm_sdei_vcpu_state {struct kvm_sdei_vcpu_regs_state normal_regs; }; +#define KVM_SDEI_CMD_GET_VERSION 0 +#define KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT 1 +#define KVM_SDEI_CMD_GET_EXPOSED_EVENT 2 +#define KVM_SDEI_CMD_SET_EXPOSED_EVENT 3 +#define KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT 4 +#define KVM_SDEI_CMD_GET_REGISTERED_EVENT 5 +#define KVM_SDEI_CMD_SET_REGISTERED_EVENT 6 + +struct kvm_sdei_cmd { +__u32 cmd; +union { +__u32 version; +__u32 count; + }; +union { +struct kvm_sdei_exposed_event_state *exposed_event_state; +struct kvm_sdei_registered_event_state *registered_event_state; + }; +}; +#endif /* !__ASSEMBLY__ */ #endif /* _UAPI__ASM_KVM_SDEI_STATE_H */
arch/arm64/kvm/arm.c +3 -0
@@ -1465,6 +1465,9 @@ long kvm_arch_vm_ioctl(struct file *filp,return -EFAULT ;return kvm_vm_ioctl_mte_copy_tags (kvm, ©_tags); } +case KVM_ARM_SDEI_COMMAND : { +return kvm_sdei_vm_ioctl (kvm, arg); + }default :return -EINVAL ; }
arch/arm64/kvm/sdei.c +302 -0
308 lines not on this page · raw message
include/uapi/linux/kvm.h +3 -0
@@ -2049,4 +2049,7 @@ struct kvm_stats_desc {/* Available with KVM_CAP_XSAVE2 */ #define KVM_GET_XSAVE2 _IOR(KVMIO, 0xcf, struct kvm_xsave) +/* Available with KVM_CAP_ARM_SDEI */ +#define KVM_ARM_SDEI_COMMAND _IOWR(KVMIO, 0xd0, struct kvm_sdei_cmd) +#endif /* __LINUX_KVM_H */
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
Quoted 9 lines · Gavin Shan
This supports ioctl commands on VM to manage the various objects. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:
*
KVM_SDEI_CMD_GET_VERSION
Retrieve the version of current implementation. It's different from the version of the followed SDEI specification. This version is used to indicates what functionalities documented in the SDEI specification have been supported or not supported.
Don't we need a way to set the version as well? KVM is very much responsible for upholding ABI of older specs. So, if a VMM and guest expect SDEI v1.1, we can't just forcibly raise it to something else during a migration.
The PSCI implementation is a great example of how KVM has grown its implementation in line with a specification, all the while preserving backwards compatibility.
Quoted 7 lines
*
KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
Return the total count of exposed events.*
KVM_SDEI_CMD_GET_EXPOSED_EVENT
*KVM_SDEI_CMD_SET_EXPOSED_EVENT
Get or set exposed event
Quoted 6 lines
*
KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
Return the total count of registered events.*
KVM_SDEI_CMD_GET_REGISTERED_EVENT
*KVM_SDEI_CMD_SET_REGISTERED_EVENT
Get or set registered event.
Any new UAPI needs to be documented in Documentation/virt/kvm/api.rst
Additionally, we desperately need a better, generic way to save/restore VM scoped state. IMO, we should only be adding ioctls if we are affording userspace a meaningful interface. Every save/restore pair of ioctls winds up wasting precious ioctl numbers and requires userspace take a change to read/write an otherwise opaque value.
Marc had made some suggestions in this area already that Raghavendra experimented with [1], and I think its time to meaningfully consider our options. Basically, KVM_GET_REG_LIST needs to convey whether a particular register is VM or vCPU state. We only need to save/restore a VM state register once. That way, userspace doesn't have to care about the underlying data and the next piece of VM state that comes along doesn't require an ioctl nr nor VMM participation.
[1]: http://lore.kernel.org/r/[email protected]
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
Hi Oliver,
Quoted 11 lines · Oliver Upton
Quoted 9 lines · Gavin Shan
This supports ioctl commands on VM to manage the various objects. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:
*
KVM_SDEI_CMD_GET_VERSION
Retrieve the version of current implementation. It's different from the version of the followed SDEI specification. This version is used to indicates what functionalities documented in the SDEI specification have been supported or not supported.
Quoted 9 lines
Don't we need a way to set the version as well? KVM is very much responsible for upholding ABI of older specs. So, if a VMM and guest expect SDEI v1.1, we can't just forcibly raise it to something else during a migration.
The PSCI implementation is a great example of how KVM has grown its implementation in line with a specification, all the while preserving backwards compatibility.
The only information feed by VMM is the exposed events. The events can't be registered from guest kernel, and raised from host to guest kernel until it's exposed by VMM. Besides, the exposed events will be defined staticly in host/KVM as we discussed on PATCH[02/22]. We also discussed to eliminate those ioctl commands. So I think we needn't to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a concern to host itself if the migration can be done through the firmware pseudo system registers since the migration compatibility is the only concern to VMM (QEMU).
Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable it.
Quoted 30 lines
Quoted 7 lines
*
KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
Return the total count of exposed events.*
KVM_SDEI_CMD_GET_EXPOSED_EVENT
*KVM_SDEI_CMD_SET_EXPOSED_EVENT
Get or set exposed eventQuoted 6 lines
*
KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNTReturn the total count of registered events.*
KVM_SDEI_CMD_GET_REGISTERED_EVENT
*KVM_SDEI_CMD_SET_REGISTERED_EVENT
Get or set registered event.Any new UAPI needs to be documented in
Documentation/virt/kvm/api.rstAdditionally, we desperately need a better, generic way to save/restore VM scoped state. IMO, we should only be adding ioctls if we are affording userspace a meaningful interface. Every save/restore pair of ioctls winds up wasting precious ioctl numbers and requires userspace take a change to read/write an otherwise opaque value.
Marc had made some suggestions in this area already that Raghavendra experimented with [1], and I think its time to meaningfully consider our options. Basically,
KVM_GET_REG_LISTneeds to convey whether a particular register is VM or vCPU state. We only need to save/restore a VM state register once. That way, userspace doesn't have to care about the underlying data and the next piece of VM state that comes along doesn't require an ioctl nr nor VMM participation.
Quoted 2 lines
Thanks for the pointer to Raghavendra's series. The firmware pseudo system registers have been classified into VM and VCPU scoped in the series. I think it fits the SDEI migration requirements very well. The shared events can even be migrated through the VM scoped firmware pseudo system registers. However, I don't plan to support it in next revision (v6) as currently needed events are all private. I may spend more time to go through Raghavendra's series later.
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
Quoted 8 lines · Gavin Shan
Quoted 4 lines
The PSCI implementation is a great example of how KVM has grown its implementation in line with a specification, all the while preserving backwards compatibility.
The only information feed by VMM is the exposed events. The events can't be registered from guest kernel, and raised from host to guest kernel until it's exposed by VMM.
I would suggest assuming that all SDEI events are exposed by default in KVM. We will not require a VMM change to enable events individually.
Quoted 7 lines
Besides, the exposed events will
be defined staticly in host/KVM as we discussed on PATCH[02/22]. We also discussed to eliminate those ioctl commands. So I think we needn't to addKVM_SDEI_CMD_SET_VERSION. Further more, the version is only a concern to host itself if the migration can be done through the firmware pseudo system registers since the migration compatibility is the only concern to VMM (QEMU).
This all needs to work just like the KVM_REG_ARM_PSCI_VERSION version, I'd recommend taking a look at how we handle that register in KVM.
Quoted 4 lines
Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable it.
As far as how it interacts with KVM, QEMU looks fine. The name of the KVM_ARM_VCPU_PSCI_0_2 bit is quite frustrating. It actually implies that KVM will enable it highest supported PSCI version. If the feature bit is cleared then you only get PSCIv0.1
However, the DT node that QEMU sets up looks a bit crusty. The properties for conveying PSCI function IDs were only ever necessary for PSCIv0.1. The only property of interest any more is 'method', to convey the SMCCC conduit instruction.
--
Thanks,
Oliver
Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
Hi Oliver,
Quoted 6 lines · Oliver Upton
Quoted 5 lines · Gavin Shan
Quoted 4 lines
The PSCI implementation is a great example of how KVM has grown its implementation in line with a specification, all the while preserving backwards compatibility.
Quoted 7 lines
Quoted 3 lines
The only information feed by VMM is the exposed events. The events can't be registered from guest kernel, and raised from host to guest kernel until it's exposed by VMM.
I would suggest assuming that all SDEI events are exposed by default in KVM. We will not require a VMM change to enable events individually.
Ok, it was exactly what I did in v4, but the event is exposed by VMM in v5. In v6, it will be staticly defined again :)
Quoted 11 lines
Quoted 7 lines
Besides, the exposed events will
be defined staticly in host/KVM as we discussed on PATCH[02/22]. We also discussed to eliminate those ioctl commands. So I think we needn't to addKVM_SDEI_CMD_SET_VERSION. Further more, the version is only a concern to host itself if the migration can be done through the firmware pseudo system registers since the migration compatibility is the only concern to VMM (QEMU).This all needs to work just like the
KVM_REG_ARM_PSCI_VERSIONversion, I'd recommend taking a look at how we handle that register in KVM.
Ok. I will do necessary investigation :)
Quoted 10 lines
Quoted 4 lines
Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable it.
As far as how it interacts with KVM, QEMU looks fine. The name of the
KVM_ARM_VCPU_PSCI_0_2bit is quite frustrating. It actually implies that KVM will enable it highest supported PSCI version. If the feature bit is cleared then you only get PSCIv0.1
Quoted 5 lines
However, the DT node that QEMU sets up looks a bit crusty. The properties for conveying PSCI function IDs were only ever necessary for PSCIv0.1. The only property of interest any more is 'method', to convey the SMCCC conduit instruction.
Ok, Thanks again for the further information about PSCI implementation. I will go through the code when I have free cycyles :)
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 19/22 KVM: arm64: Support SDEI ioctl commands on vCPU
In reply to Gavin ShanThis supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:
* KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.
* KVM_SDEI_CMD_GET_VCPU_EVENT
* KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.
* KVM_SDEI_CMD_GET_VCPU_STATE
* KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.
* KVM_SDEI_CMD_INJECT_EVENT
Inject SDEI event.
--- arch/arm64/include/asm/kvm_sdei.h | 1 + arch/arm64/include/uapi/asm/kvm_sdei_state.h | 9 + arch/arm64/kvm/arm.c | 3 + arch/arm64/kvm/sdei.c | 299 +++++++++++++++++++ 4 files changed, 312 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +1 -0
@@ -180,6 +180,7 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,int kvm_sdei_cancel_event (struct kvm_vcpu *vcpu,unsigned long num);void kvm_sdei_deliver_event (struct kvm_vcpu *vcpu);long kvm_sdei_vm_ioctl (struct kvm *kvm,unsigned long arg); +long kvm_sdei_vcpu_ioctl (struct kvm_vcpu *vcpu,unsigned long arg);void kvm_sdei_destroy_vcpu (struct kvm_vcpu *vcpu);void kvm_sdei_destroy_vm (struct kvm *kvm);
arch/arm64/include/uapi/asm/kvm_sdei_state.h +9 -0
@@ -75,6 +75,12 @@ struct kvm_sdei_vcpu_state {#define KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT 4#define KVM_SDEI_CMD_GET_REGISTERED_EVENT 5#define KVM_SDEI_CMD_SET_REGISTERED_EVENT 6 +#define KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT 7 +#define KVM_SDEI_CMD_GET_VCPU_EVENT 8 +#define KVM_SDEI_CMD_SET_VCPU_EVENT 9 +#define KVM_SDEI_CMD_GET_VCPU_STATE 10 +#define KVM_SDEI_CMD_SET_VCPU_STATE 11 +#define KVM_SDEI_CMD_INJECT_EVENT 12struct kvm_sdei_cmd {__u32 cmd;
@@ -85,6 +91,9 @@ struct kvm_sdei_cmd {union {struct kvm_sdei_exposed_event_state *exposed_event_state;struct kvm_sdei_registered_event_state *registered_event_state; +struct kvm_sdei_vcpu_event_state *vcpu_event_state; +struct kvm_sdei_vcpu_state *vcpu_state; +__u64 num; }; };
arch/arm64/kvm/arm.c +3 -0
@@ -1387,6 +1387,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,return kvm_arm_vcpu_finalize (vcpu, what); } +case KVM_ARM_SDEI_COMMAND : { +return kvm_sdei_vcpu_ioctl (vcpu, arg); + }default : r = -EINVAL ; }
arch/arm64/kvm/sdei.c +299 -0
305 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
Quoted 8 lines · Gavin Shan
This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:
*
KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.
Quoted 7 lines
*
KVM_SDEI_CMD_GET_VCPU_EVENT
*KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.*
KVM_SDEI_CMD_GET_VCPU_STATE
*KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.
All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG immediately. Just introduce new registers any time a new event comes along. The only event we have at the end of this series is the software-signaled event, with async PF coming later right?
Some special consideration is likely necessary to avoid adding a register for every u64 chunk of data. I don't think we need to afford userspace any illusion of granularity with these, and can probably lump it all under one giant pseudoregister.
Quoted 2 lines
*
KVM_SDEI_CMD_INJECT_EVENT
Inject SDEI event.
What events are we going to allow userspace to inject? IIUC, the software-signaled event is an IPI and really under the control of the guest. Async PF is entriely under KVM control.
I do agree that having some form of event injection would be great. VM providers have found it useful to allow users to NMI their VMs when they get wedged. I just believe that userspace should not be able to trigger events that have a defined meaning and are under full KVM ownership.
IMO, unless the async PF changes need to go out to userspace, you could probably skip event injection for now and only worry about SDEI within a VM.
--
Thanks,
Oliver
Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
Hi Oliver,
Quoted 22 lines · Oliver Upton
Quoted 8 lines · Gavin Shan
This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:
*
KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.Quoted 7 lines
*
KVM_SDEI_CMD_GET_VCPU_EVENT
*KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.*
KVM_SDEI_CMD_GET_VCPU_STATE
*KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG immediately. Just introduce new registers any time a new event comes along. The only event we have at the end of this series is the software-signaled event, with async PF coming later right?
Quoted 5 lines
Some special consideration is likely necessary to avoid adding a register for every u64 chunk of data. I don't think we need to afford userspace any illusion of granularity with these, and can probably lump it all under one giant pseudoregister.
Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're correct we're only concerned by software signaled event and the one for Async PF.
I didn't look into Raghavendra's series deeply. Actually, a lump of registers can be avoid after 2048 byte size is specified in its encoding. I think 2048 bytes are enough for now since there are only two supported events.
In the future, we probably have varied number of SDEI events to be migrated. In that case, we need to add a new bit to the encoding of the pseudo system register, so that VMM (QEMU) can support variable sized system register and keep reading and writing on these registers on migration:
PSEUDO_SDEI_ADDR: 64-bits in width
PSEUDO_SDEI_DATA: has varied size
PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
(2) The information to be read/written, for example the (shared/private) registered events on VM and vCPU, VCPU state.
PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be written. (2) Flags to indicate current block of information is the last one or not.
Quoted 7 lines
Quoted 2 lines
*
KVM_SDEI_CMD_INJECT_EVENT
Inject SDEI event.What events are we going to allow userspace to inject? IIUC, the software-signaled event is an IPI and really under the control of the guest. Async PF is entriely under KVM control.
Quoted 9 lines
I do agree that having some form of event injection would be great. VM providers have found it useful to allow users to NMI their VMs when they get wedged. I just believe that userspace should not be able to trigger events that have a defined meaning and are under full KVM ownership.
IMO, unless the async PF changes need to go out to userspace, you could probably skip event injection for now and only worry about SDEI within a VM.
I was overthinking on the usage of SDEI. I had the assumption that SDEI may be used by emulated devices to inject SDEI events in VMM. It can even be done through PSEUDO_SDEI_{ADDR, DATA} in future. For now, the software signaled and Async PF events are concerned, Async PF event is always raised by host/KVM. So we needn't it for now and I will drop this functionality, actually the whole ioctl commands and migration support in next respin, as you suggested :)
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
Quoted 31 lines · Gavin Shan
Hi Oliver,
Quoted 22 lines · Oliver Upton
Quoted 8 lines · Gavin Shan
This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:
*
KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.Quoted 7 lines
*
KVM_SDEI_CMD_GET_VCPU_EVENT
*KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.*
KVM_SDEI_CMD_GET_VCPU_STATE
*KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG immediately. Just introduce new registers any time a new event comes along. The only event we have at the end of this series is the software-signaled event, with async PF coming later right?
Quoted 5 lines
Some special consideration is likely necessary to avoid adding a register for every u64 chunk of data. I don't think we need to afford userspace any illusion of granularity with these, and can probably lump it all under one giant pseudoregister.
Quoted 8 lines
Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're correct we're only concerned by software signaled event and the one for Async PF.
I didn't look into Raghavendra's series deeply. Actually, a lump of registers can be avoid after 2048 byte size is specified in its encoding. I think 2048 bytes are enough for now since there are only two supported events.
When I had said 'one giant pseudoregister' I actually meant 'one pseudoregister per event', not all of SDEI into a single structure. Since most of this is in flux now, it is hard to point out what/how we should migrate from conversation alone.
And on the topic of Raghavendra's series, I do not believe it is required anymore here w/ the removal of shared events, which I'm strongly in favor of.
Let's delve deeper into migration on the next series :)
Quoted 9 lines
In the future, we probably have varied number of SDEI events to be migrated. In that case, we need to add a new bit to the encoding of the pseudo system register, so that VMM (QEMU) can support variable sized system register and keep reading and writing on these registers on migration:
PSEUDO_SDEI_ADDR: 64-bits in width
PSEUDO_SDEI_DATA: has varied size
Quoted 7 lines
PSEUDO_SDEI_ADDRis used to (1) Indicate the size ofPSEUDO_SDEI_DATA
(2) The information to be read/written, for example the (shared/private) registered events on VM and vCPU, VCPU state.
PSEUDO_SDEI_DATAis used to (1) Retrieved information or that to be written. (2) Flags to indicate current block of information is the last one or not.
I don't think we have sufficient encoding space in the register ID to allow for arbitrary length registers. Any new registers for SDEI will need to fit into one of the predefined sizes. Note that we've already conditioned userspace to handle registers this way and anything else is an ABI change.
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
Hi Oliver,
Quoted 44 lines · Oliver Upton
Quoted 29 lines · Gavin Shan
Quoted 22 lines · Oliver Upton
Quoted 8 lines · Gavin Shan
This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:
*
KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.Quoted 7 lines
*
KVM_SDEI_CMD_GET_VCPU_EVENT
*KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.*
KVM_SDEI_CMD_GET_VCPU_STATE
*KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG immediately. Just introduce new registers any time a new event comes along. The only event we have at the end of this series is the software-signaled event, with async PF coming later right?
Quoted 5 lines
Some special consideration is likely necessary to avoid adding a register for every u64 chunk of data. I don't think we need to afford userspace any illusion of granularity with these, and can probably lump it all under one giant pseudoregister.
Quoted 8 lines
Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're correct we're only concerned by software signaled event and the one for Async PF.
I didn't look into Raghavendra's series deeply. Actually, a lump of registers can be avoid after 2048 byte size is specified in its encoding. I think 2048 bytes are enough for now since there are only two supported events.
When I had said 'one giant pseudoregister' I actually meant 'one pseudoregister per event', not all of SDEI into a single structure. Since most of this is in flux now, it is hard to point out what/how we should migrate from conversation alone.
Quoted 6 lines
And on the topic of Raghavendra's series, I do not believe it is required anymore here w/ the removal of shared events, which I'm strongly in favor of.
Let's delve deeper into migration on the next series :)
Ok, Thanks for your clarification about 'one giant pseudoregister'. Lets have more discussion about the migration on next revision. To be more clear, I plan to implement the base functionality, where only the private event is supported. After it reaches into mergeable or merged, we can post the add-on series to support migration.
Quoted 17 lines
Quoted 9 lines
In the future, we probably have varied number of SDEI events to be migrated. In that case, we need to add a new bit to the encoding of the pseudo system register, so that VMM (QEMU) can support variable sized system register and keep reading and writing on these registers on migration:
PSEUDO_SDEI_ADDR: 64-bits in width
PSEUDO_SDEI_DATA: has varied sizeQuoted 7 lines
PSEUDO_SDEI_ADDRis used to (1) Indicate the size ofPSEUDO_SDEI_DATA
(2) The information to be read/written, for example the (shared/private) registered events on VM and vCPU, VCPU state.
PSEUDO_SDEI_DATAis used to (1) Retrieved information or that to be written. (2) Flags to indicate current block of information is the last one or not.
Quoted 6 lines
I don't think we have sufficient encoding space in the register ID to allow for arbitrary length registers. Any new registers for SDEI will need to fit into one of the predefined sizes. Note that we've already conditioned userspace to handle registers this way and anything else is an ABI change.
Ok, I think we need padding to structures of the event, to make the event object aligned to 64-bytes aligned, and VCPU state to 512-bytes aligned. 64-bytes and 128-bytes registers have been supported.
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 20/22 KVM: arm64: Export SDEI capability
In reply to Gavin ShanThe SDEI functionality is ready to be exported so far. This adds new capability (KVM_CAP_ARM_SDEI) and exports it.
--- Documentation/virt/kvm/api.rst | 10 ++++++++++ arch/arm64/kvm/arm.c | 3 +++ include/uapi/linux/kvm.h | 1 + 3 files changed, 14 insertions(+)
Documentation/virt/kvm/api.rst +10 -0
@@ -7575,3 +7575,13 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset of the result of KVM_CHECK_EXTENSION. KVM will forward to userspace the hypercalls whose corresponding bit is in the argument, and return ENOSYS for the others. + +8.35 KVM_CAP_ARM_SDEI +--------------------- + +:Capability: KVM_CAP_ARM_SDEI +:Architectures: arm64 + +This capability indicates that the SDEI virtual service is supported +in the host. A VMM can check whether the service is available to enable +it.
arch/arm64/kvm/arm.c +3 -0
@@ -282,6 +282,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_ARM_PTRAUTH_GENERIC : r =system_has_full_ptr_auth ();break ; +case KVM_CAP_ARM_SDEI : + r =1 ; +break ;default : r =0 ; }
include/uapi/linux/kvm.h +1 -0
@@ -1135,6 +1135,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_XSAVE2 208#define KVM_CAP_SYS_ATTRIBUTES 209#define KVM_CAP_PPC_AIL_MODE_3 210 +#define KVM_CAP_ARM_SDEI 211#ifdef KVM_CAP_IRQ_ROUTING
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 21/22 KVM: arm64: Add SDEI document
In reply to Gavin ShanThis adds one document to explain how virtualized SDEI service is implemented and supported.
--- Documentation/virt/kvm/arm/sdei.rst | 325 ++++++++++++++++++++++++++++ 1 file changed, 325 insertions(+) create mode 100644 Documentation/virt/kvm/arm/sdei.rst
Documentation/virt/kvm/arm/sdei.rst new file +325 -0
@@ -0,0 +1,325 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================================== +SDEI Virtualization Support for ARM64 +===================================== + +Arm specification DEN0054/C defines Software Delegated Exception Interface +(SDEI). It provides a mechanism for registering and servicing system events +from system firmware. The interface is offered by a higher exception level +to a lower exception level, in other words by a secure platform firmware +to hypervisor or hypervisor to OS or both. + +https://developer.arm.com/documentation/den0054/c + +KVM/arm64 implements the defined hypercalls in the specification so that +the system events can be registered and serviced from KVM hypervisor to +the guest OS. + +SDEI Event Management +===================== + +Each SDEI event is associated and identified with a 32-bit number. The +lower 24-bits is the event number, but other bits are reserved or used +for vendor defined events. In KVM/arm64 implementation, bit 22 and 23 +are further reserved to identify the events visible to the implementation. +Value 0x1 should be seen in this field for those KVM/arm64 visible events. +In the meanwhile, event number 0x0 is also supported since it is used by +SDEI_EVENT_SIGNAL hypercall. + +The SDEI event needs to be exposed through ioctl interface by VMM to +hypervisor before the guest is able to register it. The exposed event +is represented by ``struct kvm_sdei_exposed_event``. The registered event +is represented by ``struct kvm_sdei_registered_event``, whose instances +are created on SDEI_EVENT_REGISTER hypercall. There is only one registered +event instance in one particular VM no matter what event type is. The +registered event can be injected and delivered to one specific vcpu in +forms of vcpu event. ``struct kvm_sdei_vcpu_event`` describes vcpu events. +On the particular vcpu, one vcpu event instance can be shared by multiple +events. + +The execution runs into SDEI context when vcpu event is handled. The +interrupted or preempted context should be saved to vcpu state, which +is represented by ``struct kvm_sdei_vcpu``. After that, the SDEI event +handler, which was provided by SDEI_EVENT_REGISTER hypercall is invoked. +SDEI_EVENT_COMPLETE or SDEI_EVENT_COMPLETE_AND_RESUME hypercall must be +issued before the SDEI event handler is to complete. When one of these +two hypercalls is received, the interrupted or preempted context is +restored from vcpu state for execution. + +When migration happens, the status of one particular SDEI event is not +deterministic. So we need to support migration for the aforementioned +structures or objects. The information capable of migration in these +objects are put into separate data structures, which are the corresponding +state variant in kvm_sdei_state.h. Besides, ioctl command are introduced +to read and write them on the source and destination VM during migration. + +IOCTL Commands +============== + +KVM_ARM_SDEI_COMMAND +-------------------- + +:Capability: KVM_CAP_ARM_SDEI +:Type: vm ioctl, vcpu ioctl +:Parameters: struct kvm_sdei_cmd +:Returns: 0 on success, < 0 on error + +:: + + struct kvm_sdei_cmd { + __u32 cmd; + union { + __u32 version; + __u32 count; + }; + union { + struct kvm_sdei_exposed_event_state *exposed_event_state; + struct kvm_sdei_registered_event_state *registered_event_state; + struct kvm_sdei_vcpu_event_state *vcpu_event_state; + struct kvm_sdei_vcpu_state *vcpu_state; + __u64 num; + }; + }; + +The SDEI ioctl command is identified by KVM_ARM_SDEI_COMMAND and ``cmd`` +in the argument ``struct kvm_sdei_cmd`` provides further command to be +executed. + +KVM_SDEI_CMD_GET_VERSION +------------------------ + +:Type: vm ioctl +:Parameters: struct kvm_sdei_cmd +:Returns: 0 on success, < 0 on error + +On success, the implementation version is returned in ``version`` of +``struct kvm_sdei_cmd``. This version is different from that of the +followed SDEI specification. The implementation version is used to tell +the coherence extent to the following specification. For example, the +SDEI interrupt binding event is defined in SDEI v1.1 specification, +but it is not supported in current implementation. + +KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT +------------------------------------ + +:Type: vm ioctl +:Parameters: struct kvm_sdei_cmd +:Returns: 0 on success, < 0 on error + +This ioctl command is used prior to KVM_SDEI_CMD_GET_EXPOSED_EVENT, to +prepare ``exposed_event_state`` of ``struct kvm_sdei_cmd`` for that +command during migration. + +On success, the number of exposed events is returned by ``count`` +of ``struct kvm_sdei_cmd``. + +KVM_SDEI_CMD_GET_EXPOSED_EVENT +------------------------------ + +:Type: vm ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_exposed_event_state +:Returns: 0 on success, < 0 on error + +:: + + struct kvm_sdei_exposed_event_state { + __u64 num; + + __u8 type; + __u8 signaled; + __u8 priority; + __u8 padding[5]; + __u64 notifier; + }; + +This ioctl command is used to retrieve the exposed events on the source +VM during migration. + +The number of exposed events to be retrieved is specified by ``count`` +of ``struct kvm_sdei_cmd``. On success, the retrieved exposed events are +returned by ``exposed_event_state`` of ``struct kvm_sdei_state``. + +KVM_SDEI_CMD_SET_EXPOSED_EVENT +------------------------------ + +:Type: vm ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_exposed_event_state +:Returns: 0 on success, < 0 on error + +This ioctl command is used by VMM to expose SDEI events and migrate +them. The ``notifier`` of ``struct kvm_sdei_exposed_event_state`` +will be modified if the specified exposed event has been existing. + +The number of events to be exposed is specified by ``count`` of +``struct kvm_sdei_cmd``. The information about the exposed events is +passed by ``exposed_event_state`` of ``struct kvm_sdei_cmd``. + +KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT +--------------------------------------- + +:Type: vm ioctl +:Parameters: struct kvm_sdei_cmd +:Returns: 0 on success, < 0 on error + +This ioctl command is used prior to KVM_SDEI_CMD_GET_REGISTERED_EVENT, +to prepare ``registered_event_state`` of ``struct kvm_sdei_cmd`` for +that command during migration. + +On success, the number of registered events is returned by ``count`` +of ``struct kvm_sdei_cmd``. + +KVM_SDEI_CMD_GET_REGISTERED_EVENT +--------------------------------- + +:Type: vm ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_registered_event_state +:Returns: 0 on success, < 0 on error + +:: + struct kvm_sdei_registered_event_state { + __u64 num; + + __u8 route_mode; + __u8 padding[3]; + __u64 route_affinity; + __u64 ep_address[KVM_SDEI_MAX_VCPUS]; + __u64 ep_arg[KVM_SDEI_MAX_VCPUS]; + __u64 registered[KVM_SDEI_MAX_VCPUS/64]; + __u64 enabled[KVM_SDEI_MAX_VCPUS/64]; + __u64 unregister_pending[KVM_SDEI_MAX_VCPUS/64]; + }; + +This ioctl command is used to retrieve the registered events on the +source VM during migration. + +The number of registered events to be retrieved is specified by ``count`` +of ``struct kvm_sdei_cmd``. On success, the retrieved registered events +are returned by ``registered_event_state`` of ``struct kvm_sdei_state``. + +KVM_SDEI_CMD_SET_REGISTERED_EVENT +--------------------------------- + +:Type: vm ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_registered_event_state +:Returns: 0 on success, < 0 on error + +This ioctl command is used by VMM to migrate the registered events +on the destination VM. + +The number of events to be registered is specified by ``count`` of +``struct kvm_sdei_cmd``. The information about the registered events +is passed by ``registered_event_state`` of ``struct kvm_sdei_cmd``. + +KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT +--------------------------------- + +:Type: vcpu ioctl +:Parameters: struct kvm_sdei_cmd +:Returns: 0 on success, < 0 on error + +This ioctl command is used prior to KVM_SDEI_CMD_GET_VCPU_EVENT, to +prepare ``vcpu_event_state`` of ``struct kvm_sdei_cmd`` for that +command during migration. + +On success, the number of vcpu events is returned by ``count`` +of ``struct kvm_sdei_cmd``. + +KVM_SDEI_CMD_GET_VCPU_EVENT +--------------------------- + +:Type: vcpu ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_event_state +:Returns: 0 on success, < 0 on error + +:: + + struct kvm_sdei_vcpu_event_state { + __u64 num; + + __u32 event_count; + __u32 padding; + }; + +This ioctl command is used to retrieve the vcpu events on the source +VM during migration. + +The number of vcpu events to be retrieved is specified by ``count`` of +``struct kvm_sdei_cmd``. On success, the retrieved exposed events are +returned by ``vcpu_event_state`` of ``struct kvm_sdei_state``. + +KVM_SDEI_CMD_SET_VCPU_EVENT +--------------------------- + +:Type: vcpu ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_event_state +:Returns: 0 on success, < 0 on error + +This ioctl command is used by VMM to migrate the vcpu events on the +destination VM. + +The number of vcpu events to be added is specified by ``count`` of +``struct kvm_sdei_cmd``. The information about the vcpu events is +passed by ``vcpu_event_state`` of ``struct kvm_sdei_cmd``. + +KVM_SDEI_CMD_GET_VCPU_STATE +--------------------------- + +:Type: vcpu ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state +:Returns: 0 on success, < 0 on error + +:: + + struct kvm_sdei_vcpu_regs_state { + __u64 regs[18]; + __u64 pc; + __u64 pstate; + }; + + struct kvm_sdei_vcpu_state { + __u8 masked; + __u8 padding[7]; + __u64 critical_num; + __u64 normal_num; + struct kvm_sdei_vcpu_regs_state critical_regs; + struct kvm_sdei_vcpu_regs_state normal_regs; + }; + +This ioctl command is used to retrieve the vcpu state on the source VM +during migration. + +On success, the current vcpu state is returned by ``vcpu_state`` of +``struct kvm_sdei_cmd``. + +KVM_SDEI_CMD_SET_VCPU_STATE +--------------------------- + +:Type: vcpu ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state +:Returns: 0 on success, < 0 on error + +This ioctl command is used by VMM to migrate the vcpu state on the +destination VM. + +The vcpu state to be configured is passed by ``vcpu_state`` of +``struct kvm_sdei_cmd``. + +KVM_SDEI_CMD_INJECT_EVENT +------------------------- + +:Type: vcpu ioctl +:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state +:Returns: 0 on success, < 0 on error + +This ioctl command is used by VMM to inject SDEI event to the specified +vcpu. + +The number of the SDEI event to be injected is passed by ``num`` of +``struct kvm_sdei_cmd``. + +Future Work +=========== + +1. Support the routing mode and affinity for the shared events. +2. Support interrupt binding events.
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v5 22/22 KVM: selftests: Add SDEI test case
In reply to Gavin ShanThis adds SDEI self-test case where the various hypercalls are issued to default event (0x0). The default event is private, signaled and in normal priority.
By default, two vCPUs are started and the following ioctl commands or hypercalls are sent to them in sequence, to simulate how they are used in VMM and the linux guest:
kvm_check_cap(KVM_CAP_ARM_SDEI)
KVM_SDEI_CMD_GET_VERSION
KVM_SDEI_CMD_SET_EXPOSED_EVENT (expose event)
SDEI_1_0_FN_SDEI_VERSION
SDEI_1_1_FN_SDEI_FEATURES (SDEI capability probing) SDEI_1_0_FN_SDEI_SHARED_RESET (restart SDEI) SDEI_1_0_FN_SDEI_PE_UNMASK (CPU online)
SDEI_1_0_FN_SDEI_EVENT_GET_INFO
SDEI_1_0_FN_SDEI_EVENT_REGISTER (register event) SDEI_1_0_FN_SDEI_EVENT_ENABLE (enable event) SDEI_1_1_FN_SDEI_EVENT_SIGNAL (event injection)
SDEI_1_0_FN_SDEI_EVENT_DISABLE (disable event) SDEI_1_0_FN_SDEI_EVENT_UNREGISTER (unregister event) SDEI_1_0_FN_SDEI_PE_MASK (CPU offline)
--- tools/testing/selftests/kvm/Makefile | 1 + tools/testing/selftests/kvm/aarch64/sdei.c | 525 +++++++++++++++++++++ 2 files changed, 526 insertions(+) create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -105,6 +105,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list TEST_GEN_PROGS_aarch64 += aarch64/psci_cpu_on_test TEST_GEN_PROGS_aarch64 += aarch64/vgic_init TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq +TEST_GEN_PROGS_aarch64 += aarch64/sdei TEST_GEN_PROGS_aarch64 += demand_paging_test TEST_GEN_PROGS_aarch64 += dirty_log_test TEST_GEN_PROGS_aarch64 += dirty_log_perf_test
tools/testing/selftests/kvm/aarch64/sdei.c new file +525 -0
525 lines not on this page · raw message
Signature · 6 lines
2.23.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 132 lines · Gavin Shan
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei")For the design and migration needs, please refer to the document in PATCH[21/22] in this series. The series is organized as below:
PATCH[01] Introduces template for
smccc_get_argx()
PATCH[02] Adds SDEI virtualization infrastructure
PATCH[03-17] Supports various SDEI hypercalls and event handling PATCH[18-20] Adds ioctl commands to support migration and configuration
and exports SDEI capability
PATCH[21] Adds SDEI document
PATCH[22] Adds SDEI selftest caseTesting
=======[1] The selftest case included in this series works fine. The default SDEI
event, whose number is zero, can be registered, enabled, raised. The SDEI event handler can be invoked. [host]# pwd /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm [root@virtlab-arm01 kvm]# ./aarch64/sdei NR_VCPUS: 2 SDEI Event: 0x00000000 --- VERSION Version: 1.1 (vendor: 0x4b564d) --- FEATURES Shared event slots: 0 Private event slots: 0 Relative mode: No --- PRIVATE_RESET --- SHARED_RESET --- PE_UNMASK --- EVENT_GET_INFO Type: Private Priority: Normal Signaled: Yes --- EVENT_REGISTER --- EVENT_ENABLE --- EVENT_SIGNAL Handled: Yes IRQ: No Status: Registered-Enabled-Running PC/PSTATE: 000000000040232c 00000000600003c5 Regs: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 --- PE_MASK --- EVENT_DISABLE --- EVENT_UNREGISTER Result: OK[2] There are additional patches in the following repositories to create
procfs entries, allowing to inject SDEI event from host side. The SDEI client in the guest side registers the SDEI default event, whose number is zero. Also, the QEMU exports SDEI ACPI table and supports migration for SDEI. https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei") [2.1] Start the guests and migrate the source VM to the destination VM. [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 6,sockets=2,cores=3,threads=1 \ -m 1024M,slots=16,maxmem=64G \ : \ -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \ -initrd /home/gavin/sandbox/images/rootfs.cpio.xz \ -append earlycon=pl011,mmio,0x9000000 \ : [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 6,sockets=2,cores=3,threads=1 \ -m 1024M,slots=16,maxmem=64G \ : \ -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \ -initrd /home/gavin/sandbox/images/rootfs.cpio.xz \ -append earlycon=pl011,mmio,0x9000000 \ -incoming tcp:0:4444 \ : [2.2] Check kernel log on the source VM. The SDEI service is enabled and the default SDEI event (0x0) is enabled. [guest-src]# dmesg | grep -i sdei ACPI: SDEI 0x000000005BC80000 000024 \ (v00 BOCHS BXPC 00000001 BXPC 00000001) sdei: SDEIv1.1 (0x4b564d) detected in firmware. SDEI TEST: Version 1.1, Vendor 0x4b564d sdei_init: SDEI event (0x0) registered sdei_init: SDEI event (0x0) enabled (qemu) migrate -d tcp:localhost:4444 [2.3] Migrate the source VM to the destination VM. Inject SDEI event to the destination VM. The event is raised and handled. (qemu) migrate -d tcp:localhost:4444 [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1 [guest-dst]# =========== SDEI Event (CPU#1) =========== Event: 0000000000000000 Parameter: 00000000dabfdabf PC: ffff800008cbb554 PSTATE: 00000000604000c5 SP: ffff800009c7bde0 Regs: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff800009399008 ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000
Quoted 3 lines
Changelog
=========
v5:
Next time can you include a link to the cover letter of the previous patch set? It is extremely helpful for understanding the progress to date and allows reviewers to see prior feedback.
--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi Oliver,
Quoted 133 lines · Oliver Upton
Quoted 132 lines · Gavin Shan
This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:
https://developer.arm.com/documentation/den0054/c https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei")For the design and migration needs, please refer to the document in PATCH[21/22] in this series. The series is organized as below:
PATCH[01] Introduces template for
smccc_get_argx()
PATCH[02] Adds SDEI virtualization infrastructure
PATCH[03-17] Supports various SDEI hypercalls and event handling PATCH[18-20] Adds ioctl commands to support migration and configuration
and exports SDEI capability
PATCH[21] Adds SDEI document
PATCH[22] Adds SDEI selftest caseTesting
=======[1] The selftest case included in this series works fine. The default SDEI
event, whose number is zero, can be registered, enabled, raised. The SDEI event handler can be invoked. [host]# pwd /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm [root@virtlab-arm01 kvm]# ./aarch64/sdei NR_VCPUS: 2 SDEI Event: 0x00000000 --- VERSION Version: 1.1 (vendor: 0x4b564d) --- FEATURES Shared event slots: 0 Private event slots: 0 Relative mode: No --- PRIVATE_RESET --- SHARED_RESET --- PE_UNMASK --- EVENT_GET_INFO Type: Private Priority: Normal Signaled: Yes --- EVENT_REGISTER --- EVENT_ENABLE --- EVENT_SIGNAL Handled: Yes IRQ: No Status: Registered-Enabled-Running PC/PSTATE: 000000000040232c 00000000600003c5 Regs: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 --- PE_MASK --- EVENT_DISABLE --- EVENT_UNREGISTER Result: OK[2] There are additional patches in the following repositories to create
procfs entries, allowing to inject SDEI event from host side. The SDEI client in the guest side registers the SDEI default event, whose number is zero. Also, the QEMU exports SDEI ACPI table and supports migration for SDEI. https://github.com/gwshan/linux ("kvm/arm64_sdei") https://github.com/gwshan/qemu ("kvm/arm64_sdei") [2.1] Start the guests and migrate the source VM to the destination VM. [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 6,sockets=2,cores=3,threads=1 \ -m 1024M,slots=16,maxmem=64G \ : \ -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \ -initrd /home/gavin/sandbox/images/rootfs.cpio.xz \ -append earlycon=pl011,mmio,0x9000000 \ : [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 6,sockets=2,cores=3,threads=1 \ -m 1024M,slots=16,maxmem=64G \ : \ -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \ -initrd /home/gavin/sandbox/images/rootfs.cpio.xz \ -append earlycon=pl011,mmio,0x9000000 \ -incoming tcp:0:4444 \ : [2.2] Check kernel log on the source VM. The SDEI service is enabled and the default SDEI event (0x0) is enabled. [guest-src]# dmesg | grep -i sdei ACPI: SDEI 0x000000005BC80000 000024 \ (v00 BOCHS BXPC 00000001 BXPC 00000001) sdei: SDEIv1.1 (0x4b564d) detected in firmware. SDEI TEST: Version 1.1, Vendor 0x4b564d sdei_init: SDEI event (0x0) registered sdei_init: SDEI event (0x0) enabled (qemu) migrate -d tcp:localhost:4444 [2.3] Migrate the source VM to the destination VM. Inject SDEI event to the destination VM. The event is raised and handled. (qemu) migrate -d tcp:localhost:4444 [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1 [guest-dst]# =========== SDEI Event (CPU#1) =========== Event: 0000000000000000 Parameter: 00000000dabfdabf PC: ffff800008cbb554 PSTATE: 00000000604000c5 SP: ffff800009c7bde0 Regs: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff800009399008 ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000
Quoted 8 lines
Quoted 3 lines
Changelog
=========
v5:Next time can you include a link to the cover letter of the previous patch set? It is extremely helpful for understanding the progress to date and allows reviewers to see prior feedback.
Yep, I will provide the link to the cover letter of the previous version. I'm amending it this time:
https://lore.kernel.org/all/[email protected]/
Besides, I don't know what happened to my "git send-email". Some of the recipients are skipped even I have put them into the cc list. Lets amend it again to avoid resending the series.
Thanks,
Gavin
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm