Skip to contentSkip to conversation

Support SDEI Virtualization

PATCH Superseded Gavin Shan 2 participants 49 messages Mar 22, 2022 – Mar 25, 2022

Subsystems (6)
Documentation KVM Kselftest arm64 +2 more subsystems

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Gavin Shan

author
#0 lore raw

v5 cover Support SDEI Virtualization

This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:

https://developer.arm.com/documentation/den0054/c
https://github.com/gwshan/linux ("kvm/arm64_sdei")
https://github.com/gwshan/qemu ("kvm/arm64_sdei")

For the design and migration needs, please refer to the document in PATCH[21/22] in this series. The series is organized as below:

PATCH[01] Introduces template for smccc_get_argx()
PATCH[02] Adds SDEI virtualization infrastructure
PATCH[03-17] Supports various SDEI hypercalls and event handling PATCH[18-20] Adds ioctl commands to support migration and configuration
and exports SDEI capability
PATCH[21] Adds SDEI document
PATCH[22] Adds SDEI selftest case

Testing
=======

[1] The selftest case included in this series works fine. The default SDEI

    event, whose number is zero, can be registered, enabled, raised. The
    SDEI event handler can be invoked.

    [host]# pwd
    /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
    [root@virtlab-arm01 kvm]# ./aarch64/sdei 

        NR_VCPUS: 2    SDEI Event: 0x00000000

    --- VERSION
        Version:              1.1 (vendor: 0x4b564d)
    --- FEATURES
        Shared event slots:   0
        Private event slots:  0
        Relative mode:        No
    --- PRIVATE_RESET
    --- SHARED_RESET
    --- PE_UNMASK
    --- EVENT_GET_INFO
        Type:                 Private
        Priority:             Normal
        Signaled:             Yes
    --- EVENT_REGISTER
    --- EVENT_ENABLE
    --- EVENT_SIGNAL
        Handled:              Yes
        IRQ:                  No
        Status:               Registered-Enabled-Running
        PC/PSTATE:            000000000040232c 00000000600003c5
        Regs:                 0000000000000000 0000000000000000
                              0000000000000000 0000000000000000
    --- PE_MASK
    --- EVENT_DISABLE
    --- EVENT_UNREGISTER

        Result: OK

[2] There are additional patches in the following repositories to create

    procfs entries, allowing to inject SDEI event from host side. The
    SDEI client in the guest side registers the SDEI default event, whose
    number is zero. Also, the QEMU exports SDEI ACPI table and supports
    migration for SDEI.

    https://github.com/gwshan/linux    ("kvm/arm64_sdei")
    https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

    [2.1] Start the guests and migrate the source VM to the destination
          VM.

    [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
            -accel kvm -machine virt,gic-version=host                     \
            -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
            -m 1024M,slots=16,maxmem=64G                                  \
               :                                                          \
            -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
            -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
            -append earlycon=pl011,mmio,0x9000000                         \
               :

    [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
            -accel kvm -machine virt,gic-version=host                     \
            -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
            -m 1024M,slots=16,maxmem=64G                                  \
               :                                                          \
            -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
            -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
            -append earlycon=pl011,mmio,0x9000000                         \
            -incoming tcp:0:4444                                          \
               :

    [2.2] Check kernel log on the source VM. The SDEI service is enabled
          and the default SDEI event (0x0) is enabled.

     [guest-src]# dmesg | grep -i sdei
     ACPI: SDEI 0x000000005BC80000 000024 \
                (v00 BOCHS  BXPC     00000001 BXPC 00000001)
     sdei: SDEIv1.1 (0x4b564d) detected in firmware.
     SDEI TEST: Version 1.1, Vendor 0x4b564d
     sdei_init: SDEI event (0x0) registered
     sdei_init: SDEI event (0x0) enabled

 
     (qemu) migrate -d tcp:localhost:4444

    [2.3] Migrate the source VM to the destination VM. Inject SDEI event
          to the destination VM. The event is raised and handled.

    (qemu) migrate -d tcp:localhost:4444

    [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1

    [guest-dst]#
    =========== SDEI Event (CPU#1) ===========
    Event: 0000000000000000  Parameter: 00000000dabfdabf
    PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
    Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
             ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 ffff800009399008 
             ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 
             0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 
             0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 
    Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
             ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 
             ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000 

Changelog
=========
v5:
* Rebased to v5.17.rc7 (Gavin)
* Unified names for the objects, data structures, variables and functions. The events have been named as exposed, registered and vcpu event. The staes needs to be migrated
is put into kvm_sdei_state.h (Eric)

   * More inline functions to visit SDEI event's properties   (Eric)
   * Support unregistration pending state                     (Eric)
   * Support v1.1 SDEI specification                          (Eric)
   * Fold the code to inject, deliver and handle SDEI event
     from PATCH[v4 13/18/19] into PATCH[v5 13]                (Eric)
   * Simplified ioctl interface to visit all events at once   (Eric/Gavin)
   * Improved reference count and avoid its migration. Also,
     the limit to memory allocation is added based on it.     (Eric)
   * Change the return values from hypercall functions        (Eric) 
   * Validate @ksdei and @vsdi in kvm_sdei_hypercall()        (Shannon)
   * Add document to explain how SDEI virutalization and the
     migration are supported                                  (Eric)
   * Improved selftest case to inject and handle SDEI event   (Gavin)
   * Improved comments and commit logs                        (Eric)

* Address misc comments from Eric. Hopefully, all of them are covered in v5 because Eric provided lots of comments
in the last round of review (Eric)
v4:
* Rebased to v5.14.rc5 (Gavin)
v3:
* Rebased to v5.13.rc1 (Gavin)
* Use linux data types in kvm_sdei.h (Gavin)
v2:

   * Rebased to v5.11.rc6                                         (Gavin)
   * Dropped changes related to SDEI client driver                (Gavin)
   * Removed support for passthrou SDEI events                    (Gavin)
   * Redesigned data structures                                   (Gavin)

* Implementation is almost rewritten as the data structures
are totally changed (Gavin)
* Added ioctl commands to support migration (Gavin)

Gavin Shan (22):
KVM: arm64: Introduce template for inline functions
KVM: arm64: Add SDEI virtualization infrastructure
KVM: arm64: Support SDEI_VERSION hypercall
KVM: arm64: Support SDEI_EVENT_REGISTER hypercall
KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall
KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall
KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall
KVM: arm64: Support SDEI_EVENT_STATUS hypercall
KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall
KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall
KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall
KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET
KVM: arm64: Support SDEI_FEATURES hypercall
KVM: arm64: Support SDEI event injection, delivery and cancellation
KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
KVM: arm64: Support SDEI_EVENT_{COMPLETE,COMPLETE_AND_RESUME}
hypercall
KVM: arm64: Support SDEI event notifier
KVM: arm64: Support SDEI ioctl commands on VM
KVM: arm64: Support SDEI ioctl commands on vCPU
KVM: arm64: Export SDEI capability
KVM: arm64: Add SDEI document
KVM: selftests: Add SDEI test case

 Documentation/virt/kvm/api.rst               |   10 +
 Documentation/virt/kvm/arm/sdei.rst          |  325 +++
 arch/arm64/include/asm/kvm_emulate.h         |    1 +
 arch/arm64/include/asm/kvm_host.h            |    5 +
 arch/arm64/include/asm/kvm_sdei.h            |  187 ++
 arch/arm64/include/uapi/asm/kvm.h            |    1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  101 +
 arch/arm64/kvm/Makefile                      |    2 +-
 arch/arm64/kvm/arm.c                         |   20 +
 arch/arm64/kvm/hypercalls.c                  |   21 +
 arch/arm64/kvm/inject_fault.c                |   29 +
 arch/arm64/kvm/sdei.c                        | 1900 ++++++++++++++++++
 include/kvm/arm_hypercalls.h                 |   24 +-
 include/uapi/linux/arm_sdei.h                |    2 +
 include/uapi/linux/kvm.h                     |    4 +
 tools/testing/selftests/kvm/Makefile         |    1 +
 tools/testing/selftests/kvm/aarch64/sdei.c   |  525 +++++
 17 files changed, 3145 insertions(+), 13 deletions(-)
 create mode 100644 Documentation/virt/kvm/arm/sdei.rst
 create mode 100644 arch/arm64/include/asm/kvm_sdei.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
 create mode 100644 arch/arm64/kvm/sdei.c
 create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c
Signature · 1 line
2.23.0
author
#1 lore raw

PATCH v5 1/22 KVM: arm64: Introduce template for inline functions

The inline functions used to get the SMCCC parameters have same layout. It means these functions can be presented by an unified template, to make the code simplified. Besides, this adds more similar inline functions like smccc_get_arg{4,5,6,7,8}() to get more SMCCC arguments, which are needed by SDEI virtualization support.

---
 include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
include/kvm/arm_hypercalls.h +12 -12
index 0e2509d27910..d5144c852fe4 100644
@@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
 	return vcpu_get_reg(vcpu, 0);
 }
 
-static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 1);
-}
-
-static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 2);
+#define SMCCC_DECLARE_GET_ARG(reg)					\
+static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
+{									\
+	return vcpu_get_reg(vcpu, reg);					\
 }
 
-static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 3);
-}
+SMCCC_DECLARE_GET_ARG(1)
+SMCCC_DECLARE_GET_ARG(2)
+SMCCC_DECLARE_GET_ARG(3)
+SMCCC_DECLARE_GET_ARG(4)
+SMCCC_DECLARE_GET_ARG(5)
+SMCCC_DECLARE_GET_ARG(6)
+SMCCC_DECLARE_GET_ARG(7)
+SMCCC_DECLARE_GET_ARG(8)
 
 static inline void smccc_set_retval(struct kvm_vcpu *vcpu,
 				    unsigned long a0,
Signature · 1 line
2.23.0
#2 lore raw

Re: [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions

Hi Gavin,

Quoted 36 lines · Gavin Shan

The inline functions used to get the SMCCC parameters have same layout. It means these functions can be presented by an unified template, to make the code simplified. Besides, this adds more similar inline functions like smccc_get_arg{4,5,6,7,8}() to get more SMCCC arguments, which are needed by SDEI virtualization support.

---
 include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
include/kvm/arm_hypercalls.h +4 -10
index 0e2509d27910..d5144c852fe4 100644
@@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
 	return vcpu_get_reg(vcpu, 0);
 }
 
-static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 1);
-}
-
-static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 2);
+#define SMCCC_DECLARE_GET_ARG(reg)					\
+static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
+{									\
+	return vcpu_get_reg(vcpu, reg);					\
 }
 
-static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
-{

include/kvm/arm_hypercalls.h static inline u32 smccc_get_function(struct kvm_vcpu *vcpu) · line 29

Quoted 10 lines
include/kvm/arm_hypercalls.h +8 -2
-	return vcpu_get_reg(vcpu, 3);
-}
+SMCCC_DECLARE_GET_ARG(1)
+SMCCC_DECLARE_GET_ARG(2)
+SMCCC_DECLARE_GET_ARG(3)
+SMCCC_DECLARE_GET_ARG(4)
+SMCCC_DECLARE_GET_ARG(5)
+SMCCC_DECLARE_GET_ARG(6)
+SMCCC_DECLARE_GET_ARG(7)
+SMCCC_DECLARE_GET_ARG(8)

Hmm. What if we specify a single inline function where the caller passes the arg # as a parameter? We really just want to abstract away the off-by-one difference between GP registers and SMCCC arguments.

Macros generally make me uneasy for template functions, but I may be in the vocal minority on this topic :)

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#3 lore raw

Re: [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions

Hi Oliver,

Quoted 52 lines · Oliver Upton
Quoted 36 lines · Gavin Shan

The inline functions used to get the SMCCC parameters have same layout. It means these functions can be presented by an unified template, to make the code simplified. Besides, this adds more similar inline functions like smccc_get_arg{4,5,6,7,8}() to get more SMCCC arguments, which are needed by SDEI virtualization support.

---
  include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
  1 file changed, 12 insertions(+), 12 deletions(-)
include/kvm/arm_hypercalls.h +4 -10
index 0e2509d27910..d5144c852fe4 100644
@@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
  	return vcpu_get_reg(vcpu, 0);
  }
 
-static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 1);
-}
-
-static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 2);
+#define SMCCC_DECLARE_GET_ARG(reg)					\
+static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
+{									\
+	return vcpu_get_reg(vcpu, reg);					\
  }
 
-static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
-{

include/kvm/arm_hypercalls.h static inline u32 smccc_get_function(struct kvm_vcpu *vcpu) · line 29

Quoted 10 lines
include/kvm/arm_hypercalls.h +8 -2
-	return vcpu_get_reg(vcpu, 3);
-}
+SMCCC_DECLARE_GET_ARG(1)
+SMCCC_DECLARE_GET_ARG(2)
+SMCCC_DECLARE_GET_ARG(3)
+SMCCC_DECLARE_GET_ARG(4)
+SMCCC_DECLARE_GET_ARG(5)
+SMCCC_DECLARE_GET_ARG(6)
+SMCCC_DECLARE_GET_ARG(7)
+SMCCC_DECLARE_GET_ARG(8)

Hmm. What if we specify a single inline function where the caller passes the arg # as a parameter? We really just want to abstract away the off-by-one difference between GP registers and SMCCC arguments.

Quoted 3 lines

Macros generally make me uneasy for template functions, but I may be in the vocal minority on this topic :)

I think it's a good idea to have smccc_get_arg(unsigned char index). However, it will cause more code changes because the following functions have been used. Anyway, I think it's still worthy to pass @index to differentiate the argument index. I will change it accordingly in next respin.

smccc_get_arg1()
smccc_get_arg2()
smccc_get_arg3()

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#4 lore raw

PATCH v5 2/22 KVM: arm64: Add SDEI virtualization infrastructure

In reply to Gavin Shan

Software Delegated Exception Interface (SDEI) provides a mechanism for registering and servicing system events. Those system events are high priority events, which must be serviced immediately. It's going to be used by Asynchronous Page Fault (APF) to deliver notification from KVM to guest. It's noted that SDEI is defined by ARM DEN0054C specification.

This introduces SDEI virtualization infrastructure where the SDEI events are registered and manipulated by the guest through hypercall. The SDEI event is delivered to one specific vCPU by KVM once it's raised. This introduces data structures to represent the needed objects to support the feature, which is highlighted as below. As those objects could be migrated between VMs, these data structures are partially exposed to user space.

* kvm_sdei_exposed_event
The exposed events are determined and added by VMM through ioctl interface. Only the exposed events can be registered from the guest.

* kvm_sdei_registered_event
The events that have been registered from the guest through the SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.

* kvm_sdei_vcpu_event
The events that have been delivered to the target vCPU.

* kvm_sdei_vcpu
Used to save the preempted context when the SDEI event is serviced and delivered. After the SDEI event handling is completed, the execution is resumed from the preempted context.

* kvm_sdei_kvm
Place holder for the exposed and registered events.

The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for now. They will be supported in the subsequent patches.

---
 arch/arm64/include/asm/kvm_host.h            |   3 +
 arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
 arch/arm64/include/uapi/asm/kvm.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
 arch/arm64/kvm/Makefile                      |   2 +-
 arch/arm64/kvm/arm.c                         |   8 +
 arch/arm64/kvm/hypercalls.c                  |  21 ++
 arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
 include/uapi/linux/arm_sdei.h                |   2 +
 9 files changed, 523 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/kvm_sdei.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
 create mode 100644 arch/arm64/kvm/sdei.c
arch/arm64/include/asm/kvm_host.h +3 -0
index 031e3a2537fc..5d37e046a458 100644
@@ -113,6 +113,8 @@ struct kvm_arch {
 	/* Interrupt controller */
 	struct vgic_dist	vgic;
 
+	struct kvm_sdei_kvm *sdei;
+
 	/* Mandated version of PSCI */
 	u32 psci_version;
 
@@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
 	 * Anything that is not used directly from assembly code goes
 	 * here.
 	 */
+	struct kvm_sdei_vcpu *sdei;
 
 	/*
 	 * Guest registers we preserve during guest debugging.
arch/arm64/include/asm/kvm_sdei.h new file +171 -0

171 lines not on this page · raw message

arch/arm64/include/uapi/asm/kvm.h +1 -0
index 323e251ed37b..33ee95bc088e 100644
@@ -36,6 +36,7 @@
 #include <linux/types.h>
 #include <asm/ptrace.h>
 #include <asm/sve_context.h>
+#include <asm/kvm_sdei_state.h>
 
 #define __KVM_HAVE_GUEST_DEBUG
 #define __KVM_HAVE_IRQ_LINE
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +72 -0

72 lines not on this page · raw message

arch/arm64/kvm/Makefile +1 -1
index 91861fd8b897..a12903a21d1f 100644
@@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 inject_fault.o va_layout.o handle_exit.o \
 	 guest.o debug.o reset.o sys_regs.o \
 	 vgic-sys-reg-v3.o fpsimd.o pmu.o pkvm.o \
-	 arch_timer.o trng.o\
+	 arch_timer.o trng.o sdei.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/arm.c +8 -0
index 4dca6ffd03d4..96fcae5beee4 100644
@@ -38,6 +38,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_mmu.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_sdei.h>
 #include <asm/sections.h>
 
 #include <kvm/arm_hypercalls.h>
@@ -152,6 +153,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
 	kvm_vgic_early_init(kvm);
 
+	kvm_sdei_init_vm(kvm);
+
 	/* The maximum number of VCPUs is limited by the host's GIC model */
 	kvm->arch.max_vcpus = kvm_arm_default_max_vcpus();
 
@@ -179,6 +182,8 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 
 	kvm_vgic_destroy(kvm);
 
+	kvm_sdei_destroy_vm(kvm);
+
 	kvm_destroy_vcpus(kvm);
 
 	kvm_unshare_hyp(kvm, kvm + 1);
@@ -330,6 +335,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	kvm_arm_pvtime_vcpu_init(&vcpu->arch);
 
+	kvm_sdei_create_vcpu(vcpu);
+
 	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
 
 	err = kvm_vgic_vcpu_init(vcpu);
@@ -351,6 +358,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
 	kvm_timer_vcpu_terminate(vcpu);
 	kvm_pmu_vcpu_destroy(vcpu);
+	kvm_sdei_destroy_vcpu(vcpu);
 
 	kvm_arm_vcpu_destroy(vcpu);
 }
arch/arm64/kvm/hypercalls.c +21 -0
index 202b8c455724..3c20fee72bb4 100644
@@ -5,6 +5,7 @@
 #include <linux/kvm_host.h>
 
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_sdei.h>
 
 #include <kvm/arm_hypercalls.h>
 #include <kvm/arm_psci.h>
@@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 	case ARM_SMCCC_TRNG_RND32:
 	case ARM_SMCCC_TRNG_RND64:
 		return kvm_trng_call(vcpu);
+	case SDEI_1_0_FN_SDEI_VERSION:
+	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+	case SDEI_1_0_FN_SDEI_PE_MASK:
+	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+	case SDEI_1_1_FN_SDEI_FEATURES:
+		return kvm_sdei_hypercall(vcpu);
 	default:
 		return kvm_psci_call(vcpu);
 	}
arch/arm64/kvm/sdei.c new file +244 -0

244 lines not on this page · raw message

include/uapi/linux/arm_sdei.h +2 -0
index af0630ba5437..39bcf6dbbea0 100644
@@ -22,8 +22,10 @@
 #define SDEI_1_0_FN_SDEI_PE_UNMASK			SDEI_1_0_FN(0x0C)
 #define SDEI_1_0_FN_SDEI_INTERRUPT_BIND			SDEI_1_0_FN(0x0D)
 #define SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE		SDEI_1_0_FN(0x0E)
+#define SDEI_1_1_FN_SDEI_EVENT_SIGNAL			SDEI_1_0_FN(0x0F)
 #define SDEI_1_0_FN_SDEI_PRIVATE_RESET			SDEI_1_0_FN(0x11)
 #define SDEI_1_0_FN_SDEI_SHARED_RESET			SDEI_1_0_FN(0x12)
+#define SDEI_1_1_FN_SDEI_FEATURES			SDEI_1_0_FN(0x30)
 
 #define SDEI_VERSION_MAJOR_SHIFT			48
 #define SDEI_VERSION_MAJOR_MASK				0x7fff
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#5 lore raw

Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure

Hi Gavin,

Quoted 5 lines · Gavin Shan

Software Delegated Exception Interface (SDEI) provides a mechanism for registering and servicing system events. Those system events are high priority events, which must be serviced immediately. It's going to be used by Asynchronous Page Fault (APF) to deliver notification from KVM to guest. It's noted that SDEI is defined by ARM DEN0054C specification.

I'm guessing that you're using linked lists for stitching all of this together because the specification provides for 24 bits of event encoding. However, it seems that there will be a finite number of events in KVM. So the APF stuff and a software signaled event.

Given that the number of events in KVM is rather small, would it make more sense to do away with the overhead of linked lists and having the state just represented in a static or allocated array? I think you can cram all of the VM scoped event state into a single structure and hang the implementation off of that.

Quoted 20 lines

This introduces SDEI virtualization infrastructure where the SDEI events are registered and manipulated by the guest through hypercall. The SDEI event is delivered to one specific vCPU by KVM once it's raised. This introduces data structures to represent the needed objects to support the feature, which is highlighted as below. As those objects could be migrated between VMs, these data structures are partially exposed to user space.

* kvm_sdei_exposed_event
The exposed events are determined and added by VMM through ioctl interface. Only the exposed events can be registered from the guest.

* kvm_sdei_registered_event
The events that have been registered from the guest through the SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.

* kvm_sdei_vcpu_event
The events that have been delivered to the target vCPU.

Quoted 7 lines

* kvm_sdei_vcpu
Used to save the preempted context when the SDEI event is serviced and delivered. After the SDEI event handling is completed, the execution is resumed from the preempted context.

* kvm_sdei_kvm
Place holder for the exposed and registered events.

It might be a good idea to expand these a bit and move them into comments on each of the structures.

Quoted 24 lines

The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for now. They will be supported in the subsequent patches.

---
 arch/arm64/include/asm/kvm_host.h            |   3 +
 arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
 arch/arm64/include/uapi/asm/kvm.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
 arch/arm64/kvm/Makefile                      |   2 +-
 arch/arm64/kvm/arm.c                         |   8 +
 arch/arm64/kvm/hypercalls.c                  |  21 ++
 arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
 include/uapi/linux/arm_sdei.h                |   2 +
 9 files changed, 523 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/kvm_sdei.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
 create mode 100644 arch/arm64/kvm/sdei.c
arch/arm64/include/asm/kvm_host.h +0 -0
index 031e3a2537fc..5d37e046a458 100644

arch/arm64/include/asm/kvm_host.h struct kvm_arch { · line 117

Quoted 5 lines
arch/arm64/include/asm/kvm_host.h +2 -0
@@ -113,6 +113,8 @@ struct kvm_arch {
 	/* Interrupt controller */
 	struct vgic_dist	vgic;
 
+	struct kvm_sdei_kvm *sdei;
+

nit: avoid repeating 'kvm'. struct kvm_sdei should be descriptive enough :)

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 344

Quoted 9 lines
arch/arm64/include/asm/kvm_host.h +1 -0
 	/* Mandated version of PSCI */
 	u32 psci_version;
 
@@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
 	 * Anything that is not used directly from assembly code goes
 	 * here.
 	 */
+	struct kvm_sdei_vcpu *sdei;
 

nit: put your scoping tokens at the beginning of a symbol name, so 'struct kvm_vcpu_sdei'.

[...]

Quoted 25 lines
arch/arm64/kvm/hypercalls.c +10 -0
index 202b8c455724..3c20fee72bb4 100644
@@ -5,6 +5,7 @@
 #include <linux/kvm_host.h>
 
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_sdei.h>
 
 #include <kvm/arm_hypercalls.h>
 #include <kvm/arm_psci.h>
@@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 	case ARM_SMCCC_TRNG_RND32:
 	case ARM_SMCCC_TRNG_RND64:
 		return kvm_trng_call(vcpu);
+	case SDEI_1_0_FN_SDEI_VERSION:
+	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_STATUS:

arch/arm64/kvm/hypercalls.c int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) · line 173

Quoted 10 lines
arch/arm64/kvm/hypercalls.c +10 -0
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+	case SDEI_1_0_FN_SDEI_PE_MASK:
+	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+	case SDEI_1_1_FN_SDEI_FEATURES:

Consider only adding switch statements for hypercalls when they're actually implemented.

Additionally, while this isn't directly related to your patch, I do have a gripe about kvm_hvc_call_handler(). It is really ugly that we enumerate the specific hypercalls we support, and otherwise fall through to PSCI.

IMO, a cleaner approach would be to have kvm_hvc_call_handler() simply route a particular service range/service owner to the appropriate handler. We can then terminate individual hypercalls in those handlers, avoiding a catch-all switch such as this one is currently.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#6 lore raw

Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure

Hi Oliver,

Quoted 12 lines · Oliver Upton
Quoted 5 lines · Gavin Shan

Software Delegated Exception Interface (SDEI) provides a mechanism for registering and servicing system events. Those system events are high priority events, which must be serviced immediately. It's going to be used by Asynchronous Page Fault (APF) to deliver notification from KVM to guest. It's noted that SDEI is defined by ARM DEN0054C specification.

I'm guessing that you're using linked lists for stitching all of this together because the specification provides for 24 bits of event encoding. However, it seems that there will be a finite number of events in KVM. So the APF stuff and a software signaled event.

Quoted 6 lines

Given that the number of events in KVM is rather small, would it make more sense to do away with the overhead of linked lists and having the state just represented in a static or allocated array? I think you can cram all of the VM scoped event state into a single structure and hang the implementation off of that.

Yes, the number of events in KVM is small. Including the events for Async PF and the software signaled event, 8 events would be enough currently. In the meanwhile, there are several types of objects for various events. Some of them can be put into static array, while the left might need static array of pointers to avoid the linked list:

struct kvm_sdei_exposed_event/state on struct kvm_arch
size: 24 bytes
static array, 8 entries
struct kvm_sdei_registered_event/state on struct kvm_arch
size: 9KB
static array of pointers, still need allocate them dynamically, 8 entries
struct kvm_sdei_vcpu_event/state on struct kvm_vcpu_arch
size: 16 bytes
static array, 8 entries

Quoted 25 lines
Quoted 25 lines

This introduces SDEI virtualization infrastructure where the SDEI events are registered and manipulated by the guest through hypercall. The SDEI event is delivered to one specific vCPU by KVM once it's raised. This introduces data structures to represent the needed objects to support the feature, which is highlighted as below. As those objects could be migrated between VMs, these data structures are partially exposed to user space.

* kvm_sdei_exposed_event
The exposed events are determined and added by VMM through ioctl interface. Only the exposed events can be registered from the guest.

* kvm_sdei_registered_event
The events that have been registered from the guest through the SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.

* kvm_sdei_vcpu_event
The events that have been delivered to the target vCPU.

* kvm_sdei_vcpu
Used to save the preempted context when the SDEI event is serviced and delivered. After the SDEI event handling is completed, the execution is resumed from the preempted context.

Quoted 6 lines
Quoted 2 lines

* kvm_sdei_kvm
Place holder for the exposed and registered events.

It might be a good idea to expand these a bit and move them into comments on each of the structures.

Sure, I will do in next respin.

Quoted 27 lines
Quoted 27 lines

The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for now. They will be supported in the subsequent patches.

---
  arch/arm64/include/asm/kvm_host.h            |   3 +
  arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
  arch/arm64/include/uapi/asm/kvm.h            |   1 +
  arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
  arch/arm64/kvm/Makefile                      |   2 +-
  arch/arm64/kvm/arm.c                         |   8 +
  arch/arm64/kvm/hypercalls.c                  |  21 ++
  arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
  include/uapi/linux/arm_sdei.h                |   2 +
  9 files changed, 523 insertions(+), 1 deletion(-)

create mode 100644 arch/arm64/include/asm/kvm_sdei.h create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h create mode 100644 arch/arm64/kvm/sdei.c

arch/arm64/include/asm/kvm_host.h +0 -0
index 031e3a2537fc..5d37e046a458 100644
@@ -113,6 +113,8 @@ struct kvm_arch {
  	/* Interrupt controller */
  	struct vgic_dist	vgic;
 
Quoted 6 lines

arch/arm64/include/asm/kvm_host.h struct kvm_arch { · line 117

Quoted 2 lines
arch/arm64/include/asm/kvm_host.h +2 -0
+	struct kvm_sdei_kvm *sdei;
+

nit: avoid repeating 'kvm'. struct kvm_sdei should be descriptive enough :)

Indeed, "struct kvm_sdei" is better :)

Quoted 13 lines

arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 344

Quoted 9 lines
arch/arm64/include/asm/kvm_host.h +1 -0
  	/* Mandated version of PSCI */
  	u32 psci_version;
 
@@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
  	 * Anything that is not used directly from assembly code goes
  	 * here.
  	 */
+	struct kvm_sdei_vcpu *sdei;
 

nit: put your scoping tokens at the beginning of a symbol name, so 'struct kvm_vcpu_sdei'.

Quoted 2 lines

[...]

Yep, "struct kvm_vcpu_sdei" is the one I will have in next respin :)

Quoted 44 lines
Quoted 25 lines
arch/arm64/kvm/hypercalls.c +10 -0
index 202b8c455724..3c20fee72bb4 100644
@@ -5,6 +5,7 @@
  #include <linux/kvm_host.h>
 
  #include <asm/kvm_emulate.h>
+#include <asm/kvm_sdei.h>
 
  #include <kvm/arm_hypercalls.h>
  #include <kvm/arm_psci.h>
@@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
  	case ARM_SMCCC_TRNG_RND32:
  	case ARM_SMCCC_TRNG_RND64:
  		return kvm_trng_call(vcpu);
+	case SDEI_1_0_FN_SDEI_VERSION:
+	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_STATUS:

arch/arm64/kvm/hypercalls.c int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) · line 173

Quoted 10 lines
arch/arm64/kvm/hypercalls.c +10 -0
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+	case SDEI_1_0_FN_SDEI_PE_MASK:
+	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+	case SDEI_1_1_FN_SDEI_FEATURES:

Consider only adding switch statements for hypercalls when they're actually implemented.

Additionally, while this isn't directly related to your patch, I do have a gripe about kvm_hvc_call_handler(). It is really ugly that we enumerate the specific hypercalls we support, and otherwise fall through to PSCI.

Quoted 5 lines

IMO, a cleaner approach would be to have kvm_hvc_call_handler() simply route a particular service range/service owner to the appropriate handler. We can then terminate individual hypercalls in those handlers, avoiding a catch-all switch such as this one is currently.

Yes, I agree. I can have a separate patch as preparatory work to route the range of hypercalls to their owner for further handling. In this way, we can route the range of SDEI hypercalls to its own handler. I will figure it out in next respin.

Thanks,
Gavin

#7 lore raw

Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure

In reply to Gavin Shan

Hi Gavin,

More comments, didn't see exactly how all of these structures are getting used.

On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:

[...]

Quoted 16 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +10 -0
new file mode 100644
index 000000000000..b14844230117
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Definitions of various KVM SDEI event states.
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <[email protected]>
+ */
+
+#ifndef _UAPI__ASM_KVM_SDEI_STATE_H

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 20

Quoted 10 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +10 -0
+#define _UAPI__ASM_KVM_SDEI_STATE_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
+/*
+ * The software signaled event is the default one, which is
+ * defined in v1.1 specification.
+ */
+#define KVM_SDEI_INVALID_EVENT	0xFFFFFFFF

Isn't the constraint that bit 31 must be zero? (DEN 0054C 4.4 "Event number allocation")

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 24

Quoted 4 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +4 -0
+#define KVM_SDEI_DEFAULT_EVENT	0
+
+#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
+#define KVM_SDEI_MAX_EVENTS	128

I would *strongly* recommend against having these limits. I find the vCPU limit especially concerning, because we're making KVM_MAX_VCPUS ABI, which it definitely is not. Anything that deals with a vCPU should be accessed through a vCPU FD (and thus agnostic to the maximum number of vCPUs) to avoid such a complication.

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 33

Quoted 8 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +8 -0
+struct kvm_sdei_exposed_event_state {
+	__u64	num;
+
+	__u8	type;
+	__u8	signaled;
+	__u8	priority;
+	__u8	padding[5];
+	__u64	notifier;

Wait, isn't this a kernel function pointer!?

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 36

Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3 -0
+};
+
+struct kvm_sdei_registered_event_state {

You should fold these fields together with kvm_sdei_exposed_event_state into a single 'kvm_sdei_event' structure:

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 41

Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5 -0
+	__u64	num;
+
+	__u8	route_mode;
+	__u8	padding[3];
+	__u64	route_affinity;

And these shouldn't be UAPI at the VM scope. Each of these properties could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:

Quoted 11 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +11 -0
+	__u64	ep_address[KVM_SDEI_MAX_VCPUS];
+	__u64	ep_arg[KVM_SDEI_MAX_VCPUS];
+	__u64	registered[KVM_SDEI_MAX_VCPUS/64];
+	__u64	enabled[KVM_SDEI_MAX_VCPUS/64];
+	__u64	unregister_pending[KVM_SDEI_MAX_VCPUS/64];
+};
+
+struct kvm_sdei_vcpu_event_state {
+	__u64	num;
+
+	__u32	event_count;

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 62

Quoted 10 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +10 -0
+	__u32	padding;
+};
+
+struct kvm_sdei_vcpu_regs_state {
+	__u64	regs[18];
+	__u64	pc;
+	__u64	pstate;
+};
+
+struct kvm_sdei_vcpu_state {

Same goes here, I strongly recommend you try to expose this through the KVM_{GET,SET}_ONE_REG interface if at all possible since it significantly reduces the UAPI burden, both on KVM to maintain it and VMMs to actually use it.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#8 lore raw

Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure

Hi Oliver,

Quoted 3 lines · Oliver Upton

More comments, didn't see exactly how all of these structures are getting used.

Ok, thanks for your review and comments.

Quoted 25 lines · Gavin Shan

[...]

Quoted 21 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +15 -0
new file mode 100644
index 000000000000..b14844230117
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Definitions of various KVM SDEI event states.
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <[email protected]>
+ */
+
+#ifndef _UAPI__ASM_KVM_SDEI_STATE_H
+#define _UAPI__ASM_KVM_SDEI_STATE_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
Quoted 9 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 20

Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5 -0
+/*
+ * The software signaled event is the default one, which is
+ * defined in v1.1 specification.
+ */
+#define KVM_SDEI_INVALID_EVENT	0xFFFFFFFF

Isn't the constraint that bit 31 must be zero? (DEN 0054C 4.4 "Event number allocation")

Yes, bit 31 of the event number should be zero. So this is invalid event number, used by struct kvm_sdei_vcpu_state::critical_num and normal_num to indicate if there is event being handled on the corresponding vcpu. When those fields are set to KVM_SDEI_INVALID_EVENT, no event is being handled on the vcpu.

Quoted 11 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 24

Quoted 4 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +4 -0
+#define KVM_SDEI_DEFAULT_EVENT	0
+
+#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
+#define KVM_SDEI_MAX_EVENTS	128

I would *strongly* recommend against having these limits. I find the vCPU limit especially concerning, because we're making KVM_MAX_VCPUS ABI, which it definitely is not. Anything that deals with a vCPU should be accessed through a vCPU FD (and thus agnostic to the maximum number of vCPUs) to avoid such a complication.

For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled event. As you suggested on PATCH[15/22], we can't assume its usage. I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h

For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to kvm_sdei.h after static arrays to hold the data structures or their pointers are used, as you suggested early for this patch (PATCH[02/22]).

There are two types of (SDEI) events: shared and private. For the private event, it can be registered independently from the vcpus. It also means the address and argument for the entry points, corresponding to @ep_address and @ep_arg in struct kvm_sdei_registered_event, can be different on the individual vcpus. However, all the registered/enabled states and the entry point address and argument are same on all vcpus for the shared event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to represent both shared and private event.

If the data belongs to particular vcpu should be accessed through the vcpu fd, then we need to split or reorganize the data struct as below.

    /*
     * The events are exposed through ioctl interface or similar
     * mechanism (synthetic system registers?) before they can be
     * registered. struct kvm_sdei_exposed_event instance is reserved
     * from the kvm's static array on receiving the ioctl command
     * from VMM.
     */
    struct kvm_sdei_exposed_event {
        __u32   num;

        __u8    type;
        __u8    signaled;
        __u8    priority;
        __u8    padding;
    };

    /*
     * The struct kvm_sdei_registered_event instance is allocated or
     * reserved from the static array. For the shared event, the instance
     * is linked to kvm, but it will be allocated or reserved from vcpu's
     * static array and linked to the vcpu if it's a private event.
     *
     * The instance is only allocated and reserved upon SDEI_EVENT_REGISTER
     * hypercall.
     */
    struct kvm_sdei_registered_event {
        __u32   num

#define KVM_SDEI_EVENT_STATE_REGISTERED (1 << 0)
#define KVM_SDEI_EVENT_STATE_ENABLE (1 << 1)

#define KVM_SDEI_EVENT_STATE_UNREGISTER_PENDING (1 << 2)
        __u8    state;
        __u8	route_mode;
        __u8    padding[2];
        __u64   route_affinity;
        __u64	ep_address;
        __u64	ep_arg;
        __u64   notifier;
    }
Quoted 11 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 33

Quoted 8 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +8 -0
+struct kvm_sdei_exposed_event_state {
+	__u64	num;
+
+	__u8	type;
+	__u8	signaled;
+	__u8	priority;
+	__u8	padding[5];
+	__u64	notifier;

Wait, isn't this a kernel function pointer!?

Yeah, it is a kernel function pointer, used by Async PF to know if the corresponding event has been handled or not. Async PF can cancel the previously injected event for performance concerns. Either Async PF or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF, SDEI is responsible for its migration.

Quoted 7 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 36

Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3 -0
+};
+
+struct kvm_sdei_registered_event_state {

You should fold these fields together with kvm_sdei_exposed_event_state into a single 'kvm_sdei_event' structure:

@route_mode and @route_affinity can't be configured or modified until the event is registered. Besides, they're only valid to the shared events. For private events, they don't have the routing needs. It means those two fields would be part of struct kvm_sdei_registered_event instead of kvm_sdei_exposed_event.

Quoted 9 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 41

Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5 -0
+	__u64	num;
+
+	__u8	route_mode;
+	__u8	padding[3];
+	__u64	route_affinity;

And these shouldn't be UAPI at the VM scope. Each of these properties could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:

They're accessed through vcpu or kvm fd depending on what type the event is. For the VM-owned shared event, they're accessed through KVM fd. For the vcpu-owned private event, they're accessed through vcpu fd.

I'm not sure if I catch the idea to have a synthetic register and I'm to confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED" system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation defined registers can be adopted, I don't think we need to expose anything through ABI. All the operations and the needed data can be passed through the system registers.

SYS_REG_SDEI_COMMAND
Receives commands like to expose event, register event and change vcpu state etc.
SYS_REG_SDEI_DATA
The needed data corresponding to the received command.

However, I'm not positive that synthetic register can be used here. When Mark Rutland review "PATCH[RFC v1] Async PF support", the implementation defined registers can't be used in a very limited way. That time, a set of implementation defined registers are defined to identify the asynchronous page faults and access to the control data block. However, the idea was rejected. Later on, Marc recommended SDEI for Async PF.

https://www.spinics.net/lists/kvm-arm/msg40315.html

Quoted 18 lines
Quoted 18 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +18 -0
+	__u64	ep_address[KVM_SDEI_MAX_VCPUS];
+	__u64	ep_arg[KVM_SDEI_MAX_VCPUS];
+	__u64	registered[KVM_SDEI_MAX_VCPUS/64];
+	__u64	enabled[KVM_SDEI_MAX_VCPUS/64];
+	__u64	unregister_pending[KVM_SDEI_MAX_VCPUS/64];
+};
+
+struct kvm_sdei_vcpu_event_state {
+	__u64	num;
+
+	__u32	event_count;
+	__u32	padding;
+};
+
+struct kvm_sdei_vcpu_regs_state {
+	__u64	regs[18];
+	__u64	pc;
+	__u64	pstate;
Quoted 9 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 62

Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3 -0
+};
+
+struct kvm_sdei_vcpu_state {

Same goes here, I strongly recommend you try to expose this through the KVM_{GET,SET}_ONE_REG interface if at all possible since it significantly reduces the UAPI burden, both on KVM to maintain it and VMMs to actually use it.

Yeah, it's much convenient to use the implementation defined register here. However, I'm not positive if we can do this. Please see the details I provided above :)

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#9 lore raw

Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure

Quoted 20 lines · Gavin Shan
Quoted 11 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 24

Quoted 4 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +4 -0
+#define KVM_SDEI_DEFAULT_EVENT	0
+
+#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
+#define KVM_SDEI_MAX_EVENTS	128

I would *strongly* recommend against having these limits. I find the vCPU limit especially concerning, because we're making KVM_MAX_VCPUS ABI, which it definitely is not. Anything that deals with a vCPU should be accessed through a vCPU FD (and thus agnostic to the maximum number of vCPUs) to avoid such a complication.

For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled event. As you suggested on PATCH[15/22], we can't assume its usage. I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h

For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to kvm_sdei.h after static arrays to hold the data structures or their pointers are used, as you suggested early for this patch (PATCH[02/22]).

Quoted 8 lines

There are two types of (SDEI) events: shared and private. For the private event, it can be registered independently from the vcpus. It also means the address and argument for the entry points, corresponding to @ep_address and @ep_arg in struct kvm_sdei_registered_event, can be different on the individual vcpus. However, all the registered/enabled states and the entry point address and argument are same on all vcpus for the shared event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to represent both shared and private event.

You're providing a great deal of abstraction around the SDEI specification, but I really am unconvinced that KVM needs that. This series needs to add support for a single SDEI event (event 0) and async PF to follow. Since we are going to support a static set of events under KVM I believe a lot of the complexity in this design should fade away.

Quoted 12 lines
Quoted 11 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 33

Quoted 8 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +8 -0
+struct kvm_sdei_exposed_event_state {
+	__u64	num;
+
+	__u8	type;
+	__u8	signaled;
+	__u8	priority;
+	__u8	padding[5];
+	__u64	notifier;

Wait, isn't this a kernel function pointer!?

Quoted 5 lines

Yeah, it is a kernel function pointer, used by Async PF to know if the corresponding event has been handled or not. Async PF can cancel the previously injected event for performance concerns. Either Async PF or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF, SDEI is responsible for its migration.

But this is a UAPI header, why would we even consider giving userspace a window into the kernel like this? Couldn't userspace crash the kernel by writing whatever it wants to this field, knowing that it will call it as a function pointer?

Security aside, there's no guarantee that a function winds up at the same address between compiler versions or kernel versions.

Overall, I don't think that userspace should have the ability to add arbitrary SDEI events. KVM takes ownership of its own vendor-specific ABI in patch 3/22 by declaring its vendor identifier, so any new events we support must remain within KVM. There is going to be some state that will need to be migrated for KVM's SDEI events, that ought to be surfaced to userspace through the KVM_{GET,SET}_ONE_REG ioctls.

Given that there isn't much userspace involvement to make SDEI work, do you think it would be possible to drop the proposed UAPI from your series and work on enabling software-signaled SDEI events within KVM first? By this I mean a VM running under KVM shouldn't require any ioctls to make it work.

In so doing, we can discover exactly what the mechanics look like in KVM and only then talk about the necessary UAPI to migrate state. One piece of the mechanics that is top of mind which I'd like to see addressed is the use of linked lists and the preallocations that have been made in structures. KVM will know how many events exist at compile time, so we can represent these statically.

Quoted 25 lines
Quoted 7 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 36

Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3 -0
+};
+
+struct kvm_sdei_registered_event_state {

You should fold these fields together with kvm_sdei_exposed_event_state into a single 'kvm_sdei_event' structure:

@route_mode and @route_affinity can't be configured or modified until the event is registered. Besides, they're only valid to the shared events. For private events, they don't have the routing needs. It means those two fields would be part of struct kvm_sdei_registered_event instead of kvm_sdei_exposed_event.

Quoted 9 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 41

Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5 -0
+	__u64	num;
+
+	__u8	route_mode;
+	__u8	padding[3];
+	__u64	route_affinity;

And these shouldn't be UAPI at the VM scope. Each of these properties could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:

Quoted 3 lines

They're accessed through vcpu or kvm fd depending on what type the event is. For the VM-owned shared event, they're accessed through KVM fd. For the vcpu-owned private event, they're accessed through vcpu fd.

Some of the state that you represent in struct kvm_sdei_registered_event_state is really per-vCPU state. Any time that there's data available at a vCPU granularity userspace should access it with a vCPU FD.

Quoted 6 lines

I'm not sure if I catch the idea to have a synthetic register and I'm to confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED" system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation defined registers can be adopted, I don't think we need to expose anything through ABI. All the operations and the needed data can be passed through the system registers.

No, I'm not talking about the guest-facing interface. I'm talking about what gets exposed to userspace to migrate the VM's state. For parts of the guest that do not map to an architectural construct, we've defined the concept of a firmware pseudo-register. What that really means is we expose a register to userspace that is inaccessible from the guest which migrates KVM's nonarchitected state. We are abusing the fact that VMMs will save/restore whatever registers are reported on KVM_GET_REG_LIST to trick it into migrating KVM state, and it has worked quite nicely to avoid adding new ioctls for new features. It also means that older VMMs are capable of utilizing new KVM features, so long as they obey the prescribed rules.

--
Thanks,
Oliver

author
#10 lore raw

Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure

Hi Oliver,

Quoted 30 lines · Oliver Upton
Quoted 20 lines · Gavin Shan
Quoted 11 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 24

Quoted 4 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +4 -0
+#define KVM_SDEI_DEFAULT_EVENT	0
+
+#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
+#define KVM_SDEI_MAX_EVENTS	128

I would *strongly* recommend against having these limits. I find the vCPU limit especially concerning, because we're making KVM_MAX_VCPUS ABI, which it definitely is not. Anything that deals with a vCPU should be accessed through a vCPU FD (and thus agnostic to the maximum number of vCPUs) to avoid such a complication.

For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled event. As you suggested on PATCH[15/22], we can't assume its usage. I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h

For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to kvm_sdei.h after static arrays to hold the data structures or their pointers are used, as you suggested early for this patch (PATCH[02/22]).

Quoted 8 lines

There are two types of (SDEI) events: shared and private. For the private event, it can be registered independently from the vcpus. It also means the address and argument for the entry points, corresponding to @ep_address and @ep_arg in struct kvm_sdei_registered_event, can be different on the individual vcpus. However, all the registered/enabled states and the entry point address and argument are same on all vcpus for the shared event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to represent both shared and private event.

Quoted 6 lines

You're providing a great deal of abstraction around the SDEI specification, but I really am unconvinced that KVM needs that. This series needs to add support for a single SDEI event (event 0) and async PF to follow. Since we are going to support a static set of events under KVM I believe a lot of the complexity in this design should fade away.

Yeah, I think we can drop the functionality to support the shared event since both events are exposed by KVM as private events. Please refer below reply for more details.

Quoted 39 lines
Quoted 12 lines
Quoted 11 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 33

Quoted 8 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +8 -0
+struct kvm_sdei_exposed_event_state {
+	__u64	num;
+
+	__u8	type;
+	__u8	signaled;
+	__u8	priority;
+	__u8	padding[5];
+	__u64	notifier;

Wait, isn't this a kernel function pointer!?

Quoted 5 lines

Yeah, it is a kernel function pointer, used by Async PF to know if the corresponding event has been handled or not. Async PF can cancel the previously injected event for performance concerns. Either Async PF or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF, SDEI is responsible for its migration.

But this is a UAPI header, why would we even consider giving userspace a window into the kernel like this? Couldn't userspace crash the kernel by writing whatever it wants to this field, knowing that it will call it as a function pointer?

Security aside, there's no guarantee that a function winds up at the same address between compiler versions or kernel versions.

Overall, I don't think that userspace should have the ability to add arbitrary SDEI events. KVM takes ownership of its own vendor-specific ABI in patch 3/22 by declaring its vendor identifier, so any new events we support must remain within KVM. There is going to be some state that will need to be migrated for KVM's SDEI events, that ought to be surfaced to userspace through the KVM_{GET,SET}_ONE_REG ioctls.

Given that there isn't much userspace involvement to make SDEI work, do you think it would be possible to drop the proposed UAPI from your series and work on enabling software-signaled SDEI events within KVM first? By this I mean a VM running under KVM shouldn't require any ioctls to make it work.

Quoted 7 lines

In so doing, we can discover exactly what the mechanics look like in KVM and only then talk about the necessary UAPI to migrate state. One piece of the mechanics that is top of mind which I'd like to see addressed is the use of linked lists and the preallocations that have been made in structures. KVM will know how many events exist at compile time, so we can represent these statically.

For @notifier, it is set on the kernel of source VM and migrated to the kernel of destination VM. It's really concerned with security, I will drop this in next respin.

I plan to make the following changes for next revision (v6) as below. Please let me know if there are more things to be covered:

(1) Drop the code to support the shared (SDEI) events since the needed events, either software signaled or Async PF event, are all private. After that, the private event is the only type to be supported. When the shared event becomes unsupported, we can simply return SDEI_NOT_SUPPORTED for hypercall SDEI_EVENT_ROUTING_SET and SDEI_SHARED_RESET.

(2) Drop the code to support migration for now. It means the data structures defined in arch/arm64/include/uapi/asm/kvm_sdei_state.h will be combined to arch/arm64/include/asm/kvm_sdei.h. Besides, all the ioctl commands, including KVM_CAP_ARM_SDEI will be dropped either.

(3) As we discussed before, the linked lists will be replaced by the

    static arrays or preallocation. The data structures will be amended
    like below.

    (3.1) In arch/arm64/include/asm/kvm_sdei.h
 
    struct kvm_sdei_exposed_event {
        unsigned int                     num;
        unsigned char                    type;           /* reserved, should be 'private' */
        unsigned char                    signaled;
        unsigned char                    priority;
    };

    struct kvm_sdei_registered_event {
        struct kvm_sdei_exposed_event    *exposed_event;
        unsigned long                    ep_address;
        unsigned long                    ep_arg;
#define KVM_SDEI_EVENT_STATE_REGISTERED         (1UL << 0)
#define KVM_SDEI_EVENT_STATE_ENABLED            (1UL << 1)
#define KVM_SDEI_EVENT_STATE_UNREGISTER_PENDING (1UL << 2)
        unsigned long                    state;

        unsigned long                    event_count;  /* number of events pending for hanlding */
    };
       
    struct kvm_sdei_context {
        struct kvm_sdei_registered_event *registered_event;
        unsigned long                    regs[18];
        unsigned long                    pc;
        unsigned long                    pstate;
    };

    /* We even needn't a lock */
    struct kvm_sdei_vcpu {
        unsigned char                    masked;
        struct kvm_sdei_registered_event *registered_event;
        struct kvm_sdei_context          context[SDEI_EVENT_PRIORITY_CRITICAL + 1];
    };

    /* struct kvm_sdei won't be existing any more */

    (3.2) In arch/arm64/kvm/sdei.c

    static struct kvm_sdei_exposed_event exposed_events[] = {
        { .num      = SDEI_EVENT_SW_SIGNALED,    /* defined in include/uapi/linux/arm_sdei.h */
          .type     = SDEI_EVENT_TYPE_PRIVATE,
          .signaled = 1,
          .priority = SDEI_EVENT_PRIORITY_NORMAL,
        },
        {
          .num      = SDEI_EVENT_ASYNC_PF,       /* defined in include/asm/kvm_sdei.h */
          .type     = SDEI_EVENT_TYPE_PRIVATE,
          .signaled = 1,
          .priority = SDEI_EVENT_PRIORITY_CRITICAL,
        },
    };

    In kvm_sdei_create_vcpu(), the events are pre-allocated. They will be destroy
    in kvm_sdei_destroy_vcpu().
 
    struct kvm_vcpu_arch::sdei =
           kzalloc(sizeof(struct kvm_sdei_vcpu), GFP_KERNEL_ACCOUNT);
    struct kvm_sdei_vcpu::registered_event =
           kcalloc(sizeof(struct kvm_sdei_exposed_event), ARRAY_SIZE(exposed_events), GFP_KERNEL_ACCOUNT);

    In kvm_sdei_hypercall(), SDEI_NOT_SUPPORTED will be returned on hypercall
    SDEI_EVENT_ROUTING_SET and SDEI_SHARED_RESET.

(4) Stuff to be considered or discussed in future: migration and mechanism. We probably don't want to support the shared event. The function of a event is to notify from host to guest kernel. In this regard, the shared event can be replaced by the private one. With above changes in (1)/(2)/(3), we don't have any VM-scoped properties or states. It means all the properties and states are associated with VCPU. It will help to adopt 'firmware-pseudo' registers and KVM_{GET,SET}_ONE_REG during the migration in the future.

Quoted 25 lines
Quoted 25 lines
Quoted 7 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 36

Quoted 3 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +3 -0
+};
+
+struct kvm_sdei_registered_event_state {

You should fold these fields together with kvm_sdei_exposed_event_state into a single 'kvm_sdei_event' structure:

@route_mode and @route_affinity can't be configured or modified until the event is registered. Besides, they're only valid to the shared events. For private events, they don't have the routing needs. It means those two fields would be part of struct kvm_sdei_registered_event instead of kvm_sdei_exposed_event.

Quoted 9 lines

arch/arm64/include/uapi/asm/kvm_sdei_state.h line 41

Quoted 5 lines
arch/arm64/include/uapi/asm/kvm_sdei_state.h new file +5 -0
+	__u64	num;
+
+	__u8	route_mode;
+	__u8	padding[3];
+	__u64	route_affinity;

And these shouldn't be UAPI at the VM scope. Each of these properties could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:

Quoted 8 lines
Quoted 3 lines

They're accessed through vcpu or kvm fd depending on what type the event is. For the VM-owned shared event, they're accessed through KVM fd. For the vcpu-owned private event, they're accessed through vcpu fd.

Some of the state that you represent in struct kvm_sdei_registered_event_state is really per-vCPU state. Any time that there's data available at a vCPU granularity userspace should access it with a vCPU FD.

Yeah, I used "struct kvm_sdei_registered_event" to represent the shared and private event. For the shared event, they are VM scoped. However, they're VCPU scoped for the private event. As I suggested above, all the states are VCPU scoped when the private event becomes the only supported one.

Quoted 7 lines
Quoted 6 lines

I'm not sure if I catch the idea to have a synthetic register and I'm to confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED" system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation defined registers can be adopted, I don't think we need to expose anything through ABI. All the operations and the needed data can be passed through the system registers.

Quoted 12 lines

No, I'm not talking about the guest-facing interface. I'm talking about what gets exposed to userspace to migrate the VM's state. For parts of the guest that do not map to an architectural construct, we've defined the concept of a firmware pseudo-register. What that really means is we expose a register to userspace that is inaccessible from the guest which migrates KVM's nonarchitected state. We are abusing the fact that VMMs will save/restore whatever registers are reported on KVM_GET_REG_LIST to trick it into migrating KVM state, and it has worked quite nicely to avoid adding new ioctls for new features. It also means that older VMMs are capable of utilizing new KVM features, so long as they obey the prescribed rules.

Thanks for the details about the 'firmware pseudo register'. Oliver, is there exiting one in current KVM implementation? I would like to see how it's being used. It's definitely a good idea. Those non-architectural CPU properties can be mapped and migrated in a natural way. I'm not sure if we had similar mechanism or 'firmware pseudo registers' for the VM scoped properties?

Thanks,
Gavin

author
#11 lore raw

PATCH v5 3/22 KVM: arm64: Support SDEI_VERSION hypercall

In reply to Gavin Shan

This supports SDEI_VERSION hypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.

---
 arch/arm64/kvm/sdei.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/kvm/sdei.c +10 -0
index 8a9b477b8977..5a3a64cd6e84 100644
@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
 	return pending;
 }
 
+static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
+{
+	/* v1.1 and the vendor is KVM */
+	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
+	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
+	       0x4b564d;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -142,6 +150,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 
 	switch (func) {
 	case SDEI_1_0_FN_SDEI_VERSION:
+		ret = hypercall_version(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#12 lore raw

Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall

Quoted 14 lines · Gavin Shan

This supports SDEI_VERSION hypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.

---
 arch/arm64/kvm/sdei.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/kvm/sdei.c +0 -0
index 8a9b477b8977..5a3a64cd6e84 100644

arch/arm64/kvm/sdei.c static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu, · line 126

Quoted 9 lines
arch/arm64/kvm/sdei.c +6 -0
@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
 	return pending;
 }
 
+static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
+{
+	/* v1.1 and the vendor is KVM */
+	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
+	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
+	       0x4b564d;

It looks like the SDEI specification states that the vendor-defined version number is 32 bits. Could we just use one of the ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?

ASCII 'KVM' is neat, but in reality guest software will just throw it in a macro regardless. Might as well use one of the values we've already trained it to use :-)

Also, it would appear that guest discovery of SDEI relies upon KVM reporting a valid SDEI version. IMO, this patch should come at the very end when KVM actually implements SDEI.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#13 lore raw

Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall

Hi Oliver,

Quoted 29 lines · Oliver Upton
Quoted 14 lines · Gavin Shan

This supports SDEI_VERSION hypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.

---
  arch/arm64/kvm/sdei.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
arch/arm64/kvm/sdei.c +0 -0
index 8a9b477b8977..5a3a64cd6e84 100644

arch/arm64/kvm/sdei.c static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu, · line 126

Quoted 9 lines
arch/arm64/kvm/sdei.c +6 -0
@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
  	return pending;
  }
 
+static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
+{
+	/* v1.1 and the vendor is KVM */
+	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
+	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
+	       0x4b564d;

It looks like the SDEI specification states that the vendor-defined version number is 32 bits. Could we just use one of the ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?

Quoted 8 lines

ASCII 'KVM' is neat, but in reality guest software will just throw it in a macro regardless. Might as well use one of the values we've already trained it to use :-)

Also, it would appear that guest discovery of SDEI relies upon KVM reporting a valid SDEI version. IMO, this patch should come at the very end when KVM actually implements SDEI.

Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 if you agree. Its first two characters are "VM" at least.

It's fine to return the version here because the SDEI capability isn't exposed yet. It means no events have been exposed and able to be registered. However, It's also fine to move this patch after the following one:

[PATCH v5 16/22] KVM: arm64: Support SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall

Thanks,
Gavin

#14 lore raw

Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall

Quoted 35 lines · Gavin Shan

Hi Oliver,

Quoted 32 lines · Oliver Upton
Quoted 14 lines · Gavin Shan

This supports SDEI_VERSION hypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.

---
  arch/arm64/kvm/sdei.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
arch/arm64/kvm/sdei.c +0 -0
index 8a9b477b8977..5a3a64cd6e84 100644

arch/arm64/kvm/sdei.c static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu, · line 125

Quoted 8 lines
arch/arm64/kvm/sdei.c +6 -0
@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
  	return pending;
  }
+static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
+{
+	/* v1.1 and the vendor is KVM */
+	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
+	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
+	       0x4b564d;

It looks like the SDEI specification states that the vendor-defined version number is 32 bits. Could we just use one of the ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?

ASCII 'KVM' is neat, but in reality guest software will just throw it in a macro regardless. Might as well use one of the values we've already trained it to use :-)

Quoted 8 lines
Quoted 4 lines

Also, it would appear that guest discovery of SDEI relies upon KVM reporting a valid SDEI version. IMO, this patch should come at the very end when KVM actually implements SDEI.

Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 if you agree. Its first two characters are "VM" at least.

Sounds fine to me. The only other nit I'd say is we should define a macro for it too, something like:

  #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#15 lore raw

Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall

Hi Oliver,

Quoted 39 lines · Oliver Upton
Quoted 38 lines · Gavin Shan
Quoted 36 lines · Oliver Upton
Quoted 14 lines · Gavin Shan

This supports SDEI_VERSION hypercall by returning v1.1, which is the specification version we're following. The vendor is set to 'KVM'.

---
   arch/arm64/kvm/sdei.c | 10 ++++++++++
   1 file changed, 10 insertions(+)
arch/arm64/kvm/sdei.c +0 -0
index 8a9b477b8977..5a3a64cd6e84 100644

arch/arm64/kvm/sdei.c static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu, · line 125

Quoted 8 lines
arch/arm64/kvm/sdei.c +6 -0
@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
   	return pending;
   }
+static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
+{
+	/* v1.1 and the vendor is KVM */
+	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
+	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
+	       0x4b564d;

It looks like the SDEI specification states that the vendor-defined version number is 32 bits. Could we just use one of the ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?

ASCII 'KVM' is neat, but in reality guest software will just throw it in a macro regardless. Might as well use one of the values we've already trained it to use :-)

Also, it would appear that guest discovery of SDEI relies upon KVM reporting a valid SDEI version. IMO, this patch should come at the very end when KVM actually implements SDEI.

Quoted 9 lines
Quoted 3 lines

Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 if you agree. Its first two characters are "VM" at least.

Sounds fine to me. The only other nit I'd say is we should define a macro for it too, something like:

   #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2

Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h. arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by ABI.

Thanks,
Gavin

#16 lore raw

Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall

On Thu, Mar 24, 2022 at 12:07:34PM +0800, Gavin Shan wrote:

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 if you agree. Its first two characters are "VM" at least.

Sounds fine to me. The only other nit I'd say is we should define a macro for it too, something like:

Quoted 7 lines
Quoted 2 lines
   #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2

Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h. arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by ABI.

The argument could definitely be made that our vendor ID should be promoted to UAPI. Even though linux is the only known user of our vendor-specific hypercalls, nothing is stopping other software from using them. Beyond that, these values should really never change anyway.

It isn't a big deal if you add it to internal headers, either, as the only known consumer will be the kernel.

--
Thanks,
Oliver

author
#17 lore raw

Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall

Hi Oliver,

Quoted 19 lines · Oliver Upton

On Thu, Mar 24, 2022 at 12:07:34PM +0800, Gavin Shan wrote:

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Yeah, I was sticky to the pattern of "KVM". However, I think it's good to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 if you agree. Its first two characters are "VM" at least.

Sounds fine to me. The only other nit I'd say is we should define a macro for it too, something like:

Quoted 7 lines
Quoted 2 lines
    #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2

Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h. arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by ABI.

Quoted 8 lines

The argument could definitely be made that our vendor ID should be promoted to UAPI. Even though linux is the only known user of our vendor-specific hypercalls, nothing is stopping other software from using them. Beyond that, these values should really never change anyway.

It isn't a big deal if you add it to internal headers, either, as the only known consumer will be the kernel.

Sure. Lets add it into include/asm/kvm_sdei.h in next respin. We can expose it when it is needed. For now, I do think Linux is the only user.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#18 lore raw

PATCH v5 4/22 KVM: arm64: Support SDEI_EVENT_REGISTER hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_REGISTER hypercall, which is used by guest to register SDEI events. The SDEI event won't be raised until it's registered and enabled explicitly.

Only the exposed events can be registered. For shared event, the registered event instance is created. However, the instance may be not created for the private events.

---
 arch/arm64/kvm/sdei.c | 128 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 128 insertions(+)
arch/arm64/kvm/sdei.c +128 -0
index 5a3a64cd6e84..2458dc666445 100644
@@ -14,6 +14,35 @@
 #include <kvm/arm_hypercalls.h>
 #include <asm/kvm_sdei.h>
 
+static struct kvm_sdei_exposed_event *
+find_exposed_event(struct kvm *kvm, unsigned long num)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+
+	list_for_each_entry(exposed_event, &ksdei->exposed_events, link) {
+		if (exposed_event->state.num == num)
+			return exposed_event;
+	}
+
+	return NULL;
+}
+
+static struct kvm_sdei_registered_event *
+find_registered_event(struct kvm *kvm, unsigned long num)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_registered_event *registered_event;
+
+	list_for_each_entry(registered_event,
+			    &ksdei->registered_events, link) {
+		if (registered_event->state.num == num)
+			return registered_event;
+	}
+
+	return NULL;
+}
+
 static void remove_all_exposed_events(struct kvm *kvm)
 {
 	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
@@ -126,6 +155,103 @@ static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
 	       0x4b564d;
 }
 
+static unsigned long hypercall_register(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long event_ep_address = smccc_get_arg2(vcpu);
+	unsigned long event_ep_arg = smccc_get_arg3(vcpu);
+	unsigned long route_mode = smccc_get_arg4(vcpu);
+	unsigned long route_affinity = smccc_get_arg5(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	if (route_mode != SDEI_EVENT_REGISTER_RM_ANY &&
+	    route_mode != SDEI_EVENT_REGISTER_RM_PE) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * The registered event could have been existing if it's a private
+	 * one. We needn't to create another registered event instance
+	 * in this case.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	if (registered_event) {
+		exposed_event = registered_event->exposed_event;
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (kvm_sdei_is_registered(registered_event, index) ||
+		    kvm_sdei_is_unregister_pending(registered_event, index)) {
+			ret = SDEI_DENIED;
+			goto unlock;
+		}
+
+		registered_event->state.route_mode        = route_mode;
+		registered_event->state.route_affinity    = route_affinity;
+		registered_event->state.ep_address[index] = event_ep_address;
+		registered_event->state.ep_arg[index]     = event_ep_arg;
+		kvm_sdei_set_registered(registered_event, index);
+		goto unlock;
+	}
+
+	/* Check if the exposed event exists */
+	exposed_event = find_exposed_event(kvm, event_num);
+	if (!exposed_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/*
+	 * Check if the count of registered event instances exceeds
+	 * the limit.
+	 */
+	if (ksdei->registered_event_count >= KVM_SDEI_MAX_EVENTS) {
+		ret = SDEI_OUT_OF_RESOURCE;
+		goto unlock;
+	}
+
+	/* Allocate the registered event instance */
+	registered_event = kzalloc(sizeof(*registered_event),
+				   GFP_KERNEL_ACCOUNT);
+	if (!registered_event) {
+		ret = SDEI_OUT_OF_RESOURCE;
+		goto unlock;
+	}
+
+	/* Initialize the registered event state */
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	registered_event->state.num               = event_num;
+	registered_event->state.route_mode        = route_affinity;
+	registered_event->state.route_affinity    = route_affinity;
+	registered_event->state.ep_address[index] = event_ep_address;
+	registered_event->state.ep_arg[index]     = event_ep_arg;
+	registered_event->kvm                     = kvm;
+	registered_event->exposed_event           = exposed_event;
+	registered_event->vcpu_event_count        = 0;
+	kvm_sdei_set_registered(registered_event, index);
+
+	/* Add the registered event instance */
+	ksdei->registered_event_count++;
+	exposed_event->registered_event_count++;
+	list_add_tail(&registered_event->link, &ksdei->registered_events);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -153,6 +279,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_version(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+		ret = hypercall_register(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#19 lore raw

PATCH v5 5/22 KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_{ENABLE, DISABLE} hypercall. After SDEI event is registered by guest, it won't be delivered to the guest until it's enabled. For unregistration pending event, we can't enable or disable it as the registered event is going to be destroyed after current event is handled.

---
 arch/arm64/kvm/sdei.c | 49 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
arch/arm64/kvm/sdei.c +49 -0
index 2458dc666445..4ab58f264992 100644
@@ -252,6 +252,51 @@ static unsigned long hypercall_register(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_enable(struct kvm_vcpu *vcpu, bool enable)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Check if the event is registered and pending for unregistration */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Update the enablement state */
+	if (enable)
+		kvm_sdei_set_enabled(registered_event, index);
+	else
+		kvm_sdei_clear_enabled(registered_event, index);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -282,7 +327,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_register(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+		ret = hypercall_enable(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+		ret = hypercall_enable(vcpu, false);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#20 lore raw

PATCH v5 6/22 KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_CONTEXT hypercall. It's used by the guest to retrieve the registers (R0 - R17) in the preempted context in the SDEI event handler. The preempted context is saved prior to servicing or handling the SDEI event and restored after that.

---
 arch/arm64/kvm/sdei.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
arch/arm64/kvm/sdei.c +30 -0
index 4ab58f264992..4488d3f044f2 100644
@@ -297,6 +297,34 @@ static unsigned long hypercall_enable(struct kvm_vcpu *vcpu, bool enable)
 	return ret;
 }
 
+static unsigned long hypercall_context(struct kvm_vcpu *vcpu)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_regs_state *regs;
+	unsigned long param_id = smccc_get_arg1(vcpu);
+	unsigned long ret = SDEI_SUCCESS;
+
+	spin_lock(&vsdei->lock);
+
+	/* Check if the pending event exists */
+	if (!vsdei->critical_event && !vsdei->normal_event) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Fetch the requested register */
+	regs = vsdei->critical_event ? &vsdei->state.critical_regs :
+				       &vsdei->state.normal_regs;
+	if (param_id < ARRAY_SIZE(regs->regs))
+		ret = regs->regs[param_id];
+	else
+		ret = SDEI_INVALID_PARAMETERS;
+
+unlock:
+	spin_unlock(&vsdei->lock);
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -333,6 +361,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_enable(vcpu, false);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+		ret = hypercall_context(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
 	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#21 lore raw

PATCH v5 7/22 KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_UNREGISTER hypercall. It's used by the guest to unregister SDEI event. The SDEI event won't be raised to the guest after it's unregistered. The SDEI event is disabled automatically on unregistration.

The currently handled events can't be unregistered. We set the unregistration pending state for the event so that it can be unregistered when the event handler is completed by receiving SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall.

---
 arch/arm64/kvm/sdei.c | 133 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)
arch/arm64/kvm/sdei.c +133 -0
index 4488d3f044f2..36eda31e0392 100644
@@ -325,6 +325,135 @@ static unsigned long hypercall_context(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long
+unregister_one_event(struct kvm *kvm, struct kvm_vcpu *vcpu,
+		     struct kvm_sdei_registered_event *registered_event)
+{
+	struct kvm_vcpu *vcpup;
+	struct kvm_sdei_vcpu *vsdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	int index;
+	bool pending;
+	unsigned long i, ret = SDEI_SUCCESS;
+
+	/*
+	 * Cancel all vcpu events that have been queued, but not dispatched
+	 * yet. If the vcpu event has been dispatched, we should mark it
+	 * as pending for unregistration. The unregistration will be executed
+	 * when the event handler is to be completed.
+	 */
+	exposed_event = registered_event->exposed_event;
+	kvm_for_each_vcpu(i, vcpup, kvm) {
+		vsdei = vcpup->arch.sdei;
+		if (!vsdei)
+			continue;
+
+		/*
+		 * The private vcpu events are requested to be unregistered
+		 * on the specific vcpu or all vcpus. @vcpu is used to
+		 * identify the cases. For the shared vcpu events, we need
+		 * to unregister them on all vcpus.
+		 */
+		if (kvm_sdei_is_private(exposed_event->state.type) &&
+		    vcpu && vcpu != vcpup)
+			continue;
+
+		if (registered_event->vcpu_event_count > 0) {
+			spin_lock(&vsdei->lock);
+			pending = remove_all_vcpu_events(vcpup,
+					registered_event->state.num);
+			spin_unlock(&vsdei->lock);
+		} else {
+			pending = false;
+		}
+
+		/*
+		 * For the private event, the pending state for unregistration
+		 * is scattered and we need to update them individually.
+		 * However, that same state for the shared event has to be
+		 * updated at once according to @ret after the iteration is
+		 * done.
+		 */
+		ret = pending ? SDEI_PENDING : ret;
+		if (!kvm_sdei_is_private(exposed_event->state.type))
+			continue;
+
+		index = kvm_sdei_vcpu_index(vcpup, exposed_event);
+		if (pending) {
+			kvm_sdei_set_unregister_pending(registered_event,
+							index);
+		} else {
+			kvm_sdei_clear_enabled(registered_event, index);
+			kvm_sdei_clear_registered(registered_event, index);
+		}
+	}
+
+	/*
+	 * Update the pending state for unregistration for the shared event
+	 * at once.
+	 */
+	if (kvm_sdei_is_shared(exposed_event->state.type)) {
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (ret == SDEI_PENDING) {
+			kvm_sdei_set_unregister_pending(registered_event,
+							index);
+		} else {
+			kvm_sdei_clear_enabled(registered_event, index);
+			kvm_sdei_clear_registered(registered_event, index);
+		}
+	}
+
+	/* Destroy the registered event instance if needed */
+	if (kvm_sdei_none_registered(registered_event))
+		remove_one_registered_event(kvm, registered_event);
+
+	return ret;
+}
+
+static unsigned long hypercall_unregister(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num))
+		return SDEI_INVALID_PARAMETERS;
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Check if the event has been registered */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Check if the event has been pending for unregistration */
+	if (kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_PENDING;
+		goto unlock;
+	}
+
+	ret = unregister_one_event(kvm, vcpu, registered_event);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -365,7 +494,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+		ret = SDEI_NOT_SUPPORTED;
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+		ret = hypercall_unregister(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#22 lore raw

PATCH v5 8/22 KVM: arm64: Support SDEI_EVENT_STATUS hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_STATUS hypercall. It's used by the guest to retrieve the status about the specified SDEI event. A bitmap is returned to indicate the corresponding status, including registration, enablement and delivery state.

---
 arch/arm64/kvm/sdei.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
arch/arm64/kvm/sdei.c +42 -0
index 36eda31e0392..5c43c8912ea1 100644
@@ -454,6 +454,46 @@ static unsigned long hypercall_unregister(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_status(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = 0;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * Check if the registered event exists. None of the flags
+	 * will be set if it doesn't exist.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event)
+		goto unlock;
+
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (kvm_sdei_is_registered(registered_event, index))
+		ret |= (1UL << SDEI_EVENT_STATUS_REGISTERED);
+	if (kvm_sdei_is_enabled(registered_event, index))
+		ret |= (1UL << SDEI_EVENT_STATUS_ENABLED);
+	if (registered_event->vcpu_event_count > 0)
+		ret |= (1UL << SDEI_EVENT_STATUS_RUNNING);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -500,6 +540,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_unregister(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
+		ret = hypercall_status(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
 	case SDEI_1_0_FN_SDEI_PE_MASK:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#23 lore raw

PATCH v5 9/22 KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_GET_INFO hypercall. It's used by the guest to retrieve various information about the exposed or registered event, including type, signaled, routing mode and affinity. The routing mode and affinity information is only valid to the shared and registered event.

---
 arch/arm64/kvm/sdei.c | 73 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
arch/arm64/kvm/sdei.c +73 -0
index 5c43c8912ea1..4f26e5f70bff 100644
@@ -494,6 +494,77 @@ static unsigned long hypercall_status(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_info(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	struct kvm_sdei_registered_event *registered_event = NULL;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long event_info = smccc_get_arg2(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * Retrieve the information from the registered event if it exists.
+	 * Otherwise, we turn into the exposed event if needed.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	exposed_event = registered_event ? registered_event->exposed_event :
+					   find_exposed_event(kvm, event_num);
+	if (!exposed_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Retrieve the requested information */
+	switch (event_info) {
+	case SDEI_EVENT_INFO_EV_TYPE:
+		ret = exposed_event->state.type;
+		break;
+	case SDEI_EVENT_INFO_EV_SIGNALED:
+		ret = exposed_event->state.signaled;
+		break;
+	case SDEI_EVENT_INFO_EV_PRIORITY:
+		ret = exposed_event->state.priority;
+		break;
+	case SDEI_EVENT_INFO_EV_ROUTING_MODE:
+	case SDEI_EVENT_INFO_EV_ROUTING_AFF:
+		if (!kvm_sdei_is_shared(exposed_event->state.type)) {
+			ret = SDEI_INVALID_PARAMETERS;
+			break;
+		}
+
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (!registered_event ||
+		    !kvm_sdei_is_registered(registered_event, index)) {
+			ret = SDEI_DENIED;
+			break;
+		}
+
+		if (event_info == SDEI_EVENT_INFO_EV_ROUTING_MODE)
+			ret = registered_event->state.route_mode;
+		else
+			ret = registered_event->state.route_affinity;
+
+		break;
+	default:
+		ret = SDEI_INVALID_PARAMETERS;
+	}
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -543,6 +614,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_status(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+		ret = hypercall_info(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
 	case SDEI_1_0_FN_SDEI_PE_MASK:
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#24 lore raw

PATCH v5 10/22 KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_ROUTING_SET hypercall. It's used by the guest to set route mode and affinity for the shared and registered events. The request to configure the routing mode and affinity for the private events are disallowed. Besides, It's not allowed to do when the corresponding vCPU events are existing.

---
 arch/arm64/kvm/sdei.c | 62 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)
arch/arm64/kvm/sdei.c +62 -0
index 4f26e5f70bff..db82ea441eae 100644
@@ -565,6 +565,66 @@ static unsigned long hypercall_info(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_route(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long route_mode = smccc_get_arg2(vcpu);
+	unsigned long route_affinity = smccc_get_arg3(vcpu);
+	int index = 0;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	/*
+	 * FIXME: The affinity should be verified when it's supported. We
+	 * accept anything for now.
+	 */
+	if (route_mode != SDEI_EVENT_REGISTER_RM_ANY &&
+	    route_mode != SDEI_EVENT_REGISTER_RM_PE) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Check the registered event is a shared one */
+	exposed_event = registered_event->exposed_event;
+	if (!kvm_sdei_is_shared(exposed_event->state.type)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    kvm_sdei_is_enabled(registered_event, index)     ||
+	    registered_event->vcpu_event_count > 0) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Update the registered event state */
+	registered_event->state.route_mode     = route_mode;
+	registered_event->state.route_affinity = route_affinity;
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -617,6 +677,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_info(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+		ret = hypercall_route(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_PE_MASK:
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#25 lore raw

PATCH v5 11/22 KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall

In reply to Gavin Shan

This supports SDEI_PE_{MASK, UNMASK} hypercall. They are used by the guest to stop or start receiving SDEI event on the specified vCPU.

---
 arch/arm64/kvm/sdei.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/arm64/kvm/sdei.c +16 -0
index db82ea441eae..b2a916724cfa 100644
@@ -625,6 +625,18 @@ static unsigned long hypercall_route(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	unsigned long ret = SDEI_SUCCESS;
+
+	spin_lock(&vsdei->lock);
+	vsdei->state.masked = mask ? 1 : 0;
+	spin_unlock(&vsdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -680,7 +692,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_route(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_PE_MASK:
+		ret = hypercall_mask(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+		ret = hypercall_mask(vcpu, false);
+		break;
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
 	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#26 lore raw

PATCH v5 12/22 KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET

In reply to Gavin Shan

This supports SDEI_{PRIVATE, SHARED}_RESET. They are used by the guest to reset the private events on the calling vCPU or the shared events on all vCPUs.

---
 arch/arm64/kvm/sdei.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/arm64/kvm/sdei.c +31 -0
index b2a916724cfa..0dec35a0eed1 100644
@@ -637,6 +637,31 @@ static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)
 	return ret;
 }
 
+static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event, *tmp;
+	unsigned long r, ret = SDEI_SUCCESS;
+
+	spin_lock(&ksdei->lock);
+
+	list_for_each_entry_safe(registered_event, tmp,
+				 &ksdei->registered_events, link) {
+		exposed_event = registered_event->exposed_event;
+		if (private ^ kvm_sdei_is_shared(exposed_event->state.type))
+			continue;
+
+		r = unregister_one_event(kvm, NULL, registered_event);
+		ret = (r == SDEI_SUCCESS) ? ret : r;
+	}
+
+	spin_unlock(&ksdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -700,8 +725,14 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
 	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+		ret = SDEI_NOT_SUPPORTED;
+		break;
 	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+		ret = hypercall_reset(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+		ret = hypercall_reset(vcpu, false);
+		break;
 	case SDEI_1_1_FN_SDEI_FEATURES:
 	default:
 		ret = SDEI_NOT_SUPPORTED;
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#27 lore raw

PATCH v5 13/22 KVM: arm64: Support SDEI_FEATURES hypercall

In reply to Gavin Shan

This supports SDEI_FEATURES hypercall. It's used by the guest to retrieve the supported features, which are number of binding slots and relative mode for the event handler. Currently, none of them is supported.

---
 arch/arm64/kvm/sdei.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/arm64/kvm/sdei.c +16 -0
index 0dec35a0eed1..1e0ca9022eaa 100644
@@ -662,6 +662,20 @@ static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
 	return ret;
 }
 
+static unsigned long hypercall_features(struct kvm_vcpu *vcpu)
+{
+	unsigned long feature = smccc_get_arg1(vcpu);
+
+	switch (feature) {
+	case 0: /* BIND_SLOTS */
+		return 0;
+	case 1: /* RELATIVE_MODE */
+		return 0;
+	}
+
+	return SDEI_INVALID_PARAMETERS;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -734,6 +748,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_reset(vcpu, false);
 		break;
 	case SDEI_1_1_FN_SDEI_FEATURES:
+		ret = hypercall_features(vcpu);
+		break;
 	default:
 		ret = SDEI_NOT_SUPPORTED;
 	}
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#28 lore raw

PATCH v5 14/22 KVM: arm64: Support SDEI event injection, delivery and cancellation

In reply to Gavin Shan

This supports SDEI event injection, delivery and cancellation. The SDEI event is injected by kvm_sdei_inject_event(). The injected event can be cancelled by kvm_sdei_cancel_event() before it's delivered and handled.

KVM_REQ_SDEI request becomes pending once the SDEI event is injected and kvm_sdei_deliver_event() is called to accommodate the request. The injected SDEI event is delivered and handled in this way. The context for execution is switched like below:

* x0 - x17 are saved. All of them are cleared except the following registers:

x0: SDEI event number
x1: user argument associated with the SDEI event
x2: PC of the interrupted or preempted context
x3: PSTATE of the interrupted or preempted context

* PC is set to the handler of the SDEI event, which was provided during its registration. PSTATE is modified according to the SDEI specification.

* The SDEI event with normal priority can be preempted by that with critical priority. However, no one can preempt the SDEI event with critical event.

---
 arch/arm64/include/asm/kvm_host.h |   1 +
 arch/arm64/include/asm/kvm_sdei.h |   4 +
 arch/arm64/kvm/arm.c              |   3 +
 arch/arm64/kvm/sdei.c             | 284 ++++++++++++++++++++++++++++++
 4 files changed, 292 insertions(+)
arch/arm64/include/asm/kvm_host.h +1 -0
index 5d37e046a458..e2762d08ab1c 100644
@@ -46,6 +46,7 @@
 #define KVM_REQ_RECORD_STEAL	KVM_ARCH_REQ(3)
 #define KVM_REQ_RELOAD_GICv4	KVM_ARCH_REQ(4)
 #define KVM_REQ_RELOAD_PMU	KVM_ARCH_REQ(5)
+#define KVM_REQ_SDEI		KVM_ARCH_REQ(6)
 
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
 				     KVM_DIRTY_LOG_INITIALLY_SET)
arch/arm64/include/asm/kvm_sdei.h +4 -0
index 6f58a846d05c..54c730acd298 100644
@@ -165,6 +165,10 @@ KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)
 void kvm_sdei_init_vm(struct kvm *kvm);
 void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu);
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu);
+int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
+			  unsigned long num, bool immediate);
+int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
+void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
arch/arm64/kvm/arm.c +3 -0
index 96fcae5beee4..00c136a6e8df 100644
@@ -734,6 +734,9 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
 		if (kvm_check_request(KVM_REQ_VCPU_RESET, vcpu))
 			kvm_reset_vcpu(vcpu);
 
+		if (kvm_check_request(KVM_REQ_SDEI, vcpu))
+			kvm_sdei_deliver_event(vcpu);
+
 		/*
 		 * Clear IRQ_PENDING requests that were made to guarantee
 		 * that a VCPU sees new virtual interrupts.
arch/arm64/kvm/sdei.c +284 -0

302 lines not on this page · raw message

Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#29 lore raw

PATCH v5 15/22 KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest to inject SDEI event, whose number must be zero to the specified vCPU. As the routing mode and affinity isn't supported yet, the calling vCPU is assumed to be the target.

The SDEI event 0x0 is a private one, with normal priority. It's usually used for testing.

---
 arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)
arch/arm64/kvm/sdei.c +63 -1
index a24270378305..ba2ca65c871b 100644
@@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	/* @event_num must be zero */
+	if (!kvm_sdei_is_default(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_kvm;
+	}
+
+	/*
+	 * Check if the event has been registered and enabled. The
+	 * @target_pe parameter isn't checked for now and the event
+	 * is assumed to injected the current vcpu. We should fix it
+	 * when the routing mode and affinity are supported.
+	 */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    !kvm_sdei_is_enabled(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_kvm;
+	}
+
+	/* Check if the vcpu has been masked off */
+	spin_lock(&vsdei->lock);
+	if (vsdei->state.masked) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_vcpu;
+	}
+
+	/* Inject the event */
+	if (do_inject_event(vcpu, registered_event, false))
+		ret = SDEI_INVALID_PARAMETERS;
+
+unlock_vcpu:
+	spin_unlock(&vsdei->lock);
+unlock_kvm:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -827,9 +887,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		break;
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
-	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
 		ret = SDEI_NOT_SUPPORTED;
 		break;
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+		ret = hypercall_signal(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
 		ret = hypercall_reset(vcpu, true);
 		break;
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#30 lore raw

Re: [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall

Hi Gavin,

Quoted 7 lines · Gavin Shan

This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest to inject SDEI event, whose number must be zero to the specified vCPU. As the routing mode and affinity isn't supported yet, the calling vCPU is assumed to be the target.

The SDEI event 0x0 is a private one, with normal priority. It's usually used for testing.

I don't know if that is actually the case. One real use that immediately comes to mind is doing an NMI on a wedged CPU. KVM probably shouldn't glean at how the guest may use a particular call, so at most we should just point at the spec and state that event 0 is for software signaled events.

Quoted 16 lines
---
 arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)
arch/arm64/kvm/sdei.c +3 -0
index a24270378305..ba2ca65c871b 100644
@@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;

arch/arm64/kvm/sdei.c static int do_inject_event(struct kvm_vcpu *vcpu, · line 741

Quoted 10 lines
arch/arm64/kvm/sdei.c +10 -0
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	/* @event_num must be zero */
+	if (!kvm_sdei_is_default(event_num)) {

0 isn't KVM's default event. I'd argue KVM doesn't have a default event to begin with. This has a precise definition coming from the spec. In fact, 'KVM_SDEI_DEFAULT_EVENT' should probably be eliminated, and any missing SDEI definitions should be added to include/uapi/linux/arm_sdei.h.

That goes for any values coming from the specification. KVM's implementation details belong in a KVM header :)

--
Thanks,
Oliver

author
#31 lore raw

Re: [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall

Hi Oliver,

Quoted 6 lines · Oliver Upton
Quoted 5 lines · Gavin Shan

This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest to inject SDEI event, whose number must be zero to the specified vCPU. As the routing mode and affinity isn't supported yet, the calling vCPU is assumed to be the target.

Quoted 9 lines
Quoted 2 lines

The SDEI event 0x0 is a private one, with normal priority. It's usually used for testing.

I don't know if that is actually the case. One real use that immediately comes to mind is doing an NMI on a wedged CPU. KVM probably shouldn't glean at how the guest may use a particular call, so at most we should just point at the spec and state that event 0 is for software signaled events.

Yes, I agree. I will amend the change log in next respin, to make it clear. As you said, it really depend on how the event is used by the guest. We can't assume its usage.

Quoted 32 lines
Quoted 16 lines
---
  arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 63 insertions(+), 1 deletion(-)
arch/arm64/kvm/sdei.c +3 -0
index a24270378305..ba2ca65c871b 100644
@@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
  	return 0;
  }
 
+static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;

arch/arm64/kvm/sdei.c static int do_inject_event(struct kvm_vcpu *vcpu, · line 741

Quoted 10 lines
arch/arm64/kvm/sdei.c +10 -0
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	/* @event_num must be zero */
+	if (!kvm_sdei_is_default(event_num)) {

0 isn't KVM's default event. I'd argue KVM doesn't have a default event to begin with. This has a precise definition coming from the spec. In fact, 'KVM_SDEI_DEFAULT_EVENT' should probably be eliminated, and any missing SDEI definitions should be added to include/uapi/linux/arm_sdei.h.

Quoted 3 lines

That goes for any values coming from the specification. KVM's implementation details belong in a KVM header :)

Indeed. include/uapi/linux/arm_sdei.h is the right place to define the software signaled event number. The function to validate/check on it is still part of kvm_sdei.h, but its name would be something like kvm_sdei_is_sw_signaled() or kvm_sdei_is_signaled_event().

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#32 lore raw

PATCH v5 16/22 KVM: arm64: Support SDEI_EVENT_{COMPLETE,COMPLETE_AND_RESUME} hypercall

In reply to Gavin Shan

This supports SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall. They are used by the guest to notify the completion of the SDEI event in the handler. The executing context or registers are modified according to the SDEI specification like below:

* x0 - x17, PC and PState are restored to what values we had in the interrupted or preempted context.

* If it's SDEI_EVENT_COMPLETE_AND_RESUME hypercall, IRQ exception is injected.

---
 arch/arm64/include/asm/kvm_emulate.h |  1 +
 arch/arm64/include/asm/kvm_host.h    |  1 +
 arch/arm64/kvm/inject_fault.c        | 29 +++++++++++
 arch/arm64/kvm/sdei.c                | 76 +++++++++++++++++++++++++++-
 4 files changed, 106 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_emulate.h +1 -0
index d62405ce3e6d..ca9de9f24923 100644
@@ -37,6 +37,7 @@ bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
 void kvm_skip_instr32(struct kvm_vcpu *vcpu);
 
 void kvm_inject_undefined(struct kvm_vcpu *vcpu);
+void kvm_inject_irq(struct kvm_vcpu *vcpu);
 void kvm_inject_vabt(struct kvm_vcpu *vcpu);
 void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
 void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
arch/arm64/include/asm/kvm_host.h +1 -0
index e2762d08ab1c..282913e1afb0 100644
@@ -428,6 +428,7 @@ struct kvm_vcpu_arch {
 #define KVM_ARM64_EXCEPT_AA32_UND	(0 << 9)
 #define KVM_ARM64_EXCEPT_AA32_IABT	(1 << 9)
 #define KVM_ARM64_EXCEPT_AA32_DABT	(2 << 9)
+#define KVM_ARM64_EXCEPT_AA32_IRQ	(3 << 9)
 /* For AArch64: */
 #define KVM_ARM64_EXCEPT_AA64_ELx_SYNC	(0 << 9)
 #define KVM_ARM64_EXCEPT_AA64_ELx_IRQ	(1 << 9)
arch/arm64/kvm/inject_fault.c +29 -0
index b47df73e98d7..c8a8791bdf28 100644
@@ -66,6 +66,13 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
 	vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
 }
 
+static void inject_irq64(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1     |
+			     KVM_ARM64_EXCEPT_AA64_ELx_IRQ |
+			     KVM_ARM64_PENDING_EXCEPTION);
+}
+
 #define DFSR_FSC_EXTABT_LPAE	0x10
 #define DFSR_FSC_EXTABT_nLPAE	0x08
 #define DFSR_LPAE		BIT(9)
@@ -77,6 +84,12 @@ static void inject_undef32(struct kvm_vcpu *vcpu)
 			     KVM_ARM64_PENDING_EXCEPTION);
 }
 
+static void inject_irq32(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA32_IRQ |
+			     KVM_ARM64_PENDING_EXCEPTION);
+}
+
 /*
  * Modelled after TakeDataAbortException() and TakePrefetchAbortException
  * pseudocode.
@@ -160,6 +173,22 @@ void kvm_inject_undefined(struct kvm_vcpu *vcpu)
 		inject_undef64(vcpu);
 }
 
+/**
+ * kvm_inject_irq - inject an IRQ into the guest
+ * @vcpu: The vCPU in which to inject IRQ
+ *
+ * Inject IRQs to the target vCPU. It is assumed that this code is
+ * called from the VCPU thread and that the VCPU therefore is not
+ * currently executing guest code.
+ */
+void kvm_inject_irq(struct kvm_vcpu *vcpu)
+{
+	if (vcpu_el1_is_32bit(vcpu))
+		inject_irq32(vcpu);
+	else
+		inject_irq64(vcpu);
+}
+
 void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 esr)
 {
 	vcpu_set_vsesr(vcpu, esr & ESR_ELx_ISS_MASK);
arch/arm64/kvm/sdei.c +75 -1

89 lines not on this page · raw message

Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#33 lore raw

PATCH v5 17/22 KVM: arm64: Support SDEI event notifier

In reply to Gavin Shan

The owner of the SDEI event may want to know if the injected event has been delivered and handled. For example, Async PF cancels the injected SDEI event at early stage of the work, driven by the event has been completed asynchronously.

This supports SDEI event state updating by introducing notifier mechanism. It's notable the notifier (handler) should be capable of migration.

---
 arch/arm64/include/asm/kvm_sdei.h | 10 +++++++
 arch/arm64/kvm/sdei.c             | 45 +++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +10 -0
index 54c730acd298..2480ec0e9824 100644
@@ -18,6 +18,14 @@
 
 struct kvm_vcpu;
 
+typedef void (*kvm_sdei_notifier)(struct kvm_vcpu *vcpu,
+				  unsigned long num,
+				  unsigned int state);
+enum {
+	KVM_SDEI_NOTIFY_DELIVERED,
+	KVM_SDEI_NOTIFY_COMPLETED,
+};
+
 struct kvm_sdei_exposed_event {
 	struct kvm_sdei_exposed_event_state	state;
 	struct kvm				*kvm;
@@ -165,6 +173,8 @@ KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)
 void kvm_sdei_init_vm(struct kvm *kvm);
 void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu);
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu);
+int kvm_sdei_register_event_notifier(struct kvm *kvm, unsigned long num,
+				     kvm_sdei_notifier notifier);
 int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 			  unsigned long num, bool immediate);
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
arch/arm64/kvm/sdei.c +45 -0

87 lines not on this page · raw message

Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#34 lore raw

PATCH v5 18/22 KVM: arm64: Support SDEI ioctl commands on VM

In reply to Gavin Shan

This supports ioctl commands on VM to manage the various objects. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:

* KVM_SDEI_CMD_GET_VERSION
Retrieve the version of current implementation. It's different from the version of the followed SDEI specification. This version is used to indicates what functionalities documented in the SDEI specification have been supported or not supported.

* KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
Return the total count of exposed events.

* KVM_SDEI_CMD_GET_EXPOSED_EVENT
* KVM_SDEI_CMD_SET_EXPOSED_EVENT
Get or set exposed event

* KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
Return the total count of registered events.

* KVM_SDEI_CMD_GET_REGISTERED_EVENT
* KVM_SDEI_CMD_SET_REGISTERED_EVENT
Get or set registered event.

---
 arch/arm64/include/asm/kvm_sdei.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  20 ++
 arch/arm64/kvm/arm.c                         |   3 +
 arch/arm64/kvm/sdei.c                        | 302 +++++++++++++++++++
 include/uapi/linux/kvm.h                     |   3 +
 5 files changed, 329 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +1 -0
index 2480ec0e9824..64f00cc79162 100644
@@ -179,6 +179,7 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 			  unsigned long num, bool immediate);
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
 void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
+long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
arch/arm64/include/uapi/asm/kvm_sdei_state.h +20 -0
index b14844230117..2bd6d11627bc 100644
@@ -68,5 +68,25 @@ struct kvm_sdei_vcpu_state {
 	struct kvm_sdei_vcpu_regs_state	normal_regs;
 };
 
+#define KVM_SDEI_CMD_GET_VERSION		0
+#define KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT	1
+#define KVM_SDEI_CMD_GET_EXPOSED_EVENT		2
+#define KVM_SDEI_CMD_SET_EXPOSED_EVENT		3
+#define KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT	4
+#define KVM_SDEI_CMD_GET_REGISTERED_EVENT	5
+#define KVM_SDEI_CMD_SET_REGISTERED_EVENT	6
+
+struct kvm_sdei_cmd {
+	__u32                                           cmd;
+	union {
+		__u32                                   version;
+		__u32                                   count;
+	};
+	union {
+		struct kvm_sdei_exposed_event_state     *exposed_event_state;
+		struct kvm_sdei_registered_event_state  *registered_event_state;
+	};
+};
+
 #endif /* !__ASSEMBLY__ */
 #endif /* _UAPI__ASM_KVM_SDEI_STATE_H */
arch/arm64/kvm/arm.c +3 -0
index 00c136a6e8df..ebfd504a1c08 100644
@@ -1465,6 +1465,9 @@ long kvm_arch_vm_ioctl(struct file *filp,
 			return -EFAULT;
 		return kvm_vm_ioctl_mte_copy_tags(kvm, &copy_tags);
 	}
+	case KVM_ARM_SDEI_COMMAND: {
+		return kvm_sdei_vm_ioctl(kvm, arg);
+	}
 	default:
 		return -EINVAL;
 	}
arch/arm64/kvm/sdei.c +302 -0

308 lines not on this page · raw message

include/uapi/linux/kvm.h +3 -0
index 507ee1f2aa96..2d11c909ec42 100644
@@ -2049,4 +2049,7 @@ struct kvm_stats_desc {
 /* Available with KVM_CAP_XSAVE2 */
 #define KVM_GET_XSAVE2		  _IOR(KVMIO,  0xcf, struct kvm_xsave)
 
+/* Available with KVM_CAP_ARM_SDEI */
+#define KVM_ARM_SDEI_COMMAND	_IOWR(KVMIO, 0xd0, struct kvm_sdei_cmd)
+
 #endif /* __LINUX_KVM_H */
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#35 lore raw

Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM

Quoted 9 lines · Gavin Shan

This supports ioctl commands on VM to manage the various objects. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:

* KVM_SDEI_CMD_GET_VERSION
Retrieve the version of current implementation. It's different from the version of the followed SDEI specification. This version is used to indicates what functionalities documented in the SDEI specification have been supported or not supported.

Don't we need a way to set the version as well? KVM is very much responsible for upholding ABI of older specs. So, if a VMM and guest expect SDEI v1.1, we can't just forcibly raise it to something else during a migration.

The PSCI implementation is a great example of how KVM has grown its implementation in line with a specification, all the while preserving backwards compatibility.

Quoted 7 lines

* KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
Return the total count of exposed events.

* KVM_SDEI_CMD_GET_EXPOSED_EVENT
* KVM_SDEI_CMD_SET_EXPOSED_EVENT
Get or set exposed event

Quoted 6 lines

* KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
Return the total count of registered events.

* KVM_SDEI_CMD_GET_REGISTERED_EVENT
* KVM_SDEI_CMD_SET_REGISTERED_EVENT
Get or set registered event.

Any new UAPI needs to be documented in Documentation/virt/kvm/api.rst

Additionally, we desperately need a better, generic way to save/restore VM scoped state. IMO, we should only be adding ioctls if we are affording userspace a meaningful interface. Every save/restore pair of ioctls winds up wasting precious ioctl numbers and requires userspace take a change to read/write an otherwise opaque value.

Marc had made some suggestions in this area already that Raghavendra experimented with [1], and I think its time to meaningfully consider our options. Basically, KVM_GET_REG_LIST needs to convey whether a particular register is VM or vCPU state. We only need to save/restore a VM state register once. That way, userspace doesn't have to care about the underlying data and the next piece of VM state that comes along doesn't require an ioctl nr nor VMM participation.

[1]: http://lore.kernel.org/r/[email protected]

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#36 lore raw

Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM

Hi Oliver,

Quoted 11 lines · Oliver Upton
Quoted 9 lines · Gavin Shan

This supports ioctl commands on VM to manage the various objects. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:

* KVM_SDEI_CMD_GET_VERSION
Retrieve the version of current implementation. It's different from the version of the followed SDEI specification. This version is used to indicates what functionalities documented in the SDEI specification have been supported or not supported.

Quoted 9 lines

Don't we need a way to set the version as well? KVM is very much responsible for upholding ABI of older specs. So, if a VMM and guest expect SDEI v1.1, we can't just forcibly raise it to something else during a migration.

The PSCI implementation is a great example of how KVM has grown its implementation in line with a specification, all the while preserving backwards compatibility.

The only information feed by VMM is the exposed events. The events can't be registered from guest kernel, and raised from host to guest kernel until it's exposed by VMM. Besides, the exposed events will be defined staticly in host/KVM as we discussed on PATCH[02/22]. We also discussed to eliminate those ioctl commands. So I think we needn't to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a concern to host itself if the migration can be done through the firmware pseudo system registers since the migration compatibility is the only concern to VMM (QEMU).

Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable it.

Quoted 30 lines
Quoted 7 lines

* KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
Return the total count of exposed events.

* KVM_SDEI_CMD_GET_EXPOSED_EVENT
* KVM_SDEI_CMD_SET_EXPOSED_EVENT
Get or set exposed event

Quoted 6 lines

* KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT Return the total count of registered events.

* KVM_SDEI_CMD_GET_REGISTERED_EVENT
* KVM_SDEI_CMD_SET_REGISTERED_EVENT
Get or set registered event.

Any new UAPI needs to be documented in Documentation/virt/kvm/api.rst

Additionally, we desperately need a better, generic way to save/restore VM scoped state. IMO, we should only be adding ioctls if we are affording userspace a meaningful interface. Every save/restore pair of ioctls winds up wasting precious ioctl numbers and requires userspace take a change to read/write an otherwise opaque value.

Marc had made some suggestions in this area already that Raghavendra experimented with [1], and I think its time to meaningfully consider our options. Basically, KVM_GET_REG_LIST needs to convey whether a particular register is VM or vCPU state. We only need to save/restore a VM state register once. That way, userspace doesn't have to care about the underlying data and the next piece of VM state that comes along doesn't require an ioctl nr nor VMM participation.

Quoted 2 lines

[1]: http://lore.kernel.org/r/[email protected]

Thanks for the pointer to Raghavendra's series. The firmware pseudo system registers have been classified into VM and VCPU scoped in the series. I think it fits the SDEI migration requirements very well. The shared events can even be migrated through the VM scoped firmware pseudo system registers. However, I don't plan to support it in next revision (v6) as currently needed events are all private. I may spend more time to go through Raghavendra's series later.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#37 lore raw

Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM

Quoted 8 lines · Gavin Shan
Quoted 4 lines

The PSCI implementation is a great example of how KVM has grown its implementation in line with a specification, all the while preserving backwards compatibility.

The only information feed by VMM is the exposed events. The events can't be registered from guest kernel, and raised from host to guest kernel until it's exposed by VMM.

I would suggest assuming that all SDEI events are exposed by default in KVM. We will not require a VMM change to enable events individually.

Quoted 7 lines

Besides, the exposed events will
be defined staticly in host/KVM as we discussed on PATCH[02/22]. We also discussed to eliminate those ioctl commands. So I think we needn't to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a concern to host itself if the migration can be done through the firmware pseudo system registers since the migration compatibility is the only concern to VMM (QEMU).

This all needs to work just like the KVM_REG_ARM_PSCI_VERSION version, I'd recommend taking a look at how we handle that register in KVM.

Quoted 4 lines

Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable it.

As far as how it interacts with KVM, QEMU looks fine. The name of the KVM_ARM_VCPU_PSCI_0_2 bit is quite frustrating. It actually implies that KVM will enable it highest supported PSCI version. If the feature bit is cleared then you only get PSCIv0.1

However, the DT node that QEMU sets up looks a bit crusty. The properties for conveying PSCI function IDs were only ever necessary for PSCIv0.1. The only property of interest any more is 'method', to convey the SMCCC conduit instruction.

--
Thanks,
Oliver

author
#38 lore raw

Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM

Hi Oliver,

Quoted 6 lines · Oliver Upton
Quoted 5 lines · Gavin Shan
Quoted 4 lines

The PSCI implementation is a great example of how KVM has grown its implementation in line with a specification, all the while preserving backwards compatibility.

Quoted 7 lines
Quoted 3 lines

The only information feed by VMM is the exposed events. The events can't be registered from guest kernel, and raised from host to guest kernel until it's exposed by VMM.

I would suggest assuming that all SDEI events are exposed by default in KVM. We will not require a VMM change to enable events individually.

Ok, it was exactly what I did in v4, but the event is exposed by VMM in v5. In v6, it will be staticly defined again :)

Quoted 11 lines
Quoted 7 lines

Besides, the exposed events will
be defined staticly in host/KVM as we discussed on PATCH[02/22]. We also discussed to eliminate those ioctl commands. So I think we needn't to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a concern to host itself if the migration can be done through the firmware pseudo system registers since the migration compatibility is the only concern to VMM (QEMU).

This all needs to work just like the KVM_REG_ARM_PSCI_VERSION version, I'd recommend taking a look at how we handle that register in KVM.

Ok. I will do necessary investigation :)

Quoted 10 lines
Quoted 4 lines

Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable it.

As far as how it interacts with KVM, QEMU looks fine. The name of the KVM_ARM_VCPU_PSCI_0_2 bit is quite frustrating. It actually implies that KVM will enable it highest supported PSCI version. If the feature bit is cleared then you only get PSCIv0.1

Quoted 5 lines

However, the DT node that QEMU sets up looks a bit crusty. The properties for conveying PSCI function IDs were only ever necessary for PSCIv0.1. The only property of interest any more is 'method', to convey the SMCCC conduit instruction.

Ok, Thanks again for the further information about PSCI implementation. I will go through the code when I have free cycyles :)

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#39 lore raw

PATCH v5 19/22 KVM: arm64: Support SDEI ioctl commands on vCPU

In reply to Gavin Shan

This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:

* KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.

* KVM_SDEI_CMD_GET_VCPU_EVENT
* KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.

* KVM_SDEI_CMD_GET_VCPU_STATE
* KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.

* KVM_SDEI_CMD_INJECT_EVENT
Inject SDEI event.

---
 arch/arm64/include/asm/kvm_sdei.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |   9 +
 arch/arm64/kvm/arm.c                         |   3 +
 arch/arm64/kvm/sdei.c                        | 299 +++++++++++++++++++
 4 files changed, 312 insertions(+)
arch/arm64/include/asm/kvm_sdei.h +1 -0
index 64f00cc79162..ea4f222cf73d 100644
@@ -180,6 +180,7 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
 void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
 long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg);
+long kvm_sdei_vcpu_ioctl(struct kvm_vcpu *vcpu, unsigned long arg);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
arch/arm64/include/uapi/asm/kvm_sdei_state.h +9 -0
index 2bd6d11627bc..149451c5584f 100644
@@ -75,6 +75,12 @@ struct kvm_sdei_vcpu_state {
 #define KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT	4
 #define KVM_SDEI_CMD_GET_REGISTERED_EVENT	5
 #define KVM_SDEI_CMD_SET_REGISTERED_EVENT	6
+#define KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT	7
+#define KVM_SDEI_CMD_GET_VCPU_EVENT		8
+#define KVM_SDEI_CMD_SET_VCPU_EVENT		9
+#define KVM_SDEI_CMD_GET_VCPU_STATE		10
+#define KVM_SDEI_CMD_SET_VCPU_STATE		11
+#define KVM_SDEI_CMD_INJECT_EVENT		12
 
 struct kvm_sdei_cmd {
 	__u32                                           cmd;
@@ -85,6 +91,9 @@ struct kvm_sdei_cmd {
 	union {
 		struct kvm_sdei_exposed_event_state     *exposed_event_state;
 		struct kvm_sdei_registered_event_state  *registered_event_state;
+		struct kvm_sdei_vcpu_event_state	*vcpu_event_state;
+		struct kvm_sdei_vcpu_state		*vcpu_state;
+		__u64					num;
 	};
 };
 
arch/arm64/kvm/arm.c +3 -0
index ebfd504a1c08..3f532e1c4a95 100644
@@ -1387,6 +1387,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 
 		return kvm_arm_vcpu_finalize(vcpu, what);
 	}
+	case KVM_ARM_SDEI_COMMAND: {
+		return kvm_sdei_vcpu_ioctl(vcpu, arg);
+	}
 	default:
 		r = -EINVAL;
 	}
arch/arm64/kvm/sdei.c +299 -0

305 lines not on this page · raw message

Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#40 lore raw

Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU

Quoted 8 lines · Gavin Shan

This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:

* KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.

Quoted 7 lines

* KVM_SDEI_CMD_GET_VCPU_EVENT
* KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.

* KVM_SDEI_CMD_GET_VCPU_STATE
* KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.

All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG immediately. Just introduce new registers any time a new event comes along. The only event we have at the end of this series is the software-signaled event, with async PF coming later right?

Some special consideration is likely necessary to avoid adding a register for every u64 chunk of data. I don't think we need to afford userspace any illusion of granularity with these, and can probably lump it all under one giant pseudoregister.

Quoted 2 lines

* KVM_SDEI_CMD_INJECT_EVENT
Inject SDEI event.

What events are we going to allow userspace to inject? IIUC, the software-signaled event is an IPI and really under the control of the guest. Async PF is entriely under KVM control.

I do agree that having some form of event injection would be great. VM providers have found it useful to allow users to NMI their VMs when they get wedged. I just believe that userspace should not be able to trigger events that have a defined meaning and are under full KVM ownership.

IMO, unless the async PF changes need to go out to userspace, you could probably skip event injection for now and only worry about SDEI within a VM.

--
Thanks,
Oliver

author
#41 lore raw

Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU

Hi Oliver,

Quoted 22 lines · Oliver Upton
Quoted 8 lines · Gavin Shan

This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:

* KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.

Quoted 7 lines

* KVM_SDEI_CMD_GET_VCPU_EVENT
* KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.

* KVM_SDEI_CMD_GET_VCPU_STATE
* KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.

All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG immediately. Just introduce new registers any time a new event comes along. The only event we have at the end of this series is the software-signaled event, with async PF coming later right?

Quoted 5 lines

Some special consideration is likely necessary to avoid adding a register for every u64 chunk of data. I don't think we need to afford userspace any illusion of granularity with these, and can probably lump it all under one giant pseudoregister.

Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're correct we're only concerned by software signaled event and the one for Async PF.

I didn't look into Raghavendra's series deeply. Actually, a lump of registers can be avoid after 2048 byte size is specified in its encoding. I think 2048 bytes are enough for now since there are only two supported events.

In the future, we probably have varied number of SDEI events to be migrated. In that case, we need to add a new bit to the encoding of the pseudo system register, so that VMM (QEMU) can support variable sized system register and keep reading and writing on these registers on migration:

PSEUDO_SDEI_ADDR: 64-bits in width
PSEUDO_SDEI_DATA: has varied size

PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
(2) The information to be read/written, for example the (shared/private) registered events on VM and vCPU, VCPU state.

PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be written. (2) Flags to indicate current block of information is the last one or not.

Quoted 7 lines
Quoted 2 lines

* KVM_SDEI_CMD_INJECT_EVENT
Inject SDEI event.

What events are we going to allow userspace to inject? IIUC, the software-signaled event is an IPI and really under the control of the guest. Async PF is entriely under KVM control.

Quoted 9 lines

I do agree that having some form of event injection would be great. VM providers have found it useful to allow users to NMI their VMs when they get wedged. I just believe that userspace should not be able to trigger events that have a defined meaning and are under full KVM ownership.

IMO, unless the async PF changes need to go out to userspace, you could probably skip event injection for now and only worry about SDEI within a VM.

I was overthinking on the usage of SDEI. I had the assumption that SDEI may be used by emulated devices to inject SDEI events in VMM. It can even be done through PSEUDO_SDEI_{ADDR, DATA} in future. For now, the software signaled and Async PF events are concerned, Async PF event is always raised by host/KVM. So we needn't it for now and I will drop this functionality, actually the whole ioctl commands and migration support in next respin, as you suggested :)

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#42 lore raw

Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU

Quoted 31 lines · Gavin Shan

Hi Oliver,

Quoted 22 lines · Oliver Upton
Quoted 8 lines · Gavin Shan

This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:

* KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.

Quoted 7 lines

* KVM_SDEI_CMD_GET_VCPU_EVENT
* KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.

* KVM_SDEI_CMD_GET_VCPU_STATE
* KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.

All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG immediately. Just introduce new registers any time a new event comes along. The only event we have at the end of this series is the software-signaled event, with async PF coming later right?

Quoted 5 lines

Some special consideration is likely necessary to avoid adding a register for every u64 chunk of data. I don't think we need to afford userspace any illusion of granularity with these, and can probably lump it all under one giant pseudoregister.

Quoted 8 lines

Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're correct we're only concerned by software signaled event and the one for Async PF.

I didn't look into Raghavendra's series deeply. Actually, a lump of registers can be avoid after 2048 byte size is specified in its encoding. I think 2048 bytes are enough for now since there are only two supported events.

When I had said 'one giant pseudoregister' I actually meant 'one pseudoregister per event', not all of SDEI into a single structure. Since most of this is in flux now, it is hard to point out what/how we should migrate from conversation alone.

And on the topic of Raghavendra's series, I do not believe it is required anymore here w/ the removal of shared events, which I'm strongly in favor of.

Let's delve deeper into migration on the next series :)

Quoted 9 lines

In the future, we probably have varied number of SDEI events to be migrated. In that case, we need to add a new bit to the encoding of the pseudo system register, so that VMM (QEMU) can support variable sized system register and keep reading and writing on these registers on migration:

PSEUDO_SDEI_ADDR: 64-bits in width
PSEUDO_SDEI_DATA: has varied size

Quoted 7 lines

PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
(2) The information to be read/written, for example the (shared/private) registered events on VM and vCPU, VCPU state.

PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be written. (2) Flags to indicate current block of information is the last one or not.

I don't think we have sufficient encoding space in the register ID to allow for arbitrary length registers. Any new registers for SDEI will need to fit into one of the predefined sizes. Note that we've already conditioned userspace to handle registers this way and anything else is an ABI change.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#43 lore raw

Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU

Hi Oliver,

Quoted 44 lines · Oliver Upton
Quoted 29 lines · Gavin Shan
Quoted 22 lines · Oliver Upton
Quoted 8 lines · Gavin Shan

This supports ioctl commands on vCPU to manage the various object. It's primarily used by VMM to accomplish migration. The ioctl commands introduced by this are highlighted as below:

* KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
Return the total count of vCPU events, which have been queued on the target vCPU.

Quoted 7 lines

* KVM_SDEI_CMD_GET_VCPU_EVENT
* KVM_SDEI_CMD_SET_VCPU_EVENT
Get or set vCPU events.

* KVM_SDEI_CMD_GET_VCPU_STATE
* KVM_SDEI_CMD_SET_VCPU_STATE
Get or set vCPU state.

All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG immediately. Just introduce new registers any time a new event comes along. The only event we have at the end of this series is the software-signaled event, with async PF coming later right?

Quoted 5 lines

Some special consideration is likely necessary to avoid adding a register for every u64 chunk of data. I don't think we need to afford userspace any illusion of granularity with these, and can probably lump it all under one giant pseudoregister.

Quoted 8 lines

Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're correct we're only concerned by software signaled event and the one for Async PF.

I didn't look into Raghavendra's series deeply. Actually, a lump of registers can be avoid after 2048 byte size is specified in its encoding. I think 2048 bytes are enough for now since there are only two supported events.

When I had said 'one giant pseudoregister' I actually meant 'one pseudoregister per event', not all of SDEI into a single structure. Since most of this is in flux now, it is hard to point out what/how we should migrate from conversation alone.

Quoted 6 lines

And on the topic of Raghavendra's series, I do not believe it is required anymore here w/ the removal of shared events, which I'm strongly in favor of.

Let's delve deeper into migration on the next series :)

Ok, Thanks for your clarification about 'one giant pseudoregister'. Lets have more discussion about the migration on next revision. To be more clear, I plan to implement the base functionality, where only the private event is supported. After it reaches into mergeable or merged, we can post the add-on series to support migration.

Quoted 17 lines
Quoted 9 lines

In the future, we probably have varied number of SDEI events to be migrated. In that case, we need to add a new bit to the encoding of the pseudo system register, so that VMM (QEMU) can support variable sized system register and keep reading and writing on these registers on migration:

PSEUDO_SDEI_ADDR: 64-bits in width
PSEUDO_SDEI_DATA: has varied size

Quoted 7 lines

PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
(2) The information to be read/written, for example the (shared/private) registered events on VM and vCPU, VCPU state.

PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be written. (2) Flags to indicate current block of information is the last one or not.

Quoted 6 lines

I don't think we have sufficient encoding space in the register ID to allow for arbitrary length registers. Any new registers for SDEI will need to fit into one of the predefined sizes. Note that we've already conditioned userspace to handle registers this way and anything else is an ABI change.

Ok, I think we need padding to structures of the event, to make the event object aligned to 64-bytes aligned, and VCPU state to 512-bytes aligned. 64-bytes and 128-bytes registers have been supported.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#44 lore raw

PATCH v5 20/22 KVM: arm64: Export SDEI capability

In reply to Gavin Shan

The SDEI functionality is ready to be exported so far. This adds new capability (KVM_CAP_ARM_SDEI) and exports it.

---
 Documentation/virt/kvm/api.rst | 10 ++++++++++
 arch/arm64/kvm/arm.c           |  3 +++
 include/uapi/linux/kvm.h       |  1 +
 3 files changed, 14 insertions(+)
Documentation/virt/kvm/api.rst +10 -0
index 9f3172376ec3..06cf27f37b4d 100644
@@ -7575,3 +7575,13 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset
 of the result of KVM_CHECK_EXTENSION.  KVM will forward to userspace
 the hypercalls whose corresponding bit is in the argument, and return
 ENOSYS for the others.
+
+8.35 KVM_CAP_ARM_SDEI
+---------------------
+
+:Capability: KVM_CAP_ARM_SDEI
+:Architectures: arm64
+
+This capability indicates that the SDEI virtual service is supported
+in the host. A VMM can check whether the service is available to enable
+it.
arch/arm64/kvm/arm.c +3 -0
index 3f532e1c4a95..ae3b53dfe88f 100644
@@ -282,6 +282,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PTRAUTH_GENERIC:
 		r = system_has_full_ptr_auth();
 		break;
+	case KVM_CAP_ARM_SDEI:
+		r = 1;
+		break;
 	default:
 		r = 0;
 	}
include/uapi/linux/kvm.h +1 -0
index 2d11c909ec42..5772385639b8 100644
@@ -1135,6 +1135,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_XSAVE2 208
 #define KVM_CAP_SYS_ATTRIBUTES 209
 #define KVM_CAP_PPC_AIL_MODE_3 210
+#define KVM_CAP_ARM_SDEI 211
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#45 lore raw

PATCH v5 21/22 KVM: arm64: Add SDEI document

In reply to Gavin Shan

This adds one document to explain how virtualized SDEI service is implemented and supported.

---
 Documentation/virt/kvm/arm/sdei.rst | 325 ++++++++++++++++++++++++++++
 1 file changed, 325 insertions(+)
 create mode 100644 Documentation/virt/kvm/arm/sdei.rst
Documentation/virt/kvm/arm/sdei.rst new file +325 -0
new file mode 100644
index 000000000000..61213e4b9aea
@@ -0,0 +1,325 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================================
+SDEI Virtualization Support for ARM64
+=====================================
+
+Arm specification DEN0054/C defines Software Delegated Exception Interface
+(SDEI). It provides a mechanism for registering and servicing system events
+from system firmware. The interface is offered by a higher exception level
+to a lower exception level, in other words by a secure platform firmware
+to hypervisor or hypervisor to OS or both.
+
+https://developer.arm.com/documentation/den0054/c
+
+KVM/arm64 implements the defined hypercalls in the specification so that
+the system events can be registered and serviced from KVM hypervisor to
+the guest OS.
+
+SDEI Event Management
+=====================
+
+Each SDEI event is associated and identified with a 32-bit number. The
+lower 24-bits is the event number, but other bits are reserved or used
+for vendor defined events. In KVM/arm64 implementation, bit 22 and 23
+are further reserved to identify the events visible to the implementation.
+Value 0x1 should be seen in this field for those KVM/arm64 visible events.
+In the meanwhile, event number 0x0 is also supported since it is used by
+SDEI_EVENT_SIGNAL hypercall.
+
+The SDEI event needs to be exposed through ioctl interface by VMM to
+hypervisor before the guest is able to register it. The exposed event
+is represented by ``struct kvm_sdei_exposed_event``. The registered event
+is represented by ``struct kvm_sdei_registered_event``, whose instances
+are created on SDEI_EVENT_REGISTER hypercall. There is only one registered
+event instance in one particular VM no matter what event type is. The
+registered event can be injected and delivered to one specific vcpu in
+forms of vcpu event. ``struct kvm_sdei_vcpu_event`` describes vcpu events.
+On the particular vcpu, one vcpu event instance can be shared by multiple
+events.
+
+The execution runs into SDEI context when vcpu event is handled. The
+interrupted or preempted context should be saved to vcpu state, which
+is represented by ``struct kvm_sdei_vcpu``. After that, the SDEI event
+handler, which was provided by SDEI_EVENT_REGISTER hypercall is invoked.
+SDEI_EVENT_COMPLETE or SDEI_EVENT_COMPLETE_AND_RESUME hypercall must be
+issued before the SDEI event handler is to complete. When one of these
+two hypercalls is received, the interrupted or preempted context is
+restored from vcpu state for execution.
+
+When migration happens, the status of one particular SDEI event is not
+deterministic. So we need to support migration for the aforementioned
+structures or objects. The information capable of migration in these
+objects are put into separate data structures, which are the corresponding
+state variant in kvm_sdei_state.h. Besides, ioctl command are introduced
+to read and write them on the source and destination VM during migration.
+
+IOCTL Commands
+==============
+
+KVM_ARM_SDEI_COMMAND
+--------------------
+
+:Capability: KVM_CAP_ARM_SDEI
+:Type: vm ioctl, vcpu ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_cmd {
+        __u32        cmd;
+        union {
+            __u32    version;
+            __u32    count;
+        };
+        union {
+            struct kvm_sdei_exposed_event_state     *exposed_event_state;
+            struct kvm_sdei_registered_event_state  *registered_event_state;
+            struct kvm_sdei_vcpu_event_state        *vcpu_event_state;
+            struct kvm_sdei_vcpu_state              *vcpu_state;
+            __u64                                   num;
+        };
+    };
+
+The SDEI ioctl command is identified by KVM_ARM_SDEI_COMMAND and ``cmd``
+in the argument ``struct kvm_sdei_cmd`` provides further command to be
+executed.
+
+KVM_SDEI_CMD_GET_VERSION
+------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+On success, the implementation version is returned in ``version`` of
+``struct kvm_sdei_cmd``. This version is different from that of the
+followed SDEI specification. The implementation version is used to tell
+the coherence extent to the following specification. For example, the
+SDEI interrupt binding event is defined in SDEI v1.1 specification,
+but it is not supported in current implementation.
+
+KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
+------------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_EXPOSED_EVENT, to
+prepare ``exposed_event_state`` of ``struct kvm_sdei_cmd`` for that
+command during migration.
+
+On success, the number of exposed events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_EXPOSED_EVENT
+------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_exposed_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_exposed_event_state {
+        __u64   num;
+
+        __u8    type;
+        __u8    signaled;
+        __u8    priority;
+        __u8    padding[5];
+        __u64   notifier;
+    };
+
+This ioctl command is used to retrieve the exposed events on the source
+VM during migration.
+
+The number of exposed events to be retrieved is specified by ``count``
+of ``struct kvm_sdei_cmd``. On success, the retrieved exposed events are
+returned by ``exposed_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_EXPOSED_EVENT
+------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_exposed_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to expose SDEI events and migrate
+them. The ``notifier`` of ``struct kvm_sdei_exposed_event_state``
+will be modified if the specified exposed event has been existing.
+
+The number of events to be exposed is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the exposed events is
+passed by ``exposed_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
+---------------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_REGISTERED_EVENT,
+to prepare ``registered_event_state`` of ``struct kvm_sdei_cmd`` for
+that command during migration.
+
+On success, the number of registered events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_REGISTERED_EVENT
+---------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_registered_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+    struct kvm_sdei_registered_event_state {
+        __u64   num;
+
+        __u8    route_mode;
+        __u8    padding[3];
+        __u64   route_affinity;
+        __u64   ep_address[KVM_SDEI_MAX_VCPUS];
+        __u64   ep_arg[KVM_SDEI_MAX_VCPUS];
+        __u64   registered[KVM_SDEI_MAX_VCPUS/64];
+        __u64   enabled[KVM_SDEI_MAX_VCPUS/64];
+        __u64   unregister_pending[KVM_SDEI_MAX_VCPUS/64];
+    };
+
+This ioctl command is used to retrieve the registered events on the
+source VM during migration.
+
+The number of registered events to be retrieved is specified by ``count``
+of ``struct kvm_sdei_cmd``. On success, the retrieved registered events
+are returned by ``registered_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_REGISTERED_EVENT
+---------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_registered_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the registered events
+on the destination VM.
+
+The number of events to be registered is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the registered events
+is passed by ``registered_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
+---------------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_VCPU_EVENT, to
+prepare ``vcpu_event_state`` of ``struct kvm_sdei_cmd`` for that
+command during migration.
+
+On success, the number of vcpu events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_EVENT
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_vcpu_event_state {
+        __u64   num;
+
+        __u32   event_count;
+        __u32   padding;
+    };
+
+This ioctl command is used to retrieve the vcpu events on the source
+VM during migration.
+
+The number of vcpu events to be retrieved is specified by ``count`` of
+``struct kvm_sdei_cmd``. On success, the retrieved exposed events are
+returned by ``vcpu_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_VCPU_EVENT
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the vcpu events on the
+destination VM.
+
+The number of vcpu events to be added is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the vcpu events is
+passed by ``vcpu_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_STATE
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_vcpu_regs_state {
+        __u64   regs[18];
+        __u64   pc;
+        __u64   pstate;
+    };
+
+    struct kvm_sdei_vcpu_state {
+        __u8                            masked;
+        __u8                            padding[7];
+        __u64                           critical_num;
+        __u64                           normal_num;
+        struct kvm_sdei_vcpu_regs_state critical_regs;
+        struct kvm_sdei_vcpu_regs_state normal_regs;
+    };
+
+This ioctl command is used to retrieve the vcpu state on the source VM
+during migration.
+
+On success, the current vcpu state is returned by ``vcpu_state`` of
+``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_SET_VCPU_STATE
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the vcpu state on the
+destination VM.
+
+The vcpu state to be configured is passed by ``vcpu_state`` of
+``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_INJECT_EVENT
+-------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to inject SDEI event to the specified
+vcpu.
+
+The number of the SDEI event to be injected is passed by ``num`` of
+``struct kvm_sdei_cmd``.
+
+Future Work
+===========
+
+1. Support the routing mode and affinity for the shared events.
+2. Support interrupt binding events.
Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#46 lore raw

PATCH v5 22/22 KVM: selftests: Add SDEI test case

In reply to Gavin Shan

This adds SDEI self-test case where the various hypercalls are issued to default event (0x0). The default event is private, signaled and in normal priority.

By default, two vCPUs are started and the following ioctl commands or hypercalls are sent to them in sequence, to simulate how they are used in VMM and the linux guest:

kvm_check_cap(KVM_CAP_ARM_SDEI)
KVM_SDEI_CMD_GET_VERSION
KVM_SDEI_CMD_SET_EXPOSED_EVENT (expose event)

SDEI_1_0_FN_SDEI_VERSION

   SDEI_1_1_FN_SDEI_FEATURES           (SDEI capability probing)
   SDEI_1_0_FN_SDEI_SHARED_RESET       (restart SDEI)
   SDEI_1_0_FN_SDEI_PE_UNMASK          (CPU online)

SDEI_1_0_FN_SDEI_EVENT_GET_INFO

   SDEI_1_0_FN_SDEI_EVENT_REGISTER     (register event)
   SDEI_1_0_FN_SDEI_EVENT_ENABLE       (enable event)
   SDEI_1_1_FN_SDEI_EVENT_SIGNAL       (event injection)
   SDEI_1_0_FN_SDEI_EVENT_DISABLE      (disable event)
   SDEI_1_0_FN_SDEI_EVENT_UNREGISTER   (unregister event)
   SDEI_1_0_FN_SDEI_PE_MASK            (CPU offline)
---
 tools/testing/selftests/kvm/Makefile       |   1 +
 tools/testing/selftests/kvm/aarch64/sdei.c | 525 +++++++++++++++++++++
 2 files changed, 526 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c
tools/testing/selftests/kvm/Makefile +1 -0
index 17c3f0749f05..4e4c03cc316b 100644
@@ -105,6 +105,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list
 TEST_GEN_PROGS_aarch64 += aarch64/psci_cpu_on_test
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_init
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq
+TEST_GEN_PROGS_aarch64 += aarch64/sdei
 TEST_GEN_PROGS_aarch64 += demand_paging_test
 TEST_GEN_PROGS_aarch64 += dirty_log_test
 TEST_GEN_PROGS_aarch64 += dirty_log_perf_test
tools/testing/selftests/kvm/aarch64/sdei.c new file +525 -0

525 lines not on this page · raw message

Signature · 6 lines
2.23.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#47 lore raw
In reply to Gavin Shan
Quoted 132 lines · Gavin Shan

This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:

   https://developer.arm.com/documentation/den0054/c
   https://github.com/gwshan/linux    ("kvm/arm64_sdei")
   https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

For the design and migration needs, please refer to the document in PATCH[21/22] in this series. The series is organized as below:

PATCH[01] Introduces template for smccc_get_argx()
PATCH[02] Adds SDEI virtualization infrastructure
PATCH[03-17] Supports various SDEI hypercalls and event handling PATCH[18-20] Adds ioctl commands to support migration and configuration
and exports SDEI capability
PATCH[21] Adds SDEI document
PATCH[22] Adds SDEI selftest case

Testing
=======

[1] The selftest case included in this series works fine. The default SDEI

    event, whose number is zero, can be registered, enabled, raised. The
    SDEI event handler can be invoked.

    [host]# pwd
    /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
    [root@virtlab-arm01 kvm]# ./aarch64/sdei 

        NR_VCPUS: 2    SDEI Event: 0x00000000

    --- VERSION
        Version:              1.1 (vendor: 0x4b564d)
    --- FEATURES
        Shared event slots:   0
        Private event slots:  0
        Relative mode:        No
    --- PRIVATE_RESET
    --- SHARED_RESET
    --- PE_UNMASK
    --- EVENT_GET_INFO
        Type:                 Private
        Priority:             Normal
        Signaled:             Yes
    --- EVENT_REGISTER
    --- EVENT_ENABLE
    --- EVENT_SIGNAL
        Handled:              Yes
        IRQ:                  No
        Status:               Registered-Enabled-Running
        PC/PSTATE:            000000000040232c 00000000600003c5
        Regs:                 0000000000000000 0000000000000000
                              0000000000000000 0000000000000000
    --- PE_MASK
    --- EVENT_DISABLE
    --- EVENT_UNREGISTER

        Result: OK

[2] There are additional patches in the following repositories to create

    procfs entries, allowing to inject SDEI event from host side. The
    SDEI client in the guest side registers the SDEI default event, whose
    number is zero. Also, the QEMU exports SDEI ACPI table and supports
    migration for SDEI.

    https://github.com/gwshan/linux    ("kvm/arm64_sdei")
    https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

    [2.1] Start the guests and migrate the source VM to the destination
          VM.

    [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
            -accel kvm -machine virt,gic-version=host                     \
            -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
            -m 1024M,slots=16,maxmem=64G                                  \
               :                                                          \
            -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
            -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
            -append earlycon=pl011,mmio,0x9000000                         \
               :

    [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
            -accel kvm -machine virt,gic-version=host                     \
            -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
            -m 1024M,slots=16,maxmem=64G                                  \
               :                                                          \
            -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
            -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
            -append earlycon=pl011,mmio,0x9000000                         \
            -incoming tcp:0:4444                                          \
               :

    [2.2] Check kernel log on the source VM. The SDEI service is enabled
          and the default SDEI event (0x0) is enabled.

     [guest-src]# dmesg | grep -i sdei
     ACPI: SDEI 0x000000005BC80000 000024 \
                (v00 BOCHS  BXPC     00000001 BXPC 00000001)
     sdei: SDEIv1.1 (0x4b564d) detected in firmware.
     SDEI TEST: Version 1.1, Vendor 0x4b564d
     sdei_init: SDEI event (0x0) registered
     sdei_init: SDEI event (0x0) enabled

 
     (qemu) migrate -d tcp:localhost:4444

    [2.3] Migrate the source VM to the destination VM. Inject SDEI event
          to the destination VM. The event is raised and handled.

    (qemu) migrate -d tcp:localhost:4444

    [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1

    [guest-dst]#
    =========== SDEI Event (CPU#1) ===========
    Event: 0000000000000000  Parameter: 00000000dabfdabf
    PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
    Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
             ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 ffff800009399008 
             ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 
             0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 
             0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 
    Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
             ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 
             ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000 
Quoted 3 lines

Changelog
=========
v5:

Next time can you include a link to the cover letter of the previous patch set? It is extremely helpful for understanding the progress to date and allows reviewers to see prior feedback.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#48 lore raw

Hi Oliver,

Quoted 133 lines · Oliver Upton
Quoted 132 lines · Gavin Shan

This series intends to virtualize Software Delegated Exception Interface (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to deliver NMI-alike SDEI event to guest and it's needed by Async PF to deliver page-not-present notification from hypervisor to guest. The code and the required qemu changes can be found from:

    https://developer.arm.com/documentation/den0054/c
    https://github.com/gwshan/linux    ("kvm/arm64_sdei")
    https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

For the design and migration needs, please refer to the document in PATCH[21/22] in this series. The series is organized as below:

PATCH[01] Introduces template for smccc_get_argx()
PATCH[02] Adds SDEI virtualization infrastructure
PATCH[03-17] Supports various SDEI hypercalls and event handling PATCH[18-20] Adds ioctl commands to support migration and configuration
and exports SDEI capability
PATCH[21] Adds SDEI document
PATCH[22] Adds SDEI selftest case

Testing
=======

[1] The selftest case included in this series works fine. The default SDEI

     event, whose number is zero, can be registered, enabled, raised. The
     SDEI event handler can be invoked.

     [host]# pwd
     /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
     [root@virtlab-arm01 kvm]# ./aarch64/sdei

         NR_VCPUS: 2    SDEI Event: 0x00000000

     --- VERSION
         Version:              1.1 (vendor: 0x4b564d)
     --- FEATURES
         Shared event slots:   0
         Private event slots:  0
         Relative mode:        No
     --- PRIVATE_RESET
     --- SHARED_RESET
     --- PE_UNMASK
     --- EVENT_GET_INFO
         Type:                 Private
         Priority:             Normal
         Signaled:             Yes
     --- EVENT_REGISTER
     --- EVENT_ENABLE
     --- EVENT_SIGNAL
         Handled:              Yes
         IRQ:                  No
         Status:               Registered-Enabled-Running
         PC/PSTATE:            000000000040232c 00000000600003c5
         Regs:                 0000000000000000 0000000000000000
                               0000000000000000 0000000000000000
     --- PE_MASK
     --- EVENT_DISABLE
     --- EVENT_UNREGISTER

         Result: OK

[2] There are additional patches in the following repositories to create

     procfs entries, allowing to inject SDEI event from host side. The
     SDEI client in the guest side registers the SDEI default event, whose
     number is zero. Also, the QEMU exports SDEI ACPI table and supports
     migration for SDEI.

     https://github.com/gwshan/linux    ("kvm/arm64_sdei")
     https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

     [2.1] Start the guests and migrate the source VM to the destination
           VM.

     [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
             -accel kvm -machine virt,gic-version=host                     \
             -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
             -m 1024M,slots=16,maxmem=64G                                  \
                :                                                          \
             -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
             -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
             -append earlycon=pl011,mmio,0x9000000                         \
                :

     [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
             -accel kvm -machine virt,gic-version=host                     \
             -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
             -m 1024M,slots=16,maxmem=64G                                  \
                :                                                          \
             -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
             -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
             -append earlycon=pl011,mmio,0x9000000                         \
             -incoming tcp:0:4444                                          \
                :

     [2.2] Check kernel log on the source VM. The SDEI service is enabled
           and the default SDEI event (0x0) is enabled.

      [guest-src]# dmesg | grep -i sdei
      ACPI: SDEI 0x000000005BC80000 000024 \
                 (v00 BOCHS  BXPC     00000001 BXPC 00000001)
      sdei: SDEIv1.1 (0x4b564d) detected in firmware.
      SDEI TEST: Version 1.1, Vendor 0x4b564d
      sdei_init: SDEI event (0x0) registered
      sdei_init: SDEI event (0x0) enabled

  
      (qemu) migrate -d tcp:localhost:4444

     [2.3] Migrate the source VM to the destination VM. Inject SDEI event
           to the destination VM. The event is raised and handled.

     (qemu) migrate -d tcp:localhost:4444

     [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1

     [guest-dst]#
     =========== SDEI Event (CPU#1) ===========
     Event: 0000000000000000  Parameter: 00000000dabfdabf
     PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
     Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
              ffff800016c28000 0000000000000000 0000000000000000 0000000000000000
              0000000000000000 0000000000000000 0000000000000000 0000000000000000
              0000000000000000 0000000000000000 0000000000000000 0000000000000000
              0000000000000000 0000000000000000 0000000000000000 ffff800009399008
              ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18
              0000000000000000 0000000000000000 ffff000000339d00 0000000000000000
              0000000000000000 ffff800009c7bde0 ffff800008cbb5c4
     Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
              ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190
              ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000
Quoted 8 lines
Quoted 3 lines

Changelog
=========
v5:

Next time can you include a link to the cover letter of the previous patch set? It is extremely helpful for understanding the progress to date and allows reviewers to see prior feedback.

Yep, I will provide the link to the cover letter of the previous version. I'm amending it this time:

    https://lore.kernel.org/all/[email protected]/

Besides, I don't know what happened to my "git send-email". Some of the recipients are skipped even I have put them into the cc list. Lets amend it again to avoid resending the series.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm