RISC-V: KVM: VCPU reset fixes
Subsystems (4)
v3 cover RISC-V: KVM: VCPU reset fixes
Hello,
the design still requires a discussion.
[v3 1/2] removes most of the additional changes that the KVM capability was doing in v2. [v3 2/2] is new and previews a general solution to the lack of userspace control over KVM SBI.
A possible QEMU implementation for both capabilities can be seen in
https://github.com/radimkrcmar/qemu/tree/reset_fixes_v3
The next step would be to forward the HSM ecalls to QEMU.
v2: https://lore.kernel.org/kvm-riscv/[email protected]/
v1: https://lore.kernel.org/kvm-riscv/[email protected]/
Radim Krčmář (2):
RISC-V: KVM: add KVM_CAP_RISCV_MP_STATE_RESET
RISC-V: KVM: add KVM_CAP_RISCV_USERSPACE_SBI
Documentation/virt/kvm/api.rst | 22 ++++++++++++++++++++++ arch/riscv/include/asm/kvm_host.h | 6 ++++++ arch/riscv/include/asm/kvm_vcpu_sbi.h | 1 + arch/riscv/kvm/vcpu.c | 27 ++++++++++++++------------- arch/riscv/kvm/vcpu_sbi.c | 27 +++++++++++++++++++++++++-- arch/riscv/kvm/vm.c | 18 ++++++++++++++++++ include/uapi/linux/kvm.h | 2 ++ 7 files changed, 88 insertions(+), 15 deletions(-)
Signature · 1 line
2.49.0
PATCH v3 1/2 RISC-V: KVM: add KVM_CAP_RISCV_MP_STATE_RESET
Add a toggleable VM capability to reset the VCPU from userspace by setting MP_STATE_INIT_RECEIVED through IOCTL.
Reset through a mp_state to avoid adding a new IOCTL.
Do not reset on a transition from STOPPED to RUNNABLE, because it's better to avoid side effects that would complicate userspace adoption. The MP_STATE_INIT_RECEIVED is not a permanent mp_state -- IOCTL resets the VCPU while preserving the original mp_state -- because we wouldn't gain much from having a new state it in the rest of KVM, but it's a very non-standard use of the IOCTL.
---
If we want a permanent mp_state, I think that MP_STATE_UNINITIALIZED would be reasonable. KVM could reset on transition to any other state.
v3: do not allow allow userspace to set the HSM reset state [Anup]
v2: new
--- Documentation/virt/kvm/api.rst | 11 +++++++++++ arch/riscv/include/asm/kvm_host.h | 3 +++ arch/riscv/include/asm/kvm_vcpu_sbi.h | 1 + arch/riscv/kvm/vcpu.c | 27 ++++++++++++++------------- arch/riscv/kvm/vcpu_sbi.c | 17 +++++++++++++++++ arch/riscv/kvm/vm.c | 13 +++++++++++++ include/uapi/linux/kvm.h | 1 + 7 files changed, 60 insertions(+), 13 deletions(-)
Documentation/virt/kvm/api.rst +11 -0
@@ -8496,6 +8496,17 @@ aforementioned registers before the first KVM_RUN. These registers are VM scoped, meaning that the same set of values are presented on all vCPUs in a given VM. +7.43 KVM_CAP_RISCV_MP_STATE_RESET +--------------------------------- + +:Architectures: riscv +:Type: VM +:Parameters: None +:Returns: 0 on success, -EINVAL if arg[0] is not zero + +When this capability is enabled, KVM resets the VCPU when setting +MP_STATE_INIT_RECEIVED through IOCTL. The original MP_STATE is preserved. + 8. Other capabilities. ======================
arch/riscv/include/asm/kvm_host.h +3 -0
@@ -119,6 +119,9 @@ struct kvm_arch {/* AIA Guest/VM context */ struct kvm_aia aia; + +/* KVM_CAP_RISCV_MP_STATE_RESET */ +bool mp_state_reset; };struct kvm_cpu_trap {
arch/riscv/include/asm/kvm_vcpu_sbi.h +1 -0
@@ -57,6 +57,7 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu, u32 type, u64 flags);void kvm_riscv_vcpu_sbi_request_reset (struct kvm_vcpu *vcpu,unsigned long pc,unsigned long a1); +void kvm_riscv_vcpu_sbi_load_reset_state (struct kvm_vcpu *vcpu);int kvm_riscv_vcpu_sbi_return (struct kvm_vcpu *vcpu,struct kvm_run *run);int kvm_riscv_vcpu_set_reg_sbi_ext (struct kvm_vcpu *vcpu,const struct kvm_one_reg *reg);
arch/riscv/kvm/vcpu.c +14 -13
@@ -51,11 +51,11 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { sizeof(kvm_vcpu_stats_desc), }; -static void kvm_riscv_vcpu_context_reset (struct kvm_vcpu *vcpu) +static void kvm_riscv_vcpu_context_reset (struct kvm_vcpu *vcpu, +bool kvm_sbi_reset) {struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; -struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state;void *vector_datap = cntx->vector.datap;memset (cntx,0 ,sizeof (*cntx));
@@ -65,13 +65,8 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)/* Restore datap as it's not a part of the guest context. */ cntx->vector.datap = vector_datap; -/* Load SBI reset values */ - cntx->a0 = vcpu->vcpu_id; - -spin_lock (&reset_state->lock); - cntx->sepc = reset_state->pc; - cntx->a1 = reset_state->a1; -spin_unlock (&reset_state->lock); +if (kvm_sbi_reset) +kvm_riscv_vcpu_sbi_load_reset_state (vcpu);/* Setup reset state of shadow SSTATUS and HSTATUS CSRs */ cntx->sstatus =SR_SPP |SR_SPIE ;
@@ -84,7 +79,7 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu) csr->scounteren =0x7 ; } -static void kvm_riscv_reset_vcpu (struct kvm_vcpu *vcpu) +static void kvm_riscv_reset_vcpu (struct kvm_vcpu *vcpu,bool kvm_sbi_reset) {bool loaded;
@@ -100,7 +95,7 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) vcpu->arch.last_exit_cpu =-1 ; -kvm_riscv_vcpu_context_reset (vcpu); +kvm_riscv_vcpu_context_reset (vcpu, kvm_sbi_reset);kvm_riscv_vcpu_fp_reset (vcpu);
@@ -177,7 +172,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)kvm_riscv_vcpu_sbi_init (vcpu);/* Reset VCPU */ -kvm_riscv_reset_vcpu (vcpu); +kvm_riscv_reset_vcpu (vcpu, false);return 0 ; }
@@ -526,6 +521,12 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,case KVM_MP_STATE_STOPPED :__kvm_riscv_vcpu_power_off (vcpu);break ; +case KVM_MP_STATE_INIT_RECEIVED : +if (vcpu->kvm->arch.mp_state_reset) +kvm_riscv_reset_vcpu (vcpu, false); +else + ret = -EINVAL ; +break ;default : ret = -EINVAL ; }
@@ -714,7 +715,7 @@ static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu) }if (kvm_check_request (KVM_REQ_VCPU_RESET , vcpu)) -kvm_riscv_reset_vcpu (vcpu); +kvm_riscv_reset_vcpu (vcpu, true);if (kvm_check_request (KVM_REQ_UPDATE_HGATP , vcpu))kvm_riscv_gstage_update_hgatp (vcpu);
arch/riscv/kvm/vcpu_sbi.c +17 -0
@@ -167,6 +167,23 @@ void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,kvm_make_request (KVM_REQ_VCPU_RESET , vcpu); } +void kvm_riscv_vcpu_sbi_load_reset_state (struct kvm_vcpu *vcpu) +{ +struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; +struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; +struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state; + + cntx->a0 = vcpu->vcpu_id; + +spin_lock (&vcpu->arch.reset_state.lock); + cntx->sepc = reset_state->pc; + cntx->a1 = reset_state->a1; +spin_unlock (&vcpu->arch.reset_state.lock); + + cntx->sstatus &= ~SR_SIE ; + csr->vsatp =0 ; +} +int kvm_riscv_vcpu_sbi_return (struct kvm_vcpu *vcpu,struct kvm_run *run) {struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
arch/riscv/kvm/vm.c +13 -0
@@ -209,6 +209,19 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)return r; } +int kvm_vm_ioctl_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap) +{ +switch (cap->cap) { +case KVM_CAP_RISCV_MP_STATE_RESET : +if (cap->flags) +return -EINVAL ; + kvm->arch.mp_state_reset = true; +return 0 ; +default : +return -EINVAL ; + } +} +int kvm_arch_vm_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg) {return -EINVAL ;
include/uapi/linux/kvm.h +1 -0
@@ -930,6 +930,7 @@ struct kvm_enable_cap {#define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237#define KVM_CAP_X86_GUEST_MODE 238#define KVM_CAP_ARM_WRITABLE_IMP_ID_REGS 239 +#define KVM_CAP_RISCV_MP_STATE_RESET 240struct kvm_irq_routing_irqchip {__u32 irqchip;
Signature · 1 line
2.49.0
Anup Patel
maintainerRe: [PATCH v3 1/2] RISC-V: KVM: add KVM_CAP_RISCV_MP_STATE_RESET
Quoted 12 lines · Radim Krčmář
Add a toggleable VM capability to reset the VCPU from userspace by setting
MP_STATE_INIT_RECEIVEDthrough IOCTL.Reset through a
mp_stateto avoid adding a new IOCTL.
Do not reset on a transition from STOPPED to RUNNABLE, because it's better to avoid side effects that would complicate userspace adoption. TheMP_STATE_INIT_RECEIVEDis not a permanentmp_state-- IOCTL resets the VCPU while preserving the originalmp_state-- because we wouldn't gain much from having a new state it in the rest of KVM, but it's a very non-standard use of the IOCTL.
Quoted 4 lines
Signed-off-by Radim Krčmář <[email protected]>---
If we want a permanentmp_state, I think thatMP_STATE_UNINITIALIZEDwould be reasonable. KVM could reset on transition to any other state.
Yes, MP_STATE_UNINITIALIZED looks better. I also suggest that VCPU should be reset when set_mpstate() is called with MP_STATE_UNINITIALIZED and the current state is MP_STATE_STOPPED.
Regards,
Anup
Quoted 211 lines
v3: do not allow allow userspace to set the HSM reset state [Anup]
v2: new--- Documentation/virt/kvm/api.rst | 11 +++++++++++ arch/riscv/include/asm/kvm_host.h | 3 +++ arch/riscv/include/asm/kvm_vcpu_sbi.h | 1 + arch/riscv/kvm/vcpu.c | 27 ++++++++++++++------------- arch/riscv/kvm/vcpu_sbi.c | 17 +++++++++++++++++ arch/riscv/kvm/vm.c | 13 +++++++++++++ include/uapi/linux/kvm.h | 1 + 7 files changed, 60 insertions(+), 13 deletions(-)Documentation/virt/kvm/api.rst +11
-0@@ -8496,6 +8496,17 @@ aforementioned registers before the first KVM_RUN. These registers are VM scoped, meaning that the same set of values are presented on all vCPUs in a given VM. +7.43 KVM_CAP_RISCV_MP_STATE_RESET +--------------------------------- + +:Architectures: riscv +:Type: VM +:Parameters: None +:Returns: 0 on success, -EINVAL if arg[0] is not zero + +When this capability is enabled, KVM resets the VCPU when setting +MP_STATE_INIT_RECEIVED through IOCTL. The original MP_STATE is preserved. + 8. Other capabilities. ======================arch/riscv/include/asm/kvm_host.h +3
-0@@ -119,6 +119,9 @@ struct kvm_arch {/* AIA Guest/VM context */ struct kvm_aia aia; + +/* KVM_CAP_RISCV_MP_STATE_RESET */ +bool mp_state_reset; };struct kvm_cpu_trap {arch/riscv/include/asm/kvm_vcpu_sbi.h +1
-0@@ -57,6 +57,7 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu, u32 type, u64 flags);void kvm_riscv_vcpu_sbi_request_reset (struct kvm_vcpu *vcpu,unsigned long pc,unsigned long a1); +void kvm_riscv_vcpu_sbi_load_reset_state (struct kvm_vcpu *vcpu);int kvm_riscv_vcpu_sbi_return (struct kvm_vcpu *vcpu,struct kvm_run *run);int kvm_riscv_vcpu_set_reg_sbi_ext (struct kvm_vcpu *vcpu,const struct kvm_one_reg *reg);arch/riscv/kvm/vcpu.c +14
-13@@ -51,11 +51,11 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { sizeof(kvm_vcpu_stats_desc), }; -static void kvm_riscv_vcpu_context_reset (struct kvm_vcpu *vcpu) +static void kvm_riscv_vcpu_context_reset (struct kvm_vcpu *vcpu, +bool kvm_sbi_reset) {struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; -struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state;void *vector_datap = cntx->vector.datap;memset (cntx,0 ,sizeof (*cntx));@@ -65,13 +65,8 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)/* Restore datap as it's not a part of the guest context. */ cntx->vector.datap = vector_datap; -/* Load SBI reset values */ - cntx->a0 = vcpu->vcpu_id; - -spin_lock (&reset_state->lock); - cntx->sepc = reset_state->pc; - cntx->a1 = reset_state->a1; -spin_unlock (&reset_state->lock); +if (kvm_sbi_reset) +kvm_riscv_vcpu_sbi_load_reset_state (vcpu);/* Setup reset state of shadow SSTATUS and HSTATUS CSRs */ cntx->sstatus =SR_SPP |SR_SPIE ;@@ -84,7 +79,7 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu) csr->scounteren =0x7 ; } -static void kvm_riscv_reset_vcpu (struct kvm_vcpu *vcpu) +static void kvm_riscv_reset_vcpu (struct kvm_vcpu *vcpu,bool kvm_sbi_reset) {bool loaded;@@ -100,7 +95,7 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) vcpu->arch.last_exit_cpu =-1 ; -kvm_riscv_vcpu_context_reset (vcpu); +kvm_riscv_vcpu_context_reset (vcpu, kvm_sbi_reset);kvm_riscv_vcpu_fp_reset (vcpu);@@ -177,7 +172,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)kvm_riscv_vcpu_sbi_init (vcpu);/* Reset VCPU */ -kvm_riscv_reset_vcpu (vcpu); +kvm_riscv_reset_vcpu (vcpu, false);return 0 ; }@@ -526,6 +521,12 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,case KVM_MP_STATE_STOPPED :__kvm_riscv_vcpu_power_off (vcpu);break ; +case KVM_MP_STATE_INIT_RECEIVED : +if (vcpu->kvm->arch.mp_state_reset) +kvm_riscv_reset_vcpu (vcpu, false); +else + ret = -EINVAL ; +break ;default : ret = -EINVAL ; }@@ -714,7 +715,7 @@ static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu) }if (kvm_check_request (KVM_REQ_VCPU_RESET , vcpu)) -kvm_riscv_reset_vcpu (vcpu); +kvm_riscv_reset_vcpu (vcpu, true);if (kvm_check_request (KVM_REQ_UPDATE_HGATP , vcpu))kvm_riscv_gstage_update_hgatp (vcpu);arch/riscv/kvm/vcpu_sbi.c +17
-0@@ -167,6 +167,23 @@ void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,kvm_make_request (KVM_REQ_VCPU_RESET , vcpu); } +void kvm_riscv_vcpu_sbi_load_reset_state (struct kvm_vcpu *vcpu) +{ +struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; +struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; +struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state; + + cntx->a0 = vcpu->vcpu_id; + +spin_lock (&vcpu->arch.reset_state.lock); + cntx->sepc = reset_state->pc; + cntx->a1 = reset_state->a1; +spin_unlock (&vcpu->arch.reset_state.lock); + + cntx->sstatus &= ~SR_SIE ; + csr->vsatp =0 ; +} +int kvm_riscv_vcpu_sbi_return (struct kvm_vcpu *vcpu,struct kvm_run *run) {struct kvm_cpu_context *cp = &vcpu->arch.guest_context;arch/riscv/kvm/vm.c +13
-0@@ -209,6 +209,19 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)return r; } +int kvm_vm_ioctl_enable_cap (struct kvm *kvm,struct kvm_enable_cap *cap) +{ +switch (cap->cap) { +case KVM_CAP_RISCV_MP_STATE_RESET : +if (cap->flags) +return -EINVAL ; + kvm->arch.mp_state_reset = true; +return 0 ; +default : +return -EINVAL ; + } +} +int kvm_arch_vm_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg) {return -EINVAL ;include/uapi/linux/kvm.h +1
-1@@ -930,6 +930,7 @@ struct kvm_enable_cap {#define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237#define KVM_CAP_X86_GUEST_MODE 238#define KVM_CAP_ARM_WRITABLE_IMP_ID_REGS 239 +#define KVM_CAP_RISCV_MP_STATE_RESET 240struct kvm_irq_routing_irqchip {__u32 irqchip; --2.49.0
Re: [PATCH v3 1/2] RISC-V: KVM: add KVM_CAP_RISCV_MP_STATE_RESET
2025-05-16T17:55:05+05:30, Anup Patel <[email protected]>:
Quoted 18 lines · Radim Krčmář
Quoted 12 lines
Add a toggleable VM capability to reset the VCPU from userspace by setting
MP_STATE_INIT_RECEIVEDthrough IOCTL.Reset through a
mp_stateto avoid adding a new IOCTL.
Do not reset on a transition from STOPPED to RUNNABLE, because it's better to avoid side effects that would complicate userspace adoption. TheMP_STATE_INIT_RECEIVEDis not a permanentmp_state-- IOCTL resets the VCPU while preserving the originalmp_state-- because we wouldn't gain much from having a new state it in the rest of KVM, but it's a very non-standard use of the IOCTL.Quoted 4 lines
Signed-off-by Radim Krčmář <[email protected]>---
If we want a permanentmp_state, I think thatMP_STATE_UNINITIALIZEDwould be reasonable. KVM could reset on transition to any other state.
Quoted 4 lines
Yes,
MP_STATE_UNINITIALIZEDlooks better. I also suggest that VCPU should be reset whenset_mpstate()is called withMP_STATE_UNINITIALIZEDand the current state isMP_STATE_STOPPED.
That would result in two resets (stopped -> uninitialized -> *), unless we changed the logic.
Would you prefer to reset on transition to the new permanent mp_state? MP_STATE_INIT_RECEIVED seems a more fitting name for the state, then.
Thanks.
Re: [PATCH v3 1/2] RISC-V: KVM: add KVM_CAP_RISCV_MP_STATE_RESET
Quoted 25 lines · Radim Krčmář
2025-05-16T17:55:05+05:30, Anup Patel <[email protected]>:Quoted 18 lines · Radim Krčmář
Quoted 12 lines
Add a toggleable VM capability to reset the VCPU from userspace by setting
MP_STATE_INIT_RECEIVEDthrough IOCTL.Reset through a
mp_stateto avoid adding a new IOCTL.
Do not reset on a transition from STOPPED to RUNNABLE, because it's better to avoid side effects that would complicate userspace adoption. TheMP_STATE_INIT_RECEIVEDis not a permanentmp_state-- IOCTL resets the VCPU while preserving the originalmp_state-- because we wouldn't gain much from having a new state it in the rest of KVM, but it's a very non-standard use of the IOCTL.Quoted 4 lines
Signed-off-by Radim Krčmář <[email protected]>---
If we want a permanentmp_state, I think thatMP_STATE_UNINITIALIZEDwould be reasonable. KVM could reset on transition to any other state.Quoted 4 lines
Yes,
MP_STATE_UNINITIALIZEDlooks better. I also suggest that VCPU should be reset whenset_mpstate()is called withMP_STATE_UNINITIALIZEDand the current state isMP_STATE_STOPPED.
Quoted 6 lines
That would result in two resets (stopped -> uninitialized -> *), unless we changed the logic.
Would you prefer to reset on transition to the new permanent
mp_state?MP_STATE_INIT_RECEIVEDseems a more fitting name for the state, then.
Let's not introduce unnecessary state transitions and go ahead with MP_STATE_INIT_RECEIVED
Queued this patch for Linux-6.16
Thanks,
Anup
PATCH v3 2/2 RISC-V: KVM: add KVM_CAP_RISCV_USERSPACE_SBI
In reply to Radim KrčmářThe new capability allows userspace to implement SBI extensions that KVM does not handle. This allows userspace to implement any SBI ecall as userspace already has the ability to disable acceleration of selected SBI extensions.
This is a VM capability, because userspace will most likely want to have the same behavior for all VCPUs. We can easily make it both a VCPU and a VM capability if there is demand in the future.
---
v3: new
--- Documentation/virt/kvm/api.rst | 11 +++++++++++ arch/riscv/include/asm/kvm_host.h | 3 +++ arch/riscv/kvm/vcpu_sbi.c | 10 ++++++++-- arch/riscv/kvm/vm.c | 5 +++++ include/uapi/linux/kvm.h | 1 + 5 files changed, 28 insertions(+), 2 deletions(-)
Documentation/virt/kvm/api.rst +11 -0
@@ -8507,6 +8507,17 @@ given VM. When this capability is enabled, KVM resets the VCPU when setting MP_STATE_INIT_RECEIVED through IOCTL. The original MP_STATE is preserved. +7.44 KVM_CAP_RISCV_USERSPACE_SBI +-------------------------------- + +:Architectures: riscv +:Type: VM +:Parameters: None +:Returns: 0 on success, -EINVAL if arg[0] is not zero + +When this capability is enabled, KVM forwards ecalls from disabled or unknown +SBI extensions to userspace. + 8. Other capabilities. ======================
arch/riscv/include/asm/kvm_host.h +3 -0
@@ -122,6 +122,9 @@ struct kvm_arch {/* KVM_CAP_RISCV_MP_STATE_RESET */ bool mp_state_reset; + +/* KVM_CAP_RISCV_USERSPACE_SBI */ +bool userspace_sbi; };struct kvm_cpu_trap {
arch/riscv/kvm/vcpu_sbi.c +8 -2
@@ -471,8 +471,14 @@ int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run)#endif ret = sbi_ext->handler (vcpu, run, &sbi_ret); }else { -/* Return error for unsupported SBI calls */ - cp->a0 =SBI_ERR_NOT_SUPPORTED ; +if (vcpu->kvm->arch.userspace_sbi) { + next_sepc = false; + ret =0 ; +kvm_riscv_vcpu_sbi_forward (vcpu, run); + }else { +/* Return error for unsupported SBI calls */ + cp->a0 =SBI_ERR_NOT_SUPPORTED ; + } goto ecall_done; }
arch/riscv/kvm/vm.c +5 -0
@@ -217,6 +217,11 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)return -EINVAL ; kvm->arch.mp_state_reset = true;return 0 ; +case KVM_CAP_RISCV_USERSPACE_SBI : +if (cap->flags) +return -EINVAL ; + kvm->arch.userspace_sbi = true; +return 0 ;default :return -EINVAL ; }
include/uapi/linux/kvm.h +1 -0
@@ -931,6 +931,7 @@ struct kvm_enable_cap {#define KVM_CAP_X86_GUEST_MODE 238#define KVM_CAP_ARM_WRITABLE_IMP_ID_REGS 239#define KVM_CAP_RISCV_MP_STATE_RESET 240 +#define KVM_CAP_RISCV_USERSPACE_SBI 241struct kvm_irq_routing_irqchip {__u32 irqchip;
Signature · 1 line
2.49.0
Atish Patra
reviewerQuoted 8 lines · Radim KrÄmáÅ
Hello,
the design still requires a discussion.
[v3 1/2] removes most of the additional changes that the KVM capability was doing in v2. [v3 2/2] is new and previews a general solution to the lack of userspace control over KVM SBI.
I am still missing the motivation behind it. If the motivation is SBI HSM suspend, the PATCH2 doesn't achieve that as it forwards every call to the user space. Why do you want to control hsm start/stop from the user space ?
Quoted 20 lines
A possible QEMU implementation for both capabilities can be seen in
https://github.com/radimkrcmar/qemu/tree/reset_fixes_v3
The next step would be to forward the HSM ecalls to QEMU.v2: https://lore.kernel.org/kvm-riscv/[email protected]/
v1: https://lore.kernel.org/kvm-riscv/[email protected]/Radim Krčmář (2):
RISC-V: KVM: addKVM_CAP_RISCV_MP_STATE_RESET
RISC-V: KVM: addKVM_CAP_RISCV_USERSPACE_SBIDocumentation/virt/kvm/api.rst | 22 ++++++++++++++++++++++ arch/riscv/include/asm/kvm_host.h | 6 ++++++ arch/riscv/include/asm/kvm_vcpu_sbi.h | 1 + arch/riscv/kvm/vcpu.c | 27 ++++++++++++++------------- arch/riscv/kvm/vcpu_sbi.c | 27 +++++++++++++++++++++++++-- arch/riscv/kvm/vm.c | 18 ++++++++++++++++++ include/uapi/linux/kvm.h | 2 ++ 7 files changed, 88 insertions(+), 15 deletions(-)
2025-05-22T14:43:40-07:00, Atish Patra <[email protected]>:
Quoted 10 lines · Radim KrÄmáÅ
Quoted 8 lines
Hello,
the design still requires a discussion.
[v3 1/2] removes most of the additional changes that the KVM capability was doing in v2. [v3 2/2] is new and previews a general solution to the lack of userspace control over KVM SBI.
Quoted 4 lines
I am still missing the motivation behind it. If the motivation is SBI HSM suspend, the PATCH2 doesn't achieve that as it forwards every call to the user space. Why do you want to control hsm start/stop from the user space ?
HSM needs fixing, because KVM doesn't know what the state after sbi_hart_start should be.
For example, we had a discussion about scounteren and regardless of what default we choose in KVM, the userspace might want a different value. I don't think that HSM start/stop is a hot path, so trapping to userspace seems better than adding more kernel code.
Forwarding all the unimplemented SBI ecalls shouldn't be a performance issue, because S-mode software would hopefully learn after the first error and stop trying again.
Allowing userspace to fully implement the ecall instruction one of the motivations as well -- SBI is not a part of RISC-V ISA, so someone might be interested in accelerating a different M-mode software with KVM.
I'll send v4 later today -- there is a missing part in [2/2], because userspace also needs to be able to emulate the base SBI extension.
Quoted 17 lines · Radim Krčmář
2025-05-22T14:43:40-07:00, Atish Patra <[email protected]>:Quoted 10 lines · Radim KrÄmáÅ
Quoted 8 lines
Hello,
the design still requires a discussion.
[v3 1/2] removes most of the additional changes that the KVM capability was doing in v2. [v3 2/2] is new and previews a general solution to the lack of userspace control over KVM SBI.
Quoted 4 lines
I am still missing the motivation behind it. If the motivation is SBI HSM suspend, the PATCH2 doesn't achieve that as it forwards every call to the user space. Why do you want to control hsm start/stop from the user space ?
Quoted 6 lines
HSM needs fixing, because KVM doesn't know what the state after
sbi_hart_startshould be.
For example, we had a discussion about scounteren and regardless of what default we choose in KVM, the userspace might want a different value. I don't think that HSM start/stop is a hot path, so trapping to userspace seems better than adding more kernel code.
There are no implementation specific S-mode CSR reset values required at the moment. Whenever the need arises, we will extend the ONE_REG interface so that user space can specify custom CSR reset values at Guest/VM creation time. We don't need to forward SBI HSM calls to user space for custom S-mode CSR reset values.
Quoted 5 lines
Forwarding all the unimplemented SBI ecalls shouldn't be a performance issue, because S-mode software would hopefully learn after the first error and stop trying again.
Quoted 7 lines
Allowing userspace to fully implement the ecall instruction one of the motivations as well -- SBI is not a part of RISC-V ISA, so someone might be interested in accelerating a different M-mode software with KVM.
I'll send v4 later today -- there is a missing part in [2/2], because userspace also needs to be able to emulate the base SBI extension.
Emulating entire SBI in user space has may challenges, here are few:
1) SBI IPI in userspace will require an ioctl to trigger VCPU local interrupt which does not exist. We only have KVM ioctls to trigger external interrupts and MSIs.
2) SBI RFENCE in userspace will requires HFENCE operation in user space which is not allowed by RISC-V ISA.
3) SBI PMU uses Linux perf framework APIs to share counters between host and guest. The Linux perf APIs for guest perf events are not available to userspace as syscall or ioctl.
4) SBI STA uses sched_info.run_delay which I am sure is not available to user space.
5) SBI NACL when implemented will be using tons of HS-mode functionality (HS-mode CSRs, HFENCEs, etc.) to achieve the nested world-switch and none of these are accessible to userspace.
6) SBI FWFT may require programming hstateenX CSRs which are not accessible to userspace.
7) SBI DBTR requires direct coordination between the KVM RISC-V and kernel hw_breakpoint driver to share the debug triggers.
... and so on ...
Based on the above, emulating the entire SBI in user space is a non-starter. The best approach is to selectively forward SBI calls to user space where needed (e.g. SBI system reset, SBI system suspend, SBI debug console, etc.).
Regards,
Anup
2025-05-23T13:38:26+05:30, Anup Patel <[email protected]>:
Quoted 25 lines · Radim Krčmář
Quoted 17 lines
2025-05-22T14:43:40-07:00, Atish Patra <[email protected]>:Quoted 10 lines · Radim KrÄmáÅ
Quoted 8 lines
Hello,
the design still requires a discussion.
[v3 1/2] removes most of the additional changes that the KVM capability was doing in v2. [v3 2/2] is new and previews a general solution to the lack of userspace control over KVM SBI.
Quoted 4 lines
I am still missing the motivation behind it. If the motivation is SBI HSM suspend, the PATCH2 doesn't achieve that as it forwards every call to the user space. Why do you want to control hsm start/stop from the user space ?
Quoted 6 lines
HSM needs fixing, because KVM doesn't know what the state after
sbi_hart_startshould be.
For example, we had a discussion about scounteren and regardless of what default we choose in KVM, the userspace might want a different value. I don't think that HSM start/stop is a hot path, so trapping to userspace seems better than adding more kernel code.
Quoted 2 lines
There are no implementation specific S-mode CSR reset values required at the moment.
Jessica mentioned that BSD requires scounteren to be non-zero, so userspace should be able to provide that value.
I would prefer if KVM could avoid getting into those discussions. We can just just let userspace be as crazy as it wants.
Quoted 5 lines
Whenever the need arises, we will extend
theONE_REGinterface so that user space can specify custom CSR reset values at Guest/VM creation time. We don't need to forward SBI HSM calls to user space for custom S-mode CSR reset values.
The benefits of adding a new ONE_REG interface seem very small compared to the drawbacks of having extra kernel code.
If userspace would want to reset or setup new multi-VCPUs VMs often, we could add an interface that loads the whole register state from userspace in a single IOCTL, because ONE_REG is not the best interface for bulk data transfer either.
Quoted 8 lines
Quoted 8 lines
Forwarding all the unimplemented SBI ecalls shouldn't be a performance issue, because S-mode software would hopefully learn after the first error and stop trying again.
Allowing userspace to fully implement the ecall instruction one of the motivations as well -- SBI is not a part of RISC-V ISA, so someone might be interested in accelerating a different M-mode software with KVM.
Quoted 7 lines
Quoted 3 lines
I'll send v4 later today -- there is a missing part in [2/2], because userspace also needs to be able to emulate the base SBI extension.
[...] The best approach is to selectively forward SBI
calls to user space where needed (e.g. SBI system reset, SBI system suspend, SBI debug console, etc.).
That is exactly what my proposal does, it's just that the userspace says what is "needed".
If we started with this mechanism, KVM would not have needed to add SRST/SUSP/DBCN SBI emulation at all -- they would be forwarded as any other unhandled ecall.
Atish Patra
reviewerQuoted 22 lines · Radim Krčmář
2025-05-23T13:38:26+05:30, Anup Patel <[email protected]>:Quoted 21 lines · Radim Krčmář
Quoted 20 lines
2025-05-22T14:43:40-07:00, Atish Patra <[email protected]>:Quoted 5 lines · Radim KrÄmáÅ
Quoted 4 lines
Hello,
the design still requires a discussion.
Quoted 8 lines
Quoted 4 lines
[v3 1/2] removes most of the additional changes that the KVM capability was doing in v2. [v3 2/2] is new and previews a general solution to the lack of userspace control over KVM SBI.
I am still missing the motivation behind it. If the motivation is SBI HSM suspend, the PATCH2 doesn't achieve that as it forwards every call to the user space. Why do you want to control hsm start/stop from the user space ?
HSM needs fixing, because KVM doesn't know what the state after
sbi_hart_startshould be.
For example, we had a discussion about scounteren and regardless of what default we choose in KVM, the userspace might want a different value. I don't think that HSM start/stop is a hot path, so trapping to userspace seems better than adding more kernel code.
Quoted 4 lines
Quoted 2 lines
There are no implementation specific S-mode CSR reset values required at the moment.
Jessica mentioned that BSD requires scounteren to be non-zero, so userspace should be able to provide that value.
Jessica admitted that it was a bug which should be fixed.
Quoted 2 lines
I would prefer if KVM could avoid getting into those discussions. We can just just let userspace be as crazy as it wants.
The scounteren state you mentioned is already fixed now.
I would prefer to do this if there are more of these issues. Otherwise, we may gain little by just delegating more work to the userspace for no reason.
Quoted 7 lines
Quoted 5 lines
Whenever the need arises, we will extend
theONE_REGinterface so that user space can specify custom CSR reset values at Guest/VM creation time. We don't need to forward SBI HSM calls to user space for custom S-mode CSR reset values.The benefits of adding a new
ONE_REGinterface seem very small compared to the drawbacks of having extra kernel code.
How ? The extra kernel code is just few lines where it just registers a SBI extension and forwards it to the userspace. That's for the entire extension.
For extensions like HSM, only selective functions that should be forwarded to the userspace which defeats the purpose.
Let's not try to fix something that is not broken yet.
Quoted 25 lines
If userspace would want to reset or setup new multi-VCPUs VMs often, we could add an interface that loads the whole register state from userspace in a single IOCTL, because
ONE_REGis not the best interface for bulk data transfer either.Quoted 8 lines
Quoted 8 lines
Forwarding all the unimplemented SBI ecalls shouldn't be a performance issue, because S-mode software would hopefully learn after the first error and stop trying again.
Allowing userspace to fully implement the ecall instruction one of the motivations as well -- SBI is not a part of RISC-V ISA, so someone might be interested in accelerating a different M-mode software with KVM.
Quoted 6 lines
Quoted 3 lines
I'll send v4 later today -- there is a missing part in [2/2], because userspace also needs to be able to emulate the base SBI extension.
[...] The best approach is to selectively forward SBI
calls to user space where needed (e.g. SBI system reset, SBI system suspend, SBI debug console, etc.).That is exactly what my proposal does, it's just that the userspace says what is "needed".
If we started with this mechanism, KVM would not have needed to add SRST/SUSP/DBCN SBI emulation at all -- they would be forwarded as any other unhandled ecall.
Quoted 27 lines · Radim Krčmář
2025-05-23T13:38:26+05:30, Anup Patel <[email protected]>:Quoted 25 lines · Radim Krčmář
Quoted 23 lines
2025-05-22T14:43:40-07:00, Atish Patra <[email protected]>:Quoted 10 lines · Radim KrÄmáÅ
Quoted 8 lines
Hello,
the design still requires a discussion.
[v3 1/2] removes most of the additional changes that the KVM capability was doing in v2. [v3 2/2] is new and previews a general solution to the lack of userspace control over KVM SBI.
Quoted 4 lines
I am still missing the motivation behind it. If the motivation is SBI HSM suspend, the PATCH2 doesn't achieve that as it forwards every call to the user space. Why do you want to control hsm start/stop from the user space ?
HSM needs fixing, because KVM doesn't know what the state after
sbi_hart_startshould be.
For example, we had a discussion about scounteren and regardless of what default we choose in KVM, the userspace might want a different value. I don't think that HSM start/stop is a hot path, so trapping to userspace seems better than adding more kernel code.
Quoted 8 lines
Quoted 2 lines
There are no implementation specific S-mode CSR reset values required at the moment.
Jessica mentioned that BSD requires scounteren to be non-zero, so userspace should be able to provide that value.
I would prefer if KVM could avoid getting into those discussions. We can just just let userspace be as crazy as it wants.
The supervisor OS must not expect a particular state of S-mode CSRs other than what is defined in the boot protocol or the SBI specification.
Like mentioned before, scounteren setup in KVM RISC-V and OpenSBI is a HACK for buggy OSes which don't set up scounteren CSR correctly when a HART comes-up. Even KVM user space should not entertain such HACKs.
Quoted 9 lines
Quoted 6 lines
Whenever the need arises, we will extend
theONE_REGinterface so that user space can specify custom CSR reset values at Guest/VM creation time. We don't need to forward SBI HSM calls to user space for custom S-mode CSR reset values.The benefits of adding a new
ONE_REGinterface seem very small compared to the drawbacks of having extra kernel code.
Forwarding HSM stop to userspace will slow down CPU hotplug on Guest side. Further, this directly impacts SBI system suspend performance for Guest because Guest is supposed to turn-off all VCPUs except one before entering the SBI system suspend state.
Quoted 5 lines
If userspace would want to reset or setup new multi-VCPUs VMs often, we could add an interface that loads the whole register state from userspace in a single IOCTL, because
ONE_REGis not the best interface for bulk data transfer either.
Instead of inventing a new interface, we can simply improve the ONE_REG interface to allow bulk read/write of multiple ONE_REG registers which will benefit other architectures as well.
If required in the future, this bulk ONE_REG read/write interface can also be used to load reset state of VCPU CSRs.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Forwarding all the unimplemented SBI ecalls shouldn't be a performance issue, because S-mode software would hopefully learn after the first error and stop trying again.
Allowing userspace to fully implement the ecall instruction one of the motivations as well -- SBI is not a part of RISC-V ISA, so someone might be interested in accelerating a different M-mode software with KVM.
I'll send v4 later today -- there is a missing part in [2/2], because userspace also needs to be able to emulate the base SBI extension.
Quoted 6 lines
Quoted 3 lines
[...] The best approach is to selectively forward SBI
calls to user space where needed (e.g. SBI system reset, SBI system suspend, SBI debug console, etc.).That is exactly what my proposal does, it's just that the userspace says what is "needed".
Nope, the approach taken by your patch is problematic because for example userspace might disable SBI RFENCE or SBI PMU with no means to implement these SBI extensions in user space.
We can't blindly forward an SBI extension to userspace when userspace lacks the capability to implement this extension.
Quoted 4 lines
If we started with this mechanism, KVM would not have needed to add SRST/SUSP/DBCN SBI emulation at all -- they would be forwarded as any other unhandled ecall.
SBI SRST extension is implemented in kernel space because we are re-using the existing KVM_EXIT_SYSTEM_EVENT so that we can also re-use existing KVM_EXIT_SYSTEM_EVENT related code on userspace side.
SBI SUSP and DBCN are already forward to user space and we only have a minimal code in kernel space to ensure that:
1) In-kernel SBI BASE extension is aware of these extensions
2) These are forwarded to userspace only when userspace enables these extensions.
In addition to the above, we are blindly forwarding SBI experimental and vendor extensions to user space so user space can do its own thing by implementing these extensions.
Regards,
Anup