Skip to contentSkip to conversation

KVM: RISC-V: VCPU reset fixes

PATCH Superseded Radim Krčmář 4 participants 35 messages Apr 3, 2025 – May 8, 2025

Subsystems (3)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover KVM: RISC-V: VCPU reset fixes

Hello,

this series started with a simple fix [5/5], which sadly didn't fix enough without [4/5]. [1-3/5] are refactors to make the reset a bit easier to follow. ([1-3,5/5] are applicable without [4/5].)

[4/5] changes the userspace ABI and I'd like to gather your opinions on how the ABI is supposed to work.

As another proposal, what do you think about an IOCTL that allows userspace to invoke any KVM SBI ecall?
Userspace could call the KVM HSM implementation to reset and start a VCPU, but I think that kvm_mp_state is an SBI HSM interface, so we have an obscure IOCTL for it already.
I was also thinking about using KVM_MP_STATE_UNINITIALIZED to distinguish that KVM should reset the state when becoming runnable.

I recommend to start with the following hunk in [4/5], as it's the only significant part of this series:

@@ -520,6 +525,10 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,

   	switch (mp_state->mp_state) {
   	case KVM_MP_STATE_RUNNABLE:
  +		if (riscv_vcpu_supports_sbi_ext(vcpu, KVM_RISCV_SBI_EXT_HSM) &&
  +				vcpu->arch.ran_atleast_once &&
  +				kvm_riscv_vcpu_stopped(vcpu))
  +			kvm_riscv_vcpu_sbi_request_reset_from_userspace(vcpu);

Thanks.

Radim Krčmář (5):
KVM: RISC-V: refactor vector state reset
KVM: RISC-V: refactor sbi reset request
KVM: RISC-V: remove unnecessary SBI reset state
KVM: RISC-V: reset VCPU state when becoming runnable
KVM: RISC-V: reset smstateen CSRs

 arch/riscv/include/asm/kvm_aia.h         |  3 --
 arch/riscv/include/asm/kvm_host.h        | 13 +++--
 arch/riscv/include/asm/kvm_vcpu_sbi.h    |  5 ++
 arch/riscv/include/asm/kvm_vcpu_vector.h |  6 +--
 arch/riscv/kvm/aia_device.c              |  4 +-
 arch/riscv/kvm/vcpu.c                    | 69 +++++++++++++++---------
 arch/riscv/kvm/vcpu_sbi.c                | 28 ++++++++++
 arch/riscv/kvm/vcpu_sbi_hsm.c            | 13 +----
 arch/riscv/kvm/vcpu_sbi_system.c         | 10 +---
 arch/riscv/kvm/vcpu_vector.c             | 13 ++---
 10 files changed, 97 insertions(+), 67 deletions(-)
Signature · 1 line
2.48.1
#1 lore raw

PATCH 1/5 KVM: RISC-V: refactor vector state reset

Do not depend on the reset structures.

vector.datap is a kernel memory pointer that needs to be preserved as it is not a part of the guest vector data.

---
 arch/riscv/include/asm/kvm_vcpu_vector.h |  6 ++----
 arch/riscv/kvm/vcpu.c                    |  5 ++++-
 arch/riscv/kvm/vcpu_vector.c             | 13 +++++++------
 3 files changed, 13 insertions(+), 11 deletions(-)
arch/riscv/include/asm/kvm_vcpu_vector.h +2 -4
index 27f5bccdd8b0..57a798a4cb0d 100644
@@ -33,8 +33,7 @@ void kvm_riscv_vcpu_guest_vector_restore(struct kvm_cpu_context *cntx,
 					 unsigned long *isa);
 void kvm_riscv_vcpu_host_vector_save(struct kvm_cpu_context *cntx);
 void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx);
-int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
-					struct kvm_cpu_context *cntx);
+int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu);
 void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu);
 #else
 
@@ -62,8 +61,7 @@ static inline void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cn
 {
 }
 
-static inline int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
-						      struct kvm_cpu_context *cntx)
+static inline int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu)
 {
 	return 0;
 }
arch/riscv/kvm/vcpu.c +4 -1
index 60d684c76c58..2fb75288ecfe 100644
@@ -57,6 +57,7 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
 	struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr;
 	struct kvm_cpu_context *cntx = &vcpu->arch.guest_context;
 	struct kvm_cpu_context *reset_cntx = &vcpu->arch.guest_reset_context;
+	void *vector_datap = cntx->vector.datap;
 	bool loaded;
 
 	/**
@@ -79,6 +80,8 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
 
 	kvm_riscv_vcpu_fp_reset(vcpu);
 
+	/* Restore datap as it's not a part of the guest context. */
+	cntx->vector.datap = vector_datap;
 	kvm_riscv_vcpu_vector_reset(vcpu);
 
 	kvm_riscv_vcpu_timer_reset(vcpu);
@@ -143,7 +146,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 	cntx->hstatus |= HSTATUS_SPV;
 	spin_unlock(&vcpu->arch.reset_cntx_lock);
 
-	if (kvm_riscv_vcpu_alloc_vector_context(vcpu, cntx))
+	if (kvm_riscv_vcpu_alloc_vector_context(vcpu))
 		return -ENOMEM;
 
 	/* By default, make CY, TM, and IR counters accessible in VU mode */
arch/riscv/kvm/vcpu_vector.c +7 -6
index d92d1348045c..a5f88cb717f3 100644
@@ -22,6 +22,9 @@ void kvm_riscv_vcpu_vector_reset(struct kvm_vcpu *vcpu)
 	struct kvm_cpu_context *cntx = &vcpu->arch.guest_context;
 
 	cntx->sstatus &= ~SR_VS;
+
+	cntx->vector.vlenb = riscv_v_vsize / 32;
+
 	if (riscv_isa_extension_available(isa, v)) {
 		cntx->sstatus |= SR_VS_INITIAL;
 		WARN_ON(!cntx->vector.datap);
@@ -70,13 +73,11 @@ void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx)
 		__kvm_riscv_vector_restore(cntx);
 }
 
-int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
-					struct kvm_cpu_context *cntx)
+int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu)
 {
-	cntx->vector.datap = kmalloc(riscv_v_vsize, GFP_KERNEL);
-	if (!cntx->vector.datap)
+	vcpu->arch.guest_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL);
+	if (!vcpu->arch.guest_context.vector.datap)
 		return -ENOMEM;
-	cntx->vector.vlenb = riscv_v_vsize / 32;
 
 	vcpu->arch.host_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL);
 	if (!vcpu->arch.host_context.vector.datap)
@@ -87,7 +88,7 @@ int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
 
 void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu)
 {
-	kfree(vcpu->arch.guest_reset_context.vector.datap);
+	kfree(vcpu->arch.guest_context.vector.datap);
 	kfree(vcpu->arch.host_context.vector.datap);
 }
 #endif
Signature · 1 line
2.48.1
Reviewed-by on v1 1/5
#2 lore raw

Re: [PATCH 1/5] KVM: RISC-V: refactor vector state reset

Quoted 5 lines · Radim Krčmář

Do not depend on the reset structures.

vector.datap is a kernel memory pointer that needs to be preserved as it is not a part of the guest vector data.

Quoted 7 lines
---
 arch/riscv/include/asm/kvm_vcpu_vector.h |  6 ++----
 arch/riscv/kvm/vcpu.c                    |  5 ++++-
 arch/riscv/kvm/vcpu_vector.c             | 13 +++++++------
 3 files changed, 13 insertions(+), 11 deletions(-)
maintainer
#3 lore raw

Re: [PATCH 1/5] KVM: RISC-V: refactor vector state reset

In reply to Radim Krčmář
Quoted 7 lines · Radim Krčmář

Do not depend on the reset structures.

vector.datap is a kernel memory pointer that needs to be preserved as it is not a part of the guest vector data.

Queued this patch for Linux-6.16

Thanks,
Anup

Quoted 103 lines
---
 arch/riscv/include/asm/kvm_vcpu_vector.h |  6 ++----
 arch/riscv/kvm/vcpu.c                    |  5 ++++-
 arch/riscv/kvm/vcpu_vector.c             | 13 +++++++------
 3 files changed, 13 insertions(+), 11 deletions(-)
arch/riscv/include/asm/kvm_vcpu_vector.h +2 -4
index 27f5bccdd8b0..57a798a4cb0d 100644
@@ -33,8 +33,7 @@ void kvm_riscv_vcpu_guest_vector_restore(struct kvm_cpu_context *cntx,
                                         unsigned long *isa);
 void kvm_riscv_vcpu_host_vector_save(struct kvm_cpu_context *cntx);
 void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx);
-int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
-                                       struct kvm_cpu_context *cntx);
+int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu);
 void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu);
 #else
 
@@ -62,8 +61,7 @@ static inline void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cn
 {
 }
 
-static inline int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
-                                                     struct kvm_cpu_context *cntx)
+static inline int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu)
 {
        return 0;
 }
arch/riscv/kvm/vcpu.c +4 -1
index 60d684c76c58..2fb75288ecfe 100644
@@ -57,6 +57,7 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
        struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr;
        struct kvm_cpu_context *cntx = &vcpu->arch.guest_context;
        struct kvm_cpu_context *reset_cntx = &vcpu->arch.guest_reset_context;
+       void *vector_datap = cntx->vector.datap;
        bool loaded;
 
        /**
@@ -79,6 +80,8 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
 
        kvm_riscv_vcpu_fp_reset(vcpu);
 
+       /* Restore datap as it's not a part of the guest context. */
+       cntx->vector.datap = vector_datap;
        kvm_riscv_vcpu_vector_reset(vcpu);
 
        kvm_riscv_vcpu_timer_reset(vcpu);
@@ -143,7 +146,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
        cntx->hstatus |= HSTATUS_SPV;
        spin_unlock(&vcpu->arch.reset_cntx_lock);
 
-       if (kvm_riscv_vcpu_alloc_vector_context(vcpu, cntx))
+       if (kvm_riscv_vcpu_alloc_vector_context(vcpu))
                return -ENOMEM;
 
        /* By default, make CY, TM, and IR counters accessible in VU mode */
arch/riscv/kvm/vcpu_vector.c +7 -7
index d92d1348045c..a5f88cb717f3 100644
@@ -22,6 +22,9 @@ void kvm_riscv_vcpu_vector_reset(struct kvm_vcpu *vcpu)
        struct kvm_cpu_context *cntx = &vcpu->arch.guest_context;
 
        cntx->sstatus &= ~SR_VS;
+
+       cntx->vector.vlenb = riscv_v_vsize / 32;
+
        if (riscv_isa_extension_available(isa, v)) {
                cntx->sstatus |= SR_VS_INITIAL;
                WARN_ON(!cntx->vector.datap);
@@ -70,13 +73,11 @@ void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx)
                __kvm_riscv_vector_restore(cntx);
 }
 
-int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
-                                       struct kvm_cpu_context *cntx)
+int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu)
 {
-       cntx->vector.datap = kmalloc(riscv_v_vsize, GFP_KERNEL);
-       if (!cntx->vector.datap)
+       vcpu->arch.guest_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL);
+       if (!vcpu->arch.guest_context.vector.datap)
                return -ENOMEM;
-       cntx->vector.vlenb = riscv_v_vsize / 32;
 
        vcpu->arch.host_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL);
        if (!vcpu->arch.host_context.vector.datap)
@@ -87,7 +88,7 @@ int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
 
 void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu)
 {
-       kfree(vcpu->arch.guest_reset_context.vector.datap);
+       kfree(vcpu->arch.guest_context.vector.datap);
        kfree(vcpu->arch.host_context.vector.datap);
 }
 #endif
--

2.48.1

#4 lore raw

PATCH 2/5 KVM: RISC-V: refactor sbi reset request

In reply to Radim Krčmář

The same code is used twice and SBI reset sets only two variables.

---
 arch/riscv/include/asm/kvm_vcpu_sbi.h |  2 ++
 arch/riscv/kvm/vcpu_sbi.c             | 12 ++++++++++++
 arch/riscv/kvm/vcpu_sbi_hsm.c         | 13 +------------
 arch/riscv/kvm/vcpu_sbi_system.c      | 10 +---------
 4 files changed, 16 insertions(+), 21 deletions(-)
arch/riscv/include/asm/kvm_vcpu_sbi.h +2 -0
index 4ed6203cdd30..aaaa81355276 100644
@@ -55,6 +55,8 @@ void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run);
 void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 				     struct kvm_run *run,
 				     u32 type, u64 flags);
+void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1);
 int kvm_riscv_vcpu_sbi_return(struct kvm_vcpu *vcpu, struct kvm_run *run);
 int kvm_riscv_vcpu_set_reg_sbi_ext(struct kvm_vcpu *vcpu,
 				   const struct kvm_one_reg *reg);
arch/riscv/kvm/vcpu_sbi.c +12 -0
index d1c83a77735e..f58368f7df1d 100644
@@ -156,6 +156,18 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 	run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
 }
 
+void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1)
+{
+	spin_lock(&vcpu->arch.reset_cntx_lock);
+	vcpu->arch.guest_reset_context.sepc = pc;
+	vcpu->arch.guest_reset_context.a0 = vcpu->vcpu_id;
+	vcpu->arch.guest_reset_context.a1 = a1;
+	spin_unlock(&vcpu->arch.reset_cntx_lock);
+
+	kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
+}
+
 int kvm_riscv_vcpu_sbi_return(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {
 	struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
arch/riscv/kvm/vcpu_sbi_hsm.c +1 -12
index 3070bb31745d..f26207f84bab 100644
@@ -15,7 +15,6 @@
 
 static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu)
 {
-	struct kvm_cpu_context *reset_cntx;
 	struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
 	struct kvm_vcpu *target_vcpu;
 	unsigned long target_vcpuid = cp->a0;
@@ -32,17 +31,7 @@ static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu)
 		goto out;
 	}
 
-	spin_lock(&target_vcpu->arch.reset_cntx_lock);
-	reset_cntx = &target_vcpu->arch.guest_reset_context;
-	/* start address */
-	reset_cntx->sepc = cp->a1;
-	/* target vcpu id to start */
-	reset_cntx->a0 = target_vcpuid;
-	/* private data passed from kernel */
-	reset_cntx->a1 = cp->a2;
-	spin_unlock(&target_vcpu->arch.reset_cntx_lock);
-
-	kvm_make_request(KVM_REQ_VCPU_RESET, target_vcpu);
+	kvm_riscv_vcpu_sbi_request_reset(target_vcpu, cp->a1, cp->a2);
 
 	__kvm_riscv_vcpu_power_on(target_vcpu);
 
arch/riscv/kvm/vcpu_sbi_system.c +1 -9
index bc0ebba89003..359be90b0fc5 100644
@@ -13,7 +13,6 @@ static int kvm_sbi_ext_susp_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
 				    struct kvm_vcpu_sbi_return *retdata)
 {
 	struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
-	struct kvm_cpu_context *reset_cntx;
 	unsigned long funcid = cp->a6;
 	unsigned long hva, i;
 	struct kvm_vcpu *tmp;
@@ -45,14 +44,7 @@ static int kvm_sbi_ext_susp_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
 			}
 		}
 
-		spin_lock(&vcpu->arch.reset_cntx_lock);
-		reset_cntx = &vcpu->arch.guest_reset_context;
-		reset_cntx->sepc = cp->a1;
-		reset_cntx->a0 = vcpu->vcpu_id;
-		reset_cntx->a1 = cp->a2;
-		spin_unlock(&vcpu->arch.reset_cntx_lock);
-
-		kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
+		kvm_riscv_vcpu_sbi_request_reset(vcpu, cp->a1, cp->a2);
 
 		/* userspace provides the suspend implementation */
 		kvm_riscv_vcpu_sbi_forward(vcpu, run);
Signature · 1 line
2.48.1
Reviewed-by on v1 2/5
#5 lore raw

Re: [PATCH 2/5] KVM: RISC-V: refactor sbi reset request

Quoted 10 lines · Radim Krčmář

The same code is used twice and SBI reset sets only two variables.

---
 arch/riscv/include/asm/kvm_vcpu_sbi.h |  2 ++
 arch/riscv/kvm/vcpu_sbi.c             | 12 ++++++++++++
 arch/riscv/kvm/vcpu_sbi_hsm.c         | 13 +------------
 arch/riscv/kvm/vcpu_sbi_system.c      | 10 +---------
 4 files changed, 16 insertions(+), 21 deletions(-)
maintainer
#6 lore raw

Re: [PATCH 2/5] KVM: RISC-V: refactor sbi reset request

In reply to Radim Krčmář
Quoted 16 lines · Radim Krčmář

The same code is used twice and SBI reset sets only two variables.

---
 arch/riscv/include/asm/kvm_vcpu_sbi.h |  2 ++
 arch/riscv/kvm/vcpu_sbi.c             | 12 ++++++++++++
 arch/riscv/kvm/vcpu_sbi_hsm.c         | 13 +------------
 arch/riscv/kvm/vcpu_sbi_system.c      | 10 +---------
 4 files changed, 16 insertions(+), 21 deletions(-)
arch/riscv/include/asm/kvm_vcpu_sbi.h +0 -0
index 4ed6203cdd30..aaaa81355276 100644

arch/riscv/include/asm/kvm_vcpu_sbi.h void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run); · line 59

Quoted 5 lines
arch/riscv/include/asm/kvm_vcpu_sbi.h +2 -0
@@ -55,6 +55,8 @@ void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run);
 void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
                                     struct kvm_run *run,
                                     u32 type, u64 flags);
+void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1);

Use tabs for alignment instead of spaces.

Quoted 86 lines
arch/riscv/include/asm/kvm_vcpu_sbi.h +0 -0
 int kvm_riscv_vcpu_sbi_return(struct kvm_vcpu *vcpu, struct kvm_run *run);
 int kvm_riscv_vcpu_set_reg_sbi_ext(struct kvm_vcpu *vcpu,
                                   const struct kvm_one_reg *reg);
arch/riscv/kvm/vcpu_sbi.c +12 -0
index d1c83a77735e..f58368f7df1d 100644
@@ -156,6 +156,18 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
        run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
 }
 
+void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1)
+{
+       spin_lock(&vcpu->arch.reset_cntx_lock);
+       vcpu->arch.guest_reset_context.sepc = pc;
+       vcpu->arch.guest_reset_context.a0 = vcpu->vcpu_id;
+       vcpu->arch.guest_reset_context.a1 = a1;
+       spin_unlock(&vcpu->arch.reset_cntx_lock);
+
+       kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
+}
+
 int kvm_riscv_vcpu_sbi_return(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {
        struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
arch/riscv/kvm/vcpu_sbi_hsm.c +1 -12
index 3070bb31745d..f26207f84bab 100644
@@ -15,7 +15,6 @@
 
 static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu)
 {
-       struct kvm_cpu_context *reset_cntx;
        struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
        struct kvm_vcpu *target_vcpu;
        unsigned long target_vcpuid = cp->a0;
@@ -32,17 +31,7 @@ static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu)
                goto out;
        }
 
-       spin_lock(&target_vcpu->arch.reset_cntx_lock);
-       reset_cntx = &target_vcpu->arch.guest_reset_context;
-       /* start address */
-       reset_cntx->sepc = cp->a1;
-       /* target vcpu id to start */
-       reset_cntx->a0 = target_vcpuid;
-       /* private data passed from kernel */
-       reset_cntx->a1 = cp->a2;
-       spin_unlock(&target_vcpu->arch.reset_cntx_lock);
-
-       kvm_make_request(KVM_REQ_VCPU_RESET, target_vcpu);
+       kvm_riscv_vcpu_sbi_request_reset(target_vcpu, cp->a1, cp->a2);
 
        __kvm_riscv_vcpu_power_on(target_vcpu);
 
arch/riscv/kvm/vcpu_sbi_system.c +1 -10
index bc0ebba89003..359be90b0fc5 100644
@@ -13,7 +13,6 @@ static int kvm_sbi_ext_susp_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
                                    struct kvm_vcpu_sbi_return *retdata)
 {
        struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
-       struct kvm_cpu_context *reset_cntx;
        unsigned long funcid = cp->a6;
        unsigned long hva, i;
        struct kvm_vcpu *tmp;
@@ -45,14 +44,7 @@ static int kvm_sbi_ext_susp_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
                        }
                }
 
-               spin_lock(&vcpu->arch.reset_cntx_lock);
-               reset_cntx = &vcpu->arch.guest_reset_context;
-               reset_cntx->sepc = cp->a1;
-               reset_cntx->a0 = vcpu->vcpu_id;
-               reset_cntx->a1 = cp->a2;
-               spin_unlock(&vcpu->arch.reset_cntx_lock);
-
-               kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
+               kvm_riscv_vcpu_sbi_request_reset(vcpu, cp->a1, cp->a2);
 
                /* userspace provides the suspend implementation */
                kvm_riscv_vcpu_sbi_forward(vcpu, run);
--
Quoted 2 lines

2.48.1

Otherwise, it looks good to me.
I have taken care of the above comment at the time of merging this patch.

Queued this patch for Linux-6.16

Thanks,
Anup

#7 lore raw

Re: [PATCH 2/5] KVM: RISC-V: refactor sbi reset request

2025-05-07T17:31:33+05:30, Anup Patel <[email protected]>:
Quoted 10 lines · Radim Krčmář

arch/riscv/include/asm/kvm_vcpu_sbi.h void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run); · line 59

Quoted 7 lines
arch/riscv/include/asm/kvm_vcpu_sbi.h +2 -0
@@ -55,6 +55,8 @@ void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run);
 void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
                                     struct kvm_run *run,
                                     u32 type, u64 flags);
+void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1);

Use tabs for alignment instead of spaces.

Oops, I totally forgot that linux uses tabs even for alignment.

Quoted 3 lines

Otherwise, it looks good to me.
I have taken care of the above comment at the time of merging this patch.

Thanks, I'll post v2 without the three patches.

maintainer
#8 lore raw

Re: [PATCH 2/5] KVM: RISC-V: refactor sbi reset request

Quoted 13 lines · Radim Krčmář
2025-05-07T17:31:33+05:30, Anup Patel <[email protected]>:
Quoted 10 lines · Radim Krčmář

arch/riscv/include/asm/kvm_vcpu_sbi.h void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run); · line 59

Quoted 7 lines
arch/riscv/include/asm/kvm_vcpu_sbi.h +2 -0
@@ -55,6 +55,8 @@ void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run);
 void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
                                     struct kvm_run *run,
                                     u32 type, u64 flags);
+void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1);

Use tabs for alignment instead of spaces.

Quoted 7 lines

Oops, I totally forgot that linux uses tabs even for alignment.

Quoted 3 lines

Otherwise, it looks good to me.
I have taken care of the above comment at the time of merging this patch.

Thanks, I'll post v2 without the three patches.

Base your v2 upon riscv_kvm_queue branch at
https://github.com/kvm-riscv/linux.git

Regards,
Anup

#9 lore raw

PATCH 3/5 KVM: RISC-V: remove unnecessary SBI reset state

In reply to Radim Krčmář

The SBI reset state has only two variables -- pc and a1. The rest is known, so keep only the necessary information.

The reset structures make sense if we want userspace to control the reset state (which we do), but I'd still remove them now and reintroduce with the userspace interface later -- we could probably have just a single reset state per VM, instead of a reset state for each VCPU.

---
 arch/riscv/include/asm/kvm_aia.h  |  3 --
 arch/riscv/include/asm/kvm_host.h | 12 ++++---
 arch/riscv/kvm/aia_device.c       |  4 +--
 arch/riscv/kvm/vcpu.c             | 58 +++++++++++++++++--------------
 arch/riscv/kvm/vcpu_sbi.c         |  9 +++--
 5 files changed, 44 insertions(+), 42 deletions(-)
arch/riscv/include/asm/kvm_aia.h +0 -3
index 1f37b600ca47..3b643b9efc07 100644
@@ -63,9 +63,6 @@ struct kvm_vcpu_aia {
 	/* CPU AIA CSR context of Guest VCPU */
 	struct kvm_vcpu_aia_csr guest_csr;
 
-	/* CPU AIA CSR context upon Guest VCPU reset */
-	struct kvm_vcpu_aia_csr guest_reset_csr;
-
 	/* Guest physical address of IMSIC for this VCPU */
 	gpa_t		imsic_addr;
 
arch/riscv/include/asm/kvm_host.h +7 -5
index 0e9c2fab6378..0c8c9c05af91 100644
@@ -193,6 +193,12 @@ struct kvm_vcpu_smstateen_csr {
 	unsigned long sstateen0;
 };
 
+struct kvm_vcpu_reset_state {
+	spinlock_t lock;
+	unsigned long pc;
+	unsigned long a1;
+};
+
 struct kvm_vcpu_arch {
 	/* VCPU ran at least once */
 	bool ran_atleast_once;
@@ -227,12 +233,8 @@ struct kvm_vcpu_arch {
 	/* CPU Smstateen CSR context of Guest VCPU */
 	struct kvm_vcpu_smstateen_csr smstateen_csr;
 
-	/* CPU context upon Guest VCPU reset */
-	struct kvm_cpu_context guest_reset_context;
-	spinlock_t reset_cntx_lock;
+	struct kvm_vcpu_reset_state reset_state;
 
-	/* CPU CSR context upon Guest VCPU reset */
-	struct kvm_vcpu_csr guest_reset_csr;
 
 	/*
 	 * VCPU interrupts
arch/riscv/kvm/aia_device.c +1 -3
index 39cd26af5a69..43e472ff3e1a 100644
@@ -526,12 +526,10 @@ int kvm_riscv_vcpu_aia_update(struct kvm_vcpu *vcpu)
 void kvm_riscv_vcpu_aia_reset(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
-	struct kvm_vcpu_aia_csr *reset_csr =
-				&vcpu->arch.aia_context.guest_reset_csr;
 
 	if (!kvm_riscv_aia_available())
 		return;
-	memcpy(csr, reset_csr, sizeof(*csr));
+	memset(csr, 0, sizeof(*csr));
 
 	/* Proceed only if AIA was initialized successfully */
 	if (!kvm_riscv_aia_initialized(vcpu->kvm))
arch/riscv/kvm/vcpu.c +32 -26

93 lines not on this page · raw message

arch/riscv/kvm/vcpu_sbi.c +4 -5
index f58368f7df1d..3d7955e05cc3 100644
@@ -159,11 +159,10 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1)
 {
-	spin_lock(&vcpu->arch.reset_cntx_lock);
-	vcpu->arch.guest_reset_context.sepc = pc;
-	vcpu->arch.guest_reset_context.a0 = vcpu->vcpu_id;
-	vcpu->arch.guest_reset_context.a1 = a1;
-	spin_unlock(&vcpu->arch.reset_cntx_lock);
+	spin_lock(&vcpu->arch.reset_state.lock);
+	vcpu->arch.reset_state.pc = pc;
+	vcpu->arch.reset_state.a1 = a1;
+	spin_unlock(&vcpu->arch.reset_state.lock);
 
 	kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
 }
Signature · 1 line
2.48.1
Reviewed-by on v1 3/5
#10 lore raw

Re: [PATCH 3/5] KVM: RISC-V: remove unnecessary SBI reset state

Quoted 8 lines · Radim Krčmář

The SBI reset state has only two variables -- pc and a1. The rest is known, so keep only the necessary information.

The reset structures make sense if we want userspace to control the reset state (which we do), but I'd still remove them now and reintroduce with the userspace interface later -- we could probably have just a single reset state per VM, instead of a reset state for each VCPU.

Quoted 9 lines
---
 arch/riscv/include/asm/kvm_aia.h  |  3 --
 arch/riscv/include/asm/kvm_host.h | 12 ++++---
 arch/riscv/kvm/aia_device.c       |  4 +--
 arch/riscv/kvm/vcpu.c             | 58 +++++++++++++++++--------------
 arch/riscv/kvm/vcpu_sbi.c         |  9 +++--
 5 files changed, 44 insertions(+), 42 deletions(-)
maintainer
#11 lore raw

Re: [PATCH 3/5] KVM: RISC-V: remove unnecessary SBI reset state

In reply to Radim Krčmář
Quoted 8 lines · Radim Krčmář

The SBI reset state has only two variables -- pc and a1. The rest is known, so keep only the necessary information.

The reset structures make sense if we want userspace to control the reset state (which we do), but I'd still remove them now and reintroduce with the userspace interface later -- we could probably have just a single reset state per VM, instead of a reset state for each VCPU.

The SBI spec does not define the reset state of CPUs. The SBI implementations (aka KVM RISC-V or OpenSBI) or platform firmwares are free to clear additional registers as part system reset or CPU.

As part of resetting the VCPU, the in-kernel KVM clears all the registers.

The setting of PC, A0, and A1 is only an entry condition defined for CPUs brought-up using SBI HSM start or SBI System suspend.

We should not go ahead with this patch.

Regards,
Anup

Quoted 197 lines
---
 arch/riscv/include/asm/kvm_aia.h  |  3 --
 arch/riscv/include/asm/kvm_host.h | 12 ++++---
 arch/riscv/kvm/aia_device.c       |  4 +--
 arch/riscv/kvm/vcpu.c             | 58 +++++++++++++++++--------------
 arch/riscv/kvm/vcpu_sbi.c         |  9 +++--
 5 files changed, 44 insertions(+), 42 deletions(-)
arch/riscv/include/asm/kvm_aia.h +0 -3
index 1f37b600ca47..3b643b9efc07 100644
@@ -63,9 +63,6 @@ struct kvm_vcpu_aia {
        /* CPU AIA CSR context of Guest VCPU */
        struct kvm_vcpu_aia_csr guest_csr;
 
-       /* CPU AIA CSR context upon Guest VCPU reset */
-       struct kvm_vcpu_aia_csr guest_reset_csr;
-
        /* Guest physical address of IMSIC for this VCPU */
        gpa_t           imsic_addr;
 
arch/riscv/include/asm/kvm_host.h +7 -5
index 0e9c2fab6378..0c8c9c05af91 100644
@@ -193,6 +193,12 @@ struct kvm_vcpu_smstateen_csr {
        unsigned long sstateen0;
 };
 
+struct kvm_vcpu_reset_state {
+       spinlock_t lock;
+       unsigned long pc;
+       unsigned long a1;
+};
+
 struct kvm_vcpu_arch {
        /* VCPU ran at least once */
        bool ran_atleast_once;
@@ -227,12 +233,8 @@ struct kvm_vcpu_arch {
        /* CPU Smstateen CSR context of Guest VCPU */
        struct kvm_vcpu_smstateen_csr smstateen_csr;
 
-       /* CPU context upon Guest VCPU reset */
-       struct kvm_cpu_context guest_reset_context;
-       spinlock_t reset_cntx_lock;
+       struct kvm_vcpu_reset_state reset_state;
 
-       /* CPU CSR context upon Guest VCPU reset */
-       struct kvm_vcpu_csr guest_reset_csr;
 
        /*
         * VCPU interrupts
arch/riscv/kvm/aia_device.c +1 -3
index 39cd26af5a69..43e472ff3e1a 100644
@@ -526,12 +526,10 @@ int kvm_riscv_vcpu_aia_update(struct kvm_vcpu *vcpu)
 void kvm_riscv_vcpu_aia_reset(struct kvm_vcpu *vcpu)
 {
        struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
-       struct kvm_vcpu_aia_csr *reset_csr =
-                               &vcpu->arch.aia_context.guest_reset_csr;
 
        if (!kvm_riscv_aia_available())
                return;
-       memcpy(csr, reset_csr, sizeof(*csr));
+       memset(csr, 0, sizeof(*csr));
 
        /* Proceed only if AIA was initialized successfully */
        if (!kvm_riscv_aia_initialized(vcpu->kvm))
arch/riscv/kvm/vcpu.c +32 -26
index 2fb75288ecfe..b8485c1c1ce4 100644
@@ -51,13 +51,40 @@ const struct kvm_stats_header kvm_vcpu_stats_header = {
                       sizeof(kvm_vcpu_stats_desc),
 };
 
-static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
+static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)
 {
        struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
-       struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr;
        struct kvm_cpu_context *cntx = &vcpu->arch.guest_context;
-       struct kvm_cpu_context *reset_cntx = &vcpu->arch.guest_reset_context;
+       struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state;
        void *vector_datap = cntx->vector.datap;
+
+       memset(cntx, 0, sizeof(*cntx));
+       memset(csr, 0, sizeof(*csr));
+
+       /* Restore datap as it's not a part of the guest context. */
+       cntx->vector.datap = vector_datap;
+
+       /* Load SBI reset values */
+       cntx->a0 = vcpu->vcpu_id;
+
+       spin_lock(&reset_state->lock);
+       cntx->sepc = reset_state->pc;
+       cntx->a1 = reset_state->a1;
+       spin_unlock(&reset_state->lock);
+
+       /* Setup reset state of shadow SSTATUS and HSTATUS CSRs */
+       cntx->sstatus = SR_SPP | SR_SPIE;
+
+       cntx->hstatus |= HSTATUS_VTW;
+       cntx->hstatus |= HSTATUS_SPVP;
+       cntx->hstatus |= HSTATUS_SPV;
+
+       /* By default, make CY, TM, and IR counters accessible in VU mode */
+       csr->scounteren = 0x7;
+}
+
+static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
+{
        bool loaded;
 
        /**
@@ -72,16 +99,10 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
 
        vcpu->arch.last_exit_cpu = -1;
 
-       memcpy(csr, reset_csr, sizeof(*csr));
-
-       spin_lock(&vcpu->arch.reset_cntx_lock);
-       memcpy(cntx, reset_cntx, sizeof(*cntx));
-       spin_unlock(&vcpu->arch.reset_cntx_lock);
+       kvm_riscv_vcpu_context_reset(vcpu);
 
        kvm_riscv_vcpu_fp_reset(vcpu);
 
-       /* Restore datap as it's not a part of the guest context. */
-       cntx->vector.datap = vector_datap;
        kvm_riscv_vcpu_vector_reset(vcpu);
 
        kvm_riscv_vcpu_timer_reset(vcpu);
@@ -113,8 +134,6 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
 int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 {
        int rc;
-       struct kvm_cpu_context *cntx;
-       struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr;
 
        spin_lock_init(&vcpu->arch.mp_state_lock);
 
@@ -134,24 +153,11 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
        /* Setup VCPU hfence queue */
        spin_lock_init(&vcpu->arch.hfence_lock);
 
-       /* Setup reset state of shadow SSTATUS and HSTATUS CSRs */
-       spin_lock_init(&vcpu->arch.reset_cntx_lock);
-
-       spin_lock(&vcpu->arch.reset_cntx_lock);
-       cntx = &vcpu->arch.guest_reset_context;
-       cntx->sstatus = SR_SPP | SR_SPIE;
-       cntx->hstatus = 0;
-       cntx->hstatus |= HSTATUS_VTW;
-       cntx->hstatus |= HSTATUS_SPVP;
-       cntx->hstatus |= HSTATUS_SPV;
-       spin_unlock(&vcpu->arch.reset_cntx_lock);
+       spin_lock_init(&vcpu->arch.reset_state.lock);
 
        if (kvm_riscv_vcpu_alloc_vector_context(vcpu))
                return -ENOMEM;
 
-       /* By default, make CY, TM, and IR counters accessible in VU mode */
-       reset_csr->scounteren = 0x7;
-
        /* Setup VCPU timer */
        kvm_riscv_vcpu_timer_init(vcpu);
 
arch/riscv/kvm/vcpu_sbi.c +4 -6
index f58368f7df1d..3d7955e05cc3 100644
@@ -159,11 +159,10 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1)
 {
-       spin_lock(&vcpu->arch.reset_cntx_lock);
-       vcpu->arch.guest_reset_context.sepc = pc;
-       vcpu->arch.guest_reset_context.a0 = vcpu->vcpu_id;
-       vcpu->arch.guest_reset_context.a1 = a1;
-       spin_unlock(&vcpu->arch.reset_cntx_lock);
+       spin_lock(&vcpu->arch.reset_state.lock);
+       vcpu->arch.reset_state.pc = pc;
+       vcpu->arch.reset_state.a1 = a1;
+       spin_unlock(&vcpu->arch.reset_state.lock);
 
        kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
 }
--

2.48.1

#12 lore raw

Re: [PATCH 3/5] KVM: RISC-V: remove unnecessary SBI reset state

2025-04-28T17:46:01+05:30, Anup Patel <[email protected]>:
Quoted 10 lines · Radim Krčmář
Quoted 8 lines

The SBI reset state has only two variables -- pc and a1. The rest is known, so keep only the necessary information.

The reset structures make sense if we want userspace to control the reset state (which we do), but I'd still remove them now and reintroduce with the userspace interface later -- we could probably have just a single reset state per VM, instead of a reset state for each VCPU.

Quoted 7 lines

The SBI spec does not define the reset state of CPUs. The SBI implementations (aka KVM RISC-V or OpenSBI) or platform firmwares are free to clear additional registers as part system reset or CPU.

As part of resetting the VCPU, the in-kernel KVM clears all the registers.

Yes, but instead of doing a simple memset(0), KVM carriers around a lot of data with minimal information value. Reset is not really a fast path, so I think it would be good to have the code there as simple as possible.

Quoted 2 lines

The setting of PC, A0, and A1 is only an entry condition defined for CPUs brought-up using SBI HSM start or SBI System suspend.

That is why this patch has to add kvm_vcpu_reset_state, to remember the state of pc and a1. (a0 is hart id and can be figured out.)

Quoted 1 line

We should not go ahead with this patch.

This patch only does refactoring. Do you think the current reset structures are better?

Thanks.

#13 lore raw

Re: [PATCH 3/5] KVM: RISC-V: remove unnecessary SBI reset state

Quoted 28 lines · Radim Krčmář
2025-04-28T17:46:01+05:30, Anup Patel <[email protected]>:
Quoted 10 lines · Radim Krčmář
Quoted 8 lines

The SBI reset state has only two variables -- pc and a1. The rest is known, so keep only the necessary information.

The reset structures make sense if we want userspace to control the reset state (which we do), but I'd still remove them now and reintroduce with the userspace interface later -- we could probably have just a single reset state per VM, instead of a reset state for each VCPU.

Quoted 7 lines

The SBI spec does not define the reset state of CPUs. The SBI implementations (aka KVM RISC-V or OpenSBI) or platform firmwares are free to clear additional registers as part system reset or CPU.

As part of resetting the VCPU, the in-kernel KVM clears all the registers.

Yes, but instead of doing a simple memset(0), KVM carriers around a lot of data with minimal information value. Reset is not really a fast path, so I think it would be good to have the code there as simple as possible.

Quoted 2 lines

The setting of PC, A0, and A1 is only an entry condition defined for CPUs brought-up using SBI HSM start or SBI System suspend.

Quoted 8 lines

That is why this patch has to add kvm_vcpu_reset_state, to remember the state of pc and a1. (a0 is hart id and can be figured out.)

Quoted 1 line

We should not go ahead with this patch.

This patch only does refactoring. Do you think the current reset structures are better?

I am fine getting rid of reset structures as long as we have memset(0) in-place to achieve the same thing.

Regards,
Anup

maintainer
#14 lore raw

Re: [PATCH 3/5] KVM: RISC-V: remove unnecessary SBI reset state

In reply to Radim Krčmář
Quoted 10 lines · Radim Krčmář

The SBI reset state has only two variables -- pc and a1. The rest is known, so keep only the necessary information.

The reset structures make sense if we want userspace to control the reset state (which we do), but I'd still remove them now and reintroduce with the userspace interface later -- we could probably have just a single reset state per VM, instead of a reset state for each VCPU.

Queued this patch for Linux-6.16

Thanks,
Anup

Quoted 195 lines
---
 arch/riscv/include/asm/kvm_aia.h  |  3 --
 arch/riscv/include/asm/kvm_host.h | 12 ++++---
 arch/riscv/kvm/aia_device.c       |  4 +--
 arch/riscv/kvm/vcpu.c             | 58 +++++++++++++++++--------------
 arch/riscv/kvm/vcpu_sbi.c         |  9 +++--
 5 files changed, 44 insertions(+), 42 deletions(-)
arch/riscv/include/asm/kvm_aia.h +0 -3
index 1f37b600ca47..3b643b9efc07 100644
@@ -63,9 +63,6 @@ struct kvm_vcpu_aia {
        /* CPU AIA CSR context of Guest VCPU */
        struct kvm_vcpu_aia_csr guest_csr;
 
-       /* CPU AIA CSR context upon Guest VCPU reset */
-       struct kvm_vcpu_aia_csr guest_reset_csr;
-
        /* Guest physical address of IMSIC for this VCPU */
        gpa_t           imsic_addr;
 
arch/riscv/include/asm/kvm_host.h +7 -5
index 0e9c2fab6378..0c8c9c05af91 100644
@@ -193,6 +193,12 @@ struct kvm_vcpu_smstateen_csr {
        unsigned long sstateen0;
 };
 
+struct kvm_vcpu_reset_state {
+       spinlock_t lock;
+       unsigned long pc;
+       unsigned long a1;
+};
+
 struct kvm_vcpu_arch {
        /* VCPU ran at least once */
        bool ran_atleast_once;
@@ -227,12 +233,8 @@ struct kvm_vcpu_arch {
        /* CPU Smstateen CSR context of Guest VCPU */
        struct kvm_vcpu_smstateen_csr smstateen_csr;
 
-       /* CPU context upon Guest VCPU reset */
-       struct kvm_cpu_context guest_reset_context;
-       spinlock_t reset_cntx_lock;
+       struct kvm_vcpu_reset_state reset_state;
 
-       /* CPU CSR context upon Guest VCPU reset */
-       struct kvm_vcpu_csr guest_reset_csr;
 
        /*
         * VCPU interrupts
arch/riscv/kvm/aia_device.c +1 -3
index 39cd26af5a69..43e472ff3e1a 100644
@@ -526,12 +526,10 @@ int kvm_riscv_vcpu_aia_update(struct kvm_vcpu *vcpu)
 void kvm_riscv_vcpu_aia_reset(struct kvm_vcpu *vcpu)
 {
        struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
-       struct kvm_vcpu_aia_csr *reset_csr =
-                               &vcpu->arch.aia_context.guest_reset_csr;
 
        if (!kvm_riscv_aia_available())
                return;
-       memcpy(csr, reset_csr, sizeof(*csr));
+       memset(csr, 0, sizeof(*csr));
 
        /* Proceed only if AIA was initialized successfully */
        if (!kvm_riscv_aia_initialized(vcpu->kvm))
arch/riscv/kvm/vcpu.c +32 -26
index 2fb75288ecfe..b8485c1c1ce4 100644
@@ -51,13 +51,40 @@ const struct kvm_stats_header kvm_vcpu_stats_header = {
                       sizeof(kvm_vcpu_stats_desc),
 };
 
-static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
+static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)
 {
        struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
-       struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr;
        struct kvm_cpu_context *cntx = &vcpu->arch.guest_context;
-       struct kvm_cpu_context *reset_cntx = &vcpu->arch.guest_reset_context;
+       struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state;
        void *vector_datap = cntx->vector.datap;
+
+       memset(cntx, 0, sizeof(*cntx));
+       memset(csr, 0, sizeof(*csr));
+
+       /* Restore datap as it's not a part of the guest context. */
+       cntx->vector.datap = vector_datap;
+
+       /* Load SBI reset values */
+       cntx->a0 = vcpu->vcpu_id;
+
+       spin_lock(&reset_state->lock);
+       cntx->sepc = reset_state->pc;
+       cntx->a1 = reset_state->a1;
+       spin_unlock(&reset_state->lock);
+
+       /* Setup reset state of shadow SSTATUS and HSTATUS CSRs */
+       cntx->sstatus = SR_SPP | SR_SPIE;
+
+       cntx->hstatus |= HSTATUS_VTW;
+       cntx->hstatus |= HSTATUS_SPVP;
+       cntx->hstatus |= HSTATUS_SPV;
+
+       /* By default, make CY, TM, and IR counters accessible in VU mode */
+       csr->scounteren = 0x7;
+}
+
+static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
+{
        bool loaded;
 
        /**
@@ -72,16 +99,10 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
 
        vcpu->arch.last_exit_cpu = -1;
 
-       memcpy(csr, reset_csr, sizeof(*csr));
-
-       spin_lock(&vcpu->arch.reset_cntx_lock);
-       memcpy(cntx, reset_cntx, sizeof(*cntx));
-       spin_unlock(&vcpu->arch.reset_cntx_lock);
+       kvm_riscv_vcpu_context_reset(vcpu);
 
        kvm_riscv_vcpu_fp_reset(vcpu);
 
-       /* Restore datap as it's not a part of the guest context. */
-       cntx->vector.datap = vector_datap;
        kvm_riscv_vcpu_vector_reset(vcpu);
 
        kvm_riscv_vcpu_timer_reset(vcpu);
@@ -113,8 +134,6 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
 int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 {
        int rc;
-       struct kvm_cpu_context *cntx;
-       struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr;
 
        spin_lock_init(&vcpu->arch.mp_state_lock);
 
@@ -134,24 +153,11 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
        /* Setup VCPU hfence queue */
        spin_lock_init(&vcpu->arch.hfence_lock);
 
-       /* Setup reset state of shadow SSTATUS and HSTATUS CSRs */
-       spin_lock_init(&vcpu->arch.reset_cntx_lock);
-
-       spin_lock(&vcpu->arch.reset_cntx_lock);
-       cntx = &vcpu->arch.guest_reset_context;
-       cntx->sstatus = SR_SPP | SR_SPIE;
-       cntx->hstatus = 0;
-       cntx->hstatus |= HSTATUS_VTW;
-       cntx->hstatus |= HSTATUS_SPVP;
-       cntx->hstatus |= HSTATUS_SPV;
-       spin_unlock(&vcpu->arch.reset_cntx_lock);
+       spin_lock_init(&vcpu->arch.reset_state.lock);
 
        if (kvm_riscv_vcpu_alloc_vector_context(vcpu))
                return -ENOMEM;
 
-       /* By default, make CY, TM, and IR counters accessible in VU mode */
-       reset_csr->scounteren = 0x7;
-
        /* Setup VCPU timer */
        kvm_riscv_vcpu_timer_init(vcpu);
 
arch/riscv/kvm/vcpu_sbi.c +4 -6
index f58368f7df1d..3d7955e05cc3 100644
@@ -159,11 +159,10 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1)
 {
-       spin_lock(&vcpu->arch.reset_cntx_lock);
-       vcpu->arch.guest_reset_context.sepc = pc;
-       vcpu->arch.guest_reset_context.a0 = vcpu->vcpu_id;
-       vcpu->arch.guest_reset_context.a1 = a1;
-       spin_unlock(&vcpu->arch.reset_cntx_lock);
+       spin_lock(&vcpu->arch.reset_state.lock);
+       vcpu->arch.reset_state.pc = pc;
+       vcpu->arch.reset_state.a1 = a1;
+       spin_unlock(&vcpu->arch.reset_state.lock);
 
        kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
 }
--

2.48.1

#15 lore raw

Re: [PATCH 3/5] KVM: RISC-V: remove unnecessary SBI reset state

2025-05-08T11:48:00+05:30, Anup Patel <[email protected]>:
Quoted 5 lines · Radim Krčmář
Quoted 4 lines

The SBI reset state has only two variables -- pc and a1. The rest is known, so keep only the necessary information.

Quoted 8 lines
Quoted 6 lines

The reset structures make sense if we want userspace to control the reset state (which we do), but I'd still remove them now and reintroduce with the userspace interface later -- we could probably have just a single reset state per VM, instead of a reset state for each VCPU.

Queued this patch for Linux-6.16

[5/5] was already applied, which means that [3/5] would be nicer with

  memset(&vcpu->arch.smstateen_csr, 0, sizeof(vcpu->arch.smstateen_csr));

in the new function (kvm_riscv_vcpu_context_reset) where we memset(0) the other csr context.

Should I add a patch to do that in v2?

Thanks.

maintainer
#16 lore raw

Re: [PATCH 3/5] KVM: RISC-V: remove unnecessary SBI reset state

Quoted 16 lines · Radim Krčmář
2025-05-08T11:48:00+05:30, Anup Patel <[email protected]>:
Quoted 5 lines · Radim Krčmář
Quoted 4 lines

The SBI reset state has only two variables -- pc and a1. The rest is known, so keep only the necessary information.

Quoted 8 lines
Quoted 6 lines

The reset structures make sense if we want userspace to control the reset state (which we do), but I'd still remove them now and reintroduce with the userspace interface later -- we could probably have just a single reset state per VM, instead of a reset state for each VCPU.

Queued this patch for Linux-6.16

Quoted 8 lines

[5/5] was already applied, which means that [3/5] would be nicer with

  memset(&vcpu->arch.smstateen_csr, 0, sizeof(vcpu->arch.smstateen_csr));

in the new function (kvm_riscv_vcpu_context_reset) where we memset(0) the other csr context.

Should I add a patch to do that in v2?

Yes, please add it to your v2. I will update my queue accordingly.

Regards,
Anup

#17 lore raw

PATCH 4/5 KVM: RISC-V: reset VCPU state when becoming runnable

In reply to Radim Krčmář

Beware, this patch is "breaking" the userspace interface, because it fixes a KVM/QEMU bug where the boot VCPU is not being reset by KVM.

The VCPU reset paths are inconsistent right now. KVM resets VCPUs that are brought up by KVM-accelerated SBI calls, but does nothing for VCPUs brought up through ioctls.

We need to perform a KVM reset even when the VCPU is started through an ioctl. This patch is one of the ways we can achieve it.

Assume that userspace has no business setting the post-reset state. KVM is de-facto the SBI implementation, as the SBI HSM acceleration cannot be disabled and userspace cannot control the reset state, so KVM should be in full control of the post-reset state.

Do not reset the pc and a1 registers, because SBI reset is expected to provide them and KVM has no idea what these registers should be -- only the userspace knows where it put the data.

An important consideration is resume. Userspace might want to start with non-reset state. Check ran_atleast_once to allow this, because KVM-SBI HSM creates some VCPUs as STOPPED.

The drawback is that userspace can still start the boot VCPU with an incorrect reset state, because there is no way to distinguish a freshly reset new VCPU on the KVM side (userspace might set some values by mistake) from a restored VCPU (userspace must set all values).

The advantage of this solution is that it fixes current QEMU and makes some sense with the assumption that KVM implements SBI HSM. I do not like it too much, so I'd be in favor of a different solution if we can still afford to drop support for current userspaces.

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

---
 arch/riscv/include/asm/kvm_host.h     |  1 +
 arch/riscv/include/asm/kvm_vcpu_sbi.h |  3 +++
 arch/riscv/kvm/vcpu.c                 |  9 +++++++++
 arch/riscv/kvm/vcpu_sbi.c             | 21 +++++++++++++++++++--
 4 files changed, 32 insertions(+), 2 deletions(-)
arch/riscv/include/asm/kvm_host.h +1 -0
index 0c8c9c05af91..9bbf8c4a286b 100644
@@ -195,6 +195,7 @@ struct kvm_vcpu_smstateen_csr {
 
 struct kvm_vcpu_reset_state {
 	spinlock_t lock;
+	bool active;
 	unsigned long pc;
 	unsigned long a1;
 };
arch/riscv/include/asm/kvm_vcpu_sbi.h +3 -0
index aaaa81355276..2c334a87e02a 100644
@@ -57,6 +57,9 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 				     u32 type, u64 flags);
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1);
+void __kvm_riscv_vcpu_set_reset_state(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1);
+void kvm_riscv_vcpu_sbi_request_reset_from_userspace(struct kvm_vcpu *vcpu);
 int kvm_riscv_vcpu_sbi_return(struct kvm_vcpu *vcpu, struct kvm_run *run);
 int kvm_riscv_vcpu_set_reg_sbi_ext(struct kvm_vcpu *vcpu,
 				   const struct kvm_one_reg *reg);
arch/riscv/kvm/vcpu.c +9 -0
index b8485c1c1ce4..4578863a39e3 100644
@@ -58,6 +58,11 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)
 	struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state;
 	void *vector_datap = cntx->vector.datap;
 
+	spin_lock(&reset_state->lock);
+	if (!reset_state->active)
+		__kvm_riscv_vcpu_set_reset_state(vcpu, cntx->sepc, cntx->a1);
+	spin_unlock(&reset_state->lock);
+
 	memset(cntx, 0, sizeof(*cntx));
 	memset(csr, 0, sizeof(*csr));
 
@@ -520,6 +525,10 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 
 	switch (mp_state->mp_state) {
 	case KVM_MP_STATE_RUNNABLE:
+		if (riscv_vcpu_supports_sbi_ext(vcpu, KVM_RISCV_SBI_EXT_HSM) &&
+				vcpu->arch.ran_atleast_once &&
+				kvm_riscv_vcpu_stopped(vcpu))
+			kvm_riscv_vcpu_sbi_request_reset_from_userspace(vcpu);
 		WRITE_ONCE(vcpu->arch.mp_state, *mp_state);
 		break;
 	case KVM_MP_STATE_STOPPED:
arch/riscv/kvm/vcpu_sbi.c +19 -2
index 3d7955e05cc3..77f9f0bd3842 100644
@@ -156,12 +156,29 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 	run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
 }
 
+/* must be called with held vcpu->arch.reset_state.lock */
+void __kvm_riscv_vcpu_set_reset_state(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1)
+{
+	vcpu->arch.reset_state.active = true;
+	vcpu->arch.reset_state.pc = pc;
+	vcpu->arch.reset_state.a1 = a1;
+}
+
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1)
 {
 	spin_lock(&vcpu->arch.reset_state.lock);
-	vcpu->arch.reset_state.pc = pc;
-	vcpu->arch.reset_state.a1 = a1;
+	__kvm_riscv_vcpu_set_reset_state(vcpu, pc, a1);
+	spin_unlock(&vcpu->arch.reset_state.lock);
+
+	kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
+}
+
+void kvm_riscv_vcpu_sbi_request_reset_from_userspace(struct kvm_vcpu *vcpu)
+{
+	spin_lock(&vcpu->arch.reset_state.lock);
+	vcpu->arch.reset_state.active = false;
 	spin_unlock(&vcpu->arch.reset_state.lock);
 
 	kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
Signature · 1 line
2.48.1
#18 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

Quoted 6 lines · Radim Krčmář

Beware, this patch is "breaking" the userspace interface, because it fixes a KVM/QEMU bug where the boot VCPU is not being reset by KVM.

The VCPU reset paths are inconsistent right now. KVM resets VCPUs that are brought up by KVM-accelerated SBI calls, but does nothing for VCPUs brought up through ioctls.

I guess we currently expect userspace to make a series of set-one-reg ioctls in order to prepare ("reset") newly created vcpus, and I guess the problem is that KVM isn't capturing the resulting configuration in order to replay it when SBI HSM reset is invoked by the guest. But, instead of capture-replay we could just exit to userspace on an SBI HSM reset call and let userspace repeat what it did at vcpu-create time.

Quoted 4 lines

We need to perform a KVM reset even when the VCPU is started through an ioctl. This patch is one of the ways we can achieve it.

Quoted 8 lines

Assume that userspace has no business setting the post-reset state. KVM is de-facto the SBI implementation, as the SBI HSM acceleration cannot be disabled and userspace cannot control the reset state, so KVM should be in full control of the post-reset state.

Do not reset the pc and a1 registers, because SBI reset is expected to provide them and KVM has no idea what these registers should be -- only the userspace knows where it put the data.

s/userspace/guest/

Quoted 9 lines

An important consideration is resume. Userspace might want to start with non-reset state. Check ran_atleast_once to allow this, because KVM-SBI HSM creates some VCPUs as STOPPED.

The drawback is that userspace can still start the boot VCPU with an incorrect reset state, because there is no way to distinguish a freshly reset new VCPU on the KVM side (userspace might set some values by mistake) from a restored VCPU (userspace must set all values).

If there's a correct vs. incorrect reset state that KVM needs to enforce, then we'll need a different API than just a bunch of set-one-reg calls, or set/get-one-reg should be WARL for userpace.

Quoted 8 lines

The advantage of this solution is that it fixes current QEMU and makes some sense with the assumption that KVM implements SBI HSM. I do not like it too much, so I'd be in favor of a different solution if we can still afford to drop support for current userspaces.

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand.

That's what the change to kvm_arch_vcpu_ioctl_set_mpstate() in this patch is providing, right?

Quoted 2 lines

I think it would also be much better
if userspace was in control of the post-reset state.

Agreed. Can we just exit to userspace on SBI HSM reset?

Thanks,
drew

Quoted 101 lines
---
 arch/riscv/include/asm/kvm_host.h     |  1 +
 arch/riscv/include/asm/kvm_vcpu_sbi.h |  3 +++
 arch/riscv/kvm/vcpu.c                 |  9 +++++++++
 arch/riscv/kvm/vcpu_sbi.c             | 21 +++++++++++++++++++--
 4 files changed, 32 insertions(+), 2 deletions(-)
arch/riscv/include/asm/kvm_host.h +1 -0
index 0c8c9c05af91..9bbf8c4a286b 100644
@@ -195,6 +195,7 @@ struct kvm_vcpu_smstateen_csr {
 
 struct kvm_vcpu_reset_state {
 	spinlock_t lock;
+	bool active;
 	unsigned long pc;
 	unsigned long a1;
 };
arch/riscv/include/asm/kvm_vcpu_sbi.h +3 -0
index aaaa81355276..2c334a87e02a 100644
@@ -57,6 +57,9 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 				     u32 type, u64 flags);
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1);
+void __kvm_riscv_vcpu_set_reset_state(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1);
+void kvm_riscv_vcpu_sbi_request_reset_from_userspace(struct kvm_vcpu *vcpu);
 int kvm_riscv_vcpu_sbi_return(struct kvm_vcpu *vcpu, struct kvm_run *run);
 int kvm_riscv_vcpu_set_reg_sbi_ext(struct kvm_vcpu *vcpu,
 				   const struct kvm_one_reg *reg);
arch/riscv/kvm/vcpu.c +9 -0
index b8485c1c1ce4..4578863a39e3 100644
@@ -58,6 +58,11 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)
 	struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state;
 	void *vector_datap = cntx->vector.datap;
 
+	spin_lock(&reset_state->lock);
+	if (!reset_state->active)
+		__kvm_riscv_vcpu_set_reset_state(vcpu, cntx->sepc, cntx->a1);
+	spin_unlock(&reset_state->lock);
+
 	memset(cntx, 0, sizeof(*cntx));
 	memset(csr, 0, sizeof(*csr));
 
@@ -520,6 +525,10 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 
 	switch (mp_state->mp_state) {
 	case KVM_MP_STATE_RUNNABLE:
+		if (riscv_vcpu_supports_sbi_ext(vcpu, KVM_RISCV_SBI_EXT_HSM) &&
+				vcpu->arch.ran_atleast_once &&
+				kvm_riscv_vcpu_stopped(vcpu))
+			kvm_riscv_vcpu_sbi_request_reset_from_userspace(vcpu);
 		WRITE_ONCE(vcpu->arch.mp_state, *mp_state);
 		break;
 	case KVM_MP_STATE_STOPPED:
arch/riscv/kvm/vcpu_sbi.c +19 -2
index 3d7955e05cc3..77f9f0bd3842 100644
@@ -156,12 +156,29 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
 	run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
 }
 
+/* must be called with held vcpu->arch.reset_state.lock */
+void __kvm_riscv_vcpu_set_reset_state(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1)
+{
+	vcpu->arch.reset_state.active = true;
+	vcpu->arch.reset_state.pc = pc;
+	vcpu->arch.reset_state.a1 = a1;
+}
+
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1)
 {
 	spin_lock(&vcpu->arch.reset_state.lock);
-	vcpu->arch.reset_state.pc = pc;
-	vcpu->arch.reset_state.a1 = a1;
+	__kvm_riscv_vcpu_set_reset_state(vcpu, pc, a1);
+	spin_unlock(&vcpu->arch.reset_state.lock);
+
+	kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
+}
+
+void kvm_riscv_vcpu_sbi_request_reset_from_userspace(struct kvm_vcpu *vcpu)
+{
+	spin_lock(&vcpu->arch.reset_state.lock);
+	vcpu->arch.reset_state.active = false;
 	spin_unlock(&vcpu->arch.reset_state.lock);
 
 	kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
Signature · 1 line
2.48.1
#19 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

2025-04-25T15:26:08+02:00, Andrew Jones <[email protected]>:
Quoted 10 lines · Radim Krčmář
Quoted 6 lines

Beware, this patch is "breaking" the userspace interface, because it fixes a KVM/QEMU bug where the boot VCPU is not being reset by KVM.

The VCPU reset paths are inconsistent right now. KVM resets VCPUs that are brought up by KVM-accelerated SBI calls, but does nothing for VCPUs brought up through ioctls.

I guess we currently expect userspace to make a series of set-one-reg ioctls in order to prepare ("reset") newly created vcpus,

Userspace should currently get-one-reg a freshly reset VCPU to know what KVM SBI decides is the correct reset. Userspace shouldn't set-one-reg anything other than what KVM decides, hence we can currently just let KVM do it.

Quoted 3 lines

and I guess
the problem is that KVM isn't capturing the resulting configuration in order to replay it when SBI HSM reset is invoked by the guest.

That can also be a solution, but it's not possible to capture the desired reset state with current IOCTLs, because the first run of a VCPU can just as well be a resume from a mid-execution.

Quoted 4 lines

But,
instead of capture-replay we could just exit to userspace on an SBI HSM reset call and let userspace repeat what it did at vcpu-create time.

Right, I like the idea. (It doesn't fix current userspaces, though.)

Quoted 8 lines
Quoted 8 lines

We need to perform a KVM reset even when the VCPU is started through an ioctl. This patch is one of the ways we can achieve it.

Assume that userspace has no business setting the post-reset state. KVM is de-facto the SBI implementation, as the SBI HSM acceleration cannot be disabled and userspace cannot control the reset state, so KVM should be in full control of the post-reset state.

Quoted 5 lines
Quoted 3 lines

Do not reset the pc and a1 registers, because SBI reset is expected to provide them and KVM has no idea what these registers should be -- only the userspace knows where it put the data.

s/userspace/guest/

Both are correct... I should have made the context clearer here. I meant the initial hart boot, where userspace loads code/dt and sets pc/a1 to them.

Quoted 4 lines
Quoted 4 lines

An important consideration is resume. Userspace might want to start with non-reset state. Check ran_atleast_once to allow this, because KVM-SBI HSM creates some VCPUs as STOPPED.

Quoted 8 lines
Quoted 4 lines

The drawback is that userspace can still start the boot VCPU with an incorrect reset state, because there is no way to distinguish a freshly reset new VCPU on the KVM side (userspace might set some values by mistake) from a restored VCPU (userspace must set all values).

If there's a correct vs. incorrect reset state that KVM needs to enforce, then we'll need a different API than just a bunch of set-one-reg calls, or set/get-one-reg should be WARL for userpace.

Incorrect might have been too strong word... while the SBI reset state is technically UNSPECIFIED, I think it's just asking for bugs if the harts have different initial states based on their reset method.

Quoted 1 line

set/get-one-reg should be WARL for userpace.

WAAAA! :)

Quoted 10 lines
Quoted 7 lines

The advantage of this solution is that it fixes current QEMU and makes some sense with the assumption that KVM implements SBI HSM. I do not like it too much, so I'd be in favor of a different solution if we can still afford to drop support for current userspaces.

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand.

That's what the change to kvm_arch_vcpu_ioctl_set_mpstate() in this patch is providing, right?

It does. With conditions to be as compatible as possible.

Quoted 4 lines
Quoted 2 lines

I think it would also be much better
if userspace was in control of the post-reset state.

Agreed. Can we just exit to userspace on SBI HSM reset?

Yes. (It needs an userspace toggle if we care about backward-compatiblity, though.)

How much do we want to fix/break current userspaces?

Thanks.

maintainer
#20 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

In reply to Radim Krčmář
Quoted 29 lines · Radim Krčmář

Beware, this patch is "breaking" the userspace interface, because it fixes a KVM/QEMU bug where the boot VCPU is not being reset by KVM.

The VCPU reset paths are inconsistent right now. KVM resets VCPUs that are brought up by KVM-accelerated SBI calls, but does nothing for VCPUs brought up through ioctls.

We need to perform a KVM reset even when the VCPU is started through an ioctl. This patch is one of the ways we can achieve it.

Assume that userspace has no business setting the post-reset state. KVM is de-facto the SBI implementation, as the SBI HSM acceleration cannot be disabled and userspace cannot control the reset state, so KVM should be in full control of the post-reset state.

Do not reset the pc and a1 registers, because SBI reset is expected to provide them and KVM has no idea what these registers should be -- only the userspace knows where it put the data.

An important consideration is resume. Userspace might want to start with non-reset state. Check ran_atleast_once to allow this, because KVM-SBI HSM creates some VCPUs as STOPPED.

The drawback is that userspace can still start the boot VCPU with an incorrect reset state, because there is no way to distinguish a freshly reset new VCPU on the KVM side (userspace might set some values by mistake) from a restored VCPU (userspace must set all values).

Quoted 8 lines

The advantage of this solution is that it fixes current QEMU and makes some sense with the assumption that KVM implements SBI HSM. I do not like it too much, so I'd be in favor of a different solution if we can still afford to drop support for current userspaces.

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

Due to above, reasons we should not go ahead with this patch.

Regards,
Anup

Quoted 101 lines
---
 arch/riscv/include/asm/kvm_host.h     |  1 +
 arch/riscv/include/asm/kvm_vcpu_sbi.h |  3 +++
 arch/riscv/kvm/vcpu.c                 |  9 +++++++++
 arch/riscv/kvm/vcpu_sbi.c             | 21 +++++++++++++++++++--
 4 files changed, 32 insertions(+), 2 deletions(-)
arch/riscv/include/asm/kvm_host.h +1 -0
index 0c8c9c05af91..9bbf8c4a286b 100644
@@ -195,6 +195,7 @@ struct kvm_vcpu_smstateen_csr {
 
 struct kvm_vcpu_reset_state {
        spinlock_t lock;
+       bool active;
        unsigned long pc;
        unsigned long a1;
 };
arch/riscv/include/asm/kvm_vcpu_sbi.h +3 -0
index aaaa81355276..2c334a87e02a 100644
@@ -57,6 +57,9 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
                                     u32 type, u64 flags);
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1);
+void __kvm_riscv_vcpu_set_reset_state(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1);
+void kvm_riscv_vcpu_sbi_request_reset_from_userspace(struct kvm_vcpu *vcpu);
 int kvm_riscv_vcpu_sbi_return(struct kvm_vcpu *vcpu, struct kvm_run *run);
 int kvm_riscv_vcpu_set_reg_sbi_ext(struct kvm_vcpu *vcpu,
                                   const struct kvm_one_reg *reg);
arch/riscv/kvm/vcpu.c +9 -0
index b8485c1c1ce4..4578863a39e3 100644
@@ -58,6 +58,11 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)
        struct kvm_vcpu_reset_state *reset_state = &vcpu->arch.reset_state;
        void *vector_datap = cntx->vector.datap;
 
+       spin_lock(&reset_state->lock);
+       if (!reset_state->active)
+               __kvm_riscv_vcpu_set_reset_state(vcpu, cntx->sepc, cntx->a1);
+       spin_unlock(&reset_state->lock);
+
        memset(cntx, 0, sizeof(*cntx));
        memset(csr, 0, sizeof(*csr));
 
@@ -520,6 +525,10 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 
        switch (mp_state->mp_state) {
        case KVM_MP_STATE_RUNNABLE:
+               if (riscv_vcpu_supports_sbi_ext(vcpu, KVM_RISCV_SBI_EXT_HSM) &&
+                               vcpu->arch.ran_atleast_once &&
+                               kvm_riscv_vcpu_stopped(vcpu))
+                       kvm_riscv_vcpu_sbi_request_reset_from_userspace(vcpu);
                WRITE_ONCE(vcpu->arch.mp_state, *mp_state);
                break;
        case KVM_MP_STATE_STOPPED:
arch/riscv/kvm/vcpu_sbi.c +19 -3
index 3d7955e05cc3..77f9f0bd3842 100644
@@ -156,12 +156,29 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
        run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
 }
 
+/* must be called with held vcpu->arch.reset_state.lock */
+void __kvm_riscv_vcpu_set_reset_state(struct kvm_vcpu *vcpu,
+                                      unsigned long pc, unsigned long a1)
+{
+       vcpu->arch.reset_state.active = true;
+       vcpu->arch.reset_state.pc = pc;
+       vcpu->arch.reset_state.a1 = a1;
+}
+
 void kvm_riscv_vcpu_sbi_request_reset(struct kvm_vcpu *vcpu,
                                       unsigned long pc, unsigned long a1)
 {
        spin_lock(&vcpu->arch.reset_state.lock);
-       vcpu->arch.reset_state.pc = pc;
-       vcpu->arch.reset_state.a1 = a1;
+       __kvm_riscv_vcpu_set_reset_state(vcpu, pc, a1);
+       spin_unlock(&vcpu->arch.reset_state.lock);
+
+       kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
+}
+
+void kvm_riscv_vcpu_sbi_request_reset_from_userspace(struct kvm_vcpu *vcpu)
+{
+       spin_lock(&vcpu->arch.reset_state.lock);
+       vcpu->arch.reset_state.active = false;
        spin_unlock(&vcpu->arch.reset_state.lock);
 
        kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
--

2.48.1

#21 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

What would be your preferred interface for that?

Thanks.

#22 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

Quoted 23 lines · Radim Krčmář
2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

Quoted 9 lines

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

What would be your preferred interface for that?

Instead of creating a new interface, I would prefer that VCPU which initiates SBI System Reset should be resetted immediately in-kernel space before forwarding the system reset request to user space. This way we also force KVM user-space to explicitly set the PC, A0, and A1 before running the VCPU again after system reset.

Regards,
Anup

#23 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

2025-04-29T11:25:35+05:30, Anup Patel <[email protected]>:
Quoted 31 lines · Radim Krčmář
Quoted 30 lines
2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

Quoted 7 lines
Quoted 2 lines

What would be your preferred interface for that?

Instead of creating a new interface, I would prefer that VCPU which initiates SBI System Reset should be resetted immediately in-kernel space before forwarding the system reset request to user space.

The initiating VCPU might not be the boot VCPU.
It would be safer to reset all of them.

You also previously mentioned that we need to preserve the pre-reset state for userspace, which I completely agree with and it is why the reset happens later.

Quoted 3 lines

This way we also force KVM user-space to explicitly
set the PC, A0, and A1 before running the VCPU again after system reset.

We also want to consider reset from emulation outside of KVM.

There is a "simple" solution that covers everything (except speed) -- the userspace can tear down the whole VM and re-create it. Do we want to do this instead and drop all resets from KVM?

maintainer
#24 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

Quoted 36 lines · Radim Krčmář
2025-04-29T11:25:35+05:30, Anup Patel <[email protected]>:
Quoted 34 lines · Radim Krčmář
Quoted 32 lines
2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

What would be your preferred interface for that?

Quoted 7 lines
Quoted 4 lines

Instead of creating a new interface, I would prefer that VCPU which initiates SBI System Reset should be resetted immediately in-kernel space before forwarding the system reset request to user space.

The initiating VCPU might not be the boot VCPU.
It would be safer to reset all of them.

I meant initiating VCPU and not the boot VCPU. Currently, the non-initiating VCPUs are already resetted by VCPU requests so nothing special needs to be done.

Quoted 4 lines

You also previously mentioned that we need to preserve the pre-reset state for userspace, which I completely agree with and it is why the reset happens later.

Yes, that was only for debug purposes from user space. At the moment, there is no one using this for debug purposes so we can sacrifice that.

Quoted 10 lines
Quoted 4 lines

This way we also force KVM user-space to explicitly
set the PC, A0, and A1 before running the VCPU again after system reset.

We also want to consider reset from emulation outside of KVM.

There is a "simple" solution that covers everything (except speed) -- the userspace can tear down the whole VM and re-create it. Do we want to do this instead and drop all resets from KVM?

I think we should keep the VCPU resets in KVM so that handling of system reset handling in user space remains simple. The user space can also re-create the VM upon system reset but that is user space choice.

Regards,
Anup

#25 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

2025-04-29T20:31:18+05:30, Anup Patel <[email protected]>:
Quoted 42 lines · Radim Krčmář
Quoted 41 lines
2025-04-29T11:25:35+05:30, Anup Patel <[email protected]>:
Quoted 38 lines · Radim Krčmář
Quoted 23 lines
2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

Quoted 9 lines

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

What would be your preferred interface for that?

Instead of creating a new interface, I would prefer that VCPU which initiates SBI System Reset should be resetted immediately in-kernel space before forwarding the system reset request to user space.

Quoted 6 lines
Quoted 2 lines

The initiating VCPU might not be the boot VCPU.
It would be safer to reset all of them.

I meant initiating VCPU and not the boot VCPU. Currently, the non-initiating VCPUs are already resetted by VCPU requests so nothing special needs to be done.

Currently, we make the request only for VCPUs brought up by HSM -- the non-boot VCPUs. There is a single VCPU not being reset and resetting the reset initiating VCPU changes nothing. e.g.

1) VCPU 1 initiates the reset through an ecall.
2) All VCPUs are stopped and return to userspace.
3) Userspace prepares VCPU 0 as the boot VCPU.
4) VCPU 0 executes without going through KVM reset paths.

The point of this patch is to reset the boot VCPU, so we reset the VCPU that is made runnable by the KVM_SET_MP_STATE IOCTL.

For design alternatives, it is also possible to reset immediately in an IOCTL instead of making the reset request.

Quoted 7 lines
Quoted 3 lines

You also previously mentioned that we need to preserve the pre-reset state for userspace, which I completely agree with and it is why the reset happens later.

Yes, that was only for debug purposes from user space. At the moment, there is no one using this for debug purposes so we can sacrifice that.

We still can't immediately reset the boot VCPU, because it might already be in userspace. We don't really benefit from immediately resetting the initiating VCPU.
Also, making the reset request for all VCPUs from the initiating VCPU has some undesirable race conditions we would have to prevent, so I do prefer we go the IOCTL reset way.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

This way we also force KVM user-space to explicitly
set the PC, A0, and A1 before running the VCPU again after system reset.

We also want to consider reset from emulation outside of KVM.

Quoted 8 lines
Quoted 3 lines

There is a "simple" solution that covers everything (except speed) -- the userspace can tear down the whole VM and re-create it. Do we want to do this instead and drop all resets from KVM?

I think we should keep the VCPU resets in KVM so that handling of system reset handling in user space remains simple. The user space can also re-create the VM upon system reset but that is user space choice.

Ok.

maintainer
#26 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

Quoted 50 lines · Radim Krčmář
2025-04-29T20:31:18+05:30, Anup Patel <[email protected]>:
Quoted 48 lines · Radim Krčmář
Quoted 36 lines
2025-04-29T11:25:35+05:30, Anup Patel <[email protected]>:
Quoted 34 lines · Radim Krčmář
Quoted 32 lines
2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

What would be your preferred interface for that?

Quoted 7 lines
Quoted 4 lines

Instead of creating a new interface, I would prefer that VCPU which initiates SBI System Reset should be resetted immediately in-kernel space before forwarding the system reset request to user space.

The initiating VCPU might not be the boot VCPU.
It would be safer to reset all of them.

I meant initiating VCPU and not the boot VCPU. Currently, the non-initiating VCPUs are already resetted by VCPU requests so nothing special needs to be done.

There is no designated boot VCPU for KVM so let us only use the term "initiating" or "non-initiating" VCPUs in context of system reset.

Quoted 7 lines

Currently, we make the request only for VCPUs brought up by HSM -- the non-boot VCPUs. There is a single VCPU not being reset and resetting the reset initiating VCPU changes nothing. e.g.

1) VCPU 1 initiates the reset through an ecall.
2) All VCPUs are stopped and return to userspace.

When all VCPUs are stopped, all VCPUs except VCPU1 (in this example) will SLEEP because we do "kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_SLEEP)" so none of the VCPUs except VCPU1 (in this case) will return to userspace.

Quoted 2 lines

3) Userspace prepares VCPU 0 as the boot VCPU.
4) VCPU 0 executes without going through KVM reset paths.

Userspace will see a system reset event exit for the initiating VCPU by that time all other VCPUs are already sleeping with mp_state == KVM_MP_STATE_STOPPED.

Quoted 3 lines

The point of this patch is to reset the boot VCPU, so we reset the VCPU that is made runnable by the KVM_SET_MP_STATE IOCTL.

Like I said before, we don't need to do this. The initiating VCPU can be resetted just before exiting to user space for system reset event exit.

Quoted 12 lines

For design alternatives, it is also possible to reset immediately in an IOCTL instead of making the reset request.

Quoted 7 lines
Quoted 3 lines

You also previously mentioned that we need to preserve the pre-reset state for userspace, which I completely agree with and it is why the reset happens later.

Yes, that was only for debug purposes from user space. At the moment, there is no one using this for debug purposes so we can sacrifice that.

Quoted 6 lines

We still can't immediately reset the boot VCPU, because it might already be in userspace. We don't really benefit from immediately resetting the initiating VCPU.
Also, making the reset request for all VCPUs from the initiating VCPU has some undesirable race conditions we would have to prevent, so I do prefer we go the IOCTL reset way.

All VCPUs are sleeping with mp_state == KVM_MP_STATE_STOPPED when userspace sees system reset exit on the initiating VCPU so I don't see any race condition if we also reset the initiating VCPU before exiting to userspace.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

This way we also force KVM user-space to explicitly
set the PC, A0, and A1 before running the VCPU again after system reset.

We also want to consider reset from emulation outside of KVM.

There is a "simple" solution that covers everything (except speed) -- the userspace can tear down the whole VM and re-create it. Do we want to do this instead and drop all resets from KVM?

Quoted 6 lines
Quoted 4 lines

I think we should keep the VCPU resets in KVM so that handling of system reset handling in user space remains simple. The user space can also re-create the VM upon system reset but that is user space choice.

Ok.

Regards,
Anup

maintainer
#27 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

Quoted 78 lines · Anup Patel
Quoted 50 lines · Radim Krčmář
2025-04-29T20:31:18+05:30, Anup Patel <[email protected]>:
Quoted 48 lines · Radim Krčmář
Quoted 36 lines
2025-04-29T11:25:35+05:30, Anup Patel <[email protected]>:
Quoted 34 lines · Radim Krčmář
Quoted 32 lines
2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

What would be your preferred interface for that?

Quoted 7 lines
Quoted 4 lines

Instead of creating a new interface, I would prefer that VCPU which initiates SBI System Reset should be resetted immediately in-kernel space before forwarding the system reset request to user space.

The initiating VCPU might not be the boot VCPU.
It would be safer to reset all of them.

I meant initiating VCPU and not the boot VCPU. Currently, the non-initiating VCPUs are already resetted by VCPU requests so nothing special needs to be done.

There is no designated boot VCPU for KVM so let us only use the term "initiating" or "non-initiating" VCPUs in context of system reset.

Quoted 7 lines

Currently, we make the request only for VCPUs brought up by HSM -- the non-boot VCPUs. There is a single VCPU not being reset and resetting the reset initiating VCPU changes nothing. e.g.

1) VCPU 1 initiates the reset through an ecall.
2) All VCPUs are stopped and return to userspace.

When all VCPUs are stopped, all VCPUs except VCPU1 (in this example) will SLEEP because we do "kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_SLEEP)" so none of the VCPUs except VCPU1 (in this case) will return to userspace.

Quoted 2 lines

3) Userspace prepares VCPU 0 as the boot VCPU.
4) VCPU 0 executes without going through KVM reset paths.

Userspace will see a system reset event exit for the initiating VCPU by that time all other VCPUs are already sleeping with mp_state == KVM_MP_STATE_STOPPED.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

The point of this patch is to reset the boot VCPU, so we reset the VCPU that is made runnable by the KVM_SET_MP_STATE IOCTL.

Like I said before, we don't need to do this. The initiating VCPU can be resetted just before exiting to user space for system reset event exit.

Below is what I am suggesting. This change completely removes dependency of kvm_sbi_hsm_vcpu_start() on "reset" structures.

(quoted diff) +0 -0

b/arch/riscv/include/asm/kvm_host.h
index 0e9c2fab6378..6bd12469852d 100644

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -396,6 +396,7 @@ int kvm_riscv_vcpu_get_reg(struct kvm_vcpu *vcpu,
 int kvm_riscv_vcpu_set_reg(struct kvm_vcpu *vcpu,
                const struct kvm_one_reg *reg);
 
+void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu);
 int kvm_riscv_vcpu_set_interrupt(struct kvm_vcpu *vcpu, unsigned int irq);
 int kvm_riscv_vcpu_unset_interrupt(struct kvm_vcpu *vcpu, unsigned int irq);
 void kvm_riscv_vcpu_flush_interrupts(struct kvm_vcpu *vcpu);
arch/riscv/kvm/vcpu.c +1 -1
index 02635bac91f1..801c6a1a1aef 100644
@@ -51,7 +51,7 @@ const struct kvm_stats_header kvm_vcpu_stats_header = {
                sizeof(kvm_vcpu_stats_desc),
 };
 
-static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
+void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
 {
     struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
     struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr;
@@ -689,6 +689,9 @@ static void kvm_riscv_check_vcpu_requests(struct

kvm_vcpu *vcpu)

     struct rcuwait *wait = kvm_arch_vcpu_get_wait(vcpu);

     if (kvm_request_pending(vcpu)) {
+        if (kvm_check_request(KVM_REQ_VCPU_RESET, vcpu))
+            kvm_riscv_reset_vcpu(vcpu);
+
         if (kvm_check_request(KVM_REQ_SLEEP, vcpu)) {
             kvm_vcpu_srcu_read_unlock(vcpu);
             rcuwait_wait_event(wait,
@@ -705,9 +708,6 @@ static void kvm_riscv_check_vcpu_requests(struct
kvm_vcpu *vcpu)
             }
         }

- if (kvm_check_request(KVM_REQ_VCPU_RESET, vcpu))
- kvm_riscv_reset_vcpu(vcpu);
-
if (kvm_check_request(KVM_REQ_UPDATE_HGATP, vcpu))
kvm_riscv_gstage_update_hgatp(vcpu);

arch/riscv/kvm/vcpu_sbi.c +7 -1
index d1c83a77735e..79477e7f240a 100644
@@ -146,9 +146,15 @@ void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
         spin_lock(&vcpu->arch.mp_state_lock);
         WRITE_ONCE(tmp->arch.mp_state.mp_state, KVM_MP_STATE_STOPPED);
         spin_unlock(&vcpu->arch.mp_state_lock);
+        if (tmp != vcpu) {
+            kvm_make_request(KVM_REQ_SLEEP | KVM_REQ_VCPU_RESET, vcpu);
+            kvm_vcpu_kick(vcpu);
+        } else {
+            kvm_make_request(KVM_REQ_SLEEP, vcpu);
+        }
     }
-    kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_SLEEP);
 
+    kvm_riscv_reset_vcpu(vcpu);
     memset(&run->system_event, 0, sizeof(run->system_event));
     run->system_event.type = type;
     run->system_event.ndata = 1;
arch/riscv/kvm/vcpu_sbi_hsm.c +7 -11
index 3070bb31745d..30d7d59db5a5 100644
@@ -15,15 +15,15 @@
 
 static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu)
 {
-    struct kvm_cpu_context *reset_cntx;
-    struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
-    struct kvm_vcpu *target_vcpu;
+    struct kvm_cpu_context *target_cp, *cp = &vcpu->arch.guest_context;
     unsigned long target_vcpuid = cp->a0;
+    struct kvm_vcpu *target_vcpu;
     int ret = 0;
 
     target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid);
     if (!target_vcpu)
         return SBI_ERR_INVALID_PARAM;
+    target_cp = &target_vcpu->arch.guest_context;
 
     spin_lock(&target_vcpu->arch.mp_state_lock);
 
@@ -32,17 +32,12 @@ static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu)
         goto out;
     }
 
-    spin_lock(&target_vcpu->arch.reset_cntx_lock);
-    reset_cntx = &target_vcpu->arch.guest_reset_context;
     /* start address */
-    reset_cntx->sepc = cp->a1;
+    target_cp->sepc = cp->a1;
     /* target vcpu id to start */
-    reset_cntx->a0 = target_vcpuid;
+    target_cp->a0 = target_vcpuid;
     /* private data passed from kernel */
-    reset_cntx->a1 = cp->a2;
-    spin_unlock(&target_vcpu->arch.reset_cntx_lock);
-
-    kvm_make_request(KVM_REQ_VCPU_RESET, target_vcpu);
+    target_cp->a1 = cp->a2;
 
     __kvm_riscv_vcpu_power_on(target_vcpu);
 
@@ -63,6 +58,7 @@ static int kvm_sbi_hsm_vcpu_stop(struct kvm_vcpu *vcpu)
         goto out;
     }
 
+    kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
     __kvm_riscv_vcpu_power_off(vcpu);
 
 out:

Regards,
Anup

#28 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

2025-04-30T10:56:35+05:30, Anup Patel <[email protected]>:
Quoted 50 lines · Anup Patel
Quoted 49 lines · Radim Krčmář
Quoted 47 lines
2025-04-29T20:31:18+05:30, Anup Patel <[email protected]>:
Quoted 45 lines · Radim Krčmář
Quoted 43 lines
2025-04-29T11:25:35+05:30, Anup Patel <[email protected]>:
Quoted 31 lines · Radim Krčmář
Quoted 30 lines
2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

Quoted 7 lines
Quoted 2 lines

What would be your preferred interface for that?

Instead of creating a new interface, I would prefer that VCPU which initiates SBI System Reset should be resetted immediately in-kernel space before forwarding the system reset request to user space.

The initiating VCPU might not be the boot VCPU.
It would be safer to reset all of them.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

I meant initiating VCPU and not the boot VCPU. Currently, the non-initiating VCPUs are already resetted by VCPU requests so nothing special needs to be done.

There is no designated boot VCPU for KVM so let us only use the term "initiating" or "non-initiating" VCPUs in context of system reset.

That is exactly how I use it. Some VCPU will be the boot VCPU (the VCPU made runnable by KVM_SET_MP_STATE) and loaded with state from userspace.

RISC-V doesn't guarantee that the boot VCPU is the reset initiating VCPU, so I think KVM should allow it.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Currently, we make the request only for VCPUs brought up by HSM -- the non-boot VCPUs. There is a single VCPU not being reset and resetting the reset initiating VCPU changes nothing. e.g.

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

1) VCPU 1 initiates the reset through an ecall.
2) All VCPUs are stopped and return to userspace.

When all VCPUs are stopped, all VCPUs except VCPU1 (in this example) will SLEEP because we do "kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_SLEEP)" so none of the VCPUs except VCPU1 (in this case) will return to userspace.

Userspace should be able to do whatever it likes -- in my example, all the VCPUs are brought to userspace and a different boot VCPU is selected.

(Perhaps userspace wanted to record their reset pre-reset state, or
maybe it really wants to boot with a designated VCPU.)

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

3) Userspace prepares VCPU 0 as the boot VCPU.
4) VCPU 0 executes without going through KVM reset paths.

Userspace will see a system reset event exit for the initiating VCPU by that time all other VCPUs are already sleeping with mp_state == KVM_MP_STATE_STOPPED.

Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

The point of this patch is to reset the boot VCPU, so we reset the VCPU that is made runnable by the KVM_SET_MP_STATE IOCTL.

Like I said before, we don't need to do this. The initiating VCPU can be resetted just before exiting to user space for system reset event exit.

You assume initiating VCPU == boot VCPU.

We should prevent KVM_SET_MP_STATE IOCTL for all non-initiating VCPUs if we decide to accept the assumption.

I'd rather choose a different design, though.

How about a new userspace interface for IOCTL reset?
(Can be capability toggle for KVM_SET_MP_STATE or a straight new IOCTL.)

That wouldn't "fix" current userspaces, but would significantly improve the sanity of the KVM interface.

Quoted 2 lines

Below is what I am suggesting. This change completely removes dependency of kvm_sbi_hsm_vcpu_start() on "reset" structures.

I'd keep the reset structure in this series -- it's small enough and locklessly accessing the state of another VCPU needs a lot of consideration to prevent all possible race conditions.

Thanks.

maintainer
#29 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

Quoted 56 lines · Radim Krčmář
2025-04-30T10:56:35+05:30, Anup Patel <[email protected]>:
Quoted 54 lines · Anup Patel
Quoted 53 lines · Radim Krčmář
Quoted 50 lines
2025-04-29T20:31:18+05:30, Anup Patel <[email protected]>:
Quoted 48 lines · Radim Krčmář
Quoted 36 lines
2025-04-29T11:25:35+05:30, Anup Patel <[email protected]>:
Quoted 34 lines · Radim Krčmář
Quoted 32 lines
2025-04-28T17:52:25+05:30, Anup Patel <[email protected]>:
Quoted 16 lines · Radim Krčmář
Quoted 3 lines

For a cleaner solution, we should add interfaces to perform the KVM-SBI reset request on userspace demand. I think it would also be much better if userspace was in control of the post-reset state.

Apart from breaking KVM user-space, this patch is incorrect and does not align with the:
1) SBI spec
2) OS boot protocol.

The SBI spec only defines the entry state of certain CPU registers (namely, PC, A0, and A1) when CPU enters S-mode:
1) Upon SBI HSM start call from some other CPU
2) Upon resuming from non-retentive SBI HSM suspend or SBI system suspend

Quoted 4 lines

The S-mode entry state of the boot CPU is defined by the OS boot protocol and not by the SBI spec. Due to this, reason KVM RISC-V expects user-space to set up the S-mode entry state of the boot CPU upon system reset.

We can handle the initial state consistency in other patches. What needs addressing is a way to trigger the KVM reset from userspace, even if only to clear the internal KVM state.

I think mp_state is currently the best signalization that KVM should reset, so I added it there.

What would be your preferred interface for that?

Quoted 7 lines
Quoted 4 lines

Instead of creating a new interface, I would prefer that VCPU which initiates SBI System Reset should be resetted immediately in-kernel space before forwarding the system reset request to user space.

The initiating VCPU might not be the boot VCPU.
It would be safer to reset all of them.

I meant initiating VCPU and not the boot VCPU. Currently, the non-initiating VCPUs are already resetted by VCPU requests so nothing special needs to be done.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

There is no designated boot VCPU for KVM so let us only use the term "initiating" or "non-initiating" VCPUs in context of system reset.

That is exactly how I use it. Some VCPU will be the boot VCPU (the VCPU made runnable by KVM_SET_MP_STATE) and loaded with state from userspace.

RISC-V doesn't guarantee that the boot VCPU is the reset initiating VCPU, so I think KVM should allow it.

We do allow any VCPU to be the boot VCPU. I am not sure why you think otherwise.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Currently, we make the request only for VCPUs brought up by HSM -- the non-boot VCPUs. There is a single VCPU not being reset and resetting the reset initiating VCPU changes nothing. e.g.

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

1) VCPU 1 initiates the reset through an ecall.
2) All VCPUs are stopped and return to userspace.

When all VCPUs are stopped, all VCPUs except VCPU1 (in this example) will SLEEP because we do "kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_SLEEP)" so none of the VCPUs except VCPU1 (in this case) will return to userspace.

Quoted 3 lines

Userspace should be able to do whatever it likes -- in my example, all the VCPUs are brought to userspace and a different boot VCPU is selected.

In your example, the VCPU1 (initiating VCPU) need not be the boot VCPU after system reset. The user space can setup some other VCPU as boot VCPU (by setting its MPSTATE, PC, A0, and A1) and simply do ioctl_run() for the initiating VCPU without changing its MPSTATE.

Quoted 15 lines

(Perhaps userspace wanted to record their reset pre-reset state, or
maybe it really wants to boot with a designated VCPU.)

Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

3) Userspace prepares VCPU 0 as the boot VCPU.
4) VCPU 0 executes without going through KVM reset paths.

Userspace will see a system reset event exit for the initiating VCPU by that time all other VCPUs are already sleeping with mp_state == KVM_MP_STATE_STOPPED.

Quoted 3 lines

The point of this patch is to reset the boot VCPU, so we reset the VCPU that is made runnable by the KVM_SET_MP_STATE IOCTL.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

Like I said before, we don't need to do this. The initiating VCPU can be resetted just before exiting to user space for system reset event exit.

You assume initiating VCPU == boot VCPU.

We should prevent KVM_SET_MP_STATE IOCTL for all non-initiating VCPUs if we decide to accept the assumption.

There is no such assumption.

Quoted 8 lines

I'd rather choose a different design, though.

How about a new userspace interface for IOCTL reset?
(Can be capability toggle for KVM_SET_MP_STATE or a straight new IOCTL.)

That wouldn't "fix" current userspaces, but would significantly improve the sanity of the KVM interface.

I believe the current implementation needs a few improvements that's all. We certainly don't need to introduce any new IOCTL.

Also, keep in mind that so far we have avoided any RISC-V specific KVM IOCTLs and we should try to keep it that way as long as we can.

Regards,
Anup

#30 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

2025-04-30T15:47:13+05:30, Anup Patel <[email protected]>:
Quoted 11 lines · Radim Krčmář
Quoted 10 lines
2025-04-30T10:56:35+05:30, Anup Patel <[email protected]>:
Quoted 8 lines · Anup Patel
Quoted 7 lines · Radim Krčmář
Quoted 2 lines

The point of this patch is to reset the boot VCPU, so we reset the VCPU that is made runnable by the KVM_SET_MP_STATE IOCTL.

Like I said before, we don't need to do this. The initiating VCPU can be resetted just before exiting to user space for system reset event exit.

Quoted 6 lines
Quoted 4 lines

You assume initiating VCPU == boot VCPU.

We should prevent KVM_SET_MP_STATE IOCTL for all non-initiating VCPUs if we decide to accept the assumption.

There is no such assumption.

You probably haven't intended it:

1) VCPU 0 is "chilling" in userspace.
2) VCPU 1 initiates SBI reset.
3) VCPU 1 makes a reset request to VCPU 0.
4) VCPU 1 returns to userspace.
5) Userspace knows it should reset the VM.
6) VCPU 0 still hasn't entered KVM.
7) Userspace sets the initial state of VCPU 0 and enters KVM.
8) VCPU 0 is reset in KVM, because of the pending request.
9) The initial boot state from userspace is lost.

Quoted 10 lines
Quoted 7 lines

I'd rather choose a different design, though.

How about a new userspace interface for IOCTL reset?
(Can be capability toggle for KVM_SET_MP_STATE or a straight new IOCTL.)

That wouldn't "fix" current userspaces, but would significantly improve the sanity of the KVM interface.

I believe the current implementation needs a few improvements that's all. We certainly don't need to introduce any new IOCTL.

I do too. The whole patch could have been a single line:

arch/riscv/kvm/vcpu.c +1 -0
index d3d957a9e5c4..b3e6ad87e1cd 100644
@@ -511,6 +511,7 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 
 	switch (mp_state->mp_state) {
 	case KVM_MP_STATE_RUNNABLE:
+		kvm_riscv_reset_vcpu(vcpu);
 		WRITE_ONCE(vcpu->arch.mp_state, *mp_state);
 		break;
 	case KVM_MP_STATE_STOPPED:

It is the backward compatibility and trying to fix current userspaces that's making it ugly. I already gave up on the latter, so we can have a decently clean solution with the former.

Quoted 3 lines

Also, keep in mind that so far we have avoided any RISC-V specific KVM IOCTLs and we should try to keep it that way as long as we can.

We can re-use KVM_SET_MP_STATE and add a KVM capability. Userspace will opt-in to reset the VCPU through the existing IOCTL.

This design will also allow userspace to trigger a VCPU reset without tearing down the whole VM.

maintainer
#31 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

Quoted 66 lines · Radim Krčmář
2025-04-30T15:47:13+05:30, Anup Patel <[email protected]>:
Quoted 11 lines · Radim Krčmář
Quoted 10 lines
2025-04-30T10:56:35+05:30, Anup Patel <[email protected]>:
Quoted 8 lines · Anup Patel
Quoted 7 lines · Radim Krčmář
Quoted 2 lines

The point of this patch is to reset the boot VCPU, so we reset the VCPU that is made runnable by the KVM_SET_MP_STATE IOCTL.

Like I said before, we don't need to do this. The initiating VCPU can be resetted just before exiting to user space for system reset event exit.

Quoted 6 lines
Quoted 4 lines

You assume initiating VCPU == boot VCPU.

We should prevent KVM_SET_MP_STATE IOCTL for all non-initiating VCPUs if we decide to accept the assumption.

There is no such assumption.

You probably haven't intended it:

1) VCPU 0 is "chilling" in userspace.
2) VCPU 1 initiates SBI reset.
3) VCPU 1 makes a reset request to VCPU 0.
4) VCPU 1 returns to userspace.
5) Userspace knows it should reset the VM.
6) VCPU 0 still hasn't entered KVM.
7) Userspace sets the initial state of VCPU 0 and enters KVM.
8) VCPU 0 is reset in KVM, because of the pending request.
9) The initial boot state from userspace is lost.

Quoted 10 lines
Quoted 7 lines

I'd rather choose a different design, though.

How about a new userspace interface for IOCTL reset?
(Can be capability toggle for KVM_SET_MP_STATE or a straight new IOCTL.)

That wouldn't "fix" current userspaces, but would significantly improve the sanity of the KVM interface.

I believe the current implementation needs a few improvements that's all. We certainly don't need to introduce any new IOCTL.

I do too. The whole patch could have been a single line:

arch/riscv/kvm/vcpu.c +1 -0
index d3d957a9e5c4..b3e6ad87e1cd 100644
@@ -511,6 +511,7 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 
        switch (mp_state->mp_state) {
        case KVM_MP_STATE_RUNNABLE:
+               kvm_riscv_reset_vcpu(vcpu);
                WRITE_ONCE(vcpu->arch.mp_state, *mp_state);
                break;
        case KVM_MP_STATE_STOPPED:

It is the backward compatibility and trying to fix current userspaces that's making it ugly. I already gave up on the latter, so we can have a decently clean solution with the former.

Quoted 3 lines

Also, keep in mind that so far we have avoided any RISC-V specific KVM IOCTLs and we should try to keep it that way as long as we can.

Quoted 5 lines

We can re-use KVM_SET_MP_STATE and add a KVM capability. Userspace will opt-in to reset the VCPU through the existing IOCTL.

This design will also allow userspace to trigger a VCPU reset without tearing down the whole VM.

Okay, lets go ahead with a KVM capability which user space can opt-in for KVM_SET_MP_STATE ioctl().

Keep in mind that at runtime Guest can still do CPU hotplug using SBI HSM start/stop and do system suspend using SBI SUSP so we should continue to have VCPU reset requests for both these SBI extensions.

Regards,
Anup

#32 lore raw

Re: [PATCH 4/5] KVM: RISC-V: reset VCPU state when becoming runnable

2025-04-30T18:32:31+05:30, Anup Patel <[email protected]>:
Quoted 7 lines · Radim Krčmář
Quoted 5 lines

We can re-use KVM_SET_MP_STATE and add a KVM capability. Userspace will opt-in to reset the VCPU through the existing IOCTL.

This design will also allow userspace to trigger a VCPU reset without tearing down the whole VM.

Quoted 6 lines

Okay, lets go ahead with a KVM capability which user space can opt-in for KVM_SET_MP_STATE ioctl().

Keep in mind that at runtime Guest can still do CPU hotplug using SBI HSM start/stop and do system suspend using SBI SUSP so we should continue to have VCPU reset requests for both these SBI extensions.

Will do, thanks.

#33 lore raw

PATCH 5/5 KVM: RISC-V: reset smstateen CSRs

In reply to Radim Krčmář

Not resetting smstateen is a potential security hole, because VU might be able to access state that VS does not properly context-switch.

Fixes 81f0f314fec9 ("RISCV: KVM: Add sstateen0 context save/restore")
---
 arch/riscv/kvm/vcpu.c | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/kvm/vcpu.c +1 -0
index 4578863a39e3..ac0fa50bc489 100644
@@ -65,6 +65,7 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)
 
 	memset(cntx, 0, sizeof(*cntx));
 	memset(csr, 0, sizeof(*csr));
+	memset(&vcpu->arch.smstateen_csr, 0, sizeof(vcpu->arch.smstateen_csr));
 
 	/* Restore datap as it's not a part of the guest context. */
 	cntx->vector.datap = vector_datap;
Signature · 1 line
2.48.1
maintainer
#34 lore raw

Re: [PATCH 5/5] KVM: RISC-V: reset smstateen CSRs

Quoted 6 lines · Radim Krčmář

Not resetting smstateen is a potential security hole, because VU might be able to access state that VS does not properly context-switch.

Fixes 81f0f314fec9 ("RISCV: KVM: Add sstateen0 context save/restore")

This should be the first patch of the series which can be applied independently.

I have rebased and queued it as a fix for Linux-6.15.

Regards,
Anup

Quoted 19 lines
---
 arch/riscv/kvm/vcpu.c | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/kvm/vcpu.c +1 -1
index 4578863a39e3..ac0fa50bc489 100644
@@ -65,6 +65,7 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu)
 
        memset(cntx, 0, sizeof(*cntx));
        memset(csr, 0, sizeof(*csr));
+       memset(&vcpu->arch.smstateen_csr, 0, sizeof(vcpu->arch.smstateen_csr));
 
        /* Restore datap as it's not a part of the guest context. */
        cntx->vector.datap = vector_datap;
--

2.48.1