Skip to contentSkip to conversation

RISC-V: KVM: Delegate illegal instruction fault

PATCH Superseded Xu Lu 3 participants 13 messages Jun 20, 2025 – Jun 23, 2025

Subsystems (3)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Xu Lu

author
#0 lore raw

PATCH 1/1 RISC-V: KVM: Delegate illegal instruction fault

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---
 arch/riscv/include/asm/kvm_host.h | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/include/asm/kvm_host.h +1 -0
index 85cfebc32e4cf..97cc2c0dba73a 100644
@@ -48,6 +48,7 @@
 					 BIT(EXC_SYSCALL)         | \
 					 BIT(EXC_INST_PAGE_FAULT) | \
 					 BIT(EXC_LOAD_PAGE_FAULT) | \
+					 BIT(EXC_INST_ILLEGAL)    | \
 					 BIT(EXC_STORE_PAGE_FAULT))
 
 #define KVM_HIDELEG_DEFAULT		(BIT(IRQ_VS_SOFT)  | \
Signature · 1 line
2.20.1
#1 lore raw
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:

arch/riscv/include/asm/kvm_host.h line 48

Quoted 8 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+					 BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Thanks.

author
#2 lore raw

Hi Radim,

Quoted 5 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:
Quoted 3 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

Quoted 7 lines

arch/riscv/include/asm/kvm_host.h line 48

Quoted 5 lines

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                      BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

I only want to delegate it by default. And KVM may still want to delegate different exceptions for different VMs like what it does for EXC_BREAKPOINT. So maybe it is better to reserve these codes?

Quoted 4 lines

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Thanks for the reminder. I will have a test and resend the patch if it works.

Quoted 2 lines

Thanks.

Best Regards,
Xu Lu

#3 lore raw
2025-06-22T18:11:49+08:00, Xu Lu <[email protected]>:
Quoted 16 lines

Hi Radim,

Quoted 5 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:
Quoted 3 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

Quoted 7 lines

arch/riscv/include/asm/kvm_host.h line 48

Quoted 5 lines

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                      BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

Quoted 3 lines

I only want to delegate it by default. And KVM may still want to delegate different exceptions for different VMs like what it does for EXC_BREAKPOINT.

(I think we could easily reintroduce the code if KVM wants to do that in
the future. I also think that it's bad that this patch is doing an observable change without userspace involvement -- the counting of KVM SBI PMU events, but others will probably disagree with me on this.)

Quoted 1 line

So maybe it is better to reserve these codes?

Possibly, the current is acceptable if you have considered the implications on PMU events.

Quoted 5 lines
Quoted 3 lines

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Thanks for the reminder. I will have a test and resend the patch if it works.

The misaligned exceptions are already being worked on, so don't waste your time on them, sorry.

author
#4 lore raw

Hi Radim,

Quoted 27 lines · Radim Krčmář
2025-06-22T18:11:49+08:00, Xu Lu <[email protected]>:
Quoted 16 lines

Hi Radim,

Quoted 5 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:
Quoted 3 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

Quoted 7 lines

arch/riscv/include/asm/kvm_host.h line 48

Quoted 5 lines

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                      BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

Quoted 3 lines

I only want to delegate it by default. And KVM may still want to delegate different exceptions for different VMs like what it does for EXC_BREAKPOINT.

(I think we could easily reintroduce the code if KVM wants to do that in
the future. I also think that it's bad that this patch is doing an observable change without userspace involvement -- the counting of KVM SBI PMU events, but others will probably disagree with me on this.)

Quoted 4 lines
Quoted 1 line

So maybe it is better to reserve these codes?

Possibly, the current is acceptable if you have considered the implications on PMU events.

So maybe it comes back to our discussion on the difference between vs insn fault and illegal insn fault again~ In my personal opinion, it seems to be a waste of CPU resources to trap illegal instruction to HS-mode hypervisor, which does nothing but redirect it back to VS-mode guest kernel. I think it is OK (and maybe it should) to record 0 illegal instruction exits in KVM PMU. If someone wants illegal insn to trigger an vcpu exit, then an ioctl can be provided to remove the delegation like what KVM_SET_GUEST_DEBUG does.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Thanks for the reminder. I will have a test and resend the patch if it works.

The misaligned exceptions are already being worked on, so don't waste your time on them, sorry.

Thanks for the reminder too. I did not consider this before. I will leave the MISALIGNED faults alone.

Best Regards,

Xu Lu

#5 lore raw
In reply to Radim Krčmář
Quoted 12 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:

arch/riscv/include/asm/kvm_host.h line 48

Quoted 8 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+					 BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

Quoted 3 lines

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default.

Thanks,

Clément

Quoted 7 lines

Thanks.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#6 lore raw
2025-06-23T10:04:45+02:00, Clément Léger <[email protected]>:
Quoted 10 lines · Radim Krčmář
Quoted 3 lines

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default.

Yeah, I forgot about your patches that conflict with the change, thanks.

(The current KVM exception handler only forwards all the listed
exceptions, so the only observable difference this change would make is that the KVM SBI PMU event is not counted.)

author
#7 lore raw
In reply to Clément Léger

Hi Clément,

Quoted 20 lines · Clément Léger
Quoted 12 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:

arch/riscv/include/asm/kvm_host.h line 48

Quoted 8 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                     BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

Quoted 3 lines

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Quoted 5 lines

Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default.

It doesn't matter whether these exceptions are delegated in medeleg. KVM in HS-mode does not handle illegal instruction or misaligned access and only redirects them back to VS-mode. Delegating such exceptions in hedeleg helps save CPU usage even when they are not delegated in medeleg: opensbi will check whether these exceptions are delegated to VS-mode and redirect them to VS-mode if possible. There seems to be no conflicts with SSE implementation. Please correct me if I missed anything.

Best Regards,
Xu Lu

Quoted 13 lines

Thanks,

Clément

Quoted 7 lines

Thanks.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#8 lore raw
Quoted 23 lines · Xu Lu

Hi Clément,

Quoted 20 lines · Clément Léger
Quoted 15 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:

arch/riscv/include/asm/kvm_host.h line 48

Quoted 8 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                     BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Quoted 7 lines
Quoted 5 lines

Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default.

It doesn't matter whether these exceptions are delegated in medeleg.

Not sure to totally understand, but if the exceptions are not delegated in medeleg, they won't be delegated to VS-mode even though hedeleg bit is set right ? The spec says:

A synchronous trap that has been delegated to HS-mode (using medeleg) is further delegated to VS-mode if V=1 before the trap and the corresponding hedeleg bit is set.

Quoted 7 lines

KVM in HS-mode does not handle illegal instruction or misaligned access and only redirects them back to VS-mode. Delegating such exceptions in hedeleg helps save CPU usage even when they are not delegated in medeleg: opensbi will check whether these exceptions are delegated to VS-mode and redirect them to VS-mode if possible. There seems to be no conflicts with SSE implementation. Please correct me if I missed anything.

AFAIU, this means that since medeleg bit for misaligned accesses were not delegated up to the introduction of the FWFT extension, VS-mode generated misaligned accesses were handled by OpenSBI right ? Now that we are requesting openSBI to delegate misaligned accesses, HS-mode handles it's own misaligned accesses through the trap handler. With that configuration, if VS-mode generate a misaligned access, it will end up being redirected to VS-mode and won't be handle by HS-mode.

To summarize, prior to FWFT, medeleg wasn't delegating misaligned accesses to S-mode:

- VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> Back to VS-mode, misaligned access fixed up by OpenSBI

Now that Linux uses SBI FWFT to delegates misaligned accesses (without hedeleg being set for misaligned delegation, but that doesn't really matter, the outcome is the same):

- VS-mode misaligned access -> trap to HS-mode -> redirection to VS-mode, needs to handle the misaligned access by itself

This means that previously, misaligned access were silently fixed up by OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't true anymore. So, old kernel or sueprvisor software that included code to handle misaligned accesses will crash. Did I missed something ?

Note: this is not directly related to your series but my introduction of FWFT !

Thanks,

Clément

Quoted 17 lines

Best Regards,
Xu Lu

Quoted 13 lines

Thanks,

Clément

Quoted 7 lines

Thanks.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#9 lore raw

Hi Clément,

Quoted 35 lines · Clément Léger
Quoted 23 lines · Xu Lu

Hi Clément,

Quoted 20 lines · Clément Léger
Quoted 15 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:

arch/riscv/include/asm/kvm_host.h line 48

Quoted 8 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                     BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Quoted 7 lines
Quoted 5 lines

Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default.

It doesn't matter whether these exceptions are delegated in medeleg.

Quoted 7 lines

Not sure to totally understand, but if the exceptions are not delegated in medeleg, they won't be delegated to VS-mode even though hedeleg bit is set right ? The spec says:

A synchronous trap that has been delegated to HS-mode (using medeleg) is further delegated to VS-mode if V=1 before the trap and the corresponding hedeleg bit is set.

Yes, you are right. The illegal insn exception is still trapped in M mode if it is not delegated in medeleg. But delegating it in hedeleg is still useful. The opensbi will check CSR_HEDELEG in the function sbi_trap_redirect. If the exception has been delegated to VS-mode in CSR_HEDLEG, opensbi can directly return back to VS-mode, without the overhead of going back to HS-mode and then going back to VS-mode.

Quoted 19 lines
Quoted 10 lines

KVM in HS-mode does not handle illegal instruction or misaligned access and only redirects them back to VS-mode. Delegating such exceptions in hedeleg helps save CPU usage even when they are not delegated in medeleg: opensbi will check whether these exceptions are delegated to VS-mode and redirect them to VS-mode if possible. There seems to be no conflicts with SSE implementation. Please correct me if I missed anything.

AFAIU, this means that since medeleg bit for misaligned accesses were not delegated up to the introduction of the FWFT extension, VS-mode generated misaligned accesses were handled by OpenSBI right ? Now that we are requesting openSBI to delegate misaligned accesses, HS-mode handles it's own misaligned accesses through the trap handler. With that configuration, if VS-mode generate a misaligned access, it will end up being redirected to VS-mode and won't be handle by HS-mode.

Quoted 5 lines

To summarize, prior to FWFT, medeleg wasn't delegating misaligned accesses to S-mode:

- VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> Back to VS-mode, misaligned access fixed up by OpenSBI

Yes, this is what I want the procedure of handling illegal insn exceptions to be. Actually it now behaves as:

VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it -> Back to HS-mode, does nothing -> Back to VS-mode.

I want to avoid going through HS-mode.

Quoted 5 lines

Now that Linux uses SBI FWFT to delegates misaligned accesses (without hedeleg being set for misaligned delegation, but that doesn't really matter, the outcome is the same):

Quoted 8 lines

- VS-mode misaligned access -> trap to HS-mode -> redirection to VS-mode, needs to handle the misaligned access by itself

This means that previously, misaligned access were silently fixed up by OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't true anymore. So, old kernel or sueprvisor software that included code to handle misaligned accesses will crash. Did I missed something ?

Great! You make it very clear! Thanks for your explanation. But even when misalign exceptions are delegated to HS-mode, KVM seems to do nothing but redirect to VS-mode when VM get trapped due to misalign exceptions. So maybe we can directly delegate the misaligned exceptions in hedeleg too before running VCPU and retrieve them after VCPU exists. And then the handling procedure will be:

VS-mode misaligned exception -> trap to VS-mode -> VS handles it -> Back to VU-mode.

Please correct me if I missed anything.

Best Regards,

Xu Lu

Quoted 26 lines

Note: this is not directly related to your series but my introduction of FWFT !

Thanks,

Clément

Quoted 17 lines

Best Regards,
Xu Lu

Quoted 13 lines

Thanks,

Clément

Quoted 7 lines

Thanks.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#10 lore raw
Quoted 78 lines · Xu Lu

Hi Clément,

Quoted 35 lines · Clément Léger
Quoted 23 lines · Xu Lu

Hi Clément,

Quoted 20 lines · Clément Léger
Quoted 15 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:

arch/riscv/include/asm/kvm_host.h line 48

Quoted 8 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                     BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Quoted 7 lines
Quoted 5 lines

Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default.

It doesn't matter whether these exceptions are delegated in medeleg.

Quoted 7 lines

Not sure to totally understand, but if the exceptions are not delegated in medeleg, they won't be delegated to VS-mode even though hedeleg bit is set right ? The spec says:

A synchronous trap that has been delegated to HS-mode (using medeleg) is further delegated to VS-mode if V=1 before the trap and the corresponding hedeleg bit is set.

Yes, you are right. The illegal insn exception is still trapped in M mode if it is not delegated in medeleg. But delegating it in hedeleg is still useful. The opensbi will check CSR_HEDELEG in the function sbi_trap_redirect. If the exception has been delegated to VS-mode in CSR_HEDLEG, opensbi can directly return back to VS-mode, without the overhead of going back to HS-mode and then going back to VS-mode.

Quoted 19 lines
Quoted 10 lines

KVM in HS-mode does not handle illegal instruction or misaligned access and only redirects them back to VS-mode. Delegating such exceptions in hedeleg helps save CPU usage even when they are not delegated in medeleg: opensbi will check whether these exceptions are delegated to VS-mode and redirect them to VS-mode if possible. There seems to be no conflicts with SSE implementation. Please correct me if I missed anything.

AFAIU, this means that since medeleg bit for misaligned accesses were not delegated up to the introduction of the FWFT extension, VS-mode generated misaligned accesses were handled by OpenSBI right ? Now that we are requesting openSBI to delegate misaligned accesses, HS-mode handles it's own misaligned accesses through the trap handler. With that configuration, if VS-mode generate a misaligned access, it will end up being redirected to VS-mode and won't be handle by HS-mode.

Quoted 5 lines

To summarize, prior to FWFT, medeleg wasn't delegating misaligned accesses to S-mode:

- VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> Back to VS-mode, misaligned access fixed up by OpenSBI

Quoted 7 lines

Yes, this is what I want the procedure of handling illegal insn exceptions to be. Actually it now behaves as:

VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it -> Back to HS-mode, does nothing -> Back to VS-mode.

I want to avoid going through HS-mode.

Hi Xu,

Yeah, that make sense as well but that should only happen if the VS-mode requested misaligned access delegation via KVM SBI FWFT interface. I know that currently KVM does do anything useful from the misaligned exception except redirecting it to VS-mode but IMHO, that's a regression I introduced with FWFT misaligned requested delegation...

Quoted 15 lines
Quoted 6 lines

Now that Linux uses SBI FWFT to delegates misaligned accesses (without hedeleg being set for misaligned delegation, but that doesn't really matter, the outcome is the same):

Quoted 8 lines

- VS-mode misaligned access -> trap to HS-mode -> redirection to VS-mode, needs to handle the misaligned access by itself

This means that previously, misaligned access were silently fixed up by OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't true anymore. So, old kernel or sueprvisor software that included code to handle misaligned accesses will crash. Did I missed something ?

Quoted 4 lines

Great! You make it very clear! Thanks for your explanation. But even when misalign exceptions are delegated to HS-mode, KVM seems to do nothing but redirect to VS-mode when VM get trapped due to misalign exceptions.

Exactly, which is why I said that either setting hedeleg by default or not will lead to the same outcome, ie: VS-mode needs to handle access by itself (which is a regression introduced by FWFT usage).

Quoted 6 lines

So maybe we can directly delegate the misaligned exceptions in hedeleg too before running VCPU and retrieve them after VCPU exists. And then the handling procedure will be:

VS-mode misaligned exception -> trap to VS-mode -> VS handles it -> Back to VU-mode.

I'd better want to let the HS-mode handle the misaligned accesses if not requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode expected behavior. As you pointed out, this is not currently the case and the misaligned exceptions are directly redirected to VS-mode, this differs from what was actually done previously without FWFT (ie OpenSBI handles the misaligned access).

To summarize, I think HS-mode should fixup VS-mode misaligned accesses unless requested via KVM SBI FWFT interface, in which case it will delegates them (which is done by the FWFT series). This would match the HS-mode <-> OpenSBI behavior.

Thanks,

Clément

Quoted 33 lines

Please correct me if I missed anything.

Best Regards,

Xu Lu

Quoted 26 lines

Note: this is not directly related to your series but my introduction of FWFT !

Thanks,

Clément

Quoted 17 lines

Best Regards,
Xu Lu

Quoted 13 lines

Thanks,

Clément

Quoted 7 lines

Thanks.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#11 lore raw
Quoted 137 lines · Clément Léger
Quoted 78 lines · Xu Lu

Hi Clément,

Quoted 35 lines · Clément Léger
Quoted 23 lines · Xu Lu

Hi Clément,

Quoted 20 lines · Clément Léger
Quoted 15 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:

arch/riscv/include/asm/kvm_host.h line 48

Quoted 8 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                     BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Quoted 7 lines
Quoted 5 lines

Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default.

It doesn't matter whether these exceptions are delegated in medeleg.

Quoted 7 lines

Not sure to totally understand, but if the exceptions are not delegated in medeleg, they won't be delegated to VS-mode even though hedeleg bit is set right ? The spec says:

A synchronous trap that has been delegated to HS-mode (using medeleg) is further delegated to VS-mode if V=1 before the trap and the corresponding hedeleg bit is set.

Yes, you are right. The illegal insn exception is still trapped in M mode if it is not delegated in medeleg. But delegating it in hedeleg is still useful. The opensbi will check CSR_HEDELEG in the function sbi_trap_redirect. If the exception has been delegated to VS-mode in CSR_HEDLEG, opensbi can directly return back to VS-mode, without the overhead of going back to HS-mode and then going back to VS-mode.

Quoted 19 lines
Quoted 10 lines

KVM in HS-mode does not handle illegal instruction or misaligned access and only redirects them back to VS-mode. Delegating such exceptions in hedeleg helps save CPU usage even when they are not delegated in medeleg: opensbi will check whether these exceptions are delegated to VS-mode and redirect them to VS-mode if possible. There seems to be no conflicts with SSE implementation. Please correct me if I missed anything.

AFAIU, this means that since medeleg bit for misaligned accesses were not delegated up to the introduction of the FWFT extension, VS-mode generated misaligned accesses were handled by OpenSBI right ? Now that we are requesting openSBI to delegate misaligned accesses, HS-mode handles it's own misaligned accesses through the trap handler. With that configuration, if VS-mode generate a misaligned access, it will end up being redirected to VS-mode and won't be handle by HS-mode.

Quoted 5 lines

To summarize, prior to FWFT, medeleg wasn't delegating misaligned accesses to S-mode:

- VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> Back to VS-mode, misaligned access fixed up by OpenSBI

Quoted 7 lines

Yes, this is what I want the procedure of handling illegal insn exceptions to be. Actually it now behaves as:

VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it -> Back to HS-mode, does nothing -> Back to VS-mode.

I want to avoid going through HS-mode.

Hi Xu,

Yeah, that make sense as well but that should only happen if the VS-mode requested misaligned access delegation via KVM SBI FWFT interface. I know that currently KVM does do anything useful from the misaligned exception except redirecting it to VS-mode but IMHO, that's a regression I introduced with FWFT misaligned requested delegation...

Quoted 15 lines
Quoted 6 lines

Now that Linux uses SBI FWFT to delegates misaligned accesses (without hedeleg being set for misaligned delegation, but that doesn't really matter, the outcome is the same):

Quoted 8 lines

- VS-mode misaligned access -> trap to HS-mode -> redirection to VS-mode, needs to handle the misaligned access by itself

This means that previously, misaligned access were silently fixed up by OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't true anymore. So, old kernel or sueprvisor software that included code to handle misaligned accesses will crash. Did I missed something ?

Quoted 4 lines

Great! You make it very clear! Thanks for your explanation. But even when misalign exceptions are delegated to HS-mode, KVM seems to do nothing but redirect to VS-mode when VM get trapped due to misalign exceptions.

Exactly, which is why I said that either setting hedeleg by default or not will lead to the same outcome, ie: VS-mode needs to handle access by itself (which is a regression introduced by FWFT usage).

Quoted 6 lines

So maybe we can directly delegate the misaligned exceptions in hedeleg too before running VCPU and retrieve them after VCPU exists. And then the handling procedure will be:

VS-mode misaligned exception -> trap to VS-mode -> VS handles it -> Back to VU-mode.

I'd better want to let the HS-mode handle the misaligned accesses if not requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode expected behavior. As you pointed out, this is not currently the case and the misaligned exceptions are directly redirected to VS-mode, this differs from what was actually done previously without FWFT (ie OpenSBI handles the misaligned access).

Quoted 4 lines

To summarize, I think HS-mode should fixup VS-mode misaligned accesses unless requested via KVM SBI FWFT interface, in which case it will delegates them (which is done by the FWFT series). This would match the HS-mode <-> OpenSBI behavior.

Great! Roger that. Hope it can be fixed in the future.

Quoted 39 lines

Thanks,

Clément

Quoted 33 lines

Please correct me if I missed anything.

Best Regards,

Xu Lu

Quoted 26 lines

Note: this is not directly related to your series but my introduction of FWFT !

Thanks,

Clément

Quoted 17 lines

Best Regards,
Xu Lu

Quoted 13 lines

Thanks,

Clément

Quoted 7 lines

Thanks.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#12 lore raw
Quoted 138 lines · Xu Lu
Quoted 137 lines · Clément Léger
Quoted 78 lines · Xu Lu

Hi Clément,

Quoted 35 lines · Clément Léger
Quoted 23 lines · Xu Lu

Hi Clément,

Quoted 20 lines · Clément Léger
Quoted 15 lines · Radim Krčmář
2025-06-20T17:17:20+08:00, Xu Lu <[email protected]>:

arch/riscv/include/asm/kvm_host.h line 48

Quoted 8 lines

Delegate illegal instruction fault to VS mode in default to avoid such exceptions being trapped to HS and redirected back to VS.

---

arch/riscv/include/asm/kvm_host.h +1 -0
@@ -48,6 +48,7 @@
+                                     BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Quoted 7 lines
Quoted 5 lines

Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default.

It doesn't matter whether these exceptions are delegated in medeleg.

Quoted 7 lines

Not sure to totally understand, but if the exceptions are not delegated in medeleg, they won't be delegated to VS-mode even though hedeleg bit is set right ? The spec says:

A synchronous trap that has been delegated to HS-mode (using medeleg) is further delegated to VS-mode if V=1 before the trap and the corresponding hedeleg bit is set.

Yes, you are right. The illegal insn exception is still trapped in M mode if it is not delegated in medeleg. But delegating it in hedeleg is still useful. The opensbi will check CSR_HEDELEG in the function sbi_trap_redirect. If the exception has been delegated to VS-mode in CSR_HEDLEG, opensbi can directly return back to VS-mode, without the overhead of going back to HS-mode and then going back to VS-mode.

Quoted 19 lines
Quoted 10 lines

KVM in HS-mode does not handle illegal instruction or misaligned access and only redirects them back to VS-mode. Delegating such exceptions in hedeleg helps save CPU usage even when they are not delegated in medeleg: opensbi will check whether these exceptions are delegated to VS-mode and redirect them to VS-mode if possible. There seems to be no conflicts with SSE implementation. Please correct me if I missed anything.

AFAIU, this means that since medeleg bit for misaligned accesses were not delegated up to the introduction of the FWFT extension, VS-mode generated misaligned accesses were handled by OpenSBI right ? Now that we are requesting openSBI to delegate misaligned accesses, HS-mode handles it's own misaligned accesses through the trap handler. With that configuration, if VS-mode generate a misaligned access, it will end up being redirected to VS-mode and won't be handle by HS-mode.

Quoted 5 lines

To summarize, prior to FWFT, medeleg wasn't delegating misaligned accesses to S-mode:

- VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> Back to VS-mode, misaligned access fixed up by OpenSBI

Quoted 7 lines

Yes, this is what I want the procedure of handling illegal insn exceptions to be. Actually it now behaves as:

VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it -> Back to HS-mode, does nothing -> Back to VS-mode.

I want to avoid going through HS-mode.

Hi Xu,

Yeah, that make sense as well but that should only happen if the VS-mode requested misaligned access delegation via KVM SBI FWFT interface. I know that currently KVM does do anything useful from the misaligned exception except redirecting it to VS-mode but IMHO, that's a regression I introduced with FWFT misaligned requested delegation...

Quoted 15 lines
Quoted 6 lines

Now that Linux uses SBI FWFT to delegates misaligned accesses (without hedeleg being set for misaligned delegation, but that doesn't really matter, the outcome is the same):

Quoted 8 lines

- VS-mode misaligned access -> trap to HS-mode -> redirection to VS-mode, needs to handle the misaligned access by itself

This means that previously, misaligned access were silently fixed up by OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't true anymore. So, old kernel or sueprvisor software that included code to handle misaligned accesses will crash. Did I missed something ?

Quoted 4 lines

Great! You make it very clear! Thanks for your explanation. But even when misalign exceptions are delegated to HS-mode, KVM seems to do nothing but redirect to VS-mode when VM get trapped due to misalign exceptions.

Exactly, which is why I said that either setting hedeleg by default or not will lead to the same outcome, ie: VS-mode needs to handle access by itself (which is a regression introduced by FWFT usage).

Quoted 6 lines

So maybe we can directly delegate the misaligned exceptions in hedeleg too before running VCPU and retrieve them after VCPU exists. And then the handling procedure will be:

VS-mode misaligned exception -> trap to VS-mode -> VS handles it -> Back to VU-mode.

I'd better want to let the HS-mode handle the misaligned accesses if not requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode expected behavior. As you pointed out, this is not currently the case and the misaligned exceptions are directly redirected to VS-mode, this differs from what was actually done previously without FWFT (ie OpenSBI handles the misaligned access).

Quoted 6 lines
Quoted 4 lines

To summarize, I think HS-mode should fixup VS-mode misaligned accesses unless requested via KVM SBI FWFT interface, in which case it will delegates them (which is done by the FWFT series). This would match the HS-mode <-> OpenSBI behavior.

Great! Roger that. Hope it can be fixed in the future.

Ahah ! I'll take a look at it if everyone agrees that it's a regression.

Thanks,

Clément

Quoted 40 lines
Quoted 40 lines

Thanks,

Clément

Quoted 33 lines

Please correct me if I missed anything.

Best Regards,

Xu Lu

Quoted 26 lines

Note: this is not directly related to your series but my introduction of FWFT !

Thanks,

Clément

Quoted 17 lines

Best Regards,
Xu Lu

Quoted 13 lines

Thanks,

Clément

Quoted 7 lines

Thanks.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv