Skip to contentSkip to conversation

riscv: add support for SBI Supervisor Software Events

PATCH Superseded Clément Léger 3 participants 17 messages Aug 8, 2025 – Aug 28, 2025

Subsystems (4)

v6 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v6 cover riscv: add support for SBI Supervisor Software Events

The SBI Supervisor Software Events (SSE) extensions provides a mechanism to inject software events from an SBI implementation to supervisor software such that it preempts all other supervisor level traps and interrupts. This extension is introduced by the SBI v3.0 specification[1].

Various events are defined and can be send asynchronously to supervisor software (RAS, PMU, DEBUG, Asynchronous page fault) from SBI as well as platform specific events. Events can be either local (per-hart) or global. Events can be nested on top of each other based on priority and can interrupt the kernel at any time.

First patch adds the SSE definitions. Second one adds support for SSE at arch level (entry code and stack allocations) and third one at driver level. Finally, the last patch add support for SSE events in the SBI PMU driver. Additional testing for that part is highly welcomed since there are a lot of possible path that needs to be exercised.

Amongst the specific points that needs to be handle is the interruption at any point of the kernel execution and more specifically at the beginning of exception handling. Due to the fact that the exception entry implementation uses the SCRATCH CSR as both the current task struct and as the temporary register to switch the stack and save register, it is difficult to reliably get the current task struct if we get interrupted at this specific moment (ie, it might contain 0, the task pointer or tp). A fixup-like mechanism is not possible due to the nested nature of SSE which makes it really hard to obtain the original interruption site. In order to retrieve the task in a reliable manner, add an additional __sse_entry_task per_cpu array which stores the current task. Ideally, we would need to modify the way we retrieve/store the current task in exception handling so that it does not depend on the place where it's interrupted.

Contrary to pseudo NMI [2], SSE does not modifies the way interrupts are handled and does not adds any overhead to existing code. Moreover, it provides "true" NMI-like interrupts which can interrupt the kernel at any time (even in exception handling). This is particularly crucial for RAS errors which needs to be handled as fast as possible to avoid any fault propagation.

A KVM SBI SSE extension implementation is available at [2].

---

Changes in v6:
- Fix comment in assembly argument
- Check hart id to be the expected one in order to skip CPU id matching in sse assembly.

Changes in v5:
- Added a SSE test module in kselftests
- Removed an unused variable
- Applied checkpatch.pl --strict and fix all errors
- Use scope_guard(cpus_read_lock) instead of manual cpus_read_lock()
- Fix wrong variable returned in sse_get_event
- Remove useless init of events list
- Remove useless empty for loop on cpus
- Set sse_available as __ro_after_init
- Changed a few pr_debug to pr_warn
- Fix event enabled stated updated in case of failure
- Change no_lock to nolock
- Rename attr_buf to attr
- renamed sse_get_event_phys() to sse_event_get_attr_phys() and removed the second argument
- Simplify return value in sse_event_attr_set_nolock()
- Remove while loop(-EINVAL) for event cpu set call
- Renamed interrupted_state_phys to interrupted_phys
- Use scoped_guards/guard for sse_mutex
- Remove useless struct forward declaration in sse.h
- Add more explanations as to why we set SIE bit in IP
- Unconditionnally set SIE in SIP
- Move SSE_STACK_SIZE adjustement in sse_stack_alloc/free()
- Replace move instructions with mv
- Rename NR_CPUS asm symbol to ASM_NR_CPUS
- Restore SSTATUS first in sse_entry return path so that it works for double trap without any modification later.
- Implement proper per cpu revert if enable/register fails

Changes in v4:
- Fix a bug when using per_cpu ptr for local event (Andrew)
- Add sse_event_disable/enable_local()
- Add pmu_disable/pmu_enable() to disable/enable SSE event
- Update event ID description according to the latest spec
- Fix comment about arguments in handle_sse()
- Added Himanchu as a SSE reviewer
- Used SYM_DATA_*() macros instead of hardcoded labels
- Invoke softirqs only if not returning to kernel with irqs disabled
- Remove invalid state check for write attribute function.
- Remove useless bneq statement in sse_entry.S

Changes in v3:
- Split arch/driver support
- Fix potential register failure reporting
- Set a few pr_err as pr_debug
- Allow CONFIG_RISCV_SSE to be disabled
- Fix build without CONFIG_RISCV_SSE
- Remove fixup-like mechanism and use a per-cpu array
- Fixed SSCRATCH being corrupted when interrupting the kernel in early exception path.
- Split SSE assembly from entry.S
- Add Himanchu SSE mask/unmask and runtime PM support.
- Disable user memory access/floating point/vector in SSE handler
- Rebased on master

v2: https://lore.kernel.org/linux-riscv/[email protected]/

Changes in v2:
- Implemented specification v2
- Fix various error handling cases
- Added shadow stack support

v1: https://lore.kernel.org/linux-riscv/[email protected]/

Clément Léger (5):
riscv: add SBI SSE extension definitions
riscv: add support for SBI Supervisor Software Events extension
drivers: firmware: add riscv SSE support
perf: RISC-V: add support for SSE event selftests/riscv: add SSE test module

 MAINTAINERS                                   |  15 +
 arch/riscv/include/asm/asm.h                  |  14 +-
 arch/riscv/include/asm/sbi.h                  |  61 ++
 arch/riscv/include/asm/scs.h                  |   7 +
 arch/riscv/include/asm/sse.h                  |  47 ++
 arch/riscv/include/asm/switch_to.h            |  14 +
 arch/riscv/include/asm/thread_info.h          |   1 +
 arch/riscv/kernel/Makefile                    |   1 +
 arch/riscv/kernel/asm-offsets.c               |  14 +
 arch/riscv/kernel/sse.c                       | 154 ++++
 arch/riscv/kernel/sse_entry.S                 | 180 +++++
 drivers/firmware/Kconfig                      |   1 +
 drivers/firmware/Makefile                     |   1 +
 drivers/firmware/riscv/Kconfig                |  15 +
 drivers/firmware/riscv/Makefile               |   3 +
 drivers/firmware/riscv/riscv_sse.c            | 694 ++++++++++++++++++
 drivers/perf/Kconfig                          |  10 +
 drivers/perf/riscv_pmu.c                      |  19 +
 drivers/perf/riscv_pmu_sbi.c                  |  71 +-
 include/linux/perf/riscv_pmu.h                |   3 +
 include/linux/riscv_sse.h                     |  56 ++
 tools/testing/selftests/riscv/Makefile        |   2 +-
 tools/testing/selftests/riscv/sse/Makefile    |   5 +
 .../selftests/riscv/sse/module/Makefile       |  16 +
 .../riscv/sse/module/riscv_sse_test.c         | 513 +++++++++++++
 .../selftests/riscv/sse/run_sse_test.sh       |  44 ++
 26 files changed, 1947 insertions(+), 14 deletions(-)
 create mode 100644 arch/riscv/include/asm/sse.h
 create mode 100644 arch/riscv/kernel/sse.c
 create mode 100644 arch/riscv/kernel/sse_entry.S
 create mode 100644 drivers/firmware/riscv/Kconfig
 create mode 100644 drivers/firmware/riscv/Makefile
 create mode 100644 drivers/firmware/riscv/riscv_sse.c
 create mode 100644 include/linux/riscv_sse.h
 create mode 100644 tools/testing/selftests/riscv/sse/Makefile
 create mode 100644 tools/testing/selftests/riscv/sse/module/Makefile
 create mode 100644 tools/testing/selftests/riscv/sse/module/riscv_sse_test.c
 create mode 100644 tools/testing/selftests/riscv/sse/run_sse_test.sh
Signature · 1 line
2.43.0
#1 lore raw

PATCH v6 1/5 riscv: add SBI SSE extension definitions

Add needed definitions for SBI Supervisor Software Events extension [1]. This extension enables the SBI to inject events into supervisor software much like ARM SDEI.

[1] https://lists.riscv.org/g/tech-prs/message/515

---
 arch/riscv/include/asm/sbi.h | 61 ++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
arch/riscv/include/asm/sbi.h +61 -0
index 341e74238aa0..e4993fb664d2 100644
@@ -36,6 +36,7 @@ enum sbi_ext_id {
 	SBI_EXT_STA = 0x535441,
 	SBI_EXT_NACL = 0x4E41434C,
 	SBI_EXT_FWFT = 0x46574654,
+	SBI_EXT_SSE = 0x535345,
 
 	/* Experimentals extensions must lie within this range */
 	SBI_EXT_EXPERIMENTAL_START = 0x08000000,
@@ -430,6 +431,66 @@ enum sbi_fwft_feature_t {
 
 #define SBI_FWFT_SET_FLAG_LOCK			BIT(0)
 
+enum sbi_ext_sse_fid {
+	SBI_SSE_EVENT_ATTR_READ = 0,
+	SBI_SSE_EVENT_ATTR_WRITE,
+	SBI_SSE_EVENT_REGISTER,
+	SBI_SSE_EVENT_UNREGISTER,
+	SBI_SSE_EVENT_ENABLE,
+	SBI_SSE_EVENT_DISABLE,
+	SBI_SSE_EVENT_COMPLETE,
+	SBI_SSE_EVENT_SIGNAL,
+	SBI_SSE_EVENT_HART_UNMASK,
+	SBI_SSE_EVENT_HART_MASK,
+};
+
+enum sbi_sse_state {
+	SBI_SSE_STATE_UNUSED     = 0,
+	SBI_SSE_STATE_REGISTERED = 1,
+	SBI_SSE_STATE_ENABLED    = 2,
+	SBI_SSE_STATE_RUNNING    = 3,
+};
+
+/* SBI SSE Event Attributes. */
+enum sbi_sse_attr_id {
+	SBI_SSE_ATTR_STATUS		= 0x00000000,
+	SBI_SSE_ATTR_PRIO		= 0x00000001,
+	SBI_SSE_ATTR_CONFIG		= 0x00000002,
+	SBI_SSE_ATTR_PREFERRED_HART	= 0x00000003,
+	SBI_SSE_ATTR_ENTRY_PC		= 0x00000004,
+	SBI_SSE_ATTR_ENTRY_ARG		= 0x00000005,
+	SBI_SSE_ATTR_INTERRUPTED_SEPC	= 0x00000006,
+	SBI_SSE_ATTR_INTERRUPTED_FLAGS	= 0x00000007,
+	SBI_SSE_ATTR_INTERRUPTED_A6	= 0x00000008,
+	SBI_SSE_ATTR_INTERRUPTED_A7	= 0x00000009,
+
+	SBI_SSE_ATTR_MAX		= 0x0000000A
+};
+
+#define SBI_SSE_ATTR_STATUS_STATE_OFFSET	0
+#define SBI_SSE_ATTR_STATUS_STATE_MASK		0x3
+#define SBI_SSE_ATTR_STATUS_PENDING_OFFSET	2
+#define SBI_SSE_ATTR_STATUS_INJECT_OFFSET	3
+
+#define SBI_SSE_ATTR_CONFIG_ONESHOT	BIT(0)
+
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_SSTATUS_SPP	BIT(0)
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_SSTATUS_SPIE	BIT(1)
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_HSTATUS_SPV	BIT(2)
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_HSTATUS_SPVP	BIT(3)
+
+#define SBI_SSE_EVENT_LOCAL_HIGH_PRIO_RAS	0x00000000
+#define SBI_SSE_EVENT_LOCAL_DOUBLE_TRAP		0x00000001
+#define SBI_SSE_EVENT_GLOBAL_HIGH_PRIO_RAS	0x00008000
+#define SBI_SSE_EVENT_LOCAL_PMU_OVERFLOW	0x00010000
+#define SBI_SSE_EVENT_LOCAL_LOW_PRIO_RAS	0x00100000
+#define SBI_SSE_EVENT_GLOBAL_LOW_PRIO_RAS	0x00108000
+#define SBI_SSE_EVENT_LOCAL_SOFTWARE_INJECTED	0xffff0000
+#define SBI_SSE_EVENT_GLOBAL_SOFTWARE_INJECTED	0xffff8000
+
+#define SBI_SSE_EVENT_PLATFORM		BIT(14)
+#define SBI_SSE_EVENT_GLOBAL		BIT(15)
+
 /* SBI spec version fields */
 #define SBI_SPEC_VERSION_DEFAULT	0x1
 #define SBI_SPEC_VERSION_MAJOR_SHIFT	24
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v6 1/5] riscv: add SBI SSE extension definitions

Hi Clément,

Quoted 26 lines · Clément Léger

Add needed definitions for SBI Supervisor Software Events extension [1]. This extension enables the SBI to inject events into supervisor software much like ARM SDEI.

[1] https://lists.riscv.org/g/tech-prs/message/515

---
  arch/riscv/include/asm/sbi.h | 61 ++++++++++++++++++++++++++++++++++++
  1 file changed, 61 insertions(+)
arch/riscv/include/asm/sbi.h +1 -0
index 341e74238aa0..e4993fb664d2 100644
@@ -36,6 +36,7 @@ enum sbi_ext_id {
  	SBI_EXT_STA = 0x535441,
  	SBI_EXT_NACL = 0x4E41434C,
  	SBI_EXT_FWFT = 0x46574654,
+	SBI_EXT_SSE = 0x535345,
 
  	/* Experimentals extensions must lie within this range */
  	SBI_EXT_EXPERIMENTAL_START = 0x08000000,
@@ -430,6 +431,66 @@ enum sbi_fwft_feature_t {
 
  #define SBI_FWFT_SET_FLAG_LOCK			BIT(0)

arch/riscv/include/asm/sbi.h enum sbi_fwft_feature_t { · line 442

Quoted 10 lines
arch/riscv/include/asm/sbi.h +9 -0
 
+enum sbi_ext_sse_fid {
+	SBI_SSE_EVENT_ATTR_READ = 0,
+	SBI_SSE_EVENT_ATTR_WRITE,
+	SBI_SSE_EVENT_REGISTER,
+	SBI_SSE_EVENT_UNREGISTER,
+	SBI_SSE_EVENT_ENABLE,
+	SBI_SSE_EVENT_DISABLE,
+	SBI_SSE_EVENT_COMPLETE,
+	SBI_SSE_EVENT_SIGNAL,

Nit but in the specification this event is referred as "inject" instead of "signal", any reason to change that naming?

Quoted 37 lines
arch/riscv/include/asm/sbi.h +37 -0
+	SBI_SSE_EVENT_HART_UNMASK,
+	SBI_SSE_EVENT_HART_MASK,
+};
+
+enum sbi_sse_state {
+	SBI_SSE_STATE_UNUSED     = 0,
+	SBI_SSE_STATE_REGISTERED = 1,
+	SBI_SSE_STATE_ENABLED    = 2,
+	SBI_SSE_STATE_RUNNING    = 3,
+};
+
+/* SBI SSE Event Attributes. */
+enum sbi_sse_attr_id {
+	SBI_SSE_ATTR_STATUS		= 0x00000000,
+	SBI_SSE_ATTR_PRIO		= 0x00000001,
+	SBI_SSE_ATTR_CONFIG		= 0x00000002,
+	SBI_SSE_ATTR_PREFERRED_HART	= 0x00000003,
+	SBI_SSE_ATTR_ENTRY_PC		= 0x00000004,
+	SBI_SSE_ATTR_ENTRY_ARG		= 0x00000005,
+	SBI_SSE_ATTR_INTERRUPTED_SEPC	= 0x00000006,
+	SBI_SSE_ATTR_INTERRUPTED_FLAGS	= 0x00000007,
+	SBI_SSE_ATTR_INTERRUPTED_A6	= 0x00000008,
+	SBI_SSE_ATTR_INTERRUPTED_A7	= 0x00000009,
+
+	SBI_SSE_ATTR_MAX		= 0x0000000A
+};
+
+#define SBI_SSE_ATTR_STATUS_STATE_OFFSET	0
+#define SBI_SSE_ATTR_STATUS_STATE_MASK		0x3
+#define SBI_SSE_ATTR_STATUS_PENDING_OFFSET	2
+#define SBI_SSE_ATTR_STATUS_INJECT_OFFSET	3
+
+#define SBI_SSE_ATTR_CONFIG_ONESHOT	BIT(0)
+
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_SSTATUS_SPP	BIT(0)
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_SSTATUS_SPIE	BIT(1)
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_HSTATUS_SPV	BIT(2)

arch/riscv/include/asm/sbi.h enum sbi_fwft_feature_t { · line 489

Quoted 10 lines
arch/riscv/include/asm/sbi.h +10 -0
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_HSTATUS_SPVP	BIT(3)
+
+#define SBI_SSE_EVENT_LOCAL_HIGH_PRIO_RAS	0x00000000
+#define SBI_SSE_EVENT_LOCAL_DOUBLE_TRAP		0x00000001
+#define SBI_SSE_EVENT_GLOBAL_HIGH_PRIO_RAS	0x00008000
+#define SBI_SSE_EVENT_LOCAL_PMU_OVERFLOW	0x00010000
+#define SBI_SSE_EVENT_LOCAL_LOW_PRIO_RAS	0x00100000
+#define SBI_SSE_EVENT_GLOBAL_LOW_PRIO_RAS	0x00108000
+#define SBI_SSE_EVENT_LOCAL_SOFTWARE_INJECTED	0xffff0000
+#define SBI_SSE_EVENT_GLOBAL_SOFTWARE_INJECTED	0xffff8000

And here you use "injected".

Quoted 7 lines
arch/riscv/include/asm/sbi.h +4 -0
+
+#define SBI_SSE_EVENT_PLATFORM		BIT(14)
+#define SBI_SSE_EVENT_GLOBAL		BIT(15)
+
  /* SBI spec version fields */
  #define SBI_SPEC_VERSION_DEFAULT	0x1
  #define SBI_SPEC_VERSION_MAJOR_SHIFT	24
#3 lore raw

Re: [PATCH v6 1/5] riscv: add SBI SSE extension definitions

Quoted 32 lines · Alexandre Ghiti

Hi Clément,

Quoted 29 lines · Clément Léger

Add needed definitions for SBI Supervisor Software Events extension [1]. This extension enables the SBI to inject events into supervisor software much like ARM SDEI.

[1] https://lists.riscv.org/g/tech-prs/message/515

---
  arch/riscv/include/asm/sbi.h | 61 ++++++++++++++++++++++++++++++++++++

  1 file changed, 61 insertions(+)
arch/riscv/include/asm/sbi.h +5 -0
index 341e74238aa0..e4993fb664d2 100644
@@ -36,6 +36,7 @@ enum sbi_ext_id {
       SBI_EXT_STA = 0x535441,
       SBI_EXT_NACL = 0x4E41434C,
       SBI_EXT_FWFT = 0x46574654,
+    SBI_EXT_SSE = 0x535345,
         /* Experimentals extensions must lie within this range */
       SBI_EXT_EXPERIMENTAL_START = 0x08000000,
@@ -430,6 +431,66 @@ enum sbi_fwft_feature_t {
     #define SBI_FWFT_SET_FLAG_LOCK            BIT(0)
   +enum sbi_ext_sse_fid {
+    SBI_SSE_EVENT_ATTR_READ = 0,
+    SBI_SSE_EVENT_ATTR_WRITE,
+    SBI_SSE_EVENT_REGISTER,
+    SBI_SSE_EVENT_UNREGISTER,
Quoted 8 lines

arch/riscv/include/asm/sbi.h enum sbi_fwft_feature_t { · line 442

Quoted 4 lines
arch/riscv/include/asm/sbi.h +4 -0
+    SBI_SSE_EVENT_ENABLE,
+    SBI_SSE_EVENT_DISABLE,
+    SBI_SSE_EVENT_COMPLETE,
+    SBI_SSE_EVENT_SIGNAL,

Nit but in the specification this event is referred as "inject" instead of "signal", any reason to change that naming?

Hi Alex,

That's mostly for historic reason, spec changed that after a few revision. I'll update that.

Thanks,

Clément

Quoted 61 lines
Quoted 39 lines
arch/riscv/include/asm/sbi.h +37 -0
+    SBI_SSE_EVENT_HART_UNMASK,
+    SBI_SSE_EVENT_HART_MASK,
+};
+
+enum sbi_sse_state {
+    SBI_SSE_STATE_UNUSED     = 0,
+    SBI_SSE_STATE_REGISTERED = 1,
+    SBI_SSE_STATE_ENABLED    = 2,
+    SBI_SSE_STATE_RUNNING    = 3,
+};
+
+/* SBI SSE Event Attributes. */
+enum sbi_sse_attr_id {
+    SBI_SSE_ATTR_STATUS        = 0x00000000,
+    SBI_SSE_ATTR_PRIO        = 0x00000001,
+    SBI_SSE_ATTR_CONFIG        = 0x00000002,
+    SBI_SSE_ATTR_PREFERRED_HART    = 0x00000003,
+    SBI_SSE_ATTR_ENTRY_PC        = 0x00000004,
+    SBI_SSE_ATTR_ENTRY_ARG        = 0x00000005,
+    SBI_SSE_ATTR_INTERRUPTED_SEPC    = 0x00000006,
+    SBI_SSE_ATTR_INTERRUPTED_FLAGS    = 0x00000007,
+    SBI_SSE_ATTR_INTERRUPTED_A6    = 0x00000008,
+    SBI_SSE_ATTR_INTERRUPTED_A7    = 0x00000009,
+
+    SBI_SSE_ATTR_MAX        = 0x0000000A
+};
+
+#define SBI_SSE_ATTR_STATUS_STATE_OFFSET    0
+#define SBI_SSE_ATTR_STATUS_STATE_MASK        0x3
+#define SBI_SSE_ATTR_STATUS_PENDING_OFFSET    2
+#define SBI_SSE_ATTR_STATUS_INJECT_OFFSET    3
+
+#define SBI_SSE_ATTR_CONFIG_ONESHOT    BIT(0)
+
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_SSTATUS_SPP    BIT(0)
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_SSTATUS_SPIE    BIT(1)
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_HSTATUS_SPV    BIT(2)

arch/riscv/include/asm/sbi.h enum sbi_fwft_feature_t { · line 489

Quoted 10 lines
arch/riscv/include/asm/sbi.h +10 -0
+#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_HSTATUS_SPVP    BIT(3)
+
+#define SBI_SSE_EVENT_LOCAL_HIGH_PRIO_RAS    0x00000000
+#define SBI_SSE_EVENT_LOCAL_DOUBLE_TRAP        0x00000001
+#define SBI_SSE_EVENT_GLOBAL_HIGH_PRIO_RAS    0x00008000
+#define SBI_SSE_EVENT_LOCAL_PMU_OVERFLOW    0x00010000
+#define SBI_SSE_EVENT_LOCAL_LOW_PRIO_RAS    0x00100000
+#define SBI_SSE_EVENT_GLOBAL_LOW_PRIO_RAS    0x00108000
+#define SBI_SSE_EVENT_LOCAL_SOFTWARE_INJECTED    0xffff0000
+#define SBI_SSE_EVENT_GLOBAL_SOFTWARE_INJECTED    0xffff8000

And here you use "injected".

Quoted 7 lines
arch/riscv/include/asm/sbi.h +4 -0
+
+#define SBI_SSE_EVENT_PLATFORM        BIT(14)
+#define SBI_SSE_EVENT_GLOBAL        BIT(15)
+
   /* SBI spec version fields */
   #define SBI_SPEC_VERSION_DEFAULT    0x1
   #define SBI_SPEC_VERSION_MAJOR_SHIFT    24
#4 lore raw

PATCH v6 2/5 riscv: add support for SBI Supervisor Software Events extension

In reply to Clément Léger

The SBI SSE extension allows the supervisor software to be notified by the SBI of specific events that are not maskable. The context switch is handled partially by the firmware which will save registers a6 and a7. When entering kernel we can rely on these 2 registers to setup the stack and save all the registers.

Since SSE events can be delivered at any time to the kernel (including during exception handling, we need a way to locate the current_task for context tracking. On RISC-V, it is sotred in scratch when in user space or tp when in kernel space (in which case SSCRATCH is zero). But at a at the beginning of exception handling, SSCRATCH is used to swap tp and check the origin of the exception. If interrupted at that point, then, there is no way to reliably know were is located the current task_struct. Even checking the interruption location won't work as SSE event can be nested on top of each other so the original interruption site might be lost at some point. In order to retrieve it reliably, store the current task in an additional __sse_entry_task per_cpu array. This array is then used to retrieve the current task based on the hart ID that is passed to the SSE event handler in a6.

That being said, the way the current task struct is stored should probably be reworked to find a better reliable alternative.

Since each events (and each CPU for local events) have their own context and can preempt each other, allocate a stack (and a shadow stack if needed for each of them (and for each cpu for local events).

When completing the event, if we were coming from kernel with interrupts disabled, simply return there. If coming from userspace or kernel with interrupts enabled, simulate an interrupt exception by setting IE_SIE in CSR_IP to allow delivery of signals to user task. For instance this can happen, when a RAS event has been generated by a user application and a SIGBUS has been sent to a task.

---
 arch/riscv/include/asm/asm.h         |  14 ++-
 arch/riscv/include/asm/scs.h         |   7 ++
 arch/riscv/include/asm/sse.h         |  47 +++++++
 arch/riscv/include/asm/switch_to.h   |  14 +++
 arch/riscv/include/asm/thread_info.h |   1 +
 arch/riscv/kernel/Makefile           |   1 +
 arch/riscv/kernel/asm-offsets.c      |  14 +++
 arch/riscv/kernel/sse.c              | 154 +++++++++++++++++++++++
 arch/riscv/kernel/sse_entry.S        | 180 +++++++++++++++++++++++++++
 9 files changed, 429 insertions(+), 3 deletions(-)
 create mode 100644 arch/riscv/include/asm/sse.h
 create mode 100644 arch/riscv/kernel/sse.c
 create mode 100644 arch/riscv/kernel/sse_entry.S
arch/riscv/include/asm/asm.h +11 -3
index a8a2af6dfe9d..982c4be9a9c3 100644
@@ -90,16 +90,24 @@
 #define PER_CPU_OFFSET_SHIFT 3
 #endif
 
-.macro asm_per_cpu dst sym tmp
-	REG_L \tmp, TASK_TI_CPU_NUM(tp)
-	slli  \tmp, \tmp, PER_CPU_OFFSET_SHIFT
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+	slli  \tmp, \cpu, PER_CPU_OFFSET_SHIFT
 	la    \dst, __per_cpu_offset
 	add   \dst, \dst, \tmp
 	REG_L \tmp, 0(\dst)
 	la    \dst, \sym
 	add   \dst, \dst, \tmp
 .endm
+
+.macro asm_per_cpu dst sym tmp
+	REG_L \tmp, TASK_TI_CPU_NUM(tp)
+	asm_per_cpu_with_cpu \dst \sym \tmp \tmp
+.endm
 #else /* CONFIG_SMP */
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+	la    \dst, \sym
+.endm
+
 .macro asm_per_cpu dst sym tmp
 	la    \dst, \sym
 .endm
arch/riscv/include/asm/scs.h +7 -0
index 0e45db78b24b..62344daad73d 100644
@@ -18,6 +18,11 @@
 	load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp
 .endm
 
+/* Load the per-CPU IRQ shadow call stack to gp. */
+.macro scs_load_sse_stack reg_evt
+	REG_L gp, SSE_REG_EVT_SHADOW_STACK(\reg_evt)
+.endm
+
 /* Load task_scs_sp(current) to gp. */
 .macro scs_load_current
 	REG_L	gp, TASK_TI_SCS_SP(tp)
@@ -41,6 +46,8 @@
 .endm
 .macro scs_load_irq_stack tmp
 .endm
+.macro scs_load_sse_stack reg_evt
+.endm
 .macro scs_load_current
 .endm
 .macro scs_load_current_if_task_changed prev
arch/riscv/include/asm/sse.h new file +47 -0

47 lines not on this page · raw message

arch/riscv/include/asm/switch_to.h +14 -0
index 0e71eb82f920..cd1cead0c682 100644
@@ -88,6 +88,19 @@ static inline void __switch_to_envcfg(struct task_struct *next)
 			:: "r" (next->thread.envcfg) : "memory");
 }
 
+#ifdef CONFIG_RISCV_SSE
+DECLARE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+	__this_cpu_write(__sse_entry_task, next);
+}
+#else
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+}
+#endif
+
 extern struct task_struct *__switch_to(struct task_struct *,
 				       struct task_struct *);
 
@@ -122,6 +135,7 @@ do {							\
 	if (switch_to_should_flush_icache(__next))	\
 		local_flush_icache_all();		\
 	__switch_to_envcfg(__next);			\
+	__switch_sse_entry_task(__next);			\
 	((last) = __switch_to(__prev, __next));		\
 } while (0)
 
arch/riscv/include/asm/thread_info.h +1 -0
index f5916a70879a..28e9805e61fc 100644
@@ -36,6 +36,7 @@
 #define OVERFLOW_STACK_SIZE     SZ_4K
 
 #define IRQ_STACK_SIZE		THREAD_SIZE
+#define SSE_STACK_SIZE		THREAD_SIZE
 
 #ifndef __ASSEMBLY__
 
arch/riscv/kernel/Makefile +1 -0
index c7b542573407..62e4490b34ee 100644
@@ -99,6 +99,7 @@ obj-$(CONFIG_DYNAMIC_FTRACE)	+= mcount-dyn.o
 obj-$(CONFIG_PERF_EVENTS)	+= perf_callchain.o
 obj-$(CONFIG_HAVE_PERF_REGS)	+= perf_regs.o
 obj-$(CONFIG_RISCV_SBI)		+= sbi.o sbi_ecall.o
+obj-$(CONFIG_RISCV_SSE)		+= sse.o sse_entry.o
 ifeq ($(CONFIG_RISCV_SBI), y)
 obj-$(CONFIG_SMP)		+= sbi-ipi.o
 obj-$(CONFIG_SMP) += cpu_ops_sbi.o
arch/riscv/kernel/asm-offsets.c +14 -0
index 6e8c0d6feae9..315547c3a2ef 100644
@@ -14,6 +14,8 @@
 #include <asm/ptrace.h>
 #include <asm/cpu_ops_sbi.h>
 #include <asm/stacktrace.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
 #include <asm/suspend.h>
 
 void asm_offsets(void);
@@ -528,4 +530,16 @@ void asm_offsets(void)
 	DEFINE(FREGS_A6,	    offsetof(struct __arch_ftrace_regs, a6));
 	DEFINE(FREGS_A7,	    offsetof(struct __arch_ftrace_regs, a7));
 #endif
+
+#ifdef CONFIG_RISCV_SSE
+	OFFSET(SSE_REG_EVT_STACK, sse_event_arch_data, stack);
+	OFFSET(SSE_REG_EVT_SHADOW_STACK, sse_event_arch_data, shadow_stack);
+	OFFSET(SSE_REG_EVT_TMP, sse_event_arch_data, tmp);
+	OFFSET(SSE_REG_HART_ID, sse_event_arch_data, hart_id);
+	OFFSET(SSE_REG_CPU_ID, sse_event_arch_data, cpu_id);
+
+	DEFINE(SBI_EXT_SSE, SBI_EXT_SSE);
+	DEFINE(SBI_SSE_EVENT_COMPLETE, SBI_SSE_EVENT_COMPLETE);
+	DEFINE(ASM_NR_CPUS, NR_CPUS);
+#endif
 }
arch/riscv/kernel/sse.c new file +154 -0

154 lines not on this page · raw message

arch/riscv/kernel/sse_entry.S new file +180 -0

180 lines not on this page · raw message

Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH v6 2/5] riscv: add support for SBI Supervisor Software Events extension

Quoted 1 line · Clément Léger

The SBI SSE extension allows the supervisor software to be notified by

I would change "notified" with "interrupted" no?

Quoted 1 line

the SBI of specific events that are not maskable. The context switch is

Nit: s/the SBI/the SBI implementation

Quoted 6 lines

handled partially by the firmware which will save registers a6 and a7. When entering kernel we can rely on these 2 registers to setup the stack and save all the registers.

Since SSE events can be delivered at any time to the kernel (including during exception handling, we need a way to locate the current_task for

s/handling/handling)

Quoted 1 line

context tracking. On RISC-V, it is sotred in scratch when in user space

s/sotred/stored

Quoted 1 line

or tp when in kernel space (in which case SSCRATCH is zero). But at a

Remove "at a"

Quoted 3 lines

at the beginning of exception handling, SSCRATCH is used to swap tp and check the origin of the exception. If interrupted at that point, then, there is no way to reliably know were is located the current

s/were/where

Quoted 11 lines

task_struct. Even checking the interruption location won't work as SSE event can be nested on top of each other so the original interruption site might be lost at some point. In order to retrieve it reliably, store the current task in an additional __sse_entry_task per_cpu array. This array is then used to retrieve the current task based on the hart ID that is passed to the SSE event handler in a6.

That being said, the way the current task struct is stored should probably be reworked to find a better reliable alternative.

Since each events (and each CPU for local events) have their own

"Since each event has its own"

Quoted 2 lines

context and can preempt each other, allocate a stack (and a shadow stack if needed for each of them (and for each cpu for local events).

s/needed/needed)

Quoted 36 lines

When completing the event, if we were coming from kernel with interrupts disabled, simply return there. If coming from userspace or kernel with interrupts enabled, simulate an interrupt exception by setting IE_SIE in CSR_IP to allow delivery of signals to user task. For instance this can happen, when a RAS event has been generated by a user application and a SIGBUS has been sent to a task.

---
  arch/riscv/include/asm/asm.h         |  14 ++-
  arch/riscv/include/asm/scs.h         |   7 ++
  arch/riscv/include/asm/sse.h         |  47 +++++++
  arch/riscv/include/asm/switch_to.h   |  14 +++
  arch/riscv/include/asm/thread_info.h |   1 +
  arch/riscv/kernel/Makefile           |   1 +
  arch/riscv/kernel/asm-offsets.c      |  14 +++
  arch/riscv/kernel/sse.c              | 154 +++++++++++++++++++++++
  arch/riscv/kernel/sse_entry.S        | 180 +++++++++++++++++++++++++++
  9 files changed, 429 insertions(+), 3 deletions(-)

create mode 100644 arch/riscv/include/asm/sse.h create mode 100644 arch/riscv/kernel/sse.c create mode 100644 arch/riscv/kernel/sse_entry.S

arch/riscv/include/asm/asm.h +1 -3
index a8a2af6dfe9d..982c4be9a9c3 100644
@@ -90,16 +90,24 @@
  #define PER_CPU_OFFSET_SHIFT 3
  #endif
 
-.macro asm_per_cpu dst sym tmp
-	REG_L \tmp, TASK_TI_CPU_NUM(tp)
-	slli  \tmp, \tmp, PER_CPU_OFFSET_SHIFT
+.macro asm_per_cpu_with_cpu dst sym tmp cpu

arch/riscv/include/asm/asm.h line 103

Quoted 10 lines
arch/riscv/include/asm/asm.h +4 -0
+	slli  \tmp, \cpu, PER_CPU_OFFSET_SHIFT
  	la    \dst, __per_cpu_offset
  	add   \dst, \dst, \tmp
  	REG_L \tmp, 0(\dst)
  	la    \dst, \sym
  	add   \dst, \dst, \tmp
  .endm
+
+.macro asm_per_cpu dst sym tmp
+	REG_L \tmp, TASK_TI_CPU_NUM(tp)

cpu is not xlen-wide, see https://lore.kernel.org/linux-riscv/[email protected]/. Can you rebase your next version on top my fixes branch which contains this fix? Here it is https://git.kernel.org/pub/scm/linux/kernel/git/alexghiti/linux.git/log/?h=alex-fixes

Quoted 235 lines
arch/riscv/include/asm/asm.h +6 -0
+	asm_per_cpu_with_cpu \dst \sym \tmp \tmp
+.endm
  #else /* CONFIG_SMP */
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+	la    \dst, \sym
+.endm
+
  .macro asm_per_cpu dst sym tmp
  	la    \dst, \sym
  .endm
arch/riscv/include/asm/scs.h +7 -0
index 0e45db78b24b..62344daad73d 100644
@@ -18,6 +18,11 @@
  	load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp
  .endm
 
+/* Load the per-CPU IRQ shadow call stack to gp. */
+.macro scs_load_sse_stack reg_evt
+	REG_L gp, SSE_REG_EVT_SHADOW_STACK(\reg_evt)
+.endm
+
  /* Load task_scs_sp(current) to gp. */
  .macro scs_load_current
  	REG_L	gp, TASK_TI_SCS_SP(tp)
@@ -41,6 +46,8 @@
  .endm
  .macro scs_load_irq_stack tmp
  .endm
+.macro scs_load_sse_stack reg_evt
+.endm
  .macro scs_load_current
  .endm
  .macro scs_load_current_if_task_changed prev
arch/riscv/include/asm/sse.h new file +47 -0
new file mode 100644
index 000000000000..8929a268462c
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#ifndef __ASM_SSE_H
+#define __ASM_SSE_H
+
+#include <asm/sbi.h>
+
+#ifdef CONFIG_RISCV_SSE
+
+struct sse_event_interrupted_state {
+	unsigned long a6;
+	unsigned long a7;
+};
+
+struct sse_event_arch_data {
+	void *stack;
+	void *shadow_stack;
+	unsigned long tmp;
+	struct sse_event_interrupted_state interrupted;
+	unsigned long interrupted_phys;
+	u32 evt_id;
+	unsigned int hart_id;
+	unsigned int cpu_id;
+};
+
+static inline bool sse_event_is_global(u32 evt)
+{
+	return !!(evt & SBI_SSE_EVENT_GLOBAL);
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt, int cpu);
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id,
+			int cpu);
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt);
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt);
+
+void sse_handle_event(struct sse_event_arch_data *arch_evt,
+		      struct pt_regs *regs);
+asmlinkage void handle_sse(void);
+asmlinkage void do_sse(struct sse_event_arch_data *arch_evt,
+		       struct pt_regs *reg);
+
+#endif
+
+#endif
arch/riscv/include/asm/switch_to.h +14 -0
index 0e71eb82f920..cd1cead0c682 100644
@@ -88,6 +88,19 @@ static inline void __switch_to_envcfg(struct task_struct *next)
  			:: "r" (next->thread.envcfg) : "memory");
  }
 
+#ifdef CONFIG_RISCV_SSE
+DECLARE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+	__this_cpu_write(__sse_entry_task, next);
+}
+#else
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+}
+#endif
+
  extern struct task_struct *__switch_to(struct task_struct *,
  				       struct task_struct *);
 
@@ -122,6 +135,7 @@ do {							\
  	if (switch_to_should_flush_icache(__next))	\
  		local_flush_icache_all();		\
  	__switch_to_envcfg(__next);			\
+	__switch_sse_entry_task(__next);			\
  	((last) = __switch_to(__prev, __next));		\
  } while (0)
 
arch/riscv/include/asm/thread_info.h +1 -0
index f5916a70879a..28e9805e61fc 100644
@@ -36,6 +36,7 @@
  #define OVERFLOW_STACK_SIZE     SZ_4K
 
  #define IRQ_STACK_SIZE		THREAD_SIZE
+#define SSE_STACK_SIZE		THREAD_SIZE
 
  #ifndef __ASSEMBLY__
 
arch/riscv/kernel/Makefile +1 -0
index c7b542573407..62e4490b34ee 100644
@@ -99,6 +99,7 @@ obj-$(CONFIG_DYNAMIC_FTRACE)	+= mcount-dyn.o
  obj-$(CONFIG_PERF_EVENTS)	+= perf_callchain.o
  obj-$(CONFIG_HAVE_PERF_REGS)	+= perf_regs.o
  obj-$(CONFIG_RISCV_SBI)		+= sbi.o sbi_ecall.o
+obj-$(CONFIG_RISCV_SSE)		+= sse.o sse_entry.o
  ifeq ($(CONFIG_RISCV_SBI), y)
  obj-$(CONFIG_SMP)		+= sbi-ipi.o
  obj-$(CONFIG_SMP) += cpu_ops_sbi.o
arch/riscv/kernel/asm-offsets.c +14 -0
index 6e8c0d6feae9..315547c3a2ef 100644
@@ -14,6 +14,8 @@
  #include <asm/ptrace.h>
  #include <asm/cpu_ops_sbi.h>
  #include <asm/stacktrace.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
  #include <asm/suspend.h>
 
  void asm_offsets(void);
@@ -528,4 +530,16 @@ void asm_offsets(void)
  	DEFINE(FREGS_A6,	    offsetof(struct __arch_ftrace_regs, a6));
  	DEFINE(FREGS_A7,	    offsetof(struct __arch_ftrace_regs, a7));
  #endif
+
+#ifdef CONFIG_RISCV_SSE
+	OFFSET(SSE_REG_EVT_STACK, sse_event_arch_data, stack);
+	OFFSET(SSE_REG_EVT_SHADOW_STACK, sse_event_arch_data, shadow_stack);
+	OFFSET(SSE_REG_EVT_TMP, sse_event_arch_data, tmp);
+	OFFSET(SSE_REG_HART_ID, sse_event_arch_data, hart_id);
+	OFFSET(SSE_REG_CPU_ID, sse_event_arch_data, cpu_id);
+
+	DEFINE(SBI_EXT_SSE, SBI_EXT_SSE);
+	DEFINE(SBI_SSE_EVENT_COMPLETE, SBI_SSE_EVENT_COMPLETE);
+	DEFINE(ASM_NR_CPUS, NR_CPUS);
+#endif
  }
arch/riscv/kernel/sse.c new file +55 -0
new file mode 100644
index 000000000000..d2da7e23a74a
@@ -0,0 +1,154 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#include <linux/nmi.h>
+#include <linux/scs.h>
+#include <linux/bitfield.h>
+#include <linux/riscv_sse.h>
+#include <linux/percpu-defs.h>
+
+#include <asm/asm-prototypes.h>
+#include <asm/switch_to.h>
+#include <asm/irq_stack.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
+
+DEFINE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+void __weak sse_handle_event(struct sse_event_arch_data *arch_evt, struct pt_regs *regs)
+{
+}
+
+void do_sse(struct sse_event_arch_data *arch_evt, struct pt_regs *regs)
+{
+	nmi_enter();
+
+	/* Retrieve missing GPRs from SBI */
+	sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, arch_evt->evt_id,
+		  SBI_SSE_ATTR_INTERRUPTED_A6,
+		  (SBI_SSE_ATTR_INTERRUPTED_A7 - SBI_SSE_ATTR_INTERRUPTED_A6) + 1,
+		  arch_evt->interrupted_phys, 0, 0);
+
+	memcpy(&regs->a6, &arch_evt->interrupted, sizeof(arch_evt->interrupted));
+
+	sse_handle_event(arch_evt, regs);
+
+	/*
+	 * The SSE delivery path does not uses the "standard" exception path
+	 * (see sse_entry.S) and does not process any pending signal/softirqs
+	 * due to being similar to a NMI.
+	 * Some drivers (PMU, RAS) enqueue pending work that needs to be handled
+	 * as soon as possible by bottom halves. For that purpose, set the SIP
+	 * software interrupt pending bit which will force a software interrupt
+	 * to be serviced once interrupts are reenabled in the interrupted
+	 * context if they were masked or directly if unmasked.
+	 */
+	csr_set(CSR_IP, IE_SIE);
+
+	nmi_exit();
+}
+
+static void *alloc_to_stack_pointer(void *alloc)
+{
+	return alloc ? alloc + SSE_STACK_SIZE : NULL;
+}

arch/riscv/kernel/sse.c line 65

Quoted 10 lines
arch/riscv/kernel/sse.c new file +10 -0
+
+static void *stack_pointer_to_alloc(void *stack)
+{
+	return stack - SSE_STACK_SIZE;
+}
+
+#ifdef CONFIG_VMAP_STACK
+static void *sse_stack_alloc(unsigned int cpu)
+{
+	void *stack = arch_alloc_vmap_stack(SSE_STACK_SIZE, cpu_to_node(cpu));

I'm wondering what could happen if, in case of Svvptc, we take a trap in the sse path because the vmalloced stack page table entry is not yet seen by the page table walker, would that be an issue?

Quoted 53 lines
arch/riscv/kernel/sse.c new file +53 -0
+
+	return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+	vfree(stack_pointer_to_alloc(stack));
+}
+#else /* CONFIG_VMAP_STACK */
+static void *sse_stack_alloc(unsigned int cpu)
+{
+	void *stack = kmalloc(SSE_STACK_SIZE, GFP_KERNEL);
+
+	return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+	kfree(stack_pointer_to_alloc(stack));
+}
+#endif /* CONFIG_VMAP_STACK */
+
+static int sse_init_scs(int cpu, struct sse_event_arch_data *arch_evt)
+{
+	void *stack;
+
+	if (!scs_is_enabled())
+		return 0;
+
+	stack = scs_alloc(cpu_to_node(cpu));
+	if (!stack)
+		return -ENOMEM;
+
+	arch_evt->shadow_stack = stack;
+
+	return 0;
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt, int cpu)
+{
+	arch_evt->cpu_id = cpu;
+	arch_evt->hart_id = cpuid_to_hartid_map(cpu);
+}
+
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id, int cpu)
+{
+	void *stack;
+
+	arch_evt->evt_id = evt_id;
+	stack = sse_stack_alloc(cpu);
+	if (!stack)
+		return -ENOMEM;
+

arch/riscv/kernel/sse.c line 128

Quoted 10 lines
arch/riscv/kernel/sse.c new file +10 -0
+	arch_evt->stack = stack;
+
+	if (sse_init_scs(cpu, arch_evt)) {
+		sse_stack_free(arch_evt->stack);
+		return -ENOMEM;
+	}
+
+	if (sse_event_is_global(evt_id)) {
+		arch_evt->interrupted_phys =
+					virt_to_phys(&arch_evt->interrupted);

I may be missing something but I don't see why interrupted could not be a userspace address?

Quoted 98 lines
arch/riscv/kernel/sse.c new file +26 -0
+	} else {
+		arch_evt->interrupted_phys =
+				per_cpu_ptr_to_phys(&arch_evt->interrupted);
+	}
+
+	arch_sse_event_update_cpu(arch_evt, cpu);
+
+	return 0;
+}
+
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt)
+{
+	scs_free(arch_evt->shadow_stack);
+	sse_stack_free(arch_evt->stack);
+}
+
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt)
+{
+	struct sbiret sret;
+
+	sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_REGISTER, arch_evt->evt_id,
+			 (unsigned long)handle_sse, (unsigned long)arch_evt, 0,
+			 0, 0);
+
+	return sbi_err_map_linux_errno(sret.error);
+}
arch/riscv/kernel/sse_entry.S new file +66 -0
new file mode 100644
index 000000000000..112bdb7d4369
@@ -0,0 +1,180 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+
+#include <linux/init.h>
+#include <linux/linkage.h>
+
+#include <asm/asm.h>
+#include <asm/csr.h>
+#include <asm/scs.h>
+
+/* When entering handle_sse, the following registers are set:
+ * a6: contains the hartid
+ * a7: contains a sse_event_arch_data struct pointer
+ */
+SYM_CODE_START(handle_sse)
+	/* Save stack temporarily */
+	REG_S sp, SSE_REG_EVT_TMP(a7)
+	/* Set entry stack */
+	REG_L sp, SSE_REG_EVT_STACK(a7)
+
+	addi sp, sp, -(PT_SIZE_ON_STACK)
+	REG_S ra, PT_RA(sp)
+	REG_S s0, PT_S0(sp)
+	REG_S s1, PT_S1(sp)
+	REG_S s2, PT_S2(sp)
+	REG_S s3, PT_S3(sp)
+	REG_S s4, PT_S4(sp)
+	REG_S s5, PT_S5(sp)
+	REG_S s6, PT_S6(sp)
+	REG_S s7, PT_S7(sp)
+	REG_S s8, PT_S8(sp)
+	REG_S s9, PT_S9(sp)
+	REG_S s10, PT_S10(sp)
+	REG_S s11, PT_S11(sp)
+	REG_S tp, PT_TP(sp)
+	REG_S t0, PT_T0(sp)
+	REG_S t1, PT_T1(sp)
+	REG_S t2, PT_T2(sp)
+	REG_S t3, PT_T3(sp)
+	REG_S t4, PT_T4(sp)
+	REG_S t5, PT_T5(sp)
+	REG_S t6, PT_T6(sp)
+	REG_S gp, PT_GP(sp)
+	REG_S a0, PT_A0(sp)
+	REG_S a1, PT_A1(sp)
+	REG_S a2, PT_A2(sp)
+	REG_S a3, PT_A3(sp)
+	REG_S a4, PT_A4(sp)
+	REG_S a5, PT_A5(sp)
+
+	/* Retrieve entry sp */
+	REG_L a4, SSE_REG_EVT_TMP(a7)
+	/* Save CSRs */
+	csrr a0, CSR_EPC
+	csrr a1, CSR_SSTATUS
+	csrr a2, CSR_STVAL
+	csrr a3, CSR_SCAUSE
+
+	REG_S a0, PT_EPC(sp)
+	REG_S a1, PT_STATUS(sp)
+	REG_S a2, PT_BADADDR(sp)
+	REG_S a3, PT_CAUSE(sp)
+	REG_S a4, PT_SP(sp)
+

arch/riscv/kernel/sse_entry.S line 76

Quoted 10 lines
arch/riscv/kernel/sse_entry.S new file +10 -0
+	/* Disable user memory access and floating/vector computing */
+	li t0, SR_SUM | SR_FS_VS
+	csrc CSR_STATUS, t0
+
+	load_global_pointer
+	scs_load_sse_stack a7
+
+	/* Restore current task struct from __sse_entry_task */
+	li t1, ASM_NR_CPUS
+	mv t3, zero

t1 is only used in find_hart_id_slowpath and shouldn't we use the real number of present cpus instead of NR_CPUS?

t3 is clobbered by cpu_id below, is that correct?

arch/riscv/kernel/sse_entry.S line 80

Quoted 4 lines
arch/riscv/kernel/sse_entry.S new file +4 -0
+
+#ifdef CONFIG_SMP
+	REG_L t4, SSE_REG_HART_ID(a7)
+	REG_L t3, SSE_REG_CPU_ID(a7)

hart_id and cpu_id are "unsigned int", not xlen-wide fields so you must use lw here, not REG_L (a lot of bugs lately on the usage of this macro).

Quoted 8 lines
arch/riscv/kernel/sse_entry.S new file +8 -0
+
+	bne t4, a6, .Lfind_hart_id_slowpath
+
+.Lcpu_id_found:
+#endif
+	asm_per_cpu_with_cpu t2 __sse_entry_task t1 t3
+	REG_L tp, 0(t2)
+

arch/riscv/kernel/sse_entry.S line 98

Quoted 10 lines
arch/riscv/kernel/sse_entry.S new file +10 -0
+	mv a1, sp /* pt_regs on stack */
+
+	/*
+	 * Save sscratch for restoration since we might have interrupted the
+	 * kernel in early exception path and thus, we don't know the content of
+	 * sscratch.
+	 */
+	csrr s4, CSR_SSCRATCH
+	/* In-kernel scratch is 0 */
+	csrw CSR_SCRATCH, x0

csrrw s4, CSR_SSCRATCH, x0 instead?

Quoted 46 lines
arch/riscv/kernel/sse_entry.S new file +46 -0
+
+	mv a0, a7
+
+	call do_sse
+
+	csrw CSR_SSCRATCH, s4
+
+	REG_L a0, PT_STATUS(sp)
+	REG_L a1, PT_EPC(sp)
+	REG_L a2, PT_BADADDR(sp)
+	REG_L a3, PT_CAUSE(sp)
+	csrw CSR_SSTATUS, a0
+	csrw CSR_EPC, a1
+	csrw CSR_STVAL, a2
+	csrw CSR_SCAUSE, a3
+
+	REG_L ra, PT_RA(sp)
+	REG_L s0, PT_S0(sp)
+	REG_L s1, PT_S1(sp)
+	REG_L s2, PT_S2(sp)
+	REG_L s3, PT_S3(sp)
+	REG_L s4, PT_S4(sp)
+	REG_L s5, PT_S5(sp)
+	REG_L s6, PT_S6(sp)
+	REG_L s7, PT_S7(sp)
+	REG_L s8, PT_S8(sp)
+	REG_L s9, PT_S9(sp)
+	REG_L s10, PT_S10(sp)
+	REG_L s11, PT_S11(sp)
+	REG_L tp, PT_TP(sp)
+	REG_L t0, PT_T0(sp)
+	REG_L t1, PT_T1(sp)
+	REG_L t2, PT_T2(sp)
+	REG_L t3, PT_T3(sp)
+	REG_L t4, PT_T4(sp)
+	REG_L t5, PT_T5(sp)
+	REG_L t6, PT_T6(sp)
+	REG_L gp, PT_GP(sp)
+	REG_L a0, PT_A0(sp)
+	REG_L a1, PT_A1(sp)
+	REG_L a2, PT_A2(sp)
+	REG_L a3, PT_A3(sp)
+	REG_L a4, PT_A4(sp)
+	REG_L a5, PT_A5(sp)
+
+	REG_L sp, PT_SP(sp)

arch/riscv/kernel/sse_entry.S line 154

Quoted 10 lines
arch/riscv/kernel/sse_entry.S new file +10 -0
+
+	li a7, SBI_EXT_SSE
+	li a6, SBI_SSE_EVENT_COMPLETE
+	ecall
+
+#ifdef CONFIG_SMP
+.Lfind_hart_id_slowpath:
+
+/* Slowpath to find the CPU id associated to the hart id */
+la t0, __cpuid_to_hartid_map

^ Missing tab

Quoted 9 lines
arch/riscv/kernel/sse_entry.S new file +9 -0
+
+.Lhart_id_loop:
+	REG_L t2, 0(t0)
+	beq t2, a6, .Lcpu_id_found
+
+	/* Increment pointer and CPU number */
+	addi t3, t3, 1
+	addi t0, t0, RISCV_SZPTR
+	bltu t3, t1, .Lhart_id_loop

arch/riscv/kernel/sse_entry.S line 173

Quoted 10 lines
arch/riscv/kernel/sse_entry.S new file +10 -0
+
+	/*
+	 * This should never happen since we expect the hart_id to match one
+	 * of our CPU, but better be safe than sorry
+	 */
+	la tp, init_task
+	la a0, sse_hart_id_panic_string
+	la t0, panic
+	jalr t0
+

This newline is not needed ^

arch/riscv/kernel/sse_entry.S line 180

Quoted 7 lines
arch/riscv/kernel/sse_entry.S new file +7 -0
+#endif
+
+SYM_CODE_END(handle_sse)
+
+SYM_DATA_START_LOCAL(sse_hart_id_panic_string)
+    .ascii "Unable to match hart_id with cpu\0"
+SYM_DATA_END(sse_hart_id_panic_string)

Thanks,

Alex

#6 lore raw

Re: [PATCH v6 2/5] riscv: add support for SBI Supervisor Software Events extension

Quoted 109 lines · Alexandre Ghiti
Quoted 1 line · Clément Léger

The SBI SSE extension allows the supervisor software to be notified by

I would change "notified" with "interrupted" no?

Quoted 1 line

the SBI of specific events that are not maskable. The context switch is

Nit: s/the SBI/the SBI implementation

Quoted 6 lines

handled partially by the firmware which will save registers a6 and a7. When entering kernel we can rely on these 2 registers to setup the stack and save all the registers.

Since SSE events can be delivered at any time to the kernel (including during exception handling, we need a way to locate the current_task for

s/handling/handling)

Quoted 1 line

context tracking. On RISC-V, it is sotred in scratch when in user space

s/sotred/stored

Quoted 1 line

or tp when in kernel space (in which case SSCRATCH is zero). But at a

Remove "at a"

Quoted 3 lines

at the beginning of exception handling, SSCRATCH is used to swap tp and check the origin of the exception. If interrupted at that point, then, there is no way to reliably know were is located the current

s/were/where

Quoted 11 lines

task_struct. Even checking the interruption location won't work as SSE event can be nested on top of each other so the original interruption site might be lost at some point. In order to retrieve it reliably, store the current task in an additional __sse_entry_task per_cpu array. This array is then used to retrieve the current task based on the hart ID that is passed to the SSE event handler in a6.

That being said, the way the current task struct is stored should probably be reworked to find a better reliable alternative.

Since each events (and each CPU for local events) have their own

"Since each event has its own"

Quoted 2 lines

context and can preempt each other, allocate a stack (and a shadow stack if needed for each of them (and for each cpu for local events).

s/needed/needed)

Quoted 44 lines

When completing the event, if we were coming from kernel with interrupts disabled, simply return there. If coming from userspace or kernel with interrupts enabled, simulate an interrupt exception by setting IE_SIE in CSR_IP to allow delivery of signals to user task. For instance this can happen, when a RAS event has been generated by a user application and a SIGBUS has been sent to a task.

---

  arch/riscv/include/asm/asm.h         |  14 ++-
  arch/riscv/include/asm/scs.h         |   7 ++
  arch/riscv/include/asm/sse.h         |  47 +++++++

  arch/riscv/include/asm/switch_to.h   |  14 +++ arch/riscv/include/asm/thread_info.h |   1 + arch/riscv/kernel/Makefile           |   1 + arch/riscv/kernel/asm-offsets.c      |  14 +++ arch/riscv/kernel/sse.c              | 154 +++++++++++++++++++++++ arch/riscv/kernel/sse_entry.S        | 180 +++++++++++++++++++++++++++

  9 files changed, 429 insertions(+), 3 deletions(-)

  create mode 100644 arch/riscv/include/asm/sse.h create mode 100644 arch/riscv/kernel/sse.c create mode 100644 arch/riscv/kernel/sse_entry.S

arch/riscv/include/asm/asm.h +0 -0
index a8a2af6dfe9d..982c4be9a9c3 100644
@@ -90,16 +90,24 @@
   #define PER_CPU_OFFSET_SHIFT 3
   #endif

  -.macro asm_per_cpu dst sym tmp
-    REG_L \tmp, TASK_TI_CPU_NUM(tp)
-    slli  \tmp, \tmp, PER_CPU_OFFSET_SHIFT
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+    slli  \tmp, \cpu, PER_CPU_OFFSET_SHIFT
      la    \dst, __per_cpu_offset
      add   \dst, \dst, \tmp
      REG_L \tmp, 0(\dst)
      la    \dst, \sym
      add   \dst, \dst, \tmp
  .endm
+
+.macro asm_per_cpu dst sym tmp

Quoted 8 lines
Quoted 1 line

+    REG_L \tmp, TASK_TI_CPU_NUM(tp)

cpu is not xlen-wide, see https://lore.kernel.org/linux-
riscv/[email protected]/. Can you rebase your next version on top my fixes branch which contains this fix? Here it is https://git.kernel.org/pub/scm/linux/kernel/git/alexghiti/
linux.git/log/?h=alex-fixes

Hi Alex,

Acked, I'll fix that.

Quoted 250 lines
Quoted 250 lines

+    asm_per_cpu_with_cpu \dst \sym \tmp \tmp
+.endm
  #else /* CONFIG_SMP */
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+    la    \dst, \sym
+.endm
+
  .macro asm_per_cpu dst sym tmp
      la    \dst, \sym
  .endm

arch/riscv/include/asm/scs.h +0 -0
index 0e45db78b24b..62344daad73d 100644
@@ -18,6 +18,11 @@

      load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp .endm
  +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_sse_stack reg_evt
+    REG_L gp, SSE_REG_EVT_SHADOW_STACK(\reg_evt) +.endm
+
  /* Load task_scs_sp(current) to gp. */ .macro scs_load_current
      REG_L    gp, TASK_TI_SCS_SP(tp)

arch/riscv/include/asm/scs.h +0 -0
@@ -41,6 +46,8 @@

  .endm
  .macro scs_load_irq_stack tmp
  .endm
+.macro scs_load_sse_stack reg_evt
+.endm
  .macro scs_load_current
  .endm
  .macro scs_load_current_if_task_changed prev

arch/riscv/include/asm/sse.h new file +33 -0
new file mode 100644
index 000000000000..8929a268462c
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#ifndef __ASM_SSE_H
+#define __ASM_SSE_H
+
+#include <asm/sbi.h>
+
+#ifdef CONFIG_RISCV_SSE
+
+struct sse_event_interrupted_state {
+    unsigned long a6;
+    unsigned long a7;
+};
+
+struct sse_event_arch_data {
+    void *stack;
+    void *shadow_stack;
+    unsigned long tmp;
+    struct sse_event_interrupted_state interrupted;
+    unsigned long interrupted_phys;
+    u32 evt_id;
+    unsigned int hart_id;
+    unsigned int cpu_id;
+};
+
+static inline bool sse_event_is_global(u32 evt)
+{
+    return !!(evt & SBI_SSE_EVENT_GLOBAL);
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt,

int cpu);
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id,
+            int cpu);

arch/riscv/include/asm/sse.h new file +12 -0
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt);
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt);
+
+void sse_handle_event(struct sse_event_arch_data *arch_evt,
+              struct pt_regs *regs);
+asmlinkage void handle_sse(void);
+asmlinkage void do_sse(struct sse_event_arch_data *arch_evt,
+               struct pt_regs *reg);
+
+#endif
+
+#endif
arch/riscv/include/ renamed from arch/riscv/include/asm/switch_to.h +0 -0

asm/switch_to.h
index 0e71eb82f920..cd1cead0c682 100644

arch/riscv/include/asm/switch_to.h +0 -0
@@ -88,6 +88,19 @@ static inline void __switch_to_envcfg(struct
 task_struct *next)
               :: "r" (next->thread.envcfg) : "memory");
   }

  +#ifdef CONFIG_RISCV_SSE

arch/riscv/include/asm/switch_to.h +12 -0
+DECLARE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+    __this_cpu_write(__sse_entry_task, next);
+}
+#else
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+}
+#endif
+

  extern struct task_struct *__switch_to(struct task_struct *, struct task_struct *);
  @@ -122,6 +135,7 @@ do {                            \
      if (switch_to_should_flush_icache(__next))    \
          local_flush_icache_all();        \
      __switch_to_envcfg(__next);            \
+    __switch_sse_entry_task(__next);            \
      ((last) = __switch_to(__prev, __next));        \
  } while (0)
  diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/ include/asm/thread_info.h
index f5916a70879a..28e9805e61fc 100644

arch/riscv/include/asm/thread_info.h +1 -0
@@ -36,6 +36,7 @@
   #define OVERFLOW_STACK_SIZE     SZ_4K
     #define IRQ_STACK_SIZE        THREAD_SIZE
+#define SSE_STACK_SIZE        THREAD_SIZE
     #ifndef __ASSEMBLY__

  diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index c7b542573407..62e4490b34ee 100644

arch/riscv/kernel/Makefile +0 -0
@@ -99,6 +99,7 @@ obj-$(CONFIG_DYNAMIC_FTRACE)    += mcount-dyn.o

  obj-$(CONFIG_PERF_EVENTS)    += perf_callchain.o obj-$(CONFIG_HAVE_PERF_REGS)    += perf_regs.o obj-$(CONFIG_RISCV_SBI)        += sbi.o sbi_ecall.o +obj-$(CONFIG_RISCV_SSE)        += sse.o sse_entry.o ifeq ($(CONFIG_RISCV_SBI), y) obj-$(CONFIG_SMP)        += sbi-ipi.o obj-$(CONFIG_SMP) += cpu_ops_sbi.o

arch/riscv/kernel/asm- renamed from arch/riscv/kernel/asm-offsets.c +0 -0

offsets.c
index 6e8c0d6feae9..315547c3a2ef 100644

arch/riscv/kernel/asm-offsets.c +14 -0
@@ -14,6 +14,8 @@
   #include <asm/ptrace.h>
   #include <asm/cpu_ops_sbi.h>
   #include <asm/stacktrace.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
   #include <asm/suspend.h>
     void asm_offsets(void);
@@ -528,4 +530,16 @@ void asm_offsets(void)
       DEFINE(FREGS_A6,        offsetof(struct __arch_ftrace_regs, a6));
       DEFINE(FREGS_A7,        offsetof(struct __arch_ftrace_regs, a7));
   #endif
+
+#ifdef CONFIG_RISCV_SSE
+    OFFSET(SSE_REG_EVT_STACK, sse_event_arch_data, stack);
+    OFFSET(SSE_REG_EVT_SHADOW_STACK, sse_event_arch_data, shadow_stack);
+    OFFSET(SSE_REG_EVT_TMP, sse_event_arch_data, tmp);
+    OFFSET(SSE_REG_HART_ID, sse_event_arch_data, hart_id);
+    OFFSET(SSE_REG_CPU_ID, sse_event_arch_data, cpu_id);
+
+    DEFINE(SBI_EXT_SSE, SBI_EXT_SSE);
+    DEFINE(SBI_SSE_EVENT_COMPLETE, SBI_SSE_EVENT_COMPLETE);
+    DEFINE(ASM_NR_CPUS, NR_CPUS);
+#endif
   }
arch/riscv/kernel/sse.c new file +19 -0
new file mode 100644
index 000000000000..d2da7e23a74a
@@ -0,0 +1,154 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#include <linux/nmi.h>
+#include <linux/scs.h>
+#include <linux/bitfield.h>
+#include <linux/riscv_sse.h>
+#include <linux/percpu-defs.h>
+
+#include <asm/asm-prototypes.h>
+#include <asm/switch_to.h>
+#include <asm/irq_stack.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
+
+DEFINE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+void __weak sse_handle_event(struct sse_event_arch_data *arch_evt,

struct pt_regs *regs)

arch/riscv/kernel/sse.c new file +15 -0
+{
+}
+
+void do_sse(struct sse_event_arch_data *arch_evt, struct pt_regs *regs)
+{
+    nmi_enter();
+
+    /* Retrieve missing GPRs from SBI */
+    sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, arch_evt->evt_id,
+          SBI_SSE_ATTR_INTERRUPTED_A6,
+          (SBI_SSE_ATTR_INTERRUPTED_A7 - SBI_SSE_ATTR_INTERRUPTED_A6)
+ 1,
+          arch_evt->interrupted_phys, 0, 0);
+
+    memcpy(&regs->a6, &arch_evt->interrupted, sizeof(arch_evt-
Quoted 1 line

interrupted));

arch/riscv/kernel/sse.c new file +6 -0
+
+    sse_handle_event(arch_evt, regs);
+
+    /*
+     * The SSE delivery path does not uses the "standard" exception path
+     * (see sse_entry.S) and does not process any pending signal/

softirqs
+     * due to being similar to a NMI.
+     * Some drivers (PMU, RAS) enqueue pending work that needs to be handled
+     * as soon as possible by bottom halves. For that purpose, set the SIP
+     * software interrupt pending bit which will force a software interrupt
+     * to be serviced once interrupts are reenabled in the interrupted +     * context if they were masked or directly if unmasked. +     */
+    csr_set(CSR_IP, IE_SIE);

arch/riscv/kernel/sse.c new file +16 -0
+
+    nmi_exit();
+}
+
+static void *alloc_to_stack_pointer(void *alloc)
+{
+    return alloc ? alloc + SSE_STACK_SIZE : NULL;
+}
+
+static void *stack_pointer_to_alloc(void *stack)
+{
+    return stack - SSE_STACK_SIZE;
+}
+
+#ifdef CONFIG_VMAP_STACK
+static void *sse_stack_alloc(unsigned int cpu)
Quoted 8 lines
Quoted 3 lines

+{
+    void *stack = arch_alloc_vmap_stack(SSE_STACK_SIZE, cpu_to_node(cpu));

I'm wondering what could happen if, in case of Svvptc, we take a trap in the sse path because the vmalloced stack page table entry is not yet seen by the page table walker, would that be an issue?

I'll take a look at that but if it trap at the beginning of the SSE handler, it's going to clobber anything that wasn't saved. So basically fatal. Should I 'touch' the stack itself after allocation to be sure it's mapped in the page table ?

Quoted 63 lines
Quoted 63 lines
arch/riscv/kernel/sse.c new file +39 -0
+
+    return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+    vfree(stack_pointer_to_alloc(stack));
+}
+#else /* CONFIG_VMAP_STACK */
+static void *sse_stack_alloc(unsigned int cpu)
+{
+    void *stack = kmalloc(SSE_STACK_SIZE, GFP_KERNEL);
+
+    return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+    kfree(stack_pointer_to_alloc(stack));
+}
+#endif /* CONFIG_VMAP_STACK */
+
+static int sse_init_scs(int cpu, struct sse_event_arch_data *arch_evt)
+{
+    void *stack;
+
+    if (!scs_is_enabled())
+        return 0;
+
+    stack = scs_alloc(cpu_to_node(cpu));
+    if (!stack)
+        return -ENOMEM;
+
+    arch_evt->shadow_stack = stack;
+
+    return 0;
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt,

int cpu)

arch/riscv/kernel/sse.c new file +6 -0
+{
+    arch_evt->cpu_id = cpu;
+    arch_evt->hart_id = cpuid_to_hartid_map(cpu);
+}
+
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32

evt_id, int cpu)

arch/riscv/kernel/sse.c new file +14 -0
+{
+    void *stack;
+
+    arch_evt->evt_id = evt_id;
+    stack = sse_stack_alloc(cpu);
+    if (!stack)
+        return -ENOMEM;
+
+    arch_evt->stack = stack;
+
+    if (sse_init_scs(cpu, arch_evt)) {
+        sse_stack_free(arch_evt->stack);
+        return -ENOMEM;
+    }
Quoted 8 lines
Quoted 4 lines

+
+    if (sse_event_is_global(evt_id)) {
+        arch_evt->interrupted_phys =
+                    virt_to_phys(&arch_evt->interrupted);

I may be missing something but I don't see why interrupted could not be a userspace address?

I'm not sure to follow what you are saying but arch_evt is allocated via kzalloc() from the driver layer (see next commit) and is passed to arch_sse_init_event(). There is no way for this struct to be allocated/passed by userspace.

Quoted 108 lines
Quoted 108 lines

+    } else {
+        arch_evt->interrupted_phys =
+                per_cpu_ptr_to_phys(&arch_evt->interrupted); +    }

arch/riscv/kernel/sse.c new file +17 -0
+
+    arch_sse_event_update_cpu(arch_evt, cpu);
+
+    return 0;
+}
+
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt)
+{
+    scs_free(arch_evt->shadow_stack);
+    sse_stack_free(arch_evt->stack);
+}
+
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt)
+{
+    struct sbiret sret;
+
+    sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_REGISTER, arch_evt-
Quoted 1 line

evt_id,

+             (unsigned long)handle_sse, (unsigned long)arch_evt, 0, +             0, 0);

arch/riscv/kernel/sse.c new file +3 -0
+
+    return sbi_err_map_linux_errno(sret.error);
+}
arch/riscv/kernel/ renamed from arch/riscv/kernel/sse_entry.S +0 -0

sse_entry.S
new file mode 100644
index 000000000000..112bdb7d4369

arch/riscv/kernel/sse_entry.S new file +72 -0
@@ -0,0 +1,180 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+
+#include <linux/init.h>
+#include <linux/linkage.h>
+
+#include <asm/asm.h>
+#include <asm/csr.h>
+#include <asm/scs.h>
+
+/* When entering handle_sse, the following registers are set:
+ * a6: contains the hartid
+ * a7: contains a sse_event_arch_data struct pointer
+ */
+SYM_CODE_START(handle_sse)
+    /* Save stack temporarily */
+    REG_S sp, SSE_REG_EVT_TMP(a7)
+    /* Set entry stack */
+    REG_L sp, SSE_REG_EVT_STACK(a7)
+
+    addi sp, sp, -(PT_SIZE_ON_STACK)
+    REG_S ra, PT_RA(sp)
+    REG_S s0, PT_S0(sp)
+    REG_S s1, PT_S1(sp)
+    REG_S s2, PT_S2(sp)
+    REG_S s3, PT_S3(sp)
+    REG_S s4, PT_S4(sp)
+    REG_S s5, PT_S5(sp)
+    REG_S s6, PT_S6(sp)
+    REG_S s7, PT_S7(sp)
+    REG_S s8, PT_S8(sp)
+    REG_S s9, PT_S9(sp)
+    REG_S s10, PT_S10(sp)
+    REG_S s11, PT_S11(sp)
+    REG_S tp, PT_TP(sp)
+    REG_S t0, PT_T0(sp)
+    REG_S t1, PT_T1(sp)
+    REG_S t2, PT_T2(sp)
+    REG_S t3, PT_T3(sp)
+    REG_S t4, PT_T4(sp)
+    REG_S t5, PT_T5(sp)
+    REG_S t6, PT_T6(sp)
+    REG_S gp, PT_GP(sp)
+    REG_S a0, PT_A0(sp)
+    REG_S a1, PT_A1(sp)
+    REG_S a2, PT_A2(sp)
+    REG_S a3, PT_A3(sp)
+    REG_S a4, PT_A4(sp)
+    REG_S a5, PT_A5(sp)
+
+    /* Retrieve entry sp */
+    REG_L a4, SSE_REG_EVT_TMP(a7)
+    /* Save CSRs */
+    csrr a0, CSR_EPC
+    csrr a1, CSR_SSTATUS
+    csrr a2, CSR_STVAL
+    csrr a3, CSR_SCAUSE
+
+    REG_S a0, PT_EPC(sp)
+    REG_S a1, PT_STATUS(sp)
+    REG_S a2, PT_BADADDR(sp)
+    REG_S a3, PT_CAUSE(sp)
+    REG_S a4, PT_SP(sp)
+
+    /* Disable user memory access and floating/vector computing */
+    li t0, SR_SUM | SR_FS_VS
+    csrc CSR_STATUS, t0
+
+    load_global_pointer
+    scs_load_sse_stack a7
Quoted 8 lines

arch/riscv/kernel/sse_entry.S line 76

Quoted 4 lines
arch/riscv/kernel/sse_entry.S new file +4 -0
+
+    /* Restore current task struct from __sse_entry_task */
+    li t1, ASM_NR_CPUS
+    mv t3, zero

t1 is only used in find_hart_id_slowpath and shouldn't we use the real number of present cpus instead of NR_CPUS?

Indeed, that should be moved to the slow path which iterates over hart_id_to_cpu_map. This array is sized based on NR_CPUS which is why I use NR_CPUS rather than loading some other variable with the count opf onlines cpus. Moreover, the slowpath should never loop up to NR_CPUS since the SBI must passed a valid hartid index.

Quoted 2 lines

t3 is clobbered by cpu_id below, is that correct?

In case of !SMP it is kept to zero. But I'll move that in a else below.

Quoted 10 lines

arch/riscv/kernel/sse_entry.S line 80

Quoted 6 lines
arch/riscv/kernel/sse_entry.S new file +4 -0
+
+#ifdef CONFIG_SMP
+    REG_L t4, SSE_REG_HART_ID(a7)
+    REG_L t3, SSE_REG_CPU_ID(a7)

hart_id and cpu_id are "unsigned int", not xlen-wide fields so you must use lw here, not REG_L (a lot of bugs lately on the usage of this macro).

Ouch, thanks for catching that.

Quoted 16 lines
Quoted 16 lines
arch/riscv/kernel/sse_entry.S new file +13 -0
+
+    bne t4, a6, .Lfind_hart_id_slowpath
+
+.Lcpu_id_found:
+#endif
+    asm_per_cpu_with_cpu t2 __sse_entry_task t1 t3
+    REG_L tp, 0(t2)
+
+    mv a1, sp /* pt_regs on stack */
+
+    /*
+     * Save sscratch for restoration since we might have interrupted the
+     * kernel in early exception path and thus, we don't know the

content of

Quoted 8 lines
Quoted 5 lines

+     * sscratch.
+     */
+    csrr s4, CSR_SSCRATCH
+    /* In-kernel scratch is 0 */
+    csrw CSR_SCRATCH, x0

csrrw s4, CSR_SSCRATCH, x0 instead?

Ah yeah.

Thanks !

Clément

Quoted 98 lines
Quoted 48 lines
arch/riscv/kernel/sse_entry.S new file +46 -0
+
+    mv a0, a7
+
+    call do_sse
+
+    csrw CSR_SSCRATCH, s4
+
+    REG_L a0, PT_STATUS(sp)
+    REG_L a1, PT_EPC(sp)
+    REG_L a2, PT_BADADDR(sp)
+    REG_L a3, PT_CAUSE(sp)
+    csrw CSR_SSTATUS, a0
+    csrw CSR_EPC, a1
+    csrw CSR_STVAL, a2
+    csrw CSR_SCAUSE, a3
+
+    REG_L ra, PT_RA(sp)
+    REG_L s0, PT_S0(sp)
+    REG_L s1, PT_S1(sp)
+    REG_L s2, PT_S2(sp)
+    REG_L s3, PT_S3(sp)
+    REG_L s4, PT_S4(sp)
+    REG_L s5, PT_S5(sp)
+    REG_L s6, PT_S6(sp)
+    REG_L s7, PT_S7(sp)
+    REG_L s8, PT_S8(sp)
+    REG_L s9, PT_S9(sp)
+    REG_L s10, PT_S10(sp)
+    REG_L s11, PT_S11(sp)
+    REG_L tp, PT_TP(sp)
+    REG_L t0, PT_T0(sp)
+    REG_L t1, PT_T1(sp)
+    REG_L t2, PT_T2(sp)
+    REG_L t3, PT_T3(sp)
+    REG_L t4, PT_T4(sp)
+    REG_L t5, PT_T5(sp)
+    REG_L t6, PT_T6(sp)
+    REG_L gp, PT_GP(sp)
+    REG_L a0, PT_A0(sp)
+    REG_L a1, PT_A1(sp)
+    REG_L a2, PT_A2(sp)
+    REG_L a3, PT_A3(sp)
+    REG_L a4, PT_A4(sp)
+    REG_L a5, PT_A5(sp)
+
+    REG_L sp, PT_SP(sp)

arch/riscv/kernel/sse_entry.S line 154

Quoted 10 lines
arch/riscv/kernel/sse_entry.S new file +10 -0
+
+    li a7, SBI_EXT_SSE
+    li a6, SBI_SSE_EVENT_COMPLETE
+    ecall
+
+#ifdef CONFIG_SMP
+.Lfind_hart_id_slowpath:
+
+/* Slowpath to find the CPU id associated to the hart id */
+la t0, __cpuid_to_hartid_map

^ Missing tab

Quoted 9 lines
arch/riscv/kernel/sse_entry.S new file +9 -0
+
+.Lhart_id_loop:
+    REG_L t2, 0(t0)
+    beq t2, a6, .Lcpu_id_found
+
+    /* Increment pointer and CPU number */
+    addi t3, t3, 1
+    addi t0, t0, RISCV_SZPTR
+    bltu t3, t1, .Lhart_id_loop

arch/riscv/kernel/sse_entry.S line 173

Quoted 10 lines
arch/riscv/kernel/sse_entry.S new file +10 -0
+
+    /*
+     * This should never happen since we expect the hart_id to match one
+     * of our CPU, but better be safe than sorry
+     */
+    la tp, init_task
+    la a0, sse_hart_id_panic_string
+    la t0, panic
+    jalr t0
+

This newline is not needed ^

arch/riscv/kernel/sse_entry.S line 180

Quoted 7 lines
arch/riscv/kernel/sse_entry.S new file +7 -0
+#endif
+
+SYM_CODE_END(handle_sse)
+
+SYM_DATA_START_LOCAL(sse_hart_id_panic_string)
+    .ascii "Unable to match hart_id with cpu\0"
+SYM_DATA_END(sse_hart_id_panic_string)

Thanks,

Alex

#7 lore raw

Re: [PATCH v6 2/5] riscv: add support for SBI Supervisor Software Events extension

Quoted 369 lines · Clément Léger
Quoted 99 lines · Alexandre Ghiti
Quoted 1 line · Clément Léger

The SBI SSE extension allows the supervisor software to be notified by

I would change "notified" with "interrupted" no?

Quoted 1 line

the SBI of specific events that are not maskable. The context switch is

Nit: s/the SBI/the SBI implementation

Quoted 6 lines

handled partially by the firmware which will save registers a6 and a7. When entering kernel we can rely on these 2 registers to setup the stack and save all the registers.

Since SSE events can be delivered at any time to the kernel (including during exception handling, we need a way to locate the current_task for

s/handling/handling)

Quoted 1 line

context tracking. On RISC-V, it is sotred in scratch when in user space

s/sotred/stored

Quoted 1 line

or tp when in kernel space (in which case SSCRATCH is zero). But at a

Remove "at a"

Quoted 3 lines

at the beginning of exception handling, SSCRATCH is used to swap tp and check the origin of the exception. If interrupted at that point, then, there is no way to reliably know were is located the current

s/were/where

Quoted 11 lines

task_struct. Even checking the interruption location won't work as SSE event can be nested on top of each other so the original interruption site might be lost at some point. In order to retrieve it reliably, store the current task in an additional __sse_entry_task per_cpu array. This array is then used to retrieve the current task based on the hart ID that is passed to the SSE event handler in a6.

That being said, the way the current task struct is stored should probably be reworked to find a better reliable alternative.

Since each events (and each CPU for local events) have their own

"Since each event has its own"

Quoted 2 lines

context and can preempt each other, allocate a stack (and a shadow stack if needed for each of them (and for each cpu for local events).

s/needed/needed)

Quoted 42 lines

When completing the event, if we were coming from kernel with interrupts disabled, simply return there. If coming from userspace or kernel with interrupts enabled, simulate an interrupt exception by setting IE_SIE in CSR_IP to allow delivery of signals to user task. For instance this can happen, when a RAS event has been generated by a user application and a SIGBUS has been sent to a task.

---
   arch/riscv/include/asm/asm.h         |  14 ++-
   arch/riscv/include/asm/scs.h         |   7 ++
   arch/riscv/include/asm/sse.h         |  47 +++++++
   arch/riscv/include/asm/switch_to.h   |  14 +++
   arch/riscv/include/asm/thread_info.h |   1 +
   arch/riscv/kernel/Makefile           |   1 +
   arch/riscv/kernel/asm-offsets.c      |  14 +++
   arch/riscv/kernel/sse.c              | 154 +++++++++++++++++++++++
   arch/riscv/kernel/sse_entry.S        | 180 +++++++++++++++++++++++++++
   9 files changed, 429 insertions(+), 3 deletions(-)

  create mode 100644 arch/riscv/include/asm/sse.h create mode 100644 arch/riscv/kernel/sse.c create mode 100644 arch/riscv/kernel/sse_entry.S

arch/riscv/include/asm/asm.h +3 -2
index a8a2af6dfe9d..982c4be9a9c3 100644
@@ -90,16 +90,24 @@
   #define PER_CPU_OFFSET_SHIFT 3
   #endif
   -.macro asm_per_cpu dst sym tmp
-    REG_L \tmp, TASK_TI_CPU_NUM(tp)
-    slli  \tmp, \tmp, PER_CPU_OFFSET_SHIFT
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+    slli  \tmp, \cpu, PER_CPU_OFFSET_SHIFT
       la    \dst, __per_cpu_offset
       add   \dst, \dst, \tmp
       REG_L \tmp, 0(\dst)
       la    \dst, \sym
       add   \dst, \dst, \tmp
   .endm
+
Quoted 8 lines

arch/riscv/include/asm/asm.h line 103

Quoted 2 lines
arch/riscv/include/asm/asm.h +2 -0
+.macro asm_per_cpu dst sym tmp
+    REG_L \tmp, TASK_TI_CPU_NUM(tp)

cpu is not xlen-wide, see https://lore.kernel.org/linux-
riscv/[email protected]/. Can you rebase your next version on top my fixes branch which contains this fix? Here it is https://git.kernel.org/pub/scm/linux/kernel/git/alexghiti/
linux.git/log/?h=alex-fixes

Hi Alex,

Acked, I'll fix that.

Quoted 248 lines
Quoted 248 lines
arch/riscv/include/asm/asm.h +6 -0
+    asm_per_cpu_with_cpu \dst \sym \tmp \tmp
+.endm
   #else /* CONFIG_SMP */
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+    la    \dst, \sym
+.endm
+
   .macro asm_per_cpu dst sym tmp
       la    \dst, \sym
   .endm
arch/riscv/include/asm/scs.h +6 -0
index 0e45db78b24b..62344daad73d 100644
@@ -18,6 +18,11 @@
       load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp
   .endm
   +/* Load the per-CPU IRQ shadow call stack to gp. */
+.macro scs_load_sse_stack reg_evt
+    REG_L gp, SSE_REG_EVT_SHADOW_STACK(\reg_evt)
+.endm
+
   /* Load task_scs_sp(current) to gp. */
   .macro scs_load_current
       REG_L    gp, TASK_TI_SCS_SP(tp)
@@ -41,6 +46,8 @@
   .endm
   .macro scs_load_irq_stack tmp
   .endm
+.macro scs_load_sse_stack reg_evt
+.endm
   .macro scs_load_current
   .endm
   .macro scs_load_current_if_task_changed prev
arch/riscv/include/asm/sse.h new file +33 -0
new file mode 100644
index 000000000000..8929a268462c
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#ifndef __ASM_SSE_H
+#define __ASM_SSE_H
+
+#include <asm/sbi.h>
+
+#ifdef CONFIG_RISCV_SSE
+
+struct sse_event_interrupted_state {
+    unsigned long a6;
+    unsigned long a7;
+};
+
+struct sse_event_arch_data {
+    void *stack;
+    void *shadow_stack;
+    unsigned long tmp;
+    struct sse_event_interrupted_state interrupted;
+    unsigned long interrupted_phys;
+    u32 evt_id;
+    unsigned int hart_id;
+    unsigned int cpu_id;
+};
+
+static inline bool sse_event_is_global(u32 evt)
+{
+    return !!(evt & SBI_SSE_EVENT_GLOBAL);
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt,

int cpu);
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id,
+            int cpu);

arch/riscv/include/asm/sse.h new file +12 -0
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt);
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt);
+
+void sse_handle_event(struct sse_event_arch_data *arch_evt,
+              struct pt_regs *regs);
+asmlinkage void handle_sse(void);
+asmlinkage void do_sse(struct sse_event_arch_data *arch_evt,
+               struct pt_regs *reg);
+
+#endif
+
+#endif
arch/riscv/include/ renamed from arch/riscv/include/asm/switch_to.h +0 -0

asm/switch_to.h
index 0e71eb82f920..cd1cead0c682 100644

arch/riscv/include/asm/switch_to.h +13 -0
@@ -88,6 +88,19 @@ static inline void __switch_to_envcfg(struct
 task_struct *next)
               :: "r" (next->thread.envcfg) : "memory");
   }
   +#ifdef CONFIG_RISCV_SSE
+DECLARE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+    __this_cpu_write(__sse_entry_task, next);
+}
+#else
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+}
+#endif
+
   extern struct task_struct *__switch_to(struct task_struct *,
                          struct task_struct *);
   @@ -122,6 +135,7 @@ do {                            \
       if (switch_to_should_flush_icache(__next))    \
           local_flush_icache_all();        \
       __switch_to_envcfg(__next);            \
+    __switch_sse_entry_task(__next);            \
       ((last) = __switch_to(__prev, __next));        \
   } while (0)
   diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/

include/asm/thread_info.h
index f5916a70879a..28e9805e61fc 100644

arch/riscv/include/asm/thread_info.h +1 -0
@@ -36,6 +36,7 @@
   #define OVERFLOW_STACK_SIZE     SZ_4K
     #define IRQ_STACK_SIZE        THREAD_SIZE
+#define SSE_STACK_SIZE        THREAD_SIZE
     #ifndef __ASSEMBLY__
   diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile

index c7b542573407..62e4490b34ee 100644

arch/riscv/kernel/Makefile +1 -0
@@ -99,6 +99,7 @@ obj-$(CONFIG_DYNAMIC_FTRACE)    += mcount-dyn.o
   obj-$(CONFIG_PERF_EVENTS)    += perf_callchain.o
   obj-$(CONFIG_HAVE_PERF_REGS)    += perf_regs.o
   obj-$(CONFIG_RISCV_SBI)        += sbi.o sbi_ecall.o
+obj-$(CONFIG_RISCV_SSE)        += sse.o sse_entry.o
   ifeq ($(CONFIG_RISCV_SBI), y)
   obj-$(CONFIG_SMP)        += sbi-ipi.o
   obj-$(CONFIG_SMP) += cpu_ops_sbi.o
arch/riscv/kernel/asm- renamed from arch/riscv/kernel/asm-offsets.c +0 -0

offsets.c
index 6e8c0d6feae9..315547c3a2ef 100644

arch/riscv/kernel/asm-offsets.c +14 -0
@@ -14,6 +14,8 @@
   #include <asm/ptrace.h>
   #include <asm/cpu_ops_sbi.h>
   #include <asm/stacktrace.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
   #include <asm/suspend.h>
     void asm_offsets(void);
@@ -528,4 +530,16 @@ void asm_offsets(void)
       DEFINE(FREGS_A6,        offsetof(struct __arch_ftrace_regs, a6));
       DEFINE(FREGS_A7,        offsetof(struct __arch_ftrace_regs, a7));
   #endif
+
+#ifdef CONFIG_RISCV_SSE
+    OFFSET(SSE_REG_EVT_STACK, sse_event_arch_data, stack);
+    OFFSET(SSE_REG_EVT_SHADOW_STACK, sse_event_arch_data, shadow_stack);
+    OFFSET(SSE_REG_EVT_TMP, sse_event_arch_data, tmp);
+    OFFSET(SSE_REG_HART_ID, sse_event_arch_data, hart_id);
+    OFFSET(SSE_REG_CPU_ID, sse_event_arch_data, cpu_id);
+
+    DEFINE(SBI_EXT_SSE, SBI_EXT_SSE);
+    DEFINE(SBI_SSE_EVENT_COMPLETE, SBI_SSE_EVENT_COMPLETE);
+    DEFINE(ASM_NR_CPUS, NR_CPUS);
+#endif
   }
arch/riscv/kernel/sse.c new file +19 -0
new file mode 100644
index 000000000000..d2da7e23a74a
@@ -0,0 +1,154 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#include <linux/nmi.h>
+#include <linux/scs.h>
+#include <linux/bitfield.h>
+#include <linux/riscv_sse.h>
+#include <linux/percpu-defs.h>
+
+#include <asm/asm-prototypes.h>
+#include <asm/switch_to.h>
+#include <asm/irq_stack.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
+
+DEFINE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+void __weak sse_handle_event(struct sse_event_arch_data *arch_evt,

struct pt_regs *regs)

arch/riscv/kernel/sse.c new file +15 -0
+{
+}
+
+void do_sse(struct sse_event_arch_data *arch_evt, struct pt_regs *regs)
+{
+    nmi_enter();
+
+    /* Retrieve missing GPRs from SBI */
+    sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, arch_evt->evt_id,
+          SBI_SSE_ATTR_INTERRUPTED_A6,
+          (SBI_SSE_ATTR_INTERRUPTED_A7 - SBI_SSE_ATTR_INTERRUPTED_A6)
+ 1,
+          arch_evt->interrupted_phys, 0, 0);
+
+    memcpy(&regs->a6, &arch_evt->interrupted, sizeof(arch_evt-
Quoted 1 line

interrupted));

arch/riscv/kernel/sse.c new file +6 -0
+
+    sse_handle_event(arch_evt, regs);
+
+    /*
+     * The SSE delivery path does not uses the "standard" exception path
+     * (see sse_entry.S) and does not process any pending signal/

softirqs
+     * due to being similar to a NMI.
+     * Some drivers (PMU, RAS) enqueue pending work that needs to be handled
+     * as soon as possible by bottom halves. For that purpose, set the SIP
+     * software interrupt pending bit which will force a software interrupt
+     * to be serviced once interrupts are reenabled in the interrupted +     * context if they were masked or directly if unmasked. +     */
+    csr_set(CSR_IP, IE_SIE);

arch/riscv/kernel/sse.c new file +15 -0
+
+    nmi_exit();
+}
+
+static void *alloc_to_stack_pointer(void *alloc)
+{
+    return alloc ? alloc + SSE_STACK_SIZE : NULL;
+}
+
+static void *stack_pointer_to_alloc(void *stack)
+{
+    return stack - SSE_STACK_SIZE;
+}
+
+#ifdef CONFIG_VMAP_STACK
Quoted 8 lines
Quoted 4 lines
arch/riscv/kernel/sse.c new file +3 -0
+static void *sse_stack_alloc(unsigned int cpu)
+{
+    void *stack = arch_alloc_vmap_stack(SSE_STACK_SIZE,

cpu_to_node(cpu));

I'm wondering what could happen if, in case of Svvptc, we take a trap in the sse path because the vmalloced stack page table entry is not yet seen by the page table walker, would that be an issue?

Quoted 4 lines

I'll take a look at that but if it trap at the beginning of the SSE handler, it's going to clobber anything that wasn't saved. So basically fatal. Should I 'touch' the stack itself after allocation to be sure it's mapped in the page table ?

Actually no matter svvptc or not, since we don't emit a sfence.vma after each vmalloc allocation, we could trap. So maybe in that case the simplest solution is to sfence.vma right after the allocation.

Quoted 70 lines
Quoted 62 lines
Quoted 62 lines
arch/riscv/kernel/sse.c new file +39 -0
+
+    return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+    vfree(stack_pointer_to_alloc(stack));
+}
+#else /* CONFIG_VMAP_STACK */
+static void *sse_stack_alloc(unsigned int cpu)
+{
+    void *stack = kmalloc(SSE_STACK_SIZE, GFP_KERNEL);
+
+    return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+    kfree(stack_pointer_to_alloc(stack));
+}
+#endif /* CONFIG_VMAP_STACK */
+
+static int sse_init_scs(int cpu, struct sse_event_arch_data *arch_evt)
+{
+    void *stack;
+
+    if (!scs_is_enabled())
+        return 0;
+
+    stack = scs_alloc(cpu_to_node(cpu));
+    if (!stack)
+        return -ENOMEM;
+
+    arch_evt->shadow_stack = stack;
+
+    return 0;
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt,

int cpu)

arch/riscv/kernel/sse.c new file +6 -0
+{
+    arch_evt->cpu_id = cpu;
+    arch_evt->hart_id = cpuid_to_hartid_map(cpu);
+}
+
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32

evt_id, int cpu)

arch/riscv/kernel/sse.c new file +13 -0
+{
+    void *stack;
+
+    arch_evt->evt_id = evt_id;
+    stack = sse_stack_alloc(cpu);
+    if (!stack)
+        return -ENOMEM;
+
+    arch_evt->stack = stack;
+
+    if (sse_init_scs(cpu, arch_evt)) {
+        sse_stack_free(arch_evt->stack);
+        return -ENOMEM;
Quoted 8 lines
Quoted 5 lines

+    }
+
+    if (sse_event_is_global(evt_id)) {
+        arch_evt->interrupted_phys =
+                    virt_to_phys(&arch_evt->interrupted);

I may be missing something but I don't see why interrupted could not be a userspace address?

Quoted 4 lines

I'm not sure to follow what you are saying but arch_evt is allocated via kzalloc() from the driver layer (see next commit) and is passed to arch_sse_init_event(). There is no way for this struct to be allocated/passed by userspace.

My bad, I missed the '&'!

Quoted 256 lines
Quoted 107 lines
Quoted 107 lines

+    } else {
+        arch_evt->interrupted_phys =
+                per_cpu_ptr_to_phys(&arch_evt->interrupted); +    }

arch/riscv/kernel/sse.c new file +17 -0
+
+    arch_sse_event_update_cpu(arch_evt, cpu);
+
+    return 0;
+}
+
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt)
+{
+    scs_free(arch_evt->shadow_stack);
+    sse_stack_free(arch_evt->stack);
+}
+
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt)
+{
+    struct sbiret sret;
+
+    sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_REGISTER, arch_evt-
Quoted 1 line

evt_id,

+             (unsigned long)handle_sse, (unsigned long)arch_evt, 0, +             0, 0);

arch/riscv/kernel/sse.c new file +3 -0
+
+    return sbi_err_map_linux_errno(sret.error);
+}
arch/riscv/kernel/ renamed from arch/riscv/kernel/sse_entry.S +0 -0

sse_entry.S
new file mode 100644
index 000000000000..112bdb7d4369

arch/riscv/kernel/sse_entry.S new file +71 -0
@@ -0,0 +1,180 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+
+#include <linux/init.h>
+#include <linux/linkage.h>
+
+#include <asm/asm.h>
+#include <asm/csr.h>
+#include <asm/scs.h>
+
+/* When entering handle_sse, the following registers are set:
+ * a6: contains the hartid
+ * a7: contains a sse_event_arch_data struct pointer
+ */
+SYM_CODE_START(handle_sse)
+    /* Save stack temporarily */
+    REG_S sp, SSE_REG_EVT_TMP(a7)
+    /* Set entry stack */
+    REG_L sp, SSE_REG_EVT_STACK(a7)
+
+    addi sp, sp, -(PT_SIZE_ON_STACK)
+    REG_S ra, PT_RA(sp)
+    REG_S s0, PT_S0(sp)
+    REG_S s1, PT_S1(sp)
+    REG_S s2, PT_S2(sp)
+    REG_S s3, PT_S3(sp)
+    REG_S s4, PT_S4(sp)
+    REG_S s5, PT_S5(sp)
+    REG_S s6, PT_S6(sp)
+    REG_S s7, PT_S7(sp)
+    REG_S s8, PT_S8(sp)
+    REG_S s9, PT_S9(sp)
+    REG_S s10, PT_S10(sp)
+    REG_S s11, PT_S11(sp)
+    REG_S tp, PT_TP(sp)
+    REG_S t0, PT_T0(sp)
+    REG_S t1, PT_T1(sp)
+    REG_S t2, PT_T2(sp)
+    REG_S t3, PT_T3(sp)
+    REG_S t4, PT_T4(sp)
+    REG_S t5, PT_T5(sp)
+    REG_S t6, PT_T6(sp)
+    REG_S gp, PT_GP(sp)
+    REG_S a0, PT_A0(sp)
+    REG_S a1, PT_A1(sp)
+    REG_S a2, PT_A2(sp)
+    REG_S a3, PT_A3(sp)
+    REG_S a4, PT_A4(sp)
+    REG_S a5, PT_A5(sp)
+
+    /* Retrieve entry sp */
+    REG_L a4, SSE_REG_EVT_TMP(a7)
+    /* Save CSRs */
+    csrr a0, CSR_EPC
+    csrr a1, CSR_SSTATUS
+    csrr a2, CSR_STVAL
+    csrr a3, CSR_SCAUSE
+
+    REG_S a0, PT_EPC(sp)
+    REG_S a1, PT_STATUS(sp)
+    REG_S a2, PT_BADADDR(sp)
+    REG_S a3, PT_CAUSE(sp)
+    REG_S a4, PT_SP(sp)
+
+    /* Disable user memory access and floating/vector computing */
+    li t0, SR_SUM | SR_FS_VS
+    csrc CSR_STATUS, t0
+
+    load_global_pointer
Quoted 8 lines

arch/riscv/kernel/sse_entry.S line 76

Quoted 5 lines
arch/riscv/kernel/sse_entry.S new file +5 -0
+    scs_load_sse_stack a7
+
+    /* Restore current task struct from __sse_entry_task */
+    li t1, ASM_NR_CPUS
+    mv t3, zero

t1 is only used in find_hart_id_slowpath and shouldn't we use the real number of present cpus instead of NR_CPUS?

Indeed, that should be moved to the slow path which iterates over hart_id_to_cpu_map. This array is sized based on NR_CPUS which is why I use NR_CPUS rather than loading some other variable with the count opf onlines cpus. Moreover, the slowpath should never loop up to NR_CPUS since the SBI must passed a valid hartid index.

Quoted 1 line

t3 is clobbered by cpu_id below, is that correct?

In case of !SMP it is kept to zero. But I'll move that in a else below.

Quoted 8 lines

arch/riscv/kernel/sse_entry.S line 80

Quoted 5 lines
arch/riscv/kernel/sse_entry.S new file +4 -0
+
+#ifdef CONFIG_SMP
+    REG_L t4, SSE_REG_HART_ID(a7)
+    REG_L t3, SSE_REG_CPU_ID(a7)

hart_id and cpu_id are "unsigned int", not xlen-wide fields so you must use lw here, not REG_L (a lot of bugs lately on the usage of this macro).

Ouch, thanks for catching that.

Quoted 14 lines
Quoted 14 lines
arch/riscv/kernel/sse_entry.S new file +13 -0
+
+    bne t4, a6, .Lfind_hart_id_slowpath
+
+.Lcpu_id_found:
+#endif
+    asm_per_cpu_with_cpu t2 __sse_entry_task t1 t3
+    REG_L tp, 0(t2)
+
+    mv a1, sp /* pt_regs on stack */
+
+    /*
+     * Save sscratch for restoration since we might have interrupted the
+     * kernel in early exception path and thus, we don't know the
Quoted 8 lines
Quoted 6 lines

content of
+     * sscratch.
+     */
+    csrr s4, CSR_SSCRATCH
+    /* In-kernel scratch is 0 */
+    csrw CSR_SCRATCH, x0

csrrw s4, CSR_SSCRATCH, x0 instead?

Ah yeah.

Thanks !

Clément

Quoted 94 lines
Quoted 47 lines
arch/riscv/kernel/sse_entry.S new file +46 -0
+
+    mv a0, a7
+
+    call do_sse
+
+    csrw CSR_SSCRATCH, s4
+
+    REG_L a0, PT_STATUS(sp)
+    REG_L a1, PT_EPC(sp)
+    REG_L a2, PT_BADADDR(sp)
+    REG_L a3, PT_CAUSE(sp)
+    csrw CSR_SSTATUS, a0
+    csrw CSR_EPC, a1
+    csrw CSR_STVAL, a2
+    csrw CSR_SCAUSE, a3
+
+    REG_L ra, PT_RA(sp)
+    REG_L s0, PT_S0(sp)
+    REG_L s1, PT_S1(sp)
+    REG_L s2, PT_S2(sp)
+    REG_L s3, PT_S3(sp)
+    REG_L s4, PT_S4(sp)
+    REG_L s5, PT_S5(sp)
+    REG_L s6, PT_S6(sp)
+    REG_L s7, PT_S7(sp)
+    REG_L s8, PT_S8(sp)
+    REG_L s9, PT_S9(sp)
+    REG_L s10, PT_S10(sp)
+    REG_L s11, PT_S11(sp)
+    REG_L tp, PT_TP(sp)
+    REG_L t0, PT_T0(sp)
+    REG_L t1, PT_T1(sp)
+    REG_L t2, PT_T2(sp)
+    REG_L t3, PT_T3(sp)
+    REG_L t4, PT_T4(sp)
+    REG_L t5, PT_T5(sp)
+    REG_L t6, PT_T6(sp)
+    REG_L gp, PT_GP(sp)
+    REG_L a0, PT_A0(sp)
+    REG_L a1, PT_A1(sp)
+    REG_L a2, PT_A2(sp)
+    REG_L a3, PT_A3(sp)
+    REG_L a4, PT_A4(sp)
+    REG_L a5, PT_A5(sp)
+
+    REG_L sp, PT_SP(sp)

arch/riscv/kernel/sse_entry.S line 154

Quoted 10 lines
arch/riscv/kernel/sse_entry.S new file +10 -0
+
+    li a7, SBI_EXT_SSE
+    li a6, SBI_SSE_EVENT_COMPLETE
+    ecall
+
+#ifdef CONFIG_SMP
+.Lfind_hart_id_slowpath:
+
+/* Slowpath to find the CPU id associated to the hart id */
+la t0, __cpuid_to_hartid_map

^ Missing tab

Quoted 9 lines
arch/riscv/kernel/sse_entry.S new file +9 -0
+
+.Lhart_id_loop:
+    REG_L t2, 0(t0)
+    beq t2, a6, .Lcpu_id_found
+
+    /* Increment pointer and CPU number */
+    addi t3, t3, 1
+    addi t0, t0, RISCV_SZPTR
+    bltu t3, t1, .Lhart_id_loop

arch/riscv/kernel/sse_entry.S line 173

Quoted 10 lines
arch/riscv/kernel/sse_entry.S new file +10 -0
+
+    /*
+     * This should never happen since we expect the hart_id to match one
+     * of our CPU, but better be safe than sorry
+     */
+    la tp, init_task
+    la a0, sse_hart_id_panic_string
+    la t0, panic
+    jalr t0
+

This newline is not needed ^

arch/riscv/kernel/sse_entry.S line 180

Quoted 7 lines
arch/riscv/kernel/sse_entry.S new file +7 -0
+#endif
+
+SYM_CODE_END(handle_sse)
+
+SYM_DATA_START_LOCAL(sse_hart_id_panic_string)
+    .ascii "Unable to match hart_id with cpu\0"
+SYM_DATA_END(sse_hart_id_panic_string)

Thanks,

Alex

#8 lore raw

Re: [External] [PATCH v6 2/5] riscv: add support for SBI Supervisor Software Events extension

In reply to Clément Léger

Hi Clément,

Quoted 290 lines · Clément Léger

The SBI SSE extension allows the supervisor software to be notified by the SBI of specific events that are not maskable. The context switch is handled partially by the firmware which will save registers a6 and a7. When entering kernel we can rely on these 2 registers to setup the stack and save all the registers.

Since SSE events can be delivered at any time to the kernel (including during exception handling, we need a way to locate the current_task for context tracking. On RISC-V, it is sotred in scratch when in user space or tp when in kernel space (in which case SSCRATCH is zero). But at a at the beginning of exception handling, SSCRATCH is used to swap tp and check the origin of the exception. If interrupted at that point, then, there is no way to reliably know were is located the current task_struct. Even checking the interruption location won't work as SSE event can be nested on top of each other so the original interruption site might be lost at some point. In order to retrieve it reliably, store the current task in an additional __sse_entry_task per_cpu array. This array is then used to retrieve the current task based on the hart ID that is passed to the SSE event handler in a6.

That being said, the way the current task struct is stored should probably be reworked to find a better reliable alternative.

Since each events (and each CPU for local events) have their own context and can preempt each other, allocate a stack (and a shadow stack if needed for each of them (and for each cpu for local events).

When completing the event, if we were coming from kernel with interrupts disabled, simply return there. If coming from userspace or kernel with interrupts enabled, simulate an interrupt exception by setting IE_SIE in CSR_IP to allow delivery of signals to user task. For instance this can happen, when a RAS event has been generated by a user application and a SIGBUS has been sent to a task.

---
 arch/riscv/include/asm/asm.h         |  14 ++-
 arch/riscv/include/asm/scs.h         |   7 ++
 arch/riscv/include/asm/sse.h         |  47 +++++++
 arch/riscv/include/asm/switch_to.h   |  14 +++
 arch/riscv/include/asm/thread_info.h |   1 +
 arch/riscv/kernel/Makefile           |   1 +
 arch/riscv/kernel/asm-offsets.c      |  14 +++
 arch/riscv/kernel/sse.c              | 154 +++++++++++++++++++++++
 arch/riscv/kernel/sse_entry.S        | 180 +++++++++++++++++++++++++++
 9 files changed, 429 insertions(+), 3 deletions(-)
 create mode 100644 arch/riscv/include/asm/sse.h
 create mode 100644 arch/riscv/kernel/sse.c
 create mode 100644 arch/riscv/kernel/sse_entry.S
arch/riscv/include/asm/asm.h +11 -3
index a8a2af6dfe9d..982c4be9a9c3 100644
@@ -90,16 +90,24 @@
 #define PER_CPU_OFFSET_SHIFT 3
 #endif
 
-.macro asm_per_cpu dst sym tmp
-       REG_L \tmp, TASK_TI_CPU_NUM(tp)
-       slli  \tmp, \tmp, PER_CPU_OFFSET_SHIFT
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+       slli  \tmp, \cpu, PER_CPU_OFFSET_SHIFT
        la    \dst, __per_cpu_offset
        add   \dst, \dst, \tmp
        REG_L \tmp, 0(\dst)
        la    \dst, \sym
        add   \dst, \dst, \tmp
 .endm
+
+.macro asm_per_cpu dst sym tmp
+       REG_L \tmp, TASK_TI_CPU_NUM(tp)
+       asm_per_cpu_with_cpu \dst \sym \tmp \tmp
+.endm
 #else /* CONFIG_SMP */
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+       la    \dst, \sym
+.endm
+
 .macro asm_per_cpu dst sym tmp
        la    \dst, \sym
 .endm
arch/riscv/include/asm/scs.h +7 -0
index 0e45db78b24b..62344daad73d 100644
@@ -18,6 +18,11 @@
        load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp
 .endm
 
+/* Load the per-CPU IRQ shadow call stack to gp. */
+.macro scs_load_sse_stack reg_evt
+       REG_L gp, SSE_REG_EVT_SHADOW_STACK(\reg_evt)
+.endm
+
 /* Load task_scs_sp(current) to gp. */
 .macro scs_load_current
        REG_L   gp, TASK_TI_SCS_SP(tp)
@@ -41,6 +46,8 @@
 .endm
 .macro scs_load_irq_stack tmp
 .endm
+.macro scs_load_sse_stack reg_evt
+.endm
 .macro scs_load_current
 .endm
 .macro scs_load_current_if_task_changed prev
arch/riscv/include/asm/sse.h new file +47 -0
new file mode 100644
index 000000000000..8929a268462c
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#ifndef __ASM_SSE_H
+#define __ASM_SSE_H
+
+#include <asm/sbi.h>
+
+#ifdef CONFIG_RISCV_SSE
+
+struct sse_event_interrupted_state {
+       unsigned long a6;
+       unsigned long a7;
+};
+
+struct sse_event_arch_data {
+       void *stack;
+       void *shadow_stack;
+       unsigned long tmp;
+       struct sse_event_interrupted_state interrupted;
+       unsigned long interrupted_phys;
+       u32 evt_id;
+       unsigned int hart_id;
+       unsigned int cpu_id;
+};
+
+static inline bool sse_event_is_global(u32 evt)
+{
+       return !!(evt & SBI_SSE_EVENT_GLOBAL);
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt, int cpu);
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id,
+                       int cpu);
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt);
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt);
+
+void sse_handle_event(struct sse_event_arch_data *arch_evt,
+                     struct pt_regs *regs);
+asmlinkage void handle_sse(void);
+asmlinkage void do_sse(struct sse_event_arch_data *arch_evt,
+                      struct pt_regs *reg);
+
+#endif
+
+#endif
arch/riscv/include/asm/switch_to.h +14 -0
index 0e71eb82f920..cd1cead0c682 100644
@@ -88,6 +88,19 @@ static inline void __switch_to_envcfg(struct task_struct *next)
                        :: "r" (next->thread.envcfg) : "memory");
 }
 
+#ifdef CONFIG_RISCV_SSE
+DECLARE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+       __this_cpu_write(__sse_entry_task, next);
+}
+#else
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+}
+#endif
+
 extern struct task_struct *__switch_to(struct task_struct *,
                                       struct task_struct *);
 
@@ -122,6 +135,7 @@ do {                                                        \
        if (switch_to_should_flush_icache(__next))      \
                local_flush_icache_all();               \
        __switch_to_envcfg(__next);                     \
+       __switch_sse_entry_task(__next);                        \
        ((last) = __switch_to(__prev, __next));         \
 } while (0)
 
arch/riscv/include/asm/thread_info.h +1 -0
index f5916a70879a..28e9805e61fc 100644
@@ -36,6 +36,7 @@
 #define OVERFLOW_STACK_SIZE     SZ_4K
 
 #define IRQ_STACK_SIZE         THREAD_SIZE
+#define SSE_STACK_SIZE         THREAD_SIZE
 
 #ifndef __ASSEMBLY__
 
arch/riscv/kernel/Makefile +1 -0
index c7b542573407..62e4490b34ee 100644
@@ -99,6 +99,7 @@ obj-$(CONFIG_DYNAMIC_FTRACE)  += mcount-dyn.o
 obj-$(CONFIG_PERF_EVENTS)      += perf_callchain.o
 obj-$(CONFIG_HAVE_PERF_REGS)   += perf_regs.o
 obj-$(CONFIG_RISCV_SBI)                += sbi.o sbi_ecall.o
+obj-$(CONFIG_RISCV_SSE)                += sse.o sse_entry.o
 ifeq ($(CONFIG_RISCV_SBI), y)
 obj-$(CONFIG_SMP)              += sbi-ipi.o
 obj-$(CONFIG_SMP) += cpu_ops_sbi.o
arch/riscv/kernel/asm-offsets.c +14 -0
index 6e8c0d6feae9..315547c3a2ef 100644
@@ -14,6 +14,8 @@
 #include <asm/ptrace.h>
 #include <asm/cpu_ops_sbi.h>
 #include <asm/stacktrace.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
 #include <asm/suspend.h>
 
 void asm_offsets(void);
@@ -528,4 +530,16 @@ void asm_offsets(void)
        DEFINE(FREGS_A6,            offsetof(struct __arch_ftrace_regs, a6));
        DEFINE(FREGS_A7,            offsetof(struct __arch_ftrace_regs, a7));
 #endif
+
+#ifdef CONFIG_RISCV_SSE
+       OFFSET(SSE_REG_EVT_STACK, sse_event_arch_data, stack);
+       OFFSET(SSE_REG_EVT_SHADOW_STACK, sse_event_arch_data, shadow_stack);
+       OFFSET(SSE_REG_EVT_TMP, sse_event_arch_data, tmp);
+       OFFSET(SSE_REG_HART_ID, sse_event_arch_data, hart_id);
+       OFFSET(SSE_REG_CPU_ID, sse_event_arch_data, cpu_id);
+
+       DEFINE(SBI_EXT_SSE, SBI_EXT_SSE);
+       DEFINE(SBI_SSE_EVENT_COMPLETE, SBI_SSE_EVENT_COMPLETE);
+       DEFINE(ASM_NR_CPUS, NR_CPUS);
+#endif
 }
arch/riscv/kernel/sse.c new file +37 -0
new file mode 100644
index 000000000000..d2da7e23a74a
@@ -0,0 +1,154 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#include <linux/nmi.h>
+#include <linux/scs.h>
+#include <linux/bitfield.h>
+#include <linux/riscv_sse.h>
+#include <linux/percpu-defs.h>
+
+#include <asm/asm-prototypes.h>
+#include <asm/switch_to.h>
+#include <asm/irq_stack.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
+
+DEFINE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+void __weak sse_handle_event(struct sse_event_arch_data *arch_evt, struct pt_regs *regs)
+{
+}
+
+void do_sse(struct sse_event_arch_data *arch_evt, struct pt_regs *regs)
+{
+       nmi_enter();
+
+       /* Retrieve missing GPRs from SBI */
+       sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, arch_evt->evt_id,
+                 SBI_SSE_ATTR_INTERRUPTED_A6,
+                 (SBI_SSE_ATTR_INTERRUPTED_A7 - SBI_SSE_ATTR_INTERRUPTED_A6) + 1,
+                 arch_evt->interrupted_phys, 0, 0);
+
+       memcpy(&regs->a6, &arch_evt->interrupted, sizeof(arch_evt->interrupted));
+
+       sse_handle_event(arch_evt, regs);
+
+       /*

arch/riscv/kernel/sse.c line 47

Quoted 10 lines
arch/riscv/kernel/sse.c new file +10 -0
+        * The SSE delivery path does not uses the "standard" exception path
+        * (see sse_entry.S) and does not process any pending signal/softirqs
+        * due to being similar to a NMI.
+        * Some drivers (PMU, RAS) enqueue pending work that needs to be handled
+        * as soon as possible by bottom halves. For that purpose, set the SIP
+        * software interrupt pending bit which will force a software interrupt
+        * to be serviced once interrupts are reenabled in the interrupted
+        * context if they were masked or directly if unmasked.
+        */
+       csr_set(CSR_IP, IE_SIE);

When using perf record, will S mode interrupts experience starvation?

Quoted 97 lines
arch/riscv/kernel/sse.c new file +97 -0
+
+       nmi_exit();
+}
+
+static void *alloc_to_stack_pointer(void *alloc)
+{
+       return alloc ? alloc + SSE_STACK_SIZE : NULL;
+}
+
+static void *stack_pointer_to_alloc(void *stack)
+{
+       return stack - SSE_STACK_SIZE;
+}
+
+#ifdef CONFIG_VMAP_STACK
+static void *sse_stack_alloc(unsigned int cpu)
+{
+       void *stack = arch_alloc_vmap_stack(SSE_STACK_SIZE, cpu_to_node(cpu));
+
+       return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+       vfree(stack_pointer_to_alloc(stack));
+}
+#else /* CONFIG_VMAP_STACK */
+static void *sse_stack_alloc(unsigned int cpu)
+{
+       void *stack = kmalloc(SSE_STACK_SIZE, GFP_KERNEL);
+
+       return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+       kfree(stack_pointer_to_alloc(stack));
+}
+#endif /* CONFIG_VMAP_STACK */
+
+static int sse_init_scs(int cpu, struct sse_event_arch_data *arch_evt)
+{
+       void *stack;
+
+       if (!scs_is_enabled())
+               return 0;
+
+       stack = scs_alloc(cpu_to_node(cpu));
+       if (!stack)
+               return -ENOMEM;
+
+       arch_evt->shadow_stack = stack;
+
+       return 0;
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt, int cpu)
+{
+       arch_evt->cpu_id = cpu;
+       arch_evt->hart_id = cpuid_to_hartid_map(cpu);
+}
+
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id, int cpu)
+{
+       void *stack;
+
+       arch_evt->evt_id = evt_id;
+       stack = sse_stack_alloc(cpu);
+       if (!stack)
+               return -ENOMEM;
+
+       arch_evt->stack = stack;
+
+       if (sse_init_scs(cpu, arch_evt)) {
+               sse_stack_free(arch_evt->stack);
+               return -ENOMEM;
+       }
+
+       if (sse_event_is_global(evt_id)) {
+               arch_evt->interrupted_phys =
+                                       virt_to_phys(&arch_evt->interrupted);
+       } else {
+               arch_evt->interrupted_phys =
+                               per_cpu_ptr_to_phys(&arch_evt->interrupted);
+       }
+
+       arch_sse_event_update_cpu(arch_evt, cpu);
+
+       return 0;
+}
+
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt)
+{
+       scs_free(arch_evt->shadow_stack);
+       sse_stack_free(arch_evt->stack);
+}
+

arch/riscv/kernel/sse.c line 154

Quoted 10 lines
arch/riscv/kernel/sse.c new file +10 -0
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt)
+{
+       struct sbiret sret;
+
+       sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_REGISTER, arch_evt->evt_id,
+                        (unsigned long)handle_sse, (unsigned long)arch_evt, 0,
+                        0, 0);
+
+       return sbi_err_map_linux_errno(sret.error);
+}

...

Thanks,
Yunhui

#9 lore raw

Re: [External] [PATCH v6 2/5] riscv: add support for SBI Supervisor Software Events extension

Quoted 297 lines · yunhui cui

Hi Clément,

Quoted 294 lines · Clément Léger

The SBI SSE extension allows the supervisor software to be notified by the SBI of specific events that are not maskable. The context switch is handled partially by the firmware which will save registers a6 and a7. When entering kernel we can rely on these 2 registers to setup the stack and save all the registers.

Since SSE events can be delivered at any time to the kernel (including during exception handling, we need a way to locate the current_task for context tracking. On RISC-V, it is sotred in scratch when in user space or tp when in kernel space (in which case SSCRATCH is zero). But at a at the beginning of exception handling, SSCRATCH is used to swap tp and check the origin of the exception. If interrupted at that point, then, there is no way to reliably know were is located the current task_struct. Even checking the interruption location won't work as SSE event can be nested on top of each other so the original interruption site might be lost at some point. In order to retrieve it reliably, store the current task in an additional __sse_entry_task per_cpu array. This array is then used to retrieve the current task based on the hart ID that is passed to the SSE event handler in a6.

That being said, the way the current task struct is stored should probably be reworked to find a better reliable alternative.

Since each events (and each CPU for local events) have their own context and can preempt each other, allocate a stack (and a shadow stack if needed for each of them (and for each cpu for local events).

When completing the event, if we were coming from kernel with interrupts disabled, simply return there. If coming from userspace or kernel with interrupts enabled, simulate an interrupt exception by setting IE_SIE in CSR_IP to allow delivery of signals to user task. For instance this can happen, when a RAS event has been generated by a user application and a SIGBUS has been sent to a task.

---
 arch/riscv/include/asm/asm.h         |  14 ++-
 arch/riscv/include/asm/scs.h         |   7 ++
 arch/riscv/include/asm/sse.h         |  47 +++++++
 arch/riscv/include/asm/switch_to.h   |  14 +++
 arch/riscv/include/asm/thread_info.h |   1 +
 arch/riscv/kernel/Makefile           |   1 +
 arch/riscv/kernel/asm-offsets.c      |  14 +++
 arch/riscv/kernel/sse.c              | 154 +++++++++++++++++++++++
 arch/riscv/kernel/sse_entry.S        | 180 +++++++++++++++++++++++++++
 9 files changed, 429 insertions(+), 3 deletions(-)
 create mode 100644 arch/riscv/include/asm/sse.h
 create mode 100644 arch/riscv/kernel/sse.c
 create mode 100644 arch/riscv/kernel/sse_entry.S
arch/riscv/include/asm/asm.h +11 -3
index a8a2af6dfe9d..982c4be9a9c3 100644
@@ -90,16 +90,24 @@
 #define PER_CPU_OFFSET_SHIFT 3
 #endif
 
-.macro asm_per_cpu dst sym tmp
-       REG_L \tmp, TASK_TI_CPU_NUM(tp)
-       slli  \tmp, \tmp, PER_CPU_OFFSET_SHIFT
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+       slli  \tmp, \cpu, PER_CPU_OFFSET_SHIFT
        la    \dst, __per_cpu_offset
        add   \dst, \dst, \tmp
        REG_L \tmp, 0(\dst)
        la    \dst, \sym
        add   \dst, \dst, \tmp
 .endm
+
+.macro asm_per_cpu dst sym tmp
+       REG_L \tmp, TASK_TI_CPU_NUM(tp)
+       asm_per_cpu_with_cpu \dst \sym \tmp \tmp
+.endm
 #else /* CONFIG_SMP */
+.macro asm_per_cpu_with_cpu dst sym tmp cpu
+       la    \dst, \sym
+.endm
+
 .macro asm_per_cpu dst sym tmp
        la    \dst, \sym
 .endm
arch/riscv/include/asm/scs.h +7 -0
index 0e45db78b24b..62344daad73d 100644
@@ -18,6 +18,11 @@
        load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp
 .endm
 
+/* Load the per-CPU IRQ shadow call stack to gp. */
+.macro scs_load_sse_stack reg_evt
+       REG_L gp, SSE_REG_EVT_SHADOW_STACK(\reg_evt)
+.endm
+
 /* Load task_scs_sp(current) to gp. */
 .macro scs_load_current
        REG_L   gp, TASK_TI_SCS_SP(tp)
@@ -41,6 +46,8 @@
 .endm
 .macro scs_load_irq_stack tmp
 .endm
+.macro scs_load_sse_stack reg_evt
+.endm
 .macro scs_load_current
 .endm
 .macro scs_load_current_if_task_changed prev
arch/riscv/include/asm/sse.h new file +47 -0
new file mode 100644
index 000000000000..8929a268462c
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#ifndef __ASM_SSE_H
+#define __ASM_SSE_H
+
+#include <asm/sbi.h>
+
+#ifdef CONFIG_RISCV_SSE
+
+struct sse_event_interrupted_state {
+       unsigned long a6;
+       unsigned long a7;
+};
+
+struct sse_event_arch_data {
+       void *stack;
+       void *shadow_stack;
+       unsigned long tmp;
+       struct sse_event_interrupted_state interrupted;
+       unsigned long interrupted_phys;
+       u32 evt_id;
+       unsigned int hart_id;
+       unsigned int cpu_id;
+};
+
+static inline bool sse_event_is_global(u32 evt)
+{
+       return !!(evt & SBI_SSE_EVENT_GLOBAL);
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt, int cpu);
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id,
+                       int cpu);
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt);
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt);
+
+void sse_handle_event(struct sse_event_arch_data *arch_evt,
+                     struct pt_regs *regs);
+asmlinkage void handle_sse(void);
+asmlinkage void do_sse(struct sse_event_arch_data *arch_evt,
+                      struct pt_regs *reg);
+
+#endif
+
+#endif
arch/riscv/include/asm/switch_to.h +14 -0
index 0e71eb82f920..cd1cead0c682 100644
@@ -88,6 +88,19 @@ static inline void __switch_to_envcfg(struct task_struct *next)
                        :: "r" (next->thread.envcfg) : "memory");
 }
 
+#ifdef CONFIG_RISCV_SSE
+DECLARE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+       __this_cpu_write(__sse_entry_task, next);
+}
+#else
+static inline void __switch_sse_entry_task(struct task_struct *next)
+{
+}
+#endif
+
 extern struct task_struct *__switch_to(struct task_struct *,
                                       struct task_struct *);
 
@@ -122,6 +135,7 @@ do {                                                        \
        if (switch_to_should_flush_icache(__next))      \
                local_flush_icache_all();               \
        __switch_to_envcfg(__next);                     \
+       __switch_sse_entry_task(__next);                        \
        ((last) = __switch_to(__prev, __next));         \
 } while (0)
 
arch/riscv/include/asm/thread_info.h +1 -0
index f5916a70879a..28e9805e61fc 100644
@@ -36,6 +36,7 @@
 #define OVERFLOW_STACK_SIZE     SZ_4K
 
 #define IRQ_STACK_SIZE         THREAD_SIZE
+#define SSE_STACK_SIZE         THREAD_SIZE
 
 #ifndef __ASSEMBLY__
 
arch/riscv/kernel/Makefile +1 -0
index c7b542573407..62e4490b34ee 100644
@@ -99,6 +99,7 @@ obj-$(CONFIG_DYNAMIC_FTRACE)  += mcount-dyn.o
 obj-$(CONFIG_PERF_EVENTS)      += perf_callchain.o
 obj-$(CONFIG_HAVE_PERF_REGS)   += perf_regs.o
 obj-$(CONFIG_RISCV_SBI)                += sbi.o sbi_ecall.o
+obj-$(CONFIG_RISCV_SSE)                += sse.o sse_entry.o
 ifeq ($(CONFIG_RISCV_SBI), y)
 obj-$(CONFIG_SMP)              += sbi-ipi.o
 obj-$(CONFIG_SMP) += cpu_ops_sbi.o
arch/riscv/kernel/asm-offsets.c +14 -0
index 6e8c0d6feae9..315547c3a2ef 100644
@@ -14,6 +14,8 @@
 #include <asm/ptrace.h>
 #include <asm/cpu_ops_sbi.h>
 #include <asm/stacktrace.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
 #include <asm/suspend.h>
 
 void asm_offsets(void);
@@ -528,4 +530,16 @@ void asm_offsets(void)
        DEFINE(FREGS_A6,            offsetof(struct __arch_ftrace_regs, a6));
        DEFINE(FREGS_A7,            offsetof(struct __arch_ftrace_regs, a7));
 #endif
+
+#ifdef CONFIG_RISCV_SSE
+       OFFSET(SSE_REG_EVT_STACK, sse_event_arch_data, stack);
+       OFFSET(SSE_REG_EVT_SHADOW_STACK, sse_event_arch_data, shadow_stack);
+       OFFSET(SSE_REG_EVT_TMP, sse_event_arch_data, tmp);
+       OFFSET(SSE_REG_HART_ID, sse_event_arch_data, hart_id);
+       OFFSET(SSE_REG_CPU_ID, sse_event_arch_data, cpu_id);
+
+       DEFINE(SBI_EXT_SSE, SBI_EXT_SSE);
+       DEFINE(SBI_SSE_EVENT_COMPLETE, SBI_SSE_EVENT_COMPLETE);
+       DEFINE(ASM_NR_CPUS, NR_CPUS);
+#endif
 }
arch/riscv/kernel/sse.c new file +41 -0
new file mode 100644
index 000000000000..d2da7e23a74a
@@ -0,0 +1,154 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+#include <linux/nmi.h>
+#include <linux/scs.h>
+#include <linux/bitfield.h>
+#include <linux/riscv_sse.h>
+#include <linux/percpu-defs.h>
+
+#include <asm/asm-prototypes.h>
+#include <asm/switch_to.h>
+#include <asm/irq_stack.h>
+#include <asm/sbi.h>
+#include <asm/sse.h>
+
+DEFINE_PER_CPU(struct task_struct *, __sse_entry_task);
+
+void __weak sse_handle_event(struct sse_event_arch_data *arch_evt, struct pt_regs *regs)
+{
+}
+
+void do_sse(struct sse_event_arch_data *arch_evt, struct pt_regs *regs)
+{
+       nmi_enter();
+
+       /* Retrieve missing GPRs from SBI */
+       sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, arch_evt->evt_id,
+                 SBI_SSE_ATTR_INTERRUPTED_A6,
+                 (SBI_SSE_ATTR_INTERRUPTED_A7 - SBI_SSE_ATTR_INTERRUPTED_A6) + 1,
+                 arch_evt->interrupted_phys, 0, 0);
+
+       memcpy(&regs->a6, &arch_evt->interrupted, sizeof(arch_evt->interrupted));
+
+       sse_handle_event(arch_evt, regs);
+
+       /*
+        * The SSE delivery path does not uses the "standard" exception path
+        * (see sse_entry.S) and does not process any pending signal/softirqs
+        * due to being similar to a NMI.
+        * Some drivers (PMU, RAS) enqueue pending work that needs to be handled
Quoted 8 lines

arch/riscv/kernel/sse.c line 47

Quoted 6 lines
arch/riscv/kernel/sse.c new file +6 -0
+        * as soon as possible by bottom halves. For that purpose, set the SIP
+        * software interrupt pending bit which will force a software interrupt
+        * to be serviced once interrupts are reenabled in the interrupted
+        * context if they were masked or directly if unmasked.
+        */
+       csr_set(CSR_IP, IE_SIE);

When using perf record, will S mode interrupts experience starvation?

It shouldn't starve the other interrupts since after returning to S-mode (before servicing the interrupt), the hart will still be able to set other interrupts as pending which will be serviced as well I think. At least, I did not observed anything indicating that S-mode was not servicing interrupts anymore.

Clément

Quoted 112 lines
Quoted 98 lines
arch/riscv/kernel/sse.c new file +97 -0
+
+       nmi_exit();
+}
+
+static void *alloc_to_stack_pointer(void *alloc)
+{
+       return alloc ? alloc + SSE_STACK_SIZE : NULL;
+}
+
+static void *stack_pointer_to_alloc(void *stack)
+{
+       return stack - SSE_STACK_SIZE;
+}
+
+#ifdef CONFIG_VMAP_STACK
+static void *sse_stack_alloc(unsigned int cpu)
+{
+       void *stack = arch_alloc_vmap_stack(SSE_STACK_SIZE, cpu_to_node(cpu));
+
+       return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+       vfree(stack_pointer_to_alloc(stack));
+}
+#else /* CONFIG_VMAP_STACK */
+static void *sse_stack_alloc(unsigned int cpu)
+{
+       void *stack = kmalloc(SSE_STACK_SIZE, GFP_KERNEL);
+
+       return alloc_to_stack_pointer(stack);
+}
+
+static void sse_stack_free(void *stack)
+{
+       kfree(stack_pointer_to_alloc(stack));
+}
+#endif /* CONFIG_VMAP_STACK */
+
+static int sse_init_scs(int cpu, struct sse_event_arch_data *arch_evt)
+{
+       void *stack;
+
+       if (!scs_is_enabled())
+               return 0;
+
+       stack = scs_alloc(cpu_to_node(cpu));
+       if (!stack)
+               return -ENOMEM;
+
+       arch_evt->shadow_stack = stack;
+
+       return 0;
+}
+
+void arch_sse_event_update_cpu(struct sse_event_arch_data *arch_evt, int cpu)
+{
+       arch_evt->cpu_id = cpu;
+       arch_evt->hart_id = cpuid_to_hartid_map(cpu);
+}
+
+int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id, int cpu)
+{
+       void *stack;
+
+       arch_evt->evt_id = evt_id;
+       stack = sse_stack_alloc(cpu);
+       if (!stack)
+               return -ENOMEM;
+
+       arch_evt->stack = stack;
+
+       if (sse_init_scs(cpu, arch_evt)) {
+               sse_stack_free(arch_evt->stack);
+               return -ENOMEM;
+       }
+
+       if (sse_event_is_global(evt_id)) {
+               arch_evt->interrupted_phys =
+                                       virt_to_phys(&arch_evt->interrupted);
+       } else {
+               arch_evt->interrupted_phys =
+                               per_cpu_ptr_to_phys(&arch_evt->interrupted);
+       }
+
+       arch_sse_event_update_cpu(arch_evt, cpu);
+
+       return 0;
+}
+
+void arch_sse_free_event(struct sse_event_arch_data *arch_evt)
+{
+       scs_free(arch_evt->shadow_stack);
+       sse_stack_free(arch_evt->stack);
+}
+

arch/riscv/kernel/sse.c line 154

Quoted 10 lines
arch/riscv/kernel/sse.c new file +10 -0
+int arch_sse_register_event(struct sse_event_arch_data *arch_evt)
+{
+       struct sbiret sret;
+
+       sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_REGISTER, arch_evt->evt_id,
+                        (unsigned long)handle_sse, (unsigned long)arch_evt, 0,
+                        0, 0);
+
+       return sbi_err_map_linux_errno(sret.error);
+}

...

Thanks,
Yunhui

#10 lore raw

PATCH v6 3/5 drivers: firmware: add riscv SSE support

In reply to Clément Léger

Add driver level interface to use RISC-V SSE arch support. This interface allows registering SSE handlers, and receive them. This will be used by PMU and GHES driver.

---
 MAINTAINERS                        |  15 +
 drivers/firmware/Kconfig           |   1 +
 drivers/firmware/Makefile          |   1 +
 drivers/firmware/riscv/Kconfig     |  15 +
 drivers/firmware/riscv/Makefile    |   3 +
 drivers/firmware/riscv/riscv_sse.c | 694 +++++++++++++++++++++++++++++
 include/linux/riscv_sse.h          |  56 +++
 7 files changed, 785 insertions(+)
 create mode 100644 drivers/firmware/riscv/Kconfig
 create mode 100644 drivers/firmware/riscv/Makefile
 create mode 100644 drivers/firmware/riscv/riscv_sse.c
 create mode 100644 include/linux/riscv_sse.h
MAINTAINERS +15 -0
index bd62ad58a47f..114396d47f76 100644
@@ -21640,6 +21640,13 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux.git
 F:	Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
 F:	drivers/iommu/riscv/
 
+RISC-V FIRMWARE DRIVERS
+M:	Conor Dooley <[email protected]>
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/conor/linux.git
+F:	drivers/firmware/riscv/*
+
 RISC-V MICROCHIP FPGA SUPPORT
 M:	Conor Dooley <[email protected]>
 M:	Daire McNamara <[email protected]>
@@ -21704,6 +21711,14 @@ F:	arch/riscv/boot/dts/spacemit/
 N:	spacemit
 K:	spacemit
 
+RISC-V SSE DRIVER
+M:	Clément Léger <[email protected]>
+R:	Himanshu Chauhan <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/firmware/riscv/riscv_sse.c
+F:	include/linux/riscv_sse.h
+
 RISC-V THEAD SoC SUPPORT
 M:	Drew Fustini <[email protected]>
 M:	Guo Ren <[email protected]>
drivers/firmware/Kconfig +1 -0
index bbd2155d8483..1894df87b08e 100644
@@ -294,6 +294,7 @@ source "drivers/firmware/meson/Kconfig"
 source "drivers/firmware/microchip/Kconfig"
 source "drivers/firmware/psci/Kconfig"
 source "drivers/firmware/qcom/Kconfig"
+source "drivers/firmware/riscv/Kconfig"
 source "drivers/firmware/samsung/Kconfig"
 source "drivers/firmware/smccc/Kconfig"
 source "drivers/firmware/tegra/Kconfig"
drivers/firmware/Makefile +1 -0
index 4ddec2820c96..6cdd84570ea7 100644
@@ -34,6 +34,7 @@ obj-y				+= efi/
 obj-y				+= imx/
 obj-y				+= psci/
 obj-y				+= qcom/
+obj-y				+= riscv/
 obj-y				+= samsung/
 obj-y				+= smccc/
 obj-y				+= tegra/
drivers/firmware/riscv/Kconfig new file +15 -0
new file mode 100644
index 000000000000..8056ed3262d9
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menu "Risc-V Specific firmware drivers"
+depends on RISCV
+
+config RISCV_SSE
+	bool "Enable SBI Supervisor Software Events support"
+	depends on RISCV_SBI
+	default y
+	help
+	  The Supervisor Software Events support allow the SBI to deliver
+	  NMI-like notifications to the supervisor mode software. When enable,
+	  this option provides support to register callbacks on specific SSE
+	  events.
+
+endmenu
drivers/firmware/riscv/Makefile new file +3 -0
new file mode 100644
index 000000000000..4ccfcbbc28ea
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_RISCV_SSE)		+= riscv_sse.o
drivers/firmware/riscv/riscv_sse.c new file +694 -0

694 lines not on this page · raw message

include/linux/riscv_sse.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.43.0
#11 lore raw

Re: [External] [PATCH v6 3/5] drivers: firmware: add riscv SSE support

Hi Clément,

Quoted 616 lines · Clément Léger

Add driver level interface to use RISC-V SSE arch support. This interface allows registering SSE handlers, and receive them. This will be used by PMU and GHES driver.

---
 MAINTAINERS                        |  15 +
 drivers/firmware/Kconfig           |   1 +
 drivers/firmware/Makefile          |   1 +
 drivers/firmware/riscv/Kconfig     |  15 +
 drivers/firmware/riscv/Makefile    |   3 +
 drivers/firmware/riscv/riscv_sse.c | 694 +++++++++++++++++++++++++++++
 include/linux/riscv_sse.h          |  56 +++
 7 files changed, 785 insertions(+)
 create mode 100644 drivers/firmware/riscv/Kconfig
 create mode 100644 drivers/firmware/riscv/Makefile
 create mode 100644 drivers/firmware/riscv/riscv_sse.c
 create mode 100644 include/linux/riscv_sse.h
MAINTAINERS +15 -0
index bd62ad58a47f..114396d47f76 100644
@@ -21640,6 +21640,13 @@ T:     git git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux.git
 F:     Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
 F:     drivers/iommu/riscv/
 
+RISC-V FIRMWARE DRIVERS
+M:     Conor Dooley <[email protected]>
+L:     [email protected]
+S:     Maintained
+T:     git git://git.kernel.org/pub/scm/linux/kernel/git/conor/linux.git
+F:     drivers/firmware/riscv/*
+
 RISC-V MICROCHIP FPGA SUPPORT
 M:     Conor Dooley <[email protected]>
 M:     Daire McNamara <[email protected]>
@@ -21704,6 +21711,14 @@ F:     arch/riscv/boot/dts/spacemit/
 N:     spacemit
 K:     spacemit
 
+RISC-V SSE DRIVER
+M:     Clément Léger <[email protected]>
+R:     Himanshu Chauhan <[email protected]>
+L:     [email protected]
+S:     Maintained
+F:     drivers/firmware/riscv/riscv_sse.c
+F:     include/linux/riscv_sse.h
+
 RISC-V THEAD SoC SUPPORT
 M:     Drew Fustini <[email protected]>
 M:     Guo Ren <[email protected]>
drivers/firmware/Kconfig +1 -0
index bbd2155d8483..1894df87b08e 100644
@@ -294,6 +294,7 @@ source "drivers/firmware/meson/Kconfig"
 source "drivers/firmware/microchip/Kconfig"
 source "drivers/firmware/psci/Kconfig"
 source "drivers/firmware/qcom/Kconfig"
+source "drivers/firmware/riscv/Kconfig"
 source "drivers/firmware/samsung/Kconfig"
 source "drivers/firmware/smccc/Kconfig"
 source "drivers/firmware/tegra/Kconfig"
drivers/firmware/Makefile +1 -0
index 4ddec2820c96..6cdd84570ea7 100644
@@ -34,6 +34,7 @@ obj-y                         += efi/
 obj-y                          += imx/
 obj-y                          += psci/
 obj-y                          += qcom/
+obj-y                          += riscv/
 obj-y                          += samsung/
 obj-y                          += smccc/
 obj-y                          += tegra/
drivers/firmware/riscv/Kconfig new file +15 -0
new file mode 100644
index 000000000000..8056ed3262d9
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menu "Risc-V Specific firmware drivers"
+depends on RISCV
+
+config RISCV_SSE
+       bool "Enable SBI Supervisor Software Events support"
+       depends on RISCV_SBI
+       default y
+       help
+         The Supervisor Software Events support allow the SBI to deliver
+         NMI-like notifications to the supervisor mode software. When enable,
+         this option provides support to register callbacks on specific SSE
+         events.
+
+endmenu
drivers/firmware/riscv/Makefile new file +3 -0
new file mode 100644
index 000000000000..4ccfcbbc28ea
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_RISCV_SSE)                += riscv_sse.o
drivers/firmware/riscv/riscv_sse.c new file +500 -0
new file mode 100644
index 000000000000..edd107ade94a
@@ -0,0 +1,694 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+
+#define pr_fmt(fmt) "sse: " fmt
+
+#include <linux/cpu.h>
+#include <linux/cpuhotplug.h>
+#include <linux/cpu_pm.h>
+#include <linux/hardirq.h>
+#include <linux/list.h>
+#include <linux/percpu-defs.h>
+#include <linux/reboot.h>
+#include <linux/riscv_sse.h>
+#include <linux/slab.h>
+
+#include <asm/sbi.h>
+#include <asm/sse.h>
+
+struct sse_event {
+       struct list_head list;
+       u32 evt_id;
+       u32 priority;
+       sse_event_handler *handler;
+       void *handler_arg;
+       /* Only valid for global events */
+       unsigned int cpu;
+
+       union {
+               struct sse_registered_event *global;
+               struct sse_registered_event __percpu *local;
+       };
+};
+
+static int sse_hp_state;
+static bool sse_available __ro_after_init;
+static DEFINE_SPINLOCK(events_list_lock);
+static LIST_HEAD(events);
+static DEFINE_MUTEX(sse_mutex);
+
+struct sse_registered_event {
+       struct sse_event_arch_data arch;
+       struct sse_event *event;
+       unsigned long attr;
+       bool is_enabled;
+};
+
+void sse_handle_event(struct sse_event_arch_data *arch_event,
+                     struct pt_regs *regs)
+{
+       int ret;
+       struct sse_registered_event *reg_evt =
+               container_of(arch_event, struct sse_registered_event, arch);
+       struct sse_event *evt = reg_evt->event;
+
+       ret = evt->handler(evt->evt_id, evt->handler_arg, regs);
+       if (ret)
+               pr_warn("event %x handler failed with error %d\n", evt->evt_id, ret);
+}
+
+static struct sse_event *sse_event_get(u32 evt)
+{
+       struct sse_event *event = NULL;
+
+       scoped_guard(spinlock, &events_list_lock) {
+               list_for_each_entry(event, &events, list) {
+                       if (event->evt_id == evt)
+                               return event;
+               }
+       }
+
+       return NULL;
+}
+
+static phys_addr_t sse_event_get_attr_phys(struct sse_registered_event *reg_evt)
+{
+       phys_addr_t phys;
+       void *addr = &reg_evt->attr;
+
+       if (sse_event_is_global(reg_evt->event->evt_id))
+               phys = virt_to_phys(addr);
+       else
+               phys = per_cpu_ptr_to_phys(addr);
+
+       return phys;
+}
+
+static struct sse_registered_event *sse_get_reg_evt(struct sse_event *event)
+{
+       if (sse_event_is_global(event->evt_id))
+               return event->global;
+       else
+               return per_cpu_ptr(event->local, smp_processor_id());
+}
+
+static int sse_sbi_event_func(struct sse_event *event, unsigned long func)
+{
+       struct sbiret ret;
+       u32 evt = event->evt_id;
+       struct sse_registered_event *reg_evt = sse_get_reg_evt(event);
+
+       ret = sbi_ecall(SBI_EXT_SSE, func, evt, 0, 0, 0, 0, 0);
+       if (ret.error) {
+               pr_warn("Failed to execute func %lx, event %x, error %ld\n",
+                       func, evt, ret.error);
+               return sbi_err_map_linux_errno(ret.error);
+       }
+
+       if (func == SBI_SSE_EVENT_DISABLE)
+               reg_evt->is_enabled = false;
+       else if (func == SBI_SSE_EVENT_ENABLE)
+               reg_evt->is_enabled = true;
+
+       return 0;
+}
+
+int sse_event_disable_local(struct sse_event *event)
+{
+       return sse_sbi_event_func(event, SBI_SSE_EVENT_DISABLE);
+}
+EXPORT_SYMBOL_GPL(sse_event_disable_local);
+
+int sse_event_enable_local(struct sse_event *event)
+{
+       return sse_sbi_event_func(event, SBI_SSE_EVENT_ENABLE);
+}
+EXPORT_SYMBOL_GPL(sse_event_enable_local);
+
+static int sse_event_attr_get_no_lock(struct sse_registered_event *reg_evt,
+                                     unsigned long attr_id, unsigned long *val)
+{
+       struct sbiret sret;
+       u32 evt = reg_evt->event->evt_id;
+       unsigned long phys;
+
+       phys = sse_event_get_attr_phys(reg_evt);
+
+       sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, evt, attr_id, 1,
+                        phys, 0, 0);
+       if (sret.error) {
+               pr_debug("Failed to get event %x attr %lx, error %ld\n", evt,
+                        attr_id, sret.error);
+               return sbi_err_map_linux_errno(sret.error);
+       }
+
+       *val = reg_evt->attr;
+
+       return 0;
+}
+
+static int sse_event_attr_set_nolock(struct sse_registered_event *reg_evt,
+                                    unsigned long attr_id, unsigned long val)
+{
+       struct sbiret sret;
+       u32 evt = reg_evt->event->evt_id;
+       unsigned long phys;
+
+       reg_evt->attr = val;
+       phys = sse_event_get_attr_phys(reg_evt);
+
+       sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_WRITE, evt, attr_id, 1,
+                        phys, 0, 0);
+       if (sret.error)
+               pr_debug("Failed to set event %x attr %lx, error %ld\n", evt,
+                        attr_id, sret.error);
+
+       return sbi_err_map_linux_errno(sret.error);
+}
+
+static void sse_global_event_update_cpu(struct sse_event *event,
+                                       unsigned int cpu)
+{
+       struct sse_registered_event *reg_evt = event->global;
+
+       event->cpu = cpu;
+       arch_sse_event_update_cpu(&reg_evt->arch, cpu);
+}
+
+static int sse_event_set_target_cpu_nolock(struct sse_event *event,
+                                          unsigned int cpu)
+{
+       unsigned int hart_id = cpuid_to_hartid_map(cpu);
+       struct sse_registered_event *reg_evt = event->global;
+       u32 evt = event->evt_id;
+       bool was_enabled;
+       int ret;
+
+       if (!sse_event_is_global(evt))
+               return -EINVAL;
+
+       was_enabled = reg_evt->is_enabled;
+       if (was_enabled)
+               sse_event_disable_local(event);
+
+       ret = sse_event_attr_set_nolock(reg_evt, SBI_SSE_ATTR_PREFERRED_HART,
+                                       hart_id);
+       if (ret == 0)
+               sse_global_event_update_cpu(event, cpu);
+
+       if (was_enabled)
+               sse_event_enable_local(event);
+
+       return 0;
+}
+
+int sse_event_set_target_cpu(struct sse_event *event, unsigned int cpu)
+{
+       int ret;
+
+       scoped_guard(mutex, &sse_mutex) {
+               scoped_guard(cpus_read_lock) {
+                       if (!cpu_online(cpu))
+                               return -EINVAL;
+
+                       ret = sse_event_set_target_cpu_nolock(event, cpu);
+               }
+       }
+
+       return ret;
+}
+EXPORT_SYMBOL_GPL(sse_event_set_target_cpu);
+
+static int sse_event_init_registered(unsigned int cpu,
+                                    struct sse_registered_event *reg_evt,
+                                    struct sse_event *event)
+{
+       reg_evt->event = event;
+
+       return arch_sse_init_event(&reg_evt->arch, event->evt_id, cpu);
+}
+
+static void sse_event_free_registered(struct sse_registered_event *reg_evt)
+{
+       arch_sse_free_event(&reg_evt->arch);
+}
+
+static int sse_event_alloc_global(struct sse_event *event)
+{
+       int err;
+       struct sse_registered_event *reg_evt;
+
+       reg_evt = kzalloc(sizeof(*reg_evt), GFP_KERNEL);
+       if (!reg_evt)
+               return -ENOMEM;
+
+       event->global = reg_evt;
+       err = sse_event_init_registered(smp_processor_id(), reg_evt, event);
+       if (err)
+               kfree(reg_evt);
+
+       return err;
+}
+
+static int sse_event_alloc_local(struct sse_event *event)
+{
+       int err;
+       unsigned int cpu, err_cpu;
+       struct sse_registered_event *reg_evt;
+       struct sse_registered_event __percpu *reg_evts;
+
+       reg_evts = alloc_percpu(struct sse_registered_event);
+       if (!reg_evts)
+               return -ENOMEM;
+
+       event->local = reg_evts;
+
+       for_each_possible_cpu(cpu) {
+               reg_evt = per_cpu_ptr(reg_evts, cpu);
+               err = sse_event_init_registered(cpu, reg_evt, event);
+               if (err) {
+                       err_cpu = cpu;
+                       goto err_free_per_cpu;
+               }
+       }
+
+       return 0;
+
+err_free_per_cpu:
+       for_each_possible_cpu(cpu) {
+               if (cpu == err_cpu)
+                       break;
+               reg_evt = per_cpu_ptr(reg_evts, cpu);
+               sse_event_free_registered(reg_evt);
+       }
+
+       free_percpu(reg_evts);
+
+       return err;
+}
+
+static struct sse_event *sse_event_alloc(u32 evt, u32 priority,
+                                        sse_event_handler *handler, void *arg)
+{
+       int err;
+       struct sse_event *event;
+
+       event = kzalloc(sizeof(*event), GFP_KERNEL);
+       if (!event)
+               return ERR_PTR(-ENOMEM);
+
+       event->evt_id = evt;
+       event->priority = priority;
+       event->handler_arg = arg;
+       event->handler = handler;
+
+       if (sse_event_is_global(evt))
+               err = sse_event_alloc_global(event);
+       else
+               err = sse_event_alloc_local(event);
+
+       if (err) {
+               kfree(event);
+               return ERR_PTR(err);
+       }
+
+       return event;
+}
+
+static int sse_sbi_register_event(struct sse_event *event,
+                                 struct sse_registered_event *reg_evt)
+{
+       int ret;
+
+       ret = sse_event_attr_set_nolock(reg_evt, SBI_SSE_ATTR_PRIO,
+                                       event->priority);
+       if (ret)
+               return ret;
+
+       return arch_sse_register_event(&reg_evt->arch);
+}
+
+static int sse_event_register_local(struct sse_event *event)
+{
+       int ret;
+       struct sse_registered_event *reg_evt;
+
+       reg_evt = per_cpu_ptr(event->local, smp_processor_id());
+       ret = sse_sbi_register_event(event, reg_evt);
+       if (ret)
+               pr_debug("Failed to register event %x: err %d\n", event->evt_id,
+                        ret);
+
+       return ret;
+}
+
+static int sse_sbi_unregister_event(struct sse_event *event)
+{
+       return sse_sbi_event_func(event, SBI_SSE_EVENT_UNREGISTER);
+}
+
+struct sse_per_cpu_evt {
+       struct sse_event *event;
+       unsigned long func;
+       cpumask_t error;
+};
+
+static void sse_event_per_cpu_func(void *info)
+{
+       int ret;
+       struct sse_per_cpu_evt *cpu_evt = info;
+
+       if (cpu_evt->func == SBI_SSE_EVENT_REGISTER)
+               ret = sse_event_register_local(cpu_evt->event);
+       else
+               ret = sse_sbi_event_func(cpu_evt->event, cpu_evt->func);
+
+       if (ret)
+               cpumask_set_cpu(smp_processor_id(), &cpu_evt->error);
+}
+
+static void sse_event_free(struct sse_event *event)
+{
+       unsigned int cpu;
+       struct sse_registered_event *reg_evt;
+
+       if (sse_event_is_global(event->evt_id)) {
+               sse_event_free_registered(event->global);
+               kfree(event->global);
+       } else {
+               for_each_possible_cpu(cpu) {
+                       reg_evt = per_cpu_ptr(event->local, cpu);
+                       sse_event_free_registered(reg_evt);
+               }
+               free_percpu(event->local);
+       }
+
+       kfree(event);
+}
+
+static void sse_on_each_cpu(struct sse_event *event, unsigned long func,
+                           unsigned long revert_func)
+{
+       struct sse_per_cpu_evt cpu_evt;
+
+       cpu_evt.event = event;
+       cpumask_clear(&cpu_evt.error);
+       cpu_evt.func = func;
+       on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+       /*
+        * If there are some error reported by CPUs, revert event state on the
+        * other ones
+        */
+       if (!cpumask_empty(&cpu_evt.error)) {
+               cpumask_t revert;
+
+               cpumask_andnot(&revert, cpu_online_mask, &cpu_evt.error);
+               cpu_evt.func = revert_func;
+               on_each_cpu_mask(&revert, sse_event_per_cpu_func, &cpu_evt, 1);
+       }
+}
+
+int sse_event_enable(struct sse_event *event)
+{
+       int ret = 0;
+
+       scoped_guard(mutex, &sse_mutex) {
+               scoped_guard(cpus_read_lock) {
+                       if (sse_event_is_global(event->evt_id)) {
+                               ret = sse_event_enable_local(event);
+                       } else {
+                               sse_on_each_cpu(event, SBI_SSE_EVENT_ENABLE,
+                                               SBI_SSE_EVENT_DISABLE);
+                       }
+               }
+       }
+       return ret;
+}
+EXPORT_SYMBOL_GPL(sse_event_enable);
+
+static int sse_events_mask(void)
+{
+       struct sbiret ret;
+
+       ret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_HART_MASK, 0, 0, 0, 0, 0, 0);
+
+       return sbi_err_map_linux_errno(ret.error);
+}
+
+static int sse_events_unmask(void)
+{
+       struct sbiret ret;
+
+       ret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_HART_UNMASK, 0, 0, 0, 0, 0, 0);
+
+       return sbi_err_map_linux_errno(ret.error);
+}
+
+static void sse_event_disable_nolock(struct sse_event *event)
+{
+       struct sse_per_cpu_evt cpu_evt;
+
+       if (sse_event_is_global(event->evt_id)) {
+               sse_event_disable_local(event);
+       } else {
+               cpu_evt.event = event;
+               cpu_evt.func = SBI_SSE_EVENT_DISABLE;
+               on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+       }
+}
+
+void sse_event_disable(struct sse_event *event)
+{
+       scoped_guard(mutex, &sse_mutex) {
+               scoped_guard(cpus_read_lock) {
+                       sse_event_disable_nolock(event);
+               }
+       }
+}
+EXPORT_SYMBOL_GPL(sse_event_disable);
+
+struct sse_event *sse_event_register(u32 evt, u32 priority,
+                                    sse_event_handler *handler, void *arg)
+{
+       struct sse_event *event;
+       int cpu;
+       int ret = 0;
+
+       if (!sse_available)
+               return ERR_PTR(-EOPNOTSUPP);
+
+       guard(mutex)(&sse_mutex);
+       if (sse_event_get(evt))
+               return ERR_PTR(-EEXIST);
+
+       event = sse_event_alloc(evt, priority, handler, arg);
+       if (IS_ERR(event))
+               return event;
+
+       scoped_guard(cpus_read_lock) {
+               if (sse_event_is_global(evt)) {
+                       unsigned long preferred_hart;
+
+                       ret = sse_event_attr_get_no_lock(event->global,
+                                                       SBI_SSE_ATTR_PREFERRED_HART,
+                                                       &preferred_hart);
+                       if (ret)
+                               goto err_event_free;
+
+                       cpu = riscv_hartid_to_cpuid(preferred_hart);

drivers/firmware/riscv/riscv_sse.c line 510

Quoted 10 lines
drivers/firmware/riscv/riscv_sse.c new file +10 -0
+                       sse_global_event_update_cpu(event, cpu);
+
+                       ret = sse_sbi_register_event(event, event->global);
+                       if (ret)
+                               goto err_event_free;
+
+               } else {
+                       sse_on_each_cpu(event, SBI_SSE_EVENT_REGISTER,
+                                       SBI_SSE_EVENT_DISABLE);
+               }

It is necessary to check the return value; otherwise, the event might be passed as a valid value to subsequent logic, such as sse_event_enable().

Quoted 247 lines
drivers/firmware/riscv/riscv_sse.c new file +184 -0
+       }
+
+       scoped_guard(spinlock, &events_list_lock)
+               list_add(&event->list, &events);
+
+       return event;
+
+err_event_free:
+       sse_event_free(event);
+
+       return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(sse_event_register);
+
+static void sse_event_unregister_nolock(struct sse_event *event)
+{
+       struct sse_per_cpu_evt cpu_evt;
+
+       if (sse_event_is_global(event->evt_id)) {
+               sse_sbi_unregister_event(event);
+       } else {
+               cpu_evt.event = event;
+               cpu_evt.func = SBI_SSE_EVENT_UNREGISTER;
+               on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+       }
+}
+
+void sse_event_unregister(struct sse_event *event)
+{
+       scoped_guard(mutex, &sse_mutex) {
+               scoped_guard(cpus_read_lock)
+                       sse_event_unregister_nolock(event);
+
+               scoped_guard(spinlock, &events_list_lock)
+                       list_del(&event->list);
+
+               sse_event_free(event);
+       }
+}
+EXPORT_SYMBOL_GPL(sse_event_unregister);
+
+static int sse_cpu_online(unsigned int cpu)
+{
+       struct sse_event *event;
+
+       scoped_guard(spinlock, &events_list_lock) {
+               list_for_each_entry(event, &events, list) {
+                       if (sse_event_is_global(event->evt_id))
+                               continue;
+
+                       sse_event_register_local(event);
+                       if (sse_get_reg_evt(event))
+                               sse_event_enable_local(event);
+               }
+       }
+
+       /* Ready to handle events. Unmask SSE. */
+       return sse_events_unmask();
+}
+
+static int sse_cpu_teardown(unsigned int cpu)
+{
+       int ret = 0;
+       unsigned int next_cpu;
+       struct sse_event *event;
+
+       /* Mask the sse events */
+       ret = sse_events_mask();
+       if (ret)
+               return ret;
+
+       scoped_guard(spinlock, &events_list_lock) {
+               list_for_each_entry(event, &events, list) {
+                       if (!sse_event_is_global(event->evt_id)) {
+                               if (event->global->is_enabled)
+                                       sse_event_disable_local(event);
+
+                               sse_sbi_unregister_event(event);
+                               continue;
+                       }
+
+                       if (event->cpu != smp_processor_id())
+                               continue;
+
+                       /* Update destination hart for global event */
+                       next_cpu = cpumask_any_but(cpu_online_mask, cpu);
+                       ret = sse_event_set_target_cpu_nolock(event, next_cpu);
+               }
+       }
+
+       return ret;
+}
+
+static void sse_reset(void)
+{
+       struct sse_event *event;
+
+       list_for_each_entry(event, &events, list) {
+               sse_event_disable_nolock(event);
+               sse_event_unregister_nolock(event);
+       }
+}
+
+static int sse_pm_notifier(struct notifier_block *nb, unsigned long action,
+                          void *data)
+{
+       WARN_ON_ONCE(preemptible());
+
+       switch (action) {
+       case CPU_PM_ENTER:
+               sse_events_mask();
+               break;
+       case CPU_PM_EXIT:
+       case CPU_PM_ENTER_FAILED:
+               sse_events_unmask();
+               break;
+       default:
+               return NOTIFY_DONE;
+       }
+
+       return NOTIFY_OK;
+}
+
+static struct notifier_block sse_pm_nb = {
+       .notifier_call = sse_pm_notifier,
+};
+
+/*
+ * Mask all CPUs and unregister all events on panic, reboot or kexec.
+ */
+static int sse_reboot_notifier(struct notifier_block *nb, unsigned long action,
+                              void *data)
+{
+       cpuhp_remove_state(sse_hp_state);
+       sse_reset();
+
+       return NOTIFY_OK;
+}
+
+static struct notifier_block sse_reboot_nb = {
+       .notifier_call = sse_reboot_notifier,
+};
+
+static int __init sse_init(void)
+{
+       int ret;
+
+       if (sbi_probe_extension(SBI_EXT_SSE) <= 0) {
+               pr_err("Missing SBI SSE extension\n");
+               return -EOPNOTSUPP;
+       }
+       pr_info("SBI SSE extension detected\n");
+
+       ret = cpu_pm_register_notifier(&sse_pm_nb);
+       if (ret) {
+               pr_warn("Failed to register CPU PM notifier...\n");
+               return ret;
+       }
+
+       ret = register_reboot_notifier(&sse_reboot_nb);
+       if (ret) {
+               pr_warn("Failed to register reboot notifier...\n");
+               goto remove_cpupm;
+       }
+
+       ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "riscv/sse:online",
+                               sse_cpu_online, sse_cpu_teardown);
+       if (ret < 0)
+               goto remove_reboot;
+
+       sse_hp_state = ret;
+       sse_available = true;
+
+       return 0;
+
+remove_reboot:
+       unregister_reboot_notifier(&sse_reboot_nb);
+
+remove_cpupm:
+       cpu_pm_unregister_notifier(&sse_pm_nb);
+
+       return ret;
+}
+arch_initcall(sse_init);
include/linux/riscv_sse.h new file +56 -1
new file mode 100644
index 000000000000..d7bd0e22a00f
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+
+#ifndef __LINUX_RISCV_SSE_H
+#define __LINUX_RISCV_SSE_H
+
+#include <linux/types.h>
+#include <linux/linkage.h>
+
+struct sse_event;
+struct pt_regs;
+
+typedef int (sse_event_handler)(u32 event_num, void *arg, struct pt_regs *regs);
+
+#ifdef CONFIG_RISCV_SSE
+
+struct sse_event *sse_event_register(u32 event_num, u32 priority,
+                                    sse_event_handler *handler, void *arg);
+
+void sse_event_unregister(struct sse_event *evt);
+
+int sse_event_set_target_cpu(struct sse_event *sse_evt, unsigned int cpu);
+
+int sse_event_enable(struct sse_event *sse_evt);
+
+void sse_event_disable(struct sse_event *sse_evt);
+
+int sse_event_enable_local(struct sse_event *sse_evt);
+int sse_event_disable_local(struct sse_event *sse_evt);
+
+#else
+static inline struct sse_event *sse_event_register(u32 event_num, u32 priority,
+                                                  sse_event_handler *handler,
+                                                  void *arg)
+{
+       return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline void sse_event_unregister(struct sse_event *evt) {}
+
+static inline int sse_event_set_target_cpu(struct sse_event *sse_evt,
+                                          unsigned int cpu)
+{
+       return -EOPNOTSUPP;
+}
+
+static inline int sse_event_enable(struct sse_event *sse_evt)
+{
+       return -EOPNOTSUPP;
+}
+
+static inline void sse_event_disable(struct sse_event *sse_evt) {}
+#endif
+#endif /* __LINUX_RISCV_SSE_H */
--
Quoted 2 lines

2.43.0

Thanks,
Yunhui

#12 lore raw

Re: [External] [PATCH v6 3/5] drivers: firmware: add riscv SSE support

Quoted 10 lines · yunhui cui
Quoted 10 lines
                                                      &preferred_hart);
drivers/firmware/riscv/riscv_sse.c new file +9 -0
+                       if (ret)
+                               goto err_event_free;
+
+                       cpu = riscv_hartid_to_cpuid(preferred_hart);
+                       sse_global_event_update_cpu(event, cpu);
+
+                       ret = sse_sbi_register_event(event, event->global);
+                       if (ret)
+                               goto err_event_free;
Quoted 10 lines

drivers/firmware/riscv/riscv_sse.c line 510

Quoted 5 lines
drivers/firmware/riscv/riscv_sse.c new file +5 -0
+
+               } else {
+                       sse_on_each_cpu(event, SBI_SSE_EVENT_REGISTER,
+                                       SBI_SSE_EVENT_DISABLE);
+               }

It is necessary to check the return value; otherwise, the event might be passed as a valid value to subsequent logic, such as sse_event_enable().

Hi Yunhui,

Indeed, nice catch. I'll fix that.

Thanks,

Clément

#13 lore raw

Re: [PATCH v6 3/5] drivers: firmware: add riscv SSE support

In reply to Clément Léger
Quoted 6 lines · Clément Léger

Add driver level interface to use RISC-V SSE arch support. This interface allows registering SSE handlers, and receive them. This will be used by PMU and GHES driver.

Here you need to switch the Signed-off-by and the Co-developed-by

Quoted 79 lines
---
  MAINTAINERS                        |  15 +
  drivers/firmware/Kconfig           |   1 +
  drivers/firmware/Makefile          |   1 +
  drivers/firmware/riscv/Kconfig     |  15 +
  drivers/firmware/riscv/Makefile    |   3 +
  drivers/firmware/riscv/riscv_sse.c | 694 +++++++++++++++++++++++++++++
  include/linux/riscv_sse.h          |  56 +++
  7 files changed, 785 insertions(+)

create mode 100644 drivers/firmware/riscv/Kconfig create mode 100644 drivers/firmware/riscv/Makefile create mode 100644 drivers/firmware/riscv/riscv_sse.c create mode 100644 include/linux/riscv_sse.h

MAINTAINERS +15 -0
index bd62ad58a47f..114396d47f76 100644
@@ -21640,6 +21640,13 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux.git
  F:	Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
  F:	drivers/iommu/riscv/
 
+RISC-V FIRMWARE DRIVERS
+M:	Conor Dooley <[email protected]>
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/conor/linux.git
+F:	drivers/firmware/riscv/*
+
  RISC-V MICROCHIP FPGA SUPPORT
  M:	Conor Dooley <[email protected]>
  M:	Daire McNamara <[email protected]>
@@ -21704,6 +21711,14 @@ F:	arch/riscv/boot/dts/spacemit/
  N:	spacemit
  K:	spacemit
 
+RISC-V SSE DRIVER
+M:	Clément Léger <[email protected]>
+R:	Himanshu Chauhan <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/firmware/riscv/riscv_sse.c
+F:	include/linux/riscv_sse.h
+
  RISC-V THEAD SoC SUPPORT
  M:	Drew Fustini <[email protected]>
  M:	Guo Ren <[email protected]>
drivers/firmware/Kconfig +1 -0
index bbd2155d8483..1894df87b08e 100644
@@ -294,6 +294,7 @@ source "drivers/firmware/meson/Kconfig"
  source "drivers/firmware/microchip/Kconfig"
  source "drivers/firmware/psci/Kconfig"
  source "drivers/firmware/qcom/Kconfig"
+source "drivers/firmware/riscv/Kconfig"
  source "drivers/firmware/samsung/Kconfig"
  source "drivers/firmware/smccc/Kconfig"
  source "drivers/firmware/tegra/Kconfig"
drivers/firmware/Makefile +1 -0
index 4ddec2820c96..6cdd84570ea7 100644
@@ -34,6 +34,7 @@ obj-y				+= efi/
  obj-y				+= imx/
  obj-y				+= psci/
  obj-y				+= qcom/
+obj-y				+= riscv/
  obj-y				+= samsung/
  obj-y				+= smccc/
  obj-y				+= tegra/
drivers/firmware/riscv/Kconfig new file +0 -0
new file mode 100644
index 000000000000..8056ed3262d9

drivers/firmware/riscv/Kconfig line 10

Quoted 10 lines
drivers/firmware/riscv/Kconfig new file +10 -0
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menu "Risc-V Specific firmware drivers"
+depends on RISCV
+
+config RISCV_SSE
+	bool "Enable SBI Supervisor Software Events support"
+	depends on RISCV_SBI
+	default y
+	help
+	  The Supervisor Software Events support allow the SBI to deliver

s/allow/allows

drivers/firmware/riscv/Kconfig line 11

Quoted 1 line
drivers/firmware/riscv/Kconfig new file +1 -0
+	  NMI-like notifications to the supervisor mode software. When enable,

s/enable/enabled

drivers/firmware/riscv/Kconfig line 15

Quoted 4 lines
drivers/firmware/riscv/Kconfig new file +4 -0
+	  this option provides support to register callbacks on specific SSE
+	  events.
+
+endmenu

I think RISCV_SBI_SSE would be more accurate. Why do we need to gate SSE support behind this config though? I don't see similar configs for other SBI extensions.

Quoted 33 lines
drivers/firmware/riscv/Makefile new file +3 -0
new file mode 100644
index 000000000000..4ccfcbbc28ea
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_RISCV_SSE)		+= riscv_sse.o
drivers/firmware/riscv/riscv_sse.c new file +18 -0
new file mode 100644
index 000000000000..edd107ade94a
@@ -0,0 +1,694 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+
+#define pr_fmt(fmt) "sse: " fmt
+
+#include <linux/cpu.h>
+#include <linux/cpuhotplug.h>
+#include <linux/cpu_pm.h>
+#include <linux/hardirq.h>
+#include <linux/list.h>
+#include <linux/percpu-defs.h>
+#include <linux/reboot.h>
+#include <linux/riscv_sse.h>
+#include <linux/slab.h>
+
+#include <asm/sbi.h>

drivers/firmware/riscv/riscv_sse.c line 28

Quoted 10 lines
drivers/firmware/riscv/riscv_sse.c new file +10 -0
+#include <asm/sse.h>
+
+struct sse_event {
+	struct list_head list;
+	u32 evt_id;
+	u32 priority;
+	sse_event_handler *handler;
+	void *handler_arg;
+	/* Only valid for global events */
+	unsigned int cpu;

Nit: Replace unsigned int with u32 to match evt_id and priority definition? I have noticed a mix of "unsigned int" and u32 also below, maybe stick to one? Same for other types.

Quoted 26 lines
drivers/firmware/riscv/riscv_sse.c new file +26 -0
+
+	union {
+		struct sse_registered_event *global;
+		struct sse_registered_event __percpu *local;
+	};
+};
+
+static int sse_hp_state;
+static bool sse_available __ro_after_init;
+static DEFINE_SPINLOCK(events_list_lock);
+static LIST_HEAD(events);
+static DEFINE_MUTEX(sse_mutex);
+
+struct sse_registered_event {
+	struct sse_event_arch_data arch;
+	struct sse_event *event;
+	unsigned long attr;
+	bool is_enabled;
+};
+
+void sse_handle_event(struct sse_event_arch_data *arch_event,
+		      struct pt_regs *regs)
+{
+	int ret;
+	struct sse_registered_event *reg_evt =
+		container_of(arch_event, struct sse_registered_event, arch);

drivers/firmware/riscv/riscv_sse.c line 64

Quoted 10 lines
drivers/firmware/riscv/riscv_sse.c new file +10 -0
+	struct sse_event *evt = reg_evt->event;
+
+	ret = evt->handler(evt->evt_id, evt->handler_arg, regs);
+	if (ret)
+		pr_warn("event %x handler failed with error %d\n", evt->evt_id, ret);
+}
+
+static struct sse_event *sse_event_get(u32 evt)
+{
+	struct sse_event *event = NULL;

No need to initialize to NULL

Quoted 79 lines
drivers/firmware/riscv/riscv_sse.c new file +79 -0
+
+	scoped_guard(spinlock, &events_list_lock) {
+		list_for_each_entry(event, &events, list) {
+			if (event->evt_id == evt)
+				return event;
+		}
+	}
+
+	return NULL;
+}
+
+static phys_addr_t sse_event_get_attr_phys(struct sse_registered_event *reg_evt)
+{
+	phys_addr_t phys;
+	void *addr = &reg_evt->attr;
+
+	if (sse_event_is_global(reg_evt->event->evt_id))
+		phys = virt_to_phys(addr);
+	else
+		phys = per_cpu_ptr_to_phys(addr);
+
+	return phys;
+}
+
+static struct sse_registered_event *sse_get_reg_evt(struct sse_event *event)
+{
+	if (sse_event_is_global(event->evt_id))
+		return event->global;
+	else
+		return per_cpu_ptr(event->local, smp_processor_id());
+}
+
+static int sse_sbi_event_func(struct sse_event *event, unsigned long func)
+{
+	struct sbiret ret;
+	u32 evt = event->evt_id;
+	struct sse_registered_event *reg_evt = sse_get_reg_evt(event);
+
+	ret = sbi_ecall(SBI_EXT_SSE, func, evt, 0, 0, 0, 0, 0);
+	if (ret.error) {
+		pr_warn("Failed to execute func %lx, event %x, error %ld\n",
+			func, evt, ret.error);
+		return sbi_err_map_linux_errno(ret.error);
+	}
+
+	if (func == SBI_SSE_EVENT_DISABLE)
+		reg_evt->is_enabled = false;
+	else if (func == SBI_SSE_EVENT_ENABLE)
+		reg_evt->is_enabled = true;
+
+	return 0;
+}
+
+int sse_event_disable_local(struct sse_event *event)
+{
+	return sse_sbi_event_func(event, SBI_SSE_EVENT_DISABLE);
+}
+EXPORT_SYMBOL_GPL(sse_event_disable_local);
+
+int sse_event_enable_local(struct sse_event *event)
+{
+	return sse_sbi_event_func(event, SBI_SSE_EVENT_ENABLE);
+}
+EXPORT_SYMBOL_GPL(sse_event_enable_local);
+
+static int sse_event_attr_get_no_lock(struct sse_registered_event *reg_evt,
+				      unsigned long attr_id, unsigned long *val)
+{
+	struct sbiret sret;
+	u32 evt = reg_evt->event->evt_id;
+	unsigned long phys;
+
+	phys = sse_event_get_attr_phys(reg_evt);
+
+	sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, evt, attr_id, 1,
+			 phys, 0, 0);
+	if (sret.error) {
+		pr_debug("Failed to get event %x attr %lx, error %ld\n", evt,
+			 attr_id, sret.error);

drivers/firmware/riscv/riscv_sse.c line 153

Quoted 10 lines
drivers/firmware/riscv/riscv_sse.c new file +10 -0
+		return sbi_err_map_linux_errno(sret.error);
+	}
+
+	*val = reg_evt->attr;
+
+	return 0;
+}
+
+static int sse_event_attr_set_nolock(struct sse_registered_event *reg_evt,
+				     unsigned long attr_id, unsigned long val)

Nit but here you use "nolock" and for the getter above, you use "no_lock" (and while scrolling I noticed "nolock" again in sse_event_set_target_cpu_nolock()).

Quoted 552 lines
drivers/firmware/riscv/riscv_sse.c new file +541 -0
+{
+	struct sbiret sret;
+	u32 evt = reg_evt->event->evt_id;
+	unsigned long phys;
+
+	reg_evt->attr = val;
+	phys = sse_event_get_attr_phys(reg_evt);
+
+	sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_WRITE, evt, attr_id, 1,
+			 phys, 0, 0);
+	if (sret.error)
+		pr_debug("Failed to set event %x attr %lx, error %ld\n", evt,
+			 attr_id, sret.error);
+
+	return sbi_err_map_linux_errno(sret.error);
+}
+
+static void sse_global_event_update_cpu(struct sse_event *event,
+					unsigned int cpu)
+{
+	struct sse_registered_event *reg_evt = event->global;
+
+	event->cpu = cpu;
+	arch_sse_event_update_cpu(&reg_evt->arch, cpu);
+}
+
+static int sse_event_set_target_cpu_nolock(struct sse_event *event,
+					   unsigned int cpu)
+{
+	unsigned int hart_id = cpuid_to_hartid_map(cpu);
+	struct sse_registered_event *reg_evt = event->global;
+	u32 evt = event->evt_id;
+	bool was_enabled;
+	int ret;
+
+	if (!sse_event_is_global(evt))
+		return -EINVAL;
+
+	was_enabled = reg_evt->is_enabled;
+	if (was_enabled)
+		sse_event_disable_local(event);
+
+	ret = sse_event_attr_set_nolock(reg_evt, SBI_SSE_ATTR_PREFERRED_HART,
+					hart_id);
+	if (ret == 0)
+		sse_global_event_update_cpu(event, cpu);
+
+	if (was_enabled)
+		sse_event_enable_local(event);
+
+	return 0;
+}
+
+int sse_event_set_target_cpu(struct sse_event *event, unsigned int cpu)
+{
+	int ret;
+
+	scoped_guard(mutex, &sse_mutex) {
+		scoped_guard(cpus_read_lock) {
+			if (!cpu_online(cpu))
+				return -EINVAL;
+
+			ret = sse_event_set_target_cpu_nolock(event, cpu);
+		}
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(sse_event_set_target_cpu);
+
+static int sse_event_init_registered(unsigned int cpu,
+				     struct sse_registered_event *reg_evt,
+				     struct sse_event *event)
+{
+	reg_evt->event = event;
+
+	return arch_sse_init_event(&reg_evt->arch, event->evt_id, cpu);
+}
+
+static void sse_event_free_registered(struct sse_registered_event *reg_evt)
+{
+	arch_sse_free_event(&reg_evt->arch);
+}
+
+static int sse_event_alloc_global(struct sse_event *event)
+{
+	int err;
+	struct sse_registered_event *reg_evt;
+
+	reg_evt = kzalloc(sizeof(*reg_evt), GFP_KERNEL);
+	if (!reg_evt)
+		return -ENOMEM;
+
+	event->global = reg_evt;
+	err = sse_event_init_registered(smp_processor_id(), reg_evt, event);
+	if (err)
+		kfree(reg_evt);
+
+	return err;
+}
+
+static int sse_event_alloc_local(struct sse_event *event)
+{
+	int err;
+	unsigned int cpu, err_cpu;
+	struct sse_registered_event *reg_evt;
+	struct sse_registered_event __percpu *reg_evts;
+
+	reg_evts = alloc_percpu(struct sse_registered_event);
+	if (!reg_evts)
+		return -ENOMEM;
+
+	event->local = reg_evts;
+
+	for_each_possible_cpu(cpu) {
+		reg_evt = per_cpu_ptr(reg_evts, cpu);
+		err = sse_event_init_registered(cpu, reg_evt, event);
+		if (err) {
+			err_cpu = cpu;
+			goto err_free_per_cpu;
+		}
+	}
+
+	return 0;
+
+err_free_per_cpu:
+	for_each_possible_cpu(cpu) {
+		if (cpu == err_cpu)
+			break;
+		reg_evt = per_cpu_ptr(reg_evts, cpu);
+		sse_event_free_registered(reg_evt);
+	}
+
+	free_percpu(reg_evts);
+
+	return err;
+}
+
+static struct sse_event *sse_event_alloc(u32 evt, u32 priority,
+					 sse_event_handler *handler, void *arg)
+{
+	int err;
+	struct sse_event *event;
+
+	event = kzalloc(sizeof(*event), GFP_KERNEL);
+	if (!event)
+		return ERR_PTR(-ENOMEM);
+
+	event->evt_id = evt;
+	event->priority = priority;
+	event->handler_arg = arg;
+	event->handler = handler;
+
+	if (sse_event_is_global(evt))
+		err = sse_event_alloc_global(event);
+	else
+		err = sse_event_alloc_local(event);
+
+	if (err) {
+		kfree(event);
+		return ERR_PTR(err);
+	}
+
+	return event;
+}
+
+static int sse_sbi_register_event(struct sse_event *event,
+				  struct sse_registered_event *reg_evt)
+{
+	int ret;
+
+	ret = sse_event_attr_set_nolock(reg_evt, SBI_SSE_ATTR_PRIO,
+					event->priority);
+	if (ret)
+		return ret;
+
+	return arch_sse_register_event(&reg_evt->arch);
+}
+
+static int sse_event_register_local(struct sse_event *event)
+{
+	int ret;
+	struct sse_registered_event *reg_evt;
+
+	reg_evt = per_cpu_ptr(event->local, smp_processor_id());
+	ret = sse_sbi_register_event(event, reg_evt);
+	if (ret)
+		pr_debug("Failed to register event %x: err %d\n", event->evt_id,
+			 ret);
+
+	return ret;
+}
+
+static int sse_sbi_unregister_event(struct sse_event *event)
+{
+	return sse_sbi_event_func(event, SBI_SSE_EVENT_UNREGISTER);
+}
+
+struct sse_per_cpu_evt {
+	struct sse_event *event;
+	unsigned long func;
+	cpumask_t error;
+};
+
+static void sse_event_per_cpu_func(void *info)
+{
+	int ret;
+	struct sse_per_cpu_evt *cpu_evt = info;
+
+	if (cpu_evt->func == SBI_SSE_EVENT_REGISTER)
+		ret = sse_event_register_local(cpu_evt->event);
+	else
+		ret = sse_sbi_event_func(cpu_evt->event, cpu_evt->func);
+
+	if (ret)
+		cpumask_set_cpu(smp_processor_id(), &cpu_evt->error);
+}
+
+static void sse_event_free(struct sse_event *event)
+{
+	unsigned int cpu;
+	struct sse_registered_event *reg_evt;
+
+	if (sse_event_is_global(event->evt_id)) {
+		sse_event_free_registered(event->global);
+		kfree(event->global);
+	} else {
+		for_each_possible_cpu(cpu) {
+			reg_evt = per_cpu_ptr(event->local, cpu);
+			sse_event_free_registered(reg_evt);
+		}
+		free_percpu(event->local);
+	}
+
+	kfree(event);
+}
+
+static void sse_on_each_cpu(struct sse_event *event, unsigned long func,
+			    unsigned long revert_func)
+{
+	struct sse_per_cpu_evt cpu_evt;
+
+	cpu_evt.event = event;
+	cpumask_clear(&cpu_evt.error);
+	cpu_evt.func = func;
+	on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+	/*
+	 * If there are some error reported by CPUs, revert event state on the
+	 * other ones
+	 */
+	if (!cpumask_empty(&cpu_evt.error)) {
+		cpumask_t revert;
+
+		cpumask_andnot(&revert, cpu_online_mask, &cpu_evt.error);
+		cpu_evt.func = revert_func;
+		on_each_cpu_mask(&revert, sse_event_per_cpu_func, &cpu_evt, 1);
+	}
+}
+
+int sse_event_enable(struct sse_event *event)
+{
+	int ret = 0;
+
+	scoped_guard(mutex, &sse_mutex) {
+		scoped_guard(cpus_read_lock) {
+			if (sse_event_is_global(event->evt_id)) {
+				ret = sse_event_enable_local(event);
+			} else {
+				sse_on_each_cpu(event, SBI_SSE_EVENT_ENABLE,
+						SBI_SSE_EVENT_DISABLE);
+			}
+		}
+	}
+	return ret;
+}
+EXPORT_SYMBOL_GPL(sse_event_enable);
+
+static int sse_events_mask(void)
+{
+	struct sbiret ret;
+
+	ret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_HART_MASK, 0, 0, 0, 0, 0, 0);
+
+	return sbi_err_map_linux_errno(ret.error);
+}
+
+static int sse_events_unmask(void)
+{
+	struct sbiret ret;
+
+	ret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_HART_UNMASK, 0, 0, 0, 0, 0, 0);
+
+	return sbi_err_map_linux_errno(ret.error);
+}
+
+static void sse_event_disable_nolock(struct sse_event *event)
+{
+	struct sse_per_cpu_evt cpu_evt;
+
+	if (sse_event_is_global(event->evt_id)) {
+		sse_event_disable_local(event);
+	} else {
+		cpu_evt.event = event;
+		cpu_evt.func = SBI_SSE_EVENT_DISABLE;
+		on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+	}
+}
+
+void sse_event_disable(struct sse_event *event)
+{
+	scoped_guard(mutex, &sse_mutex) {
+		scoped_guard(cpus_read_lock) {
+			sse_event_disable_nolock(event);
+		}
+	}
+}
+EXPORT_SYMBOL_GPL(sse_event_disable);
+
+struct sse_event *sse_event_register(u32 evt, u32 priority,
+				     sse_event_handler *handler, void *arg)
+{
+	struct sse_event *event;
+	int cpu;
+	int ret = 0;
+
+	if (!sse_available)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	guard(mutex)(&sse_mutex);
+	if (sse_event_get(evt))
+		return ERR_PTR(-EEXIST);
+
+	event = sse_event_alloc(evt, priority, handler, arg);
+	if (IS_ERR(event))
+		return event;
+
+	scoped_guard(cpus_read_lock) {
+		if (sse_event_is_global(evt)) {
+			unsigned long preferred_hart;
+
+			ret = sse_event_attr_get_no_lock(event->global,
+							SBI_SSE_ATTR_PREFERRED_HART,
+							&preferred_hart);
+			if (ret)
+				goto err_event_free;
+
+			cpu = riscv_hartid_to_cpuid(preferred_hart);
+			sse_global_event_update_cpu(event, cpu);
+
+			ret = sse_sbi_register_event(event, event->global);
+			if (ret)
+				goto err_event_free;
+
+		} else {
+			sse_on_each_cpu(event, SBI_SSE_EVENT_REGISTER,
+					SBI_SSE_EVENT_DISABLE);
+		}
+	}
+
+	scoped_guard(spinlock, &events_list_lock)
+		list_add(&event->list, &events);
+
+	return event;
+
+err_event_free:
+	sse_event_free(event);
+
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(sse_event_register);
+
+static void sse_event_unregister_nolock(struct sse_event *event)
+{
+	struct sse_per_cpu_evt cpu_evt;
+
+	if (sse_event_is_global(event->evt_id)) {
+		sse_sbi_unregister_event(event);
+	} else {
+		cpu_evt.event = event;
+		cpu_evt.func = SBI_SSE_EVENT_UNREGISTER;
+		on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+	}
+}
+
+void sse_event_unregister(struct sse_event *event)
+{
+	scoped_guard(mutex, &sse_mutex) {
+		scoped_guard(cpus_read_lock)
+			sse_event_unregister_nolock(event);
+
+		scoped_guard(spinlock, &events_list_lock)
+			list_del(&event->list);
+
+		sse_event_free(event);
+	}
+}
+EXPORT_SYMBOL_GPL(sse_event_unregister);
+
+static int sse_cpu_online(unsigned int cpu)
+{
+	struct sse_event *event;
+
+	scoped_guard(spinlock, &events_list_lock) {
+		list_for_each_entry(event, &events, list) {
+			if (sse_event_is_global(event->evt_id))
+				continue;
+
+			sse_event_register_local(event);
+			if (sse_get_reg_evt(event))
+				sse_event_enable_local(event);
+		}
+	}
+
+	/* Ready to handle events. Unmask SSE. */
+	return sse_events_unmask();
+}
+
+static int sse_cpu_teardown(unsigned int cpu)
+{
+	int ret = 0;
+	unsigned int next_cpu;
+	struct sse_event *event;
+
+	/* Mask the sse events */
+	ret = sse_events_mask();
+	if (ret)
+		return ret;
+
+	scoped_guard(spinlock, &events_list_lock) {
+		list_for_each_entry(event, &events, list) {
+			if (!sse_event_is_global(event->evt_id)) {
+				if (event->global->is_enabled)
+					sse_event_disable_local(event);
+
+				sse_sbi_unregister_event(event);
+				continue;
+			}
+
+			if (event->cpu != smp_processor_id())
+				continue;
+
+			/* Update destination hart for global event */
+			next_cpu = cpumask_any_but(cpu_online_mask, cpu);
+			ret = sse_event_set_target_cpu_nolock(event, next_cpu);
+		}
+	}
+
+	return ret;
+}
+
+static void sse_reset(void)
+{
+	struct sse_event *event;
+
+	list_for_each_entry(event, &events, list) {
+		sse_event_disable_nolock(event);
+		sse_event_unregister_nolock(event);
+	}
+}
+
+static int sse_pm_notifier(struct notifier_block *nb, unsigned long action,
+			   void *data)
+{
+	WARN_ON_ONCE(preemptible());
+
+	switch (action) {
+	case CPU_PM_ENTER:
+		sse_events_mask();
+		break;
+	case CPU_PM_EXIT:
+	case CPU_PM_ENTER_FAILED:
+		sse_events_unmask();
+		break;
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block sse_pm_nb = {
+	.notifier_call = sse_pm_notifier,
+};
+
+/*
+ * Mask all CPUs and unregister all events on panic, reboot or kexec.
+ */
+static int sse_reboot_notifier(struct notifier_block *nb, unsigned long action,
+			       void *data)
+{
+	cpuhp_remove_state(sse_hp_state);
+	sse_reset();
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block sse_reboot_nb = {
+	.notifier_call = sse_reboot_notifier,
+};
+
+static int __init sse_init(void)
+{
+	int ret;
+
+	if (sbi_probe_extension(SBI_EXT_SSE) <= 0) {
+		pr_err("Missing SBI SSE extension\n");
+		return -EOPNOTSUPP;
+	}
+	pr_info("SBI SSE extension detected\n");
+
+	ret = cpu_pm_register_notifier(&sse_pm_nb);
+	if (ret) {
+		pr_warn("Failed to register CPU PM notifier...\n");
+		return ret;
+	}
+
+	ret = register_reboot_notifier(&sse_reboot_nb);
+	if (ret) {
+		pr_warn("Failed to register reboot notifier...\n");
+		goto remove_cpupm;
+	}
+
+	ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "riscv/sse:online",
+				sse_cpu_online, sse_cpu_teardown);
+	if (ret < 0)
+		goto remove_reboot;
+
+	sse_hp_state = ret;
+	sse_available = true;
+
+	return 0;
+
+remove_reboot:
+	unregister_reboot_notifier(&sse_reboot_nb);
+
+remove_cpupm:
+	cpu_pm_unregister_notifier(&sse_pm_nb);
+
+	return ret;
+}
+arch_initcall(sse_init);
include/linux/riscv_sse.h new file +5 -0
new file mode 100644
index 000000000000..d7bd0e22a00f
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+

include/linux/riscv_sse.h line 15

Quoted 10 lines
include/linux/riscv_sse.h new file +10 -0
+#ifndef __LINUX_RISCV_SSE_H
+#define __LINUX_RISCV_SSE_H
+
+#include <linux/types.h>
+#include <linux/linkage.h>
+
+struct sse_event;
+struct pt_regs;
+
+typedef int (sse_event_handler)(u32 event_num, void *arg, struct pt_regs *regs);

I don't see function pointer listed here https://www.kernel.org/doc/html/v4.10/process/coding-style.html#typedefs so I'd rather avoid this one.

Quoted 41 lines
include/linux/riscv_sse.h new file +41 -0
+
+#ifdef CONFIG_RISCV_SSE
+
+struct sse_event *sse_event_register(u32 event_num, u32 priority,
+				     sse_event_handler *handler, void *arg);
+
+void sse_event_unregister(struct sse_event *evt);
+
+int sse_event_set_target_cpu(struct sse_event *sse_evt, unsigned int cpu);
+
+int sse_event_enable(struct sse_event *sse_evt);
+
+void sse_event_disable(struct sse_event *sse_evt);
+
+int sse_event_enable_local(struct sse_event *sse_evt);
+int sse_event_disable_local(struct sse_event *sse_evt);
+
+#else
+static inline struct sse_event *sse_event_register(u32 event_num, u32 priority,
+						   sse_event_handler *handler,
+						   void *arg)
+{
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline void sse_event_unregister(struct sse_event *evt) {}
+
+static inline int sse_event_set_target_cpu(struct sse_event *sse_evt,
+					   unsigned int cpu)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int sse_event_enable(struct sse_event *sse_evt)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void sse_event_disable(struct sse_event *sse_evt) {}
+#endif
+#endif /* __LINUX_RISCV_SSE_H */
#14 lore raw

Re: [PATCH v6 3/5] drivers: firmware: add riscv SSE support

Hi Alex,

Quoted 11 lines · Alexandre Ghiti
Quoted 6 lines · Clément Léger

Add driver level interface to use RISC-V SSE arch support. This interface allows registering SSE handlers, and receive them. This will be used by PMU and GHES driver.

Here you need to switch the Signed-off-by and the Co-developed-by

Acked.

Quoted 103 lines
Quoted 81 lines

---
  MAINTAINERS                        |  15 + drivers/firmware/Kconfig           |   1 + drivers/firmware/Makefile          |   1 + drivers/firmware/riscv/Kconfig     |  15 + drivers/firmware/riscv/Makefile    |   3 + drivers/firmware/riscv/riscv_sse.c | 694 +++++++++++++++++++++++++++++ include/linux/riscv_sse.h          |  56 +++

  7 files changed, 785 insertions(+)

  create mode 100644 drivers/firmware/riscv/Kconfig create mode 100644 drivers/firmware/riscv/Makefile create mode 100644 drivers/firmware/riscv/riscv_sse.c create mode 100644 include/linux/riscv_sse.h

MAINTAINERS +0 -0
index bd62ad58a47f..114396d47f76 100644
@@ -21640,6 +21640,13 @@ T:    git git://git.kernel.org/pub/scm/linux/

kernel/git/iommu/linux.git
  F:    Documentation/devicetree/bindings/iommu/riscv,iommu.yaml F:    drivers/iommu/riscv/
  +RISC-V FIRMWARE DRIVERS

MAINTAINERS +6 -0
+M:    Conor Dooley <[email protected]>
+L:    [email protected]
+S:    Maintained
+T:    git git://git.kernel.org/pub/scm/linux/kernel/git/conor/linux.git
+F:    drivers/firmware/riscv/*
+

  RISC-V MICROCHIP FPGA SUPPORT
  M:    Conor Dooley <[email protected]> M:    Daire McNamara <[email protected]>

MAINTAINERS +0 -0
@@ -21704,6 +21711,14 @@ F:    arch/riscv/boot/dts/spacemit/

  N:    spacemit
  K:    spacemit
  +RISC-V SSE DRIVER

MAINTAINERS +7 -0
+M:    Clément Léger <[email protected]>
+R:    Himanshu Chauhan <[email protected]>
+L:    [email protected]
+S:    Maintained
+F:    drivers/firmware/riscv/riscv_sse.c
+F:    include/linux/riscv_sse.h
+

  RISC-V THEAD SoC SUPPORT
  M:    Drew Fustini <[email protected]>
  M:    Guo Ren <[email protected]>

drivers/firmware/Kconfig +0 -0
index bbd2155d8483..1894df87b08e 100644
@@ -294,6 +294,7 @@ source "drivers/firmware/meson/Kconfig"

  source "drivers/firmware/microchip/Kconfig"
  source "drivers/firmware/psci/Kconfig"
  source "drivers/firmware/qcom/Kconfig"
+source "drivers/firmware/riscv/Kconfig"
  source "drivers/firmware/samsung/Kconfig"
  source "drivers/firmware/smccc/Kconfig"
  source "drivers/firmware/tegra/Kconfig"

drivers/firmware/Makefile +0 -0
index 4ddec2820c96..6cdd84570ea7 100644
@@ -34,6 +34,7 @@ obj-y                += efi/

  obj-y                += imx/
  obj-y                += psci/
  obj-y                += qcom/
+obj-y                += riscv/
  obj-y                += samsung/
  obj-y                += smccc/
  obj-y                += tegra/

drivers/firmware/riscv/ renamed from drivers/firmware/riscv/Kconfig +0 -0

Kconfig
new file mode 100644
index 000000000000..8056ed3262d9

drivers/firmware/riscv/Kconfig line 10

Quoted 10 lines
drivers/firmware/riscv/Kconfig new file +10 -0
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menu "Risc-V Specific firmware drivers"
+depends on RISCV
+
+config RISCV_SSE
+    bool "Enable SBI Supervisor Software Events support"
+    depends on RISCV_SBI
+    default y
+    help
+      The Supervisor Software Events support allow the SBI to deliver

s/allow/allows

Quoted 2 lines
drivers/firmware/riscv/Kconfig new file +1 -0
+      NMI-like notifications to the supervisor mode software. When

enable,

s/enable/enabled

Quoted 9 lines

drivers/firmware/riscv/Kconfig

Quoted 4 lines

+      this option provides support to register callbacks on specific SSE +      events.

drivers/firmware/riscv/Kconfig new file +2 -0
+
+endmenu

I think RISCV_SBI_SSE would be more accurate. Why do we need to gate SSE support behind this config though? I don't see similar configs for other SBI extensions.

SSE is way more intrusive/larger than the other extensions in terms of code so it made sense to have a config option for it.

Quoted 44 lines
Quoted 44 lines
drivers/firmware/riscv/ renamed from drivers/firmware/riscv/Makefile +0 -0

Makefile
new file mode 100644
index 000000000000..4ccfcbbc28ea

drivers/firmware/riscv/Makefile new file +3 -0
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_RISCV_SSE)        += riscv_sse.o
drivers/firmware/ renamed from drivers/firmware/riscv/riscv_sse.c +0 -0

riscv/riscv_sse.c
new file mode 100644
index 000000000000..edd107ade94a

drivers/firmware/riscv/riscv_sse.c new file +25 -0
@@ -0,0 +1,694 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+
+#define pr_fmt(fmt) "sse: " fmt
+
+#include <linux/cpu.h>
+#include <linux/cpuhotplug.h>
+#include <linux/cpu_pm.h>
+#include <linux/hardirq.h>
+#include <linux/list.h>
+#include <linux/percpu-defs.h>
+#include <linux/reboot.h>
+#include <linux/riscv_sse.h>
+#include <linux/slab.h>
+
+#include <asm/sbi.h>
+#include <asm/sse.h>
+
+struct sse_event {
+    struct list_head list;
+    u32 evt_id;
+    u32 priority;
+    sse_event_handler *handler;
Quoted 8 lines

drivers/firmware/riscv/riscv_sse.c line 28

Quoted 3 lines
drivers/firmware/riscv/riscv_sse.c new file +3 -0
+    void *handler_arg;
+    /* Only valid for global events */
+    unsigned int cpu;

Nit: Replace unsigned int with u32 to match evt_id and priority definition? I have noticed a mix of "unsigned int" and u32 also below, maybe stick to one? Same for other types.

The CPU is always defined as unsigned int though this file which matches the cpu_online/cpu_offline definitions. I think keeping it like this is fine.

Quoted 136 lines
Quoted 34 lines
drivers/firmware/riscv/riscv_sse.c new file +31 -0
+
+    union {
+        struct sse_registered_event *global;
+        struct sse_registered_event __percpu *local;
+    };
+};
+
+static int sse_hp_state;
+static bool sse_available __ro_after_init;
+static DEFINE_SPINLOCK(events_list_lock);
+static LIST_HEAD(events);
+static DEFINE_MUTEX(sse_mutex);
+
+struct sse_registered_event {
+    struct sse_event_arch_data arch;
+    struct sse_event *event;
+    unsigned long attr;
+    bool is_enabled;
+};
+
+void sse_handle_event(struct sse_event_arch_data *arch_event,
+              struct pt_regs *regs)
+{
+    int ret;
+    struct sse_registered_event *reg_evt =
+        container_of(arch_event, struct sse_registered_event, arch);
+    struct sse_event *evt = reg_evt->event;
+
+    ret = evt->handler(evt->evt_id, evt->handler_arg, regs);
+    if (ret)
+        pr_warn("event %x handler failed with error %d\n", evt-
Quoted 1 line

evt_id, ret);

drivers/firmware/riscv/riscv_sse.c line 64

Quoted 5 lines
drivers/firmware/riscv/riscv_sse.c new file +5 -0
+}
+
+static struct sse_event *sse_event_get(u32 evt)
+{
+    struct sse_event *event = NULL;

No need to initialize to NULL

Quoted 92 lines
drivers/firmware/riscv/riscv_sse.c new file +12 -0
+
+    scoped_guard(spinlock, &events_list_lock) {
+        list_for_each_entry(event, &events, list) {
+            if (event->evt_id == evt)
+                return event;
+        }
+    }
+
+    return NULL;
+}
+
+static phys_addr_t sse_event_get_attr_phys(struct

sse_registered_event *reg_evt)

drivers/firmware/riscv/riscv_sse.c new file +13 -0
+{
+    phys_addr_t phys;
+    void *addr = &reg_evt->attr;
+
+    if (sse_event_is_global(reg_evt->event->evt_id))
+        phys = virt_to_phys(addr);
+    else
+        phys = per_cpu_ptr_to_phys(addr);
+
+    return phys;
+}
+
+static struct sse_registered_event *sse_get_reg_evt(struct sse_event

*event)

drivers/firmware/riscv/riscv_sse.c new file +8 -0
+{
+    if (sse_event_is_global(event->evt_id))
+        return event->global;
+    else
+        return per_cpu_ptr(event->local, smp_processor_id());
+}
+
+static int sse_sbi_event_func(struct sse_event *event, unsigned long

func)

drivers/firmware/riscv/riscv_sse.c new file +33 -0
+{
+    struct sbiret ret;
+    u32 evt = event->evt_id;
+    struct sse_registered_event *reg_evt = sse_get_reg_evt(event);
+
+    ret = sbi_ecall(SBI_EXT_SSE, func, evt, 0, 0, 0, 0, 0);
+    if (ret.error) {
+        pr_warn("Failed to execute func %lx, event %x, error %ld\n",
+            func, evt, ret.error);
+        return sbi_err_map_linux_errno(ret.error);
+    }
+
+    if (func == SBI_SSE_EVENT_DISABLE)
+        reg_evt->is_enabled = false;
+    else if (func == SBI_SSE_EVENT_ENABLE)
+        reg_evt->is_enabled = true;
+
+    return 0;
+}
+
+int sse_event_disable_local(struct sse_event *event)
+{
+    return sse_sbi_event_func(event, SBI_SSE_EVENT_DISABLE);
+}
+EXPORT_SYMBOL_GPL(sse_event_disable_local);
+
+int sse_event_enable_local(struct sse_event *event)
+{
+    return sse_sbi_event_func(event, SBI_SSE_EVENT_ENABLE);
+}
+EXPORT_SYMBOL_GPL(sse_event_enable_local);
+
+static int sse_event_attr_get_no_lock(struct sse_registered_event

*reg_evt,
+                      unsigned long attr_id, unsigned long *val)

drivers/firmware/riscv/riscv_sse.c new file +8 -0
+{
+    struct sbiret sret;
+    u32 evt = reg_evt->event->evt_id;
+    unsigned long phys;
+
+    phys = sse_event_get_attr_phys(reg_evt);
+
+    sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, evt,

attr_id, 1,
+             phys, 0, 0);
+    if (sret.error) {
+        pr_debug("Failed to get event %x attr %lx, error %ld\n", evt, +             attr_id, sret.error);
+        return sbi_err_map_linux_errno(sret.error);
+    }

drivers/firmware/riscv/riscv_sse.c new file +6 -0
+
+    *val = reg_evt->attr;
+
+    return 0;
+}
+
Quoted 8 lines
Quoted 3 lines

+static int sse_event_attr_set_nolock(struct sse_registered_event *reg_evt,
+                     unsigned long attr_id, unsigned long val)

Nit but here you use "nolock" and for the getter above, you use "no_lock" (and while scrolling I noticed "nolock" again in sse_event_set_target_cpu_nolock()).

Will use nolock like the other.

Quoted 569 lines
Quoted 569 lines
drivers/firmware/riscv/riscv_sse.c new file +9 -0
+{
+    struct sbiret sret;
+    u32 evt = reg_evt->event->evt_id;
+    unsigned long phys;
+
+    reg_evt->attr = val;
+    phys = sse_event_get_attr_phys(reg_evt);
+
+    sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_WRITE, evt,

attr_id, 1,
+             phys, 0, 0);
+    if (sret.error)
+        pr_debug("Failed to set event %x attr %lx, error %ld\n", evt, +             attr_id, sret.error);

drivers/firmware/riscv/riscv_sse.c new file +30 -0
+
+    return sbi_err_map_linux_errno(sret.error);
+}
+
+static void sse_global_event_update_cpu(struct sse_event *event,
+                    unsigned int cpu)
+{
+    struct sse_registered_event *reg_evt = event->global;
+
+    event->cpu = cpu;
+    arch_sse_event_update_cpu(&reg_evt->arch, cpu);
+}
+
+static int sse_event_set_target_cpu_nolock(struct sse_event *event,
+                       unsigned int cpu)
+{
+    unsigned int hart_id = cpuid_to_hartid_map(cpu);
+    struct sse_registered_event *reg_evt = event->global;
+    u32 evt = event->evt_id;
+    bool was_enabled;
+    int ret;
+
+    if (!sse_event_is_global(evt))
+        return -EINVAL;
+
+    was_enabled = reg_evt->is_enabled;
+    if (was_enabled)
+        sse_event_disable_local(event);
+
+    ret = sse_event_attr_set_nolock(reg_evt,

SBI_SSE_ATTR_PREFERRED_HART,
+                    hart_id);
+    if (ret == 0)
+        sse_global_event_update_cpu(event, cpu);

drivers/firmware/riscv/riscv_sse.c new file +34 -0
+
+    if (was_enabled)
+        sse_event_enable_local(event);
+
+    return 0;
+}
+
+int sse_event_set_target_cpu(struct sse_event *event, unsigned int cpu)
+{
+    int ret;
+
+    scoped_guard(mutex, &sse_mutex) {
+        scoped_guard(cpus_read_lock) {
+            if (!cpu_online(cpu))
+                return -EINVAL;
+
+            ret = sse_event_set_target_cpu_nolock(event, cpu);
+        }
+    }
+
+    return ret;
+}
+EXPORT_SYMBOL_GPL(sse_event_set_target_cpu);
+
+static int sse_event_init_registered(unsigned int cpu,
+                     struct sse_registered_event *reg_evt,
+                     struct sse_event *event)
+{
+    reg_evt->event = event;
+
+    return arch_sse_init_event(&reg_evt->arch, event->evt_id, cpu);
+}
+
+static void sse_event_free_registered(struct sse_registered_event

*reg_evt)

drivers/firmware/riscv/riscv_sse.c new file +168 -0
+{
+    arch_sse_free_event(&reg_evt->arch);
+}
+
+static int sse_event_alloc_global(struct sse_event *event)
+{
+    int err;
+    struct sse_registered_event *reg_evt;
+
+    reg_evt = kzalloc(sizeof(*reg_evt), GFP_KERNEL);
+    if (!reg_evt)
+        return -ENOMEM;
+
+    event->global = reg_evt;
+    err = sse_event_init_registered(smp_processor_id(), reg_evt, event);
+    if (err)
+        kfree(reg_evt);
+
+    return err;
+}
+
+static int sse_event_alloc_local(struct sse_event *event)
+{
+    int err;
+    unsigned int cpu, err_cpu;
+    struct sse_registered_event *reg_evt;
+    struct sse_registered_event __percpu *reg_evts;
+
+    reg_evts = alloc_percpu(struct sse_registered_event);
+    if (!reg_evts)
+        return -ENOMEM;
+
+    event->local = reg_evts;
+
+    for_each_possible_cpu(cpu) {
+        reg_evt = per_cpu_ptr(reg_evts, cpu);
+        err = sse_event_init_registered(cpu, reg_evt, event);
+        if (err) {
+            err_cpu = cpu;
+            goto err_free_per_cpu;
+        }
+    }
+
+    return 0;
+
+err_free_per_cpu:
+    for_each_possible_cpu(cpu) {
+        if (cpu == err_cpu)
+            break;
+        reg_evt = per_cpu_ptr(reg_evts, cpu);
+        sse_event_free_registered(reg_evt);
+    }
+
+    free_percpu(reg_evts);
+
+    return err;
+}
+
+static struct sse_event *sse_event_alloc(u32 evt, u32 priority,
+                     sse_event_handler *handler, void *arg)
+{
+    int err;
+    struct sse_event *event;
+
+    event = kzalloc(sizeof(*event), GFP_KERNEL);
+    if (!event)
+        return ERR_PTR(-ENOMEM);
+
+    event->evt_id = evt;
+    event->priority = priority;
+    event->handler_arg = arg;
+    event->handler = handler;
+
+    if (sse_event_is_global(evt))
+        err = sse_event_alloc_global(event);
+    else
+        err = sse_event_alloc_local(event);
+
+    if (err) {
+        kfree(event);
+        return ERR_PTR(err);
+    }
+
+    return event;
+}
+
+static int sse_sbi_register_event(struct sse_event *event,
+                  struct sse_registered_event *reg_evt)
+{
+    int ret;
+
+    ret = sse_event_attr_set_nolock(reg_evt, SBI_SSE_ATTR_PRIO,
+                    event->priority);
+    if (ret)
+        return ret;
+
+    return arch_sse_register_event(&reg_evt->arch);
+}
+
+static int sse_event_register_local(struct sse_event *event)
+{
+    int ret;
+    struct sse_registered_event *reg_evt;
+
+    reg_evt = per_cpu_ptr(event->local, smp_processor_id());
+    ret = sse_sbi_register_event(event, reg_evt);
+    if (ret)
+        pr_debug("Failed to register event %x: err %d\n", event->evt_id,
+             ret);
+
+    return ret;
+}
+
+static int sse_sbi_unregister_event(struct sse_event *event)
+{
+    return sse_sbi_event_func(event, SBI_SSE_EVENT_UNREGISTER);
+}
+
+struct sse_per_cpu_evt {
+    struct sse_event *event;
+    unsigned long func;
+    cpumask_t error;
+};
+
+static void sse_event_per_cpu_func(void *info)
+{
+    int ret;
+    struct sse_per_cpu_evt *cpu_evt = info;
+
+    if (cpu_evt->func == SBI_SSE_EVENT_REGISTER)
+        ret = sse_event_register_local(cpu_evt->event);
+    else
+        ret = sse_sbi_event_func(cpu_evt->event, cpu_evt->func);
+
+    if (ret)
+        cpumask_set_cpu(smp_processor_id(), &cpu_evt->error);
+}
+
+static void sse_event_free(struct sse_event *event)
+{
+    unsigned int cpu;
+    struct sse_registered_event *reg_evt;
+
+    if (sse_event_is_global(event->evt_id)) {
+        sse_event_free_registered(event->global);
+        kfree(event->global);
+    } else {
+        for_each_possible_cpu(cpu) {
+            reg_evt = per_cpu_ptr(event->local, cpu);
+            sse_event_free_registered(reg_evt);
+        }
+        free_percpu(event->local);
+    }
+
+    kfree(event);
+}
+
+static void sse_on_each_cpu(struct sse_event *event, unsigned long func,
+                unsigned long revert_func)
+{
+    struct sse_per_cpu_evt cpu_evt;
+
+    cpu_evt.event = event;
+    cpumask_clear(&cpu_evt.error);
+    cpu_evt.func = func;
+    on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+    /*
+     * If there are some error reported by CPUs, revert event state

on the
+     * other ones
+     */
+    if (!cpumask_empty(&cpu_evt.error)) {
+        cpumask_t revert;

drivers/firmware/riscv/riscv_sse.c new file +30 -0
+
+        cpumask_andnot(&revert, cpu_online_mask, &cpu_evt.error);
+        cpu_evt.func = revert_func;
+        on_each_cpu_mask(&revert, sse_event_per_cpu_func, &cpu_evt, 1);
+    }
+}
+
+int sse_event_enable(struct sse_event *event)
+{
+    int ret = 0;
+
+    scoped_guard(mutex, &sse_mutex) {
+        scoped_guard(cpus_read_lock) {
+            if (sse_event_is_global(event->evt_id)) {
+                ret = sse_event_enable_local(event);
+            } else {
+                sse_on_each_cpu(event, SBI_SSE_EVENT_ENABLE,
+                        SBI_SSE_EVENT_DISABLE);
+            }
+        }
+    }
+    return ret;
+}
+EXPORT_SYMBOL_GPL(sse_event_enable);
+
+static int sse_events_mask(void)
+{
+    struct sbiret ret;
+
+    ret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_HART_MASK, 0, 0, 0, 0,

0, 0);

drivers/firmware/riscv/riscv_sse.c new file +9 -0
+
+    return sbi_err_map_linux_errno(ret.error);
+}
+
+static int sse_events_unmask(void)
+{
+    struct sbiret ret;
+
+    ret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_HART_UNMASK, 0, 0, 0,

0, 0, 0);

drivers/firmware/riscv/riscv_sse.c new file +170 -0
+
+    return sbi_err_map_linux_errno(ret.error);
+}
+
+static void sse_event_disable_nolock(struct sse_event *event)
+{
+    struct sse_per_cpu_evt cpu_evt;
+
+    if (sse_event_is_global(event->evt_id)) {
+        sse_event_disable_local(event);
+    } else {
+        cpu_evt.event = event;
+        cpu_evt.func = SBI_SSE_EVENT_DISABLE;
+        on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+    }
+}
+
+void sse_event_disable(struct sse_event *event)
+{
+    scoped_guard(mutex, &sse_mutex) {
+        scoped_guard(cpus_read_lock) {
+            sse_event_disable_nolock(event);
+        }
+    }
+}
+EXPORT_SYMBOL_GPL(sse_event_disable);
+
+struct sse_event *sse_event_register(u32 evt, u32 priority,
+                     sse_event_handler *handler, void *arg)
+{
+    struct sse_event *event;
+    int cpu;
+    int ret = 0;
+
+    if (!sse_available)
+        return ERR_PTR(-EOPNOTSUPP);
+
+    guard(mutex)(&sse_mutex);
+    if (sse_event_get(evt))
+        return ERR_PTR(-EEXIST);
+
+    event = sse_event_alloc(evt, priority, handler, arg);
+    if (IS_ERR(event))
+        return event;
+
+    scoped_guard(cpus_read_lock) {
+        if (sse_event_is_global(evt)) {
+            unsigned long preferred_hart;
+
+            ret = sse_event_attr_get_no_lock(event->global,
+                            SBI_SSE_ATTR_PREFERRED_HART,
+                            &preferred_hart);
+            if (ret)
+                goto err_event_free;
+
+            cpu = riscv_hartid_to_cpuid(preferred_hart);
+            sse_global_event_update_cpu(event, cpu);
+
+            ret = sse_sbi_register_event(event, event->global);
+            if (ret)
+                goto err_event_free;
+
+        } else {
+            sse_on_each_cpu(event, SBI_SSE_EVENT_REGISTER,
+                    SBI_SSE_EVENT_DISABLE);
+        }
+    }
+
+    scoped_guard(spinlock, &events_list_lock)
+        list_add(&event->list, &events);
+
+    return event;
+
+err_event_free:
+    sse_event_free(event);
+
+    return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(sse_event_register);
+
+static void sse_event_unregister_nolock(struct sse_event *event)
+{
+    struct sse_per_cpu_evt cpu_evt;
+
+    if (sse_event_is_global(event->evt_id)) {
+        sse_sbi_unregister_event(event);
+    } else {
+        cpu_evt.event = event;
+        cpu_evt.func = SBI_SSE_EVENT_UNREGISTER;
+        on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1);
+    }
+}
+
+void sse_event_unregister(struct sse_event *event)
+{
+    scoped_guard(mutex, &sse_mutex) {
+        scoped_guard(cpus_read_lock)
+            sse_event_unregister_nolock(event);
+
+        scoped_guard(spinlock, &events_list_lock)
+            list_del(&event->list);
+
+        sse_event_free(event);
+    }
+}
+EXPORT_SYMBOL_GPL(sse_event_unregister);
+
+static int sse_cpu_online(unsigned int cpu)
+{
+    struct sse_event *event;
+
+    scoped_guard(spinlock, &events_list_lock) {
+        list_for_each_entry(event, &events, list) {
+            if (sse_event_is_global(event->evt_id))
+                continue;
+
+            sse_event_register_local(event);
+            if (sse_get_reg_evt(event))
+                sse_event_enable_local(event);
+        }
+    }
+
+    /* Ready to handle events. Unmask SSE. */
+    return sse_events_unmask();
+}
+
+static int sse_cpu_teardown(unsigned int cpu)
+{
+    int ret = 0;
+    unsigned int next_cpu;
+    struct sse_event *event;
+
+    /* Mask the sse events */
+    ret = sse_events_mask();
+    if (ret)
+        return ret;
+
+    scoped_guard(spinlock, &events_list_lock) {
+        list_for_each_entry(event, &events, list) {
+            if (!sse_event_is_global(event->evt_id)) {
+                if (event->global->is_enabled)
+                    sse_event_disable_local(event);
+
+                sse_sbi_unregister_event(event);
+                continue;
+            }
+
+            if (event->cpu != smp_processor_id())
+                continue;
+
+            /* Update destination hart for global event */
+            next_cpu = cpumask_any_but(cpu_online_mask, cpu);
+            ret = sse_event_set_target_cpu_nolock(event, next_cpu);
+        }
+    }
+
+    return ret;
+}
+
+static void sse_reset(void)
+{
+    struct sse_event *event;
+
+    list_for_each_entry(event, &events, list) {
+        sse_event_disable_nolock(event);
+        sse_event_unregister_nolock(event);
+    }
+}
+
+static int sse_pm_notifier(struct notifier_block *nb, unsigned long

action,
+               void *data)

drivers/firmware/riscv/riscv_sse.c new file +26 -0
+{
+    WARN_ON_ONCE(preemptible());
+
+    switch (action) {
+    case CPU_PM_ENTER:
+        sse_events_mask();
+        break;
+    case CPU_PM_EXIT:
+    case CPU_PM_ENTER_FAILED:
+        sse_events_unmask();
+        break;
+    default:
+        return NOTIFY_DONE;
+    }
+
+    return NOTIFY_OK;
+}
+
+static struct notifier_block sse_pm_nb = {
+    .notifier_call = sse_pm_notifier,
+};
+
+/*
+ * Mask all CPUs and unregister all events on panic, reboot or kexec.
+ */
+static int sse_reboot_notifier(struct notifier_block *nb, unsigned

long action,
+                   void *data)

drivers/firmware/riscv/riscv_sse.c new file +52 -0
+{
+    cpuhp_remove_state(sse_hp_state);
+    sse_reset();
+
+    return NOTIFY_OK;
+}
+
+static struct notifier_block sse_reboot_nb = {
+    .notifier_call = sse_reboot_notifier,
+};
+
+static int __init sse_init(void)
+{
+    int ret;
+
+    if (sbi_probe_extension(SBI_EXT_SSE) <= 0) {
+        pr_err("Missing SBI SSE extension\n");
+        return -EOPNOTSUPP;
+    }
+    pr_info("SBI SSE extension detected\n");
+
+    ret = cpu_pm_register_notifier(&sse_pm_nb);
+    if (ret) {
+        pr_warn("Failed to register CPU PM notifier...\n");
+        return ret;
+    }
+
+    ret = register_reboot_notifier(&sse_reboot_nb);
+    if (ret) {
+        pr_warn("Failed to register reboot notifier...\n");
+        goto remove_cpupm;
+    }
+
+    ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "riscv/sse:online",
+                sse_cpu_online, sse_cpu_teardown);
+    if (ret < 0)
+        goto remove_reboot;
+
+    sse_hp_state = ret;
+    sse_available = true;
+
+    return 0;
+
+remove_reboot:
+    unregister_reboot_notifier(&sse_reboot_nb);
+
+remove_cpupm:
+    cpu_pm_unregister_notifier(&sse_pm_nb);
+
+    return ret;
+}
+arch_initcall(sse_init);
include/linux/riscv_sse.h new file +12 -0
new file mode 100644
index 000000000000..d7bd0e22a00f
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2024 Rivos Inc.
+ */
+
+#ifndef __LINUX_RISCV_SSE_H
+#define __LINUX_RISCV_SSE_H
+
+#include <linux/types.h>
+#include <linux/linkage.h>
+
+struct sse_event;
Quoted 8 lines
Quoted 4 lines
include/linux/riscv_sse.h new file +3 -0
+struct pt_regs;
+
+typedef int (sse_event_handler)(u32 event_num, void *arg, struct

pt_regs *regs);

I don't see function pointer listed here https://www.kernel.org/doc/
html/v4.10/process/coding-style.html#typedefs so I'd rather avoid this one.

Although not mentionned here, there are function typedefs in the kernel:

git grep (typedef.*_fn)

This one is used at various places of the driver and is quite long so I would prefer to keep it. I'll rename it sse_event_handler_fn like a majority of the one in the kernel though.

Thanks,

Clément

Quoted 45 lines
Quoted 45 lines
include/linux/riscv_sse.h new file +9 -0
+
+#ifdef CONFIG_RISCV_SSE
+
+struct sse_event *sse_event_register(u32 event_num, u32 priority,
+                     sse_event_handler *handler, void *arg);
+
+void sse_event_unregister(struct sse_event *evt);
+
+int sse_event_set_target_cpu(struct sse_event *sse_evt, unsigned int

cpu);

include/linux/riscv_sse.h new file +10 -0
+
+int sse_event_enable(struct sse_event *sse_evt);
+
+void sse_event_disable(struct sse_event *sse_evt);
+
+int sse_event_enable_local(struct sse_event *sse_evt);
+int sse_event_disable_local(struct sse_event *sse_evt);
+
+#else
+static inline struct sse_event *sse_event_register(u32 event_num, u32

priority,
+                           sse_event_handler *handler, +                           void *arg)

include/linux/riscv_sse.h new file +20 -0
+{
+    return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline void sse_event_unregister(struct sse_event *evt) {}
+
+static inline int sse_event_set_target_cpu(struct sse_event *sse_evt,
+                       unsigned int cpu)
+{
+    return -EOPNOTSUPP;
+}
+
+static inline int sse_event_enable(struct sse_event *sse_evt)
+{
+    return -EOPNOTSUPP;
+}
+
+static inline void sse_event_disable(struct sse_event *sse_evt) {}
+#endif
+#endif /* __LINUX_RISCV_SSE_H */
#15 lore raw

PATCH v6 4/5 perf: RISC-V: add support for SSE event

In reply to Clément Léger

In order to use SSE within PMU drivers, register a SSE handler for the local PMU event. Reuse the existing overflow IRQ handler and pass appropriate pt_regs. Add a config option RISCV_PMU_SSE to select event delivery via SSE events.

---
 drivers/perf/Kconfig           | 10 +++++
 drivers/perf/riscv_pmu.c       | 19 +++++++++
 drivers/perf/riscv_pmu_sbi.c   | 71 +++++++++++++++++++++++++++++-----
 include/linux/perf/riscv_pmu.h |  3 ++
 4 files changed, 93 insertions(+), 10 deletions(-)
drivers/perf/Kconfig +10 -0
index a9188dec36fe..8cf0befaf2c0 100644
@@ -105,6 +105,16 @@ config RISCV_PMU_SBI
 	  full perf feature support i.e. counter overflow, privilege mode
 	  filtering, counter configuration.
 
+config RISCV_PMU_SSE
+	depends on RISCV_PMU && RISCV_SSE
+	bool "RISC-V PMU SSE events"
+	default n
+	help
+	  Say y if you want to use SSE events to deliver PMU interrupts. This
+	  provides a way to profile the kernel at any level by using NMI-like
+	  SSE events. SSE events being really intrusive, this option allows
+	  to select it only if needed.
+
 config STARFIVE_STARLINK_PMU
 	depends on ARCH_STARFIVE || COMPILE_TEST
 	depends on 64BIT
drivers/perf/riscv_pmu.c +19 -0
index 7644147d50b4..1eb28381b80f 100644
@@ -13,6 +13,7 @@
 #include <linux/irqdesc.h>
 #include <linux/perf/riscv_pmu.h>
 #include <linux/printk.h>
+#include <linux/riscv_sse.h>
 #include <linux/smp.h>
 #include <linux/sched_clock.h>
 
@@ -254,6 +255,22 @@ void riscv_pmu_start(struct perf_event *event, int flags)
 	perf_event_update_userpage(event);
 }
 
+static void riscv_pmu_disable(struct pmu *pmu)
+{
+	struct riscv_pmu *rvpmu = to_riscv_pmu(pmu);
+
+	if (rvpmu->sse_evt)
+		sse_event_disable_local(rvpmu->sse_evt);
+}
+
+static void riscv_pmu_enable(struct pmu *pmu)
+{
+	struct riscv_pmu *rvpmu = to_riscv_pmu(pmu);
+
+	if (rvpmu->sse_evt)
+		sse_event_enable_local(rvpmu->sse_evt);
+}
+
 static int riscv_pmu_add(struct perf_event *event, int flags)
 {
 	struct riscv_pmu *rvpmu = to_riscv_pmu(event->pmu);
@@ -411,6 +428,8 @@ struct riscv_pmu *riscv_pmu_alloc(void)
 		.event_mapped	= riscv_pmu_event_mapped,
 		.event_unmapped	= riscv_pmu_event_unmapped,
 		.event_idx	= riscv_pmu_event_idx,
+		.pmu_enable	= riscv_pmu_enable,
+		.pmu_disable	= riscv_pmu_disable,
 		.add		= riscv_pmu_add,
 		.del		= riscv_pmu_del,
 		.start		= riscv_pmu_start,
drivers/perf/riscv_pmu_sbi.c +61 -10

132 lines not on this page · raw message

include/linux/perf/riscv_pmu.h +3 -0
index 701974639ff2..d4a5c55fe077 100644
@@ -28,6 +28,8 @@
 
 #define RISCV_PMU_CONFIG1_GUEST_EVENTS 0x1
 
+struct sse_event;
+
 struct cpu_hw_events {
 	/* currently enabled events */
 	int			n_events;
@@ -54,6 +56,7 @@ struct riscv_pmu {
 	char		*name;
 
 	irqreturn_t	(*handle_irq)(int irq_num, void *dev);
+	struct sse_event *sse_evt;
 
 	unsigned long	cmask;
 	u64		(*ctr_read)(struct perf_event *event);
Signature · 1 line
2.43.0
#16 lore raw

PATCH v6 5/5 selftests/riscv: add SSE test module

In reply to Clément Léger

This module, once loaded, will execute a series of tests using the SSE framework. The provided script will check for any error reported by the test module.

---
 tools/testing/selftests/riscv/Makefile        |   2 +-
 tools/testing/selftests/riscv/sse/Makefile    |   5 +
 .../selftests/riscv/sse/module/Makefile       |  16 +
 .../riscv/sse/module/riscv_sse_test.c         | 513 ++++++++++++++++++
 .../selftests/riscv/sse/run_sse_test.sh       |  44 ++
 5 files changed, 579 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/riscv/sse/Makefile
 create mode 100644 tools/testing/selftests/riscv/sse/module/Makefile
 create mode 100644 tools/testing/selftests/riscv/sse/module/riscv_sse_test.c
 create mode 100644 tools/testing/selftests/riscv/sse/run_sse_test.sh
tools/testing/selftests/riscv/Makefile +1 -1
index 099b8c1f46f8..c62f58414b29 100644
@@ -5,7 +5,7 @@
 ARCH ?= $(shell uname -m 2>/dev/null || echo not)
 
 ifneq (,$(filter $(ARCH),riscv))
-RISCV_SUBTARGETS ?= abi hwprobe mm sigreturn vector
+RISCV_SUBTARGETS ?= abi hwprobe mm sigreturn vector sse
 else
 RISCV_SUBTARGETS :=
 endif
tools/testing/selftests/riscv/sse/Makefile new file +5 -0
new file mode 100644
index 000000000000..67eaee06f213
@@ -0,0 +1,5 @@
+TEST_GEN_MODS_DIR := module
+
+TEST_FILES := run_sse_test.sh
+
+include ../../lib.mk
tools/testing/selftests/riscv/sse/module/Makefile new file +16 -0
new file mode 100644
index 000000000000..02018f083456
@@ -0,0 +1,16 @@
+TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
+KDIR ?= /lib/modules/$(shell uname -r)/build
+
+obj-m += riscv_sse_test.o
+
+# Ensure that KDIR exists, otherwise skip the compilation
+modules:
+ifneq ("$(wildcard $(KDIR))", "")
+	$(Q)$(MAKE) -C $(KDIR) modules KBUILD_EXTMOD=$(TESTMODS_DIR)
+endif
+
+# Ensure that KDIR exists, otherwise skip the clean target
+clean:
+ifneq ("$(wildcard $(KDIR))", "")
+	$(Q)$(MAKE) -C $(KDIR) clean KBUILD_EXTMOD=$(TESTMODS_DIR)
+endif
tools/testing/selftests/riscv/sse/module/riscv_sse_test.c new file +513 -0

513 lines not on this page · raw message

tools/testing/selftests/riscv/sse/run_sse_test.sh new file +44 -0

44 lines not on this page · raw message

Signature · 1 line
2.43.0