Introduce Virtual IRQ (VIRQ) framework

From: Raymond Mao <[email protected]>

This series introduces a Virtual IRQ (VIRQ) framework for OpenSBI to support domain-aware wired interrupt couriering.

The main use case is to let OpenSBI receive a host HWIRQ in M-mode, map it into a per-channel VIRQ number space, route it to a target domain, notify the target S-mode payload, and let that payload pop/complete the pending VIRQ through an SBI ecall interface.

The implementation is organized as follows:

- add irqchip helpers for S-mode pending notification
- extend domain context switching to support VIRQ-driven cross-domain couriering and return-to-previous-domain flow
- add the core VIRQ mapping, routing, and per-(domain,hart) pending queue framework
- add a VIRQ vendor ecall extension for POP and COMPLETE
- parse sysirq routing rules from DT under /chosen/opensbi-domains
- derive APLIC target hart routing from sysirq nodes
- update the irqchip/APLIC path to support deferred completion and per-HWIRQ APLIC targets
- document the DT binding and routing rules

The DT routing model is based on "opensbi,mpxy-sysirq" nodes. Each interrupts-extended entry contributes one routed physical source, its entry index becomes the VIRQ number within the selected opensbi,mpxy-channel-id, and opensbi,domain selects the destination OpenSBI domain.

This series depends on a previous patch set:
[PATCH v2 1/3] lib: utils: irqchip: implement APLIC hwirq operation hooks
https://lore.kernel.org/opensbi/[email protected]/#t

Raymond Mao (10):
lib: irqchip: add S-mode notification helpers
lib: sbi: domain: adaptation for supporting VIRQ couriering domain
context switch
lib: sbi: Add Virtual IRQ (VIRQ) subsystem
lib: sbi: Add VIRQ ecall extension
lib: sbi: domain: add domain lookup by name
lib: utils: fdt: parse sysirq routing from DT
lib: utils: irqchip: derive APLIC targets from sysirq nodes
lib: irqchip: support deferred completion and per-HWIRQ APLIC targets
lib: sbi: domain: ensure boot_hartid is assigned
docs: domain: document sysirq VIRQ mapping and routing rules

 docs/domain_support.md                |   63 ++
 include/sbi/sbi_domain.h              |    7 +
 include/sbi/sbi_domain_context.h      |   24 +
 include/sbi/sbi_ecall_interface.h     |   26 +
 include/sbi/sbi_irqchip.h             |   24 +
 include/sbi/sbi_virq.h                |  492 +++++++++++
 include/sbi_utils/fdt/fdt_helper.h    |   17 +
 include/sbi_utils/irqchip/aplic.h     |    1 +
 lib/sbi/Kconfig                       |   10 +
 lib/sbi/objects.mk                    |    4 +
 lib/sbi/sbi_domain.c                  |   49 ++
 lib/sbi/sbi_domain_context.c          |  152 +++-
 lib/sbi/sbi_ecall_virq.c              |   56 ++
 lib/sbi/sbi_irqchip.c                 |   39 +-
 lib/sbi/sbi_trap.c                    |   16 +
 lib/sbi/sbi_virq.c                    | 1136 +++++++++++++++++++++++++
 lib/utils/fdt/fdt_domain.c            |  119 ++-
 lib/utils/fdt/fdt_helper.c            |   49 ++
 lib/utils/irqchip/aplic.c             |   60 +-
 lib/utils/irqchip/fdt_irqchip_aplic.c |  103 +++
 20 files changed, 2429 insertions(+), 18 deletions(-)
 create mode 100644 include/sbi/sbi_virq.h
 create mode 100644 lib/sbi/sbi_ecall_virq.c
 create mode 100644 lib/sbi/sbi_virq.c

2.25.1

-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Add irqchip helpers to set/clear S-mode notification (SEIP-based), In addition to set/clear, expose a get API to read the current notification state so upper layers can do edge-triggered notification.

---
 include/sbi/sbi_irqchip.h | 24 ++++++++++++++++++++++++
 lib/sbi/sbi_irqchip.c     | 18 +++++++++++++++++-
 2 files changed, 41 insertions(+), 1 deletion(-)

index 77b54110..7f23615a 100644
@@ -110,4 +110,28 @@ int sbi_irqchip_init(struct sbi_scratch *scratch, bool cold_boot);
 /** Exit interrupt controllers */
 void sbi_irqchip_exit(struct sbi_scratch *scratch);
 
+/**
+ * Notify S-mode for a pending virtual interrupt on this hart.
+ *
+ * The irqchip layer abstracts the notification mechanism; on platforms that
+ * use SEIP, this sets mip.SEIP.
+ */
+int sbi_irqchip_notify_smode_set(void);
+
+/**
+ * Clear S-mode notification for virtual interrupts on this hart.
+ *
+ * The irqchip layer abstracts the notification mechanism; on platforms that
+ * use SEIP, this clears mip.SEIP.
+ */
+void sbi_irqchip_notify_smode_clear(void);
+
+/**
+ * Read S-mode notification state for virtual interrupts on this hart.
+ *
+ * The irqchip layer abstracts the notification mechanism; on platforms that
+ * use SEIP, this reads mip.SEIP.
+ */
+bool sbi_irqchip_notify_smode_get(void);
+
 #endif

index f8599fa6..e022d534 100644
@@ -122,7 +122,7 @@ int sbi_irqchip_raw_handler_default(struct sbi_irqchip_device *chip, u32 hwirq)
 	sbi_printf("[IRQCHIP] Calling hwirq %u raw handler callback\n", hwirq);
 	rc = h->callback(hwirq, h->priv);
 
-	if (chip->hwirq_eoi) {
+	if (chip->hwirq_eoi && rc != SBI_EALREADY) {
 		sbi_printf("[IRQCHIP] Calling EOI of hwirq %u\n", hwirq);
 		chip->hwirq_eoi(chip, hwirq);
 	}

@@ -320,3 +320,19 @@ void sbi_irqchip_exit(struct sbi_scratch *scratch)
 	if (hd && hd->chip && hd->chip->process_hwirqs)
 		csr_clear(CSR_MIE, MIP_MEIP);
 }
+
+int sbi_irqchip_notify_smode_set(void)
+{
+	csr_set(CSR_MIP, MIP_SEIP);
+	return 0;
+}
+
+void sbi_irqchip_notify_smode_clear(void)
+{
+	csr_clear(CSR_MIP, MIP_SEIP);
+}
+
+bool sbi_irqchip_notify_smode_get(void)
+{
+	return !!(csr_read(CSR_MIP) & MIP_SEIP);
+}

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Prerequisite adaptations for introducing VIRQ couriering domain context switch.

MIDELEG:
- Save/restore MIDELEG in domain contexts and initialize per-domain defaults.

SEIP notification:
- Add virq_seip_notify flag, and use it to enable SEIP delegation for SEIP-notify domains.
- Add smode_notify_pending and helper function to store the S-mode notification status, and use it as a flag to fire the pending notification after domain context switch.

Return domain context switch:
- Add sbi_domain_context_exit_to_prev() to return to the previous domain without scanning for another candidate.
- Introduce per-hart deferred return flags and APIs to request/consume them.
- Perform the actual return-to-prev at the end of sbi_trap_handler() to avoid corrupting mepc during ecall handling.

Additionally, fix two return domain-switch protential issues:
- When a domain switch occurs inside sbi_trap_handler(), return the switched-to trap context from scratch instead of the original trap entry context. This prevents the trap restore path from resuming with stale state from the previous domain.
- When returning to an S-mode target domain, copy the restored CSR_SSTATUS SIE/SPIE/SPP bits into trap_ctx->regs.mstatus. The final trap exit writes CSR_MSTATUS from the trap context, so stale mstatus bits can otherwise clear S-mode interrupt enable and leave a pending SEIP undelivered.

---
 include/sbi/sbi_domain.h         |   2 +
 include/sbi/sbi_domain_context.h |  24 +++++
 lib/sbi/sbi_domain_context.c     | 152 ++++++++++++++++++++++++++++++-
 lib/sbi/sbi_trap.c               |  16 ++++
 4 files changed, 192 insertions(+), 2 deletions(-)

index 16edd4ce..c507023c 100644
@@ -217,6 +217,8 @@ struct sbi_domain {
 	bool system_suspend_allowed;
 	/** Identifies whether to include the firmware region */
 	bool fw_region_inited;
+	/** Whether to notify S-mode for VIRQ couriering */
+	bool virq_seip_notify;
 };
 
 /** The root domain instance */

index 31a3a7f8..88450fcb 100644
@@ -28,6 +28,30 @@ int sbi_domain_context_enter(struct sbi_domain *dom);
  */
 int sbi_domain_context_exit(void);
 
+/**
+ * Exit the current domain context and return to the previous context
+ * if one exists. This will not attempt to start other domains.
+ *
+ * @return 0 on success and negative error code on failure
+ */
+int sbi_domain_context_exit_to_prev(void);
+
+void sbi_domain_context_request_return_to_prev(void);
+bool sbi_domain_context_need_return_to_prev(void);
+void sbi_domain_context_mark_switched(void);
+bool sbi_domain_context_consume_switched(void);
+
+/**
+ * Mark a pending S-mode notification for a target domain context.
+ *
+ * @param dom pointer to domain
+ * @param hartindex hart index
+ *
+ * @return true if notification was already pending, false otherwise
+ */
+bool sbi_domain_context_pending_notify_smode(struct sbi_domain *dom,
+					     u32 hartindex);
+
 /**
  * Initialize domain context support
  *

index f41db4d1..79d9ca5a 100644
@@ -24,6 +24,7 @@
 #include <sbi/sbi_sse.h>
 #include <sbi/sbi_timer.h>
 #include <sbi/sbi_trap.h>
+#include <sbi/sbi_domain_context.h>
 
 static void sbi_trap_error_one(const struct sbi_trap_context *tcntx,
 			       const char *prefix, u32 hartid, u32 depth)

@@ -372,6 +373,21 @@ trap_done:
 	if (sbi_mstatus_prev_mode(regs->mstatus) != PRV_M)
 		sbi_sse_process_pending_events(regs);
 
+	if (sbi_domain_context_need_return_to_prev()) {
+		int rc = sbi_domain_context_exit_to_prev();
+
+		if (rc && rc != SBI_ENOENT)
+			sbi_printf("return_to_prev failed, rc=%d\n",
+				   rc);
+	}
+
+	if (sbi_domain_context_consume_switched()) {
+		struct sbi_trap_context *newctx = sbi_trap_get_context(scratch);
+
+		sbi_trap_set_context(scratch, newctx->prev_context);
+		return newctx;
+	}
+
 	sbi_trap_set_context(scratch, tcntx->prev_context);
 	return tcntx;
 }

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

VIRQ is an abstraction framework providing per-MPXY-channel HWIRQ<->VIRQ mapping, per-(domain,hart) VIRQ routing and couriering. It notifies S-mode payload via irqchip SEIP helper when a VIRQ is mapped/routed/enqueued and provides ecall extension for a S-mode payload to pop/complete a pending VIRQ.

VIRQ layer is composed of three major parts:
1. VIRQ mapping and allocation
- Provides a stable per-MPXY-channel mapping between a host physical interrupt endpoint (chip_uid, hwirq) and a VIRQ number.
- VIRQ number allocation uses a scalable bitmap.
2. HWIRQ->(Domain,hart) routing rules
- Routing rules are derived from sysirq nodes by interrupts-extended property, for example:
interrupts-extended =
<&aplic HWIRQx IRQ_TYPE>, // virq 0
<&aplic HWIRQy IRQ_TYPE>, // virq 1
...;
- VIRQ numbers are allocated from zero, implicit from the order of the entries in the interrupts-extended property.
- Each entry is cached as a routing rule.
- Default behavior: if an asserted HWIRQ does not match any routing rule, it will be routed to the root domain (channel 0) as a fallback.
3. Per-(domain,hart) pending queue couriering
- Each domain maintains a per-hart ring buffer queue of pending VIRQs. A courier handler enqueues VIRQs on HWIRQ assertion.
- The couriering is domain-aware. It switches to the target domain when it is not the same as the current one, and sends request for returning to the previous domain after SEIP completion.
- S-mode notification is edge-triggered based on the irqchip notify state, and is cleared only when the queue becomes empty.

---
 include/sbi/sbi_domain.h |    2 +
 include/sbi/sbi_virq.h   |  492 +++++++++++++++++
 lib/sbi/objects.mk       |    1 +
 lib/sbi/sbi_domain.c     |   10 +
 lib/sbi/sbi_virq.c       | 1136 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 1641 insertions(+)
 create mode 100644 include/sbi/sbi_virq.h
 create mode 100644 lib/sbi/sbi_virq.c

index c507023c..7e288cd8 100644
@@ -219,6 +219,8 @@ struct sbi_domain {
 	bool fw_region_inited;
 	/** Whether to notify S-mode for VIRQ couriering */
 	bool virq_seip_notify;
+	/** per-domain wired-IRQ courier state */
+	void *virq_priv;
 };
 
 /** The root domain instance */

index 07d13229..184bf173 100644
@@ -86,6 +86,7 @@ libsbi-objs-y += sbi_illegal_insn.o
 libsbi-objs-y += sbi_init.o
 libsbi-objs-y += sbi_ipi.o
 libsbi-objs-y += sbi_irqchip.o
+libsbi-objs-y += sbi_virq.o
 libsbi-objs-y += sbi_platform.o
 libsbi-objs-y += sbi_pmu.o
 libsbi-objs-y += sbi_dbtr.o

index 7030848d..2a846eea 100644
@@ -18,6 +18,7 @@
 #include <sbi/sbi_platform.h>
 #include <sbi/sbi_scratch.h>
 #include <sbi/sbi_string.h>
+#include <sbi/sbi_virq.h>
 
 SBI_LIST_HEAD(domain_list);
 

@@ -693,6 +694,15 @@ int sbi_domain_register(struct sbi_domain *dom,
 		return rc;
 	}
 
+	/* Init per-domain wired-IRQ courier state */
+	rc = sbi_virq_domain_init(dom);
+	if (rc) {
+		sbi_printf("%s: virq init failed for %s (error %d)\n",
+				__func__, dom->name, rc);
+		sbi_list_del(&dom->node);
+		return rc;
+	}
+
 	return 0;
 }
 

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Add vendor SBI extension ecall for VIRQ.
This allows S-mode payload to pop/complete the next pending VIRQ has couried into the current domain.

---
 include/sbi/sbi_ecall_interface.h | 26 ++++++++++++++
 lib/sbi/Kconfig                   | 10 ++++++
 lib/sbi/objects.mk                |  3 ++
 lib/sbi/sbi_ecall_virq.c          | 56 +++++++++++++++++++++++++++++++
 4 files changed, 95 insertions(+)
 create mode 100644 lib/sbi/sbi_ecall_virq.c

index 9a776f79..37937a0c 100644
@@ -126,6 +126,32 @@
 #define SBI_EXT_FWFT_SET		0x0
 #define SBI_EXT_FWFT_GET		0x1
 
+#ifdef CONFIG_SBI_ECALL_VIRQ
+
+/* Vendor extension base range is defined by the SBI spec. Choose a private ID. */
+#define SBI_EXT_VIRQ			0x0900524d
+
+/* Function IDs for SBI_EXT_VIRQ */
+#define SBI_EXT_VIRQ_POP		0
+#define SBI_EXT_VIRQ_COMPLETE		1
+
+/*
+ * SBI_EXT_VIRQ_POP
+ * Returns:
+ *   a0: SBI error code (0 for success)
+ *   a1: next pending VIRQ (VIRQ_INVALID if none pending)
+ */
+
+/*
+ * SBI_EXT_VIRQ_COMPLETE
+ * Input:
+ *   a0: VIRQ to complete
+ * Returns:
+ *   a0: SBI error code (0 for success)
+ */
+
+#endif
+
 enum sbi_fwft_feature_t {
 	SBI_FWFT_MISALIGNED_EXC_DELEG		= 0x0,
 	SBI_FWFT_LANDING_PAD			= 0x1,

index c6cc04bc..cbb74640 100644
@@ -69,4 +69,14 @@ config SBI_ECALL_SSE
 config SBI_ECALL_MPXY
 	bool "MPXY extension"
 	default y
+
+config SBI_ECALL_VIRQ
+	bool "VIRQ extension"
+	default y
+	help
+	  Enable the OpenSBI VIRQ ecall extension.
+	  This extension allows an S-mode payload to pop a pending
+	  virtual interrupt and complete its deferred host interrupt
+	  handling after the payload has consumed the event.
+
 endmenu

index 184bf173..ea816e92 100644
@@ -64,6 +64,9 @@ libsbi-objs-$(CONFIG_SBI_ECALL_SSE) += sbi_ecall_sse.o
 carray-sbi_ecall_exts-$(CONFIG_SBI_ECALL_MPXY) += ecall_mpxy
 libsbi-objs-$(CONFIG_SBI_ECALL_MPXY) += sbi_ecall_mpxy.o
 
+carray-sbi_ecall_exts-$(CONFIG_SBI_ECALL_VIRQ) += ecall_virq
+libsbi-objs-$(CONFIG_SBI_ECALL_VIRQ) += sbi_ecall_virq.o
+
 libsbi-objs-y += sbi_bitmap.o
 libsbi-objs-y += sbi_bitops.o
 libsbi-objs-y += sbi_console.o

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Provide a helper to resolve a domain by its DT node name, used by sysirq DT parsing.

---
 include/sbi/sbi_domain.h |  3 +++
 lib/sbi/sbi_domain.c     | 15 +++++++++++++++
 2 files changed, 18 insertions(+)

index 7e288cd8..b7267da8 100644
@@ -232,6 +232,9 @@ struct sbi_domain *sbi_hartindex_to_domain(u32 hartindex);
 /** Update HART local pointer to point to specified domain */
 void sbi_update_hartindex_to_domain(u32 hartindex, struct sbi_domain *dom);
 
+/** Find domain by DT node name (domain name) */
+struct sbi_domain *sbi_domain_find_by_name(const char *name);
+
 /** Get pointer to sbi_domain for current HART */
 #define sbi_domain_thishart_ptr() \
 	sbi_hartindex_to_domain(current_hartindex())

index 2a846eea..c33f2b3c 100644
@@ -60,6 +60,21 @@ void sbi_update_hartindex_to_domain(u32 hartindex, struct sbi_domain *dom)
 	sbi_scratch_write_type(scratch, void *, domain_hart_ptr_offset, dom);
 }
 
+struct sbi_domain *sbi_domain_find_by_name(const char *name)
+{
+	struct sbi_domain *dom;
+
+	if (!name)
+		return NULL;
+
+	sbi_domain_for_each(dom) {
+		if (!sbi_strncmp(dom->name, name, sizeof(dom->name)))
+			return dom;
+	}
+
+	return NULL;
+}
+
 bool sbi_domain_is_assigned_hart(const struct sbi_domain *dom, u32 hartindex)
 {
 	bool ret;

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Init VIRQ and parse mpxy-sysirq nodes under /chosen/opensbi-domains, pre-size per-channel VIRQ maps, and add HWIRQ routes from interrupts-extended.
Mark sysirq domains to allow SEIP-based VIRQ notification.

---
 include/sbi_utils/fdt/fdt_helper.h |  17 +++++
 lib/utils/fdt/fdt_domain.c         | 119 ++++++++++++++++++++++++++++-
 lib/utils/fdt/fdt_helper.c         |  49 ++++++++++++
 3 files changed, 183 insertions(+), 2 deletions(-)

index 04c850cc..e49a5bca 100644
@@ -12,6 +12,7 @@
 
 #include <sbi/sbi_types.h>
 #include <sbi/sbi_domain.h>
+#include <sbi/sbi_irqchip.h>
 
 struct fdt_match {
 	const char *compatible;

@@ -38,6 +39,22 @@ int fdt_parse_phandle_with_args(const void *fdt, int nodeoff,
 				const char *prop, const char *cells_prop,
 				int index, struct fdt_phandle_args *out_args);
 
+/*
+ * Parse one entry from "interrupts-extended" and return irqchip device,
+ * hwirq, and optional flags.
+ *
+ * @index: entry index within interrupts-extended
+ * @out_flags_count: in/out, on input capacity of out_flags array;
+ *                   on output number of flags returned (or total flags
+ *                   if out_flags is NULL).
+ */
+int fdt_parse_interrupts_extended_entry(const void *fdt, int nodeoff,
+					int index,
+					struct sbi_irqchip_device **out_chip,
+					u32 *out_hwirq,
+					u32 *out_flags,
+					u32 *out_flags_count);
+
 int fdt_get_node_addr_size(const void *fdt, int node, int index,
 			   uint64_t *addr, uint64_t *size);
 

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Derive a per-HWIRQ target hartindex map from each sysirq node's target domain boot-hart, and store the result in aplic_data. This is used for setting IDC target per HWIRQ under irqchip driver instead of hardcoding to a specific target.

---
 include/sbi_utils/irqchip/aplic.h     |   1 +
 lib/utils/irqchip/fdt_irqchip_aplic.c | 103 ++++++++++++++++++++++++++
 2 files changed, 104 insertions(+)

index 3461d1c7..d6e088f8 100644
@@ -47,6 +47,7 @@ struct aplic_data {
 	struct aplic_msicfg_data msicfg_smode;
 	struct aplic_delegate_data delegate[APLIC_MAX_DELEGATE];
 	u32 *idc_map;
+	u32 *hwirq_target_hartindex;
 };
 
 int aplic_cold_irqchip_init(struct aplic_data *aplic);

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Lazily resolve the active irqchip provider for each hart when the scratch-local provider is not yet populated.
Program APLIC target IDCs from the precomputed per-HWIRQ hart map, and treat SBI_EALREADY as deferred completion so the normal process path does not complete an interrupt that will be finished later by the VIRQ flow.

---
 lib/sbi/sbi_irqchip.c     | 21 ++++++++++++++
 lib/utils/irqchip/aplic.c | 60 ++++++++++++++++++++++++++++++---------
 2 files changed, 68 insertions(+), 13 deletions(-)

index e022d534..45b2992f 100644
@@ -45,11 +45,28 @@ struct sbi_irqchip_hart_data {
 static unsigned long irqchip_hart_data_off;
 static SBI_LIST_HEAD(irqchip_list);
 
+static struct sbi_irqchip_device *sbi_irqchip_find_hart_device(u32 hartindex)
+{
+	struct sbi_irqchip_device *chip;
+
+	sbi_list_for_each_entry(chip, &irqchip_list, node) {
+		if (!chip->process_hwirqs)
+			continue;
+		if (!sbi_hartmask_test_hartindex(hartindex, &chip->target_harts))
+			continue;
+		return chip;
+	}
+
+	return NULL;
+}
+
 int sbi_irqchip_process(void)
 {
 	struct sbi_irqchip_hart_data *hd;
 
 	hd = sbi_scratch_thishart_offset_ptr(irqchip_hart_data_off);
+	if (hd && !hd->chip)
+		hd->chip = sbi_irqchip_find_hart_device(current_hartindex());
 	if (!hd || !hd->chip || !hd->chip->process_hwirqs)
 		return SBI_ENODEV;
 

@@ -306,6 +323,8 @@ int sbi_irqchip_init(struct sbi_scratch *scratch, bool cold_boot)
 	}
 
 	hd = sbi_scratch_thishart_offset_ptr(irqchip_hart_data_off);
+	if (hd && !hd->chip)
+		hd->chip = sbi_irqchip_find_hart_device(current_hartindex());
 	if (hd && hd->chip && hd->chip->process_hwirqs)
 		csr_set(CSR_MIE, MIP_MEIP);
 

@@ -317,6 +336,8 @@ void sbi_irqchip_exit(struct sbi_scratch *scratch)
 	struct sbi_irqchip_hart_data *hd;
 
 	hd = sbi_scratch_thishart_offset_ptr(irqchip_hart_data_off);
+	if (hd && !hd->chip)
+		hd->chip = sbi_irqchip_find_hart_device(current_hartindex());
 	if (hd && hd->chip && hd->chip->process_hwirqs)
 		csr_clear(CSR_MIE, MIP_MEIP);
 }

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

When boot_hartid points to a hart that is not in the domain's assigned hartmask (e.g. due to cold boot hart differences), the domain startup can skip starting the intended boot hart, leading to intermittent Linux boot failures.
Scan the assigned hartmask and pick the first available hartid to guarantee a valid boot target.

---
 lib/sbi/sbi_domain.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

index c33f2b3c..ac563615 100644
@@ -819,6 +819,30 @@ int sbi_domain_startup(struct sbi_scratch *scratch, u32 cold_hartid)
 
 	/* Startup boot HART of domains */
 	sbi_domain_for_each(dom) {
+		u32 boot_hartindex = sbi_hartid_to_hartindex(dom->boot_hartid);
+		bool boot_assigned = false;
+
+		if (sbi_hartindex_valid(boot_hartindex)) {
+			spin_lock(&dom->assigned_harts_lock);
+			boot_assigned = sbi_hartmask_test_hartindex(
+				boot_hartindex, &dom->assigned_harts);
+			spin_unlock(&dom->assigned_harts_lock);
+		}
+
+		if (!boot_assigned) {
+			u32 new_hartid = -1U;
+
+			spin_lock(&dom->assigned_harts_lock);
+			sbi_hartmask_for_each_hartindex(dhart, &dom->assigned_harts) {
+				new_hartid = sbi_hartindex_to_hartid(dhart);
+				break;
+			}
+			spin_unlock(&dom->assigned_harts_lock);
+
+			if (new_hartid != -1U)
+				dom->boot_hartid = new_hartid;
+		}
+
 		/* Domain boot HART index */
 		dhart = sbi_hartid_to_hartindex(dom->boot_hartid);
 

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Document the DT binding semantics for opensbi,mpxy-sysirq nodes, including how interrupts-extended entry order determines per-channel VIRQ numbers and how opensbi,domain selects the destination OpenSBI domain.
Add a rpmi_sysirq_intc example under /chosen/opensbi-domains.

---
 docs/domain_support.md | 63 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

index 93186c4a..8d474e50 100644
@@ -198,6 +198,48 @@ The DT properties of a domain instance DT node are as follows:
 * **system-suspend-allowed** (Optional) - A boolean flag representing
   whether the domain instance is allowed to do system suspend.
 
+### Domain SysIRQ / VIRQ Routing Node
+
+The domain configuration DT node can also contain sysirq routing nodes for
+describing how a physical interrupt source should be mapped to a VIRQ and
+routed to a target domain. In local DTS overlays this node is often labeled
+`rpmi_sysirq_intc`, but OpenSBI matches it by compatible string rather than
+by node name or label.
+
+The DT properties of a domain sysirq routing DT node are as follows:
+
+* **compatible** (Mandatory) - The compatible string of the sysirq routing
+  node. This DT property should have value *"opensbi,mpxy-sysirq"*
+* **interrupt-controller** (Mandatory) - Marks the node as an interrupt
+  controller for child references.
+* **#interrupt-cells** (Mandatory) - Number of cells used by this interrupt
+  controller. Current examples use value **<1>**.
+* **interrupts-extended** (Mandatory) - The list of routed physical interrupt
+  sources. Each entry is `<&irqchip hwirq flags>`. OpenSBI interprets the
+  entry order as the VIRQ number within the selected MPXY channel, starting
+  from zero:
+  VIRQ 0 maps to entry 0, VIRQ 1 maps to entry 1, and so on.
+* **opensbi,mpxy-channel-id** (Mandatory) - The MPXY channel identifier used
+  as the VIRQ number space for this sysirq node.
+* **opensbi,domain** (Mandatory) - Phandle to the target domain instance DT
+  node. All physical interrupts listed in **interrupts-extended** are routed
+  to this domain after being mapped to VIRQs.
+
+The resulting VIRQ rule for a sysirq node is:
+
+* **mapping** - `(irqchip phandle, hwirq, entry index)` from
+  **interrupts-extended** becomes `(channel-id, virq)`
+* **routing** - `opensbi,domain` selects the destination OpenSBI domain for
+  all VIRQs created from that sysirq node
+
+In other words, a node labeled `rpmi_sysirq_intc` typically means:
+
+* the physical interrupt source is described by one **interrupts-extended**
+  entry
+* the VIRQ number is the position of that entry in the list
+* the VIRQ namespace is selected by **opensbi,mpxy-channel-id**
+* the destination domain is selected by **opensbi,domain**
+
 ### Assigning HART To Domain Instance
 
 By default, all HARTs are assigned to **the ROOT domain**. The OpenSBI

@@ -270,6 +312,27 @@ be done:
                 possible-harts = <&cpu1 &cpu2 &cpu3 &cpu4>;
                 regions = <&tmem 0x0>, <&tuart 0x0>, <&allmem 0x3f>;
             };
+
+            rpmi_sysirq_intc: interrupt-controller {
+                compatible = "opensbi,mpxy-sysirq";
+                interrupt-controller;
+                #interrupt-cells = <1>;
+
+                /*
+                 * VIRQ numbers are assigned from zero in
+                 * interrupts-extended order.
+                 */
+                interrupts-extended =
+                    <&aplic_m 10 4>, /* VIRQ 0 */
+                    <&aplic_m 20 4>, /* VIRQ 1 */
+                    <&aplic_m 21 4>; /* VIRQ 2 */
+
+                /* Select the VIRQ namespace / MPXY channel. */
+                opensbi,mpxy-channel-id = <4>;
+
+                /* Route all VIRQs from this node to udomain. */
+                opensbi,domain = <&udomain>;
+            };
         };
     };
 

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

This VIRQ abstraction is not needed with the improved SBI irqchip where each external interrupt (wired / MSI) can be uniquely identified by a tuple which SBI irqchip instance and hwirq number in the instance.

Also, the VIRQ ecall is ad-hoc and does not correspond to any SBI extension.

All-in-all the whole interrupt routing for supervisor domains must be re-written using SBI MPXY + RPMI system irq service group as the interface for supervisor domains.

The SBI MPXY is already a ratified SBI extension and OpenSBI already has an extensible MPXY framework with a number of MPXY drivers under lib/utils/mpxy. The RPMI system irq draft specifcation was already shared few months back.

Regards,
Anup

opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

Hi Anup,

As we discussed in the SSWG meeting, the plan to support the SBI MPXY
+ RPMI system irq service group is ongoing.
It will be much appreciated if you can consider to merge this series to enable the interrupt routing first before I post a separate series for RPMI on top of this.

Regards,
Raymond

opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

Like I explained above, you don't need this series for implementing SBI MPXY + RPMI system irq service group.

Regards,
Anup

opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi