Introduce Virtual IRQ (VIRQ) framework

From: Raymond Mao <[email protected]>

This series introduces a Virtual IRQ (VIRQ) framework for OpenSBI to support domain-aware wired interrupt couriering.

The main use case is to let OpenSBI receive a host HWIRQ in M-mode, map it into a per-channel VIRQ number space, route it to a target domain, notify the target S-mode payload, and let that payload claim the pending VIRQs through RPMI SYSTEM_IRQ over SBI MPXY.

The implementation is organized as follows:

- add RPMI SYSTEM_IRQ service support, including SYSIRQ_CLAIM
- add an optional experimental Yield ecall for explicit completion
- add irqchip helpers for S-mode pending notification
- extend domain context switching to support VIRQ-driven cross-domain couriering and return-to-previous-domain flow
- add the core VIRQ mapping, routing, and per-(domain,hart) pending queue framework
- parse sysirq routing rules from DT under /chosen/opensbi-domains
- derive APLIC target hart routing from sysirq nodes
- update the irqchip/APLIC path to support deferred completion and per-HWIRQ APLIC targets
- select domain boot harts before activating physical sources
- document the DT binding and routing rules

The DT routing model is based on "opensbi,mpxy-sysirq" nodes. Each interrupts-extended entry contributes one routed physical source, its entry index becomes the VIRQ number within the selected opensbi,mpxy-channel-id, and opensbi,domain selects the destination OpenSBI domain.

Claimed sources remain masked while the payload handles their device conditions. The next valid CLAIM completes the previous batch, or the payload can complete it explicitly through Yield. Return to a previously interrupted domain is deferred until pending and claimed work is drained. The Yield EID is experimental and is not an official SBI assignment.

This version reuses the existing irqchip infrastructure.

Patch 12 contains sample DT overlays for reference only and is marked [NOT-FOR-UPSTREAM].

The series are also placed at:
https://gitlab.com/riseproject/riscv-optee/opensbi/-/tree/rp022-m1

Test instructions are available at:
https://riscv-optee-docs-c9fea0.gitlab.io/rp022/milestone1/

Changes in v2:

- Rebase on master and adapt to its irqchip interfaces, dropping the old scratch-local provider lookup workaround.
- Replace the private VIRQ POP/COMPLETE ecall extension with RPMI SYSTEM_IRQ 2.0 services over SBI MPXY, including batched SYSIRQ_CLAIM.
- Add an optional experimental Yield ecall for explicit completion after device handling; retain implicit completion through the next valid CLAIM.
- Track outstanding claims per channel and target, preserve failed completions for retry, and block return switches until all pending and claimed work is drained.
- Keep SEPC separate from the M-mode trap frame's MEPC and preserve the advanced ecall return state across domain switches.
- Key routes by (chip_uid, hwirq), allowing identical HWIRQ numbers on different physical controllers without routing collisions.
- Select boot harts during domain finalization and activate couriers only after boot-hart selection and all mapping/channel setup.
- Add irqchip-independent courier registration with DT IRQ flags, target affinity and rollback; use APLIC IDC indices for affinity.
- Reject malformed DT argument lists, extra IRQ specifier cells and invalid scalar properties; update the binding documentation.

Raymond Mao (12):
lib: utils: add RPMI SYSTEM_IRQ service support
lib: sbi: add experimental Yield ECALL extension
lib: irqchip: add S-mode notification helpers
lib: sbi: domain: support VIRQ couriering domain context switch
lib: sbi: domain: ensure boot_hartid is assigned
lib: sbi: Add Virtual IRQ (VIRQ) subsystem
lib: sbi: domain: add domain lookup by name
lib: utils: irqchip: derive APLIC targets from sysirq nodes
lib: utils: irqchip: Add helper to register VIRQ courier handler
lib: utils: fdt: parse sysirq routing from DT
docs: domain: document sysirq VIRQ mapping and routing rules [NOT-FOR-UPSTREAM] platform: generic: Add sample dts overlays for
testing hwirq/domain route rules

 docs/domain_support.md                        |   55 +
 include/sbi/sbi_domain.h                      |    7 +
 include/sbi/sbi_domain_context.h              |   24 +
 include/sbi/sbi_ecall_interface.h             |    6 +
 include/sbi/sbi_irqchip.h                     |    9 +
 include/sbi/sbi_virq.h                        |  513 +++++++
 include/sbi_utils/fdt/fdt_helper.h            |    8 +
 include/sbi_utils/irqchip/aplic.h             |    4 +
 include/sbi_utils/mailbox/rpmi_msgprot.h      |   25 +
 include/sbi_utils/sysirq/rpmi_sysirq.h        |   30 +
 lib/sbi/Kconfig                               |    9 +
 lib/sbi/objects.mk                            |    4 +
 lib/sbi/sbi_domain.c                          |   61 +
 lib/sbi/sbi_domain_context.c                  |  130 +-
 lib/sbi/sbi_ecall_yield.c                     |   42 +
 lib/sbi/sbi_irqchip.c                         |   21 +-
 lib/sbi/sbi_trap.c                            |   11 +
 lib/sbi/sbi_virq.c                            | 1235 +++++++++++++++++
 lib/utils/fdt/fdt_domain.c                    |  116 +-
 lib/utils/fdt/fdt_helper.c                    |   57 +-
 lib/utils/irqchip/aplic.c                     |   57 +-
 lib/utils/irqchip/fdt_irqchip_aplic.c         |  103 ++
 lib/utils/mpxy/objects.mk                     |    2 +
 lib/utils/sysirq/rpmi_sysirq.c                |  266 ++++
 platform/generic/virt/hwirq_bind_domain.dts   |  107 ++
 .../virt/hwirq_bind_domain_linux_bmapp.dts    |  125 ++
 26 files changed, 3017 insertions(+), 10 deletions(-)
 create mode 100644 include/sbi/sbi_virq.h
 create mode 100644 include/sbi_utils/sysirq/rpmi_sysirq.h
 create mode 100644 lib/sbi/sbi_ecall_yield.c
 create mode 100644 lib/sbi/sbi_virq.c
 create mode 100644 lib/utils/sysirq/rpmi_sysirq.c
 create mode 100644 platform/generic/virt/hwirq_bind_domain.dts
 create mode 100644 platform/generic/virt/hwirq_bind_domain_linux_bmapp.dts

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Provide an SBI MPXY implementation of the RPMI SYSTEM_IRQ service group, version 2.0, with GET_ATTRIBUTES, GET_TARGET_ATTRIBUTES, GET_IRQ_ATTRIBUTES and SYSIRQ_CLAIM. Other services return NOTSUPP.

Expose backend callbacks for claiming and completing IRQs, allowing delivery implementations to register domain-owned channels. Target indices enumerate possible harts and CLAIM is restricted to the calling domain and hart.

Keep claimed sources outstanding until the next valid CLAIM, after the payload has handled the previous batch. Preserve failed completion records for retry. This completion convention introduces no new RPMI service and leaves backend device handling outside the protocol layer.

---
 include/sbi_utils/mailbox/rpmi_msgprot.h |  25 +++
 include/sbi_utils/sysirq/rpmi_sysirq.h   |  30 +++
 lib/utils/mpxy/objects.mk                |   2 +
 lib/utils/sysirq/rpmi_sysirq.c           | 266 +++++++++++++++++++++++
 4 files changed, 323 insertions(+)
 create mode 100644 include/sbi_utils/sysirq/rpmi_sysirq.h
 create mode 100644 lib/utils/sysirq/rpmi_sysirq.c

index 7ae4abe0..f9d1d177 100644
@@ -226,6 +226,7 @@ enum rpmi_servicegroup_id {
 	RPMI_SRVGRP_PERFORMANCE = 0x0000A,
 	RPMI_SRVGRP_MANAGEMENT_MODE = 0x000B,
 	RPMI_SRVGRP_LOGGING = 0x000E,
+	RPMI_SRVGRP_SYSTEM_IRQ = 0x000F,
 	RPMI_SRVGRP_ID_MAX_COUNT,
 
 	/* Reserved range for service groups */

@@ -290,6 +291,30 @@ struct rpmi_base_get_platform_info_resp {
 	char plat_info[];
 };
 
+/** RPMI System IRQ ServiceGroup Service IDs */
+enum rpmi_sysirq_service_id {
+	RPMI_SYSIRQ_SRV_ENABLE_NOTIFICATION = 0x01,
+	RPMI_SYSIRQ_SRV_GET_ATTRIBUTES = 0x02,
+	RPMI_SYSIRQ_SRV_GET_TARGET_ATTRIBUTES = 0x03,
+	RPMI_SYSIRQ_SRV_GET_IRQ_ATTRIBUTES = 0x04,
+	RPMI_SYSIRQ_SRV_SET_IRQ_STATE = 0x05,
+	RPMI_SYSIRQ_SRV_GET_IRQ_STATE = 0x06,
+	RPMI_SYSIRQ_SRV_SET_IRQ_TARGET = 0x07,
+	RPMI_SYSIRQ_SRV_GET_IRQ_TARGET = 0x08,
+	RPMI_SYSIRQ_SRV_CLAIM = 0x09,
+	RPMI_SYSIRQ_SRV_ID_MAX_COUNT,
+};
+
+struct rpmi_sysirq_claim_req {
+	u32 sys_target_index;
+};
+
+struct rpmi_sysirq_claim_resp {
+	s32 status;
+	u32 returned;
+	u32 sys_irq_index[];
+};
+
 /** RPMI System MSI ServiceGroup Service IDs */
 enum rpmi_sysmsi_service_id {
 	RPMI_SYSMSI_SRV_ENABLE_NOTIFICATION = 0x01,

new file mode 100644
index 00000000..0af2f33b
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: BSD-2-Clause */
+/*
+ * Copyright (c) 2026 RISCstar Solutions Corporation.
+ *
+ * Author: Raymond Mao <[email protected]>
+ */
+
+#ifndef __RPMI_SYSIRQ_H__
+#define __RPMI_SYSIRQ_H__
+
+#include <sbi/sbi_types.h>
+
+struct sbi_domain;
+
+/* Callbacks operate on the calling domain and hart; ops must remain valid. */
+struct rpmi_sysirq_ops {
+	int (*claim)(u32 channel_id, u32 *irqs, u32 max_count, u32 *count);
+	int (*complete)(u32 channel_id, u32 irq);
+	void (*drained)(void);
+};
+
+int rpmi_sysirq_register_channel(struct sbi_domain *dom, u32 channel_id,
+				u32 num_irqs, const struct rpmi_sysirq_ops *ops);
+
+bool rpmi_sysirq_available(void);
+
+/* Complete this domain/hart's batches, then ask backends to consider return. */
+int rpmi_sysirq_complete_thishart(u32 *completed);
+
+#endif

index 1f1bf0a0..a59a7175 100644
@@ -32,3 +32,5 @@ libsbiutils-objs-$(CONFIG_FDT_MPXY_RPMI_MM) += mpxy/fdt_mpxy_rpmi_mm.o
 
 carray-fdt_mpxy_drivers-$(CONFIG_FDT_MPXY_RPMI_LOGGING) += fdt_mpxy_rpmi_logging
 libsbiutils-objs-$(CONFIG_FDT_MPXY_RPMI_LOGGING) += mpxy/fdt_mpxy_rpmi_logging.o
+
+libsbiutils-objs-y += sysirq/rpmi_sysirq.o

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Add a Yield extension with EID 0x0800594c and FID 0. The ID is experimental and is not an official SBI assignment.

Complete the calling domain and hart's outstanding SYSIRQ batches using the same completion mechanism as the next CLAIM. Return the number completed, including partial progress on error, without claiming new IRQs. Ask the backend to consider a drained return only after completion succeeds.

Enable the extension with CONFIG_SBI_ECALL_YIELD and register it only when SYSIRQ channels exist.

---
 include/sbi/sbi_ecall_interface.h |  6 +++++
 lib/sbi/Kconfig                   |  9 +++++++
 lib/sbi/objects.mk                |  3 +++
 lib/sbi/sbi_ecall_yield.c         | 42 +++++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+)
 create mode 100644 lib/sbi/sbi_ecall_yield.c

index fd4e77ca..287d3151 100644
@@ -39,6 +39,9 @@
 #define SBI_EXT_FWFT				0x46574654
 #define SBI_EXT_MPXY				0x4D505859
 
+/* SBI extension ID for Yield. */
+#define SBI_EXT_YIELD				0x0800594c
+
 /* SBI function IDs for BASE extension*/
 #define SBI_EXT_BASE_GET_SPEC_VERSION		0x0
 #define SBI_EXT_BASE_GET_IMP_ID			0x1

@@ -122,6 +125,9 @@
 #define SBI_EXT_DBTR_TRIGGER_ENABLE	0x6
 #define SBI_EXT_DBTR_TRIGGER_DISABLE	0x7
 
+/* SBI function IDs for the Yield extension. */
+#define SBI_EXT_YIELD_YIELD	0x0
+
 /* SBI function IDs for FW feature extension */
 #define SBI_EXT_FWFT_SET		0x0
 #define SBI_EXT_FWFT_GET		0x1

index 24132eaa..894ce22f 100644
@@ -76,6 +76,15 @@ config SBI_ECALL_LEGACY
 	bool "SBI v0.1 legacy extensions"
 	default y
 
+config SBI_ECALL_YIELD
+	bool "Yield extension"
+	default y
+	help
+	  Enable the Yield ECALL extension.
+	  Yield completes the calling domain and hart's claimed SYSIRQs.
+	  A delivery backend may request a context return once work is drained.
+	  Interrupt claiming still uses RPMI SYSTEM_IRQ through SBI MPXY.
+
 config SBI_ECALL_VENDOR
 	bool "Platform-defined vendor extensions"
 	default y

index ae27d0c5..16c141fe 100644
@@ -64,6 +64,9 @@ libsbi-objs-$(CONFIG_SBI_ECALL_SSE) += sbi_ecall_sse.o
 carray-sbi_ecall_exts-$(CONFIG_SBI_ECALL_MPXY) += ecall_mpxy
 libsbi-objs-$(CONFIG_SBI_ECALL_MPXY) += sbi_ecall_mpxy.o
 
+carray-sbi_ecall_exts-$(CONFIG_SBI_ECALL_YIELD) += ecall_yield
+libsbi-objs-$(CONFIG_SBI_ECALL_YIELD) += sbi_ecall_yield.o
+
 libsbi-objs-y += sbi_bitmap.o
 libsbi-objs-y += sbi_bitops.o
 libsbi-objs-y += sbi_console.o

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Add irqchip helpers to set/clear S-mode notification (SEIP-based). In addition to set/clear, expose a get API to read the current notification state so upper layers can do edge-triggered notification.

Allow raw handlers to return SBI_EALREADY when physical completion is deferred to the recipient, instead of issuing an immediate EOI. Preserve ordinary completion behavior for other handler results.

Release handler storage when unregistering an interrupt handler so failed courier setup can roll back without leaking memory.

---
 include/sbi/sbi_irqchip.h |  9 +++++++++
 lib/sbi/sbi_irqchip.c     | 21 +++++++++++++++++++--
 2 files changed, 28 insertions(+), 2 deletions(-)

index f5dbf1c7..174fa930 100644
@@ -172,4 +172,13 @@ void sbi_irqchip_exit(struct sbi_scratch *scratch);
 bool sbi_irqchip_is_hwirq_enabled(struct sbi_irqchip_device *chip,
 				   u32 hwirq);
 
+/** Notify S-mode for a pending virtual interrupt on this hart. */
+int sbi_irqchip_notify_smode_set(void);
+
+/** Clear S-mode notification for virtual interrupts on this hart. */
+void sbi_irqchip_notify_smode_clear(void);
+
+/** Read S-mode notification state for virtual interrupts on this hart. */
+bool sbi_irqchip_notify_smode_get(void);
+
 #endif

index 1664bd2d..f9afadc0 100644
@@ -14,6 +14,7 @@
 #include <sbi/sbi_list.h>
 #include <sbi/sbi_platform.h>
 #include <sbi/sbi_scratch.h>
+#include <sbi/riscv_asm.h>
 
 /** Internal irqchip hardware interrupt data */
 struct sbi_irqchip_hwirq_data {

@@ -191,9 +192,8 @@ int sbi_irqchip_raw_handler_default(struct sbi_irqchip_device *chip, u32 hwirq)
 	if (h->callback)
 		rc = h->callback(hwirq, h->priv);
 
-	if (chip->hwirq_eoi)
+	if (chip->hwirq_eoi && rc != SBI_EALREADY)
 		chip->hwirq_eoi(chip, hwirq);
-
 	return rc;
 }
 

@@ -471,6 +471,7 @@ int sbi_irqchip_unregister_handler(struct sbi_irqchip_device *chip,
 	}
 
 	sbi_list_del(&fh->node);
+	sbi_free(fh);
 	return 0;
 }
 

@@ -618,3 +619,19 @@ skip_migrate:
 	if (hd && hd->chip && hd->chip->process_hwirqs)
 		csr_clear(CSR_MIE, MIP_MEIP);
 }
+
+int sbi_irqchip_notify_smode_set(void)
+{
+	csr_set(CSR_MIP, MIP_SEIP);
+	return 0;
+}
+
+void sbi_irqchip_notify_smode_clear(void)
+{
+	csr_clear(CSR_MIP, MIP_SEIP);
+}
+
+bool sbi_irqchip_notify_smode_get(void)
+{
+	return !!(csr_read(CSR_MIP) & MIP_SEIP);
+}

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Prerequisite adaptations for introducing VIRQ couriering domain context switch.

MIDELEG:
- Save/restore MIDELEG in domain contexts and initialize per-domain defaults.

SEIP notification:
- Add virq_seip_notify flag, and use it to enable SEIP delegation for SEIP-notify domains.
- Add smode_notify_pending and helper function to store the S-mode notification status, and use it as a flag to fire the pending notification after domain context switch.

Return domain context switch:
- Add sbi_domain_context_exit_to_prev() to return to the previous domain without scanning for another candidate.
- Introduce per-hart deferred return flags and APIs to request/consume them.
- Perform the actual return-to-prev at the end of sbi_trap_handler() to avoid corrupting mepc during ecall handling.

Additionally, fix return domain-switch issues:
- When a domain switch occurs inside sbi_trap_handler(), return the switched-to trap context from scratch instead of the original trap entry context. This prevents the trap restore path from resuming with stale state from the previous domain.
- When returning to an S-mode target domain, copy the restored CSR_SSTATUS SIE/SPIE/SPP bits into trap_ctx->regs.mstatus. The final trap exit writes CSR_MSTATUS from the trap context, so stale mstatus bits can otherwise clear S-mode interrupt enable and leave a pending SEIP undelivered.
- Keep SEPC independent of the M-mode trap frame's MEPC and preserve the updated trap frame instead of replacing its return state with live CSRs. This prevents a resumed domain from repeating an ECALL whose result and advanced return PC have already been recorded.

---
 include/sbi/sbi_domain.h         |   2 +
 include/sbi/sbi_domain_context.h |  24 ++++++
 lib/sbi/sbi_domain_context.c     | 130 ++++++++++++++++++++++++++++++-
 lib/sbi/sbi_trap.c               |  11 +++
 4 files changed, 166 insertions(+), 1 deletion(-)

index 1f773e41..31cbe01e 100644
@@ -219,6 +219,8 @@ struct sbi_domain {
 	bool system_suspend_allowed;
 	/** Identifies whether to include the firmware region */
 	bool fw_region_inited;
+	/** Whether to notify S-mode for VIRQ couriering */
+	bool virq_seip_notify;
 };
 
 /** The root domain instance */

index 31a3a7f8..88450fcb 100644
@@ -28,6 +28,30 @@ int sbi_domain_context_enter(struct sbi_domain *dom);
  */
 int sbi_domain_context_exit(void);
 
+/**
+ * Exit the current domain context and return to the previous context
+ * if one exists. This will not attempt to start other domains.
+ *
+ * @return 0 on success and negative error code on failure
+ */
+int sbi_domain_context_exit_to_prev(void);
+
+void sbi_domain_context_request_return_to_prev(void);
+bool sbi_domain_context_need_return_to_prev(void);
+void sbi_domain_context_mark_switched(void);
+bool sbi_domain_context_consume_switched(void);
+
+/**
+ * Mark a pending S-mode notification for a target domain context.
+ *
+ * @param dom pointer to domain
+ * @param hartindex hart index
+ *
+ * @return true if notification was already pending, false otherwise
+ */
+bool sbi_domain_context_pending_notify_smode(struct sbi_domain *dom,
+					     u32 hartindex);
+
 /**
  * Initialize domain context support
  *

index 16774fd2..00905497 100644
@@ -25,6 +25,7 @@
 #include <sbi/sbi_sse.h>
 #include <sbi/sbi_timer.h>
 #include <sbi/sbi_trap.h>
+#include <sbi/sbi_domain_context.h>
 
 static void sbi_trap_error_one(const struct sbi_trap_context *tcntx,
 			       const char *prefix, u32 hartid, u32 depth)

@@ -373,6 +374,16 @@ trap_done:
 	if (sbi_mstatus_prev_mode(regs->mstatus) != PRV_M)
 		sbi_sse_process_pending_events(regs);
 
+	if (sbi_domain_context_need_return_to_prev())
+		sbi_domain_context_exit_to_prev();
+
+	if (sbi_domain_context_consume_switched()) {
+		struct sbi_trap_context *newctx = sbi_trap_get_context(scratch);
+
+		sbi_trap_set_context(scratch, newctx->prev_context);
+		return newctx;
+	}
+
 	sbi_trap_set_context(scratch, tcntx->prev_context);
 	return tcntx;
 }

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

When boot_hartid points to a hart that is not in the domain's assigned hartmask (e.g. due to cold boot hart differences), the domain startup can skip starting the intended boot hart, leading to intermittent Linux boot failures.
Scan the assigned hartmask and pick the first available hartid to provide a valid boot target when assigned HARTs are available.

Select boot HARTs during domain finalization, after all domains have been registered, rather than changing them during startup. This keeps the selected target stable for physical interrupt affinity setup. Preserve the configured boot HART when no HART is initially assigned.

---
 lib/sbi/sbi_domain.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

index b3f5892d..604c5b7a 100644
@@ -799,6 +799,38 @@ int sbi_domain_root_add_memrange(unsigned long addr, unsigned long size,
 	return 0;
 }
 
+static void domain_select_boot_harts(void)
+{
+	u32 dhart;
+	struct sbi_domain *dom;
+
+	sbi_domain_for_each(dom) {
+		u32 boot_hartindex = sbi_hartid_to_hartindex(dom->boot_hartid);
+		bool boot_assigned = false;
+
+		if (sbi_hartindex_valid(boot_hartindex)) {
+			spin_lock(&dom->assigned_harts_lock);
+			boot_assigned = sbi_hartmask_test_hartindex(
+				boot_hartindex, &dom->assigned_harts);
+			spin_unlock(&dom->assigned_harts_lock);
+		}
+
+		if (!boot_assigned) {
+			u32 new_hartid = -1U;
+
+			spin_lock(&dom->assigned_harts_lock);
+			sbi_hartmask_for_each_hartindex(dhart, &dom->assigned_harts) {
+				new_hartid = sbi_hartindex_to_hartid(dhart);
+				break;
+			}
+			spin_unlock(&dom->assigned_harts_lock);
+
+			if (new_hartid != -1U)
+				dom->boot_hartid = new_hartid;
+		}
+	}
+}
+
 int sbi_domain_startup(struct sbi_scratch *scratch, u32 cold_hartid)
 {
 	int rc;

@@ -873,6 +905,9 @@ int sbi_domain_finalize(struct sbi_scratch *scratch)
 		return rc;
 	}
 
+	/* Select stable boot HARTs before programming physical IRQ targets. */
+	domain_select_boot_harts();
+
 	/*
 	 * Set the finalized flag so that the root domain
 	 * regions can't be changed.

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

VIRQ is an abstraction framework providing per-MPXY-channel HWIRQ<->VIRQ mapping, per-(domain,hart) VIRQ routing and couriering. It notifies S-mode payload via irqchip SEIP helpers when a VIRQ is mapped/routed/enqueued and provides backend callbacks for claiming and completing pending VIRQs through RPMI SYSTEM_IRQ channels.

VIRQ layer is composed of three major parts:
1. VIRQ mapping and allocation
- Provides a stable per-MPXY-channel mapping between a host physical interrupt endpoint (chip_uid, hwirq) and a VIRQ number.
- VIRQ number allocation uses a scalable bitmap.
2. Physical IRQ->(Domain,hart) routing rules
- Supports routing rules derived from sysirq nodes by the interrupts-extended property, for example:
interrupts-extended =
<&aplic HWIRQx IRQ_TYPE>, // virq 0
<&aplic HWIRQy IRQ_TYPE>, // virq 1
...;
- VIRQ numbers are allocated from zero, implicit from the order of the entries in the interrupts-extended property.
- Cache routes by (chip_uid, hwirq), rejecting duplicate physical endpoints while allowing the same HWIRQ on different controllers.
- Default behavior: if a physical endpoint does not match any routing rule, it will be routed to the root domain (channel 0) as a fallback.
3. Per-(domain,hart) pending queue couriering
- Each domain maintains a per-hart ring buffer queue of pending VIRQs. A courier handler masks the physical source before enqueueing a VIRQ on HWIRQ assertion.
- The couriering is domain-aware. It switches to the target domain when it is not the same as the current one, and requests a return to the previous domain after pending and claimed work is drained.
- S-mode notification is edge-triggered based on the irqchip notify state, and is cleared only when the queue becomes empty.

Bind VIRQ queues to the RPMI SYSTEM_IRQ backend. SYSIRQ_CLAIM removes only the selected channel's entries while preserving other entries in the shared domain queue. Index zero is a valid VIRQ. Keep claimed sources masked until device handling is followed by another CLAIM or Yield. Track outstanding claims and rearm each source through EOI/unmask before allowing the deferred return domain switch.

Provide an irqchip-independent helper for DT-driven courier registration. Configure the physical interrupt with the supplied IRQ flags and target hart, and roll back registration if target setup or interrupt enabling fails. Store the physical IRQ flags with each route and provide a setup entry point to activate couriers after domain boot HART selection.

---
 include/sbi/sbi_domain.h |    2 +
 include/sbi/sbi_virq.h   |  513 ++++++++++++++++
 lib/sbi/objects.mk       |    1 +
 lib/sbi/sbi_domain.c     |    8 +
 lib/sbi/sbi_virq.c       | 1235 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 1759 insertions(+)
 create mode 100644 include/sbi/sbi_virq.h
 create mode 100644 lib/sbi/sbi_virq.c

index 31cbe01e..3e357e67 100644
@@ -221,6 +221,8 @@ struct sbi_domain {
 	bool fw_region_inited;
 	/** Whether to notify S-mode for VIRQ couriering */
 	bool virq_seip_notify;
+	/** per-domain wired-IRQ courier state */
+	void *virq_priv;
 };
 
 /** The root domain instance */

index 16c141fe..cbd6f6bf 100644
@@ -89,6 +89,7 @@ libsbi-objs-y += sbi_illegal_insn.o
 libsbi-objs-y += sbi_init.o
 libsbi-objs-y += sbi_ipi.o
 libsbi-objs-y += sbi_irqchip.o
+libsbi-objs-y += sbi_virq.o
 libsbi-objs-y += sbi_platform.o
 libsbi-objs-y += sbi_pmp.o
 libsbi-objs-y += sbi_pmu.o

index 604c5b7a..30588678 100644
@@ -18,6 +18,7 @@
 #include <sbi/sbi_platform.h>
 #include <sbi/sbi_scratch.h>
 #include <sbi/sbi_string.h>
+#include <sbi/sbi_virq.h>
 
 SBI_LIST_HEAD(domain_list);
 

@@ -710,6 +711,13 @@ int sbi_domain_register(struct sbi_domain *dom)
 		return rc;
 	}
 
+	/* Init per-domain wired-IRQ courier state */
+	rc = sbi_virq_domain_init(dom);
+	if (rc) {
+		sbi_list_del(&dom->node);
+		return rc;
+	}
+
 	return 0;
 }
 

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Provide a helper to resolve a domain by its DT node name, used by sysirq DT parsing.

---
 include/sbi/sbi_domain.h |  3 +++
 lib/sbi/sbi_domain.c     | 15 +++++++++++++++
 2 files changed, 18 insertions(+)

index 3e357e67..3cb49007 100644
@@ -234,6 +234,9 @@ struct sbi_domain *sbi_hartindex_to_domain(u32 hartindex);
 /** Update HART local pointer to point to specified domain */
 void sbi_update_hartindex_to_domain(u32 hartindex, struct sbi_domain *dom);
 
+/** Find domain by DT node name (domain name) */
+struct sbi_domain *sbi_domain_find_by_name(const char *name);
+
 /** Get pointer to sbi_domain for current HART */
 #define sbi_domain_thishart_ptr() \
 	sbi_hartindex_to_domain(current_hartindex())

index 30588678..16df5533 100644
@@ -61,6 +61,21 @@ void sbi_update_hartindex_to_domain(u32 hartindex, struct sbi_domain *dom)
 	sbi_scratch_write_type(scratch, void *, domain_hart_ptr_offset, dom);
 }
 
+struct sbi_domain *sbi_domain_find_by_name(const char *name)
+{
+	struct sbi_domain *dom;
+
+	if (!name)
+		return NULL;
+
+	sbi_domain_for_each(dom) {
+		if (!sbi_strncmp(dom->name, name, sizeof(dom->name)))
+			return dom;
+	}
+
+	return NULL;
+}
+
 bool sbi_domain_is_assigned_hart(const struct sbi_domain *dom, u32 hartindex)
 	MUST_NOT_HOLD(&dom->assigned_harts_lock)
 {

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Derive a per-HWIRQ target hartindex map from each sysirq node's target domain boot-hart, and store the result in aplic_data. This is used for setting IDC target per HWIRQ under irqchip driver instead of hardcoding to a specific target.

---
 include/sbi_utils/irqchip/aplic.h     |   1 +
 lib/utils/irqchip/fdt_irqchip_aplic.c | 103 ++++++++++++++++++++++++++
 2 files changed, 104 insertions(+)

index 8d6d871e..76b39099 100644
@@ -48,6 +48,7 @@ struct aplic_data {
 	struct aplic_msicfg_data msicfg_smode;
 	struct aplic_delegate_data delegate[APLIC_MAX_DELEGATE];
 	u32 *idc_map;
+	u32 *hwirq_target_hartindex;
 };
 
 int aplic_cold_irqchip_init(struct aplic_data *aplic);

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Add an APLIC helper to register a VIRQ courier handler for a supplied HWIRQ and IRQ type, using the precomputed per-HWIRQ target hartindex map.

Translate OpenSBI hart indices to APLIC IDC indices when programming targets and setting affinity. Reject targets with no matching IDC instead of directing interrupts to an unrelated fallback IDC. Reuse the VIRQ registration helper to preserve the selected target after handler registration, which initially selects the calling hart.

Treat SBI_EALREADY as deferred completion rather than an error when processing hardware interrupts.

---
 include/sbi_utils/irqchip/aplic.h |  3 ++
 lib/utils/irqchip/aplic.c         | 57 +++++++++++++++++++++++++++++--
 2 files changed, 58 insertions(+), 2 deletions(-)

index 76b39099..44bbfcf6 100644
@@ -53,6 +53,9 @@ struct aplic_data {
 
 int aplic_cold_irqchip_init(struct aplic_data *aplic);
 
+int aplic_init_virq_courier(struct aplic_data *aplic, u32 hwirq,
+			    u32 hwirq_flags);
+
 void aplic_reinit_all(void);
 
 #endif

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Parse mpxy-sysirq nodes under /chosen/opensbi-domains, pre-size per-channel VIRQ maps, and add physical interrupt routes from interrupts-extended. Mark sysirq domains to allow SEIP-based VIRQ notification.

Initialize VIRQ on demand and register each domain-owned RPMI SYSTEM_IRQ channel using its channel ID and interrupt entry count. Store the physical controller, HWIRQ and IRQ flags in each route. Activate courier handlers during domain finalization after boot HART selection, setting affinity to the destination domain's final boot HART. Complete mapping and channel setup before activating physical sources.

Reject malformed phandle argument lists and insufficient IRQ flag buffers instead of truncating interrupt specifiers. Require one HWIRQ cell and one flags cell for SYSIRQ sources, and validate scalar channel and domain properties.

---
 include/sbi_utils/fdt/fdt_helper.h |   8 ++
 lib/sbi/sbi_domain.c               |   3 +
 lib/utils/fdt/fdt_domain.c         | 116 ++++++++++++++++++++++++++++-
 lib/utils/fdt/fdt_helper.c         |  57 +++++++++++++-
 4 files changed, 179 insertions(+), 5 deletions(-)

index 75a564d1..342cf738 100644
@@ -12,6 +12,7 @@
 
 #include <sbi/sbi_types.h>
 #include <sbi/sbi_domain.h>
+#include <sbi/sbi_irqchip.h>
 
 struct fdt_match {
 	const char *compatible;

@@ -38,6 +39,13 @@ int fdt_parse_phandle_with_args(const void *fdt, int nodeoff,
 				const char *prop, const char *cells_prop,
 				int index, struct fdt_phandle_args *out_args);
 
+int fdt_parse_interrupts_extended_entry(const void *fdt, int nodeoff,
+					int index,
+					struct sbi_irqchip_device **out_chip,
+					u32 *out_hwirq,
+					u32 *out_flags,
+					u32 *out_flags_count);
+
 int fdt_get_node_addr_size(const void *fdt, int node, int index,
 			   uint64_t *addr, uint64_t *size);
 

index 16df5533..36b64b32 100644
@@ -930,6 +930,9 @@ int sbi_domain_finalize(struct sbi_scratch *scratch)
 
 	/* Select stable boot HARTs before programming physical IRQ targets. */
 	domain_select_boot_harts();
+	rc = sbi_virq_setup_couriers();
+	if (rc)
+		return rc;
 
 	/*
 	 * Set the finalized flag so that the root domain

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Document the DT binding semantics for opensbi,mpxy-sysirq nodes, including how interrupts-extended entry order determines per-channel VIRQ numbers and how opensbi,domain selects the destination OpenSBI domain.
Add a SYSIRQ node example under /chosen/opensbi-domains.

Describe DT-driven courier registration and delivery to the owning domain's boot hart. Summarize interrupt notification, claiming, completion and deferred return domain switches.

---
 docs/domain_support.md | 55 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)

index 51737218..a2d40024 100644
@@ -325,3 +325,58 @@ be done:
         ...
     };
 ```
+
+### SYSIRQ Node
+
+The SYSIRQ node describes physical interrupts delivered to a domain through
+an RPMI SYSTEM_IRQ channel over SBI MPXY. It should be placed under the
+domain configuration node.
+
+The DT properties of a SYSIRQ node are as follows:
+
+* **compatible** (Mandatory) - The compatible string of the SYSIRQ node.
+  This DT property should have value *"opensbi,mpxy-sysirq"*.
+* **interrupt-controller** (Optional) - Marks the node as a virtual interrupt
+  controller when it is referenced by other DT nodes.
+* **#interrupt-cells** (Optional) - Must be <1> when the node is used as an
+  interrupt controller. The interrupt specifier is the VIRQ number.
+* **interrupts-extended** (Mandatory) - The physical interrupt sources,
+  specified as <&irqchip hwirq flags> entries.
+* **opensbi,mpxy-channel-id** (Mandatory) - The unique SBI MPXY channel ID.
+  Each channel has its own VIRQ namespace.
+* **opensbi,domain** (Mandatory) - The phandle of the destination domain
+  instance node.
+
+Each interrupts-extended entry maps a physical interrupt to a VIRQ number
+starting from zero, in entry order. All entries in the node belong to the
+specified domain. Physical sources are identified by the interrupt controller
+and HWIRQ number; the same HWIRQ number on different controllers denotes
+different sources. Each physical source can belong to only one channel.
+The physical specifier must contain one HWIRQ cell and one flags cell.
+Physical delivery targets the domain's finalized boot HART. The current
+registration path supports wired M-mode interrupt controllers with masking
+and affinity operations. The interrupt controller must be configured to
+deliver these sources to M-mode rather than delegate them to S-mode.
+
+OpenSBI establishes the channel mappings and routing rules, then selects
+domain boot HARTs before enabling physical delivery. It notifies the
+destination domain through SEIP and switches to that domain if required.
+The payload claims pending interrupts through SYSIRQ_CLAIM and handles the
+device sources before completing them through the next CLAIM or the optional
+Yield ECALL. Return to a previously interrupted domain is permitted only
+after pending and claimed interrupts are drained. This completion and
+return-switch convention is specific to OpenSBI's domain courier.
+
+For example, the following node maps physical interrupt 10 to VIRQ 0 in
+channel 4 and delivers it to the boot HART of tdomain:
+
+```dts
+sysirq {
+    compatible = "opensbi,mpxy-sysirq";
+    interrupt-controller;
+    #interrupt-cells = <1>;
+    opensbi,mpxy-channel-id = <4>;
+    opensbi,domain = <&tdomain>;
+    interrupts-extended = <&aplic_m 10 4>;
+};
+```

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi

From: Raymond Mao <[email protected]>

Introduce rpmi_sysirq_intc nodes with interrupts-extended property for VIRQ testing.
- Routing rules are derived from sysirq nodes by:
interrupts-extended =
<&aplic HWIRQx IRQ_TYPE>, // virq 0
<&aplic HWIRQy IRQ_TYPE>, // virq 1
...;
- VIRQ numbers are allocated from zero, implicit from the order of the entries in the interrupts-extended property.

During cold boot, route rules are converted and cached for VIRQ routing/couriering purpose, and DT IRQ entries activate courier delivery.

hwirq_bind_domain.dts is for testing with a single domain/hart. Domain1 owns UART hwirq 10 via 'interrupts-extended'.

hwirq_bind_domain_linux_bmapp.dts is for testing with multiple domains/harts. It models two BM domains alongside the root domain on 'QEMU virt -smp 4'. Domain1 and domain2 share hart2 as boot hart and each maps its own bm-app RAM region and the UART region; domain2 owns UART hwirq 10 via 'interrupts-extended'. Bind CPU hart2 to domain1 and set explicit init-order values to keep hart2 in domain1 for initial execution. Load the BM payloads at 0x88000000 and 0x89000000 to exercise domain1 -> domain2 -> domain1.

Keep the configured courier sources 10, 20 and 21 in the M-mode APLIC and delegate the remaining sources to its S-mode child. This preserves Linux interrupt delivery without delegating the BM domains' IRQs.

Usage:
1. Compile the overlay.

   $ dtc -@ -I dts -O dtb -o <dto_file> \
         platform/generic/virt/<dts_overlay_file>
2. Dump the base dtb from QEMU via arg
   '-machine dumpdtb=<dtb_qemu_base>'
3. Apply the overlay on the base.
   $ fdtoverlay -i <dtb_qemu_base> -o <dtb_qemu_merge> <dto_file>
4. Re-run QEMU with adding argument '-dtb <dtb_qemu_merge>'

---
 platform/generic/virt/hwirq_bind_domain.dts   | 107 +++++++++++++++
 .../virt/hwirq_bind_domain_linux_bmapp.dts    | 125 ++++++++++++++++++
 2 files changed, 232 insertions(+)
 create mode 100644 platform/generic/virt/hwirq_bind_domain.dts
 create mode 100644 platform/generic/virt/hwirq_bind_domain_linux_bmapp.dts

new file mode 100644
index 00000000..93f21d0b
@@ -0,0 +1,107 @@
+// SPDX-License-Identifier: BSD-2-Clause
+/dts-v1/;
+/plugin/;
+
+/ {
+	/*
+	 * Fragment 0: OpenSBI domain configuration under /chosen
+	 */
+	fragment@0 {
+		target-path = "/chosen";
+		__overlay__ {
+			opensbi-domains {
+				compatible = "opensbi,domain,config";
+
+				/*
+				 * DRAM region for domain1 (2GB from 0x8000_0000).
+				 * base is 64-bit: <hi lo>, order is log2(size).
+				 */
+				memregion_dram_2g: memregion_dram_2g {
+					compatible = "opensbi,domain,memregion";
+					base  = <0x0 0x80000000>;
+					order = <31>;
+				};
+
+				/*
+				 * UART MMIO region so S-mode payload can access UART registers
+				 * without fault. 0x1000_0000 size 4KB.
+				 */
+				uart0_mmio: memregion_uart0 {
+					compatible = "opensbi,domain,memregion";
+					base  = <0x0 0x10000000>;
+					order = <12>;
+				};
+
+				/*
+				 * Domain instance:
+				 * - possible-harts = <cpu0 phandle> (cpu@0 phandle is 1)
+				 * - regions use label references (fdtoverlay will fixup phandles)
+				 * - IRQ routing is defined by sysirq nodes under
+				 *   /chosen/opensbi-domains
+				 */
+				domain1: domain1 {
+					compatible = "opensbi,domain,instance";
+
+					/* cpu@0 phandle in base qemu dtb is 1 */
+					possible-harts = <0x1>;
+
+					/*
+					 * Keep permissive for test.
+					 * access=0x3f: enable all M+SU R/W/X flags (test only).
+					 */
+					regions = <&memregion_dram_2g 0x3f>,
+						  <&uart0_mmio 0x3f>;
+
+					/* Route rules are supplied via sysirq nodes. */
+				};
+
+				rpmi_sysirq_intc: interrupt-controller {
+					compatible = "opensbi,mpxy-sysirq";
+					interrupt-controller;
+					#interrupt-cells = <1>;
+
+					/*
+					 * VIRQ numbers are allocated from zero, in order.
+					 * Each entry: <&irqchip hwirq flags>
+					 * flags: IRQ_TYPE_LEVEL_HIGH = 4
+					 */
+					/*
+					 * Base DTB (qemu_linux_test.dtb) phandle:
+					 * /soc/interrupt-controller@c000000 -> 0x09
+					 */
+					interrupts-extended =
+						<0x09 10 4>, /* VIRQ 0: UART RX */
+						<0x09 20 4>, /* VIRQ 1: test */
+						<0x09 21 4>; /* VIRQ 2: test */
+
+					/* MPXY channel id allocated by system designer */
+					opensbi,mpxy-channel-id = <4>;
+
+					/* Route VIRQs to domain1 */
+					opensbi,domain = <&domain1>;
+				};
+			};
+		};
+	};
+
+	/*
+	 * Fragment 1: Assign cpu0 to domain1 using label reference
+	 */
+	fragment@1 {
+		target-path = "/cpus/cpu@0";
+		__overlay__ {
+			opensbi-domain = <&domain1>;
+		};
+	};
+
+	fragment@2 {
+		target-path = "/soc/interrupt-controller@c000000";
+		__overlay__ {
+			/* Keep courier IRQs 10, 20 and 21 in the M-mode APLIC. */
+			riscv,delegate = <0x0a 1 9>, <0x0a 11 19>,
+					 <0x0a 22 96>;
+			riscv,delegation = <0x0a 1 9>, <0x0a 11 19>,
+					   <0x0a 22 96>;
+		};
+	};
+};

2.25.1


-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi