Linux RISC-V AIA Support

We add DT bindings document for the RISC-V incoming MSI controller (IMSIC) defined by the RISC-V advanced interrupt architecture (AIA) specification.

Signed-off-by Anup Patel <[email protected]>

Reviewed-by Conor Dooley <[email protected]>

Acked-by Krzysztof Kozlowski <[email protected]>

---
 .../interrupt-controller/riscv,imsics.yaml    | 172 ++++++++++++++++++
 1 file changed, 172 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,imsics.yaml

Documentation/devicetree/bindings/interrupt-controller/riscv,imsics.yaml new file +172 -0

new file mode 100644
index 000000000000..84976f17a4a1
@@ -0,0 +1,172 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/interrupt-controller/riscv,imsics.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: RISC-V Incoming MSI Controller (IMSIC)
+
+maintainers:
+  - Anup Patel <[email protected]>
+
+description: |
+  The RISC-V advanced interrupt architecture (AIA) defines a per-CPU incoming
+  MSI controller (IMSIC) for handling MSIs in a RISC-V platform. The RISC-V
+  AIA specification can be found at https://github.com/riscv/riscv-aia.
+
+  The IMSIC is a per-CPU (or per-HART) device with separate interrupt file
+  for each privilege level (machine or supervisor). The configuration of
+  a IMSIC interrupt file is done using AIA CSRs and it also has a 4KB MMIO
+  space to receive MSIs from devices. Each IMSIC interrupt file supports a
+  fixed number of interrupt identities (to distinguish MSIs from devices)
+  which is same for given privilege level across CPUs (or HARTs).
+
+  The device tree of a RISC-V platform will have one IMSIC device tree node
+  for each privilege level (machine or supervisor) which collectively describe
+  IMSIC interrupt files at that privilege level across CPUs (or HARTs).
+
+  The arrangement of IMSIC interrupt files in MMIO space of a RISC-V platform
+  follows a particular scheme defined by the RISC-V AIA specification. A IMSIC
+  group is a set of IMSIC interrupt files co-located in MMIO space and we can
+  have multiple IMSIC groups (i.e. clusters, sockets, chiplets, etc) in a
+  RISC-V platform. The MSI target address of a IMSIC interrupt file at given
+  privilege level (machine or supervisor) encodes group index, HART index,
+  and guest index (shown below).
+
+  XLEN-1            > (HART Index MSB)                  12    0
+  |                  |                                  |     |
+  -------------------------------------------------------------
+  |xxxxxx|Group Index|xxxxxxxxxxx|HART Index|Guest Index|  0  |
+  -------------------------------------------------------------
+
+allOf:
+  - $ref: /schemas/interrupt-controller.yaml#
+  - $ref: /schemas/interrupt-controller/msi-controller.yaml#
+
+properties:
+  compatible:
+    items:
+      - enum:
+          - qemu,imsics
+      - const: riscv,imsics
+
+  reg:
+    minItems: 1
+    maxItems: 16384
+    description:
+      Base address of each IMSIC group.
+
+  interrupt-controller: true
+
+  "#interrupt-cells":
+    const: 0
+
+  msi-controller: true
+
+  "#msi-cells":
+    const: 0
+
+  interrupts-extended:
+    minItems: 1
+    maxItems: 16384
+    description:
+      This property represents the set of CPUs (or HARTs) for which given
+      device tree node describes the IMSIC interrupt files. Each node pointed
+      to should be a riscv,cpu-intc node, which has a CPU node (i.e. RISC-V
+      HART) as parent.
+
+  riscv,num-ids:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    minimum: 63
+    maximum: 2047
+    description:
+      Number of interrupt identities supported by IMSIC interrupt file.
+
+  riscv,num-guest-ids:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    minimum: 63
+    maximum: 2047
+    description:
+      Number of interrupt identities are supported by IMSIC guest interrupt
+      file. When not specified it is assumed to be same as specified by the
+      riscv,num-ids property.
+
+  riscv,guest-index-bits:
+    minimum: 0
+    maximum: 7
+    default: 0
+    description:
+      Number of guest index bits in the MSI target address.
+
+  riscv,hart-index-bits:
+    minimum: 0
+    maximum: 15
+    description:
+      Number of HART index bits in the MSI target address. When not
+      specified it is calculated based on the interrupts-extended property.
+
+  riscv,group-index-bits:
+    minimum: 0
+    maximum: 7
+    default: 0
+    description:
+      Number of group index bits in the MSI target address.
+
+  riscv,group-index-shift:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    minimum: 0
+    maximum: 55
+    default: 24
+    description:
+      The least significant bit position of the group index bits in the
+      MSI target address.
+
+required:
+  - compatible
+  - reg
+  - interrupt-controller
+  - msi-controller
+  - "#msi-cells"
+  - interrupts-extended
+  - riscv,num-ids
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    // Example 1 (Machine-level IMSIC files with just one group):
+
+    interrupt-controller@24000000 {
+      compatible = "qemu,imsics", "riscv,imsics";
+      interrupts-extended = <&cpu1_intc 11>,
+                            <&cpu2_intc 11>,
+                            <&cpu3_intc 11>,
+                            <&cpu4_intc 11>;
+      reg = <0x28000000 0x4000>;
+      interrupt-controller;
+      #interrupt-cells = <0>;
+      msi-controller;
+      #msi-cells = <0>;
+      riscv,num-ids = <127>;
+    };
+
+  - |
+    // Example 2 (Supervisor-level IMSIC files with two groups):
+
+    interrupt-controller@28000000 {
+      compatible = "qemu,imsics", "riscv,imsics";
+      interrupts-extended = <&cpu1_intc 9>,
+                            <&cpu2_intc 9>,
+                            <&cpu3_intc 9>,
+                            <&cpu4_intc 9>;
+      reg = <0x28000000 0x2000>, /* Group0 IMSICs */
+            <0x29000000 0x2000>; /* Group1 IMSICs */
+      interrupt-controller;
+      #interrupt-cells = <0>;
+      msi-controller;
+      #msi-cells = <0>;
+      riscv,num-ids = <127>;
+      riscv,group-index-bits = <1>;
+      riscv,group-index-shift = <24>;
+    };
+...

Signature · 1 line

2.34.1

Hey,

Quoted 6 lines · Anup Patel

We add DT bindings document for the RISC-V incoming MSI controller (IMSIC) defined by the RISC-V advanced interrupt architecture (AIA) specification.

Signed-off-by Anup Patel <[email protected]>

Reviewed-by Conor Dooley <[email protected]>

Just FYI, since they'll reply to this themselves, but some of the Microchip folks have run into problems with sparse hart indexes while trying to use the imsic binding to describe some configurations they have. I think there were also so problems with how to describe to a linux guest which file to use, when the first hart available to the guest does not use the first file. They'll do a better job of describing their problems than I will, so I shall leave it to them!

Cheers,
Conor.

Quoted 187 lines

Acked-by Krzysztof Kozlowski <[email protected]>

---
 .../interrupt-controller/riscv,imsics.yaml    | 172 ++++++++++++++++++
 1 file changed, 172 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,imsics.yaml

Documentation/devicetree/bindings/interrupt-controller/riscv,imsics.yaml new file +172 -1

new file mode 100644
index 000000000000..84976f17a4a1
@@ -0,0 +1,172 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/interrupt-controller/riscv,imsics.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: RISC-V Incoming MSI Controller (IMSIC)
+
+maintainers:
+  - Anup Patel <[email protected]>
+
+description: |
+  The RISC-V advanced interrupt architecture (AIA) defines a per-CPU incoming
+  MSI controller (IMSIC) for handling MSIs in a RISC-V platform. The RISC-V
+  AIA specification can be found at https://github.com/riscv/riscv-aia.
+
+  The IMSIC is a per-CPU (or per-HART) device with separate interrupt file
+  for each privilege level (machine or supervisor). The configuration of
+  a IMSIC interrupt file is done using AIA CSRs and it also has a 4KB MMIO
+  space to receive MSIs from devices. Each IMSIC interrupt file supports a
+  fixed number of interrupt identities (to distinguish MSIs from devices)
+  which is same for given privilege level across CPUs (or HARTs).
+
+  The device tree of a RISC-V platform will have one IMSIC device tree node
+  for each privilege level (machine or supervisor) which collectively describe
+  IMSIC interrupt files at that privilege level across CPUs (or HARTs).
+
+  The arrangement of IMSIC interrupt files in MMIO space of a RISC-V platform
+  follows a particular scheme defined by the RISC-V AIA specification. A IMSIC
+  group is a set of IMSIC interrupt files co-located in MMIO space and we can
+  have multiple IMSIC groups (i.e. clusters, sockets, chiplets, etc) in a
+  RISC-V platform. The MSI target address of a IMSIC interrupt file at given
+  privilege level (machine or supervisor) encodes group index, HART index,
+  and guest index (shown below).
+
+  XLEN-1            > (HART Index MSB)                  12    0
+  |                  |                                  |     |
+  -------------------------------------------------------------
+  |xxxxxx|Group Index|xxxxxxxxxxx|HART Index|Guest Index|  0  |
+  -------------------------------------------------------------
+
+allOf:
+  - $ref: /schemas/interrupt-controller.yaml#
+  - $ref: /schemas/interrupt-controller/msi-controller.yaml#
+
+properties:
+  compatible:
+    items:
+      - enum:
+          - qemu,imsics
+      - const: riscv,imsics
+
+  reg:
+    minItems: 1
+    maxItems: 16384
+    description:
+      Base address of each IMSIC group.
+
+  interrupt-controller: true
+
+  "#interrupt-cells":
+    const: 0
+
+  msi-controller: true
+
+  "#msi-cells":
+    const: 0
+
+  interrupts-extended:
+    minItems: 1
+    maxItems: 16384
+    description:
+      This property represents the set of CPUs (or HARTs) for which given
+      device tree node describes the IMSIC interrupt files. Each node pointed
+      to should be a riscv,cpu-intc node, which has a CPU node (i.e. RISC-V
+      HART) as parent.
+
+  riscv,num-ids:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    minimum: 63
+    maximum: 2047
+    description:
+      Number of interrupt identities supported by IMSIC interrupt file.
+
+  riscv,num-guest-ids:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    minimum: 63
+    maximum: 2047
+    description:
+      Number of interrupt identities are supported by IMSIC guest interrupt
+      file. When not specified it is assumed to be same as specified by the
+      riscv,num-ids property.
+
+  riscv,guest-index-bits:
+    minimum: 0
+    maximum: 7
+    default: 0
+    description:
+      Number of guest index bits in the MSI target address.
+
+  riscv,hart-index-bits:
+    minimum: 0
+    maximum: 15
+    description:
+      Number of HART index bits in the MSI target address. When not
+      specified it is calculated based on the interrupts-extended property.
+
+  riscv,group-index-bits:
+    minimum: 0
+    maximum: 7
+    default: 0
+    description:
+      Number of group index bits in the MSI target address.
+
+  riscv,group-index-shift:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    minimum: 0
+    maximum: 55
+    default: 24
+    description:
+      The least significant bit position of the group index bits in the
+      MSI target address.
+
+required:
+  - compatible
+  - reg
+  - interrupt-controller
+  - msi-controller
+  - "#msi-cells"
+  - interrupts-extended
+  - riscv,num-ids
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    // Example 1 (Machine-level IMSIC files with just one group):
+
+    interrupt-controller@24000000 {
+      compatible = "qemu,imsics", "riscv,imsics";
+      interrupts-extended = <&cpu1_intc 11>,
+                            <&cpu2_intc 11>,
+                            <&cpu3_intc 11>,
+                            <&cpu4_intc 11>;
+      reg = <0x28000000 0x4000>;
+      interrupt-controller;
+      #interrupt-cells = <0>;
+      msi-controller;
+      #msi-cells = <0>;
+      riscv,num-ids = <127>;
+    };
+
+  - |
+    // Example 2 (Supervisor-level IMSIC files with two groups):
+
+    interrupt-controller@28000000 {
+      compatible = "qemu,imsics", "riscv,imsics";
+      interrupts-extended = <&cpu1_intc 9>,
+                            <&cpu2_intc 9>,
+                            <&cpu3_intc 9>,
+                            <&cpu4_intc 9>;
+      reg = <0x28000000 0x2000>, /* Group0 IMSICs */
+            <0x29000000 0x2000>; /* Group1 IMSICs */
+      interrupt-controller;
+      #interrupt-cells = <0>;
+      msi-controller;
+      #msi-cells = <0>;
+      riscv,num-ids = <127>;
+      riscv,group-index-bits = <1>;
+      riscv,group-index-shift = <24>;
+    };
+...
--

2.34.1

Quoted 8 lines · Conor Dooley

Quoted 6 lines · Anup Patel

We add DT bindings document for the RISC-V incoming MSI controller (IMSIC) defined by the RISC-V advanced interrupt architecture (AIA) specification.

Signed-off-by Anup Patel <[email protected]>

Reviewed-by Conor Dooley <[email protected]>

Quoted 7 lines

Just FYI, since they'll reply to this themselves, but some of the Microchip folks have run into problems with sparse hart indexes while trying to use the imsic binding to describe some configurations they have. I think there were also so problems with how to describe to a linux guest which file to use, when the first hart available to the guest does not use the first file. They'll do a better job of describing their problems than I will, so I shall leave it to them!

Quoting AIA spec:
"For the purpose of locating the memory pages of interrupt files in the address space, assume each hart (or each hart within a group) has a unique hart number that may or may not be related to the unique hart identifiers (“hart IDs”) that the RISC-V Privileged Architecture assigns to harts."

It is very easy to get confused between the AIA "hart index" and "hart IDs" defined by the RISC-V Privileged specification but these are two very different things. The AIA "hart index" over here is the bits in the address of an IMSIC file.

This DT binding follows the IMSIC file arrangement in the address space as defined by the section "3.6 Arrangement of the memory regions of multiple interrupt files" of the AIA specification. This arrangement is MANDATORY for platforms having both APLIC and IMSIC because in MSI-mode the APLIC generates target MSI address based the IMSIC file arrangement described in the section "3.6 Arrangement of the memory regions of multiple interrupt files". In fact, this also applies to virtual platforms created by hypervisors (KVM, Xen, ...)

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

I think this could simplify/remove(?) the locking as well.

I realize that the series in v10, and coming with a change like this now might be a bit of a pain...

Finally, another question related to APLIC/IMSIC. AFAIU the memory map of the IMSIC regions are constrained by the APLIC, which requires a certain layout for MSI forwarding (group/hart/guest bits). Say that a system doesn't have an APLIC, couldn't the layout requirement be simplified?

Again, thanks for the hard work!
Björn

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

2) All wired IRQs for APLIC MSI-mode will also target a fixed HART hence irq_set_affinity() won't work for wired IRQs as well.

3) Contemporary storage/networking drivers which use per-core queues use irq_set_affinity() on queue IRQs to balance across cores but this will fail.

4) HART hotplug breaks because kernel irq-subsystem can't migrate the IRQs (both MSIs and Wired) targeting HART X to another HART Y when the HART X goes down.

The idea of treating per-HART MSIs as separate IRQs has been discussed in the past.

Also, the current approach is very similar to the ARM GICv3 driver where ITS LPIs across CPUs are treated as single IRQ.

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Quoted 4 lines

2) All wired IRQs for APLIC MSI-mode will also target a fixed HART hence irq_set_affinity() won't work for wired IRQs as well.

I'm not following here. Why would APLIC put a constraint here? I had a look at the specs, and I didn't see anything supporting the current scheme explicitly.

Quoted 4 lines

3) Contemporary storage/networking drivers which use per-core queues use irq_set_affinity() on queue IRQs to balance across cores but this will fail.

Or via the the managed interrupts. But this is a non-issue, as pointed out in my reply to 1.

Quoted 3 lines

4) HART hotplug breaks because kernel irq-subsystem can't migrate the IRQs (both MSIs and Wired) targeting HART X to another HART Y when the HART X goes down.

Yes, we might end up in scenarios where we can't move to a certain target cpu, but I wouldn't expect that to be a common scenario.

Quoted 2 lines

The idea of treating per-HART MSIs as separate IRQs has been discussed in the past.

Aha! I tried to look for it in lore, but didn't find any. Could you point me to those discussions?

Quoted 3 lines

Also, the current approach is very similar to the ARM GICv3 driver where ITS LPIs across CPUs are treated as single IRQ.

I'm not familiar with the GIC. Is the GICv3 design similar to IMSIC? I had the impression that the GIC had a more advanced interrupt routing mechanism, than what IMSIC exposes. I think x86 APIC takes the 1-1 approach (the folks on the To: list definitely knows! ;-)).

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Quoted 4 lines

Quoted 3 lines

2) All wired IRQs for APLIC MSI-mode will also target a fixed HART hence irq_set_affinity() won't work for wired IRQs as well.

Quoted 3 lines

I'm not following here. Why would APLIC put a constraint here? I had a look at the specs, and I didn't see anything supporting the current scheme explicitly.

Lets say the number of APLIC wired interrupts are greater than the number of per-CPU IMSIC IDs. In this case, if all wired interrupts are moved to a particular CPU then irq_set_affinity() will fail for some of the wired interrupts.

Quoted 5 lines

Quoted 2 lines

The idea of treating per-HART MSIs as separate IRQs has been discussed in the past.

Aha! I tried to look for it in lore, but didn't find any. Could you point me to those discussions?

This was done 2 years back in the AIA TG meeting when we were doing the PoC for AIA spec.

Quoted 5 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

Quoted 3 lines

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

2) All wired IRQs for APLIC MSI-mode will also target a fixed HART hence irq_set_affinity() won't work for wired IRQs as well.

Quoted 8 lines

Quoted 3 lines

I'm not following here. Why would APLIC put a constraint here? I had a look at the specs, and I didn't see anything supporting the current scheme explicitly.

Lets say the number of APLIC wired interrupts are greater than the number of per-CPU IMSIC IDs. In this case, if all wired interrupts are moved to a particular CPU then irq_set_affinity() will fail for some of the wired interrupts.

Right, it's the case of "full remote CPU" again. Thanks for clearing that up.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

The idea of treating per-HART MSIs as separate IRQs has been discussed in the past.

Aha! I tried to look for it in lore, but didn't find any. Could you point me to those discussions?

This was done 2 years back in the AIA TG meeting when we were doing the PoC for AIA spec.

Ah, too bad. Thanks regardless.

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 6 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

Quoted 2 lines

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

In the RISC-V world, we can easily define a small fast track extension based on S*csrind extension which can allow a large number of IMSIC IDs per-core.

Instead of addressing problems on a hypothetical system, I suggest we go ahead with the current approach and deal with a system having MSI over-subscription when such a system shows up.

Quoted 98 lines · Anup Patel

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 6 lines

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

I'm aware. We're on the same page here.

Quoted 9 lines

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Yes, the 1-1 approach will require an IPI to one target cpu on affinity changes, and similar on mask/unmask.

The 1-n approach, require no-IPI on affinity changes (nice!), but IPI broadcast to all cores on mask/unmask (not so nice).

Quoted 17 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 5 lines

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

You are misreading. A 128c system with 2047 MSIs per-core, will only have 16 *per-core unique* (2047/128) interrupts with the current series.

I'm not saying that each IMSIC has 16 IDs, I'm saying that in a 128c system with the maximum amount of MSIs possible in the spec, you'll end up with 16 *unique* interrupts per core.

Quoted 6 lines

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

That's one sample point, and clearly not the only one. There are *many* different usage models. Just because you *assign* MSI, doesn't mean they are firing all the time.

I can show you a couple of networking setups where this is clearly not enough. Each core has a large number of QoS queues, and each queue would very much like to have a dedicated MSI.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 5 lines

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

Again, very workload dependent.

This series does IPI broadcast on masking/unmasking, which means that cores that don't care get interrupted because, say, a network queue-pair is setup on another core.

Some workloads never change the irq affinity.

I'm just pointing out that there are pro/cons with both variants.

Quoted 2 lines

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

...and I'm arguing that we're paying a big price for that.

Quoted 3 lines

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

Yeah, but this is not the GIC. This is something that looks more like the x86 world. We'll be stuck with a lot of implementations with AIA 1.0 spec, and many cores.

Quoted 9 lines

In the RISC-V world, we can easily define a small fast track extension based on S*csrind extension which can allow a large number of IMSIC IDs per-core.

Instead of addressing problems on a hypothetical system, I suggest we go ahead with the current approach and deal with a system having MSI over-subscription when such a system shows up.

I've pointed out my concerns. We're not agreeing, but hey, I'm just one sample point here! I'll leave it here for others to chime in!

Still much appreciate all the hard work on the series!

Have a nice weekend,

Quoted 149 lines · Björn Töpel

Quoted 98 lines · Anup Patel

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 6 lines

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

I'm aware. We're on the same page here.

Quoted 9 lines

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Yes, the 1-1 approach will require an IPI to one target cpu on affinity changes, and similar on mask/unmask.

The 1-n approach, require no-IPI on affinity changes (nice!), but IPI broadcast to all cores on mask/unmask (not so nice).

Quoted 17 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 5 lines

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

Quoted 6 lines

You are misreading. A 128c system with 2047 MSIs per-core, will only have 16 *per-core unique* (2047/128) interrupts with the current series.

I'm not saying that each IMSIC has 16 IDs, I'm saying that in a 128c system with the maximum amount of MSIs possible in the spec, you'll end up with 16 *unique* interrupts per core.

-ENOPARSE

I don't see how this applies to the current approach because we treat MSI ID space as global across cores so if a system has 2047 MSIs per-core then we have 2047 MSIs across all cores.

Quoted 40 lines

Quoted 7 lines

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

That's one sample point, and clearly not the only one. There are *many* different usage models. Just because you *assign* MSI, doesn't mean they are firing all the time.

I can show you a couple of networking setups where this is clearly not enough. Each core has a large number of QoS queues, and each queue would very much like to have a dedicated MSI.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 5 lines

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

Again, very workload dependent.

This series does IPI broadcast on masking/unmasking, which means that cores that don't care get interrupted because, say, a network queue-pair is setup on another core.

Quoted 1 line

Some workloads never change the irq affinity.

There are various events which irq affinity such as irq balance, CPU hotplug, system suspend, etc.

Also, the 1-1 approach does IPI upon set_affinity, mask and unmask whereas the 1-n approach does IPI only upon mask and unmask.

Quoted 11 lines

I'm just pointing out that there are pro/cons with both variants.

Quoted 2 lines

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

...and I'm arguing that we're paying a big price for that.

Quoted 3 lines

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

Quoted 3 lines

Yeah, but this is not the GIC. This is something that looks more like the x86 world. We'll be stuck with a lot of implementations with AIA 1.0 spec, and many cores.

Well, RISC-V AIA is neigher ARM GIG not x86 APIC. All I am saying is that there are systems with large number per-core interrupt IDs for handling MSIs.

Quoted 157 lines · Anup Patel

Quoted 149 lines · Björn Töpel

Quoted 98 lines · Anup Patel

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 6 lines

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

I'm aware. We're on the same page here.

Quoted 9 lines

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Yes, the 1-1 approach will require an IPI to one target cpu on affinity changes, and similar on mask/unmask.

The 1-n approach, require no-IPI on affinity changes (nice!), but IPI broadcast to all cores on mask/unmask (not so nice).

Quoted 17 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 5 lines

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

Quoted 6 lines

You are misreading. A 128c system with 2047 MSIs per-core, will only have 16 *per-core unique* (2047/128) interrupts with the current series.

I'm not saying that each IMSIC has 16 IDs, I'm saying that in a 128c system with the maximum amount of MSIs possible in the spec, you'll end up with 16 *unique* interrupts per core.

Quoted 5 lines

-ENOPARSE

I don't see how this applies to the current approach because we treat MSI ID space as global across cores so if a system has 2047 MSIs per-core then we have 2047 MSIs across all cores.

Ok, I'll try again! :-)

Let's assume that each core in the 128c system has some per-core resources, say a two NIC queue pairs, and a storage queue pair. This will consume, e.g., 2*2 + 2 (6) MSI sources from the global namespace.

If each core does this it'll be 6*128 MSI sources of the global namespace.

The maximum number of "privates" MSI sources a core can utilize is 16.

I'm trying (it's does seem to go that well ;-)) to point out that it's only 16 unique sources per core. For, say, a 256 core system it would be
8. 2047 MSI sources in a system is not much.

Say that I want to spin up 24 NIC queues with one MSI each on each core on my 128c system. That's not possible with this series, while with an 1-1 system it wouldn't be an issue.

Clearer, or still weird?

Quoted 40 lines

Quoted 40 lines

Quoted 7 lines

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

That's one sample point, and clearly not the only one. There are *many* different usage models. Just because you *assign* MSI, doesn't mean they are firing all the time.

I can show you a couple of networking setups where this is clearly not enough. Each core has a large number of QoS queues, and each queue would very much like to have a dedicated MSI.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 5 lines

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

Again, very workload dependent.

This series does IPI broadcast on masking/unmasking, which means that cores that don't care get interrupted because, say, a network queue-pair is setup on another core.

Quoted 8 lines

Quoted 1 line

Some workloads never change the irq affinity.

There are various events which irq affinity such as irq balance, CPU hotplug, system suspend, etc.

Also, the 1-1 approach does IPI upon set_affinity, mask and unmask whereas the 1-n approach does IPI only upon mask and unmask.

An important distinction; When you say IPI on mask/unmask it is a broadcast IPI to *all* cores, which is pretty instrusive.

The 1-1 variant does an IPI to a *one* target core.

Quoted 15 lines

Quoted 14 lines

I'm just pointing out that there are pro/cons with both variants.

Quoted 2 lines

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

...and I'm arguing that we're paying a big price for that.

Quoted 3 lines

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

Yeah, but this is not the GIC. This is something that looks more like the x86 world. We'll be stuck with a lot of implementations with AIA 1.0 spec, and many cores.

Quoted 3 lines

Well, RISC-V AIA is neigher ARM GIG not x86 APIC. All I am saying is that there are systems with large number per-core interrupt IDs for handling MSIs.

Yes, and while that is nice, it's not what IMSIC is.

Quoted 259 lines · Björn Töpel

Quoted 157 lines · Anup Patel

Quoted 149 lines · Björn Töpel

Quoted 98 lines · Anup Patel

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 6 lines

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

I'm aware. We're on the same page here.

Quoted 9 lines

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Yes, the 1-1 approach will require an IPI to one target cpu on affinity changes, and similar on mask/unmask.

The 1-n approach, require no-IPI on affinity changes (nice!), but IPI broadcast to all cores on mask/unmask (not so nice).

Quoted 17 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 5 lines

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

Quoted 6 lines

You are misreading. A 128c system with 2047 MSIs per-core, will only have 16 *per-core unique* (2047/128) interrupts with the current series.

I'm not saying that each IMSIC has 16 IDs, I'm saying that in a 128c system with the maximum amount of MSIs possible in the spec, you'll end up with 16 *unique* interrupts per core.

Quoted 5 lines

-ENOPARSE

I don't see how this applies to the current approach because we treat MSI ID space as global across cores so if a system has 2047 MSIs per-core then we have 2047 MSIs across all cores.

Ok, I'll try again! :-)

Let's assume that each core in the 128c system has some per-core resources, say a two NIC queue pairs, and a storage queue pair. This will consume, e.g., 2*2 + 2 (6) MSI sources from the global namespace.

If each core does this it'll be 6*128 MSI sources of the global namespace.

The maximum number of "privates" MSI sources a core can utilize is 16.

I'm trying (it's does seem to go that well ;-)) to point out that it's only 16 unique sources per core. For, say, a 256 core system it would be
8. 2047 MSI sources in a system is not much.

Say that I want to spin up 24 NIC queues with one MSI each on each core on my 128c system. That's not possible with this series, while with an 1-1 system it wouldn't be an issue.

Clearer, or still weird?

Quoted 40 lines

Quoted 40 lines

Quoted 7 lines

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

That's one sample point, and clearly not the only one. There are *many* different usage models. Just because you *assign* MSI, doesn't mean they are firing all the time.

I can show you a couple of networking setups where this is clearly not enough. Each core has a large number of QoS queues, and each queue would very much like to have a dedicated MSI.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 5 lines

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

Again, very workload dependent.

This series does IPI broadcast on masking/unmasking, which means that cores that don't care get interrupted because, say, a network queue-pair is setup on another core.

Quoted 8 lines

Quoted 1 line

Some workloads never change the irq affinity.

There are various events which irq affinity such as irq balance, CPU hotplug, system suspend, etc.

Also, the 1-1 approach does IPI upon set_affinity, mask and unmask whereas the 1-n approach does IPI only upon mask and unmask.

An important distinction; When you say IPI on mask/unmask it is a broadcast IPI to *all* cores, which is pretty instrusive.

The 1-1 variant does an IPI to a *one* target core.

Quoted 18 lines

Quoted 7 lines

I'm just pointing out that there are pro/cons with both variants.

Quoted 2 lines

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

...and I'm arguing that we're paying a big price for that.

Quoted 7 lines

Quoted 3 lines

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

Yeah, but this is not the GIC. This is something that looks more like the x86 world. We'll be stuck with a lot of implementations with AIA 1.0 spec, and many cores.

Well, RISC-V AIA is neigher ARM GIG not x86 APIC. All I am saying is that there are systems with large number per-core interrupt IDs for handling MSIs.

Quoted 1 line

Yes, and while that is nice, it's not what IMSIC is.

Some follow-ups, after thinking more about it more over the weekend.

* Do one really need an IPI for irq_set_affinity() for the 1-1 model? Why touch the enable/disable bits when moving interrupts?

* In my book the IMSIC looks very much like the x86 LAPIC, which also has few interrupts (IMSIC <2048, LAPIC 256). The IRQ matrix allocator [1], and a scheme similar to LAPIC [2] would be a good fit. This is the 1-1 model, but more sophisticated than what I've been describing (e.g. properly handling mangaged/regular irqs). As a bonus we would get the IRQ matrix debugfs/tracepoint support.

Quoted 266 lines · Björn Töpel

Quoted 255 lines · Björn Töpel

Quoted 157 lines · Anup Patel

Quoted 149 lines · Björn Töpel

Quoted 98 lines · Anup Patel

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 4 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

Quoted 11 lines

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 6 lines

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

I'm aware. We're on the same page here.

Quoted 9 lines

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Yes, the 1-1 approach will require an IPI to one target cpu on affinity changes, and similar on mask/unmask.

The 1-n approach, require no-IPI on affinity changes (nice!), but IPI broadcast to all cores on mask/unmask (not so nice).

Quoted 17 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 5 lines

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

Quoted 6 lines

You are misreading. A 128c system with 2047 MSIs per-core, will only have 16 *per-core unique* (2047/128) interrupts with the current series.

I'm not saying that each IMSIC has 16 IDs, I'm saying that in a 128c system with the maximum amount of MSIs possible in the spec, you'll end up with 16 *unique* interrupts per core.

Quoted 5 lines

-ENOPARSE

I don't see how this applies to the current approach because we treat MSI ID space as global across cores so if a system has 2047 MSIs per-core then we have 2047 MSIs across all cores.

Ok, I'll try again! :-)

Let's assume that each core in the 128c system has some per-core resources, say a two NIC queue pairs, and a storage queue pair. This will consume, e.g., 2*2 + 2 (6) MSI sources from the global namespace.

If each core does this it'll be 6*128 MSI sources of the global namespace.

The maximum number of "privates" MSI sources a core can utilize is 16.

I'm trying (it's does seem to go that well ;-)) to point out that it's only 16 unique sources per core. For, say, a 256 core system it would be
8. 2047 MSI sources in a system is not much.

Say that I want to spin up 24 NIC queues with one MSI each on each core on my 128c system. That's not possible with this series, while with an 1-1 system it wouldn't be an issue.

Clearer, or still weird?

Quoted 40 lines

Quoted 40 lines

Quoted 7 lines

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

That's one sample point, and clearly not the only one. There are *many* different usage models. Just because you *assign* MSI, doesn't mean they are firing all the time.

I can show you a couple of networking setups where this is clearly not enough. Each core has a large number of QoS queues, and each queue would very much like to have a dedicated MSI.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 5 lines

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

Again, very workload dependent.

This series does IPI broadcast on masking/unmasking, which means that cores that don't care get interrupted because, say, a network queue-pair is setup on another core.

Quoted 8 lines

Quoted 1 line

Some workloads never change the irq affinity.

There are various events which irq affinity such as irq balance, CPU hotplug, system suspend, etc.

Also, the 1-1 approach does IPI upon set_affinity, mask and unmask whereas the 1-n approach does IPI only upon mask and unmask.

An important distinction; When you say IPI on mask/unmask it is a broadcast IPI to *all* cores, which is pretty instrusive.

The 1-1 variant does an IPI to a *one* target core.

Quoted 15 lines

Quoted 14 lines

I'm just pointing out that there are pro/cons with both variants.

Quoted 2 lines

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

...and I'm arguing that we're paying a big price for that.

Quoted 3 lines

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

Yeah, but this is not the GIC. This is something that looks more like the x86 world. We'll be stuck with a lot of implementations with AIA 1.0 spec, and many cores.

Quoted 5 lines

Quoted 3 lines

Well, RISC-V AIA is neigher ARM GIG not x86 APIC. All I am saying is that there are systems with large number per-core interrupt IDs for handling MSIs.

Yes, and while that is nice, it's not what IMSIC is.

Some follow-ups, after thinking more about it more over the weekend.

Quoted 2 lines

* Do one really need an IPI for irq_set_affinity() for the 1-1 model? Why touch the enable/disable bits when moving interrupts?

In the 1-1 model, the ID on the current HART and target HART upon irq_set_affinity will be different so we can't leave the unused ID on current HART enabled because it can lead to spurious interrupts when the ID on current HART is re-used for some other device.

There is also a possibility of receiving an interrupt while the ID was moved to a new target HART in-which case we have to detect and re-trigger interrupt on the new target HART. In fact, x86 APLIC does an IPI to take care of this case.

Quoted 8 lines

* In my book the IMSIC looks very much like the x86 LAPIC, which also has few interrupts (IMSIC <2048, LAPIC 256). The IRQ matrix allocator [1], and a scheme similar to LAPIC [2] would be a good fit. This is the 1-1 model, but more sophisticated than what I've been describing (e.g. properly handling mangaged/regular irqs). As a bonus we would get the IRQ matrix debugfs/tracepoint support.

Yes, I have been evaluating the 1-1 model for the past few days. I also have a working implementation with a simple per-CPU bitmap based allocator which handles both legacy MSI (block of 1,2,4,8,16, or 32 IDs) and MSI-X.

The irq matrix allocator needs to be improved for handling legacy MSI so initially I will post a v11 series which works for me and converging with irq matrix allocator can be future work.

Quoted 270 lines · Anup Patel

Quoted 268 lines · Björn Töpel

Quoted 255 lines · Björn Töpel

Quoted 157 lines · Anup Patel

Quoted 149 lines · Björn Töpel

Quoted 98 lines · Anup Patel

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Quoted 15 lines · Anup Patel

The RISC-V AIA specification is ratified as-per the RISC-V international
process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and each MSI is dedicated to a certain hart. The series takes the approach to say that there are, e.g., 2047 interrupts ("lnx-irq") globally. Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt sources on "other" harts are pre-allocated. On the other hand it requires to propagate irq masking to other harts via IPIs (this is mostly done up setup/tear down). It's also wasteful, because msi-irqs are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core (or a sub-set of cores). The current scheme wastes a lot of msi-irqs. If we instead used a scheme where "msi-irq == lnx-irq", instead of "lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device would like to use 5 queues (5 cores) on a 128 core system, the current scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum 2047 MSIs for all cores (whole system). A better scheme, IMO, would be to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 6 lines

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

I'm aware. We're on the same page here.

Quoted 9 lines

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Yes, the 1-1 approach will require an IPI to one target cpu on affinity changes, and similar on mask/unmask.

The 1-n approach, require no-IPI on affinity changes (nice!), but IPI broadcast to all cores on mask/unmask (not so nice).

Quoted 17 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 5 lines

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

Quoted 6 lines

You are misreading. A 128c system with 2047 MSIs per-core, will only have 16 *per-core unique* (2047/128) interrupts with the current series.

I'm not saying that each IMSIC has 16 IDs, I'm saying that in a 128c system with the maximum amount of MSIs possible in the spec, you'll end up with 16 *unique* interrupts per core.

Quoted 5 lines

-ENOPARSE

I don't see how this applies to the current approach because we treat MSI ID space as global across cores so if a system has 2047 MSIs per-core then we have 2047 MSIs across all cores.

Ok, I'll try again! :-)

Let's assume that each core in the 128c system has some per-core resources, say a two NIC queue pairs, and a storage queue pair. This will consume, e.g., 2*2 + 2 (6) MSI sources from the global namespace.

If each core does this it'll be 6*128 MSI sources of the global namespace.

The maximum number of "privates" MSI sources a core can utilize is 16.

I'm trying (it's does seem to go that well ;-)) to point out that it's only 16 unique sources per core. For, say, a 256 core system it would be
8. 2047 MSI sources in a system is not much.

Say that I want to spin up 24 NIC queues with one MSI each on each core on my 128c system. That's not possible with this series, while with an 1-1 system it wouldn't be an issue.

Clearer, or still weird?

Quoted 40 lines

Quoted 40 lines

Quoted 7 lines

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

That's one sample point, and clearly not the only one. There are *many* different usage models. Just because you *assign* MSI, doesn't mean they are firing all the time.

I can show you a couple of networking setups where this is clearly not enough. Each core has a large number of QoS queues, and each queue would very much like to have a dedicated MSI.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 5 lines

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

Again, very workload dependent.

This series does IPI broadcast on masking/unmasking, which means that cores that don't care get interrupted because, say, a network queue-pair is setup on another core.

Quoted 8 lines

Quoted 1 line

Some workloads never change the irq affinity.

There are various events which irq affinity such as irq balance, CPU hotplug, system suspend, etc.

Also, the 1-1 approach does IPI upon set_affinity, mask and unmask whereas the 1-n approach does IPI only upon mask and unmask.

An important distinction; When you say IPI on mask/unmask it is a broadcast IPI to *all* cores, which is pretty instrusive.

The 1-1 variant does an IPI to a *one* target core.

Quoted 15 lines

Quoted 14 lines

I'm just pointing out that there are pro/cons with both variants.

Quoted 2 lines

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

...and I'm arguing that we're paying a big price for that.

Quoted 3 lines

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

Yeah, but this is not the GIC. This is something that looks more like the x86 world. We'll be stuck with a lot of implementations with AIA 1.0 spec, and many cores.

Quoted 5 lines

Quoted 3 lines

Well, RISC-V AIA is neigher ARM GIG not x86 APIC. All I am saying is that there are systems with large number per-core interrupt IDs for handling MSIs.

Yes, and while that is nice, it's not what IMSIC is.

Some follow-ups, after thinking more about it more over the weekend.

* Do one really need an IPI for irq_set_affinity() for the 1-1 model? Why touch the enable/disable bits when moving interrupts?

Quoted 4 lines

In the 1-1 model, the ID on the current HART and target HART upon irq_set_affinity will be different so we can't leave the unused ID on current HART enabled because it can lead to spurious interrupts when the ID on current HART is re-used for some other device.

Hmm, is this really an actual problem, or a theoretical one? The implementation need to track what's in-use, so can we ever get into this situation?

Somewhat related; I had a similar question for imsic_pci_{un,}mask_irq() -- why not only do the the default mask operation (only pci_msi_{un,}mask_irq()), but instead propagate to the IMSIC mask/unmask?

Quoted 5 lines

There is also a possibility of receiving an interrupt while the ID was moved to a new target HART in-which case we have to detect and re-trigger interrupt on the new target HART. In fact, x86 APLIC does an IPI to take care of this case.

This case I get, and the implementation can track that both are in use. It's the spurious one that I'm dubious of (don't get).

Quoted 14 lines

Quoted 8 lines

* In my book the IMSIC looks very much like the x86 LAPIC, which also has few interrupts (IMSIC <2048, LAPIC 256). The IRQ matrix allocator [1], and a scheme similar to LAPIC [2] would be a good fit. This is the 1-1 model, but more sophisticated than what I've been describing (e.g. properly handling mangaged/regular irqs). As a bonus we would get the IRQ matrix debugfs/tracepoint support.

Yes, I have been evaluating the 1-1 model for the past few days. I also have a working implementation with a simple per-CPU bitmap based allocator which handles both legacy MSI (block of 1,2,4,8,16, or 32 IDs) and MSI-X.

Quoted 3 lines

The irq matrix allocator needs to be improved for handling legacy MSI so initially I will post a v11 series which works for me and converging with irq matrix allocator can be future work.

What's missing/needs to be improved for legacy MSI (legacy MSI == !MSI-X, right?) in the matrix allocator?

Quoted 278 lines · Björn Töpel

Quoted 274 lines · Anup Patel

Quoted 262 lines · Björn Töpel

Quoted 259 lines · Björn Töpel

Quoted 157 lines · Anup Patel

Quoted 149 lines · Björn Töpel

Quoted 98 lines · Anup Patel

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 62 lines · Anup Patel

Quoted 61 lines · Björn Töpel

Hi Anup,

Anup Patel <[email protected]> writes:

The RISC-V AIA specification is ratified as-per the RISC-V international
process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and
have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and
each MSI is dedicated to a certain hart. The series takes the approach
to say that there are, e.g., 2047 interrupts ("lnx-irq") globally.
Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the
slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt
sources on "other" harts are pre-allocated. On the other hand it
requires to propagate irq masking to other harts via IPIs (this is
mostly done up setup/tear down). It's also wasteful, because msi-irqs
are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core
(or a sub-set of cores). The current scheme wastes a lot of msi-irqs.
If we instead used a scheme where "msi-irq == lnx-irq", instead of
"lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be
a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device
would like to use 5 queues (5 cores) on a 128 core system, the current
scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts
on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of
interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no
broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum
2047 MSIs for all cores (whole system). A better scheme, IMO, would be
to expose 2047 * #harts unique MSIs.

Quoted 6 lines

Quoted 1 line

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each IRQ is not tied to a particular HART. This means we can't balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 6 lines

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

I'm aware. We're on the same page here.

Quoted 9 lines

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Yes, the 1-1 approach will require an IPI to one target cpu on affinity changes, and similar on mask/unmask.

The 1-n approach, require no-IPI on affinity changes (nice!), but IPI broadcast to all cores on mask/unmask (not so nice).

Quoted 17 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 5 lines

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

Quoted 6 lines

You are misreading. A 128c system with 2047 MSIs per-core, will only have 16 *per-core unique* (2047/128) interrupts with the current series.

I'm not saying that each IMSIC has 16 IDs, I'm saying that in a 128c system with the maximum amount of MSIs possible in the spec, you'll end up with 16 *unique* interrupts per core.

Quoted 5 lines

-ENOPARSE

I don't see how this applies to the current approach because we treat MSI ID space as global across cores so if a system has 2047 MSIs per-core then we have 2047 MSIs across all cores.

Ok, I'll try again! :-)

Let's assume that each core in the 128c system has some per-core resources, say a two NIC queue pairs, and a storage queue pair. This will consume, e.g., 2*2 + 2 (6) MSI sources from the global namespace.

If each core does this it'll be 6*128 MSI sources of the global namespace.

The maximum number of "privates" MSI sources a core can utilize is 16.

I'm trying (it's does seem to go that well ;-)) to point out that it's only 16 unique sources per core. For, say, a 256 core system it would be
8. 2047 MSI sources in a system is not much.

Say that I want to spin up 24 NIC queues with one MSI each on each core on my 128c system. That's not possible with this series, while with an 1-1 system it wouldn't be an issue.

Clearer, or still weird?

Quoted 40 lines

Quoted 40 lines

Quoted 7 lines

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

That's one sample point, and clearly not the only one. There are *many* different usage models. Just because you *assign* MSI, doesn't mean they are firing all the time.

I can show you a couple of networking setups where this is clearly not enough. Each core has a large number of QoS queues, and each queue would very much like to have a dedicated MSI.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 5 lines

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

Again, very workload dependent.

This series does IPI broadcast on masking/unmasking, which means that cores that don't care get interrupted because, say, a network queue-pair is setup on another core.

Quoted 8 lines

Quoted 1 line

Some workloads never change the irq affinity.

There are various events which irq affinity such as irq balance, CPU hotplug, system suspend, etc.

Also, the 1-1 approach does IPI upon set_affinity, mask and unmask whereas the 1-n approach does IPI only upon mask and unmask.

An important distinction; When you say IPI on mask/unmask it is a broadcast IPI to *all* cores, which is pretty instrusive.

The 1-1 variant does an IPI to a *one* target core.

Quoted 18 lines

Quoted 7 lines

I'm just pointing out that there are pro/cons with both variants.

Quoted 2 lines

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

...and I'm arguing that we're paying a big price for that.

Quoted 7 lines

Quoted 3 lines

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

Yeah, but this is not the GIC. This is something that looks more like the x86 world. We'll be stuck with a lot of implementations with AIA 1.0 spec, and many cores.

Well, RISC-V AIA is neigher ARM GIG not x86 APIC. All I am saying is that there are systems with large number per-core interrupt IDs for handling MSIs.

Quoted 6 lines

Quoted 1 line

Yes, and while that is nice, it's not what IMSIC is.

Some follow-ups, after thinking more about it more over the weekend.

* Do one really need an IPI for irq_set_affinity() for the 1-1 model? Why touch the enable/disable bits when moving interrupts?

In the 1-1 model, the ID on the current HART and target HART upon irq_set_affinity will be different so we can't leave the unused ID on current HART enabled because it can lead to spurious interrupts when the ID on current HART is re-used for some other device.

Quoted 3 lines

Hmm, is this really an actual problem, or a theoretical one? The implementation need to track what's in-use, so can we ever get into this situation?

As of now, it is theoretical but it is certainly possible to hit this issue.

Quoted 4 lines

Somewhat related; I had a similar question for imsic_pci_{un,}mask_irq() -- why not only do the the default mask operation (only pci_msi_{un,}mask_irq()), but instead propagate to the IMSIC mask/unmask?

We have hierarchical IMSIC PCI irq domain whoes parent irq domain is IMSIC base domain. Unfortunately pci_msi_[un]mask_irq() don't work for hierarchical irq domain.

Quoted 26 lines

Quoted 4 lines

There is also a possibility of receiving an interrupt while the ID was moved to a new target HART in-which case we have to detect and re-trigger interrupt on the new target HART. In fact, x86 APLIC does an IPI to take care of this case.

This case I get, and the implementation can track that both are in use. It's the spurious one that I'm dubious of (don't get).

Quoted 17 lines

Quoted 8 lines

* In my book the IMSIC looks very much like the x86 LAPIC, which also has few interrupts (IMSIC <2048, LAPIC 256). The IRQ matrix allocator [1], and a scheme similar to LAPIC [2] would be a good fit. This is the 1-1 model, but more sophisticated than what I've been describing (e.g. properly handling mangaged/regular irqs). As a bonus we would get the IRQ matrix debugfs/tracepoint support.

Yes, I have been evaluating the 1-1 model for the past few days. I also have a working implementation with a simple per-CPU bitmap based allocator which handles both legacy MSI (block of 1,2,4,8,16, or 32 IDs) and MSI-X.

The irq matrix allocator needs to be improved for handling legacy MSI so initially I will post a v11 series which works for me and converging with irq matrix allocator can be future work.

Quoted 2 lines

What's missing/needs to be improved for legacy MSI (legacy MSI == !MSI-X, right?) in the matrix allocator?

For legacy MSI, a block IDs needs to be contiguous and the number of IDs can be a power of 2.

Quoted 283 lines · Anup Patel

Quoted 281 lines · Björn Töpel

Quoted 267 lines · Anup Patel

Quoted 266 lines · Björn Töpel

Quoted 255 lines · Björn Töpel

Quoted 157 lines · Anup Patel

Quoted 149 lines · Björn Töpel

Quoted 98 lines · Anup Patel

Quoted 97 lines · Björn Töpel

Quoted 84 lines · Anup Patel

Quoted 83 lines · Björn Töpel

Thanks for the quick reply!

Quoted 68 lines · Anup Patel

On Thu, Oct 19, 2023 at 7:13 PM Björn Töpel <[email protected]> wrote:

Hi Anup,

Anup Patel <[email protected]> writes:

The RISC-V AIA specification is ratified as-per the RISC-V international
process. The latest ratified AIA specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf

At a high-level, the AIA specification adds three things:
1) AIA CSRs
- Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
- Per-HART MSI controller
- Support MSI virtualization
- Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
- Wired interrupt controller
- In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
- In Direct-mode, injects external interrupts directly into HARTs

Thanks for working on the AIA support! I had a look at the series, and
have some concerns about interrupt ID abstraction.

A bit of background, for readers not familiar with the AIA details.

IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and
each MSI is dedicated to a certain hart. The series takes the approach
to say that there are, e.g., 2047 interrupts ("lnx-irq") globally.
Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the
slice only *one* msi-irq is acutally used.

This scheme makes affinity changes more robust, because the interrupt
sources on "other" harts are pre-allocated. On the other hand it
requires to propagate irq masking to other harts via IPIs (this is
mostly done up setup/tear down). It's also wasteful, because msi-irqs
are hogged, and cannot be used.

Contemporary storage/networking drivers usually uses queues per core
(or a sub-set of cores). The current scheme wastes a lot of msi-irqs.
If we instead used a scheme where "msi-irq == lnx-irq", instead of
"lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be
a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device
would like to use 5 queues (5 cores) on a 128 core system, the current
scheme would consume 5 * 128 MSIs, instead of just 5.

On the plus side:
* Changing interrupts affinity will never fail, because the interrupts
on each hart is pre-allocated.

On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of
interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no
broadcast csr in IMSIC), and a more complex locking scheme IMSIC

Summary:
The current series caps the number of global interrupts to maximum
2047 MSIs for all cores (whole system). A better scheme, IMO, would be
to expose 2047 * #harts unique MSIs.

I think this could simplify/remove(?) the locking as well.

Exposing 2047 * #harts unique MSIs has multiple issues:
1) The irq_set_affinity() does not work for MSIs because each
IRQ is not tied to a particular HART. This means we can't
balance the IRQ processing load among HARTs.

Yes, you can balance. In your code, each *active* MSI is still bound/active to a specific hard together with the affinity mask. In an 1-1 model you would still need to track the affinity mask, but the irq_set_affinity() would be different. It would try to allocate a new MSI from the target CPU, and then switch to having that MSI active.

That's what x86 does AFAIU, which is also constrained by the # of available MSIs.

Quoted 6 lines

Quoted 2 lines

The downside, as I pointed out, is that the set affinity action can fail for a certain target CPU.

Yes, irq_set_affinity() can fail for the suggested approach plus for RISC-V AIA, one HART does not have access to other HARTs MSI enable/disable bits so the approach will also involve IPI.

Correct, but the current series does a broadcast to all cores, where the 1-1 approach is at most an IPI to a single core.

Quoted 6 lines

Quoted 3 lines

128+c machines are getting more common, and you have devices that you bring up/down on a per-core basis. Broadcasting IPIs to all cores, when dealing with a per-core activity is a pretty noisy neighbor.

Broadcast IPI in the current approach is only done upon MSI mask/unmask operation. It is not done upon set_affinity() of interrupt handling.

I'm aware. We're on the same page here.

Quoted 9 lines

Quoted 3 lines

This could be fixed in the existing 1-n approach, by not require to sync the cores that are not handling the MSI in question. "Lazy disable"

Incorrect. The approach you are suggesting involves an IPI upon every irq_set_affinity(). This is because a HART can only enable it's own MSI ID so when an IRQ is moved to from HART A to HART B with a different ID X on HART B then we will need an IPI in irq_set_affinit() to enable ID X on HART B.

Yes, the 1-1 approach will require an IPI to one target cpu on affinity changes, and similar on mask/unmask.

The 1-n approach, require no-IPI on affinity changes (nice!), but IPI broadcast to all cores on mask/unmask (not so nice).

Quoted 17 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

My concern is interrupts become a scarce resource with this implementation, but maybe my view is incorrect. I've seen bare-metal x86 systems (no VMs) with ~200 cores, and ~2000 interrupts, but maybe that is considered "a lot of interrupts".

As long as we don't get into scenarios where we're running out of interrupts, due to the software design.

Quoted 7 lines

Quoted 3 lines

The current approach is simpler and ensures irq_set_affinity always works. The limit of max 2047 IDs is sufficient for many systems (if not all).

Let me give you another view. On a 128c system each core has ~16 unique interrupts for disposal. E.g. the Intel E800 NIC has more than 2048 network queue pairs for each PF.

Quoted 5 lines

Clearly, this example is a hypothetical and represents a poorly designed platform.

Having just 16 IDs per-Core is a very poor design choice. In fact, the Server SoC spec mandates a minimum 255 IDs.

Quoted 6 lines

You are misreading. A 128c system with 2047 MSIs per-core, will only have 16 *per-core unique* (2047/128) interrupts with the current series.

I'm not saying that each IMSIC has 16 IDs, I'm saying that in a 128c system with the maximum amount of MSIs possible in the spec, you'll end up with 16 *unique* interrupts per core.

Quoted 5 lines

-ENOPARSE

I don't see how this applies to the current approach because we treat MSI ID space as global across cores so if a system has 2047 MSIs per-core then we have 2047 MSIs across all cores.

Ok, I'll try again! :-)

Let's assume that each core in the 128c system has some per-core resources, say a two NIC queue pairs, and a storage queue pair. This will consume, e.g., 2*2 + 2 (6) MSI sources from the global namespace.

If each core does this it'll be 6*128 MSI sources of the global namespace.

The maximum number of "privates" MSI sources a core can utilize is 16.

I'm trying (it's does seem to go that well ;-)) to point out that it's only 16 unique sources per core. For, say, a 256 core system it would be
8. 2047 MSI sources in a system is not much.

Say that I want to spin up 24 NIC queues with one MSI each on each core on my 128c system. That's not possible with this series, while with an 1-1 system it wouldn't be an issue.

Clearer, or still weird?

Quoted 40 lines

Quoted 40 lines

Quoted 7 lines

Regarding NICs which support a large number of queues, the driver will typically enable only one queue per-core and set the affinity to separate cores. We have user-space data plane applications based on DPDK which are capable of using a large number of NIC queues but these applications are polling based and don't use MSIs.

That's one sample point, and clearly not the only one. There are *many* different usage models. Just because you *assign* MSI, doesn't mean they are firing all the time.

I can show you a couple of networking setups where this is clearly not enough. Each core has a large number of QoS queues, and each queue would very much like to have a dedicated MSI.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

When we encounter a system requiring a large number of MSIs, we can either:
1) Extend the AIA spec to support greater than 2047 IDs
2) Re-think the approach in the IMSIC driver

The choice between #1 and #2 above depends on the
guarantees we want for irq_set_affinity().

The irq_set_affinity() behavior is better with this series, but I think the other downsides: number of available interrupt sources, and IPI broadcast are worse.

Quoted 5 lines

The IPI overhead in the approach you are suggesting will be even bad compared to the IPI overhead of the current approach because we will end-up doing IPI upon every irq_set_affinity() in the suggested approach compared to doing IPI upon every mask/unmask in the current approach.

Again, very workload dependent.

This series does IPI broadcast on masking/unmasking, which means that cores that don't care get interrupted because, say, a network queue-pair is setup on another core.

Quoted 8 lines

Quoted 1 line

Some workloads never change the irq affinity.

There are various events which irq affinity such as irq balance, CPU hotplug, system suspend, etc.

Also, the 1-1 approach does IPI upon set_affinity, mask and unmask whereas the 1-n approach does IPI only upon mask and unmask.

An important distinction; When you say IPI on mask/unmask it is a broadcast IPI to *all* cores, which is pretty instrusive.

The 1-1 variant does an IPI to a *one* target core.

Quoted 15 lines

Quoted 14 lines

I'm just pointing out that there are pro/cons with both variants.

Quoted 2 lines

The biggest advantage of the current approach is a reliable irq_set_affinity() which is a very valuable thing to have.

...and I'm arguing that we're paying a big price for that.

Quoted 3 lines

ARM systems easily support a large number of LPIs per-core.
For example, GIC-700 supports 56000 LPIs per-core.
(Refer, https://developer.arm.com/documentation/101516/0300/About-the-GIC-700/Features)

Yeah, but this is not the GIC. This is something that looks more like the x86 world. We'll be stuck with a lot of implementations with AIA 1.0 spec, and many cores.

Quoted 5 lines

Quoted 3 lines

Well, RISC-V AIA is neigher ARM GIG not x86 APIC. All I am saying is that there are systems with large number per-core interrupt IDs for handling MSIs.

Yes, and while that is nice, it's not what IMSIC is.

Some follow-ups, after thinking more about it more over the weekend.

Quoted 7 lines

Quoted 2 lines

* Do one really need an IPI for irq_set_affinity() for the 1-1 model? Why touch the enable/disable bits when moving interrupts?

In the 1-1 model, the ID on the current HART and target HART upon irq_set_affinity will be different so we can't leave the unused ID on current HART enabled because it can lead to spurious interrupts when the ID on current HART is re-used for some other device.

Hmm, is this really an actual problem, or a theoretical one? The implementation need to track what's in-use, so can we ever get into this situation?

Quoted 1 line

As of now, it is theoretical but it is certainly possible to hit this issue.

Sorry for being slow here, Anup, but could you give an example how this could happen? For me it sounds like this could only be caused by a broken (buggy) implementation?

Quoted 9 lines

Quoted 5 lines

Somewhat related; I had a similar question for imsic_pci_{un,}mask_irq() -- why not only do the the default mask operation (only pci_msi_{un,}mask_irq()), but instead propagate to the IMSIC mask/unmask?

We have hierarchical IMSIC PCI irq domain whoes parent irq domain is IMSIC base domain. Unfortunately pci_msi_[un]mask_irq() don't work for hierarchical irq domain.

Ok! Thanks for the explanation!

Quoted 26 lines

Quoted 26 lines

Quoted 4 lines

There is also a possibility of receiving an interrupt while the ID was moved to a new target HART in-which case we have to detect and re-trigger interrupt on the new target HART. In fact, x86 APLIC does an IPI to take care of this case.

This case I get, and the implementation can track that both are in use. It's the spurious one that I'm dubious of (don't get).

Quoted 17 lines

Quoted 8 lines

* In my book the IMSIC looks very much like the x86 LAPIC, which also has few interrupts (IMSIC <2048, LAPIC 256). The IRQ matrix allocator [1], and a scheme similar to LAPIC [2] would be a good fit. This is the 1-1 model, but more sophisticated than what I've been describing (e.g. properly handling mangaged/regular irqs). As a bonus we would get the IRQ matrix debugfs/tracepoint support.

Yes, I have been evaluating the 1-1 model for the past few days. I also have a working implementation with a simple per-CPU bitmap based allocator which handles both legacy MSI (block of 1,2,4,8,16, or 32 IDs) and MSI-X.

The irq matrix allocator needs to be improved for handling legacy MSI so initially I will post a v11 series which works for me and converging with irq matrix allocator can be future work.

Quoted 5 lines

Quoted 2 lines

What's missing/needs to be improved for legacy MSI (legacy MSI == !MSI-X, right?) in the matrix allocator?

For legacy MSI, a block IDs needs to be contiguous and the number of IDs can be a power of 2.

Oh, so this is not supported by the matrix allocator?