Linux RISC-V IOMMU Support

PATCH Superseded Tomasz Jeznach 7 participants 30 messages Apr 18, 2024 – Apr 25, 2024

Subsystems (4)

Device tree bindings RISC-V IOMMU RISC-V IOMMU

lore

Share & tools

lore

Tomasz Jeznach

author

2024-04-18 16:32 UTC #0 lore raw

v2 cover Linux RISC-V IOMMU Support

This patch series introduces support for RISC-V IOMMU architected hardware into the Linux kernel.

The RISC-V IOMMU specification, which this series is based on, is ratified and available at GitHub/riscv-non-isa [1].

At a high level, the RISC-V IOMMU specification defines:

1) Data structures:
- Device-context: Associates devices with address spaces and holds per-device parameters for address translations.
- Process-contexts: Associates different virtual address spaces based on device-provided process identification numbers.
- MSI page table configuration used to direct an MSI to a guest interrupt file in an IMSIC.
2) In-memory queue interface:
- Command-queue for issuing commands to the IOMMU.
- Fault/event queue for reporting faults and events.
- Page-request queue for reporting "Page Request" messages received from PCIe devices.
- Message-signaled and wire-signaled interrupt mechanisms.
3) Memory-mapped programming interface:
- Mandatory and optional register layout and description.
- Software guidelines for device initialization and capabilities discovery.

This series introduces RISC-V IOMMU hardware initialization and complete single-stage translation with paging domain support.

The patches are organized as follows:

Patch 1: Introduces minimal required device tree bindings for the driver. Patch 2: Defines RISC-V IOMMU data structures, hardware programming interface
registers layout, and minimal initialization code for enabling global pass-through for all connected masters.
Patch 3: Implements the device driver for PCIe implementation of RISC-V IOMMU
architected hardware.
Patch 4: Introduces IOMMU interfaces to the kernel subsystem.
Patch 5: Implements device directory management with discovery sequences for
I/O mapped or in-memory device directory table location, hardware capabilities discovery, and device to domain attach implementation.
Patch 6: Implements command and fault queue, and introduces directory cache
invalidation sequences.
Patch 7: Implements paging domain, with page table using the same format as the
CPU’s MMU. This patch series enables only 4K mappings; complete support for large page mappings will be introduced in follow-up patch series.

Follow-up patch series, providing large page support and updated walk cache management based on the revised specification, and complete ATS/PRI/SVA support, will be posted to GitHub [2] in the next few days.

Changes from v1:

This version includes major reorganization of the code related to queue and page table management, removal of all ATS/PRI/SVA features to be addressed in follow-up patch series, removal of unnecessary checks, and adoption of new interfaces for identity and paging domain allocations.

Apologies for the delay in sending v2 series, and thank you for valuable feedback and patience with last patch series.

Best regards,
Tomasz Jeznach

[1] link: https://github.com/riscv-non-isa/riscv-iommu
[2] link: https://github.com/tjeznach/linux
v1 link: https://lore.kernel.org/linux-iommu/[email protected]/

Tomasz Jeznach (7):
dt-bindings: iommu: riscv: Add bindings for RISC-V IOMMU iommu/riscv: Add RISC-V IOMMU platform device driver iommu/riscv: Add RISC-V IOMMU PCIe device driver iommu/riscv: Enable IOMMU registration and device probe. iommu/riscv: Device directory management. iommu/riscv: Command and fault queue support iommu/riscv: Paging domain support

 .../bindings/iommu/riscv,iommu.yaml           |  149 ++
 MAINTAINERS                                   |   14 +
 drivers/iommu/Kconfig                         |    1 +
 drivers/iommu/Makefile                        |    2 +-
 drivers/iommu/riscv/Kconfig                   |   23 +
 drivers/iommu/riscv/Makefile                  |    3 +
 drivers/iommu/riscv/iommu-bits.h              |  782 +++++++++
 drivers/iommu/riscv/iommu-pci.c               |  154 ++
 drivers/iommu/riscv/iommu-platform.c          |   94 ++
 drivers/iommu/riscv/iommu.c                   | 1441 +++++++++++++++++
 drivers/iommu/riscv/iommu.h                   |   88 +
 11 files changed, 2750 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
 create mode 100644 drivers/iommu/riscv/Kconfig
 create mode 100644 drivers/iommu/riscv/Makefile
 create mode 100644 drivers/iommu/riscv/iommu-bits.h
 create mode 100644 drivers/iommu/riscv/iommu-pci.c
 create mode 100644 drivers/iommu/riscv/iommu-platform.c
 create mode 100644 drivers/iommu/riscv/iommu.c
 create mode 100644 drivers/iommu/riscv/iommu.h

base-commit: 0bbac3facb5d6cc0171c45c9873a2dc96bea9680

Signature · 1 line

2.34.1

Tomasz Jeznach

author

2024-04-18 16:32 UTC #1 lore raw

PATCH v2 1/7 dt-bindings: iommu: riscv: Add bindings for RISC-V IOMMU

Add bindings for the RISC-V IOMMU device drivers.

Co-developed-by Anup Patel <[email protected]>

Signed-off-by Anup Patel <[email protected]>

Signed-off-by Tomasz Jeznach <[email protected]>

---
 .../bindings/iommu/riscv,iommu.yaml           | 149 ++++++++++++++++++
 MAINTAINERS                                   |   7 +
 2 files changed, 156 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/riscv,iommu.yaml

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +149 -0

new file mode 100644
index 000000000000..d6522ddd43fa
@@ -0,0 +1,149 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/iommu/riscv,iommu.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: RISC-V IOMMU Architecture Implementation
+
+maintainers:
+  - Tomasz Jeznach <[email protected]>
+
+description: |+
+  The RISC-V IOMMU provides memory address translation and isolation for
+  input and output devices, supporting per-device translation context,
+  shared process address spaces including the ATS and PRI components of
+  the PCIe specification, two stage address translation and MSI remapping.
+  It supports identical translation table format to the RISC-V address
+  translation tables with page level access and protection attributes.
+  Hardware uses in-memory command and fault reporting queues with wired
+  interrupt or MSI notifications.
+
+  Visit https://github.com/riscv-non-isa/riscv-iommu for more details.
+
+  For information on assigning RISC-V IOMMU to its peripheral devices,
+  see generic IOMMU bindings.
+
+properties:
+  # For PCIe IOMMU hardware compatible property should contain the vendor
+  # and device ID according to the PCI Bus Binding specification.
+  # Since PCI provides built-in identification methods, compatible is not
+  # actually required. For non-PCIe hardware implementations 'riscv,iommu'
+  # should be specified along with 'reg' property providing MMIO location.
+  compatible:
+    oneOf:
+      - items:
+          - const: riscv,pci-iommu
+          - const: pci1efd,edf1
+      - items:
+          - const: pci1efd,edf1
+      - items:
+          - const: riscv,iommu
+
+  reg:
+    maxItems: 1
+    description:
+      For non-PCI devices this represents base address and size of for the
+      IOMMU memory mapped registers interface.
+      For PCI IOMMU hardware implementation this should represent an address
+      of the IOMMU, as defined in the PCI Bus Binding reference. The reg
+      property is a five-cell address encoded as (phys.hi phys.mid phys.lo
+      size.hi size.lo), where phys.hi should contain the device's BDF as
+      0b00000000 bbbbbbbb dddddfff 00000000. The other cells should be zero.
+
+  '#iommu-cells':
+    const: 1
+    description:
+      Has to be one. The single cell describes the requester id emitted
+      by a master to the IOMMU.
+
+  interrupts:
+    minItems: 1
+    maxItems: 4
+    description:
+      Wired interrupt vectors available for RISC-V IOMMU to notify the
+      RISC-V HARTS. The cause to interrupt vector is software defined
+      using IVEC IOMMU register.
+
+  msi-parent: true
+
+  power-domains:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - '#iommu-cells'
+
+additionalProperties: false
+
+examples:
+  - |+
+    /* Example 1 (IOMMU device with wired interrupts) */
+    #include <dt-bindings/interrupt-controller/irq.h>
+
+    iommu1: iommu@1bccd000 {
+        compatible = "riscv,iommu";
+        reg = <0x1bccd000 0x1000>;
+        interrupt-parent = <&aplic_smode>;
+        interrupts = <32 IRQ_TYPE_LEVEL_HIGH>,
+                     <33 IRQ_TYPE_LEVEL_HIGH>,
+                     <34 IRQ_TYPE_LEVEL_HIGH>,
+                     <35 IRQ_TYPE_LEVEL_HIGH>;
+        #iommu-cells = <1>;
+    };
+
+    /* Device with two IOMMU device IDs, 0 and 7 */
+    master1 {
+        iommus = <&iommu1 0>, <&iommu1 7>;
+    };
+
+  - |+
+    /* Example 2 (IOMMU device with shared wired interrupt) */
+    #include <dt-bindings/interrupt-controller/irq.h>
+
+    iommu2: iommu@1bccd000 {
+        compatible = "riscv,iommu";
+        reg = <0x1bccd000 0x1000>;
+        interrupt-parent = <&aplic_smode>;
+        interrupts = <32 IRQ_TYPE_LEVEL_HIGH>;
+        #iommu-cells = <1>;
+    };
+
+  - |+
+    /* Example 3 (IOMMU device with MSIs) */
+    iommu3: iommu@1bcdd000 {
+        compatible = "riscv,iommu";
+        reg = <0x1bccd000 0x1000>;
+        msi-parent = <&imsics_smode>;
+        #iommu-cells = <1>;
+    };
+
+  - |+
+    /* Example 4 (IOMMU PCIe device with MSIs) */
+    bus {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        pcie@30000000 {
+            device_type = "pci";
+            #address-cells = <3>;
+            #size-cells = <2>;
+            reg = <0x0 0x30000000  0x0 0x1000000>;
+            ranges = <0x02000000 0x0 0x41000000  0x0 0x41000000  0x0 0x0f000000>;
+
+            /*
+             * The IOMMU manages all functions in this PCI domain except
+             * itself. Omit BDF 00:01.0.
+             */
+            iommu-map = <0x0 &iommu0 0x0 0x8
+                         0x9 &iommu0 0x9 0xfff7>;
+
+            /* The IOMMU programming interface uses slot 00:01.0 */
+            iommu0: iommu@1,0 {
+               compatible = "pci1efd,edf1";
+               reg = <0x800 0 0 0 0>;
+               #iommu-cells = <1>;
+            };
+        };
+    };

MAINTAINERS +7 -0

index c23fda1aa1f0..2657f9eae84c 100644
@@ -18966,6 +18966,13 @@ F:	arch/riscv/
 N:	riscv
 K:	riscv
 
+RISC-V IOMMU
+M:	Tomasz Jeznach <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Maintained
+F:	Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
+
 RISC-V MICROCHIP FPGA SUPPORT
 M:	Conor Dooley <[email protected]>
 M:	Daire McNamara <[email protected]>

Signature · 1 line

2.34.1

Conor Dooley

2024-04-18 17:04 UTC #2 lore raw

Re: [PATCH v2 1/7] dt-bindings: iommu: riscv: Add bindings for RISC-V IOMMU

Quoted 19 lines · Tomasz Jeznach

Add bindings for the RISC-V IOMMU device drivers.
Co-developed-by Anup Patel <[email protected]>
Signed-off-by Anup Patel <[email protected]>
Signed-off-by Tomasz Jeznach <[email protected]>
---
 .../bindings/iommu/riscv,iommu.yaml           | 149 ++++++++++++++++++
 MAINTAINERS                                   |   7 +
 2 files changed, 156 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +2 -0
new file mode 100644
index 000000000000..d6522ddd43fa
@@ -0,0 +1,149 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 12

Quoted 10 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +10 -0

+---
+$id: http://devicetree.org/schemas/iommu/riscv,iommu.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: RISC-V IOMMU Architecture Implementation
+
+maintainers:
+  - Tomasz Jeznach <[email protected]>
+
+description: |+

FYI, the + here is probably not needed.

Quoted 10 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +10 -0

+  The RISC-V IOMMU provides memory address translation and isolation for
+  input and output devices, supporting per-device translation context,
+  shared process address spaces including the ATS and PRI components of
+  the PCIe specification, two stage address translation and MSI remapping.
+  It supports identical translation table format to the RISC-V address
+  translation tables with page level access and protection attributes.
+  Hardware uses in-memory command and fault reporting queues with wired
+  interrupt or MSI notifications.
+
+  Visit https://github.com/riscv-non-isa/riscv-iommu for more details.

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 32

Quoted 10 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +10 -0

+
+  For information on assigning RISC-V IOMMU to its peripheral devices,
+  see generic IOMMU bindings.
+
+properties:
+  # For PCIe IOMMU hardware compatible property should contain the vendor
+  # and device ID according to the PCI Bus Binding specification.
+  # Since PCI provides built-in identification methods, compatible is not
+  # actually required. For non-PCIe hardware implementations 'riscv,iommu'
+  # should be specified along with 'reg' property providing MMIO location.

I dunno, I'd like to see soc-specific compatibles for implementations of the RISC-V IOMMU. If you need a DT compatible for use in QEMU, I'd suggest doing what was done for the aplic and having a dedicated compatible for that and disallow having "riscv,iommu" in isolation.

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 39

Quoted 7 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +7 -0

+  compatible:
+    oneOf:
+      - items:
+          - const: riscv,pci-iommu
+          - const: pci1efd,edf1
+      - items:
+          - const: pci1efd,edf1

Why are both versions allowed? If the former is more understandable, can't we just go with that?

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 41

Quoted 2 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +2 -0
+      - items:
+          - const: riscv,iommu

Other than the compatible setup I think this is pretty decent though, Conor.

Tomasz Jeznach

author

2024-04-24 22:37 UTC #3 lore raw

Re: [PATCH v2 1/7] dt-bindings: iommu: riscv: Add bindings for RISC-V IOMMU

Quoted 51 lines · Conor Dooley

Quoted 19 lines · Tomasz Jeznach

Add bindings for the RISC-V IOMMU device drivers.
Co-developed-by Anup Patel <[email protected]>
Signed-off-by Anup Patel <[email protected]>
Signed-off-by Tomasz Jeznach <[email protected]>
---
 .../bindings/iommu/riscv,iommu.yaml           | 149 ++++++++++++++++++
 MAINTAINERS                                   |   7 +
 2 files changed, 156 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +2 -0
new file mode 100644
index 000000000000..d6522ddd43fa
@@ -0,0 +1,149 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 12

Quoted 10 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +10 -0

+---
+$id: http://devicetree.org/schemas/iommu/riscv,iommu.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: RISC-V IOMMU Architecture Implementation
+
+maintainers:
+  - Tomasz Jeznach <[email protected]>
+
+description: |+

FYI, the + here is probably not needed.

Quoted 17 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +17 -0

+  The RISC-V IOMMU provides memory address translation and isolation for
+  input and output devices, supporting per-device translation context,
+  shared process address spaces including the ATS and PRI components of
+  the PCIe specification, two stage address translation and MSI remapping.
+  It supports identical translation table format to the RISC-V address
+  translation tables with page level access and protection attributes.
+  Hardware uses in-memory command and fault reporting queues with wired
+  interrupt or MSI notifications.
+
+  Visit https://github.com/riscv-non-isa/riscv-iommu for more details.
+
+  For information on assigning RISC-V IOMMU to its peripheral devices,
+  see generic IOMMU bindings.
+
+properties:
+  # For PCIe IOMMU hardware compatible property should contain the vendor
+  # and device ID according to the PCI Bus Binding specification.

Quoted 9 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 32
Quoted 3 lines
Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +3 -0
+  # Since PCI provides built-in identification methods, compatible is not
+  # actually required. For non-PCIe hardware implementations 'riscv,iommu'
+  # should be specified along with 'reg' property providing MMIO location.
I dunno, I'd like to see soc-specific compatibles for implementations of the RISC-V IOMMU. If you need a DT compatible for use in QEMU, I'd suggest doing what was done for the aplic and having a dedicated compatible for that and disallow having "riscv,iommu" in isolation.

Makes sense. Will update to something like below:
compatible:
oneOf:
- items:
- enum:
- qemu,iommu
- const: riscv,iommu
- items:
- enum:
- pci1efd,edf1
- const: riscv,pci-iommu

Quoted 8 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 39
Quoted 7 lines
Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +7 -0
+  compatible:
+    oneOf:
+      - items:
+          - const: riscv,pci-iommu
+          - const: pci1efd,edf1
+      - items:
+          - const: pci1efd,edf1

Quoted 8 lines

Why are both versions allowed? If the former is more understandable, can't we just go with that?
Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 41
Quoted 2 lines
Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +2 -0
+      - items:
+          - const: riscv,iommu
Other than the compatible setup I think this is pretty decent though, Conor.

ACK to other comments,

Thanks for the review,
- Tomasz

Conor Dooley

2024-04-25 17:11 UTC #4 lore raw

Re: [PATCH v2 1/7] dt-bindings: iommu: riscv: Add bindings for RISC-V IOMMU

Quoted 11 lines · Tomasz Jeznach

Makes sense. Will update to something like below:
compatible:
oneOf:
- items:
- enum:
- qemu,iommu
- const: riscv,iommu
- items:
- enum:
- pci1efd,edf1
- const: riscv,pci-iommu

That seems reasonable to me, addresses both what Rob and I pointed out.

Thanks,
Conor.

Rob Herring

maintainer

2024-04-22 14:04 UTC #5 lore raw

Re: [PATCH v2 1/7] dt-bindings: iommu: riscv: Add bindings for RISC-V IOMMU

In reply to Tomasz Jeznach

Quoted 44 lines · Tomasz Jeznach

Add bindings for the RISC-V IOMMU device drivers.

Co-developed-by Anup Patel <[email protected]>

Signed-off-by Anup Patel <[email protected]>

Signed-off-by Tomasz Jeznach <[email protected]>

---
 .../bindings/iommu/riscv,iommu.yaml           | 149 ++++++++++++++++++
 MAINTAINERS                                   |   7 +
 2 files changed, 156 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/riscv,iommu.yaml

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +27 -0

new file mode 100644
index 000000000000..d6522ddd43fa
@@ -0,0 +1,149 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/iommu/riscv,iommu.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: RISC-V IOMMU Architecture Implementation
+
+maintainers:
+  - Tomasz Jeznach <[email protected]>
+
+description: |+
+  The RISC-V IOMMU provides memory address translation and isolation for
+  input and output devices, supporting per-device translation context,
+  shared process address spaces including the ATS and PRI components of
+  the PCIe specification, two stage address translation and MSI remapping.
+  It supports identical translation table format to the RISC-V address
+  translation tables with page level access and protection attributes.
+  Hardware uses in-memory command and fault reporting queues with wired
+  interrupt or MSI notifications.
+
+  Visit https://github.com/riscv-non-isa/riscv-iommu for more details.
+
+  For information on assigning RISC-V IOMMU to its peripheral devices,
+  see generic IOMMU bindings.
+
+properties:

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 37

Quoted 10 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +10 -0

+  # For PCIe IOMMU hardware compatible property should contain the vendor
+  # and device ID according to the PCI Bus Binding specification.
+  # Since PCI provides built-in identification methods, compatible is not
+  # actually required. For non-PCIe hardware implementations 'riscv,iommu'
+  # should be specified along with 'reg' property providing MMIO location.
+  compatible:
+    oneOf:
+      - items:
+          - const: riscv,pci-iommu
+          - const: pci1efd,edf1

Given the PCI compatible string is a specific vendor and device, it is more specific than "riscv,pci-iommu" and should come first.

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 39

Quoted 2 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +2 -0
+      - items:
+          - const: pci1efd,edf1

Why do you need to support this without riscv,pci-iommu?

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml line 41

Quoted 2 lines

Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file +2 -0
+      - items:
+          - const: riscv,iommu

I agree with what Conor said on this.

Rob

Tomasz Jeznach

author

2024-04-18 16:32 UTC #6 lore raw

PATCH v2 2/7 iommu/riscv: Add RISC-V IOMMU platform device driver

In reply to Tomasz Jeznach

Introduce platform device driver for implementation of RISC-V IOMMU architected hardware.

Hardware interface definition located in file iommu-bits.h is based on ratified RISC-V IOMMU Architecture Specification version 1.0.0.

This patch implements platform device initialization, early check and configuration of the IOMMU interfaces and enables global pass-through address translation mode (iommu_mode == BARE), without registering hardware instance in the IOMMU subsystem.

Link https://github.com/riscv-non-isa/riscv-iommu

Co-developed-by Nick Kossifidis <[email protected]>

Signed-off-by Nick Kossifidis <[email protected]>

Co-developed-by Sebastien Boeuf <[email protected]>

Signed-off-by Sebastien Boeuf <[email protected]>

Signed-off-by Tomasz Jeznach <[email protected]>

---
 MAINTAINERS                          |   6 +
 drivers/iommu/Kconfig                |   1 +
 drivers/iommu/Makefile               |   2 +-
 drivers/iommu/riscv/Kconfig          |  16 +
 drivers/iommu/riscv/Makefile         |   2 +
 drivers/iommu/riscv/iommu-bits.h     | 707 +++++++++++++++++++++++++++
 drivers/iommu/riscv/iommu-platform.c |  94 ++++
 drivers/iommu/riscv/iommu.c          |  89 ++++
 drivers/iommu/riscv/iommu.h          |  62 +++
 9 files changed, 978 insertions(+), 1 deletion(-)
 create mode 100644 drivers/iommu/riscv/Kconfig
 create mode 100644 drivers/iommu/riscv/Makefile
 create mode 100644 drivers/iommu/riscv/iommu-bits.h
 create mode 100644 drivers/iommu/riscv/iommu-platform.c
 create mode 100644 drivers/iommu/riscv/iommu.c
 create mode 100644 drivers/iommu/riscv/iommu.h

MAINTAINERS +6 -0

index 2657f9eae84c..051599c76585 100644
@@ -18972,6 +18972,12 @@ L:	[email protected]
 L:	[email protected]
 S:	Maintained
 F:	Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
+F:	drivers/iommu/riscv/Kconfig
+F:	drivers/iommu/riscv/Makefile
+F:	drivers/iommu/riscv/iommu-bits.h
+F:	drivers/iommu/riscv/iommu-platform.c
+F:	drivers/iommu/riscv/iommu.c
+F:	drivers/iommu/riscv/iommu.h
 
 RISC-V MICROCHIP FPGA SUPPORT
 M:	Conor Dooley <[email protected]>

drivers/iommu/Kconfig +1 -0

index 0af39bbbe3a3..ae762db0365e 100644
@@ -195,6 +195,7 @@ config MSM_IOMMU
 source "drivers/iommu/amd/Kconfig"
 source "drivers/iommu/intel/Kconfig"
 source "drivers/iommu/iommufd/Kconfig"
+source "drivers/iommu/riscv/Kconfig"
 
 config IRQ_REMAP
 	bool "Support for Interrupt Remapping"

drivers/iommu/Makefile +1 -1

index 542760d963ec..5e5a83c6c2aa 100644
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-y += amd/ intel/ arm/ iommufd/
+obj-y += amd/ intel/ arm/ iommufd/ riscv/
 obj-$(CONFIG_IOMMU_API) += iommu.o
 obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o

drivers/iommu/riscv/Kconfig new file +16 -0

new file mode 100644
index 000000000000..d02326bddb4c
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# RISC-V IOMMU support
+
+config RISCV_IOMMU
+	def_bool y if RISCV && 64BIT && MMU
+	depends on RISCV && 64BIT && MMU
+	select DMA_OPS
+	select IOMMU_API
+	select IOMMU_IOVA
+	help
+	  Support for implementations of the RISC-V IOMMU architecture that
+	  complements the RISC-V MMU capabilities, providing similar address
+	  translation and protection functions for accesses from I/O devices.
+
+	  Say Y here if your SoC includes an IOMMU device implementing
+	  the RISC-V IOMMU architecture.

drivers/iommu/riscv/Makefile new file +2 -0

new file mode 100644
index 000000000000..e4c189de58d3
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-platform.o

drivers/iommu/riscv/iommu-bits.h new file +707 -0

707 lines not on this page · raw message

drivers/iommu/riscv/iommu-platform.c new file +94 -0

94 lines not on this page · raw message

drivers/iommu/riscv/iommu.c new file +89 -0

89 lines not on this page · raw message

drivers/iommu/riscv/iommu.h new file +62 -0

62 lines not on this page · raw message

Signature · 1 line

2.34.1

Robin Murphy

reviewer

2024-04-18 21:22 UTC #7 lore raw

Re: [PATCH v2 2/7] iommu/riscv: Add RISC-V IOMMU platform device driver

Quoted 40 lines · Tomasz Jeznach

Introduce platform device driver for implementation of RISC-V IOMMU architected hardware.
Hardware interface definition located in file iommu-bits.h is based on ratified RISC-V IOMMU Architecture Specification version 1.0.0.
This patch implements platform device initialization, early check and configuration of the IOMMU interfaces and enables global pass-through address translation mode (iommu_mode == BARE), without registering hardware instance in the IOMMU subsystem.
Link https://github.com/riscv-non-isa/riscv-iommu
Co-developed-by Nick Kossifidis <[email protected]>
Signed-off-by Nick Kossifidis <[email protected]>
Co-developed-by Sebastien Boeuf <[email protected]>
Signed-off-by Sebastien Boeuf <[email protected]>
Signed-off-by Tomasz Jeznach <[email protected]>
---
  MAINTAINERS                          |   6 +
  drivers/iommu/Kconfig                |   1 +
  drivers/iommu/Makefile               |   2 +-
  drivers/iommu/riscv/Kconfig          |  16 +
  drivers/iommu/riscv/Makefile         |   2 +
  drivers/iommu/riscv/iommu-bits.h     | 707 +++++++++++++++++++++++++++
  drivers/iommu/riscv/iommu-platform.c |  94 ++++
  drivers/iommu/riscv/iommu.c          |  89 ++++
  drivers/iommu/riscv/iommu.h          |  62 +++
  9 files changed, 978 insertions(+), 1 deletion(-)
create mode 100644 drivers/iommu/riscv/Kconfig create mode 100644 drivers/iommu/riscv/Makefile create mode 100644 drivers/iommu/riscv/iommu-bits.h create mode 100644 drivers/iommu/riscv/iommu-platform.c create mode 100644 drivers/iommu/riscv/iommu.c create mode 100644 drivers/iommu/riscv/iommu.h
MAINTAINERS +0 -0
index 2657f9eae84c..051599c76585 100644

MAINTAINERS L: [email protected] · line 18980

Quoted 9 lines

MAINTAINERS +6 -0

@@ -18972,6 +18972,12 @@ L:	[email protected]
  L:	[email protected]
  S:	Maintained
  F:	Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
+F:	drivers/iommu/riscv/Kconfig
+F:	drivers/iommu/riscv/Makefile
+F:	drivers/iommu/riscv/iommu-bits.h
+F:	drivers/iommu/riscv/iommu-platform.c
+F:	drivers/iommu/riscv/iommu.c
+F:	drivers/iommu/riscv/iommu.h

I'm pretty sure a single "F: drivers/iommu/riscv/" pattern will suffice.

Quoted 31 lines

MAINTAINERS +0 -0

  RISC-V MICROCHIP FPGA SUPPORT
  M:	Conor Dooley <[email protected]>

drivers/iommu/Kconfig +1 -0

index 0af39bbbe3a3..ae762db0365e 100644
@@ -195,6 +195,7 @@ config MSM_IOMMU
  source "drivers/iommu/amd/Kconfig"
  source "drivers/iommu/intel/Kconfig"
  source "drivers/iommu/iommufd/Kconfig"
+source "drivers/iommu/riscv/Kconfig"
 
  config IRQ_REMAP
  	bool "Support for Interrupt Remapping"

drivers/iommu/Makefile +1 -1

index 542760d963ec..5e5a83c6c2aa 100644
@@ -1,5 +1,5 @@
  # SPDX-License-Identifier: GPL-2.0
-obj-y += amd/ intel/ arm/ iommufd/
+obj-y += amd/ intel/ arm/ iommufd/ riscv/
  obj-$(CONFIG_IOMMU_API) += iommu.o
  obj-$(CONFIG_IOMMU_API) += iommu-traces.o
  obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o

drivers/iommu/riscv/Kconfig new file +0 -0

new file mode 100644
index 000000000000..d02326bddb4c

drivers/iommu/riscv/Kconfig line 5

Quoted 5 lines

drivers/iommu/riscv/Kconfig new file +5 -0

@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# RISC-V IOMMU support
+
+config RISCV_IOMMU
+	def_bool y if RISCV && 64BIT && MMU

Drop the dependencies here, they're already dependencies. However, also consider allowing users to configure this out without disabling IOMMU_SUPPORT entirely - I imagine other IOMMU implementations are going to end up paired wiothg RISC-V CPUs sooner or later. Furthermore, if it's a regular driver model driver, consider allowing it to build as a module. Not to mention that the help text below is rather pointless if there's no prompt offered in the first place.

drivers/iommu/riscv/Kconfig line 7

Quoted 2 lines

drivers/iommu/riscv/Kconfig new file +2 -0
+	depends on RISCV && 64BIT && MMU
+	select DMA_OPS

Drop this, you're not (and shouldn't be) architecture code implementing DMA ops.

drivers/iommu/riscv/Kconfig line 9

Quoted 2 lines

drivers/iommu/riscv/Kconfig new file +2 -0
+	select IOMMU_API
+	select IOMMU_IOVA

Drop this, you're not using the IOVA library either.

Quoted 784 lines

drivers/iommu/riscv/Kconfig new file +7 -0

+	help
+	  Support for implementations of the RISC-V IOMMU architecture that
+	  complements the RISC-V MMU capabilities, providing similar address
+	  translation and protection functions for accesses from I/O devices.
+
+	  Say Y here if your SoC includes an IOMMU device implementing
+	  the RISC-V IOMMU architecture.

drivers/iommu/riscv/Makefile new file +2 -0

new file mode 100644
index 000000000000..e4c189de58d3
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-platform.o

drivers/iommu/riscv/iommu-bits.h new file +707 -0

new file mode 100644
index 000000000000..ba093c29de9f
@@ -0,0 +1,707 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright © 2022-2024 Rivos Inc.
+ * Copyright © 2023 FORTH-ICS/CARV
+ * Copyright © 2023 RISC-V IOMMU Task Group
+ *
+ * RISC-V IOMMU - Register Layout and Data Structures.
+ *
+ * Based on the 'RISC-V IOMMU Architecture Specification', Version 1.0
+ * Published at  https://github.com/riscv-non-isa/riscv-iommu
+ *
+ */
+
+#ifndef _RISCV_IOMMU_BITS_H_
+#define _RISCV_IOMMU_BITS_H_
+
+#include <linux/types.h>
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+
+/*
+ * Chapter 5: Memory Mapped register interface
+ */
+
+/* Common field positions */
+#define RISCV_IOMMU_PPN_FIELD		GENMASK_ULL(53, 10)
+#define RISCV_IOMMU_QUEUE_LOGSZ_FIELD	GENMASK_ULL(4, 0)
+#define RISCV_IOMMU_QUEUE_INDEX_FIELD	GENMASK_ULL(31, 0)
+#define RISCV_IOMMU_QUEUE_ENABLE	BIT(0)
+#define RISCV_IOMMU_QUEUE_INTR_ENABLE	BIT(1)
+#define RISCV_IOMMU_QUEUE_MEM_FAULT	BIT(8)
+#define RISCV_IOMMU_QUEUE_OVERFLOW	BIT(9)
+#define RISCV_IOMMU_QUEUE_ACTIVE	BIT(16)
+#define RISCV_IOMMU_QUEUE_BUSY		BIT(17)
+
+#define RISCV_IOMMU_ATP_PPN_FIELD	GENMASK_ULL(43, 0)
+#define RISCV_IOMMU_ATP_MODE_FIELD	GENMASK_ULL(63, 60)
+
+/* 5.3 IOMMU Capabilities (64bits) */
+#define RISCV_IOMMU_REG_CAP		0x0000
+#define RISCV_IOMMU_CAP_VERSION		GENMASK_ULL(7, 0)
+#define RISCV_IOMMU_CAP_S_SV32		BIT_ULL(8)
+#define RISCV_IOMMU_CAP_S_SV39		BIT_ULL(9)
+#define RISCV_IOMMU_CAP_S_SV48		BIT_ULL(10)
+#define RISCV_IOMMU_CAP_S_SV57		BIT_ULL(11)
+#define RISCV_IOMMU_CAP_SVPBMT		BIT_ULL(15)
+#define RISCV_IOMMU_CAP_G_SV32		BIT_ULL(16)
+#define RISCV_IOMMU_CAP_G_SV39		BIT_ULL(17)
+#define RISCV_IOMMU_CAP_G_SV48		BIT_ULL(18)
+#define RISCV_IOMMU_CAP_G_SV57		BIT_ULL(19)
+#define RISCV_IOMMU_CAP_AMO_MRIF	BIT_ULL(21)
+#define RISCV_IOMMU_CAP_MSI_FLAT	BIT_ULL(22)
+#define RISCV_IOMMU_CAP_MSI_MRIF	BIT_ULL(23)
+#define RISCV_IOMMU_CAP_AMO_HWAD	BIT_ULL(24)
+#define RISCV_IOMMU_CAP_ATS		BIT_ULL(25)
+#define RISCV_IOMMU_CAP_T2GPA		BIT_ULL(26)
+#define RISCV_IOMMU_CAP_END		BIT_ULL(27)
+#define RISCV_IOMMU_CAP_IGS		GENMASK_ULL(29, 28)
+#define RISCV_IOMMU_CAP_HPM		BIT_ULL(30)
+#define RISCV_IOMMU_CAP_DBG		BIT_ULL(31)
+#define RISCV_IOMMU_CAP_PAS		GENMASK_ULL(37, 32)
+#define RISCV_IOMMU_CAP_PD8		BIT_ULL(38)
+#define RISCV_IOMMU_CAP_PD17		BIT_ULL(39)
+#define RISCV_IOMMU_CAP_PD20		BIT_ULL(40)
+
+#define RISCV_IOMMU_CAP_VERSION_VER_MASK	0xF0
+#define RISCV_IOMMU_CAP_VERSION_REV_MASK	0x0F
+
+/**
+ * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
+ * @RISCV_IOMMU_CAP_IGS_MSI: I/O MMU supports only MSI generation
+ * @RISCV_IOMMU_CAP_IGS_WSI: I/O MMU supports only Wired-Signaled interrupt
+ * @RISCV_IOMMU_CAP_IGS_BOTH: I/O MMU supports both MSI and WSI generation
+ * @RISCV_IOMMU_CAP_IGS_RSRV: Reserved for standard use
+ */
+enum riscv_iommu_igs_settings {
+	RISCV_IOMMU_CAP_IGS_MSI = 0,
+	RISCV_IOMMU_CAP_IGS_WSI = 1,
+	RISCV_IOMMU_CAP_IGS_BOTH = 2,
+	RISCV_IOMMU_CAP_IGS_RSRV = 3
+};
+
+/* 5.4 Features control register (32bits) */
+#define RISCV_IOMMU_REG_FCTL		0x0008
+#define RISCV_IOMMU_FCTL_BE		BIT(0)
+#define RISCV_IOMMU_FCTL_WSI		BIT(1)
+#define RISCV_IOMMU_FCTL_GXL		BIT(2)
+
+/* 5.5 Device-directory-table pointer (64bits) */
+#define RISCV_IOMMU_REG_DDTP		0x0010
+#define RISCV_IOMMU_DDTP_MODE		GENMASK_ULL(3, 0)
+#define RISCV_IOMMU_DDTP_BUSY		BIT_ULL(4)
+#define RISCV_IOMMU_DDTP_PPN		RISCV_IOMMU_PPN_FIELD
+
+/**
+ * enum riscv_iommu_ddtp_modes - I/O MMU translation modes
+ * @RISCV_IOMMU_DDTP_MODE_OFF: No inbound transactions allowed
+ * @RISCV_IOMMU_DDTP_MODE_BARE: Pass-through mode
+ * @RISCV_IOMMU_DDTP_MODE_1LVL: One-level DDT
+ * @RISCV_IOMMU_DDTP_MODE_2LVL: Two-level DDT
+ * @RISCV_IOMMU_DDTP_MODE_3LVL: Three-level DDT
+ * @RISCV_IOMMU_DDTP_MODE_MAX: Max value allowed by specification
+ */
+enum riscv_iommu_ddtp_modes {
+	RISCV_IOMMU_DDTP_MODE_OFF = 0,
+	RISCV_IOMMU_DDTP_MODE_BARE = 1,
+	RISCV_IOMMU_DDTP_MODE_1LVL = 2,
+	RISCV_IOMMU_DDTP_MODE_2LVL = 3,
+	RISCV_IOMMU_DDTP_MODE_3LVL = 4,
+	RISCV_IOMMU_DDTP_MODE_MAX = 4
+};
+
+/* 5.6 Command Queue Base (64bits) */
+#define RISCV_IOMMU_REG_CQB		0x0018
+#define RISCV_IOMMU_CQB_ENTRIES		RISCV_IOMMU_QUEUE_LOGSZ_FIELD
+#define RISCV_IOMMU_CQB_PPN		RISCV_IOMMU_PPN_FIELD
+
+/* 5.7 Command Queue head (32bits) */
+#define RISCV_IOMMU_REG_CQH		0x0020
+#define RISCV_IOMMU_CQH_INDEX		RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.8 Command Queue tail (32bits) */
+#define RISCV_IOMMU_REG_CQT		0x0024
+#define RISCV_IOMMU_CQT_INDEX		RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.9 Fault Queue Base (64bits) */
+#define RISCV_IOMMU_REG_FQB		0x0028
+#define RISCV_IOMMU_FQB_ENTRIES		RISCV_IOMMU_QUEUE_LOGSZ_FIELD
+#define RISCV_IOMMU_FQB_PPN		RISCV_IOMMU_PPN_FIELD
+
+/* 5.10 Fault Queue Head (32bits) */
+#define RISCV_IOMMU_REG_FQH		0x0030
+#define RISCV_IOMMU_FQH_INDEX		RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.11 Fault Queue tail (32bits) */
+#define RISCV_IOMMU_REG_FQT		0x0034
+#define RISCV_IOMMU_FQT_INDEX		RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.12 Page Request Queue base (64bits) */
+#define RISCV_IOMMU_REG_PQB		0x0038
+#define RISCV_IOMMU_PQB_ENTRIES		RISCV_IOMMU_QUEUE_LOGSZ_FIELD
+#define RISCV_IOMMU_PQB_PPN		RISCV_IOMMU_PPN_FIELD
+
+/* 5.13 Page Request Queue head (32bits) */
+#define RISCV_IOMMU_REG_PQH		0x0040
+#define RISCV_IOMMU_PQH_INDEX		RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.14 Page Request Queue tail (32bits) */
+#define RISCV_IOMMU_REG_PQT		0x0044
+#define RISCV_IOMMU_PQT_INDEX_MASK	RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.15 Command Queue CSR (32bits) */
+#define RISCV_IOMMU_REG_CQCSR		0x0048
+#define RISCV_IOMMU_CQCSR_CQEN		RISCV_IOMMU_QUEUE_ENABLE
+#define RISCV_IOMMU_CQCSR_CIE		RISCV_IOMMU_QUEUE_INTR_ENABLE
+#define RISCV_IOMMU_CQCSR_CQMF		RISCV_IOMMU_QUEUE_MEM_FAULT
+#define RISCV_IOMMU_CQCSR_CMD_TO	BIT(9)
+#define RISCV_IOMMU_CQCSR_CMD_ILL	BIT(10)
+#define RISCV_IOMMU_CQCSR_FENCE_W_IP	BIT(11)
+#define RISCV_IOMMU_CQCSR_CQON		RISCV_IOMMU_QUEUE_ACTIVE
+#define RISCV_IOMMU_CQCSR_BUSY		RISCV_IOMMU_QUEUE_BUSY
+
+/* 5.16 Fault Queue CSR (32bits) */
+#define RISCV_IOMMU_REG_FQCSR		0x004C
+#define RISCV_IOMMU_FQCSR_FQEN		RISCV_IOMMU_QUEUE_ENABLE
+#define RISCV_IOMMU_FQCSR_FIE		RISCV_IOMMU_QUEUE_INTR_ENABLE
+#define RISCV_IOMMU_FQCSR_FQMF		RISCV_IOMMU_QUEUE_MEM_FAULT
+#define RISCV_IOMMU_FQCSR_FQOF		RISCV_IOMMU_QUEUE_OVERFLOW
+#define RISCV_IOMMU_FQCSR_FQON		RISCV_IOMMU_QUEUE_ACTIVE
+#define RISCV_IOMMU_FQCSR_BUSY		RISCV_IOMMU_QUEUE_BUSY
+
+/* 5.17 Page Request Queue CSR (32bits) */
+#define RISCV_IOMMU_REG_PQCSR		0x0050
+#define RISCV_IOMMU_PQCSR_PQEN		RISCV_IOMMU_QUEUE_ENABLE
+#define RISCV_IOMMU_PQCSR_PIE		RISCV_IOMMU_QUEUE_INTR_ENABLE
+#define RISCV_IOMMU_PQCSR_PQMF		RISCV_IOMMU_QUEUE_MEM_FAULT
+#define RISCV_IOMMU_PQCSR_PQOF		RISCV_IOMMU_QUEUE_OVERFLOW
+#define RISCV_IOMMU_PQCSR_PQON		RISCV_IOMMU_QUEUE_ACTIVE
+#define RISCV_IOMMU_PQCSR_BUSY		RISCV_IOMMU_QUEUE_BUSY
+
+/* 5.18 Interrupt Pending Status (32bits) */
+#define RISCV_IOMMU_REG_IPSR		0x0054
+
+#define RISCV_IOMMU_INTR_CQ		0
+#define RISCV_IOMMU_INTR_FQ		1
+#define RISCV_IOMMU_INTR_PM		2
+#define RISCV_IOMMU_INTR_PQ		3
+#define RISCV_IOMMU_INTR_COUNT		4
+
+#define RISCV_IOMMU_IPSR_CIP		BIT(RISCV_IOMMU_INTR_CQ)
+#define RISCV_IOMMU_IPSR_FIP		BIT(RISCV_IOMMU_INTR_FQ)
+#define RISCV_IOMMU_IPSR_PMIP		BIT(RISCV_IOMMU_INTR_PM)
+#define RISCV_IOMMU_IPSR_PIP		BIT(RISCV_IOMMU_INTR_PQ)
+
+/* 5.19 Performance monitoring counter overflow status (32bits) */
+#define RISCV_IOMMU_REG_IOCOUNTOVF	0x0058
+#define RISCV_IOMMU_IOCOUNTOVF_CY	BIT(0)
+#define RISCV_IOMMU_IOCOUNTOVF_HPM	GENMASK_ULL(31, 1)
+
+/* 5.20 Performance monitoring counter inhibits (32bits) */
+#define RISCV_IOMMU_REG_IOCOUNTINH	0x005C
+#define RISCV_IOMMU_IOCOUNTINH_CY	BIT(0)
+#define RISCV_IOMMU_IOCOUNTINH_HPM	GENMASK(31, 1)
+
+/* 5.21 Performance monitoring cycles counter (64bits) */
+#define RISCV_IOMMU_REG_IOHPMCYCLES     0x0060
+#define RISCV_IOMMU_IOHPMCYCLES_COUNTER	GENMASK_ULL(62, 0)
+#define RISCV_IOMMU_IOHPMCYCLES_OVF	BIT_ULL(63)
+
+/* 5.22 Performance monitoring event counters (31 * 64bits) */
+#define RISCV_IOMMU_REG_IOHPMCTR_BASE	0x0068
+#define RISCV_IOMMU_REG_IOHPMCTR(_n)	(RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n) * 0x8))
+
+/* 5.23 Performance monitoring event selectors (31 * 64bits) */
+#define RISCV_IOMMU_REG_IOHPMEVT_BASE	0x0160
+#define RISCV_IOMMU_REG_IOHPMEVT(_n)	(RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
+#define RISCV_IOMMU_IOHPMEVT_CNT	31
+#define RISCV_IOMMU_IOHPMEVT_EVENT_ID	GENMASK_ULL(14, 0)
+#define RISCV_IOMMU_IOHPMEVT_DMASK	BIT_ULL(15)
+#define RISCV_IOMMU_IOHPMEVT_PID_PSCID	GENMASK_ULL(35, 16)
+#define RISCV_IOMMU_IOHPMEVT_DID_GSCID	GENMASK_ULL(59, 36)
+#define RISCV_IOMMU_IOHPMEVT_PV_PSCV	BIT_ULL(60)
+#define RISCV_IOMMU_IOHPMEVT_DV_GSCV	BIT_ULL(61)
+#define RISCV_IOMMU_IOHPMEVT_IDT	BIT_ULL(62)
+#define RISCV_IOMMU_IOHPMEVT_OF		BIT_ULL(63)
+
+/**
+ * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
+ *
+ * @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count
+ * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
+ * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
+ * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
+ * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
+ * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
+ * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
+ * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: S/VS-Stage page table walks
+ * @RISCV_IOMMU_HPMEVENT_G_WALKS: G-Stage page table walks
+ * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
+ */
+enum riscv_iommu_hpmevent_id {
+	RISCV_IOMMU_HPMEVENT_INVALID    = 0,
+	RISCV_IOMMU_HPMEVENT_URQ        = 1,
+	RISCV_IOMMU_HPMEVENT_TRQ        = 2,
+	RISCV_IOMMU_HPMEVENT_ATS_RQ     = 3,
+	RISCV_IOMMU_HPMEVENT_TLB_MISS   = 4,
+	RISCV_IOMMU_HPMEVENT_DD_WALK    = 5,
+	RISCV_IOMMU_HPMEVENT_PD_WALK    = 6,
+	RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
+	RISCV_IOMMU_HPMEVENT_G_WALKS    = 8,
+	RISCV_IOMMU_HPMEVENT_MAX        = 9
+};
+
+/* 5.24 Translation request IOVA (64bits) */
+#define RISCV_IOMMU_REG_TR_REQ_IOVA     0x0258
+#define RISCV_IOMMU_TR_REQ_IOVA_VPN	GENMASK_ULL(63, 12)
+
+/* 5.25 Translation request control (64bits) */
+#define RISCV_IOMMU_REG_TR_REQ_CTL	0x0260
+#define RISCV_IOMMU_TR_REQ_CTL_GO_BUSY	BIT_ULL(0)
+#define RISCV_IOMMU_TR_REQ_CTL_PRIV	BIT_ULL(1)
+#define RISCV_IOMMU_TR_REQ_CTL_EXE	BIT_ULL(2)
+#define RISCV_IOMMU_TR_REQ_CTL_NW	BIT_ULL(3)
+#define RISCV_IOMMU_TR_REQ_CTL_PID	GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_TR_REQ_CTL_PV	BIT_ULL(32)
+#define RISCV_IOMMU_TR_REQ_CTL_DID	GENMASK_ULL(63, 40)
+
+/* 5.26 Translation request response (64bits) */
+#define RISCV_IOMMU_REG_TR_RESPONSE	0x0268
+#define RISCV_IOMMU_TR_RESPONSE_FAULT	BIT_ULL(0)
+#define RISCV_IOMMU_TR_RESPONSE_PBMT	GENMASK_ULL(8, 7)
+#define RISCV_IOMMU_TR_RESPONSE_SZ	BIT_ULL(9)
+#define RISCV_IOMMU_TR_RESPONSE_PPN	RISCV_IOMMU_PPN_FIELD
+
+/* 5.27 Interrupt cause to vector (64bits) */
+#define RISCV_IOMMU_REG_IVEC		0x02F8
+#define RISCV_IOMMU_IVEC_CIV		GENMASK_ULL(3, 0)
+#define RISCV_IOMMU_IVEC_FIV		GENMASK_ULL(7, 4)
+#define RISCV_IOMMU_IVEC_PMIV		GENMASK_ULL(11, 8)
+#define RISCV_IOMMU_IVEC_PIV		GENMASK_ULL(15, 12)
+
+/* 5.28 MSI Configuration table (32 * 64bits) */
+#define RISCV_IOMMU_REG_MSI_CONFIG	0x0300
+#define RISCV_IOMMU_REG_MSI_ADDR(_n)	(RISCV_IOMMU_REG_MSI_CONFIG + ((_n) * 0x10))
+#define RISCV_IOMMU_MSI_ADDR		GENMASK_ULL(55, 2)
+#define RISCV_IOMMU_REG_MSI_DATA(_n)	(RISCV_IOMMU_REG_MSI_CONFIG + ((_n) * 0x10) + 0x08)
+#define RISCV_IOMMU_MSI_DATA		GENMASK_ULL(31, 0)
+#define RISCV_IOMMU_REG_MSI_VEC_CTL(_n)	(RISCV_IOMMU_REG_MSI_CONFIG + ((_n) * 0x10) + 0x0C)
+#define RISCV_IOMMU_MSI_VEC_CTL_M	BIT_ULL(0)
+
+#define RISCV_IOMMU_REG_SIZE	0x1000
+
+/*
+ * Chapter 2: Data structures
+ */
+
+/*
+ * Device Directory Table macros for non-leaf nodes
+ */
+#define RISCV_IOMMU_DDTE_VALID	BIT_ULL(0)
+#define RISCV_IOMMU_DDTE_PPN	RISCV_IOMMU_PPN_FIELD
+
+/**
+ * struct riscv_iommu_dc - Device Context
+ * @tc: Translation Control
+ * @iohgatp: I/O Hypervisor guest address translation and protection
+ *	     (Second stage context)
+ * @ta: Translation Attributes
+ * @fsc: First stage context
+ * @msiptp: MSI page table pointer
+ * @msi_addr_mask: MSI address mask
+ * @msi_addr_pattern: MSI address pattern
+ * @_reserved: Reserved for future use, padding
+ *
+ * This structure is used for leaf nodes on the Device Directory Table,
+ * in case RISCV_IOMMU_CAP_MSI_FLAT is not set, the bottom 4 fields are
+ * not present and are skipped with pointer arithmetic to avoid
+ * casting, check out riscv_iommu_get_dc().
+ * See section 2.1 for more details
+ */
+struct riscv_iommu_dc {
+	u64 tc;
+	u64 iohgatp;
+	u64 ta;
+	u64 fsc;
+	u64 msiptp;
+	u64 msi_addr_mask;
+	u64 msi_addr_pattern;
+	u64 _reserved;
+};
+
+/* Translation control fields */
+#define RISCV_IOMMU_DC_TC_V		BIT_ULL(0)
+#define RISCV_IOMMU_DC_TC_EN_ATS	BIT_ULL(1)
+#define RISCV_IOMMU_DC_TC_EN_PRI	BIT_ULL(2)
+#define RISCV_IOMMU_DC_TC_T2GPA		BIT_ULL(3)
+#define RISCV_IOMMU_DC_TC_DTF		BIT_ULL(4)
+#define RISCV_IOMMU_DC_TC_PDTV		BIT_ULL(5)
+#define RISCV_IOMMU_DC_TC_PRPR		BIT_ULL(6)
+#define RISCV_IOMMU_DC_TC_GADE		BIT_ULL(7)
+#define RISCV_IOMMU_DC_TC_SADE		BIT_ULL(8)
+#define RISCV_IOMMU_DC_TC_DPE		BIT_ULL(9)
+#define RISCV_IOMMU_DC_TC_SBE		BIT_ULL(10)
+#define RISCV_IOMMU_DC_TC_SXL		BIT_ULL(11)
+
+/* Second-stage (aka G-stage) context fields */
+#define RISCV_IOMMU_DC_IOHGATP_PPN	RISCV_IOMMU_ATP_PPN_FIELD
+#define RISCV_IOMMU_DC_IOHGATP_GSCID	GENMASK_ULL(59, 44)
+#define RISCV_IOMMU_DC_IOHGATP_MODE	RISCV_IOMMU_ATP_MODE_FIELD
+
+/**
+ * enum riscv_iommu_dc_iohgatp_modes - Guest address translation/protection modes
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_BARE: No translation/protection
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4: Sv32x4 (2-bit extension of Sv32), when fctl.GXL == 1
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4: Sv39x4 (2-bit extension of Sv39), when fctl.GXL == 0
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4: Sv48x4 (2-bit extension of Sv48), when fctl.GXL == 0
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4: Sv57x4 (2-bit extension of Sv57), when fctl.GXL == 0
+ */
+enum riscv_iommu_dc_iohgatp_modes {
+	RISCV_IOMMU_DC_IOHGATP_MODE_BARE = 0,
+	RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4 = 8,
+	RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4 = 8,
+	RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4 = 9,
+	RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4 = 10
+};
+
+/* Translation attributes fields */
+#define RISCV_IOMMU_DC_TA_PSCID		GENMASK_ULL(31, 12)
+
+/* First-stage context fields */
+#define RISCV_IOMMU_DC_FSC_PPN		RISCV_IOMMU_ATP_PPN_FIELD
+#define RISCV_IOMMU_DC_FSC_MODE		RISCV_IOMMU_ATP_MODE_FIELD
+
+/**
+ * enum riscv_iommu_dc_fsc_atp_modes - First stage address translation/protection modes
+ * @RISCV_IOMMU_DC_FSC_MODE_BARE: No translation/protection
+ * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32: Sv32, when dc.tc.SXL == 1
+ * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39: Sv39, when dc.tc.SXL == 0
+ * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48: Sv48, when dc.tc.SXL == 0
+ * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57: Sv57, when dc.tc.SXL == 0
+ * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8: 1lvl PDT, 8bit process ids
+ * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17: 2lvl PDT, 17bit process ids
+ * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20: 3lvl PDT, 20bit process ids
+ *
+ * FSC holds IOSATP when RISCV_IOMMU_DC_TC_PDTV is 0 and PDTP otherwise.
+ * IOSATP controls the first stage address translation (same as the satp register on
+ * the RISC-V MMU), and PDTP holds the process directory table, used to select a
+ * first stage page table based on a process id (for devices that support multiple
+ * process ids).
+ */
+enum riscv_iommu_dc_fsc_atp_modes {
+	RISCV_IOMMU_DC_FSC_MODE_BARE = 0,
+	RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32 = 8,
+	RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39 = 8,
+	RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48 = 9,
+	RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57 = 10,
+	RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8 = 1,
+	RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17 = 2,
+	RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20 = 3
+};
+
+/* MSI page table pointer */
+#define RISCV_IOMMU_DC_MSIPTP_PPN	RISCV_IOMMU_ATP_PPN_FIELD
+#define RISCV_IOMMU_DC_MSIPTP_MODE	RISCV_IOMMU_ATP_MODE_FIELD
+#define RISCV_IOMMU_DC_MSIPTP_MODE_OFF	0
+#define RISCV_IOMMU_DC_MSIPTP_MODE_FLAT	1
+
+/* MSI address mask */
+#define RISCV_IOMMU_DC_MSI_ADDR_MASK	GENMASK_ULL(51, 0)
+
+/* MSI address pattern */
+#define RISCV_IOMMU_DC_MSI_PATTERN	GENMASK_ULL(51, 0)
+
+/**
+ * struct riscv_iommu_pc - Process Context
+ * @ta: Translation Attributes
+ * @fsc: First stage context
+ *
+ * This structure is used for leaf nodes on the Process Directory Table
+ * See section 2.3 for more details
+ */
+struct riscv_iommu_pc {
+	u64 ta;
+	u64 fsc;
+};
+
+/* Translation attributes fields */
+#define RISCV_IOMMU_PC_TA_V	BIT_ULL(0)
+#define RISCV_IOMMU_PC_TA_ENS	BIT_ULL(1)
+#define RISCV_IOMMU_PC_TA_SUM	BIT_ULL(2)
+#define RISCV_IOMMU_PC_TA_PSCID	GENMASK_ULL(31, 12)
+
+/* First stage context fields */
+#define RISCV_IOMMU_PC_FSC_PPN	RISCV_IOMMU_ATP_PPN_FIELD
+#define RISCV_IOMMU_PC_FSC_MODE	RISCV_IOMMU_ATP_MODE_FIELD
+
+/*
+ * Chapter 3: In-memory queue interface
+ */
+
+/**
+ * struct riscv_iommu_command - Generic I/O MMU command structure
+ * @dword0: Includes the opcode and the function identifier
+ * @dword1: Opcode specific data
+ *
+ * The commands are interpreted as two 64bit fields, where the first
+ * 7bits of the first field are the opcode which also defines the
+ * command's format, followed by a 3bit field that specifies the
+ * function invoked by that command, and the rest is opcode-specific.
+ * This is a generic struct which will be populated differently
+ * according to each command. For more infos on the commands and
+ * the command queue check section 3.1.
+ */
+struct riscv_iommu_command {
+	u64 dword0;
+	u64 dword1;
+};
+
+/* Fields on dword0, common for all commands */
+#define RISCV_IOMMU_CMD_OPCODE	GENMASK_ULL(6, 0)
+#define	RISCV_IOMMU_CMD_FUNC	GENMASK_ULL(9, 7)
+
+/* 3.1.1 I/O MMU Page-table cache invalidation */
+/* Fields on dword0 */
+#define RISCV_IOMMU_CMD_IOTINVAL_OPCODE		1
+#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA	0
+#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA	1
+#define RISCV_IOMMU_CMD_IOTINVAL_AV		BIT_ULL(10)
+#define RISCV_IOMMU_CMD_IOTINVAL_PSCID		GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_CMD_IOTINVAL_PSCV		BIT_ULL(32)
+#define RISCV_IOMMU_CMD_IOTINVAL_GV		BIT_ULL(33)
+#define RISCV_IOMMU_CMD_IOTINVAL_GSCID		GENMASK_ULL(59, 44)
+/* dword1[61:10] is the 4K-aligned page address */
+#define RISCV_IOMMU_CMD_IOTINVAL_ADDR		GENMASK_ULL(61, 10)
+
+/* 3.1.2 I/O MMU Command Queue Fences */
+/* Fields on dword0 */
+#define RISCV_IOMMU_CMD_IOFENCE_OPCODE		2
+#define RISCV_IOMMU_CMD_IOFENCE_FUNC_C		0
+#define RISCV_IOMMU_CMD_IOFENCE_AV		BIT_ULL(10)
+#define RISCV_IOMMU_CMD_IOFENCE_WSI		BIT_ULL(11)
+#define RISCV_IOMMU_CMD_IOFENCE_PR		BIT_ULL(12)
+#define RISCV_IOMMU_CMD_IOFENCE_PW		BIT_ULL(13)
+#define RISCV_IOMMU_CMD_IOFENCE_DATA		GENMASK_ULL(63, 32)
+/* dword1 is the address, word-size aligned and shifted to the right by two bits. */
+
+/* 3.1.3 I/O MMU Directory cache invalidation */
+/* Fields on dword0 */
+#define RISCV_IOMMU_CMD_IODIR_OPCODE		3
+#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT	0
+#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT	1
+#define RISCV_IOMMU_CMD_IODIR_PID		GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_CMD_IODIR_DV		BIT_ULL(33)
+#define RISCV_IOMMU_CMD_IODIR_DID		GENMASK_ULL(63, 40)
+/* dword1 is reserved for standard use */
+
+/* 3.1.4 I/O MMU PCIe ATS */
+/* Fields on dword0 */
+#define RISCV_IOMMU_CMD_ATS_OPCODE		4
+#define RISCV_IOMMU_CMD_ATS_FUNC_INVAL		0
+#define RISCV_IOMMU_CMD_ATS_FUNC_PRGR		1
+#define RISCV_IOMMU_CMD_ATS_PID			GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_CMD_ATS_PV			BIT_ULL(32)
+#define RISCV_IOMMU_CMD_ATS_DSV			BIT_ULL(33)
+#define RISCV_IOMMU_CMD_ATS_RID			GENMASK_ULL(55, 40)
+#define RISCV_IOMMU_CMD_ATS_DSEG		GENMASK_ULL(63, 56)
+/* dword1 is the ATS payload, two different payload types for INVAL and PRGR */
+
+/* ATS.INVAL payload*/
+#define RISCV_IOMMU_CMD_ATS_INVAL_G		BIT_ULL(0)
+/* Bits 1 - 10 are zeroed */
+#define RISCV_IOMMU_CMD_ATS_INVAL_S		BIT_ULL(11)
+#define RISCV_IOMMU_CMD_ATS_INVAL_UADDR		GENMASK_ULL(63, 12)
+
+/* ATS.PRGR payload */
+/* Bits 0 - 31 are zeroed */
+#define RISCV_IOMMU_CMD_ATS_PRGR_PRG_INDEX	GENMASK_ULL(40, 32)
+/* Bits 41 - 43 are zeroed */
+#define RISCV_IOMMU_CMD_ATS_PRGR_RESP_CODE	GENMASK_ULL(47, 44)
+#define RISCV_IOMMU_CMD_ATS_PRGR_DST_ID		GENMASK_ULL(63, 48)
+
+/**
+ * struct riscv_iommu_fq_record - Fault/Event Queue Record
+ * @hdr: Header, includes fault/event cause, PID/DID, transaction type etc
+ * @_reserved: Low 32bits for custom use, high 32bits for standard use
+ * @iotval: Transaction-type/cause specific format
+ * @iotval2: Cause specific format
+ *
+ * The fault/event queue reports events and failures raised when
+ * processing transactions. Each record is a 32byte structure where
+ * the first dword has a fixed format for providing generic infos
+ * regarding the fault/event, and two more dwords are there for
+ * fault/event-specific information. For more details see section
+ * 3.2.
+ */
+struct riscv_iommu_fq_record {
+	u64 hdr;
+	u64 _reserved;
+	u64 iotval;
+	u64 iotval2;
+};
+
+/* Fields on header */
+#define RISCV_IOMMU_FQ_HDR_CAUSE	GENMASK_ULL(11, 0)
+#define RISCV_IOMMU_FQ_HDR_PID		GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_FQ_HDR_PV		BIT_ULL(32)
+#define RISCV_IOMMU_FQ_HDR_PRIV		BIT_ULL(33)
+#define RISCV_IOMMU_FQ_HDR_TTYPE	GENMASK_ULL(39, 34)
+#define RISCV_IOMMU_FQ_HDR_DID		GENMASK_ULL(63, 40)
+
+/**
+ * enum riscv_iommu_fq_causes - Fault/event cause values
+ * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT: Instruction access fault
+ * @RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED: Read address misaligned
+ * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT: Read load fault
+ * @RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED: Write/AMO address misaligned
+ * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT: Write/AMO access fault
+ * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S: Instruction page fault
+ * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S: Read page fault
+ * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S: Write/AMO page fault
+ * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS: Instruction guest page fault
+ * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS: Read guest page fault
+ * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS: Write/AMO guest page fault
+ * @RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED: All inbound transactions disallowed
+ * @RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT: DDT entry load access fault
+ * @RISCV_IOMMU_FQ_CAUSE_DDT_INVALID: DDT entry invalid
+ * @RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED: DDT entry misconfigured
+ * @RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED: Transaction type disallowed
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT: MSI PTE load access fault
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_INVALID: MSI PTE invalid
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED: MSI PTE misconfigured
+ * @RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT: MRIF access fault
+ * @RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT: PDT entry load access fault
+ * @RISCV_IOMMU_FQ_CAUSE_PDT_INVALID: PDT entry invalid
+ * @RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED: PDT entry misconfigured
+ * @RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED: DDT data corruption
+ * @RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED: PDT data corruption
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED: MSI page table data corruption
+ * @RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED: MRIF data corruption
+ * @RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR: Internal data path error
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT: IOMMU MSI write access fault
+ * @RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED: First/second stage page table data corruption
+ *
+ * Values are on table 11 of the spec, encodings 275 - 2047 are reserved for standard
+ * use, and 2048 - 4095 for custom use.
+ */
+enum riscv_iommu_fq_causes {
+	RISCV_IOMMU_FQ_CAUSE_INST_FAULT = 1,
+	RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED = 4,
+	RISCV_IOMMU_FQ_CAUSE_RD_FAULT = 5,
+	RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED = 6,
+	RISCV_IOMMU_FQ_CAUSE_WR_FAULT = 7,
+	RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S = 12,
+	RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S = 13,
+	RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S = 15,
+	RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS = 20,
+	RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS = 21,
+	RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS = 23,
+	RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED = 256,
+	RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT = 257,
+	RISCV_IOMMU_FQ_CAUSE_DDT_INVALID = 258,
+	RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED = 259,
+	RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED = 260,
+	RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT = 261,
+	RISCV_IOMMU_FQ_CAUSE_MSI_INVALID = 262,
+	RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED = 263,
+	RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT = 264,
+	RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT = 265,
+	RISCV_IOMMU_FQ_CAUSE_PDT_INVALID = 266,
+	RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED = 267,
+	RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED = 268,
+	RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED = 269,
+	RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED = 270,
+	RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED = 271,
+	RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR = 272,
+	RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT = 273,
+	RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED = 274
+};
+
+/**
+ * enum riscv_iommu_fq_ttypes: Fault/event transaction types
+ * @RISCV_IOMMU_FQ_TTYPE_NONE: None. Fault not caused by an inbound transaction.
+ * @RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH: Instruction fetch from untranslated address
+ * @RISCV_IOMMU_FQ_TTYPE_UADDR_RD: Read from untranslated address
+ * @RISCV_IOMMU_FQ_TTYPE_UADDR_WR: Write/AMO to untranslated address
+ * @RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH: Instruction fetch from translated address
+ * @RISCV_IOMMU_FQ_TTYPE_TADDR_RD: Read from translated address
+ * @RISCV_IOMMU_FQ_TTYPE_TADDR_WR: Write/AMO to translated address
+ * @RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ: PCIe ATS translation request
+ * @RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ: PCIe message request
+ *
+ * Values are on table 12 of the spec, type 4 and 10 - 31 are reserved for standard use
+ * and 31 - 63 for custom use.
+ */
+enum riscv_iommu_fq_ttypes {
+	RISCV_IOMMU_FQ_TTYPE_NONE = 0,
+	RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH = 1,
+	RISCV_IOMMU_FQ_TTYPE_UADDR_RD = 2,
+	RISCV_IOMMU_FQ_TTYPE_UADDR_WR = 3,
+	RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH = 5,
+	RISCV_IOMMU_FQ_TTYPE_TADDR_RD = 6,
+	RISCV_IOMMU_FQ_TTYPE_TADDR_WR = 7,
+	RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ = 8,
+	RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ = 9,
+};
+
+/**
+ * struct riscv_iommu_pq_record - PCIe Page Request record
+ * @hdr: Header, includes PID, DID etc
+ * @payload: Holds the page address, request group and permission bits
+ *
+ * For more infos on the PCIe Page Request queue see chapter 3.3.
+ */
+struct riscv_iommu_pq_record {
+	u64 hdr;
+	u64 payload;
+};
+
+/* Header fields */
+#define RISCV_IOMMU_PREQ_HDR_PID	GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_PREQ_HDR_PV		BIT_ULL(32)
+#define RISCV_IOMMU_PREQ_HDR_PRIV	BIT_ULL(33)
+#define RISCV_IOMMU_PREQ_HDR_EXEC	BIT_ULL(34)
+#define RISCV_IOMMU_PREQ_HDR_DID	GENMASK_ULL(63, 40)
+
+/* Payload fields */
+#define RISCV_IOMMU_PREQ_PAYLOAD_R	BIT_ULL(0)
+#define RISCV_IOMMU_PREQ_PAYLOAD_W	BIT_ULL(1)
+#define RISCV_IOMMU_PREQ_PAYLOAD_L	BIT_ULL(2)
+#define RISCV_IOMMU_PREQ_PAYLOAD_M	GENMASK_ULL(2, 0)	/* Mask of RWL for convenience */
+#define RISCV_IOMMU_PREQ_PRG_INDEX	GENMASK_ULL(11, 3)
+#define RISCV_IOMMU_PREQ_UADDR		GENMASK_ULL(63, 12)
+
+/**
+ * struct riscv_iommu_msi_pte - MSI Page Table Entry
+ * @pte: MSI PTE
+ * @mrif_info: Memory-resident interrupt file info
+ *
+ * The MSI Page Table is used for virtualizing MSIs, so that when
+ * a device sends an MSI to a guest, the IOMMU can reroute it
+ * by translating the MSI address, either to a guest interrupt file
+ * or a memory resident interrupt file (MRIF). Note that this page table
+ * is an array of MSI PTEs, not a multi-level pt, each entry
+ * is a leaf entry. For more infos check out the AIA spec, chapter 9.5.
+ *
+ * Also in basic mode the mrif_info field is ignored by the IOMMU and can
+ * be used by software, any other reserved fields on pte must be zeroed-out
+ * by software.
+ */
+struct riscv_iommu_msi_pte {
+	u64 pte;
+	u64 mrif_info;
+};
+
+/* Fields on pte */
+#define RISCV_IOMMU_MSI_PTE_V		BIT_ULL(0)
+#define RISCV_IOMMU_MSI_PTE_M		GENMASK_ULL(2, 1)
+#define RISCV_IOMMU_MSI_PTE_MRIF_ADDR	GENMASK_ULL(53, 7)	/* When M == 1 (MRIF mode) */
+#define RISCV_IOMMU_MSI_PTE_PPN		RISCV_IOMMU_PPN_FIELD	/* When M == 3 (basic mode) */
+#define RISCV_IOMMU_MSI_PTE_C		BIT_ULL(63)
+
+/* Fields on mrif_info */
+#define RISCV_IOMMU_MSI_MRIF_NID	GENMASK_ULL(9, 0)
+#define RISCV_IOMMU_MSI_MRIF_NPPN	RISCV_IOMMU_PPN_FIELD
+#define RISCV_IOMMU_MSI_MRIF_NID_MSB	BIT_ULL(60)
+
+#endif /* _RISCV_IOMMU_BITS_H_ */

drivers/iommu/riscv/iommu-platform.c new file +50 -0

new file mode 100644
index 000000000000..770086ae2ab3
@@ -0,0 +1,94 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * RISC-V IOMMU as a platform device
+ *
+ * Copyright © 2023 FORTH-ICS/CARV
+ * Copyright © 2023-2024 Rivos Inc.
+ *
+ * Authors
+ *	Nick Kossifidis <[email protected]>
+ *	Tomasz Jeznach <[email protected]>
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+
+#include "iommu-bits.h"
+#include "iommu.h"
+
+static int riscv_iommu_platform_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct riscv_iommu_device *iommu = NULL;
+	struct resource *res = NULL;
+	int vec;
+
+	iommu = devm_kzalloc(dev, sizeof(*iommu), GFP_KERNEL);
+	if (!iommu)
+		return -ENOMEM;
+
+	iommu->dev = dev;
+	iommu->reg = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
+	if (IS_ERR(iommu->reg))
+		return dev_err_probe(dev, PTR_ERR(iommu->reg),
+				     "could not map register region\n");
+
+	dev_set_drvdata(dev, iommu);
+
+	/* Check device reported capabilities / features. */
+	iommu->caps = riscv_iommu_readq(iommu, RISCV_IOMMU_REG_CAP);
+	iommu->fctl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL);
+
+	/* For now we only support WSI */
+	switch (FIELD_GET(RISCV_IOMMU_CAP_IGS, iommu->caps)) {
+	case RISCV_IOMMU_CAP_IGS_WSI:
+	case RISCV_IOMMU_CAP_IGS_BOTH:
+		break;
+	default:
+		return dev_err_probe(dev, -ENODEV,

drivers/iommu/riscv/iommu-platform.c line 60

Quoted 10 lines

drivers/iommu/riscv/iommu-platform.c new file +10 -0

+				     "unable to use wire-signaled interrupts\n");
+	}
+
+	iommu->irqs_count = platform_irq_count(pdev);
+	if (iommu->irqs_count <= 0)
+		return dev_err_probe(dev, -ENODEV,
+				     "no IRQ resources provided\n");
+
+	for (vec = 0; vec < iommu->irqs_count; vec++)
+		iommu->irqs[vec] = platform_get_irq(pdev, vec);

And if I've specified 97 interrupts in my DT because I won't let schema be the boss of me?

drivers/iommu/riscv/iommu-platform.c line 64

Quoted 3 lines

drivers/iommu/riscv/iommu-platform.c new file +3 -0

+	/* Enable wire-signaled interrupts, fctl.WSI */
+	if (!(iommu->fctl & RISCV_IOMMU_FCTL_WSI)) {
+		iommu->fctl ^= RISCV_IOMMU_FCTL_WSI;

Using XOR to only ever set a 0 bit to 1 seems a bit obtuse.

Quoted 7 lines

drivers/iommu/riscv/iommu-platform.c new file +7 -0

+		riscv_iommu_writel(iommu, RISCV_IOMMU_REG_FCTL, iommu->fctl);
+	}
+
+	return riscv_iommu_init(iommu);
+};
+
+static void riscv_iommu_platform_remove(struct platform_device *pdev)

drivers/iommu/riscv/iommu-platform.c line 81

Quoted 10 lines

drivers/iommu/riscv/iommu-platform.c new file +10 -0

+{
+	riscv_iommu_remove(dev_get_drvdata(&pdev->dev));
+};
+
+static const struct of_device_id riscv_iommu_of_match[] = {
+	{.compatible = "riscv,iommu",},
+	{},
+};
+
+MODULE_DEVICE_TABLE(of, riscv_iommu_of_match);

And yet it cannot be a module?

drivers/iommu/riscv/iommu-platform.c line 94

Quoted 12 lines

drivers/iommu/riscv/iommu-platform.c new file +12 -0

+static struct platform_driver riscv_iommu_platform_driver = {
+	.probe = riscv_iommu_platform_probe,
+	.remove_new = riscv_iommu_platform_remove,
+	.driver = {
+		.name = "riscv,iommu",
+		.of_match_table = riscv_iommu_of_match,
+		.suppress_bind_attrs = true,
+	},
+};
+
+module_driver(riscv_iommu_platform_driver, platform_driver_register,
+	      platform_driver_unregister);

module_platform_driver() is a thing. Or builtin_platform_driver(), as things currently stand.

Quoted 9 lines

drivers/iommu/riscv/iommu.c new file +3 -0

new file mode 100644
index 000000000000..af68c89200a9
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * IOMMU API for RISC-V IOMMU implementations.

drivers/iommu/riscv/iommu.c line 13

Quoted 10 lines

drivers/iommu/riscv/iommu.c new file +10 -0

+ *
+ * Copyright © 2022-2024 Rivos Inc.
+ * Copyright © 2023 FORTH-ICS/CARV
+ *
+ * Authors
+ *	Tomasz Jeznach <[email protected]>
+ *	Nick Kossifidis <[email protected]>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

I guess that ends up as "iommu:"? Given that that prefix already belongs to the core code, please pick something more specific.

Quoted 18 lines

drivers/iommu/riscv/iommu.c new file +18 -0

+#include <linux/compiler.h>
+#include <linux/dma-mapping.h>
+#include <linux/init.h>
+#include <linux/iommu.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "iommu-bits.h"
+#include "iommu.h"
+
+MODULE_DESCRIPTION("Driver for RISC-V IOMMU");
+MODULE_AUTHOR("Tomasz Jeznach <[email protected]>");
+MODULE_AUTHOR("Nick Kossifidis <[email protected]>");
+MODULE_LICENSE("GPL");
+
+/* Timeouts in [us] */
+#define RISCV_IOMMU_DDTP_TIMEOUT	50000
+

drivers/iommu/riscv/iommu.c line 42

Quoted 10 lines

drivers/iommu/riscv/iommu.c new file +10 -0

+static int riscv_iommu_init_check(struct riscv_iommu_device *iommu)
+{
+	u64 ddtp;
+
+	/* Hardware must be configured in OFF | BARE mode at system initialization. */
+	riscv_iommu_readq_timeout(iommu, RISCV_IOMMU_REG_DDTP,
+				  ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+				  10, RISCV_IOMMU_DDTP_TIMEOUT);
+	if (FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp) > RISCV_IOMMU_DDTP_MODE_BARE)
+		return -EBUSY;

It looks like RISC-V already supports kdump, so you probably want to be prepared to find the IOMMU with its pants down and deal with it from day one.

drivers/iommu/riscv/iommu.c line 53

Quoted 11 lines

drivers/iommu/riscv/iommu.c new file +11 -0

+
+	/* Configure accesses to in-memory data structures for CPU-native byte order. */
+	if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN) != !!(iommu->fctl & RISCV_IOMMU_FCTL_BE)) {
+		if (!(iommu->caps & RISCV_IOMMU_CAP_END))
+			return -EINVAL;
+		riscv_iommu_writel(iommu, RISCV_IOMMU_REG_FCTL,
+				   iommu->fctl ^ RISCV_IOMMU_FCTL_BE);
+		iommu->fctl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL);
+		if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN) != !!(iommu->fctl & RISCV_IOMMU_FCTL_BE))
+			return -EINVAL;
+	}

That's fun. It could be reorganised to avoid the duplicate check, but it is rather majestic as-is :)

drivers/iommu/riscv/iommu.c line 56

Quoted 3 lines

drivers/iommu/riscv/iommu.c new file +3 -0

+
+	dma_set_mask_and_coherent(iommu->dev,
+				  DMA_BIT_MASK(FIELD_GET(RISCV_IOMMU_CAP_PAS, iommu->caps)));

This isn't a check, so I would think it belongs to "_init" rather than "_init_check".

Thanks,
Robin.

Quoted 101 lines

drivers/iommu/riscv/iommu.c new file +33 -0

+
+	return 0;
+}
+
+void riscv_iommu_remove(struct riscv_iommu_device *iommu)
+{
+	iommu_device_sysfs_remove(&iommu->iommu);
+}
+
+int riscv_iommu_init(struct riscv_iommu_device *iommu)
+{
+	int rc;
+
+	rc = riscv_iommu_init_check(iommu);
+	if (rc)
+		return dev_err_probe(iommu->dev, rc, "unexpected device state\n");
+	/*
+	 * Placeholder for a complete IOMMU device initialization.
+	 * For now, only bare minimum: enable global identity mapping mode and register sysfs.
+	 */
+	riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP,
+			   FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE));
+
+	rc = iommu_device_sysfs_add(&iommu->iommu, NULL, NULL, "riscv-iommu@%s",
+				    dev_name(iommu->dev));
+	if (WARN(rc, "cannot register sysfs interface\n"))
+		goto err_sysfs;
+
+	return 0;
+
+err_sysfs:
+	return rc;
+}

drivers/iommu/riscv/iommu.h new file +62 -0

new file mode 100644
index 000000000000..700e33dc2446
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright © 2022-2024 Rivos Inc.
+ * Copyright © 2023 FORTH-ICS/CARV
+ *
+ * Authors
+ *	Tomasz Jeznach <[email protected]>
+ *	Nick Kossifidis <[email protected]>
+ */
+
+#ifndef _RISCV_IOMMU_H_
+#define _RISCV_IOMMU_H_
+
+#include <linux/iommu.h>
+#include <linux/types.h>
+#include <linux/iopoll.h>
+
+#include "iommu-bits.h"
+
+struct riscv_iommu_device {
+	/* iommu core interface */
+	struct iommu_device iommu;
+
+	/* iommu hardware */
+	struct device *dev;
+
+	/* hardware control register space */
+	void __iomem *reg;
+
+	/* supported and enabled hardware capabilities */
+	u64 caps;
+	u32 fctl;
+
+	/* available interrupt numbers, MSI or WSI */
+	unsigned int irqs[RISCV_IOMMU_INTR_COUNT];
+	unsigned int irqs_count;
+};
+
+int riscv_iommu_init(struct riscv_iommu_device *iommu);
+void riscv_iommu_remove(struct riscv_iommu_device *iommu);
+
+#define riscv_iommu_readl(iommu, addr) \
+	readl_relaxed((iommu)->reg + (addr))
+
+#define riscv_iommu_readq(iommu, addr) \
+	readq_relaxed((iommu)->reg + (addr))
+
+#define riscv_iommu_writel(iommu, addr, val) \
+	writel_relaxed((val), (iommu)->reg + (addr))
+
+#define riscv_iommu_writeq(iommu, addr, val) \
+	writeq_relaxed((val), (iommu)->reg + (addr))
+
+#define riscv_iommu_readq_timeout(iommu, addr, val, cond, delay_us, timeout_us) \
+	readx_poll_timeout(readq_relaxed, (iommu)->reg + (addr), val, cond, \
+			   delay_us, timeout_us)
+
+#define riscv_iommu_readl_timeout(iommu, addr, val, cond, delay_us, timeout_us) \
+	readx_poll_timeout(readl_relaxed, (iommu)->reg + (addr), val, cond, \
+			   delay_us, timeout_us)
+
+#endif

Tomasz Jeznach

author

2024-04-24 21:59 UTC #8 lore raw

Re: [PATCH v2 2/7] iommu/riscv: Add RISC-V IOMMU platform device driver

Quoted 37 lines · Robin Murphy

Quoted 35 lines · Tomasz Jeznach
Introduce platform device driver for implementation of RISC-V IOMMU architected hardware.
Hardware interface definition located in file iommu-bits.h is based on ratified RISC-V IOMMU Architecture Specification version 1.0.0.
This patch implements platform device initialization, early check and configuration of the IOMMU interfaces and enables global pass-through address translation mode (iommu_mode == BARE), without registering hardware instance in the IOMMU subsystem.
Link https://github.com/riscv-non-isa/riscv-iommu
Co-developed-by Nick Kossifidis <[email protected]>
Signed-off-by Nick Kossifidis <[email protected]>
Co-developed-by Sebastien Boeuf <[email protected]>
Signed-off-by Sebastien Boeuf <[email protected]>
Signed-off-by Tomasz Jeznach <[email protected]>
---
  MAINTAINERS                          |   6 +
  drivers/iommu/Kconfig                |   1 +
  drivers/iommu/Makefile               |   2 +-
  drivers/iommu/riscv/Kconfig          |  16 +
  drivers/iommu/riscv/Makefile         |   2 +
  drivers/iommu/riscv/iommu-bits.h     | 707 +++++++++++++++++++++++++++
  drivers/iommu/riscv/iommu-platform.c |  94 ++++
  drivers/iommu/riscv/iommu.c          |  89 ++++
  drivers/iommu/riscv/iommu.h          |  62 +++
  9 files changed, 978 insertions(+), 1 deletion(-)
create mode 100644 drivers/iommu/riscv/Kconfig create mode 100644 drivers/iommu/riscv/Makefile create mode 100644 drivers/iommu/riscv/iommu-bits.h create mode 100644 drivers/iommu/riscv/iommu-platform.c create mode 100644 drivers/iommu/riscv/iommu.c create mode 100644 drivers/iommu/riscv/iommu.h

Quoted 17 lines

Quoted 5 lines

MAINTAINERS +0 -0

index 2657f9eae84c..051599c76585 100644

MAINTAINERS L: [email protected] · line 18980

Quoted 9 lines

MAINTAINERS +6 -0

@@ -18972,6 +18972,12 @@ L:   [email protected]
  L:  [email protected]
  S:  Maintained
  F:  Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
+F:   drivers/iommu/riscv/Kconfig
+F:   drivers/iommu/riscv/Makefile
+F:   drivers/iommu/riscv/iommu-bits.h
+F:   drivers/iommu/riscv/iommu-platform.c
+F:   drivers/iommu/riscv/iommu.c
+F:   drivers/iommu/riscv/iommu.h

I'm pretty sure a single "F: drivers/iommu/riscv/" pattern will suffice.

Correct. But will required a workaround for pretty naive MAINTAINERS update check in scripts/checkpatch.pl:3014 in next patch.

Quoted 37 lines

Quoted 31 lines

MAINTAINERS +0 -0

  RISC-V MICROCHIP FPGA SUPPORT
  M:  Conor Dooley <[email protected]>

drivers/iommu/Kconfig +1 -0

index 0af39bbbe3a3..ae762db0365e 100644
@@ -195,6 +195,7 @@ config MSM_IOMMU
  source "drivers/iommu/amd/Kconfig"
  source "drivers/iommu/intel/Kconfig"
  source "drivers/iommu/iommufd/Kconfig"
+source "drivers/iommu/riscv/Kconfig"
 
  config IRQ_REMAP
      bool "Support for Interrupt Remapping"

drivers/iommu/Makefile +1 -1

index 542760d963ec..5e5a83c6c2aa 100644
@@ -1,5 +1,5 @@
  # SPDX-License-Identifier: GPL-2.0
-obj-y += amd/ intel/ arm/ iommufd/
+obj-y += amd/ intel/ arm/ iommufd/ riscv/
  obj-$(CONFIG_IOMMU_API) += iommu.o
  obj-$(CONFIG_IOMMU_API) += iommu-traces.o
  obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o

drivers/iommu/riscv/Kconfig new file +0 -0

new file mode 100644
index 000000000000..d02326bddb4c

drivers/iommu/riscv/Kconfig line 5

Quoted 5 lines

drivers/iommu/riscv/Kconfig new file +5 -0

@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# RISC-V IOMMU support
+
+config RISCV_IOMMU
+     def_bool y if RISCV && 64BIT && MMU

Quoted 8 lines

Drop the dependencies here, they're already dependencies. However, also consider allowing users to configure this out without disabling IOMMU_SUPPORT entirely - I imagine other IOMMU implementations are going to end up paired wiothg RISC-V CPUs sooner or later. Furthermore, if it's a regular driver model driver, consider allowing it to build as a module. Not to mention that the help text below is rather pointless if there's no prompt offered in the first place.

Ack. Changed to optional builtin driver for now, also cleaned up all module info.
I'll revisit proper module build later.

Quoted 898 lines

drivers/iommu/riscv/Kconfig line 7

Quoted 2 lines

drivers/iommu/riscv/Kconfig new file +2 -0
+     depends on RISCV && 64BIT && MMU
+     select DMA_OPS

Drop this, you're not (and shouldn't be) architecture code implementing DMA ops.

drivers/iommu/riscv/Kconfig line 9

Quoted 2 lines

drivers/iommu/riscv/Kconfig new file +2 -0
+     select IOMMU_API
+     select IOMMU_IOVA

Drop this, you're not using the IOVA library either.

Quoted 784 lines

drivers/iommu/riscv/Kconfig new file +7 -0

+     help
+       Support for implementations of the RISC-V IOMMU architecture that
+       complements the RISC-V MMU capabilities, providing similar address
+       translation and protection functions for accesses from I/O devices.
+
+       Say Y here if your SoC includes an IOMMU device implementing
+       the RISC-V IOMMU architecture.

drivers/iommu/riscv/Makefile new file +2 -0

new file mode 100644
index 000000000000..e4c189de58d3
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-platform.o

drivers/iommu/riscv/iommu-bits.h new file +707 -0

new file mode 100644
index 000000000000..ba093c29de9f
@@ -0,0 +1,707 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright © 2022-2024 Rivos Inc.
+ * Copyright © 2023 FORTH-ICS/CARV
+ * Copyright © 2023 RISC-V IOMMU Task Group
+ *
+ * RISC-V IOMMU - Register Layout and Data Structures.
+ *
+ * Based on the 'RISC-V IOMMU Architecture Specification', Version 1.0
+ * Published at  https://github.com/riscv-non-isa/riscv-iommu
+ *
+ */
+
+#ifndef _RISCV_IOMMU_BITS_H_
+#define _RISCV_IOMMU_BITS_H_
+
+#include <linux/types.h>
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+
+/*
+ * Chapter 5: Memory Mapped register interface
+ */
+
+/* Common field positions */
+#define RISCV_IOMMU_PPN_FIELD                GENMASK_ULL(53, 10)
+#define RISCV_IOMMU_QUEUE_LOGSZ_FIELD        GENMASK_ULL(4, 0)
+#define RISCV_IOMMU_QUEUE_INDEX_FIELD        GENMASK_ULL(31, 0)
+#define RISCV_IOMMU_QUEUE_ENABLE     BIT(0)
+#define RISCV_IOMMU_QUEUE_INTR_ENABLE        BIT(1)
+#define RISCV_IOMMU_QUEUE_MEM_FAULT  BIT(8)
+#define RISCV_IOMMU_QUEUE_OVERFLOW   BIT(9)
+#define RISCV_IOMMU_QUEUE_ACTIVE     BIT(16)
+#define RISCV_IOMMU_QUEUE_BUSY               BIT(17)
+
+#define RISCV_IOMMU_ATP_PPN_FIELD    GENMASK_ULL(43, 0)
+#define RISCV_IOMMU_ATP_MODE_FIELD   GENMASK_ULL(63, 60)
+
+/* 5.3 IOMMU Capabilities (64bits) */
+#define RISCV_IOMMU_REG_CAP          0x0000
+#define RISCV_IOMMU_CAP_VERSION              GENMASK_ULL(7, 0)
+#define RISCV_IOMMU_CAP_S_SV32               BIT_ULL(8)
+#define RISCV_IOMMU_CAP_S_SV39               BIT_ULL(9)
+#define RISCV_IOMMU_CAP_S_SV48               BIT_ULL(10)
+#define RISCV_IOMMU_CAP_S_SV57               BIT_ULL(11)
+#define RISCV_IOMMU_CAP_SVPBMT               BIT_ULL(15)
+#define RISCV_IOMMU_CAP_G_SV32               BIT_ULL(16)
+#define RISCV_IOMMU_CAP_G_SV39               BIT_ULL(17)
+#define RISCV_IOMMU_CAP_G_SV48               BIT_ULL(18)
+#define RISCV_IOMMU_CAP_G_SV57               BIT_ULL(19)
+#define RISCV_IOMMU_CAP_AMO_MRIF     BIT_ULL(21)
+#define RISCV_IOMMU_CAP_MSI_FLAT     BIT_ULL(22)
+#define RISCV_IOMMU_CAP_MSI_MRIF     BIT_ULL(23)
+#define RISCV_IOMMU_CAP_AMO_HWAD     BIT_ULL(24)
+#define RISCV_IOMMU_CAP_ATS          BIT_ULL(25)
+#define RISCV_IOMMU_CAP_T2GPA                BIT_ULL(26)
+#define RISCV_IOMMU_CAP_END          BIT_ULL(27)
+#define RISCV_IOMMU_CAP_IGS          GENMASK_ULL(29, 28)
+#define RISCV_IOMMU_CAP_HPM          BIT_ULL(30)
+#define RISCV_IOMMU_CAP_DBG          BIT_ULL(31)
+#define RISCV_IOMMU_CAP_PAS          GENMASK_ULL(37, 32)
+#define RISCV_IOMMU_CAP_PD8          BIT_ULL(38)
+#define RISCV_IOMMU_CAP_PD17         BIT_ULL(39)
+#define RISCV_IOMMU_CAP_PD20         BIT_ULL(40)
+
+#define RISCV_IOMMU_CAP_VERSION_VER_MASK     0xF0
+#define RISCV_IOMMU_CAP_VERSION_REV_MASK     0x0F
+
+/**
+ * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
+ * @RISCV_IOMMU_CAP_IGS_MSI: I/O MMU supports only MSI generation
+ * @RISCV_IOMMU_CAP_IGS_WSI: I/O MMU supports only Wired-Signaled interrupt
+ * @RISCV_IOMMU_CAP_IGS_BOTH: I/O MMU supports both MSI and WSI generation
+ * @RISCV_IOMMU_CAP_IGS_RSRV: Reserved for standard use
+ */
+enum riscv_iommu_igs_settings {
+     RISCV_IOMMU_CAP_IGS_MSI = 0,
+     RISCV_IOMMU_CAP_IGS_WSI = 1,
+     RISCV_IOMMU_CAP_IGS_BOTH = 2,
+     RISCV_IOMMU_CAP_IGS_RSRV = 3
+};
+
+/* 5.4 Features control register (32bits) */
+#define RISCV_IOMMU_REG_FCTL         0x0008
+#define RISCV_IOMMU_FCTL_BE          BIT(0)
+#define RISCV_IOMMU_FCTL_WSI         BIT(1)
+#define RISCV_IOMMU_FCTL_GXL         BIT(2)
+
+/* 5.5 Device-directory-table pointer (64bits) */
+#define RISCV_IOMMU_REG_DDTP         0x0010
+#define RISCV_IOMMU_DDTP_MODE                GENMASK_ULL(3, 0)
+#define RISCV_IOMMU_DDTP_BUSY                BIT_ULL(4)
+#define RISCV_IOMMU_DDTP_PPN         RISCV_IOMMU_PPN_FIELD
+
+/**
+ * enum riscv_iommu_ddtp_modes - I/O MMU translation modes
+ * @RISCV_IOMMU_DDTP_MODE_OFF: No inbound transactions allowed
+ * @RISCV_IOMMU_DDTP_MODE_BARE: Pass-through mode
+ * @RISCV_IOMMU_DDTP_MODE_1LVL: One-level DDT
+ * @RISCV_IOMMU_DDTP_MODE_2LVL: Two-level DDT
+ * @RISCV_IOMMU_DDTP_MODE_3LVL: Three-level DDT
+ * @RISCV_IOMMU_DDTP_MODE_MAX: Max value allowed by specification
+ */
+enum riscv_iommu_ddtp_modes {
+     RISCV_IOMMU_DDTP_MODE_OFF = 0,
+     RISCV_IOMMU_DDTP_MODE_BARE = 1,
+     RISCV_IOMMU_DDTP_MODE_1LVL = 2,
+     RISCV_IOMMU_DDTP_MODE_2LVL = 3,
+     RISCV_IOMMU_DDTP_MODE_3LVL = 4,
+     RISCV_IOMMU_DDTP_MODE_MAX = 4
+};
+
+/* 5.6 Command Queue Base (64bits) */
+#define RISCV_IOMMU_REG_CQB          0x0018
+#define RISCV_IOMMU_CQB_ENTRIES              RISCV_IOMMU_QUEUE_LOGSZ_FIELD
+#define RISCV_IOMMU_CQB_PPN          RISCV_IOMMU_PPN_FIELD
+
+/* 5.7 Command Queue head (32bits) */
+#define RISCV_IOMMU_REG_CQH          0x0020
+#define RISCV_IOMMU_CQH_INDEX                RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.8 Command Queue tail (32bits) */
+#define RISCV_IOMMU_REG_CQT          0x0024
+#define RISCV_IOMMU_CQT_INDEX                RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.9 Fault Queue Base (64bits) */
+#define RISCV_IOMMU_REG_FQB          0x0028
+#define RISCV_IOMMU_FQB_ENTRIES              RISCV_IOMMU_QUEUE_LOGSZ_FIELD
+#define RISCV_IOMMU_FQB_PPN          RISCV_IOMMU_PPN_FIELD
+
+/* 5.10 Fault Queue Head (32bits) */
+#define RISCV_IOMMU_REG_FQH          0x0030
+#define RISCV_IOMMU_FQH_INDEX                RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.11 Fault Queue tail (32bits) */
+#define RISCV_IOMMU_REG_FQT          0x0034
+#define RISCV_IOMMU_FQT_INDEX                RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.12 Page Request Queue base (64bits) */
+#define RISCV_IOMMU_REG_PQB          0x0038
+#define RISCV_IOMMU_PQB_ENTRIES              RISCV_IOMMU_QUEUE_LOGSZ_FIELD
+#define RISCV_IOMMU_PQB_PPN          RISCV_IOMMU_PPN_FIELD
+
+/* 5.13 Page Request Queue head (32bits) */
+#define RISCV_IOMMU_REG_PQH          0x0040
+#define RISCV_IOMMU_PQH_INDEX                RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.14 Page Request Queue tail (32bits) */
+#define RISCV_IOMMU_REG_PQT          0x0044
+#define RISCV_IOMMU_PQT_INDEX_MASK   RISCV_IOMMU_QUEUE_INDEX_FIELD
+
+/* 5.15 Command Queue CSR (32bits) */
+#define RISCV_IOMMU_REG_CQCSR                0x0048
+#define RISCV_IOMMU_CQCSR_CQEN               RISCV_IOMMU_QUEUE_ENABLE
+#define RISCV_IOMMU_CQCSR_CIE                RISCV_IOMMU_QUEUE_INTR_ENABLE
+#define RISCV_IOMMU_CQCSR_CQMF               RISCV_IOMMU_QUEUE_MEM_FAULT
+#define RISCV_IOMMU_CQCSR_CMD_TO     BIT(9)
+#define RISCV_IOMMU_CQCSR_CMD_ILL    BIT(10)
+#define RISCV_IOMMU_CQCSR_FENCE_W_IP BIT(11)
+#define RISCV_IOMMU_CQCSR_CQON               RISCV_IOMMU_QUEUE_ACTIVE
+#define RISCV_IOMMU_CQCSR_BUSY               RISCV_IOMMU_QUEUE_BUSY
+
+/* 5.16 Fault Queue CSR (32bits) */
+#define RISCV_IOMMU_REG_FQCSR                0x004C
+#define RISCV_IOMMU_FQCSR_FQEN               RISCV_IOMMU_QUEUE_ENABLE
+#define RISCV_IOMMU_FQCSR_FIE                RISCV_IOMMU_QUEUE_INTR_ENABLE
+#define RISCV_IOMMU_FQCSR_FQMF               RISCV_IOMMU_QUEUE_MEM_FAULT
+#define RISCV_IOMMU_FQCSR_FQOF               RISCV_IOMMU_QUEUE_OVERFLOW
+#define RISCV_IOMMU_FQCSR_FQON               RISCV_IOMMU_QUEUE_ACTIVE
+#define RISCV_IOMMU_FQCSR_BUSY               RISCV_IOMMU_QUEUE_BUSY
+
+/* 5.17 Page Request Queue CSR (32bits) */
+#define RISCV_IOMMU_REG_PQCSR                0x0050
+#define RISCV_IOMMU_PQCSR_PQEN               RISCV_IOMMU_QUEUE_ENABLE
+#define RISCV_IOMMU_PQCSR_PIE                RISCV_IOMMU_QUEUE_INTR_ENABLE
+#define RISCV_IOMMU_PQCSR_PQMF               RISCV_IOMMU_QUEUE_MEM_FAULT
+#define RISCV_IOMMU_PQCSR_PQOF               RISCV_IOMMU_QUEUE_OVERFLOW
+#define RISCV_IOMMU_PQCSR_PQON               RISCV_IOMMU_QUEUE_ACTIVE
+#define RISCV_IOMMU_PQCSR_BUSY               RISCV_IOMMU_QUEUE_BUSY
+
+/* 5.18 Interrupt Pending Status (32bits) */
+#define RISCV_IOMMU_REG_IPSR         0x0054
+
+#define RISCV_IOMMU_INTR_CQ          0
+#define RISCV_IOMMU_INTR_FQ          1
+#define RISCV_IOMMU_INTR_PM          2
+#define RISCV_IOMMU_INTR_PQ          3
+#define RISCV_IOMMU_INTR_COUNT               4
+
+#define RISCV_IOMMU_IPSR_CIP         BIT(RISCV_IOMMU_INTR_CQ)
+#define RISCV_IOMMU_IPSR_FIP         BIT(RISCV_IOMMU_INTR_FQ)
+#define RISCV_IOMMU_IPSR_PMIP                BIT(RISCV_IOMMU_INTR_PM)
+#define RISCV_IOMMU_IPSR_PIP         BIT(RISCV_IOMMU_INTR_PQ)
+
+/* 5.19 Performance monitoring counter overflow status (32bits) */
+#define RISCV_IOMMU_REG_IOCOUNTOVF   0x0058
+#define RISCV_IOMMU_IOCOUNTOVF_CY    BIT(0)
+#define RISCV_IOMMU_IOCOUNTOVF_HPM   GENMASK_ULL(31, 1)
+
+/* 5.20 Performance monitoring counter inhibits (32bits) */
+#define RISCV_IOMMU_REG_IOCOUNTINH   0x005C
+#define RISCV_IOMMU_IOCOUNTINH_CY    BIT(0)
+#define RISCV_IOMMU_IOCOUNTINH_HPM   GENMASK(31, 1)
+
+/* 5.21 Performance monitoring cycles counter (64bits) */
+#define RISCV_IOMMU_REG_IOHPMCYCLES     0x0060
+#define RISCV_IOMMU_IOHPMCYCLES_COUNTER      GENMASK_ULL(62, 0)
+#define RISCV_IOMMU_IOHPMCYCLES_OVF  BIT_ULL(63)
+
+/* 5.22 Performance monitoring event counters (31 * 64bits) */
+#define RISCV_IOMMU_REG_IOHPMCTR_BASE        0x0068
+#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n) * 0x8))
+
+/* 5.23 Performance monitoring event selectors (31 * 64bits) */
+#define RISCV_IOMMU_REG_IOHPMEVT_BASE        0x0160
+#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + ((_n) * 0x8))
+#define RISCV_IOMMU_IOHPMEVT_CNT     31
+#define RISCV_IOMMU_IOHPMEVT_EVENT_ID        GENMASK_ULL(14, 0)
+#define RISCV_IOMMU_IOHPMEVT_DMASK   BIT_ULL(15)
+#define RISCV_IOMMU_IOHPMEVT_PID_PSCID       GENMASK_ULL(35, 16)
+#define RISCV_IOMMU_IOHPMEVT_DID_GSCID       GENMASK_ULL(59, 36)
+#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60)
+#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61)
+#define RISCV_IOMMU_IOHPMEVT_IDT     BIT_ULL(62)
+#define RISCV_IOMMU_IOHPMEVT_OF              BIT_ULL(63)
+
+/**
+ * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier
+ *
+ * @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count
+ * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests
+ * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests
+ * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests
+ * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses
+ * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks
+ * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks
+ * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: S/VS-Stage page table walks
+ * @RISCV_IOMMU_HPMEVENT_G_WALKS: G-Stage page table walks
+ * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs
+ */
+enum riscv_iommu_hpmevent_id {
+     RISCV_IOMMU_HPMEVENT_INVALID    = 0,
+     RISCV_IOMMU_HPMEVENT_URQ        = 1,
+     RISCV_IOMMU_HPMEVENT_TRQ        = 2,
+     RISCV_IOMMU_HPMEVENT_ATS_RQ     = 3,
+     RISCV_IOMMU_HPMEVENT_TLB_MISS   = 4,
+     RISCV_IOMMU_HPMEVENT_DD_WALK    = 5,
+     RISCV_IOMMU_HPMEVENT_PD_WALK    = 6,
+     RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7,
+     RISCV_IOMMU_HPMEVENT_G_WALKS    = 8,
+     RISCV_IOMMU_HPMEVENT_MAX        = 9
+};
+
+/* 5.24 Translation request IOVA (64bits) */
+#define RISCV_IOMMU_REG_TR_REQ_IOVA     0x0258
+#define RISCV_IOMMU_TR_REQ_IOVA_VPN  GENMASK_ULL(63, 12)
+
+/* 5.25 Translation request control (64bits) */
+#define RISCV_IOMMU_REG_TR_REQ_CTL   0x0260
+#define RISCV_IOMMU_TR_REQ_CTL_GO_BUSY       BIT_ULL(0)
+#define RISCV_IOMMU_TR_REQ_CTL_PRIV  BIT_ULL(1)
+#define RISCV_IOMMU_TR_REQ_CTL_EXE   BIT_ULL(2)
+#define RISCV_IOMMU_TR_REQ_CTL_NW    BIT_ULL(3)
+#define RISCV_IOMMU_TR_REQ_CTL_PID   GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_TR_REQ_CTL_PV    BIT_ULL(32)
+#define RISCV_IOMMU_TR_REQ_CTL_DID   GENMASK_ULL(63, 40)
+
+/* 5.26 Translation request response (64bits) */
+#define RISCV_IOMMU_REG_TR_RESPONSE  0x0268
+#define RISCV_IOMMU_TR_RESPONSE_FAULT        BIT_ULL(0)
+#define RISCV_IOMMU_TR_RESPONSE_PBMT GENMASK_ULL(8, 7)
+#define RISCV_IOMMU_TR_RESPONSE_SZ   BIT_ULL(9)
+#define RISCV_IOMMU_TR_RESPONSE_PPN  RISCV_IOMMU_PPN_FIELD
+
+/* 5.27 Interrupt cause to vector (64bits) */
+#define RISCV_IOMMU_REG_IVEC         0x02F8
+#define RISCV_IOMMU_IVEC_CIV         GENMASK_ULL(3, 0)
+#define RISCV_IOMMU_IVEC_FIV         GENMASK_ULL(7, 4)
+#define RISCV_IOMMU_IVEC_PMIV                GENMASK_ULL(11, 8)
+#define RISCV_IOMMU_IVEC_PIV         GENMASK_ULL(15, 12)
+
+/* 5.28 MSI Configuration table (32 * 64bits) */
+#define RISCV_IOMMU_REG_MSI_CONFIG   0x0300
+#define RISCV_IOMMU_REG_MSI_ADDR(_n) (RISCV_IOMMU_REG_MSI_CONFIG + ((_n) * 0x10))
+#define RISCV_IOMMU_MSI_ADDR         GENMASK_ULL(55, 2)
+#define RISCV_IOMMU_REG_MSI_DATA(_n) (RISCV_IOMMU_REG_MSI_CONFIG + ((_n) * 0x10) + 0x08)
+#define RISCV_IOMMU_MSI_DATA         GENMASK_ULL(31, 0)
+#define RISCV_IOMMU_REG_MSI_VEC_CTL(_n)      (RISCV_IOMMU_REG_MSI_CONFIG + ((_n) * 0x10) + 0x0C)
+#define RISCV_IOMMU_MSI_VEC_CTL_M    BIT_ULL(0)
+
+#define RISCV_IOMMU_REG_SIZE 0x1000
+
+/*
+ * Chapter 2: Data structures
+ */
+
+/*
+ * Device Directory Table macros for non-leaf nodes
+ */
+#define RISCV_IOMMU_DDTE_VALID       BIT_ULL(0)
+#define RISCV_IOMMU_DDTE_PPN RISCV_IOMMU_PPN_FIELD
+
+/**
+ * struct riscv_iommu_dc - Device Context
+ * @tc: Translation Control
+ * @iohgatp: I/O Hypervisor guest address translation and protection
+ *        (Second stage context)
+ * @ta: Translation Attributes
+ * @fsc: First stage context
+ * @msiptp: MSI page table pointer
+ * @msi_addr_mask: MSI address mask
+ * @msi_addr_pattern: MSI address pattern
+ * @_reserved: Reserved for future use, padding
+ *
+ * This structure is used for leaf nodes on the Device Directory Table,
+ * in case RISCV_IOMMU_CAP_MSI_FLAT is not set, the bottom 4 fields are
+ * not present and are skipped with pointer arithmetic to avoid
+ * casting, check out riscv_iommu_get_dc().
+ * See section 2.1 for more details
+ */
+struct riscv_iommu_dc {
+     u64 tc;
+     u64 iohgatp;
+     u64 ta;
+     u64 fsc;
+     u64 msiptp;
+     u64 msi_addr_mask;
+     u64 msi_addr_pattern;
+     u64 _reserved;
+};
+
+/* Translation control fields */
+#define RISCV_IOMMU_DC_TC_V          BIT_ULL(0)
+#define RISCV_IOMMU_DC_TC_EN_ATS     BIT_ULL(1)
+#define RISCV_IOMMU_DC_TC_EN_PRI     BIT_ULL(2)
+#define RISCV_IOMMU_DC_TC_T2GPA              BIT_ULL(3)
+#define RISCV_IOMMU_DC_TC_DTF                BIT_ULL(4)
+#define RISCV_IOMMU_DC_TC_PDTV               BIT_ULL(5)
+#define RISCV_IOMMU_DC_TC_PRPR               BIT_ULL(6)
+#define RISCV_IOMMU_DC_TC_GADE               BIT_ULL(7)
+#define RISCV_IOMMU_DC_TC_SADE               BIT_ULL(8)
+#define RISCV_IOMMU_DC_TC_DPE                BIT_ULL(9)
+#define RISCV_IOMMU_DC_TC_SBE                BIT_ULL(10)
+#define RISCV_IOMMU_DC_TC_SXL                BIT_ULL(11)
+
+/* Second-stage (aka G-stage) context fields */
+#define RISCV_IOMMU_DC_IOHGATP_PPN   RISCV_IOMMU_ATP_PPN_FIELD
+#define RISCV_IOMMU_DC_IOHGATP_GSCID GENMASK_ULL(59, 44)
+#define RISCV_IOMMU_DC_IOHGATP_MODE  RISCV_IOMMU_ATP_MODE_FIELD
+
+/**
+ * enum riscv_iommu_dc_iohgatp_modes - Guest address translation/protection modes
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_BARE: No translation/protection
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4: Sv32x4 (2-bit extension of Sv32), when fctl.GXL == 1
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4: Sv39x4 (2-bit extension of Sv39), when fctl.GXL == 0
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4: Sv48x4 (2-bit extension of Sv48), when fctl.GXL == 0
+ * @RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4: Sv57x4 (2-bit extension of Sv57), when fctl.GXL == 0
+ */
+enum riscv_iommu_dc_iohgatp_modes {
+     RISCV_IOMMU_DC_IOHGATP_MODE_BARE = 0,
+     RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4 = 8,
+     RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4 = 8,
+     RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4 = 9,
+     RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4 = 10
+};
+
+/* Translation attributes fields */
+#define RISCV_IOMMU_DC_TA_PSCID              GENMASK_ULL(31, 12)
+
+/* First-stage context fields */
+#define RISCV_IOMMU_DC_FSC_PPN               RISCV_IOMMU_ATP_PPN_FIELD
+#define RISCV_IOMMU_DC_FSC_MODE              RISCV_IOMMU_ATP_MODE_FIELD
+
+/**
+ * enum riscv_iommu_dc_fsc_atp_modes - First stage address translation/protection modes
+ * @RISCV_IOMMU_DC_FSC_MODE_BARE: No translation/protection
+ * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32: Sv32, when dc.tc.SXL == 1
+ * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39: Sv39, when dc.tc.SXL == 0
+ * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48: Sv48, when dc.tc.SXL == 0
+ * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57: Sv57, when dc.tc.SXL == 0
+ * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8: 1lvl PDT, 8bit process ids
+ * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17: 2lvl PDT, 17bit process ids
+ * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20: 3lvl PDT, 20bit process ids
+ *
+ * FSC holds IOSATP when RISCV_IOMMU_DC_TC_PDTV is 0 and PDTP otherwise.
+ * IOSATP controls the first stage address translation (same as the satp register on
+ * the RISC-V MMU), and PDTP holds the process directory table, used to select a
+ * first stage page table based on a process id (for devices that support multiple
+ * process ids).
+ */
+enum riscv_iommu_dc_fsc_atp_modes {
+     RISCV_IOMMU_DC_FSC_MODE_BARE = 0,
+     RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32 = 8,
+     RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39 = 8,
+     RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48 = 9,
+     RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57 = 10,
+     RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8 = 1,
+     RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17 = 2,
+     RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20 = 3
+};
+
+/* MSI page table pointer */
+#define RISCV_IOMMU_DC_MSIPTP_PPN    RISCV_IOMMU_ATP_PPN_FIELD
+#define RISCV_IOMMU_DC_MSIPTP_MODE   RISCV_IOMMU_ATP_MODE_FIELD
+#define RISCV_IOMMU_DC_MSIPTP_MODE_OFF       0
+#define RISCV_IOMMU_DC_MSIPTP_MODE_FLAT      1
+
+/* MSI address mask */
+#define RISCV_IOMMU_DC_MSI_ADDR_MASK GENMASK_ULL(51, 0)
+
+/* MSI address pattern */
+#define RISCV_IOMMU_DC_MSI_PATTERN   GENMASK_ULL(51, 0)
+
+/**
+ * struct riscv_iommu_pc - Process Context
+ * @ta: Translation Attributes
+ * @fsc: First stage context
+ *
+ * This structure is used for leaf nodes on the Process Directory Table
+ * See section 2.3 for more details
+ */
+struct riscv_iommu_pc {
+     u64 ta;
+     u64 fsc;
+};
+
+/* Translation attributes fields */
+#define RISCV_IOMMU_PC_TA_V  BIT_ULL(0)
+#define RISCV_IOMMU_PC_TA_ENS        BIT_ULL(1)
+#define RISCV_IOMMU_PC_TA_SUM        BIT_ULL(2)
+#define RISCV_IOMMU_PC_TA_PSCID      GENMASK_ULL(31, 12)
+
+/* First stage context fields */
+#define RISCV_IOMMU_PC_FSC_PPN       RISCV_IOMMU_ATP_PPN_FIELD
+#define RISCV_IOMMU_PC_FSC_MODE      RISCV_IOMMU_ATP_MODE_FIELD
+
+/*
+ * Chapter 3: In-memory queue interface
+ */
+
+/**
+ * struct riscv_iommu_command - Generic I/O MMU command structure
+ * @dword0: Includes the opcode and the function identifier
+ * @dword1: Opcode specific data
+ *
+ * The commands are interpreted as two 64bit fields, where the first
+ * 7bits of the first field are the opcode which also defines the
+ * command's format, followed by a 3bit field that specifies the
+ * function invoked by that command, and the rest is opcode-specific.
+ * This is a generic struct which will be populated differently
+ * according to each command. For more infos on the commands and
+ * the command queue check section 3.1.
+ */
+struct riscv_iommu_command {
+     u64 dword0;
+     u64 dword1;
+};
+
+/* Fields on dword0, common for all commands */
+#define RISCV_IOMMU_CMD_OPCODE       GENMASK_ULL(6, 0)
+#define      RISCV_IOMMU_CMD_FUNC    GENMASK_ULL(9, 7)
+
+/* 3.1.1 I/O MMU Page-table cache invalidation */
+/* Fields on dword0 */
+#define RISCV_IOMMU_CMD_IOTINVAL_OPCODE              1
+#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA    0
+#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA   1
+#define RISCV_IOMMU_CMD_IOTINVAL_AV          BIT_ULL(10)
+#define RISCV_IOMMU_CMD_IOTINVAL_PSCID               GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_CMD_IOTINVAL_PSCV                BIT_ULL(32)
+#define RISCV_IOMMU_CMD_IOTINVAL_GV          BIT_ULL(33)
+#define RISCV_IOMMU_CMD_IOTINVAL_GSCID               GENMASK_ULL(59, 44)
+/* dword1[61:10] is the 4K-aligned page address */
+#define RISCV_IOMMU_CMD_IOTINVAL_ADDR                GENMASK_ULL(61, 10)
+
+/* 3.1.2 I/O MMU Command Queue Fences */
+/* Fields on dword0 */
+#define RISCV_IOMMU_CMD_IOFENCE_OPCODE               2
+#define RISCV_IOMMU_CMD_IOFENCE_FUNC_C               0
+#define RISCV_IOMMU_CMD_IOFENCE_AV           BIT_ULL(10)
+#define RISCV_IOMMU_CMD_IOFENCE_WSI          BIT_ULL(11)
+#define RISCV_IOMMU_CMD_IOFENCE_PR           BIT_ULL(12)
+#define RISCV_IOMMU_CMD_IOFENCE_PW           BIT_ULL(13)
+#define RISCV_IOMMU_CMD_IOFENCE_DATA         GENMASK_ULL(63, 32)
+/* dword1 is the address, word-size aligned and shifted to the right by two bits. */
+
+/* 3.1.3 I/O MMU Directory cache invalidation */
+/* Fields on dword0 */
+#define RISCV_IOMMU_CMD_IODIR_OPCODE         3
+#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT 0
+#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT 1
+#define RISCV_IOMMU_CMD_IODIR_PID            GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_CMD_IODIR_DV             BIT_ULL(33)
+#define RISCV_IOMMU_CMD_IODIR_DID            GENMASK_ULL(63, 40)
+/* dword1 is reserved for standard use */
+
+/* 3.1.4 I/O MMU PCIe ATS */
+/* Fields on dword0 */
+#define RISCV_IOMMU_CMD_ATS_OPCODE           4
+#define RISCV_IOMMU_CMD_ATS_FUNC_INVAL               0
+#define RISCV_IOMMU_CMD_ATS_FUNC_PRGR                1
+#define RISCV_IOMMU_CMD_ATS_PID                      GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_CMD_ATS_PV                       BIT_ULL(32)
+#define RISCV_IOMMU_CMD_ATS_DSV                      BIT_ULL(33)
+#define RISCV_IOMMU_CMD_ATS_RID                      GENMASK_ULL(55, 40)
+#define RISCV_IOMMU_CMD_ATS_DSEG             GENMASK_ULL(63, 56)
+/* dword1 is the ATS payload, two different payload types for INVAL and PRGR */
+
+/* ATS.INVAL payload*/
+#define RISCV_IOMMU_CMD_ATS_INVAL_G          BIT_ULL(0)
+/* Bits 1 - 10 are zeroed */
+#define RISCV_IOMMU_CMD_ATS_INVAL_S          BIT_ULL(11)
+#define RISCV_IOMMU_CMD_ATS_INVAL_UADDR              GENMASK_ULL(63, 12)
+
+/* ATS.PRGR payload */
+/* Bits 0 - 31 are zeroed */
+#define RISCV_IOMMU_CMD_ATS_PRGR_PRG_INDEX   GENMASK_ULL(40, 32)
+/* Bits 41 - 43 are zeroed */
+#define RISCV_IOMMU_CMD_ATS_PRGR_RESP_CODE   GENMASK_ULL(47, 44)
+#define RISCV_IOMMU_CMD_ATS_PRGR_DST_ID              GENMASK_ULL(63, 48)
+
+/**
+ * struct riscv_iommu_fq_record - Fault/Event Queue Record
+ * @hdr: Header, includes fault/event cause, PID/DID, transaction type etc
+ * @_reserved: Low 32bits for custom use, high 32bits for standard use
+ * @iotval: Transaction-type/cause specific format
+ * @iotval2: Cause specific format
+ *
+ * The fault/event queue reports events and failures raised when
+ * processing transactions. Each record is a 32byte structure where
+ * the first dword has a fixed format for providing generic infos
+ * regarding the fault/event, and two more dwords are there for
+ * fault/event-specific information. For more details see section
+ * 3.2.
+ */
+struct riscv_iommu_fq_record {
+     u64 hdr;
+     u64 _reserved;
+     u64 iotval;
+     u64 iotval2;
+};
+
+/* Fields on header */
+#define RISCV_IOMMU_FQ_HDR_CAUSE     GENMASK_ULL(11, 0)
+#define RISCV_IOMMU_FQ_HDR_PID               GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_FQ_HDR_PV                BIT_ULL(32)
+#define RISCV_IOMMU_FQ_HDR_PRIV              BIT_ULL(33)
+#define RISCV_IOMMU_FQ_HDR_TTYPE     GENMASK_ULL(39, 34)
+#define RISCV_IOMMU_FQ_HDR_DID               GENMASK_ULL(63, 40)
+
+/**
+ * enum riscv_iommu_fq_causes - Fault/event cause values
+ * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT: Instruction access fault
+ * @RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED: Read address misaligned
+ * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT: Read load fault
+ * @RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED: Write/AMO address misaligned
+ * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT: Write/AMO access fault
+ * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S: Instruction page fault
+ * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S: Read page fault
+ * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S: Write/AMO page fault
+ * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS: Instruction guest page fault
+ * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS: Read guest page fault
+ * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS: Write/AMO guest page fault
+ * @RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED: All inbound transactions disallowed
+ * @RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT: DDT entry load access fault
+ * @RISCV_IOMMU_FQ_CAUSE_DDT_INVALID: DDT entry invalid
+ * @RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED: DDT entry misconfigured
+ * @RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED: Transaction type disallowed
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT: MSI PTE load access fault
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_INVALID: MSI PTE invalid
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED: MSI PTE misconfigured
+ * @RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT: MRIF access fault
+ * @RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT: PDT entry load access fault
+ * @RISCV_IOMMU_FQ_CAUSE_PDT_INVALID: PDT entry invalid
+ * @RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED: PDT entry misconfigured
+ * @RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED: DDT data corruption
+ * @RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED: PDT data corruption
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED: MSI page table data corruption
+ * @RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED: MRIF data corruption
+ * @RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR: Internal data path error
+ * @RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT: IOMMU MSI write access fault
+ * @RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED: First/second stage page table data corruption
+ *
+ * Values are on table 11 of the spec, encodings 275 - 2047 are reserved for standard
+ * use, and 2048 - 4095 for custom use.
+ */
+enum riscv_iommu_fq_causes {
+     RISCV_IOMMU_FQ_CAUSE_INST_FAULT = 1,
+     RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED = 4,
+     RISCV_IOMMU_FQ_CAUSE_RD_FAULT = 5,
+     RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED = 6,
+     RISCV_IOMMU_FQ_CAUSE_WR_FAULT = 7,
+     RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S = 12,
+     RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S = 13,
+     RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S = 15,
+     RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS = 20,
+     RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS = 21,
+     RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS = 23,
+     RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED = 256,
+     RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT = 257,
+     RISCV_IOMMU_FQ_CAUSE_DDT_INVALID = 258,
+     RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED = 259,
+     RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED = 260,
+     RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT = 261,
+     RISCV_IOMMU_FQ_CAUSE_MSI_INVALID = 262,
+     RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED = 263,
+     RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT = 264,
+     RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT = 265,
+     RISCV_IOMMU_FQ_CAUSE_PDT_INVALID = 266,
+     RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED = 267,
+     RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED = 268,
+     RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED = 269,
+     RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED = 270,
+     RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED = 271,
+     RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR = 272,
+     RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT = 273,
+     RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED = 274
+};
+
+/**
+ * enum riscv_iommu_fq_ttypes: Fault/event transaction types
+ * @RISCV_IOMMU_FQ_TTYPE_NONE: None. Fault not caused by an inbound transaction.
+ * @RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH: Instruction fetch from untranslated address
+ * @RISCV_IOMMU_FQ_TTYPE_UADDR_RD: Read from untranslated address
+ * @RISCV_IOMMU_FQ_TTYPE_UADDR_WR: Write/AMO to untranslated address
+ * @RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH: Instruction fetch from translated address
+ * @RISCV_IOMMU_FQ_TTYPE_TADDR_RD: Read from translated address
+ * @RISCV_IOMMU_FQ_TTYPE_TADDR_WR: Write/AMO to translated address
+ * @RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ: PCIe ATS translation request
+ * @RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ: PCIe message request
+ *
+ * Values are on table 12 of the spec, type 4 and 10 - 31 are reserved for standard use
+ * and 31 - 63 for custom use.
+ */
+enum riscv_iommu_fq_ttypes {
+     RISCV_IOMMU_FQ_TTYPE_NONE = 0,
+     RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH = 1,
+     RISCV_IOMMU_FQ_TTYPE_UADDR_RD = 2,
+     RISCV_IOMMU_FQ_TTYPE_UADDR_WR = 3,
+     RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH = 5,
+     RISCV_IOMMU_FQ_TTYPE_TADDR_RD = 6,
+     RISCV_IOMMU_FQ_TTYPE_TADDR_WR = 7,
+     RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ = 8,
+     RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ = 9,
+};
+
+/**
+ * struct riscv_iommu_pq_record - PCIe Page Request record
+ * @hdr: Header, includes PID, DID etc
+ * @payload: Holds the page address, request group and permission bits
+ *
+ * For more infos on the PCIe Page Request queue see chapter 3.3.
+ */
+struct riscv_iommu_pq_record {
+     u64 hdr;
+     u64 payload;
+};
+
+/* Header fields */
+#define RISCV_IOMMU_PREQ_HDR_PID     GENMASK_ULL(31, 12)
+#define RISCV_IOMMU_PREQ_HDR_PV              BIT_ULL(32)
+#define RISCV_IOMMU_PREQ_HDR_PRIV    BIT_ULL(33)
+#define RISCV_IOMMU_PREQ_HDR_EXEC    BIT_ULL(34)
+#define RISCV_IOMMU_PREQ_HDR_DID     GENMASK_ULL(63, 40)
+
+/* Payload fields */
+#define RISCV_IOMMU_PREQ_PAYLOAD_R   BIT_ULL(0)
+#define RISCV_IOMMU_PREQ_PAYLOAD_W   BIT_ULL(1)
+#define RISCV_IOMMU_PREQ_PAYLOAD_L   BIT_ULL(2)
+#define RISCV_IOMMU_PREQ_PAYLOAD_M   GENMASK_ULL(2, 0)       /* Mask of RWL for convenience */
+#define RISCV_IOMMU_PREQ_PRG_INDEX   GENMASK_ULL(11, 3)
+#define RISCV_IOMMU_PREQ_UADDR               GENMASK_ULL(63, 12)
+
+/**
+ * struct riscv_iommu_msi_pte - MSI Page Table Entry
+ * @pte: MSI PTE
+ * @mrif_info: Memory-resident interrupt file info
+ *
+ * The MSI Page Table is used for virtualizing MSIs, so that when
+ * a device sends an MSI to a guest, the IOMMU can reroute it
+ * by translating the MSI address, either to a guest interrupt file
+ * or a memory resident interrupt file (MRIF). Note that this page table
+ * is an array of MSI PTEs, not a multi-level pt, each entry
+ * is a leaf entry. For more infos check out the AIA spec, chapter 9.5.
+ *
+ * Also in basic mode the mrif_info field is ignored by the IOMMU and can
+ * be used by software, any other reserved fields on pte must be zeroed-out
+ * by software.
+ */
+struct riscv_iommu_msi_pte {
+     u64 pte;
+     u64 mrif_info;
+};
+
+/* Fields on pte */
+#define RISCV_IOMMU_MSI_PTE_V                BIT_ULL(0)
+#define RISCV_IOMMU_MSI_PTE_M                GENMASK_ULL(2, 1)
+#define RISCV_IOMMU_MSI_PTE_MRIF_ADDR        GENMASK_ULL(53, 7)      /* When M == 1 (MRIF mode) */
+#define RISCV_IOMMU_MSI_PTE_PPN              RISCV_IOMMU_PPN_FIELD   /* When M == 3 (basic mode) */
+#define RISCV_IOMMU_MSI_PTE_C                BIT_ULL(63)
+
+/* Fields on mrif_info */
+#define RISCV_IOMMU_MSI_MRIF_NID     GENMASK_ULL(9, 0)
+#define RISCV_IOMMU_MSI_MRIF_NPPN    RISCV_IOMMU_PPN_FIELD
+#define RISCV_IOMMU_MSI_MRIF_NID_MSB BIT_ULL(60)
+
+#endif /* _RISCV_IOMMU_BITS_H_ */

drivers/iommu/riscv/iommu-platform.c new file +50 -0

new file mode 100644
index 000000000000..770086ae2ab3
@@ -0,0 +1,94 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * RISC-V IOMMU as a platform device
+ *
+ * Copyright © 2023 FORTH-ICS/CARV
+ * Copyright © 2023-2024 Rivos Inc.
+ *
+ * Authors
+ *   Nick Kossifidis <[email protected]>
+ *   Tomasz Jeznach <[email protected]>
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+
+#include "iommu-bits.h"
+#include "iommu.h"
+
+static int riscv_iommu_platform_probe(struct platform_device *pdev)
+{
+     struct device *dev = &pdev->dev;
+     struct riscv_iommu_device *iommu = NULL;
+     struct resource *res = NULL;
+     int vec;
+
+     iommu = devm_kzalloc(dev, sizeof(*iommu), GFP_KERNEL);
+     if (!iommu)
+             return -ENOMEM;
+
+     iommu->dev = dev;
+     iommu->reg = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
+     if (IS_ERR(iommu->reg))
+             return dev_err_probe(dev, PTR_ERR(iommu->reg),
+                                  "could not map register region\n");
+
+     dev_set_drvdata(dev, iommu);
+
+     /* Check device reported capabilities / features. */
+     iommu->caps = riscv_iommu_readq(iommu, RISCV_IOMMU_REG_CAP);
+     iommu->fctl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL);
+
+     /* For now we only support WSI */
+     switch (FIELD_GET(RISCV_IOMMU_CAP_IGS, iommu->caps)) {
+     case RISCV_IOMMU_CAP_IGS_WSI:
+     case RISCV_IOMMU_CAP_IGS_BOTH:
+             break;
+     default:
+             return dev_err_probe(dev, -ENODEV,

drivers/iommu/riscv/iommu-platform.c line 60

Quoted 10 lines

drivers/iommu/riscv/iommu-platform.c new file +10 -0

+                                  "unable to use wire-signaled interrupts\n");
+     }
+
+     iommu->irqs_count = platform_irq_count(pdev);
+     if (iommu->irqs_count <= 0)
+             return dev_err_probe(dev, -ENODEV,
+                                  "no IRQ resources provided\n");
+
+     for (vec = 0; vec < iommu->irqs_count; vec++)
+             iommu->irqs[vec] = platform_get_irq(pdev, vec);

And if I've specified 97 interrupts in my DT because I won't let schema be the boss of me?

drivers/iommu/riscv/iommu-platform.c line 64

Quoted 3 lines

drivers/iommu/riscv/iommu-platform.c new file +3 -0

+     /* Enable wire-signaled interrupts, fctl.WSI */
+     if (!(iommu->fctl & RISCV_IOMMU_FCTL_WSI)) {
+             iommu->fctl ^= RISCV_IOMMU_FCTL_WSI;

Using XOR to only ever set a 0 bit to 1 seems a bit obtuse.

Quoted 7 lines

drivers/iommu/riscv/iommu-platform.c new file +7 -0

+             riscv_iommu_writel(iommu, RISCV_IOMMU_REG_FCTL, iommu->fctl);
+     }
+
+     return riscv_iommu_init(iommu);
+};
+
+static void riscv_iommu_platform_remove(struct platform_device *pdev)

drivers/iommu/riscv/iommu-platform.c line 81

Quoted 10 lines

drivers/iommu/riscv/iommu-platform.c new file +10 -0

+{
+     riscv_iommu_remove(dev_get_drvdata(&pdev->dev));
+};
+
+static const struct of_device_id riscv_iommu_of_match[] = {
+     {.compatible = "riscv,iommu",},
+     {},
+};
+
+MODULE_DEVICE_TABLE(of, riscv_iommu_of_match);

And yet it cannot be a module?

drivers/iommu/riscv/iommu-platform.c line 94

Quoted 12 lines

drivers/iommu/riscv/iommu-platform.c new file +12 -0

+static struct platform_driver riscv_iommu_platform_driver = {
+     .probe = riscv_iommu_platform_probe,
+     .remove_new = riscv_iommu_platform_remove,
+     .driver = {
+             .name = "riscv,iommu",
+             .of_match_table = riscv_iommu_of_match,
+             .suppress_bind_attrs = true,
+     },
+};
+
+module_driver(riscv_iommu_platform_driver, platform_driver_register,
+           platform_driver_unregister);

module_platform_driver() is a thing. Or builtin_platform_driver(), as things currently stand.

Quoted 9 lines

drivers/iommu/riscv/iommu.c new file +3 -0

new file mode 100644
index 000000000000..af68c89200a9
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * IOMMU API for RISC-V IOMMU implementations.

drivers/iommu/riscv/iommu.c line 13

Quoted 10 lines

drivers/iommu/riscv/iommu.c new file +10 -0

+ *
+ * Copyright © 2022-2024 Rivos Inc.
+ * Copyright © 2023 FORTH-ICS/CARV
+ *
+ * Authors
+ *   Tomasz Jeznach <[email protected]>
+ *   Nick Kossifidis <[email protected]>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

I guess that ends up as "iommu:"? Given that that prefix already belongs to the core code, please pick something more specific.

Quoted 24 lines

drivers/iommu/riscv/iommu.c new file +24 -0

+#include <linux/compiler.h>
+#include <linux/dma-mapping.h>
+#include <linux/init.h>
+#include <linux/iommu.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "iommu-bits.h"
+#include "iommu.h"
+
+MODULE_DESCRIPTION("Driver for RISC-V IOMMU");
+MODULE_AUTHOR("Tomasz Jeznach <[email protected]>");
+MODULE_AUTHOR("Nick Kossifidis <[email protected]>");
+MODULE_LICENSE("GPL");
+
+/* Timeouts in [us] */
+#define RISCV_IOMMU_DDTP_TIMEOUT     50000
+
+static int riscv_iommu_init_check(struct riscv_iommu_device *iommu)
+{
+     u64 ddtp;
+
+     /* Hardware must be configured in OFF | BARE mode at system initialization. */
+     riscv_iommu_readq_timeout(iommu, RISCV_IOMMU_REG_DDTP,

Quoted 9 lines

drivers/iommu/riscv/iommu.c line 42
Quoted 4 lines
drivers/iommu/riscv/iommu.c new file +4 -0
+                               ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+                               10, RISCV_IOMMU_DDTP_TIMEOUT);
+     if (FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp) > RISCV_IOMMU_DDTP_MODE_BARE)
+             return -EBUSY;
It looks like RISC-V already supports kdump, so you probably want to be prepared to find the IOMMU with its pants down and deal with it from day one.

This is the simplest check/fail for the kexec and/or boot loaders leaving IOMMU translations active.
I've been already looking into kexec path to quiesce all devices and IOMMU in shutdown path.
I'm not convinced it's ready for the prime time on RISC-V, will address this in follow up patches.

Quoted 19 lines

drivers/iommu/riscv/iommu.c line 53

Quoted 11 lines

drivers/iommu/riscv/iommu.c new file +11 -0

+
+     /* Configure accesses to in-memory data structures for CPU-native byte order. */
+     if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN) != !!(iommu->fctl & RISCV_IOMMU_FCTL_BE)) {
+             if (!(iommu->caps & RISCV_IOMMU_CAP_END))
+                     return -EINVAL;
+             riscv_iommu_writel(iommu, RISCV_IOMMU_REG_FCTL,
+                                iommu->fctl ^ RISCV_IOMMU_FCTL_BE);
+             iommu->fctl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL);
+             if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN) != !!(iommu->fctl & RISCV_IOMMU_FCTL_BE))
+                     return -EINVAL;
+     }

That's fun. It could be reorganised to avoid the duplicate check, but it is rather majestic as-is :)

drivers/iommu/riscv/iommu.c line 56

Quoted 3 lines

drivers/iommu/riscv/iommu.c new file +3 -0

+
+     dma_set_mask_and_coherent(iommu->dev,
+                               DMA_BIT_MASK(FIELD_GET(RISCV_IOMMU_CAP_PAS, iommu->caps)));

Quoted 6 lines

This isn't a check, so I would think it belongs to "_init" rather than "_init_check".
Thanks,
Robin.

ACK to all other comments.

Thanks
- Tomasz

Quoted 101 lines

drivers/iommu/riscv/iommu.c new file +33 -0

+
+     return 0;
+}
+
+void riscv_iommu_remove(struct riscv_iommu_device *iommu)
+{
+     iommu_device_sysfs_remove(&iommu->iommu);
+}
+
+int riscv_iommu_init(struct riscv_iommu_device *iommu)
+{
+     int rc;
+
+     rc = riscv_iommu_init_check(iommu);
+     if (rc)
+             return dev_err_probe(iommu->dev, rc, "unexpected device state\n");
+     /*
+      * Placeholder for a complete IOMMU device initialization.
+      * For now, only bare minimum: enable global identity mapping mode and register sysfs.
+      */
+     riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP,
+                        FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE));
+
+     rc = iommu_device_sysfs_add(&iommu->iommu, NULL, NULL, "riscv-iommu@%s",
+                                 dev_name(iommu->dev));
+     if (WARN(rc, "cannot register sysfs interface\n"))
+             goto err_sysfs;
+
+     return 0;
+
+err_sysfs:
+     return rc;
+}

drivers/iommu/riscv/iommu.h new file +62 -0

new file mode 100644
index 000000000000..700e33dc2446
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright © 2022-2024 Rivos Inc.
+ * Copyright © 2023 FORTH-ICS/CARV
+ *
+ * Authors
+ *   Tomasz Jeznach <[email protected]>
+ *   Nick Kossifidis <[email protected]>
+ */
+
+#ifndef _RISCV_IOMMU_H_
+#define _RISCV_IOMMU_H_
+
+#include <linux/iommu.h>
+#include <linux/types.h>
+#include <linux/iopoll.h>
+
+#include "iommu-bits.h"
+
+struct riscv_iommu_device {
+     /* iommu core interface */
+     struct iommu_device iommu;
+
+     /* iommu hardware */
+     struct device *dev;
+
+     /* hardware control register space */
+     void __iomem *reg;
+
+     /* supported and enabled hardware capabilities */
+     u64 caps;
+     u32 fctl;
+
+     /* available interrupt numbers, MSI or WSI */
+     unsigned int irqs[RISCV_IOMMU_INTR_COUNT];
+     unsigned int irqs_count;
+};
+
+int riscv_iommu_init(struct riscv_iommu_device *iommu);
+void riscv_iommu_remove(struct riscv_iommu_device *iommu);
+
+#define riscv_iommu_readl(iommu, addr) \
+     readl_relaxed((iommu)->reg + (addr))
+
+#define riscv_iommu_readq(iommu, addr) \
+     readq_relaxed((iommu)->reg + (addr))
+
+#define riscv_iommu_writel(iommu, addr, val) \
+     writel_relaxed((val), (iommu)->reg + (addr))
+
+#define riscv_iommu_writeq(iommu, addr, val) \
+     writeq_relaxed((val), (iommu)->reg + (addr))
+
+#define riscv_iommu_readq_timeout(iommu, addr, val, cond, delay_us, timeout_us) \
+     readx_poll_timeout(readq_relaxed, (iommu)->reg + (addr), val, cond, \
+                        delay_us, timeout_us)
+
+#define riscv_iommu_readl_timeout(iommu, addr, val, cond, delay_us, timeout_us) \
+     readx_poll_timeout(readl_relaxed, (iommu)->reg + (addr), val, cond, \
+                        delay_us, timeout_us)
+
+#endif

Robin Murphy

reviewer

2024-04-25 11:23 UTC #9 lore raw

Re: [PATCH v2 2/7] iommu/riscv: Add RISC-V IOMMU platform device driver

On 24/04/2024 10:59 pm, Tomasz Jeznach wrote:
[...]

Quoted 18 lines

Quoted 8 lines

MAINTAINERS +0 -0

index 2657f9eae84c..051599c76585 100644
@@ -18972,6 +18972,12 @@ L:   [email protected]
   L:  [email protected]
   S:  Maintained
   F:  Documentation/devicetree/bindings/iommu/riscv,iommu.yaml

Quoted 9 lines

MAINTAINERS L: [email protected] · line 18980
Quoted 6 lines
MAINTAINERS +6 -0
+F:   drivers/iommu/riscv/Kconfig
+F:   drivers/iommu/riscv/Makefile
+F:   drivers/iommu/riscv/iommu-bits.h
+F:   drivers/iommu/riscv/iommu-platform.c
+F:   drivers/iommu/riscv/iommu.c
+F:   drivers/iommu/riscv/iommu.h
I'm pretty sure a single "F: drivers/iommu/riscv/" pattern will suffice.

Quoted 2 lines

Correct. But will required a workaround for pretty naive MAINTAINERS update check in scripts/checkpatch.pl:3014 in next patch.

As long as what you're doing is clearly reasonable to humans, the correct workaround for any checkpatch complaint is to ignore checkpatch.

[...]

Quoted 16 lines

Quoted 6 lines
Quoted 6 lines
MAINTAINERS +6 -0
+static int riscv_iommu_init_check(struct riscv_iommu_device *iommu)
+{
+     u64 ddtp;
+
+     /* Hardware must be configured in OFF | BARE mode at system initialization. */
+     riscv_iommu_readq_timeout(iommu, RISCV_IOMMU_REG_DDTP,
Quoted 9 lines
MAINTAINERS
Quoted 4 lines
MAINTAINERS +4 -0
+                               ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+                               10, RISCV_IOMMU_DDTP_TIMEOUT);
+     if (FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp) > RISCV_IOMMU_DDTP_MODE_BARE)
+             return -EBUSY;
It looks like RISC-V already supports kdump, so you probably want to be prepared to find the IOMMU with its pants down and deal with it from day one.

Quoted 6 lines

This is the simplest check/fail for the kexec and/or boot loaders leaving IOMMU translations active.
I've been already looking into kexec path to quiesce all devices and IOMMU in shutdown path.
I'm not convinced it's ready for the prime time on RISC-V, will address this in follow up patches.

Yeah, for regular kexec you definitely want an orderly shutdown of the IOMMU, although there's still a bit of an open question about whether it's better to actively block any remaining traffic from devices whose drivers haven't cleanly stopped them. It's in the kdump crash kernel case that you can't have any expectations and need to be able to recover the IOMMU into a usable state, since it's likely to be in the way of devices which the crash kernel wants to take over and use.

Thanks,
Robin.

Tomasz Jeznach

author

2024-04-18 16:32 UTC #10 lore raw

PATCH v2 3/7 iommu/riscv: Add RISC-V IOMMU PCIe device driver

In reply to Tomasz Jeznach

Introduce device driver for PCIe implementation
of RISC-V IOMMU architected hardware.

IOMMU hardware and system support for MSI or MSI-X is required by this implementation.

Vendor and device identifiers used in this patch matches QEMU implementation of the RISC-V IOMMU PCIe device, from Rivos VID (0x1efd) range allocated by the PCI-SIG.

Link https://lore.kernel.org/qemu-devel/[email protected]/ open here

Co-developed-by Nick Kossifidis <[email protected]>

Signed-off-by Nick Kossifidis <[email protected]>

Signed-off-by Tomasz Jeznach <[email protected]>

---
 MAINTAINERS                     |   1 +
 drivers/iommu/riscv/Kconfig     |   6 ++
 drivers/iommu/riscv/Makefile    |   1 +
 drivers/iommu/riscv/iommu-pci.c | 154 ++++++++++++++++++++++++++++++++
 4 files changed, 162 insertions(+)
 create mode 100644 drivers/iommu/riscv/iommu-pci.c

MAINTAINERS +1 -0

index 051599c76585..4da290d5e9db 100644
@@ -18975,6 +18975,7 @@ F:	Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
 F:	drivers/iommu/riscv/Kconfig
 F:	drivers/iommu/riscv/Makefile
 F:	drivers/iommu/riscv/iommu-bits.h
+F:	drivers/iommu/riscv/iommu-pci.c
 F:	drivers/iommu/riscv/iommu-platform.c
 F:	drivers/iommu/riscv/iommu.c
 F:	drivers/iommu/riscv/iommu.h

drivers/iommu/riscv/Kconfig +6 -0

index d02326bddb4c..711326992585 100644
@@ -14,3 +14,9 @@ config RISCV_IOMMU
 
 	  Say Y here if your SoC includes an IOMMU device implementing
 	  the RISC-V IOMMU architecture.
+
+config RISCV_IOMMU_PCI
+	def_bool y if RISCV_IOMMU && PCI_MSI
+	depends on RISCV_IOMMU && PCI_MSI
+	help
+	  Support for the PCI implementation of RISC-V IOMMU architecture.

drivers/iommu/riscv/Makefile +1 -0

index e4c189de58d3..f54c9ed17d41 100644
@@ -1,2 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-platform.o
+obj-$(CONFIG_RISCV_IOMMU_PCI) += iommu-pci.o

drivers/iommu/riscv/iommu-pci.c new file +154 -0

154 lines not on this page · raw message

Signature · 1 line

2.34.1

Robin Murphy

reviewer

2024-04-18 22:07 UTC #11 lore raw

Re: [PATCH v2 3/7] iommu/riscv: Add RISC-V IOMMU PCIe device driver

Quoted 39 lines · Tomasz Jeznach

Introduce device driver for PCIe implementation
of RISC-V IOMMU architected hardware.
IOMMU hardware and system support for MSI or MSI-X is required by this implementation.
Vendor and device identifiers used in this patch matches QEMU implementation of the RISC-V IOMMU PCIe device, from Rivos VID (0x1efd) range allocated by the PCI-SIG.
Link https://lore.kernel.org/qemu-devel/[email protected]/ open here
Co-developed-by Nick Kossifidis <[email protected]>
Signed-off-by Nick Kossifidis <[email protected]>
Signed-off-by Tomasz Jeznach <[email protected]>
---
  MAINTAINERS                     |   1 +
  drivers/iommu/riscv/Kconfig     |   6 ++
  drivers/iommu/riscv/Makefile    |   1 +
  drivers/iommu/riscv/iommu-pci.c | 154 ++++++++++++++++++++++++++++++++
  4 files changed, 162 insertions(+)
create mode 100644 drivers/iommu/riscv/iommu-pci.c
MAINTAINERS +1 -0
index 051599c76585..4da290d5e9db 100644
@@ -18975,6 +18975,7 @@ F:	Documentation/devicetree/bindings/iommu/riscv,iommu.yaml
  F:	drivers/iommu/riscv/Kconfig
  F:	drivers/iommu/riscv/Makefile
  F:	drivers/iommu/riscv/iommu-bits.h
+F:	drivers/iommu/riscv/iommu-pci.c
  F:	drivers/iommu/riscv/iommu-platform.c
  F:	drivers/iommu/riscv/iommu.c
  F:	drivers/iommu/riscv/iommu.h
drivers/iommu/riscv/Kconfig +0 -0
index d02326bddb4c..711326992585 100644

drivers/iommu/riscv/Kconfig config RISCV_IOMMU · line 22

Quoted 9 lines

drivers/iommu/riscv/Kconfig +6 -0

@@ -14,3 +14,9 @@ config RISCV_IOMMU
 
  	  Say Y here if your SoC includes an IOMMU device implementing
  	  the RISC-V IOMMU architecture.
+
+config RISCV_IOMMU_PCI
+	def_bool y if RISCV_IOMMU && PCI_MSI
+	depends on RISCV_IOMMU && PCI_MSI
+	help
+	  Support for the PCI implementation of RISC-V IOMMU architecture.

Similar comments as before.

Quoted 64 lines

drivers/iommu/riscv/Makefile +1 -0

index e4c189de58d3..f54c9ed17d41 100644
@@ -1,2 +1,3 @@
  # SPDX-License-Identifier: GPL-2.0-only
  obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-platform.o
+obj-$(CONFIG_RISCV_IOMMU_PCI) += iommu-pci.o

drivers/iommu/riscv/iommu-pci.c new file +50 -0

new file mode 100644
index 000000000000..9263c6e475be
@@ -0,0 +1,154 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+/*
+ * Copyright © 2022-2024 Rivos Inc.
+ * Copyright © 2023 FORTH-ICS/CARV
+ *
+ * RISCV IOMMU as a PCIe device
+ *
+ * Authors
+ *	Tomasz Jeznach <[email protected]>
+ *	Nick Kossifidis <[email protected]>
+ */
+
+#include <linux/compiler.h>
+#include <linux/init.h>
+#include <linux/iommu.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+
+#include "iommu-bits.h"
+#include "iommu.h"
+
+/* Rivos Inc. assigned PCI Vendor and Device IDs */
+#ifndef PCI_VENDOR_ID_RIVOS
+#define PCI_VENDOR_ID_RIVOS             0x1efd
+#endif
+
+#ifndef PCI_DEVICE_ID_RIVOS_IOMMU
+#define PCI_DEVICE_ID_RIVOS_IOMMU       0xedf1
+#endif
+
+static int riscv_iommu_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+	struct device *dev = &pdev->dev;
+	struct riscv_iommu_device *iommu;
+	int rc, vec;
+
+	rc = pci_enable_device_mem(pdev);
+	if (rc)
+		return rc;
+
+	rc = pci_request_mem_regions(pdev, KBUILD_MODNAME);
+	if (rc)
+		goto fail;
+
+	pci_set_master(pdev);
+
+	if (!(pci_resource_flags(pdev, 0) & IORESOURCE_MEM))
+		goto fail;

drivers/iommu/riscv/iommu-pci.c line 60

Quoted 10 lines

drivers/iommu/riscv/iommu-pci.c new file +10 -0

+
+	if (pci_resource_len(pdev, 0) < RISCV_IOMMU_REG_SIZE)
+		goto fail;
+
+	iommu = devm_kzalloc(dev, sizeof(*iommu), GFP_KERNEL);
+	if (!iommu)
+		goto fail;
+
+	iommu->dev = dev;
+	iommu->reg = pci_iomap(pdev, 0, RISCV_IOMMU_REG_SIZE);

Maybe consider some of the pcim_* devres helpers, to simplify cleanup/remove?

Quoted 21 lines

drivers/iommu/riscv/iommu-pci.c new file +21 -0

+
+	if (!iommu->reg)
+		goto fail;
+
+	dev_set_drvdata(dev, iommu);
+
+	/* Check device reported capabilities / features. */
+	iommu->caps = riscv_iommu_readq(iommu, RISCV_IOMMU_REG_CAP);
+	iommu->fctl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL);
+
+	/* The PCI driver only uses MSIs, make sure the IOMMU supports this */
+	switch (FIELD_GET(RISCV_IOMMU_CAP_IGS, iommu->caps)) {
+	case RISCV_IOMMU_CAP_IGS_MSI:
+	case RISCV_IOMMU_CAP_IGS_BOTH:
+		break;
+	default:
+		dev_err(dev, "unable to use message-signaled interrupts\n");
+		rc = -ENODEV;
+		goto fail_unmap;
+	}
+

drivers/iommu/riscv/iommu-pci.c line 91

Quoted 10 lines

drivers/iommu/riscv/iommu-pci.c new file +10 -0

+	/* Allocate and assign IRQ vectors for the various events */
+	rc = pci_alloc_irq_vectors(pdev, 1, RISCV_IOMMU_INTR_COUNT,
+				   PCI_IRQ_MSIX | PCI_IRQ_MSI);
+	if (rc <= 0) {
+		dev_err(dev, "unable to allocate irq vectors\n");
+		goto fail_unmap;
+	}
+	for (vec = 0; vec < rc; vec++) {
+		iommu->irqs[vec] = msi_get_virq(dev, vec);
+		if (!iommu->irqs[vec])

Can that ever fail if the loop is already bounded to the number of vectors successfully allocated?

Quoted 48 lines

drivers/iommu/riscv/iommu-pci.c new file +48 -0

+			break;
+	}
+	iommu->irqs_count = vec;
+
+	/* Enable message-signaled interrupts, fctl.WSI */
+	if (iommu->fctl & RISCV_IOMMU_FCTL_WSI) {
+		iommu->fctl ^= RISCV_IOMMU_FCTL_WSI;
+		riscv_iommu_writel(iommu, RISCV_IOMMU_REG_FCTL, iommu->fctl);
+	}
+
+	rc = riscv_iommu_init(iommu);
+	if (!rc)
+		return 0;
+
+fail_unmap:
+	iounmap(iommu->reg);
+	pci_free_irq_vectors(pdev);
+fail:
+	pci_release_regions(pdev);
+	pci_clear_master(pdev);
+	pci_disable_device(pdev);
+	return rc;
+}
+
+static void riscv_iommu_pci_remove(struct pci_dev *pdev)
+{
+	struct riscv_iommu_device *iommu = dev_get_drvdata(&pdev->dev);
+
+	riscv_iommu_remove(iommu);
+	iounmap(iommu->reg);
+	pci_free_irq_vectors(pdev);
+	pci_release_regions(pdev);
+	pci_clear_master(pdev);
+	pci_disable_device(pdev);
+}
+
+static const struct pci_device_id riscv_iommu_pci_tbl[] = {
+	{PCI_VENDOR_ID_RIVOS, PCI_DEVICE_ID_RIVOS_IOMMU,
+	 PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
+	{0,}
+};
+
+MODULE_DEVICE_TABLE(pci, riscv_iommu_pci_tbl);
+
+static const struct of_device_id riscv_iommu_of_match[] = {
+	{.compatible = "riscv,pci-iommu",},
+	{},
+};

drivers/iommu/riscv/iommu-pci.c line 149

Quoted 10 lines

drivers/iommu/riscv/iommu-pci.c new file +10 -0

+
+MODULE_DEVICE_TABLE(of, riscv_iommu_of_match);
+
+static struct pci_driver riscv_iommu_pci_driver = {
+	.name = KBUILD_MODNAME,
+	.id_table = riscv_iommu_pci_tbl,
+	.probe = riscv_iommu_pci_probe,
+	.remove = riscv_iommu_pci_remove,
+	.driver = {
+		.of_match_table = riscv_iommu_of_match,

Does an of_match_table serve any functional purpose for a PCI driver? I can't find any other examples of this being done.

Thanks,
Robin.

Quoted 5 lines

drivers/iommu/riscv/iommu-pci.c new file +5 -0
+		.suppress_bind_attrs = true,
+	},
+};
+
+module_pci_driver(riscv_iommu_pci_driver);

Tomasz Jeznach

author

2024-04-18 16:32 UTC #12 lore raw

PATCH v2 4/7 iommu/riscv: Enable IOMMU registration and device probe.

In reply to Tomasz Jeznach

Advertise IOMMU device and its core API.
Only minimal implementation for single identity domain type, without per-group domain protection.

Signed-off-by Tomasz Jeznach <[email protected]>

---
 drivers/iommu/riscv/iommu.c | 69 +++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

drivers/iommu/riscv/iommu.c +69 -0

index af68c89200a9..d38317cb2493 100644
@@ -18,6 +18,7 @@
 #include <linux/iommu.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/pci.h>
 
 #include "iommu-bits.h"
 #include "iommu.h"

@@ -30,6 +31,67 @@ MODULE_LICENSE("GPL");
 /* Timeouts in [us] */
 #define RISCV_IOMMU_DDTP_TIMEOUT	50000
 
+static int riscv_iommu_attach_identity_domain(struct iommu_domain *domain,
+					      struct device *dev)
+{
+	/* Global pass-through already enabled, do nothing for now. */
+	return 0;
+}
+
+static struct iommu_domain riscv_iommu_identity_domain = {
+	.type = IOMMU_DOMAIN_IDENTITY,
+	.ops = &(const struct iommu_domain_ops) {
+		.attach_dev = riscv_iommu_attach_identity_domain,
+	}
+};
+
+static int riscv_iommu_device_domain_type(struct device *dev)
+{
+	return IOMMU_DOMAIN_IDENTITY;
+}
+
+static struct iommu_group *riscv_iommu_device_group(struct device *dev)
+{
+	if (dev_is_pci(dev))
+		return pci_device_group(dev);
+	return generic_device_group(dev);
+}
+
+static int riscv_iommu_of_xlate(struct device *dev, const struct of_phandle_args *args)
+{
+	return iommu_fwspec_add_ids(dev, args->args, 1);
+}
+
+static struct iommu_device *riscv_iommu_probe_device(struct device *dev)
+{
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct riscv_iommu_device *iommu;
+
+	if (!fwspec->iommu_fwnode->dev || !fwspec->num_ids)
+		return ERR_PTR(-ENODEV);
+
+	iommu = dev_get_drvdata(fwspec->iommu_fwnode->dev);
+	if (!iommu)
+		return ERR_PTR(-ENODEV);
+
+	return &iommu->iommu;
+}
+
+static void riscv_iommu_probe_finalize(struct device *dev)
+{
+	iommu_setup_dma_ops(dev, 0, U64_MAX);
+}
+
+static const struct iommu_ops riscv_iommu_ops = {
+	.owner = THIS_MODULE,
+	.of_xlate = riscv_iommu_of_xlate,
+	.identity_domain = &riscv_iommu_identity_domain,
+	.def_domain_type = riscv_iommu_device_domain_type,
+	.device_group = riscv_iommu_device_group,
+	.probe_device = riscv_iommu_probe_device,
+	.probe_finalize = riscv_iommu_probe_finalize,
+};
+
 static int riscv_iommu_init_check(struct riscv_iommu_device *iommu)
 {
 	u64 ddtp;

@@ -60,6 +122,7 @@ static int riscv_iommu_init_check(struct riscv_iommu_device *iommu)
 
 void riscv_iommu_remove(struct riscv_iommu_device *iommu)
 {
+	iommu_device_unregister(&iommu->iommu);
 	iommu_device_sysfs_remove(&iommu->iommu);
 }

@@ -82,8 +145,14 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
 	if (WARN(rc, "cannot register sysfs interface\n"))
 		goto err_sysfs;
 
+	rc = iommu_device_register(&iommu->iommu, &riscv_iommu_ops, iommu->dev);
+	if (WARN(rc, "cannot register iommu interface\n"))
+		goto err_iommu;
+
 	return 0;
 
+err_iommu:
+	iommu_device_sysfs_remove(&iommu->iommu);
 err_sysfs:
 	return rc;
 }

Signature · 1 line

2.34.1

Tomasz Jeznach

author

2024-04-18 16:32 UTC #13 lore raw

PATCH v2 5/7 iommu/riscv: Device directory management.

In reply to Tomasz Jeznach

Introduce device context allocation and device directory tree management including capabilities discovery sequence, as described in Chapter 2.1 of the RISC-V IOMMU Architecture Specification.

Device directory mode will be auto detected using DDTP WARL property, using highest mode supported by the driver and hardware. If none supported can be configured, driver will fall back to global pass-through.

First level DDTP page can be located in I/O (detected using DDTP WARL) and system memory.

Only identity protection domain is supported by this implementation.

Co-developed-by Nick Kossifidis <[email protected]>

Signed-off-by Nick Kossifidis <[email protected]>

Signed-off-by Tomasz Jeznach <[email protected]>

---
 drivers/iommu/riscv/iommu.c | 369 +++++++++++++++++++++++++++++++++++-
 drivers/iommu/riscv/iommu.h |   5 +
 2 files changed, 365 insertions(+), 9 deletions(-)

drivers/iommu/riscv/iommu.c +360 -9

index d38317cb2493..721cc71cb959 100644
@@ -16,6 +16,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/init.h>
 #include <linux/iommu.h>
+#include <linux/iopoll.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/pci.h>

@@ -31,13 +32,350 @@ MODULE_LICENSE("GPL");
 /* Timeouts in [us] */
 #define RISCV_IOMMU_DDTP_TIMEOUT	50000
 
-static int riscv_iommu_attach_identity_domain(struct iommu_domain *domain,
-					      struct device *dev)
+/* RISC-V IOMMU PPN <> PHYS address conversions, PHYS <=> PPN[53:10] */
+#define phys_to_ppn(va)  (((va) >> 2) & (((1ULL << 44) - 1) << 10))
+#define ppn_to_phys(pn)	 (((pn) << 2) & (((1ULL << 44) - 1) << 12))
+
+#define dev_to_iommu(dev) \
+	container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu)
+
+/* Device resource-managed allocations */
+struct riscv_iommu_devres {
+	unsigned long addr;
+	unsigned int order;
+};
+
+static void riscv_iommu_devres_pages_release(struct device *dev, void *res)
+{
+	struct riscv_iommu_devres *devres = res;
+
+	free_pages(devres->addr, devres->order);
+}
+
+static int riscv_iommu_devres_pages_match(struct device *dev, void *res, void *p)
+{
+	struct riscv_iommu_devres *devres = res;
+	struct riscv_iommu_devres *target = p;
+
+	return devres->addr == target->addr;
+}
+
+static unsigned long riscv_iommu_get_pages(struct riscv_iommu_device *iommu, unsigned int order)
+{
+	struct riscv_iommu_devres *devres;
+	struct page *pages;
+
+	pages = alloc_pages_node(dev_to_node(iommu->dev),
+				 GFP_KERNEL_ACCOUNT | __GFP_ZERO, order);
+	if (unlikely(!pages)) {
+		dev_err(iommu->dev, "Page allocation failed, order %u\n", order);
+		return 0;
+	}
+
+	devres = devres_alloc(riscv_iommu_devres_pages_release,
+			      sizeof(struct riscv_iommu_devres), GFP_KERNEL);
+
+	if (unlikely(!devres)) {
+		__free_pages(pages, order);
+		return 0;
+	}
+
+	devres->addr = (unsigned long)page_address(pages);
+	devres->order = order;
+
+	devres_add(iommu->dev, devres);
+
+	return devres->addr;
+}
+
+static void riscv_iommu_free_pages(struct riscv_iommu_device *iommu, unsigned long addr)
+{
+	struct riscv_iommu_devres devres = { .addr = addr };
+
+	devres_release(iommu->dev, riscv_iommu_devres_pages_release,
+		       riscv_iommu_devres_pages_match, &devres);
+}
+
+/* Lookup and initialize device context info structure. */
+static struct riscv_iommu_dc *riscv_iommu_get_dc(struct riscv_iommu_device *iommu,
+						 unsigned int devid, bool fetch)
+{
+	const bool base_format = !(iommu->caps & RISCV_IOMMU_CAP_MSI_FLAT);
+	unsigned int depth;
+	unsigned long ddt, ptr, old, new;
+	u8 ddi_bits[3] = { 0 };
+	u64 *ddtp = NULL;
+
+	/* Make sure the mode is valid */
+	if (iommu->ddt_mode < RISCV_IOMMU_DDTP_MODE_1LVL ||
+	    iommu->ddt_mode > RISCV_IOMMU_DDTP_MODE_3LVL)
+		return NULL;
+
+	/*
+	 * Device id partitioning for base format:
+	 * DDI[0]: bits 0 - 6   (1st level) (7 bits)
+	 * DDI[1]: bits 7 - 15  (2nd level) (9 bits)
+	 * DDI[2]: bits 16 - 23 (3rd level) (8 bits)
+	 *
+	 * For extended format:
+	 * DDI[0]: bits 0 - 5   (1st level) (6 bits)
+	 * DDI[1]: bits 6 - 14  (2nd level) (9 bits)
+	 * DDI[2]: bits 15 - 23 (3rd level) (9 bits)
+	 */
+	if (base_format) {
+		ddi_bits[0] = 7;
+		ddi_bits[1] = 7 + 9;
+		ddi_bits[2] = 7 + 9 + 8;
+	} else {
+		ddi_bits[0] = 6;
+		ddi_bits[1] = 6 + 9;
+		ddi_bits[2] = 6 + 9 + 9;
+	}
+
+	/* Make sure device id is within range */
+	depth = iommu->ddt_mode - RISCV_IOMMU_DDTP_MODE_1LVL;
+	if (devid >= (1 << ddi_bits[depth]))
+		return NULL;
+
+	/* Get to the level of the non-leaf node that holds the device context */
+	for (ddtp = iommu->ddt_root; depth-- > 0;) {
+		const int split = ddi_bits[depth];
+		/*
+		 * Each non-leaf node is 64bits wide and on each level
+		 * nodes are indexed by DDI[depth].
+		 */
+		ddtp += (devid >> split) & 0x1FF;
+
+		/*
+		 * Check if this node has been populated and if not
+		 * allocate a new level and populate it.
+		 */
+		do {
+			ddt = READ_ONCE(*(unsigned long *)ddtp);
+			if (ddt & RISCV_IOMMU_DDTE_VALID) {
+				ddtp = __va(ppn_to_phys(ddt));
+				break;
+			}
+
+			/* Fetch only, do not allocate new device context. */
+			if (fetch)
+				return NULL;
+
+			ptr = riscv_iommu_get_pages(iommu, 0);
+			if (!ptr)
+				return NULL;
+
+			new = phys_to_ppn(__pa(ptr)) | RISCV_IOMMU_DDTE_VALID;
+			old = cmpxchg_relaxed((unsigned long *)ddtp, ddt, new);
+
+			if (old == ddt) {
+				ddtp = (u64 *)ptr;
+				break;
+			}
+
+			/* Race setting DDT detected, re-read and retry. */
+			riscv_iommu_free_pages(iommu, ptr);
+		} while (1);
+	}
+
+	/*
+	 * Grab the node that matches DDI[depth], note that when using base
+	 * format the device context is 4 * 64bits, and the extended format
+	 * is 8 * 64bits, hence the (3 - base_format) below.
+	 */
+	ddtp += (devid & ((64 << base_format) - 1)) << (3 - base_format);
+
+	return (struct riscv_iommu_dc *)ddtp;
+}
+
+/*
+ * Discover supported DDT modes starting from requested value,
+ * configure DDTP register with accepted mode and root DDT address.
+ * Accepted iommu->ddt_mode is updated on success.
+ */
+static int riscv_iommu_set_ddtp_mode(struct riscv_iommu_device *iommu,
+				     unsigned int ddtp_mode)
+{
+	struct device *dev = iommu->dev;
+	u64 ddtp, rq_ddtp;
+	unsigned int mode, rq_mode = ddtp_mode;
+	int rc;
+
+	rc = readq_relaxed_poll_timeout(iommu->reg + RISCV_IOMMU_REG_DDTP,
+					ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+					10, RISCV_IOMMU_DDTP_TIMEOUT);
+	if (rc < 0)
+		return -EBUSY;
+
+	/* Disallow state transition from xLVL to xLVL. */
+	switch (FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp)) {
+	case RISCV_IOMMU_DDTP_MODE_BARE:
+	case RISCV_IOMMU_DDTP_MODE_OFF:
+		break;
+	default:
+		if (rq_mode != RISCV_IOMMU_DDTP_MODE_BARE &&
+		    rq_mode != RISCV_IOMMU_DDTP_MODE_OFF)
+			return -EINVAL;
+		break;
+	}
+
+	do {
+		rq_ddtp = FIELD_PREP(RISCV_IOMMU_DDTP_MODE, rq_mode);
+		if (rq_mode > RISCV_IOMMU_DDTP_MODE_BARE)
+			rq_ddtp |= phys_to_ppn(iommu->ddt_phys);
+
+		riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP, rq_ddtp);
+
+		rc = readq_relaxed_poll_timeout(iommu->reg + RISCV_IOMMU_REG_DDTP,
+						ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+						10, RISCV_IOMMU_DDTP_TIMEOUT);
+		if (rc < 0) {
+			dev_warn(dev, "timeout when setting ddtp (ddt mode: %u, read: %llx)\n",
+				 rq_mode, ddtp);
+			return -EBUSY;
+		}
+
+		/* Verify IOMMU hardware accepts new DDTP config. */
+		mode = FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp);
+
+		if (rq_mode == mode)
+			break;
+
+		/* Hardware mandatory DDTP mode has not been accepted. */
+		if (rq_mode < RISCV_IOMMU_DDTP_MODE_1LVL && rq_ddtp != ddtp) {
+			dev_warn(dev, "DDTP update failed hw: %llx vs %llx\n", ddtp, rq_ddtp);
+			return -EINVAL;
+		}
+
+		/*
+		 * Mode field is WARL, an IOMMU may support a subset of
+		 * directory table levels in which case if we tried to set
+		 * an unsupported number of levels we'll readback either
+		 * a valid xLVL or off/bare. If we got off/bare, try again
+		 * with a smaller xLVL.
+		 */
+		if (mode < RISCV_IOMMU_DDTP_MODE_1LVL &&
+		    rq_mode > RISCV_IOMMU_DDTP_MODE_1LVL) {
+			dev_dbg(dev, "DDTP hw mode %u vs %u\n", mode, rq_mode);
+			rq_mode--;
+			continue;
+		}
+
+		/*
+		 * We tried all supported modes and IOMMU hardware failed to
+		 * accept new settings, something went very wrong since off/bare
+		 * and at least one xLVL must be supported.
+		 */
+		dev_warn(dev, "DDTP hw mode %u, failed to set %u\n", mode, ddtp_mode);
+		return -EINVAL;
+	} while (1);
+
+	iommu->ddt_mode = mode;
+	if (mode != ddtp_mode)
+		dev_warn(dev, "DDTP failover to %u mode, requested %u\n",
+			 mode, ddtp_mode);
+
+	return 0;
+}
+
+static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu)
 {
-	/* Global pass-through already enabled, do nothing for now. */
+	u64 ddtp;
+	unsigned int mode;
+
+	riscv_iommu_readq_timeout(iommu, RISCV_IOMMU_REG_DDTP,
+				  ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+				  10, RISCV_IOMMU_DDTP_TIMEOUT);
+
+	if (ddtp & RISCV_IOMMU_DDTP_BUSY)
+		return -EBUSY;
+
+	/*
+	 * It is optional for the hardware to report a fixed address for device
+	 * directory root page when DDT.MODE is OFF or BARE.
+	 */
+	mode = FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp);
+	if (mode != RISCV_IOMMU_DDTP_MODE_BARE && mode != RISCV_IOMMU_DDTP_MODE_OFF) {
+		/* Use WARL to discover hardware fixed DDT PPN */
+		riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP,
+				   FIELD_PREP(RISCV_IOMMU_DDTP_MODE, mode));
+		riscv_iommu_readl_timeout(iommu, RISCV_IOMMU_REG_DDTP,
+					  ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+					  10, RISCV_IOMMU_DDTP_TIMEOUT);
+		if (ddtp & RISCV_IOMMU_DDTP_BUSY)
+			return -EBUSY;
+
+		iommu->ddt_phys = ppn_to_phys(ddtp);
+		if (iommu->ddt_phys)
+			iommu->ddt_root = devm_ioremap(iommu->dev, iommu->ddt_phys, PAGE_SIZE);
+		if (iommu->ddt_root)
+			memset(iommu->ddt_root, 0, PAGE_SIZE);
+	}
+
+	if (!iommu->ddt_root) {
+		iommu->ddt_root = (u64 *)riscv_iommu_get_pages(iommu, 0);
+		iommu->ddt_phys = __pa(iommu->ddt_root);
+	}
+
+	if (!iommu->ddt_root)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
+				     struct device *dev,
+				     struct iommu_domain *iommu_domain)
+{
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct riscv_iommu_dc *dc;
+	u64 fsc, ta, tc;
+	int i;
+
+	if (!iommu_domain) {
+		ta = 0;
+		tc = 0;
+		fsc = 0;
+	} else if (iommu_domain->type == IOMMU_DOMAIN_IDENTITY) {
+		ta = 0;
+		tc = RISCV_IOMMU_DC_TC_V;
+		fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE);
+	} else {
+		/* This should never happen. */
+		return -ENODEV;
+	}
+
+	/* Update existing or allocate new entries in device directory */
+	for (i = 0; i < fwspec->num_ids; i++) {
+		dc = riscv_iommu_get_dc(iommu, fwspec->ids[i], !iommu_domain);
+		if (!dc && !iommu_domain)
+			continue;
+		if (!dc)
+			return -ENODEV;
+
+		/* Swap device context, update TC valid bit as the last operation */
+		xchg64(&dc->fsc, fsc);
+		xchg64(&dc->ta, ta);
+		xchg64(&dc->tc, tc);
+
+		/* Device context invalidation will be required. Ignoring for now. */
+	}
+
 	return 0;
 }
 
+static int riscv_iommu_attach_identity_domain(struct iommu_domain *iommu_domain,
+					      struct device *dev)
+{
+	struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+
+	/* Global pass-through already enabled, do nothing. */
+	if (iommu->ddt_mode == RISCV_IOMMU_DDTP_MODE_BARE)
+		return 0;
+
+	return riscv_iommu_attach_domain(iommu, dev, iommu_domain);
+}
+
 static struct iommu_domain riscv_iommu_identity_domain = {
 	.type = IOMMU_DOMAIN_IDENTITY,
 	.ops = &(const struct iommu_domain_ops) {

@@ -82,6 +420,13 @@ static void riscv_iommu_probe_finalize(struct device *dev)
 	iommu_setup_dma_ops(dev, 0, U64_MAX);
 }
 
+static void riscv_iommu_release_device(struct device *dev)
+{
+	struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+
+	riscv_iommu_attach_domain(iommu, dev, NULL);
+}
+
 static const struct iommu_ops riscv_iommu_ops = {
 	.owner = THIS_MODULE,
 	.of_xlate = riscv_iommu_of_xlate,

@@ -90,6 +435,7 @@ static const struct iommu_ops riscv_iommu_ops = {
 	.device_group = riscv_iommu_device_group,
 	.probe_device = riscv_iommu_probe_device,
 	.probe_finalize = riscv_iommu_probe_finalize,
+	.release_device = riscv_iommu_release_device,
 };
 
 static int riscv_iommu_init_check(struct riscv_iommu_device *iommu)

@@ -124,6 +470,7 @@ void riscv_iommu_remove(struct riscv_iommu_device *iommu)
 {
 	iommu_device_unregister(&iommu->iommu);
 	iommu_device_sysfs_remove(&iommu->iommu);
+	riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF);
 }
 
 int riscv_iommu_init(struct riscv_iommu_device *iommu)

@@ -133,12 +480,14 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
 	rc = riscv_iommu_init_check(iommu);
 	if (rc)
 		return dev_err_probe(iommu->dev, rc, "unexpected device state\n");
-	/*
-	 * Placeholder for a complete IOMMU device initialization.
-	 * For now, only bare minimum: enable global identity mapping mode and register sysfs.
-	 */
-	riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP,
-			   FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE));
+
+	rc = riscv_iommu_ddt_alloc(iommu);
+	if (WARN(rc, "cannot allocate device directory\n"))
+		goto err_init;
+
+	rc = riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_MAX);
+	if (WARN(rc, "cannot enable iommu device\n"))
+		goto err_init;
 
 	rc = iommu_device_sysfs_add(&iommu->iommu, NULL, NULL, "riscv-iommu@%s",
 				    dev_name(iommu->dev));

@@ -154,5 +503,7 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
 err_iommu:
 	iommu_device_sysfs_remove(&iommu->iommu);
 err_sysfs:
+	riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF);
+err_init:
 	return rc;
 }

drivers/iommu/riscv/iommu.h +5 -0

index 700e33dc2446..f1696926582c 100644
@@ -34,6 +34,11 @@ struct riscv_iommu_device {
 	/* available interrupt numbers, MSI or WSI */
 	unsigned int irqs[RISCV_IOMMU_INTR_COUNT];
 	unsigned int irqs_count;
+
+	/* device directory */
+	unsigned int ddt_mode;
+	dma_addr_t ddt_phys;
+	u64 *ddt_root;
 };
 
 int riscv_iommu_init(struct riscv_iommu_device *iommu);

Signature · 1 line

2.34.1

Jason Gunthorpe

2024-04-19 12:40 UTC #14 lore raw

Re: [PATCH v2 5/7] iommu/riscv: Device directory management.

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 40

Quoted 12 lines · Tomasz Jeznach

drivers/iommu/riscv/iommu.c +6 -2

@@ -31,13 +32,350 @@ MODULE_LICENSE("GPL");
 /* Timeouts in [us] */
 #define RISCV_IOMMU_DDTP_TIMEOUT	50000
 
-static int riscv_iommu_attach_identity_domain(struct iommu_domain *domain,
-					      struct device *dev)
+/* RISC-V IOMMU PPN <> PHYS address conversions, PHYS <=> PPN[53:10] */
+#define phys_to_ppn(va)  (((va) >> 2) & (((1ULL << 44) - 1) << 10))
+#define ppn_to_phys(pn)	 (((pn) << 2) & (((1ULL << 44) - 1) << 12))
+
+#define dev_to_iommu(dev) \
+	container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu)

We have iommu_get_iommu_dev() now

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 73

Quoted 11 lines

drivers/iommu/riscv/iommu.c +11 -0

+static unsigned long riscv_iommu_get_pages(struct riscv_iommu_device *iommu, unsigned int order)
+{
+	struct riscv_iommu_devres *devres;
+	struct page *pages;
+
+	pages = alloc_pages_node(dev_to_node(iommu->dev),
+				 GFP_KERNEL_ACCOUNT | __GFP_ZERO, order);
+	if (unlikely(!pages)) {
+		dev_err(iommu->dev, "Page allocation failed, order %u\n", order);
+		return 0;
+	}

This needs adjusting for the recently merged allocation accounting

Quoted 11 lines

drivers/iommu/riscv/iommu.c +11 -0

+static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
+				     struct device *dev,
+				     struct iommu_domain *iommu_domain)
+{
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct riscv_iommu_dc *dc;
+	u64 fsc, ta, tc;
+	int i;
+
+	if (!iommu_domain) {
+		ta = 0;

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 346

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+		tc = 0;
+		fsc = 0;
+	} else if (iommu_domain->type == IOMMU_DOMAIN_IDENTITY) {
+		ta = 0;
+		tc = RISCV_IOMMU_DC_TC_V;
+		fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE);
+	} else {
+		/* This should never happen. */
+		return -ENODEV;
+	}

Please don't write it like this. This function is already being called by functions that are already under specific ops, don't check domain->type here.

Instead have the caller compute and pass in the ta/tc/fsc values. Maybe in a tidy struct..

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 354

Quoted 7 lines

drivers/iommu/riscv/iommu.c +7 -0

+	/* Update existing or allocate new entries in device directory */
+	for (i = 0; i < fwspec->num_ids; i++) {
+		dc = riscv_iommu_get_dc(iommu, fwspec->ids[i], !iommu_domain);
+		if (!dc && !iommu_domain)
+			continue;
+		if (!dc)
+			return -ENODEV;

But if this fails some of the fwspecs were left in a weird state ?

Drivers should try hard to have attach functions that fail and make no change at all or fully succeed.

Meaning ideally preallocate any required memory before doing any change to the HW visable structures.

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 359

Quoted 5 lines

drivers/iommu/riscv/iommu.c +5 -0

+
+		/* Swap device context, update TC valid bit as the last operation */
+		xchg64(&dc->fsc, fsc);
+		xchg64(&dc->ta, ta);
+		xchg64(&dc->tc, tc);

This doesn't loook right? When you get to adding PAGING suport fsc has the page table pfn and ta has the cache tag, so this will end up tearing the data for sure, eg when asked to replace a PAGING domain with another PAGING domain? That will create a functional/security problem, right?

I would encourage you to re-use the ARM sequencing code, ideally moved to some generic helper library. Every iommu driver dealing with multi-quanta descriptors seems to have this same fundamental sequencing problem.

drivers/iommu/riscv/iommu.c static void riscv_iommu_probe_finalize(struct device *dev) · line 428

Quoted 6 lines

drivers/iommu/riscv/iommu.c +6 -0

+static void riscv_iommu_release_device(struct device *dev)
+{
+	struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+
+	riscv_iommu_attach_domain(iommu, dev, NULL);
+}

The release_domain has landed too now. Please don't invent weird NULL domain types that have special meaning. I assume clearing the V bit is a blocking behavior? So please implement a proper blocking domain and set release_domain = &riscv_iommu_blocking and just omit this release function.

Quoted 4 lines

drivers/iommu/riscv/iommu.c +0 -0

@@ -133,12 +480,14 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
 	rc = riscv_iommu_init_check(iommu);
 	if (rc)
 		return dev_err_probe(iommu->dev, rc, "unexpected device state\n");

drivers/iommu/riscv/iommu.c int riscv_iommu_init(struct riscv_iommu_device *iommu) · line 486

Quoted 10 lines

drivers/iommu/riscv/iommu.c +4 -6

-	/*
-	 * Placeholder for a complete IOMMU device initialization.
-	 * For now, only bare minimum: enable global identity mapping mode and register sysfs.
-	 */
-	riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP,
-			   FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE));
+
+	rc = riscv_iommu_ddt_alloc(iommu);
+	if (WARN(rc, "cannot allocate device directory\n"))
+		goto err_init;

memory allocation failure already makes noisy prints, more prints are not needed..

drivers/iommu/riscv/iommu.c int riscv_iommu_init(struct riscv_iommu_device *iommu) · line 490

Quoted 3 lines

drivers/iommu/riscv/iommu.c +3 -0

+	rc = riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_MAX);
+	if (WARN(rc, "cannot enable iommu device\n"))
+		goto err_init;

This is not a proper use of WARN, it should only be used for things that cannot happen not undesired error paths.

Jason

Tomasz Jeznach

author

2024-04-24 23:01 UTC #15 lore raw

Re: [PATCH v2 5/7] iommu/riscv: Device directory management.

Quoted 68 lines · Jason Gunthorpe

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 40

Quoted 12 lines · Tomasz Jeznach

drivers/iommu/riscv/iommu.c +6 -2

@@ -31,13 +32,350 @@ MODULE_LICENSE("GPL");
 /* Timeouts in [us] */
 #define RISCV_IOMMU_DDTP_TIMEOUT     50000
 
-static int riscv_iommu_attach_identity_domain(struct iommu_domain *domain,
-                                           struct device *dev)
+/* RISC-V IOMMU PPN <> PHYS address conversions, PHYS <=> PPN[53:10] */
+#define phys_to_ppn(va)  (((va) >> 2) & (((1ULL << 44) - 1) << 10))
+#define ppn_to_phys(pn)       (((pn) << 2) & (((1ULL << 44) - 1) << 12))
+
+#define dev_to_iommu(dev) \
+     container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu)

We have iommu_get_iommu_dev() now

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 73

Quoted 11 lines

drivers/iommu/riscv/iommu.c +11 -0

+static unsigned long riscv_iommu_get_pages(struct riscv_iommu_device *iommu, unsigned int order)
+{
+     struct riscv_iommu_devres *devres;
+     struct page *pages;
+
+     pages = alloc_pages_node(dev_to_node(iommu->dev),
+                              GFP_KERNEL_ACCOUNT | __GFP_ZERO, order);
+     if (unlikely(!pages)) {
+             dev_err(iommu->dev, "Page allocation failed, order %u\n", order);
+             return 0;
+     }

This needs adjusting for the recently merged allocation accounting

Quoted 11 lines

drivers/iommu/riscv/iommu.c +11 -0

+static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
+                                  struct device *dev,
+                                  struct iommu_domain *iommu_domain)
+{
+     struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+     struct riscv_iommu_dc *dc;
+     u64 fsc, ta, tc;
+     int i;
+
+     if (!iommu_domain) {
+             ta = 0;

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 346

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+             tc = 0;
+             fsc = 0;
+     } else if (iommu_domain->type == IOMMU_DOMAIN_IDENTITY) {
+             ta = 0;
+             tc = RISCV_IOMMU_DC_TC_V;
+             fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE);
+     } else {
+             /* This should never happen. */
+             return -ENODEV;
+     }

Please don't write it like this. This function is already being called by functions that are already under specific ops, don't check domain->type here.

Instead have the caller compute and pass in the ta/tc/fsc values. Maybe in a tidy struct..

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 354

Quoted 7 lines

drivers/iommu/riscv/iommu.c +7 -0

+     /* Update existing or allocate new entries in device directory */
+     for (i = 0; i < fwspec->num_ids; i++) {
+             dc = riscv_iommu_get_dc(iommu, fwspec->ids[i], !iommu_domain);
+             if (!dc && !iommu_domain)
+                     continue;
+             if (!dc)
+                     return -ENODEV;

Quoted 8 lines

But if this fails some of the fwspecs were left in a weird state ?
Drivers should try hard to have attach functions that fail and make no change at all or fully succeed.
Meaning ideally preallocate any required memory before doing any change to the HW visable structures.

Good point. Done.
Looking at the fwspec->ids[] I'm assuming nobody will add/modify the IDs after iommu_probe_device() completes.

Quoted 12 lines

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 359
Quoted 5 lines
drivers/iommu/riscv/iommu.c +5 -0
+
+             /* Swap device context, update TC valid bit as the last operation */
+             xchg64(&dc->fsc, fsc);
+             xchg64(&dc->ta, ta);
+             xchg64(&dc->tc, tc);
This doesn't loook right? When you get to adding PAGING suport fsc has the page table pfn and ta has the cache tag, so this will end up tearing the data for sure, eg when asked to replace a PAGING domain with another PAGING domain? That will create a functional/security problem, right?

Quoted 5 lines

I would encourage you to re-use the ARM sequencing code, ideally moved to some generic helper library. Every iommu driver dealing with multi-quanta descriptors seems to have this same fundamental sequencing problem.

Good point. Reworked.

Quoted 4 lines

drivers/iommu/riscv/iommu.c +4 -0

+static void riscv_iommu_release_device(struct device *dev)
+{
+     struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+

Quoted 9 lines

drivers/iommu/riscv/iommu.c static void riscv_iommu_probe_finalize(struct device *dev) · line 428
Quoted 2 lines
drivers/iommu/riscv/iommu.c +2 -0
+     riscv_iommu_attach_domain(iommu, dev, NULL);
+}
The release_domain has landed too now. Please don't invent weird NULL domain types that have special meaning. I assume clearing the V bit is a blocking behavior? So please implement a proper blocking domain and set release_domain = &riscv_iommu_blocking and just omit this release function.

Updated to use release_domain, should be cleaner now. Clearing TC.V is a blocking (but noisy) behavior, should be fine for release domain where devices should be quiesced already.

Quoted 18 lines

Quoted 4 lines

drivers/iommu/riscv/iommu.c +0 -0

@@ -133,12 +480,14 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
      rc = riscv_iommu_init_check(iommu);
      if (rc)
              return dev_err_probe(iommu->dev, rc, "unexpected device state\n");

drivers/iommu/riscv/iommu.c int riscv_iommu_init(struct riscv_iommu_device *iommu) · line 486

Quoted 10 lines

drivers/iommu/riscv/iommu.c +4 -6

-     /*
-      * Placeholder for a complete IOMMU device initialization.
-      * For now, only bare minimum: enable global identity mapping mode and register sysfs.
-      */
-     riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP,
-                        FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE));
+
+     rc = riscv_iommu_ddt_alloc(iommu);
+     if (WARN(rc, "cannot allocate device directory\n"))
+             goto err_init;

memory allocation failure already makes noisy prints, more prints are not needed..

Quoted 8 lines

drivers/iommu/riscv/iommu.c int riscv_iommu_init(struct riscv_iommu_device *iommu) · line 490
Quoted 3 lines
drivers/iommu/riscv/iommu.c +3 -0
+     rc = riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_MAX);
+     if (WARN(rc, "cannot enable iommu device\n"))
+             goto err_init;
This is not a proper use of WARN, it should only be used for things that cannot happen not undesired error paths.
Jason

Thanks, ack to all. Will push updated v3 shortly.
- Tomasz

Jason Gunthorpe

2024-04-24 23:07 UTC #16 lore raw

Re: [PATCH v2 5/7] iommu/riscv: Device directory management.

Quoted 13 lines · Tomasz Jeznach

Quoted 13 lines
drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 354
Quoted 7 lines
drivers/iommu/riscv/iommu.c +7 -0
+     /* Update existing or allocate new entries in device directory */
+     for (i = 0; i < fwspec->num_ids; i++) {
+             dc = riscv_iommu_get_dc(iommu, fwspec->ids[i], !iommu_domain);
+             if (!dc && !iommu_domain)
+                     continue;
+             if (!dc)
+                     return -ENODEV;
But if this fails some of the fwspecs were left in a weird state ?
Drivers should try hard to have attach functions that fail and make no change at all or fully succeed.

Quoted 6 lines

Quoted 2 lines
Meaning ideally preallocate any required memory before doing any change to the HW visable structures.
Good point. Done.
Looking at the fwspec->ids[] I'm assuming nobody will add/modify the IDs after iommu_probe_device() completes.

Yes

Quoted 11 lines

Quoted 11 lines
drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 359
Quoted 4 lines
drivers/iommu/riscv/iommu.c +4 -0
+             /* Swap device context, update TC valid bit as the last operation */
+             xchg64(&dc->fsc, fsc);
+             xchg64(&dc->ta, ta);
+             xchg64(&dc->tc, tc);
This doesn't loook right? When you get to adding PAGING suport fsc has the page table pfn and ta has the cache tag, so this will end up tearing the data for sure, eg when asked to replace a PAGING domain with another PAGING domain? That will create a functional/security problem, right?

Quoted 7 lines

Quoted 5 lines
I would encourage you to re-use the ARM sequencing code, ideally moved to some generic helper library. Every iommu driver dealing with multi-quanta descriptors seems to have this same fundamental sequencing problem.
Good point. Reworked.

I suppose by force clearing the v bit before starting the sequence?

That is OK but won't support some non-embedded focused features in the long run. It is a good approach to get the driver landed though.

Quoted 10 lines

Quoted 6 lines
The release_domain has landed too now. Please don't invent weird NULL domain types that have special meaning. I assume clearing the V bit is a blocking behavior? So please implement a proper blocking domain and set release_domain = &riscv_iommu_blocking and just omit this release function.
Updated to use release_domain, should be cleaner now. Clearing TC.V is a blocking (but noisy) behavior, should be fine for release domain where devices should be quiesced already.

blocking is fine to be noisy.

Jason

Baolu Lu

2024-04-22 05:13 UTC #17 lore raw

Re: [PATCH v2 5/7] iommu/riscv: Device directory management.

In reply to Tomasz Jeznach

Quoted 20 lines · Tomasz Jeznach

Introduce device context allocation and device directory tree management including capabilities discovery sequence, as described in Chapter 2.1 of the RISC-V IOMMU Architecture Specification.
Device directory mode will be auto detected using DDTP WARL property, using highest mode supported by the driver and hardware. If none supported can be configured, driver will fall back to global pass-through.
First level DDTP page can be located in I/O (detected using DDTP WARL) and system memory.
Only identity protection domain is supported by this implementation.
Co-developed-by Nick Kossifidis <[email protected]>
Signed-off-by Nick Kossifidis <[email protected]>
Signed-off-by Tomasz Jeznach <[email protected]>
---
  drivers/iommu/riscv/iommu.c | 369 +++++++++++++++++++++++++++++++++++-
  drivers/iommu/riscv/iommu.h |   5 +
  2 files changed, 365 insertions(+), 9 deletions(-)

[ ... ]

Quoted 19 lines

drivers/iommu/riscv/iommu.c +19 -0

+
+/*
+ * Discover supported DDT modes starting from requested value,
+ * configure DDTP register with accepted mode and root DDT address.
+ * Accepted iommu->ddt_mode is updated on success.
+ */
+static int riscv_iommu_set_ddtp_mode(struct riscv_iommu_device *iommu,
+				     unsigned int ddtp_mode)
+{
+	struct device *dev = iommu->dev;
+	u64 ddtp, rq_ddtp;
+	unsigned int mode, rq_mode = ddtp_mode;
+	int rc;
+
+	rc = readq_relaxed_poll_timeout(iommu->reg + RISCV_IOMMU_REG_DDTP,
+					ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+					10, RISCV_IOMMU_DDTP_TIMEOUT);
+	if (rc < 0)
+		return -EBUSY;

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 218

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+
+	/* Disallow state transition from xLVL to xLVL. */
+	switch (FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp)) {
+	case RISCV_IOMMU_DDTP_MODE_BARE:
+	case RISCV_IOMMU_DDTP_MODE_OFF:
+		break;
+	default:
+		if (rq_mode != RISCV_IOMMU_DDTP_MODE_BARE &&
+		    rq_mode != RISCV_IOMMU_DDTP_MODE_OFF)
+			return -EINVAL;

Is this check duplicate? It appears that it's always true in the default branch.

Quoted 62 lines

drivers/iommu/riscv/iommu.c +62 -0

+		break;
+	}
+
+	do {
+		rq_ddtp = FIELD_PREP(RISCV_IOMMU_DDTP_MODE, rq_mode);
+		if (rq_mode > RISCV_IOMMU_DDTP_MODE_BARE)
+			rq_ddtp |= phys_to_ppn(iommu->ddt_phys);
+
+		riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP, rq_ddtp);
+
+		rc = readq_relaxed_poll_timeout(iommu->reg + RISCV_IOMMU_REG_DDTP,
+						ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+						10, RISCV_IOMMU_DDTP_TIMEOUT);
+		if (rc < 0) {
+			dev_warn(dev, "timeout when setting ddtp (ddt mode: %u, read: %llx)\n",
+				 rq_mode, ddtp);
+			return -EBUSY;
+		}
+
+		/* Verify IOMMU hardware accepts new DDTP config. */
+		mode = FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp);
+
+		if (rq_mode == mode)
+			break;
+
+		/* Hardware mandatory DDTP mode has not been accepted. */
+		if (rq_mode < RISCV_IOMMU_DDTP_MODE_1LVL && rq_ddtp != ddtp) {
+			dev_warn(dev, "DDTP update failed hw: %llx vs %llx\n", ddtp, rq_ddtp);
+			return -EINVAL;
+		}
+
+		/*
+		 * Mode field is WARL, an IOMMU may support a subset of
+		 * directory table levels in which case if we tried to set
+		 * an unsupported number of levels we'll readback either
+		 * a valid xLVL or off/bare. If we got off/bare, try again
+		 * with a smaller xLVL.
+		 */
+		if (mode < RISCV_IOMMU_DDTP_MODE_1LVL &&
+		    rq_mode > RISCV_IOMMU_DDTP_MODE_1LVL) {
+			dev_dbg(dev, "DDTP hw mode %u vs %u\n", mode, rq_mode);
+			rq_mode--;
+			continue;
+		}
+
+		/*
+		 * We tried all supported modes and IOMMU hardware failed to
+		 * accept new settings, something went very wrong since off/bare
+		 * and at least one xLVL must be supported.
+		 */
+		dev_warn(dev, "DDTP hw mode %u, failed to set %u\n", mode, ddtp_mode);
+		return -EINVAL;
+	} while (1);
+
+	iommu->ddt_mode = mode;
+	if (mode != ddtp_mode)
+		dev_warn(dev, "DDTP failover to %u mode, requested %u\n",
+			 mode, ddtp_mode);
+
+	return 0;
+}
+

[ ... ]

Quoted 12 lines

drivers/iommu/riscv/iommu.c +12 -0

+
+static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
+				     struct device *dev,
+				     struct iommu_domain *iommu_domain)
+{
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct riscv_iommu_dc *dc;
+	u64 fsc, ta, tc;
+	int i;
+
+	if (!iommu_domain) {
+		ta = 0;

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 346

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+		tc = 0;
+		fsc = 0;
+	} else if (iommu_domain->type == IOMMU_DOMAIN_IDENTITY) {
+		ta = 0;
+		tc = RISCV_IOMMU_DC_TC_V;
+		fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE);
+	} else {
+		/* This should never happen. */
+		return -ENODEV;
+	}

Move the domain->type check code to the domain-specific ops.

Quoted 36 lines

drivers/iommu/riscv/iommu.c +29 -0

+
+	/* Update existing or allocate new entries in device directory */
+	for (i = 0; i < fwspec->num_ids; i++) {
+		dc = riscv_iommu_get_dc(iommu, fwspec->ids[i], !iommu_domain);
+		if (!dc && !iommu_domain)
+			continue;
+		if (!dc)
+			return -ENODEV;
+
+		/* Swap device context, update TC valid bit as the last operation */
+		xchg64(&dc->fsc, fsc);
+		xchg64(&dc->ta, ta);
+		xchg64(&dc->tc, tc);
+
+		/* Device context invalidation will be required. Ignoring for now. */
+	}
+
  	return 0;
  }
 
+static int riscv_iommu_attach_identity_domain(struct iommu_domain *iommu_domain,
+					      struct device *dev)
+{
+	struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+
+	/* Global pass-through already enabled, do nothing. */
+	if (iommu->ddt_mode == RISCV_IOMMU_DDTP_MODE_BARE)
+		return 0;
+
+	return riscv_iommu_attach_domain(iommu, dev, iommu_domain);
+}
+
  static struct iommu_domain riscv_iommu_identity_domain = {
  	.type = IOMMU_DOMAIN_IDENTITY,
  	.ops = &(const struct iommu_domain_ops) {

drivers/iommu/riscv/iommu.c static void riscv_iommu_probe_finalize(struct device *dev) · line 427

Quoted 8 lines

drivers/iommu/riscv/iommu.c +5 -0

@@ -82,6 +420,13 @@ static void riscv_iommu_probe_finalize(struct device *dev)
  	iommu_setup_dma_ops(dev, 0, U64_MAX);
  }
 
+static void riscv_iommu_release_device(struct device *dev)
+{
+	struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+
+	riscv_iommu_attach_domain(iommu, dev, NULL);

Attaching a NULL domain to a device has already been removed. You can use the iommu_ops->release_domain here.

drivers/iommu/riscv/iommu.c static const struct iommu_ops riscv_iommu_ops = { · line 437

Quoted 9 lines

drivers/iommu/riscv/iommu.c +2 -0

+}
+
  static const struct iommu_ops riscv_iommu_ops = {
  	.owner = THIS_MODULE,
  	.of_xlate = riscv_iommu_of_xlate,

@@ -90,6 +435,7 @@ static const struct iommu_ops riscv_iommu_ops = {
  	.device_group = riscv_iommu_device_group,
  	.probe_device = riscv_iommu_probe_device,
  	.probe_finalize = riscv_iommu_probe_finalize,

The probe_finalize op will be removed soon.

https://lore.kernel.org/linux-iommu/bebea331c1d688b34d9862eefd5ede47503961b8.1713523152.git.robin.murphy@arm.com/

Quoted 47 lines

drivers/iommu/riscv/iommu.c +12 -6

+	.release_device = riscv_iommu_release_device,
  };
 
  static int riscv_iommu_init_check(struct riscv_iommu_device *iommu)

@@ -124,6 +470,7 @@ void riscv_iommu_remove(struct riscv_iommu_device *iommu)
  {
  	iommu_device_unregister(&iommu->iommu);
  	iommu_device_sysfs_remove(&iommu->iommu);
+	riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF);
  }
 
  int riscv_iommu_init(struct riscv_iommu_device *iommu)

@@ -133,12 +480,14 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
  	rc = riscv_iommu_init_check(iommu);
  	if (rc)
  		return dev_err_probe(iommu->dev, rc, "unexpected device state\n");
-	/*
-	 * Placeholder for a complete IOMMU device initialization.
-	 * For now, only bare minimum: enable global identity mapping mode and register sysfs.
-	 */
-	riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP,
-			   FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE));
+
+	rc = riscv_iommu_ddt_alloc(iommu);
+	if (WARN(rc, "cannot allocate device directory\n"))
+		goto err_init;
+
+	rc = riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_MAX);
+	if (WARN(rc, "cannot enable iommu device\n"))
+		goto err_init;
 
  	rc = iommu_device_sysfs_add(&iommu->iommu, NULL, NULL, "riscv-iommu@%s",
  				    dev_name(iommu->dev));

@@ -154,5 +503,7 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
  err_iommu:
  	iommu_device_sysfs_remove(&iommu->iommu);
  err_sysfs:
+	riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF);
+err_init:
  	return rc;
  }

drivers/iommu/riscv/iommu.h +0 -0

index 700e33dc2446..f1696926582c 100644
@@ -34,6 +34,11 @@ struct riscv_iommu_device {
  	/* available interrupt numbers, MSI or WSI */

drivers/iommu/riscv/iommu.h struct riscv_iommu_device { · line 44

Quoted 10 lines

drivers/iommu/riscv/iommu.h +5 -0

  	unsigned int irqs[RISCV_IOMMU_INTR_COUNT];
  	unsigned int irqs_count;
+
+	/* device directory */
+	unsigned int ddt_mode;
+	dma_addr_t ddt_phys;
+	u64 *ddt_root;
  };
 
  int riscv_iommu_init(struct riscv_iommu_device *iommu);

Best regards,
baolu

Tomasz Jeznach

author

2024-04-24 23:07 UTC #18 lore raw

Re: [PATCH v2 5/7] iommu/riscv: Device directory management.

Quoted 49 lines · Baolu Lu

Quoted 20 lines · Tomasz Jeznach
Introduce device context allocation and device directory tree management including capabilities discovery sequence, as described in Chapter 2.1 of the RISC-V IOMMU Architecture Specification.
Device directory mode will be auto detected using DDTP WARL property, using highest mode supported by the driver and hardware. If none supported can be configured, driver will fall back to global pass-through.
First level DDTP page can be located in I/O (detected using DDTP WARL) and system memory.
Only identity protection domain is supported by this implementation.
Co-developed-by Nick Kossifidis <[email protected]>
Signed-off-by Nick Kossifidis <[email protected]>
Signed-off-by Tomasz Jeznach <[email protected]>
---
  drivers/iommu/riscv/iommu.c | 369 +++++++++++++++++++++++++++++++++++-
  drivers/iommu/riscv/iommu.h |   5 +
  2 files changed, 365 insertions(+), 9 deletions(-)
[ ... ]
Quoted 24 lines
drivers/iommu/riscv/iommu.c +24 -0
+
+/*
+ * Discover supported DDT modes starting from requested value,
+ * configure DDTP register with accepted mode and root DDT address.
+ * Accepted iommu->ddt_mode is updated on success.
+ */
+static int riscv_iommu_set_ddtp_mode(struct riscv_iommu_device *iommu,
+                                  unsigned int ddtp_mode)
+{
+     struct device *dev = iommu->dev;
+     u64 ddtp, rq_ddtp;
+     unsigned int mode, rq_mode = ddtp_mode;
+     int rc;
+
+     rc = readq_relaxed_poll_timeout(iommu->reg + RISCV_IOMMU_REG_DDTP,
+                                     ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+                                     10, RISCV_IOMMU_DDTP_TIMEOUT);
+     if (rc < 0)
+             return -EBUSY;
+
+     /* Disallow state transition from xLVL to xLVL. */
+     switch (FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp)) {
+     case RISCV_IOMMU_DDTP_MODE_BARE:
+     case RISCV_IOMMU_DDTP_MODE_OFF:

Quoted 9 lines

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 218
Quoted 5 lines
drivers/iommu/riscv/iommu.c +5 -0
+             break;
+     default:
+             if (rq_mode != RISCV_IOMMU_DDTP_MODE_BARE &&
+                 rq_mode != RISCV_IOMMU_DDTP_MODE_OFF)
+                     return -EINVAL;
Is this check duplicate? It appears that it's always true in the default branch.

No. The condition in the switch represents current mode, while check in default branch checks desired mode device will be configured to. I've reworked the code in v3 to be more readable.

Quoted 150 lines

Quoted 62 lines

drivers/iommu/riscv/iommu.c +62 -0

+             break;
+     }
+
+     do {
+             rq_ddtp = FIELD_PREP(RISCV_IOMMU_DDTP_MODE, rq_mode);
+             if (rq_mode > RISCV_IOMMU_DDTP_MODE_BARE)
+                     rq_ddtp |= phys_to_ppn(iommu->ddt_phys);
+
+             riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP, rq_ddtp);
+
+             rc = readq_relaxed_poll_timeout(iommu->reg + RISCV_IOMMU_REG_DDTP,
+                                             ddtp, !(ddtp & RISCV_IOMMU_DDTP_BUSY),
+                                             10, RISCV_IOMMU_DDTP_TIMEOUT);
+             if (rc < 0) {
+                     dev_warn(dev, "timeout when setting ddtp (ddt mode: %u, read: %llx)\n",
+                              rq_mode, ddtp);
+                     return -EBUSY;
+             }
+
+             /* Verify IOMMU hardware accepts new DDTP config. */
+             mode = FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp);
+
+             if (rq_mode == mode)
+                     break;
+
+             /* Hardware mandatory DDTP mode has not been accepted. */
+             if (rq_mode < RISCV_IOMMU_DDTP_MODE_1LVL && rq_ddtp != ddtp) {
+                     dev_warn(dev, "DDTP update failed hw: %llx vs %llx\n", ddtp, rq_ddtp);
+                     return -EINVAL;
+             }
+
+             /*
+              * Mode field is WARL, an IOMMU may support a subset of
+              * directory table levels in which case if we tried to set
+              * an unsupported number of levels we'll readback either
+              * a valid xLVL or off/bare. If we got off/bare, try again
+              * with a smaller xLVL.
+              */
+             if (mode < RISCV_IOMMU_DDTP_MODE_1LVL &&
+                 rq_mode > RISCV_IOMMU_DDTP_MODE_1LVL) {
+                     dev_dbg(dev, "DDTP hw mode %u vs %u\n", mode, rq_mode);
+                     rq_mode--;
+                     continue;
+             }
+
+             /*
+              * We tried all supported modes and IOMMU hardware failed to
+              * accept new settings, something went very wrong since off/bare
+              * and at least one xLVL must be supported.
+              */
+             dev_warn(dev, "DDTP hw mode %u, failed to set %u\n", mode, ddtp_mode);
+             return -EINVAL;
+     } while (1);
+
+     iommu->ddt_mode = mode;
+     if (mode != ddtp_mode)
+             dev_warn(dev, "DDTP failover to %u mode, requested %u\n",
+                      mode, ddtp_mode);
+
+     return 0;
+}
+

[ ... ]

Quoted 12 lines

drivers/iommu/riscv/iommu.c +12 -0

+
+static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
+                                  struct device *dev,
+                                  struct iommu_domain *iommu_domain)
+{
+     struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+     struct riscv_iommu_dc *dc;
+     u64 fsc, ta, tc;
+     int i;
+
+     if (!iommu_domain) {
+             ta = 0;

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 346

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+             tc = 0;
+             fsc = 0;
+     } else if (iommu_domain->type == IOMMU_DOMAIN_IDENTITY) {
+             ta = 0;
+             tc = RISCV_IOMMU_DC_TC_V;
+             fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE);
+     } else {
+             /* This should never happen. */
+             return -ENODEV;
+     }

Move the domain->type check code to the domain-specific ops.

Quoted 36 lines

drivers/iommu/riscv/iommu.c +29 -0

+
+     /* Update existing or allocate new entries in device directory */
+     for (i = 0; i < fwspec->num_ids; i++) {
+             dc = riscv_iommu_get_dc(iommu, fwspec->ids[i], !iommu_domain);
+             if (!dc && !iommu_domain)
+                     continue;
+             if (!dc)
+                     return -ENODEV;
+
+             /* Swap device context, update TC valid bit as the last operation */
+             xchg64(&dc->fsc, fsc);
+             xchg64(&dc->ta, ta);
+             xchg64(&dc->tc, tc);
+
+             /* Device context invalidation will be required. Ignoring for now. */
+     }
+
      return 0;
  }
 
+static int riscv_iommu_attach_identity_domain(struct iommu_domain *iommu_domain,
+                                           struct device *dev)
+{
+     struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+
+     /* Global pass-through already enabled, do nothing. */
+     if (iommu->ddt_mode == RISCV_IOMMU_DDTP_MODE_BARE)
+             return 0;
+
+     return riscv_iommu_attach_domain(iommu, dev, iommu_domain);
+}
+
  static struct iommu_domain riscv_iommu_identity_domain = {
      .type = IOMMU_DOMAIN_IDENTITY,
      .ops = &(const struct iommu_domain_ops) {

drivers/iommu/riscv/iommu.c static void riscv_iommu_probe_finalize(struct device *dev) · line 427

Quoted 8 lines

drivers/iommu/riscv/iommu.c +5 -0

@@ -82,6 +420,13 @@ static void riscv_iommu_probe_finalize(struct device *dev)
      iommu_setup_dma_ops(dev, 0, U64_MAX);
  }
 
+static void riscv_iommu_release_device(struct device *dev)
+{
+     struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+
+     riscv_iommu_attach_domain(iommu, dev, NULL);

Attaching a NULL domain to a device has already been removed. You can use the iommu_ops->release_domain here.

drivers/iommu/riscv/iommu.c static const struct iommu_ops riscv_iommu_ops = { · line 437

Quoted 9 lines

drivers/iommu/riscv/iommu.c +2 -0

+}
+
  static const struct iommu_ops riscv_iommu_ops = {
      .owner = THIS_MODULE,
      .of_xlate = riscv_iommu_of_xlate,

@@ -90,6 +435,7 @@ static const struct iommu_ops riscv_iommu_ops = {
      .device_group = riscv_iommu_device_group,
      .probe_device = riscv_iommu_probe_device,
      .probe_finalize = riscv_iommu_probe_finalize,

The probe_finalize op will be removed soon.

Quoted 1 line

https://lore.kernel.org/linux-iommu/bebea331c1d688b34d9862eefd5ede47503961b8.1713523152.git.robin.murphy@arm.com/

Thanks, I'm aware of the change, once this change is pulled for iommu/next I'll just remove probe_finalize.

Quoted 57 lines

drivers/iommu/riscv/iommu.c +12 -6

+     .release_device = riscv_iommu_release_device,
  };
 
  static int riscv_iommu_init_check(struct riscv_iommu_device *iommu)

@@ -124,6 +470,7 @@ void riscv_iommu_remove(struct riscv_iommu_device *iommu)
  {
      iommu_device_unregister(&iommu->iommu);
      iommu_device_sysfs_remove(&iommu->iommu);
+     riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF);
  }
 
  int riscv_iommu_init(struct riscv_iommu_device *iommu)

@@ -133,12 +480,14 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
      rc = riscv_iommu_init_check(iommu);
      if (rc)
              return dev_err_probe(iommu->dev, rc, "unexpected device state\n");
-     /*
-      * Placeholder for a complete IOMMU device initialization.
-      * For now, only bare minimum: enable global identity mapping mode and register sysfs.
-      */
-     riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP,
-                        FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE));
+
+     rc = riscv_iommu_ddt_alloc(iommu);
+     if (WARN(rc, "cannot allocate device directory\n"))
+             goto err_init;
+
+     rc = riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_MAX);
+     if (WARN(rc, "cannot enable iommu device\n"))
+             goto err_init;
 
      rc = iommu_device_sysfs_add(&iommu->iommu, NULL, NULL, "riscv-iommu@%s",
                                  dev_name(iommu->dev));

@@ -154,5 +503,7 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu)
  err_iommu:
      iommu_device_sysfs_remove(&iommu->iommu);
  err_sysfs:
+     riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF);
+err_init:
      return rc;
  }

drivers/iommu/riscv/iommu.h +5 -0

index 700e33dc2446..f1696926582c 100644
@@ -34,6 +34,11 @@ struct riscv_iommu_device {
      /* available interrupt numbers, MSI or WSI */
      unsigned int irqs[RISCV_IOMMU_INTR_COUNT];
      unsigned int irqs_count;
+
+     /* device directory */
+     unsigned int ddt_mode;
+     dma_addr_t ddt_phys;
+     u64 *ddt_root;
  };

Quoted 4 lines

drivers/iommu/riscv/iommu.h struct riscv_iommu_device { · line 44
Quoted 1 line
drivers/iommu/riscv/iommu.h +0 -0
  int riscv_iommu_init(struct riscv_iommu_device *iommu);
Best regards,
baolu

Thank you, Best.
- Tomasz

Tomasz Jeznach

author

2024-04-18 16:32 UTC #19 lore raw

PATCH v2 6/7 iommu/riscv: Command and fault queue support

In reply to Tomasz Jeznach

Introduce device command submission and fault reporting queues, as described in Chapter 3.1 and 3.2 of the RISC-V IOMMU Architecture Specification.

Command and fault queues are instantiated in contiguous system memory local to IOMMU device domain, or mapped from fixed I/O space provided by the hardware implementation. Detection of the location and maximum allowed size of the queue utilize WARL properties of queue base control register. Driver implementation will try to allocate up to 128KB of system memory, while respecting hardware supported maximum queue size.

Interrupts allocation is based on interrupt vectors availability and ditributed to all queues in simple round-robin fashion. For hardware Implementation with fixed event type to interrupt vector assignment IVEC WARL property is used to discover such mappings.

Address translation, command and queue fault handling in this change is limited to simple fault reporing without taking any action.

Signed-off-by Tomasz Jeznach <[email protected]>

---
 drivers/iommu/riscv/iommu-bits.h |  75 +++++
 drivers/iommu/riscv/iommu.c      | 473 ++++++++++++++++++++++++++++++-
 drivers/iommu/riscv/iommu.h      |  21 ++
 3 files changed, 567 insertions(+), 2 deletions(-)

drivers/iommu/riscv/iommu-bits.h +75 -0

index ba093c29de9f..40c379222821 100644
@@ -704,4 +704,79 @@ struct riscv_iommu_msi_pte {
 #define RISCV_IOMMU_MSI_MRIF_NPPN	RISCV_IOMMU_PPN_FIELD
 #define RISCV_IOMMU_MSI_MRIF_NID_MSB	BIT_ULL(60)
 
+/* Helper functions: command structure builders. */
+
+static inline void riscv_iommu_cmd_inval_vma(struct riscv_iommu_command *cmd)
+{
+	cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IOTINVAL_OPCODE) |
+		      FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA);
+	cmd->dword1 = 0;
+}
+
+static inline void riscv_iommu_cmd_inval_set_addr(struct riscv_iommu_command *cmd,
+						  u64 addr)
+{
+	cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, phys_to_pfn(addr));
+	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV;
+}
+
+static inline void riscv_iommu_cmd_inval_set_pscid(struct riscv_iommu_command *cmd,
+						   int pscid)
+{
+	cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_PSCID, pscid) |
+		       RISCV_IOMMU_CMD_IOTINVAL_PSCV;
+}
+
+static inline void riscv_iommu_cmd_inval_set_gscid(struct riscv_iommu_command *cmd,
+						   int gscid)
+{
+	cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_GSCID, gscid) |
+		       RISCV_IOMMU_CMD_IOTINVAL_GV;
+}
+
+static inline void riscv_iommu_cmd_iofence(struct riscv_iommu_command *cmd)
+{
+	cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IOFENCE_OPCODE) |
+		      FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IOFENCE_FUNC_C) |
+		      RISCV_IOMMU_CMD_IOFENCE_PR | RISCV_IOMMU_CMD_IOFENCE_PW;
+	cmd->dword1 = 0;
+}
+
+static inline void riscv_iommu_cmd_iofence_set_av(struct riscv_iommu_command *cmd,
+						  u64 addr, u32 data)
+{
+	cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IOFENCE_OPCODE) |
+		      FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IOFENCE_FUNC_C) |
+		      FIELD_PREP(RISCV_IOMMU_CMD_IOFENCE_DATA, data) |
+		      RISCV_IOMMU_CMD_IOFENCE_AV;
+	cmd->dword1 = addr >> 2;
+}
+
+static inline void riscv_iommu_cmd_iodir_inval_ddt(struct riscv_iommu_command *cmd)
+{
+	cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IODIR_OPCODE) |
+		      FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT);
+	cmd->dword1 = 0;
+}
+
+static inline void riscv_iommu_cmd_iodir_inval_pdt(struct riscv_iommu_command *cmd)
+{
+	cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IODIR_OPCODE) |
+		      FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT);
+	cmd->dword1 = 0;
+}
+
+static inline void riscv_iommu_cmd_iodir_set_did(struct riscv_iommu_command *cmd,
+						 unsigned int devid)
+{
+	cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_IODIR_DID, devid) |
+		       RISCV_IOMMU_CMD_IODIR_DV;
+}
+
+static inline void riscv_iommu_cmd_iodir_set_pid(struct riscv_iommu_command *cmd,
+						 unsigned int pasid)
+{
+	cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_IODIR_PID, pasid);
+}
+
 #endif /* _RISCV_IOMMU_BITS_H_ */

drivers/iommu/riscv/iommu.c +471 -2

526 lines not on this page · raw message

drivers/iommu/riscv/iommu.h +21 -0

index f1696926582c..03e0c45bc7e1 100644
@@ -17,6 +17,22 @@
 
 #include "iommu-bits.h"
 
+struct riscv_iommu_device;
+
+struct riscv_iommu_queue {
+	atomic_t prod;				/* unbounded producer allocation index */
+	atomic_t head;				/* unbounded shadow ring buffer consumer index */
+	atomic_t tail;				/* unbounded shadow ring buffer producer index */
+	unsigned int mask;			/* index mask, queue length - 1 */
+	unsigned int irq;			/* allocated interrupt number */
+	struct riscv_iommu_device *iommu;	/* iommu device handling the queue when active */
+	void *base;				/* ring buffer kernel pointer */
+	dma_addr_t phys;			/* ring buffer physical address */
+	u16 qbr;				/* base register offset, head and tail reference */
+	u16 qcr;				/* control and status register offset */
+	u8 qid;					/* queue identifier, same as RISCV_IOMMU_INTR_XX */
+};
+
 struct riscv_iommu_device {
 	/* iommu core interface */
 	struct iommu_device iommu;

@@ -34,6 +50,11 @@ struct riscv_iommu_device {
 	/* available interrupt numbers, MSI or WSI */
 	unsigned int irqs[RISCV_IOMMU_INTR_COUNT];
 	unsigned int irqs_count;
+	unsigned int ivec;
+
+	/* hardware queues */
+	struct riscv_iommu_queue cmdq;
+	struct riscv_iommu_queue fltq;
 
 	/* device directory */
 	unsigned int ddt_mode;

Signature · 1 line

2.34.1

Tomasz Jeznach

author

2024-04-18 16:32 UTC #20 lore raw

PATCH v2 7/7 iommu/riscv: Paging domain support

In reply to Tomasz Jeznach

Introduce first-stage address translation support.

Page table configured by the IOMMU driver will use the same format as the CPU’s MMU, and will fallback to identity translation if the page table format configured for the MMU is not supported by the IOMMU hardware.

This change introduces IOTINVAL.VMA command, required to invalidate any cached IOATC entries after mapping is updated and/or removed from the paging domain. Invalidations for the non-leaf page entries will be added to the driver code in separate patch series, following spec update to clarify non-leaf cache invalidation command. With this patch, allowing only 4K mappings and keeping non-leaf page entries in memory this should be a reasonable simplification.

Signed-off-by Tomasz Jeznach <[email protected]>

---
 drivers/iommu/riscv/Kconfig |   1 +
 drivers/iommu/riscv/iommu.c | 467 +++++++++++++++++++++++++++++++++++-
 2 files changed, 466 insertions(+), 2 deletions(-)

drivers/iommu/riscv/Kconfig +1 -0

index 711326992585..6f9fb396034a 100644
@@ -7,6 +7,7 @@ config RISCV_IOMMU
 	select DMA_OPS
 	select IOMMU_API
 	select IOMMU_IOVA
+	select IOMMU_DMA
 	help
 	  Support for implementations of the RISC-V IOMMU architecture that
 	  complements the RISC-V MMU capabilities, providing similar address

drivers/iommu/riscv/iommu.c +465 -2

516 lines not on this page · raw message

Signature · 1 line

2.34.1

Jason Gunthorpe

2024-04-19 12:56 UTC #21 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 52

Quoted 12 lines · Tomasz Jeznach

drivers/iommu/riscv/iommu.c +4 -0

index a4f74588cdc2..32ddc372432d 100644
@@ -46,6 +46,10 @@ MODULE_LICENSE("GPL");
 #define dev_to_iommu(dev) \
 	container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu)
 
+/* IOMMU PSCID allocation namespace. */
+static DEFINE_IDA(riscv_iommu_pscids);
+#define RISCV_IOMMU_MAX_PSCID		BIT(20)
+

You may consider putting this IDA in the riscv_iommu_device() and move the pscid from the domain to the bond?

Quoted 14 lines

drivers/iommu/riscv/iommu.c +7 -0

 /* Device resource-managed allocations */
 struct riscv_iommu_devres {
 	unsigned long addr;

@@ -752,12 +756,77 @@ static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu)
 	return 0;
 }
 
+struct riscv_iommu_bond {
+	struct list_head list;
+	struct rcu_head rcu;
+	struct device *dev;
+};
+
+/* This struct contains protection domain specific IOMMU driver data. */

drivers/iommu/riscv/iommu.c static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu) · line 775

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+struct riscv_iommu_domain {
+	struct iommu_domain domain;
+	struct list_head bonds;
+	int pscid;
+	int numa_node;
+	int amo_enabled:1;
+	unsigned int pgd_mode;
+	/* paging domain */
+	unsigned long pgd_root;
+};

Glad to see there is no riscv_iommu_device pointer in the domain!

drivers/iommu/riscv/iommu.c static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu) · line 799

Quoted 12 lines

drivers/iommu/riscv/iommu.c +12 -0

+static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
+				    unsigned long start, unsigned long end)
+{
+	struct riscv_iommu_bond *bond;
+	struct riscv_iommu_device *iommu;
+	struct riscv_iommu_command cmd;
+	unsigned long len = end - start + 1;
+	unsigned long iova;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(bond, &domain->bonds, list) {
+		iommu = dev_to_iommu(bond->dev);

Pedantically this locking isn't locked right, there is technically nothing that prevents bond->dev and the iommu instance struct from being freed here. eg iommufd can hit races here if userspace can hot unplug devices.

I suggest storing the iommu pointer itself in the bond instead of the device then add a synchronize_rcu() to the iommu unregister path.

drivers/iommu/riscv/iommu.c static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu) · line 810

Quoted 11 lines

drivers/iommu/riscv/iommu.c +11 -0

+		riscv_iommu_cmd_inval_vma(&cmd);
+		riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid);
+		if (len > 0 && len < RISCV_IOMMU_IOTLB_INVAL_LIMIT) {
+			for (iova = start; iova < end; iova += PAGE_SIZE) {
+				riscv_iommu_cmd_inval_set_addr(&cmd, iova);
+				riscv_iommu_cmd_send(iommu, &cmd, 0);
+			}
+		} else {
+			riscv_iommu_cmd_send(iommu, &cmd, 0);
+		}
+	}

This seems suboptimal, you probably want to copy the new design that Intel is doing where you allocate "bonds" that are already de-duplicated. Ie if I have 10 devices on the same iommu sharing the domain the above will invalidate the PSCID 10 times. It should only be done once.

ie add a "bond" for the (iommu,pscid) and refcount that based on how many devices are used. Then another "bond" for the ATS stuff eventually.

Quoted 12 lines

drivers/iommu/riscv/iommu.c +10 -0

+
+	list_for_each_entry_rcu(bond, &domain->bonds, list) {
+		iommu = dev_to_iommu(bond->dev);
+
+		riscv_iommu_cmd_iofence(&cmd);
+		riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_QUEUE_TIMEOUT);
+	}
+	rcu_read_unlock();
+}
+

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 874

Quoted 6 lines

drivers/iommu/riscv/iommu.c +2 -1

@@ -787,12 +870,390 @@ static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
 		xchg64(&dc->ta, ta);
 		xchg64(&dc->tc, tc);
 
-		/* Device context invalidation will be required. Ignoring for now. */
+		if (!(tc & RISCV_IOMMU_DC_TC_V))
+			continue;

No negative caching in HW?

Quoted 11 lines

drivers/iommu/riscv/iommu.c +11 -0

+		/* Invalidate device context cache */
+		riscv_iommu_cmd_iodir_inval_ddt(&cmd);
+		riscv_iommu_cmd_iodir_set_did(&cmd, fwspec->ids[i]);
+		riscv_iommu_cmd_send(iommu, &cmd, 0);
+
+		if (FIELD_GET(RISCV_IOMMU_PC_FSC_MODE, fsc) == RISCV_IOMMU_DC_FSC_MODE_BARE)
+			continue;
+
+		/* Invalidate last valid PSCID */
+		riscv_iommu_cmd_inval_vma(&cmd);
+		riscv_iommu_cmd_inval_set_pscid(&cmd, FIELD_GET(RISCV_IOMMU_DC_TA_PSCID, ta));

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 896

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+		riscv_iommu_cmd_send(iommu, &cmd, 0);
+	}
+
+	/* Synchronize directory update */
+	riscv_iommu_cmd_iofence(&cmd);
+	riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_IOTINVAL_TIMEOUT);
+
+	/* Track domain to devices mapping. */
+	if (bond)
+		list_add_rcu(&bond->list, &domain->bonds);

This is in the wrong order, the invalidation on the pscid needs to start before the pscid is loaded into HW in the first place otherwise concurrent invalidations may miss HW updates.

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 900

Quoted 4 lines

drivers/iommu/riscv/iommu.c +4 -0

+
+	/* Remove tracking from previous domain, if needed. */
+	iommu_domain = iommu_get_domain_for_dev(dev);
+	if (iommu_domain && !!(iommu_domain->type & __IOMMU_DOMAIN_PAGING)) {

No need for !!, && is already booleanizing

Quoted 49 lines

drivers/iommu/riscv/iommu.c +48 -0

+		domain = iommu_domain_to_riscv(iommu_domain);
+		bond = NULL;
+		rcu_read_lock();
+		list_for_each_entry_rcu(b, &domain->bonds, list) {
+			if (b->dev == dev) {
+				bond = b;
+				break;
+			}
+		}
+		rcu_read_unlock();
+
+		if (bond) {
+			list_del_rcu(&bond->list);
+			kfree_rcu(bond, rcu);
+		}
+	}
+
+	return 0;
+}
 
+static inline size_t get_page_size(size_t size)
+{
+	if (size >= IOMMU_PAGE_SIZE_512G)
+		return IOMMU_PAGE_SIZE_512G;
+	if (size >= IOMMU_PAGE_SIZE_1G)
+		return IOMMU_PAGE_SIZE_1G;
+	if (size >= IOMMU_PAGE_SIZE_2M)
+		return IOMMU_PAGE_SIZE_2M;
+	return IOMMU_PAGE_SIZE_4K;
+}
+
+#define _io_pte_present(pte)	((pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE))
+#define _io_pte_leaf(pte)	((pte) & _PAGE_LEAF)
+#define _io_pte_none(pte)	((pte) == 0)
+#define _io_pte_entry(pn, prot)	((_PAGE_PFN_MASK & ((pn) << _PAGE_PFN_SHIFT)) | (prot))
+
+static void riscv_iommu_pte_free(struct riscv_iommu_domain *domain,
+				 unsigned long pte, struct list_head *freelist)
+{
+	unsigned long *ptr;
+	int i;
+
+	if (!_io_pte_present(pte) || _io_pte_leaf(pte))
+		return;
+
+	ptr = (unsigned long *)pfn_to_virt(__page_val_to_pfn(pte));
+
+	/* Recursively free all sub page table pages */
+	for (i = 0; i < PTRS_PER_PTE; i++) {

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 985

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+		pte = READ_ONCE(ptr[i]);
+		if (!_io_pte_none(pte) && cmpxchg_relaxed(ptr + i, pte, 0) == pte)
+			riscv_iommu_pte_free(domain, pte, freelist);
+	}
+
+	if (freelist)
+		list_add_tail(&virt_to_page(ptr)->lru, freelist);
+	else
+		free_page((unsigned long)ptr);
+}

Consider putting the page table handling in its own file?

Quoted 9 lines

drivers/iommu/riscv/iommu.c +9 -0

+static int riscv_iommu_attach_paging_domain(struct iommu_domain *iommu_domain,
+					    struct device *dev)
+{
+	struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+	struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
+	struct page *page;
+
+	if (!riscv_iommu_pt_supported(iommu, domain->pgd_mode))
+		return -ENODEV;

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 1199

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+
+	domain->numa_node = dev_to_node(iommu->dev);
+	domain->amo_enabled = !!(iommu->caps & RISCV_IOMMU_CAP_AMO_HWAD);
+
+	if (!domain->pgd_root) {
+		page = alloc_pages_node(domain->numa_node,
+					GFP_KERNEL_ACCOUNT | __GFP_ZERO, 0);
+		if (!page)
+			return -ENOMEM;
+		domain->pgd_root = (unsigned long)page_to_virt(page);

The pgd_root should be allocated by the alloc_paging function, not during attach. There is no locking here that will protect against concurrent attach and also map before attach should work.

You can pick up the numa affinity from the alloc paging dev pointer (note it may be null still in some cases)

Jason

Baolu Lu

2024-04-22 07:40 UTC #22 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

Quoted 17 lines · Jason Gunthorpe

drivers/iommu/riscv/iommu.c static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu) · line 810
Quoted 11 lines
drivers/iommu/riscv/iommu.c +11 -0
+		riscv_iommu_cmd_inval_vma(&cmd);
+		riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid);
+		if (len > 0 && len < RISCV_IOMMU_IOTLB_INVAL_LIMIT) {
+			for (iova = start; iova < end; iova += PAGE_SIZE) {
+				riscv_iommu_cmd_inval_set_addr(&cmd, iova);
+				riscv_iommu_cmd_send(iommu, &cmd, 0);
+			}
+		} else {
+			riscv_iommu_cmd_send(iommu, &cmd, 0);
+		}
+	}
This seems suboptimal, you probably want to copy the new design that Intel is doing where you allocate "bonds" that are already de-duplicated. Ie if I have 10 devices on the same iommu sharing the domain the above will invalidate the PSCID 10 times. It should only be done once.

Quoted 2 lines

ie add a "bond" for the (iommu,pscid) and refcount that based on how many devices are used. Then another "bond" for the ATS stuff eventually.

The latest version is under discussion here.

https://lore.kernel.org/linux-iommu/[email protected]/

Supposedly, you can make such optimization after the base code is landed in the mainline tree if the change is big.

Best regards,
baolu

Tomasz Jeznach

author

2024-04-24 23:30 UTC #23 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

In reply to Jason Gunthorpe

Quoted 11 lines · Jason Gunthorpe

Quoted 8 lines · Tomasz Jeznach

drivers/iommu/riscv/iommu.c +0 -0

index a4f74588cdc2..32ddc372432d 100644
@@ -46,6 +46,10 @@ MODULE_LICENSE("GPL");
 #define dev_to_iommu(dev) \
      container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu)

Quoted 8 lines

drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 52
Quoted 4 lines
drivers/iommu/riscv/iommu.c +4 -0
+/* IOMMU PSCID allocation namespace. */
+static DEFINE_IDA(riscv_iommu_pscids);
+#define RISCV_IOMMU_MAX_PSCID                BIT(20)
+
You may consider putting this IDA in the riscv_iommu_device() and move the pscid from the domain to the bond?

I've been considering containing IDA inside riscv_iommu_device at some point, but it made PCSID management more complicated. In the follow up patches it is desired for PSCID to be unique across all IOMMUs in the system (within guest's GSCID), as the protection domains might (and will) be shared between more than single IOMMU device.

Quoted 45 lines

Quoted 14 lines

drivers/iommu/riscv/iommu.c +7 -0

 /* Device resource-managed allocations */
 struct riscv_iommu_devres {
      unsigned long addr;

@@ -752,12 +756,77 @@ static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu)
      return 0;
 }
 
+struct riscv_iommu_bond {
+     struct list_head list;
+     struct rcu_head rcu;
+     struct device *dev;
+};
+
+/* This struct contains protection domain specific IOMMU driver data. */

drivers/iommu/riscv/iommu.c static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu) · line 775

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+struct riscv_iommu_domain {
+     struct iommu_domain domain;
+     struct list_head bonds;
+     int pscid;
+     int numa_node;
+     int amo_enabled:1;
+     unsigned int pgd_mode;
+     /* paging domain */
+     unsigned long pgd_root;
+};

Glad to see there is no riscv_iommu_device pointer in the domain!

drivers/iommu/riscv/iommu.c static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu) · line 799

Quoted 12 lines

drivers/iommu/riscv/iommu.c +12 -0

+static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
+                                 unsigned long start, unsigned long end)
+{
+     struct riscv_iommu_bond *bond;
+     struct riscv_iommu_device *iommu;
+     struct riscv_iommu_command cmd;
+     unsigned long len = end - start + 1;
+     unsigned long iova;
+
+     rcu_read_lock();
+     list_for_each_entry_rcu(bond, &domain->bonds, list) {
+             iommu = dev_to_iommu(bond->dev);

Quoted 3 lines

I suggest storing the iommu pointer itself in the bond instead of the device then add a synchronize_rcu() to the iommu unregister path.

Very good point. Thanks for pointing this out. Reworked to add locking around list modifications (and do not incorrectly rely on iommu group mutex locks).

Quoted 12 lines

drivers/iommu/riscv/iommu.c static int riscv_iommu_ddt_alloc(struct riscv_iommu_device *iommu) · line 810

Quoted 11 lines

drivers/iommu/riscv/iommu.c +11 -0

+             riscv_iommu_cmd_inval_vma(&cmd);
+             riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid);
+             if (len > 0 && len < RISCV_IOMMU_IOTLB_INVAL_LIMIT) {
+                     for (iova = start; iova < end; iova += PAGE_SIZE) {
+                             riscv_iommu_cmd_inval_set_addr(&cmd, iova);
+                             riscv_iommu_cmd_send(iommu, &cmd, 0);
+                     }
+             } else {
+                     riscv_iommu_cmd_send(iommu, &cmd, 0);
+             }
+     }

Quoted 9 lines

This seems suboptimal, you probably want to copy the new design that Intel is doing where you allocate "bonds" that are already de-duplicated. Ie if I have 10 devices on the same iommu sharing the domain the above will invalidate the PSCID 10 times. It should only be done once.
ie add a "bond" for the (iommu,pscid) and refcount that based on how many devices are used. Then another "bond" for the ATS stuff eventually.

Agree, not perfect to send duplicate invalidations.
This should improve with follow up patchsets introducing of SVA (reusing the same, extended bond structure) and update to send IOTLB range invalidations.

For this change I've decided to go with as simple as possible implementation and over-invalidate for domains with multiple devices attached. Hope this makes sense.

Quoted 15 lines

drivers/iommu/riscv/iommu.c +10 -0

+
+     list_for_each_entry_rcu(bond, &domain->bonds, list) {
+             iommu = dev_to_iommu(bond->dev);
+
+             riscv_iommu_cmd_iofence(&cmd);
+             riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_QUEUE_TIMEOUT);
+     }
+     rcu_read_unlock();
+}
+

@@ -787,12 +870,390 @@ static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
              xchg64(&dc->ta, ta);
              xchg64(&dc->tc, tc);

Quoted 6 lines

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 874
Quoted 3 lines
drivers/iommu/riscv/iommu.c +2 -1
-             /* Device context invalidation will be required. Ignoring for now. */
+             if (!(tc & RISCV_IOMMU_DC_TC_V))
+                     continue;
No negative caching in HW?

No. Disallowed by the spec.

Quoted 86 lines

Quoted 11 lines

drivers/iommu/riscv/iommu.c +11 -0

+             /* Invalidate device context cache */
+             riscv_iommu_cmd_iodir_inval_ddt(&cmd);
+             riscv_iommu_cmd_iodir_set_did(&cmd, fwspec->ids[i]);
+             riscv_iommu_cmd_send(iommu, &cmd, 0);
+
+             if (FIELD_GET(RISCV_IOMMU_PC_FSC_MODE, fsc) == RISCV_IOMMU_DC_FSC_MODE_BARE)
+                     continue;
+
+             /* Invalidate last valid PSCID */
+             riscv_iommu_cmd_inval_vma(&cmd);
+             riscv_iommu_cmd_inval_set_pscid(&cmd, FIELD_GET(RISCV_IOMMU_DC_TA_PSCID, ta));

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 896

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+             riscv_iommu_cmd_send(iommu, &cmd, 0);
+     }
+
+     /* Synchronize directory update */
+     riscv_iommu_cmd_iofence(&cmd);
+     riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_IOTINVAL_TIMEOUT);
+
+     /* Track domain to devices mapping. */
+     if (bond)
+             list_add_rcu(&bond->list, &domain->bonds);

This is in the wrong order, the invalidation on the pscid needs to start before the pscid is loaded into HW in the first place otherwise concurrent invalidations may miss HW updates.

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 900

Quoted 4 lines

drivers/iommu/riscv/iommu.c +4 -0

+
+     /* Remove tracking from previous domain, if needed. */
+     iommu_domain = iommu_get_domain_for_dev(dev);
+     if (iommu_domain && !!(iommu_domain->type & __IOMMU_DOMAIN_PAGING)) {

No need for !!, && is already booleanizing

Quoted 53 lines

drivers/iommu/riscv/iommu.c +52 -0

+             domain = iommu_domain_to_riscv(iommu_domain);
+             bond = NULL;
+             rcu_read_lock();
+             list_for_each_entry_rcu(b, &domain->bonds, list) {
+                     if (b->dev == dev) {
+                             bond = b;
+                             break;
+                     }
+             }
+             rcu_read_unlock();
+
+             if (bond) {
+                     list_del_rcu(&bond->list);
+                     kfree_rcu(bond, rcu);
+             }
+     }
+
+     return 0;
+}
 
+static inline size_t get_page_size(size_t size)
+{
+     if (size >= IOMMU_PAGE_SIZE_512G)
+             return IOMMU_PAGE_SIZE_512G;
+     if (size >= IOMMU_PAGE_SIZE_1G)
+             return IOMMU_PAGE_SIZE_1G;
+     if (size >= IOMMU_PAGE_SIZE_2M)
+             return IOMMU_PAGE_SIZE_2M;
+     return IOMMU_PAGE_SIZE_4K;
+}
+
+#define _io_pte_present(pte) ((pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE))
+#define _io_pte_leaf(pte)    ((pte) & _PAGE_LEAF)
+#define _io_pte_none(pte)    ((pte) == 0)
+#define _io_pte_entry(pn, prot)      ((_PAGE_PFN_MASK & ((pn) << _PAGE_PFN_SHIFT)) | (prot))
+
+static void riscv_iommu_pte_free(struct riscv_iommu_domain *domain,
+                              unsigned long pte, struct list_head *freelist)
+{
+     unsigned long *ptr;
+     int i;
+
+     if (!_io_pte_present(pte) || _io_pte_leaf(pte))
+             return;
+
+     ptr = (unsigned long *)pfn_to_virt(__page_val_to_pfn(pte));
+
+     /* Recursively free all sub page table pages */
+     for (i = 0; i < PTRS_PER_PTE; i++) {
+             pte = READ_ONCE(ptr[i]);
+             if (!_io_pte_none(pte) && cmpxchg_relaxed(ptr + i, pte, 0) == pte)
+                     riscv_iommu_pte_free(domain, pte, freelist);
+     }

Quoted 9 lines

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 985
Quoted 6 lines
drivers/iommu/riscv/iommu.c +6 -0
+
+     if (freelist)
+             list_add_tail(&virt_to_page(ptr)->lru, freelist);
+     else
+             free_page((unsigned long)ptr);
+}
Consider putting the page table handling in its own file?

It was in separate file at some point, but merged to iommu.c, as its simple enough with ~300 lines only. Probably not worth separating this out.

Quoted 24 lines

Quoted 9 lines

drivers/iommu/riscv/iommu.c +9 -0

+static int riscv_iommu_attach_paging_domain(struct iommu_domain *iommu_domain,
+                                         struct device *dev)
+{
+     struct riscv_iommu_device *iommu = dev_to_iommu(dev);
+     struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
+     struct page *page;
+
+     if (!riscv_iommu_pt_supported(iommu, domain->pgd_mode))
+             return -ENODEV;

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 1199

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+
+     domain->numa_node = dev_to_node(iommu->dev);
+     domain->amo_enabled = !!(iommu->caps & RISCV_IOMMU_CAP_AMO_HWAD);
+
+     if (!domain->pgd_root) {
+             page = alloc_pages_node(domain->numa_node,
+                                     GFP_KERNEL_ACCOUNT | __GFP_ZERO, 0);
+             if (!page)
+                     return -ENOMEM;
+             domain->pgd_root = (unsigned long)page_to_virt(page);

The pgd_root should be allocated by the alloc_paging function, not during attach. There is no locking here that will protect against concurrent attach and also map before attach should work.

Quoted 3 lines

You can pick up the numa affinity from the alloc paging dev pointer (note it may be null still in some cases)

Good point. Thanks. Will send update shortly with v3.

Quoted 1 line

Jason

Ack to all other comments, thank you!
Best,
- Tomasz

Jason Gunthorpe

2024-04-24 23:39 UTC #24 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

Quoted 13 lines · Tomasz Jeznach

Quoted 4 lines
Quoted 4 lines
drivers/iommu/riscv/iommu.c +0 -0
@@ -46,6 +46,10 @@ MODULE_LICENSE("GPL");
 #define dev_to_iommu(dev) \
      container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu)
 
Quoted 8 lines
drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 52
Quoted 4 lines
drivers/iommu/riscv/iommu.c +4 -0
+/* IOMMU PSCID allocation namespace. */
+static DEFINE_IDA(riscv_iommu_pscids);
+#define RISCV_IOMMU_MAX_PSCID                BIT(20)
+
You may consider putting this IDA in the riscv_iommu_device() and move the pscid from the domain to the bond?

Quoted 5 lines

I've been considering containing IDA inside riscv_iommu_device at some point, but it made PCSID management more complicated. In the follow up patches it is desired for PSCID to be unique across all IOMMUs in the system (within guest's GSCID), as the protection domains might (and will) be shared between more than single IOMMU device.

The PCSID isn't scoped under the GSCID? That doesn't sound very good, it means VM's can't direct issue invalidation with their local view of the PCSID space?

Quoted 10 lines

Quoted 9 lines
This seems suboptimal, you probably want to copy the new design that Intel is doing where you allocate "bonds" that are already de-duplicated. Ie if I have 10 devices on the same iommu sharing the domain the above will invalidate the PSCID 10 times. It should only be done once.
ie add a "bond" for the (iommu,pscid) and refcount that based on how many devices are used. Then another "bond" for the ATS stuff eventually.

Quoted 8 lines

Agree, not perfect to send duplicate invalidations.
This should improve with follow up patchsets introducing of SVA (reusing the same, extended bond structure) and update to send IOTLB range invalidations.
For this change I've decided to go with as simple as possible implementation and over-invalidate for domains with multiple devices attached. Hope this makes sense.

It is fine as long as you do fix it..

Jason

Tomasz Jeznach

author

2024-04-24 23:54 UTC #25 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

Quoted 21 lines · Jason Gunthorpe

Quoted 13 lines · Tomasz Jeznach
Quoted 4 lines
Quoted 4 lines
drivers/iommu/riscv/iommu.c +0 -0
@@ -46,6 +46,10 @@ MODULE_LICENSE("GPL");
 #define dev_to_iommu(dev) \
      container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu)
 
Quoted 8 lines
drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 52
Quoted 4 lines
drivers/iommu/riscv/iommu.c +4 -0
+/* IOMMU PSCID allocation namespace. */
+static DEFINE_IDA(riscv_iommu_pscids);
+#define RISCV_IOMMU_MAX_PSCID                BIT(20)
+
You may consider putting this IDA in the riscv_iommu_device() and move the pscid from the domain to the bond?
Quoted 5 lines
I've been considering containing IDA inside riscv_iommu_device at some point, but it made PCSID management more complicated. In the follow up patches it is desired for PSCID to be unique across all IOMMUs in the system (within guest's GSCID), as the protection domains might (and will) be shared between more than single IOMMU device.

Quoted 4 lines

The PCSID isn't scoped under the GSCID? That doesn't sound very good, it means VM's can't direct issue invalidation with their local view of the PCSID space?

To clarify: PSCID namespace is per GSCID.
However there might be more than one IOMMU in a single system sharing the same GSCID, and with e.g. SVA domains attached to more than one IOMMU. It was simpler to manage PCSID globally.

PSCID management for the VM assigned GSCID will be the VM's responsibility.

Quoted 15 lines

Quoted 15 lines
Quoted 9 lines
This seems suboptimal, you probably want to copy the new design that Intel is doing where you allocate "bonds" that are already de-duplicated. Ie if I have 10 devices on the same iommu sharing the domain the above will invalidate the PSCID 10 times. It should only be done once.
ie add a "bond" for the (iommu,pscid) and refcount that based on how many devices are used. Then another "bond" for the ATS stuff eventually.
Agree, not perfect to send duplicate invalidations.
This should improve with follow up patchsets introducing of SVA (reusing the same, extended bond structure) and update to send IOTLB range invalidations.

Quoted 6 lines

Quoted 3 lines
For this change I've decided to go with as simple as possible implementation and over-invalidate for domains with multiple devices attached. Hope this makes sense.
It is fine as long as you do fix it..

SG. I'll have a second look if it can be fixed sooner.

Quoted 1 line

Jason

Best,
- Tomasz

Jason Gunthorpe

2024-04-25 00:48 UTC #26 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

Quoted 22 lines · Tomasz Jeznach

Quoted 21 lines · Jason Gunthorpe
Quoted 18 lines · Tomasz Jeznach
Quoted 4 lines
Quoted 4 lines
drivers/iommu/riscv/iommu.c +0 -0
@@ -46,6 +46,10 @@ MODULE_LICENSE("GPL");
 #define dev_to_iommu(dev) \
      container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu)
 
Quoted 8 lines
drivers/iommu/riscv/iommu.c MODULE_LICENSE("GPL"); · line 52
Quoted 4 lines
drivers/iommu/riscv/iommu.c +4 -0
+/* IOMMU PSCID allocation namespace. */
+static DEFINE_IDA(riscv_iommu_pscids);
+#define RISCV_IOMMU_MAX_PSCID                BIT(20)
+
You may consider putting this IDA in the riscv_iommu_device() and move the pscid from the domain to the bond?
I've been considering containing IDA inside riscv_iommu_device at some point, but it made PCSID management more complicated. In the follow up patches it is desired for PSCID to be unique across all IOMMUs in the system (within guest's GSCID), as the protection domains might (and will) be shared between more than single IOMMU device.

Quoted 8 lines

Quoted 4 lines
The PCSID isn't scoped under the GSCID? That doesn't sound very good, it means VM's can't direct issue invalidation with their local view of the PCSID space?
To clarify: PSCID namespace is per GSCID.
However there might be more than one IOMMU in a single system sharing the same GSCID

I assume this is because GSCID ends up shared with kvm?

Quoted 2 lines

and with e.g. SVA domains attached to more than one IOMMU. It was simpler to manage PCSID globally.

If the PSCID is moved into the invalidation list like Intel structured it then it doesn't matter for SVA, or really anything.

AFAIK the only reason to do otherwise is if you have a reason to share the ID with the CPU/MM and the IOMMU probably to coordinate invalidations. But if you do this then you really just always want to use the MM's global ID space in the first place...

So I'm not sure :)

Jason

Baolu Lu

2024-04-22 05:22 UTC #27 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

In reply to Tomasz Jeznach

Quoted 21 lines · Tomasz Jeznach

Introduce first-stage address translation support.
Page table configured by the IOMMU driver will use the same format as the CPU’s MMU, and will fallback to identity translation if the page table format configured for the MMU is not supported by the IOMMU hardware.
This change introduces IOTINVAL.VMA command, required to invalidate any cached IOATC entries after mapping is updated and/or removed from the paging domain. Invalidations for the non-leaf page entries will be added to the driver code in separate patch series, following spec update to clarify non-leaf cache invalidation command. With this patch, allowing only 4K mappings and keeping non-leaf page entries in memory this should be a reasonable simplification.
Signed-off-by Tomasz Jeznach <[email protected]>
---
  drivers/iommu/riscv/Kconfig |   1 +
  drivers/iommu/riscv/iommu.c | 467 +++++++++++++++++++++++++++++++++++-
  2 files changed, 466 insertions(+), 2 deletions(-)

[...]

Quoted 56 lines

drivers/iommu/riscv/iommu.c +35 -1

+
  static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
  				     struct device *dev,
  				     struct iommu_domain *iommu_domain)
  {
  	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct riscv_iommu_domain *domain;
  	struct riscv_iommu_dc *dc;
+	struct riscv_iommu_bond *bond = NULL, *b;
+	struct riscv_iommu_command cmd;
  	u64 fsc, ta, tc;
  	int i;

@@ -769,6 +838,20 @@ static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
  		ta = 0;
  		tc = RISCV_IOMMU_DC_TC_V;
  		fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE);
+	} else if (iommu_domain->type & __IOMMU_DOMAIN_PAGING) {
+		domain = iommu_domain_to_riscv(iommu_domain);
+
+		ta = FIELD_PREP(RISCV_IOMMU_PC_TA_PSCID, domain->pscid);
+		tc = RISCV_IOMMU_DC_TC_V;
+		if (domain->amo_enabled)
+			tc |= RISCV_IOMMU_DC_TC_SADE;
+		fsc = FIELD_PREP(RISCV_IOMMU_PC_FSC_MODE, domain->pgd_mode) |
+		      FIELD_PREP(RISCV_IOMMU_PC_FSC_PPN, virt_to_pfn(domain->pgd_root));
+
+		bond = kzalloc(sizeof(*bond), GFP_KERNEL);
+		if (!bond)
+			return -ENOMEM;
+		bond->dev = dev;
  	} else {
  		/* This should never happen. */
  		return -ENODEV;

@@ -787,12 +870,390 @@ static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu,
  		xchg64(&dc->ta, ta);
  		xchg64(&dc->tc, tc);
 
-		/* Device context invalidation will be required. Ignoring for now. */
+		if (!(tc & RISCV_IOMMU_DC_TC_V))
+			continue;
+
+		/* Invalidate device context cache */
+		riscv_iommu_cmd_iodir_inval_ddt(&cmd);
+		riscv_iommu_cmd_iodir_set_did(&cmd, fwspec->ids[i]);
+		riscv_iommu_cmd_send(iommu, &cmd, 0);
+
+		if (FIELD_GET(RISCV_IOMMU_PC_FSC_MODE, fsc) == RISCV_IOMMU_DC_FSC_MODE_BARE)
+			continue;
+
+		/* Invalidate last valid PSCID */
+		riscv_iommu_cmd_inval_vma(&cmd);
+		riscv_iommu_cmd_inval_set_pscid(&cmd, FIELD_GET(RISCV_IOMMU_DC_TA_PSCID, ta));
+		riscv_iommu_cmd_send(iommu, &cmd, 0);
+	}
+

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 899

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+	/* Synchronize directory update */
+	riscv_iommu_cmd_iofence(&cmd);
+	riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_IOTINVAL_TIMEOUT);
+
+	/* Track domain to devices mapping. */
+	if (bond)
+		list_add_rcu(&bond->list, &domain->bonds);
+
+	/* Remove tracking from previous domain, if needed. */
+	iommu_domain = iommu_get_domain_for_dev(dev);

Calling iommu_get_domain_for_dev() in the domain attaching path is very fragile because it heavily depends on the order of calling the attach callback and setting the domain pointer in the core.

Perhaps the driver can use dev_iommu_priv_set/get() to keep the active domain in the per-device private data?

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+	if (iommu_domain && !!(iommu_domain->type & __IOMMU_DOMAIN_PAGING)) {
+		domain = iommu_domain_to_riscv(iommu_domain);
+		bond = NULL;
+		rcu_read_lock();
+		list_for_each_entry_rcu(b, &domain->bonds, list) {
+			if (b->dev == dev) {
+				bond = b;
+				break;
+			}
+		}

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 919

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+		rcu_read_unlock();
+
+		if (bond) {
+			list_del_rcu(&bond->list);
+			kfree_rcu(bond, rcu);
+		}
+	}
+
+	return 0;
+}

Best regards,
baolu

Jason Gunthorpe

2024-04-22 19:31 UTC #28 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

Quoted 10 lines · Baolu Lu

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 899
Quoted 6 lines
drivers/iommu/riscv/iommu.c +6 -0
+	/* Track domain to devices mapping. */
+	if (bond)
+		list_add_rcu(&bond->list, &domain->bonds);
+
+	/* Remove tracking from previous domain, if needed. */
+	iommu_domain = iommu_get_domain_for_dev(dev);
Calling iommu_get_domain_for_dev() in the domain attaching path is very fragile because it heavily depends on the order of calling the attach callback and setting the domain pointer in the core.

We have a couple places doing this already, the core code accomodates it well enough for deleting from a list.. So I think it is OK to keep doing.

Quoted 14 lines

Perhaps the driver can use dev_iommu_priv_set/get() to keep the active domain in the per-device private data?
drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 910
Quoted 11 lines
drivers/iommu/riscv/iommu.c +11 -0
+	if (iommu_domain && !!(iommu_domain->type & __IOMMU_DOMAIN_PAGING)) {
+		domain = iommu_domain_to_riscv(iommu_domain);
+		bond = NULL;
+		rcu_read_lock();
+		list_for_each_entry_rcu(b, &domain->bonds, list) {
+			if (b->dev == dev) {
+				bond = b;
+				break;
+			}
+		}
+		rcu_read_unlock();

But now that I look again, this is not safe, you have to hold some kind of per-domain lock to mutate the list. rcu_*read*_lock() cannot be used for write.

Jason

Andrew Jones

2024-04-23 17:00 UTC #29 lore raw

Re: [PATCH v2 7/7] iommu/riscv: Paging domain support

In reply to Tomasz Jeznach

On Thu, Apr 18, 2024 at 09:32:25AM -0700, Tomasz Jeznach wrote:
...

Quoted 7 lines

drivers/iommu/riscv/iommu.c +7 -0

+static size_t riscv_iommu_unmap_pages(struct iommu_domain *iommu_domain,
+				      unsigned long iova, size_t pgsize, size_t pgcount,
+				      struct iommu_iotlb_gather *gather)
+{
+	struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
+	size_t size = pgcount << __ffs(pgsize);
+	unsigned long *ptr, old;

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 1118

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+	size_t unmapped = 0;
+	size_t pte_size;
+
+	while (unmapped < size) {
+		ptr = riscv_iommu_pte_fetch(domain, iova, &pte_size);
+		if (!ptr)
+			return unmapped;
+
+		/* partial unmap is not allowed, fail. */
+		if (iova & ~(pte_size - 1))

^ Shouldn't this ~ be removed?

Quoted 6 lines

drivers/iommu/riscv/iommu.c +6 -0

+			return unmapped;
+
+		old = READ_ONCE(*ptr);
+		if (cmpxchg_relaxed(ptr, old, 0) != old)
+			continue;
+

drivers/iommu/riscv/iommu.c static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, · line 1134

Quoted 10 lines

drivers/iommu/riscv/iommu.c +10 -0

+		iommu_iotlb_gather_add_page(&domain->domain, gather, iova,
+					    pte_size);
+
+		iova += pte_size;
+		unmapped += pte_size;
+	}
+
+	return unmapped;
+}
+

Thanks,
drew