QEMU RISC-V IOMMU Support
Subsystems (2)
cover QEMU RISC-V IOMMU Support
The RISC-V IOMMU specification is now ratified as-per the RISC-V international process [1]. The latest frozen specifcation can be found at:
https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
This series introduces a RISC-V IOMMU device emulation implementation with two stage address translation logic, device and process translation context mapping and queue interfaces, along with riscv/virt machine bindings (patch 5) and memory attributes extensions for PASID support (patch 3,4).
This series is based on incremental patches created during RISC-V International IOMMU Task Group discussions and specification development process, with original series available in the the maintainer's repository branch [2].
These patches can also be found in the riscv_iommu_v1 branch at:
https://github.com/tjeznach/qemu/tree/riscv_iommu_v1
To test this series, use Linux v6.5-rc2 with RISC-V IOMMU implementation available in the riscv_iommu_v1 branch at:
https://github.com/tjeznach/linux/tree/riscv_iommu_v1
References:
[1] - https://wiki.riscv.org/display/HOME/Specification+Status
[2] - https://github.com/tjeznach/qemu/tree/tjeznach/riscv-iommu-20230719
Tomasz Jeznach (5):
hw/riscv: Introduction of RISC-V IOMMU device
MAINTAINERS: Add RISC-V IOMMU maintainers
exec/memtxattr: add process identifier to the transaction attributes hw/riscv: IOMMU: use process identifier from transaction attributes. hw/riscv: virt: support for RISC-V IOMMU platform device.
MAINTAINERS | 6 + hw/riscv/Kconfig | 4 + hw/riscv/meson.build | 1 + hw/riscv/riscv-iommu-bits.h | 749 +++++++++++ hw/riscv/riscv-iommu-pci.c | 181 +++ hw/riscv/riscv-iommu-sys.c | 123 ++ hw/riscv/riscv-iommu.c | 2546 +++++++++++++++++++++++++++++++++++ hw/riscv/riscv-iommu.h | 152 +++ hw/riscv/trace-events | 14 + hw/riscv/trace.h | 2 + hw/riscv/virt.c | 100 +- include/exec/memattrs.h | 6 + include/hw/riscv/iommu.h | 40 + include/hw/riscv/virt.h | 3 + meson.build | 1 + 15 files changed, 3927 insertions(+), 1 deletion(-) create mode 100644 hw/riscv/riscv-iommu-bits.h create mode 100644 hw/riscv/riscv-iommu-pci.c create mode 100644 hw/riscv/riscv-iommu-sys.c create mode 100644 hw/riscv/riscv-iommu.c create mode 100644 hw/riscv/riscv-iommu.h create mode 100644 hw/riscv/trace-events create mode 100644 hw/riscv/trace.h create mode 100644 include/hw/riscv/iommu.h
Signature · 1 line
2.34.1
PATCH 1/5 hw/riscv: Introduction of RISC-V IOMMU device
The RISC-V IOMMU specification is now ratified as-per the RISC-V international process [1]. The latest frozen specifcation can be found at:
https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
The patch add device emulation for RISC-V IOMMU which supports device and process context lookups, command and fault queue interfaces, two stage address translation logic with Sv32, Sv39, Sv48, Sv57 addressing modes, address translation cache, MSI remapping with FLAT/MRIF modes, initial ATS and PRI interfaces, debug capabilities, hardware performance counters. Platform and PCIe device instantiation is supported, with wire-signaled and message-signaled interrupt capabilities.
Hardware interface definition file is shared with Linux kernel driver implementation, available in the maintainer's branch riscv_iommu_v1 at https://github.com/tjeznach/linux.
Co-developed-by Sebastien Boeuf <[email protected]>
Signed-off-by Sebastien Boeuf <[email protected]>
Signed-off-by Tomasz Jeznach <[email protected]>
--- hw/riscv/Kconfig | 3 + hw/riscv/meson.build | 1 + hw/riscv/riscv-iommu-bits.h | 749 +++++++++++ hw/riscv/riscv-iommu-pci.c | 181 +++ hw/riscv/riscv-iommu-sys.c | 123 ++ hw/riscv/riscv-iommu.c | 2539 +++++++++++++++++++++++++++++++++++ hw/riscv/riscv-iommu.h | 152 +++ hw/riscv/trace-events | 14 + hw/riscv/trace.h | 2 + include/hw/riscv/iommu.h | 40 + meson.build | 1 + 11 files changed, 3805 insertions(+) create mode 100644 hw/riscv/riscv-iommu-bits.h create mode 100644 hw/riscv/riscv-iommu-pci.c create mode 100644 hw/riscv/riscv-iommu-sys.c create mode 100644 hw/riscv/riscv-iommu.c create mode 100644 hw/riscv/riscv-iommu.h create mode 100644 hw/riscv/trace-events create mode 100644 hw/riscv/trace.h create mode 100644 include/hw/riscv/iommu.h
hw/riscv/Kconfig +3 -0
@@ -1,3 +1,6 @@ +config RISCV_IOMMU + bool + config RISCV_NUMA bool
hw/riscv/meson.build +1 -0
@@ -10,5 +10,6 @@ riscv_ss.add(when: 'CONFIG_SIFIVE_U', if_true: files('sifive_u.c')) riscv_ss.add(when: 'CONFIG_SPIKE', if_true: files('spike.c')) riscv_ss.add(when: 'CONFIG_MICROCHIP_PFSOC', if_true: files('microchip_pfsoc.c')) riscv_ss.add(when: 'CONFIG_ACPI', if_true: files('virt-acpi-build.c')) +riscv_ss.add(when: 'CONFIG_RISCV_IOMMU', if_true: files('riscv-iommu.c', 'riscv-iommu-pci.c', 'riscv-iommu-sys.c')) hw_arch += {'riscv': riscv_ss}
hw/riscv/riscv-iommu-bits.h new file +749 -0
749 lines not on this page · raw message
hw/riscv/riscv-iommu-pci.c new file +181 -0
181 lines not on this page · raw message
hw/riscv/riscv-iommu-sys.c new file +123 -0
123 lines not on this page · raw message
hw/riscv/riscv-iommu.c new file +2539 -0
1,935 lines not on this page · raw message
hw/riscv/riscv-iommu.h new file +152 -0
152 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.34.1
Part of the diff is not shown here. View the raw message.
Re: [PATCH 1/5] hw/riscv: Introduction of RISC-V IOMMU device
Quoted 4 lines · Tomasz Jeznach
The RISC-V IOMMU specification is now ratified as-per the RISC-V international process [1]. The latest frozen specifcation can be found at:
https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
Exciting!
Quoted 14 lines
The patch add device emulation for RISC-V IOMMU which supports device and process context lookups, command and fault queue interfaces, two stage address translation logic with Sv32, Sv39, Sv48, Sv57 addressing modes, address translation cache, MSI remapping with FLAT/MRIF modes, initial ATS and PRI interfaces, debug capabilities, hardware performance counters. Platform and PCIe device instantiation is supported, with wire-signaled and message-signaled interrupt capabilities.
Hardware interface definition file is shared with Linux kernel driver implementation, available in the maintainer's branch
riscv_iommu_v1at https://github.com/tjeznach/linux.Co-developed-by Sebastien Boeuf <[email protected]>Signed-off-by Sebastien Boeuf <[email protected]>Signed-off-by Tomasz Jeznach <[email protected]>
Quoted 12 lines
--- hw/riscv/Kconfig | 3 + hw/riscv/meson.build | 1 + hw/riscv/riscv-iommu-bits.h | 749 +++++++++++ hw/riscv/riscv-iommu-pci.c | 181 +++ hw/riscv/riscv-iommu-sys.c | 123 ++ hw/riscv/riscv-iommu.c | 2539 +++++++++++++++++++++++++++++++++++ hw/riscv/riscv-iommu.h | 152 +++ hw/riscv/trace-events | 14 + hw/riscv/trace.h | 2 + include/hw/riscv/iommu.h | 40 + meson.build | 1 +
This is a really long patch!
I think this should at least be split up to rougly each file (as long as it compiles). For example the header files could be added in a patch each. Which would reduce some of the review burden.
Quoted 50 lines
11 files changed, 3805 insertions(+) create mode 100644 hw/riscv/riscv-iommu-bits.h create mode 100644 hw/riscv/riscv-iommu-pci.c create mode 100644 hw/riscv/riscv-iommu-sys.c create mode 100644 hw/riscv/riscv-iommu.c create mode 100644 hw/riscv/riscv-iommu.h create mode 100644 hw/riscv/trace-events create mode 100644 hw/riscv/trace.h create mode 100644 include/hw/riscv/iommu.hhw/riscv/Kconfig +3
-0@@ -1,3 +1,6 @@ +config RISCV_IOMMU + bool + config RISCV_NUMA boolhw/riscv/meson.build +1
-0@@ -10,5 +10,6 @@ riscv_ss.add(when: 'CONFIG_SIFIVE_U', if_true: files('sifive_u.c')) riscv_ss.add(when: 'CONFIG_SPIKE', if_true: files('spike.c')) riscv_ss.add(when: 'CONFIG_MICROCHIP_PFSOC', if_true: files('microchip_pfsoc.c')) riscv_ss.add(when: 'CONFIG_ACPI', if_true: files('virt-acpi-build.c')) +riscv_ss.add(when: 'CONFIG_RISCV_IOMMU', if_true: files('riscv-iommu.c', 'riscv-iommu-pci.c', 'riscv-iommu-sys.c')) hw_arch += {'riscv': riscv_ss}hw/riscv/riscv-iommu-bits.h new file +12
-0@@ -0,0 +1,749 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright © 2022-2023 Rivos Inc. +* Copyright © 2023 FORTH-ICS/CARV +* Copyright © 2023 RISC-V IOMMU Task Group +* +* RISC-V Ziommu - Register Layout and Data Structures. +* +* Based on the 'RISC-V IOMMU Architecture Specification', Version 1.0 +* Published at https://github.com/riscv-non-isa/riscv-iommu +* +*/
hw/riscv/riscv-iommu-bits.h line 22
Quoted 10 lines
hw/riscv/riscv-iommu-bits.h new file +10
-0+ +#ifndef HW_RISCV_IOMMU_BITS_H +#define HW_RISCV_IOMMU_BITS_H + +/* +* This file is based on Linux RISC-V IOMMU file +* located at 'drivers/iommu/riscv/iommu-bits.h' +*/ + +#include "qemu/osdep.h"
This shouldn't be included in header files
Quoted 764 lines
hw/riscv/riscv-iommu-bits.h new file +727
-0+ +#define RISCV_IOMMU_SPEC_DOT_VER 0x010 + +#ifndef GENMASK_ULL +#define GENMASK_ULL (h, l) (((~0ULL) >> (63 - (h) + (l))) << (l)) +#endif + +/* +* Chapter 5: Memory Mapped register interface +*/ + +/* Common field positions */ +#define RISCV_IOMMU_PPN_FIELD GENMASK_ULL(53, 10) +#define RISCV_IOMMU_QUEUE_LOGSZ_FIELD GENMASK_ULL(4, 0) +#define RISCV_IOMMU_QUEUE_INDEX_FIELD GENMASK_ULL(31, 0) +#define RISCV_IOMMU_QUEUE_ENABLE BIT(0) +#define RISCV_IOMMU_QUEUE_INTR_ENABLE BIT(1) +#define RISCV_IOMMU_QUEUE_MEM_FAULT BIT(8) +#define RISCV_IOMMU_QUEUE_OVERFLOW BIT(9) +#define RISCV_IOMMU_QUEUE_ACTIVE BIT(16) +#define RISCV_IOMMU_QUEUE_BUSY BIT(17) +#define RISCV_IOMMU_ATP_PPN_FIELD GENMASK_ULL(43, 0) +#define RISCV_IOMMU_ATP_MODE_FIELD GENMASK_ULL(63, 60) + +/* 5.3 IOMMU Capabilities (64bits) */ +#define RISCV_IOMMU_REG_CAP 0x0000 +#define RISCV_IOMMU_CAP_VERSION GENMASK_ULL(7, 0) +#define RISCV_IOMMU_CAP_S_SV32 BIT_ULL(8) +#define RISCV_IOMMU_CAP_S_SV39 BIT_ULL(9) +#define RISCV_IOMMU_CAP_S_SV48 BIT_ULL(10) +#define RISCV_IOMMU_CAP_S_SV57 BIT_ULL(11) +#define RISCV_IOMMU_CAP_SVPBMT BIT_ULL(15) +#define RISCV_IOMMU_CAP_G_SV32 BIT_ULL(16) +#define RISCV_IOMMU_CAP_G_SV39 BIT_ULL(17) +#define RISCV_IOMMU_CAP_G_SV48 BIT_ULL(18) +#define RISCV_IOMMU_CAP_G_SV57 BIT_ULL(19) +#define RISCV_IOMMU_CAP_MSI_FLAT BIT_ULL(22) +#define RISCV_IOMMU_CAP_MSI_MRIF BIT_ULL(23) +#define RISCV_IOMMU_CAP_AMO BIT_ULL(24) +#define RISCV_IOMMU_CAP_ATS BIT_ULL(25) +#define RISCV_IOMMU_CAP_T2GPA BIT_ULL(26) +#define RISCV_IOMMU_CAP_END BIT_ULL(27) +#define RISCV_IOMMU_CAP_IGS GENMASK_ULL(29, 28) +#define RISCV_IOMMU_CAP_HPM BIT_ULL(30) +#define RISCV_IOMMU_CAP_DBG BIT_ULL(31) +#define RISCV_IOMMU_CAP_PAS GENMASK_ULL(37, 32) +#define RISCV_IOMMU_CAP_PD8 BIT_ULL(38) +#define RISCV_IOMMU_CAP_PD17 BIT_ULL(39) +#define RISCV_IOMMU_CAP_PD20 BIT_ULL(40) + +#define RISCV_IOMMU_CAP_VERSION_VER_MASK 0xF0 +#define RISCV_IOMMU_CAP_VERSION_REV_MASK 0x0F + +/** +* enum riscv_iommu_igs_settings - Interrupt Generation Support Settings +* @RISCV_IOMMU_CAP_IGS_MSI: I/O MMU supports only MSI generation +* @RISCV_IOMMU_CAP_IGS_WSI: I/O MMU supports only Wired-Signaled interrupt +* @RISCV_IOMMU_CAP_IGS_BOTH: I/O MMU supports both MSI and WSI generation +* @RISCV_IOMMU_CAP_IGS_RSRV: Reserved for standard use +*/ +enum riscv_iommu_igs_settings { +RISCV_IOMMU_CAP_IGS_MSI =0 , +RISCV_IOMMU_CAP_IGS_WSI =1 , +RISCV_IOMMU_CAP_IGS_BOTH =2 , +RISCV_IOMMU_CAP_IGS_RSRV =3 +}; + + +/* 5.4 Features control register (32bits) */ +#define RISCV_IOMMU_REG_FCTL 0x0008 +#define RISCV_IOMMU_FCTL_BE BIT(0) +#define RISCV_IOMMU_FCTL_WSI BIT(1) +#define RISCV_IOMMU_FCTL_GXL BIT(2) + + +/* 5.5 Device-directory-table pointer (64bits) */ +#define RISCV_IOMMU_REG_DDTP 0x0010 +#define RISCV_IOMMU_DDTP_MODE GENMASK_ULL(3, 0) +#define RISCV_IOMMU_DDTP_BUSY BIT_ULL(4) +#define RISCV_IOMMU_DDTP_PPN RISCV_IOMMU_PPN_FIELD + +/** +* enum riscv_iommu_ddtp_modes - I/O MMU translation modes +* @RISCV_IOMMU_DDTP_MODE_OFF: No inbound transactions allowed +* @RISCV_IOMMU_DDTP_MODE_BARE: Pass-through mode +* @RISCV_IOMMU_DDTP_MODE_1LVL: One-level DDT +* @RISCV_IOMMU_DDTP_MODE_2LVL: Two-level DDT +* @RISCV_IOMMU_DDTP_MODE_3LVL: Three-level DDT +*/ +enum riscv_iommu_ddtp_modes { +RISCV_IOMMU_DDTP_MODE_OFF =0 , +RISCV_IOMMU_DDTP_MODE_BARE =1 , +RISCV_IOMMU_DDTP_MODE_1LVL =2 , +RISCV_IOMMU_DDTP_MODE_2LVL =3 , +RISCV_IOMMU_DDTP_MODE_3LVL =4 , +RISCV_IOMMU_DDTP_MODE_MAX =4 +}; + + +/* 5.6 Command Queue Base (64bits) */ +#define RISCV_IOMMU_REG_CQB 0x0018 +#define RISCV_IOMMU_CQB_LOG2SZ RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_CQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.7 Command Queue head (32bits) */ +#define RISCV_IOMMU_REG_CQH 0x0020 +#define RISCV_IOMMU_CQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.8 Command Queue tail (32bits) */ +#define RISCV_IOMMU_REG_CQT 0x0024 +#define RISCV_IOMMU_CQT_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + + +/* 5.9 Fault Queue Base (64bits) */ +#define RISCV_IOMMU_REG_FQB 0x0028 +#define RISCV_IOMMU_FQB_LOG2SZ RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_FQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.10 Fault Queue Head (32bits) */ +#define RISCV_IOMMU_REG_FQH 0x0030 +#define RISCV_IOMMU_FQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.11 Fault Queue tail (32bits) */ +#define RISCV_IOMMU_REG_FQT 0x0034 +#define RISCV_IOMMU_FQT_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + + +/* 5.12 Page Request Queue base (64bits) */ +#define RISCV_IOMMU_REG_PQB 0x0038 +#define RISCV_IOMMU_PQB_LOG2SZ RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_PQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.13 Page Request Queue head (32bits) */ +#define RISCV_IOMMU_REG_PQH 0x0040 +#define RISCV_IOMMU_PQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.14 Page Request Queue tail (32bits) */ +#define RISCV_IOMMU_REG_PQT 0x0044 +#define RISCV_IOMMU_PQT_INDEX_MASK RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.15 Command Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_CQCSR 0x0048 +#define RISCV_IOMMU_CQCSR_CQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_CQCSR_CIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_CQCSR_CQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_CQCSR_CMD_TO BIT(9) +#define RISCV_IOMMU_CQCSR_CMD_ILL BIT(10) +#define RISCV_IOMMU_CQCSR_FENCE_W_IP BIT(11) +#define RISCV_IOMMU_CQCSR_CQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_CQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + + +/* 5.16 Fault Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_FQCSR 0x004C +#define RISCV_IOMMU_FQCSR_FQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_FQCSR_FIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_FQCSR_FQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_FQCSR_FQOF RISCV_IOMMU_QUEUE_OVERFLOW +#define RISCV_IOMMU_FQCSR_FQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_FQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + + +/* 5.17 Page Request Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_PQCSR 0x0050 +#define RISCV_IOMMU_PQCSR_PQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_PQCSR_PIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_PQCSR_PQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_PQCSR_PQOF RISCV_IOMMU_QUEUE_OVERFLOW +#define RISCV_IOMMU_PQCSR_PQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_PQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + + +/* 5.18 Interrupt Pending Status (32bits) */ +#define RISCV_IOMMU_REG_IPSR 0x0054 + +#define RISCV_IOMMU_INTR_CQ 0 +#define RISCV_IOMMU_INTR_FQ 1 +#define RISCV_IOMMU_INTR_PM 2 +#define RISCV_IOMMU_INTR_PQ 3 +#define RISCV_IOMMU_INTR_COUNT 4 + +#define RISCV_IOMMU_IPSR_CIP BIT(RISCV_IOMMU_INTR_CQ) +#define RISCV_IOMMU_IPSR_FIP BIT(RISCV_IOMMU_INTR_FQ) +#define RISCV_IOMMU_IPSR_PMIP BIT(RISCV_IOMMU_INTR_PM) +#define RISCV_IOMMU_IPSR_PIP BIT(RISCV_IOMMU_INTR_PQ) + +#define RISCV_IOMMU_IOCOUNT_NUM 31 + +/* 5.19 Performance monitoring counter overflow status (32bits) */ +#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058 +#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0) +#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK(31, 1) + +/* 5.20 Performance monitoring counter inhibits (32bits) */ +#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C +#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0) +#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 1) + +/* 5.21 Performance monitoring cycles counter (64bits) */ +#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060 +#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0) +#define RISCV_IOMMU_IOHPMCYCLES_OVF BIT_ULL(63) + +/* 5.22 Performance monitoring event counters (31 * 64bits) */ +#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068 +#define RISCV_IOMMU_REG_IOHPMCTR (_n) \ + (RISCV_IOMMU_REG_IOHPMCTR_BASE + (_n * 0x8)) + +/* 5.23 Performance monitoring event selectors (31 * 64bits) */ +#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160 +#define RISCV_IOMMU_REG_IOHPMEVT (_n) \ + (RISCV_IOMMU_REG_IOHPMEVT_BASE + (_n * 0x8)) +#define RISCV_IOMMU_IOHPMEVT_EVENT_ID GENMASK_ULL(14, 0) +#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15) +#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16) +#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36) +#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60) +#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61) +#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62) +#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63) + +/** +* enum RISCV_IOMMU_HPMEVENT_id - Performance-monitoring event identifier +* +* @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count +* @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests +* @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests +* @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests +* @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses +* @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks +* @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks +* @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: S/VS-Stage page table walks +* @RISCV_IOMMU_HPMEVENT_G_WALKS: G-Stage page table walks +* @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs +*/ +enum RISCV_IOMMU_HPMEVENT_id { +RISCV_IOMMU_HPMEVENT_INVALID =0 , +RISCV_IOMMU_HPMEVENT_URQ =1 , +RISCV_IOMMU_HPMEVENT_TRQ =2 , +RISCV_IOMMU_HPMEVENT_ATS_RQ =3 , +RISCV_IOMMU_HPMEVENT_TLB_MISS =4 , +RISCV_IOMMU_HPMEVENT_DD_WALK =5 , +RISCV_IOMMU_HPMEVENT_PD_WALK =6 , +RISCV_IOMMU_HPMEVENT_S_VS_WALKS =7 , +RISCV_IOMMU_HPMEVENT_G_WALKS =8 , +RISCV_IOMMU_HPMEVENT_MAX =9 +}; + +/* 5.24 Translation request IOVA (64bits) */ +#define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258 +#define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12) + +/* 5.25 Translation request control (64bits) */ +#define RISCV_IOMMU_REG_TR_REQ_CTL 0x0260 +#define RISCV_IOMMU_TR_REQ_CTL_GO_BUSY BIT_ULL(0) +#define RISCV_IOMMU_TR_REQ_CTL_PRIV BIT_ULL(1) +#define RISCV_IOMMU_TR_REQ_CTL_EXE BIT_ULL(2) +#define RISCV_IOMMU_TR_REQ_CTL_NW BIT_ULL(3) +#define RISCV_IOMMU_TR_REQ_CTL_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_TR_REQ_CTL_PV BIT_ULL(32) +#define RISCV_IOMMU_TR_REQ_CTL_DID GENMASK_ULL(63, 40) + +/* 5.26 Translation request response (64bits) */ +#define RISCV_IOMMU_REG_TR_RESPONSE 0x0268 +#define RISCV_IOMMU_TR_RESPONSE_FAULT BIT_ULL(0) +#define RISCV_IOMMU_TR_RESPONSE_PBMT GENMASK_ULL(8, 7) +#define RISCV_IOMMU_TR_RESPONSE_SZ BIT_ULL(9) +#define RISCV_IOMMU_TR_RESPONSE_PPN RISCV_IOMMU_PPN_FIELD + + +/* 5.27 Interrupt cause to vector (64bits) */ +#define RISCV_IOMMU_REG_IVEC 0x02F8 +#define RISCV_IOMMU_IVEC_CIV GENMASK_ULL(3, 0) +#define RISCV_IOMMU_IVEC_FIV GENMASK_ULL(7, 4) +#define RISCV_IOMMU_IVEC_PMIV GENMASK_ULL(11, 8) +#define RISCV_IOMMU_IVEC_PIV GENMASK_ULL(15, 12) + + +/* 5.28 MSI Configuration table (32 * 64bits) */ +#define RISCV_IOMMU_REG_MSI_CONFIG 0x0300 +#define RISCV_IOMMU_REG_MSI_ADDR (_n) \ + (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10)) +#define RISCV_IOMMU_MSI_ADDR GENMASK_ULL(55, 2) +#define RISCV_IOMMU_REG_MSI_DATA (_n) \ + (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10) + 0x08) +#define RISCV_IOMMU_MSI_DATA GENMASK_ULL(31, 0) +#define RISCV_IOMMU_REG_MSI_VEC_CTL (_n) \ + (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10) + 0x0C) +#define RISCV_IOMMU_MSI_VEC_CTL_M BIT_ULL(0) + + +#define RISCV_IOMMU_REG_SIZE 0x1000 + +/* +* Chapter 2: Data structures +*/ + +/* +* Device Directory Table macros for non-leaf nodes +*/ +#define RISCV_IOMMU_DDTE_VALID BIT_ULL(0) +#define RISCV_IOMMU_DDTE_PPN RISCV_IOMMU_PPN_FIELD + +/** +* struct riscv_iommu_dc - Device Context +* @tc: Translation Control +* @iohgatp: I/O Hypervisor guest address translation and protection +* (Second stage context) +* @ta: Translation Attributes +* @fsc: First stage context +* @msiptpt: MSI page table pointer +* @msi_addr_mask: MSI address mask +* @msi_addr_pattern: MSI address pattern +* +* This structure is used for leaf nodes on the Device Directory Table, +* in case RISCV_IOMMU_CAP_MSI_FLAT is not set, the bottom 4 fields are +* not present and are skipped with pointer arithmetic to avoid +* casting, check out riscv_iommu_get_dc(). +* See section 2.1 for more details +*/ +struct riscv_iommu_dc { +uint64_t tc; +uint64_t iohgatp; +uint64_t ta; +uint64_t fsc; +uint64_t msiptp; +uint64_t msi_addr_mask; +uint64_t msi_addr_pattern; +uint64_t _reserved; +}; + +/* Translation control fields */ +#define RISCV_IOMMU_DC_TC_V BIT_ULL(0) +#define RISCV_IOMMU_DC_TC_EN_ATS BIT_ULL(1) +#define RISCV_IOMMU_DC_TC_EN_PRI BIT_ULL(2) +#define RISCV_IOMMU_DC_TC_T2GPA BIT_ULL(3) +#define RISCV_IOMMU_DC_TC_DTF BIT_ULL(4) +#define RISCV_IOMMU_DC_TC_PDTV BIT_ULL(5) +#define RISCV_IOMMU_DC_TC_PRPR BIT_ULL(6) +#define RISCV_IOMMU_DC_TC_GADE BIT_ULL(7) +#define RISCV_IOMMU_DC_TC_SADE BIT_ULL(8) +#define RISCV_IOMMU_DC_TC_DPE BIT_ULL(9) +#define RISCV_IOMMU_DC_TC_SBE BIT_ULL(10) +#define RISCV_IOMMU_DC_TC_SXL BIT_ULL(11) + +/* Second-stage (aka G-stage) context fields */ +#define RISCV_IOMMU_DC_IOHGATP_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_IOHGATP_GSCID GENMASK_ULL(59, 44) +#define RISCV_IOMMU_DC_IOHGATP_MODE RISCV_IOMMU_ATP_MODE_FIELD + +/** +* enum riscv_iommu_dc_iohgatp_modes - Guest address +* translation/protection modes +* +* @RISCV_IOMMU_DC_IOHGATP_MODE_BARE: +* No translation/protection +* @RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4: +* Sv32x4 (2-bit extension of Sv32), when fctl.GXL == 1 +* @RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4: +* Sv39x4 (2-bit extension of Sv39), when fctl.GXL == 0 +* @RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4: +* Sv48x4 (2-bit extension of Sv48), when fctl.GXL == 0 +* @RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4: +* Sv57x4 (2-bit extension of Sv57), when fctl.GXL == 0 +*/ +enum riscv_iommu_dc_iohgatp_modes { +RISCV_IOMMU_DC_IOHGATP_MODE_BARE =0 , +RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4 =8 , +RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4 =8 , +RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4 =9 , +RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4 =10 +}; + +/* Translation attributes fields */ +#define RISCV_IOMMU_DC_TA_PSCID GENMASK_ULL(31, 12) + +/* First-stage context fields */ +#define RISCV_IOMMU_DC_FSC_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_FSC_MODE RISCV_IOMMU_ATP_MODE_FIELD + +/** +* enum riscv_iommu_dc_fsc_atp_modes - First stage address +* translation/protection modes +* +* @RISCV_IOMMU_DC_FSC_MODE_BARE: No translation/protection +* @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32: Sv32, when dc.tc.SXL == 1 +* @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39: Sv39, when dc.tc.SXL == 0 +* @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48: Sv48, when dc.tc.SXL == 0 +* @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57: Sv57, when dc.tc.SXL == 0 +* @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8: 1lvl PDT, 8bit process ids +* @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17: 2lvl PDT, 17bit process ids +* @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20: 3lvl PDT, 20bit process ids +* +* FSC holds IOSATP when RISCV_IOMMU_DC_TC_PDTV is 0 and PDTP otherwise. +* IOSATP controls the first stage address translation (same as the satp +* register on the RISC-V MMU), and PDTP holds the process directory table, +* used to select a first stage page table based on a process id (for devices +* that support multiple process ids). +*/ +enum riscv_iommu_dc_fsc_atp_modes { +RISCV_IOMMU_DC_FSC_MODE_BARE =0 , +RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32 =8 , +RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39 =8 , +RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48 =9 , +RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57 =10 , +RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8 =1 , +RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17 =2 , +RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20 =3 +}; + +/* MSI page table pointer */ +#define RISCV_IOMMU_DC_MSIPTP_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_MSIPTP_MODE RISCV_IOMMU_ATP_MODE_FIELD +#define RISCV_IOMMU_DC_MSIPTP_MODE_OFF 0 +#define RISCV_IOMMU_DC_MSIPTP_MODE_FLAT 1 + +/* MSI address mask */ +#define RISCV_IOMMU_DC_MSI_ADDR_MASK GENMASK_ULL(51, 0) + +/* MSI address pattern */ +#define RISCV_IOMMU_DC_MSI_PATTERN GENMASK_ULL(51, 0) + + +/** +* struct riscv_iommu_pc - Process Context +* @ta: Translation Attributes +* @fsc: First stage context +* +* This structure is used for leaf nodes on the Process Directory Table +* See section 2.3 for more details +*/ +struct riscv_iommu_pc { +uint64_t ta; +uint64_t fsc; +}; + +/* Translation attributes fields */ +#define RISCV_IOMMU_PC_TA_V BIT_ULL(0) +#define RISCV_IOMMU_PC_TA_ENS BIT_ULL(1) +#define RISCV_IOMMU_PC_TA_SUM BIT_ULL(2) +#define RISCV_IOMMU_PC_TA_PSCID GENMASK_ULL(31, 12) + +/* First stage context fields */ +#define RISCV_IOMMU_PC_FSC_PPN GENMASK_ULL(43, 0) +#define RISCV_IOMMU_PC_FSC_MODE GENMASK_ULL(63, 60) + + +/* +* Chapter 3: In-memory queue interface +*/ + +/** +* struct riscv_iommu_cmd - Generic I/O MMU command structure +* @dword0: Includes the opcode and the function identifier +* @dword1: Opcode specific data +* +* The commands are interpreted as two 64bit fields, where the first +* 7bits of the first field are the opcode which also defines the +* command's format, followed by a 3bit field that specifies the +* function invoked by that command, and the rest is opcode-specific. +* This is a generic struct which will be populated differently +* according to each command. For more infos on the commands and +* the command queue check section 3.1. +*/ +struct riscv_iommu_command { +uint64_t dword0; +uint64_t dword1; +}; + +/* Fields on dword0, common for all commands */ +#define RISCV_IOMMU_CMD_OPCODE GENMASK_ULL(6, 0) +#define RISCV_IOMMU_CMD_FUNC GENMASK_ULL(9, 7) + +/* 3.1.1 I/O MMU Page-table cache invalidation */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IOTINVAL_OPCODE 1 +#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA 0 +#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA 1 +#define RISCV_IOMMU_CMD_IOTINVAL_AV BIT_ULL(10) +#define RISCV_IOMMU_CMD_IOTINVAL_PSCID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_IOTINVAL_PSCV BIT_ULL(32) +#define RISCV_IOMMU_CMD_IOTINVAL_GV BIT_ULL(33) +#define RISCV_IOMMU_CMD_IOTINVAL_GSCID GENMASK_ULL(59, 44) +/* dword1 is the address, 4K-alligned and shifted to the right by two bits. */ + +/* 3.1.2 I/O MMU Command Queue Fences */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IOFENCE_OPCODE 2 +#define RISCV_IOMMU_CMD_IOFENCE_FUNC_C 0 +#define RISCV_IOMMU_CMD_IOFENCE_AV BIT_ULL(10) +#define RISCV_IOMMU_CMD_IOFENCE_WSI BIT_ULL(11) +#define RISCV_IOMMU_CMD_IOFENCE_PR BIT_ULL(12) +#define RISCV_IOMMU_CMD_IOFENCE_PW BIT_ULL(13) +#define RISCV_IOMMU_CMD_IOFENCE_DATA GENMASK_ULL(63, 32) +/* dword1 is the address, word-size alligned and shifted to the right by two bits. */ + +/* 3.1.3 I/O MMU Directory cache invalidation */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IODIR_OPCODE 3 +#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT 0 +#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT 1 +#define RISCV_IOMMU_CMD_IODIR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_IODIR_DV BIT_ULL(33) +#define RISCV_IOMMU_CMD_IODIR_DID GENMASK_ULL(63, 40) +/* dword1 is reserved for standard use */ + +/* 3.1.4 I/O MMU PCIe ATS */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_ATS_OPCODE 4 +#define RISCV_IOMMU_CMD_ATS_FUNC_INVAL 0 +#define RISCV_IOMMU_CMD_ATS_FUNC_PRGR 1 +#define RISCV_IOMMU_CMD_ATS_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_ATS_PV BIT_ULL(32) +#define RISCV_IOMMU_CMD_ATS_DSV BIT_ULL(33) +#define RISCV_IOMMU_CMD_ATS_RID GENMASK_ULL(55, 40) +#define RISCV_IOMMU_CMD_ATS_DSEG GENMASK_ULL(63, 56) +/* dword1 is the ATS payload, two different payload types for INVAL and PRGR */ + +/* ATS.INVAL payload*/ +#define RISCV_IOMMU_CMD_ATS_INVAL_G BIT_ULL(0) +/* Bits 1 - 10 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_INVAL_S BIT_ULL(11) +#define RISCV_IOMMU_CMD_ATS_INVAL_UADDR GENMASK_ULL(63, 12) + +/* ATS.PRGR payload */ +/* Bits 0 - 31 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_PRGR_PRG_INDEX GENMASK_ULL(40, 32) +/* Bits 41 - 43 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_PRGR_RESP_CODE GENMASK_ULL(47, 44) +#define RISCV_IOMMU_CMD_ATS_PRGR_DST_ID GENMASK_ULL(63, 48) + + +/** +* struct riscv_iommu_fq_record - Fault/Event Queue Record +* @hdr: Header, includes fault/event cause, PID/DID, transaction type etc +* @_reserved: Low 32bits for custom use, high 32bits for standard use +* @iotval: Transaction-type/cause specific format +* @iotval2: Cause specific format +* +* The fault/event queue reports events and failures raised when +* processing transactions. Each record is a 32byte structure where +* the first dword has a fixed format for providing generic infos +* regarding the fault/event, and two more dwords are there for +* fault/event-specific information. For more details see section +* 3.2. +*/ +struct riscv_iommu_fq_record { +uint64_t hdr; +uint64_t _reserved; +uint64_t iotval; +uint64_t iotval2; +}; + +/* Fields on header */ +#define RISCV_IOMMU_FQ_HDR_CAUSE GENMASK_ULL(11, 0) +#define RISCV_IOMMU_FQ_HDR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_FQ_HDR_PV BIT_ULL(32) +#define RISCV_IOMMU_FQ_HDR_PRIV BIT_ULL(33) +#define RISCV_IOMMU_FQ_HDR_TTYPE GENMASK_ULL(39, 34) +#define RISCV_IOMMU_FQ_HDR_DID GENMASK_ULL(63, 40) + +/** +* enum riscv_iommu_fq_causes - Fault/event cause values +* @RISCV_IOMMU_FQ_CAUSE_INST_FAULT: Instruction access fault +* @RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED: Read address misaligned +* @RISCV_IOMMU_FQ_CAUSE_RD_FAULT: Read load fault +* @RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED: Write/AMO address misaligned +* @RISCV_IOMMU_FQ_CAUSE_WR_FAULT: Write/AMO access fault +* @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S: Instruction page fault +* @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S: Read page fault +* @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S: Write/AMO page fault +* @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS: Instruction guest page fault +* @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS: Read guest page fault +* @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS: Write/AMO guest page fault +* @RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED: All inbound transactions disallowed +* @RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT: DDT entry load access fault +* @RISCV_IOMMU_FQ_CAUSE_DDT_INVALID: DDT entry invalid +* @RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED: DDT entry misconfigured +* @RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED: Transaction type disallowed +* @RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT: MSI PTE load access fault +* @RISCV_IOMMU_FQ_CAUSE_MSI_INVALID: MSI PTE invalid +* @RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED: MSI PTE misconfigured +* @RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT: MRIF access fault +* @RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT: PDT entry load access fault +* @RISCV_IOMMU_FQ_CAUSE_PDT_INVALID: PDT entry invalid +* @RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED: PDT entry misconfigured +* @RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED: DDT data corruption +* @RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED: PDT data corruption +* @RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED: MSI page table data corruption +* @RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED: MRIF data corruption +* @RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR: Internal data path error +* @RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT: IOMMU MSI write access fault +* @RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED: First/second stage page table data corruption +* +* Values are on table 11 of the spec, encodings 275 - 2047 are reserved for +* standard use, and 2048 - 4095 for custom use. +*/ +enum riscv_iommu_fq_causes { +RISCV_IOMMU_FQ_CAUSE_INST_FAULT =1 , +RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED =4 , +RISCV_IOMMU_FQ_CAUSE_RD_FAULT =5 , +RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED =6 , +RISCV_IOMMU_FQ_CAUSE_WR_FAULT =7 , +RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S =12 , +RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S =13 , +RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S =15 , +RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS =20 , +RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS =21 , +RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS =23 , +RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED =256 , +RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT =257 , +RISCV_IOMMU_FQ_CAUSE_DDT_INVALID =258 , +RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED =259 , +RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED =260 , +RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT =261 , +RISCV_IOMMU_FQ_CAUSE_MSI_INVALID =262 , +RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED =263 , +RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT =264 , +RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT =265 , +RISCV_IOMMU_FQ_CAUSE_PDT_INVALID =266 , +RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED =267 , +RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED =268 , +RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED =269 , +RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED =270 , +RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED =271 , +RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR =272 , +RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT =273 , +RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED =274 +}; + +/** +* enum riscv_iommu_fq_ttypes: Fault/event transaction types +* @RISCV_IOMMU_FQ_TTYPE_NONE: None. Fault not caused by an inbound transaction. +* @RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH: Instruction fetch from untranslated address +* @RISCV_IOMMU_FQ_TTYPE_UADDR_RD: Read from untranslated address +* @RISCV_IOMMU_FQ_TTYPE_UADDR_WR: Write/AMO to untranslated address +* @RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH: Instruction fetch from translated address +* @RISCV_IOMMU_FQ_TTYPE_TADDR_RD: Read from translated address +* @RISCV_IOMMU_FQ_TTYPE_TADDR_WR: Write/AMO to translated address +* @RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ: PCIe ATS translation request +* @RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ: PCIe message request +* +* Values are on table 12 of the spec, type 4 and 10 - 31 are reserved for +* standard use and 31 - 63 for custom use. +*/ +enum riscv_iommu_fq_ttypes { +RISCV_IOMMU_FQ_TTYPE_NONE =0 , +RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH =1 , +RISCV_IOMMU_FQ_TTYPE_UADDR_RD =2 , +RISCV_IOMMU_FQ_TTYPE_UADDR_WR =3 , +RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH =5 , +RISCV_IOMMU_FQ_TTYPE_TADDR_RD =6 , +RISCV_IOMMU_FQ_TTYPE_TADDR_WR =7 , +RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ =8 , +RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ =9 , +}; + + +/** +* struct riscv_iommu_pq_record - PCIe Page Request record +* @hdr: Header, includes PID, DID etc +* @payload: Holds the page address, request group and permission bits +* +* For more infos on the PCIe Page Request queue see chapter 3.3. +*/ +struct riscv_iommu_pq_record { +uint64_t hdr; +uint64_t payload; +}; + +/* Header fields */ +#define RISCV_IOMMU_PREQ_HDR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_PREQ_HDR_PV BIT_ULL(32) +#define RISCV_IOMMU_PREQ_HDR_PRIV BIT_ULL(33) +#define RISCV_IOMMU_PREQ_HDR_EXEC BIT_ULL(34) +#define RISCV_IOMMU_PREQ_HDR_DID GENMASK_ULL(63, 40) + +/* Payload fields */ +#define RISCV_IOMMU_PREQ_PAYLOAD_R BIT_ULL(0) +#define RISCV_IOMMU_PREQ_PAYLOAD_W BIT_ULL(1) +#define RISCV_IOMMU_PREQ_PAYLOAD_L BIT_ULL(2) +#define RISCV_IOMMU_PREQ_PAYLOAD_M GENMASK_ULL(2, 0) +#define RISCV_IOMMU_PREQ_PRG_INDEX GENMASK_ULL(11, 3) +#define RISCV_IOMMU_PREQ_UADDR GENMASK_ULL(63, 12) + + +/** +* struct riscv_iommu_msi_pte - MSI Page Table Entry +* @pte: MSI PTE +* @mrif_info: Memory-resident interrupt file info +* +* The MSI Page Table is used for virtualizing MSIs, so that when +* a device sends an MSI to a guest, the IOMMU can reroute it +* by translating the MSI address, either to a guest interrupt file +* or a memory resident interrupt file (MRIF). Note that this page table +* is an array of MSI PTEs, not a multi-level pt, each entry +* is a leaf entry. For more infos check out the the AIA spec, chapter 9.5. +* +* Also in basic mode the mrif_info field is ignored by the IOMMU and can +* be used by software, any other reserved fields on pte must be zeroed-out +* by software. +*/ +struct riscv_iommu_msi_pte { +uint64_t pte; +uint64_t mrif_info; +}; + +/* Fields on pte */ +#define RISCV_IOMMU_MSI_PTE_V BIT_ULL(0) +#define RISCV_IOMMU_MSI_PTE_M GENMASK_ULL(2, 1) + +#define RISCV_IOMMU_MSI_PTE_M_MRIF 1 +#define RISCV_IOMMU_MSI_PTE_M_BASIC 3 + +/* When M == 1 (MRIF mode) */ +#define RISCV_IOMMU_MSI_PTE_MRIF_ADDR GENMASK_ULL(53, 7) +/* When M == 3 (basic mode) */ +#define RISCV_IOMMU_MSI_PTE_PPN RISCV_IOMMU_PPN_FIELD +#define RISCV_IOMMU_MSI_PTE_C BIT_ULL(63) + +/* Fields on mrif_info */ +#define RISCV_IOMMU_MSI_MRIF_NID GENMASK_ULL(9, 0) +#define RISCV_IOMMU_MSI_MRIF_NPPN RISCV_IOMMU_PPN_FIELD +#define RISCV_IOMMU_MSI_MRIF_NID_MSB BIT_ULL(60) + + +#endif /* _RISCV_IOMMU_BITS_H_ */ hw/riscv/riscv-iommu-pci.c new file +31
-0@@ -0,0 +1,181 @@ +/* +* QEMU emulation of an RISC-V IOMMU (Ziommu) +* +* Copyright (C) 2022-2023 Rivos Inc. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#include "qemu/osdep.h" +#include "hw/pci/msi.h" +#include "hw/pci/msix.h" +#include "hw/pci/pci_bus.h" +#include "hw/qdev-properties.h" +#include "hw/riscv/riscv_hart.h" +#include "migration/vmstate.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qemu/host-utils.h" +#include "qom/object.h" + +#include "cpu_bits.h"
hw/riscv/riscv-iommu-pci.c line 41
Quoted 10 lines
hw/riscv/riscv-iommu-pci.c new file +10
-0+#include "riscv-iommu.h" +#include "riscv-iommu-bits.h" + +#ifndef PCI_VENDOR_ID_RIVOS +#define PCI_VENDOR_ID_RIVOS 0x1efd +#endif + +#ifndef PCI_DEVICE_ID_RIVOS_IOMMU +#define PCI_DEVICE_ID_RIVOS_IOMMU 0xedf1 +#endif
The file is the RISC-V IOMMU, but don't these IDs say Rivos IOMMU?
Quoted 128 lines
hw/riscv/riscv-iommu-pci.c new file +128
-0+ +/* RISC-V IOMMU PCI Device Emulation */ + +typedef struct RISCVIOMMUStatePci { +PCIDevice pci;/* Parent PCIe device state */ +MemoryRegion bar0;/* PCI BAR (including MSI-x config) */ +RISCVIOMMUState iommu;/* common IOMMU state */ +}RISCVIOMMUStatePci ; + +/* interrupt delivery callback */ +static void riscv_iommu_pci_notify (RISCVIOMMUState *iommu,unsigned vector) +{ +RISCVIOMMUStatePci *s =container_of (iommu, RISCVIOMMUStatePci, iommu); + +if (msix_enabled (&(s->pci))) { +msix_notify (&(s->pci), vector); + } +} + +static void riscv_iommu_pci_realize (PCIDevice *dev,Error **errp) +{ +RISCVIOMMUStatePci *s =DO_UPCAST (RISCVIOMMUStatePci, pci, dev); +RISCVIOMMUState *iommu = &s->iommu; +uint64_t cap = iommu->cap; +Error *err =NULL ; + +/* Set device id for trace / debug */ +DEVICE (iommu)->id =g_strdup_printf ("%02x:%02x.%01x" , +pci_dev_bus_num (dev),PCI_SLOT (dev->devfn),PCI_FUNC (dev->devfn)); + +/* Support MSI only */ + cap =set_field (cap,RISCV_IOMMU_CAP_IGS ,RISCV_IOMMU_CAP_IGS_MSI ); +qdev_prop_set_uint64 (DEVICE (dev),"capabilities" , cap); + +if (!qdev_realize (DEVICE (iommu),NULL , errp)) { +return ; + } + +memory_region_init (&s->bar0,OBJECT (s),"riscv-iommu-bar0" , +QEMU_ALIGN_UP (memory_region_size (&iommu->regs_mr),TARGET_PAGE_SIZE )); +memory_region_add_subregion (&s->bar0,0 , &iommu->regs_mr); + +pcie_endpoint_cap_init (dev,0 ); + +pci_register_bar (dev,0 ,PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , &s->bar0); + +int ret =msix_init (dev,RISCV_IOMMU_INTR_COUNT , + &s->bar0,0 ,RISCV_IOMMU_REG_MSI_CONFIG , + &s->bar0,0 ,RISCV_IOMMU_REG_MSI_CONFIG +256 ,0 , &err); + +if (ret == -ENOTSUP ) { +/* +* MSI-x is not supported by the platform. +* Driver should use timer/polling based notification handlers. +*/ +warn_report_err (err); + }else if (ret <0 ) { +error_propagate (errp, err); +return ; + }else { +/* mark all allocated MSIx vectors as used. */ +msix_vector_use (dev,RISCV_IOMMU_INTR_CQ ); +msix_vector_use (dev,RISCV_IOMMU_INTR_FQ ); +msix_vector_use (dev,RISCV_IOMMU_INTR_PM ); +msix_vector_use (dev,RISCV_IOMMU_INTR_PQ ); + iommu->notify = riscv_iommu_pci_notify; + } + +PCIBus *bus =pci_device_root_bus (dev); +if (!bus) { +error_setg (errp,"can't find PCIe root port for %02x:%02x.%x" , +pci_bus_num (pci_get_bus (dev)),PCI_SLOT (dev->devfn), +PCI_FUNC (dev->devfn)); +return ; + } + +riscv_iommu_pci_setup_iommu (iommu, bus, errp); +} + +static void riscv_iommu_pci_exit (PCIDevice *pci_dev) +{ +pci_setup_iommu (pci_device_root_bus (pci_dev),NULL ,NULL ); +} + +static const VMStateDescription riscv_iommu_vmstate = { + .name ="riscv-iommu" , + .unmigratable =1 +}; + +static void riscv_iommu_pci_init (Object *obj) +{ +RISCVIOMMUStatePci *s =RISCV_IOMMU_PCI (obj); +RISCVIOMMUState *iommu = &s->iommu; + +object_initialize_child (obj,"iommu" , iommu,TYPE_RISCV_IOMMU ); +qdev_alias_all_properties (DEVICE (iommu), obj); +} + +static Property riscv_iommu_pci_properties[] = { +DEFINE_PROP_END_OF_LIST (), +}; + +static void riscv_iommu_pci_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *k =PCI_DEVICE_CLASS (klass); + + k->realize = riscv_iommu_pci_realize; + k->exit = riscv_iommu_pci_exit; + k->vendor_id =PCI_VENDOR_ID_RIVOS ; + k->device_id =PCI_DEVICE_ID_RIVOS_IOMMU ; + k->revision =0 ; + k->class_id =0x0806 ; + dc->desc ="RISCV-IOMMU DMA Remapping device" ; + dc->vmsd = &riscv_iommu_vmstate; + dc->hotpluggable = false; + dc->user_creatable = true; +set_bit (DEVICE_CATEGORY_MISC , dc->categories); +device_class_set_props (dc, riscv_iommu_pci_properties); +} + +static const TypeInfo riscv_iommu_pci = { + .name =TYPE_RISCV_IOMMU_PCI , + .parent =TYPE_PCI_DEVICE , + .class_init = riscv_iommu_pci_class_init, + .instance_init = riscv_iommu_pci_init, + .instance_size =sizeof (RISCVIOMMUStatePci),
hw/riscv/riscv-iommu-pci.c line 179
Quoted 10 lines
hw/riscv/riscv-iommu-pci.c new file +10
-0+ .interfaces = (InterfaceInfo[]) { + {INTERFACE_PCIE_DEVICE }, + { }, + }, +}; + +static void riscv_iommu_register_pci_types (void ) +{ +type_register_static (&riscv_iommu_pci); +}
The PCIe device should be a seperate patch
Quoted 119 lines
hw/riscv/riscv-iommu-pci.c new file +2
-0+ +type_init (riscv_iommu_register_pci_types);hw/riscv/riscv-iommu-sys.c new file +111
-0@@ -0,0 +1,123 @@ +/* +* QEMU emulation of an RISC-V IOMMU (Ziommu) - Platform Device +* +* Copyright (C) 2022-2023 Rivos Inc. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#include "qemu/osdep.h" +#include "hw/pci/pci_bus.h" +#include "hw/irq.h" +#include "hw/qdev-properties.h" +#include "hw/sysbus.h" +#include "qapi/error.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qemu/host-utils.h" +#include "qemu/module.h" +#include "qemu/osdep.h" +#include "qom/object.h" + +#include "cpu_bits.h" +#include "riscv-iommu.h" +#include "riscv-iommu-bits.h" + +/* RISC-V IOMMU System Platform Device Emulation */ + +struct RISCVIOMMUStateSys { +SysBusDevice parent; +uint64_t addr; +qemu_irq irqs[4 ]; +RISCVIOMMUState iommu; +}; + +/* interrupt delivery callback */ +static void riscv_iommu_sys_notify (RISCVIOMMUState *iommu,unsigned vector) +{ +RISCVIOMMUStateSys *s =container_of (iommu, RISCVIOMMUStateSys, iommu); + +if (vector <RISCV_IOMMU_INTR_COUNT && s->irqs[vector]) { +qemu_irq_pulse (s->irqs[vector]); + } +} + +static void riscv_iommu_sys_realize (DeviceState *dev,Error **errp) +{ +RISCVIOMMUStateSys *s =RISCV_IOMMU_SYS (dev); +RISCVIOMMUState *iommu = &s->iommu; +PCIBus *pci_bus; +uint64_t cap = iommu->cap; +int i; + +/* Support WSI only */ + cap =set_field (cap,RISCV_IOMMU_CAP_IGS ,RISCV_IOMMU_CAP_IGS_WSI ); +qdev_prop_set_uint64 (dev,"capabilities" , cap); + +if (!qdev_realize (DEVICE (iommu),NULL , errp)) { +return ; + } + +sysbus_init_mmio (SYS_BUS_DEVICE (dev), &iommu->regs_mr); +if (s->addr) { +sysbus_mmio_map (SYS_BUS_DEVICE (s),0 , s->addr); + } + +for (i =0 ; i <RISCV_IOMMU_INTR_COUNT ; i++) { +sysbus_init_irq (&s->parent, &s->irqs[i]); + } + + iommu->notify = riscv_iommu_sys_notify; + + pci_bus = (PCIBus *)object_resolve_path_type ("" ,TYPE_PCI_BUS ,NULL ); +if (pci_bus) { +riscv_iommu_pci_setup_iommu (iommu, pci_bus, errp); + } +} + +static void riscv_iommu_sys_init (Object *obj) +{ +RISCVIOMMUStateSys *s =RISCV_IOMMU_SYS (obj); +RISCVIOMMUState *iommu = &s->iommu; + +object_initialize_child (obj,"iommu" , iommu,TYPE_RISCV_IOMMU ); +qdev_alias_all_properties (DEVICE (iommu), obj); +} + +static Property riscv_iommu_sys_properties[] = { +DEFINE_PROP_UINT64 ("addr" , RISCVIOMMUStateSys, addr,0 ), +DEFINE_PROP_END_OF_LIST (), +}; + +static void riscv_iommu_sys_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); + dc->realize = riscv_iommu_sys_realize; +set_bit (DEVICE_CATEGORY_MISC , dc->categories); +device_class_set_props (dc, riscv_iommu_sys_properties); +} + +static const TypeInfo riscv_iommu_sys = { + .name =TYPE_RISCV_IOMMU_SYS ,
hw/riscv/riscv-iommu-sys.c line 121
Quoted 10 lines
hw/riscv/riscv-iommu-sys.c new file +10
-0+ .parent = TYPE_SYS_BUS_DEVICE, + .class_init = riscv_iommu_sys_class_init, + .instance_init = riscv_iommu_sys_init, + .instance_size = sizeof(RISCVIOMMUStateSys), +}; + +static void riscv_iommu_register_sys (void ) +{ +type_register_static (&riscv_iommu_sys); +}
Same here
Quoted 2,795 lines
hw/riscv/riscv-iommu-sys.c new file +2
-0+ +type_init (riscv_iommu_register_sys)hw/riscv/riscv-iommu.c new file +2539
-0@@ -0,0 +1,2539 @@ +/* +* QEMU emulation of an RISC-V IOMMU (Ziommu) +* +* Copyright (C) 2021-2023, Rivos Inc. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#include "qemu/osdep.h" +#include "qom/object.h" +#include "hw/pci/pci_bus.h" +#include "hw/pci/pci_device.h" +#include "hw/qdev-properties.h" +#include "hw/riscv/riscv_hart.h" +#include "migration/vmstate.h" +#include "qapi/error.h" +#include "qemu/timer.h" + +#include "cpu_bits.h" +#include "riscv-iommu.h" +#include "riscv-iommu-bits.h" +#include "trace.h" + +#define LIMIT_CACHE_CTX (1U << 7) +#define LIMIT_CACHE_IOT (1U << 20) + +/* Physical page number coversions */ +#define PPN_PHYS (ppn) ((ppn) << TARGET_PAGE_BITS) +#define PPN_DOWN (phy) ((phy) >> TARGET_PAGE_BITS) + +typedef struct RISCVIOMMUContext RISCVIOMMUContext ; +typedef struct RISCVIOMMUEntry RISCVIOMMUEntry ; + +/* Device assigned I/O address space */ +struct RISCVIOMMUSpace { +IOMMUMemoryRegion iova_mr;/* IOVA memory region for attached device */ +AddressSpace iova_as;/* IOVA address space for attached device */ +RISCVIOMMUState *iommu;/* Managing IOMMU device state */ +uint32_t devid;/* Requester identifier, AKA device_id */ +bool notifier;/* IOMMU unmap notifier enabled */ +QLIST_ENTRY (RISCVIOMMUSpace) list; +}; + +/* Device translation context state. */ +struct RISCVIOMMUContext { +uint64_t devid:24 ;/* Requester Id, AKA device_id */ +uint64_t pasid:20 ;/* Process Address Space ID */ +uint64_t __rfu:20 ;/* reserved */ +uint64_t tc;/* Translation Control */ +uint64_t ta;/* Translation Attributes */ +uint64_t satp;/* S-Stage address translation and protection */ +uint64_t gatp;/* G-Stage address translation and protection */ +uint64_t msi_addr_mask;/* MSI filtering - address mask */ +uint64_t msi_addr_pattern;/* MSI filtering - address pattern */ +uint64_t msiptp;/* MSI redirection page table pointer */ +}; + +/* Address translation cache entry */ +struct RISCVIOMMUEntry { +uint64_t iova:44 ;/* IOVA Page Number */ +uint64_t pscid:20 ;/* Process Soft-Context identifier */ +uint64_t phys:44 ;/* Physical Page Number */ +uint64_t gscid:16 ;/* Guest Soft-Context identifier */ +uint64_t perm:2 ;/* IOMMU_RW flags */ +uint64_t __rfu:2 ; +}; + +/* IOMMU index for transactions without PASID specified. */ +#define RISCV_IOMMU_NOPASID 0 + +static void riscv_iommu_notify (RISCVIOMMUState *s,int vec) +{ +const uint32_t ipsr = +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_IPSR , (1 << vec),0 ); +const uint32_t ivec =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IVEC ); +if (s->notify && !(ipsr & (1 << vec))) { + s->notify (s, (ivec >> (vec *4 )) &0x0F ); + } +} + +static void riscv_iommu_fault (RISCVIOMMUState *s,struct riscv_iommu_fq_record *ev) +{ +uint32_t ctrl =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_FQCSR ); +uint32_t head =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_FQH ) & s->fq_mask; +uint32_t tail =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_FQT ) & s->fq_mask; +uint32_t next = (tail +1 ) & s->fq_mask; +uint32_t devid =get_field (ev->hdr,RISCV_IOMMU_FQ_HDR_DID ); + +trace_riscv_iommu_flt (s->parent_obj.id,PCI_BUS_NUM (devid),PCI_SLOT (devid), +PCI_FUNC (devid), ev->hdr, ev->iotval); + +if (!(ctrl &RISCV_IOMMU_FQCSR_FQON ) || + !!(ctrl & (RISCV_IOMMU_FQCSR_FQOF |RISCV_IOMMU_FQCSR_FQMF ))) { +return ; + } + +if (head == next) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_FQCSR ,RISCV_IOMMU_FQCSR_FQOF ,0 ); + }else { +dma_addr_t addr = s->fq_addr + tail *sizeof (*ev); +if (dma_memory_write (s->target_as, addr, ev,sizeof (*ev), +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_FQCSR ,RISCV_IOMMU_FQCSR_FQMF ,0 ); + }else { +riscv_iommu_reg_set32 (s,RISCV_IOMMU_REG_FQT , next); + } + } + +if (ctrl &RISCV_IOMMU_FQCSR_FIE ) { +riscv_iommu_notify (s,RISCV_IOMMU_INTR_FQ ); + } +} + +static void riscv_iommu_pri (RISCVIOMMUState *s, +struct riscv_iommu_pq_record *pr) +{ +uint32_t ctrl =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_PQCSR ); +uint32_t head =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_PQH ) & s->pq_mask; +uint32_t tail =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_PQT ) & s->pq_mask; +uint32_t next = (tail +1 ) & s->pq_mask; +uint32_t devid =get_field (pr->hdr,RISCV_IOMMU_PREQ_HDR_DID ); + +trace_riscv_iommu_pri (s->parent_obj.id,PCI_BUS_NUM (devid),PCI_SLOT (devid), +PCI_FUNC (devid), pr->payload); + +if (!(ctrl &RISCV_IOMMU_PQCSR_PQON ) || + !!(ctrl & (RISCV_IOMMU_PQCSR_PQOF |RISCV_IOMMU_PQCSR_PQMF ))) { +return ; + } + +if (head == next) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_PQCSR ,RISCV_IOMMU_PQCSR_PQOF ,0 ); + }else { +dma_addr_t addr = s->pq_addr + tail *sizeof (*pr); +if (dma_memory_write (s->target_as, addr, pr,sizeof (*pr), +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_PQCSR ,RISCV_IOMMU_PQCSR_PQMF ,0 ); + }else { +riscv_iommu_reg_set32 (s,RISCV_IOMMU_REG_PQT , next); + } + } + +if (ctrl &RISCV_IOMMU_PQCSR_PIE ) { +riscv_iommu_notify (s,RISCV_IOMMU_INTR_PQ ); + } +} + +static void __hpm_incr_ctr (RISCVIOMMUState *s,uint32_t ctr_idx) +{ +const uint32_t off = ctr_idx <<3 ; +uint64_t cntr_val; + +qemu_spin_lock (&s->regs_lock); + cntr_val =ldq_le_p (&s->regs_rw[RISCV_IOMMU_REG_IOHPMCTR_BASE + off]); +stq_le_p (&s->regs_rw[RISCV_IOMMU_REG_IOHPMCTR_BASE + off], cntr_val +1 ); +qemu_spin_unlock (&s->regs_lock); + +/* Handle the overflow scenario. */ +if (cntr_val ==UINT64_MAX ) { +/* +* Generate interrupt only if OF bit is clear. +1 to offset the cycle +* register OF bit. +*/ +const uint32_t ovf = +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ,BIT (ctr_idx +1 ),0 ); +if (!get_field (ovf,BIT (ctr_idx +1 ))) { +riscv_iommu_reg_mod64 (s, +RISCV_IOMMU_REG_IOHPMEVT_BASE + off, +RISCV_IOMMU_IOHPMEVT_OF , +0 ); +riscv_iommu_notify (s,RISCV_IOMMU_INTR_PM ); + } + } +} + +static void riscv_iommu_hpm_incr_ctr (RISCVIOMMUState *s,RISCVIOMMUContext *ctx, +unsigned event_id) +{ +const uint32_t inhibit =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTINH ); +uint32_t did_gscid; +uint32_t pid_pscid; +uint32_t ctr_idx; +gpointer value; +uint32_t ctrs; +uint64_t evt; + +if (!(s->cap &RISCV_IOMMU_CAP_HPM )) { +return ; + } + +pthread_rwlock_rdlock (&s->ht_lock); + value =g_hash_table_lookup (s->hpm_event_ctr_map, +GUINT_TO_POINTER (event_id)); +if (value ==NULL ) { +pthread_rwlock_unlock (&s->ht_lock); +return ; + } + +for (ctrs =GPOINTER_TO_UINT (value); ctrs !=0 ; ctrs &= ctrs -1 ) { + ctr_idx =ctz32 (ctrs); +if (get_field (inhibit,BIT (ctr_idx +1 ))) { +continue ; + } + + evt =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_IOHPMEVT_BASE + (ctr_idx <<3 )); + +/* +* It's quite possible that event ID has been changed in counter +* but hashtable hasn't been updated yet. We don't want to increment +* counter for the old event ID. +*/ +if (event_id !=get_field (evt,RISCV_IOMMU_IOHPMEVT_EVENT_ID )) { +continue ; + } + +if (get_field (evt,RISCV_IOMMU_IOHPMEVT_IDT )) { + did_gscid =get_field (ctx->gatp,RISCV_IOMMU_DC_IOHGATP_GSCID ); + pid_pscid =get_field (ctx->ta,RISCV_IOMMU_DC_TA_PSCID ); + }else { + did_gscid = ctx->devid; + pid_pscid = ctx->pasid; + } + +if (get_field (evt,RISCV_IOMMU_IOHPMEVT_PV_PSCV )) { +/* +* If the transaction does not have a valid process_id, counter +* increments if device_id matches DID_GSCID. If the transaction has +* a valid process_id, counter increments if device_id matches +* DID_GSCID and process_id matches PID_PSCID. See IOMMU +* Specification, Chapter 5.23. Performance-monitoring event +* selector. +*/ +if (ctx->pasid && +get_field (evt,RISCV_IOMMU_IOHPMEVT_PID_PSCID ) != pid_pscid) { +continue ; + } + } + +if (get_field (evt,RISCV_IOMMU_IOHPMEVT_DV_GSCV )) { +uint32_t mask = ~0 ; + +if (get_field (evt,RISCV_IOMMU_IOHPMEVT_DMASK )) { +/* +* 1001 1011 mask = GSCID +* 0000 0111 mask = mask ^ (mask + 1) +* 1111 1000 mask = ~mask; +*/ + mask =get_field (evt,RISCV_IOMMU_IOHPMEVT_DID_GSCID ); + mask = mask ^ (mask +1 ); + mask = ~mask; + } + +if ((get_field (evt,RISCV_IOMMU_IOHPMEVT_DID_GSCID ) & mask) != + (did_gscid & mask)) { +continue ; + } + } + +__hpm_incr_ctr (s, ctr_idx); + } + +pthread_rwlock_unlock (&s->ht_lock); +} + +/* Portable implementation of pext_u64, bit-mask extraction. */ +static uint64_t _pext_u64 (uint64_t val,uint64_t ext) +{ +uint64_t ret =0 ; +uint64_t rot =1 ; + +while (ext) { +if (ext &1 ) { +if (val &1 ) { + ret |= rot; + } + rot <<=1 ; + } + val >>=1 ; + ext >>=1 ; + } + +return ret; +} + +/* Check if GPA matches MSI/MRIF pattern. */ +static bool riscv_iommu_msi_check (RISCVIOMMUState *s,RISCVIOMMUContext *ctx, +dma_addr_t gpa) +{ +if (get_field (ctx->msiptp,RISCV_IOMMU_DC_MSIPTP_MODE ) != +RISCV_IOMMU_DC_MSIPTP_MODE_FLAT ) { +return false;/* Invalid MSI/MRIF mode */ + } + +if ((PPN_DOWN (gpa) ^ ctx->msi_addr_pattern) & ~ctx->msi_addr_mask) { +return false;/* GPA not in MSI range defined by AIA IMSIC rules. */ + } + +return true; +} + +/* +* RISCV IOMMU Address Translation Lookup - Page Table Walk +* +* Note: Code is based on get_physical_address() from target/riscv/cpu_helper.c +* Both implementation can be merged into single helper function in future. +* Keeping them separate for now, as error reporting and flow specifics are +* sufficiently different for separate implementation. +* +* @s : IOMMU Device State +* @ctx : Translation context for device id and process address space id. +* @iotlb : translation data: physical address and access mode. +* @gpa : provided IOVA is a guest physical address, use G-Stage only. +* @return : success or fault cause code. +*/ +static int riscv_iommu_spa_fetch (RISCVIOMMUState *s,RISCVIOMMUContext *ctx, +IOMMUTLBEntry *iotlb,bool gpa) +{ +dma_addr_t addr, base; +uint64_t satp, gatp, pte; +bool en_s, en_g; +struct { +unsigned char step; +unsigned char levels; +unsigned char ptidxbits; +unsigned char ptesize; + } sc[2 ]; +/* Translation stage phase */ +enum { +S_STAGE =0 , +G_STAGE =1 , + } pass; + + satp =get_field (ctx->satp,RISCV_IOMMU_ATP_MODE_FIELD ); + gatp =get_field (ctx->gatp,RISCV_IOMMU_ATP_MODE_FIELD ); + + en_s = satp !=RISCV_IOMMU_DC_FSC_MODE_BARE && !gpa; + en_g = gatp !=RISCV_IOMMU_DC_IOHGATP_MODE_BARE ; + +/* Early check for MSI address match when IOVA == GPA */ +if (!en_s && (iotlb->perm &IOMMU_WO ) && +riscv_iommu_msi_check (s, ctx, iotlb->iova)) { + iotlb->target_as = &s->trap_as; + iotlb->translated_addr = iotlb->iova; + iotlb->addr_mask = ~TARGET_PAGE_MASK ; +return 0 ; + } + +/* Exit early for pass-through mode. */ +if (!(en_s || en_g)) { + iotlb->translated_addr = iotlb->iova; + iotlb->addr_mask = ~TARGET_PAGE_MASK ; +/* Allow R/W in pass-through mode */ + iotlb->perm =IOMMU_RW ; +return 0 ; + } + +/* S/G translation parameters. */ +for (pass =0 ; pass <2 ; pass++) { + sc[pass].step =0 ; +if (pass ? (s->fctl &RISCV_IOMMU_FCTL_GXL ) : + (ctx->tc &RISCV_IOMMU_DC_TC_SXL )) { +/* 32bit mode for GXL/SXL == 1 */ +switch (pass ? gatp : satp) { +case RISCV_IOMMU_DC_IOHGATP_MODE_BARE : + sc[pass].levels =0 ; + sc[pass].ptidxbits =0 ; + sc[pass].ptesize =0 ; +break ; +case RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4 : +if (!(s->cap & + (pass ?RISCV_IOMMU_CAP_G_SV32 :RISCV_IOMMU_CAP_S_SV32 ))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + sc[pass].levels =2 ; + sc[pass].ptidxbits =10 ; + sc[pass].ptesize =4 ; +break ; +default : +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + }else { +/* 64bit mode for GXL/SXL == 0 */ +switch (pass ? gatp : satp) { +case RISCV_IOMMU_DC_IOHGATP_MODE_BARE : + sc[pass].levels =0 ; + sc[pass].ptidxbits =0 ; + sc[pass].ptesize =0 ; +break ; +case RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4 : +if (!(s->cap & + (pass ?RISCV_IOMMU_CAP_G_SV39 :RISCV_IOMMU_CAP_S_SV39 ))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + sc[pass].levels =3 ; + sc[pass].ptidxbits =9 ; + sc[pass].ptesize =8 ; +break ; +case RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4 : +if (!(s->cap & + (pass ?RISCV_IOMMU_CAP_G_SV48 :RISCV_IOMMU_CAP_S_SV48 ))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + sc[pass].levels =4 ; + sc[pass].ptidxbits =9 ; + sc[pass].ptesize =8 ; +break ; +case RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4 : +if (!(s->cap & + (pass ?RISCV_IOMMU_CAP_G_SV57 :RISCV_IOMMU_CAP_S_SV57 ))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + sc[pass].levels =5 ; + sc[pass].ptidxbits =9 ; + sc[pass].ptesize =8 ; +break ; +default : +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + } + }; + +/* S/G stages translation tables root pointers */ + gatp =PPN_PHYS (get_field (ctx->gatp,RISCV_IOMMU_ATP_PPN_FIELD )); + satp =PPN_PHYS (get_field (ctx->satp,RISCV_IOMMU_ATP_PPN_FIELD )); + addr = (en_s && en_g) ? satp : iotlb->iova; + base = en_g ? gatp : satp; + pass = en_g ?G_STAGE :S_STAGE ; + +do { +const unsigned widened = (pass && !sc[pass].step) ?2 :0 ; +const unsigned va_bits = widened + sc[pass].ptidxbits; +const unsigned va_skip =TARGET_PAGE_BITS + sc[pass].ptidxbits * + (sc[pass].levels -1 - sc[pass].step); +const unsigned idx = (addr >> va_skip) & ((1 << va_bits) -1 ); +const dma_addr_t pte_addr = base + idx * sc[pass].ptesize; +const bool ade = + ctx->tc & (pass ?RISCV_IOMMU_DC_TC_GADE :RISCV_IOMMU_DC_TC_SADE ); + +/* Address range check before first level lookup */ +if (!sc[pass].step) { +const uint64_t va_mask = (1ULL << (va_skip + va_bits)) -1 ; +if ((addr & va_mask) != addr) { +return RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED ; + } + } + +/* Read page table entry */ +if (dma_memory_read (s->target_as, pte_addr, &pte, + sc[pass].ptesize,MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return (iotlb->perm &IOMMU_WO ) ?RISCV_IOMMU_FQ_CAUSE_WR_FAULT + :RISCV_IOMMU_FQ_CAUSE_RD_FAULT ; + } + +if (pass ==S_STAGE ) { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_S_VS_WALKS ); + }else { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_G_WALKS ); + } + +if (sc[pass].ptesize ==4 ) { + pte = (uint64_t )le32_to_cpu (*((uint32_t *)&pte)); + }else { + pte =le64_to_cpu (pte); + } + + sc[pass].step++; +hwaddr ppn = pte >>PTE_PPN_SHIFT ; + +if (!(pte &PTE_V )) { +break ;/* Invalid PTE */ + }else if (!(pte & (PTE_R |PTE_W |PTE_X ))) { + base =PPN_PHYS (ppn);/* Inner PTE, continue walking */ + }else if ((pte & (PTE_R |PTE_W |PTE_X )) ==PTE_W ) { +break ;/* Reserved leaf PTE flags: PTE_W */ + }else if ((pte & (PTE_R |PTE_W |PTE_X )) == (PTE_W |PTE_X )) { +break ;/* Reserved leaf PTE flags: PTE_W + PTE_X */ + }else if (ppn & ((1ULL << (va_skip -TARGET_PAGE_BITS )) -1 )) { +break ;/* Misaligned PPN */ + }else if ((iotlb->perm &IOMMU_RO ) && !(pte &PTE_R )) { +break ;/* Read access check failed */ + }else if ((iotlb->perm &IOMMU_WO ) && !(pte &PTE_W )) { +break ;/* Write access check failed */ + }else if ((iotlb->perm &IOMMU_RO ) && !ade && !(pte &PTE_A )) { +break ;/* Access bit not set */ + }else if ((iotlb->perm &IOMMU_WO ) && !ade && !(pte &PTE_D )) { +break ;/* Dirty bit not set */ + }else { +/* Leaf PTE, translation completed. */ + sc[pass].step = sc[pass].levels; + base =PPN_PHYS (ppn) | (addr & ((1ULL << va_skip) -1 )); +/* Update address mask based on smallest translation granularity */ + iotlb->addr_mask &= (1ULL << va_skip) -1 ; +/* Continue with S-Stage translation? */ +if (pass && sc[0 ].step != sc[0 ].levels) { + pass =S_STAGE ; + addr = iotlb->iova; +continue ; + } +/* Translation phase completed (GPA or SPA) */ + iotlb->translated_addr = base; + iotlb->perm = (pte &PTE_W ) ? ((pte &PTE_R ) ?IOMMU_RW :IOMMU_WO ) + :IOMMU_RO ; + +/* Check MSI GPA address match */ +if (pass ==S_STAGE && (iotlb->perm &IOMMU_WO ) && +riscv_iommu_msi_check (s, ctx, base)) { +/* Trap MSI writes and return GPA address. */ + iotlb->target_as = &s->trap_as; + iotlb->addr_mask = ~TARGET_PAGE_MASK ; +return 0 ; + } + +/* Continue with G-Stage translation? */ +if (!pass && en_g) { + pass =G_STAGE ; + addr = base; + base = gatp; + sc[pass].step =0 ; +continue ; + } + +return 0 ; + } + +if (sc[pass].step == sc[pass].levels) { +break ;/* Can't find leaf PTE */ + } + +/* Continue with G-Stage translation? */ +if (!pass && en_g) { + pass =G_STAGE ; + addr = base; + base = gatp; + sc[pass].step =0 ; + } + }while (1 ); + +return (iotlb->perm &IOMMU_WO ) ? + (pass ?RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS : +RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S ) : + (pass ?RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS : +RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S ); +} + +/* Redirect MSI write for given GPA. */ +static MemTxResult riscv_iommu_msi_write (RISCVIOMMUState *s, +RISCVIOMMUContext *ctx,uint64_t gpa,uint64_t data, +unsigned size,MemTxAttrs attrs) +{ +MemTxResult res; +dma_addr_t addr; +uint64_t intn; +uint32_t n190; +uint64_t pte[2 ]; + +if (!riscv_iommu_msi_check (s, ctx, gpa)) { +return MEMTX_ACCESS_ERROR ; + } + +/* Interrupt File Number */ + intn =_pext_u64 (PPN_DOWN (gpa), ctx->msi_addr_mask); +if (intn >=256 ) { +/* Interrupt file number out of range */ +return MEMTX_ACCESS_ERROR ; + } + +/* fetch MSI PTE */ + addr =PPN_PHYS (get_field (ctx->msiptp,RISCV_IOMMU_DC_MSIPTP_PPN )); + addr = addr | (intn *sizeof (pte)); + res =dma_memory_read (s->target_as, addr, &pte,sizeof (pte), +MEMTXATTRS_UNSPECIFIED ); +if (res !=MEMTX_OK ) { +return res; + } + +le64_to_cpus (&pte[0 ]); +le64_to_cpus (&pte[1 ]); + +if (!(pte[0 ] &RISCV_IOMMU_MSI_PTE_V ) || (pte[0 ] &RISCV_IOMMU_MSI_PTE_C )) { +return MEMTX_ACCESS_ERROR ; + } + +switch (get_field (pte[0 ],RISCV_IOMMU_MSI_PTE_M )) { +case RISCV_IOMMU_MSI_PTE_M_BASIC : +/* MSI Pass-through mode */ + addr =PPN_PHYS (get_field (pte[0 ],RISCV_IOMMU_MSI_PTE_PPN )); + addr = addr | (gpa &TARGET_PAGE_MASK ); + +trace_riscv_iommu_msi (s->parent_obj.id,PCI_BUS_NUM (ctx->devid), +PCI_SLOT (ctx->devid),PCI_FUNC (ctx->devid), + gpa, addr); + +return dma_memory_write (s->target_as, addr, &data, size, attrs); +case RISCV_IOMMU_MSI_PTE_M_MRIF : +/* MRIF mode, continue. */ +break ; +default : +return MEMTX_ACCESS_ERROR ; + } + +/* +* Report an error for interrupt identities exceeding the maximum allowed +* for an IMSIC interrupt file (2047) or destination address is not 32-bit +* aligned. See IOMMU Specification, Chapter 2.3. MSI page tables. +*/ +if ((data >2047 ) || (gpa &3 )) { +return MEMTX_ACCESS_ERROR ; + } + +/* MSI MRIF mode, non atomic pending bit update */ + +/* MRIF pending bit address */ + addr =get_field (pte[0 ],RISCV_IOMMU_MSI_PTE_MRIF_ADDR ) <<9 ; + addr = addr | ((data &0x7c0 ) >>3 ); + +trace_riscv_iommu_msi (s->parent_obj.id,PCI_BUS_NUM (ctx->devid), +PCI_SLOT (ctx->devid),PCI_FUNC (ctx->devid), + gpa, addr); + +/* MRIF pending bit mask */ + data =1ULL << (data &0x03f ); + res =dma_memory_read (s->target_as, addr, &intn,sizeof (intn), attrs); +if (res !=MEMTX_OK ) { +return res; + } + intn = intn | data; + res =dma_memory_write (s->target_as, addr, &intn,sizeof (intn), attrs); +if (res !=MEMTX_OK ) { +return res; + } + +/* Get MRIF enable bits */ + addr = addr +sizeof (intn); + res =dma_memory_read (s->target_as, addr, &intn,sizeof (intn), attrs); +if (res !=MEMTX_OK ) { +return res; + } +if (!(intn & data)) { +/* notification disabled, MRIF update completed. */ +return MEMTX_OK ; + } + +/* Send notification message */ + addr =PPN_PHYS (get_field (pte[1 ],RISCV_IOMMU_MSI_MRIF_NPPN )); + n190 =get_field (pte[1 ],RISCV_IOMMU_MSI_MRIF_NID ) | + (get_field (pte[1 ],RISCV_IOMMU_MSI_MRIF_NID_MSB ) <<10 ); + + res =dma_memory_write (s->target_as, addr, &n190,sizeof (n190), attrs); +if (res !=MEMTX_OK ) { +return res; + } + +return MEMTX_OK ; +} + +/* +* Device Context format. +* +* @s : IOMMU Device State +* @return : 0: extended (64 bytes) | 1: base (32 bytes) +*/ +static int riscv_iommu_dc_is_base (RISCVIOMMUState *s) +{ +return !(s->cap &RISCV_IOMMU_CAP_MSI_FLAT ); +} + +/* +* RISC-V IOMMU Device Context Loopkup - Device Directory Tree Walk +* +* @s : IOMMU Device State +* @ctx : Device Translation Context with devid and pasid set. +* @return : success or fault code. +*/ +static int riscv_iommu_ctx_fetch (RISCVIOMMUState *s,RISCVIOMMUContext *ctx) +{ +const uint64_t ddtp = s->ddtp; +unsigned mode =get_field (ddtp,RISCV_IOMMU_DDTP_MODE ); +dma_addr_t addr =PPN_PHYS (get_field (ddtp,RISCV_IOMMU_DDTP_PPN )); +struct riscv_iommu_dc dc; +const int dc_fmt =riscv_iommu_dc_is_base (s); +const size_t dc_len =sizeof (dc) >> dc_fmt; +unsigned depth; +uint64_t de; + +switch (mode) { +case RISCV_IOMMU_DDTP_MODE_OFF : +return RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED ; + +case RISCV_IOMMU_DDTP_MODE_BARE : +/* mock up pass-through translation context */ + ctx->gatp =set_field (0 ,RISCV_IOMMU_ATP_MODE_FIELD , +RISCV_IOMMU_DC_IOHGATP_MODE_BARE ); + ctx->satp =set_field (0 ,RISCV_IOMMU_ATP_MODE_FIELD , +RISCV_IOMMU_DC_FSC_MODE_BARE ); + ctx->tc =RISCV_IOMMU_DC_TC_EN_ATS |RISCV_IOMMU_DC_TC_V ; + ctx->ta =0 ; + ctx->msiptp =0 ; +return 0 ; + +case RISCV_IOMMU_DDTP_MODE_1LVL : + depth =0 ; +break ; + +case RISCV_IOMMU_DDTP_MODE_2LVL : + depth =1 ; +break ; + +case RISCV_IOMMU_DDTP_MODE_3LVL : + depth =2 ; +break ; + +default : +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + +/* +* Check supported device id width (in bits). +* See IOMMU Specification, Chapter 6. Software guidelines. +* - if extended device-context format is used: +* 1LVL: 6, 2LVL: 15, 3LVL: 24 +* - if base device-context format is used: +* 1LVL: 7, 2LVL: 16, 3LVL: 24 +*/ +if (ctx->devid >= (1 << (depth *9 +6 + (dc_fmt && depth !=2 )))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ; + } + +/* Device directory tree walk */ +for (; depth-- >0 ; ) { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_DD_WALK ); + +/* +* Select device id index bits based on device directory tree level +* and device context format. +* See IOMMU Specification, Chapter 2. Data Structures. +* - if extended device-context format is used: +* device index: [23:15][14:6][5:0] +* - if base device-context format is used: +* device index: [23:16][15:7][6:0] +*/ +const int split = depth *9 +6 + dc_fmt; + addr |= ((ctx->devid >> split) <<3 ) & ~TARGET_PAGE_MASK ; +if (dma_memory_read (s->target_as, addr, &de,sizeof (de), +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT ; + } +le64_to_cpus (&de); +if (!(de &RISCV_IOMMU_DDTE_VALID )) { +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ;/* invalid directory entry */ + } +if (de & ~(RISCV_IOMMU_DDTE_PPN |RISCV_IOMMU_DDTE_VALID )) { +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ;/* reserved bits set. */ + } + addr =PPN_PHYS (get_field (de,RISCV_IOMMU_DDTE_PPN )); + } + +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_DD_WALK ); + +/* index into device context entry page */ + addr |= (ctx->devid * dc_len) & ~TARGET_PAGE_MASK ; + +memset (&dc,0 ,sizeof (dc)); +if (dma_memory_read (s->target_as, addr, &dc, dc_len, +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT ; + } + +/* Set translation context. */ + ctx->tc =le64_to_cpu (dc.tc); + ctx->gatp =le64_to_cpu (dc.iohgatp); + ctx->satp =le64_to_cpu (dc.fsc); + ctx->ta =le64_to_cpu (dc.ta); + ctx->msiptp =le64_to_cpu (dc.msiptp); + ctx->msi_addr_mask =le64_to_cpu (dc.msi_addr_mask); + ctx->msi_addr_pattern =le64_to_cpu (dc.msi_addr_pattern); + +if (!(ctx->tc &RISCV_IOMMU_DC_TC_V )) { +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ; + } + +/* FSC field checks */ + mode =get_field (ctx->satp,RISCV_IOMMU_DC_FSC_MODE ); + addr =PPN_PHYS (get_field (ctx->satp,RISCV_IOMMU_DC_FSC_PPN )); + +if (mode ==RISCV_IOMMU_DC_FSC_MODE_BARE ) { +/* No S-Stage translation, done. */ +return 0 ; + } + +if (!(ctx->tc &RISCV_IOMMU_DC_TC_PDTV )) { +if (ctx->pasid !=RISCV_IOMMU_NOPASID ) { +/* PASID is disabled */ +return RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED ; + } +if (mode >RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57 ) { +/* Invalid translation mode */ +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ; + } +return 0 ; + } + +if (ctx->pasid ==RISCV_IOMMU_NOPASID ) { +if (!(ctx->tc &RISCV_IOMMU_DC_TC_DPE )) { +/* No default PASID enabled, set BARE mode */ + ctx->satp =0ULL ; +return 0 ; + }else { +/* Use default PASID #0 */ + ctx->pasid =0 ; + } + } + +/* FSC.TC.PDTV enabled */ +if (mode >RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20 ) { +/* Invalid PDTP.MODE */ +return RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED ; + } + +for (depth = mode -RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8 ; depth-- >0 ; ) { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_PD_WALK ); + +/* +* Select process id index bits based on process directory tree +* level. See IOMMU Specification, 2.2. Process-Directory-Table. +*/ +const int split = depth *9 +8 ; + addr |= ((ctx->pasid >> split) <<3 ) & ~TARGET_PAGE_MASK ; +if (dma_memory_read (s->target_as, addr, &de,sizeof (de), +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT ; + } +le64_to_cpus (&de); +if (!(de &RISCV_IOMMU_PC_TA_V )) { +return RISCV_IOMMU_FQ_CAUSE_PDT_INVALID ; + } + addr =PPN_PHYS (get_field (de,RISCV_IOMMU_PC_FSC_PPN )); + } + +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_PD_WALK ); + +/* Leaf entry in PDT */ + addr |= (ctx->pasid <<4 ) & ~TARGET_PAGE_MASK ; +if (dma_memory_read (s->target_as, addr, &dc.ta,sizeof (uint64_t ) *2 , +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT ; + } + +/* Use FSC and TA from process directory entry. */ + ctx->ta =le64_to_cpu (dc.ta); + ctx->satp =le64_to_cpu (dc.fsc); + +return 0 ; +} + +/* Translation Context cache support */ +static gboolean __ctx_equal (gconstpointer v1,gconstpointer v2) +{ +RISCVIOMMUContext *c1 = (RISCVIOMMUContext *) v1; +RISCVIOMMUContext *c2 = (RISCVIOMMUContext *) v2; +return c1->devid == c2->devid && c1->pasid == c2->pasid; +} + +static guint __ctx_hash (gconstpointer v) +{ +RISCVIOMMUContext *ctx = (RISCVIOMMUContext *) v; +/* Generate simple hash of (pasid, devid), assuming 24-bit wide devid */ +return (guint)(ctx->devid) + ((guint)(ctx->pasid) <<24 ); +} + +static void __ctx_inval_devid_pasid (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUContext *ctx = (RISCVIOMMUContext *) value; +RISCVIOMMUContext *arg = (RISCVIOMMUContext *) data; +if (ctx->tc &RISCV_IOMMU_DC_TC_V && + ctx->devid == arg->devid && + ctx->pasid == arg->pasid) { + ctx->tc &= ~RISCV_IOMMU_DC_TC_V ; + } +} + +static void __ctx_inval_devid (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUContext *ctx = (RISCVIOMMUContext *) value; +RISCVIOMMUContext *arg = (RISCVIOMMUContext *) data; +if (ctx->tc &RISCV_IOMMU_DC_TC_V && + ctx->devid == arg->devid) { + ctx->tc &= ~RISCV_IOMMU_DC_TC_V ; + } +} + +static void __ctx_inval_all (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUContext *ctx = (RISCVIOMMUContext *) value; +if (ctx->tc &RISCV_IOMMU_DC_TC_V ) { + ctx->tc &= ~RISCV_IOMMU_DC_TC_V ; + } +} + +static void riscv_iommu_ctx_inval (RISCVIOMMUState *s,GHFunc func, +uint32_t devid,uint32_t pasid) +{ +GHashTable *ctx_cache; +RISCVIOMMUContext key = { + .devid = devid, + .pasid = pasid, + }; + ctx_cache =g_hash_table_ref (s->ctx_cache); +g_hash_table_foreach (ctx_cache, func, &key); +g_hash_table_unref (ctx_cache); +} + +/* Find or allocate translation context for a given {device_id, process_id} */ +static RISCVIOMMUContext *riscv_iommu_ctx (RISCVIOMMUState *s, +unsigned devid,unsigned pasid,void **ref) +{ +GHashTable *ctx_cache; +RISCVIOMMUContext *ctx; +RISCVIOMMUContext key = { + .devid = devid, + .pasid = pasid, + }; + + ctx_cache =g_hash_table_ref (s->ctx_cache); + ctx =g_hash_table_lookup (ctx_cache, &key); + +if (ctx && (ctx->tc &RISCV_IOMMU_DC_TC_V )) { + *ref = ctx_cache; +return ctx; + } + +if (g_hash_table_size (s->ctx_cache) >=LIMIT_CACHE_CTX ) { + ctx_cache =g_hash_table_new_full (__ctx_hash, __ctx_equal, + g_free,NULL ); +g_hash_table_unref (qatomic_xchg (&s->ctx_cache, ctx_cache)); + } + + ctx =g_new0 (RISCVIOMMUContext,1 ); + ctx->devid = devid; + ctx->pasid = pasid; + +int fault =riscv_iommu_ctx_fetch (s, ctx); +if (!fault) { +g_hash_table_add (ctx_cache, ctx); + *ref = ctx_cache; +return ctx; + } + +g_hash_table_unref (ctx_cache); + *ref =NULL ; + +if (!(ctx->tc &RISCV_IOMMU_DC_TC_DTF )) { +struct riscv_iommu_fq_record ev = {0 }; + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_CAUSE , fault); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_TTYPE , +RISCV_IOMMU_FQ_TTYPE_UADDR_RD ); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_DID , devid); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_PID , pasid); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_PV , !!pasid); +riscv_iommu_fault (s, &ev); + } + +g_free (ctx); +return NULL ; +} + +static void riscv_iommu_ctx_put (RISCVIOMMUState *s,void *ref) +{ +if (ref) { +g_hash_table_unref ((GHashTable *)ref); + } +} + +/* Find or allocate address space for a given device */ +static AddressSpace *riscv_iommu_space (RISCVIOMMUState *s,uint32_t devid) +{ +RISCVIOMMUSpace *as; + +/* FIXME: PCIe bus remapping for attached endpoints. */ + devid |= s->bus <<8 ; + +qemu_mutex_lock (&s->core_lock); + QLIST_FOREACH(as, &s->spaces, list) { +if (as->devid == devid) { +break ; + } + } +qemu_mutex_unlock (&s->core_lock); + +if (as ==NULL ) { +char name[64 ]; + as =g_new0 (RISCVIOMMUSpace,1 ); + + as->iommu = s; + as->devid = devid; + +snprintf (name,sizeof (name),"riscv-iommu-%04x:%02x.%d-iova" , +PCI_BUS_NUM (as->devid),PCI_SLOT (as->devid),PCI_FUNC (as->devid)); + +/* IOVA address space, untranslated addresses */ +memory_region_init_iommu (&as->iova_mr,sizeof (as->iova_mr), +TYPE_RISCV_IOMMU_MEMORY_REGION , +OBJECT (as), name,UINT64_MAX ); +address_space_init (&as->iova_as,MEMORY_REGION (&as->iova_mr), +TYPE_RISCV_IOMMU_PCI ); + +qemu_mutex_lock (&s->core_lock); +QLIST_INSERT_HEAD (&s->spaces, as, list); +qemu_mutex_unlock (&s->core_lock); + +trace_riscv_iommu_new (s->parent_obj.id,PCI_BUS_NUM (as->devid), +PCI_SLOT (as->devid),PCI_FUNC (as->devid)); + } +return &as->iova_as; +} + +/* Translation Object cache support */ +static gboolean __iot_equal (gconstpointer v1,gconstpointer v2) +{ +RISCVIOMMUEntry *t1 = (RISCVIOMMUEntry *) v1; +RISCVIOMMUEntry *t2 = (RISCVIOMMUEntry *) v2; +return t1->gscid == t2->gscid && t1->pscid == t2->pscid && + t1->iova == t2->iova; +} + +static guint __iot_hash (gconstpointer v) +{ +RISCVIOMMUEntry *t = (RISCVIOMMUEntry *) v; +return (guint )t->iova; +} + +/* GV: 1 PSCV: 1 AV: 1 */ +static void __iot_inval_pscid_iova (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; +RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; +if (iot->gscid == arg->gscid && + iot->pscid == arg->pscid && + iot->iova == arg->iova) { + iot->perm =0 ; + } +} + +/* GV: 1 PSCV: 1 AV: 0 */ +static void __iot_inval_pscid (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; +RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; +if (iot->gscid == arg->gscid && + iot->pscid == arg->pscid) { + iot->perm =0 ; + } +} + +/* GV: 1 GVMA: 1 */ +static void __iot_inval_gscid_gpa (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; +RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; +if (iot->gscid == arg->gscid) { +/* simplified cache, no GPA matching */ + iot->perm =0 ; + } +} + +/* GV: 1 GVMA: 0 */ +static void __iot_inval_gscid (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; +RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; +if (iot->gscid == arg->gscid) { + iot->perm =0 ; + } +} + +/* GV: 0 */ +static void __iot_inval_all (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; + iot->perm =0 ; +} + +/* caller should keep ref-count for iot_cache object */ +static RISCVIOMMUEntry *riscv_iommu_iot_lookup (RISCVIOMMUContext *ctx, +GHashTable *iot_cache,hwaddr iova) +{ +RISCVIOMMUEntry key = { + .gscid =get_field (ctx->gatp,RISCV_IOMMU_DC_IOHGATP_GSCID ), + .pscid =get_field (ctx->ta,RISCV_IOMMU_DC_TA_PSCID ), + .iova =PPN_DOWN (iova), + }; +return g_hash_table_lookup (iot_cache, &key); +} + +/* caller should keep ref-count for iot_cache object */ +static void riscv_iommu_iot_update (RISCVIOMMUState *s, +GHashTable *iot_cache,RISCVIOMMUEntry *iot) +{ +if (!s->iot_limit) { +return ; + } + +if (g_hash_table_size (s->iot_cache) >= s->iot_limit) { + iot_cache =g_hash_table_new_full (__iot_hash, __iot_equal, + g_free,NULL ); +g_hash_table_unref (qatomic_xchg (&s->iot_cache, iot_cache)); + } +g_hash_table_add (iot_cache, iot); +} + +static void riscv_iommu_iot_inval (RISCVIOMMUState *s,GHFunc func, +uint32_t gscid,uint32_t pscid,hwaddr iova) +{ +GHashTable *iot_cache; +RISCVIOMMUEntry key = { + .gscid = gscid, + .pscid = pscid, + .iova =PPN_DOWN (iova), + }; + + iot_cache =g_hash_table_ref (s->iot_cache); +g_hash_table_foreach (iot_cache, func, &key); +g_hash_table_unref (iot_cache); +} + +static int riscv_iommu_translate (RISCVIOMMUState *s,RISCVIOMMUContext *ctx, +IOMMUTLBEntry *iotlb,bool enable_cache) +{ +RISCVIOMMUEntry *iot; +IOMMUAccessFlags perm; +bool enable_faults; +bool enable_pasid; +bool enable_pri; +GHashTable *iot_cache; +int fault; + +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_URQ ); + + iot_cache =g_hash_table_ref (s->iot_cache); + + enable_faults = !(ctx->tc &RISCV_IOMMU_DC_TC_DTF ); +/* +* TC[32] is reserved for custom extensions, used here to temporarily +* enable automatic page-request generation for ATS queries. +*/ + enable_pri = (iotlb->perm ==IOMMU_NONE ) && (ctx->tc &BIT_ULL (32 )); + enable_pasid = (ctx->tc &RISCV_IOMMU_DC_TC_PDTV ); + +/* Check for ATS request. */ +if (iotlb->perm ==IOMMU_NONE ) { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_ATS_RQ ); +/* Check if ATS is disabled. */ +if (!(ctx->tc &RISCV_IOMMU_DC_TC_EN_ATS )) { + enable_pri = false; + fault =RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED ; + goto done; + } +trace_riscv_iommu_ats (s->parent_obj.id,PCI_BUS_NUM (ctx->devid), +PCI_SLOT (ctx->devid),PCI_FUNC (ctx->devid), iotlb->iova); + } + + iot =riscv_iommu_iot_lookup (ctx, iot_cache, iotlb->iova); + perm = iot ? iot->perm :IOMMU_NONE ; +if (perm !=IOMMU_NONE ) { + iotlb->translated_addr =PPN_PHYS (iot->phys); + iotlb->addr_mask = ~TARGET_PAGE_MASK ; + iotlb->perm = perm; + fault =0 ; + goto done; + } + +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_TLB_MISS ); + +/* Translate using device directory / page table information. */ + fault =riscv_iommu_spa_fetch (s, ctx, iotlb, false); + +if (!fault && iotlb->target_as == &s->trap_as) { +/* Do not cache trapped MSI translations */ + goto done; + } + +if (!fault && iotlb->translated_addr != iotlb->iova && enable_cache) { + iot =g_new0 (RISCVIOMMUEntry,1 ); + iot->iova =PPN_DOWN (iotlb->iova); + iot->phys =PPN_DOWN (iotlb->translated_addr); + iot->gscid =get_field (ctx->gatp,RISCV_IOMMU_DC_IOHGATP_GSCID ); + iot->pscid =get_field (ctx->ta,RISCV_IOMMU_DC_TA_PSCID ); + iot->perm = iotlb->perm; +riscv_iommu_iot_update (s, iot_cache, iot); + } + +done: +g_hash_table_unref (iot_cache); + +if (enable_pri && fault) { +struct riscv_iommu_pq_record pr = {0 }; +if (enable_pasid) { + pr.hdr =set_field (RISCV_IOMMU_PREQ_HDR_PV , +RISCV_IOMMU_PREQ_HDR_PID , ctx->pasid); + } + pr.hdr =set_field (pr.hdr,RISCV_IOMMU_PREQ_HDR_DID , ctx->devid); + pr.payload = (iotlb->iova &TARGET_PAGE_MASK ) |RISCV_IOMMU_PREQ_PAYLOAD_M ; +riscv_iommu_pri (s, &pr); +return fault; + } + +if (enable_faults && fault) { +struct riscv_iommu_fq_record ev; +const unsigned ttype = + (iotlb->perm &IOMMU_RW ) ?RISCV_IOMMU_FQ_TTYPE_UADDR_WR : + ((iotlb->perm &IOMMU_RO ) ?RISCV_IOMMU_FQ_TTYPE_UADDR_RD : +RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ ); + ev.hdr =set_field (0 ,RISCV_IOMMU_FQ_HDR_CAUSE , fault); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_TTYPE , ttype); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_PV , enable_pasid); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_PID , ctx->pasid); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_DID , ctx->devid); + ev.iotval = iotlb->iova; + ev.iotval2 = iotlb->translated_addr; + ev._reserved =0 ; +riscv_iommu_fault (s, &ev); +return fault; + } + +return 0 ; +} + +/* IOMMU Command Interface */ +static MemTxResult riscv_iommu_iofence (RISCVIOMMUState *s,bool notify, +uint64_t addr,uint32_t data) +{ +/* +* ATS processing in this implementation of the IOMMU is synchronous, +* no need to wait for completions here. +*/ +if (!notify) { +return MEMTX_OK ; + } + +return dma_memory_write (s->target_as, addr, &data,sizeof (data), +MEMTXATTRS_UNSPECIFIED ); +} + +static void riscv_iommu_ats (RISCVIOMMUState *s, +struct riscv_iommu_command *cmd,IOMMUNotifierFlag flag, +IOMMUAccessFlags perm, +void (*trace_fn)(const char *id)) +{ +RISCVIOMMUSpace *as =NULL ; +IOMMUNotifier *n; +IOMMUTLBEvent event; +uint32_t pasid; +uint32_t devid; +const bool pv = cmd->dword0 &RISCV_IOMMU_CMD_ATS_PV ; + +if (cmd->dword0 &RISCV_IOMMU_CMD_ATS_DSV ) { +/* Use device segment and requester id */ + devid =get_field (cmd->dword0, +RISCV_IOMMU_CMD_ATS_DSEG |RISCV_IOMMU_CMD_ATS_RID ); + }else { + devid =get_field (cmd->dword0,RISCV_IOMMU_CMD_ATS_RID ); + } + + pasid =get_field (cmd->dword0,RISCV_IOMMU_CMD_ATS_PID ); + +qemu_mutex_lock (&s->core_lock); + QLIST_FOREACH(as, &s->spaces, list) { +if (as->devid == devid) { +break ; + } + } +qemu_mutex_unlock (&s->core_lock); + +if (!as || !as->notifier) { +return ; + } + + event.type = flag; + event.entry.perm = perm; + event.entry.target_as = s->target_as; + + IOMMU_NOTIFIER_FOREACH(n, &as->iova_mr) { +if (!pv || n->iommu_idx == pasid) { + event.entry.iova = n->start; + event.entry.addr_mask = n->end - n->start; +trace_fn (as->iova_mr.parent_obj.name); +memory_region_notify_iommu_one (n, &event); + } + } +} + +static void riscv_iommu_ats_inval (RISCVIOMMUState *s, +struct riscv_iommu_command *cmd) +{ +return riscv_iommu_ats (s, cmd,IOMMU_NOTIFIER_DEVIOTLB_UNMAP ,IOMMU_NONE , + trace_riscv_iommu_ats_inval); +} + +static void riscv_iommu_ats_prgr (RISCVIOMMUState *s, +struct riscv_iommu_command *cmd) +{ +unsigned resp_code =get_field (cmd->dword1,RISCV_IOMMU_CMD_ATS_PRGR_RESP_CODE ); +/* Using the access flag to carry response code information */ +IOMMUAccessFlags perm = resp_code ?IOMMU_NONE :IOMMU_RW ; +return riscv_iommu_ats (s, cmd,IOMMU_NOTIFIER_MAP , perm, + trace_riscv_iommu_ats_prgr); +} + +static void riscv_iommu_process_ddtp (RISCVIOMMUState *s) +{ +uint64_t old_ddtp = s->ddtp; +uint64_t new_ddtp =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_DDTP ); +unsigned new_mode =get_field (new_ddtp,RISCV_IOMMU_DDTP_MODE ); +unsigned old_mode =get_field (old_ddtp,RISCV_IOMMU_DDTP_MODE ); +bool ok = false; + +/* +* Check for allowed DDTP.MODE transitions: +* {OFF, BARE} -> {OFF, BARE, 1LVL, 2LVL, 3LVL} +* {1LVL, 2LVL, 3LVL} -> {OFF, BARE} +*/ +if (new_mode == old_mode || + new_mode ==RISCV_IOMMU_DDTP_MODE_OFF || + new_mode ==RISCV_IOMMU_DDTP_MODE_BARE ) { + ok = true; + }else if (new_mode ==RISCV_IOMMU_DDTP_MODE_1LVL || + new_mode ==RISCV_IOMMU_DDTP_MODE_2LVL || + new_mode ==RISCV_IOMMU_DDTP_MODE_3LVL ) { + ok = old_mode ==RISCV_IOMMU_DDTP_MODE_OFF || + old_mode ==RISCV_IOMMU_DDTP_MODE_BARE ; + } + +if (ok) { +/* clear reserved and busy bits, report back sanitized version */ + new_ddtp =set_field (new_ddtp &RISCV_IOMMU_DDTP_PPN , +RISCV_IOMMU_DDTP_MODE , new_mode); + }else { + new_ddtp = old_ddtp; + } + s->ddtp = new_ddtp; + +riscv_iommu_reg_set64 (s,RISCV_IOMMU_REG_DDTP , new_ddtp); +} + +/* Command function and opcode field. */ +#define RISCV_IOMMU_CMD (func, op) (((func) << 7) | (op)) + +static void riscv_iommu_process_cq_tail (RISCVIOMMUState *s) +{ +struct riscv_iommu_command cmd; +MemTxResult res; +dma_addr_t addr; +uint32_t tail, head, ctrl; +GHFunc func; + + ctrl =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_CQCSR ); + tail =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_CQT ) & s->cq_mask; + head =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_CQH ) & s->cq_mask; + +/* Check for pending error or queue processing disabled */ +if (!(ctrl &RISCV_IOMMU_CQCSR_CQON ) || + !!(ctrl & (RISCV_IOMMU_CQCSR_CMD_ILL |RISCV_IOMMU_CQCSR_CQMF ))) { +return ; + } + +while (tail != head) { + addr = s->cq_addr + head *sizeof (cmd); + res =dma_memory_read (s->target_as, addr, &cmd,sizeof (cmd), +MEMTXATTRS_UNSPECIFIED ); + +if (res !=MEMTX_OK ) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_CQCSR ,RISCV_IOMMU_CQCSR_CQMF ,0 ); + goto fault; + } + +trace_riscv_iommu_cmd (s->parent_obj.id, cmd.dword0, cmd.dword1); + +switch (get_field (cmd.dword0,RISCV_IOMMU_CMD_OPCODE |RISCV_IOMMU_CMD_FUNC )) { +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IOFENCE_FUNC_C , +RISCV_IOMMU_CMD_IOFENCE_OPCODE ): + res =riscv_iommu_iofence (s, cmd.dword0 &RISCV_IOMMU_CMD_IOFENCE_AV , + cmd.dword1,get_field (cmd.dword0,RISCV_IOMMU_CMD_IOFENCE_DATA )); + +if (res !=MEMTX_OK ) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_CQCSR , +RISCV_IOMMU_CQCSR_CQMF ,0 ); + goto fault; + } +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA , +RISCV_IOMMU_CMD_IOTINVAL_OPCODE ): +if (cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_PSCV ) { +/* illegal command arguments IOTINVAL.GVMA & PSCV == 1 */ + goto cmd_ill; + }else if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_GV )) { +/* invalidate all cache mappings */ + func = __iot_inval_all; + }else if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_AV )) { +/* invalidate cache matching GSCID */ + func = __iot_inval_gscid; + }else { +/* invalidate cache matching GSCID and ADDR (GPA) */ + func = __iot_inval_gscid_gpa; + } +riscv_iommu_iot_inval (s, func, +get_field (cmd.dword0,RISCV_IOMMU_CMD_IOTINVAL_GSCID ),0 , + cmd.dword1 &TARGET_PAGE_MASK ); +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA , +RISCV_IOMMU_CMD_IOTINVAL_OPCODE ): +if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_GV )) { +/* invalidate all cache mappings, simplified model */ + func = __iot_inval_all; + }else if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_PSCV )) { +/* invalidate cache matching GSCID, simplified model */ + func = __iot_inval_gscid; + }else if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_AV )) { +/* invalidate cache matching GSCID and PSCID */ + func = __iot_inval_pscid; + }else { +/* invalidate cache matching GSCID and PSCID and ADDR (IOVA) */ + func = __iot_inval_pscid_iova; + } +riscv_iommu_iot_inval (s, func, +get_field (cmd.dword0,RISCV_IOMMU_CMD_IOTINVAL_GSCID ), +get_field (cmd.dword0,RISCV_IOMMU_CMD_IOTINVAL_PSCID ), + cmd.dword1 &TARGET_PAGE_MASK ); +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT , +RISCV_IOMMU_CMD_IODIR_OPCODE ): +if (!(cmd.dword0 &RISCV_IOMMU_CMD_IODIR_DV )) { +/* invalidate all device context cache mappings */ + func = __ctx_inval_all; + }else { +/* invalidate all device context matching DID */ + func = __ctx_inval_devid; + } +riscv_iommu_ctx_inval (s, func, +get_field (cmd.dword0,RISCV_IOMMU_CMD_IODIR_DID ),0 ); +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT , +RISCV_IOMMU_CMD_IODIR_OPCODE ): +if (!(cmd.dword0 &RISCV_IOMMU_CMD_IODIR_DV )) { +/* illegal command arguments IODIR_PDT & DV == 0 */ + goto cmd_ill; + }else { + func = __ctx_inval_devid_pasid; + } +riscv_iommu_ctx_inval (s, func, +get_field (cmd.dword0,RISCV_IOMMU_CMD_IODIR_DID ), +get_field (cmd.dword0,RISCV_IOMMU_CMD_IODIR_PID )); +break ; + +/* ATS commands */ +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_ATS_FUNC_INVAL , +RISCV_IOMMU_CMD_ATS_OPCODE ): +riscv_iommu_ats_inval (s, &cmd); +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_ATS_FUNC_PRGR , +RISCV_IOMMU_CMD_ATS_OPCODE ): +riscv_iommu_ats_prgr (s, &cmd); +break ; + +default : + cmd_ill: +/* Invalid instruction, do not advance instruction index. */ +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_CQCSR , +RISCV_IOMMU_CQCSR_CMD_ILL ,0 ); + goto fault; + } + +/* Advance and update head pointer after command completes. */ + head = (head +1 ) & s->cq_mask; +riscv_iommu_reg_set32 (s,RISCV_IOMMU_REG_CQH , head); + } +return ; + +fault: +if (ctrl &RISCV_IOMMU_CQCSR_CIE ) { +riscv_iommu_notify (s,RISCV_IOMMU_INTR_CQ ); + } +} + +static void riscv_iommu_process_cq_control (RISCVIOMMUState *s) +{ +uint64_t base; +uint32_t ctrl_set =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_CQCSR ); +uint32_t ctrl_clr; +bool enable = !!(ctrl_set &RISCV_IOMMU_CQCSR_CQEN ); +bool active = !!(ctrl_set &RISCV_IOMMU_CQCSR_CQON ); + +if (enable && !active) { + base =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_CQB ); + s->cq_mask = (2ULL <<get_field (base,RISCV_IOMMU_CQB_LOG2SZ )) -1 ; + s->cq_addr =PPN_PHYS (get_field (base,RISCV_IOMMU_CQB_PPN )); +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_CQT ], ~s->cq_mask); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_CQH ],0 ); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_CQT ],0 ); + ctrl_set =RISCV_IOMMU_CQCSR_CQON ; + ctrl_clr =RISCV_IOMMU_CQCSR_BUSY |RISCV_IOMMU_CQCSR_CQMF | +RISCV_IOMMU_CQCSR_CMD_ILL |RISCV_IOMMU_CQCSR_CMD_TO ; + }else if (!enable && active) { +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_CQT ], ~0 ); + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_CQCSR_BUSY |RISCV_IOMMU_CQCSR_CQON ; + }else { + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_CQCSR_BUSY ; + } + +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_CQCSR , ctrl_set, ctrl_clr); +} + +static void riscv_iommu_process_fq_control (RISCVIOMMUState *s) +{ +uint64_t base; +uint32_t ctrl_set =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_FQCSR ); +uint32_t ctrl_clr; +bool enable = !!(ctrl_set &RISCV_IOMMU_FQCSR_FQEN ); +bool active = !!(ctrl_set &RISCV_IOMMU_FQCSR_FQON ); + +if (enable && !active) { + base =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_FQB ); + s->fq_mask = (2ULL <<get_field (base,RISCV_IOMMU_FQB_LOG2SZ )) -1 ; + s->fq_addr =PPN_PHYS (get_field (base,RISCV_IOMMU_FQB_PPN )); +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_FQH ], ~s->fq_mask); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_FQH ],0 ); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_FQT ],0 ); + ctrl_set =RISCV_IOMMU_FQCSR_FQON ; + ctrl_clr =RISCV_IOMMU_FQCSR_BUSY |RISCV_IOMMU_FQCSR_FQMF | +RISCV_IOMMU_FQCSR_FQOF ; + }else if (!enable && active) { +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_FQH ], ~0 ); + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_FQCSR_BUSY |RISCV_IOMMU_FQCSR_FQON ; + }else { + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_FQCSR_BUSY ; + } + +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_FQCSR , ctrl_set, ctrl_clr); +} + +static void riscv_iommu_process_pq_control (RISCVIOMMUState *s) +{ +uint64_t base; +uint32_t ctrl_set =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_PQCSR ); +uint32_t ctrl_clr; +bool enable = !!(ctrl_set &RISCV_IOMMU_PQCSR_PQEN ); +bool active = !!(ctrl_set &RISCV_IOMMU_PQCSR_PQON ); + +if (enable && !active) { + base =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_PQB ); + s->pq_mask = (2ULL <<get_field (base,RISCV_IOMMU_PQB_LOG2SZ )) -1 ; + s->pq_addr =PPN_PHYS (get_field (base,RISCV_IOMMU_PQB_PPN )); +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_PQH ], ~s->pq_mask); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_PQH ],0 ); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_PQT ],0 ); + ctrl_set =RISCV_IOMMU_PQCSR_PQON ; + ctrl_clr =RISCV_IOMMU_PQCSR_BUSY |RISCV_IOMMU_PQCSR_PQMF | +RISCV_IOMMU_PQCSR_PQOF ; + }else if (!enable && active) { +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_PQH ], ~0 ); + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_PQCSR_BUSY |RISCV_IOMMU_PQCSR_PQON ; + }else { + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_PQCSR_BUSY ; + } + +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_PQCSR , ctrl_set, ctrl_clr); +} + +static void riscv_iommu_process_dbg (RISCVIOMMUState *s) +{ +uint64_t iova =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_TR_REQ_IOVA ); +uint64_t ctrl =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_TR_REQ_CTL ); +unsigned devid =get_field (ctrl,RISCV_IOMMU_TR_REQ_CTL_DID ); +unsigned pid =get_field (ctrl,RISCV_IOMMU_TR_REQ_CTL_PID ); +RISCVIOMMUContext *ctx; +void *ref; + +if (!(ctrl &RISCV_IOMMU_TR_REQ_CTL_GO_BUSY )) { +return ; + } + + ctx =riscv_iommu_ctx (s, devid, pid, &ref); +if (ctx ==NULL ) { +riscv_iommu_reg_set64 (s,RISCV_IOMMU_REG_TR_RESPONSE , +RISCV_IOMMU_TR_RESPONSE_FAULT | (RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED <<10 )); + }else { +IOMMUTLBEntry iotlb = { + .iova = iova, + .perm =IOMMU_NONE , + .addr_mask = ~0 , + .target_as =NULL , + }; +int fault =riscv_iommu_translate (s, ctx, &iotlb, false); +if (fault) { + iova =RISCV_IOMMU_TR_RESPONSE_FAULT | (((uint64_t ) fault) <<10 ); + }else { + iova = ((iotlb.translated_addr & ~iotlb.addr_mask) >>2 ) & +RISCV_IOMMU_TR_RESPONSE_PPN ; + } +riscv_iommu_reg_set64 (s,RISCV_IOMMU_REG_TR_RESPONSE , iova); + } + +riscv_iommu_reg_mod64 (s,RISCV_IOMMU_REG_TR_REQ_CTL ,0 , +RISCV_IOMMU_TR_REQ_CTL_GO_BUSY ); +riscv_iommu_ctx_put (s, ref); +} + +/* Core IOMMU execution activation */ +enum { +RISCV_IOMMU_EXEC_DDTP , +RISCV_IOMMU_EXEC_CQCSR , +RISCV_IOMMU_EXEC_CQT , +RISCV_IOMMU_EXEC_FQCSR , +RISCV_IOMMU_EXEC_FQH , +RISCV_IOMMU_EXEC_PQCSR , +RISCV_IOMMU_EXEC_PQH , +RISCV_IOMMU_EXEC_TR_REQUEST , +/* RISCV_IOMMU_EXEC_EXIT must be the last enum value */ +RISCV_IOMMU_EXEC_EXIT , +}; + +static void *riscv_iommu_core_proc (void * arg) +{ +RISCVIOMMUState *s = arg; +unsigned exec =0 ; +unsigned mask =0 ; + +while (!(exec &BIT (RISCV_IOMMU_EXEC_EXIT ))) { + mask = (mask ? mask :BIT (RISCV_IOMMU_EXEC_EXIT )) >>1 ; +switch (exec & mask) { +case BIT (RISCV_IOMMU_EXEC_DDTP ): +riscv_iommu_process_ddtp (s); +break ; +case BIT (RISCV_IOMMU_EXEC_CQCSR ): +riscv_iommu_process_cq_control (s); +break ; +case BIT (RISCV_IOMMU_EXEC_CQT ): +riscv_iommu_process_cq_tail (s); +break ; +case BIT (RISCV_IOMMU_EXEC_FQCSR ): +riscv_iommu_process_fq_control (s); +break ; +case BIT (RISCV_IOMMU_EXEC_FQH ): +/* NOP */ +break ; +case BIT (RISCV_IOMMU_EXEC_PQCSR ): +riscv_iommu_process_pq_control (s); +break ; +case BIT (RISCV_IOMMU_EXEC_PQH ): +/* NOP */ +break ; +case BIT (RISCV_IOMMU_EXEC_TR_REQUEST ): +riscv_iommu_process_dbg (s); +break ; + } + exec &= ~mask; +if (!exec) { +qemu_mutex_lock (&s->core_lock); + exec = s->core_exec; +while (!exec) { +qemu_cond_wait (&s->core_cond, &s->core_lock); + exec = s->core_exec; + } + s->core_exec =0 ; +qemu_mutex_unlock (&s->core_lock); + } + }; + +return NULL ; +} + +/* For now we assume IOMMU HPM frequency to be 1GHz so 1-cycle is of 1-ns. */ +static inline uint64_t __get_cycles (void ) +{ +return qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); +} + +static void __hpm_setup_timer (RISCVIOMMUState *s,uint64_t value) +{ +const uint32_t inhibit =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTINH ); +uint64_t overflow_at, overflow_ns; + +if (get_field (inhibit,RISCV_IOMMU_IOCOUNTINH_CY )) { +return ; + } + +/* +* We are using INT64_MAX here instead to UINT64_MAX because cycle counter +* has 63-bit precision and INT64_MAX is the maximum it can store. +*/ +if (value) { + overflow_ns =INT64_MAX - value +1 ; + }else { + overflow_ns =INT64_MAX ; + } + + overflow_at = (uint64_t )qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ) + overflow_ns; + +if (overflow_at >INT64_MAX ) { + s->irq_overflow_left = overflow_at -INT64_MAX ; + overflow_at =INT64_MAX ; + } + +timer_mod_anticipate_ns (s->hpm_timer, overflow_at); +} + +/* Updates the internal cycle counter state when iocntinh:CY is changed. */ +static void riscv_iommu_process_iocntinh_cy (RISCVIOMMUState *s, +bool prev_cy_inh) +{ +const uint32_t inhibit =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTINH ); + +/* We only need to process CY bit toggle. */ +if (!(inhibit ^ prev_cy_inh)) { +return ; + } + +if (!(inhibit &RISCV_IOMMU_IOCOUNTINH_CY )) { +/* +* Cycle counter is enabled. Just start the timer again and update the +* clock snapshot value to point to the current time to make sure +* iohpmcycles read is correct. +*/ + s->hpmcycle_prev =__get_cycles (); +__hpm_setup_timer (s, s->hpmcycle_val); + }else { +/* +* Cycle counter is disabled. Stop the timer and update the cycle +* counter to record the current value which is last programmed +* value + the cycles passed so far. +*/ + s->hpmcycle_val = s->hpmcycle_val + (__get_cycles () - s->hpmcycle_prev); +timer_del (s->hpm_timer); + } +} + +static void riscv_iommu_process_hpmcycle_write (RISCVIOMMUState *s) +{ +const uint64_t val =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_IOHPMCYCLES ); +const uint32_t ovf =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ); + +/* +* Clear OF bit in IOCNTOVF if it's being cleared in IOHPMCYCLES register. +*/ +if (get_field (ovf,RISCV_IOMMU_IOCOUNTOVF_CY ) && + !get_field (val,RISCV_IOMMU_IOHPMCYCLES_OVF )) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ,0 , +RISCV_IOMMU_IOCOUNTOVF_CY ); + } + + s->hpmcycle_val = val & ~RISCV_IOMMU_IOHPMCYCLES_OVF ; + s->hpmcycle_prev =__get_cycles (); +__hpm_setup_timer (s, s->hpmcycle_val); +} + +static inline bool __check_valid_event_id (unsigned event_id) +{ +return event_id >RISCV_IOMMU_HPMEVENT_INVALID && + event_id <RISCV_IOMMU_HPMEVENT_MAX ; +} + +static gboolean __hpm_event_equal (gpointer key,gpointer value,gpointer udata) +{ +uint32_t *pair = udata; + +if (GPOINTER_TO_UINT (value) & (1 << pair[0 ])) { + pair[1 ] =GPOINTER_TO_UINT (key); +return true; + } + +return false; +} + +/* Caller must check ctr_idx against hpm_ctrs to see if its supported or not. */ +static void __update_event_map (RISCVIOMMUState *s,uint64_t value, +uint32_t ctr_idx) +{ +unsigned event_id =get_field (value,RISCV_IOMMU_IOHPMEVT_EVENT_ID ); +uint32_t pair[2 ] = { ctr_idx,RISCV_IOMMU_HPMEVENT_INVALID }; +uint32_t new_value =1 << ctr_idx; +gpointer data; + +/* If EventID field is RISCV_IOMMU_HPMEVENT_INVALID remove the current mapping. */ +if (event_id ==RISCV_IOMMU_HPMEVENT_INVALID ) { + data =g_hash_table_find (s->hpm_event_ctr_map, __hpm_event_equal, pair); + + new_value =GPOINTER_TO_UINT (data) & ~(new_value); +pthread_rwlock_wrlock (&s->ht_lock); +if (new_value !=0 ) { +g_hash_table_replace (s->hpm_event_ctr_map, +GUINT_TO_POINTER (pair[1 ]), +GUINT_TO_POINTER (new_value)); + }else { +g_hash_table_remove (s->hpm_event_ctr_map, +GUINT_TO_POINTER (pair[1 ])); + } +pthread_rwlock_unlock (&s->ht_lock); + +return ; + } + +/* Update the counter mask if the event is already enabled. */ +if (g_hash_table_lookup_extended (s->hpm_event_ctr_map, +GUINT_TO_POINTER (event_id), +NULL , + &data)) { + new_value |=GPOINTER_TO_UINT (data); + } + +pthread_rwlock_wrlock (&s->ht_lock); +g_hash_table_insert (s->hpm_event_ctr_map, +GUINT_TO_POINTER (event_id), +GUINT_TO_POINTER (new_value)); +pthread_rwlock_unlock (&s->ht_lock); +} + +static void riscv_iommu_process_hpmevt_write (RISCVIOMMUState *s, +uint32_t evt_reg) +{ +const uint32_t ctr_idx = (evt_reg -RISCV_IOMMU_REG_IOHPMEVT_BASE ) >>3 ; +const uint32_t ovf =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ); +uint64_t val =riscv_iommu_reg_get64 (s, evt_reg); + +if (ctr_idx >= s->hpm_cntrs) { +return ; + } + +/* Clear OF bit in IOCNTOVF if it's being cleared in IOHPMEVT register. */ +if (get_field (ovf,BIT (ctr_idx +1 )) && !get_field (val,RISCV_IOMMU_IOHPMEVT_OF )) { +/* +1 to offset CYCLE register OF bit. */ +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ,0 ,BIT (ctr_idx +1 )); + } + +if (!__check_valid_event_id (get_field (val,RISCV_IOMMU_IOHPMEVT_EVENT_ID ))) { +/* Reset EventID (WARL) field to invalid. */ + val =set_field (val,RISCV_IOMMU_IOHPMEVT_EVENT_ID , +RISCV_IOMMU_HPMEVENT_INVALID ); +riscv_iommu_reg_set64 (s, evt_reg, val); + } + +__update_event_map (s, val, ctr_idx); +} + +static void riscv_iommu_process_hpm_writes (RISCVIOMMUState *s, +uint32_t regb, +bool prev_cy_inh) +{ +switch (regb) { +case RISCV_IOMMU_REG_IOCOUNTINH : +riscv_iommu_process_iocntinh_cy (s, prev_cy_inh); +break ; + +case RISCV_IOMMU_REG_IOHPMCYCLES : +case RISCV_IOMMU_REG_IOHPMCYCLES +4 : +riscv_iommu_process_hpmcycle_write (s); +break ; + + case RISCV_IOMMU_REG_IOHPMEVT_BASE ... +RISCV_IOMMU_REG_IOHPMEVT (RISCV_IOMMU_IOCOUNT_NUM ) +4 : +riscv_iommu_process_hpmevt_write (s, regb & ~7 ); +break ; + } +} + +static MemTxResult riscv_iommu_mmio_write (void *opaque,hwaddr addr, +uint64_t data,unsigned size,MemTxAttrs attrs) +{ +RISCVIOMMUState *s = opaque; +uint32_t regb = addr & ~3 ; +bool cy_inh = false; +uint32_t busy =0 ; +uint32_t exec =0 ; + +if (size ==0 || size >8 || (addr & (size -1 )) !=0 ) { +/* Unsupported MMIO alignment or access size */ +return MEMTX_ERROR ; + } + +if (addr + size >RISCV_IOMMU_REG_MSI_CONFIG ) { +/* Unsupported MMIO access location. */ +return MEMTX_ACCESS_ERROR ; + } + +/* Track actionable MMIO write. */ +switch (regb) { +case RISCV_IOMMU_REG_DDTP : +case RISCV_IOMMU_REG_DDTP +4 : + exec =BIT (RISCV_IOMMU_EXEC_DDTP ); + regb =RISCV_IOMMU_REG_DDTP ; + busy =RISCV_IOMMU_DDTP_BUSY ; +break ; + +case RISCV_IOMMU_REG_CQT : + exec =BIT (RISCV_IOMMU_EXEC_CQT ); +break ; + +case RISCV_IOMMU_REG_CQCSR : + exec =BIT (RISCV_IOMMU_EXEC_CQCSR ); + busy =RISCV_IOMMU_CQCSR_BUSY ; +break ; + +case RISCV_IOMMU_REG_FQH : + exec =BIT (RISCV_IOMMU_EXEC_FQH ); +break ; + +case RISCV_IOMMU_REG_FQCSR : + exec =BIT (RISCV_IOMMU_EXEC_FQCSR ); + busy =RISCV_IOMMU_FQCSR_BUSY ; +break ; + +case RISCV_IOMMU_REG_PQH : + exec =BIT (RISCV_IOMMU_EXEC_PQH ); +break ; + +case RISCV_IOMMU_REG_PQCSR : + exec =BIT (RISCV_IOMMU_EXEC_PQCSR );Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
2.34.1
Part of the diff is not shown here. View the raw message.
Re: [PATCH 1/5] hw/riscv: Introduction of RISC-V IOMMU device
Quoted 36 lines · Alistair Francis
Quoted 4 lines · Tomasz Jeznach
The RISC-V IOMMU specification is now ratified as-per the RISC-V international process [1]. The latest frozen specifcation can be found at:
https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdfExciting!
Quoted 14 lines
The patch add device emulation for RISC-V IOMMU which supports device and process context lookups, command and fault queue interfaces, two stage address translation logic with Sv32, Sv39, Sv48, Sv57 addressing modes, address translation cache, MSI remapping with FLAT/MRIF modes, initial ATS and PRI interfaces, debug capabilities, hardware performance counters. Platform and PCIe device instantiation is supported, with wire-signaled and message-signaled interrupt capabilities.
Hardware interface definition file is shared with Linux kernel driver implementation, available in the maintainer's branch
riscv_iommu_v1at https://github.com/tjeznach/linux.Co-developed-by Sebastien Boeuf <[email protected]>Signed-off-by Sebastien Boeuf <[email protected]>Signed-off-by Tomasz Jeznach <[email protected]>Quoted 12 lines
--- hw/riscv/Kconfig | 3 + hw/riscv/meson.build | 1 + hw/riscv/riscv-iommu-bits.h | 749 +++++++++++ hw/riscv/riscv-iommu-pci.c | 181 +++ hw/riscv/riscv-iommu-sys.c | 123 ++ hw/riscv/riscv-iommu.c | 2539 +++++++++++++++++++++++++++++++++++ hw/riscv/riscv-iommu.h | 152 +++ hw/riscv/trace-events | 14 + hw/riscv/trace.h | 2 + include/hw/riscv/iommu.h | 40 + meson.build | 1 +
Quoted 6 lines
This is a really long patch!
I think this should at least be split up to rougly each file (as long as it compiles). For example the header files could be added in a patch each. Which would reduce some of the review burden.
Thanks for the feedback. Definitely it will be easier for reviewers :) I will split into more reasonable patches.
Quoted 831 lines
Quoted 50 lines
11 files changed, 3805 insertions(+) create mode 100644 hw/riscv/riscv-iommu-bits.h create mode 100644 hw/riscv/riscv-iommu-pci.c create mode 100644 hw/riscv/riscv-iommu-sys.c create mode 100644 hw/riscv/riscv-iommu.c create mode 100644 hw/riscv/riscv-iommu.h create mode 100644 hw/riscv/trace-events create mode 100644 hw/riscv/trace.h create mode 100644 include/hw/riscv/iommu.hhw/riscv/Kconfig +3
-0@@ -1,3 +1,6 @@ +config RISCV_IOMMU + bool + config RISCV_NUMA boolhw/riscv/meson.build +1
-0@@ -10,5 +10,6 @@ riscv_ss.add(when: 'CONFIG_SIFIVE_U', if_true: files('sifive_u.c')) riscv_ss.add(when: 'CONFIG_SPIKE', if_true: files('spike.c')) riscv_ss.add(when: 'CONFIG_MICROCHIP_PFSOC', if_true: files('microchip_pfsoc.c')) riscv_ss.add(when: 'CONFIG_ACPI', if_true: files('virt-acpi-build.c')) +riscv_ss.add(when: 'CONFIG_RISCV_IOMMU', if_true: files('riscv-iommu.c', 'riscv-iommu-pci.c', 'riscv-iommu-sys.c')) hw_arch += {'riscv': riscv_ss}hw/riscv/riscv-iommu-bits.h new file +12
-0@@ -0,0 +1,749 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright © 2022-2023 Rivos Inc. +* Copyright © 2023 FORTH-ICS/CARV +* Copyright © 2023 RISC-V IOMMU Task Group +* +* RISC-V Ziommu - Register Layout and Data Structures. +* +* Based on the 'RISC-V IOMMU Architecture Specification', Version 1.0 +* Published at https://github.com/riscv-non-isa/riscv-iommu +* +*/ hw/riscv/riscv-iommu-bits.h line 22
Quoted 10 lines
hw/riscv/riscv-iommu-bits.h new file +10
-0+ +#ifndef HW_RISCV_IOMMU_BITS_H +#define HW_RISCV_IOMMU_BITS_H + +/* +* This file is based on Linux RISC-V IOMMU file +* located at 'drivers/iommu/riscv/iommu-bits.h' +*/ + +#include "qemu/osdep.h" This shouldn't be included in header files
Quoted 768 lines
hw/riscv/riscv-iommu-bits.h new file +727
-0+ +#define RISCV_IOMMU_SPEC_DOT_VER 0x010 + +#ifndef GENMASK_ULL +#define GENMASK_ULL (h, l) (((~0ULL) >> (63 - (h) + (l))) << (l)) +#endif + +/* +* Chapter 5: Memory Mapped register interface +*/ + +/* Common field positions */ +#define RISCV_IOMMU_PPN_FIELD GENMASK_ULL(53, 10) +#define RISCV_IOMMU_QUEUE_LOGSZ_FIELD GENMASK_ULL(4, 0) +#define RISCV_IOMMU_QUEUE_INDEX_FIELD GENMASK_ULL(31, 0) +#define RISCV_IOMMU_QUEUE_ENABLE BIT(0) +#define RISCV_IOMMU_QUEUE_INTR_ENABLE BIT(1) +#define RISCV_IOMMU_QUEUE_MEM_FAULT BIT(8) +#define RISCV_IOMMU_QUEUE_OVERFLOW BIT(9) +#define RISCV_IOMMU_QUEUE_ACTIVE BIT(16) +#define RISCV_IOMMU_QUEUE_BUSY BIT(17) +#define RISCV_IOMMU_ATP_PPN_FIELD GENMASK_ULL(43, 0) +#define RISCV_IOMMU_ATP_MODE_FIELD GENMASK_ULL(63, 60) + +/* 5.3 IOMMU Capabilities (64bits) */ +#define RISCV_IOMMU_REG_CAP 0x0000 +#define RISCV_IOMMU_CAP_VERSION GENMASK_ULL(7, 0) +#define RISCV_IOMMU_CAP_S_SV32 BIT_ULL(8) +#define RISCV_IOMMU_CAP_S_SV39 BIT_ULL(9) +#define RISCV_IOMMU_CAP_S_SV48 BIT_ULL(10) +#define RISCV_IOMMU_CAP_S_SV57 BIT_ULL(11) +#define RISCV_IOMMU_CAP_SVPBMT BIT_ULL(15) +#define RISCV_IOMMU_CAP_G_SV32 BIT_ULL(16) +#define RISCV_IOMMU_CAP_G_SV39 BIT_ULL(17) +#define RISCV_IOMMU_CAP_G_SV48 BIT_ULL(18) +#define RISCV_IOMMU_CAP_G_SV57 BIT_ULL(19) +#define RISCV_IOMMU_CAP_MSI_FLAT BIT_ULL(22) +#define RISCV_IOMMU_CAP_MSI_MRIF BIT_ULL(23) +#define RISCV_IOMMU_CAP_AMO BIT_ULL(24) +#define RISCV_IOMMU_CAP_ATS BIT_ULL(25) +#define RISCV_IOMMU_CAP_T2GPA BIT_ULL(26) +#define RISCV_IOMMU_CAP_END BIT_ULL(27) +#define RISCV_IOMMU_CAP_IGS GENMASK_ULL(29, 28) +#define RISCV_IOMMU_CAP_HPM BIT_ULL(30) +#define RISCV_IOMMU_CAP_DBG BIT_ULL(31) +#define RISCV_IOMMU_CAP_PAS GENMASK_ULL(37, 32) +#define RISCV_IOMMU_CAP_PD8 BIT_ULL(38) +#define RISCV_IOMMU_CAP_PD17 BIT_ULL(39) +#define RISCV_IOMMU_CAP_PD20 BIT_ULL(40) + +#define RISCV_IOMMU_CAP_VERSION_VER_MASK 0xF0 +#define RISCV_IOMMU_CAP_VERSION_REV_MASK 0x0F + +/** +* enum riscv_iommu_igs_settings - Interrupt Generation Support Settings +* @RISCV_IOMMU_CAP_IGS_MSI: I/O MMU supports only MSI generation +* @RISCV_IOMMU_CAP_IGS_WSI: I/O MMU supports only Wired-Signaled interrupt +* @RISCV_IOMMU_CAP_IGS_BOTH: I/O MMU supports both MSI and WSI generation +* @RISCV_IOMMU_CAP_IGS_RSRV: Reserved for standard use +*/ +enum riscv_iommu_igs_settings { +RISCV_IOMMU_CAP_IGS_MSI =0 , +RISCV_IOMMU_CAP_IGS_WSI =1 , +RISCV_IOMMU_CAP_IGS_BOTH =2 , +RISCV_IOMMU_CAP_IGS_RSRV =3 +}; + + +/* 5.4 Features control register (32bits) */ +#define RISCV_IOMMU_REG_FCTL 0x0008 +#define RISCV_IOMMU_FCTL_BE BIT(0) +#define RISCV_IOMMU_FCTL_WSI BIT(1) +#define RISCV_IOMMU_FCTL_GXL BIT(2) + + +/* 5.5 Device-directory-table pointer (64bits) */ +#define RISCV_IOMMU_REG_DDTP 0x0010 +#define RISCV_IOMMU_DDTP_MODE GENMASK_ULL(3, 0) +#define RISCV_IOMMU_DDTP_BUSY BIT_ULL(4) +#define RISCV_IOMMU_DDTP_PPN RISCV_IOMMU_PPN_FIELD + +/** +* enum riscv_iommu_ddtp_modes - I/O MMU translation modes +* @RISCV_IOMMU_DDTP_MODE_OFF: No inbound transactions allowed +* @RISCV_IOMMU_DDTP_MODE_BARE: Pass-through mode +* @RISCV_IOMMU_DDTP_MODE_1LVL: One-level DDT +* @RISCV_IOMMU_DDTP_MODE_2LVL: Two-level DDT +* @RISCV_IOMMU_DDTP_MODE_3LVL: Three-level DDT +*/ +enum riscv_iommu_ddtp_modes { +RISCV_IOMMU_DDTP_MODE_OFF =0 , +RISCV_IOMMU_DDTP_MODE_BARE =1 , +RISCV_IOMMU_DDTP_MODE_1LVL =2 , +RISCV_IOMMU_DDTP_MODE_2LVL =3 , +RISCV_IOMMU_DDTP_MODE_3LVL =4 , +RISCV_IOMMU_DDTP_MODE_MAX =4 +}; + + +/* 5.6 Command Queue Base (64bits) */ +#define RISCV_IOMMU_REG_CQB 0x0018 +#define RISCV_IOMMU_CQB_LOG2SZ RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_CQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.7 Command Queue head (32bits) */ +#define RISCV_IOMMU_REG_CQH 0x0020 +#define RISCV_IOMMU_CQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.8 Command Queue tail (32bits) */ +#define RISCV_IOMMU_REG_CQT 0x0024 +#define RISCV_IOMMU_CQT_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + + +/* 5.9 Fault Queue Base (64bits) */ +#define RISCV_IOMMU_REG_FQB 0x0028 +#define RISCV_IOMMU_FQB_LOG2SZ RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_FQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.10 Fault Queue Head (32bits) */ +#define RISCV_IOMMU_REG_FQH 0x0030 +#define RISCV_IOMMU_FQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.11 Fault Queue tail (32bits) */ +#define RISCV_IOMMU_REG_FQT 0x0034 +#define RISCV_IOMMU_FQT_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + + +/* 5.12 Page Request Queue base (64bits) */ +#define RISCV_IOMMU_REG_PQB 0x0038 +#define RISCV_IOMMU_PQB_LOG2SZ RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_PQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.13 Page Request Queue head (32bits) */ +#define RISCV_IOMMU_REG_PQH 0x0040 +#define RISCV_IOMMU_PQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.14 Page Request Queue tail (32bits) */ +#define RISCV_IOMMU_REG_PQT 0x0044 +#define RISCV_IOMMU_PQT_INDEX_MASK RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.15 Command Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_CQCSR 0x0048 +#define RISCV_IOMMU_CQCSR_CQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_CQCSR_CIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_CQCSR_CQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_CQCSR_CMD_TO BIT(9) +#define RISCV_IOMMU_CQCSR_CMD_ILL BIT(10) +#define RISCV_IOMMU_CQCSR_FENCE_W_IP BIT(11) +#define RISCV_IOMMU_CQCSR_CQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_CQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + + +/* 5.16 Fault Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_FQCSR 0x004C +#define RISCV_IOMMU_FQCSR_FQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_FQCSR_FIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_FQCSR_FQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_FQCSR_FQOF RISCV_IOMMU_QUEUE_OVERFLOW +#define RISCV_IOMMU_FQCSR_FQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_FQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + + +/* 5.17 Page Request Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_PQCSR 0x0050 +#define RISCV_IOMMU_PQCSR_PQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_PQCSR_PIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_PQCSR_PQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_PQCSR_PQOF RISCV_IOMMU_QUEUE_OVERFLOW +#define RISCV_IOMMU_PQCSR_PQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_PQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + + +/* 5.18 Interrupt Pending Status (32bits) */ +#define RISCV_IOMMU_REG_IPSR 0x0054 + +#define RISCV_IOMMU_INTR_CQ 0 +#define RISCV_IOMMU_INTR_FQ 1 +#define RISCV_IOMMU_INTR_PM 2 +#define RISCV_IOMMU_INTR_PQ 3 +#define RISCV_IOMMU_INTR_COUNT 4 + +#define RISCV_IOMMU_IPSR_CIP BIT(RISCV_IOMMU_INTR_CQ) +#define RISCV_IOMMU_IPSR_FIP BIT(RISCV_IOMMU_INTR_FQ) +#define RISCV_IOMMU_IPSR_PMIP BIT(RISCV_IOMMU_INTR_PM) +#define RISCV_IOMMU_IPSR_PIP BIT(RISCV_IOMMU_INTR_PQ) + +#define RISCV_IOMMU_IOCOUNT_NUM 31 + +/* 5.19 Performance monitoring counter overflow status (32bits) */ +#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058 +#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0) +#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK(31, 1) + +/* 5.20 Performance monitoring counter inhibits (32bits) */ +#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C +#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0) +#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 1) + +/* 5.21 Performance monitoring cycles counter (64bits) */ +#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060 +#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0) +#define RISCV_IOMMU_IOHPMCYCLES_OVF BIT_ULL(63) + +/* 5.22 Performance monitoring event counters (31 * 64bits) */ +#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068 +#define RISCV_IOMMU_REG_IOHPMCTR (_n) \ + (RISCV_IOMMU_REG_IOHPMCTR_BASE + (_n * 0x8)) + +/* 5.23 Performance monitoring event selectors (31 * 64bits) */ +#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160 +#define RISCV_IOMMU_REG_IOHPMEVT (_n) \ + (RISCV_IOMMU_REG_IOHPMEVT_BASE + (_n * 0x8)) +#define RISCV_IOMMU_IOHPMEVT_EVENT_ID GENMASK_ULL(14, 0) +#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15) +#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16) +#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36) +#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60) +#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61) +#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62) +#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63) + +/** +* enum RISCV_IOMMU_HPMEVENT_id - Performance-monitoring event identifier +* +* @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count +* @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests +* @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests +* @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests +* @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses +* @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks +* @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks +* @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: S/VS-Stage page table walks +* @RISCV_IOMMU_HPMEVENT_G_WALKS: G-Stage page table walks +* @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs +*/ +enum RISCV_IOMMU_HPMEVENT_id { +RISCV_IOMMU_HPMEVENT_INVALID =0 , +RISCV_IOMMU_HPMEVENT_URQ =1 , +RISCV_IOMMU_HPMEVENT_TRQ =2 , +RISCV_IOMMU_HPMEVENT_ATS_RQ =3 , +RISCV_IOMMU_HPMEVENT_TLB_MISS =4 , +RISCV_IOMMU_HPMEVENT_DD_WALK =5 , +RISCV_IOMMU_HPMEVENT_PD_WALK =6 , +RISCV_IOMMU_HPMEVENT_S_VS_WALKS =7 , +RISCV_IOMMU_HPMEVENT_G_WALKS =8 , +RISCV_IOMMU_HPMEVENT_MAX =9 +}; + +/* 5.24 Translation request IOVA (64bits) */ +#define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258 +#define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12) + +/* 5.25 Translation request control (64bits) */ +#define RISCV_IOMMU_REG_TR_REQ_CTL 0x0260 +#define RISCV_IOMMU_TR_REQ_CTL_GO_BUSY BIT_ULL(0) +#define RISCV_IOMMU_TR_REQ_CTL_PRIV BIT_ULL(1) +#define RISCV_IOMMU_TR_REQ_CTL_EXE BIT_ULL(2) +#define RISCV_IOMMU_TR_REQ_CTL_NW BIT_ULL(3) +#define RISCV_IOMMU_TR_REQ_CTL_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_TR_REQ_CTL_PV BIT_ULL(32) +#define RISCV_IOMMU_TR_REQ_CTL_DID GENMASK_ULL(63, 40) + +/* 5.26 Translation request response (64bits) */ +#define RISCV_IOMMU_REG_TR_RESPONSE 0x0268 +#define RISCV_IOMMU_TR_RESPONSE_FAULT BIT_ULL(0) +#define RISCV_IOMMU_TR_RESPONSE_PBMT GENMASK_ULL(8, 7) +#define RISCV_IOMMU_TR_RESPONSE_SZ BIT_ULL(9) +#define RISCV_IOMMU_TR_RESPONSE_PPN RISCV_IOMMU_PPN_FIELD + + +/* 5.27 Interrupt cause to vector (64bits) */ +#define RISCV_IOMMU_REG_IVEC 0x02F8 +#define RISCV_IOMMU_IVEC_CIV GENMASK_ULL(3, 0) +#define RISCV_IOMMU_IVEC_FIV GENMASK_ULL(7, 4) +#define RISCV_IOMMU_IVEC_PMIV GENMASK_ULL(11, 8) +#define RISCV_IOMMU_IVEC_PIV GENMASK_ULL(15, 12) + + +/* 5.28 MSI Configuration table (32 * 64bits) */ +#define RISCV_IOMMU_REG_MSI_CONFIG 0x0300 +#define RISCV_IOMMU_REG_MSI_ADDR (_n) \ + (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10)) +#define RISCV_IOMMU_MSI_ADDR GENMASK_ULL(55, 2) +#define RISCV_IOMMU_REG_MSI_DATA (_n) \ + (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10) + 0x08) +#define RISCV_IOMMU_MSI_DATA GENMASK_ULL(31, 0) +#define RISCV_IOMMU_REG_MSI_VEC_CTL (_n) \ + (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10) + 0x0C) +#define RISCV_IOMMU_MSI_VEC_CTL_M BIT_ULL(0) + + +#define RISCV_IOMMU_REG_SIZE 0x1000 + +/* +* Chapter 2: Data structures +*/ + +/* +* Device Directory Table macros for non-leaf nodes +*/ +#define RISCV_IOMMU_DDTE_VALID BIT_ULL(0) +#define RISCV_IOMMU_DDTE_PPN RISCV_IOMMU_PPN_FIELD + +/** +* struct riscv_iommu_dc - Device Context +* @tc: Translation Control +* @iohgatp: I/O Hypervisor guest address translation and protection +* (Second stage context) +* @ta: Translation Attributes +* @fsc: First stage context +* @msiptpt: MSI page table pointer +* @msi_addr_mask: MSI address mask +* @msi_addr_pattern: MSI address pattern +* +* This structure is used for leaf nodes on the Device Directory Table, +* in case RISCV_IOMMU_CAP_MSI_FLAT is not set, the bottom 4 fields are +* not present and are skipped with pointer arithmetic to avoid +* casting, check out riscv_iommu_get_dc(). +* See section 2.1 for more details +*/ +struct riscv_iommu_dc { +uint64_t tc; +uint64_t iohgatp; +uint64_t ta; +uint64_t fsc; +uint64_t msiptp; +uint64_t msi_addr_mask; +uint64_t msi_addr_pattern; +uint64_t _reserved; +}; + +/* Translation control fields */ +#define RISCV_IOMMU_DC_TC_V BIT_ULL(0) +#define RISCV_IOMMU_DC_TC_EN_ATS BIT_ULL(1) +#define RISCV_IOMMU_DC_TC_EN_PRI BIT_ULL(2) +#define RISCV_IOMMU_DC_TC_T2GPA BIT_ULL(3) +#define RISCV_IOMMU_DC_TC_DTF BIT_ULL(4) +#define RISCV_IOMMU_DC_TC_PDTV BIT_ULL(5) +#define RISCV_IOMMU_DC_TC_PRPR BIT_ULL(6) +#define RISCV_IOMMU_DC_TC_GADE BIT_ULL(7) +#define RISCV_IOMMU_DC_TC_SADE BIT_ULL(8) +#define RISCV_IOMMU_DC_TC_DPE BIT_ULL(9) +#define RISCV_IOMMU_DC_TC_SBE BIT_ULL(10) +#define RISCV_IOMMU_DC_TC_SXL BIT_ULL(11) + +/* Second-stage (aka G-stage) context fields */ +#define RISCV_IOMMU_DC_IOHGATP_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_IOHGATP_GSCID GENMASK_ULL(59, 44) +#define RISCV_IOMMU_DC_IOHGATP_MODE RISCV_IOMMU_ATP_MODE_FIELD + +/** +* enum riscv_iommu_dc_iohgatp_modes - Guest address +* translation/protection modes +* +* @RISCV_IOMMU_DC_IOHGATP_MODE_BARE: +* No translation/protection +* @RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4: +* Sv32x4 (2-bit extension of Sv32), when fctl.GXL == 1 +* @RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4: +* Sv39x4 (2-bit extension of Sv39), when fctl.GXL == 0 +* @RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4: +* Sv48x4 (2-bit extension of Sv48), when fctl.GXL == 0 +* @RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4: +* Sv57x4 (2-bit extension of Sv57), when fctl.GXL == 0 +*/ +enum riscv_iommu_dc_iohgatp_modes { +RISCV_IOMMU_DC_IOHGATP_MODE_BARE =0 , +RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4 =8 , +RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4 =8 , +RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4 =9 , +RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4 =10 +}; + +/* Translation attributes fields */ +#define RISCV_IOMMU_DC_TA_PSCID GENMASK_ULL(31, 12) + +/* First-stage context fields */ +#define RISCV_IOMMU_DC_FSC_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_FSC_MODE RISCV_IOMMU_ATP_MODE_FIELD + +/** +* enum riscv_iommu_dc_fsc_atp_modes - First stage address +* translation/protection modes +* +* @RISCV_IOMMU_DC_FSC_MODE_BARE: No translation/protection +* @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32: Sv32, when dc.tc.SXL == 1 +* @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39: Sv39, when dc.tc.SXL == 0 +* @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48: Sv48, when dc.tc.SXL == 0 +* @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57: Sv57, when dc.tc.SXL == 0 +* @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8: 1lvl PDT, 8bit process ids +* @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17: 2lvl PDT, 17bit process ids +* @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20: 3lvl PDT, 20bit process ids +* +* FSC holds IOSATP when RISCV_IOMMU_DC_TC_PDTV is 0 and PDTP otherwise. +* IOSATP controls the first stage address translation (same as the satp +* register on the RISC-V MMU), and PDTP holds the process directory table, +* used to select a first stage page table based on a process id (for devices +* that support multiple process ids). +*/ +enum riscv_iommu_dc_fsc_atp_modes { +RISCV_IOMMU_DC_FSC_MODE_BARE =0 , +RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32 =8 , +RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39 =8 , +RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48 =9 , +RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57 =10 , +RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8 =1 , +RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17 =2 , +RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20 =3 +}; + +/* MSI page table pointer */ +#define RISCV_IOMMU_DC_MSIPTP_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_MSIPTP_MODE RISCV_IOMMU_ATP_MODE_FIELD +#define RISCV_IOMMU_DC_MSIPTP_MODE_OFF 0 +#define RISCV_IOMMU_DC_MSIPTP_MODE_FLAT 1 + +/* MSI address mask */ +#define RISCV_IOMMU_DC_MSI_ADDR_MASK GENMASK_ULL(51, 0) + +/* MSI address pattern */ +#define RISCV_IOMMU_DC_MSI_PATTERN GENMASK_ULL(51, 0) + + +/** +* struct riscv_iommu_pc - Process Context +* @ta: Translation Attributes +* @fsc: First stage context +* +* This structure is used for leaf nodes on the Process Directory Table +* See section 2.3 for more details +*/ +struct riscv_iommu_pc { +uint64_t ta; +uint64_t fsc; +}; + +/* Translation attributes fields */ +#define RISCV_IOMMU_PC_TA_V BIT_ULL(0) +#define RISCV_IOMMU_PC_TA_ENS BIT_ULL(1) +#define RISCV_IOMMU_PC_TA_SUM BIT_ULL(2) +#define RISCV_IOMMU_PC_TA_PSCID GENMASK_ULL(31, 12) + +/* First stage context fields */ +#define RISCV_IOMMU_PC_FSC_PPN GENMASK_ULL(43, 0) +#define RISCV_IOMMU_PC_FSC_MODE GENMASK_ULL(63, 60) + + +/* +* Chapter 3: In-memory queue interface +*/ + +/** +* struct riscv_iommu_cmd - Generic I/O MMU command structure +* @dword0: Includes the opcode and the function identifier +* @dword1: Opcode specific data +* +* The commands are interpreted as two 64bit fields, where the first +* 7bits of the first field are the opcode which also defines the +* command's format, followed by a 3bit field that specifies the +* function invoked by that command, and the rest is opcode-specific. +* This is a generic struct which will be populated differently +* according to each command. For more infos on the commands and +* the command queue check section 3.1. +*/ +struct riscv_iommu_command { +uint64_t dword0; +uint64_t dword1; +}; + +/* Fields on dword0, common for all commands */ +#define RISCV_IOMMU_CMD_OPCODE GENMASK_ULL(6, 0) +#define RISCV_IOMMU_CMD_FUNC GENMASK_ULL(9, 7) + +/* 3.1.1 I/O MMU Page-table cache invalidation */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IOTINVAL_OPCODE 1 +#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA 0 +#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA 1 +#define RISCV_IOMMU_CMD_IOTINVAL_AV BIT_ULL(10) +#define RISCV_IOMMU_CMD_IOTINVAL_PSCID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_IOTINVAL_PSCV BIT_ULL(32) +#define RISCV_IOMMU_CMD_IOTINVAL_GV BIT_ULL(33) +#define RISCV_IOMMU_CMD_IOTINVAL_GSCID GENMASK_ULL(59, 44) +/* dword1 is the address, 4K-alligned and shifted to the right by two bits. */ + +/* 3.1.2 I/O MMU Command Queue Fences */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IOFENCE_OPCODE 2 +#define RISCV_IOMMU_CMD_IOFENCE_FUNC_C 0 +#define RISCV_IOMMU_CMD_IOFENCE_AV BIT_ULL(10) +#define RISCV_IOMMU_CMD_IOFENCE_WSI BIT_ULL(11) +#define RISCV_IOMMU_CMD_IOFENCE_PR BIT_ULL(12) +#define RISCV_IOMMU_CMD_IOFENCE_PW BIT_ULL(13) +#define RISCV_IOMMU_CMD_IOFENCE_DATA GENMASK_ULL(63, 32) +/* dword1 is the address, word-size alligned and shifted to the right by two bits. */ + +/* 3.1.3 I/O MMU Directory cache invalidation */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IODIR_OPCODE 3 +#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT 0 +#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT 1 +#define RISCV_IOMMU_CMD_IODIR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_IODIR_DV BIT_ULL(33) +#define RISCV_IOMMU_CMD_IODIR_DID GENMASK_ULL(63, 40) +/* dword1 is reserved for standard use */ + +/* 3.1.4 I/O MMU PCIe ATS */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_ATS_OPCODE 4 +#define RISCV_IOMMU_CMD_ATS_FUNC_INVAL 0 +#define RISCV_IOMMU_CMD_ATS_FUNC_PRGR 1 +#define RISCV_IOMMU_CMD_ATS_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_ATS_PV BIT_ULL(32) +#define RISCV_IOMMU_CMD_ATS_DSV BIT_ULL(33) +#define RISCV_IOMMU_CMD_ATS_RID GENMASK_ULL(55, 40) +#define RISCV_IOMMU_CMD_ATS_DSEG GENMASK_ULL(63, 56) +/* dword1 is the ATS payload, two different payload types for INVAL and PRGR */ + +/* ATS.INVAL payload*/ +#define RISCV_IOMMU_CMD_ATS_INVAL_G BIT_ULL(0) +/* Bits 1 - 10 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_INVAL_S BIT_ULL(11) +#define RISCV_IOMMU_CMD_ATS_INVAL_UADDR GENMASK_ULL(63, 12) + +/* ATS.PRGR payload */ +/* Bits 0 - 31 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_PRGR_PRG_INDEX GENMASK_ULL(40, 32) +/* Bits 41 - 43 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_PRGR_RESP_CODE GENMASK_ULL(47, 44) +#define RISCV_IOMMU_CMD_ATS_PRGR_DST_ID GENMASK_ULL(63, 48) + + +/** +* struct riscv_iommu_fq_record - Fault/Event Queue Record +* @hdr: Header, includes fault/event cause, PID/DID, transaction type etc +* @_reserved: Low 32bits for custom use, high 32bits for standard use +* @iotval: Transaction-type/cause specific format +* @iotval2: Cause specific format +* +* The fault/event queue reports events and failures raised when +* processing transactions. Each record is a 32byte structure where +* the first dword has a fixed format for providing generic infos +* regarding the fault/event, and two more dwords are there for +* fault/event-specific information. For more details see section +* 3.2. +*/ +struct riscv_iommu_fq_record { +uint64_t hdr; +uint64_t _reserved; +uint64_t iotval; +uint64_t iotval2; +}; + +/* Fields on header */ +#define RISCV_IOMMU_FQ_HDR_CAUSE GENMASK_ULL(11, 0) +#define RISCV_IOMMU_FQ_HDR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_FQ_HDR_PV BIT_ULL(32) +#define RISCV_IOMMU_FQ_HDR_PRIV BIT_ULL(33) +#define RISCV_IOMMU_FQ_HDR_TTYPE GENMASK_ULL(39, 34) +#define RISCV_IOMMU_FQ_HDR_DID GENMASK_ULL(63, 40) + +/** +* enum riscv_iommu_fq_causes - Fault/event cause values +* @RISCV_IOMMU_FQ_CAUSE_INST_FAULT: Instruction access fault +* @RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED: Read address misaligned +* @RISCV_IOMMU_FQ_CAUSE_RD_FAULT: Read load fault +* @RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED: Write/AMO address misaligned +* @RISCV_IOMMU_FQ_CAUSE_WR_FAULT: Write/AMO access fault +* @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S: Instruction page fault +* @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S: Read page fault +* @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S: Write/AMO page fault +* @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS: Instruction guest page fault +* @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS: Read guest page fault +* @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS: Write/AMO guest page fault +* @RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED: All inbound transactions disallowed +* @RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT: DDT entry load access fault +* @RISCV_IOMMU_FQ_CAUSE_DDT_INVALID: DDT entry invalid +* @RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED: DDT entry misconfigured +* @RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED: Transaction type disallowed +* @RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT: MSI PTE load access fault +* @RISCV_IOMMU_FQ_CAUSE_MSI_INVALID: MSI PTE invalid +* @RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED: MSI PTE misconfigured +* @RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT: MRIF access fault +* @RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT: PDT entry load access fault +* @RISCV_IOMMU_FQ_CAUSE_PDT_INVALID: PDT entry invalid +* @RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED: PDT entry misconfigured +* @RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED: DDT data corruption +* @RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED: PDT data corruption +* @RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED: MSI page table data corruption +* @RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED: MRIF data corruption +* @RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR: Internal data path error +* @RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT: IOMMU MSI write access fault +* @RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED: First/second stage page table data corruption +* +* Values are on table 11 of the spec, encodings 275 - 2047 are reserved for +* standard use, and 2048 - 4095 for custom use. +*/ +enum riscv_iommu_fq_causes { +RISCV_IOMMU_FQ_CAUSE_INST_FAULT =1 , +RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED =4 , +RISCV_IOMMU_FQ_CAUSE_RD_FAULT =5 , +RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED =6 , +RISCV_IOMMU_FQ_CAUSE_WR_FAULT =7 , +RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S =12 , +RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S =13 , +RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S =15 , +RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS =20 , +RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS =21 , +RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS =23 , +RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED =256 , +RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT =257 , +RISCV_IOMMU_FQ_CAUSE_DDT_INVALID =258 , +RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED =259 , +RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED =260 , +RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT =261 , +RISCV_IOMMU_FQ_CAUSE_MSI_INVALID =262 , +RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED =263 , +RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT =264 , +RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT =265 , +RISCV_IOMMU_FQ_CAUSE_PDT_INVALID =266 , +RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED =267 , +RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED =268 , +RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED =269 , +RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED =270 , +RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED =271 , +RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR =272 , +RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT =273 , +RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED =274 +}; + +/** +* enum riscv_iommu_fq_ttypes: Fault/event transaction types +* @RISCV_IOMMU_FQ_TTYPE_NONE: None. Fault not caused by an inbound transaction. +* @RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH: Instruction fetch from untranslated address +* @RISCV_IOMMU_FQ_TTYPE_UADDR_RD: Read from untranslated address +* @RISCV_IOMMU_FQ_TTYPE_UADDR_WR: Write/AMO to untranslated address +* @RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH: Instruction fetch from translated address +* @RISCV_IOMMU_FQ_TTYPE_TADDR_RD: Read from translated address +* @RISCV_IOMMU_FQ_TTYPE_TADDR_WR: Write/AMO to translated address +* @RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ: PCIe ATS translation request +* @RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ: PCIe message request +* +* Values are on table 12 of the spec, type 4 and 10 - 31 are reserved for +* standard use and 31 - 63 for custom use. +*/ +enum riscv_iommu_fq_ttypes { +RISCV_IOMMU_FQ_TTYPE_NONE =0 , +RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH =1 , +RISCV_IOMMU_FQ_TTYPE_UADDR_RD =2 , +RISCV_IOMMU_FQ_TTYPE_UADDR_WR =3 , +RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH =5 , +RISCV_IOMMU_FQ_TTYPE_TADDR_RD =6 , +RISCV_IOMMU_FQ_TTYPE_TADDR_WR =7 , +RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ =8 , +RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ =9 , +}; + + +/** +* struct riscv_iommu_pq_record - PCIe Page Request record +* @hdr: Header, includes PID, DID etc +* @payload: Holds the page address, request group and permission bits +* +* For more infos on the PCIe Page Request queue see chapter 3.3. +*/ +struct riscv_iommu_pq_record { +uint64_t hdr; +uint64_t payload; +}; + +/* Header fields */ +#define RISCV_IOMMU_PREQ_HDR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_PREQ_HDR_PV BIT_ULL(32) +#define RISCV_IOMMU_PREQ_HDR_PRIV BIT_ULL(33) +#define RISCV_IOMMU_PREQ_HDR_EXEC BIT_ULL(34) +#define RISCV_IOMMU_PREQ_HDR_DID GENMASK_ULL(63, 40) + +/* Payload fields */ +#define RISCV_IOMMU_PREQ_PAYLOAD_R BIT_ULL(0) +#define RISCV_IOMMU_PREQ_PAYLOAD_W BIT_ULL(1) +#define RISCV_IOMMU_PREQ_PAYLOAD_L BIT_ULL(2) +#define RISCV_IOMMU_PREQ_PAYLOAD_M GENMASK_ULL(2, 0) +#define RISCV_IOMMU_PREQ_PRG_INDEX GENMASK_ULL(11, 3) +#define RISCV_IOMMU_PREQ_UADDR GENMASK_ULL(63, 12) + + +/** +* struct riscv_iommu_msi_pte - MSI Page Table Entry +* @pte: MSI PTE +* @mrif_info: Memory-resident interrupt file info +* +* The MSI Page Table is used for virtualizing MSIs, so that when +* a device sends an MSI to a guest, the IOMMU can reroute it +* by translating the MSI address, either to a guest interrupt file +* or a memory resident interrupt file (MRIF). Note that this page table +* is an array of MSI PTEs, not a multi-level pt, each entry +* is a leaf entry. For more infos check out the the AIA spec, chapter 9.5. +* +* Also in basic mode the mrif_info field is ignored by the IOMMU and can +* be used by software, any other reserved fields on pte must be zeroed-out +* by software. +*/ +struct riscv_iommu_msi_pte { +uint64_t pte; +uint64_t mrif_info; +}; + +/* Fields on pte */ +#define RISCV_IOMMU_MSI_PTE_V BIT_ULL(0) +#define RISCV_IOMMU_MSI_PTE_M GENMASK_ULL(2, 1) + +#define RISCV_IOMMU_MSI_PTE_M_MRIF 1 +#define RISCV_IOMMU_MSI_PTE_M_BASIC 3 + +/* When M == 1 (MRIF mode) */ +#define RISCV_IOMMU_MSI_PTE_MRIF_ADDR GENMASK_ULL(53, 7) +/* When M == 3 (basic mode) */ +#define RISCV_IOMMU_MSI_PTE_PPN RISCV_IOMMU_PPN_FIELD +#define RISCV_IOMMU_MSI_PTE_C BIT_ULL(63) + +/* Fields on mrif_info */ +#define RISCV_IOMMU_MSI_MRIF_NID GENMASK_ULL(9, 0) +#define RISCV_IOMMU_MSI_MRIF_NPPN RISCV_IOMMU_PPN_FIELD +#define RISCV_IOMMU_MSI_MRIF_NID_MSB BIT_ULL(60) + + +#endif /* _RISCV_IOMMU_BITS_H_ */ hw/riscv/riscv-iommu-pci.c new file +35
-0@@ -0,0 +1,181 @@ +/* +* QEMU emulation of an RISC-V IOMMU (Ziommu) +* +* Copyright (C) 2022-2023 Rivos Inc. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#include "qemu/osdep.h" +#include "hw/pci/msi.h" +#include "hw/pci/msix.h" +#include "hw/pci/pci_bus.h" +#include "hw/qdev-properties.h" +#include "hw/riscv/riscv_hart.h" +#include "migration/vmstate.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qemu/host-utils.h" +#include "qom/object.h" + +#include "cpu_bits.h" +#include "riscv-iommu.h" +#include "riscv-iommu-bits.h" + +#ifndef PCI_VENDOR_ID_RIVOS
Quoted 9 lines
hw/riscv/riscv-iommu-pci.c line 41
Quoted 6 lines
hw/riscv/riscv-iommu-pci.c new file +6
-0+#define PCI_VENDOR_ID_RIVOS 0x1efd +#endif + +#ifndef PCI_DEVICE_ID_RIVOS_IOMMU +#define PCI_DEVICE_ID_RIVOS_IOMMU 0xedf1 +#endif The file is the RISC-V IOMMU, but don't these IDs say Rivos IOMMU?
This can be changed if there is a more appropriate poll of VIDs for QEMU models. As we had to pick an VID/DID, the simples was to donate one from Rivos VID pool.
Quoted 132 lines
Quoted 132 lines
hw/riscv/riscv-iommu-pci.c new file +132
-0+ +/* RISC-V IOMMU PCI Device Emulation */ + +typedef struct RISCVIOMMUStatePci { +PCIDevice pci;/* Parent PCIe device state */ +MemoryRegion bar0;/* PCI BAR (including MSI-x config) */ +RISCVIOMMUState iommu;/* common IOMMU state */ +}RISCVIOMMUStatePci ; + +/* interrupt delivery callback */ +static void riscv_iommu_pci_notify (RISCVIOMMUState *iommu,unsigned vector) +{ +RISCVIOMMUStatePci *s =container_of (iommu, RISCVIOMMUStatePci, iommu); + +if (msix_enabled (&(s->pci))) { +msix_notify (&(s->pci), vector); + } +} + +static void riscv_iommu_pci_realize (PCIDevice *dev,Error **errp) +{ +RISCVIOMMUStatePci *s =DO_UPCAST (RISCVIOMMUStatePci, pci, dev); +RISCVIOMMUState *iommu = &s->iommu; +uint64_t cap = iommu->cap; +Error *err =NULL ; + +/* Set device id for trace / debug */ +DEVICE (iommu)->id =g_strdup_printf ("%02x:%02x.%01x" , +pci_dev_bus_num (dev),PCI_SLOT (dev->devfn),PCI_FUNC (dev->devfn)); + +/* Support MSI only */ + cap =set_field (cap,RISCV_IOMMU_CAP_IGS ,RISCV_IOMMU_CAP_IGS_MSI ); +qdev_prop_set_uint64 (DEVICE (dev),"capabilities" , cap); + +if (!qdev_realize (DEVICE (iommu),NULL , errp)) { +return ; + } + +memory_region_init (&s->bar0,OBJECT (s),"riscv-iommu-bar0" , +QEMU_ALIGN_UP (memory_region_size (&iommu->regs_mr),TARGET_PAGE_SIZE )); +memory_region_add_subregion (&s->bar0,0 , &iommu->regs_mr); + +pcie_endpoint_cap_init (dev,0 ); + +pci_register_bar (dev,0 ,PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , &s->bar0); + +int ret =msix_init (dev,RISCV_IOMMU_INTR_COUNT , + &s->bar0,0 ,RISCV_IOMMU_REG_MSI_CONFIG , + &s->bar0,0 ,RISCV_IOMMU_REG_MSI_CONFIG +256 ,0 , &err); + +if (ret == -ENOTSUP ) { +/* +* MSI-x is not supported by the platform. +* Driver should use timer/polling based notification handlers. +*/ +warn_report_err (err); + }else if (ret <0 ) { +error_propagate (errp, err); +return ; + }else { +/* mark all allocated MSIx vectors as used. */ +msix_vector_use (dev,RISCV_IOMMU_INTR_CQ ); +msix_vector_use (dev,RISCV_IOMMU_INTR_FQ ); +msix_vector_use (dev,RISCV_IOMMU_INTR_PM ); +msix_vector_use (dev,RISCV_IOMMU_INTR_PQ ); + iommu->notify = riscv_iommu_pci_notify; + } + +PCIBus *bus =pci_device_root_bus (dev); +if (!bus) { +error_setg (errp,"can't find PCIe root port for %02x:%02x.%x" , +pci_bus_num (pci_get_bus (dev)),PCI_SLOT (dev->devfn), +PCI_FUNC (dev->devfn)); +return ; + } + +riscv_iommu_pci_setup_iommu (iommu, bus, errp); +} + +static void riscv_iommu_pci_exit (PCIDevice *pci_dev) +{ +pci_setup_iommu (pci_device_root_bus (pci_dev),NULL ,NULL ); +} + +static const VMStateDescription riscv_iommu_vmstate = { + .name ="riscv-iommu" , + .unmigratable =1 +}; + +static void riscv_iommu_pci_init (Object *obj) +{ +RISCVIOMMUStatePci *s =RISCV_IOMMU_PCI (obj); +RISCVIOMMUState *iommu = &s->iommu; + +object_initialize_child (obj,"iommu" , iommu,TYPE_RISCV_IOMMU ); +qdev_alias_all_properties (DEVICE (iommu), obj); +} + +static Property riscv_iommu_pci_properties[] = { +DEFINE_PROP_END_OF_LIST (), +}; + +static void riscv_iommu_pci_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *k =PCI_DEVICE_CLASS (klass); + + k->realize = riscv_iommu_pci_realize; + k->exit = riscv_iommu_pci_exit; + k->vendor_id =PCI_VENDOR_ID_RIVOS ; + k->device_id =PCI_DEVICE_ID_RIVOS_IOMMU ; + k->revision =0 ; + k->class_id =0x0806 ; + dc->desc ="RISCV-IOMMU DMA Remapping device" ; + dc->vmsd = &riscv_iommu_vmstate; + dc->hotpluggable = false; + dc->user_creatable = true; +set_bit (DEVICE_CATEGORY_MISC , dc->categories); +device_class_set_props (dc, riscv_iommu_pci_properties); +} + +static const TypeInfo riscv_iommu_pci = { + .name =TYPE_RISCV_IOMMU_PCI , + .parent =TYPE_PCI_DEVICE , + .class_init = riscv_iommu_pci_class_init, + .instance_init = riscv_iommu_pci_init, + .instance_size =sizeof (RISCVIOMMUStatePci), + .interfaces = (InterfaceInfo []) { + {INTERFACE_PCIE_DEVICE }, + { }, + },
Quoted 9 lines
hw/riscv/riscv-iommu-pci.c line 179
Quoted 6 lines
hw/riscv/riscv-iommu-pci.c new file +6
-0+}; + +static void riscv_iommu_register_pci_types (void ) +{ +type_register_static (&riscv_iommu_pci); +}The PCIe device should be a seperate patch
ack.
Quoted 123 lines
Quoted 123 lines
hw/riscv/riscv-iommu-pci.c new file +2
-0+ +type_init (riscv_iommu_register_pci_types);hw/riscv/riscv-iommu-sys.c new file +115
-0@@ -0,0 +1,123 @@ +/* +* QEMU emulation of an RISC-V IOMMU (Ziommu) - Platform Device +* +* Copyright (C) 2022-2023 Rivos Inc. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#include "qemu/osdep.h" +#include "hw/pci/pci_bus.h" +#include "hw/irq.h" +#include "hw/qdev-properties.h" +#include "hw/sysbus.h" +#include "qapi/error.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qemu/host-utils.h" +#include "qemu/module.h" +#include "qemu/osdep.h" +#include "qom/object.h" + +#include "cpu_bits.h" +#include "riscv-iommu.h" +#include "riscv-iommu-bits.h" + +/* RISC-V IOMMU System Platform Device Emulation */ + +struct RISCVIOMMUStateSys { +SysBusDevice parent; +uint64_t addr; +qemu_irq irqs[4 ]; +RISCVIOMMUState iommu; +}; + +/* interrupt delivery callback */ +static void riscv_iommu_sys_notify (RISCVIOMMUState *iommu,unsigned vector) +{ +RISCVIOMMUStateSys *s =container_of (iommu, RISCVIOMMUStateSys, iommu); + +if (vector <RISCV_IOMMU_INTR_COUNT && s->irqs[vector]) { +qemu_irq_pulse (s->irqs[vector]); + } +} + +static void riscv_iommu_sys_realize (DeviceState *dev,Error **errp) +{ +RISCVIOMMUStateSys *s =RISCV_IOMMU_SYS (dev); +RISCVIOMMUState *iommu = &s->iommu; +PCIBus *pci_bus; +uint64_t cap = iommu->cap; +int i; + +/* Support WSI only */ + cap =set_field (cap,RISCV_IOMMU_CAP_IGS ,RISCV_IOMMU_CAP_IGS_WSI ); +qdev_prop_set_uint64 (dev,"capabilities" , cap); + +if (!qdev_realize (DEVICE (iommu),NULL , errp)) { +return ; + } + +sysbus_init_mmio (SYS_BUS_DEVICE (dev), &iommu->regs_mr); +if (s->addr) { +sysbus_mmio_map (SYS_BUS_DEVICE (s),0 , s->addr); + } + +for (i =0 ; i <RISCV_IOMMU_INTR_COUNT ; i++) { +sysbus_init_irq (&s->parent, &s->irqs[i]); + } + + iommu->notify = riscv_iommu_sys_notify; + + pci_bus = (PCIBus *)object_resolve_path_type ("" ,TYPE_PCI_BUS ,NULL ); +if (pci_bus) { +riscv_iommu_pci_setup_iommu (iommu, pci_bus, errp); + } +} + +static void riscv_iommu_sys_init (Object *obj) +{ +RISCVIOMMUStateSys *s =RISCV_IOMMU_SYS (obj); +RISCVIOMMUState *iommu = &s->iommu; + +object_initialize_child (obj,"iommu" , iommu,TYPE_RISCV_IOMMU ); +qdev_alias_all_properties (DEVICE (iommu), obj); +} + +static Property riscv_iommu_sys_properties[] = { +DEFINE_PROP_UINT64 ("addr" , RISCVIOMMUStateSys, addr,0 ), +DEFINE_PROP_END_OF_LIST (), +}; + +static void riscv_iommu_sys_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); + dc->realize = riscv_iommu_sys_realize; +set_bit (DEVICE_CATEGORY_MISC , dc->categories); +device_class_set_props (dc, riscv_iommu_sys_properties); +} + +static const TypeInfo riscv_iommu_sys = { + .name =TYPE_RISCV_IOMMU_SYS , + .parent =TYPE_SYS_BUS_DEVICE , + .class_init = riscv_iommu_sys_class_init, + .instance_init = riscv_iommu_sys_init, + .instance_size =sizeof (RISCVIOMMUStateSys),
Quoted 9 lines
hw/riscv/riscv-iommu-sys.c line 121
Quoted 6 lines
hw/riscv/riscv-iommu-sys.c new file +6
-0+}; + +static void riscv_iommu_register_sys (void ) +{ +type_register_static (&riscv_iommu_sys); +}Same here
ack
Quoted 2,787 lines
Quoted 2,787 lines
hw/riscv/riscv-iommu-sys.c new file +2
-0+ +type_init (riscv_iommu_register_sys)hw/riscv/riscv-iommu.c new file +2539
-0@@ -0,0 +1,2539 @@ +/* +* QEMU emulation of an RISC-V IOMMU (Ziommu) +* +* Copyright (C) 2021-2023, Rivos Inc. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#include "qemu/osdep.h" +#include "qom/object.h" +#include "hw/pci/pci_bus.h" +#include "hw/pci/pci_device.h" +#include "hw/qdev-properties.h" +#include "hw/riscv/riscv_hart.h" +#include "migration/vmstate.h" +#include "qapi/error.h" +#include "qemu/timer.h" + +#include "cpu_bits.h" +#include "riscv-iommu.h" +#include "riscv-iommu-bits.h" +#include "trace.h" + +#define LIMIT_CACHE_CTX (1U << 7) +#define LIMIT_CACHE_IOT (1U << 20) + +/* Physical page number coversions */ +#define PPN_PHYS (ppn) ((ppn) << TARGET_PAGE_BITS) +#define PPN_DOWN (phy) ((phy) >> TARGET_PAGE_BITS) + +typedef struct RISCVIOMMUContext RISCVIOMMUContext ; +typedef struct RISCVIOMMUEntry RISCVIOMMUEntry ; + +/* Device assigned I/O address space */ +struct RISCVIOMMUSpace { +IOMMUMemoryRegion iova_mr;/* IOVA memory region for attached device */ +AddressSpace iova_as;/* IOVA address space for attached device */ +RISCVIOMMUState *iommu;/* Managing IOMMU device state */ +uint32_t devid;/* Requester identifier, AKA device_id */ +bool notifier;/* IOMMU unmap notifier enabled */ +QLIST_ENTRY (RISCVIOMMUSpace) list; +}; + +/* Device translation context state. */ +struct RISCVIOMMUContext { +uint64_t devid:24 ;/* Requester Id, AKA device_id */ +uint64_t pasid:20 ;/* Process Address Space ID */ +uint64_t __rfu:20 ;/* reserved */ +uint64_t tc;/* Translation Control */ +uint64_t ta;/* Translation Attributes */ +uint64_t satp;/* S-Stage address translation and protection */ +uint64_t gatp;/* G-Stage address translation and protection */ +uint64_t msi_addr_mask;/* MSI filtering - address mask */ +uint64_t msi_addr_pattern;/* MSI filtering - address pattern */ +uint64_t msiptp;/* MSI redirection page table pointer */ +}; + +/* Address translation cache entry */ +struct RISCVIOMMUEntry { +uint64_t iova:44 ;/* IOVA Page Number */ +uint64_t pscid:20 ;/* Process Soft-Context identifier */ +uint64_t phys:44 ;/* Physical Page Number */ +uint64_t gscid:16 ;/* Guest Soft-Context identifier */ +uint64_t perm:2 ;/* IOMMU_RW flags */ +uint64_t __rfu:2 ; +}; + +/* IOMMU index for transactions without PASID specified. */ +#define RISCV_IOMMU_NOPASID 0 + +static void riscv_iommu_notify (RISCVIOMMUState *s,int vec) +{ +const uint32_t ipsr = +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_IPSR , (1 << vec),0 ); +const uint32_t ivec =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IVEC ); +if (s->notify && !(ipsr & (1 << vec))) { + s->notify (s, (ivec >> (vec *4 )) &0x0F ); + } +} + +static void riscv_iommu_fault (RISCVIOMMUState *s,struct riscv_iommu_fq_record *ev) +{ +uint32_t ctrl =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_FQCSR ); +uint32_t head =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_FQH ) & s->fq_mask; +uint32_t tail =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_FQT ) & s->fq_mask; +uint32_t next = (tail +1 ) & s->fq_mask; +uint32_t devid =get_field (ev->hdr,RISCV_IOMMU_FQ_HDR_DID ); + +trace_riscv_iommu_flt (s->parent_obj.id,PCI_BUS_NUM (devid),PCI_SLOT (devid), +PCI_FUNC (devid), ev->hdr, ev->iotval); + +if (!(ctrl &RISCV_IOMMU_FQCSR_FQON ) || + !!(ctrl & (RISCV_IOMMU_FQCSR_FQOF |RISCV_IOMMU_FQCSR_FQMF ))) { +return ; + } + +if (head == next) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_FQCSR ,RISCV_IOMMU_FQCSR_FQOF ,0 ); + }else { +dma_addr_t addr = s->fq_addr + tail *sizeof (*ev); +if (dma_memory_write (s->target_as, addr, ev,sizeof (*ev), +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_FQCSR ,RISCV_IOMMU_FQCSR_FQMF ,0 ); + }else { +riscv_iommu_reg_set32 (s,RISCV_IOMMU_REG_FQT , next); + } + } + +if (ctrl &RISCV_IOMMU_FQCSR_FIE ) { +riscv_iommu_notify (s,RISCV_IOMMU_INTR_FQ ); + } +} + +static void riscv_iommu_pri (RISCVIOMMUState *s, +struct riscv_iommu_pq_record *pr) +{ +uint32_t ctrl =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_PQCSR ); +uint32_t head =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_PQH ) & s->pq_mask; +uint32_t tail =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_PQT ) & s->pq_mask; +uint32_t next = (tail +1 ) & s->pq_mask; +uint32_t devid =get_field (pr->hdr,RISCV_IOMMU_PREQ_HDR_DID ); + +trace_riscv_iommu_pri (s->parent_obj.id,PCI_BUS_NUM (devid),PCI_SLOT (devid), +PCI_FUNC (devid), pr->payload); + +if (!(ctrl &RISCV_IOMMU_PQCSR_PQON ) || + !!(ctrl & (RISCV_IOMMU_PQCSR_PQOF |RISCV_IOMMU_PQCSR_PQMF ))) { +return ; + } + +if (head == next) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_PQCSR ,RISCV_IOMMU_PQCSR_PQOF ,0 ); + }else { +dma_addr_t addr = s->pq_addr + tail *sizeof (*pr); +if (dma_memory_write (s->target_as, addr, pr,sizeof (*pr), +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_PQCSR ,RISCV_IOMMU_PQCSR_PQMF ,0 ); + }else { +riscv_iommu_reg_set32 (s,RISCV_IOMMU_REG_PQT , next); + } + } + +if (ctrl &RISCV_IOMMU_PQCSR_PIE ) { +riscv_iommu_notify (s,RISCV_IOMMU_INTR_PQ ); + } +} + +static void __hpm_incr_ctr (RISCVIOMMUState *s,uint32_t ctr_idx) +{ +const uint32_t off = ctr_idx <<3 ; +uint64_t cntr_val; + +qemu_spin_lock (&s->regs_lock); + cntr_val =ldq_le_p (&s->regs_rw[RISCV_IOMMU_REG_IOHPMCTR_BASE + off]); +stq_le_p (&s->regs_rw[RISCV_IOMMU_REG_IOHPMCTR_BASE + off], cntr_val +1 ); +qemu_spin_unlock (&s->regs_lock); + +/* Handle the overflow scenario. */ +if (cntr_val ==UINT64_MAX ) { +/* +* Generate interrupt only if OF bit is clear. +1 to offset the cycle +* register OF bit. +*/ +const uint32_t ovf = +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ,BIT (ctr_idx +1 ),0 ); +if (!get_field (ovf,BIT (ctr_idx +1 ))) { +riscv_iommu_reg_mod64 (s, +RISCV_IOMMU_REG_IOHPMEVT_BASE + off, +RISCV_IOMMU_IOHPMEVT_OF , +0 ); +riscv_iommu_notify (s,RISCV_IOMMU_INTR_PM ); + } + } +} + +static void riscv_iommu_hpm_incr_ctr (RISCVIOMMUState *s,RISCVIOMMUContext *ctx, +unsigned event_id) +{ +const uint32_t inhibit =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTINH ); +uint32_t did_gscid; +uint32_t pid_pscid; +uint32_t ctr_idx; +gpointer value; +uint32_t ctrs; +uint64_t evt; + +if (!(s->cap &RISCV_IOMMU_CAP_HPM )) { +return ; + } + +pthread_rwlock_rdlock (&s->ht_lock); + value =g_hash_table_lookup (s->hpm_event_ctr_map, +GUINT_TO_POINTER (event_id)); +if (value ==NULL ) { +pthread_rwlock_unlock (&s->ht_lock); +return ; + } + +for (ctrs =GPOINTER_TO_UINT (value); ctrs !=0 ; ctrs &= ctrs -1 ) { + ctr_idx =ctz32 (ctrs); +if (get_field (inhibit,BIT (ctr_idx +1 ))) { +continue ; + } + + evt =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_IOHPMEVT_BASE + (ctr_idx <<3 )); + +/* +* It's quite possible that event ID has been changed in counter +* but hashtable hasn't been updated yet. We don't want to increment +* counter for the old event ID. +*/ +if (event_id !=get_field (evt,RISCV_IOMMU_IOHPMEVT_EVENT_ID )) { +continue ; + } + +if (get_field (evt,RISCV_IOMMU_IOHPMEVT_IDT )) { + did_gscid =get_field (ctx->gatp,RISCV_IOMMU_DC_IOHGATP_GSCID ); + pid_pscid =get_field (ctx->ta,RISCV_IOMMU_DC_TA_PSCID ); + }else { + did_gscid = ctx->devid; + pid_pscid = ctx->pasid; + } + +if (get_field (evt,RISCV_IOMMU_IOHPMEVT_PV_PSCV )) { +/* +* If the transaction does not have a valid process_id, counter +* increments if device_id matches DID_GSCID. If the transaction has +* a valid process_id, counter increments if device_id matches +* DID_GSCID and process_id matches PID_PSCID. See IOMMU +* Specification, Chapter 5.23. Performance-monitoring event +* selector. +*/ +if (ctx->pasid && +get_field (evt,RISCV_IOMMU_IOHPMEVT_PID_PSCID ) != pid_pscid) { +continue ; + } + } + +if (get_field (evt,RISCV_IOMMU_IOHPMEVT_DV_GSCV )) { +uint32_t mask = ~0 ; + +if (get_field (evt,RISCV_IOMMU_IOHPMEVT_DMASK )) { +/* +* 1001 1011 mask = GSCID +* 0000 0111 mask = mask ^ (mask + 1) +* 1111 1000 mask = ~mask; +*/ + mask =get_field (evt,RISCV_IOMMU_IOHPMEVT_DID_GSCID ); + mask = mask ^ (mask +1 ); + mask = ~mask; + } + +if ((get_field (evt,RISCV_IOMMU_IOHPMEVT_DID_GSCID ) & mask) != + (did_gscid & mask)) { +continue ; + } + } + +__hpm_incr_ctr (s, ctr_idx); + } + +pthread_rwlock_unlock (&s->ht_lock); +} + +/* Portable implementation of pext_u64, bit-mask extraction. */ +static uint64_t _pext_u64 (uint64_t val,uint64_t ext) +{ +uint64_t ret =0 ; +uint64_t rot =1 ; + +while (ext) { +if (ext &1 ) { +if (val &1 ) { + ret |= rot; + } + rot <<=1 ; + } + val >>=1 ; + ext >>=1 ; + } + +return ret; +} + +/* Check if GPA matches MSI/MRIF pattern. */ +static bool riscv_iommu_msi_check (RISCVIOMMUState *s,RISCVIOMMUContext *ctx, +dma_addr_t gpa) +{ +if (get_field (ctx->msiptp,RISCV_IOMMU_DC_MSIPTP_MODE ) != +RISCV_IOMMU_DC_MSIPTP_MODE_FLAT ) { +return false;/* Invalid MSI/MRIF mode */ + } + +if ((PPN_DOWN (gpa) ^ ctx->msi_addr_pattern) & ~ctx->msi_addr_mask) { +return false;/* GPA not in MSI range defined by AIA IMSIC rules. */ + } + +return true; +} + +/* +* RISCV IOMMU Address Translation Lookup - Page Table Walk +* +* Note: Code is based on get_physical_address() from target/riscv/cpu_helper.c +* Both implementation can be merged into single helper function in future. +* Keeping them separate for now, as error reporting and flow specifics are +* sufficiently different for separate implementation. +* +* @s : IOMMU Device State +* @ctx : Translation context for device id and process address space id. +* @iotlb : translation data: physical address and access mode. +* @gpa : provided IOVA is a guest physical address, use G-Stage only. +* @return : success or fault cause code. +*/ +static int riscv_iommu_spa_fetch (RISCVIOMMUState *s,RISCVIOMMUContext *ctx, +IOMMUTLBEntry *iotlb,bool gpa) +{ +dma_addr_t addr, base; +uint64_t satp, gatp, pte; +bool en_s, en_g; +struct { +unsigned char step; +unsigned char levels; +unsigned char ptidxbits; +unsigned char ptesize; + } sc[2 ]; +/* Translation stage phase */ +enum { +S_STAGE =0 , +G_STAGE =1 , + } pass; + + satp =get_field (ctx->satp,RISCV_IOMMU_ATP_MODE_FIELD ); + gatp =get_field (ctx->gatp,RISCV_IOMMU_ATP_MODE_FIELD ); + + en_s = satp !=RISCV_IOMMU_DC_FSC_MODE_BARE && !gpa; + en_g = gatp !=RISCV_IOMMU_DC_IOHGATP_MODE_BARE ; + +/* Early check for MSI address match when IOVA == GPA */ +if (!en_s && (iotlb->perm &IOMMU_WO ) && +riscv_iommu_msi_check (s, ctx, iotlb->iova)) { + iotlb->target_as = &s->trap_as; + iotlb->translated_addr = iotlb->iova; + iotlb->addr_mask = ~TARGET_PAGE_MASK ; +return 0 ; + } + +/* Exit early for pass-through mode. */ +if (!(en_s || en_g)) { + iotlb->translated_addr = iotlb->iova; + iotlb->addr_mask = ~TARGET_PAGE_MASK ; +/* Allow R/W in pass-through mode */ + iotlb->perm =IOMMU_RW ; +return 0 ; + } + +/* S/G translation parameters. */ +for (pass =0 ; pass <2 ; pass++) { + sc[pass].step =0 ; +if (pass ? (s->fctl &RISCV_IOMMU_FCTL_GXL ) : + (ctx->tc &RISCV_IOMMU_DC_TC_SXL )) { +/* 32bit mode for GXL/SXL == 1 */ +switch (pass ? gatp : satp) { +case RISCV_IOMMU_DC_IOHGATP_MODE_BARE : + sc[pass].levels =0 ; + sc[pass].ptidxbits =0 ; + sc[pass].ptesize =0 ; +break ; +case RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4 : +if (!(s->cap & + (pass ?RISCV_IOMMU_CAP_G_SV32 :RISCV_IOMMU_CAP_S_SV32 ))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + sc[pass].levels =2 ; + sc[pass].ptidxbits =10 ; + sc[pass].ptesize =4 ; +break ; +default : +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + }else { +/* 64bit mode for GXL/SXL == 0 */ +switch (pass ? gatp : satp) { +case RISCV_IOMMU_DC_IOHGATP_MODE_BARE : + sc[pass].levels =0 ; + sc[pass].ptidxbits =0 ; + sc[pass].ptesize =0 ; +break ; +case RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4 : +if (!(s->cap & + (pass ?RISCV_IOMMU_CAP_G_SV39 :RISCV_IOMMU_CAP_S_SV39 ))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + sc[pass].levels =3 ; + sc[pass].ptidxbits =9 ; + sc[pass].ptesize =8 ; +break ; +case RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4 : +if (!(s->cap & + (pass ?RISCV_IOMMU_CAP_G_SV48 :RISCV_IOMMU_CAP_S_SV48 ))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + sc[pass].levels =4 ; + sc[pass].ptidxbits =9 ; + sc[pass].ptesize =8 ; +break ; +case RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4 : +if (!(s->cap & + (pass ?RISCV_IOMMU_CAP_G_SV57 :RISCV_IOMMU_CAP_S_SV57 ))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + sc[pass].levels =5 ; + sc[pass].ptidxbits =9 ; + sc[pass].ptesize =8 ; +break ; +default : +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + } + }; + +/* S/G stages translation tables root pointers */ + gatp =PPN_PHYS (get_field (ctx->gatp,RISCV_IOMMU_ATP_PPN_FIELD )); + satp =PPN_PHYS (get_field (ctx->satp,RISCV_IOMMU_ATP_PPN_FIELD )); + addr = (en_s && en_g) ? satp : iotlb->iova; + base = en_g ? gatp : satp; + pass = en_g ?G_STAGE :S_STAGE ; + +do { +const unsigned widened = (pass && !sc[pass].step) ?2 :0 ; +const unsigned va_bits = widened + sc[pass].ptidxbits; +const unsigned va_skip =TARGET_PAGE_BITS + sc[pass].ptidxbits * + (sc[pass].levels -1 - sc[pass].step); +const unsigned idx = (addr >> va_skip) & ((1 << va_bits) -1 ); +const dma_addr_t pte_addr = base + idx * sc[pass].ptesize; +const bool ade = + ctx->tc & (pass ?RISCV_IOMMU_DC_TC_GADE :RISCV_IOMMU_DC_TC_SADE ); + +/* Address range check before first level lookup */ +if (!sc[pass].step) { +const uint64_t va_mask = (1ULL << (va_skip + va_bits)) -1 ; +if ((addr & va_mask) != addr) { +return RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED ; + } + } + +/* Read page table entry */ +if (dma_memory_read (s->target_as, pte_addr, &pte, + sc[pass].ptesize,MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return (iotlb->perm &IOMMU_WO ) ?RISCV_IOMMU_FQ_CAUSE_WR_FAULT + :RISCV_IOMMU_FQ_CAUSE_RD_FAULT ; + } + +if (pass ==S_STAGE ) { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_S_VS_WALKS ); + }else { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_G_WALKS ); + } + +if (sc[pass].ptesize ==4 ) { + pte = (uint64_t )le32_to_cpu (*((uint32_t *)&pte)); + }else { + pte =le64_to_cpu (pte); + } + + sc[pass].step++; +hwaddr ppn = pte >>PTE_PPN_SHIFT ; + +if (!(pte &PTE_V )) { +break ;/* Invalid PTE */ + }else if (!(pte & (PTE_R |PTE_W |PTE_X ))) { + base =PPN_PHYS (ppn);/* Inner PTE, continue walking */ + }else if ((pte & (PTE_R |PTE_W |PTE_X )) ==PTE_W ) { +break ;/* Reserved leaf PTE flags: PTE_W */ + }else if ((pte & (PTE_R |PTE_W |PTE_X )) == (PTE_W |PTE_X )) { +break ;/* Reserved leaf PTE flags: PTE_W + PTE_X */ + }else if (ppn & ((1ULL << (va_skip -TARGET_PAGE_BITS )) -1 )) { +break ;/* Misaligned PPN */ + }else if ((iotlb->perm &IOMMU_RO ) && !(pte &PTE_R )) { +break ;/* Read access check failed */ + }else if ((iotlb->perm &IOMMU_WO ) && !(pte &PTE_W )) { +break ;/* Write access check failed */ + }else if ((iotlb->perm &IOMMU_RO ) && !ade && !(pte &PTE_A )) { +break ;/* Access bit not set */ + }else if ((iotlb->perm &IOMMU_WO ) && !ade && !(pte &PTE_D )) { +break ;/* Dirty bit not set */ + }else { +/* Leaf PTE, translation completed. */ + sc[pass].step = sc[pass].levels; + base =PPN_PHYS (ppn) | (addr & ((1ULL << va_skip) -1 )); +/* Update address mask based on smallest translation granularity */ + iotlb->addr_mask &= (1ULL << va_skip) -1 ; +/* Continue with S-Stage translation? */ +if (pass && sc[0 ].step != sc[0 ].levels) { + pass =S_STAGE ; + addr = iotlb->iova; +continue ; + } +/* Translation phase completed (GPA or SPA) */ + iotlb->translated_addr = base; + iotlb->perm = (pte &PTE_W ) ? ((pte &PTE_R ) ?IOMMU_RW :IOMMU_WO ) + :IOMMU_RO ; + +/* Check MSI GPA address match */ +if (pass ==S_STAGE && (iotlb->perm &IOMMU_WO ) && +riscv_iommu_msi_check (s, ctx, base)) { +/* Trap MSI writes and return GPA address. */ + iotlb->target_as = &s->trap_as; + iotlb->addr_mask = ~TARGET_PAGE_MASK ; +return 0 ; + } + +/* Continue with G-Stage translation? */ +if (!pass && en_g) { + pass =G_STAGE ; + addr = base; + base = gatp; + sc[pass].step =0 ; +continue ; + } + +return 0 ; + } + +if (sc[pass].step == sc[pass].levels) { +break ;/* Can't find leaf PTE */ + } + +/* Continue with G-Stage translation? */ +if (!pass && en_g) { + pass =G_STAGE ; + addr = base; + base = gatp; + sc[pass].step =0 ; + } + }while (1 ); + +return (iotlb->perm &IOMMU_WO ) ? + (pass ?RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS : +RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S ) : + (pass ?RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS : +RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S ); +} + +/* Redirect MSI write for given GPA. */ +static MemTxResult riscv_iommu_msi_write (RISCVIOMMUState *s, +RISCVIOMMUContext *ctx,uint64_t gpa,uint64_t data, +unsigned size,MemTxAttrs attrs) +{ +MemTxResult res; +dma_addr_t addr; +uint64_t intn; +uint32_t n190; +uint64_t pte[2 ]; + +if (!riscv_iommu_msi_check (s, ctx, gpa)) { +return MEMTX_ACCESS_ERROR ; + } + +/* Interrupt File Number */ + intn =_pext_u64 (PPN_DOWN (gpa), ctx->msi_addr_mask); +if (intn >=256 ) { +/* Interrupt file number out of range */ +return MEMTX_ACCESS_ERROR ; + } + +/* fetch MSI PTE */ + addr =PPN_PHYS (get_field (ctx->msiptp,RISCV_IOMMU_DC_MSIPTP_PPN )); + addr = addr | (intn *sizeof (pte)); + res =dma_memory_read (s->target_as, addr, &pte,sizeof (pte), +MEMTXATTRS_UNSPECIFIED ); +if (res !=MEMTX_OK ) { +return res; + } + +le64_to_cpus (&pte[0 ]); +le64_to_cpus (&pte[1 ]); + +if (!(pte[0 ] &RISCV_IOMMU_MSI_PTE_V ) || (pte[0 ] &RISCV_IOMMU_MSI_PTE_C )) { +return MEMTX_ACCESS_ERROR ; + } + +switch (get_field (pte[0 ],RISCV_IOMMU_MSI_PTE_M )) { +case RISCV_IOMMU_MSI_PTE_M_BASIC : +/* MSI Pass-through mode */ + addr =PPN_PHYS (get_field (pte[0 ],RISCV_IOMMU_MSI_PTE_PPN )); + addr = addr | (gpa &TARGET_PAGE_MASK ); + +trace_riscv_iommu_msi (s->parent_obj.id,PCI_BUS_NUM (ctx->devid), +PCI_SLOT (ctx->devid),PCI_FUNC (ctx->devid), + gpa, addr); + +return dma_memory_write (s->target_as, addr, &data, size, attrs); +case RISCV_IOMMU_MSI_PTE_M_MRIF : +/* MRIF mode, continue. */ +break ; +default : +return MEMTX_ACCESS_ERROR ; + } + +/* +* Report an error for interrupt identities exceeding the maximum allowed +* for an IMSIC interrupt file (2047) or destination address is not 32-bit +* aligned. See IOMMU Specification, Chapter 2.3. MSI page tables. +*/ +if ((data >2047 ) || (gpa &3 )) { +return MEMTX_ACCESS_ERROR ; + } + +/* MSI MRIF mode, non atomic pending bit update */ + +/* MRIF pending bit address */ + addr =get_field (pte[0 ],RISCV_IOMMU_MSI_PTE_MRIF_ADDR ) <<9 ; + addr = addr | ((data &0x7c0 ) >>3 ); + +trace_riscv_iommu_msi (s->parent_obj.id,PCI_BUS_NUM (ctx->devid), +PCI_SLOT (ctx->devid),PCI_FUNC (ctx->devid), + gpa, addr); + +/* MRIF pending bit mask */ + data =1ULL << (data &0x03f ); + res =dma_memory_read (s->target_as, addr, &intn,sizeof (intn), attrs); +if (res !=MEMTX_OK ) { +return res; + } + intn = intn | data; + res =dma_memory_write (s->target_as, addr, &intn,sizeof (intn), attrs); +if (res !=MEMTX_OK ) { +return res; + } + +/* Get MRIF enable bits */ + addr = addr +sizeof (intn); + res =dma_memory_read (s->target_as, addr, &intn,sizeof (intn), attrs); +if (res !=MEMTX_OK ) { +return res; + } +if (!(intn & data)) { +/* notification disabled, MRIF update completed. */ +return MEMTX_OK ; + } + +/* Send notification message */ + addr =PPN_PHYS (get_field (pte[1 ],RISCV_IOMMU_MSI_MRIF_NPPN )); + n190 =get_field (pte[1 ],RISCV_IOMMU_MSI_MRIF_NID ) | + (get_field (pte[1 ],RISCV_IOMMU_MSI_MRIF_NID_MSB ) <<10 ); + + res =dma_memory_write (s->target_as, addr, &n190,sizeof (n190), attrs); +if (res !=MEMTX_OK ) { +return res; + } + +return MEMTX_OK ; +} + +/* +* Device Context format. +* +* @s : IOMMU Device State +* @return : 0: extended (64 bytes) | 1: base (32 bytes) +*/ +static int riscv_iommu_dc_is_base (RISCVIOMMUState *s) +{ +return !(s->cap &RISCV_IOMMU_CAP_MSI_FLAT ); +} + +/* +* RISC-V IOMMU Device Context Loopkup - Device Directory Tree Walk +* +* @s : IOMMU Device State +* @ctx : Device Translation Context with devid and pasid set. +* @return : success or fault code. +*/ +static int riscv_iommu_ctx_fetch (RISCVIOMMUState *s,RISCVIOMMUContext *ctx) +{ +const uint64_t ddtp = s->ddtp; +unsigned mode =get_field (ddtp,RISCV_IOMMU_DDTP_MODE ); +dma_addr_t addr =PPN_PHYS (get_field (ddtp,RISCV_IOMMU_DDTP_PPN )); +struct riscv_iommu_dc dc; +const int dc_fmt =riscv_iommu_dc_is_base (s); +const size_t dc_len =sizeof (dc) >> dc_fmt; +unsigned depth; +uint64_t de; + +switch (mode) { +case RISCV_IOMMU_DDTP_MODE_OFF : +return RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED ; + +case RISCV_IOMMU_DDTP_MODE_BARE : +/* mock up pass-through translation context */ + ctx->gatp =set_field (0 ,RISCV_IOMMU_ATP_MODE_FIELD , +RISCV_IOMMU_DC_IOHGATP_MODE_BARE ); + ctx->satp =set_field (0 ,RISCV_IOMMU_ATP_MODE_FIELD , +RISCV_IOMMU_DC_FSC_MODE_BARE ); + ctx->tc =RISCV_IOMMU_DC_TC_EN_ATS |RISCV_IOMMU_DC_TC_V ; + ctx->ta =0 ; + ctx->msiptp =0 ; +return 0 ; + +case RISCV_IOMMU_DDTP_MODE_1LVL : + depth =0 ; +break ; + +case RISCV_IOMMU_DDTP_MODE_2LVL : + depth =1 ; +break ; + +case RISCV_IOMMU_DDTP_MODE_3LVL : + depth =2 ; +break ; + +default : +return RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED ; + } + +/* +* Check supported device id width (in bits). +* See IOMMU Specification, Chapter 6. Software guidelines. +* - if extended device-context format is used: +* 1LVL: 6, 2LVL: 15, 3LVL: 24 +* - if base device-context format is used: +* 1LVL: 7, 2LVL: 16, 3LVL: 24 +*/ +if (ctx->devid >= (1 << (depth *9 +6 + (dc_fmt && depth !=2 )))) { +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ; + } + +/* Device directory tree walk */ +for (; depth-- >0 ; ) { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_DD_WALK ); + +/* +* Select device id index bits based on device directory tree level +* and device context format. +* See IOMMU Specification, Chapter 2. Data Structures. +* - if extended device-context format is used: +* device index: [23:15][14:6][5:0] +* - if base device-context format is used: +* device index: [23:16][15:7][6:0] +*/ +const int split = depth *9 +6 + dc_fmt; + addr |= ((ctx->devid >> split) <<3 ) & ~TARGET_PAGE_MASK ; +if (dma_memory_read (s->target_as, addr, &de,sizeof (de), +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT ; + } +le64_to_cpus (&de); +if (!(de &RISCV_IOMMU_DDTE_VALID )) { +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ;/* invalid directory entry */ + } +if (de & ~(RISCV_IOMMU_DDTE_PPN |RISCV_IOMMU_DDTE_VALID )) { +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ;/* reserved bits set. */ + } + addr =PPN_PHYS (get_field (de,RISCV_IOMMU_DDTE_PPN )); + } + +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_DD_WALK ); + +/* index into device context entry page */ + addr |= (ctx->devid * dc_len) & ~TARGET_PAGE_MASK ; + +memset (&dc,0 ,sizeof (dc)); +if (dma_memory_read (s->target_as, addr, &dc, dc_len, +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT ; + } + +/* Set translation context. */ + ctx->tc =le64_to_cpu (dc.tc); + ctx->gatp =le64_to_cpu (dc.iohgatp); + ctx->satp =le64_to_cpu (dc.fsc); + ctx->ta =le64_to_cpu (dc.ta); + ctx->msiptp =le64_to_cpu (dc.msiptp); + ctx->msi_addr_mask =le64_to_cpu (dc.msi_addr_mask); + ctx->msi_addr_pattern =le64_to_cpu (dc.msi_addr_pattern); + +if (!(ctx->tc &RISCV_IOMMU_DC_TC_V )) { +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ; + } + +/* FSC field checks */ + mode =get_field (ctx->satp,RISCV_IOMMU_DC_FSC_MODE ); + addr =PPN_PHYS (get_field (ctx->satp,RISCV_IOMMU_DC_FSC_PPN )); + +if (mode ==RISCV_IOMMU_DC_FSC_MODE_BARE ) { +/* No S-Stage translation, done. */ +return 0 ; + } + +if (!(ctx->tc &RISCV_IOMMU_DC_TC_PDTV )) { +if (ctx->pasid !=RISCV_IOMMU_NOPASID ) { +/* PASID is disabled */ +return RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED ; + } +if (mode >RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57 ) { +/* Invalid translation mode */ +return RISCV_IOMMU_FQ_CAUSE_DDT_INVALID ; + } +return 0 ; + } + +if (ctx->pasid ==RISCV_IOMMU_NOPASID ) { +if (!(ctx->tc &RISCV_IOMMU_DC_TC_DPE )) { +/* No default PASID enabled, set BARE mode */ + ctx->satp =0ULL ; +return 0 ; + }else { +/* Use default PASID #0 */ + ctx->pasid =0 ; + } + } + +/* FSC.TC.PDTV enabled */ +if (mode >RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20 ) { +/* Invalid PDTP.MODE */ +return RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED ; + } + +for (depth = mode -RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8 ; depth-- >0 ; ) { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_PD_WALK ); + +/* +* Select process id index bits based on process directory tree +* level. See IOMMU Specification, 2.2. Process-Directory-Table. +*/ +const int split = depth *9 +8 ; + addr |= ((ctx->pasid >> split) <<3 ) & ~TARGET_PAGE_MASK ; +if (dma_memory_read (s->target_as, addr, &de,sizeof (de), +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT ; + } +le64_to_cpus (&de); +if (!(de &RISCV_IOMMU_PC_TA_V )) { +return RISCV_IOMMU_FQ_CAUSE_PDT_INVALID ; + } + addr =PPN_PHYS (get_field (de,RISCV_IOMMU_PC_FSC_PPN )); + } + +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_PD_WALK ); + +/* Leaf entry in PDT */ + addr |= (ctx->pasid <<4 ) & ~TARGET_PAGE_MASK ; +if (dma_memory_read (s->target_as, addr, &dc.ta,sizeof (uint64_t ) *2 , +MEMTXATTRS_UNSPECIFIED ) !=MEMTX_OK ) { +return RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT ; + } + +/* Use FSC and TA from process directory entry. */ + ctx->ta =le64_to_cpu (dc.ta); + ctx->satp =le64_to_cpu (dc.fsc); + +return 0 ; +} + +/* Translation Context cache support */ +static gboolean __ctx_equal (gconstpointer v1,gconstpointer v2) +{ +RISCVIOMMUContext *c1 = (RISCVIOMMUContext *) v1; +RISCVIOMMUContext *c2 = (RISCVIOMMUContext *) v2; +return c1->devid == c2->devid && c1->pasid == c2->pasid; +} + +static guint __ctx_hash (gconstpointer v) +{ +RISCVIOMMUContext *ctx = (RISCVIOMMUContext *) v; +/* Generate simple hash of (pasid, devid), assuming 24-bit wide devid */ +return (guint)(ctx->devid) + ((guint)(ctx->pasid) <<24 ); +} + +static void __ctx_inval_devid_pasid (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUContext *ctx = (RISCVIOMMUContext *) value; +RISCVIOMMUContext *arg = (RISCVIOMMUContext *) data; +if (ctx->tc &RISCV_IOMMU_DC_TC_V && + ctx->devid == arg->devid && + ctx->pasid == arg->pasid) { + ctx->tc &= ~RISCV_IOMMU_DC_TC_V ; + } +} + +static void __ctx_inval_devid (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUContext *ctx = (RISCVIOMMUContext *) value; +RISCVIOMMUContext *arg = (RISCVIOMMUContext *) data; +if (ctx->tc &RISCV_IOMMU_DC_TC_V && + ctx->devid == arg->devid) { + ctx->tc &= ~RISCV_IOMMU_DC_TC_V ; + } +} + +static void __ctx_inval_all (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUContext *ctx = (RISCVIOMMUContext *) value; +if (ctx->tc &RISCV_IOMMU_DC_TC_V ) { + ctx->tc &= ~RISCV_IOMMU_DC_TC_V ; + } +} + +static void riscv_iommu_ctx_inval (RISCVIOMMUState *s,GHFunc func, +uint32_t devid,uint32_t pasid) +{ +GHashTable *ctx_cache; +RISCVIOMMUContext key = { + .devid = devid, + .pasid = pasid, + }; + ctx_cache =g_hash_table_ref (s->ctx_cache); +g_hash_table_foreach (ctx_cache, func, &key); +g_hash_table_unref (ctx_cache); +} + +/* Find or allocate translation context for a given {device_id, process_id} */ +static RISCVIOMMUContext *riscv_iommu_ctx (RISCVIOMMUState *s, +unsigned devid,unsigned pasid,void **ref) +{ +GHashTable *ctx_cache; +RISCVIOMMUContext *ctx; +RISCVIOMMUContext key = { + .devid = devid, + .pasid = pasid, + }; + + ctx_cache =g_hash_table_ref (s->ctx_cache); + ctx =g_hash_table_lookup (ctx_cache, &key); + +if (ctx && (ctx->tc &RISCV_IOMMU_DC_TC_V )) { + *ref = ctx_cache; +return ctx; + } + +if (g_hash_table_size (s->ctx_cache) >=LIMIT_CACHE_CTX ) { + ctx_cache =g_hash_table_new_full (__ctx_hash, __ctx_equal, + g_free,NULL ); +g_hash_table_unref (qatomic_xchg (&s->ctx_cache, ctx_cache)); + } + + ctx =g_new0 (RISCVIOMMUContext,1 ); + ctx->devid = devid; + ctx->pasid = pasid; + +int fault =riscv_iommu_ctx_fetch (s, ctx); +if (!fault) { +g_hash_table_add (ctx_cache, ctx); + *ref = ctx_cache; +return ctx; + } + +g_hash_table_unref (ctx_cache); + *ref =NULL ; + +if (!(ctx->tc &RISCV_IOMMU_DC_TC_DTF )) { +struct riscv_iommu_fq_record ev = {0 }; + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_CAUSE , fault); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_TTYPE , +RISCV_IOMMU_FQ_TTYPE_UADDR_RD ); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_DID , devid); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_PID , pasid); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_PV , !!pasid); +riscv_iommu_fault (s, &ev); + } + +g_free (ctx); +return NULL ; +} + +static void riscv_iommu_ctx_put (RISCVIOMMUState *s,void *ref) +{ +if (ref) { +g_hash_table_unref ((GHashTable *)ref); + } +} + +/* Find or allocate address space for a given device */ +static AddressSpace *riscv_iommu_space (RISCVIOMMUState *s,uint32_t devid) +{ +RISCVIOMMUSpace *as; + +/* FIXME: PCIe bus remapping for attached endpoints. */ + devid |= s->bus <<8 ; + +qemu_mutex_lock (&s->core_lock); + QLIST_FOREACH(as, &s->spaces, list) { +if (as->devid == devid) { +break ; + } + } +qemu_mutex_unlock (&s->core_lock); + +if (as ==NULL ) { +char name[64 ]; + as =g_new0 (RISCVIOMMUSpace,1 ); + + as->iommu = s; + as->devid = devid; + +snprintf (name,sizeof (name),"riscv-iommu-%04x:%02x.%d-iova" , +PCI_BUS_NUM (as->devid),PCI_SLOT (as->devid),PCI_FUNC (as->devid)); + +/* IOVA address space, untranslated addresses */ +memory_region_init_iommu (&as->iova_mr,sizeof (as->iova_mr), +TYPE_RISCV_IOMMU_MEMORY_REGION , +OBJECT (as), name,UINT64_MAX ); +address_space_init (&as->iova_as,MEMORY_REGION (&as->iova_mr), +TYPE_RISCV_IOMMU_PCI ); + +qemu_mutex_lock (&s->core_lock); +QLIST_INSERT_HEAD (&s->spaces, as, list); +qemu_mutex_unlock (&s->core_lock); + +trace_riscv_iommu_new (s->parent_obj.id,PCI_BUS_NUM (as->devid), +PCI_SLOT (as->devid),PCI_FUNC (as->devid)); + } +return &as->iova_as; +} + +/* Translation Object cache support */ +static gboolean __iot_equal (gconstpointer v1,gconstpointer v2) +{ +RISCVIOMMUEntry *t1 = (RISCVIOMMUEntry *) v1; +RISCVIOMMUEntry *t2 = (RISCVIOMMUEntry *) v2; +return t1->gscid == t2->gscid && t1->pscid == t2->pscid && + t1->iova == t2->iova; +} + +static guint __iot_hash (gconstpointer v) +{ +RISCVIOMMUEntry *t = (RISCVIOMMUEntry *) v; +return (guint )t->iova; +} + +/* GV: 1 PSCV: 1 AV: 1 */ +static void __iot_inval_pscid_iova (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; +RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; +if (iot->gscid == arg->gscid && + iot->pscid == arg->pscid && + iot->iova == arg->iova) { + iot->perm =0 ; + } +} + +/* GV: 1 PSCV: 1 AV: 0 */ +static void __iot_inval_pscid (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; +RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; +if (iot->gscid == arg->gscid && + iot->pscid == arg->pscid) { + iot->perm =0 ; + } +} + +/* GV: 1 GVMA: 1 */ +static void __iot_inval_gscid_gpa (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; +RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; +if (iot->gscid == arg->gscid) { +/* simplified cache, no GPA matching */ + iot->perm =0 ; + } +} + +/* GV: 1 GVMA: 0 */ +static void __iot_inval_gscid (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; +RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; +if (iot->gscid == arg->gscid) { + iot->perm =0 ; + } +} + +/* GV: 0 */ +static void __iot_inval_all (gpointer key,gpointer value,gpointer data) +{ +RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; + iot->perm =0 ; +} + +/* caller should keep ref-count for iot_cache object */ +static RISCVIOMMUEntry *riscv_iommu_iot_lookup (RISCVIOMMUContext *ctx, +GHashTable *iot_cache,hwaddr iova) +{ +RISCVIOMMUEntry key = { + .gscid =get_field (ctx->gatp,RISCV_IOMMU_DC_IOHGATP_GSCID ), + .pscid =get_field (ctx->ta,RISCV_IOMMU_DC_TA_PSCID ), + .iova =PPN_DOWN (iova), + }; +return g_hash_table_lookup (iot_cache, &key); +} + +/* caller should keep ref-count for iot_cache object */ +static void riscv_iommu_iot_update (RISCVIOMMUState *s, +GHashTable *iot_cache,RISCVIOMMUEntry *iot) +{ +if (!s->iot_limit) { +return ; + } + +if (g_hash_table_size (s->iot_cache) >= s->iot_limit) { + iot_cache =g_hash_table_new_full (__iot_hash, __iot_equal, + g_free,NULL ); +g_hash_table_unref (qatomic_xchg (&s->iot_cache, iot_cache)); + } +g_hash_table_add (iot_cache, iot); +} + +static void riscv_iommu_iot_inval (RISCVIOMMUState *s,GHFunc func, +uint32_t gscid,uint32_t pscid,hwaddr iova) +{ +GHashTable *iot_cache; +RISCVIOMMUEntry key = { + .gscid = gscid, + .pscid = pscid, + .iova =PPN_DOWN (iova), + }; + + iot_cache =g_hash_table_ref (s->iot_cache); +g_hash_table_foreach (iot_cache, func, &key); +g_hash_table_unref (iot_cache); +} + +static int riscv_iommu_translate (RISCVIOMMUState *s,RISCVIOMMUContext *ctx, +IOMMUTLBEntry *iotlb,bool enable_cache) +{ +RISCVIOMMUEntry *iot; +IOMMUAccessFlags perm; +bool enable_faults; +bool enable_pasid; +bool enable_pri; +GHashTable *iot_cache; +int fault; + +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_URQ ); + + iot_cache =g_hash_table_ref (s->iot_cache); + + enable_faults = !(ctx->tc &RISCV_IOMMU_DC_TC_DTF ); +/* +* TC[32] is reserved for custom extensions, used here to temporarily +* enable automatic page-request generation for ATS queries. +*/ + enable_pri = (iotlb->perm ==IOMMU_NONE ) && (ctx->tc &BIT_ULL (32 )); + enable_pasid = (ctx->tc &RISCV_IOMMU_DC_TC_PDTV ); + +/* Check for ATS request. */ +if (iotlb->perm ==IOMMU_NONE ) { +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_ATS_RQ ); +/* Check if ATS is disabled. */ +if (!(ctx->tc &RISCV_IOMMU_DC_TC_EN_ATS )) { + enable_pri = false; + fault =RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED ; + goto done; + } +trace_riscv_iommu_ats (s->parent_obj.id,PCI_BUS_NUM (ctx->devid), +PCI_SLOT (ctx->devid),PCI_FUNC (ctx->devid), iotlb->iova); + } + + iot =riscv_iommu_iot_lookup (ctx, iot_cache, iotlb->iova); + perm = iot ? iot->perm :IOMMU_NONE ; +if (perm !=IOMMU_NONE ) { + iotlb->translated_addr =PPN_PHYS (iot->phys); + iotlb->addr_mask = ~TARGET_PAGE_MASK ; + iotlb->perm = perm; + fault =0 ; + goto done; + } + +riscv_iommu_hpm_incr_ctr (s, ctx,RISCV_IOMMU_HPMEVENT_TLB_MISS ); + +/* Translate using device directory / page table information. */ + fault =riscv_iommu_spa_fetch (s, ctx, iotlb, false); + +if (!fault && iotlb->target_as == &s->trap_as) { +/* Do not cache trapped MSI translations */ + goto done; + } + +if (!fault && iotlb->translated_addr != iotlb->iova && enable_cache) { + iot =g_new0 (RISCVIOMMUEntry,1 ); + iot->iova =PPN_DOWN (iotlb->iova); + iot->phys =PPN_DOWN (iotlb->translated_addr); + iot->gscid =get_field (ctx->gatp,RISCV_IOMMU_DC_IOHGATP_GSCID ); + iot->pscid =get_field (ctx->ta,RISCV_IOMMU_DC_TA_PSCID ); + iot->perm = iotlb->perm; +riscv_iommu_iot_update (s, iot_cache, iot); + } + +done: +g_hash_table_unref (iot_cache); + +if (enable_pri && fault) { +struct riscv_iommu_pq_record pr = {0 }; +if (enable_pasid) { + pr.hdr =set_field (RISCV_IOMMU_PREQ_HDR_PV , +RISCV_IOMMU_PREQ_HDR_PID , ctx->pasid); + } + pr.hdr =set_field (pr.hdr,RISCV_IOMMU_PREQ_HDR_DID , ctx->devid); + pr.payload = (iotlb->iova &TARGET_PAGE_MASK ) |RISCV_IOMMU_PREQ_PAYLOAD_M ; +riscv_iommu_pri (s, &pr); +return fault; + } + +if (enable_faults && fault) { +struct riscv_iommu_fq_record ev; +const unsigned ttype = + (iotlb->perm &IOMMU_RW ) ?RISCV_IOMMU_FQ_TTYPE_UADDR_WR : + ((iotlb->perm &IOMMU_RO ) ?RISCV_IOMMU_FQ_TTYPE_UADDR_RD : +RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ ); + ev.hdr =set_field (0 ,RISCV_IOMMU_FQ_HDR_CAUSE , fault); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_TTYPE , ttype); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_PV , enable_pasid); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_PID , ctx->pasid); + ev.hdr =set_field (ev.hdr,RISCV_IOMMU_FQ_HDR_DID , ctx->devid); + ev.iotval = iotlb->iova; + ev.iotval2 = iotlb->translated_addr; + ev._reserved =0 ; +riscv_iommu_fault (s, &ev); +return fault; + } + +return 0 ; +} + +/* IOMMU Command Interface */ +static MemTxResult riscv_iommu_iofence (RISCVIOMMUState *s,bool notify, +uint64_t addr,uint32_t data) +{ +/* +* ATS processing in this implementation of the IOMMU is synchronous, +* no need to wait for completions here. +*/ +if (!notify) { +return MEMTX_OK ; + } + +return dma_memory_write (s->target_as, addr, &data,sizeof (data), +MEMTXATTRS_UNSPECIFIED ); +} + +static void riscv_iommu_ats (RISCVIOMMUState *s, +struct riscv_iommu_command *cmd,IOMMUNotifierFlag flag, +IOMMUAccessFlags perm, +void (*trace_fn)(const char *id)) +{ +RISCVIOMMUSpace *as =NULL ; +IOMMUNotifier *n; +IOMMUTLBEvent event; +uint32_t pasid; +uint32_t devid; +const bool pv = cmd->dword0 &RISCV_IOMMU_CMD_ATS_PV ; + +if (cmd->dword0 &RISCV_IOMMU_CMD_ATS_DSV ) { +/* Use device segment and requester id */ + devid =get_field (cmd->dword0, +RISCV_IOMMU_CMD_ATS_DSEG |RISCV_IOMMU_CMD_ATS_RID ); + }else { + devid =get_field (cmd->dword0,RISCV_IOMMU_CMD_ATS_RID ); + } + + pasid =get_field (cmd->dword0,RISCV_IOMMU_CMD_ATS_PID ); + +qemu_mutex_lock (&s->core_lock); + QLIST_FOREACH(as, &s->spaces, list) { +if (as->devid == devid) { +break ; + } + } +qemu_mutex_unlock (&s->core_lock); + +if (!as || !as->notifier) { +return ; + } + + event.type = flag; + event.entry.perm = perm; + event.entry.target_as = s->target_as; + + IOMMU_NOTIFIER_FOREACH(n, &as->iova_mr) { +if (!pv || n->iommu_idx == pasid) { + event.entry.iova = n->start; + event.entry.addr_mask = n->end - n->start; +trace_fn (as->iova_mr.parent_obj.name); +memory_region_notify_iommu_one (n, &event); + } + } +} + +static void riscv_iommu_ats_inval (RISCVIOMMUState *s, +struct riscv_iommu_command *cmd) +{ +return riscv_iommu_ats (s, cmd,IOMMU_NOTIFIER_DEVIOTLB_UNMAP ,IOMMU_NONE , + trace_riscv_iommu_ats_inval); +} + +static void riscv_iommu_ats_prgr (RISCVIOMMUState *s, +struct riscv_iommu_command *cmd) +{ +unsigned resp_code =get_field (cmd->dword1,RISCV_IOMMU_CMD_ATS_PRGR_RESP_CODE ); +/* Using the access flag to carry response code information */ +IOMMUAccessFlags perm = resp_code ?IOMMU_NONE :IOMMU_RW ; +return riscv_iommu_ats (s, cmd,IOMMU_NOTIFIER_MAP , perm, + trace_riscv_iommu_ats_prgr); +} + +static void riscv_iommu_process_ddtp (RISCVIOMMUState *s) +{ +uint64_t old_ddtp = s->ddtp; +uint64_t new_ddtp =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_DDTP ); +unsigned new_mode =get_field (new_ddtp,RISCV_IOMMU_DDTP_MODE ); +unsigned old_mode =get_field (old_ddtp,RISCV_IOMMU_DDTP_MODE ); +bool ok = false; + +/* +* Check for allowed DDTP.MODE transitions: +* {OFF, BARE} -> {OFF, BARE, 1LVL, 2LVL, 3LVL} +* {1LVL, 2LVL, 3LVL} -> {OFF, BARE} +*/ +if (new_mode == old_mode || + new_mode ==RISCV_IOMMU_DDTP_MODE_OFF || + new_mode ==RISCV_IOMMU_DDTP_MODE_BARE ) { + ok = true; + }else if (new_mode ==RISCV_IOMMU_DDTP_MODE_1LVL || + new_mode ==RISCV_IOMMU_DDTP_MODE_2LVL || + new_mode ==RISCV_IOMMU_DDTP_MODE_3LVL ) { + ok = old_mode ==RISCV_IOMMU_DDTP_MODE_OFF || + old_mode ==RISCV_IOMMU_DDTP_MODE_BARE ; + } + +if (ok) { +/* clear reserved and busy bits, report back sanitized version */ + new_ddtp =set_field (new_ddtp &RISCV_IOMMU_DDTP_PPN , +RISCV_IOMMU_DDTP_MODE , new_mode); + }else { + new_ddtp = old_ddtp; + } + s->ddtp = new_ddtp; + +riscv_iommu_reg_set64 (s,RISCV_IOMMU_REG_DDTP , new_ddtp); +} + +/* Command function and opcode field. */ +#define RISCV_IOMMU_CMD (func, op) (((func) << 7) | (op)) + +static void riscv_iommu_process_cq_tail (RISCVIOMMUState *s) +{ +struct riscv_iommu_command cmd; +MemTxResult res; +dma_addr_t addr; +uint32_t tail, head, ctrl; +GHFunc func; + + ctrl =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_CQCSR ); + tail =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_CQT ) & s->cq_mask; + head =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_CQH ) & s->cq_mask; + +/* Check for pending error or queue processing disabled */ +if (!(ctrl &RISCV_IOMMU_CQCSR_CQON ) || + !!(ctrl & (RISCV_IOMMU_CQCSR_CMD_ILL |RISCV_IOMMU_CQCSR_CQMF ))) { +return ; + } + +while (tail != head) { + addr = s->cq_addr + head *sizeof (cmd); + res =dma_memory_read (s->target_as, addr, &cmd,sizeof (cmd), +MEMTXATTRS_UNSPECIFIED ); + +if (res !=MEMTX_OK ) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_CQCSR ,RISCV_IOMMU_CQCSR_CQMF ,0 ); + goto fault; + } + +trace_riscv_iommu_cmd (s->parent_obj.id, cmd.dword0, cmd.dword1); + +switch (get_field (cmd.dword0,RISCV_IOMMU_CMD_OPCODE |RISCV_IOMMU_CMD_FUNC )) { +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IOFENCE_FUNC_C , +RISCV_IOMMU_CMD_IOFENCE_OPCODE ): + res =riscv_iommu_iofence (s, cmd.dword0 &RISCV_IOMMU_CMD_IOFENCE_AV , + cmd.dword1,get_field (cmd.dword0,RISCV_IOMMU_CMD_IOFENCE_DATA )); + +if (res !=MEMTX_OK ) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_CQCSR , +RISCV_IOMMU_CQCSR_CQMF ,0 ); + goto fault; + } +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA , +RISCV_IOMMU_CMD_IOTINVAL_OPCODE ): +if (cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_PSCV ) { +/* illegal command arguments IOTINVAL.GVMA & PSCV == 1 */ + goto cmd_ill; + }else if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_GV )) { +/* invalidate all cache mappings */ + func = __iot_inval_all; + }else if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_AV )) { +/* invalidate cache matching GSCID */ + func = __iot_inval_gscid; + }else { +/* invalidate cache matching GSCID and ADDR (GPA) */ + func = __iot_inval_gscid_gpa; + } +riscv_iommu_iot_inval (s, func, +get_field (cmd.dword0,RISCV_IOMMU_CMD_IOTINVAL_GSCID ),0 , + cmd.dword1 &TARGET_PAGE_MASK ); +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA , +RISCV_IOMMU_CMD_IOTINVAL_OPCODE ): +if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_GV )) { +/* invalidate all cache mappings, simplified model */ + func = __iot_inval_all; + }else if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_PSCV )) { +/* invalidate cache matching GSCID, simplified model */ + func = __iot_inval_gscid; + }else if (!(cmd.dword0 &RISCV_IOMMU_CMD_IOTINVAL_AV )) { +/* invalidate cache matching GSCID and PSCID */ + func = __iot_inval_pscid; + }else { +/* invalidate cache matching GSCID and PSCID and ADDR (IOVA) */ + func = __iot_inval_pscid_iova; + } +riscv_iommu_iot_inval (s, func, +get_field (cmd.dword0,RISCV_IOMMU_CMD_IOTINVAL_GSCID ), +get_field (cmd.dword0,RISCV_IOMMU_CMD_IOTINVAL_PSCID ), + cmd.dword1 &TARGET_PAGE_MASK ); +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT , +RISCV_IOMMU_CMD_IODIR_OPCODE ): +if (!(cmd.dword0 &RISCV_IOMMU_CMD_IODIR_DV )) { +/* invalidate all device context cache mappings */ + func = __ctx_inval_all; + }else { +/* invalidate all device context matching DID */ + func = __ctx_inval_devid; + } +riscv_iommu_ctx_inval (s, func, +get_field (cmd.dword0,RISCV_IOMMU_CMD_IODIR_DID ),0 ); +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT , +RISCV_IOMMU_CMD_IODIR_OPCODE ): +if (!(cmd.dword0 &RISCV_IOMMU_CMD_IODIR_DV )) { +/* illegal command arguments IODIR_PDT & DV == 0 */ + goto cmd_ill; + }else { + func = __ctx_inval_devid_pasid; + } +riscv_iommu_ctx_inval (s, func, +get_field (cmd.dword0,RISCV_IOMMU_CMD_IODIR_DID ), +get_field (cmd.dword0,RISCV_IOMMU_CMD_IODIR_PID )); +break ; + +/* ATS commands */ +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_ATS_FUNC_INVAL , +RISCV_IOMMU_CMD_ATS_OPCODE ): +riscv_iommu_ats_inval (s, &cmd); +break ; + +case RISCV_IOMMU_CMD (RISCV_IOMMU_CMD_ATS_FUNC_PRGR , +RISCV_IOMMU_CMD_ATS_OPCODE ): +riscv_iommu_ats_prgr (s, &cmd); +break ; + +default : + cmd_ill: +/* Invalid instruction, do not advance instruction index. */ +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_CQCSR , +RISCV_IOMMU_CQCSR_CMD_ILL ,0 ); + goto fault; + } + +/* Advance and update head pointer after command completes. */ + head = (head +1 ) & s->cq_mask; +riscv_iommu_reg_set32 (s,RISCV_IOMMU_REG_CQH , head); + } +return ; + +fault: +if (ctrl &RISCV_IOMMU_CQCSR_CIE ) { +riscv_iommu_notify (s,RISCV_IOMMU_INTR_CQ ); + } +} + +static void riscv_iommu_process_cq_control (RISCVIOMMUState *s) +{ +uint64_t base; +uint32_t ctrl_set =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_CQCSR ); +uint32_t ctrl_clr; +bool enable = !!(ctrl_set &RISCV_IOMMU_CQCSR_CQEN ); +bool active = !!(ctrl_set &RISCV_IOMMU_CQCSR_CQON ); + +if (enable && !active) { + base =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_CQB ); + s->cq_mask = (2ULL <<get_field (base,RISCV_IOMMU_CQB_LOG2SZ )) -1 ; + s->cq_addr =PPN_PHYS (get_field (base,RISCV_IOMMU_CQB_PPN )); +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_CQT ], ~s->cq_mask); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_CQH ],0 ); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_CQT ],0 ); + ctrl_set =RISCV_IOMMU_CQCSR_CQON ; + ctrl_clr =RISCV_IOMMU_CQCSR_BUSY |RISCV_IOMMU_CQCSR_CQMF | +RISCV_IOMMU_CQCSR_CMD_ILL |RISCV_IOMMU_CQCSR_CMD_TO ; + }else if (!enable && active) { +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_CQT ], ~0 ); + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_CQCSR_BUSY |RISCV_IOMMU_CQCSR_CQON ; + }else { + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_CQCSR_BUSY ; + } + +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_CQCSR , ctrl_set, ctrl_clr); +} + +static void riscv_iommu_process_fq_control (RISCVIOMMUState *s) +{ +uint64_t base; +uint32_t ctrl_set =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_FQCSR ); +uint32_t ctrl_clr; +bool enable = !!(ctrl_set &RISCV_IOMMU_FQCSR_FQEN ); +bool active = !!(ctrl_set &RISCV_IOMMU_FQCSR_FQON ); + +if (enable && !active) { + base =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_FQB ); + s->fq_mask = (2ULL <<get_field (base,RISCV_IOMMU_FQB_LOG2SZ )) -1 ; + s->fq_addr =PPN_PHYS (get_field (base,RISCV_IOMMU_FQB_PPN )); +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_FQH ], ~s->fq_mask); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_FQH ],0 ); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_FQT ],0 ); + ctrl_set =RISCV_IOMMU_FQCSR_FQON ; + ctrl_clr =RISCV_IOMMU_FQCSR_BUSY |RISCV_IOMMU_FQCSR_FQMF | +RISCV_IOMMU_FQCSR_FQOF ; + }else if (!enable && active) { +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_FQH ], ~0 ); + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_FQCSR_BUSY |RISCV_IOMMU_FQCSR_FQON ; + }else { + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_FQCSR_BUSY ; + } + +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_FQCSR , ctrl_set, ctrl_clr); +} + +static void riscv_iommu_process_pq_control (RISCVIOMMUState *s) +{ +uint64_t base; +uint32_t ctrl_set =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_PQCSR ); +uint32_t ctrl_clr; +bool enable = !!(ctrl_set &RISCV_IOMMU_PQCSR_PQEN ); +bool active = !!(ctrl_set &RISCV_IOMMU_PQCSR_PQON ); + +if (enable && !active) { + base =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_PQB ); + s->pq_mask = (2ULL <<get_field (base,RISCV_IOMMU_PQB_LOG2SZ )) -1 ; + s->pq_addr =PPN_PHYS (get_field (base,RISCV_IOMMU_PQB_PPN )); +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_PQH ], ~s->pq_mask); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_PQH ],0 ); +stl_le_p (&s->regs_rw[RISCV_IOMMU_REG_PQT ],0 ); + ctrl_set =RISCV_IOMMU_PQCSR_PQON ; + ctrl_clr =RISCV_IOMMU_PQCSR_BUSY |RISCV_IOMMU_PQCSR_PQMF | +RISCV_IOMMU_PQCSR_PQOF ; + }else if (!enable && active) { +stl_le_p (&s->regs_ro[RISCV_IOMMU_REG_PQH ], ~0 ); + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_PQCSR_BUSY |RISCV_IOMMU_PQCSR_PQON ; + }else { + ctrl_set =0 ; + ctrl_clr =RISCV_IOMMU_PQCSR_BUSY ; + } + +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_PQCSR , ctrl_set, ctrl_clr); +} + +static void riscv_iommu_process_dbg (RISCVIOMMUState *s) +{ +uint64_t iova =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_TR_REQ_IOVA ); +uint64_t ctrl =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_TR_REQ_CTL ); +unsigned devid =get_field (ctrl,RISCV_IOMMU_TR_REQ_CTL_DID ); +unsigned pid =get_field (ctrl,RISCV_IOMMU_TR_REQ_CTL_PID ); +RISCVIOMMUContext *ctx; +void *ref; + +if (!(ctrl &RISCV_IOMMU_TR_REQ_CTL_GO_BUSY )) { +return ; + } + + ctx =riscv_iommu_ctx (s, devid, pid, &ref); +if (ctx ==NULL ) { +riscv_iommu_reg_set64 (s,RISCV_IOMMU_REG_TR_RESPONSE , +RISCV_IOMMU_TR_RESPONSE_FAULT | (RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED <<10 )); + }else { +IOMMUTLBEntry iotlb = { + .iova = iova, + .perm =IOMMU_NONE , + .addr_mask = ~0 , + .target_as =NULL , + }; +int fault =riscv_iommu_translate (s, ctx, &iotlb, false); +if (fault) { + iova =RISCV_IOMMU_TR_RESPONSE_FAULT | (((uint64_t ) fault) <<10 ); + }else { + iova = ((iotlb.translated_addr & ~iotlb.addr_mask) >>2 ) & +RISCV_IOMMU_TR_RESPONSE_PPN ; + } +riscv_iommu_reg_set64 (s,RISCV_IOMMU_REG_TR_RESPONSE , iova); + } + +riscv_iommu_reg_mod64 (s,RISCV_IOMMU_REG_TR_REQ_CTL ,0 , +RISCV_IOMMU_TR_REQ_CTL_GO_BUSY ); +riscv_iommu_ctx_put (s, ref); +} + +/* Core IOMMU execution activation */ +enum { +RISCV_IOMMU_EXEC_DDTP , +RISCV_IOMMU_EXEC_CQCSR , +RISCV_IOMMU_EXEC_CQT , +RISCV_IOMMU_EXEC_FQCSR , +RISCV_IOMMU_EXEC_FQH , +RISCV_IOMMU_EXEC_PQCSR , +RISCV_IOMMU_EXEC_PQH , +RISCV_IOMMU_EXEC_TR_REQUEST , +/* RISCV_IOMMU_EXEC_EXIT must be the last enum value */ +RISCV_IOMMU_EXEC_EXIT , +}; + +static void *riscv_iommu_core_proc (void * arg) +{ +RISCVIOMMUState *s = arg; +unsigned exec =0 ; +unsigned mask =0 ; + +while (!(exec &BIT (RISCV_IOMMU_EXEC_EXIT ))) { + mask = (mask ? mask :BIT (RISCV_IOMMU_EXEC_EXIT )) >>1 ; +switch (exec & mask) { +case BIT (RISCV_IOMMU_EXEC_DDTP ): +riscv_iommu_process_ddtp (s); +break ; +case BIT (RISCV_IOMMU_EXEC_CQCSR ): +riscv_iommu_process_cq_control (s); +break ; +case BIT (RISCV_IOMMU_EXEC_CQT ): +riscv_iommu_process_cq_tail (s); +break ; +case BIT (RISCV_IOMMU_EXEC_FQCSR ): +riscv_iommu_process_fq_control (s); +break ; +case BIT (RISCV_IOMMU_EXEC_FQH ): +/* NOP */ +break ; +case BIT (RISCV_IOMMU_EXEC_PQCSR ): +riscv_iommu_process_pq_control (s); +break ; +case BIT (RISCV_IOMMU_EXEC_PQH ): +/* NOP */ +break ; +case BIT (RISCV_IOMMU_EXEC_TR_REQUEST ): +riscv_iommu_process_dbg (s); +break ; + } + exec &= ~mask; +if (!exec) { +qemu_mutex_lock (&s->core_lock); + exec = s->core_exec; +while (!exec) { +qemu_cond_wait (&s->core_cond, &s->core_lock); + exec = s->core_exec; + } + s->core_exec =0 ; +qemu_mutex_unlock (&s->core_lock); + } + }; + +return NULL ; +} + +/* For now we assume IOMMU HPM frequency to be 1GHz so 1-cycle is of 1-ns. */ +static inline uint64_t __get_cycles (void ) +{ +return qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); +} + +static void __hpm_setup_timer (RISCVIOMMUState *s,uint64_t value) +{ +const uint32_t inhibit =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTINH ); +uint64_t overflow_at, overflow_ns; + +if (get_field (inhibit,RISCV_IOMMU_IOCOUNTINH_CY )) { +return ; + } + +/* +* We are using INT64_MAX here instead to UINT64_MAX because cycle counter +* has 63-bit precision and INT64_MAX is the maximum it can store. +*/ +if (value) { + overflow_ns =INT64_MAX - value +1 ; + }else { + overflow_ns =INT64_MAX ; + } + + overflow_at = (uint64_t )qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ) + overflow_ns; + +if (overflow_at >INT64_MAX ) { + s->irq_overflow_left = overflow_at -INT64_MAX ; + overflow_at =INT64_MAX ; + } + +timer_mod_anticipate_ns (s->hpm_timer, overflow_at); +} + +/* Updates the internal cycle counter state when iocntinh:CY is changed. */ +static void riscv_iommu_process_iocntinh_cy (RISCVIOMMUState *s, +bool prev_cy_inh) +{ +const uint32_t inhibit =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTINH ); + +/* We only need to process CY bit toggle. */ +if (!(inhibit ^ prev_cy_inh)) { +return ; + } + +if (!(inhibit &RISCV_IOMMU_IOCOUNTINH_CY )) { +/* +* Cycle counter is enabled. Just start the timer again and update the +* clock snapshot value to point to the current time to make sure +* iohpmcycles read is correct. +*/ + s->hpmcycle_prev =__get_cycles (); +__hpm_setup_timer (s, s->hpmcycle_val); + }else { +/* +* Cycle counter is disabled. Stop the timer and update the cycle +* counter to record the current value which is last programmed +* value + the cycles passed so far. +*/ + s->hpmcycle_val = s->hpmcycle_val + (__get_cycles () - s->hpmcycle_prev); +timer_del (s->hpm_timer); + } +} + +static void riscv_iommu_process_hpmcycle_write (RISCVIOMMUState *s) +{ +const uint64_t val =riscv_iommu_reg_get64 (s,RISCV_IOMMU_REG_IOHPMCYCLES ); +const uint32_t ovf =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ); + +/* +* Clear OF bit in IOCNTOVF if it's being cleared in IOHPMCYCLES register. +*/ +if (get_field (ovf,RISCV_IOMMU_IOCOUNTOVF_CY ) && + !get_field (val,RISCV_IOMMU_IOHPMCYCLES_OVF )) { +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ,0 , +RISCV_IOMMU_IOCOUNTOVF_CY ); + } + + s->hpmcycle_val = val & ~RISCV_IOMMU_IOHPMCYCLES_OVF ; + s->hpmcycle_prev =__get_cycles (); +__hpm_setup_timer (s, s->hpmcycle_val); +} + +static inline bool __check_valid_event_id (unsigned event_id) +{ +return event_id >RISCV_IOMMU_HPMEVENT_INVALID && + event_id <RISCV_IOMMU_HPMEVENT_MAX ; +} + +static gboolean __hpm_event_equal (gpointer key,gpointer value,gpointer udata) +{ +uint32_t *pair = udata; + +if (GPOINTER_TO_UINT (value) & (1 << pair[0 ])) { + pair[1 ] =GPOINTER_TO_UINT (key); +return true; + } + +return false; +} + +/* Caller must check ctr_idx against hpm_ctrs to see if its supported or not. */ +static void __update_event_map (RISCVIOMMUState *s,uint64_t value, +uint32_t ctr_idx) +{ +unsigned event_id =get_field (value,RISCV_IOMMU_IOHPMEVT_EVENT_ID ); +uint32_t pair[2 ] = { ctr_idx,RISCV_IOMMU_HPMEVENT_INVALID }; +uint32_t new_value =1 << ctr_idx; +gpointer data; + +/* If EventID field is RISCV_IOMMU_HPMEVENT_INVALID remove the current mapping. */ +if (event_id ==RISCV_IOMMU_HPMEVENT_INVALID ) { + data =g_hash_table_find (s->hpm_event_ctr_map, __hpm_event_equal, pair); + + new_value =GPOINTER_TO_UINT (data) & ~(new_value); +pthread_rwlock_wrlock (&s->ht_lock); +if (new_value !=0 ) { +g_hash_table_replace (s->hpm_event_ctr_map, +GUINT_TO_POINTER (pair[1 ]), +GUINT_TO_POINTER (new_value)); + }else { +g_hash_table_remove (s->hpm_event_ctr_map, +GUINT_TO_POINTER (pair[1 ])); + } +pthread_rwlock_unlock (&s->ht_lock); + +return ; + } + +/* Update the counter mask if the event is already enabled. */ +if (g_hash_table_lookup_extended (s->hpm_event_ctr_map, +GUINT_TO_POINTER (event_id), +NULL , + &data)) { + new_value |=GPOINTER_TO_UINT (data); + } + +pthread_rwlock_wrlock (&s->ht_lock); +g_hash_table_insert (s->hpm_event_ctr_map, +GUINT_TO_POINTER (event_id), +GUINT_TO_POINTER (new_value)); +pthread_rwlock_unlock (&s->ht_lock); +} + +static void riscv_iommu_process_hpmevt_write (RISCVIOMMUState *s, +uint32_t evt_reg) +{ +const uint32_t ctr_idx = (evt_reg -RISCV_IOMMU_REG_IOHPMEVT_BASE ) >>3 ; +const uint32_t ovf =riscv_iommu_reg_get32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ); +uint64_t val =riscv_iommu_reg_get64 (s, evt_reg); + +if (ctr_idx >= s->hpm_cntrs) { +return ; + } + +/* Clear OF bit in IOCNTOVF if it's being cleared in IOHPMEVT register. */ +if (get_field (ovf,BIT (ctr_idx +1 )) && !get_field (val,RISCV_IOMMU_IOHPMEVT_OF )) { +/* +1 to offset CYCLE register OF bit. */ +riscv_iommu_reg_mod32 (s,RISCV_IOMMU_REG_IOCOUNTOVF ,0 ,BIT (ctr_idx +1 )); + } + +if (!__check_valid_event_id (get_field (val,RISCV_IOMMU_IOHPMEVT_EVENT_ID ))) { +/* Reset EventID (WARL) field to invalid. */ + val =set_field (val,RISCV_IOMMU_IOHPMEVT_EVENT_ID , +RISCV_IOMMU_HPMEVENT_INVALID ); +riscv_iommu_reg_set64 (s, evt_reg, val); + } + +__update_event_map (s, val, ctr_idx); +} + +static void riscv_iommu_process_hpm_writes (RISCVIOMMUState *s, +uint32_t regb, +bool prev_cy_inh) +{ +switch (regb) { +case RISCV_IOMMU_REG_IOCOUNTINH : +riscv_iommu_process_iocntinh_cy (s, prev_cy_inh); +break ; + +case RISCV_IOMMU_REG_IOHPMCYCLES : +case RISCV_IOMMU_REG_IOHPMCYCLES +4 : +riscv_iommu_process_hpmcycle_write (s); +break ; + + case RISCV_IOMMU_REG_IOHPMEVT_BASE ... +RISCV_IOMMU_REG_IOHPMEVT (RISCV_IOMMU_IOCOUNT_NUM ) +4 : +riscv_iommu_process_hpmevt_write (s, regb & ~7 ); +break ; + } +} + +static MemTxResult riscv_iommu_mmio_write (void *opaque,hwaddr addr, +uint64_t data,unsigned size,MemTxAttrs attrs) +{ +RISCVIOMMUState *s = opaque; +uint32_t regb = addr & ~3 ; +bool cy_inh = false; +uint32_t busy =0 ; +uint32_t exec =0 ; + +if (size ==0 || size >8 || (addr & (size -1 )) !=0 ) { +/* Unsupported MMIO alignment or access size */ +return MEMTX_ERROR ; + } + +if (addr + size >RISCV_IOMMU_REG_MSI_CONFIG ) { +/* Unsupported MMIO access location. */ +return MEMTX_ACCESS_ERROR ; + } + +/* Track actionable MMIO write. */ +switch (regb) { +case RISCV_IOMMU_REG_DDTP : +case RISCV_IOMMU_REG_DDTP +4 : + exec =BIT (RISCV_IOMMU_EXEC_DDTP ); + regb =RISCV_IOMMU_REG_DDTP ; + busy =RISCV_IOMMU_DDTP_BUSY ; +break ; + +case RISCV_IOMMU_REG_CQT : + exec =BIT (RISCV_IOMMU_EXEC_CQT ); +break ; + +case RISCV_IOMMU_REG_CQCSR : + exec =BIT (RISCV_IOMMU_EXEC_CQCSR ); + busy =RISCV_IOMMU_CQCSR_BUSY ; +break ; + +case RISCV_IOMMU_REG_FQH : + exec =BIT (RISCV_IOMMU_EXEC_FQH ); +break ; + +case RISCV_IOMMU_REG_FQCSR : + exec =BIT (RISCV_IOMMU_EXEC_FQCSR ); + busy =RISCV_IOMMU_FQCSR_BUSY ; +break ; + +case RISCV_IOMMU_REG_PQH : + exec =BIT (RISCV_IOMMU_EXEC_PQH ); +break ; + +case RISCV_IOMMU_REG_PQCSR : + exec =BIT (RISCV_IOMMU_EXEC_PQCSR );Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Quoted 8 lines
Quoted 8 lines
meson.build +1
-1Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
2.34.1
best,
- Tomasz
Part of the diff is not shown here. View the raw message.
PATCH 2/5 MAINTAINERS: Add RISC-V IOMMU maintainers
In reply to Tomasz JeznachSigned-off-by Tomasz Jeznach <[email protected]>
--- MAINTAINERS | 6 ++++++ 1 file changed, 6 insertions(+)
MAINTAINERS +6 -0
@@ -1971,6 +1971,12 @@ F: include/hw/i2c/ppc4xx_i2c.h F: hw/intc/ppc-uic.c F: include/hw/intc/ppc-uic.h +RISC-V IOMMU +M: Tomasz Jeznach <[email protected]> +L: [email protected] +S: Maintained +F: hw/riscv/riscv-iommu* + Character devices M: Marc-André Lureau <[email protected]> R: Paolo Bonzini <[email protected]>
Signature · 1 line
2.34.1
PATCH 3/5 exec/memtxattr: add process identifier to the transaction attributes
In reply to Tomasz JeznachExtend memory transaction attributes with process identifier to allow per-request address translation logic to use requester_id / process_id to identify memory mapping (e.g. enabling IOMMU w/ PASID translations).
Signed-off-by Tomasz Jeznach <[email protected]>
--- include/exec/memattrs.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/exec/memattrs.h +6 -0
@@ -64,6 +64,12 @@ typedef struct MemTxAttrs { unsigned int target_tlb_bit0 : 1; unsigned int target_tlb_bit1 : 1; unsigned int target_tlb_bit2 : 1; + +/* +* PCI PASID support: Limited to 8 bits process identifier. +*/ + unsigned int pasid:8; + } MemTxAttrs;/* Bus masters which don't specify any attributes will get this,
Signature · 1 line
2.34.1
PATCH 4/5 hw/riscv: IOMMU: use process identifier from transaction attributes.
In reply to Tomasz JeznachUse iommu index as process identifier, linking transaction memory attributes with translation request.
Signed-off-by Tomasz Jeznach <[email protected]>
--- hw/riscv/riscv-iommu.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
hw/riscv/riscv-iommu.c +9 -2
@@ -2236,6 +2236,12 @@ static void riscv_iommu_realize(DeviceState *dev, Error **errp)/* Report QEMU target physical address space limits */ s->cap =set_field (s->cap,RISCV_IOMMU_CAP_PAS ,TARGET_PHYS_ADDR_SPACE_BITS ); +/* Restricted to the size of MemTxAttrs.pasid field. */ +if (s->cap &RISCV_IOMMU_CAP_PD8 ) { +MemTxAttrs attrs = { .pasid = ~0 }; + s->pasid_bits =ctz32 (~((unsigned )attrs.pasid)); + } +/* Adjust reported PD capabilities */ if (s->pasid_bits <20 ) { s->cap &= ~RISCV_IOMMU_CAP_PD20 ;
@@ -2506,12 +2512,13 @@ void riscv_iommu_pci_setup_iommu(RISCVIOMMUState *iommu, PCIBus *bus,static int riscv_iommu_memory_region_index (IOMMUMemoryRegion *iommu_mr,MemTxAttrs attrs) { -return RISCV_IOMMU_NOPASID ; +return attrs.unspecified ?RISCV_IOMMU_NOPASID : (int )attrs.pasid; }static int riscv_iommu_memory_region_index_len (IOMMUMemoryRegion *iommu_mr) { -return 1 ; +RISCVIOMMUSpace *as =container_of (iommu_mr, RISCVIOMMUSpace, iova_mr); +return 1 << as->iommu->pasid_bits; }static void riscv_iommu_memory_region_init (ObjectClass *klass,void *data)
Signature · 1 line
2.34.1
PATCH 5/5 hw/riscv: virt: support for RISC-V IOMMU platform device.
In reply to Tomasz JeznachAdding virt machine property 'iommu' to enable/disable IOMMU support, with platform RISC-V IOMMU device implementation.
Generate device tree entry for riscv-iommu device, along with mapping all PCI device identifiers to the single IOMMU device instance.
Signed-off-by Tomasz Jeznach <[email protected]>
--- hw/riscv/Kconfig | 1 + hw/riscv/virt.c | 100 +++++++++++++++++++++++++++++++++++++++- include/hw/riscv/virt.h | 3 ++ 3 files changed, 103 insertions(+), 1 deletion(-)
hw/riscv/Kconfig +1 -0
@@ -41,6 +41,7 @@ config RISCV_VIRT select SERIAL select RISCV_ACLINT select RISCV_APLIC + select RISCV_IOMMU select RISCV_IMSIC select SIFIVE_PLIC select SIFIVE_TEST
hw/riscv/virt.c +99 -1
154 lines not on this page · raw message
include/hw/riscv/virt.h +3 -0
@@ -59,6 +59,7 @@ struct RISCVVirtState {char *oem_id;char *oem_table_id;OnOffAuto acpi; +OnOffAuto iommu;const MemMapEntry *memmap; };
@@ -73,6 +74,7 @@ enum {VIRT_APLIC_M ,VIRT_APLIC_S ,VIRT_UART0 , +VIRT_IOMMU ,VIRT_VIRTIO ,VIRT_FW_CFG , VIRT_IMSIC_M,
@@ -91,6 +93,7 @@ enum {VIRTIO_IRQ =1 ,/* 1 to 8 */ VIRTIO_COUNT =8 ,PCIE_IRQ =0x20 ,/* 32 to 35 */ +IOMMU_IRQ =60 ,/* 60 to 63 */ VIRT_PLATFORM_BUS_IRQ = 64, /* 64 to 95 */ };
Signature · 1 line
2.34.1
Re: [PATCH 5/5] hw/riscv: virt: support for RISC-V IOMMU platform device.
Quoted 40 lines · Tomasz Jeznach
Adding virt machine property 'iommu' to enable/disable IOMMU support, with platform RISC-V IOMMU device implementation.
Generate device tree entry for riscv-iommu device, along with mapping all PCI device identifiers to the single IOMMU device instance.
Signed-off-by Tomasz Jeznach <[email protected]>--- hw/riscv/Kconfig | 1 + hw/riscv/virt.c | 100 +++++++++++++++++++++++++++++++++++++++- include/hw/riscv/virt.h | 3 ++ 3 files changed, 103 insertions(+), 1 deletion(-)hw/riscv/Kconfig +1
-0@@ -41,6 +41,7 @@ config RISCV_VIRT select SERIAL select RISCV_ACLINT select RISCV_APLIC + select RISCV_IOMMU select RISCV_IMSIC select SIFIVE_PLIC select SIFIVE_TESThw/riscv/virt.c +1
-0@@ -32,6 +32,7 @@#include "hw/core/sysbus-fdt.h" #include "target/riscv/pmu.h" #include "hw/riscv/riscv_hart.h" +#include "hw/riscv/iommu.h" #include "hw/riscv/virt.h" #include "hw/riscv/boot.h" #include "hw/riscv/numa.h"
hw/riscv/virt.c static const MemMapEntry virt_memmap[] = { · line 93
Quoted 6 lines
hw/riscv/virt.c +2
-1@@ -88,7 +89,8 @@ static const MemMapEntry virt_memmap[] = { [VIRT_APLIC_M] = { 0xc000000, APLIC_SIZE(VIRT_CPUS_MAX) }, [VIRT_APLIC_S] = { 0xd000000, APLIC_SIZE(VIRT_CPUS_MAX) }, [VIRT_UART0] = { 0x10000000, 0x100 }, - [VIRT_VIRTIO] = { 0x10001000, 0x1000 }, + [VIRT_IOMMU] = { 0x10001000, 0x1000 }, + [VIRT_VIRTIO] = { 0x10008000, 0x1000 }, /* VIRTIO_COUNT */
We shouldn't change existing addresses
Alistair
Quoted 180 lines
hw/riscv/virt.c +96
-0[VIRT_FW_CFG] = { 0x10100000, 0x18 }, [VIRT_FLASH] = { 0x20000000, 0x4000000 }, [VIRT_IMSIC_M] = { 0x24000000, VIRT_IMSIC_MAX_SIZE },@@ -1019,6 +1021,44 @@ static void create_fdt_fw_cfg(RISCVVirtState *s, const MemMapEntry *memmap)g_free (nodename); } +static void create_fdt_iommu (RISCVVirtState *s,const MemMapEntry *memmap, +uint32_t irq_mmio_phandle) +{ +MachineState *ms =MACHINE (s); +uint32_t iommu_phandle; +const char *irq_names[] = {"cmdq" ,"fltq" ,"pm" ,"priq" }; +char *iommu_node; +char *pci_node; + + pci_node =g_strdup_printf ("/soc/pci@%" PRIx64, memmap[VIRT_PCIE_ECAM ].base); + iommu_node =g_strdup_printf ("/soc/iommu@%" PRIx64, memmap[VIRT_IOMMU ].base); + + iommu_phandle =qemu_fdt_alloc_phandle (ms->fdt); +qemu_fdt_add_subnode (ms->fdt, iommu_node); +qemu_fdt_setprop_string (ms->fdt, iommu_node,"compatible" ,"riscv,iommu" ); +qemu_fdt_setprop_cell (ms->fdt, iommu_node,"#iommu-cells" ,1 ); +qemu_fdt_setprop_cell (ms->fdt, iommu_node,"phandle" , iommu_phandle); +qemu_fdt_setprop_cells (ms->fdt, iommu_node,"reg" , +0x0 , memmap[VIRT_IOMMU ].base,0x0 , memmap[VIRT_IOMMU ].size); +qemu_fdt_setprop_cell (ms->fdt, iommu_node,"interrupt-parent" , irq_mmio_phandle); +qemu_fdt_setprop_string_array (ms->fdt, iommu_node,"interrupt-names" , + (char **) &irq_names,ARRAY_SIZE (irq_names)); +qemu_fdt_setprop_cells (ms->fdt, iommu_node,"interrupts" , +IOMMU_IRQ +0 ,0x4 , +IOMMU_IRQ +1 ,0x4 , +IOMMU_IRQ +2 ,0x4 , +IOMMU_IRQ +3 ,0x4 ); +qemu_fdt_setprop_cells (ms->fdt, pci_node,"iommu-map" , +0x0 , iommu_phandle,0x0 ,0xffff ); +g_free (iommu_node); +g_free (pci_node); +} + +static bool virt_is_iommu_enabled (RISCVVirtState *s) +{ +return s->iommu !=ON_OFF_AUTO_OFF ; +} +static void create_fdt (RISCVVirtState *s,const MemMapEntry *memmap) {MachineState *ms =MACHINE (s);@@ -1051,6 +1091,10 @@ static void create_fdt(RISCVVirtState *s, const MemMapEntry *memmap)create_fdt_pcie (s, memmap, irq_pcie_phandle, msi_pcie_phandle); +if (virt_is_iommu_enabled (s)) { +create_fdt_iommu (s, memmap, irq_mmio_phandle); + } +create_fdt_reset (s, memmap, &phandle);create_fdt_uart (s, memmap, irq_mmio_phandle);@@ -1210,6 +1254,31 @@ static DeviceState *virt_create_aia(RISCVVirtAIAType aia_type, int aia_guests,return aplic_m; } +static DeviceState *virt_create_iommu (RISCVVirtState *s,DeviceState *irqchip) +{ +DeviceState *iommu; +int i; + + iommu =qdev_new (TYPE_RISCV_IOMMU_SYS ); + +if (s->aia_type !=VIRT_AIA_TYPE_APLIC_IMSIC ) { +/* Disable MSI_FLAT [22], MSI_MRIF [23] if IMSIC is not enabled. */ +qdev_prop_set_uint64 (iommu,"capabilities" , ~(BIT_ULL (22 ) |BIT_ULL (23 ))); + } + +/* Fixed base register address */ +qdev_prop_set_uint64 (iommu,"addr" , virt_memmap[VIRT_IOMMU ].base); + +sysbus_realize_and_unref (SYS_BUS_DEVICE (iommu), &error_fatal); + +for (i =0 ; i <4 ; i++) { +sysbus_connect_irq (SYS_BUS_DEVICE (iommu), i, +qdev_get_gpio_in (irqchip,IOMMU_IRQ + i)); + } + +return iommu; +} +static void create_platform_bus (RISCVVirtState *s,DeviceState *irqchip) {DeviceState *dev;@@ -1506,6 +1575,10 @@ static void virt_machine_init(MachineState *machine)create_platform_bus (s, mmio_irqchip); +if (virt_is_iommu_enabled (s)) { +virt_create_iommu (s, mmio_irqchip); + } +serial_mm_init (system_memory, memmap[VIRT_UART0 ].base,0 ,qdev_get_gpio_in (mmio_irqchip,UART0_IRQ ),399193 ,serial_hd (0 ),DEVICE_LITTLE_ENDIAN );@@ -1533,6 +1606,7 @@ static void virt_machine_instance_init(Object *obj) s->oem_id =g_strndup (ACPI_BUILD_APPNAME6 ,6 ); s->oem_table_id =g_strndup (ACPI_BUILD_APPNAME8 ,8 ); s->acpi =ON_OFF_AUTO_AUTO ; + s->iommu =ON_OFF_AUTO_AUTO ; }static char *virt_get_aia_guests (Object *obj,Error **errp)@@ -1607,6 +1681,23 @@ static void virt_set_aclint(Object *obj, bool value, Error **errp) s->have_aclint = value; } +static void virt_get_iommu (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +RISCVVirtState *s =RISCV_VIRT_MACHINE (obj); +OnOffAuto iommu = s->iommu; + +visit_type_OnOffAuto (v, name, &iommu, errp); +} + +static void virt_set_iommu (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +RISCVVirtState *s =RISCV_VIRT_MACHINE (obj); + +visit_type_OnOffAuto (v, name, &s->iommu, errp); +} +bool virt_is_acpi_enabled (RISCVVirtState *s) {return s->acpi !=ON_OFF_AUTO_OFF ;@@ -1683,6 +1774,13 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)machine_class_allow_dynamic_sysbus_dev (mc,TYPE_TPM_TIS_SYSBUS );#endif +machine_class_allow_dynamic_sysbus_dev (mc,TYPE_RISCV_IOMMU_SYS ); +object_class_property_add (oc,"iommu" ,"OnOffAuto" , + virt_get_iommu, virt_set_iommu, +NULL ,NULL ); +object_class_property_set_description (oc,"iommu" , +"Set on/off to enable/disable emulating RISC-V IOMMU platform device" ); +if (tcg_enabled ()) {object_class_property_add_bool (oc,"aclint" , virt_get_aclint, virt_set_aclint);include/hw/riscv/virt.h +3
-1@@ -59,6 +59,7 @@ struct RISCVVirtState {char *oem_id;char *oem_table_id;OnOffAuto acpi; +OnOffAuto iommu;const MemMapEntry *memmap; };@@ -73,6 +74,7 @@ enum {VIRT_APLIC_M ,VIRT_APLIC_S ,VIRT_UART0 , +VIRT_IOMMU ,VIRT_VIRTIO ,VIRT_FW_CFG , VIRT_IMSIC_M,@@ -91,6 +93,7 @@ enum {VIRTIO_IRQ =1 ,/* 1 to 8 */ VIRTIO_COUNT =8 ,PCIE_IRQ =0x20 ,/* 32 to 35 */ +IOMMU_IRQ =60 ,/* 60 to 63 */ VIRT_PLATFORM_BUS_IRQ = 64, /* 64 to 95 */ }; --2.34.1
Re: [PATCH 5/5] hw/riscv: virt: support for RISC-V IOMMU platform device.
Quoted 42 lines · Alistair Francis
Quoted 40 lines · Tomasz Jeznach
Adding virt machine property 'iommu' to enable/disable IOMMU support, with platform RISC-V IOMMU device implementation.
Generate device tree entry for riscv-iommu device, along with mapping all PCI device identifiers to the single IOMMU device instance.
Signed-off-by Tomasz Jeznach <[email protected]>--- hw/riscv/Kconfig | 1 + hw/riscv/virt.c | 100 +++++++++++++++++++++++++++++++++++++++- include/hw/riscv/virt.h | 3 ++ 3 files changed, 103 insertions(+), 1 deletion(-)hw/riscv/Kconfig +1
-0@@ -41,6 +41,7 @@ config RISCV_VIRT select SERIAL select RISCV_ACLINT select RISCV_APLIC + select RISCV_IOMMU select RISCV_IMSIC select SIFIVE_PLIC select SIFIVE_TESThw/riscv/virt.c +1
-0@@ -32,6 +32,7 @@#include "hw/core/sysbus-fdt.h" #include "target/riscv/pmu.h" #include "hw/riscv/riscv_hart.h" +#include "hw/riscv/iommu.h" #include "hw/riscv/virt.h" #include "hw/riscv/boot.h" #include "hw/riscv/numa.h" @@ -88,7 +89,8 @@ static const MemMapEntry virt_memmap[] = {
Quoted 9 lines
hw/riscv/virt.c static const MemMapEntry virt_memmap[] = { · line 93
Quoted 6 lines
hw/riscv/virt.c +2
-1[VIRT_APLIC_M] = { 0xc000000, APLIC_SIZE(VIRT_CPUS_MAX) }, [VIRT_APLIC_S] = { 0xd000000, APLIC_SIZE(VIRT_CPUS_MAX) }, [VIRT_UART0] = { 0x10000000, 0x100 }, - [VIRT_VIRTIO] = { 0x10001000, 0x1000 }, + [VIRT_IOMMU] = { 0x10001000, 0x1000 }, + [VIRT_VIRTIO] = { 0x10008000, 0x1000 }, /* VIRTIO_COUNT */We shouldn't change existing addresses
OK, I'll find another 4k window for IOMMU.
Quoted 178 lines
Alistair
Quoted 176 lines
hw/riscv/virt.c +96
-0[VIRT_FW_CFG] = { 0x10100000, 0x18 }, [VIRT_FLASH] = { 0x20000000, 0x4000000 }, [VIRT_IMSIC_M] = { 0x24000000, VIRT_IMSIC_MAX_SIZE },@@ -1019,6 +1021,44 @@ static void create_fdt_fw_cfg(RISCVVirtState *s, const MemMapEntry *memmap)g_free (nodename); } +static void create_fdt_iommu (RISCVVirtState *s,const MemMapEntry *memmap, +uint32_t irq_mmio_phandle) +{ +MachineState *ms =MACHINE (s); +uint32_t iommu_phandle; +const char *irq_names[] = {"cmdq" ,"fltq" ,"pm" ,"priq" }; +char *iommu_node; +char *pci_node; + + pci_node =g_strdup_printf ("/soc/pci@%" PRIx64, memmap[VIRT_PCIE_ECAM ].base); + iommu_node =g_strdup_printf ("/soc/iommu@%" PRIx64, memmap[VIRT_IOMMU ].base); + + iommu_phandle =qemu_fdt_alloc_phandle (ms->fdt); +qemu_fdt_add_subnode (ms->fdt, iommu_node); +qemu_fdt_setprop_string (ms->fdt, iommu_node,"compatible" ,"riscv,iommu" ); +qemu_fdt_setprop_cell (ms->fdt, iommu_node,"#iommu-cells" ,1 ); +qemu_fdt_setprop_cell (ms->fdt, iommu_node,"phandle" , iommu_phandle); +qemu_fdt_setprop_cells (ms->fdt, iommu_node,"reg" , +0x0 , memmap[VIRT_IOMMU ].base,0x0 , memmap[VIRT_IOMMU ].size); +qemu_fdt_setprop_cell (ms->fdt, iommu_node,"interrupt-parent" , irq_mmio_phandle); +qemu_fdt_setprop_string_array (ms->fdt, iommu_node,"interrupt-names" , + (char **) &irq_names,ARRAY_SIZE (irq_names)); +qemu_fdt_setprop_cells (ms->fdt, iommu_node,"interrupts" , +IOMMU_IRQ +0 ,0x4 , +IOMMU_IRQ +1 ,0x4 , +IOMMU_IRQ +2 ,0x4 , +IOMMU_IRQ +3 ,0x4 ); +qemu_fdt_setprop_cells (ms->fdt, pci_node,"iommu-map" , +0x0 , iommu_phandle,0x0 ,0xffff ); +g_free (iommu_node); +g_free (pci_node); +} + +static bool virt_is_iommu_enabled (RISCVVirtState *s) +{ +return s->iommu !=ON_OFF_AUTO_OFF ; +} +static void create_fdt (RISCVVirtState *s,const MemMapEntry *memmap) {MachineState *ms =MACHINE (s);@@ -1051,6 +1091,10 @@ static void create_fdt(RISCVVirtState *s, const MemMapEntry *memmap)create_fdt_pcie (s, memmap, irq_pcie_phandle, msi_pcie_phandle); +if (virt_is_iommu_enabled (s)) { +create_fdt_iommu (s, memmap, irq_mmio_phandle); + } +create_fdt_reset (s, memmap, &phandle);create_fdt_uart (s, memmap, irq_mmio_phandle);@@ -1210,6 +1254,31 @@ static DeviceState *virt_create_aia(RISCVVirtAIAType aia_type, int aia_guests,return aplic_m; } +static DeviceState *virt_create_iommu (RISCVVirtState *s,DeviceState *irqchip) +{ +DeviceState *iommu; +int i; + + iommu =qdev_new (TYPE_RISCV_IOMMU_SYS ); + +if (s->aia_type !=VIRT_AIA_TYPE_APLIC_IMSIC ) { +/* Disable MSI_FLAT [22], MSI_MRIF [23] if IMSIC is not enabled. */ +qdev_prop_set_uint64 (iommu,"capabilities" , ~(BIT_ULL (22 ) |BIT_ULL (23 ))); + } + +/* Fixed base register address */ +qdev_prop_set_uint64 (iommu,"addr" , virt_memmap[VIRT_IOMMU ].base); + +sysbus_realize_and_unref (SYS_BUS_DEVICE (iommu), &error_fatal); + +for (i =0 ; i <4 ; i++) { +sysbus_connect_irq (SYS_BUS_DEVICE (iommu), i, +qdev_get_gpio_in (irqchip,IOMMU_IRQ + i)); + } + +return iommu; +} +static void create_platform_bus (RISCVVirtState *s,DeviceState *irqchip) {DeviceState *dev;@@ -1506,6 +1575,10 @@ static void virt_machine_init(MachineState *machine)create_platform_bus (s, mmio_irqchip); +if (virt_is_iommu_enabled (s)) { +virt_create_iommu (s, mmio_irqchip); + } +serial_mm_init (system_memory, memmap[VIRT_UART0 ].base,0 ,qdev_get_gpio_in (mmio_irqchip,UART0_IRQ ),399193 ,serial_hd (0 ),DEVICE_LITTLE_ENDIAN );@@ -1533,6 +1606,7 @@ static void virt_machine_instance_init(Object *obj) s->oem_id =g_strndup (ACPI_BUILD_APPNAME6 ,6 ); s->oem_table_id =g_strndup (ACPI_BUILD_APPNAME8 ,8 ); s->acpi =ON_OFF_AUTO_AUTO ; + s->iommu =ON_OFF_AUTO_AUTO ; }static char *virt_get_aia_guests (Object *obj,Error **errp)@@ -1607,6 +1681,23 @@ static void virt_set_aclint(Object *obj, bool value, Error **errp) s->have_aclint = value; } +static void virt_get_iommu (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +RISCVVirtState *s =RISCV_VIRT_MACHINE (obj); +OnOffAuto iommu = s->iommu; + +visit_type_OnOffAuto (v, name, &iommu, errp); +} + +static void virt_set_iommu (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +RISCVVirtState *s =RISCV_VIRT_MACHINE (obj); + +visit_type_OnOffAuto (v, name, &s->iommu, errp); +} +bool virt_is_acpi_enabled (RISCVVirtState *s) {return s->acpi !=ON_OFF_AUTO_OFF ;@@ -1683,6 +1774,13 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)machine_class_allow_dynamic_sysbus_dev (mc,TYPE_TPM_TIS_SYSBUS );#endif +machine_class_allow_dynamic_sysbus_dev (mc,TYPE_RISCV_IOMMU_SYS ); +object_class_property_add (oc,"iommu" ,"OnOffAuto" , + virt_get_iommu, virt_set_iommu, +NULL ,NULL ); +object_class_property_set_description (oc,"iommu" , +"Set on/off to enable/disable emulating RISC-V IOMMU platform device" ); +if (tcg_enabled ()) {object_class_property_add_bool (oc,"aclint" , virt_get_aclint, virt_set_aclint);include/hw/riscv/virt.h +3
-0@@ -59,6 +59,7 @@ struct RISCVVirtState {char *oem_id;char *oem_table_id;OnOffAuto acpi; +OnOffAuto iommu;const MemMapEntry *memmap; };@@ -73,6 +74,7 @@ enum {VIRT_APLIC_M ,VIRT_APLIC_S ,VIRT_UART0 , +VIRT_IOMMU ,VIRT_VIRTIO ,VIRT_FW_CFG , VIRT_IMSIC_M,@@ -91,6 +93,7 @@ enum {VIRTIO_IRQ =1 ,/* 1 to 8 */ VIRTIO_COUNT =8 ,PCIE_IRQ =0x20 ,/* 32 to 35 */ +IOMMU_IRQ =60 ,/* 60 to 63 */ VIRT_PLATFORM_BUS_IRQ = 64, /* 64 to 95 */ };
Quoted 4 lines
Quoted 4 lines
--
2.34.1
best
- Tomasz