make MSI IOVA base address and its length configurable
Subsystems (5)
cover make MSI IOVA base address and its length configurable
Hi,
Currently, the MSI_IOVA_BASE address is hard-coded to 0x80000000, assuming that all platforms have this address available for MSI IOVA reservation. However, this is not always the case, as some platforms reserve this address for other purposes. Consequently, these platforms cannot reserve the MSI_IOVA_BASE address for MSI.
There was an [1] attempt to fix this problem by passing the MSI IOVA base as a kernel command line parameter. In the previous attempt, Will suggested reserving the MSI IOVA at runtime whenever there is a conflict with the default MSI_IOVA_BASE. However, dynamically reserving this address has debuggability concerns, as it becomes difficult to track IOMMU mapping failures.
This patch series aims to address the issue by introducing a new DTS property, "arm,smmu-pci-msi-iova-data". This property allows the configuration of MSI IOVA with a custom MSI base address and a custom length for IOMMU/SMMU drivers. It accommodates platforms that do not have the default MSI base address available for MSI reservation.
[1]: https://lore.kernel.org/lkml/[email protected]/
Thanks,
Shyam
Shyam Saini (3):
dt-bindings: iommu: add "arm,smmu-pci-msi-iova-data" property
iommu: consolidate MSI_IOVA macro definitions
arm-smmu: use dts passed MSI IOVA address and length
.../bindings/iommu/arm,smmu-v3.yaml | 12 +++++ .../devicetree/bindings/iommu/arm,smmu.yaml | 12 +++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 -- drivers/iommu/arm/arm-smmu/arm-smmu.c | 11 +++-- drivers/iommu/virtio-iommu.c | 8 ++-- include/linux/iommu.h | 44 +++++++++++++++++++ 7 files changed, 86 insertions(+), 14 deletions(-)
Signature · 1 line
2.34.1
PATCH 1/3 dt-bindings: iommu: add "arm,smmu-pci-msi-iova-data" property
By default ARM SMMU drivers use MSI_IOVA_BASE macro to reserve PCI MSI IOVA memory range, this assumes that all the platforms have MSI_IOVA_BASE address available for MSI reservation. However, this is not always the case, as some platforms may have the default address reserved for some other purposes and as a consequence ARM SMMU drivers can't reserve MSI memory for those platforms.
To address this issue, add a new dts property "arm,smmu-pci-msi-iova-data" which can be used to hold custom PCI MSI IOVA address and its address length. This property can be passed to ARM SMMU drivers via device tree to reserve specified memory range for MSI.
--- .../devicetree/bindings/iommu/arm,smmu-v3.yaml | 12 ++++++++++++ .../devicetree/bindings/iommu/arm,smmu.yaml | 12 ++++++++++++ 2 files changed, 24 insertions(+)
Documentation/devicetree/bindings/iommu/arm,smmu-v3.yaml +12 -0
@@ -56,6 +56,17 @@ properties: NOTE: this only applies to the SMMU itself, not masters connected upstream of the SMMU. + arm,smmu-pci-msi-iova-data: + $ref: /schemas/types.yaml#/definitions/uint32-array + description: | + Specifies a custom PCI MSI base I/O Virtual Address and its memory range + size for ARM SMMU drivers. This allows setting a custom address and + memory size pair if the default MSI_IOVA_BASE_DEFAULT address and + MSI_IOVA_LENGTH_DEFAULT size are not suitable for the intended platform. + items: + - description: MSI IOVA base address + - description: MSI IOVA address length + msi-parent: true hisilicon,broken-prefetch-cmd:
@@ -92,4 +103,5 @@ examples: dma-coherent; #iommu-cells = <1>; msi-parent = <&its 0xff0000>; + arm,smmu-pci-msi-iova-data = <0xa0000000 0x100000>; };
Documentation/devicetree/bindings/iommu/arm,smmu.yaml +12 -0
@@ -207,6 +207,17 @@ properties: NOTE: this only applies to the SMMU itself, not masters connected upstream of the SMMU. + arm,smmu-pci-msi-iova-data: + $ref: /schemas/types.yaml#/definitions/uint32-array + description: | + Specifies a custom PCI MSI base I/O Virtual Address and its memory range + size for ARM SMMU drivers. This allows setting a custom address and + memory size pair if the default MSI_IOVA_BASE_DEFAULT address and + MSI_IOVA_LENGTH_DEFAULT size are not suitable for the intended platform. + items: + - description: MSI IOVA base address + - description: MSI IOVA address length + calxeda,smmu-secure-config-access: type: boolean description:
@@ -679,6 +690,7 @@ examples: #iommu-cells = <1>; /* always ignore appended 5-bit TBU number */ stream-match-mask = <0x7c00>; + arm,smmu-pci-msi-iova-data = <0xa0000000 0x100000>; }; bus {
Signature · 1 line
2.34.1
PATCH 2/3 iommu: consolidate MSI_IOVA macro definitions
In reply to Shyam SainiMSI_IOVA* macros are common among different iommu/smu drivers, so move them to common iommu.h header file.
This doesn't change anything wrt functionality of the IOMMU subsystem.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 --- drivers/iommu/arm/arm-smmu/arm-smmu.c | 3 --- drivers/iommu/virtio-iommu.c | 2 -- include/linux/iommu.h | 3 +++ 4 files changed, 3 insertions(+), 8 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +0 -3
@@ -502,9 +502,6 @@ static inline unsigned int arm_smmu_cdtab_l2_idx(unsigned int ssid)#define ARM_SMMU_POLL_TIMEOUT_US 1000000/* 1s! */ #define ARM_SMMU_POLL_SPIN_COUNT 10 -#define MSI_IOVA_BASE 0x8000000 -#define MSI_IOVA_LENGTH 0x100000 -enum pri_resp {PRI_RESP_DENY =0 ,PRI_RESP_FAIL =1 ,
drivers/iommu/arm/arm-smmu/arm-smmu.c +0 -3
@@ -49,9 +49,6 @@*/ #define QCOM_DUMMY_VAL -1 -#define MSI_IOVA_BASE 0x8000000 -#define MSI_IOVA_LENGTH 0x100000 -static int force_stage;module_param (force_stage, int,S_IRUGO ); MODULE_PARM_DESC(force_stage,
drivers/iommu/virtio-iommu.c +0 -2
@@ -24,8 +24,6 @@#include "dma-iommu.h" -#define MSI_IOVA_BASE 0x8000000 -#define MSI_IOVA_LENGTH 0x100000#define VIOMMU_REQUEST_VQ 0#define VIOMMU_EVENT_VQ 1
include/linux/iommu.h +3 -0
@@ -1505,6 +1505,9 @@ static inline void iommu_debugfs_setup(void) {}#ifdef CONFIG_IOMMU_DMA #include <linux/msi.h> +#define MSI_IOVA_BASE 0x8000000 +#define MSI_IOVA_LENGTH 0x100000 +int iommu_get_msi_cookie (struct iommu_domain *domain,dma_addr_t base);int iommu_dma_prepare_msi (struct msi_desc *desc,phys_addr_t msi_addr);
Signature · 1 line
2.34.1
PATCH 3/3 arm-smmu: use dts passed MSI IOVA address and length
In reply to Shyam SainiCurrently ARM SMMU drivers hardcode PCI MSI IOVA address. Not all the platform have same memory mappings and some platform could have this address already being mapped for something else. This can lead to collision and as a consequence the MSI IOVA addr range is never reserved.
Fix this by passing PCI MSI IOVA address via dts property "arm,smmu-pci-msi-iova-data". Likewise this property can be used to set custom MSI IOVA address length, by default it should be set to MSI IOVA default length value i.e, 0x100000.
If this property is not found in the dtb for the given platform then the driver falls back on the default MSI IOVA address.
Since this change allows configuration of custom MSI IOVA base address and length, rename existing as MSI_IOVA* macros accordingly.
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++- drivers/iommu/arm/arm-smmu/arm-smmu.c | 10 ++++- drivers/iommu/virtio-iommu.c | 6 +-- include/linux/iommu.h | 45 ++++++++++++++++++++- 4 files changed, 62 insertions(+), 9 deletions(-)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +8 -2
@@ -32,6 +32,8 @@#include "arm-smmu-v3.h" #include "../../dma-iommu.h" +struct msi_iova_data msi_data_smmu_v3; +static bool disable_msipolling;module_param (disable_msipolling, bool,0444 ); MODULE_PARM_DESC(disable_msipolling,
@@ -3540,8 +3542,9 @@ static void arm_smmu_get_resv_regions(struct device *dev,struct iommu_resv_region *region;int prot =IOMMU_WRITE |IOMMU_NOEXEC |IOMMU_MMIO ; - region =iommu_alloc_resv_region (MSI_IOVA_BASE ,MSI_IOVA_LENGTH , - prot,IOMMU_RESV_SW_MSI ,GFP_KERNEL ); + region =iommu_alloc_resv_region (msi_data_smmu_v3.msi_iova_base, + msi_data_smmu_v3.msi_iova_length, prot, +IOMMU_RESV_SW_MSI ,GFP_KERNEL );if (!region)return ;
@@ -4569,6 +4572,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,struct device *dev = &pdev->dev;u32 cells;int ret = -EINVAL ; +struct msi_iova_data *msi_data_ptr = &msi_data_smmu_v3; + +iommu_configure_msi_iova (dev,"arm,smmu-pci-msi-iova-data" , msi_data_ptr);if (of_property_read_u32 (dev->of_node,"#iommu-cells" , &cells))dev_err (dev,"missing #iommu-cells property\n" );
drivers/iommu/arm/arm-smmu/arm-smmu.c +8 -2
@@ -49,6 +49,8 @@*/ #define QCOM_DUMMY_VAL -1 +struct msi_iova_data msi_data; +static int force_stage;module_param (force_stage, int,S_IRUGO ); MODULE_PARM_DESC(force_stage,
@@ -1593,8 +1595,9 @@ static void arm_smmu_get_resv_regions(struct device *dev,struct iommu_resv_region *region;int prot =IOMMU_WRITE |IOMMU_NOEXEC |IOMMU_MMIO ; - region =iommu_alloc_resv_region (MSI_IOVA_BASE ,MSI_IOVA_LENGTH , - prot,IOMMU_RESV_SW_MSI ,GFP_KERNEL ); + region =iommu_alloc_resv_region (msi_data.msi_iova_base, + msi_data.msi_iova_length, prot, +IOMMU_RESV_SW_MSI ,GFP_KERNEL );if (!region)return ;
@@ -2029,6 +2032,9 @@ static int arm_smmu_device_dt_probe(struct arm_smmu_device *smmu,const struct arm_smmu_match_data *data;struct device *dev = smmu->dev;bool legacy_binding; +struct msi_iova_data *msi_data_ptr = &msi_data; + +iommu_configure_msi_iova (dev,"arm,smmu-pci-msi-iova-data" , msi_data_ptr);if (of_property_read_u32 (dev->of_node,"#global-interrupts" , global_irqs)) return dev_err_probe(dev, -ENODEV,
drivers/iommu/virtio-iommu.c +3 -3
@@ -958,9 +958,9 @@ static void viommu_get_resv_regions(struct device *dev, struct list_head *head)* software-mapped region. */ if (!msi) { - msi =iommu_alloc_resv_region (MSI_IOVA_BASE ,MSI_IOVA_LENGTH , - prot,IOMMU_RESV_SW_MSI , -GFP_KERNEL ); + msi =iommu_alloc_resv_region (MSI_IOVA_BASE_DEFAULT , +MSI_IOVA_LENGTH_DEFAULT , prot, +IOMMU_RESV_SW_MSI ,GFP_KERNEL );if (!msi)return ;
include/linux/iommu.h +43 -2
57 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 6 lines · Shyam Saini
Hi,
Currently, the
MSI_IOVA_BASEaddress is hard-coded to0x80000000, assuming that all platforms have this address available for MSI IOVA reservation. However, this is not always the case, as some platforms reserve this address for other purposes.
Can you explain this some more? This address is in the kernel controlled IOVA space, there are few ways a platform can impact this.
How is the platform impacting it? Is the non-functional IOVA always reflected in the iommu_get_resv_regions()?
Why not avoid this conflict in your platform software?
Quoted 2 lines
There was an [1] attempt to fix this problem by passing the MSI IOVA base as a kernel command line parameter.
Yuk
Quoted 5 lines
In the previous attempt,
Will suggested reserving the MSI IOVA at runtime whenever there is a conflict with the defaultMSI_IOVA_BASE. However, dynamically reserving this address has debuggability concerns, as it becomes difficult to track IOMMU mapping failures.
Still, this approach seems like the best to me..
Quoted 5 lines
This patch series aims to address the issue by introducing a new DTS property, "arm,smmu-pci-msi-iova-data". This property allows the configuration of MSI IOVA with a custom MSI base address and a custom length for IOMMU/SMMU drivers. It accommodates platforms that do not have the default MSI base address available for MSI reservation.
My understand was using DT to set kernel configurables was frowned upon? Ultimately MSI_IOVA_BASE is an arbitary choice by kernel software.
Jason
Hi Jason,
Quoted 8 lines · On Mon, 20 Jan 2025 10:26:43 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 6 lines · Shyam Saini
Hi,
Currently, the
MSI_IOVA_BASEaddress is hard-coded to0x80000000, assuming that all platforms have this address available for MSI IOVA reservation. However, this is not always the case, as some platforms reserve this address for other purposes.
Quoted 5 lines
Can you explain this some more? This address is in the kernel controlled IOVA space, there are few ways a platform can impact this.
How is the platform impacting it? Is the non-functional IOVA always reflected in the
iommu_get_resv_regions()?
I don't know the platform impact but just to clarify, are you asking whether this non-functional IOVA is also under IORT RMR or other FW tables? I don't think it is.
But this special IOVA is reflected in iommu_get_resv_regions() the same way as the hardcoded MSI_IOVA_BASE. So each iommu group's reserved_regions should show.
Quoted 1 line
Why not avoid this conflict in your platform software?
I had the same question but it seems there is not enough difference (than the standard smmu) to justify a platform code. i.e. platform specific iommu_get_resv_regions(), is that what you are suggesting?
Quoted 25 lines
Quoted 2 lines
There was an [1] attempt to fix this problem by passing the MSI IOVA base as a kernel command line parameter.
Yuk
Quoted 5 lines
In the previous attempt,
Will suggested reserving the MSI IOVA at runtime whenever there is a conflict with the defaultMSI_IOVA_BASE. However, dynamically reserving this address has debuggability concerns, as it becomes difficult to track IOMMU mapping failures.Still, this approach seems like the best to me..
Quoted 6 lines
This patch series aims to address the issue by introducing a new DTS property, "arm,smmu-pci-msi-iova-data". This property allows the configuration of MSI IOVA with a custom MSI base address and a custom length for IOMMU/SMMU drivers. It accommodates platforms that do not have the default MSI base address available for MSI reservation.
My understand was using DT to set kernel configurables was frowned upon? Ultimately
MSI_IOVA_BASEis an arbitary choice by kernel software.Jason
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Jacob Pan
Quoted 11 lines · Shyam Saini
Quoted 6 lines
Hi,
Currently, the
MSI_IOVA_BASEaddress is hard-coded to0x80000000, assuming that all platforms have this address available for MSI IOVA reservation. However, this is not always the case, as some platforms reserve this address for other purposes.Can you explain this some more? This address is in the kernel controlled IOVA space, there are few ways a platform can impact this.
Quoted 6 lines
Quoted 2 lines
How is the platform impacting it? Is the non-functional IOVA always reflected in the
iommu_get_resv_regions()?I don't know the platform impact but just to clarify, are you asking whether this non-functional IOVA is also under IORT RMR or other FW tables? I don't think it is.
No, I'm asking how can you possibly have a HW platform where MSI_IOVA_BASE is unable to be used for DMA?
MSI_IOVA_BASE is 128M, and most ARM platforms put DRAM starting at
0. Most ARM VMMs put DRAM starting at 0 too.
So a platform saying that DMA to 128M doesn't work is pretty broken, to the point it is hard to believe there is a HW issue at work here?
Quoted 3 lines
But this special IOVA is reflected in
iommu_get_resv_regions()the same way as the hardcodedMSI_IOVA_BASE. So each iommu group'sreserved_regionsshould show.
That's great
Quoted 4 lines
Quoted 1 line
Why not avoid this conflict in your platform software?
I had the same question but it seems there is not enough difference (than the standard smmu) to justify a platform code. i.e. platform specific
iommu_get_resv_regions(), is that what you are suggesting?
And here I mean, why not stop marking it reserved in the ACPI/DT inside your firwmare or hypervisor?
This smells like some SW component using the same address Linux uses for some odd purpose. Just change it and let Linux keep using the address it wants?
Jason
Hi Jason,
Apologies for delayed reponse,
Quoted 20 lines · Jacob Pan
Quoted 12 lines
Quoted 12 lines · Shyam Saini
Quoted 6 lines
Hi,
Currently, the
MSI_IOVA_BASEaddress is hard-coded to0x80000000, assuming that all platforms have this address available for MSI IOVA reservation. However, this is not always the case, as some platforms reserve this address for other purposes.Can you explain this some more? This address is in the kernel controlled IOVA space, there are few ways a platform can impact this.
Quoted 6 lines
Quoted 2 lines
How is the platform impacting it? Is the non-functional IOVA always reflected in the
iommu_get_resv_regions()?I don't know the platform impact but just to clarify, are you asking whether this non-functional IOVA is also under IORT RMR or other FW tables? I don't think it is.
Quoted 8 lines
No, I'm asking how can you possibly have a HW platform where
MSI_IOVA_BASEis unable to be used for DMA?
MSI_IOVA_BASEis 128M, and most ARM platforms put DRAM starting at
0. Most ARM VMMs put DRAM starting at 0 too.So a platform saying that DMA to 128M doesn't work is pretty broken, to the point it is hard to believe there is a HW issue at work here?
Correct, this is limitation with our HW.
Since we can't fix it in hardware, we would need to fix it in Linux.
Quoted 6 lines
Quoted 3 lines
But this special IOVA is reflected in
iommu_get_resv_regions()the same way as the hardcodedMSI_IOVA_BASE. So each iommu group'sreserved_regionsshould show.That's great
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Why not avoid this conflict in your platform software?
I had the same question but it seems there is not enough difference (than the standard smmu) to justify a platform code. i.e. platform specific
iommu_get_resv_regions(), is that what you are suggesting?And here I mean, why not stop marking it reserved in the ACPI/DT inside your firwmare or hypervisor?
even if we reserve it in dts we would still need some address reserved for MSI IOVA
Quoted 3 lines
This smells like some SW component using the same address Linux uses for some odd purpose. Just change it and let Linux keep using the address it wants?
Unfortunately, it is an HW issue.
Are you okay with this passing custom MSI_IOVA via DTS approach ?
Thanks,
Shyam
Quoted 1 line · Shyam Saini
Unfortunately, it is an HW issue.
Well, that's pretty bad to have built HW that can't DMA to low addresses at all.. But OK.
Quoted 1 line
Are you okay with this passing custom
MSI_IOVAvia DTS approach ?
It isn't up to me, but I've understood the DT maintainers would reject this as it isn't is describing HW but just a random Linux software knob.
I think you should make selecting the sw_msi dynamic in Linux.
Jason
Hi Jason,
Quoted 13 lines · Shyam Saini
Quoted 2 lines
Unfortunately, it is an HW issue.
Well, that's pretty bad to have built HW that can't DMA to low addresses at all.. But OK.
Quoted 1 line
Are you okay with this passing custom
MSI_IOVAvia DTS approach ?It isn't up to me, but I've understood the DT maintainers would reject this as it isn't is describing HW but just a random Linux software knob.
If i understood correctly MSI window IOVA is hw property, if yes then it should be accepted by DTS folks, did i misundertstand that?
Quoted 1 line
I think you should make selecting the
sw_msidynamic in Linux.
My understanding is that if we have to make it dynamic, we have to use iova allocator that would need iova_domain as a member of struct arm_smmu_domain,
and allocate iova for MSI window dynamically using alloc_iova() in arm_smmu_get_resv_regions() is that what you meant when you mentioned selecting the sw_msi dynamically?
later we still have to [1] re-adjust the final iova list to IOVA range - resereved regions (msi)
[1] https://elixir.bootlin.com/linux/v6.13.1/source/drivers/vfio/vfio_iommu_type1.c#L2232
so either static or dynamic, we seems to need MSI allocation
Please let me know what you think about this?
Thanks,
Shyam
Quoted 11 lines · Shyam Saini
Hi Jason,
Quoted 9 lines · Shyam Saini
Quoted 2 lines
Unfortunately, it is an HW issue.
Well, that's pretty bad to have built HW that can't DMA to low addresses at all.. But OK.
Quoted 1 line
Are you okay with this passing custom
MSI_IOVAvia DTS approach ?
Quoted 7 lines
Quoted 4 lines
It isn't up to me, but I've understood the DT maintainers would reject this as it isn't is describing HW but just a random Linux software knob.
If i understood correctly MSI window IOVA is hw property, if yes then it should be accepted by DTS folks, did i misundertstand that?
It is not a HW property. The MSI window in the SMMU is entirely up to software.
Yuor HW proprerty is the regions of IOVA that do not function in the SMMU.
Quoted 7 lines
Quoted 1 line
I think you should make selecting the
sw_msidynamic in Linux.My understanding is that if we have to make it dynamic, we have to use iova allocator that would need
iova_domainas a member of structarm_smmu_domain, and allocate iova for MSI window dynamically usingalloc_iova()inarm_smmu_get_resv_regions()is that what you meant when you mentioned selecting thesw_msidynamically?
I would not structure it like that..
The simplest thing would be to have the SMMU driver have a list of potential addresses and select the first one that does not intersect with the non-working IOVA ranges.
Jason