Skip to contentSkip to conversation

PCI: dwc: Strengthen the MSI address allocation logic

PATCH Superseded Ajay Agarwal 2 participants 3 messages Feb 21, 2024

Subsystems (3)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 PCI: dwc: Strengthen the MSI address allocation logic

There can be platforms that do not use/have 32-bit DMA addresses. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.

Try to allocate a 32-bit msi_data. If this allocation fails, attempt a 64-bit address allocation. Please note that if the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address only will not work.

---
Changelog since v4:
- Remove the 'DW_PCIE_CAP_MSI_DATA_SET' flag
- Refactor the comments and msi_data allocation logic

Changelog since v3:
- Add a new controller cap flag 'DW_PCIE_CAP_MSI_DATA_SET'
- Refactor the comments and print statements

Changelog since v2:
- If the vendor driver has setup the msi_data, use the same

Changelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails

---
 .../pci/controller/dwc/pcie-designware-host.c | 21 ++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +14 -7
index d5fc31f8345f..9c905e5c4904 100644
@@ -379,15 +379,22 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
 	 * memory.
 	 */
 	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
-	if (ret)
-		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
+	if (!ret)
+		msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
+						GFP_KERNEL);
 
-	msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
-					GFP_KERNEL);
 	if (!msi_vaddr) {
-		dev_err(dev, "Failed to alloc and map MSI data\n");
-		dw_pcie_free_msi(pp);
-		return -ENOMEM;
+		dev_warn(dev, "Failed to configure 32-bit MSI address. Devices with only 32-bit MSI support may not work properly\n");
+		ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(64));
+		if (!ret)
+			msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
+							GFP_KERNEL);
+
+		if (!msi_vaddr) {
+			dev_err(dev, "Failed to configure MSI address\n");
+			dw_pcie_free_msi(pp);
+			return -ENOMEM;
+		}
 	}
 
 	return 0;
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
#1 lore raw
Quoted 35 lines · Ajay Agarwal

There can be platforms that do not use/have 32-bit DMA addresses. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.

Try to allocate a 32-bit msi_data. If this allocation fails, attempt a 64-bit address allocation. Please note that if the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address only will not work.

---
Changelog since v4:
- Remove the 'DW_PCIE_CAP_MSI_DATA_SET' flag
- Refactor the comments and msi_data allocation logic

Changelog since v3:
- Add a new controller cap flag 'DW_PCIE_CAP_MSI_DATA_SET'
- Refactor the comments and print statements

Changelog since v2:
- If the vendor driver has setup the msi_data, use the same

Changelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails

---
 .../pci/controller/dwc/pcie-designware-host.c | 21 ++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +0 -0
index d5fc31f8345f..9c905e5c4904 100644

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 385

Quoted 9 lines
drivers/pci/controller/dwc/pcie-designware-host.c +3 -2
@@ -379,15 +379,22 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
 	 * memory.
 	 */
 	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
-	if (ret)
-		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
 
+	if (!ret)
+		msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
+						GFP_KERNEL);

msi_vaddr will be left uninitialized if the mask setting fails. Robin had it initialized in v2 discussion:
https://lore.kernel.org/linux-pci/[email protected]/

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)

Quoted 9 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1 -5
-	msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
-					GFP_KERNEL);
 	if (!msi_vaddr) {
-		dev_err(dev, "Failed to alloc and map MSI data\n");
-		dw_pcie_free_msi(pp);
-		return -ENOMEM;
 
+		dev_warn(dev, "Failed to configure 32-bit MSI address. Devices with only 32-bit MSI support may not work properly\n");

Can we short it up already? I suggested something like "Failed to allocate 32-bit MSI target address" in v2. It means the problem with 32-bit MSIs which can be perceived as the possible reason of the respective peripheral devices not working correctly.

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 389

Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(64));
+		if (!ret)

Redundant check. It was discussed some time ago. See the comment from Robin:
https://lore.kernel.org/linux-pci/[email protected]/

-Serge(y)

Quoted 14 lines
drivers/pci/controller/dwc/pcie-designware-host.c +8 -0
+			msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
+							GFP_KERNEL);
+
+		if (!msi_vaddr) {
+			dev_err(dev, "Failed to configure MSI address\n");
+			dw_pcie_free_msi(pp);
+			return -ENOMEM;
+		}
 	}
 

return 0;

Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog
#2 lore raw
Quoted 42 lines · Serge Semin
Quoted 40 lines · Ajay Agarwal

There can be platforms that do not use/have 32-bit DMA addresses. The current implementation of 32-bit IOVA allocation can fail for such platforms, eventually leading to the probe failure.

Try to allocate a 32-bit msi_data. If this allocation fails, attempt a 64-bit address allocation. Please note that if the 64-bit MSI address is allocated, then the EPs supporting 32-bit MSI address only will not work.

---
Changelog since v4:
- Remove the 'DW_PCIE_CAP_MSI_DATA_SET' flag
- Refactor the comments and msi_data allocation logic

Changelog since v3:
- Add a new controller cap flag 'DW_PCIE_CAP_MSI_DATA_SET'
- Refactor the comments and print statements

Changelog since v2:
- If the vendor driver has setup the msi_data, use the same

Changelog since v1:
- Use reserved memory, if it exists, to setup the MSI data
- Fallback to 64-bit IOVA allocation if 32-bit allocation fails

---
 .../pci/controller/dwc/pcie-designware-host.c | 21 ++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +0 -2
index d5fc31f8345f..9c905e5c4904 100644
@@ -379,15 +379,22 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
 	 * memory.
 	 */
 	ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
-	if (ret)
-		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
Quoted 8 lines

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 384

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +3 -0
+	if (!ret)
+		msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
+						GFP_KERNEL);

msi_vaddr will be left uninitialized if the mask setting fails. Robin had it initialized in v2 discussion:
https://lore.kernel.org/linux-pci/[email protected]/

ACK. Will initialize to NULL in the next version.

Quoted 9 lines
Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c +0 -5
-	msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
-					GFP_KERNEL);
 	if (!msi_vaddr) {
-		dev_err(dev, "Failed to alloc and map MSI data\n");
-		dw_pcie_free_msi(pp);
-		return -ENOMEM;
Quoted 7 lines

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 387

Quoted 1 line
drivers/pci/controller/dwc/pcie-designware-host.c +1 -0
+		dev_warn(dev, "Failed to configure 32-bit MSI address. Devices with only 32-bit MSI support may not work properly\n");

Can we short it up already? I suggested something like "Failed to allocate 32-bit MSI target address" in v2. It means the problem with 32-bit MSIs which can be perceived as the possible reason of the respective peripheral devices not working correctly.

ACK. Will shorten the message in the next version.

Quoted 7 lines

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 389

Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(64));
+		if (!ret)

Redundant check. It was discussed some time ago. See the comment from Robin:
https://lore.kernel.org/linux-pci/[email protected]/

ACK. Will remove the check in the next version.

Quoted 16 lines

-Serge(y)

Quoted 14 lines
drivers/pci/controller/dwc/pcie-designware-host.c +8 -0
+			msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data,
+							GFP_KERNEL);
+
+		if (!msi_vaddr) {
+			dev_err(dev, "Failed to configure MSI address\n");
+			dw_pcie_free_msi(pp);
+			return -ENOMEM;
+		}
 	}
 
 	return 0;
Signature · 1 line
2.44.0.rc0.258.g7320e95886-goog