PCI: dwc: Add support for 64-bit MSI target addresses

Re-work the msi_msg DMA allocation logic to use dma_alloc_coherent() which uses the coherent DMA mask to try and return an allocation within the DMA mask limits. This allows kernel configurations that disable ZONE_DMA32 to continue supporting a 32-bit DMA mask. Without this patch, the PCIe host device will fail to probe when ZONE_DMA32 is disabled.

Fixes 35797e672ff0 ("PCI: dwc: Fix MSI msi_msg DMA mapping")

Reported-by Isaac J. Manjarres <[email protected]>

Signed-off-by Will McVicker <[email protected]>

---
 .../pci/controller/dwc/pcie-designware-host.c | 23 ++++++++-----------
 1 file changed, 9 insertions(+), 14 deletions(-)

drivers/pci/controller/dwc/pcie-designware-host.c +4 -4

index 7746f94a715f..8f2222f51671 100644
@@ -272,9 +272,9 @@ static void dw_pcie_free_msi(struct dw_pcie_rp *pp)
 		struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 		struct device *dev = pci->dev;
 
-		dma_unmap_page(dev, pp->msi_data, PAGE_SIZE, DMA_FROM_DEVICE);
-		if (pp->msi_page)
-			__free_page(pp->msi_page);
+		dma_free_coherent(dev, PAGE_SIZE, pp->msi_page, pp->msi_data);
+		pp->msi_data = 0;
+		pp->msi_page = NULL;
 	}
 }
 

@@ -375,22 +375,17 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
 						    dw_chained_msi_isr, pp);
 	}
 
-	ret = dma_set_mask(dev, DMA_BIT_MASK(32));
+	ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
 	if (ret)
 		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
 

drivers/pci/controller/dwc/pcie-designware-host.c +3 -2

+	if (!pp->msi_page) {
+		dev_err(dev, "Failed to alloc and map MSI data\n");
 		pp->msi_data = 0;
 		dw_pcie_free_msi(pp);
-
-		return ret;
+		return -ENOMEM;
 	}
 
 	return 0;

Signature · 1 line

2.37.1.559.g78731f0fdb-goog

Quoted 39 lines · Will McVicker

Re-work the msi_msg DMA allocation logic to use dma_alloc_coherent() which uses the coherent DMA mask to try and return an allocation within the DMA mask limits. This allows kernel configurations that disable ZONE_DMA32 to continue supporting a 32-bit DMA mask. Without this patch, the PCIe host device will fail to probe when ZONE_DMA32 is disabled.

Fixes 35797e672ff0 ("PCI: dwc: Fix MSI msi_msg DMA mapping")

Reported-by Isaac J. Manjarres <[email protected]>

Signed-off-by Will McVicker <[email protected]>

---
 .../pci/controller/dwc/pcie-designware-host.c | 23 ++++++++-----------
 1 file changed, 9 insertions(+), 14 deletions(-)

drivers/pci/controller/dwc/pcie-designware-host.c +4 -4

index 7746f94a715f..8f2222f51671 100644
@@ -272,9 +272,9 @@ static void dw_pcie_free_msi(struct dw_pcie_rp *pp)
 		struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 		struct device *dev = pci->dev;
 
-		dma_unmap_page(dev, pp->msi_data, PAGE_SIZE, DMA_FROM_DEVICE);
-		if (pp->msi_page)
-			__free_page(pp->msi_page);
+		dma_free_coherent(dev, PAGE_SIZE, pp->msi_page, pp->msi_data);
+		pp->msi_data = 0;
+		pp->msi_page = NULL;
 	}
 }
 

@@ -375,22 +375,17 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
 						    dw_chained_msi_isr, pp);
 	}
 
-	ret = dma_set_mask(dev, DMA_BIT_MASK(32));
+	ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
 	if (ret)
 		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
 

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) · line 383

Quoted 10 lines

drivers/pci/controller/dwc/pcie-designware-host.c +2 -8

-	pp->msi_page = alloc_page(GFP_DMA32);
-	pp->msi_data = dma_map_page(dev, pp->msi_page, 0,
-				    PAGE_SIZE, DMA_FROM_DEVICE);
-	ret = dma_mapping_error(dev, pp->msi_data);
-	if (ret) {
-		dev_err(pci->dev, "Failed to map MSI data\n");
-		__free_page(pp->msi_page);
-		pp->msi_page = NULL;
+	pp->msi_page = dma_alloc_coherent(dev, PAGE_SIZE, &pp->msi_data,
+					  GFP_KERNEL);

You can use the managed version, dmam_alloc_coherent(), and avoid the freeing yourself. Also with that, I think you don't need 'msi_page'?

Also, no need to alloc a whole page. A u32 or u64? should be fine. The write never makes it to memory, so doesn't really matter.

Quoted 9 lines

drivers/pci/controller/dwc/pcie-designware-host.c +3 -2

+	if (!pp->msi_page) {
+		dev_err(dev, "Failed to alloc and map MSI data\n");
 		pp->msi_data = 0;
 		dw_pcie_free_msi(pp);
-
-		return ret;
+		return -ENOMEM;
 	}
 

Quoted 4 lines

drivers/pci/controller/dwc/pcie-designware-host.c +0 -0

 	return 0;

Signature · 1 line

2.37.1.559.g78731f0fdb-goog

Since not all devices require a 32-bit MSI address, add support to the PCIe host driver to allow setting the DMA mask to 64-bits. This allows kernels to disable ZONE_DMA32 and bounce buffering (swiotlb) without risking not being able to get a 32-bit address during DMA allocation. Basically, in the slim chance that there are no 32-bit allocations available, the current PCIe host driver will fail to allocate the msi_msg page due to a DMA address overflow (seen in [1]). With this patch, the PCIe driver can advertise 64-bit support via it's MSI capabilities to hint to the PCIe host driver to set the DMA mask to 64-bits.

[1] https://lore.kernel.org/all/[email protected]/

Reported-by kernel test robot <[email protected]>

Signed-off-by Will McVicker <[email protected]>

---
 drivers/pci/controller/dwc/pcie-designware-host.c | 14 ++++++++++++--
 drivers/pci/controller/dwc/pcie-designware.c      |  9 +++++++++
 drivers/pci/controller/dwc/pcie-designware.h      |  1 +
 3 files changed, 22 insertions(+), 2 deletions(-)