Skip to contentSkip to conversation

PCI: dwc: Use multiple ATU regions for large bridge windows

PATCH Applied Randolph 5 participants 5 messages Jan 9 – Jan 22

Subsystems (3)
lore
Share & tools
lore

Randolph

author
#0 lore raw

PATCH v2 1/1 PCI: dwc: Use multiple ATU regions for large bridge windows

From: Samuel Holland <[email protected]>

Some SoCs may allocate more address space for a bridge window than can be covered by a single ATU region. Allow using a larger bridge window by allocating multiple adjacent ATU regions.

---
Since our changes depend on the original patch and no updated revision has been posted by the original author, we reached out to Samuel Holland directly and received his approval.I have consolidated the required changes and am resending them as v2.

---
 .../pci/controller/dwc/pcie-designware-host.c | 72 ++++++++++++++-----
 1 file changed, 55 insertions(+), 17 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +55 -17
index 372207c33a85..771e71b40f76 100644
@@ -903,29 +903,49 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 
 	i = 0;
 	resource_list_for_each_entry(entry, &pp->bridge->windows) {
+		resource_size_t res_size;
+
 		if (resource_type(entry->res) != IORESOURCE_MEM)
 			continue;
 
-		if (pci->num_ob_windows <= ++i)
+		if (pci->num_ob_windows <= i + 1)
 			break;
 
-		atu.index = i;
 		atu.type = PCIE_ATU_TYPE_MEM;
 		atu.parent_bus_addr = entry->res->start - pci->parent_bus_offset;
 		atu.pci_addr = entry->res->start - entry->offset;
 
 		/* Adjust iATU size if MSG TLP region was allocated before */
 		if (pp->msg_res && pp->msg_res->parent == entry->res)
-			atu.size = resource_size(entry->res) -
+			res_size = resource_size(entry->res) -
 					resource_size(pp->msg_res);
 		else
-			atu.size = resource_size(entry->res);
+			res_size = resource_size(entry->res);
+
+		while (res_size > 0) {
+			/*
+			 * Make sure to fail probe if we run out of windows
+			 * in the middle and we would end up only partially
+			 * mapping a single resource
+			 */
+			if (pci->num_ob_windows <= ++i) {
+				dev_err(pci->dev, "Exhausted outbound windows mapping %pr\n",
+					entry->res);
+				return -ENOMEM;
+			}
+			atu.index = i;
+			atu.size = MIN(pci->region_limit + 1, res_size);
 
-		ret = dw_pcie_prog_outbound_atu(pci, &atu);
-		if (ret) {
-			dev_err(pci->dev, "Failed to set MEM range %pr\n",
-				entry->res);
-			return ret;
+			ret = dw_pcie_prog_outbound_atu(pci, &atu);
+			if (ret) {
+				dev_err(pci->dev, "Failed to set MEM range %pr\n",
+					entry->res);
+				return ret;
+			}
+
+			atu.parent_bus_addr += atu.size;
+			atu.pci_addr += atu.size;
+			res_size -= atu.size;
 		}
 	}
 
@@ -956,20 +976,38 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 
 	i = 0;
 	resource_list_for_each_entry(entry, &pp->bridge->dma_ranges) {
+		resource_size_t res_start, res_size, window_size;
+
 		if (resource_type(entry->res) != IORESOURCE_MEM)
 			continue;
 
 		if (pci->num_ib_windows <= i)
 			break;
 
-		ret = dw_pcie_prog_inbound_atu(pci, i++, PCIE_ATU_TYPE_MEM,
-					       entry->res->start,
-					       entry->res->start - entry->offset,
-					       resource_size(entry->res));
-		if (ret) {
-			dev_err(pci->dev, "Failed to set DMA range %pr\n",
-				entry->res);
-			return ret;
+		res_size = resource_size(entry->res);
+		res_start = entry->res->start;
+		while (res_size >= 0) {
+			/*
+			 * Make sure to fail probe if we run out of windows
+			 * in the middle and we would end up only partially
+			 * mapping a single resource
+			 */
+			if (pci->num_ib_windows <= i) {
+				dev_err(pci->dev, "Exhausted inbound windows mapping %pr\n",
+					entry->res);
+				return -ENOMEM;
+			}
+			window_size = MIN(pci->region_limit + 1, res_size);
+			ret = dw_pcie_prog_inbound_atu(pci, i++, PCIE_ATU_TYPE_MEM, res_start,
+						       res_start - entry->offset, window_size);
+			if (ret) {
+				dev_err(pci->dev, "Failed to set DMA range %pr\n",
+					entry->res);
+				return ret;
+			}
+
+			res_start += window_size;
+			res_size -= window_size;
 		}
 	}
 
Signature · 1 line
2.34.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 6 lines · Randolph

From: Samuel Holland <[email protected]>

Some SoCs may allocate more address space for a bridge window than can be covered by a single ATU region. Allow using a larger bridge window by allocating multiple adjacent ATU regions.

Quoted 7 lines

---

#2 lore raw
In reply to Randolph

Hi Randolph—

Thanks for sending this again!

Quoted 98 lines · Randolph

From: Samuel Holland <[email protected]>

Some SoCs may allocate more address space for a bridge window than can be covered by a single ATU region. Allow using a larger bridge window by allocating multiple adjacent ATU regions.

---
Since our changes depend on the original patch and no updated revision has been posted by the original author, we reached out to Samuel Holland directly and received his approval.I have consolidated the required changes and am resending them as v2.

---
 .../pci/controller/dwc/pcie-designware-host.c | 72 ++++++++++++++-----
 1 file changed, 55 insertions(+), 17 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +31 -10
index 372207c33a85..771e71b40f76 100644
@@ -903,29 +903,49 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 
        i = 0;
        resource_list_for_each_entry(entry, &pp->bridge->windows) {
+               resource_size_t res_size;
+
                if (resource_type(entry->res) != IORESOURCE_MEM)
                        continue;
 
-               if (pci->num_ob_windows <= ++i)
+               if (pci->num_ob_windows <= i + 1)
                        break;
 
-               atu.index = i;
                atu.type = PCIE_ATU_TYPE_MEM;
                atu.parent_bus_addr = entry->res->start - pci->parent_bus_offset;
                atu.pci_addr = entry->res->start - entry->offset;
 
                /* Adjust iATU size if MSG TLP region was allocated before */
                if (pp->msg_res && pp->msg_res->parent == entry->res)
-                       atu.size = resource_size(entry->res) -
+                       res_size = resource_size(entry->res) -
                                        resource_size(pp->msg_res);
                else
-                       atu.size = resource_size(entry->res);
+                       res_size = resource_size(entry->res);
+
+               while (res_size > 0) {
+                       /*
+                        * Make sure to fail probe if we run out of windows
+                        * in the middle and we would end up only partially
+                        * mapping a single resource
+                        */
+                       if (pci->num_ob_windows <= ++i) {
+                               dev_err(pci->dev, "Exhausted outbound windows mapping %pr\n",
+                                       entry->res);
+                               return -ENOMEM;
+                       }
+                       atu.index = i;
+                       atu.size = MIN(pci->region_limit + 1, res_size);
 
-               ret = dw_pcie_prog_outbound_atu(pci, &atu);
-               if (ret) {
-                       dev_err(pci->dev, "Failed to set MEM range %pr\n",
-                               entry->res);
-                       return ret;
+                       ret = dw_pcie_prog_outbound_atu(pci, &atu);
+                       if (ret) {
+                               dev_err(pci->dev, "Failed to set MEM range %pr\n",
+                                       entry->res);
+                               return ret;
+                       }
+
+                       atu.parent_bus_addr += atu.size;
+                       atu.pci_addr += atu.size;
+                       res_size -= atu.size;
                }
        }
 
@@ -956,20 +976,38 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 
        i = 0;
        resource_list_for_each_entry(entry, &pp->bridge->dma_ranges) {
+               resource_size_t res_start, res_size, window_size;
+
                if (resource_type(entry->res) != IORESOURCE_MEM)
                        continue;
 
                if (pci->num_ib_windows <= i)
                        break;
 
-               ret = dw_pcie_prog_inbound_atu(pci, i++, PCIE_ATU_TYPE_MEM,

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp) · line 989

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +3 -7
-                                              entry->res->start,
-                                              entry->res->start - entry->offset,
-                                              resource_size(entry->res));
-               if (ret) {
-                       dev_err(pci->dev, "Failed to set DMA range %pr\n",
-                               entry->res);
-                       return ret;
+               res_size = resource_size(entry->res);
+               res_start = entry->res->start;
+               while (res_size >= 0) {

I think this should be > 0 instead of >= 0 like the loop from the outbound window. Looking back it is actually my own mistake from the reply I sent to v1 that you faithfully replicated. One of those times you wish for git blame-someone-else :^)

Hopefully this can just get folded in as the patch is taken to avoid a v3.

Quoted 25 lines
drivers/pci/controller/dwc/pcie-designware-host.c +21 -1
+                       /*
+                        * Make sure to fail probe if we run out of windows
+                        * in the middle and we would end up only partially
+                        * mapping a single resource
+                        */
+                       if (pci->num_ib_windows <= i) {
+                               dev_err(pci->dev, "Exhausted inbound windows mapping %pr\n",
+                                       entry->res);
+                               return -ENOMEM;
+                       }
+                       window_size = MIN(pci->region_limit + 1, res_size);
+                       ret = dw_pcie_prog_inbound_atu(pci, i++, PCIE_ATU_TYPE_MEM, res_start,
+                                                      res_start - entry->offset, window_size);
+                       if (ret) {
+                               dev_err(pci->dev, "Failed to set DMA range %pr\n",
+                                       entry->res);
+                               return ret;
+                       }
+
+                       res_start += window_size;
+                       res_size -= window_size;
                }
        }
 
--
Quoted 2 lines

2.34.1

Best—Charlie

#3 lore raw
Quoted 119 lines · Charles Mirabile

Hi Randolph—

Thanks for sending this again!

Quoted 98 lines · Randolph

From: Samuel Holland <[email protected]>

Some SoCs may allocate more address space for a bridge window than can be covered by a single ATU region. Allow using a larger bridge window by allocating multiple adjacent ATU regions.

---
Since our changes depend on the original patch and no updated revision has been posted by the original author, we reached out to Samuel Holland directly and received his approval.I have consolidated the required changes and am resending them as v2.

---
 .../pci/controller/dwc/pcie-designware-host.c | 72 ++++++++++++++-----
 1 file changed, 55 insertions(+), 17 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +31 -10
index 372207c33a85..771e71b40f76 100644
@@ -903,29 +903,49 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 
        i = 0;
        resource_list_for_each_entry(entry, &pp->bridge->windows) {
+               resource_size_t res_size;
+
                if (resource_type(entry->res) != IORESOURCE_MEM)
                        continue;
 
-               if (pci->num_ob_windows <= ++i)
+               if (pci->num_ob_windows <= i + 1)
                        break;
 
-               atu.index = i;
                atu.type = PCIE_ATU_TYPE_MEM;
                atu.parent_bus_addr = entry->res->start - pci->parent_bus_offset;
                atu.pci_addr = entry->res->start - entry->offset;
 
                /* Adjust iATU size if MSG TLP region was allocated before */
                if (pp->msg_res && pp->msg_res->parent == entry->res)
-                       atu.size = resource_size(entry->res) -
+                       res_size = resource_size(entry->res) -
                                        resource_size(pp->msg_res);
                else
-                       atu.size = resource_size(entry->res);
+                       res_size = resource_size(entry->res);
+
+               while (res_size > 0) {
+                       /*
+                        * Make sure to fail probe if we run out of windows
+                        * in the middle and we would end up only partially
+                        * mapping a single resource
+                        */
+                       if (pci->num_ob_windows <= ++i) {
+                               dev_err(pci->dev, "Exhausted outbound windows mapping %pr\n",
+                                       entry->res);
+                               return -ENOMEM;
+                       }
+                       atu.index = i;
+                       atu.size = MIN(pci->region_limit + 1, res_size);
 
-               ret = dw_pcie_prog_outbound_atu(pci, &atu);
-               if (ret) {
-                       dev_err(pci->dev, "Failed to set MEM range %pr\n",
-                               entry->res);
-                       return ret;
+                       ret = dw_pcie_prog_outbound_atu(pci, &atu);
+                       if (ret) {
+                               dev_err(pci->dev, "Failed to set MEM range %pr\n",
+                                       entry->res);
+                               return ret;
+                       }
+
+                       atu.parent_bus_addr += atu.size;
+                       atu.pci_addr += atu.size;
+                       res_size -= atu.size;
                }
        }
 
@@ -956,20 +976,38 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 
        i = 0;
        resource_list_for_each_entry(entry, &pp->bridge->dma_ranges) {
+               resource_size_t res_start, res_size, window_size;
+
                if (resource_type(entry->res) != IORESOURCE_MEM)
                        continue;
 
                if (pci->num_ib_windows <= i)
                        break;
 
-               ret = dw_pcie_prog_inbound_atu(pci, i++, PCIE_ATU_TYPE_MEM,

drivers/pci/controller/dwc/pcie-designware-host.c static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp) · line 989

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +3 -7
-                                              entry->res->start,
-                                              entry->res->start - entry->offset,
-                                              resource_size(entry->res));
-               if (ret) {
-                       dev_err(pci->dev, "Failed to set DMA range %pr\n",
-                               entry->res);
-                       return ret;
+               res_size = resource_size(entry->res);
+               res_start = entry->res->start;
+               while (res_size >= 0) {

I think this should be > 0 instead of >= 0 like the loop from the outbound window. Looking back it is actually my own mistake from the reply I sent to v1 that you faithfully replicated. One of those times you wish for git blame-someone-else :^)

Quoted 2 lines

Hopefully this can just get folded in as the patch is taken to avoid a v3.

Squashed this change while applying, thanks!

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#4 lore raw
In reply to Randolph
Quoted 5 lines · Randolph

Some SoCs may allocate more address space for a bridge window than can be covered by a single ATU region. Allow using a larger bridge window by allocating multiple adjacent ATU regions.

Applied, thanks!

[1/1] PCI: dwc: Use multiple ATU regions for large bridge windows

      commit: e9a5415adb209f86a05e55b850127ada82e070f1

Best regards,

Signature · 1 line
Manivannan Sadhasivam <[email protected]>