Skip to contentSkip to conversation

PCI: dwc: Fix BAR0 wrong map to iATU6 after root complex reinit endpoint

PATCH In review Frank Li 4 participants 17 messages Dec 19, 2023 – Mar 4, 2024

Subsystems (4)
lore
Share & tools
lore

Frank Li

author maintainer
#0 lore raw

PATCH 1/1 PCI: dwc: Fix BAR0 wrong map to iATU6 after root complex reinit endpoint

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

 drivers/pci/controller/dwc/pcie-designware-ep.c | 11 ++++++++---
 drivers/pci/controller/dwc/pcie-designware.h    |  2 +-
 2 files changed, 9 insertions(+), 4 deletions(-)
drivers/pci/controller/dwc/pcie-designware-ep.c +8 -3
index f6207989fc6ad..0ff5cd64f49b0 100644
@@ -174,7 +174,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
 	u32 free_win;
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
-	if (!ep->bar_to_atu[bar])
+	if (ep->bar_to_atu[bar] < 0)
 		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
 	else
 		free_win = ep->bar_to_atu[bar];
@@ -228,14 +228,17 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	enum pci_barno bar = epf_bar->barno;
-	u32 atu_index = ep->bar_to_atu[bar];
+	s8 atu_index = ep->bar_to_atu[bar];
+
+	if (atu_index < 0)
+		return;
 
 	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
 
 	dw_pcie_disable_atu(pci, PCIE_ATU_REGION_DIR_IB, atu_index);
 	clear_bit(atu_index, ep->ib_window_map);
 	ep->epf_bar[bar] = NULL;
-	ep->bar_to_atu[bar] = 0;
+	ep->bar_to_atu[bar] = -1;
 }
 
 static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
@@ -767,6 +770,8 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 		return -ENOMEM;
 	ep->outbound_addr = addr;
 
+	memset(ep->bar_to_atu, -1, sizeof(ep->bar_to_atu));
+
 	epc = devm_pci_epc_create(dev, &epc_ops);
 	if (IS_ERR(epc)) {
 		dev_err(dev, "Failed to create epc device\n");
drivers/pci/controller/dwc/pcie-designware.h +1 -1
index 55ff76e3d3846..5879907c5cf25 100644
@@ -362,7 +362,7 @@ struct dw_pcie_ep {
 	phys_addr_t		phys_base;
 	size_t			addr_size;
 	size_t			page_size;
-	u8			bar_to_atu[PCI_STD_NUM_BARS];
+	s8			bar_to_atu[PCI_STD_NUM_BARS];
 	phys_addr_t		*outbound_addr;
 	unsigned long		*ib_window_map;
 	unsigned long		*ob_window_map;
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH 1/1 PCI: dwc: Fix BAR0 wrong map to iATU6 after root complex reinit endpoint

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

 drivers/pci/controller/dwc/pcie-designware-ep.c | 11 ++++++++---
 drivers/pci/controller/dwc/pcie-designware.h    |  2 +-
 2 files changed, 9 insertions(+), 4 deletions(-)
drivers/pci/controller/dwc/pcie-designware-ep.c +8 -3
index f6207989fc6ad..0ff5cd64f49b0 100644
@@ -174,7 +174,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
 	u32 free_win;
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
-	if (!ep->bar_to_atu[bar])
+	if (ep->bar_to_atu[bar] < 0)
 		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
 	else
 		free_win = ep->bar_to_atu[bar];
@@ -228,14 +228,17 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	enum pci_barno bar = epf_bar->barno;
-	u32 atu_index = ep->bar_to_atu[bar];
+	s8 atu_index = ep->bar_to_atu[bar];
+
+	if (atu_index < 0)
+		return;
 
 	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
 
 	dw_pcie_disable_atu(pci, PCIE_ATU_REGION_DIR_IB, atu_index);
 	clear_bit(atu_index, ep->ib_window_map);
 	ep->epf_bar[bar] = NULL;
-	ep->bar_to_atu[bar] = 0;
+	ep->bar_to_atu[bar] = -1;
 }
 
 static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
@@ -767,6 +770,8 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 		return -ENOMEM;
 	ep->outbound_addr = addr;
 
+	memset(ep->bar_to_atu, -1, sizeof(ep->bar_to_atu));
+
 	epc = devm_pci_epc_create(dev, &epc_ops);
 	if (IS_ERR(epc)) {
 		dev_err(dev, "Failed to create epc device\n");
drivers/pci/controller/dwc/pcie-designware.h +1 -1
index 55ff76e3d3846..5879907c5cf25 100644
@@ -362,7 +362,7 @@ struct dw_pcie_ep {
 	phys_addr_t		phys_base;
 	size_t			addr_size;
 	size_t			page_size;
-	u8			bar_to_atu[PCI_STD_NUM_BARS];
+	s8			bar_to_atu[PCI_STD_NUM_BARS];
 	phys_addr_t		*outbound_addr;
 	unsigned long		*ib_window_map;
 	unsigned long		*ob_window_map;
Signature · 1 line
2.34.1
Reviewed-by on v1 1/1
#2 lore raw
In reply to Frank Li
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

Quoted 63 lines
 drivers/pci/controller/dwc/pcie-designware-ep.c | 11 ++++++++---
 drivers/pci/controller/dwc/pcie-designware.h    |  2 +-
 2 files changed, 9 insertions(+), 4 deletions(-)
drivers/pci/controller/dwc/pcie-designware-ep.c +8 -3
index f6207989fc6ad..0ff5cd64f49b0 100644
@@ -174,7 +174,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
 	u32 free_win;
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
-	if (!ep->bar_to_atu[bar])
+	if (ep->bar_to_atu[bar] < 0)
 		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
 	else
 		free_win = ep->bar_to_atu[bar];
@@ -228,14 +228,17 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	enum pci_barno bar = epf_bar->barno;
-	u32 atu_index = ep->bar_to_atu[bar];
+	s8 atu_index = ep->bar_to_atu[bar];
+
+	if (atu_index < 0)
+		return;
 
 	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
 
 	dw_pcie_disable_atu(pci, PCIE_ATU_REGION_DIR_IB, atu_index);
 	clear_bit(atu_index, ep->ib_window_map);
 	ep->epf_bar[bar] = NULL;
-	ep->bar_to_atu[bar] = 0;
+	ep->bar_to_atu[bar] = -1;
 }
 
 static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
@@ -767,6 +770,8 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 		return -ENOMEM;
 	ep->outbound_addr = addr;
 
+	memset(ep->bar_to_atu, -1, sizeof(ep->bar_to_atu));
+
 	epc = devm_pci_epc_create(dev, &epc_ops);
 	if (IS_ERR(epc)) {
 		dev_err(dev, "Failed to create epc device\n");
drivers/pci/controller/dwc/pcie-designware.h +1 -1
index 55ff76e3d3846..5879907c5cf25 100644
@@ -362,7 +362,7 @@ struct dw_pcie_ep {
 	phys_addr_t		phys_base;
 	size_t			addr_size;
 	size_t			page_size;
-	u8			bar_to_atu[PCI_STD_NUM_BARS];
+	s8			bar_to_atu[PCI_STD_NUM_BARS];
 	phys_addr_t		*outbound_addr;
 	unsigned long		*ib_window_map;
 	unsigned long		*ob_window_map;
Signature · 1 line
2.34.1
author maintainer
#3 lore raw
Quoted 29 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Quoted 7 lines

Hello Frank,

Thank you for sending a proper fix for this!

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

Frank

Quoted 66 lines
Quoted 63 lines
 drivers/pci/controller/dwc/pcie-designware-ep.c | 11 ++++++++---
 drivers/pci/controller/dwc/pcie-designware.h    |  2 +-
 2 files changed, 9 insertions(+), 4 deletions(-)
drivers/pci/controller/dwc/pcie-designware-ep.c +8 -3
index f6207989fc6ad..0ff5cd64f49b0 100644
@@ -174,7 +174,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
 	u32 free_win;
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
-	if (!ep->bar_to_atu[bar])
+	if (ep->bar_to_atu[bar] < 0)
 		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
 	else
 		free_win = ep->bar_to_atu[bar];
@@ -228,14 +228,17 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	enum pci_barno bar = epf_bar->barno;
-	u32 atu_index = ep->bar_to_atu[bar];
+	s8 atu_index = ep->bar_to_atu[bar];
+
+	if (atu_index < 0)
+		return;
 
 	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
 
 	dw_pcie_disable_atu(pci, PCIE_ATU_REGION_DIR_IB, atu_index);
 	clear_bit(atu_index, ep->ib_window_map);
 	ep->epf_bar[bar] = NULL;
-	ep->bar_to_atu[bar] = 0;
+	ep->bar_to_atu[bar] = -1;
 }
 
 static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
@@ -767,6 +770,8 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 		return -ENOMEM;
 	ep->outbound_addr = addr;
 
+	memset(ep->bar_to_atu, -1, sizeof(ep->bar_to_atu));
+
 	epc = devm_pci_epc_create(dev, &epc_ops);
 	if (IS_ERR(epc)) {
 		dev_err(dev, "Failed to create epc device\n");
drivers/pci/controller/dwc/pcie-designware.h +1 -1
index 55ff76e3d3846..5879907c5cf25 100644
@@ -362,7 +362,7 @@ struct dw_pcie_ep {
 	phys_addr_t		phys_base;
 	size_t			addr_size;
 	size_t			page_size;
-	u8			bar_to_atu[PCI_STD_NUM_BARS];
+	s8			bar_to_atu[PCI_STD_NUM_BARS];
 	phys_addr_t		*outbound_addr;
 	unsigned long		*ib_window_map;
 	unsigned long		*ob_window_map;
Signature · 1 line
2.34.1
#4 lore raw
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Kind regards,
Niklas

#5 lore raw

Hello Frank,

Quoted 44 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

Quoted 8 lines

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Do you plan to send out a v2?

Kind regards,
Niklas

author maintainer
#6 lore raw
Quoted 50 lines · Niklas Cassel

Hello Frank,

Quoted 47 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Quoted 7 lines
Quoted 5 lines

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Do you plan to send out a v2?

It is easy to change to u16. But I hope Manivannan Sadhasivam have time to review it.

Frank

Quoted 4 lines

Kind regards,
Niklas

author maintainer
#7 lore raw
In reply to Niklas Cassel
Quoted 50 lines · Niklas Cassel

Hello Frank,

Quoted 47 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Quoted 7 lines
Quoted 5 lines

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Do you plan to send out a v2?

@mani:
Do you have any comments about this fixes except u8's problem?

Frank

Quoted 4 lines

Kind regards,
Niklas

#8 lore raw
Quoted 54 lines · Frank Li
Quoted 53 lines · Niklas Cassel

Hello Frank,

Quoted 50 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Do you plan to send out a v2?

@mani:
Do you have any comments about this fixes except u8's problem?

IMO, the core issue lies in the EPF driver. It calls set_bar() during init, but it is not clearing the BARs with clear_bar() during LINK_DOWN.

If the BARs were cleared properly, then we would not see this issue.

I'm planning to do a cleanup of the behavior of EPF with core_init_notifier and it should get addressed there.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#9 lore raw
Quoted 63 lines · Manivannan Sadhasivam
Quoted 54 lines · Frank Li
Quoted 53 lines · Niklas Cassel

Hello Frank,

Quoted 50 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Do you plan to send out a v2?

@mani:
Do you have any comments about this fixes except u8's problem?

Quoted 7 lines

IMO, the core issue lies in the EPF driver. It calls set_bar() during init, but it is not clearing the BARs with clear_bar() during LINK_DOWN.

If the BARs were cleared properly, then we would not see this issue.

I'm planning to do a cleanup of the behavior of EPF with core_init_notifier and it should get addressed there.

Hello Mani, Frank,

Please reconsider respinning this.

I think that the invalid usage of find_first_zero_bit() is very bad, because other people might copy the find_first_zero_bit() usage here.

find_first_zero_bit() can (and will) return 0, so this check is just bad:

if (!ep->bar_to_atu[bar])

        free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);

...

ep->bar_to_atu[bar] = free_win;
set_bit(free_win, ep->ib_window_map);

No matter which code Mani adds that cleans up the BARs, will not change that the logic is this function is just broken.

Kind regards,
Niklas

author maintainer
#10 lore raw
Quoted 87 lines · Niklas Cassel
Quoted 63 lines · Manivannan Sadhasivam
Quoted 54 lines · Frank Li
Quoted 53 lines · Niklas Cassel

Hello Frank,

Quoted 50 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Do you plan to send out a v2?

@mani:
Do you have any comments about this fixes except u8's problem?

Quoted 7 lines

IMO, the core issue lies in the EPF driver. It calls set_bar() during init, but it is not clearing the BARs with clear_bar() during LINK_DOWN.

If the BARs were cleared properly, then we would not see this issue.

I'm planning to do a cleanup of the behavior of EPF with core_init_notifier and it should get addressed there.

Hello Mani, Frank,

Please reconsider respinning this.

I think that the invalid usage of find_first_zero_bit() is very bad, because other people might copy the find_first_zero_bit() usage here.

find_first_zero_bit() can (and will) return 0, so this check is just bad:

if (!ep->bar_to_atu[bar])

        free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);

...

Quoted 7 lines

ep->bar_to_atu[bar] = free_win;
set_bit(free_win, ep->ib_window_map);

No matter which code Mani adds that cleans up the BARs, will not change that the logic is this function is just broken.

Hello Mani:
if you agree this fix, I can respin this patch by fix s8 problem.

Frank

Quoted 4 lines

Kind regards,
Niklas

#11 lore raw
In reply to Niklas Cassel
Quoted 44 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

Quoted 9 lines

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Agree. I cannot suggest a better alternative. So let's go with this. But please add a comment before bar_to_atu assignment to make it clear. Like,

	/*
	 * Always increment free_win before assignment, since value 0 is used to
	 * identify unallocated mapping.
	 */
	ep->bar_to_atu[bar] = free_win + 1;

- Mani

Quoted 3 lines

Kind regards,
Niklas

Signature · 1 line
மணிவண்ணன் சதாசிவம்
author maintainer
#12 lore raw
Quoted 55 lines · Manivannan Sadhasivam
Quoted 44 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

Quoted 9 lines

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Quoted 8 lines

Agree. I cannot suggest a better alternative. So let's go with this. But please add a comment before bar_to_atu assignment to make it clear. Like,

	/*
	 * Always increment free_win before assignment, since value 0 is used to
	 * identify unallocated mapping.
	 */
	ep->bar_to_atu[bar] = free_win + 1;

This patch already change to use "-1" as free. Only issue for this patch is that use 's16' instead of 's8' becasue max ATU number is 255.

Frank

Quoted 9 lines

- Mani

Quoted 3 lines

Kind regards,
Niklas

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#13 lore raw
Quoted 59 lines · Frank Li
Quoted 58 lines · Manivannan Sadhasivam
Quoted 44 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

Quoted 9 lines

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Agree. I cannot suggest a better alternative. So let's go with this. But please add a comment before bar_to_atu assignment to make it clear. Like,

Quoted 9 lines
Quoted 5 lines
	/*
	 * Always increment free_win before assignment, since value 0 is used to
	 * identify unallocated mapping.
	 */
	ep->bar_to_atu[bar] = free_win + 1;

This patch already change to use "-1" as free. Only issue for this patch is that use 's16' instead of 's8' becasue max ATU number is 255.

Niklas's initial suggestion of keeping u8 for the array and 0 as the unallocated placeholder sounds good to me. Please use that instead.

Even though iATU window index starts from 0, the comment I suggested will clarify what this bar_to_atu[] does.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
author maintainer
#14 lore raw
Quoted 66 lines · Manivannan Sadhasivam
Quoted 65 lines · Frank Li
Quoted 63 lines · Manivannan Sadhasivam
Quoted 44 lines · Niklas Cassel
Quoted 34 lines · Frank Li
Quoted 33 lines · Niklas Cassel
Quoted 19 lines · Frank Li

dw_pcie_ep_inbound_atu()

{
	...
	if (!ep->bar_to_atu[bar])
		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
	else
		free_win = ep->bar_to_atu[bar];
	...
}

The atu index 0 is valid case for atu number. The find_first_zero_bit() will return 6 when second time call into this function if atu is 0. Suppose it should use branch 'free_win = ep->bar_to_atu[bar]'.

Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have not allocate atu to the bar.

Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u

Quoted 8 lines
Fixes 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")

---

Notes:
@Niklas:
I have not test your case. I should be equal to previous's fix in
mail list.

Hello Frank,

Thank you for sending a proper fix for this!

Quoted 8 lines
Quoted 3 lines

Personally, I slightly prefer your fix that saves the iatu index + 1, and keeps 0 to mean unused. That way, you don't need the memset, and you don't need to change the type to signed, but either way is fine by me, so:

index + 1 don't match hardware iATU index. It will be confused because other parts is 0 based.

So I choose "-1" as free iATU.

Quoted 9 lines

A s8 can hold a max value of 127.
CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.

Since the DWC code can be synthesized with 256 iATUs, your code will not work on systems with 128 or more iATUs.

If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs.

Agree. I cannot suggest a better alternative. So let's go with this. But please add a comment before bar_to_atu assignment to make it clear. Like,

	/*
	 * Always increment free_win before assignment, since value 0 is used to
	 * identify unallocated mapping.
	 */
	ep->bar_to_atu[bar] = free_win + 1;
Quoted 7 lines
Quoted 3 lines

This patch already change to use "-1" as free. Only issue for this patch is that use 's16' instead of 's8' becasue max ATU number is 255.

Niklas's initial suggestion of keeping u8 for the array and 0 as the unallocated placeholder sounds good to me. Please use that instead.

It is impossible to keep u8, because 255 + 1 will 0 for u8. Previously Niklas's initial suggestion have not consider this condition. If u8 have to change to u16 or s16 anyways, I prefer use -1 as free.

Frank

Quoted 7 lines

Even though iATU window index starts from 0, the comment I suggested will clarify what this bar_to_atu[] does.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#15 lore raw
Quoted 8 lines · Frank Li
Quoted 4 lines

Niklas's initial suggestion of keeping u8 for the array and 0 as the unallocated placeholder sounds good to me. Please use that instead.

It is impossible to keep u8, because 255 + 1 will 0 for u8. Previously Niklas's initial suggestion have not consider this condition. If u8 have to change to u16 or s16 anyways, I prefer use -1 as free.

Well, to be fair, my suggestion was:
"If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs."

But we have this define:
drivers/pci/controller/dwc/pcie-designware.h:#define MAX_IATU_IN 256 (Even if it isn't used anywhere.)

But as ridiculous as it may seem to support that many inbound ranges, that is the max number of windows supported by the hardware, so why not just let the driver support the max supported by the hardware?

We are talking about:

struct dw_pcie_ep {
	...
        u8                      bar_to_atu[PCI_STD_NUM_BARS];
	...
}

where PCI_STD_NUM_BARS == 6.

And where struct dw_pcie_ep is kzalloced for what I assume is all drivers.

So I'm actually for your idea of changing it to u16, or even unsigned int.

If the code is simplified if we use a u16 or unsigned int (because we don't need any weird if (val < MAX_IATU_IN - 1) check), then I'm all for it.

What I personally don't like with your patch in $subject,
was that you changed both dw_pcie_ep_clear_bar() to set the "clear value" to -1, but you also need a
memset(ep->bar_to_atu, -1, sizeof(ep->bar_to_atu)); in dw_pcie_ep_init().

I much prefer to have 0 as the default/unused value, because it will automatically get set when you do kzalloc().
Seeing a memset -1 just looks a bit hackish to be, but I realize that it is personal preference.

If it is super important to save 8 bytes from the heap, then I would even prefer changing the MAX_IATU_IN 256 to something smaller, like 127 or whatever, just as long as we don't need that extra memset -1 :)

Kind regards,
Niklas

author maintainer
#16 lore raw
Quoted 52 lines · Niklas Cassel
Quoted 8 lines · Frank Li
Quoted 4 lines

Niklas's initial suggestion of keeping u8 for the array and 0 as the unallocated placeholder sounds good to me. Please use that instead.

It is impossible to keep u8, because 255 + 1 will 0 for u8. Previously Niklas's initial suggestion have not consider this condition. If u8 have to change to u16 or s16 anyways, I prefer use -1 as free.

Well, to be fair, my suggestion was:
"If we continue to use a u8, and offset the saved value by one, we will at least be able to support 255-1 == 254 iATUs."

But we have this define:
drivers/pci/controller/dwc/pcie-designware.h:#define MAX_IATU_IN 256 (Even if it isn't used anywhere.)

But as ridiculous as it may seem to support that many inbound ranges, that is the max number of windows supported by the hardware, so why not just let the driver support the max supported by the hardware?

We are talking about:

struct dw_pcie_ep {
	...
        u8                      bar_to_atu[PCI_STD_NUM_BARS];
	...
}

where PCI_STD_NUM_BARS == 6.

And where struct dw_pcie_ep is kzalloced for what I assume is all drivers.

So I'm actually for your idea of changing it to u16, or even unsigned int.

If the code is simplified if we use a u16 or unsigned int (because we don't need any weird if (val < MAX_IATU_IN - 1) check), then I'm all for it.

What I personally don't like with your patch in $subject,
was that you changed both dw_pcie_ep_clear_bar() to set the "clear value" to -1, but you also need a
memset(ep->bar_to_atu, -1, sizeof(ep->bar_to_atu)); in dw_pcie_ep_init().

I much prefer to have 0 as the default/unused value, because it will automatically get set when you do kzalloc().
Seeing a memset -1 just looks a bit hackish to be, but I realize that it is personal preference.

Quoted 3 lines

If it is super important to save 8 bytes from the heap, then I would even prefer changing the MAX_IATU_IN 256 to something smaller, like 127 or whatever, just as long as we don't need that extra memset -1 :)

Okay, Let me work on '0' version.

Frank

Quoted 4 lines

Kind regards,
Niklas