Skip to contentSkip to conversation

pcie-xilinx: Support reset GPIO for PERST#

PATCH Superseded Mike Looijmans 2 participants 4 messages Apr 9, 2025 – Apr 10, 2025 1 message not in the archive

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.d67fd5a1-30a5-4d5a-a4ef-dde83ebcdd8d@emailsignatures365.codetwo.com

#1 lore raw

PATCH v3 1/2 pcie-xilinx: Wait for link-up status during initialization

When the driver loads, the transceiver may still be in the state of setting up a link. Wait for that to complete before continuing. This fixes that the PCIe core does not work when loading the PL bitstream from userspace. There's only milliseconds between the FPGA boot and the core initializing in that case, and the link won't be up yet. The design only worked when the FPGA was programmed in the bootloader, as that will give the system hundreds of milliseconds to boot.

As the PCIe spec allows up to 100 ms time to establish a link, we'll allow up to 200ms before giving up.

---

(no changes since v2)

Changes in v2:
Split into "reset GPIO" and "wait for link" patches Add timeout explanation

 drivers/pci/controller/pcie-xilinx.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/pci/controller/pcie-xilinx.c +15 -1
index 0b534f73a942..2e59b91f43e0 100644
@@ -15,6 +15,7 @@
 #include <linux/irqdomain.h>
 #include <linux/kernel.h>
 #include <linux/init.h>
+#include <linux/iopoll.h>
 #include <linux/msi.h>
 #include <linux/of_address.h>
 #include <linux/of_pci.h>
@@ -126,6 +127,19 @@ static inline bool xilinx_pcie_link_up(struct xilinx_pcie *pcie)
 		XILINX_PCIE_REG_PSCR_LNKUP) ? 1 : 0;
 }
 
+static int xilinx_pci_wait_link_up(struct xilinx_pcie *pcie)
+{
+	u32 val;
+
+	/*
+	 * PCIe r6.0, sec 6.6.1 provides 100ms timeout. Since this is FPGA
+	 * fabric, we're more lenient and allow 200 ms for link training.
+	 */
+	return readl_poll_timeout(pcie->reg_base + XILINX_PCIE_REG_PSCR, val,
+			(val & XILINX_PCIE_REG_PSCR_LNKUP), 2 * USEC_PER_MSEC,
+			200 * USEC_PER_MSEC);
+}
+
 /**
  * xilinx_pcie_clear_err_interrupts - Clear Error Interrupts
  * @pcie: PCIe port information
@@ -493,7 +507,7 @@ static void xilinx_pcie_init_port(struct xilinx_pcie *pcie)
 {
 	struct device *dev = pcie->dev;
 
-	if (xilinx_pcie_link_up(pcie))
+	if (!xilinx_pci_wait_link_up(pcie))
 		dev_info(dev, "PCIe Link is UP\n");
 	else
 		dev_info(dev, "PCIe Link is DOWN\n");
Signature · 18 lines
2.43.0


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: [email protected]
W: www.topic.nl

Please consider the environment before printing this e-mail
#2 lore raw

PATCH v3 2/2 pcie-xilinx: Support reset GPIO for PERST#

Support providing the PERST# reset signal through a devicetree binding. Thus the system no longer relies on external components to perform the bus reset.

---

Changes in v3:
Include linux/gpio/consumer.h
Message "Failed to request reset GPIO"
Use PCIE_T_PVPERL_MS and PCIE_T_RRS_READY_MS
Dropped dt-bindings patch, not needed

Changes in v2:
Split into "reset GPIO" and "wait for link" patches Handle GPIO defer and/or errors

 drivers/pci/controller/pcie-xilinx.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/pci/controller/pcie-xilinx.c +14 -0
index 2e59b91f43e0..8e8ca9891b36 100644
@@ -10,6 +10,7 @@
  * ARM PCI Host generic driver.
  */
 
+#include <linux/gpio/consumer.h>
 #include <linux/interrupt.h>
 #include <linux/irq.h>
 #include <linux/irqdomain.h>
@@ -577,11 +578,17 @@ static int xilinx_pcie_probe(struct platform_device *pdev)
 	struct device *dev = &pdev->dev;
 	struct xilinx_pcie *pcie;
 	struct pci_host_bridge *bridge;
+	struct gpio_desc *perst_gpio;
 	int err;
 
 	if (!dev->of_node)
 		return -ENODEV;
 
+	perst_gpio = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_HIGH);
+	if (IS_ERR(perst_gpio))
+		return dev_err_probe(dev, PTR_ERR(perst_gpio),
+				     "Failed to request reset GPIO\n");
+
 	bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
 	if (!bridge)
 		return -ENODEV;
@@ -596,6 +603,13 @@ static int xilinx_pcie_probe(struct platform_device *pdev)
 		return err;
 	}
 
+	if (perst_gpio) {
+		msleep(PCIE_T_PVPERL_MS); /* Minimum assertion time */
+		gpiod_set_value_cansleep(perst_gpio, 0);
+		/* Initial delay to provide endpoint time to initialize */
+		msleep(PCIE_T_RRS_READY_MS);
+	}
+
 	xilinx_pcie_init_port(pcie);
 
 	err = xilinx_pcie_init_irq_domain(pcie);
Signature · 18 lines
2.43.0


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: [email protected]
W: www.topic.nl

Please consider the environment before printing this e-mail
#3 lore raw

Re: [PATCH v3 1/2] pcie-xilinx: Wait for link-up status during initialization

In reply to Mike Looijmans

Please make the subject line match previous changes to this driver. See "git log --oneline drivers/pci/controller/pcie-xilinx.c"

Quoted 10 lines · Mike Looijmans

When the driver loads, the transceiver may still be in the state of setting up a link. Wait for that to complete before continuing. This fixes that the PCIe core does not work when loading the PL bitstream from userspace. There's only milliseconds between the FPGA boot and the core initializing in that case, and the link won't be up yet. The design only worked when the FPGA was programmed in the bootloader, as that will give the system hundreds of milliseconds to boot.

As the PCIe spec allows up to 100 ms time to establish a link, we'll allow up to 200ms before giving up.

This sounds like there's still a race between userspace loading the PL bitstream and the driver waiting for link up, but we're just waiting longer in the kernel so userspace has more chance of winning the race. Is that true?

Quoted 5 lines
drivers/pci/controller/pcie-xilinx.c +1 -0
@@ -126,6 +127,19 @@ static inline bool xilinx_pcie_link_up(struct xilinx_pcie *pcie)
 		XILINX_PCIE_REG_PSCR_LNKUP) ? 1 : 0;
 }
 
+static int xilinx_pci_wait_link_up(struct xilinx_pcie *pcie)

drivers/pci/controller/pcie-xilinx.c static inline bool xilinx_pcie_link_up(struct xilinx_pcie *pcie) · line 140

Quoted 10 lines
drivers/pci/controller/pcie-xilinx.c +10 -0
+{
+	u32 val;
+
+	/*
+	 * PCIe r6.0, sec 6.6.1 provides 100ms timeout. Since this is FPGA
+	 * fabric, we're more lenient and allow 200 ms for link training.
+	 */
+	return readl_poll_timeout(pcie->reg_base + XILINX_PCIE_REG_PSCR, val,
+			(val & XILINX_PCIE_REG_PSCR_LNKUP), 2 * USEC_PER_MSEC,
+			200 * USEC_PER_MSEC);

There should be a #define in drivers/pci/pci.h for this 100ms value that you can use here to connect this more closely with the spec.

Maybe there's a way to use read_poll_timeout(), readx_poll_timeout(), or something similar so we can use xilinx_pcie_link_up() directly instead of reimplementing it here?

Quoted 14 lines
drivers/pci/controller/pcie-xilinx.c +3 -1
+}
+
 /**
  * xilinx_pcie_clear_err_interrupts - Clear Error Interrupts
  * @pcie: PCIe port information
@@ -493,7 +507,7 @@ static void xilinx_pcie_init_port(struct xilinx_pcie *pcie)
 {
 	struct device *dev = pcie->dev;
 
-	if (xilinx_pcie_link_up(pcie))
+	if (!xilinx_pci_wait_link_up(pcie))
 		dev_info(dev, "PCIe Link is UP\n");
 	else
 		dev_info(dev, "PCIe Link is DOWN\n");
#4 lore raw

Re: [PATCH v3 1/2] pcie-xilinx: Wait for link-up status during initialization

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: [email protected]
W: www.topic.nl

Please consider the environment before printing this e-mail

Quoted 12 lines · Bjorn Helgaas

Please make the subject line match previous changes to this driver. See "git log --oneline drivers/pci/controller/pcie-xilinx.c"

Quoted 8 lines · Mike Looijmans

When the driver loads, the transceiver may still be in the state of setting up a link. Wait for that to complete before continuing. This fixes that the PCIe core does not work when loading the PL bitstream from userspace. There's only milliseconds between the FPGA boot and the core initializing in that case, and the link won't be up yet. The design only worked when the FPGA was programmed in the bootloader, as that will give the system hundreds of milliseconds to boot.

Quoted 6 lines
Quoted 2 lines

As the PCIe spec allows up to 100 ms time to establish a link, we'll allow up to 200ms before giving up.

This sounds like there's still a race between userspace loading the PL bitstream and the driver waiting for link up, but we're just waiting longer in the kernel so userspace has more chance of winning the race. Is that true?

No, that's not the case here. The PCIe (host) core is what is in the PL bitstream. Devicetree overlay and FPGA support take care of that part, so the PL is programmed and the PCIe core is available when this driver probes.

The issue is with the endpoint on the other side of the PL, most likely an NVME drive, WiFi card or PCIe switch (nothing inside the FPGA, the purpose of the PCIe core here is to communicate with something external over PCIe).

The endpoint gets reset by what amounts to black magic (external circuit, something in the PL, or just nothing at all). This driver assumes that that has already happened, and immediately starts training. This works on Xilinx' reference designs that program the PL in their proprietary bootloader. If the PL was programmed from within Linux, this driver will probe within milliseconds after programming the PL, and in that case, the link won't be up yet.

The second patch in this series adds a PERST# GPIO support, so that the "black magic" can also be replaced with a proper reset.

Quoted 5 lines
Quoted 5 lines
drivers/pci/controller/pcie-xilinx.c +0 -0
@@ -126,6 +127,19 @@ static inline bool xilinx_pcie_link_up(struct xilinx_pcie *pcie)
  		XILINX_PCIE_REG_PSCR_LNKUP) ? 1 : 0;
  }
 
Quoted 13 lines

drivers/pci/controller/pcie-xilinx.c static inline bool xilinx_pcie_link_up(struct xilinx_pcie *pcie) · line 140

Quoted 11 lines
drivers/pci/controller/pcie-xilinx.c +11 -0
+static int xilinx_pci_wait_link_up(struct xilinx_pcie *pcie)
+{
+	u32 val;
+
+	/*
+	 * PCIe r6.0, sec 6.6.1 provides 100ms timeout. Since this is FPGA
+	 * fabric, we're more lenient and allow 200 ms for link training.
+	 */
+	return readl_poll_timeout(pcie->reg_base + XILINX_PCIE_REG_PSCR, val,
+			(val & XILINX_PCIE_REG_PSCR_LNKUP), 2 * USEC_PER_MSEC,
+			200 * USEC_PER_MSEC);

There should be a #define in drivers/pci/pci.h for this 100ms value that you can use here to connect this more closely with the spec.

That'd be "PCIE_T_RRS_READY_MS". Experience learns that adhering too closely to this spec is a good way to make your system fail though, so most host controllers are more lenient, e.g. rockchip uses 500ms.

Quoted 4 lines

Maybe there's a way to use read_poll_timeout(), readx_poll_timeout(), or something similar so we can use xilinx_pcie_link_up() directly instead of reimplementing it here?

Other way around would be easier, just call this again when it wants to know if the link is up, maybe with a 0 timeout (which allows the compiler to remove the loop).

Quoted 15 lines
Quoted 15 lines
drivers/pci/controller/pcie-xilinx.c +3 -1
+}
+
  /**
   * xilinx_pcie_clear_err_interrupts - Clear Error Interrupts
   * @pcie: PCIe port information
@@ -493,7 +507,7 @@ static void xilinx_pcie_init_port(struct xilinx_pcie *pcie)
  {
  	struct device *dev = pcie->dev;
 
-	if (xilinx_pcie_link_up(pcie))
+	if (!xilinx_pci_wait_link_up(pcie))
  		dev_info(dev, "PCIe Link is UP\n");
  	else
  		dev_info(dev, "PCIe Link is DOWN\n");