Skip to contentSkip to conversation

PCI: dwc: Make Link Up IRQ logic handle already powered on PCIe switches

PATCH In review Niklas Cassel 4 participants 12 messages Dec 1, 2025 – Dec 22, 2025

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 PCI: dwc: Make Link Up IRQ logic handle already powered on PCIe switches

The DWC glue drivers always call pci_host_probe() during probe(), which will allocate upstream bridge resources and enumerate the bus.

For controllers without Link Up IRQ support, pci_host_probe() is called after dw_pcie_wait_for_link(), which will also wait the time required by the PCIe specification before performing PCI Configuration Space reads.

For controllers with Link Up IRQ support, the pci_host_probe() call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.

For controllers with Link Up IRQ support, since the pci_host_probe() call is done without any delay (link training might still be ongoing), it is very unlikely that this scan will find any devices. Once the Link Up IRQ triggers, the Link Up IRQ handler will call pci_rescan_bus().

This works fine for PCIe endpoints connected to the Root Port, since they don't extend the bus. However, if the pci_rescan_bus() call detects a PCIe switch, then there will be a problem when the downstream busses starts showing up, because the PCIe controller is not hotplug capable, so we are not allowed to extend the subordinate bus number after the initial scan, resulting in error messages such as:

pci_bus 0004:43: busn_res: can not insert [bus 43-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:43: busn_res: [bus 43-41] end is updated to 43
pci_bus 0004:43: busn_res: can not insert [bus 43] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:00.0: devices behind bridge are unusable because [bus 43] cannot be assigned for them pci_bus 0004:44: busn_res: can not insert [bus 44-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:44: busn_res: [bus 44-41] end is updated to 44
pci_bus 0004:44: busn_res: can not insert [bus 44] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:02.0: devices behind bridge are unusable because [bus 44] cannot be assigned for them pci_bus 0004:45: busn_res: can not insert [bus 45-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:45: busn_res: [bus 45-41] end is updated to 45
pci_bus 0004:45: busn_res: can not insert [bus 45] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:06.0: devices behind bridge are unusable because [bus 45] cannot be assigned for them pci_bus 0004:46: busn_res: can not insert [bus 46-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:46: busn_res: [bus 46-41] end is updated to 46
pci_bus 0004:46: busn_res: can not insert [bus 46] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:0e.0: devices behind bridge are unusable because [bus 46] cannot be assigned for them pci_bus 0004:42: busn_res: [bus 42-41] end is updated to 46
pci_bus 0004:42: busn_res: can not insert [bus 42-46] under [bus 41] (conflicts with (null) [bus 41]) pci 0004:41:00.0: devices behind bridge are unusable because [bus 42-46] cannot be assigned for them pcieport 0004:40:00.0: bridge has subordinate 41 but max busn 46

While we would like to set the is_hotplug_bridge flag (quirk_hotplug_bridge()), many embedded SoCs that use the DWC controller have synthesized the controller without hot-plug support. Thus, the Link Up IRQ logic is only mimicking hot-plug functionality, i.e. it is not compliant with the PCI Hot-Plug Specification, so we cannot make use of the is_hotplug_bridge flag.

In order to let the Link Up IRQ logic handle PCIe switches that are already powered on (PCIe switches that not powered on already need to implement a pwrctrl driver), don't perform a pci_host_probe() call during probe() (which disregards the delays required by the PCIe specification).

Instead let the first Link Up IRQ call pci_host_probe(). Any follow up Link Up IRQ will call pci_rescan_bus().

The IRQ name in /proc/interrupts for the pcie-qcom driver is renamed in order to not dereference pp->bridge->bus before it has been assigned.

Fixes ec9fd499b9c6 ("PCI: dw-rockchip: Don't wait for link since we can detect Link Up")
Fixes 0e0b45ab5d77 ("PCI: dw-rockchip: Enumerate endpoints based on dll_link_up IRQ")

---
Changes since v1:
-Rename the IRQ on pcie-qcom to not depend on pp->bridge->bus (Mani) -Make sure that ret is initialized in dw_pcie_host_init() error path (Dan)

 .../pci/controller/dwc/pcie-designware-host.c | 71 ++++++++++++++++---
 drivers/pci/controller/dwc/pcie-designware.h  |  5 ++
 drivers/pci/controller/dwc/pcie-dw-rockchip.c |  5 +-
 drivers/pci/controller/dwc/pcie-qcom.c        |  9 +--
 4 files changed, 71 insertions(+), 19 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +62 -9
index e92513c5bda51..bed7b309f6d9e 100644
@@ -565,6 +565,59 @@ static int dw_pcie_host_get_resources(struct dw_pcie_rp *pp)
 	return 0;
 }
 
+static int dw_pcie_host_initial_scan(struct dw_pcie_rp *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct pci_host_bridge *bridge = pp->bridge;
+	int ret;
+
+	ret = pci_host_probe(bridge);
+	if (ret)
+		return ret;
+
+	if (pp->ops->post_init)
+		pp->ops->post_init(pp);
+
+	dwc_pcie_debugfs_init(pci, DW_PCIE_RC_TYPE);
+
+	return 0;
+}
+
+void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp)
+{
+	if (!pp->initial_linkup_irq_done) {
+		int ret;
+
+		ret = dw_pcie_host_initial_scan(pp);
+		if (ret) {
+			struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+			struct device *dev = pci->dev;
+
+			dev_err(dev, "Initial scan from IRQ failed: %d\n", ret);
+
+			dw_pcie_stop_link(pci);
+
+			dw_pcie_edma_remove(pci);
+
+			if (pp->has_msi_ctrl)
+				dw_pcie_free_msi(pp);
+
+			if (pp->ops->deinit)
+				pp->ops->deinit(pp);
+
+			if (pp->cfg)
+				pci_ecam_free(pp->cfg);
+		} else {
+			pp->initial_linkup_irq_done = true;
+		}
+	} else {
+		/* Rescan the bus to enumerate endpoint devices */
+		pci_lock_rescan_remove();
+		pci_rescan_bus(pp->bridge->bus);
+		pci_unlock_rescan_remove();
+	}
+}
+
 int dw_pcie_host_init(struct dw_pcie_rp *pp)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
@@ -669,18 +722,18 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
 	 * If there is no Link Up IRQ, we should not bypass the delay
 	 * because that would require users to manually rescan for devices.
 	 */
-	if (!pp->use_linkup_irq)
+	if (!pp->use_linkup_irq) {
 		/* Ignore errors, the link may come up later */
 		dw_pcie_wait_for_link(pci);
 
-	ret = pci_host_probe(bridge);
-	if (ret)
-		goto err_stop_link;
-
-	if (pp->ops->post_init)
-		pp->ops->post_init(pp);
-
-	dwc_pcie_debugfs_init(pci, DW_PCIE_RC_TYPE);
+		/*
+		 * For platforms with Link Up IRQ, initial scan will be done
+		 * on first Link Up IRQ.
+		 */
+		ret = dw_pcie_host_initial_scan(pp);
+		if (ret)
+			goto err_stop_link;
+	}
 
 	return 0;
 
drivers/pci/controller/dwc/pcie-designware.h +5 -0
index e995f692a1ecd..a31bd93490dcd 100644
@@ -427,6 +427,7 @@ struct dw_pcie_rp {
 	int			msg_atu_index;
 	struct resource		*msg_res;
 	bool			use_linkup_irq;
+	bool			initial_linkup_irq_done;
 	struct pci_eq_presets	presets;
 	struct pci_config_window *cfg;
 	bool			ecam_enabled;
@@ -807,6 +808,7 @@ void dw_pcie_msi_init(struct dw_pcie_rp *pp);
 int dw_pcie_msi_host_init(struct dw_pcie_rp *pp);
 void dw_pcie_free_msi(struct dw_pcie_rp *pp);
 int dw_pcie_setup_rc(struct dw_pcie_rp *pp);
+void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp);
 int dw_pcie_host_init(struct dw_pcie_rp *pp);
 void dw_pcie_host_deinit(struct dw_pcie_rp *pp);
 int dw_pcie_allocate_domains(struct dw_pcie_rp *pp);
@@ -844,6 +846,9 @@ static inline int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 	return 0;
 }
 
+static inline void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp)
+{ }
+
 static inline int dw_pcie_host_init(struct dw_pcie_rp *pp)
 {
 	return 0;
drivers/pci/controller/dwc/pcie-dw-rockchip.c +1 -4
index 3e2752c7dd096..8f2cc1ef25e3d 100644
@@ -466,10 +466,7 @@ static irqreturn_t rockchip_pcie_rc_sys_irq_thread(int irq, void *arg)
 		if (rockchip_pcie_link_up(pci)) {
 			msleep(PCIE_RESET_CONFIG_WAIT_MS);
 			dev_dbg(dev, "Received Link up event. Starting enumeration!\n");
-			/* Rescan the bus to enumerate endpoint devices */
-			pci_lock_rescan_remove();
-			pci_rescan_bus(pp->bridge->bus);
-			pci_unlock_rescan_remove();
+			dw_pcie_handle_link_up_irq(pp);
 		}
 	}
 
drivers/pci/controller/dwc/pcie-qcom.c +3 -6
index c48a20602d7fa..04f29cd8d8881 100644
@@ -1617,10 +1617,7 @@ static irqreturn_t qcom_pcie_global_irq_thread(int irq, void *data)
 	if (FIELD_GET(PARF_INT_ALL_LINK_UP, status)) {
 		msleep(PCIE_RESET_CONFIG_WAIT_MS);
 		dev_dbg(dev, "Received Link up event. Starting enumeration!\n");
-		/* Rescan the bus to enumerate endpoint devices */
-		pci_lock_rescan_remove();
-		pci_rescan_bus(pp->bridge->bus);
-		pci_unlock_rescan_remove();
+		dw_pcie_handle_link_up_irq(pp);
 
 		qcom_pcie_icc_opp_update(pcie);
 	} else {
@@ -1937,8 +1934,8 @@ static int qcom_pcie_probe(struct platform_device *pdev)
 		goto err_phy_exit;
 	}
 
-	name = devm_kasprintf(dev, GFP_KERNEL, "qcom_pcie_global_irq%d",
-			      pci_domain_nr(pp->bridge->bus));
+	name = devm_kasprintf(dev, GFP_KERNEL, "qcom_pcie_global_irq_%pOFP",
+			      dev->of_node);
 	if (!name) {
 		ret = -ENOMEM;
 		goto err_host_deinit;
Signature · 1 line
2.52.0
#2 lore raw
In reply to Niklas Cassel

On Mon, Dec 01, 2025 at 07:36:35AM +0100, Niklas Cassel wrote:

Mani, could you perhaps try this series on Qcom?

I personally prefer this series over a revert because:
It allows to us to still use a PCIe endpoint that comes up after the main system has booted, e.g. when the PCIe endpoint is a board running pci-epf-test, without the need for a manual rescan on be bus.

If we go with revert instead, this very nice feature would be gone, and the user would need to do a manual rescan of the bus.

One could even argue that that is a user visible regression.

Kind regards,
Niklas

#3 lore raw
In reply to Niklas Cassel
Quoted 156 lines · Niklas Cassel

The DWC glue drivers always call pci_host_probe() during probe(), which will allocate upstream bridge resources and enumerate the bus.

For controllers without Link Up IRQ support, pci_host_probe() is called after dw_pcie_wait_for_link(), which will also wait the time required by the PCIe specification before performing PCI Configuration Space reads.

For controllers with Link Up IRQ support, the pci_host_probe() call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.

For controllers with Link Up IRQ support, since the pci_host_probe() call is done without any delay (link training might still be ongoing), it is very unlikely that this scan will find any devices. Once the Link Up IRQ triggers, the Link Up IRQ handler will call pci_rescan_bus().

This works fine for PCIe endpoints connected to the Root Port, since they don't extend the bus. However, if the pci_rescan_bus() call detects a PCIe switch, then there will be a problem when the downstream busses starts showing up, because the PCIe controller is not hotplug capable, so we are not allowed to extend the subordinate bus number after the initial scan, resulting in error messages such as:

pci_bus 0004:43: busn_res: can not insert [bus 43-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:43: busn_res: [bus 43-41] end is updated to 43
pci_bus 0004:43: busn_res: can not insert [bus 43] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:00.0: devices behind bridge are unusable because [bus 43] cannot be assigned for them pci_bus 0004:44: busn_res: can not insert [bus 44-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:44: busn_res: [bus 44-41] end is updated to 44
pci_bus 0004:44: busn_res: can not insert [bus 44] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:02.0: devices behind bridge are unusable because [bus 44] cannot be assigned for them pci_bus 0004:45: busn_res: can not insert [bus 45-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:45: busn_res: [bus 45-41] end is updated to 45
pci_bus 0004:45: busn_res: can not insert [bus 45] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:06.0: devices behind bridge are unusable because [bus 45] cannot be assigned for them pci_bus 0004:46: busn_res: can not insert [bus 46-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:46: busn_res: [bus 46-41] end is updated to 46
pci_bus 0004:46: busn_res: can not insert [bus 46] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:0e.0: devices behind bridge are unusable because [bus 46] cannot be assigned for them pci_bus 0004:42: busn_res: [bus 42-41] end is updated to 46
pci_bus 0004:42: busn_res: can not insert [bus 42-46] under [bus 41] (conflicts with (null) [bus 41]) pci 0004:41:00.0: devices behind bridge are unusable because [bus 42-46] cannot be assigned for them pcieport 0004:40:00.0: bridge has subordinate 41 but max busn 46

While we would like to set the is_hotplug_bridge flag (quirk_hotplug_bridge()), many embedded SoCs that use the DWC controller have synthesized the controller without hot-plug support. Thus, the Link Up IRQ logic is only mimicking hot-plug functionality, i.e. it is not compliant with the PCI Hot-Plug Specification, so we cannot make use of the is_hotplug_bridge flag.

In order to let the Link Up IRQ logic handle PCIe switches that are already powered on (PCIe switches that not powered on already need to implement a pwrctrl driver), don't perform a pci_host_probe() call during probe() (which disregards the delays required by the PCIe specification).

Instead let the first Link Up IRQ call pci_host_probe(). Any follow up Link Up IRQ will call pci_rescan_bus().

The IRQ name in /proc/interrupts for the pcie-qcom driver is renamed in order to not dereference pp->bridge->bus before it has been assigned.

Fixes ec9fd499b9c6 ("PCI: dw-rockchip: Don't wait for link since we can detect Link Up")
Fixes 0e0b45ab5d77 ("PCI: dw-rockchip: Enumerate endpoints based on dll_link_up IRQ")

---
Changes since v1:

(quoted diff) +0 -2
-Rename the IRQ on pcie-qcom to not depend on pp->bridge->bus (Mani)
-Make sure that ret is initialized in dw_pcie_host_init() error path (Dan)
 
  .../pci/controller/dwc/pcie-designware-host.c | 71 ++++++++++++++++---
  drivers/pci/controller/dwc/pcie-designware.h  |  5 ++
  drivers/pci/controller/dwc/pcie-dw-rockchip.c |  5 +-
  drivers/pci/controller/dwc/pcie-qcom.c        |  9 +--
  4 files changed, 71 insertions(+), 19 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +54 -6
index e92513c5bda51..bed7b309f6d9e 100644
@@ -565,6 +565,59 @@ static int dw_pcie_host_get_resources(struct dw_pcie_rp *pp)
  	return 0;
  }
 
+static int dw_pcie_host_initial_scan(struct dw_pcie_rp *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct pci_host_bridge *bridge = pp->bridge;
+	int ret;
+
+	ret = pci_host_probe(bridge);
+	if (ret)
+		return ret;
+
+	if (pp->ops->post_init)
+		pp->ops->post_init(pp);
+
+	dwc_pcie_debugfs_init(pci, DW_PCIE_RC_TYPE);
+
+	return 0;
+}
+
+void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp)
+{
+	if (!pp->initial_linkup_irq_done) {
+		int ret;
+
+		ret = dw_pcie_host_initial_scan(pp);
+		if (ret) {
+			struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+			struct device *dev = pci->dev;
+
+			dev_err(dev, "Initial scan from IRQ failed: %d\n", ret);
+
+			dw_pcie_stop_link(pci);
+
+			dw_pcie_edma_remove(pci);
+
+			if (pp->has_msi_ctrl)
+				dw_pcie_free_msi(pp);
+
+			if (pp->ops->deinit)
+				pp->ops->deinit(pp);
+
+			if (pp->cfg)
+				pci_ecam_free(pp->cfg);
+		} else {
+			pp->initial_linkup_irq_done = true;
+		}
+	} else {
+		/* Rescan the bus to enumerate endpoint devices */
+		pci_lock_rescan_remove();
+		pci_rescan_bus(pp->bridge->bus);
+		pci_unlock_rescan_remove();
+	}
+}
+
  int dw_pcie_host_init(struct dw_pcie_rp *pp)
  {
  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
@@ -669,18 +722,18 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
  	 * If there is no Link Up IRQ, we should not bypass the delay
  	 * because that would require users to manually rescan for devices.
  	 */
-	if (!pp->use_linkup_irq)
+	if (!pp->use_linkup_irq) {
  		/* Ignore errors, the link may come up later */
  		dw_pcie_wait_for_link(pci);
 
-	ret = pci_host_probe(bridge);
-	if (ret)
-		goto err_stop_link;
-
-	if (pp->ops->post_init)

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct dw_pcie_rp *pp) · line 735

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +7 -3
-		pp->ops->post_init(pp);
-
-	dwc_pcie_debugfs_init(pci, DW_PCIE_RC_TYPE);
+		/*
+		 * For platforms with Link Up IRQ, initial scan will be done
+		 * on first Link Up IRQ.
+		 */
+		ret = dw_pcie_host_initial_scan(pp);
+		if (ret)
+			goto err_stop_link;

This will cause breakages in qcom platforms, qcom platforms use pwrctrl framework for enabling power to the endpoints. And the pwrctrl drivers will be created as part of the initial scan. As we skipping the initial scan here, pwrctrl driver will be never created and the power to the endpoint will be never enabled. This will cause link to be never up.

- Krishna Chaitanya.

Quoted 77 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1 -0
+	}
 
  	return 0;
 
drivers/pci/controller/dwc/pcie-designware.h +5 -0
index e995f692a1ecd..a31bd93490dcd 100644
@@ -427,6 +427,7 @@ struct dw_pcie_rp {
  	int			msg_atu_index;
  	struct resource		*msg_res;
  	bool			use_linkup_irq;
+	bool			initial_linkup_irq_done;
  	struct pci_eq_presets	presets;
  	struct pci_config_window *cfg;
  	bool			ecam_enabled;
@@ -807,6 +808,7 @@ void dw_pcie_msi_init(struct dw_pcie_rp *pp);
  int dw_pcie_msi_host_init(struct dw_pcie_rp *pp);
  void dw_pcie_free_msi(struct dw_pcie_rp *pp);
  int dw_pcie_setup_rc(struct dw_pcie_rp *pp);
+void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp);
  int dw_pcie_host_init(struct dw_pcie_rp *pp);
  void dw_pcie_host_deinit(struct dw_pcie_rp *pp);
  int dw_pcie_allocate_domains(struct dw_pcie_rp *pp);
@@ -844,6 +846,9 @@ static inline int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
  	return 0;
  }
 
+static inline void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp)
+{ }
+
  static inline int dw_pcie_host_init(struct dw_pcie_rp *pp)
  {
  	return 0;
drivers/pci/controller/dwc/pcie-dw-rockchip.c +1 -4
index 3e2752c7dd096..8f2cc1ef25e3d 100644
@@ -466,10 +466,7 @@ static irqreturn_t rockchip_pcie_rc_sys_irq_thread(int irq, void *arg)
  		if (rockchip_pcie_link_up(pci)) {
  			msleep(PCIE_RESET_CONFIG_WAIT_MS);
  			dev_dbg(dev, "Received Link up event. Starting enumeration!\n");
-			/* Rescan the bus to enumerate endpoint devices */
-			pci_lock_rescan_remove();
-			pci_rescan_bus(pp->bridge->bus);
-			pci_unlock_rescan_remove();
+			dw_pcie_handle_link_up_irq(pp);
  		}
  	}
 
drivers/pci/controller/dwc/pcie-qcom.c +3 -6
index c48a20602d7fa..04f29cd8d8881 100644
@@ -1617,10 +1617,7 @@ static irqreturn_t qcom_pcie_global_irq_thread(int irq, void *data)
  	if (FIELD_GET(PARF_INT_ALL_LINK_UP, status)) {
  		msleep(PCIE_RESET_CONFIG_WAIT_MS);
  		dev_dbg(dev, "Received Link up event. Starting enumeration!\n");
-		/* Rescan the bus to enumerate endpoint devices */
-		pci_lock_rescan_remove();
-		pci_rescan_bus(pp->bridge->bus);
-		pci_unlock_rescan_remove();
+		dw_pcie_handle_link_up_irq(pp);
 
  		qcom_pcie_icc_opp_update(pcie);
  	} else {
@@ -1937,8 +1934,8 @@ static int qcom_pcie_probe(struct platform_device *pdev)
  		goto err_phy_exit;
  	}
 
-	name = devm_kasprintf(dev, GFP_KERNEL, "qcom_pcie_global_irq%d",
-			      pci_domain_nr(pp->bridge->bus));
+	name = devm_kasprintf(dev, GFP_KERNEL, "qcom_pcie_global_irq_%pOFP",
+			      dev->of_node);
  	if (!name) {
  		ret = -ENOMEM;
  		goto err_host_deinit;
#4 lore raw

Hello Krishna,

Currently:
For controllers with Link Up IRQ support, the pci_host_probe() call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.

This seems quite bad.

A device might not be fully initialized during during the time of these PCI Configuration Space reads, but might still return some bogus values that are actually different from the Configuration Space reads if done after respecting the delays mandated by the PCIe specification.

I think the options are:
1) Keep the pci_host_probe() call in dw_pcie_host_init() for controllers with Link Up IRQ support, but make sure that we respect the delays also in this case.
or
2) Remove the pci_host_probe() call from dw_pcie_host_init(), and make sure that pci_host_probe() is done by the first Link Up IRQ
(i.e. what this patch does).

One big thing with using the Link Up IRQ is to not do any delay during PCIe controller driver's probe(), which reduces startup time, exactly as your commit message in commit 36971d6c5a9a ("PCI: qcom: Don't wait for link if we can detect Link Up") explains.
Therefore, I don't think that 1) is a good solution, so that leaves us with 2).

If pwrctrl drivers are created as part of the pci_host_probe() call, I think that perhaps an alternative would be to create an explict pwrctrl_init() function, and let the PCI controller drivers that actually use pwrctrl call that from their probe().
(And just remove the same from pci_host_probe() ?)

In fact, looking at your suggested patches (that hasn't landed yet):
[PATCH 3/5] PCI/pwrctrl: Add APIs for explicitly creating and destroying pwrctrl devices [PATCH 5/5] PCI/pwrctrl: Switch to the new pwrctrl APIs

https://lore.kernel.org/linux-pci/[email protected]/
https://lore.kernel.org/linux-pci/[email protected]/

Seem to do exactly that:
Call pci_pwrctrl_create_devices() explicitly from the PCIe controller drivers directly, and removes the pci_pwrctrl_create_device() call from pci_host_probe().

So I don't really understand your concern with this series, at least not if it goes on top of your series:
https://lore.kernel.org/linux-pci/[email protected]/

Kind regards,
Niklas

#5 lore raw
Quoted 45 lines · Niklas Cassel

Hello Krishna,

Currently:
For controllers with Link Up IRQ support, the pci_host_probe() call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.

This seems quite bad.

A device might not be fully initialized during during the time of these PCI Configuration Space reads, but might still return some bogus values that are actually different from the Configuration Space reads if done after respecting the delays mandated by the PCIe specification.

I think the options are:
1) Keep the pci_host_probe() call in dw_pcie_host_init() for controllers with Link Up IRQ support, but make sure that we respect the delays also in this case.
or
2) Remove the pci_host_probe() call from dw_pcie_host_init(), and make sure that pci_host_probe() is done by the first Link Up IRQ (i.e. what this patch does).

One big thing with using the Link Up IRQ is to not do any delay during PCIe controller driver's probe(), which reduces startup time, exactly as your commit message in commit 36971d6c5a9a ("PCI: qcom: Don't wait for link if we can detect Link Up") explains.
Therefore, I don't think that 1) is a good solution, so that leaves us with 2).

If pwrctrl drivers are created as part of the pci_host_probe() call, I think that perhaps an alternative would be to create an explict pwrctrl_init() function, and let the PCI controller drivers that actually use pwrctrl call that from their probe().
(And just remove the same from pci_host_probe() ?)

In fact, looking at your suggested patches (that hasn't landed yet):
[PATCH 3/5] PCI/pwrctrl: Add APIs for explicitly creating and destroying pwrctrl devices [PATCH 5/5] PCI/pwrctrl: Switch to the new pwrctrl APIs

https://lore.kernel.org/linux-pci/[email protected]/
https://lore.kernel.org/linux-pci/[email protected]/

Quoted 7 lines

Seem to do exactly that:
Call pci_pwrctrl_create_devices() explicitly from the PCIe controller drivers directly, and removes the pci_pwrctrl_create_device() call from pci_host_probe().

So I don't really understand your concern with this series, at least not if it goes on top of your series:
https://lore.kernel.org/linux-pci/[email protected]/

Hi Niklas,

If this series goes on top of the our series i.e pwrctrl rework series, I don't have any concerns. My only concern is link up IRQ never fires if this patch goes before series.

- Krishna Chaitanya.

Quoted 4 lines

Kind regards,
Niklas

#6 lore raw
Quoted 5 lines · Krishna Chaitanya Chundru
Quoted 4 lines · Niklas Cassel

So I don't really understand your concern with this series, at least not if it goes on top of your series:
https://lore.kernel.org/linux-pci/[email protected]/

Quoted 7 lines

Hi Niklas,

If this series goes on top of the our series i.e pwrctrl rework series, I don't have any concerns.
My only concern is link up IRQ never fires if this patch goes before series.

- Krishna Chaitanya.

This patch missed the v6.19 merge window (and so did the pwrctrl rework series), so as long as Mani queues up both for v6.20 (with the pwrctrl rework series getting applied first), I think we are good.

Kind regards,
Niklas

#7 lore raw
Quoted 12 lines · Niklas Cassel
Quoted 11 lines · Krishna Chaitanya Chundru
Quoted 4 lines · Niklas Cassel

So I don't really understand your concern with this series, at least not if it goes on top of your series:
https://lore.kernel.org/linux-pci/[email protected]/

Hi Niklas,

If this series goes on top of the our series i.e pwrctrl rework series, I don't have any concerns.
My only concern is link up IRQ never fires if this patch goes before series.

Quoted 6 lines
Quoted 1 line

- Krishna Chaitanya.

This patch missed the v6.19 merge window (and so did the pwrctrl rework series), so as long as Mani queues up both for v6.20 (with the pwrctrl rework series getting applied first), I think we are good.

The plan is to merge pwrctrl series to v6.20 (unless we get strong objections), but once that happens, Qcom doesn't need this patch.

So it'd be good if you can just limit this patch to just Rockchip. Then once the Rockchip also moves to pwrctrl, we can revert this patch (also the whole IRQ based link up since the Root Port is not hotplug capable).

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#8 lore raw

Hello Mani,

Quoted 11 lines · Manivannan Sadhasivam
Quoted 4 lines

This patch missed the v6.19 merge window (and so did the pwrctrl rework series), so as long as Mani queues up both for v6.20 (with the pwrctrl rework series getting applied first), I think we are good.

The plan is to merge pwrctrl series to v6.20 (unless we get strong objections), but once that happens, Qcom doesn't need this patch.

So it'd be good if you can just limit this patch to just Rockchip. Then once the Rockchip also moves to pwrctrl, we can revert this patch (also the whole IRQ based link up since the Root Port is not hotplug capable).

The main reason why I did not like a revert, was because it would remove the nice feature where the bus is enumerated automatically on link up.

I assumed that the plan was to add Link Up IRQ support in pwrctrl in the future.

If that is not the case, and the plan is instead to eventually remove the existing Link Up IRQ support, then perhaps you could simply apply patches from:
https://lore.kernel.org/linux-pci/[email protected]/

Either the whole series or just patch 1 and 2, maintainer's choice :)

Kind regards,
Niklas

#9 lore raw
Quoted 15 lines · Niklas Cassel

Hello Mani,

Quoted 11 lines · Manivannan Sadhasivam
Quoted 4 lines

This patch missed the v6.19 merge window (and so did the pwrctrl rework series), so as long as Mani queues up both for v6.20 (with the pwrctrl rework series getting applied first), I think we are good.

The plan is to merge pwrctrl series to v6.20 (unless we get strong objections), but once that happens, Qcom doesn't need this patch.

So it'd be good if you can just limit this patch to just Rockchip. Then once the Rockchip also moves to pwrctrl, we can revert this patch (also the whole IRQ based link up since the Root Port is not hotplug capable).

Quoted 6 lines

The main reason why I did not like a revert, was because it would remove the nice feature where the bus is enumerated automatically on link up.

I assumed that the plan was to add Link Up IRQ support in pwrctrl in the future.

As I dig more and more into pwrctrl, it becomes clear to me that emulating hotplug on non-hotplug capable hw is a bad idea. Though it gives a nice user experience in majority of the cases, it also gives issue in some cases like switch. So pwrctrl will just power on all devices/slots before initial bus scan and will not do any link up based enumeration later. And neither the controller driver.

Quoted 7 lines

If that is not the case, and the plan is instead to eventually remove the existing Link Up IRQ support, then perhaps you could simply apply patches from:
https://lore.kernel.org/linux-pci/[email protected]/

Either the whole series or just patch 1 and 2, maintainer's choice :)

I tried applying the whole series, but it fails at patch 2. Could you please rebase against, pci/controller/dwc and repost?

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#10 lore raw
In reply to Niklas Cassel
Quoted 59 lines · Niklas Cassel

The DWC glue drivers always call pci_host_probe() during probe(), which will allocate upstream bridge resources and enumerate the bus.

For controllers without Link Up IRQ support, pci_host_probe() is called after dw_pcie_wait_for_link(), which will also wait the time required by the PCIe specification before performing PCI Configuration Space reads.

For controllers with Link Up IRQ support, the pci_host_probe() call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.

For controllers with Link Up IRQ support, since the pci_host_probe() call is done without any delay (link training might still be ongoing), it is very unlikely that this scan will find any devices. Once the Link Up IRQ triggers, the Link Up IRQ handler will call pci_rescan_bus().

This works fine for PCIe endpoints connected to the Root Port, since they don't extend the bus. However, if the pci_rescan_bus() call detects a PCIe switch, then there will be a problem when the downstream busses starts showing up, because the PCIe controller is not hotplug capable, so we are not allowed to extend the subordinate bus number after the initial scan, resulting in error messages such as:

pci_bus 0004:43: busn_res: can not insert [bus 43-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:43: busn_res: [bus 43-41] end is updated to 43
pci_bus 0004:43: busn_res: can not insert [bus 43] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:00.0: devices behind bridge are unusable because [bus 43] cannot be assigned for them pci_bus 0004:44: busn_res: can not insert [bus 44-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:44: busn_res: [bus 44-41] end is updated to 44
pci_bus 0004:44: busn_res: can not insert [bus 44] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:02.0: devices behind bridge are unusable because [bus 44] cannot be assigned for them pci_bus 0004:45: busn_res: can not insert [bus 45-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:45: busn_res: [bus 45-41] end is updated to 45
pci_bus 0004:45: busn_res: can not insert [bus 45] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:06.0: devices behind bridge are unusable because [bus 45] cannot be assigned for them pci_bus 0004:46: busn_res: can not insert [bus 46-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:46: busn_res: [bus 46-41] end is updated to 46
pci_bus 0004:46: busn_res: can not insert [bus 46] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:0e.0: devices behind bridge are unusable because [bus 46] cannot be assigned for them pci_bus 0004:42: busn_res: [bus 42-41] end is updated to 46
pci_bus 0004:42: busn_res: can not insert [bus 42-46] under [bus 41] (conflicts with (null) [bus 41]) pci 0004:41:00.0: devices behind bridge are unusable because [bus 42-46] cannot be assigned for them pcieport 0004:40:00.0: bridge has subordinate 41 but max busn 46

While we would like to set the is_hotplug_bridge flag (quirk_hotplug_bridge()), many embedded SoCs that use the DWC controller have synthesized the controller without hot-plug support. Thus, the Link Up IRQ logic is only mimicking hot-plug functionality, i.e. it is not compliant with the PCI Hot-Plug Specification, so we cannot make use of the is_hotplug_bridge flag.

In order to let the Link Up IRQ logic handle PCIe switches that are already powered on (PCIe switches that not powered on already need to implement a pwrctrl driver), don't perform a pci_host_probe() call during probe() (which disregards the delays required by the PCIe specification).

Instead let the first Link Up IRQ call pci_host_probe(). Any follow up Link Up IRQ will call pci_rescan_bus().

Quoted 8 lines

The IRQ name in /proc/interrupts for the pcie-qcom driver is renamed in order to not dereference pp->bridge->bus before it has been assigned.

Fixes ec9fd499b9c6 ("PCI: dw-rockchip: Don't wait for link since we can detect Link Up")
Fixes 0e0b45ab5d77 ("PCI: dw-rockchip: Enumerate endpoints based on dll_link_up IRQ")

Hi Niklas,

Now this patch looks interesting (and managed to catch my attention despite being a controllers/ patch).

You're only talking about bus number allocations here but we did hit a similar problem with bridge window allocations where not enough information is available at the time of the initial scan + resource allocation (currently it is one of the issues that prevents fixing one resource overlap bug):

https://lore.kernel.org/linux-pci/[email protected]/

(Please check also Mani's reply that relates to pwrctrl.)

So I definitely like the general direction this patch goes (though I'm a bit worried though if the downstream busses trickle in, the resource allocation problems will not be fully solved but maybe that fear is not warranted, I don't know).

Signature · 1 line
 i.
Quoted 177 lines

---
Changes since v1:

(quoted diff) +0 -2
-Rename the IRQ on pcie-qcom to not depend on pp->bridge->bus (Mani)
-Make sure that ret is initialized in dw_pcie_host_init() error path (Dan)
 
 .../pci/controller/dwc/pcie-designware-host.c | 71 ++++++++++++++++---
 drivers/pci/controller/dwc/pcie-designware.h  |  5 ++
 drivers/pci/controller/dwc/pcie-dw-rockchip.c |  5 +-
 drivers/pci/controller/dwc/pcie-qcom.c        |  9 +--
 4 files changed, 71 insertions(+), 19 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +62 -9
index e92513c5bda51..bed7b309f6d9e 100644
@@ -565,6 +565,59 @@ static int dw_pcie_host_get_resources(struct dw_pcie_rp *pp)
 	return 0;
 }
 
+static int dw_pcie_host_initial_scan(struct dw_pcie_rp *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct pci_host_bridge *bridge = pp->bridge;
+	int ret;
+
+	ret = pci_host_probe(bridge);
+	if (ret)
+		return ret;
+
+	if (pp->ops->post_init)
+		pp->ops->post_init(pp);
+
+	dwc_pcie_debugfs_init(pci, DW_PCIE_RC_TYPE);
+
+	return 0;
+}
+
+void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp)
+{
+	if (!pp->initial_linkup_irq_done) {
+		int ret;
+
+		ret = dw_pcie_host_initial_scan(pp);
+		if (ret) {
+			struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+			struct device *dev = pci->dev;
+
+			dev_err(dev, "Initial scan from IRQ failed: %d\n", ret);
+
+			dw_pcie_stop_link(pci);
+
+			dw_pcie_edma_remove(pci);
+
+			if (pp->has_msi_ctrl)
+				dw_pcie_free_msi(pp);
+
+			if (pp->ops->deinit)
+				pp->ops->deinit(pp);
+
+			if (pp->cfg)
+				pci_ecam_free(pp->cfg);
+		} else {
+			pp->initial_linkup_irq_done = true;
+		}
+	} else {
+		/* Rescan the bus to enumerate endpoint devices */
+		pci_lock_rescan_remove();
+		pci_rescan_bus(pp->bridge->bus);
+		pci_unlock_rescan_remove();
+	}
+}
+
 int dw_pcie_host_init(struct dw_pcie_rp *pp)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
@@ -669,18 +722,18 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
 	 * If there is no Link Up IRQ, we should not bypass the delay
 	 * because that would require users to manually rescan for devices.
 	 */
-	if (!pp->use_linkup_irq)
+	if (!pp->use_linkup_irq) {
 		/* Ignore errors, the link may come up later */
 		dw_pcie_wait_for_link(pci);
 
-	ret = pci_host_probe(bridge);
-	if (ret)
-		goto err_stop_link;
-
-	if (pp->ops->post_init)
-		pp->ops->post_init(pp);
-
-	dwc_pcie_debugfs_init(pci, DW_PCIE_RC_TYPE);
+		/*
+		 * For platforms with Link Up IRQ, initial scan will be done
+		 * on first Link Up IRQ.
+		 */
+		ret = dw_pcie_host_initial_scan(pp);
+		if (ret)
+			goto err_stop_link;
+	}
 
 	return 0;
 
drivers/pci/controller/dwc/pcie-designware.h +5 -0
index e995f692a1ecd..a31bd93490dcd 100644
@@ -427,6 +427,7 @@ struct dw_pcie_rp {
 	int			msg_atu_index;
 	struct resource		*msg_res;
 	bool			use_linkup_irq;
+	bool			initial_linkup_irq_done;
 	struct pci_eq_presets	presets;
 	struct pci_config_window *cfg;
 	bool			ecam_enabled;
@@ -807,6 +808,7 @@ void dw_pcie_msi_init(struct dw_pcie_rp *pp);
 int dw_pcie_msi_host_init(struct dw_pcie_rp *pp);
 void dw_pcie_free_msi(struct dw_pcie_rp *pp);
 int dw_pcie_setup_rc(struct dw_pcie_rp *pp);
+void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp);
 int dw_pcie_host_init(struct dw_pcie_rp *pp);
 void dw_pcie_host_deinit(struct dw_pcie_rp *pp);
 int dw_pcie_allocate_domains(struct dw_pcie_rp *pp);
@@ -844,6 +846,9 @@ static inline int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 	return 0;
 }
 
+static inline void dw_pcie_handle_link_up_irq(struct dw_pcie_rp *pp)
+{ }
+
 static inline int dw_pcie_host_init(struct dw_pcie_rp *pp)
 {
 	return 0;
drivers/pci/controller/dwc/pcie-dw-rockchip.c +1 -4
index 3e2752c7dd096..8f2cc1ef25e3d 100644
@@ -466,10 +466,7 @@ static irqreturn_t rockchip_pcie_rc_sys_irq_thread(int irq, void *arg)
 		if (rockchip_pcie_link_up(pci)) {
 			msleep(PCIE_RESET_CONFIG_WAIT_MS);
 			dev_dbg(dev, "Received Link up event. Starting enumeration!\n");
-			/* Rescan the bus to enumerate endpoint devices */
-			pci_lock_rescan_remove();
-			pci_rescan_bus(pp->bridge->bus);
-			pci_unlock_rescan_remove();
+			dw_pcie_handle_link_up_irq(pp);
 		}
 	}
 
drivers/pci/controller/dwc/pcie-qcom.c +3 -6
index c48a20602d7fa..04f29cd8d8881 100644
@@ -1617,10 +1617,7 @@ static irqreturn_t qcom_pcie_global_irq_thread(int irq, void *data)
 	if (FIELD_GET(PARF_INT_ALL_LINK_UP, status)) {
 		msleep(PCIE_RESET_CONFIG_WAIT_MS);
 		dev_dbg(dev, "Received Link up event. Starting enumeration!\n");
-		/* Rescan the bus to enumerate endpoint devices */
-		pci_lock_rescan_remove();
-		pci_rescan_bus(pp->bridge->bus);
-		pci_unlock_rescan_remove();
+		dw_pcie_handle_link_up_irq(pp);
 
 		qcom_pcie_icc_opp_update(pcie);
 	} else {
@@ -1937,8 +1934,8 @@ static int qcom_pcie_probe(struct platform_device *pdev)
 		goto err_phy_exit;
 	}
 
-	name = devm_kasprintf(dev, GFP_KERNEL, "qcom_pcie_global_irq%d",
-			      pci_domain_nr(pp->bridge->bus));
+	name = devm_kasprintf(dev, GFP_KERNEL, "qcom_pcie_global_irq_%pOFP",
+			      dev->of_node);
 	if (!name) {
 		ret = -ENOMEM;
 		goto err_host_deinit;
#11 lore raw

Hello Ilpo,

Quoted 11 lines · Ilpo Järvinen

Now this patch looks interesting (and managed to catch my attention despite being a controllers/ patch).

You're only talking about bus number allocations here but we did hit a similar problem with bridge window allocations where not enough information is available at the time of the initial scan + resource allocation (currently it is one of the issues that prevents fixing one resource overlap bug):

https://lore.kernel.org/linux-pci/[email protected]/

I'm quite sure that Val's problem will be solved by applying:
https://lore.kernel.org/linux-pci/[email protected]/

If not, he could try applying the above series, and then this patch on top.

Kind regards,
Niklas