PCI: dwc: Make Link Up IRQ logic handle already powered on PCIe switches
Subsystems (6)
PATCH v2 1/1 PCI: dwc: Make Link Up IRQ logic handle already powered on PCIe switches
The DWC glue drivers always call pci_host_probe() during probe(), which will allocate upstream bridge resources and enumerate the bus.
For controllers without Link Up IRQ support, pci_host_probe() is called after dw_pcie_wait_for_link(), which will also wait the time required by the PCIe specification before performing PCI Configuration Space reads.
For controllers with Link Up IRQ support, the pci_host_probe() call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.
For controllers with Link Up IRQ support, since the pci_host_probe() call is done without any delay (link training might still be ongoing), it is very unlikely that this scan will find any devices. Once the Link Up IRQ triggers, the Link Up IRQ handler will call pci_rescan_bus().
This works fine for PCIe endpoints connected to the Root Port, since they don't extend the bus. However, if the pci_rescan_bus() call detects a PCIe switch, then there will be a problem when the downstream busses starts showing up, because the PCIe controller is not hotplug capable, so we are not allowed to extend the subordinate bus number after the initial scan, resulting in error messages such as:
pci_bus 0004:43: busn_res: can not insert [bus 43-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:43: busn_res: [bus 43-41] end is updated to 43pci_bus 0004:43: busn_res: can not insert [bus 43] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:00.0: devices behind bridge are unusable because [bus 43] cannot be assigned for them pci_bus 0004:44: busn_res: can not insert [bus 44-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:44: busn_res: [bus 44-41] end is updated to 44pci_bus 0004:44: busn_res: can not insert [bus 44] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:02.0: devices behind bridge are unusable because [bus 44] cannot be assigned for them pci_bus 0004:45: busn_res: can not insert [bus 45-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:45: busn_res: [bus 45-41] end is updated to 45pci_bus 0004:45: busn_res: can not insert [bus 45] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:06.0: devices behind bridge are unusable because [bus 45] cannot be assigned for them pci_bus 0004:46: busn_res: can not insert [bus 46-41] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci_bus 0004:46: busn_res: [bus 46-41] end is updated to 46pci_bus 0004:46: busn_res: can not insert [bus 46] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:0e.0: devices behind bridge are unusable because [bus 46] cannot be assigned for them pci_bus 0004:42: busn_res: [bus 42-41] end is updated to 46pci_bus 0004:42: busn_res: can not insert [bus 42-46] under [bus 41] (conflicts with (null) [bus 41]) pci 0004:41:00.0: devices behind bridge are unusable because [bus 42-46] cannot be assigned for them pcieport 0004:40:00.0: bridge has subordinate 41 but max busn 46
While we would like to set the is_hotplug_bridge flag (quirk_hotplug_bridge()), many embedded SoCs that use the DWC controller have synthesized the controller without hot-plug support. Thus, the Link Up IRQ logic is only mimicking hot-plug functionality, i.e. it is not compliant with the PCI Hot-Plug Specification, so we cannot make use of the is_hotplug_bridge flag.
In order to let the Link Up IRQ logic handle PCIe switches that are already powered on (PCIe switches that not powered on already need to implement a pwrctrl driver), don't perform a pci_host_probe() call during probe() (which disregards the delays required by the PCIe specification).
Instead let the first Link Up IRQ call pci_host_probe(). Any follow up Link Up IRQ will call pci_rescan_bus().
The IRQ name in /proc/interrupts for the pcie-qcom driver is renamed in order to not dereference pp->bridge->bus before it has been assigned.
---
Changes since v1:
-Rename the IRQ on pcie-qcom to not depend on pp->bridge->bus (Mani) -Make sure that ret is initialized in dw_pcie_host_init() error path (Dan)
.../pci/controller/dwc/pcie-designware-host.c | 71 ++++++++++++++++--- drivers/pci/controller/dwc/pcie-designware.h | 5 ++ drivers/pci/controller/dwc/pcie-dw-rockchip.c | 5 +- drivers/pci/controller/dwc/pcie-qcom.c | 9 +-- 4 files changed, 71 insertions(+), 19 deletions(-)
drivers/pci/controller/dwc/pcie-designware-host.c +62 -9
@@ -565,6 +565,59 @@ static int dw_pcie_host_get_resources(struct dw_pcie_rp *pp)return 0 ; } +static int dw_pcie_host_initial_scan (struct dw_pcie_rp *pp) +{ +struct dw_pcie *pci =to_dw_pcie_from_pp (pp); +struct pci_host_bridge *bridge = pp->bridge; +int ret; + + ret =pci_host_probe (bridge); +if (ret) +return ret; + +if (pp->ops->post_init) + pp->ops->post_init (pp); + +dwc_pcie_debugfs_init (pci,DW_PCIE_RC_TYPE ); + +return 0 ; +} + +void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp) +{ +if (!pp->initial_linkup_irq_done) { +int ret; + + ret =dw_pcie_host_initial_scan (pp); +if (ret) { +struct dw_pcie *pci =to_dw_pcie_from_pp (pp); +struct device *dev = pci->dev; + +dev_err (dev,"Initial scan from IRQ failed: %d\n" , ret); + +dw_pcie_stop_link (pci); + +dw_pcie_edma_remove (pci); + +if (pp->has_msi_ctrl) +dw_pcie_free_msi (pp); + +if (pp->ops->deinit) + pp->ops->deinit (pp); + +if (pp->cfg) +pci_ecam_free (pp->cfg); + }else { + pp->initial_linkup_irq_done = true; + } + }else { +/* Rescan the bus to enumerate endpoint devices */ +pci_lock_rescan_remove (); +pci_rescan_bus (pp->bridge->bus); +pci_unlock_rescan_remove (); + } +} +int dw_pcie_host_init (struct dw_pcie_rp *pp) {struct dw_pcie *pci =to_dw_pcie_from_pp (pp);
@@ -669,18 +722,18 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)* If there is no Link Up IRQ, we should not bypass the delay * because that would require users to manually rescan for devices. */ -if (!pp->use_linkup_irq) +if (!pp->use_linkup_irq) {/* Ignore errors, the link may come up later */ dw_pcie_wait_for_link (pci); - ret =pci_host_probe (bridge); -if (ret) - goto err_stop_link; - -if (pp->ops->post_init) - pp->ops->post_init (pp); - -dwc_pcie_debugfs_init (pci,DW_PCIE_RC_TYPE ); +/* +* For platforms with Link Up IRQ, initial scan will be done +* on first Link Up IRQ. +*/ + ret =dw_pcie_host_initial_scan (pp); +if (ret) + goto err_stop_link; + }return 0 ;
drivers/pci/controller/dwc/pcie-designware.h +5 -0
@@ -427,6 +427,7 @@ struct dw_pcie_rp {int msg_atu_index;struct resource *msg_res;bool use_linkup_irq; +bool initial_linkup_irq_done;struct pci_eq_presets presets;struct pci_config_window *cfg;bool ecam_enabled;
@@ -807,6 +808,7 @@ void dw_pcie_msi_init(struct dw_pcie_rp *pp);int dw_pcie_msi_host_init (struct dw_pcie_rp *pp);void dw_pcie_free_msi (struct dw_pcie_rp *pp);int dw_pcie_setup_rc (struct dw_pcie_rp *pp); +void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp);int dw_pcie_host_init (struct dw_pcie_rp *pp);void dw_pcie_host_deinit (struct dw_pcie_rp *pp);int dw_pcie_allocate_domains (struct dw_pcie_rp *pp);
@@ -844,6 +846,9 @@ static inline int dw_pcie_setup_rc(struct dw_pcie_rp *pp)return 0 ; } +static inline void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp) +{ } +static inline int dw_pcie_host_init (struct dw_pcie_rp *pp) {return 0 ;
drivers/pci/controller/dwc/pcie-dw-rockchip.c +1 -4
@@ -466,10 +466,7 @@ static irqreturn_t rockchip_pcie_rc_sys_irq_thread(int irq, void *arg)if (rockchip_pcie_link_up (pci)) {msleep (PCIE_RESET_CONFIG_WAIT_MS );dev_dbg (dev,"Received Link up event. Starting enumeration!\n" ); -/* Rescan the bus to enumerate endpoint devices */ -pci_lock_rescan_remove (); -pci_rescan_bus (pp->bridge->bus); -pci_unlock_rescan_remove (); +dw_pcie_handle_link_up_irq (pp); } }
drivers/pci/controller/dwc/pcie-qcom.c +3 -6
@@ -1617,10 +1617,7 @@ static irqreturn_t qcom_pcie_global_irq_thread(int irq, void *data)if (FIELD_GET (PARF_INT_ALL_LINK_UP , status)) {msleep (PCIE_RESET_CONFIG_WAIT_MS );dev_dbg (dev,"Received Link up event. Starting enumeration!\n" ); -/* Rescan the bus to enumerate endpoint devices */ -pci_lock_rescan_remove (); -pci_rescan_bus (pp->bridge->bus); -pci_unlock_rescan_remove (); +dw_pcie_handle_link_up_irq (pp);qcom_pcie_icc_opp_update (pcie); }else {
@@ -1937,8 +1934,8 @@ static int qcom_pcie_probe(struct platform_device *pdev) goto err_phy_exit; } - name =devm_kasprintf (dev,GFP_KERNEL ,"qcom_pcie_global_irq%d" , -pci_domain_nr (pp->bridge->bus)); + name =devm_kasprintf (dev,GFP_KERNEL ,"qcom_pcie_global_irq_%pOFP" , + dev->of_node);if (!name) { ret = -ENOMEM ; goto err_host_deinit;
Signature · 1 line
2.52.0
Forgot to pick up tags from v1, adding them here so that e.g. b4 will pick them up:
On Mon, Dec 01, 2025 at 07:36:35AM +0100, Niklas Cassel wrote:
Mani, could you perhaps try this series on Qcom?
I personally prefer this series over a revert because:
It allows to us to still use a PCIe endpoint that comes up after the main system has booted, e.g. when the PCIe endpoint is a board running pci-epf-test, without the need for a manual rescan on be bus.
If we go with revert instead, this very nice feature would be gone, and the user would need to do a manual rescan of the bus.
One could even argue that that is a user visible regression.
Kind regards,
Niklas
Quoted 156 lines · Niklas Cassel
The DWC glue drivers always call
pci_host_probe()duringprobe(), which will allocate upstream bridge resources and enumerate the bus.For controllers without Link Up IRQ support,
pci_host_probe()is called afterdw_pcie_wait_for_link(), which will also wait the time required by the PCIe specification before performing PCI Configuration Space reads.For controllers with Link Up IRQ support, the
pci_host_probe()call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.For controllers with Link Up IRQ support, since the
pci_host_probe()call is done without any delay (link training might still be ongoing), it is very unlikely that this scan will find any devices. Once the Link Up IRQ triggers, the Link Up IRQ handler will callpci_rescan_bus().This works fine for PCIe endpoints connected to the Root Port, since they don't extend the bus. However, if the
pci_rescan_bus()call detects a PCIe switch, then there will be a problem when the downstream busses starts showing up, because the PCIe controller is not hotplug capable, so we are not allowed to extend the subordinate bus number after the initial scan, resulting in error messages such as:
pci_bus0004:43:busn_res: can not insert [bus 43-41] under [bus 42-41] (conflicts with (null) [bus 42-41])pci_bus0004:43:busn_res: [bus 43-41] end is updated to 43pci_bus0004:43:busn_res: can not insert [bus 43] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:00.0: devices behind bridge are unusable because [bus 43] cannot be assigned for thempci_bus0004:44:busn_res: can not insert [bus 44-41] under [bus 42-41] (conflicts with (null) [bus 42-41])pci_bus0004:44:busn_res: [bus 44-41] end is updated to 44pci_bus0004:44:busn_res: can not insert [bus 44] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:02.0: devices behind bridge are unusable because [bus 44] cannot be assigned for thempci_bus0004:45:busn_res: can not insert [bus 45-41] under [bus 42-41] (conflicts with (null) [bus 42-41])pci_bus0004:45:busn_res: [bus 45-41] end is updated to 45pci_bus0004:45:busn_res: can not insert [bus 45] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:06.0: devices behind bridge are unusable because [bus 45] cannot be assigned for thempci_bus0004:46:busn_res: can not insert [bus 46-41] under [bus 42-41] (conflicts with (null) [bus 42-41])pci_bus0004:46:busn_res: [bus 46-41] end is updated to 46pci_bus0004:46:busn_res: can not insert [bus 46] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:0e.0: devices behind bridge are unusable because [bus 46] cannot be assigned for thempci_bus0004:42:busn_res: [bus 42-41] end is updated to 46pci_bus0004:42:busn_res: can not insert [bus 42-46] under [bus 41] (conflicts with (null) [bus 41]) pci 0004:41:00.0: devices behind bridge are unusable because [bus 42-46] cannot be assigned for them pcieport 0004:40:00.0: bridge has subordinate 41 but max busn 46While we would like to set the
is_hotplug_bridgeflag (quirk_hotplug_bridge()), many embedded SoCs that use the DWC controller have synthesized the controller without hot-plug support. Thus, the Link Up IRQ logic is only mimicking hot-plug functionality, i.e. it is not compliant with the PCI Hot-Plug Specification, so we cannot make use of theis_hotplug_bridgeflag.In order to let the Link Up IRQ logic handle PCIe switches that are already powered on (PCIe switches that not powered on already need to implement a pwrctrl driver), don't perform a
pci_host_probe()call duringprobe()(which disregards the delays required by the PCIe specification).Instead let the first Link Up IRQ call
pci_host_probe(). Any follow up Link Up IRQ will callpci_rescan_bus().The IRQ name in
/proc/interruptsfor the pcie-qcom driver is renamed in order to not dereferencepp->bridge->busbefore it has been assigned.Fixes ec9fd499b9c6 ("PCI: dw-rockchip: Don't wait for link since we can detect Link Up")Fixes 0e0b45ab5d77 ("PCI: dw-rockchip: Enumerate endpoints based on dll_link_up IRQ")Reported-by FUKAUMI Naoki <[email protected]>Signed-off-by Niklas Cassel <[email protected]>---
Changes since v1:(quoted diff) +0
-2-Rename the IRQ on pcie-qcom to not depend on pp->bridge->bus (Mani) -Make sure that ret is initialized in dw_pcie_host_init() error path (Dan) .../pci/controller/dwc/pcie-designware-host.c | 71 ++++++++++++++++--- drivers/pci/controller/dwc/pcie-designware.h | 5 ++ drivers/pci/controller/dwc/pcie-dw-rockchip.c | 5 +- drivers/pci/controller/dwc/pcie-qcom.c | 9 +-- 4 files changed, 71 insertions(+), 19 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +54
-6@@ -565,6 +565,59 @@ static int dw_pcie_host_get_resources(struct dw_pcie_rp *pp)return 0 ; } +static int dw_pcie_host_initial_scan (struct dw_pcie_rp *pp) +{ +struct dw_pcie *pci =to_dw_pcie_from_pp (pp); +struct pci_host_bridge *bridge = pp->bridge; +int ret; + + ret =pci_host_probe (bridge); +if (ret) +return ret; + +if (pp->ops->post_init) + pp->ops->post_init (pp); + +dwc_pcie_debugfs_init (pci,DW_PCIE_RC_TYPE ); + +return 0 ; +} + +void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp) +{ +if (!pp->initial_linkup_irq_done) { +int ret; + + ret =dw_pcie_host_initial_scan (pp); +if (ret) { +struct dw_pcie *pci =to_dw_pcie_from_pp (pp); +struct device *dev = pci->dev; + +dev_err (dev,"Initial scan from IRQ failed: %d\n" , ret); + +dw_pcie_stop_link (pci); + +dw_pcie_edma_remove (pci); + +if (pp->has_msi_ctrl) +dw_pcie_free_msi (pp); + +if (pp->ops->deinit) + pp->ops->deinit (pp); + +if (pp->cfg) +pci_ecam_free (pp->cfg); + }else { + pp->initial_linkup_irq_done = true; + } + }else { +/* Rescan the bus to enumerate endpoint devices */ +pci_lock_rescan_remove (); +pci_rescan_bus (pp->bridge->bus); +pci_unlock_rescan_remove (); + } +} +int dw_pcie_host_init (struct dw_pcie_rp *pp) {struct dw_pcie *pci =to_dw_pcie_from_pp (pp);@@ -669,18 +722,18 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)* If there is no Link Up IRQ, we should not bypass the delay * because that would require users to manually rescan for devices. */ -if (!pp->use_linkup_irq) +if (!pp->use_linkup_irq) {/* Ignore errors, the link may come up later */ dw_pcie_wait_for_link (pci); - ret =pci_host_probe (bridge); -if (ret) - goto err_stop_link; - -if (pp->ops->post_init)
drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_host_init(struct dw_pcie_rp *pp) · line 735
Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +7
-3- pp->ops->post_init (pp); - -dwc_pcie_debugfs_init (pci,DW_PCIE_RC_TYPE ); +/* +* For platforms with Link Up IRQ, initial scan will be done +* on first Link Up IRQ. +*/ + ret =dw_pcie_host_initial_scan (pp); +if (ret) + goto err_stop_link;
This will cause breakages in qcom platforms, qcom platforms use pwrctrl framework for enabling power to the endpoints. And the pwrctrl drivers will be created as part of the initial scan. As we skipping the initial scan here, pwrctrl driver will be never created and the power to the endpoint will be never enabled. This will cause link to be never up.
- Krishna Chaitanya.
Quoted 77 lines
drivers/pci/controller/dwc/pcie-designware-host.c +1
-0+ }return 0 ;drivers/pci/controller/dwc/pcie-designware.h +5
-0@@ -427,6 +427,7 @@ struct dw_pcie_rp {int msg_atu_index;struct resource *msg_res;bool use_linkup_irq; +bool initial_linkup_irq_done;struct pci_eq_presets presets;struct pci_config_window *cfg;bool ecam_enabled;@@ -807,6 +808,7 @@ void dw_pcie_msi_init(struct dw_pcie_rp *pp);int dw_pcie_msi_host_init (struct dw_pcie_rp *pp);void dw_pcie_free_msi (struct dw_pcie_rp *pp);int dw_pcie_setup_rc (struct dw_pcie_rp *pp); +void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp);int dw_pcie_host_init (struct dw_pcie_rp *pp);void dw_pcie_host_deinit (struct dw_pcie_rp *pp);int dw_pcie_allocate_domains (struct dw_pcie_rp *pp);@@ -844,6 +846,9 @@ static inline int dw_pcie_setup_rc(struct dw_pcie_rp *pp)return 0 ; } +static inline void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp) +{ } +static inline int dw_pcie_host_init (struct dw_pcie_rp *pp) {return 0 ;drivers/pci/controller/dwc/pcie-dw-rockchip.c +1
-4@@ -466,10 +466,7 @@ static irqreturn_t rockchip_pcie_rc_sys_irq_thread(int irq, void *arg)if (rockchip_pcie_link_up (pci)) {msleep (PCIE_RESET_CONFIG_WAIT_MS );dev_dbg (dev,"Received Link up event. Starting enumeration!\n" ); -/* Rescan the bus to enumerate endpoint devices */ -pci_lock_rescan_remove (); -pci_rescan_bus (pp->bridge->bus); -pci_unlock_rescan_remove (); +dw_pcie_handle_link_up_irq (pp); } }drivers/pci/controller/dwc/pcie-qcom.c +3
-6@@ -1617,10 +1617,7 @@ static irqreturn_t qcom_pcie_global_irq_thread(int irq, void *data)if (FIELD_GET (PARF_INT_ALL_LINK_UP , status)) {msleep (PCIE_RESET_CONFIG_WAIT_MS );dev_dbg (dev,"Received Link up event. Starting enumeration!\n" ); -/* Rescan the bus to enumerate endpoint devices */ -pci_lock_rescan_remove (); -pci_rescan_bus (pp->bridge->bus); -pci_unlock_rescan_remove (); +dw_pcie_handle_link_up_irq (pp);qcom_pcie_icc_opp_update (pcie); }else {@@ -1937,8 +1934,8 @@ static int qcom_pcie_probe(struct platform_device *pdev) goto err_phy_exit; } - name =devm_kasprintf (dev,GFP_KERNEL ,"qcom_pcie_global_irq%d" , -pci_domain_nr (pp->bridge->bus)); + name =devm_kasprintf (dev,GFP_KERNEL ,"qcom_pcie_global_irq_%pOFP" , + dev->of_node);if (!name) { ret = -ENOMEM ; goto err_host_deinit;
Hello Krishna,
Currently:
For controllers with Link Up IRQ support, the pci_host_probe() call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.
This seems quite bad.
A device might not be fully initialized during during the time of these PCI Configuration Space reads, but might still return some bogus values that are actually different from the Configuration Space reads if done after respecting the delays mandated by the PCIe specification.
I think the options are:
1) Keep the pci_host_probe() call in dw_pcie_host_init() for controllers with Link Up IRQ support, but make sure that we respect the delays also in this case.
or
2) Remove the pci_host_probe() call from dw_pcie_host_init(), and make sure that pci_host_probe() is done by the first Link Up IRQ
(i.e. what this patch does).
One big thing with using the Link Up IRQ is to not do any delay during PCIe controller driver's probe(), which reduces startup time, exactly as your commit message in commit 36971d6c5a9a ("PCI: qcom: Don't wait for link if we can detect Link Up") explains.
Therefore, I don't think that 1) is a good solution, so that leaves us with 2).
If pwrctrl drivers are created as part of the pci_host_probe() call, I think that perhaps an alternative would be to create an explict pwrctrl_init() function, and let the PCI controller drivers that actually use pwrctrl call that from their probe().
(And just remove the same from pci_host_probe() ?)
In fact, looking at your suggested patches (that hasn't landed yet):
[PATCH 3/5] PCI/pwrctrl: Add APIs for explicitly creating and destroying pwrctrl devices [PATCH 5/5] PCI/pwrctrl: Switch to the new pwrctrl APIs
https://lore.kernel.org/linux-pci/[email protected]/
https://lore.kernel.org/linux-pci/[email protected]/
Seem to do exactly that:
Call pci_pwrctrl_create_devices() explicitly from the PCIe controller drivers directly, and removes the pci_pwrctrl_create_device() call from pci_host_probe().
So I don't really understand your concern with this series, at least not if it goes on top of your series:
https://lore.kernel.org/linux-pci/[email protected]/
Kind regards,
Niklas
Quoted 45 lines · Niklas Cassel
Hello Krishna,
Currently:
For controllers with Link Up IRQ support, thepci_host_probe()call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.This seems quite bad.
A device might not be fully initialized during during the time of these PCI Configuration Space reads, but might still return some bogus values that are actually different from the Configuration Space reads if done after respecting the delays mandated by the PCIe specification.
I think the options are:
1) Keep thepci_host_probe()call indw_pcie_host_init()for controllers with Link Up IRQ support, but make sure that we respect the delays also in this case.
or
2) Remove thepci_host_probe()call fromdw_pcie_host_init(), and make sure thatpci_host_probe()is done by the first Link Up IRQ (i.e. what this patch does).One big thing with using the Link Up IRQ is to not do any delay during PCIe controller driver's
probe(), which reduces startup time, exactly as your commit message in commit36971d6c5a9a("PCI: qcom: Don't wait for link if we can detect Link Up") explains.
Therefore, I don't think that 1) is a good solution, so that leaves us with 2).If pwrctrl drivers are created as part of the
pci_host_probe()call, I think that perhaps an alternative would be to create an explictpwrctrl_init()function, and let the PCI controller drivers that actually use pwrctrl call that from theirprobe().
(And just remove the same frompci_host_probe()?)In fact, looking at your suggested patches (that hasn't landed yet):
[PATCH 3/5] PCI/pwrctrl: Add APIs for explicitly creating and destroying pwrctrl devices [PATCH 5/5] PCI/pwrctrl: Switch to the new pwrctrl APIshttps://lore.kernel.org/linux-pci/[email protected]/
https://lore.kernel.org/linux-pci/[email protected]/
Quoted 7 lines
Seem to do exactly that:
Callpci_pwrctrl_create_devices()explicitly from the PCIe controller drivers directly, and removes thepci_pwrctrl_create_device()call frompci_host_probe().So I don't really understand your concern with this series, at least not if it goes on top of your series:
https://lore.kernel.org/linux-pci/[email protected]/
Hi Niklas,
If this series goes on top of the our series i.e pwrctrl rework series, I don't have any concerns. My only concern is link up IRQ never fires if this patch goes before series.
- Krishna Chaitanya.
Quoted 4 lines
Kind regards,
Niklas
Quoted 5 lines · Krishna Chaitanya Chundru
Quoted 4 lines · Niklas Cassel
So I don't really understand your concern with this series, at least not if it goes on top of your series:
https://lore.kernel.org/linux-pci/[email protected]/
Quoted 7 lines
Hi Niklas,
If this series goes on top of the our series i.e pwrctrl rework series, I don't have any concerns.
My only concern is link up IRQ never fires if this patch goes before series.- Krishna Chaitanya.
This patch missed the v6.19 merge window (and so did the pwrctrl rework series), so as long as Mani queues up both for v6.20 (with the pwrctrl rework series getting applied first), I think we are good.
Kind regards,
Niklas
Manivannan Sadhasivam
maintainerQuoted 12 lines · Niklas Cassel
Quoted 11 lines · Krishna Chaitanya Chundru
Quoted 4 lines · Niklas Cassel
So I don't really understand your concern with this series, at least not if it goes on top of your series:
https://lore.kernel.org/linux-pci/[email protected]/Hi Niklas,
If this series goes on top of the our series i.e pwrctrl rework series, I don't have any concerns.
My only concern is link up IRQ never fires if this patch goes before series.
Quoted 6 lines
Quoted 1 line
- Krishna Chaitanya.
This patch missed the v6.19 merge window (and so did the pwrctrl rework series), so as long as Mani queues up both for v6.20 (with the pwrctrl rework series getting applied first), I think we are good.
The plan is to merge pwrctrl series to v6.20 (unless we get strong objections), but once that happens, Qcom doesn't need this patch.
So it'd be good if you can just limit this patch to just Rockchip. Then once the Rockchip also moves to pwrctrl, we can revert this patch (also the whole IRQ based link up since the Root Port is not hotplug capable).
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Hello Mani,
Quoted 11 lines · Manivannan Sadhasivam
Quoted 4 lines
This patch missed the v6.19 merge window (and so did the pwrctrl rework series), so as long as Mani queues up both for v6.20 (with the pwrctrl rework series getting applied first), I think we are good.
The plan is to merge pwrctrl series to v6.20 (unless we get strong objections), but once that happens, Qcom doesn't need this patch.
So it'd be good if you can just limit this patch to just Rockchip. Then once the Rockchip also moves to pwrctrl, we can revert this patch (also the whole IRQ based link up since the Root Port is not hotplug capable).
The main reason why I did not like a revert, was because it would remove the nice feature where the bus is enumerated automatically on link up.
I assumed that the plan was to add Link Up IRQ support in pwrctrl in the future.
If that is not the case, and the plan is instead to eventually remove the existing Link Up IRQ support, then perhaps you could simply apply patches from:
https://lore.kernel.org/linux-pci/[email protected]/
Either the whole series or just patch 1 and 2, maintainer's choice :)
Kind regards,
Niklas
Manivannan Sadhasivam
maintainerQuoted 15 lines · Niklas Cassel
Hello Mani,
Quoted 11 lines · Manivannan Sadhasivam
Quoted 4 lines
This patch missed the v6.19 merge window (and so did the pwrctrl rework series), so as long as Mani queues up both for v6.20 (with the pwrctrl rework series getting applied first), I think we are good.
The plan is to merge pwrctrl series to v6.20 (unless we get strong objections), but once that happens, Qcom doesn't need this patch.
So it'd be good if you can just limit this patch to just Rockchip. Then once the Rockchip also moves to pwrctrl, we can revert this patch (also the whole IRQ based link up since the Root Port is not hotplug capable).
Quoted 6 lines
The main reason why I did not like a revert, was because it would remove the nice feature where the bus is enumerated automatically on link up.
I assumed that the plan was to add Link Up IRQ support in pwrctrl in the future.
As I dig more and more into pwrctrl, it becomes clear to me that emulating hotplug on non-hotplug capable hw is a bad idea. Though it gives a nice user experience in majority of the cases, it also gives issue in some cases like switch. So pwrctrl will just power on all devices/slots before initial bus scan and will not do any link up based enumeration later. And neither the controller driver.
Quoted 7 lines
If that is not the case, and the plan is instead to eventually remove the existing Link Up IRQ support, then perhaps you could simply apply patches from:
https://lore.kernel.org/linux-pci/[email protected]/Either the whole series or just patch 1 and 2, maintainer's choice :)
I tried applying the whole series, but it fails at patch 2. Could you please rebase against, pci/controller/dwc and repost?
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 59 lines · Niklas Cassel
The DWC glue drivers always call
pci_host_probe()duringprobe(), which will allocate upstream bridge resources and enumerate the bus.For controllers without Link Up IRQ support,
pci_host_probe()is called afterdw_pcie_wait_for_link(), which will also wait the time required by the PCIe specification before performing PCI Configuration Space reads.For controllers with Link Up IRQ support, the
pci_host_probe()call (which will perform PCI Configuration Space reads) is done without any of the delays mandated by the PCIe specification.For controllers with Link Up IRQ support, since the
pci_host_probe()call is done without any delay (link training might still be ongoing), it is very unlikely that this scan will find any devices. Once the Link Up IRQ triggers, the Link Up IRQ handler will callpci_rescan_bus().This works fine for PCIe endpoints connected to the Root Port, since they don't extend the bus. However, if the
pci_rescan_bus()call detects a PCIe switch, then there will be a problem when the downstream busses starts showing up, because the PCIe controller is not hotplug capable, so we are not allowed to extend the subordinate bus number after the initial scan, resulting in error messages such as:
pci_bus0004:43:busn_res: can not insert [bus 43-41] under [bus 42-41] (conflicts with (null) [bus 42-41])pci_bus0004:43:busn_res: [bus 43-41] end is updated to 43pci_bus0004:43:busn_res: can not insert [bus 43] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:00.0: devices behind bridge are unusable because [bus 43] cannot be assigned for thempci_bus0004:44:busn_res: can not insert [bus 44-41] under [bus 42-41] (conflicts with (null) [bus 42-41])pci_bus0004:44:busn_res: [bus 44-41] end is updated to 44pci_bus0004:44:busn_res: can not insert [bus 44] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:02.0: devices behind bridge are unusable because [bus 44] cannot be assigned for thempci_bus0004:45:busn_res: can not insert [bus 45-41] under [bus 42-41] (conflicts with (null) [bus 42-41])pci_bus0004:45:busn_res: [bus 45-41] end is updated to 45pci_bus0004:45:busn_res: can not insert [bus 45] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:06.0: devices behind bridge are unusable because [bus 45] cannot be assigned for thempci_bus0004:46:busn_res: can not insert [bus 46-41] under [bus 42-41] (conflicts with (null) [bus 42-41])pci_bus0004:46:busn_res: [bus 46-41] end is updated to 46pci_bus0004:46:busn_res: can not insert [bus 46] under [bus 42-41] (conflicts with (null) [bus 42-41]) pci 0004:42:0e.0: devices behind bridge are unusable because [bus 46] cannot be assigned for thempci_bus0004:42:busn_res: [bus 42-41] end is updated to 46pci_bus0004:42:busn_res: can not insert [bus 42-46] under [bus 41] (conflicts with (null) [bus 41]) pci 0004:41:00.0: devices behind bridge are unusable because [bus 42-46] cannot be assigned for them pcieport 0004:40:00.0: bridge has subordinate 41 but max busn 46While we would like to set the
is_hotplug_bridgeflag (quirk_hotplug_bridge()), many embedded SoCs that use the DWC controller have synthesized the controller without hot-plug support. Thus, the Link Up IRQ logic is only mimicking hot-plug functionality, i.e. it is not compliant with the PCI Hot-Plug Specification, so we cannot make use of theis_hotplug_bridgeflag.In order to let the Link Up IRQ logic handle PCIe switches that are already powered on (PCIe switches that not powered on already need to implement a pwrctrl driver), don't perform a
pci_host_probe()call duringprobe()(which disregards the delays required by the PCIe specification).Instead let the first Link Up IRQ call
pci_host_probe(). Any follow up Link Up IRQ will callpci_rescan_bus().
Quoted 8 lines
The IRQ name in
/proc/interruptsfor the pcie-qcom driver is renamed in order to not dereferencepp->bridge->busbefore it has been assigned.Fixes ec9fd499b9c6 ("PCI: dw-rockchip: Don't wait for link since we can detect Link Up")Fixes 0e0b45ab5d77 ("PCI: dw-rockchip: Enumerate endpoints based on dll_link_up IRQ")Reported-by FUKAUMI Naoki <[email protected]>Signed-off-by Niklas Cassel <[email protected]>
Hi Niklas,
Now this patch looks interesting (and managed to catch my attention despite being a controllers/ patch).
You're only talking about bus number allocations here but we did hit a similar problem with bridge window allocations where not enough information is available at the time of the initial scan + resource allocation (currently it is one of the issues that prevents fixing one resource overlap bug):
https://lore.kernel.org/linux-pci/[email protected]/
(Please check also Mani's reply that relates to pwrctrl.)
So I definitely like the general direction this patch goes (though I'm a bit worried though if the downstream busses trickle in, the resource allocation problems will not be fully solved but maybe that fear is not warranted, I don't know).
Signature · 1 line
i.
Quoted 177 lines
---
Changes since v1:(quoted diff) +0
-2-Rename the IRQ on pcie-qcom to not depend on pp->bridge->bus (Mani) -Make sure that ret is initialized in dw_pcie_host_init() error path (Dan) .../pci/controller/dwc/pcie-designware-host.c | 71 ++++++++++++++++--- drivers/pci/controller/dwc/pcie-designware.h | 5 ++ drivers/pci/controller/dwc/pcie-dw-rockchip.c | 5 +- drivers/pci/controller/dwc/pcie-qcom.c | 9 +-- 4 files changed, 71 insertions(+), 19 deletions(-)drivers/pci/controller/dwc/pcie-designware-host.c +62
-9@@ -565,6 +565,59 @@ static int dw_pcie_host_get_resources(struct dw_pcie_rp *pp)return 0 ; } +static int dw_pcie_host_initial_scan (struct dw_pcie_rp *pp) +{ +struct dw_pcie *pci =to_dw_pcie_from_pp (pp); +struct pci_host_bridge *bridge = pp->bridge; +int ret; + + ret =pci_host_probe (bridge); +if (ret) +return ret; + +if (pp->ops->post_init) + pp->ops->post_init (pp); + +dwc_pcie_debugfs_init (pci,DW_PCIE_RC_TYPE ); + +return 0 ; +} + +void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp) +{ +if (!pp->initial_linkup_irq_done) { +int ret; + + ret =dw_pcie_host_initial_scan (pp); +if (ret) { +struct dw_pcie *pci =to_dw_pcie_from_pp (pp); +struct device *dev = pci->dev; + +dev_err (dev,"Initial scan from IRQ failed: %d\n" , ret); + +dw_pcie_stop_link (pci); + +dw_pcie_edma_remove (pci); + +if (pp->has_msi_ctrl) +dw_pcie_free_msi (pp); + +if (pp->ops->deinit) + pp->ops->deinit (pp); + +if (pp->cfg) +pci_ecam_free (pp->cfg); + }else { + pp->initial_linkup_irq_done = true; + } + }else { +/* Rescan the bus to enumerate endpoint devices */ +pci_lock_rescan_remove (); +pci_rescan_bus (pp->bridge->bus); +pci_unlock_rescan_remove (); + } +} +int dw_pcie_host_init (struct dw_pcie_rp *pp) {struct dw_pcie *pci =to_dw_pcie_from_pp (pp);@@ -669,18 +722,18 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)* If there is no Link Up IRQ, we should not bypass the delay * because that would require users to manually rescan for devices. */ -if (!pp->use_linkup_irq) +if (!pp->use_linkup_irq) {/* Ignore errors, the link may come up later */ dw_pcie_wait_for_link (pci); - ret =pci_host_probe (bridge); -if (ret) - goto err_stop_link; - -if (pp->ops->post_init) - pp->ops->post_init (pp); - -dwc_pcie_debugfs_init (pci,DW_PCIE_RC_TYPE ); +/* +* For platforms with Link Up IRQ, initial scan will be done +* on first Link Up IRQ. +*/ + ret =dw_pcie_host_initial_scan (pp); +if (ret) + goto err_stop_link; + }return 0 ;drivers/pci/controller/dwc/pcie-designware.h +5
-0@@ -427,6 +427,7 @@ struct dw_pcie_rp {int msg_atu_index;struct resource *msg_res;bool use_linkup_irq; +bool initial_linkup_irq_done;struct pci_eq_presets presets;struct pci_config_window *cfg;bool ecam_enabled;@@ -807,6 +808,7 @@ void dw_pcie_msi_init(struct dw_pcie_rp *pp);int dw_pcie_msi_host_init (struct dw_pcie_rp *pp);void dw_pcie_free_msi (struct dw_pcie_rp *pp);int dw_pcie_setup_rc (struct dw_pcie_rp *pp); +void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp);int dw_pcie_host_init (struct dw_pcie_rp *pp);void dw_pcie_host_deinit (struct dw_pcie_rp *pp);int dw_pcie_allocate_domains (struct dw_pcie_rp *pp);@@ -844,6 +846,9 @@ static inline int dw_pcie_setup_rc(struct dw_pcie_rp *pp)return 0 ; } +static inline void dw_pcie_handle_link_up_irq (struct dw_pcie_rp *pp) +{ } +static inline int dw_pcie_host_init (struct dw_pcie_rp *pp) {return 0 ;drivers/pci/controller/dwc/pcie-dw-rockchip.c +1
-4@@ -466,10 +466,7 @@ static irqreturn_t rockchip_pcie_rc_sys_irq_thread(int irq, void *arg)if (rockchip_pcie_link_up (pci)) {msleep (PCIE_RESET_CONFIG_WAIT_MS );dev_dbg (dev,"Received Link up event. Starting enumeration!\n" ); -/* Rescan the bus to enumerate endpoint devices */ -pci_lock_rescan_remove (); -pci_rescan_bus (pp->bridge->bus); -pci_unlock_rescan_remove (); +dw_pcie_handle_link_up_irq (pp); } }drivers/pci/controller/dwc/pcie-qcom.c +3
-6@@ -1617,10 +1617,7 @@ static irqreturn_t qcom_pcie_global_irq_thread(int irq, void *data)if (FIELD_GET (PARF_INT_ALL_LINK_UP , status)) {msleep (PCIE_RESET_CONFIG_WAIT_MS );dev_dbg (dev,"Received Link up event. Starting enumeration!\n" ); -/* Rescan the bus to enumerate endpoint devices */ -pci_lock_rescan_remove (); -pci_rescan_bus (pp->bridge->bus); -pci_unlock_rescan_remove (); +dw_pcie_handle_link_up_irq (pp);qcom_pcie_icc_opp_update (pcie); }else {@@ -1937,8 +1934,8 @@ static int qcom_pcie_probe(struct platform_device *pdev) goto err_phy_exit; } - name =devm_kasprintf (dev,GFP_KERNEL ,"qcom_pcie_global_irq%d" , -pci_domain_nr (pp->bridge->bus)); + name =devm_kasprintf (dev,GFP_KERNEL ,"qcom_pcie_global_irq_%pOFP" , + dev->of_node);if (!name) { ret = -ENOMEM ; goto err_host_deinit;
Hello Ilpo,
Quoted 11 lines · Ilpo Järvinen
Now this patch looks interesting (and managed to catch my attention despite being a controllers/ patch).
You're only talking about bus number allocations here but we did hit a similar problem with bridge window allocations where not enough information is available at the time of the initial scan + resource allocation (currently it is one of the issues that prevents fixing one resource overlap bug):
I'm quite sure that Val's problem will be solved by applying:
https://lore.kernel.org/linux-pci/[email protected]/
If not, he could try applying the above series, and then this patch on top.
Kind regards,
Niklas