Skip to contentSkip to conversation

PCI: Enable runtime pm of the host bridge

PATCH In review Krishna chaitanya chundru 7 participants 22 messages Nov 11, 2024 – Jan 28, 2025

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

v7 cover PCI: Enable runtime pm of the host bridge

It is a property of the runtime PM framework that it can only follow continuous dependency chains. That is, if there is a device with runtime PM disabled in a dependency chain, runtime PM cannot be enabled for devices below it and above it in that chain both at the same time.

Since runtime PM is disabled for host bridge, the state of the child devices under the host bridge is not taken into account by PM framework for the top level parent, PCIe controller. So PM framework, allows the controller driver to enter runtime PM irrespective of the state of the devices under the host bridge.

So enable runtime pm of the host bridge device, so that dependency chain in maintained between endpoint devices and the controller driver.

PM framework expectes parent runtime pm enabled before enabling runtime pm of the child. As PCIe starfive device is enabling runtime pm after the pci_host_probe which enables runtime pm of the child device i.e for the bridge device a warning is shown saying "pcie-starfive 940000000.pcie:
Enabling runtime PM for inactive device with active children" and also shows possible circular locking dependency detected message.

As it is must to enable parent device's runtime PM before enabling child's runtime pm as the pcie-starfive device runtime pm is enabled after child runtime starfive device is seeing the warning.

In the first patch fix the pcie-starfive driver by enabling runtime pm before calling pci_host_probe().

All other PCIe controller drivers are enabling runtime pm before calling pci_host_probe() which is as expected so don't require any fix like pcie-starfive driver.

---
Changes in v7:
- Bring the cover letter details to commit message as suggested by bjorn.
- Add a comment in the driver to ensure pm_runtime_enable is called
before host bridge pm_runtime_enable().
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- include the patch by mayank which fixes runtime pm enable order for
pcie starfive driver.
Changes in v5:
- call pm_runtime_no_callbacks() as suggested by Rafael.
- include the commit texts as suggested by Rafael.
- Link to v4: https://lore.kernel.org/linux-pci/[email protected]/
Changes in v4:
- Changed pm_runtime_enable() to devm_pm_runtime_enable() (suggested by mayank)
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes in v3:
- Moved the runtime API call's from the dwc driver to PCI framework
as it is applicable for all (suggested by mani)
- Updated the commit message.
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Updated commit message as suggested by mani.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Krishna chaitanya chundru (1):
PCI: Enable runtime pm of the host bridge

Mayank Rana (1):
PCI: starfive: Enable PCIe controller's runtime PM before probing host bridge

 drivers/pci/controller/plda/pcie-starfive.c | 10 +++++++---
 drivers/pci/probe.c                         | 11 +++++++++++
 2 files changed, 18 insertions(+), 3 deletions(-)

---
base-commit: 2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2
change-id: 20241016-runtime_pm-d3dbf41736b6

Best regards,

Signature · 1 line
Krishna chaitanya chundru <[email protected]>
#1 lore raw

PATCH v7 1/2 PCI: starfive: Enable PCIe controller's runtime PM before probing host bridge

From: Mayank Rana <[email protected]>

PCIe controller device (i.e. PCIe starfive device) is parent to PCIe host bridge device. To enable runtime PM of PCIe host bridge device (child device), it is must to enable parent device's runtime PM to avoid seeing the below warning from PM core:

pcie-starfive 940000000.pcie: Enabling runtime PM for inactive device with active children

Fix this issue by enabling starfive pcie controller device's runtime PM before calling pci_host_probe() in plda_pcie_host_init().

---
v3->v6:
- no change
Link to v3: https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
v2->v3:
- Update commit description based on Mani's feedback
- Updated Reviewed-by tag
Link to v2: https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/

v1->v2: Updated commit description based on Bjorn's feedback
Link to v1: https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/

---
 drivers/pci/controller/plda/pcie-starfive.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/pci/controller/plda/pcie-starfive.c +7 -3
index c9933ecf6833..0564fdce47c2 100644
@@ -404,6 +404,9 @@ static int starfive_pcie_probe(struct platform_device *pdev)
 	if (ret)
 		return ret;
 
+	pm_runtime_enable(&pdev->dev);
+	pm_runtime_get_sync(&pdev->dev);
+
 	plda->host_ops = &sf_host_ops;
 	plda->num_events = PLDA_MAX_EVENT_NUM;
 	/* mask doorbell event */
@@ -413,11 +416,12 @@ static int starfive_pcie_probe(struct platform_device *pdev)
 	plda->events_bitmap <<= PLDA_NUM_DMA_EVENTS;
 	ret = plda_pcie_host_init(&pcie->plda, &starfive_pcie_ops,
 				  &stf_pcie_event);
-	if (ret)
+	if (ret) {
+		pm_runtime_put_sync(&pdev->dev);
+		pm_runtime_disable(&pdev->dev);
 		return ret;
+	}
 
-	pm_runtime_enable(&pdev->dev);
-	pm_runtime_get_sync(&pdev->dev);
 	platform_set_drvdata(pdev, pcie);
 
 	return 0;
Signature · 1 line
2.34.1
#2 lore raw

PATCH v7 2/2 PCI: Enable runtime pm of the host bridge

In reply to Krishna chaitanya chundru

The Controller driver is the parent device of the PCIe host bridge, PCI-PCI bridge and PCIe endpoint as shown below.

PCIe controller(Top level parent & parent of host bridge)
|
v
PCIe Host bridge(Parent of PCI-PCI bridge)
|
v
PCI-PCI bridge(Parent of endpoint driver)
|
v
PCIe endpoint driver

Now, when the controller device goes to runtime suspend, PM framework will check the runtime PM state of the child device (host bridge) and will find it to be disabled. So it will allow the parent (controller device) to go to runtime suspend. Only if the child device's state was 'active' it will prevent the parent to get suspended.

It is a property of the runtime PM framework that it can only follow continuous dependency chains. That is, if there is a device with runtime PM disabled in a dependency chain, runtime PM cannot be enabled for devices below it and above it in that chain both at the same time.

Since runtime PM is disabled for host bridge, the state of the child devices under the host bridge is not taken into account by PM framework for the top level parent, PCIe controller. So PM framework, allows the controller driver to enter runtime PM irrespective of the state of the devices under the host bridge. And this causes the topology breakage and also possible PM issues like controller driver goes to runtime suspend while endpoint driver is doing some transfers.

Because of the above, in order to enable runtime PM for a PCIe controller device, one needs to ensure that runtime PM is enabled for all devices in every dependency chain between it and any PCIe endpoint (as runtime PM is enabled for PCIe endpoints).

This means that runtime PM needs to be enabled for the host bridge device, which is present in all of these dependency chains.

After this change, the host bridge device will be runtime-suspended by the runtime PM framework automatically after suspending its last child and it will be runtime-resumed automatically before resuming its first child which will allow the runtime PM framework to track dependencies between the host bridge device and all of its descendants.

PM framework expectes parent runtime pm enabled before enabling runtime pm of the child. Ensure pm_runtime_enable() is called for the controller drivers, before calling pci_host_probe() as pm frameworks expects if the parent device supports runtime pm then it needs to enabled before child runtime pm.

---
Changes in v7:
- update commit text & and add a comment in the driver as suggested by bjorn. Changes in v6:
- no change
Changes in v5:
- call pm_runtime_no_callbacks() as suggested by Rafael.
- include the commit texts as suggested by Rafael.
- Link to v4: https://lore.kernel.org/linux-pci/[email protected]/
Changes in v4:
- Changed pm_runtime_enable() to devm_pm_runtime_enable() (suggested by mayank)
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes in v3:
- Moved the runtime API call's from the dwc driver to PCI framework
as it is applicable for all (suggested by mani)
- Updated the commit message.
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Updated commit message as suggested by mani.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
 drivers/pci/probe.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/pci/probe.c +11 -0
index 4f68414c3086..ee30e6024f52 100644
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
 		pcie_bus_configure_settings(child);
 
 	pci_bus_add_devices(bus);
+
+	/*
+	 * Ensure pm_runtime_enable() is called for the controller drivers,
+	 * before calling pci_host_probe() as pm frameworks expects if the
+	 * parent device supports runtime pm then it needs to enabled before
+	 * child runtime pm.
+	 */
+	pm_runtime_set_active(&bridge->dev);
+	pm_runtime_no_callbacks(&bridge->dev);
+	devm_pm_runtime_enable(&bridge->dev);
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(pci_host_probe);
Signature · 1 line
2.34.1
#3 lore raw

…

Quoted 1 line

PM framework expectes parent runtime pm enabled before enabling runtime

expects? PM?

Quoted 1 line

pm of the child. …

PM?

Quoted 1 line

drivers, before calling pci_host_probe() as pm frameworks expects if the

PM framework?

Quoted 1 line

parent device supports runtime pm then it needs to enabled before child

                                 PM?
Quoted 1 line

runtime pm.

          PM?

Can any tags (like “Fixes” and “Cc”) become helpful for the proposed change?

Regards,
Markus

#4 lore raw
Quoted 21 lines · Markus Elfring

…

Quoted 1 line

PM framework expectes parent runtime pm enabled before enabling runtime

expects? PM?

Quoted 1 line

pm of the child. …

   PM?
Quoted 1 line

drivers, before calling pci_host_probe() as pm frameworks expects if the

PM framework?

Quoted 1 line

parent device supports runtime pm then it needs to enabled before child

                                  PM?
Quoted 6 lines
Quoted 1 line

runtime pm.

           PM?

Can any tags (like “Fixes” and “Cc”) become helpful for the proposed change?

Bjorn, This problem was present from starting on wards not sure what is the fix tag we need to use here, is it fine if we use fix tag as below. as at this function only we are trying add the fix.

Fixes 49b8e3f3ed1d4 ("PCI: Add generic function to probe PCI host controllers")

- Krishna Chaitanya.

Quoted 3 lines

Regards,
Markus

maintainer
#5 lore raw
In reply to Krishna chaitanya chundru
Quoted 31 lines · Krishna chaitanya chundru

The Controller driver is the parent device of the PCIe host bridge, PCI-PCI bridge and PCIe endpoint as shown below.

PCIe controller(Top level parent & parent of host bridge)
|
v
PCIe Host bridge(Parent of PCI-PCI bridge)
|
v
PCI-PCI bridge(Parent of endpoint driver)
|
v
PCIe endpoint driver

Now, when the controller device goes to runtime suspend, PM framework will check the runtime PM state of the child device (host bridge) and will find it to be disabled. So it will allow the parent (controller device) to go to runtime suspend. Only if the child device's state was 'active' it will prevent the parent to get suspended.

drivers/pci/probe.c +4 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
 		pcie_bus_configure_settings(child);
 
 	pci_bus_add_devices(bus);
+
+	/*
+	 * Ensure pm_runtime_enable() is called for the controller drivers,
+	 * before calling pci_host_probe() as pm frameworks expects if the

drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 10 lines
drivers/pci/probe.c +7 -0
+	 * parent device supports runtime pm then it needs to enabled before
+	 * child runtime pm.
+	 */
+	pm_runtime_set_active(&bridge->dev);
+	pm_runtime_no_callbacks(&bridge->dev);
+	devm_pm_runtime_enable(&bridge->dev);
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

which may have unpopulated ports (this laptop SKU does not have a modem).

Reverting dc421bb3c0db ("PCI: Enable runtime PM of the host bridge") makes the warning go away.

Johan

#regzbot introduced: dc421bb3c0db

#6 lore raw
Quoted 38 lines · Johan Hovold
Quoted 37 lines · Krishna chaitanya chundru

The Controller driver is the parent device of the PCIe host bridge, PCI-PCI bridge and PCIe endpoint as shown below.

PCIe controller(Top level parent & parent of host bridge)
|
v
PCIe Host bridge(Parent of PCI-PCI bridge)
|
v
PCI-PCI bridge(Parent of endpoint driver)
|
v
PCIe endpoint driver

Now, when the controller device goes to runtime suspend, PM framework will check the runtime PM state of the child device (host bridge) and will find it to be disabled. So it will allow the parent (controller device) to go to runtime suspend. Only if the child device's state was 'active' it will prevent the parent to get suspended.

drivers/pci/probe.c +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
  		pcie_bus_configure_settings(child);
 
  	pci_bus_add_devices(bus);
+
+	/*
+	 * Ensure pm_runtime_enable() is called for the controller drivers,
+	 * before calling pci_host_probe() as pm frameworks expects if the
+	 * parent device supports runtime pm then it needs to enabled before
+	 * child runtime pm.
+	 */
+	pm_runtime_set_active(&bridge->dev);
+	pm_runtime_no_callbacks(&bridge->dev);
+	devm_pm_runtime_enable(&bridge->dev);
Quoted 9 lines

drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
drivers/pci/probe.c +1 -0
+
  	return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Hi Johan,

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

Quoted 2 lines

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

Quoted 2 lines

which may have unpopulated ports (this laptop SKU does not have a modem).

Can you confirm if this warning goes away if there is some endpoint connected to it.

- Krishna Chaitanya.

Quoted 7 lines

Reverting dc421bb3c0db ("PCI: Enable runtime PM of the host bridge") makes the warning go away.

Johan

#regzbot introduced: dc421bb3c0db

maintainer
#7 lore raw
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
  		pcie_bus_configure_settings(child);
 
  	pci_bus_add_devices(bus);
+
+	/*
+	 * Ensure pm_runtime_enable() is called for the controller drivers,
+	 * before calling pci_host_probe() as pm frameworks expects if the
+	 * parent device supports runtime pm then it needs to enabled before
+	 * child runtime pm.
+	 */
+	pm_runtime_set_active(&bridge->dev);
+	pm_runtime_no_callbacks(&bridge->dev);
+	devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
drivers/pci/probe.c +1 -0
+
  	return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 4 lines
Quoted 1 line

which may have unpopulated ports (this laptop SKU does not have a modem).

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Johan

#8 lore raw

+ Ulf (for the runtime PM related question)

Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
  		pcie_bus_configure_settings(child);
 
  	pci_bus_add_devices(bus);
+
+	/*
+	 * Ensure pm_runtime_enable() is called for the controller drivers,
+	 * before calling pci_host_probe() as pm frameworks expects if the
+	 * parent device supports runtime pm then it needs to enabled before
+	 * child runtime pm.
+	 */
+	pm_runtime_set_active(&bridge->dev);
+	pm_runtime_no_callbacks(&bridge->dev);
+	devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
drivers/pci/probe.c +1 -0
+
  	return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

which may have unpopulated ports (this laptop SKU does not have a modem).

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.

For the sake of reference, PCI host bridge is the parent of PCI bridge.

Looking at where the warning gets triggered (in pm_runtime_enable()), we have the below checks:

dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_children
atomic_read(&dev->power.child_count) > 0

When pm_runtime_enable() gets called for PCI host bridge:

dev->power.runtime_status = RPM_SUSPENDED
dev->power.ignore_children = 0
dev->power.child_count = 1

First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.

Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.

This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I'm not sure of what the fix is in this case. But removing the DPM_FLAG_SMART_SUSPEND flag from PCI bridge driver (portdrv) makes the warning go away. This indicates that something is wrong with the DPM_FLAG_SMART_SUSPEND flag handling in PM core.

Ulf/Rafael, thoughts?

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#9 lore raw
Quoted 97 lines · Manivannan Sadhasivam
Quoted 42 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
  		pcie_bus_configure_settings(child);
 
  	pci_bus_add_devices(bus);
+
+	/*
+	 * Ensure pm_runtime_enable() is called for the controller drivers,
+	 * before calling pci_host_probe() as pm frameworks expects if the
+	 * parent device supports runtime pm then it needs to enabled before
+	 * child runtime pm.
+	 */
+	pm_runtime_set_active(&bridge->dev);
+	pm_runtime_no_callbacks(&bridge->dev);
+	devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
drivers/pci/probe.c +1 -0
+
  	return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 2 lines
Quoted 2 lines

which may have unpopulated ports (this laptop SKU does not have a modem).

Quoted 7 lines
Quoted 2 lines

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.

For the sake of reference, PCI host bridge is the parent of PCI bridge.

Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:

dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_children
atomic_read(&dev->power.child_count) > 0

When pm_runtime_enable() gets called for PCI host bridge:

dev->power.runtime_status = RPM_SUSPENDED
dev->power.ignore_children = 0
dev->power.child_count = 1

First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.

Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.

This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

Quoted 6 lines

I'm not sure of what the fix is in this case. But removing the DPM_FLAG_SMART_SUSPEND flag from PCI bridge driver (portdrv) makes the warning go away. This indicates that something is wrong with the DPM_FLAG_SMART_SUSPEND flag handling in PM core.

Ulf/Rafael, thoughts?

What's the plan for this? Does anybody have a proposal?

IIUC there is no functional issue, but the new warning must be fixed, and it would sure be nice to do it before v6.13. If there *is* a functional problem, we need to consider a revert ASAP.

Bjorn

#10 lore raw
Quoted 98 lines · Bjorn Helgaas
Quoted 97 lines · Manivannan Sadhasivam
Quoted 42 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
  		pcie_bus_configure_settings(child);
 
  	pci_bus_add_devices(bus);
+
+	/*
+	 * Ensure pm_runtime_enable() is called for the controller drivers,
+	 * before calling pci_host_probe() as pm frameworks expects if the
+	 * parent device supports runtime pm then it needs to enabled before
+	 * child runtime pm.
+	 */
+	pm_runtime_set_active(&bridge->dev);
+	pm_runtime_no_callbacks(&bridge->dev);
+	devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
drivers/pci/probe.c +1 -0
+
  	return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 2 lines
Quoted 2 lines

which may have unpopulated ports (this laptop SKU does not have a modem).

Quoted 7 lines
Quoted 2 lines

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.

For the sake of reference, PCI host bridge is the parent of PCI bridge.

Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:

dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_children
atomic_read(&dev->power.child_count) > 0

When pm_runtime_enable() gets called for PCI host bridge:

dev->power.runtime_status = RPM_SUSPENDED
dev->power.ignore_children = 0
dev->power.child_count = 1

First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.

Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.

This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

Quoted 9 lines
Quoted 6 lines

I'm not sure of what the fix is in this case. But removing the DPM_FLAG_SMART_SUSPEND flag from PCI bridge driver (portdrv) makes the warning go away. This indicates that something is wrong with the DPM_FLAG_SMART_SUSPEND flag handling in PM core.

Ulf/Rafael, thoughts?

What's the plan for this? Does anybody have a proposal?

TBH, I don't know how to fix this issue in a proper way. I need inputs from Rafael/Ulf.

Quoted 4 lines

IIUC there is no functional issue, but the new warning must be fixed, and it would sure be nice to do it before v6.13. If there *is* a functional problem, we need to consider a revert ASAP.

There is no functional problem that I'm aware of, so revert is not warranted.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
maintainer
#11 lore raw
Quoted 19 lines · Manivannan Sadhasivam
Quoted 7 lines · Bjorn Helgaas
Quoted 5 lines · Manivannan Sadhasivam
Quoted 3 lines · Johan Hovold
Quoted 3 lines
Quoted 3 lines

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

which may have unpopulated ports (this laptop SKU does not have a modem).

What's the plan for this? Does anybody have a proposal?

TBH, I don't know how to fix this issue in a proper way. I need inputs from Rafael/Ulf.

Quoted 6 lines
Quoted 4 lines

IIUC there is no functional issue, but the new warning must be fixed, and it would sure be nice to do it before v6.13. If there *is* a functional problem, we need to consider a revert ASAP.

There is no functional problem that I'm aware of, so revert is not warranted.

I'd argue for reverting the offending commit as that is the only way to make sure that the new warning is ever addressed.

Vendors unfortunately do not a have a good track record of following up and fixing issues like this.

Judging from a quick look at the code (and the commit message of the patch in question), no host controller driver depends on the commit in question as the ones that do enable runtime PM just resume unconditionally at probe() currently (i.e. effectively ignores the state of their children).

Johan

#12 lore raw
Quoted 25 lines · Johan Hovold
Quoted 24 lines · Manivannan Sadhasivam
Quoted 7 lines · Bjorn Helgaas
Quoted 5 lines · Manivannan Sadhasivam
Quoted 3 lines · Johan Hovold
Quoted 3 lines
Quoted 3 lines

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

which may have unpopulated ports (this laptop SKU does not have a modem).

What's the plan for this? Does anybody have a proposal?

TBH, I don't know how to fix this issue in a proper way. I need inputs from Rafael/Ulf.

Quoted 4 lines

IIUC there is no functional issue, but the new warning must be fixed, and it would sure be nice to do it before v6.13. If there *is* a functional problem, we need to consider a revert ASAP.

Quoted 5 lines
Quoted 1 line

There is no functional problem that I'm aware of, so revert is not warranted.

I'd argue for reverting the offending commit as that is the only way to make sure that the new warning is ever addressed.

How come reverting becomes the *only* way to address the issue? There seems to be nothing wrong with the commit in question and the same pattern in being used in other drivers as well. The issue looks to be in the PM core.

Moreover, the warning is not causing any functional issue as far as I know. So just reverting the commit that took so much effort to get merged for the sake of hiding a warning doesn't feel right to me.

Quoted 3 lines

Vendors unfortunately do not a have a good track record of following up and fixing issues like this.

I'm not relying on vendors here. I am looking into this issue and would like to fix the warning/issue on my own. That's why I shared my observation with Ulf/Rafael.

Quoted 6 lines

Judging from a quick look at the code (and the commit message of the patch in question), no host controller driver depends on the commit in question as the ones that do enable runtime PM just resume unconditionally at probe() currently (i.e. effectively ignores the state of their children).

Right. There are a couple of pieces that needs to be fixed to have the runtime PM working from top to bottom in the PCIe hierarchy. This is also one more reason why I believe that the commit wouldn't have caused any functional issue.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
maintainer
#13 lore raw
Quoted 6 lines · Manivannan Sadhasivam
Quoted 2 lines · Johan Hovold

I'd argue for reverting the offending commit as that is the only way to make sure that the new warning is ever addressed.

How come reverting becomes the *only* way to address the issue?

I didn't say it was the only way to address the issue, just that it's the only way to make sure that the new warning gets addressed. Once code is upstream, some vendors tend to lose interest.

Quoted 3 lines

There seems to
be nothing wrong with the commit in question and the same pattern in being used in other drivers as well. The issue looks to be in the PM core.

After taking a closer look now, I agree that the underlying issue seems to be in PM core.

Possibly introduced by Rafael's commit 6e176bf8d461 ("PM: sleep: core:
Do not skip callbacks in the resume phase") which moved the set_active() call from resume_noirq() to resume_early().

Quoted 3 lines

Moreover, the warning is not causing any functional issue as far as I know. So just reverting the commit that took so much effort to get merged for the sake of hiding a warning doesn't feel right to me.

My point was simply that this commit introduced a new warning in 6.13, and there is still no fix available. The code is also effectively dead, well aside from triggering the warning, and runtime suspending the host controller cannot even be tested with mainline yet (and this was unfortunately not made clear in the commit message).

The change should have been part of a series that actually enabled the functionality and not just a potential piece of it which cannot yet be tested. Also, for Qualcomm controllers, we don't even yet have proper suspend so it's probably not a good idea to throw runtime PM into the mix there just yet.

But, sure, a revert would have made more sense last week. I guess you have a few more weeks to address this now. We can always backport a revert once rc1 is out.

Johan

maintainer
#14 lore raw
Quoted 6 lines · Johan Hovold

After taking a closer look now, I agree that the underlying issue seems to be in PM core.

Possibly introduced by Rafael's commit 6e176bf8d461 ("PM: sleep: core:
Do not skip callbacks in the resume phase") which moved the set_active() call from resume_noirq() to resume_early().

Scratch the last paragraph, I misread the diff, sorry.

Johan

#15 lore raw
In reply to Johan Hovold
Quoted 33 lines · Johan Hovold
Quoted 6 lines · Manivannan Sadhasivam
Quoted 2 lines · Johan Hovold

I'd argue for reverting the offending commit as that is the only way to make sure that the new warning is ever addressed.

How come reverting becomes the *only* way to address the issue?

I didn't say it was the only way to address the issue, just that it's the only way to make sure that the new warning gets addressed. Once code is upstream, some vendors tend to lose interest.

Quoted 3 lines

There seems to
be nothing wrong with the commit in question and the same pattern in being used in other drivers as well. The issue looks to be in the PM core.

After taking a closer look now, I agree that the underlying issue seems to be in PM core.

Possibly introduced by Rafael's commit 6e176bf8d461 ("PM: sleep: core:
Do not skip callbacks in the resume phase") which moved the set_active() call from resume_noirq() to resume_early().

Quoted 3 lines

Moreover, the warning is not causing any functional issue as far as I know. So just reverting the commit that took so much effort to get merged for the sake of hiding a warning doesn't feel right to me.

My point was simply that this commit introduced a new warning in 6.13, and there is still no fix available. The code is also effectively dead, well aside from triggering the warning, and runtime suspending the host controller cannot even be tested with mainline yet (and this was unfortunately not made clear in the commit message).

Quoted 6 lines

The change should have been part of a series that actually enabled the functionality and not just a potential piece of it which cannot yet be tested. Also, for Qualcomm controllers, we don't even yet have proper suspend so it's probably not a good idea to throw runtime PM into the mix there just yet.

There are multiple pieces needed to be stitch together to enable runtime PM in the PCIe topology. And each one of them seemed to be controversial enough (due to common code etc...). So that's the reason they were sent separately. Though I must admit that the full picture and the limitation of not being able to exercise the runtime PM should've been mentioned.

Quoted 4 lines

But, sure, a revert would have made more sense last week. I guess you have a few more weeks to address this now. We can always backport a revert once rc1 is out.

Sure. I've pinged Ulf offline and he promised to look into it asap.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#16 lore raw
In reply to Manivannan Sadhasivam
Quoted 86 lines · Manivannan Sadhasivam

+ Ulf (for the runtime PM related question)

Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
                 pcie_bus_configure_settings(child);
 
         pci_bus_add_devices(bus);
+
+        /*
+         * Ensure pm_runtime_enable() is called for the controller drivers,
+         * before calling pci_host_probe() as pm frameworks expects if the
+         * parent device supports runtime pm then it needs to enabled before
+         * child runtime pm.
+         */
+        pm_runtime_set_active(&bridge->dev);
+        pm_runtime_no_callbacks(&bridge->dev);
+        devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
drivers/pci/probe.c +1 -0
+
         return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

which may have unpopulated ports (this laptop SKU does not have a modem).

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.

For the sake of reference, PCI host bridge is the parent of PCI bridge.

Looking at where the warning gets triggered (in pm_runtime_enable()), we have the below checks:

dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_children
atomic_read(&dev->power.child_count) > 0

When pm_runtime_enable() gets called for PCI host bridge:

dev->power.runtime_status = RPM_SUSPENDED
dev->power.ignore_children = 0
dev->power.child_count = 1

First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.

Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.

Quoted 6 lines

This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

Thanks for the detailed analysis, much appreciated.

So this seems to boil down to the fact that the PM core calls pm_runtime_set_active() for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.

Quoted 5 lines

I'm not sure of what the fix is in this case. But removing the DPM_FLAG_SMART_SUSPEND flag from PCI bridge driver (portdrv) makes the warning go away. This indicates that something is wrong with the DPM_FLAG_SMART_SUSPEND flag handling in PM core.

As an intermediate step, perhaps the easiest thing here is to remove the DPM_FLAG_SMART_SUSPEND flag from the PCI bridge? Ideally it should not break anything, but we probably would lose some optimizations.

Another option would be to use pm_suspend_ignore_children(), but I am not sure if that would be the correct thing to do here.

Quoted 7 lines

Ulf/Rafael, thoughts?

- Mani

--
மணிவண்ணன் சதாசிவம்

Kind regards
Uffe

#17 lore raw
Quoted 94 lines · Ulf Hansson
Quoted 91 lines · Manivannan Sadhasivam

+ Ulf (for the runtime PM related question)

Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
(quoted diff) +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
                 pcie_bus_configure_settings(child);
 
         pci_bus_add_devices(bus);
+
+        /*
+         * Ensure pm_runtime_enable() is called for the controller drivers,
+         * before calling pci_host_probe() as pm frameworks expects if the
+         * parent device supports runtime pm then it needs to enabled before
+         * child runtime pm.
+         */
+        pm_runtime_set_active(&bridge->dev);
+        pm_runtime_no_callbacks(&bridge->dev);
+        devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

Review int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
(quoted diff) +1 -0
+
         return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

which may have unpopulated ports (this laptop SKU does not have a modem).

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.

For the sake of reference, PCI host bridge is the parent of PCI bridge.

Looking at where the warning gets triggered (in pm_runtime_enable()), we have the below checks:

dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_children
atomic_read(&dev->power.child_count) > 0

When pm_runtime_enable() gets called for PCI host bridge:

dev->power.runtime_status = RPM_SUSPENDED
dev->power.ignore_children = 0
dev->power.child_count = 1

First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.

Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.

This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():

Quoted 8 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

Thanks for the detailed analysis, much appreciated.

So this seems to boil down to the fact that the PM core calls pm_runtime_set_active() for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.

Actually, not really.

The status of the child and the child count of the parent have no meaning until runtime PM is enabled for the parent. They can be manipulated freely before this happens with no consequences and all will be fine as long as those settings are consistent when runtime PM is enabled for the parent.

Now, they aren't consistent at that point because dev_pm_skip_suspend() returns false for the parent as it has DPM_FLAG_SMART_SUSPEND clear.

To me, this looks like a coding mistake because all devices that have power.must_resume set should also be set to RPM_ACTIVE before re-enabling runtime PM for them, so the attached patch should work.

---
 drivers/base/power/main.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/base/power/main.c +3 -3
@@ -656,12 +656,12 @@
 	 * so change its status accordingly.
 	 *
 	 * Otherwise, the device is going to be resumed, so set its PM-runtime
-	 * status to "active", but do that only if DPM_FLAG_SMART_SUSPEND is set
-	 * to avoid confusing drivers that don't use it.
+	 * status to "active" unless its power.must_resume flag is clear, in
+	 * which case it is not necessary to update its PM-runtime status.
 	 */
 	if (skip_resume)
 		pm_runtime_set_suspended(dev);
-	else if (dev_pm_skip_suspend(dev))
+	else if (dev->power.must_resume)
 		pm_runtime_set_active(dev);
 
 	if (dev->pm_domain) {
#18 lore raw
Quoted 113 lines · Rafael J. Wysocki
Quoted 94 lines · Ulf Hansson
Quoted 91 lines · Manivannan Sadhasivam

+ Ulf (for the runtime PM related question)

Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
(quoted diff) +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
                 pcie_bus_configure_settings(child);
 
         pci_bus_add_devices(bus);
+
+        /*
+         * Ensure pm_runtime_enable() is called for the controller drivers,
+         * before calling pci_host_probe() as pm frameworks expects if the
+         * parent device supports runtime pm then it needs to enabled before
+         * child runtime pm.
+         */
+        pm_runtime_set_active(&bridge->dev);
+        pm_runtime_no_callbacks(&bridge->dev);
+        devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

Review int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
(quoted diff) +1 -0
+
         return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

which may have unpopulated ports (this laptop SKU does not have a modem).

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.

For the sake of reference, PCI host bridge is the parent of PCI bridge.

Looking at where the warning gets triggered (in pm_runtime_enable()), we have the below checks:

dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_children
atomic_read(&dev->power.child_count) > 0

When pm_runtime_enable() gets called for PCI host bridge:

dev->power.runtime_status = RPM_SUSPENDED
dev->power.ignore_children = 0
dev->power.child_count = 1

First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.

Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.

This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():

Quoted 8 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

Thanks for the detailed analysis, much appreciated.

So this seems to boil down to the fact that the PM core calls pm_runtime_set_active() for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.

Actually, not really.

The status of the child and the child count of the parent have no meaning until runtime PM is enabled for the parent. They can be manipulated freely before this happens with no consequences and all will be fine as long as those settings are consistent when runtime PM is enabled for the parent.

Quoted 7 lines

Now, they aren't consistent at that point because dev_pm_skip_suspend() returns false for the parent as it has DPM_FLAG_SMART_SUSPEND clear.

To me, this looks like a coding mistake because all devices that have power.must_resume set should also be set to RPM_ACTIVE before re-enabling runtime PM for them, so the attached patch should work.

Having reflected on it a bit I think that it's better to avoid changing the existing behavior too much, so attached is a new version of the patch.

It is along the same lines as before, but it doesn't go as far as the previous version. Namely, in addition to what the existing code does, it will cause the runtime PM status to be set to RPM_ACTIVE for the devices whose dependents will have it set which should address the problem at hand if I'm not mistaken.

I'd appreciated giving it a go on a system where the warning is printed.

Thanks!

---
 drivers/base/power/main.c |   29 ++++++++++++++++++++---------
 include/linux/pm.h        |    1 +
 2 files changed, 21 insertions(+), 9 deletions(-)
drivers/base/power/main.c +20 -9
@@ -656,13 +656,15 @@
 	 * so change its status accordingly.
 	 *
 	 * Otherwise, the device is going to be resumed, so set its PM-runtime
-	 * status to "active", but do that only if DPM_FLAG_SMART_SUSPEND is set
-	 * to avoid confusing drivers that don't use it.
+	 * status to "active" unless its power.set_active flag is clear, in
+	 * which case it is not necessary to update its PM-runtime status.
 	 */
-	if (skip_resume)
+	if (skip_resume) {
 		pm_runtime_set_suspended(dev);
-	else if (dev_pm_skip_suspend(dev))
+	} else if (dev->power.set_active) {
 		pm_runtime_set_active(dev);
+		dev->power.set_active = false;
+	}
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -1189,18 +1191,24 @@
 	return PMSG_ON;
 }
 
-static void dpm_superior_set_must_resume(struct device *dev)
+static void dpm_superior_set_must_resume(struct device *dev, bool set_active)
 {
 	struct device_link *link;
 	int idx;
 
-	if (dev->parent)
+	if (dev->parent) {
 		dev->parent->power.must_resume = true;
+		if (set_active)
+			dev->parent->power.set_active = true;
+	}
 
 	idx = device_links_read_lock();
 
-	list_for_each_entry_rcu_locked(link, &dev->links.suppliers, c_node)
+	list_for_each_entry_rcu_locked(link, &dev->links.suppliers, c_node) {
 		link->supplier->power.must_resume = true;
+		if (set_active)
+			link->supplier->power.set_active = true;
+	}
 
 	device_links_read_unlock(idx);
 }
@@ -1278,8 +1286,11 @@
 	      dev->power.may_skip_resume))
 		dev->power.must_resume = true;
 
-	if (dev->power.must_resume)
-		dpm_superior_set_must_resume(dev);
+	if (dev->power.must_resume) {
+		dev->power.set_active = dev->power.set_active ||
+			dev_pm_test_driver_flags(dev, DPM_FLAG_SMART_SUSPEND);
+		dpm_superior_set_must_resume(dev, dev->power.set_active);
+	}
 
 Complete:
 	complete_all(&dev->power.completion);
include/linux/pm.h +1 -0
@@ -683,6 +683,7 @@
 	bool			no_pm_callbacks:1;	/* Owned by the PM core */
 	bool			async_in_progress:1;	/* Owned by the PM core */
 	bool			must_resume:1;		/* Owned by the PM core */
+	bool			set_active:1;		/* Owned by the PM core */
 	bool			may_skip_resume:1;	/* Set by subsystems */
 #else
 	bool			should_wakeup:1;
#19 lore raw
Quoted 126 lines · Rafael J. Wysocki
Quoted 113 lines · Rafael J. Wysocki
Quoted 94 lines · Ulf Hansson
Quoted 91 lines · Manivannan Sadhasivam

+ Ulf (for the runtime PM related question)

Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
(quoted diff) +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
                 pcie_bus_configure_settings(child);
 
         pci_bus_add_devices(bus);
+
+        /*
+         * Ensure pm_runtime_enable() is called for the controller drivers,
+         * before calling pci_host_probe() as pm frameworks expects if the
+         * parent device supports runtime pm then it needs to enabled before
+         * child runtime pm.
+         */
+        pm_runtime_set_active(&bridge->dev);
+        pm_runtime_no_callbacks(&bridge->dev);
+        devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

Review int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
(quoted diff) +1 -0
+
         return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

which may have unpopulated ports (this laptop SKU does not have a modem).

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.

For the sake of reference, PCI host bridge is the parent of PCI bridge.

Looking at where the warning gets triggered (in pm_runtime_enable()), we have the below checks:

dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_children
atomic_read(&dev->power.child_count) > 0

When pm_runtime_enable() gets called for PCI host bridge:

dev->power.runtime_status = RPM_SUSPENDED
dev->power.ignore_children = 0
dev->power.child_count = 1

First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.

Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.

This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():

Quoted 8 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

Thanks for the detailed analysis, much appreciated.

So this seems to boil down to the fact that the PM core calls pm_runtime_set_active() for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.

Actually, not really.

The status of the child and the child count of the parent have no meaning until runtime PM is enabled for the parent. They can be manipulated freely before this happens with no consequences and all will be fine as long as those settings are consistent when runtime PM is enabled for the parent.

Quoted 7 lines

Now, they aren't consistent at that point because dev_pm_skip_suspend() returns false for the parent as it has DPM_FLAG_SMART_SUSPEND clear.

To me, this looks like a coding mistake because all devices that have power.must_resume set should also be set to RPM_ACTIVE before re-enabling runtime PM for them, so the attached patch should work.

Having reflected on it a bit I think that it's better to avoid changing the existing behavior too much, so attached is a new version of the patch.

Quoted 8 lines

It is along the same lines as before, but it doesn't go as far as the previous version. Namely, in addition to what the existing code does, it will cause the runtime PM status to be set to RPM_ACTIVE for the devices whose dependents will have it set which should address the problem at hand if I'm not mistaken.

I'd appreciated giving it a go on a system where the warning is printed.

This patch indeed makes the warning go away and I don't spot any other issues. So you can add my Tested-by tag while submitting the fix.

Thanks a lot!

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#20 lore raw
Quoted 137 lines · Manivannan Sadhasivam
Quoted 126 lines · Rafael J. Wysocki
Quoted 113 lines · Rafael J. Wysocki
Quoted 94 lines · Ulf Hansson
Quoted 91 lines · Manivannan Sadhasivam

+ Ulf (for the runtime PM related question)

Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
(quoted diff) +10 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)
                 pcie_bus_configure_settings(child);
 
         pci_bus_add_devices(bus);
+
+        /*
+         * Ensure pm_runtime_enable() is called for the controller drivers,
+         * before calling pci_host_probe() as pm frameworks expects if the
+         * parent device supports runtime pm then it needs to enabled before
+         * child runtime pm.
+         */
+        pm_runtime_set_active(&bridge->dev);
+        pm_runtime_no_callbacks(&bridge->dev);
+        devm_pm_runtime_enable(&bridge->dev);
Quoted 8 lines

Review int pci_host_probe(struct pci_host_bridge *bridge) · line 3122

Quoted 4 lines
(quoted diff) +1 -0
+
         return 0;
  }
  EXPORT_SYMBOL_GPL(pci_host_probe);

I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:

Quoted 3 lines

Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.

No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.

Quoted 3 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

I believe this is not causing any functional issues.

It still needs to be fixed.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

which may have unpopulated ports (this laptop SKU does not have a modem).

Can you confirm if this warning goes away if there is some endpoint connected to it.

I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.

Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.

For the sake of reference, PCI host bridge is the parent of PCI bridge.

Looking at where the warning gets triggered (in pm_runtime_enable()), we have the below checks:

dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_children
atomic_read(&dev->power.child_count) > 0

When pm_runtime_enable() gets called for PCI host bridge:

dev->power.runtime_status = RPM_SUSPENDED
dev->power.ignore_children = 0
dev->power.child_count = 1

First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.

Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.

This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():

Quoted 8 lines
Quoted 1 line

pci0004:00: pcie4: Enabling runtime PM for inactive device with active children

Thanks for the detailed analysis, much appreciated.

So this seems to boil down to the fact that the PM core calls pm_runtime_set_active() for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.

Actually, not really.

The status of the child and the child count of the parent have no meaning until runtime PM is enabled for the parent. They can be manipulated freely before this happens with no consequences and all will be fine as long as those settings are consistent when runtime PM is enabled for the parent.

Quoted 7 lines

Now, they aren't consistent at that point because dev_pm_skip_suspend() returns false for the parent as it has DPM_FLAG_SMART_SUSPEND clear.

To me, this looks like a coding mistake because all devices that have power.must_resume set should also be set to RPM_ACTIVE before re-enabling runtime PM for them, so the attached patch should work.

Having reflected on it a bit I think that it's better to avoid changing the existing behavior too much, so attached is a new version of the patch.

Quoted 8 lines

It is along the same lines as before, but it doesn't go as far as the previous version. Namely, in addition to what the existing code does, it will cause the runtime PM status to be set to RPM_ACTIVE for the devices whose dependents will have it set which should address the problem at hand if I'm not mistaken.

I'd appreciated giving it a go on a system where the warning is printed.

Quoted 6 lines

This patch indeed makes the warning go away and I don't spot any other issues. So you can add my Tested-by tag while submitting the fix.

Thanks a lot!

Thank you!

#21 lore raw
In reply to Krishna chaitanya chundru
Quoted 26 lines · Krishna chaitanya chundru

It is a property of the runtime PM framework that it can only follow continuous dependency chains. That is, if there is a device with runtime PM disabled in a dependency chain, runtime PM cannot be enabled for devices below it and above it in that chain both at the same time.

Since runtime PM is disabled for host bridge, the state of the child devices under the host bridge is not taken into account by PM framework for the top level parent, PCIe controller. So PM framework, allows the controller driver to enter runtime PM irrespective of the state of the devices under the host bridge.

So enable runtime pm of the host bridge device, so that dependency chain in maintained between endpoint devices and the controller driver.

PM framework expectes parent runtime pm enabled before enabling runtime pm of the child. As PCIe starfive device is enabling runtime pm after the pci_host_probe which enables runtime pm of the child device i.e for the bridge device a warning is shown saying "pcie-starfive 940000000.pcie:
Enabling runtime PM for inactive device with active children" and also shows possible circular locking dependency detected message.

As it is must to enable parent device's runtime PM before enabling child's runtime pm as the pcie-starfive device runtime pm is enabled after child runtime starfive device is seeing the warning.

Quoted 8 lines

In the first patch fix the pcie-starfive driver by enabling runtime pm before calling pci_host_probe().

All other PCIe controller drivers are enabling runtime pm before calling pci_host_probe() which is as expected so don't require any fix like pcie-starfive driver.

Provisionally applied to pci/pm for v6.13. I see a typo or two, so the commit logs will likely be updated, but I pushed the branch to get build testing started.

Quoted 43 lines

---
Changes in v7:
- Bring the cover letter details to commit message as suggested by bjorn.
- Add a comment in the driver to ensure pm_runtime_enable is called
before host bridge pm_runtime_enable().
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- include the patch by mayank which fixes runtime pm enable order for
pcie starfive driver.
Changes in v5:
- call pm_runtime_no_callbacks() as suggested by Rafael.
- include the commit texts as suggested by Rafael.
- Link to v4: https://lore.kernel.org/linux-pci/[email protected]/
Changes in v4:
- Changed pm_runtime_enable() to devm_pm_runtime_enable() (suggested by mayank)
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes in v3:
- Moved the runtime API call's from the dwc driver to PCI framework
as it is applicable for all (suggested by mani)
- Updated the commit message.
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Updated commit message as suggested by mani.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Krishna chaitanya chundru (1):
PCI: Enable runtime pm of the host bridge

Mayank Rana (1):
PCI: starfive: Enable PCIe controller's runtime PM before probing host bridge

 drivers/pci/controller/plda/pcie-starfive.c | 10 +++++++---
 drivers/pci/probe.c                         | 11 +++++++++++
 2 files changed, 18 insertions(+), 3 deletions(-)

---
base-commit: 2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2
change-id: 20241016-runtime_pm-d3dbf41736b6

Best regards,

Signature · 1 line
Krishna chaitanya chundru <[email protected]>