PCI: Enable runtime pm of the host bridge
Subsystems (8)
v7 cover PCI: Enable runtime pm of the host bridge
It is a property of the runtime PM framework that it can only follow continuous dependency chains. That is, if there is a device with runtime PM disabled in a dependency chain, runtime PM cannot be enabled for devices below it and above it in that chain both at the same time.
Since runtime PM is disabled for host bridge, the state of the child devices under the host bridge is not taken into account by PM framework for the top level parent, PCIe controller. So PM framework, allows the controller driver to enter runtime PM irrespective of the state of the devices under the host bridge.
So enable runtime pm of the host bridge device, so that dependency chain in maintained between endpoint devices and the controller driver.
PM framework expectes parent runtime pm enabled before enabling runtime pm of the child. As PCIe starfive device is enabling runtime pm after the pci_host_probe which enables runtime pm of the child device i.e for the bridge device a warning is shown saying "pcie-starfive 940000000.pcie:
Enabling runtime PM for inactive device with active children" and also shows possible circular locking dependency detected message.
As it is must to enable parent device's runtime PM before enabling child's runtime pm as the pcie-starfive device runtime pm is enabled after child runtime starfive device is seeing the warning.
In the first patch fix the pcie-starfive driver by enabling runtime pm before calling pci_host_probe().
All other PCIe controller drivers are enabling runtime pm before calling pci_host_probe() which is as expected so don't require any fix like pcie-starfive driver.
---
Changes in v7:
- Bring the cover letter details to commit message as suggested by bjorn.
- Add a comment in the driver to ensure pm_runtime_enable is called
before host bridge pm_runtime_enable().
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- include the patch by mayank which fixes runtime pm enable order for
pcie starfive driver.
Changes in v5:
- call pm_runtime_no_callbacks() as suggested by Rafael.
- include the commit texts as suggested by Rafael.
- Link to v4: https://lore.kernel.org/linux-pci/[email protected]/
Changes in v4:
- Changed pm_runtime_enable() to devm_pm_runtime_enable() (suggested by mayank)
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes in v3:
- Moved the runtime API call's from the dwc driver to PCI framework
as it is applicable for all (suggested by mani)
- Updated the commit message.
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Updated commit message as suggested by mani.
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Krishna chaitanya chundru (1):
PCI: Enable runtime pm of the host bridge
Mayank Rana (1):
PCI: starfive: Enable PCIe controller's runtime PM before probing host bridge
drivers/pci/controller/plda/pcie-starfive.c | 10 +++++++--- drivers/pci/probe.c | 11 +++++++++++ 2 files changed, 18 insertions(+), 3 deletions(-)
---
base-commit: 2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2
change-id: 20241016-runtime_pm-d3dbf41736b6
Best regards,
Signature · 1 line
Krishna chaitanya chundru <[email protected]>
PATCH v7 1/2 PCI: starfive: Enable PCIe controller's runtime PM before probing host bridge
From: Mayank Rana <[email protected]>
PCIe controller device (i.e. PCIe starfive device) is parent to PCIe host bridge device. To enable runtime PM of PCIe host bridge device (child device), it is must to enable parent device's runtime PM to avoid seeing the below warning from PM core:
pcie-starfive 940000000.pcie: Enabling runtime PM for inactive device with active children
Fix this issue by enabling starfive pcie controller device's runtime PM before calling pci_host_probe() in plda_pcie_host_init().
---v3->v6:
- no change
Link to v3: https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/v2->v3:
- Update commit description based on Mani's feedback
- Updated Reviewed-by tag
Link to v2: https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
v1->v2: Updated commit description based on Bjorn's feedback
Link to v1: https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
--- drivers/pci/controller/plda/pcie-starfive.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/pci/controller/plda/pcie-starfive.c +7 -3
@@ -404,6 +404,9 @@ static int starfive_pcie_probe(struct platform_device *pdev)if (ret)return ret; +pm_runtime_enable (&pdev->dev); +pm_runtime_get_sync (&pdev->dev); + plda->host_ops = &sf_host_ops; plda->num_events =PLDA_MAX_EVENT_NUM ;/* mask doorbell event */
@@ -413,11 +416,12 @@ static int starfive_pcie_probe(struct platform_device *pdev) plda->events_bitmap <<=PLDA_NUM_DMA_EVENTS ; ret =plda_pcie_host_init (&pcie->plda, &starfive_pcie_ops, &stf_pcie_event); -if (ret) +if (ret) { +pm_runtime_put_sync (&pdev->dev); +pm_runtime_disable (&pdev->dev);return ret; + } -pm_runtime_enable (&pdev->dev); -pm_runtime_get_sync (&pdev->dev);platform_set_drvdata (pdev, pcie);return 0 ;
Signature · 1 line
2.34.1
PATCH v7 2/2 PCI: Enable runtime pm of the host bridge
In reply to Krishna chaitanya chundruThe Controller driver is the parent device of the PCIe host bridge, PCI-PCI bridge and PCIe endpoint as shown below.
PCIe controller(Top level parent & parent of host bridge)
|
v
PCIe Host bridge(Parent of PCI-PCI bridge)
|
v
PCI-PCI bridge(Parent of endpoint driver)
|
v
PCIe endpoint driver
Now, when the controller device goes to runtime suspend, PM framework will check the runtime PM state of the child device (host bridge) and will find it to be disabled. So it will allow the parent (controller device) to go to runtime suspend. Only if the child device's state was 'active' it will prevent the parent to get suspended.
It is a property of the runtime PM framework that it can only follow continuous dependency chains. That is, if there is a device with runtime PM disabled in a dependency chain, runtime PM cannot be enabled for devices below it and above it in that chain both at the same time.
Since runtime PM is disabled for host bridge, the state of the child devices under the host bridge is not taken into account by PM framework for the top level parent, PCIe controller. So PM framework, allows the controller driver to enter runtime PM irrespective of the state of the devices under the host bridge. And this causes the topology breakage and also possible PM issues like controller driver goes to runtime suspend while endpoint driver is doing some transfers.
Because of the above, in order to enable runtime PM for a PCIe controller device, one needs to ensure that runtime PM is enabled for all devices in every dependency chain between it and any PCIe endpoint (as runtime PM is enabled for PCIe endpoints).
This means that runtime PM needs to be enabled for the host bridge device, which is present in all of these dependency chains.
After this change, the host bridge device will be runtime-suspended by the runtime PM framework automatically after suspending its last child and it will be runtime-resumed automatically before resuming its first child which will allow the runtime PM framework to track dependencies between the host bridge device and all of its descendants.
PM framework expectes parent runtime pm enabled before enabling runtime pm of the child. Ensure pm_runtime_enable() is called for the controller drivers, before calling pci_host_probe() as pm frameworks expects if the parent device supports runtime pm then it needs to enabled before child runtime pm.
---
Changes in v7:
- update commit text & and add a comment in the driver as suggested by bjorn. Changes in v6:
- no change
Changes in v5:
- call pm_runtime_no_callbacks() as suggested by Rafael.
- include the commit texts as suggested by Rafael.
- Link to v4: https://lore.kernel.org/linux-pci/[email protected]/
Changes in v4:
- Changed pm_runtime_enable() to devm_pm_runtime_enable() (suggested by mayank)
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes in v3:
- Moved the runtime API call's from the dwc driver to PCI framework
as it is applicable for all (suggested by mani)
- Updated the commit message.
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Updated commit message as suggested by mani.
- Link to v1: https://lore.kernel.org/r/[email protected]
--- drivers/pci/probe.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/pci/probe.c +11 -0
@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)pcie_bus_configure_settings (child);pci_bus_add_devices (bus); + +/* +* Ensure pm_runtime_enable() is called for the controller drivers, +* before calling pci_host_probe() as pm frameworks expects if the +* parent device supports runtime pm then it needs to enabled before +* child runtime pm. +*/ +pm_runtime_set_active (&bridge->dev); +pm_runtime_no_callbacks (&bridge->dev); +devm_pm_runtime_enable (&bridge->dev); +return 0 ; }EXPORT_SYMBOL_GPL (pci_host_probe);
Signature · 1 line
2.34.1
…
Quoted 1 line
PM framework expectes parent runtime pm enabled before enabling runtime
expects? PM?
Quoted 1 line
pm of the child. …
PM?
Quoted 1 line
drivers, before calling
pci_host_probe()as pm frameworks expects if the
PM framework?
Quoted 1 line
parent device supports runtime pm then it needs to enabled before child
PM?
Quoted 1 line
runtime pm.
PM?
Can any tags (like “Fixes” and “Cc”) become helpful for the proposed change?
Regards,
Markus
Quoted 21 lines · Markus Elfring
…
Quoted 1 line
PM framework expectes parent runtime pm enabled before enabling runtime
expects? PM?
Quoted 1 line
pm of the child. …
PM?Quoted 1 line
drivers, before calling
pci_host_probe()as pm frameworks expects if thePM framework?
Quoted 1 line
parent device supports runtime pm then it needs to enabled before child
PM?
Quoted 6 lines
Quoted 1 line
runtime pm.
PM?Can any tags (like “Fixes” and “Cc”) become helpful for the proposed change?
Bjorn, This problem was present from starting on wards not sure what is the fix tag we need to use here, is it fine if we use fix tag as below. as at this function only we are trying add the fix.
- Krishna Chaitanya.
Quoted 3 lines
Regards,
Markus
Johan Hovold
maintainerQuoted 31 lines · Krishna chaitanya chundru
The Controller driver is the parent device of the PCIe host bridge, PCI-PCI bridge and PCIe endpoint as shown below.
PCIe
controller(Top level parent & parent of host bridge)
|
v
PCIe Host bridge(Parent of PCI-PCI bridge)
|
v
PCI-PCI bridge(Parent of endpoint driver)
|
v
PCIe endpoint driverNow, when the controller device goes to runtime suspend, PM framework will check the runtime PM state of the child device (host bridge) and will find it to be disabled. So it will allow the parent (controller device) to go to runtime suspend. Only if the child device's state was 'active' it will prevent the parent to get suspended.
Reviewed-by Manivannan Sadhasivam <[email protected]>Signed-off-by Krishna chaitanya chundru <[email protected]>drivers/pci/probe.c +4
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)pcie_bus_configure_settings (child);pci_bus_add_devices (bus); + +/* +* Ensure pm_runtime_enable() is called for the controller drivers, +* before calling pci_host_probe() as pm frameworks expects if the
drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 10 lines
drivers/pci/probe.c +7
-0+* parent device supports runtime pm then it needs to enabled before +* child runtime pm. +*/ +pm_runtime_set_active (&bridge->dev); +pm_runtime_no_callbacks (&bridge->dev); +devm_pm_runtime_enable (&bridge->dev); +return 0 ; }EXPORT_SYMBOL_GPL (pci_host_probe);
I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
which may have unpopulated ports (this laptop SKU does not have a modem).
Reverting dc421bb3c0db ("PCI: Enable runtime PM of the host bridge") makes the warning go away.
Johan
#regzbot introduced: dc421bb3c0db
Quoted 38 lines · Johan Hovold
Quoted 37 lines · Krishna chaitanya chundru
The Controller driver is the parent device of the PCIe host bridge, PCI-PCI bridge and PCIe endpoint as shown below.
PCIe
controller(Top level parent & parent of host bridge)
|
v
PCIe Host bridge(Parent of PCI-PCI bridge)
|
v
PCI-PCI bridge(Parent of endpoint driver)
|
v
PCIe endpoint driverNow, when the controller device goes to runtime suspend, PM framework will check the runtime PM state of the child device (host bridge) and will find it to be disabled. So it will allow the parent (controller device) to go to runtime suspend. Only if the child device's state was 'active' it will prevent the parent to get suspended.
Reviewed-by Manivannan Sadhasivam <[email protected]>Signed-off-by Krishna chaitanya chundru <[email protected]>drivers/pci/probe.c +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)pcie_bus_configure_settings (child);pci_bus_add_devices (bus); + +/* +* Ensure pm_runtime_enable() is called for the controller drivers, +* before calling pci_host_probe() as pm frameworks expects if the +* parent device supports runtime pm then it needs to enabled before +* child runtime pm. +*/ +pm_runtime_set_active (&bridge->dev); +pm_runtime_no_callbacks (&bridge->dev); +devm_pm_runtime_enable (&bridge->dev);
Quoted 9 lines
drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
drivers/pci/probe.c +1
-0+return 0 ; }EXPORT_SYMBOL_GPL (pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Hi Johan,
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
Quoted 2 lines
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
Quoted 2 lines
which may have unpopulated ports (this laptop SKU does not have a modem).
Can you confirm if this warning goes away if there is some endpoint connected to it.
- Krishna Chaitanya.
Quoted 7 lines
Reverting
dc421bb3c0db("PCI: Enable runtime PM of the host bridge") makes the warning go away.Johan
#regzbot introduced:
dc421bb3c0db
Johan Hovold
maintainerQuoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)pcie_bus_configure_settings (child);pci_bus_add_devices (bus); + +/* +* Ensure pm_runtime_enable() is called for the controller drivers, +* before calling pci_host_probe() as pm frameworks expects if the +* parent device supports runtime pm then it needs to enabled before +* child runtime pm. +*/ +pm_runtime_set_active (&bridge->dev); +pm_runtime_no_callbacks (&bridge->dev); +devm_pm_runtime_enable (&bridge->dev);Quoted 8 lines
drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
drivers/pci/probe.c +1
-0+return 0 ; }EXPORT_SYMBOL_GPL (pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 4 lines
Quoted 1 line
which may have unpopulated ports (this laptop SKU does not have a modem).
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Johan
+ Ulf (for the runtime PM related question)
Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)pcie_bus_configure_settings (child);pci_bus_add_devices (bus); + +/* +* Ensure pm_runtime_enable() is called for the controller drivers, +* before calling pci_host_probe() as pm frameworks expects if the +* parent device supports runtime pm then it needs to enabled before +* child runtime pm. +*/ +pm_runtime_set_active (&bridge->dev); +pm_runtime_no_callbacks (&bridge->dev); +devm_pm_runtime_enable (&bridge->dev);Quoted 8 lines
drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
drivers/pci/probe.c +1
-0+return 0 ; }EXPORT_SYMBOL_GPL (pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
which may have unpopulated ports (this laptop SKU does not have a modem).
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.
For the sake of reference, PCI host bridge is the parent of PCI bridge.
Looking at where the warning gets triggered (in pm_runtime_enable()), we have the below checks:
dev->power.runtime_status == RPM_SUSPENDED
!dev->power.ignore_childrenatomic_read(&dev->power.child_count) > 0
When pm_runtime_enable() gets called for PCI host bridge:
dev->power.runtime_status = RPM_SUSPENDEDdev->power.ignore_children = 0dev->power.child_count = 1
First 2 passes seem legit, but the issue is with the 3rd one. Here, the child_count of 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to be RPM_SUSPENDED as the resume process should first resume the parent (PCI host bridge). But this is not the case here.
Then looking at where the child_count gets incremented, it leads to pm_runtime_set_active() of device_resume_noirq(). pm_runtime_set_active() is only called for a device if dev_pm_skip_suspend() succeeds, which requires DPM_FLAG_SMART_SUSPEND flag to be set and the device to be runtime suspended.
This criteria matches for PCI bridge. So its status was set to 'RPM_ACTIVE' even though the parent PCI host bridge was still in the RPM_SUSPENDED state. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge when pm_runtime_enable() is called from device_resume_early():
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I'm not sure of what the fix is in this case. But removing the DPM_FLAG_SMART_SUSPEND flag from PCI bridge driver (portdrv) makes the warning go away. This indicates that something is wrong with the DPM_FLAG_SMART_SUSPEND flag handling in PM core.
Ulf/Rafael, thoughts?
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 97 lines · Manivannan Sadhasivam
Quoted 42 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)pcie_bus_configure_settings (child);pci_bus_add_devices (bus); + +/* +* Ensure pm_runtime_enable() is called for the controller drivers, +* before calling pci_host_probe() as pm frameworks expects if the +* parent device supports runtime pm then it needs to enabled before +* child runtime pm. +*/ +pm_runtime_set_active (&bridge->dev); +pm_runtime_no_callbacks (&bridge->dev); +devm_pm_runtime_enable (&bridge->dev);Quoted 8 lines
drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
drivers/pci/probe.c +1
-0+return 0 ; }EXPORT_SYMBOL_GPL (pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 2 lines
Quoted 2 lines
which may have unpopulated ports (this laptop SKU does not have a modem).
Quoted 7 lines
Quoted 2 lines
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.
For the sake of reference, PCI host bridge is the parent of PCI bridge.
Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:
dev->power.runtime_status==RPM_SUSPENDED
!dev->power.ignore_childrenatomic_read(&dev->power.child_count)> 0When
pm_runtime_enable()gets called for PCI host bridge:
dev->power.runtime_status=RPM_SUSPENDEDdev->power.ignore_children= 0dev->power.child_count= 1First 2 passes seem legit, but the issue is with the 3rd one. Here, the
child_countof 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to beRPM_SUSPENDEDas the resume process should first resume the parent (PCI host bridge). But this is not the case here.Then looking at where the
child_countgets incremented, it leads topm_runtime_set_active()ofdevice_resume_noirq().pm_runtime_set_active()is only called for a device ifdev_pm_skip_suspend()succeeds, which requiresDPM_FLAG_SMART_SUSPENDflag to be set and the device to be runtime suspended.This criteria matches for PCI bridge. So its status was set to '
RPM_ACTIVE' even though the parent PCI host bridge was still in theRPM_SUSPENDEDstate. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge whenpm_runtime_enable()is called fromdevice_resume_early():pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
Quoted 6 lines
I'm not sure of what the fix is in this case. But removing the
DPM_FLAG_SMART_SUSPENDflag from PCI bridge driver (portdrv) makes the warning go away. This indicates that something is wrong with theDPM_FLAG_SMART_SUSPENDflag handling in PM core.Ulf/Rafael, thoughts?
What's the plan for this? Does anybody have a proposal?
IIUC there is no functional issue, but the new warning must be fixed, and it would sure be nice to do it before v6.13. If there *is* a functional problem, we need to consider a revert ASAP.
Bjorn
Quoted 98 lines · Bjorn Helgaas
Quoted 97 lines · Manivannan Sadhasivam
Quoted 42 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)pcie_bus_configure_settings (child);pci_bus_add_devices (bus); + +/* +* Ensure pm_runtime_enable() is called for the controller drivers, +* before calling pci_host_probe() as pm frameworks expects if the +* parent device supports runtime pm then it needs to enabled before +* child runtime pm. +*/ +pm_runtime_set_active (&bridge->dev); +pm_runtime_no_callbacks (&bridge->dev); +devm_pm_runtime_enable (&bridge->dev);Quoted 8 lines
drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
drivers/pci/probe.c +1
-0+return 0 ; }EXPORT_SYMBOL_GPL (pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 2 lines
Quoted 2 lines
which may have unpopulated ports (this laptop SKU does not have a modem).
Quoted 7 lines
Quoted 2 lines
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.
For the sake of reference, PCI host bridge is the parent of PCI bridge.
Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:
dev->power.runtime_status==RPM_SUSPENDED
!dev->power.ignore_childrenatomic_read(&dev->power.child_count)> 0When
pm_runtime_enable()gets called for PCI host bridge:
dev->power.runtime_status=RPM_SUSPENDEDdev->power.ignore_children= 0dev->power.child_count= 1First 2 passes seem legit, but the issue is with the 3rd one. Here, the
child_countof 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to beRPM_SUSPENDEDas the resume process should first resume the parent (PCI host bridge). But this is not the case here.Then looking at where the
child_countgets incremented, it leads topm_runtime_set_active()ofdevice_resume_noirq().pm_runtime_set_active()is only called for a device ifdev_pm_skip_suspend()succeeds, which requiresDPM_FLAG_SMART_SUSPENDflag to be set and the device to be runtime suspended.This criteria matches for PCI bridge. So its status was set to '
RPM_ACTIVE' even though the parent PCI host bridge was still in theRPM_SUSPENDEDstate. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge whenpm_runtime_enable()is called fromdevice_resume_early():pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
Quoted 9 lines
Quoted 6 lines
I'm not sure of what the fix is in this case. But removing the
DPM_FLAG_SMART_SUSPENDflag from PCI bridge driver (portdrv) makes the warning go away. This indicates that something is wrong with theDPM_FLAG_SMART_SUSPENDflag handling in PM core.Ulf/Rafael, thoughts?
What's the plan for this? Does anybody have a proposal?
TBH, I don't know how to fix this issue in a proper way. I need inputs from Rafael/Ulf.
Quoted 4 lines
IIUC there is no functional issue, but the new warning must be fixed, and it would sure be nice to do it before v6.13. If there *is* a functional problem, we need to consider a revert ASAP.
There is no functional problem that I'm aware of, so revert is not warranted.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Johan Hovold
maintainerQuoted 19 lines · Manivannan Sadhasivam
Quoted 7 lines · Bjorn Helgaas
Quoted 5 lines · Manivannan Sadhasivam
Quoted 3 lines · Johan Hovold
Quoted 3 lines
Quoted 3 lines
I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
which may have unpopulated ports (this laptop SKU does not have a modem).
What's the plan for this? Does anybody have a proposal?
TBH, I don't know how to fix this issue in a proper way. I need inputs from Rafael/Ulf.
Quoted 6 lines
Quoted 4 lines
IIUC there is no functional issue, but the new warning must be fixed, and it would sure be nice to do it before v6.13. If there *is* a functional problem, we need to consider a revert ASAP.
There is no functional problem that I'm aware of, so revert is not warranted.
I'd argue for reverting the offending commit as that is the only way to make sure that the new warning is ever addressed.
Vendors unfortunately do not a have a good track record of following up and fixing issues like this.
Judging from a quick look at the code (and the commit message of the patch in question), no host controller driver depends on the commit in question as the ones that do enable runtime PM just resume unconditionally at probe() currently (i.e. effectively ignores the state of their children).
Johan
Quoted 25 lines · Johan Hovold
Quoted 24 lines · Manivannan Sadhasivam
Quoted 7 lines · Bjorn Helgaas
Quoted 5 lines · Manivannan Sadhasivam
Quoted 3 lines · Johan Hovold
Quoted 3 lines
Quoted 3 lines
I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
which may have unpopulated ports (this laptop SKU does not have a modem).
What's the plan for this? Does anybody have a proposal?
TBH, I don't know how to fix this issue in a proper way. I need inputs from Rafael/Ulf.
Quoted 4 lines
IIUC there is no functional issue, but the new warning must be fixed, and it would sure be nice to do it before v6.13. If there *is* a functional problem, we need to consider a revert ASAP.
Quoted 5 lines
Quoted 1 line
There is no functional problem that I'm aware of, so revert is not warranted.
I'd argue for reverting the offending commit as that is the only way to make sure that the new warning is ever addressed.
How come reverting becomes the *only* way to address the issue? There seems to be nothing wrong with the commit in question and the same pattern in being used in other drivers as well. The issue looks to be in the PM core.
Moreover, the warning is not causing any functional issue as far as I know. So just reverting the commit that took so much effort to get merged for the sake of hiding a warning doesn't feel right to me.
Quoted 3 lines
Vendors unfortunately do not a have a good track record of following up and fixing issues like this.
I'm not relying on vendors here. I am looking into this issue and would like to fix the warning/issue on my own. That's why I shared my observation with Ulf/Rafael.
Quoted 6 lines
Judging from a quick look at the code (and the commit message of the patch in question), no host controller driver depends on the commit in question as the ones that do enable runtime PM just resume unconditionally at
probe()currently (i.e. effectively ignores the state of their children).
Right. There are a couple of pieces that needs to be fixed to have the runtime PM working from top to bottom in the PCIe hierarchy. This is also one more reason why I believe that the commit wouldn't have caused any functional issue.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Johan Hovold
maintainerQuoted 6 lines · Manivannan Sadhasivam
Quoted 2 lines · Johan Hovold
I'd argue for reverting the offending commit as that is the only way to make sure that the new warning is ever addressed.
How come reverting becomes the *only* way to address the issue?
I didn't say it was the only way to address the issue, just that it's the only way to make sure that the new warning gets addressed. Once code is upstream, some vendors tend to lose interest.
Quoted 3 lines
There seems to
be nothing wrong with the commit in question and the same pattern in being used in other drivers as well. The issue looks to be in the PM core.
After taking a closer look now, I agree that the underlying issue seems to be in PM core.
Possibly introduced by Rafael's commit 6e176bf8d461 ("PM: sleep: core:
Do not skip callbacks in the resume phase") which moved the set_active() call from resume_noirq() to resume_early().
Quoted 3 lines
Moreover, the warning is not causing any functional issue as far as I know. So just reverting the commit that took so much effort to get merged for the sake of hiding a warning doesn't feel right to me.
My point was simply that this commit introduced a new warning in 6.13, and there is still no fix available. The code is also effectively dead, well aside from triggering the warning, and runtime suspending the host controller cannot even be tested with mainline yet (and this was unfortunately not made clear in the commit message).
The change should have been part of a series that actually enabled the functionality and not just a potential piece of it which cannot yet be tested. Also, for Qualcomm controllers, we don't even yet have proper suspend so it's probably not a good idea to throw runtime PM into the mix there just yet.
But, sure, a revert would have made more sense last week. I guess you have a few more weeks to address this now. We can always backport a revert once rc1 is out.
Johan
Johan Hovold
maintainerQuoted 6 lines · Johan Hovold
After taking a closer look now, I agree that the underlying issue seems to be in PM core.
Possibly introduced by Rafael's commit
6e176bf8d461("PM: sleep: core:
Do not skip callbacks in the resume phase") which moved theset_active()call fromresume_noirq()toresume_early().
Scratch the last paragraph, I misread the diff, sorry.
Johan
Quoted 33 lines · Johan Hovold
Quoted 6 lines · Manivannan Sadhasivam
Quoted 2 lines · Johan Hovold
I'd argue for reverting the offending commit as that is the only way to make sure that the new warning is ever addressed.
How come reverting becomes the *only* way to address the issue?
I didn't say it was the only way to address the issue, just that it's the only way to make sure that the new warning gets addressed. Once code is upstream, some vendors tend to lose interest.
Quoted 3 lines
There seems to
be nothing wrong with the commit in question and the same pattern in being used in other drivers as well. The issue looks to be in the PM core.After taking a closer look now, I agree that the underlying issue seems to be in PM core.
Possibly introduced by Rafael's commit
6e176bf8d461("PM: sleep: core:
Do not skip callbacks in the resume phase") which moved theset_active()call fromresume_noirq()toresume_early().Quoted 3 lines
Moreover, the warning is not causing any functional issue as far as I know. So just reverting the commit that took so much effort to get merged for the sake of hiding a warning doesn't feel right to me.
My point was simply that this commit introduced a new warning in 6.13, and there is still no fix available. The code is also effectively dead, well aside from triggering the warning, and runtime suspending the host controller cannot even be tested with mainline yet (and this was unfortunately not made clear in the commit message).
Quoted 6 lines
The change should have been part of a series that actually enabled the functionality and not just a potential piece of it which cannot yet be tested. Also, for Qualcomm controllers, we don't even yet have proper suspend so it's probably not a good idea to throw runtime PM into the mix there just yet.
There are multiple pieces needed to be stitch together to enable runtime PM in the PCIe topology. And each one of them seemed to be controversial enough (due to common code etc...). So that's the reason they were sent separately. Though I must admit that the full picture and the limitation of not being able to exercise the runtime PM should've been mentioned.
Quoted 4 lines
But, sure, a revert would have made more sense last week. I guess you have a few more weeks to address this now. We can always backport a revert once rc1 is out.
Sure. I've pinged Ulf offline and he promised to look into it asap.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 86 lines · Manivannan Sadhasivam
+ Ulf (for the runtime PM related question)
Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
drivers/pci/probe.c +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge)pcie_bus_configure_settings (child);pci_bus_add_devices (bus); + +/* +* Ensure pm_runtime_enable() is called for the controller drivers, +* before calling pci_host_probe() as pm frameworks expects if the +* parent device supports runtime pm then it needs to enabled before +* child runtime pm. +*/ +pm_runtime_set_active (&bridge->dev); +pm_runtime_no_callbacks (&bridge->dev); +devm_pm_runtime_enable (&bridge->dev);Quoted 8 lines
drivers/pci/probe.c int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
drivers/pci/probe.c +1
-0+return 0 ; }EXPORT_SYMBOL_GPL (pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
which may have unpopulated ports (this laptop SKU does not have a modem).
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.
For the sake of reference, PCI host bridge is the parent of PCI bridge.
Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:
dev->power.runtime_status==RPM_SUSPENDED
!dev->power.ignore_childrenatomic_read(&dev->power.child_count)> 0When
pm_runtime_enable()gets called for PCI host bridge:
dev->power.runtime_status=RPM_SUSPENDEDdev->power.ignore_children= 0dev->power.child_count= 1First 2 passes seem legit, but the issue is with the 3rd one. Here, the
child_countof 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to beRPM_SUSPENDEDas the resume process should first resume the parent (PCI host bridge). But this is not the case here.Then looking at where the
child_countgets incremented, it leads topm_runtime_set_active()ofdevice_resume_noirq().pm_runtime_set_active()is only called for a device ifdev_pm_skip_suspend()succeeds, which requiresDPM_FLAG_SMART_SUSPENDflag to be set and the device to be runtime suspended.
Quoted 6 lines
This criteria matches for PCI bridge. So its status was set to '
RPM_ACTIVE' even though the parent PCI host bridge was still in theRPM_SUSPENDEDstate. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge whenpm_runtime_enable()is called fromdevice_resume_early():pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
Thanks for the detailed analysis, much appreciated.
So this seems to boil down to the fact that the PM core calls pm_runtime_set_active() for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.
Quoted 5 lines
I'm not sure of what the fix is in this case. But removing the
DPM_FLAG_SMART_SUSPENDflag from PCI bridge driver (portdrv) makes the warning go away. This indicates that something is wrong with theDPM_FLAG_SMART_SUSPENDflag handling in PM core.
As an intermediate step, perhaps the easiest thing here is to remove the DPM_FLAG_SMART_SUSPEND flag from the PCI bridge? Ideally it should not break anything, but we probably would lose some optimizations.
Another option would be to use pm_suspend_ignore_children(), but I am not sure if that would be the correct thing to do here.
Quoted 7 lines
Ulf/Rafael, thoughts?
- Mani
--
மணிவண்ணன் சதாசிவம்
Kind regards
Uffe
Rafael J. Wysocki
maintainerQuoted 94 lines · Ulf Hansson
Quoted 91 lines · Manivannan Sadhasivam
+ Ulf (for the runtime PM related question)
Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
(quoted diff) +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge) pcie_bus_configure_settings(child); pci_bus_add_devices(bus); + + /* + * Ensure pm_runtime_enable() is called for the controller drivers, + * before calling pci_host_probe() as pm frameworks expects if the + * parent device supports runtime pm then it needs to enabled before + * child runtime pm. + */ + pm_runtime_set_active(&bridge->dev); + pm_runtime_no_callbacks(&bridge->dev); + devm_pm_runtime_enable(&bridge->dev);Quoted 8 lines
Review int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
(quoted diff) +1
-0+ return 0; } EXPORT_SYMBOL_GPL(pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
which may have unpopulated ports (this laptop SKU does not have a modem).
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.
For the sake of reference, PCI host bridge is the parent of PCI bridge.
Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:
dev->power.runtime_status==RPM_SUSPENDED
!dev->power.ignore_childrenatomic_read(&dev->power.child_count)> 0When
pm_runtime_enable()gets called for PCI host bridge:
dev->power.runtime_status=RPM_SUSPENDEDdev->power.ignore_children= 0dev->power.child_count= 1First 2 passes seem legit, but the issue is with the 3rd one. Here, the
child_countof 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to beRPM_SUSPENDEDas the resume process should first resume the parent (PCI host bridge). But this is not the case here.Then looking at where the
child_countgets incremented, it leads topm_runtime_set_active()ofdevice_resume_noirq().pm_runtime_set_active()is only called for a device ifdev_pm_skip_suspend()succeeds, which requiresDPM_FLAG_SMART_SUSPENDflag to be set and the device to be runtime suspended.This criteria matches for PCI bridge. So its status was set to '
RPM_ACTIVE' even though the parent PCI host bridge was still in theRPM_SUSPENDEDstate. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge whenpm_runtime_enable()is called fromdevice_resume_early():
Quoted 8 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
Thanks for the detailed analysis, much appreciated.
So this seems to boil down to the fact that the PM core calls
pm_runtime_set_active()for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.
Actually, not really.
The status of the child and the child count of the parent have no meaning until runtime PM is enabled for the parent. They can be manipulated freely before this happens with no consequences and all will be fine as long as those settings are consistent when runtime PM is enabled for the parent.
Now, they aren't consistent at that point because dev_pm_skip_suspend() returns false for the parent as it has DPM_FLAG_SMART_SUSPEND clear.
To me, this looks like a coding mistake because all devices that have power.must_resume set should also be set to RPM_ACTIVE before re-enabling runtime PM for them, so the attached patch should work.
--- drivers/base/power/main.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/base/power/main.c +3 -3
@@ -656,12 +656,12 @@* so change its status accordingly. * * Otherwise, the device is going to be resumed, so set its PM-runtime -* status to "active", but do that only if DPM_FLAG_SMART_SUSPEND is set -* to avoid confusing drivers that don't use it. +* status to "active" unless its power.must_resume flag is clear, in +* which case it is not necessary to update its PM-runtime status. */ if (skip_resume)pm_runtime_set_suspended (dev); -else if (dev_pm_skip_suspend (dev)) +else if (dev->power.must_resume)pm_runtime_set_active (dev);if (dev->pm_domain) {
Rafael J. Wysocki
maintainerQuoted 113 lines · Rafael J. Wysocki
Quoted 94 lines · Ulf Hansson
Quoted 91 lines · Manivannan Sadhasivam
+ Ulf (for the runtime PM related question)
Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
(quoted diff) +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge) pcie_bus_configure_settings(child); pci_bus_add_devices(bus); + + /* + * Ensure pm_runtime_enable() is called for the controller drivers, + * before calling pci_host_probe() as pm frameworks expects if the + * parent device supports runtime pm then it needs to enabled before + * child runtime pm. + */ + pm_runtime_set_active(&bridge->dev); + pm_runtime_no_callbacks(&bridge->dev); + devm_pm_runtime_enable(&bridge->dev);Quoted 8 lines
Review int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
(quoted diff) +1
-0+ return 0; } EXPORT_SYMBOL_GPL(pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
which may have unpopulated ports (this laptop SKU does not have a modem).
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.
For the sake of reference, PCI host bridge is the parent of PCI bridge.
Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:
dev->power.runtime_status==RPM_SUSPENDED
!dev->power.ignore_childrenatomic_read(&dev->power.child_count)> 0When
pm_runtime_enable()gets called for PCI host bridge:
dev->power.runtime_status=RPM_SUSPENDEDdev->power.ignore_children= 0dev->power.child_count= 1First 2 passes seem legit, but the issue is with the 3rd one. Here, the
child_countof 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to beRPM_SUSPENDEDas the resume process should first resume the parent (PCI host bridge). But this is not the case here.Then looking at where the
child_countgets incremented, it leads topm_runtime_set_active()ofdevice_resume_noirq().pm_runtime_set_active()is only called for a device ifdev_pm_skip_suspend()succeeds, which requiresDPM_FLAG_SMART_SUSPENDflag to be set and the device to be runtime suspended.This criteria matches for PCI bridge. So its status was set to '
RPM_ACTIVE' even though the parent PCI host bridge was still in theRPM_SUSPENDEDstate. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge whenpm_runtime_enable()is called fromdevice_resume_early():Quoted 8 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
Thanks for the detailed analysis, much appreciated.
So this seems to boil down to the fact that the PM core calls
pm_runtime_set_active()for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.Actually, not really.
The status of the child and the child count of the parent have no meaning until runtime PM is enabled for the parent. They can be manipulated freely before this happens with no consequences and all will be fine as long as those settings are consistent when runtime PM is enabled for the parent.
Quoted 7 lines
Now, they aren't consistent at that point because
dev_pm_skip_suspend()returns false for the parent as it hasDPM_FLAG_SMART_SUSPENDclear.To me, this looks like a coding mistake because all devices that have power.must_resume set should also be set to
RPM_ACTIVEbefore re-enabling runtime PM for them, so the attached patch should work.
Having reflected on it a bit I think that it's better to avoid changing the existing behavior too much, so attached is a new version of the patch.
It is along the same lines as before, but it doesn't go as far as the previous version. Namely, in addition to what the existing code does, it will cause the runtime PM status to be set to RPM_ACTIVE for the devices whose dependents will have it set which should address the problem at hand if I'm not mistaken.
I'd appreciated giving it a go on a system where the warning is printed.
Thanks!
--- drivers/base/power/main.c | 29 ++++++++++++++++++++--------- include/linux/pm.h | 1 + 2 files changed, 21 insertions(+), 9 deletions(-)
drivers/base/power/main.c +20 -9
@@ -656,13 +656,15 @@* so change its status accordingly. * * Otherwise, the device is going to be resumed, so set its PM-runtime -* status to "active", but do that only if DPM_FLAG_SMART_SUSPEND is set -* to avoid confusing drivers that don't use it. +* status to "active" unless its power.set_active flag is clear, in +* which case it is not necessary to update its PM-runtime status. */ -if (skip_resume) +if (skip_resume) {pm_runtime_set_suspended (dev); -else if (dev_pm_skip_suspend (dev)) + }else if (dev->power.set_active) {pm_runtime_set_active (dev); + dev->power.set_active = false; + }if (dev->pm_domain) { info ="noirq power domain " ;
@@ -1189,18 +1191,24 @@return PMSG_ON ; } -static void dpm_superior_set_must_resume (struct device *dev) +static void dpm_superior_set_must_resume (struct device *dev,bool set_active) {struct device_link *link;int idx; -if (dev->parent) +if (dev->parent) { dev->parent->power.must_resume = true; +if (set_active) + dev->parent->power.set_active = true; + } idx =device_links_read_lock (); -list_for_each_entry_rcu_locked (link, &dev->links.suppliers, c_node) +list_for_each_entry_rcu_locked (link, &dev->links.suppliers, c_node) { link->supplier->power.must_resume = true; +if (set_active) + link->supplier->power.set_active = true; + }device_links_read_unlock (idx); }
@@ -1278,8 +1286,11 @@ dev->power.may_skip_resume)) dev->power.must_resume = true; -if (dev->power.must_resume) -dpm_superior_set_must_resume (dev); +if (dev->power.must_resume) { + dev->power.set_active = dev->power.set_active || +dev_pm_test_driver_flags (dev,DPM_FLAG_SMART_SUSPEND ); +dpm_superior_set_must_resume (dev, dev->power.set_active); + } Complete:complete_all (&dev->power.completion);
include/linux/pm.h +1 -0
@@ -683,6 +683,7 @@ bool no_pm_callbacks:1; /* Owned by the PM core */ bool async_in_progress:1; /* Owned by the PM core */ bool must_resume:1; /* Owned by the PM core */ + bool set_active:1; /* Owned by the PM core */ bool may_skip_resume:1; /* Set by subsystems */#else bool should_wakeup:1;
Manivannan Sadhasivam
Tested-byQuoted 126 lines · Rafael J. Wysocki
Quoted 113 lines · Rafael J. Wysocki
Quoted 94 lines · Ulf Hansson
Quoted 91 lines · Manivannan Sadhasivam
+ Ulf (for the runtime PM related question)
Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
(quoted diff) +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge) pcie_bus_configure_settings(child); pci_bus_add_devices(bus); + + /* + * Ensure pm_runtime_enable() is called for the controller drivers, + * before calling pci_host_probe() as pm frameworks expects if the + * parent device supports runtime pm then it needs to enabled before + * child runtime pm. + */ + pm_runtime_set_active(&bridge->dev); + pm_runtime_no_callbacks(&bridge->dev); + devm_pm_runtime_enable(&bridge->dev);Quoted 8 lines
Review int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
(quoted diff) +1
-0+ return 0; } EXPORT_SYMBOL_GPL(pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
which may have unpopulated ports (this laptop SKU does not have a modem).
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.
For the sake of reference, PCI host bridge is the parent of PCI bridge.
Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:
dev->power.runtime_status==RPM_SUSPENDED
!dev->power.ignore_childrenatomic_read(&dev->power.child_count)> 0When
pm_runtime_enable()gets called for PCI host bridge:
dev->power.runtime_status=RPM_SUSPENDEDdev->power.ignore_children= 0dev->power.child_count= 1First 2 passes seem legit, but the issue is with the 3rd one. Here, the
child_countof 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to beRPM_SUSPENDEDas the resume process should first resume the parent (PCI host bridge). But this is not the case here.Then looking at where the
child_countgets incremented, it leads topm_runtime_set_active()ofdevice_resume_noirq().pm_runtime_set_active()is only called for a device ifdev_pm_skip_suspend()succeeds, which requiresDPM_FLAG_SMART_SUSPENDflag to be set and the device to be runtime suspended.This criteria matches for PCI bridge. So its status was set to '
RPM_ACTIVE' even though the parent PCI host bridge was still in theRPM_SUSPENDEDstate. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge whenpm_runtime_enable()is called fromdevice_resume_early():Quoted 8 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
Thanks for the detailed analysis, much appreciated.
So this seems to boil down to the fact that the PM core calls
pm_runtime_set_active()for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.Actually, not really.
The status of the child and the child count of the parent have no meaning until runtime PM is enabled for the parent. They can be manipulated freely before this happens with no consequences and all will be fine as long as those settings are consistent when runtime PM is enabled for the parent.
Quoted 7 lines
Now, they aren't consistent at that point because
dev_pm_skip_suspend()returns false for the parent as it hasDPM_FLAG_SMART_SUSPENDclear.To me, this looks like a coding mistake because all devices that have power.must_resume set should also be set to
RPM_ACTIVEbefore re-enabling runtime PM for them, so the attached patch should work.Having reflected on it a bit I think that it's better to avoid changing the existing behavior too much, so attached is a new version of the patch.
Quoted 8 lines
It is along the same lines as before, but it doesn't go as far as the previous version. Namely, in addition to what the existing code does, it will cause the runtime PM status to be set to
RPM_ACTIVEfor the devices whose dependents will have it set which should address the problem at hand if I'm not mistaken.I'd appreciated giving it a go on a system where the warning is printed.
This patch indeed makes the warning go away and I don't spot any other issues. So you can add my Tested-by tag while submitting the fix.
Thanks a lot!
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Rafael J. Wysocki
maintainerQuoted 137 lines · Manivannan Sadhasivam
Quoted 126 lines · Rafael J. Wysocki
Quoted 113 lines · Rafael J. Wysocki
Quoted 94 lines · Ulf Hansson
Quoted 91 lines · Manivannan Sadhasivam
+ Ulf (for the runtime PM related question)
Quoted 39 lines · Johan Hovold
Quoted 25 lines · Krishna Chaitanya Chundru
Quoted 14 lines · Johan Hovold
Quoted 14 lines
(quoted diff) +10
-0@@ -3106,6 +3106,17 @@ int pci_host_probe(struct pci_host_bridge *bridge) pcie_bus_configure_settings(child); pci_bus_add_devices(bus); + + /* + * Ensure pm_runtime_enable() is called for the controller drivers, + * before calling pci_host_probe() as pm frameworks expects if the + * parent device supports runtime pm then it needs to enabled before + * child runtime pm. + */ + pm_runtime_set_active(&bridge->dev); + pm_runtime_no_callbacks(&bridge->dev); + devm_pm_runtime_enable(&bridge->dev);Quoted 8 lines
Review int pci_host_probe(struct pci_host_bridge *bridge) · line 3122
Quoted 4 lines
(quoted diff) +1
-0+ return 0; } EXPORT_SYMBOL_GPL(pci_host_probe);I just noticed that this change in 6.13-rc1 is causing the following warning on resume from suspend on machines like the Lenovo ThinkPad X13s:
Quoted 3 lines
Can you confirm if you are seeing this issue is seen in the boot-up case also. As this part of the code executes only at the boot time and will not have effect in resume from suspend.
No, I only see it during resume. And enabling runtime PM can (and in this case, obviously does) impact system suspend as well.
Quoted 3 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
I believe this is not causing any functional issues.
It still needs to be fixed.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
which may have unpopulated ports (this laptop SKU does not have a modem).
Can you confirm if this warning goes away if there is some endpoint connected to it.
I don't have anything to connect to the slot in this machine, but this seems to be the case as I do not see this warning for the populated slots, nor on the CRD reference design which has a modem on PCIe4.
Yes, this is only happening for unpopulated slots and the warning shows up only if runtime PM is enabled for both PCI bridge and host bridge. This patch enables the runtime PM for host bridge and if the PCI bridge runtime PM is also enabled (only happens now for ACPI/BIOS based platforms), then the warning shows up only if the PCI bridge was RPM suspended (mostly happens if there was no device connected) during the system wide resume time.
For the sake of reference, PCI host bridge is the parent of PCI bridge.
Looking at where the warning gets triggered (in
pm_runtime_enable()), we have the below checks:
dev->power.runtime_status==RPM_SUSPENDED
!dev->power.ignore_childrenatomic_read(&dev->power.child_count)> 0When
pm_runtime_enable()gets called for PCI host bridge:
dev->power.runtime_status=RPM_SUSPENDEDdev->power.ignore_children= 0dev->power.child_count= 1First 2 passes seem legit, but the issue is with the 3rd one. Here, the
child_countof 1 means that the PCI host bridge has an 'active' child (which is the PCI bridge). The PCI bridge was supposed to beRPM_SUSPENDEDas the resume process should first resume the parent (PCI host bridge). But this is not the case here.Then looking at where the
child_countgets incremented, it leads topm_runtime_set_active()ofdevice_resume_noirq().pm_runtime_set_active()is only called for a device ifdev_pm_skip_suspend()succeeds, which requiresDPM_FLAG_SMART_SUSPENDflag to be set and the device to be runtime suspended.This criteria matches for PCI bridge. So its status was set to '
RPM_ACTIVE' even though the parent PCI host bridge was still in theRPM_SUSPENDEDstate. I don't think this is a valid condition as seen from the warning triggered for PCI host bridge whenpm_runtime_enable()is called fromdevice_resume_early():Quoted 8 lines
Quoted 1 line
pci0004:00: pcie4: Enabling runtime PM for inactive device with active children
Thanks for the detailed analysis, much appreciated.
So this seems to boil down to the fact that the PM core calls
pm_runtime_set_active()for a device, when it really should not. If there is a clever way to avoid that, I think we need Rafael's opinion on.Actually, not really.
The status of the child and the child count of the parent have no meaning until runtime PM is enabled for the parent. They can be manipulated freely before this happens with no consequences and all will be fine as long as those settings are consistent when runtime PM is enabled for the parent.
Quoted 7 lines
Now, they aren't consistent at that point because
dev_pm_skip_suspend()returns false for the parent as it hasDPM_FLAG_SMART_SUSPENDclear.To me, this looks like a coding mistake because all devices that have power.must_resume set should also be set to
RPM_ACTIVEbefore re-enabling runtime PM for them, so the attached patch should work.Having reflected on it a bit I think that it's better to avoid changing the existing behavior too much, so attached is a new version of the patch.
Quoted 8 lines
It is along the same lines as before, but it doesn't go as far as the previous version. Namely, in addition to what the existing code does, it will cause the runtime PM status to be set to
RPM_ACTIVEfor the devices whose dependents will have it set which should address the problem at hand if I'm not mistaken.I'd appreciated giving it a go on a system where the warning is printed.
Quoted 6 lines
This patch indeed makes the warning go away and I don't spot any other issues. So you can add my Tested-by tag while submitting the fix.
Thanks a lot!
Thank you!
Quoted 26 lines · Krishna chaitanya chundru
It is a property of the runtime PM framework that it can only follow continuous dependency chains. That is, if there is a device with runtime PM disabled in a dependency chain, runtime PM cannot be enabled for devices below it and above it in that chain both at the same time.
Since runtime PM is disabled for host bridge, the state of the child devices under the host bridge is not taken into account by PM framework for the top level parent, PCIe controller. So PM framework, allows the controller driver to enter runtime PM irrespective of the state of the devices under the host bridge.
So enable runtime pm of the host bridge device, so that dependency chain in maintained between endpoint devices and the controller driver.
PM framework expectes parent runtime pm enabled before enabling runtime pm of the child. As PCIe starfive device is enabling runtime pm after the
pci_host_probewhich enables runtime pm of the child device i.e for the bridge device a warning is shown saying "pcie-starfive 940000000.pcie:
Enabling runtime PM for inactive device with active children" and also shows possible circular locking dependency detected message.As it is must to enable parent device's runtime PM before enabling child's runtime pm as the pcie-starfive device runtime pm is enabled after child runtime starfive device is seeing the warning.
Quoted 8 lines
In the first patch fix the pcie-starfive driver by enabling runtime pm before calling
pci_host_probe().All other PCIe controller drivers are enabling runtime pm before calling
pci_host_probe()which is as expected so don't require any fix like pcie-starfive driver.Signed-off-by Krishna chaitanya chundru <[email protected]>
Provisionally applied to pci/pm for v6.13. I see a typo or two, so the commit logs will likely be updated, but I pushed the branch to get build testing started.
Quoted 43 lines
---
Changes in v7:
- Bring the cover letter details to commit message as suggested by bjorn.
- Add a comment in the driver to ensurepm_runtime_enableis called
before host bridgepm_runtime_enable().
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- include the patch by mayank which fixes runtime pm enable order for
pcie starfive driver.
Changes in v5:
- callpm_runtime_no_callbacks()as suggested by Rafael.
- include the commit texts as suggested by Rafael.
- Link to v4: https://lore.kernel.org/linux-pci/[email protected]/
Changes in v4:
- Changedpm_runtime_enable()todevm_pm_runtime_enable()(suggested by mayank)
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes in v3:
- Moved the runtime API call's from the dwc driver to PCI framework
as it is applicable for all (suggested by mani)
- Updated the commit message.
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Updated commit message as suggested by mani.
- Link to v1: https://lore.kernel.org/r/[email protected]---
Krishna chaitanya chundru (1):
PCI: Enable runtime pm of the host bridgeMayank Rana (1):
PCI: starfive: Enable PCIe controller's runtime PM before probing host bridgedrivers/pci/controller/plda/pcie-starfive.c | 10 +++++++--- drivers/pci/probe.c | 11 +++++++++++ 2 files changed, 18 insertions(+), 3 deletions(-)---
base-commit:2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2
change-id: 20241016-runtime_pm-d3dbf41736b6Best regards,
Signature · 1 line
Krishna chaitanya chundru <[email protected]>