PCI/AER: Prevent runtime power management during recovery
Subsystems (2)
PATCH 1/1 PCI/AER: Prevent runtime power management during recovery
A bridge that supports D3 but not hotplug will be subject to runtime power management placing it in a non-operation power state if it doesn't have any devices attached. This patch will prevent this power management during error recovery so that the rescan at the end may be successful.
--- drivers/pci/pcie/err.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/pci/pcie/err.c +3 -0
@@ -16,6 +16,7 @@#include <linux/kernel.h> #include <linux/errno.h> #include <linux/aer.h> +#include <linux/pm_runtime.h> #include "portdrv.h" #include "../pci.h"
@@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) udev = dev->bus->self; parent = udev->subordinate; +pm_runtime_forbid (&udev->dev);pci_lock_rescan_remove ();list_for_each_entry_safe_reverse (pdev, temp, &parent->devices, bus_list) {
@@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) }pci_unlock_rescan_remove (); +pm_runtime_allow (&udev->dev); }/**
Signature · 1 line
2.14.3
Quoted 4 lines · Keith Busch
A bridge that supports D3 but not hotplug will be subject to runtime power management placing it in a non-operation power state if it doesn't have any devices attached. This patch will prevent this power management during error recovery so that the rescan at the end may be successful.
If there is no card connected, why would the bridge observe a fatal error?
Signature · 3 lines
Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
Quoted 7 lines · Sinan Kaya
Quoted 4 lines · Keith Busch
A bridge that supports D3 but not hotplug will be subject to runtime power management placing it in a non-operation power state if it doesn't have any devices attached. This patch will prevent this power management during error recovery so that the rescan at the end may be successful.
If there is no card connected, why would the bridge observe a fatal error?
fatal error could be coming from anywhere let us say below RootPort.. and RP observes it and decides to take the tree down.... Is that the case Keith is talking about ?
Why will re-enumeration be a problem even if runtime PM is active ? I assume that enumeration will get bridge out of D3.
Quoted 9 lines · On Tue, Jun 12, 2018 at 01:21:53PM +0530, [email protected] wrote:
Quoted 8 lines · Sinan Kaya
Quoted 6 lines · Keith Busch
A bridge that supports D3 but not hotplug will be subject to runtime power management placing it in a non-operation power state if it doesn't
have any devices attached. This patch will prevent this power management
during error recovery so that the rescan at the end may be successful.
Quoted 6 lines
Quoted 2 lines
If there is no card connected, why would the bridge observe a fatal error?
fatal error could be coming from anywhere let us say below RootPort.. and RP observes it and decides to take the tree down.... Is that the case Keith is talking about ?
Right, the err fatal handling removes all the devices below a bridge, making that bridge allowed for run time d3.
Quoted 2 lines
Why will re-enumeration be a problem even if runtime PM is active ? I assume that enumeration will get bridge out of D3.
That doesn't seem to be the case.
[email protected]
Reviewed-byQuoted 17 lines · Keith Busch
Quoted 9 lines · On Tue, Jun 12, 2018 at 01:21:53PM +0530, [email protected] wrote:
Quoted 8 lines · Sinan Kaya
Quoted 6 lines · Keith Busch
A bridge that supports D3 but not hotplug will be subject to runtime power management placing it in a non-operation power state if it doesn't
have any devices attached. This patch will prevent this power management
during error recovery so that the rescan at the end may be successful.Quoted 6 lines
Quoted 2 lines
If there is no card connected, why would the bridge observe a fatal error?
fatal error could be coming from anywhere let us say below RootPort.. and RP observes it and decides to take the tree down.... Is that the case Keith is talking about ?
Quoted 7 lines
Right, the err fatal handling removes all the devices below a bridge, making that bridge allowed for run time d3.
Quoted 2 lines
Why will re-enumeration be a problem even if runtime PM is active ? I assume that enumeration will get bridge out of D3.
That doesn't seem to be the case.
reset_link(udev, service); will initiate SBR, and which should do hot reset the bridge and bring it out of D3 ! but it seems even SBR is not doing it for you.
Although I am not sure if the EP has to be designed to reset its config space upon SBR.. even if they are, some just might not do it all correctly !
The code looks okay to me anyway.
[+cc Rafael, linux-pm for a PM perspective]
Quoted 4 lines · Keith Busch
A bridge that supports D3 but not hotplug will be subject to runtime power management placing it in a non-operation power state if it doesn't have any devices attached. This patch will prevent this power management during error recovery so that the rescan at the end may be successful.
I assume you're seeing a problem that's fixed by this patch?
I'm not very familiar with the pm_runtime_*() interfaces. The fact that drivers/pci/ only contains two calls to pm_runtime_forbid() and two to pm_runtime_allow() (one in a quirk) makes me wonder if this is the right approach.
Do we need something similar in other enumeration paths, e.g., boot-time enumeration, hot-add, sysfs "rescan" entry, etc?
Quoted 37 lines
Cc Oza Pawandeep <[email protected]>Signed-off-by Keith Busch <[email protected]>--- drivers/pci/pcie/err.c | 3 +++ 1 file changed, 3 insertions(+)drivers/pci/pcie/err.c +3
-0@@ -16,6 +16,7 @@#include <linux/kernel.h> #include <linux/errno.h> #include <linux/aer.h> +#include <linux/pm_runtime.h> #include "portdrv.h" #include "../pci.h" @@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) udev = dev->bus->self; parent = udev->subordinate; +pm_runtime_forbid (&udev->dev);pci_lock_rescan_remove ();list_for_each_entry_safe_reverse (pdev, temp, &parent->devices, bus_list) {@@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) }pci_unlock_rescan_remove (); +pm_runtime_allow (&udev->dev); }/** Signature · 1 line
2.14.3
Quoted 6 lines · Bjorn Helgaas
Quoted 4 lines · Keith Busch
A bridge that supports D3 but not hotplug will be subject to runtime power management placing it in a non-operation power state if it doesn't have any devices attached. This patch will prevent this power management during error recovery so that the rescan at the end may be successful.
Quoted 6 lines
I assume you're seeing a problem that's fixed by this patch?
I'm not very familiar with the pm_runtime_*() interfaces. The fact that
drivers/pci/only contains two calls topm_runtime_forbid()and two topm_runtime_allow()(one in a quirk) makes me wonder if this is the right approach.
The use of pm_runtime_forbid() / pm_runtime_allow() is wrong here. The user may forbid and allow runtime PM on a device by writing "on" or "auto" to its power/control file in sysfs. The two functions have the same effect, but the user may override them at any time. If the intention is to ensure the bridge is runtime resumed while a rescan is in progess, the appropriate functions to call are pm_runtime_get_sync() and pm_runtime_put().
Quoted 2 lines
Do we need something similar in other enumeration paths, e.g., boot-time enumeration, hot-add, sysfs "rescan" entry, etc?
Definitely not on boot-time enumeration. PCI devices are kept in D0 if not bound to a driver, and the PCI core first scans all devices, only then attaches drivers.
As for hot-add, hotplug bridges are currently always kept in D0, but I've posted a patch series two weeks ago to add runtime PM. Did you get a chance to look at it? Patch [28/32] of that series resumes the bridge to D0 on hot-add ("PCI: pciehp: Resume to D0 on enable/disable").
As for sysfs "rescan", this does lack runtime PM calls currently, as do many other sysfs accessors. The above-mentioned series adds runtime PM to sysfs "reset" in patch [30/32] ("PCI: sysfs:
Resume to D0 on function reset"). The patch does not depend on the preceding patches in the series, it could be applied out of order. Amending all the other accessors would be another big project. It's not as trivial as sprinkling a few runtime PM calls, it requires careful thought. E.g. if only config space is accessed, it's sufficient to resume the parent with pci_config_pm_runtime_get() instead of the device itself with pm_runtime_get_sync(). In other cases the information is read from config space even though it's cached, e.g. secondary_bus_number_show(). In those cases it might be better to always return the cached data instead of adding runtime PM calls.
FWIW, from a quick look proc.c seems fine as far as runtime PM is concerned.
Thanks,
Lukas
Quoted 37 lines
Quoted 37 lines
Cc Oza Pawandeep <[email protected]>Signed-off-by Keith Busch <[email protected]>--- drivers/pci/pcie/err.c | 3 +++ 1 file changed, 3 insertions(+)drivers/pci/pcie/err.c +3
-0@@ -16,6 +16,7 @@#include <linux/kernel.h> #include <linux/errno.h> #include <linux/aer.h> +#include <linux/pm_runtime.h> #include "portdrv.h" #include "../pci.h" @@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) udev = dev->bus->self; parent = udev->subordinate; +pm_runtime_forbid (&udev->dev);pci_lock_rescan_remove ();list_for_each_entry_safe_reverse (pdev, temp, &parent->devices, bus_list) {@@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) }pci_unlock_rescan_remove (); +pm_runtime_allow (&udev->dev); }/** Signature · 1 line
2.14.3