Skip to contentSkip to conversation

PCI/portdrv: Don't disable pci device during shutdown

RFC PATCH In review Huacai Chen 2 participants 3 messages Aug 1, 2020 – Sep 11, 2020 PCI

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 PCI/portdrv: Don't disable pci device during shutdown

Use separate remove()/shutdown() callback, and don't disable pci device during shutdown. This can avoid some poweroff/reboot failures.

The poweroff/reboot failures can easily reproduce on Loongson platforms. I think this is not a Loongson-specific problem, instead, is a problem related to some specific PCI hosts. On some x86 platforms, radeon/amdgpu devices can cause the same problem, and commit faefba95c9e8ca3a523831c2e ("drm/amdgpu: just suspend the hw on pci shutdown") can resolve it.

Radeon driver is more difficult than amdgpu due to its confusing symbol names, and I have maintained an out-of-tree patch for a long time [1]. Recently, we found more and more devices can cause the same problem, and it is very difficult to modify all problematic drivers as radeon/amdgpu does. So, I think modify the PCIe port driver is a simple and effective way.

[1] https://github.com/chenhuacai/linux/commit/6612f9c1fc290d42a14618ce9a7d03014d8ebb1a

---
 drivers/pci/pcie/portdrv.h      |  2 +-
 drivers/pci/pcie/portdrv_core.c |  6 ++++--
 drivers/pci/pcie/portdrv_pci.c  | 15 +++++++++++++--
 3 files changed, 18 insertions(+), 5 deletions(-)
drivers/pci/pcie/portdrv.h +1 -1
index af7cf23..22ba7b9 100644
@@ -123,7 +123,7 @@ int pcie_port_device_resume(struct device *dev);
 int pcie_port_device_runtime_suspend(struct device *dev);
 int pcie_port_device_runtime_resume(struct device *dev);
 #endif
-void pcie_port_device_remove(struct pci_dev *dev);
+void pcie_port_device_remove(struct pci_dev *dev, bool disable);
 int __must_check pcie_port_bus_register(void);
 void pcie_port_bus_unregister(void);
 
drivers/pci/pcie/portdrv_core.c +4 -2
index 50a9522..aa165be 100644
@@ -487,11 +487,13 @@ EXPORT_SYMBOL_GPL(pcie_port_find_device);
  * Remove PCI Express port service devices associated with given port and
  * disable MSI-X or MSI for the port.
  */
-void pcie_port_device_remove(struct pci_dev *dev)
+void pcie_port_device_remove(struct pci_dev *dev, bool disable)
 {
 	device_for_each_child(&dev->dev, NULL, remove_iter);
 	pci_free_irq_vectors(dev);
-	pci_disable_device(dev);
+
+	if (disable)
+		pci_disable_device(dev);
 }
 
 /**
drivers/pci/pcie/portdrv_pci.c +13 -2
index 3acf151..fcd3d2e 100644
@@ -142,7 +142,18 @@ static void pcie_portdrv_remove(struct pci_dev *dev)
 		pm_runtime_dont_use_autosuspend(&dev->dev);
 	}
 
-	pcie_port_device_remove(dev);
+	pcie_port_device_remove(dev, true);
+}
+
+static void pcie_portdrv_shutdown(struct pci_dev *dev)
+{
+	if (pci_bridge_d3_possible(dev)) {
+		pm_runtime_forbid(&dev->dev);
+		pm_runtime_get_noresume(&dev->dev);
+		pm_runtime_dont_use_autosuspend(&dev->dev);
+	}
+
+	pcie_port_device_remove(dev, false);
 }
 
 static pci_ers_result_t pcie_portdrv_error_detected(struct pci_dev *dev,
@@ -211,7 +222,7 @@ static struct pci_driver pcie_portdriver = {
 
 	.probe		= pcie_portdrv_probe,
 	.remove		= pcie_portdrv_remove,
-	.shutdown	= pcie_portdrv_remove,
+	.shutdown	= pcie_portdrv_shutdown,
 
 	.err_handler	= &pcie_portdrv_err_handler,
 
Signature · 1 line
2.7.0
#1 lore raw

[+cc Tiezhu]

Quoted 9 lines · Huacai Chen

Use separate remove()/shutdown() callback, and don't disable pci device during shutdown. This can avoid some poweroff/reboot failures.

The poweroff/reboot failures can easily reproduce on Loongson platforms. I think this is not a Loongson-specific problem, instead, is a problem related to some specific PCI hosts. On some x86 platforms, radeon/amdgpu devices can cause the same problem, and commit faefba95c9e8ca3a523831c2e ("drm/amdgpu: just suspend the hw on pci shutdown") can resolve it.

Quoted 6 lines

Radeon driver is more difficult than amdgpu due to its confusing symbol names, and I have maintained an out-of-tree patch for a long time [1]. Recently, we found more and more devices can cause the same problem, and it is very difficult to modify all problematic drivers as radeon/amdgpu does. So, I think modify the PCIe port driver is a simple and effective way.

Sounds plausible, but I don't know what the actual problem is, other than "poweroff/reboot doesn't work, and this patch 'fixes' it".

Quoted 30 lines

[1] https://github.com/chenhuacai/linux/commit/6612f9c1fc290d42a14618ce9a7d03014d8ebb1a

---
 drivers/pci/pcie/portdrv.h      |  2 +-
 drivers/pci/pcie/portdrv_core.c |  6 ++++--
 drivers/pci/pcie/portdrv_pci.c  | 15 +++++++++++++--
 3 files changed, 18 insertions(+), 5 deletions(-)
drivers/pci/pcie/portdrv.h +1 -1
index af7cf23..22ba7b9 100644
@@ -123,7 +123,7 @@ int pcie_port_device_resume(struct device *dev);
 int pcie_port_device_runtime_suspend(struct device *dev);
 int pcie_port_device_runtime_resume(struct device *dev);
 #endif
-void pcie_port_device_remove(struct pci_dev *dev);
+void pcie_port_device_remove(struct pci_dev *dev, bool disable);
 int __must_check pcie_port_bus_register(void);
 void pcie_port_bus_unregister(void);
 
drivers/pci/pcie/portdrv_core.c +0 -0
index 50a9522..aa165be 100644
@@ -487,11 +487,13 @@ EXPORT_SYMBOL_GPL(pcie_port_find_device);
  * Remove PCI Express port service devices associated with given port and
  * disable MSI-X or MSI for the port.
  */

drivers/pci/pcie/portdrv_core.c EXPORT_SYMBOL_GPL(pcie_port_find_device); · line 497

Quoted 10 lines
drivers/pci/pcie/portdrv_core.c +4 -2
-void pcie_port_device_remove(struct pci_dev *dev)
+void pcie_port_device_remove(struct pci_dev *dev, bool disable)
 {
 	device_for_each_child(&dev->dev, NULL, remove_iter);
 	pci_free_irq_vectors(dev);
-	pci_disable_device(dev);
+
+	if (disable)
+		pci_disable_device(dev);
 }

portdrv can only be built statically (not as a module), and I don't think there's value in being able to unbind it. So I think we should remove its .remove() method. That would be a separate patch, of course, but it should make this one quite a bit simpler.

Quoted 10 lines
drivers/pci/pcie/portdrv_core.c +0 -0
 /**
drivers/pci/pcie/portdrv_pci.c +0 -1
index 3acf151..fcd3d2e 100644
@@ -142,7 +142,18 @@ static void pcie_portdrv_remove(struct pci_dev *dev)
 		pm_runtime_dont_use_autosuspend(&dev->dev);
 	}
 
-	pcie_port_device_remove(dev);

drivers/pci/pcie/portdrv_pci.c static void pcie_portdrv_remove(struct pci_dev *dev) · line 154

Quoted 10 lines
drivers/pci/pcie/portdrv_pci.c +10 -0
+	pcie_port_device_remove(dev, true);
+}
+
+static void pcie_portdrv_shutdown(struct pci_dev *dev)
+{
+	if (pci_bridge_d3_possible(dev)) {
+		pm_runtime_forbid(&dev->dev);
+		pm_runtime_get_noresume(&dev->dev);
+		pm_runtime_dont_use_autosuspend(&dev->dev);
+	}

I admit to being ignorant about how all this PM stuff works. A comment here about what's going on and why we need to check pci_bridge_d3_possible() would be useful.

Quoted 16 lines
drivers/pci/pcie/portdrv_pci.c +2 -1
+	pcie_port_device_remove(dev, false);
 }
 
 static pci_ers_result_t pcie_portdrv_error_detected(struct pci_dev *dev,
@@ -211,7 +222,7 @@ static struct pci_driver pcie_portdriver = {
 
 	.probe		= pcie_portdrv_probe,
 	.remove		= pcie_portdrv_remove,
-	.shutdown	= pcie_portdrv_remove,
+	.shutdown	= pcie_portdrv_shutdown,
 
 	.err_handler	= &pcie_portdrv_err_handler,
 
Signature · 1 line
2.7.0
#2 lore raw

Hi, all,

Quoted 20 lines · Bjorn Helgaas

[+cc Tiezhu]

Quoted 9 lines · Huacai Chen

Use separate remove()/shutdown() callback, and don't disable pci device during shutdown. This can avoid some poweroff/reboot failures.

The poweroff/reboot failures can easily reproduce on Loongson platforms. I think this is not a Loongson-specific problem, instead, is a problem related to some specific PCI hosts. On some x86 platforms, radeon/amdgpu devices can cause the same problem, and commit faefba95c9e8ca3a523831c2e ("drm/amdgpu: just suspend the hw on pci shutdown") can resolve it.

Quoted 6 lines

Radeon driver is more difficult than amdgpu due to its confusing symbol names, and I have maintained an out-of-tree patch for a long time [1]. Recently, we found more and more devices can cause the same problem, and it is very difficult to modify all problematic drivers as radeon/amdgpu does. So, I think modify the PCIe port driver is a simple and effective way.

Quoted 2 lines

Sounds plausible, but I don't know what the actual problem is, other than "poweroff/reboot doesn't work, and this patch 'fixes' it".

After all discussions (including those in the other thread), I think the actual problem is "every driver should implement their shutdown callback correctly as radeon/amdgpu does, otherwise there will be poweroff failures because of device's BUS MASTER".

So, I think I should give two patches, the first remove the "remove" callback, and the second just simply remove the "pci_disable_device()" instead of modifying a ton of drivers, right?

Huacai

Quoted 88 lines
Quoted 31 lines

[1] https://github.com/chenhuacai/linux/commit/6612f9c1fc290d42a14618ce9a7d03014d8ebb1a

---
 drivers/pci/pcie/portdrv.h      |  2 +-
 drivers/pci/pcie/portdrv_core.c |  6 ++++--
 drivers/pci/pcie/portdrv_pci.c  | 15 +++++++++++++--
 3 files changed, 18 insertions(+), 5 deletions(-)
drivers/pci/pcie/portdrv.h +1 -1
index af7cf23..22ba7b9 100644
@@ -123,7 +123,7 @@ int pcie_port_device_resume(struct device *dev);
 int pcie_port_device_runtime_suspend(struct device *dev);
 int pcie_port_device_runtime_resume(struct device *dev);
 #endif
-void pcie_port_device_remove(struct pci_dev *dev);
+void pcie_port_device_remove(struct pci_dev *dev, bool disable);
 int __must_check pcie_port_bus_register(void);
 void pcie_port_bus_unregister(void);
 
drivers/pci/pcie/portdrv_core.c +0 -0
index 50a9522..aa165be 100644
@@ -487,11 +487,13 @@ EXPORT_SYMBOL_GPL(pcie_port_find_device);
  * Remove PCI Express port service devices associated with given port and
  * disable MSI-X or MSI for the port.
  */

drivers/pci/pcie/portdrv_core.c EXPORT_SYMBOL_GPL(pcie_port_find_device); · line 497

Quoted 10 lines
drivers/pci/pcie/portdrv_core.c +4 -2
-void pcie_port_device_remove(struct pci_dev *dev)
+void pcie_port_device_remove(struct pci_dev *dev, bool disable)
 {
      device_for_each_child(&dev->dev, NULL, remove_iter);
      pci_free_irq_vectors(dev);
-     pci_disable_device(dev);
+
+     if (disable)
+             pci_disable_device(dev);
 }

portdrv can only be built statically (not as a module), and I don't think there's value in being able to unbind it. So I think we should remove its .remove() method. That would be a separate patch, of course, but it should make this one quite a bit simpler.

Quoted 10 lines
drivers/pci/pcie/portdrv_core.c +0 -0
 /**
drivers/pci/pcie/portdrv_pci.c +0 -1
index 3acf151..fcd3d2e 100644
@@ -142,7 +142,18 @@ static void pcie_portdrv_remove(struct pci_dev *dev)
              pm_runtime_dont_use_autosuspend(&dev->dev);
      }
 
-     pcie_port_device_remove(dev);

drivers/pci/pcie/portdrv_pci.c static void pcie_portdrv_remove(struct pci_dev *dev) · line 154

Quoted 10 lines
drivers/pci/pcie/portdrv_pci.c +10 -0
+     pcie_port_device_remove(dev, true);
+}
+
+static void pcie_portdrv_shutdown(struct pci_dev *dev)
+{
+     if (pci_bridge_d3_possible(dev)) {
+             pm_runtime_forbid(&dev->dev);
+             pm_runtime_get_noresume(&dev->dev);
+             pm_runtime_dont_use_autosuspend(&dev->dev);
+     }

I admit to being ignorant about how all this PM stuff works. A comment here about what's going on and why we need to check pci_bridge_d3_possible() would be useful.

Quoted 16 lines
drivers/pci/pcie/portdrv_pci.c +2 -2
+     pcie_port_device_remove(dev, false);
 }
 
 static pci_ers_result_t pcie_portdrv_error_detected(struct pci_dev *dev,
@@ -211,7 +222,7 @@ static struct pci_driver pcie_portdriver = {
 
      .probe          = pcie_portdrv_probe,
      .remove         = pcie_portdrv_remove,
-     .shutdown       = pcie_portdrv_remove,
+     .shutdown       = pcie_portdrv_shutdown,
 
      .err_handler    = &pcie_portdrv_err_handler,
 
--

2.7.0