Skip to contentSkip to conversation

PCI/portdrv: Remove the .remove() method in pcie_portdriver

PATCH In review Huacai Chen 4 participants 9 messages Sep 11, 2020 – Sep 15, 2020 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 PCI/portdrv: Remove the .remove() method in pcie_portdriver

As Bjorn Helgaas said, portdrv can only be built statically (not as a module), so the .remove() method in pcie_portdriver is useless. So just remove it.

BTW, rename pcie_portdrv_remove() to pcie_portdrv_shutdown() since it is only used by the .shutdown() method now.

---
 drivers/pci/pcie/portdrv_pci.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/pci/pcie/portdrv_pci.c +2 -3
index 3a3ce40..4e0af0f 100644
@@ -134,7 +134,7 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
 	return 0;
 }
 
-static void pcie_portdrv_remove(struct pci_dev *dev)
+static void pcie_portdrv_shutdown(struct pci_dev *dev)
 {
 	if (pci_bridge_d3_possible(dev)) {
 		pm_runtime_forbid(&dev->dev);
@@ -210,8 +210,7 @@ static struct pci_driver pcie_portdriver = {
 	.id_table	= &port_pci_ids[0],
 
 	.probe		= pcie_portdrv_probe,
-	.remove		= pcie_portdrv_remove,
-	.shutdown	= pcie_portdrv_remove,
+	.shutdown	= pcie_portdrv_shutdown,
 
 	.err_handler	= &pcie_portdrv_err_handler,
 
Signature · 1 line
2.7.0
#1 lore raw

PATCH 2/2 PCI/portdrv: Don't disable pci device during shutdown

Don't call pci_disable_device() in pcie_port_device_remove() during the portdrv's shutdown. This can avoid some poweroff/reboot failures.

The poweroff/reboot failures can easily reproduce on Loongson platforms. I think this is not a Loongson-specific problem, instead, is a problem related to some specific PCI hosts. On some x86 platforms, radeon/amdgpu devices can cause the same problem, and commit faefba95c9e8ca3a523831c2e ("drm/amdgpu: just suspend the hw on pci shutdown") can resolve it.

Radeon driver is more difficult than amdgpu due to its confusing symbol names, and I have maintained an out-of-tree patch for a long time [1]. Recently, we found more and more devices can cause the same problem, and it is very difficult to modify all problematic drivers as radeon/amdgpu does. So, I think modify the PCIe port driver is a simple and effective way.

[1] https://github.com/chenhuacai/linux/commit/6612f9c1fc290d42a14618ce9a7d03014d8ebb1a

---
 drivers/pci/pcie/portdrv_core.c | 1 -
 1 file changed, 1 deletion(-)
drivers/pci/pcie/portdrv_core.c +0 -1
index 50a9522..1991aca 100644
@@ -491,7 +491,6 @@ void pcie_port_device_remove(struct pci_dev *dev)
 {
 	device_for_each_child(&dev->dev, NULL, remove_iter);
 	pci_free_irq_vectors(dev);
-	pci_disable_device(dev);
 }
 
 /**
Signature · 1 line
2.7.0
#2 lore raw

Re: [PATCH 2/2] PCI/portdrv: Don't disable pci device during shutdown

Quoted 9 lines · Huacai Chen

Don't call pci_disable_device() in pcie_port_device_remove() during the portdrv's shutdown. This can avoid some poweroff/reboot failures.

The poweroff/reboot failures can easily reproduce on Loongson platforms. I think this is not a Loongson-specific problem, instead, is a problem related to some specific PCI hosts. On some x86 platforms, radeon/amdgpu devices can cause the same problem, and commit faefba95c9e8ca3a523831c2e ("drm/amdgpu: just suspend the hw on pci shutdown") can resolve it.

Quoted 6 lines

Radeon driver is more difficult than amdgpu due to its confusing symbol names, and I have maintained an out-of-tree patch for a long time [1]. Recently, we found more and more devices can cause the same problem, and it is very difficult to modify all problematic drivers as radeon/amdgpu does. So, I think modify the PCIe port driver is a simple and effective way.

This needs to explain in more detail what the failure is and how this patch fixes it.

The main thing pci_disable_device() does is turn off bus mastering, so I assume this has to do with DMA during shutdown or reboot. This change is in portdrv, so it affects PCIe Root Ports and Switch Ports, which of course are type 1 (bridge) devices. Clearing PCI_COMMAND_MASTER on bridges prevents them from forwarding memory or I/O requests in the upstream direction, i.e., it prevents DMA from devices below the bridge.

But if the problem is DMA, the same problem may occur with Root Complex Integrated Endpoints or conventional PCI devices, since portdrv may not be involved in those topologies.

Quoted 23 lines

[1] https://github.com/chenhuacai/linux/commit/6612f9c1fc290d42a14618ce9a7d03014d8ebb1a

---
 drivers/pci/pcie/portdrv_core.c | 1 -
 1 file changed, 1 deletion(-)
drivers/pci/pcie/portdrv_core.c +0 -1
index 50a9522..1991aca 100644
@@ -491,7 +491,6 @@ void pcie_port_device_remove(struct pci_dev *dev)
 {
 	device_for_each_child(&dev->dev, NULL, remove_iter);
 	pci_free_irq_vectors(dev);
-	pci_disable_device(dev);
 }
 
 /**
Signature · 1 line
2.7.0
#3 lore raw

RE: [PATCH 2/2] PCI/portdrv: Don't disable pci device during shutdown

[AMD Public Use]

Quoted 40 lines

-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Sunday, September 13, 2020 12:05 PM
To: Huacai Chen <[email protected]>
Cc: Bjorn Helgaas <[email protected]>; Deucher, Alexander <[email protected]>; [email protected]; Huacai Chen <[email protected]>; Jiaxun Yang <[email protected]>; Tiezhu Yang <[email protected]>
Subject: Re: [PATCH 2/2] PCI/portdrv: Don't disable pci device during shutdown

Quoted 10 lines · Huacai Chen

Don't call pci_disable_device() in pcie_port_device_remove() during the portdrv's shutdown. This can avoid some poweroff/reboot failures.

The poweroff/reboot failures can easily reproduce on Loongson platforms. I think this is not a Loongson-specific problem, instead, is a problem related to some specific PCI hosts. On some x86 platforms, radeon/amdgpu devices can cause the same problem, and commit faefba95c9e8ca3a523831c2e
("drm/amdgpu: just suspend the hw on pci shutdown") can resolve it.

Quoted 2 lines

Radeon driver is more difficult than amdgpu due to its confusing symbol names, and I have maintained an out-of-tree patch for a long time

[1].

Quoted 4 lines

Recently, we found more and more devices can cause the same problem, and it is very difficult to modify all problematic drivers as radeon/amdgpu does. So, I think modify the PCIe port driver is a simple and effective way.

This needs to explain in more detail what the failure is and how this patch fixes it.

The main thing pci_disable_device() does is turn off bus mastering, so I assume this has to do with DMA during shutdown or reboot. This change is in portdrv, so it affects PCIe Root Ports and Switch Ports, which of course are type 1 (bridge) devices. Clearing PCI_COMMAND_MASTER on bridges prevents them from forwarding memory or I/O requests in the upstream direction, i.e., it prevents DMA from devices below the bridge.

Quoted 3 lines

But if the problem is DMA, the same problem may occur with Root Complex Integrated Endpoints or conventional PCI devices, since portdrv may not be involved in those topologies.

I'm not sure I understand what the point of this patch is. Isn't the whole point of the shutdown callback to cleanly tear down whatever tasks are happening on the device? It could be DMA or stuff happening on the device itself (e.g., microcontrollers on the devices, etc.). Most of the complications in GPU drivers are due to the lifetime issues between drm and other subsystems.

Alex

Quoted 40 lines
Quoted 3 lines

[1]

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith

Quoted 1 line

ub.com%2Fchenhuacai%2Flinux%2Fcommit%2F6612f9c1fc290d42a14618ce9a 7d030

Quoted 1 line

14d8ebb1a&amp;data=02%7C01%7Calexander.deucher%40amd.com%7C1cc 5cca01b5

Quoted 1 line

74485a0aa08d857feb88e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C 0%7C637

Quoted 1 line

356098841869775&amp;sdata=HJmniTV2jJLEiOh3UCFpqPuGeq38y6crax2ahZa 5Eqc%

drivers/pci/pcie/portdrv_core.c void pcie_port_device_remove(struct pci_dev

Quoted 13 lines

3D&amp;reserved=0

---
 drivers/pci/pcie/portdrv_core.c | 1 -
 1 file changed, 1 deletion(-)
(quoted diff) +0 -0

b/drivers/pci/pcie/portdrv_core.c index 50a9522..1991aca 100644

drivers/pci/pcie/portdrv_core.c +0 -0
@@ -491,7 +491,6 @@ void pcie_port_device_remove(struct pci_dev

*dev)

Quoted 10 lines
drivers/pci/pcie/portdrv_core.c +0 -2
 {
 	device_for_each_child(&dev->dev, NULL, remove_iter);
 	pci_free_irq_vectors(dev);
-	pci_disable_device(dev);
 }
 
 /**
--

2.7.0

#4 lore raw
In reply to Huacai Chen
Quoted 3 lines · Huacai Chen

As Bjorn Helgaas said, portdrv can only be built statically (not as a module), so the .remove() method in pcie_portdriver is useless. So just remove it.

No, PCIe switches (containing upstream and downstream PCIe ports) can be hot-plugged and hot-removed at runtime. Every Thunderbolt device contains a PCIe switch and is hot-pluggable. We do want to clean up a hot-removed PCIe port properly.

Thanks,

Lukas

Quoted 20 lines
drivers/pci/pcie/portdrv_pci.c +2 -3
@@ -134,7 +134,7 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
 	return 0;
 }
 
-static void pcie_portdrv_remove(struct pci_dev *dev)
+static void pcie_portdrv_shutdown(struct pci_dev *dev)
 {
 	if (pci_bridge_d3_possible(dev)) {
 		pm_runtime_forbid(&dev->dev);
@@ -210,8 +210,7 @@ static struct pci_driver pcie_portdriver = {
 	.id_table	= &port_pci_ids[0],
 
 	.probe		= pcie_portdrv_probe,
-	.remove		= pcie_portdrv_remove,
-	.shutdown	= pcie_portdrv_remove,
+	.shutdown	= pcie_portdrv_shutdown,
 
 	.err_handler	= &pcie_portdrv_err_handler,
#5 lore raw
Quoted 9 lines · Lukas Wunner
Quoted 3 lines · Huacai Chen

As Bjorn Helgaas said, portdrv can only be built statically (not as a module), so the .remove() method in pcie_portdriver is useless. So just remove it.

No, PCIe switches (containing upstream and downstream PCIe ports) can be hot-plugged and hot-removed at runtime. Every Thunderbolt device contains a PCIe switch and is hot-pluggable. We do want to clean up a hot-removed PCIe port properly.

Right, sorry, I was thinking only of driver unbinding, not of device removal. Sorry to have wasted your time.

Quoted 20 lines
Quoted 20 lines
drivers/pci/pcie/portdrv_pci.c +2 -3
@@ -134,7 +134,7 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
 	return 0;
 }
 
-static void pcie_portdrv_remove(struct pci_dev *dev)
+static void pcie_portdrv_shutdown(struct pci_dev *dev)
 {
 	if (pci_bridge_d3_possible(dev)) {
 		pm_runtime_forbid(&dev->dev);
@@ -210,8 +210,7 @@ static struct pci_driver pcie_portdriver = {
 	.id_table	= &port_pci_ids[0],
 
 	.probe		= pcie_portdrv_probe,
-	.remove		= pcie_portdrv_remove,
-	.shutdown	= pcie_portdrv_remove,
+	.shutdown	= pcie_portdrv_shutdown,
 
 	.err_handler	= &pcie_portdrv_err_handler,
#6 lore raw

[AMD Public Use]

Quoted 17 lines

-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Sunday, September 13, 2020 11:43 AM
To: Lukas Wunner <[email protected]>
Cc: Huacai Chen <[email protected]>; Bjorn Helgaas <[email protected]>; Deucher, Alexander <[email protected]>; [email protected]; Huacai Chen <[email protected]>; Jiaxun Yang <[email protected]>
Subject: Re: [PATCH 1/2] PCI/portdrv: Remove the .remove() method in pcie_portdriver

Quoted 5 lines · Lukas Wunner
Quoted 3 lines · Huacai Chen

As Bjorn Helgaas said, portdrv can only be built statically (not as a module), so the .remove() method in pcie_portdriver is useless. So just remove it.

Quoted 8 lines
Quoted 4 lines

No, PCIe switches (containing upstream and downstream PCIe ports) can be hot-plugged and hot-removed at runtime. Every Thunderbolt device contains a PCIe switch and is hot-pluggable. We do want to clean up a hot-removed PCIe port properly.

Right, sorry, I was thinking only of driver unbinding, not of device removal. Sorry to have wasted your time.

FWIW, our newer GPUs have both upstream and downstream ports that are part of the device.

Slightly off topic, but does the current pm code handle these cases correctly? ACPI related power handling doesn't seem to work correctly for these devices in laptops where the GPU power control is handled by ACPI.

Alex

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
drivers/pci/pcie/portdrv_pci.c +2 -3
@@ -134,7 +134,7 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
 	return 0;
 }
 
-static void pcie_portdrv_remove(struct pci_dev *dev)
+static void pcie_portdrv_shutdown(struct pci_dev *dev)
 {
 	if (pci_bridge_d3_possible(dev)) {
 		pm_runtime_forbid(&dev->dev);
@@ -210,8 +210,7 @@ static struct pci_driver pcie_portdriver = {
 	.id_table	= &port_pci_ids[0],
 
 	.probe		= pcie_portdrv_probe,
-	.remove		= pcie_portdrv_remove,
-	.shutdown	= pcie_portdrv_remove,
+	.shutdown	= pcie_portdrv_shutdown,
 
 	.err_handler	= &pcie_portdrv_err_handler,
#7 lore raw

[+cc Rafael]

Quoted 7 lines · Alexander

FWIW, our newer GPUs have both upstream and downstream ports that are part of the device.

Slightly off topic, but does the current pm code handle these cases correctly? ACPI related power handling doesn't seem to work correctly for these devices in laptops where the GPU power control is handled by ACPI.

I guess these newer GPUs basically have a PCIe switch embedded in them? The picture in my head is this:

	     +--------------------------------------+
  +----+     |+--------+   +----------+   +--------+|
  |Root+------+Switch  +---+Switch    +---+GPU     ||
  |Port|     ||Upstream|   |Downstream|   |Endpoint||
  +----+     ||Port    |   |Port      |   |        ||
	     |+--------+   +----------+   +--------+|
	     +--------------------------------------+

Is that accurate? If not, can you share "lspci -vv" output so we can figure it out?

The PCI PM code is *supposed* to work with arbitrary hierarchies, so assuming your devices are PCIe spec-compliant, it should work.

It sounds like ACPI PM is involved as well, and I can't speak to that side at all. But I know Rafael can :)

Bjorn

#8 lore raw
In reply to Deucher, Alexander
Quoted 7 lines · Alexander

FWIW, our newer GPUs have both upstream and downstream ports that are part of the device.

Slightly off topic, but does the current pm code handle these cases correctly? ACPI related power handling doesn't seem to work correctly for these devices in laptops where the GPU power control is handled by ACPI.

PCIe ports are only suspended to D3 if pci_bridge_d3_possible() in drivers/pci/pci.c returns true. In particular, if the downstream ports are hotplug-capable, they will *not* be suspended to D3 unless "pcie_port_pm=force" is specified on the command line. There was a report of MCEs on Xeon-SP servers if hotplug ports were runtime suspended, hence this rule. Also, non-hotplug ports are not suspended if the BIOS is older than 2015.

If the downstream ports are not suspended, by consequence the upstream port above them isn't either. So if the GPU is powered down by an ACPI _PR3 method for the upstream port, that method may not be executed.

I think the _PR3 method for GPUs was located in the Root Port's namespace so far. If it's moved to a port below that, then it may be necessary to adjust GPU driver code, e.g. where pci_pr3_present() is called (but that's only called by nouveau and hda_intel.c AFAICS).

Thanks,

Lukas