Skip to contentSkip to conversation

PCI: vmd: Fix secondary bus reset for Intel bridges

PATCH Superseded [email protected] 4 participants 9 messages Sep 23, 2022 – Nov 3, 2022

Subsystems (3)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: vmd: Fix secondary bus reset for Intel bridges

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.

---
 drivers/pci/controller/vmd.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
Signature · 1 line
2.25.1
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 11 lines · On 9/23/2022 2:37 PM, [email protected] wrote:

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.

Quoted 27 lines
---
 drivers/pci/controller/vmd.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
#2 lore raw

Hi,

Please let me know if something else is needed.

Thanks,
Francisco.

Quoted 43 lines · Jonathan Derrick
Quoted 11 lines · On 9/23/2022 2:37 PM, [email protected] wrote:

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.

Quoted 27 lines
---
  drivers/pci/controller/vmd.c | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
  	pci_scan_child_bus(vmd->bus);
  	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));
+			break;
+		}
+	}
+
  	pci_assign_unassigned_bus_resources(vmd->bus);
 
  	/*
#3 lore raw
Quoted 3 lines · Francisco

Hi,

Please let me know if something else is needed.

We're in the merge window now, so it's too late for v6.1, but we'll start merging v6.2 changes about Oct 17.

Quoted 40 lines · Jonathan Derrick
Quoted 39 lines · On 9/23/2022 2:37 PM, [email protected] wrote:
Quoted 11 lines

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.

Quoted 25 lines
---
  drivers/pci/controller/vmd.c | 12 ++++++++++--
  1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
  	pci_scan_child_bus(vmd->bus);
  	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));
+			break;
+		}
+	}
+
  	pci_assign_unassigned_bus_resources(vmd->bus);
  	/*
#4 lore raw
In reply to [email protected]
Quoted 8 lines · On Fri, Sep 23, 2022 at 01:37:57PM -0700, [email protected]…

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.

What kind of problem does this fix? I guess some devices below a VMD need to be reset before we can use them?

As a rule, Linux doesn't reset PCI devices at boot, so I'm just wondering what's different about these.

If this fixes a problem, it's also nice if we can include a symptom in the commit log so if people are seeing the problem, they can find the solution, or distros can tell whether they need to include it.

Quoted 32 lines
---
 drivers/pci/controller/vmd.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
Signature · 1 line
2.25.1
#5 lore raw

PATCH v2 1/1 PCI: vmd: Fix secondary bus reset for Intel bridges

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, pci_reset_bus applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent.

In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when host reboots are performed. Commit id “6aab5622296b990024ee67dd7efa7d143e7558d0” attempted to fix this, but later we discovered that the code inside pci_reset_bus wasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.

---
 drivers/pci/controller/vmd.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
Signature · 9 lines
2.25.1

Hi Bjorn,

I updated the commit message with more details. Hopefully, this will 
clarify its purpose.

Thanks,
Francisco.
#6 lore raw

[+cc Alex, Myron]

Quoted 7 lines · On Mon, Oct 31, 2022 at 02:45:01PM -0700, [email protected]…

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, pci_reset_bus applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent.

Quoted 7 lines

In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when host reboots are performed. Commit id “6aab5622296b990024ee67dd7efa7d143e7558d0” attempted to fix this, but later we discovered that the code inside pci_reset_bus wasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.

Did you mean "guest reboots" above? If the *host* reboots, I assume everybody (host and guests) starts over, so a reset wouldn't really apply.

Is the scenario that the VMD device is passed through to a guest, and the guest OS is running vmd_probe() and vmd_enable_domain()?

I thought VFIO already had something to reset devices between guests. But maybe this is different because from the point of view of VFIO, the pass-through happens only once, and during that single session, the guest OS reboots several times, so you want vmd_probe() to reset the downstream devices?

Should this have a Fixes: tag for 6aab5622296b?

s/pci/PCI/ above in English text.

Also add "()" after function names.

Use the typical 12-char SHA1 + subject citation, e.g., 6aab5622296b ("PCI: vmd: Clean up domain before enumeration").

Quoted 39 lines
---
 drivers/pci/controller/vmd.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
Signature · 9 lines
2.25.1

Hi Bjorn,

I updated the commit message with more details. Hopefully, this will 
clarify its purpose.

Thanks,
Francisco.
#7 lore raw

Thanks for including Alex and Myron.

Quoted 18 lines · Bjorn Helgaas

[+cc Alex, Myron]

Quoted 7 lines · On Mon, Oct 31, 2022 at 02:45:01PM -0700, [email protected]…

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, pci_reset_bus applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent.

Quoted 7 lines

In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when host reboots are performed. Commit id “6aab5622296b990024ee67dd7efa7d143e7558d0” attempted to fix this, but later we discovered that the code inside pci_reset_bus wasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.

Quoted 4 lines

Did you mean "guest reboots" above? If the *host* reboots, I assume everybody (host and guests) starts over, so a reset wouldn't really apply.

Correct, I meant guest reboots.

Quoted 8 lines

Is the scenario that the VMD device is passed through to a guest, and the guest OS is running vmd_probe() and vmd_enable_domain()?

I thought VFIO already had something to reset devices between guests. But maybe this is different because from the point of view of VFIO, the pass-through happens only once, and during that single session, the guest OS reboots several times, so you want vmd_probe() to reset the downstream devices?

Right, this change just makes sure pci_bus_reset(), introduced in 6aab5622296b ("PCI: vmd: Clean up domain before enumeration"), is sending pci_bridge_secondary_bus_reset().

I'll issue a v3 with the suggestions below.

Quoted 42 lines

Should this have a Fixes: tag for 6aab5622296b?

s/pci/PCI/ above in English text.

Also add "()" after function names.

Use the typical 12-char SHA1 + subject citation, e.g., 6aab5622296b ("PCI: vmd: Clean up domain before enumeration").

Quoted 32 lines
---
 drivers/pci/controller/vmd.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
Signature · 1 line
2.25.1
Quoted 7 lines
Quoted 7 lines

Hi Bjorn,

I updated the commit message with more details. Hopefully, this will clarify its purpose.

Thanks,
Francisco.

Thanks!

#8 lore raw
In reply to Bjorn Helgaas
Quoted 52 lines · On Wed, 2 Nov 2022 18:42:21 -0500 Bjorn Helgaas <[email protected]> wrote:

[+cc Alex, Myron]

Quoted 7 lines · On Mon, Oct 31, 2022 at 02:45:01PM -0700, [email protected]…

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, pci_reset_bus applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent.

Quoted 7 lines

In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when host reboots are performed. Commit id “6aab5622296b990024ee67dd7efa7d143e7558d0” attempted to fix this, but later we discovered that the code inside pci_reset_bus wasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.

Did you mean "guest reboots" above? If the *host* reboots, I assume everybody (host and guests) starts over, so a reset wouldn't really apply.

Is the scenario that the VMD device is passed through to a guest, and the guest OS is running vmd_probe() and vmd_enable_domain()?

I thought VFIO already had something to reset devices between guests. But maybe this is different because from the point of view of VFIO, the pass-through happens only once, and during that single session, the guest OS reboots several times, so you want vmd_probe() to reset the downstream devices?

Should this have a Fixes: tag for 6aab5622296b?

s/pci/PCI/ above in English text.

Also add "()" after function names.

Use the typical 12-char SHA1 + subject citation, e.g., 6aab5622296b ("PCI: vmd: Clean up domain before enumeration").

Quoted 12 lines
---
 drivers/pci/controller/vmd.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +0 -0
index e06e9f4fc50f..34d6ba675440 100644
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
Quoted 10 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 867

Quoted 10 lines
drivers/pci/controller/vmd.c +6 -2
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			pci_reset_bus(list_first_entry(&child->devices,
+						       struct pci_dev,
+						       bus_list));

Do you want to test the return value here to avoid another case of not actually doing what we expect it to do? WARN_ON perhaps? Thanks,

Alex

Quoted 18 lines
Quoted 17 lines
drivers/pci/controller/vmd.c +4 -0
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
Signature · 9 lines
2.25.1

Hi Bjorn,

I updated the commit message with more details. Hopefully, this will 
clarify its purpose.

Thanks,
Francisco.