Skip to contentSkip to conversation

PCI: vmd: Fix secondary bus reset for Intel bridges

PATCH Superseded [email protected] 2 participants 3 messages Nov 3, 2022 – Dec 5, 2022

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 PCI: vmd: Fix secondary bus reset for Intel bridges

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, pci_reset_bus() applies a reset to the parent of the PCI device supplied as argument, but in this case it failed because there wasn't a parent.

In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when guest reboots are performed. Commit id 6aab5622296b ("PCI: vmd: Clean up domain before enumeration") attempted to fix this, but later we discovered that the code inside pci_reset_bus() wasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.

---
V3:
- Add WARN_ON
- Include Jonathan as reviewer
- Update commit message
V2:
- Update commit message

 drivers/pci/controller/vmd.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +11 -2
index e06e9f4fc50f..2406be6644f3 100644
@@ -859,8 +859,17 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			ret = pci_reset_bus(list_first_entry(&child->devices,
+							     struct pci_dev,
+							     bus_list));
+			WARN_ON(ret);
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
Signature · 1 line
2.25.1
#1 lore raw
Quoted 43 lines · On 11/3/2022 1:14 PM, [email protected] wrote:

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, pci_reset_bus() applies a reset to the parent of the PCI device supplied as argument, but in this case it failed because there wasn't a parent.

In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when guest reboots are performed. Commit id 6aab5622296b ("PCI: vmd: Clean up domain before enumeration") attempted to fix this, but later we discovered that the code inside pci_reset_bus() wasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.

---
V3:
- Add WARN_ON
- Include Jonathan as reviewer
- Update commit message
V2:
- Update commit message

 drivers/pci/controller/vmd.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +4 -2
index e06e9f4fc50f..2406be6644f3 100644
@@ -859,8 +859,17 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			ret = pci_reset_bus(list_first_entry(&child->devices,

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 875

Quoted 10 lines
drivers/pci/controller/vmd.c +7 -0
+							     struct pci_dev,
+							     bus_list));
+			WARN_ON(ret);
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*

Hi,

Just a gentle reminder for this one

Best wishes,
Francisco.

#2 lore raw
Quoted 7 lines · Francisco
Quoted 6 lines · On 11/3/2022 1:14 PM, [email protected] wrote:

From: Francisco Munoz <[email protected]>

The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, pci_reset_bus() applies a reset to the parent of the PCI device supplied as argument, but in this case it failed because there wasn't a parent.

Please add a comment in the *code* as well and explain why it is so.

Quoted 7 lines
Quoted 7 lines

In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when guest reboots are performed. Commit id 6aab5622296b ("PCI: vmd: Clean up domain before enumeration") attempted to fix this, but later we discovered that the code inside pci_reset_bus() wasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.

I am not sure this helps much - I'd rather explain why the device hierarchy in VMD is as it is - when that's explained is rather clear why the current reset is NOP and this patch is needed.

Quoted 20 lines
Quoted 20 lines

---
V3:
- Add WARN_ON
- Include Jonathan as reviewer
- Update commit message
V2:
- Update commit message

 drivers/pci/controller/vmd.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +0 -0
index e06e9f4fc50f..2406be6644f3 100644
@@ -859,8 +859,17 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
Quoted 11 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 868

Quoted 11 lines
drivers/pci/controller/vmd.c +7 -2
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			ret = pci_reset_bus(list_first_entry(&child->devices,
+							     struct pci_dev,
+							     bus_list));
+			WARN_ON(ret);

Technically you are adding a WARN_ON() here to make sure that a failed reset is detected - I am not sure a backtrace is really required, a pci_warn() maybe ?

Lorenzo

Quoted 15 lines

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 875

Quoted 7 lines
drivers/pci/controller/vmd.c +4 -0
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*

Hi,

Just a gentle reminder for this one

Best wishes,
Francisco.