PCI: vmd: Fix secondary bus reset for Intel bridges
Subsystems (3)
PATCH 1/1 PCI: vmd: Fix secondary bus reset for Intel bridges
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.
--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list)); +break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/*
Signature · 1 line
2.25.1
Quoted 11 lines · On 9/23/2022 2:37 PM, [email protected] wrote:
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.
Signed-off-by Francisco Munoz <[email protected]>Reviewed-by Nirmal Patel <[email protected]>
Quoted 27 lines
--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +10
-2@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list)); +break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/*
Hi,
Please let me know if something else is needed.
Thanks,
Francisco.
Quoted 43 lines · Jonathan Derrick
Quoted 11 lines · On 9/23/2022 2:37 PM, [email protected] wrote:
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.
Signed-off-by Francisco Munoz <[email protected]>Reviewed-by Nirmal Patel <[email protected]>Reviewed-by Jonathan Derrick <[email protected]>Quoted 27 lines
--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +10
-2@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list)); +break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/*
Quoted 3 lines · Francisco
Hi,
Please let me know if something else is needed.
We're in the merge window now, so it's too late for v6.1, but we'll start merging v6.2 changes about Oct 17.
Quoted 40 lines · Jonathan Derrick
Quoted 39 lines · On 9/23/2022 2:37 PM, [email protected] wrote:
Quoted 11 lines
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.
Signed-off-by Francisco Munoz <[email protected]>Reviewed-by Nirmal Patel <[email protected]>Reviewed-by Jonathan Derrick <[email protected]>Quoted 25 lines
--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +10
-2@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list)); +break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/*
Quoted 8 lines · On Fri, Sep 23, 2022 at 01:37:57PM -0700, [email protected]…
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, the reset API applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent. This change feeds a child device of an Intel Bridge to the reset API and internally the reset is applied to its parent.
What kind of problem does this fix? I guess some devices below a VMD need to be reset before we can use them?
As a rule, Linux doesn't reset PCI devices at boot, so I'm just wondering what's different about these.
If this fixes a problem, it's also nice if we can include a symptom in the commit log so if people are seeing the problem, they can find the solution, or distros can tell whether they need to include it.
Quoted 32 lines
Signed-off-by Francisco Munoz <[email protected]>Reviewed-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +10
-2@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list)); +break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/* Signature · 1 line
2.25.1
PATCH v2 1/1 PCI: vmd: Fix secondary bus reset for Intel bridges
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally, pci_reset_bus applies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent.
In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when host reboots are performed. Commit id “6aab5622296b990024ee67dd7efa7d143e7558d0” attempted to fix this, but later we discovered that the code inside pci_reset_bus wasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.
--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list)); +break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/*
Signature · 9 lines
2.25.1 Hi Bjorn, I updated the commit message with more details. Hopefully, this will clarify its purpose. Thanks, Francisco.
[+cc Alex, Myron]
Quoted 7 lines · On Mon, Oct 31, 2022 at 02:45:01PM -0700, [email protected]…
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally,
pci_reset_busapplies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent.
Quoted 7 lines
In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when host reboots are performed. Commit id “6aab5622296b990024ee67dd7efa7d143e7558d0” attempted to fix this, but later we discovered that the code inside
pci_reset_buswasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.
Did you mean "guest reboots" above? If the *host* reboots, I assume everybody (host and guests) starts over, so a reset wouldn't really apply.
Is the scenario that the VMD device is passed through to a guest, and the guest OS is running vmd_probe() and vmd_enable_domain()?
I thought VFIO already had something to reset devices between guests. But maybe this is different because from the point of view of VFIO, the pass-through happens only once, and during that single session, the guest OS reboots several times, so you want vmd_probe() to reset the downstream devices?
Should this have a Fixes: tag for 6aab5622296b?
s/pci/PCI/ above in English text.
Also add "()" after function names.
Use the typical 12-char SHA1 + subject citation, e.g., 6aab5622296b ("PCI: vmd: Clean up domain before enumeration").
Quoted 39 lines
Signed-off-by Francisco Munoz <[email protected]>Reviewed-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +10
-2@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list)); +break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/* Signature · 9 lines
2.25.1 Hi Bjorn, I updated the commit message with more details. Hopefully, this will clarify its purpose. Thanks, Francisco.
Thanks for including Alex and Myron.
Quoted 18 lines · Bjorn Helgaas
[+cc Alex, Myron]
Quoted 7 lines · On Mon, Oct 31, 2022 at 02:45:01PM -0700, [email protected]…
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally,
pci_reset_busapplies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent.Quoted 7 lines
In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when host reboots are performed. Commit id “6aab5622296b990024ee67dd7efa7d143e7558d0” attempted to fix this, but later we discovered that the code inside
pci_reset_buswasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.
Quoted 4 lines
Did you mean "guest reboots" above? If the *host* reboots, I assume everybody (host and guests) starts over, so a reset wouldn't really apply.
Correct, I meant guest reboots.
Quoted 8 lines
Is the scenario that the VMD device is passed through to a guest, and the guest OS is running
vmd_probe()andvmd_enable_domain()?I thought VFIO already had something to reset devices between guests. But maybe this is different because from the point of view of VFIO, the pass-through happens only once, and during that single session, the guest OS reboots several times, so you want
vmd_probe()to reset the downstream devices?
Right, this change just makes sure pci_bus_reset(), introduced in 6aab5622296b ("PCI: vmd: Clean up domain before enumeration"), is sending pci_bridge_secondary_bus_reset().
I'll issue a v3 with the suggestions below.
Quoted 42 lines
Should this have a Fixes: tag for
6aab5622296b?s/pci/PCI/ above in English text.
Also add "()" after function names.
Use the typical 12-char SHA1 + subject citation, e.g.,
6aab5622296b("PCI: vmd: Clean up domain before enumeration").Quoted 32 lines
Signed-off-by Francisco Munoz <[email protected]>Reviewed-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +10
-2@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list)); +break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/* Signature · 1 line
2.25.1
Quoted 7 lines
Quoted 7 lines
Hi Bjorn,
I updated the commit message with more details. Hopefully, this will clarify its purpose.
Thanks,
Francisco.
Thanks!
Quoted 52 lines · On Wed, 2 Nov 2022 18:42:21 -0500 Bjorn Helgaas <[email protected]> wrote:
[+cc Alex, Myron]
Quoted 7 lines · On Mon, Oct 31, 2022 at 02:45:01PM -0700, [email protected]…
From: Francisco Munoz <[email protected]>
The reset was never applied in the current implementation because Intel Bridges owned by VMD are parentless. Internally,
pci_reset_busapplies a reset to the parent of the pci device supplied as argument, but in this case it failed because there wasn't a parent.Quoted 7 lines
In more detail, this change allows the VMD driver to enumerate NVMe devices in pass-through configurations when host reboots are performed. Commit id “6aab5622296b990024ee67dd7efa7d143e7558d0” attempted to fix this, but later we discovered that the code inside
pci_reset_buswasn’t triggering secondary bus resets. Therefore, we updated the parameters passed to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in VT-d pass-through scenarios.Did you mean "guest reboots" above? If the *host* reboots, I assume everybody (host and guests) starts over, so a reset wouldn't really apply.
Is the scenario that the VMD device is passed through to a guest, and the guest OS is running
vmd_probe()andvmd_enable_domain()?I thought VFIO already had something to reset devices between guests. But maybe this is different because from the point of view of VFIO, the pass-through happens only once, and during that single session, the guest OS reboots several times, so you want
vmd_probe()to reset the downstream devices?Should this have a Fixes: tag for
6aab5622296b?s/pci/PCI/ above in English text.
Also add "()" after function names.
Use the typical 12-char SHA1 + subject citation, e.g.,
6aab5622296b("PCI: vmd: Clean up domain before enumeration").Quoted 12 lines
Signed-off-by Francisco Munoz <[email protected]>Reviewed-by Nirmal Patel <[email protected]>--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +0
-0@@ -859,8 +859,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
Quoted 10 lines
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 867
Quoted 10 lines
drivers/pci/controller/vmd.c +6
-2pci_scan_child_bus (vmd->bus);vmd_domain_reset (vmd); -list_for_each_entry (child, &vmd->bus->children, node) -pci_reset_bus (child->self); + +list_for_each_entry (child, &vmd->bus->children, node) { +if (!list_empty (&child->devices)) { +pci_reset_bus (list_first_entry (&child->devices, + struct pci_dev, + bus_list));
Do you want to test the return value here to avoid another case of not actually doing what we expect it to do? WARN_ON perhaps? Thanks,
Alex
Quoted 18 lines
Quoted 17 lines
drivers/pci/controller/vmd.c +4
-0+break ; + } + } +pci_assign_unassigned_bus_resources (vmd->bus);/* Signature · 9 lines
2.25.1 Hi Bjorn, I updated the commit message with more details. Hopefully, this will clarify its purpose. Thanks, Francisco.