PCI/MSI: Fix MSI IRQ domains for SR-IOV
PATCH 1/1 PCI/MSI: Fix MSI IRQ domains for SR-IOV
SR-IOV creates a virtual bus where bus->self is NULL. This results in a segfault as VFs are added and we scan for an MSI domain without taking that into account. Detect this and scan up to the parent bus until we find a real bridge.
--- drivers/pci/probe.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
drivers/pci/probe.c +11 -6
@@ -676,15 +676,20 @@ static struct irq_domain *pci_host_bridge_msi_domain(struct pci_bus *bus)static void pci_set_bus_msi_domain (struct pci_bus *bus) {struct irq_domain *d; +struct pci_bus *b;/* -* Either bus is the root, and we must obtain it from the -* firmware, or we inherit it from the bridge device. +* The bus can be a root bus, a subordinate bus, or a virtual bus +* created by an SR-IOV device. Walk up to the first bridge device +* found or derive the domain from the host bridge. */ -if (pci_is_root_bus (bus)) - d =pci_host_bridge_msi_domain (bus); -else - d =dev_get_msi_domain (&bus->self->dev); +for (b = bus, d =NULL ; !d && !pci_is_root_bus (b); b = b->parent) { +if (b->self) + d =dev_get_msi_domain (&b->self->dev); + } + +if (!d) + d =pci_host_bridge_msi_domain (b);dev_set_msi_domain (&bus->dev, d); }
On Fri, 18 Sep 2015 15:08:54 -0600
Alex Williamson <[email protected]> wrote:
Hi Alex,
Quoted 4 lines
SR-IOV creates a virtual bus where
bus->selfis NULL. This results in a segfault as VFs are added and we scan for an MSI domain without taking that into account. Detect this and scan up to the parent bus until we find a real bridge.
Irk. Sorry about the breakage.
Quoted 28 lines
Fixes 44aa0c657e3e ("PCI/MSI: Add hooks to populate the msi_domain field")Signed-off-by Alex Williamson <[email protected]>--- drivers/pci/probe.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)drivers/pci/probe.c +4
-6@@ -676,15 +676,20 @@ static struct irq_domain *pci_host_bridge_msi_domain(struct pci_bus *bus)static void pci_set_bus_msi_domain (struct pci_bus *bus) {struct irq_domain *d; +struct pci_bus *b;/* -* Either bus is the root, and we must obtain it from the -* firmware, or we inherit it from the bridge device. +* The bus can be a root bus, a subordinate bus, or a virtual bus +* created by an SR-IOV device. Walk up to the first bridge device +* found or derive the domain from the host bridge. */ -if (pci_is_root_bus (bus)) - d =pci_host_bridge_msi_domain (bus); -else - d =dev_get_msi_domain (&bus->self->dev);
drivers/pci/probe.c static struct irq_domain *pci_host_bridge_msi_domain(struct pci_bus *bus) · line 695
Quoted 10 lines
drivers/pci/probe.c +7
-0+for (b = bus, d =NULL ; !d && !pci_is_root_bus (b); b = b->parent) { +if (b->self) + d =dev_get_msi_domain (&b->self->dev); + } + +if (!d) + d =pci_host_bridge_msi_domain (b);dev_set_msi_domain (&bus->dev, d); }
Out of curiosity, is this a common behaviour? I've tested the original code with an Intel i350 Ethernet interface (IGB+IGBVF), and used it with virtual functions on my arm64-based Seattle, without any issue. Do we have divergent implementations of the same functionality in the kernel? Otherwise:
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny.
Quoted 47 lines · Marc Zyngier
On Fri, 18 Sep 2015 15:08:54 -0600
Alex Williamson <[email protected]> wrote:Hi Alex,
Quoted 4 lines
SR-IOV creates a virtual bus where
bus->selfis NULL. This results in a segfault as VFs are added and we scan for an MSI domain without taking that into account. Detect this and scan up to the parent bus until we find a real bridge.Irk. Sorry about the breakage.
Quoted 35 lines
Fixes 44aa0c657e3e ("PCI/MSI: Add hooks to populate the msi_domain field")Signed-off-by Alex Williamson <[email protected]>--- drivers/pci/probe.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)drivers/pci/probe.c +11
-6@@ -676,15 +676,20 @@ static struct irq_domain *pci_host_bridge_msi_domain(struct pci_bus *bus)static void pci_set_bus_msi_domain (struct pci_bus *bus) {struct irq_domain *d; +struct pci_bus *b;/* -* Either bus is the root, and we must obtain it from the -* firmware, or we inherit it from the bridge device. +* The bus can be a root bus, a subordinate bus, or a virtual bus +* created by an SR-IOV device. Walk up to the first bridge device +* found or derive the domain from the host bridge. */ -if (pci_is_root_bus (bus)) - d =pci_host_bridge_msi_domain (bus); -else - d =dev_get_msi_domain (&bus->self->dev); +for (b = bus, d =NULL ; !d && !pci_is_root_bus (b); b = b->parent) { +if (b->self) + d =dev_get_msi_domain (&b->self->dev); + } + +if (!d) + d =pci_host_bridge_msi_domain (b);
Quoted 9 lines
drivers/pci/probe.c static struct irq_domain *pci_host_bridge_msi_domain(struct pci_bus *bus) · line 695
Quoted 3 lines
drivers/pci/probe.c +0
-0dev_set_msi_domain (&bus->dev, d); }Out of curiosity, is this a common behaviour? I've tested the original code with an Intel i350 Ethernet interface (IGB+IGBVF), and used it with virtual functions on my arm64-based Seattle, without any issue. Do we have divergent implementations of the same functionality in the kernel? Otherwise:
Thanks for the review Marc. I believe this is a property coming out of the core PCI IOV code:
drivers/pci/iov.c:virtfn_add_bus()
child = pci_add_new_bus(bus, NULL, busnr);
That second arg is the dev for the new bus, which gets passed as 'bridge' to:
drivers/pci/probe.c:pci_alloc_child_bus()
child->self = bridge;
Resulting in our bus->self == NULL issue.
We have the following call path to virtfn_add_bus:
pci_enable_sriov
sriov_enable
virtfn_add
virtfn_add_bus
The only thing unique I can think of for my system is that I'm using the max_vfs module option for igb, but it's not apparent to me how that would trigger anything different through here. Thanks,
Alex
[+cc Joerg]
Quoted 7 lines · Alex Williamson
SR-IOV creates a virtual bus where
bus->selfis NULL. This results in a segfault as VFs are added and we scan for an MSI domain without taking that into account. Detect this and scan up to the parent bus until we find a real bridge.Fixes 44aa0c657e3e ("PCI/MSI: Add hooks to populate the msi_domain field")Signed-off-by Alex Williamson <[email protected]>
Applied to for-linus with changelog below for v4.3, thanks!
PCI/MSI: Fix MSI IRQ domains for VFs on virtual buses
SR-IOV creates a virtual bus where bus->self is NULL. When we add VFs and scan for an MSI domain, pci_set_bus_msi_domain() dereferences bus->self, which causes a kernel NULL pointer dereference oops.
Scan up to the parent bus until we find a real bridge where we can get the MSI domain.