PCI/x86: Interface for testing multivector MSI support

For the most part multivector MSI is not supported and drivers and hardware wanting multiple vectors opt for MSI-X instead. It seems though that having the ability to query the arch/platform code to determine whether allocating multiple MSI vectors will ever succeed is a useful thing. For instance, vfio-pci can use this to determine whether to expose multiple MSI vectors to the user. If we know we cannot ever support more than one vector, we have a better shot at the userspace driver working, especially if it's a guest OS, if we only expose one vector as being available in the interface.

Signed-off-by Alex Williamson <[email protected]>

---

 drivers/pci/msi.c   |    5 +++++
 include/linux/msi.h |    1 +
 2 files changed, 6 insertions(+)

drivers/pci/msi.c +0 -0

index 9fab30a..36b503a 100644

Quoted 22 lines · Alex Williamson

For the most part multivector MSI is not supported and drivers and hardware wanting multiple vectors opt for MSI-X instead. It seems though that having the ability to query the arch/platform code to determine whether allocating multiple MSI vectors will ever succeed is a useful thing. For instance, vfio-pci can use this to determine whether to expose multiple MSI vectors to the user. If we know we cannot ever support more than one vector, we have a better shot at the userspace driver working, especially if it's a guest OS, if we only expose one vector as being available in the interface.

Signed-off-by Alex Williamson <[email protected]>

---

 drivers/pci/msi.c   |    5 +++++
 include/linux/msi.h |    1 +
 2 files changed, 6 insertions(+)

drivers/pci/msi.c +0 -0

index 9fab30a..36b503a 100644

drivers/pci/msi.c int __weak arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) · line 82

Quoted 4 lines

drivers/pci/msi.c +1 -0

@@ -79,6 +79,11 @@ int __weak arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 	return 0;
 }
 
+bool __weak arch_supports_multivector_msi(struct pci_dev *dev)

Please not another weak arch function. We are in the process to reduce them not to extend them.

arch_supports is pretty much wrong here anyway. We are moving away from arch MSI dependencies simply because it's not a arch property per se.

Multi MSI is a property of the underlying interrupt controllers and there might be several MSI interrupt domains on a given system. They can or cannot support multi MSI.

The current x86 implementation is a tangled maze and Jiang is in the process to distangle it completely. Until thats done x86 is not going to add new ad hoc interfaces.

Once we converted everything over to the new hierarchical irqdomains we can add such an interface to the code, but for now I'm not accepting anything like that into x86 msi related code.

Quoted 39 lines · Thomas Gleixner

Quoted 22 lines · Alex Williamson

For the most part multivector MSI is not supported and drivers and hardware wanting multiple vectors opt for MSI-X instead. It seems though that having the ability to query the arch/platform code to determine whether allocating multiple MSI vectors will ever succeed is a useful thing. For instance, vfio-pci can use this to determine whether to expose multiple MSI vectors to the user. If we know we cannot ever support more than one vector, we have a better shot at the userspace driver working, especially if it's a guest OS, if we only expose one vector as being available in the interface.

Signed-off-by Alex Williamson <[email protected]>

---

 drivers/pci/msi.c   |    5 +++++
 include/linux/msi.h |    1 +
 2 files changed, 6 insertions(+)

drivers/pci/msi.c +0 -0

index 9fab30a..36b503a 100644

drivers/pci/msi.c int __weak arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) · line 82

Quoted 4 lines

drivers/pci/msi.c +1 -0

@@ -79,6 +79,11 @@ int __weak arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 	return 0;
 }
 
+bool __weak arch_supports_multivector_msi(struct pci_dev *dev)

Please not another weak arch function. We are in the process to reduce them not to extend them.

arch_supports is pretty much wrong here anyway. We are moving away from arch MSI dependencies simply because it's not a arch property per se.

Multi MSI is a property of the underlying interrupt controllers and there might be several MSI interrupt domains on a given system. They can or cannot support multi MSI.

Quoted 7 lines

The current x86 implementation is a tangled maze and Jiang is in the process to distangle it completely. Until thats done x86 is not going to add new ad hoc interfaces.

Once we converted everything over to the new hierarchical irqdomains we can add such an interface to the code, but for now I'm not accepting anything like that into x86 msi related code.

Ok, I guess I can do some ugliness with associating the IOMMU IRQ remapping capability with multivector MSI support within an #ifdef x86 block. Gross, but I think that's as accurate as I can get w/o a hook through the MSI code. Is there any target for the refactoring you mention? Thanks,

Alex

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

I'd like to make vfio-pci capable of manipulating the device exposed to the user such that if the host can only support a single MSI vector then we hide the fact that the device itself may actually be able to support more. When we virtualize PCI config space and interrupt setup there's no PCI protocol for the device failing to allocate the number of vectors that it said were available. If the userspace driver is a guest operating system, it certainly doesn't expect this to fail. I don't think we can ever guarantee that a multi-vector request will succeed, but we can certainly guarantee that it will fail if the platform doesn't support it.

An example device is the Atheros AR93xxx running in a Windows 7 VM. Both the device and the guest OS support multiple MSI vectors. With interrupt remapping, such that the host supports multivector, the device works well in the guest. With interrupt remapping disabled, the device is far less reliable because of the mismatch in MSI programming vs driver configuration and often fails. If vfio-pci can test whether multiple vectors are supported, then we can make it work reliably in both cases by adjusting the exposed MSI capability, like in this patch that would follow this series:

https://github.com/awilliam/linux-vfio/commit/9ace67515680

With this series, only x86 w/ interrupt remapping will advertise support for multiple MSI vectors. In surveying the code, I couldn't find any other archs that allowed it, but I'll take corrections if that's untrue. Thanks,

---

Alex Williamson (3):
PCI: Extend and export pci_msi_supported() for multivector MSI PCI/x86: Add arch_supports_multivector_msi() hook PCI/MSI: Initial hook for archs to declare multivector MSI support

 arch/x86/include/asm/x86_init.h |    1 +
 arch/x86/kernel/x86_init.c      |    6 ++++++
 drivers/iommu/irq_remapping.c   |    6 ++++++
 drivers/pci/msi.c               |   20 +++++++++++++++++---
 include/linux/msi.h             |    1 +
 include/linux/pci.h             |    3 +++
 6 files changed, 34 insertions(+), 3 deletions(-)

Quoted 23 lines · Alex Williamson

I'd like to make vfio-pci capable of manipulating the device exposed to the user such that if the host can only support a single MSI vector then we hide the fact that the device itself may actually be able to support more. When we virtualize PCI config space and interrupt setup there's no PCI protocol for the device failing to allocate the number of vectors that it said were available. If the userspace driver is a guest operating system, it certainly doesn't expect this to fail. I don't think we can ever guarantee that a multi-vector request will succeed, but we can certainly guarantee that it will fail if the platform doesn't support it.

An example device is the Atheros AR93xxx running in a Windows 7 VM. Both the device and the guest OS support multiple MSI vectors. With interrupt remapping, such that the host supports multivector, the device works well in the guest. With interrupt remapping disabled, the device is far less reliable because of the mismatch in MSI programming vs driver configuration and often fails. If vfio-pci can test whether multiple vectors are supported, then we can make it work reliably in both cases by adjusting the exposed MSI capability, like in this patch that would follow this series:

https://github.com/awilliam/linux-vfio/commit/9ace67515680

Quoted 4 lines

With this series, only x86 w/ interrupt remapping will advertise support for multiple MSI vectors. In surveying the code, I couldn't find any other archs that allowed it, but I'll take corrections if that's untrue. Thanks,

Per Thomas' comments and your possible workaround if we don't have pci_msi_supported(), I'm going to ignore these for now. Let me know if you disagree.