vfio-pci: Fix Nvidia MSI ACK through 0x88000 quirk
PATCH 1/1 vfio-pci: Fix Nvidia MSI ACK through 0x88000 quirk
When MSI is enabled on Nvidia GeForce cards the driver seems to acknowledge the interrupt by writing a 0xff byte to the MSI capability ID register using the PCI config space mirror at offset 0x88000 from BAR0. Without this, the device will only fire a single interrupt. VFIO handles the PCI capability ID/next registers as virtual w/o write support, so any write through config space is currently dropped. Add a check for this and allow the write through the BAR window. The registers are read-only anyway.
---
We might be able to do something with an ioeventfd to make the QEMU component of this asynchronous... tbd.
hw/misc/vfio.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
hw/misc/vfio.c +29 -1
@@ -1811,6 +1811,34 @@ static void vfio_probe_nvidia_bar5_window_quirk(VFIODevice *vdev, int nr) vdev->host.function); } +static void vfio_nvidia_88000_quirk_write (void *opaque,hwaddr addr, +uint64_t data,unsigned size) +{ +VFIOQuirk *quirk = opaque; +VFIODevice *vdev = quirk->vdev; +PCIDevice *pdev = &vdev->pdev; +hwaddr base = quirk->data.address_match &TARGET_PAGE_MASK ; + +vfio_generic_quirk_write (opaque, addr, data, size); + +/* +* Nvidia seems to acknowledge MSI interrupts by writing 0xff to the +* MSI capability ID register. Both the ID and next register are +* read-only, so we allow writes covering either of those to real hw. +* NB - only fixed for the 0x88000 MMIO window. +*/ +if ((pdev->cap_present &QEMU_PCI_CAP_MSI ) && +vfio_range_contained (addr, size, pdev->msi_cap,PCI_MSI_FLAGS )) { +vfio_bar_write (&vdev->bars[quirk->data.bar], addr + base, data, size); + } +} + +static const MemoryRegionOps vfio_nvidia_88000_quirk = { + .read = vfio_generic_quirk_read, + .write = vfio_nvidia_88000_quirk_write, + .endianness =DEVICE_LITTLE_ENDIAN , +}; +/* * Finally, BAR0 itself. We want to redirect any accesses to either * 0x1800 or 0x88000 through the PCI config space access functions.
@@ -1837,7 +1865,7 @@ static void vfio_probe_nvidia_bar0_88000_quirk(VFIODevice *vdev, int nr) quirk->data.address_mask =PCIE_CONFIG_SPACE_SIZE -1 ; quirk->data.bar = nr; -memory_region_init_io (&quirk->mem,OBJECT (vdev), &vfio_generic_quirk, +memory_region_init_io (&quirk->mem,OBJECT (vdev), &vfio_nvidia_88000_quirk, quirk,"vfio-nvidia-bar0-88000-quirk" ,TARGET_PAGE_ALIGN (quirk->data.address_mask +1 )); memory_region_add_subregion_overlap(&vdev->bars[nr].mem,
Quoted 8 lines · Alex Williamson
When MSI is enabled on Nvidia GeForce cards the driver seems to acknowledge the interrupt by writing a
0xffbyte to the MSI capability ID register using the PCI config space mirror at offset0x88000from BAR0. Without this, the device will only fire a single interrupt. VFIO handles the PCI capability ID/next registers as virtual w/o write support, so any write through config space is currently dropped. Add a check for this and allow the write through the BAR window. The registers are read-only anyway.
This is only half the truth, I'm afraid if I'm right its much worse than that.
At least on some GPUs the MSI ack is done via PCI config space itself, and on some its done via the mirror, and yes it matters on some cards which way it works.
Dave.
Quoted 11 lines · Dave Airlie
Quoted 8 lines · Alex Williamson
When MSI is enabled on Nvidia GeForce cards the driver seems to acknowledge the interrupt by writing a
0xffbyte to the MSI capability ID register using the PCI config space mirror at offset0x88000from BAR0. Without this, the device will only fire a single interrupt. VFIO handles the PCI capability ID/next registers as virtual w/o write support, so any write through config space is currently dropped. Add a check for this and allow the write through the BAR window. The registers are read-only anyway.
Quoted 5 lines
This is only half the truth, I'm afraid if I'm right its much worse than that.
At least on some GPUs the MSI ack is done via PCI config space itself, and on some its done via the mirror, and yes it matters on some cards which way it works.
I was hoping that wouldn't be the case since it seems fairly universal that PCI config space access should be considered slow and avoided for things like this. But, I suppose with MMConfig it's no worse than device MMIO space.
On my GTX660 I did actually test both paths. The existing quirk converts all accesses into the config space mirror into proper config space accesses, so I changed the kernel to not drop the write. Then I figured I'd rather solve this in QEMU using the mirror directly. So in my case it didn't matter which path.
I'll keep this in mind though, I may want to enable kernel support for this first and advertise it via a flag on the config space region, then I can avoid doing a double write in QEMU and really breaking the state machine. Thanks for the feedback,
Alex
Quoted 14 lines · Alex Williamson
Quoted 13 lines · Dave Airlie
Quoted 8 lines · Alex Williamson
When MSI is enabled on Nvidia GeForce cards the driver seems to acknowledge the interrupt by writing a
0xffbyte to the MSI capability ID register using the PCI config space mirror at offset0x88000from BAR0. Without this, the device will only fire a single interrupt. VFIO handles the PCI capability ID/next registers as virtual w/o write support, so any write through config space is currently dropped. Add a check for this and allow the write through the BAR window. The registers are read-only anyway.This is only half the truth, I'm afraid if I'm right its much worse than that.
Quoted 8 lines
Quoted 3 lines
At least on some GPUs the MSI ack is done via PCI config space itself, and on some its done via the mirror, and yes it matters on some cards which way it works.
I was hoping that wouldn't be the case since it seems fairly universal that PCI config space access should be considered slow and avoided for things like this. But, I suppose with MMConfig it's no worse than device MMIO space.
For reference dig around in here
http://www.mail-archive.com/[email protected]/msg14437.html
nvidia commented on it as well, it may be they require the slowness or just some other coherency issue.
Dave.