PCI/VMD: White list for fast interrupt handlers
PATCH 1/1 PCI/VMD: White list for fast interrupt handlers
Devices with slow interrupt handlers are significantly harming performance when their interrupt vector is shared with a fast device. This patch creates a class code white list for devices with known fast interrupt handlers, and all other devices will share a single vector so they don't interfere with performance.
At the moment, only the NVM Express class code is on the list, but more may be added if VMD users desire to use other low-latency devices in these domains.
--- drivers/pci/host/vmd.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/pci/host/vmd.c +12 -1
@@ -175,9 +175,20 @@ static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *dint i, best =1 ;unsigned long flags; -if (pci_is_bridge (msi_desc_to_pci_dev (desc)) || vmd->msix_count ==1 ) +if (vmd->msix_count ==1 )return &vmd->irqs[0 ]; +/* +* White list for fast-interrupt handlers. All others will share the +* "slow" interrupt vector. +*/ +switch (msi_desc_to_pci_dev (desc)->class) { +case PCI_CLASS_STORAGE_EXPRESS : +break ; +default : +return &vmd->irqs[0 ]; + } +raw_spin_lock_irqsave (&list_lock, flags);for (i =1 ; i < vmd->msix_count; i++)if (vmd->irqs[i].count < vmd->irqs[best].count)
Signature · 1 line
2.14.3
Quoted 9 lines · Keith Busch
Devices with slow interrupt handlers are significantly harming performance when their interrupt vector is shared with a fast device. This patch creates a class code white list for devices with known fast interrupt handlers, and all other devices will share a single vector so they don't interfere with performance.
At the moment, only the NVM Express class code is on the list, but more may be added if VMD users desire to use other low-latency devices in these domains.
I think this is far too much of a hack. Just don't use VMD if your
care about performance.
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Keith Busch
Devices with slow interrupt handlers are significantly harming performance when their interrupt vector is shared with a fast device. This patch creates a class code white list for devices with known fast interrupt handlers, and all other devices will share a single vector so they don't interfere with performance.
Quoted 6 lines
Quoted 3 lines
At the moment, only the NVM Express class code is on the list, but more may be added if VMD users desire to use other low-latency devices in these domains.
I think this is far too much of a hack. Just don't use VMD if your
care about performance.
I'm not aware of an easier way you can direct-assign an entire PCIe domain to a virtual machine. :)
Derrick, Jonathan
Acked-byQuoted 18 lines · Keith Busch
Quoted 17 lines · Christoph Hellwig
Quoted 15 lines · Keith Busch
Devices with slow interrupt handlers are significantly harming performance
when their interrupt vector is shared with a fast device. This patch
creates a class code white list for devices with known fast interrupt
handlers, and all other devices will share a single vector so they don't
interfere with performance.At the moment, only the NVM Express class code is on the list, but more
may be added if VMD users desire to use other low-latency devices in
these domains.
Quoted 7 lines
Quoted 3 lines
I think this is far too much of a hack. Just don't use VMD if
your
care about performance.I'm not aware of an easier way you can direct-assign an entire PCIe domain
to a virtual machine. :)
It's fine with me
K0xvcmVuem8NCg0KT24gRnJpLCAyMDE4LTA1LTExIGF0IDA5OjM5IC0wNjAwLCBKb25hdGhhbiBE ZXJyaWNrIHdyb3RlOg0KPiBPbiBXZWQsIDIwMTgtMDUtMDkgYXQgMDk6MjYgLTA2MDAsIEtlaXRo IEJ1c2NoIHdyb3RlOg0KPiA+IE9uIFR1ZSwgTWF5IDA4LCAyMDE4IGF0IDA5OjM4OjI4UE0gLTA3 MDAsIENocmlzdG9waCBIZWxsd2lnIHdyb3RlOg0KPiA+ID4gT24gVHVlLCBNYXkgMDgsIDIwMTgg YXQgMTA6MDA6MjJBTSAtMDYwMCwgS2VpdGggQnVzY2ggd3JvdGU6DQo+ID4gPiA+IERldmljZXMg d2l0aCBzbG93IGludGVycnVwdCBoYW5kbGVycyBhcmUgc2lnbmlmaWNhbnRseSBoYXJtaW5nDQo+ ID4gPiA+IHBlcmZvcm1hbmNlDQo+ID4gPiA+IHdoZW4gdGhlaXIgaW50ZXJydXB0IHZlY3RvciBp cyBzaGFyZWQgd2l0aCBhIGZhc3QgZGV2aWNlLiBUaGlzDQo+ID4gPiA+IHBhdGNoDQo+ID4gPiA+ IGNyZWF0ZXMgYSBjbGFzcyBjb2RlIHdoaXRlIGxpc3QgZm9yIGRldmljZXMgd2l0aCBrbm93biBm YXN0DQo+ID4gPiA+IGludGVycnVwdA0KPiA+ID4gPiBoYW5kbGVycywgYW5kIGFsbCBvdGhlciBk ZXZpY2VzIHdpbGwgc2hhcmUgYSBzaW5nbGUgdmVjdG9yIHNvDQo+ID4gPiA+IHRoZXkgZG9uJ3QN Cj4gPiA+ID4gaW50ZXJmZXJlIHdpdGggcGVyZm9ybWFuY2UuDQo+ID4gPiA+IA0KPiA+ID4gPiBB dCB0aGUgbW9tZW50LCBvbmx5IHRoZSBOVk0gRXhwcmVzcyBjbGFzcyBjb2RlIGlzIG9uIHRoZSBs aXN0LA0KPiA+ID4gPiBidXQgbW9yZQ0KPiA+ID4gPiBtYXkgYmUgYWRkZWQgaWYgVk1EIHVzZXJz IGRlc2lyZSB0byB1c2Ugb3RoZXIgbG93LWxhdGVuY3kNCj4gPiA+ID4gZGV2aWNlcw0KPiA+ID4g PiBpbg0KPiA+ID4gPiB0aGVzZSBkb21haW5zLg0KPiA+ID4gDQo+ID4gPiBJIHRoaW5rIHRoaXMg aXMgZmFyIHRvbyBtdWNoIG9mIGEgaGFjay4gICBKdXN0IGRvbid0IHVzZSBWTUQgaWYNCj4gPiA+ IHlvdXINCj4gPiA+IGNhcmUgYWJvdXQgcGVyZm9ybWFuY2UuDQo+ID4gDQo+ID4gSSdtIG5vdCBh d2FyZSBvZiBhbiBlYXNpZXIgd2F5IHlvdSBjYW4gZGlyZWN0LWFzc2lnbiBhbiBlbnRpcmUgUENJ ZQ0KPiA+IGRvbWFpbg0KPiA+IHRvIGEgdmlydHVhbCBtYWNoaW5lLiA6KQ0KPiANCj4gSXQncyBm aW5lIHdpdGggbWUNCj4gDQo+IEFja2VkLWJ5OiBKb24gRGVycmljazogPGpvbmF0aGFuLmRlcnJp Y2tAaW50ZWwuY29tPg0KDQoNCldlJ3ZlIHNlZW4gdGhpcyBhY3R1YWxseSBmaXggYW4gaXNzdWUg d2l0aCBvbmUgdmVuZG9yJ3MgbXVsdGktZnVuY3Rpb24NCnN3aXRjaC4NCkknZCBsaWtlIHRvIHNl ZSBpdCBnZXQgaW50byA0LjE4IGlmIHBvc3NpYmxl
Quoted 31 lines · Jonathan
+Lorenzo
Quoted 28 lines · Jonathan Derrick
Quoted 26 lines · Keith Busch
Quoted 18 lines · Christoph Hellwig
Quoted 10 lines · Keith Busch
Devices with slow interrupt handlers are significantly harming performance
when their interrupt vector is shared with a fast device. This patch
creates a class code white list for devices with known fast interrupt
handlers, and all other devices will share a single vector so they don't
interfere with performance.Quoted 6 lines
At the moment, only the NVM Express class code is on the list, but more
may be added if VMD users desire to use other low-latency devices
in
these domains.Quoted 3 lines
I think this is far too much of a hack. Just don't use VMD if
your
care about performance.I'm not aware of an easier way you can direct-assign an entire PCIe domain
to a virtual machine. :)
Quoted 8 lines
Quoted 3 lines
It's fine with me
Acked-by Jon Derrick: <[email protected]>We've seen this actually fix an issue with one vendor's multi-function switch.
I'd like to see it get into 4.18 if possible
Sorry for the delay in getting back to this.
It seems like Christoph is not too happy about this patch, it is your code so I would apply it unless there is a cleaner alternative so please do let me know.
Thanks,
Lorenzo
Quoted 9 lines · Lorenzo Pieralisi
Quoted 2 lines · Jonathan
We've seen this actually fix an issue with one vendor's multi-function switch. I'd like to see it get into 4.18 if possible
Sorry for the delay in getting back to this.
It seems like Christoph is not too happy about this patch, it is your code so I would apply it unless there is a cleaner alternative so please do let me know.
VMD hardware really relies on the host software to bring all the logic, and it was intended to prioritize PCIe attached storage. The patch just captures that intention, so yes please, let's queue this up.
I understands Christoph's lack of enthusiasm for this, but at least this isn't a PCIe HBA with proprietary interfaces and firmware. :)
Lorenzo Pieralisi
AppliedQuoted 11 lines · Keith Busch
Quoted 9 lines · Lorenzo Pieralisi
Quoted 2 lines · Jonathan
We've seen this actually fix an issue with one vendor's multi-function switch. I'd like to see it get into 4.18 if possible
Sorry for the delay in getting back to this.
It seems like Christoph is not too happy about this patch, it is your code so I would apply it unless there is a cleaner alternative so please do let me know.
Quoted 6 lines
VMD hardware really relies on the host software to bring all the logic, and it was intended to prioritize PCIe attached storage. The patch just captures that intention, so yes please, let's queue this up.
I understands Christoph's lack of enthusiasm for this, but at least this isn't a PCIe HBA with proprietary interfaces and firmware. :)
Applied to pci/vmd with a slightly updated changelog, thanks.
Lorenzo
Quoted 2 lines · Keith Busch
I'm not aware of an easier way you can direct-assign an entire PCIe domain to a virtual machine. :)
libvirt will handle it just fine for you.