RFC vfio/pci: Back guest interrupts from Interrupt Message Store (IMS)
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v3 RFC Latest26 patches 13 repliesIn review
- v2 RFC18 patches 2 repliesSuperseded
- v1 RFC3 patches 6 repliesSuperseded
Changes in v3 from the cover letter
- RFC V2: https://lore.kernel.org/lkml/[email protected]/
- Still submiting this as RFC series. I believe that this now matches the expectatations raised during earlier reviews. If you agree this is
the right direction then I can drop the RFC prefix on next submission. If you do not agree then please do let me know where I missed expectations.
- First patch (PCI/MSI: Provide stubs for IMS functions)
has been submitted upstream separately and is queued for inclusion during the next merge window. I do still include it in this series to avoid the noise about issues that bots will find when checking this series without it included.
https://lore.kernel.org/lkml/169757242009.3135.5502383859327174030.tip-bot2@tip-bot2/
- Eliminated duplicate code between the PCI passthrough device backend and the IMS backend through more abstraction within the interrupt management frontend. (Kevin)
- Emulated interrupts are now managed by the interrupt management frontend and no longer unique to IMS. (Jason and Kevin)
- Since being an emulated interrupt is a persistent property there is
a new functional change to PCI interrupt management in that per-interrupt contexts (managed by frontend) are now persistent (they remain allocated until device release).
- Most of the patches from RFC V2 look the same with more patches
added to support the additional abstraction needed to eliminate the duplicate code. The IMS support was refactored to benefit from the
new abstraction. Please refer to individual patches for specific changes.
Patches in v3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/26 | PCI/MSI: Provide stubs for IMS functionsinclude/ | +26 | 0 replies | |
| 2/26 | vfio/pci: Move PCI specific check from wrapper to PCI functiondrivers/ | +4 | 0 replies | |
| 3/26 | vfio/pci: Use unsigned int instead of unsigneddrivers/ | +24 | 0 replies | |
| 4/26 | vfio/pci: Make core interrupt callbacks accessible to all virtual devicesdrivers/ drivers/ drivers/ +1 more | +34 | 0 replies | |
| 5/26 | vfio/pci: Split PCI interrupt management into front and backenddrivers/ drivers/ include/ | +55 | 0 replies | |
| 6/26 | vfio/pci: Separate MSI and MSI-X handlingdrivers/ include/ | +16 | 0 replies | |
| 7/26 | vfio/pci: Move interrupt eventfd to interrupt contextdrivers/ drivers/ include/ | +35 | 0 replies | |
| 8/26 | vfio/pci: Move mutex acquisition into functiondrivers/ drivers/ | +8 | 0 replies | |
| 9/26 | vfio/pci: Move per-interrupt contexts to generic interrupt structdrivers/ drivers/ include/ | +7 | 0 replies | |
| 10/26 | vfio/pci: Move IRQ type to generic interrupt contextdrivers/ drivers/ drivers/ +1 more | +16 | 0 replies | |
| 11/26 | vfio/pci: Provide interrupt context to irq_is() and is_irq_none()drivers/ | +12 | 0 replies | |
| 12/26 | vfio/pci: Provide interrupt context to generic opsdrivers/ | +21 | 0 replies | |
| 13/26 | vfio/pci: Provide interrupt context to vfio_msi_enable() and vfio_msi_disable()drivers/ | +11 | 0 replies | |
| 14/26 | vfio/pci: Let interrupt management backend interpret interrupt indexdrivers/ | +21 | 0 replies | |
| 15/26 | vfio/pci: Move generic code to frontenddrivers/ | +14 | 0 replies | |
| 16/26 | vfio/pci: Split interrupt context initializationdrivers/ | +14 | 0 replies | |
| 17/26 | vfio/pci: Make vfio_pci_set_irqs_ioctl() availabledrivers/ include/ | +4 | 0 replies | |
| 18/26 | vfio/pci: Preserve per-interrupt contextsdrivers/ | +29 | 0 replies | |
| 19/26 | vfio/pci: Store Linux IRQ number in per-interrupt contextdrivers/ | +7 | 0 replies | |
| 20/26 | vfio/pci: Separate frontend and backend code during interrupt enable/disabledrivers/ | +70 | 0 replies | |
| 21/26 | vfio/pci: Replace backend specific calls with callbacksdrivers/ include/ | +26 | 0 replies | |
| 22/26 | vfio/pci: Introduce backend specific context initializerdrivers/ include/ | +10 | 0 replies | |
| 23/26 | vfio/pci: Support emulated interruptsdrivers/ include/ | +88 | 0 replies | |
| 24/26 | vfio/pci: Add core IMS supportdrivers/ include/ | +185 | 0 replies | |
| 25/26 | vfio/pci: Add accessor for IMS indexdrivers/ include/ | +26 | 0 replies | |
| 26/26 | vfio/pci: Support IMS cookie modificationdrivers/ include/ | +56 | 0 replies |
Reviews trailers per patch
None of the 26 patches has a Reviewed-by or Acked-by.
| Patch | Dave Jiang |
|---|---|
| 1/26 | |
| 2/26 | |
| 3/26 | |
| 4/26 | |
| 5/26 | |
| 6/26 | |
| 7/26 | |
| 8/26 | |
| 9/26 | |
| 10/26 | |
| 11/26 | |
| 12/26 | |
| 13/26 | |
| 14/26 | |
| 15/26 | |
| 16/26 | |
| 17/26 | |
| 18/26 | |
| 19/26 | |
| 20/26 | |
| 21/26 | |
| 22/26 | |
| 23/26 | Originally-by |
| 24/26 | |
| 25/26 | Originally-by |
| 26/26 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [RFC PATCH V3 00/26] vfio/pci: Back guest interrupts from Interrupt Message Store (IMS)
Changes since RFC V2:
- RFC V2: https://lore.kernel.org/lkml/[email protected]/
- Still submiting this as RFC series. I believe that this now matches the expectatations raised during earlier reviews. If you agree this is
the right direction then I can drop the RFC prefix on next submission. If you do not agree then please do let me know where I missed expectations.
- First patch (PCI/MSI: Provide stubs for IMS functions)
has been submitted upstream separately and is queued for inclusion during the next merge window. I do still include it in this series to avoid the noise about issues that bots will find when checking this series without it included.
https://lore.kernel.org/lkml/169757242009.3135.5502383859327174030.tip-bot2@tip-bot2/
- Eliminated duplicate code between the PCI passthrough device backend and the IMS backend through more abstraction within the interrupt management frontend. (Kevin)
- Emulated interrupts are now managed by the interrupt management frontend and no longer unique to IMS. (Jason and Kevin)
- Since being an emulated interrupt is a persistent property there is
a new functional change to PCI interrupt management in that per-interrupt contexts (managed by frontend) are now persistent (they remain allocated until device release).
- Most of the patches from RFC V2 look the same with more patches
added to support the additional abstraction needed to eliminate the duplicate code. The IMS support was refactored to benefit from the
new abstraction. Please refer to individual patches for specific changes.
Changes since RFC V1:
- RFC V1: https://lore.kernel.org/lkml/[email protected]/
- This is a complete rewrite based on feedback from Jason and Kevin. Primarily the transition is to make IMS a new backend of MSI-X
emulation: VFIO PCI transitions to be an interrupt management frontend with existing interrupt management for PCI passthrough devices as a backend and IMS interrupt management introduced as a new backend.
The first part of the series splits VFIO PCI interrupt
management into a "frontend" and "backend" with the existing PCI interrupt management as its first backend. The second part of the series adds IMS interrupt management as a new interrupt management backend.
This is a significant change from RFC V1 as well as in the impact of the changes on existing VFIO PCI. This was done in response to feedback that I hope I understood as intended. If I did not get it right, please do point out to me where I went astray and I'd be
happy to rewrite. Of course, suggestions for improvement will
be much appreciated.
Hi Everybody,
With Interrupt Message Store (IMS) support introduced in commit 0194425af0c8 ("PCI/MSI: Provide IMS (Interrupt Message Store) support") a device can create a secondary interrupt domain that works side by side with MSI-X on the same device. IMS allows for implementation-specific interrupt storage that is managed by the implementation specific interrupt chip associated with the IMS domain at the time it (the IMS domain) is created for the device via pci_create_ims_domain().
An example usage of IMS is for devices that can have their resources assigned to guests with varying granularity. For example, an accelerator device may support many workqueues and a single workqueue can be composed into a virtual device for use by a guest. Using IMS interrupts for the guest preserves MSI-X for host usage while allowing a significantly larger number of interrupt vectors than allowed by MSI-X. All while enabling usage of the same device driver within the host and guest.
This series introduces IMS support to VFIO PCI for use by virtual devices that support MSI-X interrupts that are backed by IMS interrupts on the host. Specifically, that means that when the virtual device's VFIO_DEVICE_SET_IRQS ioctl() receives a "trigger interrupt" (VFIO_IRQ_SET_ACTION_TRIGGER) for a MSI-X index then VFIO PCI IMS allocates/frees an IMS interrupt on the host.
VFIO PCI assumes that it is managing interrupts of a passthrough PCI device. Split VFIO PCI into a "frontend" and "backend" to support interrupt management for virtual devices that are not passthrough PCI devices. The VFIO PCI frontend directs guest requests to the appropriate backend. Existing interrupt management for passthrough PCI devices is the first backend, guest MSI-X interrupts backed by IMS interrupts on the host is the new backend (VFIO PCI IMS).
An IMS interrupt is allocated via pci_ims_alloc_irq() that requires an implementation specific cookie that is opaque to VFIO PCI IMS. This can be a PASID, queue ID, pointer etc. During initialization VFIO PCI IMS learns which PCI device to operate on and what the default cookie should be for any new interrupt allocation. VFIO PCI IMS can also associate a unique cookie with each vector.
Guests may access a virtual device via both 'direct-path', where the guest interacts directly with the underlying hardware, and 'intercepted path', where the virtual device emulates operations. VFIO PCI supports emulated interrupts (better naming suggestions are welcome) to handle 'intercepted path' operations where completion interrupts are signaled from the virtual device, not the underlying hardware.
This has been tested with a yet to be published VFIO driver for the Intel Data Accelerators (IDXD) present in Intel Xeon CPUs.
While this series contains a working implementation it is presented as an RFC with the goal to obtain feedback on whether VFIO PCI IMS is appropriate for inclusion into VFIO and whether it is (or could be adapted to be) appropriate for support of other planned IMS usages you may be aware of.
Any feedback will be greatly appreciated.
Reinette
Reinette Chatre (26):
PCI/MSI: Provide stubs for IMS functions
vfio/pci: Move PCI specific check from wrapper to PCI function vfio/pci: Use unsigned int instead of unsigned
vfio/pci: Make core interrupt callbacks accessible to all virtual
devices
vfio/pci: Split PCI interrupt management into front and backend vfio/pci: Separate MSI and MSI-X handling
vfio/pci: Move interrupt eventfd to interrupt context vfio/pci: Move mutex acquisition into function
vfio/pci: Move per-interrupt contexts to generic interrupt struct vfio/pci: Move IRQ type to generic interrupt context vfio/pci: Provide interrupt context to irq_is() and is_irq_none() vfio/pci: Provide interrupt context to generic ops vfio/pci: Provide interrupt context to vfio_msi_enable() and
vfio_msi_disable()
vfio/pci: Let interrupt management backend interpret interrupt index vfio/pci: Move generic code to frontend
vfio/pci: Split interrupt context initialization vfio/pci: Make vfio_pci_set_irqs_ioctl() available vfio/pci: Preserve per-interrupt contexts
vfio/pci: Store Linux IRQ number in per-interrupt context vfio/pci: Separate frontend and backend code during interrupt
enable/disable
vfio/pci: Replace backend specific calls with callbacks vfio/pci: Introduce backend specific context initializer vfio/pci: Support emulated interrupts
vfio/pci: Add core IMS support
vfio/pci: Add accessor for IMS index
vfio/pci: Support IMS cookie modification
drivers/vfio/pci/vfio_pci_config.c | 2 +- drivers/vfio/pci/vfio_pci_core.c | 50 +- drivers/vfio/pci/vfio_pci_intrs.c | 758 ++++++++++++++++++++++++----- drivers/vfio/pci/vfio_pci_priv.h | 2 +- include/linux/pci.h | 34 +- include/linux/vfio_pci_core.h | 87 +++- 6 files changed, 756 insertions(+), 177 deletions(-)
base-commit: 611da07b89fdd53f140d7b33013f255bf0ed8f34
Signature · 1 line
2.34.1