Skip to content

KVM: Improve IRQ assignment for device passthrough

Jan Kiszka kvm 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest9 patches 25 repliesIn review
  2. v24 patches 20 repliesSuperseded

Changes in v4 from the cover letter

- Allow user space to push its INTx mask state to the kernel and respect this on IRQ masking and delivery
- Specify IRQF_ONESHOT for threaded IRQ of assigned device
- Switch IRQ subsystem locking from RCU to SRCU
- Fix for SRCU struct leakage
- Small cleanup for kvm_vm_ioctl_assigned_device
- Documentation for device assignment API (please check carefully if I got it right)

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9KVM: Fix srcu struct leakagevirt/kvm/kvm_main.c+7 -82 replies
2/9KVM: Switch IRQ subsystem to SRCUinclude/linux/kvm_host.h virt/kvm/irq_comm.c virt/kvm/kvm_main.c+23 -142 replies
3/9KVM: Clear assigned guest IRQ on releasevirt/kvm/assigned-dev.c+3 -05 replies
4/9KVM: Switch assigned device IRQ forwarding to threaded handlerinclude/linux/kvm_host.h virt/kvm/assigned-dev.c+36 -832 replies
5/9KVM: Refactor IRQ names of assigned devicesinclude/linux/kvm_host.h virt/kvm/assigned-dev.c+7 -50 replies
6/9KVM: Save/restore state of assigned PCI devicevirt/kvm/assigned-dev.c+4 -16 replies
7/9KVM: Clean up kvm_vm_ioctl_assigned_devicevirt/kvm/assigned-dev.c+4 -50 replies
8/9KVM: Document device assigment APIDocumentation/kvm/api.txt+178 -00 replies
9/9KVM: Allow host IRQ sharing for passed-through PCI 2.3 devicesDocumentation/kvm/api.txt arch/x86/kvm/x86.c include/linux/kvm.h +2 more+299 -235 replies

Reviews trailers per patch

1 of 9 patches has a Reviewed-by or Acked-by.

PatchMichael S. Tsirkin
1/9A-b
2/9
3/9
4/9
5/9
6/9
7/9
8/9
9/9

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v4 thread

Cover letter [PATCH v4 0/9] KVM: Improve IRQ assignment for device passthrough

Nine patches (yeah, it's getting more and more) to improve "classic" device assigment /wrt IRQs. Highlight is the last one that resolves the host IRQ sharing issue for all PCI 2.3 devices. Quite essential when passing non-MSI-ready devices like many USB host controllers.

As there were concerns regarding the overhead of IRQ masking via the PCI config space, I did some micro-benchmarks. Well, the concerns are valid:

disable_irq_nosync:           ~600 cycles
pci_2_3_irq_check_and_mask:  ~6000 cycles (EHCI)
                            ~22000 cycles (AR9287, with peaks >100000)

Specifically the varying impact of the device like in the Atheros case is worrying (this device is actually known to cause horrible latencies to the host, but who knows what other devices do). So I decided to go with PCI-2.3 masking as default off in the to-be-sent qemu-kvm patch. Maybe something to consider vor VFIO as well.

Changes in v4:
- Allow user space to push its INTx mask state to the kernel and respect this on IRQ masking and delivery
- Specify IRQF_ONESHOT for threaded IRQ of assigned device
- Switch IRQ subsystem locking from RCU to SRCU
- Fix for SRCU struct leakage
- Small cleanup for kvm_vm_ioctl_assigned_device
- Documentation for device assignment API (please check carefully if I got it right)

Changes in v3:
- Save/restore PCI device state across guest usage
- Make PCI-2.3-based IRQ sharing configurable by user space
- Fix potential nesting of pci_block_user_cfg_access
- Do not track PCI-level IRQ masking in software, user space may change it concurrently
- Optimize kvm_assigned_dev_ack_irq for the case another IRQ is already pending
- Cleanups according to review comments

Changes in v2:
- Reworked IRQ forwarding path to use threaded IRQs (direct signalling from IRQ context does not work out of the box and may be too lengthy)
- Refactored host IRQ naming of assigned devices (cosmetic change)
- Avoid unmask on ack when the next IRQ is pending, rather reassert the guest line (PCI-2.3 patch)
- Refactored PCI-2.3 patch (but still no control knob for shared mode - is that a must?)

Jan Kiszka (9):
KVM: Fix srcu struct leakage
KVM: Switch IRQ subsystem to SRCU
KVM: Clear assigned guest IRQ on release
KVM: Switch assigned device IRQ forwarding to threaded handler
KVM: Refactor IRQ names of assigned devices
KVM: Save/restore state of assigned PCI device
KVM: Clean up kvm_vm_ioctl_assigned_device
KVM: Document device assigment API
KVM: Allow host IRQ sharing for passed-through PCI 2.3 devices

 Documentation/kvm/api.txt |  203 ++++++++++++++++++++++++
 arch/x86/kvm/x86.c        |    1 +
 include/linux/kvm.h       |    6 +
 include/linux/kvm_host.h  |   16 +--
 virt/kvm/assigned-dev.c   |  383 ++++++++++++++++++++++++++++++++++----------
 virt/kvm/irq_comm.c       |   31 ++--
 virt/kvm/kvm_main.c       |   20 ++-
 7 files changed, 541 insertions(+), 119 deletions(-)