vfio/pci: Fix up breakage against split irqchip and INTx
v2 cover vfio/pci: Fix up breakage against split irqchip and INTx
v2:
- pick tags
- don't register resamplefd with KVM kernel when the userspace resamplefd path is enabled (should enable fast path on new kernels)
- fix resamplefd mem leak
- fix commit message of patch 4 [Eric]
- let kvm_resample_fd_notify() return a boolean, skip ioapic check if returned true
- more comments here and there in the code to state the fact that userspace ioapic irr & remote-irr are bypassed [Paolo]
VFIO INTx is not working with split irqchip. On new kernels KVM_IRQFD will directly fail with resamplefd attached so QEMU will automatically fallback to the INTx slow path. However on old kernels it's still broken.
Only until recently I noticed that this could also break PXE boot for assigned NICs [1]. My wild guess is that the PXE ROM will be mostly using INTx as well, which means we can't bypass that even if we enables MSI for the guest kernel.
This series tries to first fix this issue function-wise, then speed up for the INTx again with resamplefd (mostly following the ideas proposed by Paolo one year ago [2]). My TCP_RR test shows that:
- Before this series: this is broken, no number to show
- After patch 1 (enable slow path): get 63% perf comparing to full kernel irqchip
- After whole series (enable fast path partly, irq injection will be the same as fast path, however userspace needs to intercept for EOI broadcast to resamplefd, though should still be faster than the MMIO trick for intx eoi): get 93% perf comparing to full kernel irqchip, which is a 46% performance boost
I think we can consider to apply patch 1 even sooner than the rest of the series to unbreak intx+split first.
The whole test matrix for reference:
|----------+---------+-------------------+--------------+--------------------| | IRQ type | irqchip | TCP_STREAM (Gbps) | TCP_RR (pps) | note | |----------+---------+-------------------+--------------+--------------------| | msi | on | 9.39 | 17567 | | | nomsi | on | 9.29 | 14056 | | | msi | split | 9.36 | 17330 | | | nomsi | split | / | / | currently broken | | nomsi | split | 8.98 | 8977 | after patch 1 | | nomsi | split | 9.21 | 13142 | after whole series | |----------+---------+-------------------+--------------+--------------------|
Any review comment is welcomed. Thanks,
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1786404
[2] https://patchwork.kernel.org/patch/10738541/#22609933
Peter Xu (5):
vfio/pci: Disable INTx fast path if using split irqchip vfio/pci: Use kvm_irqchip_add_irqfd_notifier_gsi() for irqfds
KVM: Pass EventNotifier into kvm_irqchip_assign_irqfd
KVM: Kick resamplefd for split kernel irqchip
Revert "vfio/pci: Disable INTx fast path if using split irqchip"
accel/kvm/kvm-all.c | 101 +++++++++++++++++++++++++++++++++++++---- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++- hw/vfio/pci.c | 37 ++++++--------- include/sysemu/kvm.h | 7 +++ 5 files changed, 137 insertions(+), 32 deletions(-)
Signature · 1 line
2.24.1
PATCH v2 1/5 vfio/pci: Disable INTx fast path if using split irqchip
It's currently broken. Let's use the slow path to at least make it functional.
--- hw/vfio/pci.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
hw/vfio/pci.c +12 -0
@@ -128,6 +128,18 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp)return ; } +if (kvm_irqchip_is_split ()) { +/* +* VFIO INTx is currently not working with split kernel +* irqchip for level triggered interrupts. Go the slow path +* as long as split is enabled so we can be at least +* functional (even with poor performance). +* +* TODO: Remove this after all things fixed up. +*/ +return ; + } +/* Get to a known interrupt state */ qemu_set_fd_handler (irqfd.fd,NULL ,NULL , vdev);vfio_mask_single_irqindex (&vdev->vbasedev,VFIO_PCI_INTX_IRQ_INDEX );
Signature · 1 line
2.24.1
PATCH v2 2/5 vfio/pci: Use kvm_irqchip_add_irqfd_notifier_gsi() for irqfds
In reply to Peter XuVFIO is currently the only one left that is not using the generic function (kvm_irqchip_add_irqfd_notifier_gsi()) to register irqfds. Let VFIO use the common framework too.
Follow up patches will introduce extra features for kvm irqfd, so that VFIO can easily leverage that after the switch.
--- hw/vfio/pci.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-)
hw/vfio/pci.c +15 -22
@@ -115,11 +115,7 @@ static void vfio_intx_eoi(VFIODevice *vbasedev)static void vfio_intx_enable_kvm (VFIOPCIDevice *vdev,Error **errp) {#ifdef CONFIG_KVM -struct kvm_irqfd irqfd = { - .fd =event_notifier_get_fd (&vdev->intx.interrupt), - .gsi = vdev->intx.route.irq, - .flags =KVM_IRQFD_FLAG_RESAMPLE , - }; +int irq_fd =event_notifier_get_fd (&vdev->intx.interrupt);Error *err =NULL ; if (vdev->no_kvm_intx || !kvm_irqfds_enabled() ||
@@ -141,7 +137,7 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) }/* Get to a known interrupt state */ -qemu_set_fd_handler (irqfd.fd,NULL ,NULL , vdev); +qemu_set_fd_handler (irq_fd,NULL ,NULL , vdev);vfio_mask_single_irqindex (&vdev->vbasedev,VFIO_PCI_INTX_IRQ_INDEX ); vdev->intx.pending = false;pci_irq_deassert (&vdev->pdev);
@@ -152,17 +148,18 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) goto fail; } -/* KVM triggers it, VFIO listens for it */ - irqfd.resamplefd =event_notifier_get_fd (&vdev->intx.unmask); - -if (kvm_vm_ioctl (kvm_state,KVM_IRQFD , &irqfd)) { +if (kvm_irqchip_add_irqfd_notifier_gsi (kvm_state, + &vdev->intx.interrupt, + &vdev->intx.unmask, + vdev->intx.route.irq)) {error_setg_errno (errp, errno,"failed to setup resample irqfd" ); goto fail_irqfd; }if (vfio_set_irq_signaling (&vdev->vbasedev,VFIO_PCI_INTX_IRQ_INDEX ,0 ,VFIO_IRQ_SET_ACTION_UNMASK , - irqfd.resamplefd, &err)) { +event_notifier_get_fd (&vdev->intx.unmask), + &err)) {error_propagate (errp, err); goto fail_vfio; }
@@ -177,12 +174,12 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp)return ; fail_vfio: - irqfd.flags =KVM_IRQFD_FLAG_DEASSIGN ; -kvm_vm_ioctl (kvm_state,KVM_IRQFD , &irqfd); +kvm_irqchip_remove_irqfd_notifier_gsi (kvm_state, &vdev->intx.interrupt, + vdev->intx.route.irq); fail_irqfd:event_notifier_cleanup (&vdev->intx.unmask); fail: -qemu_set_fd_handler (irqfd.fd, vfio_intx_interrupt,NULL , vdev); +qemu_set_fd_handler (irq_fd, vfio_intx_interrupt,NULL , vdev);vfio_unmask_single_irqindex (&vdev->vbasedev,VFIO_PCI_INTX_IRQ_INDEX );#endif }
@@ -190,12 +187,6 @@ fail:static void vfio_intx_disable_kvm (VFIOPCIDevice *vdev) {#ifdef CONFIG_KVM -struct kvm_irqfd irqfd = { - .fd =event_notifier_get_fd (&vdev->intx.interrupt), - .gsi = vdev->intx.route.irq, - .flags =KVM_IRQFD_FLAG_DEASSIGN , - }; -if (!vdev->intx.kvm_accel) {return ; }
@@ -209,7 +200,8 @@ static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev)pci_irq_deassert (&vdev->pdev);/* Tell KVM to stop listening for an INTx irqfd */ -if (kvm_vm_ioctl (kvm_state,KVM_IRQFD , &irqfd)) { +if (kvm_irqchip_remove_irqfd_notifier_gsi (kvm_state, &vdev->intx.interrupt, + vdev->intx.route.irq)) {error_report ("vfio: Error: Failed to disable INTx irqfd: %m" ); }
@@ -217,7 +209,8 @@ static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev)event_notifier_cleanup (&vdev->intx.unmask);/* QEMU starts listening for interrupt events. */ -qemu_set_fd_handler (irqfd.fd, vfio_intx_interrupt,NULL , vdev); +qemu_set_fd_handler (event_notifier_get_fd (&vdev->intx.interrupt), + vfio_intx_interrupt,NULL , vdev); vdev->intx.kvm_accel = false;
Signature · 1 line
2.24.1
PATCH v2 3/5 KVM: Pass EventNotifier into kvm_irqchip_assign_irqfd
In reply to Peter XuSo that kvm_irqchip_assign_irqfd() can have access to the EventNotifiers, especially the resample event. It is needed in follow up patch to cache and kick resamplefds from QEMU.
--- accel/kvm/kvm-all.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
accel/kvm/kvm-all.c +10 -6
@@ -1628,9 +1628,13 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg,return kvm_update_routing_entry (s, &kroute); } -static int kvm_irqchip_assign_irqfd (KVMState *s,int fd,int rfd,int virq, +static int kvm_irqchip_assign_irqfd (KVMState *s,EventNotifier *event, +EventNotifier *resample,int virq,bool assign) { +int fd =event_notifier_get_fd (event); +int rfd = resample ?event_notifier_get_fd (resample) :-1 ; +struct kvm_irqfd irqfd = { .fd = fd, .gsi = virq,
@@ -1735,7 +1739,9 @@ int kvm_irqchip_add_hv_sint_route(KVMState *s, uint32_t vcpu, uint32_t sint)return -ENOSYS ; } -static int kvm_irqchip_assign_irqfd (KVMState *s,int fd,int virq,bool assign) +static int kvm_irqchip_assign_irqfd (KVMState *s,EventNotifier *event, +EventNotifier *resample,int virq, +bool assign) {abort (); }
@@ -1749,15 +1755,13 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg)int kvm_irqchip_add_irqfd_notifier_gsi (KVMState *s,EventNotifier *n,EventNotifier *rn,int virq) { -return kvm_irqchip_assign_irqfd (s,event_notifier_get_fd (n), - rn ?event_notifier_get_fd (rn) :-1 , virq, true); +return kvm_irqchip_assign_irqfd (s, n, rn, virq, true); }int kvm_irqchip_remove_irqfd_notifier_gsi (KVMState *s,EventNotifier *n,int virq) { -return kvm_irqchip_assign_irqfd (s,event_notifier_get_fd (n),-1 , virq, - false); +return kvm_irqchip_assign_irqfd (s, n,NULL , virq, false); } int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
Signature · 1 line
2.24.1
PATCH v2 4/5 KVM: Kick resamplefd for split kernel irqchip
In reply to Peter XuThis is majorly only for X86 because that's the only one that supports split irqchip for now.
When the irqchip is split, we face a dilemma that KVM irqfd will be enabled, however the slow irqchip is still running in the userspace. It means that the resamplefd in the kernel irqfds won't take any effect and it will miss to ack INTx interrupts on EOIs.
One example is split irqchip with VFIO INTx, which will break if we use the VFIO INTx fast path.
This patch can potentially supports the VFIO fast path again for INTx, that the IRQ delivery will still use the fast path, while we don't need to trap MMIOs in QEMU for the device to emulate the EIOs (see the callers of vfio_eoi() hook). However the EOI of the INTx will still need to be done from the userspace by caching all the resamplefds in QEMU and kick properly for IOAPIC EOI broadcast.
This is tricky because in this case the userspace ioapic irr & remote-irr will be bypassed. However such a change will greatly boost performance for assigned devices using INTx irqs (TCP_RR boosts 46% after this patch applied).
When the userspace is responsible for the resamplefd kickup, don't register it on the kvm_irqfd anymore, because on newer kernels (after commit 654f1f13ea56, 5.2+) the KVM_IRQFD will fail if with both split irqchip and resamplefd. This will make sure that the fast path will work for all supported kernels.
https://patchwork.kernel.org/patch/10738541/#22609933
--- accel/kvm/kvm-all.c | 85 +++++++++++++++++++++++++++++++++++++++++- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++++- include/sysemu/kvm.h | 7 ++++ 4 files changed, 112 insertions(+), 4 deletions(-)
accel/kvm/kvm-all.c +83 -2
@@ -159,9 +159,65 @@ static const KVMCapabilityInfo kvm_required_capabilites[] = {static NotifierList kvm_irqchip_change_notifiers =NOTIFIER_LIST_INITIALIZER (kvm_irqchip_change_notifiers); +struct KVMResampleFd { +int gsi; +EventNotifier *resample_event; +QLIST_ENTRY (KVMResampleFd) node; +}; +typedef struct KVMResampleFd KVMResampleFd ; + +/* +* Only used with split irqchip where we need to do the resample fd +* kick for the kernel from userspace. +*/ +static QLIST_HEAD(, KVMResampleFd) kvm_resample_fd_list = +QLIST_HEAD_INITIALIZER (kvm_resample_fd_list); +#define kvm_slots_lock (kml) qemu_mutex_lock(&(kml)->slots_lock)#define kvm_slots_unlock (kml) qemu_mutex_unlock(&(kml)->slots_lock) +static inline void kvm_resample_fd_remove (int gsi) +{ +KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +QLIST_REMOVE (rfd, node); +g_free (rfd); +break ; + } + } +} + +static inline void kvm_resample_fd_insert (int gsi,EventNotifier *event) +{ +KVMResampleFd *rfd =g_new0 (KVMResampleFd,1 ); + + rfd->gsi = gsi; + rfd->resample_event = event; + +QLIST_INSERT_HEAD (&kvm_resample_fd_list, rfd, node); +} + +bool kvm_resample_fd_notify (int gsi) +{ +KVMResampleFd *rfd; + +if (!kvm_irqchip_is_split ()) { +return false; + } + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +event_notifier_set (rfd->resample_event); +trace_kvm_resample_fd_notify (gsi); +return true; + } + } + +return false; +} +int kvm_get_max_memslots (void ) {KVMState *s =KVM_STATE (current_accel ());
@@ -1642,8 +1698,33 @@ static int kvm_irqchip_assign_irqfd(KVMState *s, EventNotifier *event, };if (rfd !=-1 ) { - irqfd.flags |=KVM_IRQFD_FLAG_RESAMPLE ; - irqfd.resamplefd = rfd; +assert (assign); +if (kvm_irqchip_is_split ()) { +/* +* When the slow irqchip (e.g. IOAPIC) is in the +* userspace, KVM kernel resamplefd will not work because +* the EOI of the interrupt will be delivered to userspace +* instead, so the KVM kernel resamplefd kick will be +* skipped. The userspace here mimics what the kernel +* provides with resamplefd, remember the resamplefd and +* kick it when we receive EOI of this IRQ. +* +* This is hackery because IOAPIC is mostly bypassed +* (except EOI broadcasts) when irqfd is used. However +* this can bring much performance back for split irqchip +* with INTx IRQs (for VFIO, this gives 93% perf of the +* full fast path, which is 46% perf boost comparing to +* the INTx slow path). +*/ +kvm_resample_fd_insert (virq, resample); + }else { + irqfd.flags |=KVM_IRQFD_FLAG_RESAMPLE ; + irqfd.resamplefd = rfd; + } + }else if (!assign) { +if (kvm_irqchip_is_split ()) { +kvm_resample_fd_remove (virq); + } }if (!kvm_irqfds_enabled ()) {
accel/kvm/trace-events +1 -0
@@ -16,4 +16,5 @@ kvm_set_ioeventfd_mmio(int fd, uint64_t addr, uint32_t val, bool assign, uint32_ kvm_set_ioeventfd_pio(int fd, uint16_t addr, uint32_t val, bool assign, uint32_t size, bool datamatch) "fd: %d @0x%x val=0x%x assign: %d size: %d match: %d" kvm_set_user_memory(uint32_t slot, uint32_t flags, uint64_t guest_phys_addr, uint64_t memory_size, uint64_t userspace_addr, int ret) "Slot#%d flags=0x%x gpa=0x%"PRIx64 " size=0x%"PRIx64 " ua=0x%"PRIx64 " ret=%d" kvm_clear_dirty_log(uint32_t slot, uint64_t start, uint32_t size) "slot#%"PRId32" start 0x%"PRIx64" size 0x%"PRIx32 +kvm_resample_fd_notify(int gsi) "gsi %d"
hw/intc/ioapic.c +21 -2
@@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector)for (n =0 ; n <IOAPIC_NUM_PINS ; n++) { entry = s->ioredtbl[n]; -if ((entry &IOAPIC_VECTOR_MASK ) != vector || - ((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) !=IOAPIC_TRIGGER_LEVEL ) { +if ((entry &IOAPIC_VECTOR_MASK ) != vector) { +continue ; + } + +/* +* When IOAPIC is in the userspace while APIC is still in +* the kernel (i.e., split irqchip), we have a trick to +* kick the resamplefd logic for registered irqfds from +* userspace to deactivate the IRQ. When that happens, it +* means the irq bypassed userspace IOAPIC (so the irr and +* remote-irr of the table entry should be bypassed too +* even if interrupt come), then we don't need to clear +* the remote-IRR and check irr again because they'll +* always be zeros. +*/ +if (kvm_resample_fd_notify (n)) { +continue ; + } + +if (((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) != +IOAPIC_TRIGGER_LEVEL ) {continue ; }
include/sysemu/kvm.h +7 -0
@@ -555,4 +555,11 @@ int kvm_set_one_reg(CPUState *cs, uint64_t id, void *source);int kvm_get_one_reg (CPUState *cs,uint64_t id,void *target);struct ppc_radix_page_info *kvm_get_radix_page_info (void );int kvm_get_max_memslots (void ); + +/* +* Notify resamplefd for EOI of specific interrupts. Returns true +* when one resamplefd is notified, false if no such IRQ found. +*/ +bool kvm_resample_fd_notify (int gsi); +#endif
Signature · 1 line
2.24.1
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
Hi Peter,
Quoted 29 lines · Peter Xu
This is majorly only for X86 because that's the only one that supports split irqchip for now.
When the irqchip is split, we face a dilemma that KVM irqfd will be enabled, however the slow irqchip is still running in the userspace. It means that the resamplefd in the kernel irqfds won't take any effect and it will miss to ack INTx interrupts on EOIs.
One example is split irqchip with VFIO INTx, which will break if we use the VFIO INTx fast path.
This patch can potentially supports the VFIO fast path again for INTx, that the IRQ delivery will still use the fast path, while we don't need to trap MMIOs in QEMU for the device to emulate the EIOs (see the callers of
vfio_eoi()hook). However the EOI of the INTx will still need to be done from the userspace by caching all the resamplefds in QEMU and kick properly for IOAPIC EOI broadcast.This is tricky because in this case the userspace ioapic irr & remote-irr will be bypassed. However such a change will greatly boost performance for assigned devices using INTx irqs (
TCP_RRboosts 46% after this patch applied).When the userspace is responsible for the resamplefd kickup, don't register it on the
kvm_irqfdanymore, because on newer kernels (after commit654f1f13ea56, 5.2+) theKVM_IRQFDwill fail if with both split irqchip and resamplefd. This will make sure that the fast path will work for all supported kernels.
Quoted 4 lines
https://patchwork.kernel.org/patch/10738541/#22609933
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Peter Xu <[email protected]>
Thanks
Eric
Quoted 174 lines
--- accel/kvm/kvm-all.c | 85 +++++++++++++++++++++++++++++++++++++++++- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++++- include/sysemu/kvm.h | 7 ++++ 4 files changed, 112 insertions(+), 4 deletions(-)accel/kvm/kvm-all.c +83
-2@@ -159,9 +159,65 @@ static const KVMCapabilityInfo kvm_required_capabilites[] = {static NotifierList kvm_irqchip_change_notifiers =NOTIFIER_LIST_INITIALIZER (kvm_irqchip_change_notifiers); +struct KVMResampleFd { +int gsi; +EventNotifier *resample_event; +QLIST_ENTRY (KVMResampleFd) node; +}; +typedef struct KVMResampleFd KVMResampleFd ; + +/* +* Only used with split irqchip where we need to do the resample fd +* kick for the kernel from userspace. +*/ +static QLIST_HEAD(, KVMResampleFd) kvm_resample_fd_list = +QLIST_HEAD_INITIALIZER (kvm_resample_fd_list); +#define kvm_slots_lock (kml) qemu_mutex_lock(&(kml)->slots_lock)#define kvm_slots_unlock (kml) qemu_mutex_unlock(&(kml)->slots_lock) +static inline void kvm_resample_fd_remove (int gsi) +{ +KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +QLIST_REMOVE (rfd, node); +g_free (rfd); +break ; + } + } +} + +static inline void kvm_resample_fd_insert (int gsi,EventNotifier *event) +{ +KVMResampleFd *rfd =g_new0 (KVMResampleFd,1 ); + + rfd->gsi = gsi; + rfd->resample_event = event; + +QLIST_INSERT_HEAD (&kvm_resample_fd_list, rfd, node); +} + +bool kvm_resample_fd_notify (int gsi) +{ +KVMResampleFd *rfd; + +if (!kvm_irqchip_is_split ()) { +return false; + } + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +event_notifier_set (rfd->resample_event); +trace_kvm_resample_fd_notify (gsi); +return true; + } + } + +return false; +} +int kvm_get_max_memslots (void ) {KVMState *s =KVM_STATE (current_accel ());@@ -1642,8 +1698,33 @@ static int kvm_irqchip_assign_irqfd(KVMState *s, EventNotifier *event, };if (rfd !=-1 ) { - irqfd.flags |=KVM_IRQFD_FLAG_RESAMPLE ; - irqfd.resamplefd = rfd; +assert (assign); +if (kvm_irqchip_is_split ()) { +/* +* When the slow irqchip (e.g. IOAPIC) is in the +* userspace, KVM kernel resamplefd will not work because +* the EOI of the interrupt will be delivered to userspace +* instead, so the KVM kernel resamplefd kick will be +* skipped. The userspace here mimics what the kernel +* provides with resamplefd, remember the resamplefd and +* kick it when we receive EOI of this IRQ. +* +* This is hackery because IOAPIC is mostly bypassed +* (except EOI broadcasts) when irqfd is used. However +* this can bring much performance back for split irqchip +* with INTx IRQs (for VFIO, this gives 93% perf of the +* full fast path, which is 46% perf boost comparing to +* the INTx slow path). +*/ +kvm_resample_fd_insert (virq, resample); + }else { + irqfd.flags |=KVM_IRQFD_FLAG_RESAMPLE ; + irqfd.resamplefd = rfd; + } + }else if (!assign) { +if (kvm_irqchip_is_split ()) { +kvm_resample_fd_remove (virq); + } }if (!kvm_irqfds_enabled ()) {accel/kvm/trace-events +1
-0@@ -16,4 +16,5 @@ kvm_set_ioeventfd_mmio(int fd, uint64_t addr, uint32_t val, bool assign, uint32_ kvm_set_ioeventfd_pio(int fd, uint16_t addr, uint32_t val, bool assign, uint32_t size, bool datamatch) "fd: %d @0x%x val=0x%x assign: %d size: %d match: %d" kvm_set_user_memory(uint32_t slot, uint32_t flags, uint64_t guest_phys_addr, uint64_t memory_size, uint64_t userspace_addr, int ret) "Slot#%d flags=0x%x gpa=0x%"PRIx64 " size=0x%"PRIx64 " ua=0x%"PRIx64 " ret=%d" kvm_clear_dirty_log(uint32_t slot, uint64_t start, uint32_t size) "slot#%"PRId32" start 0x%"PRIx64" size 0x%"PRIx32 +kvm_resample_fd_notify(int gsi) "gsi %d"hw/intc/ioapic.c +21
-2@@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector)for (n =0 ; n <IOAPIC_NUM_PINS ; n++) { entry = s->ioredtbl[n]; -if ((entry &IOAPIC_VECTOR_MASK ) != vector || - ((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) !=IOAPIC_TRIGGER_LEVEL ) { +if ((entry &IOAPIC_VECTOR_MASK ) != vector) { +continue ; + } + +/* +* When IOAPIC is in the userspace while APIC is still in +* the kernel (i.e., split irqchip), we have a trick to +* kick the resamplefd logic for registered irqfds from +* userspace to deactivate the IRQ. When that happens, it +* means the irq bypassed userspace IOAPIC (so the irr and +* remote-irr of the table entry should be bypassed too +* even if interrupt come), then we don't need to clear +* the remote-IRR and check irr again because they'll +* always be zeros. +*/ +if (kvm_resample_fd_notify (n)) { +continue ; + } + +if (((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) != +IOAPIC_TRIGGER_LEVEL ) {continue ; }include/sysemu/kvm.h +7
-0@@ -555,4 +555,11 @@ int kvm_set_one_reg(CPUState *cs, uint64_t id, void *source);int kvm_get_one_reg (CPUState *cs,uint64_t id,void *target);struct ppc_radix_page_info *kvm_get_radix_page_info (void );int kvm_get_max_memslots (void ); + +/* +* Notify resamplefd for EOI of specific interrupts. Returns true +* when one resamplefd is notified, false if no such IRQ found. +*/ +bool kvm_resample_fd_notify (int gsi); +#endif
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
In reply to Peter XuQuoted 85 lines · On Fri, 28 Feb 2020 11:15:02 -0500 Peter Xu <[email protected]> wrote:
This is majorly only for X86 because that's the only one that supports split irqchip for now.
When the irqchip is split, we face a dilemma that KVM irqfd will be enabled, however the slow irqchip is still running in the userspace. It means that the resamplefd in the kernel irqfds won't take any effect and it will miss to ack INTx interrupts on EOIs.
One example is split irqchip with VFIO INTx, which will break if we use the VFIO INTx fast path.
This patch can potentially supports the VFIO fast path again for INTx, that the IRQ delivery will still use the fast path, while we don't need to trap MMIOs in QEMU for the device to emulate the EIOs (see the callers of
vfio_eoi()hook). However the EOI of the INTx will still need to be done from the userspace by caching all the resamplefds in QEMU and kick properly for IOAPIC EOI broadcast.This is tricky because in this case the userspace ioapic irr & remote-irr will be bypassed. However such a change will greatly boost performance for assigned devices using INTx irqs (
TCP_RRboosts 46% after this patch applied).When the userspace is responsible for the resamplefd kickup, don't register it on the
kvm_irqfdanymore, because on newer kernels (after commit654f1f13ea56, 5.2+) theKVM_IRQFDwill fail if with both split irqchip and resamplefd. This will make sure that the fast path will work for all supported kernels.https://patchwork.kernel.org/patch/10738541/#22609933
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Peter Xu <[email protected]>--- accel/kvm/kvm-all.c | 85 +++++++++++++++++++++++++++++++++++++++++- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++++- include/sysemu/kvm.h | 7 ++++ 4 files changed, 112 insertions(+), 4 deletions(-)accel/kvm/kvm-all.c +34
-0@@ -159,9 +159,65 @@ static const KVMCapabilityInfo kvm_required_capabilites[] = {static NotifierList kvm_irqchip_change_notifiers =NOTIFIER_LIST_INITIALIZER (kvm_irqchip_change_notifiers); +struct KVMResampleFd { +int gsi; +EventNotifier *resample_event; +QLIST_ENTRY (KVMResampleFd) node; +}; +typedef struct KVMResampleFd KVMResampleFd ; + +/* +* Only used with split irqchip where we need to do the resample fd +* kick for the kernel from userspace. +*/ +static QLIST_HEAD(, KVMResampleFd) kvm_resample_fd_list = +QLIST_HEAD_INITIALIZER (kvm_resample_fd_list); +#define kvm_slots_lock (kml) qemu_mutex_lock(&(kml)->slots_lock)#define kvm_slots_unlock (kml) qemu_mutex_unlock(&(kml)->slots_lock) +static inline void kvm_resample_fd_remove (int gsi) +{ +KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +QLIST_REMOVE (rfd, node); +g_free (rfd); +break ; + } + } +} + +static inline void kvm_resample_fd_insert (int gsi,EventNotifier *event) +{ +KVMResampleFd *rfd =g_new0 (KVMResampleFd,1 ); + + rfd->gsi = gsi; + rfd->resample_event = event; +
accel/kvm/kvm-all.c static const KVMCapabilityInfo kvm_required_capabilites[] = { · line 208
Quoted 10 lines
accel/kvm/kvm-all.c +10
-0+QLIST_INSERT_HEAD (&kvm_resample_fd_list, rfd, node); +} + +bool kvm_resample_fd_notify (int gsi) +{ +KVMResampleFd *rfd; + +if (!kvm_irqchip_is_split ()) { +return false; + }
Nit, checking split irqchip here seems unnecessary. We're only adding and removing list entries based on split irqchip below, so the list would be empty anyway, unless another user comes along that might have a reason for this functionality that isn't as tied to split irqchip.
Overall the series looks like a big improvement versus falling back to our crappy generic EOI hackery with split irqchip. Thanks,
Alex
Quoted 111 lines
accel/kvm/kvm-all.c +39
-2+ + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +event_notifier_set (rfd->resample_event); +trace_kvm_resample_fd_notify (gsi); +return true; + } + } + +return false; +} +int kvm_get_max_memslots (void ) {KVMState *s =KVM_STATE (current_accel ());@@ -1642,8 +1698,33 @@ static int kvm_irqchip_assign_irqfd(KVMState *s, EventNotifier *event, };if (rfd !=-1 ) { - irqfd.flags |=KVM_IRQFD_FLAG_RESAMPLE ; - irqfd.resamplefd = rfd; +assert (assign); +if (kvm_irqchip_is_split ()) { +/* +* When the slow irqchip (e.g. IOAPIC) is in the +* userspace, KVM kernel resamplefd will not work because +* the EOI of the interrupt will be delivered to userspace +* instead, so the KVM kernel resamplefd kick will be +* skipped. The userspace here mimics what the kernel +* provides with resamplefd, remember the resamplefd and +* kick it when we receive EOI of this IRQ. +* +* This is hackery because IOAPIC is mostly bypassed +* (except EOI broadcasts) when irqfd is used. However +* this can bring much performance back for split irqchip +* with INTx IRQs (for VFIO, this gives 93% perf of the +* full fast path, which is 46% perf boost comparing to +* the INTx slow path). +*/ +kvm_resample_fd_insert (virq, resample); + }else { + irqfd.flags |=KVM_IRQFD_FLAG_RESAMPLE ; + irqfd.resamplefd = rfd; + } + }else if (!assign) { +if (kvm_irqchip_is_split ()) { +kvm_resample_fd_remove (virq); + } }if (!kvm_irqfds_enabled ()) {accel/kvm/trace-events +1
-0@@ -16,4 +16,5 @@ kvm_set_ioeventfd_mmio(int fd, uint64_t addr, uint32_t val, bool assign, uint32_ kvm_set_ioeventfd_pio(int fd, uint16_t addr, uint32_t val, bool assign, uint32_t size, bool datamatch) "fd: %d @0x%x val=0x%x assign: %d size: %d match: %d" kvm_set_user_memory(uint32_t slot, uint32_t flags, uint64_t guest_phys_addr, uint64_t memory_size, uint64_t userspace_addr, int ret) "Slot#%d flags=0x%x gpa=0x%"PRIx64 " size=0x%"PRIx64 " ua=0x%"PRIx64 " ret=%d" kvm_clear_dirty_log(uint32_t slot, uint64_t start, uint32_t size) "slot#%"PRId32" start 0x%"PRIx64" size 0x%"PRIx32 +kvm_resample_fd_notify(int gsi) "gsi %d"hw/intc/ioapic.c +21
-2@@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector)for (n =0 ; n <IOAPIC_NUM_PINS ; n++) { entry = s->ioredtbl[n]; -if ((entry &IOAPIC_VECTOR_MASK ) != vector || - ((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) !=IOAPIC_TRIGGER_LEVEL ) { +if ((entry &IOAPIC_VECTOR_MASK ) != vector) { +continue ; + } + +/* +* When IOAPIC is in the userspace while APIC is still in +* the kernel (i.e., split irqchip), we have a trick to +* kick the resamplefd logic for registered irqfds from +* userspace to deactivate the IRQ. When that happens, it +* means the irq bypassed userspace IOAPIC (so the irr and +* remote-irr of the table entry should be bypassed too +* even if interrupt come), then we don't need to clear +* the remote-IRR and check irr again because they'll +* always be zeros. +*/ +if (kvm_resample_fd_notify (n)) { +continue ; + } + +if (((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) != +IOAPIC_TRIGGER_LEVEL ) {continue ; }include/sysemu/kvm.h +7
-0@@ -555,4 +555,11 @@ int kvm_set_one_reg(CPUState *cs, uint64_t id, void *source);int kvm_get_one_reg (CPUState *cs,uint64_t id,void *target);struct ppc_radix_page_info *kvm_get_radix_page_info (void );int kvm_get_max_memslots (void ); + +/* +* Notify resamplefd for EOI of specific interrupts. Returns true +* when one resamplefd is notified, false if no such IRQ found. +*/ +bool kvm_resample_fd_notify (int gsi); +#endif
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
On Thu, Mar 05, 2020 at 04:58:57PM -0700, Alex Williamson wrote:
Hi, Alex,
[...]
Quoted 4 lines
Quoted 4 lines
accel/kvm/kvm-all.c +4
-0+bool kvm_resample_fd_notify (int gsi) +{ +KVMResampleFd *rfd; +
Quoted 8 lines
accel/kvm/kvm-all.c static const KVMCapabilityInfo kvm_required_capabilites[] = { · line 208
Quoted 3 lines
accel/kvm/kvm-all.c +3
-0+if (!kvm_irqchip_is_split ()) { +return false; + }Nit, checking split irqchip here seems unnecessary. We're only adding and removing list entries based on split irqchip below, so the list would be empty anyway, unless another user comes along that might have a reason for this functionality that isn't as tied to split irqchip.
Right, now it's more or less a hint to readers, and we can remove it. I'll see whether I'll repost a new version, and I'll drop it if so.
Quoted 3 lines
Overall the series looks like a big improvement versus falling back to our crappy generic EOI hackery with split irqchip. Thanks,
Yes I was pretty happy to see the numbers too when I first tested the series, after all I was still uncertain about how much overhead the userspace EOI would take on the irq return path. It turns out that the injection seems to be more important.
In all cases, major credits go to Paolo for the idea. :)
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
Quoted 26 lines · On Thu, 5 Mar 2020 19:43:24 -0500 Peter Xu <[email protected]> wrote:
On Thu, Mar 05, 2020 at 04:58:57PM -0700, Alex Williamson wrote:
Hi, Alex,
[...]
Quoted 4 lines
Quoted 4 lines
accel/kvm/kvm-all.c +4
-0+bool kvm_resample_fd_notify (int gsi) +{ +KVMResampleFd *rfd; +Quoted 8 lines
accel/kvm/kvm-all.c static const KVMCapabilityInfo kvm_required_capabilites[] = { · line 208
Quoted 3 lines
accel/kvm/kvm-all.c +3
-0+if (!kvm_irqchip_is_split ()) { +return false; + }Nit, checking split irqchip here seems unnecessary. We're only adding and removing list entries based on split irqchip below, so the list would be empty anyway, unless another user comes along that might have a reason for this functionality that isn't as tied to split irqchip.
Right, now it's more or less a hint to readers, and we can remove it. I'll see whether I'll repost a new version, and I'll drop it if so.
Quoted 3 lines
Overall the series looks like a big improvement versus falling back to our crappy generic EOI hackery with split irqchip. Thanks,
Quoted 6 lines
Yes I was pretty happy to see the numbers too when I first tested the series, after all I was still uncertain about how much overhead the userspace EOI would take on the irq return path. It turns out that the injection seems to be more important.
In all cases, major credits go to Paolo for the idea. :)
Hey Peter, I'm trying to test this myself and my VM just hangs as soon as I enable split irqchip. It boots up to discovering the virtio disks, then nothing more. My host kernel is 5.3.7-301.fc31.x86_64, QEMU is 373c7068dd61 + this patch series. VM script is:
/usr/local/bin/qemu-system-x86_64 \
-S \
-machine pc-q35-5.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel-irqchip=split \
-cpu host \
-m 2048 \
-smp 2,sockets=2,cores=1,threads=1 \
-no-user-config \
-nodefaults \
-monitor stdio \
-serial none \
-parallel none \
-no-hpet \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-drive file=/var/lib/libvirt/images/fedora31-1.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
-device virtio-blk-pci,scsi=off,bus=pci.3,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-vnc :0 \
-device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
-device vfio-pci,host=02:00.0,id=e1000e,bus=pci.2,addr=0x0
Guest has pci=nomsi on the kernel command line. It boots with irqchip=on, also boots with x-no-kvm-intx=on as an arg to the vfio-pci device. I'm afraid there's a regression here unless I'm failing to add something necessary for split irqchip. Thanks,
Alex
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
In reply to Peter XuQuoted 110 lines · On Fri, 28 Feb 2020 11:15:02 -0500 Peter Xu <[email protected]> wrote:
This is majorly only for X86 because that's the only one that supports split irqchip for now.
When the irqchip is split, we face a dilemma that KVM irqfd will be enabled, however the slow irqchip is still running in the userspace. It means that the resamplefd in the kernel irqfds won't take any effect and it will miss to ack INTx interrupts on EOIs.
One example is split irqchip with VFIO INTx, which will break if we use the VFIO INTx fast path.
This patch can potentially supports the VFIO fast path again for INTx, that the IRQ delivery will still use the fast path, while we don't need to trap MMIOs in QEMU for the device to emulate the EIOs (see the callers of
vfio_eoi()hook). However the EOI of the INTx will still need to be done from the userspace by caching all the resamplefds in QEMU and kick properly for IOAPIC EOI broadcast.This is tricky because in this case the userspace ioapic irr & remote-irr will be bypassed. However such a change will greatly boost performance for assigned devices using INTx irqs (
TCP_RRboosts 46% after this patch applied).When the userspace is responsible for the resamplefd kickup, don't register it on the
kvm_irqfdanymore, because on newer kernels (after commit654f1f13ea56, 5.2+) theKVM_IRQFDwill fail if with both split irqchip and resamplefd. This will make sure that the fast path will work for all supported kernels.https://patchwork.kernel.org/patch/10738541/#22609933
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Peter Xu <[email protected]>--- accel/kvm/kvm-all.c | 85 +++++++++++++++++++++++++++++++++++++++++- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++++- include/sysemu/kvm.h | 7 ++++ 4 files changed, 112 insertions(+), 4 deletions(-)accel/kvm/kvm-all.c +56
-0@@ -159,9 +159,65 @@ static const KVMCapabilityInfo kvm_required_capabilites[] = {static NotifierList kvm_irqchip_change_notifiers =NOTIFIER_LIST_INITIALIZER (kvm_irqchip_change_notifiers); +struct KVMResampleFd { +int gsi; +EventNotifier *resample_event; +QLIST_ENTRY (KVMResampleFd) node; +}; +typedef struct KVMResampleFd KVMResampleFd ; + +/* +* Only used with split irqchip where we need to do the resample fd +* kick for the kernel from userspace. +*/ +static QLIST_HEAD(, KVMResampleFd) kvm_resample_fd_list = +QLIST_HEAD_INITIALIZER (kvm_resample_fd_list); +#define kvm_slots_lock (kml) qemu_mutex_lock(&(kml)->slots_lock)#define kvm_slots_unlock (kml) qemu_mutex_unlock(&(kml)->slots_lock) +static inline void kvm_resample_fd_remove (int gsi) +{ +KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +QLIST_REMOVE (rfd, node); +g_free (rfd); +break ; + } + } +} + +static inline void kvm_resample_fd_insert (int gsi,EventNotifier *event) +{ +KVMResampleFd *rfd =g_new0 (KVMResampleFd,1 ); + + rfd->gsi = gsi; + rfd->resample_event = event; + +QLIST_INSERT_HEAD (&kvm_resample_fd_list, rfd, node); +} + +bool kvm_resample_fd_notify (int gsi) +{ +KVMResampleFd *rfd; + +if (!kvm_irqchip_is_split ()) { +return false; + } + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +event_notifier_set (rfd->resample_event); +trace_kvm_resample_fd_notify (gsi); +return true; + } + } + +return false; +} +int kvm_get_max_memslots (void ) {KVMState *s =KVM_STATE (current_accel ());
[snip]
Quoted 18 lines
hw/intc/ioapic.c +8
-2@@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector)for (n =0 ; n <IOAPIC_NUM_PINS ; n++) { entry = s->ioredtbl[n]; -if ((entry &IOAPIC_VECTOR_MASK ) != vector || - ((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) !=IOAPIC_TRIGGER_LEVEL ) { +if ((entry &IOAPIC_VECTOR_MASK ) != vector) { +continue ; + } + +/* +* When IOAPIC is in the userspace while APIC is still in +* the kernel (i.e., split irqchip), we have a trick to +* kick the resamplefd logic for registered irqfds from
hw/intc/ioapic.c void ioapic_eoi_broadcast(int vector) · line 256
Quoted 10 lines
hw/intc/ioapic.c +10
-0+* userspace to deactivate the IRQ. When that happens, it +* means the irq bypassed userspace IOAPIC (so the irr and +* remote-irr of the table entry should be bypassed too +* even if interrupt come), then we don't need to clear +* the remote-IRR and check irr again because they'll +* always be zeros. +*/ +if (kvm_resample_fd_notify (n)) { +continue ; + }
It seems the problem I reported is here. In my configuration virtio-blk and an assigned e1000e share an interrupt. virtio-blk is initializing and apparently triggers an interrupt. The vfio-pci device is configured for INTx though not active yet, but kvm_resample_fd_notify() kicks the fd here, so we continue. If I remove the continue here both devices seem to work, but I don't claim to understand the condition we're trying to continue for here yet. This series needs more testing with shared interrupts. Thanks,
Alex
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
Quoted 143 lines · On Mon, 9 Mar 2020 16:10:59 -0600 Alex Williamson <[email protected]>…
Quoted 110 lines · On Fri, 28 Feb 2020 11:15:02 -0500 Peter Xu <[email protected]> wrote:
This is majorly only for X86 because that's the only one that supports split irqchip for now.
When the irqchip is split, we face a dilemma that KVM irqfd will be enabled, however the slow irqchip is still running in the userspace. It means that the resamplefd in the kernel irqfds won't take any effect and it will miss to ack INTx interrupts on EOIs.
One example is split irqchip with VFIO INTx, which will break if we use the VFIO INTx fast path.
This patch can potentially supports the VFIO fast path again for INTx, that the IRQ delivery will still use the fast path, while we don't need to trap MMIOs in QEMU for the device to emulate the EIOs (see the callers of
vfio_eoi()hook). However the EOI of the INTx will still need to be done from the userspace by caching all the resamplefds in QEMU and kick properly for IOAPIC EOI broadcast.This is tricky because in this case the userspace ioapic irr & remote-irr will be bypassed. However such a change will greatly boost performance for assigned devices using INTx irqs (
TCP_RRboosts 46% after this patch applied).When the userspace is responsible for the resamplefd kickup, don't register it on the
kvm_irqfdanymore, because on newer kernels (after commit654f1f13ea56, 5.2+) theKVM_IRQFDwill fail if with both split irqchip and resamplefd. This will make sure that the fast path will work for all supported kernels.https://patchwork.kernel.org/patch/10738541/#22609933
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Peter Xu <[email protected]>--- accel/kvm/kvm-all.c | 85 +++++++++++++++++++++++++++++++++++++++++- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++++- include/sysemu/kvm.h | 7 ++++ 4 files changed, 112 insertions(+), 4 deletions(-)accel/kvm/kvm-all.c +56
-0@@ -159,9 +159,65 @@ static const KVMCapabilityInfo kvm_required_capabilites[] = {static NotifierList kvm_irqchip_change_notifiers =NOTIFIER_LIST_INITIALIZER (kvm_irqchip_change_notifiers); +struct KVMResampleFd { +int gsi; +EventNotifier *resample_event; +QLIST_ENTRY (KVMResampleFd) node; +}; +typedef struct KVMResampleFd KVMResampleFd ; + +/* +* Only used with split irqchip where we need to do the resample fd +* kick for the kernel from userspace. +*/ +static QLIST_HEAD(, KVMResampleFd) kvm_resample_fd_list = +QLIST_HEAD_INITIALIZER (kvm_resample_fd_list); +#define kvm_slots_lock (kml) qemu_mutex_lock(&(kml)->slots_lock)#define kvm_slots_unlock (kml) qemu_mutex_unlock(&(kml)->slots_lock) +static inline void kvm_resample_fd_remove (int gsi) +{ +KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +QLIST_REMOVE (rfd, node); +g_free (rfd); +break ; + } + } +} + +static inline void kvm_resample_fd_insert (int gsi,EventNotifier *event) +{ +KVMResampleFd *rfd =g_new0 (KVMResampleFd,1 ); + + rfd->gsi = gsi; + rfd->resample_event = event; + +QLIST_INSERT_HEAD (&kvm_resample_fd_list, rfd, node); +} + +bool kvm_resample_fd_notify (int gsi) +{ +KVMResampleFd *rfd; + +if (!kvm_irqchip_is_split ()) { +return false; + } + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +event_notifier_set (rfd->resample_event); +trace_kvm_resample_fd_notify (gsi); +return true; + } + } + +return false; +} +int kvm_get_max_memslots (void ) {KVMState *s =KVM_STATE (current_accel ());[snip]
Quoted 18 lines
hw/intc/ioapic.c +8
-2@@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector)for (n =0 ; n <IOAPIC_NUM_PINS ; n++) { entry = s->ioredtbl[n]; -if ((entry &IOAPIC_VECTOR_MASK ) != vector || - ((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) !=IOAPIC_TRIGGER_LEVEL ) { +if ((entry &IOAPIC_VECTOR_MASK ) != vector) { +continue ; + } + +/* +* When IOAPIC is in the userspace while APIC is still in +* the kernel (i.e., split irqchip), we have a trick to +* kick the resamplefd logic for registered irqfds from hw/intc/ioapic.c void ioapic_eoi_broadcast(int vector) · line 256
Quoted 10 lines
hw/intc/ioapic.c +10
-0+* userspace to deactivate the IRQ. When that happens, it +* means the irq bypassed userspace IOAPIC (so the irr and +* remote-irr of the table entry should be bypassed too +* even if interrupt come), then we don't need to clear +* the remote-IRR and check irr again because they'll +* always be zeros. +*/ +if (kvm_resample_fd_notify (n)) { +continue ; + }
Quoted 8 lines
It seems the problem I reported is here. In my configuration virtio-blk and an assigned e1000e share an interrupt. virtio-blk is initializing and apparently triggers an interrupt. The vfio-pci device is configured for INTx though not active yet, but
kvm_resample_fd_notify()kicks the fd here, so we continue. If I remove the continue here both devices seem to work, but I don't claim to understand the condition we're trying to continue for here yet. This series needs more testing with shared interrupts. Thanks,
I'm also curious how this ended up between testing whether the vector is masked and testing that it's level triggered. We shouldn't have any edge triggered resamplers. I find however that if I move the resampler notify to after the remote IRR test, my NIC gets starved of interrupts. So empirically, it seems kvm_resample_fd_notify() should be a void function called unconditionally between the original mask+level check removed above and the IRR check below. Thanks,
Alex
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
On Mon, Mar 09, 2020 at 04:33:59PM -0600, Alex Williamson wrote:
[...]
Quoted 38 lines
Quoted 29 lines
Quoted 18 lines
hw/intc/ioapic.c +8
-2@@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector)for (n =0 ; n <IOAPIC_NUM_PINS ; n++) { entry = s->ioredtbl[n]; -if ((entry &IOAPIC_VECTOR_MASK ) != vector || - ((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) !=IOAPIC_TRIGGER_LEVEL ) { +if ((entry &IOAPIC_VECTOR_MASK ) != vector) { +continue ; + } + +/* +* When IOAPIC is in the userspace while APIC is still in +* the kernel (i.e., split irqchip), we have a trick to +* kick the resamplefd logic for registered irqfds from hw/intc/ioapic.c void ioapic_eoi_broadcast(int vector) · line 256
Quoted 10 lines
hw/intc/ioapic.c +10
-0+* userspace to deactivate the IRQ. When that happens, it +* means the irq bypassed userspace IOAPIC (so the irr and +* remote-irr of the table entry should be bypassed too +* even if interrupt come), then we don't need to clear +* the remote-IRR and check irr again because they'll +* always be zeros. +*/ +if (kvm_resample_fd_notify (n)) { +continue ; + }Quoted 8 lines
It seems the problem I reported is here. In my configuration virtio-blk and an assigned e1000e share an interrupt. virtio-blk is initializing and apparently triggers an interrupt. The vfio-pci device is configured for INTx though not active yet, but
kvm_resample_fd_notify()kicks the fd here, so we continue. If I remove the continue here both devices seem to work, but I don't claim to understand the condition we're trying to continue for here yet. This series needs more testing with shared interrupts. Thanks,
Quoted 3 lines
I'm also curious how this ended up between testing whether the vector is masked and testing that it's level triggered. We shouldn't have any edge triggered resamplers.
We had a similar discussion in V1 (with Paolo):
https://patchwork.kernel.org/patch/11407441/#23190891
So my understanding is that VFIO will unmask the intx IRQ when it comes, and register the resamplefd too, no matter whether it's level triggered (at least from what the code does). Am I right?
Quoted 5 lines
I find however that if I move the resampler
notify to after the remote IRR test, my NIC gets starved of interrupts. So empirically, it seemskvm_resample_fd_notify()should be a void function called unconditionally between the original mask+level check removed above and the IRR check below. Thanks,
Yes IMHO we can't move that notify() to be after the remote IRR check because the IRR and remote IRR will be completely bypassed for the assigned device. In other words, even if the interrupt has arrived for the assigned device, both IRR and remote IRR should always be zero (assuming the virtio-blk device doesn't generate any IRQ). So we probably still need to do the notify even if remote-irr is not set.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
Quoted 46 lines · On Mon, 9 Mar 2020 20:38:08 -0400 Peter Xu <[email protected]> wrote:
On Mon, Mar 09, 2020 at 04:33:59PM -0600, Alex Williamson wrote:
[...]
Quoted 38 lines
Quoted 29 lines
Quoted 18 lines
hw/intc/ioapic.c +8
-2@@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector)for (n =0 ; n <IOAPIC_NUM_PINS ; n++) { entry = s->ioredtbl[n]; -if ((entry &IOAPIC_VECTOR_MASK ) != vector || - ((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) !=IOAPIC_TRIGGER_LEVEL ) { +if ((entry &IOAPIC_VECTOR_MASK ) != vector) { +continue ; + } + +/* +* When IOAPIC is in the userspace while APIC is still in +* the kernel (i.e., split irqchip), we have a trick to +* kick the resamplefd logic for registered irqfds from hw/intc/ioapic.c void ioapic_eoi_broadcast(int vector) · line 256
Quoted 10 lines
hw/intc/ioapic.c +10
-0+* userspace to deactivate the IRQ. When that happens, it +* means the irq bypassed userspace IOAPIC (so the irr and +* remote-irr of the table entry should be bypassed too +* even if interrupt come), then we don't need to clear +* the remote-IRR and check irr again because they'll +* always be zeros. +*/ +if (kvm_resample_fd_notify (n)) { +continue ; + }Quoted 8 lines
It seems the problem I reported is here. In my configuration virtio-blk and an assigned e1000e share an interrupt. virtio-blk is initializing and apparently triggers an interrupt. The vfio-pci device is configured for INTx though not active yet, but
kvm_resample_fd_notify()kicks the fd here, so we continue. If I remove the continue here both devices seem to work, but I don't claim to understand the condition we're trying to continue for here yet. This series needs more testing with shared interrupts. Thanks,Quoted 3 lines
I'm also curious how this ended up between testing whether the vector is masked and testing that it's level triggered. We shouldn't have any edge triggered resamplers.
Quoted 7 lines
We had a similar discussion in V1 (with Paolo):
https://patchwork.kernel.org/patch/11407441/#23190891
So my understanding is that VFIO will unmask the intx IRQ when it comes, and register the resamplefd too, no matter whether it's level triggered (at least from what the code does). Am I right?
As Paolo replied in your previous discussion, INTx is always level triggered.
Quoted 6 lines
Quoted 5 lines
I find however that if I move the resampler
notify to after the remote IRR test, my NIC gets starved of interrupts. So empirically, it seemskvm_resample_fd_notify()should be a void function called unconditionally between the original mask+level check removed above and the IRR check below. Thanks,
Quoted 6 lines
Yes IMHO we can't move that
notify()to be after the remote IRR check because the IRR and remote IRR will be completely bypassed for the assigned device. In other words, even if the interrupt has arrived for the assigned device, both IRR and remote IRR should always be zero (assuming the virtio-blk device doesn't generate any IRQ). So we probably still need to do the notify even if remote-irr is not set.
Yep. Thanks,
Alex
Re: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip
In reply to Alex WilliamsonQuoted 143 lines · Alex Williamson
Quoted 110 lines · On Fri, 28 Feb 2020 11:15:02 -0500 Peter Xu <[email protected]> wrote:
This is majorly only for X86 because that's the only one that supports split irqchip for now.
When the irqchip is split, we face a dilemma that KVM irqfd will be enabled, however the slow irqchip is still running in the userspace. It means that the resamplefd in the kernel irqfds won't take any effect and it will miss to ack INTx interrupts on EOIs.
One example is split irqchip with VFIO INTx, which will break if we use the VFIO INTx fast path.
This patch can potentially supports the VFIO fast path again for INTx, that the IRQ delivery will still use the fast path, while we don't need to trap MMIOs in QEMU for the device to emulate the EIOs (see the callers of
vfio_eoi()hook). However the EOI of the INTx will still need to be done from the userspace by caching all the resamplefds in QEMU and kick properly for IOAPIC EOI broadcast.This is tricky because in this case the userspace ioapic irr & remote-irr will be bypassed. However such a change will greatly boost performance for assigned devices using INTx irqs (
TCP_RRboosts 46% after this patch applied).When the userspace is responsible for the resamplefd kickup, don't register it on the
kvm_irqfdanymore, because on newer kernels (after commit654f1f13ea56, 5.2+) theKVM_IRQFDwill fail if with both split irqchip and resamplefd. This will make sure that the fast path will work for all supported kernels.https://patchwork.kernel.org/patch/10738541/#22609933
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Peter Xu <[email protected]>--- accel/kvm/kvm-all.c | 85 +++++++++++++++++++++++++++++++++++++++++- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++++- include/sysemu/kvm.h | 7 ++++ 4 files changed, 112 insertions(+), 4 deletions(-)accel/kvm/kvm-all.c +56
-0@@ -159,9 +159,65 @@ static const KVMCapabilityInfo kvm_required_capabilites[] = {static NotifierList kvm_irqchip_change_notifiers =NOTIFIER_LIST_INITIALIZER (kvm_irqchip_change_notifiers); +struct KVMResampleFd { +int gsi; +EventNotifier *resample_event; +QLIST_ENTRY (KVMResampleFd) node; +}; +typedef struct KVMResampleFd KVMResampleFd ; + +/* +* Only used with split irqchip where we need to do the resample fd +* kick for the kernel from userspace. +*/ +static QLIST_HEAD(, KVMResampleFd) kvm_resample_fd_list = +QLIST_HEAD_INITIALIZER (kvm_resample_fd_list); +#define kvm_slots_lock (kml) qemu_mutex_lock(&(kml)->slots_lock)#define kvm_slots_unlock (kml) qemu_mutex_unlock(&(kml)->slots_lock) +static inline void kvm_resample_fd_remove (int gsi) +{ +KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +QLIST_REMOVE (rfd, node); +g_free (rfd); +break ; + } + } +} + +static inline void kvm_resample_fd_insert (int gsi,EventNotifier *event) +{ +KVMResampleFd *rfd =g_new0 (KVMResampleFd,1 ); + + rfd->gsi = gsi; + rfd->resample_event = event; + +QLIST_INSERT_HEAD (&kvm_resample_fd_list, rfd, node); +} + +bool kvm_resample_fd_notify (int gsi) +{ +KVMResampleFd *rfd; + +if (!kvm_irqchip_is_split ()) { +return false; + } + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { +if (rfd->gsi == gsi) { +event_notifier_set (rfd->resample_event); +trace_kvm_resample_fd_notify (gsi); +return true; + } + } + +return false; +} +int kvm_get_max_memslots (void ) {KVMState *s =KVM_STATE (current_accel ());[snip]
Quoted 18 lines
hw/intc/ioapic.c +8
-2@@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector)for (n =0 ; n <IOAPIC_NUM_PINS ; n++) { entry = s->ioredtbl[n]; -if ((entry &IOAPIC_VECTOR_MASK ) != vector || - ((entry >>IOAPIC_LVT_TRIGGER_MODE_SHIFT ) &1 ) !=IOAPIC_TRIGGER_LEVEL ) { +if ((entry &IOAPIC_VECTOR_MASK ) != vector) { +continue ; + } + +/* +* When IOAPIC is in the userspace while APIC is still in +* the kernel (i.e., split irqchip), we have a trick to +* kick the resamplefd logic for registered irqfds from hw/intc/ioapic.c void ioapic_eoi_broadcast(int vector) · line 256
Quoted 10 lines
hw/intc/ioapic.c +10
-0+* userspace to deactivate the IRQ. When that happens, it +* means the irq bypassed userspace IOAPIC (so the irr and +* remote-irr of the table entry should be bypassed too +* even if interrupt come), then we don't need to clear +* the remote-IRR and check irr again because they'll +* always be zeros. +*/ +if (kvm_resample_fd_notify (n)) { +continue ; + }
Quoted 8 lines
It seems the problem I reported is here. In my configuration virtio-blk and an assigned e1000e share an interrupt. virtio-blk is initializing and apparently triggers an interrupt. The vfio-pci device is configured for INTx though not active yet, but
kvm_resample_fd_notify()kicks the fd here, so we continue. If I remove the continue here both devices seem to work, but I don't claim to understand the condition we're trying to continue for here yet. This series needs more testing with shared interrupts. Thanks,
Hi, Alex,
The "continue" was there only because I wanted to skip updating remote IRR and so on, considering that it won't be useful after all if the userspace irqchip is bypassed here. However I totally overlooked the fact that this exact IRQ can be shared by another device that is using the same IRQ while it was not bypassed by the kernel instead.
Here's my guess on what should have happened...
- The first IRQ of the virtio-blk device will work, which will set IRR bit, and at last it'll also set remote-irr of the IRQ, showing that it's in service
- When the virtio-blk wants to EOI the irq, it noticed that it's registered with resamplefd (which is actually the nic's rather than the virtio-blk device), then it ignored the update to remote-irr assuming that it'll always be zero (but actually it's just set).
- When the virtio-blk wants to send further IRQs (starting from the 2nd one), it will try to set irr again, but this time since the remote-irr is still set, it'll ignore the interrupt because it thought the IRQ is still in service (while it's not). This corresponds to where "coalesce==true" in ioapic_service():
if (coalesce) {
/* We are level triggered interrupts, and the
* guest should be still working on previous one,
* so skip it. */
continue;
}
With that, I think your proposed solution should be the right fix, that we should still keep the whole IOAPIC EOI path even if the IRQ is registered with resamplefd, because that can be used by the other device.
I'll also test this senario tomorrow.
Thanks!
Signature · 1 line
Peter Xu
PATCH v2 5/5 Revert "vfio/pci: Disable INTx fast path if using split irqchip"
In reply to Peter XuWith the resamplefd list introduced, we can savely enable VFIO INTx fast path again with split irqchip so it can still be faster than the complete slow path.
--- hw/vfio/pci.c | 12 ------------ 1 file changed, 12 deletions(-)
hw/vfio/pci.c +0 -12
@@ -124,18 +124,6 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp)return ; } -if (kvm_irqchip_is_split ()) { -/* -* VFIO INTx is currently not working with split kernel -* irqchip for level triggered interrupts. Go the slow path -* as long as split is enabled so we can be at least -* functional (even with poor performance). -* -* TODO: Remove this after all things fixed up. -*/ -return ; - } -/* Get to a known interrupt state */ qemu_set_fd_handler (irq_fd,NULL ,NULL , vdev);vfio_mask_single_irqindex (&vdev->vbasedev,VFIO_PCI_INTX_IRQ_INDEX );
Signature · 1 line
2.24.1
Re: [PATCH v2 5/5] Revert "vfio/pci: Disable INTx fast path if using split irqchip"
Hi Peter,
Quoted 5 lines · Peter Xu
With the resamplefd list introduced, we can savely enable VFIO INTx fast path again with split irqchip so it can still be faster than the complete slow path.
Signed-off-by Peter Xu <[email protected]>
Thanks
Eric
Quoted 28 lines
--- hw/vfio/pci.c | 12 ------------ 1 file changed, 12 deletions(-)hw/vfio/pci.c +0
-12@@ -124,18 +124,6 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp)return ; } -if (kvm_irqchip_is_split ()) { -/* -* VFIO INTx is currently not working with split kernel -* irqchip for level triggered interrupts. Go the slow path -* as long as split is enabled so we can be at least -* functional (even with poor performance). -* -* TODO: Remove this after all things fixed up. -*/ -return ; - } -/* Get to a known interrupt state */ qemu_set_fd_handler (irq_fd,NULL ,NULL , vdev);vfio_mask_single_irqindex (&vdev->vbasedev,VFIO_PCI_INTX_IRQ_INDEX );
Hi Peter,
Quoted 52 lines · Peter Xu
v2:
- pick tags
- don't register resamplefd with KVM kernel when the userspace resamplefd path is enabled (should enable fast path on new kernels)
- fix resamplefd mem leak
- fix commit message of patch 4 [Eric]
- letkvm_resample_fd_notify()return a boolean, skip ioapic check if returned true
- more comments here and there in the code to state the fact that userspace ioapic irr & remote-irr are bypassed [Paolo]VFIO INTx is not working with split irqchip. On new kernels
KVM_IRQFDwill directly fail with resamplefd attached so QEMU will automatically fallback to the INTx slow path. However on old kernels it's still broken.Only until recently I noticed that this could also break PXE boot for assigned NICs [1]. My wild guess is that the PXE ROM will be mostly using INTx as well, which means we can't bypass that even if we enables MSI for the guest kernel.
This series tries to first fix this issue function-wise, then speed up for the INTx again with resamplefd (mostly following the ideas proposed by Paolo one year ago [2]). My
TCP_RRtest shows that:- Before this series: this is broken, no number to show
- After patch 1 (enable slow path): get 63% perf comparing to full kernel irqchip
- After whole series (enable fast path partly, irq injection will be the same as fast path, however userspace needs to intercept for EOI broadcast to resamplefd, though should still be faster than the MMIO trick for intx eoi): get 93% perf comparing to full kernel irqchip, which is a 46% performance boost
I think we can consider to apply patch 1 even sooner than the rest of the series to unbreak intx+split first.
The whole test matrix for reference:
|----------+---------+-------------------+--------------+--------------------| | IRQ type | irqchip | TCP_STREAM (Gbps) | TCP_RR (pps) | note | |----------+---------+-------------------+--------------+--------------------| | msi | on | 9.39 | 17567 | | | nomsi | on | 9.29 | 14056 | | | msi | split | 9.36 | 17330 | | | nomsi | split | / | / | currently broken | | nomsi | split | 8.98 | 8977 | after patch 1 | | nomsi | split | 9.21 | 13142 | after whole series | |----------+---------+-------------------+--------------+--------------------|
Quoted 4 lines
Any review comment is welcomed. Thanks,
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1786404
[2] https://patchwork.kernel.org/patch/10738541/#22609933
For the whole series:
using a 5.2-rc1 kernel with reverted "654f1f13ea56 kvm: Check irqchip mode before assign irqfd" and guest booting with nomsi.
Thanks
Eric
Quoted 15 lines
Peter Xu (5):
vfio/pci: Disable INTx fast path if using split irqchip vfio/pci: Usekvm_irqchip_add_irqfd_notifier_gsi()for irqfds
KVM: Pass EventNotifier intokvm_irqchip_assign_irqfd
KVM: Kick resamplefd for split kernel irqchip
Revert "vfio/pci: Disable INTx fast path if using split irqchip"accel/kvm/kvm-all.c | 101 +++++++++++++++++++++++++++++++++++++---- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++- hw/vfio/pci.c | 37 ++++++--------- include/sysemu/kvm.h | 7 +++ 5 files changed, 137 insertions(+), 32 deletions(-)