KVM platform device passthrough
v18 cover KVM platform device passthrough
Those remaining patches aim at enabling irqfd usage in the VFIO platform device
They can be found at :
http://git.linaro.org/people/eric.auger/qemu.git
(branch vfio_integ_v18)
Tested by Vikram Sethi on a device featuring 2 MMIO regions. Code also exercised successfully by VOSYS for AMBA derivative series.
Thanks
Eric
History:
v17 -> v18:
- remove found in vfio_start_irqfd_injection
- unconditionnally set sbc->connect_irq_notifier in the class_init function and do the proper checks in vfio_start_irqfd_injection
v16 -> v17:
- add Paolo's A-b on [1-3]
- Take into account Peter's comments
- add [kvm: some fixes to kvm_resamplefds_allowed] to bring non-KVM version of kvm_resamplefds_allowed
- remove #ifdef CONFIG_KVM in last patch
v15 -> v16:
- no code change
- add Vikram's T-b
- rebase on 0915aed
- remove linux header update patch file
v14 -> v15:
- add Peter R-b on sysbus: add irq_routing_notifier
- correct g_malloc0_n usage in skeleton
- correct return values of reset related functions
- include Cornelia's patch for header update
v13 -> v14:
- remove v13 9, 10, 11 patch files and replace them by a single patch file "sysbus: add irq_routing_notifier".
- in skeleton, fix ENAMETOOLONG sign
- remove VFIOINTp virtualID in "add irq assignment" patch file
- removed trace_vfio_platform_start_eventfd
v12 -> v13:
- header update but same update was already sent by Cornelia
- Rework VFIO signaling & irqfd setup: restored 2-step setup featuring eventfd setup on realize and then irqfd setup on irq binding.
- irqfd setup now uses kvm_irqchip_add_irqfd_notifier and sysbus irq_set_hook override. This leads to the introduction of 6 patch files enabling those 2 features. Paolo advised to introduce kvm_irqchip_add_irqfd_notifier series in the VFIO one. I did the same for irq_set_hook series but if it is better I can submit it aside.
- above changes made possible to remove
x hw/vfio/platform: add capability to start IRQ propagation" x hw/arm/virt: start VFIO IRQ propagation
- in sysbus-fdt.c, use platform_bus_get_mmio_addr instead of deprecated mmio[0] property. Thanks to Bharat who pointed this issue out. also cpu_to_be32 was used for size and base (Vikram input) .
- in skeleton misc corrections following Alex review.
v11->v12:
- add x-mmap property definition, without which the default value of vbasedev.allow_mmap is false, hence preventing the reg space from being mmapped.
v10->v11:
- rebase onto v2.3.0-rc0 (mainly related to PCIe support in virt)
- add dma-coherent property for calxeda midway (fix revealed by removal of kernel-side "vfio: type1: support for ARM SMMUS with VFIO_IOMMU_TYPE1")
- virt modifications to start VFIO IRQ forwarding are now in a separate patch
- rearrange linux header exports (those are still partial exports waiting for definitive 4.1-rc0)
- take into account Alex Bennée comments:
- use g_malloc0_n instead of g_malloc0
- use block declarations when possible
- rework readlink returned value treatment
- use g_strlcat in place strncat
- re-arrange mutex locking for multiple IRQ support (user-side handled eventfds)
- use g_snprintf instead of snprintf
- change the order of functions to avoid pre-declaration in platform.c
- add flags in VFIOINTp struct to detect whether the IRQ is automasked
- some comment rewriting
v9->v10:
- rebase on "vfio: cleanup vfio_get_device error path, remove vfio_populate_device": vfio_populate_device no more called in vfio_get_device but in vfio_base_device_init
- update VFIO header according to vfio platform driver v13 (no AMBA)
v8->v9:
- rebase on 2.2.0 and machvirt dynamic sysbus instantiation v10
- v8 1-11 were pulled
- patch files related to forwarding are moved in a seperate series since it depends on kernel series still in RFC.
- introduction of basic VFIO platform device split into 3 patch files to ease the review (hope it will help).
- add an author in platform.c
- add deallocation in vfio_populate_device error case
- add patch file doing the VFIO header sync
- use VFIO_DEVICE_FLAGS_PLATFORM in vfio_populate_device
- rename calxeda_xgmac.c into calxeda-xgmac.c
- sysbus-fdt: add_calxeda_midway_xgmac_fdt_node g_free in case of errors
- reword of linux-headers patch files
v7->v8:
- rebase on v2.2.0-rc3 and integrate
"Add skip_dump flag to ignore memory region during dump"
- KVM header evolution with subindex addition in kvm_arch_forwarded_irq
- split [PATCH v7 03/16] hw/vfio/pci: introduce VFIODevice into 4 patches
- vfio_compute_needs_reset does not return bool anymore
- add some comments about exposed MMIO region and IRQ in calxeda xgmac device
- vfio_[un]mask_irqindex renamed into vfio_[un]mask_single_irqindex
- rework IRQ startup: former machine init done notifier is replaced by a reset notifier. machine file passes the interrupt controller DeviceState handle (not the platform bus first irq parameter).
- sysbus-fdt:
- move the add_fdt_node_functions array declaration between the device specific code and the generic code to avoid forward declarations of decice specific functions
- rename add_basic_vfio_fdt_node into add_calxeda_midway_xgmac_fdt_node emphasizing the fact it is xgmac specific
v6->v7:
- fake injection test modality removed
- VFIO_DEVICE_TYPE_PLATFORM only introduced with VFIO platform
- new helper functions to start VFIO IRQ on machine init done notifier (introduced in hw/vfio/platform: add vfio-platform support and notifier registration invoked in hw/arm/virt: add support for VFIO devices). vfio_start_irq_injection is replaced by vfio_register_irq_starter.
v5->v6:
- rebase on 2.1rc5 PCI code
- forwarded IRQ first integraton
- vfio_device property renamed into host property
- split IRQ setup in different functions that match the 3 supported injection techniques (user handled eventfd, irqfd, forwarded IRQ):
removes dynamic switch between injection methods
- introduce fake interrupts as a test modality:
x makes possible to test multiple IRQ user-side handling. x this is a test feature only: enable to trigger a fd as if the
real physical IRQ hit. No virtual IRQ is injected into the guest but handling is simulated so that the state machine can be tested
- user handled eventfd:
x add mutex to protect IRQ state & list manipulation,
x correct misleading comment in vfio_intp_interrupt.
x Fix bugs using fake interrupt modality
- irqfd no more advertised in this patchset (handled in [3])
- VFIOPlatformDeviceClass becomes abstract and Calxeda xgmac device and class is re-introduced (as per v4)
- all DPRINTF removed in platform and replaced by trace-points
- corrects compilation with configure --disable-kvm
- simplifies the split for vfio_get_device and introduce a unique specialized function named vfio_populate_device
- group_list renamed into vfio_group_list
- hw/arm/dyn_sysbus_devtree.c currently only support vfio-calxeda-xgmac instantiation. Needs to be specialized for other VFIO devices
- fix 2 bugs in dyn_sysbus_devtree(reg_attr index and compat)
v4->v5:
- rebase on v2.1.0 PCI code
- take into account Alex Williamson comments on PCI code rework
- trace updates in vfio_region_write/read
- remove fd from VFIORegion
- get/put ckeanup
- bug fix: bar region's vbasedev field duly initialization
- misc cleanups in platform device
- device tree node generation removed from device and handled in hw/arm/dyn_sysbus_devtree.c
- remove "hw/vfio: add an example calxeda_xgmac": with removal of device tree node generation we do not have so many things to implement in that derived device yet. May be re-introduced later on if needed typically for reset/migration.
- no GSI routing table anymore
v3->v4 changes (Eric Auger, Alvise Rigo)
- rebase on last VFIO PCI code (v2.1.0-rc0)
- full git history rework to ease PCI code change review
- mv include files in hw/vfio
- DPRINTF reformatting temporarily moved out
- support of VFIO virq (removal of resamplefd handler on user-side)
- integration with sysbus dynamic instantiation framwork
- removal of unrealize and cleanup routines until it is better understood what is really needed
- Support of VFIO for Amba devices should be handled in an inherited device to specialize the device tree generation (clock handle currently missing in framework however)
- "Always use eventfd as notifying mechanism" temporarily moved out
- static instantiation is not mainstream (although it remains possible) note if static instantiation is used, irqfd must be setup in machine file when virtual IRQ is known
- create the GSI routing table on qemu side
v2->v3 changes (Alvise Rigo, Eric Auger):
- Following Alex W recommandations, further efforts to factorize the code between PCI:introduction of VFIODevice and VFIORegion as base classes
- unique reset handler for platform and PCI
- cleanup following Kim's comments
- multiple IRQ support mechanics should be in place although not tested
- Better handling of MMIO multiple regions
- New features and fixes by Alvise (multiple compat string, exec flag, force eventfd usage, amba device tree support)
- irqfd support
v1->v2 changes (Kim Phillips, Eric Auger):
- IRQ initial support (legacy mode where eventfds are handled on user side)
- hacked dynamic instantiation
v1 (Kim Phillips):
- initial split between PCI and platform
- MMIO support only
- static instantiation
Best Regards
Eric
Eric Auger (6):
kvm: rename kvm_irqchip_[add,remove]_irqfd_notifier with gsi suffix kvm-all.c: add qemu_irq/gsi hash table and utility routines
intc: arm_gic_kvm: set the qemu_irq/gsi mapping
sysbus: add irq_routing_notifier
kvm: some fixes to kvm_resamplefds_allowed hw/vfio/platform: add irqfd support
hw/core/sysbus.c | 6 +++ hw/intc/arm_gic_kvm.c | 6 +++ hw/s390x/virtio-ccw.c | 8 ++-- hw/vfio/pci.c | 6 +-- hw/vfio/platform.c | 100 ++++++++++++++++++++++++++++++++++++++++ hw/virtio/virtio-pci.c | 4 +- include/hw/sysbus.h | 1 + include/hw/vfio/vfio-platform.h | 2 + include/sysemu/kvm.h | 12 ++++- kvm-all.c | 40 ++++++++++++++-- kvm-stub.c | 8 ++-- trace-events | 1 + 12 files changed, 177 insertions(+), 17 deletions(-)
Signature · 1 line
1.8.3.2
PATCH v18 1/6 kvm: rename kvm_irqchip_[add, remove]_irqfd_notifier with gsi suffix
Anticipating for the introduction of new add/remove functions taking a qemu_irq parameter, let's rename existing ones with a gsi suffix.
---
v16 -> v17:
- added Paolo's A-b
v15 -> v16:
- added Vikram's T-b
- resolve rebase conflict in kvm.h
--- hw/s390x/virtio-ccw.c | 8 ++++---- hw/vfio/pci.c | 6 +++--- hw/virtio/virtio-pci.c | 4 ++-- include/sysemu/kvm.h | 7 ++++--- kvm-all.c | 7 ++++--- kvm-stub.c | 7 ++++--- 6 files changed, 21 insertions(+), 18 deletions(-)
hw/s390x/virtio-ccw.c +4 -4
@@ -1186,8 +1186,8 @@ static int virtio_ccw_add_irqfd(VirtioCcwDevice *dev, int n)VirtQueue *vq =virtio_get_queue (vdev, n);EventNotifier *notifier =virtio_queue_get_guest_notifier (vq); -return kvm_irqchip_add_irqfd_notifier (kvm_state, notifier,NULL , - dev->routes.gsi[n]); +return kvm_irqchip_add_irqfd_notifier_gsi (kvm_state, notifier,NULL , + dev->routes.gsi[n]); }static void virtio_ccw_remove_irqfd (VirtioCcwDevice *dev,int n)
@@ -1197,8 +1197,8 @@ static void virtio_ccw_remove_irqfd(VirtioCcwDevice *dev, int n)EventNotifier *notifier =virtio_queue_get_guest_notifier (vq);int ret; - ret =kvm_irqchip_remove_irqfd_notifier (kvm_state, notifier, - dev->routes.gsi[n]); + ret =kvm_irqchip_remove_irqfd_notifier_gsi (kvm_state, notifier, + dev->routes.gsi[n]);assert (ret ==0 ); }
hw/vfio/pci.c +3 -3
@@ -597,7 +597,7 @@ static void vfio_add_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage *msg,return ; } -if (kvm_irqchip_add_irqfd_notifier (kvm_state, &vector->kvm_interrupt, +if (kvm_irqchip_add_irqfd_notifier_gsi (kvm_state, &vector->kvm_interrupt,NULL , virq) <0 ) {kvm_irqchip_release_virq (kvm_state, virq);event_notifier_cleanup (&vector->kvm_interrupt);
@@ -609,8 +609,8 @@ static void vfio_add_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage *msg,static void vfio_remove_kvm_msi_virq (VFIOMSIVector *vector) { -kvm_irqchip_remove_irqfd_notifier (kvm_state, &vector->kvm_interrupt, - vector->virq); +kvm_irqchip_remove_irqfd_notifier_gsi (kvm_state, &vector->kvm_interrupt, + vector->virq);kvm_irqchip_release_virq (kvm_state, vector->virq); vector->virq =-1 ;event_notifier_cleanup (&vector->kvm_interrupt);
hw/virtio/virtio-pci.c +2 -2
@@ -495,7 +495,7 @@ static int kvm_virtio_pci_irqfd_use(VirtIOPCIProxy *proxy,VirtQueue *vq =virtio_get_queue (vdev, queue_no);EventNotifier *n =virtio_queue_get_guest_notifier (vq);int ret; - ret =kvm_irqchip_add_irqfd_notifier (kvm_state, n,NULL , irqfd->virq); + ret =kvm_irqchip_add_irqfd_notifier_gsi (kvm_state, n,NULL , irqfd->virq);return ret; }
@@ -509,7 +509,7 @@ static void kvm_virtio_pci_irqfd_release(VirtIOPCIProxy *proxy,VirtIOIRQFD *irqfd = &proxy->vector_irqfd[vector];int ret; - ret =kvm_irqchip_remove_irqfd_notifier (kvm_state, n, irqfd->virq); + ret =kvm_irqchip_remove_irqfd_notifier_gsi (kvm_state, n, irqfd->virq);assert (ret ==0 ); }
include/sysemu/kvm.h +4 -3
@@ -416,9 +416,10 @@ void kvm_irqchip_release_virq(KVMState *s, int virq);int kvm_irqchip_add_adapter_route (KVMState *s,AdapterInfo *adapter); -int kvm_irqchip_add_irqfd_notifier (KVMState *s,EventNotifier *n, -EventNotifier *rn,int virq); -int kvm_irqchip_remove_irqfd_notifier (KVMState *s,EventNotifier *n,int virq); +int kvm_irqchip_add_irqfd_notifier_gsi (KVMState *s,EventNotifier *n, +EventNotifier *rn,int virq); +int kvm_irqchip_remove_irqfd_notifier_gsi (KVMState *s,EventNotifier *n, +int virq);void kvm_pc_gsi_handler (void *opaque,int n,int level);void kvm_pc_setup_irq_routing (bool pci_enabled);void kvm_init_irq_routing (KVMState *s);
kvm-all.c +4 -3
@@ -1325,14 +1325,15 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg) }#endif /* !KVM_CAP_IRQ_ROUTING */ -int kvm_irqchip_add_irqfd_notifier (KVMState *s,EventNotifier *n, -EventNotifier *rn,int virq) +int kvm_irqchip_add_irqfd_notifier_gsi (KVMState *s,EventNotifier *n, +EventNotifier *rn,int virq) {return kvm_irqchip_assign_irqfd (s,event_notifier_get_fd (n), rn ?event_notifier_get_fd (rn) :-1 , virq, true); } -int kvm_irqchip_remove_irqfd_notifier (KVMState *s,EventNotifier *n,int virq) +int kvm_irqchip_remove_irqfd_notifier_gsi (KVMState *s,EventNotifier *n, +int virq) {return kvm_irqchip_assign_irqfd (s,event_notifier_get_fd (n),-1 , virq, false);
kvm-stub.c +4 -3
@@ -137,13 +137,14 @@ int kvm_irqchip_add_adapter_route(KVMState *s, AdapterInfo *adapter)return -ENOSYS ; } -int kvm_irqchip_add_irqfd_notifier (KVMState *s,EventNotifier *n, -EventNotifier *rn,int virq) +int kvm_irqchip_add_irqfd_notifier_gsi (KVMState *s,EventNotifier *n, +EventNotifier *rn,int virq) {return -ENOSYS ; } -int kvm_irqchip_remove_irqfd_notifier (KVMState *s,EventNotifier *n,int virq) +int kvm_irqchip_remove_irqfd_notifier_gsi (KVMState *s,EventNotifier *n, +int virq) {return -ENOSYS ; }
Signature · 1 line
1.8.3.2
Re: [Qemu-devel] [PATCH v18 1/6] kvm: rename kvm_irqchip_[add, remove]_irqfd_notifier with gsi suffix
Quoted 6 lines · Eric Auger
Anticipating for the introduction of new add/remove functions taking a
qemu_irqparameter, let's rename existing ones with a gsi suffix.Signed-off-by Eric Auger <[email protected]>Tested-by Vikram Sethi <[email protected]>Acked-by Paolo Bonzini <[email protected]>
thanks
-- PMM
PATCH v18 2/6 kvm-all.c: add qemu_irq/gsi hash table and utility routines
In reply to Eric AugerVFIO platform device needs to setup irqfd but it does not know the gsi corresponding to the device qemu_irq. This patch proposes to store a hash table in kvm_state using the qemu_irq as key and the gsi as a value.
kvm_irqchip_set_qemuirq_gsi allows to insert such a pair. The interrupt controller is supposed to use it.
kvm_irqchip_[add, remove]_irqfd_notifier allows to setup/tear down irqfd directly from the qemu_irq.
---
v16 -> v17
- replace GPOINTER_TO_UINT by GPOINTER_TO_INT
- add Paolo's ack
v15 -> v16:
- Added Vikram's T-b
v13 -> v14:
- correct checkpatch warning
v2 -> v3 (integration into VFIO series v13):
- rename kvm_irqchip_[add, remove]_qemuirq_irqfd_notifier into kvm_irqchip_[add, remove]_irqfd_notifier. Possible since legacy functions were also renamed with _gsi suffix.
V1 -> v2:
- qemu_irq get_gsi callback replaced by hash table stored in kvm
--- include/sysemu/kvm.h | 6 ++++++ kvm-all.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/sysemu/kvm.h +6 -0
@@ -19,6 +19,7 @@#include "qemu/queue.h" #include "qom/cpu.h" #include "exec/memattrs.h" +#include "hw/irq.h" #ifdef CONFIG_KVM #include <linux/kvm.h>
@@ -420,6 +421,11 @@ int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n, EventNotifier *rn, int virq);int kvm_irqchip_remove_irqfd_notifier_gsi (KVMState *s,EventNotifier *n,int virq); +int kvm_irqchip_add_irqfd_notifier (KVMState *s,EventNotifier *n, +EventNotifier *rn,qemu_irq irq); +int kvm_irqchip_remove_irqfd_notifier (KVMState *s,EventNotifier *n, +qemu_irq irq); +void kvm_irqchip_set_qemuirq_gsi (KVMState *s,qemu_irq irq,int gsi);void kvm_pc_gsi_handler (void *opaque,int n,int level);void kvm_pc_setup_irq_routing (bool pci_enabled);void kvm_init_irq_routing (KVMState *s);
kvm-all.c +33 -0
@@ -37,6 +37,7 @@#include "exec/address-spaces.h" #include "qemu/event_notifier.h" #include "trace.h" +#include "hw/irq.h" #include "hw/boards.h"
@@ -98,6 +99,7 @@ struct KVMState* unsigned, and treating them as signed here can break things */ unsigned irq_set_ioctl;unsigned int sigmask_len; +GHashTable *gsimap;#ifdef KVM_CAP_IRQ_ROUTING struct kvm_irq_routing *irq_routes;int nr_allocated_irq_routes;
@@ -1339,6 +1341,35 @@ int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n, false); } +int kvm_irqchip_add_irqfd_notifier (KVMState *s,EventNotifier *n, +EventNotifier *rn,qemu_irq irq) +{ +gpointer key, gsi; +gboolean found =g_hash_table_lookup_extended (s->gsimap, irq, &key, &gsi); + +if (!found) { +return -ENXIO ; + } +return kvm_irqchip_add_irqfd_notifier_gsi (s, n, rn,GPOINTER_TO_INT (gsi)); +} + +int kvm_irqchip_remove_irqfd_notifier (KVMState *s,EventNotifier *n, +qemu_irq irq) +{ +gpointer key, gsi; +gboolean found =g_hash_table_lookup_extended (s->gsimap, irq, &key, &gsi); + +if (!found) { +return -ENXIO ; + } +return kvm_irqchip_remove_irqfd_notifier_gsi (s, n,GPOINTER_TO_INT (gsi)); +} + +void kvm_irqchip_set_qemuirq_gsi (KVMState *s,qemu_irq irq,int gsi) +{ +g_hash_table_insert (s->gsimap, irq,GINT_TO_POINTER (gsi)); +} +static int kvm_irqchip_create (MachineState *machine,KVMState *s) {int ret;
@@ -1371,6 +1402,8 @@ static int kvm_irqchip_create(MachineState *machine, KVMState *s)kvm_init_irq_routing (s); + s->gsimap =g_hash_table_new (g_direct_hash, g_direct_equal); +return 0 ; }
Signature · 1 line
1.8.3.2
Re: [Qemu-devel] [PATCH v18 2/6] kvm-all.c: add qemu_irq/gsi hash table and utility routines
Quoted 8 lines · Eric Auger
VFIO platform device needs to setup irqfd but it does not know the gsi corresponding to the device
qemu_irq. This patch proposes to store a hash table inkvm_stateusing theqemu_irqas key and the gsi as a value.
kvm_irqchip_set_qemuirq_gsiallows to insert such a pair. The interrupt controller is supposed to use it.
Quoted 6 lines
kvm_irqchip_[add, remove]_irqfd_notifier allows to setup/tear down irqfd directly from the
qemu_irq.Signed-off-by Eric Auger <[email protected]>Tested-by Vikram Sethi <[email protected]>Acked-by Paolo Bonzini <[email protected]>
thanks
-- PMM
PATCH v18 3/6 intc: arm_gic_kvm: set the qemu_irq/gsi mapping
In reply to Eric AugerThe arm_gic_kvm now calls kvm_irqchip_set_qemuirq_gsi to build the hash table storing qemu_irq/gsi mappings. From that point on irqfd can be setup directly from the qemu_irq using kvm_irqchip_add_irqfd_notifier.
---
v16 -> v17:
- add Paolo's A-b
v15 -> v16:
- added Vikram's T-b
- Resolve rebase conflict
v2 -> v3:
- kvm_irqchip_add_qemuirq_irqfd_notifier renamed into kvm_irqchip_add_irqfd_notifier
--- hw/intc/arm_gic_kvm.c | 6 ++++++ 1 file changed, 6 insertions(+)
hw/intc/arm_gic_kvm.c +6 -0
@@ -570,6 +570,12 @@ static void kvm_arm_gic_realize(DeviceState *dev, Error **errp)*/ i += (GIC_INTERNAL * s->num_cpu);qdev_init_gpio_in (dev, kvm_arm_gic_set_irq, i); + +for (i =0 ; i < s->num_irq -GIC_INTERNAL ; i++) { +qemu_irq irq =qdev_get_gpio_in (dev, i); +kvm_irqchip_set_qemuirq_gsi (kvm_state, irq, i); + } +/* We never use our outbound IRQ/FIQ lines but provide them so that * we maintain the same interface as the non-KVM GIC. */
Signature · 1 line
1.8.3.2
Re: [Qemu-devel] [PATCH v18 3/6] intc: arm_gic_kvm: set the qemu_irq/gsi mapping
Quoted 8 lines · Eric Auger
The
arm_gic_kvmnow callskvm_irqchip_set_qemuirq_gsito build the hash table storing qemu_irq/gsi mappings. From that point on irqfd can be setup directly from theqemu_irqusingkvm_irqchip_add_irqfd_notifier.Signed-off-by Eric Auger <[email protected]>Tested-by Vikram Sethi <[email protected]>Acked-by Paolo Bonzini <[email protected]>
thanks
-- PMM
PATCH v18 4/6 sysbus: add irq_routing_notifier
In reply to Eric AugerAdd a new connect_irq_notifier notifier in the SysBusDeviceClass. This notifier, if populated, is called after sysbus_connect_irq.
This mechanism is used to setup VFIO signaling once VFIO platform devices get attached to their platform bus, on a machine init done notifier.
---
v15 -> v16:
- add Vikram's T-b
v14 -> v15:
- add Peter R-b
v2 -> v3 (integrated into this series v14):
- rename irq_routing_notifier into connect_irq_notifier
v1 -> v2:
- duly put the notifier in the class and not in the device
--- hw/core/sysbus.c | 6 ++++++ include/hw/sysbus.h | 1 + 2 files changed, 7 insertions(+)
hw/core/sysbus.c +6 -0
@@ -109,7 +109,13 @@ qemu_irq sysbus_get_connected_irq(SysBusDevice *dev, int n)void sysbus_connect_irq (SysBusDevice *dev,int n,qemu_irq irq) { +SysBusDeviceClass *sbd =SYS_BUS_DEVICE_GET_CLASS (dev); +qdev_connect_gpio_out_named (DEVICE (dev),SYSBUS_DEVICE_GPIO_IRQ , n, irq); + +if (sbd->connect_irq_notifier) { + sbd->connect_irq_notifier (dev, irq); + } }/* Check whether an MMIO region exists */
include/hw/sysbus.h +1 -0
@@ -41,6 +41,7 @@ typedef struct SysBusDeviceClass {/*< public >*/ int (*init)(SysBusDevice *dev); +void (*connect_irq_notifier)(SysBusDevice *dev,qemu_irq irq); } SysBusDeviceClass;struct SysBusDevice {
Signature · 1 line
1.8.3.2
Re: [Qemu-devel] [PATCH v18 4/6] sysbus: add irq_routing_notifier
Quoted 10 lines · Eric Auger
Add a new
connect_irq_notifiernotifier in the SysBusDeviceClass. This notifier, if populated, is called aftersysbus_connect_irq.This mechanism is used to setup VFIO signaling once VFIO platform devices get attached to their platform bus, on a machine init done notifier.
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Crosthwaite <[email protected]>Tested-by Vikram Sethi <[email protected]>
thanks
-- PMM
PATCH v18 5/6 kvm: some fixes to kvm_resamplefds_allowed
In reply to Eric AugerCommit f41389ae3c54b introduced kvm_resamplefds_enabled() and associated kvm_resamplefds_allowed boolean. This patch adds non-KVM version for kvm_resamplefds_enabled and also declares kvm_resamplefds_allowed in kvm-stub as it is done for fellow kvm_irqfds_allowed.
--- include/sysemu/kvm.h | 1 + kvm-stub.c | 1 + 2 files changed, 2 insertions(+)
include/sysemu/kvm.h +1 -0
@@ -152,6 +152,7 @@ extern bool kvm_readonly_mem_allowed;#define kvm_halt_in_kernel () (false)#define kvm_eventfds_enabled () (false)#define kvm_irqfds_enabled () (false) +#define kvm_resamplefds_enabled () (false)#define kvm_msi_via_irqfd_enabled () (false)#define kvm_gsi_routing_allowed () (false)#define kvm_gsi_direct_mapping () (false)
kvm-stub.c +1 -0
@@ -24,6 +24,7 @@ bool kvm_kernel_irqchip;bool kvm_async_interrupts_allowed;bool kvm_eventfds_allowed;bool kvm_irqfds_allowed; +bool kvm_resamplefds_allowed;bool kvm_msi_via_irqfd_allowed;bool kvm_gsi_routing_allowed;bool kvm_gsi_direct_mapping;
Signature · 1 line
1.8.3.2
Re: [Qemu-devel] [PATCH v18 5/6] kvm: some fixes to kvm_resamplefds_allowed
Quoted 37 lines · Eric Auger
Commit
f41389ae3c54bintroducedkvm_resamplefds_enabled()and associatedkvm_resamplefds_allowedboolean. This patch adds non-KVM version forkvm_resamplefds_enabledand also declareskvm_resamplefds_allowedin kvm-stub as it is done for fellowkvm_irqfds_allowed.Signed-off-by Eric Auger <[email protected]>--- include/sysemu/kvm.h | 1 + kvm-stub.c | 1 + 2 files changed, 2 insertions(+)include/sysemu/kvm.h +1
-0@@ -152,6 +152,7 @@ extern bool kvm_readonly_mem_allowed;#define kvm_halt_in_kernel () (false)#define kvm_eventfds_enabled () (false)#define kvm_irqfds_enabled () (false) +#define kvm_resamplefds_enabled () (false)#define kvm_msi_via_irqfd_enabled () (false)#define kvm_gsi_routing_allowed () (false)#define kvm_gsi_direct_mapping () (false)kvm-stub.c +1
-1@@ -24,6 +24,7 @@ bool kvm_kernel_irqchip;bool kvm_async_interrupts_allowed;bool kvm_eventfds_allowed;bool kvm_irqfds_allowed; +bool kvm_resamplefds_allowed;bool kvm_msi_via_irqfd_allowed;bool kvm_gsi_routing_allowed;bool kvm_gsi_direct_mapping; --
Quoted 1 line
1.8.3.2
though "kvm: implement kvm_resamplefds_enabled() for !CONFIG_KVM" might be a more informative subject line if you need to respin for some other reason.
thanks
-- PMM
Re: [Qemu-devel] [PATCH v18 5/6] kvm: some fixes to kvm_resamplefds_allowed
Quoted 42 lines · Peter Maydell
Quoted 37 lines · Eric Auger
Commit
f41389ae3c54bintroducedkvm_resamplefds_enabled()and associatedkvm_resamplefds_allowedboolean. This patch adds non-KVM version forkvm_resamplefds_enabledand also declareskvm_resamplefds_allowedin kvm-stub as it is done for fellowkvm_irqfds_allowed.Signed-off-by Eric Auger <[email protected]>--- include/sysemu/kvm.h | 1 + kvm-stub.c | 1 + 2 files changed, 2 insertions(+)include/sysemu/kvm.h +1
-0@@ -152,6 +152,7 @@ extern bool kvm_readonly_mem_allowed;#define kvm_halt_in_kernel () (false)#define kvm_eventfds_enabled () (false)#define kvm_irqfds_enabled () (false) +#define kvm_resamplefds_enabled () (false)#define kvm_msi_via_irqfd_enabled () (false)#define kvm_gsi_routing_allowed () (false)#define kvm_gsi_direct_mapping () (false)kvm-stub.c +1
-1@@ -24,6 +24,7 @@ bool kvm_kernel_irqchip;bool kvm_async_interrupts_allowed;bool kvm_eventfds_allowed;bool kvm_irqfds_allowed; +bool kvm_resamplefds_allowed;bool kvm_msi_via_irqfd_allowed;bool kvm_gsi_routing_allowed;bool kvm_gsi_direct_mapping; --Quoted 1 line
1.8.3.2
Reviewed-by Peter Maydell <[email protected]>
Quoted 3 lines
though "kvm: implement
kvm_resamplefds_enabled()for !CONFIG_KVM" might be a more informative subject line if you need to respin for some other reason.
OK. Thanks for the whole review
Best Regards
Eric
Quoted 4 lines
thanks
-- PMM
PATCH v18 6/6 hw/vfio/platform: add irqfd support
In reply to Eric AugerThis patch aims at optimizing IRQ handling using irqfd framework.
Instead of handling the eventfds on user-side they are handled on kernel side using
- the KVM irqfd framework,
- the VFIO driver virqfd framework.
the virtual IRQ completion is trapped at interrupt controller This removes the need for fast/slow path swap.
Overall this brings significant performance improvements.
---
v17 -> v18:
- remove found in vfio_start_irqfd_injection
- unconditionnally set sbc->connect_irq_notifier in the class_init function and do the proper check in vfio_start_irqfd_injection
v16 -> v17:
- Take into account Peter's comments
- most importantly remove #ifdef CONFIG_KVM
v15 -> v16:
- add Vikram's T-b
v13 -> v14:
- use connect_irq_notifier
- remove trace_vfio_platform_start_eventfd
v12 -> v13:
- setup the new mechanism for starting irqfd, based on LinkPropertySetter override
- use kvm_irqchip_[add,remove]_irqfd_notifier new functions: no need to bother about gsi (hence virtualID could be removed with small change in trace-events)
v10 -> v11:
- Add Alex' Reviewed-by
- introduce kvm_accel in this patch and initialize it
v5 -> v6
- rely on kvm_irqfds_enabled() and kvm_resamplefds_enabled()
- guard KVM code with #ifdef CONFIG_KVM
v3 -> v4:
[Alvise Rigo]
Use of VFIO Platform driver v6 unmask/virqfd feature and removal of resamplefd handler. Physical IRQ unmasking is now done in VFIO driver.
v3:
[Eric Auger]
initial support with resamplefd handled on QEMU side since the unmask was not supported on VFIO platform driver v5.
--- hw/vfio/platform.c | 100 ++++++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-platform.h | 2 + trace-events | 1 + 3 files changed, 103 insertions(+)
hw/vfio/platform.c +100 -0
@@ -26,6 +26,7 @@#include "hw/sysbus.h" #include "trace.h" #include "hw/platform-bus.h" +#include "sysemu/kvm.h" /* * Functions used whatever the injection method
@@ -51,6 +52,7 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev, intp->pin = info.index; intp->flags = info.flags; intp->state =VFIO_IRQ_INACTIVE ; + intp->kvm_accel = false;sysbus_init_irq (sbdev, &intp->qemuirq);
@@ -61,6 +63,13 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,error_report ("vfio: Error: trigger event_notifier_init failed " );return NULL ; } +/* Get an eventfd for resample/unmask */ + ret =event_notifier_init (&intp->unmask,0 ); +if (ret) { +g_free (intp); +error_report ("vfio: Error: resamplefd event_notifier_init failed" ); +return NULL ; + }QLIST_INSERT_HEAD (&vdev->intp_list, intp, next);return intp;
@@ -315,6 +324,94 @@ static int vfio_start_eventfd_injection(VFIOINTp *intp)return ret; } +/* +* Functions used for irqfd +*/ + +/** +* vfio_set_resample_eventfd - sets the resamplefd for an IRQ +* @intp: the IRQ struct handle +* programs the VFIO driver to unmask this IRQ when the +* intp->unmask eventfd is triggered +*/ +static int vfio_set_resample_eventfd (VFIOINTp *intp) +{ +VFIODevice *vbasedev = &intp->vdev->vbasedev; +struct vfio_irq_set *irq_set; +int argsz, ret; +int32_t *pfd; + + argsz =sizeof (*irq_set) +sizeof (*pfd); + irq_set =g_malloc0 (argsz); + irq_set->argsz = argsz; + irq_set->flags =VFIO_IRQ_SET_DATA_EVENTFD |VFIO_IRQ_SET_ACTION_UNMASK ; + irq_set->index = intp->pin; + irq_set->start =0 ; + irq_set->count =1 ; + pfd = (int32_t *)&irq_set->data; + *pfd =event_notifier_get_fd (&intp->unmask); +qemu_set_fd_handler (*pfd,NULL ,NULL ,NULL ); + ret =ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set); +g_free (irq_set); +if (ret <0 ) { +error_report ("vfio: Failed to set resample eventfd: %m" ); + } +return ret; +} + +static void vfio_start_irqfd_injection (SysBusDevice *sbdev,qemu_irq irq) +{ +VFIOPlatformDevice *vdev =VFIO_PLATFORM_DEVICE (sbdev); +VFIOINTp *intp; + +if (!kvm_irqfds_enabled () || !kvm_resamplefds_enabled () || + !vdev->irqfd_allowed) { +return ; + } + + QLIST_FOREACH(intp, &vdev->intp_list, next) { +if (intp->qemuirq == irq) { +break ; + } + } +assert (intp); + +/* Get to a known interrupt state */ +qemu_set_fd_handler (event_notifier_get_fd (&intp->interrupt), +NULL ,NULL , vdev); + +vfio_mask_single_irqindex (&vdev->vbasedev, intp->pin); +qemu_set_irq (intp->qemuirq,0 ); + +if (kvm_irqchip_add_irqfd_notifier (kvm_state, &intp->interrupt, + &intp->unmask, irq) <0 ) { + goto fail_irqfd; + } + +if (vfio_set_trigger_eventfd (intp,NULL ) <0 ) { + goto fail_vfio; + } +if (vfio_set_resample_eventfd (intp) <0 ) { + goto fail_vfio; + } + +/* Let's resume injection with irqfd setup */ +vfio_unmask_single_irqindex (&vdev->vbasedev, intp->pin); + + intp->kvm_accel = true; + +trace_vfio_platform_start_irqfd_injection (intp->pin, +event_notifier_get_fd (&intp->interrupt), +event_notifier_get_fd (&intp->unmask)); +return ; +fail_vfio: +kvm_irqchip_remove_irqfd_notifier (kvm_state, &intp->interrupt, irq); +fail_irqfd: +vfio_start_eventfd_injection (intp); +vfio_unmask_single_irqindex (&vdev->vbasedev, intp->pin); +return ; +} +/* VFIO skeleton */ static void vfio_platform_compute_needs_reset (VFIODevice *vbasedev)
@@ -584,17 +681,20 @@ static Property vfio_platform_dev_properties[] = {DEFINE_PROP_BOOL ("x-mmap" , VFIOPlatformDevice, vbasedev.allow_mmap, true),DEFINE_PROP_UINT32 ("mmap-timeout-ms" , VFIOPlatformDevice, mmap_timeout,1100 ), +DEFINE_PROP_BOOL ("x-irqfd" , VFIOPlatformDevice, irqfd_allowed, true), DEFINE_PROP_END_OF_LIST(), };static void vfio_platform_class_init (ObjectClass *klass,void *data) {DeviceClass *dc =DEVICE_CLASS (klass); +SysBusDeviceClass *sbc =SYS_BUS_DEVICE_CLASS (klass); dc->realize = vfio_platform_realize; dc->props = vfio_platform_dev_properties; dc->vmsd = &vfio_platform_vmstate; dc->desc ="VFIO-based platform device assignment" ; + sbc->connect_irq_notifier = vfio_start_irqfd_injection;set_bit (DEVICE_CATEGORY_MISC , dc->categories); }
include/hw/vfio/vfio-platform.h +2 -0
@@ -41,6 +41,7 @@ typedef struct VFIOINTp {int state;/* inactive, pending, active */ uint8_t pin;/* index */ uint32_t flags;/* IRQ info flags */ +bool kvm_accel;/* set when QEMU bypass through KVM enabled */ } VFIOINTp;/* function type for user side eventfd handler */
@@ -57,6 +58,7 @@ typedef struct VFIOPlatformDevice {uint32_t mmap_timeout;/* delay to re-enable mmaps after interrupt */ QEMUTimer *mmap_timer;/* allows fast-path resume after IRQ hit */ QemuMutex intp_mutex;/* protect the intp_list IRQ state */ +bool irqfd_allowed;/* debug option to force irqfd on/off */ } VFIOPlatformDevice; typedef struct VFIOPlatformDeviceClass {
trace-events +1 -0
@@ -1594,6 +1594,7 @@ vfio_platform_intp_interrupt(int pin, int fd) "Inject IRQ #%d (fd = %d)" vfio_platform_intp_inject_pending_lockheld(int pin, int fd) "Inject pending IRQ #%d (fd = %d)" vfio_platform_populate_interrupts(int pin, int count, int flags) "- IRQ index %d: count %d, flags=0x%x" vfio_intp_interrupt_set_pending(int index) "irq %d is set PENDING" +vfio_platform_start_irqfd_injection(int index, int fd, int resamplefd) "IRQ index=%d, fd = %d, resamplefd = %d" #hw/acpi/memory_hotplug.c mhp_acpi_invalid_slot_selected(uint32_t slot) "0x%"PRIx32
Signature · 1 line
1.8.3.2
Re: [Qemu-devel] [PATCH v18 6/6] hw/vfio/platform: add irqfd support
Quoted 10 lines · Eric Auger
This patch aims at optimizing IRQ handling using irqfd framework.
Instead of handling the eventfds on user-side they are handled on kernel side using
- the KVM irqfd framework,
- the VFIO driver virqfd framework.the virtual IRQ completion is trapped at interrupt controller This removes the need for fast/slow path swap.
Quoted 6 lines
Overall this brings significant performance improvements.
Signed-off-by Alvise Rigo <[email protected]>Signed-off-by Eric Auger <[email protected]>Reviewed-by Alex Bennée <[email protected]>Tested-by Vikram Sethi <[email protected]>
(by which I mean that I can't see anything obviously wrong and am happy for this mostly-ARM-related feature to go in, but am not qualified to review or test it; I think we've agreed that this patchset will all go in via Alex Williamson's tree.)
thanks
-- PMM