vfio-user client
v5 cover vfio-user client
The series contains an implementation of a vfio-user client in QEMU.
The vfio-user protocol allows for implementing (PCI) devices in another userspace process; SPDK is one example, which includes a virtual NVMe implementation.
The vfio-user framework consists of 3 parts:
1) The VFIO user protocol specification.
2) A client - the VFIO device in QEMU that encapsulates VFIO messages and sends them to the server.
3) A server - a remote process that emulates a device.
This patchset implements parts 1 and 2.
It has been tested against libvfio-user test servers as well as SPDK. A functional test is still being worked on.
A previous version of this series can be found at
https://lore.kernel.org/qemu-devel/[email protected]/
Changes since last series:
- fixed SPDX identifier nits
- code review changes for error handling
thanks
john
John Levon (18):
vfio-user: add vfio-user class and container
vfio-user: connect vfio proxy to remote server
vfio-user: implement message receive infrastructure
vfio-user: implement message send infrastructure
vfio-user: implement VFIO_USER_DEVICE_GET_INFO
vfio-user: implement VFIO_USER_DEVICE_GET_REGION_INFO
vfio-user: implement VFIO_USER_REGION_READ/WRITE
vfio-user: set up PCI in vfio_user_pci_realize()
vfio-user: implement VFIO_USER_DEVICE_GET/SET_IRQ*
vfio-user: forward MSI-X PBA BAR accesses to server
vfio-user: set up container access to the proxy
vfio-user: implement VFIO_USER_DEVICE_RESET
vfio-user: implement VFIO_USER_DMA_MAP/UNMAP
vfio-user: implement VFIO_USER_DMA_READ/WRITE
vfio-user: add 'x-msg-timeout' option
vfio-user: support posted writes
vfio-user: add coalesced posted writes
docs: add vfio-user documentation
Thanos Makatos (1):
vfio-user: introduce vfio-user protocol specification
MAINTAINERS | 11 +- docs/interop/index.rst | 1 + docs/interop/vfio-user.rst | 1520 +++++++++++++++++++++++++ docs/system/device-emulation.rst | 1 + docs/system/devices/vfio-user.rst | 26 + meson.build | 1 + hw/vfio-user/container.h | 23 + hw/vfio-user/device.h | 24 + hw/vfio-user/protocol.h | 242 ++++ hw/vfio-user/proxy.h | 135 +++ hw/vfio-user/trace.h | 4 + hw/vfio/pci.h | 1 + include/hw/vfio/vfio-container-base.h | 1 + include/hw/vfio/vfio-device.h | 2 + hw/vfio-user/container.c | 370 ++++++ hw/vfio-user/device.c | 441 +++++++ hw/vfio-user/pci.c | 475 ++++++++ hw/vfio-user/proxy.c | 1356 ++++++++++++++++++++++ hw/Kconfig | 1 + hw/meson.build | 1 + hw/vfio-user/Kconfig | 7 + hw/vfio-user/meson.build | 11 + hw/vfio-user/trace-events | 20 + 23 files changed, 4673 insertions(+), 1 deletion(-) create mode 100644 docs/interop/vfio-user.rst create mode 100644 docs/system/devices/vfio-user.rst create mode 100644 hw/vfio-user/container.h create mode 100644 hw/vfio-user/device.h create mode 100644 hw/vfio-user/protocol.h create mode 100644 hw/vfio-user/proxy.h create mode 100644 hw/vfio-user/trace.h create mode 100644 hw/vfio-user/container.c create mode 100644 hw/vfio-user/device.c create mode 100644 hw/vfio-user/pci.c create mode 100644 hw/vfio-user/proxy.c create mode 100644 hw/vfio-user/Kconfig create mode 100644 hw/vfio-user/meson.build create mode 100644 hw/vfio-user/trace-events
Signature · 1 line
2.43.0
PATCH v5 1/19 vfio-user: add vfio-user class and container
Introduce basic plumbing for vfio-user with CONFIG_VFIO_USER.
We introduce VFIOUserContainer in hw/vfio-user/container.c, which is a container type for the "IOMMU" type "vfio-iommu-user", and share some common container code from hw/vfio/container.c.
Add hw/vfio-user/pci.c for instantiating VFIOUserPCIDevice objects, sharing some common code from hw/vfio/pci.c.
--- MAINTAINERS | 8 + hw/vfio-user/container.h | 21 +++ include/hw/vfio/vfio-container-base.h | 1 + hw/vfio-user/container.c | 208 ++++++++++++++++++++++++++ hw/vfio-user/pci.c | 185 +++++++++++++++++++++++ hw/Kconfig | 1 + hw/meson.build | 1 + hw/vfio-user/Kconfig | 7 + hw/vfio-user/meson.build | 9 ++ 9 files changed, 441 insertions(+) create mode 100644 hw/vfio-user/container.h create mode 100644 hw/vfio-user/container.c create mode 100644 hw/vfio-user/pci.c create mode 100644 hw/vfio-user/Kconfig create mode 100644 hw/vfio-user/meson.build
MAINTAINERS +8 -0
@@ -4253,6 +4253,14 @@ F: hw/remote/iommu.c F: include/hw/remote/iommu.h F: tests/functional/test_multiprocess.py +VFIO-USER: +M: John Levon <[email protected]> +M: Thanos Makatos <[email protected]> +S: Supported +F: hw/vfio-user/* +F: include/hw/vfio-user/* +F: subprojects/libvfio-user + EBPF: M: Jason Wang <[email protected]> R: Andrew Melnychenko <[email protected]>
hw/vfio-user/container.h new file +21 -0
@@ -0,0 +1,21 @@ +/* +* vfio-user specific definitions. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef HW_VFIO_USER_CONTAINER_H +#define HW_VFIO_USER_CONTAINER_H + +#include "qemu/osdep.h" + +#include "hw/vfio/vfio-container-base.h" + +/* MMU container sub-class for vfio-user. */ +typedef struct VFIOUserContainer { +VFIOContainerBase bcontainer; +}VFIOUserContainer ; + +OBJECT_DECLARE_SIMPLE_TYPE (VFIOUserContainer,VFIO_IOMMU_USER ); + +#endif /* HW_VFIO_USER_CONTAINER_H */
include/hw/vfio/vfio-container-base.h +1 -0
@@ -109,6 +109,7 @@ vfio_container_get_page_size_mask(const VFIOContainerBase *bcontainer)#define TYPE_VFIO_IOMMU_LEGACY TYPE_VFIO_IOMMU "-legacy"#define TYPE_VFIO_IOMMU_SPAPR TYPE_VFIO_IOMMU "-spapr"#define TYPE_VFIO_IOMMU_IOMMUFD TYPE_VFIO_IOMMU "-iommufd" +#define TYPE_VFIO_IOMMU_USER TYPE_VFIO_IOMMU "-user"OBJECT_DECLARE_TYPE (VFIOContainerBase, VFIOIOMMUClass,VFIO_IOMMU )
hw/vfio-user/container.c new file +208 -0
208 lines not on this page · raw message
hw/vfio-user/pci.c new file +185 -0
185 lines not on this page · raw message
hw/Kconfig +1 -0
@@ -42,6 +42,7 @@ source ufs/Kconfig source usb/Kconfig source virtio/Kconfig source vfio/Kconfig +source vfio-user/Kconfig source vmapple/Kconfig source xen/Kconfig source watchdog/Kconfig
hw/meson.build +1 -0
@@ -39,6 +39,7 @@ subdir('uefi') subdir('ufs') subdir('usb') subdir('vfio') +subdir('vfio-user') subdir('virtio') subdir('vmapple') subdir('watchdog')
hw/vfio-user/Kconfig new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-or-later + +config VFIO_USER + bool + default y + depends on VFIO_PCI +
hw/vfio-user/meson.build new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0-or-later + +vfio_user_ss = ss.source_set() +vfio_user_ss.add(files( + 'container.c', + 'pci.c', +)) + +system_ss.add_all(when: 'CONFIG_VFIO_USER', if_true: vfio_user_ss)
Signature · 1 line
2.43.0
Re: [PATCH v5 01/19] vfio-user: add vfio-user class and container
Quoted 6 lines · John Levon
Introduce basic plumbing for vfio-user with
CONFIG_VFIO_USER.We introduce VFIOUserContainer in
hw/vfio-user/container.c, which is a container type for the "IOMMU" type "vfio-iommu-user", and share some common container code fromhw/vfio/container.c.
Quoted 7 lines
Add
hw/vfio-user/pci.cfor instantiating VFIOUserPCIDevice objects, sharing some common code fromhw/vfio/pci.c.Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Thanks,
C.
Re: [PATCH v5 01/19] vfio-user: add vfio-user class and container
In reply to John Levon+Steve
Quoted 169 lines · John Levon
Introduce basic plumbing for vfio-user with
CONFIG_VFIO_USER.We introduce VFIOUserContainer in
hw/vfio-user/container.c, which is a container type for the "IOMMU" type "vfio-iommu-user", and share some common container code fromhw/vfio/container.c.Add
hw/vfio-user/pci.cfor instantiating VFIOUserPCIDevice objects, sharing some common code fromhw/vfio/pci.c.Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>--- MAINTAINERS | 8 + hw/vfio-user/container.h | 21 +++ include/hw/vfio/vfio-container-base.h | 1 + hw/vfio-user/container.c | 208 ++++++++++++++++++++++++++ hw/vfio-user/pci.c | 185 +++++++++++++++++++++++ hw/Kconfig | 1 + hw/meson.build | 1 + hw/vfio-user/Kconfig | 7 + hw/vfio-user/meson.build | 9 ++ 9 files changed, 441 insertions(+)create mode 100644
hw/vfio-user/container.h
create mode 100644hw/vfio-user/container.c
create mode 100644hw/vfio-user/pci.c
create mode 100644 hw/vfio-user/Kconfig
create mode 100644 hw/vfio-user/meson.buildMAINTAINERS +8
-0@@ -4253,6 +4253,14 @@ F: hw/remote/iommu.c F: include/hw/remote/iommu.h F: tests/functional/test_multiprocess.py +VFIO-USER: +M: John Levon <[email protected]> +M: Thanos Makatos <[email protected]> +S: Supported +F: hw/vfio-user/* +F: include/hw/vfio-user/* +F: subprojects/libvfio-user + EBPF: M: Jason Wang <[email protected]> R: Andrew Melnychenko <[email protected]>hw/vfio-user/container.h new file +21
-0@@ -0,0 +1,21 @@ +/* +* vfio-user specific definitions. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef HW_VFIO_USER_CONTAINER_H +#define HW_VFIO_USER_CONTAINER_H + +#include "qemu/osdep.h" + +#include "hw/vfio/vfio-container-base.h" + +/* MMU container sub-class for vfio-user. */ +typedef struct VFIOUserContainer { +VFIOContainerBase bcontainer; +}VFIOUserContainer ; + +OBJECT_DECLARE_SIMPLE_TYPE (VFIOUserContainer,VFIO_IOMMU_USER ); + +#endif /* HW_VFIO_USER_CONTAINER_H */ include/hw/vfio/vfio-container-base.h +1
-0@@ -109,6 +109,7 @@ vfio_container_get_page_size_mask(const VFIOContainerBase *bcontainer)#define TYPE_VFIO_IOMMU_LEGACY TYPE_VFIO_IOMMU "-legacy"#define TYPE_VFIO_IOMMU_SPAPR TYPE_VFIO_IOMMU "-spapr"#define TYPE_VFIO_IOMMU_IOMMUFD TYPE_VFIO_IOMMU "-iommufd" +#define TYPE_VFIO_IOMMU_USER TYPE_VFIO_IOMMU "-user"OBJECT_DECLARE_TYPE (VFIOContainerBase, VFIOIOMMUClass,VFIO_IOMMU )hw/vfio-user/container.c new file +75
-0@@ -0,0 +1,208 @@ +/* +* Container for vfio-user IOMMU type: rather than communicating with the kernel +* vfio driver, we communicate over a socket to a server using the vfio-user +* protocol. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include <sys/ioctl.h> +#include <linux/vfio.h> +#include "qemu/osdep.h" + +#include "hw/vfio-user/container.h" +#include "hw/vfio/vfio-cpr.h" +#include "hw/vfio/vfio-device.h" +#include "hw/vfio/vfio-listener.h" +#include "qapi/error.h" + +static int vfio_user_dma_unmap (const VFIOContainerBase *bcontainer, +hwaddr iova,ram_addr_t size, +IOMMUTLBEntry *iotlb,bool unmap_all) +{ +return -ENOTSUP ; +} + +static int vfio_user_dma_map (const VFIOContainerBase *bcontainer,hwaddr iova, +ram_addr_t size,void *vaddr,bool readonly, +MemoryRegion *mrp) +{ +return -ENOTSUP ; +} + +static int +vfio_user_set_dirty_page_tracking (const VFIOContainerBase *bcontainer, +bool start,Error **errp) +{ +error_setg_errno (errp,ENOTSUP ,"Not supported" ); +return -ENOTSUP ; +} + +static int vfio_user_query_dirty_bitmap (const VFIOContainerBase *bcontainer, +VFIOBitmap *vbmap,hwaddr iova, +hwaddr size,Error **errp) +{ +error_setg_errno (errp,ENOTSUP ,"Not supported" ); +return -ENOTSUP ; +} + +static bool vfio_user_setup (VFIOContainerBase *bcontainer,Error **errp) +{ +error_setg_errno (errp,ENOTSUP ,"Not supported" ); +return -ENOTSUP ; +} + +static VFIOUserContainer *vfio_user_create_container (Error **errp) +{ +VFIOUserContainer *container; + + container =VFIO_IOMMU_USER (object_new (TYPE_VFIO_IOMMU_USER )); +return container; +} + +/* +* Try to mirror vfio_container_connect() as much as possible. +*/ +static VFIOUserContainer * +vfio_user_container_connect (AddressSpace *as,Error **errp) +{ +VFIOContainerBase *bcontainer; +VFIOUserContainer *container; +VFIOAddressSpace *space; +VFIOIOMMUClass *vioc; + + space =vfio_address_space_get (as); +
hw/vfio-user/container.c line 85
Quoted 10 lines
hw/vfio-user/container.c new file +10
-0+ container =vfio_user_create_container (errp); +if (!container) { + goto put_space_exit; + } + + bcontainer = &container->bcontainer; + +if (!vfio_cpr_register_container (bcontainer, errp)) { + goto free_container_exit; + }
I missed the CPR usage in vfio-user when reviewing and I think it should be addressed in a separate series. This also interferes with Steve's "live update" changes.
John,
Could please drop the CPR register ?
Thanks,
C.
Quoted 366 lines
hw/vfio-user/container.c new file +123
-0+ + vioc =VFIO_IOMMU_GET_CLASS (bcontainer); +assert (vioc->setup); + +if (!vioc->setup (bcontainer, errp)) { + goto unregister_container_exit; + } + +vfio_address_space_insert (space, bcontainer); + +if (!vfio_listener_register (bcontainer, errp)) { + goto listener_release_exit; + } + + bcontainer->initialized = true; + +return container; + +listener_release_exit: +vfio_listener_unregister (bcontainer); +if (vioc->release) { + vioc->release (bcontainer); + } + +unregister_container_exit: +vfio_cpr_unregister_container (bcontainer); + +free_container_exit: +object_unref (container); + +put_space_exit: +vfio_address_space_put (space); + +return NULL ; +} + +static void vfio_user_container_disconnect (VFIOUserContainer *container) +{ +VFIOContainerBase *bcontainer = &container->bcontainer; +VFIOIOMMUClass *vioc =VFIO_IOMMU_GET_CLASS (bcontainer); + +vfio_listener_unregister (bcontainer); +if (vioc->release) { + vioc->release (bcontainer); + } + +VFIOAddressSpace *space = bcontainer->space; + +vfio_cpr_unregister_container (bcontainer); +object_unref (container); + +vfio_address_space_put (space); +} + +static bool vfio_user_device_get (VFIOUserContainer *container, +VFIODevice *vbasedev,Error **errp) +{ +struct vfio_device_info info = {0 }; + + vbasedev->fd =-1 ; + +vfio_device_prepare (vbasedev, &container->bcontainer, &info); + +return true; +} + +/* +* vfio_user_device_attach: attach a device to a new container. +*/ +static bool vfio_user_device_attach (const char *name,VFIODevice *vbasedev, +AddressSpace *as,Error **errp) +{ +VFIOUserContainer *container; + + container =vfio_user_container_connect (as, errp); +if (container ==NULL ) { +error_prepend (errp,"failed to connect proxy" ); +return false; + } + +return vfio_user_device_get (container, vbasedev, errp); +} + +static void vfio_user_device_detach (VFIODevice *vbasedev) +{ +VFIOUserContainer *container =container_of (vbasedev->bcontainer, + VFIOUserContainer, bcontainer); + +vfio_device_unprepare (vbasedev); + +vfio_user_container_disconnect (container); +} + +static int vfio_user_pci_hot_reset (VFIODevice *vbasedev,bool single) +{ +/* ->needs_reset is always false for vfio-user. */ +return 0 ; +} + +static void vfio_iommu_user_class_init (ObjectClass *klass,const void *data) +{ +VFIOIOMMUClass *vioc =VFIO_IOMMU_CLASS (klass); + + vioc->setup = vfio_user_setup; + vioc->dma_map = vfio_user_dma_map; + vioc->dma_unmap = vfio_user_dma_unmap; + vioc->attach_device = vfio_user_device_attach; + vioc->detach_device = vfio_user_device_detach; + vioc->set_dirty_page_tracking = vfio_user_set_dirty_page_tracking; + vioc->query_dirty_bitmap = vfio_user_query_dirty_bitmap; + vioc->pci_hot_reset = vfio_user_pci_hot_reset; +}; + +static const TypeInfo types[] = { + { + .name =TYPE_VFIO_IOMMU_USER , + .parent =TYPE_VFIO_IOMMU , + .instance_size =sizeof (VFIOUserContainer), + .class_init = vfio_iommu_user_class_init, + }, +}; + +DEFINE_TYPES (types)hw/vfio-user/pci.c new file +185
-0@@ -0,0 +1,185 @@ +/* +* vfio PCI device over a UNIX socket. +* +* Copyright © 2018, 2021 Oracle and/or its affiliates. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include <sys/ioctl.h> +#include "qemu/osdep.h" +#include "qapi-visit-sockets.h" + +#include "hw/qdev-properties.h" +#include "hw/vfio/pci.h" + +#define TYPE_VFIO_USER_PCI "vfio-user-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) + +struct VFIOUserPCIDevice { +VFIOPCIDevice device; +SocketAddress *socket; +}; + +/* +* Emulated devices don't use host hot reset +*/ +static void vfio_user_compute_needs_reset (VFIODevice *vbasedev) +{ + vbasedev->needs_reset = false; +} + +static Object *vfio_user_pci_get_object (VFIODevice *vbasedev) +{ +VFIOUserPCIDevice *vdev =container_of (vbasedev, VFIOUserPCIDevice, + device.vbasedev); + +return OBJECT (vdev); +} + +static VFIODeviceOps vfio_user_pci_ops = { + .vfio_compute_needs_reset = vfio_user_compute_needs_reset, + .vfio_eoi = vfio_pci_intx_eoi, + .vfio_get_object = vfio_user_pci_get_object, +/* No live migration support yet. */ + .vfio_save_config =NULL , + .vfio_load_config =NULL , +}; + +static void vfio_user_pci_realize (PCIDevice *pdev,Error **errp) +{ +ERRP_GUARD (); +VFIOUserPCIDevice *udev =VFIO_USER_PCI (pdev); +VFIOPCIDevice *vdev =VFIO_PCI_BASE (pdev); +VFIODevice *vbasedev = &vdev->vbasedev; +const char *sock_name; +AddressSpace *as; + +if (!udev->socket) { +error_setg (errp,"No socket specified" ); +error_append_hint (errp,"e.g. -device '{" +"\"driver\":\"vfio-user-pci\", " +"\"socket\": {\"path\": \"/tmp/vfio-user.sock\", " +"\"type\": \"unix\"}'" +"}'\n" ); +return ; + } + + sock_name = udev->socket->u.q_unix.path; + + vbasedev->name =g_strdup_printf ("vfio-user:%s" , sock_name); + +/* +* vfio-user devices are effectively mdevs (don't use a host iommu). +*/ + vbasedev->mdev = true; + + as =pci_device_iommu_address_space (pdev); +if (!vfio_device_attach_by_iommu_type (TYPE_VFIO_IOMMU_USER , + vbasedev->name, vbasedev, + as, errp)) { +error_prepend (errp,VFIO_MSG_PREFIX , vbasedev->name); +return ; + } +} + +static void vfio_user_instance_init (Object *obj) +{ +PCIDevice *pci_dev =PCI_DEVICE (obj); +VFIOPCIDevice *vdev =VFIO_PCI_BASE (obj); +VFIODevice *vbasedev = &vdev->vbasedev; + +device_add_bootindex_property (obj, &vdev->bootindex, +"bootindex" ,NULL , + &pci_dev->qdev); + vdev->host.domain = ~0U ; + vdev->host.bus = ~0U ; + vdev->host.slot = ~0U ; + vdev->host.function = ~0U ; + +vfio_device_init (vbasedev,VFIO_DEVICE_TYPE_PCI , &vfio_user_pci_ops, +DEVICE (vdev), false); + + vdev->nv_gpudirect_clique =0xFF ; + +/* +* QEMU_PCI_CAP_EXPRESS initialization does not depend on QEMU command +* line, therefore, no need to wait to realize like other devices. +*/ + pci_dev->cap_present |=QEMU_PCI_CAP_EXPRESS ; +} + +static void vfio_user_instance_finalize (Object *obj) +{ +VFIOPCIDevice *vdev =VFIO_PCI_BASE (obj); + +vfio_pci_put_device (vdev); +} + +static const Property vfio_user_pci_dev_properties[] = { +DEFINE_PROP_UINT32 ("x-pci-vendor-id" , VFIOPCIDevice, + vendor_id,PCI_ANY_ID ), +DEFINE_PROP_UINT32 ("x-pci-device-id" , VFIOPCIDevice, + device_id,PCI_ANY_ID ), +DEFINE_PROP_UINT32 ("x-pci-sub-vendor-id" , VFIOPCIDevice, + sub_vendor_id,PCI_ANY_ID ), +DEFINE_PROP_UINT32 ("x-pci-sub-device-id" , VFIOPCIDevice, + sub_device_id,PCI_ANY_ID ), +}; + +static void vfio_user_pci_set_socket (Object *obj,Visitor *v,const char *name, +void *opaque,Error **errp) +{ +VFIOUserPCIDevice *udev =VFIO_USER_PCI (obj); +bool success; + +qapi_free_SocketAddress (udev->socket); + + udev->socket =NULL ; + + success =visit_type_SocketAddress (v, name, &udev->socket, errp); + +if (!success) { +return ; + } + +if (udev->socket->type !=SOCKET_ADDRESS_TYPE_UNIX ) { +error_setg (errp,"Unsupported socket type %s" , +SocketAddressType_str (udev->socket->type)); +qapi_free_SocketAddress (udev->socket); + udev->socket =NULL ; +return ; + } +} + +static void vfio_user_pci_dev_class_init (ObjectClass *klass,const void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *pdc =PCI_DEVICE_CLASS (klass); + +device_class_set_props (dc, vfio_user_pci_dev_properties); + +object_class_property_add (klass,"socket" ,"SocketAddress" ,NULL , + vfio_user_pci_set_socket,NULL ,NULL ); +object_class_property_set_description (klass,"socket" , +"SocketAddress (UNIX sockets only)" ); + + dc->desc ="VFIO over socket PCI device assignment" ; + pdc->realize = vfio_user_pci_realize; +} + +static const TypeInfo vfio_user_pci_dev_info = { + .name =TYPE_VFIO_USER_PCI , + .parent =TYPE_VFIO_PCI_BASE , + .instance_size =sizeof (VFIOUserPCIDevice), + .class_init = vfio_user_pci_dev_class_init, + .instance_init = vfio_user_instance_init, + .instance_finalize = vfio_user_instance_finalize, +}; + +static void register_vfio_user_dev_type (void ) +{ +type_register_static (&vfio_user_pci_dev_info); +} + +type_init (register_vfio_user_dev_type)hw/Kconfig +1
-0@@ -42,6 +42,7 @@ source ufs/Kconfig source usb/Kconfig source virtio/Kconfig source vfio/Kconfig +source vfio-user/Kconfig source vmapple/Kconfig source xen/Kconfig source watchdog/Kconfighw/meson.build +1
-0@@ -39,6 +39,7 @@ subdir('uefi') subdir('ufs') subdir('usb') subdir('vfio') +subdir('vfio-user') subdir('virtio') subdir('vmapple') subdir('watchdog')hw/vfio-user/Kconfig new file +7
-0@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-or-later + +config VFIO_USER + bool + default y + depends on VFIO_PCI +hw/vfio-user/meson.build new file +9
-0@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0-or-later + +vfio_user_ss = ss.source_set() +vfio_user_ss.add(files( + 'container.c', + 'pci.c', +)) + +system_ss.add_all(when: 'CONFIG_VFIO_USER', if_true: vfio_user_ss)
PATCH v5 2/19 vfio-user: connect vfio proxy to remote server
In reply to John LevonIntroduce the vfio-user "proxy": this is the client code responsible for sending and receiving vfio-user messages across the control socket.
The new files hw/vfio-user/proxy.[ch] contain some basic plumbing for managing the proxy; initialize the proxy during realization of the VFIOUserPCIDevice instance.
--- hw/vfio-user/proxy.h | 79 +++++++++++++++++ include/hw/vfio/vfio-device.h | 2 + hw/vfio-user/pci.c | 22 +++++ hw/vfio-user/proxy.c | 162 ++++++++++++++++++++++++++++++++++ hw/vfio-user/meson.build | 1 + 5 files changed, 266 insertions(+) create mode 100644 hw/vfio-user/proxy.h create mode 100644 hw/vfio-user/proxy.c
hw/vfio-user/proxy.h new file +79 -0
@@ -0,0 +1,79 @@ +#ifndef VFIO_USER_PROXY_H +#define VFIO_USER_PROXY_H + +/* +* vfio protocol over a UNIX socket. +* +* Copyright © 2018, 2021 Oracle and/or its affiliates. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "io/channel.h" +#include "io/channel-socket.h" + +typedef struct { +int send_fds; +int recv_fds; +int *fds; +}VFIOUserFDs ; + +enum msg_type { +VFIO_MSG_NONE , +VFIO_MSG_ASYNC , +VFIO_MSG_WAIT , +VFIO_MSG_NOWAIT , +VFIO_MSG_REQ , +}; + +typedef struct VFIOUserMsg { +QTAILQ_ENTRY (VFIOUserMsg) next; +VFIOUserFDs *fds; +uint32_t rsize; +uint32_t id; +QemuCond cv; +bool complete; +enum msg_type type; +}VFIOUserMsg ; + + +enum proxy_state { +VFIO_PROXY_CONNECTED =1 , +VFIO_PROXY_ERROR =2 , +VFIO_PROXY_CLOSING =3 , +VFIO_PROXY_CLOSED =4 , +}; + +typedef QTAILQ_HEAD(VFIOUserMsgQ, VFIOUserMsg) VFIOUserMsgQ; + +typedef struct VFIOUserProxy { +QLIST_ENTRY (VFIOUserProxy) next; +char *sockname; +struct QIOChannel *ioc; +void (*request)(void *opaque,VFIOUserMsg *msg); +void *req_arg; +int flags; +QemuCond close_cv; +AioContext *ctx; +QEMUBH *req_bh; + +/* +* above only changed when BQL is held +* below are protected by per-proxy lock +*/ +QemuMutex lock; +VFIOUserMsgQ free; +VFIOUserMsgQ pending; +VFIOUserMsgQ incoming; +VFIOUserMsgQ outgoing; +VFIOUserMsg *last_nowait; +enum proxy_state state; +}VFIOUserProxy ; + +/* VFIOProxy flags */ +#define VFIO_PROXY_CLIENT 0x1 + +VFIOUserProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp); +void vfio_user_disconnect (VFIOUserProxy *proxy); + +#endif /* VFIO_USER_PROXY_H */
include/hw/vfio/vfio-device.h +2 -0
@@ -47,6 +47,7 @@ typedef struct VFIOMigration VFIOMigration;typedef struct IOMMUFDBackend IOMMUFDBackend ;typedef struct VFIOIOASHwpt VFIOIOASHwpt ; +typedef struct VFIOUserProxy VFIOUserProxy ; typedef struct VFIODevice { QLIST_ENTRY(VFIODevice) next;
@@ -88,6 +89,7 @@ typedef struct VFIODevice {struct vfio_region_info **reginfo;int *region_fds;VFIODeviceCPR cpr; +VFIOUserProxy *proxy; } VFIODevice;struct VFIODeviceOps {
hw/vfio-user/pci.c +22 -0
@@ -12,6 +12,7 @@#include "hw/qdev-properties.h" #include "hw/vfio/pci.h" +#include "hw/vfio-user/proxy.h" #define TYPE_VFIO_USER_PCI "vfio-user-pci"OBJECT_DECLARE_SIMPLE_TYPE (VFIOUserPCIDevice,VFIO_USER_PCI )
@@ -54,6 +55,8 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)VFIODevice *vbasedev = &vdev->vbasedev;const char *sock_name;AddressSpace *as; +SocketAddress addr; +VFIOUserProxy *proxy;if (!udev->socket) {error_setg (errp,"No socket specified" );
@@ -69,6 +72,15 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->name =g_strdup_printf ("vfio-user:%s" , sock_name); +memset (&addr,0 ,sizeof (addr)); + addr.type =SOCKET_ADDRESS_TYPE_UNIX ; + addr.u.q_unix.path = (char *)sock_name; + proxy =vfio_user_connect_dev (&addr, errp); +if (!proxy) { +return ; + } + vbasedev->proxy = proxy; +/* * vfio-user devices are effectively mdevs (don't use a host iommu). */
@@ -112,8 +124,13 @@ static void vfio_user_instance_init(Object *obj)static void vfio_user_instance_finalize (Object *obj) {VFIOPCIDevice *vdev =VFIO_PCI_BASE (obj); +VFIODevice *vbasedev = &vdev->vbasedev;vfio_pci_put_device (vdev); + +if (vbasedev->proxy !=NULL ) { +vfio_user_disconnect (vbasedev->proxy); + } }static const Property vfio_user_pci_dev_properties[] = {
@@ -133,6 +150,11 @@ static void vfio_user_pci_set_socket(Object *obj, Visitor *v, const char *name,VFIOUserPCIDevice *udev =VFIO_USER_PCI (obj);bool success; +if (udev->device.vbasedev.proxy) { +error_setg (errp,"Proxy is connected" ); +return ; + } +qapi_free_SocketAddress (udev->socket); udev->socket =NULL ;
hw/vfio-user/proxy.c new file +162 -0
162 lines not on this page · raw message
hw/vfio-user/meson.build +1 -0
@@ -4,6 +4,7 @@ vfio_user_ss = ss.source_set() vfio_user_ss.add(files( 'container.c', 'pci.c', + 'proxy.c', )) system_ss.add_all(when: 'CONFIG_VFIO_USER', if_true: vfio_user_ss)
Signature · 1 line
2.43.0
Re: [PATCH v5 02/19] vfio-user: connect vfio proxy to remote server
Quoted 18 lines · John Levon
Introduce the vfio-user "proxy": this is the client code responsible for sending and receiving vfio-user messages across the control socket.
The new files hw/vfio-user/proxy.[ch] contain some basic plumbing for managing the proxy; initialize the proxy during realization of the VFIOUserPCIDevice instance.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>--- hw/vfio-user/proxy.h | 79 +++++++++++++++++ include/hw/vfio/vfio-device.h | 2 + hw/vfio-user/pci.c | 22 +++++ hw/vfio-user/proxy.c | 162 ++++++++++++++++++++++++++++++++++ hw/vfio-user/meson.build | 1 + 5 files changed, 266 insertions(+)
Quoted 3 lines
create mode 100644
hw/vfio-user/proxy.h
create mode 100644hw/vfio-user/proxy.c
Thanks,
C.
PATCH v5 3/19 vfio-user: implement message receive infrastructure
In reply to John LevonAdd the basic implementation for receiving vfio-user messages from the control socket.
--- meson.build | 1 + hw/vfio-user/protocol.h | 53 +++++ hw/vfio-user/proxy.h | 11 + hw/vfio-user/trace.h | 4 + hw/vfio-user/pci.c | 11 + hw/vfio-user/proxy.c | 411 ++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 8 + 7 files changed, 499 insertions(+) create mode 100644 hw/vfio-user/protocol.h create mode 100644 hw/vfio-user/trace.h create mode 100644 hw/vfio-user/trace-events
meson.build +1 -0
@@ -3683,6 +3683,7 @@ if have_system 'hw/ufs', 'hw/usb', 'hw/vfio', + 'hw/vfio-user', 'hw/virtio', 'hw/vmapple', 'hw/watchdog',
hw/vfio-user/protocol.h new file +53 -0
53 lines not on this page · raw message
hw/vfio-user/proxy.h +11 -0
@@ -12,6 +12,9 @@#include "io/channel.h" #include "io/channel-socket.h" +#include "qemu/sockets.h" +#include "hw/vfio-user/protocol.h" + typedef struct { int send_fds; int recv_fds;
@@ -28,6 +31,7 @@ enum msg_type { typedef struct VFIOUserMsg { QTAILQ_ENTRY(VFIOUserMsg) next; + VFIOUserHdr *hdr; VFIOUserFDs *fds; uint32_t rsize; uint32_t id;
@@ -67,13 +71,20 @@ typedef struct VFIOUserProxy {VFIOUserMsgQ incoming;VFIOUserMsgQ outgoing;VFIOUserMsg *last_nowait; +VFIOUserMsg *part_recv; +size_t recv_left; enum proxy_state state; } VFIOUserProxy;/* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 +typedef struct VFIODevice VFIODevice ; +VFIOUserProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp);void vfio_user_disconnect (VFIOUserProxy *proxy); +void vfio_user_set_handler (VFIODevice *vbasedev, +void (*handler)(void *opaque,VFIOUserMsg *msg), +void *reqarg);#endif /* VFIO_USER_PROXY_H */
hw/vfio-user/trace.h new file +4 -0
@@ -0,0 +1,4 @@ +/* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ +#include "trace/trace-hw_vfio_user.h"
hw/vfio-user/pci.c +11 -0
@@ -22,6 +22,16 @@ struct VFIOUserPCIDevice {SocketAddress *socket; }; +/* +* Incoming request message callback. +* +* Runs off main loop, so BQL held. +*/ +static void vfio_user_pci_process_req (void *opaque,VFIOUserMsg *msg) +{ + +} +/* * Emulated devices don't use host hot reset */
@@ -80,6 +90,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)return ; } vbasedev->proxy = proxy; +vfio_user_set_handler (vbasedev, vfio_user_pci_process_req, vdev);/* * vfio-user devices are effectively mdevs (don't use a host iommu).
hw/vfio-user/proxy.c +411 -0
460 lines not on this page · raw message
hw/vfio-user/trace-events new file +8 -0
@@ -0,0 +1,8 @@ +# See docs/devel/tracing.rst for syntax documentation. +# +# SPDX-License-Identifier: GPL-2.0-or-later + +# common.c +vfio_user_recv_hdr(const char *name, uint16_t id, uint16_t cmd, uint32_t size, uint32_t flags) " (%s) id 0x%x cmd 0x%x size 0x%x flags 0x%x" +vfio_user_recv_read(uint16_t id, int read) " id 0x%x read 0x%x" +vfio_user_recv_request(uint16_t cmd) " command 0x%x"
Signature · 1 line
2.43.0
Re: [PATCH v5 03/19] vfio-user: implement message receive infrastructure
Quoted 7 lines · John Levon
Add the basic implementation for receiving vfio-user messages from the control socket.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Thanks,
C.
PATCH v5 4/19 vfio-user: implement message send infrastructure
In reply to John LevonAdd plumbing for sending vfio-user messages on the control socket. Add initial version negotation on connection.
--- hw/vfio-user/protocol.h | 62 +++++ hw/vfio-user/proxy.h | 9 + hw/vfio-user/pci.c | 20 +- hw/vfio-user/proxy.c | 515 ++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 2 + 5 files changed, 606 insertions(+), 2 deletions(-)
hw/vfio-user/protocol.h +62 -0
@@ -50,4 +50,66 @@ enum vfio_user_command {#define VFIO_USER_NO_REPLY 0x10#define VFIO_USER_ERROR 0x20 + +/* +* VFIO_USER_VERSION +*/ +typedef struct { +VFIOUserHdr hdr; +uint16_t major; +uint16_t minor; +char capabilities[]; +}VFIOUserVersion ; + +#define VFIO_USER_MAJOR_VER 0 +#define VFIO_USER_MINOR_VER 0 + +#define VFIO_USER_CAP "capabilities" + +/* "capabilities" members */ +#define VFIO_USER_CAP_MAX_FDS "max_msg_fds" +#define VFIO_USER_CAP_MAX_XFER "max_data_xfer_size" +#define VFIO_USER_CAP_PGSIZES "pgsizes" +#define VFIO_USER_CAP_MAP_MAX "max_dma_maps" +#define VFIO_USER_CAP_MIGR "migration" + +/* "migration" members */ +#define VFIO_USER_CAP_PGSIZE "pgsize" +#define VFIO_USER_CAP_MAX_BITMAP "max_bitmap_size" + +/* +* Max FDs mainly comes into play when a device supports multiple interrupts +* where each ones uses an eventfd to inject it into the guest. +* It is clamped by the the number of FDs the qio channel supports in a +* single message. +*/ +#define VFIO_USER_DEF_MAX_FDS 8 +#define VFIO_USER_MAX_MAX_FDS 16 + +/* +* Max transfer limits the amount of data in region and DMA messages. +* Region R/W will be very small (limited by how much a single instruction +* can process) so just use a reasonable limit here. +*/ +#define VFIO_USER_DEF_MAX_XFER (1024 * 1024) +#define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) + +/* +* Default pagesizes supported is 4k. +*/ +#define VFIO_USER_DEF_PGSIZE 4096 + +/* +* Default max number of DMA mappings is stolen from the +* linux kernel "dma_entry_limit" +*/ +#define VFIO_USER_DEF_MAP_MAX 65535 + +/* +* Default max bitmap size is also take from the linux kernel, +* where usage of signed ints limits the VA range to 2^31 bytes. +* Dividing that by the number of bits per byte yields 256MB +*/ +#define VFIO_USER_DEF_MAX_BITMAP (256 * 1024 * 1024) +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio-user/proxy.h +9 -0
@@ -37,6 +37,7 @@ typedef struct VFIOUserMsg {uint32_t id;QemuCond cv;bool complete; +bool pending;enum msg_type type; } VFIOUserMsg;
@@ -56,6 +57,12 @@ typedef struct VFIOUserProxy {struct QIOChannel *ioc;void (*request)(void *opaque,VFIOUserMsg *msg);void *req_arg; +uint64_t max_xfer_size; +uint64_t max_send_fds; +uint64_t max_dma; +uint64_t dma_pgsizes; +uint64_t max_bitmap; +uint64_t migr_pgsize;int flags;QemuCond close_cv;AioContext *ctx;
@@ -78,6 +85,7 @@ typedef struct VFIOUserProxy {/* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 +#define VFIO_PROXY_FORCE_QUEUED 0x4typedef struct VFIODevice VFIODevice ;
@@ -86,5 +94,6 @@ void vfio_user_disconnect(VFIOUserProxy *proxy);void vfio_user_set_handler (VFIODevice *vbasedev,void (*handler)(void *opaque,VFIOUserMsg *msg),void *reqarg); +bool vfio_user_validate_version (VFIOUserProxy *proxy,Error **errp);#endif /* VFIO_USER_PROXY_H */
hw/vfio-user/pci.c +18 -2
@@ -20,6 +20,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI)struct VFIOUserPCIDevice {VFIOPCIDevice device;SocketAddress *socket; +bool send_queued;/* all sends are queued */ };/*
@@ -92,6 +93,16 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->proxy = proxy;vfio_user_set_handler (vbasedev, vfio_user_pci_process_req, vdev); + vbasedev->name =g_strdup_printf ("vfio-user:%s" , sock_name); + +if (udev->send_queued) { + proxy->flags |=VFIO_PROXY_FORCE_QUEUED ; + } + +if (!vfio_user_validate_version (proxy, errp)) { + goto error; + } +/* * vfio-user devices are effectively mdevs (don't use a host iommu). */
@@ -101,9 +112,13 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)if (!vfio_device_attach_by_iommu_type (TYPE_VFIO_IOMMU_USER , vbasedev->name, vbasedev, as, errp)) { -error_prepend (errp,VFIO_MSG_PREFIX , vbasedev->name); -return ; + goto error; } + +return ; + +error: +error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name); }static void vfio_user_instance_init (Object *obj)
@@ -153,6 +168,7 @@ static const Property vfio_user_pci_dev_properties[] = { sub_vendor_id, PCI_ANY_ID), DEFINE_PROP_UINT32("x-pci-sub-device-id", VFIOPCIDevice, sub_device_id, PCI_ANY_ID), + DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), }; static void vfio_user_pci_set_socket(Object *obj, Visitor *v, const char *name,
hw/vfio-user/proxy.c +515 -0
568 lines not on this page · raw message
hw/vfio-user/trace-events +2 -0
@@ -6,3 +6,5 @@ vfio_user_recv_hdr(const char *name, uint16_t id, uint16_t cmd, uint32_t size, uint32_t flags) " (%s) id 0x%x cmd 0x%x size 0x%x flags 0x%x" vfio_user_recv_read(uint16_t id, int read) " id 0x%x read 0x%x" vfio_user_recv_request(uint16_t cmd) " command 0x%x" +vfio_user_send_write(uint16_t id, int wrote) " id 0x%x wrote 0x%x" +vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d minor %d caps: %s"
Signature · 1 line
2.43.0
Re: [PATCH v5 04/19] vfio-user: implement message send infrastructure
Quoted 7 lines · John Levon
Add plumbing for sending vfio-user messages on the control socket. Add initial version negotation on connection.
Originally-by John Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John Levon <[email protected]>
Quoted 8 lines
--- hw/vfio-user/protocol.h | 62 +++++ hw/vfio-user/proxy.h | 9 + hw/vfio-user/pci.c | 20 +- hw/vfio-user/proxy.c | 515 ++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 2 + 5 files changed, 606 insertions(+), 2 deletions(-)
Thanks,
C.
PATCH v5 5/19 vfio-user: implement VFIO_USER_DEVICE_GET_INFO
In reply to John LevonAdd support for getting basic device information.
--- hw/vfio-user/device.h | 20 ++++++++++++++ hw/vfio-user/protocol.h | 12 +++++++++ hw/vfio-user/proxy.h | 7 +++++ hw/vfio-user/container.c | 8 +++++- hw/vfio-user/device.c | 55 +++++++++++++++++++++++++++++++++++++++ hw/vfio-user/proxy.c | 10 +++---- hw/vfio-user/meson.build | 1 + hw/vfio-user/trace-events | 1 + 8 files changed, 107 insertions(+), 7 deletions(-) create mode 100644 hw/vfio-user/device.h create mode 100644 hw/vfio-user/device.c
hw/vfio-user/device.h new file +20 -0
@@ -0,0 +1,20 @@ +#ifndef VFIO_USER_DEVICE_H +#define VFIO_USER_DEVICE_H + +/* +* vfio protocol over a UNIX socket device handling. +* +* Copyright © 2018, 2021 Oracle and/or its affiliates. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "linux/vfio.h" + +#include "hw/vfio-user/proxy.h" + +bool vfio_user_get_device_info (VFIOUserProxy *proxy, +struct vfio_device_info *info,Error **errp); + +#endif /* VFIO_USER_DEVICE_H */
hw/vfio-user/protocol.h +12 -0
@@ -112,4 +112,16 @@ typedef struct {*/ #define VFIO_USER_DEF_MAX_BITMAP (256 * 1024 * 1024) +/* +* VFIO_USER_DEVICE_GET_INFO +* imported from struct vfio_device_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t num_regions; +uint32_t num_irqs; +}VFIOUserDeviceInfo ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio-user/proxy.h +7 -0
@@ -12,7 +12,9 @@#include "io/channel.h" #include "io/channel-socket.h" +#include "qemu/queue.h" #include "qemu/sockets.h" +#include "qemu/thread.h" #include "hw/vfio-user/protocol.h" typedef struct {
@@ -96,4 +98,9 @@ void vfio_user_set_handler(VFIODevice *vbasedev, void *reqarg);bool vfio_user_validate_version (VFIOUserProxy *proxy,Error **errp); +void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd, +uint32_t size,uint32_t flags); +bool vfio_user_send_wait (VFIOUserProxy *proxy,VFIOUserHdr *hdr, +VFIOUserFDs *fds,int rsize,Error **errp); +#endif /* VFIO_USER_PROXY_H */
hw/vfio-user/container.c +7 -1
@@ -11,6 +11,7 @@#include "qemu/osdep.h" #include "hw/vfio-user/container.h" +#include "hw/vfio-user/device.h" #include "hw/vfio/vfio-cpr.h" #include "hw/vfio/vfio-device.h" #include "hw/vfio/vfio-listener.h"
@@ -140,7 +141,12 @@ static void vfio_user_container_disconnect(VFIOUserContainer *container)static bool vfio_user_device_get (VFIOUserContainer *container,VFIODevice *vbasedev,Error **errp) { -struct vfio_device_info info = {0 }; +struct vfio_device_info info = { .argsz =sizeof (info) }; + + +if (!vfio_user_get_device_info (vbasedev->proxy, &info, errp)) { +return false; + } vbasedev->fd =-1 ;
hw/vfio-user/device.c new file +55 -0
55 lines not on this page · raw message
hw/vfio-user/proxy.c +4 -6
@@ -35,8 +35,6 @@ static void vfio_user_send(void *opaque);static void vfio_user_cb (void *opaque);static void vfio_user_request (void *opaque); -static void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd, -uint32_t size,uint32_t flags);static inline void vfio_user_set_error (VFIOUserHdr *hdr,uint32_t err) {
@@ -626,8 +624,8 @@ static bool vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg,* * In either case, the caller must free @hdr and @fds if needed. */ -static bool vfio_user_send_wait (VFIOUserProxy *proxy,VFIOUserHdr *hdr, -VFIOUserFDs *fds,int rsize,Error **errp) +bool vfio_user_send_wait (VFIOUserProxy *proxy,VFIOUserHdr *hdr, +VFIOUserFDs *fds,int rsize,Error **errp) {VFIOUserMsg *msg;bool ok = false;
@@ -802,8 +800,8 @@ void vfio_user_disconnect(VFIOUserProxy *proxy)g_free (proxy); } -static void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd, -uint32_t size,uint32_t flags) +void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd, +uint32_t size,uint32_t flags) {static uint16_t next_id;
hw/vfio-user/meson.build +1 -0
@@ -3,6 +3,7 @@ vfio_user_ss = ss.source_set() vfio_user_ss.add(files( 'container.c', + 'device.c', 'pci.c', 'proxy.c', ))
hw/vfio-user/trace-events +1 -0
@@ -8,3 +8,4 @@ vfio_user_recv_read(uint16_t id, int read) " id 0x%x read 0x%x" vfio_user_recv_request(uint16_t cmd) " command 0x%x" vfio_user_send_write(uint16_t id, int wrote) " id 0x%x wrote 0x%x" vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d minor %d caps: %s" +vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d"
Signature · 1 line
2.43.0
Re: [PATCH v5 05/19] vfio-user: implement VFIO_USER_DEVICE_GET_INFO
Quoted 16 lines · John Levon
Add support for getting basic device information.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>--- hw/vfio-user/device.h | 20 ++++++++++++++ hw/vfio-user/protocol.h | 12 +++++++++ hw/vfio-user/proxy.h | 7 +++++ hw/vfio-user/container.c | 8 +++++- hw/vfio-user/device.c | 55 +++++++++++++++++++++++++++++++++++++++ hw/vfio-user/proxy.c | 10 +++---- hw/vfio-user/meson.build | 1 + hw/vfio-user/trace-events | 1 + 8 files changed, 107 insertions(+), 7 deletions(-)
Quoted 3 lines
create mode 100644
hw/vfio-user/device.h
create mode 100644hw/vfio-user/device.c
Thanks,
C.
PATCH v5 6/19 vfio-user: implement VFIO_USER_DEVICE_GET_REGION_INFO
In reply to John LevonAdd support for getting region info for vfio-user. As vfio-user has one fd per region, enable ->use_region_fds.
--- hw/vfio-user/device.h | 2 ++ hw/vfio-user/protocol.h | 14 ++++++++ hw/vfio-user/proxy.h | 1 + hw/vfio-user/device.c | 74 +++++++++++++++++++++++++++++++++++++++ hw/vfio-user/pci.c | 11 ++++++ hw/vfio-user/trace-events | 1 + 6 files changed, 103 insertions(+)
hw/vfio-user/device.h +2 -0
@@ -17,4 +17,6 @@bool vfio_user_get_device_info (VFIOUserProxy *proxy,struct vfio_device_info *info,Error **errp); +extern VFIODeviceIOOps vfio_user_device_io_ops_sock; +#endif /* VFIO_USER_DEVICE_H */
hw/vfio-user/protocol.h +14 -0
@@ -124,4 +124,18 @@ typedef struct { uint32_t num_irqs; } VFIOUserDeviceInfo; +/* +* VFIO_USER_DEVICE_GET_REGION_INFO +* imported from struct vfio_region_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t cap_offset; +uint64_t size; +uint64_t offset; +}VFIOUserRegionInfo ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio-user/proxy.h +1 -0
@@ -15,6 +15,7 @@#include "qemu/queue.h" #include "qemu/sockets.h" #include "qemu/thread.h" +#include "hw/vfio/vfio-device.h" #include "hw/vfio-user/protocol.h" typedef struct {
hw/vfio-user/device.c +74 -0
77 lines not on this page · raw message
hw/vfio-user/pci.c +11 -0
@@ -12,6 +12,7 @@#include "hw/qdev-properties.h" #include "hw/vfio/pci.h" +#include "hw/vfio-user/device.h" #include "hw/vfio-user/proxy.h" #define TYPE_VFIO_USER_PCI "vfio-user-pci"
@@ -103,11 +104,21 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto error; } +/* +* Use socket-based device I/O instead of vfio kernel driver. +*/ + vbasedev->io_ops = &vfio_user_device_io_ops_sock; +/* * vfio-user devices are effectively mdevs (don't use a host iommu). */ vbasedev->mdev = true; +/* +* Enable per-region fds. +*/ + vbasedev->use_region_fds = true; + as =pci_device_iommu_address_space (pdev); if (!vfio_device_attach_by_iommu_type(TYPE_VFIO_IOMMU_USER, vbasedev->name, vbasedev,
hw/vfio-user/trace-events +1 -0
@@ -9,3 +9,4 @@ vfio_user_recv_request(uint16_t cmd) " command 0x%x" vfio_user_send_write(uint16_t id, int wrote) " id 0x%x wrote 0x%x" vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d minor %d caps: %s" vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d" +vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index %d flags 0x%x size 0x%"PRIx64
Signature · 1 line
2.43.0
Re: [PATCH v5 06/19] vfio-user: implement VFIO_USER_DEVICE_GET_REGION_INFO
Quoted 7 lines · John Levon
Add support for getting region info for vfio-user. As vfio-user has one fd per region, enable ->use_region_fds.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Quoted 8 lines
--- hw/vfio-user/device.h | 2 ++ hw/vfio-user/protocol.h | 14 ++++++++ hw/vfio-user/proxy.h | 1 + hw/vfio-user/device.c | 74 +++++++++++++++++++++++++++++++++++++++ hw/vfio-user/pci.c | 11 ++++++ hw/vfio-user/trace-events | 1 + 6 files changed, 103 insertions(+)
Thanks,
C.
PATCH v5 7/19 vfio-user: implement VFIO_USER_REGION_READ/WRITE
In reply to John Levon--- hw/vfio-user/protocol.h | 12 ++++++ hw/vfio-user/device.c | 79 +++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 1 + 3 files changed, 92 insertions(+)
hw/vfio-user/protocol.h +12 -0
@@ -138,4 +138,16 @@ typedef struct { uint64_t offset; } VFIOUserRegionInfo; +/* +* VFIO_USER_REGION_READ +* VFIO_USER_REGION_WRITE +*/ +typedef struct { +VFIOUserHdr hdr; +uint64_t offset; +uint32_t region; +uint32_t count; +char data[]; +}VFIOUserRegionRW ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio-user/device.c +79 -0
88 lines not on this page · raw message
hw/vfio-user/trace-events +1 -0
@@ -10,3 +10,4 @@ vfio_user_send_write(uint16_t id, int wrote) " id 0x%x wrote 0x%x" vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d minor %d caps: %s" vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d" vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index %d flags 0x%x size 0x%"PRIx64 +vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d"
Signature · 1 line
2.43.0
Re: [PATCH v5 07/19] vfio-user: implement VFIO_USER_REGION_READ/WRITE
Quoted 10 lines · John Levon
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>--- hw/vfio-user/protocol.h | 12 ++++++ hw/vfio-user/device.c | 79 +++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 1 + 3 files changed, 92 insertions(+)
Thanks,
C.
PATCH v5 8/19 vfio-user: set up PCI in vfio_user_pci_realize()
In reply to John LevonRe-use PCI setup functions from hw/vfio/pci.c to realize the vfio-user PCI device.
--- hw/vfio-user/pci.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
hw/vfio-user/pci.c +29 -0
@@ -126,10 +126,39 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto error; } +if (!vfio_pci_populate_device (vdev, errp)) { + goto error; + } + +if (!vfio_pci_config_setup (vdev, errp)) { + goto error; + } + +/* +* vfio_pci_config_setup will have registered the device's BARs +* and setup any MSIX BARs, so errors after it succeeds must +* use out_teardown +*/ + +if (!vfio_pci_add_capabilities (vdev, errp)) { + goto out_teardown; + } + +if (!vfio_pci_interrupt_setup (vdev, errp)) { + goto out_teardown; + } + +vfio_pci_register_err_notifier (vdev); +vfio_pci_register_req_notifier (vdev); +return ; +out_teardown: +vfio_pci_teardown_msi (vdev); +vfio_pci_bars_exit (vdev); error:error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name); +vfio_pci_put_device (vdev); }static void vfio_user_instance_init (Object *obj)
Signature · 1 line
2.43.0
Re: [PATCH v5 08/19] vfio-user: set up PCI in vfio_user_pci_realize()
Quoted 7 lines · John Levon
Re-use PCI setup functions from
hw/vfio/pci.cto realize the vfio-user PCI device.Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Thanks,
C.
PATCH v5 9/19 vfio-user: implement VFIO_USER_DEVICE_GET/SET_IRQ*
In reply to John LevonIRQ setup uses the same semantics as the traditional vfio path, but we need to share the corresponding file descriptors with the server as necessary.
--- hw/vfio-user/protocol.h | 25 +++++++ hw/vfio-user/device.c | 138 ++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 2 + 3 files changed, 165 insertions(+)
hw/vfio-user/protocol.h +25 -0
@@ -138,6 +138,31 @@ typedef struct {uint64_t offset; } VFIOUserRegionInfo; +/* +* VFIO_USER_DEVICE_GET_IRQ_INFO +* imported from struct vfio_irq_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t count; +}VFIOUserIRQInfo ; + +/* +* VFIO_USER_DEVICE_SET_IRQS +* imported from struct vfio_irq_set +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t start; +uint32_t count; +}VFIOUserIRQSet ; +/* * VFIO_USER_REGION_READ * VFIO_USER_REGION_WRITE
hw/vfio-user/device.c +138 -0
150 lines not on this page · raw message
hw/vfio-user/trace-events +2 -0
@@ -11,3 +11,5 @@ vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d m vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d" vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index %d flags 0x%x size 0x%"PRIx64 vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d" +vfio_user_get_irq_info(uint32_t index, uint32_t flags, uint32_t count) " index %d flags 0x%x count %d" +vfio_user_set_irqs(uint32_t index, uint32_t start, uint32_t count, uint32_t flags) " index %d start %d count %d flags 0x%x"
Signature · 1 line
2.43.0
Re: [PATCH v5 09/19] vfio-user: implement VFIO_USER_DEVICE_GET/SET_IRQ*
Quoted 8 lines · John Levon
IRQ setup uses the same semantics as the traditional vfio path, but we need to share the corresponding file descriptors with the server as necessary.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Quoted 6 lines
--- hw/vfio-user/protocol.h | 25 +++++++ hw/vfio-user/device.c | 138 ++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 2 + 3 files changed, 165 insertions(+)
Thanks,
C.
PATCH v5 10/19 vfio-user: forward MSI-X PBA BAR accesses to server
In reply to John LevonFor vfio-user, the server holds the pending IRQ state; set up an I/O region for the MSI-X PBA so we can ask the server for this state on a PBA read.
--- hw/vfio/pci.h | 1 + hw/vfio-user/pci.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+)
hw/vfio/pci.h +1 -0
@@ -116,6 +116,7 @@ typedef struct VFIOMSIXInfo {uint32_t pba_offset;unsigned long *pending;bool noresize; +MemoryRegion *pba_region; } VFIOMSIXInfo;/*
hw/vfio-user/pci.c +64 -0
82 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v5 10/19] vfio-user: forward MSI-X PBA BAR accesses to server
Quoted 4 lines · John Levon
For vfio-user, the server holds the pending IRQ state; set up an I/O region for the MSI-X PBA so we can ask the server for this state on a PBA read.
Quoted 9 lines
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>--- hw/vfio/pci.h | 1 + hw/vfio-user/pci.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+)
Thanks,
C.
PATCH v5 11/19 vfio-user: set up container access to the proxy
In reply to John LevonThe user container will shortly need access to the underlying vfio-user proxy; set this up.
--- hw/vfio-user/container.h | 2 ++ hw/vfio-user/container.c | 43 +++++++++++++++++++++++++++++++--------- 2 files changed, 36 insertions(+), 9 deletions(-)
hw/vfio-user/container.h +2 -0
@@ -10,10 +10,12 @@#include "qemu/osdep.h" #include "hw/vfio/vfio-container-base.h" +#include "hw/vfio-user/proxy.h" /* MMU container sub-class for vfio-user. */ typedef struct VFIOUserContainer {VFIOContainerBase bcontainer; +VFIOUserProxy *proxy; }VFIOUserContainer ;OBJECT_DECLARE_SIMPLE_TYPE (VFIOUserContainer,VFIO_IOMMU_USER );
hw/vfio-user/container.c +34 -9
103 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v5 11/19] vfio-user: set up container access to the proxy
Quoted 11 lines · John Levon
The user container will shortly need access to the underlying vfio-user proxy; set this up.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>--- hw/vfio-user/container.h | 2 ++ hw/vfio-user/container.c | 43 +++++++++++++++++++++++++++++++--------- 2 files changed, 36 insertions(+), 9 deletions(-)
Thanks,
C.
PATCH v5 12/19 vfio-user: implement VFIO_USER_DEVICE_RESET
In reply to John LevonHook this call up to the legacy reset handler for vfio-user-pci.
--- hw/vfio-user/device.h | 2 ++ hw/vfio-user/device.c | 18 ++++++++++++++++++ hw/vfio-user/pci.c | 15 +++++++++++++++ 3 files changed, 35 insertions(+)
hw/vfio-user/device.h +2 -0
@@ -17,6 +17,8 @@bool vfio_user_get_device_info (VFIOUserProxy *proxy,struct vfio_device_info *info,Error **errp); +void vfio_user_device_reset (VFIOUserProxy *proxy); +extern VFIODeviceIOOps vfio_user_device_io_ops_sock;#endif /* VFIO_USER_DEVICE_H */
hw/vfio-user/device.c +18 -0
@@ -54,6 +54,24 @@ bool vfio_user_get_device_info(VFIOUserProxy *proxy,return true; } +void vfio_user_device_reset (VFIOUserProxy *proxy) +{ +Error *local_err =NULL ; +VFIOUserHdr hdr; + +vfio_user_request_msg (&hdr,VFIO_USER_DEVICE_RESET ,sizeof (hdr),0 ); + +if (!vfio_user_send_wait (proxy, &hdr,NULL ,0 , &local_err)) { +error_prepend (&local_err,"%s: " , __func__); +error_report_err (local_err); +return ; + } + +if (hdr.flags &VFIO_USER_ERROR ) { +error_printf ("reset reply error %d\n" , hdr.error_reply); + } +} +static int vfio_user_get_region_info (VFIOUserProxy *proxy,struct vfio_region_info *info,VFIOUserFDs *fds)
hw/vfio-user/pci.c +15 -0
@@ -263,6 +263,20 @@ static void vfio_user_instance_finalize(Object *obj) } } +static void vfio_user_pci_reset (DeviceState *dev) +{ +VFIOPCIDevice *vdev =VFIO_PCI_BASE (dev); +VFIODevice *vbasedev = &vdev->vbasedev; + +vfio_pci_pre_reset (vdev); + +if (vbasedev->reset_works) { +vfio_user_device_reset (vbasedev->proxy); + } + +vfio_pci_post_reset (vdev); +} +static const Property vfio_user_pci_dev_properties[] = {DEFINE_PROP_UINT32 ("x-pci-vendor-id" , VFIOPCIDevice, vendor_id,PCI_ANY_ID ),
@@ -310,6 +324,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, const void *data)DeviceClass *dc =DEVICE_CLASS (klass);PCIDeviceClass *pdc =PCI_DEVICE_CLASS (klass); +device_class_set_legacy_reset (dc, vfio_user_pci_reset);device_class_set_props (dc, vfio_user_pci_dev_properties); object_class_property_add(klass, "socket", "SocketAddress", NULL,
Signature · 1 line
2.43.0
Re: [PATCH v5 12/19] vfio-user: implement VFIO_USER_DEVICE_RESET
Quoted 6 lines · John Levon
Hook this call up to the legacy reset handler for vfio-user-pci.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Quoted 6 lines
--- hw/vfio-user/device.h | 2 ++ hw/vfio-user/device.c | 18 ++++++++++++++++++ hw/vfio-user/pci.c | 15 +++++++++++++++ 3 files changed, 35 insertions(+)
Thanks,
C.
PATCH v5 13/19 vfio-user: implement VFIO_USER_DMA_MAP/UNMAP
In reply to John LevonFrom: John Levon <[email protected]>
When the vfio-user container gets mapping updates, share them with the vfio-user by sending a message; this can include the region fd, allowing the server to directly mmap() the region as needed.
For performance, we only wait for the message responses when we're doing with a series of updates via the listener_commit() callback.
--- hw/vfio-user/protocol.h | 32 +++++++++ hw/vfio-user/proxy.h | 6 ++ hw/vfio-user/container.c | 135 +++++++++++++++++++++++++++++++++++++- hw/vfio-user/proxy.c | 84 +++++++++++++++++++++++- hw/vfio-user/trace-events | 4 ++ 5 files changed, 257 insertions(+), 4 deletions(-)
hw/vfio-user/protocol.h +32 -0
@@ -112,6 +112,31 @@ typedef struct {*/ #define VFIO_USER_DEF_MAX_BITMAP (256 * 1024 * 1024) +/* +* VFIO_USER_DMA_MAP +* imported from struct vfio_iommu_type1_dma_map +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint64_t offset;/* FD offset */ +uint64_t iova; +uint64_t size; +}VFIOUserDMAMap ; + +/* +* VFIO_USER_DMA_UNMAP +* imported from struct vfio_iommu_type1_dma_unmap +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint64_t iova; +uint64_t size; +}VFIOUserDMAUnmap ; +/* * VFIO_USER_DEVICE_GET_INFO * imported from struct vfio_device_info
@@ -175,4 +200,11 @@ typedef struct { char data[]; } VFIOUserRegionRW; +/*imported from struct vfio_bitmap */ +typedef struct { +uint64_t pgsize; +uint64_t size; +char data[]; +}VFIOUserBitmap ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio-user/proxy.h +6 -0
@@ -70,6 +70,7 @@ typedef struct VFIOUserProxy {QemuCond close_cv;AioContext *ctx;QEMUBH *req_bh; +bool async_ops;/* * above only changed when BQL is held
@@ -99,9 +100,14 @@ void vfio_user_set_handler(VFIODevice *vbasedev, void *reqarg);bool vfio_user_validate_version (VFIOUserProxy *proxy,Error **errp); +VFIOUserFDs *vfio_user_getfds (int numfds); +void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd,uint32_t size,uint32_t flags); +void vfio_user_wait_reqs (VFIOUserProxy *proxy);bool vfio_user_send_wait (VFIOUserProxy *proxy,VFIOUserHdr *hdr,VFIOUserFDs *fds,int rsize,Error **errp); +bool vfio_user_send_nowait (VFIOUserProxy *proxy,VFIOUserHdr *hdr, +VFIOUserFDs *fds,int rsize,Error **errp);#endif /* VFIO_USER_PROXY_H */
hw/vfio-user/container.c +133 -2
162 lines not on this page · raw message
hw/vfio-user/proxy.c +82 -2
108 lines not on this page · raw message
hw/vfio-user/trace-events +4 -0
@@ -13,3 +13,7 @@ vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d" vfio_user_get_irq_info(uint32_t index, uint32_t flags, uint32_t count) " index %d flags 0x%x count %d" vfio_user_set_irqs(uint32_t index, uint32_t start, uint32_t count, uint32_t flags) " index %d start %d count %d flags 0x%x" + +# container.c +vfio_user_dma_map(uint64_t iova, uint64_t size, uint64_t off, uint32_t flags, bool async_ops) " iova 0x%"PRIx64" size 0x%"PRIx64" off 0x%"PRIx64" flags 0x%x async_ops %d" +vfio_user_dma_unmap(uint64_t iova, uint64_t size, uint32_t flags, bool async_ops) " iova 0x%"PRIx64" size 0x%"PRIx64" flags 0x%x async_ops %d"
Signature · 1 line
2.43.0
Re: [PATCH v5 13/19] vfio-user: implement VFIO_USER_DMA_MAP/UNMAP
Quoted 13 lines · John Levon
From: John Levon <[email protected]>
When the vfio-user container gets mapping updates, share them with the vfio-user by sending a message; this can include the region fd, allowing the server to directly
mmap()the region as needed.For performance, we only wait for the message responses when we're doing with a series of updates via the
listener_commit()callback.Originally-by John Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John Levon <[email protected]>
Quoted 8 lines
--- hw/vfio-user/protocol.h | 32 +++++++++ hw/vfio-user/proxy.h | 6 ++ hw/vfio-user/container.c | 135 +++++++++++++++++++++++++++++++++++++- hw/vfio-user/proxy.c | 84 +++++++++++++++++++++++- hw/vfio-user/trace-events | 4 ++ 5 files changed, 257 insertions(+), 4 deletions(-)
Thanks,
C.
PATCH v5 14/19 vfio-user: implement VFIO_USER_DMA_READ/WRITE
In reply to John LevonUnlike most other messages, this is a server->client message, for when a server wants to do "DMA"; this is slow, so normally the server has memory directly mapped instead.
--- hw/vfio-user/protocol.h | 13 ++++- hw/vfio-user/proxy.h | 3 ++ hw/vfio-user/pci.c | 111 ++++++++++++++++++++++++++++++++++++++++ hw/vfio-user/proxy.c | 97 +++++++++++++++++++++++++++++++++++ 4 files changed, 223 insertions(+), 1 deletion(-)
hw/vfio-user/protocol.h +12 -1
@@ -200,7 +200,18 @@ typedef struct {char data[]; } VFIOUserRegionRW; -/*imported from struct vfio_bitmap */ +/* +* VFIO_USER_DMA_READ +* VFIO_USER_DMA_WRITE +*/ +typedef struct { +VFIOUserHdr hdr; +uint64_t offset; +uint32_t count; +char data[]; +}VFIOUserDMARW ; + +/* imported from struct vfio_bitmap */ typedef struct { uint64_t pgsize; uint64_t size;
hw/vfio-user/proxy.h +3 -0
@@ -101,6 +101,7 @@ void vfio_user_set_handler(VFIODevice *vbasedev,bool vfio_user_validate_version (VFIOUserProxy *proxy,Error **errp);VFIOUserFDs *vfio_user_getfds (int numfds); +void vfio_user_putfds (VFIOUserMsg *msg);void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd,uint32_t size,uint32_t flags);
@@ -109,5 +110,7 @@ bool vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize, Error **errp);bool vfio_user_send_nowait (VFIOUserProxy *proxy,VFIOUserHdr *hdr,VFIOUserFDs *fds,int rsize,Error **errp); +void vfio_user_send_reply (VFIOUserProxy *proxy,VFIOUserHdr *hdr,int size); +void vfio_user_send_error (VFIOUserProxy *proxy,VFIOUserHdr *hdr,int error);#endif /* VFIO_USER_PROXY_H */
hw/vfio-user/pci.c +111 -0
130 lines not on this page · raw message
hw/vfio-user/proxy.c +97 -0
115 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v5 14/19] vfio-user: implement VFIO_USER_DMA_READ/WRITE
Quoted 8 lines · John Levon
Unlike most other messages, this is a
server->clientmessage, for when a server wants to do "DMA"; this is slow, so normally the server has memory directly mapped instead.Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Quoted 7 lines
--- hw/vfio-user/protocol.h | 13 ++++- hw/vfio-user/proxy.h | 3 ++ hw/vfio-user/pci.c | 111 ++++++++++++++++++++++++++++++++++++++++ hw/vfio-user/proxy.c | 97 +++++++++++++++++++++++++++++++++++ 4 files changed, 223 insertions(+), 1 deletion(-)
Thanks,
C.
PATCH v5 15/19 vfio-user: add 'x-msg-timeout' option
In reply to John LevonBy default, the vfio-user subsystem will wait 5 seconds for a message reply from the server. Add an option to allow this to be configurable.
--- hw/vfio-user/proxy.h | 1 + hw/vfio-user/pci.c | 5 +++++ hw/vfio-user/proxy.c | 7 ++++--- 3 files changed, 10 insertions(+), 3 deletions(-)
hw/vfio-user/proxy.h +1 -0
@@ -67,6 +67,7 @@ typedef struct VFIOUserProxy {uint64_t max_bitmap;uint64_t migr_pgsize;int flags; +uint32_t wait_time;QemuCond close_cv;AioContext *ctx;QEMUBH *req_bh;
hw/vfio-user/pci.c +5 -0
@@ -23,6 +23,7 @@ struct VFIOUserPCIDevice {VFIOPCIDevice device;SocketAddress *socket;bool send_queued;/* all sends are queued */ +uint32_t wait_time;/* timeout for message replies */ };/*
@@ -267,6 +268,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) proxy->flags |=VFIO_PROXY_FORCE_QUEUED ; } +/* user specified or 5 sec default */ + proxy->wait_time = udev->wait_time; +if (!vfio_user_validate_version (proxy, errp)) { goto error; }
@@ -398,6 +402,7 @@ static const Property vfio_user_pci_dev_properties[] = {DEFINE_PROP_UINT32 ("x-pci-sub-device-id" , VFIOPCIDevice, sub_device_id,PCI_ANY_ID ),DEFINE_PROP_BOOL ("x-send-queued" , VFIOUserPCIDevice, send_queued, false), + DEFINE_PROP_UINT32("x-msg-timeout", VFIOUserPCIDevice, wait_time, 5000), }; static void vfio_user_pci_set_socket(Object *obj, Visitor *v, const char *name,
hw/vfio-user/proxy.c +4 -3
@@ -21,7 +21,6 @@#include "qemu/main-loop.h" #include "system/iothread.h" -static int wait_time =5000 ;/* wait up to 5 sec for busy servers */ static IOThread *vfio_user_iothread;static void vfio_user_shutdown (VFIOUserProxy *proxy);
@@ -686,7 +685,8 @@ bool vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr,if (ok) {while (!msg->complete) { -if (!qemu_cond_timedwait (&msg->cv, &proxy->lock, wait_time)) { +if (!qemu_cond_timedwait (&msg->cv, &proxy->lock, + proxy->wait_time)) {VFIOUserMsgQ *list; list = msg->pending ? &proxy->pending : &proxy->outgoing;
@@ -758,7 +758,8 @@ void vfio_user_wait_reqs(VFIOUserProxy *proxy) msg->type =VFIO_MSG_WAIT ; proxy->last_nowait =NULL ;while (!msg->complete) { -if (!qemu_cond_timedwait (&msg->cv, &proxy->lock, wait_time)) { +if (!qemu_cond_timedwait (&msg->cv, &proxy->lock, + proxy->wait_time)) {VFIOUserMsgQ *list; list = msg->pending ? &proxy->pending : &proxy->outgoing;
Signature · 1 line
2.43.0
Re: [PATCH v5 15/19] vfio-user: add 'x-msg-timeout' option
Quoted 7 lines · John Levon
By default, the vfio-user subsystem will wait 5 seconds for a message reply from the server. Add an option to allow this to be configurable.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Quoted 6 lines
--- hw/vfio-user/proxy.h | 1 + hw/vfio-user/pci.c | 5 +++++ hw/vfio-user/proxy.c | 7 ++++--- 3 files changed, 10 insertions(+), 3 deletions(-)
Thanks,
C.
PATCH v5 16/19 vfio-user: support posted writes
In reply to John LevonSupport an asynchronous send of a vfio-user socket message (no wait for a reply) when the write is posted. This is only safe when no regions are mappable by the VM. Add an option to explicitly disable this as well.
--- hw/vfio-user/proxy.h | 6 ++++++ hw/vfio-user/device.c | 45 +++++++++++++++++++++++++++++++++++++++---- hw/vfio-user/pci.c | 6 ++++++ hw/vfio-user/proxy.c | 12 ++++++++++-- 4 files changed, 63 insertions(+), 6 deletions(-)
hw/vfio-user/proxy.h +6 -0
@@ -91,6 +91,7 @@ typedef struct VFIOUserProxy {/* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1#define VFIO_PROXY_FORCE_QUEUED 0x4 +#define VFIO_PROXY_NO_POST 0x8typedef struct VFIODevice VFIODevice ;
@@ -104,6 +105,8 @@ bool vfio_user_validate_version(VFIOUserProxy *proxy, Error **errp);VFIOUserFDs *vfio_user_getfds (int numfds);void vfio_user_putfds (VFIOUserMsg *msg); +void vfio_user_disable_posted_writes (VFIOUserProxy *proxy); +void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd,uint32_t size,uint32_t flags);void vfio_user_wait_reqs (VFIOUserProxy *proxy);
@@ -111,6 +114,9 @@ bool vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize, Error **errp);bool vfio_user_send_nowait (VFIOUserProxy *proxy,VFIOUserHdr *hdr,VFIOUserFDs *fds,int rsize,Error **errp); +bool vfio_user_send_async (VFIOUserProxy *proxy,VFIOUserHdr *hdr, +VFIOUserFDs *fds,Error **errp); +void vfio_user_send_reply (VFIOUserProxy *proxy,VFIOUserHdr *hdr,int size);void vfio_user_send_error (VFIOUserProxy *proxy,VFIOUserHdr *hdr,int error);
hw/vfio-user/device.c +41 -4
90 lines not on this page · raw message
hw/vfio-user/pci.c +6 -0
@@ -24,6 +24,7 @@ struct VFIOUserPCIDevice {SocketAddress *socket;bool send_queued;/* all sends are queued */ uint32_t wait_time;/* timeout for message replies */ +bool no_post;/* all region writes are sync */ };/*
@@ -268,6 +269,10 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) proxy->flags |=VFIO_PROXY_FORCE_QUEUED ; } +if (udev->no_post) { + proxy->flags |=VFIO_PROXY_NO_POST ; + } +/* user specified or 5 sec default */ proxy->wait_time = udev->wait_time;
@@ -403,6 +408,7 @@ static const Property vfio_user_pci_dev_properties[] = { sub_device_id, PCI_ANY_ID),DEFINE_PROP_BOOL ("x-send-queued" , VFIOUserPCIDevice, send_queued, false),DEFINE_PROP_UINT32 ("x-msg-timeout" , VFIOUserPCIDevice, wait_time,5000 ), + DEFINE_PROP_BOOL("x-no-posted-writes", VFIOUserPCIDevice, no_post, false), }; static void vfio_user_pci_set_socket(Object *obj, Visitor *v, const char *name,
hw/vfio-user/proxy.c +10 -2
@@ -714,8 +714,8 @@ bool vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr,* In either case, ownership of @hdr and @fds is taken, and the caller must * *not* free them itself. */ -static bool vfio_user_send_async (VFIOUserProxy *proxy,VFIOUserHdr *hdr, -VFIOUserFDs *fds,Error **errp) +bool vfio_user_send_async (VFIOUserProxy *proxy,VFIOUserHdr *hdr, +VFIOUserFDs *fds,Error **errp) {VFIOUserMsg *msg;
@@ -844,6 +844,14 @@ void vfio_user_putfds(VFIOUserMsg *msg) msg->fds =NULL ; } +void +vfio_user_disable_posted_writes (VFIOUserProxy *proxy) +{ + WITH_QEMU_LOCK_GUARD(&proxy->lock) { + proxy->flags |=VFIO_PROXY_NO_POST ; + } +} + static QLIST_HEAD(, VFIOUserProxy) vfio_user_sockets =QLIST_HEAD_INITIALIZER (vfio_user_sockets);
Signature · 1 line
2.43.0
Re: [PATCH v5 16/19] vfio-user: support posted writes
Quoted 4 lines · John Levon
Support an asynchronous send of a vfio-user socket message (no wait for a reply) when the write is posted. This is only safe when no regions are mappable by the VM. Add an option to explicitly disable this as well.
Quoted 8 lines
Signed-off-by John Levon <[email protected]>--- hw/vfio-user/proxy.h | 6 ++++++ hw/vfio-user/device.c | 45 +++++++++++++++++++++++++++++++++++++++---- hw/vfio-user/pci.c | 6 ++++++ hw/vfio-user/proxy.c | 12 ++++++++++-- 4 files changed, 63 insertions(+), 6 deletions(-)
Thanks,
C.
PATCH v5 17/19 vfio-user: add coalesced posted writes
In reply to John LevonAdd new message to send multiple writes to server in a single message. Prevents the outgoing queue from overflowing when a long latency operation is followed by a series of posted writes.
--- hw/vfio-user/protocol.h | 21 ++++++++++ hw/vfio-user/proxy.h | 12 ++++++ hw/vfio-user/device.c | 40 +++++++++++++++++++ hw/vfio-user/proxy.c | 84 +++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 1 + 5 files changed, 158 insertions(+)
hw/vfio-user/protocol.h +21 -0
@@ -39,6 +39,7 @@ enum vfio_user_command {VFIO_USER_DMA_WRITE =12 ,VFIO_USER_DEVICE_RESET =13 ,VFIO_USER_DIRTY_PAGES =14 , +VFIO_USER_REGION_WRITE_MULTI =15 , VFIO_USER_MAX, };
@@ -72,6 +73,7 @@ typedef struct {#define VFIO_USER_CAP_PGSIZES "pgsizes"#define VFIO_USER_CAP_MAP_MAX "max_dma_maps"#define VFIO_USER_CAP_MIGR "migration" +#define VFIO_USER_CAP_MULTI "write_multiple"/* "migration" members */ #define VFIO_USER_CAP_PGSIZE "pgsize"
@@ -218,4 +220,23 @@ typedef struct { char data[]; } VFIOUserBitmap; +/* +* VFIO_USER_REGION_WRITE_MULTI +*/ +#define VFIO_USER_MULTI_DATA 8 +#define VFIO_USER_MULTI_MAX 200 + +typedef struct { +uint64_t offset; +uint32_t region; +uint32_t count; +char data[VFIO_USER_MULTI_DATA ]; +}VFIOUserWROne ; + +typedef struct { +VFIOUserHdr hdr; +uint64_t wr_cnt; +VFIOUserWROne wrs[VFIO_USER_MULTI_MAX ]; +}VFIOUserWRMulti ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio-user/proxy.h +12 -0
@@ -85,6 +85,8 @@ typedef struct VFIOUserProxy {VFIOUserMsg *last_nowait;VFIOUserMsg *part_recv;size_t recv_left; +VFIOUserWRMulti *wr_multi; +int num_outgoing;enum proxy_state state; } VFIOUserProxy;
@@ -92,6 +94,11 @@ typedef struct VFIOUserProxy {#define VFIO_PROXY_CLIENT 0x1#define VFIO_PROXY_FORCE_QUEUED 0x4#define VFIO_PROXY_NO_POST 0x8 +#define VFIO_PROXY_USE_MULTI 0x16 + +/* coalescing high and low water marks for VFIOProxy num_outgoing */ +#define VFIO_USER_OUT_HIGH 1024 +#define VFIO_USER_OUT_LOW 128typedef struct VFIODevice VFIODevice ;
@@ -120,4 +127,9 @@ bool vfio_user_send_async(VFIOUserProxy *proxy, VFIOUserHdr *hdr,void vfio_user_send_reply (VFIOUserProxy *proxy,VFIOUserHdr *hdr,int size);void vfio_user_send_error (VFIOUserProxy *proxy,VFIOUserHdr *hdr,int error); +void vfio_user_flush_multi (VFIOUserProxy *proxy); +void vfio_user_create_multi (VFIOUserProxy *proxy); +void vfio_user_add_multi (VFIOUserProxy *proxy,uint8_t index, +off_t offset,uint32_t count,void *data); +#endif /* VFIO_USER_PROXY_H */
hw/vfio-user/device.c +40 -0
@@ -9,6 +9,8 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "qemu/error-report.h" +#include "qemu/lockable.h" +#include "qemu/thread.h" #include "hw/vfio-user/device.h" #include "hw/vfio-user/trace.h"
@@ -337,6 +339,7 @@ static int vfio_user_device_io_region_write(VFIODevice *vbasedev, uint8_t index,VFIOUserProxy *proxy = vbasedev->proxy;int size =sizeof (*msgp) + count;Error *local_err =NULL ; +bool can_multi;int flags =0 ;int ret;
@@ -352,6 +355,43 @@ static int vfio_user_device_io_region_write(VFIODevice *vbasedev, uint8_t index, flags |=VFIO_USER_NO_REPLY ; } +/* write eligible to be in a WRITE_MULTI msg ? */ + can_multi = (proxy->flags &VFIO_PROXY_USE_MULTI ) && post && + count <=VFIO_USER_MULTI_DATA ; + +/* +* This should be a rare case, so first check without the lock, +* if we're wrong, vfio_send_queued() will flush any posted writes +* we missed here +*/ +if (proxy->wr_multi !=NULL || + (proxy->num_outgoing >VFIO_USER_OUT_HIGH && can_multi)) { + +/* +* re-check with lock +* +* if already building a WRITE_MULTI msg, +* add this one if possible else flush pending before +* sending the current one +* +* else if outgoing queue is over the highwater, +* start a new WRITE_MULTI message +*/ + WITH_QEMU_LOCK_GUARD(&proxy->lock) { +if (proxy->wr_multi !=NULL ) { +if (can_multi) { +vfio_user_add_multi (proxy, index, off, count, data); +return count; + } +vfio_user_flush_multi (proxy); + }else if (proxy->num_outgoing >VFIO_USER_OUT_HIGH && can_multi) { +vfio_user_create_multi (proxy); +vfio_user_add_multi (proxy, index, off, count, data); +return count; + } + } + } + msgp =g_malloc0 (size);vfio_user_request_msg (&msgp->hdr,VFIO_USER_REGION_WRITE , size, flags); msgp->offset = off;
hw/vfio-user/proxy.c +84 -0
146 lines not on this page · raw message
hw/vfio-user/trace-events +1 -0
@@ -13,6 +13,7 @@ vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d" vfio_user_get_irq_info(uint32_t index, uint32_t flags, uint32_t count) " index %d flags 0x%x count %d" vfio_user_set_irqs(uint32_t index, uint32_t start, uint32_t count, uint32_t flags) " index %d start %d count %d flags 0x%x" +vfio_user_wrmulti(const char *s, uint64_t wr_cnt) " %s count 0x%"PRIx64 # container.c vfio_user_dma_map(uint64_t iova, uint64_t size, uint64_t off, uint32_t flags, bool async_ops) " iova 0x%"PRIx64" size 0x%"PRIx64" off 0x%"PRIx64" flags 0x%x async_ops %d"
Signature · 1 line
2.43.0
Re: [PATCH v5 17/19] vfio-user: add coalesced posted writes
Quoted 8 lines · John Levon
Add new message to send multiple writes to server in a single message. Prevents the outgoing queue from overflowing when a long latency operation is followed by a series of posted writes.
Originally-by John Johnson <[email protected]>Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Signed-off-by John Levon <[email protected]>
Quoted 8 lines
--- hw/vfio-user/protocol.h | 21 ++++++++++ hw/vfio-user/proxy.h | 12 ++++++ hw/vfio-user/device.c | 40 +++++++++++++++++++ hw/vfio-user/proxy.c | 84 +++++++++++++++++++++++++++++++++++++++ hw/vfio-user/trace-events | 1 + 5 files changed, 158 insertions(+)
Thanks,
C.
PATCH v5 18/19 docs: add vfio-user documentation
In reply to John LevonAdd some basic documentation on vfio-user usage.
--- docs/system/device-emulation.rst | 1 + docs/system/devices/vfio-user.rst | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+) create mode 100644 docs/system/devices/vfio-user.rst
docs/system/device-emulation.rst +1 -0
@@ -85,6 +85,7 @@ Emulated Devices devices/can.rst devices/ccid.rst devices/cxl.rst + devices/vfio-user.rst devices/ivshmem.rst devices/ivshmem-flat.rst devices/keyboard.rst
docs/system/devices/vfio-user.rst new file +26 -0
@@ -0,0 +1,26 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +========= +vfio-user +========= + +QEMU includes a ``vfio-user`` client. The ``vfio-user`` specification allows for +implementing (PCI) devices in userspace outside of QEMU; it is similar to +``vhost-user`` in this respect (see :doc:`vhost-user`), but can emulate arbitrary +PCI devices, not just ``virtio``. Whereas ``vfio`` is handled by the host +kernel, ``vfio-user``, while similar in implementation, is handled entirely in +userspace. + +For example, SPDK includes a virtual PCI NVMe controller implementation; by +setting up a ``vfio-user`` UNIX socket between QEMU and SPDK, a VM can send NVMe +I/O to the SPDK process. + +Presuming a suitable ``vfio-user`` server has opened a socket at +``/tmp/vfio-user.sock``, a device can be configured with for example: + +.. code-block:: console + +-device '{"driver": "vfio-user-pci","socket": {"path": "/tmp/vfio-user.sock", "type": "unix"}}' + +See `libvfio-user <https://github.com/nutanix/libvfio-user/>`_ for further +information.
Signature · 1 line
2.43.0
Re: [PATCH v5 18/19] docs: add vfio-user documentation
Quoted 8 lines · John Levon
Add some basic documentation on vfio-user usage.
Signed-off-by John Levon <[email protected]>--- docs/system/device-emulation.rst | 1 + docs/system/devices/vfio-user.rst | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+)create mode 100644
docs/system/devices/vfio-user.rst
Thanks,
C.
PATCH v5 19/19 vfio-user: introduce vfio-user protocol specification
In reply to John LevonFrom: Thanos Makatos <[email protected]>
This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts.
It has been earlier discussed as an RFC in:
"RFC: use VFIO over a UNIX domain socket to implement device offloading"
--- MAINTAINERS | 3 +- docs/interop/index.rst | 1 + docs/interop/vfio-user.rst | 1520 ++++++++++++++++++++++++++++++++++++ 3 files changed, 1523 insertions(+), 1 deletion(-) create mode 100644 docs/interop/vfio-user.rst
MAINTAINERS +2 -1
@@ -4246,7 +4246,6 @@ F: hw/remote/proxy-memory-listener.c F: include/hw/remote/proxy-memory-listener.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h -F: subprojects/libvfio-user F: hw/remote/vfio-user-obj.c F: include/hw/remote/vfio-user-obj.h F: hw/remote/iommu.c
@@ -4257,6 +4256,8 @@ VFIO-USER: M: John Levon <[email protected]> M: Thanos Makatos <[email protected]> S: Supported +F: docs/interop/vfio-user.rst +F: docs/system/devices/vfio-user.rst F: hw/vfio-user/* F: include/hw/vfio-user/* F: subprojects/libvfio-user
docs/interop/index.rst +1 -0
@@ -25,6 +25,7 @@ are useful for making QEMU interoperate with other software. qemu-ga-ref qemu-qmp-ref qemu-storage-daemon-qmp-ref + vfio-user vhost-user vhost-user-gpu vhost-vdpa
docs/interop/vfio-user.rst new file +1520 -0
1,520 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v5 19/19] vfio-user: introduce vfio-user protocol specification
Quoted 10 lines · John Levon
From: Thanos Makatos <[email protected]>
This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts.
It has been earlier discussed as an RFC in:
"RFC: use VFIO over a UNIX domain socket to implement device offloading"
Quoted 8 lines
Signed-off-by Thanos Makatos <[email protected]>Signed-off-by John Levon <[email protected]>--- MAINTAINERS | 3 +- docs/interop/index.rst | 1 + docs/interop/vfio-user.rst | 1520 ++++++++++++++++++++++++++++++++++++ 3 files changed, 1523 insertions(+), 1 deletion(-)create mode 100644
docs/interop/vfio-user.rst
Thanks,
C.
Cédric Le Goater
AppliedQuoted 75 lines · John Levon
The series contains an implementation of a vfio-user client in QEMU.
The vfio-user protocol allows for implementing (PCI) devices in another userspace process; SPDK is one example, which includes a virtual NVMe implementation.
The vfio-user framework consists of 3 parts:
1) The VFIO user protocol specification.
2) A client - the VFIO device in QEMU that encapsulates VFIO messages and sends them to the server.
3) A server - a remote process that emulates a device.This patchset implements parts 1 and 2.
It has been tested against libvfio-user test servers as well as SPDK. A functional test is still being worked on.
A previous version of this series can be found at
https://lore.kernel.org/qemu-devel/[email protected]/Changes since last series:
- fixed SPDX identifier nits
- code review changes for error handlingthanks
johnJohn Levon (18):
vfio-user: add vfio-user class and container
vfio-user: connect vfio proxy to remote server
vfio-user: implement message receive infrastructure
vfio-user: implement message send infrastructure
vfio-user: implementVFIO_USER_DEVICE_GET_INFO
vfio-user: implementVFIO_USER_DEVICE_GET_REGION_INFO
vfio-user: implement VFIO_USER_REGION_READ/WRITE
vfio-user: set up PCI invfio_user_pci_realize()
vfio-user: implement VFIO_USER_DEVICE_GET/SET_IRQ*
vfio-user: forward MSI-X PBA BAR accesses to server
vfio-user: set up container access to the proxy
vfio-user: implementVFIO_USER_DEVICE_RESET
vfio-user: implement VFIO_USER_DMA_MAP/UNMAP
vfio-user: implement VFIO_USER_DMA_READ/WRITE
vfio-user: add 'x-msg-timeout' option
vfio-user: support posted writes
vfio-user: add coalesced posted writes
docs: add vfio-user documentationThanos Makatos (1):
vfio-user: introduce vfio-user protocol specificationMAINTAINERS | 11 +- docs/interop/index.rst | 1 + docs/interop/vfio-user.rst | 1520 +++++++++++++++++++++++++ docs/system/device-emulation.rst | 1 + docs/system/devices/vfio-user.rst | 26 + meson.build | 1 + hw/vfio-user/container.h | 23 + hw/vfio-user/device.h | 24 + hw/vfio-user/protocol.h | 242 ++++ hw/vfio-user/proxy.h | 135 +++ hw/vfio-user/trace.h | 4 + hw/vfio/pci.h | 1 + include/hw/vfio/vfio-container-base.h | 1 + include/hw/vfio/vfio-device.h | 2 + hw/vfio-user/container.c | 370 ++++++ hw/vfio-user/device.c | 441 +++++++ hw/vfio-user/pci.c | 475 ++++++++ hw/vfio-user/proxy.c | 1356 ++++++++++++++++++++++ hw/Kconfig | 1 + hw/meson.build | 1 + hw/vfio-user/Kconfig | 7 + hw/vfio-user/meson.build | 11 + hw/vfio-user/trace-events | 20 + 23 files changed, 4673 insertions(+), 1 deletion(-)
Quoted 15 lines
create mode 100644
docs/interop/vfio-user.rstcreate mode 100644docs/system/devices/vfio-user.rstcreate mode 100644hw/vfio-user/container.hcreate mode 100644hw/vfio-user/device.hcreate mode 100644hw/vfio-user/protocol.hcreate mode 100644hw/vfio-user/proxy.hcreate mode 100644hw/vfio-user/trace.hcreate mode 100644hw/vfio-user/container.ccreate mode 100644hw/vfio-user/device.ccreate mode 100644hw/vfio-user/pci.ccreate mode 100644hw/vfio-user/proxy.ccreate mode 100644 hw/vfio-user/Kconfig create mode 100644 hw/vfio-user/meson.build create mode 100644 hw/vfio-user/trace-events
Applied to vfio-next.
Thanks,
C.