vfio-user implementation
v2 cover vfio-user implementation
Hi
This is v2 of the RFC patches for vfio-user multi-process QEMU project[1].
Thank you for the review of v1 of the RFC patches.
vfio-user is a protocol that allows a device to be emulated in a separate process outside of QEMU. It encapsulates the messages sent from QEMU to the kernel VFIO driver, and sends them to a remote process over a UNIX socket.
The vfio-user framework consists of 3 parts:
1) The protocol specification.
2) A server - the VFIO generic device in QEMU that exchanges the protocol messages with the client.
3) A client - remote process that emulates a device.
This patchset implements parts 1 and 2.
The protocol's specification can be found here [2]:
We also include this as the first patch of the series.
The libvfio-user project (https://github.com/nutanix/libvfio-user)
can be used by a remote process to handle the protocol to implement the third part.
We also worked on implementing a client and will be sending this patch series shortly.
Contributors:
John G Johnson <[email protected]>
John Levon <[email protected]>
Thanos Makatos <[email protected]>
Elena Ufimtseva <[email protected]>
Jagannathan Raman <[email protected]>
Changes in v2:
- combine some patches with relevant functionality.
- use SocketAddress with idea to modify later the command line options.
- define protocol bits in user-protocol.h.
- use QEMU_LOCK_GUARD where appropriate.
- fix the locking when event signaling.
- do not drop BQL on dma map/unmap.
- added checks for message sizes in communication functions.
John Johnson (15):
vfio-user: add VFIO base abstract class
vfio-user: Define type vfio_user_pci_dev_info
vfio-user: connect vfio proxy to remote server
vfio-user: define VFIO Proxy and communication functions
vfio-user: negotiate version with remote server
vfio-user: get device info
vfio-user: get region info
vfio-user: region read/write
vfio-user: pci_user_realize PCI setup
vfio-user: get and set IRQs
vfio-user: proxy container connect/disconnect
vfio-user: dma map/unmap operations
vfio-user: dma read/write operations
vfio-user: pci reset
vfio-user: migration support
Thanos Makatos (1):
vfio-user: introduce vfio-user protocol specification
docs/devel/index.rst | 1 + docs/devel/vfio-user.rst | 1809 +++++++++++++++++++++++++++++++++ hw/vfio/pci.h | 25 +- hw/vfio/user-protocol.h | 210 ++++ hw/vfio/user.h | 95 ++ include/hw/vfio/vfio-common.h | 9 + hw/vfio/common.c | 296 +++++- hw/vfio/migration.c | 34 +- hw/vfio/pci.c | 571 +++++++++-- hw/vfio/user.c | 1104 ++++++++++++++++++++ MAINTAINERS | 11 + hw/vfio/meson.build | 1 + 12 files changed, 4062 insertions(+), 104 deletions(-) create mode 100644 docs/devel/vfio-user.rst create mode 100644 hw/vfio/user-protocol.h create mode 100644 hw/vfio/user.h create mode 100644 hw/vfio/user.c
Signature · 1 line
2.25.1
RFC PATCH v2 1/16 vfio-user: introduce vfio-user protocol specification
From: Thanos Makatos <[email protected]>
This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts.
This patch is sourced from:
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
It has been earlier discussed as an RFC in:
"RFC: use VFIO over a UNIX domain socket to implement device offloading"
--- docs/devel/index.rst | 1 + docs/devel/vfio-user.rst | 1809 ++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 6 + 3 files changed, 1816 insertions(+) create mode 100644 docs/devel/vfio-user.rst
docs/devel/index.rst +1 -0
@@ -44,3 +44,4 @@ modifying QEMU's source code. vfio-migration qapi-code-gen writing-qmp-commands + vfio-user
docs/devel/vfio-user.rst new file +1809 -0
1,809 lines not on this page · raw message
MAINTAINERS +6 -0
@@ -1879,6 +1879,12 @@ F: hw/vfio/ap.c F: docs/system/s390x/vfio-ap.rst L: [email protected] +vfio-user +M: John G Johnson <[email protected]> +M: Thanos Makatos <[email protected]> +S: Supported +F: docs/devel/vfio-user.rst + vhost M: Michael S. Tsirkin <[email protected]> S: Supported
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification
docs/devel/vfio-user.rst line 218
Quoted 6 lines · On Mon, 16 Aug 2021 09:42:34 -0700 Elena Ufimtseva…
docs/devel/vfio-user.rst new file +6
-0+Authentication +-------------- + +For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files, +therefore it is up to the management layer to set up the socket as required. +Socket types than span guests or hosts will require a proper authentication
s/than/that/
...
Quoted 40 lines
+``VFIO_USER_DMA_UNMAP``
+-----------------------docs/devel/vfio-user.rst new file +38
-0+ +This command message is sent by the client to the server to inform it that a +DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command +message, is no longer available for DMA. It typically occurs when memory is +subtracted from the client or if the client uses a vIOMMU. The DMA region is +described by the following structure: + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++--------------+--------+------------------------+ +| Name | Offset | Size | ++==============+========+========================+ +| argsz | 0 | 4 | ++--------------+--------+------------------------+ +| flags | 4 | 4 | ++--------------+--------+------------------------+ +| | +-----+-----------------------+ | +| | | Bit | Definition | | +| | +=====+=======================+ | +| | | 0 | get dirty page bitmap | | +| | +-----+-----------------------+ | ++--------------+--------+------------------------+ +| address | 8 | 8 | ++--------------+--------+------------------------+ +| size | 16 | 8 | ++--------------+--------+------------------------+ + +* *argsz* is the maximum size of the reply payload. +* *flags* contains the following DMA region attributes: + + * *get dirty page bitmap* indicates that a dirty page bitmap must be + populated before unmapping the DMA region. The client must provide a + `VFIO Bitmap`_ structure, explained below, immediately following this + entry. +
docs/devel/vfio-user.rst line 637
Quoted 10 lines
docs/devel/vfio-user.rst new file +10
-0+* *address* is the base DMA address of the DMA region. +* *size* is the size of the DMA region. + +The address and size of the DMA region being unmapped must match exactly a +previous mapping. The size of request message depends on whether or not the +*get dirty page bitmap* bit is set in Flags: + +* If not set, the size of the total request message is: 16 + 24. + +* If set, the size of the total request message is: 16 + 24 + 16.
The address/size paradigm falls into the same issues as the vfio kernel interface where we can't map or unmap the entire 64-bit address space, ie. size is limited to 2^64 - 1. The kernel interface also requires PAGE_SIZE granularity for the DMA, which means the practical limit is 2^64 - PAGE_SIZE. If we had a redo on the kernel interface we'd use start/end so we can express a size of (end - start + 1).
Is following the vfio kernel interface sufficiently worthwhile for compatibility to incur this same limitation? I don't recall if we've already discussed this, but perhaps worth a note in this design doc if similarity to the kernel interface is being favored here. See for example QEMU commit 1b296c3def4b ("vfio: Don't issue full 2^64 unmap"). Thanks,
Alex
Re: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification
On 18. Aug 2021, at 01:04, Alex Williamson <[email protected]<mailto:[email protected]>> wrote:
On Mon, 16 Aug 2021 09:42:34 -0700
Elena Ufimtseva <[email protected]<mailto:[email protected]>> wrote:
+Authentication
+--------------
+
+For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files, +therefore it is up to the management layer to set up the socket as required. +Socket types than span guests or hosts will require a proper authentication
s/than/that/
Ack
...
+``VFIO_USER_DMA_UNMAP``
+-----------------------
+
+This command message is sent by the client to the server to inform it that a +DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command +message, is no longer available for DMA. It typically occurs when memory is +subtracted from the client or if the client uses a vIOMMU. The DMA region is +described by the following structure:
+
+Request
+^^^^^^^
+
+The request payload for this message is a structure of the following format:
+
++--------------+--------+------------------------+
+| Name | Offset | Size |
++==============+========+========================+
+| argsz | 0 | 4 |
++--------------+--------+------------------------+
+| flags | 4 | 4 |
++--------------+--------+------------------------+
+| | +-----+-----------------------+ | +| | | Bit | Definition | | +| | +=====+=======================+ | +| | | 0 | get dirty page bitmap | | +| | +-----+-----------------------+ |
++--------------+--------+------------------------+
+| address | 8 | 8 |
++--------------+--------+------------------------+
+| size | 16 | 8 |
++--------------+--------+------------------------+
+
+* *argsz* is the maximum size of the reply payload.
+* *flags* contains the following DMA region attributes:
+
+ * *get dirty page bitmap* indicates that a dirty page bitmap must be
+ populated before unmapping the DMA region. The client must provide a + `VFIO Bitmap`_ structure, explained below, immediately following this + entry.
+
+* *address* is the base DMA address of the DMA region.
+* *size* is the size of the DMA region.
+
+The address and size of the DMA region being unmapped must match exactly a +previous mapping. The size of request message depends on whether or not the +*get dirty page bitmap* bit is set in Flags:
+
+* If not set, the size of the total request message is: 16 + 24. +
+* If set, the size of the total request message is: 16 + 24 + 16.
The address/size paradigm falls into the same issues as the vfio kernel interface where we can't map or unmap the entire 64-bit address space, ie. size is limited to 2^64 - 1. The kernel interface also requires PAGE_SIZE granularity for the DMA, which means the practical limit is 2^64 - PAGE_SIZE. If we had a redo on the kernel interface we'd use start/end so we can express a size of (end - start + 1).
Is following the vfio kernel interface sufficiently worthwhile for compatibility to incur this same limitation? I don't recall if we've already discussed this, but perhaps worth a note in this design doc if similarity to the kernel interface is being favored here. See for example QEMU commit 1b296c3def4b ("vfio: Don't issue full 2^64 unmap”). I cannot think of any reason that we need to have this limitation in vfio-user. Opened https://github.com/nutanix/libvfio-user/issues/593 to address this issue.
Thanks,
-Swapnil
Re: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification
In reply to Alex WilliamsonQuoted 12 lines · Alex Williamson
The address/size paradigm falls into the same issues as the vfio kernel interface where we can't map or unmap the entire 64-bit address space, ie. size is limited to 2^64 - 1. The kernel interface also requires
PAGE_SIZEgranularity for the DMA, which means the practical limit is 2^64 -PAGE_SIZE. If we had a redo on the kernel interface we'd use start/end so we can express a size of (end - start + 1).
Quoted 7 lines
Is following the vfio kernel interface sufficiently worthwhile for compatibility to incur this same limitation? I don't recall if we've already discussed this, but perhaps worth a note in this design doc if similarity to the kernel interface is being favored here. See for example QEMU commit
1b296c3def4b("vfio: Don't issue full 2^64 unmap"). Thanks,
I’d prefer to stay as close to the kernel i/f as we can.
An earlier version of the spec used a vhost-user derived structure for MAP & UNMAP. This made it more difficult to add the bitmap field when vfio added migration capability, so we switched to the ioctl() structure.
It looks like vfio_dma_unmap() takes a 64b ‘size’ arg
(ram_addr_t) How did you unmap an entire 64b address space? The comment there mentions a bug where iova+size wraps the end of the 64b space.
JJ
Re: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification
Quoted 19 lines · On Thu, 19 Aug 2021 15:32:16 +0000 John Johnson <[email protected]>…
Quoted 10 lines · Alex Williamson
The address/size paradigm falls into the same issues as the vfio kernel interface where we can't map or unmap the entire 64-bit address space, ie. size is limited to 2^64 - 1. The kernel interface also requires
PAGE_SIZEgranularity for the DMA, which means the practical limit is 2^64 -PAGE_SIZE. If we had a redo on the kernel interface we'd use start/end so we can express a size of (end - start + 1).Quoted 7 lines
Is following the vfio kernel interface sufficiently worthwhile for compatibility to incur this same limitation? I don't recall if we've already discussed this, but perhaps worth a note in this design doc if similarity to the kernel interface is being favored here. See for example QEMU commit
1b296c3def4b("vfio: Don't issue full 2^64 unmap"). Thanks,
Quoted 8 lines
I’d prefer to stay as close to the kernel i/f as we can.
An earlier version of the spec used a vhost-user derived structure for MAP & UNMAP. This made it more difficult to add the bitmap field when vfio added migration capability, so we switched to theioctl()structure.It looks like
vfio_dma_unmap()takes a 64b ‘size’ arg
(ram_addr_t) How did you unmap an entire 64b address space?
It's called from the MemoryListener which operates on MemoryRegionSections, which uses Int128 that get's chunked to ram_addr_t for vfio_dma_unmap(). We do now have VFIO_DMA_UNMAP_FLAG_ALL in the kernel API which gives us an option to clear the whole 64bit address space in one ioctl, but it's not a high priority to make use of in QEMU since it still needs to handle older kernels.
Quoted 2 lines
The comment there mentions a bug where iova+size wraps the end of the 64b space.
Right, that's a separate issue that's just a bug in the kernel. That's been fixed, but the QEMU code exists for now as a workaround for any broken kernels in the wild. Thanks,
Alex
RFC PATCH v2 2/16 vfio-user: add VFIO base abstract class
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
Add an abstract base class both the kernel driver and user socket implementations can use to share code.
--- hw/vfio/pci.h | 16 +++++++++++-- hw/vfio/pci.c | 63 ++++++++++++++++++++++++++++++++------------------- 2 files changed, 54 insertions(+), 25 deletions(-)
hw/vfio/pci.h +14 -2
@@ -114,8 +114,13 @@ typedef struct VFIOMSIXInfo {unsigned long *pending; } VFIOMSIXInfo; -#define TYPE_VFIO_PCI "vfio-pci" -OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI) +/* +* TYPE_VFIO_PCI_BASE is an abstract type used to share code +* between VFIO implementations that use a kernel driver +* with those that use user sockets. +*/ +#define TYPE_VFIO_PCI_BASE "vfio-pci-base" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_BASE)struct VFIOPCIDevice {PCIDevice pdev;
@@ -175,6 +180,13 @@ struct VFIOPCIDevice {Notifier irqchip_change_notifier; }; +#define TYPE_VFIO_PCI "vfio-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOKernPCIDevice, VFIO_PCI) + +struct VFIOKernPCIDevice { +VFIOPCIDevice device; +}; +/* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ static inline bool vfio_pci_is (VFIOPCIDevice *vdev,uint32_t vendor,uint32_t device) {
hw/vfio/pci.c +40 -23
163 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH v2 3/16 vfio-user: Define type vfio_user_pci_dev_info
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
New class for vfio-user with its class and instance constructors and destructors, and its pci ops.
--- hw/vfio/pci.h | 9 ++++++ hw/vfio/pci.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+)
hw/vfio/pci.h +9 -0
@@ -187,6 +187,15 @@ struct VFIOKernPCIDevice {VFIOPCIDevice device; }; +#define TYPE_VFIO_USER_PCI "vfio-user-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) + +struct VFIOUserPCIDevice { +VFIOPCIDevice device; +char *sock_name; +bool secure_dma;/* disable shared mem for DMA */ +}; +/* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ static inline bool vfio_pci_is (VFIOPCIDevice *vdev,uint32_t vendor,uint32_t device) {
hw/vfio/pci.c +86 -0
89 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 03/16] vfio-user: Define type vfio_user_pci_dev_info
hw/vfio/pci.c static void register_vfio_pci_dev_type(void) · line 3388
Quoted 5 lines · Elena Ufimtseva
hw/vfio/pci.c +5
-0+static Property vfio_user_pci_dev_properties[] = { +DEFINE_PROP_STRING ("socket" , VFIOUserPCIDevice, sock_name), +DEFINE_PROP_BOOL ("secure-dma" , VFIOUserPCIDevice, secure_dma, false), +DEFINE_PROP_END_OF_LIST (), +};
Are we missing out on properties that could be common for all VFIO PCI devices like x-pci-vendor-id, x-pci-device-id, etc?
Stefan
RFC PATCH v2 4/16 vfio-user: connect vfio proxy to remote server
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user.h | 66 ++++++++++++++ include/hw/vfio/vfio-common.h | 2 + hw/vfio/pci.c | 29 ++++++ hw/vfio/user.c | 160 ++++++++++++++++++++++++++++++++++ MAINTAINERS | 4 + hw/vfio/meson.build | 1 + 6 files changed, 262 insertions(+) create mode 100644 hw/vfio/user.h create mode 100644 hw/vfio/user.c
hw/vfio/user.h new file +66 -0
@@ -0,0 +1,66 @@ +#ifndef VFIO_USER_H +#define VFIO_USER_H + +/* +* vfio protocol over a UNIX socket. +* +* Copyright © 2018, 2021 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL, version 2. See +* the COPYING file in the top-level directory. +* +*/ + +typedef struct { +int send_fds; +int recv_fds; +int *fds; +}VFIOUserFDs ; + +typedef struct VFIOUserReply { +QTAILQ_ENTRY (VFIOUserReply) next; +VFIOUserFDs *fds; +uint32_t rsize; +uint32_t id; +QemuCond cv; +bool complete; +bool nowait; +}VFIOUserReply ; + + +enum proxy_state { +VFIO_PROXY_CONNECTED =1 , +VFIO_PROXY_RECV_ERROR =2 , +VFIO_PROXY_CLOSING =3 , +VFIO_PROXY_CLOSED =4 , +}; + +typedef struct VFIOProxy { +QLIST_ENTRY (VFIOProxy) next; +char *sockname; +struct QIOChannel *ioc; +int (*request)(void *opaque,char *buf,VFIOUserFDs *fds); +void *reqarg; +int flags; +QemuCond close_cv; + +/* +* above only changed when BQL is held +* below are protected by per-proxy lock +*/ +QemuMutex lock; + QTAILQ_HEAD(, VFIOUserReply) free; + QTAILQ_HEAD(, VFIOUserReply) pending; +VFIOUserReply *last_nowait; +enum proxy_state state; +bool close_wait; +}VFIOProxy ; + +/* VFIOProxy flags */ +#define VFIO_PROXY_CLIENT 0x1 +#define VFIO_PROXY_SECURE 0x2 + +VFIOProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp); +void vfio_user_disconnect (VFIOProxy *proxy); + +#endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +2 -0
@@ -75,6 +75,7 @@ typedef struct VFIOAddressSpace { } VFIOAddressSpace;struct VFIOGroup ; +typedef struct VFIOProxy VFIOProxy ; typedef struct VFIOContainer { VFIOAddressSpace *space;
@@ -143,6 +144,7 @@ typedef struct VFIODevice {VFIOMigration *migration;Error *migration_blocker;OnOffAuto pre_copy_dirty_page_tracking; +VFIOProxy *proxy; } VFIODevice;struct VFIODeviceOps {
hw/vfio/pci.c +29 -0
@@ -42,6 +42,7 @@#include "qapi/error.h" #include "migration/blocker.h" #include "migration/qemu-file.h" +#include "hw/vfio/user.h" #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug"
@@ -3361,13 +3362,35 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)VFIOUserPCIDevice *udev =VFIO_USER_PCI (pdev);VFIOPCIDevice *vdev =VFIO_PCI_BASE (pdev);VFIODevice *vbasedev = &vdev->vbasedev; +SocketAddress addr; +VFIOProxy *proxy; +Error *err =NULL ; +/* +* TODO: make option parser understand SocketAddress +* and use that instead of having scaler options +* for each socket type. +*/ if (!udev->sock_name) {error_setg (errp,"No socket specified" );error_append_hint (errp,"Use -device vfio-user-pci,socket=<name>\n" );return ; } +memset (&addr,0 ,sizeof (addr)); + addr.type =SOCKET_ADDRESS_TYPE_UNIX ; + addr.u.q_unix.path = udev->sock_name; + proxy =vfio_user_connect_dev (&addr, &err); +if (!proxy) { +error_setg (errp,"Remote proxy not found" ); +return ; + } + vbasedev->proxy = proxy; + +if (udev->secure_dma) { + proxy->flags |=VFIO_PROXY_SECURE ; + } + vbasedev->name =g_strdup_printf ("VFIO user <%s>" , udev->sock_name); vbasedev->dev =DEVICE (vdev); vbasedev->fd =-1 ;
@@ -3379,6 +3402,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)static void vfio_user_instance_finalize (Object *obj) { +VFIOPCIDevice *vdev =VFIO_PCI_BASE (obj); +VFIODevice *vbasedev = &vdev->vbasedev; + +vfio_put_device (vdev); + +vfio_user_disconnect (vbasedev->proxy); }static Property vfio_user_pci_dev_properties[] = {
hw/vfio/user.c new file +160 -0
160 lines not on this page · raw message
MAINTAINERS +4 -0
@@ -1882,8 +1882,12 @@ L: [email protected] vfio-user M: John G Johnson <[email protected]> M: Thanos Makatos <[email protected]> +M: Elena Ufimtseva <[email protected]> +M: Jagannathan Raman <[email protected]> S: Supported F: docs/devel/vfio-user.rst +F: hw/vfio/user.c +F: hw/vfio/user.h vhost M: Michael S. Tsirkin <[email protected]>
hw/vfio/meson.build +1 -0
@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'display.c', 'pci-quirks.c', 'pci.c', + 'user.c', )) vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c')) vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
hw/vfio/meson.build vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( · line 14
Quoted 12 lines · On Mon, 16 Aug 2021 09:42:37 -0700 Elena Ufimtseva…
hw/vfio/meson.build +1
-0@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'display.c', 'pci-quirks.c', 'pci.c', + 'user.c', )) vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c')) vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))
Wouldn't it make sense to be able to configure QEMU with any combination of vfio-pci and/or vfio-user-pci support rather than statically tying vfio-user-pci to vfio-pci? Not to mention that doing so would help to more formally define the interface operations between kernel and user options, for example fewer tests of vbasedev->proxy and perhaps more abstraction through ops structures. Thanks,
Alex
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 18 lines · Alex Williamson
Quoted 11 lines · On Mon, 16 Aug 2021 09:42:37 -0700 Elena Ufimtseva…
hw/vfio/meson.build +1
-0@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'display.c', 'pci-quirks.c', 'pci.c', + 'user.c', )) vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c'))
Quoted 9 lines
hw/vfio/meson.build vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( · line 14
Quoted 1 line
hw/vfio/meson.build +0
-0vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))Wouldn't it make sense to be able to configure QEMU with any combination of vfio-pci and/or vfio-user-pci support rather than statically tying vfio-user-pci to vfio-pci? Not to mention that doing so would help to more formally define the interface operations between kernel and user options, for example fewer tests of
vbasedev->proxyand perhaps more abstraction through ops structures. Thanks,
We can certainly add another config option for vfio-user. As far as an ops vector vs vbasedev->proxy tests, it’s a
matter personal preference. I prefer the changes inline when they are this small, but we can make a vector if that’s what you want.
JJ
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
In reply to Elena Ufimtsevahw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3371
Quoted 11 lines · Elena Ufimtseva
hw/vfio/pci.c +6
-0@@ -3361,13 +3362,35 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)VFIOUserPCIDevice *udev =VFIO_USER_PCI (pdev);VFIOPCIDevice *vdev =VFIO_PCI_BASE (pdev);VFIODevice *vbasedev = &vdev->vbasedev; +SocketAddress addr; +VFIOProxy *proxy; +Error *err =NULL ; +/* +* TODO: make option parser understand SocketAddress +* and use that instead of having scaler options
s/scaler/scalar/
Quoted 13 lines
hw/vfio/user.c new file +13
-0+VFIOProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp) +{ +VFIOProxy *proxy; +QIOChannelSocket *sioc; +QIOChannel *ioc; +char *sockname; + +if (addr->type !=SOCKET_ADDRESS_TYPE_UNIX ) { +error_setg (errp,"vfio_user_connect - bad address family" ); +return NULL ; + } + sockname = addr->u.q_unix.path; +
hw/vfio/user.c line 86
Quoted 10 lines
hw/vfio/user.c new file +10
-0+ sioc =qio_channel_socket_new (); + ioc =QIO_CHANNEL (sioc); +if (qio_channel_socket_connect_sync (sioc, addr, errp)) { +object_unref (OBJECT (ioc)); +return NULL ; + } +qio_channel_set_blocking (ioc, true,NULL ); + + proxy =g_malloc0 (sizeof (VFIOProxy)); + proxy->sockname = sockname;
sockname is addr->u.q_unix.path, so there's an assumption that the lifetime of the addr argument is at least as long as the proxy object's lifetime. This doesn't seem to be the case in vfio_user_pci_realize() since the SocketAddress variable is declared on the stack.
I suggest making SocketAddress *addr const so it's obvious that this function just reads it (doesn't take ownership of the pointer) and copying the UNIX domain socket path with g_strdup() to avoid the dangling pointer.
hw/vfio/user.c line 94
Quoted 8 lines
hw/vfio/user.c new file +8
-0+ proxy->ioc = ioc; + proxy->flags =VFIO_PROXY_CLIENT ; + proxy->state =VFIO_PROXY_CONNECTED ; +qemu_cond_init (&proxy->close_cv); + +if (vfio_user_iothread ==NULL ) { + vfio_user_iothread =iothread_create ("VFIO user" , errp); + }
Why is a dedicated IOThread needed for VFIO user?
hw/vfio/user.c line 103
Quoted 9 lines
hw/vfio/user.c new file +9
-0+ +qemu_mutex_init (&proxy->lock); +QTAILQ_INIT (&proxy->free); +QTAILQ_INIT (&proxy->pending); +QLIST_INSERT_HEAD (&vfio_user_sockets, proxy, next); + +return proxy; +} +
/* Called with the BQL */
Quoted 25 lines
hw/vfio/user.c new file +25
-0+void vfio_user_disconnect (VFIOProxy *proxy) +{ +VFIOUserReply *r1, *r2; + +qemu_mutex_lock (&proxy->lock); + +/* our side is quitting */ +if (proxy->state ==VFIO_PROXY_CONNECTED ) { +vfio_user_shutdown (proxy); +if (!QTAILQ_EMPTY (&proxy->pending)) { +error_printf ("vfio_user_disconnect: outstanding requests\n" ); + } + } +object_unref (OBJECT (proxy->ioc)); + proxy->ioc =NULL ; + + proxy->state =VFIO_PROXY_CLOSING ; + QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) { +qemu_cond_destroy (&r1->cv); +QTAILQ_REMOVE (&proxy->pending, r1, next); +g_free (r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) { +qemu_cond_destroy (&r1->cv); +QTAILQ_REMOVE (&proxy->free, r1, next);
hw/vfio/user.c line 138
Quoted 10 lines
hw/vfio/user.c new file +10
-0+g_free (r1); + } + +/* +* Make sure the iothread isn't blocking anywhere +* with a ref to this proxy by waiting for a BH +* handler to run after the proxy fd handlers were +* deleted above. +*/ + proxy->close_wait =1 ;
Please use true. '1' is shorter but it's less obvious to the reader (I had to jump to the definition to check whether this field was bool or int).
hw/vfio/user.c line 143
Quoted 5 lines
hw/vfio/user.c new file +5
-0+aio_bh_schedule_oneshot (iothread_get_aio_context (vfio_user_iothread), + vfio_user_cb, proxy); + +/* drop locks so the iothread can make progress */ +qemu_mutex_unlock_iothread ();
Why does the BQL needs to be dropped so vfio_user_iothread can make progress?
Quoted 1 line
+
qemu_cond_wait(&proxy->close_cv, &proxy->lock);
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 13 lines · Stefan Hajnoczi
Quoted 8 lines · Elena Ufimtseva
hw/vfio/pci.c +3
-0@@ -3361,13 +3362,35 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)VFIOUserPCIDevice *udev =VFIO_USER_PCI (pdev);VFIOPCIDevice *vdev =VFIO_PCI_BASE (pdev);VFIODevice *vbasedev = &vdev->vbasedev; +SocketAddress addr; +VFIOProxy *proxy; +Error *err =NULL ;
Quoted 6 lines
hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3371
Quoted 3 lines
hw/vfio/pci.c +3
-0+/* +* TODO: make option parser understand SocketAddress +* and use that instead of having scaler options s/scaler/scalar/
OK
Quoted 29 lines
Quoted 13 lines
hw/vfio/user.c new file +13
-0+VFIOProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp) +{ +VFIOProxy *proxy; +QIOChannelSocket *sioc; +QIOChannel *ioc; +char *sockname; + +if (addr->type !=SOCKET_ADDRESS_TYPE_UNIX ) { +error_setg (errp,"vfio_user_connect - bad address family" ); +return NULL ; + } + sockname = addr->u.q_unix.path; +hw/vfio/user.c line 86
Quoted 10 lines
hw/vfio/user.c new file +10
-0+ sioc =qio_channel_socket_new (); + ioc =QIO_CHANNEL (sioc); +if (qio_channel_socket_connect_sync (sioc, addr, errp)) { +object_unref (OBJECT (ioc)); +return NULL ; + } +qio_channel_set_blocking (ioc, true,NULL ); + + proxy =g_malloc0 (sizeof (VFIOProxy)); + proxy->sockname = sockname;sockname is
addr->u.q_unix.path, so there's an assumption that the lifetime of the addr argument is at least as long as the proxy object's lifetime. This doesn't seem to be the case invfio_user_pci_realize()since the SocketAddress variable is declared on the stack.
Quoted 5 lines
I suggest making SocketAddress *addr const so it's obvious that this function just reads it (doesn't take ownership of the pointer) and copying the UNIX domain socket path with
g_strdup()to avoid the dangling pointer.
OK
Quoted 11 lines
hw/vfio/user.c line 94
Quoted 8 lines
hw/vfio/user.c new file +8
-0+ proxy->ioc = ioc; + proxy->flags =VFIO_PROXY_CLIENT ; + proxy->state =VFIO_PROXY_CONNECTED ; +qemu_cond_init (&proxy->close_cv); + +if (vfio_user_iothread ==NULL ) { + vfio_user_iothread =iothread_create ("VFIO user" , errp); + }Why is a dedicated IOThread needed for VFIO user?
It seemed the best model for inbound message processing. Most messages
are replies, so the receiver will either signal a thread waiting the reply or report any errors from the server if there is no waiter. None of this requires the BQL.
If the message is a request - which currently only happens if device
DMA targets guest memory that wasn’t mmap()d by QEMU or if the ’secure-dma’ option is used - then the receiver can then acquire BQL so it can call the VFIO code to handle the request.
Quoted 11 lines
hw/vfio/user.c line 103
Quoted 9 lines
hw/vfio/user.c new file +9
-0+ +qemu_mutex_init (&proxy->lock); +QTAILQ_INIT (&proxy->free); +QTAILQ_INIT (&proxy->pending); +QLIST_INSERT_HEAD (&vfio_user_sockets, proxy, next); + +return proxy; +} +/* Called with the BQL */
OK
Quoted 31 lines
Quoted 31 lines
hw/vfio/user.c new file +31
-0+void vfio_user_disconnect (VFIOProxy *proxy) +{ +VFIOUserReply *r1, *r2; + +qemu_mutex_lock (&proxy->lock); + +/* our side is quitting */ +if (proxy->state ==VFIO_PROXY_CONNECTED ) { +vfio_user_shutdown (proxy); +if (!QTAILQ_EMPTY (&proxy->pending)) { +error_printf ("vfio_user_disconnect: outstanding requests\n" ); + } + } +object_unref (OBJECT (proxy->ioc)); + proxy->ioc =NULL ; + + proxy->state =VFIO_PROXY_CLOSING ; + QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) { +qemu_cond_destroy (&r1->cv); +QTAILQ_REMOVE (&proxy->pending, r1, next); +g_free (r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) { +qemu_cond_destroy (&r1->cv); +QTAILQ_REMOVE (&proxy->free, r1, next); +g_free (r1); + } + +/* +* Make sure the iothread isn't blocking anywhere +* with a ref to this proxy by waiting for a BH
Quoted 9 lines
hw/vfio/user.c line 138
Quoted 4 lines
hw/vfio/user.c new file +4
-0+* handler to run after the proxy fd handlers were +* deleted above. +*/ + proxy->close_wait =1 ;Please use true. '1' is shorter but it's less obvious to the reader (I had to jump to the definition to check whether this field was bool or int).
I assume this is also true for the other boolean struct members
I’ve added.
Quoted 9 lines
hw/vfio/user.c line 143
Quoted 5 lines
hw/vfio/user.c new file +5
-0+aio_bh_schedule_oneshot (iothread_get_aio_context (vfio_user_iothread), + vfio_user_cb, proxy); + +/* drop locks so the iothread can make progress */ +qemu_mutex_unlock_iothread ();Why does the BQL needs to be dropped so
vfio_user_iothreadcan make progress?
See above. Acquiring BQL by the iothread is rare, but I have to
handle the case where a disconnect is concurrent with an incoming request message that is waiting for the BQL. See the proxy state check after BQL is acquired in vfio_user_recv()
Quoted 1 line
Quoted 1 line
+
qemu_cond_wait(&proxy->close_cv, &proxy->lock);
JJ
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 19 lines · John Johnson
Quoted 11 lines · Stefan Hajnoczi
hw/vfio/user.c line 94
Quoted 8 lines · Elena Ufimtseva
hw/vfio/user.c new file +8
-0+ proxy->ioc = ioc; + proxy->flags =VFIO_PROXY_CLIENT ; + proxy->state =VFIO_PROXY_CONNECTED ; +qemu_cond_init (&proxy->close_cv); + +if (vfio_user_iothread ==NULL ) { + vfio_user_iothread =iothread_create ("VFIO user" , errp); + }Quoted 2 lines
Why is a dedicated IOThread needed for VFIO user?
It seemed the best model for inbound message processing. Most messages
are replies, so the receiver will either signal a thread waiting the reply or report any errors from the server if there is no waiter. None of this requires the BQL.
Quoted 4 lines
If the message is a request - which currently only happens if device
DMA targets guest memory that wasn’t mmap()d by QEMU or if the ’secure-dma’ option is used - then the receiver can then acquire BQL so it can call the VFIO code to handle the request.
QEMU is generally event-driven and the APIs are designed for that style. Threads in QEMU are there for parallelism or resource control, everything else is event-driven.
It's not clear to me if there is a reason why the message processing must be done in a separate thread or whether it is just done this way because the code was written in multi-threaded style instead of event-driven style.
You mentioned other threads waiting for replies. Which threads are they?
Quoted 7 lines
Quoted 4 lines
Please use true. '1' is shorter but it's less obvious to the reader (I had to jump to the definition to check whether this field was bool or int).
I assume this is also true for the other boolean struct members
I’ve added.
Yes, please. QEMU uses bool and true/false.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
hw/vfio/user.c new file +5
-0+aio_bh_schedule_oneshot (iothread_get_aio_context (vfio_user_iothread), + vfio_user_cb, proxy); + +/* drop locks so the iothread can make progress */ +qemu_mutex_unlock_iothread ();
Quoted 8 lines
Quoted 3 lines
Why does the BQL needs to be dropped so
vfio_user_iothreadcan make progress?See above. Acquiring BQL by the iothread is rare, but I have to
handle the case where a disconnect is concurrent with an incoming request message that is waiting for the BQL. See the proxy state check after BQL is acquired invfio_user_recv()
Here is how this code looks when written using coroutines (this is from nbd/server.c):
static coroutine_fn void nbd_trip(void *opaque)
{
...
req = nbd_request_get(client);
ret = nbd_co_receive_request(req, &request, &local_err);
client->recv_coroutine = NULL; if (client->closing) {
/*
* The client may be closed when we are blocked in
* nbd_co_receive_request()
*/
goto done;
}It's the same check. The code is inverted: the server reads the next request using nbd_co_receive_request() (this coroutine function can yield while waiting for data on the socket).
This way the network communication code doesn't need to know how messages will by processed by the client or server. There is no need for if (isreply) { qemu_cond_signal(&reply->cv); } else { proxy->request(proxy->reqarg, buf, &reqfds); }. The callbacks and threads aren't hardcoded into the network communication code.
This goes back to the question earlier about why a dedicated thread is necessary here. I suggest writing the network communication code using coroutines. That way the code is easier to read (no callbacks or thread synchronization), there are fewer thread-safety issues to worry about, and users or management tools don't need to know about additional threads (e.g. CPU/NUMA affinity).
Stefan
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 11 lines · Stefan Hajnoczi
This way the network communication code doesn't need to know how messages will by processed by the client or server. There is no need for if (isreply) {
qemu_cond_signal(&reply->cv); } else {proxy->request(proxy->reqarg, buf, &reqfds); }. The callbacks and threads aren't hardcoded into the network communication code.
I fear we are talking past each other. The vfio-user protocol
is bi-directional. e.g., the client both sends requests to the server and receives requests from the server on the same socket. No matter what threading model we use, the receive algorithm will be:
read message header
if it’s a reply
schedule the thread waiting for the reply
else
run a callback to process the request
The only way I can see changing this is to establish two
uni-directional sockets: one for requests outbound to the server, and one for requests inbound from the server.
This is the reason I chose the iothread model. It can run
independently of any vCPU/main threads waiting for replies and of the callback thread. I did muddle this idea by having the iothread become a callback thread by grabbing BQL and running the callback inline when it receives a request from the server, but if you like a pure event driven model, I can make incoming requests kick a BH from the main loop. e.g.,
if it’s a reply
qemu_cond_signal(reply cv)
else
qemu_bh_schedule(proxy bh)
That would avoid disconnect having to handle the iothread
blocked on BQL.
Quoted 7 lines
This goes back to the question earlier about why a dedicated thread is necessary here. I suggest writing the network communication code using coroutines. That way the code is easier to read (no callbacks or thread synchronization), there are fewer thread-safety issues to worry about, and users or management tools don't need to know about additional threads (e.g. CPU/NUMA affinity).
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.
JJ
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 55 lines · John Johnson
Quoted 11 lines · Stefan Hajnoczi
This way the network communication code doesn't need to know how messages will by processed by the client or server. There is no need for if (isreply) {
qemu_cond_signal(&reply->cv); } else {proxy->request(proxy->reqarg, buf, &reqfds); }. The callbacks and threads aren't hardcoded into the network communication code.I fear we are talking past each other. The vfio-user protocol
is bi-directional. e.g., the client both sends requests to the server and receives requests from the server on the same socket. No matter what threading model we use, the receive algorithm will be:read message header
if it’s a reply
schedule the thread waiting for the reply
else
run a callback to process the requestThe only way I can see changing this is to establish two
uni-directional sockets: one for requests outbound to the server, and one for requests inbound from the server.This is the reason I chose the iothread model. It can run
independently of any vCPU/main threads waiting for replies and of the callback thread. I did muddle this idea by having the iothread become a callback thread by grabbing BQL and running the callback inline when it receives a request from the server, but if you like a pure event driven model, I can make incoming requests kick a BH from the main loop. e.g.,if it’s a reply
qemu_cond_signal(reply cv)
else
qemu_bh_schedule(proxy bh)That would avoid disconnect having to handle the iothread
blocked on BQL.Quoted 7 lines
This goes back to the question earlier about why a dedicated thread is necessary here. I suggest writing the network communication code using coroutines. That way the code is easier to read (no callbacks or thread synchronization), there are fewer thread-safety issues to worry about, and users or management tools don't need to know about additional threads (e.g. CPU/NUMA affinity).
Quoted 3 lines
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.
This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().
(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)
Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply = find_reply(...)
qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else
QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
if (pending_reqs_was_empty) {
qemu_coroutine_enter(process_request_co);
}The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.
Stefan
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 34 lines · Stefan Hajnoczi
Quoted 5 lines · John Johnson
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from
vfio_user_send_recv()orvfio_user_pci_process_req().(Although
vfio_user_pci_process_req()could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in casevfio_user_pci_process_req()yields while processing a request.)Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply = find_reply(...) qemu_coroutine_enter(reply->co) // instead of signalling reply->cv else QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }
Quoted 3 lines
The
pending_reqsqueue holds incoming requests that theprocess_request_cocoroutine processes.
How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller.
JJ
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 39 lines · John Johnson
Quoted 34 lines · Stefan Hajnoczi
Quoted 5 lines · John Johnson
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from
vfio_user_send_recv()orvfio_user_pci_process_req().(Although
vfio_user_pci_process_req()could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in casevfio_user_pci_process_req()yields while processing a request.)Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply = find_reply(...) qemu_coroutine_enter(reply->co) // instead of signalling reply->cv else QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }Quoted 3 lines
The
pending_reqsqueue holds incoming requests that theprocess_request_cocoroutine processes.
Quoted 4 lines
How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I thinkcoroutine_yield()returns to itscoroutine_enter()caller.
A vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.
For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:
Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);
BDRV_POLL_WHILE() boils down to a loop like this:
while ((cond)) {
aio_poll(ctx, true);
}I also want to check that I understand the scenarios in which the vfio-user communication code is used:
1. vhost-user-server
The vfio-user communication code should run in a given AioContext (it will be the main loop by default but maybe the user will be able to configure a specific IOThread in the future).
2. vCPU thread vfio-user clients
The vfio-user communication code is called from the vCPU thread where the proxy device executes. The MemoryRegion->read()/write() callbacks are synchronous, so the thread needs to wait for a vfio-user reply before it can return.
Is this what you had in mind?
Stefan
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
In reply to John JohnsonQuoted 62 lines
Quoted 43 lines · John Johnson
Quoted 35 lines · Stefan Hajnoczi
Quoted 5 lines · John Johnson
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from
vfio_user_send_recv()orvfio_user_pci_process_req().(Although
vfio_user_pci_process_req()could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in casevfio_user_pci_process_req()yields while processing a request.)Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply = find_reply(...) qemu_coroutine_enter(reply->co) // instead of signalling reply->cv else QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }Quoted 3 lines
The
pending_reqsqueue holds incoming requests that theprocess_request_cocoroutine processes.Quoted 4 lines
How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I thinkcoroutine_yield()returns to itscoroutine_enter()callerA vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.
For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:
Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);
Quoted 6 lines
BDRV_POLL_WHILE()boils down to a loop like this:while ((cond)) { aio_poll(ctx, true); }
I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.
Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.
Quoted 9 lines
I also want to check that I understand the scenarios in which the vfio-user communication code is used:
1. vhost-user-server
The vfio-user communication code should run in a given AioContext (it will be the main loop by default but maybe the user will be able to configure a specific IOThread in the future).
Jag would know more, but I believe it runs off the main loop.
Running it in an iothread doesn’t gain much, since it needs BQL to run the device emulation code.
Quoted 8 lines
2. vCPU thread vfio-user clients
The vfio-user communication code is called from the vCPU thread where the proxy device executes. The MemoryRegion->read()/write() callbacks are synchronous, so the thread needs to wait for a vfio-user reply before it can return.
Is this what you had in mind?
The client is also called from the main thread - the GET_*
messages from vfio_user_pci_realize() as well as MAP/DEMAP messages from guest address space change transactions. It is also called by the migration thread, which is a separate thread that does not run holding BQL.
JJ
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 29 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from
vfio_user_send_recv()orvfio_user_pci_process_req().(Although
vfio_user_pci_process_req()could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in casevfio_user_pci_process_req()yields while processing a request.)Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply = find_reply(...) qemu_coroutine_enter(reply->co) // instead of signalling reply->cv else QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }Quoted 3 lines
The
pending_reqsqueue holds incoming requests that theprocess_request_cocoroutine processes.Quoted 4 lines
How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I thinkcoroutine_yield()returns to itscoroutine_enter()callerA vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.
For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:
Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);Quoted 6 lines
BDRV_POLL_WHILE()boils down to a loop like this:while ((cond)) { aio_poll(ctx, true); }I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.
Quoted 4 lines
Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.
Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:
if it's a reply
reply = find_reply(...)
qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else
QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
if (pending_reqs_was_empty) {
qemu_coroutine_enter(process_request_co);
}The qemu_coroutine_enter(reply->co) call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signalling reply->cv in the existing patch.
Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call aio_poll() (which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.
Quoted 22 lines
Quoted 9 lines
I also want to check that I understand the scenarios in which the vfio-user communication code is used:
1. vhost-user-server
The vfio-user communication code should run in a given AioContext (it will be the main loop by default but maybe the user will be able to configure a specific IOThread in the future).
Jag would know more, but I believe it runs off the main loop.
Running it in an iothread doesn’t gain much, since it needs BQL to run the device emulation code.Quoted 7 lines
2. vCPU thread vfio-user clients
The vfio-user communication code is called from the vCPU thread where the proxy device executes. The MemoryRegion->read()/write() callbacks are synchronous, so the thread needs to wait for a vfio-user reply before it can return.
Quoted 7 lines
Quoted 1 line
Is this what you had in mind?
The client is also called from the main thread - the GET_*
messages fromvfio_user_pci_realize()as well as MAP/DEMAP messages from guest address space change transactions. It is also called by the migration thread, which is a separate thread that does not run holding BQL.
Thanks for mentioning those additional cases.
Stefan
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 81 lines · Stefan Hajnoczi
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 24 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from
vfio_user_send_recv()orvfio_user_pci_process_req().(Although
vfio_user_pci_process_req()could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in casevfio_user_pci_process_req()yields while processing a request.)Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply =find_reply(...)
qemu_coroutine_enter(reply->co)// instead of signallingreply->cv
elseQuoted 8 lines
QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }The
pending_reqsqueue holds incoming requests that theprocess_request_cocoroutine processes.Quoted 4 lines
How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I thinkcoroutine_yield()returns to itscoroutine_enter()callerA vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.
For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:
Coroutine *co =
qemu_coroutine_create(co_entry, &rwco);bdrv_coroutine_enter(blk_bs(blk), co);BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);Quoted 6 lines
BDRV_POLL_WHILE()boils down to a loop like this:while ((cond)) { aio_poll(ctx, true);}
I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.Quoted 4 lines
Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:
Quoted 9 lines
if it's a reply
reply = find_reply(...) qemu_coroutine_enter(reply->co) // instead of signalling reply->cv else QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }
Sorry, I was assuming when you said the coroutine will block withaio_poll(), you implied it would also read messages from the socket.
Quoted 5 lines
The
qemu_coroutine_enter(reply->co)call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signallingreply->cvin the existing patch.
Yes, the only difference is it would be woken on each message,
even though it doesn’t read them. Which is what I think you’re addressing below.
Quoted 6 lines
Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call
aio_poll()(which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.
Wouldn’t you need to acquire BQL twice for every vCPU reply: once to
run the receiver coroutine, and once when the vCPU thread wakes up and wants to return to the VFIO code. The migration thread would also add a BQL dependency, where it didn’t have one before.
Is your objection with using an iothread, or using a separate thread?
I can change to using qemu_thread_create() and running aio_poll() from the thread routine, instead of creating an iothread.
On a related subject:
Quoted 7 lines · Stefan Hajnoczi
Review
Quoted 2 lines
(quoted diff) +2
-0+ ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds, + &local_err);This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).
This shouldn’t block if the emulation process sends less than sizeof(msg)
bytes. qio_channel_readv() will eventually call recvmsg(), which only blocks a short read if MSG_WAITALL is set, and it’s not set in this case. recvmsg() will return the data available, and vfio_user_recv() will treat a short read as an error.
JJ
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 91 lines · John Johnson
Quoted 81 lines · Stefan Hajnoczi
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 24 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from
vfio_user_send_recv()orvfio_user_pci_process_req().(Although
vfio_user_pci_process_req()could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in casevfio_user_pci_process_req()yields while processing a request.)Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply =find_reply(...)
qemu_coroutine_enter(reply->co)// instead of signallingreply->cv
elseQuoted 8 lines
QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }The
pending_reqsqueue holds incoming requests that theprocess_request_cocoroutine processes.Quoted 4 lines
How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I thinkcoroutine_yield()returns to itscoroutine_enter()callerA vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.
For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:
Coroutine *co =
qemu_coroutine_create(co_entry, &rwco);bdrv_coroutine_enter(blk_bs(blk), co);BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);Quoted 6 lines
BDRV_POLL_WHILE()boils down to a loop like this:while ((cond)) { aio_poll(ctx, true);}
I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.Quoted 4 lines
Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:
Quoted 9 lines
if it's a reply
reply = find_reply(...) qemu_coroutine_enter(reply->co) // instead of signalling reply->cv else QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }
Quoted 2 lines
Sorry, I was assuming when you said the coroutine will block with
aio_poll(), you implied it would also read messages from the socket.
The vCPU thread calls aio_poll() outside the coroutine, similar to the block/block-backend.c:blk_prw() example I posted above:
Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);
(BDRV_POLL_WHILE() is a aio_poll() loop.)
The coroutine isn't aware of aio_poll(), it just yields when it needs to wait.
Quoted 17 lines
Quoted 5 lines
The
qemu_coroutine_enter(reply->co)call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signallingreply->cvin the existing patch.Yes, the only difference is it would be woken on each message,
even though it doesn’t read them. Which is what I think you’re addressing below.Quoted 6 lines
Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call
aio_poll()(which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.
Quoted 4 lines
Wouldn’t you need to acquire BQL twice for every vCPU reply: once to
run the receiver coroutine, and once when the vCPU thread wakes up and wants to return to the VFIO code. The migration thread would also add a BQL dependency, where it didn’t have one before.
If aio_poll() is used then the vCPU thread doesn't drop the BQL at all. The vCPU thread sends the message and waits for the reply while other BQL threads are locked out.
If a condvar or similar mechanism is used then the vCPU sends the message, drops the BQL, and waits on the condvar. The main loop thread runs the receiver coroutine and re-enters the coroutine, which signals the condvar. The vCPU then re-acquires the BQL.
Quoted 3 lines
Is your objection with using an iothread, or using a separate thread?
I can change to usingqemu_thread_create()and runningaio_poll()from the thread routine, instead of creating an iothread.
The vfio-user communication code shouldn't need to worry about threads or locks. The code can be written in terms of AioContext so the caller can use it from various environments without hardcoding details of the BQL or threads into the communication code. This makes it easier to understand and less tightly coupled.
I'll try to sketch how that could work:
The main concept is VFIOProxy, which has a QIOChannel (the socket connection) and its main API is:
void coroutine_fn vfio_user_co_send_recv(VFIOProxy *proxy,
VFIOUserHdr *msg, VFIOUserFDs *fds, int rsize, int flags);
There is also a request callback for processing incoming requests:
void coroutine_fn (*request)(void *opaque, char *buf,
VFIOUserFDs *fds);
The main loop thread can either use vfio_user_co_send_recv() from coroutine context or use this blocking wrapper:
typedef struct {
VFIOProxy *proxy;
VFIOUserHdr *msg;
VFIOUserFDs *fds;
int rsize;
int flags;
bool done;
} VFIOUserSendRecvData;
static void coroutine_fn vfu_send_recv_co(void *opaque)
{
VFIOUserSendRecvData *data = opaque;
vfio_user_co_send_recv(data->proxy, data->msg, data->fds,
data->rsize, data->flags);
data->done = true;
} /* A blocking version of vfio_user_co_send_recv() */ void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags)
{
VFIOUserSendRecvData data = {
.proxy = proxy,
.msg = msg,
.fds = fds,
.rsize = rsize,
.flags = flags,
};
Coroutine *co = qemu_coroutine_create(vfu_send_recv_co, &data);
qemu_coroutine_enter(co);
while (!data.done) {
aio_poll(proxy->ioc->ctx, true);
}}
The vCPU thread can use vfio_user_send_recv() if it wants, although the BQL will be held, preventing other threads from making progress. That can be avoided by writing a similar wrapper that uses a QemuSemaphore:
typedef struct {
VFIOProxy *proxy;
VFIOUserHdr *msg;
VFIOUserFDs *fds;
int rsize;
int flags;
QemuSemaphore sem;
} VFIOUserSendRecvData;
static void coroutine_fn vfu_send_recv_co(void *opaque)
{
VFIOUserSendRecvData *data = opaque;
vfio_user_co_send_recv(data->proxy, data->msg, data->fds,
data->rsize, data->flags);
qemu_sem_post(&data->sem);
} /*
* A blocking version of vfio_user_co_send_recv() that relies on
* another thread to run the event loop. This can be used from vCPU
* threads to avoid hogging the BQL.
*/
void vfio_user_vcpu_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags)
{
VFIOUserSendRecvData data = {
.proxy = proxy,
.msg = msg,
.fds = fds,
.rsize = rsize,
.flags = flags,
};
Coroutine *co = qemu_coroutine_create(vfu_vcpu_send_recv_co, &data);
qemu_sem_init(&data.sem, 0);
qemu_coroutine_enter(co);
qemu_mutex_unlock_iothread();
qemu_sem_wait(&data.sem);
qemu_mutex_lock_iothread();
qemu_sem_destroy(&data.sem);}
With vfio_user_vcpu_send_recv() the vCPU thread doesn't call aio_poll() itself but instead relies on the main loop thread to run the event loop.
By writing coroutines that run in proxy->ioc->ctx we keep the threading model and locking in the caller. The communication code isn't aware of or tied to specific threads. It's possible to drop proxy->lock because state is only changed from within the AioContext, not multiple threads that may run in parallel. I think this makes the communication code simpler and cleaner.
It's possible to use IOThreads with this approach: set the QIOChannel's AioContext to the IOThread AioContext. However, I don't think we can do this in the vhost-user server yet because QEMU's device models expect to run with the BQL and not in an IOThread.
I didn't go into detail about how vfio_user_co_send_recv() is implemented. Please let me know if you want me to share ideas about that, but it's what we've already discussed with a "receiver" coroutine that re-enters the reply coroutines or calls ->request(). A CoMutex is needed to around qio_channel_write_all() to ensure that coroutines sending messages don't interleave partial writes if the socket sndbuf is exhausted.
Quoted 13 lines
On a related subject:
Quoted 7 lines · Stefan Hajnoczi
Review
Quoted 2 lines
(quoted diff) +2
-0+ ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds, + &local_err);This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).
Quoted 4 lines
This shouldn’t block if the emulation process sends less than sizeof(msg)
bytes.qio_channel_readv()will eventually callrecvmsg(), which only blocks a short read ifMSG_WAITALLis set, and it’s not set in this case.recvmsg()will return the data available, andvfio_user_recv()will treat a short read as an error.
That's true but vfio_user_recv() can still block layer on: if only sizeof(msg) bytes are available and msg.size > sizeof(msg) then the second call blocks.
msgleft = msg.size - sizeof(msg);
if (msgleft != 0) {
ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err);I think either code should be non-blocking or it shouldn't be. Writing code that is partially non-blocking is asking for trouble because it's not obvious where it can block and misbehaving or malicious programs can cause it to block.
Stefan
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 133 lines · Stefan Hajnoczi
Quoted 91 lines · John Johnson
Quoted 81 lines · Stefan Hajnoczi
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 25 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from
vfio_user_send_recv()orvfio_user_pci_process_req().(Although
vfio_user_pci_process_req()could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in casevfio_user_pci_process_req()yields while processing a request.)Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply =find_reply(...)
qemu_coroutine_enter(reply->co)// instead of signallingreply->cv
else
QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);Quoted 7 lines
if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co);}
The
pending_reqsqueue holds incoming requests that theprocess_request_cocoroutine processes.Quoted 4 lines
How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I thinkcoroutine_yield()returns to itscoroutine_enter()callerA vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.
For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:
Coroutine *co =
qemu_coroutine_create(co_entry, &rwco);bdrv_coroutine_enter(blk_bs(blk), co);BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);Quoted 6 lines
BDRV_POLL_WHILE()boils down to a loop like this:while ((cond)) {
aio_poll(ctx, true);
}I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.Quoted 4 lines
Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:
Quoted 9 lines
if it's a reply
reply = find_reply(...) qemu_coroutine_enter(reply->co) // instead of signalling reply->cv else QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }Quoted 2 lines
Sorry, I was assuming when you said the coroutine will block with
aio_poll(), you implied it would also read messages from the socket.The vCPU thread calls
aio_poll()outside the coroutine, similar to the block/block-backend.c:blk_prw() example I posted above:Coroutine *co =
qemu_coroutine_create(co_entry, &rwco);bdrv_coroutine_enter(blk_bs(blk), co);BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);(
BDRV_POLL_WHILE()is aaio_poll()loop.)The coroutine isn't aware of
aio_poll(), it just yields when it needs to wait.Quoted 17 lines
Quoted 5 lines
The
qemu_coroutine_enter(reply->co)call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signallingreply->cvin the existing patch.Yes, the only difference is it would be woken on each message,
even though it doesn’t read them. Which is what I think you’re addressing below.Quoted 6 lines
Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call
aio_poll()(which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.Quoted 4 lines
Wouldn’t you need to acquire BQL twice for every vCPU reply: once to
run the receiver coroutine, and once when the vCPU thread wakes up and wants to return to the VFIO code. The migration thread would also add a BQL dependency, where it didn’t have one before.
Quoted 9 lines
If
aio_poll()is used then the vCPU thread doesn't drop the BQL at all. The vCPU thread sends the message and waits for the reply while other BQL threads are locked out.If a condvar or similar mechanism is used then the vCPU sends the message, drops the BQL, and waits on the condvar. The main loop thread runs the receiver coroutine and re-enters the coroutine, which signals the condvar. The vCPU then re-acquires the BQL.
I understand this. The point I was trying to make was you'd need
to acquire BQL twice for every reply: once by the main loop before it runs the receiver coroutine and once after the vCPU wakes up. That would seem to increase latency over the iothread model.
Quoted 108 lines
Quoted 3 lines
Is your objection with using an iothread, or using a separate thread?
I can change to usingqemu_thread_create()and runningaio_poll()from the thread routine, instead of creating an iothread.The vfio-user communication code shouldn't need to worry about threads or locks. The code can be written in terms of AioContext so the caller can use it from various environments without hardcoding details of the BQL or threads into the communication code. This makes it easier to understand and less tightly coupled.
I'll try to sketch how that could work:
The main concept is VFIOProxy, which has a QIOChannel (the socket connection) and its main API is:
void
coroutine_fnvfio_user_co_send_recv(VFIOProxy *proxy,VFIOUserHdr *msg, VFIOUserFDs *fds, int rsize, int flags);There is also a request callback for processing incoming requests:
void
coroutine_fn(*request)(void *opaque, char *buf,VFIOUserFDs *fds);The main loop thread can either use
vfio_user_co_send_recv()from coroutine context or use this blocking wrapper:typedef struct { VFIOProxy *proxy; VFIOUserHdr *msg; VFIOUserFDs *fds; int rsize; int flags; bool done;} VFIOUserSendRecvData;
static void
coroutine_fnvfu_send_recv_co(void *opaque){ VFIOUserSendRecvData *data = opaque; vfio_user_co_send_recv(data->proxy, data->msg, data->fds, data->rsize, data->flags); data->done = true;}
/* A blocking version of
vfio_user_co_send_recv()*/ void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags){ VFIOUserSendRecvData data = { .proxy = proxy, .msg = msg, .fds = fds, .rsize = rsize, .flags = flags, }; Coroutine *co = qemu_coroutine_create(vfu_send_recv_co, &data); qemu_coroutine_enter(co); while (!data.done) { aio_poll(proxy->ioc->ctx, true); }}
The vCPU thread can use
vfio_user_send_recv()if it wants, although the BQL will be held, preventing other threads from making progress. That can be avoided by writing a similar wrapper that uses a QemuSemaphore:typedef struct { VFIOProxy *proxy; VFIOUserHdr *msg; VFIOUserFDs *fds; int rsize; int flags; QemuSemaphore sem;} VFIOUserSendRecvData;
static void
coroutine_fnvfu_send_recv_co(void *opaque){ VFIOUserSendRecvData *data = opaque; vfio_user_co_send_recv(data->proxy, data->msg, data->fds, data->rsize, data->flags); qemu_sem_post(&data->sem);}
/*
* A blocking version ofvfio_user_co_send_recv()that relies on
* another thread to run the event loop. This can be used from vCPU
* threads to avoid hogging the BQL.
*/
void vfio_user_vcpu_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags){ VFIOUserSendRecvData data = { .proxy = proxy, .msg = msg, .fds = fds, .rsize = rsize, .flags = flags, }; Coroutine *co = qemu_coroutine_create(vfu_vcpu_send_recv_co, &data); qemu_sem_init(&data.sem, 0); qemu_coroutine_enter(co); qemu_mutex_unlock_iothread(); qemu_sem_wait(&data.sem); qemu_mutex_lock_iothread(); qemu_sem_destroy(&data.sem);
Quoted 5 lines
}
With
vfio_user_vcpu_send_recv()the vCPU thread doesn't callaio_poll()itself but instead relies on the main loop thread to run the event loop.
I think this means I need 2 send algorithms: one for when called
from the main loop, and another for when called outside the main loop (vCPU or migration). I can’t use the semaphore version from the main loop, since blocking the main loop would prevent the receiver routine from being scheduled, so I’d want to use aio_poll() there.
Some vfio_user calls can come from either place (e.g., realize
uses REGION_READ to read the device config space, and vCPU uses it on a guest load to the device), so I’d need to detect which thread I’m running in to choose the right sender.
Quoted 12 lines
By writing coroutines that run in
proxy->ioc->ctxwe keep the threading model and locking in the caller. The communication code isn't aware of or tied to specific threads. It's possible to dropproxy->lockbecause state is only changed from within the AioContext, not multiple threads that may run in parallel. I think this makes the communication code simpler and cleaner.It's possible to use IOThreads with this approach: set the QIOChannel's AioContext to the IOThread AioContext. However, I don't think we can do this in the vhost-user server yet because QEMU's device models expect to run with the BQL and not in an IOThread.
Quoted 8 lines
I didn't go into detail about how
vfio_user_co_send_recv()is implemented. Please let me know if you want me to share ideas about that, but it's what we've already discussed with a "receiver" coroutine that re-enters the reply coroutines or calls ->request(). A CoMutex is needed to aroundqio_channel_write_all()to ensure that coroutines sending messages don't interleave partial writes if the socket sndbuf is exhausted.
Here is where I questioned how coroutines work across threads.
When the reply waiter is not the main loop, would the receiver coroutine re-enter the reply coroutine or signal the condvar it is waiting on?
Quoted 22 lines
Quoted 13 lines
On a related subject:
Quoted 7 lines · Stefan Hajnoczi
Review
Quoted 2 lines
(quoted diff) +2
-0+ ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds, + &local_err);This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).
Quoted 4 lines
This shouldn’t block if the emulation process sends less than sizeof(msg)
bytes.qio_channel_readv()will eventually callrecvmsg(), which only blocks a short read ifMSG_WAITALLis set, and it’s not set in this case.recvmsg()will return the data available, andvfio_user_recv()will treat a short read as an error.That's true but
vfio_user_recv()can still block layer on: if only sizeof(msg) bytes are available and msg.size > sizeof(msg) then the second call blocks.
Quoted 9 lines
msgleft = msg.size - sizeof(msg);
if (msgleft != 0) { ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err);I think either code should be non-blocking or it shouldn't be. Writing code that is partially non-blocking is asking for trouble because it's not obvious where it can block and misbehaving or malicious programs can cause it to block.
I wonder if I should just go fully non-blocking, and have the
senders queue messages for the sending routine, and have the receiving routine either signal a reply waiter or schedule a request handling routine.
JJ
Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server
Quoted 143 lines · John Johnson
Quoted 133 lines · Stefan Hajnoczi
Quoted 91 lines · John Johnson
Quoted 81 lines · Stefan Hajnoczi
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 25 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from
vfio_user_send_recv()orvfio_user_pci_process_req().(Although
vfio_user_pci_process_req()could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in casevfio_user_pci_process_req()yields while processing a request.)Going back to what you mentioned above, the receiver coroutine does something like this:
if it's a reply
reply =find_reply(...)
qemu_coroutine_enter(reply->co)// instead of signallingreply->cv
else
QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);Quoted 7 lines
if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co);}
The
pending_reqsqueue holds incoming requests that theprocess_request_cocoroutine processes.Quoted 4 lines
How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I thinkcoroutine_yield()returns to itscoroutine_enter()callerA vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.
For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:
Coroutine *co =
qemu_coroutine_create(co_entry, &rwco);bdrv_coroutine_enter(blk_bs(blk), co);BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);Quoted 6 lines
BDRV_POLL_WHILE()boils down to a loop like this:while ((cond)) {
aio_poll(ctx, true);
}I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.Quoted 4 lines
Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:
Quoted 9 lines
if it's a reply
reply = find_reply(...) qemu_coroutine_enter(reply->co) // instead of signalling reply->cv else QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next); if (pending_reqs_was_empty) { qemu_coroutine_enter(process_request_co); }Quoted 2 lines
Sorry, I was assuming when you said the coroutine will block with
aio_poll(), you implied it would also read messages from the socket.The vCPU thread calls
aio_poll()outside the coroutine, similar to the block/block-backend.c:blk_prw() example I posted above:Coroutine *co =
qemu_coroutine_create(co_entry, &rwco);bdrv_coroutine_enter(blk_bs(blk), co);BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);(
BDRV_POLL_WHILE()is aaio_poll()loop.)The coroutine isn't aware of
aio_poll(), it just yields when it needs to wait.Quoted 17 lines
Quoted 5 lines
The
qemu_coroutine_enter(reply->co)call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signallingreply->cvin the existing patch.Yes, the only difference is it would be woken on each message,
even though it doesn’t read them. Which is what I think you’re addressing below.Quoted 6 lines
Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call
aio_poll()(which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.Quoted 4 lines
Wouldn’t you need to acquire BQL twice for every vCPU reply: once to
run the receiver coroutine, and once when the vCPU thread wakes up and wants to return to the VFIO code. The migration thread would also add a BQL dependency, where it didn’t have one before.Quoted 9 lines
If
aio_poll()is used then the vCPU thread doesn't drop the BQL at all. The vCPU thread sends the message and waits for the reply while other BQL threads are locked out.If a condvar or similar mechanism is used then the vCPU sends the message, drops the BQL, and waits on the condvar. The main loop thread runs the receiver coroutine and re-enters the coroutine, which signals the condvar. The vCPU then re-acquires the BQL.
Quoted 4 lines
I understand this. The point I was trying to make was you'd need
to acquire BQL twice for every reply: once by the main loop before it runs the receiver coroutine and once after the vCPU wakes up. That would seem to increase latency over the iothread model.
Yes, but if minimizing latency is critical then you can use the aio_poll() approach. It's fastest since it doesn't context switch or drop the BQL.
Regarding vfio-user performance in general, devices should use ioeventfd and/or mmap regions to avoid going through the VMM in performance-critical code paths.
Quoted 120 lines
Quoted 108 lines
Quoted 3 lines
Is your objection with using an iothread, or using a separate thread?
I can change to usingqemu_thread_create()and runningaio_poll()from the thread routine, instead of creating an iothread.The vfio-user communication code shouldn't need to worry about threads or locks. The code can be written in terms of AioContext so the caller can use it from various environments without hardcoding details of the BQL or threads into the communication code. This makes it easier to understand and less tightly coupled.
I'll try to sketch how that could work:
The main concept is VFIOProxy, which has a QIOChannel (the socket connection) and its main API is:
void
coroutine_fnvfio_user_co_send_recv(VFIOProxy *proxy,VFIOUserHdr *msg, VFIOUserFDs *fds, int rsize, int flags);There is also a request callback for processing incoming requests:
void
coroutine_fn(*request)(void *opaque, char *buf,VFIOUserFDs *fds);The main loop thread can either use
vfio_user_co_send_recv()from coroutine context or use this blocking wrapper:typedef struct { VFIOProxy *proxy; VFIOUserHdr *msg; VFIOUserFDs *fds; int rsize; int flags; bool done;} VFIOUserSendRecvData;
static void
coroutine_fnvfu_send_recv_co(void *opaque){ VFIOUserSendRecvData *data = opaque; vfio_user_co_send_recv(data->proxy, data->msg, data->fds, data->rsize, data->flags); data->done = true;}
/* A blocking version of
vfio_user_co_send_recv()*/ void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags){ VFIOUserSendRecvData data = { .proxy = proxy, .msg = msg, .fds = fds, .rsize = rsize, .flags = flags, }; Coroutine *co = qemu_coroutine_create(vfu_send_recv_co, &data); qemu_coroutine_enter(co); while (!data.done) { aio_poll(proxy->ioc->ctx, true); }}
The vCPU thread can use
vfio_user_send_recv()if it wants, although the BQL will be held, preventing other threads from making progress. That can be avoided by writing a similar wrapper that uses a QemuSemaphore:typedef struct { VFIOProxy *proxy; VFIOUserHdr *msg; VFIOUserFDs *fds; int rsize; int flags; QemuSemaphore sem;} VFIOUserSendRecvData;
static void
coroutine_fnvfu_send_recv_co(void *opaque){ VFIOUserSendRecvData *data = opaque; vfio_user_co_send_recv(data->proxy, data->msg, data->fds, data->rsize, data->flags); qemu_sem_post(&data->sem);}
/*
* A blocking version ofvfio_user_co_send_recv()that relies on
* another thread to run the event loop. This can be used from vCPU
* threads to avoid hogging the BQL.
*/
void vfio_user_vcpu_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags){ VFIOUserSendRecvData data = { .proxy = proxy, .msg = msg, .fds = fds, .rsize = rsize, .flags = flags, }; Coroutine *co = qemu_coroutine_create(vfu_vcpu_send_recv_co, &data); qemu_sem_init(&data.sem, 0); qemu_coroutine_enter(co); qemu_mutex_unlock_iothread(); qemu_sem_wait(&data.sem); qemu_mutex_lock_iothread(); qemu_sem_destroy(&data.sem);Quoted 5 lines
}
With
vfio_user_vcpu_send_recv()the vCPU thread doesn't callaio_poll()itself but instead relies on the main loop thread to run the event loop.I think this means I need 2 send algorithms: one for when called
from the main loop, and another for when called outside the main loop (vCPU or migration). I can’t use the semaphore version from the main loop, since blocking the main loop would prevent the receiver routine from being scheduled, so I’d want to useaio_poll()there.
Quoted 4 lines
Some
vfio_usercalls can come from either place (e.g., realize
usesREGION_READto read the device config space, and vCPU uses it on a guest load to the device), so I’d need to detect which thread I’m running in to choose the right sender.
The semaphore version is not really necessary, although it allows other BQL threads to make progress while we wait for a reply (but is slower, as you pointed out).
The aio_poll() approach can be used from either thread.
Quoted 21 lines
Quoted 12 lines
By writing coroutines that run in
proxy->ioc->ctxwe keep the threading model and locking in the caller. The communication code isn't aware of or tied to specific threads. It's possible to dropproxy->lockbecause state is only changed from within the AioContext, not multiple threads that may run in parallel. I think this makes the communication code simpler and cleaner.It's possible to use IOThreads with this approach: set the QIOChannel's AioContext to the IOThread AioContext. However, I don't think we can do this in the vhost-user server yet because QEMU's device models expect to run with the BQL and not in an IOThread.
Quoted 8 lines
I didn't go into detail about how
vfio_user_co_send_recv()is implemented. Please let me know if you want me to share ideas about that, but it's what we've already discussed with a "receiver" coroutine that re-enters the reply coroutines or calls ->request(). A CoMutex is needed to aroundqio_channel_write_all()to ensure that coroutines sending messages don't interleave partial writes if the socket sndbuf is exhausted.
Quoted 3 lines
Here is where I questioned how coroutines work across threads.
When the reply waiter is not the main loop, would the receiver coroutine re-enter the reply coroutine or signal the condvar it is waiting on?
The global AioContext (qemu_get_aio_context()) is shared by multiple thread and mutual exclusion is provided by the BQL. Any of the threads can run the coroutines and during a coroutine's lifetime it can run in multiple threads (but never simultaneously in multiple threads).
The vfu_send_recv_co() coroutine above starts in the vCPU thread but then yields and is re-entered from the main loop thread. The receiver coroutine re-enters vfu_send_recv_co() in the main loop thread where it calls qemu_sem_post() to wake up the vCPU thread.
(IOThread AioContexts are not shared by multiple threads, so in that case we don't need to worry about threads since everything is done in the IOThread.)
Quoted 32 lines
Quoted 22 lines
Quoted 13 lines
On a related subject:
Quoted 7 lines · Stefan Hajnoczi
Review
Quoted 2 lines
(quoted diff) +2
-0+ ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds, + &local_err);This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).
Quoted 4 lines
This shouldn’t block if the emulation process sends less than sizeof(msg)
bytes.qio_channel_readv()will eventually callrecvmsg(), which only blocks a short read ifMSG_WAITALLis set, and it’s not set in this case.recvmsg()will return the data available, andvfio_user_recv()will treat a short read as an error.That's true but
vfio_user_recv()can still block layer on: if only sizeof(msg) bytes are available and msg.size > sizeof(msg) then the second call blocks.Quoted 9 lines
msgleft = msg.size - sizeof(msg);
if (msgleft != 0) { ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err);I think either code should be non-blocking or it shouldn't be. Writing code that is partially non-blocking is asking for trouble because it's not obvious where it can block and misbehaving or malicious programs can cause it to block.
Quoted 4 lines
I wonder if I should just go fully non-blocking, and have the
senders queue messages for the sending routine, and have the receiving routine either signal a reply waiter or schedule a request handling routine.
That sounds good.
If messages are sent from coroutine context rather than plain functions then a separate sender isn't needed, they can use a CoMutex for mutual exclusion/queuing instead of an explicit send queue and sender routine.
Stefan
RFC PATCH v2 5/16 vfio-user: define VFIO Proxy and communication functions
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 62 +++++++++ hw/vfio/user.h | 8 ++ hw/vfio/pci.c | 6 + hw/vfio/user.c | 289 ++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 5 files changed, 366 insertions(+) create mode 100644 hw/vfio/user-protocol.h
hw/vfio/user-protocol.h new file +62 -0
@@ -0,0 +1,62 @@ +#ifndef VFIO_USER_PROTOCOL_H +#define VFIO_USER_PROTOCOL_H + +/* +* vfio protocol over a UNIX socket. +* +* Copyright © 2018, 2021 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL, version 2. See +* the COPYING file in the top-level directory. +* +* Each message has a standard header that describes the command +* being sent, which is almost always a VFIO ioctl(). +* +* The header may be followed by command-specific data, such as the +* region and offset info for read and write commands. +*/ + +typedef struct { +uint16_t id; +uint16_t command; +uint32_t size; +uint32_t flags; +uint32_t error_reply; +}VFIOUserHdr ; + +/* VFIOUserHdr commands */ +enum vfio_user_command { +VFIO_USER_VERSION =1 , +VFIO_USER_DMA_MAP =2 , +VFIO_USER_DMA_UNMAP =3 , +VFIO_USER_DEVICE_GET_INFO =4 , +VFIO_USER_DEVICE_GET_REGION_INFO =5 , +VFIO_USER_DEVICE_GET_REGION_IO_FDS =6 , +VFIO_USER_DEVICE_GET_IRQ_INFO =7 , +VFIO_USER_DEVICE_SET_IRQS =8 , +VFIO_USER_REGION_READ =9 , +VFIO_USER_REGION_WRITE =10 , +VFIO_USER_DMA_READ =11 , +VFIO_USER_DMA_WRITE =12 , +VFIO_USER_DEVICE_RESET =13 , +VFIO_USER_DIRTY_PAGES =14 , +VFIO_USER_MAX , +}; + +/* VFIOUserHdr flags */ +#define VFIO_USER_REQUEST 0x0 +#define VFIO_USER_REPLY 0x1 +#define VFIO_USER_TYPE 0xF + +#define VFIO_USER_NO_REPLY 0x10 +#define VFIO_USER_ERROR 0x20 + + +#define VFIO_USER_DEF_MAX_FDS 8 +#define VFIO_USER_MAX_MAX_FDS 16 + +#define VFIO_USER_DEF_MAX_XFER (1024 * 1024) +#define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) + + +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +8 -0
@@ -11,6 +11,8 @@* */ +#include "user-protocol.h" + typedef struct { int send_fds; int recv_fds;
@@ -19,6 +21,7 @@ typedef struct { typedef struct VFIOUserReply { QTAILQ_ENTRY(VFIOUserReply) next; + VFIOUserHdr *msg; VFIOUserFDs *fds; uint32_t rsize; uint32_t id;
@@ -62,5 +65,10 @@ typedef struct VFIOProxy {VFIOProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp);void vfio_user_disconnect (VFIOProxy *proxy); +void vfio_user_set_reqhandler (VFIODevice *vbasdev, +int (*handler)(void *opaque,char *buf, +VFIOUserFDs *fds), +void *reqarg); +void vfio_user_send_reply (VFIOProxy *proxy,char *buf,int ret);#endif /* VFIO_USER_H */
hw/vfio/pci.c +6 -0
@@ -3333,6 +3333,11 @@ type_init(register_vfio_pci_dev_type)* vfio-user routines. */ +static int vfio_user_pci_process_req (void *opaque,char *buf,VFIOUserFDs *fds) +{ +return 0 ; +} +/* * Emulated devices don't use host hot reset */
@@ -3386,6 +3391,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)return ; } vbasedev->proxy = proxy; +vfio_user_set_reqhandler (vbasedev, vfio_user_pci_process_req, vdev);if (udev->secure_dma) { proxy->flags |=VFIO_PROXY_SECURE ;
hw/vfio/user.c +289 -0
315 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -1888,6 +1888,7 @@ S: Supported F: docs/devel/vfio-user.rst F: hw/vfio/user.c F: hw/vfio/user.h +F: hw/vfio/user-protocol.h vhost M: Michael S. Tsirkin <[email protected]>
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 05/16] vfio-user: define VFIO Proxy and communication functions
hw/vfio/user.h typedef struct VFIOProxy { · line 68
Quoted 5 lines · Elena Ufimtseva
hw/vfio/user.h +1
-0@@ -62,5 +65,10 @@ typedef struct VFIOProxy {VFIOProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp);void vfio_user_disconnect (VFIOProxy *proxy); +void vfio_user_set_reqhandler(VFIODevice *vbasdev,
"vbasedev" for consistency?
hw/vfio/user.h typedef struct VFIOProxy { · line 71
Quoted 3 lines
hw/vfio/user.h +3
-0+int (*handler)(void *opaque,char *buf, +VFIOUserFDs *fds), + void *reqarg);
The handler callback is undocumented. What context does it run in, what do the arguments mean, and what should the function return? Please document it so it's easy for others to modify this code in the future without reverse-engineering the assumptions behind it.
Quoted 17 lines
hw/vfio/user.c +17
-0+void vfio_user_recv (void *opaque) +{ +VFIODevice *vbasedev = opaque; +VFIOProxy *proxy = vbasedev->proxy; +VFIOUserReply *reply =NULL ; + g_autofree int *fdp = NULL; +VFIOUserFDs reqfds = {0 ,0 , fdp }; +VFIOUserHdr msg; +struct iovec iov = { + .iov_base = &msg, + .iov_len =sizeof (msg), + }; +bool isreply; +int i, ret; +size_t msgleft, numfds =0 ; +char *data =NULL ; + g_autofree char *buf = NULL;
hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 162
Quoted 10 lines
hw/vfio/user.c +10
-0+Error *local_err =NULL ; + +qemu_mutex_lock (&proxy->lock); +if (proxy->state ==VFIO_PROXY_CLOSING ) { +qemu_mutex_unlock (&proxy->lock); +return ; + } + + ret =qio_channel_readv_full (proxy->ioc, &iov,1 , &fdp, &numfds, + &local_err);
This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).
While we're blocked in this function the proxy device cannot be hot-removed since proxy->lock is held.
It would more robust to use of the event loop to avoid blocking. There could be a per-connection receiver coroutine that calls qio_channel_readv_full_all_eof() (it yields the coroutine if reading would block).
Quoted 16 lines
hw/vfio/user.c +16
-0+/* +* Replies signal a waiter, requests get processed by vfio code +* that may assume the iothread lock is held. +*/ +if (isreply) { + reply->complete =1 ; +if (!reply->nowait) { +qemu_cond_signal (&reply->cv); + }else { +if (msg.flags &VFIO_USER_ERROR ) { +error_printf ("vfio_user_rcv error reply on async request " ); +error_printf ("command %x error %s\n" , msg.command, +strerror (msg.error_reply)); + } +/* just free it if no one is waiting */ + reply->nowait =0 ;
hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 286
Quoted 10 lines
hw/vfio/user.c +10
-0+if (proxy->last_nowait == reply) { + proxy->last_nowait =NULL ; + } +g_free (reply->msg); +QTAILQ_INSERT_HEAD (&proxy->free, reply, next); + } +qemu_mutex_unlock (&proxy->lock); + }else { +qemu_mutex_unlock (&proxy->lock); +qemu_mutex_lock_iothread ();
The fact that proxy->request() runs with the BQL suggests that VFIO communication should take place in the main event loop thread instead of a separate IOThread.
hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 293
Quoted 7 lines
hw/vfio/user.c +7
-0+/* +* make sure proxy wasn't closed while we waited +* checking state without holding the proxy lock is safe +* since it's only set to CLOSING when BQL is held +*/ +if (proxy->state !=VFIO_PROXY_CLOSING ) { + ret = proxy->request (proxy->reqarg, buf, &reqfds);
The request() callback in an earlier patch is a noop for the client implementation. Who frees passed fds?
Quoted 12 lines
hw/vfio/user.c +12
-0+if (ret <0 && !(msg.flags &VFIO_USER_NO_REPLY )) { +vfio_user_send_reply (proxy, buf, ret); + } + } +qemu_mutex_unlock_iothread (); + } +return ; + +fatal: +vfio_user_shutdown (proxy); + proxy->state =VFIO_PROXY_RECV_ERROR ; +
hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 315
Quoted 10 lines
hw/vfio/user.c +10
-0+err: +for (i =0 ; i < numfds; i++) { +close (fdp[i]); + } +if (reply !=NULL ) { +/* force an error to keep sending thread from hanging */ + reply->msg->flags |=VFIO_USER_ERROR ; + reply->msg->error_reply =EINVAL ; + reply->complete =1 ; +qemu_cond_signal (&reply->cv);
What about fd passing? The actual fds have been closed already in fdp[] but reply has a copy too.
What about the nowait case? If no one is waiting on reply->cv so this reply will be leaked?
Stefan
Re: [PATCH RFC v2 05/16] vfio-user: define VFIO Proxy and communication functions
Quoted 13 lines · Stefan Hajnoczi
hw/vfio/user.h typedef struct VFIOProxy { · line 68
Quoted 5 lines · Elena Ufimtseva
hw/vfio/user.h +1
-0@@ -62,5 +65,10 @@ typedef struct VFIOProxy {VFIOProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp);void vfio_user_disconnect (VFIOProxy *proxy); +void vfio_user_set_reqhandler(VFIODevice *vbasdev,"vbasedev" for consistency?
OK
Quoted 9 lines
hw/vfio/user.h typedef struct VFIOProxy { · line 71
Quoted 3 lines
hw/vfio/user.h +3
-0+int (*handler)(void *opaque,char *buf, +VFIOUserFDs *fds), + void *reqarg);The handler callback is undocumented. What context does it run in, what do the arguments mean, and what should the function return? Please document it so it's easy for others to modify this code in the future without reverse-engineering the assumptions behind it.
OK
Quoted 33 lines
Quoted 17 lines
hw/vfio/user.c +17
-0+void vfio_user_recv (void *opaque) +{ +VFIODevice *vbasedev = opaque; +VFIOProxy *proxy = vbasedev->proxy; +VFIOUserReply *reply =NULL ; + g_autofree int *fdp = NULL; +VFIOUserFDs reqfds = {0 ,0 , fdp }; +VFIOUserHdr msg; +struct iovec iov = { + .iov_base = &msg, + .iov_len =sizeof (msg), + }; +bool isreply; +int i, ret; +size_t msgleft, numfds =0 ; +char *data =NULL ; + g_autofree char *buf = NULL;hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 162
Quoted 10 lines
hw/vfio/user.c +10
-0+Error *local_err =NULL ; + +qemu_mutex_lock (&proxy->lock); +if (proxy->state ==VFIO_PROXY_CLOSING ) { +qemu_mutex_unlock (&proxy->lock); +return ; + } + + ret =qio_channel_readv_full (proxy->ioc, &iov,1 , &fdp, &numfds, + &local_err);This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).
Quoted 8 lines
While we're blocked in this function the proxy device cannot be hot-removed since
proxy->lockis held.It would more robust to use of the event loop to avoid blocking. There could be a per-connection receiver coroutine that calls
qio_channel_readv_full_all_eof()(it yields the coroutine if reading would block).
I thought the main loop uses BQL, which I don’t need for most
message processing. The blocking behavior can be fixed with FIONREAD beforehand to detect a message with fewer bytes than in a header.
Quoted 27 lines
Quoted 16 lines
hw/vfio/user.c +16
-0+/* +* Replies signal a waiter, requests get processed by vfio code +* that may assume the iothread lock is held. +*/ +if (isreply) { + reply->complete =1 ; +if (!reply->nowait) { +qemu_cond_signal (&reply->cv); + }else { +if (msg.flags &VFIO_USER_ERROR ) { +error_printf ("vfio_user_rcv error reply on async request " ); +error_printf ("command %x error %s\n" , msg.command, +strerror (msg.error_reply)); + } +/* just free it if no one is waiting */ + reply->nowait =0 ;hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 286
Quoted 10 lines
hw/vfio/user.c +10
-0+if (proxy->last_nowait == reply) { + proxy->last_nowait =NULL ; + } +g_free (reply->msg); +QTAILQ_INSERT_HEAD (&proxy->free, reply, next); + } +qemu_mutex_unlock (&proxy->lock); + }else { +qemu_mutex_unlock (&proxy->lock); +qemu_mutex_lock_iothread ();
Quoted 4 lines
The fact that
proxy->request()runs with the BQL suggests that VFIO communication should take place in the main event loop thread instead of a separate IOThread.
See the last reply. Using the main event loop optimizes the
least common case.
Quoted 11 lines
hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 293
Quoted 7 lines
hw/vfio/user.c +7
-0+/* +* make sure proxy wasn't closed while we waited +* checking state without holding the proxy lock is safe +* since it's only set to CLOSING when BQL is held +*/ +if (proxy->state !=VFIO_PROXY_CLOSING ) { + ret = proxy->request (proxy->reqarg, buf, &reqfds);The
request()callback in an earlier patch is a noop for the client implementation. Who frees passed fds?
Right now no server->client requests send fd’s, but I do need
a single point where they are consumed if an error is returned.
Quoted 17 lines
Quoted 17 lines
hw/vfio/user.c +17
-0+if (ret <0 && !(msg.flags &VFIO_USER_NO_REPLY )) { +vfio_user_send_reply (proxy, buf, ret); + } + } +qemu_mutex_unlock_iothread (); + } +return ; + +fatal: +vfio_user_shutdown (proxy); + proxy->state =VFIO_PROXY_RECV_ERROR ; + +err: +for (i =0 ; i < numfds; i++) { +close (fdp[i]); + } +if (reply !=NULL ) {
Quoted 9 lines
hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 315
Quoted 5 lines
hw/vfio/user.c +5
-0+/* force an error to keep sending thread from hanging */ + reply->msg->flags |=VFIO_USER_ERROR ; + reply->msg->error_reply =EINVAL ; + reply->complete =1 ; +qemu_cond_signal (&reply->cv);What about fd passing? The actual fds have been closed already in fdp[] but reply has a copy too.
If the sender gets an error, it won’t be using the fd’s. I
can zero reply->fds to make this clearer.
Quoted 3 lines
What about the nowait case? If no one is waiting on
reply->cvso this reply will be leaked?
This looks like a leak.
JJ
Re: [PATCH RFC v2 05/16] vfio-user: define VFIO Proxy and communication functions
Quoted 71 lines · John Johnson
Quoted 13 lines · Stefan Hajnoczi
hw/vfio/user.h typedef struct VFIOProxy { · line 68
Quoted 5 lines · Elena Ufimtseva
hw/vfio/user.h +1
-0@@ -62,5 +65,10 @@ typedef struct VFIOProxy {VFIOProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp);void vfio_user_disconnect (VFIOProxy *proxy); +void vfio_user_set_reqhandler(VFIODevice *vbasdev,"vbasedev" for consistency?
OK
Quoted 9 lines
hw/vfio/user.h typedef struct VFIOProxy { · line 71
Quoted 3 lines
hw/vfio/user.h +3
-0+int (*handler)(void *opaque,char *buf, +VFIOUserFDs *fds), + void *reqarg);The handler callback is undocumented. What context does it run in, what do the arguments mean, and what should the function return? Please document it so it's easy for others to modify this code in the future without reverse-engineering the assumptions behind it.
OK
Quoted 33 lines
Quoted 17 lines
hw/vfio/user.c +17
-0+void vfio_user_recv (void *opaque) +{ +VFIODevice *vbasedev = opaque; +VFIOProxy *proxy = vbasedev->proxy; +VFIOUserReply *reply =NULL ; + g_autofree int *fdp = NULL; +VFIOUserFDs reqfds = {0 ,0 , fdp }; +VFIOUserHdr msg; +struct iovec iov = { + .iov_base = &msg, + .iov_len =sizeof (msg), + }; +bool isreply; +int i, ret; +size_t msgleft, numfds =0 ; +char *data =NULL ; + g_autofree char *buf = NULL;hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 162
Quoted 10 lines
hw/vfio/user.c +10
-0+Error *local_err =NULL ; + +qemu_mutex_lock (&proxy->lock); +if (proxy->state ==VFIO_PROXY_CLOSING ) { +qemu_mutex_unlock (&proxy->lock); +return ; + } + + ret =qio_channel_readv_full (proxy->ioc, &iov,1 , &fdp, &numfds, + &local_err);This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).
Quoted 8 lines
While we're blocked in this function the proxy device cannot be hot-removed since
proxy->lockis held.It would more robust to use of the event loop to avoid blocking. There could be a per-connection receiver coroutine that calls
qio_channel_readv_full_all_eof()(it yields the coroutine if reading would block).
Quoted 3 lines
I thought the main loop uses BQL, which I don’t need for most
message processing. The blocking behavior can be fixed with FIONREAD beforehand to detect a message with fewer bytes than in a header.
It's I/O-bound work, exactly what the main loop was intended for.
I'm not sure the BQL can be avoided anyway:
- The vfio-user client runs under the BQL (a vCPU thread).
- The vfio-user server needs to hold the BQL since most QEMU device models assume they are running under the BQL.
The network communication code doesn't need to know about the BQL though. Event-driven code (code that runs in an AioContext) can rely on the fact that its callbacks only execute in the AioContext, i.e. in one thread at any given time.
The code probably doesn't need explicit BQL lock/unlock and can run safely in another IOThread if the user wishes (I would leave that up to the user, e.g. -device vfio-user-pci,iothread=iothread0, instead of creating a dedicated IOThread that is shared for all vfio-user communication). See nbd/server.c for an example of doing event-driven network I/O with coroutines.
Stefan
RFC PATCH v2 6/16 vfio-user: negotiate version with remote server
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 23 ++++ hw/vfio/user.h | 1 + hw/vfio/pci.c | 9 ++ hw/vfio/user.c | 267 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 300 insertions(+)
hw/vfio/user-protocol.h +23 -0
@@ -52,6 +52,29 @@ enum vfio_user_command {#define VFIO_USER_ERROR 0x20 +/* +* VFIO_USER_VERSION +*/ +typedef struct { +VFIOUserHdr hdr; +uint16_t major; +uint16_t minor; +char capabilities[]; +}VFIOUserVersion ; + +#define VFIO_USER_MAJOR_VER 0 +#define VFIO_USER_MINOR_VER 0 + +#define VFIO_USER_CAP "capabilities" + +/* "capabilities" members */ +#define VFIO_USER_CAP_MAX_FDS "max_msg_fds" +#define VFIO_USER_CAP_MAX_XFER "max_data_xfer_size" +#define VFIO_USER_CAP_MIGR "migration" + +/* "migration" member */ +#define VFIO_USER_CAP_PGSIZE "pgsize" +#define VFIO_USER_DEF_MAX_FDS 8#define VFIO_USER_MAX_MAX_FDS 16
hw/vfio/user.h +1 -0
@@ -70,5 +70,6 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, VFIOUserFDs *fds), void *reqarg);void vfio_user_send_reply (VFIOProxy *proxy,char *buf,int ret); +int vfio_user_validate_version (VFIODevice *vbasedev,Error **errp);#endif /* VFIO_USER_H */
hw/vfio/pci.c +9 -0
@@ -3397,6 +3397,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) proxy->flags |=VFIO_PROXY_SECURE ; } +vfio_user_validate_version (vbasedev, &err); +if (err !=NULL ) { +error_propagate (errp, err); + goto error; + } + vbasedev->name =g_strdup_printf ("VFIO user <%s>" , udev->sock_name); vbasedev->dev =DEVICE (vdev); vbasedev->fd =-1 ;
@@ -3404,6 +3410,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; +error: +vfio_user_disconnect (proxy); +error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name); }static void vfio_user_instance_finalize (Object *obj)
hw/vfio/user.c +267 -0
292 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 06/16] vfio-user: negotiate version with remote server
Quoted 18 lines · Elena Ufimtseva
hw/vfio/pci.c +6
-0@@ -3397,6 +3397,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) proxy->flags |=VFIO_PROXY_SECURE ; } +vfio_user_validate_version (vbasedev, &err); +if (err !=NULL ) { +error_propagate (errp, err); + goto error; + } + vbasedev->name =g_strdup_printf ("VFIO user <%s>" , udev->sock_name); vbasedev->dev =DEVICE (vdev); vbasedev->fd =-1 ;
hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3413
Quoted 4 lines
hw/vfio/pci.c +1
-0@@ -3404,6 +3410,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; +error:
Missing return before error label? We shouldn't disconnect in the success case.
Quoted 28 lines
hw/vfio/pci.c +2
-0+vfio_user_disconnect (proxy); +error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name); }static void vfio_user_instance_finalize (Object *obj)hw/vfio/user.c +7
-0@@ -23,9 +23,16 @@#include "io/channel-socket.h" #include "io/channel-util.h" #include "sysemu/iothread.h" +#include "qapi/qmp/qdict.h" +#include "qapi/qmp/qjson.h" +#include "qapi/qmp/qnull.h" +#include "qapi/qmp/qstring.h" +#include "qapi/qmp/qnum.h" #include "user.h" static uint64_t max_xfer_size =VFIO_USER_DEF_MAX_XFER ; +static uint64_t max_send_fds =VFIO_USER_DEF_MAX_FDS ; +static int wait_time =1000 ;/* wait 1 sec for replies */ static IOThread *vfio_user_iothread;static void vfio_user_shutdown (VFIOProxy *proxy);@@ -34,7 +41,14 @@ static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, VFIOUserFDs *fds);
hw/vfio/user.c static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, · line 51
Quoted 10 lines
hw/vfio/user.c +7
-0static void vfio_user_send (VFIOProxy *proxy,VFIOUserHdr *msg,VFIOUserFDs *fds); +static void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd, +uint32_t size,uint32_t flags); +static void vfio_user_send_recv (VFIOProxy *proxy,VFIOUserHdr *msg, +VFIOUserFDs *fds,int rsize,int flags); +/* vfio_user_send_recv flags */ +#define NOWAIT 0x1/* do not wait for reply */ +#define NOIOLOCK 0x2/* do not drop iolock */
Please use "BQL", it's a widely used term while "iolock" isn't used:
s/IOLOCK/BQL/
Quoted 11 lines
hw/vfio/user.c +4
-0/* * Functions called by main, CPU, or iothread threads @@ -333,6 +347,79 @@ static void vfio_user_cb(void *opaque)* Functions called by main or CPU threads */ +static void vfio_user_send_recv (VFIOProxy *proxy,VFIOUserHdr *msg, +VFIOUserFDs *fds,int rsize,int flags) +{ +VFIOUserReply *reply;
hw/vfio/user.c static void vfio_user_cb(void *opaque) · line 363
Quoted 10 lines
hw/vfio/user.c +10
-0+bool iolock =0 ; + +if (msg->flags &VFIO_USER_NO_REPLY ) { +error_printf ("vfio_user_send_recv on async message\n" ); +return ; + } + +/* +* We may block later, so use a per-proxy lock and let +* the iothreads run while we sleep unless told no to
s/no/not/
Quoted 19 lines
hw/vfio/user.c +19
-0+int vfio_user_validate_version (VFIODevice *vbasedev,Error **errp) +{ + g_autofree VFIOUserVersion *msgp; +GString *caps; +int size, caplen; + + caps =caps_json (); + caplen = caps->len +1 ; + size =sizeof (*msgp) + caplen; + msgp =g_malloc0 (size); + +vfio_user_request_msg (&msgp->hdr,VFIO_USER_VERSION , size,0 ); + msgp->major =VFIO_USER_MAJOR_VER ; + msgp->minor =VFIO_USER_MINOR_VER ; +memcpy (&msgp->capabilities, caps->str, caplen); +g_string_free (caps, true); + +vfio_user_send_recv (vbasedev->proxy, &msgp->hdr,NULL ,0 ,0 ); +if (msgp->hdr.flags &VFIO_USER_ERROR ) {
hw/vfio/user.c void vfio_user_disconnect(VFIOProxy *proxy) · line 711
Quoted 10 lines
hw/vfio/user.c +10
-0+error_setg_errno (errp, msgp->hdr.error_reply,"version reply" ); +return -1 ; + } + +if (msgp->major !=VFIO_USER_MAJOR_VER || + msgp->minor >VFIO_USER_MINOR_VER ) { +error_setg (errp,"incompatible server version" ); +return -1 ; + } +if (caps_check (msgp->minor, (char *)msgp +sizeof (*msgp), errp) !=0 ) {
The reply is untrusted so we cannot treat it as a NUL-terminated string yet. The final byte msgp->capabilities[] needs to be checked first.
Please be careful about input validation, I might miss something so it's best if you audit the patches too. QEMU must not trust the device emulation process and vice versa.
Quoted 8 lines
hw/vfio/user.c +5
-0+return -1 ; + } + +return 0 ; +}Signature · 1 line
2.25.1
Re: [PATCH RFC v2 06/16] vfio-user: negotiate version with remote server
Quoted 26 lines · Stefan Hajnoczi
Quoted 21 lines · Elena Ufimtseva
hw/vfio/pci.c +6
-0@@ -3397,6 +3397,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) proxy->flags |=VFIO_PROXY_SECURE ; } +vfio_user_validate_version (vbasedev, &err); +if (err !=NULL ) { +error_propagate (errp, err); + goto error; + } + vbasedev->name =g_strdup_printf ("VFIO user <%s>" , udev->sock_name); vbasedev->dev =DEVICE (vdev); vbasedev->fd =-1 ;@@ -3404,6 +3410,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops;
Quoted 5 lines
hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3413
Quoted 1 line
hw/vfio/pci.c +1
-0+error:Missing return before error label? We shouldn't disconnect in the success case.
The return ended up in a later patch.
Quoted 35 lines
Quoted 35 lines
hw/vfio/pci.c +2
-0+vfio_user_disconnect (proxy); +error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name); }static void vfio_user_instance_finalize (Object *obj)hw/vfio/user.c +11
-0@@ -23,9 +23,16 @@#include "io/channel-socket.h" #include "io/channel-util.h" #include "sysemu/iothread.h" +#include "qapi/qmp/qdict.h" +#include "qapi/qmp/qjson.h" +#include "qapi/qmp/qnull.h" +#include "qapi/qmp/qstring.h" +#include "qapi/qmp/qnum.h" #include "user.h" static uint64_t max_xfer_size =VFIO_USER_DEF_MAX_XFER ; +static uint64_t max_send_fds =VFIO_USER_DEF_MAX_FDS ; +static int wait_time =1000 ;/* wait 1 sec for replies */ static IOThread *vfio_user_iothread;static void vfio_user_shutdown (VFIOProxy *proxy);@@ -34,7 +41,14 @@ static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, VFIOUserFDs *fds);static void vfio_user_send (VFIOProxy *proxy,VFIOUserHdr *msg,VFIOUserFDs *fds); +static void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd, +uint32_t size,uint32_t flags); +static void vfio_user_send_recv (VFIOProxy *proxy,VFIOUserHdr *msg, +VFIOUserFDs *fds,int rsize,int flags);
Quoted 7 lines
hw/vfio/user.c static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, · line 51
Quoted 3 lines
hw/vfio/user.c +3
-0+/* vfio_user_send_recv flags */ +#define NOWAIT 0x1/* do not wait for reply */ +#define NOIOLOCK 0x2/* do not drop iolock */ Please use "BQL", it's a widely used term while "iolock" isn't used:
s/IOLOCK/BQL/
OK
Quoted 7 lines
Quoted 7 lines
hw/vfio/user.c +0
-0/* * Functions called by main, CPU, or iothread threads @@ -333,6 +347,79 @@ static void vfio_user_cb(void *opaque)* Functions called by main or CPU threads */
Quoted 16 lines
Quoted 4 lines
hw/vfio/user.c +4
-0+static void vfio_user_send_recv (VFIOProxy *proxy,VFIOUserHdr *msg, +VFIOUserFDs *fds,int rsize,int flags) +{ +VFIOUserReply *reply;hw/vfio/user.c static void vfio_user_cb(void *opaque) · line 363
Quoted 10 lines
hw/vfio/user.c +10
-0+bool iolock =0 ; + +if (msg->flags &VFIO_USER_NO_REPLY ) { +error_printf ("vfio_user_send_recv on async message\n" ); +return ; + } + +/* +* We may block later, so use a per-proxy lock and let +* the iothreads run while we sleep unless told no to s/no/not/
OK
Quoted 34 lines
Quoted 20 lines
hw/vfio/user.c +19
-0+int vfio_user_validate_version (VFIODevice *vbasedev,Error **errp) +{ + g_autofree VFIOUserVersion *msgp; +GString *caps; +int size, caplen; + + caps =caps_json (); + caplen = caps->len +1 ; + size =sizeof (*msgp) + caplen; + msgp =g_malloc0 (size); + +vfio_user_request_msg (&msgp->hdr,VFIO_USER_VERSION , size,0 ); + msgp->major =VFIO_USER_MAJOR_VER ; + msgp->minor =VFIO_USER_MINOR_VER ; +memcpy (&msgp->capabilities, caps->str, caplen); +g_string_free (caps, true); + +vfio_user_send_recv (vbasedev->proxy, &msgp->hdr,NULL ,0 ,0 ); +if (msgp->hdr.flags &VFIO_USER_ERROR ) {hw/vfio/user.c void vfio_user_disconnect(VFIOProxy *proxy) · line 711
Quoted 10 lines
hw/vfio/user.c +10
-0+error_setg_errno (errp, msgp->hdr.error_reply,"version reply" ); +return -1 ; + } + +if (msgp->major !=VFIO_USER_MAJOR_VER || + msgp->minor >VFIO_USER_MINOR_VER ) { +error_setg (errp,"incompatible server version" ); +return -1 ; + } +if (caps_check (msgp->minor, (char *)msgp +sizeof (*msgp), errp) !=0 ) {The reply is untrusted so we cannot treat it as a NUL-terminated string yet. The final byte msgp->capabilities[] needs to be checked first.
Quoted 4 lines
Please be careful about input validation, I might miss something so it's best if you audit the patches too. QEMU must not trust the device emulation process and vice versa.
This message is consumed by vfio-user, so I can check for valid
replies, but for most messages this checking will have to be done at in the VFIO common or bus-specific code, as vfio-user doens’t know valid data from invalid.
JJ
Quoted 7 lines
Quoted 7 lines
hw/vfio/user.c +5
-0+return -1 ; + } + +return 0 ; +}Signature · 1 line
2.25.1
Re: [PATCH RFC v2 06/16] vfio-user: negotiate version with remote server
Quoted 40 lines · John Johnson
Quoted 35 lines · Stefan Hajnoczi
Quoted 19 lines · Elena Ufimtseva
hw/vfio/user.c +19
-0+int vfio_user_validate_version (VFIODevice *vbasedev,Error **errp) +{ + g_autofree VFIOUserVersion *msgp; +GString *caps; +int size, caplen; + + caps =caps_json (); + caplen = caps->len +1 ; + size =sizeof (*msgp) + caplen; + msgp =g_malloc0 (size); + +vfio_user_request_msg (&msgp->hdr,VFIO_USER_VERSION , size,0 ); + msgp->major =VFIO_USER_MAJOR_VER ; + msgp->minor =VFIO_USER_MINOR_VER ; +memcpy (&msgp->capabilities, caps->str, caplen); +g_string_free (caps, true); + +vfio_user_send_recv (vbasedev->proxy, &msgp->hdr,NULL ,0 ,0 ); +if (msgp->hdr.flags &VFIO_USER_ERROR ) {hw/vfio/user.c void vfio_user_disconnect(VFIOProxy *proxy) · line 711
Quoted 10 lines
hw/vfio/user.c +10
-0+error_setg_errno (errp, msgp->hdr.error_reply,"version reply" ); +return -1 ; + } + +if (msgp->major !=VFIO_USER_MAJOR_VER || + msgp->minor >VFIO_USER_MINOR_VER ) { +error_setg (errp,"incompatible server version" ); +return -1 ; + } +if (caps_check (msgp->minor, (char *)msgp +sizeof (*msgp), errp) !=0 ) {The reply is untrusted so we cannot treat it as a NUL-terminated string yet. The final byte msgp->capabilities[] needs to be checked first.
Quoted 4 lines
Please be careful about input validation, I might miss something so it's best if you audit the patches too. QEMU must not trust the device emulation process and vice versa.
Quoted 4 lines
This message is consumed by vfio-user, so I can check for valid
replies, but for most messages this checking will have to be done at in the VFIO common or bus-specific code, as vfio-user doens’t know valid data from invalid.
Good point. Today's VFIO code trusts the replies because they come from the kernel. Now that they can come from an untrusted source they must be validated and common VFIO code may need to change its trust model.
Stefan
RFC PATCH v2 7/16 vfio-user: get device info
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 13 +++++++++++++ hw/vfio/user.h | 1 + hw/vfio/pci.c | 13 +++++++++++++ hw/vfio/user.c | 20 ++++++++++++++++++++ 4 files changed, 47 insertions(+)
hw/vfio/user-protocol.h +13 -0
@@ -82,4 +82,17 @@ typedef struct {#define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) +/* +* VFIO_USER_DEVICE_GET_INFO +* imported from struct_device_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t num_regions; +uint32_t num_irqs; +uint32_t cap_offset; +}VFIOUserDeviceInfo ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +1 -0
@@ -71,5 +71,6 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, void *reqarg);void vfio_user_send_reply (VFIOProxy *proxy,char *buf,int ret);int vfio_user_validate_version (VFIODevice *vbasedev,Error **errp); +int vfio_user_get_info (VFIODevice *vbasedev);#endif /* VFIO_USER_H */
hw/vfio/pci.c +13 -0
@@ -3369,6 +3369,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)VFIODevice *vbasedev = &vdev->vbasedev;SocketAddress addr;VFIOProxy *proxy; +int ret;Error *err =NULL ;/*
@@ -3410,6 +3411,18 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; + ret =vfio_user_get_info (&vdev->vbasedev); +if (ret) { +error_setg_errno (errp, -ret,"get info failure" ); + goto error; + } + +vfio_populate_device (vdev, &err); +if (err) { +error_propagate (errp, err); + goto error; + } + error:vfio_user_disconnect (proxy);error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name);
hw/vfio/user.c +20 -0
@@ -714,3 +714,23 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp)return 0 ; } + +int vfio_user_get_info (VFIODevice *vbasedev) +{ +VFIOUserDeviceInfo msg; + +memset (&msg,0 ,sizeof (msg)); +vfio_user_request_msg (&msg.hdr,VFIO_USER_DEVICE_GET_INFO ,sizeof (msg),0 ); + msg.argsz =sizeof (struct vfio_device_info ); + +vfio_user_send_recv (vbasedev->proxy, &msg.hdr,NULL ,0 ,0 ); +if (msg.hdr.flags &VFIO_USER_ERROR ) { +return -msg.hdr.error_reply; + } + + vbasedev->num_irqs = msg.num_irqs; + vbasedev->num_regions = msg.num_regions; + vbasedev->flags = msg.flags; + vbasedev->reset_works = !!(msg.flags &VFIO_DEVICE_FLAGS_RESET ); +return 0 ; +}
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 07/16] vfio-user: get device info
Quoted 7 lines · Elena Ufimtseva
hw/vfio/user.c +7
-0+int vfio_user_get_info (VFIODevice *vbasedev) +{ +VFIOUserDeviceInfo msg; + +memset (&msg,0 ,sizeof (msg)); +vfio_user_request_msg (&msg.hdr,VFIO_USER_DEVICE_GET_INFO ,sizeof (msg),0 ); + msg.argsz =sizeof (struct vfio_device_info );
hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 734
Quoted 10 lines
hw/vfio/user.c +10
-0+ +vfio_user_send_recv (vbasedev->proxy, &msg.hdr,NULL ,0 ,0 ); +if (msg.hdr.flags &VFIO_USER_ERROR ) { +return -msg.hdr.error_reply; + } + + vbasedev->num_irqs = msg.num_irqs; + vbasedev->num_regions = msg.num_regions; + vbasedev->flags = msg.flags; + vbasedev->reset_works = !!(msg.flags &VFIO_DEVICE_FLAGS_RESET );
No input validation. I haven't checked what happens when num_irqs, num_regions, or flags are bogus but it's a little concerning. Unlike kernel VFIO, we do not trust these values.
Stefan
Re: [PATCH RFC v2 07/16] vfio-user: get device info
Quoted 18 lines · Stefan Hajnoczi
Quoted 13 lines · Elena Ufimtseva
hw/vfio/user.c +13
-0+int vfio_user_get_info (VFIODevice *vbasedev) +{ +VFIOUserDeviceInfo msg; + +memset (&msg,0 ,sizeof (msg)); +vfio_user_request_msg (&msg.hdr,VFIO_USER_DEVICE_GET_INFO ,sizeof (msg),0 ); + msg.argsz =sizeof (struct vfio_device_info ); + +vfio_user_send_recv (vbasedev->proxy, &msg.hdr,NULL ,0 ,0 ); +if (msg.hdr.flags &VFIO_USER_ERROR ) { +return -msg.hdr.error_reply; + } +
Quoted 9 lines
hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 734
Quoted 4 lines
hw/vfio/user.c +4
-0+ vbasedev->num_irqs = msg.num_irqs; + vbasedev->num_regions = msg.num_regions; + vbasedev->flags = msg.flags; + vbasedev->reset_works = !!(msg.flags &VFIO_DEVICE_FLAGS_RESET );No input validation. I haven't checked what happens when
num_irqs,num_regions, or flags are bogus but it's a little concerning. Unlike kernel VFIO, we do not trust these values.
As in the last reply, vfio-user doesn’t know valid values
from invalid, so I need to re-work this so the PCI-specific code that calls vfio-user_get_info() can test for invalid values.
JJ
Re: [PATCH RFC v2 07/16] vfio-user: get device info
Quoted 28 lines · John Johnson
Quoted 18 lines · Stefan Hajnoczi
Quoted 13 lines · Elena Ufimtseva
hw/vfio/user.c +13
-0+int vfio_user_get_info (VFIODevice *vbasedev) +{ +VFIOUserDeviceInfo msg; + +memset (&msg,0 ,sizeof (msg)); +vfio_user_request_msg (&msg.hdr,VFIO_USER_DEVICE_GET_INFO ,sizeof (msg),0 ); + msg.argsz =sizeof (struct vfio_device_info ); + +vfio_user_send_recv (vbasedev->proxy, &msg.hdr,NULL ,0 ,0 ); +if (msg.hdr.flags &VFIO_USER_ERROR ) { +return -msg.hdr.error_reply; + } +Quoted 9 lines
hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 734
Quoted 4 lines
hw/vfio/user.c +4
-0+ vbasedev->num_irqs = msg.num_irqs; + vbasedev->num_regions = msg.num_regions; + vbasedev->flags = msg.flags; + vbasedev->reset_works = !!(msg.flags &VFIO_DEVICE_FLAGS_RESET );No input validation. I haven't checked what happens when
num_irqs,num_regions, or flags are bogus but it's a little concerning. Unlike kernel VFIO, we do not trust these values.
Quoted 3 lines
As in the last reply, vfio-user doesn’t know valid values
from invalid, so I need to re-work this so the PCI-specific code that calls vfio-user_get_info() can test for invalid values.
Sounds good. I won't look further for missing input validation in the VFIO message contents in this revision of the patch series. Once you're happy with input validation I'll look at the code from this angle again.
Stefan
RFC PATCH v2 8/16 vfio-user: get region info
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 14 +++++++ hw/vfio/user.h | 2 + include/hw/vfio/vfio-common.h | 3 ++ hw/vfio/common.c | 76 ++++++++++++++++++++++++++++++++++- hw/vfio/user.c | 33 +++++++++++++++ 5 files changed, 126 insertions(+), 2 deletions(-)
hw/vfio/user-protocol.h +14 -0
@@ -95,4 +95,18 @@ typedef struct { uint32_t cap_offset; } VFIOUserDeviceInfo; +/* +* VFIO_USER_DEVICE_GET_REGION_INFO +* imported from struct_vfio_region_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t cap_offset; +uint64_t size; +uint64_t offset; +}VFIOUserRegionInfo ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +2 -0
@@ -72,5 +72,7 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev,void vfio_user_send_reply (VFIOProxy *proxy,char *buf,int ret);int vfio_user_validate_version (VFIODevice *vbasedev,Error **errp);int vfio_user_get_info (VFIODevice *vbasedev); +int vfio_user_get_region_info (VFIODevice *vbasedev,int index, +struct vfio_region_info *info,VFIOUserFDs *fds);#endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +3 -0
@@ -56,6 +56,7 @@ typedef struct VFIORegion {uint32_t nr_mmaps;VFIOMmap *mmaps;uint8_t nr;/* cache the region number for debug */ +int remfd;/* fd if exported from remote process */ } VFIORegion; typedef struct VFIOMigration {
@@ -145,6 +146,8 @@ typedef struct VFIODevice {Error *migration_blocker;OnOffAuto pre_copy_dirty_page_tracking;VFIOProxy *proxy; +struct vfio_region_info **regions; +int *regfds; } VFIODevice;struct VFIODeviceOps {
hw/vfio/common.c +74 -2
136 lines not on this page · raw message
hw/vfio/user.c +33 -0
@@ -734,3 +734,36 @@ int vfio_user_get_info(VFIODevice *vbasedev) vbasedev->reset_works = !!(msg.flags &VFIO_DEVICE_FLAGS_RESET );return 0 ; } + +int vfio_user_get_region_info (VFIODevice *vbasedev,int index, +struct vfio_region_info *info,VFIOUserFDs *fds) +{ + g_autofree VFIOUserRegionInfo *msgp = NULL; +int size; + +/* data returned can be larger than vfio_region_info */ +if (info->argsz <sizeof (*info)) { +error_printf ("vfio_user_get_region_info argsz too small\n" ); +return -EINVAL ; + } +if (fds !=NULL && fds->send_fds !=0 ) { +error_printf ("vfio_user_get_region_info can't send FDs\n" ); +return -EINVAL ; + } + + size = info->argsz +sizeof (VFIOUserHdr); + msgp =g_malloc0 (size); + +vfio_user_request_msg (&msgp->hdr,VFIO_USER_DEVICE_GET_REGION_INFO , +sizeof (*msgp),0 ); + msgp->argsz = info->argsz; + msgp->index = info->index; + +vfio_user_send_recv (vbasedev->proxy, &msgp->hdr, fds, size,0 ); +if (msgp->hdr.flags &VFIO_USER_ERROR ) { +return -msgp->hdr.error_reply; + } + +memcpy (info, &msgp->argsz, info->argsz); +return 0 ; +}
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 08/16] vfio-user: get region info
hw/vfio/common.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 1524
Quoted 11 lines · Elena Ufimtseva
hw/vfio/common.c +7
-0@@ -1514,6 +1515,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,return true; } +static int vfio_get_region_info_remfd (VFIODevice *vbasedev,int index) +{ +struct vfio_region_info *info; + +if (vbasedev->regions ==NULL || vbasedev->regions[index] ==NULL ) { +vfio_get_region_info (vbasedev, index, &info); + }
Maybe this will be called from other places in the future, but the vfio_region_setup() caller below already invoked vfio_get_region_info() so I'm not sure it's necessary to do this again?
Perhaps add an int *remfd argument to vfio_get_region_info(). That way vfio_get_region_info_remfd() isn't necessary.
Quoted 12 lines
hw/vfio/common.c +8
-0@@ -2410,6 +2442,24 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) {size_t argsz =sizeof (struct vfio_region_info ); +int fd =-1 ; +int ret; + +/* create region cache */ +if (vbasedev->regions ==NULL ) { + vbasedev->regions = g_new0(struct vfio_region_info *, + vbasedev->num_regions); +if (vbasedev->proxy !=NULL ) {
hw/vfio/common.c int vfio_get_region_info(VFIODevice *vbasedev, int index, · line 2462
Quoted 10 lines
hw/vfio/common.c +10
-0+ vbasedev->regfds =g_new0 (int, vbasedev->num_regions); + } + } +/* check cache */ +if (vbasedev->regions[index] !=NULL ) { + *info =g_malloc0 (vbasedev->regions[index]->argsz); +memcpy (*info, vbasedev->regions[index], + vbasedev->regions[index]->argsz); +return 0 ; + }
Why is it necessary to introduce a cache? Is it to avoid passing the same fd multiple times?
Quoted 28 lines
hw/vfio/common.c +13
-1*info =g_malloc0 (argsz);@@ -2417,7 +2467,17 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, retry: (*info)->argsz = argsz; -if (ioctl (vbasedev->fd,VFIO_DEVICE_GET_REGION_INFO , *info)) { +if (vbasedev->proxy !=NULL ) { +VFIOUserFDs fds = {0 ,1 , &fd}; + + ret =vfio_user_get_region_info (vbasedev, index, *info, &fds); + }else { + ret =ioctl (vbasedev->fd,VFIO_DEVICE_GET_REGION_INFO , *info); +if (ret <0 ) { + ret = -errno; + } + } +if (ret !=0 ) {g_free (*info); *info =NULL ;return -errno;@@ -2426,10 +2486,22 @@ retry:if ((*info)->argsz > argsz) { argsz = (*info)->argsz; *info =g_realloc (*info, argsz); +if (fd !=-1 ) { +close (fd);
hw/vfio/common.c retry: · line 2500
Quoted 10 lines
hw/vfio/common.c +6
-0+ fd =-1 ; + } goto retry; } +/* fill cache */ + vbasedev->regions[index] =g_malloc0 (argsz); +memcpy (vbasedev->regions[index], *info, argsz); + *vbasedev->regions[index] = **info;
The previous line already copied the contents of *info. What is the purpose of this assignment?
Quoted 12 lines
hw/vfio/common.c +4
-0+if (vbasedev->regfds !=NULL ) { + vbasedev->regfds[index] = fd; + } +return 0 ; }hw/vfio/user.c +0
-0
hw/vfio/user.c int vfio_user_get_info(VFIODevice *vbasedev) · line 742
Quoted 9 lines
hw/vfio/user.c +6
-0@@ -734,3 +734,36 @@ int vfio_user_get_info(VFIODevice *vbasedev) vbasedev->reset_works = !!(msg.flags &VFIO_DEVICE_FLAGS_RESET );return 0 ; } + +int vfio_user_get_region_info (VFIODevice *vbasedev,int index, +struct vfio_region_info *info,VFIOUserFDs *fds) +{ + g_autofree VFIOUserRegionInfo *msgp = NULL; +int size;
Please use uint32_t size instead of int size to prevent possible signedness issues:
- VFIOUserRegionInfo->argsz is uint32_t.
- sizeof(VFIOUserHdr) is size_t.
- The vfio_user_request_msg() size argument is uint32_t.
Re: [PATCH RFC v2 08/16] vfio-user: get region info
Quoted 21 lines · Stefan Hajnoczi
hw/vfio/common.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 1524
Quoted 11 lines · Elena Ufimtseva
hw/vfio/common.c +7
-0@@ -1514,6 +1515,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,return true; } +static int vfio_get_region_info_remfd (VFIODevice *vbasedev,int index) +{ +struct vfio_region_info *info; + +if (vbasedev->regions ==NULL || vbasedev->regions[index] ==NULL ) { +vfio_get_region_info (vbasedev, index, &info); + }Maybe this will be called from other places in the future, but the
vfio_region_setup()caller below already invokedvfio_get_region_info()so I'm not sure it's necessary to do this again?
Quoted 3 lines
Perhaps add an int *remfd argument to
vfio_get_region_info(). That wayvfio_get_region_info_remfd()isn't necessary.
I think they could be combined, but the region capabilities are
retrieved with separate calls to vfio_get_region_info_cap(), so I followed that precedent.
Quoted 17 lines
Quoted 17 lines
hw/vfio/common.c +13
-0@@ -2410,6 +2442,24 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) {size_t argsz =sizeof (struct vfio_region_info ); +int fd =-1 ; +int ret; + +/* create region cache */ +if (vbasedev->regions ==NULL ) { + vbasedev->regions = g_new0(struct vfio_region_info *, + vbasedev->num_regions); +if (vbasedev->proxy !=NULL ) { + vbasedev->regfds =g_new0 (int, vbasedev->num_regions); + } + } +/* check cache */ +if (vbasedev->regions[index] !=NULL ) {
Quoted 9 lines
hw/vfio/common.c int vfio_get_region_info(VFIODevice *vbasedev, int index, · line 2462
Quoted 5 lines
hw/vfio/common.c +5
-0+ *info =g_malloc0 (vbasedev->regions[index]->argsz); +memcpy (*info, vbasedev->regions[index], + vbasedev->regions[index]->argsz); +return 0 ; + }Why is it necessary to introduce a cache? Is it to avoid passing the same fd multiple times?
Yes. For polled regions, the server returns an FD with each
message, so there would be an FD leak if I didn’t check that the region hadn’t already returned one. Since I had to cache the FD anyway, I cached the whole struct.
Quoted 34 lines
Quoted 34 lines
hw/vfio/common.c +0
-0*info =g_malloc0 (argsz);@@ -2417,7 +2467,17 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index,retry:
(*info)->argsz = argsz;hw/vfio/common.c +15
-1-if (ioctl (vbasedev->fd,VFIO_DEVICE_GET_REGION_INFO , *info)) { +if (vbasedev->proxy !=NULL ) { +VFIOUserFDs fds = {0 ,1 , &fd}; + + ret =vfio_user_get_region_info (vbasedev, index, *info, &fds); + }else { + ret =ioctl (vbasedev->fd,VFIO_DEVICE_GET_REGION_INFO , *info); +if (ret <0 ) { + ret = -errno; + } + } +if (ret !=0 ) {g_free (*info); *info =NULL ;return -errno;@@ -2426,10 +2486,22 @@ retry:if ((*info)->argsz > argsz) { argsz = (*info)->argsz; *info =g_realloc (*info, argsz); +if (fd !=-1 ) { +close (fd); + fd =-1 ; + } goto retry; }
Quoted 8 lines
hw/vfio/common.c retry: · line 2500
Quoted 4 lines
hw/vfio/common.c +4
-0+/* fill cache */ + vbasedev->regions[index] =g_malloc0 (argsz); +memcpy (vbasedev->regions[index], *info, argsz); + *vbasedev->regions[index] = **info;The previous line already copied the contents of *info. What is the purpose of this assignment?
That might be a mis-merge. The struct assignment was a bug
fixed several revs ago with the memcpy() call.
Quoted 19 lines
Quoted 19 lines
hw/vfio/common.c +4
-0+if (vbasedev->regfds !=NULL ) { + vbasedev->regfds[index] = fd; + } +return 0 ; }hw/vfio/user.c +4
-0@@ -734,3 +734,36 @@ int vfio_user_get_info(VFIODevice *vbasedev) vbasedev->reset_works = !!(msg.flags &VFIO_DEVICE_FLAGS_RESET );return 0 ; } + +int vfio_user_get_region_info (VFIODevice *vbasedev,int index, +struct vfio_region_info *info,VFIOUserFDs *fds) +{
Quoted 8 lines
hw/vfio/user.c int vfio_user_get_info(VFIODevice *vbasedev) · line 742
Quoted 2 lines
hw/vfio/user.c +2
-0+ g_autofree VFIOUserRegionInfo *msgp = NULL; +int size;Please use
uint32_tsize instead of int size to prevent possible signedness issues:
-VFIOUserRegionInfo->argszisuint32_t.
- sizeof(VFIOUserHdr) issize_t.
- Thevfio_user_request_msg()size argument isuint32_t.
OK
JJ
Re: [PATCH RFC v2 08/16] vfio-user: get region info
Quoted 57 lines · John Johnson
Quoted 21 lines · Stefan Hajnoczi
hw/vfio/common.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 1524
Quoted 11 lines · Elena Ufimtseva
hw/vfio/common.c +7
-0@@ -1514,6 +1515,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,return true; } +static int vfio_get_region_info_remfd (VFIODevice *vbasedev,int index) +{ +struct vfio_region_info *info; + +if (vbasedev->regions ==NULL || vbasedev->regions[index] ==NULL ) { +vfio_get_region_info (vbasedev, index, &info); + }Maybe this will be called from other places in the future, but the
vfio_region_setup()caller below already invokedvfio_get_region_info()so I'm not sure it's necessary to do this again?Quoted 3 lines
Perhaps add an int *remfd argument to
vfio_get_region_info(). That wayvfio_get_region_info_remfd()isn't necessary.I think they could be combined, but the region capabilities are
retrieved with separate calls tovfio_get_region_info_cap(), so I followed that precedent.Quoted 17 lines
Quoted 17 lines
hw/vfio/common.c +13
-0@@ -2410,6 +2442,24 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) {size_t argsz =sizeof (struct vfio_region_info ); +int fd =-1 ; +int ret; + +/* create region cache */ +if (vbasedev->regions ==NULL ) { + vbasedev->regions = g_new0(struct vfio_region_info *, + vbasedev->num_regions); +if (vbasedev->proxy !=NULL ) { + vbasedev->regfds =g_new0 (int, vbasedev->num_regions); + } + } +/* check cache */ +if (vbasedev->regions[index] !=NULL ) {Quoted 9 lines
hw/vfio/common.c int vfio_get_region_info(VFIODevice *vbasedev, int index, · line 2462
Quoted 5 lines
hw/vfio/common.c +5
-0+ *info =g_malloc0 (vbasedev->regions[index]->argsz); +memcpy (*info, vbasedev->regions[index], + vbasedev->regions[index]->argsz); +return 0 ; + }Why is it necessary to introduce a cache? Is it to avoid passing the same fd multiple times?
Quoted 4 lines
Yes. For polled regions, the server returns an FD with each
message, so there would be an FD leak if I didn’t check that the region hadn’t already returned one. Since I had to cache the FD anyway, I cached the whole struct.
If vfio_get_region_info() takes an int *fd argument then fd ownership becomes explicit and the need for the cache falls away. Maybe Alex has a preference for how to structure the code to track per-region fds.
Stefan
RFC PATCH v2 9/16 vfio-user: region read/write
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 12 ++++++++++++ hw/vfio/user.h | 4 ++++ hw/vfio/common.c | 16 +++++++++++++-- hw/vfio/user.c | 43 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 73 insertions(+), 2 deletions(-)
hw/vfio/user-protocol.h +12 -0
@@ -109,4 +109,16 @@ typedef struct { uint64_t offset; } VFIOUserRegionInfo; +/* +* VFIO_USER_REGION_READ +* VFIO_USER_REGION_WRITE +*/ +typedef struct { +VFIOUserHdr hdr; +uint64_t offset; +uint32_t region; +uint32_t count; +char data[]; +}VFIOUserRegionRW ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +4 -0
@@ -74,5 +74,9 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);int vfio_user_get_info (VFIODevice *vbasedev);int vfio_user_get_region_info (VFIODevice *vbasedev,int index,struct vfio_region_info *info,VFIOUserFDs *fds); +int vfio_user_region_read (VFIODevice *vbasedev,uint32_t index,uint64_t offset, +uint32_t count,void *data); +int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index, +uint64_t offset,uint32_t count,void *data);#endif /* VFIO_USER_H */
hw/vfio/common.c +14 -2
@@ -215,6 +215,7 @@ void vfio_region_write(void *opaque, hwaddr addr,uint32_t dword;uint64_t qword; } buf; +int ret;switch (size) {case 1 :
@@ -234,7 +235,12 @@ void vfio_region_write(void *opaque, hwaddr addr,break ; } -if (pwrite (vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_region_write (vbasedev, region->nr, addr, size, &data); + }else { + ret =pwrite (vbasedev->fd, &buf, size, region->fd_offset + addr); + } +if (ret != size) { error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64 ",%d) failed: %m", __func__, vbasedev->name, region->nr,
@@ -266,8 +272,14 @@ uint64_t vfio_region_read(void *opaque,uint64_t qword; } buf;uint64_t data =0 ; +int ret; -if (pread (vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_region_read (vbasedev, region->nr, addr, size, &buf); + }else { + ret =pread (vbasedev->fd, &buf, size, region->fd_offset + addr); + } +if (ret != size) {error_report ("%s(%s:region%d+0x%" HWADDR_PRIx", %d) failed: %m" , __func__, vbasedev->name, region->nr, addr, size);
hw/vfio/user.c +43 -0
46 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 09/16] vfio-user: region read/write
Quoted 13 lines · Elena Ufimtseva
hw/vfio/common.c +1
-0@@ -215,6 +215,7 @@ void vfio_region_write(void *opaque, hwaddr addr,uint32_t dword;uint64_t qword; } buf; +int ret;switch (size) {case 1 :
hw/vfio/common.c void vfio_region_write(void *opaque, hwaddr addr, · line 242
Quoted 9 lines
hw/vfio/common.c +5
-1@@ -234,7 +235,12 @@ void vfio_region_write(void *opaque, hwaddr addr,break ; } -if (pwrite (vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_region_write (vbasedev, region->nr, addr, size, &data); + }else { + ret =pwrite (vbasedev->fd, &buf, size, region->fd_offset + addr); + }
The vfio-user spec says everything is little-endian. Why does vfio_user_region_write() take the host-endian uint64_t data value instead of the little-endian buf value?
Quoted 58 lines
hw/vfio/common.c +8
-1+if (ret != size) { error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64 ",%d) failed: %m", __func__, vbasedev->name, region->nr,@@ -266,8 +272,14 @@ uint64_t vfio_region_read(void *opaque,uint64_t qword; } buf;uint64_t data =0 ; +int ret; -if (pread (vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_region_read (vbasedev, region->nr, addr, size, &buf); + }else { + ret =pread (vbasedev->fd, &buf, size, region->fd_offset + addr); + } +if (ret != size) {error_report ("%s(%s:region%d+0x%" HWADDR_PRIx", %d) failed: %m" , __func__, vbasedev->name, region->nr, addr, size);hw/vfio/user.c +30
-0@@ -767,3 +767,46 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,memcpy (info, &msgp->argsz, info->argsz);return 0 ; } + +int vfio_user_region_read (VFIODevice *vbasedev,uint32_t index,uint64_t offset, +uint32_t count,void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; +int size =sizeof (*msgp) + count; + + msgp =g_malloc0 (size); +vfio_user_request_msg (&msgp->hdr,VFIO_USER_REGION_READ ,sizeof (*msgp),0 ); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + +vfio_user_send_recv (vbasedev->proxy, &msgp->hdr,NULL , size,0 ); +if (msgp->hdr.flags &VFIO_USER_ERROR ) { +return -msgp->hdr.error_reply; + }else if (msgp->count > count) { +return -E2BIG ; + }else { +memcpy (data, &msgp->data, msgp->count); + } + +return msgp->count; +} + +int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index, +uint64_t offset,uint32_t count,void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; +int size =sizeof (*msgp) + count;
hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 809
Quoted 10 lines
hw/vfio/user.c +10
-0+ + msgp =g_malloc0 (size); +vfio_user_request_msg (&msgp->hdr,VFIO_USER_REGION_WRITE , size, +VFIO_USER_NO_REPLY ); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; +memcpy (&msgp->data, data, count); + +vfio_user_send (vbasedev->proxy, &msgp->hdr,NULL );
Are VFIO region writes posted writes (VFIO_USER_NO_REPLY)? This can be a problem if the device driver performs a write to the region followed by another access (e.g. to an mmap region) and expected the write to complete before the second access takes place.
Re: [PATCH RFC v2 09/16] vfio-user: region read/write
In reply to Elena Ufimtsevahw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803
Quoted 9 lines · Elena Ufimtseva
hw/vfio/user.c +9
-0+int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index, +uint64_t offset,uint32_t count,void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; +int size =sizeof (*msgp) + count; + + msgp =g_malloc0 (size); +vfio_user_request_msg (&msgp->hdr,VFIO_USER_REGION_WRITE , size, +VFIO_USER_NO_REPLY );
Mirroring https://github.com/oracle/qemu/issues/10 here for visibility:
Currently, vfio_user_region_write uses VFIO_USER_NO_REPLY unconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.
regards
john
Re: [PATCH RFC v2 09/16] vfio-user: region read/write
Quoted 18 lines · John Levon
hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803
Quoted 9 lines · Elena Ufimtseva
hw/vfio/user.c +9
-0+int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index, +uint64_t offset,uint32_t count,void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; +int size =sizeof (*msgp) + count; + + msgp =g_malloc0 (size); +vfio_user_request_msg (&msgp->hdr,VFIO_USER_REGION_WRITE , size, +VFIO_USER_NO_REPLY );Mirroring https://github.com/oracle/qemu/issues/10 here for visibility:
Quoted 5 lines
Currently,
vfio_user_region_writeusesVFIO_USER_NO_REPLYunconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.
I’m not sure following the PCI spec (mem writes posted, config & IO
are not) completely solves the issue if the device uses sparse mmap. A store to went over the socket can be passed by a load that goes directly to memory, which could break a driver that assumes a load completion means older stores to the same device have also completed.
JJ
Re: [PATCH RFC v2 09/16] vfio-user: region read/write
Quoted 21 lines · John Johnson
Quoted 16 lines · John Levon
hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803
Quoted 9 lines · Elena Ufimtseva
hw/vfio/user.c +9
-0+int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index, +uint64_t offset,uint32_t count,void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; +int size =sizeof (*msgp) + count; + + msgp =g_malloc0 (size); +vfio_user_request_msg (&msgp->hdr,VFIO_USER_REGION_WRITE , size, +VFIO_USER_NO_REPLY );Mirroring https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_oracle_qemu_issues_10&d=DwIGaQ&c=s883GpUCOChKOHiocYtGcg&r=v7SNLJqx7b9Vfc7ZO82Wg4nnZ8O5XkACFQ30bVKxotI&m=PJ390CfKPdTFUffSi02whMSqey2en8OJv7dm9VAQKI0&s=Mfp1xRKET3LEucEeZwUVUvSJ3V0zzGEktOzFwMsTfEE&e= here for visibility:
Quoted 4 lines
Currently,
vfio_user_region_writeusesVFIO_USER_NO_REPLYunconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.
Quoted 5 lines
I’m not sure following the PCI spec (mem writes posted, config & IO
are not) completely solves the issue if the device uses sparse mmap. A store to went over the socket can be passed by a load that goes directly to memory, which could break a driver that assumes a load completion means older stores to the same device have also completed.
Sure, but sparse mmaps are under the device's control - so wouldn't that be something of a "don't do that" scenario?
regards
john
Re: [PATCH RFC v2 09/16] vfio-user: region read/write
Quoted 27 lines · John Levon
Quoted 21 lines · John Johnson
Quoted 20 lines · John Levon
hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803
Quoted 9 lines · Elena Ufimtseva
hw/vfio/user.c +9
-0+int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index, +uint64_t offset,uint32_t count,void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; +int size =sizeof (*msgp) + count; + + msgp =g_malloc0 (size); +vfio_user_request_msg (&msgp->hdr,VFIO_USER_REGION_WRITE , size, +VFIO_USER_NO_REPLY );Mirroring https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_oracle_qemu_issues_10&d=DwIGaQ&c=s883GpUCOChKOHiocYtGcg&r=v7SNLJqx7b9Vfc7ZO82Wg4nnZ8O5XkACFQ30bVKxotI&m=PJ390CfKPdTFUffSi02whMSqey2en8OJv7dm9VAQKI0&s=Mfp1xRKET3LEucEeZwUVUvSJ3V0zzGEktOzFwMsTfEE&e= here for visibility:
Currently,
vfio_user_region_writeusesVFIO_USER_NO_REPLYunconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.
Quoted 9 lines
Quoted 5 lines
I’m not sure following the PCI spec (mem writes posted, config & IO
are not) completely solves the issue if the device uses sparse mmap. A store to went over the socket can be passed by a load that goes directly to memory, which could break a driver that assumes a load completion means older stores to the same device have also completed.Sure, but sparse mmaps are under the device's control - so wouldn't that be something of a "don't do that" scenario?
The sparse mmaps are under the emulation program’s control, but it
doesn’t know what registers the guest device driver is using to force stores to complete. The Linux device drivers doc on kernel.org just says the driver must read from the same device.
JJ
https://www.kernel.org/doc/Documentation/driver-api/device-io.rst
While the basic functions are defined to be synchronous with respect to each other and ordered with respect to each other the busses the devices sit on may themselves have asynchronicity. In particular many authors are burned by the fact that PCI bus writes are posted asynchronously. A driver author must issue a read from the same device to ensure that writes have occurred in the specific cases the author cares.
Re: [PATCH RFC v2 09/16] vfio-user: region read/write
Quoted 25 lines · John Johnson
Quoted 25 lines
Quoted 24 lines
Quoted 14 lines · Elena Ufimtseva
hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803
Quoted 10 lines
hw/vfio/user.c +9
-0+int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index, +uint64_t offset,uint32_t count,void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; +int size =sizeof (*msgp) + count; + + msgp =g_malloc0 (size); +vfio_user_request_msg (&msgp->hdr,VFIO_USER_REGION_WRITE , size, +VFIO_USER_NO_REPLY );Mirroring https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_oracle_qemu_issues_10&d=DwIGaQ&c=s883GpUCOChKOHiocYtGcg&r=v7SNLJqx7b9Vfc7ZO82Wg4nnZ8O5XkACFQ30bVKxotI&m=PJ390CfKPdTFUffSi02whMSqey2en8OJv7dm9VAQKI0&s=Mfp1xRKET3LEucEeZwUVUvSJ3V0zzGEktOzFwMsTfEE&e= here for visibility:
Quoted 4 lines
Currently,
vfio_user_region_writeusesVFIO_USER_NO_REPLYunconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.I’m not sure following the PCI spec (mem writes posted, config & IO
are not) completely solves the issue if the device uses sparse mmap. A store to went over the socket can be passed by a load that goes directly to memory, which could break a driver that assumes a load completion means older stores to the same device have also completed.
Quoted 7 lines
Quoted 2 lines
Sure, but sparse mmaps are under the device's control - so wouldn't that be something of a "don't do that" scenario?
The sparse mmaps are under the emulation program’s control, but it
doesn’t know what registers the guest device driver is using to force stores to complete. The Linux device drivers doc on kernel.org just says the driver must read from the same device.
Sure, but any device where that is important wouldn't use the sparse mmaps, no?
There's no other alternative.
regards
john
RFC PATCH v2 10/16 vfio-user: pci_user_realize PCI setup
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
PCI BARs read from remote device
PCI config reads/writes sent to remote server
--- hw/vfio/pci.c | 210 +++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 175 insertions(+), 35 deletions(-)
hw/vfio/pci.c +175 -35
@@ -807,8 +807,14 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev)memset (vdev->rom,0xff , size);while (size) { - bytes =pread (vdev->vbasedev.fd, vdev->rom + off, - size, vdev->rom_offset + off); +if (vdev->vbasedev.proxy !=NULL ) { + bytes =vfio_user_region_read (&vdev->vbasedev, +VFIO_PCI_ROM_REGION_INDEX , + off, size, vdev->rom + off); + }else { + bytes =pread (vdev->vbasedev.fd, vdev->rom + off, + size, vdev->rom_offset + off); + }if (bytes ==0 ) {break ; }else if (bytes >0 ) {
@@ -927,12 +933,28 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev)* Use the same size ROM BAR as the physical device. The contents * will get filled in later when the guest tries to read it. */ -if (pread (fd, &orig,4 , offset) !=4 || -pwrite (fd, &size,4 , offset) !=4 || -pread (fd, &size,4 , offset) !=4 || -pwrite (fd, &orig,4 , offset) !=4 ) { -error_report ("%s(%s) failed: %m" , __func__, vdev->vbasedev.name); -return ; +if (vdev->vbasedev.proxy !=NULL ) { +if (vfio_user_region_read (&vdev->vbasedev,VFIO_PCI_CONFIG_REGION_INDEX , +PCI_ROM_ADDRESS ,4 , &orig) !=4 || +vfio_user_region_write (&vdev->vbasedev, +VFIO_PCI_CONFIG_REGION_INDEX , +PCI_ROM_ADDRESS ,4 , &size) !=4 || +vfio_user_region_read (&vdev->vbasedev,VFIO_PCI_CONFIG_REGION_INDEX , +PCI_ROM_ADDRESS ,4 , &size) !=4 || +vfio_user_region_write (&vdev->vbasedev, +VFIO_PCI_CONFIG_REGION_INDEX , +PCI_ROM_ADDRESS ,4 , &orig) !=4 ) { +error_report ("%s(%s) failed: %m" , __func__, vdev->vbasedev.name); +return ; + } + }else { +if (pread (fd, &orig,4 , offset) !=4 || +pwrite (fd, &size,4 , offset) !=4 || +pread (fd, &size,4 , offset) !=4 || +pwrite (fd, &orig,4 , offset) !=4 ) { +error_report ("%s(%s) failed: %m" , __func__, vdev->vbasedev.name); +return ; + } } size = ~(le32_to_cpu (size) &PCI_ROM_ADDRESS_MASK ) +1 ;
@@ -1123,8 +1145,14 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len)if (~emu_bits & (0xffffffffU >> (32 - len *8 ))) {ssize_t ret; - ret =pread (vdev->vbasedev.fd, &phys_val, len, - vdev->config_offset + addr); +if (vdev->vbasedev.proxy !=NULL ) { + ret =vfio_user_region_read (&vdev->vbasedev, +VFIO_PCI_CONFIG_REGION_INDEX , + addr, len, &phys_val); + }else { + ret =pread (vdev->vbasedev.fd, &phys_val, len, + vdev->config_offset + addr); + }if (ret != len) {error_report ("%s(%s, 0x%x, 0x%x) failed: %m" , __func__, vdev->vbasedev.name, addr, len);
@@ -1145,12 +1173,20 @@ void vfio_pci_write_config(PCIDevice *pdev, {VFIOPCIDevice *vdev =VFIO_PCI_BASE (pdev);uint32_t val_le =cpu_to_le32 (val); +int ret;trace_vfio_pci_write_config (vdev->vbasedev.name, addr, val, len);/* Write everything to VFIO, let it filter out what we can't write */ -if (pwrite (vdev->vbasedev.fd, &val_le, len, vdev->config_offset + addr) - != len) { +if (vdev->vbasedev.proxy !=NULL ) { + ret =vfio_user_region_write (&vdev->vbasedev, +VFIO_PCI_CONFIG_REGION_INDEX , + addr, len, &val_le); + }else { + ret =pwrite (vdev->vbasedev.fd, &val_le, len, + vdev->config_offset + addr); + } +if (ret != len) {error_report ("%s(%s, 0x%x, 0x%x, 0x%x) failed: %m" , __func__, vdev->vbasedev.name, addr, val, len); }
@@ -1240,10 +1276,15 @@ static int vfio_msi_setup(VFIOPCIDevice *vdev, int pos, Error **errp)int ret, entries;Error *err =NULL ; -if (pread (vdev->vbasedev.fd, &ctrl,sizeof (ctrl), - vdev->config_offset + pos +PCI_CAP_FLAGS ) !=sizeof (ctrl)) { -error_setg_errno (errp, errno,"failed reading MSI PCI_CAP_FLAGS" ); -return -errno; +if (vdev->vbasedev.proxy !=NULL ) { +/* during setup, config space was initialized from remote */ +memcpy (&ctrl, vdev->pdev.config + pos +PCI_CAP_FLAGS ,sizeof (ctrl)); + }else { +if (pread (vdev->vbasedev.fd, &ctrl,sizeof (ctrl), + vdev->config_offset + pos +PCI_CAP_FLAGS ) !=sizeof (ctrl)) { +error_setg_errno (errp, errno,"failed reading MSI PCI_CAP_FLAGS" ); +return -errno; + } } ctrl =le16_to_cpu (ctrl);
@@ -1456,22 +1497,30 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp)return ; } -if (pread (fd, &ctrl,sizeof (ctrl), - vdev->config_offset + pos +PCI_MSIX_FLAGS ) !=sizeof (ctrl)) { -error_setg_errno (errp, errno,"failed to read PCI MSIX FLAGS" ); -return ; - } +if (vdev->vbasedev.proxy !=NULL ) { +/* during setup, config space was initialized from remote */ +memcpy (&ctrl, vdev->pdev.config + pos +PCI_MSIX_FLAGS ,sizeof (ctrl)); +memcpy (&table, vdev->pdev.config + pos +PCI_MSIX_TABLE ,sizeof (table)); +memcpy (&pba, vdev->pdev.config + pos +PCI_MSIX_PBA ,sizeof (pba)); + }else { +if (pread (fd, &ctrl,sizeof (ctrl), + vdev->config_offset + pos +PCI_MSIX_FLAGS ) !=sizeof (ctrl)) { +error_setg_errno (errp, errno,"failed to read PCI MSIX FLAGS" ); +return ; + } -if (pread (fd, &table,sizeof (table), - vdev->config_offset + pos +PCI_MSIX_TABLE ) !=sizeof (table)) { -error_setg_errno (errp, errno,"failed to read PCI MSIX TABLE" ); -return ; - } +if (pread (fd, &table,sizeof (table), + vdev->config_offset + pos + +PCI_MSIX_TABLE ) !=sizeof (table)) { +error_setg_errno (errp, errno,"failed to read PCI MSIX TABLE" ); +return ; + } -if (pread (fd, &pba,sizeof (pba), - vdev->config_offset + pos +PCI_MSIX_PBA ) !=sizeof (pba)) { -error_setg_errno (errp, errno,"failed to read PCI MSIX PBA" ); -return ; +if (pread (fd, &pba,sizeof (pba), + vdev->config_offset + pos +PCI_MSIX_PBA ) !=sizeof (pba)) { +error_setg_errno (errp, errno,"failed to read PCI MSIX PBA" ); +return ; + } } ctrl =le16_to_cpu (ctrl);
@@ -1619,11 +1668,17 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) }/* Determine what type of BAR this is for registration */ - ret =pread (vdev->vbasedev.fd, &pci_bar,sizeof (pci_bar), - vdev->config_offset +PCI_BASE_ADDRESS_0 + (4 * nr)); -if (ret !=sizeof (pci_bar)) { -error_report ("vfio: Failed to read BAR %d (%m)" , nr); -return ; +if (vdev->vbasedev.proxy !=NULL ) { +/* during setup, config space was initialized from remote */ +memcpy (&pci_bar, vdev->pdev.config +PCI_BASE_ADDRESS_0 + (4 * nr), +sizeof (pci_bar)); + }else { + ret =pread (vdev->vbasedev.fd, &pci_bar,sizeof (pci_bar), + vdev->config_offset +PCI_BASE_ADDRESS_0 + (4 * nr)); +if (ret !=sizeof (pci_bar)) { +error_report ("vfio: Failed to read BAR %d (%m)" , nr); +return ; + } } pci_bar =le32_to_cpu (pci_bar);
@@ -3423,6 +3478,91 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto error; } +/* Get a copy of config space */ + ret =vfio_user_region_read (vbasedev,VFIO_PCI_CONFIG_REGION_INDEX ,0 , +MIN (pci_config_size (pdev), vdev->config_size), + pdev->config); +if (ret < (int )MIN (pci_config_size (&vdev->pdev), vdev->config_size)) { +error_setg_errno (errp, -ret,"failed to read device config space" ); + goto error; + } + +/* vfio emulates a lot for us, but some bits need extra love */ + vdev->emulated_config_bits =g_malloc0 (vdev->config_size); + +/* QEMU can choose to expose the ROM or not */ +memset (vdev->emulated_config_bits +PCI_ROM_ADDRESS ,0xff ,4 ); +/* QEMU can also add or extend BARs */ +memset (vdev->emulated_config_bits +PCI_BASE_ADDRESS_0 ,0xff ,6 *4 ); + vdev->vendor_id =pci_get_word (pdev->config +PCI_VENDOR_ID ); + vdev->device_id =pci_get_word (pdev->config +PCI_DEVICE_ID ); + +/* QEMU can change multi-function devices to single function, or reverse */ + vdev->emulated_config_bits[PCI_HEADER_TYPE ] = +PCI_HEADER_TYPE_MULTI_FUNCTION ; + +/* Restore or clear multifunction, this is always controlled by QEMU */ +if (vdev->pdev.cap_present &QEMU_PCI_CAP_MULTIFUNCTION ) { + vdev->pdev.config[PCI_HEADER_TYPE ] |=PCI_HEADER_TYPE_MULTI_FUNCTION ; + }else { + vdev->pdev.config[PCI_HEADER_TYPE ] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION ; + } + +/* +* Clear host resource mapping info. If we choose not to register a +* BAR, such as might be the case with the option ROM, we can get +* confusing, unwritable, residual addresses from the host here. +*/ +memset (&vdev->pdev.config[PCI_BASE_ADDRESS_0 ],0 ,24 ); +memset (&vdev->pdev.config[PCI_ROM_ADDRESS ],0 ,4 ); + +vfio_pci_size_rom (vdev); + +vfio_bars_prepare (vdev); + +vfio_msix_early_setup (vdev, &err); +if (err) { +error_propagate (errp, err); + goto error; + } + +vfio_bars_register (vdev); + + ret =vfio_add_capabilities (vdev, errp); +if (ret) { + goto out_teardown; + } + +/* QEMU emulates all of MSI & MSIX */ +if (pdev->cap_present &QEMU_PCI_CAP_MSIX ) { +memset (vdev->emulated_config_bits + pdev->msix_cap,0xff , +MSIX_CAP_LENGTH ); + } + +if (pdev->cap_present &QEMU_PCI_CAP_MSI ) { +memset (vdev->emulated_config_bits + pdev->msi_cap,0xff , + vdev->msi_cap_size); + } + +if (vdev->pdev.config[PCI_INTERRUPT_PIN ] !=0 ) { + vdev->intx.mmap_timer =timer_new_ms (QEMU_CLOCK_VIRTUAL , + vfio_intx_mmap_enable, vdev); +pci_device_set_intx_routing_notifier (&vdev->pdev, + vfio_intx_routing_notifier); + vdev->irqchip_change_notifier.notify = vfio_irqchip_change; +kvm_irqchip_add_change_notifier (&vdev->irqchip_change_notifier); + ret =vfio_intx_enable (vdev, errp); +if (ret) { + goto out_deregister; + } + } + +out_deregister: +pci_device_set_intx_routing_notifier (&vdev->pdev,NULL ); +kvm_irqchip_remove_change_notifier (&vdev->irqchip_change_notifier); +out_teardown: +vfio_teardown_msi (vdev); +vfio_bars_exit (vdev); error:vfio_user_disconnect (proxy);error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name);
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 10/16] vfio-user: pci_user_realize PCI setup
Quoted 48 lines · Elena Ufimtseva
hw/vfio/pci.c +44
-0@@ -3423,6 +3478,91 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto error; } +/* Get a copy of config space */ + ret =vfio_user_region_read (vbasedev,VFIO_PCI_CONFIG_REGION_INDEX ,0 , +MIN (pci_config_size (pdev), vdev->config_size), + pdev->config); +if (ret < (int )MIN (pci_config_size (&vdev->pdev), vdev->config_size)) { +error_setg_errno (errp, -ret,"failed to read device config space" ); + goto error; + } + +/* vfio emulates a lot for us, but some bits need extra love */ + vdev->emulated_config_bits =g_malloc0 (vdev->config_size); + +/* QEMU can choose to expose the ROM or not */ +memset (vdev->emulated_config_bits +PCI_ROM_ADDRESS ,0xff ,4 ); +/* QEMU can also add or extend BARs */ +memset (vdev->emulated_config_bits +PCI_BASE_ADDRESS_0 ,0xff ,6 *4 ); + vdev->vendor_id =pci_get_word (pdev->config +PCI_VENDOR_ID ); + vdev->device_id =pci_get_word (pdev->config +PCI_DEVICE_ID ); + +/* QEMU can change multi-function devices to single function, or reverse */ + vdev->emulated_config_bits[PCI_HEADER_TYPE ] = +PCI_HEADER_TYPE_MULTI_FUNCTION ; + +/* Restore or clear multifunction, this is always controlled by QEMU */ +if (vdev->pdev.cap_present &QEMU_PCI_CAP_MULTIFUNCTION ) { + vdev->pdev.config[PCI_HEADER_TYPE ] |=PCI_HEADER_TYPE_MULTI_FUNCTION ; + }else { + vdev->pdev.config[PCI_HEADER_TYPE ] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION ; + } + +/* +* Clear host resource mapping info. If we choose not to register a +* BAR, such as might be the case with the option ROM, we can get +* confusing, unwritable, residual addresses from the host here. +*/ +memset (&vdev->pdev.config[PCI_BASE_ADDRESS_0 ],0 ,24 ); +memset (&vdev->pdev.config[PCI_ROM_ADDRESS ],0 ,4 ); + +vfio_pci_size_rom (vdev); + +vfio_bars_prepare (vdev); + +vfio_msix_early_setup (vdev, &err); +if (err) {
hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3534
Quoted 10 lines
hw/vfio/pci.c +10
-0+error_propagate (errp, err); + goto error; + } + +vfio_bars_register (vdev); + + ret =vfio_add_capabilities (vdev, errp); +if (ret) { + goto out_teardown; + }
I haven't audited the common code to find places where the contents of the PCI Configuration Space are trusted. Input validation may need to be performed on offsets and other inputs that we read from the device.
Otherwise:
RFC PATCH v2 11/16 vfio-user: get and set IRQs
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 25 ++++++++++ hw/vfio/user.h | 2 + hw/vfio/common.c | 26 ++++++++-- hw/vfio/pci.c | 31 ++++++++++-- hw/vfio/user.c | 106 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 181 insertions(+), 9 deletions(-)
hw/vfio/user-protocol.h +25 -0
@@ -109,6 +109,31 @@ typedef struct {uint64_t offset; } VFIOUserRegionInfo; +/* +* VFIO_USER_DEVICE_GET_IRQ_INFO +* imported from struct vfio_irq_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t count; +}VFIOUserIRQInfo ; + +/* +* VFIO_USER_DEVICE_SET_IRQS +* imported from struct vfio_irq_set +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t start; +uint32_t count; +}VFIOUserIRQSet ; +/* * VFIO_USER_REGION_READ * VFIO_USER_REGION_WRITE
hw/vfio/user.h +2 -0
@@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);int vfio_user_get_info (VFIODevice *vbasedev);int vfio_user_get_region_info (VFIODevice *vbasedev,int index,struct vfio_region_info *info,VFIOUserFDs *fds); +int vfio_user_get_irq_info (VFIODevice *vbasedev,struct vfio_irq_info *info); +int vfio_user_set_irqs (VFIODevice *vbasedev,struct vfio_irq_set *irq);int vfio_user_region_read (VFIODevice *vbasedev,uint32_t index,uint64_t offset,uint32_t count,void *data); int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
hw/vfio/common.c +21 -5
@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index) .count = 0, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }void vfio_unmask_single_irqindex (VFIODevice *vbasedev,int index)
@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }void vfio_mask_single_irqindex (VFIODevice *vbasedev,int index)
@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }static inline const char *action_to_str (int action)
@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, pfd = (int32_t *)&irq_set->data; *pfd = fd; -if (ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set)) { - ret = -errno; +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_set_irqs (vbasedev, irq_set); + }else { +if (ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set)) { + ret = -errno; + } }g_free (irq_set);
hw/vfio/pci.c +27 -4
@@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) fds[i] = fd; } - ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_SET_IRQS , irq_set); +if (vdev->vbasedev.proxy !=NULL ) { + ret =vfio_user_set_irqs (&vdev->vbasedev, irq_set); + }else { + ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_SET_IRQS , irq_set); + }g_free (irq_set);
@@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index =VFIO_PCI_ERR_IRQ_INDEX ; - ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info); +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_get_irq_info (vbasedev, &irq_info); + }else { + ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info); + } + +if (ret) {/* This can fail for an old kernel or legacy PCI dev */ trace_vfio_populate_device_get_irq_info_failure (strerror (errno));
@@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)return ; } -if (ioctl (vdev->vbasedev.fd, -VFIO_DEVICE_GET_IRQ_INFO , &irq_info) <0 || irq_info.count <1 ) { +if (vdev->vbasedev.proxy !=NULL ) { +if (vfio_user_get_irq_info (&vdev->vbasedev, &irq_info) <0 ) { +return ; + } + }else { +if (ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info) <0 ) { +return ; + } + } +if (irq_info.count <1 ) {return ; }
@@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } } +vfio_register_err_notifier (vdev); +vfio_register_req_notifier (vdev); + +return ; + out_deregister:pci_device_set_intx_routing_notifier (&vdev->pdev,NULL );kvm_irqchip_remove_change_notifier (&vdev->irqchip_change_notifier);
hw/vfio/user.c +106 -0
112 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 11/16] vfio-user: get and set IRQs
Quoted 197 lines · Elena Ufimtseva
From: John Johnson <[email protected]>
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/vfio/user-protocol.h | 25 ++++++++++ hw/vfio/user.h | 2 + hw/vfio/common.c | 26 ++++++++-- hw/vfio/pci.c | 31 ++++++++++-- hw/vfio/user.c | 106 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 181 insertions(+), 9 deletions(-)hw/vfio/user-protocol.h +25
-0@@ -109,6 +109,31 @@ typedef struct {uint64_t offset; } VFIOUserRegionInfo; +/* +* VFIO_USER_DEVICE_GET_IRQ_INFO +* imported from struct vfio_irq_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t count; +}VFIOUserIRQInfo ; + +/* +* VFIO_USER_DEVICE_SET_IRQS +* imported from struct vfio_irq_set +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t start; +uint32_t count; +}VFIOUserIRQSet ; +/* * VFIO_USER_REGION_READ * VFIO_USER_REGION_WRITE hw/vfio/user.h +2
-0@@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);int vfio_user_get_info (VFIODevice *vbasedev);int vfio_user_get_region_info (VFIODevice *vbasedev,int index,struct vfio_region_info *info,VFIOUserFDs *fds); +int vfio_user_get_irq_info (VFIODevice *vbasedev,struct vfio_irq_info *info); +int vfio_user_set_irqs (VFIODevice *vbasedev,struct vfio_irq_set *irq);int vfio_user_region_read (VFIODevice *vbasedev,uint32_t index,uint64_t offset,uint32_t count,void *data); int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,hw/vfio/common.c +21
-5@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index) .count = 0, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }void vfio_unmask_single_irqindex (VFIODevice *vbasedev,int index)@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }void vfio_mask_single_irqindex (VFIODevice *vbasedev,int index)@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }static inline const char *action_to_str (int action)@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, pfd = (int32_t *)&irq_set->data; *pfd = fd; -if (ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set)) { - ret = -errno; +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_set_irqs (vbasedev, irq_set); + }else { +if (ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set)) { + ret = -errno; + } }g_free (irq_set);hw/vfio/pci.c +27
-4@@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) fds[i] = fd; } - ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_SET_IRQS , irq_set); +if (vdev->vbasedev.proxy !=NULL ) { + ret =vfio_user_set_irqs (&vdev->vbasedev, irq_set); + }else { + ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_SET_IRQS , irq_set); + }g_free (irq_set);@@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index =VFIO_PCI_ERR_IRQ_INDEX ; - ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info); +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_get_irq_info (vbasedev, &irq_info); + }else { + ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info); + } + +if (ret) {/* This can fail for an old kernel or legacy PCI dev */ trace_vfio_populate_device_get_irq_info_failure (strerror (errno));@@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)return ; } -if (ioctl (vdev->vbasedev.fd, -VFIO_DEVICE_GET_IRQ_INFO , &irq_info) <0 || irq_info.count <1 ) { +if (vdev->vbasedev.proxy !=NULL ) { +if (vfio_user_get_irq_info (&vdev->vbasedev, &irq_info) <0 ) { +return ; + } + }else { +if (ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info) <0 ) { +return ; + } + } +if (irq_info.count <1 ) {return ; }@@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } } +vfio_register_err_notifier (vdev); +vfio_register_req_notifier (vdev); + +return ; + out_deregister:pci_device_set_intx_routing_notifier (&vdev->pdev,NULL );kvm_irqchip_remove_change_notifier (&vdev->irqchip_change_notifier);hw/vfio/user.c +6
-0@@ -768,6 +768,112 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,return 0 ; } +int vfio_user_get_irq_info (VFIODevice *vbasedev,struct vfio_irq_info *info) +{ +VFIOUserIRQInfo msg; + +memset (&msg,0 ,sizeof (msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO,
hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 786
Quoted 10 lines
hw/vfio/user.c +10
-0+ sizeof(msg), 0); + msg.argsz = info->argsz; + msg.index = info->index; + +vfio_user_send_recv (vbasedev->proxy, &msg.hdr,NULL ,0 ,0 ); +if (msg.hdr.flags &VFIO_USER_ERROR ) { +return -msg.hdr.error_reply; + } + +memcpy (info, &msg.argsz,sizeof (*info));
Should this be info.count = msg.count instead? Not sure why argsz is used here.
Also, I just noticed the lack of endianness conversion in this patch series. The spec says values are little-endian but these patches mostly use host-endian. Did I miss something?
Re: [PATCH RFC v2 11/16] vfio-user: get and set IRQs
Quoted 207 lines · Stefan Hajnoczi
Quoted 202 lines · Elena Ufimtseva
From: John Johnson <[email protected]>
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/vfio/user-protocol.h | 25 ++++++++++ hw/vfio/user.h | 2 + hw/vfio/common.c | 26 ++++++++-- hw/vfio/pci.c | 31 ++++++++++-- hw/vfio/user.c | 106 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 181 insertions(+), 9 deletions(-)hw/vfio/user-protocol.h +25
-0@@ -109,6 +109,31 @@ typedef struct {uint64_t offset; } VFIOUserRegionInfo; +/* +* VFIO_USER_DEVICE_GET_IRQ_INFO +* imported from struct vfio_irq_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t count; +}VFIOUserIRQInfo ; + +/* +* VFIO_USER_DEVICE_SET_IRQS +* imported from struct vfio_irq_set +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t start; +uint32_t count; +}VFIOUserIRQSet ; +/* * VFIO_USER_REGION_READ * VFIO_USER_REGION_WRITE hw/vfio/user.h +2
-0@@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);int vfio_user_get_info (VFIODevice *vbasedev);int vfio_user_get_region_info (VFIODevice *vbasedev,int index,struct vfio_region_info *info,VFIOUserFDs *fds); +int vfio_user_get_irq_info (VFIODevice *vbasedev,struct vfio_irq_info *info); +int vfio_user_set_irqs (VFIODevice *vbasedev,struct vfio_irq_set *irq);int vfio_user_region_read (VFIODevice *vbasedev,uint32_t index,uint64_t offset,uint32_t count,void *data); int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,hw/vfio/common.c +21
-5@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index) .count = 0, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }void vfio_unmask_single_irqindex (VFIODevice *vbasedev,int index)@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }void vfio_mask_single_irqindex (VFIODevice *vbasedev,int index)@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }static inline const char *action_to_str (int action)@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, pfd = (int32_t *)&irq_set->data; *pfd = fd; -if (ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set)) { - ret = -errno; +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_set_irqs (vbasedev, irq_set); + }else { +if (ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set)) { + ret = -errno; + } }g_free (irq_set);hw/vfio/pci.c +27
-4@@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) fds[i] = fd; } - ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_SET_IRQS , irq_set); +if (vdev->vbasedev.proxy !=NULL ) { + ret =vfio_user_set_irqs (&vdev->vbasedev, irq_set); + }else { + ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_SET_IRQS , irq_set); + }g_free (irq_set);@@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index =VFIO_PCI_ERR_IRQ_INDEX ; - ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info); +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_get_irq_info (vbasedev, &irq_info); + }else { + ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info); + } + +if (ret) {/* This can fail for an old kernel or legacy PCI dev */ trace_vfio_populate_device_get_irq_info_failure (strerror (errno));@@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)return ; } -if (ioctl (vdev->vbasedev.fd, -VFIO_DEVICE_GET_IRQ_INFO , &irq_info) <0 || irq_info.count <1 ) { +if (vdev->vbasedev.proxy !=NULL ) { +if (vfio_user_get_irq_info (&vdev->vbasedev, &irq_info) <0 ) { +return ; + } + }else { +if (ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info) <0 ) { +return ; + } + } +if (irq_info.count <1 ) {return ; }@@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } } +vfio_register_err_notifier (vdev); +vfio_register_req_notifier (vdev); + +return ; +
out_deregister:pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);hw/vfio/user.c +11
-0@@ -768,6 +768,112 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,return 0 ; } +int vfio_user_get_irq_info (VFIODevice *vbasedev,struct vfio_irq_info *info) +{ +VFIOUserIRQInfo msg; + +memset (&msg,0 ,sizeof (msg)); +vfio_user_request_msg (&msg.hdr,VFIO_USER_DEVICE_GET_IRQ_INFO , +sizeof (msg),0 ); + msg.argsz = info->argsz; + msg.index = info->index; + +vfio_user_send_recv (vbasedev->proxy, &msg.hdr,NULL ,0 ,0 );
Quoted 8 lines
hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 786
Quoted 5 lines
hw/vfio/user.c +5
-0+if (msg.hdr.flags &VFIO_USER_ERROR ) { +return -msg.hdr.error_reply; + } + +memcpy (info, &msg.argsz,sizeof (*info));Should this be info.count = msg.count instead? Not sure why argsz is used here.
It’s copying the entire returned vfio_irq_info struct, which starts
at &msg.argsz.
Quoted 4 lines
Also, I just noticed the lack of endianness conversion in this patch series. The spec says values are little-endian but these patches mostly use host-endian. Did I miss something?
I had thought we were using host endianness for UNIX sockets and
were deferring the cross endianness issue to when we support TCP, but the spec does say all LE.
JJ
Re: [PATCH RFC v2 11/16] vfio-user: get and set IRQs
Quoted 216 lines · John Johnson
Quoted 207 lines · Stefan Hajnoczi
Quoted 202 lines · Elena Ufimtseva
From: John Johnson <[email protected]>
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/vfio/user-protocol.h | 25 ++++++++++ hw/vfio/user.h | 2 + hw/vfio/common.c | 26 ++++++++-- hw/vfio/pci.c | 31 ++++++++++-- hw/vfio/user.c | 106 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 181 insertions(+), 9 deletions(-)hw/vfio/user-protocol.h +25
-0@@ -109,6 +109,31 @@ typedef struct {uint64_t offset; } VFIOUserRegionInfo; +/* +* VFIO_USER_DEVICE_GET_IRQ_INFO +* imported from struct vfio_irq_info +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t count; +}VFIOUserIRQInfo ; + +/* +* VFIO_USER_DEVICE_SET_IRQS +* imported from struct vfio_irq_set +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint32_t index; +uint32_t start; +uint32_t count; +}VFIOUserIRQSet ; +/* * VFIO_USER_REGION_READ * VFIO_USER_REGION_WRITE hw/vfio/user.h +2
-0@@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);int vfio_user_get_info (VFIODevice *vbasedev);int vfio_user_get_region_info (VFIODevice *vbasedev,int index,struct vfio_region_info *info,VFIOUserFDs *fds); +int vfio_user_get_irq_info (VFIODevice *vbasedev,struct vfio_irq_info *info); +int vfio_user_set_irqs (VFIODevice *vbasedev,struct vfio_irq_set *irq);int vfio_user_region_read (VFIODevice *vbasedev,uint32_t index,uint64_t offset,uint32_t count,void *data); int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,hw/vfio/common.c +21
-5@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index) .count = 0, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }void vfio_unmask_single_irqindex (VFIODevice *vbasedev,int index)@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }void vfio_mask_single_irqindex (VFIODevice *vbasedev,int index)@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; -ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); +if (vbasedev->proxy !=NULL ) { +vfio_user_set_irqs (vbasedev, &irq_set); + }else { +ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , &irq_set); + } }static inline const char *action_to_str (int action)@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, pfd = (int32_t *)&irq_set->data; *pfd = fd; -if (ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set)) { - ret = -errno; +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_set_irqs (vbasedev, irq_set); + }else { +if (ioctl (vbasedev->fd,VFIO_DEVICE_SET_IRQS , irq_set)) { + ret = -errno; + } }g_free (irq_set);hw/vfio/pci.c +27
-4@@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) fds[i] = fd; } - ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_SET_IRQS , irq_set); +if (vdev->vbasedev.proxy !=NULL ) { + ret =vfio_user_set_irqs (&vdev->vbasedev, irq_set); + }else { + ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_SET_IRQS , irq_set); + }g_free (irq_set);@@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index =VFIO_PCI_ERR_IRQ_INDEX ; - ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info); +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_get_irq_info (vbasedev, &irq_info); + }else { + ret =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info); + } + +if (ret) {/* This can fail for an old kernel or legacy PCI dev */ trace_vfio_populate_device_get_irq_info_failure (strerror (errno));@@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)return ; } -if (ioctl (vdev->vbasedev.fd, -VFIO_DEVICE_GET_IRQ_INFO , &irq_info) <0 || irq_info.count <1 ) { +if (vdev->vbasedev.proxy !=NULL ) { +if (vfio_user_get_irq_info (&vdev->vbasedev, &irq_info) <0 ) { +return ; + } + }else { +if (ioctl (vdev->vbasedev.fd,VFIO_DEVICE_GET_IRQ_INFO , &irq_info) <0 ) { +return ; + } + } +if (irq_info.count <1 ) {return ; }@@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } } +vfio_register_err_notifier (vdev); +vfio_register_req_notifier (vdev); + +return ; +
out_deregister:pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);hw/vfio/user.c +11
-0@@ -768,6 +768,112 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,return 0 ; } +int vfio_user_get_irq_info (VFIODevice *vbasedev,struct vfio_irq_info *info) +{ +VFIOUserIRQInfo msg; + +memset (&msg,0 ,sizeof (msg)); +vfio_user_request_msg (&msg.hdr,VFIO_USER_DEVICE_GET_IRQ_INFO , +sizeof (msg),0 ); + msg.argsz = info->argsz; + msg.index = info->index; + +vfio_user_send_recv (vbasedev->proxy, &msg.hdr,NULL ,0 ,0 );Quoted 8 lines
hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 786
Quoted 5 lines
hw/vfio/user.c +5
-0+if (msg.hdr.flags &VFIO_USER_ERROR ) { +return -msg.hdr.error_reply; + } + +memcpy (info, &msg.argsz,sizeof (*info));Should this be info.count = msg.count instead? Not sure why argsz is used here.
Quoted 2 lines
It’s copying the entire returned
vfio_irq_infostruct, which starts
at &msg.argsz.
That makes sense, I missed it. Thanks!
Stefan
RFC PATCH v2 12/16 vfio-user: proxy container connect/disconnect
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- include/hw/vfio/vfio-common.h | 3 ++ hw/vfio/common.c | 84 +++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 22 +++++++++ 3 files changed, 109 insertions(+)
include/hw/vfio/vfio-common.h +3 -0
@@ -91,6 +91,7 @@ typedef struct VFIOContainer {uint64_t max_dirty_bitmap_size;unsigned long pgsizes;unsigned int dma_max_mappings; +VFIOProxy *proxy; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list; QLIST_HEAD(, VFIOGroup) group_list;
@@ -217,6 +218,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp);void vfio_put_group (VFIOGroup *group);int vfio_get_device (VFIOGroup *group,const char *name,VFIODevice *vbasedev,Error **errp); +void vfio_connect_proxy (VFIOProxy *proxy,VFIOGroup *group,AddressSpace *as); +void vfio_disconnect_proxy (VFIOGroup *group);extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList;
hw/vfio/common.c +84 -0
96 lines not on this page · raw message
hw/vfio/pci.c +22 -0
@@ -3442,6 +3442,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)VFIODevice *vbasedev = &vdev->vbasedev;SocketAddress addr;VFIOProxy *proxy; +VFIOGroup *group =NULL ;int ret;Error *err =NULL ;
@@ -3484,6 +3485,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; +/* +* each device gets its own group and container +* make them unrelated to any host IOMMU groupings +*/ + group =g_malloc0 (sizeof (*group)); + group->fd =-1 ; + group->groupid =-1 ; +QLIST_INIT (&group->device_list); +QLIST_INSERT_HEAD (&group->device_list, vbasedev, next); + vbasedev->group = group; + +vfio_connect_proxy (proxy, group,pci_device_iommu_address_space (pdev)); + ret =vfio_user_get_info (&vdev->vbasedev);if (ret) {error_setg_errno (errp, -ret,"get info failure" );
@@ -3587,6 +3601,9 @@ out_teardown:vfio_teardown_msi (vdev);vfio_bars_exit (vdev); error: +if (group !=NULL ) { +vfio_disconnect_proxy (group); + }vfio_user_disconnect (proxy);error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name); }
@@ -3595,6 +3612,11 @@ static void vfio_user_instance_finalize(Object *obj) {VFIOPCIDevice *vdev =VFIO_PCI_BASE (obj);VFIODevice *vbasedev = &vdev->vbasedev; +VFIOGroup *group = vbasedev->group; + +vfio_disconnect_proxy (group); +g_free (group); + vbasedev->group =NULL ;vfio_put_device (vdev);
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 12/16] vfio-user: proxy container connect/disconnect
Quoted 10 lines · Elena Ufimtseva
From: John Johnson <[email protected]>
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/vfio/vfio-common.h | 3 ++ hw/vfio/common.c | 84 +++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 22 +++++++++ 3 files changed, 109 insertions(+)
Alex: I'm not familiar enough with hw/vfio/ to review this in depth. You might have suggestions on how to unify the vfio-user and vfio kernel concepts of groups and containers.
Quoted 167 lines
include/hw/vfio/vfio-common.h +3
-0@@ -91,6 +91,7 @@ typedef struct VFIOContainer {uint64_t max_dirty_bitmap_size;unsigned long pgsizes;unsigned int dma_max_mappings; +VFIOProxy *proxy; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list; QLIST_HEAD(, VFIOGroup) group_list;@@ -217,6 +218,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp);void vfio_put_group (VFIOGroup *group);int vfio_get_device (VFIOGroup *group,const char *name,VFIODevice *vbasedev,Error **errp); +void vfio_connect_proxy (VFIOProxy *proxy,VFIOGroup *group,AddressSpace *as); +void vfio_disconnect_proxy (VFIOGroup *group);extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList;hw/vfio/common.c +84
-0@@ -2249,6 +2249,55 @@ put_space_exit:return ret; } +void vfio_connect_proxy (VFIOProxy *proxy,VFIOGroup *group,AddressSpace *as) +{ +VFIOAddressSpace *space; +VFIOContainer *container; + +if (QLIST_EMPTY (&vfio_group_list)) { +qemu_register_reset (vfio_reset_handler,NULL ); + } + +QLIST_INSERT_HEAD (&vfio_group_list, group, next); + +/* +* try to mirror vfio_connect_container() +* as much as possible +*/ + + space =vfio_get_address_space (as); + + container =g_malloc0 (sizeof (*container)); + container->space = space; + container->fd =-1 ; +QLIST_INIT (&container->giommu_list); +QLIST_INIT (&container->hostwin_list); + container->proxy = proxy; + +/* +* The proxy uses a SW IOMMU in lieu of the HW one +* used in the ioctl() version. Use TYPE1 with the +* target's page size for maximum capatibility +*/ + container->iommu_type =VFIO_TYPE1_IOMMU ; +vfio_host_win_add (container,0 , (hwaddr)-1 ,TARGET_PAGE_SIZE ); + container->pgsizes =TARGET_PAGE_SIZE ; + + container->dirty_pages_supported = true; + container->max_dirty_bitmap_size =VFIO_USER_DEF_MAX_XFER ; + container->dirty_pgsizes =TARGET_PAGE_SIZE ; + +QLIST_INIT (&container->group_list); +QLIST_INSERT_HEAD (&space->containers, container, next); + + group->container = container; +QLIST_INSERT_HEAD (&container->group_list, group, container_next); + + container->listener = vfio_memory_listener; +memory_listener_register (&container->listener, container->space->as); + container->initialized = true; +} +static void vfio_disconnect_container (VFIOGroup *group) {VFIOContainer *container = group->container;@@ -2291,6 +2340,41 @@ static void vfio_disconnect_container(VFIOGroup *group) } } +void vfio_disconnect_proxy (VFIOGroup *group) +{ +VFIOContainer *container = group->container; +VFIOAddressSpace *space = container->space; +VFIOGuestIOMMU *giommu, *tmp; + +/* +* try to mirror vfio_disconnect_container() +* as much as possible, knowing each device +* is in one group and one container +*/ + +QLIST_REMOVE (group, container_next); + group->container =NULL ; + +/* +* Explicitly release the listener first before unset container, +* since unset may destroy the backend container if it's the last +* group. +*/ +memory_listener_unregister (&container->listener); + +QLIST_REMOVE (container, next); + + QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, tmp) { +memory_region_unregister_iommu_notifier ( +MEMORY_REGION (giommu->iommu), &giommu->n); +QLIST_REMOVE (giommu, giommu_next); +g_free (giommu); + } + +g_free (container); +vfio_put_address_space (space); +} +VFIOGroup *vfio_get_group (int groupid,AddressSpace *as,Error **errp) {VFIOGroup *group;hw/vfio/pci.c +17
-0@@ -3442,6 +3442,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)VFIODevice *vbasedev = &vdev->vbasedev;SocketAddress addr;VFIOProxy *proxy; +VFIOGroup *group =NULL ;int ret;Error *err =NULL ;@@ -3484,6 +3485,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; +/* +* each device gets its own group and container +* make them unrelated to any host IOMMU groupings +*/ + group =g_malloc0 (sizeof (*group)); + group->fd =-1 ; + group->groupid =-1 ; +QLIST_INIT (&group->device_list); +QLIST_INSERT_HEAD (&group->device_list, vbasedev, next); + vbasedev->group = group; + +vfio_connect_proxy (proxy, group,pci_device_iommu_address_space (pdev)); + ret =vfio_user_get_info (&vdev->vbasedev);if (ret) {error_setg_errno (errp, -ret,"get info failure" );@@ -3587,6 +3601,9 @@ out_teardown:vfio_teardown_msi (vdev);vfio_bars_exit (vdev); error: +if (group !=NULL ) { +vfio_disconnect_proxy (group); + }vfio_user_disconnect (proxy);error_prepend (errp,VFIO_MSG_PREFIX , vdev->vbasedev.name); }
hw/vfio/pci.c static void vfio_user_instance_finalize(Object *obj) · line 3619
Quoted 8 lines
hw/vfio/pci.c +5
-0@@ -3595,6 +3612,11 @@ static void vfio_user_instance_finalize(Object *obj) {VFIOPCIDevice *vdev =VFIO_PCI_BASE (obj);VFIODevice *vbasedev = &vdev->vbasedev; +VFIOGroup *group = vbasedev->group; + +vfio_disconnect_proxy (group); +g_free (group); + vbasedev->group =NULL ;
Can vfio_put_group() be used instead? I'm worried that the cleanup code will be duplicated or become inconsistent if it's not shared.
Also, vfio_instance_finalize() calls vfio_put_group() after vfio_put_device(). Does this code intentionally take advantage of the if (!vbasedev->group) early return in vfio_put_base_device()? This is non-obvious. I recommend unifying the device and group cleanup instead of special-casing it here (this is fragile!).
Quoted 6 lines
hw/vfio/pci.c +0
-0vfio_put_device (vdev);Signature · 1 line
2.25.1
RFC PATCH v2 13/16 vfio-user: dma map/unmap operations
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 32 +++++++++ hw/vfio/user.h | 6 ++ include/hw/vfio/vfio-common.h | 1 + hw/vfio/common.c | 71 ++++++++++++++++--- hw/vfio/user.c | 124 ++++++++++++++++++++++++++++++++++ 5 files changed, 226 insertions(+), 8 deletions(-)
hw/vfio/user-protocol.h +32 -0
@@ -82,6 +82,31 @@ typedef struct {#define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) +/* +* VFIO_USER_DMA_MAP +* imported from struct vfio_iommu_type1_dma_map +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint64_t offset;/* FD offset */ +uint64_t iova; +uint64_t size; +}VFIOUserDMAMap ; + +/* +* VFIO_USER_DMA_UNMAP +* imported from struct vfio_iommu_type1_dma_unmap +*/ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +uint64_t iova; +uint64_t size; +}VFIOUserDMAUnmap ; +/* * VFIO_USER_DEVICE_GET_INFO * imported from struct_device_info
@@ -146,4 +171,11 @@ typedef struct { char data[]; } VFIOUserRegionRW; +/*imported from struct vfio_bitmap */ +typedef struct { +uint64_t pgsize; +uint64_t size; +char data[]; +}VFIOUserBitmap ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +6 -0
@@ -71,6 +71,11 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, void *reqarg);void vfio_user_send_reply (VFIOProxy *proxy,char *buf,int ret);int vfio_user_validate_version (VFIODevice *vbasedev,Error **errp); +int vfio_user_dma_map (VFIOProxy *proxy,struct vfio_iommu_type1_dma_map *map, +VFIOUserFDs *fds,bool will_commit); +int vfio_user_dma_unmap (VFIOProxy *proxy, +struct vfio_iommu_type1_dma_unmap *unmap, +struct vfio_bitmap *bitmap,bool will_commit);int vfio_user_get_info (VFIODevice *vbasedev);int vfio_user_get_region_info (VFIODevice *vbasedev,int index,struct vfio_region_info *info,VFIOUserFDs *fds);
@@ -80,5 +85,6 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data);int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index,uint64_t offset,uint32_t count,void *data); +void vfio_user_drain_reqs (VFIOProxy *proxy);#endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +1 -0
@@ -87,6 +87,7 @@ typedef struct VFIOContainer {Error *error;bool initialized;bool dirty_pages_supported; +bool will_commit;uint64_t dirty_pgsizes;uint64_t max_dirty_bitmap_size;unsigned long pgsizes;
hw/vfio/common.c +63 -8
172 lines not on this page · raw message
hw/vfio/user.c +124 -0
136 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 13/16] vfio-user: dma map/unmap operations
Quoted 29 lines · Elena Ufimtseva
hw/vfio/user.c +29
-0+void vfio_user_drain_reqs (VFIOProxy *proxy) +{ +VFIOUserReply *reply; +bool iolock =0 ; + +/* +* Any DMA map/unmap requests sent in the middle +* of a memory region transaction were sent async. +* Wait for them here. +*/ +QEMU_LOCK_GUARD (&proxy->lock); +if (proxy->last_nowait !=NULL ) { + iolock =qemu_mutex_iothread_locked (); +if (iolock) { +qemu_mutex_unlock_iothread (); + } + + reply = proxy->last_nowait; + reply->nowait =0 ; +while (reply->complete ==0 ) { +if (!qemu_cond_timedwait (&reply->cv, &proxy->lock, wait_time)) { +error_printf ("vfio_drain_reqs - timed out\n" ); +break ; + } + } + +if (reply->msg->flags &VFIO_USER_ERROR ) { +error_printf ("vfio_user_rcv error reply on async request " ); + error_printf("command %x error %s\n", reply->msg->command,
hw/vfio/user.c static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg, · line 449
Quoted 10 lines
hw/vfio/user.c +10
-0+ strerror(reply->msg->error_reply)); + } + proxy->last_nowait =NULL ; +g_free (reply->msg); +QTAILQ_INSERT_HEAD (&proxy->free, reply, next); + } + +if (iolock) { +qemu_mutex_lock_iothread (); + }
Not sure this lock ordering is correct. Above we acquire proxy->lock while holding the BQL and here we acquire the BQL while holding proxy->lock. If another thread (e.g. a vCPU thread) does something similar this is the ABBA lock ordering problem.
The more obviously correct way to write this is:
WITH_QEMU_LOCK_GUARD(&proxy->lock) {
...
}
if (iolock) {
qemu_mutex_lock_iothread();
}hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 762
Quoted 13 lines
hw/vfio/user.c +6
-0+} +static void vfio_user_request_msg (VFIOUserHdr *hdr,uint16_t cmd,uint32_t size,uint32_t flags) {@@ -715,6 +756,89 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp)return 0 ; } +int vfio_user_dma_map (VFIOProxy *proxy,struct vfio_iommu_type1_dma_map *map, +VFIOUserFDs *fds,bool will_commit) +{ +VFIOUserDMAMap *msgp =g_malloc (sizeof (*msgp));
Is this zero-initialized anywhere to guarantee that QEMU memory isn't exposed to the VFIO device emulation program?
hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 766
Quoted 4 lines
hw/vfio/user.c +4
-0+int ret, flags; + +/* commit will wait, so send async without dropping BQL */ + flags = will_commit ? (NOIOLOCK |NOWAIT ) :0 ;
Why is this distinction between will_commit and !will_commit necessary? I get a sense that the network communications code drops the BQL and that's undesirable here for some reason. I wonder why the code doesn't take the NOWAIT code path all the time?
RFC PATCH v2 14/16 vfio-user: dma read/write operations
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 11 +++++++ hw/vfio/user.h | 1 + hw/vfio/pci.c | 63 ++++++++++++++++++++++++++++++++++++++++- hw/vfio/user.c | 7 ++++- 4 files changed, 80 insertions(+), 2 deletions(-)
hw/vfio/user-protocol.h +11 -0
@@ -171,6 +171,17 @@ typedef struct {char data[]; } VFIOUserRegionRW; +/* +* VFIO_USER_DMA_READ +* VFIO_USER_DMA_WRITE +*/ +typedef struct { +VFIOUserHdr hdr; +uint64_t offset; +uint32_t count; +char data[]; +}VFIOUserDMARW ; +/*imported from struct vfio_bitmap */ typedef struct { uint64_t pgsize;
hw/vfio/user.h +1 -0
@@ -65,6 +65,7 @@ typedef struct VFIOProxy {VFIOProxy *vfio_user_connect_dev (SocketAddress *addr,Error **errp);void vfio_user_disconnect (VFIOProxy *proxy); +uint64_t vfio_user_max_xfer (void ); void vfio_user_set_reqhandler(VFIODevice *vbasdev, int (*handler)(void *opaque, char *buf, VFIOUserFDs *fds),
hw/vfio/pci.c +62 -1
73 lines not on this page · raw message
hw/vfio/user.c +6 -1
@@ -54,6 +54,11 @@ static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,* Functions called by main, CPU, or iothread threads */ +uint64_t vfio_user_max_xfer (void ) +{ +return max_xfer_size; +} +static void vfio_user_shutdown (VFIOProxy *proxy) {qio_channel_shutdown (proxy->ioc,QIO_CHANNEL_SHUTDOWN_READ ,NULL );
@@ -251,7 +256,7 @@ void vfio_user_recv(void *opaque) *reply->msg = msg; data = (char *)reply->msg +sizeof (msg); }else { -if (msg.size > max_xfer_size) { +if (msg.size > max_xfer_size +sizeof (VFIOUserDMARW)) {error_setg (&local_err,"vfio_user_recv request larger than max" ); goto fatal; }
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 14/16] vfio-user: dma read/write operations
Quoted 5 lines · Elena Ufimtseva
hw/vfio/pci.c +0
-0
hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3413
Quoted 9 lines
hw/vfio/pci.c +4
-1@@ -3406,11 +3406,72 @@ type_init(register_vfio_pci_dev_type)* vfio-user routines. */ -static int vfio_user_pci_process_req (void *opaque,char *buf,VFIOUserFDs *fds) +static int vfio_user_dma_read (VFIOPCIDevice *vdev,VFIOUserDMARW *msg) { +PCIDevice *pdev = &vdev->pdev; +char *buf; +int size = msg->count +sizeof (VFIOUserDMARW);
The caller has only checked that hdr->size is large enough for VFIOUserHdr, not VFIOUserDMARW. We must not access VFIOUserDMARW fields until this has been checked.
Size should be size_t to avoid signedness issues.
Even then, this can overflow on 32-bit hosts so I suggest moving this arithmetic expression below the msg->count > vfio_user_max_xfer() check. That way it's clear that overflow cannot happen.
hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3420
Quoted 7 lines
hw/vfio/pci.c +7
-0+ +if (msg->hdr.flags &VFIO_USER_NO_REPLY ) { +return -EINVAL ; + } +if (msg->count >vfio_user_max_xfer ()) { +return -E2BIG ; + }
Does vfio-user allow the request to be smaller than the reply? In other words, is it okay that we're not checking msg->count against hdr->size?
hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3425
Quoted 5 lines
hw/vfio/pci.c +5
-0+ + buf =g_malloc0 (size); +memcpy (buf, msg,sizeof (*msg)); + +pci_dma_read (pdev, msg->offset, buf +sizeof (*msg), msg->count);
The vfio-user spec doesn't go into errors but pci_dma_read() can return errors. Hmm...
hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3436
Quoted 11 lines
hw/vfio/pci.c +8
-0+ +vfio_user_send_reply (vdev->vbasedev.proxy, buf, size); +g_free (buf);return 0 ; } +static int vfio_user_dma_write (VFIOPCIDevice *vdev, +VFIOUserDMARW *msg) +{ +PCIDevice *pdev = &vdev->pdev; +char *buf = (char *)msg +sizeof (*msg);
Or:
char *buf = msg->data;
hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3441
Quoted 5 lines
hw/vfio/pci.c +5
-0+ +/* make sure transfer count isn't larger than the message data */ +if (msg->count > msg->hdr.size -sizeof (*msg)) { +return -E2BIG ; + }
msg->count cannot be accessed until we have checked that msg->hdr.size is large enough for VFIOUserDMARW. Adding the check also eliminates the underflow in the subtraction if msg->hdr.size was smaller than sizeof(VFIOUserDMARW).
Quoted 8 lines
hw/vfio/pci.c +8
-0+ +pci_dma_write (pdev, msg->offset, buf, msg->count); + +if ((msg->hdr.flags &VFIO_USER_NO_REPLY ) ==0 ) { +vfio_user_send_reply (vdev->vbasedev.proxy, (char *)msg, +sizeof (msg->hdr)); + } +return 0 ;
hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3459
Quoted 10 lines
hw/vfio/pci.c +10
-0+} + +static int vfio_user_pci_process_req (void *opaque,char *buf,VFIOUserFDs *fds) +{ +VFIOPCIDevice *vdev = opaque; +VFIOUserHdr *hdr = (VFIOUserHdr *)buf; +int ret; + +if (fds->recv_fds !=0 ) { +return -EINVAL ;
Where are the fds closed?
Re: [PATCH RFC v2 14/16] vfio-user: dma read/write operations
Quoted 8 lines · Stefan Hajnoczi
hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3425
Quoted 5 lines
hw/vfio/pci.c +5
-0+ + buf =g_malloc0 (size); +memcpy (buf, msg,sizeof (*msg)); + +pci_dma_read (pdev, msg->offset, buf +sizeof (*msg), msg->count);The vfio-user spec doesn't go into errors but
pci_dma_read()can return errors. Hmm...
It's certainly under-specified in the spec, but in terms of the library, we do return EINVAL if we decide something invalid happened...
regards
john
RFC PATCH v2 15/16 vfio-user: pci reset
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user.h | 1 + hw/vfio/pci.c | 29 ++++++++++++++++++++++++++--- hw/vfio/user.c | 12 ++++++++++++ 3 files changed, 39 insertions(+), 3 deletions(-)
hw/vfio/user.h +1 -0
@@ -86,6 +86,7 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data);int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index,uint64_t offset,uint32_t count,void *data); +void vfio_user_reset (VFIODevice *vbasedev);void vfio_user_drain_reqs (VFIOProxy *proxy);#endif /* VFIO_USER_H */
hw/vfio/pci.c +26 -3
@@ -2229,8 +2229,9 @@ static void vfio_pci_pre_reset(VFIOPCIDevice *vdev)static void vfio_pci_post_reset (VFIOPCIDevice *vdev) { +VFIODevice *vbasedev = &vdev->vbasedev;Error *err =NULL ; -int nr; +int ret, nr;vfio_intx_enable (vdev, &err);if (err) {
@@ -2238,11 +2239,18 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev) }for (nr =0 ; nr <PCI_NUM_REGIONS -1 ; ++nr) { -off_t addr = vdev->config_offset +PCI_BASE_ADDRESS_0 + (4 * nr); +off_t addr =PCI_BASE_ADDRESS_0 + (4 * nr);uint32_t val =0 ;uint32_t len =sizeof (val); -if (pwrite (vdev->vbasedev.fd, &val, len, addr) != len) { +if (vbasedev->proxy !=NULL ) { + ret =vfio_user_region_write (vbasedev,VFIO_PCI_CONFIG_REGION_INDEX , + addr, len, &val); + }else { + ret =pwrite (vdev->vbasedev.fd, &val, len, + vdev->config_offset + addr); + } +if (ret != len) {error_report ("%s(%s) reset bar %d failed: %m" , __func__, vdev->vbasedev.name, nr); }
@@ -3684,6 +3692,20 @@ static void vfio_user_instance_finalize(Object *obj)vfio_user_disconnect (vbasedev->proxy); } +static void vfio_user_pci_reset (DeviceState *dev) +{ +VFIOPCIDevice *vdev =VFIO_PCI_BASE (dev); +VFIODevice *vbasedev = &vdev->vbasedev; + +vfio_pci_pre_reset (vdev); + +if (vbasedev->reset_works) { +vfio_user_reset (vbasedev); + } + +vfio_pci_post_reset (vdev); +} +static Property vfio_user_pci_dev_properties[] = {DEFINE_PROP_STRING ("socket" , VFIOUserPCIDevice, sock_name),DEFINE_PROP_BOOL ("secure-dma" , VFIOUserPCIDevice, secure_dma, false),
@@ -3695,6 +3717,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data)DeviceClass *dc =DEVICE_CLASS (klass);PCIDeviceClass *pdc =PCI_DEVICE_CLASS (klass); + dc->reset = vfio_user_pci_reset;device_class_set_props (dc, vfio_user_pci_dev_properties); dc->desc ="VFIO over socket PCI device assignment" ; pdc->realize = vfio_user_pci_realize;
hw/vfio/user.c +12 -0
@@ -1045,3 +1045,15 @@ int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,return count; } + +void vfio_user_reset (VFIODevice *vbasedev) +{ +VFIOUserHdr msg; + +vfio_user_request_msg (&msg,VFIO_USER_DEVICE_RESET ,sizeof (msg),0 ); + +vfio_user_send_recv (vbasedev->proxy, &msg,NULL ,0 ,0 ); +if (msg.flags &VFIO_USER_ERROR ) { +error_printf ("reset reply error %d\n" , msg.error_reply); + } +}
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 15/16] vfio-user: pci reset
Quoted 48 lines · Elena Ufimtseva
From: John Johnson <[email protected]>
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/vfio/user.h | 1 + hw/vfio/pci.c | 29 ++++++++++++++++++++++++++--- hw/vfio/user.c | 12 ++++++++++++ 3 files changed, 39 insertions(+), 3 deletions(-)hw/vfio/user.h +1
-0@@ -86,6 +86,7 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data);int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index,uint64_t offset,uint32_t count,void *data); +void vfio_user_reset (VFIODevice *vbasedev);void vfio_user_drain_reqs (VFIOProxy *proxy);#endif /* VFIO_USER_H */ hw/vfio/pci.c +3
-3@@ -2229,8 +2229,9 @@ static void vfio_pci_pre_reset(VFIOPCIDevice *vdev)static void vfio_pci_post_reset (VFIOPCIDevice *vdev) { +VFIODevice *vbasedev = &vdev->vbasedev;Error *err =NULL ; -int nr; +int ret, nr;vfio_intx_enable (vdev, &err);if (err) {@@ -2238,11 +2239,18 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev) }for (nr =0 ; nr <PCI_NUM_REGIONS -1 ; ++nr) { -off_t addr = vdev->config_offset +PCI_BASE_ADDRESS_0 + (4 * nr); +off_t addr =PCI_BASE_ADDRESS_0 + (4 * nr);uint32_t val =0 ;uint32_t len =sizeof (val); -if (pwrite (vdev->vbasedev.fd, &val, len, addr) != len) {
hw/vfio/pci.c static void vfio_pci_post_reset(VFIOPCIDevice *vdev) · line 2255
Quoted 10 lines
hw/vfio/pci.c +8
-0+if (vbasedev->proxy !=NULL ) { + ret =vfio_user_region_write (vbasedev,VFIO_PCI_CONFIG_REGION_INDEX , + addr, len, &val); + }else { + ret =pwrite (vdev->vbasedev.fd, &val, len, + vdev->config_offset + addr); + } +if (ret != len) {error_report ("%s(%s) reset bar %d failed: %m" , __func__, vdev->vbasedev.name, nr);
The %m format string assumes vfio_user_region_write() sets errno. I don't think it does. We're relying on vfio_user_region_write() never failing here, which is true at the moment but not nice.
Quoted 35 lines
hw/vfio/pci.c +15
-0}@@ -3684,6 +3692,20 @@ static void vfio_user_instance_finalize(Object *obj)vfio_user_disconnect (vbasedev->proxy); } +static void vfio_user_pci_reset (DeviceState *dev) +{ +VFIOPCIDevice *vdev =VFIO_PCI_BASE (dev); +VFIODevice *vbasedev = &vdev->vbasedev; + +vfio_pci_pre_reset (vdev); + +if (vbasedev->reset_works) { +vfio_user_reset (vbasedev); + } + +vfio_pci_post_reset (vdev); +} +static Property vfio_user_pci_dev_properties[] = {DEFINE_PROP_STRING ("socket" , VFIOUserPCIDevice, sock_name),DEFINE_PROP_BOOL ("secure-dma" , VFIOUserPCIDevice, secure_dma, false),@@ -3695,6 +3717,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data)DeviceClass *dc =DEVICE_CLASS (klass);PCIDeviceClass *pdc =PCI_DEVICE_CLASS (klass); + dc->reset = vfio_user_pci_reset;device_class_set_props (dc, vfio_user_pci_dev_properties); dc->desc ="VFIO over socket PCI device assignment" ; pdc->realize = vfio_user_pci_realize;hw/vfio/user.c +0
-0
hw/vfio/user.c int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, · line 1051
Quoted 7 lines
hw/vfio/user.c +4
-0@@ -1045,3 +1045,15 @@ int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,return count; } + +void vfio_user_reset (VFIODevice *vbasedev) +{ +VFIOUserHdr msg;
Maybe add "= {}" to ensure it's zero-initialized?
Quoted 11 lines
hw/vfio/user.c +8
-0+ +vfio_user_request_msg (&msg,VFIO_USER_DEVICE_RESET ,sizeof (msg),0 ); + +vfio_user_send_recv (vbasedev->proxy, &msg,NULL ,0 ,0 ); +if (msg.flags &VFIO_USER_ERROR ) { +error_printf ("reset reply error %d\n" , msg.error_reply); + } +}Signature · 1 line
2.25.1
RFC PATCH v2 16/16 vfio-user: migration support
In reply to Elena UfimtsevaFrom: John Johnson <[email protected]>
--- hw/vfio/user-protocol.h | 18 +++++++++++++++++ hw/vfio/user.h | 3 +++ hw/vfio/common.c | 23 ++++++++++++++++----- hw/vfio/migration.c | 34 +++++++++++++++++-------------- hw/vfio/pci.c | 12 +++++++++++ hw/vfio/user.c | 45 +++++++++++++++++++++++++++++++++++++++++ 6 files changed, 115 insertions(+), 20 deletions(-)
hw/vfio/user-protocol.h +18 -0
@@ -182,6 +182,10 @@ typedef struct {char data[]; } VFIOUserDMARW; +/* +* VFIO_USER_DIRTY_PAGES +*/ +/*imported from struct vfio_bitmap */ typedef struct { uint64_t pgsize;
@@ -189,4 +193,18 @@ typedef struct { char data[]; } VFIOUserBitmap; +/* imported from struct vfio_iommu_type1_dirty_bitmap_get */ +typedef struct { +uint64_t iova; +uint64_t size; +VFIOUserBitmap bitmap; +}VFIOUserBitmapRange ; + +/* imported from struct vfio_iommu_type1_dirty_bitmap */ +typedef struct { +VFIOUserHdr hdr; +uint32_t argsz; +uint32_t flags; +}VFIOUserDirtyPages ; +#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +3 -0
@@ -87,6 +87,9 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,int vfio_user_region_write (VFIODevice *vbasedev,uint32_t index,uint64_t offset,uint32_t count,void *data);void vfio_user_reset (VFIODevice *vbasedev); +int vfio_user_dirty_bitmap (VFIOProxy *proxy, +struct vfio_iommu_type1_dirty_bitmap *bitmap, +struct vfio_iommu_type1_dirty_bitmap_get *range);void vfio_user_drain_reqs (VFIOProxy *proxy);#endif /* VFIO_USER_H */
hw/vfio/common.c +18 -5
@@ -1303,10 +1303,19 @@ static void vfio_set_dirty_page_tracking(VFIOContainer *container, bool start) dirty.flags =VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP ; } - ret =ioctl (container->fd,VFIO_IOMMU_DIRTY_PAGES , &dirty); -if (ret) { -error_report ("Failed to set dirty tracking flag 0x%x errno: %d" , - dirty.flags, errno); +if (container->proxy !=NULL ) { + ret =vfio_user_dirty_bitmap (container->proxy, &dirty,NULL ); +if (ret) { +error_report ("Failed to set dirty tracking flag 0x%x errno: %d" , + dirty.flags, -ret); + } + }else { + ret =ioctl (container->fd,VFIO_IOMMU_DIRTY_PAGES , &dirty); +if (ret) { +error_report ("Failed to set dirty tracking flag 0x%x errno: %d" , + dirty.flags, errno); + ret = -errno; + } } }
@@ -1356,7 +1365,11 @@ static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, goto err_out; } - ret =ioctl (container->fd,VFIO_IOMMU_DIRTY_PAGES , dbitmap); +if (container->proxy !=NULL ) { + ret =vfio_user_dirty_bitmap (container->proxy, dbitmap, range); + }else { + ret =ioctl (container->fd,VFIO_IOMMU_DIRTY_PAGES , dbitmap); + }if (ret) { error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova,
hw/vfio/migration.c +19 -15
@@ -27,6 +27,7 @@#include "pci.h" #include "trace.h" #include "hw/hw.h" +#include "user.h" /* * Flags to be used as unique delimiters for VFIO devices in the migration
@@ -49,10 +50,18 @@ static int64_t bytes_transferred;static inline int vfio_mig_access (VFIODevice *vbasedev,void *val,int count,off_t off,bool iswrite) { +VFIORegion *region = &vbasedev->migration->region;int ret; - ret = iswrite ?pwrite (vbasedev->fd, val, count, off) : -pread (vbasedev->fd, val, count, off); +if (vbasedev->proxy !=NULL ) { + ret = iswrite ? +vfio_user_region_write (vbasedev, region->nr, off, count, val) : +vfio_user_region_read (vbasedev, region->nr, off, count, val); + }else { + off += region->fd_offset; + ret = iswrite ?pwrite (vbasedev->fd, val, count, off) : +pread (vbasedev->fd, val, count, off); + }if (ret < count) { error_report("vfio_mig_%s %d byte %s: failed at offset 0x%" HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count,
@@ -111,9 +120,7 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask, uint32_t value) {VFIOMigration *migration = vbasedev->migration; -VFIORegion *region = &migration->region; -off_t dev_state_off = region->fd_offset + -VFIO_MIG_STRUCT_OFFSET (device_state); +off_t dev_state_off =VFIO_MIG_STRUCT_OFFSET (device_state);uint32_t device_state;int ret;
@@ -201,13 +208,13 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size)int ret; ret =vfio_mig_read (vbasedev, &data_offset,sizeof (data_offset), - region->fd_offset +VFIO_MIG_STRUCT_OFFSET (data_offset)); +VFIO_MIG_STRUCT_OFFSET (data_offset));if (ret <0 ) {return ret; } ret =vfio_mig_read (vbasedev, &data_size,sizeof (data_size), - region->fd_offset +VFIO_MIG_STRUCT_OFFSET (data_size)); +VFIO_MIG_STRUCT_OFFSET (data_size));if (ret <0 ) {return ret; }
@@ -233,8 +240,7 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) } buf_allocated = true; - ret =vfio_mig_read (vbasedev, buf, sec_size, - region->fd_offset + data_offset); + ret =vfio_mig_read (vbasedev, buf, sec_size, data_offset);if (ret <0 ) {g_free (buf);return ret;
@@ -269,7 +275,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, do { ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); + VFIO_MIG_STRUCT_OFFSET(data_offset)); if (ret < 0) { return ret; }
@@ -309,8 +315,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,qemu_get_buffer (f, buf, sec_size);if (buf_alloc) { - ret =vfio_mig_write (vbasedev, buf, sec_size, - region->fd_offset + data_offset); + ret =vfio_mig_write (vbasedev, buf, sec_size, data_offset);g_free (buf);if (ret <0 ) {
@@ -322,7 +327,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, } ret =vfio_mig_write (vbasedev, &report_size,sizeof (report_size), - region->fd_offset +VFIO_MIG_STRUCT_OFFSET (data_size)); +VFIO_MIG_STRUCT_OFFSET (data_size));if (ret <0 ) {return ret; }
@@ -334,12 +339,11 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,static int vfio_update_pending (VFIODevice *vbasedev) {VFIOMigration *migration = vbasedev->migration; -VFIORegion *region = &migration->region;uint64_t pending_bytes =0 ;int ret; ret =vfio_mig_read (vbasedev, &pending_bytes,sizeof (pending_bytes), - region->fd_offset +VFIO_MIG_STRUCT_OFFSET (pending_bytes)); +VFIO_MIG_STRUCT_OFFSET (pending_bytes));if (ret <0 ) { migration->pending_bytes =0 ;return ret;
hw/vfio/pci.c +12 -0
@@ -3658,6 +3658,13 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } } +if (!pdev->failover_pair_id) { + ret =vfio_migration_probe (&vdev->vbasedev, errp); +if (ret) { +error_report ("%s: Migration disabled" , vdev->vbasedev.name); + } + } +vfio_register_err_notifier (vdev);vfio_register_req_notifier (vdev);
@@ -3709,6 +3716,11 @@ static void vfio_user_pci_reset(DeviceState *dev)static Property vfio_user_pci_dev_properties[] = {DEFINE_PROP_STRING ("socket" , VFIOUserPCIDevice, sock_name),DEFINE_PROP_BOOL ("secure-dma" , VFIOUserPCIDevice, secure_dma, false), +DEFINE_PROP_BOOL ("x-enable-migration" , VFIOPCIDevice, + vbasedev.enable_migration, false), +DEFINE_PROP_ON_OFF_AUTO ("x-pre-copy-dirty-page-tracking" , VFIOPCIDevice, + vbasedev.pre_copy_dirty_page_tracking, +ON_OFF_AUTO_ON ),DEFINE_PROP_END_OF_LIST (), };
hw/vfio/user.c +45 -0
48 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH RFC v2 16/16] vfio-user: migration support
hw/vfio/common.c static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, · line 1375
Quoted 13 lines · Elena Ufimtseva
hw/vfio/common.c +5
-1@@ -1356,7 +1365,11 @@ static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, goto err_out; } - ret =ioctl (container->fd,VFIO_IOMMU_DIRTY_PAGES , dbitmap); +if (container->proxy !=NULL ) { + ret =vfio_user_dirty_bitmap (container->proxy, dbitmap, range); + }else { + ret =ioctl (container->fd,VFIO_IOMMU_DIRTY_PAGES , dbitmap); + }if (ret) { error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova,
This error_report() relies on errno. vfio_user_region_write() doesn't set errno.
Quoted 23 lines
hw/vfio/migration.c +4
-2@@ -27,6 +27,7 @@#include "pci.h" #include "trace.h" #include "hw/hw.h" +#include "user.h" /* * Flags to be used as unique delimiters for VFIO devices in the migration @@ -49,10 +50,18 @@ static int64_t bytes_transferred;static inline int vfio_mig_access (VFIODevice *vbasedev,void *val,int count,off_t off,bool iswrite) { +VFIORegion *region = &vbasedev->migration->region;int ret; - ret = iswrite ?pwrite (vbasedev->fd, val, count, off) : -pread (vbasedev->fd, val, count, off); +if (vbasedev->proxy !=NULL ) { + ret = iswrite ?
hw/vfio/migration.c static int64_t bytes_transferred; · line 67
Quoted 10 lines
hw/vfio/migration.c +7
-0+ vfio_user_region_write(vbasedev, region->nr, off, count, val) : +vfio_user_region_read (vbasedev, region->nr, off, count, val); + }else { + off += region->fd_offset; + ret = iswrite ?pwrite (vbasedev->fd, val, count, off) : +pread (vbasedev->fd, val, count, off); + }if (ret < count) { error_report("vfio_mig_%s %d byte %s: failed at offset 0x%" HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count,
Another errno user. I haven't exhaustively audited all the code for these issues. Please take a look.
v2 cover vfio-user server in QEMU
In reply to Elena UfimtsevaHi,
This series depends on the following series from
Elena Ufimtseva <[email protected]>:
[PATCH RFC v2 00/16] vfio-user implementation
Thank you for your feedback for the v1 patches!
https://www.mail-archive.com/[email protected]/msg825021.html
We have incorporated the following feedback from v1 of the review cycle:
[PATCH RFC server v2 01/11] vfio-user: build library
- Using cmake subproject to build libvfio-user
[PATCH RFC server v2 02/11] vfio-user: define vfio-user object
- Added check to confirm that TYPE_REMOTE_MACHINE is used with TYPE_VFU_OBJECT
[PATCH RFC server v2 04/11] vfio-user: find and init PCI device
- Removed call to vfu_pci_set_id()
- Added check to confirm that TYPE_PCI_DEVICE is used with TYPE_VFU_OBJECT
[PATCH RFC server v2 05/11] vfio-user: run vfio-user context
- Using QEMU main-loop to drive the vfu_ctx (using
vfu_get_poll_fd() & qemu_set_fd_handler())
- Set vfu_ctx to non-blocking mode (LIBVFIO_USER_FLAG_ATTACH_NB)
- Modified how QEMU attaches to the vfu_ctx
[PATCH RFC server v2 06/11] handle PCI config space accesses
- Broke-up PCI config space access to 4-byte accesses
[PATCH RFC server v2 07/11] vfio-user: handle DMA mappings
- Received feedback to assert that vfu_dma_info_t->vaddr is not NULL - unable to do it as it appears to be a valid case.
[PATCH RFC server v2 10/11] register handlers to facilitate migration
- Migrate only one device's data per contect
Would appreciate if you could kindly review this v2 series. Looking forward to your comments.
Thank you!
Jagannathan Raman (11):
vfio-user: build library
vfio-user: define vfio-user object
vfio-user: instantiate vfio-user context
vfio-user: find and init PCI device
vfio-user: run vfio-user context
vfio-user: handle PCI config space accesses
vfio-user: handle DMA mappings
vfio-user: handle PCI BAR accesses
vfio-user: handle device interrupts
vfio-user: register handlers to facilitate migration
vfio-user: acceptance test
configure | 11 + meson.build | 28 ++ qapi/qom.json | 20 +- include/hw/remote/iohub.h | 2 + migration/savevm.h | 2 + hw/remote/iohub.c | 5 + hw/remote/vfio-user-obj.c | 803 ++++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 73 ++++ .gitmodules | 3 + MAINTAINERS | 9 + hw/remote/meson.build | 3 + hw/remote/trace-events | 10 + subprojects/libvfio-user | 1 + tests/acceptance/vfio-user.py | 94 +++++ 14 files changed, 1062 insertions(+), 2 deletions(-) create mode 100644 hw/remote/vfio-user-obj.c create mode 160000 subprojects/libvfio-user create mode 100644 tests/acceptance/vfio-user.py
Signature · 1 line
1.8.3.1
RFC PATCH v2 1/11 vfio-user: build library
add the libvfio-user library as a submodule. build it as a cmake subproject.
--- configure | 11 +++++++++++ meson.build | 28 ++++++++++++++++++++++++++++ .gitmodules | 3 +++ MAINTAINERS | 7 +++++++ hw/remote/meson.build | 2 ++ subprojects/libvfio-user | 1 + 6 files changed, 52 insertions(+) create mode 160000 subprojects/libvfio-user
configure +11 -0
@@ -4291,6 +4291,17 @@ but not implemented on your system" fi ########################################## +# check for multiprocess + +case "$multiprocess" in + auto | enabled ) + if test "$git_submodules_action" != "ignore"; then + git_submodules="${git_submodules} libvfio-user" + fi + ;; +esac + +########################################## # End of CC checks # After here, no more $cc or $ld runs
meson.build +28 -0
@@ -1898,6 +1898,34 @@ if get_option('cfi') and slirp_opt == 'system' + ' Please configure with --enable-slirp=git') endif +vfiouser = not_found +if have_system and multiprocess_allowed + have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/Makefile') + + if not have_internal + error('libvfio-user source not found - please pull git submodule') + endif + + json_c = dependency('json-c', required: false) + if not json_c.found() + json_c = dependency('libjson-c') + endif + + cmake = import('cmake') + + vfiouser_subproj = cmake.subproject('libvfio-user') + + vfiouser_sl = vfiouser_subproj.dependency('vfio-user-static') + + # Although cmake links the json-c library with vfio-user-static + # target, that info is not available to meson via cmake.subproject. + # As such, we have to separately declare the json-c dependency here. + # This appears to be a current limitation of using cmake inside meson. + # libvfio-user is planning a switch to meson in the future, which + # would address this item automatically. + vfiouser = declare_dependency(dependencies: [vfiouser_sl, json_c]) +endif + fdt = not_found fdt_opt = get_option('fdt') if have_system
.gitmodules +3 -0
@@ -64,3 +64,6 @@ [submodule "roms/vbootrom"] path = roms/vbootrom url = https://gitlab.com/qemu-project/vbootrom.git +[submodule "subprojects/libvfio-user"] + path = subprojects/libvfio-user + url = https://github.com/nutanix/libvfio-user.git
MAINTAINERS +7 -0
@@ -3361,6 +3361,13 @@ F: semihosting/ F: include/semihosting/ F: tests/tcg/multiarch/arm-compat-semi/ +libvfio-user Library +M: Thanos Makatos <[email protected]> +M: John Levon <[email protected]> +T: https://github.com/nutanix/libvfio-user.git +S: Maintained +F: subprojects/libvfio-user/* + Multi-process QEMU M: Elena Ufimtseva <[email protected]> M: Jagannathan Raman <[email protected]>
hw/remote/meson.build +2 -0
@@ -7,6 +7,8 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: vfiouser) + specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c')) specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
subprojects/libvfio-user new file +1 -0
@@ -0,0 +1 @@ +Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 01/11] vfio-user: build library
Quoted 52 lines · Jag Raman
add the libvfio-user library as a submodule. build it as a cmake subproject.
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
configure | 11 +++++++++++ meson.build | 28 ++++++++++++++++++++++++++++ .gitmodules | 3 +++ MAINTAINERS | 7 +++++++ hw/remote/meson.build | 2 ++ subprojects/libvfio-user | 1 + 6 files changed, 52 insertions(+)create mode 160000 subprojects/libvfio-user
configure +0
-0@@ -4291,6 +4291,17 @@ but not implemented on your system"fi
##########################################
configure +11
-0+# check for multiprocess + +case "$multiprocess" in + auto | enabled ) + if test "$git_submodules_action" != "ignore"; then + git_submodules="${git_submodules} libvfio-user" + fi + ;; +esac + +########################################### End of CC checks
# After here, no more $cc or $ld runsmeson.build +0
-0@@ -1898,6 +1898,34 @@ if get_option('cfi') and slirp_opt == 'system' + ' Please configure with --enable-slirp=git')endif
meson.build +2
-0+vfiouser = not_found +if have_system and multiprocess_allowed
meson.build if get_option('cfi') and slirp_opt == 'system' · line 1912
Quoted 10 lines
meson.build +10
-0+ have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/Makefile') + + if not have_internal + error('libvfio-user source not found - please pull git submodule') + endif + + json_c = dependency('json-c', required: false) + if not json_c.found() + json_c = dependency('libjson-c') + endif
One of the things we’re wondering is about this json-c package that we need to build libvfio-user library.
The gitlab runners typically don’t have this package installed, as such the gitlab builds fail. Wondering if there's a way to install this package for all QEMU builds?
We checked out the various jobs defined in “.gitlab-ci.d/buildtest.yml” - there is a “before_script” keyword which we could use to install this package. The “before_script” keyword appears to be run every time before a job’s script is executed. But this option appears to be per job/build. Wondering if there's a distro-independent global way to install a required package for all builds.
Thank you!
--
Jag
Quoted 71 lines
meson.build +16
-0+ + cmake = import('cmake') + + vfiouser_subproj = cmake.subproject('libvfio-user') + + vfiouser_sl = vfiouser_subproj.dependency('vfio-user-static') + + # Although cmake links the json-c library with vfio-user-static + # target, that info is not available to meson via cmake.subproject. + # As such, we have to separately declare the json-c dependency here. + # This appears to be a current limitation of using cmake inside meson. + # libvfio-user is planning a switch to meson in the future, which + # would address this item automatically. + vfiouser = declare_dependency(dependencies: [vfiouser_sl, json_c]) +endif +fdt =
not_foundfdt_opt=get_option('fdt')
ifhave_system.gitmodules +0
-0@@ -64,3 +64,6 @@[submodule "roms/vbootrom"]
path = roms/vbootrom
url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "subprojects/libvfio-user"].gitmodules +2
-0+ path = subprojects/libvfio-user + url = https://github.com/nutanix/libvfio-user.gitMAINTAINERS +0
-0@@ -3361,6 +3361,13 @@ F: semihosting/F:
include/semihosting/
F: tests/tcg/multiarch/arm-compat-semi/MAINTAINERS +7
-0+libvfio-user Library +M: Thanos Makatos <[email protected]> +M: John Levon <[email protected]> +T: https://github.com/nutanix/libvfio-user.git +S: Maintained +F: subprojects/libvfio-user/* +Multi-process QEMU
M: Elena Ufimtseva <[email protected]>
M: Jagannathan Raman <[email protected]>hw/remote/meson.build +2
-0@@ -7,6 +7,8 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: vfiouser) + specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c')) specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))subprojects/libvfio-user new file +1
-0@@ -0,0 +1 @@ +Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 01/11] vfio-user: build library
In reply to Jagannathan RamanQuoted 8 lines · Jagannathan Raman
meson.build +0
-0@@ -1898,6 +1898,34 @@ if get_option('cfi') and slirp_opt == 'system' + ' Please configure with --enable-slirp=git') endif
meson.build if get_option('cfi') and slirp_opt == 'system' · line 1910
Quoted 10 lines
meson.build +10
-0+vfiouser = not_found +if have_system and multiprocess_allowed + have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/Makefile') + + if not have_internal + error('libvfio-user source not found - please pull git submodule') + endif + + json_c = dependency('json-c', required: false) + if not json_c.found()
Indentation is off.
Quoted 26 lines
meson.build +18
-0+ json_c = dependency('libjson-c') + endif + + cmake = import('cmake') + + vfiouser_subproj = cmake.subproject('libvfio-user') + + vfiouser_sl = vfiouser_subproj.dependency('vfio-user-static') + + # Although cmake links the json-c library with vfio-user-static + # target, that info is not available to meson via cmake.subproject. + # As such, we have to separately declare the json-c dependency here. + # This appears to be a current limitation of using cmake inside meson. + # libvfio-user is planning a switch to meson in the future, which + # would address this item automatically. + vfiouser = declare_dependency(dependencies: [vfiouser_sl, json_c]) +endif + fdt = not_found fdt_opt = get_option('fdt') if have_system.gitmodules +0
-0
.gitmodules line 69
Quoted 6 lines
.gitmodules +3
-0@@ -64,3 +64,6 @@ [submodule "roms/vbootrom"] path = roms/vbootrom url = https://gitlab.com/qemu-project/vbootrom.git +[submodule "subprojects/libvfio-user"] + path = subprojects/libvfio-user + url = https://github.com/nutanix/libvfio-user.git
Once this is merged I'll set up a gitlab.com/qemu-project/libvfio-user.git mirror. This ensures that no matter what happens with upstream libvfio-user.git, the source code that QEMU builds against will remain archived/available.
Quoted 5 lines
MAINTAINERS +0
-0
MAINTAINERS F: semihosting/ · line 3369
Quoted 9 lines
MAINTAINERS +6
-0@@ -3361,6 +3361,13 @@ F: semihosting/ F: include/semihosting/ F: tests/tcg/multiarch/arm-compat-semi/ +libvfio-user Library +M: Thanos Makatos <[email protected]> +M: John Levon <[email protected]> +T: https://github.com/nutanix/libvfio-user.git +S: Maintained +F: subprojects/libvfio-user/*
A MAINTAINERS entry isn't necessary for git submodules. This could become outdated. People should look at the upstream project instead for information on maintainership and how to contribute.
Re: [PATCH RFC server v2 01/11] vfio-user: build library
Quoted 18 lines · Stefan Hajnoczi
.gitmodules line 69
Quoted 11 lines · Jagannathan Raman
.gitmodules +3
-0@@ -64,3 +64,6 @@ [submodule "roms/vbootrom"] path = roms/vbootrom url = https://gitlab.com/qemu-project/vbootrom.git +[submodule "subprojects/libvfio-user"] + path = subprojects/libvfio-user + url = https://github.com/nutanix/libvfio-user.gitOnce this is merged I'll set up a gitlab.com/qemu-project/libvfio-user.git mirror. This ensures that no matter what happens with upstream libvfio-user.git, the source code that QEMU builds against will remain archived/available.
Can we do it the other way around? When the series is OK to be merged, setup the https://gitlab.com/qemu-project/libvfio-user.git mirror and have the submodule point to it?
Re: [PATCH RFC server v2 01/11] vfio-user: build library
Quoted 20 lines · Philippe Mathieu-Daudé
Quoted 18 lines · Stefan Hajnoczi
.gitmodules line 69
Quoted 11 lines · Jagannathan Raman
.gitmodules +3
-0@@ -64,3 +64,6 @@ [submodule "roms/vbootrom"] path = roms/vbootrom url = https://gitlab.com/qemu-project/vbootrom.git +[submodule "subprojects/libvfio-user"] + path = subprojects/libvfio-user + url = https://github.com/nutanix/libvfio-user.gitOnce this is merged I'll set up a gitlab.com/qemu-project/libvfio-user.git mirror. This ensures that no matter what happens with upstream libvfio-user.git, the source code that QEMU builds against will remain archived/available.
Quoted 3 lines
Can we do it the other way around? When the series is OK to be merged, setup the https://gitlab.com/qemu-project/libvfio-user.git mirror and have the submodule point to it?
Yes, good idea.
Stefan
Re: [PATCH RFC server v2 01/11] vfio-user: build library
In reply to Jagannathan RamanQuoted 23 lines · Jagannathan Raman
add the libvfio-user library as a submodule. build it as a cmake subproject.
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- configure | 11 +++++++++++ meson.build | 28 ++++++++++++++++++++++++++++ .gitmodules | 3 +++ MAINTAINERS | 7 +++++++ hw/remote/meson.build | 2 ++ subprojects/libvfio-user | 1 + 6 files changed, 52 insertions(+) create mode 160000 subprojects/libvfio-usersubprojects/libvfio-user new file +0
-0
subprojects/libvfio-user line 1
Quoted 1 line
subprojects/libvfio-user new file +1
-0@@ -0,0 +1 @@ +Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446
Could we point to a sha1 of a released tag instead?
Re: [PATCH RFC server v2 01/11] vfio-user: build library
Quoted 21 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Jagannathan Raman
add the libvfio-user library as a submodule. build it as a cmake subproject.
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
configure | 11 +++++++++++ meson.build | 28 ++++++++++++++++++++++++++++ .gitmodules | 3 +++ MAINTAINERS | 7 +++++++ hw/remote/meson.build | 2 ++ subprojects/libvfio-user | 1 + 6 files changed, 52 insertions(+)create mode 160000 subprojects/libvfio-user
Quoted 10 lines
subprojects/libvfio-user line 1
Quoted 8 lines
subprojects/libvfio-user new file +1
-0@@ -0,0 +1 @@ +Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446Could we point to a sha1 of a released tag instead?
OK, will do.
--
Jag
Quoted 1 line
Re: [PATCH RFC server v2 01/11] vfio-user: build library
In reply to Philippe Mathieu-DaudéQuoted 17 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Jagannathan Raman
add the libvfio-user library as a submodule. build it as a cmake subproject.
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- configure | 11 +++++++++++ meson.build | 28 ++++++++++++++++++++++++++++ .gitmodules | 3 +++ MAINTAINERS | 7 +++++++ hw/remote/meson.build | 2 ++ subprojects/libvfio-user | 1 + 6 files changed, 52 insertions(+) create mode 160000 subprojects/libvfio-user
Quoted 9 lines
subprojects/libvfio-user line 1
Quoted 7 lines
subprojects/libvfio-user new file +1
-0@@ -0,0 +1 @@ +Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446Could we point to a sha1 of a released tag instead?
We don't have releases (yet) partly because we haven't yet stabilized the API.
regards
john
Re: [PATCH RFC server v2 01/11] vfio-user: build library
Quoted 28 lines · John Levon
Quoted 17 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Jagannathan Raman
add the libvfio-user library as a submodule. build it as a cmake subproject.
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- configure | 11 +++++++++++ meson.build | 28 ++++++++++++++++++++++++++++ .gitmodules | 3 +++ MAINTAINERS | 7 +++++++ hw/remote/meson.build | 2 ++ subprojects/libvfio-user | 1 + 6 files changed, 52 insertions(+) create mode 160000 subprojects/libvfio-userQuoted 9 lines
subprojects/libvfio-user line 1
Quoted 7 lines
subprojects/libvfio-user new file +1
-0@@ -0,0 +1 @@ +Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446Could we point to a sha1 of a released tag instead?
Quoted 1 line
We don't have releases (yet) partly because we haven't yet stabilized the API.
OK. Maybe acceptable, up to the maintainer then ¯\_(ツ)_/¯
RFC PATCH v2 2/11 vfio-user: define vfio-user object
In reply to Jagannathan RamanDefine vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object
--- qapi/qom.json | 20 ++++++- hw/remote/vfio-user-obj.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + hw/remote/meson.build | 1 + hw/remote/trace-events | 3 + 5 files changed, 168 insertions(+), 2 deletions(-) create mode 100644 hw/remote/vfio-user-obj.c
qapi/qom.json +18 -2
@@ -689,6 +689,20 @@ 'data': { 'fd': 'str', 'devid': 'str' } } ## +# @VfioUserProperties: +# +# Properties for vfio-user objects. +# +# @socket: path to be used as socket by the libvfiouser library +# +# @devid: the id of the device to be associated with the file descriptor +# +# Since: 6.0 +## +{ 'struct': 'VfioUserProperties', + 'data': { 'socket': 'str', 'devid': 'str' } } + +## # @RngProperties: # # Properties for objects of classes derived from rng.
@@ -812,7 +826,8 @@ 'tls-creds-psk', 'tls-creds-x509', 'tls-cipher-suites', - 'x-remote-object' + 'x-remote-object', + 'vfio-user' ] } ##
@@ -868,7 +883,8 @@ 'tls-creds-psk': 'TlsCredsPskProperties', 'tls-creds-x509': 'TlsCredsX509Properties', 'tls-cipher-suites': 'TlsCredsProperties', - 'x-remote-object': 'RemoteObjectProperties' + 'x-remote-object': 'RemoteObjectProperties', + 'vfio-user': 'VfioUserProperties' } } ##
hw/remote/vfio-user-obj.c new file +145 -0
145 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -3391,6 +3391,7 @@ F: hw/remote/proxy-memory-listener.c F: include/hw/remote/proxy-memory-listener.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h +F: hw/remote/vfio-user-obj.c EBPF: M: Jason Wang <[email protected]>
hw/remote/meson.build +1 -0
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('vfio-user-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: vfiouser)
hw/remote/trace-events +3 -0
@@ -2,3 +2,6 @@ mpqemu_send_io_error(int cmd, int size, int nfds) "send command %d size %d, %d file descriptors to remote process" mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, %d file descriptors to remote process" + +# vfio-user-obj.c +vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 02/11] vfio-user: define vfio-user object
Quoted 26 lines · Jagannathan Raman
Define vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- qapi/qom.json | 20 ++++++- hw/remote/vfio-user-obj.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + hw/remote/meson.build | 1 + hw/remote/trace-events | 3 + 5 files changed, 168 insertions(+), 2 deletions(-) create mode 100644 hw/remote/vfio-user-obj.cqapi/qom.json +2
-0@@ -689,6 +689,20 @@ 'data': { 'fd': 'str', 'devid': 'str' } } ## +# @VfioUserProperties: +#
qapi/qom.json line 703
Quoted 10 lines
qapi/qom.json +10
-0+# Properties for vfio-user objects. +# +# @socket: path to be used as socket by the libvfiouser library +# +# @devid: the id of the device to be associated with the file descriptor +# +# Since: 6.0 +## +{ 'struct': 'VfioUserProperties', + 'data': { 'socket': 'str', 'devid': 'str' } }
Please use 'SocketAddress' for socket instead of 'str'. That way file descriptor passing is easy to support and additional socket address families can be supported in the future.
Quoted 16 lines
qapi/qom.json +4
-1+ +## # @RngProperties: # # Properties for objects of classes derived from rng.@@ -812,7 +826,8 @@ 'tls-creds-psk', 'tls-creds-x509', 'tls-cipher-suites', - 'x-remote-object' + 'x-remote-object', + 'vfio-user' ] } ##
qapi/qom.json line 887
Quoted 6 lines
qapi/qom.json +2
-1@@ -868,7 +883,8 @@ 'tls-creds-psk': 'TlsCredsPskProperties', 'tls-creds-x509': 'TlsCredsX509Properties', 'tls-cipher-suites': 'TlsCredsProperties', - 'x-remote-object': 'RemoteObjectProperties' + 'x-remote-object': 'RemoteObjectProperties', + 'vfio-user': 'VfioUserProperties'
"vfio-user" doesn't communicate whether this is a client or server. Is "vfio-user-server" clearer?
Quoted 15 lines
qapi/qom.json +0
-0} } ##hw/remote/vfio-user-obj.c new file +6
-0@@ -0,0 +1,145 @@ +/** +* QEMU vfio-user server object +* +* Copyright © 2021 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL-v2, version 2 or later.
hw/remote/vfio-user-obj.c line 16
Quoted 10 lines
hw/remote/vfio-user-obj.c new file +10
-0+* +* See the COPYING file in the top-level directory. +* +*/ + +/** +* Usage: add options: +* -machine x-remote +* -device <PCI-device>,id=<pci-dev-id> +* -object vfio-user,id=<id>,socket=<socket-path>,devid=<pci-dev-id>
I suggest renaming devid= to device= or pci-device= (similar to drive= and netdev=) for consistency and to avoid confusion with PCI Device IDs.
hw/remote/meson.build remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) · line 9
Quoted 9 lines
hw/remote/meson.build +1
-0@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('vfio-user-obj.c'))
If you use CONFIG_VFIO_USER_SERVER then it's easier to separate mpqemu from vfio-user. Sharing CONFIG_MULTIPROCESS could become messy later.
Re: [PATCH RFC server v2 02/11] vfio-user: define vfio-user object
Quoted 37 lines · Stefan Hajnoczi
Quoted 32 lines · Jagannathan Raman
Define vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
qapi/qom.json | 20 ++++++- hw/remote/vfio-user-obj.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + hw/remote/meson.build | 1 + hw/remote/trace-events | 3 + 5 files changed, 168 insertions(+), 2 deletions(-)create mode 100644
hw/remote/vfio-user-obj.cqapi/qom.json +8
-0@@ -689,6 +689,20 @@ 'data': { 'fd': 'str', 'devid': 'str' } } ## +# @VfioUserProperties: +# +# Properties for vfio-user objects. +# +# @socket: path to be used as socket by the libvfiouser library +# +# @devid: the id of the device to be associated with the file descriptor +#
Quoted 8 lines
qapi/qom.json line 703
Quoted 4 lines
qapi/qom.json +4
-0+# Since: 6.0 +## +{ 'struct': 'VfioUserProperties', + 'data': { 'socket': 'str', 'devid': 'str' } }Please use 'SocketAddress' for socket instead of 'str'. That way file descriptor passing is easy to support and additional socket address families can be supported in the future.
OK, will do.
Quoted 15 lines
Quoted 15 lines
qapi/qom.json +4
-1+ +## # @RngProperties: # # Properties for objects of classes derived from rng.@@ -812,7 +826,8 @@ 'tls-creds-psk', 'tls-creds-x509', 'tls-cipher-suites', - 'x-remote-object' + 'x-remote-object', + 'vfio-user' ] }
Quoted 11 lines
qapi/qom.json line 887
Quoted 8 lines
qapi/qom.json +2
-1##@@ -868,7 +883,8 @@ 'tls-creds-psk': 'TlsCredsPskProperties', 'tls-creds-x509': 'TlsCredsX509Properties', 'tls-cipher-suites': 'TlsCredsProperties', - 'x-remote-object': 'RemoteObjectProperties' + 'x-remote-object': 'RemoteObjectProperties', + 'vfio-user': 'VfioUserProperties'"vfio-user" doesn't communicate whether this is a client or server. Is "vfio-user-server" clearer?
“vfio-user-server” sounds better.
Quoted 21 lines
Quoted 21 lines
qapi/qom.json +0
-0} } ##hw/remote/vfio-user-obj.c new file +11
-0@@ -0,0 +1,145 @@ +/** +* QEMU vfio-user server object +* +* Copyright © 2021 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL-v2, version 2 or later. +* +* See the COPYING file in the top-level directory. +* +*/ +
Quoted 8 lines
hw/remote/vfio-user-obj.c line 16
Quoted 5 lines
hw/remote/vfio-user-obj.c new file +5
-0+/** +* Usage: add options: +* -machine x-remote +* -device <PCI-device>,id=<pci-dev-id> +* -object vfio-user,id=<id>,socket=<socket-path>,devid=<pci-dev-id> I suggest renaming devid= to device= or pci-device= (similar to drive= and netdev=) for consistency and to avoid confusion with PCI Device IDs.
OK, will do.
Quoted 13 lines
hw/remote/meson.build remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) · line 9
Quoted 10 lines
hw/remote/meson.build +1
-0@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('vfio-user-obj.c'))If you use
CONFIG_VFIO_USER_SERVERthen it's easier to separate mpqemu from vfio-user. SharingCONFIG_MULTIPROCESScould become messy later.
OK, got it.
--
Jag
RFC PATCH v2 3/11 vfio-user: instantiate vfio-user context
In reply to Jagannathan Ramancreate a context with the vfio-user library to run a PCI device
--- hw/remote/vfio-user-obj.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
hw/remote/vfio-user-obj.c +29 -0
@@ -27,11 +27,17 @@#include "qemu/osdep.h" #include "qemu-common.h" +#include <errno.h> +#include "qom/object.h" #include "qom/object_interfaces.h" #include "qemu/error-report.h" #include "trace.h" #include "sysemu/runstate.h" +#include "qemu/notify.h" +#include "qapi/error.h" +#include "sysemu/sysemu.h" +#include "libvfio-user.h" #define TYPE_VFU_OBJECT "vfio-user"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )
@@ -51,6 +57,10 @@ struct VfuObject {char *socket;char *devid; + +Notifier machine_done; + +vfu_ctx_t *vfu_ctx; };static void vfu_object_set_socket (Object *obj,const char *str,Error **errp)
@@ -75,9 +85,23 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)trace_vfu_prop ("devid" , str); } +static void vfu_object_machine_done (Notifier *notifier,void *data) +{ +VfuObject *o =container_of (notifier, VfuObject, machine_done); + + o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket,0 , + o,VFU_DEV_TYPE_PCI ); +if (o->vfu_ctx ==NULL ) { +error_setg (&error_abort,"vfu: Failed to create context - %s" , +strerror (errno)); +return ; + } +} +static void vfu_object_init (Object *obj) {VfuObjectClass *k =VFU_OBJECT_GET_CLASS (obj); +VfuObject *o =VFU_OBJECT (obj);if (!object_dynamic_cast (OBJECT (current_machine),TYPE_REMOTE_MACHINE )) { error_report("vfu: %s only compatible with %s machine",
@@ -92,6 +116,9 @@ static void vfu_object_init(Object *obj) } k->nr_devs++; + + o->machine_done.notify = vfu_object_machine_done; +qemu_add_machine_init_done_notifier (&o->machine_done); }static void vfu_object_finalize (Object *obj)
@@ -101,6 +128,8 @@ static void vfu_object_finalize(Object *obj) k->nr_devs--; +vfu_destroy_ctx (o->vfu_ctx); +g_free (o->socket);g_free (o->devid);
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 03/11] vfio-user: instantiate vfio-user context
Quoted 14 lines · Jagannathan Raman
create a context with the vfio-user library to run a PCI device
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/vfio-user-obj.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)hw/remote/vfio-user-obj.c +0
-0
hw/remote/vfio-user-obj.c line 30
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -27,11 +27,17 @@#include "qemu/osdep.h" #include "qemu-common.h" +#include <errno.h>
qemu/osdep.h already includes <errno.h>
Quoted 25 lines
hw/remote/vfio-user-obj.c +9
-0+#include "qom/object.h" #include "qom/object_interfaces.h" #include "qemu/error-report.h" #include "trace.h" #include "sysemu/runstate.h" +#include "qemu/notify.h" +#include "qapi/error.h" +#include "sysemu/sysemu.h" +#include "libvfio-user.h" #define TYPE_VFU_OBJECT "vfio-user"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -51,6 +57,10 @@ struct VfuObject {char *socket;char *devid; + +Notifier machine_done; + +vfu_ctx_t *vfu_ctx; };static void vfu_object_set_socket (Object *obj,const char *str,Error **errp)
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 88
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -75,9 +85,23 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)trace_vfu_prop ("devid" , str); } +static void vfu_object_machine_done (Notifier *notifier,void *data)
Please document the reason for using a machine init done notifier.
Quoted 30 lines
hw/remote/vfio-user-obj.c +16
-0+{ +VfuObject *o =container_of (notifier, VfuObject, machine_done); + + o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket,0 , + o,VFU_DEV_TYPE_PCI ); +if (o->vfu_ctx ==NULL ) { +error_setg (&error_abort,"vfu: Failed to create context - %s" , +strerror (errno)); +return ; + } +} +static void vfu_object_init (Object *obj) {VfuObjectClass *k =VFU_OBJECT_GET_CLASS (obj); +VfuObject *o =VFU_OBJECT (obj);if (!object_dynamic_cast (OBJECT (current_machine),TYPE_REMOTE_MACHINE )) { error_report("vfu: %s only compatible with %s machine",@@ -92,6 +116,9 @@ static void vfu_object_init(Object *obj) } k->nr_devs++; + + o->machine_done.notify = vfu_object_machine_done; +qemu_add_machine_init_done_notifier (&o->machine_done); }static void vfu_object_finalize (Object *obj)
hw/remote/vfio-user-obj.c static void vfu_object_finalize(Object *obj) · line 131
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -101,6 +128,8 @@ static void vfu_object_finalize(Object *obj) k->nr_devs--; +vfu_destroy_ctx (o->vfu_ctx);
Will this function ever be called before vfu_object_machine_done() is called? In that case vfu_ctx isn't initialized.
Re: [PATCH RFC server v2 03/11] vfio-user: instantiate vfio-user context
Quoted 49 lines · Stefan Hajnoczi
Quoted 14 lines · Jagannathan Raman
create a context with the vfio-user library to run a PCI device
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/vfio-user-obj.c| 29 +++++++++++++++++++++++++++++1 file changed, 29 insertions(+)hw/remote/vfio-user-obj.c +0
-0hw/remote/vfio-user-obj.c line 30
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -27,11 +27,17 @@#include "qemu/osdep.h" #include "qemu-common.h" +#include <errno.h>
qemu/osdep.halready includes <errno.h>Quoted 23 lines
hw/remote/vfio-user-obj.c +9
-0+#include "qom/object.h" #include "qom/object_interfaces.h" #include "qemu/error-report.h" #include "trace.h" #include "sysemu/runstate.h" +#include "qemu/notify.h" +#include "qapi/error.h" +#include "sysemu/sysemu.h" +#include "libvfio-user.h" #define TYPE_VFU_OBJECT "vfio-user"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -51,6 +57,10 @@ struct VfuObject {char *socket;char *devid; + +Notifier machine_done; + +vfu_ctx_t *vfu_ctx; };
Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 88
Quoted 6 lines
hw/remote/vfio-user-obj.c +1
-0static void vfu_object_set_socket (Object *obj,const char *str,Error **errp)@@ -75,9 +85,23 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)trace_vfu_prop ("devid" , str); } +static void vfu_object_machine_done (Notifier *notifier,void *data)Please document the reason for using a machine init done notifier.
OK, will do.
Quoted 32 lines
Quoted 32 lines
hw/remote/vfio-user-obj.c +16
-0+{ +VfuObject *o =container_of (notifier, VfuObject, machine_done); + + o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket,0 , + o,VFU_DEV_TYPE_PCI ); +if (o->vfu_ctx ==NULL ) { +error_setg (&error_abort,"vfu: Failed to create context - %s" , +strerror (errno)); +return ; + } +} +static void vfu_object_init (Object *obj) {VfuObjectClass *k =VFU_OBJECT_GET_CLASS (obj); +VfuObject *o =VFU_OBJECT (obj);if (!object_dynamic_cast (OBJECT (current_machine),TYPE_REMOTE_MACHINE )) { error_report("vfu: %s only compatible with %s machine",@@ -92,6 +116,9 @@ static void vfu_object_init(Object *obj) } k->nr_devs++; + + o->machine_done.notify = vfu_object_machine_done; +qemu_add_machine_init_done_notifier (&o->machine_done); }static void vfu_object_finalize (Object *obj)@@ -101,6 +128,8 @@ static void vfu_object_finalize(Object *obj)
Quoted 6 lines
hw/remote/vfio-user-obj.c static void vfu_object_finalize(Object *obj) · line 131
Quoted 3 lines
hw/remote/vfio-user-obj.c +1
-0k->nr_devs--; +vfu_destroy_ctx (o->vfu_ctx);Will this function ever be called before
vfu_object_machine_done()is called? In that casevfu_ctxisn't initialized.
There are some case where vfu_object_finalize() could be called before vfu_object_machine_done() executes. In that case o->vfu_ctx would be NULL - we didn’t account for that before.
vfu_destroy_ctx() does check for NULL - however, we’ll add a check here as well in case vfu_destroy_ctx() changes in the future.
--
Jag
RFC PATCH v2 4/11 vfio-user: find and init PCI device
In reply to Jagannathan RamanFind the PCI device with specified id. Initialize the device context with the QEMU PCI device
--- hw/remote/vfio-user-obj.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
hw/remote/vfio-user-obj.c +28 -0
@@ -38,6 +38,8 @@#include "qapi/error.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" +#include "hw/qdev-core.h" +#include "hw/pci/pci.h" #define TYPE_VFU_OBJECT "vfio-user"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )
@@ -61,6 +63,8 @@ struct VfuObject {Notifier machine_done;vfu_ctx_t *vfu_ctx; + +PCIDevice *pci_dev; };static void vfu_object_set_socket (Object *obj,const char *str,Error **errp)
@@ -88,6 +92,8 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)static void vfu_object_machine_done (Notifier *notifier,void *data) {VfuObject *o =container_of (notifier, VfuObject, machine_done); +DeviceState *dev =NULL ; +int ret; o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket,0 , o,VFU_DEV_TYPE_PCI );
@@ -96,6 +102,28 @@ static void vfu_object_machine_done(Notifier *notifier, void *data) strerror(errno));return ; } + + dev =qdev_find_recursive (sysbus_get_default (), o->devid); +if (dev ==NULL ) { +error_setg (&error_abort,"vfu: Device %s not found" , o->devid); +return ; + } + +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (&error_abort,"vfu: %s not a PCI devices" , o->devid); +return ; + } + + o->pci_dev =PCI_DEVICE (dev); + + ret =vfu_pci_init (o->vfu_ctx,VFU_PCI_TYPE_CONVENTIONAL , +PCI_HEADER_TYPE_NORMAL ,0 ); +if (ret <0 ) { +error_setg (&error_abort, +"vfu: Failed to attach PCI device %s to context - %s" , + o->devid,strerror (errno)); +return ; + } }static void vfu_object_init (Object *obj)
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 04/11] vfio-user: find and init PCI device
Quoted 10 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +6
-0@@ -96,6 +102,28 @@ static void vfu_object_machine_done(Notifier *notifier, void *data) strerror(errno));return ; } + + dev =qdev_find_recursive (sysbus_get_default (), o->devid); +if (dev ==NULL ) { +error_setg (&error_abort,"vfu: Device %s not found" , o->devid); +return ; + }
hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 120
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+ +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (&error_abort,"vfu: %s not a PCI devices" , o->devid); +return ; + } + + o->pci_dev =PCI_DEVICE (dev); + + ret =vfu_pci_init (o->vfu_ctx,VFU_PCI_TYPE_CONVENTIONAL , +PCI_HEADER_TYPE_NORMAL ,0 );
What is needed to support PCI Express?
Re: [PATCH RFC server v2 04/11] vfio-user: find and init PCI device
Quoted 19 lines · Stefan Hajnoczi
Quoted 14 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +10
-0@@ -96,6 +102,28 @@ static void vfu_object_machine_done(Notifier *notifier, void *data) strerror(errno));return ; } + + dev =qdev_find_recursive (sysbus_get_default (), o->devid); +if (dev ==NULL ) { +error_setg (&error_abort,"vfu: Device %s not found" , o->devid); +return ; + } + +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (&error_abort,"vfu: %s not a PCI devices" , o->devid); +return ;
Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 120
Quoted 6 lines
hw/remote/vfio-user-obj.c +6
-0+ } + + o->pci_dev =PCI_DEVICE (dev); + + ret =vfu_pci_init (o->vfu_ctx,VFU_PCI_TYPE_CONVENTIONAL , +PCI_HEADER_TYPE_NORMAL ,0 );What is needed to support PCI Express?
I think we could check if o->pci_dev supports QEMU_PCI_CAP_EXPRESS, and based on that choose if we should use VFU_PCI_TYPE_CONVENTIONAL or VFU_PCI_TYPE_EXPRESS.
pci_is_express() is already doing that, although it’s a private function now. It’s a good time to export it.
--
Jag
RFC PATCH v2 5/11 vfio-user: run vfio-user context
In reply to Jagannathan RamanSetup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
--- hw/remote/vfio-user-obj.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 1 deletion(-)
hw/remote/vfio-user-obj.c +70 -1
@@ -35,6 +35,7 @@#include "trace.h" #include "sysemu/runstate.h" #include "qemu/notify.h" +#include "qemu/thread.h" #include "qapi/error.h" #include "sysemu/sysemu.h" #include "libvfio-user.h"
@@ -65,6 +66,8 @@ struct VfuObject {vfu_ctx_t *vfu_ctx;PCIDevice *pci_dev; + +int vfu_poll_fd; };static void vfu_object_set_socket (Object *obj,const char *str,Error **errp)
@@ -89,13 +92,67 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)trace_vfu_prop ("devid" , str); } +static void vfu_object_ctx_run (void *opaque) +{ +VfuObject *o = opaque; +int ret =-1 ; + +while (ret !=0 ) { + ret =vfu_run_ctx (o->vfu_ctx); +if (ret <0 ) { +if (errno ==EINTR ) { +continue ; + }else if (errno ==ENOTCONN ) { +qemu_set_fd_handler (o->vfu_poll_fd,NULL ,NULL ,NULL ); + o->vfu_poll_fd =-1 ; +object_unparent (OBJECT (o)); +break ; + }else { +error_setg (&error_abort,"vfu: Failed to run device %s - %s" , + o->devid,strerror (errno)); +break ; + } + } + } +} + +static void *vfu_object_attach_ctx (void *opaque) +{ +VfuObject *o = opaque; +int ret; + +retry_attach: + ret =vfu_attach_ctx (o->vfu_ctx); +if (ret <0 && (errno ==EAGAIN || errno ==EWOULDBLOCK )) { + goto retry_attach; + }else if (ret <0 ) { +error_setg (&error_abort, +"vfu: Failed to attach device %s to context - %s" , + o->devid,strerror (errno)); +return NULL ; + } + + o->vfu_poll_fd =vfu_get_poll_fd (o->vfu_ctx); +if (o->vfu_poll_fd <0 ) { +error_setg (&error_abort,"vfu: Failed to get poll fd %s" , o->devid); +return NULL ; + } + +qemu_set_fd_handler (o->vfu_poll_fd, vfu_object_ctx_run, +NULL , o); + +return NULL ; +} +static void vfu_object_machine_done (Notifier *notifier,void *data) {VfuObject *o =container_of (notifier, VfuObject, machine_done);DeviceState *dev =NULL ; +QemuThread thread;int ret; - o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket,0 , + o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket, +LIBVFIO_USER_FLAG_ATTACH_NB , o,VFU_DEV_TYPE_PCI );if (o->vfu_ctx ==NULL ) { error_setg(&error_abort, "vfu: Failed to create context - %s",
@@ -124,6 +181,16 @@ static void vfu_object_machine_done(Notifier *notifier, void *data) o->devid, strerror(errno));return ; } + + ret =vfu_realize_ctx (o->vfu_ctx); +if (ret <0 ) { +error_setg (&error_abort,"vfu: Failed to realize device %s- %s" , + o->devid,strerror (errno)); +return ; + } + +qemu_thread_create (&thread, o->socket, vfu_object_attach_ctx, o, +QEMU_THREAD_DETACHED ); }static void vfu_object_init (Object *obj)
@@ -147,6 +214,8 @@ static void vfu_object_init(Object *obj) o->machine_done.notify = vfu_object_machine_done;qemu_add_machine_init_done_notifier (&o->machine_done); + + o->vfu_poll_fd =-1 ; }static void vfu_object_finalize (Object *obj)
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context
Quoted 58 lines · Jagannathan Raman
Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/vfio-user-obj.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 1 deletion(-)hw/remote/vfio-user-obj.c +25
-0@@ -35,6 +35,7 @@#include "trace.h" #include "sysemu/runstate.h" #include "qemu/notify.h" +#include "qemu/thread.h" #include "qapi/error.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" @@ -65,6 +66,8 @@ struct VfuObject {vfu_ctx_t *vfu_ctx;PCIDevice *pci_dev; + +int vfu_poll_fd; };static void vfu_object_set_socket (Object *obj,const char *str,Error **errp)@@ -89,13 +92,67 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)trace_vfu_prop ("devid" , str); } +static void vfu_object_ctx_run (void *opaque) +{ +VfuObject *o = opaque; +int ret =-1 ; + +while (ret !=0 ) { + ret =vfu_run_ctx (o->vfu_ctx); +if (ret <0 ) { +if (errno ==EINTR ) { +continue ; + }else if (errno ==ENOTCONN ) { +qemu_set_fd_handler (o->vfu_poll_fd,NULL ,NULL ,NULL ); + o->vfu_poll_fd =-1 ; +object_unparent (OBJECT (o)); +break ; + }else { +error_setg (&error_abort,"vfu: Failed to run device %s - %s" , + o->devid,strerror (errno)); +break ; + } + } + }
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 126
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+} + +static void *vfu_object_attach_ctx (void *opaque) +{ +VfuObject *o = opaque; +int ret; + +retry_attach: + ret =vfu_attach_ctx (o->vfu_ctx); +if (ret <0 && (errno ==EAGAIN || errno ==EWOULDBLOCK )) {
Does this loop consume 100% CPU since this is non-blocking?
Is it possible to register the fd with a QEMU AioContext instead of spawning a separate thread?
libvfio-user has non-blocking listen_fd but conn_fd is always blocking. This means ATTACH_NB is not useful because vfu_attach_ctx() is actually blocking. I think this means vfu_run_ctx() is also blocking in some places and QEMU's event loop might hang :(.
Can you make libvfio-user non-blocking in order to solve these issues?
Quoted 37 lines
hw/remote/vfio-user-obj.c +23
-1+ goto retry_attach; + }else if (ret <0 ) { +error_setg (&error_abort, +"vfu: Failed to attach device %s to context - %s" , + o->devid,strerror (errno)); +return NULL ; + } + + o->vfu_poll_fd =vfu_get_poll_fd (o->vfu_ctx); +if (o->vfu_poll_fd <0 ) { +error_setg (&error_abort,"vfu: Failed to get poll fd %s" , o->devid); +return NULL ; + } + +qemu_set_fd_handler (o->vfu_poll_fd, vfu_object_ctx_run, +NULL , o); + +return NULL ; +} +static void vfu_object_machine_done (Notifier *notifier,void *data) {VfuObject *o =container_of (notifier, VfuObject, machine_done);DeviceState *dev =NULL ; +QemuThread thread;int ret; - o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket,0 , + o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket, +LIBVFIO_USER_FLAG_ATTACH_NB , o,VFU_DEV_TYPE_PCI );if (o->vfu_ctx ==NULL ) { error_setg(&error_abort, "vfu: Failed to create context - %s",@@ -124,6 +181,16 @@ static void vfu_object_machine_done(Notifier *notifier, void *data) o->devid, strerror(errno));return ; }
hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 193
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+ + ret =vfu_realize_ctx (o->vfu_ctx); +if (ret <0 ) { +error_setg (&error_abort,"vfu: Failed to realize device %s- %s" , + o->devid,strerror (errno)); +return ; + } + +qemu_thread_create (&thread, o->socket, vfu_object_attach_ctx, o, +QEMU_THREAD_DETACHED );
Is this thread leaked when the object is destroyed?
Quoted 15 lines
hw/remote/vfio-user-obj.c +2
-0}static void vfu_object_init (Object *obj)@@ -147,6 +214,8 @@ static void vfu_object_init(Object *obj) o->machine_done.notify = vfu_object_machine_done;qemu_add_machine_init_done_notifier (&o->machine_done); + + o->vfu_poll_fd =-1 ; }static void vfu_object_finalize (Object *obj)Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context
Quoted 10 lines · Stefan Hajnoczi
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 126
Quoted 8 lines
hw/remote/vfio-user-obj.c +8
-0+static void *vfu_object_attach_ctx (void *opaque) +{ +VfuObject *o = opaque; +int ret; + +retry_attach: + ret =vfu_attach_ctx (o->vfu_ctx); +if (ret <0 && (errno ==EAGAIN || errno ==EWOULDBLOCK )) {Does this loop consume 100% CPU since this is non-blocking?
Looks like it. Instead after vfu_create_ctx, there should be a vfu_get_poll_fd() to get the listen socket, then a qemu_set_fd_handler(vfu_object_attach_ctx) to handle the attach when the listen socket is ready, modulo the below part.
Quoted 1 line
libvfio-user has non-blocking
listen_fdbutconn_fdis always blocking.
It is, but in vfu_run_ctx(), we poll on it:
```
790 if (vfu_ctx->flags & LIBVFIO_USER_FLAG_ATTACH_NB) {
791 sock_flags = MSG_DONTWAIT | MSG_WAITALL;
792 }
793 return get_msg(hdr, sizeof(*hdr), fds, nr_fds, ts->conn_fd, sock_flags);
```Quoted 2 lines
This means
ATTACH_NBis not useful becausevfu_attach_ctx()is actually blocking.
You're correct that vfu_attach_ctx is in fact partially blocking: after accepting the connection, we call negotiate(), which can indeed block waiting if the client hasn't sent anything.
Quoted 1 line
I think this means
vfu_run_ctx()is also blocking in some places
Correct. There's a presumption that if a message is ready, we can read it all without blocking, and equally that we can write to the socket without blocking.
The library docs are not at all clear on this point.
Quoted 3 lines
and QEMU's event loop might hang :(
Can you make libvfio-user non-blocking in order to solve these issues?
I presume you're concerned about the security aspect: a malicious client could withhold a write, and hence hang the device server.
Problem is the libvfio-user API is synchronous: there's no way to return half-way through a vfu_attach_ctx() (or a vfu_run_ctx() after we read the header) then resume.
We'd have to have a whole separate API to do that, so a separate thread seems a better approach?
regards
john
Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context
Quoted 19 lines · John Levon
Quoted 10 lines · Stefan Hajnoczi
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 126
Quoted 8 lines
hw/remote/vfio-user-obj.c +8
-0+static void *vfu_object_attach_ctx (void *opaque) +{ +VfuObject *o = opaque; +int ret; + +retry_attach: + ret =vfu_attach_ctx (o->vfu_ctx); +if (ret <0 && (errno ==EAGAIN || errno ==EWOULDBLOCK )) {Does this loop consume 100% CPU since this is non-blocking?
Looks like it. Instead after
vfu_create_ctx, there should be avfu_get_poll_fd()to get the listen socket, then aqemu_set_fd_handler(vfu_object_attach_ctx)to handle the attach when the listen socket is ready, modulo the below part.Quoted 1 line
libvfio-user has non-blocking
listen_fdbutconn_fdis always blocking.
Quoted 8 lines
It is, but in
vfu_run_ctx(), we poll on it:``` 790 if (vfu_ctx->flags & LIBVFIO_USER_FLAG_ATTACH_NB) { 791 sock_flags = MSG_DONTWAIT | MSG_WAITALL; 792 } 793 return get_msg(hdr, sizeof(*hdr), fds, nr_fds, ts->conn_fd, sock_flags); ```
This is only used for the request header. Other I/O is blocking.
Quoted 22 lines
Quoted 3 lines
This means
ATTACH_NBis not useful becausevfu_attach_ctx()is actually blocking.You're correct that
vfu_attach_ctxis in fact partially blocking: after accepting the connection, we callnegotiate(), which can indeed block waiting if the client hasn't sent anything.Quoted 1 line
I think this means
vfu_run_ctx()is also blocking in some placesCorrect. There's a presumption that if a message is ready, we can read it all without blocking, and equally that we can write to the socket without blocking.
The library docs are not at all clear on this point.
Quoted 3 lines
and QEMU's event loop might hang :(
Can you make libvfio-user non-blocking in order to solve these issues?
I presume you're concerned about the security aspect: a malicious client could withhold a write, and hence hang the device server.
Quoted 6 lines
Problem is the libvfio-user API is synchronous: there's no way to return half-way through a
vfu_attach_ctx()(or avfu_run_ctx()after we read the header) then resume.We'd have to have a whole separate API to do that, so a separate thread seems a better approach?
Whether to support non-blocking properly in libvfio-user is a decision for you. If libvfio-user doesn't support non-blocking, then QEMU should run a dedicated thread instead of the partially non-blocking approach in this patch.
A non-blocking approach is nice when there are many devices hosted in a single process or a lot of async replies (which requires extra thread synchronization with the blocking approach).
Stefan
Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context
Quoted 7 lines · Stefan Hajnoczi
Quoted 2 lines
We'd have to have a whole separate API to do that, so a separate thread seems a better approach?
Whether to support non-blocking properly in libvfio-user is a decision for you. If libvfio-user doesn't support non-blocking, then QEMU should run a dedicated thread instead of the partially non-blocking approach in this patch.
Right, sure. At this point we don't have any plans to implement a separate async API due to the amount of work involved.
Quoted 3 lines
A non-blocking approach is nice when there are many devices hosted in a single process or a lot of async replies (which requires extra thread synchronization with the blocking approach).
I suppose this would be more of a problem with devices where the I/O path has to be handled via the socket.
regards
john
Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context
Quoted 13 lines · John Levon
Quoted 7 lines · Stefan Hajnoczi
Quoted 2 lines
We'd have to have a whole separate API to do that, so a separate thread seems a better approach?
Whether to support non-blocking properly in libvfio-user is a decision for you. If libvfio-user doesn't support non-blocking, then QEMU should run a dedicated thread instead of the partially non-blocking approach in this patch.
Right, sure. At this point we don't have any plans to implement a separate async API due to the amount of work involved.
Quoted 6 lines
Quoted 3 lines
A non-blocking approach is nice when there are many devices hosted in a single process or a lot of async replies (which requires extra thread synchronization with the blocking approach).
I suppose this would be more of a problem with devices where the I/O path has to be handled via the socket.
Yes, exactly. I think it shouldn't be a problem when shared memory is used and the irqfd (eventfd) mechanism is used for IRQs. It becomes slow when there's no shared memory or if raising IRQs requires protocol messages.
Stefan
RFC PATCH v2 6/11 vfio-user: handle PCI config space accesses
In reply to Jagannathan RamanDefine and register handlers for PCI config space accesses
--- hw/remote/vfio-user-obj.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 2 files changed, 46 insertions(+)
hw/remote/vfio-user-obj.c +44 -0
@@ -36,6 +36,7 @@#include "sysemu/runstate.h" #include "qemu/notify.h" #include "qemu/thread.h" +#include "qemu/main-loop.h" #include "qapi/error.h" #include "sysemu/sysemu.h" #include "libvfio-user.h"
@@ -144,6 +145,38 @@ retry_attach:return NULL ; } +static ssize_t vfu_object_cfg_access (vfu_ctx_t *vfu_ctx,char *const buf, +size_t count,loff_t offset, +const bool is_write) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint32_t pci_access_width =sizeof (uint32_t ); +size_t bytes = count; +uint32_t val =0 ; +char *ptr = buf; +int len; + +while (bytes >0 ) { + len = (bytes > pci_access_width) ? pci_access_width : bytes; +if (is_write) { +memcpy (&val, ptr, len); +pci_default_write_config (PCI_DEVICE (o->pci_dev), + offset, val, len); +trace_vfu_cfg_write (offset, val); + }else { + val =pci_default_read_config (PCI_DEVICE (o->pci_dev), + offset, len); +memcpy (ptr, &val, len); +trace_vfu_cfg_read (offset, val); + } + offset += len; + ptr += len; + bytes -= len; + } + +return count; +} +static void vfu_object_machine_done (Notifier *notifier,void *data) {VfuObject *o =container_of (notifier, VfuObject, machine_done);
@@ -182,6 +215,17 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)return ; } + ret =vfu_setup_region (o->vfu_ctx,VFU_PCI_DEV_CFG_REGION_IDX , +pci_config_size (o->pci_dev), &vfu_object_cfg_access, +VFU_REGION_FLAG_RW |VFU_REGION_FLAG_ALWAYS_CB , +NULL ,0 ,-1 ,0 ); +if (ret <0 ) { +error_setg (&error_abort, +"vfu: Failed to setup config space handlers for %s- %s" , + o->devid,strerror (errno)); +return ; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
@@ -5,3 +5,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, # vfio-user-obj.c vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s" +vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" +vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 06/11] vfio-user: handle PCI config space accesses
Quoted 12 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +12
-0+static ssize_t vfu_object_cfg_access (vfu_ctx_t *vfu_ctx,char *const buf, +size_t count,loff_t offset, +const bool is_write) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint32_t pci_access_width =sizeof (uint32_t ); +size_t bytes = count; +uint32_t val =0 ; +char *ptr = buf; +int len; + +while (bytes >0 ) {
hw/remote/vfio-user-obj.c retry_attach: · line 169
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+ len = (bytes > pci_access_width) ? pci_access_width : bytes; +if (is_write) { +memcpy (&val, ptr, len); +pci_default_write_config (PCI_DEVICE (o->pci_dev), + offset, val, len); +trace_vfu_cfg_write (offset, val); + }else { + val =pci_default_read_config (PCI_DEVICE (o->pci_dev), + offset, len); +memcpy (ptr, &val, len);
pci_default_read_config() returns a host-endian 32-bit value. This code looks wrong because it copies different bytes on big- and little-endian hosts.
Review
Quoted 2 lines
(quoted diff) +2
-0+ trace_vfu_cfg_read(offset, val); + }
Why call pci_default_read/write_config() directly instead of pci_dev->config_read/write()?
Re: [PATCH RFC server v2 06/11] vfio-user: handle PCI config space accesses
Quoted 23 lines · Stefan Hajnoczi
Quoted 18 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +18
-0+static ssize_t vfu_object_cfg_access (vfu_ctx_t *vfu_ctx,char *const buf, +size_t count,loff_t offset, +const bool is_write) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint32_t pci_access_width =sizeof (uint32_t ); +size_t bytes = count; +uint32_t val =0 ; +char *ptr = buf; +int len; + +while (bytes >0 ) { + len = (bytes > pci_access_width) ? pci_access_width : bytes; +if (is_write) { +memcpy (&val, ptr, len); +pci_default_write_config (PCI_DEVICE (o->pci_dev), + offset, val, len); +trace_vfu_cfg_write (offset, val);
Quoted 8 lines
hw/remote/vfio-user-obj.c retry_attach: · line 169
Quoted 4 lines
hw/remote/vfio-user-obj.c +4
-0+ }else { + val =pci_default_read_config (PCI_DEVICE (o->pci_dev), + offset, len); +memcpy (ptr, &val, len);
pci_default_read_config()returns a host-endian 32-bit value. This code looks wrong because it copies different bytes on big- and little-endian hosts.
I’ll collect more details on this one, trying to wrap my head around it.
Concerning config space writes, it doesn’t look like we need to perform any conversion as the store operation automatically happens in the CPU’s native format when we do something like the following:
PCIDevice->config[addr] = val;
Concerning config read, I observed that pci_default_read_config() performs le32_to_cpu() conversion. May be we should not rely on it doing the conversion.
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ trace_vfu_cfg_read(offset, val); + }Why call pci_default_read/write_config() directly instead of pci_dev->config_read/write()?
That makes sense - we should be calling pci_dev->config_read/write().
After performing pci_dev->config_read(), I’ll convert it to the CPU’s endianness format using le32_to_cpu(). On big-endian systems, it would re-order the bytes, and on little-endian systems it would be a no-op.
--
Jag
Re: [PATCH RFC server v2 06/11] vfio-user: handle PCI config space accesses
Quoted 32 lines · Jag Raman
Quoted 23 lines · Stefan Hajnoczi
Quoted 18 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +18
-0+static ssize_t vfu_object_cfg_access (vfu_ctx_t *vfu_ctx,char *const buf, +size_t count,loff_t offset, +const bool is_write) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint32_t pci_access_width =sizeof (uint32_t ); +size_t bytes = count; +uint32_t val =0 ; +char *ptr = buf; +int len; + +while (bytes >0 ) { + len = (bytes > pci_access_width) ? pci_access_width : bytes; +if (is_write) { +memcpy (&val, ptr, len); +pci_default_write_config (PCI_DEVICE (o->pci_dev), + offset, val, len); +trace_vfu_cfg_write (offset, val);Quoted 8 lines
hw/remote/vfio-user-obj.c retry_attach: · line 169
Quoted 4 lines
hw/remote/vfio-user-obj.c +4
-0+ }else { + val =pci_default_read_config (PCI_DEVICE (o->pci_dev), + offset, len); +memcpy (ptr, &val, len);
pci_default_read_config()returns a host-endian 32-bit value. This code looks wrong because it copies different bytes on big- and little-endian hosts.
Quoted 6 lines
I’ll collect more details on this one, trying to wrap my head around it.
Concerning config space writes, it doesn’t look like we need to perform any conversion as the store operation automatically happens in the CPU’s native format when we do something like the following:
PCIDevice->config[addr] = val;
PCIDevice->config is uint8_t*. Endianness isn't an issue with single byte accesses.
Quoted 4 lines
Concerning config read, I observed that
pci_default_read_config()performsle32_to_cpu()conversion. May be we should not rely on it doing the conversion.
Yes, ->config_read() returns a host-endian 32-bit value and ->config_write() receives a host-endian 32-bit value (it has a bit-shifting loop that implicitly handles endianness conversion).
Stefan
RFC PATCH v2 7/11 vfio-user: handle DMA mappings
In reply to Jagannathan RamanDefine and register callbacks to manage the RAM regions used for device DMA
--- hw/remote/vfio-user-obj.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 2 files changed, 52 insertions(+)
hw/remote/vfio-user-obj.c +50 -0
@@ -177,6 +177,49 @@ static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,return count; } +static void dma_register (vfu_ctx_t *vfu_ctx,vfu_dma_info_t *info) +{ +MemoryRegion *subregion =NULL ; + g_autofree char *name = NULL; +static unsigned int suffix; +struct iovec *iov = &info->iova; + +if (!info->vaddr) { +return ; + } + + name =g_strdup_printf ("remote-mem-%u" , suffix++); + + subregion =g_new0 (MemoryRegion,1 ); + +memory_region_init_ram_ptr (subregion,NULL , name, + iov->iov_len, info->vaddr); + +memory_region_add_subregion (get_system_memory (), (hwaddr )iov->iov_base, + subregion); + +trace_vfu_dma_register ((uint64_t )iov->iov_base, iov->iov_len); +} + +static int dma_unregister (vfu_ctx_t *vfu_ctx,vfu_dma_info_t *info) +{ +MemoryRegion *mr =NULL ; +ram_addr_t offset; + + mr =memory_region_from_host (info->vaddr, &offset); +if (!mr) { +return 0 ; + } + +memory_region_del_subregion (get_system_memory (), mr); + +object_unparent ((OBJECT (mr))); + +trace_vfu_dma_unregister ((uint64_t )info->iova.iov_base); + +return 0 ; +} +static void vfu_object_machine_done (Notifier *notifier,void *data) {VfuObject *o =container_of (notifier, VfuObject, machine_done);
@@ -226,6 +269,13 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)return ; } + ret =vfu_setup_device_dma (o->vfu_ctx, &dma_register, &dma_unregister); +if (ret <0 ) { +error_setg (&error_abort,"vfu: Failed to setup DMA handlers for %s" , + o->devid); +return ; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
@@ -7,3 +7,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s" vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x" +vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes" +vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 07/11] vfio-user: handle DMA mappings
Quoted 10 lines · Jagannathan Raman
Define and register callbacks to manage the RAM regions used for device DMA
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/vfio-user-obj.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 2 files changed, 52 insertions(+)
RFC PATCH v2 8/11 vfio-user: handle PCI BAR accesses
In reply to Jagannathan RamanDetermine the BARs used by the PCI device and register handlers to manage the access to the same.
--- hw/remote/vfio-user-obj.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 + 2 files changed, 97 insertions(+)
hw/remote/vfio-user-obj.c +95 -0
@@ -220,6 +220,99 @@ static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)return 0 ; } +static ssize_t vfu_object_bar_rw (PCIDevice *pci_dev,hwaddr addr,size_t count, +char *const buf,const bool is_write, +uint8_t type) +{ +AddressSpace *as =NULL ; +MemTxResult res; + +if (type ==PCI_BASE_ADDRESS_SPACE_MEMORY ) { + as =pci_device_iommu_address_space (pci_dev); + }else { + as = &address_space_io; + } + +trace_vfu_bar_rw_enter (is_write ?"Write" :"Read" , (uint64_t )addr); + + res =address_space_rw (as, addr,MEMTXATTRS_UNSPECIFIED , (void *)buf, + (hwaddr )count, is_write); +if (res !=MEMTX_OK ) { +warn_report ("vfu: failed to %s 0x%" PRIx64"" , + is_write ?"write to" :"read from" , + addr); +return -1 ; + } + +trace_vfu_bar_rw_exit (is_write ?"Write" :"Read" , (uint64_t )addr); + +return count; +} + +/** +* VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs. +* +* To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would +* define vfu_object_bar2_handler +*/ +#define VFU_OBJECT_BAR_HANDLER (BAR_NO ) \ + static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx, \ + char * const buf, size_t count, \ + loff_t offset, const bool is_write) \ + { \ + VfuObject *o = vfu_get_private(vfu_ctx); \ + hwaddr addr = (hwaddr)(pci_get_long(o->pci_dev->config + \ + PCI_BASE_ADDRESS_0 + \ + (4 * BAR_NO)) + offset); \ + \ + return vfu_object_bar_rw(o->pci_dev, addr, count, buf, is_write, \ + o->pci_dev->io_regions[BAR_NO].type); \ + } \ + +VFU_OBJECT_BAR_HANDLER(0) +VFU_OBJECT_BAR_HANDLER(1) +VFU_OBJECT_BAR_HANDLER(2) +VFU_OBJECT_BAR_HANDLER(3) +VFU_OBJECT_BAR_HANDLER(4) +VFU_OBJECT_BAR_HANDLER(5) + +static vfu_region_access_cb_t *vfu_object_bar_handlers[PCI_NUM_REGIONS ] = { + &vfu_object_bar0_handler, + &vfu_object_bar1_handler, + &vfu_object_bar2_handler, + &vfu_object_bar3_handler, + &vfu_object_bar4_handler, + &vfu_object_bar5_handler, +}; + +/** +* vfu_object_register_bars - Identify active BAR regions of pdev and setup +* callbacks to handle read/write accesses +*/ +static void vfu_object_register_bars (vfu_ctx_t *vfu_ctx,PCIDevice *pdev) +{ +uint32_t orig_val, new_val; +int i, size; + +for (i =0 ; i <PCI_NUM_REGIONS ; i++) { + orig_val =pci_default_read_config (pdev, +PCI_BASE_ADDRESS_0 + (4 * i),4 ); + new_val =0xffffffff ; +pci_default_write_config (pdev, +PCI_BASE_ADDRESS_0 + (4 * i), new_val,4 ); + new_val =pci_default_read_config (pdev, +PCI_BASE_ADDRESS_0 + (4 * i),4 ); + size = (~(new_val &0xFFFFFFF0 )) +1 ; +pci_default_write_config (pdev,PCI_BASE_ADDRESS_0 + (4 * i), + orig_val,4 ); +if (size) { +vfu_setup_region (vfu_ctx,VFU_PCI_DEV_BAR0_REGION_IDX + i, size, + vfu_object_bar_handlers[i],VFU_REGION_FLAG_RW , +NULL ,0 ,-1 ,0 ); + } + } +} +static void vfu_object_machine_done (Notifier *notifier,void *data) {VfuObject *o =container_of (notifier, VfuObject, machine_done);
@@ -276,6 +369,8 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)return ; } +vfu_object_register_bars (o->vfu_ctx, o->pci_dev); + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
@@ -9,3 +9,5 @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x" vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes" vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" +vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64"" +vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 08/11] vfio-user: handle PCI BAR accesses
Quoted 5 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +5
-0+/** +* VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs. +* +* To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would +* define vfu_object_bar2_handler
hw/remote/vfio-user-obj.c static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info) · line 266
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+*/ +#define VFU_OBJECT_BAR_HANDLER (BAR_NO ) \ + static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx, \ + char * const buf, size_t count, \ + loff_t offset, const bool is_write) \ + { \ + VfuObject *o = vfu_get_private(vfu_ctx); \ + hwaddr addr = (hwaddr)(pci_get_long(o->pci_dev->config + \ + PCI_BASE_ADDRESS_0 + \ + (4 * BAR_NO)) + offset); \
Does this handle 64-bit BARs?
hw/remote/vfio-user-obj.c static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info) · line 299
Quoted 12 lines
hw/remote/vfio-user-obj.c +12
-0+/** +* vfu_object_register_bars - Identify active BAR regions of pdev and setup +* callbacks to handle read/write accesses +*/ +static void vfu_object_register_bars (vfu_ctx_t *vfu_ctx,PCIDevice *pdev) +{ +uint32_t orig_val, new_val; +int i, size; + +for (i =0 ; i <PCI_NUM_REGIONS ; i++) { + orig_val =pci_default_read_config (pdev, +PCI_BASE_ADDRESS_0 + (4 * i),4 );
Same question as in an earlier patch: should we call pdev->read_config()?
Re: [PATCH RFC server v2 08/11] vfio-user: handle PCI BAR accesses
Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +9
-0+/** +* VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs. +* +* To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would +* define vfu_object_bar2_handler +*/ +#define VFU_OBJECT_BAR_HANDLER (BAR_NO ) \ + static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx, \ + char * const buf, size_t count, \
Quoted 8 lines
hw/remote/vfio-user-obj.c static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info) · line 266
Quoted 6 lines
hw/remote/vfio-user-obj.c +6
-0+ loff_t offset, const bool is_write) \ + { \ + VfuObject *o = vfu_get_private(vfu_ctx); \ +hwaddr addr = (hwaddr)(pci_get_long (o->pci_dev->config + \ +PCI_BASE_ADDRESS_0 + \ + (4 *BAR_NO )) + offset); \Does this handle 64-bit BARs?
It presently only handles 32-bit BARs. We’ll add support for 64-bit BARs in the next rev of this series.
Quoted 15 lines
hw/remote/vfio-user-obj.c static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info) · line 299
Quoted 13 lines
hw/remote/vfio-user-obj.c +12
-0+/** +* vfu_object_register_bars - Identify active BAR regions of pdev and setup +* callbacks to handle read/write accesses +*/ +static void vfu_object_register_bars (vfu_ctx_t *vfu_ctx,PCIDevice *pdev) +{ +uint32_t orig_val, new_val; +int i, size; + +for (i =0 ; i <PCI_NUM_REGIONS ; i++) { + orig_val =pci_default_read_config (pdev, +PCI_BASE_ADDRESS_0 + (4 * i),4 );Same question as in an earlier patch: should we call
pdev->read_config()?
Sure, will do.
--
Jag
RFC PATCH v2 9/11 vfio-user: handle device interrupts
In reply to Jagannathan RamanForward remote device's interrupts to the guest
--- include/hw/remote/iohub.h | 2 ++ hw/remote/iohub.c | 5 +++++ hw/remote/vfio-user-obj.c | 30 ++++++++++++++++++++++++++++++ hw/remote/trace-events | 1 + 4 files changed, 38 insertions(+)
include/hw/remote/iohub.h +2 -0
@@ -15,6 +15,7 @@#include "qemu/event_notifier.h" #include "qemu/thread-posix.h" #include "hw/remote/mpqemu-link.h" +#include "libvfio-user.h" #define REMOTE_IOHUB_NB_PIRQS PCI_DEVFN_MAX
@@ -30,6 +31,7 @@ typedef struct RemoteIOHubState {unsigned int irq_level[REMOTE_IOHUB_NB_PIRQS ];ResampleToken token[REMOTE_IOHUB_NB_PIRQS ];QemuMutex irq_level_lock[REMOTE_IOHUB_NB_PIRQS ]; +vfu_ctx_t *vfu_ctx[REMOTE_IOHUB_NB_PIRQS ]; } RemoteIOHubState;int remote_iohub_map_irq (PCIDevice *pci_dev,int intx);
hw/remote/iohub.c +5 -0
@@ -18,6 +18,7 @@#include "hw/remote/machine.h" #include "hw/remote/iohub.h" #include "qemu/main-loop.h" +#include "trace.h" void remote_iohub_init (RemoteIOHubState *iohub) {
@@ -62,6 +63,10 @@ void remote_iohub_set_irq(void *opaque, int pirq, int level)QEMU_LOCK_GUARD (&iohub->irq_level_lock[pirq]);if (level) { +if (iohub->vfu_ctx[pirq]) { +trace_vfu_interrupt (pirq); +vfu_irq_trigger (iohub->vfu_ctx[pirq],0 ); + }if (++iohub->irq_level[pirq] ==1 ) {event_notifier_set (&iohub->irqfds[pirq]); }
hw/remote/vfio-user-obj.c +30 -0
@@ -42,6 +42,9 @@#include "libvfio-user.h" #include "hw/qdev-core.h" #include "hw/pci/pci.h" +#include "hw/boards.h" +#include "hw/remote/iohub.h" +#include "hw/remote/machine.h" #define TYPE_VFU_OBJECT "vfio-user"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )
@@ -313,6 +316,26 @@ static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev) } } +static int vfu_object_setup_irqs (vfu_ctx_t *vfu_ctx,PCIDevice *pci_dev) +{ +RemoteMachineState *machine =REMOTE_MACHINE (current_machine); +RemoteIOHubState *iohub = &machine->iohub; +int pirq, intx, ret; + + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_INTX_IRQ ,1 ); +if (ret <0 ) { +return ret; + } + + intx =pci_get_byte (pci_dev->config +PCI_INTERRUPT_PIN ) -1 ; + + pirq =remote_iohub_map_irq (pci_dev, intx); + + iohub->vfu_ctx[pirq] = vfu_ctx; + +return 0 ; +} +static void vfu_object_machine_done (Notifier *notifier,void *data) {VfuObject *o =container_of (notifier, VfuObject, machine_done);
@@ -371,6 +394,13 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)vfu_object_register_bars (o->vfu_ctx, o->pci_dev); + ret =vfu_object_setup_irqs (o->vfu_ctx, o->pci_dev); +if (ret <0 ) { +error_setg (&error_abort,"vfu: Failed to setup interrupts for %s" , + o->devid); +return ; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +1 -0
@@ -11,3 +11,4 @@ vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %z vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64"" vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64"" +vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 09/11] vfio-user: handle device interrupts
Quoted 11 lines · Jagannathan Raman
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/remote/iohub.h | 2 ++ hw/remote/iohub.c | 5 +++++ hw/remote/vfio-user-obj.c | 30 ++++++++++++++++++++++++++++++ hw/remote/trace-events | 1 + 4 files changed, 38 insertions(+)
RFC PATCH v2 10/11 vfio-user: register handlers to facilitate migration
In reply to Jagannathan RamanStore and load the device's state during migration. use libvfio-user's handlers for this purpose
--- migration/savevm.h | 2 + hw/remote/vfio-user-obj.c | 313 ++++++++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 73 +++++++++++ 3 files changed, 388 insertions(+)
migration/savevm.h +2 -0
@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f);int qemu_savevm_state_complete_precopy_non_iterable (QEMUFile *f,bool in_postcopy,bool inactivate_disks); +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev); +int qemu_remote_loadvm (QEMUFile *f);#endif
hw/remote/vfio-user-obj.c +313 -0
349 lines not on this page · raw message
migration/savevm.c +73 -0
85 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 10/11] vfio-user: register handlers to facilitate migration
Quoted 4 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +4
-0+static ssize_t vfu_mig_buf_read (void *opaque,uint8_t *buf,int64_t pos, +size_t size,Error **errp) +{ +VfuObject *o = opaque;
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 152
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+ +if (pos > o->vfu_mig_buf_size) { + size =0 ; + }else if ((pos + size) > o->vfu_mig_buf_size) { + size = o->vfu_mig_buf_size; + } + +memcpy (buf, (o->vfu_mig_buf + pos), size); + + o->vfu_mig_buf_size -= size;
This looks strange. pos increases each time we're called. We seem to be truncating the buffer on each read. Should this line be dropped? Did you test live migration (maybe this code needs more debugging)?
Quoted 60 lines
hw/remote/vfio-user-obj.c +60
-0+ +return size; +} + +static ssize_t vfu_mig_buf_write (void *opaque,struct iovec *iov,int iovcnt, +int64_t pos,Error **errp) +{ +VfuObject *o = opaque; +uint64_t end = pos +iov_size (iov, iovcnt); +int i; + +if (end > o->vfu_mig_buf_size) { + o->vfu_mig_buf =g_realloc (o->vfu_mig_buf, end); + } + +for (i =0 ; i < iovcnt; i++) { +memcpy ((o->vfu_mig_buf + o->vfu_mig_buf_size), iov[i].iov_base, + iov[i].iov_len); + o->vfu_mig_buf_size += iov[i].iov_len; + o->vfu_mig_buf_pending += iov[i].iov_len; + } + +return iov_size (iov, iovcnt); +} + +static int vfu_mig_buf_shutdown (void *opaque,bool rd,bool wr,Error **errp) +{ +VfuObject *o = opaque; + + o->vfu_mig_buf_size =0 ; + +g_free (o->vfu_mig_buf); + +return 0 ; +} + +static const QEMUFileOps vfu_mig_fops_save = { + .writev_buffer = vfu_mig_buf_write, + .shut_down = vfu_mig_buf_shutdown, +}; + +static const QEMUFileOps vfu_mig_fops_load = { + .get_buffer = vfu_mig_buf_read, + .shut_down = vfu_mig_buf_shutdown, +}; + +/** +* handlers for vfu_migration_callbacks_t +* +* The libvfio-user library accesses these handlers to drive the migration +* at the remote end, and also to transport the data stored in vfu_mig_buf +* +*/ +static void vfu_mig_state_precopy (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_save, false);
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 222
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+ } + + ret =qemu_remote_savevm (o->vfu_mig_file,DEVICE (o->pci_dev)); +if (ret) { +qemu_file_shutdown (o->vfu_mig_file); +return ; + } + +qemu_fflush (o->vfu_mig_file); +}
Are you sure pre-copy is the state where you want to serialize the savevm data? IIUC pre-copy is the iterative state while the device is still running (e.g. when copying RAM but before devices are stopped). I expected savevm to happen when we reach stop-and-copy.
The reason why this matters is that we're saving the state of the device while the guest is still running and possibly interacting with the device. The destination won't have the final state of the device, it will have an earlier state of the device when we started migrating RAM!
Maybe I'm wrong, please double-check, but this looks like a bug.
Quoted 15 lines
hw/remote/vfio-user-obj.c +15
-0+ +static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +VfuObjectClass *k =VFU_OBJECT_GET_CLASS (OBJECT (o)); +static int migrated_devs; +Error *local_err =NULL ; +int ret; + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +error_setg (&error_abort,"vfu: failed to restore device state" ); +return ; + } +
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 247
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+if (++migrated_devs == k->nr_devs) { +bdrv_invalidate_cache_all (&local_err); +if (local_err) { +error_report_err (local_err); +return ; + } + +vm_start (); + } +}
This looks like it's intended for the destination side. Does this code work on the source side if the device is transitioned back into the running state?
Quoted 97 lines
hw/remote/vfio-user-obj.c +97
-0+ +static void vfu_mig_state_stop (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +VfuObjectClass *k =VFU_OBJECT_GET_CLASS (OBJECT (o)); +static int migrated_devs; + +/** +* note: calling bdrv_inactivate_all() is not the best approach. +* +* Ideally, we would identify the block devices (if any) indirectly +* linked (such as via a scs-hd device) to each of the migrated devices, +* and inactivate them individually. This is essential while operating +* the server in a storage daemon mode, with devices from different VMs. +* +* However, we currently don't have this capability. As such, we need to +* inactivate all devices at the same time when migration is completed. +*/ +if (++migrated_devs == k->nr_devs) { +bdrv_inactivate_all (); + } +} + +static int vfu_mig_transition (vfu_ctx_t *vfu_ctx,vfu_migr_state_t state) +{ +switch (state) { +case VFU_MIGR_STATE_RESUME : +case VFU_MIGR_STATE_STOP_AND_COPY : +break ; +case VFU_MIGR_STATE_STOP : +vfu_mig_state_stop (vfu_ctx); +break ; +case VFU_MIGR_STATE_PRE_COPY : +vfu_mig_state_precopy (vfu_ctx); +break ; +case VFU_MIGR_STATE_RUNNING : +if (!runstate_is_running ()) { +vfu_mig_state_running (vfu_ctx); + } +break ; +default : +warn_report ("vfu: Unknown migration state %d" , state); + } + +return 0 ; +} + +static uint64_t vfu_mig_get_pending_bytes (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +return o->vfu_mig_buf_pending; +} + +static int vfu_mig_prepare_data (vfu_ctx_t *vfu_ctx,uint64_t *offset, +uint64_t *size) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (offset) { + *offset =0 ; + } + +if (size) { + *size = o->vfu_mig_buf_size; + } + +return 0 ; +} + +static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (offset > o->vfu_mig_buf_size) { +return -1 ; + } + +if ((offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - check pending_bytes" ); + size = o->vfu_mig_buf_size - offset; + } + +memcpy (buf, (o->vfu_mig_buf + offset), size); + + o->vfu_mig_buf_pending -= size; + +return size; +} + +static ssize_t vfu_mig_write_data (vfu_ctx_t *vfu_ctx,void *data, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t end = offset + size; +
hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 354
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+if (end > o->vfu_mig_buf_size) { + o->vfu_mig_buf =g_realloc (o->vfu_mig_buf, end); + o->vfu_mig_buf_size = end; + } + +memcpy ((o->vfu_mig_buf + offset), data, size); + +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + }
Why open the file here where it's not accessed? I expected this to happen at the point where data has been fully written and we call qemu_remote_loadvm().
Quoted 35 lines
hw/remote/vfio-user-obj.c +21
-0+ +return size; +} + +static int vfu_mig_data_written (vfu_ctx_t *vfu_ctx,uint64_t count) +{ +return 0 ; +} + +static const vfu_migration_callbacks_t vfu_mig_cbs = { + .version =VFU_MIGR_CALLBACKS_VERS , + .transition = &vfu_mig_transition, + .get_pending_bytes = &vfu_mig_get_pending_bytes, + .prepare_data = &vfu_mig_prepare_data, + .read_data = &vfu_mig_read_data, + .data_written = &vfu_mig_data_written, + .write_data = &vfu_mig_write_data, +}; +static void vfu_object_ctx_run (void *opaque) {VfuObject *o = opaque;@@ -340,6 +615,7 @@ static void vfu_object_machine_done(Notifier *notifier, void *data) {VfuObject *o =container_of (notifier, VfuObject, machine_done);DeviceState *dev =NULL ; +size_t migr_area_size;QemuThread thread;int ret;@@ -401,6 +677,35 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)return ; } +/*
hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 690
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+* TODO: The 0x20000 number used below is a temporary. We are working on +* a cleaner fix for this. +* +* The libvfio-user library assumes that the remote knows the size of +* the data to be migrated at boot time, but that is not the case with +* VMSDs, as it can contain a variable-size buffer. 0x20000 is used +* as a sufficiently large buffer to demonstrate migration, but that +* cannot be used as a solution. +* +*/
libvfio-user has the vfu_migration_callbacks_t interface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?
RFC PATCH v2 11/11 vfio-user: acceptance test
In reply to Jagannathan RamanAcceptance test for libvfio-user in QEMU
--- MAINTAINERS | 1 + tests/acceptance/vfio-user.py | 94 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 tests/acceptance/vfio-user.py
MAINTAINERS +1 -0
@@ -3392,6 +3392,7 @@ F: include/hw/remote/proxy-memory-listener.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h F: hw/remote/vfio-user-obj.c +F: tests/acceptance/vfio-user.py EBPF: M: Jason Wang <[email protected]>
tests/acceptance/vfio-user.py new file +94 -0
94 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH RFC server v2 00/11] vfio-user server in QEMU
In reply to Jagannathan RamanQuoted 5 lines · Jagannathan Raman
Hi,
This series depends on the following series from
Elena Ufimtseva <[email protected]>:
[PATCH RFC v2 00/16] vfio-user implementation
Please send future revisions as separate email threads. Tools have trouble separating your series from the one you replied to.
You can use "Based-on" to let CI know that Elena's series needs to be applied first:
Based-on: <[email protected]>
Re: [PATCH RFC server v2 00/11] vfio-user server in QEMU
Quoted 11 lines · Stefan Hajnoczi
Quoted 5 lines · Jagannathan Raman
Hi,
This series depends on the following series from
Elena Ufimtseva <[email protected]>:
[PATCH RFC v2 00/16] vfio-user implementation
Quoted 7 lines
Please send future revisions as separate email threads. Tools have trouble separating your series from the one you replied to.
You can use "Based-on" to let CI know that Elena's series needs to be applied first:
Based-on: <[email protected]>
Thank you for letting us know, Stefan! Will do going forward.
Re: [PATCH RFC server v2 00/11] vfio-user server in QEMU
In reply to Jagannathan RamanHi Jag,
I have finished reviewing these patches and left comments. I didn't take a look at the libvfio-user's implementation.
Stefan
Re: [PATCH RFC server v2 00/11] vfio-user server in QEMU
Quoted 7 lines · Stefan Hajnoczi
Hi Jag,
I have finished reviewing these patches and left comments. I didn't take a look at the libvfio-user's implementation.
Thank you for you comments, Stefan - we’ll get cracking on them. :)
--
Jag
Quoted 2 lines
Stefan