Skip to contentSkip to conversation

vfio-user implementation

RFC PATCH In review Elena Ufimtseva 8 participants 108 messages Aug 16, 2021 – Sep 16, 2021 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

v2 cover vfio-user implementation

Hi

This is v2 of the RFC patches for vfio-user multi-process QEMU project[1].

Thank you for the review of v1 of the RFC patches.

vfio-user is a protocol that allows a device to be emulated in a separate process outside of QEMU. It encapsulates the messages sent from QEMU to the kernel VFIO driver, and sends them to a remote process over a UNIX socket.

The vfio-user framework consists of 3 parts:
1) The protocol specification.
2) A server - the VFIO generic device in QEMU that exchanges the protocol messages with the client.
3) A client - remote process that emulates a device.

This patchset implements parts 1 and 2.
The protocol's specification can be found here [2]:
We also include this as the first patch of the series.

The libvfio-user project (https://github.com/nutanix/libvfio-user)
can be used by a remote process to handle the protocol to implement the third part.
We also worked on implementing a client and will be sending this patch series shortly.

Contributors:

John G Johnson <[email protected]>
John Levon <[email protected]>
Thanos Makatos <[email protected]>
Elena Ufimtseva <[email protected]>
Jagannathan Raman <[email protected]>

Changes in v2:
- combine some patches with relevant functionality.
- use SocketAddress with idea to modify later the command line options.
- define protocol bits in user-protocol.h.
- use QEMU_LOCK_GUARD where appropriate.
- fix the locking when event signaling.
- do not drop BQL on dma map/unmap.
- added checks for message sizes in communication functions.

John Johnson (15):
vfio-user: add VFIO base abstract class
vfio-user: Define type vfio_user_pci_dev_info
vfio-user: connect vfio proxy to remote server
vfio-user: define VFIO Proxy and communication functions
vfio-user: negotiate version with remote server
vfio-user: get device info
vfio-user: get region info
vfio-user: region read/write
vfio-user: pci_user_realize PCI setup
vfio-user: get and set IRQs
vfio-user: proxy container connect/disconnect
vfio-user: dma map/unmap operations
vfio-user: dma read/write operations
vfio-user: pci reset
vfio-user: migration support

Thanos Makatos (1):
vfio-user: introduce vfio-user protocol specification

 docs/devel/index.rst          |    1 +
 docs/devel/vfio-user.rst      | 1809 +++++++++++++++++++++++++++++++++
 hw/vfio/pci.h                 |   25 +-
 hw/vfio/user-protocol.h       |  210 ++++
 hw/vfio/user.h                |   95 ++
 include/hw/vfio/vfio-common.h |    9 +
 hw/vfio/common.c              |  296 +++++-
 hw/vfio/migration.c           |   34 +-
 hw/vfio/pci.c                 |  571 +++++++++--
 hw/vfio/user.c                | 1104 ++++++++++++++++++++
 MAINTAINERS                   |   11 +
 hw/vfio/meson.build           |    1 +
 12 files changed, 4062 insertions(+), 104 deletions(-)
 create mode 100644 docs/devel/vfio-user.rst
 create mode 100644 hw/vfio/user-protocol.h
 create mode 100644 hw/vfio/user.h
 create mode 100644 hw/vfio/user.c
Signature · 1 line
2.25.1
#1 lore raw

RFC PATCH v2 1/16 vfio-user: introduce vfio-user protocol specification

From: Thanos Makatos <[email protected]>

This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts.

This patch is sourced from:
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/

It has been earlier discussed as an RFC in:
"RFC: use VFIO over a UNIX domain socket to implement device offloading"

---
 docs/devel/index.rst     |    1 +
 docs/devel/vfio-user.rst | 1809 ++++++++++++++++++++++++++++++++++++++
 MAINTAINERS              |    6 +
 3 files changed, 1816 insertions(+)
 create mode 100644 docs/devel/vfio-user.rst
docs/devel/index.rst +1 -0
index 5522db7241..304ca1c12f 100644
@@ -44,3 +44,4 @@ modifying QEMU's source code.
    vfio-migration
    qapi-code-gen
    writing-qmp-commands
+   vfio-user
docs/devel/vfio-user.rst new file +1809 -0

1,809 lines not on this page · raw message

MAINTAINERS +6 -0
index 694973ed23..d838b9e3f2 100644
@@ -1879,6 +1879,12 @@ F: hw/vfio/ap.c
 F: docs/system/s390x/vfio-ap.rst
 L: [email protected]
 
+vfio-user
+M: John G Johnson <[email protected]>
+M: Thanos Makatos <[email protected]>
+S: Supported
+F: docs/devel/vfio-user.rst
+
 vhost
 M: Michael S. Tsirkin <[email protected]>
 S: Supported
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification

docs/devel/vfio-user.rst line 218

Quoted 6 lines · On Mon, 16 Aug 2021 09:42:34 -0700 Elena Ufimtseva…
docs/devel/vfio-user.rst new file +6 -0
+Authentication
+--------------
+
+For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files,
+therefore it is up to the management layer to set up the socket as required.
+Socket types than span guests or hosts will require a proper authentication

s/than/that/

...

Quoted 40 lines

+``VFIO_USER_DMA_UNMAP``
+-----------------------

docs/devel/vfio-user.rst new file +38 -0
+
+This command message is sent by the client to the server to inform it that a
+DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command
+message, is no longer available for DMA. It typically occurs when memory is
+subtracted from the client or if the client uses a vIOMMU. The DMA region is
+described by the following structure:
+
+Request
+^^^^^^^
+
+The request payload for this message is a structure of the following format:
+
++--------------+--------+------------------------+
+| Name         | Offset | Size                   |
++==============+========+========================+
+| argsz        | 0      | 4                      |
++--------------+--------+------------------------+
+| flags        | 4      | 4                      |
++--------------+--------+------------------------+
+|              | +-----+-----------------------+ |
+|              | | Bit | Definition            | |
+|              | +=====+=======================+ |
+|              | | 0   | get dirty page bitmap | |
+|              | +-----+-----------------------+ |
++--------------+--------+------------------------+
+| address      | 8      | 8                      |
++--------------+--------+------------------------+
+| size         | 16     | 8                      |
++--------------+--------+------------------------+
+
+* *argsz* is the maximum size of the reply payload.
+* *flags* contains the following DMA region attributes:
+
+  * *get dirty page bitmap* indicates that a dirty page bitmap must be
+    populated before unmapping the DMA region. The client must provide a
+    `VFIO Bitmap`_ structure, explained below, immediately following this
+    entry.
+

docs/devel/vfio-user.rst line 637

Quoted 10 lines
docs/devel/vfio-user.rst new file +10 -0
+* *address* is the base DMA address of the DMA region.
+* *size* is the size of the DMA region.
+
+The address and size of the DMA region being unmapped must match exactly a
+previous mapping. The size of request message depends on whether or not the
+*get dirty page bitmap* bit is set in Flags:
+
+* If not set, the size of the total request message is: 16 + 24.
+
+* If set, the size of the total request message is: 16 + 24 + 16.

The address/size paradigm falls into the same issues as the vfio kernel interface where we can't map or unmap the entire 64-bit address space, ie. size is limited to 2^64 - 1. The kernel interface also requires PAGE_SIZE granularity for the DMA, which means the practical limit is 2^64 - PAGE_SIZE. If we had a redo on the kernel interface we'd use start/end so we can express a size of (end - start + 1).

Is following the vfio kernel interface sufficiently worthwhile for compatibility to incur this same limitation? I don't recall if we've already discussed this, but perhaps worth a note in this design doc if similarity to the kernel interface is being favored here. See for example QEMU commit 1b296c3def4b ("vfio: Don't issue full 2^64 unmap"). Thanks,

Alex

#3 lore raw

Re: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification

On 18. Aug 2021, at 01:04, Alex Williamson <[email protected]<mailto:[email protected]>> wrote:

On Mon, 16 Aug 2021 09:42:34 -0700
Elena Ufimtseva <[email protected]<mailto:[email protected]>> wrote:
+Authentication
+--------------
+
+For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files, +therefore it is up to the management layer to set up the socket as required. +Socket types than span guests or hosts will require a proper authentication

s/than/that/
Ack

...
+``VFIO_USER_DMA_UNMAP``
+-----------------------
+
+This command message is sent by the client to the server to inform it that a +DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command +message, is no longer available for DMA. It typically occurs when memory is +subtracted from the client or if the client uses a vIOMMU. The DMA region is +described by the following structure:
+
+Request
+^^^^^^^
+
+The request payload for this message is a structure of the following format:
+
++--------------+--------+------------------------+
+| Name | Offset | Size |
++==============+========+========================+
+| argsz | 0 | 4 |
++--------------+--------+------------------------+
+| flags | 4 | 4 |
++--------------+--------+------------------------+

+|              | +-----+-----------------------+ |
+|              | | Bit | Definition            | |
+|              | +=====+=======================+ |
+|              | | 0   | get dirty page bitmap | |
+|              | +-----+-----------------------+ |

++--------------+--------+------------------------+
+| address | 8 | 8 |
++--------------+--------+------------------------+
+| size | 16 | 8 |
++--------------+--------+------------------------+
+
+* *argsz* is the maximum size of the reply payload.
+* *flags* contains the following DMA region attributes:
+
+ * *get dirty page bitmap* indicates that a dirty page bitmap must be

+    populated before unmapping the DMA region. The client must provide a
+    `VFIO Bitmap`_ structure, explained below, immediately following this
+    entry.

+
+* *address* is the base DMA address of the DMA region.
+* *size* is the size of the DMA region.
+
+The address and size of the DMA region being unmapped must match exactly a +previous mapping. The size of request message depends on whether or not the +*get dirty page bitmap* bit is set in Flags:
+
+* If not set, the size of the total request message is: 16 + 24. +
+* If set, the size of the total request message is: 16 + 24 + 16.

The address/size paradigm falls into the same issues as the vfio kernel interface where we can't map or unmap the entire 64-bit address space, ie. size is limited to 2^64 - 1. The kernel interface also requires PAGE_SIZE granularity for the DMA, which means the practical limit is 2^64 - PAGE_SIZE. If we had a redo on the kernel interface we'd use start/end so we can express a size of (end - start + 1).

Is following the vfio kernel interface sufficiently worthwhile for compatibility to incur this same limitation? I don't recall if we've already discussed this, but perhaps worth a note in this design doc if similarity to the kernel interface is being favored here. See for example QEMU commit 1b296c3def4b ("vfio: Don't issue full 2^64 unmap”). I cannot think of any reason that we need to have this limitation in vfio-user. Opened https://github.com/nutanix/libvfio-user/issues/593 to address this issue.

Thanks,
-Swapnil

#4 lore raw

Re: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification

In reply to Alex Williamson
Quoted 12 lines · Alex Williamson

The address/size paradigm falls into the same issues as the vfio kernel interface where we can't map or unmap the entire 64-bit address space, ie. size is limited to 2^64 - 1. The kernel interface also requires PAGE_SIZE granularity for the DMA, which means the practical limit is 2^64 - PAGE_SIZE. If we had a redo on the kernel interface we'd use start/end so we can express a size of (end - start + 1).

Quoted 7 lines

Is following the vfio kernel interface sufficiently worthwhile for compatibility to incur this same limitation? I don't recall if we've already discussed this, but perhaps worth a note in this design doc if similarity to the kernel interface is being favored here. See for example QEMU commit 1b296c3def4b ("vfio: Don't issue full 2^64 unmap"). Thanks,

I’d prefer to stay as close to the kernel i/f as we can.
An earlier version of the spec used a vhost-user derived structure for MAP & UNMAP. This made it more difficult to add the bitmap field when vfio added migration capability, so we switched to the ioctl() structure.

It looks like vfio_dma_unmap() takes a 64b ‘size’ arg
(ram_addr_t) How did you unmap an entire 64b address space? The comment there mentions a bug where iova+size wraps the end of the 64b space.

JJ

#5 lore raw

Re: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification

Quoted 19 lines · On Thu, 19 Aug 2021 15:32:16 +0000 John Johnson <[email protected]>…
Quoted 10 lines · Alex Williamson

The address/size paradigm falls into the same issues as the vfio kernel interface where we can't map or unmap the entire 64-bit address space, ie. size is limited to 2^64 - 1. The kernel interface also requires PAGE_SIZE granularity for the DMA, which means the practical limit is 2^64 - PAGE_SIZE. If we had a redo on the kernel interface we'd use start/end so we can express a size of (end - start + 1).

Quoted 7 lines

Is following the vfio kernel interface sufficiently worthwhile for compatibility to incur this same limitation? I don't recall if we've already discussed this, but perhaps worth a note in this design doc if similarity to the kernel interface is being favored here. See for example QEMU commit 1b296c3def4b ("vfio: Don't issue full 2^64 unmap"). Thanks,

Quoted 8 lines

I’d prefer to stay as close to the kernel i/f as we can.
An earlier version of the spec used a vhost-user derived structure for MAP & UNMAP. This made it more difficult to add the bitmap field when vfio added migration capability, so we switched to the ioctl() structure.

It looks like vfio_dma_unmap() takes a 64b ‘size’ arg
(ram_addr_t) How did you unmap an entire 64b address space?

It's called from the MemoryListener which operates on MemoryRegionSections, which uses Int128 that get's chunked to ram_addr_t for vfio_dma_unmap(). We do now have VFIO_DMA_UNMAP_FLAG_ALL in the kernel API which gives us an option to clear the whole 64bit address space in one ioctl, but it's not a high priority to make use of in QEMU since it still needs to handle older kernels.

Quoted 2 lines

The comment there mentions a bug where iova+size wraps the end of the 64b space.

Right, that's a separate issue that's just a bug in the kernel. That's been fixed, but the QEMU code exists for now as a workaround for any broken kernels in the wild. Thanks,

Alex

#6 lore raw

RFC PATCH v2 2/16 vfio-user: add VFIO base abstract class

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

Add an abstract base class both the kernel driver and user socket implementations can use to share code.

---
 hw/vfio/pci.h | 16 +++++++++++--
 hw/vfio/pci.c | 63 ++++++++++++++++++++++++++++++++-------------------
 2 files changed, 54 insertions(+), 25 deletions(-)
hw/vfio/pci.h +14 -2
index 64777516d1..bbc78aaeb3 100644
@@ -114,8 +114,13 @@ typedef struct VFIOMSIXInfo {
     unsigned long *pending;
 } VFIOMSIXInfo;
 
-#define TYPE_VFIO_PCI "vfio-pci"
-OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI)
+/*
+ * TYPE_VFIO_PCI_BASE is an abstract type used to share code
+ * between VFIO implementations that use a kernel driver
+ * with those that use user sockets.
+ */
+#define TYPE_VFIO_PCI_BASE "vfio-pci-base"
+OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_BASE)
 
 struct VFIOPCIDevice {
     PCIDevice pdev;
@@ -175,6 +180,13 @@ struct VFIOPCIDevice {
     Notifier irqchip_change_notifier;
 };
 
+#define TYPE_VFIO_PCI "vfio-pci"
+OBJECT_DECLARE_SIMPLE_TYPE(VFIOKernPCIDevice, VFIO_PCI)
+
+struct VFIOKernPCIDevice {
+    VFIOPCIDevice device;
+};
+
 /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */
 static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device)
 {
hw/vfio/pci.c +40 -23

163 lines not on this page · raw message

Signature · 1 line
2.25.1
#7 lore raw

RFC PATCH v2 3/16 vfio-user: Define type vfio_user_pci_dev_info

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

New class for vfio-user with its class and instance constructors and destructors, and its pci ops.

---
 hw/vfio/pci.h |  9 ++++++
 hw/vfio/pci.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
hw/vfio/pci.h +9 -0
index bbc78aaeb3..08ac6475a4 100644
@@ -187,6 +187,15 @@ struct VFIOKernPCIDevice {
     VFIOPCIDevice device;
 };
 
+#define TYPE_VFIO_USER_PCI "vfio-user-pci"
+OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI)
+
+struct VFIOUserPCIDevice {
+    VFIOPCIDevice device;
+    char *sock_name;
+    bool secure_dma; /* disable shared mem for DMA */
+};
+
 /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */
 static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device)
 {
hw/vfio/pci.c +86 -0

89 lines not on this page · raw message

Signature · 1 line
2.25.1
#8 lore raw

Re: [PATCH RFC v2 03/16] vfio-user: Define type vfio_user_pci_dev_info

hw/vfio/pci.c static void register_vfio_pci_dev_type(void) · line 3388

Quoted 5 lines · Elena Ufimtseva
hw/vfio/pci.c +5 -0
+static Property vfio_user_pci_dev_properties[] = {
+    DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),
+    DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false),
+    DEFINE_PROP_END_OF_LIST(),
+};

Are we missing out on properties that could be common for all VFIO PCI devices like x-pci-vendor-id, x-pci-device-id, etc?

Stefan

#9 lore raw

RFC PATCH v2 4/16 vfio-user: connect vfio proxy to remote server

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user.h                |  66 ++++++++++++++
 include/hw/vfio/vfio-common.h |   2 +
 hw/vfio/pci.c                 |  29 ++++++
 hw/vfio/user.c                | 160 ++++++++++++++++++++++++++++++++++
 MAINTAINERS                   |   4 +
 hw/vfio/meson.build           |   1 +
 6 files changed, 262 insertions(+)
 create mode 100644 hw/vfio/user.h
 create mode 100644 hw/vfio/user.c
hw/vfio/user.h new file +66 -0
new file mode 100644
index 0000000000..62b2d03d56
@@ -0,0 +1,66 @@
+#ifndef VFIO_USER_H
+#define VFIO_USER_H
+
+/*
+ * vfio protocol over a UNIX socket.
+ *
+ * Copyright © 2018, 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+typedef struct {
+    int send_fds;
+    int recv_fds;
+    int *fds;
+} VFIOUserFDs;
+
+typedef struct VFIOUserReply {
+    QTAILQ_ENTRY(VFIOUserReply) next;
+    VFIOUserFDs *fds;
+    uint32_t rsize;
+    uint32_t id;
+    QemuCond cv;
+    bool complete;
+    bool nowait;
+} VFIOUserReply;
+
+
+enum proxy_state {
+    VFIO_PROXY_CONNECTED = 1,
+    VFIO_PROXY_RECV_ERROR = 2,
+    VFIO_PROXY_CLOSING = 3,
+    VFIO_PROXY_CLOSED = 4,
+};
+
+typedef struct VFIOProxy {
+    QLIST_ENTRY(VFIOProxy) next;
+    char *sockname;
+    struct QIOChannel *ioc;
+    int (*request)(void *opaque, char *buf, VFIOUserFDs *fds);
+    void *reqarg;
+    int flags;
+    QemuCond close_cv;
+
+    /*
+     * above only changed when BQL is held
+     * below are protected by per-proxy lock
+     */
+    QemuMutex lock;
+    QTAILQ_HEAD(, VFIOUserReply) free;
+    QTAILQ_HEAD(, VFIOUserReply) pending;
+    VFIOUserReply *last_nowait;
+    enum proxy_state state;
+    bool close_wait;
+} VFIOProxy;
+
+/* VFIOProxy flags */
+#define VFIO_PROXY_CLIENT       0x1
+#define VFIO_PROXY_SECURE       0x2
+
+VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp);
+void vfio_user_disconnect(VFIOProxy *proxy);
+
+#endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +2 -0
index 8af11b0a76..f43dc6e5d0 100644
@@ -75,6 +75,7 @@ typedef struct VFIOAddressSpace {
 } VFIOAddressSpace;
 
 struct VFIOGroup;
+typedef struct VFIOProxy VFIOProxy;
 
 typedef struct VFIOContainer {
     VFIOAddressSpace *space;
@@ -143,6 +144,7 @@ typedef struct VFIODevice {
     VFIOMigration *migration;
     Error *migration_blocker;
     OnOffAuto pre_copy_dirty_page_tracking;
+    VFIOProxy *proxy;
 } VFIODevice;
 
 struct VFIODeviceOps {
hw/vfio/pci.c +29 -0
index d642aafb7f..7c2d245ca5 100644
@@ -42,6 +42,7 @@
 #include "qapi/error.h"
 #include "migration/blocker.h"
 #include "migration/qemu-file.h"
+#include "hw/vfio/user.h"
 
 #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug"
 
@@ -3361,13 +3362,35 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev);
     VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev);
     VFIODevice *vbasedev = &vdev->vbasedev;
+    SocketAddress addr;
+    VFIOProxy *proxy;
+    Error *err = NULL;
 
+    /*
+     * TODO: make option parser understand SocketAddress
+     * and use that instead of having scaler options
+     * for each socket type.
+     */
     if (!udev->sock_name) {
         error_setg(errp, "No socket specified");
         error_append_hint(errp, "Use -device vfio-user-pci,socket=<name>\n");
         return;
     }
 
+    memset(&addr, 0, sizeof(addr));
+    addr.type = SOCKET_ADDRESS_TYPE_UNIX;
+    addr.u.q_unix.path = udev->sock_name;
+    proxy = vfio_user_connect_dev(&addr, &err);
+    if (!proxy) {
+        error_setg(errp, "Remote proxy not found");
+        return;
+    }
+    vbasedev->proxy = proxy;
+
+    if (udev->secure_dma) {
+        proxy->flags |= VFIO_PROXY_SECURE;
+    }
+
     vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name);
     vbasedev->dev = DEVICE(vdev);
     vbasedev->fd = -1;
@@ -3379,6 +3402,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
 
 static void vfio_user_instance_finalize(Object *obj)
 {
+    VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj);
+    VFIODevice *vbasedev = &vdev->vbasedev;
+
+    vfio_put_device(vdev);
+
+    vfio_user_disconnect(vbasedev->proxy);
 }
 
 static Property vfio_user_pci_dev_properties[] = {
hw/vfio/user.c new file +160 -0

160 lines not on this page · raw message

MAINTAINERS +4 -0
index d838b9e3f2..f429bab391 100644
@@ -1882,8 +1882,12 @@ L: [email protected]
 vfio-user
 M: John G Johnson <[email protected]>
 M: Thanos Makatos <[email protected]>
+M: Elena Ufimtseva <[email protected]>
+M: Jagannathan Raman <[email protected]>
 S: Supported
 F: docs/devel/vfio-user.rst
+F: hw/vfio/user.c
+F: hw/vfio/user.h
 
 vhost
 M: Michael S. Tsirkin <[email protected]>
hw/vfio/meson.build +1 -0
index da9af297a0..739b30be73 100644
@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files(
   'display.c',
   'pci-quirks.c',
   'pci.c',
+  'user.c',
 ))
 vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c'))
 vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))
Signature · 1 line
2.25.1
#10 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

hw/vfio/meson.build vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( · line 14

Quoted 12 lines · On Mon, 16 Aug 2021 09:42:37 -0700 Elena Ufimtseva…
hw/vfio/meson.build +1 -0
index da9af297a0..739b30be73 100644
@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files(
   'display.c',
   'pci-quirks.c',
   'pci.c',
+  'user.c',
 ))
 vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c'))
 vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))

Wouldn't it make sense to be able to configure QEMU with any combination of vfio-pci and/or vfio-user-pci support rather than statically tying vfio-user-pci to vfio-pci? Not to mention that doing so would help to more formally define the interface operations between kernel and user options, for example fewer tests of vbasedev->proxy and perhaps more abstraction through ops structures. Thanks,

Alex

#11 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 18 lines · Alex Williamson
Quoted 11 lines · On Mon, 16 Aug 2021 09:42:37 -0700 Elena Ufimtseva…
hw/vfio/meson.build +1 -0
index da9af297a0..739b30be73 100644
@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files(
  'display.c',
  'pci-quirks.c',
  'pci.c',
+  'user.c',
 ))
 vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c'))
Quoted 9 lines

hw/vfio/meson.build vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( · line 14

Quoted 1 line
hw/vfio/meson.build +0 -0
 vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))

Wouldn't it make sense to be able to configure QEMU with any combination of vfio-pci and/or vfio-user-pci support rather than statically tying vfio-user-pci to vfio-pci? Not to mention that doing so would help to more formally define the interface operations between kernel and user options, for example fewer tests of vbasedev->proxy and perhaps more abstraction through ops structures. Thanks,

	We can certainly add another config option for vfio-user.

	As far as an ops vector vs vbasedev->proxy tests, it’s a

matter personal preference. I prefer the changes inline when they are this small, but we can make a vector if that’s what you want.

JJ

#12 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

In reply to Elena Ufimtseva

hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3371

Quoted 11 lines · Elena Ufimtseva
hw/vfio/pci.c +6 -0
@@ -3361,13 +3362,35 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev);
     VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev);
     VFIODevice *vbasedev = &vdev->vbasedev;
+    SocketAddress addr;
+    VFIOProxy *proxy;
+    Error *err = NULL;
 
+    /*
+     * TODO: make option parser understand SocketAddress
+     * and use that instead of having scaler options

s/scaler/scalar/

Quoted 13 lines
hw/vfio/user.c new file +13 -0
+VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp)
+{
+    VFIOProxy *proxy;
+    QIOChannelSocket *sioc;
+    QIOChannel *ioc;
+    char *sockname;
+
+    if (addr->type != SOCKET_ADDRESS_TYPE_UNIX) {
+        error_setg(errp, "vfio_user_connect - bad address family");
+        return NULL;
+    }
+    sockname = addr->u.q_unix.path;
+

hw/vfio/user.c line 86

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+    sioc = qio_channel_socket_new();
+    ioc = QIO_CHANNEL(sioc);
+    if (qio_channel_socket_connect_sync(sioc, addr, errp)) {
+        object_unref(OBJECT(ioc));
+        return NULL;
+    }
+    qio_channel_set_blocking(ioc, true, NULL);
+
+    proxy = g_malloc0(sizeof(VFIOProxy));
+    proxy->sockname = sockname;

sockname is addr->u.q_unix.path, so there's an assumption that the lifetime of the addr argument is at least as long as the proxy object's lifetime. This doesn't seem to be the case in vfio_user_pci_realize() since the SocketAddress variable is declared on the stack.

I suggest making SocketAddress *addr const so it's obvious that this function just reads it (doesn't take ownership of the pointer) and copying the UNIX domain socket path with g_strdup() to avoid the dangling pointer.

hw/vfio/user.c line 94

Quoted 8 lines
hw/vfio/user.c new file +8 -0
+    proxy->ioc = ioc;
+    proxy->flags = VFIO_PROXY_CLIENT;
+    proxy->state = VFIO_PROXY_CONNECTED;
+    qemu_cond_init(&proxy->close_cv);
+
+    if (vfio_user_iothread == NULL) {
+        vfio_user_iothread = iothread_create("VFIO user", errp);
+    }

Why is a dedicated IOThread needed for VFIO user?

hw/vfio/user.c line 103

Quoted 9 lines
hw/vfio/user.c new file +9 -0
+
+    qemu_mutex_init(&proxy->lock);
+    QTAILQ_INIT(&proxy->free);
+    QTAILQ_INIT(&proxy->pending);
+    QLIST_INSERT_HEAD(&vfio_user_sockets, proxy, next);
+
+    return proxy;
+}
+

/* Called with the BQL */

Quoted 25 lines
hw/vfio/user.c new file +25 -0
+void vfio_user_disconnect(VFIOProxy *proxy)
+{
+    VFIOUserReply *r1, *r2;
+
+    qemu_mutex_lock(&proxy->lock);
+
+    /* our side is quitting */
+    if (proxy->state == VFIO_PROXY_CONNECTED) {
+        vfio_user_shutdown(proxy);
+        if (!QTAILQ_EMPTY(&proxy->pending)) {
+            error_printf("vfio_user_disconnect: outstanding requests\n");
+        }
+    }
+    object_unref(OBJECT(proxy->ioc));
+    proxy->ioc = NULL;
+
+    proxy->state = VFIO_PROXY_CLOSING;
+    QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) {
+        qemu_cond_destroy(&r1->cv);
+        QTAILQ_REMOVE(&proxy->pending, r1, next);
+        g_free(r1);
+    }
+    QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) {
+        qemu_cond_destroy(&r1->cv);
+        QTAILQ_REMOVE(&proxy->free, r1, next);

hw/vfio/user.c line 138

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+        g_free(r1);
+    }
+
+    /*
+     * Make sure the iothread isn't blocking anywhere
+     * with a ref to this proxy by waiting for a BH
+     * handler to run after the proxy fd handlers were
+     * deleted above.
+     */
+    proxy->close_wait = 1;

Please use true. '1' is shorter but it's less obvious to the reader (I had to jump to the definition to check whether this field was bool or int).

hw/vfio/user.c line 143

Quoted 5 lines
hw/vfio/user.c new file +5 -0
+    aio_bh_schedule_oneshot(iothread_get_aio_context(vfio_user_iothread),
+                            vfio_user_cb, proxy);
+
+    /* drop locks so the iothread can make progress */
+    qemu_mutex_unlock_iothread();

Why does the BQL needs to be dropped so vfio_user_iothread can make progress?

Quoted 1 line

+ qemu_cond_wait(&proxy->close_cv, &proxy->lock);

#13 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 13 lines · Stefan Hajnoczi
Quoted 8 lines · Elena Ufimtseva
hw/vfio/pci.c +3 -0
@@ -3361,13 +3362,35 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
    VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev);
    VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev);
    VFIODevice *vbasedev = &vdev->vbasedev;
+    SocketAddress addr;
+    VFIOProxy *proxy;
+    Error *err = NULL;
 
Quoted 6 lines

hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3371

Quoted 3 lines
hw/vfio/pci.c +3 -0
+    /*
+     * TODO: make option parser understand SocketAddress
+     * and use that instead of having scaler options

s/scaler/scalar/

OK

Quoted 29 lines
Quoted 13 lines
hw/vfio/user.c new file +13 -0
+VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp)
+{
+    VFIOProxy *proxy;
+    QIOChannelSocket *sioc;
+    QIOChannel *ioc;
+    char *sockname;
+
+    if (addr->type != SOCKET_ADDRESS_TYPE_UNIX) {
+        error_setg(errp, "vfio_user_connect - bad address family");
+        return NULL;
+    }
+    sockname = addr->u.q_unix.path;
+

hw/vfio/user.c line 86

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+    sioc = qio_channel_socket_new();
+    ioc = QIO_CHANNEL(sioc);
+    if (qio_channel_socket_connect_sync(sioc, addr, errp)) {
+        object_unref(OBJECT(ioc));
+        return NULL;
+    }
+    qio_channel_set_blocking(ioc, true, NULL);
+
+    proxy = g_malloc0(sizeof(VFIOProxy));
+    proxy->sockname = sockname;

sockname is addr->u.q_unix.path, so there's an assumption that the lifetime of the addr argument is at least as long as the proxy object's lifetime. This doesn't seem to be the case in vfio_user_pci_realize() since the SocketAddress variable is declared on the stack.

Quoted 5 lines

I suggest making SocketAddress *addr const so it's obvious that this function just reads it (doesn't take ownership of the pointer) and copying the UNIX domain socket path with g_strdup() to avoid the dangling pointer.

OK

Quoted 11 lines

hw/vfio/user.c line 94

Quoted 8 lines
hw/vfio/user.c new file +8 -0
+    proxy->ioc = ioc;
+    proxy->flags = VFIO_PROXY_CLIENT;
+    proxy->state = VFIO_PROXY_CONNECTED;
+    qemu_cond_init(&proxy->close_cv);
+
+    if (vfio_user_iothread == NULL) {
+        vfio_user_iothread = iothread_create("VFIO user", errp);
+    }

Why is a dedicated IOThread needed for VFIO user?

It seemed the best model for inbound message processing. Most messages
are replies, so the receiver will either signal a thread waiting the reply or report any errors from the server if there is no waiter. None of this requires the BQL.

If the message is a request - which currently only happens if device
DMA targets guest memory that wasn’t mmap()d by QEMU or if the ’secure-dma’ option is used - then the receiver can then acquire BQL so it can call the VFIO code to handle the request.

Quoted 11 lines

hw/vfio/user.c line 103

Quoted 9 lines
hw/vfio/user.c new file +9 -0
+
+    qemu_mutex_init(&proxy->lock);
+    QTAILQ_INIT(&proxy->free);
+    QTAILQ_INIT(&proxy->pending);
+    QLIST_INSERT_HEAD(&vfio_user_sockets, proxy, next);
+
+    return proxy;
+}
+

/* Called with the BQL */

OK

Quoted 31 lines
Quoted 31 lines
hw/vfio/user.c new file +31 -0
+void vfio_user_disconnect(VFIOProxy *proxy)
+{
+    VFIOUserReply *r1, *r2;
+
+    qemu_mutex_lock(&proxy->lock);
+
+    /* our side is quitting */
+    if (proxy->state == VFIO_PROXY_CONNECTED) {
+        vfio_user_shutdown(proxy);
+        if (!QTAILQ_EMPTY(&proxy->pending)) {
+            error_printf("vfio_user_disconnect: outstanding requests\n");
+        }
+    }
+    object_unref(OBJECT(proxy->ioc));
+    proxy->ioc = NULL;
+
+    proxy->state = VFIO_PROXY_CLOSING;
+    QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) {
+        qemu_cond_destroy(&r1->cv);
+        QTAILQ_REMOVE(&proxy->pending, r1, next);
+        g_free(r1);
+    }
+    QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) {
+        qemu_cond_destroy(&r1->cv);
+        QTAILQ_REMOVE(&proxy->free, r1, next);
+        g_free(r1);
+    }
+
+    /*
+     * Make sure the iothread isn't blocking anywhere
+     * with a ref to this proxy by waiting for a BH
Quoted 9 lines

hw/vfio/user.c line 138

Quoted 4 lines
hw/vfio/user.c new file +4 -0
+     * handler to run after the proxy fd handlers were
+     * deleted above.
+     */
+    proxy->close_wait = 1;

Please use true. '1' is shorter but it's less obvious to the reader (I had to jump to the definition to check whether this field was bool or int).

I assume this is also true for the other boolean struct members
I’ve added.

Quoted 9 lines

hw/vfio/user.c line 143

Quoted 5 lines
hw/vfio/user.c new file +5 -0
+    aio_bh_schedule_oneshot(iothread_get_aio_context(vfio_user_iothread),
+                            vfio_user_cb, proxy);
+
+    /* drop locks so the iothread can make progress */
+    qemu_mutex_unlock_iothread();

Why does the BQL needs to be dropped so vfio_user_iothread can make progress?

See above. Acquiring BQL by the iothread is rare, but I have to
handle the case where a disconnect is concurrent with an incoming request message that is waiting for the BQL. See the proxy state check after BQL is acquired in vfio_user_recv()

Quoted 1 line
Quoted 1 line

+ qemu_cond_wait(&proxy->close_cv, &proxy->lock);

JJ

#14 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 19 lines · John Johnson
Quoted 11 lines · Stefan Hajnoczi

hw/vfio/user.c line 94

Quoted 8 lines · Elena Ufimtseva
hw/vfio/user.c new file +8 -0
+    proxy->ioc = ioc;
+    proxy->flags = VFIO_PROXY_CLIENT;
+    proxy->state = VFIO_PROXY_CONNECTED;
+    qemu_cond_init(&proxy->close_cv);
+
+    if (vfio_user_iothread == NULL) {
+        vfio_user_iothread = iothread_create("VFIO user", errp);
+    }
Quoted 2 lines

Why is a dedicated IOThread needed for VFIO user?

It seemed the best model for inbound message processing. Most messages
are replies, so the receiver will either signal a thread waiting the reply or report any errors from the server if there is no waiter. None of this requires the BQL.

Quoted 4 lines

If the message is a request - which currently only happens if device
DMA targets guest memory that wasn’t mmap()d by QEMU or if the ’secure-dma’ option is used - then the receiver can then acquire BQL so it can call the VFIO code to handle the request.

QEMU is generally event-driven and the APIs are designed for that style. Threads in QEMU are there for parallelism or resource control, everything else is event-driven.

It's not clear to me if there is a reason why the message processing must be done in a separate thread or whether it is just done this way because the code was written in multi-threaded style instead of event-driven style.

You mentioned other threads waiting for replies. Which threads are they?

Quoted 7 lines
Quoted 4 lines

Please use true. '1' is shorter but it's less obvious to the reader (I had to jump to the definition to check whether this field was bool or int).

I assume this is also true for the other boolean struct members
I’ve added.

Yes, please. QEMU uses bool and true/false.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
hw/vfio/user.c new file +5 -0
+    aio_bh_schedule_oneshot(iothread_get_aio_context(vfio_user_iothread),
+                            vfio_user_cb, proxy);
+
+    /* drop locks so the iothread can make progress */
+    qemu_mutex_unlock_iothread();
Quoted 8 lines
Quoted 3 lines

Why does the BQL needs to be dropped so vfio_user_iothread can make progress?

See above. Acquiring BQL by the iothread is rare, but I have to
handle the case where a disconnect is concurrent with an incoming request message that is waiting for the BQL. See the proxy state check after BQL is acquired in vfio_user_recv()

Here is how this code looks when written using coroutines (this is from nbd/server.c):

  static coroutine_fn void nbd_trip(void *opaque)
  {
      ...
      req = nbd_request_get(client);
      ret = nbd_co_receive_request(req, &request, &local_err);
      client->recv_coroutine = NULL;
      if (client->closing) {
          /*
           * The client may be closed when we are blocked in
           * nbd_co_receive_request()
           */
          goto done;
      }

It's the same check. The code is inverted: the server reads the next request using nbd_co_receive_request() (this coroutine function can yield while waiting for data on the socket).

This way the network communication code doesn't need to know how messages will by processed by the client or server. There is no need for if (isreply) { qemu_cond_signal(&reply->cv); } else { proxy->request(proxy->reqarg, buf, &reqfds); }. The callbacks and threads aren't hardcoded into the network communication code.

This goes back to the question earlier about why a dedicated thread is necessary here. I suggest writing the network communication code using coroutines. That way the code is easier to read (no callbacks or thread synchronization), there are fewer thread-safety issues to worry about, and users or management tools don't need to know about additional threads (e.g. CPU/NUMA affinity).

Stefan

#15 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 11 lines · Stefan Hajnoczi

This way the network communication code doesn't need to know how messages will by processed by the client or server. There is no need for if (isreply) { qemu_cond_signal(&reply->cv); } else { proxy->request(proxy->reqarg, buf, &reqfds); }. The callbacks and threads aren't hardcoded into the network communication code.

I fear we are talking past each other. The vfio-user protocol
is bi-directional. e.g., the client both sends requests to the server and receives requests from the server on the same socket. No matter what threading model we use, the receive algorithm will be:

read message header
if it’s a reply
schedule the thread waiting for the reply
else
run a callback to process the request

The only way I can see changing this is to establish two
uni-directional sockets: one for requests outbound to the server, and one for requests inbound from the server.

This is the reason I chose the iothread model. It can run
independently of any vCPU/main threads waiting for replies and of the callback thread. I did muddle this idea by having the iothread become a callback thread by grabbing BQL and running the callback inline when it receives a request from the server, but if you like a pure event driven model, I can make incoming requests kick a BH from the main loop. e.g.,

if it’s a reply
qemu_cond_signal(reply cv)
else
qemu_bh_schedule(proxy bh)

That would avoid disconnect having to handle the iothread
blocked on BQL.

Quoted 7 lines

This goes back to the question earlier about why a dedicated thread is necessary here. I suggest writing the network communication code using coroutines. That way the code is easier to read (no callbacks or thread synchronization), there are fewer thread-safety issues to worry about, and users or management tools don't need to know about additional threads (e.g. CPU/NUMA affinity).

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

JJ

#16 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 55 lines · John Johnson
Quoted 11 lines · Stefan Hajnoczi

This way the network communication code doesn't need to know how messages will by processed by the client or server. There is no need for if (isreply) { qemu_cond_signal(&reply->cv); } else { proxy->request(proxy->reqarg, buf, &reqfds); }. The callbacks and threads aren't hardcoded into the network communication code.

I fear we are talking past each other. The vfio-user protocol
is bi-directional. e.g., the client both sends requests to the server and receives requests from the server on the same socket. No matter what threading model we use, the receive algorithm will be:

read message header
if it’s a reply
schedule the thread waiting for the reply
else
run a callback to process the request

The only way I can see changing this is to establish two
uni-directional sockets: one for requests outbound to the server, and one for requests inbound from the server.

This is the reason I chose the iothread model. It can run
independently of any vCPU/main threads waiting for replies and of the callback thread. I did muddle this idea by having the iothread become a callback thread by grabbing BQL and running the callback inline when it receives a request from the server, but if you like a pure event driven model, I can make incoming requests kick a BH from the main loop. e.g.,

if it’s a reply
qemu_cond_signal(reply cv)
else
qemu_bh_schedule(proxy bh)

That would avoid disconnect having to handle the iothread
blocked on BQL.

Quoted 7 lines

This goes back to the question earlier about why a dedicated thread is necessary here. I suggest writing the network communication code using coroutines. That way the code is easier to read (no callbacks or thread synchronization), there are fewer thread-safety issues to worry about, and users or management tools don't need to know about additional threads (e.g. CPU/NUMA affinity).

Quoted 3 lines

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

  if it's a reply
      reply = find_reply(...)
      qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
  else
      QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
      if (pending_reqs_was_empty) {
          qemu_coroutine_enter(process_request_co);
      }

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

Stefan

#17 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 34 lines · Stefan Hajnoczi
Quoted 5 lines · John Johnson

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

if it's a reply

     reply = find_reply(...)
     qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
 else
     QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
     if (pending_reqs_was_empty) {
         qemu_coroutine_enter(process_request_co);
     }
Quoted 3 lines

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller.

JJ

#18 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 39 lines · John Johnson
Quoted 34 lines · Stefan Hajnoczi
Quoted 5 lines · John Johnson

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

if it's a reply

     reply = find_reply(...)
     qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
 else
     QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
     if (pending_reqs_was_empty) {
         qemu_coroutine_enter(process_request_co);
     }
Quoted 3 lines

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

Quoted 4 lines

How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller.

A vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.

For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:

  Coroutine *co = qemu_coroutine_create(co_entry, &rwco);
  bdrv_coroutine_enter(blk_bs(blk), co);
  BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);

BDRV_POLL_WHILE() boils down to a loop like this:

  while ((cond)) {
    aio_poll(ctx, true);
  }

I also want to check that I understand the scenarios in which the vfio-user communication code is used:

1. vhost-user-server

The vfio-user communication code should run in a given AioContext (it will be the main loop by default but maybe the user will be able to configure a specific IOThread in the future).

2. vCPU thread vfio-user clients

The vfio-user communication code is called from the vCPU thread where the proxy device executes. The MemoryRegion->read()/write() callbacks are synchronous, so the thread needs to wait for a vfio-user reply before it can return.

Is this what you had in mind?

Stefan

#19 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

In reply to John Johnson
Quoted 62 lines
Quoted 43 lines · John Johnson
Quoted 35 lines · Stefan Hajnoczi
Quoted 5 lines · John Johnson

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

if it's a reply

    reply = find_reply(...)
    qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else
    QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
    if (pending_reqs_was_empty) {
        qemu_coroutine_enter(process_request_co);
    }
Quoted 3 lines

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

Quoted 4 lines

How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller

A vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.

For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:

  Coroutine *co = qemu_coroutine_create(co_entry, &rwco);
  bdrv_coroutine_enter(blk_bs(blk), co);
  BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);
Quoted 6 lines

BDRV_POLL_WHILE() boils down to a loop like this:

  while ((cond)) {
    aio_poll(ctx, true);
  }

I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.

Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.

Quoted 9 lines

I also want to check that I understand the scenarios in which the vfio-user communication code is used:

1. vhost-user-server

The vfio-user communication code should run in a given AioContext (it will be the main loop by default but maybe the user will be able to configure a specific IOThread in the future).

Jag would know more, but I believe it runs off the main loop.
Running it in an iothread doesn’t gain much, since it needs BQL to run the device emulation code.

Quoted 8 lines

2. vCPU thread vfio-user clients

The vfio-user communication code is called from the vCPU thread where the proxy device executes. The MemoryRegion->read()/write() callbacks are synchronous, so the thread needs to wait for a vfio-user reply before it can return.

Is this what you had in mind?

The client is also called from the main thread - the GET_*
messages from vfio_user_pci_realize() as well as MAP/DEMAP messages from guest address space change transactions. It is also called by the migration thread, which is a separate thread that does not run holding BQL.

JJ

#20 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 29 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

if it's a reply

    reply = find_reply(...)
    qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else
    QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
    if (pending_reqs_was_empty) {
        qemu_coroutine_enter(process_request_co);
    }
Quoted 3 lines

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

Quoted 4 lines

How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller

A vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.

For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:

  Coroutine *co = qemu_coroutine_create(co_entry, &rwco);
  bdrv_coroutine_enter(blk_bs(blk), co);
  BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);
Quoted 6 lines

BDRV_POLL_WHILE() boils down to a loop like this:

  while ((cond)) {
    aio_poll(ctx, true);
  }

I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.

Quoted 4 lines

Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.

Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:

  if it's a reply
      reply = find_reply(...)
      qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
  else
      QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
      if (pending_reqs_was_empty) {
          qemu_coroutine_enter(process_request_co);
      }

The qemu_coroutine_enter(reply->co) call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signalling reply->cv in the existing patch.

Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call aio_poll() (which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.

Quoted 22 lines
Quoted 9 lines

I also want to check that I understand the scenarios in which the vfio-user communication code is used:

1. vhost-user-server

The vfio-user communication code should run in a given AioContext (it will be the main loop by default but maybe the user will be able to configure a specific IOThread in the future).

Jag would know more, but I believe it runs off the main loop.
Running it in an iothread doesn’t gain much, since it needs BQL to run the device emulation code.

Quoted 7 lines

2. vCPU thread vfio-user clients

The vfio-user communication code is called from the vCPU thread where the proxy device executes. The MemoryRegion->read()/write() callbacks are synchronous, so the thread needs to wait for a vfio-user reply before it can return.

Quoted 7 lines
Quoted 1 line

Is this what you had in mind?

The client is also called from the main thread - the GET_*
messages from vfio_user_pci_realize() as well as MAP/DEMAP messages from guest address space change transactions. It is also called by the migration thread, which is a separate thread that does not run holding BQL.

Thanks for mentioning those additional cases.

Stefan

#21 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 81 lines · Stefan Hajnoczi
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 24 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

if it's a reply
reply = find_reply(...)
qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else

Quoted 8 lines
   QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
   if (pending_reqs_was_empty) {
       qemu_coroutine_enter(process_request_co);
   }

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

Quoted 4 lines

How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller

A vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.

For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:

Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);

Quoted 6 lines

BDRV_POLL_WHILE() boils down to a loop like this:

 while ((cond)) {
   aio_poll(ctx, true);

}

I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.

Quoted 4 lines

Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.

Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:

Quoted 9 lines

if it's a reply

     reply = find_reply(...)
     qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
 else
     QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
     if (pending_reqs_was_empty) {
         qemu_coroutine_enter(process_request_co);
     }

Sorry, I was assuming when you said the coroutine will block with
aio_poll(), you implied it would also read messages from the socket.

Quoted 5 lines

The qemu_coroutine_enter(reply->co) call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signalling reply->cv in the existing patch.

Yes, the only difference is it would be woken on each message,
even though it doesn’t read them. Which is what I think you’re addressing below.

Quoted 6 lines

Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call aio_poll() (which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.

Wouldn’t you need to acquire BQL twice for every vCPU reply: once to
run the receiver coroutine, and once when the vCPU thread wakes up and wants to return to the VFIO code. The migration thread would also add a BQL dependency, where it didn’t have one before.

Is your objection with using an iothread, or using a separate thread?
I can change to using qemu_thread_create() and running aio_poll() from the thread routine, instead of creating an iothread.

On a related subject:

Quoted 7 lines · Stefan Hajnoczi

Review

Quoted 2 lines
(quoted diff) +2 -0
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);

This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).

This shouldn’t block if the emulation process sends less than sizeof(msg)
bytes. qio_channel_readv() will eventually call recvmsg(), which only blocks a short read if MSG_WAITALL is set, and it’s not set in this case. recvmsg() will return the data available, and vfio_user_recv() will treat a short read as an error.

JJ

#22 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 91 lines · John Johnson
Quoted 81 lines · Stefan Hajnoczi
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 24 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

if it's a reply
reply = find_reply(...)
qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else

Quoted 8 lines
   QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
   if (pending_reqs_was_empty) {
       qemu_coroutine_enter(process_request_co);
   }

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

Quoted 4 lines

How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller

A vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.

For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:

Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);

Quoted 6 lines

BDRV_POLL_WHILE() boils down to a loop like this:

 while ((cond)) {
   aio_poll(ctx, true);

}

I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.

Quoted 4 lines

Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.

Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:

Quoted 9 lines

if it's a reply

     reply = find_reply(...)
     qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
 else
     QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
     if (pending_reqs_was_empty) {
         qemu_coroutine_enter(process_request_co);
     }
Quoted 2 lines

Sorry, I was assuming when you said the coroutine will block with
aio_poll(), you implied it would also read messages from the socket.

The vCPU thread calls aio_poll() outside the coroutine, similar to the block/block-backend.c:blk_prw() example I posted above:

  Coroutine *co = qemu_coroutine_create(co_entry, &rwco);
  bdrv_coroutine_enter(blk_bs(blk), co);
  BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);

(BDRV_POLL_WHILE() is a aio_poll() loop.)

The coroutine isn't aware of aio_poll(), it just yields when it needs to wait.

Quoted 17 lines
Quoted 5 lines

The qemu_coroutine_enter(reply->co) call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signalling reply->cv in the existing patch.

Yes, the only difference is it would be woken on each message,
even though it doesn’t read them. Which is what I think you’re addressing below.

Quoted 6 lines

Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call aio_poll() (which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.

Quoted 4 lines

Wouldn’t you need to acquire BQL twice for every vCPU reply: once to
run the receiver coroutine, and once when the vCPU thread wakes up and wants to return to the VFIO code. The migration thread would also add a BQL dependency, where it didn’t have one before.

If aio_poll() is used then the vCPU thread doesn't drop the BQL at all. The vCPU thread sends the message and waits for the reply while other BQL threads are locked out.

If a condvar or similar mechanism is used then the vCPU sends the message, drops the BQL, and waits on the condvar. The main loop thread runs the receiver coroutine and re-enters the coroutine, which signals the condvar. The vCPU then re-acquires the BQL.

Quoted 3 lines

Is your objection with using an iothread, or using a separate thread?
I can change to using qemu_thread_create() and running aio_poll() from the thread routine, instead of creating an iothread.

The vfio-user communication code shouldn't need to worry about threads or locks. The code can be written in terms of AioContext so the caller can use it from various environments without hardcoding details of the BQL or threads into the communication code. This makes it easier to understand and less tightly coupled.

I'll try to sketch how that could work:

The main concept is VFIOProxy, which has a QIOChannel (the socket connection) and its main API is:

void coroutine_fn vfio_user_co_send_recv(VFIOProxy *proxy,

          VFIOUserHdr *msg, VFIOUserFDs *fds, int rsize, int flags);

There is also a request callback for processing incoming requests:

void coroutine_fn (*request)(void *opaque, char *buf,

                              VFIOUserFDs *fds);

The main loop thread can either use vfio_user_co_send_recv() from coroutine context or use this blocking wrapper:

  typedef struct {
      VFIOProxy *proxy;
      VFIOUserHdr *msg;
      VFIOUserFDs *fds;
      int rsize;
      int flags;
      bool done;
  } VFIOUserSendRecvData;

  static void coroutine_fn vfu_send_recv_co(void *opaque)
  {
      VFIOUserSendRecvData *data = opaque;
      vfio_user_co_send_recv(data->proxy, data->msg, data->fds,
                             data->rsize, data->flags);
      data->done = true;
  }

/* A blocking version of vfio_user_co_send_recv() */ void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags)

  {
      VFIOUserSendRecvData data = {
          .proxy = proxy,
	  .msg = msg,
	  .fds = fds,
	  .rsize = rsize,
	  .flags = flags,
      };
      Coroutine *co = qemu_coroutine_create(vfu_send_recv_co, &data);
      qemu_coroutine_enter(co);
      while (!data.done) {
          aio_poll(proxy->ioc->ctx, true);
      }

}

The vCPU thread can use vfio_user_send_recv() if it wants, although the BQL will be held, preventing other threads from making progress. That can be avoided by writing a similar wrapper that uses a QemuSemaphore:

  typedef struct {
      VFIOProxy *proxy;
      VFIOUserHdr *msg;
      VFIOUserFDs *fds;
      int rsize;
      int flags;
      QemuSemaphore sem;
  } VFIOUserSendRecvData;

  static void coroutine_fn vfu_send_recv_co(void *opaque)
  {
      VFIOUserSendRecvData *data = opaque;
      vfio_user_co_send_recv(data->proxy, data->msg, data->fds,
                             data->rsize, data->flags);
      qemu_sem_post(&data->sem);
  }

/*
* A blocking version of vfio_user_co_send_recv() that relies on
* another thread to run the event loop. This can be used from vCPU
* threads to avoid hogging the BQL.
*/
void vfio_user_vcpu_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags)

  {
      VFIOUserSendRecvData data = {
          .proxy = proxy,
	  .msg = msg,
	  .fds = fds,
	  .rsize = rsize,
	  .flags = flags,
      };
      Coroutine *co = qemu_coroutine_create(vfu_vcpu_send_recv_co, &data);

      qemu_sem_init(&data.sem, 0);

      qemu_coroutine_enter(co);

      qemu_mutex_unlock_iothread();
      qemu_sem_wait(&data.sem);
      qemu_mutex_lock_iothread();

      qemu_sem_destroy(&data.sem);

}

With vfio_user_vcpu_send_recv() the vCPU thread doesn't call aio_poll() itself but instead relies on the main loop thread to run the event loop.

By writing coroutines that run in proxy->ioc->ctx we keep the threading model and locking in the caller. The communication code isn't aware of or tied to specific threads. It's possible to drop proxy->lock because state is only changed from within the AioContext, not multiple threads that may run in parallel. I think this makes the communication code simpler and cleaner.

It's possible to use IOThreads with this approach: set the QIOChannel's AioContext to the IOThread AioContext. However, I don't think we can do this in the vhost-user server yet because QEMU's device models expect to run with the BQL and not in an IOThread.

I didn't go into detail about how vfio_user_co_send_recv() is implemented. Please let me know if you want me to share ideas about that, but it's what we've already discussed with a "receiver" coroutine that re-enters the reply coroutines or calls ->request(). A CoMutex is needed to around qio_channel_write_all() to ensure that coroutines sending messages don't interleave partial writes if the socket sndbuf is exhausted.

Quoted 13 lines

On a related subject:

Quoted 7 lines · Stefan Hajnoczi

Review

Quoted 2 lines
(quoted diff) +2 -0
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);

This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).

Quoted 4 lines

This shouldn’t block if the emulation process sends less than sizeof(msg)
bytes. qio_channel_readv() will eventually call recvmsg(), which only blocks a short read if MSG_WAITALL is set, and it’s not set in this case. recvmsg() will return the data available, and vfio_user_recv() will treat a short read as an error.

That's true but vfio_user_recv() can still block layer on: if only sizeof(msg) bytes are available and msg.size > sizeof(msg) then the second call blocks.

  msgleft = msg.size - sizeof(msg);
  if (msgleft != 0) {
      ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err);

I think either code should be non-blocking or it shouldn't be. Writing code that is partially non-blocking is asking for trouble because it's not obvious where it can block and misbehaving or malicious programs can cause it to block.

Stefan

#23 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 133 lines · Stefan Hajnoczi
Quoted 91 lines · John Johnson
Quoted 81 lines · Stefan Hajnoczi
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 25 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

if it's a reply
reply = find_reply(...)
qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else
QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);

Quoted 7 lines
  if (pending_reqs_was_empty) {
      qemu_coroutine_enter(process_request_co);

}

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

Quoted 4 lines

How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller

A vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.

For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:

Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);

Quoted 6 lines

BDRV_POLL_WHILE() boils down to a loop like this:

while ((cond)) {
aio_poll(ctx, true);
}

I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.

Quoted 4 lines

Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.

Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:

Quoted 9 lines

if it's a reply

    reply = find_reply(...)
    qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else
    QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
    if (pending_reqs_was_empty) {
        qemu_coroutine_enter(process_request_co);
    }
Quoted 2 lines

Sorry, I was assuming when you said the coroutine will block with
aio_poll(), you implied it would also read messages from the socket.

The vCPU thread calls aio_poll() outside the coroutine, similar to the block/block-backend.c:blk_prw() example I posted above:

Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);

(BDRV_POLL_WHILE() is a aio_poll() loop.)

The coroutine isn't aware of aio_poll(), it just yields when it needs to wait.

Quoted 17 lines
Quoted 5 lines

The qemu_coroutine_enter(reply->co) call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signalling reply->cv in the existing patch.

Yes, the only difference is it would be woken on each message,
even though it doesn’t read them. Which is what I think you’re addressing below.

Quoted 6 lines

Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call aio_poll() (which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.

Quoted 4 lines

Wouldn’t you need to acquire BQL twice for every vCPU reply: once to
run the receiver coroutine, and once when the vCPU thread wakes up and wants to return to the VFIO code. The migration thread would also add a BQL dependency, where it didn’t have one before.

Quoted 9 lines

If aio_poll() is used then the vCPU thread doesn't drop the BQL at all. The vCPU thread sends the message and waits for the reply while other BQL threads are locked out.

If a condvar or similar mechanism is used then the vCPU sends the message, drops the BQL, and waits on the condvar. The main loop thread runs the receiver coroutine and re-enters the coroutine, which signals the condvar. The vCPU then re-acquires the BQL.

I understand this. The point I was trying to make was you'd need
to acquire BQL twice for every reply: once by the main loop before it runs the receiver coroutine and once after the vCPU wakes up. That would seem to increase latency over the iothread model.

Quoted 108 lines
Quoted 3 lines

Is your objection with using an iothread, or using a separate thread?
I can change to using qemu_thread_create() and running aio_poll() from the thread routine, instead of creating an iothread.

The vfio-user communication code shouldn't need to worry about threads or locks. The code can be written in terms of AioContext so the caller can use it from various environments without hardcoding details of the BQL or threads into the communication code. This makes it easier to understand and less tightly coupled.

I'll try to sketch how that could work:

The main concept is VFIOProxy, which has a QIOChannel (the socket connection) and its main API is:

void coroutine_fn vfio_user_co_send_recv(VFIOProxy *proxy,

         VFIOUserHdr *msg, VFIOUserFDs *fds, int rsize, int flags);

There is also a request callback for processing incoming requests:

void coroutine_fn (*request)(void *opaque, char *buf,

                             VFIOUserFDs *fds);

The main loop thread can either use vfio_user_co_send_recv() from coroutine context or use this blocking wrapper:

 typedef struct {
     VFIOProxy *proxy;
     VFIOUserHdr *msg;
     VFIOUserFDs *fds;
     int rsize;
     int flags;
     bool done;

} VFIOUserSendRecvData;

static void coroutine_fn vfu_send_recv_co(void *opaque)

 {
     VFIOUserSendRecvData *data = opaque;
     vfio_user_co_send_recv(data->proxy, data->msg, data->fds,
                            data->rsize, data->flags);
     data->done = true;

}

/* A blocking version of vfio_user_co_send_recv() */ void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags)

 {
     VFIOUserSendRecvData data = {
         .proxy = proxy,
	  .msg = msg,
	  .fds = fds,
	  .rsize = rsize,
	  .flags = flags,
     };
     Coroutine *co = qemu_coroutine_create(vfu_send_recv_co, &data);
     qemu_coroutine_enter(co);
     while (!data.done) {
         aio_poll(proxy->ioc->ctx, true);
     }

}

The vCPU thread can use vfio_user_send_recv() if it wants, although the BQL will be held, preventing other threads from making progress. That can be avoided by writing a similar wrapper that uses a QemuSemaphore:

 typedef struct {
     VFIOProxy *proxy;
     VFIOUserHdr *msg;
     VFIOUserFDs *fds;
     int rsize;
     int flags;
     QemuSemaphore sem;

} VFIOUserSendRecvData;

static void coroutine_fn vfu_send_recv_co(void *opaque)

 {
     VFIOUserSendRecvData *data = opaque;
     vfio_user_co_send_recv(data->proxy, data->msg, data->fds,
                            data->rsize, data->flags);
     qemu_sem_post(&data->sem);

}

/*
* A blocking version of vfio_user_co_send_recv() that relies on
* another thread to run the event loop. This can be used from vCPU
* threads to avoid hogging the BQL.
*/
void vfio_user_vcpu_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags)

 {
     VFIOUserSendRecvData data = {
         .proxy = proxy,
	  .msg = msg,
	  .fds = fds,
	  .rsize = rsize,
	  .flags = flags,
     };
     Coroutine *co = qemu_coroutine_create(vfu_vcpu_send_recv_co, &data);

     qemu_sem_init(&data.sem, 0);

     qemu_coroutine_enter(co);

     qemu_mutex_unlock_iothread();
     qemu_sem_wait(&data.sem);
     qemu_mutex_lock_iothread();

     qemu_sem_destroy(&data.sem);
Quoted 5 lines

}

With vfio_user_vcpu_send_recv() the vCPU thread doesn't call aio_poll() itself but instead relies on the main loop thread to run the event loop.

I think this means I need 2 send algorithms: one for when called
from the main loop, and another for when called outside the main loop (vCPU or migration). I can’t use the semaphore version from the main loop, since blocking the main loop would prevent the receiver routine from being scheduled, so I’d want to use aio_poll() there.

Some vfio_user calls can come from either place (e.g., realize
uses REGION_READ to read the device config space, and vCPU uses it on a guest load to the device), so I’d need to detect which thread I’m running in to choose the right sender.

Quoted 12 lines

By writing coroutines that run in proxy->ioc->ctx we keep the threading model and locking in the caller. The communication code isn't aware of or tied to specific threads. It's possible to drop proxy->lock because state is only changed from within the AioContext, not multiple threads that may run in parallel. I think this makes the communication code simpler and cleaner.

It's possible to use IOThreads with this approach: set the QIOChannel's AioContext to the IOThread AioContext. However, I don't think we can do this in the vhost-user server yet because QEMU's device models expect to run with the BQL and not in an IOThread.

Quoted 8 lines

I didn't go into detail about how vfio_user_co_send_recv() is implemented. Please let me know if you want me to share ideas about that, but it's what we've already discussed with a "receiver" coroutine that re-enters the reply coroutines or calls ->request(). A CoMutex is needed to around qio_channel_write_all() to ensure that coroutines sending messages don't interleave partial writes if the socket sndbuf is exhausted.

Here is where I questioned how coroutines work across threads.
When the reply waiter is not the main loop, would the receiver coroutine re-enter the reply coroutine or signal the condvar it is waiting on?

Quoted 22 lines
Quoted 13 lines

On a related subject:

Quoted 7 lines · Stefan Hajnoczi

Review

Quoted 2 lines
(quoted diff) +2 -0
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);

This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).

Quoted 4 lines

This shouldn’t block if the emulation process sends less than sizeof(msg)
bytes. qio_channel_readv() will eventually call recvmsg(), which only blocks a short read if MSG_WAITALL is set, and it’s not set in this case. recvmsg() will return the data available, and vfio_user_recv() will treat a short read as an error.

That's true but vfio_user_recv() can still block layer on: if only sizeof(msg) bytes are available and msg.size > sizeof(msg) then the second call blocks.

Quoted 9 lines

msgleft = msg.size - sizeof(msg);

 if (msgleft != 0) {
     ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err);

I think either code should be non-blocking or it shouldn't be. Writing code that is partially non-blocking is asking for trouble because it's not obvious where it can block and misbehaving or malicious programs can cause it to block.

I wonder if I should just go fully non-blocking, and have the
senders queue messages for the sending routine, and have the receiving routine either signal a reply waiter or schedule a request handling routine.

JJ

#24 lore raw

Re: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server

Quoted 143 lines · John Johnson
Quoted 133 lines · Stefan Hajnoczi
Quoted 91 lines · John Johnson
Quoted 81 lines · Stefan Hajnoczi
Quoted 68 lines · John Johnson
Quoted 54 lines
Quoted 35 lines · John Johnson
Quoted 25 lines · Stefan Hajnoczi
Quoted 3 lines · John Johnson

I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving asynchronously to the sends.

This can be done with a receiver coroutine. Its job is to be the only thing that reads vfio-user messages from the socket. A receiver coroutine reads messages from the socket and wakes up the waiting coroutine that yielded from vfio_user_send_recv() or vfio_user_pci_process_req().

(Although vfio_user_pci_process_req() could be called directly from the receiver coroutine, it seems safer to have a separate coroutine that processes requests so that the receiver isn't blocked in case vfio_user_pci_process_req() yields while processing a request.)

Going back to what you mentioned above, the receiver coroutine does something like this:

if it's a reply
reply = find_reply(...)
qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else
QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);

Quoted 7 lines
  if (pending_reqs_was_empty) {
      qemu_coroutine_enter(process_request_co);

}

The pending_reqs queue holds incoming requests that the process_request_co coroutine processes.

Quoted 4 lines

How do coroutines work across threads? There can be multiple vCPU
threads waiting for replies, and I think the receiver coroutine will be running in the main loop thread. Where would a vCPU block waiting for a reply? I think coroutine_yield() returns to its coroutine_enter() caller

A vCPU thread holding the BQL can iterate the event loop if it has reached a synchronous point that needs to wait for a reply before returning. I think we have this situation when a MemoryRegion is accessed on the proxy device.

For example, block/block-backend.c:blk_prw() kicks off a coroutine and then runs the event loop until the coroutine finishes:

Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);

Quoted 6 lines

BDRV_POLL_WHILE() boils down to a loop like this:

while ((cond)) {
aio_poll(ctx, true);
}

I think that would make vCPUs sending requests and the
receiver coroutine all poll on the same socket. If the “wrong” routine reads the message, I’d need a second level of synchronization to pass the message to the “right” one. e.g., if the vCPU coroutine reads a request, it needs to pass it to the receiver; if the receiver coroutine reads a reply, it needs to pass it to a vCPU.

Quoted 4 lines

Avoiding this complexity is one of the reasons I went with
a separate thread that only reads the socket over the mp-qemu model, which does have the sender poll, but doesn’t need to handle incoming requests.

Only one coroutine reads from the socket, the "receiver" coroutine. In a previous reply I sketched what the receiver does:

Quoted 9 lines

if it's a reply

    reply = find_reply(...)
    qemu_coroutine_enter(reply->co) // instead of signalling reply->cv
else
    QSIMPLEQ_INSERT_TAIL(&pending_reqs, request, next);
    if (pending_reqs_was_empty) {
        qemu_coroutine_enter(process_request_co);
    }
Quoted 2 lines

Sorry, I was assuming when you said the coroutine will block with
aio_poll(), you implied it would also read messages from the socket.

The vCPU thread calls aio_poll() outside the coroutine, similar to the block/block-backend.c:blk_prw() example I posted above:

Coroutine *co = qemu_coroutine_create(co_entry, &rwco); bdrv_coroutine_enter(blk_bs(blk), co); BDRV_POLL_WHILE(blk_bs(blk), rwco.ret == NOT_DONE);

(BDRV_POLL_WHILE() is a aio_poll() loop.)

The coroutine isn't aware of aio_poll(), it just yields when it needs to wait.

Quoted 17 lines
Quoted 5 lines

The qemu_coroutine_enter(reply->co) call re-enters the coroutine that was created by the vCPU thread. Is this the "second level of synchronization" that you described? It's very similar to signalling reply->cv in the existing patch.

Yes, the only difference is it would be woken on each message,
even though it doesn’t read them. Which is what I think you’re addressing below.

Quoted 6 lines

Now I'm actually thinking about whether this can be improved by keeping the condvar so that the vCPU thread doesn't need to call aio_poll() (which is awkward because it doesn't drop the BQL and therefore blocks other vCPUs from making progress). That approach wouldn't require a dedicated thread for vfio-user.

Quoted 4 lines

Wouldn’t you need to acquire BQL twice for every vCPU reply: once to
run the receiver coroutine, and once when the vCPU thread wakes up and wants to return to the VFIO code. The migration thread would also add a BQL dependency, where it didn’t have one before.

Quoted 9 lines

If aio_poll() is used then the vCPU thread doesn't drop the BQL at all. The vCPU thread sends the message and waits for the reply while other BQL threads are locked out.

If a condvar or similar mechanism is used then the vCPU sends the message, drops the BQL, and waits on the condvar. The main loop thread runs the receiver coroutine and re-enters the coroutine, which signals the condvar. The vCPU then re-acquires the BQL.

Quoted 4 lines

I understand this. The point I was trying to make was you'd need
to acquire BQL twice for every reply: once by the main loop before it runs the receiver coroutine and once after the vCPU wakes up. That would seem to increase latency over the iothread model.

Yes, but if minimizing latency is critical then you can use the aio_poll() approach. It's fastest since it doesn't context switch or drop the BQL.

Regarding vfio-user performance in general, devices should use ioeventfd and/or mmap regions to avoid going through the VMM in performance-critical code paths.

Quoted 120 lines
Quoted 108 lines
Quoted 3 lines

Is your objection with using an iothread, or using a separate thread?
I can change to using qemu_thread_create() and running aio_poll() from the thread routine, instead of creating an iothread.

The vfio-user communication code shouldn't need to worry about threads or locks. The code can be written in terms of AioContext so the caller can use it from various environments without hardcoding details of the BQL or threads into the communication code. This makes it easier to understand and less tightly coupled.

I'll try to sketch how that could work:

The main concept is VFIOProxy, which has a QIOChannel (the socket connection) and its main API is:

void coroutine_fn vfio_user_co_send_recv(VFIOProxy *proxy,

         VFIOUserHdr *msg, VFIOUserFDs *fds, int rsize, int flags);

There is also a request callback for processing incoming requests:

void coroutine_fn (*request)(void *opaque, char *buf,

                             VFIOUserFDs *fds);

The main loop thread can either use vfio_user_co_send_recv() from coroutine context or use this blocking wrapper:

 typedef struct {
     VFIOProxy *proxy;
     VFIOUserHdr *msg;
     VFIOUserFDs *fds;
     int rsize;
     int flags;
     bool done;

} VFIOUserSendRecvData;

static void coroutine_fn vfu_send_recv_co(void *opaque)

 {
     VFIOUserSendRecvData *data = opaque;
     vfio_user_co_send_recv(data->proxy, data->msg, data->fds,
                            data->rsize, data->flags);
     data->done = true;

}

/* A blocking version of vfio_user_co_send_recv() */ void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags)

 {
     VFIOUserSendRecvData data = {
         .proxy = proxy,
	  .msg = msg,
	  .fds = fds,
	  .rsize = rsize,
	  .flags = flags,
     };
     Coroutine *co = qemu_coroutine_create(vfu_send_recv_co, &data);
     qemu_coroutine_enter(co);
     while (!data.done) {
         aio_poll(proxy->ioc->ctx, true);
     }

}

The vCPU thread can use vfio_user_send_recv() if it wants, although the BQL will be held, preventing other threads from making progress. That can be avoided by writing a similar wrapper that uses a QemuSemaphore:

 typedef struct {
     VFIOProxy *proxy;
     VFIOUserHdr *msg;
     VFIOUserFDs *fds;
     int rsize;
     int flags;
     QemuSemaphore sem;

} VFIOUserSendRecvData;

static void coroutine_fn vfu_send_recv_co(void *opaque)

 {
     VFIOUserSendRecvData *data = opaque;
     vfio_user_co_send_recv(data->proxy, data->msg, data->fds,
                            data->rsize, data->flags);
     qemu_sem_post(&data->sem);

}

/*
* A blocking version of vfio_user_co_send_recv() that relies on
* another thread to run the event loop. This can be used from vCPU
* threads to avoid hogging the BQL.
*/
void vfio_user_vcpu_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
VFIOUserFDs *fds, int rsize, int flags)

 {
     VFIOUserSendRecvData data = {
         .proxy = proxy,
	  .msg = msg,
	  .fds = fds,
	  .rsize = rsize,
	  .flags = flags,
     };
     Coroutine *co = qemu_coroutine_create(vfu_vcpu_send_recv_co, &data);

     qemu_sem_init(&data.sem, 0);

     qemu_coroutine_enter(co);

     qemu_mutex_unlock_iothread();
     qemu_sem_wait(&data.sem);
     qemu_mutex_lock_iothread();

     qemu_sem_destroy(&data.sem);
Quoted 5 lines

}

With vfio_user_vcpu_send_recv() the vCPU thread doesn't call aio_poll() itself but instead relies on the main loop thread to run the event loop.

I think this means I need 2 send algorithms: one for when called
from the main loop, and another for when called outside the main loop (vCPU or migration). I can’t use the semaphore version from the main loop, since blocking the main loop would prevent the receiver routine from being scheduled, so I’d want to use aio_poll() there.

Quoted 4 lines

Some vfio_user calls can come from either place (e.g., realize
uses REGION_READ to read the device config space, and vCPU uses it on a guest load to the device), so I’d need to detect which thread I’m running in to choose the right sender.

The semaphore version is not really necessary, although it allows other BQL threads to make progress while we wait for a reply (but is slower, as you pointed out).

The aio_poll() approach can be used from either thread.

Quoted 21 lines
Quoted 12 lines

By writing coroutines that run in proxy->ioc->ctx we keep the threading model and locking in the caller. The communication code isn't aware of or tied to specific threads. It's possible to drop proxy->lock because state is only changed from within the AioContext, not multiple threads that may run in parallel. I think this makes the communication code simpler and cleaner.

It's possible to use IOThreads with this approach: set the QIOChannel's AioContext to the IOThread AioContext. However, I don't think we can do this in the vhost-user server yet because QEMU's device models expect to run with the BQL and not in an IOThread.

Quoted 8 lines

I didn't go into detail about how vfio_user_co_send_recv() is implemented. Please let me know if you want me to share ideas about that, but it's what we've already discussed with a "receiver" coroutine that re-enters the reply coroutines or calls ->request(). A CoMutex is needed to around qio_channel_write_all() to ensure that coroutines sending messages don't interleave partial writes if the socket sndbuf is exhausted.

Quoted 3 lines

Here is where I questioned how coroutines work across threads.
When the reply waiter is not the main loop, would the receiver coroutine re-enter the reply coroutine or signal the condvar it is waiting on?

The global AioContext (qemu_get_aio_context()) is shared by multiple thread and mutual exclusion is provided by the BQL. Any of the threads can run the coroutines and during a coroutine's lifetime it can run in multiple threads (but never simultaneously in multiple threads).

The vfu_send_recv_co() coroutine above starts in the vCPU thread but then yields and is re-entered from the main loop thread. The receiver coroutine re-enters vfu_send_recv_co() in the main loop thread where it calls qemu_sem_post() to wake up the vCPU thread.

(IOThread AioContexts are not shared by multiple threads, so in that case we don't need to worry about threads since everything is done in the IOThread.)

Quoted 32 lines
Quoted 22 lines
Quoted 13 lines

On a related subject:

Quoted 7 lines · Stefan Hajnoczi

Review

Quoted 2 lines
(quoted diff) +2 -0
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);

This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).

Quoted 4 lines

This shouldn’t block if the emulation process sends less than sizeof(msg)
bytes. qio_channel_readv() will eventually call recvmsg(), which only blocks a short read if MSG_WAITALL is set, and it’s not set in this case. recvmsg() will return the data available, and vfio_user_recv() will treat a short read as an error.

That's true but vfio_user_recv() can still block layer on: if only sizeof(msg) bytes are available and msg.size > sizeof(msg) then the second call blocks.

Quoted 9 lines

msgleft = msg.size - sizeof(msg);

 if (msgleft != 0) {
     ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err);

I think either code should be non-blocking or it shouldn't be. Writing code that is partially non-blocking is asking for trouble because it's not obvious where it can block and misbehaving or malicious programs can cause it to block.

Quoted 4 lines

I wonder if I should just go fully non-blocking, and have the
senders queue messages for the sending routine, and have the receiving routine either signal a reply waiter or schedule a request handling routine.

That sounds good.

If messages are sent from coroutine context rather than plain functions then a separate sender isn't needed, they can use a CoMutex for mutual exclusion/queuing instead of an explicit send queue and sender routine.

Stefan

#25 lore raw

RFC PATCH v2 5/16 vfio-user: define VFIO Proxy and communication functions

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h |  62 +++++++++
 hw/vfio/user.h          |   8 ++
 hw/vfio/pci.c           |   6 +
 hw/vfio/user.c          | 289 ++++++++++++++++++++++++++++++++++++++++
 MAINTAINERS             |   1 +
 5 files changed, 366 insertions(+)
 create mode 100644 hw/vfio/user-protocol.h
hw/vfio/user-protocol.h new file +62 -0
new file mode 100644
index 0000000000..27062cb910
@@ -0,0 +1,62 @@
+#ifndef VFIO_USER_PROTOCOL_H
+#define VFIO_USER_PROTOCOL_H
+
+/*
+ * vfio protocol over a UNIX socket.
+ *
+ * Copyright © 2018, 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Each message has a standard header that describes the command
+ * being sent, which is almost always a VFIO ioctl().
+ *
+ * The header may be followed by command-specific data, such as the
+ * region and offset info for read and write commands.
+ */
+
+typedef struct {
+    uint16_t id;
+    uint16_t command;
+    uint32_t size;
+    uint32_t flags;
+    uint32_t error_reply;
+} VFIOUserHdr;
+
+/* VFIOUserHdr commands */
+enum vfio_user_command {
+    VFIO_USER_VERSION                   = 1,
+    VFIO_USER_DMA_MAP                   = 2,
+    VFIO_USER_DMA_UNMAP                 = 3,
+    VFIO_USER_DEVICE_GET_INFO           = 4,
+    VFIO_USER_DEVICE_GET_REGION_INFO    = 5,
+    VFIO_USER_DEVICE_GET_REGION_IO_FDS  = 6,
+    VFIO_USER_DEVICE_GET_IRQ_INFO       = 7,
+    VFIO_USER_DEVICE_SET_IRQS           = 8,
+    VFIO_USER_REGION_READ               = 9,
+    VFIO_USER_REGION_WRITE              = 10,
+    VFIO_USER_DMA_READ                  = 11,
+    VFIO_USER_DMA_WRITE                 = 12,
+    VFIO_USER_DEVICE_RESET              = 13,
+    VFIO_USER_DIRTY_PAGES               = 14,
+    VFIO_USER_MAX,
+};
+
+/* VFIOUserHdr flags */
+#define VFIO_USER_REQUEST       0x0
+#define VFIO_USER_REPLY         0x1
+#define VFIO_USER_TYPE          0xF
+
+#define VFIO_USER_NO_REPLY      0x10
+#define VFIO_USER_ERROR         0x20
+
+
+#define VFIO_USER_DEF_MAX_FDS   8
+#define VFIO_USER_MAX_MAX_FDS   16
+
+#define VFIO_USER_DEF_MAX_XFER  (1024 * 1024)
+#define VFIO_USER_MAX_MAX_XFER  (64 * 1024 * 1024)
+
+
+#endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +8 -0
index 62b2d03d56..905e374e12 100644
@@ -11,6 +11,8 @@
  *
  */
 
+#include "user-protocol.h"
+
 typedef struct {
     int send_fds;
     int recv_fds;
@@ -19,6 +21,7 @@ typedef struct {
 
 typedef struct VFIOUserReply {
     QTAILQ_ENTRY(VFIOUserReply) next;
+    VFIOUserHdr *msg;
     VFIOUserFDs *fds;
     uint32_t rsize;
     uint32_t id;
@@ -62,5 +65,10 @@ typedef struct VFIOProxy {
 
 VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp);
 void vfio_user_disconnect(VFIOProxy *proxy);
+void vfio_user_set_reqhandler(VFIODevice *vbasdev,
+                              int (*handler)(void *opaque, char *buf,
+                                             VFIOUserFDs *fds),
+                                             void *reqarg);
+void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +6 -0
index 7c2d245ca5..7005d9f891 100644
@@ -3333,6 +3333,11 @@ type_init(register_vfio_pci_dev_type)
  * vfio-user routines.
  */
 
+static int vfio_user_pci_process_req(void *opaque, char *buf, VFIOUserFDs *fds)
+{
+    return 0;
+}
+
 /*
  * Emulated devices don't use host hot reset
  */
@@ -3386,6 +3391,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         return;
     }
     vbasedev->proxy = proxy;
+    vfio_user_set_reqhandler(vbasedev, vfio_user_pci_process_req, vdev);
 
     if (udev->secure_dma) {
         proxy->flags |= VFIO_PROXY_SECURE;
hw/vfio/user.c +289 -0

315 lines not on this page · raw message

MAINTAINERS +1 -0
index f429bab391..52d37dd088 100644
@@ -1888,6 +1888,7 @@ S: Supported
 F: docs/devel/vfio-user.rst
 F: hw/vfio/user.c
 F: hw/vfio/user.h
+F: hw/vfio/user-protocol.h
 
 vhost
 M: Michael S. Tsirkin <[email protected]>
Signature · 1 line
2.25.1
#26 lore raw

Re: [PATCH RFC v2 05/16] vfio-user: define VFIO Proxy and communication functions

hw/vfio/user.h typedef struct VFIOProxy { · line 68

Quoted 5 lines · Elena Ufimtseva
hw/vfio/user.h +1 -0
@@ -62,5 +65,10 @@ typedef struct VFIOProxy {
 
 VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp);
 void vfio_user_disconnect(VFIOProxy *proxy);
+void vfio_user_set_reqhandler(VFIODevice *vbasdev,

"vbasedev" for consistency?

hw/vfio/user.h typedef struct VFIOProxy { · line 71

Quoted 3 lines
hw/vfio/user.h +3 -0
+                              int (*handler)(void *opaque, char *buf,
+                                             VFIOUserFDs *fds),
+                                             void *reqarg);

The handler callback is undocumented. What context does it run in, what do the arguments mean, and what should the function return? Please document it so it's easy for others to modify this code in the future without reverse-engineering the assumptions behind it.

Quoted 17 lines
hw/vfio/user.c +17 -0
+void vfio_user_recv(void *opaque)
+{
+    VFIODevice *vbasedev = opaque;
+    VFIOProxy *proxy = vbasedev->proxy;
+    VFIOUserReply *reply = NULL;
+    g_autofree int *fdp = NULL;
+    VFIOUserFDs reqfds = { 0, 0, fdp };
+    VFIOUserHdr msg;
+    struct iovec iov = {
+        .iov_base = &msg,
+        .iov_len = sizeof(msg),
+    };
+    bool isreply;
+    int i, ret;
+    size_t msgleft, numfds = 0;
+    char *data = NULL;
+    g_autofree char *buf = NULL;

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 162

Quoted 10 lines
hw/vfio/user.c +10 -0
+    Error *local_err = NULL;
+
+    qemu_mutex_lock(&proxy->lock);
+    if (proxy->state == VFIO_PROXY_CLOSING) {
+        qemu_mutex_unlock(&proxy->lock);
+        return;
+    }
+
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);

This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).

While we're blocked in this function the proxy device cannot be hot-removed since proxy->lock is held.

It would more robust to use of the event loop to avoid blocking. There could be a per-connection receiver coroutine that calls qio_channel_readv_full_all_eof() (it yields the coroutine if reading would block).

Quoted 16 lines
hw/vfio/user.c +16 -0
+    /*
+     * Replies signal a waiter, requests get processed by vfio code
+     * that may assume the iothread lock is held.
+     */
+    if (isreply) {
+        reply->complete = 1;
+        if (!reply->nowait) {
+            qemu_cond_signal(&reply->cv);
+        } else {
+            if (msg.flags & VFIO_USER_ERROR) {
+                error_printf("vfio_user_rcv error reply on async request ");
+                error_printf("command %x error %s\n", msg.command,
+                             strerror(msg.error_reply));
+            }
+            /* just free it if no one is waiting */
+            reply->nowait = 0;

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 286

Quoted 10 lines
hw/vfio/user.c +10 -0
+            if (proxy->last_nowait == reply) {
+                proxy->last_nowait = NULL;
+            }
+            g_free(reply->msg);
+            QTAILQ_INSERT_HEAD(&proxy->free, reply, next);
+        }
+        qemu_mutex_unlock(&proxy->lock);
+    } else {
+        qemu_mutex_unlock(&proxy->lock);
+        qemu_mutex_lock_iothread();

The fact that proxy->request() runs with the BQL suggests that VFIO communication should take place in the main event loop thread instead of a separate IOThread.

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 293

Quoted 7 lines
hw/vfio/user.c +7 -0
+        /*
+         * make sure proxy wasn't closed while we waited
+         * checking state without holding the proxy lock is safe
+         * since it's only set to CLOSING when BQL is held
+         */
+        if (proxy->state != VFIO_PROXY_CLOSING) {
+            ret = proxy->request(proxy->reqarg, buf, &reqfds);

The request() callback in an earlier patch is a noop for the client implementation. Who frees passed fds?

Quoted 12 lines
hw/vfio/user.c +12 -0
+            if (ret < 0 && !(msg.flags & VFIO_USER_NO_REPLY)) {
+                vfio_user_send_reply(proxy, buf, ret);
+            }
+        }
+        qemu_mutex_unlock_iothread();
+    }
+    return;
+
+fatal:
+    vfio_user_shutdown(proxy);
+    proxy->state = VFIO_PROXY_RECV_ERROR;
+

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 315

Quoted 10 lines
hw/vfio/user.c +10 -0
+err:
+    for (i = 0; i < numfds; i++) {
+        close(fdp[i]);
+    }
+    if (reply != NULL) {
+        /* force an error to keep sending thread from hanging */
+        reply->msg->flags |= VFIO_USER_ERROR;
+        reply->msg->error_reply = EINVAL;
+        reply->complete = 1;
+        qemu_cond_signal(&reply->cv);

What about fd passing? The actual fds have been closed already in fdp[] but reply has a copy too.

What about the nowait case? If no one is waiting on reply->cv so this reply will be leaked?

Stefan

#27 lore raw

Re: [PATCH RFC v2 05/16] vfio-user: define VFIO Proxy and communication functions

Quoted 13 lines · Stefan Hajnoczi

hw/vfio/user.h typedef struct VFIOProxy { · line 68

Quoted 5 lines · Elena Ufimtseva
hw/vfio/user.h +1 -0
@@ -62,5 +65,10 @@ typedef struct VFIOProxy {
 
 VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp);
 void vfio_user_disconnect(VFIOProxy *proxy);
+void vfio_user_set_reqhandler(VFIODevice *vbasdev,

"vbasedev" for consistency?

OK

Quoted 9 lines

hw/vfio/user.h typedef struct VFIOProxy { · line 71

Quoted 3 lines
hw/vfio/user.h +3 -0
+                              int (*handler)(void *opaque, char *buf,
+                                             VFIOUserFDs *fds),
+                                             void *reqarg);

The handler callback is undocumented. What context does it run in, what do the arguments mean, and what should the function return? Please document it so it's easy for others to modify this code in the future without reverse-engineering the assumptions behind it.

OK

Quoted 33 lines
Quoted 17 lines
hw/vfio/user.c +17 -0
+void vfio_user_recv(void *opaque)
+{
+    VFIODevice *vbasedev = opaque;
+    VFIOProxy *proxy = vbasedev->proxy;
+    VFIOUserReply *reply = NULL;
+    g_autofree int *fdp = NULL;
+    VFIOUserFDs reqfds = { 0, 0, fdp };
+    VFIOUserHdr msg;
+    struct iovec iov = {
+        .iov_base = &msg,
+        .iov_len = sizeof(msg),
+    };
+    bool isreply;
+    int i, ret;
+    size_t msgleft, numfds = 0;
+    char *data = NULL;
+    g_autofree char *buf = NULL;

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 162

Quoted 10 lines
hw/vfio/user.c +10 -0
+    Error *local_err = NULL;
+
+    qemu_mutex_lock(&proxy->lock);
+    if (proxy->state == VFIO_PROXY_CLOSING) {
+        qemu_mutex_unlock(&proxy->lock);
+        return;
+    }
+
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);

This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).

Quoted 8 lines

While we're blocked in this function the proxy device cannot be hot-removed since proxy->lock is held.

It would more robust to use of the event loop to avoid blocking. There could be a per-connection receiver coroutine that calls qio_channel_readv_full_all_eof() (it yields the coroutine if reading would block).

I thought the main loop uses BQL, which I don’t need for most
message processing. The blocking behavior can be fixed with FIONREAD beforehand to detect a message with fewer bytes than in a header.

Quoted 27 lines
Quoted 16 lines
hw/vfio/user.c +16 -0
+    /*
+     * Replies signal a waiter, requests get processed by vfio code
+     * that may assume the iothread lock is held.
+     */
+    if (isreply) {
+        reply->complete = 1;
+        if (!reply->nowait) {
+            qemu_cond_signal(&reply->cv);
+        } else {
+            if (msg.flags & VFIO_USER_ERROR) {
+                error_printf("vfio_user_rcv error reply on async request ");
+                error_printf("command %x error %s\n", msg.command,
+                             strerror(msg.error_reply));
+            }
+            /* just free it if no one is waiting */
+            reply->nowait = 0;

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 286

Quoted 10 lines
hw/vfio/user.c +10 -0
+            if (proxy->last_nowait == reply) {
+                proxy->last_nowait = NULL;
+            }
+            g_free(reply->msg);
+            QTAILQ_INSERT_HEAD(&proxy->free, reply, next);
+        }
+        qemu_mutex_unlock(&proxy->lock);
+    } else {
+        qemu_mutex_unlock(&proxy->lock);
+        qemu_mutex_lock_iothread();
Quoted 4 lines

The fact that proxy->request() runs with the BQL suggests that VFIO communication should take place in the main event loop thread instead of a separate IOThread.

See the last reply. Using the main event loop optimizes the
least common case.

Quoted 11 lines

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 293

Quoted 7 lines
hw/vfio/user.c +7 -0
+        /*
+         * make sure proxy wasn't closed while we waited
+         * checking state without holding the proxy lock is safe
+         * since it's only set to CLOSING when BQL is held
+         */
+        if (proxy->state != VFIO_PROXY_CLOSING) {
+            ret = proxy->request(proxy->reqarg, buf, &reqfds);

The request() callback in an earlier patch is a noop for the client implementation. Who frees passed fds?

	Right now no server->client requests send fd’s, but I do need

a single point where they are consumed if an error is returned.

Quoted 17 lines
Quoted 17 lines
hw/vfio/user.c +17 -0
+            if (ret < 0 && !(msg.flags & VFIO_USER_NO_REPLY)) {
+                vfio_user_send_reply(proxy, buf, ret);
+            }
+        }
+        qemu_mutex_unlock_iothread();
+    }
+    return;
+
+fatal:
+    vfio_user_shutdown(proxy);
+    proxy->state = VFIO_PROXY_RECV_ERROR;
+
+err:
+    for (i = 0; i < numfds; i++) {
+        close(fdp[i]);
+    }
+    if (reply != NULL) {
Quoted 9 lines

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 315

Quoted 5 lines
hw/vfio/user.c +5 -0
+        /* force an error to keep sending thread from hanging */
+        reply->msg->flags |= VFIO_USER_ERROR;
+        reply->msg->error_reply = EINVAL;
+        reply->complete = 1;
+        qemu_cond_signal(&reply->cv);

What about fd passing? The actual fds have been closed already in fdp[] but reply has a copy too.

If the sender gets an error, it won’t be using the fd’s. I
can zero reply->fds to make this clearer.

Quoted 3 lines

What about the nowait case? If no one is waiting on reply->cv so this reply will be leaked?

This looks like a leak.

JJ

#28 lore raw

Re: [PATCH RFC v2 05/16] vfio-user: define VFIO Proxy and communication functions

Quoted 71 lines · John Johnson
Quoted 13 lines · Stefan Hajnoczi

hw/vfio/user.h typedef struct VFIOProxy { · line 68

Quoted 5 lines · Elena Ufimtseva
hw/vfio/user.h +1 -0
@@ -62,5 +65,10 @@ typedef struct VFIOProxy {
 
 VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp);
 void vfio_user_disconnect(VFIOProxy *proxy);
+void vfio_user_set_reqhandler(VFIODevice *vbasdev,

"vbasedev" for consistency?

OK

Quoted 9 lines

hw/vfio/user.h typedef struct VFIOProxy { · line 71

Quoted 3 lines
hw/vfio/user.h +3 -0
+                              int (*handler)(void *opaque, char *buf,
+                                             VFIOUserFDs *fds),
+                                             void *reqarg);

The handler callback is undocumented. What context does it run in, what do the arguments mean, and what should the function return? Please document it so it's easy for others to modify this code in the future without reverse-engineering the assumptions behind it.

OK

Quoted 33 lines
Quoted 17 lines
hw/vfio/user.c +17 -0
+void vfio_user_recv(void *opaque)
+{
+    VFIODevice *vbasedev = opaque;
+    VFIOProxy *proxy = vbasedev->proxy;
+    VFIOUserReply *reply = NULL;
+    g_autofree int *fdp = NULL;
+    VFIOUserFDs reqfds = { 0, 0, fdp };
+    VFIOUserHdr msg;
+    struct iovec iov = {
+        .iov_base = &msg,
+        .iov_len = sizeof(msg),
+    };
+    bool isreply;
+    int i, ret;
+    size_t msgleft, numfds = 0;
+    char *data = NULL;
+    g_autofree char *buf = NULL;

hw/vfio/user.c static void vfio_user_shutdown(VFIOProxy *proxy) · line 162

Quoted 10 lines
hw/vfio/user.c +10 -0
+    Error *local_err = NULL;
+
+    qemu_mutex_lock(&proxy->lock);
+    if (proxy->state == VFIO_PROXY_CLOSING) {
+        qemu_mutex_unlock(&proxy->lock);
+        return;
+    }
+
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);

This is a blocking call. My understanding is that the IOThread is shared by all vfio-user devices, so other devices will have to wait if one of them is acting up (e.g. the device emulation process sent less than sizeof(msg) bytes).

Quoted 8 lines

While we're blocked in this function the proxy device cannot be hot-removed since proxy->lock is held.

It would more robust to use of the event loop to avoid blocking. There could be a per-connection receiver coroutine that calls qio_channel_readv_full_all_eof() (it yields the coroutine if reading would block).

Quoted 3 lines

I thought the main loop uses BQL, which I don’t need for most
message processing. The blocking behavior can be fixed with FIONREAD beforehand to detect a message with fewer bytes than in a header.

It's I/O-bound work, exactly what the main loop was intended for.

I'm not sure the BQL can be avoided anyway:
- The vfio-user client runs under the BQL (a vCPU thread).
- The vfio-user server needs to hold the BQL since most QEMU device models assume they are running under the BQL.

The network communication code doesn't need to know about the BQL though. Event-driven code (code that runs in an AioContext) can rely on the fact that its callbacks only execute in the AioContext, i.e. in one thread at any given time.

The code probably doesn't need explicit BQL lock/unlock and can run safely in another IOThread if the user wishes (I would leave that up to the user, e.g. -device vfio-user-pci,iothread=iothread0, instead of creating a dedicated IOThread that is shared for all vfio-user communication). See nbd/server.c for an example of doing event-driven network I/O with coroutines.

Stefan

#29 lore raw

RFC PATCH v2 6/16 vfio-user: negotiate version with remote server

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h |  23 ++++
 hw/vfio/user.h          |   1 +
 hw/vfio/pci.c           |   9 ++
 hw/vfio/user.c          | 267 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 300 insertions(+)
hw/vfio/user-protocol.h +23 -0
index 27062cb910..14b762d1ad 100644
@@ -52,6 +52,29 @@ enum vfio_user_command {
 #define VFIO_USER_ERROR         0x20
 
 
+/*
+ * VFIO_USER_VERSION
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint16_t major;
+    uint16_t minor;
+    char capabilities[];
+} VFIOUserVersion;
+
+#define VFIO_USER_MAJOR_VER     0
+#define VFIO_USER_MINOR_VER     0
+
+#define VFIO_USER_CAP           "capabilities"
+
+/* "capabilities" members */
+#define VFIO_USER_CAP_MAX_FDS   "max_msg_fds"
+#define VFIO_USER_CAP_MAX_XFER  "max_data_xfer_size"
+#define VFIO_USER_CAP_MIGR      "migration"
+
+/* "migration" member */
+#define VFIO_USER_CAP_PGSIZE    "pgsize"
+
 #define VFIO_USER_DEF_MAX_FDS   8
 #define VFIO_USER_MAX_MAX_FDS   16
 
hw/vfio/user.h +1 -0
index 905e374e12..cab957ba7a 100644
@@ -70,5 +70,6 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev,
                                              VFIOUserFDs *fds),
                                              void *reqarg);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
+int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +9 -0
index 7005d9f891..eae33e746f 100644
@@ -3397,6 +3397,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         proxy->flags |= VFIO_PROXY_SECURE;
     }
 
+    vfio_user_validate_version(vbasedev, &err);
+    if (err != NULL) {
+        error_propagate(errp, err);
+        goto error;
+    }
+
     vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name);
     vbasedev->dev = DEVICE(vdev);
     vbasedev->fd = -1;
@@ -3404,6 +3410,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     vbasedev->no_mmap = false;
     vbasedev->ops = &vfio_user_pci_ops;
 
+error:
+    vfio_user_disconnect(proxy);
+    error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name);
 }
 
 static void vfio_user_instance_finalize(Object *obj)
hw/vfio/user.c +267 -0

292 lines not on this page · raw message

Signature · 1 line
2.25.1
#30 lore raw

Re: [PATCH RFC v2 06/16] vfio-user: negotiate version with remote server

Quoted 18 lines · Elena Ufimtseva
hw/vfio/pci.c +6 -0
index 7005d9f891..eae33e746f 100644
@@ -3397,6 +3397,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         proxy->flags |= VFIO_PROXY_SECURE;
     }
 
+    vfio_user_validate_version(vbasedev, &err);
+    if (err != NULL) {
+        error_propagate(errp, err);
+        goto error;
+    }
+
     vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name);
     vbasedev->dev = DEVICE(vdev);
     vbasedev->fd = -1;

hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3413

Quoted 4 lines
hw/vfio/pci.c +1 -0
@@ -3404,6 +3410,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     vbasedev->no_mmap = false;
     vbasedev->ops = &vfio_user_pci_ops;
 
+error:

Missing return before error label? We shouldn't disconnect in the success case.

Quoted 28 lines
hw/vfio/pci.c +2 -0
+    vfio_user_disconnect(proxy);
+    error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name);
 }
 
 static void vfio_user_instance_finalize(Object *obj)
hw/vfio/user.c +7 -0
index 2fcc77d997..e89464a571 100644
@@ -23,9 +23,16 @@
 #include "io/channel-socket.h"
 #include "io/channel-util.h"
 #include "sysemu/iothread.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qjson.h"
+#include "qapi/qmp/qnull.h"
+#include "qapi/qmp/qstring.h"
+#include "qapi/qmp/qnum.h"
 #include "user.h"
 
 static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER;
+static uint64_t max_send_fds = VFIO_USER_DEF_MAX_FDS;
+static int wait_time = 1000;   /* wait 1 sec for replies */
 static IOThread *vfio_user_iothread;
 
 static void vfio_user_shutdown(VFIOProxy *proxy);
@@ -34,7 +41,14 @@ static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg,
                                   VFIOUserFDs *fds);

hw/vfio/user.c static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, · line 51

Quoted 10 lines
hw/vfio/user.c +7 -0
 static void vfio_user_send(VFIOProxy *proxy, VFIOUserHdr *msg,
                            VFIOUserFDs *fds);
+static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd,
+                                  uint32_t size, uint32_t flags);
+static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
+                                VFIOUserFDs *fds, int rsize, int flags);
 
+/* vfio_user_send_recv flags */
+#define NOWAIT          0x1  /* do not wait for reply */
+#define NOIOLOCK        0x2  /* do not drop iolock */

Please use "BQL", it's a widely used term while "iolock" isn't used:
s/IOLOCK/BQL/

Quoted 11 lines
hw/vfio/user.c +4 -0
 /*
  * Functions called by main, CPU, or iothread threads
@@ -333,6 +347,79 @@ static void vfio_user_cb(void *opaque)
  * Functions called by main or CPU threads
  */
 
+static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
+                                VFIOUserFDs *fds, int rsize, int flags)
+{
+    VFIOUserReply *reply;

hw/vfio/user.c static void vfio_user_cb(void *opaque) · line 363

Quoted 10 lines
hw/vfio/user.c +10 -0
+    bool iolock = 0;
+
+    if (msg->flags & VFIO_USER_NO_REPLY) {
+        error_printf("vfio_user_send_recv on async message\n");
+        return;
+    }
+
+    /*
+     * We may block later, so use a per-proxy lock and let
+     * the iothreads run while we sleep unless told no to

s/no/not/

Quoted 19 lines
hw/vfio/user.c +19 -0
+int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp)
+{
+    g_autofree VFIOUserVersion *msgp;
+    GString *caps;
+    int size, caplen;
+
+    caps = caps_json();
+    caplen = caps->len + 1;
+    size = sizeof(*msgp) + caplen;
+    msgp = g_malloc0(size);
+
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_VERSION, size, 0);
+    msgp->major = VFIO_USER_MAJOR_VER;
+    msgp->minor = VFIO_USER_MINOR_VER;
+    memcpy(&msgp->capabilities, caps->str, caplen);
+    g_string_free(caps, true);
+
+    vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, 0, 0);
+    if (msgp->hdr.flags & VFIO_USER_ERROR) {

hw/vfio/user.c void vfio_user_disconnect(VFIOProxy *proxy) · line 711

Quoted 10 lines
hw/vfio/user.c +10 -0
+        error_setg_errno(errp, msgp->hdr.error_reply, "version reply");
+        return -1;
+    }
+
+    if (msgp->major != VFIO_USER_MAJOR_VER ||
+        msgp->minor > VFIO_USER_MINOR_VER) {
+        error_setg(errp, "incompatible server version");
+        return -1;
+    }
+    if (caps_check(msgp->minor, (char *)msgp + sizeof(*msgp), errp) != 0) {

The reply is untrusted so we cannot treat it as a NUL-terminated string yet. The final byte msgp->capabilities[] needs to be checked first.

Please be careful about input validation, I might miss something so it's best if you audit the patches too. QEMU must not trust the device emulation process and vice versa.

Quoted 8 lines
hw/vfio/user.c +5 -0
+        return -1;
+    }
+
+    return 0;
+}
Signature · 1 line
2.25.1
#31 lore raw

Re: [PATCH RFC v2 06/16] vfio-user: negotiate version with remote server

Quoted 26 lines · Stefan Hajnoczi
Quoted 21 lines · Elena Ufimtseva
hw/vfio/pci.c +6 -0
index 7005d9f891..eae33e746f 100644
@@ -3397,6 +3397,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
        proxy->flags |= VFIO_PROXY_SECURE;
    }
 
+    vfio_user_validate_version(vbasedev, &err);
+    if (err != NULL) {
+        error_propagate(errp, err);
+        goto error;
+    }
+
    vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name);
    vbasedev->dev = DEVICE(vdev);
    vbasedev->fd = -1;
@@ -3404,6 +3410,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
    vbasedev->no_mmap = false;
    vbasedev->ops = &vfio_user_pci_ops;
 
Quoted 5 lines

hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3413

Quoted 1 line
hw/vfio/pci.c +1 -0
+error:

Missing return before error label? We shouldn't disconnect in the success case.

The return ended up in a later patch.

Quoted 35 lines
Quoted 35 lines
hw/vfio/pci.c +2 -0
+    vfio_user_disconnect(proxy);
+    error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name);
 }
 
 static void vfio_user_instance_finalize(Object *obj)
hw/vfio/user.c +11 -0
index 2fcc77d997..e89464a571 100644
@@ -23,9 +23,16 @@
 #include "io/channel-socket.h"
 #include "io/channel-util.h"
 #include "sysemu/iothread.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qjson.h"
+#include "qapi/qmp/qnull.h"
+#include "qapi/qmp/qstring.h"
+#include "qapi/qmp/qnum.h"
 #include "user.h"
 
 static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER;
+static uint64_t max_send_fds = VFIO_USER_DEF_MAX_FDS;
+static int wait_time = 1000;   /* wait 1 sec for replies */
 static IOThread *vfio_user_iothread;
 
 static void vfio_user_shutdown(VFIOProxy *proxy);
@@ -34,7 +41,14 @@ static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg,
                                  VFIOUserFDs *fds);
 static void vfio_user_send(VFIOProxy *proxy, VFIOUserHdr *msg,
                           VFIOUserFDs *fds);
+static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd,
+                                  uint32_t size, uint32_t flags);
+static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
+                                VFIOUserFDs *fds, int rsize, int flags);
 
Quoted 7 lines

hw/vfio/user.c static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, · line 51

Quoted 3 lines
hw/vfio/user.c +3 -0
+/* vfio_user_send_recv flags */
+#define NOWAIT          0x1  /* do not wait for reply */
+#define NOIOLOCK        0x2  /* do not drop iolock */

Please use "BQL", it's a widely used term while "iolock" isn't used:
s/IOLOCK/BQL/

OK

Quoted 7 lines
Quoted 7 lines
hw/vfio/user.c +0 -0
 /*
 * Functions called by main, CPU, or iothread threads
@@ -333,6 +347,79 @@ static void vfio_user_cb(void *opaque)
 * Functions called by main or CPU threads
 */
 
Quoted 16 lines
Quoted 4 lines
hw/vfio/user.c +4 -0
+static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
+                                VFIOUserFDs *fds, int rsize, int flags)
+{
+    VFIOUserReply *reply;

hw/vfio/user.c static void vfio_user_cb(void *opaque) · line 363

Quoted 10 lines
hw/vfio/user.c +10 -0
+    bool iolock = 0;
+
+    if (msg->flags & VFIO_USER_NO_REPLY) {
+        error_printf("vfio_user_send_recv on async message\n");
+        return;
+    }
+
+    /*
+     * We may block later, so use a per-proxy lock and let
+     * the iothreads run while we sleep unless told no to

s/no/not/

OK

Quoted 34 lines
Quoted 20 lines
hw/vfio/user.c +19 -0
+int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp)
+{
+    g_autofree VFIOUserVersion *msgp;
+    GString *caps;
+    int size, caplen;
+
+    caps = caps_json();
+    caplen = caps->len + 1;
+    size = sizeof(*msgp) + caplen;
+    msgp = g_malloc0(size);
+
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_VERSION, size, 0);
+    msgp->major = VFIO_USER_MAJOR_VER;
+    msgp->minor = VFIO_USER_MINOR_VER;
+    memcpy(&msgp->capabilities, caps->str, caplen);
+    g_string_free(caps, true);
+
+    vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, 0, 0);
+    if (msgp->hdr.flags & VFIO_USER_ERROR) {

hw/vfio/user.c void vfio_user_disconnect(VFIOProxy *proxy) · line 711

Quoted 10 lines
hw/vfio/user.c +10 -0
+        error_setg_errno(errp, msgp->hdr.error_reply, "version reply");
+        return -1;
+    }
+
+    if (msgp->major != VFIO_USER_MAJOR_VER ||
+        msgp->minor > VFIO_USER_MINOR_VER) {
+        error_setg(errp, "incompatible server version");
+        return -1;
+    }
+    if (caps_check(msgp->minor, (char *)msgp + sizeof(*msgp), errp) != 0) {

The reply is untrusted so we cannot treat it as a NUL-terminated string yet. The final byte msgp->capabilities[] needs to be checked first.

Quoted 4 lines

Please be careful about input validation, I might miss something so it's best if you audit the patches too. QEMU must not trust the device emulation process and vice versa.

This message is consumed by vfio-user, so I can check for valid
replies, but for most messages this checking will have to be done at in the VFIO common or bus-specific code, as vfio-user doens’t know valid data from invalid.

JJ

Quoted 7 lines
Quoted 7 lines
hw/vfio/user.c +5 -0
+        return -1;
+    }
+
+    return 0;
+}
Signature · 1 line
2.25.1
#32 lore raw

Re: [PATCH RFC v2 06/16] vfio-user: negotiate version with remote server

Quoted 40 lines · John Johnson
Quoted 35 lines · Stefan Hajnoczi
Quoted 19 lines · Elena Ufimtseva
hw/vfio/user.c +19 -0
+int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp)
+{
+    g_autofree VFIOUserVersion *msgp;
+    GString *caps;
+    int size, caplen;
+
+    caps = caps_json();
+    caplen = caps->len + 1;
+    size = sizeof(*msgp) + caplen;
+    msgp = g_malloc0(size);
+
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_VERSION, size, 0);
+    msgp->major = VFIO_USER_MAJOR_VER;
+    msgp->minor = VFIO_USER_MINOR_VER;
+    memcpy(&msgp->capabilities, caps->str, caplen);
+    g_string_free(caps, true);
+
+    vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, 0, 0);
+    if (msgp->hdr.flags & VFIO_USER_ERROR) {

hw/vfio/user.c void vfio_user_disconnect(VFIOProxy *proxy) · line 711

Quoted 10 lines
hw/vfio/user.c +10 -0
+        error_setg_errno(errp, msgp->hdr.error_reply, "version reply");
+        return -1;
+    }
+
+    if (msgp->major != VFIO_USER_MAJOR_VER ||
+        msgp->minor > VFIO_USER_MINOR_VER) {
+        error_setg(errp, "incompatible server version");
+        return -1;
+    }
+    if (caps_check(msgp->minor, (char *)msgp + sizeof(*msgp), errp) != 0) {

The reply is untrusted so we cannot treat it as a NUL-terminated string yet. The final byte msgp->capabilities[] needs to be checked first.

Quoted 4 lines

Please be careful about input validation, I might miss something so it's best if you audit the patches too. QEMU must not trust the device emulation process and vice versa.

Quoted 4 lines

This message is consumed by vfio-user, so I can check for valid
replies, but for most messages this checking will have to be done at in the VFIO common or bus-specific code, as vfio-user doens’t know valid data from invalid.

Good point. Today's VFIO code trusts the replies because they come from the kernel. Now that they can come from an untrusted source they must be validated and common VFIO code may need to change its trust model.

Stefan

#33 lore raw

RFC PATCH v2 7/16 vfio-user: get device info

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h | 13 +++++++++++++
 hw/vfio/user.h          |  1 +
 hw/vfio/pci.c           | 13 +++++++++++++
 hw/vfio/user.c          | 20 ++++++++++++++++++++
 4 files changed, 47 insertions(+)
hw/vfio/user-protocol.h +13 -0
index 14b762d1ad..13e44ebf1c 100644
@@ -82,4 +82,17 @@ typedef struct {
 #define VFIO_USER_MAX_MAX_XFER  (64 * 1024 * 1024)
 
 
+/*
+ * VFIO_USER_DEVICE_GET_INFO
+ * imported from struct_device_info
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t num_regions;
+    uint32_t num_irqs;
+    uint32_t cap_offset;
+} VFIOUserDeviceInfo;
+
 #endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +1 -0
index cab957ba7a..82044e7e78 100644
@@ -71,5 +71,6 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev,
                                              void *reqarg);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
+int vfio_user_get_info(VFIODevice *vbasedev);
 
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +13 -0
index eae33e746f..63aa2441f0 100644
@@ -3369,6 +3369,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     VFIODevice *vbasedev = &vdev->vbasedev;
     SocketAddress addr;
     VFIOProxy *proxy;
+    int ret;
     Error *err = NULL;
 
     /*
@@ -3410,6 +3411,18 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     vbasedev->no_mmap = false;
     vbasedev->ops = &vfio_user_pci_ops;
 
+    ret = vfio_user_get_info(&vdev->vbasedev);
+    if (ret) {
+        error_setg_errno(errp, -ret, "get info failure");
+        goto error;
+    }
+
+    vfio_populate_device(vdev, &err);
+    if (err) {
+        error_propagate(errp, err);
+        goto error;
+    }
+
 error:
     vfio_user_disconnect(proxy);
     error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name);
hw/vfio/user.c +20 -0
index e89464a571..b584b8e0f2 100644
@@ -714,3 +714,23 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp)
 
     return 0;
 }
+
+int vfio_user_get_info(VFIODevice *vbasedev)
+{
+    VFIOUserDeviceInfo msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0);
+    msg.argsz = sizeof(struct vfio_device_info);
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0);
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
+    vbasedev->num_irqs = msg.num_irqs;
+    vbasedev->num_regions = msg.num_regions;
+    vbasedev->flags = msg.flags;
+    vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET);
+    return 0;
+}
Signature · 1 line
2.25.1
#34 lore raw

Re: [PATCH RFC v2 07/16] vfio-user: get device info

Quoted 7 lines · Elena Ufimtseva
hw/vfio/user.c +7 -0
+int vfio_user_get_info(VFIODevice *vbasedev)
+{
+    VFIOUserDeviceInfo msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0);
+    msg.argsz = sizeof(struct vfio_device_info);

hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 734

Quoted 10 lines
hw/vfio/user.c +10 -0
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0);
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
+    vbasedev->num_irqs = msg.num_irqs;
+    vbasedev->num_regions = msg.num_regions;
+    vbasedev->flags = msg.flags;
+    vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET);

No input validation. I haven't checked what happens when num_irqs, num_regions, or flags are bogus but it's a little concerning. Unlike kernel VFIO, we do not trust these values.

Stefan

#35 lore raw

Re: [PATCH RFC v2 07/16] vfio-user: get device info

Quoted 18 lines · Stefan Hajnoczi
Quoted 13 lines · Elena Ufimtseva
hw/vfio/user.c +13 -0
+int vfio_user_get_info(VFIODevice *vbasedev)
+{
+    VFIOUserDeviceInfo msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0);
+    msg.argsz = sizeof(struct vfio_device_info);
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0);
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
Quoted 9 lines

hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 734

Quoted 4 lines
hw/vfio/user.c +4 -0
+    vbasedev->num_irqs = msg.num_irqs;
+    vbasedev->num_regions = msg.num_regions;
+    vbasedev->flags = msg.flags;
+    vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET);

No input validation. I haven't checked what happens when num_irqs, num_regions, or flags are bogus but it's a little concerning. Unlike kernel VFIO, we do not trust these values.

As in the last reply, vfio-user doesn’t know valid values
from invalid, so I need to re-work this so the PCI-specific code that calls vfio-user_get_info() can test for invalid values.

JJ

#36 lore raw

Re: [PATCH RFC v2 07/16] vfio-user: get device info

Quoted 28 lines · John Johnson
Quoted 18 lines · Stefan Hajnoczi
Quoted 13 lines · Elena Ufimtseva
hw/vfio/user.c +13 -0
+int vfio_user_get_info(VFIODevice *vbasedev)
+{
+    VFIOUserDeviceInfo msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0);
+    msg.argsz = sizeof(struct vfio_device_info);
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0);
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
Quoted 9 lines

hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 734

Quoted 4 lines
hw/vfio/user.c +4 -0
+    vbasedev->num_irqs = msg.num_irqs;
+    vbasedev->num_regions = msg.num_regions;
+    vbasedev->flags = msg.flags;
+    vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET);

No input validation. I haven't checked what happens when num_irqs, num_regions, or flags are bogus but it's a little concerning. Unlike kernel VFIO, we do not trust these values.

Quoted 3 lines

As in the last reply, vfio-user doesn’t know valid values
from invalid, so I need to re-work this so the PCI-specific code that calls vfio-user_get_info() can test for invalid values.

Sounds good. I won't look further for missing input validation in the VFIO message contents in this revision of the patch series. Once you're happy with input validation I'll look at the code from this angle again.

Stefan

#37 lore raw

RFC PATCH v2 8/16 vfio-user: get region info

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h       | 14 +++++++
 hw/vfio/user.h                |  2 +
 include/hw/vfio/vfio-common.h |  3 ++
 hw/vfio/common.c              | 76 ++++++++++++++++++++++++++++++++++-
 hw/vfio/user.c                | 33 +++++++++++++++
 5 files changed, 126 insertions(+), 2 deletions(-)
hw/vfio/user-protocol.h +14 -0
index 13e44ebf1c..104bf4ff31 100644
@@ -95,4 +95,18 @@ typedef struct {
     uint32_t cap_offset;
 } VFIOUserDeviceInfo;
 
+/*
+ * VFIO_USER_DEVICE_GET_REGION_INFO
+ * imported from struct_vfio_region_info
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t cap_offset;
+    uint64_t size;
+    uint64_t offset;
+} VFIOUserRegionInfo;
+
 #endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +2 -0
index 82044e7e78..f0122539ba 100644
@@ -72,5 +72,7 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev,
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 int vfio_user_get_info(VFIODevice *vbasedev);
+int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
+                              struct vfio_region_info *info, VFIOUserFDs *fds);
 
 #endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +3 -0
index f43dc6e5d0..bdd25a546c 100644
@@ -56,6 +56,7 @@ typedef struct VFIORegion {
     uint32_t nr_mmaps;
     VFIOMmap *mmaps;
     uint8_t nr; /* cache the region number for debug */
+    int remfd; /* fd if exported from remote process */
 } VFIORegion;
 
 typedef struct VFIOMigration {
@@ -145,6 +146,8 @@ typedef struct VFIODevice {
     Error *migration_blocker;
     OnOffAuto pre_copy_dirty_page_tracking;
     VFIOProxy *proxy;
+    struct vfio_region_info **regions;
+    int *regfds;
 } VFIODevice;
 
 struct VFIODeviceOps {
hw/vfio/common.c +74 -2

136 lines not on this page · raw message

hw/vfio/user.c +33 -0
index b584b8e0f2..91b51f37df 100644
@@ -734,3 +734,36 @@ int vfio_user_get_info(VFIODevice *vbasedev)
     vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET);
     return 0;
 }
+
+int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
+                              struct vfio_region_info *info, VFIOUserFDs *fds)
+{
+    g_autofree VFIOUserRegionInfo *msgp = NULL;
+    int size;
+
+    /* data returned can be larger than vfio_region_info */
+    if (info->argsz < sizeof(*info)) {
+        error_printf("vfio_user_get_region_info argsz too small\n");
+        return -EINVAL;
+    }
+    if (fds != NULL && fds->send_fds != 0) {
+        error_printf("vfio_user_get_region_info can't send FDs\n");
+        return -EINVAL;
+    }
+
+    size = info->argsz + sizeof(VFIOUserHdr);
+    msgp = g_malloc0(size);
+
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_GET_REGION_INFO,
+                          sizeof(*msgp), 0);
+    msgp->argsz = info->argsz;
+    msgp->index = info->index;
+
+    vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, fds, size, 0);
+    if (msgp->hdr.flags & VFIO_USER_ERROR) {
+        return -msgp->hdr.error_reply;
+    }
+
+    memcpy(info, &msgp->argsz, info->argsz);
+    return 0;
+}
Signature · 1 line
2.25.1
#38 lore raw

Re: [PATCH RFC v2 08/16] vfio-user: get region info

hw/vfio/common.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 1524

Quoted 11 lines · Elena Ufimtseva
hw/vfio/common.c +7 -0
@@ -1514,6 +1515,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,
     return true;
 }
 
+static int vfio_get_region_info_remfd(VFIODevice *vbasedev, int index)
+{
+    struct vfio_region_info *info;
+
+    if (vbasedev->regions == NULL || vbasedev->regions[index] == NULL) {
+        vfio_get_region_info(vbasedev, index, &info);
+    }

Maybe this will be called from other places in the future, but the vfio_region_setup() caller below already invoked vfio_get_region_info() so I'm not sure it's necessary to do this again?

Perhaps add an int *remfd argument to vfio_get_region_info(). That way vfio_get_region_info_remfd() isn't necessary.

Quoted 12 lines
hw/vfio/common.c +8 -0
@@ -2410,6 +2442,24 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index,
                          struct vfio_region_info **info)
 {
     size_t argsz = sizeof(struct vfio_region_info);
+    int fd = -1;
+    int ret;
+
+    /* create region cache */
+    if (vbasedev->regions == NULL) {
+        vbasedev->regions = g_new0(struct vfio_region_info *,
+                                   vbasedev->num_regions);
+        if (vbasedev->proxy != NULL) {

hw/vfio/common.c int vfio_get_region_info(VFIODevice *vbasedev, int index, · line 2462

Quoted 10 lines
hw/vfio/common.c +10 -0
+            vbasedev->regfds = g_new0(int, vbasedev->num_regions);
+        }
+    }
+    /* check cache */
+    if (vbasedev->regions[index] != NULL) {
+        *info = g_malloc0(vbasedev->regions[index]->argsz);
+        memcpy(*info, vbasedev->regions[index],
+               vbasedev->regions[index]->argsz);
+        return 0;
+    }

Why is it necessary to introduce a cache? Is it to avoid passing the same fd multiple times?

Quoted 28 lines
hw/vfio/common.c +13 -1
     *info = g_malloc0(argsz);
 
@@ -2417,7 +2467,17 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index,
 retry:
     (*info)->argsz = argsz;
 
-    if (ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info)) {
+    if (vbasedev->proxy != NULL) {
+        VFIOUserFDs fds = { 0, 1, &fd};
+
+        ret = vfio_user_get_region_info(vbasedev, index, *info, &fds);
+    } else {
+        ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info);
+        if (ret < 0) {
+            ret = -errno;
+        }
+    }
+    if (ret != 0) {
         g_free(*info);
         *info = NULL;
         return -errno;
@@ -2426,10 +2486,22 @@ retry:
     if ((*info)->argsz > argsz) {
         argsz = (*info)->argsz;
         *info = g_realloc(*info, argsz);
+        if (fd != -1) {
+            close(fd);

hw/vfio/common.c retry: · line 2500

Quoted 10 lines
hw/vfio/common.c +6 -0
+            fd = -1;
+        }
 
         goto retry;
     }
 
+    /* fill cache */
+    vbasedev->regions[index] = g_malloc0(argsz);
+    memcpy(vbasedev->regions[index], *info, argsz);
+    *vbasedev->regions[index] = **info;

The previous line already copied the contents of *info. What is the purpose of this assignment?

Quoted 12 lines
hw/vfio/common.c +4 -0
+    if (vbasedev->regfds != NULL) {
+        vbasedev->regfds[index] = fd;
+    }
+
     return 0;
 }
 
hw/vfio/user.c +0 -0
index b584b8e0f2..91b51f37df 100644

hw/vfio/user.c int vfio_user_get_info(VFIODevice *vbasedev) · line 742

Quoted 9 lines
hw/vfio/user.c +6 -0
@@ -734,3 +734,36 @@ int vfio_user_get_info(VFIODevice *vbasedev)
     vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET);
     return 0;
 }
+
+int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
+                              struct vfio_region_info *info, VFIOUserFDs *fds)
+{
+    g_autofree VFIOUserRegionInfo *msgp = NULL;
+    int size;

Please use uint32_t size instead of int size to prevent possible signedness issues:
- VFIOUserRegionInfo->argsz is uint32_t.
- sizeof(VFIOUserHdr) is size_t.
- The vfio_user_request_msg() size argument is uint32_t.

#39 lore raw

Re: [PATCH RFC v2 08/16] vfio-user: get region info

Quoted 21 lines · Stefan Hajnoczi

hw/vfio/common.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 1524

Quoted 11 lines · Elena Ufimtseva
hw/vfio/common.c +7 -0
@@ -1514,6 +1515,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,
    return true;
 }
 
+static int vfio_get_region_info_remfd(VFIODevice *vbasedev, int index)
+{
+    struct vfio_region_info *info;
+
+    if (vbasedev->regions == NULL || vbasedev->regions[index] == NULL) {
+        vfio_get_region_info(vbasedev, index, &info);
+    }

Maybe this will be called from other places in the future, but the vfio_region_setup() caller below already invoked vfio_get_region_info() so I'm not sure it's necessary to do this again?

Quoted 3 lines

Perhaps add an int *remfd argument to vfio_get_region_info(). That way vfio_get_region_info_remfd() isn't necessary.

I think they could be combined, but the region capabilities are
retrieved with separate calls to vfio_get_region_info_cap(), so I followed that precedent.

Quoted 17 lines
Quoted 17 lines
hw/vfio/common.c +13 -0
@@ -2410,6 +2442,24 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index,
                         struct vfio_region_info **info)
 {
    size_t argsz = sizeof(struct vfio_region_info);
+    int fd = -1;
+    int ret;
+
+    /* create region cache */
+    if (vbasedev->regions == NULL) {
+        vbasedev->regions = g_new0(struct vfio_region_info *,
+                                   vbasedev->num_regions);
+        if (vbasedev->proxy != NULL) {
+            vbasedev->regfds = g_new0(int, vbasedev->num_regions);
+        }
+    }
+    /* check cache */
+    if (vbasedev->regions[index] != NULL) {
Quoted 9 lines

hw/vfio/common.c int vfio_get_region_info(VFIODevice *vbasedev, int index, · line 2462

Quoted 5 lines
hw/vfio/common.c +5 -0
+        *info = g_malloc0(vbasedev->regions[index]->argsz);
+        memcpy(*info, vbasedev->regions[index],
+               vbasedev->regions[index]->argsz);
+        return 0;
+    }

Why is it necessary to introduce a cache? Is it to avoid passing the same fd multiple times?

Yes. For polled regions, the server returns an FD with each
message, so there would be an FD leak if I didn’t check that the region hadn’t already returned one. Since I had to cache the FD anyway, I cached the whole struct.

Quoted 34 lines
Quoted 34 lines
hw/vfio/common.c +0 -0
    *info = g_malloc0(argsz);
 
@@ -2417,7 +2467,17 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index,

retry:

    (*info)->argsz = argsz;
hw/vfio/common.c +15 -1
-    if (ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info)) {
+    if (vbasedev->proxy != NULL) {
+        VFIOUserFDs fds = { 0, 1, &fd};
+
+        ret = vfio_user_get_region_info(vbasedev, index, *info, &fds);
+    } else {
+        ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info);
+        if (ret < 0) {
+            ret = -errno;
+        }
+    }
+    if (ret != 0) {
        g_free(*info);
        *info = NULL;
        return -errno;
@@ -2426,10 +2486,22 @@ retry:
    if ((*info)->argsz > argsz) {
        argsz = (*info)->argsz;
        *info = g_realloc(*info, argsz);
+        if (fd != -1) {
+            close(fd);
+            fd = -1;
+        }
 
        goto retry;
    }
 
Quoted 8 lines

hw/vfio/common.c retry: · line 2500

Quoted 4 lines
hw/vfio/common.c +4 -0
+    /* fill cache */
+    vbasedev->regions[index] = g_malloc0(argsz);
+    memcpy(vbasedev->regions[index], *info, argsz);
+    *vbasedev->regions[index] = **info;

The previous line already copied the contents of *info. What is the purpose of this assignment?

That might be a mis-merge. The struct assignment was a bug
fixed several revs ago with the memcpy() call.

Quoted 19 lines
Quoted 19 lines
hw/vfio/common.c +4 -0
+    if (vbasedev->regfds != NULL) {
+        vbasedev->regfds[index] = fd;
+    }
+
    return 0;
 }
 
hw/vfio/user.c +4 -0
index b584b8e0f2..91b51f37df 100644
@@ -734,3 +734,36 @@ int vfio_user_get_info(VFIODevice *vbasedev)
    vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET);
    return 0;
 }
+
+int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
+                              struct vfio_region_info *info, VFIOUserFDs *fds)
+{
Quoted 8 lines

hw/vfio/user.c int vfio_user_get_info(VFIODevice *vbasedev) · line 742

Quoted 2 lines
hw/vfio/user.c +2 -0
+    g_autofree VFIOUserRegionInfo *msgp = NULL;
+    int size;

Please use uint32_t size instead of int size to prevent possible signedness issues:
- VFIOUserRegionInfo->argsz is uint32_t.
- sizeof(VFIOUserHdr) is size_t.
- The vfio_user_request_msg() size argument is uint32_t.

OK

JJ

#40 lore raw

Re: [PATCH RFC v2 08/16] vfio-user: get region info

Quoted 57 lines · John Johnson
Quoted 21 lines · Stefan Hajnoczi

hw/vfio/common.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 1524

Quoted 11 lines · Elena Ufimtseva
hw/vfio/common.c +7 -0
@@ -1514,6 +1515,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,
    return true;
 }
 
+static int vfio_get_region_info_remfd(VFIODevice *vbasedev, int index)
+{
+    struct vfio_region_info *info;
+
+    if (vbasedev->regions == NULL || vbasedev->regions[index] == NULL) {
+        vfio_get_region_info(vbasedev, index, &info);
+    }

Maybe this will be called from other places in the future, but the vfio_region_setup() caller below already invoked vfio_get_region_info() so I'm not sure it's necessary to do this again?

Quoted 3 lines

Perhaps add an int *remfd argument to vfio_get_region_info(). That way vfio_get_region_info_remfd() isn't necessary.

I think they could be combined, but the region capabilities are
retrieved with separate calls to vfio_get_region_info_cap(), so I followed that precedent.

Quoted 17 lines
Quoted 17 lines
hw/vfio/common.c +13 -0
@@ -2410,6 +2442,24 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index,
                         struct vfio_region_info **info)
 {
    size_t argsz = sizeof(struct vfio_region_info);
+    int fd = -1;
+    int ret;
+
+    /* create region cache */
+    if (vbasedev->regions == NULL) {
+        vbasedev->regions = g_new0(struct vfio_region_info *,
+                                   vbasedev->num_regions);
+        if (vbasedev->proxy != NULL) {
+            vbasedev->regfds = g_new0(int, vbasedev->num_regions);
+        }
+    }
+    /* check cache */
+    if (vbasedev->regions[index] != NULL) {
Quoted 9 lines

hw/vfio/common.c int vfio_get_region_info(VFIODevice *vbasedev, int index, · line 2462

Quoted 5 lines
hw/vfio/common.c +5 -0
+        *info = g_malloc0(vbasedev->regions[index]->argsz);
+        memcpy(*info, vbasedev->regions[index],
+               vbasedev->regions[index]->argsz);
+        return 0;
+    }

Why is it necessary to introduce a cache? Is it to avoid passing the same fd multiple times?

Quoted 4 lines

Yes. For polled regions, the server returns an FD with each
message, so there would be an FD leak if I didn’t check that the region hadn’t already returned one. Since I had to cache the FD anyway, I cached the whole struct.

If vfio_get_region_info() takes an int *fd argument then fd ownership becomes explicit and the need for the cache falls away. Maybe Alex has a preference for how to structure the code to track per-region fds.

Stefan

#41 lore raw

RFC PATCH v2 9/16 vfio-user: region read/write

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h | 12 ++++++++++++
 hw/vfio/user.h          |  4 ++++
 hw/vfio/common.c        | 16 +++++++++++++--
 hw/vfio/user.c          | 43 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 73 insertions(+), 2 deletions(-)
hw/vfio/user-protocol.h +12 -0
index 104bf4ff31..56904cf872 100644
@@ -109,4 +109,16 @@ typedef struct {
     uint64_t offset;
 } VFIOUserRegionInfo;
 
+/*
+ * VFIO_USER_REGION_READ
+ * VFIO_USER_REGION_WRITE
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint64_t offset;
+    uint32_t region;
+    uint32_t count;
+    char data[];
+} VFIOUserRegionRW;
+
 #endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +4 -0
index f0122539ba..02f832a173 100644
@@ -74,5 +74,9 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 int vfio_user_get_info(VFIODevice *vbasedev);
 int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
                               struct vfio_region_info *info, VFIOUserFDs *fds);
+int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
+                          uint32_t count, void *data);
+int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
+                           uint64_t offset, uint32_t count, void *data);
 
 #endif /* VFIO_USER_H */
hw/vfio/common.c +14 -2
index 7d667b0533..a8b1ea9358 100644
@@ -215,6 +215,7 @@ void vfio_region_write(void *opaque, hwaddr addr,
         uint32_t dword;
         uint64_t qword;
     } buf;
+    int ret;
 
     switch (size) {
     case 1:
@@ -234,7 +235,12 @@ void vfio_region_write(void *opaque, hwaddr addr,
         break;
     }
 
-    if (pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) {
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_region_write(vbasedev, region->nr, addr, size, &data);
+    } else {
+        ret = pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr);
+    }
+    if (ret != size) {
         error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64
                      ",%d) failed: %m",
                      __func__, vbasedev->name, region->nr,
@@ -266,8 +272,14 @@ uint64_t vfio_region_read(void *opaque,
         uint64_t qword;
     } buf;
     uint64_t data = 0;
+    int ret;
 
-    if (pread(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) {
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_region_read(vbasedev, region->nr, addr, size, &buf);
+    } else {
+        ret = pread(vbasedev->fd, &buf, size, region->fd_offset + addr);
+    }
+    if (ret != size) {
         error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %m",
                      __func__, vbasedev->name, region->nr,
                      addr, size);
hw/vfio/user.c +43 -0

46 lines not on this page · raw message

Signature · 1 line
2.25.1
#42 lore raw

Re: [PATCH RFC v2 09/16] vfio-user: region read/write

Quoted 13 lines · Elena Ufimtseva
hw/vfio/common.c +1 -0
index 7d667b0533..a8b1ea9358 100644
@@ -215,6 +215,7 @@ void vfio_region_write(void *opaque, hwaddr addr,
         uint32_t dword;
         uint64_t qword;
     } buf;
+    int ret;
 
     switch (size) {
     case 1:

hw/vfio/common.c void vfio_region_write(void *opaque, hwaddr addr, · line 242

Quoted 9 lines
hw/vfio/common.c +5 -1
@@ -234,7 +235,12 @@ void vfio_region_write(void *opaque, hwaddr addr,
         break;
     }
 
-    if (pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) {
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_region_write(vbasedev, region->nr, addr, size, &data);
+    } else {
+        ret = pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr);
+    }

The vfio-user spec says everything is little-endian. Why does vfio_user_region_write() take the host-endian uint64_t data value instead of the little-endian buf value?

Quoted 58 lines
hw/vfio/common.c +8 -1
+    if (ret != size) {
         error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64
                      ",%d) failed: %m",
                      __func__, vbasedev->name, region->nr,
@@ -266,8 +272,14 @@ uint64_t vfio_region_read(void *opaque,
         uint64_t qword;
     } buf;
     uint64_t data = 0;
+    int ret;
 
-    if (pread(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) {
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_region_read(vbasedev, region->nr, addr, size, &buf);
+    } else {
+        ret = pread(vbasedev->fd, &buf, size, region->fd_offset + addr);
+    }
+    if (ret != size) {
         error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %m",
                      __func__, vbasedev->name, region->nr,
                      addr, size);
hw/vfio/user.c +30 -0
index 91b51f37df..83235b2411 100644
@@ -767,3 +767,46 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
     memcpy(info, &msgp->argsz, info->argsz);
     return 0;
 }
+
+int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
+                                 uint32_t count, void *data)
+{
+    g_autofree VFIOUserRegionRW *msgp = NULL;
+    int size = sizeof(*msgp) + count;
+
+    msgp = g_malloc0(size);
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_READ, sizeof(*msgp), 0);
+    msgp->offset = offset;
+    msgp->region = index;
+    msgp->count = count;
+
+    vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, size, 0);
+    if (msgp->hdr.flags & VFIO_USER_ERROR) {
+        return -msgp->hdr.error_reply;
+    } else if (msgp->count > count) {
+        return -E2BIG;
+    } else {
+        memcpy(data, &msgp->data, msgp->count);
+    }
+
+    return msgp->count;
+}
+
+int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
+                           uint64_t offset, uint32_t count, void *data)
+{
+    g_autofree VFIOUserRegionRW *msgp = NULL;
+    int size = sizeof(*msgp) + count;

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 809

Quoted 10 lines
hw/vfio/user.c +10 -0
+
+    msgp = g_malloc0(size);
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size,
+                          VFIO_USER_NO_REPLY);
+    msgp->offset = offset;
+    msgp->region = index;
+    msgp->count = count;
+    memcpy(&msgp->data, data, count);
+
+    vfio_user_send(vbasedev->proxy, &msgp->hdr, NULL);

Are VFIO region writes posted writes (VFIO_USER_NO_REPLY)? This can be a problem if the device driver performs a write to the region followed by another access (e.g. to an mmap region) and expected the write to complete before the second access takes place.

#43 lore raw

Re: [PATCH RFC v2 09/16] vfio-user: region read/write

In reply to Elena Ufimtseva

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803

Quoted 9 lines · Elena Ufimtseva
hw/vfio/user.c +9 -0
+int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
+                           uint64_t offset, uint32_t count, void *data)
+{
+    g_autofree VFIOUserRegionRW *msgp = NULL;
+    int size = sizeof(*msgp) + count;
+
+    msgp = g_malloc0(size);
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size,
+                          VFIO_USER_NO_REPLY);

Mirroring https://github.com/oracle/qemu/issues/10 here for visibility:

Currently, vfio_user_region_write uses VFIO_USER_NO_REPLY unconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.

regards
john

#44 lore raw

Re: [PATCH RFC v2 09/16] vfio-user: region read/write

Quoted 18 lines · John Levon

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803

Quoted 9 lines · Elena Ufimtseva
hw/vfio/user.c +9 -0
+int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
+                           uint64_t offset, uint32_t count, void *data)
+{
+    g_autofree VFIOUserRegionRW *msgp = NULL;
+    int size = sizeof(*msgp) + count;
+
+    msgp = g_malloc0(size);
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size,
+                          VFIO_USER_NO_REPLY);

Mirroring https://github.com/oracle/qemu/issues/10 here for visibility:

Quoted 5 lines

Currently, vfio_user_region_write uses VFIO_USER_NO_REPLY unconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.

I’m not sure following the PCI spec (mem writes posted, config & IO
are not) completely solves the issue if the device uses sparse mmap. A store to went over the socket can be passed by a load that goes directly to memory, which could break a driver that assumes a load completion means older stores to the same device have also completed.

JJ

#45 lore raw

Re: [PATCH RFC v2 09/16] vfio-user: region read/write

Quoted 21 lines · John Johnson
Quoted 16 lines · John Levon

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803

Quoted 9 lines · Elena Ufimtseva
hw/vfio/user.c +9 -0
+int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
+                           uint64_t offset, uint32_t count, void *data)
+{
+    g_autofree VFIOUserRegionRW *msgp = NULL;
+    int size = sizeof(*msgp) + count;
+
+    msgp = g_malloc0(size);
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size,
+                          VFIO_USER_NO_REPLY);

Mirroring https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_oracle_qemu_issues_10&d=DwIGaQ&c=s883GpUCOChKOHiocYtGcg&r=v7SNLJqx7b9Vfc7ZO82Wg4nnZ8O5XkACFQ30bVKxotI&m=PJ390CfKPdTFUffSi02whMSqey2en8OJv7dm9VAQKI0&s=Mfp1xRKET3LEucEeZwUVUvSJ3V0zzGEktOzFwMsTfEE&e= here for visibility:

Quoted 4 lines

Currently, vfio_user_region_write uses VFIO_USER_NO_REPLY unconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.

Quoted 5 lines

I’m not sure following the PCI spec (mem writes posted, config & IO
are not) completely solves the issue if the device uses sparse mmap. A store to went over the socket can be passed by a load that goes directly to memory, which could break a driver that assumes a load completion means older stores to the same device have also completed.

Sure, but sparse mmaps are under the device's control - so wouldn't that be something of a "don't do that" scenario?

regards
john

#46 lore raw

Re: [PATCH RFC v2 09/16] vfio-user: region read/write

Quoted 27 lines · John Levon
Quoted 21 lines · John Johnson
Quoted 20 lines · John Levon

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803

Quoted 9 lines · Elena Ufimtseva
hw/vfio/user.c +9 -0
+int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
+                           uint64_t offset, uint32_t count, void *data)
+{
+    g_autofree VFIOUserRegionRW *msgp = NULL;
+    int size = sizeof(*msgp) + count;
+
+    msgp = g_malloc0(size);
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size,
+                          VFIO_USER_NO_REPLY);

Mirroring https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_oracle_qemu_issues_10&d=DwIGaQ&c=s883GpUCOChKOHiocYtGcg&r=v7SNLJqx7b9Vfc7ZO82Wg4nnZ8O5XkACFQ30bVKxotI&m=PJ390CfKPdTFUffSi02whMSqey2en8OJv7dm9VAQKI0&s=Mfp1xRKET3LEucEeZwUVUvSJ3V0zzGEktOzFwMsTfEE&e= here for visibility:

Currently, vfio_user_region_write uses VFIO_USER_NO_REPLY unconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.

Quoted 9 lines
Quoted 5 lines

I’m not sure following the PCI spec (mem writes posted, config & IO
are not) completely solves the issue if the device uses sparse mmap. A store to went over the socket can be passed by a load that goes directly to memory, which could break a driver that assumes a load completion means older stores to the same device have also completed.

Sure, but sparse mmaps are under the device's control - so wouldn't that be something of a "don't do that" scenario?

The sparse mmaps are under the emulation program’s control, but it
doesn’t know what registers the guest device driver is using to force stores to complete. The Linux device drivers doc on kernel.org just says the driver must read from the same device.

JJ

https://www.kernel.org/doc/Documentation/driver-api/device-io.rst

While the basic functions are defined to be synchronous with respect to each other and ordered with respect to each other the busses the devices sit on may themselves have asynchronicity. In particular many authors are burned by the fact that PCI bus writes are posted asynchronously. A driver author must issue a read from the same device to ensure that writes have occurred in the specific cases the author cares.

#47 lore raw

Re: [PATCH RFC v2 09/16] vfio-user: region read/write

Quoted 25 lines · John Johnson
Quoted 25 lines
Quoted 24 lines
Quoted 14 lines · Elena Ufimtseva

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 803

Quoted 10 lines
hw/vfio/user.c +9 -0
+int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
+                           uint64_t offset, uint32_t count, void *data)
+{
+    g_autofree VFIOUserRegionRW *msgp = NULL;
+    int size = sizeof(*msgp) + count;
+
+    msgp = g_malloc0(size);
+    vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size,
+                          VFIO_USER_NO_REPLY);

Mirroring https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_oracle_qemu_issues_10&d=DwIGaQ&c=s883GpUCOChKOHiocYtGcg&r=v7SNLJqx7b9Vfc7ZO82Wg4nnZ8O5XkACFQ30bVKxotI&m=PJ390CfKPdTFUffSi02whMSqey2en8OJv7dm9VAQKI0&s=Mfp1xRKET3LEucEeZwUVUvSJ3V0zzGEktOzFwMsTfEE&e= here for visibility:

Quoted 4 lines

Currently, vfio_user_region_write uses VFIO_USER_NO_REPLY unconditionally, meaning essentially all writes are posted. But that shouldn't be the case, for example for PCI config space, where it's expected that writes will wait for an ack before the VCPU continues.

I’m not sure following the PCI spec (mem writes posted, config & IO
are not) completely solves the issue if the device uses sparse mmap. A store to went over the socket can be passed by a load that goes directly to memory, which could break a driver that assumes a load completion means older stores to the same device have also completed.

Quoted 7 lines
Quoted 2 lines

Sure, but sparse mmaps are under the device's control - so wouldn't that be something of a "don't do that" scenario?

The sparse mmaps are under the emulation program’s control, but it
doesn’t know what registers the guest device driver is using to force stores to complete. The Linux device drivers doc on kernel.org just says the driver must read from the same device.

Sure, but any device where that is important wouldn't use the sparse mmaps, no?

There's no other alternative.

regards
john

#48 lore raw

RFC PATCH v2 10/16 vfio-user: pci_user_realize PCI setup

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

PCI BARs read from remote device
PCI config reads/writes sent to remote server

---
 hw/vfio/pci.c | 210 +++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 175 insertions(+), 35 deletions(-)
hw/vfio/pci.c +175 -35
index 63aa2441f0..ea0df8be65 100644
@@ -807,8 +807,14 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev)
     memset(vdev->rom, 0xff, size);
 
     while (size) {
-        bytes = pread(vdev->vbasedev.fd, vdev->rom + off,
-                      size, vdev->rom_offset + off);
+        if (vdev->vbasedev.proxy != NULL) {
+            bytes = vfio_user_region_read(&vdev->vbasedev,
+                                          VFIO_PCI_ROM_REGION_INDEX,
+                                          off, size, vdev->rom + off);
+        } else {
+            bytes = pread(vdev->vbasedev.fd, vdev->rom + off,
+                          size, vdev->rom_offset + off);
+        }
         if (bytes == 0) {
             break;
         } else if (bytes > 0) {
@@ -927,12 +933,28 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev)
      * Use the same size ROM BAR as the physical device.  The contents
      * will get filled in later when the guest tries to read it.
      */
-    if (pread(fd, &orig, 4, offset) != 4 ||
-        pwrite(fd, &size, 4, offset) != 4 ||
-        pread(fd, &size, 4, offset) != 4 ||
-        pwrite(fd, &orig, 4, offset) != 4) {
-        error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name);
-        return;
+    if (vdev->vbasedev.proxy != NULL) {
+        if (vfio_user_region_read(&vdev->vbasedev, VFIO_PCI_CONFIG_REGION_INDEX,
+                                  PCI_ROM_ADDRESS, 4, &orig) != 4 ||
+            vfio_user_region_write(&vdev->vbasedev,
+                                   VFIO_PCI_CONFIG_REGION_INDEX,
+                                   PCI_ROM_ADDRESS, 4, &size) != 4 ||
+            vfio_user_region_read(&vdev->vbasedev, VFIO_PCI_CONFIG_REGION_INDEX,
+                                  PCI_ROM_ADDRESS, 4, &size) != 4 ||
+            vfio_user_region_write(&vdev->vbasedev,
+                                   VFIO_PCI_CONFIG_REGION_INDEX,
+                                   PCI_ROM_ADDRESS, 4, &orig) != 4) {
+            error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name);
+            return;
+        }
+    } else {
+        if (pread(fd, &orig, 4, offset) != 4 ||
+            pwrite(fd, &size, 4, offset) != 4 ||
+            pread(fd, &size, 4, offset) != 4 ||
+            pwrite(fd, &orig, 4, offset) != 4) {
+            error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name);
+            return;
+        }
     }
 
     size = ~(le32_to_cpu(size) & PCI_ROM_ADDRESS_MASK) + 1;
@@ -1123,8 +1145,14 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len)
     if (~emu_bits & (0xffffffffU >> (32 - len * 8))) {
         ssize_t ret;
 
-        ret = pread(vdev->vbasedev.fd, &phys_val, len,
-                    vdev->config_offset + addr);
+        if (vdev->vbasedev.proxy != NULL) {
+            ret = vfio_user_region_read(&vdev->vbasedev,
+                                        VFIO_PCI_CONFIG_REGION_INDEX,
+                                        addr, len, &phys_val);
+        } else {
+            ret = pread(vdev->vbasedev.fd, &phys_val, len,
+                        vdev->config_offset + addr);
+        }
         if (ret != len) {
             error_report("%s(%s, 0x%x, 0x%x) failed: %m",
                          __func__, vdev->vbasedev.name, addr, len);
@@ -1145,12 +1173,20 @@ void vfio_pci_write_config(PCIDevice *pdev,
 {
     VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev);
     uint32_t val_le = cpu_to_le32(val);
+    int ret;
 
     trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len);
 
     /* Write everything to VFIO, let it filter out what we can't write */
-    if (pwrite(vdev->vbasedev.fd, &val_le, len, vdev->config_offset + addr)
-                != len) {
+    if (vdev->vbasedev.proxy != NULL) {
+        ret = vfio_user_region_write(&vdev->vbasedev,
+                                     VFIO_PCI_CONFIG_REGION_INDEX,
+                                     addr, len, &val_le);
+    } else {
+        ret = pwrite(vdev->vbasedev.fd, &val_le, len,
+                     vdev->config_offset + addr);
+    }
+    if (ret != len) {
         error_report("%s(%s, 0x%x, 0x%x, 0x%x) failed: %m",
                      __func__, vdev->vbasedev.name, addr, val, len);
     }
@@ -1240,10 +1276,15 @@ static int vfio_msi_setup(VFIOPCIDevice *vdev, int pos, Error **errp)
     int ret, entries;
     Error *err = NULL;
 
-    if (pread(vdev->vbasedev.fd, &ctrl, sizeof(ctrl),
-              vdev->config_offset + pos + PCI_CAP_FLAGS) != sizeof(ctrl)) {
-        error_setg_errno(errp, errno, "failed reading MSI PCI_CAP_FLAGS");
-        return -errno;
+    if (vdev->vbasedev.proxy != NULL) {
+        /* during setup, config space was initialized from remote */
+        memcpy(&ctrl, vdev->pdev.config + pos + PCI_CAP_FLAGS, sizeof(ctrl));
+    } else {
+        if (pread(vdev->vbasedev.fd, &ctrl, sizeof(ctrl),
+                  vdev->config_offset + pos + PCI_CAP_FLAGS) != sizeof(ctrl)) {
+            error_setg_errno(errp, errno, "failed reading MSI PCI_CAP_FLAGS");
+            return -errno;
+        }
     }
     ctrl = le16_to_cpu(ctrl);
 
@@ -1456,22 +1497,30 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp)
         return;
     }
 
-    if (pread(fd, &ctrl, sizeof(ctrl),
-              vdev->config_offset + pos + PCI_MSIX_FLAGS) != sizeof(ctrl)) {
-        error_setg_errno(errp, errno, "failed to read PCI MSIX FLAGS");
-        return;
-    }
+    if (vdev->vbasedev.proxy != NULL) {
+        /* during setup, config space was initialized from remote */
+        memcpy(&ctrl, vdev->pdev.config + pos + PCI_MSIX_FLAGS, sizeof(ctrl));
+        memcpy(&table, vdev->pdev.config + pos + PCI_MSIX_TABLE, sizeof(table));
+        memcpy(&pba, vdev->pdev.config + pos + PCI_MSIX_PBA, sizeof(pba));
+    } else {
+        if (pread(fd, &ctrl, sizeof(ctrl),
+                  vdev->config_offset + pos + PCI_MSIX_FLAGS) != sizeof(ctrl)) {
+            error_setg_errno(errp, errno, "failed to read PCI MSIX FLAGS");
+            return;
+        }
 
-    if (pread(fd, &table, sizeof(table),
-              vdev->config_offset + pos + PCI_MSIX_TABLE) != sizeof(table)) {
-        error_setg_errno(errp, errno, "failed to read PCI MSIX TABLE");
-        return;
-    }
+        if (pread(fd, &table, sizeof(table),
+                  vdev->config_offset + pos +
+                  PCI_MSIX_TABLE) != sizeof(table)) {
+            error_setg_errno(errp, errno, "failed to read PCI MSIX TABLE");
+            return;
+        }
 
-    if (pread(fd, &pba, sizeof(pba),
-              vdev->config_offset + pos + PCI_MSIX_PBA) != sizeof(pba)) {
-        error_setg_errno(errp, errno, "failed to read PCI MSIX PBA");
-        return;
+        if (pread(fd, &pba, sizeof(pba),
+                  vdev->config_offset + pos + PCI_MSIX_PBA) != sizeof(pba)) {
+            error_setg_errno(errp, errno, "failed to read PCI MSIX PBA");
+            return;
+        }
     }
 
     ctrl = le16_to_cpu(ctrl);
@@ -1619,11 +1668,17 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr)
     }
 
     /* Determine what type of BAR this is for registration */
-    ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar),
-                vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr));
-    if (ret != sizeof(pci_bar)) {
-        error_report("vfio: Failed to read BAR %d (%m)", nr);
-        return;
+    if (vdev->vbasedev.proxy != NULL) {
+        /* during setup, config space was initialized from remote */
+        memcpy(&pci_bar, vdev->pdev.config + PCI_BASE_ADDRESS_0 + (4 * nr),
+               sizeof(pci_bar));
+    } else {
+        ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar),
+                    vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr));
+        if (ret != sizeof(pci_bar)) {
+            error_report("vfio: Failed to read BAR %d (%m)", nr);
+            return;
+        }
     }
 
     pci_bar = le32_to_cpu(pci_bar);
@@ -3423,6 +3478,91 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         goto error;
     }
 
+    /* Get a copy of config space */
+    ret = vfio_user_region_read(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, 0,
+                                MIN(pci_config_size(pdev), vdev->config_size),
+                                pdev->config);
+    if (ret < (int)MIN(pci_config_size(&vdev->pdev), vdev->config_size)) {
+        error_setg_errno(errp, -ret, "failed to read device config space");
+        goto error;
+    }
+
+    /* vfio emulates a lot for us, but some bits need extra love */
+    vdev->emulated_config_bits = g_malloc0(vdev->config_size);
+
+    /* QEMU can choose to expose the ROM or not */
+    memset(vdev->emulated_config_bits + PCI_ROM_ADDRESS, 0xff, 4);
+    /* QEMU can also add or extend BARs */
+    memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4);
+    vdev->vendor_id = pci_get_word(pdev->config + PCI_VENDOR_ID);
+    vdev->device_id = pci_get_word(pdev->config + PCI_DEVICE_ID);
+
+    /* QEMU can change multi-function devices to single function, or reverse */
+    vdev->emulated_config_bits[PCI_HEADER_TYPE] =
+                                              PCI_HEADER_TYPE_MULTI_FUNCTION;
+
+    /* Restore or clear multifunction, this is always controlled by QEMU */
+    if (vdev->pdev.cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
+        vdev->pdev.config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION;
+    } else {
+        vdev->pdev.config[PCI_HEADER_TYPE] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION;
+    }
+
+    /*
+     * Clear host resource mapping info.  If we choose not to register a
+     * BAR, such as might be the case with the option ROM, we can get
+     * confusing, unwritable, residual addresses from the host here.
+     */
+    memset(&vdev->pdev.config[PCI_BASE_ADDRESS_0], 0, 24);
+    memset(&vdev->pdev.config[PCI_ROM_ADDRESS], 0, 4);
+
+    vfio_pci_size_rom(vdev);
+
+    vfio_bars_prepare(vdev);
+
+    vfio_msix_early_setup(vdev, &err);
+    if (err) {
+        error_propagate(errp, err);
+        goto error;
+    }
+
+    vfio_bars_register(vdev);
+
+    ret = vfio_add_capabilities(vdev, errp);
+    if (ret) {
+        goto out_teardown;
+    }
+
+    /* QEMU emulates all of MSI & MSIX */
+    if (pdev->cap_present & QEMU_PCI_CAP_MSIX) {
+        memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff,
+               MSIX_CAP_LENGTH);
+    }
+
+    if (pdev->cap_present & QEMU_PCI_CAP_MSI) {
+        memset(vdev->emulated_config_bits + pdev->msi_cap, 0xff,
+               vdev->msi_cap_size);
+    }
+
+    if (vdev->pdev.config[PCI_INTERRUPT_PIN] != 0) {
+        vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL,
+                                             vfio_intx_mmap_enable, vdev);
+        pci_device_set_intx_routing_notifier(&vdev->pdev,
+                                             vfio_intx_routing_notifier);
+        vdev->irqchip_change_notifier.notify = vfio_irqchip_change;
+        kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier);
+        ret = vfio_intx_enable(vdev, errp);
+        if (ret) {
+            goto out_deregister;
+        }
+    }
+
+out_deregister:
+    pci_device_set_intx_routing_notifier(&vdev->pdev, NULL);
+    kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);
+out_teardown:
+    vfio_teardown_msi(vdev);
+    vfio_bars_exit(vdev);
 error:
     vfio_user_disconnect(proxy);
     error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name);
Signature · 1 line
2.25.1
Reviewed-by on v2 10/16
#49 lore raw

Re: [PATCH RFC v2 10/16] vfio-user: pci_user_realize PCI setup

Quoted 48 lines · Elena Ufimtseva
hw/vfio/pci.c +44 -0
@@ -3423,6 +3478,91 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         goto error;
     }
 
+    /* Get a copy of config space */
+    ret = vfio_user_region_read(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, 0,
+                                MIN(pci_config_size(pdev), vdev->config_size),
+                                pdev->config);
+    if (ret < (int)MIN(pci_config_size(&vdev->pdev), vdev->config_size)) {
+        error_setg_errno(errp, -ret, "failed to read device config space");
+        goto error;
+    }
+
+    /* vfio emulates a lot for us, but some bits need extra love */
+    vdev->emulated_config_bits = g_malloc0(vdev->config_size);
+
+    /* QEMU can choose to expose the ROM or not */
+    memset(vdev->emulated_config_bits + PCI_ROM_ADDRESS, 0xff, 4);
+    /* QEMU can also add or extend BARs */
+    memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4);
+    vdev->vendor_id = pci_get_word(pdev->config + PCI_VENDOR_ID);
+    vdev->device_id = pci_get_word(pdev->config + PCI_DEVICE_ID);
+
+    /* QEMU can change multi-function devices to single function, or reverse */
+    vdev->emulated_config_bits[PCI_HEADER_TYPE] =
+                                              PCI_HEADER_TYPE_MULTI_FUNCTION;
+
+    /* Restore or clear multifunction, this is always controlled by QEMU */
+    if (vdev->pdev.cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
+        vdev->pdev.config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION;
+    } else {
+        vdev->pdev.config[PCI_HEADER_TYPE] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION;
+    }
+
+    /*
+     * Clear host resource mapping info.  If we choose not to register a
+     * BAR, such as might be the case with the option ROM, we can get
+     * confusing, unwritable, residual addresses from the host here.
+     */
+    memset(&vdev->pdev.config[PCI_BASE_ADDRESS_0], 0, 24);
+    memset(&vdev->pdev.config[PCI_ROM_ADDRESS], 0, 4);
+
+    vfio_pci_size_rom(vdev);
+
+    vfio_bars_prepare(vdev);
+
+    vfio_msix_early_setup(vdev, &err);
+    if (err) {

hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3534

Quoted 10 lines
hw/vfio/pci.c +10 -0
+        error_propagate(errp, err);
+        goto error;
+    }
+
+    vfio_bars_register(vdev);
+
+    ret = vfio_add_capabilities(vdev, errp);
+    if (ret) {
+        goto out_teardown;
+    }

I haven't audited the common code to find places where the contents of the PCI Configuration Space are trusted. Input validation may need to be performed on offsets and other inputs that we read from the device.

Otherwise:

#50 lore raw

RFC PATCH v2 11/16 vfio-user: get and set IRQs

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h |  25 ++++++++++
 hw/vfio/user.h          |   2 +
 hw/vfio/common.c        |  26 ++++++++--
 hw/vfio/pci.c           |  31 ++++++++++--
 hw/vfio/user.c          | 106 ++++++++++++++++++++++++++++++++++++++++
 5 files changed, 181 insertions(+), 9 deletions(-)
hw/vfio/user-protocol.h +25 -0
index 56904cf872..5614efa0a4 100644
@@ -109,6 +109,31 @@ typedef struct {
     uint64_t offset;
 } VFIOUserRegionInfo;
 
+/*
+ * VFIO_USER_DEVICE_GET_IRQ_INFO
+ * imported from struct vfio_irq_info
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t count;
+} VFIOUserIRQInfo;
+
+/*
+ * VFIO_USER_DEVICE_SET_IRQS
+ * imported from struct vfio_irq_set
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t start;
+    uint32_t count;
+} VFIOUserIRQSet;
+
 /*
  * VFIO_USER_REGION_READ
  * VFIO_USER_REGION_WRITE
hw/vfio/user.h +2 -0
index 02f832a173..248ad80943 100644
@@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 int vfio_user_get_info(VFIODevice *vbasedev);
 int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
                               struct vfio_region_info *info, VFIOUserFDs *fds);
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info);
+int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq);
 int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
                           uint32_t count, void *data);
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
hw/vfio/common.c +21 -5
index a8b1ea9358..9fe3e05dc6 100644
@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index)
         .count = 0,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
         .count = 1,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
         .count = 1,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 static inline const char *action_to_str(int action)
@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex,
     pfd = (int32_t *)&irq_set->data;
     *pfd = fd;
 
-    if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
-        ret = -errno;
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_set_irqs(vbasedev, irq_set);
+    } else {
+        if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
+            ret = -errno;
+        }
     }
     g_free(irq_set);
 
hw/vfio/pci.c +27 -4
index ea0df8be65..282de6a30b 100644
@@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix)
         fds[i] = fd;
     }
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    if (vdev->vbasedev.proxy != NULL) {
+        ret = vfio_user_set_irqs(&vdev->vbasedev, irq_set);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    }
 
     g_free(irq_set);
 
@@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
 
     irq_info.index = VFIO_PCI_ERR_IRQ_INDEX;
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_get_irq_info(vbasedev, &irq_info);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    }
+
+
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
         trace_vfio_populate_device_get_irq_info_failure(strerror(errno));
@@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)
         return;
     }
 
-    if (ioctl(vdev->vbasedev.fd,
-              VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) {
+    if (vdev->vbasedev.proxy != NULL) {
+        if (vfio_user_get_irq_info(&vdev->vbasedev, &irq_info) < 0) {
+            return;
+        }
+    } else {
+        if (ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0) {
+            return;
+        }
+    }
+    if (irq_info.count < 1) {
         return;
     }
 
@@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         }
     }
 
+    vfio_register_err_notifier(vdev);
+    vfio_register_req_notifier(vdev);
+
+    return;
+
 out_deregister:
     pci_device_set_intx_routing_notifier(&vdev->pdev, NULL);
     kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);
hw/vfio/user.c +106 -0

112 lines not on this page · raw message

Signature · 1 line
2.25.1
#51 lore raw

Re: [PATCH RFC v2 11/16] vfio-user: get and set IRQs

Quoted 197 lines · Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h |  25 ++++++++++
 hw/vfio/user.h          |   2 +
 hw/vfio/common.c        |  26 ++++++++--
 hw/vfio/pci.c           |  31 ++++++++++--
 hw/vfio/user.c          | 106 ++++++++++++++++++++++++++++++++++++++++
 5 files changed, 181 insertions(+), 9 deletions(-)
hw/vfio/user-protocol.h +25 -0
index 56904cf872..5614efa0a4 100644
@@ -109,6 +109,31 @@ typedef struct {
     uint64_t offset;
 } VFIOUserRegionInfo;
 
+/*
+ * VFIO_USER_DEVICE_GET_IRQ_INFO
+ * imported from struct vfio_irq_info
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t count;
+} VFIOUserIRQInfo;
+
+/*
+ * VFIO_USER_DEVICE_SET_IRQS
+ * imported from struct vfio_irq_set
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t start;
+    uint32_t count;
+} VFIOUserIRQSet;
+
 /*
  * VFIO_USER_REGION_READ
  * VFIO_USER_REGION_WRITE
hw/vfio/user.h +2 -0
index 02f832a173..248ad80943 100644
@@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 int vfio_user_get_info(VFIODevice *vbasedev);
 int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
                               struct vfio_region_info *info, VFIOUserFDs *fds);
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info);
+int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq);
 int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
                           uint32_t count, void *data);
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
hw/vfio/common.c +21 -5
index a8b1ea9358..9fe3e05dc6 100644
@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index)
         .count = 0,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
         .count = 1,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
         .count = 1,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 static inline const char *action_to_str(int action)
@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex,
     pfd = (int32_t *)&irq_set->data;
     *pfd = fd;
 
-    if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
-        ret = -errno;
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_set_irqs(vbasedev, irq_set);
+    } else {
+        if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
+            ret = -errno;
+        }
     }
     g_free(irq_set);
 
hw/vfio/pci.c +27 -4
index ea0df8be65..282de6a30b 100644
@@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix)
         fds[i] = fd;
     }
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    if (vdev->vbasedev.proxy != NULL) {
+        ret = vfio_user_set_irqs(&vdev->vbasedev, irq_set);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    }
 
     g_free(irq_set);
 
@@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
 
     irq_info.index = VFIO_PCI_ERR_IRQ_INDEX;
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_get_irq_info(vbasedev, &irq_info);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    }
+
+
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
         trace_vfio_populate_device_get_irq_info_failure(strerror(errno));
@@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)
         return;
     }
 
-    if (ioctl(vdev->vbasedev.fd,
-              VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) {
+    if (vdev->vbasedev.proxy != NULL) {
+        if (vfio_user_get_irq_info(&vdev->vbasedev, &irq_info) < 0) {
+            return;
+        }
+    } else {
+        if (ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0) {
+            return;
+        }
+    }
+    if (irq_info.count < 1) {
         return;
     }
 
@@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         }
     }
 
+    vfio_register_err_notifier(vdev);
+    vfio_register_req_notifier(vdev);
+
+    return;
+
 out_deregister:
     pci_device_set_intx_routing_notifier(&vdev->pdev, NULL);
     kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);
hw/vfio/user.c +6 -0
index 83235b2411..b68ca1279d 100644
@@ -768,6 +768,112 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
     return 0;
 }
 
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info)
+{
+    VFIOUserIRQInfo msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO,

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 786

Quoted 10 lines
hw/vfio/user.c +10 -0
+                          sizeof(msg), 0);
+    msg.argsz = info->argsz;
+    msg.index = info->index;
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0);
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
+    memcpy(info, &msg.argsz, sizeof(*info));

Should this be info.count = msg.count instead? Not sure why argsz is used here.

Also, I just noticed the lack of endianness conversion in this patch series. The spec says values are little-endian but these patches mostly use host-endian. Did I miss something?

#52 lore raw

Re: [PATCH RFC v2 11/16] vfio-user: get and set IRQs

Quoted 207 lines · Stefan Hajnoczi
Quoted 202 lines · Elena Ufimtseva

From: John Johnson <[email protected]>

---

hw/vfio/user-protocol.h |  25 ++++++++++
hw/vfio/user.h          |   2 +
hw/vfio/common.c        |  26 ++++++++--
hw/vfio/pci.c           |  31 ++++++++++--
hw/vfio/user.c          | 106 ++++++++++++++++++++++++++++++++++++++++
5 files changed, 181 insertions(+), 9 deletions(-)
hw/vfio/user-protocol.h +25 -0
index 56904cf872..5614efa0a4 100644
@@ -109,6 +109,31 @@ typedef struct {
    uint64_t offset;
 } VFIOUserRegionInfo;
 
+/*
+ * VFIO_USER_DEVICE_GET_IRQ_INFO
+ * imported from struct vfio_irq_info
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t count;
+} VFIOUserIRQInfo;
+
+/*
+ * VFIO_USER_DEVICE_SET_IRQS
+ * imported from struct vfio_irq_set
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t start;
+    uint32_t count;
+} VFIOUserIRQSet;
+
 /*
 * VFIO_USER_REGION_READ
 * VFIO_USER_REGION_WRITE
hw/vfio/user.h +2 -0
index 02f832a173..248ad80943 100644
@@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 int vfio_user_get_info(VFIODevice *vbasedev);
 int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
                              struct vfio_region_info *info, VFIOUserFDs *fds);
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info);
+int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq);
 int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
                          uint32_t count, void *data);
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
hw/vfio/common.c +21 -5
index a8b1ea9358..9fe3e05dc6 100644
@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index)
        .count = 0,
    };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
        .count = 1,
    };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
        .count = 1,
    };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 static inline const char *action_to_str(int action)
@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex,
    pfd = (int32_t *)&irq_set->data;
    *pfd = fd;
 
-    if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
-        ret = -errno;
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_set_irqs(vbasedev, irq_set);
+    } else {
+        if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
+            ret = -errno;
+        }
    }
    g_free(irq_set);
 
hw/vfio/pci.c +27 -4
index ea0df8be65..282de6a30b 100644
@@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix)
        fds[i] = fd;
    }
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    if (vdev->vbasedev.proxy != NULL) {
+        ret = vfio_user_set_irqs(&vdev->vbasedev, irq_set);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    }
 
    g_free(irq_set);
 
@@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
 
    irq_info.index = VFIO_PCI_ERR_IRQ_INDEX;
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_get_irq_info(vbasedev, &irq_info);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    }
+
+
    if (ret) {
        /* This can fail for an old kernel or legacy PCI dev */
        trace_vfio_populate_device_get_irq_info_failure(strerror(errno));
@@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)
        return;
    }
 
-    if (ioctl(vdev->vbasedev.fd,
-              VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) {
+    if (vdev->vbasedev.proxy != NULL) {
+        if (vfio_user_get_irq_info(&vdev->vbasedev, &irq_info) < 0) {
+            return;
+        }
+    } else {
+        if (ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0) {
+            return;
+        }
+    }
+    if (irq_info.count < 1) {
        return;
    }
 
@@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
        }
    }
 
+    vfio_register_err_notifier(vdev);
+    vfio_register_req_notifier(vdev);
+
+    return;
+

out_deregister:

    pci_device_set_intx_routing_notifier(&vdev->pdev, NULL);
    kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);
hw/vfio/user.c +11 -0
index 83235b2411..b68ca1279d 100644
@@ -768,6 +768,112 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
    return 0;
 }
 
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info)
+{
+    VFIOUserIRQInfo msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO,
+                          sizeof(msg), 0);
+    msg.argsz = info->argsz;
+    msg.index = info->index;
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0);
Quoted 8 lines

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 786

Quoted 5 lines
hw/vfio/user.c +5 -0
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
+    memcpy(info, &msg.argsz, sizeof(*info));

Should this be info.count = msg.count instead? Not sure why argsz is used here.

It’s copying the entire returned vfio_irq_info struct, which starts
at &msg.argsz.

Quoted 4 lines

Also, I just noticed the lack of endianness conversion in this patch series. The spec says values are little-endian but these patches mostly use host-endian. Did I miss something?

I had thought we were using host endianness for UNIX sockets and
were deferring the cross endianness issue to when we support TCP, but the spec does say all LE.

JJ

#53 lore raw

Re: [PATCH RFC v2 11/16] vfio-user: get and set IRQs

Quoted 216 lines · John Johnson
Quoted 207 lines · Stefan Hajnoczi
Quoted 202 lines · Elena Ufimtseva

From: John Johnson <[email protected]>

---

hw/vfio/user-protocol.h |  25 ++++++++++
hw/vfio/user.h          |   2 +
hw/vfio/common.c        |  26 ++++++++--
hw/vfio/pci.c           |  31 ++++++++++--
hw/vfio/user.c          | 106 ++++++++++++++++++++++++++++++++++++++++
5 files changed, 181 insertions(+), 9 deletions(-)
hw/vfio/user-protocol.h +25 -0
index 56904cf872..5614efa0a4 100644
@@ -109,6 +109,31 @@ typedef struct {
    uint64_t offset;
 } VFIOUserRegionInfo;
 
+/*
+ * VFIO_USER_DEVICE_GET_IRQ_INFO
+ * imported from struct vfio_irq_info
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t count;
+} VFIOUserIRQInfo;
+
+/*
+ * VFIO_USER_DEVICE_SET_IRQS
+ * imported from struct vfio_irq_set
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t start;
+    uint32_t count;
+} VFIOUserIRQSet;
+
 /*
 * VFIO_USER_REGION_READ
 * VFIO_USER_REGION_WRITE
hw/vfio/user.h +2 -0
index 02f832a173..248ad80943 100644
@@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 int vfio_user_get_info(VFIODevice *vbasedev);
 int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
                              struct vfio_region_info *info, VFIOUserFDs *fds);
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info);
+int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq);
 int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
                          uint32_t count, void *data);
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
hw/vfio/common.c +21 -5
index a8b1ea9358..9fe3e05dc6 100644
@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index)
        .count = 0,
    };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
        .count = 1,
    };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
        .count = 1,
    };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 static inline const char *action_to_str(int action)
@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex,
    pfd = (int32_t *)&irq_set->data;
    *pfd = fd;
 
-    if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
-        ret = -errno;
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_set_irqs(vbasedev, irq_set);
+    } else {
+        if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
+            ret = -errno;
+        }
    }
    g_free(irq_set);
 
hw/vfio/pci.c +27 -4
index ea0df8be65..282de6a30b 100644
@@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix)
        fds[i] = fd;
    }
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    if (vdev->vbasedev.proxy != NULL) {
+        ret = vfio_user_set_irqs(&vdev->vbasedev, irq_set);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    }
 
    g_free(irq_set);
 
@@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
 
    irq_info.index = VFIO_PCI_ERR_IRQ_INDEX;
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_get_irq_info(vbasedev, &irq_info);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    }
+
+
    if (ret) {
        /* This can fail for an old kernel or legacy PCI dev */
        trace_vfio_populate_device_get_irq_info_failure(strerror(errno));
@@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)
        return;
    }
 
-    if (ioctl(vdev->vbasedev.fd,
-              VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) {
+    if (vdev->vbasedev.proxy != NULL) {
+        if (vfio_user_get_irq_info(&vdev->vbasedev, &irq_info) < 0) {
+            return;
+        }
+    } else {
+        if (ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0) {
+            return;
+        }
+    }
+    if (irq_info.count < 1) {
        return;
    }
 
@@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
        }
    }
 
+    vfio_register_err_notifier(vdev);
+    vfio_register_req_notifier(vdev);
+
+    return;
+

out_deregister:

    pci_device_set_intx_routing_notifier(&vdev->pdev, NULL);
    kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier);
hw/vfio/user.c +11 -0
index 83235b2411..b68ca1279d 100644
@@ -768,6 +768,112 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
    return 0;
 }
 
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info)
+{
+    VFIOUserIRQInfo msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO,
+                          sizeof(msg), 0);
+    msg.argsz = info->argsz;
+    msg.index = info->index;
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0);
Quoted 8 lines

hw/vfio/user.c int vfio_user_get_region_info(VFIODevice *vbasedev, int index, · line 786

Quoted 5 lines
hw/vfio/user.c +5 -0
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
+    memcpy(info, &msg.argsz, sizeof(*info));

Should this be info.count = msg.count instead? Not sure why argsz is used here.

Quoted 2 lines

It’s copying the entire returned vfio_irq_info struct, which starts
at &msg.argsz.

That makes sense, I missed it. Thanks!

Stefan

#54 lore raw

RFC PATCH v2 12/16 vfio-user: proxy container connect/disconnect

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 include/hw/vfio/vfio-common.h |  3 ++
 hw/vfio/common.c              | 84 +++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 | 22 +++++++++
 3 files changed, 109 insertions(+)
include/hw/vfio/vfio-common.h +3 -0
index bdd25a546c..688660c28d 100644
@@ -91,6 +91,7 @@ typedef struct VFIOContainer {
     uint64_t max_dirty_bitmap_size;
     unsigned long pgsizes;
     unsigned int dma_max_mappings;
+    VFIOProxy *proxy;
     QLIST_HEAD(, VFIOGuestIOMMU) giommu_list;
     QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list;
     QLIST_HEAD(, VFIOGroup) group_list;
@@ -217,6 +218,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp);
 void vfio_put_group(VFIOGroup *group);
 int vfio_get_device(VFIOGroup *group, const char *name,
                     VFIODevice *vbasedev, Error **errp);
+void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as);
+void vfio_disconnect_proxy(VFIOGroup *group);
 
 extern const MemoryRegionOps vfio_region_ops;
 typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList;
hw/vfio/common.c +84 -0

96 lines not on this page · raw message

hw/vfio/pci.c +22 -0
index 282de6a30b..2c9fcb2fa9 100644
@@ -3442,6 +3442,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     VFIODevice *vbasedev = &vdev->vbasedev;
     SocketAddress addr;
     VFIOProxy *proxy;
+    VFIOGroup *group = NULL;
     int ret;
     Error *err = NULL;
 
@@ -3484,6 +3485,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     vbasedev->no_mmap = false;
     vbasedev->ops = &vfio_user_pci_ops;
 
+    /*
+     * each device gets its own group and container
+     * make them unrelated to any host IOMMU groupings
+     */
+    group = g_malloc0(sizeof(*group));
+    group->fd = -1;
+    group->groupid = -1;
+    QLIST_INIT(&group->device_list);
+    QLIST_INSERT_HEAD(&group->device_list, vbasedev, next);
+    vbasedev->group = group;
+
+    vfio_connect_proxy(proxy, group, pci_device_iommu_address_space(pdev));
+
     ret = vfio_user_get_info(&vdev->vbasedev);
     if (ret) {
         error_setg_errno(errp, -ret, "get info failure");
@@ -3587,6 +3601,9 @@ out_teardown:
     vfio_teardown_msi(vdev);
     vfio_bars_exit(vdev);
 error:
+    if (group != NULL) {
+        vfio_disconnect_proxy(group);
+    }
     vfio_user_disconnect(proxy);
     error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name);
 }
@@ -3595,6 +3612,11 @@ static void vfio_user_instance_finalize(Object *obj)
 {
     VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj);
     VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIOGroup *group = vbasedev->group;
+
+    vfio_disconnect_proxy(group);
+    g_free(group);
+    vbasedev->group = NULL;
 
     vfio_put_device(vdev);
 
Signature · 1 line
2.25.1
#55 lore raw

Re: [PATCH RFC v2 12/16] vfio-user: proxy container connect/disconnect

Quoted 10 lines · Elena Ufimtseva

From: John Johnson <[email protected]>

---
 include/hw/vfio/vfio-common.h |  3 ++
 hw/vfio/common.c              | 84 +++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 | 22 +++++++++
 3 files changed, 109 insertions(+)

Alex: I'm not familiar enough with hw/vfio/ to review this in depth. You might have suggestions on how to unify the vfio-user and vfio kernel concepts of groups and containers.

Quoted 167 lines
include/hw/vfio/vfio-common.h +3 -0
index bdd25a546c..688660c28d 100644
@@ -91,6 +91,7 @@ typedef struct VFIOContainer {
     uint64_t max_dirty_bitmap_size;
     unsigned long pgsizes;
     unsigned int dma_max_mappings;
+    VFIOProxy *proxy;
     QLIST_HEAD(, VFIOGuestIOMMU) giommu_list;
     QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list;
     QLIST_HEAD(, VFIOGroup) group_list;
@@ -217,6 +218,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp);
 void vfio_put_group(VFIOGroup *group);
 int vfio_get_device(VFIOGroup *group, const char *name,
                     VFIODevice *vbasedev, Error **errp);
+void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as);
+void vfio_disconnect_proxy(VFIOGroup *group);
 
 extern const MemoryRegionOps vfio_region_ops;
 typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList;
hw/vfio/common.c +84 -0
index 9fe3e05dc6..57b9e111e6 100644
@@ -2249,6 +2249,55 @@ put_space_exit:
     return ret;
 }
 
+void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as)
+{
+    VFIOAddressSpace *space;
+    VFIOContainer *container;
+
+    if (QLIST_EMPTY(&vfio_group_list)) {
+        qemu_register_reset(vfio_reset_handler, NULL);
+    }
+
+    QLIST_INSERT_HEAD(&vfio_group_list, group, next);
+
+    /*
+     * try to mirror vfio_connect_container()
+     * as much as possible
+     */
+
+    space = vfio_get_address_space(as);
+
+    container = g_malloc0(sizeof(*container));
+    container->space = space;
+    container->fd = -1;
+    QLIST_INIT(&container->giommu_list);
+    QLIST_INIT(&container->hostwin_list);
+    container->proxy = proxy;
+
+    /*
+     * The proxy uses a SW IOMMU in lieu of the HW one
+     * used in the ioctl() version.  Use TYPE1 with the
+     * target's page size for maximum capatibility
+     */
+    container->iommu_type = VFIO_TYPE1_IOMMU;
+    vfio_host_win_add(container, 0, (hwaddr)-1, TARGET_PAGE_SIZE);
+    container->pgsizes = TARGET_PAGE_SIZE;
+
+    container->dirty_pages_supported = true;
+    container->max_dirty_bitmap_size = VFIO_USER_DEF_MAX_XFER;
+    container->dirty_pgsizes = TARGET_PAGE_SIZE;
+
+    QLIST_INIT(&container->group_list);
+    QLIST_INSERT_HEAD(&space->containers, container, next);
+
+    group->container = container;
+    QLIST_INSERT_HEAD(&container->group_list, group, container_next);
+
+    container->listener = vfio_memory_listener;
+    memory_listener_register(&container->listener, container->space->as);
+    container->initialized = true;
+}
+
 static void vfio_disconnect_container(VFIOGroup *group)
 {
     VFIOContainer *container = group->container;
@@ -2291,6 +2340,41 @@ static void vfio_disconnect_container(VFIOGroup *group)
     }
 }
 
+void vfio_disconnect_proxy(VFIOGroup *group)
+{
+    VFIOContainer *container = group->container;
+    VFIOAddressSpace *space = container->space;
+    VFIOGuestIOMMU *giommu, *tmp;
+
+    /*
+     * try to mirror vfio_disconnect_container()
+     * as much as possible, knowing each device
+     * is in one group and one container
+     */
+
+    QLIST_REMOVE(group, container_next);
+    group->container = NULL;
+
+    /*
+     * Explicitly release the listener first before unset container,
+     * since unset may destroy the backend container if it's the last
+     * group.
+     */
+    memory_listener_unregister(&container->listener);
+
+    QLIST_REMOVE(container, next);
+
+    QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, tmp) {
+        memory_region_unregister_iommu_notifier(
+            MEMORY_REGION(giommu->iommu), &giommu->n);
+        QLIST_REMOVE(giommu, giommu_next);
+        g_free(giommu);
+    }
+
+    g_free(container);
+    vfio_put_address_space(space);
+}
+
 VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp)
 {
     VFIOGroup *group;
hw/vfio/pci.c +17 -0
index 282de6a30b..2c9fcb2fa9 100644
@@ -3442,6 +3442,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     VFIODevice *vbasedev = &vdev->vbasedev;
     SocketAddress addr;
     VFIOProxy *proxy;
+    VFIOGroup *group = NULL;
     int ret;
     Error *err = NULL;
 
@@ -3484,6 +3485,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     vbasedev->no_mmap = false;
     vbasedev->ops = &vfio_user_pci_ops;
 
+    /*
+     * each device gets its own group and container
+     * make them unrelated to any host IOMMU groupings
+     */
+    group = g_malloc0(sizeof(*group));
+    group->fd = -1;
+    group->groupid = -1;
+    QLIST_INIT(&group->device_list);
+    QLIST_INSERT_HEAD(&group->device_list, vbasedev, next);
+    vbasedev->group = group;
+
+    vfio_connect_proxy(proxy, group, pci_device_iommu_address_space(pdev));
+
     ret = vfio_user_get_info(&vdev->vbasedev);
     if (ret) {
         error_setg_errno(errp, -ret, "get info failure");
@@ -3587,6 +3601,9 @@ out_teardown:
     vfio_teardown_msi(vdev);
     vfio_bars_exit(vdev);
 error:
+    if (group != NULL) {
+        vfio_disconnect_proxy(group);
+    }
     vfio_user_disconnect(proxy);
     error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name);
 }

hw/vfio/pci.c static void vfio_user_instance_finalize(Object *obj) · line 3619

Quoted 8 lines
hw/vfio/pci.c +5 -0
@@ -3595,6 +3612,11 @@ static void vfio_user_instance_finalize(Object *obj)
 {
     VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj);
     VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIOGroup *group = vbasedev->group;
+
+    vfio_disconnect_proxy(group);
+    g_free(group);
+    vbasedev->group = NULL;

Can vfio_put_group() be used instead? I'm worried that the cleanup code will be duplicated or become inconsistent if it's not shared.

Also, vfio_instance_finalize() calls vfio_put_group() after vfio_put_device(). Does this code intentionally take advantage of the if (!vbasedev->group) early return in vfio_put_base_device()? This is non-obvious. I recommend unifying the device and group cleanup instead of special-casing it here (this is fragile!).

Quoted 6 lines
hw/vfio/pci.c +0 -0
     vfio_put_device(vdev);
 
Signature · 1 line
2.25.1
#56 lore raw

RFC PATCH v2 13/16 vfio-user: dma map/unmap operations

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h       |  32 +++++++++
 hw/vfio/user.h                |   6 ++
 include/hw/vfio/vfio-common.h |   1 +
 hw/vfio/common.c              |  71 ++++++++++++++++---
 hw/vfio/user.c                | 124 ++++++++++++++++++++++++++++++++++
 5 files changed, 226 insertions(+), 8 deletions(-)
hw/vfio/user-protocol.h +32 -0
index 5614efa0a4..ca53fce5f4 100644
@@ -82,6 +82,31 @@ typedef struct {
 #define VFIO_USER_MAX_MAX_XFER  (64 * 1024 * 1024)
 
 
+/*
+ * VFIO_USER_DMA_MAP
+ * imported from struct vfio_iommu_type1_dma_map
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint64_t offset;    /* FD offset */
+    uint64_t iova;
+    uint64_t size;
+} VFIOUserDMAMap;
+
+/*
+ * VFIO_USER_DMA_UNMAP
+ * imported from struct vfio_iommu_type1_dma_unmap
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint64_t iova;
+    uint64_t size;
+} VFIOUserDMAUnmap;
+
 /*
  * VFIO_USER_DEVICE_GET_INFO
  * imported from struct_device_info
@@ -146,4 +171,11 @@ typedef struct {
     char data[];
 } VFIOUserRegionRW;
 
+/*imported from struct vfio_bitmap */
+typedef struct {
+    uint64_t pgsize;
+    uint64_t size;
+    char data[];
+} VFIOUserBitmap;
+
 #endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +6 -0
index 248ad80943..7786ab57c5 100644
@@ -71,6 +71,11 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev,
                                              void *reqarg);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
+int vfio_user_dma_map(VFIOProxy *proxy, struct vfio_iommu_type1_dma_map *map,
+                      VFIOUserFDs *fds, bool will_commit);
+int vfio_user_dma_unmap(VFIOProxy *proxy,
+                        struct vfio_iommu_type1_dma_unmap *unmap,
+                        struct vfio_bitmap *bitmap, bool will_commit);
 int vfio_user_get_info(VFIODevice *vbasedev);
 int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
                               struct vfio_region_info *info, VFIOUserFDs *fds);
@@ -80,5 +85,6 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
                           uint32_t count, void *data);
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
                            uint64_t offset, uint32_t count, void *data);
+void vfio_user_drain_reqs(VFIOProxy *proxy);
 
 #endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +1 -0
index 688660c28d..13d1d14c3b 100644
@@ -87,6 +87,7 @@ typedef struct VFIOContainer {
     Error *error;
     bool initialized;
     bool dirty_pages_supported;
+    bool will_commit;
     uint64_t dirty_pgsizes;
     uint64_t max_dirty_bitmap_size;
     unsigned long pgsizes;
hw/vfio/common.c +63 -8

172 lines not on this page · raw message

hw/vfio/user.c +124 -0

136 lines not on this page · raw message

Signature · 1 line
2.25.1
#57 lore raw

Re: [PATCH RFC v2 13/16] vfio-user: dma map/unmap operations

Quoted 29 lines · Elena Ufimtseva
hw/vfio/user.c +29 -0
+void vfio_user_drain_reqs(VFIOProxy *proxy)
+{
+    VFIOUserReply *reply;
+    bool iolock = 0;
+
+    /*
+     * Any DMA map/unmap requests sent in the middle
+     * of a memory region transaction were sent async.
+     * Wait for them here.
+     */
+    QEMU_LOCK_GUARD(&proxy->lock);
+    if (proxy->last_nowait != NULL) {
+        iolock = qemu_mutex_iothread_locked();
+        if (iolock) {
+            qemu_mutex_unlock_iothread();
+        }
+
+        reply = proxy->last_nowait;
+        reply->nowait = 0;
+        while (reply->complete == 0) {
+            if (!qemu_cond_timedwait(&reply->cv, &proxy->lock, wait_time)) {
+                error_printf("vfio_drain_reqs - timed out\n");
+                break;
+            }
+        }
+
+        if (reply->msg->flags & VFIO_USER_ERROR) {
+            error_printf("vfio_user_rcv error reply on async request ");
+            error_printf("command %x error %s\n", reply->msg->command,

hw/vfio/user.c static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg, · line 449

Quoted 10 lines
hw/vfio/user.c +10 -0
+                         strerror(reply->msg->error_reply));
+        }
+        proxy->last_nowait = NULL;
+        g_free(reply->msg);
+        QTAILQ_INSERT_HEAD(&proxy->free, reply, next);
+    }
+
+    if (iolock) {
+        qemu_mutex_lock_iothread();
+    }

Not sure this lock ordering is correct. Above we acquire proxy->lock while holding the BQL and here we acquire the BQL while holding proxy->lock. If another thread (e.g. a vCPU thread) does something similar this is the ABBA lock ordering problem.

The more obviously correct way to write this is:

  WITH_QEMU_LOCK_GUARD(&proxy->lock) {
      ...
  }

  if (iolock) {
      qemu_mutex_lock_iothread();
  }

hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 762

Quoted 13 lines
hw/vfio/user.c +6 -0
+}
+
 static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd,
                                   uint32_t size, uint32_t flags)
 {
@@ -715,6 +756,89 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp)
     return 0;
 }
 
+int vfio_user_dma_map(VFIOProxy *proxy, struct vfio_iommu_type1_dma_map *map,
+                      VFIOUserFDs *fds, bool will_commit)
+{
+    VFIOUserDMAMap *msgp = g_malloc(sizeof(*msgp));

Is this zero-initialized anywhere to guarantee that QEMU memory isn't exposed to the VFIO device emulation program?

hw/vfio/user.c int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) · line 766

Quoted 4 lines
hw/vfio/user.c +4 -0
+    int ret, flags;
+
+    /* commit will wait, so send async without dropping BQL */
+    flags = will_commit ? (NOIOLOCK | NOWAIT) : 0;

Why is this distinction between will_commit and !will_commit necessary? I get a sense that the network communications code drops the BQL and that's undesirable here for some reason. I wonder why the code doesn't take the NOWAIT code path all the time?

#58 lore raw

RFC PATCH v2 14/16 vfio-user: dma read/write operations

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h | 11 +++++++
 hw/vfio/user.h          |  1 +
 hw/vfio/pci.c           | 63 ++++++++++++++++++++++++++++++++++++++++-
 hw/vfio/user.c          |  7 ++++-
 4 files changed, 80 insertions(+), 2 deletions(-)
hw/vfio/user-protocol.h +11 -0
index ca53fce5f4..c5d9473f8f 100644
@@ -171,6 +171,17 @@ typedef struct {
     char data[];
 } VFIOUserRegionRW;
 
+/*
+ * VFIO_USER_DMA_READ
+ * VFIO_USER_DMA_WRITE
+ */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint64_t offset;
+    uint32_t count;
+    char data[];
+} VFIOUserDMARW;
+
 /*imported from struct vfio_bitmap */
 typedef struct {
     uint64_t pgsize;
hw/vfio/user.h +1 -0
index 7786ab57c5..32e8b70d28 100644
@@ -65,6 +65,7 @@ typedef struct VFIOProxy {
 
 VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp);
 void vfio_user_disconnect(VFIOProxy *proxy);
+uint64_t vfio_user_max_xfer(void);
 void vfio_user_set_reqhandler(VFIODevice *vbasdev,
                               int (*handler)(void *opaque, char *buf,
                                              VFIOUserFDs *fds),
hw/vfio/pci.c +62 -1

73 lines not on this page · raw message

hw/vfio/user.c +6 -1
index 06bcd46e60..fcc041959c 100644
@@ -54,6 +54,11 @@ static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg,
  * Functions called by main, CPU, or iothread threads
  */
 
+uint64_t vfio_user_max_xfer(void)
+{
+    return max_xfer_size;
+}
+
 static void vfio_user_shutdown(VFIOProxy *proxy)
 {
     qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL);
@@ -251,7 +256,7 @@ void vfio_user_recv(void *opaque)
         *reply->msg = msg;
         data = (char *)reply->msg + sizeof(msg);
     } else {
-        if (msg.size > max_xfer_size) {
+        if (msg.size > max_xfer_size + sizeof(VFIOUserDMARW)) {
             error_setg(&local_err, "vfio_user_recv request larger than max");
             goto fatal;
         }
Signature · 1 line
2.25.1
#59 lore raw

Re: [PATCH RFC v2 14/16] vfio-user: dma read/write operations

Quoted 5 lines · Elena Ufimtseva
hw/vfio/pci.c +0 -0
index 2c9fcb2fa9..29a874c066 100644

hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3413

Quoted 9 lines
hw/vfio/pci.c +4 -1
@@ -3406,11 +3406,72 @@ type_init(register_vfio_pci_dev_type)
  * vfio-user routines.
  */
 
-static int vfio_user_pci_process_req(void *opaque, char *buf, VFIOUserFDs *fds)
+static int vfio_user_dma_read(VFIOPCIDevice *vdev, VFIOUserDMARW *msg)
 {
+    PCIDevice *pdev = &vdev->pdev;
+    char *buf;
+    int size = msg->count + sizeof(VFIOUserDMARW);

The caller has only checked that hdr->size is large enough for VFIOUserHdr, not VFIOUserDMARW. We must not access VFIOUserDMARW fields until this has been checked.

Size should be size_t to avoid signedness issues.

Even then, this can overflow on 32-bit hosts so I suggest moving this arithmetic expression below the msg->count > vfio_user_max_xfer() check. That way it's clear that overflow cannot happen.

hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3420

Quoted 7 lines
hw/vfio/pci.c +7 -0
+
+    if (msg->hdr.flags & VFIO_USER_NO_REPLY) {
+        return -EINVAL;
+    }
+    if (msg->count > vfio_user_max_xfer()) {
+        return -E2BIG;
+    }

Does vfio-user allow the request to be smaller than the reply? In other words, is it okay that we're not checking msg->count against hdr->size?

hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3425

Quoted 5 lines
hw/vfio/pci.c +5 -0
+
+    buf = g_malloc0(size);
+    memcpy(buf, msg, sizeof(*msg));
+
+    pci_dma_read(pdev, msg->offset, buf + sizeof(*msg), msg->count);

The vfio-user spec doesn't go into errors but pci_dma_read() can return errors. Hmm...

hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3436

Quoted 11 lines
hw/vfio/pci.c +8 -0
+
+    vfio_user_send_reply(vdev->vbasedev.proxy, buf, size);
+    g_free(buf);
     return 0;
 }
 
+static int vfio_user_dma_write(VFIOPCIDevice *vdev,
+                               VFIOUserDMARW *msg)
+{
+    PCIDevice *pdev = &vdev->pdev;
+    char *buf = (char *)msg + sizeof(*msg);

Or:

  char *buf = msg->data;

hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3441

Quoted 5 lines
hw/vfio/pci.c +5 -0
+
+    /* make sure transfer count isn't larger than the message data */
+    if (msg->count > msg->hdr.size - sizeof(*msg)) {
+        return -E2BIG;
+    }

msg->count cannot be accessed until we have checked that msg->hdr.size is large enough for VFIOUserDMARW. Adding the check also eliminates the underflow in the subtraction if msg->hdr.size was smaller than sizeof(VFIOUserDMARW).

Quoted 8 lines
hw/vfio/pci.c +8 -0
+
+    pci_dma_write(pdev, msg->offset, buf, msg->count);
+
+    if ((msg->hdr.flags & VFIO_USER_NO_REPLY) == 0) {
+        vfio_user_send_reply(vdev->vbasedev.proxy, (char *)msg,
+                             sizeof(msg->hdr));
+    }
+    return 0;

hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3459

Quoted 10 lines
hw/vfio/pci.c +10 -0
+}
+
+static int vfio_user_pci_process_req(void *opaque, char *buf, VFIOUserFDs *fds)
+{
+    VFIOPCIDevice *vdev = opaque;
+    VFIOUserHdr *hdr = (VFIOUserHdr *)buf;
+    int ret;
+
+    if (fds->recv_fds != 0) {
+        return -EINVAL;

Where are the fds closed?

#60 lore raw

Re: [PATCH RFC v2 14/16] vfio-user: dma read/write operations

Quoted 8 lines · Stefan Hajnoczi

hw/vfio/pci.c type_init(register_vfio_pci_dev_type) · line 3425

Quoted 5 lines
hw/vfio/pci.c +5 -0
+
+    buf = g_malloc0(size);
+    memcpy(buf, msg, sizeof(*msg));
+
+    pci_dma_read(pdev, msg->offset, buf + sizeof(*msg), msg->count);

The vfio-user spec doesn't go into errors but pci_dma_read() can return errors. Hmm...

It's certainly under-specified in the spec, but in terms of the library, we do return EINVAL if we decide something invalid happened...

regards
john

#61 lore raw

RFC PATCH v2 15/16 vfio-user: pci reset

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user.h |  1 +
 hw/vfio/pci.c  | 29 ++++++++++++++++++++++++++---
 hw/vfio/user.c | 12 ++++++++++++
 3 files changed, 39 insertions(+), 3 deletions(-)
hw/vfio/user.h +1 -0
index 32e8b70d28..5d4d0a43ba 100644
@@ -86,6 +86,7 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
                           uint32_t count, void *data);
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
                            uint64_t offset, uint32_t count, void *data);
+void vfio_user_reset(VFIODevice *vbasedev);
 void vfio_user_drain_reqs(VFIOProxy *proxy);
 
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +26 -3
index 29a874c066..4b933ed10f 100644
@@ -2229,8 +2229,9 @@ static void vfio_pci_pre_reset(VFIOPCIDevice *vdev)
 
 static void vfio_pci_post_reset(VFIOPCIDevice *vdev)
 {
+    VFIODevice *vbasedev = &vdev->vbasedev;
     Error *err = NULL;
-    int nr;
+    int ret, nr;
 
     vfio_intx_enable(vdev, &err);
     if (err) {
@@ -2238,11 +2239,18 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev)
     }
 
     for (nr = 0; nr < PCI_NUM_REGIONS - 1; ++nr) {
-        off_t addr = vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr);
+        off_t addr = PCI_BASE_ADDRESS_0 + (4 * nr);
         uint32_t val = 0;
         uint32_t len = sizeof(val);
 
-        if (pwrite(vdev->vbasedev.fd, &val, len, addr) != len) {
+        if (vbasedev->proxy != NULL) {
+            ret = vfio_user_region_write(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX,
+                                         addr, len, &val);
+        } else {
+            ret = pwrite(vdev->vbasedev.fd, &val, len,
+                         vdev->config_offset + addr);
+        }
+        if (ret != len) {
             error_report("%s(%s) reset bar %d failed: %m", __func__,
                          vdev->vbasedev.name, nr);
         }
@@ -3684,6 +3692,20 @@ static void vfio_user_instance_finalize(Object *obj)
     vfio_user_disconnect(vbasedev->proxy);
 }
 
+static void vfio_user_pci_reset(DeviceState *dev)
+{
+    VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev);
+    VFIODevice *vbasedev = &vdev->vbasedev;
+
+    vfio_pci_pre_reset(vdev);
+
+    if (vbasedev->reset_works) {
+        vfio_user_reset(vbasedev);
+    }
+
+    vfio_pci_post_reset(vdev);
+}
+
 static Property vfio_user_pci_dev_properties[] = {
     DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),
     DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false),
@@ -3695,6 +3717,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data)
     DeviceClass *dc = DEVICE_CLASS(klass);
     PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass);
 
+    dc->reset = vfio_user_pci_reset;
     device_class_set_props(dc, vfio_user_pci_dev_properties);
     dc->desc = "VFIO over socket PCI device assignment";
     pdc->realize = vfio_user_pci_realize;
hw/vfio/user.c +12 -0
index fcc041959c..7de2125346 100644
@@ -1045,3 +1045,15 @@ int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
 
     return count;
 }
+
+void vfio_user_reset(VFIODevice *vbasedev)
+{
+    VFIOUserHdr msg;
+
+    vfio_user_request_msg(&msg, VFIO_USER_DEVICE_RESET, sizeof(msg), 0);
+
+    vfio_user_send_recv(vbasedev->proxy, &msg, NULL, 0, 0);
+    if (msg.flags & VFIO_USER_ERROR) {
+        error_printf("reset reply error %d\n", msg.error_reply);
+    }
+}
Signature · 1 line
2.25.1
#62 lore raw

Re: [PATCH RFC v2 15/16] vfio-user: pci reset

Quoted 48 lines · Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user.h |  1 +
 hw/vfio/pci.c  | 29 ++++++++++++++++++++++++++---
 hw/vfio/user.c | 12 ++++++++++++
 3 files changed, 39 insertions(+), 3 deletions(-)
hw/vfio/user.h +1 -0
index 32e8b70d28..5d4d0a43ba 100644
@@ -86,6 +86,7 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
                           uint32_t count, void *data);
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
                            uint64_t offset, uint32_t count, void *data);
+void vfio_user_reset(VFIODevice *vbasedev);
 void vfio_user_drain_reqs(VFIOProxy *proxy);
 
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +3 -3
index 29a874c066..4b933ed10f 100644
@@ -2229,8 +2229,9 @@ static void vfio_pci_pre_reset(VFIOPCIDevice *vdev)
 
 static void vfio_pci_post_reset(VFIOPCIDevice *vdev)
 {
+    VFIODevice *vbasedev = &vdev->vbasedev;
     Error *err = NULL;
-    int nr;
+    int ret, nr;
 
     vfio_intx_enable(vdev, &err);
     if (err) {
@@ -2238,11 +2239,18 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev)
     }
 
     for (nr = 0; nr < PCI_NUM_REGIONS - 1; ++nr) {
-        off_t addr = vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr);
+        off_t addr = PCI_BASE_ADDRESS_0 + (4 * nr);
         uint32_t val = 0;
         uint32_t len = sizeof(val);
 
-        if (pwrite(vdev->vbasedev.fd, &val, len, addr) != len) {

hw/vfio/pci.c static void vfio_pci_post_reset(VFIOPCIDevice *vdev) · line 2255

Quoted 10 lines
hw/vfio/pci.c +8 -0
+        if (vbasedev->proxy != NULL) {
+            ret = vfio_user_region_write(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX,
+                                         addr, len, &val);
+        } else {
+            ret = pwrite(vdev->vbasedev.fd, &val, len,
+                         vdev->config_offset + addr);
+        }
+        if (ret != len) {
             error_report("%s(%s) reset bar %d failed: %m", __func__,
                          vdev->vbasedev.name, nr);

The %m format string assumes vfio_user_region_write() sets errno. I don't think it does. We're relying on vfio_user_region_write() never failing here, which is true at the moment but not nice.

Quoted 35 lines
hw/vfio/pci.c +15 -0
         }
@@ -3684,6 +3692,20 @@ static void vfio_user_instance_finalize(Object *obj)
     vfio_user_disconnect(vbasedev->proxy);
 }
 
+static void vfio_user_pci_reset(DeviceState *dev)
+{
+    VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev);
+    VFIODevice *vbasedev = &vdev->vbasedev;
+
+    vfio_pci_pre_reset(vdev);
+
+    if (vbasedev->reset_works) {
+        vfio_user_reset(vbasedev);
+    }
+
+    vfio_pci_post_reset(vdev);
+}
+
 static Property vfio_user_pci_dev_properties[] = {
     DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),
     DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false),
@@ -3695,6 +3717,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data)
     DeviceClass *dc = DEVICE_CLASS(klass);
     PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass);
 
+    dc->reset = vfio_user_pci_reset;
     device_class_set_props(dc, vfio_user_pci_dev_properties);
     dc->desc = "VFIO over socket PCI device assignment";
     pdc->realize = vfio_user_pci_realize;
hw/vfio/user.c +0 -0
index fcc041959c..7de2125346 100644

hw/vfio/user.c int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, · line 1051

Quoted 7 lines
hw/vfio/user.c +4 -0
@@ -1045,3 +1045,15 @@ int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
 
     return count;
 }
+
+void vfio_user_reset(VFIODevice *vbasedev)
+{
+    VFIOUserHdr msg;

Maybe add "= {}" to ensure it's zero-initialized?

Quoted 11 lines
hw/vfio/user.c +8 -0
+
+    vfio_user_request_msg(&msg, VFIO_USER_DEVICE_RESET, sizeof(msg), 0);
+
+    vfio_user_send_recv(vbasedev->proxy, &msg, NULL, 0, 0);
+    if (msg.flags & VFIO_USER_ERROR) {
+        error_printf("reset reply error %d\n", msg.error_reply);
+    }
+}
Signature · 1 line
2.25.1
#63 lore raw

RFC PATCH v2 16/16 vfio-user: migration support

In reply to Elena Ufimtseva

From: John Johnson <[email protected]>

---
 hw/vfio/user-protocol.h | 18 +++++++++++++++++
 hw/vfio/user.h          |  3 +++
 hw/vfio/common.c        | 23 ++++++++++++++++-----
 hw/vfio/migration.c     | 34 +++++++++++++++++--------------
 hw/vfio/pci.c           | 12 +++++++++++
 hw/vfio/user.c          | 45 +++++++++++++++++++++++++++++++++++++++++
 6 files changed, 115 insertions(+), 20 deletions(-)
hw/vfio/user-protocol.h +18 -0
index c5d9473f8f..bad067a570 100644
@@ -182,6 +182,10 @@ typedef struct {
     char data[];
 } VFIOUserDMARW;
 
+/*
+ * VFIO_USER_DIRTY_PAGES
+ */
+
 /*imported from struct vfio_bitmap */
 typedef struct {
     uint64_t pgsize;
@@ -189,4 +193,18 @@ typedef struct {
     char data[];
 } VFIOUserBitmap;
 
+/* imported from struct vfio_iommu_type1_dirty_bitmap_get */
+typedef struct {
+    uint64_t iova;
+    uint64_t size;
+    VFIOUserBitmap bitmap;
+} VFIOUserBitmapRange;
+
+/* imported from struct vfio_iommu_type1_dirty_bitmap */
+typedef struct {
+    VFIOUserHdr hdr;
+    uint32_t argsz;
+    uint32_t flags;
+} VFIOUserDirtyPages;
+
 #endif /* VFIO_USER_PROTOCOL_H */
hw/vfio/user.h +3 -0
index 5d4d0a43ba..905e0ee28d 100644
@@ -87,6 +87,9 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
                            uint64_t offset, uint32_t count, void *data);
 void vfio_user_reset(VFIODevice *vbasedev);
+int vfio_user_dirty_bitmap(VFIOProxy *proxy,
+                           struct vfio_iommu_type1_dirty_bitmap *bitmap,
+                           struct vfio_iommu_type1_dirty_bitmap_get *range);
 void vfio_user_drain_reqs(VFIOProxy *proxy);
 
 #endif /* VFIO_USER_H */
hw/vfio/common.c +18 -5
index a532e52bcf..09d0147df2 100644
@@ -1303,10 +1303,19 @@ static void vfio_set_dirty_page_tracking(VFIOContainer *container, bool start)
         dirty.flags = VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP;
     }
 
-    ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty);
-    if (ret) {
-        error_report("Failed to set dirty tracking flag 0x%x errno: %d",
-                     dirty.flags, errno);
+    if (container->proxy != NULL) {
+        ret = vfio_user_dirty_bitmap(container->proxy, &dirty, NULL);
+        if (ret) {
+            error_report("Failed to set dirty tracking flag 0x%x errno: %d",
+                         dirty.flags, -ret);
+        }
+    } else {
+        ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty);
+        if (ret) {
+            error_report("Failed to set dirty tracking flag 0x%x errno: %d",
+                         dirty.flags, errno);
+            ret = -errno;
+        }
     }
 }
 
@@ -1356,7 +1365,11 @@ static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova,
         goto err_out;
     }
 
-    ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap);
+    if (container->proxy != NULL) {
+        ret = vfio_user_dirty_bitmap(container->proxy, dbitmap, range);
+    } else {
+        ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap);
+    }
     if (ret) {
         error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64
                 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova,
hw/vfio/migration.c +19 -15
index 82f654afb6..89926a3b01 100644
@@ -27,6 +27,7 @@
 #include "pci.h"
 #include "trace.h"
 #include "hw/hw.h"
+#include "user.h"
 
 /*
  * Flags to be used as unique delimiters for VFIO devices in the migration
@@ -49,10 +50,18 @@ static int64_t bytes_transferred;
 static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count,
                                   off_t off, bool iswrite)
 {
+    VFIORegion *region = &vbasedev->migration->region;
     int ret;
 
-    ret = iswrite ? pwrite(vbasedev->fd, val, count, off) :
-                    pread(vbasedev->fd, val, count, off);
+    if (vbasedev->proxy != NULL) {
+        ret = iswrite ?
+            vfio_user_region_write(vbasedev, region->nr, off, count, val) :
+            vfio_user_region_read(vbasedev, region->nr, off, count, val);
+    } else {
+        off += region->fd_offset;
+        ret = iswrite ? pwrite(vbasedev->fd, val, count, off) :
+                        pread(vbasedev->fd, val, count, off);
+    }
     if (ret < count) {
         error_report("vfio_mig_%s %d byte %s: failed at offset 0x%"
                      HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count,
@@ -111,9 +120,7 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask,
                                     uint32_t value)
 {
     VFIOMigration *migration = vbasedev->migration;
-    VFIORegion *region = &migration->region;
-    off_t dev_state_off = region->fd_offset +
-                          VFIO_MIG_STRUCT_OFFSET(device_state);
+    off_t dev_state_off = VFIO_MIG_STRUCT_OFFSET(device_state);
     uint32_t device_state;
     int ret;
 
@@ -201,13 +208,13 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size)
     int ret;
 
     ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset),
-                      region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset));
+                        VFIO_MIG_STRUCT_OFFSET(data_offset));
     if (ret < 0) {
         return ret;
     }
 
     ret = vfio_mig_read(vbasedev, &data_size, sizeof(data_size),
-                        region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size));
+                        VFIO_MIG_STRUCT_OFFSET(data_size));
     if (ret < 0) {
         return ret;
     }
@@ -233,8 +240,7 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size)
             }
             buf_allocated = true;
 
-            ret = vfio_mig_read(vbasedev, buf, sec_size,
-                                region->fd_offset + data_offset);
+            ret = vfio_mig_read(vbasedev, buf, sec_size, data_offset);
             if (ret < 0) {
                 g_free(buf);
                 return ret;
@@ -269,7 +275,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
 
     do {
         ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset),
-                      region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset));
+                            VFIO_MIG_STRUCT_OFFSET(data_offset));
         if (ret < 0) {
             return ret;
         }
@@ -309,8 +315,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
             qemu_get_buffer(f, buf, sec_size);
 
             if (buf_alloc) {
-                ret = vfio_mig_write(vbasedev, buf, sec_size,
-                        region->fd_offset + data_offset);
+                ret = vfio_mig_write(vbasedev, buf, sec_size, data_offset);
                 g_free(buf);
 
                 if (ret < 0) {
@@ -322,7 +327,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
         }
 
         ret = vfio_mig_write(vbasedev, &report_size, sizeof(report_size),
-                        region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size));
+                             VFIO_MIG_STRUCT_OFFSET(data_size));
         if (ret < 0) {
             return ret;
         }
@@ -334,12 +339,11 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
 static int vfio_update_pending(VFIODevice *vbasedev)
 {
     VFIOMigration *migration = vbasedev->migration;
-    VFIORegion *region = &migration->region;
     uint64_t pending_bytes = 0;
     int ret;
 
     ret = vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes),
-                    region->fd_offset + VFIO_MIG_STRUCT_OFFSET(pending_bytes));
+                        VFIO_MIG_STRUCT_OFFSET(pending_bytes));
     if (ret < 0) {
         migration->pending_bytes = 0;
         return ret;
hw/vfio/pci.c +12 -0
index 4b933ed10f..976fb89786 100644
@@ -3658,6 +3658,13 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         }
     }
 
+    if (!pdev->failover_pair_id) {
+        ret = vfio_migration_probe(&vdev->vbasedev, errp);
+        if (ret) {
+            error_report("%s: Migration disabled", vdev->vbasedev.name);
+        }
+    }
+
     vfio_register_err_notifier(vdev);
     vfio_register_req_notifier(vdev);
 
@@ -3709,6 +3716,11 @@ static void vfio_user_pci_reset(DeviceState *dev)
 static Property vfio_user_pci_dev_properties[] = {
     DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),
     DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false),
+    DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
+                     vbasedev.enable_migration, false),
+    DEFINE_PROP_ON_OFF_AUTO("x-pre-copy-dirty-page-tracking", VFIOPCIDevice,
+                            vbasedev.pre_copy_dirty_page_tracking,
+                            ON_OFF_AUTO_ON),
     DEFINE_PROP_END_OF_LIST(),
 };
 
hw/vfio/user.c +45 -0

48 lines not on this page · raw message

Signature · 1 line
2.25.1
#64 lore raw

Re: [PATCH RFC v2 16/16] vfio-user: migration support

hw/vfio/common.c static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, · line 1375

Quoted 13 lines · Elena Ufimtseva
hw/vfio/common.c +5 -1
@@ -1356,7 +1365,11 @@ static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova,
         goto err_out;
     }
 
-    ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap);
+    if (container->proxy != NULL) {
+        ret = vfio_user_dirty_bitmap(container->proxy, dbitmap, range);
+    } else {
+        ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap);
+    }
     if (ret) {
         error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64
                 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova,

This error_report() relies on errno. vfio_user_region_write() doesn't set errno.

Quoted 23 lines
hw/vfio/migration.c +4 -2
index 82f654afb6..89926a3b01 100644
@@ -27,6 +27,7 @@
 #include "pci.h"
 #include "trace.h"
 #include "hw/hw.h"
+#include "user.h"
 
 /*
  * Flags to be used as unique delimiters for VFIO devices in the migration
@@ -49,10 +50,18 @@ static int64_t bytes_transferred;
 static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count,
                                   off_t off, bool iswrite)
 {
+    VFIORegion *region = &vbasedev->migration->region;
     int ret;
 
-    ret = iswrite ? pwrite(vbasedev->fd, val, count, off) :
-                    pread(vbasedev->fd, val, count, off);
+    if (vbasedev->proxy != NULL) {
+        ret = iswrite ?

hw/vfio/migration.c static int64_t bytes_transferred; · line 67

Quoted 10 lines
hw/vfio/migration.c +7 -0
+            vfio_user_region_write(vbasedev, region->nr, off, count, val) :
+            vfio_user_region_read(vbasedev, region->nr, off, count, val);
+    } else {
+        off += region->fd_offset;
+        ret = iswrite ? pwrite(vbasedev->fd, val, count, off) :
+                        pread(vbasedev->fd, val, count, off);
+    }
     if (ret < count) {
         error_report("vfio_mig_%s %d byte %s: failed at offset 0x%"
                      HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count,

Another errno user. I haven't exhaustively audited all the code for these issues. Please take a look.

#65 lore raw

v2 cover vfio-user server in QEMU

In reply to Elena Ufimtseva

Hi,

This series depends on the following series from
Elena Ufimtseva <[email protected]>:
[PATCH RFC v2 00/16] vfio-user implementation

Thank you for your feedback for the v1 patches!
https://www.mail-archive.com/[email protected]/msg825021.html

We have incorporated the following feedback from v1 of the review cycle:

[PATCH RFC server v2 01/11] vfio-user: build library
- Using cmake subproject to build libvfio-user

[PATCH RFC server v2 02/11] vfio-user: define vfio-user object
- Added check to confirm that TYPE_REMOTE_MACHINE is used with TYPE_VFU_OBJECT

[PATCH RFC server v2 04/11] vfio-user: find and init PCI device
- Removed call to vfu_pci_set_id()
- Added check to confirm that TYPE_PCI_DEVICE is used with TYPE_VFU_OBJECT

[PATCH RFC server v2 05/11] vfio-user: run vfio-user context
- Using QEMU main-loop to drive the vfu_ctx (using

    vfu_get_poll_fd() & qemu_set_fd_handler())

- Set vfu_ctx to non-blocking mode (LIBVFIO_USER_FLAG_ATTACH_NB)
- Modified how QEMU attaches to the vfu_ctx

[PATCH RFC server v2 06/11] handle PCI config space accesses
- Broke-up PCI config space access to 4-byte accesses

[PATCH RFC server v2 07/11] vfio-user: handle DMA mappings
- Received feedback to assert that vfu_dma_info_t->vaddr is not NULL - unable to do it as it appears to be a valid case.

[PATCH RFC server v2 10/11] register handlers to facilitate migration
- Migrate only one device's data per contect

Would appreciate if you could kindly review this v2 series. Looking forward to your comments.

Thank you!

Jagannathan Raman (11):
vfio-user: build library
vfio-user: define vfio-user object
vfio-user: instantiate vfio-user context
vfio-user: find and init PCI device
vfio-user: run vfio-user context
vfio-user: handle PCI config space accesses
vfio-user: handle DMA mappings
vfio-user: handle PCI BAR accesses
vfio-user: handle device interrupts
vfio-user: register handlers to facilitate migration
vfio-user: acceptance test

 configure                     |  11 +
 meson.build                   |  28 ++
 qapi/qom.json                 |  20 +-
 include/hw/remote/iohub.h     |   2 +
 migration/savevm.h            |   2 +
 hw/remote/iohub.c             |   5 +
 hw/remote/vfio-user-obj.c     | 803 ++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c            |  73 ++++
 .gitmodules                   |   3 +
 MAINTAINERS                   |   9 +
 hw/remote/meson.build         |   3 +
 hw/remote/trace-events        |  10 +
 subprojects/libvfio-user      |   1 +
 tests/acceptance/vfio-user.py |  94 +++++
 14 files changed, 1062 insertions(+), 2 deletions(-)
 create mode 100644 hw/remote/vfio-user-obj.c
 create mode 160000 subprojects/libvfio-user
 create mode 100644 tests/acceptance/vfio-user.py
Signature · 1 line
1.8.3.1
#66 lore raw

RFC PATCH v2 1/11 vfio-user: build library

add the libvfio-user library as a submodule. build it as a cmake subproject.

---
 configure                | 11 +++++++++++
 meson.build              | 28 ++++++++++++++++++++++++++++
 .gitmodules              |  3 +++
 MAINTAINERS              |  7 +++++++
 hw/remote/meson.build    |  2 ++
 subprojects/libvfio-user |  1 +
 6 files changed, 52 insertions(+)
 create mode 160000 subprojects/libvfio-user
configure +11 -0
index 9a79a00..794e900 100755
@@ -4291,6 +4291,17 @@ but not implemented on your system"
 fi
 
 ##########################################
+# check for multiprocess
+
+case "$multiprocess" in
+  auto | enabled )
+    if test "$git_submodules_action" != "ignore"; then
+      git_submodules="${git_submodules} libvfio-user"
+    fi
+    ;;
+esac
+
+##########################################
 # End of CC checks
 # After here, no more $cc or $ld runs
 
meson.build +28 -0
index bf63784..2b2d5c2 100644
@@ -1898,6 +1898,34 @@ if get_option('cfi') and slirp_opt == 'system'
          + ' Please configure with --enable-slirp=git')
 endif
 
+vfiouser = not_found
+if have_system and multiprocess_allowed
+  have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/Makefile')
+
+  if not have_internal
+    error('libvfio-user source not found - please pull git submodule')
+  endif
+
+  json_c = dependency('json-c', required: false)
+    if not json_c.found()
+      json_c = dependency('libjson-c')
+  endif
+
+  cmake = import('cmake')
+
+  vfiouser_subproj = cmake.subproject('libvfio-user')
+
+  vfiouser_sl = vfiouser_subproj.dependency('vfio-user-static')
+
+  # Although cmake links the json-c library with vfio-user-static
+  # target, that info is not available to meson via cmake.subproject.
+  # As such, we have to separately declare the json-c dependency here.
+  # This appears to be a current limitation of using cmake inside meson.
+  # libvfio-user is planning a switch to meson in the future, which
+  # would address this item automatically.
+  vfiouser = declare_dependency(dependencies: [vfiouser_sl, json_c])
+endif
+
 fdt = not_found
 fdt_opt = get_option('fdt')
 if have_system
.gitmodules +3 -0
index 08b1b48..cfeea7c 100644
@@ -64,3 +64,6 @@
 [submodule "roms/vbootrom"]
 	path = roms/vbootrom
 	url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "subprojects/libvfio-user"]
+	path = subprojects/libvfio-user
+	url = https://github.com/nutanix/libvfio-user.git
MAINTAINERS +7 -0
index 4039d3c..0c5a18e 100644
@@ -3361,6 +3361,13 @@ F: semihosting/
 F: include/semihosting/
 F: tests/tcg/multiarch/arm-compat-semi/
 
+libvfio-user Library
+M: Thanos Makatos <[email protected]>
+M: John Levon <[email protected]>
+T: https://github.com/nutanix/libvfio-user.git
+S: Maintained
+F: subprojects/libvfio-user/*
+
 Multi-process QEMU
 M: Elena Ufimtseva <[email protected]>
 M: Jagannathan Raman <[email protected]>
hw/remote/meson.build +2 -0
index e6a5574..fb35fb8 100644
@@ -7,6 +7,8 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
 
+remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: vfiouser)
+
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c'))
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
 
subprojects/libvfio-user new file +1 -0
new file mode 160000
index 0000000..647c934
@@ -0,0 +1 @@
+Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446
Signature · 1 line
1.8.3.1
#67 lore raw

Re: [PATCH RFC server v2 01/11] vfio-user: build library

Quoted 52 lines · Jag Raman

add the libvfio-user library as a submodule. build it as a cmake subproject.

---

configure                | 11 +++++++++++
meson.build              | 28 ++++++++++++++++++++++++++++
.gitmodules              |  3 +++
MAINTAINERS              |  7 +++++++
hw/remote/meson.build    |  2 ++
subprojects/libvfio-user |  1 +
6 files changed, 52 insertions(+)

create mode 160000 subprojects/libvfio-user

configure +0 -0
index 9a79a00..794e900 100755
@@ -4291,6 +4291,17 @@ but not implemented on your system"

fi

##########################################

configure +11 -0
+# check for multiprocess
+
+case "$multiprocess" in
+  auto | enabled )
+    if test "$git_submodules_action" != "ignore"; then
+      git_submodules="${git_submodules} libvfio-user"
+    fi
+    ;;
+esac
+
+##########################################

# End of CC checks
# After here, no more $cc or $ld runs

meson.build +0 -0
index bf63784..2b2d5c2 100644
@@ -1898,6 +1898,34 @@ if get_option('cfi') and slirp_opt == 'system'
         + ' Please configure with --enable-slirp=git')

endif

meson.build +2 -0
+vfiouser = not_found
+if have_system and multiprocess_allowed

meson.build if get_option('cfi') and slirp_opt == 'system' · line 1912

Quoted 10 lines
meson.build +10 -0
+  have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/Makefile')
+
+  if not have_internal
+    error('libvfio-user source not found - please pull git submodule')
+  endif
+
+  json_c = dependency('json-c', required: false)
+    if not json_c.found()
+      json_c = dependency('libjson-c')
+  endif

One of the things we’re wondering is about this json-c package that we need to build libvfio-user library.

The gitlab runners typically don’t have this package installed, as such the gitlab builds fail. Wondering if there's a way to install this package for all QEMU builds?

We checked out the various jobs defined in “.gitlab-ci.d/buildtest.yml” - there is a “before_script” keyword which we could use to install this package. The “before_script” keyword appears to be run every time before a job’s script is executed. But this option appears to be per job/build. Wondering if there's a distro-independent global way to install a required package for all builds.

Thank you!
--
Jag

Quoted 71 lines
meson.build +16 -0
+
+  cmake = import('cmake')
+
+  vfiouser_subproj = cmake.subproject('libvfio-user')
+
+  vfiouser_sl = vfiouser_subproj.dependency('vfio-user-static')
+
+  # Although cmake links the json-c library with vfio-user-static
+  # target, that info is not available to meson via cmake.subproject.
+  # As such, we have to separately declare the json-c dependency here.
+  # This appears to be a current limitation of using cmake inside meson.
+  # libvfio-user is planning a switch to meson in the future, which
+  # would address this item automatically.
+  vfiouser = declare_dependency(dependencies: [vfiouser_sl, json_c])
+endif
+

fdt = not_found
fdt_opt = get_option('fdt')
if have_system

.gitmodules +0 -0
index 08b1b48..cfeea7c 100644
@@ -64,3 +64,6 @@

[submodule "roms/vbootrom"]
path = roms/vbootrom
url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "subprojects/libvfio-user"]

.gitmodules +2 -0
+	path = subprojects/libvfio-user
+	url = https://github.com/nutanix/libvfio-user.git
MAINTAINERS +0 -0
index 4039d3c..0c5a18e 100644
@@ -3361,6 +3361,13 @@ F: semihosting/

F: include/semihosting/
F: tests/tcg/multiarch/arm-compat-semi/

MAINTAINERS +7 -0
+libvfio-user Library
+M: Thanos Makatos <[email protected]>
+M: John Levon <[email protected]>
+T: https://github.com/nutanix/libvfio-user.git
+S: Maintained
+F: subprojects/libvfio-user/*
+

Multi-process QEMU
M: Elena Ufimtseva <[email protected]>
M: Jagannathan Raman <[email protected]>

hw/remote/meson.build +2 -0
index e6a5574..fb35fb8 100644
@@ -7,6 +7,8 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
 
+remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: vfiouser)
+
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c'))
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
 
subprojects/libvfio-user new file +1 -0
new file mode 160000
index 0000000..647c934
@@ -0,0 +1 @@
+Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446
Signature · 1 line
1.8.3.1
#68 lore raw

Re: [PATCH RFC server v2 01/11] vfio-user: build library

In reply to Jagannathan Raman
Quoted 8 lines · Jagannathan Raman
meson.build +0 -0
index bf63784..2b2d5c2 100644
@@ -1898,6 +1898,34 @@ if get_option('cfi') and slirp_opt == 'system'
          + ' Please configure with --enable-slirp=git')
 endif
 

meson.build if get_option('cfi') and slirp_opt == 'system' · line 1910

Quoted 10 lines
meson.build +10 -0
+vfiouser = not_found
+if have_system and multiprocess_allowed
+  have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/Makefile')
+
+  if not have_internal
+    error('libvfio-user source not found - please pull git submodule')
+  endif
+
+  json_c = dependency('json-c', required: false)
+    if not json_c.found()

Indentation is off.

Quoted 26 lines
meson.build +18 -0
+      json_c = dependency('libjson-c')
+  endif
+
+  cmake = import('cmake')
+
+  vfiouser_subproj = cmake.subproject('libvfio-user')
+
+  vfiouser_sl = vfiouser_subproj.dependency('vfio-user-static')
+
+  # Although cmake links the json-c library with vfio-user-static
+  # target, that info is not available to meson via cmake.subproject.
+  # As such, we have to separately declare the json-c dependency here.
+  # This appears to be a current limitation of using cmake inside meson.
+  # libvfio-user is planning a switch to meson in the future, which
+  # would address this item automatically.
+  vfiouser = declare_dependency(dependencies: [vfiouser_sl, json_c])
+endif
+
 fdt = not_found
 fdt_opt = get_option('fdt')
 if have_system
.gitmodules +0 -0
index 08b1b48..cfeea7c 100644

.gitmodules line 69

Quoted 6 lines
.gitmodules +3 -0
@@ -64,3 +64,6 @@
 [submodule "roms/vbootrom"]
 	path = roms/vbootrom
 	url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "subprojects/libvfio-user"]
+	path = subprojects/libvfio-user
+	url = https://github.com/nutanix/libvfio-user.git

Once this is merged I'll set up a gitlab.com/qemu-project/libvfio-user.git mirror. This ensures that no matter what happens with upstream libvfio-user.git, the source code that QEMU builds against will remain archived/available.

Quoted 5 lines
MAINTAINERS +0 -0
index 4039d3c..0c5a18e 100644

MAINTAINERS F: semihosting/ · line 3369

Quoted 9 lines
MAINTAINERS +6 -0
@@ -3361,6 +3361,13 @@ F: semihosting/
 F: include/semihosting/
 F: tests/tcg/multiarch/arm-compat-semi/
 
+libvfio-user Library
+M: Thanos Makatos <[email protected]>
+M: John Levon <[email protected]>
+T: https://github.com/nutanix/libvfio-user.git
+S: Maintained
+F: subprojects/libvfio-user/*

A MAINTAINERS entry isn't necessary for git submodules. This could become outdated. People should look at the upstream project instead for information on maintainership and how to contribute.

#69 lore raw

Re: [PATCH RFC server v2 01/11] vfio-user: build library

Quoted 18 lines · Stefan Hajnoczi

.gitmodules line 69

Quoted 11 lines · Jagannathan Raman
.gitmodules +3 -0
index 08b1b48..cfeea7c 100644
@@ -64,3 +64,6 @@
 [submodule "roms/vbootrom"]
 	path = roms/vbootrom
 	url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "subprojects/libvfio-user"]
+	path = subprojects/libvfio-user
+	url = https://github.com/nutanix/libvfio-user.git

Once this is merged I'll set up a gitlab.com/qemu-project/libvfio-user.git mirror. This ensures that no matter what happens with upstream libvfio-user.git, the source code that QEMU builds against will remain archived/available.

Can we do it the other way around? When the series is OK to be merged, setup the https://gitlab.com/qemu-project/libvfio-user.git mirror and have the submodule point to it?

#70 lore raw

Re: [PATCH RFC server v2 01/11] vfio-user: build library

Quoted 20 lines · Philippe Mathieu-Daudé
Quoted 18 lines · Stefan Hajnoczi

.gitmodules line 69

Quoted 11 lines · Jagannathan Raman
.gitmodules +3 -0
index 08b1b48..cfeea7c 100644
@@ -64,3 +64,6 @@
 [submodule "roms/vbootrom"]
 	path = roms/vbootrom
 	url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "subprojects/libvfio-user"]
+	path = subprojects/libvfio-user
+	url = https://github.com/nutanix/libvfio-user.git

Once this is merged I'll set up a gitlab.com/qemu-project/libvfio-user.git mirror. This ensures that no matter what happens with upstream libvfio-user.git, the source code that QEMU builds against will remain archived/available.

Quoted 3 lines

Can we do it the other way around? When the series is OK to be merged, setup the https://gitlab.com/qemu-project/libvfio-user.git mirror and have the submodule point to it?

Yes, good idea.

Stefan

#71 lore raw

Re: [PATCH RFC server v2 01/11] vfio-user: build library

In reply to Jagannathan Raman
Quoted 23 lines · Jagannathan Raman

add the libvfio-user library as a submodule. build it as a cmake subproject.

---
 configure                | 11 +++++++++++
 meson.build              | 28 ++++++++++++++++++++++++++++
 .gitmodules              |  3 +++
 MAINTAINERS              |  7 +++++++
 hw/remote/meson.build    |  2 ++
 subprojects/libvfio-user |  1 +
 6 files changed, 52 insertions(+)
 create mode 160000 subprojects/libvfio-user
subprojects/libvfio-user new file +0 -0
new file mode 160000
index 0000000..647c934

subprojects/libvfio-user line 1

Quoted 1 line
subprojects/libvfio-user new file +1 -0
@@ -0,0 +1 @@
+Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446

Could we point to a sha1 of a released tag instead?

#72 lore raw

Re: [PATCH RFC server v2 01/11] vfio-user: build library

Quoted 21 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Jagannathan Raman

add the libvfio-user library as a submodule. build it as a cmake subproject.

---

configure                | 11 +++++++++++
meson.build              | 28 ++++++++++++++++++++++++++++
.gitmodules              |  3 +++
MAINTAINERS              |  7 +++++++
hw/remote/meson.build    |  2 ++
subprojects/libvfio-user |  1 +
6 files changed, 52 insertions(+)

create mode 160000 subprojects/libvfio-user

Quoted 10 lines

subprojects/libvfio-user line 1

Quoted 8 lines
subprojects/libvfio-user new file +1 -0
new file mode 160000
index 0000000..647c934
@@ -0,0 +1 @@
+Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446

Could we point to a sha1 of a released tag instead?

OK, will do.

--
Jag

Quoted 1 line
#73 lore raw

Re: [PATCH RFC server v2 01/11] vfio-user: build library

In reply to Philippe Mathieu-Daudé
Quoted 17 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Jagannathan Raman

add the libvfio-user library as a submodule. build it as a cmake subproject.

---
 configure                | 11 +++++++++++
 meson.build              | 28 ++++++++++++++++++++++++++++
 .gitmodules              |  3 +++
 MAINTAINERS              |  7 +++++++
 hw/remote/meson.build    |  2 ++
 subprojects/libvfio-user |  1 +
 6 files changed, 52 insertions(+)
 create mode 160000 subprojects/libvfio-user
Quoted 9 lines

subprojects/libvfio-user line 1

Quoted 7 lines
subprojects/libvfio-user new file +1 -0
new file mode 160000
index 0000000..647c934
@@ -0,0 +1 @@
+Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446

Could we point to a sha1 of a released tag instead?

We don't have releases (yet) partly because we haven't yet stabilized the API.

regards
john

#74 lore raw

Re: [PATCH RFC server v2 01/11] vfio-user: build library

Quoted 28 lines · John Levon
Quoted 17 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Jagannathan Raman

add the libvfio-user library as a submodule. build it as a cmake subproject.

---
 configure                | 11 +++++++++++
 meson.build              | 28 ++++++++++++++++++++++++++++
 .gitmodules              |  3 +++
 MAINTAINERS              |  7 +++++++
 hw/remote/meson.build    |  2 ++
 subprojects/libvfio-user |  1 +
 6 files changed, 52 insertions(+)
 create mode 160000 subprojects/libvfio-user
Quoted 9 lines

subprojects/libvfio-user line 1

Quoted 7 lines
subprojects/libvfio-user new file +1 -0
new file mode 160000
index 0000000..647c934
@@ -0,0 +1 @@
+Subproject commit 647c9341d2e06266a710ddd075f69c95dd3b8446

Could we point to a sha1 of a released tag instead?

Quoted 1 line

We don't have releases (yet) partly because we haven't yet stabilized the API.

OK. Maybe acceptable, up to the maintainer then ¯\_(ツ)_/¯

#75 lore raw

RFC PATCH v2 2/11 vfio-user: define vfio-user object

In reply to Jagannathan Raman

Define vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object

---
 qapi/qom.json             |  20 ++++++-
 hw/remote/vfio-user-obj.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++
 MAINTAINERS               |   1 +
 hw/remote/meson.build     |   1 +
 hw/remote/trace-events    |   3 +
 5 files changed, 168 insertions(+), 2 deletions(-)
 create mode 100644 hw/remote/vfio-user-obj.c
qapi/qom.json +18 -2
index a25616b..3e941ee 100644
@@ -689,6 +689,20 @@
   'data': { 'fd': 'str', 'devid': 'str' } }
 
 ##
+# @VfioUserProperties:
+#
+# Properties for vfio-user objects.
+#
+# @socket: path to be used as socket by the libvfiouser library
+#
+# @devid: the id of the device to be associated with the file descriptor
+#
+# Since: 6.0
+##
+{ 'struct': 'VfioUserProperties',
+  'data': { 'socket': 'str', 'devid': 'str' } }
+
+##
 # @RngProperties:
 #
 # Properties for objects of classes derived from rng.
@@ -812,7 +826,8 @@
     'tls-creds-psk',
     'tls-creds-x509',
     'tls-cipher-suites',
-    'x-remote-object'
+    'x-remote-object',
+    'vfio-user'
   ] }
 
 ##
@@ -868,7 +883,8 @@
       'tls-creds-psk':              'TlsCredsPskProperties',
       'tls-creds-x509':             'TlsCredsX509Properties',
       'tls-cipher-suites':          'TlsCredsProperties',
-      'x-remote-object':            'RemoteObjectProperties'
+      'x-remote-object':            'RemoteObjectProperties',
+      'vfio-user':                  'VfioUserProperties'
   } }
 
 ##
hw/remote/vfio-user-obj.c new file +145 -0

145 lines not on this page · raw message

MAINTAINERS +1 -0
index 0c5a18e..f9d8092 100644
@@ -3391,6 +3391,7 @@ F: hw/remote/proxy-memory-listener.c
 F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
+F: hw/remote/vfio-user-obj.c
 
 EBPF:
 M: Jason Wang <[email protected]>
hw/remote/meson.build +1 -0
index fb35fb8..cd44dfc 100644
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
+remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('vfio-user-obj.c'))
 
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: vfiouser)
 
hw/remote/trace-events +3 -0
index 0b23974..7da12f0 100644
@@ -2,3 +2,6 @@
 
 mpqemu_send_io_error(int cmd, int size, int nfds) "send command %d size %d, %d file descriptors to remote process"
 mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, %d file descriptors to remote process"
+
+# vfio-user-obj.c
+vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
Signature · 1 line
1.8.3.1
#76 lore raw

Re: [PATCH RFC server v2 02/11] vfio-user: define vfio-user object

Quoted 26 lines · Jagannathan Raman

Define vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object

---
 qapi/qom.json             |  20 ++++++-
 hw/remote/vfio-user-obj.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++
 MAINTAINERS               |   1 +
 hw/remote/meson.build     |   1 +
 hw/remote/trace-events    |   3 +
 5 files changed, 168 insertions(+), 2 deletions(-)
 create mode 100644 hw/remote/vfio-user-obj.c
qapi/qom.json +2 -0
index a25616b..3e941ee 100644
@@ -689,6 +689,20 @@
   'data': { 'fd': 'str', 'devid': 'str' } }
 
 ##
+# @VfioUserProperties:
+#

qapi/qom.json line 703

Quoted 10 lines
qapi/qom.json +10 -0
+# Properties for vfio-user objects.
+#
+# @socket: path to be used as socket by the libvfiouser library
+#
+# @devid: the id of the device to be associated with the file descriptor
+#
+# Since: 6.0
+##
+{ 'struct': 'VfioUserProperties',
+  'data': { 'socket': 'str', 'devid': 'str' } }

Please use 'SocketAddress' for socket instead of 'str'. That way file descriptor passing is easy to support and additional socket address families can be supported in the future.

Quoted 16 lines
qapi/qom.json +4 -1
+
+##
 # @RngProperties:
 #
 # Properties for objects of classes derived from rng.
@@ -812,7 +826,8 @@
     'tls-creds-psk',
     'tls-creds-x509',
     'tls-cipher-suites',
-    'x-remote-object'
+    'x-remote-object',
+    'vfio-user'
   ] }
 
 ##

qapi/qom.json line 887

Quoted 6 lines
qapi/qom.json +2 -1
@@ -868,7 +883,8 @@
       'tls-creds-psk':              'TlsCredsPskProperties',
       'tls-creds-x509':             'TlsCredsX509Properties',
       'tls-cipher-suites':          'TlsCredsProperties',
-      'x-remote-object':            'RemoteObjectProperties'
+      'x-remote-object':            'RemoteObjectProperties',
+      'vfio-user':                  'VfioUserProperties'

"vfio-user" doesn't communicate whether this is a client or server. Is "vfio-user-server" clearer?

Quoted 15 lines
qapi/qom.json +0 -0
   } }
 
 ##
hw/remote/vfio-user-obj.c new file +6 -0
new file mode 100644
index 0000000..4a1e297
@@ -0,0 +1,145 @@
+/**
+ * QEMU vfio-user server object
+ *
+ * Copyright © 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL-v2, version 2 or later.

hw/remote/vfio-user-obj.c line 16

Quoted 10 lines
hw/remote/vfio-user-obj.c new file +10 -0
+ *
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+/**
+ * Usage: add options:
+ *     -machine x-remote
+ *     -device <PCI-device>,id=<pci-dev-id>
+ *     -object vfio-user,id=<id>,socket=<socket-path>,devid=<pci-dev-id>

I suggest renaming devid= to device= or pci-device= (similar to drive= and netdev=) for consistency and to avoid confusion with PCI Device IDs.

hw/remote/meson.build remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) · line 9

Quoted 9 lines
hw/remote/meson.build +1 -0
index fb35fb8..cd44dfc 100644
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
+remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('vfio-user-obj.c'))

If you use CONFIG_VFIO_USER_SERVER then it's easier to separate mpqemu from vfio-user. Sharing CONFIG_MULTIPROCESS could become messy later.

#77 lore raw

Re: [PATCH RFC server v2 02/11] vfio-user: define vfio-user object

Quoted 37 lines · Stefan Hajnoczi
Quoted 32 lines · Jagannathan Raman

Define vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object

---

qapi/qom.json             |  20 ++++++-
hw/remote/vfio-user-obj.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++
MAINTAINERS               |   1 +
hw/remote/meson.build     |   1 +
hw/remote/trace-events    |   3 +
5 files changed, 168 insertions(+), 2 deletions(-)

create mode 100644 hw/remote/vfio-user-obj.c

qapi/qom.json +8 -0
index a25616b..3e941ee 100644
@@ -689,6 +689,20 @@
  'data': { 'fd': 'str', 'devid': 'str' } }
 
 ##
+# @VfioUserProperties:
+#
+# Properties for vfio-user objects.
+#
+# @socket: path to be used as socket by the libvfiouser library
+#
+# @devid: the id of the device to be associated with the file descriptor
+#
Quoted 8 lines

qapi/qom.json line 703

Quoted 4 lines
qapi/qom.json +4 -0
+# Since: 6.0
+##
+{ 'struct': 'VfioUserProperties',
+  'data': { 'socket': 'str', 'devid': 'str' } }

Please use 'SocketAddress' for socket instead of 'str'. That way file descriptor passing is easy to support and additional socket address families can be supported in the future.

OK, will do.

Quoted 15 lines
Quoted 15 lines
qapi/qom.json +4 -1
+
+##
 # @RngProperties:
 #
 # Properties for objects of classes derived from rng.
@@ -812,7 +826,8 @@
    'tls-creds-psk',
    'tls-creds-x509',
    'tls-cipher-suites',
-    'x-remote-object'
+    'x-remote-object',
+    'vfio-user'
  ] }
 
Quoted 11 lines

qapi/qom.json line 887

Quoted 8 lines
qapi/qom.json +2 -1
 ##
@@ -868,7 +883,8 @@
      'tls-creds-psk':              'TlsCredsPskProperties',
      'tls-creds-x509':             'TlsCredsX509Properties',
      'tls-cipher-suites':          'TlsCredsProperties',
-      'x-remote-object':            'RemoteObjectProperties'
+      'x-remote-object':            'RemoteObjectProperties',
+      'vfio-user':                  'VfioUserProperties'

"vfio-user" doesn't communicate whether this is a client or server. Is "vfio-user-server" clearer?

“vfio-user-server” sounds better.

Quoted 21 lines
Quoted 21 lines
qapi/qom.json +0 -0
  } }
 
 ##
hw/remote/vfio-user-obj.c new file +11 -0
new file mode 100644
index 0000000..4a1e297
@@ -0,0 +1,145 @@
+/**
+ * QEMU vfio-user server object
+ *
+ * Copyright © 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL-v2, version 2 or later.
+ *
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
Quoted 8 lines

hw/remote/vfio-user-obj.c line 16

Quoted 5 lines
hw/remote/vfio-user-obj.c new file +5 -0
+/**
+ * Usage: add options:
+ *     -machine x-remote
+ *     -device <PCI-device>,id=<pci-dev-id>
+ *     -object vfio-user,id=<id>,socket=<socket-path>,devid=<pci-dev-id>

I suggest renaming devid= to device= or pci-device= (similar to drive= and netdev=) for consistency and to avoid confusion with PCI Device IDs.

OK, will do.

Quoted 13 lines

hw/remote/meson.build remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) · line 9

Quoted 10 lines
hw/remote/meson.build +1 -0
index fb35fb8..cd44dfc 100644
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
+remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('vfio-user-obj.c'))

If you use CONFIG_VFIO_USER_SERVER then it's easier to separate mpqemu from vfio-user. Sharing CONFIG_MULTIPROCESS could become messy later.

OK, got it.

--
Jag

#78 lore raw

RFC PATCH v2 3/11 vfio-user: instantiate vfio-user context

In reply to Jagannathan Raman

create a context with the vfio-user library to run a PCI device

---
 hw/remote/vfio-user-obj.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
hw/remote/vfio-user-obj.c +29 -0
index 4a1e297..99d3dd1 100644
@@ -27,11 +27,17 @@
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 
+#include <errno.h>
+
 #include "qom/object.h"
 #include "qom/object_interfaces.h"
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "sysemu/runstate.h"
+#include "qemu/notify.h"
+#include "qapi/error.h"
+#include "sysemu/sysemu.h"
+#include "libvfio-user.h"
 
 #define TYPE_VFU_OBJECT "vfio-user"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -51,6 +57,10 @@ struct VfuObject {
 
     char *socket;
     char *devid;
+
+    Notifier machine_done;
+
+    vfu_ctx_t *vfu_ctx;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -75,9 +85,23 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
     trace_vfu_prop("devid", str);
 }
 
+static void vfu_object_machine_done(Notifier *notifier, void *data)
+{
+    VfuObject *o = container_of(notifier, VfuObject, machine_done);
+
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
+                                o, VFU_DEV_TYPE_PCI);
+    if (o->vfu_ctx == NULL) {
+        error_setg(&error_abort, "vfu: Failed to create context - %s",
+                   strerror(errno));
+        return;
+    }
+}
+
 static void vfu_object_init(Object *obj)
 {
     VfuObjectClass *k = VFU_OBJECT_GET_CLASS(obj);
+    VfuObject *o = VFU_OBJECT(obj);
 
     if (!object_dynamic_cast(OBJECT(current_machine), TYPE_REMOTE_MACHINE)) {
         error_report("vfu: %s only compatible with %s machine",
@@ -92,6 +116,9 @@ static void vfu_object_init(Object *obj)
     }
 
     k->nr_devs++;
+
+    o->machine_done.notify = vfu_object_machine_done;
+    qemu_add_machine_init_done_notifier(&o->machine_done);
 }
 
 static void vfu_object_finalize(Object *obj)
@@ -101,6 +128,8 @@ static void vfu_object_finalize(Object *obj)
 
     k->nr_devs--;
 
+    vfu_destroy_ctx(o->vfu_ctx);
+
     g_free(o->socket);
     g_free(o->devid);
 
Signature · 1 line
1.8.3.1
#79 lore raw

Re: [PATCH RFC server v2 03/11] vfio-user: instantiate vfio-user context

Quoted 14 lines · Jagannathan Raman

create a context with the vfio-user library to run a PCI device

---
 hw/remote/vfio-user-obj.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
hw/remote/vfio-user-obj.c +0 -0
index 4a1e297..99d3dd1 100644

hw/remote/vfio-user-obj.c line 30

Quoted 4 lines
hw/remote/vfio-user-obj.c +1 -0
@@ -27,11 +27,17 @@
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 
+#include <errno.h>

qemu/osdep.h already includes <errno.h>

Quoted 25 lines
hw/remote/vfio-user-obj.c +9 -0
+
 #include "qom/object.h"
 #include "qom/object_interfaces.h"
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "sysemu/runstate.h"
+#include "qemu/notify.h"
+#include "qapi/error.h"
+#include "sysemu/sysemu.h"
+#include "libvfio-user.h"
 
 #define TYPE_VFU_OBJECT "vfio-user"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -51,6 +57,10 @@ struct VfuObject {
 
     char *socket;
     char *devid;
+
+    Notifier machine_done;
+
+    vfu_ctx_t *vfu_ctx;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 88

Quoted 4 lines
hw/remote/vfio-user-obj.c +1 -0
@@ -75,9 +85,23 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
     trace_vfu_prop("devid", str);
 }
 
+static void vfu_object_machine_done(Notifier *notifier, void *data)

Please document the reason for using a machine init done notifier.

Quoted 30 lines
hw/remote/vfio-user-obj.c +16 -0
+{
+    VfuObject *o = container_of(notifier, VfuObject, machine_done);
+
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
+                                o, VFU_DEV_TYPE_PCI);
+    if (o->vfu_ctx == NULL) {
+        error_setg(&error_abort, "vfu: Failed to create context - %s",
+                   strerror(errno));
+        return;
+    }
+}
+
 static void vfu_object_init(Object *obj)
 {
     VfuObjectClass *k = VFU_OBJECT_GET_CLASS(obj);
+    VfuObject *o = VFU_OBJECT(obj);
 
     if (!object_dynamic_cast(OBJECT(current_machine), TYPE_REMOTE_MACHINE)) {
         error_report("vfu: %s only compatible with %s machine",
@@ -92,6 +116,9 @@ static void vfu_object_init(Object *obj)
     }
 
     k->nr_devs++;
+
+    o->machine_done.notify = vfu_object_machine_done;
+    qemu_add_machine_init_done_notifier(&o->machine_done);
 }
 
 static void vfu_object_finalize(Object *obj)

hw/remote/vfio-user-obj.c static void vfu_object_finalize(Object *obj) · line 131

Quoted 4 lines
hw/remote/vfio-user-obj.c +1 -0
@@ -101,6 +128,8 @@ static void vfu_object_finalize(Object *obj)
 
     k->nr_devs--;
 
+    vfu_destroy_ctx(o->vfu_ctx);

Will this function ever be called before vfu_object_machine_done() is called? In that case vfu_ctx isn't initialized.

#80 lore raw

Re: [PATCH RFC server v2 03/11] vfio-user: instantiate vfio-user context

Quoted 49 lines · Stefan Hajnoczi
Quoted 14 lines · Jagannathan Raman

create a context with the vfio-user library to run a PCI device

---
hw/remote/vfio-user-obj.c | 29 +++++++++++++++++++++++++++++

1 file changed, 29 insertions(+)
hw/remote/vfio-user-obj.c +0 -0
index 4a1e297..99d3dd1 100644

hw/remote/vfio-user-obj.c line 30

Quoted 4 lines
hw/remote/vfio-user-obj.c +1 -0
@@ -27,11 +27,17 @@
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 
+#include <errno.h>

qemu/osdep.h already includes <errno.h>

Quoted 23 lines
hw/remote/vfio-user-obj.c +9 -0
+
 #include "qom/object.h"
 #include "qom/object_interfaces.h"
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "sysemu/runstate.h"
+#include "qemu/notify.h"
+#include "qapi/error.h"
+#include "sysemu/sysemu.h"
+#include "libvfio-user.h"
 
 #define TYPE_VFU_OBJECT "vfio-user"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -51,6 +57,10 @@ struct VfuObject {
 
    char *socket;
    char *devid;
+
+    Notifier machine_done;
+
+    vfu_ctx_t *vfu_ctx;
 };
 
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 88

Quoted 6 lines
hw/remote/vfio-user-obj.c +1 -0
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -75,9 +85,23 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
    trace_vfu_prop("devid", str);
 }
 
+static void vfu_object_machine_done(Notifier *notifier, void *data)

Please document the reason for using a machine init done notifier.

OK, will do.

Quoted 32 lines
Quoted 32 lines
hw/remote/vfio-user-obj.c +16 -0
+{
+    VfuObject *o = container_of(notifier, VfuObject, machine_done);
+
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
+                                o, VFU_DEV_TYPE_PCI);
+    if (o->vfu_ctx == NULL) {
+        error_setg(&error_abort, "vfu: Failed to create context - %s",
+                   strerror(errno));
+        return;
+    }
+}
+
 static void vfu_object_init(Object *obj)
 {
    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(obj);
+    VfuObject *o = VFU_OBJECT(obj);
 
    if (!object_dynamic_cast(OBJECT(current_machine), TYPE_REMOTE_MACHINE)) {
        error_report("vfu: %s only compatible with %s machine",
@@ -92,6 +116,9 @@ static void vfu_object_init(Object *obj)
    }
 
    k->nr_devs++;
+
+    o->machine_done.notify = vfu_object_machine_done;
+    qemu_add_machine_init_done_notifier(&o->machine_done);
 }
 
 static void vfu_object_finalize(Object *obj)
@@ -101,6 +128,8 @@ static void vfu_object_finalize(Object *obj)
 
Quoted 6 lines

hw/remote/vfio-user-obj.c static void vfu_object_finalize(Object *obj) · line 131

Quoted 3 lines
hw/remote/vfio-user-obj.c +1 -0
    k->nr_devs--;
 
+    vfu_destroy_ctx(o->vfu_ctx);

Will this function ever be called before vfu_object_machine_done() is called? In that case vfu_ctx isn't initialized.

There are some case where vfu_object_finalize() could be called before vfu_object_machine_done() executes. In that case o->vfu_ctx would be NULL - we didn’t account for that before.

vfu_destroy_ctx() does check for NULL - however, we’ll add a check here as well in case vfu_destroy_ctx() changes in the future.

--
Jag

#81 lore raw

RFC PATCH v2 4/11 vfio-user: find and init PCI device

In reply to Jagannathan Raman

Find the PCI device with specified id. Initialize the device context with the QEMU PCI device

---
 hw/remote/vfio-user-obj.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
hw/remote/vfio-user-obj.c +28 -0
index 99d3dd1..5ae0991 100644
@@ -38,6 +38,8 @@
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
+#include "hw/qdev-core.h"
+#include "hw/pci/pci.h"
 
 #define TYPE_VFU_OBJECT "vfio-user"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -61,6 +63,8 @@ struct VfuObject {
     Notifier machine_done;
 
     vfu_ctx_t *vfu_ctx;
+
+    PCIDevice *pci_dev;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -88,6 +92,8 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
+    DeviceState *dev = NULL;
+    int ret;
 
     o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
                                 o, VFU_DEV_TYPE_PCI);
@@ -96,6 +102,28 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
                    strerror(errno));
         return;
     }
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->devid);
+    if (dev == NULL) {
+        error_setg(&error_abort, "vfu: Device %s not found", o->devid);
+        return;
+    }
+
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(&error_abort, "vfu: %s not a PCI devices", o->devid);
+        return;
+    }
+
+    o->pci_dev = PCI_DEVICE(dev);
+
+    ret = vfu_pci_init(o->vfu_ctx, VFU_PCI_TYPE_CONVENTIONAL,
+                       PCI_HEADER_TYPE_NORMAL, 0);
+    if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to attach PCI device %s to context - %s",
+                   o->devid, strerror(errno));
+        return;
+    }
 }
 
 static void vfu_object_init(Object *obj)
Signature · 1 line
1.8.3.1
#82 lore raw

Re: [PATCH RFC server v2 04/11] vfio-user: find and init PCI device

Quoted 10 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +6 -0
@@ -96,6 +102,28 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
                    strerror(errno));
         return;
     }
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->devid);
+    if (dev == NULL) {
+        error_setg(&error_abort, "vfu: Device %s not found", o->devid);
+        return;
+    }

hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 120

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(&error_abort, "vfu: %s not a PCI devices", o->devid);
+        return;
+    }
+
+    o->pci_dev = PCI_DEVICE(dev);
+
+    ret = vfu_pci_init(o->vfu_ctx, VFU_PCI_TYPE_CONVENTIONAL,
+                       PCI_HEADER_TYPE_NORMAL, 0);

What is needed to support PCI Express?

#83 lore raw

Re: [PATCH RFC server v2 04/11] vfio-user: find and init PCI device

Quoted 19 lines · Stefan Hajnoczi
Quoted 14 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +10 -0
@@ -96,6 +102,28 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
                   strerror(errno));
        return;
    }
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->devid);
+    if (dev == NULL) {
+        error_setg(&error_abort, "vfu: Device %s not found", o->devid);
+        return;
+    }
+
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(&error_abort, "vfu: %s not a PCI devices", o->devid);
+        return;
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 120

Quoted 6 lines
hw/remote/vfio-user-obj.c +6 -0
+    }
+
+    o->pci_dev = PCI_DEVICE(dev);
+
+    ret = vfu_pci_init(o->vfu_ctx, VFU_PCI_TYPE_CONVENTIONAL,
+                       PCI_HEADER_TYPE_NORMAL, 0);

What is needed to support PCI Express?

I think we could check if o->pci_dev supports QEMU_PCI_CAP_EXPRESS, and based on that choose if we should use VFU_PCI_TYPE_CONVENTIONAL or VFU_PCI_TYPE_EXPRESS.

pci_is_express() is already doing that, although it’s a private function now. It’s a good time to export it.

--
Jag

#84 lore raw

RFC PATCH v2 5/11 vfio-user: run vfio-user context

In reply to Jagannathan Raman

Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it

---
 hw/remote/vfio-user-obj.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 70 insertions(+), 1 deletion(-)
hw/remote/vfio-user-obj.c +70 -1
index 5ae0991..0726eb9 100644
@@ -35,6 +35,7 @@
 #include "trace.h"
 #include "sysemu/runstate.h"
 #include "qemu/notify.h"
+#include "qemu/thread.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
@@ -65,6 +66,8 @@ struct VfuObject {
     vfu_ctx_t *vfu_ctx;
 
     PCIDevice *pci_dev;
+
+    int vfu_poll_fd;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -89,13 +92,67 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
     trace_vfu_prop("devid", str);
 }
 
+static void vfu_object_ctx_run(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret = -1;
+
+    while (ret != 0) {
+        ret = vfu_run_ctx(o->vfu_ctx);
+        if (ret < 0) {
+            if (errno == EINTR) {
+                continue;
+            } else if (errno == ENOTCONN) {
+                qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);
+                o->vfu_poll_fd = -1;
+                object_unparent(OBJECT(o));
+                break;
+            } else {
+                error_setg(&error_abort, "vfu: Failed to run device %s - %s",
+                           o->devid, strerror(errno));
+                 break;
+            }
+        }
+    }
+}
+
+static void *vfu_object_attach_ctx(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret;
+
+retry_attach:
+    ret = vfu_attach_ctx(o->vfu_ctx);
+    if (ret < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
+        goto retry_attach;
+    } else if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to attach device %s to context - %s",
+                   o->devid, strerror(errno));
+        return NULL;
+    }
+
+    o->vfu_poll_fd = vfu_get_poll_fd(o->vfu_ctx);
+    if (o->vfu_poll_fd < 0) {
+        error_setg(&error_abort, "vfu: Failed to get poll fd %s", o->devid);
+        return NULL;
+    }
+
+    qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_ctx_run,
+                        NULL, o);
+
+    return NULL;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
     DeviceState *dev = NULL;
+    QemuThread thread;
     int ret;
 
-    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket,
+                                LIBVFIO_USER_FLAG_ATTACH_NB,
                                 o, VFU_DEV_TYPE_PCI);
     if (o->vfu_ctx == NULL) {
         error_setg(&error_abort, "vfu: Failed to create context - %s",
@@ -124,6 +181,16 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
                    o->devid, strerror(errno));
         return;
     }
+
+    ret = vfu_realize_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
+                   o->devid, strerror(errno));
+        return;
+    }
+
+    qemu_thread_create(&thread, o->socket, vfu_object_attach_ctx, o,
+                       QEMU_THREAD_DETACHED);
 }
 
 static void vfu_object_init(Object *obj)
@@ -147,6 +214,8 @@ static void vfu_object_init(Object *obj)
 
     o->machine_done.notify = vfu_object_machine_done;
     qemu_add_machine_init_done_notifier(&o->machine_done);
+
+    o->vfu_poll_fd = -1;
 }
 
 static void vfu_object_finalize(Object *obj)
Signature · 1 line
1.8.3.1
#85 lore raw

Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context

Quoted 58 lines · Jagannathan Raman

Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it

---
 hw/remote/vfio-user-obj.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 70 insertions(+), 1 deletion(-)
hw/remote/vfio-user-obj.c +25 -0
index 5ae0991..0726eb9 100644
@@ -35,6 +35,7 @@
 #include "trace.h"
 #include "sysemu/runstate.h"
 #include "qemu/notify.h"
+#include "qemu/thread.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
@@ -65,6 +66,8 @@ struct VfuObject {
     vfu_ctx_t *vfu_ctx;
 
     PCIDevice *pci_dev;
+
+    int vfu_poll_fd;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -89,13 +92,67 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
     trace_vfu_prop("devid", str);
 }
 
+static void vfu_object_ctx_run(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret = -1;
+
+    while (ret != 0) {
+        ret = vfu_run_ctx(o->vfu_ctx);
+        if (ret < 0) {
+            if (errno == EINTR) {
+                continue;
+            } else if (errno == ENOTCONN) {
+                qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);
+                o->vfu_poll_fd = -1;
+                object_unparent(OBJECT(o));
+                break;
+            } else {
+                error_setg(&error_abort, "vfu: Failed to run device %s - %s",
+                           o->devid, strerror(errno));
+                 break;
+            }
+        }
+    }

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 126

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+}
+
+static void *vfu_object_attach_ctx(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret;
+
+retry_attach:
+    ret = vfu_attach_ctx(o->vfu_ctx);
+    if (ret < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {

Does this loop consume 100% CPU since this is non-blocking?

Is it possible to register the fd with a QEMU AioContext instead of spawning a separate thread?

libvfio-user has non-blocking listen_fd but conn_fd is always blocking. This means ATTACH_NB is not useful because vfu_attach_ctx() is actually blocking. I think this means vfu_run_ctx() is also blocking in some places and QEMU's event loop might hang :(.

Can you make libvfio-user non-blocking in order to solve these issues?

Quoted 37 lines
hw/remote/vfio-user-obj.c +23 -1
+        goto retry_attach;
+    } else if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to attach device %s to context - %s",
+                   o->devid, strerror(errno));
+        return NULL;
+    }
+
+    o->vfu_poll_fd = vfu_get_poll_fd(o->vfu_ctx);
+    if (o->vfu_poll_fd < 0) {
+        error_setg(&error_abort, "vfu: Failed to get poll fd %s", o->devid);
+        return NULL;
+    }
+
+    qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_ctx_run,
+                        NULL, o);
+
+    return NULL;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
     DeviceState *dev = NULL;
+    QemuThread thread;
     int ret;
 
-    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket,
+                                LIBVFIO_USER_FLAG_ATTACH_NB,
                                 o, VFU_DEV_TYPE_PCI);
     if (o->vfu_ctx == NULL) {
         error_setg(&error_abort, "vfu: Failed to create context - %s",
@@ -124,6 +181,16 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
                    o->devid, strerror(errno));
         return;
     }

hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 193

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+
+    ret = vfu_realize_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
+                   o->devid, strerror(errno));
+        return;
+    }
+
+    qemu_thread_create(&thread, o->socket, vfu_object_attach_ctx, o,
+                       QEMU_THREAD_DETACHED);

Is this thread leaked when the object is destroyed?

Quoted 15 lines
hw/remote/vfio-user-obj.c +2 -0
 }
 
 static void vfu_object_init(Object *obj)
@@ -147,6 +214,8 @@ static void vfu_object_init(Object *obj)
 
     o->machine_done.notify = vfu_object_machine_done;
     qemu_add_machine_init_done_notifier(&o->machine_done);
+
+    o->vfu_poll_fd = -1;
 }
 
 static void vfu_object_finalize(Object *obj)
Signature · 1 line
1.8.3.1
#86 lore raw

Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context

Quoted 10 lines · Stefan Hajnoczi

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 126

Quoted 8 lines
hw/remote/vfio-user-obj.c +8 -0
+static void *vfu_object_attach_ctx(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret;
+
+retry_attach:
+    ret = vfu_attach_ctx(o->vfu_ctx);
+    if (ret < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {

Does this loop consume 100% CPU since this is non-blocking?

Looks like it. Instead after vfu_create_ctx, there should be a vfu_get_poll_fd() to get the listen socket, then a qemu_set_fd_handler(vfu_object_attach_ctx) to handle the attach when the listen socket is ready, modulo the below part.

Quoted 1 line

libvfio-user has non-blocking listen_fd but conn_fd is always blocking.

It is, but in vfu_run_ctx(), we poll on it:

```
790     if (vfu_ctx->flags & LIBVFIO_USER_FLAG_ATTACH_NB) {                          
791         sock_flags = MSG_DONTWAIT | MSG_WAITALL;                                 
792     }                                                                            
793     return get_msg(hdr, sizeof(*hdr), fds, nr_fds, ts->conn_fd, sock_flags);     
```
Quoted 2 lines

This means ATTACH_NB is not useful because vfu_attach_ctx() is actually blocking.

You're correct that vfu_attach_ctx is in fact partially blocking: after accepting the connection, we call negotiate(), which can indeed block waiting if the client hasn't sent anything.

Quoted 1 line

I think this means vfu_run_ctx() is also blocking in some places

Correct. There's a presumption that if a message is ready, we can read it all without blocking, and equally that we can write to the socket without blocking.

The library docs are not at all clear on this point.

Quoted 3 lines

and QEMU's event loop might hang :(

Can you make libvfio-user non-blocking in order to solve these issues?

I presume you're concerned about the security aspect: a malicious client could withhold a write, and hence hang the device server.

Problem is the libvfio-user API is synchronous: there's no way to return half-way through a vfu_attach_ctx() (or a vfu_run_ctx() after we read the header) then resume.

We'd have to have a whole separate API to do that, so a separate thread seems a better approach?

regards
john

#87 lore raw

Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context

Quoted 19 lines · John Levon
Quoted 10 lines · Stefan Hajnoczi

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 126

Quoted 8 lines
hw/remote/vfio-user-obj.c +8 -0
+static void *vfu_object_attach_ctx(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret;
+
+retry_attach:
+    ret = vfu_attach_ctx(o->vfu_ctx);
+    if (ret < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {

Does this loop consume 100% CPU since this is non-blocking?

Looks like it. Instead after vfu_create_ctx, there should be a vfu_get_poll_fd() to get the listen socket, then a qemu_set_fd_handler(vfu_object_attach_ctx) to handle the attach when the listen socket is ready, modulo the below part.

Quoted 1 line

libvfio-user has non-blocking listen_fd but conn_fd is always blocking.

Quoted 8 lines

It is, but in vfu_run_ctx(), we poll on it:

```
790     if (vfu_ctx->flags & LIBVFIO_USER_FLAG_ATTACH_NB) {                          
791         sock_flags = MSG_DONTWAIT | MSG_WAITALL;                                 
792     }                                                                            
793     return get_msg(hdr, sizeof(*hdr), fds, nr_fds, ts->conn_fd, sock_flags);     
```

This is only used for the request header. Other I/O is blocking.

Quoted 22 lines
Quoted 3 lines

This means ATTACH_NB is not useful because vfu_attach_ctx() is actually blocking.

You're correct that vfu_attach_ctx is in fact partially blocking: after accepting the connection, we call negotiate(), which can indeed block waiting if the client hasn't sent anything.

Quoted 1 line

I think this means vfu_run_ctx() is also blocking in some places

Correct. There's a presumption that if a message is ready, we can read it all without blocking, and equally that we can write to the socket without blocking.

The library docs are not at all clear on this point.

Quoted 3 lines

and QEMU's event loop might hang :(

Can you make libvfio-user non-blocking in order to solve these issues?

I presume you're concerned about the security aspect: a malicious client could withhold a write, and hence hang the device server.

Quoted 6 lines

Problem is the libvfio-user API is synchronous: there's no way to return half-way through a vfu_attach_ctx() (or a vfu_run_ctx() after we read the header) then resume.

We'd have to have a whole separate API to do that, so a separate thread seems a better approach?

Whether to support non-blocking properly in libvfio-user is a decision for you. If libvfio-user doesn't support non-blocking, then QEMU should run a dedicated thread instead of the partially non-blocking approach in this patch.

A non-blocking approach is nice when there are many devices hosted in a single process or a lot of async replies (which requires extra thread synchronization with the blocking approach).

Stefan

#88 lore raw

Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context

Quoted 7 lines · Stefan Hajnoczi
Quoted 2 lines

We'd have to have a whole separate API to do that, so a separate thread seems a better approach?

Whether to support non-blocking properly in libvfio-user is a decision for you. If libvfio-user doesn't support non-blocking, then QEMU should run a dedicated thread instead of the partially non-blocking approach in this patch.

Right, sure. At this point we don't have any plans to implement a separate async API due to the amount of work involved.

Quoted 3 lines

A non-blocking approach is nice when there are many devices hosted in a single process or a lot of async replies (which requires extra thread synchronization with the blocking approach).

I suppose this would be more of a problem with devices where the I/O path has to be handled via the socket.

regards
john

#89 lore raw

Re: [PATCH RFC server v2 05/11] vfio-user: run vfio-user context

Quoted 13 lines · John Levon
Quoted 7 lines · Stefan Hajnoczi
Quoted 2 lines

We'd have to have a whole separate API to do that, so a separate thread seems a better approach?

Whether to support non-blocking properly in libvfio-user is a decision for you. If libvfio-user doesn't support non-blocking, then QEMU should run a dedicated thread instead of the partially non-blocking approach in this patch.

Right, sure. At this point we don't have any plans to implement a separate async API due to the amount of work involved.

Quoted 6 lines
Quoted 3 lines

A non-blocking approach is nice when there are many devices hosted in a single process or a lot of async replies (which requires extra thread synchronization with the blocking approach).

I suppose this would be more of a problem with devices where the I/O path has to be handled via the socket.

Yes, exactly. I think it shouldn't be a problem when shared memory is used and the irqfd (eventfd) mechanism is used for IRQs. It becomes slow when there's no shared memory or if raising IRQs requires protocol messages.

Stefan

#90 lore raw

RFC PATCH v2 6/11 vfio-user: handle PCI config space accesses

In reply to Jagannathan Raman

Define and register handlers for PCI config space accesses

---
 hw/remote/vfio-user-obj.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 46 insertions(+)
hw/remote/vfio-user-obj.c +44 -0
index 0726eb9..13011ce 100644
@@ -36,6 +36,7 @@
 #include "sysemu/runstate.h"
 #include "qemu/notify.h"
 #include "qemu/thread.h"
+#include "qemu/main-loop.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
@@ -144,6 +145,38 @@ retry_attach:
     return NULL;
 }
 
+static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
+                                     size_t count, loff_t offset,
+                                     const bool is_write)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint32_t pci_access_width = sizeof(uint32_t);
+    size_t bytes = count;
+    uint32_t val = 0;
+    char *ptr = buf;
+    int len;
+
+    while (bytes > 0) {
+        len = (bytes > pci_access_width) ? pci_access_width : bytes;
+        if (is_write) {
+            memcpy(&val, ptr, len);
+            pci_default_write_config(PCI_DEVICE(o->pci_dev),
+                                     offset, val, len);
+            trace_vfu_cfg_write(offset, val);
+        } else {
+            val = pci_default_read_config(PCI_DEVICE(o->pci_dev),
+                                          offset, len);
+            memcpy(ptr, &val, len);
+            trace_vfu_cfg_read(offset, val);
+        }
+        offset += len;
+        ptr += len;
+        bytes -= len;
+    }
+
+    return count;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -182,6 +215,17 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
         return;
     }
 
+    ret = vfu_setup_region(o->vfu_ctx, VFU_PCI_DEV_CFG_REGION_IDX,
+                           pci_config_size(o->pci_dev), &vfu_object_cfg_access,
+                           VFU_REGION_FLAG_RW | VFU_REGION_FLAG_ALWAYS_CB,
+                           NULL, 0, -1, 0);
+    if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to setup config space handlers for %s- %s",
+                   o->devid, strerror(errno));
+        return;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
index 7da12f0..2ef7884 100644
@@ -5,3 +5,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
 
 # vfio-user-obj.c
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
+vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
+vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
Signature · 1 line
1.8.3.1
#91 lore raw

Re: [PATCH RFC server v2 06/11] vfio-user: handle PCI config space accesses

Quoted 12 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +12 -0
+static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
+                                     size_t count, loff_t offset,
+                                     const bool is_write)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint32_t pci_access_width = sizeof(uint32_t);
+    size_t bytes = count;
+    uint32_t val = 0;
+    char *ptr = buf;
+    int len;
+
+    while (bytes > 0) {

hw/remote/vfio-user-obj.c retry_attach: · line 169

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+        len = (bytes > pci_access_width) ? pci_access_width : bytes;
+        if (is_write) {
+            memcpy(&val, ptr, len);
+            pci_default_write_config(PCI_DEVICE(o->pci_dev),
+                                     offset, val, len);
+            trace_vfu_cfg_write(offset, val);
+        } else {
+            val = pci_default_read_config(PCI_DEVICE(o->pci_dev),
+                                          offset, len);
+            memcpy(ptr, &val, len);

pci_default_read_config() returns a host-endian 32-bit value. This code looks wrong because it copies different bytes on big- and little-endian hosts.

Review

Quoted 2 lines
(quoted diff) +2 -0
+            trace_vfu_cfg_read(offset, val);
+        }

Why call pci_default_read/write_config() directly instead of pci_dev->config_read/write()?

#92 lore raw

Re: [PATCH RFC server v2 06/11] vfio-user: handle PCI config space accesses

Quoted 23 lines · Stefan Hajnoczi
Quoted 18 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +18 -0
+static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
+                                     size_t count, loff_t offset,
+                                     const bool is_write)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint32_t pci_access_width = sizeof(uint32_t);
+    size_t bytes = count;
+    uint32_t val = 0;
+    char *ptr = buf;
+    int len;
+
+    while (bytes > 0) {
+        len = (bytes > pci_access_width) ? pci_access_width : bytes;
+        if (is_write) {
+            memcpy(&val, ptr, len);
+            pci_default_write_config(PCI_DEVICE(o->pci_dev),
+                                     offset, val, len);
+            trace_vfu_cfg_write(offset, val);
Quoted 8 lines

hw/remote/vfio-user-obj.c retry_attach: · line 169

Quoted 4 lines
hw/remote/vfio-user-obj.c +4 -0
+        } else {
+            val = pci_default_read_config(PCI_DEVICE(o->pci_dev),
+                                          offset, len);
+            memcpy(ptr, &val, len);

pci_default_read_config() returns a host-endian 32-bit value. This code looks wrong because it copies different bytes on big- and little-endian hosts.

I’ll collect more details on this one, trying to wrap my head around it.

Concerning config space writes, it doesn’t look like we need to perform any conversion as the store operation automatically happens in the CPU’s native format when we do something like the following:
PCIDevice->config[addr] = val;

Concerning config read, I observed that pci_default_read_config() performs le32_to_cpu() conversion. May be we should not rely on it doing the conversion.

Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+            trace_vfu_cfg_read(offset, val);
+        }

Why call pci_default_read/write_config() directly instead of pci_dev->config_read/write()?

That makes sense - we should be calling pci_dev->config_read/write().

After performing pci_dev->config_read(), I’ll convert it to the CPU’s endianness format using le32_to_cpu(). On big-endian systems, it would re-order the bytes, and on little-endian systems it would be a no-op.

--
Jag

#93 lore raw

Re: [PATCH RFC server v2 06/11] vfio-user: handle PCI config space accesses

Quoted 32 lines · Jag Raman
Quoted 23 lines · Stefan Hajnoczi
Quoted 18 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +18 -0
+static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
+                                     size_t count, loff_t offset,
+                                     const bool is_write)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint32_t pci_access_width = sizeof(uint32_t);
+    size_t bytes = count;
+    uint32_t val = 0;
+    char *ptr = buf;
+    int len;
+
+    while (bytes > 0) {
+        len = (bytes > pci_access_width) ? pci_access_width : bytes;
+        if (is_write) {
+            memcpy(&val, ptr, len);
+            pci_default_write_config(PCI_DEVICE(o->pci_dev),
+                                     offset, val, len);
+            trace_vfu_cfg_write(offset, val);
Quoted 8 lines

hw/remote/vfio-user-obj.c retry_attach: · line 169

Quoted 4 lines
hw/remote/vfio-user-obj.c +4 -0
+        } else {
+            val = pci_default_read_config(PCI_DEVICE(o->pci_dev),
+                                          offset, len);
+            memcpy(ptr, &val, len);

pci_default_read_config() returns a host-endian 32-bit value. This code looks wrong because it copies different bytes on big- and little-endian hosts.

Quoted 6 lines

I’ll collect more details on this one, trying to wrap my head around it.

Concerning config space writes, it doesn’t look like we need to perform any conversion as the store operation automatically happens in the CPU’s native format when we do something like the following:
PCIDevice->config[addr] = val;

PCIDevice->config is uint8_t*. Endianness isn't an issue with single byte accesses.

Quoted 4 lines

Concerning config read, I observed that pci_default_read_config() performs le32_to_cpu() conversion. May be we should not rely on it doing the conversion.

Yes, ->config_read() returns a host-endian 32-bit value and ->config_write() receives a host-endian 32-bit value (it has a bit-shifting loop that implicitly handles endianness conversion).

Stefan

#94 lore raw

RFC PATCH v2 7/11 vfio-user: handle DMA mappings

In reply to Jagannathan Raman

Define and register callbacks to manage the RAM regions used for device DMA

---
 hw/remote/vfio-user-obj.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 52 insertions(+)
hw/remote/vfio-user-obj.c +50 -0
index 13011ce..76fb2d4 100644
@@ -177,6 +177,49 @@ static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
     return count;
 }
 
+static void dma_register(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    MemoryRegion *subregion = NULL;
+    g_autofree char *name = NULL;
+    static unsigned int suffix;
+    struct iovec *iov = &info->iova;
+
+    if (!info->vaddr) {
+        return;
+    }
+
+    name = g_strdup_printf("remote-mem-%u", suffix++);
+
+    subregion = g_new0(MemoryRegion, 1);
+
+    memory_region_init_ram_ptr(subregion, NULL, name,
+                               iov->iov_len, info->vaddr);
+
+    memory_region_add_subregion(get_system_memory(), (hwaddr)iov->iov_base,
+                                subregion);
+
+    trace_vfu_dma_register((uint64_t)iov->iov_base, iov->iov_len);
+}
+
+static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    MemoryRegion *mr = NULL;
+    ram_addr_t offset;
+
+    mr = memory_region_from_host(info->vaddr, &offset);
+    if (!mr) {
+        return 0;
+    }
+
+    memory_region_del_subregion(get_system_memory(), mr);
+
+    object_unparent((OBJECT(mr)));
+
+    trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
+
+    return 0;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -226,6 +269,13 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
         return;
     }
 
+    ret = vfu_setup_device_dma(o->vfu_ctx, &dma_register, &dma_unregister);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to setup DMA handlers for %s",
+                   o->devid);
+        return;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
index 2ef7884..f945c7e 100644
@@ -7,3 +7,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
 vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
+vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
+vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
Signature · 1 line
1.8.3.1
Reviewed-by on v2 7/11
#95 lore raw

Re: [PATCH RFC server v2 07/11] vfio-user: handle DMA mappings

Quoted 10 lines · Jagannathan Raman

Define and register callbacks to manage the RAM regions used for device DMA

---
 hw/remote/vfio-user-obj.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 52 insertions(+)
#96 lore raw

RFC PATCH v2 8/11 vfio-user: handle PCI BAR accesses

In reply to Jagannathan Raman

Determine the BARs used by the PCI device and register handlers to manage the access to the same.

---
 hw/remote/vfio-user-obj.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 +
 2 files changed, 97 insertions(+)
hw/remote/vfio-user-obj.c +95 -0
index 76fb2d4..299c938 100644
@@ -220,6 +220,99 @@ static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
     return 0;
 }
 
+static ssize_t vfu_object_bar_rw(PCIDevice *pci_dev, hwaddr addr, size_t count,
+                                 char * const buf, const bool is_write,
+                                 uint8_t type)
+{
+    AddressSpace *as = NULL;
+    MemTxResult res;
+
+    if (type == PCI_BASE_ADDRESS_SPACE_MEMORY) {
+        as = pci_device_iommu_address_space(pci_dev);
+    } else {
+        as = &address_space_io;
+    }
+
+    trace_vfu_bar_rw_enter(is_write ? "Write" : "Read", (uint64_t)addr);
+
+    res = address_space_rw(as, addr, MEMTXATTRS_UNSPECIFIED, (void *)buf,
+                           (hwaddr)count, is_write);
+    if (res != MEMTX_OK) {
+        warn_report("vfu: failed to %s 0x%"PRIx64"",
+                    is_write ? "write to" : "read from",
+                    addr);
+        return -1;
+    }
+
+    trace_vfu_bar_rw_exit(is_write ? "Write" : "Read", (uint64_t)addr);
+
+    return count;
+}
+
+/**
+ * VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs.
+ *
+ * To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would
+ * define vfu_object_bar2_handler
+ */
+#define VFU_OBJECT_BAR_HANDLER(BAR_NO)                                         \
+    static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx,        \
+                                        char * const buf, size_t count,        \
+                                        loff_t offset, const bool is_write)    \
+    {                                                                          \
+        VfuObject *o = vfu_get_private(vfu_ctx);                               \
+        hwaddr addr = (hwaddr)(pci_get_long(o->pci_dev->config +               \
+                                            PCI_BASE_ADDRESS_0 +               \
+                                            (4 * BAR_NO)) + offset);           \
+                                                                               \
+        return vfu_object_bar_rw(o->pci_dev, addr, count, buf, is_write,       \
+                                 o->pci_dev->io_regions[BAR_NO].type);         \
+    }                                                                          \
+
+VFU_OBJECT_BAR_HANDLER(0)
+VFU_OBJECT_BAR_HANDLER(1)
+VFU_OBJECT_BAR_HANDLER(2)
+VFU_OBJECT_BAR_HANDLER(3)
+VFU_OBJECT_BAR_HANDLER(4)
+VFU_OBJECT_BAR_HANDLER(5)
+
+static vfu_region_access_cb_t *vfu_object_bar_handlers[PCI_NUM_REGIONS] = {
+    &vfu_object_bar0_handler,
+    &vfu_object_bar1_handler,
+    &vfu_object_bar2_handler,
+    &vfu_object_bar3_handler,
+    &vfu_object_bar4_handler,
+    &vfu_object_bar5_handler,
+};
+
+/**
+ * vfu_object_register_bars - Identify active BAR regions of pdev and setup
+ *                            callbacks to handle read/write accesses
+ */
+static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
+{
+    uint32_t orig_val, new_val;
+    int i, size;
+
+    for (i = 0; i < PCI_NUM_REGIONS; i++) {
+        orig_val = pci_default_read_config(pdev,
+                                           PCI_BASE_ADDRESS_0 + (4 * i), 4);
+        new_val = 0xffffffff;
+        pci_default_write_config(pdev,
+                                 PCI_BASE_ADDRESS_0 + (4 * i), new_val, 4);
+        new_val = pci_default_read_config(pdev,
+                                          PCI_BASE_ADDRESS_0 + (4 * i), 4);
+        size = (~(new_val & 0xFFFFFFF0)) + 1;
+        pci_default_write_config(pdev, PCI_BASE_ADDRESS_0 + (4 * i),
+                                 orig_val, 4);
+        if (size) {
+            vfu_setup_region(vfu_ctx, VFU_PCI_DEV_BAR0_REGION_IDX + i, size,
+                             vfu_object_bar_handlers[i], VFU_REGION_FLAG_RW,
+                             NULL, 0, -1, 0);
+        }
+    }
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -276,6 +369,8 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
         return;
     }
 
+    vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
index f945c7e..f3f65e2 100644
@@ -9,3 +9,5 @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
 vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
 vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
+vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
+vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
Signature · 1 line
1.8.3.1
#97 lore raw

Re: [PATCH RFC server v2 08/11] vfio-user: handle PCI BAR accesses

Quoted 5 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +5 -0
+/**
+ * VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs.
+ *
+ * To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would
+ * define vfu_object_bar2_handler

hw/remote/vfio-user-obj.c static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info) · line 266

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+ */
+#define VFU_OBJECT_BAR_HANDLER(BAR_NO)                                         \
+    static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx,        \
+                                        char * const buf, size_t count,        \
+                                        loff_t offset, const bool is_write)    \
+    {                                                                          \
+        VfuObject *o = vfu_get_private(vfu_ctx);                               \
+        hwaddr addr = (hwaddr)(pci_get_long(o->pci_dev->config +               \
+                                            PCI_BASE_ADDRESS_0 +               \
+                                            (4 * BAR_NO)) + offset);           \

Does this handle 64-bit BARs?

hw/remote/vfio-user-obj.c static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info) · line 299

Quoted 12 lines
hw/remote/vfio-user-obj.c +12 -0
+/**
+ * vfu_object_register_bars - Identify active BAR regions of pdev and setup
+ *                            callbacks to handle read/write accesses
+ */
+static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
+{
+    uint32_t orig_val, new_val;
+    int i, size;
+
+    for (i = 0; i < PCI_NUM_REGIONS; i++) {
+        orig_val = pci_default_read_config(pdev,
+                                           PCI_BASE_ADDRESS_0 + (4 * i), 4);

Same question as in an earlier patch: should we call pdev->read_config()?

#98 lore raw

Re: [PATCH RFC server v2 08/11] vfio-user: handle PCI BAR accesses

Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +9 -0
+/**
+ * VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs.
+ *
+ * To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would
+ * define vfu_object_bar2_handler
+ */
+#define VFU_OBJECT_BAR_HANDLER(BAR_NO)                                         \
+    static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx,        \
+                                        char * const buf, size_t count,        \
Quoted 8 lines

hw/remote/vfio-user-obj.c static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info) · line 266

Quoted 6 lines
hw/remote/vfio-user-obj.c +6 -0
+                                        loff_t offset, const bool is_write)    \
+    {                                                                          \
+        VfuObject *o = vfu_get_private(vfu_ctx);                               \
+        hwaddr addr = (hwaddr)(pci_get_long(o->pci_dev->config +               \
+                                            PCI_BASE_ADDRESS_0 +               \
+                                            (4 * BAR_NO)) + offset);           \

Does this handle 64-bit BARs?

It presently only handles 32-bit BARs. We’ll add support for 64-bit BARs in the next rev of this series.

Quoted 15 lines

hw/remote/vfio-user-obj.c static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info) · line 299

Quoted 13 lines
hw/remote/vfio-user-obj.c +12 -0
+/**
+ * vfu_object_register_bars - Identify active BAR regions of pdev and setup
+ *                            callbacks to handle read/write accesses
+ */
+static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
+{
+    uint32_t orig_val, new_val;
+    int i, size;
+
+    for (i = 0; i < PCI_NUM_REGIONS; i++) {
+        orig_val = pci_default_read_config(pdev,
+                                           PCI_BASE_ADDRESS_0 + (4 * i), 4);

Same question as in an earlier patch: should we call pdev->read_config()?

Sure, will do.

--
Jag

#99 lore raw

RFC PATCH v2 9/11 vfio-user: handle device interrupts

In reply to Jagannathan Raman

Forward remote device's interrupts to the guest

---
 include/hw/remote/iohub.h |  2 ++
 hw/remote/iohub.c         |  5 +++++
 hw/remote/vfio-user-obj.c | 30 ++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  1 +
 4 files changed, 38 insertions(+)
include/hw/remote/iohub.h +2 -0
index 0bf98e0..d5bd0b0 100644
@@ -15,6 +15,7 @@
 #include "qemu/event_notifier.h"
 #include "qemu/thread-posix.h"
 #include "hw/remote/mpqemu-link.h"
+#include "libvfio-user.h"
 
 #define REMOTE_IOHUB_NB_PIRQS    PCI_DEVFN_MAX
 
@@ -30,6 +31,7 @@ typedef struct RemoteIOHubState {
     unsigned int irq_level[REMOTE_IOHUB_NB_PIRQS];
     ResampleToken token[REMOTE_IOHUB_NB_PIRQS];
     QemuMutex irq_level_lock[REMOTE_IOHUB_NB_PIRQS];
+    vfu_ctx_t *vfu_ctx[REMOTE_IOHUB_NB_PIRQS];
 } RemoteIOHubState;
 
 int remote_iohub_map_irq(PCIDevice *pci_dev, int intx);
hw/remote/iohub.c +5 -0
index 547d597..9410233 100644
@@ -18,6 +18,7 @@
 #include "hw/remote/machine.h"
 #include "hw/remote/iohub.h"
 #include "qemu/main-loop.h"
+#include "trace.h"
 
 void remote_iohub_init(RemoteIOHubState *iohub)
 {
@@ -62,6 +63,10 @@ void remote_iohub_set_irq(void *opaque, int pirq, int level)
     QEMU_LOCK_GUARD(&iohub->irq_level_lock[pirq]);
 
     if (level) {
+        if (iohub->vfu_ctx[pirq]) {
+            trace_vfu_interrupt(pirq);
+            vfu_irq_trigger(iohub->vfu_ctx[pirq], 0);
+        }
         if (++iohub->irq_level[pirq] == 1) {
             event_notifier_set(&iohub->irqfds[pirq]);
         }
hw/remote/vfio-user-obj.c +30 -0
index 299c938..92605ed 100644
@@ -42,6 +42,9 @@
 #include "libvfio-user.h"
 #include "hw/qdev-core.h"
 #include "hw/pci/pci.h"
+#include "hw/boards.h"
+#include "hw/remote/iohub.h"
+#include "hw/remote/machine.h"
 
 #define TYPE_VFU_OBJECT "vfio-user"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -313,6 +316,26 @@ static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
     }
 }
 
+static int vfu_object_setup_irqs(vfu_ctx_t *vfu_ctx, PCIDevice *pci_dev)
+{
+    RemoteMachineState *machine = REMOTE_MACHINE(current_machine);
+    RemoteIOHubState *iohub = &machine->iohub;
+    int pirq, intx, ret;
+
+    ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_INTX_IRQ, 1);
+    if (ret < 0) {
+        return ret;
+    }
+
+    intx = pci_get_byte(pci_dev->config + PCI_INTERRUPT_PIN) - 1;
+
+    pirq = remote_iohub_map_irq(pci_dev, intx);
+
+    iohub->vfu_ctx[pirq] = vfu_ctx;
+
+    return 0;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -371,6 +394,13 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
 
     vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
 
+    ret = vfu_object_setup_irqs(o->vfu_ctx, o->pci_dev);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to setup interrupts for %s",
+                   o->devid);
+        return;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +1 -0
index f3f65e2..b419d6f 100644
@@ -11,3 +11,4 @@ vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %z
 vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
 vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
 vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
+vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"
Signature · 1 line
1.8.3.1
Reviewed-by on v2 9/11
#100 lore raw

Re: [PATCH RFC server v2 09/11] vfio-user: handle device interrupts

Quoted 11 lines · Jagannathan Raman

Forward remote device's interrupts to the guest

---
 include/hw/remote/iohub.h |  2 ++
 hw/remote/iohub.c         |  5 +++++
 hw/remote/vfio-user-obj.c | 30 ++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  1 +
 4 files changed, 38 insertions(+)
#101 lore raw

RFC PATCH v2 10/11 vfio-user: register handlers to facilitate migration

In reply to Jagannathan Raman

Store and load the device's state during migration. use libvfio-user's handlers for this purpose

---
 migration/savevm.h        |   2 +
 hw/remote/vfio-user-obj.c | 313 ++++++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c        |  73 +++++++++++
 3 files changed, 388 insertions(+)
migration/savevm.h +2 -0
index 6461342..8007064 100644
@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
 int qemu_load_device_state(QEMUFile *f);
 int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
         bool in_postcopy, bool inactivate_disks);
+int qemu_remote_savevm(QEMUFile *f, DeviceState *dev);
+int qemu_remote_loadvm(QEMUFile *f);
 
 #endif
hw/remote/vfio-user-obj.c +313 -0

349 lines not on this page · raw message

migration/savevm.c +73 -0

85 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#102 lore raw

Re: [PATCH RFC server v2 10/11] vfio-user: register handlers to facilitate migration

Quoted 4 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +4 -0
+static ssize_t vfu_mig_buf_read(void *opaque, uint8_t *buf, int64_t pos,
+                                size_t size, Error **errp)
+{
+    VfuObject *o = opaque;

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 152

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+
+    if (pos > o->vfu_mig_buf_size) {
+        size = 0;
+    } else if ((pos + size) > o->vfu_mig_buf_size) {
+        size = o->vfu_mig_buf_size;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + pos), size);
+
+    o->vfu_mig_buf_size -= size;

This looks strange. pos increases each time we're called. We seem to be truncating the buffer on each read. Should this line be dropped? Did you test live migration (maybe this code needs more debugging)?

Quoted 60 lines
hw/remote/vfio-user-obj.c +60 -0
+
+    return size;
+}
+
+static ssize_t vfu_mig_buf_write(void *opaque, struct iovec *iov, int iovcnt,
+                                 int64_t pos, Error **errp)
+{
+    VfuObject *o = opaque;
+    uint64_t end = pos + iov_size(iov, iovcnt);
+    int i;
+
+    if (end > o->vfu_mig_buf_size) {
+        o->vfu_mig_buf = g_realloc(o->vfu_mig_buf, end);
+    }
+
+    for (i = 0; i < iovcnt; i++) {
+        memcpy((o->vfu_mig_buf + o->vfu_mig_buf_size), iov[i].iov_base,
+               iov[i].iov_len);
+        o->vfu_mig_buf_size += iov[i].iov_len;
+        o->vfu_mig_buf_pending += iov[i].iov_len;
+    }
+
+    return iov_size(iov, iovcnt);
+}
+
+static int vfu_mig_buf_shutdown(void *opaque, bool rd, bool wr, Error **errp)
+{
+    VfuObject *o = opaque;
+
+    o->vfu_mig_buf_size = 0;
+
+    g_free(o->vfu_mig_buf);
+
+    return 0;
+}
+
+static const QEMUFileOps vfu_mig_fops_save = {
+    .writev_buffer  = vfu_mig_buf_write,
+    .shut_down      = vfu_mig_buf_shutdown,
+};
+
+static const QEMUFileOps vfu_mig_fops_load = {
+    .get_buffer     = vfu_mig_buf_read,
+    .shut_down      = vfu_mig_buf_shutdown,
+};
+
+/**
+ * handlers for vfu_migration_callbacks_t
+ *
+ * The libvfio-user library accesses these handlers to drive the migration
+ * at the remote end, and also to transport the data stored in vfu_mig_buf
+ *
+ */
+static void vfu_mig_state_precopy(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    int ret;
+
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_save, false);

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 222

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+    }
+
+    ret = qemu_remote_savevm(o->vfu_mig_file, DEVICE(o->pci_dev));
+    if (ret) {
+        qemu_file_shutdown(o->vfu_mig_file);
+        return;
+    }
+
+    qemu_fflush(o->vfu_mig_file);
+}

Are you sure pre-copy is the state where you want to serialize the savevm data? IIUC pre-copy is the iterative state while the device is still running (e.g. when copying RAM but before devices are stopped). I expected savevm to happen when we reach stop-and-copy.

The reason why this matters is that we're saving the state of the device while the guest is still running and possibly interacting with the device. The destination won't have the final state of the device, it will have an earlier state of the device when we started migrating RAM!

Maybe I'm wrong, please double-check, but this looks like a bug.

Quoted 15 lines
hw/remote/vfio-user-obj.c +15 -0
+
+static void vfu_mig_state_running(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(OBJECT(o));
+    static int migrated_devs;
+    Error *local_err = NULL;
+    int ret;
+
+    ret = qemu_remote_loadvm(o->vfu_mig_file);
+    if (ret) {
+        error_setg(&error_abort, "vfu: failed to restore device state");
+        return;
+    }
+

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 247

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+    if (++migrated_devs == k->nr_devs) {
+        bdrv_invalidate_cache_all(&local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            return;
+        }
+
+        vm_start();
+    }
+}

This looks like it's intended for the destination side. Does this code work on the source side if the device is transitioned back into the running state?

Quoted 97 lines
hw/remote/vfio-user-obj.c +97 -0
+
+static void vfu_mig_state_stop(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(OBJECT(o));
+    static int migrated_devs;
+
+    /**
+     * note: calling bdrv_inactivate_all() is not the best approach.
+     *
+     *  Ideally, we would identify the block devices (if any) indirectly
+     *  linked (such as via a scs-hd device) to each of the migrated devices,
+     *  and inactivate them individually. This is essential while operating
+     *  the server in a storage daemon mode, with devices from different VMs.
+     *
+     *  However, we currently don't have this capability. As such, we need to
+     *  inactivate all devices at the same time when migration is completed.
+     */
+    if (++migrated_devs == k->nr_devs) {
+        bdrv_inactivate_all();
+    }
+}
+
+static int vfu_mig_transition(vfu_ctx_t *vfu_ctx, vfu_migr_state_t state)
+{
+    switch (state) {
+    case VFU_MIGR_STATE_RESUME:
+    case VFU_MIGR_STATE_STOP_AND_COPY:
+        break;
+    case VFU_MIGR_STATE_STOP:
+        vfu_mig_state_stop(vfu_ctx);
+        break;
+    case VFU_MIGR_STATE_PRE_COPY:
+        vfu_mig_state_precopy(vfu_ctx);
+        break;
+    case VFU_MIGR_STATE_RUNNING:
+        if (!runstate_is_running()) {
+            vfu_mig_state_running(vfu_ctx);
+        }
+        break;
+    default:
+        warn_report("vfu: Unknown migration state %d", state);
+    }
+
+    return 0;
+}
+
+static uint64_t vfu_mig_get_pending_bytes(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    return o->vfu_mig_buf_pending;
+}
+
+static int vfu_mig_prepare_data(vfu_ctx_t *vfu_ctx, uint64_t *offset,
+                                uint64_t *size)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset) {
+        *offset = 0;
+    }
+
+    if (size) {
+        *size = o->vfu_mig_buf_size;
+    }
+
+    return 0;
+}
+
+static ssize_t vfu_mig_read_data(vfu_ctx_t *vfu_ctx, void *buf,
+                                 uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset > o->vfu_mig_buf_size) {
+        return -1;
+    }
+
+    if ((offset + size) > o->vfu_mig_buf_size) {
+        warn_report("vfu: buffer overflow - check pending_bytes");
+        size = o->vfu_mig_buf_size - offset;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + offset), size);
+
+    o->vfu_mig_buf_pending -= size;
+
+    return size;
+}
+
+static ssize_t vfu_mig_write_data(vfu_ctx_t *vfu_ctx, void *data,
+                                  uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint64_t end = offset + size;
+

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 354

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+    if (end > o->vfu_mig_buf_size) {
+        o->vfu_mig_buf = g_realloc(o->vfu_mig_buf, end);
+        o->vfu_mig_buf_size = end;
+    }
+
+    memcpy((o->vfu_mig_buf + offset), data, size);
+
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_load, false);
+    }

Why open the file here where it's not accessed? I expected this to happen at the point where data has been fully written and we call qemu_remote_loadvm().

Quoted 35 lines
hw/remote/vfio-user-obj.c +21 -0
+
+    return size;
+}
+
+static int vfu_mig_data_written(vfu_ctx_t *vfu_ctx, uint64_t count)
+{
+    return 0;
+}
+
+static const vfu_migration_callbacks_t vfu_mig_cbs = {
+    .version = VFU_MIGR_CALLBACKS_VERS,
+    .transition = &vfu_mig_transition,
+    .get_pending_bytes = &vfu_mig_get_pending_bytes,
+    .prepare_data = &vfu_mig_prepare_data,
+    .read_data = &vfu_mig_read_data,
+    .data_written = &vfu_mig_data_written,
+    .write_data = &vfu_mig_write_data,
+};
+
 static void vfu_object_ctx_run(void *opaque)
 {
     VfuObject *o = opaque;
@@ -340,6 +615,7 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
     DeviceState *dev = NULL;
+    size_t migr_area_size;
     QemuThread thread;
     int ret;
 
@@ -401,6 +677,35 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
         return;
     }
 
+    /*

hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 690

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+     * TODO: The 0x20000 number used below is a temporary. We are working on
+     *     a cleaner fix for this.
+     *
+     *     The libvfio-user library assumes that the remote knows the size of
+     *     the data to be migrated at boot time, but that is not the case with
+     *     VMSDs, as it can contain a variable-size buffer. 0x20000 is used
+     *     as a sufficiently large buffer to demonstrate migration, but that
+     *     cannot be used as a solution.
+     *
+     */

libvfio-user has the vfu_migration_callbacks_t interface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?

#103 lore raw

RFC PATCH v2 11/11 vfio-user: acceptance test

In reply to Jagannathan Raman

Acceptance test for libvfio-user in QEMU

---
 MAINTAINERS                   |  1 +
 tests/acceptance/vfio-user.py | 94 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100644 tests/acceptance/vfio-user.py
MAINTAINERS +1 -0
index f9d8092..2c7332b 100644
@@ -3392,6 +3392,7 @@ F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
 F: hw/remote/vfio-user-obj.c
+F: tests/acceptance/vfio-user.py
 
 EBPF:
 M: Jason Wang <[email protected]>
tests/acceptance/vfio-user.py new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#104 lore raw

Re: [PATCH RFC server v2 00/11] vfio-user server in QEMU

In reply to Jagannathan Raman
Quoted 5 lines · Jagannathan Raman

Hi,

This series depends on the following series from
Elena Ufimtseva <[email protected]>:
[PATCH RFC v2 00/16] vfio-user implementation

Please send future revisions as separate email threads. Tools have trouble separating your series from the one you replied to.

You can use "Based-on" to let CI know that Elena's series needs to be applied first:

Based-on: <[email protected]>

#105 lore raw

Re: [PATCH RFC server v2 00/11] vfio-user server in QEMU

Quoted 11 lines · Stefan Hajnoczi
Quoted 5 lines · Jagannathan Raman

Hi,

This series depends on the following series from
Elena Ufimtseva <[email protected]>:
[PATCH RFC v2 00/16] vfio-user implementation

Quoted 7 lines

Please send future revisions as separate email threads. Tools have trouble separating your series from the one you replied to.

You can use "Based-on" to let CI know that Elena's series needs to be applied first:

Based-on: <[email protected]>

Thank you for letting us know, Stefan! Will do going forward.

#107 lore raw

Re: [PATCH RFC server v2 00/11] vfio-user server in QEMU

Quoted 7 lines · Stefan Hajnoczi

Hi Jag,
I have finished reviewing these patches and left comments. I didn't take a look at the libvfio-user's implementation.

Thank you for you comments, Stefan - we’ll get cracking on them. :)

--
Jag

Quoted 2 lines

Stefan