Skip to contentSkip to conversation

vfio-user implementation

RFC PATCH Superseded Elena Ufimtseva 9 participants 55 messages Jul 19, 2021 – Aug 16, 2021 Arm/Amlogic Meson SoC support

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover vfio-user implementation

Hi

We are happy to introduce the next stage of the multi-process QEMU project[1].

vfio-user is a protocol that allows a device to be emulated in a separate process outside of QEMU. It encapsulates the messages sent from QEMU to the kernel VFIO driver, and sends them to a remote process over a UNIX socket.

The vfio-user framework consists of 3 parts:
1) The protocol specification.
2) A server - the VFIO generic device in QEMU that exchanges the protocol messages with the client.
3) A client - remote process that emulates a device.

This patchset implements parts 1 and 2.
The protocol's specification can be found here [2]:
We also include this as the first patch of the series.

The libvfio-user project (https://github.com/nutanix/libvfio-user)
can be used by a remote process to handle the protocol to implement the third part.
We also worked on implementing a client and will be sending this patch series shortly.

Contributors:

John G Johnson <[email protected]>
John Levon <[email protected]>
Thanos Makatos <[email protected]>
Elena Ufimtseva <[email protected]>
Jagannathan Raman <[email protected]>

Please send your comments and questions!

Thank you.

References:
[1] https://wiki.qemu.org/Features/MultiProcessQEMU
[2] https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/

John G Johnson (18):
vfio-user: add VFIO base abstract class
vfio-user: define VFIO Proxy and communication functions
vfio-user: Define type vfio_user_pci_dev_info
vfio-user: connect vfio proxy to remote server
vfio-user: negotiate protocol with remote server
vfio-user: define vfio-user pci ops
vfio-user: VFIO container setup & teardown
vfio-user: get device info and get irq info
vfio-user: device region read/write
vfio-user: get region and DMA map/unmap operations
vfio-user: probe remote device's BARs
vfio-user: respond to remote DMA read/write requests
vfio_user: setup MSI/X interrupts and PCI config operations
vfio-user: vfio user device realize
vfio-user: pci reset
vfio-user: probe remote device ROM BAR
vfio-user: migration support
vfio-user: add migration cli options and version negotiation

Thanos Makatos (1):
vfio-user: introduce vfio-user protocol specification

 docs/devel/index.rst          |    1 +
 docs/devel/vfio-user.rst      | 1809 +++++++++++++++++++++++++++++++++
 hw/vfio/pci.h                 |   25 +-
 hw/vfio/user.h                |  279 +++++
 include/hw/vfio/vfio-common.h |    8 +
 hw/vfio/common.c              |  273 ++++-
 hw/vfio/migration.c           |   35 +-
 hw/vfio/pci.c                 |  547 ++++++++--
 hw/vfio/user.c                |  997 ++++++++++++++++++
 MAINTAINERS                   |   10 +
 hw/vfio/meson.build           |    1 +
 11 files changed, 3879 insertions(+), 106 deletions(-)
 create mode 100644 docs/devel/vfio-user.rst
 create mode 100644 hw/vfio/user.h
 create mode 100644 hw/vfio/user.c
Signature · 1 line
2.25.1
#1 lore raw

RFC PATCH 1/19 vfio-user: introduce vfio-user protocol specification

From: Thanos Makatos <[email protected]>

This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts.

This patch is sourced from:
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/

It has been earlier discussed as an RFC in:
"RFC: use VFIO over a UNIX domain socket to implement device offloading"

---
 docs/devel/index.rst     |    1 +
 docs/devel/vfio-user.rst | 1809 ++++++++++++++++++++++++++++++++++++++
 MAINTAINERS              |    6 +
 3 files changed, 1816 insertions(+)
 create mode 100644 docs/devel/vfio-user.rst
docs/devel/index.rst +1 -0
index 153979caf4..215eb4ff7a 100644
@@ -42,3 +42,4 @@ modifying QEMU's source code.
    multi-process
    ebpf_rss
    vfio-migration
+   vfio-user
docs/devel/vfio-user.rst new file +1809 -0

1,809 lines not on this page · raw message

MAINTAINERS +6 -0
index 4256ad1adb..12d69f3a45 100644
@@ -1880,6 +1880,12 @@ F: hw/vfio/ap.c
 F: docs/system/s390x/vfio-ap.rst
 L: [email protected]
 
+vfio-user
+M: John G Johnson <[email protected]>
+M: Thanos Makatos <[email protected]>
+S: Supported
+F: docs/devel/vfio-user.rst
+
 vhost
 M: Michael S. Tsirkin <[email protected]>
 S: Supported
Signature · 1 line
2.25.1
#2 lore raw

RFC PATCH 2/19 vfio-user: add VFIO base abstract class

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Add an abstract base class both the kernel driver and user socket implementations can use to share code.

---
 hw/vfio/pci.h | 25 ++++++++++++++++++--
 hw/vfio/pci.c | 63 ++++++++++++++++++++++++++++++++-------------------
 2 files changed, 63 insertions(+), 25 deletions(-)
hw/vfio/pci.h +23 -2
index 64777516d1..ba2f51d98f 100644
@@ -114,8 +114,13 @@ typedef struct VFIOMSIXInfo {
     unsigned long *pending;
 } VFIOMSIXInfo;
 
-#define TYPE_VFIO_PCI "vfio-pci"
-OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI)
+/*
+ * TYPE_VFIO_PCI_BASE is an abstract type used to share code
+ * between VFIO implementations that use a kernel driver
+ * with those that use user sockers.
+ */
+#define TYPE_VFIO_PCI_BASE "vfio-pci-base"
+OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_BASE)
 
 struct VFIOPCIDevice {
     PCIDevice pdev;
@@ -175,6 +180,22 @@ struct VFIOPCIDevice {
     Notifier irqchip_change_notifier;
 };
 
+#define TYPE_VFIO_PCI "vfio-pci"
+OBJECT_DECLARE_SIMPLE_TYPE(VFIOKernPCIDevice, VFIO_PCI)
+
+struct VFIOKernPCIDevice {
+    VFIOPCIDevice device;
+};
+
+#define TYPE_VFIO_USER_PCI "vfio-user-pci"
+OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI)
+
+struct VFIOUserPCIDevice {
+    VFIOPCIDevice device;
+    char *sock_name;
+    bool secure;
+};
+
 /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */
 static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device)
 {
hw/vfio/pci.c +40 -23

163 lines not on this page · raw message

Signature · 1 line
2.25.1
#3 lore raw

RFC PATCH 3/19 vfio-user: define VFIO Proxy and communication functions

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Add user.c and user.h files for vfio-user with the basic send and receive functions.

---
 hw/vfio/user.h                | 120 ++++++++++++++
 include/hw/vfio/vfio-common.h |   2 +
 hw/vfio/user.c                | 286 ++++++++++++++++++++++++++++++++++
 MAINTAINERS                   |   4 +
 hw/vfio/meson.build           |   1 +
 5 files changed, 413 insertions(+)
 create mode 100644 hw/vfio/user.h
 create mode 100644 hw/vfio/user.c
hw/vfio/user.h new file +120 -0
new file mode 100644
index 0000000000..cdbc074579
@@ -0,0 +1,120 @@
+#ifndef VFIO_USER_H
+#define VFIO_USER_H
+
+/*
+ * vfio protocol over a UNIX socket.
+ *
+ * Copyright © 2018, 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Each message has a standard header that describes the command
+ * being sent, which is almost always a VFIO ioctl().
+ *
+ * The header may be followed by command-specfic data, such as the
+ * region and offset info for read and write commands.
+ */
+
+/* commands */
+enum vfio_user_command {
+    VFIO_USER_VERSION                   = 1,
+    VFIO_USER_DMA_MAP                   = 2,
+    VFIO_USER_DMA_UNMAP                 = 3,
+    VFIO_USER_DEVICE_GET_INFO           = 4,
+    VFIO_USER_DEVICE_GET_REGION_INFO    = 5,
+    VFIO_USER_DEVICE_GET_REGION_IO_FDS  = 6,
+    VFIO_USER_DEVICE_GET_IRQ_INFO       = 7,
+    VFIO_USER_DEVICE_SET_IRQS           = 8,
+    VFIO_USER_REGION_READ               = 9,
+    VFIO_USER_REGION_WRITE              = 10,
+    VFIO_USER_DMA_READ                  = 11,
+    VFIO_USER_DMA_WRITE                 = 12,
+    VFIO_USER_DEVICE_RESET              = 13,
+    VFIO_USER_DIRTY_PAGES               = 14,
+    VFIO_USER_MAX,
+};
+
+/* flags */
+#define VFIO_USER_REQUEST       0x0
+#define VFIO_USER_REPLY         0x1
+#define VFIO_USER_TYPE          0xF
+
+#define VFIO_USER_NO_REPLY      0x10
+#define VFIO_USER_ERROR         0x20
+
+typedef struct vfio_user_hdr {
+    uint16_t id;
+    uint16_t command;
+    uint32_t size;
+    uint32_t flags;
+    uint32_t error_reply;
+} vfio_user_hdr_t;
+
+/*
+ * VFIO_USER_VERSION
+ */
+#define VFIO_USER_MAJOR_VER     0
+#define VFIO_USER_MINOR_VER     0
+
+struct vfio_user_version {
+    vfio_user_hdr_t hdr;
+    uint16_t major;
+    uint16_t minor;
+    char capabilities[];
+};
+
+#define VFIO_USER_DEF_MAX_FDS   8
+#define VFIO_USER_MAX_MAX_FDS   16
+
+#define VFIO_USER_DEF_MAX_XFER  (1024 * 1024)
+#define VFIO_USER_MAX_MAX_XFER  (64 * 1024 * 1024)
+
+typedef struct VFIOUserFDs {
+    int send_fds;
+    int recv_fds;
+    int *fds;
+} VFIOUserFDs;
+
+typedef struct VFIOUserReply {
+    QTAILQ_ENTRY(VFIOUserReply) next;
+    vfio_user_hdr_t *msg;
+    VFIOUserFDs *fds;
+    int rsize;
+    uint32_t id;
+    QemuCond cv;
+    uint8_t complete;
+} VFIOUserReply;
+
+enum proxy_state {
+    CONNECTED = 1,
+    RECV_ERROR = 2,
+    CLOSING = 3,
+    CLOSED = 4,
+};
+
+typedef struct VFIOProxy {
+    QLIST_ENTRY(VFIOProxy) next;
+    char *sockname;
+    struct QIOChannel *ioc;
+    int (*request)(void *opaque, char *buf, VFIOUserFDs *fds);
+    void *reqarg;
+    int flags;
+    QemuCond close_cv;
+
+    /*
+     * above only changed when iolock is held
+     * below are protected by per-proxy lock
+     */
+    QemuMutex lock;
+    QTAILQ_HEAD(, VFIOUserReply) free;
+    QTAILQ_HEAD(, VFIOUserReply) pending;
+    enum proxy_state state;
+    int close_wait;
+} VFIOProxy;
+
+#define VFIO_PROXY_CLIENT       0x1
+
+void vfio_user_recv(void *opaque);
+void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
+#endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +2 -0
index 8af11b0a76..f43dc6e5d0 100644
@@ -75,6 +75,7 @@ typedef struct VFIOAddressSpace {
 } VFIOAddressSpace;
 
 struct VFIOGroup;
+typedef struct VFIOProxy VFIOProxy;
 
 typedef struct VFIOContainer {
     VFIOAddressSpace *space;
@@ -143,6 +144,7 @@ typedef struct VFIODevice {
     VFIOMigration *migration;
     Error *migration_blocker;
     OnOffAuto pre_copy_dirty_page_tracking;
+    VFIOProxy *proxy;
 } VFIODevice;
 
 struct VFIODeviceOps {
hw/vfio/user.c new file +286 -0

286 lines not on this page · raw message

MAINTAINERS +4 -0
index 12d69f3a45..aa4df6c418 100644
@@ -1883,8 +1883,12 @@ L: [email protected]
 vfio-user
 M: John G Johnson <[email protected]>
 M: Thanos Makatos <[email protected]>
+M: Elena Ufimtseva <[email protected]>
+M: Jagannathan Raman <[email protected]>
 S: Supported
 F: docs/devel/vfio-user.rst
+F: hw/vfio/user.c
+F: hw/vfio/user.h
 
 vhost
 M: Michael S. Tsirkin <[email protected]>
hw/vfio/meson.build +1 -0
index da9af297a0..739b30be73 100644
@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files(
   'display.c',
   'pci-quirks.c',
   'pci.c',
+  'user.c',
 ))
 vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c'))
 vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))
Signature · 1 line
2.25.1
#4 lore raw

Re: [PATCH RFC 03/19] vfio-user: define VFIO Proxy and communication functions

Quoted 8 lines · Elena Ufimtseva

From: John G Johnson <[email protected]>

Add user.c and user.h files for vfio-user with the basic send and receive functions.

Quoted 9 lines
---
 hw/vfio/user.h                | 120 ++++++++++++++
 include/hw/vfio/vfio-common.h |   2 +
 hw/vfio/user.c                | 286 ++++++++++++++++++++++++++++++++++
 MAINTAINERS                   |   4 +
 hw/vfio/meson.build           |   1 +
 5 files changed, 413 insertions(+)
 create mode 100644 hw/vfio/user.h
 create mode 100644 hw/vfio/user.c

The multi-threading, coroutine, and blocking I/O requirements of vfio_user_recv() and vfio_user_send_reply() are unclear to me. Please document them so it's clear what environment they can be called from. I guess they are not called from coroutines and proxy->ioc is a blocking IOChannel?

Quoted 49 lines
hw/vfio/user.h new file +42 -0
new file mode 100644
index 0000000000..cdbc074579
@@ -0,0 +1,120 @@
+#ifndef VFIO_USER_H
+#define VFIO_USER_H
+
+/*
+ * vfio protocol over a UNIX socket.
+ *
+ * Copyright © 2018, 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Each message has a standard header that describes the command
+ * being sent, which is almost always a VFIO ioctl().
+ *
+ * The header may be followed by command-specfic data, such as the
+ * region and offset info for read and write commands.
+ */
+
+/* commands */
+enum vfio_user_command {
+    VFIO_USER_VERSION                   = 1,
+    VFIO_USER_DMA_MAP                   = 2,
+    VFIO_USER_DMA_UNMAP                 = 3,
+    VFIO_USER_DEVICE_GET_INFO           = 4,
+    VFIO_USER_DEVICE_GET_REGION_INFO    = 5,
+    VFIO_USER_DEVICE_GET_REGION_IO_FDS  = 6,
+    VFIO_USER_DEVICE_GET_IRQ_INFO       = 7,
+    VFIO_USER_DEVICE_SET_IRQS           = 8,
+    VFIO_USER_REGION_READ               = 9,
+    VFIO_USER_REGION_WRITE              = 10,
+    VFIO_USER_DMA_READ                  = 11,
+    VFIO_USER_DMA_WRITE                 = 12,
+    VFIO_USER_DEVICE_RESET              = 13,
+    VFIO_USER_DIRTY_PAGES               = 14,
+    VFIO_USER_MAX,
+};
+
+/* flags */
+#define VFIO_USER_REQUEST       0x0
+#define VFIO_USER_REPLY         0x1
+#define VFIO_USER_TYPE          0xF
+

hw/vfio/user.h line 52

Quoted 10 lines
hw/vfio/user.h new file +10 -0
+#define VFIO_USER_NO_REPLY      0x10
+#define VFIO_USER_ERROR         0x20
+
+typedef struct vfio_user_hdr {
+    uint16_t id;
+    uint16_t command;
+    uint32_t size;
+    uint32_t flags;
+    uint32_t error_reply;
+} vfio_user_hdr_t;

Please use QEMU coding style in QEMU code (i.e. not shared with Linux or external libraries):

  typedef struct {
      ...
  } VfioUserHdr;

You can also specify the struct VfioUserHdr tag if you want but it's only necessary to reference the struct before the end of the typedef definition.

https://qemu-project.gitlab.io/qemu/devel/style.html

Quoted 15 lines
hw/vfio/user.h new file +15 -0
+
+/*
+ * VFIO_USER_VERSION
+ */
+#define VFIO_USER_MAJOR_VER     0
+#define VFIO_USER_MINOR_VER     0
+
+struct vfio_user_version {
+    vfio_user_hdr_t hdr;
+    uint16_t major;
+    uint16_t minor;
+    char capabilities[];
+};
+
+#define VFIO_USER_DEF_MAX_FDS   8

hw/vfio/user.h line 77

Quoted 10 lines
hw/vfio/user.h new file +10 -0
+#define VFIO_USER_MAX_MAX_FDS   16
+
+#define VFIO_USER_DEF_MAX_XFER  (1024 * 1024)
+#define VFIO_USER_MAX_MAX_XFER  (64 * 1024 * 1024)
+
+typedef struct VFIOUserFDs {
+    int send_fds;
+    int recv_fds;
+    int *fds;
+} VFIOUserFDs;

I think around here we switch from vfio-user spec definitions to QEMU implementation details. It might be nice to keep the vfio-user spec definitions in a separate header file so the boundary is clear.

hw/vfio/user.h line 86

Quoted 9 lines
hw/vfio/user.h new file +9 -0
+
+typedef struct VFIOUserReply {
+    QTAILQ_ENTRY(VFIOUserReply) next;
+    vfio_user_hdr_t *msg;
+    VFIOUserFDs *fds;
+    int rsize;
+    uint32_t id;
+    QemuCond cv;
+    uint8_t complete;

Please use bool.

hw/vfio/user.h line 94

Quoted 8 lines
hw/vfio/user.h new file +8 -0
+} VFIOUserReply;
+
+enum proxy_state {
+    CONNECTED = 1,
+    RECV_ERROR = 2,
+    CLOSING = 3,
+    CLOSED = 4,
+};

These enum values probably need a prefix (VFIO_PROXY_*). Generic short names like CONNECTED, CLOSED, etc could lead to namespace collisions. Enum constants are in the global namespace.

hw/vfio/user.h line 106

Quoted 12 lines
hw/vfio/user.h new file +12 -0
+
+typedef struct VFIOProxy {
+    QLIST_ENTRY(VFIOProxy) next;
+    char *sockname;
+    struct QIOChannel *ioc;
+    int (*request)(void *opaque, char *buf, VFIOUserFDs *fds);
+    void *reqarg;
+    int flags;
+    QemuCond close_cv;
+
+    /*
+     * above only changed when iolock is held

Please use "BQL" instead of "iolock". git grep shows many results for BQL and the only result for iolock is in mpqemu code.

hw/vfio/user.h line 113

Quoted 7 lines
hw/vfio/user.h new file +7 -0
+     * below are protected by per-proxy lock
+     */
+    QemuMutex lock;
+    QTAILQ_HEAD(, VFIOUserReply) free;
+    QTAILQ_HEAD(, VFIOUserReply) pending;
+    enum proxy_state state;
+    int close_wait;

Is this a bool? Please use bool.

hw/vfio/user.h line 116

Quoted 3 lines
hw/vfio/user.h new file +3 -0
+} VFIOProxy;
+
+#define VFIO_PROXY_CLIENT       0x1

A comment that shows which field VFIO_PROXY_CLIENT relates would make this clearer:

  /* VFIOProxy->flags */
  #define VFIO_PROXY_CLIENT 0x1
Quoted 60 lines
hw/vfio/user.h new file +4 -0
+
+void vfio_user_recv(void *opaque);
+void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
+#endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +2 -0
index 8af11b0a76..f43dc6e5d0 100644
@@ -75,6 +75,7 @@ typedef struct VFIOAddressSpace {
 } VFIOAddressSpace;
 
 struct VFIOGroup;
+typedef struct VFIOProxy VFIOProxy;
 
 typedef struct VFIOContainer {
     VFIOAddressSpace *space;
@@ -143,6 +144,7 @@ typedef struct VFIODevice {
     VFIOMigration *migration;
     Error *migration_blocker;
     OnOffAuto pre_copy_dirty_page_tracking;
+    VFIOProxy *proxy;
 } VFIODevice;
 
 struct VFIODeviceOps {
hw/vfio/user.c new file +30 -0
new file mode 100644
index 0000000000..021d5540e0
@@ -0,0 +1,286 @@
+/*
+ * vfio protocol over a UNIX socket.
+ *
+ * Copyright © 2018, 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include <linux/vfio.h>
+#include <sys/ioctl.h>
+
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "qemu/main-loop.h"
+#include "hw/hw.h"
+#include "hw/vfio/vfio-common.h"
+#include "hw/vfio/vfio.h"
+#include "qemu/sockets.h"
+#include "io/channel.h"
+#include "io/channel-util.h"
+#include "sysemu/iothread.h"
+#include "user.h"
+
+static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER;
+static IOThread *vfio_user_iothread;
+static void vfio_user_send_locked(VFIOProxy *proxy, vfio_user_hdr_t *msg,
+                                  VFIOUserFDs *fds);

hw/vfio/user.c line 40

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg,
+                           VFIOUserFDs *fds);
+static void vfio_user_shutdown(VFIOProxy *proxy);
+
+static void vfio_user_shutdown(VFIOProxy *proxy)
+{
+    qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL);
+    qio_channel_set_aio_fd_handler(proxy->ioc,
+                                   iothread_get_aio_context(vfio_user_iothread),
+                                   NULL, NULL, NULL);

There is no other qio_channel_set_aio_fd_handler() call in this patch. Why is this one necessary?

Quoted 8 lines
hw/vfio/user.c new file +8 -0
+}
+
+void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret)
+{
+    vfio_user_hdr_t *hdr = (vfio_user_hdr_t *)buf;
+
+    /*
+     * convert header to associated reply

hw/vfio/user.c line 58

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+     * positive ret is reply size, negative is error code
+     */
+    hdr->flags = VFIO_USER_REPLY;
+    if (ret > 0) {
+        hdr->size = ret;
+    } else if (ret < 0) {
+        hdr->flags |= VFIO_USER_ERROR;
+        hdr->error_reply = -ret;
+        hdr->size = sizeof(*hdr);
+    }

assert(ret != 0)? That case doesn't seem to be defined so maybe an assertion is worthwhile.

Quoted 14 lines
hw/vfio/user.c new file +14 -0
+    vfio_user_send(proxy, hdr, NULL);
+}
+
+void vfio_user_recv(void *opaque)
+{
+    VFIODevice *vbasedev = opaque;
+    VFIOProxy *proxy = vbasedev->proxy;
+    VFIOUserReply *reply = NULL;
+    g_autofree int *fdp = NULL;
+    VFIOUserFDs reqfds = { 0, 0, fdp };
+    vfio_user_hdr_t msg;
+    struct iovec iov = {
+        .iov_base = &msg,
+        .iov_len = sizeof(msg),

hw/vfio/user.c line 82

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+    };
+    int isreply, i, ret;
+    size_t msgleft, numfds = 0;
+    char *data = NULL;
+    g_autofree char *buf = NULL;
+    Error *local_err = NULL;
+
+    qemu_mutex_lock(&proxy->lock);
+    if (proxy->state == CLOSING) {
+        qemu_mutex_unlock(&proxy->lock);

QEMU_LOCK_GUARD() automatically unlocks mutexes when the function returns and is less error-prone than manual lock/unlock calls.

hw/vfio/user.c line 90

Quoted 8 lines
hw/vfio/user.c new file +8 -0
+        return;
+    }
+
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);
+    if (ret <= 0) {
+        /* read error or other side closed connection */
+        error_setg_errno(&local_err, errno, "vfio_user_recv read error");

This will trigger an assertion failure when local_err was already set by qio_channel_readv_full():

static void error_setv(Error **errp,
const char *src, int line, const char *func, ErrorClass err_class, const char *fmt, va_list ap, const char *suffix)

  {
      Error *err;
      int saved_errno = errno;
  
      if (errp == NULL) {
          return;
      }
      assert(*errp == NULL);
      ^^^^^^^^^^^^^^^^^^^^^^

I think this error_setg_errno() call should be dropped. You can use error_prepend() if you'd like to add more information to the error message from qio_channel_readv_full().

Quoted 19 lines
hw/vfio/user.c new file +19 -0
+        goto fatal;
+    }
+
+    if (ret < sizeof(msg)) {
+        error_setg(&local_err, "vfio_user_recv short read of header");
+        goto err;
+    }
+
+    /*
+     * For replies, find the matching pending request
+     */
+    switch (msg.flags & VFIO_USER_TYPE) {
+    case VFIO_USER_REQUEST:
+        isreply = 0;
+        break;
+    case VFIO_USER_REPLY:
+        isreply = 1;
+        break;
+    default:

hw/vfio/user.c line 119

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+        error_setg(&local_err, "vfio_user_recv unknown message type");
+        goto err;
+    }
+
+    if (isreply) {
+        QTAILQ_FOREACH(reply, &proxy->pending, next) {
+            if (msg.id == reply->id) {
+                break;
+            }
+        }

I'm surprised to see this loop since proxy->lock prevents additional requests from being sent while we're trying to receive a message. Can there really be multiple replies pending with this locking scheme?

Quoted 24 lines
hw/vfio/user.c new file +24 -0
+        if (reply == NULL) {
+            error_setg(&local_err, "vfio_user_recv unexpected reply");
+            goto err;
+        }
+        QTAILQ_REMOVE(&proxy->pending, reply, next);
+
+        /*
+         * Process any received FDs
+         */
+        if (numfds != 0) {
+            if (reply->fds == NULL || reply->fds->recv_fds < numfds) {
+                error_setg(&local_err, "vfio_user_recv unexpected FDs");
+                goto err;
+            }
+            reply->fds->recv_fds = numfds;
+            memcpy(reply->fds->fds, fdp, numfds * sizeof(int));
+        }
+
+    } else {
+        /*
+         * The client doesn't expect any FDs in requests, but
+         * they will be expected on the server
+         */
+        if (numfds != 0 && (proxy->flags & VFIO_PROXY_CLIENT)) {

hw/vfio/user.c line 153

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+            error_setg(&local_err, "vfio_user_recv fd in client reply");
+            goto err;
+        }
+        reqfds.recv_fds = numfds;
+    }
+
+    /*
+     * put the whole message into a single buffer
+     */
+    msgleft = msg.size - sizeof(msg);

msg.size has not been validated so this could underflow. Please validate all inputs so malicious servers/clients cannot crash or compromise the program.

hw/vfio/user.c line 155

Quoted 2 lines
hw/vfio/user.c new file +2 -0
+    if (isreply) {
+        if (msg.size > reply->rsize) {

rsize is an int. Should it be uint32_t like msg.size?

hw/vfio/user.c line 166

Quoted 11 lines
hw/vfio/user.c new file +11 -0
+            error_setg(&local_err,
+                       "vfio_user_recv reply larger than recv buffer");
+            goto fatal;
+        }
+        *reply->msg = msg;
+        data = (char *)reply->msg + sizeof(msg);
+    } else {
+        if (msg.size > max_xfer_size) {
+            error_setg(&local_err, "vfio_user_recv request larger than max");
+            goto fatal;
+        }

Missing check to prevent buffer overflow:

  if (msg.size < sizeof(msg)) {
      error_setg(&local_err, "vfio_user_recv request too small");
      goto fatal;
  }
Quoted 14 lines
hw/vfio/user.c new file +14 -0
+        buf = g_malloc0(msg.size);
+        memcpy(buf, &msg, sizeof(msg));
+        data = buf + sizeof(msg);
+    }
+
+    if (msgleft != 0) {
+        ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err);
+        if (ret < 0) {
+            goto fatal;
+        }
+        if (ret != msgleft) {
+            error_setg(&local_err, "vfio_user_recv short read of msg body");
+            goto err;
+        }

hw/vfio/user.c line 190

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+    }
+
+    /*
+     * Replies signal a waiter, requests get processed by vfio code
+     * that may assume the iothread lock is held.
+     */
+    qemu_mutex_unlock(&proxy->lock);
+    if (isreply) {
+        reply->complete = 1;
+        qemu_cond_signal(&reply->cv);

signal must be called with the mutex held to avoid race conditions. If the waiter acquires the lock and still sees complete == 0, then we signal before wait is entered, the signal is missed and the waiter is stuck.

hw/vfio/user.c line 196

Quoted 6 lines
hw/vfio/user.c new file +6 -0
+    } else {
+        qemu_mutex_lock_iothread();
+        /*
+         * make sure proxy wasn't closed while we waited
+         * checking without holding the proxy lock is safe
+         * since state is only set to CLOSING when iolock is held

s/iolock/the BQL/

Quoted 16 lines
hw/vfio/user.c new file +16 -0
+         */
+        if (proxy->state != CLOSING) {
+            ret = proxy->request(proxy->reqarg, buf, &reqfds);
+            if (ret < 0 && !(msg.flags & VFIO_USER_NO_REPLY)) {
+                vfio_user_send_reply(proxy, buf, ret);
+            }
+        }
+        qemu_mutex_unlock_iothread();
+    }
+
+    return;
+ fatal:
+    vfio_user_shutdown(proxy);
+    proxy->state = RECV_ERROR;
+
+ err:

hw/vfio/user.c line 222

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+    qemu_mutex_unlock(&proxy->lock);
+    for (i = 0; i < numfds; i++) {
+        close(fdp[i]);
+    }
+    if (reply != NULL) {
+        /* force an error to keep sending thread from hanging */
+        reply->msg->flags |= VFIO_USER_ERROR;
+        reply->msg->error_reply = EINVAL;
+        reply->complete = 1;
+        qemu_cond_signal(&reply->cv);

This has the race condition too.

Quoted 32 lines
hw/vfio/user.c new file +32 -0
+    }
+    error_report_err(local_err);
+}
+
+static void vfio_user_send_locked(VFIOProxy *proxy, vfio_user_hdr_t *msg,
+                                  VFIOUserFDs *fds)
+{
+    struct iovec iov = {
+        .iov_base = msg,
+        .iov_len = msg->size,
+    };
+    size_t numfds = 0;
+    int msgleft, ret, *fdp = NULL;
+    char *buf;
+    Error *local_err = NULL;
+
+    if (proxy->state != CONNECTED) {
+        msg->flags |= VFIO_USER_ERROR;
+        msg->error_reply = ECONNRESET;
+        return;
+    }
+
+    if (fds != NULL && fds->send_fds != 0) {
+        numfds = fds->send_fds;
+        fdp = fds->fds;
+    }
+    ret = qio_channel_writev_full(proxy->ioc, &iov, 1, fdp, numfds, &local_err);
+    if (ret < 0) {
+        goto err;
+    }
+    if (ret == msg->size) {
+        return;

hw/vfio/user.c line 264

Quoted 10 lines
hw/vfio/user.c new file +10 -0
+    }
+
+    buf = iov.iov_base + ret;
+    msgleft = iov.iov_len - ret;
+    do {
+        ret = qio_channel_write(proxy->ioc, buf, msgleft, &local_err);
+        if (ret < 0) {
+            goto err;
+        }
+        buf += ret, msgleft -= ret;

Please use semicolon. Comma operators are rare, requiring readers to check their exact semantics. There is no need to use comma here.

hw/vfio/user.c line 269

Quoted 5 lines
hw/vfio/user.c new file +5 -0
+    } while (msgleft != 0);
+    return;
+
+ err:
+    error_report_err(local_err);

State remains unchanged and msg->error_reply isn't set?

Quoted 49 lines
hw/vfio/user.c new file +17 -0
+}
+
+static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg,
+                           VFIOUserFDs *fds)
+{
+    bool iolock = qemu_mutex_iothread_locked();
+
+    if (iolock) {
+        qemu_mutex_unlock_iothread();
+    }
+    qemu_mutex_lock(&proxy->lock);
+    vfio_user_send_locked(proxy, msg, fds);
+    qemu_mutex_unlock(&proxy->lock);
+    if (iolock) {
+        qemu_mutex_lock_iothread();
+    }
+}
MAINTAINERS +4 -0
index 12d69f3a45..aa4df6c418 100644
@@ -1883,8 +1883,12 @@ L: [email protected]
 vfio-user
 M: John G Johnson <[email protected]>
 M: Thanos Makatos <[email protected]>
+M: Elena Ufimtseva <[email protected]>
+M: Jagannathan Raman <[email protected]>
 S: Supported
 F: docs/devel/vfio-user.rst
+F: hw/vfio/user.c
+F: hw/vfio/user.h
 
 vhost
 M: Michael S. Tsirkin <[email protected]>
hw/vfio/meson.build +1 -0
index da9af297a0..739b30be73 100644
@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files(
   'display.c',
   'pci-quirks.c',
   'pci.c',
+  'user.c',
 ))
 vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c'))
 vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))
Signature · 1 line
2.25.1
#5 lore raw

Re: [PATCH RFC 03/19] vfio-user: define VFIO Proxy and communication functions

Quoted 23 lines · Stefan Hajnoczi
Quoted 9 lines · Elena Ufimtseva

From: John G Johnson <[email protected]>

Add user.c and user.h files for vfio-user with the basic send and receive functions.

---

Quoted 8 lines
hw/vfio/user.h                | 120 ++++++++++++++
include/hw/vfio/vfio-common.h |   2 +
hw/vfio/user.c                | 286 ++++++++++++++++++++++++++++++++++
MAINTAINERS                   |   4 +
hw/vfio/meson.build           |   1 +
5 files changed, 413 insertions(+)

create mode 100644 hw/vfio/user.h
create mode 100644 hw/vfio/user.c

Quoted 6 lines

The multi-threading, coroutine, and blocking I/O requirements of vfio_user_recv() and vfio_user_send_reply() are unclear to me. Please document them so it's clear what environment they can be called from. I guess they are not called from coroutines and proxy->ioc is a blocking IOChannel?

Yes to both, moreover, a block comment above vfio_user_recv() would
be useful. The call to setup vfio_user_recv() as the socket handler isn’t in this patch, do you want the series re-org’d?

Quoted 65 lines
Quoted 49 lines
hw/vfio/user.h new file +42 -0
new file mode 100644
index 0000000000..cdbc074579
@@ -0,0 +1,120 @@
+#ifndef VFIO_USER_H
+#define VFIO_USER_H
+
+/*
+ * vfio protocol over a UNIX socket.
+ *
+ * Copyright © 2018, 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Each message has a standard header that describes the command
+ * being sent, which is almost always a VFIO ioctl().
+ *
+ * The header may be followed by command-specfic data, such as the
+ * region and offset info for read and write commands.
+ */
+
+/* commands */
+enum vfio_user_command {
+    VFIO_USER_VERSION                   = 1,
+    VFIO_USER_DMA_MAP                   = 2,
+    VFIO_USER_DMA_UNMAP                 = 3,
+    VFIO_USER_DEVICE_GET_INFO           = 4,
+    VFIO_USER_DEVICE_GET_REGION_INFO    = 5,
+    VFIO_USER_DEVICE_GET_REGION_IO_FDS  = 6,
+    VFIO_USER_DEVICE_GET_IRQ_INFO       = 7,
+    VFIO_USER_DEVICE_SET_IRQS           = 8,
+    VFIO_USER_REGION_READ               = 9,
+    VFIO_USER_REGION_WRITE              = 10,
+    VFIO_USER_DMA_READ                  = 11,
+    VFIO_USER_DMA_WRITE                 = 12,
+    VFIO_USER_DEVICE_RESET              = 13,
+    VFIO_USER_DIRTY_PAGES               = 14,
+    VFIO_USER_MAX,
+};
+
+/* flags */
+#define VFIO_USER_REQUEST       0x0
+#define VFIO_USER_REPLY         0x1
+#define VFIO_USER_TYPE          0xF
+

hw/vfio/user.h line 52

Quoted 10 lines
hw/vfio/user.h new file +10 -0
+#define VFIO_USER_NO_REPLY      0x10
+#define VFIO_USER_ERROR         0x20
+
+typedef struct vfio_user_hdr {
+    uint16_t id;
+    uint16_t command;
+    uint32_t size;
+    uint32_t flags;
+    uint32_t error_reply;
+} vfio_user_hdr_t;

Please use QEMU coding style in QEMU code (i.e. not shared with Linux or external libraries):

 typedef struct {
     ...
Quoted 8 lines

} VfioUserHdr;

You can also specify the struct VfioUserHdr tag if you want but it's only necessary to reference the struct before the end of the typedef definition.

https://qemu-project.gitlab.io/qemu/devel/style.html

OK

Quoted 21 lines
Quoted 21 lines
hw/vfio/user.h new file +21 -0
+
+/*
+ * VFIO_USER_VERSION
+ */
+#define VFIO_USER_MAJOR_VER     0
+#define VFIO_USER_MINOR_VER     0
+
+struct vfio_user_version {
+    vfio_user_hdr_t hdr;
+    uint16_t major;
+    uint16_t minor;
+    char capabilities[];
+};
+
+#define VFIO_USER_DEF_MAX_FDS   8
+#define VFIO_USER_MAX_MAX_FDS   16
+
+#define VFIO_USER_DEF_MAX_XFER  (1024 * 1024)
+#define VFIO_USER_MAX_MAX_XFER  (64 * 1024 * 1024)
+
+typedef struct VFIOUserFDs {
Quoted 9 lines

hw/vfio/user.h line 77

Quoted 4 lines
hw/vfio/user.h new file +4 -0
+    int send_fds;
+    int recv_fds;
+    int *fds;
+} VFIOUserFDs;

I think around here we switch from vfio-user spec definitions to QEMU implementation details. It might be nice to keep the vfio-user spec definitions in a separate header file so the boundary is clear.

OK

Quoted 12 lines

hw/vfio/user.h line 86

Quoted 9 lines
hw/vfio/user.h new file +9 -0
+
+typedef struct VFIOUserReply {
+    QTAILQ_ENTRY(VFIOUserReply) next;
+    vfio_user_hdr_t *msg;
+    VFIOUserFDs *fds;
+    int rsize;
+    uint32_t id;
+    QemuCond cv;
+    uint8_t complete;

Please use bool.

OK

Quoted 4 lines
Quoted 4 lines
hw/vfio/user.h new file +4 -0
+} VFIOUserReply;
+
+enum proxy_state {
+    CONNECTED = 1,
Quoted 9 lines

hw/vfio/user.h line 94

Quoted 4 lines
hw/vfio/user.h new file +4 -0
+    RECV_ERROR = 2,
+    CLOSING = 3,
+    CLOSED = 4,
+};

These enum values probably need a prefix (VFIO_PROXY_*). Generic short names like CONNECTED, CLOSED, etc could lead to namespace collisions. Enum constants are in the global namespace.

OK

Quoted 7 lines
Quoted 7 lines
hw/vfio/user.h new file +7 -0
+
+typedef struct VFIOProxy {
+    QLIST_ENTRY(VFIOProxy) next;
+    char *sockname;
+    struct QIOChannel *ioc;
+    int (*request)(void *opaque, char *buf, VFIOUserFDs *fds);
+    void *reqarg;
Quoted 9 lines

hw/vfio/user.h line 106

Quoted 5 lines
hw/vfio/user.h new file +5 -0
+    int flags;
+    QemuCond close_cv;
+
+    /*
+     * above only changed when iolock is held

Please use "BQL" instead of "iolock". git grep shows many results for BQL and the only result for iolock is in mpqemu code.

OK

Quoted 9 lines

hw/vfio/user.h line 113

Quoted 7 lines
hw/vfio/user.h new file +7 -0
+     * below are protected by per-proxy lock
+     */
+    QemuMutex lock;
+    QTAILQ_HEAD(, VFIOUserReply) free;
+    QTAILQ_HEAD(, VFIOUserReply) pending;
+    enum proxy_state state;
+    int close_wait;

Is this a bool? Please use bool.

yes it’s a bool

Quoted 10 lines

hw/vfio/user.h line 116

Quoted 4 lines
hw/vfio/user.h new file +3 -0
+} VFIOProxy;
+
+#define VFIO_PROXY_CLIENT       0x1

A comment that shows which field VFIO_PROXY_CLIENT relates would make this clearer:

/* VFIOProxy->flags */
#define VFIO_PROXY_CLIENT 0x1

OK

Quoted 65 lines
Quoted 65 lines
hw/vfio/user.h new file +4 -0
+
+void vfio_user_recv(void *opaque);
+void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
+#endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +2 -0
index 8af11b0a76..f43dc6e5d0 100644
@@ -75,6 +75,7 @@ typedef struct VFIOAddressSpace {
 } VFIOAddressSpace;
 
 struct VFIOGroup;
+typedef struct VFIOProxy VFIOProxy;
 
 typedef struct VFIOContainer {
    VFIOAddressSpace *space;
@@ -143,6 +144,7 @@ typedef struct VFIODevice {
    VFIOMigration *migration;
    Error *migration_blocker;
    OnOffAuto pre_copy_dirty_page_tracking;
+    VFIOProxy *proxy;
 } VFIODevice;
 
 struct VFIODeviceOps {
hw/vfio/user.c new file +35 -0
new file mode 100644
index 0000000000..021d5540e0
@@ -0,0 +1,286 @@
+/*
+ * vfio protocol over a UNIX socket.
+ *
+ * Copyright © 2018, 2021 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include <linux/vfio.h>
+#include <sys/ioctl.h>
+
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "qemu/main-loop.h"
+#include "hw/hw.h"
+#include "hw/vfio/vfio-common.h"
+#include "hw/vfio/vfio.h"
+#include "qemu/sockets.h"
+#include "io/channel.h"
+#include "io/channel-util.h"
+#include "sysemu/iothread.h"
+#include "user.h"
+
+static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER;
+static IOThread *vfio_user_iothread;
+static void vfio_user_send_locked(VFIOProxy *proxy, vfio_user_hdr_t *msg,
+                                  VFIOUserFDs *fds);
+static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg,
+                           VFIOUserFDs *fds);
+static void vfio_user_shutdown(VFIOProxy *proxy);
+
+static void vfio_user_shutdown(VFIOProxy *proxy)
Quoted 9 lines

hw/vfio/user.c line 40

Quoted 5 lines
hw/vfio/user.c new file +5 -0
+{
+    qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL);
+    qio_channel_set_aio_fd_handler(proxy->ioc,
+                                   iothread_get_aio_context(vfio_user_iothread),
+                                   NULL, NULL, NULL);

There is no other qio_channel_set_aio_fd_handler() call in this patch. Why is this one necessary?

See first comment.

Quoted 13 lines
Quoted 13 lines
hw/vfio/user.c new file +13 -0
+}
+
+void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret)
+{
+    vfio_user_hdr_t *hdr = (vfio_user_hdr_t *)buf;
+
+    /*
+     * convert header to associated reply
+     * positive ret is reply size, negative is error code
+     */
+    hdr->flags = VFIO_USER_REPLY;
+    if (ret > 0) {
+        hdr->size = ret;
Quoted 9 lines

hw/vfio/user.c line 58

Quoted 5 lines
hw/vfio/user.c new file +5 -0
+    } else if (ret < 0) {
+        hdr->flags |= VFIO_USER_ERROR;
+        hdr->error_reply = -ret;
+        hdr->size = sizeof(*hdr);
+    }

assert(ret != 0)? That case doesn't seem to be defined so maybe an assertion is worthwhile.

I should test for positive size less than the header size as an error.

Quoted 19 lines
Quoted 19 lines
hw/vfio/user.c new file +19 -0
+    vfio_user_send(proxy, hdr, NULL);
+}
+
+void vfio_user_recv(void *opaque)
+{
+    VFIODevice *vbasedev = opaque;
+    VFIOProxy *proxy = vbasedev->proxy;
+    VFIOUserReply *reply = NULL;
+    g_autofree int *fdp = NULL;
+    VFIOUserFDs reqfds = { 0, 0, fdp };
+    vfio_user_hdr_t msg;
+    struct iovec iov = {
+        .iov_base = &msg,
+        .iov_len = sizeof(msg),
+    };
+    int isreply, i, ret;
+    size_t msgleft, numfds = 0;
+    char *data = NULL;
+    g_autofree char *buf = NULL;
Quoted 9 lines

hw/vfio/user.c line 82

Quoted 5 lines
hw/vfio/user.c new file +5 -0
+    Error *local_err = NULL;
+
+    qemu_mutex_lock(&proxy->lock);
+    if (proxy->state == CLOSING) {
+        qemu_mutex_unlock(&proxy->lock);

QEMU_LOCK_GUARD() automatically unlocks mutexes when the function returns and is less error-prone than manual lock/unlock calls.

will look into it

Quoted 26 lines

hw/vfio/user.c line 90

Quoted 8 lines
hw/vfio/user.c new file +8 -0
+        return;
+    }
+
+    ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds,
+                                 &local_err);
+    if (ret <= 0) {
+        /* read error or other side closed connection */
+        error_setg_errno(&local_err, errno, "vfio_user_recv read error");

This will trigger an assertion failure when local_err was already set by qio_channel_readv_full():

static void error_setv(Error **errp,
const char *src, int line, const char *func, ErrorClass err_class, const char *fmt, va_list ap, const char *suffix)

 {
     Error *err;
     int saved_errno = errno;

     if (errp == NULL) {
         return;
     }
     assert(*errp == NULL);
     ^^^^^^^^^^^^^^^^^^^^^^
Quoted 4 lines

I think this error_setg_errno() call should be dropped. You can use error_prepend() if you'd like to add more information to the error message from qio_channel_readv_full().

OK

Quoted 25 lines
Quoted 25 lines
hw/vfio/user.c new file +25 -0
+        goto fatal;
+    }
+
+    if (ret < sizeof(msg)) {
+        error_setg(&local_err, "vfio_user_recv short read of header");
+        goto err;
+    }
+
+    /*
+     * For replies, find the matching pending request
+     */
+    switch (msg.flags & VFIO_USER_TYPE) {
+    case VFIO_USER_REQUEST:
+        isreply = 0;
+        break;
+    case VFIO_USER_REPLY:
+        isreply = 1;
+        break;
+    default:
+        error_setg(&local_err, "vfio_user_recv unknown message type");
+        goto err;
+    }
+
+    if (isreply) {
+        QTAILQ_FOREACH(reply, &proxy->pending, next) {
Quoted 9 lines

hw/vfio/user.c line 119

Quoted 4 lines
hw/vfio/user.c new file +4 -0
+            if (msg.id == reply->id) {
+                break;
+            }
+        }

I'm surprised to see this loop since proxy->lock prevents additional requests from being sent while we're trying to receive a message. Can there really be multiple replies pending with this locking scheme?

I didn’t want to assume that was always true. Note an email
exchange with Peter Xu where I can drop BQL in the middle of a memory region transaction that causes dma_map/unmap messages to be sent. The fix to that issue will be to send the messages async, then wait for the youngest reply when the transaction commits.

Quoted 30 lines
Quoted 30 lines
hw/vfio/user.c new file +30 -0
+        if (reply == NULL) {
+            error_setg(&local_err, "vfio_user_recv unexpected reply");
+            goto err;
+        }
+        QTAILQ_REMOVE(&proxy->pending, reply, next);
+
+        /*
+         * Process any received FDs
+         */
+        if (numfds != 0) {
+            if (reply->fds == NULL || reply->fds->recv_fds < numfds) {
+                error_setg(&local_err, "vfio_user_recv unexpected FDs");
+                goto err;
+            }
+            reply->fds->recv_fds = numfds;
+            memcpy(reply->fds->fds, fdp, numfds * sizeof(int));
+        }
+
+    } else {
+        /*
+         * The client doesn't expect any FDs in requests, but
+         * they will be expected on the server
+         */
+        if (numfds != 0 && (proxy->flags & VFIO_PROXY_CLIENT)) {
+            error_setg(&local_err, "vfio_user_recv fd in client reply");
+            goto err;
+        }
+        reqfds.recv_fds = numfds;
+    }
+
Quoted 9 lines

hw/vfio/user.c line 153

Quoted 4 lines
hw/vfio/user.c new file +4 -0
+    /*
+     * put the whole message into a single buffer
+     */
+    msgleft = msg.size - sizeof(msg);

msg.size has not been validated so this could underflow. Please validate all inputs so malicious servers/clients cannot crash or compromise the program.

OK

Quoted 5 lines

hw/vfio/user.c line 155

Quoted 2 lines
hw/vfio/user.c new file +2 -0
+    if (isreply) {
+        if (msg.size > reply->rsize) {

rsize is an int. Should it be uint32_t like msg.size?

OK

Quoted 12 lines

hw/vfio/user.c line 166

Quoted 11 lines
hw/vfio/user.c new file +11 -0
+            error_setg(&local_err,
+                       "vfio_user_recv reply larger than recv buffer");
+            goto fatal;
+        }
+        *reply->msg = msg;
+        data = (char *)reply->msg + sizeof(msg);
+    } else {
+        if (msg.size > max_xfer_size) {
+            error_setg(&local_err, "vfio_user_recv request larger than max");
+            goto fatal;
+        }
Quoted 7 lines

Missing check to prevent buffer overflow:

 if (msg.size < sizeof(msg)) {
     error_setg(&local_err, "vfio_user_recv request too small");
     goto fatal;

}

I will put this check up before the msgleft calculation in
the review comment above.

Quoted 21 lines
Quoted 21 lines
hw/vfio/user.c new file +21 -0
+        buf = g_malloc0(msg.size);
+        memcpy(buf, &msg, sizeof(msg));
+        data = buf + sizeof(msg);
+    }
+
+    if (msgleft != 0) {
+        ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err);
+        if (ret < 0) {
+            goto fatal;
+        }
+        if (ret != msgleft) {
+            error_setg(&local_err, "vfio_user_recv short read of msg body");
+            goto err;
+        }
+    }
+
+    /*
+     * Replies signal a waiter, requests get processed by vfio code
+     * that may assume the iothread lock is held.
+     */
+    qemu_mutex_unlock(&proxy->lock);
Quoted 9 lines

hw/vfio/user.c line 190

Quoted 3 lines
hw/vfio/user.c new file +3 -0
+    if (isreply) {
+        reply->complete = 1;
+        qemu_cond_signal(&reply->cv);

signal must be called with the mutex held to avoid race conditions. If the waiter acquires the lock and still sees complete == 0, then we signal before wait is entered, the signal is missed and the waiter is stuck.

Yes, this is a bug

Quoted 9 lines

hw/vfio/user.c line 196

Quoted 6 lines
hw/vfio/user.c new file +6 -0
+    } else {
+        qemu_mutex_lock_iothread();
+        /*
+         * make sure proxy wasn't closed while we waited
+         * checking without holding the proxy lock is safe
+         * since state is only set to CLOSING when iolock is held

s/iolock/the BQL/

OK

Quoted 20 lines
Quoted 20 lines
hw/vfio/user.c new file +20 -0
+         */
+        if (proxy->state != CLOSING) {
+            ret = proxy->request(proxy->reqarg, buf, &reqfds);
+            if (ret < 0 && !(msg.flags & VFIO_USER_NO_REPLY)) {
+                vfio_user_send_reply(proxy, buf, ret);
+            }
+        }
+        qemu_mutex_unlock_iothread();
+    }
+
+    return;
+ fatal:
+    vfio_user_shutdown(proxy);
+    proxy->state = RECV_ERROR;
+
+ err:
+    qemu_mutex_unlock(&proxy->lock);
+    for (i = 0; i < numfds; i++) {
+        close(fdp[i]);
+    }
Quoted 9 lines

hw/vfio/user.c line 222

Quoted 6 lines
hw/vfio/user.c new file +6 -0
+    if (reply != NULL) {
+        /* force an error to keep sending thread from hanging */
+        reply->msg->flags |= VFIO_USER_ERROR;
+        reply->msg->error_reply = EINVAL;
+        reply->complete = 1;
+        qemu_cond_signal(&reply->cv);

This has the race condition too.

Yes

Quoted 37 lines
Quoted 37 lines
hw/vfio/user.c new file +37 -0
+    }
+    error_report_err(local_err);
+}
+
+static void vfio_user_send_locked(VFIOProxy *proxy, vfio_user_hdr_t *msg,
+                                  VFIOUserFDs *fds)
+{
+    struct iovec iov = {
+        .iov_base = msg,
+        .iov_len = msg->size,
+    };
+    size_t numfds = 0;
+    int msgleft, ret, *fdp = NULL;
+    char *buf;
+    Error *local_err = NULL;
+
+    if (proxy->state != CONNECTED) {
+        msg->flags |= VFIO_USER_ERROR;
+        msg->error_reply = ECONNRESET;
+        return;
+    }
+
+    if (fds != NULL && fds->send_fds != 0) {
+        numfds = fds->send_fds;
+        fdp = fds->fds;
+    }
+    ret = qio_channel_writev_full(proxy->ioc, &iov, 1, fdp, numfds, &local_err);
+    if (ret < 0) {
+        goto err;
+    }
+    if (ret == msg->size) {
+        return;
+    }
+
+    buf = iov.iov_base + ret;
+    msgleft = iov.iov_len - ret;
+    do {
Quoted 9 lines

hw/vfio/user.c line 264

Quoted 5 lines
hw/vfio/user.c new file +5 -0
+        ret = qio_channel_write(proxy->ioc, buf, msgleft, &local_err);
+        if (ret < 0) {
+            goto err;
+        }
+        buf += ret, msgleft -= ret;

Please use semicolon. Comma operators are rare, requiring readers to check their exact semantics. There is no need to use comma here.

OK

Quoted 8 lines

hw/vfio/user.c line 269

Quoted 5 lines
hw/vfio/user.c new file +5 -0
+    } while (msgleft != 0);
+    return;
+
+ err:
+    error_report_err(local_err);

State remains unchanged and msg->error_reply isn't set?

They should be set.

JJ

Quoted 49 lines
Quoted 49 lines
hw/vfio/user.c new file +17 -0
+}
+
+static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg,
+                           VFIOUserFDs *fds)
+{
+    bool iolock = qemu_mutex_iothread_locked();
+
+    if (iolock) {
+        qemu_mutex_unlock_iothread();
+    }
+    qemu_mutex_lock(&proxy->lock);
+    vfio_user_send_locked(proxy, msg, fds);
+    qemu_mutex_unlock(&proxy->lock);
+    if (iolock) {
+        qemu_mutex_lock_iothread();
+    }
+}
MAINTAINERS +0 -0
index 12d69f3a45..aa4df6c418 100644
@@ -1883,8 +1883,12 @@ L: [email protected]

vfio-user
M: John G Johnson <[email protected]>
M: Thanos Makatos <[email protected]>

MAINTAINERS +2 -0
+M: Elena Ufimtseva <[email protected]>
+M: Jagannathan Raman <[email protected]>

S: Supported
F: docs/devel/vfio-user.rst

MAINTAINERS +2 -0
+F: hw/vfio/user.c
+F: hw/vfio/user.h

vhost
M: Michael S. Tsirkin <[email protected]>

hw/vfio/meson.build +1 -0
index da9af297a0..739b30be73 100644
@@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files(
  'display.c',
  'pci-quirks.c',
  'pci.c',
+  'user.c',
 ))
 vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c'))
 vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c'))
Signature · 1 line
2.25.1
#6 lore raw

Re: [PATCH RFC 03/19] vfio-user: define VFIO Proxy and communication functions

Quoted 30 lines · John Johnson
Quoted 23 lines · Stefan Hajnoczi
Quoted 9 lines · Elena Ufimtseva

From: John G Johnson <[email protected]>

Add user.c and user.h files for vfio-user with the basic send and receive functions.

---

Quoted 8 lines
hw/vfio/user.h                | 120 ++++++++++++++
include/hw/vfio/vfio-common.h |   2 +
hw/vfio/user.c                | 286 ++++++++++++++++++++++++++++++++++
MAINTAINERS                   |   4 +
hw/vfio/meson.build           |   1 +
5 files changed, 413 insertions(+)

create mode 100644 hw/vfio/user.h
create mode 100644 hw/vfio/user.c

Quoted 6 lines

The multi-threading, coroutine, and blocking I/O requirements of vfio_user_recv() and vfio_user_send_reply() are unclear to me. Please document them so it's clear what environment they can be called from. I guess they are not called from coroutines and proxy->ioc is a blocking IOChannel?

Quoted 3 lines

Yes to both, moreover, a block comment above vfio_user_recv() would
be useful. The call to setup vfio_user_recv() as the socket handler isn’t in this patch, do you want the series re-org’d?

That would help with review, thanks!

Stefan

#7 lore raw

RFC PATCH 4/19 vfio-user: Define type vfio_user_pci_dev_info

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

New class for vfio-user with its class and instance constructors and destructors.

---
 hw/vfio/pci.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
hw/vfio/pci.c +49 -0
index bea95efc33..554b562769 100644
@@ -42,6 +42,7 @@
 #include "qapi/error.h"
 #include "migration/blocker.h"
 #include "migration/qemu-file.h"
+#include "hw/vfio/user.h"
 
 #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug"
 
@@ -3326,3 +3327,51 @@ static void register_vfio_pci_dev_type(void)
 }
 
 type_init(register_vfio_pci_dev_type)
+
+static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
+{
+    ERRP_GUARD();
+    VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev);
+
+    if (!udev->sock_name) {
+        error_setg(errp, "No socket specified");
+        error_append_hint(errp, "Use -device vfio-user-pci,socket=<name>\n");
+        return;
+    }
+}
+
+static void vfio_user_instance_finalize(Object *obj)
+{
+}
+
+static Property vfio_user_pci_dev_properties[] = {
+    DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),
+    DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass);
+
+    device_class_set_props(dc, vfio_user_pci_dev_properties);
+    dc->desc = "VFIO over socket PCI device assignment";
+    pdc->realize = vfio_user_pci_realize;
+}
+
+static const TypeInfo vfio_user_pci_dev_info = {
+    .name = TYPE_VFIO_USER_PCI,
+    .parent = TYPE_VFIO_PCI_BASE,
+    .instance_size = sizeof(VFIOUserPCIDevice),
+    .class_init = vfio_user_pci_dev_class_init,
+    .instance_init = vfio_instance_init,
+    .instance_finalize = vfio_user_instance_finalize,
+};
+
+static void register_vfio_user_dev_type(void)
+{
+    type_register_static(&vfio_user_pci_dev_info);
+}
+
+type_init(register_vfio_user_dev_type)
Signature · 1 line
2.25.1
#8 lore raw

Re: [PATCH RFC 04/19] vfio-user: Define type vfio_user_pci_dev_info

Quoted 37 lines · Elena Ufimtseva

From: John G Johnson <[email protected]>

New class for vfio-user with its class and instance constructors and destructors.

---
 hw/vfio/pci.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
hw/vfio/pci.c +10 -0
index bea95efc33..554b562769 100644
@@ -42,6 +42,7 @@
 #include "qapi/error.h"
 #include "migration/blocker.h"
 #include "migration/qemu-file.h"
+#include "hw/vfio/user.h"
 
 #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug"
 
@@ -3326,3 +3327,51 @@ static void register_vfio_pci_dev_type(void)
 }
 
 type_init(register_vfio_pci_dev_type)
+
+static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
+{
+    ERRP_GUARD();
+    VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev);
+
+    if (!udev->sock_name) {
+        error_setg(errp, "No socket specified");
+        error_append_hint(errp, "Use -device vfio-user-pci,socket=<name>\n");

hw/vfio/pci.c static void register_vfio_pci_dev_type(void) · line 3348

Quoted 10 lines
hw/vfio/pci.c +10 -0
+        return;
+    }
+}
+
+static void vfio_user_instance_finalize(Object *obj)
+{
+}
+
+static Property vfio_user_pci_dev_properties[] = {
+    DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),

Please use SocketAddress so that alternative socket connection details can be supported without inventing custom syntax for vfio-user-pci. For example, file descriptor passing should be possible.

I think this requires a bit of command-line parsing work, so don't worry about it for now, but please add a TODO comment. When the -device vfio-user-pci syntax is finalized (i.e. when the code is merged and the device name doesn't start with the experimental x- prefix), then it needs to be solved.

hw/vfio/pci.c static void register_vfio_pci_dev_type(void) · line 3349

Quoted 1 line
hw/vfio/pci.c +1 -0
+    DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false),

I'm not sure what "secure-dma" means and the "secure" variable name is even more inscrutable. Does this mean don't share memory so that each DMA access is checked individually?

Quoted 31 lines
hw/vfio/pci.c +28 -0
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass);
+
+    device_class_set_props(dc, vfio_user_pci_dev_properties);
+    dc->desc = "VFIO over socket PCI device assignment";
+    pdc->realize = vfio_user_pci_realize;
+}
+
+static const TypeInfo vfio_user_pci_dev_info = {
+    .name = TYPE_VFIO_USER_PCI,
+    .parent = TYPE_VFIO_PCI_BASE,
+    .instance_size = sizeof(VFIOUserPCIDevice),
+    .class_init = vfio_user_pci_dev_class_init,
+    .instance_init = vfio_instance_init,
+    .instance_finalize = vfio_user_instance_finalize,
+};
+
+static void register_vfio_user_dev_type(void)
+{
+    type_register_static(&vfio_user_pci_dev_info);
+}
+
+type_init(register_vfio_user_dev_type)
Signature · 1 line
2.25.1
#9 lore raw

Re: [PATCH RFC 04/19] vfio-user: Define type vfio_user_pci_dev_info

Quoted 57 lines · Stefan Hajnoczi
Quoted 37 lines · Elena Ufimtseva

From: John G Johnson <[email protected]>

New class for vfio-user with its class and instance constructors and destructors.

---
hw/vfio/pci.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 49 insertions(+)
hw/vfio/pci.c +10 -0
index bea95efc33..554b562769 100644
@@ -42,6 +42,7 @@
 #include "qapi/error.h"
 #include "migration/blocker.h"
 #include "migration/qemu-file.h"
+#include "hw/vfio/user.h"
 
 #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug"
 
@@ -3326,3 +3327,51 @@ static void register_vfio_pci_dev_type(void)
 }
 
 type_init(register_vfio_pci_dev_type)
+
+static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
+{
+    ERRP_GUARD();
+    VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev);
+
+    if (!udev->sock_name) {
+        error_setg(errp, "No socket specified");
+        error_append_hint(errp, "Use -device vfio-user-pci,socket=<name>\n");

hw/vfio/pci.c static void register_vfio_pci_dev_type(void) · line 3348

Quoted 10 lines
hw/vfio/pci.c +10 -0
+        return;
+    }
+}
+
+static void vfio_user_instance_finalize(Object *obj)
+{
+}
+
+static Property vfio_user_pci_dev_properties[] = {
+    DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),

Please use SocketAddress so that alternative socket connection details can be supported without inventing custom syntax for vfio-user-pci. For example, file descriptor passing should be possible.

Quoted 6 lines

I think this requires a bit of command-line parsing work, so don't worry about it for now, but please add a TODO comment. When the -device vfio-user-pci syntax is finalized (i.e. when the code is merged and the device name doesn't start with the experimental x- prefix), then it needs to be solved.

What do you want the options to look like at the endgame? I’d
rather work backward from that than have several different flavors of options as new socket options are added. I did look at -chardev socket, and it was confusing enough that I went for the simple string.

Quoted 6 lines

hw/vfio/pci.c static void register_vfio_pci_dev_type(void) · line 3349

Quoted 1 line
hw/vfio/pci.c +1 -0
+    DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false),

I'm not sure what "secure-dma" means and the "secure" variable name is even more inscrutable. Does this mean don't share memory so that each DMA access is checked individually?

Yes. Do you have another name you’d prefer? “no-shared-mem”?

JJ

Quoted 31 lines
Quoted 31 lines
hw/vfio/pci.c +28 -0
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass);
+
+    device_class_set_props(dc, vfio_user_pci_dev_properties);
+    dc->desc = "VFIO over socket PCI device assignment";
+    pdc->realize = vfio_user_pci_realize;
+}
+
+static const TypeInfo vfio_user_pci_dev_info = {
+    .name = TYPE_VFIO_USER_PCI,
+    .parent = TYPE_VFIO_PCI_BASE,
+    .instance_size = sizeof(VFIOUserPCIDevice),
+    .class_init = vfio_user_pci_dev_class_init,
+    .instance_init = vfio_instance_init,
+    .instance_finalize = vfio_user_instance_finalize,
+};
+
+static void register_vfio_user_dev_type(void)
+{
+    type_register_static(&vfio_user_pci_dev_info);
+}
+
+type_init(register_vfio_user_dev_type)
Signature · 1 line
2.25.1
#10 lore raw

Re: [PATCH RFC 04/19] vfio-user: Define type vfio_user_pci_dev_info

Quoted 64 lines · John Johnson
Quoted 57 lines · Stefan Hajnoczi
Quoted 37 lines · Elena Ufimtseva

From: John G Johnson <[email protected]>

New class for vfio-user with its class and instance constructors and destructors.

---
hw/vfio/pci.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 49 insertions(+)
hw/vfio/pci.c +10 -0
index bea95efc33..554b562769 100644
@@ -42,6 +42,7 @@
 #include "qapi/error.h"
 #include "migration/blocker.h"
 #include "migration/qemu-file.h"
+#include "hw/vfio/user.h"
 
 #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug"
 
@@ -3326,3 +3327,51 @@ static void register_vfio_pci_dev_type(void)
 }
 
 type_init(register_vfio_pci_dev_type)
+
+static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
+{
+    ERRP_GUARD();
+    VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev);
+
+    if (!udev->sock_name) {
+        error_setg(errp, "No socket specified");
+        error_append_hint(errp, "Use -device vfio-user-pci,socket=<name>\n");

hw/vfio/pci.c static void register_vfio_pci_dev_type(void) · line 3348

Quoted 10 lines
hw/vfio/pci.c +10 -0
+        return;
+    }
+}
+
+static void vfio_user_instance_finalize(Object *obj)
+{
+}
+
+static Property vfio_user_pci_dev_properties[] = {
+    DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),

Please use SocketAddress so that alternative socket connection details can be supported without inventing custom syntax for vfio-user-pci. For example, file descriptor passing should be possible.

Quoted 6 lines

I think this requires a bit of command-line parsing work, so don't worry about it for now, but please add a TODO comment. When the -device vfio-user-pci syntax is finalized (i.e. when the code is merged and the device name doesn't start with the experimental x- prefix), then it needs to be solved.

Quoted 4 lines

What do you want the options to look like at the endgame? I’d
rather work backward from that than have several different flavors of options as new socket options are added. I did look at -chardev socket, and it was confusing enough that I went for the simple string.

The standard socket syntax is present in qemu-storage-daemon's --export and --nbd-server options:

addr.type=inet,addr.host=<host>,addr.port=<port> addr.type=unix,addr.path=<socket-path> addr.type=fd,addr.str=<fd>

--export and --nbd-server use QAPI to generate parsers for these options (they use 'SocketAddress' from qapi/sockets.json). I'm not sure whether it's easier to reuse the QAPI parser or to simply add qdev properties mimicking the same syntax. Either way, there should probably be a common qdev property API for SocketAddress values.

Quoted 8 lines
Quoted 6 lines

hw/vfio/pci.c static void register_vfio_pci_dev_type(void) · line 3349

Quoted 1 line
hw/vfio/pci.c +1 -0
+    DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false),

I'm not sure what "secure-dma" means and the "secure" variable name is even more inscrutable. Does this mean don't share memory so that each DMA access is checked individually?

Yes. Do you have another name you’d prefer? “no-shared-mem”?

I'm not sure other property names are much clearer, so feel free to stick with "secure-dma". Renaming the "secure" field to "secure_dma" and adding a comment that clarifies its purpose would be enough.

Here are some options:
- The vfio-user protocol message for sharing memory is called VFIO_USER_DMA_MAP. The option could be dma-map=on|off (default on). But this is based on protocol internals and may not be clear to users.
- shared-mem=on|off
- shared-ram=on|off

Thanks,
Stefan

#11 lore raw

RFC PATCH 5/19 vfio-user: connect vfio proxy to remote server

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

---
 hw/vfio/user.h |  2 ++
 hw/vfio/pci.c  | 16 ++++++++++
 hw/vfio/user.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+)
hw/vfio/user.h +2 -0
index cdbc074579..12106ccb6a 100644
@@ -117,4 +117,6 @@ typedef struct VFIOProxy {
 
 void vfio_user_recv(void *opaque);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
+VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp);
+void vfio_user_disconnect(VFIOProxy *proxy);
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +16 -0
index 554b562769..1effdcd5c0 100644
@@ -3332,16 +3332,32 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
 {
     ERRP_GUARD();
     VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev);
+    VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev);
+    VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIOProxy *proxy;
+    Error *err = NULL;
 
     if (!udev->sock_name) {
         error_setg(errp, "No socket specified");
         error_append_hint(errp, "Use -device vfio-user-pci,socket=<name>\n");
         return;
     }
+    proxy = vfio_user_connect_dev(udev->sock_name, &err);
+    if (!proxy) {
+        error_setg(errp, "Remote proxy not found");
+        return;
+    }
+    vbasedev->proxy = proxy;
 }
 
 static void vfio_user_instance_finalize(Object *obj)
 {
+    VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj);
+    VFIODevice *vbasedev = &vdev->vbasedev;
+
+    vfio_put_device(vdev);
+
+    vfio_user_disconnect(vbasedev->proxy);
 }
 
 static Property vfio_user_pci_dev_properties[] = {
hw/vfio/user.c +87 -0

90 lines not on this page · raw message

Signature · 1 line
2.25.1
#12 lore raw

RFC PATCH 6/19 vfio-user: negotiate protocol with remote server

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Send version and capabilities and validate reply.

---
 hw/vfio/user.h |   8 ++
 hw/vfio/pci.c  |  10 +++
 hw/vfio/user.c | 223 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 241 insertions(+)
hw/vfio/user.h +8 -0
index 12106ccb6a..844496ef82 100644
@@ -64,6 +64,13 @@ struct vfio_user_version {
     char capabilities[];
 };
 
+
+#define VFIO_USER_CAP           "capabilities"
+
+/* "capabilities" members */
+#define VFIO_USER_CAP_MAX_FDS   "max_msg_fds"
+#define VFIO_USER_CAP_MAX_XFER  "max_data_xfer_size"
+
 #define VFIO_USER_DEF_MAX_FDS   8
 #define VFIO_USER_MAX_MAX_FDS   16
 
@@ -119,4 +126,5 @@ void vfio_user_recv(void *opaque);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp);
 void vfio_user_disconnect(VFIOProxy *proxy);
+int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +10 -0
index 1effdcd5c0..8ca1431cca 100644
@@ -3348,6 +3348,16 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         return;
     }
     vbasedev->proxy = proxy;
+
+    vfio_user_validate_version(vbasedev, &err);
+    if (err != NULL) {
+        error_propagate(errp, err);
+        goto error;
+    }
+    return;
+
+ error:
+    error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name);
 }
 
 static void vfio_user_instance_finalize(Object *obj)
hw/vfio/user.c +223 -0

250 lines not on this page · raw message

Signature · 1 line
2.25.1
#13 lore raw

RFC PATCH 7/19 vfio-user: define vfio-user pci ops

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

---
 hw/vfio/pci.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
hw/vfio/pci.c +31 -0
index 8ca1431cca..388b7d82d7 100644
@@ -3328,6 +3328,29 @@ static void register_vfio_pci_dev_type(void)
 
 type_init(register_vfio_pci_dev_type)
 
+/*
+ * Emulated devices don't use host hot reset
+ */
+static int vfio_user_pci_no_reset(VFIODevice *vbasedev)
+{
+    error_printf("vfio-user - no hot reset\n");
+    return 0;
+}
+
+static void vfio_user_pci_not_needed(VFIODevice *vbasedev)
+{
+    vbasedev->needs_reset = false;
+}
+
+static VFIODeviceOps vfio_user_pci_ops = {
+    .vfio_compute_needs_reset = vfio_user_pci_not_needed,
+    .vfio_hot_reset_multi = vfio_user_pci_no_reset,
+    .vfio_eoi = vfio_intx_eoi,
+    .vfio_get_object = vfio_pci_get_object,
+    .vfio_save_config = vfio_pci_save_config,
+    .vfio_load_config = vfio_pci_load_config,
+};
+
 static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
 {
     ERRP_GUARD();
@@ -3354,6 +3377,14 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         error_propagate(errp, err);
         goto error;
     }
+
+    vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name);
+    vbasedev->dev = DEVICE(vdev);
+    vbasedev->fd = -1;
+    vbasedev->type = VFIO_DEVICE_TYPE_PCI;
+    vbasedev->no_mmap = false;
+    vbasedev->ops = &vfio_user_pci_ops;
+
     return;
 
  error:
Signature · 1 line
2.25.1
#14 lore raw

RFC PATCH 8/19 vfio-user: VFIO container setup & teardown

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Create SW-emulated containers and groups for vfio-user in lieu of the host IOMMU based ones used by the kernel driver VFIO implementation.

---
 include/hw/vfio/vfio-common.h |  3 ++
 hw/vfio/common.c              | 70 +++++++++++++++++++++++++++++++++++
 hw/vfio/pci.c                 | 19 ++++++++++
 3 files changed, 92 insertions(+)
include/hw/vfio/vfio-common.h +3 -0
index f43dc6e5d0..491a92b4f5 100644
@@ -90,6 +90,7 @@ typedef struct VFIOContainer {
     uint64_t max_dirty_bitmap_size;
     unsigned long pgsizes;
     unsigned int dma_max_mappings;
+    VFIOProxy *proxy;
     QLIST_HEAD(, VFIOGuestIOMMU) giommu_list;
     QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list;
     QLIST_HEAD(, VFIOGroup) group_list;
@@ -214,6 +215,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp);
 void vfio_put_group(VFIOGroup *group);
 int vfio_get_device(VFIOGroup *group, const char *name,
                     VFIODevice *vbasedev, Error **errp);
+void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as);
+void vfio_disconnect_proxy(VFIOGroup *group);
 
 extern const MemoryRegionOps vfio_region_ops;
 typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList;
hw/vfio/common.c +70 -0

82 lines not on this page · raw message

hw/vfio/pci.c +19 -0
index 388b7d82d7..5ed42ad858 100644
@@ -3358,6 +3358,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev);
     VFIODevice *vbasedev = &vdev->vbasedev;
     VFIOProxy *proxy;
+    VFIOGroup *group = NULL;
     Error *err = NULL;
 
     if (!udev->sock_name) {
@@ -3385,6 +3386,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     vbasedev->no_mmap = false;
     vbasedev->ops = &vfio_user_pci_ops;
 
+    /*
+     * each device gets its own group and container
+     * make them unrelated to any host IOMMU groupings
+     */
+    group = g_malloc0(sizeof(*group));
+    group->fd = -1;
+    group->groupid = -1;
+    QLIST_INIT(&group->device_list);
+    QLIST_INSERT_HEAD(&group->device_list, vbasedev, next);
+    vbasedev->group = group;
+
+    vfio_connect_proxy(proxy, group, pci_device_iommu_address_space(pdev));
+
     return;
 
  error:
@@ -3395,6 +3409,11 @@ static void vfio_user_instance_finalize(Object *obj)
 {
     VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj);
     VFIODevice *vbasedev = &vdev->vbasedev;
+    VFIOGroup *group = vbasedev->group;
+
+    vfio_disconnect_proxy(group);
+    g_free(group);
+    vbasedev->group = NULL;
 
     vfio_put_device(vdev);
 
Signature · 1 line
2.25.1
#15 lore raw

RFC PATCH 9/19 vfio-user: get device info and get irq info

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Send VFIO_USER_DEVICE_GET_INFO and
VFIO_USER_DEVICE_GET_IRQ_INFO commands.

---
 hw/vfio/user.h | 27 +++++++++++++++++++++++++++
 hw/vfio/pci.c  | 32 +++++++++++++++++++++++++++++---
 hw/vfio/user.c | 40 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 96 insertions(+), 3 deletions(-)
hw/vfio/user.h +27 -0
index 844496ef82..9f51e14c7c 100644
@@ -122,9 +122,36 @@ typedef struct VFIOProxy {
 
 #define VFIO_PROXY_CLIENT       0x1
 
+/*
+ * VFIO_USER_DEVICE_GET_INFO
+ * imported from struct_device_info
+ */
+struct vfio_user_device_info {
+    vfio_user_hdr_t hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t num_regions;
+    uint32_t num_irqs;
+    uint32_t cap_offset;
+};
+
+/*
+ * VFIO_USER_DEVICE_GET_IRQ_INFO
+ * imported from struct vfio_irq_info
+ */
+struct vfio_user_irq_info {
+    vfio_user_hdr_t hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t count;
+};
+
 void vfio_user_recv(void *opaque);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp);
 void vfio_user_disconnect(VFIOProxy *proxy);
 int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
+int vfio_user_get_info(VFIODevice *vbasedev);
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info);
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +29 -3
index 5ed42ad858..029a191bcb 100644
@@ -2620,7 +2620,12 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp)
 
     irq_info.index = VFIO_PCI_ERR_IRQ_INDEX;
 
-    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_get_irq_info(vbasedev, &irq_info);
+    } else {
+        ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info);
+    }
+
     if (ret) {
         /* This can fail for an old kernel or legacy PCI dev */
         trace_vfio_populate_device_get_irq_info_failure(strerror(errno));
@@ -2739,8 +2744,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev)
         return;
     }
 
-    if (ioctl(vdev->vbasedev.fd,
-              VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) {
+    if (vdev->vbasedev.proxy != NULL) {
+        if (vfio_user_get_irq_info(&vdev->vbasedev, &irq_info) < 0) {
+            return;
+        }
+    } else {
+        if (ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0) {
+            return;
+        }
+    }
+    if (irq_info.count < 1) {
         return;
     }
 
@@ -3359,6 +3372,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     VFIODevice *vbasedev = &vdev->vbasedev;
     VFIOProxy *proxy;
     VFIOGroup *group = NULL;
+    int ret;
     Error *err = NULL;
 
     if (!udev->sock_name) {
@@ -3399,6 +3413,18 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
 
     vfio_connect_proxy(proxy, group, pci_device_iommu_address_space(pdev));
 
+    ret = vfio_user_get_info(&vdev->vbasedev);
+    if (ret) {
+        error_setg_errno(errp, -ret, "get info failure");
+        goto error;
+    }
+
+    vfio_populate_device(vdev, &err);
+    if (err) {
+        error_propagate(errp, err);
+        goto error;
+    }
+
     return;
 
  error:
hw/vfio/user.c +40 -0
index 24dd45b55d..a282b7b7b8 100644
@@ -508,6 +508,27 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp)
     return 0;
 }
 
+int vfio_user_get_info(VFIODevice *vbasedev)
+{
+    struct vfio_user_device_info msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0);
+    msg.argsz = sizeof(struct vfio_device_info);
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0);
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
+    vbasedev->num_irqs = msg.num_irqs;
+    vbasedev->num_regions = msg.num_regions;
+    vbasedev->flags = msg.flags;
+    vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET);
+    return 0;
+
+}
+
 static QLIST_HEAD(, VFIOProxy) vfio_user_sockets =
     QLIST_HEAD_INITIALIZER(vfio_user_sockets);
 
@@ -594,3 +615,22 @@ void vfio_user_disconnect(VFIOProxy *proxy)
 
     g_free(proxy);
 }
+
+int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info)
+{
+    struct vfio_user_irq_info msg;
+
+    memset(&msg, 0, sizeof(msg));
+    vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO,
+                          sizeof(msg), 0);
+    msg.argsz = info->argsz;
+    msg.index = info->index;
+
+    vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0);
+    if (msg.hdr.flags & VFIO_USER_ERROR) {
+        return -msg.hdr.error_reply;
+    }
+
+    memcpy(info, &msg.argsz, sizeof(*info));
+    return 0;
+}
Signature · 1 line
2.25.1
#16 lore raw

RFC PATCH 10/19 vfio-user: device region read/write

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Send VFIO_REGION_READ and VFIO_REGION_WRITE commands.

---
 hw/vfio/user.h   | 16 ++++++++++++++++
 hw/vfio/common.c | 17 +++++++++++++++--
 hw/vfio/pci.c    | 13 +++++++++++++
 hw/vfio/user.c   | 45 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 89 insertions(+), 2 deletions(-)
hw/vfio/user.h +16 -0
index 9f51e14c7c..17c4d90ef1 100644
@@ -147,6 +147,18 @@ struct vfio_user_irq_info {
     uint32_t count;
 };
 
+/*
+ * VFIO_USER_REGION_READ
+ * VFIO_USER_REGION_WRITE
+ */
+struct vfio_user_region_rw {
+    vfio_user_hdr_t hdr;
+    uint64_t offset;
+    uint32_t region;
+    uint32_t count;
+    char data[];
+};
+
 void vfio_user_recv(void *opaque);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp);
@@ -154,4 +166,8 @@ void vfio_user_disconnect(VFIOProxy *proxy);
 int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp);
 int vfio_user_get_info(VFIODevice *vbasedev);
 int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info);
+int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
+                          uint32_t count, void *data);
+int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
+                           uint64_t offset, uint32_t count, void *data);
 #endif /* VFIO_USER_H */
hw/vfio/common.c +15 -2
index 45acdeeb46..74041cc438 100644
@@ -40,6 +40,7 @@
 #include "trace.h"
 #include "qapi/error.h"
 #include "migration/migration.h"
+#include "hw/vfio/user.h"
 
 VFIOGroupList vfio_group_list =
     QLIST_HEAD_INITIALIZER(vfio_group_list);
@@ -214,6 +215,7 @@ void vfio_region_write(void *opaque, hwaddr addr,
         uint32_t dword;
         uint64_t qword;
     } buf;
+    int ret;
 
     switch (size) {
     case 1:
@@ -233,7 +235,12 @@ void vfio_region_write(void *opaque, hwaddr addr,
         break;
     }
 
-    if (pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) {
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_region_write(vbasedev, region->nr, addr, size, &data);
+    } else {
+        ret = pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr);
+    }
+    if (ret != size) {
         error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64
                      ",%d) failed: %m",
                      __func__, vbasedev->name, region->nr,
@@ -265,8 +272,14 @@ uint64_t vfio_region_read(void *opaque,
         uint64_t qword;
     } buf;
     uint64_t data = 0;
+    int ret;
 
-    if (pread(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) {
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_region_read(vbasedev, region->nr, addr, size, &buf);
+    } else {
+        ret = pread(vbasedev->fd, &buf, size, region->fd_offset + addr);
+    }
+    if (ret != size) {
         error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %m",
                      __func__, vbasedev->name, region->nr,
                      addr, size);
hw/vfio/pci.c +13 -0
index 029a191bcb..1054978e5e 100644
@@ -3424,6 +3424,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         error_propagate(errp, err);
         goto error;
     }
+    /* Get a copy of config space */
+    ret = vfio_user_region_read(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, 0,
+                                MIN(pci_config_size(pdev), vdev->config_size),
+                                pdev->config);
+    if (ret < 0) {
+        goto error;
+    }
+
+    /* vfio emulates a lot for us, but some bits need extra love */
+    vdev->emulated_config_bits = g_malloc0(vdev->config_size);
+
+    /* QEMU can also add or extend BARs */
+    memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4);
 
     return;
 
hw/vfio/user.c +45 -0

48 lines not on this page · raw message

Signature · 1 line
2.25.1
#17 lore raw

RFC PATCH 11/19 vfio-user: get region and DMA map/unmap operations

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Send VFIO_USER_DEVICE_GET_REGION_INFO to get device regions and VFIO_USER_DMA_MAP/UNMAP to tell remote server the DMA addresses it can access.

---
 hw/vfio/user.h                |  54 ++++++++++++++++++
 include/hw/vfio/vfio-common.h |   2 +
 hw/vfio/common.c              |  84 +++++++++++++++++++++++++---
 hw/vfio/pci.c                 |   4 ++
 hw/vfio/user.c                | 100 ++++++++++++++++++++++++++++++++++
 5 files changed, 236 insertions(+), 8 deletions(-)
hw/vfio/user.h +54 -0
index 17c4d90ef1..351fdb3ee1 100644
@@ -121,6 +121,7 @@ typedef struct VFIOProxy {
 } VFIOProxy;
 
 #define VFIO_PROXY_CLIENT       0x1
+#define VFIO_PROXY_SECURE       0x2
 
 /*
  * VFIO_USER_DEVICE_GET_INFO
@@ -159,6 +160,52 @@ struct vfio_user_region_rw {
     char data[];
 };
 
+/*
+ * VFIO_USER_DMA_MAP
+ * imported from struct vfio_iommu_type1_dma_map
+ */
+struct vfio_user_dma_map {
+    vfio_user_hdr_t hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint64_t offset;    /* FD offset */
+    uint64_t iova;
+    uint64_t size;
+};
+
+/*imported from struct vfio_bitmap */
+struct vfio_user_bitmap {
+    uint64_t pgsize;
+    uint64_t size;
+    char data[];
+};
+
+/*
+ * VFIO_USER_DMA_UNMAP
+ * imported from struct vfio_iommu_type1_dma_unmap
+ */
+struct vfio_user_dma_unmap {
+    vfio_user_hdr_t hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint64_t iova;
+    uint64_t size;
+};
+
+/*
+ * VFIO_USER_DEVICE_GET_REGION_INFO
+ * imported from struct_vfio_region_info
+ */
+struct vfio_user_region_info {
+    vfio_user_hdr_t hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t cap_offset;
+    uint64_t size;
+    uint64_t offset;
+};
+
 void vfio_user_recv(void *opaque);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp);
@@ -170,4 +217,11 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset,
                           uint32_t count, void *data);
 int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index,
                            uint64_t offset, uint32_t count, void *data);
+int vfio_user_dma_map(VFIOProxy *proxy, struct vfio_iommu_type1_dma_map *map,
+                      VFIOUserFDs *fds);
+int vfio_user_dma_unmap(VFIOProxy *proxy,
+                        struct vfio_iommu_type1_dma_unmap *unmap,
+                        struct vfio_bitmap *bitmap);
+int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
+                              struct vfio_region_info *info, VFIOUserFDs *fds);
 #endif /* VFIO_USER_H */
include/hw/vfio/vfio-common.h +2 -0
index 491a92b4f5..d7b717594b 100644
@@ -146,6 +146,8 @@ typedef struct VFIODevice {
     Error *migration_blocker;
     OnOffAuto pre_copy_dirty_page_tracking;
     VFIOProxy *proxy;
+    struct vfio_region_info **regions;
+    int *regfds;
 } VFIODevice;
 
 struct VFIODeviceOps {
hw/vfio/common.c +76 -8

172 lines not on this page · raw message

hw/vfio/pci.c +4 -0
index 1054978e5e..054e673552 100644
@@ -3387,6 +3387,10 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     }
     vbasedev->proxy = proxy;
 
+    if (udev->secure) {
+        proxy->flags |= VFIO_PROXY_SECURE;
+    }
+
     vfio_user_validate_version(vbasedev, &err);
     if (err != NULL) {
         error_propagate(errp, err);
hw/vfio/user.c +100 -0

103 lines not on this page · raw message

Signature · 1 line
2.25.1
#18 lore raw

RFC PATCH 12/19 vfio-user: probe remote device's BARs

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

---
 hw/vfio/pci.c | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)
hw/vfio/pci.c +27 -5
index 054e673552..a8d2e59470 100644
@@ -1619,11 +1619,17 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr)
     }
 
     /* Determine what type of BAR this is for registration */
-    ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar),
-                vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr));
-    if (ret != sizeof(pci_bar)) {
-        error_report("vfio: Failed to read BAR %d (%m)", nr);
-        return;
+    if (vdev->vbasedev.proxy != NULL) {
+        /* during setup, config space was initialized from remote */
+        memcpy(&pci_bar, vdev->pdev.config + PCI_BASE_ADDRESS_0 + (4 * nr),
+               sizeof(pci_bar));
+    } else {
+        ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar),
+                    vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr));
+        if (ret != sizeof(pci_bar)) {
+            error_report("vfio: Failed to read BAR %d (%m)", nr);
+            return;
+        }
     }
 
     pci_bar = le32_to_cpu(pci_bar);
@@ -3442,6 +3448,22 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     /* QEMU can also add or extend BARs */
     memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4);
 
+    /*
+     * Local QEMU overrides aren't allowed
+     * They must be done in the device process
+     */
+    if (pdev->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
+        error_setg(errp, "Multi-function must be specified by device process");
+        goto error;
+    }
+    if (pdev->romfile) {
+        error_setg(errp, "Romfile must be specified by device process");
+        goto error;
+    }
+
+    vfio_bars_prepare(vdev);
+
+
     return;
 
  error:
Signature · 1 line
2.25.1
#19 lore raw

Re: [PATCH RFC 12/19] vfio-user: probe remote device's BARs

Quoted 6 lines · On Sun, 18 Jul 2021 23:27:51 -0700 Elena Ufimtseva…
hw/vfio/pci.c +2 -0
@@ -3442,6 +3448,22 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
     /* QEMU can also add or extend BARs */
     memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4);
 
+    /*
+     * Local QEMU overrides aren't allowed

hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3462

Quoted 10 lines
hw/vfio/pci.c +10 -0
+     * They must be done in the device process
+     */
+    if (pdev->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
+        error_setg(errp, "Multi-function must be specified by device process");
+        goto error;
+    }
+    if (pdev->romfile) {
+        error_setg(errp, "Romfile must be specified by device process");
+        goto error;
+    }

For what reason? PCI functions can operate completely independently, there could be different servers for different functions, a QEMU user may wish to apply a different option ROM image than provided by the server. This all creates unnecessary incompatibilities. Thanks,

Alex

#20 lore raw

Re: [PATCH RFC 12/19] vfio-user: probe remote device's BARs

Quoted 19 lines · Alex Williamson
Quoted 13 lines · On Sun, 18 Jul 2021 23:27:51 -0700 Elena Ufimtseva…
hw/vfio/pci.c +9 -0
@@ -3442,6 +3448,22 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
    /* QEMU can also add or extend BARs */
    memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4);
 
+    /*
+     * Local QEMU overrides aren't allowed
+     * They must be done in the device process
+     */
+    if (pdev->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
+        error_setg(errp, "Multi-function must be specified by device process");
+        goto error;
+    }
+    if (pdev->romfile) {
Quoted 9 lines

hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3462

Quoted 3 lines
hw/vfio/pci.c +3 -0
+        error_setg(errp, "Romfile must be specified by device process");
+        goto error;
+    }

For what reason? PCI functions can operate completely independently, there could be different servers for different functions, a QEMU user may wish to apply a different option ROM image than provided by the server. This all creates unnecessary incompatibilities. Thanks,

The idea is to have all the device options specified on the remote
server, and have the vfio client just be a pass-through device. I thought having them specified in 2 places would cause more confusion.

JJ

#21 lore raw

Re: [PATCH RFC 12/19] vfio-user: probe remote device's BARs

Quoted 27 lines · On Tue, 20 Jul 2021 01:39:21 +0000 John Johnson <[email protected]>…
Quoted 17 lines · Alex Williamson
Quoted 13 lines · On Sun, 18 Jul 2021 23:27:51 -0700 Elena Ufimtseva…
hw/vfio/pci.c +9 -0
@@ -3442,6 +3448,22 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
    /* QEMU can also add or extend BARs */
    memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4);
 
+    /*
+     * Local QEMU overrides aren't allowed
+     * They must be done in the device process
+     */
+    if (pdev->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
+        error_setg(errp, "Multi-function must be specified by device process");
+        goto error;
+    }
+    if (pdev->romfile) {
Quoted 9 lines

hw/vfio/pci.c static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) · line 3462

Quoted 3 lines
hw/vfio/pci.c +3 -0
+        error_setg(errp, "Romfile must be specified by device process");
+        goto error;
+    }  

For what reason? PCI functions can operate completely independently, there could be different servers for different functions, a QEMU user may wish to apply a different option ROM image than provided by the server. This all creates unnecessary incompatibilities. Thanks,

Quoted 3 lines

The idea is to have all the device options specified on the remote
server, and have the vfio client just be a pass-through device. I thought having them specified in 2 places would cause more confusion.

IMO, the server has no business making such configuration restrictions. It's the client's decision if it wants to create multi-function topologies or override the option rom. Same for whether it wants to override or virtualize capabilities. All of this should just work as-is; it's actually additional code required and knowledge through the management stack to prevent it. Thanks,

Alex

#22 lore raw

RFC PATCH 13/19 vfio-user: respond to remote DMA read/write requests

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

---
 hw/vfio/user.h | 16 ++++++++++++
 hw/vfio/pci.c  | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++
 hw/vfio/user.c | 21 +++++++++++++++-
 3 files changed, 103 insertions(+), 1 deletion(-)
hw/vfio/user.h +16 -0
index 351fdb3ee1..d08d94ed92 100644
@@ -206,6 +206,17 @@ struct vfio_user_region_info {
     uint64_t offset;
 };
 
+/*
+ * VFIO_USER_DMA_READ
+ * VFIO_USER_DMA_WRITE
+ */
+struct vfio_user_dma_rw {
+    vfio_user_hdr_t hdr;
+    uint64_t offset;
+    uint32_t count;
+    char data[];
+};
+
 void vfio_user_recv(void *opaque);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp);
@@ -224,4 +235,9 @@ int vfio_user_dma_unmap(VFIOProxy *proxy,
                         struct vfio_bitmap *bitmap);
 int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
                               struct vfio_region_info *info, VFIOUserFDs *fds);
+uint64_t vfio_user_max_xfer(void);
+void vfio_user_set_reqhandler(VFIODevice *vbasdev,
+                              int (*handler)(void *opaque, char *buf,
+                                             VFIOUserFDs *fds),
+                                             void *reqarg);
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +67 -0

79 lines not on this page · raw message

hw/vfio/user.c +20 -1
index eea8b9b402..8bedbc19f3 100644
@@ -42,6 +42,11 @@ static void vfio_user_request_msg(vfio_user_hdr_t *hdr, uint16_t cmd,
 static void vfio_user_send_recv(VFIOProxy *proxy, vfio_user_hdr_t *msg,
                                 VFIOUserFDs *fds, int rsize);
 
+uint64_t vfio_user_max_xfer(void)
+{
+    return max_xfer_size;
+}
+
 static void vfio_user_shutdown(VFIOProxy *proxy)
 {
     qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL);
@@ -236,7 +241,7 @@ void vfio_user_recv(void *opaque)
         *reply->msg = msg;
         data = (char *)reply->msg + sizeof(msg);
     } else {
-        if (msg.size > max_xfer_size) {
+        if (msg.size > max_xfer_size + sizeof(struct vfio_user_dma_rw)) {
             error_setg(&local_err, "vfio_user_recv request larger than max");
             goto fatal;
         }
@@ -779,3 +784,17 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index,
     memcpy(info, &msgp->argsz, info->argsz);
     return 0;
 }
+
+void vfio_user_set_reqhandler(VFIODevice *vbasedev,
+                              int (*handler)(void *opaque, char *buf,
+                                             VFIOUserFDs *fds),
+                              void *reqarg)
+{
+    VFIOProxy *proxy = vbasedev->proxy;
+
+    proxy->request = handler;
+    proxy->reqarg = reqarg;
+    qio_channel_set_aio_fd_handler(proxy->ioc,
+                                   iothread_get_aio_context(vfio_user_iothread),
+                                   vfio_user_recv, NULL, vbasedev);
+}
Signature · 1 line
2.25.1
#23 lore raw

RFC PATCH 14/19 vfio_user: setup MSI/X interrupts and PCI config operations

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Send VFIO_USER_DEVICE_SET_IRQS to setup interrup configuration. vfio_pci_write_config/vfio_pci_read_config iforms the remote server of PCI config space reads and writes.

---
 hw/vfio/user.h   | 14 ++++++++
 hw/vfio/common.c | 26 ++++++++++++---
 hw/vfio/pci.c    | 71 ++++++++++++++++++++++++++++-----------
 hw/vfio/user.c   | 87 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 173 insertions(+), 25 deletions(-)
hw/vfio/user.h +14 -0
index d08d94ed92..afb85952da 100644
@@ -217,6 +217,19 @@ struct vfio_user_dma_rw {
     char data[];
 };
 
+/*
+ * VFIO_USER_DEVICE_SET_IRQS
+ * imported from struct vfio_irq_set
+ */
+struct vfio_user_irq_set {
+    vfio_user_hdr_t hdr;
+    uint32_t argsz;
+    uint32_t flags;
+    uint32_t index;
+    uint32_t start;
+    uint32_t count;
+};
+
 void vfio_user_recv(void *opaque);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp);
@@ -240,4 +253,5 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev,
                               int (*handler)(void *opaque, char *buf,
                                              VFIOUserFDs *fds),
                                              void *reqarg);
+int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq);
 #endif /* VFIO_USER_H */
hw/vfio/common.c +21 -5
index 52a092e168..9b68416599 100644
@@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index)
         .count = 0,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index)
         .count = 1,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
@@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index)
         .count = 1,
     };
 
-    ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    if (vbasedev->proxy != NULL) {
+        vfio_user_set_irqs(vbasedev, &irq_set);
+    } else {
+        ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set);
+    }
 }
 
 static inline const char *action_to_str(int action)
@@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex,
     pfd = (int32_t *)&irq_set->data;
     *pfd = fd;
 
-    if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
-        ret = -errno;
+    if (vbasedev->proxy != NULL) {
+        ret = vfio_user_set_irqs(vbasedev, irq_set);
+    } else {
+        if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) {
+            ret = -errno;
+        }
     }
     g_free(irq_set);
 
hw/vfio/pci.c +51 -20

113 lines not on this page · raw message

hw/vfio/user.c +87 -0

90 lines not on this page · raw message

Signature · 1 line
2.25.1
#24 lore raw

RFC PATCH 15/19 vfio-user: vfio user device realize

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Setup INTx interrupts and a device region info cache for remote device info.

---
 include/hw/vfio/vfio-common.h |  1 +
 hw/vfio/common.c              | 33 ++++++++++++++++++-
 hw/vfio/pci.c                 | 61 ++++++++++++++++++++++++++++++++---
 hw/vfio/user.c                | 20 ++++++++++++
 4 files changed, 109 insertions(+), 6 deletions(-)
include/hw/vfio/vfio-common.h +1 -0
index d7b717594b..688660c28d 100644
@@ -56,6 +56,7 @@ typedef struct VFIORegion {
     uint32_t nr_mmaps;
     VFIOMmap *mmaps;
     uint8_t nr; /* cache the region number for debug */
+    int remfd; /* fd if exported from remote process */
 } VFIORegion;
 
 typedef struct VFIOMigration {
hw/vfio/common.c +32 -1
index 9b68416599..953d9e7b55 100644
@@ -1571,6 +1571,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,
     return true;
 }
 
+static int vfio_get_region_info_remfd(VFIODevice *vbasedev, int index)
+{
+    struct vfio_region_info *info;
+
+    if (vbasedev->regions == NULL || vbasedev->regions[index] == NULL) {
+        vfio_get_region_info(vbasedev, index, &info);
+    }
+    return vbasedev->regfds != NULL ? vbasedev->regfds[index] : -1;
+}
+
 static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
                                           struct vfio_region_info *info)
 {
@@ -1624,6 +1634,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
     region->size = info->size;
     region->fd_offset = info->offset;
     region->nr = index;
+    region->remfd = vfio_get_region_info_remfd(vbasedev, index);
 
     if (region->size) {
         region->mem = g_new0(MemoryRegion, 1);
@@ -1667,6 +1678,7 @@ int vfio_region_mmap(VFIORegion *region)
 {
     int i, prot = 0;
     char *name;
+    int fd;
 
     if (!region->mem) {
         return 0;
@@ -1675,9 +1687,11 @@ int vfio_region_mmap(VFIORegion *region)
     prot |= region->flags & VFIO_REGION_INFO_FLAG_READ ? PROT_READ : 0;
     prot |= region->flags & VFIO_REGION_INFO_FLAG_WRITE ? PROT_WRITE : 0;
 
+    fd = region->remfd != -1 ? region->remfd : region->vbasedev->fd;
+
     for (i = 0; i < region->nr_mmaps; i++) {
         region->mmaps[i].mmap = mmap(NULL, region->mmaps[i].size, prot,
-                                     MAP_SHARED, region->vbasedev->fd,
+                                     MAP_SHARED, fd,
                                      region->fd_offset +
                                      region->mmaps[i].offset);
         if (region->mmaps[i].mmap == MAP_FAILED) {
@@ -2524,6 +2538,23 @@ int vfio_get_device(VFIOGroup *group, const char *name,
 
 void vfio_put_base_device(VFIODevice *vbasedev)
 {
+    if (vbasedev->regions != NULL) {
+        int i;
+
+        for (i = 0; i < vbasedev->num_regions; i++) {
+            if (vbasedev->regfds != NULL && vbasedev->regfds[i] != -1) {
+                close(vbasedev->regfds[i]);
+            }
+            g_free(vbasedev->regions[i]);
+        }
+        g_free(vbasedev->regions);
+        vbasedev->regions = NULL;
+        if (vbasedev->regfds != NULL) {
+            g_free(vbasedev->regfds);
+            vbasedev->regfds = NULL;
+        }
+    }
+
     if (!vbasedev->group) {
         return;
     }
hw/vfio/pci.c +56 -5

86 lines not on this page · raw message

hw/vfio/user.c +20 -0
index 6afbde8ba8..0fd7e01986 100644
@@ -574,6 +574,16 @@ VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp)
     return proxy;
 }
 
+static void vfio_user_cb(void *opaque)
+{
+    VFIOProxy *proxy = opaque;
+
+    qemu_mutex_lock(&proxy->lock);
+    proxy->state = CLOSED;
+    qemu_mutex_unlock(&proxy->lock);
+    qemu_cond_signal(&proxy->close_cv);
+}
+
 void vfio_user_disconnect(VFIOProxy *proxy)
 {
     VFIOUserReply *r1, *r2;
@@ -601,6 +611,16 @@ void vfio_user_disconnect(VFIOProxy *proxy)
         g_free(r1);
     }
 
+    /*
+     * Make sure the iothread isn't blocking anywhere
+     * with a ref to this proxy by waiting for a BH
+     * handler to run after the proxy fd handlers were
+     * deleted above.
+     */
+    proxy->close_wait = 1;
+    aio_bh_schedule_oneshot(iothread_get_aio_context(vfio_user_iothread),
+                            vfio_user_cb, proxy);
+
     /* drop locks so the iothread can make progress */
     qemu_mutex_unlock_iothread();
     qemu_cond_wait(&proxy->close_cv, &proxy->lock);
Signature · 1 line
2.25.1
#25 lore raw

RFC PATCH 16/19 vfio-user: pci reset

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Send VFIO_USER_DEVICE_RESET to reset remote device.

---
 hw/vfio/user.h |  1 +
 hw/vfio/pci.c  | 29 ++++++++++++++++++++++++++---
 hw/vfio/user.c | 12 ++++++++++++
 3 files changed, 39 insertions(+), 3 deletions(-)
hw/vfio/user.h +1 -0
index afb85952da..95c2fb1707 100644
@@ -254,4 +254,5 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev,
                                              VFIOUserFDs *fds),
                                              void *reqarg);
 int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq);
+void vfio_user_reset(VFIODevice *vbasedev);
 #endif /* VFIO_USER_H */
hw/vfio/pci.c +26 -3
index 52af5a1061..a6c28dac03 100644
@@ -2212,8 +2212,9 @@ static void vfio_pci_pre_reset(VFIOPCIDevice *vdev)
 
 static void vfio_pci_post_reset(VFIOPCIDevice *vdev)
 {
+    VFIODevice *vbasedev = &vdev->vbasedev;
     Error *err = NULL;
-    int nr;
+    int ret, nr;
 
     vfio_intx_enable(vdev, &err);
     if (err) {
@@ -2221,11 +2222,18 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev)
     }
 
     for (nr = 0; nr < PCI_NUM_REGIONS - 1; ++nr) {
-        off_t addr = vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr);
+        off_t addr = PCI_BASE_ADDRESS_0 + (4 * nr);
         uint32_t val = 0;
         uint32_t len = sizeof(val);
 
-        if (pwrite(vdev->vbasedev.fd, &val, len, addr) != len) {
+        if (vbasedev->proxy != NULL) {
+            ret = vfio_user_region_write(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX,
+                                         addr, len, &val);
+        } else {
+            ret = pwrite(vdev->vbasedev.fd, &val, len,
+                         vdev->config_offset + addr);
+        }
+        if (ret != len) {
             error_report("%s(%s) reset bar %d failed: %m", __func__,
                          vdev->vbasedev.name, nr);
         }
@@ -3634,6 +3642,20 @@ static void vfio_user_instance_finalize(Object *obj)
     vfio_user_disconnect(vbasedev->proxy);
 }
 
+static void vfio_user_pci_reset(DeviceState *dev)
+{
+    VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev);
+    VFIODevice *vbasedev = &vdev->vbasedev;
+
+    vfio_pci_pre_reset(vdev);
+
+    if (vbasedev->reset_works) {
+        vfio_user_reset(vbasedev);
+    }
+
+    vfio_pci_post_reset(vdev);
+}
+
 static Property vfio_user_pci_dev_properties[] = {
     DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),
     DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false),
@@ -3645,6 +3667,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data)
     DeviceClass *dc = DEVICE_CLASS(klass);
     PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass);
 
+    dc->reset = vfio_user_pci_reset;
     device_class_set_props(dc, vfio_user_pci_dev_properties);
     dc->desc = "VFIO over socket PCI device assignment";
     pdc->realize = vfio_user_pci_realize;
hw/vfio/user.c +12 -0
index 0fd7e01986..8917596a2f 100644
@@ -905,3 +905,15 @@ int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq)
 
     return 0;
 }
+
+void vfio_user_reset(VFIODevice *vbasedev)
+{
+    vfio_user_hdr_t msg;
+
+    vfio_user_request_msg(&msg, VFIO_USER_DEVICE_RESET, sizeof(msg), 0);
+
+    vfio_user_send_recv(vbasedev->proxy, &msg, NULL, 0);
+    if (msg.flags & VFIO_USER_ERROR) {
+        error_printf("reset reply error %d\n", msg.error_reply);
+    }
+}
Signature · 1 line
2.25.1
#26 lore raw

RFC PATCH 17/19 vfio-user: probe remote device ROM BAR

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

---
 hw/vfio/pci.c | 38 ++++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)
hw/vfio/pci.c +30 -8
index a6c28dac03..bed8eaa4c2 100644
@@ -816,8 +816,14 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev)
     memset(vdev->rom, 0xff, size);
 
     while (size) {
-        bytes = pread(vdev->vbasedev.fd, vdev->rom + off,
-                      size, vdev->rom_offset + off);
+        if (vdev->vbasedev.proxy != NULL) {
+            bytes = vfio_user_region_read(&vdev->vbasedev,
+                                          VFIO_PCI_ROM_REGION_INDEX,
+                                          off, size, vdev->rom + off);
+        } else {
+            bytes = pread(vdev->vbasedev.fd, vdev->rom + off,
+                          size, vdev->rom_offset + off);
+        }
         if (bytes == 0) {
             break;
         } else if (bytes > 0) {
@@ -936,12 +942,28 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev)
      * Use the same size ROM BAR as the physical device.  The contents
      * will get filled in later when the guest tries to read it.
      */
-    if (pread(fd, &orig, 4, offset) != 4 ||
-        pwrite(fd, &size, 4, offset) != 4 ||
-        pread(fd, &size, 4, offset) != 4 ||
-        pwrite(fd, &orig, 4, offset) != 4) {
-        error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name);
-        return;
+    if (vdev->vbasedev.proxy != NULL) {
+        if (vfio_user_region_read(&vdev->vbasedev, VFIO_PCI_CONFIG_REGION_INDEX,
+                                  PCI_ROM_ADDRESS, 4, &orig) != 4 ||
+            vfio_user_region_write(&vdev->vbasedev,
+                                   VFIO_PCI_CONFIG_REGION_INDEX,
+                                   PCI_ROM_ADDRESS, 4, &size) != 4 ||
+            vfio_user_region_read(&vdev->vbasedev, VFIO_PCI_CONFIG_REGION_INDEX,
+                                  PCI_ROM_ADDRESS, 4, &size) != 4 ||
+            vfio_user_region_write(&vdev->vbasedev,
+                                   VFIO_PCI_CONFIG_REGION_INDEX,
+                                   PCI_ROM_ADDRESS, 4, &orig) != 4) {
+            error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name);
+            return;
+        }
+    } else {
+        if (pread(fd, &orig, 4, offset) != 4 ||
+            pwrite(fd, &size, 4, offset) != 4 ||
+            pread(fd, &size, 4, offset) != 4 ||
+            pwrite(fd, &orig, 4, offset) != 4) {
+            error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name);
+            return;
+        }
     }
 
     size = ~(le32_to_cpu(size) & PCI_ROM_ADDRESS_MASK) + 1;
Signature · 1 line
2.25.1
#27 lore raw

RFC PATCH 18/19 vfio-user: migration support

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

Send migration region operations to remote server. Send VFIO_USER_USER_DIRTY_PAGES to get remote dirty bitmap.

---
 hw/vfio/user.h      | 17 +++++++++++++++
 hw/vfio/common.c    | 51 ++++++++++++++++++++++++++++++++++++---------
 hw/vfio/migration.c | 35 ++++++++++++++++++-------------
 hw/vfio/pci.c       |  7 +++++++
 hw/vfio/user.c      | 45 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 130 insertions(+), 25 deletions(-)
hw/vfio/user.h +17 -0
index 95c2fb1707..eeb328c0a9 100644
@@ -230,6 +230,20 @@ struct vfio_user_irq_set {
     uint32_t count;
 };
 
+/* imported from struct vfio_iommu_type1_dirty_bitmap_get */
+struct vfio_user_bitmap_range {
+    uint64_t iova;
+    uint64_t size;
+    struct vfio_user_bitmap bitmap;
+};
+
+/* imported from struct vfio_iommu_type1_dirty_bitmap */
+struct vfio_user_dirty_pages {
+    vfio_user_hdr_t hdr;
+    uint32_t argsz;
+    uint32_t flags;
+};
+
 void vfio_user_recv(void *opaque);
 void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret);
 VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp);
@@ -255,4 +269,7 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev,
                                              void *reqarg);
 int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq);
 void vfio_user_reset(VFIODevice *vbasedev);
+int vfio_user_dirty_bitmap(VFIOProxy *proxy,
+                           struct vfio_iommu_type1_dirty_bitmap *bitmap,
+                           struct vfio_iommu_type1_dirty_bitmap_get *range);
 #endif /* VFIO_USER_H */
hw/vfio/common.c +41 -10

89 lines not on this page · raw message

hw/vfio/migration.c +20 -15
index 82f654afb6..8005b1171a 100644
@@ -27,6 +27,7 @@
 #include "pci.h"
 #include "trace.h"
 #include "hw/hw.h"
+#include "user.h"
 
 /*
  * Flags to be used as unique delimiters for VFIO devices in the migration
@@ -49,10 +50,18 @@ static int64_t bytes_transferred;
 static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count,
                                   off_t off, bool iswrite)
 {
+    VFIORegion *region = &vbasedev->migration->region;
     int ret;
 
-    ret = iswrite ? pwrite(vbasedev->fd, val, count, off) :
-                    pread(vbasedev->fd, val, count, off);
+    if (vbasedev->proxy != NULL) {
+        ret = iswrite ?
+            vfio_user_region_write(vbasedev, region->nr, off, count, val) :
+            vfio_user_region_read(vbasedev, region->nr, off, count, val);
+    } else {
+        off += region->fd_offset;
+        ret = iswrite ? pwrite(vbasedev->fd, val, count, off) :
+                        pread(vbasedev->fd, val, count, off);
+    }
     if (ret < count) {
         error_report("vfio_mig_%s %d byte %s: failed at offset 0x%"
                      HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count,
@@ -111,9 +120,7 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask,
                                     uint32_t value)
 {
     VFIOMigration *migration = vbasedev->migration;
-    VFIORegion *region = &migration->region;
-    off_t dev_state_off = region->fd_offset +
-                          VFIO_MIG_STRUCT_OFFSET(device_state);
+    off_t dev_state_off = VFIO_MIG_STRUCT_OFFSET(device_state);
     uint32_t device_state;
     int ret;
 
@@ -201,13 +208,13 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size)
     int ret;
 
     ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset),
-                      region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset));
+                        VFIO_MIG_STRUCT_OFFSET(data_offset));
     if (ret < 0) {
         return ret;
     }
 
     ret = vfio_mig_read(vbasedev, &data_size, sizeof(data_size),
-                        region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size));
+                        VFIO_MIG_STRUCT_OFFSET(data_size));
     if (ret < 0) {
         return ret;
     }
@@ -233,8 +240,7 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size)
             }
             buf_allocated = true;
 
-            ret = vfio_mig_read(vbasedev, buf, sec_size,
-                                region->fd_offset + data_offset);
+            ret = vfio_mig_read(vbasedev, buf, sec_size, data_offset);
             if (ret < 0) {
                 g_free(buf);
                 return ret;
@@ -269,7 +275,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
 
     do {
         ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset),
-                      region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset));
+                            VFIO_MIG_STRUCT_OFFSET(data_offset));
         if (ret < 0) {
             return ret;
         }
@@ -309,8 +315,8 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
             qemu_get_buffer(f, buf, sec_size);
 
             if (buf_alloc) {
-                ret = vfio_mig_write(vbasedev, buf, sec_size,
-                        region->fd_offset + data_offset);
+
+                ret = vfio_mig_write(vbasedev, buf, sec_size, data_offset);
                 g_free(buf);
 
                 if (ret < 0) {
@@ -322,7 +328,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
         }
 
         ret = vfio_mig_write(vbasedev, &report_size, sizeof(report_size),
-                        region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size));
+                             VFIO_MIG_STRUCT_OFFSET(data_size));
         if (ret < 0) {
             return ret;
         }
@@ -334,12 +340,11 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev,
 static int vfio_update_pending(VFIODevice *vbasedev)
 {
     VFIOMigration *migration = vbasedev->migration;
-    VFIORegion *region = &migration->region;
     uint64_t pending_bytes = 0;
     int ret;
 
     ret = vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes),
-                    region->fd_offset + VFIO_MIG_STRUCT_OFFSET(pending_bytes));
+                        VFIO_MIG_STRUCT_OFFSET(pending_bytes));
     if (ret < 0) {
         migration->pending_bytes = 0;
         return ret;
hw/vfio/pci.c +7 -0
index bed8eaa4c2..36f8524e7c 100644
@@ -3633,6 +3633,13 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp)
         }
     }
 
+    if (!pdev->failover_pair_id) {
+        ret = vfio_migration_probe(&vdev->vbasedev, errp);
+        if (ret) {
+            error_report("%s: Migration disabled", vdev->vbasedev.name);
+        }
+    }
+
     vfio_register_err_notifier(vdev);
     vfio_register_req_notifier(vdev);
 
hw/vfio/user.c +45 -0

48 lines not on this page · raw message

Signature · 1 line
2.25.1
#28 lore raw

RFC PATCH 19/19 vfio-user: add migration cli options and version negotiation

In reply to Elena Ufimtseva

From: John G Johnson <[email protected]>

---
 hw/vfio/user.h |  4 ++++
 hw/vfio/pci.c  |  5 +++++
 hw/vfio/user.c | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 42 insertions(+)
hw/vfio/user.h +4 -0
index eeb328c0a9..5542aa1932 100644
@@ -70,6 +70,10 @@ struct vfio_user_version {
 /* "capabilities" members */
 #define VFIO_USER_CAP_MAX_FDS   "max_msg_fds"
 #define VFIO_USER_CAP_MAX_XFER  "max_data_xfer_size"
+#define VFIO_USER_CAP_MIGR      "migration"
+
+/* "migration" member */
+#define VFIO_USER_CAP_PGSIZE    "pgsize"
 
 #define VFIO_USER_DEF_MAX_FDS   8
 #define VFIO_USER_MAX_MAX_FDS   16
hw/vfio/pci.c +5 -0
index 36f8524e7c..2f97160147 100644
@@ -3688,6 +3688,11 @@ static void vfio_user_pci_reset(DeviceState *dev)
 static Property vfio_user_pci_dev_properties[] = {
     DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name),
     DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false),
+    DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
+                     vbasedev.enable_migration, false),
+    DEFINE_PROP_ON_OFF_AUTO("x-pre-copy-dirty-page-tracking", VFIOPCIDevice,
+                            vbasedev.pre_copy_dirty_page_tracking,
+                            ON_OFF_AUTO_ON),
     DEFINE_PROP_END_OF_LIST(),
 };
 
hw/vfio/user.c +33 -0
index eceaeeccea..23ace82bbb 100644
@@ -393,6 +393,23 @@ static int caps_parse(QDict *qdict, struct cap_entry caps[], Error **errp)
     return 0;
 }
 
+static int check_pgsize(QObject *qobj, Error **errp)
+{
+    QNum *qn = qobject_to(QNum, qobj);
+    uint64_t pgsize;
+
+    if (qn == NULL || !qnum_get_try_uint(qn, &pgsize)) {
+        error_setg(errp, "malformed %s", VFIO_USER_CAP_PGSIZE);
+        return -1;
+    }
+    return pgsize == 4096 ? 0 : -1;
+}
+
+static struct cap_entry caps_migr[] = {
+    { VFIO_USER_CAP_PGSIZE, check_pgsize },
+    { NULL }
+};
+
 static int check_max_fds(QObject *qobj, Error **errp)
 {
     QNum *qn = qobject_to(QNum, qobj);
@@ -417,9 +434,21 @@ static int check_max_xfer(QObject *qobj, Error **errp)
     return 0;
 }
 
+static int check_migr(QObject *qobj, Error **errp)
+{
+    QDict *qdict = qobject_to(QDict, qobj);
+
+    if (qdict == NULL || caps_parse(qdict, caps_migr, errp)) {
+        error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS);
+        return -1;
+    }
+    return 0;
+}
+
 static struct cap_entry caps_cap[] = {
     { VFIO_USER_CAP_MAX_FDS, check_max_fds },
     { VFIO_USER_CAP_MAX_XFER, check_max_xfer },
+    { VFIO_USER_CAP_MIGR, check_migr },
     { NULL }
 };
 
@@ -466,8 +495,12 @@ static GString *caps_json(void)
 {
     QDict *dict = qdict_new();
     QDict *capdict = qdict_new();
+    QDict *migdict = qdict_new();
     GString *str;
 
+    qdict_put_int(migdict, VFIO_USER_CAP_PGSIZE, 4096);
+    qdict_put_obj(capdict, VFIO_USER_CAP_MIGR, QOBJECT(migdict));
+
     qdict_put_int(capdict, VFIO_USER_CAP_MAX_FDS, VFIO_USER_MAX_MAX_FDS);
     qdict_put_int(capdict, VFIO_USER_CAP_MAX_XFER, VFIO_USER_DEF_MAX_XFER);
 
Signature · 1 line
2.25.1
#29 lore raw

cover vfio-user server in QEMU

In reply to Elena Ufimtseva

Hi,

This series adds on to the following series from
Elena Ufimtseva <[email protected]>:
[PATCH RFC 00/19] vfio-user implementation

QEMU enabled out-of-process device emulation with multi-process [1]. multi-process used a custom protocol to interact between the client and server, which is not desirable.

The vfio-user user protocol [2] implements a VFIO based mechanism to interact between the client and server. Since VFIO is a well-established specification, it is preferable in terms of maintenance. It makes sense for multi-process to switch to the vfio-user protocol.

Nutanix implemented the vfio-user protocol in their libvfio-user library. The source for this library is located below:
https://github.com/nutanix/libvfio-user

Elena previously sent the patches for the vfio-user client.

This series implements a vfio-user server for QEMU. It includes the libvfio-user as a git submodule to QEMU, and builds it along with QEMU.

We would like to make the following notes:
- Some of the existing multi-process code would become obsolete, and would need to be removed. This series does not remove them to keep the number of patches to a minimum. We will address them subsequently.
- The libvfio-user library needs json-c package to build. It looks like the GitLab CI images used for build test don't have this package. As such it causes build failure.

The patches from both series are available in the following github repo:
https://github.com/oracle/qemu.git
The vfio-user-client-server branch provides the same patches along with a python script (scripts/vfiouser-launcher.py) to launch the VM.

Contributors:
John G Johnson <[email protected]>
John Levon <[email protected]>
Thanos Makatos <[email protected]>
Elena Ufimtseva <[email protected]>
Jagannathan Raman <[email protected]>

We are looking forward to your comments and questions.

Thank you!

[1]: https://patchew.org/QEMU/[email protected]/
[2]: https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/

Jagannathan Raman (11):
vfio-user: build library
vfio-user: define vfio-user object
vfio-user: instantiate vfio-user context
vfio-user: find and init PCI device
vfio-user: run vfio-user context
vfio-user: handle PCI config space accesses
vfio-user: handle DMA mappings
vfio-user: handle PCI BAR accesses
vfio-user: handle device interrupts
vfio-user: register handlers to facilitate migration
vfio-user: acceptance test

 configure                     |  11 +
 meson.build                   |  35 ++
 qapi/qom.json                 |  20 +-
 include/hw/remote/iohub.h     |   2 +
 migration/savevm.h            |   2 +
 hw/remote/iohub.c             |   6 +
 hw/remote/vfio-user-obj.c     | 754 ++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c            |  63 ++++
 .gitmodules                   |   3 +
 MAINTAINERS                   |   9 +
 hw/remote/meson.build         |   3 +
 hw/remote/trace-events        |  10 +
 libvfio-user                  |   1 +
 tests/acceptance/vfio-user.py |  94 ++++++
 14 files changed, 1011 insertions(+), 2 deletions(-)
 create mode 100644 hw/remote/vfio-user-obj.c
 create mode 160000 libvfio-user
 create mode 100644 tests/acceptance/vfio-user.py
Signature · 1 line
1.8.3.1
#30 lore raw

RFC PATCH 1/11 vfio-user: build library

add the libvfio-user library as a submodule. build it as part of QEMU

---
 configure             | 11 +++++++++++
 meson.build           | 35 +++++++++++++++++++++++++++++++++++
 .gitmodules           |  3 +++
 MAINTAINERS           |  7 +++++++
 hw/remote/meson.build |  2 ++
 libvfio-user          |  1 +
 6 files changed, 59 insertions(+)
 create mode 160000 libvfio-user
configure +11 -0
index 49b5481..bc1c961 100755
@@ -4297,6 +4297,17 @@ but not implemented on your system"
 fi
 
 ##########################################
+# check for multiprocess
+
+case "$multiprocess" in
+  auto | enabled )
+    if test "$git_submodules_action" != "ignore"; then
+      git_submodules="${git_submodules} libvfio-user"
+    fi
+    ;;
+esac
+
+##########################################
 # End of CC checks
 # After here, no more $cc or $ld runs
 
meson.build +35 -0
index 6e4d2d8..f2f9f86 100644
@@ -1894,6 +1894,41 @@ if get_option('cfi') and slirp_opt == 'system'
          + ' Please configure with --enable-slirp=git')
 endif
 
+vfiouser = not_found
+if have_system and multiprocess_allowed
+  have_internal = fs.exists(meson.current_source_dir() / 'libvfio-user/Makefile')
+
+  if not have_internal
+    error('libvfio-user source not found - please pull git submodule')
+  endif
+
+  vfiouser_files = [
+    'libvfio-user/lib/dma.c',
+    'libvfio-user/lib/irq.c',
+    'libvfio-user/lib/libvfio-user.c',
+    'libvfio-user/lib/migration.c',
+    'libvfio-user/lib/pci.c',
+    'libvfio-user/lib/pci_caps.c',
+    'libvfio-user/lib/tran_sock.c',
+  ]
+
+  vfiouser_inc = include_directories('libvfio-user/include', 'libvfio-user/lib')
+
+  json_c = dependency('json-c', required: false)
+  if not json_c.found()
+    json_c = dependency('libjson-c')
+  endif
+
+  libvfiouser = static_library('vfiouser',
+                               build_by_default: false,
+                               sources: vfiouser_files,
+                               dependencies: json_c,
+                               include_directories: vfiouser_inc)
+
+  vfiouser = declare_dependency(link_with: libvfiouser,
+                                include_directories: vfiouser_inc)
+endif
+
 fdt = not_found
 fdt_opt = get_option('fdt')
 if have_system
.gitmodules +3 -0
index 08b1b48..a583a39 100644
@@ -64,3 +64,6 @@
 [submodule "roms/vbootrom"]
 	path = roms/vbootrom
 	url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "libvfio-user"]
+	path = libvfio-user
+	url = https://github.com/nutanix/libvfio-user.git
MAINTAINERS +7 -0
index aa4df6c..99646e7 100644
@@ -3350,6 +3350,13 @@ F: semihosting/
 F: include/semihosting/
 F: tests/tcg/multiarch/arm-compat-semi/
 
+libvfio-user Library
+M: Thanos Makatos <[email protected]>
+M: John Levon <[email protected]>
+T: https://github.com/nutanix/libvfio-user.git
+S: Maintained
+F: libvfio-user/*
+
 Multi-process QEMU
 M: Elena Ufimtseva <[email protected]>
 M: Jagannathan Raman <[email protected]>
hw/remote/meson.build +2 -0
index e6a5574..fb35fb8 100644
@@ -7,6 +7,8 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
 
+remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: vfiouser)
+
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c'))
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
 
libvfio-user new file +1 -0
new file mode 160000
index 0000000..2a0a929
@@ -0,0 +1 @@
+Subproject commit 2a0a92912d598de871ab47c034432c5fa6546dc4
Signature · 1 line
1.8.3.1
#31 lore raw

Re: [PATCH RFC server 01/11] vfio-user: build library

Quoted 34 lines · Jagannathan Raman

add the libvfio-user library as a submodule. build it as part of QEMU

meson.build +24 -0
index 6e4d2d8..f2f9f86 100644
@@ -1894,6 +1894,41 @@ if get_option('cfi') and slirp_opt == 'system'
          + ' Please configure with --enable-slirp=git')
 endif
 
+vfiouser = not_found
+if have_system and multiprocess_allowed
+  have_internal = fs.exists(meson.current_source_dir() / 'libvfio-user/Makefile')
+
+  if not have_internal
+    error('libvfio-user source not found - please pull git submodule')
+  endif
+
+  vfiouser_files = [
+    'libvfio-user/lib/dma.c',
+    'libvfio-user/lib/irq.c',
+    'libvfio-user/lib/libvfio-user.c',
+    'libvfio-user/lib/migration.c',
+    'libvfio-user/lib/pci.c',
+    'libvfio-user/lib/pci_caps.c',
+    'libvfio-user/lib/tran_sock.c',
+  ]
+
+  vfiouser_inc = include_directories('libvfio-user/include', 'libvfio-user/lib')
+
+  json_c = dependency('json-c', required: false)
+  if not json_c.found()
+    json_c = dependency('libjson-c')
+  endif

meson.build if get_option('cfi') and slirp_opt == 'system' · line 1930

Quoted 10 lines
meson.build +10 -0
+
+  libvfiouser = static_library('vfiouser',
+                               build_by_default: false,
+                               sources: vfiouser_files,
+                               dependencies: json_c,
+                               include_directories: vfiouser_inc)
+
+  vfiouser = declare_dependency(link_with: libvfiouser,
+                                include_directories: vfiouser_inc)
+endif

Why this way, rather than recursing into the submodule? Seems a bit fragile to encode details of the library here.

regards
john

#32 lore raw

Re: [PATCH RFC server 01/11] vfio-user: build library

Quoted 45 lines · John Levon
Quoted 39 lines · Jagannathan Raman

add the libvfio-user library as a submodule. build it as part of QEMU

meson.build +0 -0
index 6e4d2d8..f2f9f86 100644
@@ -1894,6 +1894,41 @@ if get_option('cfi') and slirp_opt == 'system'
         + ' Please configure with --enable-slirp=git')

endif

meson.build +29 -0
+vfiouser = not_found
+if have_system and multiprocess_allowed
+  have_internal = fs.exists(meson.current_source_dir() / 'libvfio-user/Makefile')
+
+  if not have_internal
+    error('libvfio-user source not found - please pull git submodule')
+  endif
+
+  vfiouser_files = [
+    'libvfio-user/lib/dma.c',
+    'libvfio-user/lib/irq.c',
+    'libvfio-user/lib/libvfio-user.c',
+    'libvfio-user/lib/migration.c',
+    'libvfio-user/lib/pci.c',
+    'libvfio-user/lib/pci_caps.c',
+    'libvfio-user/lib/tran_sock.c',
+  ]
+
+  vfiouser_inc = include_directories('libvfio-user/include', 'libvfio-user/lib')
+
+  json_c = dependency('json-c', required: false)
+  if not json_c.found()
+    json_c = dependency('libjson-c')
+  endif
+
+  libvfiouser = static_library('vfiouser',
+                               build_by_default: false,
+                               sources: vfiouser_files,
+                               dependencies: json_c,
Quoted 8 lines

meson.build if get_option('cfi') and slirp_opt == 'system' · line 1930

Quoted 5 lines
meson.build +5 -0
+                               include_directories: vfiouser_inc)
+
+  vfiouser = declare_dependency(link_with: libvfiouser,
+                                include_directories: vfiouser_inc)
+endif

Why this way, rather than recursing into the submodule? Seems a bit fragile to encode details of the library here.

+maintainers of meson.build. I apologize for not adding them when I sent the patches out initially. I copied the email list from Elena, but Elena did not make any changes to meson.build - stupid me.

John,

This way appears to be present convention with QEMU - I’m also not very clear
on the reason for it.

For example submodules such as slirp (libslirp), capstone (libcapstone), dtc (libfdt) are built this way.

I’m guessing it’s because QEMU doesn’t build all parts of a submodule. For example, QEMU only builds libfdt in the doc submodule. Similarly, libvfio-user only builds the core library without building the tests and samples.

Quoted 3 lines

regards
john

#33 lore raw

Re: [PATCH RFC server 01/11] vfio-user: build library

Hi

Quoted 63 lines · Jag Raman
Quoted 16 lines · John Levon
Quoted 10 lines · Jagannathan Raman

add the libvfio-user library as a submodule. build it as part of QEMU

meson.build +0 -0
index 6e4d2d8..f2f9f86 100644
@@ -1894,6 +1894,41 @@ if get_option('cfi') and slirp_opt == 'system'
         + ' Please configure with --enable-slirp=git')

endif

Quoted 3 lines

meson.build if get_option('cfi') and slirp_opt == 'system'

Quoted 3 lines
meson.build +3 -0
+vfiouser = not_found
+if have_system and multiprocess_allowed
+  have_internal = fs.exists(meson.current_source_dir() /

'libvfio-user/Makefile')

Quoted 8 lines
Quoted 8 lines
meson.build +8 -0
+
+  if not have_internal
+    error('libvfio-user source not found - please pull git submodule')
+  endif
+
+  vfiouser_files = [
+    'libvfio-user/lib/dma.c',
+    'libvfio-user/lib/irq.c',
Quoted 8 lines

meson.build if get_option('cfi') and slirp_opt == 'system'

Quoted 8 lines
meson.build +8 -0
+    'libvfio-user/lib/libvfio-user.c',
+    'libvfio-user/lib/migration.c',
+    'libvfio-user/lib/pci.c',
+    'libvfio-user/lib/pci_caps.c',
+    'libvfio-user/lib/tran_sock.c',
+  ]
+
+  vfiouser_inc = include_directories('libvfio-user/include',

'libvfio-user/lib')

Quoted 9 lines
Quoted 9 lines
meson.build +9 -0
+
+  json_c = dependency('json-c', required: false)
+  if not json_c.found()
+    json_c = dependency('libjson-c')
+  endif
+
+  libvfiouser = static_library('vfiouser',
+                               build_by_default: false,
+                               sources: vfiouser_files,
Quoted 8 lines

meson.build if get_option('cfi') and slirp_opt == 'system' · line 1930

Quoted 6 lines
meson.build +6 -0
+                               dependencies: json_c,
+                               include_directories: vfiouser_inc)
+
+  vfiouser = declare_dependency(link_with: libvfiouser,
+                                include_directories: vfiouser_inc)
+endif

Why this way, rather than recursing into the submodule? Seems a bit

fragile to

Quoted 1 line

encode details of the library here.

+maintainers of meson.build. I apologize for not adding them when I sent the
patches out initially. I copied the email list from Elena, but Elena did not make
any changes to meson.build - stupid me.

Quoted 9 lines

John,

This way appears to be present convention with QEMU - I’m also not
very clear
on the reason for it.

For example submodules such as slirp (libslirp), capstone (libcapstone), dtc (libfdt) are built this way.

For slirp and dtc, we are eventually going to use meson subproject(). No idea about capstone.

Quoted 7 lines

I’m guessing it’s because QEMU doesn’t build all parts of a submodule. For example, QEMU only builds libfdt in the doc submodule. Similarly, libvfio-user only builds the core library without building the tests and samples.

You can give subproject options to build limited parts.

Fwiw, since libvfio-user uses cmake, we may be able to use meson cmake.subproject() (https://mesonbuild.com/CMake-module.html).

Signature · 1 line
Marc-André Lureau
#34 lore raw

Re: [PATCH RFC server 01/11] vfio-user: build library

Quoted 15 lines · Marc-André Lureau
Quoted 6 lines
Quoted 5 lines

meson.build if get_option('cfi') and slirp_opt == 'system' · line 1926

Quoted 5 lines
meson.build +5 -0
+  libvfiouser = static_library('vfiouser',
+                               build_by_default: false,
+                               sources: vfiouser_files,
+                               dependencies: json_c,
+                               include_directories: vfiouser_inc)
Quoted 9 lines

This way appears to be present convention with QEMU - I’m also not
very clear
on the reason for it.

I’m guessing it’s because QEMU doesn’t build all parts of a submodule. For example, QEMU only builds libfdt in the doc submodule. Similarly, libvfio-user only builds the core library without building the tests and samples.

Quoted 4 lines

You can give subproject options to build limited parts.

Fwiw, since libvfio-user uses cmake, we may be able to use meson cmake.subproject() (https://mesonbuild.com/CMake-module.html).

That'd be great. We also briefly discussed moving away from cmake anyway - since both SPDK and qemu are meson-based, it seems like it would make sense. I'd prefer it to be easy to regularly update libvfio-user within these projects.

Ideally, running qemu tests would actually run libvfio-user tests too, for some level of assurance on the library's internal expectations.

regards
john

#35 lore raw

RFC PATCH 2/11 vfio-user: define vfio-user object

In reply to Jagannathan Raman

Define vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object

---
 qapi/qom.json             |  20 ++++++-
 hw/remote/vfio-user-obj.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++
 MAINTAINERS               |   1 +
 hw/remote/meson.build     |   1 +
 hw/remote/trace-events    |   3 +
 5 files changed, 164 insertions(+), 2 deletions(-)
 create mode 100644 hw/remote/vfio-user-obj.c
qapi/qom.json +18 -2
index 652be31..e0716d2 100644
@@ -684,6 +684,20 @@
   'data': { 'fd': 'str', 'devid': 'str' } }
 
 ##
+# @VfioUserProperties:
+#
+# Properties for vfio-user objects.
+#
+# @socket: path to be used as socket by the libvfiouser library
+#
+# @devid: the id of the device to be associated with the file descriptor
+#
+# Since: 6.0
+##
+{ 'struct': 'VfioUserProperties',
+  'data': { 'socket': 'str', 'devid': 'str' } }
+
+##
 # @RngProperties:
 #
 # Properties for objects of classes derived from rng.
@@ -807,7 +821,8 @@
     'tls-creds-psk',
     'tls-creds-x509',
     'tls-cipher-suites',
-    'x-remote-object'
+    'x-remote-object',
+    'vfio-user'
   ] }
 
 ##
@@ -863,7 +878,8 @@
       'tls-creds-psk':              'TlsCredsPskProperties',
       'tls-creds-x509':             'TlsCredsX509Properties',
       'tls-cipher-suites':          'TlsCredsProperties',
-      'x-remote-object':            'RemoteObjectProperties'
+      'x-remote-object':            'RemoteObjectProperties',
+      'vfio-user':                  'VfioUserProperties'
   } }
 
 ##
hw/remote/vfio-user-obj.c new file +141 -0

141 lines not on this page · raw message

MAINTAINERS +1 -0
index 99646e7..46ab6b6 100644
@@ -3380,6 +3380,7 @@ F: hw/remote/proxy-memory-listener.c
 F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
+F: hw/remote/vfio-user-obj.c
 
 EBPF:
 M: Jason Wang <[email protected]>
hw/remote/meson.build +1 -0
index fb35fb8..cd44dfc 100644
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
+remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('vfio-user-obj.c'))
 
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: vfiouser)
 
hw/remote/trace-events +3 -0
index 0b23974..7da12f0 100644
@@ -2,3 +2,6 @@
 
 mpqemu_send_io_error(int cmd, int size, int nfds) "send command %d size %d, %d file descriptors to remote process"
 mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, %d file descriptors to remote process"
+
+# vfio-user-obj.c
+vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
Signature · 1 line
1.8.3.1
#36 lore raw

RFC PATCH 3/11 vfio-user: instantiate vfio-user context

In reply to Jagannathan Raman

create a context with the vfio-user library for a device

---
 hw/remote/vfio-user-obj.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
hw/remote/vfio-user-obj.c +30 -0
index 5098169..adb3193 100644
@@ -27,11 +27,18 @@
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 
+#include <errno.h>
+
 #include "qom/object.h"
 #include "qom/object_interfaces.h"
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "sysemu/runstate.h"
+#include "qemu/notify.h"
+#include "qapi/error.h"
+#include "sysemu/sysemu.h"
+
+#include "libvfio-user/include/libvfio-user.h"
 
 #define TYPE_VFU_OBJECT "vfio-user"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -51,6 +58,10 @@ struct VfuObject {
 
     char *socket;
     char *devid;
+
+    Notifier machine_done;
+
+    vfu_ctx_t *vfu_ctx;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -75,9 +86,23 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
     trace_vfu_prop("devid", str);
 }
 
+static void vfu_object_machine_done(Notifier *notifier, void *data)
+{
+    VfuObject *o = container_of(notifier, VfuObject, machine_done);
+
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
+                                o, VFU_DEV_TYPE_PCI);
+    if (o->vfu_ctx == NULL) {
+        error_setg(&error_abort, "vfu: Failed to create context - %s",
+                   strerror(errno));
+        return;
+    }
+}
+
 static void vfu_object_init(Object *obj)
 {
     VfuObjectClass *k = VFU_OBJECT_GET_CLASS(obj);
+    VfuObject *o = VFU_OBJECT(obj);
 
     /* Add test for remote machine and PCI device */
 
@@ -88,6 +113,9 @@ static void vfu_object_init(Object *obj)
     }
 
     k->nr_devs++;
+
+    o->machine_done.notify = vfu_object_machine_done;
+    qemu_add_machine_init_done_notifier(&o->machine_done);
 }
 
 static void vfu_object_finalize(Object *obj)
@@ -97,6 +125,8 @@ static void vfu_object_finalize(Object *obj)
 
     k->nr_devs--;
 
+    vfu_destroy_ctx(o->vfu_ctx);
+
     g_free(o->socket);
     g_free(o->devid);
 
Signature · 1 line
1.8.3.1
#37 lore raw

RFC PATCH 4/11 vfio-user: find and init PCI device

In reply to Jagannathan Raman

Find the PCI device with specified id. Initialize the device context with the QEMU PCI device

---
 hw/remote/vfio-user-obj.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
hw/remote/vfio-user-obj.c +28 -0
index adb3193..e362709 100644
@@ -37,6 +37,8 @@
 #include "qemu/notify.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
+#include "hw/qdev-core.h"
+#include "hw/pci/pci.h"
 
 #include "libvfio-user/include/libvfio-user.h"
 
@@ -62,6 +64,8 @@ struct VfuObject {
     Notifier machine_done;
 
     vfu_ctx_t *vfu_ctx;
+
+    PCIDevice *pci_dev;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -89,6 +93,8 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
+    DeviceState *dev = NULL;
+    int ret;
 
     o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
                                 o, VFU_DEV_TYPE_PCI);
@@ -97,6 +103,28 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
                    strerror(errno));
         return;
     }
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->devid);
+    if (dev == NULL) {
+        error_setg(&error_abort, "vfu: Device %s not found", o->devid);
+        return;
+    }
+    o->pci_dev = PCI_DEVICE(dev);
+
+    ret = vfu_pci_init(o->vfu_ctx, VFU_PCI_TYPE_CONVENTIONAL,
+                       PCI_HEADER_TYPE_NORMAL, 0);
+    if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to attach PCI device %s to context - %s",
+                   o->devid, strerror(errno));
+        return;
+    }
+
+    vfu_pci_set_id(o->vfu_ctx,
+                   pci_get_word(o->pci_dev->config + PCI_VENDOR_ID),
+                   pci_get_word(o->pci_dev->config + PCI_DEVICE_ID),
+                   pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_VENDOR_ID),
+                   pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_ID));
 }
 
 static void vfu_object_init(Object *obj)
Signature · 1 line
1.8.3.1
#38 lore raw

Re: [PATCH RFC server 04/11] vfio-user: find and init PCI device

hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 127

Quoted 5 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +5 -0
+    vfu_pci_set_id(o->vfu_ctx,
+                   pci_get_word(o->pci_dev->config + PCI_VENDOR_ID),
+                   pci_get_word(o->pci_dev->config + PCI_DEVICE_ID),
+                   pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_VENDOR_ID),
+                   pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_ID));

Since you handle all config space accesses yourselves, is there even any need for this?

regards
john

#39 lore raw

Re: [PATCH RFC server 04/11] vfio-user: find and init PCI device

Quoted 14 lines · John Levon

hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 127

Quoted 5 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +5 -0
+    vfu_pci_set_id(o->vfu_ctx,
+                   pci_get_word(o->pci_dev->config + PCI_VENDOR_ID),
+                   pci_get_word(o->pci_dev->config + PCI_DEVICE_ID),
+                   pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_VENDOR_ID),
+                   pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_ID));

Since you handle all config space accesses yourselves, is there even any need for this?

I think that makes sense. Since the QEMU server handles all the config space accesses, it’s not necessary to register the device’s vendor/device ID with the library.

Thank you!
--
Jag

Quoted 3 lines

regards
john

#40 lore raw

RFC PATCH 5/11 vfio-user: run vfio-user context

In reply to Jagannathan Raman

Setup a separate thread to run the vfio-user context. The thread acts as the main loop for the device.

---
 hw/remote/vfio-user-obj.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
hw/remote/vfio-user-obj.c +44 -0
index e362709..6a2d0f5 100644
@@ -35,6 +35,7 @@
 #include "trace.h"
 #include "sysemu/runstate.h"
 #include "qemu/notify.h"
+#include "qemu/thread.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "hw/qdev-core.h"
@@ -66,6 +67,8 @@ struct VfuObject {
     vfu_ctx_t *vfu_ctx;
 
     PCIDevice *pci_dev;
+
+    QemuThread vfu_ctx_thread;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -90,6 +93,44 @@ static void vfu_object_set_devid(Object *obj, const char *str, Error **errp)
     trace_vfu_prop("devid", str);
 }
 
+static void *vfu_object_ctx_run(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret;
+
+    ret = vfu_realize_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
+                   o->devid, strerror(errno));
+        return NULL;
+    }
+
+    ret = vfu_attach_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to attach device %s to context - %s",
+                   o->devid, strerror(errno));
+        return NULL;
+    }
+
+    do {
+        ret = vfu_run_ctx(o->vfu_ctx);
+        if (ret < 0) {
+            if (errno == EINTR) {
+                ret = 0;
+            } else if (errno == ENOTCONN) {
+                object_unparent(OBJECT(o));
+                break;
+            } else {
+                error_setg(&error_abort, "vfu: Failed to run device %s - %s",
+                           o->devid, strerror(errno));
+            }
+        }
+    } while (ret == 0);
+
+    return NULL;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -125,6 +166,9 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
                    pci_get_word(o->pci_dev->config + PCI_DEVICE_ID),
                    pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_VENDOR_ID),
                    pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_ID));
+
+    qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner", vfu_object_ctx_run,
+                       o, QEMU_THREAD_JOINABLE);
 }
 
 static void vfu_object_init(Object *obj)
Signature · 1 line
1.8.3.1
#41 lore raw

RE: [PATCH RFC server 05/11] vfio-user: run vfio-user context

Quoted 11 lines

-----Original Message-----
From: Jagannathan Raman <[email protected]>
Sent: 19 July 2021 21:00
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; John Levon <[email protected]>; [email protected]; Thanos Makatos <[email protected]>; Swapnil Ingle <[email protected]>; [email protected]
Subject: [PATCH RFC server 05/11] vfio-user: run vfio-user context

Quoted 2 lines

Setup a separate thread to run the vfio-user context. The thread acts as the main loop for the device.

In your "vfio-user: instantiate vfio-user context" patch you create the vfu context in blocking-mode, so the only way to run device emulation is in a separate thread. Were you going to create a separate thread anyway? You can run device emulation in polling mode therefore you can avoid creating a separate thread, thus saving resources. Do plan to do that in the future?

Quoted 91 lines
---
 hw/remote/vfio-user-obj.c | 44
++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
hw/remote/vfio-user-obj.c +41 -0
index e362709..6a2d0f5 100644
@@ -35,6 +35,7 @@
 #include "trace.h"
 #include "sysemu/runstate.h"
 #include "qemu/notify.h"
+#include "qemu/thread.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "hw/qdev-core.h"
@@ -66,6 +67,8 @@ struct VfuObject {
     vfu_ctx_t *vfu_ctx;
 
     PCIDevice *pci_dev;
+
+    QemuThread vfu_ctx_thread;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -90,6 +93,44 @@ static void vfu_object_set_devid(Object *obj, const
 char *str, Error **errp)
     trace_vfu_prop("devid", str);
 }
 
+static void *vfu_object_ctx_run(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret;
+
+    ret = vfu_realize_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
+                   o->devid, strerror(errno));
+        return NULL;
+    }
+
+    ret = vfu_attach_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to attach device %s to context - %s",
+                   o->devid, strerror(errno));
+        return NULL;
+    }
+
+    do {
+        ret = vfu_run_ctx(o->vfu_ctx);
+        if (ret < 0) {
+            if (errno == EINTR) {
+                ret = 0;
+            } else if (errno == ENOTCONN) {
+                object_unparent(OBJECT(o));
+                break;
+            } else {
+                error_setg(&error_abort, "vfu: Failed to run device %s - %s",
+                           o->devid, strerror(errno));
+            }
+        }
+    } while (ret == 0);
+
+    return NULL;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
hw/remote/vfio-user-obj.c +3 -0
@@ -125,6 +166,9 @@ static void vfu_object_machine_done(Notifier
 *notifier, void *data)
                    pci_get_word(o->pci_dev->config + PCI_DEVICE_ID),
                    pci_get_word(o->pci_dev->config +
 PCI_SUBSYSTEM_VENDOR_ID),
                    pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_ID));
+
+    qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner",
 vfu_object_ctx_run,
+                       o, QEMU_THREAD_JOINABLE);

}

static void vfu_object_init(Object *obj)
--
1.8.3.1

#42 lore raw

Re: [PATCH RFC server 05/11] vfio-user: run vfio-user context

Quoted 15 lines · Thanos Makatos
Quoted 12 lines

-----Original Message-----
From: Jagannathan Raman <[email protected]>
Sent: 19 July 2021 21:00
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; John Levon <[email protected]>; [email protected]; Thanos Makatos <[email protected]>; Swapnil Ingle <[email protected]>; [email protected]
Subject: [PATCH RFC server 05/11] vfio-user: run vfio-user context

Quoted 5 lines
Quoted 2 lines

Setup a separate thread to run the vfio-user context. The thread acts as the main loop for the device.

In your "vfio-user: instantiate vfio-user context" patch you create the vfu context in blocking-mode, so the only way to run device emulation is in a separate thread. Were you going to create a separate thread anyway? You can run device emulation in polling mode therefore you can avoid creating a separate thread, thus saving resources. Do plan to do that in the future?

Thanks for the information about the Blocking and Non-Blocking mode.

I’d like to explain why we are using a separate thread presently and check with you if it’s possible to poll on multiple vfu contexts at the same time (similar to select/poll for fds).

Concerning my understanding on how devices are executed in QEMU, QEMU initializes the device instance - where the device registers callbacks for BAR and config space accesses. The device is then subsequently driven by these callbacks - whenever the vcpu thread tries to access the BAR addresses or places a config space access to the PCI bus, the vcpu exits to QEMU which handles these accesses. As such, the device is driven by the vcpu thread. Since there are no vcpu threads in the remote process, we created a separate thread as a replacement. As you can see already, this thread blocks on vfu_run_ctx() which I believe polls on the socket for messages from client.

If there is a way to run multiple vfu contexts at the same time, that would help with conserving threads on the host CPU. For example, if there’s a way to add vfu contexts to a list of contexts that expect messages from client, that could be a good idea. Alternatively, this QEMU server could also implement a similar mechanism to group all non-blocking vfu contexts to just a single thread, instead of having separate threads for each context.

--
Jag

Quoted 93 lines
Quoted 92 lines

---
hw/remote/vfio-user-obj.c | 44
++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 44 insertions(+)
hw/remote/vfio-user-obj.c +41 -0
index e362709..6a2d0f5 100644
@@ -35,6 +35,7 @@
 #include "trace.h"
 #include "sysemu/runstate.h"
 #include "qemu/notify.h"
+#include "qemu/thread.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "hw/qdev-core.h"
@@ -66,6 +67,8 @@ struct VfuObject {
    vfu_ctx_t *vfu_ctx;
 
    PCIDevice *pci_dev;
+
+    QemuThread vfu_ctx_thread;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)
@@ -90,6 +93,44 @@ static void vfu_object_set_devid(Object *obj, const
 char *str, Error **errp)
    trace_vfu_prop("devid", str);
 }
 
+static void *vfu_object_ctx_run(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret;
+
+    ret = vfu_realize_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to realize device %s- %s",
+                   o->devid, strerror(errno));
+        return NULL;
+    }
+
+    ret = vfu_attach_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to attach device %s to context - %s",
+                   o->devid, strerror(errno));
+        return NULL;
+    }
+
+    do {
+        ret = vfu_run_ctx(o->vfu_ctx);
+        if (ret < 0) {
+            if (errno == EINTR) {
+                ret = 0;
+            } else if (errno == ENOTCONN) {
+                object_unparent(OBJECT(o));
+                break;
+            } else {
+                error_setg(&error_abort, "vfu: Failed to run device %s - %s",
+                           o->devid, strerror(errno));
+            }
+        }
+    } while (ret == 0);
+
+    return NULL;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
    VfuObject *o = container_of(notifier, VfuObject, machine_done);
hw/remote/vfio-user-obj.c +3 -0
@@ -125,6 +166,9 @@ static void vfu_object_machine_done(Notifier
 *notifier, void *data)
                   pci_get_word(o->pci_dev->config + PCI_DEVICE_ID),
                   pci_get_word(o->pci_dev->config +
 PCI_SUBSYSTEM_VENDOR_ID),
                   pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_ID));
+
+    qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner",
 vfu_object_ctx_run,
+                       o, QEMU_THREAD_JOINABLE);

}

static void vfu_object_init(Object *obj)
--
1.8.3.1

#43 lore raw

Re: [PATCH RFC server 05/11] vfio-user: run vfio-user context

Quoted 17 lines · Jag Raman

Thanks for the information about the Blocking and Non-Blocking mode.

I’d like to explain why we are using a separate thread presently and check with you if it’s possible to poll on multiple vfu contexts at the same time (similar to select/poll for fds).

Concerning my understanding on how devices are executed in QEMU, QEMU initializes the device instance - where the device registers callbacks for BAR and config space accesses. The device is then subsequently driven by these callbacks - whenever the vcpu thread tries to access the BAR addresses or places a config space access to the PCI bus, the vcpu exits to QEMU which handles these accesses. As such, the device is driven by the vcpu thread. Since there are no vcpu threads in the remote process, we created a separate thread as a replacement. As you can see already, this thread blocks on vfu_run_ctx() which I believe polls on the socket for messages from client.

Quoted 7 lines

If there is a way to run multiple vfu contexts at the same time, that would help with conserving threads on the host CPU. For example, if there’s a way to add vfu contexts to a list of contexts that expect messages from client, that could be a good idea. Alternatively, this QEMU server could also implement a similar mechanism to group all non-blocking vfu contexts to just a single thread, instead of having separate threads for each context.

You can use vfu_get_poll_fd() to retrieve the underlying socket fd (simplest would be to do this after vfu_attach_ctx(), but that might depend), then poll on the fd set, doing vfu_run_ctx() when the fd is ready. An async hangup on the socket would show up as ENOTCONN, in which case you'd remove the fd from the set.

Note that we're not completely async yet (e.g. the actual socket read/writes are synchronous). In practice that's not typically an issue but it could be if you wanted to support multiple VMs from a single server, etc.

regards
john

#44 lore raw

Re: [PATCH RFC server 05/11] vfio-user: run vfio-user context

Quoted 31 lines · John Levon
Quoted 17 lines · Jag Raman

Thanks for the information about the Blocking and Non-Blocking mode.

I’d like to explain why we are using a separate thread presently and check with you if it’s possible to poll on multiple vfu contexts at the same time (similar to select/poll for fds).

Concerning my understanding on how devices are executed in QEMU, QEMU initializes the device instance - where the device registers callbacks for BAR and config space accesses. The device is then subsequently driven by these callbacks - whenever the vcpu thread tries to access the BAR addresses or places a config space access to the PCI bus, the vcpu exits to QEMU which handles these accesses. As such, the device is driven by the vcpu thread. Since there are no vcpu threads in the remote process, we created a separate thread as a replacement. As you can see already, this thread blocks on vfu_run_ctx() which I believe polls on the socket for messages from client.

Quoted 7 lines

If there is a way to run multiple vfu contexts at the same time, that would help with conserving threads on the host CPU. For example, if there’s a way to add vfu contexts to a list of contexts that expect messages from client, that could be a good idea. Alternatively, this QEMU server could also implement a similar mechanism to group all non-blocking vfu contexts to just a single thread, instead of having separate threads for each context.

Quoted 5 lines

You can use vfu_get_poll_fd() to retrieve the underlying socket fd (simplest would be to do this after vfu_attach_ctx(), but that might depend), then poll on the fd set, doing vfu_run_ctx() when the fd is ready. An async hangup on the socket would show up as ENOTCONN, in which case you'd remove the fd from the set.

OK sounds good, will check this model out. Thank you!

--
Jag

Quoted 8 lines

Note that we're not completely async yet (e.g. the actual socket read/writes are synchronous). In practice that's not typically an issue but it could be if you wanted to support multiple VMs from a single server, etc.

regards
john

#45 lore raw

RFC PATCH 6/11 vfio-user: handle PCI config space accesses

In reply to Jagannathan Raman

Define and register handlers for PCI config space accesses

---
 hw/remote/vfio-user-obj.c | 41 +++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 43 insertions(+)
hw/remote/vfio-user-obj.c +41 -0
index 6a2d0f5..60d9fa8 100644
@@ -36,6 +36,7 @@
 #include "sysemu/runstate.h"
 #include "qemu/notify.h"
 #include "qemu/thread.h"
+#include "qemu/main-loop.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "hw/qdev-core.h"
@@ -131,6 +132,35 @@ static void *vfu_object_ctx_run(void *opaque)
     return NULL;
 }
 
+static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
+                                     size_t count, loff_t offset,
+                                     const bool is_write)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint32_t val = 0;
+    int i;
+
+    qemu_mutex_lock_iothread();
+
+    for (i = 0; i < count; i++) {
+        if (is_write) {
+            val = *((uint8_t *)(buf + i));
+            trace_vfu_cfg_write((offset + i), val);
+            pci_default_write_config(PCI_DEVICE(o->pci_dev),
+                                     (offset + i), val, 1);
+        } else {
+            val = pci_default_read_config(PCI_DEVICE(o->pci_dev),
+                                          (offset + i), 1);
+            *((uint8_t *)(buf + i)) = (uint8_t)val;
+            trace_vfu_cfg_read((offset + i), val);
+        }
+    }
+
+    qemu_mutex_unlock_iothread();
+
+    return count;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -167,6 +197,17 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
                    pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_VENDOR_ID),
                    pci_get_word(o->pci_dev->config + PCI_SUBSYSTEM_ID));
 
+    ret = vfu_setup_region(o->vfu_ctx, VFU_PCI_DEV_CFG_REGION_IDX,
+                           pci_config_size(o->pci_dev), &vfu_object_cfg_access,
+                           VFU_REGION_FLAG_RW | VFU_REGION_FLAG_ALWAYS_CB,
+                           NULL, 0, -1, 0);
+    if (ret < 0) {
+        error_setg(&error_abort,
+                   "vfu: Failed to setup config space handlers for %s- %s",
+                   o->devid, strerror(errno));
+        return;
+    }
+
     qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner", vfu_object_ctx_run,
                        o, QEMU_THREAD_JOINABLE);
 }
hw/remote/trace-events +2 -0
index 7da12f0..2ef7884 100644
@@ -5,3 +5,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
 
 # vfio-user-obj.c
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
+vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
+vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
Signature · 1 line
1.8.3.1
#46 lore raw

Re: [PATCH RFC server 06/11] vfio-user: handle PCI config space accesses

Quoted 39 lines · Jagannathan Raman

Define and register handlers for PCI config space accesses

---
 hw/remote/vfio-user-obj.c | 41 +++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 43 insertions(+)
hw/remote/vfio-user-obj.c +14 -0
index 6a2d0f5..60d9fa8 100644
@@ -36,6 +36,7 @@
 #include "sysemu/runstate.h"
 #include "qemu/notify.h"
 #include "qemu/thread.h"
+#include "qemu/main-loop.h"
 #include "qapi/error.h"
 #include "sysemu/sysemu.h"
 #include "hw/qdev-core.h"
@@ -131,6 +132,35 @@ static void *vfu_object_ctx_run(void *opaque)
     return NULL;
 }
 
+static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
+                                     size_t count, loff_t offset,
+                                     const bool is_write)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint32_t val = 0;
+    int i;
+
+    qemu_mutex_lock_iothread();
+
+    for (i = 0; i < count; i++) {
+        if (is_write) {
+            val = *((uint8_t *)(buf + i));

hw/remote/vfio-user-obj.c static void *vfu_object_ctx_run(void *opaque) · line 157

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+            trace_vfu_cfg_write((offset + i), val);
+            pci_default_write_config(PCI_DEVICE(o->pci_dev),
+                                     (offset + i), val, 1);
+        } else {
+            val = pci_default_read_config(PCI_DEVICE(o->pci_dev),
+                                          (offset + i), 1);
+            *((uint8_t *)(buf + i)) = (uint8_t)val;
+            trace_vfu_cfg_read((offset + i), val);
+        }
+    }

Is it always OK to split up the access into single bytes like this?

regards
john

#47 lore raw

RFC PATCH 7/11 vfio-user: handle DMA mappings

In reply to Jagannathan Raman

Define and register callbacks to manage the RAM regions used for device DMA

---
 hw/remote/vfio-user-obj.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 60 insertions(+)
hw/remote/vfio-user-obj.c +58 -0
index 60d9fa8..d158a7f 100644
@@ -161,6 +161,57 @@ static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
     return count;
 }
 
+static void dma_register(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    MemoryRegion *subregion = NULL;
+    g_autofree char *name = NULL;
+    static unsigned int suffix;
+    struct iovec *iov = &info->iova;
+
+    if (!info->vaddr) {
+        return;
+    }
+
+    name = g_strdup_printf("remote-mem-%u", suffix++);
+
+    subregion = g_new0(MemoryRegion, 1);
+
+    qemu_mutex_lock_iothread();
+
+    memory_region_init_ram_ptr(subregion, NULL, name,
+                               iov->iov_len, info->vaddr);
+
+    memory_region_add_subregion(get_system_memory(), (hwaddr)iov->iov_base,
+                                subregion);
+
+    qemu_mutex_unlock_iothread();
+
+    trace_vfu_dma_register((uint64_t)iov->iov_base, iov->iov_len);
+}
+
+static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    MemoryRegion *mr = NULL;
+    ram_addr_t offset;
+
+    mr = memory_region_from_host(info->vaddr, &offset);
+    if (!mr) {
+        return 0;
+    }
+
+    qemu_mutex_lock_iothread();
+
+    memory_region_del_subregion(get_system_memory(), mr);
+
+    object_unparent((OBJECT(mr)));
+
+    qemu_mutex_unlock_iothread();
+
+    trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
+
+    return 0;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -208,6 +259,13 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
         return;
     }
 
+    ret = vfu_setup_device_dma(o->vfu_ctx, &dma_register, &dma_unregister);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to setup DMA handlers for %s",
+                   o->devid);
+        return;
+    }
+
     qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner", vfu_object_ctx_run,
                        o, QEMU_THREAD_JOINABLE);
 }
hw/remote/trace-events +2 -0
index 2ef7884..f945c7e 100644
@@ -7,3 +7,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
 vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
+vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
+vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
Signature · 1 line
1.8.3.1
#48 lore raw

RE: [PATCH RFC server 07/11] vfio-user: handle DMA mappings

Quoted 32 lines

-----Original Message-----
From: Jagannathan Raman <[email protected]>
Sent: 19 July 2021 21:00
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; John Levon <[email protected]>; [email protected]; Thanos Makatos <[email protected]>; Swapnil Ingle <[email protected]>; [email protected]
Subject: [PATCH RFC server 07/11] vfio-user: handle DMA mappings

Define and register callbacks to manage the RAM regions used for device DMA

---
 hw/remote/vfio-user-obj.c | 58
+++++++++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 60 insertions(+)
hw/remote/vfio-user-obj.c +0 -0
index 60d9fa8..d158a7f 100644
@@ -161,6 +161,57 @@ static ssize_t vfu_object_cfg_access(vfu_ctx_t
 *vfu_ctx, char * const buf,
     return count;
 }
 

hw/remote/vfio-user-obj.c static ssize_t vfu_object_cfg_access(vfu_ctx_t · line 174

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+static void dma_register(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    MemoryRegion *subregion = NULL;
+    g_autofree char *name = NULL;
+    static unsigned int suffix;
+    struct iovec *iov = &info->iova;
+
+    if (!info->vaddr) {
+        return;
+    }

This shouldn't happen, you can replace it with an assert if you want.

Quoted 16 lines
hw/remote/vfio-user-obj.c +15 -0
+
+    name = g_strdup_printf("remote-mem-%u", suffix++);
+
+    subregion = g_new0(MemoryRegion, 1);
+
+    qemu_mutex_lock_iothread();
+
+    memory_region_init_ram_ptr(subregion, NULL, name,
+                               iov->iov_len, info->vaddr);
+
+    memory_region_add_subregion(get_system_memory(), (hwaddr)iov-
 >iov_base,
+                                subregion);
+
+    qemu_mutex_unlock_iothread();
+

hw/remote/vfio-user-obj.c static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf, · line 198

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+    trace_vfu_dma_register((uint64_t)iov->iov_base, iov->iov_len);
+}
+
+static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    MemoryRegion *mr = NULL;
+    ram_addr_t offset;
+
+    mr = memory_region_from_host(info->vaddr, &offset);
+    if (!mr) {

Is this expected? If not then should we at least log something?

Quoted 50 lines
hw/remote/vfio-user-obj.c +16 -0
+        return 0;
+    }
+
+    qemu_mutex_lock_iothread();
+
+    memory_region_del_subregion(get_system_memory(), mr);
+
+    object_unparent((OBJECT(mr)));
+
+    qemu_mutex_unlock_iothread();
+
+    trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
+
+    return 0;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
hw/remote/vfio-user-obj.c +7 -0
@@ -208,6 +259,13 @@ static void vfu_object_machine_done(Notifier
 *notifier, void *data)
         return;
     }
 
+    ret = vfu_setup_device_dma(o->vfu_ctx, &dma_register,
 &dma_unregister);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to setup DMA handlers for %s",
+                   o->devid);
+        return;
+    }
+
     qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner",

vfu_object_ctx_run,

                        o, QEMU_THREAD_JOINABLE);

}

hw/remote/trace-events +1 -0
index 2ef7884..f945c7e 100644
@@ -7,3 +7,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed
 to receive %d size %d,
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
 vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
+vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA

0x%"PRIx64", %zu bytes"
+vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" --
1.8.3.1

#49 lore raw

RFC PATCH 8/11 vfio-user: handle PCI BAR accesses

In reply to Jagannathan Raman

Determine the BARs used by the PCI device and register handlers to manage the access to the same.

---
 hw/remote/vfio-user-obj.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 +
 2 files changed, 97 insertions(+)
hw/remote/vfio-user-obj.c +95 -0
index d158a7f..9853feb 100644
@@ -212,6 +212,99 @@ static int dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
     return 0;
 }
 
+static ssize_t vfu_object_bar_rw(PCIDevice *pci_dev, hwaddr addr, size_t count,
+                                 char * const buf, const bool is_write,
+                                 uint8_t type)
+{
+    AddressSpace *as = NULL;
+    MemTxResult res;
+
+    if (type == PCI_BASE_ADDRESS_SPACE_MEMORY) {
+        as = pci_device_iommu_address_space(pci_dev);
+    } else {
+        as = &address_space_io;
+    }
+
+    trace_vfu_bar_rw_enter(is_write ? "Write" : "Read", (uint64_t)addr);
+
+    res = address_space_rw(as, addr, MEMTXATTRS_UNSPECIFIED, (void *)buf,
+                           (hwaddr)count, is_write);
+    if (res != MEMTX_OK) {
+        warn_report("vfu: failed to %s 0x%"PRIx64"",
+                    is_write ? "write to" : "read from",
+                    addr);
+        return -1;
+    }
+
+    trace_vfu_bar_rw_exit(is_write ? "Write" : "Read", (uint64_t)addr);
+
+    return count;
+}
+
+/**
+ * VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs.
+ *
+ * To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would
+ * define vfu_object_bar2_handler
+ */
+#define VFU_OBJECT_BAR_HANDLER(BAR_NO)                                         \
+    static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx,        \
+                                        char * const buf, size_t count,        \
+                                        loff_t offset, const bool is_write)    \
+    {                                                                          \
+        VfuObject *o = vfu_get_private(vfu_ctx);                               \
+        hwaddr addr = (hwaddr)(pci_get_long(o->pci_dev->config +               \
+                                            PCI_BASE_ADDRESS_0 +               \
+                                            (4 * BAR_NO)) + offset);           \
+                                                                               \
+        return vfu_object_bar_rw(o->pci_dev, addr, count, buf, is_write,       \
+                                 o->pci_dev->io_regions[BAR_NO].type);         \
+    }                                                                          \
+
+VFU_OBJECT_BAR_HANDLER(0)
+VFU_OBJECT_BAR_HANDLER(1)
+VFU_OBJECT_BAR_HANDLER(2)
+VFU_OBJECT_BAR_HANDLER(3)
+VFU_OBJECT_BAR_HANDLER(4)
+VFU_OBJECT_BAR_HANDLER(5)
+
+static vfu_region_access_cb_t *vfu_object_bar_handlers[PCI_NUM_REGIONS] = {
+    &vfu_object_bar0_handler,
+    &vfu_object_bar1_handler,
+    &vfu_object_bar2_handler,
+    &vfu_object_bar3_handler,
+    &vfu_object_bar4_handler,
+    &vfu_object_bar5_handler,
+};
+
+/**
+ * vfu_object_register_bars - Identify active BAR regions of pdev and setup
+ *                            callbacks to handle read/write accesses
+ */
+static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
+{
+    uint32_t orig_val, new_val;
+    int i, size;
+
+    for (i = 0; i < PCI_NUM_REGIONS; i++) {
+        orig_val = pci_default_read_config(pdev,
+                                           PCI_BASE_ADDRESS_0 + (4 * i), 4);
+        new_val = 0xffffffff;
+        pci_default_write_config(pdev,
+                                 PCI_BASE_ADDRESS_0 + (4 * i), new_val, 4);
+        new_val = pci_default_read_config(pdev,
+                                          PCI_BASE_ADDRESS_0 + (4 * i), 4);
+        size = (~(new_val & 0xFFFFFFF0)) + 1;
+        pci_default_write_config(pdev, PCI_BASE_ADDRESS_0 + (4 * i),
+                                 orig_val, 4);
+        if (size) {
+            vfu_setup_region(vfu_ctx, VFU_PCI_DEV_BAR0_REGION_IDX + i, size,
+                             vfu_object_bar_handlers[i], VFU_REGION_FLAG_RW,
+                             NULL, 0, -1, 0);
+        }
+    }
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -266,6 +359,8 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
         return;
     }
 
+    vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
+
     qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner", vfu_object_ctx_run,
                        o, QEMU_THREAD_JOINABLE);
 }
hw/remote/trace-events +2 -0
index f945c7e..f3f65e2 100644
@@ -9,3 +9,5 @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
 vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
 vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
+vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
+vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
Signature · 1 line
1.8.3.1
#50 lore raw

RFC PATCH 9/11 vfio-user: handle device interrupts

In reply to Jagannathan Raman

Forward remote device's interrupts to the guest

---
 include/hw/remote/iohub.h |  2 ++
 hw/remote/iohub.c         |  6 ++++++
 hw/remote/vfio-user-obj.c | 30 ++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  1 +
 4 files changed, 39 insertions(+)
include/hw/remote/iohub.h +2 -0
index 0bf98e0..132f496 100644
@@ -15,6 +15,7 @@
 #include "qemu/event_notifier.h"
 #include "qemu/thread-posix.h"
 #include "hw/remote/mpqemu-link.h"
+#include "libvfio-user/include/libvfio-user.h"
 
 #define REMOTE_IOHUB_NB_PIRQS    PCI_DEVFN_MAX
 
@@ -30,6 +31,7 @@ typedef struct RemoteIOHubState {
     unsigned int irq_level[REMOTE_IOHUB_NB_PIRQS];
     ResampleToken token[REMOTE_IOHUB_NB_PIRQS];
     QemuMutex irq_level_lock[REMOTE_IOHUB_NB_PIRQS];
+    vfu_ctx_t *vfu_ctx[REMOTE_IOHUB_NB_PIRQS];
 } RemoteIOHubState;
 
 int remote_iohub_map_irq(PCIDevice *pci_dev, int intx);
hw/remote/iohub.c +6 -0
index 547d597..241c8d7 100644
@@ -18,6 +18,8 @@
 #include "hw/remote/machine.h"
 #include "hw/remote/iohub.h"
 #include "qemu/main-loop.h"
+#include "libvfio-user/include/libvfio-user.h"
+#include "trace.h"
 
 void remote_iohub_init(RemoteIOHubState *iohub)
 {
@@ -62,6 +64,10 @@ void remote_iohub_set_irq(void *opaque, int pirq, int level)
     QEMU_LOCK_GUARD(&iohub->irq_level_lock[pirq]);
 
     if (level) {
+        if (iohub->vfu_ctx[pirq]) {
+            trace_vfu_interrupt(pirq);
+            vfu_irq_trigger(iohub->vfu_ctx[pirq], 0);
+        }
         if (++iohub->irq_level[pirq] == 1) {
             event_notifier_set(&iohub->irqfds[pirq]);
         }
hw/remote/vfio-user-obj.c +30 -0
index 9853feb..d2a2e51 100644
@@ -41,6 +41,9 @@
 #include "sysemu/sysemu.h"
 #include "hw/qdev-core.h"
 #include "hw/pci/pci.h"
+#include "hw/boards.h"
+#include "hw/remote/iohub.h"
+#include "hw/remote/machine.h"
 
 #include "libvfio-user/include/libvfio-user.h"
 
@@ -305,6 +308,26 @@ static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
     }
 }
 
+static int vfu_object_setup_irqs(vfu_ctx_t *vfu_ctx, PCIDevice *pci_dev)
+{
+    RemoteMachineState *machine = REMOTE_MACHINE(current_machine);
+    RemoteIOHubState *iohub = &machine->iohub;
+    int pirq, intx, ret;
+
+    ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_INTX_IRQ, 1);
+    if (ret < 0) {
+        return ret;
+    }
+
+    intx = pci_get_byte(pci_dev->config + PCI_INTERRUPT_PIN) - 1;
+
+    pirq = remote_iohub_map_irq(pci_dev, intx);
+
+    iohub->vfu_ctx[pirq] = vfu_ctx;
+
+    return 0;
+}
+
 static void vfu_object_machine_done(Notifier *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
@@ -361,6 +384,13 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
 
     vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
 
+    ret = vfu_object_setup_irqs(o->vfu_ctx, o->pci_dev);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to setup interrupts for %s",
+                   o->devid);
+        return;
+    }
+
     qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner", vfu_object_ctx_run,
                        o, QEMU_THREAD_JOINABLE);
 }
hw/remote/trace-events +1 -0
index f3f65e2..b419d6f 100644
@@ -11,3 +11,4 @@ vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %z
 vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
 vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
 vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
+vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"
Signature · 1 line
1.8.3.1
#51 lore raw

RFC PATCH 10/11 vfio-user: register handlers to facilitate migration

In reply to Jagannathan Raman

Store and load the device's state using handlers for live migration

---
 migration/savevm.h        |   2 +
 hw/remote/vfio-user-obj.c | 287 ++++++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c        |  63 ++++++++++
 3 files changed, 352 insertions(+)
migration/savevm.h +2 -0
index 6461342..71d1733 100644
@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
 int qemu_load_device_state(QEMUFile *f);
 int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
         bool in_postcopy, bool inactivate_disks);
+int qemu_remote_savevm(QEMUFile *f);
+int qemu_remote_loadvm(QEMUFile *f);
 
 #endif
hw/remote/vfio-user-obj.c +287 -0

323 lines not on this page · raw message

migration/savevm.c +63 -0

75 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#52 lore raw

RE: [PATCH RFC server 10/11] vfio-user: register handlers to facilitate migration

Quoted 55 lines

-----Original Message-----
From: Jagannathan Raman <[email protected]>
Sent: 19 July 2021 21:00
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; John Levon <[email protected]>; [email protected]; Thanos Makatos <[email protected]>; Swapnil Ingle <[email protected]>; [email protected]
Subject: [PATCH RFC server 10/11] vfio-user: register handlers to facilitate migration

Store and load the device's state using handlers for live migration

---
 migration/savevm.h        |   2 +
 hw/remote/vfio-user-obj.c | 287
++++++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c        |  63 ++++++++++
 3 files changed, 352 insertions(+)
migration/savevm.h +2 -0
index 6461342..71d1733 100644
@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f,
 MigrationIncomingState *mis);
 int qemu_load_device_state(QEMUFile *f);
 int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
         bool in_postcopy, bool inactivate_disks);
+int qemu_remote_savevm(QEMUFile *f);
+int qemu_remote_loadvm(QEMUFile *f);
 

#endif

hw/remote/vfio-user-obj.c +4 -0
index d2a2e51..5948576 100644
@@ -44,6 +44,10 @@
 #include "hw/boards.h"
 #include "hw/remote/iohub.h"
 #include "hw/remote/machine.h"
+#include "migration/qemu-file.h"
+#include "migration/savevm.h"
+#include "migration/global_state.h"
+#include "block/block.h"
 
 #include "libvfio-user/include/libvfio-user.h"
 
@@ -73,6 +77,31 @@ struct VfuObject {
     PCIDevice *pci_dev;
 

hw/remote/vfio-user-obj.c struct VfuObject { · line 88

Quoted 10 lines
hw/remote/vfio-user-obj.c +9 -0
     QemuThread vfu_ctx_thread;
+
+    /*
+     * vfu_mig_buf holds the migration data. In the remote process, this
+     * buffer replaces the role of an IO channel which links the source
+     * and the destination.
+     *
+     * Whenever the client QEMU process initiates migration, the libvfio-user
+     * library notifies that to this server. The remote/server QEMU sets up a
+     * QEMUFile object using this buffer as backend. The remote passes this

Can we use remote/server more consistently? E.g. "remote process" or "server" instead of just "remote"? (makes me think of git remotes :D)

hw/remote/vfio-user-obj.c struct VfuObject { · line 89

Quoted 1 line
hw/remote/vfio-user-obj.c +1 -0
+     * object to its migration subsystem, and it slirps the VMSDs of all its

By "slirps" do you mean transfer from the client to the server over the SLiRP network?

hw/remote/vfio-user-obj.c struct VfuObject { · line 90

Quoted 1 line
hw/remote/vfio-user-obj.c +1 -0
+     * devices and stores them in this buffer.

Isn't this a per-device object? If so, then why do we store the VMSDs of *all* the devices in a single device's buffer? I think I'm missing something here.

hw/remote/vfio-user-obj.c struct VfuObject { · line 93

Quoted 3 lines
hw/remote/vfio-user-obj.c +3 -0
+     *
+     * libvfio-user library subsequetly asks the remote for any data that needs
+     * to be moved over to the destination using its vfu_migration_callbacks_t

It's not obvious to me, is this the libvfio-user library running at the server?

hw/remote/vfio-user-obj.c struct VfuObject { · line 94

Quoted 1 line
hw/remote/vfio-user-obj.c +1 -0
+     * APIs. The remote hands over this buffer as data at this time.

Hands over the buffer to whom?

hw/remote/vfio-user-obj.c struct VfuObject { · line 98

Quoted 4 lines
hw/remote/vfio-user-obj.c +4 -0
+     *
+     * A reverse of this process happens at the destination.
+     */
+    uint8_t *vfu_mig_buf;

Does the above description refer to a typical use case of the VFIO migration protocol where data is copied in an iterative manner (implemented in libvfio-user by the migration callbacks)? Is this what you're documenting here?

Quoted 10 lines
hw/remote/vfio-user-obj.c +6 -0
+
+    uint64_t vfu_mig_buf_size;
+
+    uint64_t vfu_mig_buf_pending;
+
+    QEMUFile *vfu_mig_file;
 };
 
 static void vfu_object_set_socket(Object *obj, const char *str, Error **errp)

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const · line 134

Quoted 9 lines
hw/remote/vfio-user-obj.c +5 -0
@@ -97,6 +126,226 @@ static void vfu_object_set_devid(Object *obj, const
 char *str, Error **errp)
     trace_vfu_prop("devid", str);
 }
 
+/**
+ * Migration helper functions
+ *
+ * vfu_mig_buf_read & vfu_mig_buf_write are used by QEMU's migration
+ * subsystem - qemu_remote_savevm & qemu_remote_loadvm.

vfu_mig_buf_read is used by qemu_remote_loadvm and vfu_mig_buf_write is used by qemu_remote_savevm, right? The order they're written suggests the opposite.

Quoted 113 lines

savevm/loadvm
+ * call these functions via QEMUFileOps to save/load the VMSD of all
+ * the devices into vfu_mig_buf
+ *
+ */

hw/remote/vfio-user-obj.c +40 -0
+static ssize_t vfu_mig_buf_read(void *opaque, uint8_t *buf, int64_t pos,
+                                size_t size, Error **errp)
+{
+    VfuObject *o = opaque;
+
+    if (pos > o->vfu_mig_buf_size) {
+        size = 0;
+    } else if ((pos + size) > o->vfu_mig_buf_size) {
+        size = o->vfu_mig_buf_size;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + pos), size);
+
+    o->vfu_mig_buf_size -= size;
+
+    return size;
+}
+
+static ssize_t vfu_mig_buf_write(void *opaque, struct iovec *iov, int iovcnt,
+                                 int64_t pos, Error **errp)
+{
+    VfuObject *o = opaque;
+    uint64_t end = pos + iov_size(iov, iovcnt);
+    int i;
+
+    if (end > o->vfu_mig_buf_size) {
+        o->vfu_mig_buf = g_realloc(o->vfu_mig_buf, end);
+    }
+
+    for (i = 0; i < iovcnt; i++) {
+        memcpy((o->vfu_mig_buf + o->vfu_mig_buf_size), iov[i].iov_base,
+               iov[i].iov_len);
+        o->vfu_mig_buf_size += iov[i].iov_len;
+        o->vfu_mig_buf_pending += iov[i].iov_len;
+    }
+
+    return iov_size(iov, iovcnt);
+}
+
+static int vfu_mig_buf_shutdown(void *opaque, bool rd, bool wr, Error

**errp)

hw/remote/vfio-user-obj.c +67 -0
+{
+    VfuObject *o = opaque;
+
+    o->vfu_mig_buf_size = 0;
+
+    g_free(o->vfu_mig_buf);
+
+    return 0;
+}
+
+static const QEMUFileOps vfu_mig_fops_save = {
+    .writev_buffer  = vfu_mig_buf_write,
+    .shut_down      = vfu_mig_buf_shutdown,
+};
+
+static const QEMUFileOps vfu_mig_fops_load = {
+    .get_buffer     = vfu_mig_buf_read,
+    .shut_down      = vfu_mig_buf_shutdown,
+};
+
+/**
+ * handlers for vfu_migration_callbacks_t
+ *
+ * The libvfio-user library accesses these handlers to drive the migration
+ * at the remote end, and also to transport the data stored in vfu_mig_buf
+ *
+ */
+static void vfu_mig_state_precopy(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    int ret;
+
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_save);
+    }
+
+    global_state_store();
+
+    ret = qemu_remote_savevm(o->vfu_mig_file);
+    if (ret) {
+        qemu_file_shutdown(o->vfu_mig_file);
+        return;
+    }
+
+    qemu_fflush(o->vfu_mig_file);
+
+    bdrv_inactivate_all();
+}
+
+static void vfu_mig_state_running(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    Error *local_err = NULL;
+    int ret;
+
+    ret = qemu_remote_loadvm(o->vfu_mig_file);
+    if (ret) {
+        error_setg(&error_abort, "vfu: failed to restore device state");
+        return;
+    }
+
+    bdrv_invalidate_cache_all(&local_err);
+    if (local_err) {
+        error_report_err(local_err);
+        return;
+    }
+

hw/remote/vfio-user-obj.c static void vfu_object_set_devid(Object *obj, const char *str, Error **errp) · line 254

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+    vm_start();
+}
+
+static int vfu_mig_transition(vfu_ctx_t *vfu_ctx, vfu_migr_state_t state)
+{
+    switch (state) {
+    case VFU_MIGR_STATE_RESUME:
+    case VFU_MIGR_STATE_STOP_AND_COPY:
+    case VFU_MIGR_STATE_STOP:
+        break;

Can you explain why we don't have to do anything in the above cases?

Quoted 114 lines
hw/remote/vfio-user-obj.c +95 -0
+    case VFU_MIGR_STATE_PRE_COPY:
+        vfu_mig_state_precopy(vfu_ctx);
+        break;
+    case VFU_MIGR_STATE_RUNNING:
+        if (!runstate_is_running()) {
+            vfu_mig_state_running(vfu_ctx);
+        }
+        break;
+    default:
+        warn_report("vfu: Unknown migration state %d", state);
+    }
+
+    return 0;
+}
+
+static uint64_t vfu_mig_get_pending_bytes(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    return o->vfu_mig_buf_pending;
+}
+
+static int vfu_mig_prepare_data(vfu_ctx_t *vfu_ctx, uint64_t *offset,
+                                uint64_t *size)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset) {
+        *offset = 0;
+    }
+
+    if (size) {
+        *size = o->vfu_mig_buf_size;
+    }
+
+    return 0;
+}
+
+static ssize_t vfu_mig_read_data(vfu_ctx_t *vfu_ctx, void *buf,
+                                 uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset > o->vfu_mig_buf_size) {
+        return -1;
+    }
+
+    if ((offset + size) > o->vfu_mig_buf_size) {
+        warn_report("vfu: buffer overflow - check pending_bytes");
+        size = o->vfu_mig_buf_size - offset;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + offset), size);
+
+    o->vfu_mig_buf_pending -= size;
+
+    return size;
+}
+
+static ssize_t vfu_mig_write_data(vfu_ctx_t *vfu_ctx, void *data,
+                                  uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint64_t end = offset + size;
+
+    if (end > o->vfu_mig_buf_size) {
+        o->vfu_mig_buf = g_realloc(o->vfu_mig_buf, end);
+        o->vfu_mig_buf_size = end;
+    }
+
+    memcpy((o->vfu_mig_buf + offset), data, size);
+
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_load);
+    }
+
+    return size;
+}
+
+static int vfu_mig_data_written(vfu_ctx_t *vfu_ctx, uint64_t count)
+{
+    return 0;
+}
+
+static const vfu_migration_callbacks_t vfu_mig_cbs = {
+    .version = VFU_MIGR_CALLBACKS_VERS,
+    .transition = &vfu_mig_transition,
+    .get_pending_bytes = &vfu_mig_get_pending_bytes,
+    .prepare_data = &vfu_mig_prepare_data,
+    .read_data = &vfu_mig_read_data,
+    .data_written = &vfu_mig_data_written,
+    .write_data = &vfu_mig_write_data,
+};
+
 static void *vfu_object_ctx_run(void *opaque)
 {
     VfuObject *o = opaque;
@@ -332,6 +581,7 @@ static void vfu_object_machine_done(Notifier
 *notifier, void *data)
 {
     VfuObject *o = container_of(notifier, VfuObject, machine_done);
     DeviceState *dev = NULL;
+    size_t migr_area_size;
     int ret;
 
     o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket, 0,
hw/remote/vfio-user-obj.c +2 -0
@@ -391,6 +641,35 @@ static void vfu_object_machine_done(Notifier
 *notifier, void *data)
         return;
     }
 
+    /*
+     * TODO: The 0x20000 number used below is a temporary. We are

working on

hw/remote/vfio-user-obj.c static void vfu_object_machine_done(Notifier *notifier, void *data) · line 654

Quoted 9 lines
hw/remote/vfio-user-obj.c +9 -0
+     *     a cleaner fix for this.
+     *
+     *     The libvfio-user library assumes that the remote knows the size of
+     *     the data to be migrated at boot time, but that is not the case with
+     *     VMSDs, as it can contain a variable-size buffer. 0x20000 is used
+     *     as a sufficiently large buffer to demonstrate migration, but that
+     *     cannot be used as a solution.
+     *
+     */

The size of the migration region dictates the amount of migration data that can be produced/consumed in one-go, it's not necessarily the total size of the migration data produced/consumed throughout the migration operation.

Quoted 119 lines
hw/remote/vfio-user-obj.c +18 -0
+    ret = vfu_setup_region(o->vfu_ctx, VFU_PCI_DEV_MIGR_REGION_IDX,
+                           0x20000, NULL,
+                           VFU_REGION_FLAG_RW, NULL, 0, -1, 0);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to register migration BAR %s- %s",
+                   o->devid, strerror(errno));
+        return;
+    }
+
+    migr_area_size = vfu_get_migr_register_area_size();
+    ret = vfu_setup_device_migration_callbacks(o->vfu_ctx, &vfu_mig_cbs,
+                                               migr_area_size);
+    if (ret < 0) {
+        error_setg(&error_abort, "vfu: Failed to setup migration %s- %s",
+                   o->devid, strerror(errno));
+        return;
+    }
+
     qemu_thread_create(&o->vfu_ctx_thread, "VFU ctx runner",

vfu_object_ctx_run,

                        o, QEMU_THREAD_JOINABLE);

}

hw/remote/vfio-user-obj.c +8 -0
@@ -412,6 +691,14 @@ static void vfu_object_init(Object *obj)
 
     o->machine_done.notify = vfu_object_machine_done;
     qemu_add_machine_init_done_notifier(&o->machine_done);
+
+    o->vfu_mig_file = NULL;
+
+    o->vfu_mig_buf = NULL;
+
+    o->vfu_mig_buf_size = 0;
+
+    o->vfu_mig_buf_pending = 0;
 }
 
 static void vfu_object_finalize(Object *obj)
migration/savevm.c +27 -0
index 72848b9..c2279af 100644
@@ -1603,6 +1603,33 @@ static int qemu_savevm_state(QEMUFile *f, Error
 **errp)
     return ret;
 }
 
+int qemu_remote_savevm(QEMUFile *f)
+{
+    SaveStateEntry *se;
+    int ret;
+
+    QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
+        if (!se->vmsd || !vmstate_save_needed(se->vmsd, se->opaque)) {
+            continue;
+        }
+
+        save_section_header(f, se, QEMU_VM_SECTION_FULL);
+
+        ret = vmstate_save(f, se, NULL);
+        if (ret) {
+            qemu_file_set_error(f, ret);
+            return ret;
+        }
+
+        save_section_footer(f, se);
+    }
+
+    qemu_put_byte(f, QEMU_VM_EOF);
+    qemu_fflush(f);
+
+    return 0;
+}
+
 void qemu_savevm_live_state(QEMUFile *f)
 {
     /* save QEMU_VM_SECTION_END section */
migration/savevm.c +36 -0
@@ -2443,6 +2470,42 @@ qemu_loadvm_section_start_full(QEMUFile *f,
 MigrationIncomingState *mis)
     return 0;
 }
 
+int qemu_remote_loadvm(QEMUFile *f)
+{
+    uint8_t section_type;
+    int ret = 0;
+
+    qemu_mutex_lock_iothread();
+
+    while (true) {
+        section_type = qemu_get_byte(f);
+
+        if (qemu_file_get_error(f)) {
+            ret = qemu_file_get_error(f);
+            break;
+        }
+
+        switch (section_type) {
+        case QEMU_VM_SECTION_FULL:
+            ret = qemu_loadvm_section_start_full(f, NULL);
+            if (ret < 0) {
+                break;
+            }
+            break;
+        case QEMU_VM_EOF:
+            goto out;
+        default:
+            ret = -EINVAL;
+            goto out;
+        }
+    }
+
+out:
+    qemu_mutex_unlock_iothread();
+
+    return ret;
+}
+
 static int
 qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState
Quoted 4 lines

*mis)
{
--
1.8.3.1

My background in implementing device migration with libvfio-user is for a specific device, it seems to me that you're using this functionality differently? Maybe that's why I'm getting confused. If this is the case, could you explain in more detail how you're using libvfio-user here?

#53 lore raw

RFC PATCH 11/11 vfio-user: acceptance test

In reply to Jagannathan Raman

Acceptance test for libvfio-user in QEMU

---
 MAINTAINERS                   |  1 +
 tests/acceptance/vfio-user.py | 94 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100644 tests/acceptance/vfio-user.py
MAINTAINERS +1 -0
index 46ab6b6..644bd35 100644
@@ -3381,6 +3381,7 @@ F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
 F: hw/remote/vfio-user-obj.c
+F: tests/acceptance/vfio-user.py
 
 EBPF:
 M: Jason Wang <[email protected]>
tests/acceptance/vfio-user.py new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#54 lore raw

RE: [PATCH RFC server 11/11] vfio-user: acceptance test

Quoted 71 lines

-----Original Message-----
From: Jagannathan Raman <[email protected]>
Sent: 19 July 2021 21:00
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; John Levon <[email protected]>; [email protected]; Thanos Makatos <[email protected]>; Swapnil Ingle <[email protected]>; [email protected]
Subject: [PATCH RFC server 11/11] vfio-user: acceptance test

Acceptance test for libvfio-user in QEMU

---
 MAINTAINERS                   |  1 +
 tests/acceptance/vfio-user.py | 94
+++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100644 tests/acceptance/vfio-user.py
MAINTAINERS +1 -0
index 46ab6b6..644bd35 100644
@@ -3381,6 +3381,7 @@ F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
 F: hw/remote/vfio-user-obj.c
+F: tests/acceptance/vfio-user.py
 
 EBPF:
 M: Jason Wang <[email protected]>
tests/acceptance/vfio-user.py new file +28 -0
new file mode 100644
index 0000000..ef318d9
@@ -0,0 +1,94 @@
+# vfio-user protocol sanity test
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+
+import os
+import socket
+import uuid
+
+from avocado_qemu import Test
+from avocado_qemu import wait_for_console_pattern
+from avocado_qemu import exec_command
+from avocado_qemu import exec_command_and_wait_for_pattern
+
+class VfioUser(Test):
+    """
+    :avocado: tags=vfiouser
+    """
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+
+    def do_test(self, kernel_url, initrd_url, kernel_command_line,
+                machine_type):
+        """Main test method"""
+        self.require_accelerator('kvm')
+
+        kernel_path = self.fetch_asset(kernel_url)
+        initrd_path = self.fetch_asset(initrd_url)

tests/acceptance/vfio-user.py line 38

Quoted 10 lines
tests/acceptance/vfio-user.py new file +10 -0
+
+        socket = os.path.join('/tmp', str(uuid.uuid4()))
+        if os.path.exists(socket):
+            os.remove(socket)
+
+        # Create remote process
+        remote_vm = self.get_vm()
+        remote_vm.add_args('-machine', 'x-remote')
+        remote_vm.add_args('-nodefaults')
+        remote_vm.add_args('-device', 'lsi53c895a,id=lsi1')

IIUC the LSI controller will now be a migratable device and migration will be handled by vfu_mig_transition() introduced in your "vfio-user: register handlers to facilitate migration" patch. In vfu_mig_transition(), you don’t copy migration data in the VFU_MIGR_STATE_STOP_AND_COPY case but only in VFU_MIGR_STATE_PRE_COPY, however I believe that in VFIO it's possible to jump from the running state straight to the stop-and-copy state. Are you relying on QEMU not doing this?

Quoted 76 lines
tests/acceptance/vfio-user.py new file +32 -0
+        remote_vm.add_args('-object', 'vfio-user,id=vfioobj1,'
+                           'devid=lsi1,socket='+socket)
+        remote_vm.launch()
+
+        # Create proxy process
+        self.vm.set_console()
+        self.vm.add_args('-machine', machine_type)
+        self.vm.add_args('-accel', 'kvm')
+        self.vm.add_args('-cpu', 'host')
+        self.vm.add_args('-object',
+                         'memory-backend-memfd,id=sysmem-file,size=2G')
+        self.vm.add_args('--numa', 'node,memdev=sysmem-file')
+        self.vm.add_args('-m', '2048')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-initrd', initrd_path,
+                         '-append', kernel_command_line)
+        self.vm.add_args('-device',
+                         'vfio-user-pci,'
+                         'socket='+socket)
+        self.vm.launch()
+        wait_for_console_pattern(self, 'as init process',
+                                 'Kernel panic - not syncing')
+        exec_command(self, 'mount -t sysfs sysfs /sys')
+        exec_command_and_wait_for_pattern(self,
+                                          'cat /sys/bus/pci/devices/*/uevent',
+                                          'PCI_ID=1000:0012')
+
+    def test_multiprocess_x86_64(self):
+        """
+        :avocado: tags=arch:x86_64
+        """
+        kernel_url = ('https://urldefense.proofpoint.com/v2/url?u=https-

3A__archives.fedoraproject.org_pub_archive_fedora&d=DwIBAg&c=s883G pUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6ogtti46atk736SI4vgsJiUKIyDE&m= 4nAZXeA9xd82TON6H7CXF5LVa0jKBAJkyu0Y- curSd4&s=hP6IktdmIVlw3gMuZlWRkPvFq9OzjUji6sb_28sapwk&e= '

tests/acceptance/vfio-user.py new file +3 -0
+                      '/linux/releases/31/Everything/x86_64/os/images'
+                      '/pxeboot/vmlinuz')
+        initrd_url = ('https://urldefense.proofpoint.com/v2/url?u=https-

3A__archives.fedoraproject.org_pub_archive_fedora&d=DwIBAg&c=s883G pUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6ogtti46atk736SI4vgsJiUKIyDE&m= 4nAZXeA9xd82TON6H7CXF5LVa0jKBAJkyu0Y- curSd4&s=hP6IktdmIVlw3gMuZlWRkPvFq9OzjUji6sb_28sapwk&e= '

tests/acceptance/vfio-user.py new file +6 -0
+                      '/linux/releases/31/Everything/x86_64/os/images'
+                      '/pxeboot/initrd.img')
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyS0 rdinit=/bin/bash')
+        machine_type = 'pc'
+        self.do_test(kernel_url, initrd_url, kernel_command_line,

machine_type)

tests/acceptance/vfio-user.py new file +6 -0
+
+    def test_multiprocess_aarch64(self):
+        """
+        :avocado: tags=arch:aarch64
+        """
+        kernel_url = ('https://urldefense.proofpoint.com/v2/url?u=https-

3A__archives.fedoraproject.org_pub_archive_fedora&d=DwIBAg&c=s883G pUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6ogtti46atk736SI4vgsJiUKIyDE&m= 4nAZXeA9xd82TON6H7CXF5LVa0jKBAJkyu0Y- curSd4&s=hP6IktdmIVlw3gMuZlWRkPvFq9OzjUji6sb_28sapwk&e= '

tests/acceptance/vfio-user.py new file +3 -0
+                      '/linux/releases/31/Everything/aarch64/os/images'
+                      '/pxeboot/vmlinuz')
+        initrd_url = ('https://urldefense.proofpoint.com/v2/url?u=https-

3A__archives.fedoraproject.org_pub_archive_fedora&d=DwIBAg&c=s883G pUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6ogtti46atk736SI4vgsJiUKIyDE&m= 4nAZXeA9xd82TON6H7CXF5LVa0jKBAJkyu0Y- curSd4&s=hP6IktdmIVlw3gMuZlWRkPvFq9OzjUji6sb_28sapwk&e= '

tests/acceptance/vfio-user.py new file +6 -0
+                      '/linux/releases/31/Everything/aarch64/os/images'
+                      '/pxeboot/initrd.img')
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'rdinit=/bin/bash console=ttyAMA0')
+        machine_type = 'virt,gic-version=3'
+        self.do_test(kernel_url, initrd_url, kernel_command_line,

machine_type)
--
1.8.3.1