Skip to contentSkip to conversation

vhost-user block device backend implementation

PATCH Superseded Coiby Xu 5 participants 21 messages Jun 4, 2020 – Jun 16, 2020 Block layer

v8 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Coiby Xu

author
#0 lore raw

v8 cover vhost-user block device backend implementation

v8
- re-try connecting to socket server to fix asan error
- fix license naming issue

v7
- fix docker-test-debug@fedora errors by freeing malloced memory

v6
- add missing license header and include guard
- vhost-user server only serve one client one time
- fix a bug in custom vu_message_read
- using qemu-storage-daemon to start vhost-user-blk-server
- a bug fix to pass docker-test-clang@ubuntu

v5:
* re-use vu_kick_cb in libvhost-user
* keeping processing VhostUserMsg in the same coroutine until there is detachment/attachment of AIOContext
* Spawn separate coroutine for each VuVirtqElement
* Other changes including relocating vhost-user-blk-server.c, coding style etc.

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

v3:
* separate generic vhost-user-server code from vhost-user-blk-server code
* re-write vu_message_read and kick hander function as coroutines to directly call blk_co_preadv, blk_co_pwritev, etc.
* add aio_context notifier functions to support multi-threading model
* other fixes regarding coding style, warning report, etc.

v2:
* Only enable this feature for Linux because eventfd is a Linux-specific feature

This patch series is an implementation of vhost-user block device backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be started using object_add,

(qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend. For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (4):
Allow vu_message_read to be replaced generic vhost user server
vhost-user block device backend server new qTest case to test the vhost-user-blk-server

 block/Makefile.objs                        |   1 +
 block/export/vhost-user-blk-server.c       | 716 ++++++++++++++++++++
 block/export/vhost-user-blk-server.h       |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c      |  11 +-
 contrib/libvhost-user/libvhost-user.h      |  21 +
 softmmu/vl.c                               |   4 +
 tests/Makefile.include                     |   3 +-
 tests/qtest/Makefile.include               |   2 +
 tests/qtest/libqos/vhost-user-blk.c        | 130 ++++
 tests/qtest/libqos/vhost-user-blk.h        |  44 ++
 tests/qtest/libqtest.c                     |  54 +-
 tests/qtest/libqtest.h                     |  38 ++
 tests/qtest/vhost-user-blk-test.c          | 737 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |   2 +
 tools/virtiofsd/fuse_virtio.c              |   4 +-
 util/Makefile.objs                         |   1 +
 util/vhost-user-server.c                   | 406 ++++++++++++
 util/vhost-user-server.h                   |  59 ++
 19 files changed, 2229 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.26.2

author
#1 lore raw

PATCH v8 1/4 Allow vu_message_read to be replaced

Allow vu_message_read to be replaced by one which will make use of the QIOChannel functions. Thus reading vhost-user message won't stall the guest.

---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c      | 11 ++++++-----
 contrib/libvhost-user/libvhost-user.h      | 21 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |  2 ++
 tools/virtiofsd/fuse_virtio.c              |  4 ++--
 5 files changed, 32 insertions(+), 8 deletions(-)
contrib/libvhost-user/libvhost-user-glib.c +1 -1
index 53f1ca4cdd..0df2ec9271 100644
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
     g_assert(dev);
     g_assert(iface);
 
-    if (!vu_init(&dev->parent, max_queues, socket, panic, set_watch,
+    if (!vu_init(&dev->parent, max_queues, socket, panic, NULL, set_watch,
                  remove_watch, iface)) {
         return false;
     }
contrib/libvhost-user/libvhost-user.c +6 -5
index 3bca996c62..0c7368baa2 100644
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -412,7 +410,7 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg *vmsg)
         goto out;
     }
 
-    if (!vu_message_read(dev, dev->slave_fd, &msg_reply)) {
+    if (!dev->read_msg(dev, dev->slave_fd, &msg_reply)) {
         goto out;
     }
 
@@ -647,7 +645,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg *vmsg)
     /* Wait for QEMU to confirm that it's registered the handler for the
      * faults.
      */
-    if (!vu_message_read(dev, dev->sock, vmsg) ||
+    if (!dev->read_msg(dev, dev->sock, vmsg) ||
         vmsg->size != sizeof(vmsg->payload.u64) ||
         vmsg->payload.u64 != 0) {
         vu_panic(dev, "failed to receive valid ack for postcopy set-mem-table");
@@ -1653,7 +1651,7 @@ vu_dispatch(VuDev *dev)
     int reply_requested;
     bool need_reply, success = false;
 
-    if (!vu_message_read(dev, dev->sock, &vmsg)) {
+    if (!dev->read_msg(dev, dev->sock, &vmsg)) {
         goto end;
     }
 
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }
@@ -1751,6 +1750,7 @@ vu_init(VuDev *dev,
         uint16_t max_queues,
         int socket,
         vu_panic_cb panic,
+        vu_read_msg_cb read_msg,
         vu_set_watch_cb set_watch,
         vu_remove_watch_cb remove_watch,
         const VuDevIface *iface)
@@ -1768,6 +1768,7 @@ vu_init(VuDev *dev,
 
     dev->sock = socket;
     dev->panic = panic;
+    dev->read_msg = read_msg ? read_msg : vu_message_read;
     dev->set_watch = set_watch;
     dev->remove_watch = remove_watch;
     dev->iface = iface;
contrib/libvhost-user/libvhost-user.h +21 -0
index f30394fab6..d756da8548 100644
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
     VHOST_SET_CONFIG_TYPE_MASTER = 0,
     VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -205,6 +207,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
                                   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -373,6 +376,23 @@ struct VuDev {
     bool broken;
     uint16_t max_queues;
 
+    /* @read_msg: custom method to read vhost-user message
+     *
+     * Read data from vhost_user socket fd and fill up
+     * the passed VhostUserMsg *vmsg struct.
+     *
+     * If reading fails, it should close the received set of file
+     * descriptors as socket message's auxiliary data.
+     *
+     * For the details, please refer to vu_message_read in libvhost-user.c
+     * which will be used by default if not custom method is provided when
+     * calling vu_init
+     *
+     * Returns: true if vhost-user message successfully received,
+     *          otherwise return false.
+     *
+     */
+    vu_read_msg_cb read_msg;
     /* @set_watch: add or update the given fd to the watch set,
      * call cb when condition is met */
     vu_set_watch_cb set_watch;
@@ -416,6 +436,7 @@ bool vu_init(VuDev *dev,
              uint16_t max_queues,
              int socket,
              vu_panic_cb panic,
+             vu_read_msg_cb read_msg,
              vu_set_watch_cb set_watch,
              vu_remove_watch_cb remove_watch,
              const VuDevIface *iface);
tests/vhost-user-bridge.c +2 -0
index 6c3d490611..bd43607a4d 100644
@@ -520,6 +520,7 @@ vubr_accept_cb(int sock, void *ctx)
                  VHOST_USER_BRIDGE_MAX_QUEUES,
                  conn_fd,
                  vubr_panic,
+                 NULL,
                  vubr_set_watch,
                  vubr_remove_watch,
                  &vuiface)) {
@@ -573,6 +574,7 @@ vubr_new(const char *path, bool client)
                      VHOST_USER_BRIDGE_MAX_QUEUES,
                      dev->sock,
                      vubr_panic,
+                     NULL,
                      vubr_set_watch,
                      vubr_remove_watch,
                      &vuiface)) {
tools/virtiofsd/fuse_virtio.c +2 -2
index 3b6d16a041..666945c897 100644
@@ -980,8 +980,8 @@ int virtio_session_mount(struct fuse_session *se)
     se->vu_socketfd = data_sock;
     se->virtio_dev->se = se;
     pthread_rwlock_init(&se->virtio_dev->vu_dispatch_rwlock, NULL);
-    vu_init(&se->virtio_dev->dev, 2, se->vu_socketfd, fv_panic, fv_set_watch,
-            fv_remove_watch, &fv_iface);
+    vu_init(&se->virtio_dev->dev, 2, se->vu_socketfd, fv_panic, NULL,
+            fv_set_watch, fv_remove_watch, &fv_iface);
 
     return 0;
 }
Signature · 1 line
2.26.2
Reviewed-by on v8 1/4
#2 lore raw

Re: [PATCH v8 1/4] Allow vu_message_read to be replaced

Quoted 4 lines · Coiby Xu

Allow vu_message_read to be replaced by one which will make use of the QIOChannel functions. Thus reading vhost-user message won't stall the guest.

Quoted 8 lines
---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c      | 11 ++++++-----
 contrib/libvhost-user/libvhost-user.h      | 21 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |  2 ++
 tools/virtiofsd/fuse_virtio.c              |  4 ++--
 5 files changed, 32 insertions(+), 8 deletions(-)
Reviewed-by on v8 1/4
#3 lore raw

Re: [PATCH v8 1/4] Allow vu_message_read to be replaced

In reply to Coiby Xu
Quoted 6 lines · Coiby Xu

Allow vu_message_read to be replaced by one which will make use of the QIOChannel functions. Thus reading vhost-user message won't stall the guest.

---

Quoted 186 lines
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c      | 11 ++++++-----
 contrib/libvhost-user/libvhost-user.h      | 21 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |  2 ++
 tools/virtiofsd/fuse_virtio.c              |  4 ++--
 5 files changed, 32 insertions(+), 8 deletions(-)
contrib/libvhost-user/libvhost-user-glib.c +1 -1
index 53f1ca4cdd..0df2ec9271 100644
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
     g_assert(dev);
     g_assert(iface);
 
-    if (!vu_init(&dev->parent, max_queues, socket, panic, set_watch,
+    if (!vu_init(&dev->parent, max_queues, socket, panic, NULL, set_watch,
                  remove_watch, iface)) {
         return false;
     }
contrib/libvhost-user/libvhost-user.c +2 -4
index 3bca996c62..0c7368baa2 100644
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -412,7 +410,7 @@ vu_process_message_reply(VuDev *dev, const
 VhostUserMsg *vmsg)
         goto out;
     }
 
-    if (!vu_message_read(dev, dev->slave_fd, &msg_reply)) {
+    if (!dev->read_msg(dev, dev->slave_fd, &msg_reply)) {
         goto out;
     }
@@ -647,7 +645,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev,
 VhostUserMsg *vmsg)
     /* Wait for QEMU to confirm that it's registered the handler for the
      * faults.
      */
-    if (!vu_message_read(dev, dev->sock, vmsg) ||
+    if (!dev->read_msg(dev, dev->sock, vmsg) ||
         vmsg->size != sizeof(vmsg->payload.u64) ||
         vmsg->payload.u64 != 0) {

vu_panic(dev, "failed to receive valid ack for postcopy
set-mem-table");

contrib/libvhost-user/libvhost-user.c +4 -1
@@ -1653,7 +1651,7 @@ vu_dispatch(VuDev *dev)
     int reply_requested;
     bool need_reply, success = false;
 
-    if (!vu_message_read(dev, dev->sock, &vmsg)) {
+    if (!dev->read_msg(dev, dev->sock, &vmsg)) {
         goto end;
     }
 
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }
@@ -1751,6 +1750,7 @@ vu_init(VuDev *dev,
         uint16_t max_queues,
         int socket,
         vu_panic_cb panic,
+        vu_read_msg_cb read_msg,
         vu_set_watch_cb set_watch,
         vu_remove_watch_cb remove_watch,
         const VuDevIface *iface)
@@ -1768,6 +1768,7 @@ vu_init(VuDev *dev,
 
     dev->sock = socket;
     dev->panic = panic;
+    dev->read_msg = read_msg ? read_msg : vu_message_read;
     dev->set_watch = set_watch;
     dev->remove_watch = remove_watch;
     dev->iface = iface;
contrib/libvhost-user/libvhost-user.h +3 -0
index f30394fab6..d756da8548 100644
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
     VHOST_SET_CONFIG_TYPE_MASTER = 0,
     VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -205,6 +207,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
                                   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool
 started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);

typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t
len);

contrib/libvhost-user/libvhost-user.h +18 -0
@@ -373,6 +376,23 @@ struct VuDev {
     bool broken;
     uint16_t max_queues;
 
+    /* @read_msg: custom method to read vhost-user message
+     *
+     * Read data from vhost_user socket fd and fill up
+     * the passed VhostUserMsg *vmsg struct.
+     *
+     * If reading fails, it should close the received set of file
+     * descriptors as socket message's auxiliary data.
+     *
+     * For the details, please refer to vu_message_read in libvhost-user.c
+     * which will be used by default if not custom method is provided when
+     * calling vu_init
+     *
+     * Returns: true if vhost-user message successfully received,
+     *          otherwise return false.
+     *
+     */
+    vu_read_msg_cb read_msg;
     /* @set_watch: add or update the given fd to the watch set,
      * call cb when condition is met */
     vu_set_watch_cb set_watch;
@@ -416,6 +436,7 @@ bool vu_init(VuDev *dev,
              uint16_t max_queues,
              int socket,
              vu_panic_cb panic,
+             vu_read_msg_cb read_msg,
              vu_set_watch_cb set_watch,
              vu_remove_watch_cb remove_watch,
              const VuDevIface *iface);
tests/vhost-user-bridge.c +2 -0
index 6c3d490611..bd43607a4d 100644
@@ -520,6 +520,7 @@ vubr_accept_cb(int sock, void *ctx)
                  VHOST_USER_BRIDGE_MAX_QUEUES,
                  conn_fd,
                  vubr_panic,
+                 NULL,
                  vubr_set_watch,
                  vubr_remove_watch,
                  &vuiface)) {
@@ -573,6 +574,7 @@ vubr_new(const char *path, bool client)
                      VHOST_USER_BRIDGE_MAX_QUEUES,
                      dev->sock,
                      vubr_panic,
+                     NULL,
                      vubr_set_watch,
                      vubr_remove_watch,
                      &vuiface)) {
tools/virtiofsd/fuse_virtio.c +2 -2
index 3b6d16a041..666945c897 100644
@@ -980,8 +980,8 @@ int virtio_session_mount(struct fuse_session *se)
     se->vu_socketfd = data_sock;
     se->virtio_dev->se = se;
     pthread_rwlock_init(&se->virtio_dev->vu_dispatch_rwlock, NULL);
-    vu_init(&se->virtio_dev->dev, 2, se->vu_socketfd, fv_panic,
 fv_set_watch,
-            fv_remove_watch, &fv_iface);
+    vu_init(&se->virtio_dev->dev, 2, se->vu_socketfd, fv_panic, NULL,
+            fv_set_watch, fv_remove_watch, &fv_iface);
 
     return 0;

}
--
2.26.2

Signature · 1 line
Marc-André Lureau
author
#4 lore raw

PATCH v8 2/4 generic vhost user server

In reply to Coiby Xu

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

---
 util/Makefile.objs       |   1 +
 util/vhost-user-server.c | 406 +++++++++++++++++++++++++++++++++++++++
 util/vhost-user-server.h |  59 ++++++
 3 files changed, 466 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
util/Makefile.objs +1 -0
index fe339c2636..f54a6c80ec 100644
@@ -40,6 +40,7 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
util/vhost-user-server.c new file +406 -0

406 lines not on this page · raw message

util/vhost-user-server.h new file +59 -0

59 lines not on this page · raw message

Signature · 1 line
2.26.2
#5 lore raw

Re: [PATCH v8 2/4] generic vhost user server

Quoted 53 lines · Coiby Xu
util/vhost-user-server.c new file +53 -0
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    struct iovec iov = {
+        .iov_base = (char *)vmsg,
+        .iov_len = VHOST_USER_HDR_SIZE,
+    };
+    int rc, read_bytes = 0;
+    Error *local_err = NULL;
+    /*
+     * Store fds/nfds returned from qio_channel_readv_full into
+     * temporary variables.
+     *
+     * VhostUserMsg is a packed structure, gcc will complain about passing
+     * pointer to a packed structure member if we pass &VhostUserMsg.fd_num
+     * and &VhostUserMsg.fds directly when calling qio_channel_readv_full,
+     * thus two temporary variables nfds and fds are used here.
+     */
+    size_t nfds = 0, nfds_t = 0;
+    int *fds = NULL, *fds_t = NULL;
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    QIOChannel *ioc = NULL;
+
+    if (conn_fd == server->sioc->fd) {
+        ioc = server->ioc;
+    } else {
+        /* Slave communication will also use this function to read msg */
+        ioc = slave_io_channel(server, conn_fd, &local_err);
+    }
+
+    if (!ioc) {
+        error_report_err(local_err);
+        goto fail;
+    }
+
+    assert(qemu_in_coroutine());
+    do {
+        /*
+         * qio_channel_readv_full may have short reads, keeping calling it
+         * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+         */
+        rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &local_err);
+        if (rc < 0) {
+            if (rc == QIO_CHANNEL_ERR_BLOCK) {
+                qio_channel_yield(ioc, G_IO_IN);
+                continue;
+            } else {
+                error_report_err(local_err);
+                return false;
+            }
+        }
+        read_bytes += rc;
+        if (nfds_t > 0) {

util/vhost-user-server.c line 148

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+            fds = g_renew(int, fds, nfds + nfds_t);
+            memcpy(fds + nfds, fds_t, nfds_t *sizeof(int));
+            nfds += nfds_t;
+            if (nfds > VHOST_MEMORY_MAX_NREGIONS) {
+                error_report("A maximum of %d fds are allowed, "
+                             "however got %lu fds now",
+                             VHOST_MEMORY_MAX_NREGIONS, nfds);
+                goto fail;
+            }
+            g_free(fds_t);

I'm not sure why the temporary fds[] array is necessary. Copying the fds directly into vmsg->fds would be simpler:

  if (nfds + nfds_t > G_N_ELEMENTS(vmsg->fds)) {
      error_report("A maximum of %d fds are allowed, "
                   "however got %lu fds now",
                   VHOST_MEMORY_MAX_NREGIONS, nfds);
      goto fail;
  }
  memcpy(vmsg->fds + nfds, fds_t, nfds_t * sizeof(vds->fds[0]));
  nfds += nfds_t;

Did I misunderstand how this works?

Quoted 142 lines
util/vhost-user-server.c new file +142 -0
+        }
+        if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+            break;
+        }
+        iov.iov_base = (char *)vmsg + read_bytes;
+        iov.iov_len = VHOST_USER_HDR_SIZE - read_bytes;
+    } while (true);
+
+    vmsg->fd_num = nfds;
+    if (nfds > 0) {
+        memcpy(vmsg->fds, fds, nfds * sizeof(int));
+    }
+    g_free(fds);
+    /* qio_channel_readv_full will make socket fds blocking, unblock them */
+    vmsg_unblock_fds(vmsg);
+    if (vmsg->size > sizeof(vmsg->payload)) {
+        error_report("Error: too big message request: %d, "
+                     "size: vmsg->size: %u, "
+                     "while sizeof(vmsg->payload) = %zu",
+                     vmsg->request, vmsg->size, sizeof(vmsg->payload));
+        goto fail;
+    }
+
+    struct iovec iov_payload = {
+        .iov_base = (char *)&vmsg->payload,
+        .iov_len = vmsg->size,
+    };
+    if (vmsg->size) {
+        rc = qio_channel_readv_all_eof(ioc, &iov_payload, 1, &local_err);
+        if (rc == -1) {
+            error_report_err(local_err);
+            goto fail;
+        }
+    }
+
+    return true;
+
+fail:
+    vmsg_close_fds(vmsg);
+
+    return false;
+}
+
+
+static void vu_client_start(VuServer *server);
+static coroutine_fn void vu_client_trip(void *opaque)
+{
+    VuServer *server = opaque;
+
+    while (!server->aio_context_changed && server->sioc) {
+        vu_dispatch(&server->vu_dev);
+    }
+
+    if (server->aio_context_changed && server->sioc) {
+        server->aio_context_changed = false;
+        vu_client_start(server);
+    }
+}
+
+static void vu_client_start(VuServer *server)
+{
+    server->co_trip = qemu_coroutine_create(vu_client_trip, server);
+    aio_co_enter(server->ctx, server->co_trip);
+}
+
+/*
+ * a wrapper for vu_kick_cb
+ *
+ * since aio_dispatch can only pass one user data pointer to the
+ * callback function, pack VuDev and pvt into a struct. Then unpack it
+ * and pass them to vu_kick_cb
+ */
+static void kick_handler(void *opaque)
+{
+    KickInfo *kick_info = opaque;
+    kick_info->cb(kick_info->vu_dev, 0, (void *) kick_info->index);
+}
+
+
+static void
+set_watch(VuDev *vu_dev, int fd, int vu_evt,
+          vu_watch_cb cb, void *pvt)
+{
+
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+    long index = (intptr_t) pvt;
+    g_assert(cb);
+    KickInfo *kick_info = &server->kick_info[index];
+    if (!kick_info->cb) {
+        kick_info->fd = fd;
+        kick_info->cb = cb;
+        qemu_set_nonblock(fd);
+        aio_set_fd_handler(server->ioc->ctx, fd, false, kick_handler,
+                           NULL, NULL, kick_info);
+        kick_info->vu_dev = vu_dev;
+    }
+}
+
+
+static void remove_watch(VuDev *vu_dev, int fd)
+{
+    VuServer *server;
+    int i;
+    int index = -1;
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+
+    server = container_of(vu_dev, VuServer, vu_dev);
+    for (i = 0; i < vu_dev->max_queues; i++) {
+        if (server->kick_info[i].fd == fd) {
+            index = i;
+            break;
+        }
+    }
+
+    if (index == -1) {
+        return;
+    }
+    server->kick_info[i].cb = NULL;
+    aio_set_fd_handler(server->ioc->ctx, fd, false, NULL, NULL, NULL, NULL);
+}
+
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+                      gpointer opaque)
+{
+    VuServer *server = opaque;
+
+    if (server->sioc) {
+        warn_report("Only one vhost-user client is allowed to "
+                    "connect the server one time");
+        return;
+    }
+
+    if (!vu_init(&server->vu_dev, server->max_queues, sioc->fd, panic_cb,
+                 vu_message_read, set_watch, remove_watch, server->vu_iface)) {
+        error_report("Failed to initialized libvhost-user");
+        return;
+    }
+

util/vhost-user-server.c line 300

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+    /*
+     * Unset the callback function for network listener to make another
+     * vhost-user client keeping waiting until this client disconnects
+     */
+    qio_net_listener_set_client_func(server->listener,
+                                     NULL,
+                                     NULL,
+                                     NULL);
+    server->sioc = sioc;
+    server->kick_info = g_new0(KickInfo, server->max_queues);

Where is kick_info freed?

Review

Quoted 2 lines
(quoted diff) +2 -0
+    /*
+     * Increase the object reference, so cioc will not freed by

s/cioc/sioc/

util/vhost-user-server.c line 309

Quoted 7 lines
util/vhost-user-server.c new file +7 -0
+     * qio_net_listener_channel_func which will call object_unref(OBJECT(sioc))
+     */
+    object_ref(OBJECT(server->sioc));
+    qio_channel_set_name(QIO_CHANNEL(sioc), "vhost-user client");
+    server->ioc = QIO_CHANNEL(sioc);
+    object_ref(OBJECT(server->ioc));
+    object_ref(OBJECT(sioc));

Why are there two object_refs for sioc and where is unref called?

Quoted 5 lines
util/vhost-user-server.c new file +5 -0
+    qio_channel_attach_aio_context(server->ioc, server->ctx);
+    qio_channel_set_blocking(QIO_CHANNEL(server->sioc), false, NULL);
+    vu_client_start(server);
+}
+

util/vhost-user-server.c line 324

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+
+void vhost_user_server_stop(VuServer *server)
+{
+    if (!server) {
+        return;
+    }
+
+    if (server->sioc) {
+        close_client(server);
+        object_unref(OBJECT(server->sioc));

This call is object_unref(NULL) since close_client() does server->sioc = NULL.

Quoted 66 lines
util/vhost-user-server.c new file +66 -0
+    }
+
+    if (server->listener) {
+        qio_net_listener_disconnect(server->listener);
+        object_unref(OBJECT(server->listener));
+    }
+}
+
+static void detach_context(VuServer *server)
+{
+    int i;
+    AioContext *ctx = server->ioc->ctx;
+    qio_channel_detach_aio_context(server->ioc);
+    for (i = 0; i < server->vu_dev.max_queues; i++) {
+        if (server->kick_info[i].cb) {
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, false, NULL,
+                               NULL, NULL, NULL);
+        }
+    }
+}
+
+static void attach_context(VuServer *server, AioContext *ctx)
+{
+    int i;
+    qio_channel_attach_aio_context(server->ioc, ctx);
+    server->aio_context_changed = true;
+    if (server->co_trip) {
+        aio_co_schedule(ctx, server->co_trip);
+    }
+    for (i = 0; i < server->vu_dev.max_queues; i++) {
+        if (server->kick_info[i].cb) {
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, false,
+                               kick_handler, NULL, NULL,
+                               &server->kick_info[i]);
+        }
+    }
+}
+
+void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server)
+{
+    server->ctx = ctx ? ctx : qemu_get_aio_context();
+    if (!server->sioc) {
+        return;
+    }
+    if (ctx) {
+        attach_context(server, ctx);
+    } else {
+        detach_context(server);
+    }
+}
+
+
+bool vhost_user_server_start(uint16_t max_queues,
+                             SocketAddress *socket_addr,
+                             AioContext *ctx,
+                             VuServer *server,
+                             void *device_panic_notifier,
+                             const VuDevIface *vu_iface,
+                             Error **errp)
+{
+    server->listener = qio_net_listener_new();
+    if (qio_net_listener_open_sync(server->listener, socket_addr, 1,
+                                   errp) < 0) {
+        goto error;
+    }
+

util/vhost-user-server.c line 400

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+    qio_net_listener_set_name(server->listener, "vhost-user-backend-listener");
+
+    server->vu_iface = vu_iface;
+    server->max_queues = max_queues;
+    server->ctx = ctx;
+    server->device_panic_notifier = device_panic_notifier;
+    qio_net_listener_set_client_func(server->listener,
+                                     vu_accept,
+                                     server,
+                                     NULL);

The qio_net_listener_set_client_func() call uses the default GMainContext but we have an AioContext *ctx argument. This is surprising. I would expect the socket to be handled in the AioContext.

Can you clarify how this should work?

util/vhost-user-server.c line 404

Quoted 4 lines
util/vhost-user-server.c new file +4 -0
+
+    return true;
+error:
+    g_free(server);

It's surprising that this function frees the server argument when an error occurs. vhost_user_server_stop() does not free server. I suggest letting the caller free server since they own the object.

Quoted 49 lines
(quoted diff) +2 -0
+    return false;
+}
util/vhost-user-server.h new file +41 -0
new file mode 100644
index 0000000000..4315556b66
@@ -0,0 +1,59 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+
+#ifndef VHOST_USER_SERVER_H
+#define VHOST_USER_SERVER_H
+
+#include "contrib/libvhost-user/libvhost-user.h"
+#include "io/channel-socket.h"
+#include "io/channel-file.h"
+#include "io/net-listener.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "standard-headers/linux/virtio_blk.h"
+
+typedef struct KickInfo {
+    VuDev *vu_dev;
+    int fd; /*kick fd*/
+    long index; /*queue index*/
+    vu_watch_cb cb;
+} KickInfo;
+
+typedef struct VuServer {
+    QIONetListener *listener;
+    AioContext *ctx;
+    void (*device_panic_notifier)(struct VuServer *server) ;
+    int max_queues;
+    const VuDevIface *vu_iface;
+    VuDev vu_dev;
+    QIOChannel *ioc; /* The I/O channel with the client */
+    QIOChannelSocket *sioc; /* The underlying data channel with the client */
+    /* IOChannel for fd provided via VHOST_USER_SET_SLAVE_REQ_FD */
+    QIOChannel *ioc_slave;
+    QIOChannelSocket *sioc_slave;
+    Coroutine *co_trip; /* coroutine for processing VhostUserMsg */
+    KickInfo *kick_info; /* an array with the length of the queue number */

util/vhost-user-server.h line 51

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    /* restart coroutine co_trip if AIOContext is changed */
+    bool aio_context_changed;
+} VuServer;
+
+
+bool vhost_user_server_start(uint16_t max_queues,
+                             SocketAddress *unix_socket,
+                             AioContext *ctx,
+                             VuServer *server,
+                             void *device_panic_notifier,

Please declare the function pointer type:

typedef void DevicePanicNotifierFn(struct VuServer *server);

Then the argument list can use DevicePanicNotifierFn *device_panic_notifier instead of void *.

util/vhost-user-server.h line 57

Quoted 6 lines
util/vhost-user-server.h new file +6 -0
+                             const VuDevIface *vu_iface,
+                             Error **errp);
+
+void vhost_user_server_stop(VuServer *server);
+
+void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server);

If you send another revision, please make VuServer *server the first argument of vhost_user_server_start() and vhost_user_server_set_aio_context(). Functions usually have the object they act on as the first argument.

author
#6 lore raw

Re: [PATCH v8 2/4] generic vhost user server

Quoted 77 lines · Stefan Hajnoczi
Quoted 53 lines · Coiby Xu
util/vhost-user-server.c new file +53 -0
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    struct iovec iov = {
+        .iov_base = (char *)vmsg,
+        .iov_len = VHOST_USER_HDR_SIZE,
+    };
+    int rc, read_bytes = 0;
+    Error *local_err = NULL;
+    /*
+     * Store fds/nfds returned from qio_channel_readv_full into
+     * temporary variables.
+     *
+     * VhostUserMsg is a packed structure, gcc will complain about passing
+     * pointer to a packed structure member if we pass &VhostUserMsg.fd_num
+     * and &VhostUserMsg.fds directly when calling qio_channel_readv_full,
+     * thus two temporary variables nfds and fds are used here.
+     */
+    size_t nfds = 0, nfds_t = 0;
+    int *fds = NULL, *fds_t = NULL;
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    QIOChannel *ioc = NULL;
+
+    if (conn_fd == server->sioc->fd) {
+        ioc = server->ioc;
+    } else {
+        /* Slave communication will also use this function to read msg */
+        ioc = slave_io_channel(server, conn_fd, &local_err);
+    }
+
+    if (!ioc) {
+        error_report_err(local_err);
+        goto fail;
+    }
+
+    assert(qemu_in_coroutine());
+    do {
+        /*
+         * qio_channel_readv_full may have short reads, keeping calling it
+         * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+         */
+        rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &local_err);
+        if (rc < 0) {
+            if (rc == QIO_CHANNEL_ERR_BLOCK) {
+                qio_channel_yield(ioc, G_IO_IN);
+                continue;
+            } else {
+                error_report_err(local_err);
+                return false;
+            }
+        }
+        read_bytes += rc;
+        if (nfds_t > 0) {

util/vhost-user-server.c line 148

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+            fds = g_renew(int, fds, nfds + nfds_t);
+            memcpy(fds + nfds, fds_t, nfds_t *sizeof(int));
+            nfds += nfds_t;
+            if (nfds > VHOST_MEMORY_MAX_NREGIONS) {
+                error_report("A maximum of %d fds are allowed, "
+                             "however got %lu fds now",
+                             VHOST_MEMORY_MAX_NREGIONS, nfds);
+                goto fail;
+            }
+            g_free(fds_t);

I'm not sure why the temporary fds[] array is necessary. Copying the fds directly into vmsg->fds would be simpler:

if (nfds + nfds_t > G_N_ELEMENTS(vmsg->fds)) {
error_report("A maximum of %d fds are allowed, "
"however got %lu fds now", VHOST_MEMORY_MAX_NREGIONS, nfds);
goto fail;
}
memcpy(vmsg->fds + nfds, fds_t, nfds_t * sizeof(vds->fds[0])); nfds += nfds_t;

Quoted 1 line

Did I misunderstand how this works?

No, the temporary fds[] array is not necessary. Thank for the simplication!

Quoted 168 lines
Quoted 142 lines
util/vhost-user-server.c new file +142 -0
+        }
+        if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+            break;
+        }
+        iov.iov_base = (char *)vmsg + read_bytes;
+        iov.iov_len = VHOST_USER_HDR_SIZE - read_bytes;
+    } while (true);
+
+    vmsg->fd_num = nfds;
+    if (nfds > 0) {
+        memcpy(vmsg->fds, fds, nfds * sizeof(int));
+    }
+    g_free(fds);
+    /* qio_channel_readv_full will make socket fds blocking, unblock them */
+    vmsg_unblock_fds(vmsg);
+    if (vmsg->size > sizeof(vmsg->payload)) {
+        error_report("Error: too big message request: %d, "
+                     "size: vmsg->size: %u, "
+                     "while sizeof(vmsg->payload) = %zu",
+                     vmsg->request, vmsg->size, sizeof(vmsg->payload));
+        goto fail;
+    }
+
+    struct iovec iov_payload = {
+        .iov_base = (char *)&vmsg->payload,
+        .iov_len = vmsg->size,
+    };
+    if (vmsg->size) {
+        rc = qio_channel_readv_all_eof(ioc, &iov_payload, 1, &local_err);
+        if (rc == -1) {
+            error_report_err(local_err);
+            goto fail;
+        }
+    }
+
+    return true;
+
+fail:
+    vmsg_close_fds(vmsg);
+
+    return false;
+}
+
+
+static void vu_client_start(VuServer *server);
+static coroutine_fn void vu_client_trip(void *opaque)
+{
+    VuServer *server = opaque;
+
+    while (!server->aio_context_changed && server->sioc) {
+        vu_dispatch(&server->vu_dev);
+    }
+
+    if (server->aio_context_changed && server->sioc) {
+        server->aio_context_changed = false;
+        vu_client_start(server);
+    }
+}
+
+static void vu_client_start(VuServer *server)
+{
+    server->co_trip = qemu_coroutine_create(vu_client_trip, server);
+    aio_co_enter(server->ctx, server->co_trip);
+}
+
+/*
+ * a wrapper for vu_kick_cb
+ *
+ * since aio_dispatch can only pass one user data pointer to the
+ * callback function, pack VuDev and pvt into a struct. Then unpack it
+ * and pass them to vu_kick_cb
+ */
+static void kick_handler(void *opaque)
+{
+    KickInfo *kick_info = opaque;
+    kick_info->cb(kick_info->vu_dev, 0, (void *) kick_info->index);
+}
+
+
+static void
+set_watch(VuDev *vu_dev, int fd, int vu_evt,
+          vu_watch_cb cb, void *pvt)
+{
+
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+    long index = (intptr_t) pvt;
+    g_assert(cb);
+    KickInfo *kick_info = &server->kick_info[index];
+    if (!kick_info->cb) {
+        kick_info->fd = fd;
+        kick_info->cb = cb;
+        qemu_set_nonblock(fd);
+        aio_set_fd_handler(server->ioc->ctx, fd, false, kick_handler,
+                           NULL, NULL, kick_info);
+        kick_info->vu_dev = vu_dev;
+    }
+}
+
+
+static void remove_watch(VuDev *vu_dev, int fd)
+{
+    VuServer *server;
+    int i;
+    int index = -1;
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+
+    server = container_of(vu_dev, VuServer, vu_dev);
+    for (i = 0; i < vu_dev->max_queues; i++) {
+        if (server->kick_info[i].fd == fd) {
+            index = i;
+            break;
+        }
+    }
+
+    if (index == -1) {
+        return;
+    }
+    server->kick_info[i].cb = NULL;
+    aio_set_fd_handler(server->ioc->ctx, fd, false, NULL, NULL, NULL, NULL);
+}
+
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+                      gpointer opaque)
+{
+    VuServer *server = opaque;
+
+    if (server->sioc) {
+        warn_report("Only one vhost-user client is allowed to "
+                    "connect the server one time");
+        return;
+    }
+
+    if (!vu_init(&server->vu_dev, server->max_queues, sioc->fd, panic_cb,
+                 vu_message_read, set_watch, remove_watch, server->vu_iface)) {
+        error_report("Failed to initialized libvhost-user");
+        return;
+    }
+

util/vhost-user-server.c line 300

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+    /*
+     * Unset the callback function for network listener to make another
+     * vhost-user client keeping waiting until this client disconnects
+     */
+    qio_net_listener_set_client_func(server->listener,
+                                     NULL,
+                                     NULL,
+                                     NULL);
+    server->sioc = sioc;
+    server->kick_info = g_new0(KickInfo, server->max_queues);

Where is kick_info freed?

Review

Quoted 2 lines
(quoted diff) +2 -0
+    /*
+     * Increase the object reference, so cioc will not freed by

s/cioc/sioc/

util/vhost-user-server.c line 309

Quoted 7 lines
util/vhost-user-server.c new file +7 -0
+     * qio_net_listener_channel_func which will call object_unref(OBJECT(sioc))
+     */
+    object_ref(OBJECT(server->sioc));
+    qio_channel_set_name(QIO_CHANNEL(sioc), "vhost-user client");
+    server->ioc = QIO_CHANNEL(sioc);
+    object_ref(OBJECT(server->ioc));
+    object_ref(OBJECT(sioc));
Quoted 1 line

Why are there two object_refs for sioc and where is unref called?

Thank you for pointing out the errors regarding memory deallocation and the typo.

Quoted 100 lines
Quoted 5 lines
util/vhost-user-server.c new file +5 -0
+    qio_channel_attach_aio_context(server->ioc, server->ctx);
+    qio_channel_set_blocking(QIO_CHANNEL(server->sioc), false, NULL);
+    vu_client_start(server);
+}
+

util/vhost-user-server.c line 324

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+
+void vhost_user_server_stop(VuServer *server)
+{
+    if (!server) {
+        return;
+    }
+
+    if (server->sioc) {
+        close_client(server);
+        object_unref(OBJECT(server->sioc));

This call is object_unref(NULL) since close_client() does server->sioc = NULL.

Quoted 66 lines
util/vhost-user-server.c new file +66 -0
+    }
+
+    if (server->listener) {
+        qio_net_listener_disconnect(server->listener);
+        object_unref(OBJECT(server->listener));
+    }
+}
+
+static void detach_context(VuServer *server)
+{
+    int i;
+    AioContext *ctx = server->ioc->ctx;
+    qio_channel_detach_aio_context(server->ioc);
+    for (i = 0; i < server->vu_dev.max_queues; i++) {
+        if (server->kick_info[i].cb) {
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, false, NULL,
+                               NULL, NULL, NULL);
+        }
+    }
+}
+
+static void attach_context(VuServer *server, AioContext *ctx)
+{
+    int i;
+    qio_channel_attach_aio_context(server->ioc, ctx);
+    server->aio_context_changed = true;
+    if (server->co_trip) {
+        aio_co_schedule(ctx, server->co_trip);
+    }
+    for (i = 0; i < server->vu_dev.max_queues; i++) {
+        if (server->kick_info[i].cb) {
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, false,
+                               kick_handler, NULL, NULL,
+                               &server->kick_info[i]);
+        }
+    }
+}
+
+void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server)
+{
+    server->ctx = ctx ? ctx : qemu_get_aio_context();
+    if (!server->sioc) {
+        return;
+    }
+    if (ctx) {
+        attach_context(server, ctx);
+    } else {
+        detach_context(server);
+    }
+}
+
+
+bool vhost_user_server_start(uint16_t max_queues,
+                             SocketAddress *socket_addr,
+                             AioContext *ctx,
+                             VuServer *server,
+                             void *device_panic_notifier,
+                             const VuDevIface *vu_iface,
+                             Error **errp)
+{
+    server->listener = qio_net_listener_new();
+    if (qio_net_listener_open_sync(server->listener, socket_addr, 1,
+                                   errp) < 0) {
+        goto error;
+    }
+

util/vhost-user-server.c line 400

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+    qio_net_listener_set_name(server->listener, "vhost-user-backend-listener");
+
+    server->vu_iface = vu_iface;
+    server->max_queues = max_queues;
+    server->ctx = ctx;
+    server->device_panic_notifier = device_panic_notifier;
+    qio_net_listener_set_client_func(server->listener,
+                                     vu_accept,
+                                     server,
+                                     NULL);

The qio_net_listener_set_client_func() call uses the default GMainContext but we have an AioContext *ctx argument. This is surprising. I would expect the socket to be handled in the AioContext.

Quoted 1 line

Can you clarify how this should work?

Yes, the vhost-user server will accept new client connections in the default GMainText. But vhost-user message and kick event will be processed in the block drive's AIOContext and if the block drive's AIOContext is changed, these tasks will also be move to the new AIOContext.

Btw, I intended to use chardev to help manage client connections after limiting one vhost-user server to serve one client one time. But according to Mark and other people[1], converting all chardev functions to AIO doesn't seem to be worth the effort.

[1] https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg01485.html

Quoted 76 lines

util/vhost-user-server.c line 404

Quoted 4 lines
util/vhost-user-server.c new file +4 -0
+
+    return true;
+error:
+    g_free(server);

It's surprising that this function frees the server argument when an error occurs. vhost_user_server_stop() does not free server. I suggest letting the caller free server since they own the object.

Quoted 49 lines
(quoted diff) +2 -0
+    return false;
+}
util/vhost-user-server.h new file +41 -0
new file mode 100644
index 0000000000..4315556b66
@@ -0,0 +1,59 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+
+#ifndef VHOST_USER_SERVER_H
+#define VHOST_USER_SERVER_H
+
+#include "contrib/libvhost-user/libvhost-user.h"
+#include "io/channel-socket.h"
+#include "io/channel-file.h"
+#include "io/net-listener.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "standard-headers/linux/virtio_blk.h"
+
+typedef struct KickInfo {
+    VuDev *vu_dev;
+    int fd; /*kick fd*/
+    long index; /*queue index*/
+    vu_watch_cb cb;
+} KickInfo;
+
+typedef struct VuServer {
+    QIONetListener *listener;
+    AioContext *ctx;
+    void (*device_panic_notifier)(struct VuServer *server) ;
+    int max_queues;
+    const VuDevIface *vu_iface;
+    VuDev vu_dev;
+    QIOChannel *ioc; /* The I/O channel with the client */
+    QIOChannelSocket *sioc; /* The underlying data channel with the client */
+    /* IOChannel for fd provided via VHOST_USER_SET_SLAVE_REQ_FD */
+    QIOChannel *ioc_slave;
+    QIOChannelSocket *sioc_slave;
+    Coroutine *co_trip; /* coroutine for processing VhostUserMsg */
+    KickInfo *kick_info; /* an array with the length of the queue number */

util/vhost-user-server.h line 51

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    /* restart coroutine co_trip if AIOContext is changed */
+    bool aio_context_changed;
+} VuServer;
+
+
+bool vhost_user_server_start(uint16_t max_queues,
+                             SocketAddress *unix_socket,
+                             AioContext *ctx,
+                             VuServer *server,
+                             void *device_panic_notifier,

Please declare the function pointer type:

typedef void DevicePanicNotifierFn(struct VuServer *server);

Then the argument list can use DevicePanicNotifierFn *device_panic_notifier instead of void *.

Quoted 11 lines

util/vhost-user-server.h line 57

Quoted 6 lines
util/vhost-user-server.h new file +6 -0
+                             const VuDevIface *vu_iface,
+                             Error **errp);
+
+void vhost_user_server_stop(VuServer *server);
+
+void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server);

If you send another revision, please make VuServer *server the first argument of vhost_user_server_start() and vhost_user_server_set_aio_context(). Functions usually have the object they act on as the first argument.

Thank you! These issues have been addressed in v9.

--
Best regards,
Coiby

author
#7 lore raw

PATCH v8 3/4 vhost-user block device backend server

In reply to Coiby Xu

By making use of libvhost-user, block device drive can be shared to the connected vhost-user client. Only one client can connect to the server one time.

Since vhost-user-server needs a block drive to be created first, delay the creation of this object.

---
 block/Makefile.objs                  |   1 +
 block/export/vhost-user-blk-server.c | 716 +++++++++++++++++++++++++++
 block/export/vhost-user-blk-server.h |  34 ++
 softmmu/vl.c                         |   4 +
 4 files changed, 755 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
block/Makefile.objs +1 -0
index 3635b6b4c1..0eb7eff470 100644
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o ../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
block/export/vhost-user-blk-server.c new file +716 -0

716 lines not on this page · raw message

block/export/vhost-user-blk-server.h new file +34 -0
new file mode 100644
index 0000000000..6a393ed49d
@@ -0,0 +1,34 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+
+#ifndef VHOST_USER_BLK_SERVER_H
+#define VHOST_USER_BLK_SERVER_H
+#include "util/vhost-user-server.h"
+
+typedef struct VuBlockDev VuBlockDev;
+#define TYPE_VHOST_USER_BLK_SERVER "vhost-user-blk-server"
+#define VHOST_USER_BLK_SERVER(obj) \
+   OBJECT_CHECK(VuBlockDev, obj, TYPE_VHOST_USER_BLK_SERVER)
+
+/* vhost user block device */
+struct VuBlockDev {
+    Object parent_obj;
+    char *node_name;
+    SocketAddress *addr;
+    AioContext *ctx;
+    VuServer vu_server;
+    uint32_t blk_size;
+    BlockBackend *backend;
+    QIOChannelSocket *sioc;
+    QTAILQ_ENTRY(VuBlockDev) next;
+    struct virtio_blk_config blkcfg;
+    bool writable;
+};
+
+#endif /* VHOST_USER_BLK_SERVER_H */
softmmu/vl.c +4 -0
index ae5451bc23..e4549871e1 100644
@@ -2520,6 +2520,10 @@ static bool object_create_initial(const char *type, QemuOpts *opts)
     }
 #endif
 
+    /* Reason: vhost-user-blk-server property "node-name" */
+    if (g_str_equal(type, "vhost-user-blk-server")) {
+        return false;
+    }
     /*
      * Reason: filter-* property "netdev" etc.
      */
Signature · 1 line
2.26.2
#8 lore raw

Re: [PATCH v8 3/4] vhost-user block device backend server

Quoted 14 lines · Coiby Xu
block/export/vhost-user-blk-server.c new file +14 -0
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+    struct req_data *data = opaque;
+    VuServer *server = data->server;
+    VuVirtq *vq = data->vq;
+    VuVirtqElement *elem = data->elem;
+    uint32_t type;
+    VuBlockReq *req;
+
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    BlockBackend *backend = vdev_blk->backend;
+
+    struct iovec *in_iov = elem->in_sg;
+    struct iovec *out_iov = elem->out_sg;

block/export/vhost-user-blk-server.c line 124

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    unsigned in_num = elem->in_num;
+    unsigned out_num = elem->out_num;
+    /* refer to hw/block/virtio_blk.c */
+    if (elem->out_num < 1 || elem->in_num < 1) {
+        error_report("virtio-blk request missing headers");
+        free(elem);
+        return;
+    }
+
+    req = g_new0(VuBlockReq, 1);

elem was allocated with enough space for VuBlockReq. Can this allocation be eliminated?

typedef struct VuBlockReq {
- VuVirtqElement *elem;

+     VuVirtqElement elem;
      int64_t sector_num;
      size_t size;
      struct virtio_blk_inhdr *in;
      struct virtio_blk_outhdr out;
      VuServer *server;
      struct VuVirtq *vq;

} VuBlockReq;

  req = vu_queue_pop(vu_dev, vq, sizeof(*req));
Quoted 15 lines
block/export/vhost-user-blk-server.c new file +15 -0
+    req->server = server;
+    req->vq = vq;
+    req->elem = elem;
+
+    if (unlikely(iov_to_buf(out_iov, out_num, 0, &req->out,
+                            sizeof(req->out)) != sizeof(req->out))) {
+        error_report("virtio-blk request outhdr too short");
+        goto err;
+    }
+
+    iov_discard_front(&out_iov, &out_num, sizeof(req->out));
+
+    if (in_iov[in_num - 1].iov_len < sizeof(struct virtio_blk_inhdr)) {
+        error_report("virtio-blk request inhdr too short");
+        goto err;

block/export/vhost-user-blk-server.c line 149

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    }
+
+    /* We always touch the last byte, so just see how big in_iov is.  */
+    req->in = (void *)in_iov[in_num - 1].iov_base
+              + in_iov[in_num - 1].iov_len
+              - sizeof(struct virtio_blk_inhdr);
+    iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
+
+
+    type = le32toh(req->out.type);

This implementation assumes the request is always little-endian. This is true for VIRTIO 1.0+ but not for older versions. Please check that VIRTIO_F_VERSION_1 has been set.

In QEMU code the le32_to_cpu(), le64_to_cpu(), etc are common used instead of le32toh(), etc.

block/export/vhost-user-blk-server.c line 158

Quoted 9 lines
block/export/vhost-user-blk-server.c new file +9 -0
+    switch (type & ~VIRTIO_BLK_T_BARRIER) {
+    case VIRTIO_BLK_T_IN:
+    case VIRTIO_BLK_T_OUT: {
+        ssize_t ret = 0;
+        bool is_write = type & VIRTIO_BLK_T_OUT;
+        req->sector_num = le64toh(req->out.sector);
+
+        int64_t offset = req->sector_num * vdev_blk->blk_size;
+        QEMUIOVector *qiov = g_new0(QEMUIOVector, 1);

This can be allocated on the stack:

QEMUIOVector qiov;

Review

Quoted 2 lines
(quoted diff) +2 -0
+static void vhost_user_blk_server_free(VuBlockDev *vu_block_device)
+{

I'm unsure why this is a separate from vu_block_free(). Neither of these functions actually free VuBlockDev, so the name could be changed to vhost_user_blk_server_stop().

block/export/vhost-user-blk-server.c line 492

Quoted 11 lines
block/export/vhost-user-blk-server.c new file +11 -0
+    if (!vu_block_device) {
+        return;
+    }
+    vhost_user_server_stop(&vu_block_device->vu_server);
+    vu_block_free(vu_block_device);
+
+}
+
+/*
+ * A exported drive can serve multiple multiple clients simutateously,
+ * thus no need to export the same drive twice.

This comment is outdated. Only one client is served at a time.

Quoted 4 lines
block/export/vhost-user-blk-server.c new file +4 -0
+static void vhost_user_blk_server_start(VuBlockDev *vu_block_device,
+                                        Error **errp)
+{
+

block/export/vhost-user-blk-server.c line 535

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    const char *name = vu_block_device->node_name;
+    SocketAddress *addr = vu_block_device->addr;
+    char *unix_socket = vu_block_device->addr->u.q_unix.path;
+
+    if (vu_block_dev_find(name)) {
+        error_setg(errp, "Vhost-user-blk server with node-name '%s' "
+                   "has already been started",
+                   name);
+        return;
+    }

I think blk_new() permissions should prevent multiple writers. Having multiple readers would be okay. Therefore this check can be removed.

block/export/vhost-user-blk-server.c line 541

Quoted 6 lines
block/export/vhost-user-blk-server.c new file +6 -0
+
+    if (vu_block_dev_find_by_unix_socket(unix_socket)) {
+        error_setg(errp, "Vhost-user-blk server with with socket_path '%s' "
+                   "has already been started", unix_socket);
+        return;
+    }

Is it a problem if the same path is reused? I don't see an issue if the user creates a vhost-user-blk server, connects a client, unlinks the UNIX domain socket, and creates a new vhost-user-blk server with the same path. It might be a little confusing but if the user wants to do it, I don't see a reason to stop them.

block/export/vhost-user-blk-server.c line 553

Quoted 12 lines
block/export/vhost-user-blk-server.c new file +12 -0
+
+    if (!vu_block_init(vu_block_device, errp)) {
+        return;
+    }
+
+
+    AioContext *ctx = bdrv_get_aio_context(blk_bs(vu_block_device->backend));
+
+    if (!vhost_user_server_start(VHOST_USER_BLK_MAX_QUEUES, addr, ctx,
+                                 &vu_block_device->vu_server,
+                                 NULL, &vu_block_iface,
+                                 errp)) {

In the previous patch I mentioned that calling g_free(server) is probably unexpected and here is an example of why that can be a problem. vu_server is a struct field, not an independent heap-allocated object, so calling g_free(server) will result in undefined behavior (freeing an object that was not allocated with g_new()).

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+        goto error;
+    }
+
+    QTAILQ_INSERT_TAIL(&vu_block_devs, vu_block_device, next);
+    blk_add_aio_context_notifier(vu_block_device->backend, blk_aio_attached,
+                                 blk_aio_detach, vu_block_device);
+    return;
+
+ error:
+    vu_block_free(vu_block_device);

block/export/vhost-user-blk-server.c line 573

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+}
+
+static void vu_set_node_name(Object *obj, const char *value, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    if (vus->node_name) {
+        error_setg(errp, "evdev property already set");
+        return;
+    }

Setting it twice is okay, we just need to g_free(vus->node_name).

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+
+    vus->node_name = g_strdup(value);
+}
+
+static char *vu_get_node_name(Object *obj, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    return g_strdup(vus->node_name);
+}
+

block/export/vhost-user-blk-server.c line 593

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+
+static void vu_set_unix_socket(Object *obj, const char *value,
+                               Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    if (vus->addr) {
+        error_setg(errp, "unix_socket property already set");
+        return;
+    }

Setting it twice is okay, we just need to g_free(vus->addr->u.q_unix.path) and g_free(vus->addr).

Quoted 8 lines
block/export/vhost-user-blk-server.c new file +8 -0
+static void vu_set_blk_size(Object *obj, Visitor *v, const char *name,
+                            void *opaque, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    Error *local_err = NULL;
+    uint32_t value;
+

block/export/vhost-user-blk-server.c line 646

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    visit_type_uint32(v, name, &value, &local_err);
+    if (local_err) {
+        goto out;
+    }
+    if (value != BDRV_SECTOR_SIZE && value != 4096) {
+        error_setg(&local_err,
+                   "Property '%s.%s' can only take value 512 or 4096",
+                   object_get_typename(obj), name);
+        goto out;
+    }

Please see hw/core/qdev-properties.c:set_blocksize() for input validation checks (min=512, max=32768, must be a power of 2). This code can be moved into a common utility function so that both hw/core/qdev-properties.c and vhost-user-blk-server.c can use it.

Quoted 12 lines
block/export/vhost-user-blk-server.c new file +12 -0
+
+    vus->blk_size = value;
+
+out:
+    error_propagate(errp, local_err);
+    vus->blk_size = value;
+}
+
+
+static void vhost_user_blk_server_instance_finalize(Object *obj)
+{
+    VuBlockDev *vub = VHOST_USER_BLK_SERVER(obj);

block/export/vhost-user-blk-server.c line 668

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+
+    vhost_user_blk_server_free(vub);
+}
+
+static void vhost_user_blk_server_complete(UserCreatable *obj, Error **errp)
+{
+    Error *local_error = NULL;
+    VuBlockDev *vub = VHOST_USER_BLK_SERVER(obj);
+
+    vhost_user_blk_server_start(vub, &local_error);

After this call succeeds the properties should become read-only ("writable", "node-name", "unix-socket", etc) to prevent modification at runtime.

I think the easiest way to do that is by keeping a bool field in VuBlockDev that the property setter functions can check.

Quoted 18 lines
block/export/vhost-user-blk-server.c new file +18 -0
+
+    if (local_error) {
+        error_propagate(errp, local_error);
+        return;
+    }
+}
+
+static void vhost_user_blk_server_class_init(ObjectClass *klass,
+                                             void *class_data)
+{
+    UserCreatableClass *ucc = USER_CREATABLE_CLASS(klass);
+    ucc->complete = vhost_user_blk_server_complete;
+
+    object_class_property_add_bool(klass, "writable",
+                                   vu_get_block_writable,
+                                   vu_set_block_writable);
+
+    object_class_property_add_str(klass, "node-name",

block/export/vhost-user-blk-server.c line 696

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+                                  vu_get_node_name,
+                                  vu_set_node_name);
+
+    object_class_property_add_str(klass, "unix-socket",
+                                  vu_get_unix_socket,
+                                  vu_set_unix_socket);
+
+    object_class_property_add(klass, "blk-size", "uint32",
+                              vu_get_blk_size, vu_set_blk_size,
+                              NULL, NULL);

include/hw/block/block.h:DEFINE_BLOCK_PROPERTIES_BASE calls this property "logical_block_size". Please use the same name for consistency.

author
#9 lore raw

Re: [PATCH v8 3/4] vhost-user block device backend server

Quoted 30 lines · Stefan Hajnoczi
Quoted 14 lines · Coiby Xu
block/export/vhost-user-blk-server.c new file +14 -0
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+    struct req_data *data = opaque;
+    VuServer *server = data->server;
+    VuVirtq *vq = data->vq;
+    VuVirtqElement *elem = data->elem;
+    uint32_t type;
+    VuBlockReq *req;
+
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    BlockBackend *backend = vdev_blk->backend;
+
+    struct iovec *in_iov = elem->in_sg;
+    struct iovec *out_iov = elem->out_sg;

block/export/vhost-user-blk-server.c line 124

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    unsigned in_num = elem->in_num;
+    unsigned out_num = elem->out_num;
+    /* refer to hw/block/virtio_blk.c */
+    if (elem->out_num < 1 || elem->in_num < 1) {
+        error_report("virtio-blk request missing headers");
+        free(elem);
+        return;
+    }
+
+    req = g_new0(VuBlockReq, 1);

elem was allocated with enough space for VuBlockReq. Can this allocation be eliminated?

typedef struct VuBlockReq {

Review

Quoted 9 lines
(quoted diff) +1 -1
-     VuVirtqElement *elem;
+     VuVirtqElement elem;
     int64_t sector_num;
     size_t size;
     struct virtio_blk_inhdr *in;
     struct virtio_blk_outhdr out;
     VuServer *server;
     struct VuVirtq *vq;
 } VuBlockReq;

Thank you for review this patch. Other issues for this patch have been addressed in v9 except for this one. I'm not sure what you mean. I can't find a way that doesn't require to allocate a VuBlockReq struct.

--
Best regards,
Coiby

author
#10 lore raw

PATCH v8 4/4 new qTest case to test the vhost-user-blk-server

In reply to Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize. Since vhost-user server can only server one client one time, two instances of qemu-storage-daemon are launched for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will send "quit" command to qemu-storage-daemon's QMP monitor. So a function is added to libqtest.c to establish socket connection with socket server.

---
 tests/Makefile.include              |   3 +-
 tests/qtest/Makefile.include        |   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +++++
 tests/qtest/libqos/vhost-user-blk.h |  44 ++
 tests/qtest/libqtest.c              |  54 +-
 tests/qtest/libqtest.h              |  38 ++
 tests/qtest/vhost-user-blk-test.c   | 737 ++++++++++++++++++++++++++++
 7 files changed, 976 insertions(+), 32 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
tests/Makefile.include +2 -1
index 6e3d6370df..d8578346b0 100644
@@ -636,7 +636,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all $(check-qtest-y)
 	$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) $(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
 	  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
-	  QTEST_QEMU_IMG=qemu-img$(EXESUF))
+	  QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+	  QTEST_QEMU_STORAGE_DAEMON_BINARY=./qemu-storage-daemon$(EXESUF))
 
 check-unit: $(check-unit-y)
 	$(call do_test_human, $^)
tests/qtest/Makefile.include +2 -0
index 9e5a51d033..b6f081cb26 100644
@@ -186,6 +186,7 @@ libqos-obj-y += tests/qtest/libqos/virtio.o
 libqos-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 libqos-obj-y += tests/qtest/libqos/virtio-balloon.o
 libqos-obj-y += tests/qtest/libqos/virtio-blk.o
+libqos-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 libqos-obj-y += tests/qtest/libqos/virtio-mmio.o
 libqos-obj-y += tests/qtest/libqos/virtio-net.o
 libqos-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -230,6 +231,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
tests/qtest/libqos/vhost-user-blk.c new file +130 -0

130 lines not on this page · raw message

tests/qtest/libqos/vhost-user-blk.h new file +44 -0

44 lines not on this page · raw message

tests/qtest/libqtest.c +23 -31

69 lines not on this page · raw message

tests/qtest/libqtest.h +38 -0
index f5cf93c386..a6774d8781 100644
@@ -20,8 +20,38 @@
 #include "qapi/qmp/qobject.h"
 #include "qapi/qmp/qdict.h"
 
+#define MAX_IRQ 256
+
 typedef struct QTestState QTestState;
 
+typedef void (*QTestSendFn)(QTestState *s, const char *buf);
+typedef void (*ExternalSendFn)(void *s, const char *buf);
+typedef GString* (*QTestRecvFn)(QTestState *);
+
+typedef struct QTestClientTransportOps {
+    QTestSendFn     send;      /* for sending qtest commands */
+
+    /*
+     * use external_send to send qtest command strings through functions which
+     * do not accept a QTestState as the first parameter.
+     */
+    ExternalSendFn  external_send;
+
+    QTestRecvFn     recv_line; /* for receiving qtest command responses */
+} QTestTransportOps;
+
+struct QTestState {
+    int fd;
+    int qmp_fd;
+    pid_t qemu_pid;  /* our child QEMU process */
+    int wstatus;
+    int expected_status;
+    bool big_endian;
+    bool irq_level[MAX_IRQ];
+    GString *rx;
+    QTestTransportOps ops;
+};
+
 /**
  * qtest_initf:
  * @fmt...: Format for creating other arguments to pass to QEMU, formatted
@@ -626,6 +656,14 @@ void qtest_add_data_func_full(const char *str, void *data,
 
 void qtest_add_abrt_handler(GHookFunc fn, const void *data);
 
+/**
+ * qtest_socket_client:
+ * @server_socket_path: the socket server's path
+ *
+ * Connect to a socket server.
+ */
+int qtest_socket_client(char *server_socket_path);
+
 /**
  * qtest_qmp_assert_success:
  * @qts: QTestState instance to operate on
tests/qtest/vhost-user-blk-test.c new file +737 -0

737 lines not on this page · raw message

Signature · 1 line
2.26.2
#11 lore raw

Re: [PATCH v8 4/4] new qTest case to test the vhost-user-blk-server

Quoted 23 lines · Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize. Since vhost-user server can only server one client one time, two instances of qemu-storage-daemon are launched for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will send "quit" command to qemu-storage-daemon's QMP monitor. So a function is added to libqtest.c to establish socket connection with socket server.

---
 tests/Makefile.include              |   3 +-
 tests/qtest/Makefile.include        |   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +++++
 tests/qtest/libqos/vhost-user-blk.h |  44 ++
 tests/qtest/libqtest.c              |  54 +-
 tests/qtest/libqtest.h              |  38 ++
 tests/qtest/vhost-user-blk-test.c   | 737 ++++++++++++++++++++++++++++
 7 files changed, 976 insertions(+), 32 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

[...]

Quoted 7 lines

new file mode 100644
index 0000000000..3de9c59194

tests/qtest/libqos/vhost-user-blk.c new file +2 -0
@@ -0,0 +1,130 @@
+/*
+ * libqos driver framework

tests/qtest/libqos/vhost-user-blk.c line 12

Quoted 10 lines
tests/qtest/libqos/vhost-user-blk.c new file +10 -0
+ *
+ * Based on tests/qtest/libqos/virtio-blk.c
+ *
+ * Copyright (c) 2020 Coiby Xu <[email protected]>
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1 as published by the Free Software Foundation.

Thanks for the update! ...

[...]

Quoted 6 lines
tests/qtest/libqos/vhost-user-blk.h new file +0 -0
new file mode 100644
index 0000000000..ef4ef09cca

tests/qtest/libqos/vhost-user-blk.h line 8

Quoted 8 lines
tests/qtest/libqos/vhost-user-blk.h new file +8 -0
@@ -0,0 +1,44 @@
+/*
+ * libqos driver framework
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.

... but you've missed the header here :-(

[...]

Quoted 31 lines
tests/qtest/libqtest.c +0 -20
index 49075b55a1..a7b7c96206 100644
@@ -31,40 +31,9 @@
 #include "qapi/qmp/qlist.h"
 #include "qapi/qmp/qstring.h"
 
-#define MAX_IRQ 256
 #define SOCKET_TIMEOUT 50
 #define SOCKET_MAX_FDS 16
 
-
-typedef void (*QTestSendFn)(QTestState *s, const char *buf);
-typedef void (*ExternalSendFn)(void *s, const char *buf);
-typedef GString* (*QTestRecvFn)(QTestState *);
-
-typedef struct QTestClientTransportOps {
-    QTestSendFn     send;      /* for sending qtest commands */
-
-    /*
-     * use external_send to send qtest command strings through functions which
-     * do not accept a QTestState as the first parameter.
-     */
-    ExternalSendFn  external_send;
-
-    QTestRecvFn     recv_line; /* for receiving qtest command responses */
-} QTestTransportOps;
-
-struct QTestState
-{

tests/qtest/libqtest.c line 66

Quoted 10 lines
tests/qtest/libqtest.c +0 -10
-    int fd;
-    int qmp_fd;
-    pid_t qemu_pid;  /* our child QEMU process */
-    int wstatus;
-    int expected_status;
-    bool big_endian;
-    bool irq_level[MAX_IRQ];
-    GString *rx;
-    QTestTransportOps ops;
-};

Why do you have to move struct QTestState and friends to the header instead? I'd prefer if we could keep it here if possible?

Thomas

author
#12 lore raw

Re: [PATCH v8 4/4] new qTest case to test the vhost-user-blk-server

Quoted 8 lines · Thomas Huth
Quoted 8 lines
tests/qtest/libqos/vhost-user-blk.h new file +2 -0
new file mode 100644
index 0000000000..ef4ef09cca
@@ -0,0 +1,44 @@
+/*
+ * libqos driver framework
Quoted 8 lines

tests/qtest/libqos/vhost-user-blk.h line 8

Quoted 6 lines
tests/qtest/libqos/vhost-user-blk.h new file +6 -0
+ *
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.

... but you've missed the header here :-(

Thank you for reminding me of this issue!

Quoted 36 lines
Quoted 36 lines
tests/qtest/libqtest.c +0 -25
index 49075b55a1..a7b7c96206 100644
@@ -31,40 +31,9 @@
 #include "qapi/qmp/qlist.h"
 #include "qapi/qmp/qstring.h"
 
-#define MAX_IRQ 256
 #define SOCKET_TIMEOUT 50
 #define SOCKET_MAX_FDS 16
 
-
-typedef void (*QTestSendFn)(QTestState *s, const char *buf);
-typedef void (*ExternalSendFn)(void *s, const char *buf);
-typedef GString* (*QTestRecvFn)(QTestState *);
-
-typedef struct QTestClientTransportOps {
-    QTestSendFn     send;      /* for sending qtest commands */
-
-    /*
-     * use external_send to send qtest command strings through functions which
-     * do not accept a QTestState as the first parameter.
-     */
-    ExternalSendFn  external_send;
-
-    QTestRecvFn     recv_line; /* for receiving qtest command responses */
-} QTestTransportOps;
-
-struct QTestState
-{
-    int fd;
-    int qmp_fd;
-    pid_t qemu_pid;  /* our child QEMU process */
-    int wstatus;
-    int expected_status;
Quoted 8 lines

tests/qtest/libqtest.c line 66

Quoted 5 lines
tests/qtest/libqtest.c +0 -5
-    bool big_endian;
-    bool irq_level[MAX_IRQ];
-    GString *rx;
-    QTestTransportOps ops;
-};

Why do you have to move struct QTestState and friends to the header instead? I'd prefer if we could keep it here if possible?

tests/qtest/vhost-user-blk-test.c needs to talk to qemu-storage-daemon's QMP. Thus I g_new0 a QTestState struct to make use of related functions like qtest_qmp and this requires the QTestState struct definition.

#13 lore raw

Re: [PATCH v8 4/4] new qTest case to test the vhost-user-blk-server

Quoted 69 lines · Coiby Xu
Quoted 10 lines · Thomas Huth
Quoted 10 lines
tests/qtest/libqos/vhost-user-blk.h new file +3 -0
new file mode 100644
index 0000000000..ef4ef09cca
@@ -0,0 +1,44 @@
+/*
+ * libqos driver framework
+ *
Quoted 8 lines
Quoted 6 lines
tests/qtest/libqos/vhost-user-blk.h new file +1 -0
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito

<[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.

... but you've missed the header here :-(

Thank you for reminding me of this issue!

Quoted 38 lines
Quoted 38 lines
tests/qtest/libqtest.c +0 -11
index 49075b55a1..a7b7c96206 100644
@@ -31,40 +31,9 @@
  #include "qapi/qmp/qlist.h"
  #include "qapi/qmp/qstring.h"
 
-#define MAX_IRQ 256
  #define SOCKET_TIMEOUT 50
  #define SOCKET_MAX_FDS 16
 
-
-typedef void (*QTestSendFn)(QTestState *s, const char *buf);
-typedef void (*ExternalSendFn)(void *s, const char *buf);
-typedef GString* (*QTestRecvFn)(QTestState *);
-
-typedef struct QTestClientTransportOps {
-    QTestSendFn     send;      /* for sending qtest commands */
-
-    /*
-     * use external_send to send qtest command strings through

functions which
-     * do not accept a QTestState as the first parameter. -     */
-    ExternalSendFn  external_send;
-
-    QTestRecvFn     recv_line; /* for receiving qtest command responses */

tests/qtest/libqtest.c +0 -9
-} QTestTransportOps;
-
-struct QTestState
-{
-    int fd;
-    int qmp_fd;
-    pid_t qemu_pid;  /* our child QEMU process */
-    int wstatus;
-    int expected_status;
Quoted 8 lines
Quoted 5 lines

-    bool big_endian;
-    bool irq_level[MAX_IRQ];
-    GString *rx;
-    QTestTransportOps ops;
-};

Why do you have to move struct QTestState and friends to the header instead? I'd prefer if we could keep it here if possible?

Quoted 3 lines

tests/qtest/vhost-user-blk-test.c needs to talk to qemu-storage-daemon's QMP. Thus I g_new0 a QTestState struct to make use of related functions like qtest_qmp and this requires the QTestState struct definition.

Hm, ok, could that maybe be solved by introducing a wrapper function to libqtest.c instead? Something like qtest_create_state_with_qmp_fd() or so? Moving a define with a generic name like MAX_IRQ to a header really does not sound like a good idea to me, so if that idea with the wrapper function does not work out, could you please at least rename MAX_IRQ to QTEST_MAX_IRQ or something similar?

Thanks,
Thomas

author
#14 lore raw

Re: [PATCH v8 4/4] new qTest case to test the vhost-user-blk-server

Quoted 74 lines · Thomas Huth
Quoted 69 lines · Coiby Xu
Quoted 10 lines · Thomas Huth
Quoted 10 lines
tests/qtest/libqos/vhost-user-blk.h new file +3 -0
new file mode 100644
index 0000000000..ef4ef09cca
@@ -0,0 +1,44 @@
+/*
+ * libqos driver framework
+ *
Quoted 8 lines
Quoted 6 lines
tests/qtest/libqos/vhost-user-blk.h new file +1 -0
+ * Copyright (c) 2018 Emanuele Giuseppe Esposito

<[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2 as published by the Free Software Foundation.

... but you've missed the header here :-(

Thank you for reminding me of this issue!

Quoted 38 lines
Quoted 38 lines
tests/qtest/libqtest.c +0 -11
index 49075b55a1..a7b7c96206 100644
@@ -31,40 +31,9 @@
  #include "qapi/qmp/qlist.h"
  #include "qapi/qmp/qstring.h"
 
-#define MAX_IRQ 256
  #define SOCKET_TIMEOUT 50
  #define SOCKET_MAX_FDS 16
 
-
-typedef void (*QTestSendFn)(QTestState *s, const char *buf);
-typedef void (*ExternalSendFn)(void *s, const char *buf);
-typedef GString* (*QTestRecvFn)(QTestState *);
-
-typedef struct QTestClientTransportOps {
-    QTestSendFn     send;      /* for sending qtest commands */
-
-    /*
-     * use external_send to send qtest command strings through

functions which
-     * do not accept a QTestState as the first parameter. -     */
-    ExternalSendFn  external_send;
-
-    QTestRecvFn     recv_line; /* for receiving qtest command responses */

tests/qtest/libqtest.c +0 -9
-} QTestTransportOps;
-
-struct QTestState
-{
-    int fd;
-    int qmp_fd;
-    pid_t qemu_pid;  /* our child QEMU process */
-    int wstatus;
-    int expected_status;
Quoted 8 lines
Quoted 5 lines

-    bool big_endian;
-    bool irq_level[MAX_IRQ];
-    GString *rx;
-    QTestTransportOps ops;
-};

Why do you have to move struct QTestState and friends to the header instead? I'd prefer if we could keep it here if possible?

Quoted 3 lines

tests/qtest/vhost-user-blk-test.c needs to talk to qemu-storage-daemon's QMP. Thus I g_new0 a QTestState struct to make use of related functions like qtest_qmp and this requires the QTestState struct definition.

Quoted 6 lines

Hm, ok, could that maybe be solved by introducing a wrapper function to libqtest.c instead? Something like qtest_create_state_with_qmp_fd() or so? Moving a define with a generic name like MAX_IRQ to a header really does not sound like a good idea to me, so if that idea with the wrapper function does not work out, could you please at least rename MAX_IRQ to QTEST_MAX_IRQ or something similar?

I didn't realize the QTestState struct is supposed to be hidden from the user and not directly accessible. To typedef a struct in a header file and define the struct in the c file is a new trick for me:)

This idea of creating a wrapper function qtest_create_state_with_qmp_fd works as expected. Thank you!

#15 lore raw
In reply to Coiby Xu

Hi Coiby Xu,

Quoted 85 lines · Coiby Xu

v8
- re-try connecting to socket server to fix asan error
- fix license naming issue

v7
- fix docker-test-debug@fedora errors by freeing malloced memory

v6
- add missing license header and include guard
- vhost-user server only serve one client one time
- fix a bug in custom vu_message_read
- using qemu-storage-daemon to start vhost-user-blk-server
- a bug fix to pass docker-test-clang@ubuntu

v5:
* re-use vu_kick_cb in libvhost-user
* keeping processing VhostUserMsg in the same coroutine until there is detachment/attachment of AIOContext
* Spawn separate coroutine for each VuVirtqElement
* Other changes including relocating vhost-user-blk-server.c, coding style etc.

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

v3:
* separate generic vhost-user-server code from vhost-user-blk-server code
* re-write vu_message_read and kick hander function as coroutines to directly call blk_co_preadv, blk_co_pwritev, etc.
* add aio_context notifier functions to support multi-threading model
* other fixes regarding coding style, warning report, etc.

v2:
* Only enable this feature for Linux because eventfd is a Linux-specific feature

This patch series is an implementation of vhost-user block device backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be started using object_add,

(qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend. For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (4):
Allow vu_message_read to be replaced generic vhost user server
vhost-user block device backend server new qTest case to test the vhost-user-blk-server

 block/Makefile.objs                        |   1 +
 block/export/vhost-user-blk-server.c       | 716 ++++++++++++++++++++
 block/export/vhost-user-blk-server.h       |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c      |  11 +-
 contrib/libvhost-user/libvhost-user.h      |  21 +
 softmmu/vl.c                               |   4 +
 tests/Makefile.include                     |   3 +-
 tests/qtest/Makefile.include               |   2 +
 tests/qtest/libqos/vhost-user-blk.c        | 130 ++++
 tests/qtest/libqos/vhost-user-blk.h        |  44 ++
 tests/qtest/libqtest.c                     |  54 +-
 tests/qtest/libqtest.h                     |  38 ++
 tests/qtest/vhost-user-blk-test.c          | 737 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |   2 +
 tools/virtiofsd/fuse_virtio.c              |   4 +-
 util/Makefile.objs                         |   1 +
 util/vhost-user-server.c                   | 406 ++++++++++++
 util/vhost-user-server.h                   |  59 ++
Quoted 9 lines
 19 files changed, 2229 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

Should we add an entry in the MAINTAINERS file for some of the new files? (e.g. util/vhost-user-server.*)

Thanks,
Stefano

author
#16 lore raw

Hi Stefano Garzarella,

Quoted 98 lines · Stefano Garzarella

Hi Coiby Xu,

Quoted 85 lines · Coiby Xu

v8
- re-try connecting to socket server to fix asan error
- fix license naming issue

v7
- fix docker-test-debug@fedora errors by freeing malloced memory

v6
- add missing license header and include guard
- vhost-user server only serve one client one time
- fix a bug in custom vu_message_read
- using qemu-storage-daemon to start vhost-user-blk-server
- a bug fix to pass docker-test-clang@ubuntu

v5:
* re-use vu_kick_cb in libvhost-user
* keeping processing VhostUserMsg in the same coroutine until there is detachment/attachment of AIOContext
* Spawn separate coroutine for each VuVirtqElement
* Other changes including relocating vhost-user-blk-server.c, coding style etc.

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

v3:
* separate generic vhost-user-server code from vhost-user-blk-server code
* re-write vu_message_read and kick hander function as coroutines to directly call blk_co_preadv, blk_co_pwritev, etc.
* add aio_context notifier functions to support multi-threading model
* other fixes regarding coding style, warning report, etc.

v2:
* Only enable this feature for Linux because eventfd is a Linux-specific feature

This patch series is an implementation of vhost-user block device backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be started using object_add,

(qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend. For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (4):
Allow vu_message_read to be replaced generic vhost user server
vhost-user block device backend server new qTest case to test the vhost-user-blk-server

 block/Makefile.objs                        |   1 +
 block/export/vhost-user-blk-server.c       | 716 ++++++++++++++++++++
 block/export/vhost-user-blk-server.h       |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c      |  11 +-
 contrib/libvhost-user/libvhost-user.h      |  21 +
 softmmu/vl.c                               |   4 +
 tests/Makefile.include                     |   3 +-
 tests/qtest/Makefile.include               |   2 +
 tests/qtest/libqos/vhost-user-blk.c        | 130 ++++
 tests/qtest/libqos/vhost-user-blk.h        |  44 ++
 tests/qtest/libqtest.c                     |  54 +-
 tests/qtest/libqtest.h                     |  38 ++
 tests/qtest/vhost-user-blk-test.c          | 737 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |   2 +
 tools/virtiofsd/fuse_virtio.c              |   4 +-
 util/Makefile.objs                         |   1 +
 util/vhost-user-server.c                   | 406 ++++++++++++
 util/vhost-user-server.h                   |  59 ++
Quoted 9 lines
 19 files changed, 2229 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
Quoted 2 lines

Should we add an entry in the MAINTAINERS file for some of the new files? (e.g. util/vhost-user-server.*)

Yes, please. Thank you!

--
Best regards,
Coiby

#17 lore raw
Quoted 101 lines · Coiby Xu

Hi Stefano Garzarella,

Quoted 98 lines · Stefano Garzarella

Hi Coiby Xu,

Quoted 94 lines · Coiby Xu

v8
- re-try connecting to socket server to fix asan error
- fix license naming issue

v7
- fix docker-test-debug@fedora errors by freeing malloced memory

v6
- add missing license header and include guard
- vhost-user server only serve one client one time
- fix a bug in custom vu_message_read
- using qemu-storage-daemon to start vhost-user-blk-server
- a bug fix to pass docker-test-clang@ubuntu

v5:
* re-use vu_kick_cb in libvhost-user
* keeping processing VhostUserMsg in the same coroutine until there is detachment/attachment of AIOContext
* Spawn separate coroutine for each VuVirtqElement
* Other changes including relocating vhost-user-blk-server.c, coding style etc.

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

v3:
* separate generic vhost-user-server code from vhost-user-blk-server code
* re-write vu_message_read and kick hander function as coroutines to directly call blk_co_preadv, blk_co_pwritev, etc.
* add aio_context notifier functions to support multi-threading model
* other fixes regarding coding style, warning report, etc.

v2:
* Only enable this feature for Linux because eventfd is a Linux-specific feature

This patch series is an implementation of vhost-user block device backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be started using object_add,

(qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend. For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (4):
Allow vu_message_read to be replaced generic vhost user server
vhost-user block device backend server new qTest case to test the vhost-user-blk-server

 block/Makefile.objs                        |   1 +
 block/export/vhost-user-blk-server.c       | 716 ++++++++++++++++++++
 block/export/vhost-user-blk-server.h       |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c      |  11 +-
 contrib/libvhost-user/libvhost-user.h      |  21 +
 softmmu/vl.c                               |   4 +
 tests/Makefile.include                     |   3 +-
 tests/qtest/Makefile.include               |   2 +
 tests/qtest/libqos/vhost-user-blk.c        | 130 ++++
 tests/qtest/libqos/vhost-user-blk.h        |  44 ++
 tests/qtest/libqtest.c                     |  54 +-
 tests/qtest/libqtest.h                     |  38 ++
 tests/qtest/vhost-user-blk-test.c          | 737 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |   2 +
 tools/virtiofsd/fuse_virtio.c              |   4 +-
 util/Makefile.objs                         |   1 +
 util/vhost-user-server.c                   | 406 ++++++++++++
 util/vhost-user-server.h                   |  59 ++
 19 files changed, 2229 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
Quoted 4 lines
Quoted 2 lines

Should we add an entry in the MAINTAINERS file for some of the new files? (e.g. util/vhost-user-server.*)

Yes, please. Thank you!

I think the best thing should be to edit MAINTAINERS in this series, since you're adding new files, but I don't know who will maintain them ;-)

Thanks,
Stefano

author
#18 lore raw
Quoted 102 lines · Stefano Garzarella
Quoted 101 lines · Coiby Xu

Hi Stefano Garzarella,

Quoted 98 lines · Stefano Garzarella

Hi Coiby Xu,

Quoted 94 lines · Coiby Xu

v8
- re-try connecting to socket server to fix asan error
- fix license naming issue

v7
- fix docker-test-debug@fedora errors by freeing malloced memory

v6
- add missing license header and include guard
- vhost-user server only serve one client one time
- fix a bug in custom vu_message_read
- using qemu-storage-daemon to start vhost-user-blk-server
- a bug fix to pass docker-test-clang@ubuntu

v5:
* re-use vu_kick_cb in libvhost-user
* keeping processing VhostUserMsg in the same coroutine until there is detachment/attachment of AIOContext
* Spawn separate coroutine for each VuVirtqElement
* Other changes including relocating vhost-user-blk-server.c, coding style etc.

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

v3:
* separate generic vhost-user-server code from vhost-user-blk-server code
* re-write vu_message_read and kick hander function as coroutines to directly call blk_co_preadv, blk_co_pwritev, etc.
* add aio_context notifier functions to support multi-threading model
* other fixes regarding coding style, warning report, etc.

v2:
* Only enable this feature for Linux because eventfd is a Linux-specific feature

This patch series is an implementation of vhost-user block device backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be started using object_add,

(qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend. For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (4):
Allow vu_message_read to be replaced generic vhost user server
vhost-user block device backend server new qTest case to test the vhost-user-blk-server

 block/Makefile.objs                        |   1 +
 block/export/vhost-user-blk-server.c       | 716 ++++++++++++++++++++
 block/export/vhost-user-blk-server.h       |  34 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c      |  11 +-
 contrib/libvhost-user/libvhost-user.h      |  21 +
 softmmu/vl.c                               |   4 +
 tests/Makefile.include                     |   3 +-
 tests/qtest/Makefile.include               |   2 +
 tests/qtest/libqos/vhost-user-blk.c        | 130 ++++
 tests/qtest/libqos/vhost-user-blk.h        |  44 ++
 tests/qtest/libqtest.c                     |  54 +-
 tests/qtest/libqtest.h                     |  38 ++
 tests/qtest/vhost-user-blk-test.c          | 737 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |   2 +
 tools/virtiofsd/fuse_virtio.c              |   4 +-
 util/Makefile.objs                         |   1 +
 util/vhost-user-server.c                   | 406 ++++++++++++
 util/vhost-user-server.h                   |  59 ++
 19 files changed, 2229 insertions(+), 40 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Should we add an entry in the MAINTAINERS file for some of the new files? (e.g. util/vhost-user-server.*)

Yes, please. Thank you!

I think the best thing should be to edit MAINTAINERS in this series, since you're adding new files, but I don't know who will maintain them ;-)

Thank you for the explanation! I thought the MAINTAINERS file is supposed to be treated in a special way:)

--
Best regards,
Coiby

#19 lore raw
In reply to Coiby Xu
Quoted 3 lines · Coiby Xu

v8
- re-try connecting to socket server to fix asan error
- fix license naming issue

Great, thanks for posting these patches!

I have posted feedback. I'd like to merge this soon. If you are busy I can send you patches that address the comments I've made, please let me know.

Thanks,
Stefan

author
#20 lore raw
Quoted 10 lines · Stefan Hajnoczi
Quoted 3 lines · Coiby Xu

v8
- re-try connecting to socket server to fix asan error
- fix license naming issue

Great, thanks for posting these patches!

I have posted feedback. I'd like to merge this soon. If you are busy I can send you patches that address the comments I've made, please let me know.

Thank you for reviewing my work! I'll post v9 to address all the comments this weekend, does you think it's soon enough?

Best regards,
Coiby