Skip to contentSkip to conversation

vhost-user block device backend implementation

PATCH Superseded Coiby Xu 6 participants 22 messages Feb 18, 2020 – Feb 28, 2020 Kbuild

v4 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Coiby Xu

author
#0 lore raw

v4 cover vhost-user block device backend implementation

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

v3:
* separate generic vhost-user-server code from vhost-user-blk-server code
* re-write vu_message_read and kick hander function as coroutines to directly call blk_co_preadv, blk_co_pwritev, etc.
* add aio_context notifier functions to support multi-threading model
* other fixes regarding coding style, warning report, etc.

v2:
* Only enable this feauture for Linux because eventfd is a Linux-specific feature

This patch series is an implementation of vhost-user block device backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be started using object_add,

(qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512

Then vhost-user client can connect to the server backend. For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mouting it.

Coiby Xu (5):
extend libvhost to support IOThread and coroutine
generic vhost user server
vhost-user block device backend server
a standone-alone tool to directly share disk image file via vhost-user
protocol
new qTest case to test the vhost-user-blk-server

 Makefile                              |   4 +
 Makefile.target                       |   1 +
 backends/Makefile.objs                |   2 +
 backends/vhost-user-blk-server.c      | 718 ++++++++++++++++++++++++++
 backends/vhost-user-blk-server.h      |  21 +
 configure                             |   3 +
 contrib/libvhost-user/libvhost-user.c |  54 +-
 contrib/libvhost-user/libvhost-user.h |  38 +-
 qemu-vu.c                             | 252 +++++++++
 tests/Makefile.include                |   3 +-
 tests/qtest/Makefile.include          |   2 +
 tests/qtest/libqos/vhost-user-blk.c   | 126 +++++
 tests/qtest/libqos/vhost-user-blk.h   |  44 ++
 tests/qtest/vhost-user-blk-test.c     | 694 +++++++++++++++++++++++++
 util/Makefile.objs                    |   3 +
 util/vhost-user-server.c              | 427 +++++++++++++++
 util/vhost-user-server.h              |  56 ++
 vl.c                                  |   4 +
 18 files changed, 2444 insertions(+), 8 deletions(-)
 create mode 100644 backends/vhost-user-blk-server.c
 create mode 100644 backends/vhost-user-blk-server.h
 create mode 100644 qemu-vu.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.25.0

author
#1 lore raw

PATCH v4 1/5 extend libvhost to support IOThread and coroutine

Previously libvhost dispatch events in its own GMainContext. Now vhost-user client's kick event can be dispatched in block device drive's AioContext thus IOThread is supported. And also allow vu_message_read and vu_kick_cb to be replaced so QEMU can run them as coroutines.

---
 contrib/libvhost-user/libvhost-user.c | 54 ++++++++++++++++++++++++---
 contrib/libvhost-user/libvhost-user.h | 38 ++++++++++++++++++-
 2 files changed, 85 insertions(+), 7 deletions(-)
contrib/libvhost-user/libvhost-user.c +49 -5
index b89bf18501..f95664bb22 100644
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -260,7 +258,7 @@ have_userfault(void)
 }
 
 static bool
-vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+vu_message_read_(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {
     char control[CMSG_SPACE(VHOST_MEMORY_MAX_NREGIONS * sizeof(int))] = { };
     struct iovec iov = {
@@ -328,6 +326,17 @@ fail:
     return false;
 }
 
+static bool vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    vu_read_msg_cb read_msg;
+    if (dev->co_iface) {
+        read_msg = dev->co_iface->read_msg;
+    } else {
+        read_msg = vu_message_read_;
+    }
+    return read_msg(dev, conn_fd, vmsg);
+}
+
 static bool
 vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {
@@ -1075,9 +1084,14 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
     }
 
     if (dev->vq[index].kick_fd != -1 && dev->vq[index].handler) {
+        if (dev->set_watch_packed_data) {
+            dev->set_watch_packed_data(dev, dev->vq[index].kick_fd, VU_WATCH_IN,
+                                       dev->co_iface->kick_callback,
+                                       (void *)(long)index);
+        } else {
         dev->set_watch(dev, dev->vq[index].kick_fd, VU_WATCH_IN,
                        vu_kick_cb, (void *)(long)index);
-
+        }
         DPRINT("Waiting for kicks on fd: %d for vq: %d\n",
                dev->vq[index].kick_fd, index);
     }
@@ -1097,8 +1111,14 @@ void vu_set_queue_handler(VuDev *dev, VuVirtq *vq,
     vq->handler = handler;
     if (vq->kick_fd >= 0) {
         if (handler) {
+            if (dev->set_watch_packed_data) {
+                dev->set_watch_packed_data(dev, vq->kick_fd, VU_WATCH_IN,
+                                           dev->co_iface->kick_callback,
+                                           (void *)(long)qidx);
+            } else {
             dev->set_watch(dev, vq->kick_fd, VU_WATCH_IN,
                            vu_kick_cb, (void *)(long)qidx);
+            }
         } else {
             dev->remove_watch(dev, vq->kick_fd);
         }
@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            /* remove watch for kick_fd
+             * When client process is running in gdb and
+             * quit command is run in gdb, QEMU will still dispatch the event
+             * which will cause segment fault in the callback function
+             */
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }
@@ -1682,7 +1708,7 @@ vu_init(VuDev *dev,
 
     assert(max_queues > 0);
     assert(socket >= 0);
-    assert(set_watch);
+    /* assert(set_watch); */
     assert(remove_watch);
     assert(iface);
     assert(panic);
@@ -1715,6 +1741,24 @@ vu_init(VuDev *dev,
     return true;
 }
 
+bool
+vu_init_packed_data(VuDev *dev,
+        uint16_t max_queues,
+        int socket,
+        vu_panic_cb panic,
+        vu_set_watch_cb_packed_data set_watch_packed_data,
+        vu_remove_watch_cb remove_watch,
+        const VuDevIface *iface,
+        const CoIface *co_iface)
+{
+    if (vu_init(dev, max_queues, socket, panic, NULL, remove_watch, iface)) {
+        dev->set_watch_packed_data = set_watch_packed_data;
+        dev->co_iface = co_iface;
+        return true;
+    }
+    return false;
+}
+
 VuVirtq *
 vu_get_queue(VuDev *dev, int qidx)
 {
contrib/libvhost-user/libvhost-user.h +36 -2
index 5cb7708559..6aadeaa0f2 100644
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
     VHOST_SET_CONFIG_TYPE_MASTER = 0,
     VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -201,6 +203,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
                                   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -208,6 +211,20 @@ typedef int (*vu_set_config_cb) (VuDev *dev, const uint8_t *data,
                                  uint32_t offset, uint32_t size,
                                  uint32_t flags);
 
+typedef void (*vu_watch_cb_packed_data) (void *packed_data);
+
+typedef void (*vu_set_watch_cb_packed_data) (VuDev *dev, int fd, int condition,
+                                             vu_watch_cb_packed_data cb,
+                                             void *data);
+/*
+ * allowing vu_read_msg_cb and kick_callback to be replaced so QEMU
+ * can run them as coroutines
+ */
+typedef struct CoIface {
+    vu_read_msg_cb read_msg;
+    vu_watch_cb_packed_data kick_callback;
+} CoIface;
+
 typedef struct VuDevIface {
     /* called by VHOST_USER_GET_FEATURES to get the features bitmask */
     vu_get_features_cb get_features;
@@ -372,7 +389,8 @@ struct VuDev {
     /* @set_watch: add or update the given fd to the watch set,
      * call cb when condition is met */
     vu_set_watch_cb set_watch;
-
+    /* AIO dispatch will only one data pointer to callback function */
+    vu_set_watch_cb_packed_data set_watch_packed_data;
     /* @remove_watch: remove the given fd from the watch set */
     vu_remove_watch_cb remove_watch;
 
@@ -380,7 +398,7 @@ struct VuDev {
      * re-initialize */
     vu_panic_cb panic;
     const VuDevIface *iface;
-
+    const CoIface *co_iface;
     /* Postcopy data */
     int postcopy_ufd;
     bool postcopy_listening;
@@ -417,6 +435,22 @@ bool vu_init(VuDev *dev,
              const VuDevIface *iface);
 
 
+/**
+ * vu_init_packed_data:
+ * Same as vu_init except for set_watch_packed_data which will pack
+ * two parameters into a struct thus QEMU aio_dispatch can pass the
+ * required data to callback function.
+ *
+ * Returns: true on success, false on failure.
+ **/
+bool vu_init_packed_data(VuDev *dev,
+                         uint16_t max_queues,
+                         int socket,
+                         vu_panic_cb panic,
+                         vu_set_watch_cb_packed_data set_watch_packed_data,
+                         vu_remove_watch_cb remove_watch,
+                         const VuDevIface *iface,
+                         const CoIface *co_iface);
 /**
  * vu_deinit:
  * @dev: a VuDev context
Signature · 1 line
2.25.0
#2 lore raw

Re: [PATCH v4 1/5] extend libvhost to support IOThread and coroutine

contrib/libvhost-user/libvhost-user.c fail: · line 338

Quoted 10 lines · Coiby Xu
contrib/libvhost-user/libvhost-user.c +10 -0
+static bool vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    vu_read_msg_cb read_msg;
+    if (dev->co_iface) {
+        read_msg = dev->co_iface->read_msg;
+    } else {
+        read_msg = vu_message_read_;
+    }
+    return read_msg(dev, conn_fd, vmsg);
+}

libvhost-user is already partially asynchronous (->set/remove_watch() are used for eventfds).

Can you make the vhost-user socket I/O asynchronous too?

Quoted 5 lines
contrib/libvhost-user/libvhost-user.c +1 -0
+
 static bool
 vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {

contrib/libvhost-user/libvhost-user.c vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) · line 1093

Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +5 -0
@@ -1075,9 +1084,14 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
     }
 
     if (dev->vq[index].kick_fd != -1 && dev->vq[index].handler) {
+        if (dev->set_watch_packed_data) {
+            dev->set_watch_packed_data(dev, dev->vq[index].kick_fd, VU_WATCH_IN,
+                                       dev->co_iface->kick_callback,
+                                       (void *)(long)index);
+        } else {
         dev->set_watch(dev, dev->vq[index].kick_fd, VU_WATCH_IN,
                        vu_kick_cb, (void *)(long)index);

Why is it necessary to replace vu_kick_cb()?

The user can set a custom vq->handler() function with vu_set_queue_handler().

Coroutines aren't needed for this part since eventfd_read() always returns right away.

contrib/libvhost-user/libvhost-user.c vu_deinit(VuDev *dev) · line 1654

Quoted 9 lines
contrib/libvhost-user/libvhost-user.c +5 -0
@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            /* remove watch for kick_fd
+             * When client process is running in gdb and
+             * quit command is run in gdb, QEMU will still dispatch the event
+             * which will cause segment fault in the callback function
+             */

Code and comments in libvhost-user should not refer to QEMU specifics. Removing the watch is a good idea regardless of the application or event loop implementation. No comment is needed here.

contrib/libvhost-user/libvhost-user.c vu_init(VuDev *dev, · line 1711

Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +2 -1
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }
@@ -1682,7 +1708,7 @@ vu_init(VuDev *dev,
 
     assert(max_queues > 0);
     assert(socket >= 0);
-    assert(set_watch);
+    /* assert(set_watch); */

?

contrib/libvhost-user/libvhost-user.h struct VuDev { · line 392

Quoted 6 lines
contrib/libvhost-user/libvhost-user.h +1 -1
@@ -372,7 +389,8 @@ struct VuDev {
     /* @set_watch: add or update the given fd to the watch set,
      * call cb when condition is met */
     vu_set_watch_cb set_watch;
-
+    /* AIO dispatch will only one data pointer to callback function */

I don't understand what this comment is saying.

#3 lore raw

Re: [PATCH v4 1/5] extend libvhost to support IOThread and coroutine

In reply to Coiby Xu

Am 18.02.2020 um 06:07 hat Coiby Xu geschrieben:

Quoted 39 lines

Previously libvhost dispatch events in its own GMainContext. Now vhost-user client's kick event can be dispatched in block device drive's AioContext thus IOThread is supported. And also allow vu_message_read and vu_kick_cb to be replaced so QEMU can run them as coroutines.

---
 contrib/libvhost-user/libvhost-user.c | 54 ++++++++++++++++++++++++---
 contrib/libvhost-user/libvhost-user.h | 38 ++++++++++++++++++-
 2 files changed, 85 insertions(+), 7 deletions(-)
contrib/libvhost-user/libvhost-user.h +6 -0
index 5cb7708559..6aadeaa0f2 100644
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
     VHOST_SET_CONFIG_TYPE_MASTER = 0,
     VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -201,6 +203,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
                                   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -208,6 +211,20 @@ typedef int (*vu_set_config_cb) (VuDev *dev, const uint8_t *data,
                                  uint32_t offset, uint32_t size,
                                  uint32_t flags);
 
+typedef void (*vu_watch_cb_packed_data) (void *packed_data);
+
+typedef void (*vu_set_watch_cb_packed_data) (VuDev *dev, int fd, int condition,

contrib/libvhost-user/libvhost-user.h typedef int (*vu_set_config_cb) (VuDev *dev, const uint8_t *data, · line 226

Quoted 10 lines
contrib/libvhost-user/libvhost-user.h +10 -0
+                                             vu_watch_cb_packed_data cb,
+                                             void *data);
+/*
+ * allowing vu_read_msg_cb and kick_callback to be replaced so QEMU
+ * can run them as coroutines
+ */
+typedef struct CoIface {
+    vu_read_msg_cb read_msg;
+    vu_watch_cb_packed_data kick_callback;
+} CoIface;

I think this should be part of VuDevIface, so that it becomes a properly integrated part of the design instead of an adapter hacked on top.

Quoted 23 lines
contrib/libvhost-user/libvhost-user.h +3 -2
 typedef struct VuDevIface {
     /* called by VHOST_USER_GET_FEATURES to get the features bitmask */
     vu_get_features_cb get_features;
@@ -372,7 +389,8 @@ struct VuDev {
     /* @set_watch: add or update the given fd to the watch set,
      * call cb when condition is met */
     vu_set_watch_cb set_watch;
-
+    /* AIO dispatch will only one data pointer to callback function */
+    vu_set_watch_cb_packed_data set_watch_packed_data;
     /* @remove_watch: remove the given fd from the watch set */
     vu_remove_watch_cb remove_watch;
 
@@ -380,7 +398,7 @@ struct VuDev {
      * re-initialize */
     vu_panic_cb panic;
     const VuDevIface *iface;
-
+    const CoIface *co_iface;
     /* Postcopy data */
     int postcopy_ufd;
     bool postcopy_listening;

contrib/libvhost-user/libvhost-user.h bool vu_init(VuDev *dev, · line 441

Quoted 7 lines
contrib/libvhost-user/libvhost-user.h +4 -0
@@ -417,6 +435,22 @@ bool vu_init(VuDev *dev,
              const VuDevIface *iface);
 
 
+/**
+ * vu_init_packed_data:
+ * Same as vu_init except for set_watch_packed_data which will pack
+ * two parameters into a struct

Be specific: Which two parameters and which struct?

I think it would be more helpful to name the function after the additional piece of information that it uses rather than the fact that it stores it internally in a struct.

We have:

typedef void (*vu_set_watch_cb) (VuDev *dev, int fd, int condition,

                                 vu_watch_cb cb, void *data);
typedef void (*vu_set_watch_cb_packed_data) (VuDev *dev, int fd, int condition,
                                             vu_watch_cb_packed_data cb,
                                             void *data);

Without looking at the implementation, they have the same set of parameters. I suspect that the difference is in the content of *data, but since it is declared void*, I suppose it's treated as an opaque data type and will only be passed unchanged (and uninspected) to cb.

If so, there is no differene between both types.

Quoted 33 lines

thus QEMU aio_dispatch can pass the
+ * required data to callback function.
+ *
+ * Returns: true on success, false on failure.
+ **/

contrib/libvhost-user/libvhost-user.h +8 -0
+bool vu_init_packed_data(VuDev *dev,
+                         uint16_t max_queues,
+                         int socket,
+                         vu_panic_cb panic,
+                         vu_set_watch_cb_packed_data set_watch_packed_data,
+                         vu_remove_watch_cb remove_watch,
+                         const VuDevIface *iface,
+                         const CoIface *co_iface);
 /**
  * vu_deinit:
  * @dev: a VuDev context
Signature · 1 line
2.25.0
contrib/libvhost-user/libvhost-user.c +0 -2
index b89bf18501..f95664bb22 100644
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0

contrib/libvhost-user/libvhost-user.c have_userfault(void) · line 261

Quoted 5 lines
contrib/libvhost-user/libvhost-user.c +1 -1
@@ -260,7 +258,7 @@ have_userfault(void)
 }
 
 static bool
-vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+vu_message_read_(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)

Just adding a trailing underscore isn't a good name. It doesn't tell the reader what the difference between vu_message_read_ and vu_message_read is.

Quoted 7 lines
contrib/libvhost-user/libvhost-user.c +0 -0
 {
     char control[CMSG_SPACE(VHOST_MEMORY_MAX_NREGIONS * sizeof(int))] = { };
     struct iovec iov = {
@@ -328,6 +326,17 @@ fail:
     return false;
 }
 

contrib/libvhost-user/libvhost-user.c fail: · line 338

Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +10 -0
+static bool vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    vu_read_msg_cb read_msg;
+    if (dev->co_iface) {
+        read_msg = dev->co_iface->read_msg;
+    } else {
+        read_msg = vu_message_read_;
+    }
+    return read_msg(dev, conn_fd, vmsg);
+}

If you change VuDevIface so that it contains the fields of CoIface directly, you can just initialise dev->iface->read_msg with what is called vu_message_read_() now for the non-QEMU case, and this whole wrapper becomes unnecessary because the code path is the same for both cases.

Quoted 6 lines
contrib/libvhost-user/libvhost-user.c +0 -0
 static bool
 vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg *vmsg)
 {
@@ -1075,9 +1084,14 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
     }
 

contrib/libvhost-user/libvhost-user.c vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) · line 1094

Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +6 -1
     if (dev->vq[index].kick_fd != -1 && dev->vq[index].handler) {
+        if (dev->set_watch_packed_data) {
+            dev->set_watch_packed_data(dev, dev->vq[index].kick_fd, VU_WATCH_IN,
+                                       dev->co_iface->kick_callback,
+                                       (void *)(long)index);
+        } else {
         dev->set_watch(dev, dev->vq[index].kick_fd, VU_WATCH_IN,
                        vu_kick_cb, (void *)(long)index);
-
+        }

Indentation is off here.

Also, this is almost exactly the same code for both cases. If you generalise things to have a dev->iface->kick_callback that can be initialised with vu_kick_cb in the non-QEMU case, you get rid of this duplication, too.

Quoted 5 lines
contrib/libvhost-user/libvhost-user.c +0 -0
         DPRINT("Waiting for kicks on fd: %d for vq: %d\n",
                dev->vq[index].kick_fd, index);
     }
@@ -1097,8 +1111,14 @@ void vu_set_queue_handler(VuDev *dev, VuVirtq *vq,
     vq->handler = handler;

contrib/libvhost-user/libvhost-user.c void vu_set_queue_handler(VuDev *dev, VuVirtq *vq, · line 1121

Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +6 -0
     if (vq->kick_fd >= 0) {
         if (handler) {
+            if (dev->set_watch_packed_data) {
+                dev->set_watch_packed_data(dev, vq->kick_fd, VU_WATCH_IN,
+                                           dev->co_iface->kick_callback,
+                                           (void *)(long)qidx);
+            } else {
             dev->set_watch(dev, vq->kick_fd, VU_WATCH_IN,
                            vu_kick_cb, (void *)(long)qidx);
+            }

Same as above. (Indentation and duplicated code.)

contrib/libvhost-user/libvhost-user.c vu_deinit(VuDev *dev) · line 1654

Quoted 12 lines
contrib/libvhost-user/libvhost-user.c +5 -0
         } else {
             dev->remove_watch(dev, vq->kick_fd);
         }
@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            /* remove watch for kick_fd
+             * When client process is running in gdb and
+             * quit command is run in gdb, QEMU will still dispatch the event
+             * which will cause segment fault in the callback function
+             */

Reformat this comment to use a consistent line width, maybe like this:

/*
* remove watch for kick_fd.
*
* When client process is running in gdb and quit command is
* run in gdb, QEMU will still dispatch the event which will
* cause segment fault in the callback function */

I'm not sure what the comment wants to tell me: Is this an existing problem in the code that we can run into segfaults, or do we remove the watch to avoid segfaults?

contrib/libvhost-user/libvhost-user.c vu_init(VuDev *dev, · line 1711

Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +2 -1
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }
@@ -1682,7 +1708,7 @@ vu_init(VuDev *dev,
 
     assert(max_queues > 0);
     assert(socket >= 0);
-    assert(set_watch);
+    /* assert(set_watch); */

Don't leave commented code around. Either leave it in, or remove it completely.

I think this one should be left in. If you integrate CoIface into VuDevIface, the assertion will hold true again.

Quoted 14 lines
contrib/libvhost-user/libvhost-user.c +7 -0
     assert(remove_watch);
     assert(iface);
     assert(panic);
@@ -1715,6 +1741,24 @@ vu_init(VuDev *dev,
     return true;
 }
 
+bool
+vu_init_packed_data(VuDev *dev,
+        uint16_t max_queues,
+        int socket,
+        vu_panic_cb panic,
+        vu_set_watch_cb_packed_data set_watch_packed_data,
+        vu_remove_watch_cb remove_watch,

contrib/libvhost-user/libvhost-user.c vu_init(VuDev *dev, · line 1760

Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +10 -0
+        const VuDevIface *iface,
+        const CoIface *co_iface)
+{
+    if (vu_init(dev, max_queues, socket, panic, NULL, remove_watch, iface)) {
+        dev->set_watch_packed_data = set_watch_packed_data;
+        dev->co_iface = co_iface;
+        return true;
+    }
+    return false;
+}

With the integrated VuDevIface, this wrapper becomes unnecessary.

Kevin

author
#4 lore raw

PATCH v4 2/5 generic vhost user server

In reply to Coiby Xu

Sharing QEMU devices via vhost-user protocol

---
 util/Makefile.objs       |   3 +
 util/vhost-user-server.c | 427 +++++++++++++++++++++++++++++++++++++++
 util/vhost-user-server.h |  56 +++++
 3 files changed, 486 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
util/Makefile.objs +3 -0
index 11262aafaf..5e450e501c 100644
@@ -36,6 +36,9 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+ifdef CONFIG_LINUX
+util-obj-y += vhost-user-server.o
+endif
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
util/vhost-user-server.c new file +427 -0

427 lines not on this page · raw message

util/vhost-user-server.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.25.0
#5 lore raw

Re: [PATCH v4 2/5] generic vhost user server

Am 18.02.2020 um 06:07 hat Coiby Xu geschrieben:

Quoted 17 lines

Sharing QEMU devices via vhost-user protocol

---
 util/Makefile.objs       |   3 +
 util/vhost-user-server.c | 427 +++++++++++++++++++++++++++++++++++++++
 util/vhost-user-server.h |  56 +++++
 3 files changed, 486 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
util/vhost-user-server.h new file +0 -0
new file mode 100644
index 0000000000..ff6d3145cd

util/vhost-user-server.h line 8

Quoted 8 lines
util/vhost-user-server.h new file +8 -0
@@ -0,0 +1,56 @@
+#include "io/channel-socket.h"
+#include "io/channel-file.h"
+#include "io/net-listener.h"
+#include "contrib/libvhost-user/libvhost-user.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "qemu/error-report.h"
+
+typedef struct VuClient VuClient;

I find the terminology a bit confusing here: VuClient is really the connection to a single client, but it's part of the server. The name gives the impression as if this were client-side code. (This is something that already tends to confuse me in the NBD code.)

I'm not sure what a better name could be, though. Maybe VuServerConnevtion or VuExportClient or VuExportConnection?

Quoted 7 lines
util/vhost-user-server.h new file +7 -0
+typedef struct VuServer {
+    QIONetListener *listener;
+    AioContext *ctx;
+    QTAILQ_HEAD(, VuClient) clients;
+    void (*device_panic_notifier)(struct VuClient *client) ;
+    int max_queues;
+    const VuDevIface *vu_iface;

util/vhost-user-server.h line 26

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    /*
+     * @ptr_in_device: VuServer pointer memory location in vhost-user device
+     * struct, so later container_of can be used to get device destruct
+     */
+    void *ptr_in_device;
+    bool close;
+} VuServer;
+
+typedef struct kick_info {
+    VuDev *vu_dev;

I suppose this could specifically be VuClient?

Quoted 4 lines
util/vhost-user-server.h new file +4 -0
+    int fd; /*kick fd*/
+    long index; /*queue index*/
+    QIOChannel *ioc; /*I/O channel for kick fd*/
+    QIOChannelFile *fioc; /*underlying data channel for kick fd*/

util/vhost-user-server.h line 40

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    Coroutine *co;
+} kick_info;
+
+struct VuClient {
+    VuDev parent;
+    VuServer *server;
+    QIOChannel *ioc; /* The current I/O channel */
+    QIOChannelSocket *sioc; /* The underlying data channel */
+    Coroutine *co_trip;
+    struct kick_info *kick_info;

If each struct kick_info (btw, QEMU coding style requires CamelCase) has exactly one VuClient and each VuClient has exactly on kick_info, should this be a single struct containing both?

[ Coming back from reading the code below - it's because this is in fact an array. This should be made clear in the definition. ]

Quoted 6 lines
util/vhost-user-server.h new file +6 -0
+    QTAILQ_ENTRY(VuClient) next;
+    bool closed;
+};
+
+
+VuServer *vhost_user_server_start(uint16_t max_queues,

util/vhost-user-server.h line 56

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+                                  SocketAddress *unix_socket,
+                                  AioContext *ctx,
+                                  void *server_ptr,
+                                  void *device_panic_notifier,
+                                  const VuDevIface *vu_iface,
+                                  Error **errp);
+
+void vhost_user_server_stop(VuServer *server);
+
+void change_vu_context(AioContext *ctx, VuServer *server);

Let's call this vhost_user_server_set_aio_context() for consistency.

Quoted 57 lines
util/Makefile.objs +3 -0
index 11262aafaf..5e450e501c 100644
@@ -36,6 +36,9 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+ifdef CONFIG_LINUX
+util-obj-y += vhost-user-server.o
+endif
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
util/vhost-user-server.c new file +37 -0
new file mode 100644
index 0000000000..70ff6d6701
@@ -0,0 +1,427 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include <sys/eventfd.h>
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+    int i;
+    for (i = 0; i < vmsg->fd_num; i++) {
+        close(vmsg->fds[i]);
+    }
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+    int i;
+    for (i = 0; i < vmsg->fd_num; i++) {
+        qemu_set_nonblock(vmsg->fds[i]);
+    }
+}
+
+
+static void close_client(VuClient *client)
+{
+    vu_deinit(&client->parent);
+    client->sioc = NULL;
+    object_unref(OBJECT(client->ioc));
+    client->closed = true;
+

util/vhost-user-server.c line 47

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+    if (buf) {
+        error_report("vu_panic: %s", buf);
+    }
+
+    VuClient *client = container_of(vu_dev, VuClient, parent);
+    VuServer *server = client->server;

Please put declarations at the start of the block.

Quoted 22 lines
util/vhost-user-server.c new file +22 -0
+    if (!client->closed) {
+        close_client(client);
+        QTAILQ_REMOVE(&server->clients, client, next);
+    }
+
+    if (server->device_panic_notifier) {
+        server->device_panic_notifier(client);
+    }
+}
+
+
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    struct iovec iov = {
+        .iov_base = (char *)vmsg,
+        .iov_len = VHOST_USER_HDR_SIZE,
+    };
+    int rc, read_bytes = 0;
+    /*
+     * VhostUserMsg is a packed structure, gcc will complain about passing

util/vhost-user-server.c line 79

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+     * pointer to a packed structure member if we pass &VhostUserMsg.fd_num
+     * and &VhostUserMsg.fds directly when calling qio_channel_readv_full,
+     * thus two temporary variables nfds and fds are used here.
+     */
+    size_t nfds = 0, nfds_t = 0;
+    int *fds = NULL, *fds_t = NULL;
+    VuClient *client = container_of(vu_dev, VuClient, parent);
+    QIOChannel *ioc = client->ioc;
+
+    Error *erp;

The convention is to call this local_err. It should be initialised as NULL.

util/vhost-user-server.c line 92

Quoted 13 lines
util/vhost-user-server.c new file +13 -0
+    assert(qemu_in_coroutine());
+    do {
+        /*
+         * qio_channel_readv_full may have short reads, keeping calling it
+         * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+         */
+        rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &erp);
+        if (rc < 0) {
+            if (rc == QIO_CHANNEL_ERR_BLOCK) {
+                qio_channel_yield(ioc, G_IO_IN);
+                continue;
+            } else {
+                error_report("Error while recvmsg: %s", strerror(errno));

I don't think, qio_channel_*() promise anything about the value in errno. (They also don't promise to use recvmsg().)

Instead, use error_report_err() because erp contains the real error message.

Quoted 23 lines
util/vhost-user-server.c new file +23 -0
+                return false;
+            }
+        }
+        read_bytes += rc;
+        fds = g_renew(int, fds_t, nfds + nfds_t);
+        memcpy(fds + nfds, fds_t, nfds_t);
+        nfds += nfds_t;
+        if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+            break;
+        }
+    } while (true);
+
+    vmsg->fd_num = nfds;
+    memcpy(vmsg->fds, fds, nfds * sizeof(int));
+    g_free(fds);
+    /* qio_channel_readv_full will make socket fds blocking, unblock them */
+    vmsg_unblock_fds(vmsg);
+    if (vmsg->size > sizeof(vmsg->payload)) {
+        error_report("Error: too big message request: %d, "
+                     "size: vmsg->size: %u, "
+                     "while sizeof(vmsg->payload) = %zu",
+                     vmsg->request, vmsg->size, sizeof(vmsg->payload));
+        goto fail;

util/vhost-user-server.c line 125

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+    }
+
+    struct iovec iov_payload = {
+        .iov_base = (char *)&vmsg->payload,
+        .iov_len = vmsg->size,
+    };
+    if (vmsg->size) {
+        rc = qio_channel_readv_all_eof(ioc, &iov_payload, 1, &erp);
+        if (rc == -1) {
+            error_report("Error while reading: %s", strerror(errno));

error_report_err() again.

Quoted 15 lines
util/vhost-user-server.c new file +15 -0
+            goto fail;
+        }
+    }
+
+    return true;
+
+fail:
+    vmsg_close_fds(vmsg);
+
+    return false;
+}
+
+
+static coroutine_fn void vu_client_next_trip(VuClient *client);
+

util/vhost-user-server.c line 150

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+static coroutine_fn void vu_client_trip(void *opaque)
+{
+    VuClient *client = opaque;
+
+    vu_dispatch(&client->parent);
+    client->co_trip = NULL;
+    if (!client->closed) {
+        vu_client_next_trip(client);
+    }
+}

The last part is very untypical coroutine code: It says that we want to spawn a new coroutine with vu_client_trip() as its entry point, and then terminates the current one.

Why don't we just put the whole thing in a while (!client->closed) loop and stay in the same coroutine instead of terminating the old one and starting a new one all the time?

util/vhost-user-server.c line 164

Quoted 13 lines
util/vhost-user-server.c new file +13 -0
+static coroutine_fn void vu_client_next_trip(VuClient *client)
+{
+    if (!client->co_trip) {
+        client->co_trip = qemu_coroutine_create(vu_client_trip, client);
+        aio_co_schedule(client->ioc->ctx, client->co_trip);
+    }
+}
+
+static void vu_client_start(VuClient *client)
+{
+    client->co_trip = qemu_coroutine_create(vu_client_trip, client);
+    aio_co_enter(client->ioc->ctx, client->co_trip);
+}

This is essentially a duplicate of vu_client_next_trip(). The only place where it is called (vu_accept()) knows that client->co_trip is already NULL, so it could just call vu_client_next_trip().

Or in fact, if vu_client_trip() gets turned into a loop, it's vu_client_next_trip() that becomes unnecessary.

Quoted 21 lines
util/vhost-user-server.c new file +21 -0
+static void coroutine_fn vu_kick_cb_next(VuClient *client,
+                                          kick_info *data);
+
+static void coroutine_fn vu_kick_cb(void *opaque)
+{
+    kick_info *data = (kick_info *) opaque;
+    int index = data->index;
+    VuDev *dev = data->vu_dev;
+    VuClient *client;
+    client = container_of(dev, VuClient, parent);
+    VuVirtq *vq = &dev->vq[index];
+    int sock = vq->kick_fd;
+    if (sock == -1) {
+        return;
+    }
+    assert(sock == data->fd);
+    eventfd_t kick_data;
+    ssize_t rc;
+    /*
+     * When eventfd is closed, the revent is POLLNVAL (=G_IO_NVAL) and
+     * reading eventfd will return errno=EBADF (Bad file number).

util/vhost-user-server.c line 196

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+     * Calling qio_channel_yield(ioc, G_IO_IN) will set reading handler
+     * for QIOChannel, but aio_dispatch_handlers will only dispatch
+     * G_IO_IN | G_IO_HUP | G_IO_ERR revents while ignoring
+     * G_IO_NVAL (POLLNVAL) revents.
+     *
+     * Thus when eventfd is closed by vhost-user client, QEMU will ignore
+     * G_IO_NVAL and keeping polling by repeatedly calling qemu_poll_ns which
+     * will lead to 100% CPU usage.
+     *
+     * To aovid this issue, make sure set_watch and remove_watch use the same

s/aovid/avoid/

util/vhost-user-server.c line 201

Quoted 5 lines
util/vhost-user-server.c new file +5 -0
+     * AIOContext for QIOChannel. Thus remove_watch will eventually succefully
+     * remove eventfd from the set of file descriptors polled for
+     * corresponding GSource.
+     */
+    rc = read(sock, &kick_data, sizeof(eventfd_t));

Why not a QIOChannel function like for vu_message_read() above?

util/vhost-user-server.c line 213

Quoted 12 lines
util/vhost-user-server.c new file +12 -0
+    if (rc != sizeof(eventfd_t)) {
+        if (errno == EAGAIN) {
+            qio_channel_yield(data->ioc, G_IO_IN);
+        } else if (errno != EINTR) {
+            data->co = NULL;
+            return;
+        }
+    } else {
+        vq->handler(dev, index);
+    }
+    data->co = NULL;
+    vu_kick_cb_next(client, data);

This can be a loop, too, instead of terminating the coroutine and starting a new one for the same function.

util/vhost-user-server.c line 224

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+}
+
+static void coroutine_fn vu_kick_cb_next(VuClient *client,
+                                          kick_info *cb_data)
+{
+    if (!cb_data->co) {
+        cb_data->co = qemu_coroutine_create(vu_kick_cb, cb_data);
+        aio_co_schedule(client->ioc->ctx, cb_data->co);
+    }
+}

Kevin

author
#6 lore raw

Re: [PATCH v4 2/5] generic vhost user server

Quoted 13 lines
Quoted 12 lines
util/vhost-user-server.c new file +12 -0
+static coroutine_fn void vu_client_next_trip(VuClient *client);
+
+static coroutine_fn void vu_client_trip(void *opaque)
+{
+    VuClient *client = opaque;
+
+    vu_dispatch(&client->parent);
+    client->co_trip = NULL;
+    if (!client->closed) {
+        vu_client_next_trip(client);
+    }
+}
Quoted 1 line
Quoted 1 line

The last part is very untypical coroutine code: It says that we want to

spawn a new coroutine with vu_client_trip() as its entry point, and then terminates the current one.

Quoted 1 line
Quoted 1 line

Why don't we just put the whole thing in a while (!client->closed) loop

and stay in the same coroutine instead of terminating the old one and starting a new one all the time?

Quoted 7 lines
Quoted 7 lines
util/vhost-user-server.c new file +7 -0
+static coroutine_fn void vu_client_next_trip(VuClient *client)
+{
+    if (!client->co_trip) {
+        client->co_trip = qemu_coroutine_create(vu_client_trip, client);
+        aio_co_schedule(client->ioc->ctx, client->co_trip);
+    }
+}
Quoted 8 lines

util/vhost-user-server.c line 164

Quoted 6 lines
util/vhost-user-server.c new file +6 -0
+
+static void vu_client_start(VuClient *client)
+{
+    client->co_trip = qemu_coroutine_create(vu_client_trip, client);
+    aio_co_enter(client->ioc->ctx, client->co_trip);
+}

This is essentially a duplicate of vu_client_next_trip(). The only

place where it is called (vu_accept()) knows that client->co_trip is already NULL, so it could just call vu_client_next_trip().

Quoted 2 lines

Or in fact, if vu_client_trip() gets turned into a loop, it's vu_client_next_trip() that becomes unnecessary.

This part of code is an imitation of nbd_client_trip in nbd/server.c. I think the reason to repeatedly create/start/terminate vu_client_trip is to support BlockBackendAioNotifier. In v5, I will keep running the spawned coroutine in a loop until being informed of the change of AioContext of the block device backend, i.e. vu_client_trip will only be restarted when the block device backend is attached to a different AiOContext.

Quoted 13 lines

util/vhost-user-server.c line 213

Quoted 12 lines
util/vhost-user-server.c new file +12 -0
+    if (rc != sizeof(eventfd_t)) {
+        if (errno == EAGAIN) {
+            qio_channel_yield(data->ioc, G_IO_IN);
+        } else if (errno != EINTR) {
+            data->co = NULL;
+            return;
+        }
+    } else {
+        vq->handler(dev, index);
+    }
+    data->co = NULL;
+    vu_kick_cb_next(client, data);
Quoted 1 line

This can be a loop, too, instead of terminating the coroutine and

starting a new one for the same function.

In v5, I plan to use aio_set_fd_handler to set a read hander which is a wrapper for vu_kick_cb to deal with kick events since eventfd doesn't have the short read issue like socket. Thus vu_kick_cb in libvhost-user can be re-used. My only concern is if this could lead to worse performance in comparison to keep reading from eventfd until getting EAGAIN errno.

Quoted 385 lines · Kevin Wolf

Am 18.02.2020 um 06:07 hat Coiby Xu geschrieben:

Quoted 17 lines

Sharing QEMU devices via vhost-user protocol

---
 util/Makefile.objs       |   3 +
 util/vhost-user-server.c | 427 +++++++++++++++++++++++++++++++++++++++
 util/vhost-user-server.h |  56 +++++
 3 files changed, 486 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
util/vhost-user-server.h new file +0 -0
new file mode 100644
index 0000000000..ff6d3145cd

util/vhost-user-server.h line 8

Quoted 8 lines
util/vhost-user-server.h new file +8 -0
@@ -0,0 +1,56 @@
+#include "io/channel-socket.h"
+#include "io/channel-file.h"
+#include "io/net-listener.h"
+#include "contrib/libvhost-user/libvhost-user.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "qemu/error-report.h"
+
+typedef struct VuClient VuClient;

I find the terminology a bit confusing here: VuClient is really the connection to a single client, but it's part of the server. The name gives the impression as if this were client-side code. (This is something that already tends to confuse me in the NBD code.)

I'm not sure what a better name could be, though. Maybe VuServerConnevtion or VuExportClient or VuExportConnection?

Quoted 7 lines
util/vhost-user-server.h new file +7 -0
+typedef struct VuServer {
+    QIONetListener *listener;
+    AioContext *ctx;
+    QTAILQ_HEAD(, VuClient) clients;
+    void (*device_panic_notifier)(struct VuClient *client) ;
+    int max_queues;
+    const VuDevIface *vu_iface;

util/vhost-user-server.h line 26

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    /*
+     * @ptr_in_device: VuServer pointer memory location in vhost-user device
+     * struct, so later container_of can be used to get device destruct
+     */
+    void *ptr_in_device;
+    bool close;
+} VuServer;
+
+typedef struct kick_info {
+    VuDev *vu_dev;

I suppose this could specifically be VuClient?

Quoted 4 lines
util/vhost-user-server.h new file +4 -0
+    int fd; /*kick fd*/
+    long index; /*queue index*/
+    QIOChannel *ioc; /*I/O channel for kick fd*/
+    QIOChannelFile *fioc; /*underlying data channel for kick fd*/

util/vhost-user-server.h line 40

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    Coroutine *co;
+} kick_info;
+
+struct VuClient {
+    VuDev parent;
+    VuServer *server;
+    QIOChannel *ioc; /* The current I/O channel */
+    QIOChannelSocket *sioc; /* The underlying data channel */
+    Coroutine *co_trip;
+    struct kick_info *kick_info;

If each struct kick_info (btw, QEMU coding style requires CamelCase) has exactly one VuClient and each VuClient has exactly on kick_info, should this be a single struct containing both?

[ Coming back from reading the code below - it's because this is in fact an array. This should be made clear in the definition. ]

Quoted 6 lines
util/vhost-user-server.h new file +6 -0
+    QTAILQ_ENTRY(VuClient) next;
+    bool closed;
+};
+
+
+VuServer *vhost_user_server_start(uint16_t max_queues,

util/vhost-user-server.h line 56

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+                                  SocketAddress *unix_socket,
+                                  AioContext *ctx,
+                                  void *server_ptr,
+                                  void *device_panic_notifier,
+                                  const VuDevIface *vu_iface,
+                                  Error **errp);
+
+void vhost_user_server_stop(VuServer *server);
+
+void change_vu_context(AioContext *ctx, VuServer *server);

Let's call this vhost_user_server_set_aio_context() for consistency.

Quoted 57 lines
util/Makefile.objs +3 -0
index 11262aafaf..5e450e501c 100644
@@ -36,6 +36,9 @@ util-obj-y += readline.o
 util-obj-y += rcu.o
 util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+ifdef CONFIG_LINUX
+util-obj-y += vhost-user-server.o
+endif
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
util/vhost-user-server.c new file +37 -0
new file mode 100644
index 0000000000..70ff6d6701
@@ -0,0 +1,427 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include <sys/eventfd.h>
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+    int i;
+    for (i = 0; i < vmsg->fd_num; i++) {
+        close(vmsg->fds[i]);
+    }
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+    int i;
+    for (i = 0; i < vmsg->fd_num; i++) {
+        qemu_set_nonblock(vmsg->fds[i]);
+    }
+}
+
+
+static void close_client(VuClient *client)
+{
+    vu_deinit(&client->parent);
+    client->sioc = NULL;
+    object_unref(OBJECT(client->ioc));
+    client->closed = true;
+

util/vhost-user-server.c line 47

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+    if (buf) {
+        error_report("vu_panic: %s", buf);
+    }
+
+    VuClient *client = container_of(vu_dev, VuClient, parent);
+    VuServer *server = client->server;

Please put declarations at the start of the block.

Quoted 22 lines
util/vhost-user-server.c new file +22 -0
+    if (!client->closed) {
+        close_client(client);
+        QTAILQ_REMOVE(&server->clients, client, next);
+    }
+
+    if (server->device_panic_notifier) {
+        server->device_panic_notifier(client);
+    }
+}
+
+
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    struct iovec iov = {
+        .iov_base = (char *)vmsg,
+        .iov_len = VHOST_USER_HDR_SIZE,
+    };
+    int rc, read_bytes = 0;
+    /*
+     * VhostUserMsg is a packed structure, gcc will complain about passing

util/vhost-user-server.c line 79

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+     * pointer to a packed structure member if we pass &VhostUserMsg.fd_num
+     * and &VhostUserMsg.fds directly when calling qio_channel_readv_full,
+     * thus two temporary variables nfds and fds are used here.
+     */
+    size_t nfds = 0, nfds_t = 0;
+    int *fds = NULL, *fds_t = NULL;
+    VuClient *client = container_of(vu_dev, VuClient, parent);
+    QIOChannel *ioc = client->ioc;
+
+    Error *erp;

The convention is to call this local_err. It should be initialised as NULL.

util/vhost-user-server.c line 92

Quoted 13 lines
util/vhost-user-server.c new file +13 -0
+    assert(qemu_in_coroutine());
+    do {
+        /*
+         * qio_channel_readv_full may have short reads, keeping calling it
+         * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+         */
+        rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &erp);
+        if (rc < 0) {
+            if (rc == QIO_CHANNEL_ERR_BLOCK) {
+                qio_channel_yield(ioc, G_IO_IN);
+                continue;
+            } else {
+                error_report("Error while recvmsg: %s", strerror(errno));

I don't think, qio_channel_*() promise anything about the value in errno. (They also don't promise to use recvmsg().)

Instead, use error_report_err() because erp contains the real error message.

Quoted 23 lines
util/vhost-user-server.c new file +23 -0
+                return false;
+            }
+        }
+        read_bytes += rc;
+        fds = g_renew(int, fds_t, nfds + nfds_t);
+        memcpy(fds + nfds, fds_t, nfds_t);
+        nfds += nfds_t;
+        if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+            break;
+        }
+    } while (true);
+
+    vmsg->fd_num = nfds;
+    memcpy(vmsg->fds, fds, nfds * sizeof(int));
+    g_free(fds);
+    /* qio_channel_readv_full will make socket fds blocking, unblock them */
+    vmsg_unblock_fds(vmsg);
+    if (vmsg->size > sizeof(vmsg->payload)) {
+        error_report("Error: too big message request: %d, "
+                     "size: vmsg->size: %u, "
+                     "while sizeof(vmsg->payload) = %zu",
+                     vmsg->request, vmsg->size, sizeof(vmsg->payload));
+        goto fail;

util/vhost-user-server.c line 125

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+    }
+
+    struct iovec iov_payload = {
+        .iov_base = (char *)&vmsg->payload,
+        .iov_len = vmsg->size,
+    };
+    if (vmsg->size) {
+        rc = qio_channel_readv_all_eof(ioc, &iov_payload, 1, &erp);
+        if (rc == -1) {
+            error_report("Error while reading: %s", strerror(errno));

error_report_err() again.

Quoted 15 lines
util/vhost-user-server.c new file +15 -0
+            goto fail;
+        }
+    }
+
+    return true;
+
+fail:
+    vmsg_close_fds(vmsg);
+
+    return false;
+}
+
+
+static coroutine_fn void vu_client_next_trip(VuClient *client);
+

util/vhost-user-server.c line 150

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+static coroutine_fn void vu_client_trip(void *opaque)
+{
+    VuClient *client = opaque;
+
+    vu_dispatch(&client->parent);
+    client->co_trip = NULL;
+    if (!client->closed) {
+        vu_client_next_trip(client);
+    }
+}

The last part is very untypical coroutine code: It says that we want to spawn a new coroutine with vu_client_trip() as its entry point, and then terminates the current one.

Why don't we just put the whole thing in a while (!client->closed) loop and stay in the same coroutine instead of terminating the old one and starting a new one all the time?

util/vhost-user-server.c line 164

Quoted 13 lines
util/vhost-user-server.c new file +13 -0
+static coroutine_fn void vu_client_next_trip(VuClient *client)
+{
+    if (!client->co_trip) {
+        client->co_trip = qemu_coroutine_create(vu_client_trip, client);
+        aio_co_schedule(client->ioc->ctx, client->co_trip);
+    }
+}
+
+static void vu_client_start(VuClient *client)
+{
+    client->co_trip = qemu_coroutine_create(vu_client_trip, client);
+    aio_co_enter(client->ioc->ctx, client->co_trip);
+}

This is essentially a duplicate of vu_client_next_trip(). The only place where it is called (vu_accept()) knows that client->co_trip is already NULL, so it could just call vu_client_next_trip().

Or in fact, if vu_client_trip() gets turned into a loop, it's vu_client_next_trip() that becomes unnecessary.

Quoted 21 lines
util/vhost-user-server.c new file +21 -0
+static void coroutine_fn vu_kick_cb_next(VuClient *client,
+                                          kick_info *data);
+
+static void coroutine_fn vu_kick_cb(void *opaque)
+{
+    kick_info *data = (kick_info *) opaque;
+    int index = data->index;
+    VuDev *dev = data->vu_dev;
+    VuClient *client;
+    client = container_of(dev, VuClient, parent);
+    VuVirtq *vq = &dev->vq[index];
+    int sock = vq->kick_fd;
+    if (sock == -1) {
+        return;
+    }
+    assert(sock == data->fd);
+    eventfd_t kick_data;
+    ssize_t rc;
+    /*
+     * When eventfd is closed, the revent is POLLNVAL (=G_IO_NVAL) and
+     * reading eventfd will return errno=EBADF (Bad file number).

util/vhost-user-server.c line 196

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+     * Calling qio_channel_yield(ioc, G_IO_IN) will set reading handler
+     * for QIOChannel, but aio_dispatch_handlers will only dispatch
+     * G_IO_IN | G_IO_HUP | G_IO_ERR revents while ignoring
+     * G_IO_NVAL (POLLNVAL) revents.
+     *
+     * Thus when eventfd is closed by vhost-user client, QEMU will ignore
+     * G_IO_NVAL and keeping polling by repeatedly calling qemu_poll_ns which
+     * will lead to 100% CPU usage.
+     *
+     * To aovid this issue, make sure set_watch and remove_watch use the same

s/aovid/avoid/

util/vhost-user-server.c line 201

Quoted 5 lines
util/vhost-user-server.c new file +5 -0
+     * AIOContext for QIOChannel. Thus remove_watch will eventually succefully
+     * remove eventfd from the set of file descriptors polled for
+     * corresponding GSource.
+     */
+    rc = read(sock, &kick_data, sizeof(eventfd_t));

Why not a QIOChannel function like for vu_message_read() above?

util/vhost-user-server.c line 213

Quoted 12 lines
util/vhost-user-server.c new file +12 -0
+    if (rc != sizeof(eventfd_t)) {
+        if (errno == EAGAIN) {
+            qio_channel_yield(data->ioc, G_IO_IN);
+        } else if (errno != EINTR) {
+            data->co = NULL;
+            return;
+        }
+    } else {
+        vq->handler(dev, index);
+    }
+    data->co = NULL;
+    vu_kick_cb_next(client, data);

This can be a loop, too, instead of terminating the coroutine and starting a new one for the same function.

util/vhost-user-server.c line 224

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+}
+
+static void coroutine_fn vu_kick_cb_next(VuClient *client,
+                                          kick_info *cb_data)
+{
+    if (!cb_data->co) {
+        cb_data->co = qemu_coroutine_create(vu_kick_cb, cb_data);
+        aio_co_schedule(client->ioc->ctx, cb_data->co);
+    }
+}

Kevin

Signature · 2 lines
Best regards,
Coiby
author
#7 lore raw

PATCH v4 3/5 vhost-user block device backend server

In reply to Coiby Xu

By making use of libvhost, multiple block device drives can be exported and each drive can serve multiple clients simultaneously. Since vhost-user-server needs a block drive to be created first, delay the creation of this object.

---
 Makefile.target                  |   1 +
 backends/Makefile.objs           |   2 +
 backends/vhost-user-blk-server.c | 718 +++++++++++++++++++++++++++++++
 backends/vhost-user-blk-server.h |  21 +
 vl.c                             |   4 +
 5 files changed, 746 insertions(+)
 create mode 100644 backends/vhost-user-blk-server.c
 create mode 100644 backends/vhost-user-blk-server.h
Makefile.target +1 -0
index 6e61f607b1..8c6c01eb3a 100644
@@ -159,6 +159,7 @@ obj-y += monitor/
 obj-y += qapi/
 obj-y += memory.o
 obj-y += memory_mapping.o
+obj-$(CONFIG_LINUX) += ../contrib/libvhost-user/libvhost-user.o
 obj-y += migration/ram.o
 LIBS := $(libs_softmmu) $(LIBS)
 
backends/Makefile.objs +2 -0
index 28a847cd57..4e7be731e0 100644
@@ -14,6 +14,8 @@ common-obj-y += cryptodev-vhost.o
 common-obj-$(CONFIG_VHOST_CRYPTO) += cryptodev-vhost-user.o
 endif
 
+common-obj-$(CONFIG_LINUX) += vhost-user-blk-server.o
+
 common-obj-$(call land,$(CONFIG_VHOST_USER),$(CONFIG_VIRTIO)) += vhost-user.o
 
 common-obj-$(CONFIG_LINUX) += hostmem-memfd.o
backends/vhost-user-blk-server.c new file +718 -0

718 lines not on this page · raw message

backends/vhost-user-blk-server.h new file +21 -0
new file mode 100644
index 0000000000..03381d11f4
@@ -0,0 +1,21 @@
+#include "util/vhost-user-server.h"
+typedef struct VuBlockDev VuBlockDev;
+#define TYPE_VHOST_USER_BLK_SERVER "vhost-user-blk-server"
+#define VHOST_USER_BLK_SERVER(obj) \
+   OBJECT_CHECK(VuBlockDev, obj, TYPE_VHOST_USER_BLK_SERVER)
+
+/* vhost user block device */
+struct VuBlockDev {
+    Object parent_obj;
+    char *node_name;
+    SocketAddress *addr;
+    bool exit_when_panic;
+    AioContext *ctx;
+    VuServer *vu_server;
+    uint32_t blk_size;
+    BlockBackend *backend;
+    QIOChannelSocket *sioc;
+    QTAILQ_ENTRY(VuBlockDev) next;
+    struct virtio_blk_config blkcfg;
+    bool writable;
+};
vl.c +4 -0
index 794f2e5733..0332ab70da 100644
@@ -2538,6 +2538,10 @@ static bool object_create_initial(const char *type, QemuOpts *opts)
     }
 #endif
 
+    /* Reason: vhost-user-server property "name" */
+    if (g_str_equal(type, "vhost-user-blk-server")) {
+        return false;
+    }
     /*
      * Reason: filter-* property "netdev" etc.
      */
Signature · 1 line
2.25.0
#8 lore raw

Re: [PATCH v4 3/5] vhost-user block device backend server

Am 18.02.2020 um 06:07 hat Coiby Xu geschrieben:

backends/vhost-user-blk-server.c

Quoted 46 lines

By making use of libvhost, multiple block device drives can be exported and each drive can serve multiple clients simultaneously. Since vhost-user-server needs a block drive to be created first, delay the creation of this object.

---
 Makefile.target                  |   1 +
 backends/Makefile.objs           |   2 +
 backends/vhost-user-blk-server.c | 718 +++++++++++++++++++++++++++++++
 backends/vhost-user-blk-server.h |  21 +
 vl.c                             |   4 +
 5 files changed, 746 insertions(+)
 create mode 100644 backends/vhost-user-blk-server.c
 create mode 100644 backends/vhost-user-blk-server.h
Makefile.target +1 -0
index 6e61f607b1..8c6c01eb3a 100644
@@ -159,6 +159,7 @@ obj-y += monitor/
 obj-y += qapi/
 obj-y += memory.o
 obj-y += memory_mapping.o
+obj-$(CONFIG_LINUX) += ../contrib/libvhost-user/libvhost-user.o
 obj-y += migration/ram.o
 LIBS := $(libs_softmmu) $(LIBS)
 
backends/Makefile.objs +2 -0
index 28a847cd57..4e7be731e0 100644
@@ -14,6 +14,8 @@ common-obj-y += cryptodev-vhost.o
 common-obj-$(CONFIG_VHOST_CRYPTO) += cryptodev-vhost-user.o
 endif
 
+common-obj-$(CONFIG_LINUX) += vhost-user-blk-server.o
+
 common-obj-$(call land,$(CONFIG_VHOST_USER),$(CONFIG_VIRTIO)) += vhost-user.o
 
 common-obj-$(CONFIG_LINUX) += hostmem-memfd.o
backends/vhost-user-blk-server.c new file +0 -0
new file mode 100644
index 0000000000..1bf7f7b544

Please move this to block/export/vhost-user-blk.c. This will automatically have it covered as a block layer component in MAINTAINERS, and it will provide a place where other exports can be put, too.

Quoted 116 lines
backends/vhost-user-blk-server.c new file +115 -0
@@ -0,0 +1,718 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+
+enum {
+    VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+    unsigned char status;
+};
+
+static QTAILQ_HEAD(, VuBlockDev) vu_block_devs =
+                                 QTAILQ_HEAD_INITIALIZER(vu_block_devs);
+
+
+typedef struct VuBlockReq {
+    VuVirtqElement *elem;
+    int64_t sector_num;
+    size_t size;
+    struct virtio_blk_inhdr *in;
+    struct virtio_blk_outhdr out;
+    VuClient *client;
+    struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+    VuDev *vu_dev = &req->client->parent;
+
+    /* IO size with 1 extra status byte */
+    vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+    vu_queue_notify(vu_dev, req->vq);
+
+    if (req->elem) {
+        free(req->elem);
+    }
+
+    g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_client(VuClient *client)
+{
+    return container_of(client->server->ptr_in_device, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+                              uint32_t iovcnt, uint32_t type)
+{
+    struct virtio_blk_discard_write_zeroes desc;
+    ssize_t size = iov_to_buf(iov, iovcnt, 0, &desc, sizeof(desc));
+    if (unlikely(size != sizeof(desc))) {
+        error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+        return -EINVAL;
+    }
+
+    VuBlockDev *vdev_blk = get_vu_block_device_by_client(req->client);
+    uint64_t range[2] = { le64toh(desc.sector) << 9,
+                          le32toh(desc.num_sectors) << 9 };
+    if (type == VIRTIO_BLK_T_DISCARD) {
+        if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+            return 0;
+        }
+    } else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+        if (blk_co_pwrite_zeroes(vdev_blk->backend,
+                                 range[0], range[1], 0) == 0) {
+            return 0;
+        }
+    }
+
+    return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+    VuBlockDev *vdev_blk = get_vu_block_device_by_client(req->client);
+    BlockBackend *backend = vdev_blk->backend;
+    blk_co_flush(backend);
+}
+
+
+static int coroutine_fn vu_block_virtio_process_req(VuClient *client,
+                                                    VuVirtq *vq)
+{
+    VuDev *vu_dev = &client->parent;
+    VuVirtqElement *elem;
+    uint32_t type;
+    VuBlockReq *req;
+
+    VuBlockDev *vdev_blk = get_vu_block_device_by_client(client);
+    BlockBackend *backend = vdev_blk->backend;
+    elem = vu_queue_pop(vu_dev, vq, sizeof(VuVirtqElement) +
+                                    sizeof(VuBlockReq));
+    if (!elem) {
+        return -1;
+    }
+
+    struct iovec *in_iov = elem->in_sg;
+    struct iovec *out_iov = elem->out_sg;
+    unsigned in_num = elem->in_num;
+    unsigned out_num = elem->out_num;
+    /* refer to hw/block/virtio_blk.c */

backends/vhost-user-blk-server.c line 125

Quoted 10 lines
backends/vhost-user-blk-server.c new file +10 -0
+    if (elem->out_num < 1 || elem->in_num < 1) {
+        error_report("virtio-blk request missing headers");
+        free(elem);
+        return -EINVAL;
+    }
+
+    req = g_new0(VuBlockReq, 1);
+    req->client = client;
+    req->vq = vq;
+    req->elem = elem;

You can keep req on the stack, it will be freed before this function returns.

Quoted 30 lines
backends/vhost-user-blk-server.c new file +30 -0
+    if (unlikely(iov_to_buf(out_iov, out_num, 0, &req->out,
+                            sizeof(req->out)) != sizeof(req->out))) {
+        error_report("virtio-blk request outhdr too short");
+        goto err;
+    }
+
+    iov_discard_front(&out_iov, &out_num, sizeof(req->out));
+
+    if (in_iov[in_num - 1].iov_len < sizeof(struct virtio_blk_inhdr)) {
+        error_report("virtio-blk request inhdr too short");
+        goto err;
+    }
+
+    /* We always touch the last byte, so just see how big in_iov is.  */
+    req->in = (void *)in_iov[in_num - 1].iov_base
+              + in_iov[in_num - 1].iov_len
+              - sizeof(struct virtio_blk_inhdr);
+    iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
+
+
+    type = le32toh(req->out.type);
+    switch (type & ~VIRTIO_BLK_T_BARRIER) {
+    case VIRTIO_BLK_T_IN:
+    case VIRTIO_BLK_T_OUT: {
+        ssize_t ret = 0;
+        bool is_write = type & VIRTIO_BLK_T_OUT;
+        req->sector_num = le64toh(req->out.sector);
+
+        int64_t offset = req->sector_num * vdev_blk->blk_size;
+        QEMUIOVector *qiov = g_new0(QEMUIOVector, 1);

backends/vhost-user-blk-server.c line 166

Quoted 10 lines
backends/vhost-user-blk-server.c new file +10 -0
+        if (is_write) {
+            qemu_iovec_init_external(qiov, out_iov, out_num);
+            ret = blk_co_pwritev(backend, offset, qiov->size,
+                                 qiov, 0);
+        } else {
+            qemu_iovec_init_external(qiov, in_iov, in_num);
+            ret = blk_co_preadv(backend, offset, qiov->size,
+                                 qiov, 0);
+        }
+        aio_wait_kick();

What purpose does this aio_wait_kick() serve? Usually you do this if you want to signal completion fo something to a different thread, but I don't see that we changed any control state that could be visible to other threads.

Quoted 30 lines
backends/vhost-user-blk-server.c new file +30 -0
+        if (ret >= 0) {
+            req->in->status = VIRTIO_BLK_S_OK;
+        } else {
+            req->in->status = VIRTIO_BLK_S_IOERR;
+        }
+        vu_block_req_complete(req);
+        break;
+    }
+    case VIRTIO_BLK_T_FLUSH:
+        vu_block_flush(req);
+        req->in->status = VIRTIO_BLK_S_OK;
+        vu_block_req_complete(req);
+        break;
+    case VIRTIO_BLK_T_GET_ID: {
+        size_t size = MIN(iov_size(&elem->in_sg[0], in_num),
+                          VIRTIO_BLK_ID_BYTES);
+        snprintf(elem->in_sg[0].iov_base, size, "%s", "vhost_user_blk_server");
+        req->in->status = VIRTIO_BLK_S_OK;
+        req->size = elem->in_sg[0].iov_len;
+        vu_block_req_complete(req);
+        break;
+    }
+    case VIRTIO_BLK_T_DISCARD:
+    case VIRTIO_BLK_T_WRITE_ZEROES: {
+        int rc;
+        rc = vu_block_discard_write_zeroes(req, &elem->out_sg[1],
+                                           out_num, type);
+        if (rc == 0) {
+            req->in->status = VIRTIO_BLK_S_OK;
+        } else {

backends/vhost-user-blk-server.c line 206

Quoted 10 lines
backends/vhost-user-blk-server.c new file +10 -0
+            req->in->status = VIRTIO_BLK_S_IOERR;
+        }
+        vu_block_req_complete(req);
+        break;
+    }
+    default:
+        req->in->status = VIRTIO_BLK_S_UNSUPP;
+        vu_block_req_complete(req);
+        break;
+    }

You have vu_block_req_complete() as the last thing in every branch. You could have it just once after the switch block.

backends/vhost-user-blk-server.c line 214

Quoted 7 lines
backends/vhost-user-blk-server.c new file +7 -0
+    return 0;
+
+err:
+    free(elem);
+    g_free(req);
+    return -EINVAL;
+}

Okay, so vu_block_virtio_process_req() takes a single request from the virtqueue and processes it. This makes sense as a building block, but it doesn't allow parallelism yet. I expect to see the creation of multiple coroutines below that can run in parallel.

Quoted 5 lines
backends/vhost-user-blk-server.c new file +5 -0
+
+static void vu_block_process_vq(VuDev *vu_dev, int idx)
+{
+    VuClient *client;
+    VuVirtq *vq;

backends/vhost-user-blk-server.c line 230

Quoted 10 lines
backends/vhost-user-blk-server.c new file +10 -0
+    int ret;
+
+    client = container_of(vu_dev, VuClient, parent);
+    assert(client);
+
+    vq = vu_get_queue(vu_dev, idx);
+    assert(vq);
+
+    while (1) {
+        ret = vu_block_virtio_process_req(client, vq);

If you call vu_block_virtio_process_req(), then this one need to be a coroutine_fn, too.

Quoted 5 lines
backends/vhost-user-blk-server.c new file +5 -0
+        if (ret) {
+            break;
+        }
+    }
+}

backends/vhost-user-blk-server.c line 245

Quoted 10 lines
backends/vhost-user-blk-server.c new file +10 -0
+
+static void vu_block_queue_set_started(VuDev *vu_dev, int idx, bool started)
+{
+    VuVirtq *vq;
+
+    assert(vu_dev);
+
+    vq = vu_get_queue(vu_dev, idx);
+    vu_set_queue_handler(vu_dev, vq, started ? vu_block_process_vq : NULL);
+}

If vu_block_process_vq() is used as a vq->handler, will it not be called outside coroutine context? How can this work when vu_block_virtio_process_req() required to be run in coroutine context?

Another problem is that vu_block_process_vq() runs a loop as long as there are requests that aren't completed yet. During this time, vu_kick_cb() will block. I don't think this is what was intended. Instead, we should start the requests (without waiting for their completion) and return.

vl.c static bool object_create_initial(const char *type, QemuOpts *opts) · line 2541

Quoted 9 lines
vl.c +1 -0
index 794f2e5733..0332ab70da 100644
@@ -2538,6 +2538,10 @@ static bool object_create_initial(const char *type, QemuOpts *opts)
     }
 #endif
 
+    /* Reason: vhost-user-server property "name" */

"node-name" now.

vl.c static bool object_create_initial(const char *type, QemuOpts *opts) · line 2547

Quoted 6 lines
vl.c +3 -0
+    if (g_str_equal(type, "vhost-user-blk-server")) {
+        return false;
+    }
     /*
      * Reason: filter-* property "netdev" etc.
      */

Kevin

author
#9 lore raw

PATCH v4 4/5 a standone-alone tool to directly share disk image file via vhost-user protocol

In reply to Coiby Xu

vhost-user-blk could have played as vhost-user backend but it only supports raw file and don't support VIRTIO_BLK_T_DISCARD and VIRTIO_BLK_T_WRITE_ZEROES operations on raw file (ioctl(fd, BLKDISCARD) is only valid for real block device).

In the future Kevin's qemu-storage-daemon will be used to replace this tool.

---
 Makefile  |   4 +
 configure |   3 +
 qemu-vu.c | 252 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 259 insertions(+)
 create mode 100644 qemu-vu.c
Makefile +4 -0
index b5a7377cb1..74fb109675 100644
@@ -572,6 +572,10 @@ qemu-img.o: qemu-img-cmds.h
 
 qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
+
+ifdef CONFIG_LINUX
+qemu-vu$(EXESUF): qemu-vu.o backends/vhost-user-blk-server.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) libvhost-user.a
+endif
 qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 
 qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
configure +3 -0
index 6f5d850949..2b846cecf0 100755
@@ -6239,6 +6239,9 @@ if test "$want_tools" = "yes" ; then
   if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then
     tools="qemu-nbd\$(EXESUF) $tools"
   fi
+  if [ "$linux" = "yes" ] ; then
+    tools="qemu-vu\$(EXESUF) $tools"
+  fi
   if [ "$ivshmem" = "yes" ]; then
     tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
   fi
qemu-vu.c new file +252 -0

252 lines not on this page · raw message

Signature · 1 line
2.25.0
author
#10 lore raw

PATCH v4 5/5 new qTest case to test the vhost-user-blk-server

In reply to Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize.

---
 tests/Makefile.include              |   3 +-
 tests/qtest/Makefile.include        |   2 +
 tests/qtest/libqos/vhost-user-blk.c | 126 +++++
 tests/qtest/libqos/vhost-user-blk.h |  44 ++
 tests/qtest/vhost-user-blk-test.c   | 694 ++++++++++++++++++++++++++++
 5 files changed, 868 insertions(+), 1 deletion(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
tests/Makefile.include +2 -1
index 2f1cafed72..4b8637b5d4 100644
@@ -629,7 +629,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all $(check-qtest-y)
 	$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) $(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
 	  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
-	  QTEST_QEMU_IMG=qemu-img$(EXESUF))
+	  QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+	  QTEST_QEMU_VU_BINARY=./qemu-vu$(EXESUF))
 
 check-unit: $(check-unit-y)
 	$(call do_test_human, $^)
tests/qtest/Makefile.include +2 -0
index eb0f23b108..f587fe9f4d 100644
@@ -182,6 +182,7 @@ qos-test-obj-y += tests/qtest/libqos/virtio.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 qos-test-obj-y += tests/qtest/libqos/virtio-balloon.o
 qos-test-obj-y += tests/qtest/libqos/virtio-blk.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 qos-test-obj-y += tests/qtest/libqos/virtio-mmio.o
 qos-test-obj-y += tests/qtest/libqos/virtio-net.o
 qos-test-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -224,6 +225,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
tests/qtest/libqos/vhost-user-blk.c new file +126 -0

126 lines not on this page · raw message

tests/qtest/libqos/vhost-user-blk.h new file +44 -0

44 lines not on this page · raw message

tests/qtest/vhost-user-blk-test.c new file +694 -0

694 lines not on this page · raw message

Signature · 1 line
2.25.0
#11 lore raw
In reply to Coiby Xu
Quoted 4 lines · Coiby Xu

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

Thanks! I think the vhost-user server code can be simplified if libvhost-user uses the event loop for asynchronous socket I/O. Then it's no longer necessary to duplicate vu_message_read() and vu_kick_cb(). I've replied to Patch 1 and we can discuss on IRC if you want to chat about it.

I've also CCed Marc-André to see what he thinks about removing the blocking recv from libvhost-user and instead using the event loop (just like for eventfds).

Stefan

author
#12 lore raw

Hi Stefan,

Thank you for reviewing my code!

I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.

If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event, i.e. use vu_dispatch as the read handler, then we can re-use vu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation of poll() and ppoll() is not affected by the O_NONBLOCK flag" despite that we use qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler to make recv non-blocking.

Previously I needed to run customized vu_kick_cb as coroutines to call blk_co_readv/blk_co_writev directly. After taking Kevin's feedback on v4 into consideration, now I use aio_set_fd_handler to set a read handler for kick_fd and this read handler will then call vu_kick_cb.

Quoted 18 lines · Stefan Hajnoczi
Quoted 4 lines · Coiby Xu

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

Thanks! I think the vhost-user server code can be simplified if libvhost-user uses the event loop for asynchronous socket I/O. Then it's no longer necessary to duplicate vu_message_read() and vu_kick_cb(). I've replied to Patch 1 and we can discuss on IRC if you want to chat about it.

I've also CCed Marc-André to see what he thinks about removing the blocking recv from libvhost-user and instead using the event loop (just like for eventfds).

Stefan

Signature · 2 lines
Best regards,
Coiby
#13 lore raw
Quoted 7 lines · Coiby Xu

Hi Stefan,

Thank you for reviewing my code!

I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.

Quoted 7 lines

If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event, i.e. use vu_dispatch as the read handler, then we can re-use vu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation of poll() and ppoll() is not affected by the O_NONBLOCK flag" despite that we use qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler to make recv non-blocking.

I'm not sure I understand. poll() just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (without O_NONBLOCK) or return only 1 byte (with O_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.

But please go ahead and send the next revision and I'll take a look.

Stefan

author
#14 lore raw

Thank you for reminding me of this socket short read issue! It seems we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

Quoted 24 lines · Stefan Hajnoczi
Quoted 7 lines · Coiby Xu

Hi Stefan,

Thank you for reviewing my code!

I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.

Quoted 7 lines

If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event, i.e. use vu_dispatch as the read handler, then we can re-use vu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation of poll() and ppoll() is not affected by the O_NONBLOCK flag" despite that we use qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler to make recv non-blocking.

I'm not sure I understand. poll() just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (without O_NONBLOCK) or return only 1 byte (with O_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.

Quoted 3 lines

But please go ahead and send the next revision and I'll take a look.

Stefan

--
Best regards,
Coiby

#15 lore raw

Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:

Quoted 7 lines

Thank you for reminding me of this socket short read issue! It seems we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Kevin

Quoted 35 lines · Stefan Hajnoczi
Quoted 24 lines · Coiby Xu
Quoted 7 lines

Hi Stefan,

Thank you for reviewing my code!

I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.

Quoted 7 lines

If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event, i.e. use vu_dispatch as the read handler, then we can re-use vu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation of poll() and ppoll() is not affected by the O_NONBLOCK flag" despite that we use qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler to make recv non-blocking.

I'm not sure I understand. poll() just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (without O_NONBLOCK) or return only 1 byte (with O_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.

Quoted 3 lines

But please go ahead and send the next revision and I'll take a look.

Stefan

--
Best regards,
Coiby

author
#16 lore raw
Quoted 10 lines
Quoted 6 lines

we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting vu_message_read to address the short read issue. In v3 & v4, I extended libvhost-user to allow vu_message_read to be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.

Quoted 52 lines · Kevin Wolf

Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:

Quoted 7 lines

Thank you for reminding me of this socket short read issue! It seems we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Kevin

Quoted 35 lines · Stefan Hajnoczi
Quoted 24 lines · Coiby Xu
Quoted 7 lines

Hi Stefan,

Thank you for reviewing my code!

I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.

Quoted 7 lines

If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event, i.e. use vu_dispatch as the read handler, then we can re-use vu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation of poll() and ppoll() is not affected by the O_NONBLOCK flag" despite that we use qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler to make recv non-blocking.

I'm not sure I understand. poll() just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (without O_NONBLOCK) or return only 1 byte (with O_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.

Quoted 3 lines

But please go ahead and send the next revision and I'll take a look.

Stefan

--
Best regards,
Coiby

Signature · 2 lines
Best regards,
Coiby
#17 lore raw

Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Quoted 6 lines

But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting vu_message_read to address the short read issue. In v3 & v4, I extended libvhost-user to allow vu_message_read to be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.

The way your latest patches have it, with a separate read function, works for me.

You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.

The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until read() returns enough bytes is not acceptable in QEMU.

Kevin

Quoted 59 lines · Kevin Wolf
Quoted 52 lines

Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:

Quoted 7 lines

Thank you for reminding me of this socket short read issue! It seems we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Kevin

Quoted 35 lines · Stefan Hajnoczi
Quoted 24 lines · Coiby Xu
Quoted 7 lines

Hi Stefan,

Thank you for reviewing my code!

I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.

Quoted 7 lines

If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event, i.e. use vu_dispatch as the read handler, then we can re-use vu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation of poll() and ppoll() is not affected by the O_NONBLOCK flag" despite that we use qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler to make recv non-blocking.

I'm not sure I understand. poll() just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (without O_NONBLOCK) or return only 1 byte (with O_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.

Quoted 3 lines

But please go ahead and send the next revision and I'll take a look.

Stefan

--
Best regards,
Coiby

Signature · 2 lines
Best regards,
Coiby
#18 lore raw

Hi

Quoted 20 lines · Kevin Wolf

Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Quoted 6 lines

But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting vu_message_read to address the short read issue. In v3 & v4, I extended libvhost-user to allow vu_message_read to be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.

Quoted 2 lines

The way your latest patches have it, with a separate read function, works for me.

Done right, I am not against it, fwiw

Quoted 9 lines

You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.

The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until read() returns enough bytes is not acceptable in QEMU.

In the context of vhost-user, local unix sockets with short messages (do we have >1k messages?), I am not sure if this is really a problem.

And isn't it possible to run libvhost-user in its own thread for this series?

Quoted 63 lines

Kevin

Quoted 59 lines · Kevin Wolf
Quoted 10 lines

Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:

Quoted 7 lines

Thank you for reminding me of this socket short read issue! It seems we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

Quoted 42 lines

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Kevin

Quoted 35 lines · Stefan Hajnoczi
Quoted 24 lines · Coiby Xu
Quoted 7 lines

Hi Stefan,

Thank you for reviewing my code!

I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.

Quoted 7 lines

If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event, i.e. use vu_dispatch as the read handler, then we can re-use vu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation of poll() and ppoll() is not affected by the O_NONBLOCK flag" despite that we use qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler to make recv non-blocking.

I'm not sure I understand. poll() just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (without O_NONBLOCK) or return only 1 byte (with O_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.

Quoted 3 lines

But please go ahead and send the next revision and I'll take a look.

Stefan

--
Best regards,
Coiby

--
Best regards,
Coiby

#19 lore raw

Am 27.02.2020 um 12:07 hat Marc-André Lureau geschrieben:

Quoted 30 lines · Kevin Wolf
Quoted 19 lines

Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Quoted 6 lines

But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting vu_message_read to address the short read issue. In v3 & v4, I extended libvhost-user to allow vu_message_read to be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.

Quoted 2 lines

The way your latest patches have it, with a separate read function, works for me.

Done right, I am not against it, fwiw

Quoted 5 lines

You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.

Quoted 7 lines
Quoted 4 lines

The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until read() returns enough bytes is not acceptable in QEMU.

In the context of vhost-user, local unix sockets with short messages (do we have >1k messages?), I am not sure if this is really a problem.

I'm not sure how much of a problem it is in practice, and whether we can consider the vhost-user client trusted. But using QIOChannel from within a coroutine just avoids the problem completely, so it feels like a natural choice to just do that.

Quoted 2 lines

And isn't it possible to run libvhost-user in its own thread for this series?

You need to run the actual block I/O requests in the iothread where the block device happens to run. So if you move the message processing to a separate thread, you would have to communicate between threads for the actual request processing. Possible, but probably slower than necessary and certainly more complex.

Kevin

#20 lore raw
Quoted 36 lines · Kevin Wolf

Am 27.02.2020 um 12:07 hat Marc-André Lureau geschrieben:

Quoted 35 lines · Kevin Wolf
Quoted 19 lines

Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Quoted 6 lines

But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting vu_message_read to address the short read issue. In v3 & v4, I extended libvhost-user to allow vu_message_read to be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.

Quoted 2 lines

The way your latest patches have it, with a separate read function, works for me.

Done right, I am not against it, fwiw

Quoted 9 lines

You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.

The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until read() returns enough bytes is not acceptable in QEMU.

Quoted 7 lines
Quoted 2 lines

In the context of vhost-user, local unix sockets with short messages (do we have >1k messages?), I am not sure if this is really a problem.

I'm not sure how much of a problem it is in practice, and whether we can consider the vhost-user client trusted. But using QIOChannel from within a coroutine just avoids the problem completely, so it feels like a natural choice to just do that.

It isn't clear to me that we have a consitent plan for how we intend libvhost-user to develop & what it is permitted to use. What information I see in the source code and in this thread are contradictory.

For example, in the text quoted above:

"libvhost-user is supposed to be indepdent from the main QEMU code."

which did match my overall understanding too. At the top of libvhost-user.c there is a comment

/* this code avoids GLib dependency */

but a few lines later it does

  #include "qemu/atomic.h"
  #include "qemu/osdep.h"
  #include "qemu/memfd.h"

and in the Makefile we link it to much of QEMU util code:

  libvhost-user.a: $(libvhost-user-obj-y) $(util-obj-y) $(stub-obj-y)

this in turn pulls in GLib code, and looking at symbols we can see over 100 GLib functions used:

  $ nm ./libvhost-user.a | grep 'U g_' | sort | uniq | wc -l
  128

And over 200 QEMU source object files included:

  $ nm ./libvhost-user.a | grep '.o:' | sort | wc -l
  224

So unless I'm missing something, we have lost any independance from both QEMU and GLib code that we might have had in the past.

Note this also has licensing implications, as I expect this means that via the QEMU source objects it pulls in, libvhost-user.a has become a GPLv2-only combined work, not a GPLv2-or-later combined work.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#21 lore raw

Hi

Quoted 75 lines · Daniel P. Berrangé
Quoted 36 lines · Kevin Wolf

Am 27.02.2020 um 12:07 hat Marc-André Lureau geschrieben:

Quoted 35 lines · Kevin Wolf
Quoted 19 lines

Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.

In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.

Quoted 6 lines

But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting vu_message_read to address the short read issue. In v3 & v4, I extended libvhost-user to allow vu_message_read to be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.

Quoted 2 lines

The way your latest patches have it, with a separate read function, works for me.

Done right, I am not against it, fwiw

Quoted 9 lines

You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.

The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until read() returns enough bytes is not acceptable in QEMU.

Quoted 7 lines
Quoted 2 lines

In the context of vhost-user, local unix sockets with short messages (do we have >1k messages?), I am not sure if this is really a problem.

I'm not sure how much of a problem it is in practice, and whether we can consider the vhost-user client trusted. But using QIOChannel from within a coroutine just avoids the problem completely, so it feels like a natural choice to just do that.

It isn't clear to me that we have a consitent plan for how we intend libvhost-user to develop & what it is permitted to use. What information I see in the source code and in this thread are contradictory.

For example, in the text quoted above:

"libvhost-user is supposed to be indepdent from the main QEMU code."

which did match my overall understanding too. At the top of libvhost-user.c there is a comment

   /* this code avoids GLib dependency */

but a few lines later it does

  #include "qemu/atomic.h"
  #include "qemu/osdep.h"
  #include "qemu/memfd.h"

and in the Makefile we link it to much of QEMU util code:

  libvhost-user.a: $(libvhost-user-obj-y) $(util-obj-y) $(stub-obj-y)

this in turn pulls in GLib code, and looking at symbols we can see over 100 GLib functions used:

  $ nm ./libvhost-user.a | grep 'U g_' | sort | uniq | wc -l
  128
Quoted 7 lines

And over 200 QEMU source object files included:

  $ nm ./libvhost-user.a | grep '.o:' | sort | wc -l
  224

So unless I'm missing something, we have lost any independance from both QEMU and GLib code that we might have had in the past.

Yep, I think this is mostly due to commit 5f9ff1eff38 ("libvhost-user:
Support tracking inflight I/O in shared memory")

It may not be that hard to bring back glib independence. Is it worth it though?

Quoted 5 lines

Note this also has licensing implications, as I expect this means that via the QEMU source objects it pulls in, libvhost-user.a has become a GPLv2-only combined work, not a GPLv2-or-later combined work.

libvhost-user.c is GPLv2-or-later because tests/vhost-user-bridge.c was and is still as well. Do we need to change that?

Quoted 8 lines

Regards,
Daniel
--

|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Signature · 1 line
Marc-André Lureau