Skip to contentSkip to conversation

vhost-user block device backend implementation

PATCH Superseded Coiby Xu 6 participants 51 messages Jun 14, 2020 – Sep 18, 2020 Block layer

v9 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Coiby Xu

author
#0 lore raw

v9 cover vhost-user block device backend implementation

v9
- move logical block size check function to a utility function
- fix issues regarding license, coding style, memory deallocation, etc.

v8
- re-try connecting to socket server to fix asan error
- fix license naming issue

v7
- fix docker-test-debug@fedora errors by freeing malloced memory

v6
- add missing license header and include guard
- vhost-user server only serve one client one time
- fix a bug in custom vu_message_read
- using qemu-storage-daemon to start vhost-user-blk-server
- a bug fix to pass docker-test-clang@ubuntu

v5:
* re-use vu_kick_cb in libvhost-user
* keeping processing VhostUserMsg in the same coroutine until there is detachment/attachment of AIOContext
* Spawn separate coroutine for each VuVirtqElement
* Other changes including relocating vhost-user-blk-server.c, coding style etc.

v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.

v3:
* separate generic vhost-user-server code from vhost-user-blk-server code
* re-write vu_message_read and kick hander function as coroutines to directly call blk_co_preadv, blk_co_pwritev, etc.
* add aio_context notifier functions to support multi-threading model
* other fixes regarding coding style, warning report, etc.

v2:
* Only enable this feature for Linux because eventfd is a Linux-specific feature

This patch series is an implementation of vhost-user block device backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be started using object_add,

(qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512

Then vhost-user client can connect to the server backend. For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (5):
Allow vu_message_read to be replaced
generic vhost user server
move logical block size check function to a common utility function vhost-user block device backend server
new qTest case to test the vhost-user-blk-server

 block/Makefile.objs                        |   1 +
 block/export/vhost-user-blk-server.c       | 669 +++++++++++++++++++
 block/export/vhost-user-blk-server.h       |  35 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c      |  11 +-
 contrib/libvhost-user/libvhost-user.h      |  21 +
 hw/core/qdev-properties.c                  |  18 +-
 softmmu/vl.c                               |   4 +
 tests/Makefile.include                     |   3 +-
 tests/qtest/Makefile.include               |   2 +
 tests/qtest/libqos/vhost-user-blk.c        | 130 ++++
 tests/qtest/libqos/vhost-user-blk.h        |  48 ++
 tests/qtest/libqtest.c                     |  35 +-
 tests/qtest/libqtest.h                     |  17 +
 tests/qtest/vhost-user-blk-test.c          | 739 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |   2 +
 tools/virtiofsd/fuse_virtio.c              |   4 +-
 util/Makefile.objs                         |   2 +
 util/block-helpers.c                       |  46 ++
 util/block-helpers.h                       |   7 +
 util/vhost-user-server.c                   | 400 +++++++++++
 util/vhost-user-server.h                   |  61 ++
 22 files changed, 2231 insertions(+), 26 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/block-helpers.c
 create mode 100644 util/block-helpers.h
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.27.0

author
#1 lore raw

PATCH v9 1/5 Allow vu_message_read to be replaced

Allow vu_message_read to be replaced by one which will make use of the QIOChannel functions. Thus reading vhost-user message won't stall the guest.

---
 contrib/libvhost-user/libvhost-user-glib.c |  2 +-
 contrib/libvhost-user/libvhost-user.c      | 11 ++++++-----
 contrib/libvhost-user/libvhost-user.h      | 21 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |  2 ++
 tools/virtiofsd/fuse_virtio.c              |  4 ++--
 5 files changed, 32 insertions(+), 8 deletions(-)
contrib/libvhost-user/libvhost-user-glib.c +1 -1
index 53f1ca4cdd..0df2ec9271 100644
@@ -147,7 +147,7 @@ vug_init(VugDev *dev, uint16_t max_queues, int socket,
     g_assert(dev);
     g_assert(iface);
 
-    if (!vu_init(&dev->parent, max_queues, socket, panic, set_watch,
+    if (!vu_init(&dev->parent, max_queues, socket, panic, NULL, set_watch,
                  remove_watch, iface)) {
         return false;
     }
contrib/libvhost-user/libvhost-user.c +6 -5
index 3bca996c62..0c7368baa2 100644
@@ -67,8 +67,6 @@
 /* The version of inflight buffer */
 #define INFLIGHT_VERSION 1
 
-#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
-
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION 1
 #define LIBVHOST_USER_DEBUG 0
@@ -412,7 +410,7 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg *vmsg)
         goto out;
     }
 
-    if (!vu_message_read(dev, dev->slave_fd, &msg_reply)) {
+    if (!dev->read_msg(dev, dev->slave_fd, &msg_reply)) {
         goto out;
     }
 
@@ -647,7 +645,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg *vmsg)
     /* Wait for QEMU to confirm that it's registered the handler for the
      * faults.
      */
-    if (!vu_message_read(dev, dev->sock, vmsg) ||
+    if (!dev->read_msg(dev, dev->sock, vmsg) ||
         vmsg->size != sizeof(vmsg->payload.u64) ||
         vmsg->payload.u64 != 0) {
         vu_panic(dev, "failed to receive valid ack for postcopy set-mem-table");
@@ -1653,7 +1651,7 @@ vu_dispatch(VuDev *dev)
     int reply_requested;
     bool need_reply, success = false;
 
-    if (!vu_message_read(dev, dev->sock, &vmsg)) {
+    if (!dev->read_msg(dev, dev->sock, &vmsg)) {
         goto end;
     }
 
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }
@@ -1751,6 +1750,7 @@ vu_init(VuDev *dev,
         uint16_t max_queues,
         int socket,
         vu_panic_cb panic,
+        vu_read_msg_cb read_msg,
         vu_set_watch_cb set_watch,
         vu_remove_watch_cb remove_watch,
         const VuDevIface *iface)
@@ -1768,6 +1768,7 @@ vu_init(VuDev *dev,
 
     dev->sock = socket;
     dev->panic = panic;
+    dev->read_msg = read_msg ? read_msg : vu_message_read;
     dev->set_watch = set_watch;
     dev->remove_watch = remove_watch;
     dev->iface = iface;
contrib/libvhost-user/libvhost-user.h +21 -0
index f30394fab6..d756da8548 100644
@@ -30,6 +30,8 @@
 
 #define VHOST_MEMORY_MAX_NREGIONS 8
 
+#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64)
+
 typedef enum VhostSetConfigType {
     VHOST_SET_CONFIG_TYPE_MASTER = 0,
     VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -205,6 +207,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);
 typedef void (*vu_set_features_cb) (VuDev *dev, uint64_t features);
 typedef int (*vu_process_msg_cb) (VuDev *dev, VhostUserMsg *vmsg,
                                   int *do_reply);
+typedef bool (*vu_read_msg_cb) (VuDev *dev, int sock, VhostUserMsg *vmsg);
 typedef void (*vu_queue_set_started_cb) (VuDev *dev, int qidx, bool started);
 typedef bool (*vu_queue_is_processed_in_order_cb) (VuDev *dev, int qidx);
 typedef int (*vu_get_config_cb) (VuDev *dev, uint8_t *config, uint32_t len);
@@ -373,6 +376,23 @@ struct VuDev {
     bool broken;
     uint16_t max_queues;
 
+    /* @read_msg: custom method to read vhost-user message
+     *
+     * Read data from vhost_user socket fd and fill up
+     * the passed VhostUserMsg *vmsg struct.
+     *
+     * If reading fails, it should close the received set of file
+     * descriptors as socket message's auxiliary data.
+     *
+     * For the details, please refer to vu_message_read in libvhost-user.c
+     * which will be used by default if not custom method is provided when
+     * calling vu_init
+     *
+     * Returns: true if vhost-user message successfully received,
+     *          otherwise return false.
+     *
+     */
+    vu_read_msg_cb read_msg;
     /* @set_watch: add or update the given fd to the watch set,
      * call cb when condition is met */
     vu_set_watch_cb set_watch;
@@ -416,6 +436,7 @@ bool vu_init(VuDev *dev,
              uint16_t max_queues,
              int socket,
              vu_panic_cb panic,
+             vu_read_msg_cb read_msg,
              vu_set_watch_cb set_watch,
              vu_remove_watch_cb remove_watch,
              const VuDevIface *iface);
tests/vhost-user-bridge.c +2 -0
index 6c3d490611..bd43607a4d 100644
@@ -520,6 +520,7 @@ vubr_accept_cb(int sock, void *ctx)
                  VHOST_USER_BRIDGE_MAX_QUEUES,
                  conn_fd,
                  vubr_panic,
+                 NULL,
                  vubr_set_watch,
                  vubr_remove_watch,
                  &vuiface)) {
@@ -573,6 +574,7 @@ vubr_new(const char *path, bool client)
                      VHOST_USER_BRIDGE_MAX_QUEUES,
                      dev->sock,
                      vubr_panic,
+                     NULL,
                      vubr_set_watch,
                      vubr_remove_watch,
                      &vuiface)) {
tools/virtiofsd/fuse_virtio.c +2 -2
index 3b6d16a041..666945c897 100644
@@ -980,8 +980,8 @@ int virtio_session_mount(struct fuse_session *se)
     se->vu_socketfd = data_sock;
     se->virtio_dev->se = se;
     pthread_rwlock_init(&se->virtio_dev->vu_dispatch_rwlock, NULL);
-    vu_init(&se->virtio_dev->dev, 2, se->vu_socketfd, fv_panic, fv_set_watch,
-            fv_remove_watch, &fv_iface);
+    vu_init(&se->virtio_dev->dev, 2, se->vu_socketfd, fv_panic, NULL,
+            fv_set_watch, fv_remove_watch, &fv_iface);
 
     return 0;
 }
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH v9 1/5] Allow vu_message_read to be replaced

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Quoted 5 lines

Allow vu_message_read to be replaced by one which will make use of the QIOChannel functions. Thus reading vhost-user message won't stall the guest.

_vu_queue_notify() still has a direct call of vu_message_read() instead of using the pointer. Is this intentional?

Renaming the function would make sure that such semantic merge conflicts don't stay unnoticed.

contrib/libvhost-user/libvhost-user.c vu_deinit(VuDev *dev) · line 1708

Quoted 8 lines
contrib/libvhost-user/libvhost-user.c +1 -0
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }

This hunk looks unrelated.

Kevin

author
#3 lore raw

Re: [PATCH v9 1/5] Allow vu_message_read to be replaced

Quoted 9 lines · Kevin Wolf

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Quoted 5 lines

Allow vu_message_read to be replaced by one which will make use of the QIOChannel functions. Thus reading vhost-user message won't stall the guest.

_vu_queue_notify() still has a direct call of vu_message_read() instead of using the pointer. Is this intentional?

This is a mistake. Thank you for reminding me!

Quoted 2 lines

Renaming the function would make sure that such semantic merge conflicts don't stay unnoticed.

Thank you for this tip! Do you suggest renaming the function only for the purpose of testing or should I adopt a name when submitting the patch? For the latter case, I will change it to vu_message_read_default then.

Quoted 10 lines

contrib/libvhost-user/libvhost-user.c vu_deinit(VuDev *dev) · line 1708

Quoted 8 lines
contrib/libvhost-user/libvhost-user.c +1 -0
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }

This hunk looks unrelated.

In v4, I made the comment to explain why it's needed. But libvhost-user is supposed to be independent from QEMU, so Stefan suggested to remove it,

Quoted 13 lines

Review vu_deinit(VuDev *dev) · line 1654

Quoted 9 lines
(quoted diff) +5 -0
@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            /* remove watch for kick_fd
+             * When client process is running in gdb and
+             * quit command is run in gdb, QEMU will still dispatch the event
+             * which will cause segment fault in the callback function
+             */

Code and comments in libvhost-user should not refer to QEMU specifics. Removing the watch is a good idea regardless of the application or event loop implementation. No comment is needed here.

Review

Quoted 6 lines
(quoted diff) +6 -0
+            /* remove watch for kick_fd
+             * When client process is running in gdb and
+             * quit command is run in gdb, QEMU will still dispatch the event
+             * which will cause segment fault in the callback function
+             */
+            dev->remove_watch(dev, vq->kick_fd);

--
Best regards,
Coiby

#4 lore raw

Re: [PATCH v9 1/5] Allow vu_message_read to be replaced

Am 24.06.2020 um 05:36 hat Coiby Xu geschrieben:

Quoted 13 lines · Kevin Wolf
Quoted 9 lines

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Quoted 5 lines

Allow vu_message_read to be replaced by one which will make use of the QIOChannel functions. Thus reading vhost-user message won't stall the guest.

_vu_queue_notify() still has a direct call of vu_message_read() instead of using the pointer. Is this intentional?

This is a mistake. Thank you for reminding me!

Quoted 7 lines
Quoted 2 lines

Renaming the function would make sure that such semantic merge conflicts don't stay unnoticed.

Thank you for this tip! Do you suggest renaming the function only for the purpose of testing or should I adopt a name when submitting the patch? For the latter case, I will change it to vu_message_read_default then.

I think I would rename it permanently and keep the new name for the actual submission. The reason is that if someone else is working on a series adding new references, the compiler would catch it there, too.

vu_message_read_default() sounds good to me.

Quoted 11 lines
Quoted 10 lines

contrib/libvhost-user/libvhost-user.c vu_deinit(VuDev *dev) · line 1708

Quoted 8 lines
contrib/libvhost-user/libvhost-user.c +1 -0
@@ -1704,6 +1702,7 @@ vu_deinit(VuDev *dev)
         }
 
         if (vq->kick_fd != -1) {
+            dev->remove_watch(dev, vq->kick_fd);
             close(vq->kick_fd);
             vq->kick_fd = -1;
         }

This hunk looks unrelated.

Quoted 2 lines

In v4, I made the comment to explain why it's needed. But libvhost-user is supposed to be independent from QEMU, so Stefan suggested to remove it,

Yes, I saw the reason why you need it in later patches.

If you can remove it completely, that is even better, but otherwise I would make the addition only later (either in the patch that actually needs it or in a new separate patch) because it's not really part of "allowing vu_message_read to be replaced", as the commit message says.

Kevin

author
#5 lore raw

PATCH v9 2/5 generic vhost user server

In reply to Coiby Xu

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

---
 util/Makefile.objs       |   1 +
 util/vhost-user-server.c | 400 +++++++++++++++++++++++++++++++++++++++
 util/vhost-user-server.h |  61 ++++++
 3 files changed, 462 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
util/Makefile.objs +1 -0
index cc5e37177a..b4d4af06dc 100644
@@ -66,6 +66,7 @@ util-obj-y += hbitmap.o
 util-obj-y += main-loop.o
 util-obj-y += nvdimm-utils.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += qemu-sockets.o
util/vhost-user-server.c new file +400 -0

400 lines not on this page · raw message

util/vhost-user-server.h new file +61 -0

61 lines not on this page · raw message

Signature · 1 line
2.27.0
#6 lore raw

Re: [PATCH v9 2/5] generic vhost user server

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Quoted 59 lines

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

---
 util/Makefile.objs       |   1 +
 util/vhost-user-server.c | 400 +++++++++++++++++++++++++++++++++++++++
 util/vhost-user-server.h |  61 ++++++
 3 files changed, 462 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
util/Makefile.objs +1 -0
index cc5e37177a..b4d4af06dc 100644
@@ -66,6 +66,7 @@ util-obj-y += hbitmap.o
 util-obj-y += main-loop.o
 util-obj-y += nvdimm-utils.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += qemu-sockets.o
util/vhost-user-server.c new file +28 -0
new file mode 100644
index 0000000000..393beeb6b9
@@ -0,0 +1,400 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include <sys/eventfd.h>
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+    int i;
+    for (i = 0; i < vmsg->fd_num; i++) {
+        close(vmsg->fds[i]);
+    }
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+    int i;
+    for (i = 0; i < vmsg->fd_num; i++) {
+        qemu_set_nonblock(vmsg->fds[i]);
+    }
+}

util/vhost-user-server.c line 38

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+                      gpointer opaque);
+
+static void close_client(VuServer *server)
+{
+    vu_deinit(&server->vu_dev);
+    object_unref(OBJECT(server->sioc));
+    object_unref(OBJECT(server->ioc));
+    server->sioc_slave = NULL;

Where is sioc_slave closed/freed?

util/vhost-user-server.c line 47

Quoted 9 lines
util/vhost-user-server.c new file +9 -0
+    object_unref(OBJECT(server->ioc_slave));
+    /*
+     * Set the callback function for network listener so another
+     * vhost-user client can connect to this server
+     */
+    qio_net_listener_set_client_func(server->listener,
+                                     vu_accept,
+                                     server,
+                                     NULL);

If connecting another client to the server should work, don't we have to set at least server->sioc = NULL so that vu_accept() won't error out?

Quoted 145 lines
util/vhost-user-server.c new file +145 -0
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+
+    if (buf) {
+        error_report("vu_panic: %s", buf);
+    }
+
+    if (server->sioc) {
+        close_client(server);
+        server->sioc = NULL;
+    }
+
+    if (server->device_panic_notifier) {
+        server->device_panic_notifier(server);
+    }
+}
+
+static QIOChannel *slave_io_channel(VuServer *server, int fd,
+                                    Error **local_err)
+{
+    if (server->sioc_slave) {
+        if (fd == server->sioc_slave->fd) {
+            return server->ioc_slave;
+        }
+    } else {
+        server->sioc_slave = qio_channel_socket_new_fd(fd, local_err);
+        if (!*local_err) {
+            server->ioc_slave = QIO_CHANNEL(server->sioc_slave);
+            return server->ioc_slave;
+        }
+    }
+
+    return NULL;
+}
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    struct iovec iov = {
+        .iov_base = (char *)vmsg,
+        .iov_len = VHOST_USER_HDR_SIZE,
+    };
+    int rc, read_bytes = 0;
+    Error *local_err = NULL;
+    /*
+     * Store fds/nfds returned from qio_channel_readv_full into
+     * temporary variables.
+     *
+     * VhostUserMsg is a packed structure, gcc will complain about passing
+     * pointer to a packed structure member if we pass &VhostUserMsg.fd_num
+     * and &VhostUserMsg.fds directly when calling qio_channel_readv_full,
+     * thus two temporary variables nfds and fds are used here.
+     */
+    size_t nfds = 0, nfds_t = 0;
+    int *fds_t = NULL;
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    QIOChannel *ioc = NULL;
+
+    if (conn_fd == server->sioc->fd) {
+        ioc = server->ioc;
+    } else {
+        /* Slave communication will also use this function to read msg */
+        ioc = slave_io_channel(server, conn_fd, &local_err);
+    }
+
+    if (!ioc) {
+        error_report_err(local_err);
+        goto fail;
+    }
+
+    assert(qemu_in_coroutine());
+    do {
+        /*
+         * qio_channel_readv_full may have short reads, keeping calling it
+         * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+         */
+        rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &local_err);
+        if (rc < 0) {
+            if (rc == QIO_CHANNEL_ERR_BLOCK) {
+                qio_channel_yield(ioc, G_IO_IN);
+                continue;
+            } else {
+                error_report_err(local_err);
+                return false;
+            }
+        }
+        read_bytes += rc;
+        if (nfds_t > 0) {
+            if (nfds + nfds_t > G_N_ELEMENTS(vmsg->fds)) {
+                error_report("A maximum of %d fds are allowed, "
+                             "however got %lu fds now",
+                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
+                goto fail;
+            }
+            memcpy(vmsg->fds + nfds, fds_t,
+                   nfds_t *sizeof(vmsg->fds[0]));
+            nfds += nfds_t;
+            g_free(fds_t);
+        }
+        if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+            break;
+        }
+        iov.iov_base = (char *)vmsg + read_bytes;
+        iov.iov_len = VHOST_USER_HDR_SIZE - read_bytes;
+    } while (true);
+
+    vmsg->fd_num = nfds;
+    /* qio_channel_readv_full will make socket fds blocking, unblock them */
+    vmsg_unblock_fds(vmsg);
+    if (vmsg->size > sizeof(vmsg->payload)) {
+        error_report("Error: too big message request: %d, "
+                     "size: vmsg->size: %u, "
+                     "while sizeof(vmsg->payload) = %zu",
+                     vmsg->request, vmsg->size, sizeof(vmsg->payload));
+        goto fail;
+    }
+
+    struct iovec iov_payload = {
+        .iov_base = (char *)&vmsg->payload,
+        .iov_len = vmsg->size,
+    };
+    if (vmsg->size) {
+        rc = qio_channel_readv_all_eof(ioc, &iov_payload, 1, &local_err);
+        if (rc == -1) {
+            error_report_err(local_err);
+            goto fail;
+        }
+    }
+
+    return true;
+
+fail:
+    vmsg_close_fds(vmsg);
+
+    return false;
+}
+
+
+static void vu_client_start(VuServer *server);
+static coroutine_fn void vu_client_trip(void *opaque)
+{
+    VuServer *server = opaque;

util/vhost-user-server.c line 202

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+
+    while (!server->aio_context_changed && server->sioc) {
+        vu_dispatch(&server->vu_dev);
+    }
+
+    if (server->aio_context_changed && server->sioc) {
+        server->aio_context_changed = false;
+        vu_client_start(server);
+    }
+}

This is somewhat convoluted, but ok. As soon as my patch "util/async:
Add aio_co_reschedule_self()" is merged, we can use it to simplify this a bit.

Quoted 102 lines
util/vhost-user-server.c new file +102 -0
+static void vu_client_start(VuServer *server)
+{
+    server->co_trip = qemu_coroutine_create(vu_client_trip, server);
+    aio_co_enter(server->ctx, server->co_trip);
+}
+
+/*
+ * a wrapper for vu_kick_cb
+ *
+ * since aio_dispatch can only pass one user data pointer to the
+ * callback function, pack VuDev and pvt into a struct. Then unpack it
+ * and pass them to vu_kick_cb
+ */
+static void kick_handler(void *opaque)
+{
+    KickInfo *kick_info = opaque;
+    kick_info->cb(kick_info->vu_dev, 0, (void *) kick_info->index);
+}
+
+
+static void
+set_watch(VuDev *vu_dev, int fd, int vu_evt,
+          vu_watch_cb cb, void *pvt)
+{
+
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+    long index = (intptr_t) pvt;
+    g_assert(cb);
+    KickInfo *kick_info = &server->kick_info[index];
+    if (!kick_info->cb) {
+        kick_info->fd = fd;
+        kick_info->cb = cb;
+        qemu_set_nonblock(fd);
+        aio_set_fd_handler(server->ioc->ctx, fd, false, kick_handler,
+                           NULL, NULL, kick_info);
+        kick_info->vu_dev = vu_dev;
+    }
+}
+
+
+static void remove_watch(VuDev *vu_dev, int fd)
+{
+    VuServer *server;
+    int i;
+    int index = -1;
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+
+    server = container_of(vu_dev, VuServer, vu_dev);
+    for (i = 0; i < vu_dev->max_queues; i++) {
+        if (server->kick_info[i].fd == fd) {
+            index = i;
+            break;
+        }
+    }
+
+    if (index == -1) {
+        return;
+    }
+    server->kick_info[i].cb = NULL;
+    aio_set_fd_handler(server->ioc->ctx, fd, false, NULL, NULL, NULL, NULL);
+}
+
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+                      gpointer opaque)
+{
+    VuServer *server = opaque;
+
+    if (server->sioc) {
+        warn_report("Only one vhost-user client is allowed to "
+                    "connect the server one time");
+        return;
+    }
+
+    if (!vu_init(&server->vu_dev, server->max_queues, sioc->fd, panic_cb,
+                 vu_message_read, set_watch, remove_watch, server->vu_iface)) {
+        error_report("Failed to initialized libvhost-user");
+        return;
+    }
+
+    /*
+     * Unset the callback function for network listener to make another
+     * vhost-user client keeping waiting until this client disconnects
+     */
+    qio_net_listener_set_client_func(server->listener,
+                                     NULL,
+                                     NULL,
+                                     NULL);
+    server->sioc = sioc;
+    server->kick_info = g_new0(KickInfo, server->max_queues);
+    /*
+     * Increase the object reference, so sioc will not freed by
+     * qio_net_listener_channel_func which will call object_unref(OBJECT(sioc))
+     */
+    object_ref(OBJECT(server->sioc));
+    qio_channel_set_name(QIO_CHANNEL(sioc), "vhost-user client");
+    server->ioc = QIO_CHANNEL(sioc);
+    object_ref(OBJECT(server->ioc));
+    qio_channel_attach_aio_context(server->ioc, server->ctx);

util/vhost-user-server.c line 315

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+    qio_channel_set_blocking(QIO_CHANNEL(server->sioc), false, NULL);
+    vu_client_start(server);
+}
+
+
+void vhost_user_server_stop(VuServer *server)
+{
+    if (!server) {
+        return;
+    }

There is no reason why the caller should even pass NULL.

util/vhost-user-server.c line 319

Quoted 3 lines
util/vhost-user-server.c new file +3 -0
+    if (server->sioc) {
+        close_client(server);
+        object_unref(OBJECT(server->sioc));

close_client() already unrefs it. Do we really hold two references? If so, why?

I can see that vu_accept() takes an extra reference, but the comment there says this is because QIOChannel takes ownership.

util/vhost-user-server.c line 327

Quoted 8 lines
util/vhost-user-server.c new file +8 -0
+    }
+
+    if (server->listener) {
+        qio_net_listener_disconnect(server->listener);
+        object_unref(OBJECT(server->listener));
+    }
+
+    g_free(server->kick_info);

Don't we need to wait for co_trip to terminate somewhere? Probably before freeing any objects because it could still use them.

I assume vhost_user_server_stop() is always called from the main thread whereas co_trip runs in the server AioContext, so extra care is necessary.

Quoted 21 lines
util/vhost-user-server.c new file +21 -0
+}
+
+static void detach_context(VuServer *server)
+{
+    int i;
+    AioContext *ctx = server->ioc->ctx;
+    qio_channel_detach_aio_context(server->ioc);
+    for (i = 0; i < server->vu_dev.max_queues; i++) {
+        if (server->kick_info[i].cb) {
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, false, NULL,
+                               NULL, NULL, NULL);
+        }
+    }
+}
+
+static void attach_context(VuServer *server, AioContext *ctx)
+{
+    int i;
+    qio_channel_attach_aio_context(server->ioc, ctx);
+    server->aio_context_changed = true;
+    if (server->co_trip) {

util/vhost-user-server.c line 358

Quoted 10 lines
util/vhost-user-server.c new file +10 -0
+        aio_co_schedule(ctx, server->co_trip);
+    }
+    for (i = 0; i < server->vu_dev.max_queues; i++) {
+        if (server->kick_info[i].cb) {
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, false,
+                               kick_handler, NULL, NULL,
+                               &server->kick_info[i]);
+        }
+    }
+}

There is a lot of duplication between detach_context() and attach_context(). I think implementing this directly in vhost_user_server_set_aio_context() for both cases at once would result in simpler code.

util/vhost-user-server.c line 371

Quoted 12 lines
util/vhost-user-server.c new file +12 -0
+void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server)
+{
+    server->ctx = ctx ? ctx : qemu_get_aio_context();
+    if (!server->sioc) {
+        return;
+    }
+    if (ctx) {
+        attach_context(server, ctx);
+    } else {
+        detach_context(server);
+    }
+}

What happens if the VuServer is already attached to an AioContext and you change it to another AioContext? Shouldn't it be detached from the old context and attached to the new one instead of only doing the latter?

util/vhost-user-server.c line 381

Quoted 9 lines
util/vhost-user-server.c new file +9 -0
+
+bool vhost_user_server_start(VuServer *server,
+                             SocketAddress *socket_addr,
+                             AioContext *ctx,
+                             uint16_t max_queues,
+                             DevicePanicNotifierFn *device_panic_notifier,
+                             const VuDevIface *vu_iface,
+                             Error **errp)
+{

I think this is the function that is supposed to initialise the VuServer object, so would it be better to first zero it out completely?

Or alternatively assign it completely like this (which automatically zeroes any unspecified field):

    *server = (VuServer) {
        .vu_iface       = vu_iface,
        .max_queues     = max_queues,
        ...
    }
Quoted 46 lines
util/vhost-user-server.c new file +19 -0
+    server->listener = qio_net_listener_new();
+    if (qio_net_listener_open_sync(server->listener, socket_addr, 1,
+                                   errp) < 0) {
+        return false;
+    }
+
+    qio_net_listener_set_name(server->listener, "vhost-user-backend-listener");
+
+    server->vu_iface = vu_iface;
+    server->max_queues = max_queues;
+    server->ctx = ctx;
+    server->device_panic_notifier = device_panic_notifier;
+    qio_net_listener_set_client_func(server->listener,
+                                     vu_accept,
+                                     server,
+                                     NULL);
+
+    return true;
+}
util/vhost-user-server.h new file +21 -0
new file mode 100644
index 0000000000..5baf58f96a
@@ -0,0 +1,61 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+
+#ifndef VHOST_USER_SERVER_H
+#define VHOST_USER_SERVER_H
+
+#include "contrib/libvhost-user/libvhost-user.h"
+#include "io/channel-socket.h"
+#include "io/channel-file.h"
+#include "io/net-listener.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "standard-headers/linux/virtio_blk.h"
+
+typedef struct KickInfo {

util/vhost-user-server.h line 31

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    VuDev *vu_dev;
+    int fd; /*kick fd*/
+    long index; /*queue index*/
+    vu_watch_cb cb;
+} KickInfo;
+
+typedef struct VuServer {
+    QIONetListener *listener;
+    AioContext *ctx;
+    void (*device_panic_notifier)(struct VuServer *server) ;

Extra space before the semicolon.

util/vhost-user-server.h line 41

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    int max_queues;
+    const VuDevIface *vu_iface;
+    VuDev vu_dev;
+    QIOChannel *ioc; /* The I/O channel with the client */
+    QIOChannelSocket *sioc; /* The underlying data channel with the client */
+    /* IOChannel for fd provided via VHOST_USER_SET_SLAVE_REQ_FD */
+    QIOChannel *ioc_slave;
+    QIOChannelSocket *sioc_slave;
+    Coroutine *co_trip; /* coroutine for processing VhostUserMsg */
+    KickInfo *kick_info; /* an array with the length of the queue number */

"an array with @max_queues elements"?

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    /* restart coroutine co_trip if AIOContext is changed */
+    bool aio_context_changed;
+} VuServer;
+
+
+typedef void DevicePanicNotifierFn(struct VuServer *server);
+
+bool vhost_user_server_start(VuServer *server,
+                             SocketAddress *unix_socket,
+                             AioContext *ctx,

util/vhost-user-server.h line 61

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+                             uint16_t max_queues,
+                             DevicePanicNotifierFn *device_panic_notifier,
+                             const VuDevIface *vu_iface,
+                             Error **errp);
+
+void vhost_user_server_stop(VuServer *server);
+
+void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server);
+
+#endif /* VHOST_USER_SERVER_H */

Kevin

author
#7 lore raw

Re: [PATCH v9 2/5] generic vhost user server

Quoted 64 lines · Kevin Wolf

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Quoted 63 lines

Sharing QEMU devices via vhost-user protocol.

Only one vhost-user client can connect to the server one time.

---
 util/Makefile.objs       |   1 +
 util/vhost-user-server.c | 400 +++++++++++++++++++++++++++++++++++++++
 util/vhost-user-server.h |  61 ++++++
 3 files changed, 462 insertions(+)
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h
util/Makefile.objs +1 -0
index cc5e37177a..b4d4af06dc 100644
@@ -66,6 +66,7 @@ util-obj-y += hbitmap.o
 util-obj-y += main-loop.o
 util-obj-y += nvdimm-utils.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
 util-obj-y += qemu-sockets.o
util/vhost-user-server.c new file +32 -0
new file mode 100644
index 0000000000..393beeb6b9
@@ -0,0 +1,400 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include <sys/eventfd.h>
+#include "qemu/main-loop.h"
+#include "vhost-user-server.h"
+
+static void vmsg_close_fds(VhostUserMsg *vmsg)
+{
+    int i;
+    for (i = 0; i < vmsg->fd_num; i++) {
+        close(vmsg->fds[i]);
+    }
+}
+
+static void vmsg_unblock_fds(VhostUserMsg *vmsg)
+{
+    int i;
+    for (i = 0; i < vmsg->fd_num; i++) {
+        qemu_set_nonblock(vmsg->fds[i]);
+    }
+}
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+                      gpointer opaque);
+
Quoted 8 lines

util/vhost-user-server.c line 38

Quoted 6 lines
util/vhost-user-server.c new file +6 -0
+static void close_client(VuServer *server)
+{
+    vu_deinit(&server->vu_dev);
+    object_unref(OBJECT(server->sioc));
+    object_unref(OBJECT(server->ioc));
+    server->sioc_slave = NULL;

Where is sioc_slave closed/freed?

Thank you for pointing out my neglect! When working on v10, I realized communication on the slave channel can't be done easily in a coroutine. So I simply dropped the support.

Quoted 10 lines

util/vhost-user-server.c line 47

Quoted 9 lines
util/vhost-user-server.c new file +9 -0
+    object_unref(OBJECT(server->ioc_slave));
+    /*
+     * Set the callback function for network listener so another
+     * vhost-user client can connect to this server
+     */
+    qio_net_listener_set_client_func(server->listener,
+                                     vu_accept,
+                                     server,
+                                     NULL);
Quoted 2 lines

If connecting another client to the server should work, don't we have to set at least server->sioc = NULL so that vu_accept() won't error out?

Previously I set server->sioc = NULL in the panic_cb, i.e. only when the client disconnects, because I thought it's different from the case that the server is shutdown. But this differentiating is not necessary. In v10, I has moved server->sioc = NULL into close_client.

Quoted 152 lines
Quoted 152 lines
util/vhost-user-server.c new file +151 -0
+}
+
+static void panic_cb(VuDev *vu_dev, const char *buf)
+{
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+
+    if (buf) {
+        error_report("vu_panic: %s", buf);
+    }
+
+    if (server->sioc) {
+        close_client(server);
+        server->sioc = NULL;
+    }
+
+    if (server->device_panic_notifier) {
+        server->device_panic_notifier(server);
+    }
+}
+
+static QIOChannel *slave_io_channel(VuServer *server, int fd,
+                                    Error **local_err)
+{
+    if (server->sioc_slave) {
+        if (fd == server->sioc_slave->fd) {
+            return server->ioc_slave;
+        }
+    } else {
+        server->sioc_slave = qio_channel_socket_new_fd(fd, local_err);
+        if (!*local_err) {
+            server->ioc_slave = QIO_CHANNEL(server->sioc_slave);
+            return server->ioc_slave;
+        }
+    }
+
+    return NULL;
+}
+
+static bool coroutine_fn
+vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
+{
+    struct iovec iov = {
+        .iov_base = (char *)vmsg,
+        .iov_len = VHOST_USER_HDR_SIZE,
+    };
+    int rc, read_bytes = 0;
+    Error *local_err = NULL;
+    /*
+     * Store fds/nfds returned from qio_channel_readv_full into
+     * temporary variables.
+     *
+     * VhostUserMsg is a packed structure, gcc will complain about passing
+     * pointer to a packed structure member if we pass &VhostUserMsg.fd_num
+     * and &VhostUserMsg.fds directly when calling qio_channel_readv_full,
+     * thus two temporary variables nfds and fds are used here.
+     */
+    size_t nfds = 0, nfds_t = 0;
+    int *fds_t = NULL;
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    QIOChannel *ioc = NULL;
+
+    if (conn_fd == server->sioc->fd) {
+        ioc = server->ioc;
+    } else {
+        /* Slave communication will also use this function to read msg */
+        ioc = slave_io_channel(server, conn_fd, &local_err);
+    }
+
+    if (!ioc) {
+        error_report_err(local_err);
+        goto fail;
+    }
+
+    assert(qemu_in_coroutine());
+    do {
+        /*
+         * qio_channel_readv_full may have short reads, keeping calling it
+         * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
+         */
+        rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &local_err);
+        if (rc < 0) {
+            if (rc == QIO_CHANNEL_ERR_BLOCK) {
+                qio_channel_yield(ioc, G_IO_IN);
+                continue;
+            } else {
+                error_report_err(local_err);
+                return false;
+            }
+        }
+        read_bytes += rc;
+        if (nfds_t > 0) {
+            if (nfds + nfds_t > G_N_ELEMENTS(vmsg->fds)) {
+                error_report("A maximum of %d fds are allowed, "
+                             "however got %lu fds now",
+                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
+                goto fail;
+            }
+            memcpy(vmsg->fds + nfds, fds_t,
+                   nfds_t *sizeof(vmsg->fds[0]));
+            nfds += nfds_t;
+            g_free(fds_t);
+        }
+        if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
+            break;
+        }
+        iov.iov_base = (char *)vmsg + read_bytes;
+        iov.iov_len = VHOST_USER_HDR_SIZE - read_bytes;
+    } while (true);
+
+    vmsg->fd_num = nfds;
+    /* qio_channel_readv_full will make socket fds blocking, unblock them */
+    vmsg_unblock_fds(vmsg);
+    if (vmsg->size > sizeof(vmsg->payload)) {
+        error_report("Error: too big message request: %d, "
+                     "size: vmsg->size: %u, "
+                     "while sizeof(vmsg->payload) = %zu",
+                     vmsg->request, vmsg->size, sizeof(vmsg->payload));
+        goto fail;
+    }
+
+    struct iovec iov_payload = {
+        .iov_base = (char *)&vmsg->payload,
+        .iov_len = vmsg->size,
+    };
+    if (vmsg->size) {
+        rc = qio_channel_readv_all_eof(ioc, &iov_payload, 1, &local_err);
+        if (rc == -1) {
+            error_report_err(local_err);
+            goto fail;
+        }
+    }
+
+    return true;
+
+fail:
+    vmsg_close_fds(vmsg);
+
+    return false;
+}
+
+
+static void vu_client_start(VuServer *server);
+static coroutine_fn void vu_client_trip(void *opaque)
+{
+    VuServer *server = opaque;
+
+    while (!server->aio_context_changed && server->sioc) {
+        vu_dispatch(&server->vu_dev);
+    }
+
+    if (server->aio_context_changed && server->sioc) {
Quoted 8 lines

util/vhost-user-server.c line 202

Quoted 4 lines
util/vhost-user-server.c new file +4 -0
+        server->aio_context_changed = false;
+        vu_client_start(server);
+    }
+}

This is somewhat convoluted, but ok. As soon as my patch "util/async:
Add aio_co_reschedule_self()" is merged, we can use it to simplify this a bit.

I will simplify this when your patch is merged.

Quoted 107 lines
Quoted 107 lines
util/vhost-user-server.c new file +106 -0
+static void vu_client_start(VuServer *server)
+{
+    server->co_trip = qemu_coroutine_create(vu_client_trip, server);
+    aio_co_enter(server->ctx, server->co_trip);
+}
+
+/*
+ * a wrapper for vu_kick_cb
+ *
+ * since aio_dispatch can only pass one user data pointer to the
+ * callback function, pack VuDev and pvt into a struct. Then unpack it
+ * and pass them to vu_kick_cb
+ */
+static void kick_handler(void *opaque)
+{
+    KickInfo *kick_info = opaque;
+    kick_info->cb(kick_info->vu_dev, 0, (void *) kick_info->index);
+}
+
+
+static void
+set_watch(VuDev *vu_dev, int fd, int vu_evt,
+          vu_watch_cb cb, void *pvt)
+{
+
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+    long index = (intptr_t) pvt;
+    g_assert(cb);
+    KickInfo *kick_info = &server->kick_info[index];
+    if (!kick_info->cb) {
+        kick_info->fd = fd;
+        kick_info->cb = cb;
+        qemu_set_nonblock(fd);
+        aio_set_fd_handler(server->ioc->ctx, fd, false, kick_handler,
+                           NULL, NULL, kick_info);
+        kick_info->vu_dev = vu_dev;
+    }
+}
+
+
+static void remove_watch(VuDev *vu_dev, int fd)
+{
+    VuServer *server;
+    int i;
+    int index = -1;
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+
+    server = container_of(vu_dev, VuServer, vu_dev);
+    for (i = 0; i < vu_dev->max_queues; i++) {
+        if (server->kick_info[i].fd == fd) {
+            index = i;
+            break;
+        }
+    }
+
+    if (index == -1) {
+        return;
+    }
+    server->kick_info[i].cb = NULL;
+    aio_set_fd_handler(server->ioc->ctx, fd, false, NULL, NULL, NULL, NULL);
+}
+
+
+static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
+                      gpointer opaque)
+{
+    VuServer *server = opaque;
+
+    if (server->sioc) {
+        warn_report("Only one vhost-user client is allowed to "
+                    "connect the server one time");
+        return;
+    }
+
+    if (!vu_init(&server->vu_dev, server->max_queues, sioc->fd, panic_cb,
+                 vu_message_read, set_watch, remove_watch, server->vu_iface)) {
+        error_report("Failed to initialized libvhost-user");
+        return;
+    }
+
+    /*
+     * Unset the callback function for network listener to make another
+     * vhost-user client keeping waiting until this client disconnects
+     */
+    qio_net_listener_set_client_func(server->listener,
+                                     NULL,
+                                     NULL,
+                                     NULL);
+    server->sioc = sioc;
+    server->kick_info = g_new0(KickInfo, server->max_queues);
+    /*
+     * Increase the object reference, so sioc will not freed by
+     * qio_net_listener_channel_func which will call object_unref(OBJECT(sioc))
+     */
+    object_ref(OBJECT(server->sioc));
+    qio_channel_set_name(QIO_CHANNEL(sioc), "vhost-user client");
+    server->ioc = QIO_CHANNEL(sioc);
+    object_ref(OBJECT(server->ioc));
+    qio_channel_attach_aio_context(server->ioc, server->ctx);
+    qio_channel_set_blocking(QIO_CHANNEL(server->sioc), false, NULL);
+    vu_client_start(server);
+}
+
Quoted 8 lines

util/vhost-user-server.c line 315

Quoted 6 lines
util/vhost-user-server.c new file +6 -0
+
+void vhost_user_server_stop(VuServer *server)
+{
+    if (!server) {
+        return;
+    }

There is no reason why the caller should even pass NULL.

Removed in v10.

Quoted 9 lines

util/vhost-user-server.c line 319

Quoted 3 lines
util/vhost-user-server.c new file +3 -0
+    if (server->sioc) {
+        close_client(server);
+        object_unref(OBJECT(server->sioc));

close_client() already unrefs it. Do we really hold two references? If so, why?

I can see that vu_accept() takes an extra reference, but the comment there says this is because QIOChannel takes ownership.

This is my neglect! Thank you!

Quoted 44 lines

util/vhost-user-server.c line 327

Quoted 8 lines
util/vhost-user-server.c new file +8 -0
+    }
+
+    if (server->listener) {
+        qio_net_listener_disconnect(server->listener);
+        object_unref(OBJECT(server->listener));
+    }
+
+    g_free(server->kick_info);

Don't we need to wait for co_trip to terminate somewhere? Probably before freeing any objects because it could still use them.

I assume vhost_user_server_stop() is always called from the main thread whereas co_trip runs in the server AioContext, so extra care is necessary.

Quoted 28 lines
util/vhost-user-server.c new file +28 -0
+}
+
+static void detach_context(VuServer *server)
+{
+    int i;
+    AioContext *ctx = server->ioc->ctx;
+    qio_channel_detach_aio_context(server->ioc);
+    for (i = 0; i < server->vu_dev.max_queues; i++) {
+        if (server->kick_info[i].cb) {
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, false, NULL,
+                               NULL, NULL, NULL);
+        }
+    }
+}
+
+static void attach_context(VuServer *server, AioContext *ctx)
+{
+    int i;
+    qio_channel_attach_aio_context(server->ioc, ctx);
+    server->aio_context_changed = true;
+    if (server->co_trip) {
+        aio_co_schedule(ctx, server->co_trip);
+    }
+    for (i = 0; i < server->vu_dev.max_queues; i++) {
+        if (server->kick_info[i].cb) {
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, false,
+                               kick_handler, NULL, NULL,
+                               &server->kick_info[i]);
Quoted 8 lines

util/vhost-user-server.c line 27

Quoted 3 lines
util/vhost-user-server.c new file +3 -0
+        }
+    }
+}

There is a lot of duplication between detach_context() and attach_context(). I think implementing this directly in vhost_user_server_set_aio_context() for both cases at once would result in simpler code.

Thank you for the advice! In v10, both cases have been dealt with in vhost_user_server_set_aio_context since in both cases we need to iterate over the kick handlers.

Quoted 10 lines
Quoted 10 lines
util/vhost-user-server.c new file +9 -0
+void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server)
+{
+    server->ctx = ctx ? ctx : qemu_get_aio_context();
+    if (!server->sioc) {
+        return;
+    }
+    if (ctx) {
+        attach_context(server, ctx);
+    } else {
Quoted 8 lines

util/vhost-user-server.c line 371

Quoted 3 lines
util/vhost-user-server.c new file +3 -0
+        detach_context(server);
+    }
+}

What happens if the VuServer is already attached to an AioContext and you change it to another AioContext? Shouldn't it be detached from the old context and attached to the new one instead of only doing the latter?

Based on my understanding, when there's a change of a block drive's AioConext, it will first call context detachment hook and then call context attachment hook. So this is not an issue.

Quoted 14 lines

util/vhost-user-server.c line 381

Quoted 10 lines
util/vhost-user-server.c new file +9 -0
+
+bool vhost_user_server_start(VuServer *server,
+                             SocketAddress *socket_addr,
+                             AioContext *ctx,
+                             uint16_t max_queues,
+                             DevicePanicNotifierFn *device_panic_notifier,
+                             const VuDevIface *vu_iface,
+                             Error **errp)
+{

I think this is the function that is supposed to initialise the VuServer object, so would it be better to first zero it out completely?

Quoted 8 lines

Or alternatively assign it completely like this (which automatically zeroes any unspecified field):

   *server = (VuServer) {
       .vu_iface       = vu_iface,
       .max_queues     = max_queues,
       ...
   }

Thank you for the suggestion!

Quoted 71 lines
Quoted 47 lines
util/vhost-user-server.c new file +19 -0
+    server->listener = qio_net_listener_new();
+    if (qio_net_listener_open_sync(server->listener, socket_addr, 1,
+                                   errp) < 0) {
+        return false;
+    }
+
+    qio_net_listener_set_name(server->listener, "vhost-user-backend-listener");
+
+    server->vu_iface = vu_iface;
+    server->max_queues = max_queues;
+    server->ctx = ctx;
+    server->device_panic_notifier = device_panic_notifier;
+    qio_net_listener_set_client_func(server->listener,
+                                     vu_accept,
+                                     server,
+                                     NULL);
+
+    return true;
+}
util/vhost-user-server.h new file +21 -0
new file mode 100644
index 0000000000..5baf58f96a
@@ -0,0 +1,61 @@
+/*
+ * Sharing QEMU devices via vhost-user protocol
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+
+#ifndef VHOST_USER_SERVER_H
+#define VHOST_USER_SERVER_H
+
+#include "contrib/libvhost-user/libvhost-user.h"
+#include "io/channel-socket.h"
+#include "io/channel-file.h"
+#include "io/net-listener.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "standard-headers/linux/virtio_blk.h"
+
+typedef struct KickInfo {

util/vhost-user-server.h line 31

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    VuDev *vu_dev;
+    int fd; /*kick fd*/
+    long index; /*queue index*/
+    vu_watch_cb cb;
+} KickInfo;
+
+typedef struct VuServer {
+    QIONetListener *listener;
+    AioContext *ctx;
+    void (*device_panic_notifier)(struct VuServer *server) ;

Extra space before the semicolon.

util/vhost-user-server.h line 41

Quoted 10 lines
util/vhost-user-server.h new file +10 -0
+    int max_queues;
+    const VuDevIface *vu_iface;
+    VuDev vu_dev;
+    QIOChannel *ioc; /* The I/O channel with the client */
+    QIOChannelSocket *sioc; /* The underlying data channel with the client */
+    /* IOChannel for fd provided via VHOST_USER_SET_SLAVE_REQ_FD */
+    QIOChannel *ioc_slave;
+    QIOChannelSocket *sioc_slave;
+    Coroutine *co_trip; /* coroutine for processing VhostUserMsg */
+    KickInfo *kick_info; /* an array with the length of the queue number */
Quoted 1 line

"an array with @max_queues elements"?

As following Stefan's advice, a linked list is used. So this problem disappears.

Quoted 14 lines
Quoted 14 lines
util/vhost-user-server.h new file +14 -0
+    /* restart coroutine co_trip if AIOContext is changed */
+    bool aio_context_changed;
+} VuServer;
+
+
+typedef void DevicePanicNotifierFn(struct VuServer *server);
+
+bool vhost_user_server_start(VuServer *server,
+                             SocketAddress *unix_socket,
+                             AioContext *ctx,
+                             uint16_t max_queues,
+                             DevicePanicNotifierFn *device_panic_notifier,
+                             const VuDevIface *vu_iface,
+                             Error **errp);
Quoted 9 lines

util/vhost-user-server.h line 61

Quoted 6 lines
util/vhost-user-server.h new file +6 -0
+
+void vhost_user_server_stop(VuServer *server);
+
+void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server);
+
+#endif /* VHOST_USER_SERVER_H */

Kevin

Thank you for reviewing the code!

--
Best regards,
Coiby

#8 lore raw

PATCH 1/6 vhost-user-server: fix VHOST_MEMORY_MAX_REGIONS compiler error

In reply to Coiby Xu

Commit b650d5f4b1cd3f9f8c4fdb319838c5c1e0695e41 ("Lift max ram slots limit in libvhost-user") renamed this constant. Use the array size instead of hard-coding a particular constant in the error message.

---
 util/vhost-user-server.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
util/vhost-user-server.c +2 -2
index 393beeb6b9..e94a8d8a83 100644
@@ -137,9 +137,9 @@ vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
         read_bytes += rc;
         if (nfds_t > 0) {
             if (nfds + nfds_t > G_N_ELEMENTS(vmsg->fds)) {
-                error_report("A maximum of %d fds are allowed, "
+                error_report("A maximum of %zu fds are allowed, "
                              "however got %lu fds now",
-                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
+                             G_N_ELEMENTS(vmsg->fds), nfds + nfds_t);
                 goto fail;
             }
             memcpy(vmsg->fds + nfds, fds_t,
Signature · 1 line
2.26.2
#9 lore raw

PATCH 2/6 vhost-user-server: drop unused #include <eventfd.h>

---
 util/vhost-user-server.c | 1 -
 1 file changed, 1 deletion(-)
util/vhost-user-server.c +0 -1
index e94a8d8a83..49ada8bc78 100644
@@ -7,7 +7,6 @@
  * later.  See the COPYING file in the top-level directory.
  */
 #include "qemu/osdep.h"
-#include <sys/eventfd.h>
 #include "qemu/main-loop.h"
 #include "vhost-user-server.h"
 
Signature · 1 line
2.26.2
author
#10 lore raw

Re: [PATCH 2/6] vhost-user-server: drop unused #include <eventfd.h>

Quoted 18 lines · Stefan Hajnoczi

---
util/vhost-user-server.c | 1 -

1 file changed, 1 deletion(-)
util/vhost-user-server.c +0 -2
index e94a8d8a83..49ada8bc78 100644
@@ -7,7 +7,6 @@
 * later.  See the COPYING file in the top-level directory.
 */
 #include "qemu/osdep.h"
-#include <sys/eventfd.h>
 #include "qemu/main-loop.h"
 #include "vhost-user-server.h"
 
--
Quoted 2 lines

2.26.2

All the patches have been applied to v10. I'm curious how do you find this issue. Is there a tool to detect this issue or simply you are so familiar with the QEMU code that you can spot it very easily?

--
Best regards,
Coiby

#11 lore raw

Re: [PATCH 2/6] vhost-user-server: drop unused #include <eventfd.h>

Quoted 18 lines · Coiby Xu
Quoted 17 lines · Stefan Hajnoczi

---
util/vhost-user-server.c | 1 -

1 file changed, 1 deletion(-)
util/vhost-user-server.c +0 -1
index e94a8d8a83..49ada8bc78 100644
@@ -7,7 +7,6 @@
 * later.  See the COPYING file in the top-level directory.
 */
 #include "qemu/osdep.h"
-#include <sys/eventfd.h>
 #include "qemu/main-loop.h"
 #include "vhost-user-server.h"
 
Quoted 7 lines
Quoted 3 lines

--
2.26.2

All the patches have been applied to v10. I'm curious how do you find this issue. Is there a tool to detect this issue or simply you are so familiar with the QEMU code that you can spot it very easily?

No, I didn't use a tool.

When looking at the code I wondered if the #include was really necessary. So I deleted the #include and recompiled to check that the build still works.

Stefan

#12 lore raw

PATCH 3/6 vhost-user-server: adjust vhost_user_server_set_aio_context() arguments

In reply to Stefan Hajnoczi

vhost_user_server_set_aio_context() operates on a VuServer object. Make that the first argument of the function since it is conventional to define functions with the object they act on as the first argument. In other words, obj_action(obj, args...) is commonly used and not obj_action(arg1, ..., obj, ...).

---
 util/vhost-user-server.h | 2 +-
 util/vhost-user-server.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
util/vhost-user-server.h +1 -1
index 5baf58f96a..584aab3da5 100644
@@ -56,6 +56,6 @@ bool vhost_user_server_start(VuServer *server,
 
 void vhost_user_server_stop(VuServer *server);
 
-void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server);
+void vhost_user_server_set_aio_context(VuServer *server, AioContext *ctx);
 
 #endif /* VHOST_USER_SERVER_H */
util/vhost-user-server.c +1 -1
index 49ada8bc78..5230ba3883 100644
@@ -356,7 +356,7 @@ static void attach_context(VuServer *server, AioContext *ctx)
     }
 }
 
-void vhost_user_server_set_aio_context(AioContext *ctx, VuServer *server)
+void vhost_user_server_set_aio_context(VuServer *server, AioContext *ctx)
 {
     server->ctx = ctx ? ctx : qemu_get_aio_context();
     if (!server->sioc) {
Signature · 1 line
2.26.2
#13 lore raw

PATCH 4/6 vhost-user-server: mark fd handlers "external"

In reply to Stefan Hajnoczi

The event loop has the concept of "external" fd handlers that process requests from outside clients such as the guest. External fd handlers are disabled during critical sections where new requests are not allowed.

The vhost-user-server seems like an "external" client to me and therefore should mark its file descriptors "external".

---
 util/vhost-user-server.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
util/vhost-user-server.c +4 -4
index 5230ba3883..a5785cbf86 100644
@@ -235,7 +235,7 @@ set_watch(VuDev *vu_dev, int fd, int vu_evt,
         kick_info->fd = fd;
         kick_info->cb = cb;
         qemu_set_nonblock(fd);
-        aio_set_fd_handler(server->ioc->ctx, fd, false, kick_handler,
+        aio_set_fd_handler(server->ioc->ctx, fd, true, kick_handler,
                            NULL, NULL, kick_info);
         kick_info->vu_dev = vu_dev;
     }
@@ -262,7 +262,7 @@ static void remove_watch(VuDev *vu_dev, int fd)
         return;
     }
     server->kick_info[i].cb = NULL;
-    aio_set_fd_handler(server->ioc->ctx, fd, false, NULL, NULL, NULL, NULL);
+    aio_set_fd_handler(server->ioc->ctx, fd, true, NULL, NULL, NULL, NULL);
 }
 
 
@@ -333,7 +333,7 @@ static void detach_context(VuServer *server)
     qio_channel_detach_aio_context(server->ioc);
     for (i = 0; i < server->vu_dev.max_queues; i++) {
         if (server->kick_info[i].cb) {
-            aio_set_fd_handler(ctx, server->kick_info[i].fd, false, NULL,
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, true, NULL,
                                NULL, NULL, NULL);
         }
     }
@@ -349,7 +349,7 @@ static void attach_context(VuServer *server, AioContext *ctx)
     }
     for (i = 0; i < server->vu_dev.max_queues; i++) {
         if (server->kick_info[i].cb) {
-            aio_set_fd_handler(ctx, server->kick_info[i].fd, false,
+            aio_set_fd_handler(ctx, server->kick_info[i].fd, true,
                                kick_handler, NULL, NULL,
                                &server->kick_info[i]);
         }
Signature · 1 line
2.26.2
#14 lore raw

PATCH 5/6 vhost-user-server: fix s/initialized/initialize/ typo

In reply to Stefan Hajnoczi

The present tense is used here.

---
 util/vhost-user-server.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
util/vhost-user-server.c +1 -1
index a5785cbf86..42a51d190c 100644
@@ -279,7 +279,7 @@ static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
 
     if (!vu_init(&server->vu_dev, server->max_queues, sioc->fd, panic_cb,
                  vu_message_read, set_watch, remove_watch, server->vu_iface)) {
-        error_report("Failed to initialized libvhost-user");
+        error_report("Failed to initialize libvhost-user");
         return;
     }
 
Signature · 1 line
2.26.2
#15 lore raw

PATCH 6/6 vhost-user-server: use DevicePanicNotifierFn everywhere

In reply to Stefan Hajnoczi

Use the DevicePanicNotifierFn typedef instead of open-coding the function pointer. Writing the code this way avoids duplicating the function prototype.

Also use the VuServer typedef instead of struct VuServer as required by QEMU's coding style.

---
 util/vhost-user-server.h | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
util/vhost-user-server.h +6 -6
index 584aab3da5..37aca1e5aa 100644
@@ -25,10 +25,13 @@ typedef struct KickInfo {
     vu_watch_cb cb;
 } KickInfo;
 
-typedef struct VuServer {
+typedef struct VuServer VuServer;
+typedef void DevicePanicNotifierFn(VuServer *server);
+
+struct VuServer {
     QIONetListener *listener;
     AioContext *ctx;
-    void (*device_panic_notifier)(struct VuServer *server) ;
+    DevicePanicNotifierFn *device_panic_notifier;
     int max_queues;
     const VuDevIface *vu_iface;
     VuDev vu_dev;
@@ -41,10 +44,7 @@ typedef struct VuServer {
     KickInfo *kick_info; /* an array with the length of the queue number */
     /* restart coroutine co_trip if AIOContext is changed */
     bool aio_context_changed;
-} VuServer;
-
-
-typedef void DevicePanicNotifierFn(struct VuServer *server);
+};
 
 bool vhost_user_server_start(VuServer *server,
                              SocketAddress *unix_socket,
Signature · 1 line
2.26.2
#16 lore raw

Re: [PATCH v9 2/5] generic vhost user server

In reply to Coiby Xu

util/vhost-user-server.c line 220

Quoted 11 lines · Coiby Xu
util/vhost-user-server.c new file +11 -0
+/*
+ * a wrapper for vu_kick_cb
+ *
+ * since aio_dispatch can only pass one user data pointer to the
+ * callback function, pack VuDev and pvt into a struct. Then unpack it
+ * and pass them to vu_kick_cb
+ */
+static void kick_handler(void *opaque)
+{
+    KickInfo *kick_info = opaque;
+    kick_info->cb(kick_info->vu_dev, 0, (void *) kick_info->index);

Where is kick_info->index assigned? It appears to be NULL in all cases.

util/vhost-user-server.c line 232

Quoted 12 lines
util/vhost-user-server.c new file +12 -0
+}
+
+
+static void
+set_watch(VuDev *vu_dev, int fd, int vu_evt,
+          vu_watch_cb cb, void *pvt)
+{
+
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+    long index = (intptr_t) pvt;

The meaning of the pvt argument is not defined in the library interface. set_watch() callbacks shouldn't interpret pvt.

You could modify libvhost-user to explicitly pass the virtqueue index (or -1 if the fd is not associated with a virtqueue), but it's nice to avoid libvhost-user API changes so that existing libvhost-user applications don't require modifications.

What I would do here is to change the ->kick_info[] data struct. How about a linked list of VuFdWatch objects? That way the code can handle any number of fd watches and doesn't make assumptions about virtqueues. set_watch() is a generic fd monitoring interface and doesn't need to be tied to virtqueues.

author
#17 lore raw

Re: [PATCH v9 2/5] generic vhost user server

Quoted 36 lines · Stefan Hajnoczi

util/vhost-user-server.c line 220

Quoted 11 lines · Coiby Xu
util/vhost-user-server.c new file +11 -0
+/*
+ * a wrapper for vu_kick_cb
+ *
+ * since aio_dispatch can only pass one user data pointer to the
+ * callback function, pack VuDev and pvt into a struct. Then unpack it
+ * and pass them to vu_kick_cb
+ */
+static void kick_handler(void *opaque)
+{
+    KickInfo *kick_info = opaque;
+    kick_info->cb(kick_info->vu_dev, 0, (void *) kick_info->index);

Where is kick_info->index assigned? It appears to be NULL in all cases.

util/vhost-user-server.c line 232

Quoted 12 lines
util/vhost-user-server.c new file +12 -0
+}
+
+
+static void
+set_watch(VuDev *vu_dev, int fd, int vu_evt,
+          vu_watch_cb cb, void *pvt)
+{
+
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    g_assert(vu_dev);
+    g_assert(fd >= 0);
+    long index = (intptr_t) pvt;

The meaning of the pvt argument is not defined in the library interface. set_watch() callbacks shouldn't interpret pvt.

You could modify libvhost-user to explicitly pass the virtqueue index (or -1 if the fd is not associated with a virtqueue), but it's nice to avoid libvhost-user API changes so that existing libvhost-user applications don't require modifications.

Quoted 5 lines

What I would do here is to change the ->kick_info[] data struct. How about a linked list of VuFdWatch objects? That way the code can handle any number of fd watches and doesn't make assumptions about virtqueues. set_watch() is a generic fd monitoring interface and doesn't need to be tied to virtqueues.

A linked list of VuFdWatch objects has been adopted in v10. Thank you!

--
Best regards,
Coiby

author
#18 lore raw

PATCH v9 3/5 move logical block size check function to a common utility function

In reply to Coiby Xu

Move logical block size check function in hw/core/qdev-properties.c:set_blocksize() to util/block-helpers.c

---
 hw/core/qdev-properties.c | 18 +++------------
 util/Makefile.objs        |  1 +
 util/block-helpers.c      | 46 +++++++++++++++++++++++++++++++++++++++
 util/block-helpers.h      |  7 ++++++
 4 files changed, 57 insertions(+), 15 deletions(-)
 create mode 100644 util/block-helpers.c
 create mode 100644 util/block-helpers.h
hw/core/qdev-properties.c +3 -15
index cc924815da..a4a6aa5204 100644
@@ -14,6 +14,7 @@
 #include "qapi/visitor.h"
 #include "chardev/char.h"
 #include "qemu/uuid.h"
+#include "util/block-helpers.h"
 
 void qdev_prop_set_after_realize(DeviceState *dev, const char *name,
                                   Error **errp)
@@ -736,8 +737,6 @@ static void set_blocksize(Object *obj, Visitor *v, const char *name,
     Property *prop = opaque;
     uint16_t value, *ptr = qdev_get_prop_ptr(dev, prop);
     Error *local_err = NULL;
-    const int64_t min = 512;
-    const int64_t max = 32768;
 
     if (dev->realized) {
         qdev_prop_set_after_realize(dev, name, errp);
@@ -749,21 +748,10 @@ static void set_blocksize(Object *obj, Visitor *v, const char *name,
         error_propagate(errp, local_err);
         return;
     }
-    /* value of 0 means "unset" */
-    if (value && (value < min || value > max)) {
-        error_setg(errp, QERR_PROPERTY_VALUE_OUT_OF_RANGE,
-                   dev->id ? : "", name, (int64_t)value, min, max);
+    check_logical_block_size(dev->id ? : "", name, value, errp);
+    if (errp) {
         return;
     }
-
-    /* We rely on power-of-2 blocksizes for bitmasks */
-    if ((value & (value - 1)) != 0) {
-        error_setg(errp,
-                  "Property %s.%s doesn't take value '%" PRId64 "', it's not a power of 2",
-                  dev->id ?: "", name, (int64_t)value);
-        return;
-    }
-
     *ptr = value;
 }
 
util/Makefile.objs +1 -0
index b4d4af06dc..fa5380ddab 100644
@@ -66,6 +66,7 @@ util-obj-y += hbitmap.o
 util-obj-y += main-loop.o
 util-obj-y += nvdimm-utils.o
 util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
+util-obj-y += block-helpers.o
 util-obj-$(CONFIG_LINUX) += vhost-user-server.o
 util-obj-y += qemu-coroutine-sleep.o
 util-obj-y += qemu-co-shared-resource.o
util/block-helpers.c new file +46 -0

46 lines not on this page · raw message

util/block-helpers.h new file +7 -0
new file mode 100644
index 0000000000..f06be282a1
@@ -0,0 +1,7 @@
+#ifndef BLOCK_HELPERS_H
+#define BLOCK_HELPERS_H
+
+void check_logical_block_size(const char *id, const char *name, uint16_t value,
+                     Error **errp);
+
+#endif /* BLOCK_HELPERS_H */
Signature · 1 line
2.27.0
#19 lore raw

Re: [PATCH v9 3/5] move logical block size check function to a common utility function

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Quoted 3 lines

Move logical block size check function in hw/core/qdev-properties.c:set_blocksize() to util/block-helpers.c

Just a heads-up that you'll probably need to rebase this after the pull request I sent yesterday is merged because it changes the block size properties.

Kevin

#20 lore raw

PATCH 1/6 block-helpers: move MIN/MAX_BLOCK_SIZE constants into header file

In reply to Coiby Xu

Move the constants from hw/core/qdev-properties.c to util/block-helpers.h so that knowledge of the min/max values is encapsulated in block-helpers code.

Callers should not assume specific min/max values. In fact, the values in hw/core/qdev-properties.c and util/block-helpers.c did not match. Use the hw/core/qdev-properties.c values since that's what existing code expects.

---
 util/block-helpers.h      | 12 ++++++++++++
 hw/core/qdev-properties.c | 11 -----------
 util/block-helpers.c      |  7 ++-----
 3 files changed, 14 insertions(+), 16 deletions(-)
util/block-helpers.h +12 -0
index f06be282a1..46975ca7af 100644
@@ -1,6 +1,18 @@
 #ifndef BLOCK_HELPERS_H
 #define BLOCK_HELPERS_H
 
+#include "qemu/units.h"
+
+/* lower limit is sector size */
+#define MIN_BLOCK_SIZE          INT64_C(512)
+#define MIN_BLOCK_SIZE_STR      "512 B"
+/*
+ * upper limit is arbitrary, 2 MiB looks sufficient for all sensible uses, and
+ * matches qcow2 cluster size limit
+ */
+#define MAX_BLOCK_SIZE          (2 * MiB)
+#define MAX_BLOCK_SIZE_STR      "2 MiB"
+
 void check_logical_block_size(const char *id, const char *name, uint16_t value,
                      Error **errp);
 
hw/core/qdev-properties.c +0 -11
index b478f100af..03981feb02 100644
@@ -14,7 +14,6 @@
 #include "qapi/visitor.h"
 #include "chardev/char.h"
 #include "qemu/uuid.h"
-#include "qemu/units.h"
 #include "util/block-helpers.h"
 
 void qdev_prop_set_after_realize(DeviceState *dev, const char *name,
@@ -782,16 +781,6 @@ const PropertyInfo qdev_prop_size32 = {
 
 /* --- blocksize --- */
 
-/* lower limit is sector size */
-#define MIN_BLOCK_SIZE          512
-#define MIN_BLOCK_SIZE_STR      "512 B"
-/*
- * upper limit is arbitrary, 2 MiB looks sufficient for all sensible uses, and
- * matches qcow2 cluster size limit
- */
-#define MAX_BLOCK_SIZE          (2 * MiB)
-#define MAX_BLOCK_SIZE_STR      "2 MiB"
-
 static void set_blocksize(Object *obj, Visitor *v, const char *name,
                           void *opaque, Error **errp)
 {
util/block-helpers.c +2 -5
index d31309cc0e..089fe3401d 100644
@@ -25,13 +25,10 @@
 void check_logical_block_size(const char *id, const char *name, uint16_t value,
                      Error **errp)
 {
-    const int64_t min = 512;
-    const int64_t max = 32768;
-
     /* value of 0 means "unset" */
-    if (value && (value < min || value > max)) {
+    if (value && (value < MIN_BLOCK_SIZE || value > MAX_BLOCK_SIZE)) {
         error_setg(errp, QERR_PROPERTY_VALUE_OUT_OF_RANGE,
-                   id, name, (int64_t)value, min, max);
+                   id, name, value, MIN_BLOCK_SIZE, MAX_BLOCK_SIZE);
         return;
     }
 
Signature · 1 line
2.26.2
#21 lore raw

PATCH 2/6 block-helpers: switch to int64_t block size values

The uint16_t type is too small for MAX_BLOCK_SIZE (2 MiB). The int64_t type is widely used in QEMU as a type for disk offsets and sizes, so it's an appropriate type to use here. It will work for all callers.

---
 util/block-helpers.h | 2 +-
 util/block-helpers.c | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
util/block-helpers.h +1 -1
index 46975ca7af..ec6421560c 100644
@@ -13,7 +13,7 @@
 #define MAX_BLOCK_SIZE          (2 * MiB)
 #define MAX_BLOCK_SIZE_STR      "2 MiB"
 
-void check_logical_block_size(const char *id, const char *name, uint16_t value,
+void check_logical_block_size(const char *id, const char *name, int64_t value,
                      Error **errp);
 
 #endif /* BLOCK_HELPERS_H */
util/block-helpers.c +2 -2
index 089fe3401d..9e68954c46 100644
@@ -22,7 +22,7 @@
  *
  *  Moved from hw/core/qdev-properties.c:set_blocksize()
  */
-void check_logical_block_size(const char *id, const char *name, uint16_t value,
+void check_logical_block_size(const char *id, const char *name, int64_t value,
                      Error **errp)
 {
     /* value of 0 means "unset" */
@@ -37,7 +37,7 @@ void check_logical_block_size(const char *id, const char *name, uint16_t value,
         error_setg(errp,
                    "Property %s.%s doesn't take value '%" PRId64
                    "', it's not a power of 2",
-                   id, name, (int64_t)value);
+                   id, name, value);
         return;
     }
 }
Signature · 1 line
2.26.2
#22 lore raw

PATCH 3/6 block-helpers: rename check_logical_block_size() to check_block_size()

In reply to Stefan Hajnoczi

Block size checking is the same whether it's a physical, logical, or other block size value. Use a more general name to show this function can be used in other cases too (just like the qdev property that this code originally comes from).

---
 util/block-helpers.h      | 4 ++--
 hw/core/qdev-properties.c | 2 +-
 util/block-helpers.c      | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)
util/block-helpers.h +2 -2
index ec6421560c..b53295a529 100644
@@ -13,7 +13,7 @@
 #define MAX_BLOCK_SIZE          (2 * MiB)
 #define MAX_BLOCK_SIZE_STR      "2 MiB"
 
-void check_logical_block_size(const char *id, const char *name, int64_t value,
-                     Error **errp);
+void check_block_size(const char *id, const char *name, int64_t value,
+                      Error **errp);
 
 #endif /* BLOCK_HELPERS_H */
hw/core/qdev-properties.c +1 -1
index 03981feb02..28a6d8b2ee 100644
@@ -800,7 +800,7 @@ static void set_blocksize(Object *obj, Visitor *v, const char *name,
         error_propagate(errp, local_err);
         return;
     }
-    check_logical_block_size(dev->id ? : "", name, value, errp);
+    check_block_size(dev->id ? : "", name, value, errp);
     if (errp) {
         return;
     }
util/block-helpers.c +2 -2
index 9e68954c46..51d9d02c43 100644
@@ -22,8 +22,8 @@
  *
  *  Moved from hw/core/qdev-properties.c:set_blocksize()
  */
-void check_logical_block_size(const char *id, const char *name, int64_t value,
-                     Error **errp)
+void check_block_size(const char *id, const char *name, int64_t value,
+                      Error **errp)
 {
     /* value of 0 means "unset" */
     if (value && (value < MIN_BLOCK_SIZE || value > MAX_BLOCK_SIZE)) {
Signature · 1 line
2.26.2
#23 lore raw

PATCH 4/6 block-helpers: use local_err in case errp is NULL

In reply to Stefan Hajnoczi

If we use errp then we won't know when check_block_size() fails when errp is NULL.

The purpose of local_err is to detect an error has occurred even if the caller doesn't care about the specific error and has passed a NULL errp.

---
 hw/core/qdev-properties.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
hw/core/qdev-properties.c +3 -2
index 28a6d8b2ee..0a651c7d32 100644
@@ -800,8 +800,9 @@ static void set_blocksize(Object *obj, Visitor *v, const char *name,
         error_propagate(errp, local_err);
         return;
     }
-    check_block_size(dev->id ? : "", name, value, errp);
-    if (errp) {
+    check_block_size(dev->id ? : "", name, value, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
         return;
     }
     *ptr = value;
Signature · 1 line
2.26.2
#24 lore raw

PATCH 5/6 block-helpers: keep the copyright line from the original file

In reply to Stefan Hajnoczi

The check_block_size() code comes from hw/core/qdev-properties.c. Keep the copyright.

---
 util/block-helpers.c | 1 +
 1 file changed, 1 insertion(+)
util/block-helpers.c +1 -0
index 51d9d02c43..9d12368032 100644
@@ -1,6 +1,7 @@
 /*
  * Block utility functions
  *
+ * Copyright IBM, Corp. 2011
  * Copyright (c) 2020 Coiby Xu <[email protected]>
  *
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
Signature · 1 line
2.26.2
#25 lore raw

PATCH 6/6 block-helpers: update doc comment in gtkdoc style

In reply to Stefan Hajnoczi

QEMU uses the gtkdoc style for API doc comments. For examples, see include/qom/object.h.

Fully document the function with up-to-date information (the min/max values were outdated).

---
 util/block-helpers.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
util/block-helpers.c +10 -8
index 9d12368032..c4851432f5 100644
@@ -13,15 +13,17 @@
 #include "qapi/qmp/qerror.h"
 #include "block-helpers.h"
 
-/*
- * Logical block size input validation
+/**
+ * check_block_size:
+ * @id: The unique ID of the object
+ * @name: The name of the property being validated
+ * @value: The block size in bytes
+ * @errp: A pointer to an area to store an error
  *
- * The size should meet the following conditions:
- * 1. min=512
- * 2. max=32768
- * 3. a power of 2
- *
- *  Moved from hw/core/qdev-properties.c:set_blocksize()
+ * This function checks that the block size meets the following conditions:
+ * 1. At least MIN_BLOCK_SIZE
+ * 2. No larger than MAX_BLOCK_SIZE
+ * 3. A power of 2
  */
 void check_block_size(const char *id, const char *name, int64_t value,
                       Error **errp)
Signature · 1 line
2.26.2
author
#26 lore raw

PATCH v9 4/5 vhost-user block device backend server

In reply to Coiby Xu

By making use of libvhost-user, block device drive can be shared to the connected vhost-user client. Only one client can connect to the server one time.

Since vhost-user-server needs a block drive to be created first, delay the creation of this object.

---
 block/Makefile.objs                  |   1 +
 block/export/vhost-user-blk-server.c | 669 +++++++++++++++++++++++++++
 block/export/vhost-user-blk-server.h |  35 ++
 softmmu/vl.c                         |   4 +
 4 files changed, 709 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
block/Makefile.objs +1 -0
index 3635b6b4c1..0eb7eff470 100644
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o ../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
block/export/vhost-user-blk-server.c new file +669 -0

669 lines not on this page · raw message

block/export/vhost-user-blk-server.h new file +35 -0
new file mode 100644
index 0000000000..5398e5d352
@@ -0,0 +1,35 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+
+#ifndef VHOST_USER_BLK_SERVER_H
+#define VHOST_USER_BLK_SERVER_H
+#include "util/vhost-user-server.h"
+
+typedef struct VuBlockDev VuBlockDev;
+#define TYPE_VHOST_USER_BLK_SERVER "vhost-user-blk-server"
+#define VHOST_USER_BLK_SERVER(obj) \
+   OBJECT_CHECK(VuBlockDev, obj, TYPE_VHOST_USER_BLK_SERVER)
+
+/* vhost user block device */
+struct VuBlockDev {
+    Object parent_obj;
+    char *node_name;
+    SocketAddress *addr;
+    AioContext *ctx;
+    VuServer vu_server;
+    bool running;
+    uint32_t blk_size;
+    BlockBackend *backend;
+    QIOChannelSocket *sioc;
+    QTAILQ_ENTRY(VuBlockDev) next;
+    struct virtio_blk_config blkcfg;
+    bool writable;
+};
+
+#endif /* VHOST_USER_BLK_SERVER_H */
softmmu/vl.c +4 -0
index 05d1a4cb6b..838df3e57a 100644
@@ -2520,6 +2520,10 @@ static bool object_create_initial(const char *type, QemuOpts *opts)
     }
 #endif
 
+    /* Reason: vhost-user-blk-server property "node-name" */
+    if (g_str_equal(type, "vhost-user-blk-server")) {
+        return false;
+    }
     /*
      * Reason: filter-* property "netdev" etc.
      */
Signature · 1 line
2.27.0
#27 lore raw

Re: [PATCH v9 4/5] vhost-user block device backend server

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Quoted 257 lines

By making use of libvhost-user, block device drive can be shared to the connected vhost-user client. Only one client can connect to the server one time.

Since vhost-user-server needs a block drive to be created first, delay the creation of this object.

---
 block/Makefile.objs                  |   1 +
 block/export/vhost-user-blk-server.c | 669 +++++++++++++++++++++++++++
 block/export/vhost-user-blk-server.h |  35 ++
 softmmu/vl.c                         |   4 +
 4 files changed, 709 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
block/Makefile.objs +1 -0
index 3635b6b4c1..0eb7eff470 100644
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o ../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
block/export/vhost-user-blk-server.c new file +222 -0
new file mode 100644
index 0000000000..bbf2ceaa9b
@@ -0,0 +1,669 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+#include "util/block-helpers.h"
+
+enum {
+    VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+    unsigned char status;
+};
+
+
+typedef struct VuBlockReq {
+    VuVirtqElement *elem;
+    int64_t sector_num;
+    size_t size;
+    struct virtio_blk_inhdr *in;
+    struct virtio_blk_outhdr out;
+    VuServer *server;
+    struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+    VuDev *vu_dev = &req->server->vu_dev;
+
+    /* IO size with 1 extra status byte */
+    vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+    vu_queue_notify(vu_dev, req->vq);
+
+    if (req->elem) {
+        free(req->elem);
+    }
+
+    g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
+{
+    return container_of(server, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+                              uint32_t iovcnt, uint32_t type)
+{
+    struct virtio_blk_discard_write_zeroes desc;
+    ssize_t size = iov_to_buf(iov, iovcnt, 0, &desc, sizeof(desc));
+    if (unlikely(size != sizeof(desc))) {
+        error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+        return -EINVAL;
+    }
+
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+    uint64_t range[2] = { le64_to_cpu(desc.sector) << 9,
+                          le32_to_cpu(desc.num_sectors) << 9 };
+    if (type == VIRTIO_BLK_T_DISCARD) {
+        if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+            return 0;
+        }
+    } else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+        if (blk_co_pwrite_zeroes(vdev_blk->backend,
+                                 range[0], range[1], 0) == 0) {
+            return 0;
+        }
+    }
+
+    return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+    BlockBackend *backend = vdev_blk->backend;
+    blk_co_flush(backend);
+}
+
+
+struct req_data {
+    VuServer *server;
+    VuVirtq *vq;
+    VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+    struct req_data *data = opaque;
+    VuServer *server = data->server;
+    VuVirtq *vq = data->vq;
+    VuVirtqElement *elem = data->elem;
+    uint32_t type;
+    VuBlockReq *req;
+
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    BlockBackend *backend = vdev_blk->backend;
+
+    struct iovec *in_iov = elem->in_sg;
+    struct iovec *out_iov = elem->out_sg;
+    unsigned in_num = elem->in_num;
+    unsigned out_num = elem->out_num;
+    /* refer to hw/block/virtio_blk.c */
+    if (elem->out_num < 1 || elem->in_num < 1) {
+        error_report("virtio-blk request missing headers");
+        free(elem);
+        return;
+    }
+
+    req = g_new0(VuBlockReq, 1);
+    req->server = server;
+    req->vq = vq;
+    req->elem = elem;
+
+    if (unlikely(iov_to_buf(out_iov, out_num, 0, &req->out,
+                            sizeof(req->out)) != sizeof(req->out))) {
+        error_report("virtio-blk request outhdr too short");
+        goto err;
+    }
+
+    iov_discard_front(&out_iov, &out_num, sizeof(req->out));
+
+    if (in_iov[in_num - 1].iov_len < sizeof(struct virtio_blk_inhdr)) {
+        error_report("virtio-blk request inhdr too short");
+        goto err;
+    }
+
+    /* We always touch the last byte, so just see how big in_iov is.  */
+    req->in = (void *)in_iov[in_num - 1].iov_base
+              + in_iov[in_num - 1].iov_len
+              - sizeof(struct virtio_blk_inhdr);
+    iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
+
+
+    type = le32_to_cpu(req->out.type);
+    switch (type & ~VIRTIO_BLK_T_BARRIER) {
+    case VIRTIO_BLK_T_IN:
+    case VIRTIO_BLK_T_OUT: {
+        ssize_t ret = 0;
+        bool is_write = type & VIRTIO_BLK_T_OUT;
+        req->sector_num = le64_to_cpu(req->out.sector);
+
+        int64_t offset = req->sector_num * vdev_blk->blk_size;
+        QEMUIOVector qiov;
+        if (is_write) {
+            qemu_iovec_init_external(&qiov, out_iov, out_num);
+            ret = blk_co_pwritev(backend, offset, qiov.size,
+                                 &qiov, 0);
+        } else {
+            qemu_iovec_init_external(&qiov, in_iov, in_num);
+            ret = blk_co_preadv(backend, offset, qiov.size,
+                                &qiov, 0);
+        }
+        if (ret >= 0) {
+            req->in->status = VIRTIO_BLK_S_OK;
+        } else {
+            req->in->status = VIRTIO_BLK_S_IOERR;
+        }
+        break;
+    }
+    case VIRTIO_BLK_T_FLUSH:
+        vu_block_flush(req);
+        req->in->status = VIRTIO_BLK_S_OK;
+        break;
+    case VIRTIO_BLK_T_GET_ID: {
+        size_t size = MIN(iov_size(&elem->in_sg[0], in_num),
+                          VIRTIO_BLK_ID_BYTES);
+        snprintf(elem->in_sg[0].iov_base, size, "%s", "vhost_user_blk_server");
+        req->in->status = VIRTIO_BLK_S_OK;
+        req->size = elem->in_sg[0].iov_len;
+        break;
+    }
+    case VIRTIO_BLK_T_DISCARD:
+    case VIRTIO_BLK_T_WRITE_ZEROES: {
+        int rc;
+        rc = vu_block_discard_write_zeroes(req, &elem->out_sg[1],
+                                           out_num, type);
+        if (rc == 0) {
+            req->in->status = VIRTIO_BLK_S_OK;
+        } else {
+            req->in->status = VIRTIO_BLK_S_IOERR;
+        }
+        break;
+    }
+    default:
+        req->in->status = VIRTIO_BLK_S_UNSUPP;
+        break;
+    }
+
+    vu_block_req_complete(req);
+    return;
+
+err:
+    free(elem);
+    g_free(req);
+    return;
+}
+
+
+
+static void vu_block_process_vq(VuDev *vu_dev, int idx)
+{
+    VuServer *server;
+    VuVirtq *vq;
+
+    server = container_of(vu_dev, VuServer, vu_dev);
+    assert(server);
+
+    vq = vu_get_queue(vu_dev, idx);
+    assert(vq);

block/export/vhost-user-blk-server.c line 232

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    VuVirtqElement *elem;
+    while (1) {
+        elem = vu_queue_pop(vu_dev, vq, sizeof(VuVirtqElement) +
+                                    sizeof(VuBlockReq));
+        if (elem) {
+            struct req_data req_data = {
+                .server = server,
+                .vq = vq,
+                .elem = elem
+            };

This is on the stack of the function.

block/export/vhost-user-blk-server.c line 235

Quoted 3 lines
block/export/vhost-user-blk-server.c new file +3 -0
+            Coroutine *co = qemu_coroutine_create(vu_block_virtio_process_req,
+                                                  &req_data);
+            aio_co_enter(server->ioc->ctx, co);

Therefore, this code is only correct, if co accesses the data only while the function has not returned yet.

This function is called in the context of vu_dispatch(), which in turn is called from vu_client_trip(). So we already run in a coroutine. In this case, aio_co_enter() only schedules co to run after the current coroutine yields or terminates. In other words, this looks wrong to me because req_data will be accessed when it's long out of scope.

I think we need to malloc it.

Quoted 71 lines
block/export/vhost-user-blk-server.c new file +71 -0
+        } else {
+            break;
+        }
+    }
+}
+
+static void vu_block_queue_set_started(VuDev *vu_dev, int idx, bool started)
+{
+    VuVirtq *vq;
+
+    assert(vu_dev);
+
+    vq = vu_get_queue(vu_dev, idx);
+    vu_set_queue_handler(vu_dev, vq, started ? vu_block_process_vq : NULL);
+}
+
+static uint64_t vu_block_get_features(VuDev *dev)
+{
+    uint64_t features;
+    VuServer *server = container_of(dev, VuServer, vu_dev);
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    features = 1ull << VIRTIO_BLK_F_SIZE_MAX |
+               1ull << VIRTIO_BLK_F_SEG_MAX |
+               1ull << VIRTIO_BLK_F_TOPOLOGY |
+               1ull << VIRTIO_BLK_F_BLK_SIZE |
+               1ull << VIRTIO_BLK_F_FLUSH |
+               1ull << VIRTIO_BLK_F_DISCARD |
+               1ull << VIRTIO_BLK_F_WRITE_ZEROES |
+               1ull << VIRTIO_BLK_F_CONFIG_WCE |
+               1ull << VIRTIO_F_VERSION_1 |
+               1ull << VIRTIO_RING_F_INDIRECT_DESC |
+               1ull << VIRTIO_RING_F_EVENT_IDX |
+               1ull << VHOST_USER_F_PROTOCOL_FEATURES;
+
+    if (!vdev_blk->writable) {
+        features |= 1ull << VIRTIO_BLK_F_RO;
+    }
+
+    return features;
+}
+
+static uint64_t vu_block_get_protocol_features(VuDev *dev)
+{
+    return 1ull << VHOST_USER_PROTOCOL_F_CONFIG |
+           1ull << VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD;
+}
+
+static int
+vu_block_get_config(VuDev *vu_dev, uint8_t *config, uint32_t len)
+{
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    memcpy(config, &vdev_blk->blkcfg, len);
+
+    return 0;
+}
+
+static int
+vu_block_set_config(VuDev *vu_dev, const uint8_t *data,
+                    uint32_t offset, uint32_t size, uint32_t flags)
+{
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    uint8_t wce;
+
+    /* don't support live migration */
+    if (flags != VHOST_SET_CONFIG_TYPE_MASTER) {
+        return -EINVAL;
+    }
+
+

block/export/vhost-user-blk-server.c line 316

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    if (offset != offsetof(struct virtio_blk_config, wce) ||
+        size != 1) {
+        return -EINVAL;
+    }
+
+    wce = *data;
+    if (wce == vdev_blk->blkcfg.wce) {
+        /* Do nothing as same with old configuration */
+        return 0;
+    }

This check is unnecessary. Nothing bad happens if you set the same value again.

Quoted 71 lines
block/export/vhost-user-blk-server.c new file +71 -0
+    vdev_blk->blkcfg.wce = wce;
+    blk_set_enable_write_cache(vdev_blk->backend, wce);
+    return 0;
+}
+
+
+/*
+ * When the client disconnects, it sends a VHOST_USER_NONE request
+ * and vu_process_message will simple call exit which cause the VM
+ * to exit abruptly.
+ * To avoid this issue,  process VHOST_USER_NONE request ahead
+ * of vu_process_message.
+ *
+ */
+static int vu_block_process_msg(VuDev *dev, VhostUserMsg *vmsg, int *do_reply)
+{
+    if (vmsg->request == VHOST_USER_NONE) {
+        dev->panic(dev, "disconnect");
+        return true;
+    }
+    return false;
+}
+
+
+static const VuDevIface vu_block_iface = {
+    .get_features          = vu_block_get_features,
+    .queue_set_started     = vu_block_queue_set_started,
+    .get_protocol_features = vu_block_get_protocol_features,
+    .get_config            = vu_block_get_config,
+    .set_config            = vu_block_set_config,
+    .process_msg           = vu_block_process_msg,
+};
+
+static void blk_aio_attached(AioContext *ctx, void *opaque)
+{
+    VuBlockDev *vub_dev = opaque;
+    aio_context_acquire(ctx);
+    vhost_user_server_set_aio_context(ctx, &vub_dev->vu_server);
+    aio_context_release(ctx);
+}
+
+static void blk_aio_detach(void *opaque)
+{
+    VuBlockDev *vub_dev = opaque;
+    AioContext *ctx = vub_dev->vu_server.ctx;
+    aio_context_acquire(ctx);
+    vhost_user_server_set_aio_context(NULL, &vub_dev->vu_server);
+    aio_context_release(ctx);
+}
+
+
+static void
+vu_block_initialize_config(BlockDriverState *bs,
+                           struct virtio_blk_config *config, uint32_t blk_size)
+{
+    config->capacity = bdrv_getlength(bs) >> BDRV_SECTOR_BITS;
+    config->blk_size = blk_size;
+    config->size_max = 0;
+    config->seg_max = 128 - 2;
+    config->min_io_size = 1;
+    config->opt_io_size = 1;
+    config->num_queues = VHOST_USER_BLK_MAX_QUEUES;
+    config->max_discard_sectors = 32768;
+    config->max_discard_seg = 1;
+    config->discard_sector_alignment = config->blk_size >> 9;
+    config->max_write_zeroes_sectors = 32768;
+    config->max_write_zeroes_seg = 1;
+}
+
+
+static VuBlockDev *vu_block_init(VuBlockDev *vu_block_device, Error **errp)

block/export/vhost-user-blk-server.c line 398

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+{
+
+    BlockBackend *blk;
+    Error *local_error = NULL;
+    const char *node_name = vu_block_device->node_name;
+    bool writable = vu_block_device->writable;
+    /*
+     * Don't allow resize while the vhost user server is running,
+     * otherwise we don't care what happens with the node.
+     */

I think this comment belong to the blk_new() below where the shared permissions are specified.

Quoted 72 lines
block/export/vhost-user-blk-server.c new file +72 -0
+    uint64_t perm = BLK_PERM_CONSISTENT_READ;
+    int ret;
+
+    AioContext *ctx;
+
+    BlockDriverState *bs = bdrv_lookup_bs(node_name, node_name, &local_error);
+
+    if (!bs) {
+        error_propagate(errp, local_error);
+        return NULL;
+    }
+
+    if (bdrv_is_read_only(bs)) {
+        writable = false;
+    }
+
+    if (writable) {
+        perm |= BLK_PERM_WRITE;
+    }
+
+    ctx = bdrv_get_aio_context(bs);
+    aio_context_acquire(ctx);
+    bdrv_invalidate_cache(bs, NULL);
+    aio_context_release(ctx);
+
+    blk = blk_new(bdrv_get_aio_context(bs), perm,
+                  BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED |
+                  BLK_PERM_WRITE | BLK_PERM_GRAPH_MOD);
+    ret = blk_insert_bs(blk, bs, errp);
+
+    if (ret < 0) {
+        goto fail;
+    }
+
+    blk_set_enable_write_cache(blk, false);
+
+    blk_set_allow_aio_context_change(blk, true);
+
+    vu_block_device->blkcfg.wce = 0;
+    vu_block_device->backend = blk;
+    if (!vu_block_device->blk_size) {
+        vu_block_device->blk_size = BDRV_SECTOR_SIZE;
+    }
+    vu_block_device->blkcfg.blk_size = vu_block_device->blk_size;
+    blk_set_guest_block_size(blk, vu_block_device->blk_size);
+    vu_block_initialize_config(bs, &vu_block_device->blkcfg,
+                                   vu_block_device->blk_size);
+    return vu_block_device;
+
+fail:
+    blk_unref(blk);
+    return NULL;
+}
+
+static void vhost_user_blk_server_stop(VuBlockDev *vu_block_device)
+{
+    if (!vu_block_device) {
+        return;
+    }
+
+    vhost_user_server_stop(&vu_block_device->vu_server);
+
+    if (vu_block_device->backend) {
+        blk_remove_aio_context_notifier(vu_block_device->backend, blk_aio_attached,
+                                        blk_aio_detach, vu_block_device);
+    }
+
+    blk_unref(vu_block_device->backend);
+
+}
+
+

block/export/vhost-user-blk-server.c line 480

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+static void vhost_user_blk_server_start(VuBlockDev *vu_block_device,
+                                        Error **errp)
+{
+    SocketAddress *addr = vu_block_device->addr;
+
+    if (!vu_block_init(vu_block_device, errp)) {
+        return;
+    }
+
+    AioContext *ctx = bdrv_get_aio_context(blk_bs(vu_block_device->backend));

Please move declarations to the top of the function.

Quoted 4 lines
block/export/vhost-user-blk-server.c new file +4 -0
+    if (!vhost_user_server_start(&vu_block_device->vu_server, addr, ctx,
+                                 VHOST_USER_BLK_MAX_QUEUES,
+                                 NULL, &vu_block_iface,
+                                 errp)) {

block/export/vhost-user-blk-server.c line 495

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+        goto error;
+    }
+
+    blk_add_aio_context_notifier(vu_block_device->backend, blk_aio_attached,
+                                 blk_aio_detach, vu_block_device);
+    vu_block_device->running = true;
+    return;
+
+ error:
+    vhost_user_blk_server_stop(vu_block_device);

vu_block_device hasn't been fully set up. You need to undo only vu_block_init(). You must not call vhost_user_server_stop().

block/export/vhost-user-blk-server.c line 498

Quoted 3 lines
block/export/vhost-user-blk-server.c new file +3 -0
+}
+
+static bool vu_prop_modificable(VuBlockDev *vus, Error **errp)

The word is "modifiable".

block/export/vhost-user-blk-server.c line 503

Quoted 5 lines
block/export/vhost-user-blk-server.c new file +5 -0
+{
+    if (vus->running) {
+            error_setg(errp, "The property can't be modified "
+                    "while the server is running");
+            return false;

The indentation is off here.

block/export/vhost-user-blk-server.c line 514

Quoted 11 lines
block/export/vhost-user-blk-server.c new file +11 -0
+    }
+    return true;
+}
+static void vu_set_node_name(Object *obj, const char *value, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    if (vus->node_name) {
+        if (!vu_prop_modificable(vus, errp)) {
+            return;
+        }

Why don't we need to check vu_prop_modificable() when the property isn't set yet? I assume it's because the server can't even be started without a node name, but it would be more obviously correct if the check were done unconditionally.

Quoted 12 lines
block/export/vhost-user-blk-server.c new file +12 -0
+        g_free(vus->node_name);
+    }
+
+    vus->node_name = g_strdup(value);
+}
+
+static char *vu_get_node_name(Object *obj, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    return g_strdup(vus->node_name);
+}
+

block/export/vhost-user-blk-server.c line 536

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+
+static void vu_set_unix_socket(Object *obj, const char *value,
+                               Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    if (vus->addr) {
+        if (!vu_prop_modificable(vus, errp)) {
+            return;
+        }

Same here.

Quoted 59 lines
block/export/vhost-user-blk-server.c new file +59 -0
+        g_free(vus->addr->u.q_unix.path);
+        g_free(vus->addr);
+    }
+
+    SocketAddress *addr = g_new0(SocketAddress, 1);
+    addr->type = SOCKET_ADDRESS_TYPE_UNIX;
+    addr->u.q_unix.path = g_strdup(value);
+    vus->addr = addr;
+}
+
+static char *vu_get_unix_socket(Object *obj, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    return g_strdup(vus->addr->u.q_unix.path);
+}
+
+static bool vu_get_block_writable(Object *obj, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    return vus->writable;
+}
+
+static void vu_set_block_writable(Object *obj, bool value, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    if (!vu_prop_modificable(vus, errp)) {
+            return;
+    }
+
+    vus->writable = value;
+}
+
+static void vu_get_blk_size(Object *obj, Visitor *v, const char *name,
+                            void *opaque, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    uint32_t value = vus->blk_size;
+
+    visit_type_uint32(v, name, &value, errp);
+}
+
+static void vu_set_blk_size(Object *obj, Visitor *v, const char *name,
+                            void *opaque, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    Error *local_err = NULL;
+    uint32_t value;
+
+    if (!vu_prop_modificable(vus, errp)) {
+            return;
+    }
+
+    visit_type_uint32(v, name, &value, &local_err);
+    if (local_err) {
+        goto out;
+    }
+

block/export/vhost-user-blk-server.c line 605

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    check_logical_block_size(object_get_typename(obj), name, value, &local_err);
+    if (local_err) {
+        goto out;
+    }
+
+    vus->blk_size = value;
+
+out:
+    error_propagate(errp, local_err);
+    vus->blk_size = value;

Surely you don't want to set the value here, when some check failed?

Quoted 11 lines
block/export/vhost-user-blk-server.c new file +11 -0
+}
+
+
+static void vhost_user_blk_server_instance_finalize(Object *obj)
+{
+    VuBlockDev *vub = VHOST_USER_BLK_SERVER(obj);
+
+    vhost_user_blk_server_stop(vub);
+}
+
+static void vhost_user_blk_server_complete(UserCreatable *obj, Error **errp)

block/export/vhost-user-blk-server.c line 626

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+{
+    Error *local_error = NULL;
+    VuBlockDev *vub = VHOST_USER_BLK_SERVER(obj);
+
+    vhost_user_blk_server_start(vub, &local_error);
+
+    if (local_error) {
+        error_propagate(errp, local_error);
+        return;
+    }

If you don't do anything with local_error (which is named inconsistently with local_err used above), you can just directly pass errp to vhost_user_blk_server_start().

Quoted 32 lines
block/export/vhost-user-blk-server.c new file +32 -0
+}
+
+static void vhost_user_blk_server_class_init(ObjectClass *klass,
+                                             void *class_data)
+{
+    UserCreatableClass *ucc = USER_CREATABLE_CLASS(klass);
+    ucc->complete = vhost_user_blk_server_complete;
+
+    object_class_property_add_bool(klass, "writable",
+                                   vu_get_block_writable,
+                                   vu_set_block_writable);
+
+    object_class_property_add_str(klass, "node-name",
+                                  vu_get_node_name,
+                                  vu_set_node_name);
+
+    object_class_property_add_str(klass, "unix-socket",
+                                  vu_get_unix_socket,
+                                  vu_set_unix_socket);
+
+    object_class_property_add(klass, "logical-block-size", "uint32",
+                              vu_get_blk_size, vu_set_blk_size,
+                              NULL, NULL);
+}
+
+static const TypeInfo vhost_user_blk_server_info = {
+    .name = TYPE_VHOST_USER_BLK_SERVER,
+    .parent = TYPE_OBJECT,
+    .instance_size = sizeof(VuBlockDev),
+    .instance_finalize = vhost_user_blk_server_instance_finalize,
+    .class_init = vhost_user_blk_server_class_init,
+    .interfaces = (InterfaceInfo[]) {

block/export/vhost-user-blk-server.c line 668

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+        {TYPE_USER_CREATABLE},
+        {}
+    },
+};
+
+static void vhost_user_blk_server_register_types(void)
+{
+    type_register_static(&vhost_user_blk_server_info);
+}
+

Please remove the trailing empty line.

Compared to the last version that I reviewed, this seems to get the architecture for concurrent requests right, which is an important improvement. I feel we're getting quite close to mergable now.

Kevin

author
#28 lore raw

Re: [PATCH v9 4/5] vhost-user block device backend server

Quoted 278 lines · Kevin Wolf

Am 14.06.2020 um 20:39 hat Coiby Xu geschrieben:

Quoted 257 lines

By making use of libvhost-user, block device drive can be shared to the connected vhost-user client. Only one client can connect to the server one time.

Since vhost-user-server needs a block drive to be created first, delay the creation of this object.

---
 block/Makefile.objs                  |   1 +
 block/export/vhost-user-blk-server.c | 669 +++++++++++++++++++++++++++
 block/export/vhost-user-blk-server.h |  35 ++
 softmmu/vl.c                         |   4 +
 4 files changed, 709 insertions(+)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
block/Makefile.objs +1 -0
index 3635b6b4c1..0eb7eff470 100644
@@ -24,6 +24,7 @@ block-obj-y += throttle-groups.o
 block-obj-$(CONFIG_LINUX) += nvme.o
 
 block-obj-y += nbd.o
+block-obj-$(CONFIG_LINUX) += export/vhost-user-blk-server.o ../contrib/libvhost-user/libvhost-user.o
 block-obj-$(CONFIG_SHEEPDOG) += sheepdog.o
 block-obj-$(CONFIG_LIBISCSI) += iscsi.o
 block-obj-$(if $(CONFIG_LIBISCSI),y,n) += iscsi-opts.o
block/export/vhost-user-blk-server.c new file +222 -0
new file mode 100644
index 0000000000..bbf2ceaa9b
@@ -0,0 +1,669 @@
+/*
+ * Sharing QEMU block devices via vhost-user protocal
+ *
+ * Author: Coiby Xu <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "vhost-user-blk-server.h"
+#include "qapi/error.h"
+#include "qom/object_interfaces.h"
+#include "sysemu/block-backend.h"
+#include "util/block-helpers.h"
+
+enum {
+    VHOST_USER_BLK_MAX_QUEUES = 1,
+};
+struct virtio_blk_inhdr {
+    unsigned char status;
+};
+
+
+typedef struct VuBlockReq {
+    VuVirtqElement *elem;
+    int64_t sector_num;
+    size_t size;
+    struct virtio_blk_inhdr *in;
+    struct virtio_blk_outhdr out;
+    VuServer *server;
+    struct VuVirtq *vq;
+} VuBlockReq;
+
+
+static void vu_block_req_complete(VuBlockReq *req)
+{
+    VuDev *vu_dev = &req->server->vu_dev;
+
+    /* IO size with 1 extra status byte */
+    vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+    vu_queue_notify(vu_dev, req->vq);
+
+    if (req->elem) {
+        free(req->elem);
+    }
+
+    g_free(req);
+}
+
+static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
+{
+    return container_of(server, VuBlockDev, vu_server);
+}
+
+static int coroutine_fn
+vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
+                              uint32_t iovcnt, uint32_t type)
+{
+    struct virtio_blk_discard_write_zeroes desc;
+    ssize_t size = iov_to_buf(iov, iovcnt, 0, &desc, sizeof(desc));
+    if (unlikely(size != sizeof(desc))) {
+        error_report("Invalid size %ld, expect %ld", size, sizeof(desc));
+        return -EINVAL;
+    }
+
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+    uint64_t range[2] = { le64_to_cpu(desc.sector) << 9,
+                          le32_to_cpu(desc.num_sectors) << 9 };
+    if (type == VIRTIO_BLK_T_DISCARD) {
+        if (blk_co_pdiscard(vdev_blk->backend, range[0], range[1]) == 0) {
+            return 0;
+        }
+    } else if (type == VIRTIO_BLK_T_WRITE_ZEROES) {
+        if (blk_co_pwrite_zeroes(vdev_blk->backend,
+                                 range[0], range[1], 0) == 0) {
+            return 0;
+        }
+    }
+
+    return -EINVAL;
+}
+
+
+static void coroutine_fn vu_block_flush(VuBlockReq *req)
+{
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
+    BlockBackend *backend = vdev_blk->backend;
+    blk_co_flush(backend);
+}
+
+
+struct req_data {
+    VuServer *server;
+    VuVirtq *vq;
+    VuVirtqElement *elem;
+};
+
+static void coroutine_fn vu_block_virtio_process_req(void *opaque)
+{
+    struct req_data *data = opaque;
+    VuServer *server = data->server;
+    VuVirtq *vq = data->vq;
+    VuVirtqElement *elem = data->elem;
+    uint32_t type;
+    VuBlockReq *req;
+
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    BlockBackend *backend = vdev_blk->backend;
+
+    struct iovec *in_iov = elem->in_sg;
+    struct iovec *out_iov = elem->out_sg;
+    unsigned in_num = elem->in_num;
+    unsigned out_num = elem->out_num;
+    /* refer to hw/block/virtio_blk.c */
+    if (elem->out_num < 1 || elem->in_num < 1) {
+        error_report("virtio-blk request missing headers");
+        free(elem);
+        return;
+    }
+
+    req = g_new0(VuBlockReq, 1);
+    req->server = server;
+    req->vq = vq;
+    req->elem = elem;
+
+    if (unlikely(iov_to_buf(out_iov, out_num, 0, &req->out,
+                            sizeof(req->out)) != sizeof(req->out))) {
+        error_report("virtio-blk request outhdr too short");
+        goto err;
+    }
+
+    iov_discard_front(&out_iov, &out_num, sizeof(req->out));
+
+    if (in_iov[in_num - 1].iov_len < sizeof(struct virtio_blk_inhdr)) {
+        error_report("virtio-blk request inhdr too short");
+        goto err;
+    }
+
+    /* We always touch the last byte, so just see how big in_iov is.  */
+    req->in = (void *)in_iov[in_num - 1].iov_base
+              + in_iov[in_num - 1].iov_len
+              - sizeof(struct virtio_blk_inhdr);
+    iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
+
+
+    type = le32_to_cpu(req->out.type);
+    switch (type & ~VIRTIO_BLK_T_BARRIER) {
+    case VIRTIO_BLK_T_IN:
+    case VIRTIO_BLK_T_OUT: {
+        ssize_t ret = 0;
+        bool is_write = type & VIRTIO_BLK_T_OUT;
+        req->sector_num = le64_to_cpu(req->out.sector);
+
+        int64_t offset = req->sector_num * vdev_blk->blk_size;
+        QEMUIOVector qiov;
+        if (is_write) {
+            qemu_iovec_init_external(&qiov, out_iov, out_num);
+            ret = blk_co_pwritev(backend, offset, qiov.size,
+                                 &qiov, 0);
+        } else {
+            qemu_iovec_init_external(&qiov, in_iov, in_num);
+            ret = blk_co_preadv(backend, offset, qiov.size,
+                                &qiov, 0);
+        }
+        if (ret >= 0) {
+            req->in->status = VIRTIO_BLK_S_OK;
+        } else {
+            req->in->status = VIRTIO_BLK_S_IOERR;
+        }
+        break;
+    }
+    case VIRTIO_BLK_T_FLUSH:
+        vu_block_flush(req);
+        req->in->status = VIRTIO_BLK_S_OK;
+        break;
+    case VIRTIO_BLK_T_GET_ID: {
+        size_t size = MIN(iov_size(&elem->in_sg[0], in_num),
+                          VIRTIO_BLK_ID_BYTES);
+        snprintf(elem->in_sg[0].iov_base, size, "%s", "vhost_user_blk_server");
+        req->in->status = VIRTIO_BLK_S_OK;
+        req->size = elem->in_sg[0].iov_len;
+        break;
+    }
+    case VIRTIO_BLK_T_DISCARD:
+    case VIRTIO_BLK_T_WRITE_ZEROES: {
+        int rc;
+        rc = vu_block_discard_write_zeroes(req, &elem->out_sg[1],
+                                           out_num, type);
+        if (rc == 0) {
+            req->in->status = VIRTIO_BLK_S_OK;
+        } else {
+            req->in->status = VIRTIO_BLK_S_IOERR;
+        }
+        break;
+    }
+    default:
+        req->in->status = VIRTIO_BLK_S_UNSUPP;
+        break;
+    }
+
+    vu_block_req_complete(req);
+    return;
+
+err:
+    free(elem);
+    g_free(req);
+    return;
+}
+
+
+
+static void vu_block_process_vq(VuDev *vu_dev, int idx)
+{
+    VuServer *server;
+    VuVirtq *vq;
+
+    server = container_of(vu_dev, VuServer, vu_dev);
+    assert(server);
+
+    vq = vu_get_queue(vu_dev, idx);
+    assert(vq);

block/export/vhost-user-blk-server.c line 232

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+    VuVirtqElement *elem;
+    while (1) {
+        elem = vu_queue_pop(vu_dev, vq, sizeof(VuVirtqElement) +
+                                    sizeof(VuBlockReq));
+        if (elem) {
+            struct req_data req_data = {
+                .server = server,
+                .vq = vq,
+                .elem = elem
+            };

This is on the stack of the function.

block/export/vhost-user-blk-server.c line 235

Quoted 3 lines
block/export/vhost-user-blk-server.c new file +3 -0
+            Coroutine *co = qemu_coroutine_create(vu_block_virtio_process_req,
+                                                  &req_data);
+            aio_co_enter(server->ioc->ctx, co);

Therefore, this code is only correct, if co accesses the data only while the function has not returned yet.

Quoted 7 lines

This function is called in the context of vu_dispatch(), which in turn is called from vu_client_trip(). So we already run in a coroutine. In this case, aio_co_enter() only schedules co to run after the current coroutine yields or terminates. In other words, this looks wrong to me because req_data will be accessed when it's long out of scope.

I think we need to malloc it.

vu_dispatch is only for reading vhost-user message. This function is called by the kick handler which is no longer run as a coroutine since I followed the advice to make better use of contrib/libvhost. Although no error has appeared or been reported by address sanitizer with --enable-sanitizers, using malloc is the correct way. I've fixed it in v10.

Quoted 77 lines
Quoted 77 lines
block/export/vhost-user-blk-server.c new file +76 -0
+        } else {
+            break;
+        }
+    }
+}
+
+static void vu_block_queue_set_started(VuDev *vu_dev, int idx, bool started)
+{
+    VuVirtq *vq;
+
+    assert(vu_dev);
+
+    vq = vu_get_queue(vu_dev, idx);
+    vu_set_queue_handler(vu_dev, vq, started ? vu_block_process_vq : NULL);
+}
+
+static uint64_t vu_block_get_features(VuDev *dev)
+{
+    uint64_t features;
+    VuServer *server = container_of(dev, VuServer, vu_dev);
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    features = 1ull << VIRTIO_BLK_F_SIZE_MAX |
+               1ull << VIRTIO_BLK_F_SEG_MAX |
+               1ull << VIRTIO_BLK_F_TOPOLOGY |
+               1ull << VIRTIO_BLK_F_BLK_SIZE |
+               1ull << VIRTIO_BLK_F_FLUSH |
+               1ull << VIRTIO_BLK_F_DISCARD |
+               1ull << VIRTIO_BLK_F_WRITE_ZEROES |
+               1ull << VIRTIO_BLK_F_CONFIG_WCE |
+               1ull << VIRTIO_F_VERSION_1 |
+               1ull << VIRTIO_RING_F_INDIRECT_DESC |
+               1ull << VIRTIO_RING_F_EVENT_IDX |
+               1ull << VHOST_USER_F_PROTOCOL_FEATURES;
+
+    if (!vdev_blk->writable) {
+        features |= 1ull << VIRTIO_BLK_F_RO;
+    }
+
+    return features;
+}
+
+static uint64_t vu_block_get_protocol_features(VuDev *dev)
+{
+    return 1ull << VHOST_USER_PROTOCOL_F_CONFIG |
+           1ull << VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD;
+}
+
+static int
+vu_block_get_config(VuDev *vu_dev, uint8_t *config, uint32_t len)
+{
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    memcpy(config, &vdev_blk->blkcfg, len);
+
+    return 0;
+}
+
+static int
+vu_block_set_config(VuDev *vu_dev, const uint8_t *data,
+                    uint32_t offset, uint32_t size, uint32_t flags)
+{
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
+    uint8_t wce;
+
+    /* don't support live migration */
+    if (flags != VHOST_SET_CONFIG_TYPE_MASTER) {
+        return -EINVAL;
+    }
+
+
+    if (offset != offsetof(struct virtio_blk_config, wce) ||
+        size != 1) {
+        return -EINVAL;
+    }
+
Quoted 8 lines

block/export/vhost-user-blk-server.c line 316

Quoted 5 lines
block/export/vhost-user-blk-server.c new file +5 -0
+    wce = *data;
+    if (wce == vdev_blk->blkcfg.wce) {
+        /* Do nothing as same with old configuration */
+        return 0;
+    }

This check is unnecessary. Nothing bad happens if you set the same value again.

Yes, removing it also simplifies the code.

Quoted 77 lines
Quoted 77 lines
block/export/vhost-user-blk-server.c new file +76 -0
+    vdev_blk->blkcfg.wce = wce;
+    blk_set_enable_write_cache(vdev_blk->backend, wce);
+    return 0;
+}
+
+
+/*
+ * When the client disconnects, it sends a VHOST_USER_NONE request
+ * and vu_process_message will simple call exit which cause the VM
+ * to exit abruptly.
+ * To avoid this issue,  process VHOST_USER_NONE request ahead
+ * of vu_process_message.
+ *
+ */
+static int vu_block_process_msg(VuDev *dev, VhostUserMsg *vmsg, int *do_reply)
+{
+    if (vmsg->request == VHOST_USER_NONE) {
+        dev->panic(dev, "disconnect");
+        return true;
+    }
+    return false;
+}
+
+
+static const VuDevIface vu_block_iface = {
+    .get_features          = vu_block_get_features,
+    .queue_set_started     = vu_block_queue_set_started,
+    .get_protocol_features = vu_block_get_protocol_features,
+    .get_config            = vu_block_get_config,
+    .set_config            = vu_block_set_config,
+    .process_msg           = vu_block_process_msg,
+};
+
+static void blk_aio_attached(AioContext *ctx, void *opaque)
+{
+    VuBlockDev *vub_dev = opaque;
+    aio_context_acquire(ctx);
+    vhost_user_server_set_aio_context(ctx, &vub_dev->vu_server);
+    aio_context_release(ctx);
+}
+
+static void blk_aio_detach(void *opaque)
+{
+    VuBlockDev *vub_dev = opaque;
+    AioContext *ctx = vub_dev->vu_server.ctx;
+    aio_context_acquire(ctx);
+    vhost_user_server_set_aio_context(NULL, &vub_dev->vu_server);
+    aio_context_release(ctx);
+}
+
+
+static void
+vu_block_initialize_config(BlockDriverState *bs,
+                           struct virtio_blk_config *config, uint32_t blk_size)
+{
+    config->capacity = bdrv_getlength(bs) >> BDRV_SECTOR_BITS;
+    config->blk_size = blk_size;
+    config->size_max = 0;
+    config->seg_max = 128 - 2;
+    config->min_io_size = 1;
+    config->opt_io_size = 1;
+    config->num_queues = VHOST_USER_BLK_MAX_QUEUES;
+    config->max_discard_sectors = 32768;
+    config->max_discard_seg = 1;
+    config->discard_sector_alignment = config->blk_size >> 9;
+    config->max_write_zeroes_sectors = 32768;
+    config->max_write_zeroes_seg = 1;
+}
+
+
+static VuBlockDev *vu_block_init(VuBlockDev *vu_block_device, Error **errp)
+{
+
+    BlockBackend *blk;
+    Error *local_error = NULL;
+    const char *node_name = vu_block_device->node_name;
Quoted 8 lines

block/export/vhost-user-blk-server.c line 398

Quoted 5 lines
block/export/vhost-user-blk-server.c new file +5 -0
+    bool writable = vu_block_device->writable;
+    /*
+     * Don't allow resize while the vhost user server is running,
+     * otherwise we don't care what happens with the node.
+     */

I think this comment belong to the blk_new() below where the shared permissions are specified.

Yes, it makes more sense to put the comment right above blk_new)_.

Quoted 77 lines
Quoted 77 lines
block/export/vhost-user-blk-server.c new file +76 -0
+    uint64_t perm = BLK_PERM_CONSISTENT_READ;
+    int ret;
+
+    AioContext *ctx;
+
+    BlockDriverState *bs = bdrv_lookup_bs(node_name, node_name, &local_error);
+
+    if (!bs) {
+        error_propagate(errp, local_error);
+        return NULL;
+    }
+
+    if (bdrv_is_read_only(bs)) {
+        writable = false;
+    }
+
+    if (writable) {
+        perm |= BLK_PERM_WRITE;
+    }
+
+    ctx = bdrv_get_aio_context(bs);
+    aio_context_acquire(ctx);
+    bdrv_invalidate_cache(bs, NULL);
+    aio_context_release(ctx);
+
+    blk = blk_new(bdrv_get_aio_context(bs), perm,
+                  BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED |
+                  BLK_PERM_WRITE | BLK_PERM_GRAPH_MOD);
+    ret = blk_insert_bs(blk, bs, errp);
+
+    if (ret < 0) {
+        goto fail;
+    }
+
+    blk_set_enable_write_cache(blk, false);
+
+    blk_set_allow_aio_context_change(blk, true);
+
+    vu_block_device->blkcfg.wce = 0;
+    vu_block_device->backend = blk;
+    if (!vu_block_device->blk_size) {
+        vu_block_device->blk_size = BDRV_SECTOR_SIZE;
+    }
+    vu_block_device->blkcfg.blk_size = vu_block_device->blk_size;
+    blk_set_guest_block_size(blk, vu_block_device->blk_size);
+    vu_block_initialize_config(bs, &vu_block_device->blkcfg,
+                                   vu_block_device->blk_size);
+    return vu_block_device;
+
+fail:
+    blk_unref(blk);
+    return NULL;
+}
+
+static void vhost_user_blk_server_stop(VuBlockDev *vu_block_device)
+{
+    if (!vu_block_device) {
+        return;
+    }
+
+    vhost_user_server_stop(&vu_block_device->vu_server);
+
+    if (vu_block_device->backend) {
+        blk_remove_aio_context_notifier(vu_block_device->backend, blk_aio_attached,
+                                        blk_aio_detach, vu_block_device);
+    }
+
+    blk_unref(vu_block_device->backend);
+
+}
+
+
+static void vhost_user_blk_server_start(VuBlockDev *vu_block_device,
+                                        Error **errp)
+{
+    SocketAddress *addr = vu_block_device->addr;
Quoted 8 lines

block/export/vhost-user-blk-server.c line 480

Quoted 6 lines
block/export/vhost-user-blk-server.c new file +6 -0
+
+    if (!vu_block_init(vu_block_device, errp)) {
+        return;
+    }
+
+    AioContext *ctx = bdrv_get_aio_context(blk_bs(vu_block_device->backend));

Please move declarations to the top of the function.

Been fixed in v10.

Quoted 19 lines
Quoted 5 lines
block/export/vhost-user-blk-server.c new file +4 -0
+    if (!vhost_user_server_start(&vu_block_device->vu_server, addr, ctx,
+                                 VHOST_USER_BLK_MAX_QUEUES,
+                                 NULL, &vu_block_iface,
+                                 errp)) {

block/export/vhost-user-blk-server.c line 495

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+        goto error;
+    }
+
+    blk_add_aio_context_notifier(vu_block_device->backend, blk_aio_attached,
+                                 blk_aio_detach, vu_block_device);
+    vu_block_device->running = true;
+    return;
+
+ error:
+    vhost_user_blk_server_stop(vu_block_device);

vu_block_device hasn't been fully set up. You need to undo only vu_block_init(). You must not call vhost_user_server_stop().

Quoted 5 lines

block/export/vhost-user-blk-server.c line 498

Quoted 3 lines
block/export/vhost-user-blk-server.c new file +3 -0
+}
+
+static bool vu_prop_modificable(VuBlockDev *vus, Error **errp)

The word is "modifiable".

Thank you for correcting my spelling!

Quoted 42 lines

block/export/vhost-user-blk-server.c line 503

Quoted 6 lines
block/export/vhost-user-blk-server.c new file +5 -0
+{
+    if (vus->running) {
+            error_setg(errp, "The property can't be modified "
+                    "while the server is running");
+            return false;

The indentation is off here.

block/export/vhost-user-blk-server.c line 514

Quoted 11 lines
block/export/vhost-user-blk-server.c new file +11 -0
+    }
+    return true;
+}
+static void vu_set_node_name(Object *obj, const char *value, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    if (vus->node_name) {
+        if (!vu_prop_modificable(vus, errp)) {
+            return;
+        }

Why don't we need to check vu_prop_modificable() when the property isn't set yet? I assume it's because the server can't even be started without a node name, but it would be more obviously correct if the check were done unconditionally.

Quoted 16 lines
block/export/vhost-user-blk-server.c new file +16 -0
+        g_free(vus->node_name);
+    }
+
+    vus->node_name = g_strdup(value);
+}
+
+static char *vu_get_node_name(Object *obj, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    return g_strdup(vus->node_name);
+}
+
+
+static void vu_set_unix_socket(Object *obj, const char *value,
+                               Error **errp)
+{
Quoted 8 lines

block/export/vhost-user-blk-server.c line 536

Quoted 6 lines
block/export/vhost-user-blk-server.c new file +6 -0
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    if (vus->addr) {
+        if (!vu_prop_modificable(vus, errp)) {
+            return;
+        }

Same here.

It makes the code more readable. Thank you!

Quoted 64 lines
Quoted 64 lines
block/export/vhost-user-blk-server.c new file +63 -0
+        g_free(vus->addr->u.q_unix.path);
+        g_free(vus->addr);
+    }
+
+    SocketAddress *addr = g_new0(SocketAddress, 1);
+    addr->type = SOCKET_ADDRESS_TYPE_UNIX;
+    addr->u.q_unix.path = g_strdup(value);
+    vus->addr = addr;
+}
+
+static char *vu_get_unix_socket(Object *obj, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    return g_strdup(vus->addr->u.q_unix.path);
+}
+
+static bool vu_get_block_writable(Object *obj, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    return vus->writable;
+}
+
+static void vu_set_block_writable(Object *obj, bool value, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    if (!vu_prop_modificable(vus, errp)) {
+            return;
+    }
+
+    vus->writable = value;
+}
+
+static void vu_get_blk_size(Object *obj, Visitor *v, const char *name,
+                            void *opaque, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+    uint32_t value = vus->blk_size;
+
+    visit_type_uint32(v, name, &value, errp);
+}
+
+static void vu_set_blk_size(Object *obj, Visitor *v, const char *name,
+                            void *opaque, Error **errp)
+{
+    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
+
+    Error *local_err = NULL;
+    uint32_t value;
+
+    if (!vu_prop_modificable(vus, errp)) {
+            return;
+    }
+
+    visit_type_uint32(v, name, &value, &local_err);
+    if (local_err) {
+        goto out;
+    }
+
+    check_logical_block_size(object_get_typename(obj), name, value, &local_err);
+    if (local_err) {
+        goto out;
+    }
Quoted 8 lines

block/export/vhost-user-blk-server.c line 605

Quoted 6 lines
block/export/vhost-user-blk-server.c new file +6 -0
+
+    vus->blk_size = value;
+
+out:
+    error_propagate(errp, local_err);
+    vus->blk_size = value;

Surely you don't want to set the value here, when some check failed?

Yes, I must have left this line of code here by mistake.

Quoted 72 lines
Quoted 12 lines
block/export/vhost-user-blk-server.c new file +11 -0
+}
+
+
+static void vhost_user_blk_server_instance_finalize(Object *obj)
+{
+    VuBlockDev *vub = VHOST_USER_BLK_SERVER(obj);
+
+    vhost_user_blk_server_stop(vub);
+}
+
+static void vhost_user_blk_server_complete(UserCreatable *obj, Error **errp)

block/export/vhost-user-blk-server.c line 626

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+{
+    Error *local_error = NULL;
+    VuBlockDev *vub = VHOST_USER_BLK_SERVER(obj);
+
+    vhost_user_blk_server_start(vub, &local_error);
+
+    if (local_error) {
+        error_propagate(errp, local_error);
+        return;
+    }

If you don't do anything with local_error (which is named inconsistently with local_err used above), you can just directly pass errp to vhost_user_blk_server_start().

Quoted 32 lines
block/export/vhost-user-blk-server.c new file +32 -0
+}
+
+static void vhost_user_blk_server_class_init(ObjectClass *klass,
+                                             void *class_data)
+{
+    UserCreatableClass *ucc = USER_CREATABLE_CLASS(klass);
+    ucc->complete = vhost_user_blk_server_complete;
+
+    object_class_property_add_bool(klass, "writable",
+                                   vu_get_block_writable,
+                                   vu_set_block_writable);
+
+    object_class_property_add_str(klass, "node-name",
+                                  vu_get_node_name,
+                                  vu_set_node_name);
+
+    object_class_property_add_str(klass, "unix-socket",
+                                  vu_get_unix_socket,
+                                  vu_set_unix_socket);
+
+    object_class_property_add(klass, "logical-block-size", "uint32",
+                              vu_get_blk_size, vu_set_blk_size,
+                              NULL, NULL);
+}
+
+static const TypeInfo vhost_user_blk_server_info = {
+    .name = TYPE_VHOST_USER_BLK_SERVER,
+    .parent = TYPE_OBJECT,
+    .instance_size = sizeof(VuBlockDev),
+    .instance_finalize = vhost_user_blk_server_instance_finalize,
+    .class_init = vhost_user_blk_server_class_init,
+    .interfaces = (InterfaceInfo[]) {

block/export/vhost-user-blk-server.c line 668

Quoted 10 lines
block/export/vhost-user-blk-server.c new file +10 -0
+        {TYPE_USER_CREATABLE},
+        {}
+    },
+};
+
+static void vhost_user_blk_server_register_types(void)
+{
+    type_register_static(&vhost_user_blk_server_info);
+}
+

Please remove the trailing empty line.

Quoted 6 lines

Compared to the last version that I reviewed, this seems to get the architecture for concurrent requests right, which is an important improvement. I feel we're getting quite close to mergable now.

Kevin

Thank you for the guidance and helpful feedbacks along the way:)

--
Best regards,
Coiby

#29 lore raw

PATCH 1/2 vhost-user-blk-server: adjust vhost_user_server_set_aio_context() arguments

In reply to Coiby Xu

The function arguments were reordered in a previous patch. Use the new ordering.

---
 block/export/vhost-user-blk-server.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block/export/vhost-user-blk-server.c +2 -2
index bbf2ceaa9b..bed3c43737 100644
@@ -352,7 +352,7 @@ static void blk_aio_attached(AioContext *ctx, void *opaque)
 {
     VuBlockDev *vub_dev = opaque;
     aio_context_acquire(ctx);
-    vhost_user_server_set_aio_context(ctx, &vub_dev->vu_server);
+    vhost_user_server_set_aio_context(&vub_dev->vu_server, ctx);
     aio_context_release(ctx);
 }
 
@@ -361,7 +361,7 @@ static void blk_aio_detach(void *opaque)
     VuBlockDev *vub_dev = opaque;
     AioContext *ctx = vub_dev->vu_server.ctx;
     aio_context_acquire(ctx);
-    vhost_user_server_set_aio_context(NULL, &vub_dev->vu_server);
+    vhost_user_server_set_aio_context(&vub_dev->vu_server, NULL);
     aio_context_release(ctx);
 }
 
Signature · 1 line
2.26.2
#30 lore raw

PATCH 2/2 vhost-user-blk-server: rename check_logical_block_size() to check_block_size()

---
 block/export/vhost-user-blk-server.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/export/vhost-user-blk-server.c +1 -1
index bed3c43737..f3fada5b37 100644
@@ -593,7 +593,7 @@ static void vu_set_blk_size(Object *obj, Visitor *v, const char *name,
         goto out;
     }
 
-    check_logical_block_size(object_get_typename(obj), name, value, &local_err);
+    check_block_size(object_get_typename(obj), name, value, &local_err);
     if (local_err) {
         goto out;
     }
Signature · 1 line
2.26.2
author
#31 lore raw

PATCH v9 5/5 new qTest case to test the vhost-user-blk-server

In reply to Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize. Since vhost-user server can only server one client one time, two instances of qemu-storage-daemon are launched for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will send "quit" command to qemu-storage-daemon's QMP monitor. So a function is added to libqtest.c to establish socket connection with socket server.

---
 tests/Makefile.include              |   3 +-
 tests/qtest/Makefile.include        |   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +++++
 tests/qtest/libqos/vhost-user-blk.h |  48 ++
 tests/qtest/libqtest.c              |  35 +-
 tests/qtest/libqtest.h              |  17 +
 tests/qtest/vhost-user-blk-test.c   | 739 ++++++++++++++++++++++++++++
 7 files changed, 971 insertions(+), 3 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
tests/Makefile.include +2 -1
index c2397de8ed..303235b40f 100644
@@ -638,7 +638,8 @@ endef
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all $(check-qtest-y)
 	$(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) $(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \
 	  QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
-	  QTEST_QEMU_IMG=qemu-img$(EXESUF))
+	  QTEST_QEMU_IMG=./qemu-img$(EXESUF) \
+	  QTEST_QEMU_STORAGE_DAEMON_BINARY=./qemu-storage-daemon$(EXESUF))
 
 check-unit: $(check-unit-y)
 	$(call do_test_human, $^)
tests/qtest/Makefile.include +2 -0
index 9e5a51d033..b6f081cb26 100644
@@ -186,6 +186,7 @@ libqos-obj-y += tests/qtest/libqos/virtio.o
 libqos-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o
 libqos-obj-y += tests/qtest/libqos/virtio-balloon.o
 libqos-obj-y += tests/qtest/libqos/virtio-blk.o
+libqos-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o
 libqos-obj-y += tests/qtest/libqos/virtio-mmio.o
 libqos-obj-y += tests/qtest/libqos/virtio-net.o
 libqos-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -230,6 +231,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += tests/qtest/vhost-user-test.o $(chardev
 qos-test-obj-y += tests/qtest/virtio-test.o
 qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o
 qos-test-obj-y += tests/qtest/virtio-blk-test.o
+qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o
 qos-test-obj-y += tests/qtest/virtio-net-test.o
 qos-test-obj-y += tests/qtest/virtio-rng-test.o
 qos-test-obj-y += tests/qtest/virtio-scsi-test.o
tests/qtest/libqos/vhost-user-blk.c new file +130 -0

130 lines not on this page · raw message

tests/qtest/libqos/vhost-user-blk.h new file +48 -0

48 lines not on this page · raw message

tests/qtest/libqtest.c +33 -2
index 49075b55a1..02cc09f893 100644
@@ -52,8 +52,7 @@ typedef struct QTestClientTransportOps {
     QTestRecvFn     recv_line; /* for receiving qtest command responses */
 } QTestTransportOps;
 
-struct QTestState
-{
+struct QTestState {
     int fd;
     int qmp_fd;
     pid_t qemu_pid;  /* our child QEMU process */
@@ -608,6 +607,38 @@ QDict *qtest_qmp_receive(QTestState *s)
     return qmp_fd_receive(s->qmp_fd);
 }
 
+QTestState *qtest_create_state_with_qmp_fd(int fd)
+{
+    QTestState *qmp_test_state = g_new0(QTestState, 1);
+    qmp_test_state->qmp_fd = fd;
+    return qmp_test_state;
+}
+
+int qtest_socket_client(char *server_socket_path)
+{
+    struct sockaddr_un serv_addr;
+    int sock;
+    int ret;
+    int retries = 0;
+    sock = socket(PF_UNIX, SOCK_STREAM, 0);
+    g_assert_cmpint(sock, !=, -1);
+    serv_addr.sun_family = AF_UNIX;
+    snprintf(serv_addr.sun_path, sizeof(serv_addr.sun_path), "%s",
+             server_socket_path);
+
+    do {
+        ret = connect(sock, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
+        if (ret == 0) {
+            break;
+        }
+        retries += 1;
+        g_usleep(G_USEC_PER_SEC);
+    } while (retries < 3);
+
+    g_assert_cmpint(ret, ==, 0);
+    return sock;
+}
+
 /**
  * Allow users to send a message without waiting for the reply,
  * in the case that they choose to discard all replies up until
tests/qtest/libqtest.h +17 -0
index f5cf93c386..c73c0a9bbe 100644
@@ -132,6 +132,23 @@ void qtest_qmp_send(QTestState *s, const char *fmt, ...)
 void qtest_qmp_send_raw(QTestState *s, const char *fmt, ...)
     GCC_FMT_ATTR(2, 3);
 
+/**
+ * qtest_socket_client:
+ * @server_socket_path: the socket server's path
+ *
+ * Connect to a socket server.
+ */
+int qtest_socket_client(char *server_socket_path);
+
+/**
+ * qtest_create_state_with_qmp_fd:
+ * @fd: socket fd
+ *
+ * Wrap socket fd in QTestState to make use of qtest_qmp*
+ * functions
+ */
+QTestState *qtest_create_state_with_qmp_fd(int fd);
+
 /**
  * qtest_vqmp_fds:
  * @s: #QTestState instance to operate on.
tests/qtest/vhost-user-blk-test.c new file +739 -0

739 lines not on this page · raw message

Signature · 1 line
2.27.0
#32 lore raw

Re: [PATCH v9 5/5] new qTest case to test the vhost-user-blk-server

Quoted 11 lines · Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize. Since vhost-user server can only server one client one time, two instances of qemu-storage-daemon are launched for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will send "quit" command to qemu-storage-daemon's QMP monitor. So a function is added to libqtest.c to establish socket connection with socket server.

Quoted 12 lines
---
 tests/Makefile.include              |   3 +-
 tests/qtest/Makefile.include        |   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +++++
 tests/qtest/libqos/vhost-user-blk.h |  48 ++
 tests/qtest/libqtest.c              |  35 +-
 tests/qtest/libqtest.h              |  17 +
 tests/qtest/vhost-user-blk-test.c   | 739 ++++++++++++++++++++++++++++
 7 files changed, 971 insertions(+), 3 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

This test case fails for me:

qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Success (0)
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Success (0)
qemu-system-x86_64: Failed to read msg header. Read -1 instead of 12. Original request 11.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Input/output error (5)

Does "make -j4 check" pass for you?

Stefan

author
#33 lore raw

Re: [PATCH v9 5/5] new qTest case to test the vhost-user-blk-server

Quoted 38 lines · Stefan Hajnoczi
Quoted 11 lines · Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize. Since vhost-user server can only server one client one time, two instances of qemu-storage-daemon are launched for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will send "quit" command to qemu-storage-daemon's QMP monitor. So a function is added to libqtest.c to establish socket connection with socket server.

Quoted 12 lines
---
 tests/Makefile.include              |   3 +-
 tests/qtest/Makefile.include        |   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +++++
 tests/qtest/libqos/vhost-user-blk.h |  48 ++
 tests/qtest/libqtest.c              |  35 +-
 tests/qtest/libqtest.h              |  17 +
 tests/qtest/vhost-user-blk-test.c   | 739 ++++++++++++++++++++++++++++
 7 files changed, 971 insertions(+), 3 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

This test case fails for me:

qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Success (0)
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Success (0)
qemu-system-x86_64: Failed to read msg header. Read -1 instead of 12. Original request 11.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Input/output error (5)

Quoted 1 line

Does "make -j4 check" pass for you?

Actually it's a success since it won't fail CI. The reason for the occurrence of these dubious messages is after finishing the tests, vhost-user-blk-server is stopped before qemu-system-x86_64 is destroyed. I'll see if I can find a way to kill qemu-system-x86_64 first.

--
Best regards,
Coiby

#34 lore raw

Re: [PATCH v9 5/5] new qTest case to test the vhost-user-blk-server

Quoted 39 lines · Coiby Xu
Quoted 38 lines · Stefan Hajnoczi
Quoted 11 lines · Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize. Since vhost-user server can only server one client one time, two instances of qemu-storage-daemon are launched for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will send "quit" command to qemu-storage-daemon's QMP monitor. So a function is added to libqtest.c to establish socket connection with socket server.

Quoted 12 lines
---
 tests/Makefile.include              |   3 +-
 tests/qtest/Makefile.include        |   2 +
 tests/qtest/libqos/vhost-user-blk.c | 130 +++++
 tests/qtest/libqos/vhost-user-blk.h |  48 ++
 tests/qtest/libqtest.c              |  35 +-
 tests/qtest/libqtest.h              |  17 +
 tests/qtest/vhost-user-blk-test.c   | 739 ++++++++++++++++++++++++++++
 7 files changed, 971 insertions(+), 3 deletions(-)
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

This test case fails for me:

qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Success (0)
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Success (0)
qemu-system-x86_64: Failed to read msg header. Read -1 instead of 12. Original request 11.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Input/output error (5)

Quoted 6 lines
Quoted 1 line

Does "make -j4 check" pass for you?

Actually it's a success since it won't fail CI. The reason for the occurrence of these dubious messages is after finishing the tests, vhost-user-blk-server is stopped before qemu-system-x86_64 is destroyed. I'll see if I can find a way to kill qemu-system-x86_64 first.

Maybe I didn't even notice whether it was passing or failing and just got scared by these messages! :)

Thanks for explaining. It would be good to terminate cleanly to avoid confusing users.

Stefan

#35 lore raw

Re: [PATCH v9 5/5] new qTest case to test the vhost-user-blk-server

In reply to Coiby Xu
Quoted 12 lines · Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize. Since vhost-user server can only server one client one time, two instances of qemu-storage-daemon are launched for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will send "quit" command to qemu-storage-daemon's QMP monitor. So a function is added to libqtest.c to establish socket connection with socket server.

---

[...]

Quoted 28 lines
tests/qtest/libqtest.c +11 -2
index 49075b55a1..02cc09f893 100644
@@ -52,8 +52,7 @@ typedef struct QTestClientTransportOps {
      QTestRecvFn     recv_line; /* for receiving qtest command responses */
  } QTestTransportOps;
 
-struct QTestState
-{
+struct QTestState {
      int fd;
      int qmp_fd;
      pid_t qemu_pid;  /* our child QEMU process */
@@ -608,6 +607,38 @@ QDict *qtest_qmp_receive(QTestState *s)
      return qmp_fd_receive(s->qmp_fd);
  }
 
+QTestState *qtest_create_state_with_qmp_fd(int fd)
+{
+    QTestState *qmp_test_state = g_new0(QTestState, 1);
+    qmp_test_state->qmp_fd = fd;
+    return qmp_test_state;
+}
+
+int qtest_socket_client(char *server_socket_path)
+{
+    struct sockaddr_un serv_addr;

tests/qtest/libqtest.c QDict *qtest_qmp_receive(QTestState *s) · line 629

Quoted 10 lines
tests/qtest/libqtest.c +10 -0
+    int sock;
+    int ret;
+    int retries = 0;
+    sock = socket(PF_UNIX, SOCK_STREAM, 0);
+    g_assert_cmpint(sock, !=, -1);
+    serv_addr.sun_family = AF_UNIX;
+    snprintf(serv_addr.sun_path, sizeof(serv_addr.sun_path), "%s",
+             server_socket_path);
+
+    do {

Why not simply:

for (retries = 0; retries < 3; retries++)

?

Quoted 11 lines
tests/qtest/libqtest.c +11 -0
+        ret = connect(sock, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
+        if (ret == 0) {
+            break;
+        }
+        retries += 1;
+        g_usleep(G_USEC_PER_SEC);
+    } while (retries < 3);
+
+    g_assert_cmpint(ret, ==, 0);
+    return sock;
+}

[...]

Quoted 18 lines
tests/qtest/vhost-user-blk-test.c new file +12 -0
new file mode 100644
index 0000000000..56e3d8f338
@@ -0,0 +1,739 @@
+/*
+ * QTest testcase for VirtIO Block Device
+ *
+ * Copyright (c) 2014 SUSE LINUX Products GmbH
+ * Copyright (c) 2014 Marc Marí
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest-single.h"

tests/qtest/vhost-user-blk-test.c line 22

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+#include "qemu/bswap.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "standard-headers/linux/virtio_pci.h"
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+#include "libqos/libqos-pc.h"
+
+/* TODO actually test the results and get rid of this */
+#define qmp_discard_response(...) qobject_unref(qmp(__VA_ARGS__))

Please no more qmp_discard_response() in new code!

Quoted 19 lines
tests/qtest/vhost-user-blk-test.c new file +19 -0
+#define TEST_IMAGE_SIZE         (64 * 1024 * 1024)
+#define QVIRTIO_BLK_TIMEOUT_US  (30 * 1000 * 1000)
+#define PCI_SLOT_HP             0x06
+
+typedef struct QVirtioBlkReq {
+    uint32_t type;
+    uint32_t ioprio;
+    uint64_t sector;
+    char *data;
+    uint8_t status;
+} QVirtioBlkReq;
+
+
+#ifdef HOST_WORDS_BIGENDIAN
+static const bool host_is_big_endian = true;
+#else
+static const bool host_is_big_endian; /* false */
+#endif
+

tests/qtest/vhost-user-blk-test.c line 52

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+static inline void virtio_blk_fix_request(QVirtioDevice *d, QVirtioBlkReq *req)
+{
+    if (qvirtio_is_big_endian(d) != host_is_big_endian) {
+        req->type = bswap32(req->type);
+        req->ioprio = bswap32(req->ioprio);
+        req->sector = bswap64(req->sector);
+    }
+}
+
+

One empty line should be enough ;-)

Quoted 106 lines
tests/qtest/vhost-user-blk-test.c new file +106 -0
+static inline void virtio_blk_fix_dwz_hdr(QVirtioDevice *d,
+    struct virtio_blk_discard_write_zeroes *dwz_hdr)
+{
+    if (qvirtio_is_big_endian(d) != host_is_big_endian) {
+        dwz_hdr->sector = bswap64(dwz_hdr->sector);
+        dwz_hdr->num_sectors = bswap32(dwz_hdr->num_sectors);
+        dwz_hdr->flags = bswap32(dwz_hdr->flags);
+    }
+}
+
+static uint64_t virtio_blk_request(QGuestAllocator *alloc, QVirtioDevice *d,
+                                   QVirtioBlkReq *req, uint64_t data_size)
+{
+    uint64_t addr;
+    uint8_t status = 0xFF;
+
+    switch (req->type) {
+    case VIRTIO_BLK_T_IN:
+    case VIRTIO_BLK_T_OUT:
+        g_assert_cmpuint(data_size % 512, ==, 0);
+        break;
+    case VIRTIO_BLK_T_DISCARD:
+    case VIRTIO_BLK_T_WRITE_ZEROES:
+        g_assert_cmpuint(data_size %
+                         sizeof(struct virtio_blk_discard_write_zeroes), ==, 0);
+        break;
+    default:
+        g_assert_cmpuint(data_size, ==, 0);
+    }
+
+    addr = guest_alloc(alloc, sizeof(*req) + data_size);
+
+    virtio_blk_fix_request(d, req);
+
+    memwrite(addr, req, 16);
+    memwrite(addr + 16, req->data, data_size);
+    memwrite(addr + 16 + data_size, &status, sizeof(status));
+
+    return addr;
+}
+
+/* Returns the request virtqueue so the caller can perform further tests */
+static QVirtQueue *test_basic(QVirtioDevice *dev, QGuestAllocator *alloc)
+{
+    QVirtioBlkReq req;
+    uint64_t req_addr;
+    uint64_t capacity;
+    uint64_t features;
+    uint32_t free_head;
+    uint8_t status;
+    char *data;
+    QTestState *qts = global_qtest;
+    QVirtQueue *vq;
+
+    features = qvirtio_get_features(dev);
+    features = features & ~(QVIRTIO_F_BAD_FEATURE |
+                    (1u << VIRTIO_RING_F_INDIRECT_DESC) |
+                    (1u << VIRTIO_RING_F_EVENT_IDX) |
+                    (1u << VIRTIO_BLK_F_SCSI));
+    qvirtio_set_features(dev, features);
+
+    capacity = qvirtio_config_readq(dev, 0);
+    g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512);
+
+    vq = qvirtqueue_setup(dev, alloc, 0);
+
+    qvirtio_set_driver_ok(dev);
+
+    /* Write and read with 3 descriptor layout */
+    /* Write request */
+    req.type = VIRTIO_BLK_T_OUT;
+    req.ioprio = 1;
+    req.sector = 0;
+    req.data = g_malloc0(512);
+    strcpy(req.data, "TEST");
+
+    req_addr = virtio_blk_request(alloc, dev, &req, 512);
+
+    g_free(req.data);
+
+    free_head = qvirtqueue_add(qts, vq, req_addr, 16, false, true);
+    qvirtqueue_add(qts, vq, req_addr + 16, 512, false, true);
+    qvirtqueue_add(qts, vq, req_addr + 528, 1, true, false);
+
+    qvirtqueue_kick(qts, dev, vq, free_head);
+
+    qvirtio_wait_used_elem(qts, dev, vq, free_head, NULL,
+                           QVIRTIO_BLK_TIMEOUT_US);
+    status = readb(req_addr + 528);
+    g_assert_cmpint(status, ==, 0);
+
+    guest_free(alloc, req_addr);
+
+    /* Read request */
+    req.type = VIRTIO_BLK_T_IN;
+    req.ioprio = 1;
+    req.sector = 0;
+    req.data = g_malloc0(512);
+
+    req_addr = virtio_blk_request(alloc, dev, &req, 512);
+
+    g_free(req.data);
+
+    free_head = qvirtqueue_add(qts, vq, req_addr, 16, false, true);
+    qvirtqueue_add(qts, vq, req_addr + 16, 512, true, true);
+    qvirtqueue_add(qts, vq, req_addr + 528, 1, true, false);

tests/qtest/vhost-user-blk-test.c line 168

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+
+    qvirtqueue_kick(qts, dev, vq, free_head);
+
+    qvirtio_wait_used_elem(qts, dev, vq, free_head, NULL,
+                           QVIRTIO_BLK_TIMEOUT_US);
+    status = readb(req_addr + 528);
+    g_assert_cmpint(status, ==, 0);
+
+    data = g_malloc0(512);
+    memread(req_addr + 16, data, 512);

Since you have a "qts" variable here anyway, could you please use qtest_memread(qts, ...) here instead? (also in the other spots where you use memread and memwrite if possible) ... in case we ever have to introduce multiple test states later, we then don't have to rewrite the code anymore. (generally, it's nice to avoid libqtest-single.h nowadays and only use libqtest.h if possible)

[...]

Quoted 76 lines
tests/qtest/vhost-user-blk-test.c new file +76 -0
+static void indirect(void *obj, void *u_data, QGuestAllocator *t_alloc)
+{
+    QVirtQueue *vq;
+    QVhostUserBlk *blk_if = obj;
+    QVirtioDevice *dev = blk_if->vdev;
+    QVirtioBlkReq req;
+    QVRingIndirectDesc *indirect;
+    uint64_t req_addr;
+    uint64_t capacity;
+    uint64_t features;
+    uint32_t free_head;
+    uint8_t status;
+    char *data;
+    QTestState *qts = global_qtest;
+
+    features = qvirtio_get_features(dev);
+    g_assert_cmphex(features & (1u << VIRTIO_RING_F_INDIRECT_DESC), !=, 0);
+    features = features & ~(QVIRTIO_F_BAD_FEATURE |
+                            (1u << VIRTIO_RING_F_EVENT_IDX) |
+                            (1u << VIRTIO_BLK_F_SCSI));
+    qvirtio_set_features(dev, features);
+
+    capacity = qvirtio_config_readq(dev, 0);
+    g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512);
+
+    vq = qvirtqueue_setup(dev, t_alloc, 0);
+    qvirtio_set_driver_ok(dev);
+
+    /* Write request */
+    req.type = VIRTIO_BLK_T_OUT;
+    req.ioprio = 1;
+    req.sector = 0;
+    req.data = g_malloc0(512);
+    strcpy(req.data, "TEST");
+
+    req_addr = virtio_blk_request(t_alloc, dev, &req, 512);
+
+    g_free(req.data);
+
+    indirect = qvring_indirect_desc_setup(qts, dev, t_alloc, 2);
+    qvring_indirect_desc_add(dev, qts, indirect, req_addr, 528, false);
+    qvring_indirect_desc_add(dev, qts, indirect, req_addr + 528, 1, true);
+    free_head = qvirtqueue_add_indirect(qts, vq, indirect);
+    qvirtqueue_kick(qts, dev, vq, free_head);
+
+    qvirtio_wait_used_elem(qts, dev, vq, free_head, NULL,
+                           QVIRTIO_BLK_TIMEOUT_US);
+    status = readb(req_addr + 528);
+    g_assert_cmpint(status, ==, 0);
+
+    g_free(indirect);
+    guest_free(t_alloc, req_addr);
+
+    /* Read request */
+    req.type = VIRTIO_BLK_T_IN;
+    req.ioprio = 1;
+    req.sector = 0;
+    req.data = g_malloc0(512);
+    strcpy(req.data, "TEST");
+
+    req_addr = virtio_blk_request(t_alloc, dev, &req, 512);
+
+    g_free(req.data);
+
+    indirect = qvring_indirect_desc_setup(qts, dev, t_alloc, 2);
+    qvring_indirect_desc_add(dev, qts, indirect, req_addr, 16, false);
+    qvring_indirect_desc_add(dev, qts, indirect, req_addr + 16, 513, true);
+    free_head = qvirtqueue_add_indirect(qts, vq, indirect);
+    qvirtqueue_kick(qts, dev, vq, free_head);
+
+    qvirtio_wait_used_elem(qts, dev, vq, free_head, NULL,
+                           QVIRTIO_BLK_TIMEOUT_US);
+    status = readb(req_addr + 528);
+    g_assert_cmpint(status, ==, 0);
+
+    data = g_malloc0(512);

tests/qtest/vhost-user-blk-test.c line 415

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+    memread(req_addr + 16, data, 512);
+    g_assert_cmpstr(data, ==, "TEST");
+    g_free(data);
+
+    g_free(indirect);
+    guest_free(t_alloc, req_addr);
+    qvirtqueue_cleanup(dev->bus, vq, t_alloc);
+}
+
+

One empty line is enough.

[...]

tests/qtest/vhost-user-blk-test.c line 609

Quoted 8 lines
tests/qtest/vhost-user-blk-test.c new file +8 -0
+static void drive_destroy(void *path)
+{
+    unlink(path);
+    g_free(path);
+    qos_invalidate_command_line();
+}
+
+

dito.

[...]

Quoted 11 lines
tests/qtest/vhost-user-blk-test.c new file +11 -0
+static char *drive_create(void)
+{
+    int fd, ret;
+    /** vhost-user-blk won't recognize drive located in /tmp */
+    char *t_path = g_strdup("qtest.XXXXXX");
+
+    /** Create a temporary raw image */
+    fd = mkstemp(t_path);
+    g_assert_cmpint(fd, >=, 0);
+    ret = ftruncate(fd, TEST_IMAGE_SIZE);
+    g_assert_cmpint(ret, ==, 0);

tests/qtest/vhost-user-blk-test.c line 630

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+    close(fd);
+
+    g_test_queue_destroy(drive_destroy, t_path);
+    return t_path;
+}
+
+static char sock_path_tempate[] = "/tmp/qtest.vhost_user_blk.XXXXXX";
+static char qmp_sock_path_tempate[] = "/tmp/qtest.vhost_user_blk.qmp.XXXXXX";
+
+

dito.

Quoted 7 lines
tests/qtest/vhost-user-blk-test.c new file +7 -0
+static void quit_storage_daemon(void *qmp_test_state)
+{
+    qobject_unref(qtest_qmp((QTestState *)qmp_test_state, "{ 'execute': 'quit' }"));
+    g_free(qmp_test_state);
+}
+
+static char *start_vhost_user_blk(void)

tests/qtest/vhost-user-blk-test.c line 647

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+{
+    int fd, qmp_fd;
+    char *sock_path = g_strdup(sock_path_tempate);
+    char *qmp_sock_path = g_strdup(qmp_sock_path_tempate);
+    QTestState *qmp_test_state;
+    fd = mkstemp(sock_path);
+    g_assert_cmpint(fd, >=, 0);
+    g_test_queue_destroy(drive_destroy, sock_path);
+
+

dito.

Quoted 7 lines
tests/qtest/vhost-user-blk-test.c new file +7 -0
+    qmp_fd = mkstemp(qmp_sock_path);
+    g_assert_cmpint(qmp_fd, >=, 0);
+    g_test_queue_destroy(drive_destroy, qmp_sock_path);
+
+    /* create image file */
+    const char *img_path = drive_create();
+

tests/qtest/vhost-user-blk-test.c line 664

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+    const char *vhost_user_blk_bin = qtest_qemu_storage_daemon_binary();
+    gchar *command = g_strdup_printf(
+            "exec %s "
+            "--blockdev driver=file,node-name=disk,filename=%s "
+            "--object vhost-user-blk-server,id=disk,unix-socket=%s,"
+            "node-name=disk,writable=on "
+            "--chardev socket,id=qmp,path=%s,server,nowait --monitor chardev=qmp",
+            vhost_user_blk_bin, img_path, sock_path, qmp_sock_path);
+
+

dito.

Quoted 12 lines
tests/qtest/vhost-user-blk-test.c new file +12 -0
+    g_test_message("starting vhost-user backend: %s", command);
+    pid_t pid = fork();
+    if (pid == 0) {
+        execlp("/bin/sh", "sh", "-c", command, NULL);
+        exit(1);
+    }
+    g_free(command);
+
+    qmp_test_state = qtest_create_state_with_qmp_fd(
+                             qtest_socket_client(qmp_sock_path));
+    /*
+     * Ask qemu-storage-daemon to quit so it

tests/qtest/vhost-user-blk-test.c line 686

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+     * will not block scripts/tap-driver.pl.
+     */
+    g_test_queue_destroy(quit_storage_daemon, qmp_test_state);
+
+    qobject_unref(qtest_qmp(qmp_test_state,
+                  "{ 'execute': 'qmp_capabilities' }"));
+    return sock_path;
+}
+
+

dito

tests/qtest/vhost-user-blk-test.c line 696

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+static void *vhost_user_blk_test_setup(GString *cmd_line, void *arg)
+{
+    char *sock_path1 = start_vhost_user_blk();
+    g_string_append_printf(cmd_line,
+                           " -object memory-backend-memfd,id=mem,size=128M,share=on -numa node,memdev=mem "
+                           "-chardev socket,id=char1,path=%s ", sock_path1);
+    return arg;
+}
+
+

dito

Quoted 34 lines
tests/qtest/vhost-user-blk-test.c new file +33 -0
+/*
+ * Setup for hotplug.
+ *
+ * Since vhost-user server only serves one vhost-user client one time,
+ * another exprot
+ *
+ */
+static void *vhost_user_blk_hotplug_test_setup(GString *cmd_line, void *arg)
+{
+    vhost_user_blk_test_setup(cmd_line, arg);
+    char *sock_path2 = start_vhost_user_blk();
+    /* "-chardev socket,id=char2" is used for pci_hotplug*/
+    g_string_append_printf(cmd_line, "-chardev socket,id=char2,path=%s",
+                           sock_path2);
+    return arg;
+}
+
+static void register_vhost_user_blk_test(void)
+{
+    QOSGraphTestOptions opts = {
+        .before = vhost_user_blk_test_setup,
+    };
+
+    /*
+     * tests for vhost-user-blk and vhost-user-blk-pci
+     * The tests are borrowed from tests/virtio-blk-test.c. But some tests
+     * regarding block_resize don't work for vhost-user-blk.
+     * vhost-user-blk device doesn't have -drive, so tests containing
+     * block_resize are also abandoned,
+     *  - config
+     *  - resize
+     */
+    qos_add_test("basic", "vhost-user-blk", basic, &opts);

tests/qtest/vhost-user-blk-test.c line 739

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+    qos_add_test("indirect", "vhost-user-blk", indirect, &opts);
+    qos_add_test("idx", "vhost-user-blk-pci", idx, &opts);
+    qos_add_test("nxvirtq", "vhost-user-blk-pci",
+                 test_nonexistent_virtqueue, &opts);
+
+    opts.before = vhost_user_blk_hotplug_test_setup;
+    qos_add_test("hotplug", "vhost-user-blk-pci", pci_hotplug, &opts);
+}
+
+libqos_init(register_vhost_user_blk_test);

Thomas

author
#36 lore raw

Re: [PATCH v9 5/5] new qTest case to test the vhost-user-blk-server

Quoted 53 lines · Thomas Huth
Quoted 12 lines · Coiby Xu

This test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize. Since vhost-user server can only server one client one time, two instances of qemu-storage-daemon are launched for the hotplug test.

In order to not block scripts/tap-driver.pl, vhost-user-blk-server will send "quit" command to qemu-storage-daemon's QMP monitor. So a function is added to libqtest.c to establish socket connection with socket server.

---

[...]

Quoted 26 lines
tests/qtest/libqtest.c +11 -2
index 49075b55a1..02cc09f893 100644
@@ -52,8 +52,7 @@ typedef struct QTestClientTransportOps {
     QTestRecvFn     recv_line; /* for receiving qtest command responses */
 } QTestTransportOps;
-struct QTestState
-{
+struct QTestState {
     int fd;
     int qmp_fd;
     pid_t qemu_pid;  /* our child QEMU process */
@@ -608,6 +607,38 @@ QDict *qtest_qmp_receive(QTestState *s)
     return qmp_fd_receive(s->qmp_fd);
 }
+QTestState *qtest_create_state_with_qmp_fd(int fd)
+{
+    QTestState *qmp_test_state = g_new0(QTestState, 1);
+    qmp_test_state->qmp_fd = fd;
+    return qmp_test_state;
+}
+
+int qtest_socket_client(char *server_socket_path)
+{
+    struct sockaddr_un serv_addr;

tests/qtest/libqtest.c QDict *qtest_qmp_receive(QTestState *s) · line 628

Quoted 10 lines
tests/qtest/libqtest.c +10 -0
+    int sock;
+    int ret;
+    int retries = 0;
+    sock = socket(PF_UNIX, SOCK_STREAM, 0);
+    g_assert_cmpint(sock, !=, -1);
+    serv_addr.sun_family = AF_UNIX;
+    snprintf(serv_addr.sun_path, sizeof(serv_addr.sun_path), "%s",
+             server_socket_path);
+
+    do {

Why not simply:

Quoted 3 lines

for (retries = 0; retries < 3; retries++)

?

Thank you for the advice which has been applied to v10.

Quoted 34 lines
Quoted 11 lines
tests/qtest/libqtest.c +11 -0
+        ret = connect(sock, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
+        if (ret == 0) {
+            break;
+        }
+        retries += 1;
+        g_usleep(G_USEC_PER_SEC);
+    } while (retries < 3);
+
+    g_assert_cmpint(ret, ==, 0);
+    return sock;
+}

[...]

Quoted 22 lines
tests/qtest/vhost-user-blk-test.c new file +16 -0
new file mode 100644
index 0000000000..56e3d8f338
@@ -0,0 +1,739 @@
+/*
+ * QTest testcase for VirtIO Block Device
+ *
+ * Copyright (c) 2014 SUSE LINUX Products GmbH
+ * Copyright (c) 2014 Marc Marí
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest-single.h"
+#include "qemu/bswap.h"
+#include "qemu/module.h"
+#include "standard-headers/linux/virtio_blk.h"
+#include "standard-headers/linux/virtio_pci.h"
Quoted 8 lines

tests/qtest/vhost-user-blk-test.c line 22

Quoted 6 lines
tests/qtest/vhost-user-blk-test.c new file +6 -0
+#include "libqos/qgraph.h"
+#include "libqos/vhost-user-blk.h"
+#include "libqos/libqos-pc.h"
+
+/* TODO actually test the results and get rid of this */
+#define qmp_discard_response(...) qobject_unref(qmp(__VA_ARGS__))

Please no more qmp_discard_response() in new code!

I planned to check if the response has the dict key "return" or "QMP". But when sometimes the SHUTDOWN event is returned which doesn't has the key "QMP". I'm not sure if QMI will change in the fure. So I use qobject_unref instead which I don't think will affect the correctness of the tests.

Quoted 149 lines
Quoted 19 lines
tests/qtest/vhost-user-blk-test.c new file +19 -0
+#define TEST_IMAGE_SIZE         (64 * 1024 * 1024)
+#define QVIRTIO_BLK_TIMEOUT_US  (30 * 1000 * 1000)
+#define PCI_SLOT_HP             0x06
+
+typedef struct QVirtioBlkReq {
+    uint32_t type;
+    uint32_t ioprio;
+    uint64_t sector;
+    char *data;
+    uint8_t status;
+} QVirtioBlkReq;
+
+
+#ifdef HOST_WORDS_BIGENDIAN
+static const bool host_is_big_endian = true;
+#else
+static const bool host_is_big_endian; /* false */
+#endif
+

tests/qtest/vhost-user-blk-test.c line 52

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+static inline void virtio_blk_fix_request(QVirtioDevice *d, QVirtioBlkReq *req)
+{
+    if (qvirtio_is_big_endian(d) != host_is_big_endian) {
+        req->type = bswap32(req->type);
+        req->ioprio = bswap32(req->ioprio);
+        req->sector = bswap64(req->sector);
+    }
+}
+
+

One empty line should be enough ;-)

Quoted 106 lines
tests/qtest/vhost-user-blk-test.c new file +106 -0
+static inline void virtio_blk_fix_dwz_hdr(QVirtioDevice *d,
+    struct virtio_blk_discard_write_zeroes *dwz_hdr)
+{
+    if (qvirtio_is_big_endian(d) != host_is_big_endian) {
+        dwz_hdr->sector = bswap64(dwz_hdr->sector);
+        dwz_hdr->num_sectors = bswap32(dwz_hdr->num_sectors);
+        dwz_hdr->flags = bswap32(dwz_hdr->flags);
+    }
+}
+
+static uint64_t virtio_blk_request(QGuestAllocator *alloc, QVirtioDevice *d,
+                                   QVirtioBlkReq *req, uint64_t data_size)
+{
+    uint64_t addr;
+    uint8_t status = 0xFF;
+
+    switch (req->type) {
+    case VIRTIO_BLK_T_IN:
+    case VIRTIO_BLK_T_OUT:
+        g_assert_cmpuint(data_size % 512, ==, 0);
+        break;
+    case VIRTIO_BLK_T_DISCARD:
+    case VIRTIO_BLK_T_WRITE_ZEROES:
+        g_assert_cmpuint(data_size %
+                         sizeof(struct virtio_blk_discard_write_zeroes), ==, 0);
+        break;
+    default:
+        g_assert_cmpuint(data_size, ==, 0);
+    }
+
+    addr = guest_alloc(alloc, sizeof(*req) + data_size);
+
+    virtio_blk_fix_request(d, req);
+
+    memwrite(addr, req, 16);
+    memwrite(addr + 16, req->data, data_size);
+    memwrite(addr + 16 + data_size, &status, sizeof(status));
+
+    return addr;
+}
+
+/* Returns the request virtqueue so the caller can perform further tests */
+static QVirtQueue *test_basic(QVirtioDevice *dev, QGuestAllocator *alloc)
+{
+    QVirtioBlkReq req;
+    uint64_t req_addr;
+    uint64_t capacity;
+    uint64_t features;
+    uint32_t free_head;
+    uint8_t status;
+    char *data;
+    QTestState *qts = global_qtest;
+    QVirtQueue *vq;
+
+    features = qvirtio_get_features(dev);
+    features = features & ~(QVIRTIO_F_BAD_FEATURE |
+                    (1u << VIRTIO_RING_F_INDIRECT_DESC) |
+                    (1u << VIRTIO_RING_F_EVENT_IDX) |
+                    (1u << VIRTIO_BLK_F_SCSI));
+    qvirtio_set_features(dev, features);
+
+    capacity = qvirtio_config_readq(dev, 0);
+    g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512);
+
+    vq = qvirtqueue_setup(dev, alloc, 0);
+
+    qvirtio_set_driver_ok(dev);
+
+    /* Write and read with 3 descriptor layout */
+    /* Write request */
+    req.type = VIRTIO_BLK_T_OUT;
+    req.ioprio = 1;
+    req.sector = 0;
+    req.data = g_malloc0(512);
+    strcpy(req.data, "TEST");
+
+    req_addr = virtio_blk_request(alloc, dev, &req, 512);
+
+    g_free(req.data);
+
+    free_head = qvirtqueue_add(qts, vq, req_addr, 16, false, true);
+    qvirtqueue_add(qts, vq, req_addr + 16, 512, false, true);
+    qvirtqueue_add(qts, vq, req_addr + 528, 1, true, false);
+
+    qvirtqueue_kick(qts, dev, vq, free_head);
+
+    qvirtio_wait_used_elem(qts, dev, vq, free_head, NULL,
+                           QVIRTIO_BLK_TIMEOUT_US);
+    status = readb(req_addr + 528);
+    g_assert_cmpint(status, ==, 0);
+
+    guest_free(alloc, req_addr);
+
+    /* Read request */
+    req.type = VIRTIO_BLK_T_IN;
+    req.ioprio = 1;
+    req.sector = 0;
+    req.data = g_malloc0(512);
+
+    req_addr = virtio_blk_request(alloc, dev, &req, 512);
+
+    g_free(req.data);
+
+    free_head = qvirtqueue_add(qts, vq, req_addr, 16, false, true);
+    qvirtqueue_add(qts, vq, req_addr + 16, 512, true, true);
+    qvirtqueue_add(qts, vq, req_addr + 528, 1, true, false);

tests/qtest/vhost-user-blk-test.c line 168

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+
+    qvirtqueue_kick(qts, dev, vq, free_head);
+
+    qvirtio_wait_used_elem(qts, dev, vq, free_head, NULL,
+                           QVIRTIO_BLK_TIMEOUT_US);
+    status = readb(req_addr + 528);
+    g_assert_cmpint(status, ==, 0);
+
+    data = g_malloc0(512);
+    memread(req_addr + 16, data, 512);
Quoted 7 lines

Since you have a "qts" variable here anyway, could you please use qtest_memread(qts, ...) here instead? (also in the other spots where you use memread and memwrite if possible) ... in case we ever have to introduce multiple test states later, we then don't have to rewrite the code anymore.
(generally, it's nice to avoid libqtest-single.h nowadays and only use libqtest.h if possible)

qtest_memread(qts, ...) has been used in v10. But libqtest-single.h is still needed to access
QTestState *global_qtest __attribute__((common, weak));.

Quoted 243 lines

[...]

Quoted 76 lines
tests/qtest/vhost-user-blk-test.c new file +76 -0
+static void indirect(void *obj, void *u_data, QGuestAllocator *t_alloc)
+{
+    QVirtQueue *vq;
+    QVhostUserBlk *blk_if = obj;
+    QVirtioDevice *dev = blk_if->vdev;
+    QVirtioBlkReq req;
+    QVRingIndirectDesc *indirect;
+    uint64_t req_addr;
+    uint64_t capacity;
+    uint64_t features;
+    uint32_t free_head;
+    uint8_t status;
+    char *data;
+    QTestState *qts = global_qtest;
+
+    features = qvirtio_get_features(dev);
+    g_assert_cmphex(features & (1u << VIRTIO_RING_F_INDIRECT_DESC), !=, 0);
+    features = features & ~(QVIRTIO_F_BAD_FEATURE |
+                            (1u << VIRTIO_RING_F_EVENT_IDX) |
+                            (1u << VIRTIO_BLK_F_SCSI));
+    qvirtio_set_features(dev, features);
+
+    capacity = qvirtio_config_readq(dev, 0);
+    g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512);
+
+    vq = qvirtqueue_setup(dev, t_alloc, 0);
+    qvirtio_set_driver_ok(dev);
+
+    /* Write request */
+    req.type = VIRTIO_BLK_T_OUT;
+    req.ioprio = 1;
+    req.sector = 0;
+    req.data = g_malloc0(512);
+    strcpy(req.data, "TEST");
+
+    req_addr = virtio_blk_request(t_alloc, dev, &req, 512);
+
+    g_free(req.data);
+
+    indirect = qvring_indirect_desc_setup(qts, dev, t_alloc, 2);
+    qvring_indirect_desc_add(dev, qts, indirect, req_addr, 528, false);
+    qvring_indirect_desc_add(dev, qts, indirect, req_addr + 528, 1, true);
+    free_head = qvirtqueue_add_indirect(qts, vq, indirect);
+    qvirtqueue_kick(qts, dev, vq, free_head);
+
+    qvirtio_wait_used_elem(qts, dev, vq, free_head, NULL,
+                           QVIRTIO_BLK_TIMEOUT_US);
+    status = readb(req_addr + 528);
+    g_assert_cmpint(status, ==, 0);
+
+    g_free(indirect);
+    guest_free(t_alloc, req_addr);
+
+    /* Read request */
+    req.type = VIRTIO_BLK_T_IN;
+    req.ioprio = 1;
+    req.sector = 0;
+    req.data = g_malloc0(512);
+    strcpy(req.data, "TEST");
+
+    req_addr = virtio_blk_request(t_alloc, dev, &req, 512);
+
+    g_free(req.data);
+
+    indirect = qvring_indirect_desc_setup(qts, dev, t_alloc, 2);
+    qvring_indirect_desc_add(dev, qts, indirect, req_addr, 16, false);
+    qvring_indirect_desc_add(dev, qts, indirect, req_addr + 16, 513, true);
+    free_head = qvirtqueue_add_indirect(qts, vq, indirect);
+    qvirtqueue_kick(qts, dev, vq, free_head);
+
+    qvirtio_wait_used_elem(qts, dev, vq, free_head, NULL,
+                           QVIRTIO_BLK_TIMEOUT_US);
+    status = readb(req_addr + 528);
+    g_assert_cmpint(status, ==, 0);
+
+    data = g_malloc0(512);

tests/qtest/vhost-user-blk-test.c line 415

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+    memread(req_addr + 16, data, 512);
+    g_assert_cmpstr(data, ==, "TEST");
+    g_free(data);
+
+    g_free(indirect);
+    guest_free(t_alloc, req_addr);
+    qvirtqueue_cleanup(dev->bus, vq, t_alloc);
+}
+
+

One empty line is enough.

[...]

tests/qtest/vhost-user-blk-test.c line 609

Quoted 8 lines
tests/qtest/vhost-user-blk-test.c new file +8 -0
+static void drive_destroy(void *path)
+{
+    unlink(path);
+    g_free(path);
+    qos_invalidate_command_line();
+}
+
+

dito.

[...]

Quoted 11 lines
tests/qtest/vhost-user-blk-test.c new file +11 -0
+static char *drive_create(void)
+{
+    int fd, ret;
+    /** vhost-user-blk won't recognize drive located in /tmp */
+    char *t_path = g_strdup("qtest.XXXXXX");
+
+    /** Create a temporary raw image */
+    fd = mkstemp(t_path);
+    g_assert_cmpint(fd, >=, 0);
+    ret = ftruncate(fd, TEST_IMAGE_SIZE);
+    g_assert_cmpint(ret, ==, 0);

tests/qtest/vhost-user-blk-test.c line 630

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+    close(fd);
+
+    g_test_queue_destroy(drive_destroy, t_path);
+    return t_path;
+}
+
+static char sock_path_tempate[] = "/tmp/qtest.vhost_user_blk.XXXXXX";
+static char qmp_sock_path_tempate[] = "/tmp/qtest.vhost_user_blk.qmp.XXXXXX";
+
+

dito.

Quoted 7 lines
tests/qtest/vhost-user-blk-test.c new file +7 -0
+static void quit_storage_daemon(void *qmp_test_state)
+{
+    qobject_unref(qtest_qmp((QTestState *)qmp_test_state, "{ 'execute': 'quit' }"));
+    g_free(qmp_test_state);
+}
+
+static char *start_vhost_user_blk(void)

tests/qtest/vhost-user-blk-test.c line 647

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+{
+    int fd, qmp_fd;
+    char *sock_path = g_strdup(sock_path_tempate);
+    char *qmp_sock_path = g_strdup(qmp_sock_path_tempate);
+    QTestState *qmp_test_state;
+    fd = mkstemp(sock_path);
+    g_assert_cmpint(fd, >=, 0);
+    g_test_queue_destroy(drive_destroy, sock_path);
+
+

dito.

Quoted 7 lines
tests/qtest/vhost-user-blk-test.c new file +7 -0
+    qmp_fd = mkstemp(qmp_sock_path);
+    g_assert_cmpint(qmp_fd, >=, 0);
+    g_test_queue_destroy(drive_destroy, qmp_sock_path);
+
+    /* create image file */
+    const char *img_path = drive_create();
+

tests/qtest/vhost-user-blk-test.c line 664

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+    const char *vhost_user_blk_bin = qtest_qemu_storage_daemon_binary();
+    gchar *command = g_strdup_printf(
+            "exec %s "
+            "--blockdev driver=file,node-name=disk,filename=%s "
+            "--object vhost-user-blk-server,id=disk,unix-socket=%s,"
+            "node-name=disk,writable=on "
+            "--chardev socket,id=qmp,path=%s,server,nowait --monitor chardev=qmp",
+            vhost_user_blk_bin, img_path, sock_path, qmp_sock_path);
+
+

dito.

Quoted 12 lines
tests/qtest/vhost-user-blk-test.c new file +12 -0
+    g_test_message("starting vhost-user backend: %s", command);
+    pid_t pid = fork();
+    if (pid == 0) {
+        execlp("/bin/sh", "sh", "-c", command, NULL);
+        exit(1);
+    }
+    g_free(command);
+
+    qmp_test_state = qtest_create_state_with_qmp_fd(
+                             qtest_socket_client(qmp_sock_path));
+    /*
+     * Ask qemu-storage-daemon to quit so it

tests/qtest/vhost-user-blk-test.c line 686

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+     * will not block scripts/tap-driver.pl.
+     */
+    g_test_queue_destroy(quit_storage_daemon, qmp_test_state);
+
+    qobject_unref(qtest_qmp(qmp_test_state,
+                  "{ 'execute': 'qmp_capabilities' }"));
+    return sock_path;
+}
+
+

dito

tests/qtest/vhost-user-blk-test.c line 696

Quoted 10 lines
tests/qtest/vhost-user-blk-test.c new file +10 -0
+static void *vhost_user_blk_test_setup(GString *cmd_line, void *arg)
+{
+    char *sock_path1 = start_vhost_user_blk();
+    g_string_append_printf(cmd_line,
+                           " -object memory-backend-memfd,id=mem,size=128M,share=on -numa node,memdev=mem "
+                           "-chardev socket,id=char1,path=%s ", sock_path1);
+    return arg;
+}
+
+

dito

Quoted 38 lines
tests/qtest/vhost-user-blk-test.c new file +38 -0
+/*
+ * Setup for hotplug.
+ *
+ * Since vhost-user server only serves one vhost-user client one time,
+ * another exprot
+ *
+ */
+static void *vhost_user_blk_hotplug_test_setup(GString *cmd_line, void *arg)
+{
+    vhost_user_blk_test_setup(cmd_line, arg);
+    char *sock_path2 = start_vhost_user_blk();
+    /* "-chardev socket,id=char2" is used for pci_hotplug*/
+    g_string_append_printf(cmd_line, "-chardev socket,id=char2,path=%s",
+                           sock_path2);
+    return arg;
+}
+
+static void register_vhost_user_blk_test(void)
+{
+    QOSGraphTestOptions opts = {
+        .before = vhost_user_blk_test_setup,
+    };
+
+    /*
+     * tests for vhost-user-blk and vhost-user-blk-pci
+     * The tests are borrowed from tests/virtio-blk-test.c. But some tests
+     * regarding block_resize don't work for vhost-user-blk.
+     * vhost-user-blk device doesn't have -drive, so tests containing
+     * block_resize are also abandoned,
+     *  - config
+     *  - resize
+     */
+    qos_add_test("basic", "vhost-user-blk", basic, &opts);
+    qos_add_test("indirect", "vhost-user-blk", indirect, &opts);
+    qos_add_test("idx", "vhost-user-blk-pci", idx, &opts);
+    qos_add_test("nxvirtq", "vhost-user-blk-pci",
+                 test_nonexistent_virtqueue, &opts);
+
Quoted 8 lines

tests/qtest/vhost-user-blk-test.c line 739

Quoted 6 lines
tests/qtest/vhost-user-blk-test.c new file +5 -0
+    opts.before = vhost_user_blk_hotplug_test_setup;
+    qos_add_test("hotplug", "vhost-user-blk-pci", pci_hotplug, &opts);
+}
+
+libqos_init(register_vhost_user_blk_test);

Thomas

All extra empty lines have been removed in v10:)

--
Best regards,
Coiby

#37 lore raw
In reply to Coiby Xu
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===

CC hw/gpio/trace.o
CC hw/riscv/trace.o
/tmp/qemu-test/src/util/vhost-user-server.c: In function 'vu_message_read':
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: 'VHOST_MEMORY_MAX_NREGIONS' undeclared (first use in this function)

                              VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                              ^

/tmp/qemu-test/src/util/vhost-user-server.c:142:30: note: each undeclared identifier is reported only once for each function it appears in
make: *** [util/vhost-user-server.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
File "./tests/docker/docker.py", line 669, in <module>
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=408a16efe57d44399707cea73c0c46cb', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-hyuz9qap/src/docker-src.2020-06-14-15.10.05.3902:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=408a16efe57d44399707cea73c0c46cb
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-hyuz9qap/src'
make: *** [docker-run-test-quick@centos7] Error 2

real 2m3.838s
user 0m8.880s

The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#38 lore raw
In reply to Coiby Xu
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===

  CC      stubs/vm-stop.o
  CC      ui/input-keymap.o
  CC      qemu-keymap.o
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                             ^

1 error generated.
make: *** [/tmp/qemu-test/src/rules.mak:69: util/vhost-user-server.o] Error 1
make: *** Waiting for unfinished jobs....
CC contrib/elf2dmp/main.o
Traceback (most recent call last):
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=99a7788ed7f54df1b0ebb7f91beb522e', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-9goaol7z/src/docker-src.2020-06-14-15.12.59.10439:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=99a7788ed7f54df1b0ebb7f91beb522e
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-9goaol7z/src'
make: *** [docker-run-test-debug@fedora] Error 2

real 3m29.335s
user 0m8.313s

The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

author
#39 lore raw
Quoted 24 lines · On Sun, Jun 14, 2020 at 12:16:28PM -0700, [email protected] wrote:

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===

 CC      stubs/vm-stop.o
 CC      ui/input-keymap.o
 CC      qemu-keymap.o
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
                            VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                            ^
Quoted 2 lines

The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.

I couldn't re-produce this error locally for both docker-test-quick@centos7 and this docker test. And I can't see any reason for this error to occur since VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h which has been included by util/vhost-user-server.h.

--
Best regards,
Coiby

#40 lore raw
Quoted 25 lines · Coiby Xu
Quoted 24 lines · On Sun, Jun 14, 2020 at 12:16:28PM -0700, [email protected] wrote:

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===

 CC      stubs/vm-stop.o
 CC      ui/input-keymap.o
 CC      qemu-keymap.o
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
                            VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                            ^
Quoted 7 lines
Quoted 2 lines

The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.

I couldn't re-produce this error locally for both docker-test-quick@centos7 and this docker test. And I can't see any reason for this error to occur since VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h which has been included by util/vhost-user-server.h.

Please see the recent change in commit b650d5f4b1cd3f9f8c4fdb319838c5c1e0695e41 ("Lift max ram slots limit in libvhost-user").

The error can be solved by replacing VHOST_MEMORY_MAX_NREGIONS with VHOST_MEMORY_BASELINE_NREGIONS in util/vhost-user-server.c.

Stefan

author
#41 lore raw
Quoted 34 lines · Stefan Hajnoczi
Quoted 25 lines · Coiby Xu
Quoted 24 lines · On Sun, Jun 14, 2020 at 12:16:28PM -0700, [email protected] wrote:

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===

 CC      stubs/vm-stop.o
 CC      ui/input-keymap.o
 CC      qemu-keymap.o
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
                            VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                            ^
Quoted 7 lines
Quoted 2 lines

The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.

I couldn't re-produce this error locally for both docker-test-quick@centos7 and this docker test. And I can't see any reason for this error to occur since VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h which has been included by util/vhost-user-server.h.

Quoted 6 lines

Please see the recent change in commit b650d5f4b1cd3f9f8c4fdb319838c5c1e0695e41 ("Lift max ram slots limit in libvhost-user").

The error can be solved by replacing VHOST_MEMORY_MAX_NREGIONS with VHOST_MEMORY_BASELINE_NREGIONS in util/vhost-user-server.c.

Thank you for the clarification! I did run "git pull" when checking this error. It seems there is a delay when pulling updates from git://git.qemu.org/qemu.git.

--
Best regards,
Coiby

#42 lore raw
In reply to Coiby Xu
Quoted 25 lines · Coiby Xu
Quoted 24 lines · On Sun, Jun 14, 2020 at 12:16:28PM -0700, [email protected] wrote:

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===

 CC      stubs/vm-stop.o
 CC      ui/input-keymap.o
 CC      qemu-keymap.o
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
                            VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                            ^
Quoted 7 lines
Quoted 2 lines

The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.

I couldn't re-produce this error locally for both docker-test-quick@centos7 and this docker test. And I can't see any reason for this error to occur since VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h which has been included by util/vhost-user-server.h.

Using G_N_ELEMENTS(vmsg->fds) instead of VHOST_MEMORY_MAX_NREGIONS is an even cleaner fix.

Stefan

author
#43 lore raw
Quoted 34 lines · Stefan Hajnoczi
Quoted 25 lines · Coiby Xu
Quoted 24 lines · On Sun, Jun 14, 2020 at 12:16:28PM -0700, [email protected] wrote:

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===

 CC      stubs/vm-stop.o
 CC      ui/input-keymap.o
 CC      qemu-keymap.o
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
                            VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                            ^
Quoted 7 lines
Quoted 2 lines

The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.

I couldn't re-produce this error locally for both docker-test-quick@centos7 and this docker test. And I can't see any reason for this error to occur since VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h which has been included by util/vhost-user-server.h.

Quoted 4 lines

Using G_N_ELEMENTS(vmsg->fds) instead of VHOST_MEMORY_MAX_NREGIONS is an even cleaner fix.

Stefan

Thank you for this cleaner fix!

--
Best regards,
Coiby

#44 lore raw
In reply to Coiby Xu
Quoted 3 lines · Coiby Xu

v9
- move logical block size check function to a utility function
- fix issues regarding license, coding style, memory deallocation, etc.

I have replied with patches that you can consider squashing into your series. I was testing this patch series and decided it was easier to send code than to go back and write review comments since I was already on a git branch.

My patches can be combined into your original patches using "git rebase -i" and the "fixup" or "squash" directive.

Please add my Signed-off-by: line to affected patches when squashing patches so that the git log records that I have confirmed that I have permission to contribute this code.

If you have questions about any of the patches, please let me know.

Stefan

author
#45 lore raw
Quoted 10 lines · Stefan Hajnoczi
Quoted 3 lines · Coiby Xu

v9
- move logical block size check function to a utility function
- fix issues regarding license, coding style, memory deallocation, etc.

I have replied with patches that you can consider squashing into your series. I was testing this patch series and decided it was easier to send code than to go back and write review comments since I was already on a git branch.

Quoted 6 lines

My patches can be combined into your original patches using "git rebase -i" and the "fixup" or "squash" directive.

Please add my Signed-off-by: line to affected patches when squashing patches so that the git log records that I have confirmed that I have permission to contribute this code.

I was thinking on how to incorporate your work while reading the emails. You just provide the instruction! Thank you!

--
Best regards,
Coiby

author
#46 lore raw
In reply to Stefan Hajnoczi
Quoted 10 lines · Stefan Hajnoczi
Quoted 3 lines · Coiby Xu

v9
- move logical block size check function to a utility function
- fix issues regarding license, coding style, memory deallocation, etc.

I have replied with patches that you can consider squashing into your series. I was testing this patch series and decided it was easier to send code than to go back and write review comments since I was already on a git branch.

Quoted 8 lines

My patches can be combined into your original patches using "git rebase -i" and the "fixup" or "squash" directive.

Please add my Signed-off-by: line to affected patches when squashing patches so that the git log records that I have confirmed that I have permission to contribute this code.

If you have questions about any of the patches, please let me know.

Besides your Signed-off-by: line, shouldn't I also add copyright info to the affected files as follows?

* Copyright (c) 2020 Coiby Xu <[email protected]>
*
* Copyright (c) 2020 Red Hat, Inc., Stefan Hajnoczi <[email protected]> *

--
Best regards,
Coiby

#47 lore raw
Quoted 18 lines · Coiby Xu
Quoted 17 lines · Stefan Hajnoczi
Quoted 3 lines · Coiby Xu

v9
- move logical block size check function to a utility function
- fix issues regarding license, coding style, memory deallocation, etc.

I have replied with patches that you can consider squashing into your series. I was testing this patch series and decided it was easier to send code than to go back and write review comments since I was already on a git branch.

My patches can be combined into your original patches using "git rebase -i" and the "fixup" or "squash" directive.

Please add my Signed-off-by: line to affected patches when squashing patches so that the git log records that I have confirmed that I have permission to contribute this code.

Quoted 8 lines
Quoted 1 line

If you have questions about any of the patches, please let me know.

Besides your Signed-off-by: line, shouldn't I also add copyright info to the affected files as follows?

* Copyright (c) 2020 Coiby Xu <[email protected]>
*
* Copyright (c) 2020 Red Hat, Inc., Stefan Hajnoczi <[email protected]>

The following would be good:

* Copyright (c) 2020 Red Hat, Inc.

Thanks,
Stefan

#48 lore raw
In reply to Coiby Xu
Quoted 3 lines · Coiby Xu

v9
- move logical block size check function to a utility function
- fix issues regarding license, coding style, memory deallocation, etc.

Great to see you are back, Coiby! Looking forward to reviewing v10.

Stefan

#49 lore raw
In reply to Coiby Xu
Quoted 3 lines · Coiby Xu

v9
- move logical block size check function to a utility function
- fix issues regarding license, coding style, memory deallocation, etc.

Hi,
Any update on v10?

Please let me know if there's anything I can do to help.

Stefan

author
#50 lore raw
Quoted 11 lines · Stefan Hajnoczi
Quoted 3 lines · Coiby Xu

v9
- move logical block size check function to a utility function
- fix issues regarding license, coding style, memory deallocation, etc.

Hi,
Any update on v10?

Please let me know if there's anything I can do to help.

Stefan

Thank you for ping me! v10 has been submitted.

--
Best regards,
Coiby