Skip to contentSkip to conversation

block/export: convert vhost-user-blk-server to block exports API

PATCH Superseded Stefan Hajnoczi 2 participants 14 messages Sep 22, 2020 – Sep 23, 2020

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover block/export: convert vhost-user-blk-server to block exports API

This patch series converts Coiby Xu's vhost-user-blk-server from a QOM object to the block exports API. The block exports API provides a standard QMP and command-line interface for managing block exports (NBD, FUSE, vhost-user-blk, etc). A fair amount of init/shutdown code is removed because the block exports API already takes care of that functionality.

Most of the patches are vhost-user-blk-server cleanups.

The syntax for launching qemu-storage-daemon is:

  $ qemu-storage-daemon \
      --blockdev file,node-name=drive0,filename=test.img \
      --export vhost-user-blk,node-name=drive0,id=export0,writable=on,unix-socket=/tmp/vhost-user-blk.sock

QEMU can connect to the vhost-user-blk export like this:

  $ qemu-system-x86_64 \
      -M accel=kvm,memory-backend=mem \
      -m 1G \
      -object memory-backend-memfd,size=1G,id=mem \
      -cpu host \
      -chardev socket,path=/tmp/vhost-user-blk.sock,id=char0 \
      -device vhost-user-blk-pci,chardev=char0

Based-on: [email protected] ("[PATCH v10 0/7] vhost-user block device backend implementation")
Based-on: [email protected] ("[PATCH 00/29] block/export: Add infrastructure and QAPI for block exports")

Stefan Hajnoczi (11):
block/export: shorten serial string to fit util/vhost-user-server: s/fileds/fields/ typo fix util/vhost-user-server: drop unnecessary QOM cast util/vhost-user-server: drop unnecessary watch deletion block/export: consolidate request structs into VuBlockReq util/vhost-user-server: drop unused DevicePanicNotifier util/vhost-user-server: fix memory leak in vu_message_read() util/vhost-user-server: check EOF when reading payload util/vhost-user-server: rework vu_client_trip() coroutine lifecycle block/export: report flush errors
block/export: convert vhost-user-blk server to block export API

 qapi/block-export.json               |  19 +-
 block/export/vhost-user-blk-server.h |  23 +-
 util/vhost-user-server.h             |  32 +-
 block/export/export.c                |   8 +-
 block/export/vhost-user-blk-server.c | 534 ++++++++-------------------
 util/vhost-user-server.c             | 322 ++++++++--------
 block/export/meson.build             |   1 +
 block/meson.build                    |   1 -
 8 files changed, 360 insertions(+), 580 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/11 block/export: shorten serial string to fit

The serial field is only 20 bytes long. Shorten the string so it fits.

This fixes the following compiler warning:

../block/export/vhost-user-blk-server.c:178:50: error: ‘%s’ directive output truncated writing 21 bytes into a region of size 20 [-Werror=format-truncation=]

  178 |         snprintf(elem->in_sg[0].iov_base, size, "%s", "vhost_user_blk_server");
      |                                                  ^~   ~~~~~~~~~~~~~~~~~~~~~~~
---
 block/export/vhost-user-blk-server.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/export/vhost-user-blk-server.c +1 -1
index ec78130f09..fb7764a730 100644
@@ -175,7 +175,7 @@ static void coroutine_fn vu_block_virtio_process_req(void *opaque)
     case VIRTIO_BLK_T_GET_ID: {
         size_t size = MIN(iov_size(&elem->in_sg[0], in_num),
                           VIRTIO_BLK_ID_BYTES);
-        snprintf(elem->in_sg[0].iov_base, size, "%s", "vhost_user_blk_server");
+        snprintf(elem->in_sg[0].iov_base, size, "%s", "vhost_user_blk");
         req->in->status = VIRTIO_BLK_S_OK;
         req->size = elem->in_sg[0].iov_len;
         break;
Signature · 1 line
2.26.2
#2 lore raw

PATCH 2/11 util/vhost-user-server: s/fileds/fields/ typo fix

In reply to Stefan Hajnoczi
---
 util/vhost-user-server.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
util/vhost-user-server.c +1 -1
index 7b50a2b1fd..2cd0cf8277 100644
@@ -407,7 +407,7 @@ bool vhost_user_server_start(VuServer *server,
         return false;
     }
 
-    /* zero out unspecified fileds */
+    /* zero out unspecified fields */
     *server = (VuServer) {
         .listener              = listener,
         .vu_iface              = vu_iface,
Signature · 1 line
2.26.2
#3 lore raw

PATCH 3/11 util/vhost-user-server: drop unnecessary QOM cast

In reply to Stefan Hajnoczi

We already have access to the value with the correct type (ioc and sioc are the same QIOChannel).

---
 util/vhost-user-server.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
util/vhost-user-server.c +1 -1
index 2cd0cf8277..ebe47885f5 100644
@@ -337,7 +337,7 @@ static void vu_accept(QIONetListener *listener, QIOChannelSocket *sioc,
     server->ioc = QIO_CHANNEL(sioc);
     object_ref(OBJECT(server->ioc));
     qio_channel_attach_aio_context(server->ioc, server->ctx);
-    qio_channel_set_blocking(QIO_CHANNEL(server->sioc), false, NULL);
+    qio_channel_set_blocking(server->ioc, false, NULL);
     vu_client_start(server);
 }
 
Signature · 1 line
2.26.2
#4 lore raw

PATCH 4/11 util/vhost-user-server: drop unnecessary watch deletion

In reply to Stefan Hajnoczi

Explicitly deleting watches is not necessary since libvhost-user calls remove_watch() during vu_deinit(). Add an assertion to check this though.

---
 util/vhost-user-server.c | 19 ++++---------------
 1 file changed, 4 insertions(+), 15 deletions(-)
util/vhost-user-server.c +4 -15
index ebe47885f5..ebb850731b 100644
@@ -48,21 +48,6 @@ static void close_client(VuServer *server)
     /* When this is set vu_client_trip will stop new processing vhost-user message */
     server->sioc = NULL;
 
-    VuFdWatch *vu_fd_watch, *next;
-    QTAILQ_FOREACH_SAFE(vu_fd_watch, &server->vu_fd_watches, next, next) {
-        aio_set_fd_handler(server->ioc->ctx, vu_fd_watch->fd, true, NULL,
-                           NULL, NULL, NULL);
-    }
-
-    while (!QTAILQ_EMPTY(&server->vu_fd_watches)) {
-        QTAILQ_FOREACH_SAFE(vu_fd_watch, &server->vu_fd_watches, next, next) {
-            if (!vu_fd_watch->processing) {
-                QTAILQ_REMOVE(&server->vu_fd_watches, vu_fd_watch, next);
-                g_free(vu_fd_watch);
-            }
-        }
-    }
-
     while (server->processing_msg) {
         if (server->ioc->read_coroutine) {
             server->ioc->read_coroutine = NULL;
@@ -73,6 +58,10 @@ static void close_client(VuServer *server)
     }
 
     vu_deinit(&server->vu_dev);
+
+    /* vu_deinit() should have called remove_watch() */
+    assert(QTAILQ_EMPTY(&server->vu_fd_watches));
+
     object_unref(OBJECT(sioc));
     object_unref(OBJECT(server->ioc));
 }
Signature · 1 line
2.26.2
#5 lore raw

PATCH 5/11 block/export: consolidate request structs into VuBlockReq

In reply to Stefan Hajnoczi

Only one struct is needed per request. Drop req_data and the separate VuBlockReq instance. Instead let vu_queue_pop() allocate everything at once.

This fixes the req_data memory leak in vu_block_virtio_process_req().

---
 block/export/vhost-user-blk-server.c | 68 +++++++++-------------------
 1 file changed, 21 insertions(+), 47 deletions(-)
block/export/vhost-user-blk-server.c +21 -47
index fb7764a730..ef07a87eb1 100644
@@ -25,7 +25,7 @@ struct virtio_blk_inhdr {
 };
 
 typedef struct VuBlockReq {
-    VuVirtqElement *elem;
+    VuVirtqElement elem;
     int64_t sector_num;
     size_t size;
     struct virtio_blk_inhdr *in;
@@ -39,14 +39,10 @@ static void vu_block_req_complete(VuBlockReq *req)
     VuDev *vu_dev = &req->server->vu_dev;
 
     /* IO size with 1 extra status byte */
-    vu_queue_push(vu_dev, req->vq, req->elem, req->size + 1);
+    vu_queue_push(vu_dev, req->vq, &req->elem, req->size + 1);
     vu_queue_notify(vu_dev, req->vq);
 
-    if (req->elem) {
-        free(req->elem);
-    }
-
-    g_free(req);
+    free(req);
 }
 
 static VuBlockDev *get_vu_block_device_by_server(VuServer *server)
@@ -89,20 +85,12 @@ static void coroutine_fn vu_block_flush(VuBlockReq *req)
     blk_co_flush(backend);
 }
 
-struct req_data {
-    VuServer *server;
-    VuVirtq *vq;
-    VuVirtqElement *elem;
-};
-
 static void coroutine_fn vu_block_virtio_process_req(void *opaque)
 {
-    struct req_data *data = opaque;
-    VuServer *server = data->server;
-    VuVirtq *vq = data->vq;
-    VuVirtqElement *elem = data->elem;
+    VuBlockReq *req = opaque;
+    VuServer *server = req->server;
+    VuVirtqElement *elem = &req->elem;
     uint32_t type;
-    VuBlockReq *req;
 
     VuBlockDev *vdev_blk = get_vu_block_device_by_server(server);
     BlockBackend *backend = vdev_blk->backend;
@@ -111,18 +99,13 @@ static void coroutine_fn vu_block_virtio_process_req(void *opaque)
     struct iovec *out_iov = elem->out_sg;
     unsigned in_num = elem->in_num;
     unsigned out_num = elem->out_num;
+
     /* refer to hw/block/virtio_blk.c */
     if (elem->out_num < 1 || elem->in_num < 1) {
         error_report("virtio-blk request missing headers");
-        free(elem);
-        return;
+        goto err;
     }
 
-    req = g_new0(VuBlockReq, 1);
-    req->server = server;
-    req->vq = vq;
-    req->elem = elem;
-
     if (unlikely(iov_to_buf(out_iov, out_num, 0, &req->out,
                             sizeof(req->out)) != sizeof(req->out))) {
         error_report("virtio-blk request outhdr too short");
@@ -202,36 +185,27 @@ static void coroutine_fn vu_block_virtio_process_req(void *opaque)
 
 err:
     free(elem);
-    g_free(req);
-    return;
 }
 
 static void vu_block_process_vq(VuDev *vu_dev, int idx)
 {
-    VuServer *server;
-    VuVirtq *vq;
-    struct req_data *req_data;
+    VuServer *server = container_of(vu_dev, VuServer, vu_dev);
+    VuVirtq *vq = vu_get_queue(vu_dev, idx);
 
-    server = container_of(vu_dev, VuServer, vu_dev);
-    assert(server);
-
-    vq = vu_get_queue(vu_dev, idx);
-    assert(vq);
-    VuVirtqElement *elem;
     while (1) {
-        elem = vu_queue_pop(vu_dev, vq, sizeof(VuVirtqElement) +
-                                    sizeof(VuBlockReq));
-        if (elem) {
-            req_data = g_new0(struct req_data, 1);
-            req_data->server = server;
-            req_data->vq = vq;
-            req_data->elem = elem;
-            Coroutine *co = qemu_coroutine_create(vu_block_virtio_process_req,
-                                                  req_data);
-            aio_co_enter(server->ioc->ctx, co);
-        } else {
+        VuBlockReq *req;
+
+        req = vu_queue_pop(vu_dev, vq, sizeof(VuBlockReq));
+        if (!req) {
             break;
         }
+
+        req->server = server;
+        req->vq = vq;
+
+        Coroutine *co =
+            qemu_coroutine_create(vu_block_virtio_process_req, req);
+        qemu_coroutine_enter(co);
     }
 }
 
Signature · 1 line
2.26.2
#6 lore raw

PATCH 6/11 util/vhost-user-server: drop unused DevicePanicNotifier

In reply to Stefan Hajnoczi

The device panic notifier callback is not used. Drop it.

---
 util/vhost-user-server.h             | 3 ---
 block/export/vhost-user-blk-server.c | 3 +--
 util/vhost-user-server.c             | 6 ------
 3 files changed, 1 insertion(+), 11 deletions(-)
util/vhost-user-server.h +0 -3
index 5232f96718..92177fc911 100644
@@ -29,12 +29,10 @@ typedef struct VuFdWatch {
 } VuFdWatch;
 
 typedef struct VuServer VuServer;
-typedef void DevicePanicNotifierFn(VuServer *server);
 
 struct VuServer {
     QIONetListener *listener;
     AioContext *ctx;
-    DevicePanicNotifierFn *device_panic_notifier;
     int max_queues;
     const VuDevIface *vu_iface;
     VuDev vu_dev;
@@ -54,7 +52,6 @@ bool vhost_user_server_start(VuServer *server,
                              SocketAddress *unix_socket,
                              AioContext *ctx,
                              uint16_t max_queues,
-                             DevicePanicNotifierFn *device_panic_notifier,
                              const VuDevIface *vu_iface,
                              Error **errp);
 
block/export/vhost-user-blk-server.c +1 -2
index ef07a87eb1..f2bfddbf26 100644
@@ -439,8 +439,7 @@ static void vhost_user_blk_server_start(VuBlockDev *vu_block_device,
     ctx = bdrv_get_aio_context(blk_bs(vu_block_device->backend));
 
     if (!vhost_user_server_start(&vu_block_device->vu_server, addr, ctx,
-                                 VHOST_USER_BLK_MAX_QUEUES,
-                                 NULL, &vu_block_iface,
+                                 VHOST_USER_BLK_MAX_QUEUES, &vu_block_iface,
                                  errp)) {
         goto error;
     }
util/vhost-user-server.c +0 -6
index ebb850731b..892815827d 100644
@@ -81,10 +81,6 @@ static void panic_cb(VuDev *vu_dev, const char *buf)
         close_client(server);
     }
 
-    if (server->device_panic_notifier) {
-        server->device_panic_notifier(server);
-    }
-
     /*
      * Set the callback function for network listener so another
      * vhost-user client can connect to this server
@@ -385,7 +381,6 @@ bool vhost_user_server_start(VuServer *server,
                              SocketAddress *socket_addr,
                              AioContext *ctx,
                              uint16_t max_queues,
-                             DevicePanicNotifierFn *device_panic_notifier,
                              const VuDevIface *vu_iface,
                              Error **errp)
 {
@@ -402,7 +397,6 @@ bool vhost_user_server_start(VuServer *server,
         .vu_iface              = vu_iface,
         .max_queues            = max_queues,
         .ctx                   = ctx,
-        .device_panic_notifier = device_panic_notifier,
     };
 
     qio_net_listener_set_name(server->listener, "vhost-user-backend-listener");
Signature · 1 line
2.26.2
#7 lore raw

PATCH 7/11 util/vhost-user-server: fix memory leak in vu_message_read()

In reply to Stefan Hajnoczi

fds[] is leaked when qio_channel_readv_full() fails.

Use vmsg->fds[] instead of keeping a local fds[] array. Then we can reuse goto fail to clean up fds. vmsg->fd_num must be zeroed before the loop to make this safe.

---
 util/vhost-user-server.c | 50 ++++++++++++++++++----------------------
 1 file changed, 23 insertions(+), 27 deletions(-)
util/vhost-user-server.c +23 -27
index 892815827d..5a60e2ca2a 100644
@@ -100,21 +100,11 @@ vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
     };
     int rc, read_bytes = 0;
     Error *local_err = NULL;
-    /*
-     * Store fds/nfds returned from qio_channel_readv_full into
-     * temporary variables.
-     *
-     * VhostUserMsg is a packed structure, gcc will complain about passing
-     * pointer to a packed structure member if we pass &VhostUserMsg.fd_num
-     * and &VhostUserMsg.fds directly when calling qio_channel_readv_full,
-     * thus two temporary variables nfds and fds are used here.
-     */
-    size_t nfds = 0, nfds_t = 0;
     const size_t max_fds = G_N_ELEMENTS(vmsg->fds);
-    int *fds_t = NULL;
     VuServer *server = container_of(vu_dev, VuServer, vu_dev);
     QIOChannel *ioc = server->ioc;
 
+    vmsg->fd_num = 0;
     if (!ioc) {
         error_report_err(local_err);
         goto fail;
@@ -122,41 +112,47 @@ vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
 
     assert(qemu_in_coroutine());
     do {
+        size_t nfds = 0;
+        int *fds = NULL;
+
         /*
          * qio_channel_readv_full may have short reads, keeping calling it
          * until getting VHOST_USER_HDR_SIZE or 0 bytes in total
          */
-        rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &local_err);
+        rc = qio_channel_readv_full(ioc, &iov, 1, &fds, &nfds, &local_err);
         if (rc < 0) {
             if (rc == QIO_CHANNEL_ERR_BLOCK) {
+                assert(local_err == NULL);
                 qio_channel_yield(ioc, G_IO_IN);
                 continue;
             } else {
                 error_report_err(local_err);
-                return false;
+                goto fail;
             }
         }
-        read_bytes += rc;
-        if (nfds_t > 0) {
-            if (nfds + nfds_t > max_fds) {
+
+        if (nfds > 0) {
+            if (vmsg->fd_num + nfds > max_fds) {
                 error_report("A maximum of %zu fds are allowed, "
                              "however got %lu fds now",
-                             max_fds, nfds + nfds_t);
+                             max_fds, vmsg->fd_num + nfds);
+                g_free(fds);
                 goto fail;
             }
-            memcpy(vmsg->fds + nfds, fds_t,
-                   nfds_t *sizeof(vmsg->fds[0]));
-            nfds += nfds_t;
-            g_free(fds_t);
+            memcpy(vmsg->fds + vmsg->fd_num, fds, nfds * sizeof(vmsg->fds[0]));
+            vmsg->fd_num += nfds;
+            g_free(fds);
         }
-        if (read_bytes == VHOST_USER_HDR_SIZE || rc == 0) {
-            break;
+
+        if (rc == 0) { /* socket closed */
+            goto fail;
         }
-        iov.iov_base = (char *)vmsg + read_bytes;
-        iov.iov_len = VHOST_USER_HDR_SIZE - read_bytes;
-    } while (true);
 
-    vmsg->fd_num = nfds;
+        iov.iov_base += rc;
+        iov.iov_len -= rc;
+        read_bytes += rc;
+    } while (read_bytes != VHOST_USER_HDR_SIZE);
+
     /* qio_channel_readv_full will make socket fds blocking, unblock them */
     vmsg_unblock_fds(vmsg);
     if (vmsg->size > sizeof(vmsg->payload)) {
Signature · 1 line
2.26.2
#8 lore raw

PATCH 8/11 util/vhost-user-server: check EOF when reading payload

In reply to Stefan Hajnoczi

Unexpected EOF is an error that must be reported.

---
 util/vhost-user-server.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
util/vhost-user-server.c +4 -2
index 5a60e2ca2a..ec555abcb2 100644
@@ -169,8 +169,10 @@ vu_message_read(VuDev *vu_dev, int conn_fd, VhostUserMsg *vmsg)
     };
     if (vmsg->size) {
         rc = qio_channel_readv_all_eof(ioc, &iov_payload, 1, &local_err);
-        if (rc == -1) {
-            error_report_err(local_err);
+        if (rc != 1) {
+            if (local_err) {
+                error_report_err(local_err);
+            }
             goto fail;
         }
     }
Signature · 1 line
2.26.2
#9 lore raw

PATCH 9/11 util/vhost-user-server: rework vu_client_trip() coroutine lifecycle

In reply to Stefan Hajnoczi

The vu_client_trip() coroutine is leaked during AioContext switching. It is also unsafe to destroy the vu_dev in panic_cb() since its callers still access it in some cases.

Rework the lifecycle to solve these safety issues.

---
 util/vhost-user-server.h             |  29 ++--
 block/export/vhost-user-blk-server.c |   9 +-
 util/vhost-user-server.c             | 245 +++++++++++++++------------
 3 files changed, 155 insertions(+), 128 deletions(-)
util/vhost-user-server.h +16 -13
index 92177fc911..0da4c2cc4c 100644
@@ -19,34 +19,36 @@
 #include "qapi/error.h"
 #include "standard-headers/linux/virtio_blk.h"
 
+/* A kick fd that we monitor on behalf of libvhost-user */
 typedef struct VuFdWatch {
     VuDev *vu_dev;
     int fd; /*kick fd*/
     void *pvt;
     vu_watch_cb cb;
-    bool processing;
     QTAILQ_ENTRY(VuFdWatch) next;
 } VuFdWatch;
 
-typedef struct VuServer VuServer;
-
-struct VuServer {
+/**
+ * VuServer:
+ * A vhost-user server instance with user-defined VuDevIface callbacks.
+ * Vhost-user device backends can be implemented using VuServer. VuDevIface
+ * callbacks and virtqueue kicks run in the given AioContext.
+ */
+typedef struct {
     QIONetListener *listener;
+    QEMUBH *restart_listener_bh;
     AioContext *ctx;
     int max_queues;
     const VuDevIface *vu_iface;
+
+    /* Protected by ctx lock */
     VuDev vu_dev;
     QIOChannel *ioc; /* The I/O channel with the client */
     QIOChannelSocket *sioc; /* The underlying data channel with the client */
-    /* IOChannel for fd provided via VHOST_USER_SET_SLAVE_REQ_FD */
-    QIOChannel *ioc_slave;
-    QIOChannelSocket *sioc_slave;
-    Coroutine *co_trip; /* coroutine for processing VhostUserMsg */
     QTAILQ_HEAD(, VuFdWatch) vu_fd_watches;
-    /* restart coroutine co_trip if AIOContext is changed */
-    bool aio_context_changed;
-    bool processing_msg;
-};
+
+    Coroutine *co_trip; /* coroutine for processing VhostUserMsg */
+} VuServer;
 
 bool vhost_user_server_start(VuServer *server,
                              SocketAddress *unix_socket,
@@ -57,6 +59,7 @@ bool vhost_user_server_start(VuServer *server,
 
 void vhost_user_server_stop(VuServer *server);
 
-void vhost_user_server_set_aio_context(VuServer *server, AioContext *ctx);
+void vhost_user_server_attach_aio_context(VuServer *server, AioContext *ctx);
+void vhost_user_server_detach_aio_context(VuServer *server);
 
 #endif /* VHOST_USER_SERVER_H */
block/export/vhost-user-blk-server.c +2 -7
index f2bfddbf26..b609a3e4d6 100644
@@ -313,18 +313,13 @@ static const VuDevIface vu_block_iface = {
 static void blk_aio_attached(AioContext *ctx, void *opaque)
 {
     VuBlockDev *vub_dev = opaque;
-    aio_context_acquire(ctx);
-    vhost_user_server_set_aio_context(&vub_dev->vu_server, ctx);
-    aio_context_release(ctx);
+    vhost_user_server_attach_aio_context(&vub_dev->vu_server, ctx);
 }
 
 static void blk_aio_detach(void *opaque)
 {
     VuBlockDev *vub_dev = opaque;
-    AioContext *ctx = vub_dev->vu_server.ctx;
-    aio_context_acquire(ctx);
-    vhost_user_server_set_aio_context(&vub_dev->vu_server, NULL);
-    aio_context_release(ctx);
+    vhost_user_server_detach_aio_context(&vub_dev->vu_server);
 }
 
 static void
util/vhost-user-server.c +137 -108

330 lines not on this page · raw message

Signature · 1 line
2.26.2
#10 lore raw

PATCH 10/11 block/export: report flush errors

In reply to Stefan Hajnoczi

Propagate the flush return value since errors are possible.

---
 block/export/vhost-user-blk-server.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
block/export/vhost-user-blk-server.c +7 -4
index b609a3e4d6..44d3c45fa2 100644
@@ -78,11 +78,11 @@ vu_block_discard_write_zeroes(VuBlockReq *req, struct iovec *iov,
     return -EINVAL;
 }
 
-static void coroutine_fn vu_block_flush(VuBlockReq *req)
+static int coroutine_fn vu_block_flush(VuBlockReq *req)
 {
     VuBlockDev *vdev_blk = get_vu_block_device_by_server(req->server);
     BlockBackend *backend = vdev_blk->backend;
-    blk_co_flush(backend);
+    return blk_co_flush(backend);
 }
 
 static void coroutine_fn vu_block_virtio_process_req(void *opaque)
@@ -152,8 +152,11 @@ static void coroutine_fn vu_block_virtio_process_req(void *opaque)
         break;
     }
     case VIRTIO_BLK_T_FLUSH:
-        vu_block_flush(req);
-        req->in->status = VIRTIO_BLK_S_OK;
+        if (vu_block_flush(req) == 0) {
+            req->in->status = VIRTIO_BLK_S_OK;
+        } else {
+            req->in->status = VIRTIO_BLK_S_IOERR;
+        }
         break;
     case VIRTIO_BLK_T_GET_ID: {
         size_t size = MIN(iov_size(&elem->in_sg[0], in_num),
Signature · 1 line
2.26.2
#11 lore raw

PATCH 11/11 block/export: convert vhost-user-blk server to block export API

In reply to Stefan Hajnoczi

Use the new QAPI block exports API instead of defining our own QOM objects.

This is a large change because the lifecycle of VuBlockDev needs to follow BlockExportDriver. QOM properties are replaced by QAPI options objects.

VuBlockDev is renamed VuBlkExport and contains a BlockExport field. Several fields can be dropped since BlockExport already has equivalents.

The file names and meson build integration will be adjusted in a future patch. libvhost-user should probably be built as a static library that is linked into QEMU instead of as a .c file that results in duplicate compilation.

The new command-line syntax is:

  $ qemu-storage-daemon \
      --blockdev file,node-name=drive0,filename=test.img \
      --export vhost-user-blk,node-name=drive0,id=export0,unix-socket=/tmp/vhost-user-blk.sock

Note that unix-socket is optional because we may wish to accept chardevs too in the future.

---
 qapi/block-export.json               |  19 +-
 block/export/vhost-user-blk-server.h |  23 +-
 block/export/export.c                |   8 +-
 block/export/vhost-user-blk-server.c | 461 ++++++++-------------------
 block/export/meson.build             |   1 +
 block/meson.build                    |   1 -
 6 files changed, 156 insertions(+), 357 deletions(-)
qapi/block-export.json +17 -2
index ace0d66e17..840dcbe833 100644
@@ -84,6 +84,19 @@
   'data': { '*name': 'str', '*description': 'str',
             '*bitmap': 'str' } }
 
+##
+# @BlockExportOptionsVhostUserBlk:
+#
+# A vhost-user-blk block export.
+#
+# @unix-socket: Path where the vhost-user UNIX domain socket will be created.
+# @logical-block-size: Logical block size in bytes.
+#
+# Since: 5.2
+##
+{ 'struct': 'BlockExportOptionsVhostUserBlk',
+  'data': { '*unix-socket': 'str', '*logical-block-size': 'size' } }
+
 ##
 # @NbdServerAddOptions:
 #
@@ -180,11 +193,12 @@
 # An enumeration of block export types
 #
 # @nbd: NBD export
+# @vhost-user-blk: vhost-user-blk export (since 5.2)
 #
 # Since: 4.2
 ##
 { 'enum': 'BlockExportType',
-  'data': [ 'nbd' ] }
+  'data': [ 'nbd', 'vhost-user-blk' ] }
 
 ##
 # @BlockExportOptions:
@@ -213,7 +227,8 @@
             '*writethrough': 'bool' },
   'discriminator': 'type',
   'data': {
-      'nbd': 'BlockExportOptionsNbd'
+      'nbd': 'BlockExportOptionsNbd',
+      'vhost-user-blk': 'BlockExportOptionsVhostUserBlk'
    } }
 
 ##
block/export/vhost-user-blk-server.h +3 -20
index f06f37c4c8..fcf46fc8a5 100644
@@ -10,27 +10,10 @@
 
 #ifndef VHOST_USER_BLK_SERVER_H
 #define VHOST_USER_BLK_SERVER_H
-#include "util/vhost-user-server.h"
 
-typedef struct VuBlockDev VuBlockDev;
-#define TYPE_VHOST_USER_BLK_SERVER "vhost-user-blk-server"
-#define VHOST_USER_BLK_SERVER(obj) \
-   OBJECT_CHECK(VuBlockDev, obj, TYPE_VHOST_USER_BLK_SERVER)
+#include "block/export.h"
 
-/* vhost user block device */
-struct VuBlockDev {
-    Object parent_obj;
-    char *node_name;
-    SocketAddress *addr;
-    AioContext *ctx;
-    VuServer vu_server;
-    bool running;
-    uint32_t blk_size;
-    BlockBackend *backend;
-    QIOChannelSocket *sioc;
-    QTAILQ_ENTRY(VuBlockDev) next;
-    struct virtio_blk_config blkcfg;
-    bool writable;
-};
+/* For block/export/export.c */
+extern const BlockExportDriver blk_exp_vhost_user_blk;
 
 #endif /* VHOST_USER_BLK_SERVER_H */
block/export/export.c +7 -1
index 87516d1e05..d0c7590911 100644
@@ -17,6 +17,9 @@
 #include "sysemu/block-backend.h"
 #include "block/export.h"
 #include "block/nbd.h"
+#if CONFIG_LINUX
+#include "block/export/vhost-user-blk-server.h"
+#endif
 #include "qapi/error.h"
 #include "qapi/qapi-commands-block-export.h"
 #include "qapi/qapi-events-block-export.h"
@@ -24,6 +27,9 @@
 
 static const BlockExportDriver *blk_exp_drivers[] = {
     &blk_exp_nbd,
+#if CONFIG_LINUX
+    &blk_exp_vhost_user_blk,
+#endif
 };
 
 /* Only accessed from the main thread */
@@ -123,7 +129,7 @@ BlockExport *blk_exp_add(BlockExportOptions *export, Error **errp)
     assert(drv->instance_size >= sizeof(BlockExport));
     exp = g_malloc0(drv->instance_size);
     *exp = (BlockExport) {
-        .drv        = &blk_exp_nbd,
+        .drv        = drv,
         .refcount   = 1,
         .user_owned = true,
         .id         = g_strdup(export->id),
block/export/vhost-user-blk-server.c +128 -333

650 lines not on this page · raw message

block/export/meson.build +1 -0
index 558ef35d38..ef3a9576f7 100644
@@ -1 +1,2 @@
 block_ss.add(files('export.c'))
+block_ss.add(when: 'CONFIG_LINUX', if_true: files('vhost-user-blk-server.c', '../../contrib/libvhost-user/libvhost-user.c'))
block/meson.build +0 -1
index cd52104c84..0b38dc36f7 100644
@@ -60,7 +60,6 @@ block_ss.add(when: 'CONFIG_WIN32', if_true: files('file-win32.c', 'win32-aio.c')
 block_ss.add(when: 'CONFIG_POSIX', if_true: [files('file-posix.c'), coref, iokit])
 block_ss.add(when: 'CONFIG_LIBISCSI', if_true: files('iscsi-opts.c'))
 block_ss.add(when: 'CONFIG_LINUX', if_true: files('nvme.c'))
-block_ss.add(when: 'CONFIG_LINUX', if_true: files('export/vhost-user-blk-server.c', '../contrib/libvhost-user/libvhost-user.c'))
 block_ss.add(when: 'CONFIG_REPLICATION', if_true: files('replication.c'))
 block_ss.add(when: 'CONFIG_SHEEPDOG', if_true: files('sheepdog.c'))
 block_ss.add(when: ['CONFIG_LINUX_AIO', libaio], if_true: files('linux-aio.c'))
Signature · 1 line
2.26.2
#12 lore raw

Re: [PATCH 11/11] block/export: convert vhost-user-blk server to block export API

Quoted 15 lines · Stefan Hajnoczi

Use the new QAPI block exports API instead of defining our own QOM objects.

This is a large change because the lifecycle of VuBlockDev needs to follow BlockExportDriver. QOM properties are replaced by QAPI options objects.

VuBlockDev is renamed VuBlkExport and contains a BlockExport field. Several fields can be dropped since BlockExport already has equivalents.

The file names and meson build integration will be adjusted in a future patch. libvhost-user should probably be built as a static library that is linked into QEMU instead of as a .c file that results in duplicate compilation.

Quoted 8 lines

The new command-line syntax is:

  $ qemu-storage-daemon \
      --blockdev file,node-name=drive0,filename=test.img \
      --export vhost-user-blk,node-name=drive0,id=export0,unix-socket=/tmp/vhost-user-blk.sock

Note that unix-socket is optional because we may wish to accept chardevs too in the future.

It's optional in the QAPI schema, but the code cosunming the --export appears to require it.

There is no need to make it optional now just in case: Changing an option parameter from mandatory to optional is backward-compatible.

Quoted 20 lines
---
 qapi/block-export.json               |  19 +-
 block/export/vhost-user-blk-server.h |  23 +-
 block/export/export.c                |   8 +-
 block/export/vhost-user-blk-server.c | 461 ++++++++-------------------
 block/export/meson.build             |   1 +
 block/meson.build                    |   1 -
 6 files changed, 156 insertions(+), 357 deletions(-)
qapi/block-export.json +2 -0
index ace0d66e17..840dcbe833 100644
@@ -84,6 +84,19 @@
   'data': { '*name': 'str', '*description': 'str',
             '*bitmap': 'str' } }
 
+##
+# @BlockExportOptionsVhostUserBlk:

qapi/block-export.json line 98

Quoted 10 lines
qapi/block-export.json +10 -0
+#
+# A vhost-user-blk block export.
+#
+# @unix-socket: Path where the vhost-user UNIX domain socket will be created.
+# @logical-block-size: Logical block size in bytes.
+#
+# Since: 5.2
+##
+{ 'struct': 'BlockExportOptionsVhostUserBlk',
+  'data': { '*unix-socket': 'str', '*logical-block-size': 'size' } }

This is where we make @unix-socket optional.

Default behavior is not documented.

Quoted 28 lines
qapi/block-export.json +5 -2
+
 ##
 # @NbdServerAddOptions:
 #
@@ -180,11 +193,12 @@
 # An enumeration of block export types
 #
 # @nbd: NBD export
+# @vhost-user-blk: vhost-user-blk export (since 5.2)
 #
 # Since: 4.2
 ##
 { 'enum': 'BlockExportType',
-  'data': [ 'nbd' ] }
+  'data': [ 'nbd', 'vhost-user-blk' ] }
 
 ##
 # @BlockExportOptions:
@@ -213,7 +227,8 @@
             '*writethrough': 'bool' },
   'discriminator': 'type',
   'data': {
-      'nbd': 'BlockExportOptionsNbd'
+      'nbd': 'BlockExportOptionsNbd',
+      'vhost-user-blk': 'BlockExportOptionsVhostUserBlk'
    } }
 
 ##

[...]

Quoted 4 lines
block/export/vhost-user-blk-server.c +0 -0
index 44d3c45fa2..9908b3287e 100644

[...]

Quoted 47 lines
block/export/vhost-user-blk-server.c +10 -36
-static char *vu_get_unix_socket(Object *obj, Error **errp)
+static int vu_blk_exp_create(BlockExport *exp, BlockExportOptions *opts,
+                             Error **errp)
 {
-    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
-    return g_strdup(vus->addr->u.q_unix.path);
-}
-
-static bool vu_get_block_writable(Object *obj, Error **errp)
-{
-    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
-    return vus->writable;
-}
-
-static void vu_set_block_writable(Object *obj, bool value, Error **errp)
-{
-    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
-
-    if (!vu_prop_modifiable(vus, errp)) {
-            return;
-    }
-
-    vus->writable = value;
-}
-
-static void vu_get_blk_size(Object *obj, Visitor *v, const char *name,
-                            void *opaque, Error **errp)
-{
-    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
-    uint32_t value = vus->blk_size;
-
-    visit_type_uint32(v, name, &value, errp);
-}
-
-static void vu_set_blk_size(Object *obj, Visitor *v, const char *name,
-                            void *opaque, Error **errp)
-{
-    VuBlockDev *vus = VHOST_USER_BLK_SERVER(obj);
-
+    VuBlkExport *vexp = container_of(exp, VuBlkExport, export);
+    BlockExportOptionsVhostUserBlk *vu_opts = &opts->u.vhost_user_blk;
+    SocketAddress addr = {
+        .type = SOCKET_ADDRESS_TYPE_UNIX,
+        .u.q_unix.path = vu_opts->has_unix_socket ?
+                         vu_opts->unix_socket :
+                         NULL,
+    };

block/export/vhost-user-blk-server.c vu_block_initialize_config(BlockDriverState *bs, · line 377

Quoted 10 lines
block/export/vhost-user-blk-server.c +4 -3
     Error *local_err = NULL;
-    uint32_t value;
+    uint64_t logical_block_size;
 
-    if (!vu_prop_modifiable(vus, errp)) {
-            return;
+    if (!vu_opts->has_unix_socket) {
+        error_setg(errp, "Missing unix-socket path to listen on");
+        return -EINVAL;
     }

This is where we require @unix-socket.

Quoted 66 lines
block/export/vhost-user-blk-server.c +28 -32
-    visit_type_uint32(v, name, &value, &local_err);
-    if (local_err) {
-        goto out;
-    }
+    vexp->writable = opts->writable;
+    vexp->blkcfg.wce = 0;
 
-    check_block_size(object_get_typename(obj), name, value, &local_err);
+    if (vu_opts->has_logical_block_size) {
+        logical_block_size = vu_opts->logical_block_size;
+    } else {
+        logical_block_size = BDRV_SECTOR_SIZE;
+    }
+    check_block_size(exp->id, "logical-block-size", logical_block_size,
+                     &local_err);
     if (local_err) {
-        goto out;
+        error_propagate(errp, local_err);
+        return -EINVAL;
+    }
+    vexp->blk_size = logical_block_size;
+    blk_set_guest_block_size(exp->blk, logical_block_size);
+    vu_blk_initialize_config(blk_bs(exp->blk), &vexp->blkcfg,
+                               logical_block_size);
+
+    blk_set_allow_aio_context_change(exp->blk, true);
+    blk_add_aio_context_notifier(exp->blk, blk_aio_attached, blk_aio_detach,
+                                 vexp);
+
+    if (!vhost_user_server_start(&vexp->vu_server, &addr, exp->ctx,
+                                 VHOST_USER_BLK_MAX_QUEUES, &vu_blk_iface,
+                                 errp)) {
+        blk_remove_aio_context_notifier(exp->blk, blk_aio_attached,
+                                        blk_aio_detach, vexp);
+        return -EADDRNOTAVAIL;
     }
 
-    vus->blk_size = value;
-
-out:
-    error_propagate(errp, local_err);
-}
-
-static void vhost_user_blk_server_instance_finalize(Object *obj)
-{
-    VuBlockDev *vub = VHOST_USER_BLK_SERVER(obj);
-
-    vhost_user_blk_server_stop(vub);
-
-    /*
-     * Unlike object_property_add_str, object_class_property_add_str
-     * doesn't have a release method. Thus manual memory freeing is
-     * needed.
-     */
-    free_socket_addr(vub->addr);
-    g_free(vub->node_name);
-}
-
-static void vhost_user_blk_server_complete(UserCreatable *obj, Error **errp)
-{
-    VuBlockDev *vub = VHOST_USER_BLK_SERVER(obj);
-
-    vhost_user_blk_server_start(vub, errp);
+    return 0;
 }

[...]

#13 lore raw

Re: [PATCH 11/11] block/export: convert vhost-user-blk server to block export API

Quoted 23 lines · Markus Armbruster
Quoted 21 lines · Stefan Hajnoczi

Use the new QAPI block exports API instead of defining our own QOM objects.

This is a large change because the lifecycle of VuBlockDev needs to follow BlockExportDriver. QOM properties are replaced by QAPI options objects.

VuBlockDev is renamed VuBlkExport and contains a BlockExport field. Several fields can be dropped since BlockExport already has equivalents.

The file names and meson build integration will be adjusted in a future patch. libvhost-user should probably be built as a static library that is linked into QEMU instead of as a .c file that results in duplicate compilation.

The new command-line syntax is:

  $ qemu-storage-daemon \
      --blockdev file,node-name=drive0,filename=test.img \
      --export vhost-user-blk,node-name=drive0,id=export0,unix-socket=/tmp/vhost-user-blk.sock
Quoted 8 lines
Quoted 2 lines

Note that unix-socket is optional because we may wish to accept chardevs too in the future.

It's optional in the QAPI schema, but the code cosunming the --export appears to require it.

There is no need to make it optional now just in case: Changing an option parameter from mandatory to optional is backward-compatible.

Good point, it should be mandatory.

Stefan