vhost-user block device backend implementation
v4 cover vhost-user block device backend implementation
v4:
* add object properties in class_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.
v3:
* separate generic vhost-user-server code from vhost-user-blk-server code
* re-write vu_message_read and kick hander function as coroutines to directly call blk_co_preadv, blk_co_pwritev, etc.
* add aio_context notifier functions to support multi-threading model
* other fixes regarding coding style, warning report, etc.
v2:
* Only enable this feauture for Linux because eventfd is a Linux-specific feature
This patch series is an implementation of vhost-user block device backend server, thanks to Stefan and Kevin's guidance.
Vhost-user block device backend server is a UserCreatable object and can be started using object_add,
(qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512 (qemu) object_del ID
or appending the "-object" option when starting QEMU,
$ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,blk-size=512
Then vhost-user client can connect to the server backend. For example, QEMU could act as a client,
$ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1
And guest OS could access this vhost-user block device after mouting it.
Coiby Xu (5):
extend libvhost to support IOThread and coroutine
generic vhost user server
vhost-user block device backend server
a standone-alone tool to directly share disk image file via vhost-user
protocol
new qTest case to test the vhost-user-blk-server
Makefile | 4 + Makefile.target | 1 + backends/Makefile.objs | 2 + backends/vhost-user-blk-server.c | 718 ++++++++++++++++++++++++++ backends/vhost-user-blk-server.h | 21 + configure | 3 + contrib/libvhost-user/libvhost-user.c | 54 +- contrib/libvhost-user/libvhost-user.h | 38 +- qemu-vu.c | 252 +++++++++ tests/Makefile.include | 3 +- tests/qtest/Makefile.include | 2 + tests/qtest/libqos/vhost-user-blk.c | 126 +++++ tests/qtest/libqos/vhost-user-blk.h | 44 ++ tests/qtest/vhost-user-blk-test.c | 694 +++++++++++++++++++++++++ util/Makefile.objs | 3 + util/vhost-user-server.c | 427 +++++++++++++++ util/vhost-user-server.h | 56 ++ vl.c | 4 + 18 files changed, 2444 insertions(+), 8 deletions(-) create mode 100644 backends/vhost-user-blk-server.c create mode 100644 backends/vhost-user-blk-server.h create mode 100644 qemu-vu.c create mode 100644 tests/qtest/libqos/vhost-user-blk.c create mode 100644 tests/qtest/libqos/vhost-user-blk.h create mode 100644 tests/qtest/vhost-user-blk-test.c create mode 100644 util/vhost-user-server.c create mode 100644 util/vhost-user-server.h
--
2.25.0
PATCH v4 1/5 extend libvhost to support IOThread and coroutine
Previously libvhost dispatch events in its own GMainContext. Now vhost-user client's kick event can be dispatched in block device drive's AioContext thus IOThread is supported. And also allow vu_message_read and vu_kick_cb to be replaced so QEMU can run them as coroutines.
--- contrib/libvhost-user/libvhost-user.c | 54 ++++++++++++++++++++++++--- contrib/libvhost-user/libvhost-user.h | 38 ++++++++++++++++++- 2 files changed, 85 insertions(+), 7 deletions(-)
contrib/libvhost-user/libvhost-user.c +49 -5
@@ -67,8 +67,6 @@/* The version of inflight buffer */ #define INFLIGHT_VERSION 1 -#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) -/* The version of the protocol we support */ #define VHOST_USER_VERSION 1#define LIBVHOST_USER_DEBUG 0
@@ -260,7 +258,7 @@ have_userfault(void) }static bool -vu_message_read (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) +vu_message_read_ (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) {char control[CMSG_SPACE (VHOST_MEMORY_MAX_NREGIONS *sizeof (int ))] = { };struct iovec iov = {
@@ -328,6 +326,17 @@ fail:return false; } +static bool vu_message_read (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) +{ +vu_read_msg_cb read_msg; +if (dev->co_iface) { + read_msg = dev->co_iface->read_msg; + }else { + read_msg = vu_message_read_; + } +return read_msg (dev, conn_fd, vmsg); +} +static bool vu_message_write (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) {
@@ -1075,9 +1084,14 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) }if (dev->vq[index].kick_fd !=-1 && dev->vq[index].handler) { +if (dev->set_watch_packed_data) { + dev->set_watch_packed_data (dev, dev->vq[index].kick_fd,VU_WATCH_IN , + dev->co_iface->kick_callback, + (void *)(long )index); + }else { dev->set_watch (dev, dev->vq[index].kick_fd,VU_WATCH_IN , vu_kick_cb, (void *)(long )index); - + }DPRINT ("Waiting for kicks on fd: %d for vq: %d\n" , dev->vq[index].kick_fd, index); }
@@ -1097,8 +1111,14 @@ void vu_set_queue_handler(VuDev *dev, VuVirtq *vq, vq->handler = handler;if (vq->kick_fd >=0 ) {if (handler) { +if (dev->set_watch_packed_data) { + dev->set_watch_packed_data (dev, vq->kick_fd,VU_WATCH_IN , + dev->co_iface->kick_callback, + (void *)(long )qidx); + }else { dev->set_watch (dev, vq->kick_fd,VU_WATCH_IN , vu_kick_cb, (void *)(long )qidx); + } }else { dev->remove_watch (dev, vq->kick_fd); }
@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev) }if (vq->kick_fd !=-1 ) { +/* remove watch for kick_fd +* When client process is running in gdb and +* quit command is run in gdb, QEMU will still dispatch the event +* which will cause segment fault in the callback function +*/ + dev->remove_watch (dev, vq->kick_fd);close (vq->kick_fd); vq->kick_fd =-1 ; }
@@ -1682,7 +1708,7 @@ vu_init(VuDev *dev,assert (max_queues >0 );assert (socket >=0 ); -assert (set_watch); +/* assert(set_watch); */ assert (remove_watch);assert (iface);assert (panic);
@@ -1715,6 +1741,24 @@ vu_init(VuDev *dev,return true; } +bool +vu_init_packed_data (VuDev *dev, +uint16_t max_queues, +int socket, +vu_panic_cb panic, +vu_set_watch_cb_packed_data set_watch_packed_data, +vu_remove_watch_cb remove_watch, +const VuDevIface *iface, +const CoIface *co_iface) +{ +if (vu_init (dev, max_queues, socket, panic,NULL , remove_watch, iface)) { + dev->set_watch_packed_data = set_watch_packed_data; + dev->co_iface = co_iface; +return true; + } +return false; +} +VuVirtq *vu_get_queue (VuDev *dev,int qidx) {
contrib/libvhost-user/libvhost-user.h +36 -2
@@ -30,6 +30,8 @@#define VHOST_MEMORY_MAX_NREGIONS 8 +#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) + typedef enum VhostSetConfigType { VHOST_SET_CONFIG_TYPE_MASTER = 0, VHOST_SET_CONFIG_TYPE_MIGRATION = 1,
@@ -201,6 +203,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);typedef void (*vu_set_features_cb ) (VuDev *dev,uint64_t features);typedef int (*vu_process_msg_cb ) (VuDev *dev,VhostUserMsg *vmsg,int *do_reply); +typedef bool (*vu_read_msg_cb ) (VuDev *dev,int sock,VhostUserMsg *vmsg);typedef void (*vu_queue_set_started_cb ) (VuDev *dev,int qidx,bool started);typedef bool (*vu_queue_is_processed_in_order_cb ) (VuDev *dev,int qidx);typedef int (*vu_get_config_cb ) (VuDev *dev,uint8_t *config,uint32_t len);
@@ -208,6 +211,20 @@ typedef int (*vu_set_config_cb) (VuDev *dev, const uint8_t *data, uint32_t offset, uint32_t size, uint32_t flags); +typedef void (*vu_watch_cb_packed_data ) (void *packed_data); + +typedef void (*vu_set_watch_cb_packed_data ) (VuDev *dev,int fd,int condition, +vu_watch_cb_packed_data cb, +void *data); +/* +* allowing vu_read_msg_cb and kick_callback to be replaced so QEMU +* can run them as coroutines +*/ +typedef struct CoIface { +vu_read_msg_cb read_msg; +vu_watch_cb_packed_data kick_callback; +}CoIface ; + typedef struct VuDevIface { /* called by VHOST_USER_GET_FEATURES to get the features bitmask */ vu_get_features_cb get_features;
@@ -372,7 +389,8 @@ struct VuDev {/* @set_watch: add or update the given fd to the watch set, * call cb when condition is met */ vu_set_watch_cb set_watch; - +/* AIO dispatch will only one data pointer to callback function */ +vu_set_watch_cb_packed_data set_watch_packed_data;/* @remove_watch: remove the given fd from the watch set */ vu_remove_watch_cb remove_watch;
@@ -380,7 +398,7 @@ struct VuDev {* re-initialize */ vu_panic_cb panic;const VuDevIface *iface; - +const CoIface *co_iface;/* Postcopy data */ int postcopy_ufd;bool postcopy_listening;
@@ -417,6 +435,22 @@ bool vu_init(VuDev *dev, const VuDevIface *iface); +/** +* vu_init_packed_data: +* Same as vu_init except for set_watch_packed_data which will pack +* two parameters into a struct thus QEMU aio_dispatch can pass the +* required data to callback function. +* +* Returns: true on success, false on failure. +**/ +bool vu_init_packed_data (VuDev *dev, +uint16_t max_queues, +int socket, +vu_panic_cb panic, +vu_set_watch_cb_packed_data set_watch_packed_data, +vu_remove_watch_cb remove_watch, +const VuDevIface *iface, +const CoIface *co_iface);/** * vu_deinit: * @dev: a VuDev context
Signature · 1 line
2.25.0
Re: [PATCH v4 1/5] extend libvhost to support IOThread and coroutine
contrib/libvhost-user/libvhost-user.c fail: · line 338
Quoted 10 lines · Coiby Xu
contrib/libvhost-user/libvhost-user.c +10
-0+static bool vu_message_read (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) +{ +vu_read_msg_cb read_msg; +if (dev->co_iface) { + read_msg = dev->co_iface->read_msg; + }else { + read_msg = vu_message_read_; + } +return read_msg (dev, conn_fd, vmsg); +}
libvhost-user is already partially asynchronous (->set/remove_watch() are used for eventfds).
Can you make the vhost-user socket I/O asynchronous too?
Quoted 5 lines
contrib/libvhost-user/libvhost-user.c +1
-0+static bool vu_message_write (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) {
contrib/libvhost-user/libvhost-user.c vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) · line 1093
Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +5
-0@@ -1075,9 +1084,14 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) }if (dev->vq[index].kick_fd !=-1 && dev->vq[index].handler) { +if (dev->set_watch_packed_data) { + dev->set_watch_packed_data (dev, dev->vq[index].kick_fd,VU_WATCH_IN , + dev->co_iface->kick_callback, + (void *)(long )index); + }else { dev->set_watch (dev, dev->vq[index].kick_fd,VU_WATCH_IN , vu_kick_cb, (void *)(long )index);
Why is it necessary to replace vu_kick_cb()?
The user can set a custom vq->handler() function with vu_set_queue_handler().
Coroutines aren't needed for this part since eventfd_read() always returns right away.
contrib/libvhost-user/libvhost-user.c vu_deinit(VuDev *dev) · line 1654
Quoted 9 lines
contrib/libvhost-user/libvhost-user.c +5
-0@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev) }if (vq->kick_fd !=-1 ) { +/* remove watch for kick_fd +* When client process is running in gdb and +* quit command is run in gdb, QEMU will still dispatch the event +* which will cause segment fault in the callback function +*/
Code and comments in libvhost-user should not refer to QEMU specifics. Removing the watch is a good idea regardless of the application or event loop implementation. No comment is needed here.
contrib/libvhost-user/libvhost-user.c vu_init(VuDev *dev, · line 1711
Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +2
-1+ dev->remove_watch (dev, vq->kick_fd);close (vq->kick_fd); vq->kick_fd =-1 ; }@@ -1682,7 +1708,7 @@ vu_init(VuDev *dev,assert (max_queues >0 );assert (socket >=0 ); -assert (set_watch); +/* assert(set_watch); */
?
contrib/libvhost-user/libvhost-user.h struct VuDev { · line 392
Quoted 6 lines
contrib/libvhost-user/libvhost-user.h +1
-1@@ -372,7 +389,8 @@ struct VuDev {/* @set_watch: add or update the given fd to the watch set, * call cb when condition is met */ vu_set_watch_cb set_watch; - +/* AIO dispatch will only one data pointer to callback function */
I don't understand what this comment is saying.
Re: [PATCH v4 1/5] extend libvhost to support IOThread and coroutine
In reply to Coiby XuAm 18.02.2020 um 06:07 hat Coiby Xu geschrieben:
Quoted 39 lines
Previously libvhost dispatch events in its own GMainContext. Now vhost-user client's kick event can be dispatched in block device drive's AioContext thus IOThread is supported. And also allow
vu_message_readandvu_kick_cbto be replaced so QEMU can run them as coroutines.Signed-off-by Coiby Xu <[email protected]>--- contrib/libvhost-user/libvhost-user.c | 54 ++++++++++++++++++++++++--- contrib/libvhost-user/libvhost-user.h | 38 ++++++++++++++++++- 2 files changed, 85 insertions(+), 7 deletions(-)contrib/libvhost-user/libvhost-user.h +6
-0@@ -30,6 +30,8 @@#define VHOST_MEMORY_MAX_NREGIONS 8 +#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) + typedef enum VhostSetConfigType { VHOST_SET_CONFIG_TYPE_MASTER = 0, VHOST_SET_CONFIG_TYPE_MIGRATION = 1,@@ -201,6 +203,7 @@ typedef uint64_t (*vu_get_features_cb) (VuDev *dev);typedef void (*vu_set_features_cb ) (VuDev *dev,uint64_t features);typedef int (*vu_process_msg_cb ) (VuDev *dev,VhostUserMsg *vmsg,int *do_reply); +typedef bool (*vu_read_msg_cb ) (VuDev *dev,int sock,VhostUserMsg *vmsg);typedef void (*vu_queue_set_started_cb ) (VuDev *dev,int qidx,bool started);typedef bool (*vu_queue_is_processed_in_order_cb ) (VuDev *dev,int qidx);typedef int (*vu_get_config_cb ) (VuDev *dev,uint8_t *config,uint32_t len);@@ -208,6 +211,20 @@ typedef int (*vu_set_config_cb) (VuDev *dev, const uint8_t *data, uint32_t offset, uint32_t size, uint32_t flags); +typedef void (*vu_watch_cb_packed_data ) (void *packed_data); + +typedef void (*vu_set_watch_cb_packed_data) (VuDev *dev, int fd, int condition,
contrib/libvhost-user/libvhost-user.h typedef int (*vu_set_config_cb) (VuDev *dev, const uint8_t *data, · line 226
Quoted 10 lines
contrib/libvhost-user/libvhost-user.h +10
-0+vu_watch_cb_packed_data cb, + void *data); +/* +* allowing vu_read_msg_cb and kick_callback to be replaced so QEMU +* can run them as coroutines +*/ +typedef struct CoIface { +vu_read_msg_cb read_msg; +vu_watch_cb_packed_data kick_callback; +}CoIface ;
I think this should be part of VuDevIface, so that it becomes a properly integrated part of the design instead of an adapter hacked on top.
Quoted 23 lines
contrib/libvhost-user/libvhost-user.h +3
-2typedef struct VuDevIface { /* called by VHOST_USER_GET_FEATURES to get the features bitmask */ vu_get_features_cb get_features;@@ -372,7 +389,8 @@ struct VuDev {/* @set_watch: add or update the given fd to the watch set, * call cb when condition is met */ vu_set_watch_cb set_watch; - +/* AIO dispatch will only one data pointer to callback function */ +vu_set_watch_cb_packed_data set_watch_packed_data;/* @remove_watch: remove the given fd from the watch set */ vu_remove_watch_cb remove_watch;@@ -380,7 +398,7 @@ struct VuDev {* re-initialize */ vu_panic_cb panic;const VuDevIface *iface; - +const CoIface *co_iface;/* Postcopy data */ int postcopy_ufd;bool postcopy_listening;
contrib/libvhost-user/libvhost-user.h bool vu_init(VuDev *dev, · line 441
Quoted 7 lines
contrib/libvhost-user/libvhost-user.h +4
-0@@ -417,6 +435,22 @@ bool vu_init(VuDev *dev, const VuDevIface *iface); +/** +* vu_init_packed_data: +* Same as vu_init except for set_watch_packed_data which will pack +* two parameters into a struct
Be specific: Which two parameters and which struct?
I think it would be more helpful to name the function after the additional piece of information that it uses rather than the fact that it stores it internally in a struct.
We have:
typedef void (*vu_set_watch_cb) (VuDev *dev, int fd, int condition,
vu_watch_cb cb, void *data);
typedef void (*vu_set_watch_cb_packed_data) (VuDev *dev, int fd, int condition,
vu_watch_cb_packed_data cb,
void *data);Without looking at the implementation, they have the same set of parameters. I suspect that the difference is in the content of *data, but since it is declared void*, I suppose it's treated as an opaque data type and will only be passed unchanged (and uninspected) to cb.
If so, there is no differene between both types.
Quoted 33 lines
thus QEMU
aio_dispatchcan pass the
+ * required data to callback function.
+ *
+ * Returns: true on success, false on failure.
+ **/contrib/libvhost-user/libvhost-user.h +8
-0+bool vu_init_packed_data (VuDev *dev, +uint16_t max_queues, +int socket, +vu_panic_cb panic, +vu_set_watch_cb_packed_data set_watch_packed_data, +vu_remove_watch_cb remove_watch, +const VuDevIface *iface, +const CoIface *co_iface);/** * vu_deinit: * @dev: a VuDev context Signature · 1 line
2.25.0contrib/libvhost-user/libvhost-user.c +0
-2@@ -67,8 +67,6 @@/* The version of inflight buffer */ #define INFLIGHT_VERSION 1 -#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) -/* The version of the protocol we support */ #define VHOST_USER_VERSION 1#define LIBVHOST_USER_DEBUG 0
contrib/libvhost-user/libvhost-user.c have_userfault(void) · line 261
Quoted 5 lines
contrib/libvhost-user/libvhost-user.c +1
-1@@ -260,7 +258,7 @@ have_userfault(void) }static bool -vu_message_read (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) +vu_message_read_ (VuDev *dev,int conn_fd,VhostUserMsg *vmsg)
Just adding a trailing underscore isn't a good name. It doesn't tell the reader what the difference between vu_message_read_ and vu_message_read is.
Quoted 7 lines
contrib/libvhost-user/libvhost-user.c +0
-0{char control[CMSG_SPACE (VHOST_MEMORY_MAX_NREGIONS *sizeof (int ))] = { };struct iovec iov = {@@ -328,6 +326,17 @@ fail:return false; }
contrib/libvhost-user/libvhost-user.c fail: · line 338
Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +10
-0+static bool vu_message_read (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) +{ +vu_read_msg_cb read_msg; +if (dev->co_iface) { + read_msg = dev->co_iface->read_msg; + }else { + read_msg = vu_message_read_; + } +return read_msg (dev, conn_fd, vmsg); +}
If you change VuDevIface so that it contains the fields of CoIface directly, you can just initialise dev->iface->read_msg with what is called vu_message_read_() now for the non-QEMU case, and this whole wrapper becomes unnecessary because the code path is the same for both cases.
Quoted 6 lines
contrib/libvhost-user/libvhost-user.c +0
-0static bool vu_message_write (VuDev *dev,int conn_fd,VhostUserMsg *vmsg) {@@ -1075,9 +1084,14 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) }
contrib/libvhost-user/libvhost-user.c vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) · line 1094
Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +6
-1if (dev->vq[index].kick_fd !=-1 && dev->vq[index].handler) { +if (dev->set_watch_packed_data) { + dev->set_watch_packed_data (dev, dev->vq[index].kick_fd,VU_WATCH_IN , + dev->co_iface->kick_callback, + (void *)(long )index); + }else { dev->set_watch (dev, dev->vq[index].kick_fd,VU_WATCH_IN , vu_kick_cb, (void *)(long )index); - + }
Indentation is off here.
Also, this is almost exactly the same code for both cases. If you generalise things to have a dev->iface->kick_callback that can be initialised with vu_kick_cb in the non-QEMU case, you get rid of this duplication, too.
Quoted 5 lines
contrib/libvhost-user/libvhost-user.c +0
-0DPRINT ("Waiting for kicks on fd: %d for vq: %d\n" , dev->vq[index].kick_fd, index); }@@ -1097,8 +1111,14 @@ void vu_set_queue_handler(VuDev *dev, VuVirtq *vq, vq->handler = handler;
contrib/libvhost-user/libvhost-user.c void vu_set_queue_handler(VuDev *dev, VuVirtq *vq, · line 1121
Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +6
-0if (vq->kick_fd >=0 ) {if (handler) { +if (dev->set_watch_packed_data) { + dev->set_watch_packed_data (dev, vq->kick_fd,VU_WATCH_IN , + dev->co_iface->kick_callback, + (void *)(long )qidx); + }else { dev->set_watch (dev, vq->kick_fd,VU_WATCH_IN , vu_kick_cb, (void *)(long )qidx); + }
Same as above. (Indentation and duplicated code.)
contrib/libvhost-user/libvhost-user.c vu_deinit(VuDev *dev) · line 1654
Quoted 12 lines
contrib/libvhost-user/libvhost-user.c +5
-0}else { dev->remove_watch (dev, vq->kick_fd); }@@ -1627,6 +1647,12 @@ vu_deinit(VuDev *dev) }if (vq->kick_fd !=-1 ) { +/* remove watch for kick_fd +* When client process is running in gdb and +* quit command is run in gdb, QEMU will still dispatch the event +* which will cause segment fault in the callback function +*/
Reformat this comment to use a consistent line width, maybe like this:
/*
* remove watch for kick_fd.
*
* When client process is running in gdb and quit command is
* run in gdb, QEMU will still dispatch the event which will
* cause segment fault in the callback function */
I'm not sure what the comment wants to tell me: Is this an existing problem in the code that we can run into segfaults, or do we remove the watch to avoid segfaults?
contrib/libvhost-user/libvhost-user.c vu_init(VuDev *dev, · line 1711
Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +2
-1+ dev->remove_watch (dev, vq->kick_fd);close (vq->kick_fd); vq->kick_fd =-1 ; }@@ -1682,7 +1708,7 @@ vu_init(VuDev *dev,assert (max_queues >0 );assert (socket >=0 ); -assert (set_watch); +/* assert(set_watch); */
Don't leave commented code around. Either leave it in, or remove it completely.
I think this one should be left in. If you integrate CoIface into VuDevIface, the assertion will hold true again.
Quoted 14 lines
contrib/libvhost-user/libvhost-user.c +7
-0assert (remove_watch);assert (iface);assert (panic);@@ -1715,6 +1741,24 @@ vu_init(VuDev *dev,return true; } +bool +vu_init_packed_data(VuDev *dev, + uint16_t max_queues, + int socket, + vu_panic_cb panic, + vu_set_watch_cb_packed_data set_watch_packed_data, + vu_remove_watch_cb remove_watch,
contrib/libvhost-user/libvhost-user.c vu_init(VuDev *dev, · line 1760
Quoted 10 lines
contrib/libvhost-user/libvhost-user.c +10
-0+const VuDevIface *iface, + const CoIface *co_iface) +{ + if (vu_init(dev, max_queues, socket, panic, NULL, remove_watch, iface)) { + dev->set_watch_packed_data = set_watch_packed_data; + dev->co_iface = co_iface; +return true; + } +return false; +}
With the integrated VuDevIface, this wrapper becomes unnecessary.
Kevin
PATCH v4 2/5 generic vhost user server
In reply to Coiby XuSharing QEMU devices via vhost-user protocol
--- util/Makefile.objs | 3 + util/vhost-user-server.c | 427 +++++++++++++++++++++++++++++++++++++++ util/vhost-user-server.h | 56 +++++ 3 files changed, 486 insertions(+) create mode 100644 util/vhost-user-server.c create mode 100644 util/vhost-user-server.h
util/Makefile.objs +3 -0
@@ -36,6 +36,9 @@ util-obj-y += readline.o util-obj-y += rcu.o util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o +ifdef CONFIG_LINUX +util-obj-y += vhost-user-server.o +endif util-obj-y += qemu-coroutine-sleep.o util-obj-y += qemu-co-shared-resource.o util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).o
util/vhost-user-server.c new file +427 -0
427 lines not on this page · raw message
util/vhost-user-server.h new file +56 -0
56 lines not on this page · raw message
Signature · 1 line
2.25.0
Re: [PATCH v4 2/5] generic vhost user server
Am 18.02.2020 um 06:07 hat Coiby Xu geschrieben:
Quoted 17 lines
Sharing QEMU devices via vhost-user protocol
Signed-off-by Coiby Xu <[email protected]>--- util/Makefile.objs | 3 + util/vhost-user-server.c | 427 +++++++++++++++++++++++++++++++++++++++ util/vhost-user-server.h | 56 +++++ 3 files changed, 486 insertions(+) create mode 100644 util/vhost-user-server.c create mode 100644 util/vhost-user-server.hutil/vhost-user-server.h new file +0
-0
util/vhost-user-server.h line 8
Quoted 8 lines
util/vhost-user-server.h new file +8
-0@@ -0,0 +1,56 @@ +#include "io/channel-socket.h" +#include "io/channel-file.h" +#include "io/net-listener.h" +#include "contrib/libvhost-user/libvhost-user.h" +#include "standard-headers/linux/virtio_blk.h" +#include "qemu/error-report.h" + +typedef struct VuClient VuClient ;
I find the terminology a bit confusing here: VuClient is really the connection to a single client, but it's part of the server. The name gives the impression as if this were client-side code. (This is something that already tends to confuse me in the NBD code.)
I'm not sure what a better name could be, though. Maybe VuServerConnevtion or VuExportClient or VuExportConnection?
Quoted 7 lines
util/vhost-user-server.h new file +7
-0+typedef struct VuServer { + QIONetListener *listener; + AioContext *ctx; + QTAILQ_HEAD(, VuClient) clients; + void (*device_panic_notifier)(struct VuClient *client) ; + int max_queues; + const VuDevIface *vu_iface;
util/vhost-user-server.h line 26
Quoted 10 lines
util/vhost-user-server.h new file +10
-0+/* +* @ptr_in_device: VuServer pointer memory location in vhost-user device +* struct, so later container_of can be used to get device destruct +*/ +void *ptr_in_device; +bool close; +} VuServer; + +typedef struct kick_info { + VuDev *vu_dev;
I suppose this could specifically be VuClient?
Quoted 4 lines
util/vhost-user-server.h new file +4
-0+int fd;/*kick fd*/ +long index;/*queue index*/ +QIOChannel *ioc;/*I/O channel for kick fd*/ +QIOChannelFile *fioc;/*underlying data channel for kick fd*/
util/vhost-user-server.h line 40
Quoted 10 lines
util/vhost-user-server.h new file +10
-0+Coroutine *co; +} kick_info; + +struct VuClient { +VuDev parent; +VuServer *server; +QIOChannel *ioc;/* The current I/O channel */ +QIOChannelSocket *sioc;/* The underlying data channel */ +Coroutine *co_trip; +struct kick_info *kick_info;
If each struct kick_info (btw, QEMU coding style requires CamelCase) has exactly one VuClient and each VuClient has exactly on kick_info, should this be a single struct containing both?
[ Coming back from reading the code below - it's because this is in fact an array. This should be made clear in the definition. ]
Quoted 6 lines
util/vhost-user-server.h new file +6
-0+QTAILQ_ENTRY (VuClient) next; +bool closed; +}; + + +VuServer *vhost_user_server_start(uint16_t max_queues,
util/vhost-user-server.h line 56
Quoted 10 lines
util/vhost-user-server.h new file +10
-0+ SocketAddress *unix_socket, + AioContext *ctx, + void *server_ptr, + void *device_panic_notifier, + const VuDevIface *vu_iface, + Error **errp); + +void vhost_user_server_stop (VuServer *server); + +void change_vu_context (AioContext *ctx,VuServer *server);
Let's call this vhost_user_server_set_aio_context() for consistency.
Quoted 57 lines
util/Makefile.objs +3
-0@@ -36,6 +36,9 @@ util-obj-y += readline.o util-obj-y += rcu.o util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o +ifdef CONFIG_LINUX +util-obj-y += vhost-user-server.o +endif util-obj-y += qemu-coroutine-sleep.o util-obj-y += qemu-co-shared-resource.o util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).outil/vhost-user-server.c new file +37
-0@@ -0,0 +1,427 @@ +/* +* Sharing QEMU devices via vhost-user protocol +* +* Author: Coiby Xu <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or +* later. See the COPYING file in the top-level directory. +*/ +#include "qemu/osdep.h" +#include <sys/eventfd.h> +#include "qemu/main-loop.h" +#include "vhost-user-server.h" + +static void vmsg_close_fds (VhostUserMsg *vmsg) +{ +int i; +for (i =0 ; i < vmsg->fd_num; i++) { +close (vmsg->fds[i]); + } +} + +static void vmsg_unblock_fds (VhostUserMsg *vmsg) +{ +int i; +for (i =0 ; i < vmsg->fd_num; i++) { +qemu_set_nonblock (vmsg->fds[i]); + } +} + + +static void close_client (VuClient *client) +{ +vu_deinit (&client->parent); + client->sioc =NULL ; +object_unref (OBJECT (client->ioc)); + client->closed = true; +
util/vhost-user-server.c line 47
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+} + +static void panic_cb (VuDev *vu_dev,const char *buf) +{ +if (buf) { +error_report ("vu_panic: %s" , buf); + } + +VuClient *client =container_of (vu_dev, VuClient, parent); +VuServer *server = client->server;
Please put declarations at the start of the block.
Quoted 22 lines
util/vhost-user-server.c new file +22
-0+if (!client->closed) { +close_client (client); +QTAILQ_REMOVE (&server->clients, client, next); + } + +if (server->device_panic_notifier) { + server->device_panic_notifier (client); + } +} + + + +static bool coroutine_fn +vu_message_read (VuDev *vu_dev,int conn_fd,VhostUserMsg *vmsg) +{ +struct iovec iov = { + .iov_base = (char *)vmsg, + .iov_len =VHOST_USER_HDR_SIZE , + }; +int rc, read_bytes =0 ; +/* +* VhostUserMsg is a packed structure, gcc will complain about passing
util/vhost-user-server.c line 79
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+* pointer to a packed structure member if we pass &VhostUserMsg.fd_num +* and &VhostUserMsg.fds directly when calling qio_channel_readv_full, +* thus two temporary variables nfds and fds are used here. +*/ +size_t nfds =0 , nfds_t =0 ; +int *fds =NULL , *fds_t =NULL ; +VuClient *client =container_of (vu_dev, VuClient, parent); +QIOChannel *ioc = client->ioc; + +Error *erp;
The convention is to call this local_err. It should be initialised as NULL.
util/vhost-user-server.c line 92
Quoted 13 lines
util/vhost-user-server.c new file +13
-0+assert (qemu_in_coroutine ()); + do { + /* + * qio_channel_readv_full may have short reads, keeping calling it + * until getting VHOST_USER_HDR_SIZE or 0 bytes in total + */ + rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &erp); + if (rc < 0) { + if (rc == QIO_CHANNEL_ERR_BLOCK) { + qio_channel_yield(ioc, G_IO_IN); + continue; + } else { + error_report("Error while recvmsg: %s", strerror(errno));
I don't think, qio_channel_*() promise anything about the value in errno. (They also don't promise to use recvmsg().)
Instead, use error_report_err() because erp contains the real error message.
Quoted 23 lines
util/vhost-user-server.c new file +23
-0+return false; + } + } + read_bytes += rc; + fds =g_renew (int, fds_t, nfds + nfds_t); +memcpy (fds + nfds, fds_t, nfds_t); + nfds += nfds_t; +if (read_bytes ==VHOST_USER_HDR_SIZE || rc ==0 ) { +break ; + } + }while (true); + + vmsg->fd_num = nfds; +memcpy (vmsg->fds, fds, nfds *sizeof (int )); +g_free (fds); +/* qio_channel_readv_full will make socket fds blocking, unblock them */ +vmsg_unblock_fds (vmsg); +if (vmsg->size >sizeof (vmsg->payload)) { +error_report ("Error: too big message request: %d, " +"size: vmsg->size: %u, " +"while sizeof(vmsg->payload) = %zu" , + vmsg->request, vmsg->size,sizeof (vmsg->payload)); + goto fail;
util/vhost-user-server.c line 125
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+ } + +struct iovec iov_payload = { + .iov_base = (char *)&vmsg->payload, + .iov_len = vmsg->size, + }; +if (vmsg->size) { + rc =qio_channel_readv_all_eof (ioc, &iov_payload,1 , &erp); +if (rc ==-1 ) { +error_report ("Error while reading: %s" ,strerror (errno));
error_report_err() again.
Quoted 15 lines
util/vhost-user-server.c new file +15
-0+ goto fail; + } + } + +return true; + +fail: +vmsg_close_fds (vmsg); + +return false; +} + + +static coroutine_fn void vu_client_next_trip(VuClient *client); +
util/vhost-user-server.c line 150
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+static coroutine_fn void vu_client_trip(void *opaque) +{ +VuClient *client = opaque; + +vu_dispatch (&client->parent); + client->co_trip =NULL ; +if (!client->closed) { +vu_client_next_trip (client); + } +}
The last part is very untypical coroutine code: It says that we want to spawn a new coroutine with vu_client_trip() as its entry point, and then terminates the current one.
Why don't we just put the whole thing in a while (!client->closed) loop and stay in the same coroutine instead of terminating the old one and starting a new one all the time?
util/vhost-user-server.c line 164
Quoted 13 lines
util/vhost-user-server.c new file +13
-0+static coroutine_fn void vu_client_next_trip(VuClient *client) +{ +if (!client->co_trip) { + client->co_trip =qemu_coroutine_create (vu_client_trip, client); +aio_co_schedule (client->ioc->ctx, client->co_trip); + } +} + +static void vu_client_start (VuClient *client) +{ + client->co_trip =qemu_coroutine_create (vu_client_trip, client); +aio_co_enter (client->ioc->ctx, client->co_trip); +}
This is essentially a duplicate of vu_client_next_trip(). The only place where it is called (vu_accept()) knows that client->co_trip is already NULL, so it could just call vu_client_next_trip().
Or in fact, if vu_client_trip() gets turned into a loop, it's vu_client_next_trip() that becomes unnecessary.
Quoted 21 lines
util/vhost-user-server.c new file +21
-0+static void coroutine_fn vu_kick_cb_next(VuClient *client, + kick_info *data); + +static void coroutine_fn vu_kick_cb(void *opaque) +{ +kick_info *data = (kick_info *) opaque; +int index = data->index; +VuDev *dev = data->vu_dev; +VuClient *client; + client =container_of (dev, VuClient, parent); +VuVirtq *vq = &dev->vq[index]; +int sock = vq->kick_fd; +if (sock ==-1 ) { +return ; + } +assert (sock == data->fd); +eventfd_t kick_data; +ssize_t rc; +/* +* When eventfd is closed, the revent is POLLNVAL (=G_IO_NVAL) and +* reading eventfd will return errno=EBADF (Bad file number).
util/vhost-user-server.c line 196
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+* Calling qio_channel_yield(ioc, G_IO_IN) will set reading handler +* for QIOChannel, but aio_dispatch_handlers will only dispatch +* G_IO_IN | G_IO_HUP | G_IO_ERR revents while ignoring +* G_IO_NVAL (POLLNVAL) revents. +* +* Thus when eventfd is closed by vhost-user client, QEMU will ignore +* G_IO_NVAL and keeping polling by repeatedly calling qemu_poll_ns which +* will lead to 100% CPU usage. +* +* To aovid this issue, make sure set_watch and remove_watch use the same
s/aovid/avoid/
util/vhost-user-server.c line 201
Quoted 5 lines
util/vhost-user-server.c new file +5
-0+* AIOContext for QIOChannel. Thus remove_watch will eventually succefully +* remove eventfd from the set of file descriptors polled for +* corresponding GSource. +*/ + rc =read (sock, &kick_data,sizeof (eventfd_t));
Why not a QIOChannel function like for vu_message_read() above?
util/vhost-user-server.c line 213
Quoted 12 lines
util/vhost-user-server.c new file +12
-0+if (rc !=sizeof (eventfd_t)) { +if (errno ==EAGAIN ) { +qio_channel_yield (data->ioc,G_IO_IN ); + }else if (errno !=EINTR ) { + data->co =NULL ; +return ; + } + }else { + vq->handler (dev, index); + } + data->co =NULL ; +vu_kick_cb_next (client, data);
This can be a loop, too, instead of terminating the coroutine and starting a new one for the same function.
util/vhost-user-server.c line 224
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+} + +static void coroutine_fn vu_kick_cb_next(VuClient *client, +kick_info *cb_data) +{ +if (!cb_data->co) { + cb_data->co =qemu_coroutine_create (vu_kick_cb, cb_data); +aio_co_schedule (client->ioc->ctx, cb_data->co); + } +}
Kevin
Re: [PATCH v4 2/5] generic vhost user server
Quoted 13 lines
Quoted 12 lines
util/vhost-user-server.c new file +12
-0+static coroutine_fn void vu_client_next_trip(VuClient *client); + +static coroutine_fn void vu_client_trip(void *opaque) +{ +VuClient *client = opaque; + +vu_dispatch (&client->parent); + client->co_trip =NULL ; +if (!client->closed) { +vu_client_next_trip (client); + } +}
Quoted 1 line
Quoted 1 line
The last part is very untypical coroutine code: It says that we want to
spawn a new coroutine with vu_client_trip() as its entry point, and then terminates the current one.
Quoted 1 line
Quoted 1 line
Why don't we just put the whole thing in a while (!client->closed) loop
and stay in the same coroutine instead of terminating the old one and starting a new one all the time?
Quoted 7 lines
Quoted 7 lines
util/vhost-user-server.c new file +7
-0+static coroutine_fn void vu_client_next_trip(VuClient *client) +{ +if (!client->co_trip) { + client->co_trip =qemu_coroutine_create (vu_client_trip, client); +aio_co_schedule (client->ioc->ctx, client->co_trip); + } +}
Quoted 8 lines
util/vhost-user-server.c line 164
Quoted 6 lines
util/vhost-user-server.c new file +6
-0+ +static void vu_client_start (VuClient *client) +{ + client->co_trip =qemu_coroutine_create (vu_client_trip, client); +aio_co_enter (client->ioc->ctx, client->co_trip); +}This is essentially a duplicate of
vu_client_next_trip(). The only
place where it is called (vu_accept()) knows that client->co_trip is already NULL, so it could just call vu_client_next_trip().
Quoted 2 lines
Or in fact, if
vu_client_trip()gets turned into a loop, it'svu_client_next_trip()that becomes unnecessary.
This part of code is an imitation of nbd_client_trip in nbd/server.c. I think the reason to repeatedly create/start/terminate vu_client_trip is to support BlockBackendAioNotifier. In v5, I will keep running the spawned coroutine in a loop until being informed of the change of AioContext of the block device backend, i.e. vu_client_trip will only be restarted when the block device backend is attached to a different AiOContext.
Quoted 13 lines
util/vhost-user-server.c line 213
Quoted 12 lines
util/vhost-user-server.c new file +12
-0+if (rc !=sizeof (eventfd_t)) { +if (errno ==EAGAIN ) { +qio_channel_yield (data->ioc,G_IO_IN ); + }else if (errno !=EINTR ) { + data->co =NULL ; +return ; + } + }else { + vq->handler (dev, index); + } + data->co =NULL ; +vu_kick_cb_next (client, data);
Quoted 1 line
This can be a loop, too, instead of terminating the coroutine and
starting a new one for the same function.
In v5, I plan to use aio_set_fd_handler to set a read hander which is a wrapper for vu_kick_cb to deal with kick events since eventfd doesn't have the short read issue like socket. Thus vu_kick_cb in libvhost-user can be re-used. My only concern is if this could lead to worse performance in comparison to keep reading from eventfd until getting EAGAIN errno.
Quoted 385 lines · Kevin Wolf
Am 18.02.2020 um 06:07 hat Coiby Xu geschrieben:
Quoted 17 lines
Sharing QEMU devices via vhost-user protocol
Signed-off-by Coiby Xu <[email protected]>--- util/Makefile.objs | 3 + util/vhost-user-server.c | 427 +++++++++++++++++++++++++++++++++++++++ util/vhost-user-server.h | 56 +++++ 3 files changed, 486 insertions(+) create mode 100644 util/vhost-user-server.c create mode 100644 util/vhost-user-server.hutil/vhost-user-server.h new file +0
-0util/vhost-user-server.h line 8
Quoted 8 lines
util/vhost-user-server.h new file +8
-0@@ -0,0 +1,56 @@ +#include "io/channel-socket.h" +#include "io/channel-file.h" +#include "io/net-listener.h" +#include "contrib/libvhost-user/libvhost-user.h" +#include "standard-headers/linux/virtio_blk.h" +#include "qemu/error-report.h" + +typedef struct VuClient VuClient ;I find the terminology a bit confusing here: VuClient is really the connection to a single client, but it's part of the server. The name gives the impression as if this were client-side code. (This is something that already tends to confuse me in the NBD code.)
I'm not sure what a better name could be, though. Maybe VuServerConnevtion or VuExportClient or VuExportConnection?
Quoted 7 lines
util/vhost-user-server.h new file +7
-0+typedef struct VuServer { + QIONetListener *listener; + AioContext *ctx; + QTAILQ_HEAD(, VuClient) clients; + void (*device_panic_notifier)(struct VuClient *client) ; + int max_queues; + const VuDevIface *vu_iface;util/vhost-user-server.h line 26
Quoted 10 lines
util/vhost-user-server.h new file +10
-0+/* +* @ptr_in_device: VuServer pointer memory location in vhost-user device +* struct, so later container_of can be used to get device destruct +*/ +void *ptr_in_device; +bool close; +} VuServer; + +typedef struct kick_info { + VuDev *vu_dev;I suppose this could specifically be VuClient?
Quoted 4 lines
util/vhost-user-server.h new file +4
-0+int fd;/*kick fd*/ +long index;/*queue index*/ +QIOChannel *ioc;/*I/O channel for kick fd*/ +QIOChannelFile *fioc;/*underlying data channel for kick fd*/ util/vhost-user-server.h line 40
Quoted 10 lines
util/vhost-user-server.h new file +10
-0+Coroutine *co; +} kick_info; + +struct VuClient { +VuDev parent; +VuServer *server; +QIOChannel *ioc;/* The current I/O channel */ +QIOChannelSocket *sioc;/* The underlying data channel */ +Coroutine *co_trip; +struct kick_info *kick_info;If each struct
kick_info(btw, QEMU coding style requires CamelCase) has exactly one VuClient and each VuClient has exactly onkick_info, should this be a single struct containing both?[ Coming back from reading the code below - it's because this is in fact an array. This should be made clear in the definition. ]
Quoted 6 lines
util/vhost-user-server.h new file +6
-0+QTAILQ_ENTRY (VuClient) next; +bool closed; +}; + + +VuServer *vhost_user_server_start(uint16_t max_queues,util/vhost-user-server.h line 56
Quoted 10 lines
util/vhost-user-server.h new file +10
-0+ SocketAddress *unix_socket, + AioContext *ctx, + void *server_ptr, + void *device_panic_notifier, + const VuDevIface *vu_iface, + Error **errp); + +void vhost_user_server_stop (VuServer *server); + +void change_vu_context (AioContext *ctx,VuServer *server);Let's call this
vhost_user_server_set_aio_context()for consistency.Quoted 57 lines
util/Makefile.objs +3
-0@@ -36,6 +36,9 @@ util-obj-y += readline.o util-obj-y += rcu.o util-obj-$(CONFIG_MEMBARRIER) += sys_membarrier.o util-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o +ifdef CONFIG_LINUX +util-obj-y += vhost-user-server.o +endif util-obj-y += qemu-coroutine-sleep.o util-obj-y += qemu-co-shared-resource.o util-obj-y += coroutine-$(CONFIG_COROUTINE_BACKEND).outil/vhost-user-server.c new file +37
-0@@ -0,0 +1,427 @@ +/* +* Sharing QEMU devices via vhost-user protocol +* +* Author: Coiby Xu <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or +* later. See the COPYING file in the top-level directory. +*/ +#include "qemu/osdep.h" +#include <sys/eventfd.h> +#include "qemu/main-loop.h" +#include "vhost-user-server.h" + +static void vmsg_close_fds (VhostUserMsg *vmsg) +{ +int i; +for (i =0 ; i < vmsg->fd_num; i++) { +close (vmsg->fds[i]); + } +} + +static void vmsg_unblock_fds (VhostUserMsg *vmsg) +{ +int i; +for (i =0 ; i < vmsg->fd_num; i++) { +qemu_set_nonblock (vmsg->fds[i]); + } +} + + +static void close_client (VuClient *client) +{ +vu_deinit (&client->parent); + client->sioc =NULL ; +object_unref (OBJECT (client->ioc)); + client->closed = true; +util/vhost-user-server.c line 47
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+} + +static void panic_cb (VuDev *vu_dev,const char *buf) +{ +if (buf) { +error_report ("vu_panic: %s" , buf); + } + +VuClient *client =container_of (vu_dev, VuClient, parent); +VuServer *server = client->server;Please put declarations at the start of the block.
Quoted 22 lines
util/vhost-user-server.c new file +22
-0+if (!client->closed) { +close_client (client); +QTAILQ_REMOVE (&server->clients, client, next); + } + +if (server->device_panic_notifier) { + server->device_panic_notifier (client); + } +} + + + +static bool coroutine_fn +vu_message_read (VuDev *vu_dev,int conn_fd,VhostUserMsg *vmsg) +{ +struct iovec iov = { + .iov_base = (char *)vmsg, + .iov_len =VHOST_USER_HDR_SIZE , + }; +int rc, read_bytes =0 ; +/* +* VhostUserMsg is a packed structure, gcc will complain about passing util/vhost-user-server.c line 79
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+* pointer to a packed structure member if we pass &VhostUserMsg.fd_num +* and &VhostUserMsg.fds directly when calling qio_channel_readv_full, +* thus two temporary variables nfds and fds are used here. +*/ +size_t nfds =0 , nfds_t =0 ; +int *fds =NULL , *fds_t =NULL ; +VuClient *client =container_of (vu_dev, VuClient, parent); +QIOChannel *ioc = client->ioc; + +Error *erp;The convention is to call this
local_err. It should be initialised as NULL.util/vhost-user-server.c line 92
Quoted 13 lines
util/vhost-user-server.c new file +13
-0+assert (qemu_in_coroutine ()); + do { + /* + * qio_channel_readv_full may have short reads, keeping calling it + * until getting VHOST_USER_HDR_SIZE or 0 bytes in total + */ + rc = qio_channel_readv_full(ioc, &iov, 1, &fds_t, &nfds_t, &erp); + if (rc < 0) { + if (rc == QIO_CHANNEL_ERR_BLOCK) { + qio_channel_yield(ioc, G_IO_IN); + continue; + } else { + error_report("Error while recvmsg: %s", strerror(errno));I don't think, qio_channel_*() promise anything about the value in errno. (They also don't promise to use
recvmsg().)Instead, use
error_report_err()because erp contains the real error message.Quoted 23 lines
util/vhost-user-server.c new file +23
-0+return false; + } + } + read_bytes += rc; + fds =g_renew (int, fds_t, nfds + nfds_t); +memcpy (fds + nfds, fds_t, nfds_t); + nfds += nfds_t; +if (read_bytes ==VHOST_USER_HDR_SIZE || rc ==0 ) { +break ; + } + }while (true); + + vmsg->fd_num = nfds; +memcpy (vmsg->fds, fds, nfds *sizeof (int )); +g_free (fds); +/* qio_channel_readv_full will make socket fds blocking, unblock them */ +vmsg_unblock_fds (vmsg); +if (vmsg->size >sizeof (vmsg->payload)) { +error_report ("Error: too big message request: %d, " +"size: vmsg->size: %u, " +"while sizeof(vmsg->payload) = %zu" , + vmsg->request, vmsg->size,sizeof (vmsg->payload)); + goto fail;util/vhost-user-server.c line 125
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+ } + +struct iovec iov_payload = { + .iov_base = (char *)&vmsg->payload, + .iov_len = vmsg->size, + }; +if (vmsg->size) { + rc =qio_channel_readv_all_eof (ioc, &iov_payload,1 , &erp); +if (rc ==-1 ) { +error_report ("Error while reading: %s" ,strerror (errno));
error_report_err()again.Quoted 15 lines
util/vhost-user-server.c new file +15
-0+ goto fail; + } + } + +return true; + +fail: +vmsg_close_fds (vmsg); + +return false; +} + + +static coroutine_fn void vu_client_next_trip(VuClient *client); +util/vhost-user-server.c line 150
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+static coroutine_fn void vu_client_trip(void *opaque) +{ +VuClient *client = opaque; + +vu_dispatch (&client->parent); + client->co_trip =NULL ; +if (!client->closed) { +vu_client_next_trip (client); + } +}The last part is very untypical coroutine code: It says that we want to spawn a new coroutine with
vu_client_trip()as its entry point, and then terminates the current one.Why don't we just put the whole thing in a while (!client->closed) loop and stay in the same coroutine instead of terminating the old one and starting a new one all the time?
util/vhost-user-server.c line 164
Quoted 13 lines
util/vhost-user-server.c new file +13
-0+static coroutine_fn void vu_client_next_trip(VuClient *client) +{ +if (!client->co_trip) { + client->co_trip =qemu_coroutine_create (vu_client_trip, client); +aio_co_schedule (client->ioc->ctx, client->co_trip); + } +} + +static void vu_client_start (VuClient *client) +{ + client->co_trip =qemu_coroutine_create (vu_client_trip, client); +aio_co_enter (client->ioc->ctx, client->co_trip); +}This is essentially a duplicate of
vu_client_next_trip(). The only place where it is called (vu_accept()) knows thatclient->co_tripis already NULL, so it could just callvu_client_next_trip().Or in fact, if
vu_client_trip()gets turned into a loop, it'svu_client_next_trip()that becomes unnecessary.Quoted 21 lines
util/vhost-user-server.c new file +21
-0+static void coroutine_fn vu_kick_cb_next(VuClient *client, + kick_info *data); + +static void coroutine_fn vu_kick_cb(void *opaque) +{ +kick_info *data = (kick_info *) opaque; +int index = data->index; +VuDev *dev = data->vu_dev; +VuClient *client; + client =container_of (dev, VuClient, parent); +VuVirtq *vq = &dev->vq[index]; +int sock = vq->kick_fd; +if (sock ==-1 ) { +return ; + } +assert (sock == data->fd); +eventfd_t kick_data; +ssize_t rc; +/* +* When eventfd is closed, the revent is POLLNVAL (=G_IO_NVAL) and +* reading eventfd will return errno=EBADF (Bad file number). util/vhost-user-server.c line 196
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+* Calling qio_channel_yield(ioc, G_IO_IN) will set reading handler +* for QIOChannel, but aio_dispatch_handlers will only dispatch +* G_IO_IN | G_IO_HUP | G_IO_ERR revents while ignoring +* G_IO_NVAL (POLLNVAL) revents. +* +* Thus when eventfd is closed by vhost-user client, QEMU will ignore +* G_IO_NVAL and keeping polling by repeatedly calling qemu_poll_ns which +* will lead to 100% CPU usage. +* +* To aovid this issue, make sure set_watch and remove_watch use the same s/aovid/avoid/
util/vhost-user-server.c line 201
Quoted 5 lines
util/vhost-user-server.c new file +5
-0+* AIOContext for QIOChannel. Thus remove_watch will eventually succefully +* remove eventfd from the set of file descriptors polled for +* corresponding GSource. +*/ + rc =read (sock, &kick_data,sizeof (eventfd_t));Why not a QIOChannel function like for
vu_message_read()above?util/vhost-user-server.c line 213
Quoted 12 lines
util/vhost-user-server.c new file +12
-0+if (rc !=sizeof (eventfd_t)) { +if (errno ==EAGAIN ) { +qio_channel_yield (data->ioc,G_IO_IN ); + }else if (errno !=EINTR ) { + data->co =NULL ; +return ; + } + }else { + vq->handler (dev, index); + } + data->co =NULL ; +vu_kick_cb_next (client, data);This can be a loop, too, instead of terminating the coroutine and starting a new one for the same function.
util/vhost-user-server.c line 224
Quoted 10 lines
util/vhost-user-server.c new file +10
-0+} + +static void coroutine_fn vu_kick_cb_next(VuClient *client, +kick_info *cb_data) +{ +if (!cb_data->co) { + cb_data->co =qemu_coroutine_create (vu_kick_cb, cb_data); +aio_co_schedule (client->ioc->ctx, cb_data->co); + } +}Kevin
Signature · 2 lines
Best regards, Coiby
PATCH v4 3/5 vhost-user block device backend server
In reply to Coiby XuBy making use of libvhost, multiple block device drives can be exported and each drive can serve multiple clients simultaneously. Since vhost-user-server needs a block drive to be created first, delay the creation of this object.
--- Makefile.target | 1 + backends/Makefile.objs | 2 + backends/vhost-user-blk-server.c | 718 +++++++++++++++++++++++++++++++ backends/vhost-user-blk-server.h | 21 + vl.c | 4 + 5 files changed, 746 insertions(+) create mode 100644 backends/vhost-user-blk-server.c create mode 100644 backends/vhost-user-blk-server.h
Makefile.target +1 -0
@@ -159,6 +159,7 @@ obj-y += monitor/ obj-y += qapi/ obj-y += memory.o obj-y += memory_mapping.o +obj-$(CONFIG_LINUX) += ../contrib/libvhost-user/libvhost-user.o obj-y += migration/ram.o LIBS := $(libs_softmmu) $(LIBS)
backends/Makefile.objs +2 -0
@@ -14,6 +14,8 @@ common-obj-y += cryptodev-vhost.o common-obj-$(CONFIG_VHOST_CRYPTO) += cryptodev-vhost-user.o endif +common-obj-$(CONFIG_LINUX) += vhost-user-blk-server.o + common-obj-$(call land,$(CONFIG_VHOST_USER),$(CONFIG_VIRTIO)) += vhost-user.o common-obj-$(CONFIG_LINUX) += hostmem-memfd.o
backends/vhost-user-blk-server.c new file +718 -0
718 lines not on this page · raw message
backends/vhost-user-blk-server.h new file +21 -0
@@ -0,0 +1,21 @@ +#include "util/vhost-user-server.h" +typedef struct VuBlockDev VuBlockDev ; +#define TYPE_VHOST_USER_BLK_SERVER "vhost-user-blk-server" +#define VHOST_USER_BLK_SERVER (obj) \ + OBJECT_CHECK(VuBlockDev, obj, TYPE_VHOST_USER_BLK_SERVER) + +/* vhost user block device */ +struct VuBlockDev { +Object parent_obj; +char *node_name; +SocketAddress *addr; +bool exit_when_panic; +AioContext *ctx; +VuServer *vu_server; +uint32_t blk_size; +BlockBackend *backend; +QIOChannelSocket *sioc; +QTAILQ_ENTRY (VuBlockDev) next; +struct virtio_blk_config blkcfg; +bool writable; +};
vl.c +4 -0
@@ -2538,6 +2538,10 @@ static bool object_create_initial(const char *type, QemuOpts *opts) }#endif +/* Reason: vhost-user-server property "name" */ +if (g_str_equal (type,"vhost-user-blk-server" )) { +return false; + }/* * Reason: filter-* property "netdev" etc. */
Signature · 1 line
2.25.0
Re: [PATCH v4 3/5] vhost-user block device backend server
Am 18.02.2020 um 06:07 hat Coiby Xu geschrieben:
backends/vhost-user-blk-server.c
Quoted 46 lines
By making use of libvhost, multiple block device drives can be exported and each drive can serve multiple clients simultaneously. Since vhost-user-server needs a block drive to be created first, delay the creation of this object.
Signed-off-by Coiby Xu <[email protected]>--- Makefile.target | 1 + backends/Makefile.objs | 2 + backends/vhost-user-blk-server.c | 718 +++++++++++++++++++++++++++++++ backends/vhost-user-blk-server.h | 21 + vl.c | 4 + 5 files changed, 746 insertions(+) create mode 100644 backends/vhost-user-blk-server.c create mode 100644 backends/vhost-user-blk-server.hMakefile.target +1
-0@@ -159,6 +159,7 @@ obj-y += monitor/ obj-y += qapi/ obj-y += memory.o obj-y += memory_mapping.o +obj-$(CONFIG_LINUX) += ../contrib/libvhost-user/libvhost-user.o obj-y += migration/ram.o LIBS := $(libs_softmmu) $(LIBS)backends/Makefile.objs +2
-0@@ -14,6 +14,8 @@ common-obj-y += cryptodev-vhost.o common-obj-$(CONFIG_VHOST_CRYPTO) += cryptodev-vhost-user.o endif +common-obj-$(CONFIG_LINUX) += vhost-user-blk-server.o + common-obj-$(call land,$(CONFIG_VHOST_USER),$(CONFIG_VIRTIO)) += vhost-user.o common-obj-$(CONFIG_LINUX) += hostmem-memfd.obackends/vhost-user-blk-server.c new file +0
-0
Please move this to block/export/vhost-user-blk.c. This will automatically have it covered as a block layer component in MAINTAINERS, and it will provide a place where other exports can be put, too.
Quoted 116 lines
backends/vhost-user-blk-server.c new file +115
-0@@ -0,0 +1,718 @@ +/* +* Sharing QEMU block devices via vhost-user protocal +* +* Author: Coiby Xu <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or +* later. See the COPYING file in the top-level directory. +*/ +#include "qemu/osdep.h" +#include "block/block.h" +#include "vhost-user-blk-server.h" +#include "qapi/error.h" +#include "qom/object_interfaces.h" +#include "sysemu/block-backend.h" + +enum { +VHOST_USER_BLK_MAX_QUEUES =1 , +}; +struct virtio_blk_inhdr { +unsigned char status; +}; + +static QTAILQ_HEAD(, VuBlockDev) vu_block_devs = +QTAILQ_HEAD_INITIALIZER (vu_block_devs); + + +typedef struct VuBlockReq { +VuVirtqElement *elem; +int64_t sector_num; +size_t size; +struct virtio_blk_inhdr *in; +struct virtio_blk_outhdr out; +VuClient *client; +struct VuVirtq *vq; +}VuBlockReq ; + + +static void vu_block_req_complete (VuBlockReq *req) +{ +VuDev *vu_dev = &req->client->parent; + +/* IO size with 1 extra status byte */ +vu_queue_push (vu_dev, req->vq, req->elem, req->size +1 ); +vu_queue_notify (vu_dev, req->vq); + +if (req->elem) { +free (req->elem); + } + +g_free (req); +} + +static VuBlockDev *get_vu_block_device_by_client (VuClient *client) +{ +return container_of (client->server->ptr_in_device, VuBlockDev, vu_server); +} + +static int coroutine_fn +vu_block_discard_write_zeroes (VuBlockReq *req,struct iovec *iov, +uint32_t iovcnt,uint32_t type) +{ +struct virtio_blk_discard_write_zeroes desc; +ssize_t size =iov_to_buf (iov, iovcnt,0 , &desc,sizeof (desc)); +if (unlikely (size !=sizeof (desc))) { +error_report ("Invalid size %ld, expect %ld" , size,sizeof (desc)); +return -EINVAL ; + } + +VuBlockDev *vdev_blk =get_vu_block_device_by_client (req->client); +uint64_t range[2 ] = {le64toh (desc.sector) <<9 , +le32toh (desc.num_sectors) <<9 }; +if (type ==VIRTIO_BLK_T_DISCARD ) { +if (blk_co_pdiscard (vdev_blk->backend, range[0 ], range[1 ]) ==0 ) { +return 0 ; + } + }else if (type ==VIRTIO_BLK_T_WRITE_ZEROES ) { +if (blk_co_pwrite_zeroes (vdev_blk->backend, + range[0 ], range[1 ],0 ) ==0 ) { +return 0 ; + } + } + +return -EINVAL ; +} + + +static void coroutine_fn vu_block_flush(VuBlockReq *req) +{ +VuBlockDev *vdev_blk =get_vu_block_device_by_client (req->client); +BlockBackend *backend = vdev_blk->backend; +blk_co_flush (backend); +} + + +static int coroutine_fn vu_block_virtio_process_req(VuClient *client, +VuVirtq *vq) +{ +VuDev *vu_dev = &client->parent; +VuVirtqElement *elem; +uint32_t type; +VuBlockReq *req; + +VuBlockDev *vdev_blk =get_vu_block_device_by_client (client); +BlockBackend *backend = vdev_blk->backend; + elem =vu_queue_pop (vu_dev, vq,sizeof (VuVirtqElement) + +sizeof (VuBlockReq)); +if (!elem) { +return -1 ; + } + +struct iovec *in_iov = elem->in_sg; +struct iovec *out_iov = elem->out_sg; +unsigned in_num = elem->in_num; +unsigned out_num = elem->out_num; +/* refer to hw/block/virtio_blk.c */
backends/vhost-user-blk-server.c line 125
Quoted 10 lines
backends/vhost-user-blk-server.c new file +10
-0+if (elem->out_num <1 || elem->in_num <1 ) { +error_report ("virtio-blk request missing headers" ); +free (elem); +return -EINVAL ; + } + + req =g_new0 (VuBlockReq,1 ); + req->client = client; + req->vq = vq; + req->elem = elem;
You can keep req on the stack, it will be freed before this function returns.
Quoted 30 lines
backends/vhost-user-blk-server.c new file +30
-0+if (unlikely (iov_to_buf (out_iov, out_num,0 , &req->out, +sizeof (req->out)) !=sizeof (req->out))) { +error_report ("virtio-blk request outhdr too short" ); + goto err; + } + +iov_discard_front (&out_iov, &out_num,sizeof (req->out)); + +if (in_iov[in_num -1 ].iov_len <sizeof (struct virtio_blk_inhdr )) { +error_report ("virtio-blk request inhdr too short" ); + goto err; + } + +/* We always touch the last byte, so just see how big in_iov is. */ + req->in = (void *)in_iov[in_num -1 ].iov_base + + in_iov[in_num -1 ].iov_len + -sizeof (struct virtio_blk_inhdr ); +iov_discard_back (in_iov, &in_num,sizeof (struct virtio_blk_inhdr )); + + + type =le32toh (req->out.type); +switch (type & ~VIRTIO_BLK_T_BARRIER ) { +case VIRTIO_BLK_T_IN : +case VIRTIO_BLK_T_OUT : { +ssize_t ret =0 ; +bool is_write = type &VIRTIO_BLK_T_OUT ; + req->sector_num =le64toh (req->out.sector); + +int64_t offset = req->sector_num * vdev_blk->blk_size; +QEMUIOVector *qiov =g_new0 (QEMUIOVector,1 );
backends/vhost-user-blk-server.c line 166
Quoted 10 lines
backends/vhost-user-blk-server.c new file +10
-0+if (is_write) { +qemu_iovec_init_external (qiov, out_iov, out_num); + ret =blk_co_pwritev (backend, offset, qiov->size, + qiov,0 ); + }else { +qemu_iovec_init_external (qiov, in_iov, in_num); + ret =blk_co_preadv (backend, offset, qiov->size, + qiov,0 ); + } +aio_wait_kick ();
What purpose does this aio_wait_kick() serve? Usually you do this if you want to signal completion fo something to a different thread, but I don't see that we changed any control state that could be visible to other threads.
Quoted 30 lines
backends/vhost-user-blk-server.c new file +30
-0+if (ret >=0 ) { + req->in->status =VIRTIO_BLK_S_OK ; + }else { + req->in->status =VIRTIO_BLK_S_IOERR ; + } +vu_block_req_complete (req); +break ; + } +case VIRTIO_BLK_T_FLUSH : +vu_block_flush (req); + req->in->status =VIRTIO_BLK_S_OK ; +vu_block_req_complete (req); +break ; +case VIRTIO_BLK_T_GET_ID : { +size_t size =MIN (iov_size (&elem->in_sg[0 ], in_num), +VIRTIO_BLK_ID_BYTES ); +snprintf (elem->in_sg[0 ].iov_base, size,"%s" ,"vhost_user_blk_server" ); + req->in->status =VIRTIO_BLK_S_OK ; + req->size = elem->in_sg[0 ].iov_len; +vu_block_req_complete (req); +break ; + } +case VIRTIO_BLK_T_DISCARD : +case VIRTIO_BLK_T_WRITE_ZEROES : { +int rc; + rc =vu_block_discard_write_zeroes (req, &elem->out_sg[1 ], + out_num, type); +if (rc ==0 ) { + req->in->status =VIRTIO_BLK_S_OK ; + }else {
backends/vhost-user-blk-server.c line 206
Quoted 10 lines
backends/vhost-user-blk-server.c new file +10
-0+ req->in->status =VIRTIO_BLK_S_IOERR ; + } +vu_block_req_complete (req); +break ; + } +default : + req->in->status =VIRTIO_BLK_S_UNSUPP ; +vu_block_req_complete (req); +break ; + }
You have vu_block_req_complete() as the last thing in every branch. You could have it just once after the switch block.
backends/vhost-user-blk-server.c line 214
Quoted 7 lines
backends/vhost-user-blk-server.c new file +7
-0+return 0 ; + +err: +free (elem); +g_free (req); +return -EINVAL ; +}
Okay, so vu_block_virtio_process_req() takes a single request from the virtqueue and processes it. This makes sense as a building block, but it doesn't allow parallelism yet. I expect to see the creation of multiple coroutines below that can run in parallel.
Quoted 5 lines
backends/vhost-user-blk-server.c new file +5
-0+ +static void vu_block_process_vq (VuDev *vu_dev,int idx) +{ +VuClient *client; +VuVirtq *vq;
backends/vhost-user-blk-server.c line 230
Quoted 10 lines
backends/vhost-user-blk-server.c new file +10
-0+int ret; + + client =container_of (vu_dev, VuClient, parent); +assert (client); + + vq =vu_get_queue (vu_dev, idx); +assert (vq); + +while (1 ) { + ret =vu_block_virtio_process_req (client, vq);
If you call vu_block_virtio_process_req(), then this one need to be a coroutine_fn, too.
Quoted 5 lines
backends/vhost-user-blk-server.c new file +5
-0+if (ret) { +break ; + } + } +}
backends/vhost-user-blk-server.c line 245
Quoted 10 lines
backends/vhost-user-blk-server.c new file +10
-0+ +static void vu_block_queue_set_started (VuDev *vu_dev,int idx,bool started) +{ +VuVirtq *vq; + +assert (vu_dev); + + vq =vu_get_queue (vu_dev, idx); +vu_set_queue_handler (vu_dev, vq, started ? vu_block_process_vq :NULL ); +}
If vu_block_process_vq() is used as a vq->handler, will it not be called outside coroutine context? How can this work when vu_block_virtio_process_req() required to be run in coroutine context?
Another problem is that vu_block_process_vq() runs a loop as long as there are requests that aren't completed yet. During this time, vu_kick_cb() will block. I don't think this is what was intended. Instead, we should start the requests (without waiting for their completion) and return.
vl.c static bool object_create_initial(const char *type, QemuOpts *opts) · line 2541
Quoted 9 lines
vl.c +1
-0@@ -2538,6 +2538,10 @@ static bool object_create_initial(const char *type, QemuOpts *opts) }#endif +/* Reason: vhost-user-server property "name" */
"node-name" now.
vl.c static bool object_create_initial(const char *type, QemuOpts *opts) · line 2547
Quoted 6 lines
vl.c +3
-0+if (g_str_equal (type,"vhost-user-blk-server" )) { +return false; + }/* * Reason: filter-* property "netdev" etc. */
Kevin
PATCH v4 4/5 a standone-alone tool to directly share disk image file via vhost-user protocol
In reply to Coiby Xuvhost-user-blk could have played as vhost-user backend but it only supports raw file and don't support VIRTIO_BLK_T_DISCARD and VIRTIO_BLK_T_WRITE_ZEROES operations on raw file (ioctl(fd, BLKDISCARD) is only valid for real block device).
In the future Kevin's qemu-storage-daemon will be used to replace this tool.
--- Makefile | 4 + configure | 3 + qemu-vu.c | 252 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 259 insertions(+) create mode 100644 qemu-vu.c
Makefile +4 -0
@@ -572,6 +572,10 @@ qemu-img.o: qemu-img-cmds.h qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) + +ifdef CONFIG_LINUX +qemu-vu$(EXESUF): qemu-vu.o backends/vhost-user-blk-server.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) libvhost-user.a +endif qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
configure +3 -0
@@ -6239,6 +6239,9 @@ if test "$want_tools" = "yes" ; then if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then tools="qemu-nbd\$(EXESUF) $tools" fi + if [ "$linux" = "yes" ] ; then + tools="qemu-vu\$(EXESUF) $tools" + fi if [ "$ivshmem" = "yes" ]; then tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools" fi
qemu-vu.c new file +252 -0
252 lines not on this page · raw message
Signature · 1 line
2.25.0
PATCH v4 5/5 new qTest case to test the vhost-user-blk-server
In reply to Coiby XuThis test case has the same tests as tests/virtio-blk-test.c except for tests have block_resize.
--- tests/Makefile.include | 3 +- tests/qtest/Makefile.include | 2 + tests/qtest/libqos/vhost-user-blk.c | 126 +++++ tests/qtest/libqos/vhost-user-blk.h | 44 ++ tests/qtest/vhost-user-blk-test.c | 694 ++++++++++++++++++++++++++++ 5 files changed, 868 insertions(+), 1 deletion(-) create mode 100644 tests/qtest/libqos/vhost-user-blk.c create mode 100644 tests/qtest/libqos/vhost-user-blk.h create mode 100644 tests/qtest/vhost-user-blk-test.c
tests/Makefile.include +2 -1
@@ -629,7 +629,8 @@ endef $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: %-softmmu/all $(check-qtest-y) $(call do_test_human,$(check-qtest-$*-y:%=tests/qtest/%$(EXESUF)) $(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)), \ QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \ - QTEST_QEMU_IMG=qemu-img$(EXESUF)) + QTEST_QEMU_IMG=./qemu-img$(EXESUF) \ + QTEST_QEMU_VU_BINARY=./qemu-vu$(EXESUF)) check-unit: $(check-unit-y) $(call do_test_human, $^)
tests/qtest/Makefile.include +2 -0
@@ -182,6 +182,7 @@ qos-test-obj-y += tests/qtest/libqos/virtio.o qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/libqos/virtio-9p.o qos-test-obj-y += tests/qtest/libqos/virtio-balloon.o qos-test-obj-y += tests/qtest/libqos/virtio-blk.o +qos-test-obj-$(CONFIG_LINUX) += tests/qtest/libqos/vhost-user-blk.o qos-test-obj-y += tests/qtest/libqos/virtio-mmio.o qos-test-obj-y += tests/qtest/libqos/virtio-net.o qos-test-obj-y += tests/qtest/libqos/virtio-pci.o
@@ -224,6 +225,7 @@ qos-test-obj-$(CONFIG_VHOST_NET_USER) += tests/qtest/vhost-user-test.o $(chardev qos-test-obj-y += tests/qtest/virtio-test.o qos-test-obj-$(CONFIG_VIRTFS) += tests/qtest/virtio-9p-test.o qos-test-obj-y += tests/qtest/virtio-blk-test.o +qos-test-obj-$(CONFIG_LINUX) += tests/qtest/vhost-user-blk-test.o qos-test-obj-y += tests/qtest/virtio-net-test.o qos-test-obj-y += tests/qtest/virtio-rng-test.o qos-test-obj-y += tests/qtest/virtio-scsi-test.o
tests/qtest/libqos/vhost-user-blk.c new file +126 -0
126 lines not on this page · raw message
tests/qtest/libqos/vhost-user-blk.h new file +44 -0
44 lines not on this page · raw message
tests/qtest/vhost-user-blk-test.c new file +694 -0
694 lines not on this page · raw message
Signature · 1 line
2.25.0
Quoted 4 lines · Coiby Xu
v4:
* add object properties inclass_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.
Thanks! I think the vhost-user server code can be simplified if libvhost-user uses the event loop for asynchronous socket I/O. Then it's no longer necessary to duplicate vu_message_read() and vu_kick_cb(). I've replied to Patch 1 and we can discuss on IRC if you want to chat about it.
I've also CCed Marc-André to see what he thinks about removing the blocking recv from libvhost-user and instead using the event loop (just like for eventfds).
Stefan
Hi Stefan,
Thank you for reviewing my code!
I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.
If we use qio_channel_set_aio_fd_handler to monitor G_IO_IN event, i.e. use vu_dispatch as the read handler, then we can re-use vu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation of poll() and ppoll() is not affected by the O_NONBLOCK flag" despite that we use qio_channel_set_blocking before calling qio_channel_set_aio_fd_handler to make recv non-blocking.
Previously I needed to run customized vu_kick_cb as coroutines to call blk_co_readv/blk_co_writev directly. After taking Kevin's feedback on v4 into consideration, now I use aio_set_fd_handler to set a read handler for kick_fd and this read handler will then call vu_kick_cb.
Quoted 18 lines · Stefan Hajnoczi
Quoted 4 lines · Coiby Xu
v4:
* add object properties inclass_init
* relocate vhost-user-blk-test
* other changes including using SocketAddress, coding style, etc.Thanks! I think the vhost-user server code can be simplified if libvhost-user uses the event loop for asynchronous socket I/O. Then it's no longer necessary to duplicate
vu_message_read()andvu_kick_cb(). I've replied to Patch 1 and we can discuss on IRC if you want to chat about it.I've also CCed Marc-André to see what he thinks about removing the blocking recv from libvhost-user and instead using the event loop (just like for eventfds).
Stefan
Signature · 2 lines
Best regards, Coiby
Quoted 7 lines · Coiby Xu
Hi Stefan,
Thank you for reviewing my code!
I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.
Quoted 7 lines
If we use
qio_channel_set_aio_fd_handlerto monitorG_IO_INevent, i.e. usevu_dispatchas the read handler, then we can re-usevu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation ofpoll()andppoll()is not affected by theO_NONBLOCKflag" despite that we useqio_channel_set_blockingbefore callingqio_channel_set_aio_fd_handlerto make recv non-blocking.
I'm not sure I understand. poll() just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (without O_NONBLOCK) or return only 1 byte (with O_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.
But please go ahead and send the next revision and I'll take a look.
Stefan
Thank you for reminding me of this socket short read issue! It seems we still need customized vu_message_read because libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user's vu_message_read by making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.
Quoted 24 lines · Stefan Hajnoczi
Quoted 7 lines · Coiby Xu
Hi Stefan,
Thank you for reviewing my code!
I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.
Quoted 7 lines
If we use
qio_channel_set_aio_fd_handlerto monitorG_IO_INevent, i.e. usevu_dispatchas the read handler, then we can re-usevu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation ofpoll()andppoll()is not affected by theO_NONBLOCKflag" despite that we useqio_channel_set_blockingbefore callingqio_channel_set_aio_fd_handlerto make recv non-blocking.I'm not sure I understand.
poll()just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (withoutO_NONBLOCK) or return only 1 byte (withO_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.
Quoted 3 lines
But please go ahead and send the next revision and I'll take a look.
Stefan
--
Best regards,
Coiby
Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:
Quoted 7 lines
Thank you for reminding me of this socket short read issue! It seems we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.
In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Kevin
Quoted 35 lines · Stefan Hajnoczi
Quoted 24 lines · Coiby Xu
Quoted 7 lines
Hi Stefan,
Thank you for reviewing my code!
I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.
Quoted 7 lines
If we use
qio_channel_set_aio_fd_handlerto monitorG_IO_INevent, i.e. usevu_dispatchas the read handler, then we can re-usevu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation ofpoll()andppoll()is not affected by theO_NONBLOCKflag" despite that we useqio_channel_set_blockingbefore callingqio_channel_set_aio_fd_handlerto make recv non-blocking.I'm not sure I understand.
poll()just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (withoutO_NONBLOCK) or return only 1 byte (withO_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.Quoted 3 lines
But please go ahead and send the next revision and I'll take a look.
Stefan
--
Best regards,
Coiby
Quoted 10 lines
Quoted 6 lines
we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting vu_message_read to address the short read issue. In v3 & v4, I extended libvhost-user to allow vu_message_read to be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.
Quoted 52 lines · Kevin Wolf
Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:
Quoted 7 lines
Thank you for reminding me of this socket short read issue! It seems we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Kevin
Quoted 35 lines · Stefan Hajnoczi
Quoted 24 lines · Coiby Xu
Quoted 7 lines
Hi Stefan,
Thank you for reviewing my code!
I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.
Quoted 7 lines
If we use
qio_channel_set_aio_fd_handlerto monitorG_IO_INevent, i.e. usevu_dispatchas the read handler, then we can re-usevu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation ofpoll()andppoll()is not affected by theO_NONBLOCKflag" despite that we useqio_channel_set_blockingbefore callingqio_channel_set_aio_fd_handlerto make recv non-blocking.I'm not sure I understand.
poll()just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (withoutO_NONBLOCK) or return only 1 byte (withO_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.Quoted 3 lines
But please go ahead and send the next revision and I'll take a look.
Stefan
--
Best regards,
Coiby
Signature · 2 lines
Best regards, Coiby
Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Quoted 6 lines
But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting
vu_message_readto address the short read issue. In v3 & v4, I extended libvhost-user to allowvu_message_readto be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.
The way your latest patches have it, with a separate read function, works for me.
You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.
The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until read() returns enough bytes is not acceptable in QEMU.
Kevin
Quoted 59 lines · Kevin Wolf
Quoted 52 lines
Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:
Quoted 7 lines
Thank you for reminding me of this socket short read issue! It seems we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Kevin
Quoted 35 lines · Stefan Hajnoczi
Quoted 24 lines · Coiby Xu
Quoted 7 lines
Hi Stefan,
Thank you for reviewing my code!
I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.
Quoted 7 lines
If we use
qio_channel_set_aio_fd_handlerto monitorG_IO_INevent, i.e. usevu_dispatchas the read handler, then we can re-usevu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation ofpoll()andppoll()is not affected by theO_NONBLOCKflag" despite that we useqio_channel_set_blockingbefore callingqio_channel_set_aio_fd_handlerto make recv non-blocking.I'm not sure I understand.
poll()just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (withoutO_NONBLOCK) or return only 1 byte (withO_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.Quoted 3 lines
But please go ahead and send the next revision and I'll take a look.
Stefan
--
Best regards,
CoibySignature · 2 lines
Best regards, Coiby
Hi
Quoted 20 lines · Kevin Wolf
Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Quoted 6 lines
But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting
vu_message_readto address the short read issue. In v3 & v4, I extended libvhost-user to allowvu_message_readto be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.
Quoted 2 lines
The way your latest patches have it, with a separate read function, works for me.
Done right, I am not against it, fwiw
Quoted 9 lines
You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.
The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until
read()returns enough bytes is not acceptable in QEMU.
In the context of vhost-user, local unix sockets with short messages (do we have >1k messages?), I am not sure if this is really a problem.
And isn't it possible to run libvhost-user in its own thread for this series?
Quoted 63 lines
Kevin
Quoted 59 lines · Kevin Wolf
Quoted 10 lines
Am 27.02.2020 um 10:53 hat Coiby Xu geschrieben:
Quoted 7 lines
Thank you for reminding me of this socket short read issue! It seems we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.Quoted 42 lines
In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Kevin
Quoted 35 lines · Stefan Hajnoczi
Quoted 24 lines · Coiby Xu
Quoted 7 lines
Hi Stefan,
Thank you for reviewing my code!
I tried to reach you on IRC. But somehow either you missed my message or I missed your reply. So I will reply by email instead.
Quoted 7 lines
If we use
qio_channel_set_aio_fd_handlerto monitorG_IO_INevent, i.e. usevu_dispatchas the read handler, then we can re-usevu_message_read. And "removing the blocking recv from libvhost-user" isn't necessary because "the operation ofpoll()andppoll()is not affected by theO_NONBLOCKflag" despite that we useqio_channel_set_blockingbefore callingqio_channel_set_aio_fd_handlerto make recv non-blocking.I'm not sure I understand.
poll()just says whether the file descriptor is readable. It does not say whether enough bytes are readable :). So our callback will be invoked if there is 1 byte ready, but when we try to read 20 bytes either it will block (withoutO_NONBLOCK) or return only 1 byte (withO_NONBLOCK). Neither case is okay, so I expect that code changes will be necessary.Quoted 3 lines
But please go ahead and send the next revision and I'll take a look.
Stefan
--
Best regards,
Coiby--
Best regards,
Coiby
Am 27.02.2020 um 12:07 hat Marc-André Lureau geschrieben:
Quoted 30 lines · Kevin Wolf
Quoted 19 lines
Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Quoted 6 lines
But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting
vu_message_readto address the short read issue. In v3 & v4, I extended libvhost-user to allowvu_message_readto be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.Quoted 2 lines
The way your latest patches have it, with a separate read function, works for me.
Done right, I am not against it, fwiw
Quoted 5 lines
You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.
Quoted 7 lines
Quoted 4 lines
The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until
read()returns enough bytes is not acceptable in QEMU.In the context of vhost-user, local unix sockets with short messages (do we have >1k messages?), I am not sure if this is really a problem.
I'm not sure how much of a problem it is in practice, and whether we can consider the vhost-user client trusted. But using QIOChannel from within a coroutine just avoids the problem completely, so it feels like a natural choice to just do that.
Quoted 2 lines
And isn't it possible to run libvhost-user in its own thread for this series?
You need to run the actual block I/O requests in the iothread where the block device happens to run. So if you move the message processing to a separate thread, you would have to communicate between threads for the actual request processing. Possible, but probably slower than necessary and certainly more complex.
Kevin
Quoted 36 lines · Kevin Wolf
Am 27.02.2020 um 12:07 hat Marc-André Lureau geschrieben:
Quoted 35 lines · Kevin Wolf
Quoted 19 lines
Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Quoted 6 lines
But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting
vu_message_readto address the short read issue. In v3 & v4, I extended libvhost-user to allowvu_message_readto be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.Quoted 2 lines
The way your latest patches have it, with a separate read function, works for me.
Done right, I am not against it, fwiw
Quoted 9 lines
You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.
The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until
read()returns enough bytes is not acceptable in QEMU.
Quoted 7 lines
Quoted 2 lines
In the context of vhost-user, local unix sockets with short messages (do we have >1k messages?), I am not sure if this is really a problem.
I'm not sure how much of a problem it is in practice, and whether we can consider the vhost-user client trusted. But using QIOChannel from within a coroutine just avoids the problem completely, so it feels like a natural choice to just do that.
It isn't clear to me that we have a consitent plan for how we intend libvhost-user to develop & what it is permitted to use. What information I see in the source code and in this thread are contradictory.
For example, in the text quoted above:
"libvhost-user is supposed to be indepdent from the main QEMU code."
which did match my overall understanding too. At the top of libvhost-user.c there is a comment
/* this code avoids GLib dependency */
but a few lines later it does
#include "qemu/atomic.h" #include "qemu/osdep.h" #include "qemu/memfd.h"
and in the Makefile we link it to much of QEMU util code:
libvhost-user.a: $(libvhost-user-obj-y) $(util-obj-y) $(stub-obj-y)
this in turn pulls in GLib code, and looking at symbols we can see over 100 GLib functions used:
$ nm ./libvhost-user.a | grep 'U g_' | sort | uniq | wc -l 128
And over 200 QEMU source object files included:
$ nm ./libvhost-user.a | grep '.o:' | sort | wc -l 224
So unless I'm missing something, we have lost any independance from both QEMU and GLib code that we might have had in the past.
Note this also has licensing implications, as I expect this means that via the QEMU source objects it pulls in, libvhost-user.a has become a GPLv2-only combined work, not a GPLv2-or-later combined work.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi
Quoted 75 lines · Daniel P. Berrangé
Quoted 36 lines · Kevin Wolf
Am 27.02.2020 um 12:07 hat Marc-André Lureau geschrieben:
Quoted 35 lines · Kevin Wolf
Quoted 19 lines
Am 27.02.2020 um 11:28 hat Coiby Xu geschrieben:
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
we still need customized
vu_message_readbecause libvhost-user assumes we will always get a full-size VhostUserMsg and hasn't taken care of this short read case. I will improve libvhost-user'svu_message_readby making it keep reading from socket util getting enough bytes. I assume short read is a rare case thus introduced performance penalty would be negligible.In any case, please make sure that we use the QIOChannel functions called from a coroutine in QEMU so that it will never block, but the coroutine can just yield while it's waiting for more bytes.
Quoted 6 lines
But if I am not wrong, libvhost-user is supposed to be indepdent from the main QEMU code. So it can't use the QIOChannel functions if we simply modify exiting
vu_message_readto address the short read issue. In v3 & v4, I extended libvhost-user to allowvu_message_readto be replaced by one which will depend on the main QEMU code. I'm not sure which way is better.Quoted 2 lines
The way your latest patches have it, with a separate read function, works for me.
Done right, I am not against it, fwiw
Quoted 9 lines
You could probably change libvhost-user to reimplement the same functionality, and it might be an improvement for other users of the library, but it's also code duplication and doesn't provide more value in the context of the vhost-user export in QEMU.
The point that's really important to me is just that we never block when we run inside QEMU because that would actually stall the guest. This means busy waiting in a tight loop until
read()returns enough bytes is not acceptable in QEMU.Quoted 7 lines
Quoted 2 lines
In the context of vhost-user, local unix sockets with short messages (do we have >1k messages?), I am not sure if this is really a problem.
I'm not sure how much of a problem it is in practice, and whether we can consider the vhost-user client trusted. But using QIOChannel from within a coroutine just avoids the problem completely, so it feels like a natural choice to just do that.
It isn't clear to me that we have a consitent plan for how we intend libvhost-user to develop & what it is permitted to use. What information I see in the source code and in this thread are contradictory.
For example, in the text quoted above:
"libvhost-user is supposed to be indepdent from the main QEMU code."
which did match my overall understanding too. At the top of
libvhost-user.cthere is a comment/* this code avoids GLib dependency */but a few lines later it does
#include "qemu/atomic.h" #include "qemu/osdep.h" #include "qemu/memfd.h"and in the
Makefilewe link it to much of QEMU util code:libvhost-user.a: $(libvhost-user-obj-y) $(util-obj-y) $(stub-obj-y)this in turn pulls in GLib code, and looking at symbols we can see over 100 GLib functions used:
$ nm ./libvhost-user.a | grep 'U g_' | sort | uniq | wc -l 128
Quoted 7 lines
And over 200 QEMU source object files included:
$ nm ./libvhost-user.a | grep '.o:' | sort | wc -l 224So unless I'm missing something, we have lost any independance from both QEMU and GLib code that we might have had in the past.
Yep, I think this is mostly due to commit 5f9ff1eff38 ("libvhost-user:
Support tracking inflight I/O in shared memory")
It may not be that hard to bring back glib independence. Is it worth it though?
Quoted 5 lines
Note this also has licensing implications, as I expect this means that via the QEMU source objects it pulls in, libvhost-user.a has become a GPLv2-only combined work, not a GPLv2-or-later combined work.
libvhost-user.c is GPLv2-or-later because tests/vhost-user-bridge.c was and is still as well. Do we need to change that?
Quoted 8 lines
Regards,
Daniel
--|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Signature · 1 line
Marc-André Lureau