vhost-user multiple queue support
v7 cover vhost-user multiple queue support
Hi,
Here is the updated patch set for enabling vhost-user multiple queue.
This patch set introduces 2 more vhost user messages: VHOST_USER_GET_QUEUE_NUM, for querying how many queues the backend supports, and VHOST_USER_SET_VRING_FLAG, for enabling/disabling a specific virt queue.
Both of th two new messages are treated as vhost protocol extension, and that's why Michaels's patch "vhost-user: add protocol feature negotiation" is also included here.
Patch 1-5 are all prepare work for actually enabling multiple queue.
Patch 6 is the key patch for enabling multiple queue, which address two major concerns from Michael: no feedback from backend if it can't support # of requested queues, and all messages are sent N time. It also fixes an hidden bug.
Patch 7 introduces the VHOST_USER_SET_VRING_FLAG message.
Note that I haven't done any formal test yet, it just passes build test and basic functional test, such as it does exit when backend doesn't support # of requested queues. Here I sent it out just for more comments, and for avoiding spending too much effort on the wrong track.
Thanks.
--yliu
---
Changchun Ouyang (1):
vhost-user: add a new message to disable/enable a specific virt queue.
Michael S. Tsirkin (1):
vhost-user: add protocol feature negotiation
Ouyang Changchun (1):
vhost-user: add multiple queue support
Yuanhan Liu (4):
vhost-user: use VHOST_USER_XXX macro for switch statement
vhost: rename VHOST_RESET_OWNER to VHOST_RESET_DEVICE
vhost-user: add VHOST_USER_GET_QUEUE_NUM message
vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
docs/specs/vhost-user.txt | 75 +++++++++++++++++++- hw/net/vhost_net.c | 45 +++++++++--- hw/net/virtio-net.c | 2 + hw/virtio/vhost-user.c | 140 ++++++++++++++++++++++++++++++++------ include/hw/virtio/vhost-backend.h | 2 + include/hw/virtio/vhost.h | 2 + include/net/net.h | 1 + include/net/vhost_net.h | 2 + linux-headers/linux/vhost.h | 2 +- net/vhost-user.c | 135 +++++++++++++++++++++++++----------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- tests/vhost-user-test.c | 2 +- 13 files changed, 340 insertions(+), 79 deletions(-)
Signature · 1 line
1.9.0
PATCH v7 1/7 vhost-user: use VHOST_USER_XXX macro for switch statement
So that we could let vhost_user_call to handle extented requests, such as VHOST_USER_GET/SET_PROTOCOL_FEATURES, instead of invoking vhost_user_read/write and constructing the msg again by ourself.
--- hw/virtio/vhost-user.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-)
hw/virtio/vhost-user.c +21 -16
@@ -193,27 +193,32 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); - msg_request =vhost_user_request_translate (request); +/* only translate vhost ioctl requests */ +if (request >VHOST_USER_MAX ) + msg_request =vhost_user_request_translate (request); +else + msg_request = request; + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ; -switch (request) { -case VHOST_GET_FEATURES : +switch (msg_request) { +case VHOST_USER_GET_FEATURES : need_reply =1 ;break ; -case VHOST_SET_FEATURES : -case VHOST_SET_LOG_BASE : +case VHOST_USER_SET_FEATURES : +case VHOST_USER_SET_LOG_BASE : msg.u64 = *((__u64 *) arg); msg.size =sizeof (m.u64);break ; -case VHOST_SET_OWNER : -case VHOST_RESET_OWNER : +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_OWNER :break ; -case VHOST_SET_MEM_TABLE : +case VHOST_USER_SET_MEM_TABLE :for (i =0 ; i < dev->mem->nregions; ++i) {struct vhost_memory_region *reg = dev->mem->regions + i;ram_addr_t ram_addr;
@@ -246,30 +251,30 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,break ; -case VHOST_SET_LOG_FD : +case VHOST_USER_SET_LOG_FD : fds[fd_num++] = *((int *) arg);break ; -case VHOST_SET_VRING_NUM : -case VHOST_SET_VRING_BASE : +case VHOST_USER_SET_VRING_NUM : +case VHOST_USER_SET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); msg.size =sizeof (m.state);break ; -case VHOST_GET_VRING_BASE : +case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); msg.size =sizeof (m.state); need_reply =1 ;break ; -case VHOST_SET_VRING_ADDR : +case VHOST_USER_SET_VRING_ADDR :memcpy (&msg.addr, arg,sizeof (struct vhost_vring_addr )); msg.size =sizeof (m.addr);break ; -case VHOST_SET_VRING_KICK : -case VHOST_SET_VRING_CALL : -case VHOST_SET_VRING_ERR : +case VHOST_USER_SET_VRING_KICK : +case VHOST_USER_SET_VRING_CALL : +case VHOST_USER_SET_VRING_ERR : file = arg; msg.u64 = file->index &VHOST_USER_VRING_IDX_MASK ; msg.size =sizeof (m.u64);
Signature · 1 line
1.9.0
PATCH v7 2/7 vhost-user: add protocol feature negotiation
In reply to Yuanhan LiuFrom: "Michael S. Tsirkin" <[email protected]>
Support a separate bitmask for vhost-user protocol features, and messages to get/set protocol features.
Invoke them at init.
No features are defined yet.
v2: leverage vhost_user_call for request handling -- Yuanhan Liu
--- docs/specs/vhost-user.txt | 37 +++++++++++++++++++++++++++++++++++++ hw/net/vhost_net.c | 2 ++ hw/virtio/vhost-user.c | 28 ++++++++++++++++++++++++++++ include/hw/virtio/vhost.h | 1 + 4 files changed, 68 insertions(+)
docs/specs/vhost-user.txt +37 -0
@@ -113,6 +113,7 @@ message replies. Most of the requests don't require replies. Here is a list of the ones that do: * VHOST_GET_FEATURES + * VHOST_GET_PROTOCOL_FEATURES * VHOST_GET_VRING_BASE There are several messages that the master sends with file descriptors passed
@@ -127,6 +128,13 @@ in the ancillary data: If Master is unable to send the full message or receives a wrong reply it will close the connection. An optional reconnection mechanism can be implemented. +Any protocol extensions are gated by protocol feature bits, +which allows full backwards compatibility on both master +and slave. +As older slaves don't support negotiating protocol features, +a feature bit was dedicated for this purpose: +#define VHOST_USER_F_PROTOCOL_FEATURES 30 + Message types -------------
@@ -138,6 +146,8 @@ Message types Slave payload: u64 Get from the underlying vhost implementation the features bitmask. + Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for + VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES. * VHOST_USER_SET_FEATURES
@@ -146,6 +156,33 @@ Message types Master payload: u64 Enable features in the underlying vhost implementation using a bitmask. + Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for + VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES. + + * VHOST_USER_GET_PROTOCOL_FEATURES + + Id: 15 + Equivalent ioctl: VHOST_GET_FEATURES + Master payload: N/A + Slave payload: u64 + + Get the protocol feature bitmask from the underlying vhost implementation. + Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in + VHOST_USER_GET_FEATURES. + Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support + this message even before VHOST_USER_SET_FEATURES was called. + + * VHOST_USER_SET_PROTOCOL_FEATURES + + Id: 16 + Ioctl: VHOST_SET_FEATURES + Master payload: u64 + + Enable protocol features in the underlying vhost implementation. + Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in + VHOST_USER_GET_FEATURES. + Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support + this message even before VHOST_USER_SET_FEATURES was called. * VHOST_USER_SET_OWNER
hw/net/vhost_net.c +2 -0
@@ -152,8 +152,10 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) net->dev.backend_features =qemu_has_vnet_hdr (options->net_backend) ?0 : (1ULL <<VHOST_NET_F_VIRTIO_NET_HDR ); net->backend = r; + net->dev.protocol_features =0 ; }else { net->dev.backend_features =0 ; + net->dev.protocol_features =0 ; net->backend =-1 ; } net->nc = options->net_backend;
hw/virtio/vhost-user.c +28 -0
@@ -24,6 +24,8 @@#include <linux/vhost.h> #define VHOST_MEMORY_MAX_NREGIONS 8 +#define VHOST_USER_F_PROTOCOL_FEATURES 30 +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x0ULL typedef enum VhostUserRequest { VHOST_USER_NONE = 0,
@@ -41,6 +43,8 @@ typedef enum VhostUserRequest {VHOST_USER_SET_VRING_KICK =12 ,VHOST_USER_SET_VRING_CALL =13 ,VHOST_USER_SET_VRING_ERR =14 , +VHOST_USER_GET_PROTOCOL_FEATURES =15 , +VHOST_USER_SET_PROTOCOL_FEATURES =16 , VHOST_USER_MAX } VhostUserRequest;
@@ -205,11 +209,13 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES : need_reply =1 ;break ;case VHOST_USER_SET_FEATURES :case VHOST_USER_SET_LOG_BASE : +case VHOST_USER_SET_PROTOCOL_FEATURES : msg.u64 = *((__u64 *) arg); msg.size =sizeof (m.u64);break ;
@@ -307,6 +313,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES :if (msg.size !=sizeof (m.u64)) {error_report ("Received bad msg size." );return -1 ;
@@ -332,10 +339,31 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,static int vhost_user_init (struct vhost_dev *dev,void *opaque) { +unsigned long long features; +int err; +assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); dev->opaque = opaque; + err =vhost_user_call (dev,VHOST_USER_GET_FEATURES , &features); +if (err <0 ) +return err; + +if (__virtio_has_feature (features,VHOST_USER_F_PROTOCOL_FEATURES )) { + dev->backend_features |=1ULL <<VHOST_USER_F_PROTOCOL_FEATURES ; + + err =vhost_user_call (dev,VHOST_USER_GET_PROTOCOL_FEATURES , &features); +if (err <0 ) +return err; + + dev->protocol_features = features &VHOST_USER_PROTOCOL_FEATURE_MASK ; + err =vhost_user_call (dev,VHOST_USER_SET_PROTOCOL_FEATURES , + &dev->protocol_features); +if (err <0 ) +return err; + } +return 0 ; }
include/hw/virtio/vhost.h +1 -0
@@ -47,6 +47,7 @@ struct vhost_dev {unsigned long long features;unsigned long long acked_features;unsigned long long backend_features; +unsigned long long protocol_features;bool started;bool log_enabled;unsigned long long log_size;
Signature · 1 line
1.9.0
PATCH v7 3/7 vhost: rename VHOST_RESET_OWNER to VHOST_RESET_DEVICE
In reply to Yuanhan LiuQuote from Michael:
We really should rename VHOST_RESET_OWNER to VHOST_RESET_DEVICE.
--- docs/specs/vhost-user.txt | 4 ++-- hw/net/vhost_net.c | 2 +- hw/virtio/vhost-user.c | 6 +++--- linux-headers/linux/vhost.h | 2 +- tests/vhost-user-test.c | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-)
docs/specs/vhost-user.txt +2 -2
@@ -194,10 +194,10 @@ Message types as an owner of the session. This can be used on the Slave as a "session start" flag. - * VHOST_USER_RESET_OWNER + * VHOST_USER_RESET_DEVICE Id: 4 - Equivalent ioctl: VHOST_RESET_OWNER + Equivalent ioctl: VHOST_RESET_DEVICE Master payload: N/A Issued when a new connection is about to be closed. The Master will no
hw/net/vhost_net.c +1 -1
@@ -287,7 +287,7 @@ static void vhost_net_stop_one(struct vhost_net *net, }else if (net->nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) {for (file.index =0 ; file.index < net->dev.nvqs; ++file.index) {const VhostOps *vhost_ops = net->dev.vhost_ops; -int r = vhost_ops->vhost_call (&net->dev,VHOST_RESET_OWNER , +int r = vhost_ops->vhost_call (&net->dev,VHOST_RESET_DEVICE ,NULL );assert (r >=0 ); }
hw/virtio/vhost-user.c +3 -3
@@ -32,7 +32,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_FEATURES =1 ,VHOST_USER_SET_FEATURES =2 ,VHOST_USER_SET_OWNER =3 , -VHOST_USER_RESET_OWNER =4 , +VHOST_USER_RESET_DEVICE =4 ,VHOST_USER_SET_MEM_TABLE =5 ,VHOST_USER_SET_LOG_BASE =6 , VHOST_USER_SET_LOG_FD = 7,
@@ -98,7 +98,7 @@ static unsigned long int ioctl_to_vhost_user_request[VHOST_USER_MAX] = {VHOST_GET_FEATURES ,/* VHOST_USER_GET_FEATURES */ VHOST_SET_FEATURES ,/* VHOST_USER_SET_FEATURES */ VHOST_SET_OWNER ,/* VHOST_USER_SET_OWNER */ -VHOST_RESET_OWNER ,/* VHOST_USER_RESET_OWNER */ +VHOST_RESET_DEVICE ,/* VHOST_USER_RESET_DEVICE */ VHOST_SET_MEM_TABLE ,/* VHOST_USER_SET_MEM_TABLE */ VHOST_SET_LOG_BASE ,/* VHOST_USER_SET_LOG_BASE */ VHOST_SET_LOG_FD, /* VHOST_USER_SET_LOG_FD */
@@ -221,7 +221,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,break ;case VHOST_USER_SET_OWNER : -case VHOST_USER_RESET_OWNER : +case VHOST_USER_RESET_DEVICE :break ;case VHOST_USER_SET_MEM_TABLE :
linux-headers/linux/vhost.h +1 -1
@@ -78,7 +78,7 @@ struct vhost_memory {#define VHOST_SET_OWNER _IO(VHOST_VIRTIO, 0x01)/* Give up ownership, and reset the device to default values. * Allows subsequent call to VHOST_OWNER_SET to succeed. */ -#define VHOST_RESET_OWNER _IO(VHOST_VIRTIO, 0x02) +#define VHOST_RESET_DEVICE _IO(VHOST_VIRTIO, 0x02)/* Set up/modify memory layout */ #define VHOST_SET_MEM_TABLE _IOW(VHOST_VIRTIO, 0x03, struct vhost_memory)
tests/vhost-user-test.c +1 -1
@@ -58,7 +58,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_FEATURES =1 ,VHOST_USER_SET_FEATURES =2 ,VHOST_USER_SET_OWNER =3 , -VHOST_USER_RESET_OWNER =4 , +VHOST_USER_RESET_DEVICE =4 ,VHOST_USER_SET_MEM_TABLE =5 ,VHOST_USER_SET_LOG_BASE =6 , VHOST_USER_SET_LOG_FD = 7,
Signature · 1 line
1.9.0
PATCH v7 4/7 vhost-user: add VHOST_USER_GET_QUEUE_NUM message
In reply to Yuanhan LiuThis is for querying how many queues the backend supports if it has mq support(when VHOST_USER_PROTOCOL_F_MQ flag is set from the quried protocol features).
vhost_net_get_max_queues() is the interface to export that value, and to tell if the backend supports # of queues user requested, which is done in the following patch.
--- docs/specs/vhost-user.txt | 11 +++++++++++ hw/net/vhost_net.c | 7 +++++++ hw/virtio/vhost-user.c | 14 +++++++++++++- include/hw/virtio/vhost.h | 1 + include/net/vhost_net.h | 1 + 5 files changed, 33 insertions(+), 1 deletion(-)
docs/specs/vhost-user.txt +11 -0
@@ -301,3 +301,14 @@ Message types Bits (0-7) of the payload contain the vring index. Bit 8 is the invalid FD flag. This flag is set when there is no file descriptor in the ancillary data. + + * VHOST_USER_GET_QUEUE_NUM + + Id: 17 + Equivalent ioctl: N/A + Master payload: N/A + Slave payload: u64 + + Query how many queues the backend supports. This request should be + sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol + features by VHOST_USER_GET_PROTOCOL_FEATURES.
hw/net/vhost_net.c +7 -0
@@ -122,6 +122,11 @@ void vhost_net_ack_features(struct vhost_net *net, uint64_t features)vhost_ack_features (&net->dev,vhost_net_get_feature_bits (net), features); } +uint64_t vhost_net_get_max_queues (VHostNetState *net) +{ +return net->dev.max_queues; +} +static int vhost_net_get_fd (NetClientState *backend) {switch (backend->info->type) {
@@ -144,6 +149,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) goto fail; } + net->dev.max_queues =1 ; +if (backend_kernel) { r =vhost_net_get_fd (options->net_backend);if (r <0 ) {
hw/virtio/vhost-user.c +13 -1
@@ -25,7 +25,9 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x0ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL + +#define VHOST_USER_PROTOCOL_F_MQ 0 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,
@@ -45,6 +47,7 @@ typedef enum VhostUserRequest {VHOST_USER_SET_VRING_ERR =14 ,VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 , +VHOST_USER_GET_QUEUE_NUM =17 , VHOST_USER_MAX } VhostUserRequest;
@@ -210,6 +213,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES :case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_GET_QUEUE_NUM : need_reply =1 ;break ;
@@ -314,6 +318,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES :case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_GET_QUEUE_NUM :if (msg.size !=sizeof (m.u64)) {error_report ("Received bad msg size." );return -1 ;
@@ -362,6 +367,13 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque) &dev->protocol_features);if (err <0 )return err; + +/* query the max queues we support if backend supports Multiple Queue */ +if (dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_MQ )) { + err =vhost_user_call (dev,VHOST_USER_GET_QUEUE_NUM , &dev->max_queues); +if (err <0 ) +return err; + } }return 0 ;
include/hw/virtio/vhost.h +1 -0
@@ -48,6 +48,7 @@ struct vhost_dev {unsigned long long acked_features;unsigned long long backend_features;unsigned long long protocol_features; +unsigned long long max_queues;bool started;bool log_enabled;unsigned long long log_size;
include/net/vhost_net.h +1 -0
@@ -13,6 +13,7 @@ typedef struct VhostNetOptions {void *opaque; } VhostNetOptions; +uint64_t vhost_net_get_max_queues (VHostNetState *net);struct vhost_net *vhost_net_init (VhostNetOptions *options);int vhost_net_start (VirtIODevice *dev,NetClientState *ncs,int total_queues);
Signature · 1 line
1.9.0
PATCH v7 5/7 vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
In reply to Yuanhan LiuSo that we could use the `vq_index' as well in the vhost_net_init stage, which is required when adding vhost-user multiple-queue support, where we need the vq_index to indicate which queue pair we are gonna initiate.
vhost-user has no multiple queue support yet, hence no queue_index set before. Here is a quick set to 0 at net_vhost_user_init() stage, and it will be set properly soon in the next patch.
--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)
hw/net/vhost_net.c +7 -9
@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;
@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend; +vhost_net_set_vq_index (net, net->nc->queue_index *2 ); + net->dev.nvqs =2 ; net->dev.vqs = net->vqs;
@@ -196,11 +203,6 @@ fail:return NULL ; } -static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) -{ - net->dev.vq_index = vq_index; -} -static int vhost_net_set_vnet_endian (VirtIODevice *dev,NetClientState *peer,bool set) {
@@ -325,10 +327,6 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, goto err; } -for (i =0 ; i < total_queues; i++) { -vhost_net_set_vq_index (get_vhost_net (ncs[i].peer), i *2 ); - } - r = k->set_guest_notifiers (qbus->parent, total_queues *2 , true);if (r <0 ) {error_report ("Error binding guest notifier: %d" , -r);
net/vhost-user.c +1 -0
@@ -146,6 +146,7 @@ static int net_vhost_user_init(NetClientState *peer, const char *device,/* We don't provide a receive callback */ s->nc.receive_disabled =1 ; s->chr = chr; + nc->queue_index =0 ;qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s);
Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
Quoted 32 lines · Yuanhan Liu
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 5 lines
hw/net/vhost_net.c +2
-0@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend; +vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +
This breaks vhost kernel multiqueue since queue_index was not initialized at this time. We do this in set_netdev() instead of setting it in each kind of netdev.
Quoted 31 lines
hw/net/vhost_net.c +0
-9net->dev.nvqs =2 ; net->dev.vqs = net->vqs;@@ -196,11 +203,6 @@ fail:return NULL ; } -static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) -{ - net->dev.vq_index = vq_index; -} -static int vhost_net_set_vnet_endian (VirtIODevice *dev,NetClientState *peer,bool set) {@@ -325,10 +327,6 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, goto err; } -for (i =0 ; i < total_queues; i++) { -vhost_net_set_vq_index (get_vhost_net (ncs[i].peer), i *2 ); - } - r = k->set_guest_notifiers (qbus->parent, total_queues *2 , true);if (r <0 ) {error_report ("Error binding guest notifier: %d" , -r);net/vhost-user.c +0
-0
net/vhost-user.c static int net_vhost_user_init(NetClientState *peer, const char *device, · line 150
Quoted 5 lines
net/vhost-user.c +1
-0@@ -146,6 +146,7 @@ static int net_vhost_user_init(NetClientState *peer, const char *device,/* We don't provide a receive callback */ s->nc.receive_disabled =1 ; s->chr = chr; + nc->queue_index =0 ;
Fail to understand why this is needed. It will be set to correct value in set_netdev().
Quoted 2 lines
net/vhost-user.c +0
-0qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s);
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
Quoted 38 lines · Jason Wang
Quoted 35 lines · Yuanhan Liu
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend;
Quoted 5 lines
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 2 lines
hw/net/vhost_net.c +2
-0+vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +This breaks vhost kernel multiqueue since
queue_indexwas not initialized at this time.
Right, thanks for pointing it out.
Quoted 2 lines
We do this in
set_netdev()instead of setting
it in each kind of netdev.
Can we move it to net_init_tap() for setting the right queue_index for each nc?
Or, can we call vhost_net_set_vq_index twice, one at vhost_net_init(for vhost-user mq support), another one at vhost_net_start(for vhost kernel mq support)?
Or, do you have better ideas?
Quoted 32 lines
Quoted 32 lines
hw/net/vhost_net.c +0
-9net->dev.nvqs =2 ; net->dev.vqs = net->vqs;@@ -196,11 +203,6 @@ fail:return NULL ; } -static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) -{ - net->dev.vq_index = vq_index; -} -static int vhost_net_set_vnet_endian (VirtIODevice *dev,NetClientState *peer,bool set) {@@ -325,10 +327,6 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, goto err; } -for (i =0 ; i < total_queues; i++) { -vhost_net_set_vq_index (get_vhost_net (ncs[i].peer), i *2 ); - } - r = k->set_guest_notifiers (qbus->parent, total_queues *2 , true);if (r <0 ) {error_report ("Error binding guest notifier: %d" , -r);net/vhost-user.c +0
-0@@ -146,6 +146,7 @@ static int net_vhost_user_init(NetClientState *peer, const char *device,
Quoted 8 lines
net/vhost-user.c static int net_vhost_user_init(NetClientState *peer, const char *device, · line 150
Quoted 5 lines
net/vhost-user.c +1
-0/* We don't provide a receive callback */ s->nc.receive_disabled =1 ; s->chr = chr; + nc->queue_index =0 ;Fail to understand why this is needed. It will be set to correct value in
set_netdev().
Right, it's unnecessary.
BTW, May I ask you a quick question: why set_netdev is invoked after tap init, while it is invoked before vhost-user init?
--yliu
Quoted 3 lines
Quoted 3 lines
net/vhost-user.c +0
-0qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s);
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
Quoted 46 lines · Yuanhan Liu
Quoted 37 lines · Jason Wang
Quoted 35 lines · Yuanhan Liu
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend;Quoted 4 lines
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 2 lines
hw/net/vhost_net.c +2
-0+vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +This breaks vhost kernel multiqueue since
queue_indexwas not initialized at this time.Right, thanks for pointing it out.
Quoted 2 lines
We do this in
set_netdev()instead of setting
it in each kind of netdev.
Quoted 8 lines
Can we move it to
net_init_tap()for setting the rightqueue_indexfor each nc?Or, can we call
vhost_net_set_vq_indextwice, one atvhost_net_init(for vhost-user mq support), another one atvhost_net_start(for vhost kernel mq support)?Or, do you have better ideas?
I think setting queue_index in net_init_tap() looks ok. But a question is that why need we do this at so early stage? ( Even before its peers is connected.)
Quoted 39 lines
Quoted 31 lines
Quoted 31 lines
hw/net/vhost_net.c +0
-9net->dev.nvqs =2 ; net->dev.vqs = net->vqs;@@ -196,11 +203,6 @@ fail:return NULL ; } -static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) -{ - net->dev.vq_index = vq_index; -} -static int vhost_net_set_vnet_endian (VirtIODevice *dev,NetClientState *peer,bool set) {@@ -325,10 +327,6 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, goto err; } -for (i =0 ; i < total_queues; i++) { -vhost_net_set_vq_index (get_vhost_net (ncs[i].peer), i *2 ); - } - r = k->set_guest_notifiers (qbus->parent, total_queues *2 , true);if (r <0 ) {error_report ("Error binding guest notifier: %d" , -r);net/vhost-user.c +0
-0Quoted 8 lines
net/vhost-user.c static int net_vhost_user_init(NetClientState *peer, const char *device, · line 150
Quoted 6 lines
net/vhost-user.c +1
-0@@ -146,6 +146,7 @@ static int net_vhost_user_init(NetClientState *peer, const char *device,/* We don't provide a receive callback */ s->nc.receive_disabled =1 ; s->chr = chr; + nc->queue_index =0 ;Fail to understand why this is needed. It will be set to correct value in
set_netdev().
Quoted 6 lines
Right, it's unnecessary.
BTW, May I ask you a quick question: why
set_netdevis invoked after tap init, while it is invoked before vhost-user init?--yliu
I think the reason is that you must initialize all netdevs before a device tries to connect them.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
net/vhost-user.c +0
-0qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s);
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
Quoted 52 lines · Jason Wang
Quoted 49 lines · Yuanhan Liu
Quoted 37 lines · Jason Wang
Quoted 35 lines · Yuanhan Liu
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend;Quoted 4 lines
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 2 lines
hw/net/vhost_net.c +2
-0+vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +This breaks vhost kernel multiqueue since
queue_indexwas not initialized at this time.Right, thanks for pointing it out.
Quoted 2 lines
We do this in
set_netdev()instead of setting
it in each kind of netdev.Can we move it to
net_init_tap()for setting the rightqueue_indexfor each nc?
Quoted 7 lines
Quoted 5 lines
Or, can we call
vhost_net_set_vq_indextwice, one atvhost_net_init(for vhost-user mq support), another one atvhost_net_start(for vhost kernel mq support)?Or, do you have better ideas?
I think setting
queue_indexinnet_init_tap()looks ok.
Good to know.
Quoted 3 lines
But a question
is that why need we do this at so early stage? ( Even before its peers is connected.)
For vhost-user multiple queues support, we will invoke vhost_net_init() N times for each queue pair, and hence we need to distinguish which queue it is while sending messages like VHOST_SET_VRING_CALL for initializing corresponding queue pair.
Does that make sense to you?
Quoted 42 lines
Quoted 42 lines
Quoted 32 lines
Quoted 32 lines
hw/net/vhost_net.c +0
-9net->dev.nvqs =2 ; net->dev.vqs = net->vqs;@@ -196,11 +203,6 @@ fail:return NULL ; } -static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) -{ - net->dev.vq_index = vq_index; -} -static int vhost_net_set_vnet_endian (VirtIODevice *dev,NetClientState *peer,bool set) {@@ -325,10 +327,6 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, goto err; } -for (i =0 ; i < total_queues; i++) { -vhost_net_set_vq_index (get_vhost_net (ncs[i].peer), i *2 ); - } - r = k->set_guest_notifiers (qbus->parent, total_queues *2 , true);if (r <0 ) {error_report ("Error binding guest notifier: %d" , -r);net/vhost-user.c +0
-0Quoted 8 lines
net/vhost-user.c static int net_vhost_user_init(NetClientState *peer, const char *device, · line 150
Quoted 6 lines
net/vhost-user.c +1
-0@@ -146,6 +146,7 @@ static int net_vhost_user_init(NetClientState *peer, const char *device,/* We don't provide a receive callback */ s->nc.receive_disabled =1 ; s->chr = chr; + nc->queue_index =0 ;Fail to understand why this is needed. It will be set to correct value in
set_netdev().Right, it's unnecessary.
Quoted 7 lines
Quoted 4 lines
BTW, May I ask you a quick question: why
set_netdevis invoked after tap init, while it is invoked before vhost-user init?--yliuI think the reason is that you must initialize all netdevs before a device tries to connect them.
Thanks!
--yliu
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
Quoted 65 lines · Yuanhan Liu
Quoted 52 lines · Jason Wang
Quoted 52 lines · Yuanhan Liu
Quoted 49 lines
Quoted 49 lines · Jason Wang
Quoted 37 lines
Quoted 37 lines · Yuanhan Liu
Quoted 35 lines
Quoted 35 lines
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend;Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 2 lines
hw/net/vhost_net.c +2
-0+vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +This breaks vhost kernel multiqueue since
queue_indexwas not initialized at this time.Right, thanks for pointing it out.
Quoted 2 lines
Quoted 2 lines
We do this in
set_netdev()instead of setting
it in each kind of netdev.Can we move it to
net_init_tap()for setting the rightqueue_indexfor each nc?Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Or, can we call
vhost_net_set_vq_indextwice, one atvhost_net_init(for vhost-user mq support), another one atvhost_net_start(for vhost kernel mq support)?Or, do you have better ideas?
I think setting
queue_indexinnet_init_tap()looks ok.Good to know.
Quoted 3 lines
Quoted 3 lines
But a question
is that why need we do this at so early stage? ( Even before its peers is connected.)
Quoted 7 lines
For vhost-user multiple queues support, we will invoke
vhost_net_init()N times for each queue pair, and hence we need to distinguish which queue it is while sending messages likeVHOST_SET_VRING_CALLfor initializing corresponding queue pair.Does that make sense to you?
Not sure. Since current codes works for vhost-kernel. (vhost_net_init() was also called N times). We don't want to break existed vhost-kernel API when developing multiqueue. For each virtqueue TX/RX pair, we have one vhost net device and it has no knowledge for the others (which was hide by qemu). So VHOST_SET_VRING_CALL works without any change here.
For the case here, since you still have multiple instances of vhost_net structure. Maybe the vhost-user backend can distinguish form this?
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
Quoted 76 lines · Jason Wang
Quoted 65 lines · Yuanhan Liu
Quoted 52 lines · Jason Wang
Quoted 52 lines · Yuanhan Liu
Quoted 49 lines
Quoted 49 lines · Jason Wang
Quoted 37 lines
Quoted 37 lines · Yuanhan Liu
Quoted 35 lines
Quoted 35 lines
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend;Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 2 lines
hw/net/vhost_net.c +2
-0+vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +This breaks vhost kernel multiqueue since
queue_indexwas not initialized at this time.Right, thanks for pointing it out.
Quoted 2 lines
Quoted 2 lines
We do this in
set_netdev()instead of setting
it in each kind of netdev.Can we move it to
net_init_tap()for setting the rightqueue_indexfor each nc?Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Or, can we call
vhost_net_set_vq_indextwice, one atvhost_net_init(for vhost-user mq support), another one atvhost_net_start(for vhost kernel mq support)?Or, do you have better ideas?
I think setting
queue_indexinnet_init_tap()looks ok.Good to know.
Quoted 3 lines
Quoted 3 lines
But a question
is that why need we do this at so early stage? ( Even before its peers is connected.)Quoted 7 lines
For vhost-user multiple queues support, we will invoke
vhost_net_init()N times for each queue pair, and hence we need to distinguish which queue it is while sending messages likeVHOST_SET_VRING_CALLfor initializing corresponding queue pair.Does that make sense to you?
Quoted 8 lines
Not sure. Since current codes works for vhost-kernel. (
vhost_net_init()was also called N times). We don't want to break existed vhost-kernel API when developing multiqueue. For each virtqueue TX/RX pair, we have one vhost net device and it has no knowledge for the others (which was hide by qemu). SoVHOST_SET_VRING_CALLworks without any change here.For the case here, since you still have multiple instances of
vhost_netstructure. Maybe the vhost-user backend can distinguish form this?
Yeah, I guess that's the difference between vhost-user and vhost-kernel. Vhost-kernel opens a char device(/dev/vhost-net) for each vhost_dev, hence it's distinguishable. But for vhost-user, all vhost_dev share one char device(a socket) for communication, hence, it's not distinguishable.
I was thinking maybe we could export vhost_net_set_vq_index() and invoke it at net/vhost-user.c, so that we break nothing, and in the meantime, it keeps the logic inside vhost-user.
What do you think?
--yliu
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
Quoted 78 lines · Yuanhan Liu
Quoted 77 lines · Jason Wang
Quoted 62 lines · Yuanhan Liu
Quoted 50 lines · Jason Wang
Quoted 50 lines · Yuanhan Liu
Quoted 48 lines
Quoted 48 lines · Jason Wang
Quoted 36 lines
Quoted 36 lines · Yuanhan Liu
Quoted 35 lines
Quoted 35 lines
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend;Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 2 lines
hw/net/vhost_net.c +2
-0+vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +This breaks vhost kernel multiqueue since
queue_indexwas not initialized at this time.Right, thanks for pointing it out.
Quoted 2 lines
Quoted 2 lines
We do this in
set_netdev()instead of setting
it in each kind of netdev.Can we move it to
net_init_tap()for setting the rightqueue_indexfor each nc?Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Or, can we call
vhost_net_set_vq_indextwice, one atvhost_net_init(for vhost-user mq support), another one atvhost_net_start(for vhost kernel mq support)?Or, do you have better ideas?
I think setting
queue_indexinnet_init_tap()looks ok.Good to know.
Quoted 3 lines
Quoted 3 lines
But a question
is that why need we do this at so early stage? ( Even before its peers is connected.)Quoted 7 lines
For vhost-user multiple queues support, we will invoke
vhost_net_init()N times for each queue pair, and hence we need to distinguish which queue it is while sending messages likeVHOST_SET_VRING_CALLfor initializing corresponding queue pair.Does that make sense to you?
Not sure. Since current codes works for vhost-kernel. (
vhost_net_init()was also called N times). We don't want to break existed vhost-kernel API when developing multiqueue. For each virtqueue TX/RX pair, we have one vhost net device and it has no knowledge for the others (which was hide by qemu). SoVHOST_SET_VRING_CALLworks without any change here.
Quoted 6 lines
Quoted 2 lines
For the case here, since you still have multiple instances of
vhost_netstructure. Maybe the vhost-user backend can distinguish form this?Yeah, I guess that's the difference between vhost-user and vhost-kernel. Vhost-kernel opens a char device(/dev/vhost-net) for each
vhost_dev, hence it's distinguishable. But for vhost-user, allvhost_devshare one char device(a socket) for communication, hence, it's not distinguishable.
How about using individual socket in this case? This seems can also minimize the changes of backend.
Quoted 9 lines
I was thinking maybe we could export
vhost_net_set_vq_index()and invoke it atnet/vhost-user.c, so that we break nothing, and in the meantime, it keeps the logic inside vhost-user.What do you think?
--yliu
Sounds work. Then I believe you will need to set queue_index in vhost_user initialization code?
Thanks
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
Quoted 83 lines · Jason Wang
Quoted 80 lines · Yuanhan Liu
Quoted 79 lines · Jason Wang
Quoted 62 lines · Yuanhan Liu
Quoted 50 lines · Jason Wang
Quoted 50 lines · Yuanhan Liu
Quoted 48 lines
Quoted 48 lines · Jason Wang
Quoted 36 lines
Quoted 36 lines · Yuanhan Liu
Quoted 35 lines
Quoted 35 lines
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend;Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 2 lines
hw/net/vhost_net.c +2
-0+vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +This breaks vhost kernel multiqueue since
queue_indexwas not initialized at this time.Right, thanks for pointing it out.
Quoted 2 lines
Quoted 2 lines
We do this in
set_netdev()instead of setting
it in each kind of netdev.Can we move it to
net_init_tap()for setting the rightqueue_indexfor each nc?Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Or, can we call
vhost_net_set_vq_indextwice, one atvhost_net_init(for vhost-user mq support), another one atvhost_net_start(for vhost kernel mq support)?Or, do you have better ideas?
I think setting
queue_indexinnet_init_tap()looks ok.Good to know.
Quoted 3 lines
Quoted 3 lines
But a question
is that why need we do this at so early stage? ( Even before its peers is connected.)Quoted 7 lines
For vhost-user multiple queues support, we will invoke
vhost_net_init()N times for each queue pair, and hence we need to distinguish which queue it is while sending messages likeVHOST_SET_VRING_CALLfor initializing corresponding queue pair.Does that make sense to you?
Not sure. Since current codes works for vhost-kernel. (
vhost_net_init()was also called N times). We don't want to break existed vhost-kernel API when developing multiqueue. For each virtqueue TX/RX pair, we have one vhost net device and it has no knowledge for the others (which was hide by qemu). SoVHOST_SET_VRING_CALLworks without any change here.For the case here, since you still have multiple instances of
vhost_netstructure. Maybe the vhost-user backend can distinguish form this?
Quoted 7 lines
Quoted 4 lines
Yeah, I guess that's the difference between vhost-user and vhost-kernel. Vhost-kernel opens a char device(/dev/vhost-net) for each
vhost_dev, hence it's distinguishable. But for vhost-user, allvhost_devshare one char device(a socket) for communication, hence, it's not distinguishable.How about using individual socket in this case? This seems can also minimize the changes of backend.
I doubt that will work, for the socket is given at cmd line by "-chardev" option. You can image that it's not flexible when you want to have 10 queues, or more.
Quoted 6 lines
Quoted 6 lines
I was thinking maybe we could export
vhost_net_set_vq_index()and invoke it atnet/vhost-user.c, so that we break nothing, and in the meantime, it keeps the logic inside vhost-user.
Quoted 6 lines
Quoted 4 lines
What do you think?
--yliuSounds work.
Good.
Quoted 2 lines
Then I believe you will need to set
queue_indexinvhost_userinitialization code?
Yes. I then will go this way if no objection.
--yliu
Re: [Qemu-devel] [PATCH 5/7] vhost_net: move vhost_net_set_vq_index ahead at vhost_net_init
In reply to Jason WangQuoted 96 lines · Jason Wang
Quoted 78 lines · Yuanhan Liu
Quoted 77 lines · Jason Wang
Quoted 62 lines · Yuanhan Liu
Quoted 50 lines · Jason Wang
Quoted 50 lines · Yuanhan Liu
Quoted 48 lines
Quoted 48 lines · Jason Wang
Quoted 36 lines
Quoted 36 lines · Yuanhan Liu
Quoted 35 lines
Quoted 35 lines
So that we could use the `vq_index' as well in the
vhost_net_initstage, which is required when adding vhost-user multiple-queue support, where we need thevq_indexto indicate which queue pair we are gonna initiate.vhost-user has no multiple queue support yet, hence no
queue_indexset before. Here is a quick set to 0 atnet_vhost_user_init()stage, and it will be set properly soon in the next patch.Signed-off-by Yuanhan Liu <[email protected]>--- hw/net/vhost_net.c | 16 +++++++--------- net/vhost-user.c | 1 + 2 files changed, 8 insertions(+), 9 deletions(-)hw/net/vhost_net.c +5
-0@@ -138,6 +138,11 @@ static int vhost_net_get_fd(NetClientState *backend) } } +static void vhost_net_set_vq_index (struct vhost_net *net,int vq_index) +{ + net->dev.vq_index = vq_index; +} +struct vhost_net *vhost_net_init (VhostNetOptions *options) {int r;@@ -167,6 +172,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) } net->nc = options->net_backend;Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
hw/net/vhost_net.c struct vhost_net *vhost_net_init(VhostNetOptions *options) · line 176
Quoted 2 lines
hw/net/vhost_net.c +2
-0+vhost_net_set_vq_index (net, net->nc->queue_index *2 ); +This breaks vhost kernel multiqueue since
queue_indexwas not initialized at this time.Right, thanks for pointing it out.
Quoted 2 lines
Quoted 2 lines
We do this in
set_netdev()instead of setting
it in each kind of netdev.Can we move it to
net_init_tap()for setting the rightqueue_indexfor each nc?Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Or, can we call
vhost_net_set_vq_indextwice, one atvhost_net_init(for vhost-user mq support), another one atvhost_net_start(for vhost kernel mq support)?Or, do you have better ideas?
I think setting
queue_indexinnet_init_tap()looks ok.Good to know.
Quoted 3 lines
Quoted 3 lines
But a question
is that why need we do this at so early stage? ( Even before its peers is connected.)Quoted 7 lines
For vhost-user multiple queues support, we will invoke
vhost_net_init()N times for each queue pair, and hence we need to distinguish which queue it is while sending messages likeVHOST_SET_VRING_CALLfor initializing corresponding queue pair.Does that make sense to you?
Not sure. Since current codes works for vhost-kernel. (
vhost_net_init()was also called N times). We don't want to break existed vhost-kernel API when developing multiqueue. For each virtqueue TX/RX pair, we have one vhost net device and it has no knowledge for the others (which was hide by qemu). SoVHOST_SET_VRING_CALLworks without any change here.Quoted 6 lines
Quoted 2 lines
For the case here, since you still have multiple instances of
vhost_netstructure. Maybe the vhost-user backend can distinguish form this?Yeah, I guess that's the difference between vhost-user and vhost-kernel. Vhost-kernel opens a char device(/dev/vhost-net) for each
vhost_dev, hence it's distinguishable. But for vhost-user, allvhost_devshare one char device(a socket) for communication, hence, it's not distinguishable.How about using individual socket in this case? This seems can also minimize the changes of backend.
Quoted 5 lines
I was thinking maybe we could export
vhost_net_set_vq_index()and invoke it atnet/vhost-user.c, so that we break nothing, and in the meantime, it keeps the logic inside vhost-user.
Quoted 7 lines
Quoted 4 lines
What do you think?
--yliuSounds work. Then I believe you will need to set
queue_indexinvhost_userinitialization code?
Nah, it will not work, as vhost_net_set_vq_index() needs a vhost_net as it's parameter: you can't do that before vhost_net_init(), but it's useless to do that after vhost_net_init(). My bad for not being aware it in the first time.
How about following then?
Thanks.
--yliu
---
hw/net/vhost_net.c +4 -1
@@ -148,6 +148,7 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options)fprintf (stderr,"vhost-net requires net backend to be setup\n" ); goto fail; } + net->nc = options->net_backend; net->dev.max_queues =1 ;
@@ -164,8 +165,10 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) net->dev.backend_features =0 ; net->dev.protocol_features =0 ; net->backend =-1 ; + +/* vhost-user needs vq_index to initiate a specific queue pair */ + net->dev.vq_index = net->nc->queue_index *2 ; } - net->nc = options->net_backend; net->dev.nvqs =2 ; net->dev.vqs = net->vqs;
PATCH v7 6/7 vhost-user: add multiple queue support
In reply to Yuanhan LiuFrom: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and vhost_net pair for each queue.
What differs from last version is that this patch addresses two major concerns from Michael, and fixes one hidden bug.
- Concern #1: no feedback when the backend can't support # of requested queues(by providing queues=# option).
Here we address this issue by querying VHOST_USER_PROTOCOL_F_MQ protocol features first, if not set, it means the backend don't support mq feature, and let max_queues be 1. Otherwise, we send another message, VHOST_USER_GET_QUEUE_NUM, for getting the max_queues the backend supports.
At vhost-user initiation stage(net_vhost_user_start), we then initiate one queue first, which, in the meantime, also gets the max_queues. We then do a simple compare: if requested_queues > max_queues, we exit(I guess it's safe to exit here, as the vm is not running yet).
- Concern #2: some messages are sent more times than necessary.
We came an agreement with Michael that we could categorize vhost user messages to 2 types: none-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function vhost_user_one_time_request(), which lists following messages as none-vring specific messages:
VHOST_USER_GET_FEATURES
VHOST_USER_SET_FEATURES
VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUM
For above messages, we simply ignore them when they are not sent the first time.
I also observed a hidden bug from last version. We register the char dev event handler N times, which is not necessary, as well as buggy: A later register overwrites the former one, as qemu_chr_add_handlers() will not chain those handlers, but instead overwrites the old one. So, in theory, invoking qemu_chr_add_handlers N times will not end up with calling the handler N times.
However, the reason the handler is invoked N times is because we start the backend(the connection server) first, and hence when net_vhost_user_init() is executed, the connection is already established, and qemu_chr_add_handlers() then invoke the handler immediately, which just looks like we invoke the handler(net_vhost_user_event) directly from net_vhost_user_init().
The solution I came up with is to make VhostUserState as an upper level structure, making it includes N nc and vhost_net pairs:
struct VhostUserNetPeer {
NetClientState *nc;
VHostNetState *vhost_net;
};
typedef struct VhostUserState {
CharDriverState *chr;
bool running;
int queues;
struct VhostUserNetPeer peers[];
} VhostUserState;--- docs/specs/vhost-user.txt | 13 +++++ hw/virtio/vhost-user.c | 31 ++++++++++- include/net/net.h | 1 + net/vhost-user.c | 136 ++++++++++++++++++++++++++++++++-------------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- 6 files changed, 146 insertions(+), 46 deletions(-)
docs/specs/vhost-user.txt +13 -0
@@ -135,6 +135,19 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +------------------- +Multiple queue is treated as a protocal extension, hence the slave has to +implement protocol features first. Multiple queues is supported only when +the protocol feature VHOST_USER_PROTOCOL_F_MQ(bit 0) is set. + +The max # of queues the slave support can be queried with message +VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the # of requested +queues is bigger than that. + +As all queues share one connection, the master use a unique index for each +queue in the sent message to identify one specified queue. + Message types -------------
hw/virtio/vhost-user.c +30 -1
@@ -187,6 +187,23 @@ static int vhost_user_write(struct vhost_dev *dev, VhostUserMsg *msg, 0 : -1; } +static bool vhost_user_one_time_request (VhostUserRequest request) +{ +switch (request) { +case VHOST_USER_GET_FEATURES : +case VHOST_USER_SET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_SET_PROTOCOL_FEATURES : +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_DEVICE : +case VHOST_USER_SET_MEM_TABLE : +case VHOST_USER_GET_QUEUE_NUM : +return true; +default : +return false; + } +} +static int vhost_user_call (struct vhost_dev *dev,unsigned long int request,void *arg) {
@@ -206,6 +223,14 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,else msg_request = request; +/* +* For none-vring specific requests, like VHOST_USER_GET_FEATURES, +* we just need send it once in the first time. For later such +* request, we just ignore it. +*/ +if (vhost_user_one_time_request (msg_request) && dev->vq_index !=0 ) +return 0 ; + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ;
@@ -268,17 +293,20 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_NUM :case VHOST_USER_SET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state); need_reply =1 ;break ;case VHOST_USER_SET_VRING_ADDR :memcpy (&msg.addr, arg,sizeof (struct vhost_vring_addr )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.addr);break ;
@@ -286,7 +314,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_CALL :case VHOST_USER_SET_VRING_ERR : file = arg; - msg.u64 = file->index &VHOST_USER_VRING_IDX_MASK ; + msg.u64 = (file->index + dev->vq_index) &VHOST_USER_VRING_IDX_MASK ; msg.size =sizeof (m.u64);if (ioeventfd_enabled () && file->fd >0 ) { fds[fd_num++] = file->fd;
@@ -330,6 +358,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,error_report ("Received bad msg size." );return -1 ; } + msg.state.index -= dev->vq_index;memcpy (arg, &msg.state,sizeof (struct vhost_vring_state ));break ;default :
include/net/net.h +1 -0
@@ -92,6 +92,7 @@ struct NetClientState {NetClientDestructor *destructor;unsigned int queue_index; unsigned rxfilter_notify_enabled:1; +void *opaque; }; typedef struct NICState {
net/vhost-user.c +94 -42
222 lines not on this page · raw message
qapi-schema.json +5 -1
@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } } ## # @NetClientOptions
qemu-options.hx +3 -2
@@ -1963,13 +1963,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single netdev. @code{-net} and @code{-device} with parameter @option{vlan} create the required hub automatically. -@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off] +@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n] Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should be a unix domain socket backed one. The vhost-user uses a specifically defined protocol to pass vhost ioctl replacement messages to an application on the other end of the socket. On non-MSIX guests, the feature can be forced with -@var{vhostforce}. +@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to +be created for multiqueue vhost-user. Example: @example
Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
Quoted 7 lines · Yuanhan Liu
From: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.
Reviewing grammar and interface only:
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 140
Quoted 8 lines
+++
b/docs/specs/vhost-user.txtdocs/specs/vhost-user.txt +3
-0@@ -135,6 +135,19 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +------------------- +Multiple queue is treated as a protocal extension, hence the slave has to
s/protocal/protocol/
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 144
Quoted 4 lines
docs/specs/vhost-user.txt +4
-0+implement protocol features first. Multiple queues is supported only when +the protocol feature VHOST_USER_PROTOCOL_F_MQ(bit 0) is set. + +The max # of queues the slave support can be queried with message
s/#/number/
s/support/supports/
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 145
Quoted 1 line
docs/specs/vhost-user.txt +1
-0+VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the # of requested
s/#/number/
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 148
Quoted 3 lines
docs/specs/vhost-user.txt +3
-0+queues is bigger than that. + +As all queues share one connection, the master use a unique index for each
s/use/uses/
Quoted 9 lines
docs/specs/vhost-user.txt +4
-0+queue in the sent message to identify one specified queue. + +++ b/qapi-schema.json@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user
docs/specs/vhost-user.txt line 2492
Quoted 10 lines
docs/specs/vhost-user.txt +4
-1+# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } }
Looks okay.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
Quoted 10 lines · Eric Blake
Quoted 7 lines · Yuanhan Liu
From: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.Reviewing grammar and interface only:
Thanks, and will fix them in next patch.
--yliu
Quoted 56 lines
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 140
Quoted 9 lines
+++
b/docs/specs/vhost-user.txtdocs/specs/vhost-user.txt +3
-0@@ -135,6 +135,19 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +------------------- +Multiple queue is treated as a protocal extension, hence the slave has tos/protocal/protocol/
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 144
Quoted 4 lines
docs/specs/vhost-user.txt +4
-0+implement protocol features first. Multiple queues is supported only when +the protocol feature VHOST_USER_PROTOCOL_F_MQ(bit 0) is set. + +The max # of queues the slave support can be queried with messages/#/number/
s/support/supports/docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 145
Quoted 1 line
docs/specs/vhost-user.txt +1
-0+VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the # of requesteds/#/number/
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 148
Quoted 3 lines
docs/specs/vhost-user.txt +3
-0+queues is bigger than that. + +As all queues share one connection, the master use a unique index for eachs/use/uses/
Quoted 9 lines
docs/specs/vhost-user.txt +4
-0+queue in the sent message to identify one specified queue. + +++ b/qapi-schema.json@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-userdocs/specs/vhost-user.txt line 2492
Quoted 10 lines
docs/specs/vhost-user.txt +4
-1+# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } }Looks okay.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
In reply to Yuanhan LiuQuoted 28 lines
-----Original Message-----
From: Yuanhan Liu [mailto:[email protected]]
Sent: Tuesday, September 8, 2015 3:39 PM
To: [email protected]
Cc: [email protected]; Ouyang, Changchun; Yuanhan Liu
Subject: [PATCH 6/7] vhost-user: add multiple queue supportFrom: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.static int vhost_user_start(VhostUserState *s) { VhostNetOptions options;net/vhost-user.c +4
-2+VHostNetState *vhost_net; +int max_queues; +int i =0 ; -if (vhost_user_running (s)) { +if (s->running)return 0 ; - }
net/vhost-user.c typedef struct VhostUserChardevProps { · line 60
Quoted 10 lines
net/vhost-user.c +5
-2options.backend_type =VHOST_BACKEND_TYPE_USER ; - options.net_backend = &s->nc; options.opaque = s->chr; - s->vhost_net =vhost_net_init (&options); + options.net_backend = s->peers[i].nc; + vhost_net = s->peers[i++].vhost_net =vhost_net_init (&options); + + max_queues =vhost_net_get_max_queues (vhost_net); +if (s->queues >= max_queues) {
use '>' rather than '>=' here?
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
Quoted 32 lines · Changchun
Quoted 32 lines
-----Original Message-----
From: Yuanhan Liu [mailto:[email protected]]
Sent: Tuesday, September 8, 2015 3:39 PM
To: [email protected]
Cc: [email protected]; Ouyang, Changchun; Yuanhan Liu
Subject: [PATCH 6/7] vhost-user: add multiple queue supportFrom: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.static int vhost_user_start(VhostUserState *s) { VhostNetOptions options;net/vhost-user.c +4
-3+VHostNetState *vhost_net; +int max_queues; +int i =0 ; -if (vhost_user_running (s)) { +if (s->running)return 0 ; - } options.backend_type =VHOST_BACKEND_TYPE_USER ; - options.net_backend = &s->nc; options.opaque = s->chr;
Quoted 8 lines
net/vhost-user.c typedef struct VhostUserChardevProps { · line 60
Quoted 6 lines
net/vhost-user.c +5
-1- s->vhost_net =vhost_net_init (&options); + options.net_backend = s->peers[i].nc; + vhost_net = s->peers[i++].vhost_net =vhost_net_init (&options); + + max_queues =vhost_net_get_max_queues (vhost_net); +if (s->queues >= max_queues) {use '>' rather than '>=' here?
Right, and thanks!
--yliu
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
In reply to Yuanhan LiuQuoted 97 lines · Yuanhan Liu
From: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.What differs from last version is that this patch addresses two major concerns from Michael, and fixes one hidden bug.
- Concern #1: no feedback when the backend can't support # of requested queues(by providing queues=# option).
Here we address this issue by querying
VHOST_USER_PROTOCOL_F_MQprotocol features first, if not set, it means the backend don't support mq feature, and letmax_queuesbe 1. Otherwise, we send another message,VHOST_USER_GET_QUEUE_NUM, for getting themax_queuesthe backend supports.At vhost-user initiation
stage(net_vhost_user_start), we then initiate one queue first, which, in the meantime, also gets themax_queues. We then do a simple compare: ifrequested_queues>max_queues, weexit(I guess it's safe to exit here, as the vm is not running yet).- Concern #2: some messages are sent more times than necessary.
We came an agreement with Michael that we could categorize vhost user messages to 2 types: none-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function
vhost_user_one_time_request(), which lists following messages as none-vring specific messages:
VHOST_USER_GET_FEATURES
VHOST_USER_SET_FEATURES
VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUMFor above messages, we simply ignore them when they are not sent the first time.
I also observed a hidden bug from last version. We register the char dev event handler N times, which is not necessary, as well as buggy: A later register overwrites the former one, as
qemu_chr_add_handlers()will not chain those handlers, but instead overwrites the old one. So, in theory, invokingqemu_chr_add_handlersN times will not end up with calling the handler N times.However, the reason the handler is invoked N times is because we start the backend(the connection server) first, and hence when
net_vhost_user_init()is executed, the connection is already established, andqemu_chr_add_handlers()then invoke the handler immediately, which just looks like we invoke thehandler(net_vhost_user_event)directly fromnet_vhost_user_init().The solution I came up with is to make VhostUserState as an upper level structure, making it includes N nc and
vhost_netpairs:struct VhostUserNetPeer { NetClientState *nc; VHostNetState *vhost_net; }; typedef struct VhostUserState { CharDriverState *chr; bool running; int queues; struct VhostUserNetPeer peers[]; } VhostUserState;Signed-off-by Nikolay Nikolaev <[email protected]>Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>--- docs/specs/vhost-user.txt | 13 +++++ hw/virtio/vhost-user.c | 31 ++++++++++- include/net/net.h | 1 + net/vhost-user.c | 136 ++++++++++++++++++++++++++++++++-------------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- 6 files changed, 146 insertions(+), 46 deletions(-)docs/specs/vhost-user.txt +3
-0@@ -135,6 +135,19 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +------------------- +Multiple queue is treated as a protocal extension, hence the slave has to
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 150
Quoted 10 lines
docs/specs/vhost-user.txt +10
-0+implement protocol features first. Multiple queues is supported only when +the protocol feature VHOST_USER_PROTOCOL_F_MQ(bit 0) is set. + +The max # of queues the slave support can be queried with message +VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the # of requested +queues is bigger than that. + +As all queues share one connection, the master use a unique index for each +queue in the sent message to identify one specified queue. +
Please also document that only 2 queues are enabled initially. More queues are enabled dynamically.
To enable more queues, the new message needs to be sent (added by patch 7).
Quoted 376 lines
docs/specs/vhost-user.txt +0
-0Message types -------------hw/virtio/vhost-user.c +30
-1@@ -187,6 +187,23 @@ static int vhost_user_write(struct vhost_dev *dev, VhostUserMsg *msg, 0 : -1; } +static bool vhost_user_one_time_request (VhostUserRequest request) +{ +switch (request) { +case VHOST_USER_GET_FEATURES : +case VHOST_USER_SET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_SET_PROTOCOL_FEATURES : +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_DEVICE : +case VHOST_USER_SET_MEM_TABLE : +case VHOST_USER_GET_QUEUE_NUM : +return true; +default : +return false; + } +} +static int vhost_user_call (struct vhost_dev *dev,unsigned long int request,void *arg) {@@ -206,6 +223,14 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,else msg_request = request; +/* +* For none-vring specific requests, like VHOST_USER_GET_FEATURES, +* we just need send it once in the first time. For later such +* request, we just ignore it. +*/ +if (vhost_user_one_time_request (msg_request) && dev->vq_index !=0 ) +return 0 ; + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ;@@ -268,17 +293,20 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_NUM :case VHOST_USER_SET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state); need_reply =1 ;break ;case VHOST_USER_SET_VRING_ADDR :memcpy (&msg.addr, arg,sizeof (struct vhost_vring_addr )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.addr);break ;@@ -286,7 +314,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_CALL :case VHOST_USER_SET_VRING_ERR : file = arg; - msg.u64 = file->index &VHOST_USER_VRING_IDX_MASK ; + msg.u64 = (file->index + dev->vq_index) &VHOST_USER_VRING_IDX_MASK ; msg.size =sizeof (m.u64);if (ioeventfd_enabled () && file->fd >0 ) { fds[fd_num++] = file->fd;@@ -330,6 +358,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,error_report ("Received bad msg size." );return -1 ; } + msg.state.index -= dev->vq_index;memcpy (arg, &msg.state,sizeof (struct vhost_vring_state ));break ;default :include/net/net.h +1
-0@@ -92,6 +92,7 @@ struct NetClientState {NetClientDestructor *destructor;unsigned int queue_index; unsigned rxfilter_notify_enabled:1; +void *opaque; }; typedef struct NICState {net/vhost-user.c +94
-42@@ -15,10 +15,16 @@#include "qemu/config-file.h" #include "qemu/error-report.h" +struct VhostUserNetPeer { +NetClientState *nc; +VHostNetState *vhost_net; +}; +typedef struct VhostUserState { -NetClientState nc;CharDriverState *chr; -VHostNetState *vhost_net; +bool running; +int queues; +struct VhostUserNetPeer peers[]; }VhostUserState ; typedef struct VhostUserChardevProps {@@ -29,48 +35,75 @@ typedef struct VhostUserChardevProps {VHostNetState *vhost_user_get_vhost_net (NetClientState *nc) { -VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); +VhostUserState *s = nc->opaque;assert (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); -return s->vhost_net; -} - -static int vhost_user_running (VhostUserState *s) -{ -return (s->vhost_net) ?1 :0 ; +return s->peers[nc->queue_index].vhost_net; }static int vhost_user_start (VhostUserState *s) {VhostNetOptions options; +VHostNetState *vhost_net; +int max_queues; +int i =0 ; -if (vhost_user_running (s)) { +if (s->running)return 0 ; - } options.backend_type =VHOST_BACKEND_TYPE_USER ; - options.net_backend = &s->nc; options.opaque = s->chr; - s->vhost_net =vhost_net_init (&options); + options.net_backend = s->peers[i].nc; + vhost_net = s->peers[i++].vhost_net =vhost_net_init (&options); + + max_queues =vhost_net_get_max_queues (vhost_net); +if (s->queues >= max_queues) { +error_report ("you are asking more queues than supported: %d\n" , + max_queues); +return -1 ; + } + +for (; i < s->queues; i++) { + options.net_backend = s->peers[i].nc; + + s->peers[i].vhost_net =vhost_net_init (&options); +if (!s->peers[i].vhost_net) +return -1 ; + } + s->running = true; -return vhost_user_running (s) ?0 :-1 ; +return 0 ; }static void vhost_user_stop (VhostUserState *s) { -if (vhost_user_running (s)) { -vhost_net_cleanup (s->vhost_net); +int i; +VHostNetState *vhost_net; + +if (!s->running) +return ; + +for (i =0 ; i < s->queues; i++) { + vhost_net = s->peers[i].vhost_net; +if (vhost_net) +vhost_net_cleanup (vhost_net); } - s->vhost_net =0 ; + s->running = false; }static void vhost_user_cleanup (NetClientState *nc) { -VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); +VhostUserState *s = nc->opaque; +VHostNetState *vhost_net = s->peers[nc->queue_index].vhost_net; + +if (vhost_net) +vhost_net_cleanup (vhost_net); -vhost_user_stop (s);qemu_purge_queued_packets (nc); + +if (nc->queue_index == s->queues -1 ) +free (s); }static bool vhost_user_has_vnet_hdr (NetClientState *nc)@@ -89,7 +122,7 @@ static bool vhost_user_has_ufo(NetClientState *nc)static NetClientInfo net_vhost_user_info = { .type =NET_CLIENT_OPTIONS_KIND_VHOST_USER , - .size =sizeof (VhostUserState), + .size =sizeof (NetClientState), .cleanup = vhost_user_cleanup, .has_vnet_hdr = vhost_user_has_vnet_hdr, .has_ufo = vhost_user_has_ufo,@@ -97,18 +130,25 @@ static NetClientInfo net_vhost_user_info = {static void net_vhost_link_down (VhostUserState *s,bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < s->queues; i++) { + nc = s->peers[i].nc; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } -if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->info->link_status_changed) { + nc->info->link_status_changed (nc); + } + +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }@@ -118,7 +158,8 @@ static void net_vhost_user_event(void *opaque, int event)switch (event) {case CHR_EVENT_OPENED : -vhost_user_start (s); +if (vhost_user_start (s) <0 ) +exit (1 );net_vhost_link_down (s, false);error_report ("chardev \"%s\" went up" , s->chr->label);break ;@@ -131,24 +172,28 @@ static void net_vhost_user_event(void *opaque, int event) }static int net_vhost_user_init (NetClientState *peer,const char *device, -const char *name,CharDriverState *chr) +const char *name,VhostUserState *s) {NetClientState *nc; -VhostUserState *s; +CharDriverState *chr = s->chr; +int i; - nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); +for (i =0 ; i < s->queues; i++) { + nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); -snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user to %s" , - chr->label); +snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user%d to %s" , + i, chr->label); - s =DO_UPCAST (VhostUserState, nc, nc); +/* We don't provide a receive callback */ + nc->receive_disabled =1 ; -/* We don't provide a receive callback */ - s->nc.receive_disabled =1 ; - s->chr = chr; - nc->queue_index =0 ; + nc->queue_index = i; + nc->opaque = s; -qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s); + s->peers[i].nc = nc; + } + +qemu_chr_add_handlers (chr,NULL ,NULL , net_vhost_user_event, s);return 0 ; }@@ -227,8 +272,10 @@ static int net_vhost_check_net(void *opaque, QemuOpts *opts, Error **errp)int net_init_vhost_user (const NetClientOptions *opts,const char *name,NetClientState *peer,Error **errp) { +int queues;const NetdevVhostUserOptions *vhost_user_opts;CharDriverState *chr; +VhostUserState *s;assert (opts->kind ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); vhost_user_opts = opts->vhost_user;@@ -244,6 +291,11 @@ int net_init_vhost_user(const NetClientOptions *opts, const char *name,return -1 ; } + queues = vhost_user_opts->has_queues ? vhost_user_opts->queues :1 ; + s =g_malloc0 (sizeof (VhostUserState) + + queues *sizeof (struct VhostUserNetPeer )); + s->queues = queues; + s->chr = chr; -return net_vhost_user_init (peer,"vhost_user" , name, chr); +return net_vhost_user_init (peer,"vhost_user" , name, s); }qapi-schema.json +5
-1@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } } ## # @NetClientOptionsqemu-options.hx +3
-2@@ -1963,13 +1963,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single netdev. @code{-net} and @code{-device} with parameter @option{vlan} create the required hub automatically. -@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off] +@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n] Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should be a unix domain socket backed one. The vhost-user uses a specifically defined protocol to pass vhost ioctl replacement messages to an application on the other end of the socket. On non-MSIX guests, the feature can be forced with -@var{vhostforce}. +@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to +be created for multiqueue vhost-user. Example: @exampleSignature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
Quoted 105 lines · Michael S. Tsirkin
Quoted 104 lines · Yuanhan Liu
From: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.What differs from last version is that this patch addresses two major concerns from Michael, and fixes one hidden bug.
- Concern #1: no feedback when the backend can't support # of requested queues(by providing queues=# option).
Here we address this issue by querying
VHOST_USER_PROTOCOL_F_MQprotocol features first, if not set, it means the backend don't support mq feature, and letmax_queuesbe 1. Otherwise, we send another message,VHOST_USER_GET_QUEUE_NUM, for getting themax_queuesthe backend supports.At vhost-user initiation
stage(net_vhost_user_start), we then initiate one queue first, which, in the meantime, also gets themax_queues. We then do a simple compare: ifrequested_queues>max_queues, weexit(I guess it's safe to exit here, as the vm is not running yet).- Concern #2: some messages are sent more times than necessary.
We came an agreement with Michael that we could categorize vhost user messages to 2 types: none-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function
vhost_user_one_time_request(), which lists following messages as none-vring specific messages:
VHOST_USER_GET_FEATURES
VHOST_USER_SET_FEATURES
VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUMFor above messages, we simply ignore them when they are not sent the first time.
I also observed a hidden bug from last version. We register the char dev event handler N times, which is not necessary, as well as buggy: A later register overwrites the former one, as
qemu_chr_add_handlers()will not chain those handlers, but instead overwrites the old one. So, in theory, invokingqemu_chr_add_handlersN times will not end up with calling the handler N times.However, the reason the handler is invoked N times is because we start the backend(the connection server) first, and hence when
net_vhost_user_init()is executed, the connection is already established, andqemu_chr_add_handlers()then invoke the handler immediately, which just looks like we invoke thehandler(net_vhost_user_event)directly fromnet_vhost_user_init().The solution I came up with is to make VhostUserState as an upper level structure, making it includes N nc and
vhost_netpairs:struct VhostUserNetPeer { NetClientState *nc; VHostNetState *vhost_net; }; typedef struct VhostUserState { CharDriverState *chr; bool running; int queues; struct VhostUserNetPeer peers[]; } VhostUserState;Signed-off-by Nikolay Nikolaev <[email protected]>Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>--- docs/specs/vhost-user.txt | 13 +++++ hw/virtio/vhost-user.c | 31 ++++++++++- include/net/net.h | 1 + net/vhost-user.c | 136 ++++++++++++++++++++++++++++++++-------------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- 6 files changed, 146 insertions(+), 46 deletions(-)docs/specs/vhost-user.txt +10
-0@@ -135,6 +135,19 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +------------------- +Multiple queue is treated as a protocal extension, hence the slave has to +implement protocol features first. Multiple queues is supported only when +the protocol feature VHOST_USER_PROTOCOL_F_MQ(bit 0) is set. + +The max # of queues the slave support can be queried with message +VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the # of requested +queues is bigger than that. +
Quoted 8 lines
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 150
Quoted 3 lines
docs/specs/vhost-user.txt +3
-0+As all queues share one connection, the master use a unique index for each +queue in the sent message to identify one specified queue. +Please also document that only 2 queues are enabled initially. More queues are enabled dynamically.
To enable more queues, the new message needs to be sent (added by patch 7).
Hi Michael,
Will do that in next version. Besides that, do you have more comments? Say, does this patch address your two concerns?
--yliu
Quoted 377 lines
Quoted 377 lines
docs/specs/vhost-user.txt +0
-0Message types -------------hw/virtio/vhost-user.c +30
-1@@ -187,6 +187,23 @@ static int vhost_user_write(struct vhost_dev *dev, VhostUserMsg *msg, 0 : -1; } +static bool vhost_user_one_time_request (VhostUserRequest request) +{ +switch (request) { +case VHOST_USER_GET_FEATURES : +case VHOST_USER_SET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_SET_PROTOCOL_FEATURES : +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_DEVICE : +case VHOST_USER_SET_MEM_TABLE : +case VHOST_USER_GET_QUEUE_NUM : +return true; +default : +return false; + } +} +static int vhost_user_call (struct vhost_dev *dev,unsigned long int request,void *arg) {@@ -206,6 +223,14 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,else msg_request = request; +/* +* For none-vring specific requests, like VHOST_USER_GET_FEATURES, +* we just need send it once in the first time. For later such +* request, we just ignore it. +*/ +if (vhost_user_one_time_request (msg_request) && dev->vq_index !=0 ) +return 0 ; + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ;@@ -268,17 +293,20 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_NUM :case VHOST_USER_SET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state); need_reply =1 ;break ;case VHOST_USER_SET_VRING_ADDR :memcpy (&msg.addr, arg,sizeof (struct vhost_vring_addr )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.addr);break ;@@ -286,7 +314,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_CALL :case VHOST_USER_SET_VRING_ERR : file = arg; - msg.u64 = file->index &VHOST_USER_VRING_IDX_MASK ; + msg.u64 = (file->index + dev->vq_index) &VHOST_USER_VRING_IDX_MASK ; msg.size =sizeof (m.u64);if (ioeventfd_enabled () && file->fd >0 ) { fds[fd_num++] = file->fd;@@ -330,6 +358,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,error_report ("Received bad msg size." );return -1 ; } + msg.state.index -= dev->vq_index;memcpy (arg, &msg.state,sizeof (struct vhost_vring_state ));break ;default :include/net/net.h +1
-0@@ -92,6 +92,7 @@ struct NetClientState {NetClientDestructor *destructor;unsigned int queue_index; unsigned rxfilter_notify_enabled:1; +void *opaque; }; typedef struct NICState {net/vhost-user.c +94
-42@@ -15,10 +15,16 @@#include "qemu/config-file.h" #include "qemu/error-report.h" +struct VhostUserNetPeer { +NetClientState *nc; +VHostNetState *vhost_net; +}; +typedef struct VhostUserState { -NetClientState nc;CharDriverState *chr; -VHostNetState *vhost_net; +bool running; +int queues; +struct VhostUserNetPeer peers[]; }VhostUserState ; typedef struct VhostUserChardevProps {@@ -29,48 +35,75 @@ typedef struct VhostUserChardevProps {VHostNetState *vhost_user_get_vhost_net (NetClientState *nc) { -VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); +VhostUserState *s = nc->opaque;assert (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); -return s->vhost_net; -} - -static int vhost_user_running (VhostUserState *s) -{ -return (s->vhost_net) ?1 :0 ; +return s->peers[nc->queue_index].vhost_net; }static int vhost_user_start (VhostUserState *s) {VhostNetOptions options; +VHostNetState *vhost_net; +int max_queues; +int i =0 ; -if (vhost_user_running (s)) { +if (s->running)return 0 ; - } options.backend_type =VHOST_BACKEND_TYPE_USER ; - options.net_backend = &s->nc; options.opaque = s->chr; - s->vhost_net =vhost_net_init (&options); + options.net_backend = s->peers[i].nc; + vhost_net = s->peers[i++].vhost_net =vhost_net_init (&options); + + max_queues =vhost_net_get_max_queues (vhost_net); +if (s->queues >= max_queues) { +error_report ("you are asking more queues than supported: %d\n" , + max_queues); +return -1 ; + } + +for (; i < s->queues; i++) { + options.net_backend = s->peers[i].nc; + + s->peers[i].vhost_net =vhost_net_init (&options); +if (!s->peers[i].vhost_net) +return -1 ; + } + s->running = true; -return vhost_user_running (s) ?0 :-1 ; +return 0 ; }static void vhost_user_stop (VhostUserState *s) { -if (vhost_user_running (s)) { -vhost_net_cleanup (s->vhost_net); +int i; +VHostNetState *vhost_net; + +if (!s->running) +return ; + +for (i =0 ; i < s->queues; i++) { + vhost_net = s->peers[i].vhost_net; +if (vhost_net) +vhost_net_cleanup (vhost_net); } - s->vhost_net =0 ; + s->running = false; }static void vhost_user_cleanup (NetClientState *nc) { -VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); +VhostUserState *s = nc->opaque; +VHostNetState *vhost_net = s->peers[nc->queue_index].vhost_net; + +if (vhost_net) +vhost_net_cleanup (vhost_net); -vhost_user_stop (s);qemu_purge_queued_packets (nc); + +if (nc->queue_index == s->queues -1 ) +free (s); }static bool vhost_user_has_vnet_hdr (NetClientState *nc)@@ -89,7 +122,7 @@ static bool vhost_user_has_ufo(NetClientState *nc)static NetClientInfo net_vhost_user_info = { .type =NET_CLIENT_OPTIONS_KIND_VHOST_USER , - .size =sizeof (VhostUserState), + .size =sizeof (NetClientState), .cleanup = vhost_user_cleanup, .has_vnet_hdr = vhost_user_has_vnet_hdr, .has_ufo = vhost_user_has_ufo,@@ -97,18 +130,25 @@ static NetClientInfo net_vhost_user_info = {static void net_vhost_link_down (VhostUserState *s,bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < s->queues; i++) { + nc = s->peers[i].nc; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } -if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->info->link_status_changed) { + nc->info->link_status_changed (nc); + } + +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }@@ -118,7 +158,8 @@ static void net_vhost_user_event(void *opaque, int event)switch (event) {case CHR_EVENT_OPENED : -vhost_user_start (s); +if (vhost_user_start (s) <0 ) +exit (1 );net_vhost_link_down (s, false);error_report ("chardev \"%s\" went up" , s->chr->label);break ;@@ -131,24 +172,28 @@ static void net_vhost_user_event(void *opaque, int event) }static int net_vhost_user_init (NetClientState *peer,const char *device, -const char *name,CharDriverState *chr) +const char *name,VhostUserState *s) {NetClientState *nc; -VhostUserState *s; +CharDriverState *chr = s->chr; +int i; - nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); +for (i =0 ; i < s->queues; i++) { + nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); -snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user to %s" , - chr->label); +snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user%d to %s" , + i, chr->label); - s =DO_UPCAST (VhostUserState, nc, nc); +/* We don't provide a receive callback */ + nc->receive_disabled =1 ; -/* We don't provide a receive callback */ - s->nc.receive_disabled =1 ; - s->chr = chr; - nc->queue_index =0 ; + nc->queue_index = i; + nc->opaque = s; -qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s); + s->peers[i].nc = nc; + } + +qemu_chr_add_handlers (chr,NULL ,NULL , net_vhost_user_event, s);return 0 ; }@@ -227,8 +272,10 @@ static int net_vhost_check_net(void *opaque, QemuOpts *opts, Error **errp)int net_init_vhost_user (const NetClientOptions *opts,const char *name,NetClientState *peer,Error **errp) { +int queues;const NetdevVhostUserOptions *vhost_user_opts;CharDriverState *chr; +VhostUserState *s;assert (opts->kind ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); vhost_user_opts = opts->vhost_user;@@ -244,6 +291,11 @@ int net_init_vhost_user(const NetClientOptions *opts, const char *name,return -1 ; } + queues = vhost_user_opts->has_queues ? vhost_user_opts->queues :1 ; + s =g_malloc0 (sizeof (VhostUserState) + + queues *sizeof (struct VhostUserNetPeer )); + s->queues = queues; + s->chr = chr; -return net_vhost_user_init (peer,"vhost_user" , name, chr); +return net_vhost_user_init (peer,"vhost_user" , name, s); }qapi-schema.json +5
-1@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } } ## # @NetClientOptionsqemu-options.hx +3
-2@@ -1963,13 +1963,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single netdev. @code{-net} and @code{-device} with parameter @option{vlan} create the required hub automatically. -@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off] +@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n] Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should be a unix domain socket backed one. The vhost-user uses a specifically defined protocol to pass vhost ioctl replacement messages to an application on the other end of the socket. On non-MSIX guests, the feature can be forced with -@var{vhostforce}. +@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to +be created for multiqueue vhost-user. Example: @exampleSignature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
In reply to Yuanhan LiuQuoted 97 lines · Yuanhan Liu
From: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.What differs from last version is that this patch addresses two major concerns from Michael, and fixes one hidden bug.
- Concern #1: no feedback when the backend can't support # of requested queues(by providing queues=# option).
Here we address this issue by querying
VHOST_USER_PROTOCOL_F_MQprotocol features first, if not set, it means the backend don't support mq feature, and letmax_queuesbe 1. Otherwise, we send another message,VHOST_USER_GET_QUEUE_NUM, for getting themax_queuesthe backend supports.At vhost-user initiation
stage(net_vhost_user_start), we then initiate one queue first, which, in the meantime, also gets themax_queues. We then do a simple compare: ifrequested_queues>max_queues, weexit(I guess it's safe to exit here, as the vm is not running yet).- Concern #2: some messages are sent more times than necessary.
We came an agreement with Michael that we could categorize vhost user messages to 2 types: none-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function
vhost_user_one_time_request(), which lists following messages as none-vring specific messages:
VHOST_USER_GET_FEATURES
VHOST_USER_SET_FEATURES
VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUMFor above messages, we simply ignore them when they are not sent the first time.
I also observed a hidden bug from last version. We register the char dev event handler N times, which is not necessary, as well as buggy: A later register overwrites the former one, as
qemu_chr_add_handlers()will not chain those handlers, but instead overwrites the old one. So, in theory, invokingqemu_chr_add_handlersN times will not end up with calling the handler N times.However, the reason the handler is invoked N times is because we start the backend(the connection server) first, and hence when
net_vhost_user_init()is executed, the connection is already established, andqemu_chr_add_handlers()then invoke the handler immediately, which just looks like we invoke thehandler(net_vhost_user_event)directly fromnet_vhost_user_init().The solution I came up with is to make VhostUserState as an upper level structure, making it includes N nc and
vhost_netpairs:struct VhostUserNetPeer { NetClientState *nc; VHostNetState *vhost_net; }; typedef struct VhostUserState { CharDriverState *chr; bool running; int queues; struct VhostUserNetPeer peers[]; } VhostUserState;Signed-off-by Nikolay Nikolaev <[email protected]>Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>--- docs/specs/vhost-user.txt | 13 +++++ hw/virtio/vhost-user.c | 31 ++++++++++- include/net/net.h | 1 + net/vhost-user.c | 136 ++++++++++++++++++++++++++++++++-------------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- 6 files changed, 146 insertions(+), 46 deletions(-)docs/specs/vhost-user.txt +3
-0@@ -135,6 +135,19 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +------------------- +Multiple queue is treated as a protocal extension, hence the slave has to
docs/specs/vhost-user.txt As older slaves don't support negotiating protocol features, · line 150
Quoted 10 lines
docs/specs/vhost-user.txt +10
-0+implement protocol features first. Multiple queues is supported only when +the protocol feature VHOST_USER_PROTOCOL_F_MQ(bit 0) is set. + +The max # of queues the slave support can be queried with message +VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the # of requested +queues is bigger than that. + +As all queues share one connection, the master use a unique index for each +queue in the sent message to identify one specified queue. +
Please also document that only 2 queues are enabled initially. More queues are enabled dynamically.
To enable more queues, the new message needs to be sent (added by patch 7).
Quoted 376 lines
docs/specs/vhost-user.txt +0
-0Message types -------------hw/virtio/vhost-user.c +30
-1@@ -187,6 +187,23 @@ static int vhost_user_write(struct vhost_dev *dev, VhostUserMsg *msg, 0 : -1; } +static bool vhost_user_one_time_request (VhostUserRequest request) +{ +switch (request) { +case VHOST_USER_GET_FEATURES : +case VHOST_USER_SET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_SET_PROTOCOL_FEATURES : +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_DEVICE : +case VHOST_USER_SET_MEM_TABLE : +case VHOST_USER_GET_QUEUE_NUM : +return true; +default : +return false; + } +} +static int vhost_user_call (struct vhost_dev *dev,unsigned long int request,void *arg) {@@ -206,6 +223,14 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,else msg_request = request; +/* +* For none-vring specific requests, like VHOST_USER_GET_FEATURES, +* we just need send it once in the first time. For later such +* request, we just ignore it. +*/ +if (vhost_user_one_time_request (msg_request) && dev->vq_index !=0 ) +return 0 ; + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ;@@ -268,17 +293,20 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_NUM :case VHOST_USER_SET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state); need_reply =1 ;break ;case VHOST_USER_SET_VRING_ADDR :memcpy (&msg.addr, arg,sizeof (struct vhost_vring_addr )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.addr);break ;@@ -286,7 +314,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_CALL :case VHOST_USER_SET_VRING_ERR : file = arg; - msg.u64 = file->index &VHOST_USER_VRING_IDX_MASK ; + msg.u64 = (file->index + dev->vq_index) &VHOST_USER_VRING_IDX_MASK ; msg.size =sizeof (m.u64);if (ioeventfd_enabled () && file->fd >0 ) { fds[fd_num++] = file->fd;@@ -330,6 +358,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,error_report ("Received bad msg size." );return -1 ; } + msg.state.index -= dev->vq_index;memcpy (arg, &msg.state,sizeof (struct vhost_vring_state ));break ;default :include/net/net.h +1
-0@@ -92,6 +92,7 @@ struct NetClientState {NetClientDestructor *destructor;unsigned int queue_index; unsigned rxfilter_notify_enabled:1; +void *opaque; }; typedef struct NICState {net/vhost-user.c +94
-42@@ -15,10 +15,16 @@#include "qemu/config-file.h" #include "qemu/error-report.h" +struct VhostUserNetPeer { +NetClientState *nc; +VHostNetState *vhost_net; +}; +typedef struct VhostUserState { -NetClientState nc;CharDriverState *chr; -VHostNetState *vhost_net; +bool running; +int queues; +struct VhostUserNetPeer peers[]; }VhostUserState ; typedef struct VhostUserChardevProps {@@ -29,48 +35,75 @@ typedef struct VhostUserChardevProps {VHostNetState *vhost_user_get_vhost_net (NetClientState *nc) { -VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); +VhostUserState *s = nc->opaque;assert (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); -return s->vhost_net; -} - -static int vhost_user_running (VhostUserState *s) -{ -return (s->vhost_net) ?1 :0 ; +return s->peers[nc->queue_index].vhost_net; }static int vhost_user_start (VhostUserState *s) {VhostNetOptions options; +VHostNetState *vhost_net; +int max_queues; +int i =0 ; -if (vhost_user_running (s)) { +if (s->running)return 0 ; - } options.backend_type =VHOST_BACKEND_TYPE_USER ; - options.net_backend = &s->nc; options.opaque = s->chr; - s->vhost_net =vhost_net_init (&options); + options.net_backend = s->peers[i].nc; + vhost_net = s->peers[i++].vhost_net =vhost_net_init (&options); + + max_queues =vhost_net_get_max_queues (vhost_net); +if (s->queues >= max_queues) { +error_report ("you are asking more queues than supported: %d\n" , + max_queues); +return -1 ; + } + +for (; i < s->queues; i++) { + options.net_backend = s->peers[i].nc; + + s->peers[i].vhost_net =vhost_net_init (&options); +if (!s->peers[i].vhost_net) +return -1 ; + } + s->running = true; -return vhost_user_running (s) ?0 :-1 ; +return 0 ; }static void vhost_user_stop (VhostUserState *s) { -if (vhost_user_running (s)) { -vhost_net_cleanup (s->vhost_net); +int i; +VHostNetState *vhost_net; + +if (!s->running) +return ; + +for (i =0 ; i < s->queues; i++) { + vhost_net = s->peers[i].vhost_net; +if (vhost_net) +vhost_net_cleanup (vhost_net); } - s->vhost_net =0 ; + s->running = false; }static void vhost_user_cleanup (NetClientState *nc) { -VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); +VhostUserState *s = nc->opaque; +VHostNetState *vhost_net = s->peers[nc->queue_index].vhost_net; + +if (vhost_net) +vhost_net_cleanup (vhost_net); -vhost_user_stop (s);qemu_purge_queued_packets (nc); + +if (nc->queue_index == s->queues -1 ) +free (s); }static bool vhost_user_has_vnet_hdr (NetClientState *nc)@@ -89,7 +122,7 @@ static bool vhost_user_has_ufo(NetClientState *nc)static NetClientInfo net_vhost_user_info = { .type =NET_CLIENT_OPTIONS_KIND_VHOST_USER , - .size =sizeof (VhostUserState), + .size =sizeof (NetClientState), .cleanup = vhost_user_cleanup, .has_vnet_hdr = vhost_user_has_vnet_hdr, .has_ufo = vhost_user_has_ufo,@@ -97,18 +130,25 @@ static NetClientInfo net_vhost_user_info = {static void net_vhost_link_down (VhostUserState *s,bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < s->queues; i++) { + nc = s->peers[i].nc; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } -if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->info->link_status_changed) { + nc->info->link_status_changed (nc); + } + +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }@@ -118,7 +158,8 @@ static void net_vhost_user_event(void *opaque, int event)switch (event) {case CHR_EVENT_OPENED : -vhost_user_start (s); +if (vhost_user_start (s) <0 ) +exit (1 );net_vhost_link_down (s, false);error_report ("chardev \"%s\" went up" , s->chr->label);break ;@@ -131,24 +172,28 @@ static void net_vhost_user_event(void *opaque, int event) }static int net_vhost_user_init (NetClientState *peer,const char *device, -const char *name,CharDriverState *chr) +const char *name,VhostUserState *s) {NetClientState *nc; -VhostUserState *s; +CharDriverState *chr = s->chr; +int i; - nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); +for (i =0 ; i < s->queues; i++) { + nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); -snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user to %s" , - chr->label); +snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user%d to %s" , + i, chr->label); - s =DO_UPCAST (VhostUserState, nc, nc); +/* We don't provide a receive callback */ + nc->receive_disabled =1 ; -/* We don't provide a receive callback */ - s->nc.receive_disabled =1 ; - s->chr = chr; - nc->queue_index =0 ; + nc->queue_index = i; + nc->opaque = s; -qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s); + s->peers[i].nc = nc; + } + +qemu_chr_add_handlers (chr,NULL ,NULL , net_vhost_user_event, s);return 0 ; }@@ -227,8 +272,10 @@ static int net_vhost_check_net(void *opaque, QemuOpts *opts, Error **errp)int net_init_vhost_user (const NetClientOptions *opts,const char *name,NetClientState *peer,Error **errp) { +int queues;const NetdevVhostUserOptions *vhost_user_opts;CharDriverState *chr; +VhostUserState *s;assert (opts->kind ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); vhost_user_opts = opts->vhost_user;@@ -244,6 +291,11 @@ int net_init_vhost_user(const NetClientOptions *opts, const char *name,return -1 ; } + queues = vhost_user_opts->has_queues ? vhost_user_opts->queues :1 ; + s =g_malloc0 (sizeof (VhostUserState) + + queues *sizeof (struct VhostUserNetPeer )); + s->queues = queues; + s->chr = chr; -return net_vhost_user_init (peer,"vhost_user" , name, chr); +return net_vhost_user_init (peer,"vhost_user" , name, s); }qapi-schema.json +5
-1@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } } ## # @NetClientOptionsqemu-options.hx +3
-2@@ -1963,13 +1963,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single netdev. @code{-net} and @code{-device} with parameter @option{vlan} create the required hub automatically. -@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off] +@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n] Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should be a unix domain socket backed one. The vhost-user uses a specifically defined protocol to pass vhost ioctl replacement messages to an application on the other end of the socket. On non-MSIX guests, the feature can be forced with -@var{vhostforce}. +@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to +be created for multiqueue vhost-user. Example: @exampleSignature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
In reply to Yuanhan LiuQuoted 62 lines · Yuanhan Liu
From: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.What differs from last version is that this patch addresses two major concerns from Michael, and fixes one hidden bug.
- Concern #1: no feedback when the backend can't support # of requested queues(by providing queues=# option).
Here we address this issue by querying
VHOST_USER_PROTOCOL_F_MQprotocol features first, if not set, it means the backend don't support mq feature, and letmax_queuesbe 1. Otherwise, we send another message,VHOST_USER_GET_QUEUE_NUM, for getting themax_queuesthe backend supports.At vhost-user initiation
stage(net_vhost_user_start), we then initiate one queue first, which, in the meantime, also gets themax_queues. We then do a simple compare: ifrequested_queues>max_queues, weexit(I guess it's safe to exit here, as the vm is not running yet).- Concern #2: some messages are sent more times than necessary.
We came an agreement with Michael that we could categorize vhost user messages to 2 types: none-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function
vhost_user_one_time_request(), which lists following messages as none-vring specific messages:
VHOST_USER_GET_FEATURES
VHOST_USER_SET_FEATURES
VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUMFor above messages, we simply ignore them when they are not sent the first time.
I also observed a hidden bug from last version. We register the char dev event handler N times, which is not necessary, as well as buggy: A later register overwrites the former one, as
qemu_chr_add_handlers()will not chain those handlers, but instead overwrites the old one. So, in theory, invokingqemu_chr_add_handlersN times will not end up with calling the handler N times.However, the reason the handler is invoked N times is because we start the backend(the connection server) first, and hence when
net_vhost_user_init()is executed, the connection is already established, andqemu_chr_add_handlers()then invoke the handler immediately, which just looks like we invoke thehandler(net_vhost_user_event)directly fromnet_vhost_user_init().The solution I came up with is to make VhostUserState as an upper level structure, making it includes N nc and
vhost_netpairs:
Quoted 11 lines
struct VhostUserNetPeer {
NetClientState *nc;
VHostNetState *vhost_net;
};
typedef struct VhostUserState {
CharDriverState *chr;
bool running;
int queues;
struct VhostUserNetPeer peers[];
} VhostUserState;Haven't looked at previous versions but alternatively, can you do this by:
- keep current VhostUserState
- allocate multiple VhostUserState
- register the char handler only on VhostUserState[0]
- enumerate all others in handler by qemu_find_net_clients_except() ?
Quoted 81 lines
Signed-off-by Nikolay Nikolaev <[email protected]>Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>--- docs/specs/vhost-user.txt | 13 +++++ hw/virtio/vhost-user.c | 31 ++++++++++- include/net/net.h | 1 + net/vhost-user.c | 136 ++++++++++++++++++++++++++++++++-------------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- 6 files changed, 146 insertions(+), 46 deletions(-)docs/specs/vhost-user.txt +13
-0@@ -135,6 +135,19 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +------------------- +Multiple queue is treated as a protocal extension, hence the slave has to +implement protocol features first. Multiple queues is supported only when +the protocol feature VHOST_USER_PROTOCOL_F_MQ(bit 0) is set. + +The max # of queues the slave support can be queried with message +VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the # of requested +queues is bigger than that. + +As all queues share one connection, the master use a unique index for each +queue in the sent message to identify one specified queue. + Message types -------------hw/virtio/vhost-user.c +25
-0@@ -187,6 +187,23 @@ static int vhost_user_write(struct vhost_dev *dev, VhostUserMsg *msg, 0 : -1; } +static bool vhost_user_one_time_request (VhostUserRequest request) +{ +switch (request) { +case VHOST_USER_GET_FEATURES : +case VHOST_USER_SET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_SET_PROTOCOL_FEATURES : +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_DEVICE : +case VHOST_USER_SET_MEM_TABLE : +case VHOST_USER_GET_QUEUE_NUM : +return true; +default : +return false; + } +} +static int vhost_user_call (struct vhost_dev *dev,unsigned long int request,void *arg) {@@ -206,6 +223,14 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,else msg_request = request; +/* +* For none-vring specific requests, like VHOST_USER_GET_FEATURES, +* we just need send it once in the first time. For later such +* request, we just ignore it. +*/ +if (vhost_user_one_time_request (msg_request) && dev->vq_index !=0 ) +return 0 ; + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ;
hw/virtio/vhost-user.c static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, · line 302
Quoted 10 lines
hw/virtio/vhost-user.c +2
-0@@ -268,17 +293,20 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_NUM :case VHOST_USER_SET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); + msg.addr.index += dev->vq_index;
Let's don't do this silently here. Since we've done a minus in vhost_virtqueue_start(), I think we could introduce a new callback and do correct calculation there.
Quoted 93 lines
hw/virtio/vhost-user.c +3
-1msg.size =sizeof (m.state); need_reply =1 ;break ;case VHOST_USER_SET_VRING_ADDR :memcpy (&msg.addr, arg,sizeof (struct vhost_vring_addr )); + msg.addr.index += dev->vq_index; msg.size =sizeof (m.addr);break ;@@ -286,7 +314,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_CALL :case VHOST_USER_SET_VRING_ERR : file = arg; - msg.u64 = file->index &VHOST_USER_VRING_IDX_MASK ; + msg.u64 = (file->index + dev->vq_index) &VHOST_USER_VRING_IDX_MASK ; msg.size =sizeof (m.u64);if (ioeventfd_enabled () && file->fd >0 ) { fds[fd_num++] = file->fd;@@ -330,6 +358,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,error_report ("Received bad msg size." );return -1 ; } + msg.state.index -= dev->vq_index;memcpy (arg, &msg.state,sizeof (struct vhost_vring_state ));break ;default :include/net/net.h +1
-0@@ -92,6 +92,7 @@ struct NetClientState {NetClientDestructor *destructor;unsigned int queue_index; unsigned rxfilter_notify_enabled:1; +void *opaque; }; typedef struct NICState {net/vhost-user.c +14
-12@@ -15,10 +15,16 @@#include "qemu/config-file.h" #include "qemu/error-report.h" +struct VhostUserNetPeer { +NetClientState *nc; +VHostNetState *vhost_net; +}; +typedef struct VhostUserState { -NetClientState nc;CharDriverState *chr; -VHostNetState *vhost_net; +bool running; +int queues; +struct VhostUserNetPeer peers[]; }VhostUserState ; typedef struct VhostUserChardevProps {@@ -29,48 +35,75 @@ typedef struct VhostUserChardevProps {VHostNetState *vhost_user_get_vhost_net (NetClientState *nc) { -VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); +VhostUserState *s = nc->opaque;assert (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); -return s->vhost_net; -} - -static int vhost_user_running (VhostUserState *s) -{ -return (s->vhost_net) ?1 :0 ; +return s->peers[nc->queue_index].vhost_net; }static int vhost_user_start (VhostUserState *s) {VhostNetOptions options; +VHostNetState *vhost_net; +int max_queues; +int i =0 ; -if (vhost_user_running (s)) { +if (s->running)return 0 ; - } options.backend_type =VHOST_BACKEND_TYPE_USER ; - options.net_backend = &s->nc; options.opaque = s->chr;
net/vhost-user.c typedef struct VhostUserChardevProps { · line 63
Quoted 10 lines
net/vhost-user.c +8
-1- s->vhost_net =vhost_net_init (&options); + options.net_backend = s->peers[i].nc; + vhost_net = s->peers[i++].vhost_net =vhost_net_init (&options); + + max_queues =vhost_net_get_max_queues (vhost_net); +if (s->queues >= max_queues) { +error_report ("you are asking more queues than supported: %d\n" , + max_queues); +return -1 ;
Memory leak here?
net/vhost-user.c typedef struct VhostUserChardevProps { · line 72
Quoted 9 lines
net/vhost-user.c +9
-0+ } + +for (; i < s->queues; i++) { + options.net_backend = s->peers[i].nc; + + s->peers[i].vhost_net =vhost_net_init (&options); +if (!s->peers[i].vhost_net) +return -1 ; + }
Maybe you can initialize all vhost_net in one loop, and check for the max_queues afterwards to simplify the code.
Quoted 121 lines
net/vhost-user.c +49
-23+ s->running = true; -return vhost_user_running (s) ?0 :-1 ; +return 0 ; }static void vhost_user_stop (VhostUserState *s) { -if (vhost_user_running (s)) { -vhost_net_cleanup (s->vhost_net); +int i; +VHostNetState *vhost_net; + +if (!s->running) +return ; + +for (i =0 ; i < s->queues; i++) { + vhost_net = s->peers[i].vhost_net; +if (vhost_net) +vhost_net_cleanup (vhost_net); } - s->vhost_net =0 ; + s->running = false; }static void vhost_user_cleanup (NetClientState *nc) { -VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); +VhostUserState *s = nc->opaque; +VHostNetState *vhost_net = s->peers[nc->queue_index].vhost_net; + +if (vhost_net) +vhost_net_cleanup (vhost_net); -vhost_user_stop (s);qemu_purge_queued_packets (nc); + +if (nc->queue_index == s->queues -1 ) +free (s); }static bool vhost_user_has_vnet_hdr (NetClientState *nc)@@ -89,7 +122,7 @@ static bool vhost_user_has_ufo(NetClientState *nc)static NetClientInfo net_vhost_user_info = { .type =NET_CLIENT_OPTIONS_KIND_VHOST_USER , - .size =sizeof (VhostUserState), + .size =sizeof (NetClientState), .cleanup = vhost_user_cleanup, .has_vnet_hdr = vhost_user_has_vnet_hdr, .has_ufo = vhost_user_has_ufo,@@ -97,18 +130,25 @@ static NetClientInfo net_vhost_user_info = {static void net_vhost_link_down (VhostUserState *s,bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < s->queues; i++) { + nc = s->peers[i].nc; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } -if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->info->link_status_changed) { + nc->info->link_status_changed (nc); + } + +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }@@ -118,7 +158,8 @@ static void net_vhost_user_event(void *opaque, int event)switch (event) {case CHR_EVENT_OPENED : -vhost_user_start (s); +if (vhost_user_start (s) <0 ) +exit (1 );net_vhost_link_down (s, false);error_report ("chardev \"%s\" went up" , s->chr->label);break ;@@ -131,24 +172,28 @@ static void net_vhost_user_event(void *opaque, int event) }static int net_vhost_user_init (NetClientState *peer,const char *device, -const char *name,CharDriverState *chr) +const char *name,VhostUserState *s) {NetClientState *nc; -VhostUserState *s; +CharDriverState *chr = s->chr; +int i; - nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); +for (i =0 ; i < s->queues; i++) { + nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); -snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user to %s" , - chr->label); +snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user%d to %s" , + i, chr->label); - s =DO_UPCAST (VhostUserState, nc, nc); +/* We don't provide a receive callback */ + nc->receive_disabled =1 ;
net/vhost-user.c static void net_vhost_user_event(void *opaque, int event) · line 193
Quoted 10 lines
net/vhost-user.c +3
-5-/* We don't provide a receive callback */ - s->nc.receive_disabled =1 ; - s->chr = chr; - nc->queue_index =0 ; + nc->queue_index = i; + nc->opaque = s; -qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s); + s->peers[i].nc = nc;
So actually "peers" is confusing. Better rename it to "ncs" or something else.
Quoted 73 lines
net/vhost-user.c +11
-1+ } + +qemu_chr_add_handlers (chr,NULL ,NULL , net_vhost_user_event, s);return 0 ; }@@ -227,8 +272,10 @@ static int net_vhost_check_net(void *opaque, QemuOpts *opts, Error **errp)int net_init_vhost_user (const NetClientOptions *opts,const char *name,NetClientState *peer,Error **errp) { +int queues;const NetdevVhostUserOptions *vhost_user_opts;CharDriverState *chr; +VhostUserState *s;assert (opts->kind ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); vhost_user_opts = opts->vhost_user;@@ -244,6 +291,11 @@ int net_init_vhost_user(const NetClientOptions *opts, const char *name,return -1 ; } + queues = vhost_user_opts->has_queues ? vhost_user_opts->queues :1 ; + s =g_malloc0 (sizeof (VhostUserState) + + queues *sizeof (struct VhostUserNetPeer )); + s->queues = queues; + s->chr = chr; -return net_vhost_user_init (peer,"vhost_user" , name, chr); +return net_vhost_user_init (peer,"vhost_user" , name, s); }qapi-schema.json +5
-1@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } } ## # @NetClientOptionsqemu-options.hx +3
-2@@ -1963,13 +1963,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single netdev. @code{-net} and @code{-device} with parameter @option{vlan} create the required hub automatically. -@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off] +@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n] Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should be a unix domain socket backed one. The vhost-user uses a specifically defined protocol to pass vhost ioctl replacement messages to an application on the other end of the socket. On non-MSIX guests, the feature can be forced with -@var{vhostforce}. +@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to +be created for multiqueue vhost-user. Example: @example
Re: [Qemu-devel] [PATCH 6/7] vhost-user: add multiple queue support
Quoted 77 lines · Jason Wang
Quoted 62 lines · Yuanhan Liu
From: Ouyang Changchun <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
Here is the latest version for adding vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.What differs from last version is that this patch addresses two major concerns from Michael, and fixes one hidden bug.
- Concern #1: no feedback when the backend can't support # of requested queues(by providing queues=# option).
Here we address this issue by querying
VHOST_USER_PROTOCOL_F_MQprotocol features first, if not set, it means the backend don't support mq feature, and letmax_queuesbe 1. Otherwise, we send another message,VHOST_USER_GET_QUEUE_NUM, for getting themax_queuesthe backend supports.At vhost-user initiation
stage(net_vhost_user_start), we then initiate one queue first, which, in the meantime, also gets themax_queues. We then do a simple compare: ifrequested_queues>max_queues, weexit(I guess it's safe to exit here, as the vm is not running yet).- Concern #2: some messages are sent more times than necessary.
We came an agreement with Michael that we could categorize vhost user messages to 2 types: none-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function
vhost_user_one_time_request(), which lists following messages as none-vring specific messages:
VHOST_USER_GET_FEATURES
VHOST_USER_SET_FEATURES
VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUMFor above messages, we simply ignore them when they are not sent the first time.
I also observed a hidden bug from last version. We register the char dev event handler N times, which is not necessary, as well as buggy: A later register overwrites the former one, as
qemu_chr_add_handlers()will not chain those handlers, but instead overwrites the old one. So, in theory, invokingqemu_chr_add_handlersN times will not end up with calling the handler N times.However, the reason the handler is invoked N times is because we start the backend(the connection server) first, and hence when
net_vhost_user_init()is executed, the connection is already established, andqemu_chr_add_handlers()then invoke the handler immediately, which just looks like we invoke thehandler(net_vhost_user_event)directly fromnet_vhost_user_init().The solution I came up with is to make VhostUserState as an upper level structure, making it includes N nc and
vhost_netpairs:Quoted 11 lines
struct VhostUserNetPeer { NetClientState *nc; VHostNetState *vhost_net; }; typedef struct VhostUserState { CharDriverState *chr; bool running; int queues; struct VhostUserNetPeer peers[]; } VhostUserState;
Quoted 6 lines
Haven't looked at previous versions but alternatively, can you do this by:
- keep current VhostUserState
- allocate multiple VhostUserState
- register the char handler only on VhostUserState[0]
- enumerate all others in handler byqemu_find_net_clients_except()?
That does sound work and much better! I will give it a try soon. Thanks!
Quoted 6 lines
Quoted 6 lines
Signed-off-by Nikolay Nikolaev <[email protected]>Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>---
...
Quoted 7 lines
Quoted 3 lines
case VHOST_USER_GET_VRING_BASE: memcpy(&msg.state, arg, sizeof(struct vhost_vring_state));+ msg.addr.index +=
dev->vq_index;Let's don't do this silently here. Since we've done a minus in
vhost_virtqueue_start(), I think we could introduce a new callback and do correct calculation there.
Good point, I will try it as well.
Quoted 11 lines
net/vhost-user.c typedef struct VhostUserChardevProps { · line 63
Quoted 9 lines
net/vhost-user.c +8
-1- s->vhost_net =vhost_net_init (&options); + options.net_backend = s->peers[i].nc; + vhost_net = s->peers[i++].vhost_net =vhost_net_init (&options); + + max_queues =vhost_net_get_max_queues (vhost_net); +if (s->queues >= max_queues) { +error_report ("you are asking more queues than supported: %d\n" , + max_queues); +return -1 ;Memory leak here?
Yeah, it is. But Does it matter? Since we will call exit() soon, which will reclaim everything in the end. Or, even though, it's still needed?
Either way, I can do that.
Quoted 13 lines
net/vhost-user.c typedef struct VhostUserChardevProps { · line 72
Quoted 10 lines
net/vhost-user.c +9
-0+ } + +for (; i < s->queues; i++) { + options.net_backend = s->peers[i].nc; + + s->peers[i].vhost_net =vhost_net_init (&options); +if (!s->peers[i].vhost_net) +return -1 ; + }Maybe you can initialize all
vhost_netin one loop, and check for themax_queuesafterwards to simplify the code.
Okay.
Quoted 1 line
Quoted 1 line
...
Quoted 29 lines
Quoted 29 lines
static int net_vhost_user_init(NetClientState *peer, const char *device,
net/vhost-user.c +11
-10- const char *name, CharDriverState *chr) + const char *name, VhostUserState *s) { NetClientState *nc; -VhostUserState *s; +CharDriverState *chr = s->chr; +int i; - nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); +for (i =0 ; i < s->queues; i++) { + nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); -snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user to %s" , - chr->label); +snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user%d to %s" , + i, chr->label); - s =DO_UPCAST (VhostUserState, nc, nc); +/* We don't provide a receive callback */ + nc->receive_disabled =1 ; -/* We don't provide a receive callback */ - s->nc.receive_disabled =1 ; - s->chr = chr; - nc->queue_index =0 ; + nc->queue_index = i; + nc->opaque = s;
Quoted 5 lines
net/vhost-user.c static void net_vhost_user_event(void *opaque, int event) · line 193
Quoted 2 lines
net/vhost-user.c +1
-1-qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s); + s->peers[i].nc = nc;So actually "peers" is confusing. Better rename it to "ncs" or something else.
Yeah, that's why I renamed it to pairs in later version. But anyway, I guess it's not needed anymore with qemu_find_net_clients_except(), right? :)
Thanks!
--yliu
PATCH v7 7/7 vhost-user: add a new message to disable/enable a specific virt queue.
In reply to Yuanhan LiuFrom: Changchun Ouyang <[email protected]>
Add a new message, VHOST_USER_SET_VRING_FLAG, to enable and disable a specific virt queue, which is similar to attach/detach queue for tap device.
virtio driver on guest doesn't have to use max virt queue pair, it could enable any number of virt queue ranging from 1 to max virt queue pair.
--- docs/specs/vhost-user.txt | 10 ++++++++++ hw/net/vhost_net.c | 18 ++++++++++++++++++ hw/net/virtio-net.c | 2 ++ hw/virtio/vhost-user.c | 32 ++++++++++++++++++++++++++++---- include/hw/virtio/vhost-backend.h | 2 ++ include/net/vhost_net.h | 1 + 6 files changed, 61 insertions(+), 4 deletions(-)
docs/specs/vhost-user.txt +10 -0
@@ -325,3 +325,13 @@ Message types Query how many queues the backend supports. This request should be sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol features by VHOST_USER_GET_PROTOCOL_FEATURES. + + * VHOST_USER_SET_VRING_FLAG + + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Set the flag(1 for enable and 0 for disable) to signal slave to enable + or disable corresponding virt queue. This request should be sent only + when the protocol feature bit VHOST_USER_PROTOCOL_F_VRING_FLAG is set.
hw/net/vhost_net.c +18 -0
@@ -418,6 +418,19 @@ VHostNetState *get_vhost_net(NetClientState *nc) return vhost_net; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +if (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) { +VHostNetState *net =get_vhost_net (nc); +const VhostOps *vhost_ops = net->dev.vhost_ops; +if (vhost_ops->vhost_backend_mq_set_vring_flag) +return vhost_ops->vhost_backend_mq_set_vring_flag (&net->dev, enable); + } + +return 0 ; +} +#else struct vhost_net *vhost_net_init (VhostNetOptions *options) {
@@ -463,4 +476,9 @@ VHostNetState *get_vhost_net(NetClientState *nc) {return 0 ; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +return 0 ; +}#endif
hw/net/virtio-net.c +2 -0
@@ -407,6 +407,7 @@ static int peer_attach(VirtIONet *n, int index) }if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,1 );return 0 ; }
@@ -422,6 +423,7 @@ static int peer_detach(VirtIONet *n, int index) }if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,0 );return 0 ; }
hw/virtio/vhost-user.c +28 -4
@@ -25,9 +25,10 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0xfULL -#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_VRING_FLAG 1 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,
@@ -48,6 +49,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 ,VHOST_USER_GET_QUEUE_NUM =17 , +VHOST_USER_SET_VRING_FLAG =18 , VHOST_USER_MAX } VhostUserRequest;
@@ -296,6 +298,11 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ; +case VHOST_USER_SET_VRING_FLAG : + msg.state.index = dev->vq_index; + msg.state.num = *(int *)arg; + msg.size =sizeof (msg.state); +break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state ));
@@ -408,6 +415,22 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)return 0 ; } +static int vhost_user_set_vring_flag (struct vhost_dev *dev,int enable) +{ +int err; + +assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); + +if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_VRING_FLAG ))) +return -1 ; + + err =vhost_user_call (dev,VHOST_USER_SET_VRING_FLAG , &enable); +if (err <0 ) +return err; + +return 0 ; +} +static int vhost_user_cleanup (struct vhost_dev *dev) {assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER );
@@ -421,5 +444,6 @@ const VhostOps user_ops = { .backend_type = VHOST_BACKEND_TYPE_USER, .vhost_call = vhost_user_call, .vhost_backend_init = vhost_user_init, - .vhost_backend_cleanup = vhost_user_cleanup - }; + .vhost_backend_cleanup = vhost_user_cleanup, + .vhost_backend_mq_set_vring_flag = vhost_user_set_vring_flag, +};
include/hw/virtio/vhost-backend.h +2 -0
@@ -24,12 +24,14 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, void *arg);typedef int (*vhost_backend_init )(struct vhost_dev *dev,void *opaque);typedef int (*vhost_backend_cleanup )(struct vhost_dev *dev); +typedef int (*vhost_backend_mq_set_vring_flag )(struct vhost_dev *dev,int enable);typedef struct VhostOps {VhostBackendType backend_type;vhost_call vhost_call;vhost_backend_init vhost_backend_init;vhost_backend_cleanup vhost_backend_cleanup; +vhost_backend_mq_set_vring_flag vhost_backend_mq_set_vring_flag; }VhostOps ;extern const VhostOps user_ops;
include/net/vhost_net.h +1 -0
@@ -28,4 +28,5 @@ bool vhost_net_virtqueue_pending(VHostNetState *net, int n);void vhost_net_virtqueue_mask (VHostNetState *net,VirtIODevice *dev,int idx,bool mask);VHostNetState *get_vhost_net (NetClientState *nc); +int vhost_set_vring_flag (NetClientState * nc,int enable);#endif
Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
Quoted 6 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
Add a new message,
VHOST_USER_SET_VRING_FLAG, to enable and disable a specific virt queue, which is similar to attach/detach queue for tap device.
Quoted 6 lines
virtio driver on guest doesn't have to use max virt queue pair, it could enable any number of virt queue ranging from 1 to max virt queue pair.
Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>
Fine, but I would like to make this dependent on the MQ flag.
Quoted 14 lines
--- docs/specs/vhost-user.txt | 10 ++++++++++ hw/net/vhost_net.c | 18 ++++++++++++++++++ hw/net/virtio-net.c | 2 ++ hw/virtio/vhost-user.c | 32 ++++++++++++++++++++++++++++---- include/hw/virtio/vhost-backend.h | 2 ++ include/net/vhost_net.h | 1 + 6 files changed, 61 insertions(+), 4 deletions(-)docs/specs/vhost-user.txt +0
-0
docs/specs/vhost-user.txt Message types · line 329
Quoted 5 lines
docs/specs/vhost-user.txt +2
-0@@ -325,3 +325,13 @@ Message types Query how many queues the backend supports. This request should be sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol features by VHOST_USER_GET_PROTOCOL_FEATURES. + + * VHOST_USER_SET_VRING_FLAG
SET_VRIHG_ENABLE would be a better name.
docs/specs/vhost-user.txt Message types · line 335
Quoted 6 lines
docs/specs/vhost-user.txt +6
-0+ + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Set the flag(1 for enable and 0 for disable) to signal slave to enable
Space before ( please.
Quoted 7 lines
docs/specs/vhost-user.txt +2
-0+ or disable corresponding virt queue. This request should be sent only + when the protocol feature bit VHOST_USER_PROTOCOL_F_VRING_FLAG is set.hw/net/vhost_net.c +0
-0
hw/net/vhost_net.c VHostNetState *get_vhost_net(NetClientState *nc) · line 426
Quoted 9 lines
hw/net/vhost_net.c +6
-0@@ -418,6 +418,19 @@ VHostNetState *get_vhost_net(NetClientState *nc)return vhost_net; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +if (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) { +VHostNetState *net =get_vhost_net (nc); +const VhostOps *vhost_ops = net->dev.vhost_ops;
add an empty line after declaration pls.
hw/net/vhost_net.c VHostNetState *get_vhost_net(NetClientState *nc) · line 428
Quoted 2 lines
hw/net/vhost_net.c +2
-0+if (vhost_ops->vhost_backend_mq_set_vring_flag) +return vhost_ops->vhost_backend_mq_set_vring_flag (&net->dev, enable);
Coding style violation:
line too long, and should use {}.
Quoted 31 lines
hw/net/vhost_net.c +10
-0+ } + +return 0 ; +} +#else struct vhost_net *vhost_net_init (VhostNetOptions *options) {@@ -463,4 +476,9 @@ VHostNetState *get_vhost_net(NetClientState *nc) {return 0 ; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +return 0 ; +}#endif hw/net/virtio-net.c +1
-0@@ -407,6 +407,7 @@ static int peer_attach(VirtIONet *n, int index) }if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,1 );return 0 ; }
hw/net/virtio-net.c static int peer_detach(VirtIONet *n, int index) · line 429
Quoted 7 lines
hw/net/virtio-net.c +1
-0@@ -422,6 +423,7 @@ static int peer_detach(VirtIONet *n, int index) }if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,0 );return 0 ; }
makes no sense to call it for !== tap only. Either call this unconditionally, or just for vhost user.
Quoted 50 lines
hw/virtio/vhost-user.c +18
-2@@ -25,9 +25,10 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0xfULL -#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_VRING_FLAG 1 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,@@ -48,6 +49,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 ,VHOST_USER_GET_QUEUE_NUM =17 , +VHOST_USER_SET_VRING_FLAG =18 , VHOST_USER_MAX } VhostUserRequest;@@ -296,6 +298,11 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ; +case VHOST_USER_SET_VRING_FLAG : + msg.state.index = dev->vq_index; + msg.state.num = *(int *)arg; + msg.size =sizeof (msg.state); +break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state ));@@ -408,6 +415,22 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)return 0 ; } +static int vhost_user_set_vring_flag (struct vhost_dev *dev,int enable) +{ +int err; + +assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); + +if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_VRING_FLAG ))) +return -1 ; +
hw/virtio/vhost-user.c static int vhost_user_init(struct vhost_dev *dev, void *opaque) · line 436
Quoted 10 lines
hw/virtio/vhost-user.c +7
-0+ err =vhost_user_call (dev,VHOST_USER_SET_VRING_FLAG , &enable); +if (err <0 ) +return err; + +return 0 ; +} +static int vhost_user_cleanup (struct vhost_dev *dev) {assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER );
In fact, it's a per VQ pair call the way it's coded. So pls fix it, call it for all VQs.
Quoted 15 lines
hw/virtio/vhost-user.c +3
-2@@ -421,5 +444,6 @@ const VhostOps user_ops = { .backend_type = VHOST_BACKEND_TYPE_USER, .vhost_call = vhost_user_call, .vhost_backend_init = vhost_user_init, - .vhost_backend_cleanup = vhost_user_cleanup - }; + .vhost_backend_cleanup = vhost_user_cleanup, + .vhost_backend_mq_set_vring_flag = vhost_user_set_vring_flag, +};include/hw/virtio/vhost-backend.h +0
-0@@ -24,12 +24,14 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, void *arg);
include/hw/virtio/vhost-backend.h typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, · line 34
Quoted 10 lines
include/hw/virtio/vhost-backend.h +2
-0typedef int (*vhost_backend_init )(struct vhost_dev *dev,void *opaque);typedef int (*vhost_backend_cleanup )(struct vhost_dev *dev); +typedef int (*vhost_backend_mq_set_vring_flag )(struct vhost_dev *dev,int enable); typedef struct VhostOps { VhostBackendType backend_type; vhost_call vhost_call; vhost_backend_init vhost_backend_init; vhost_backend_cleanup vhost_backend_cleanup; + vhost_backend_mq_set_vring_flag vhost_backend_mq_set_vring_flag;
That's quite ugly: mq is a vhost net thing. Why not enable each VQ?
Quoted 15 lines
include/hw/virtio/vhost-backend.h +0
-0} VhostOps;extern const VhostOps user_ops;include/net/vhost_net.h +1
-0@@ -28,4 +28,5 @@ bool vhost_net_virtqueue_pending(VHostNetState *net, int n);void vhost_net_virtqueue_mask (VHostNetState *net,VirtIODevice *dev,int idx,bool mask);VHostNetState *get_vhost_net (NetClientState *nc); +int vhost_set_vring_flag (NetClientState * nc,int enable);#endif Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
Add a new message,
VHOST_USER_SET_VRING_FLAG, to enable and disable a specific virt queue, which is similar to attach/detach queue for tap device.
Quoted 8 lines
Quoted 6 lines
virtio driver on guest doesn't have to use max virt queue pair, it could enable any number of virt queue ranging from 1 to max virt queue pair.
Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>Fine, but I would like to make this dependent on the MQ flag.
Reasonable.
Quoted 11 lines
Quoted 11 lines
--- docs/specs/vhost-user.txt | 10 ++++++++++ hw/net/vhost_net.c | 18 ++++++++++++++++++ hw/net/virtio-net.c | 2 ++ hw/virtio/vhost-user.c | 32 ++++++++++++++++++++++++++++---- include/hw/virtio/vhost-backend.h | 2 ++ include/net/vhost_net.h | 1 + 6 files changed, 61 insertions(+), 4 deletions(-)
Quoted 13 lines
docs/specs/vhost-user.txt Message types · line 329
Quoted 10 lines
docs/specs/vhost-user.txt +2
-0@@ -325,3 +325,13 @@ Message types Query how many queues the backend supports. This request should be sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol features by VHOST_USER_GET_PROTOCOL_FEATURES. + + * VHOST_USER_SET_VRING_FLAG
SET_VRIHG_ENABLEwould be a better name.
Okay.
Quoted 9 lines
docs/specs/vhost-user.txt Message types · line 335
Quoted 7 lines
docs/specs/vhost-user.txt +6
-0+ + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Set the flag(1 for enable and 0 for disable) to signal slave to enableSpace before ( please.
Okay.
Quoted 9 lines
Quoted 9 lines
docs/specs/vhost-user.txt +2
-0+ or disable corresponding virt queue. This request should be sent only + when the protocol feature bit VHOST_USER_PROTOCOL_F_VRING_FLAG is set.hw/net/vhost_net.c +0
-0@@ -418,6 +418,19 @@ VHostNetState *get_vhost_net(NetClientState *nc)
Quoted 10 lines
hw/net/vhost_net.c VHostNetState *get_vhost_net(NetClientState *nc) · line 426
Quoted 8 lines
hw/net/vhost_net.c +6
-0return vhost_net; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +if (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) { +VHostNetState *net =get_vhost_net (nc); +const VhostOps *vhost_ops = net->dev.vhost_ops;add an empty line after declaration pls.
Got it, thanks.
Quoted 6 lines
hw/net/vhost_net.c VHostNetState *get_vhost_net(NetClientState *nc) · line 428
Quoted 3 lines
hw/net/vhost_net.c +2
-0+if (vhost_ops->vhost_backend_mq_set_vring_flag) +return vhost_ops->vhost_backend_mq_set_vring_flag (&net->dev, enable);Coding style violation:
line too long,
Yeah, 81 chars. I thougth that's kind of acceptable. But, anyway, since you pointed it out, I will fix it next time.
Quoted 1 line
and should use {}.
TBH, I seldom use {} for one statement, and you could see that a lot from this patch set. Is it a must in qemu community? If so, I could fix them all.
Quoted 34 lines
Quoted 34 lines
hw/net/vhost_net.c +10
-0+ } + +return 0 ; +} +#else struct vhost_net *vhost_net_init (VhostNetOptions *options) {@@ -463,4 +476,9 @@ VHostNetState *get_vhost_net(NetClientState *nc) {return 0 ; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +return 0 ; +}#endif hw/net/virtio-net.c +1
-0@@ -407,6 +407,7 @@ static int peer_attach(VirtIONet *n, int index) }if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,1 );return 0 ; }@@ -422,6 +423,7 @@ static int peer_detach(VirtIONet *n, int index) }
Quoted 8 lines
hw/net/virtio-net.c static int peer_detach(VirtIONet *n, int index) · line 429
Quoted 5 lines
hw/net/virtio-net.c +1
-0if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,0 );return 0 ; }makes no sense to call it for !== tap only. Either call this unconditionally, or just for vhost user.
Agreed. I will make it vhost-usre specific then.
Quoted 57 lines
Quoted 57 lines
hw/virtio/vhost-user.c +23
-2@@ -25,9 +25,10 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0xfULL -#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_VRING_FLAG 1 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,@@ -48,6 +49,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 ,VHOST_USER_GET_QUEUE_NUM =17 , +VHOST_USER_SET_VRING_FLAG =18 , VHOST_USER_MAX } VhostUserRequest;@@ -296,6 +298,11 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ; +case VHOST_USER_SET_VRING_FLAG : + msg.state.index = dev->vq_index; + msg.state.num = *(int *)arg; + msg.size =sizeof (msg.state); +break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state ));@@ -408,6 +415,22 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)return 0 ; } +static int vhost_user_set_vring_flag (struct vhost_dev *dev,int enable) +{ +int err; + +assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); + +if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_VRING_FLAG ))) +return -1 ; + + err =vhost_user_call (dev,VHOST_USER_SET_VRING_FLAG , &enable); +if (err <0 ) +return err; + +return 0 ;
Quoted 8 lines
hw/virtio/vhost-user.c static int vhost_user_init(struct vhost_dev *dev, void *opaque) · line 436
Quoted 5 lines
hw/virtio/vhost-user.c +2
-0+} +static int vhost_user_cleanup (struct vhost_dev *dev) {assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER );In fact, it's a per VQ pair call the way it's coded. So pls fix it, call it for all VQs.
Let me try.
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-user.c +3
-2@@ -421,5 +444,6 @@ const VhostOps user_ops = { .backend_type = VHOST_BACKEND_TYPE_USER, .vhost_call = vhost_user_call, .vhost_backend_init = vhost_user_init, - .vhost_backend_cleanup = vhost_user_cleanup - }; + .vhost_backend_cleanup = vhost_user_cleanup, + .vhost_backend_mq_set_vring_flag = vhost_user_set_vring_flag, +};include/hw/virtio/vhost-backend.h +1
-0@@ -24,12 +24,14 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, void *arg);typedef int (*vhost_backend_init )(struct vhost_dev *dev,void *opaque);typedef int (*vhost_backend_cleanup )(struct vhost_dev *dev); +typedef int (*vhost_backend_mq_set_vring_flag )(struct vhost_dev *dev,int enable);
Quoted 9 lines
include/hw/virtio/vhost-backend.h typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, · line 34
Quoted 6 lines
include/hw/virtio/vhost-backend.h +1
-0typedef struct VhostOps { VhostBackendType backend_type; vhost_call vhost_call; vhost_backend_init vhost_backend_init; vhost_backend_cleanup vhost_backend_cleanup; + vhost_backend_mq_set_vring_flag vhost_backend_mq_set_vring_flag;That's quite ugly: mq is a vhost net thing. Why not enable each VQ?
To define it as `vhost_backend_enable_vring'?
Thanks.
--yliu
Quoted 16 lines
Quoted 16 lines
include/hw/virtio/vhost-backend.h +0
-0} VhostOps;extern const VhostOps user_ops;include/net/vhost_net.h +1
-0@@ -28,4 +28,5 @@ bool vhost_net_virtqueue_pending(VHostNetState *net, int n);void vhost_net_virtqueue_mask (VHostNetState *net,VirtIODevice *dev,int idx,bool mask);VHostNetState *get_vhost_net (NetClientState *nc); +int vhost_set_vring_flag (NetClientState * nc,int enable);#endif Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
Quoted 87 lines · Yuanhan Liu
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
Add a new message,
VHOST_USER_SET_VRING_FLAG, to enable and disable a specific virt queue, which is similar to attach/detach queue for tap device.Quoted 8 lines
Quoted 6 lines
virtio driver on guest doesn't have to use max virt queue pair, it could enable any number of virt queue ranging from 1 to max virt queue pair.
Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>Fine, but I would like to make this dependent on the MQ flag.
Reasonable.
Quoted 11 lines
Quoted 11 lines
--- docs/specs/vhost-user.txt | 10 ++++++++++ hw/net/vhost_net.c | 18 ++++++++++++++++++ hw/net/virtio-net.c | 2 ++ hw/virtio/vhost-user.c | 32 ++++++++++++++++++++++++++++---- include/hw/virtio/vhost-backend.h | 2 ++ include/net/vhost_net.h | 1 + 6 files changed, 61 insertions(+), 4 deletions(-)Quoted 13 lines
docs/specs/vhost-user.txt Message types · line 329
Quoted 10 lines
docs/specs/vhost-user.txt +2
-0@@ -325,3 +325,13 @@ Message types Query how many queues the backend supports. This request should be sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol features by VHOST_USER_GET_PROTOCOL_FEATURES. + + * VHOST_USER_SET_VRING_FLAG
SET_VRIHG_ENABLEwould be a better name.Okay.
Quoted 9 lines
docs/specs/vhost-user.txt Message types · line 335
Quoted 7 lines
docs/specs/vhost-user.txt +6
-0+ + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Set the flag(1 for enable and 0 for disable) to signal slave to enableSpace before ( please.
Okay.
Quoted 9 lines
Quoted 9 lines
docs/specs/vhost-user.txt +2
-0+ or disable corresponding virt queue. This request should be sent only + when the protocol feature bit VHOST_USER_PROTOCOL_F_VRING_FLAG is set.hw/net/vhost_net.c +0
-0@@ -418,6 +418,19 @@ VHostNetState *get_vhost_net(NetClientState *nc)Quoted 10 lines
hw/net/vhost_net.c VHostNetState *get_vhost_net(NetClientState *nc) · line 426
Quoted 8 lines
hw/net/vhost_net.c +6
-0return vhost_net; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +if (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) { +VHostNetState *net =get_vhost_net (nc); +const VhostOps *vhost_ops = net->dev.vhost_ops;add an empty line after declaration pls.
Got it, thanks.
Quoted 6 lines
hw/net/vhost_net.c VHostNetState *get_vhost_net(NetClientState *nc) · line 428
Quoted 3 lines
hw/net/vhost_net.c +2
-0+if (vhost_ops->vhost_backend_mq_set_vring_flag) +return vhost_ops->vhost_backend_mq_set_vring_flag (&net->dev, enable);Coding style violation:
line too long,
Quoted 7 lines
Yeah, 81 chars. I thougth that's kind of acceptable. But, anyway, since you pointed it out, I will fix it next time.
Quoted 1 line
and should use {}.
TBH, I seldom use {} for one statement, and you could see that a lot from this patch set. Is it a must in qemu community?
Yes it is. It's different from the kernel.
Quoted 146 lines
If so, I could fix them all.
Quoted 34 lines
Quoted 34 lines
hw/net/vhost_net.c +10
-0+ } + +return 0 ; +} +#else struct vhost_net *vhost_net_init (VhostNetOptions *options) {@@ -463,4 +476,9 @@ VHostNetState *get_vhost_net(NetClientState *nc) {return 0 ; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +return 0 ; +}#endif hw/net/virtio-net.c +1
-0@@ -407,6 +407,7 @@ static int peer_attach(VirtIONet *n, int index) }if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,1 );return 0 ; }@@ -422,6 +423,7 @@ static int peer_detach(VirtIONet *n, int index) }Quoted 8 lines
hw/net/virtio-net.c static int peer_detach(VirtIONet *n, int index) · line 429
Quoted 5 lines
hw/net/virtio-net.c +1
-0if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,0 );return 0 ; }makes no sense to call it for !== tap only. Either call this unconditionally, or just for vhost user.
Agreed. I will make it vhost-usre specific then.
Quoted 57 lines
Quoted 57 lines
hw/virtio/vhost-user.c +23
-2@@ -25,9 +25,10 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0xfULL -#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_VRING_FLAG 1 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,@@ -48,6 +49,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 ,VHOST_USER_GET_QUEUE_NUM =17 , +VHOST_USER_SET_VRING_FLAG =18 , VHOST_USER_MAX } VhostUserRequest;@@ -296,6 +298,11 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ; +case VHOST_USER_SET_VRING_FLAG : + msg.state.index = dev->vq_index; + msg.state.num = *(int *)arg; + msg.size =sizeof (msg.state); +break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state ));@@ -408,6 +415,22 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)return 0 ; } +static int vhost_user_set_vring_flag (struct vhost_dev *dev,int enable) +{ +int err; + +assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); + +if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_VRING_FLAG ))) +return -1 ; + + err =vhost_user_call (dev,VHOST_USER_SET_VRING_FLAG , &enable); +if (err <0 ) +return err; + +return 0 ;Quoted 8 lines
hw/virtio/vhost-user.c static int vhost_user_init(struct vhost_dev *dev, void *opaque) · line 436
Quoted 5 lines
hw/virtio/vhost-user.c +2
-0+} +static int vhost_user_cleanup (struct vhost_dev *dev) {assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER );In fact, it's a per VQ pair call the way it's coded. So pls fix it, call it for all VQs.
Let me try.
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-user.c +3
-2@@ -421,5 +444,6 @@ const VhostOps user_ops = { .backend_type = VHOST_BACKEND_TYPE_USER, .vhost_call = vhost_user_call, .vhost_backend_init = vhost_user_init, - .vhost_backend_cleanup = vhost_user_cleanup - }; + .vhost_backend_cleanup = vhost_user_cleanup, + .vhost_backend_mq_set_vring_flag = vhost_user_set_vring_flag, +};include/hw/virtio/vhost-backend.h +1
-0@@ -24,12 +24,14 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, void *arg);typedef int (*vhost_backend_init )(struct vhost_dev *dev,void *opaque);typedef int (*vhost_backend_cleanup )(struct vhost_dev *dev); +typedef int (*vhost_backend_mq_set_vring_flag )(struct vhost_dev *dev,int enable);Quoted 9 lines
include/hw/virtio/vhost-backend.h typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, · line 34
Quoted 6 lines
include/hw/virtio/vhost-backend.h +1
-0typedef struct VhostOps { VhostBackendType backend_type; vhost_call vhost_call; vhost_backend_init vhost_backend_init; vhost_backend_cleanup vhost_backend_cleanup; + vhost_backend_mq_set_vring_flag vhost_backend_mq_set_vring_flag;That's quite ugly: mq is a vhost net thing. Why not enable each VQ?
Quoted 5 lines
To define it as `vhost_backend_enable_vring'?
Thanks.
--yliu
Sure.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
include/hw/virtio/vhost-backend.h +0
-0} VhostOps;extern const VhostOps user_ops;include/net/vhost_net.h +1
-0@@ -28,4 +28,5 @@ bool vhost_net_virtqueue_pending(VHostNetState *net, int n);void vhost_net_virtqueue_mask (VHostNetState *net,VirtIODevice *dev,int idx,bool mask);VHostNetState *get_vhost_net (NetClientState *nc); +int vhost_set_vring_flag (NetClientState * nc,int enable);#endif Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
In reply to Yuanhan LiuQuoted 87 lines · Yuanhan Liu
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
Add a new message,
VHOST_USER_SET_VRING_FLAG, to enable and disable a specific virt queue, which is similar to attach/detach queue for tap device.Quoted 8 lines
Quoted 6 lines
virtio driver on guest doesn't have to use max virt queue pair, it could enable any number of virt queue ranging from 1 to max virt queue pair.
Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>Fine, but I would like to make this dependent on the MQ flag.
Reasonable.
Quoted 11 lines
Quoted 11 lines
--- docs/specs/vhost-user.txt | 10 ++++++++++ hw/net/vhost_net.c | 18 ++++++++++++++++++ hw/net/virtio-net.c | 2 ++ hw/virtio/vhost-user.c | 32 ++++++++++++++++++++++++++++---- include/hw/virtio/vhost-backend.h | 2 ++ include/net/vhost_net.h | 1 + 6 files changed, 61 insertions(+), 4 deletions(-)Quoted 13 lines
docs/specs/vhost-user.txt Message types · line 329
Quoted 10 lines
docs/specs/vhost-user.txt +2
-0@@ -325,3 +325,13 @@ Message types Query how many queues the backend supports. This request should be sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol features by VHOST_USER_GET_PROTOCOL_FEATURES. + + * VHOST_USER_SET_VRING_FLAG
SET_VRIHG_ENABLEwould be a better name.Okay.
Quoted 9 lines
docs/specs/vhost-user.txt Message types · line 335
Quoted 7 lines
docs/specs/vhost-user.txt +6
-0+ + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Set the flag(1 for enable and 0 for disable) to signal slave to enableSpace before ( please.
Okay.
Quoted 9 lines
Quoted 9 lines
docs/specs/vhost-user.txt +2
-0+ or disable corresponding virt queue. This request should be sent only + when the protocol feature bit VHOST_USER_PROTOCOL_F_VRING_FLAG is set.hw/net/vhost_net.c +0
-0@@ -418,6 +418,19 @@ VHostNetState *get_vhost_net(NetClientState *nc)Quoted 10 lines
hw/net/vhost_net.c VHostNetState *get_vhost_net(NetClientState *nc) · line 426
Quoted 8 lines
hw/net/vhost_net.c +6
-0return vhost_net; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +if (nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) { +VHostNetState *net =get_vhost_net (nc); +const VhostOps *vhost_ops = net->dev.vhost_ops;add an empty line after declaration pls.
Got it, thanks.
Quoted 6 lines
hw/net/vhost_net.c VHostNetState *get_vhost_net(NetClientState *nc) · line 428
Quoted 3 lines
hw/net/vhost_net.c +2
-0+if (vhost_ops->vhost_backend_mq_set_vring_flag) +return vhost_ops->vhost_backend_mq_set_vring_flag (&net->dev, enable);Coding style violation:
line too long,
Quoted 7 lines
Yeah, 81 chars. I thougth that's kind of acceptable. But, anyway, since you pointed it out, I will fix it next time.
Quoted 1 line
and should use {}.
TBH, I seldom use {} for one statement, and you could see that a lot from this patch set. Is it a must in qemu community?
Yes it is. It's different from the kernel.
Quoted 146 lines
If so, I could fix them all.
Quoted 34 lines
Quoted 34 lines
hw/net/vhost_net.c +10
-0+ } + +return 0 ; +} +#else struct vhost_net *vhost_net_init (VhostNetOptions *options) {@@ -463,4 +476,9 @@ VHostNetState *get_vhost_net(NetClientState *nc) {return 0 ; } + +int vhost_set_vring_flag (NetClientState *nc,int enable) +{ +return 0 ; +}#endif hw/net/virtio-net.c +1
-0@@ -407,6 +407,7 @@ static int peer_attach(VirtIONet *n, int index) }if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,1 );return 0 ; }@@ -422,6 +423,7 @@ static int peer_detach(VirtIONet *n, int index) }Quoted 8 lines
hw/net/virtio-net.c static int peer_detach(VirtIONet *n, int index) · line 429
Quoted 5 lines
hw/net/virtio-net.c +1
-0if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) { +vhost_set_vring_flag (nc->peer,0 );return 0 ; }makes no sense to call it for !== tap only. Either call this unconditionally, or just for vhost user.
Agreed. I will make it vhost-usre specific then.
Quoted 57 lines
Quoted 57 lines
hw/virtio/vhost-user.c +23
-2@@ -25,9 +25,10 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0xfULL -#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_MQ 0 +#define VHOST_USER_PROTOCOL_F_VRING_FLAG 1 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,@@ -48,6 +49,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 ,VHOST_USER_GET_QUEUE_NUM =17 , +VHOST_USER_SET_VRING_FLAG =18 , VHOST_USER_MAX } VhostUserRequest;@@ -296,6 +298,11 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, msg.addr.index += dev->vq_index; msg.size =sizeof (m.state);break ; +case VHOST_USER_SET_VRING_FLAG : + msg.state.index = dev->vq_index; + msg.state.num = *(int *)arg; + msg.size =sizeof (msg.state); +break ;case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state ));@@ -408,6 +415,22 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)return 0 ; } +static int vhost_user_set_vring_flag (struct vhost_dev *dev,int enable) +{ +int err; + +assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); + +if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_VRING_FLAG ))) +return -1 ; + + err =vhost_user_call (dev,VHOST_USER_SET_VRING_FLAG , &enable); +if (err <0 ) +return err; + +return 0 ;Quoted 8 lines
hw/virtio/vhost-user.c static int vhost_user_init(struct vhost_dev *dev, void *opaque) · line 436
Quoted 5 lines
hw/virtio/vhost-user.c +2
-0+} +static int vhost_user_cleanup (struct vhost_dev *dev) {assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER );In fact, it's a per VQ pair call the way it's coded. So pls fix it, call it for all VQs.
Let me try.
Quoted 21 lines
Quoted 21 lines
hw/virtio/vhost-user.c +3
-2@@ -421,5 +444,6 @@ const VhostOps user_ops = { .backend_type = VHOST_BACKEND_TYPE_USER, .vhost_call = vhost_user_call, .vhost_backend_init = vhost_user_init, - .vhost_backend_cleanup = vhost_user_cleanup - }; + .vhost_backend_cleanup = vhost_user_cleanup, + .vhost_backend_mq_set_vring_flag = vhost_user_set_vring_flag, +};include/hw/virtio/vhost-backend.h +1
-0@@ -24,12 +24,14 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, void *arg);typedef int (*vhost_backend_init )(struct vhost_dev *dev,void *opaque);typedef int (*vhost_backend_cleanup )(struct vhost_dev *dev); +typedef int (*vhost_backend_mq_set_vring_flag )(struct vhost_dev *dev,int enable);Quoted 9 lines
include/hw/virtio/vhost-backend.h typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, · line 34
Quoted 6 lines
include/hw/virtio/vhost-backend.h +1
-0typedef struct VhostOps { VhostBackendType backend_type; vhost_call vhost_call; vhost_backend_init vhost_backend_init; vhost_backend_cleanup vhost_backend_cleanup; + vhost_backend_mq_set_vring_flag vhost_backend_mq_set_vring_flag;That's quite ugly: mq is a vhost net thing. Why not enable each VQ?
Quoted 5 lines
To define it as `vhost_backend_enable_vring'?
Thanks.
--yliu
Sure.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
include/hw/virtio/vhost-backend.h +0
-0} VhostOps;extern const VhostOps user_ops;include/net/vhost_net.h +1
-0@@ -28,4 +28,5 @@ bool vhost_net_virtqueue_pending(VHostNetState *net, int n);void vhost_net_virtqueue_mask (VHostNetState *net,VirtIODevice *dev,int idx,bool mask);VHostNetState *get_vhost_net (NetClientState *nc); +int vhost_set_vring_flag (NetClientState * nc,int enable);#endif Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
In reply to Michael S. TsirkinQuoted 1 line · Michael S. Tsirkin
On Tue, Sep 08, 2015 at 03:38:47PM +0800, Yuanhan Liu wrote:
[snip ....]
Quoted 5 lines
Quoted 5 lines
hw/virtio/vhost-user.c +0
-0@@ -25,9 +25,10 @@
[snip ....]
Quoted 6 lines
Quoted 3 lines
static int
vhost_user_cleanup(struct vhost_dev *dev){ assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER);In fact, it's a per VQ pair call the way it's coded. So pls fix it, call it for all VQs.
I'm trying to address this comment, and I'm a bit confused now: this call is based on one 'vhost_net' structure, which is related to one 'vhost_dev' structure. One 'vhost_dev', on the other hand, represents two VQs.
So, you are suggesting to call vhost_user_cleanup() twice at it's upper level?
--yliu
---
hw/virtio/vhost.c +4 -1
@@ -988,7 +988,10 @@ void vhost_dev_cleanup(struct vhost_dev *hdev) }g_free (hdev->mem);g_free (hdev->mem_sections); - hdev->vhost_ops->vhost_backend_cleanup (hdev); + +for (i =0 ; i < hdev->nvqs; i++) { + hdev->vhost_ops->vhost_backend_cleanup (hdev, i); + } }/* Stop processing guest IO notifications in qemu.