vhost-user multiple queue support
v10 cover vhost-user multiple queue support
Hi,
Here is the updated patch set for enabling vhost-user multiple queue. And I did proper and formal testing this time.
This patch set introduces 2 more vhost user messages: VHOST_USER_GET_QUEUE_NUM, for querying how many queues the backend supports, and VHOST_USER_SET_VRING_ENABLE, for enabling/disabling a specific virt queue.
Both of the two new messages are treated as vhost protocol extension, and that's why Michaels's patch "vhost-user: add protocol feature negotiation" is also included here.
Patch 1-5 are all prepare works for actually enabling multiple queue.
Patch 6 is the major patch for enabling multiple queue, which also tries to address two major concerns from Michael: no feedback from backend if it can't support # of requested queues, and all messages are sent N time. It also fixes a hidden bug.
Patch 7 introduces the VHOST_USER_SET_VRING_ENABLE message, to enable or disable a specific vring.
v10: - typo fixes pointed out by Eric
- [PATCH 6]: don't treat VHOST_USER_SET/GET_[PROTOCOL]_FEATURES as one time request, as the two feature bits need to be stored at per-device.
v9: - Per suggested by Jason Wang, patch 5 introduces a new vhost
backend method: vhost_backend_get_vq_index().
- Use qemu_find_net_clients_except() at net/vhost-user.c for gathering all related ncs so that we could register chr dev event handler once. Which is also suggested by Jason Wang.
Thanks.
--yliu
---
Changchun Ouyang (2):
vhost-user: add multiple queue support
vhost-user: add a new message to disable/enable a specific virt queue.
Michael S. Tsirkin (1):
vhost-user: add protocol feature negotiation
Yuanhan Liu (4):
vhost-user: use VHOST_USER_XXX macro for switch statement
vhost: rename VHOST_RESET_OWNER to VHOST_RESET_DEVICE
vhost-user: add VHOST_USER_GET_QUEUE_NUM message
vhost: introduce vhost_backend_get_vq_index method
docs/specs/vhost-user.txt | 77 +++++++++++++++++++- hw/net/vhost_net.c | 39 ++++++++-- hw/net/virtio-net.c | 8 +++ hw/virtio/vhost-backend.c | 10 ++- hw/virtio/vhost-user.c | 139 ++++++++++++++++++++++++++++++------ hw/virtio/vhost.c | 20 +++--- include/hw/virtio/vhost-backend.h | 4 ++ include/hw/virtio/vhost.h | 2 + include/net/vhost_net.h | 3 + linux-headers/linux/vhost.h | 2 +- net/vhost-user.c | 146 +++++++++++++++++++++++++++----------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- tests/vhost-user-test.c | 2 +- 14 files changed, 378 insertions(+), 85 deletions(-)
Signature · 1 line
1.9.0
PATCH v10 1/7 vhost-user: use VHOST_USER_XXX macro for switch statement
So that we could let vhost_user_call to handle extented requests, such as VHOST_USER_GET/SET_PROTOCOL_FEATURES, instead of invoking vhost_user_read/write and constructing the msg again by ourself.
--- hw/virtio/vhost-user.c | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-)
hw/virtio/vhost-user.c +22 -16
@@ -193,27 +193,33 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); - msg_request =vhost_user_request_translate (request); +/* only translate vhost ioctl requests */ +if (request >VHOST_USER_MAX ) { + msg_request =vhost_user_request_translate (request); + }else { + msg_request = request; + } + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ; -switch (request) { -case VHOST_GET_FEATURES : +switch (msg_request) { +case VHOST_USER_GET_FEATURES : need_reply =1 ;break ; -case VHOST_SET_FEATURES : -case VHOST_SET_LOG_BASE : +case VHOST_USER_SET_FEATURES : +case VHOST_USER_SET_LOG_BASE : msg.u64 = *((__u64 *) arg); msg.size =sizeof (m.u64);break ; -case VHOST_SET_OWNER : -case VHOST_RESET_OWNER : +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_OWNER :break ; -case VHOST_SET_MEM_TABLE : +case VHOST_USER_SET_MEM_TABLE :for (i =0 ; i < dev->mem->nregions; ++i) {struct vhost_memory_region *reg = dev->mem->regions + i;ram_addr_t ram_addr;
@@ -246,30 +252,30 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,break ; -case VHOST_SET_LOG_FD : +case VHOST_USER_SET_LOG_FD : fds[fd_num++] = *((int *) arg);break ; -case VHOST_SET_VRING_NUM : -case VHOST_SET_VRING_BASE : +case VHOST_USER_SET_VRING_NUM : +case VHOST_USER_SET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); msg.size =sizeof (m.state);break ; -case VHOST_GET_VRING_BASE : +case VHOST_USER_GET_VRING_BASE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); msg.size =sizeof (m.state); need_reply =1 ;break ; -case VHOST_SET_VRING_ADDR : +case VHOST_USER_SET_VRING_ADDR :memcpy (&msg.addr, arg,sizeof (struct vhost_vring_addr )); msg.size =sizeof (m.addr);break ; -case VHOST_SET_VRING_KICK : -case VHOST_SET_VRING_CALL : -case VHOST_SET_VRING_ERR : +case VHOST_USER_SET_VRING_KICK : +case VHOST_USER_SET_VRING_CALL : +case VHOST_USER_SET_VRING_ERR : file = arg; msg.u64 = file->index &VHOST_USER_VRING_IDX_MASK ; msg.size =sizeof (m.u64);
Signature · 1 line
1.9.0
PATCH v10 2/7 vhost-user: add protocol feature negotiation
In reply to Yuanhan LiuFrom: "Michael S. Tsirkin" <[email protected]>
Support a separate bitmask for vhost-user protocol features, and messages to get/set protocol features.
Invoke them at init.
No features are defined yet.
[ leverage vhost_user_call for request handling -- Yuanhan Liu ]
--- docs/specs/vhost-user.txt | 37 +++++++++++++++++++++++++++++++++++++ hw/net/vhost_net.c | 2 ++ hw/virtio/vhost-user.c | 31 +++++++++++++++++++++++++++++++ include/hw/virtio/vhost.h | 1 + 4 files changed, 71 insertions(+)
docs/specs/vhost-user.txt +37 -0
@@ -113,6 +113,7 @@ message replies. Most of the requests don't require replies. Here is a list of the ones that do: * VHOST_GET_FEATURES + * VHOST_GET_PROTOCOL_FEATURES * VHOST_GET_VRING_BASE There are several messages that the master sends with file descriptors passed
@@ -127,6 +128,13 @@ in the ancillary data: If Master is unable to send the full message or receives a wrong reply it will close the connection. An optional reconnection mechanism can be implemented. +Any protocol extensions are gated by protocol feature bits, +which allows full backwards compatibility on both master +and slave. +As older slaves don't support negotiating protocol features, +a feature bit was dedicated for this purpose: +#define VHOST_USER_F_PROTOCOL_FEATURES 30 + Message types -------------
@@ -138,6 +146,8 @@ Message types Slave payload: u64 Get from the underlying vhost implementation the features bitmask. + Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for + VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES. * VHOST_USER_SET_FEATURES
@@ -146,6 +156,33 @@ Message types Master payload: u64 Enable features in the underlying vhost implementation using a bitmask. + Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for + VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES. + + * VHOST_USER_GET_PROTOCOL_FEATURES + + Id: 15 + Equivalent ioctl: VHOST_GET_FEATURES + Master payload: N/A + Slave payload: u64 + + Get the protocol feature bitmask from the underlying vhost implementation. + Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in + VHOST_USER_GET_FEATURES. + Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support + this message even before VHOST_USER_SET_FEATURES was called. + + * VHOST_USER_SET_PROTOCOL_FEATURES + + Id: 16 + Ioctl: VHOST_SET_FEATURES + Master payload: u64 + + Enable protocol features in the underlying vhost implementation. + Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in + VHOST_USER_GET_FEATURES. + Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support + this message even before VHOST_USER_SET_FEATURES was called. * VHOST_USER_SET_OWNER
hw/net/vhost_net.c +2 -0
@@ -152,8 +152,10 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) net->dev.backend_features =qemu_has_vnet_hdr (options->net_backend) ?0 : (1ULL <<VHOST_NET_F_VIRTIO_NET_HDR ); net->backend = r; + net->dev.protocol_features =0 ; }else { net->dev.backend_features =0 ; + net->dev.protocol_features =0 ; net->backend =-1 ; } net->nc = options->net_backend;
hw/virtio/vhost-user.c +31 -0
@@ -24,6 +24,8 @@#include <linux/vhost.h> #define VHOST_MEMORY_MAX_NREGIONS 8 +#define VHOST_USER_F_PROTOCOL_FEATURES 30 +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x0ULL typedef enum VhostUserRequest { VHOST_USER_NONE = 0,
@@ -41,6 +43,8 @@ typedef enum VhostUserRequest {VHOST_USER_SET_VRING_KICK =12 ,VHOST_USER_SET_VRING_CALL =13 ,VHOST_USER_SET_VRING_ERR =14 , +VHOST_USER_GET_PROTOCOL_FEATURES =15 , +VHOST_USER_SET_PROTOCOL_FEATURES =16 , VHOST_USER_MAX } VhostUserRequest;
@@ -206,11 +210,13 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES : need_reply =1 ;break ;case VHOST_USER_SET_FEATURES :case VHOST_USER_SET_LOG_BASE : +case VHOST_USER_SET_PROTOCOL_FEATURES : msg.u64 = *((__u64 *) arg); msg.size =sizeof (m.u64);break ;
@@ -308,6 +314,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES : +case VHOST_USER_GET_PROTOCOL_FEATURES :if (msg.size !=sizeof (m.u64)) {error_report ("Received bad msg size." );return -1 ;
@@ -333,10 +340,34 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,static int vhost_user_init (struct vhost_dev *dev,void *opaque) { +unsigned long long features; +int err; +assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); dev->opaque = opaque; + err =vhost_user_call (dev,VHOST_USER_GET_FEATURES , &features); +if (err <0 ) { +return err; + } + +if (virtio_has_feature (features,VHOST_USER_F_PROTOCOL_FEATURES )) { + dev->backend_features |=1ULL <<VHOST_USER_F_PROTOCOL_FEATURES ; + + err =vhost_user_call (dev,VHOST_USER_GET_PROTOCOL_FEATURES , &features); +if (err <0 ) { +return err; + } + + dev->protocol_features = features &VHOST_USER_PROTOCOL_FEATURE_MASK ; + err =vhost_user_call (dev,VHOST_USER_SET_PROTOCOL_FEATURES , + &dev->protocol_features); +if (err <0 ) { +return err; + } + } +return 0 ; }
include/hw/virtio/vhost.h +1 -0
@@ -47,6 +47,7 @@ struct vhost_dev {unsigned long long features;unsigned long long acked_features;unsigned long long backend_features; +unsigned long long protocol_features;bool started;bool log_enabled;unsigned long long log_size;
Signature · 1 line
1.9.0
PATCH v10 3/7 vhost: rename VHOST_RESET_OWNER to VHOST_RESET_DEVICE
In reply to Yuanhan LiuQuote from Michael:
We really should rename VHOST_RESET_OWNER to VHOST_RESET_DEVICE.
--- docs/specs/vhost-user.txt | 4 ++-- hw/net/vhost_net.c | 2 +- hw/virtio/vhost-user.c | 6 +++--- linux-headers/linux/vhost.h | 2 +- tests/vhost-user-test.c | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-)
docs/specs/vhost-user.txt +2 -2
@@ -194,10 +194,10 @@ Message types as an owner of the session. This can be used on the Slave as a "session start" flag. - * VHOST_USER_RESET_OWNER + * VHOST_USER_RESET_DEVICE Id: 4 - Equivalent ioctl: VHOST_RESET_OWNER + Equivalent ioctl: VHOST_RESET_DEVICE Master payload: N/A Issued when a new connection is about to be closed. The Master will no
hw/net/vhost_net.c +1 -1
@@ -287,7 +287,7 @@ static void vhost_net_stop_one(struct vhost_net *net, }else if (net->nc->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) {for (file.index =0 ; file.index < net->dev.nvqs; ++file.index) {const VhostOps *vhost_ops = net->dev.vhost_ops; -int r = vhost_ops->vhost_call (&net->dev,VHOST_RESET_OWNER , +int r = vhost_ops->vhost_call (&net->dev,VHOST_RESET_DEVICE ,NULL );assert (r >=0 ); }
hw/virtio/vhost-user.c +3 -3
@@ -32,7 +32,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_FEATURES =1 ,VHOST_USER_SET_FEATURES =2 ,VHOST_USER_SET_OWNER =3 , -VHOST_USER_RESET_OWNER =4 , +VHOST_USER_RESET_DEVICE =4 ,VHOST_USER_SET_MEM_TABLE =5 ,VHOST_USER_SET_LOG_BASE =6 , VHOST_USER_SET_LOG_FD = 7,
@@ -98,7 +98,7 @@ static unsigned long int ioctl_to_vhost_user_request[VHOST_USER_MAX] = {VHOST_GET_FEATURES ,/* VHOST_USER_GET_FEATURES */ VHOST_SET_FEATURES ,/* VHOST_USER_SET_FEATURES */ VHOST_SET_OWNER ,/* VHOST_USER_SET_OWNER */ -VHOST_RESET_OWNER ,/* VHOST_USER_RESET_OWNER */ +VHOST_RESET_DEVICE ,/* VHOST_USER_RESET_DEVICE */ VHOST_SET_MEM_TABLE ,/* VHOST_USER_SET_MEM_TABLE */ VHOST_SET_LOG_BASE ,/* VHOST_USER_SET_LOG_BASE */ VHOST_SET_LOG_FD, /* VHOST_USER_SET_LOG_FD */
@@ -222,7 +222,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,break ;case VHOST_USER_SET_OWNER : -case VHOST_USER_RESET_OWNER : +case VHOST_USER_RESET_DEVICE :break ;case VHOST_USER_SET_MEM_TABLE :
linux-headers/linux/vhost.h +1 -1
@@ -78,7 +78,7 @@ struct vhost_memory {#define VHOST_SET_OWNER _IO(VHOST_VIRTIO, 0x01)/* Give up ownership, and reset the device to default values. * Allows subsequent call to VHOST_OWNER_SET to succeed. */ -#define VHOST_RESET_OWNER _IO(VHOST_VIRTIO, 0x02) +#define VHOST_RESET_DEVICE _IO(VHOST_VIRTIO, 0x02)/* Set up/modify memory layout */ #define VHOST_SET_MEM_TABLE _IOW(VHOST_VIRTIO, 0x03, struct vhost_memory)
tests/vhost-user-test.c +1 -1
@@ -58,7 +58,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_FEATURES =1 ,VHOST_USER_SET_FEATURES =2 ,VHOST_USER_SET_OWNER =3 , -VHOST_USER_RESET_OWNER =4 , +VHOST_USER_RESET_DEVICE =4 ,VHOST_USER_SET_MEM_TABLE =5 ,VHOST_USER_SET_LOG_BASE =6 , VHOST_USER_SET_LOG_FD = 7,
Signature · 1 line
1.9.0
PATCH v10 4/7 vhost-user: add VHOST_USER_GET_QUEUE_NUM message
In reply to Yuanhan LiuThis is for querying how many queues the backend supports if it has mq support(when VHOST_USER_PROTOCOL_F_MQ flag is set from the quried protocol features).
vhost_net_get_max_queues() is the interface to export that value, and to tell if the backend supports # of queues user requested, which is done in the following patch.
--- docs/specs/vhost-user.txt | 11 +++++++++++ hw/net/vhost_net.c | 7 +++++++ hw/virtio/vhost-user.c | 15 ++++++++++++++- include/hw/virtio/vhost.h | 1 + include/net/vhost_net.h | 1 + 5 files changed, 34 insertions(+), 1 deletion(-)
docs/specs/vhost-user.txt +11 -0
@@ -301,3 +301,14 @@ Message types Bits (0-7) of the payload contain the vring index. Bit 8 is the invalid FD flag. This flag is set when there is no file descriptor in the ancillary data. + + * VHOST_USER_GET_QUEUE_NUM + + Id: 17 + Equivalent ioctl: N/A + Master payload: N/A + Slave payload: u64 + + Query how many queues the backend supports. This request should be + sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol + features by VHOST_USER_GET_PROTOCOL_FEATURES.
hw/net/vhost_net.c +7 -0
@@ -122,6 +122,11 @@ void vhost_net_ack_features(struct vhost_net *net, uint64_t features)vhost_ack_features (&net->dev,vhost_net_get_feature_bits (net), features); } +uint64_t vhost_net_get_max_queues (VHostNetState *net) +{ +return net->dev.max_queues; +} +static int vhost_net_get_fd (NetClientState *backend) {switch (backend->info->type) {
@@ -144,6 +149,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) goto fail; } + net->dev.max_queues =1 ; +if (backend_kernel) { r =vhost_net_get_fd (options->net_backend);if (r <0 ) {
hw/virtio/vhost-user.c +14 -1
@@ -25,7 +25,9 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x0ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL + +#define VHOST_USER_PROTOCOL_F_MQ 0 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,
@@ -45,6 +47,7 @@ typedef enum VhostUserRequest {VHOST_USER_SET_VRING_ERR =14 ,VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 , +VHOST_USER_GET_QUEUE_NUM =17 , VHOST_USER_MAX } VhostUserRequest;
@@ -211,6 +214,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES :case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_GET_QUEUE_NUM : need_reply =1 ;break ;
@@ -315,6 +319,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES :case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_GET_QUEUE_NUM :if (msg.size !=sizeof (m.u64)) {error_report ("Received bad msg size." );return -1 ;
@@ -366,6 +371,14 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)if (err <0 ) {return err; } + +/* query the max queues we support if backend supports Multiple Queue */ +if (dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_MQ )) { + err =vhost_user_call (dev,VHOST_USER_GET_QUEUE_NUM , &dev->max_queues); +if (err <0 ) { +return err; + } + } }return 0 ;
include/hw/virtio/vhost.h +1 -0
@@ -48,6 +48,7 @@ struct vhost_dev {unsigned long long acked_features;unsigned long long backend_features;unsigned long long protocol_features; +unsigned long long max_queues;bool started;bool log_enabled;unsigned long long log_size;
include/net/vhost_net.h +1 -0
@@ -13,6 +13,7 @@ typedef struct VhostNetOptions {void *opaque; } VhostNetOptions; +uint64_t vhost_net_get_max_queues (VHostNetState *net);struct vhost_net *vhost_net_init (VhostNetOptions *options);int vhost_net_start (VirtIODevice *dev,NetClientState *ncs,int total_queues);
Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH v10 4/7] vhost-user: add VHOST_USER_GET_QUEUE_NUM message
Quoted 41 lines · Yuanhan Liu
This is for querying how many queues the backend supports if it has mq
support(when VHOST_USER_PROTOCOL_F_MQ flag is set from the quried protocol features).
vhost_net_get_max_queues()is the interface to export that value, and to tell if the backend supports # of queues user requested, which is done in the following patch.Signed-off-by Yuanhan Liu <[email protected]>--- docs/specs/vhost-user.txt | 11 +++++++++++ hw/net/vhost_net.c | 7 +++++++ hw/virtio/vhost-user.c | 15 ++++++++++++++- include/hw/virtio/vhost.h | 1 + include/net/vhost_net.h | 1 + 5 files changed, 34 insertions(+), 1 deletion(-)docs/specs/vhost-user.txt +11
-0@@ -301,3 +301,14 @@ Message types Bits (0-7) of the payload contain the vring index. Bit 8 is the invalid FD flag. This flag is set when there is no file descriptor in the ancillary data. + + * VHOST_USER_GET_QUEUE_NUM + + Id: 17 + Equivalent ioctl: N/A + Master payload: N/A + Slave payload: u64 + + Query how many queues the backend supports. This request should be + sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol + features by VHOST_USER_GET_PROTOCOL_FEATURES.hw/net/vhost_net.c +0
-0
hw/net/vhost_net.c void vhost_net_ack_features(struct vhost_net *net, uint64_t features) · line 128
Quoted 7 lines
hw/net/vhost_net.c +4
-0@@ -122,6 +122,11 @@ void vhost_net_ack_features(struct vhost_net *net, uint64_t features)vhost_ack_features (&net->dev,vhost_net_get_feature_bits (net), features); } +uint64_t vhost_net_get_max_queues (VHostNetState *net) +{ +return net->dev.max_queues; +}
Looks like need a dummy function for !CONFIG_VHOST_NET.
Quoted 91 lines
hw/net/vhost_net.c +3
-0+static int vhost_net_get_fd (NetClientState *backend) {switch (backend->info->type) {@@ -144,6 +149,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) goto fail; } + net->dev.max_queues =1 ; +if (backend_kernel) { r =vhost_net_get_fd (options->net_backend);if (r <0 ) {hw/virtio/vhost-user.c +14
-1@@ -25,7 +25,9 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x0ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL + +#define VHOST_USER_PROTOCOL_F_MQ 0 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,@@ -45,6 +47,7 @@ typedef enum VhostUserRequest {VHOST_USER_SET_VRING_ERR =14 ,VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 , +VHOST_USER_GET_QUEUE_NUM =17 , VHOST_USER_MAX } VhostUserRequest;@@ -211,6 +214,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES :case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_GET_QUEUE_NUM : need_reply =1 ;break ;@@ -315,6 +319,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES :case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_GET_QUEUE_NUM :if (msg.size !=sizeof (m.u64)) {error_report ("Received bad msg size." );return -1 ;@@ -366,6 +371,14 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)if (err <0 ) {return err; } + +/* query the max queues we support if backend supports Multiple Queue */ +if (dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_MQ )) { + err =vhost_user_call (dev,VHOST_USER_GET_QUEUE_NUM , &dev->max_queues); +if (err <0 ) { +return err; + } + } }return 0 ;include/hw/virtio/vhost.h +1
-0@@ -48,6 +48,7 @@ struct vhost_dev {unsigned long long acked_features;unsigned long long backend_features;unsigned long long protocol_features; +unsigned long long max_queues;bool started;bool log_enabled;unsigned long long log_size;include/net/vhost_net.h +1
-0@@ -13,6 +13,7 @@ typedef struct VhostNetOptions {void *opaque; } VhostNetOptions; +uint64_t vhost_net_get_max_queues (VHostNetState *net);struct vhost_net *vhost_net_init (VhostNetOptions *options);int vhost_net_start (VirtIODevice *dev,NetClientState *ncs,int total_queues);
Re: [Qemu-devel] [PATCH v10 4/7] vhost-user: add VHOST_USER_GET_QUEUE_NUM message
Quoted 47 lines · Jason Wang
Quoted 44 lines · Yuanhan Liu
This is for querying how many queues the backend supports if it has mq
support(when VHOST_USER_PROTOCOL_F_MQ flag is set from the quried protocol features).
vhost_net_get_max_queues()is the interface to export that value, and to tell if the backend supports # of queues user requested, which is done in the following patch.Signed-off-by Yuanhan Liu <[email protected]>--- docs/specs/vhost-user.txt | 11 +++++++++++ hw/net/vhost_net.c | 7 +++++++ hw/virtio/vhost-user.c | 15 ++++++++++++++- include/hw/virtio/vhost.h | 1 + include/net/vhost_net.h | 1 + 5 files changed, 34 insertions(+), 1 deletion(-)docs/specs/vhost-user.txt +11
-0@@ -301,3 +301,14 @@ Message types Bits (0-7) of the payload contain the vring index. Bit 8 is the invalid FD flag. This flag is set when there is no file descriptor in the ancillary data. + + * VHOST_USER_GET_QUEUE_NUM + + Id: 17 + Equivalent ioctl: N/A + Master payload: N/A + Slave payload: u64 + + Query how many queues the backend supports. This request should be + sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol + features by VHOST_USER_GET_PROTOCOL_FEATURES.hw/net/vhost_net.c +0
-0@@ -122,6 +122,11 @@ void vhost_net_ack_features(struct vhost_net *net, uint64_t features)vhost_ack_features (&net->dev,vhost_net_get_feature_bits (net), features); }
Quoted 6 lines
hw/net/vhost_net.c void vhost_net_ack_features(struct vhost_net *net, uint64_t features) · line 128
Quoted 4 lines
hw/net/vhost_net.c +4
-0+uint64_t vhost_net_get_max_queues (VHostNetState *net) +{ +return net->dev.max_queues; +}Looks like need a dummy function for !CONFIG_VHOST_NET.
Yes, it is. A such build error report has been already reported by some one.
--yliu
Quoted 92 lines
Quoted 92 lines
hw/net/vhost_net.c +3
-0+static int vhost_net_get_fd (NetClientState *backend) {switch (backend->info->type) {@@ -144,6 +149,8 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) goto fail; } + net->dev.max_queues =1 ; +if (backend_kernel) { r =vhost_net_get_fd (options->net_backend);if (r <0 ) {hw/virtio/vhost-user.c +14
-1@@ -25,7 +25,9 @@#define VHOST_MEMORY_MAX_NREGIONS 8#define VHOST_USER_F_PROTOCOL_FEATURES 30 -#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x0ULL +#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL + +#define VHOST_USER_PROTOCOL_F_MQ 0 typedef enum VhostUserRequest { VHOST_USER_NONE = 0,@@ -45,6 +47,7 @@ typedef enum VhostUserRequest {VHOST_USER_SET_VRING_ERR =14 ,VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 , +VHOST_USER_GET_QUEUE_NUM =17 , VHOST_USER_MAX } VhostUserRequest;@@ -211,6 +214,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES :case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_GET_QUEUE_NUM : need_reply =1 ;break ;@@ -315,6 +319,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,switch (msg_request) {case VHOST_USER_GET_FEATURES :case VHOST_USER_GET_PROTOCOL_FEATURES : +case VHOST_USER_GET_QUEUE_NUM :if (msg.size !=sizeof (m.u64)) {error_report ("Received bad msg size." );return -1 ;@@ -366,6 +371,14 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)if (err <0 ) {return err; } + +/* query the max queues we support if backend supports Multiple Queue */ +if (dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_MQ )) { + err =vhost_user_call (dev,VHOST_USER_GET_QUEUE_NUM , &dev->max_queues); +if (err <0 ) { +return err; + } + } }return 0 ;include/hw/virtio/vhost.h +1
-0@@ -48,6 +48,7 @@ struct vhost_dev {unsigned long long acked_features;unsigned long long backend_features;unsigned long long protocol_features; +unsigned long long max_queues;bool started;bool log_enabled;unsigned long long log_size;include/net/vhost_net.h +1
-0@@ -13,6 +13,7 @@ typedef struct VhostNetOptions {void *opaque; } VhostNetOptions; +uint64_t vhost_net_get_max_queues (VHostNetState *net);struct vhost_net *vhost_net_init (VhostNetOptions *options);int vhost_net_start (VirtIODevice *dev,NetClientState *ncs,int total_queues);
PATCH v10 5/7 vhost: introduce vhost_backend_get_vq_index method
In reply to Yuanhan LiuMinusing the idx with the base(dev->vq_index) for vhost-kernel, and then adding it back for vhost-user doesn't seem right. Here introduces a new method vhost_backend_get_vq_index() for getting the right vq index for following vhost messages calls.
--- hw/virtio/vhost-backend.c | 10 +++++++++- hw/virtio/vhost-user.c | 12 ++++++++++-- hw/virtio/vhost.c | 15 ++++++--------- include/hw/virtio/vhost-backend.h | 2 ++ 4 files changed, 27 insertions(+), 12 deletions(-)
hw/virtio/vhost-backend.c +9 -1
@@ -42,11 +42,19 @@ static int vhost_kernel_cleanup(struct vhost_dev *dev)return close (fd); } +static int vhost_kernel_get_vq_index (struct vhost_dev *dev,int idx) +{ +assert (idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); + +return idx - dev->vq_index; +} +static const VhostOps kernel_ops = { .backend_type =VHOST_BACKEND_TYPE_KERNEL , .vhost_call = vhost_kernel_call, .vhost_backend_init = vhost_kernel_init, - .vhost_backend_cleanup = vhost_kernel_cleanup + .vhost_backend_cleanup = vhost_kernel_cleanup, + .vhost_backend_get_vq_index = vhost_kernel_get_vq_index, };int vhost_set_backend_type (struct vhost_dev *dev,VhostBackendType backend_type)
hw/virtio/vhost-user.c +10 -2
@@ -393,9 +393,17 @@ static int vhost_user_cleanup(struct vhost_dev *dev)return 0 ; } +static int vhost_user_get_vq_index (struct vhost_dev *dev,int idx) +{ +assert (idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); + +return idx; +} +const VhostOps user_ops = { .backend_type =VHOST_BACKEND_TYPE_USER , .vhost_call = vhost_user_call, .vhost_backend_init = vhost_user_init, - .vhost_backend_cleanup = vhost_user_cleanup - }; + .vhost_backend_cleanup = vhost_user_cleanup, + .vhost_backend_get_vq_index = vhost_user_get_vq_index, +};
hw/virtio/vhost.c +6 -9
@@ -719,7 +719,7 @@ static int vhost_virtqueue_start(struct vhost_dev *dev, {hwaddr s, l, a;int r; -int vhost_vq_index = idx - dev->vq_index; +int vhost_vq_index = dev->vhost_ops->vhost_backend_get_vq_index (dev, idx);struct vhost_vring_file file = { .index = vhost_vq_index };
@@ -728,7 +728,6 @@ static int vhost_virtqueue_start(struct vhost_dev *dev, };struct VirtQueue *vvq =virtio_get_queue (vdev, idx); -assert (idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); vq->num = state.num =virtio_queue_get_num (vdev, idx); r = dev->vhost_ops->vhost_call (dev,VHOST_SET_VRING_NUM , &state);
@@ -822,12 +821,12 @@ static void vhost_virtqueue_stop(struct vhost_dev *dev,struct vhost_virtqueue *vq, unsigned idx) { - int vhost_vq_index = idx - dev->vq_index; + int vhost_vq_index = dev->vhost_ops->vhost_backend_get_vq_index(dev, idx);struct vhost_vring_state state = { .index = vhost_vq_index, };int r; -assert (idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); + r = dev->vhost_ops->vhost_call (dev,VHOST_GET_VRING_BASE , &state);if (r <0 ) {fprintf (stderr,"vhost VQ %d ring restore failed: %d\n" , idx, r);
@@ -1066,17 +1065,15 @@ void vhost_virtqueue_mask(struct vhost_dev *hdev, VirtIODevice *vdev, int n, {struct VirtQueue *vvq =virtio_get_queue (vdev, n);int r, index = n - hdev->vq_index; +struct vhost_vring_file file; -assert (n >= hdev->vq_index && n < hdev->vq_index + hdev->nvqs); - -struct vhost_vring_file file = { - .index = index - };if (mask) { file.fd =event_notifier_get_fd (&hdev->vqs[index].masked_notifier); }else { file.fd =event_notifier_get_fd (virtio_queue_get_guest_notifier (vvq)); } + + file.index = hdev->vhost_ops->vhost_backend_get_vq_index (hdev, n); r = hdev->vhost_ops->vhost_call (hdev,VHOST_SET_VRING_CALL , &file);assert (r >=0 ); }
include/hw/virtio/vhost-backend.h +2 -0
@@ -24,12 +24,14 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, void *arg);typedef int (*vhost_backend_init )(struct vhost_dev *dev,void *opaque);typedef int (*vhost_backend_cleanup )(struct vhost_dev *dev); +typedef int (*vhost_backend_get_vq_index )(struct vhost_dev *dev,int idx);typedef struct VhostOps {VhostBackendType backend_type;vhost_call vhost_call;vhost_backend_init vhost_backend_init;vhost_backend_cleanup vhost_backend_cleanup; +vhost_backend_get_vq_index vhost_backend_get_vq_index; }VhostOps ;extern const VhostOps user_ops;
Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH v10 5/7] vhost: introduce vhost_backend_get_vq_index method
Quoted 7 lines · Yuanhan Liu
Minusing the idx with the
base(dev->vq_index)for vhost-kernel, and then adding it back for vhost-user doesn't seem right. Here introduces a new methodvhost_backend_get_vq_index()for getting the right vq index for following vhost messages calls.Suggested-by Jason Wang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>
Quoted 132 lines
--- hw/virtio/vhost-backend.c | 10 +++++++++- hw/virtio/vhost-user.c | 12 ++++++++++-- hw/virtio/vhost.c | 15 ++++++--------- include/hw/virtio/vhost-backend.h | 2 ++ 4 files changed, 27 insertions(+), 12 deletions(-)hw/virtio/vhost-backend.c +9
-1@@ -42,11 +42,19 @@ static int vhost_kernel_cleanup(struct vhost_dev *dev)return close (fd); } +static int vhost_kernel_get_vq_index (struct vhost_dev *dev,int idx) +{ +assert (idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); + +return idx - dev->vq_index; +} +static const VhostOps kernel_ops = { .backend_type =VHOST_BACKEND_TYPE_KERNEL , .vhost_call = vhost_kernel_call, .vhost_backend_init = vhost_kernel_init, - .vhost_backend_cleanup = vhost_kernel_cleanup + .vhost_backend_cleanup = vhost_kernel_cleanup, + .vhost_backend_get_vq_index = vhost_kernel_get_vq_index, };int vhost_set_backend_type (struct vhost_dev *dev,VhostBackendType backend_type)hw/virtio/vhost-user.c +10
-2@@ -393,9 +393,17 @@ static int vhost_user_cleanup(struct vhost_dev *dev)return 0 ; } +static int vhost_user_get_vq_index (struct vhost_dev *dev,int idx) +{ +assert (idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); + +return idx; +} +const VhostOps user_ops = { .backend_type =VHOST_BACKEND_TYPE_USER , .vhost_call = vhost_user_call, .vhost_backend_init = vhost_user_init, - .vhost_backend_cleanup = vhost_user_cleanup - }; + .vhost_backend_cleanup = vhost_user_cleanup, + .vhost_backend_get_vq_index = vhost_user_get_vq_index, +};hw/virtio/vhost.c +6
-9@@ -719,7 +719,7 @@ static int vhost_virtqueue_start(struct vhost_dev *dev, {hwaddr s, l, a;int r; -int vhost_vq_index = idx - dev->vq_index; +int vhost_vq_index = dev->vhost_ops->vhost_backend_get_vq_index (dev, idx);struct vhost_vring_file file = { .index = vhost_vq_index };@@ -728,7 +728,6 @@ static int vhost_virtqueue_start(struct vhost_dev *dev, };struct VirtQueue *vvq =virtio_get_queue (vdev, idx); -assert (idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); vq->num = state.num =virtio_queue_get_num (vdev, idx); r = dev->vhost_ops->vhost_call (dev,VHOST_SET_VRING_NUM , &state);@@ -822,12 +821,12 @@ static void vhost_virtqueue_stop(struct vhost_dev *dev,struct vhost_virtqueue *vq, unsigned idx) { - int vhost_vq_index = idx - dev->vq_index; + int vhost_vq_index = dev->vhost_ops->vhost_backend_get_vq_index(dev, idx);struct vhost_vring_state state = { .index = vhost_vq_index, };int r; -assert (idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); + r = dev->vhost_ops->vhost_call (dev,VHOST_GET_VRING_BASE , &state);if (r <0 ) {fprintf (stderr,"vhost VQ %d ring restore failed: %d\n" , idx, r);@@ -1066,17 +1065,15 @@ void vhost_virtqueue_mask(struct vhost_dev *hdev, VirtIODevice *vdev, int n, {struct VirtQueue *vvq =virtio_get_queue (vdev, n);int r, index = n - hdev->vq_index; +struct vhost_vring_file file; -assert (n >= hdev->vq_index && n < hdev->vq_index + hdev->nvqs); - -struct vhost_vring_file file = { - .index = index - };if (mask) { file.fd =event_notifier_get_fd (&hdev->vqs[index].masked_notifier); }else { file.fd =event_notifier_get_fd (virtio_queue_get_guest_notifier (vvq)); } + + file.index = hdev->vhost_ops->vhost_backend_get_vq_index (hdev, n); r = hdev->vhost_ops->vhost_call (hdev,VHOST_SET_VRING_CALL , &file);assert (r >=0 ); }include/hw/virtio/vhost-backend.h +2
-0@@ -24,12 +24,14 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request, void *arg);typedef int (*vhost_backend_init )(struct vhost_dev *dev,void *opaque);typedef int (*vhost_backend_cleanup )(struct vhost_dev *dev); +typedef int (*vhost_backend_get_vq_index )(struct vhost_dev *dev,int idx);typedef struct VhostOps {VhostBackendType backend_type;vhost_call vhost_call;vhost_backend_init vhost_backend_init;vhost_backend_cleanup vhost_backend_cleanup; +vhost_backend_get_vq_index vhost_backend_get_vq_index; }VhostOps ;extern const VhostOps user_ops;
Re: [Qemu-devel] [PATCH v10 5/7] vhost: introduce vhost_backend_get_vq_index method
In reply to Yuanhan LiuQuoted 1 line · Yuanhan Liu
Minusing the idx with the
base(dev->vq_index)for vhost-kernel, and
s/Minusing/Subtracting/
Quoted 7 lines
then adding it back for vhost-user doesn't seem right. Here introduces a new method
vhost_backend_get_vq_index()for getting the right vq index for following vhost messages calls.Suggested-by Jason Wang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>---
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v10 6/7 vhost-user: add multiple queue support
In reply to Yuanhan LiuFrom: Changchun Ouyang <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
This patch adds vhost-user multiple queue support, by creating a nc and vhost_net pair for each queue.
Qemu exits if find that the backend can't support number of requested queues (by providing queues=# option). The max number is queried by a new message, VHOST_USER_GET_QUEUE_NUM, and is sent only when protocol feature VHOST_USER_PROTOCOL_F_MQ is present first.
The max queue check is done at vhost-user initiation stage. We initiate one queue first, which, in the meantime, also gets the max_queues the backend supports.
In older version, it was reported that some messages are sent more times than necessary. Here we came an agreement with Michael that we could categorize vhost user messages to 2 types: non-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function vhost_user_one_time_request(), which lists following messages as non-vring specific messages:
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUM
For above messages, we simply ignore them when they are not sent the first time.
---
v10: don't treat VHOST_USER_SET/GET_[PROTOCOL]_FEATURES as one time
request, as the two feature bits need to be stored at per-device.
v9: per suggested by Jason Wang, we could invoke qemu_chr_add_handlers()
once only, and invoke qemu_find_net_clients_except() at the handler to gather all related ncs, for initiating all queues. Which addresses a hidden bug in older verion in a more QEMU-like way.
v8: set net->dev.vq_index correctly inside vhost_net_init() based on the
value from net->nc->queue_index. --- docs/specs/vhost-user.txt | 15 +++++ hw/net/vhost_net.c | 10 ++-- hw/virtio/vhost-user.c | 22 +++++++ hw/virtio/vhost.c | 5 +- net/vhost-user.c | 146 +++++++++++++++++++++++++++++++++------------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- 7 files changed, 159 insertions(+), 50 deletions(-)
docs/specs/vhost-user.txt +15 -0
@@ -135,6 +135,21 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +---------------------- + +Multiple queue is treated as a protocol extension, hence the slave has to +implement protocol features first. The multiple queues feature is supported +only when the protocol feature VHOST_USER_PROTOCOL_F_MQ (bit 0) is set: +#define VHOST_USER_PROTOCOL_F_MQ 0 + +The max number of queues the slave supports can be queried with message +VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of +requested queues is bigger than that. + +As all queues share one connection, the master uses a unique index for each +queue in the sent message to identify a specified queue. + Message types -------------
hw/net/vhost_net.c +6 -4
@@ -148,8 +148,11 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options)fprintf (stderr,"vhost-net requires net backend to be setup\n" ); goto fail; } + net->nc = options->net_backend; net->dev.max_queues =1 ; + net->dev.nvqs =2 ; + net->dev.vqs = net->vqs;if (backend_kernel) { r =vhost_net_get_fd (options->net_backend);
@@ -164,11 +167,10 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) net->dev.backend_features =0 ; net->dev.protocol_features =0 ; net->backend =-1 ; - } - net->nc = options->net_backend; - net->dev.nvqs =2 ; - net->dev.vqs = net->vqs; +/* vhost-user needs vq_index to initiate a specific queue pair */ + net->dev.vq_index = net->nc->queue_index * net->dev.nvqs; + } r =vhost_dev_init (&net->dev, options->opaque, options->backend_type);
hw/virtio/vhost-user.c +22 -0
@@ -187,6 +187,19 @@ static int vhost_user_write(struct vhost_dev *dev, VhostUserMsg *msg, 0 : -1; } +static bool vhost_user_one_time_request (VhostUserRequest request) +{ +switch (request) { +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_DEVICE : +case VHOST_USER_SET_MEM_TABLE : +case VHOST_USER_GET_QUEUE_NUM : +return true; +default : +return false; + } +} +static int vhost_user_call (struct vhost_dev *dev,unsigned long int request,void *arg) {
@@ -207,6 +220,15 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, msg_request = request; } +/* +* For non-vring specific requests, like VHOST_USER_SET_MEM_TABLE, +* we just need send it once in the first time. For later such +* request, we just ignore it. +*/ +if (vhost_user_one_time_request (msg_request) && dev->vq_index !=0 ) { +return 0 ; + } + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ;
hw/virtio/vhost.c +3 -2
@@ -874,8 +874,9 @@ static void vhost_eventfd_del(MemoryListener *listener,static int vhost_virtqueue_init (struct vhost_dev *dev,struct vhost_virtqueue *vq,int n) { +int vhost_vq_index = dev->vhost_ops->vhost_backend_get_vq_index (dev, n);struct vhost_vring_file file = { - .index = n, + .index = vhost_vq_index, };int r =event_notifier_init (&vq->masked_notifier,0 );if (r <0 ) {
@@ -926,7 +927,7 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque, }for (i =0 ; i < hdev->nvqs; ++i) { - r =vhost_virtqueue_init (hdev, hdev->vqs + i, i); + r =vhost_virtqueue_init (hdev, hdev->vqs + i, hdev->vq_index + i);if (r <0 ) { goto fail_vq; }
net/vhost-user.c +105 -41
213 lines not on this page · raw message
qapi-schema.json +5 -1
@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } } ## # @NetClientOptions
qemu-options.hx +3 -2
@@ -1990,13 +1990,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single netdev. @code{-net} and @code{-device} with parameter @option{vlan} create the required hub automatically. -@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off] +@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n] Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should be a unix domain socket backed one. The vhost-user uses a specifically defined protocol to pass vhost ioctl replacement messages to an application on the other end of the socket. On non-MSIX guests, the feature can be forced with -@var{vhostforce}. +@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to +be created for multiqueue vhost-user. Example: @example
Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH v10 6/7] vhost-user: add multiple queue support
Quoted 305 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
This patch adds vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.Qemu exits if find that the backend can't support number of requested queues (by providing queues=# option). The max number is queried by a new message,
VHOST_USER_GET_QUEUE_NUM, and is sent only when protocol featureVHOST_USER_PROTOCOL_F_MQis present first.The max queue check is done at vhost-user initiation stage. We initiate one queue first, which, in the meantime, also gets the
max_queuesthe backend supports.In older version, it was reported that some messages are sent more times than necessary. Here we came an agreement with Michael that we could categorize vhost user messages to 2 types: non-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function
vhost_user_one_time_request(), which lists following messages as non-vring specific messages:
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUMFor above messages, we simply ignore them when they are not sent the first time.
Signed-off-by Nikolay Nikolaev <[email protected]>Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>---
v10: don't treat VHOST_USER_SET/GET_[PROTOCOL]_FEATURES as one time
request, as the two feature bits need to be stored at per-device.v9: per suggested by Jason Wang, we could invoke
qemu_chr_add_handlers()
once only, and invokeqemu_find_net_clients_except()at the handler to gather all related ncs, for initiating all queues. Which addresses a hidden bug in older verion in a more QEMU-like way.v8: set
net->dev.vq_indexcorrectly insidevhost_net_init()based on thevalue from net->nc->queue_index. --- docs/specs/vhost-user.txt | 15 +++++ hw/net/vhost_net.c | 10 ++-- hw/virtio/vhost-user.c | 22 +++++++ hw/virtio/vhost.c | 5 +- net/vhost-user.c | 146 +++++++++++++++++++++++++++++++++------------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- 7 files changed, 159 insertions(+), 50 deletions(-)docs/specs/vhost-user.txt +15
-0@@ -135,6 +135,21 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +---------------------- + +Multiple queue is treated as a protocol extension, hence the slave has to +implement protocol features first. The multiple queues feature is supported +only when the protocol feature VHOST_USER_PROTOCOL_F_MQ (bit 0) is set: +#define VHOST_USER_PROTOCOL_F_MQ 0 + +The max number of queues the slave supports can be queried with message +VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of +requested queues is bigger than that. + +As all queues share one connection, the master uses a unique index for each +queue in the sent message to identify a specified queue. + Message types -------------hw/net/vhost_net.c +6
-4@@ -148,8 +148,11 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options)fprintf (stderr,"vhost-net requires net backend to be setup\n" ); goto fail; } + net->nc = options->net_backend; net->dev.max_queues =1 ; + net->dev.nvqs =2 ; + net->dev.vqs = net->vqs;if (backend_kernel) { r =vhost_net_get_fd (options->net_backend);@@ -164,11 +167,10 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) net->dev.backend_features =0 ; net->dev.protocol_features =0 ; net->backend =-1 ; - } - net->nc = options->net_backend; - net->dev.nvqs =2 ; - net->dev.vqs = net->vqs; +/* vhost-user needs vq_index to initiate a specific queue pair */ + net->dev.vq_index = net->nc->queue_index * net->dev.nvqs; + } r =vhost_dev_init (&net->dev, options->opaque, options->backend_type);hw/virtio/vhost-user.c +22
-0@@ -187,6 +187,19 @@ static int vhost_user_write(struct vhost_dev *dev, VhostUserMsg *msg, 0 : -1; } +static bool vhost_user_one_time_request (VhostUserRequest request) +{ +switch (request) { +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_DEVICE : +case VHOST_USER_SET_MEM_TABLE : +case VHOST_USER_GET_QUEUE_NUM : +return true; +default : +return false; + } +} +static int vhost_user_call (struct vhost_dev *dev,unsigned long int request,void *arg) {@@ -207,6 +220,15 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, msg_request = request; } +/* +* For non-vring specific requests, like VHOST_USER_SET_MEM_TABLE, +* we just need send it once in the first time. For later such +* request, we just ignore it. +*/ +if (vhost_user_one_time_request (msg_request) && dev->vq_index !=0 ) { +return 0 ; + } + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ;hw/virtio/vhost.c +3
-2@@ -874,8 +874,9 @@ static void vhost_eventfd_del(MemoryListener *listener,static int vhost_virtqueue_init (struct vhost_dev *dev,struct vhost_virtqueue *vq,int n) { +int vhost_vq_index = dev->vhost_ops->vhost_backend_get_vq_index (dev, n);struct vhost_vring_file file = { - .index = n, + .index = vhost_vq_index, };int r =event_notifier_init (&vq->masked_notifier,0 );if (r <0 ) {@@ -926,7 +927,7 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque, }for (i =0 ; i < hdev->nvqs; ++i) { - r =vhost_virtqueue_init (hdev, hdev->vqs + i, i); + r =vhost_virtqueue_init (hdev, hdev->vqs + i, hdev->vq_index + i);if (r <0 ) { goto fail_vq; }net/vhost-user.c +69
-24@@ -39,37 +39,77 @@ static int vhost_user_running(VhostUserState *s)return (s->vhost_net) ?1 :0 ; } -static int vhost_user_start (VhostUserState *s) +static void vhost_user_stop (int queues,NetClientState *ncs[]) { -VhostNetOptions options; - -if (vhost_user_running (s)) { -return 0 ; - } +VhostUserState *s; +int i; - options.backend_type =VHOST_BACKEND_TYPE_USER ; - options.net_backend = &s->nc; - options.opaque = s->chr; +for (i =0 ; i < queues; i++) { +assert (ncs[i]->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); - s->vhost_net =vhost_net_init (&options); + s =DO_UPCAST (VhostUserState, nc, ncs[i]); +if (!vhost_user_running (s)) { +continue ; + } -return vhost_user_running (s) ?0 :-1 ; +if (s->vhost_net) { +vhost_net_cleanup (s->vhost_net); + s->vhost_net =NULL ; + } + } } -static void vhost_user_stop (VhostUserState *s) +static int vhost_user_start (int queues,NetClientState *ncs[]) { -if (vhost_user_running (s)) { -vhost_net_cleanup (s->vhost_net); +VhostNetOptions options; +VhostUserState *s; +int max_queues; +int i; + + options.backend_type =VHOST_BACKEND_TYPE_USER ; + +for (i =0 ; i < queues; i++) { +assert (ncs[i]->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); + + s =DO_UPCAST (VhostUserState, nc, ncs[i]); +if (vhost_user_running (s)) { +continue ; + } + + options.net_backend = ncs[i]; + options.opaque = s->chr; + s->vhost_net =vhost_net_init (&options); +if (!s->vhost_net) { +error_report ("failed to init vhost_net for queue %d\n" , i); + goto err; + } + +if (i ==0 ) { + max_queues =vhost_net_get_max_queues (s->vhost_net); +if (queues > max_queues) { +error_report ("you are asking more queues than " +"supported: %d\n" , max_queues); + goto err; + } + } } - s->vhost_net =0 ; +return 0 ; + +err: +vhost_user_stop (i +1 , ncs); +return -1 ; }static void vhost_user_cleanup (NetClientState *nc) {VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); -vhost_user_stop (s); +if (s->vhost_net) { +vhost_net_cleanup (s->vhost_net); + s->vhost_net =NULL ; + } +qemu_purge_queued_packets (nc); }@@ -95,59 +135,81 @@ static NetClientInfo net_vhost_user_info = { .has_ufo = vhost_user_has_ufo, }; -static void net_vhost_link_down (VhostUserState *s,bool link_down) +static void net_vhost_link_down (int queues,NetClientState *ncs[], +bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < queues; i++) { + nc = ncs[i]; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } + +if (nc->info->link_status_changed) {
net/vhost-user.c static NetClientInfo net_vhost_user_info = { · line 161
Quoted 10 lines
net/vhost-user.c +5
-2+ nc->info->link_status_changed (nc); + } -if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }
The semantics is a little bit difference with qmp_set_link. What's the reason for this? If there's no specific reason, probably, we could just reuse qmp_set_link() (or part of) here?
Other looks good to me.
Quoted 128 lines
net/vhost-user.c +31
-15static void net_vhost_user_event (void *opaque,int event) { -VhostUserState *s = opaque; +const char *name = opaque; +NetClientState *ncs[MAX_QUEUE_NUM ]; +VhostUserState *s; +int queues; + queues =qemu_find_net_clients_except (name, ncs, +NET_CLIENT_OPTIONS_KIND_NIC , +MAX_QUEUE_NUM ); + s =DO_UPCAST (VhostUserState, nc, ncs[0 ]);switch (event) {case CHR_EVENT_OPENED : -vhost_user_start (s); -net_vhost_link_down (s, false); +if (vhost_user_start (queues, ncs) <0 ) { +exit (1 ); + } +net_vhost_link_down (queues, ncs, false);error_report ("chardev \"%s\" went up" , s->chr->label);break ;case CHR_EVENT_CLOSED : -net_vhost_link_down (s, true); -vhost_user_stop (s); +net_vhost_link_down (queues, ncs, true); +vhost_user_stop (queues, ncs);error_report ("chardev \"%s\" went down" , s->chr->label);break ; } }static int net_vhost_user_init (NetClientState *peer,const char *device, -const char *name,CharDriverState *chr) +const char *name,CharDriverState *chr, +int queues) {NetClientState *nc;VhostUserState *s; +int i; - nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); +for (i =0 ; i < queues; i++) { + nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); -snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user to %s" , - chr->label); +snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user%d to %s" , + i, chr->label); - s =DO_UPCAST (VhostUserState, nc, nc); +/* We don't provide a receive callback */ + nc->receive_disabled =1 ; + nc->queue_index = i; -/* We don't provide a receive callback */ - s->nc.receive_disabled =1 ; - s->chr = chr; + s =DO_UPCAST (VhostUserState, nc, nc); + s->chr = chr; + } -qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s); +qemu_chr_add_handlers (chr,NULL ,NULL , net_vhost_user_event, (void *)name);return 0 ; }@@ -226,6 +288,7 @@ static int net_vhost_check_net(void *opaque, QemuOpts *opts, Error **errp)int net_init_vhost_user (const NetClientOptions *opts,const char *name,NetClientState *peer,Error **errp) { +int queues;const NetdevVhostUserOptions *vhost_user_opts;CharDriverState *chr;@@ -243,6 +306,7 @@ int net_init_vhost_user(const NetClientOptions *opts, const char *name,return -1 ; } + queues = vhost_user_opts->has_queues ? vhost_user_opts->queues :1 ; -return net_vhost_user_init (peer,"vhost_user" , name, chr); +return net_vhost_user_init (peer,"vhost_user" , name, chr, queues); }qapi-schema.json +5
-1@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } } ## # @NetClientOptionsqemu-options.hx +3
-2@@ -1990,13 +1990,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single netdev. @code{-net} and @code{-device} with parameter @option{vlan} create the required hub automatically. -@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off] +@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n] Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should be a unix domain socket backed one. The vhost-user uses a specifically defined protocol to pass vhost ioctl replacement messages to an application on the other end of the socket. On non-MSIX guests, the feature can be forced with -@var{vhostforce}. +@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to +be created for multiqueue vhost-user. Example: @example
Re: [Qemu-devel] [PATCH v10 6/7] vhost-user: add multiple queue support
Quoted 2 lines · Jason Wang
[...]
Quoted 27 lines
Quoted 27 lines
net/vhost-user.c +15
-8-static void net_vhost_link_down (VhostUserState *s,bool link_down) +static void net_vhost_link_down (int queues,NetClientState *ncs[], +bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < queues; i++) { + nc = ncs[i]; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } + +if (nc->info->link_status_changed) { + nc->info->link_status_changed (nc); + }
Quoted 11 lines
net/vhost-user.c static NetClientInfo net_vhost_user_info = { · line 161
Quoted 7 lines
net/vhost-user.c +3
-2-if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }The semantics is a little bit difference with
qmp_set_link. What's the reason for this? If there's no specific reason, probably, we could just reuseqmp_set_link()(or part of) here?
I did this just because I refactored the vhost_user init code, making the char dev handler to initiate all ncs. Hence I turned net_vhost_link_down() to handle all ncs, just like what I did for vhost_user_start().
And TBH, I'm not aware of qmp_set_link() before, and after reading the code, I do think we can simply invoke it instead. And thanks for the info.
Quoted 2 lines
Other looks good to me.
Thanks for the review. May I add your reviewed-by if I fix above issue?
--yliu
Re: [Qemu-devel] [PATCH v10 6/7] vhost-user: add multiple queue support
Quoted 44 lines · Yuanhan Liu
Quoted 1 line · Jason Wang
[...]
Quoted 27 lines
Quoted 27 lines
net/vhost-user.c +15
-8-static void net_vhost_link_down (VhostUserState *s,bool link_down) +static void net_vhost_link_down (int queues,NetClientState *ncs[], +bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < queues; i++) { + nc = ncs[i]; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } + +if (nc->info->link_status_changed) { + nc->info->link_status_changed (nc); + }Quoted 10 lines
net/vhost-user.c static NetClientInfo net_vhost_user_info = { · line 161
Quoted 7 lines
net/vhost-user.c +3
-2-if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }The semantics is a little bit difference with
qmp_set_link. What's the reason for this? If there's no specific reason, probably, we could just reuseqmp_set_link()(or part of) here?I did this just because I refactored the
vhost_userinit code, making the char dev handler to initiate all ncs. Hence I turnednet_vhost_link_down()to handle all ncs, just like what I did forvhost_user_start().
Quoted 8 lines
And TBH, I'm not aware of
qmp_set_link()before, and after reading the code, I do think we can simply invoke it instead. And thanks for the info.Quoted 1 line
Other looks good to me.
Thanks for the review. May I add your reviewed-by if I fix above issue?
--yliu
I prefer to add it myself :)
Thanks.
Re: [Qemu-devel] [PATCH v10 6/7] vhost-user: add multiple queue support
Quoted 51 lines · Jason Wang
Quoted 48 lines · Yuanhan Liu
Quoted 1 line · Jason Wang
[...]
Quoted 27 lines
Quoted 27 lines
net/vhost-user.c +15
-8-static void net_vhost_link_down (VhostUserState *s,bool link_down) +static void net_vhost_link_down (int queues,NetClientState *ncs[], +bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < queues; i++) { + nc = ncs[i]; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } + +if (nc->info->link_status_changed) { + nc->info->link_status_changed (nc); + }Quoted 10 lines
net/vhost-user.c static NetClientInfo net_vhost_user_info = { · line 161
Quoted 7 lines
net/vhost-user.c +3
-2-if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }The semantics is a little bit difference with
qmp_set_link. What's the reason for this? If there's no specific reason, probably, we could just reuseqmp_set_link()(or part of) here?I did this just because I refactored the
vhost_userinit code, making the char dev handler to initiate all ncs. Hence I turnednet_vhost_link_down()to handle all ncs, just like what I did forvhost_user_start().And TBH, I'm not aware of
qmp_set_link()before, and after reading the code, I do think we can simply invoke it instead. And thanks for the info.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Other looks good to me.
Thanks for the review. May I add your reviewed-by if I fix above issue?
--yliuI prefer to add it myself :)
Fair enough; I meant to save your effort :-)
--yliu
Re: [Qemu-devel] [PATCH v10 6/7] vhost-user: add multiple queue support
In reply to Yuanhan LiuQuoted 30 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
This patch adds vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.Qemu exits if find that the backend can't support number of requested queues (by providing queues=# option). The max number is queried by a new message,
VHOST_USER_GET_QUEUE_NUM, and is sent only when protocol featureVHOST_USER_PROTOCOL_F_MQis present first.The max queue check is done at vhost-user initiation stage. We initiate one queue first, which, in the meantime, also gets the
max_queuesthe backend supports.In older version, it was reported that some messages are sent more times than necessary. Here we came an agreement with Michael that we could categorize vhost user messages to 2 types: non-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function
vhost_user_one_time_request(), which lists following messages as non-vring specific messages:
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUM
Quoted 6 lines
For above messages, we simply ignore them when they are not sent the first time.
Signed-off-by Nikolay Nikolaev <[email protected]>Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>
Eric, could you review schema changes please?
Quoted 408 lines
---
v10: don't treat VHOST_USER_SET/GET_[PROTOCOL]_FEATURES as one time
request, as the two feature bits need to be stored at per-device.v9: per suggested by Jason Wang, we could invoke
qemu_chr_add_handlers()
once only, and invokeqemu_find_net_clients_except()at the handler to gather all related ncs, for initiating all queues. Which addresses a hidden bug in older verion in a more QEMU-like way.v8: set
net->dev.vq_indexcorrectly insidevhost_net_init()based on thevalue from net->nc->queue_index. --- docs/specs/vhost-user.txt | 15 +++++ hw/net/vhost_net.c | 10 ++-- hw/virtio/vhost-user.c | 22 +++++++ hw/virtio/vhost.c | 5 +- net/vhost-user.c | 146 +++++++++++++++++++++++++++++++++------------- qapi-schema.json | 6 +- qemu-options.hx | 5 +- 7 files changed, 159 insertions(+), 50 deletions(-)docs/specs/vhost-user.txt +15
-0@@ -135,6 +135,21 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Multiple queue support +---------------------- + +Multiple queue is treated as a protocol extension, hence the slave has to +implement protocol features first. The multiple queues feature is supported +only when the protocol feature VHOST_USER_PROTOCOL_F_MQ (bit 0) is set: +#define VHOST_USER_PROTOCOL_F_MQ 0 + +The max number of queues the slave supports can be queried with message +VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of +requested queues is bigger than that. + +As all queues share one connection, the master uses a unique index for each +queue in the sent message to identify a specified queue. + Message types -------------hw/net/vhost_net.c +6
-4@@ -148,8 +148,11 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options)fprintf (stderr,"vhost-net requires net backend to be setup\n" ); goto fail; } + net->nc = options->net_backend; net->dev.max_queues =1 ; + net->dev.nvqs =2 ; + net->dev.vqs = net->vqs;if (backend_kernel) { r =vhost_net_get_fd (options->net_backend);@@ -164,11 +167,10 @@ struct vhost_net *vhost_net_init(VhostNetOptions *options) net->dev.backend_features =0 ; net->dev.protocol_features =0 ; net->backend =-1 ; - } - net->nc = options->net_backend; - net->dev.nvqs =2 ; - net->dev.vqs = net->vqs; +/* vhost-user needs vq_index to initiate a specific queue pair */ + net->dev.vq_index = net->nc->queue_index * net->dev.nvqs; + } r =vhost_dev_init (&net->dev, options->opaque, options->backend_type);hw/virtio/vhost-user.c +22
-0@@ -187,6 +187,19 @@ static int vhost_user_write(struct vhost_dev *dev, VhostUserMsg *msg, 0 : -1; } +static bool vhost_user_one_time_request (VhostUserRequest request) +{ +switch (request) { +case VHOST_USER_SET_OWNER : +case VHOST_USER_RESET_DEVICE : +case VHOST_USER_SET_MEM_TABLE : +case VHOST_USER_GET_QUEUE_NUM : +return true; +default : +return false; + } +} +static int vhost_user_call (struct vhost_dev *dev,unsigned long int request,void *arg) {@@ -207,6 +220,15 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request, msg_request = request; } +/* +* For non-vring specific requests, like VHOST_USER_SET_MEM_TABLE, +* we just need send it once in the first time. For later such +* request, we just ignore it. +*/ +if (vhost_user_one_time_request (msg_request) && dev->vq_index !=0 ) { +return 0 ; + } + msg.request = msg_request; msg.flags =VHOST_USER_VERSION ; msg.size =0 ;hw/virtio/vhost.c +3
-2@@ -874,8 +874,9 @@ static void vhost_eventfd_del(MemoryListener *listener,static int vhost_virtqueue_init (struct vhost_dev *dev,struct vhost_virtqueue *vq,int n) { +int vhost_vq_index = dev->vhost_ops->vhost_backend_get_vq_index (dev, n);struct vhost_vring_file file = { - .index = n, + .index = vhost_vq_index, };int r =event_notifier_init (&vq->masked_notifier,0 );if (r <0 ) {@@ -926,7 +927,7 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque, }for (i =0 ; i < hdev->nvqs; ++i) { - r =vhost_virtqueue_init (hdev, hdev->vqs + i, i); + r =vhost_virtqueue_init (hdev, hdev->vqs + i, hdev->vq_index + i);if (r <0 ) { goto fail_vq; }net/vhost-user.c +105
-41@@ -39,37 +39,77 @@ static int vhost_user_running(VhostUserState *s)return (s->vhost_net) ?1 :0 ; } -static int vhost_user_start (VhostUserState *s) +static void vhost_user_stop (int queues,NetClientState *ncs[]) { -VhostNetOptions options; - -if (vhost_user_running (s)) { -return 0 ; - } +VhostUserState *s; +int i; - options.backend_type =VHOST_BACKEND_TYPE_USER ; - options.net_backend = &s->nc; - options.opaque = s->chr; +for (i =0 ; i < queues; i++) { +assert (ncs[i]->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); - s->vhost_net =vhost_net_init (&options); + s =DO_UPCAST (VhostUserState, nc, ncs[i]); +if (!vhost_user_running (s)) { +continue ; + } -return vhost_user_running (s) ?0 :-1 ; +if (s->vhost_net) { +vhost_net_cleanup (s->vhost_net); + s->vhost_net =NULL ; + } + } } -static void vhost_user_stop (VhostUserState *s) +static int vhost_user_start (int queues,NetClientState *ncs[]) { -if (vhost_user_running (s)) { -vhost_net_cleanup (s->vhost_net); +VhostNetOptions options; +VhostUserState *s; +int max_queues; +int i; + + options.backend_type =VHOST_BACKEND_TYPE_USER ; + +for (i =0 ; i < queues; i++) { +assert (ncs[i]->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ); + + s =DO_UPCAST (VhostUserState, nc, ncs[i]); +if (vhost_user_running (s)) { +continue ; + } + + options.net_backend = ncs[i]; + options.opaque = s->chr; + s->vhost_net =vhost_net_init (&options); +if (!s->vhost_net) { +error_report ("failed to init vhost_net for queue %d\n" , i); + goto err; + } + +if (i ==0 ) { + max_queues =vhost_net_get_max_queues (s->vhost_net); +if (queues > max_queues) { +error_report ("you are asking more queues than " +"supported: %d\n" , max_queues); + goto err; + } + } } - s->vhost_net =0 ; +return 0 ; + +err: +vhost_user_stop (i +1 , ncs); +return -1 ; }static void vhost_user_cleanup (NetClientState *nc) {VhostUserState *s =DO_UPCAST (VhostUserState, nc, nc); -vhost_user_stop (s); +if (s->vhost_net) { +vhost_net_cleanup (s->vhost_net); + s->vhost_net =NULL ; + } +qemu_purge_queued_packets (nc); }@@ -95,59 +135,81 @@ static NetClientInfo net_vhost_user_info = { .has_ufo = vhost_user_has_ufo, }; -static void net_vhost_link_down (VhostUserState *s,bool link_down) +static void net_vhost_link_down (int queues,NetClientState *ncs[], +bool link_down) { - s->nc.link_down = link_down; +NetClientState *nc; +int i; -if (s->nc.peer) { - s->nc.peer->link_down = link_down; - } +for (i =0 ; i < queues; i++) { + nc = ncs[i]; -if (s->nc.info->link_status_changed) { - s->nc.info->link_status_changed (&s->nc); - } + nc->link_down = link_down; + +if (nc->peer) { + nc->peer->link_down = link_down; + } + +if (nc->info->link_status_changed) { + nc->info->link_status_changed (nc); + } -if (s->nc.peer && s->nc.peer->info->link_status_changed) { - s->nc.peer->info->link_status_changed (s->nc.peer); +if (nc->peer && nc->peer->info->link_status_changed) { + nc->peer->info->link_status_changed (nc->peer); + } } }static void net_vhost_user_event (void *opaque,int event) { -VhostUserState *s = opaque; +const char *name = opaque; +NetClientState *ncs[MAX_QUEUE_NUM ]; +VhostUserState *s; +int queues; + queues =qemu_find_net_clients_except (name, ncs, +NET_CLIENT_OPTIONS_KIND_NIC , +MAX_QUEUE_NUM ); + s =DO_UPCAST (VhostUserState, nc, ncs[0 ]);switch (event) {case CHR_EVENT_OPENED : -vhost_user_start (s); -net_vhost_link_down (s, false); +if (vhost_user_start (queues, ncs) <0 ) { +exit (1 ); + } +net_vhost_link_down (queues, ncs, false);error_report ("chardev \"%s\" went up" , s->chr->label);break ;case CHR_EVENT_CLOSED : -net_vhost_link_down (s, true); -vhost_user_stop (s); +net_vhost_link_down (queues, ncs, true); +vhost_user_stop (queues, ncs);error_report ("chardev \"%s\" went down" , s->chr->label);break ; } }static int net_vhost_user_init (NetClientState *peer,const char *device, -const char *name,CharDriverState *chr) +const char *name,CharDriverState *chr, +int queues) {NetClientState *nc;VhostUserState *s; +int i; - nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); +for (i =0 ; i < queues; i++) { + nc =qemu_new_net_client (&net_vhost_user_info, peer, device, name); -snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user to %s" , - chr->label); +snprintf (nc->info_str,sizeof (nc->info_str),"vhost-user%d to %s" , + i, chr->label); - s =DO_UPCAST (VhostUserState, nc, nc); +/* We don't provide a receive callback */ + nc->receive_disabled =1 ; + nc->queue_index = i; -/* We don't provide a receive callback */ - s->nc.receive_disabled =1 ; - s->chr = chr; + s =DO_UPCAST (VhostUserState, nc, nc); + s->chr = chr; + } -qemu_chr_add_handlers (s->chr,NULL ,NULL , net_vhost_user_event, s); +qemu_chr_add_handlers (chr,NULL ,NULL , net_vhost_user_event, (void *)name);return 0 ; }@@ -226,6 +288,7 @@ static int net_vhost_check_net(void *opaque, QemuOpts *opts, Error **errp)int net_init_vhost_user (const NetClientOptions *opts,const char *name,NetClientState *peer,Error **errp) { +int queues;const NetdevVhostUserOptions *vhost_user_opts;CharDriverState *chr;@@ -243,6 +306,7 @@ int net_init_vhost_user(const NetClientOptions *opts, const char *name,return -1 ; } + queues = vhost_user_opts->has_queues ? vhost_user_opts->queues :1 ; -return net_vhost_user_init (peer,"vhost_user" , name, chr); +return net_vhost_user_init (peer,"vhost_user" , name, chr, queues); }qapi-schema.json +5
-1@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5) +# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } } ## # @NetClientOptionsqemu-options.hx +3
-2@@ -1990,13 +1990,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single netdev. @code{-net} and @code{-device} with parameter @option{vlan} create the required hub automatically. -@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off] +@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n] Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should be a unix domain socket backed one. The vhost-user uses a specifically defined protocol to pass vhost ioctl replacement messages to an application on the other end of the socket. On non-MSIX guests, the feature can be forced with -@var{vhostforce}. +@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to +be created for multiqueue vhost-user. Example: @exampleSignature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH v10 6/7] vhost-user: add multiple queue support
In reply to Yuanhan LiuQuoted 8 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
This patch is initially based a patch from Nikolay Nikolaev.
This patch adds vhost-user multiple queue support, by creating a nc and
vhost_netpair for each queue.Qemu exits if find that the backend can't support number of requested
s/support/support the/
Quoted 22 lines
queues (by providing queues=# option). The max number is queried by a new message,
VHOST_USER_GET_QUEUE_NUM, and is sent only when protocol featureVHOST_USER_PROTOCOL_F_MQis present first.The max queue check is done at vhost-user initiation stage. We initiate one queue first, which, in the meantime, also gets the
max_queuesthe backend supports.In older version, it was reported that some messages are sent more times than necessary. Here we came an agreement with Michael that we could categorize vhost user messages to 2 types: non-vring specific messages, which should be sent only once, and vring specific messages, which should be sent per queue.
Here I introduced a helper function
vhost_user_one_time_request(), which lists following messages as non-vring specific messages:
VHOST_USER_SET_OWNER
VHOST_USER_RESET_DEVICE
VHOST_USER_SET_MEM_TABLE
VHOST_USER_GET_QUEUE_NUM
Quoted 8 lines
For above messages, we simply ignore them when they are not sent the first time.
Signed-off-by Nikolay Nikolaev <[email protected]>Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>---
Just focusing on the interface portions:
Quoted 7 lines
+++
b/qapi-schema.jsonqapi-schema.json +2
-0@@ -2480,12 +2480,16 @@ # # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false). # +# @queues: #optional number of queues to be created for multiqueue vhost-user +# (default: 1) (Since 2.5)
qapi-schema.json line 2493
Quoted 10 lines
qapi-schema.json +3
-1+# # Since 2.1 ## { 'struct': 'NetdevVhostUserOptions', 'data': { 'chardev': 'str', - '*vhostforce': 'bool' } } + '*vhostforce': 'bool', + '*queues': 'int' } }
Discoverable via introspection, and should work with the patches from Zoltan and me that get QMP netdev_add converted over to qapi.
Interface is good to go.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 6/7] vhost-user: add multiple queue support
In reply to Yuanhan LiuOn Fri, Sep 18, 2015 at 10:58:43PM +0800, Yuanhan Liu wrote:
[...]
Quoted 4 lines
net/vhost-user.c +0
-0
[...]
net/vhost-user.c static int vhost_user_running(VhostUserState *s) · line 88
Quoted 2 lines
net/vhost-user.c +2
-0+if (i ==0 ) { + max_queues =vhost_net_get_max_queues (s->vhost_net);
This breaks arm-softmmu (and any target that doesn't have CONFIG_VHOST_NET):
$ make subdir-arm-softmmu
CC arm-softmmu/hw/net/vhost_net.o
LINK arm-softmmu/qemu-system-arm
../net/vhost-user.o: In function `vhost_user_start':
/home/ehabkost/rh/proj/virt/qemu/net/vhost-user.c:88: undefined reference to `vhost_net_get_max_queues'
collect2: error: ld returned 1 exit status
Makefile:193: recipe for target 'qemu-system-arm' failed
make[1]: *** [qemu-system-arm] Error 1
Makefile:184: recipe for target 'subdir-arm-softmmu' failed
make: *** [subdir-arm-softmmu] Error 2Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH v10 6/7] vhost-user: add multiple queue support
Quoted 11 lines · Eduardo Habkost
On Fri, Sep 18, 2015 at 10:58:43PM +0800, Yuanhan Liu wrote:
[...]Quoted 4 lines
net/vhost-user.c +0
-0[...]
net/vhost-user.c static int vhost_user_running(VhostUserState *s) · line 88
Quoted 2 lines
net/vhost-user.c +2
-0+if (i ==0 ) { + max_queues =vhost_net_get_max_queues (s->vhost_net);This breaks arm-softmmu (and any target that doesn't have
CONFIG_VHOST_NET):
Thanks for testing. Will fix it next version.
--yliu
Quoted 14 lines
$ make subdir-arm-softmmu CC arm-softmmu/hw/net/vhost_net.o LINK arm-softmmu/qemu-system-arm ../net/vhost-user.o: In function `vhost_user_start': /home/ehabkost/rh/proj/virt/qemu/net/vhost-user.c:88: undefined reference to `vhost_net_get_max_queues' collect2: error: ld returned 1 exit status Makefile:193: recipe for target 'qemu-system-arm' failed make[1]: *** [qemu-system-arm] Error 1 Makefile:184: recipe for target 'subdir-arm-softmmu' failed make: *** [subdir-arm-softmmu] Error 2Signature · 1 line
Eduardo
PATCH v10 7/7 vhost-user: add a new message to disable/enable a specific virt queue.
In reply to Yuanhan LiuFrom: Changchun Ouyang <[email protected]>
Add a new message, VHOST_USER_SET_VRING_ENABLE, to enable or disable a specific virt queue, which is similar to attach/detach queue for tap device.
virtio driver on guest doesn't have to use max virt queue pair, it could enable any number of virt queue ranging from 1 to max virt queue pair.
--- docs/specs/vhost-user.txt | 12 +++++++++++- hw/net/vhost_net.c | 18 ++++++++++++++++++ hw/net/virtio-net.c | 8 ++++++++ hw/virtio/vhost-user.c | 19 +++++++++++++++++++ include/hw/virtio/vhost-backend.h | 2 ++ include/net/vhost_net.h | 2 ++ 6 files changed, 60 insertions(+), 1 deletion(-)
docs/specs/vhost-user.txt +11 -1
@@ -148,7 +148,9 @@ VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of requested queues is bigger than that. As all queues share one connection, the master uses a unique index for each -queue in the sent message to identify a specified queue. +queue in the sent message to identify a specified queue. One queue pairs +is enabled initially. More queues are enabled dynamically, by sending +message VHOST_USER_SET_VRING_ENABLE. Message types -------------
@@ -327,3 +329,11 @@ Message types Query how many queues the backend supports. This request should be sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol features by VHOST_USER_GET_PROTOCOL_FEATURES. + + * VHOST_USER_SET_VRING_ENABLE + + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Signal slave to enable or disable corresponding vring.
hw/net/vhost_net.c +18 -0
@@ -422,6 +422,19 @@ VHostNetState *get_vhost_net(NetClientState *nc) return vhost_net; } + +int vhost_set_vring_enable (NetClientState *nc,int enable) +{ +VHostNetState *net =get_vhost_net (nc); +const VhostOps *vhost_ops = net->dev.vhost_ops; + +if (vhost_ops->vhost_backend_set_vring_enable) { +return vhost_ops->vhost_backend_set_vring_enable (&net->dev, enable); + } + +return 0 ; +} +#else struct vhost_net *vhost_net_init (VhostNetOptions *options) {
@@ -467,4 +480,9 @@ VHostNetState *get_vhost_net(NetClientState *nc) {return 0 ; } + +int vhost_set_vring_enable (NetClientState *nc,int enable) +{ +return 0 ; +}#endif
hw/net/virtio-net.c +8 -0
@@ -406,6 +406,10 @@ static int peer_attach(VirtIONet *n, int index)return 0 ; } +if (nc->peer->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) { +vhost_set_vring_enable (nc->peer,1 ); + } +if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) {return 0 ; }
@@ -421,6 +425,10 @@ static int peer_detach(VirtIONet *n, int index)return 0 ; } +if (nc->peer->info->type ==NET_CLIENT_OPTIONS_KIND_VHOST_USER ) { +vhost_set_vring_enable (nc->peer,0 ); + } +if (nc->peer->info->type !=NET_CLIENT_OPTIONS_KIND_TAP ) {return 0 ; }
hw/virtio/vhost-user.c +19 -0
@@ -48,6 +48,7 @@ typedef enum VhostUserRequest {VHOST_USER_GET_PROTOCOL_FEATURES =15 ,VHOST_USER_SET_PROTOCOL_FEATURES =16 ,VHOST_USER_GET_QUEUE_NUM =17 , +VHOST_USER_SET_VRING_ENABLE =18 , VHOST_USER_MAX } VhostUserRequest;
@@ -290,6 +291,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,case VHOST_USER_SET_VRING_NUM :case VHOST_USER_SET_VRING_BASE : +case VHOST_USER_SET_VRING_ENABLE :memcpy (&msg.state, arg,sizeof (struct vhost_vring_state )); msg.size =sizeof (m.state);break ;
@@ -406,6 +408,22 @@ static int vhost_user_init(struct vhost_dev *dev, void *opaque)return 0 ; } +static int vhost_user_set_vring_enable (struct vhost_dev *dev,int enable) +{ +struct vhost_vring_state state = { + .index = dev->vq_index, + .num = enable, + }; + +assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER ); + +if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_MQ ))) { +return -1 ; + } + +return vhost_user_call (dev,VHOST_USER_SET_VRING_ENABLE , &state); +} +static int vhost_user_cleanup (struct vhost_dev *dev) {assert (dev->vhost_ops->backend_type ==VHOST_BACKEND_TYPE_USER );
@@ -428,4 +446,5 @@ const VhostOps user_ops = { .vhost_backend_init = vhost_user_init, .vhost_backend_cleanup = vhost_user_cleanup, .vhost_backend_get_vq_index = vhost_user_get_vq_index, + .vhost_backend_set_vring_enable = vhost_user_set_vring_enable, };
include/hw/virtio/vhost-backend.h +2 -0
@@ -25,6 +25,7 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request,typedef int (*vhost_backend_init )(struct vhost_dev *dev,void *opaque);typedef int (*vhost_backend_cleanup )(struct vhost_dev *dev);typedef int (*vhost_backend_get_vq_index )(struct vhost_dev *dev,int idx); +typedef int (*vhost_backend_set_vring_enable )(struct vhost_dev *dev,int enable); typedef struct VhostOps { VhostBackendType backend_type;
@@ -32,6 +33,7 @@ typedef struct VhostOps {vhost_backend_init vhost_backend_init;vhost_backend_cleanup vhost_backend_cleanup;vhost_backend_get_vq_index vhost_backend_get_vq_index; +vhost_backend_set_vring_enable vhost_backend_set_vring_enable; } VhostOps;extern const VhostOps user_ops;
include/net/vhost_net.h +2 -0
@@ -28,4 +28,6 @@ bool vhost_net_virtqueue_pending(VHostNetState *net, int n);void vhost_net_virtqueue_mask (VHostNetState *net,VirtIODevice *dev,int idx,bool mask);VHostNetState *get_vhost_net (NetClientState *nc); + +int vhost_set_vring_enable (NetClientState * nc,int enable);#endif
Signature · 1 line
1.9.0
Re: [Qemu-devel] [PATCH v10 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
Quoted 26 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
Add a new message,
VHOST_USER_SET_VRING_ENABLE, to enable or disable a specific virt queue, which is similar to attach/detach queue for tap device.virtio driver on guest doesn't have to use max virt queue pair, it could enable any number of virt queue ranging from 1 to max virt queue pair.
Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>--- docs/specs/vhost-user.txt | 12 +++++++++++- hw/net/vhost_net.c | 18 ++++++++++++++++++ hw/net/virtio-net.c | 8 ++++++++ hw/virtio/vhost-user.c | 19 +++++++++++++++++++ include/hw/virtio/vhost-backend.h | 2 ++ include/net/vhost_net.h | 2 ++ 6 files changed, 60 insertions(+), 1 deletion(-)docs/specs/vhost-user.txt +0
-0
docs/specs/vhost-user.txt VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of · line 151
Quoted 5 lines
docs/specs/vhost-user.txt +1
-1@@ -148,7 +148,9 @@ VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of requested queues is bigger than that. As all queues share one connection, the master uses a unique index for each -queue in the sent message to identify a specified queue. +queue in the sent message to identify a specified queue. One queue pairs
s/pairs/pair/
Quoted 7 lines
docs/specs/vhost-user.txt +2
-0+is enabled initially. More queues are enabled dynamically, by sending +message VHOST_USER_SET_VRING_ENABLE. Message types -------------@@ -327,3 +329,11 @@ Message types Query how many queues the backend supports. This request should be
docs/specs/vhost-user.txt Message types · line 339
Quoted 10 lines
docs/specs/vhost-user.txt +8
-0sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol features by VHOST_USER_GET_PROTOCOL_FEATURES. + + * VHOST_USER_SET_VRING_ENABLE + + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Signal slave to enable or disable corresponding vring.
Does there need to be any QMP control to manually change a given queue, or is it all used under the hood with no need for management apps to care other than their initial request of max queues?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
Quoted 29 lines · Eric Blake
Quoted 28 lines · Yuanhan Liu
From: Changchun Ouyang <[email protected]>
Add a new message,
VHOST_USER_SET_VRING_ENABLE, to enable or disable a specific virt queue, which is similar to attach/detach queue for tap device.virtio driver on guest doesn't have to use max virt queue pair, it could enable any number of virt queue ranging from 1 to max virt queue pair.
Signed-off-by Changchun Ouyang <[email protected]>Signed-off-by Yuanhan Liu <[email protected]>--- docs/specs/vhost-user.txt | 12 +++++++++++- hw/net/vhost_net.c | 18 ++++++++++++++++++ hw/net/virtio-net.c | 8 ++++++++ hw/virtio/vhost-user.c | 19 +++++++++++++++++++ include/hw/virtio/vhost-backend.h | 2 ++ include/net/vhost_net.h | 2 ++ 6 files changed, 60 insertions(+), 1 deletion(-)docs/specs/vhost-user.txt +0
-0@@ -148,7 +148,9 @@ VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of requested queues is bigger than that.
Quoted 5 lines
docs/specs/vhost-user.txt VHOST_USER_GET_PROTOCOL_FEATURES. Master should stop when the number of · line 151
Quoted 3 lines
docs/specs/vhost-user.txt +1
-1As all queues share one connection, the master uses a unique index for each -queue in the sent message to identify a specified queue. +queue in the sent message to identify a specified queue. One queue pairss/pairs/pair/
Thanks.
Quoted 14 lines
Quoted 14 lines
docs/specs/vhost-user.txt +6
-0+is enabled initially. More queues are enabled dynamically, by sending +message VHOST_USER_SET_VRING_ENABLE. Message types -------------@@ -327,3 +329,11 @@ Message types Query how many queues the backend supports. This request should be sent only when VHOST_USER_PROTOCOL_F_MQ is set in quried protocol features by VHOST_USER_GET_PROTOCOL_FEATURES. + + * VHOST_USER_SET_VRING_ENABLE + + Id: 18
Quoted 8 lines
docs/specs/vhost-user.txt Message types · line 339
Quoted 4 lines
docs/specs/vhost-user.txt +4
-0+ Equivalent ioctl: N/A + Master payload: vring state description + + Signal slave to enable or disable corresponding vring.Does there need to be any QMP control to manually change a given queue, or is it all used under the hood with no need for management apps to care other than their initial request of max queues?
TBH, I don't know. As far as I know, there is only one queue pair will be enabled by default, and it's user's job to enable (or disable) more queue pairs, say, by ethtool:
# ethtool -L eth0 combined <queue pair number>
Which ends up sending the VHOST_USER_SET_VRING_ENABLE to actually enable (or disable) a specific queue pairs.
Does that answer your question?
--yliu
Re: [Qemu-devel] [PATCH v10 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
Quoted 12 lines · Yuanhan Liu
Quoted 11 lines
docs/specs/vhost-user.txt Message types · line 339
Quoted 7 lines
+ *
VHOST_USER_SET_VRING_ENABLEdocs/specs/vhost-user.txt +6
-0+ + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Signal slave to enable or disable corresponding vring.Does there need to be any QMP control to manually change a given queue, or is it all used under the hood with no need for management apps to care other than their initial request of max queues?
Quoted 8 lines
TBH, I don't know. As far as I know, there is only one queue pair will be enabled by default, and it's user's job to enable (or disable) more queue pairs, say, by ethtool:
# ethtool -L eth0 combined <queue pair number>Which ends up sending the
VHOST_USER_SET_VRING_ENABLEto actually enable (or disable) a specific queue pairs.
So if I'm understanding, it is ethtool, not qemu, that is used to turn on additional queues, and therefore we don't need a QMP command in qemu to control things.
Quoted 2 lines
Does that answer your question?
I think so, but I'll let other reviewers more familiar with this area of code give a final say.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 7/7] vhost-user: add a new message to disable/enable a specific virt queue.
Quoted 18 lines · Eric Blake
Quoted 16 lines · Yuanhan Liu
Quoted 11 lines
docs/specs/vhost-user.txt Message types · line 339
Quoted 7 lines
+ *
VHOST_USER_SET_VRING_ENABLEdocs/specs/vhost-user.txt +6
-0+ + Id: 18 + Equivalent ioctl: N/A + Master payload: vring state description + + Signal slave to enable or disable corresponding vring.Does there need to be any QMP control to manually change a given queue, or is it all used under the hood with no need for management apps to care other than their initial request of max queues?
TBH, I don't know. As far as I know, there is only one queue pair will be enabled by default, and it's user's job to enable (or disable) more queue pairs, say, by ethtool:
Quoted 8 lines
Quoted 4 lines
# ethtool -L eth0 combined <queue pair number>Which ends up sending the
VHOST_USER_SET_VRING_ENABLEto actually enable (or disable) a specific queue pairs.So if I'm understanding, it is ethtool, not qemu, that is used to turn on additional queues, and therefore we don't need a QMP command in qemu to control things.
I guess so, and that's what Michael told me before.
Quoted 4 lines
Quoted 1 line
Does that answer your question?
I think so, but I'll let other reviewers more familiar with this area of code give a final say.
Michael?
--yliu