vhost-user: multiqueue support

Vhost-user will implement the multiqueueu support in a similar way to what vhost already has - a separate thread for each queue.

To enable multiquue funcionality - a new command line parameter "queues" is introduced for the vhost-user netdev.

Signed-off-by Nikolay Nikolaev <[email protected]>

---
 docs/specs/vhost-user.txt |    7 +++++++
 hw/virtio/vhost-user.c    |    6 +++++-
 net/vhost-user.c          |   35 +++++++++++++++++++++++------------
 qapi-schema.json          |    5 ++++-
 qemu-options.hx           |    5 +++--
 5 files changed, 42 insertions(+), 16 deletions(-)

[...]

net/vhost-user.c +2 -1

@@ -134,25 +134,27 @@ static void net_vhost_user_event(void *opaque, int event)
 
 static int net_vhost_user_init(NetClientState *peer, const char *device,
                                const char *name, CharDriverState *chr,
-                               bool vhostforce)
+                               bool vhostforce, uint32_t queues)
 {
     NetClientState *nc;
     VhostUserState *s;
+    int i;

Hi Nikolay,

Quoted 27 lines · Nikolay Nikolaev

Vhost-user will implement the multiqueueu support in a similar way to what vhost already has - a separate thread for each queue.

To enable multiquue funcionality - a new command line parameter "queues" is introduced for the vhost-user netdev.

Signed-off-by Nikolay Nikolaev <[email protected]>

---
 docs/specs/vhost-user.txt |    7 +++++++
 hw/virtio/vhost-user.c    |    6 +++++-
 net/vhost-user.c          |   35 +++++++++++++++++++++++------------
 qapi-schema.json          |    5 ++++-
 qemu-options.hx           |    5 +++--
 5 files changed, 42 insertions(+), 16 deletions(-)

[...]

net/vhost-user.c +2 -1

@@ -134,25 +134,27 @@ static void net_vhost_user_event(void *opaque, int event)
 
 static int net_vhost_user_init(NetClientState *peer, const char *device,
                                const char *name, CharDriverState *chr,
-                               bool vhostforce)
+                               bool vhostforce, uint32_t queues)
 {
     NetClientState *nc;
     VhostUserState *s;
+    int i;

net/vhost-user.c static void net_vhost_user_event(void *opaque, int event) · line 148

Quoted 10 lines

net/vhost-user.c +4 -3

 
-    nc = qemu_new_net_client(&net_vhost_user_info, peer, device, name);
+    for (i = 0; i < queues; i++) {
+        nc = qemu_new_net_client(&net_vhost_user_info, peer, device, name);
 
-    snprintf(nc->info_str, sizeof(nc->info_str), "vhost-user to %s",
-             chr->label);
+        snprintf(nc->info_str, sizeof(nc->info_str), "vhost-user%d to %s",
+                 i, chr->label);
 

Now that there several vhost-user are pointing to the same unix socket, it could make sense to display "nc->info_str" instead of "s->chr->label" in net_vhost_user_event(). Something like that:

net/vhost-user.c +2 -2

@@ -122,36 +122,39 @@ static void net_vhost_user_event(void *opaque, int
 event)
      case CHR_EVENT_OPENED:
          vhost_user_start(s);
          net_vhost_link_down(s, false);
-        error_report("chardev \"%s\" went up\n", s->chr->label);
+        error_report("chardev \"%s\" went up\n", s->nc.info_str);
          break;
      case CHR_EVENT_CLOSED:
          net_vhost_link_down(s, true);
          vhost_user_stop(s);
-        error_report("chardev \"%s\" went down\n", s->chr->label);
+        error_report("chardev \"%s\" went down\n", s->nc.info_str);
          break;
      }
  }
 
 

Also, another comment: if I understand well, the messages like VHOST_USER_SET_OWNER, VHOST_USER_SET_FEATURES, VHOST_SET_MEM_TABLE, (...) will be send once per queue pair and not once per device.

I don't think it's a problem, but maybe it deserves a small comment in the protocol documentation.

Apart from these 2 small comments, the approach looks correct, so

Acked-by Olivier Matz <[email protected]>

Regards,
Olivier

Vhost-user will implement the multiqueueu support in a similar way to what vhost already has - a separate thread for each queue.

To enable multiquue funcionality - a new command line parameter "queues" is introduced for the vhost-user netdev.

Signed-off-by Nikolay Nikolaev <[email protected]>

---

Vhost-user will implement the multiqueueu support in a similar way to what vhost already has - a separate thread for each queue.

To enable multiquue funcionality - a new command line parameter "queues" is introduced for the vhost-user netdev.

Signed-off-by Nikolay Nikolaev <[email protected]>

---
 docs/specs/vhost-user.txt |    7 +++++++
 hw/virtio/vhost-user.c    |    6 +++++-
 net/vhost-user.c          |   35 +++++++++++++++++++++++------------
 qapi-schema.json          |    5 ++++-
 qemu-options.hx           |    5 +++--
 5 files changed, 42 insertions(+), 16 deletions(-)

docs/specs/vhost-user.txt +0 -0

index 650bb18..d3857f5 100644
@@ -127,6 +127,13 @@ in the ancillary data:
 If Master is unable to send the full message or receives a wrong reply it will
 close the connection. An optional reconnection mechanism can be implemented.
 

hw/virtio/vhost-user.c +5 -1

index aefe0bb..83ebcaa 100644
@@ -253,17 +253,20 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,
     case VHOST_SET_VRING_NUM:
     case VHOST_SET_VRING_BASE:
         memcpy(&msg.state, arg, sizeof(struct vhost_vring_state));
+        msg.state.index += dev->vq_index;
         msg.size = sizeof(m.state);
         break;
 
     case VHOST_GET_VRING_BASE:
         memcpy(&msg.state, arg, sizeof(struct vhost_vring_state));
+        msg.state.index += dev->vq_index;
         msg.size = sizeof(m.state);
         need_reply = 1;
         break;
 
     case VHOST_SET_VRING_ADDR:
         memcpy(&msg.addr, arg, sizeof(struct vhost_vring_addr));
+        msg.addr.index += dev->vq_index;
         msg.size = sizeof(m.addr);
         break;
 

@@ -271,7 +274,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,
     case VHOST_SET_VRING_CALL:
     case VHOST_SET_VRING_ERR:
         file = arg;
-        msg.u64 = file->index & VHOST_USER_VRING_IDX_MASK;
+        msg.u64 = (file->index + dev->vq_index) & VHOST_USER_VRING_IDX_MASK;
         msg.size = sizeof(m.u64);
         if (ioeventfd_enabled() && file->fd > 0) {
             fds[fd_num++] = file->fd;

@@ -313,6 +316,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,
                 error_report("Received bad msg size.\n");
                 return -1;
             }
+            msg.state.index -= dev->vq_index;
             memcpy(arg, &msg.state, sizeof(struct vhost_vring_state));
             break;
         default:

net/vhost-user.c +23 -12

index 24e050c..1ea2f98 100644
@@ -134,25 +134,27 @@ static void net_vhost_user_event(void *opaque, int event)
 
 static int net_vhost_user_init(NetClientState *peer, const char *device,
                                const char *name, CharDriverState *chr,
-                               bool vhostforce)
+                               bool vhostforce, uint32_t queues)
 {
     NetClientState *nc;
     VhostUserState *s;
+    int i;
 
-    nc = qemu_new_net_client(&net_vhost_user_info, peer, device, name);
+    for (i = 0; i < queues; i++) {
+        nc = qemu_new_net_client(&net_vhost_user_info, peer, device, name);
 
-    snprintf(nc->info_str, sizeof(nc->info_str), "vhost-user to %s",
-             chr->label);
+        snprintf(nc->info_str, sizeof(nc->info_str), "vhost-user%d to %s",
+                 i, chr->label);
 
-    s = DO_UPCAST(VhostUserState, nc, nc);
+        s = DO_UPCAST(VhostUserState, nc, nc);
 
-    /* We don't provide a receive callback */
-    s->nc.receive_disabled = 1;
-    s->chr = chr;
-    s->vhostforce = vhostforce;
-
-    qemu_chr_add_handlers(s->chr, NULL, NULL, net_vhost_user_event, s);
+        /* We don't provide a receive callback */
+        s->nc.receive_disabled = 1;
+        s->chr = chr;
+        s->vhostforce = vhostforce;
 
+        qemu_chr_add_handlers(s->chr, NULL, NULL, net_vhost_user_event, s);
+    }
     return 0;
 }
 

@@ -228,6 +230,7 @@ static int net_vhost_check_net(QemuOpts *opts, void *opaque)
 int net_init_vhost_user(const NetClientOptions *opts, const char *name,
                         NetClientState *peer)
 {
+    uint32_t queues;
     const NetdevVhostUserOptions *vhost_user_opts;
     CharDriverState *chr;
     bool vhostforce;

@@ -254,5 +257,13 @@ int net_init_vhost_user(const NetClientOptions *opts, const char *name,
         vhostforce = false;
     }
 
-    return net_vhost_user_init(peer, "vhost_user", name, chr, vhostforce);
+    /* number of queues for multiqueue */
+    if (vhost_user_opts->has_queues) {
+        queues = vhost_user_opts->queues;
+    } else {
+        queues = 1;
+    }
+
+    return net_vhost_user_init(peer, "vhost_user", name, chr, vhostforce,
+                               queues);
 }

qapi-schema.json +4 -1

index 9ffdcf8..aa3bb6f 100644
@@ -2208,12 +2208,15 @@
 #
 # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false).
 #
+# @queues: #optional number of queues to be created for multiqueue vhost-user
+#
 # Since 2.1
 ##
 { 'type': 'NetdevVhostUserOptions',
   'data': {
     'chardev':        'str',
-    '*vhostforce':    'bool' } }
+    '*vhostforce':    'bool',
+    '*queues':        'uint32' } }
 
 ##
 # @NetClientOptions

qemu-options.hx +3 -2

index 64af16d..23f010f 100644
@@ -1893,13 +1893,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single
 netdev.  @code{-net} and @code{-device} with parameter @option{vlan} create the
 required hub automatically.
 
-@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off]
+@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n]
 
 Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should
 be a unix domain socket backed one. The vhost-user uses a specifically defined
 protocol to pass vhost ioctl replacement messages to an application on the other
 end of the socket. On non-MSIX guests, the feature can be forced with
-@var{vhostforce}.
+@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to
+be created for multiqueue vhost-user.
 
 Example:
 @example

Quoted 7 lines · Nikolay Nikolaev

Vhost-user will implement the multiqueueu support in a similar way to what vhost already has - a separate thread for each queue.

To enable multiquue funcionality - a new command line parameter "queues" is introduced for the vhost-user netdev.

Signed-off-by Nikolay Nikolaev <[email protected]>

Nikolay - plan to repost addressing comments?

Quoted 18 lines

---
 docs/specs/vhost-user.txt |    7 +++++++
 hw/virtio/vhost-user.c    |    6 +++++-
 net/vhost-user.c          |   35 +++++++++++++++++++++++------------
 qapi-schema.json          |    5 ++++-
 qemu-options.hx           |    5 +++--
 5 files changed, 42 insertions(+), 16 deletions(-)

docs/specs/vhost-user.txt +0 -0

index 650bb18..d3857f5 100644
@@ -127,6 +127,13 @@ in the ancillary data:
 If Master is unable to send the full message or receives a wrong reply it will
 close the connection. An optional reconnection mechanism can be implemented.
 

docs/specs/vhost-user.txt in the ancillary data: · line 139

Quoted 10 lines

docs/specs/vhost-user.txt +7 -0

+Multi queue suport
+---------------------
+The protocol supports multiple queues by setting all index fields in the sent
+messages to a value calculated by the following formula:
+<queue idx> + <vring idx>
+The <queue idx> is increased by 2.
+
 Message types
 -------------
 

How is the support negotiated though?

What if I set queues=N with a legacy backend that
does not support multiqueue?

Quoted 142 lines

hw/virtio/vhost-user.c +5 -1

index aefe0bb..83ebcaa 100644
@@ -253,17 +253,20 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,
     case VHOST_SET_VRING_NUM:
     case VHOST_SET_VRING_BASE:
         memcpy(&msg.state, arg, sizeof(struct vhost_vring_state));
+        msg.state.index += dev->vq_index;
         msg.size = sizeof(m.state);
         break;
 
     case VHOST_GET_VRING_BASE:
         memcpy(&msg.state, arg, sizeof(struct vhost_vring_state));
+        msg.state.index += dev->vq_index;
         msg.size = sizeof(m.state);
         need_reply = 1;
         break;
 
     case VHOST_SET_VRING_ADDR:
         memcpy(&msg.addr, arg, sizeof(struct vhost_vring_addr));
+        msg.addr.index += dev->vq_index;
         msg.size = sizeof(m.addr);
         break;
 

@@ -271,7 +274,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,
     case VHOST_SET_VRING_CALL:
     case VHOST_SET_VRING_ERR:
         file = arg;
-        msg.u64 = file->index & VHOST_USER_VRING_IDX_MASK;
+        msg.u64 = (file->index + dev->vq_index) & VHOST_USER_VRING_IDX_MASK;
         msg.size = sizeof(m.u64);
         if (ioeventfd_enabled() && file->fd > 0) {
             fds[fd_num++] = file->fd;

@@ -313,6 +316,7 @@ static int vhost_user_call(struct vhost_dev *dev, unsigned long int request,
                 error_report("Received bad msg size.\n");
                 return -1;
             }
+            msg.state.index -= dev->vq_index;
             memcpy(arg, &msg.state, sizeof(struct vhost_vring_state));
             break;
         default:

net/vhost-user.c +23 -12

index 24e050c..1ea2f98 100644
@@ -134,25 +134,27 @@ static void net_vhost_user_event(void *opaque, int event)
 
 static int net_vhost_user_init(NetClientState *peer, const char *device,
                                const char *name, CharDriverState *chr,
-                               bool vhostforce)
+                               bool vhostforce, uint32_t queues)
 {
     NetClientState *nc;
     VhostUserState *s;
+    int i;
 
-    nc = qemu_new_net_client(&net_vhost_user_info, peer, device, name);
+    for (i = 0; i < queues; i++) {
+        nc = qemu_new_net_client(&net_vhost_user_info, peer, device, name);
 
-    snprintf(nc->info_str, sizeof(nc->info_str), "vhost-user to %s",
-             chr->label);
+        snprintf(nc->info_str, sizeof(nc->info_str), "vhost-user%d to %s",
+                 i, chr->label);
 
-    s = DO_UPCAST(VhostUserState, nc, nc);
+        s = DO_UPCAST(VhostUserState, nc, nc);
 
-    /* We don't provide a receive callback */
-    s->nc.receive_disabled = 1;
-    s->chr = chr;
-    s->vhostforce = vhostforce;
-
-    qemu_chr_add_handlers(s->chr, NULL, NULL, net_vhost_user_event, s);
+        /* We don't provide a receive callback */
+        s->nc.receive_disabled = 1;
+        s->chr = chr;
+        s->vhostforce = vhostforce;
 
+        qemu_chr_add_handlers(s->chr, NULL, NULL, net_vhost_user_event, s);
+    }
     return 0;
 }
 

@@ -228,6 +230,7 @@ static int net_vhost_check_net(QemuOpts *opts, void *opaque)
 int net_init_vhost_user(const NetClientOptions *opts, const char *name,
                         NetClientState *peer)
 {
+    uint32_t queues;
     const NetdevVhostUserOptions *vhost_user_opts;
     CharDriverState *chr;
     bool vhostforce;

@@ -254,5 +257,13 @@ int net_init_vhost_user(const NetClientOptions *opts, const char *name,
         vhostforce = false;
     }
 
-    return net_vhost_user_init(peer, "vhost_user", name, chr, vhostforce);
+    /* number of queues for multiqueue */
+    if (vhost_user_opts->has_queues) {
+        queues = vhost_user_opts->queues;
+    } else {
+        queues = 1;
+    }
+
+    return net_vhost_user_init(peer, "vhost_user", name, chr, vhostforce,
+                               queues);
 }

qapi-schema.json +4 -1

index 9ffdcf8..aa3bb6f 100644
@@ -2208,12 +2208,15 @@
 #
 # @vhostforce: #optional vhost on for non-MSIX virtio guests (default: false).
 #
+# @queues: #optional number of queues to be created for multiqueue vhost-user
+#
 # Since 2.1
 ##
 { 'type': 'NetdevVhostUserOptions',
   'data': {
     'chardev':        'str',
-    '*vhostforce':    'bool' } }
+    '*vhostforce':    'bool',
+    '*queues':        'uint32' } }
 
 ##
 # @NetClientOptions

qemu-options.hx +1 -1

index 64af16d..23f010f 100644
@@ -1893,13 +1893,14 @@ The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single
 netdev.  @code{-net} and @code{-device} with parameter @option{vlan} create the
 required hub automatically.
 
-@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off]
+@item -netdev vhost-user,chardev=@var{id}[,vhostforce=on|off][,queues=n]
 

qemu-options.hx The hubport netdev lets you connect a NIC to a QEMU "vlan" instead of a single · line 1906

Quoted 10 lines

qemu-options.hx +2 -1

 Establish a vhost-user netdev, backed by a chardev @var{id}. The chardev should
 be a unix domain socket backed one. The vhost-user uses a specifically defined
 protocol to pass vhost ioctl replacement messages to an application on the other
 end of the socket. On non-MSIX guests, the feature can be forced with
-@var{vhostforce}.
+@var{vhostforce}. Use 'queues=@var{n}' to specify the number of queues to
+be created for multiqueue vhost-user.
 
 Example:
 @example

--
You received this message because you are subscribed to the Google Groups "Snabb Switch development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send an email to [email protected].
Visit this group at http://groups.google.com/group/snabb-devel.