Implement multiqueue virtio-net
Subsystems (6)
v3 cover Implement multiqueue virtio-net
Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.
Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:
struct send_queue {
struct virtqueue *svq;
struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
} ____cacheline_aligned_in_smp; and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.
Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:
vhost=on -> #txqueues = 1
vhost=on,mq=on -> #txqueues = 4
vhost=on,mq=on,numtxqs=2 -> #txqueues = 2
vhost=on,mq=on,numtxqs=8 -> #txqueues = 8 Performance (guest -> local host):
-----------------------------------System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).
______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________
1 4.46 -1.96 .19 -12.50 -6.06 2 4.93 -1.16 2.10 0 -2.38 4 46.17 64.77 33.72 19.51 -2.48 8 47.89 70.00 36.23 41.46 13.35 16 48.97 80.44 40.67 21.11 -5.46 24 49.03 78.78 41.22 20.51 -4.78 32 51.11 77.15 42.42 15.81 -6.87 40 51.60 71.65 42.43 9.75 -8.94 48 50.10 69.55 42.85 11.80 -5.81 64 46.24 68.42 42.67 14.18 -3.28 80 46.37 63.13 41.62 7.43 -6.73 96 46.40 63.31 42.20 9.36 -4.78 128 50.43 62.79 42.16 13.11 -1.23
________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%
______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________
1 -.76 -1.56 2.33 0 3.03 2 17.41 11.11 11.41 0 -4.76 4 42.12 55.11 30.20 19.51 .62 8 54.69 80.00 39.22 24.39 -3.88 16 54.77 81.62 40.89 20.34 -6.58 24 54.66 79.68 41.57 15.49 -8.99 32 54.92 76.82 41.79 17.59 -5.70 40 51.79 68.56 40.53 15.31 -3.87 48 51.72 66.40 40.84 9.72 -7.13 64 51.11 63.94 41.10 5.93 -8.82 80 46.51 59.50 39.80 9.33 -4.18 96 47.72 57.75 39.84 4.20 -7.62 128 54.35 58.95 40.66 3.24 -8.63
________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%
______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________
1 -1.43 -3.52 1.55 0 3.03 2 33.09 21.63 20.12 -10.00 -9.52 4 67.17 94.60 44.28 19.51 -11.80 8 75.72 108.14 49.15 25.00 -10.71 16 80.34 101.77 52.94 25.93 -4.49 24 70.84 93.12 43.62 27.63 -5.03 32 69.01 94.16 47.33 29.68 -1.51 40 58.56 63.47 25.91 -3.92 -25.85 48 61.16 74.70 34.88 .89 -22.08 64 54.37 69.09 26.80 -6.68 -30.04 80 36.22 22.73 -2.97 -8.25 -27.23 96 41.51 50.59 13.24 9.84 -16.77 128 48.98 38.15 6.41 -.33 -22.80
________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%
______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________
1 7.62 -38.03 -26.26 -50.00 -33.33 2 28.95 20.46 21.62 0 -7.14 4 84.05 60.79 45.74 -2.43 -12.42 8 86.43 79.57 50.32 15.85 -3.10 16 88.63 99.48 58.17 9.47 -13.10 24 74.65 80.87 41.99 -1.81 -22.89 32 63.86 59.21 23.58 -18.13 -36.37 40 64.79 60.53 22.23 -15.77 -35.84 48 49.68 26.93 .51 -36.40 -49.61 64 54.69 36.50 5.41 -26.59 -43.23 80 45.06 12.72 -13.25 -37.79 -52.08 96 40.21 -3.16 -24.53 -39.92 -52.97 128 36.33 -33.19 -43.66 -5.68 -20.49
________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%
---
RFC PATCH v3 1/4 Change virtqueue structure
Move queue_index from virtio_pci_vq_info to virtqueue. This allows callback handlers to figure out the queue number for the vq that needs attention.
--- drivers/virtio/virtio_pci.c | 10 +++------- include/linux/virtio.h | 1 + 2 files changed, 4 insertions(+), 7 deletions(-)
include/linux/virtio.h +1 -0
@@ -22,6 +22,7 @@ struct virtqueue {void (*callback)(struct virtqueue *vq);const char *name;struct virtio_device *vdev; +int queue_index;/* the index of the queue */ void *priv; };
drivers/virtio/virtio_pci.c +3 -7
@@ -75,9 +75,6 @@ struct virtio_pci_vq_info/* the number of entries in the queue */ int num; -/* the index of the queue */ -int queue_index; -/* the virtual address of the ring queue */ void *queue;
@@ -185,11 +182,10 @@ static void vp_reset(struct virtio_devicstatic void vp_notify (struct virtqueue *vq) {struct virtio_pci_device *vp_dev =to_vp_device (vq->vdev); -struct virtio_pci_vq_info *info = vq->priv;/* we write the queue's selector into the notification register to * signal the other end */ -iowrite16 (info->queue_index, vp_dev->ioaddr +VIRTIO_PCI_QUEUE_NOTIFY ); +iowrite16 (vq->queue_index, vp_dev->ioaddr +VIRTIO_PCI_QUEUE_NOTIFY ); }/* Handle a configuration change: Tell driver if it wants to know. */
@@ -385,7 +381,6 @@ static struct virtqueue *setup_vq(structif (!info)return ERR_PTR (-ENOMEM ); - info->queue_index = index; info->num = num; info->msix_vector = msix_vec;
@@ -408,6 +403,7 @@ static struct virtqueue *setup_vq(struct goto out_activate_queue; } + vq->queue_index = index; vq->priv = info; info->vq = vq;
@@ -446,7 +442,7 @@ static void vp_del_vq(struct virtqueue *list_del (&info->node);spin_unlock_irqrestore (&vp_dev->lock, flags); -iowrite16 (info->queue_index, vp_dev->ioaddr +VIRTIO_PCI_QUEUE_SEL ); +iowrite16 (vq->queue_index, vp_dev->ioaddr +VIRTIO_PCI_QUEUE_SEL );if (vp_dev->msix_enabled) { iowrite16(VIRTIO_MSI_NO_VECTOR,
RFC PATCH v3 2/4 Changes for virtio-net
In reply to Krishna KumarImplement mq virtio-net driver.
Though struct virtio_net_config changes, it works with old qemu's since the last element is not accessed, unless qemu sets VIRTIO_NET_F_NUMTXQS. Patch also adds a macro for the maximum number of TX vq's (VIRTIO_MAX_SQ) that the user can specify.
---
drivers/net/virtio_net.c | 234 ++++++++++++++++++++++++++--------- include/linux/virtio_net.h | 6 2 files changed, 185 insertions(+), 55 deletions(-)
include/linux/virtio_net.h +6 -0
@@ -7,6 +7,9 @@#include <linux/virtio_config.h> #include <linux/if_ether.h> +/* Maximum number of TX queues supported */ +#define VIRTIO_MAX_SQ 32 +/* The feature bitmap for virtio net */ #define VIRTIO_NET_F_CSUM 0/* Host handles pkts w/ partial csum */ #define VIRTIO_NET_F_GUEST_CSUM 1/* Guest handles pkts w/ partial csum */
@@ -26,6 +29,7 @@#define VIRTIO_NET_F_CTRL_RX 18/* Control channel RX mode support */ #define VIRTIO_NET_F_CTRL_VLAN 19/* Control channel VLAN filtering */ #define VIRTIO_NET_F_CTRL_RX_EXTRA 20/* Extra RX mode control support */ +#define VIRTIO_NET_F_NUMTXQS 21/* Device supports multiple TX queue */ #define VIRTIO_NET_S_LINK_UP 1/* Link is up */
@@ -34,6 +38,8 @@ struct virtio_net_config {__u8 mac[6 ];/* See VIRTIO_NET_F_STATUS and VIRTIO_NET_S_* above */ __u16 status; +/* number of transmit queues */ +__u16 numtxqs; } __attribute__((packed));/* This is the first element of the scatter-gather list. If you don't
drivers/net/virtio_net.c +179 -55
378 lines not on this page · raw message
RFC PATCH v3 3/4 Changes for vhost
In reply to Krishna KumarChanges for mq vhost.
vhost_net_open is changed to allocate a vhost_net and return. The remaining initializations are delayed till SET_OWNER. SET_OWNER is changed so that the argument is used to determine how many txqs to use. Unmodified qemu's will pass NULL, so this is recognized and handled as numtxqs=1.
Besides changing handle_tx to use 'vq', this patch also changes handle_rx to take vq as parameter. The mq RX patch requires this change, but till then it is consistent (and less confusing) to make the interfaces for handling rx and tx similar.
vhost thread handling for RX and TX is as follows. The first vhost thread handles RX traffic, while the remaining threads handles TX. The number of threads is <= #txqs, and threads handle more than one txq when #txqs is more than MAX_VHOST_THREADS (4). When guest is started with >1 txqs and there is only one stream of traffic from the guest, that is recognized and handled such that vhost[0] processes both RX and TX. This can change dynamically. vhost_poll has a new element - find_vq(), which allows optimizing some code for cases where numtxqs=1 or a packet on vhost[0] needs processing.
--- drivers/vhost/net.c | 284 ++++++++++++++++++++++++++-------------- drivers/vhost/vhost.c | 275 ++++++++++++++++++++++++++++---------- drivers/vhost/vhost.h | 42 +++++ 3 files changed, 430 insertions(+), 171 deletions(-)
drivers/vhost/vhost.h +36 -6
@@ -35,11 +35,13 @@ struct vhost_poll {wait_queue_t wait;struct vhost_work work;unsigned long mask; -struct vhost_dev *dev; +struct vhost_virtqueue *(*find_vq)(struct vhost_poll *poll); +struct vhost_virtqueue *vq;/* points back to vq */ };void vhost_poll_init (struct vhost_poll *poll,vhost_work_fn_t fn, -unsigned long mask,struct vhost_dev *dev); +unsigned long mask,struct vhost_virtqueue *vq, +int single_queue);void vhost_poll_start (struct vhost_poll *poll,struct file *file);void vhost_poll_stop (struct vhost_poll *poll);void vhost_poll_flush (struct vhost_poll *poll);
@@ -108,6 +110,10 @@ struct vhost_virtqueue {/* Log write descriptors */ void __user *log_base;struct vhost_log *log; +struct task_struct *worker;/* vhost for this vq, can be shared */ +spinlock_t *work_lock; +struct list_head *work_list; +int qnum;/* 0 for RX, 1 -> n-1 for TX */ };struct vhost_dev {
@@ -119,15 +125,39 @@ struct vhost_dev {struct mutex mutex;unsigned acked_features;struct vhost_virtqueue *vqs; +unsigned long *jiffies;int nvqs;struct file *log_file;struct eventfd_ctx *log_ctx; -spinlock_t work_lock; -struct list_head work_list; -struct task_struct *worker; +spinlock_t *work_lock; +struct list_head *work_list; }; -long vhost_dev_init (struct vhost_dev *,struct vhost_virtqueue *vqs,int nvqs); +/* +* Define maximum number of TX threads, and use that to have a maximum +* number of vhost threads to handle RX & TX. First thread handles RX. +* If guest is started with #txqs=1, only one vhost thread is started. +* Else, upto MAX_VHOST_THREADS are started where th[0] handles RX and +* remaining handles TX. However, vhost_poll_queue has an optimization +* where th[0] is selected for both RX & TX if there is only one flow. +*/ +#define MAX_TXQ_THREADS 4 +#define MAX_VHOST_THREADS (MAX_TXQ_THREADS + 1) + +static inline int get_nvhosts (int nvqs) +{ +int num_vhosts = nvqs -1 ; + +if (nvqs >2 ) + num_vhosts =min_t (int, nvqs,MAX_VHOST_THREADS ); + +return num_vhosts; +} + +int vhost_setup_vqs (struct vhost_dev *dev,int numtxqs); +void vhost_free_vqs (struct vhost_dev *dev); +long vhost_dev_init (struct vhost_dev *,struct vhost_virtqueue *vqs,int nvqs, +int nvhosts);long vhost_dev_check_owner (struct vhost_dev *);long vhost_dev_reset_owner (struct vhost_dev *);void vhost_dev_cleanup (struct vhost_dev *);
drivers/vhost/net.c +189 -95
465 lines not on this page · raw message
drivers/vhost/vhost.c +205 -70
411 lines not on this page · raw message
RFC PATCH v3 4/4 qemu changes
In reply to Krishna KumarChanges in qemu to support mq TX.
---
hw/vhost.c | 7 ++++-- hw/vhost.h | 2 - hw/vhost_net.c | 16 +++++++++---- hw/vhost_net.h | 2 - hw/virtio-net.c | 53 ++++++++++++++++++++++++++++++++-------------- hw/virtio-net.h | 2 + hw/virtio-pci.c | 2 + net.c | 17 ++++++++++++++ net.h | 1 net/tap.c | 34 ++++++++++++++++++++++++++--- 10 files changed, 107 insertions(+), 29 deletions(-)
hw/vhost.c +5 -2
@@ -580,7 +580,7 @@ static void vhost_virtqueue_cleanup(stru 0, virtio_queue_get_desc_size(vdev, idx)); } -int vhost_dev_init (struct vhost_dev *hdev,int devfd) +int vhost_dev_init (struct vhost_dev *hdev,int devfd,int numtxqs) {uint64_t features;int r;
@@ -592,11 +592,14 @@ int vhost_dev_init(struct vhost_dev *hdereturn -errno; } } - r =ioctl (hdev->control,VHOST_SET_OWNER ,NULL ); + + r =ioctl (hdev->control,VHOST_SET_OWNER , numtxqs);if (r <0 ) { goto fail; } + hdev->nvqs = numtxqs +1 ; + r =ioctl (hdev->control,VHOST_GET_FEATURES , &features);if (r <0 ) { goto fail;
hw/vhost.h +1 -1
@@ -40,7 +40,7 @@ struct vhost_dev {unsigned long long log_size; }; -int vhost_dev_init (struct vhost_dev *hdev,int devfd); +int vhost_dev_init (struct vhost_dev *hdev,int devfd,int numtxqs);void vhost_dev_cleanup (struct vhost_dev *hdev);int vhost_dev_start (struct vhost_dev *hdev,VirtIODevice *vdev);void vhost_dev_stop (struct vhost_dev *hdev,VirtIODevice *vdev);
hw/vhost_net.c +11 -5
@@ -36,7 +36,8 @@struct vhost_net {struct vhost_dev dev; -struct vhost_virtqueue vqs[2 ]; +struct vhost_virtqueue *vqs; +int nvqs;int backend;VLANClientState *vc; };
@@ -81,7 +82,8 @@ static int vhost_net_get_fd(VLANClientSt } } -struct vhost_net *vhost_net_init (VLANClientState *backend,int devfd) +struct vhost_net *vhost_net_init (VLANClientState *backend,int devfd, +int numtxqs) {int r;struct vhost_net *net =qemu_malloc (sizeof *net);
@@ -98,10 +100,14 @@ struct vhost_net *vhost_net_init(VLANCli (1 <<VHOST_NET_F_VIRTIO_NET_HDR ); net->backend = r; - r =vhost_dev_init (&net->dev, devfd); + r =vhost_dev_init (&net->dev, devfd, numtxqs);if (r <0 ) { goto fail; } + + net->nvqs = numtxqs +1 ; + net->vqs =qemu_malloc (net->nvqs * (sizeof *net->vqs)); +if (!tap_has_vnet_hdr_len (backend,sizeof (struct virtio_net_hdr_mrg_rxbuf ))) { net->dev.features &= ~(1 <<VIRTIO_NET_F_MRG_RXBUF );
@@ -131,7 +137,6 @@ int vhost_net_start(struct vhost_net *ne sizeof(struct virtio_net_hdr_mrg_rxbuf)); } - net->dev.nvqs =2 ; net->dev.vqs = net->vqs; r =vhost_dev_start (&net->dev, dev);if (r <0 ) {
@@ -188,7 +193,8 @@ void vhost_net_cleanup(struct vhost_net qemu_free(net); }#else -struct vhost_net *vhost_net_init (VLANClientState *backend,int devfd) +struct vhost_net *vhost_net_init (VLANClientState *backend,int devfd, +int nvqs) {return NULL ; }
hw/vhost_net.h +1 -1
@@ -6,7 +6,7 @@struct vhost_net ;typedef struct vhost_net VHostNetState ; -VHostNetState *vhost_net_init (VLANClientState *backend,int devfd); +VHostNetState *vhost_net_init (VLANClientState *backend,int devfd,int nvqs);int vhost_net_start (VHostNetState *net,VirtIODevice *dev);void vhost_net_stop (VHostNetState *net,VirtIODevice *dev);
hw/virtio-net.c +37 -16
156 lines not on this page · raw message
hw/virtio-net.h +2 -0
@@ -44,6 +44,7 @@#define VIRTIO_NET_F_CTRL_RX 18/* Control channel RX mode support */ #define VIRTIO_NET_F_CTRL_VLAN 19/* Control channel VLAN filtering */ #define VIRTIO_NET_F_CTRL_RX_EXTRA 20/* Extra RX mode control support */ +#define VIRTIO_NET_F_NUMTXQS 21/* Supports multiple TX queues */ #define VIRTIO_NET_S_LINK_UP 1/* Link is up */
@@ -72,6 +73,7 @@ struct virtio_net_configuint8_t mac[ETH_ALEN ];/* See VIRTIO_NET_F_STATUS and VIRTIO_NET_S_* above */ uint16_t status; +uint16_t numtxqs;/* number of transmit queues */ } __attribute__((packed));/* This is the first element of the scatter-gather list. If you don't
hw/virtio-pci.c +2 -0
@@ -99,6 +99,7 @@ typedef struct {uint32_t addr;uint32_t class_code;uint32_t nvectors; +uint32_t mq;BlockConf block;NICConf nic;uint32_t host_features;
@@ -788,6 +789,7 @@ static PCIDeviceInfo virtio_info[] = { .romfile = "pxe-virtio.bin", .qdev.props = (Property []) {DEFINE_PROP_UINT32 ("vectors" , VirtIOPCIProxy, nvectors,3 ), +DEFINE_PROP_UINT32 ("mq" , VirtIOPCIProxy, mq,1 ),DEFINE_VIRTIO_NET_FEATURES (VirtIOPCIProxy, host_features),DEFINE_NIC_PROPERTIES (VirtIOPCIProxy, nic), DEFINE_PROP_UINT32("x-txtimer", VirtIOPCIProxy,
net/tap.c +30 -4
@@ -320,13 +320,14 @@ static NetClientInfo net_tap_info = {static TAPState *net_tap_fd_init (VLANState *vlan,const char *model,const char *name, -int fd, +int fd,int numtxqs,int vnet_hdr) {VLANClientState *nc;TAPState *s; nc =qemu_new_net_client (&net_tap_info, vlan,NULL , model, name); + nc->numtxqs = numtxqs; s =DO_UPCAST (TAPState, nc, nc);
@@ -424,6 +425,27 @@ int net_init_tap(QemuOpts *opts, Monitor {TAPState *s;int fd, vnet_hdr =0 ; +int vhost; +int numtxqs =1 ; + + vhost =qemu_opt_get_bool (opts,"vhost" ,0 ); + +/* +* We support multiple tx queues if: +* 1. smp > 1 +* 2. vhost=on +* 3. mq=on +* In this case, #txqueues = #cpus. This value can be changed by +* using the "numtxqs" option. +*/ +if (vhost && smp_cpus >1 ) { +if (qemu_opt_get_bool (opts,"mq" ,0 )) { +#define VIRTIO_MAX_TXQS 32 +int dflt =MIN (smp_cpus,VIRTIO_MAX_TXQS ); + + numtxqs =qemu_opt_get_number (opts,"numtxqs" , dflt); + } + }if (qemu_opt_get (opts,"fd" )) { if (qemu_opt_get(opts, "ifname") ||
@@ -457,7 +479,7 @@ int net_init_tap(QemuOpts *opts, Monitor } } - s =net_tap_fd_init (vlan,"tap" , name, fd, vnet_hdr); + s =net_tap_fd_init (vlan,"tap" , name, fd, numtxqs, vnet_hdr);if (!s) {close (fd);return -1 ;
@@ -486,7 +508,7 @@ int net_init_tap(QemuOpts *opts, Monitor } } -if (qemu_opt_get_bool (opts,"vhost" , !!qemu_opt_get (opts,"vhostfd" ))) { +if (vhost) {int vhostfd, r;if (qemu_opt_get (opts,"vhostfd" )) { r =net_handle_fd_param (mon,qemu_opt_get (opts,"vhostfd" ));
@@ -497,9 +519,13 @@ int net_init_tap(QemuOpts *opts, Monitor }else { vhostfd =-1 ; } - s->vhost_net =vhost_net_init (&s->nc, vhostfd); + s->vhost_net =vhost_net_init (&s->nc, vhostfd, numtxqs);if (!s->vhost_net) {error_report ("vhost-net requested but could not be initialized" ); +if (numtxqs >1 ) { +error_report ("Need vhost support for numtxqs > 1, exiting..." ); +exit (1 ); + }return -1 ; } }else if (qemu_opt_get (opts,"vhostfd" )) {
net.c +17 -0
@@ -849,6 +849,15 @@ static int net_init_nic(QemuOpts *opts,return -1 ; } +if (nd->netdev->numtxqs >1 && nd->nvectors ==DEV_NVECTORS_UNSPECIFIED ) { +/* +* User specified mq for guest, but no "vectors=", tune +* it automatically to 'numtxqs' TX + 1 RX + 1 controlq. +*/ + nd->nvectors = nd->netdev->numtxqs +1 +1 ; +monitor_printf (mon,"nvectors tuned to %d\n" , nd->nvectors); + } + nd->used =1 ; nb_nics++;
@@ -992,6 +1001,14 @@ static const struct { }, #ifndef _WIN32 { + .name = "mq", + .type = QEMU_OPT_BOOL, + .help = "enable multiqueue on network i/f", + }, { + .name = "numtxqs", + .type = QEMU_OPT_NUMBER, + .help = "optional number of TX queues, if mq is enabled", + }, { .name = "fd", .type = QEMU_OPT_STRING, .help = "file descriptor of an already opened tap",
net.h +1 -0
@@ -62,6 +62,7 @@ struct VLANClientState {struct VLANState *vlan;VLANClientState *peer;NetQueue *send_queue; +int numtxqs;char *model;char *name;char info_str[256 ];
Quoted 1 line
Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:
Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.
Some more test results:
_____________________________________________________
Host->Guest BW (numtxqs=2) # BW% CPU% RCPU% SD% RSD% _____________________________________________________ 1 5.53 .31 .67 -5.88 0 2 -2.11 -1.01 -2.08 4.34 0 4 13.53 10.77 13.87 -1.96 0 8 34.22 22.80 30.53 -8.46 -2.50 16 30.89 24.06 35.17 -5.20 3.20 24 33.22 26.30 43.39 -5.17 7.58 32 30.85 27.27 47.74 -.59 15.51 40 33.80 27.33 48.00 -7.42 7.59 48 45.93 26.33 45.46 -12.24 1.10 64 33.51 27.11 45.00 -3.27 10.30 80 39.28 29.21 52.33 -4.88 12.17 96 32.05 31.01 57.72 -1.02 19.05 128 35.66 32.04 60.00 -.66 20.41
_____________________________________________________
BW: 23.5% CPU/RCPU: 28.6%,51.2% SD/RSD: -2.6%,15.8%
____________________________________________________Guest->Host 512 byte (numtxqs=2):
# BW% CPU% RCPU% SD% RSD%
_____________________________________________________
1 3.02 -3.84 -4.76 -12.50 -7.69 2 52.77 -15.73 -8.66 -45.31 -40.33 4 -23.14 13.84 7.50 50.58 40.81 8 -21.44 28.08 16.32 63.06 47.43 16 33.53 46.50 27.19 7.61 -6.60 24 55.77 42.81 30.49 -8.65 -16.48 32 52.59 38.92 29.08 -9.18 -15.63 40 50.92 36.11 28.92 -10.59 -15.30 48 46.63 34.73 28.17 -7.83 -12.32 64 45.56 37.12 28.81 -5.05 -10.80 80 44.55 36.60 28.45 -4.95 -10.61 96 43.02 35.97 28.89 -.11 -5.31 128 38.54 33.88 27.19 -4.79 -9.54
_____________________________________________________
BW: 34.4% CPU/RCPU: 35.9%,27.8% SD/RSD: -4.1%,-9.3%
Thanks,
- KK
Quoted 127 lines
[v3 RFC PATCH 0/4] Implement multiqueue virtio-net
Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.
Changes from rev2:
------------------
1. Define (invirtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:struct send_queue { struct virtqueue *svq; struct scatterlist tx_sg[MAX_SKB_FRAGS + 2]; } ____cacheline_aligned_in_smp;and to statically allocate '
VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:vhost=on -> #txqueues = 1 vhost=on,mq=on -> #txqueues = 4 vhost=on,mq=on,numtxqs=2 -> #txqueues = 2 vhost=on,mq=on,numtxqs=8 -> #txqueues = 8Performance (guest -> local host): -----------------------------------System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 4.46 -1.96 .19 -12.50 -6.06 2 4.93 -1.16 2.10 0 -2.38 4 46.17 64.77 33.72 19.51 -2.48 8 47.89 70.00 36.23 41.46 13.35 16 48.97 80.44 40.67 21.11 -5.46 24 49.03 78.78 41.22 20.51 -4.78 32 51.11 77.15 42.42 15.81 -6.87 40 51.60 71.65 42.43 9.75 -8.94 48 50.10 69.55 42.85 11.80 -5.81 64 46.24 68.42 42.67 14.18 -3.28 80 46.37 63.13 41.62 7.43 -6.73 96 46.40 63.31 42.20 9.36 -4.78 128 50.43 62.79 42.16 13.11 -1.23________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -.76 -1.56 2.33 0 3.03 2 17.41 11.11 11.41 0 -4.76 4 42.12 55.11 30.20 19.51 .62 8 54.69 80.00 39.22 24.39 -3.88 16 54.77 81.62 40.89 20.34 -6.58 24 54.66 79.68 41.57 15.49 -8.99 32 54.92 76.82 41.79 17.59 -5.70 40 51.79 68.56 40.53 15.31 -3.87 48 51.72 66.40 40.84 9.72 -7.13 64 51.11 63.94 41.10 5.93 -8.82 80 46.51 59.50 39.80 9.33 -4.18 96 47.72 57.75 39.84 4.20 -7.62 128 54.35 58.95 40.66 3.24 -8.63________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -1.43 -3.52 1.55 0 3.03 2 33.09 21.63 20.12 -10.00 -9.52 4 67.17 94.60 44.28 19.51 -11.80 8 75.72 108.14 49.15 25.00 -10.71 16 80.34 101.77 52.94 25.93 -4.49 24 70.84 93.12 43.62 27.63 -5.03 32 69.01 94.16 47.33 29.68 -1.51 40 58.56 63.47 25.91 -3.92 -25.85 48 61.16 74.70 34.88 .89 -22.08 64 54.37 69.09 26.80 -6.68 -30.04 80 36.22 22.73 -2.97 -8.25 -27.23 96 41.51 50.59 13.24 9.84 -16.77 128 48.98 38.15 6.41 -.33 -22.80________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________1 7.62 -38.03 -26.26 -50.00 -33.33 2 28.95 20.46 21.62 0 -7.14 4 84.05 60.79 45.74 -2.43 -12.42 8 86.43 79.57 50.32 15.85 -3.10 16 88.63 99.48 58.17 9.47 -13.10 24 74.65 80.87 41.99 -1.81 -22.89 32 63.86 59.21 23.58 -18.13 -36.37 40 64.79 60.53 22.23 -15.77 -35.84 48 49.68 26.93 .51 -36.40 -49.61 64 54.69 36.50 5.41 -26.59 -43.23 80 45.06 12.72 -13.25 -37.79 -52.08 96 40.21 -3.16 -24.53 -39.92 -52.97 128 36.33 -33.19 -43.66 -5.68 -20.49________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%Signed-off-by Krishna Kumar <[email protected]>
Michael S. Tsirkin
maintainerQuoted 4 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line
Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:
Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.
I am trying to wrap my head around kernel/user interface here. E.g., will we need another incompatible change when we add multiple RX queues? Also need to think about how robust our single stream heuristic is, e.g. what are the chances it will misdetect a bidirectional
UDP stream as a single TCP?
Quoted 175 lines
Some more test results:
_____________________________________________________Host->Guest BW (numtxqs=2) # BW% CPU% RCPU% SD% RSD% _____________________________________________________ 1 5.53 .31 .67 -5.88 0 2 -2.11 -1.01 -2.08 4.34 0 4 13.53 10.77 13.87 -1.96 0 8 34.22 22.80 30.53 -8.46 -2.50 16 30.89 24.06 35.17 -5.20 3.20 24 33.22 26.30 43.39 -5.17 7.58 32 30.85 27.27 47.74 -.59 15.51 40 33.80 27.33 48.00 -7.42 7.59 48 45.93 26.33 45.46 -12.24 1.10 64 33.51 27.11 45.00 -3.27 10.30 80 39.28 29.21 52.33 -4.88 12.17 96 32.05 31.01 57.72 -1.02 19.05 128 35.66 32.04 60.00 -.66 20.41_____________________________________________________
BW: 23.5% CPU/RCPU: 28.6%,51.2% SD/RSD: -2.6%,15.8%____________________________________________________
Guest->Host512 byte (numtxqs=2):
# BW% CPU% RCPU% SD% RSD%
_____________________________________________________1 3.02 -3.84 -4.76 -12.50 -7.69 2 52.77 -15.73 -8.66 -45.31 -40.33 4 -23.14 13.84 7.50 50.58 40.81 8 -21.44 28.08 16.32 63.06 47.43 16 33.53 46.50 27.19 7.61 -6.60 24 55.77 42.81 30.49 -8.65 -16.48 32 52.59 38.92 29.08 -9.18 -15.63 40 50.92 36.11 28.92 -10.59 -15.30 48 46.63 34.73 28.17 -7.83 -12.32 64 45.56 37.12 28.81 -5.05 -10.80 80 44.55 36.60 28.45 -4.95 -10.61 96 43.02 35.97 28.89 -.11 -5.31 128 38.54 33.88 27.19 -4.79 -9.54_____________________________________________________
BW: 34.4% CPU/RCPU: 35.9%,27.8% SD/RSD: -4.1%,-9.3%Thanks,
- KK
Quoted 127 lines
[v3 RFC PATCH 0/4] Implement multiqueue virtio-net
Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.
Changes from rev2:
------------------
1. Define (invirtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:struct send_queue { struct virtqueue *svq; struct scatterlist tx_sg[MAX_SKB_FRAGS + 2]; } ____cacheline_aligned_in_smp;and to statically allocate '
VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:vhost=on -> #txqueues = 1 vhost=on,mq=on -> #txqueues = 4 vhost=on,mq=on,numtxqs=2 -> #txqueues = 2 vhost=on,mq=on,numtxqs=8 -> #txqueues = 8Performance (guest -> local host): -----------------------------------System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 4.46 -1.96 .19 -12.50 -6.06 2 4.93 -1.16 2.10 0 -2.38 4 46.17 64.77 33.72 19.51 -2.48 8 47.89 70.00 36.23 41.46 13.35 16 48.97 80.44 40.67 21.11 -5.46 24 49.03 78.78 41.22 20.51 -4.78 32 51.11 77.15 42.42 15.81 -6.87 40 51.60 71.65 42.43 9.75 -8.94 48 50.10 69.55 42.85 11.80 -5.81 64 46.24 68.42 42.67 14.18 -3.28 80 46.37 63.13 41.62 7.43 -6.73 96 46.40 63.31 42.20 9.36 -4.78 128 50.43 62.79 42.16 13.11 -1.23________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -.76 -1.56 2.33 0 3.03 2 17.41 11.11 11.41 0 -4.76 4 42.12 55.11 30.20 19.51 .62 8 54.69 80.00 39.22 24.39 -3.88 16 54.77 81.62 40.89 20.34 -6.58 24 54.66 79.68 41.57 15.49 -8.99 32 54.92 76.82 41.79 17.59 -5.70 40 51.79 68.56 40.53 15.31 -3.87 48 51.72 66.40 40.84 9.72 -7.13 64 51.11 63.94 41.10 5.93 -8.82 80 46.51 59.50 39.80 9.33 -4.18 96 47.72 57.75 39.84 4.20 -7.62 128 54.35 58.95 40.66 3.24 -8.63________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -1.43 -3.52 1.55 0 3.03 2 33.09 21.63 20.12 -10.00 -9.52 4 67.17 94.60 44.28 19.51 -11.80 8 75.72 108.14 49.15 25.00 -10.71 16 80.34 101.77 52.94 25.93 -4.49 24 70.84 93.12 43.62 27.63 -5.03 32 69.01 94.16 47.33 29.68 -1.51 40 58.56 63.47 25.91 -3.92 -25.85 48 61.16 74.70 34.88 .89 -22.08 64 54.37 69.09 26.80 -6.68 -30.04 80 36.22 22.73 -2.97 -8.25 -27.23 96 41.51 50.59 13.24 9.84 -16.77 128 48.98 38.15 6.41 -.33 -22.80________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________1 7.62 -38.03 -26.26 -50.00 -33.33 2 28.95 20.46 21.62 0 -7.14 4 84.05 60.79 45.74 -2.43 -12.42 8 86.43 79.57 50.32 15.85 -3.10 16 88.63 99.48 58.17 9.47 -13.10 24 74.65 80.87 41.99 -1.81 -22.89 32 63.86 59.21 23.58 -18.13 -36.37 40 64.79 60.53 22.23 -15.77 -35.84 48 49.68 26.93 .51 -36.40 -49.61 64 54.69 36.50 5.41 -26.59 -43.23 80 45.06 12.72 -13.25 -37.79 -52.08 96 40.21 -3.16 -24.53 -39.92 -52.97 128 36.33 -33.19 -43.66 -5.68 -20.49________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%Signed-off-by Krishna Kumar <[email protected]>
"Michael S. Tsirkin" <[email protected]> wrote on 10/25/2010 09:47:18 PM:
Quoted 6 lines
Quoted 2 lines
Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.
I am trying to wrap my head around kernel/user interface here. E.g., will we need another incompatible change when we add multiple RX queues?
Though I added a 'mq' option to qemu, there shouldn't be any incompatibility between old and new qemu's wrt vhost and virtio-net drivers. So the old qemu will run new host and new guest without issues, and new qemu can also run old host and old guest. Multiple RXQ will also not add any incompatibility.
With MQ RX, I will be able to remove the hueristic (idea from David Stevens). The idea is: Guest sends out packets on, say TXQ#2, vhost#2 processes the packets but packets going out from host to guest might be sent out on a different RXQ, say RXQ#4. Guest receives the packet on RXQ#4, and all future responses on that connection are sent on TXQ#4. Now vhost#4 processes both RX and TX packets for this connection. Without needing to hash on the connection, guest can make sure that the same vhost thread will handle a single connection.
Quoted 3 lines
Also need to think about how robust our single stream heuristic is, e.g. what are the chances it will misdetect a bidirectional UDP stream as a single TCP?
I think it should not happen. The hueristic code gets called for handling just the transmit packets, packets that vhost sends out to the guest skip this path.
I tested unidirectional and bidirectional UDP to confirm:
8 iterations of iperf tests, each iteration of 15 secs, result is the sum of all 8 iterations in Gbits/sec
__________________________________________
Uni-directional Bi-directional
Org New Org New
__________________________________________
71.78 71.77 71.74 72.07
__________________________________________
Thanks,
- KK
A message that is not in the archive (sent off-list or not yet received): OF5C53E9CF.FFDF2CE7-ON652577C8.00191D14-652577C8.001C2154@LocalDomain
Krishna Kumar2/India/IBM wrote on 10/26/2010 10:40:35 AM:
Quoted 32 lines
Quoted 3 lines
I am trying to wrap my head around kernel/user interface here. E.g., will we need another incompatible change when we add multiple RX queues?
Though I added a 'mq' option to qemu, there shouldn't be any incompatibility between old and new qemu's wrt vhost and virtio-net drivers. So the old qemu will run new host and new guest without issues, and new qemu can also run old host and old guest. Multiple RXQ will also not add any incompatibility.
With MQ RX, I will be able to remove the hueristic (idea from David Stevens). The idea is: Guest sends out packets on, say TXQ#2, vhost#2 processes the packets but packets going out from host to guest might be sent out on a different RXQ, say RXQ#4. Guest receives the packet on RXQ#4, and all future responses on that connection are sent on TXQ#4. Now vhost#4 processes both RX and TX packets for this connection. Without needing to hash on the connection, guest can make sure that the same vhost thread will handle a single connection.
Quoted 3 lines
Also need to think about how robust our single stream heuristic is, e.g. what are the chances it will misdetect a bidirectional UDP stream as a single TCP?
I think it should not happen. The hueristic code gets called for handling just the transmit packets, packets that vhost sends out to the guest skip this path.
I tested unidirectional and bidirectional UDP to confirm:
Quoted 8 lines
8 iterations of iperf tests, each iteration of 15 secs, result is the sum of all 8 iterations in Gbits/sec
__________________________________________
Uni-directional Bi-directional
Org New Org New
__________________________________________
71.78 71.77 71.74 72.07
__________________________________________
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
------ numtxqs=8, vhosts=5 ---------
# BW% CPU% SD%
------------------------------------
1 .49 1.07 0 2 23.51 52.51 26.66 4 75.17 72.43 8.57 8 86.54 80.21 27.85 16 92.37 85.99 6.27 24 91.37 84.91 8.41 32 89.78 82.90 3.31 48 89.85 79.95 -3.57 64 85.83 80.28 2.22 80 88.90 79.47 -23.18 96 90.12 79.98 14.71 128 86.13 80.60 4.42
------------------------------------
BW: 71.3%, CPU: 80.4%, SD: 1.2%
------ numtxqs=16, vhosts=5 --------
# BW% CPU% SD%
------------------------------------
1 1.80 0 0 2 19.81 50.68 26.66 4 57.31 52.77 8.57 8 108.44 88.19 -5.21 16 106.09 85.03 -4.44 24 102.34 84.23 -.82 32 102.77 82.71 -5.81 48 100.00 79.62 -7.29 64 96.86 79.75 -6.10 80 99.26 79.82 -27.34 96 94.79 80.02 -5.08 128 98.14 81.15 -15.25
------------------------------------
BW: 77.9%, CPU: 80.4%, SD: -13.6%
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 3 lines · On Tue, Oct 26, 2010 at 02:38:53PM +0530, Krishna Kumar2 wrote:
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?
Signature · 1 line
MST
Quoted 9 lines
"Michael S. Tsirkin" <[email protected]>
Quoted 3 lines · On Tue, Oct 26, 2010 at 02:38:53PM +0530, Krishna Kumar2 wrote:
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?
Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding. I started binding vhost thread after Avi suggested it in response to my v1 patch (he suggested some more that I haven't done), and have been doing only this tuning ever since. This is part of his mail for the tuning:
Quoted 5 lines
vhost:
thread #0: CPU0
thread #1: CPU1
thread #2: CPU2
thread #3: CPU3
I simply bound each thread to CPU0-3 instead.
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 7 lines · On Tue, Oct 26, 2010 at 03:31:39PM +0530, Krishna Kumar2 wrote:
Quoted 7 lines
"Michael S. Tsirkin" <[email protected]>
Quoted 3 lines · On Tue, Oct 26, 2010 at 02:38:53PM +0530, Krishna Kumar2 wrote:
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
Quoted 5 lines
Quoted 2 lines
Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.
If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.
Quoted 17 lines
I started binding vhost thread
after Avi suggested it in response to my v1 patch (he suggested some more that I haven't done), and have been doing only this tuning ever since. This is part of his mail for the tuning:Quoted 5 lines
vhost:
thread #0: CPU0
thread #1: CPU1
thread #2: CPU2
thread #3: CPU3I simply bound each thread to CPU0-3 instead.
Thanks,
- KK
"Michael S. Tsirkin" <[email protected]> wrote on 10/26/2010 04:39:13 PM:
(merging two posts into one)
Quoted 4 lines
I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
For external host, there is a contention among different queues (vhosts) when packets are processed in tun/bridge, unless I implement MQ TX for macvtap (tun/bridge?). So my testing shows a small improvement (1 to 1.5% average) in BW and a rise in SD (between 10-15%). For remote host, I think tun/macvtap needs MQ TX support?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.
If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.
I will get a test run with and without binding and post the results later today.
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 9 lines · On Thu, Oct 28, 2010 at 10:44:14AM +0530, Krishna Kumar2 wrote:
"Michael S. Tsirkin" <[email protected]> wrote on 10/26/2010 04:39:13 PM:
(merging two posts into one)
Quoted 4 lines
I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
Quoted 6 lines
For external host, there is a contention among different queues (vhosts) when packets are processed in tun/bridge, unless I implement MQ TX for macvtap (tun/bridge?). So my testing shows a small improvement (1 to 1.5% average) in BW and a rise in SD (between 10-15%). For remote host, I think tun/macvtap needs MQ TX support?
Confused. I thought this *is* with a multiqueue tun/macvtap? bridge does not do any queueing AFAIK ...
I think we need to fix the contention. With migration what was guest to host a minute ago might become guest to external now ...
Quoted 19 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.
If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.
I will get a test run with and without binding and post the results later today.
Thanks,
- KK
Quoted 14 lines
"Michael S. Tsirkin" <[email protected]>
Quoted 11 lines
Quoted 4 lines
I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
For external host, there is a contention among different queues (vhosts) when packets are processed in tun/bridge, unless I implement MQ TX for macvtap (tun/bridge?). So my testing shows a small improvement (1 to 1.5% average) in BW and a rise in SD (between 10-15%). For remote host, I think tun/macvtap needs MQ TX support?
Quoted 4 lines
Confused. I thought this *is* with a multiqueue tun/macvtap? bridge does not do any queueing AFAIK ...
I think we need to fix the contention. With migration what was guest to host a minute ago might become guest to external now ...
Macvtap RX is MQ but not TX. I don't think MQ TX support is required for macvtap, though. Is it enough for existing macvtap sendmsg to work, since it calls dev_queue_xmit which selects the txq for the outgoing device?
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 19 lines · On Thu, Oct 28, 2010 at 11:42:05AM +0530, Krishna Kumar2 wrote:
Quoted 14 lines
"Michael S. Tsirkin" <[email protected]>
Quoted 11 lines
Quoted 4 lines
I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
For external host, there is a contention among different queues (vhosts) when packets are processed in tun/bridge, unless I implement MQ TX for macvtap (tun/bridge?). So my testing shows a small improvement (1 to 1.5% average) in BW and a rise in SD (between 10-15%). For remote host, I think tun/macvtap needs MQ TX support?
Quoted 4 lines
Confused. I thought this *is* with a multiqueue tun/macvtap? bridge does not do any queueing AFAIK ...
I think we need to fix the contention. With migration what was guest to host a minute ago might become guest to external now ...
Quoted 8 lines
Macvtap RX is MQ but not TX. I don't think MQ TX support is required for macvtap, though. Is it enough for existing macvtap sendmsg to work, since it calls
dev_queue_xmitwhich selects the txq for the outgoing device?Thanks,
- KK
I think there would be an issue with using a single poll notifier and contention on send buffer atomic variable.
Is tun different than macvtap? We need to support both long term ...
Signature · 1 line
MST
A message that is not in the archive (sent off-list or not yet received): OFC29C4491.59069AD1-ON652577CA.00170F0D-652577CA.001C76C8@LocalDomain
Quoted 9 lines
Krishna Kumar2/India/IBM wrote on 10/28/2010 10:44:14 AM:
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.
If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.
I will get a test run with and without binding and post the results later today.
Correction: The result with binding is is much better for SD/CPU compared to without-binding:
_____________________________________________________
numtxqs=8,vhosts=5, Bind vs No-bind
# BW% CPU% RCPU% SD% RSD%
_____________________________________________________
1 11.25 10.77 1.89 0 -6.06 2 18.66 7.20 7.20 -14.28 -7.40 4 4.24 -1.27 1.56 -2.70 -.98 8 14.91 -3.79 5.46 -12.19 -3.76 16 12.32 -8.67 4.63 -35.97 -26.66 24 11.68 -7.83 5.10 -40.73 -32.37 32 13.09 -10.51 6.57 -51.52 -42.28 40 11.04 -4.12 11.23 -50.69 -42.81 48 8.61 -10.30 6.04 -62.38 -55.54 64 7.55 -6.05 6.41 -61.20 -56.04 80 8.74 -11.45 6.29 -72.65 -67.17 96 9.84 -6.01 9.87 -69.89 -64.78 128 5.57 -6.23 8.99 -75.03 -70.97
_____________________________________________________
BW: 10.4%, CPU/RCPU: -7.4%,7.7%, SD: -70.5%,-65.7%
Notes:
1. All my test results earlier was binding vhost to cpus 0-3 for both org and new kernel.
2. I am not using MST's use_mq patch, only mainline kernel. However, I reported earlier that I got better results with that patch. The result for MQ vs MQ+use_mm patch (from my earlier mail):
BW: 0 CPU/RCPU: -4.2,-6.1 SD/RSD: -13.1,-15.6
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 12 lines · On Thu, Oct 28, 2010 at 12:48:57PM +0530, Krishna Kumar2 wrote:
Quoted 12 lines
Krishna Kumar2/India/IBM wrote on 10/28/2010 10:44:14 AM:
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.
I will get a test run with and without binding and post the results later today.
Correction: The result with binding is is much better for SD/CPU compared to without-binding:
Can you pls ty finding out why that is? Is some thread bouncing between CPUs? Does a wrong numa node get picked up?
In practice users are very unlikely to pin threads to CPUs.
Signature · 1 line
MST
Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · On Thu, Oct 28, 2010 at 12:48:57PM +0530, Krishna Kumar2 wrote:
Quoted 1 line
Krishna Kumar2/India/IBM wrote on 10/28/2010 10:44:14 AM:
In practice users are very unlikely to pin threads to CPUs.
I may be misunderstanding what you're referring to. It caught my attention since I'm working on a configuration to do what you say is unlikely, so I'll chime in for what it's worth.
An option in Vyatta allows assigning CPU affinity to network adapters, since apparently seperate L2 caches can have a significant impact on throughput.
Although much of their focus seems to be on commercial virtualization platforms, I do see quite a few forum posts with regard to KVM. Mabye this still qualifies as an edge case, but as for virtualized routing theirs seems to offer the most functionality.
http://www.vyatta.org/forum/viewtopic.php?t=2697
-cb
Michael S. Tsirkin
maintainerQuoted 12 lines · On Thu, Oct 28, 2010 at 12:48:57PM +0530, Krishna Kumar2 wrote:
Quoted 12 lines
Krishna Kumar2/India/IBM wrote on 10/28/2010 10:44:14 AM:
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):
Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.
I will get a test run with and without binding and post the results later today.
Correction: The result with binding is is much better for SD/CPU compared to without-binding:
Something that was suggested to me off-list is trying to set smp affinity for NIC: in host to guest case probably virtio-net, for external to guest the host NIC as well.
Signature · 1 line
MST
Michael S. Tsirkin
maintainerQuoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line
Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:
Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.
Some more test results:
_____________________________________________________Host->Guest BW (numtxqs=2)# BW% CPU% RCPU% SD% RSD%
_____________________________________________________
I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
Signature · 1 line
MST
"Michael S. Tsirkin" <[email protected]> wrote on 10/26/2010 02:27:09 PM:
Quoted 14 lines
Re: [v3 RFC PATCH 0/4] Implement multiqueue virtio-net
Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line
Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:
Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.
Some more test results:
_____________________________________________________Host->Guest BW (numtxqs=2)# BW% CPU% RCPU% SD% RSD%
_____________________________________________________
Quoted 4 lines
I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
I had to make a few changes to qemu (and a minor change in macvtap driver) to get multiple TXQ support using macvtap working. The NIC is a ixgbe card.
__________________________________________________________________________
Org vs New (I/O: 512 bytes, #numtxqs=2, #vhosts=3) # BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%) __________________________________________________________________________ 1 14367 13142 (-8.5) 56 62 (10.7) 8 8 (0) 2 3652 3855 (5.5) 37 35 (-5.4) 7 6 (-14.2) 4 12529 12059 (-3.7) 65 77 (18.4) 35 35 (0) 8 13912 14668 (5.4) 288 332 (15.2) 175 184 (5.1) 16 13433 14455 (7.6) 1218 1321 (8.4) 920 943 (2.5) 24 12750 13477 (5.7) 2876 2985 (3.7) 2514 2348 (-6.6) 32 11729 12632 (7.6) 5299 5332 (.6) 4934 4497 (-8.8) 40 11061 11923 (7.7) 8482 8364 (-1.3) 8374 7495 (-10.4) 48 10624 11267 (6.0) 12329 12258 (-.5) 12762 11538 (-9.5) 64 10524 10596 (.6) 21689 22859 (5.3) 23626 22403 (-5.1) 80 9856 10284 (4.3) 35769 36313 (1.5) 39932 36419 (-8.7) 96 9691 10075 (3.9) 52357 52259 (-.1) 58676 53463 (-8.8) 128 9351 9794 (4.7) 114707 94275 (-17.8) 114050 97337 (-14.6)
__________________________________________________________________________
Avg: BW: (3.3) SD: (-7.3) RSD: (-11.0)
__________________________________________________________________________
Org vs New (I/O: 1K, #numtxqs=8, #vhosts=5) # BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%) __________________________________________________________________________ 1 16509 15985 (-3.1) 45 47 (4.4) 7 7 (0) 2 6963 4499 (-35.3) 17 51 (200.0) 7 7 (0) 4 12932 11080 (-14.3) 49 74 (51.0) 35 35 (0) 8 13878 14095 (1.5) 223 292 (30.9) 175 181 (3.4) 16 13440 13698 (1.9) 980 1131 (15.4) 926 942 (1.7) 24 12680 12927 (1.9) 2387 2463 (3.1) 2526 2342 (-7.2) 32 11714 12261 (4.6) 4506 4486 (-.4) 4941 4463 (-9.6) 40 11059 11651 (5.3) 7244 7081 (-2.2) 8349 7437 (-10.9) 48 10580 11095 (4.8) 10811 10500 (-2.8) 12809 11403 (-10.9) 64 10569 10566 (0) 19194 19270 (.3) 23648 21717 (-8.1) 80 9827 10753 (9.4) 31668 29425 (-7.0) 39991 33824 (-15.4) 96 10043 10150 (1.0) 45352 44227 (-2.4) 57766 51131 (-11.4) 128 9360 9979 (6.6) 92058 79198 (-13.9) 114381 92873 (-18.8)
__________________________________________________________________________
Avg: BW: (-.5) SD: (-7.5) RSD: (-14.7)
Is there anything else you would like me to test/change, or shall I submit the next version (with the above macvtap changes)?
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 65 lines · On Tue, Nov 09, 2010 at 10:08:21AM +0530, Krishna Kumar2 wrote:
"Michael S. Tsirkin" <[email protected]> wrote on 10/26/2010 02:27:09 PM:
Quoted 14 lines
Re: [v3 RFC PATCH 0/4] Implement multiqueue virtio-net
Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line
Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:
Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.
Some more test results:
_____________________________________________________Host->Guest BW (numtxqs=2)# BW% CPU% RCPU% SD% RSD%
_____________________________________________________Quoted 4 lines
I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
I had to make a few changes to qemu (and a minor change in macvtap driver) to get multiple TXQ support using macvtap working. The NIC is a ixgbe card.
__________________________________________________________________________
Org vs New (I/O: 512 bytes, #numtxqs=2, #vhosts=3) # BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%) __________________________________________________________________________ 1 14367 13142 (-8.5) 56 62 (10.7) 8 8 (0) 2 3652 3855 (5.5) 37 35 (-5.4) 7 6 (-14.2) 4 12529 12059 (-3.7) 65 77 (18.4) 35 35 (0) 8 13912 14668 (5.4) 288 332 (15.2) 175 184 (5.1) 16 13433 14455 (7.6) 1218 1321 (8.4) 920 943 (2.5) 24 12750 13477 (5.7) 2876 2985 (3.7) 2514 2348 (-6.6) 32 11729 12632 (7.6) 5299 5332 (.6) 4934 4497 (-8.8) 40 11061 11923 (7.7) 8482 8364 (-1.3) 8374 7495 (-10.4) 48 10624 11267 (6.0) 12329 12258 (-.5) 12762 11538 (-9.5) 64 10524 10596 (.6) 21689 22859 (5.3) 23626 22403 (-5.1) 80 9856 10284 (4.3) 35769 36313 (1.5) 39932 36419 (-8.7) 96 9691 10075 (3.9) 52357 52259 (-.1) 58676 53463 (-8.8) 128 9351 9794 (4.7) 114707 94275 (-17.8) 114050 97337 (-14.6)__________________________________________________________________________
Avg: BW: (3.3) SD: (-7.3) RSD: (-11.0)__________________________________________________________________________
Org vs New (I/O: 1K, #numtxqs=8, #vhosts=5) # BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%) __________________________________________________________________________ 1 16509 15985 (-3.1) 45 47 (4.4) 7 7 (0) 2 6963 4499 (-35.3) 17 51 (200.0) 7 7 (0) 4 12932 11080 (-14.3) 49 74 (51.0) 35 35 (0) 8 13878 14095 (1.5) 223 292 (30.9) 175 181 (3.4) 16 13440 13698 (1.9) 980 1131 (15.4) 926 942 (1.7) 24 12680 12927 (1.9) 2387 2463 (3.1) 2526 2342 (-7.2) 32 11714 12261 (4.6) 4506 4486 (-.4) 4941 4463 (-9.6) 40 11059 11651 (5.3) 7244 7081 (-2.2) 8349 7437 (-10.9) 48 10580 11095 (4.8) 10811 10500 (-2.8) 12809 11403 (-10.9) 64 10569 10566 (0) 19194 19270 (.3) 23648 21717 (-8.1) 80 9827 10753 (9.4) 31668 29425 (-7.0) 39991 33824 (-15.4) 96 10043 10150 (1.0) 45352 44227 (-2.4) 57766 51131 (-11.4) 128 9360 9979 (6.6) 92058 79198 (-13.9) 114381 92873 (-18.8)__________________________________________________________________________
Avg: BW: (-.5) SD: (-7.5) RSD: (-14.7)
Quoted 6 lines
Is there anything else you would like me to test/change, or shall I submit the next version (with the above macvtap changes)?
Thanks,
- KK
Something strange here, right?
1. You are consistently getting >10G/s here, and even with a single stream?
2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.
Is your card MQ?
Signature · 1 line
MST
"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 06:52:39 PM:
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Re: [v3 RFC PATCH 0/4] Implement multiqueue virtio-net
Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line
Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:
Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.
Some more test results:
_____________________________________________________Host->Guest BW (numtxqs=2)# BW% CPU% RCPU% SD% RSD%
_____________________________________________________I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
Quoted 5 lines
Quoted 5 lines
I had to make a few changes to qemu (and a minor change in macvtap driver) to get multiple TXQ support using macvtap working. The NIC is a ixgbe card.
__________________________________________________________________________
Quoted 3 lines
Quoted 3 lines
Org vs New (I/O: 512 bytes, #numtxqs=2, #vhosts=3)# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)
__________________________________________________________________________
Quoted 8 lines
Quoted 8 lines
1 14367 13142 (-8.5) 56 62 (10.7) 8 8 (0) 2 3652 3855 (5.5) 37 35 (-5.4) 7 6 (-14.2) 4 12529 12059 (-3.7) 65 77 (18.4) 35 35 (0) 8 13912 14668 (5.4) 288 332 (15.2) 175 184 (5.1) 16 13433 14455 (7.6) 1218 1321 (8.4) 920 943 (2.5) 24 12750 13477 (5.7) 2876 2985 (3.7) 2514 2348 (-6.6) 32 11729 12632 (7.6) 5299 5332 (.6) 4934 4497 (-8.8) 40 11061 11923 (7.7) 8482 8364 (-1.3) 8374 7495
(-10.4)
Quoted 1 line
Quoted 1 line
48 10624 11267 (6.0) 12329 12258 (-.5) 12762 11538
(-9.5)
Quoted 1 line
Quoted 1 line
64 10524 10596 (.6) 21689 22859 (5.3) 23626 22403
(-5.1)
Quoted 1 line
Quoted 1 line
80 9856 10284 (4.3) 35769 36313 (1.5) 39932 36419
(-8.7)
Quoted 1 line
Quoted 1 line
96 9691 10075 (3.9) 52357 52259 (-.1) 58676 53463
(-8.8)
Quoted 1 line
Quoted 1 line
128 9351 9794 (4.7) 114707 94275 (-17.8) 114050 97337
(-14.6)
Quoted 1 line
Quoted 1 line
__________________________________________________________________________
Quoted 3 lines
Quoted 3 lines
Avg: BW: (3.3) SD: (-7.3) RSD: (-11.0)
__________________________________________________________________________
Quoted 3 lines
Quoted 3 lines
Org vs New (I/O: 1K, #numtxqs=8, #vhosts=5)# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)
__________________________________________________________________________
Quoted 9 lines
Quoted 9 lines
1 16509 15985 (-3.1) 45 47 (4.4) 7 7 (0) 2 6963 4499 (-35.3) 17 51 (200.0) 7 7 (0) 4 12932 11080 (-14.3) 49 74 (51.0) 35 35 (0) 8 13878 14095 (1.5) 223 292 (30.9) 175 181 (3.4) 16 13440 13698 (1.9) 980 1131 (15.4) 926 942 (1.7) 24 12680 12927 (1.9) 2387 2463 (3.1) 2526 2342 (-7.2) 32 11714 12261 (4.6) 4506 4486 (-.4) 4941 4463 (-9.6) 40 11059 11651 (5.3) 7244 7081 (-2.2) 8349 7437 (-10.9) 48 10580 11095 (4.8) 10811 10500 (-2.8) 12809 11403
(-10.9)
Quoted 2 lines
Quoted 2 lines
64 10569 10566 (0) 19194 19270 (.3) 23648 21717 (-8.1) 80 9827 10753 (9.4) 31668 29425 (-7.0) 39991 33824
(-15.4)
Quoted 1 line
Quoted 1 line
96 10043 10150 (1.0) 45352 44227 (-2.4) 57766 51131
(-11.4)
Quoted 1 line
Quoted 1 line
128 9360 9979 (6.6) 92058 79198 (-13.9) 114381 92873
(-18.8)
Quoted 1 line
Quoted 1 line
__________________________________________________________________________
Quoted 11 lines
Quoted 8 lines
Avg: BW: (-.5) SD: (-7.5) RSD: (-14.7)
Is there anything else you would like me to test/change, or shall I submit the next version (with the above macvtap changes)?
Thanks,
- KK
Something strange here, right?
1. You are consistently getting >10G/s here, and even with a single
stream?
Sorry, I should have mentioned this though I had stated in my earlier mails. Each test result has two iterations, each of 60 seconds, except when #netperfs is 1 for which I do 10 iteration (sum across 10 iterations). I started doing many more iterations for 1 netperf after finding the issue earlier with single stream. So the BW is only 4.5-7 Gbps.
Quoted 3 lines
2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.
(doubling happens consistently for guest -> host, but never for remote host) I tried 512/txqs=2 and 1024/txqs=8 to get a varied testing scenario. In first case, there is a slight improvement in BW and good reduction in SD. In the second case, only SD improves (though BW drops for 2 stream for some reason). In both cases, BW and SD improves as the number of sessions increase.
Quoted 1 line
Is your card MQ?
Yes, the card is MQ. ixgbe 10g card.
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 88 lines · On Tue, Nov 09, 2010 at 08:58:44PM +0530, Krishna Kumar2 wrote:
"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 06:52:39 PM:
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Re: [v3 RFC PATCH 0/4] Implement multiqueue virtio-net
Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line
Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:
Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.
Some more test results:
_____________________________________________________Host->Guest BW (numtxqs=2)# BW% CPU% RCPU% SD% RSD%
_____________________________________________________I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.
Quoted 5 lines
Quoted 5 lines
I had to make a few changes to qemu (and a minor change in macvtap driver) to get multiple TXQ support using macvtap working. The NIC is a ixgbe card.
__________________________________________________________________________
Quoted 3 lines
Quoted 3 lines
Org vs New (I/O: 512 bytes, #numtxqs=2, #vhosts=3)# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)
__________________________________________________________________________
Quoted 8 lines
Quoted 8 lines
1 14367 13142 (-8.5) 56 62 (10.7) 8 8 (0) 2 3652 3855 (5.5) 37 35 (-5.4) 7 6 (-14.2) 4 12529 12059 (-3.7) 65 77 (18.4) 35 35 (0) 8 13912 14668 (5.4) 288 332 (15.2) 175 184 (5.1) 16 13433 14455 (7.6) 1218 1321 (8.4) 920 943 (2.5) 24 12750 13477 (5.7) 2876 2985 (3.7) 2514 2348 (-6.6) 32 11729 12632 (7.6) 5299 5332 (.6) 4934 4497 (-8.8) 40 11061 11923 (7.7) 8482 8364 (-1.3) 8374 7495(-10.4)
Quoted 1 line
Quoted 1 line
48 10624 11267 (6.0) 12329 12258 (-.5) 12762 11538
(-9.5)
Quoted 1 line
Quoted 1 line
64 10524 10596 (.6) 21689 22859 (5.3) 23626 22403
(-5.1)
Quoted 1 line
Quoted 1 line
80 9856 10284 (4.3) 35769 36313 (1.5) 39932 36419
(-8.7)
Quoted 1 line
Quoted 1 line
96 9691 10075 (3.9) 52357 52259 (-.1) 58676 53463
(-8.8)
Quoted 1 line
Quoted 1 line
128 9351 9794 (4.7) 114707 94275 (-17.8) 114050 97337
(-14.6)
Quoted 1 line
Quoted 1 line
__________________________________________________________________________
Quoted 3 lines
Quoted 3 lines
Avg: BW: (3.3) SD: (-7.3) RSD: (-11.0)
__________________________________________________________________________
Quoted 3 lines
Quoted 3 lines
Org vs New (I/O: 1K, #numtxqs=8, #vhosts=5)# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)
__________________________________________________________________________
Quoted 9 lines
Quoted 9 lines
1 16509 15985 (-3.1) 45 47 (4.4) 7 7 (0) 2 6963 4499 (-35.3) 17 51 (200.0) 7 7 (0) 4 12932 11080 (-14.3) 49 74 (51.0) 35 35 (0) 8 13878 14095 (1.5) 223 292 (30.9) 175 181 (3.4) 16 13440 13698 (1.9) 980 1131 (15.4) 926 942 (1.7) 24 12680 12927 (1.9) 2387 2463 (3.1) 2526 2342 (-7.2) 32 11714 12261 (4.6) 4506 4486 (-.4) 4941 4463 (-9.6) 40 11059 11651 (5.3) 7244 7081 (-2.2) 8349 7437 (-10.9) 48 10580 11095 (4.8) 10811 10500 (-2.8) 12809 11403(-10.9)
Quoted 2 lines
Quoted 2 lines
64 10569 10566 (0) 19194 19270 (.3) 23648 21717 (-8.1) 80 9827 10753 (9.4) 31668 29425 (-7.0) 39991 33824(-15.4)
Quoted 1 line
Quoted 1 line
96 10043 10150 (1.0) 45352 44227 (-2.4) 57766 51131
(-11.4)
Quoted 1 line
Quoted 1 line
128 9360 9979 (6.6) 92058 79198 (-13.9) 114381 92873
(-18.8)
Quoted 1 line
Quoted 1 line
__________________________________________________________________________
Quoted 9 lines
Quoted 8 lines
Avg: BW: (-.5) SD: (-7.5) RSD: (-14.7)
Is there anything else you would like me to test/change, or shall I submit the next version (with the above macvtap changes)?
Thanks,
- KK
Quoted 8 lines
Quoted 2 lines
Something strange here, right?
1. You are consistently getting >10G/s here, and even with a singlestream?
Sorry, I should have mentioned this though I had stated in my earlier mails. Each test result has two iterations, each of 60 seconds, except when #netperfs is 1 for which I do 10 iteration (sum across 10 iterations).
So need to divide the number by 10?
Quoted 8 lines
I started doing many more iterations
for 1 netperf after finding the issue earlier with single stream. So the BW is only 4.5-7 Gbps.Quoted 3 lines
2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.
Quoted 6 lines
(doubling happens consistently for guest -> host, but never for remote host) I tried 512/txqs=2 and 1024/txqs=8 to get a varied testing scenario. In first case, there is a slight improvement in BW and good reduction in SD. In the second case, only SD improves (though BW drops for 2 stream for some reason). In both cases, BW and SD improves as the number of sessions increase.
I guess this is another indication that something's wrong. We are quite far from line rate, the fact BW does not scale means there's some contention in the code.
Quoted 7 lines
Quoted 1 line
Is your card MQ?
Yes, the card is MQ. ixgbe 10g card.
Thanks,
- KK
"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 09:03:25 PM:
Quoted 10 lines
Quoted 8 lines
Quoted 2 lines
Something strange here, right?
1. You are consistently getting >10G/s here, and even with a singlestream?
Sorry, I should have mentioned this though I had stated in my earlier mails. Each test result has two iterations, each of 60 seconds, except when #netperfs is 1 for which I do 10 iteration (sum across 10 iterations).
So need to divide the number by 10?
Yes, that is what I get with 512/1K macvtap I/O size :)
Quoted 8 lines
Quoted 8 lines
I started doing many more iterations
for 1 netperf after finding the issue earlier with single stream. So the BW is only 4.5-7 Gbps.Quoted 3 lines
2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.
Quoted 8 lines
Quoted 6 lines
(doubling happens consistently for guest -> host, but never for remote host) I tried 512/txqs=2 and 1024/txqs=8 to get a varied testing scenario. In first case, there is a slight improvement in BW and good reduction in SD. In the second case, only SD improves (though BW drops for 2 stream for some reason). In both cases, BW and SD improves as the number of sessions increase.
I guess this is another indication that something's wrong.
The patch - both virtio-net and vhost-net, doesn't have any locking/mutex's/ or any synchronization method. Guest -> host performance improvement of upto 100% shows the patch is not doing anything wrong.
Quoted 2 lines
We are quite far from line rate, the fact BW does not scale means there's some contention in the code.
Attaining line speed with macvtap seems to be a generic issue and unrelated to my patch specifically. IMHO if there is nothing wrong in the code (review) and is accepted, it will benefit as others can also help to find what needs to be implemented in vhost/macvtap/qemu to get line speed for guest->remote-host.
PS: bare-metal performance for host->remote-host is also
2.7 Gbps and 2.8 Gbps for 512/1024 for the same card.
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 30 lines · On Tue, Nov 09, 2010 at 10:54:57PM +0530, Krishna Kumar2 wrote:
"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 09:03:25 PM:
Quoted 10 lines
Quoted 8 lines
Quoted 2 lines
Something strange here, right?
1. You are consistently getting >10G/s here, and even with a singlestream?
Sorry, I should have mentioned this though I had stated in my earlier mails. Each test result has two iterations, each of 60 seconds, except when #netperfs is 1 for which I do 10 iteration (sum across 10 iterations).
So need to divide the number by 10?
Yes, that is what I get with 512/1K macvtap I/O size :)
Quoted 15 lines
Quoted 14 lines
I started doing many more iterations
for 1 netperf after finding the issue earlier with single stream. So the BW is only 4.5-7 Gbps.Quoted 3 lines
2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.
(doubling happens consistently for guest -> host, but never for remote host) I tried 512/txqs=2 and 1024/txqs=8 to get a varied testing scenario. In first case, there is a slight improvement in BW and good reduction in SD. In the second case, only SD improves (though BW drops for 2 stream for some reason). In both cases, BW and SD improves as the number of sessions increase.
Quoted 6 lines
Quoted 1 line
I guess this is another indication that something's wrong.
The patch - both virtio-net and vhost-net, doesn't have any locking/mutex's/ or any synchronization method. Guest -> host performance improvement of upto 100% shows the patch is not doing anything wrong.
My concern is this: we don't seem to do anything in tap or macvtap to help packets from separate virtio queues get to separate queues in the hardware device and to avoid reordering when we do this.
- skb_tx_hash calculation will get different results
- hash math that e.g. tcp does will run on guest and seems to be discarded
etc
Maybe it's as simple as some tap/macvtap ioctls to set up the queue number in skbs. Or maybe we need to pass the skb hash from guest to host. It's this last option that should make us especially cautios as it'll affect guest/host interface.
Also see d5a9e24afb4ab38110ebb777588ea0bd0eacbd0a: if we have hardware which records an RX queue, it appears important to pass that info to guest and to use that in selecting the TX queue. Of course we won't see this in netperf runs but this needs to be given thought too - supporting this seems to suggest either sticking the hash in the virtio net header for both tx and rx, or using multiplease RX queues.
Quoted 8 lines
Quoted 2 lines
We are quite far from line rate, the fact BW does not scale means there's some contention in the code.
Attaining line speed with macvtap seems to be a generic issue and unrelated to my patch specifically. IMHO if there is nothing wrong in the code (review) and is accepted, it will benefit as others can also help to find what needs to be implemented in vhost/macvtap/qemu to get line speed for guest->remote-host.
No problem, I will queue these patches in some branch
to help enable cooperation, as well as help you
iterate with incremental patches instead of resending it all each time.
Quoted 4 lines
PS: bare-metal performance for host->remote-host is also
2.7 Gbps and 2.8 Gbps for 512/1024 for the same card.Thanks,
You mean native linux BW does not scale for your host with
# of connections either? I guess this just means need another setup for testing?
Quoted 1 line
- KK
A message that is not in the archive (sent off-list or not yet received): OF24E08752.2087FFA4-ON652577D6.00532DF1-652577D6.0054B291@LocalDomain
MQ performance on other cards (cxgb3)
I had sent this mail to Michael last week - he agrees that I should share this information on the list:
On latest net-next-2.6, virtio-net (guest->host) results are:
______________________________________________________________
SQ vs MQ (#txqs=8) # BW1 BW2 (%) CPU1 CPU2 (%) RCPU1 RCPU2 (%) _______________________________________________________________ 1 105774 112256 (6.1) 257 255 (-.7) 532 549 (3.1) 2 20842 30674 (47.1) 107 150 (40.1) 208 279 (34.1) 4 22500 31953 (42.0) 241 409 (69.7) 467 619 (32.5) 8 22416 44507 (98.5) 477 1039 (117.8) 960 1459 (51.9) 16 22605 45372 (100.7) 905 2060 (127.6) 1895 2962 (56.3) 24 23192 44201 (90.5) 1360 3028 (122.6) 2833 4437 (56.6) 32 23158 43394 (87.3) 1811 3957 (118.4) 3770 5936 (57.4) 40 23322 42550 (82.4) 2276 4986 (119.0) 4711 7417 (57.4) 48 23564 41931 (77.9) 2757 5966 (116.3) 5653 8896 (57.3) 64 23949 41092 (71.5) 3788 7898 (108.5) 7609 11826 (55.4) 80 23256 41343 (77.7) 4597 9887 (115.0) 9503 14801 (55.7) 96 23310 40645 (74.3) 5588 11758 (110.4) 11381 17761 (56.0) 128 24095 41082 (70.5) 7587 15574 (105.2) 15029 23716 (57.8)
______________________________________________________________
Avg: BW: (58.3) CPU: (110.8) RCPU: (55.9)
It's true that average CPU% on guest is almost double that of the BW improvement. But I don't think this is due to the patch (driver does no synchronization, etc). To compare MQ vs SQ on a 10G card, I ran the same test from host to remote host across cxgb3. The results are somewhat similar:
(I changed cxgb_open on the client system to:
netif_set_real_num_tx_queues(dev, 1); err = netif_set_real_num_rx_queues(dev, 1);
to simulate single queue (SQ))
_____________________________________
cxgb3 SQ vs cxgb3 MQ
# BW1 BW2 (%) CPU1 CPU2 (%)
_____________________________________
1 8301 8315 (.1) 5 4.66 (-6.6) 2 9395 9380 (-.1) 16 16 (0) 4 9411 9414 (0) 33 26 (-21.2) 8 9411 9398 (-.1) 60 62 (3.3) 16 9412 9413 (0) 116 117 (.8) 24 9442 9963 (5.5) 179 198 (10.6) 32 10031 10025 (0) 230 249 (8.2) 40 9953 10024 (.7) 300 312 (4.0) 48 10002 10015 (.1) 351 376 (7.1) 64 10022 10024 (0) 494 515 (4.2) 80 8894 10011 (12.5) 537 630 (17.3) 96 8465 9907 (17.0) 612 749 (22.3) 128 7541 9617 (27.5) 760 989 (30.1)
_____________________________________
Avg: BW: (3.8) CPU: (14.8)
(Each case runs runs once for 60 secs)
The BW increased modestly but CPU increased much more. I assume the change I made above to convert the driver from MQ to SQ is not incorrect.
Thanks,
- KK
Quoted 3 lines · Krishna Kumar
Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.
Hi Krishna,
I have a few questions about the results below:
1. Are the (%) comparisons between non-mq and mq virtio?
2. Was UDP or TCP used?
3. What was the transmit size (-m option to netperf)?
Also, I'm interested to know what the status of these patches is. Are you planing a fresh series?
Quoted 128 lines
Changes from rev2:
------------------
1. Define (invirtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:struct send_queue { struct virtqueue *svq; struct scatterlist tx_sg[MAX_SKB_FRAGS + 2]; } ____cacheline_aligned_in_smp;and to statically allocate '
VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:vhost=on -> #txqueues = 1 vhost=on,mq=on -> #txqueues = 4 vhost=on,mq=on,numtxqs=2 -> #txqueues = 2 vhost=on,mq=on,numtxqs=8 -> #txqueues = 8Performance (guest -> local host): -----------------------------------System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 4.46 -1.96 .19 -12.50 -6.06 2 4.93 -1.16 2.10 0 -2.38 4 46.17 64.77 33.72 19.51 -2.48 8 47.89 70.00 36.23 41.46 13.35 16 48.97 80.44 40.67 21.11 -5.46 24 49.03 78.78 41.22 20.51 -4.78 32 51.11 77.15 42.42 15.81 -6.87 40 51.60 71.65 42.43 9.75 -8.94 48 50.10 69.55 42.85 11.80 -5.81 64 46.24 68.42 42.67 14.18 -3.28 80 46.37 63.13 41.62 7.43 -6.73 96 46.40 63.31 42.20 9.36 -4.78 128 50.43 62.79 42.16 13.11 -1.23________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -.76 -1.56 2.33 0 3.03 2 17.41 11.11 11.41 0 -4.76 4 42.12 55.11 30.20 19.51 .62 8 54.69 80.00 39.22 24.39 -3.88 16 54.77 81.62 40.89 20.34 -6.58 24 54.66 79.68 41.57 15.49 -8.99 32 54.92 76.82 41.79 17.59 -5.70 40 51.79 68.56 40.53 15.31 -3.87 48 51.72 66.40 40.84 9.72 -7.13 64 51.11 63.94 41.10 5.93 -8.82 80 46.51 59.50 39.80 9.33 -4.18 96 47.72 57.75 39.84 4.20 -7.62 128 54.35 58.95 40.66 3.24 -8.63________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -1.43 -3.52 1.55 0 3.03 2 33.09 21.63 20.12 -10.00 -9.52 4 67.17 94.60 44.28 19.51 -11.80 8 75.72 108.14 49.15 25.00 -10.71 16 80.34 101.77 52.94 25.93 -4.49 24 70.84 93.12 43.62 27.63 -5.03 32 69.01 94.16 47.33 29.68 -1.51 40 58.56 63.47 25.91 -3.92 -25.85 48 61.16 74.70 34.88 .89 -22.08 64 54.37 69.09 26.80 -6.68 -30.04 80 36.22 22.73 -2.97 -8.25 -27.23 96 41.51 50.59 13.24 9.84 -16.77 128 48.98 38.15 6.41 -.33 -22.80________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________1 7.62 -38.03 -26.26 -50.00 -33.33 2 28.95 20.46 21.62 0 -7.14 4 84.05 60.79 45.74 -2.43 -12.42 8 86.43 79.57 50.32 15.85 -3.10 16 88.63 99.48 58.17 9.47 -13.10 24 74.65 80.87 41.99 -1.81 -22.89 32 63.86 59.21 23.58 -18.13 -36.37 40 64.79 60.53 22.23 -15.77 -35.84 48 49.68 26.93 .51 -36.40 -49.61 64 54.69 36.50 5.41 -26.59 -43.23 80 45.06 12.72 -13.25 -37.79 -52.08 96 40.21 -3.16 -24.53 -39.92 -52.97 128 36.33 -33.19 -43.66 -5.68 -20.49________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%Signed-off-by Krishna Kumar <[email protected]>---
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Simon Horman <[email protected]> wrote on 02/22/2011 01:17:09 PM:
Hi Simon,
Quoted 3 lines
I have a few questions about the results below:
1. Are the (%) comparisons between non-mq and mq virtio?
Yes - mainline kernel with transmit-only MQ patch.
Quoted 1 line
2. Was UDP or TCP used?
TCP. I had done some initial testing on UDP, but don't have the results now as it is really old. But I will be running it again.
Quoted 1 line
3. What was the transmit size (-m option to netperf)?
I didn't use the -m option, so it defaults to 16K. The script does:
netperf -t TCP_STREAM -c -C -l 60 -H $SERVER
Quoted 2 lines
Also, I'm interested to know what the status of these patches is. Are you planing a fresh series?
Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?
The TX-only patch helped the guest TX path but didn't help host->guest much (as tested using TCP_MAERTS from the guest). But with the TX+RX patch, both directions are getting improvements. Remote testing is still to be done.
Thanks,
- KK
Quoted 118 lines
Quoted 118 lines
Changes from rev2:
------------------
1. Define (invirtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:struct send_queue { struct virtqueue *svq; struct scatterlist tx_sg[MAX_SKB_FRAGS + 2]; } ____cacheline_aligned_in_smp;and to statically allocate '
VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:vhost=on -> #txqueues = 1 vhost=on,mq=on -> #txqueues = 4 vhost=on,mq=on,numtxqs=2 -> #txqueues = 2 vhost=on,mq=on,numtxqs=8 -> #txqueues = 8Performance (guest -> local host): -----------------------------------System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 4.46 -1.96 .19 -12.50 -6.06 2 4.93 -1.16 2.10 0 -2.38 4 46.17 64.77 33.72 19.51 -2.48 8 47.89 70.00 36.23 41.46 13.35 16 48.97 80.44 40.67 21.11 -5.46 24 49.03 78.78 41.22 20.51 -4.78 32 51.11 77.15 42.42 15.81 -6.87 40 51.60 71.65 42.43 9.75 -8.94 48 50.10 69.55 42.85 11.80 -5.81 64 46.24 68.42 42.67 14.18 -3.28 80 46.37 63.13 41.62 7.43 -6.73 96 46.40 63.31 42.20 9.36 -4.78 128 50.43 62.79 42.16 13.11 -1.23________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -.76 -1.56 2.33 0 3.03 2 17.41 11.11 11.41 0 -4.76 4 42.12 55.11 30.20 19.51 .62 8 54.69 80.00 39.22 24.39 -3.88 16 54.77 81.62 40.89 20.34 -6.58 24 54.66 79.68 41.57 15.49 -8.99 32 54.92 76.82 41.79 17.59 -5.70 40 51.79 68.56 40.53 15.31 -3.87 48 51.72 66.40 40.84 9.72 -7.13 64 51.11 63.94 41.10 5.93 -8.82 80 46.51 59.50 39.80 9.33 -4.18 96 47.72 57.75 39.84 4.20 -7.62 128 54.35 58.95 40.66 3.24 -8.63________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -1.43 -3.52 1.55 0 3.03 2 33.09 21.63 20.12 -10.00 -9.52 4 67.17 94.60 44.28 19.51 -11.80 8 75.72 108.14 49.15 25.00 -10.71 16 80.34 101.77 52.94 25.93 -4.49 24 70.84 93.12 43.62 27.63 -5.03 32 69.01 94.16 47.33 29.68 -1.51 40 58.56 63.47 25.91 -3.92 -25.85 48 61.16 74.70 34.88 .89 -22.08 64 54.37 69.09 26.80 -6.68 -30.04 80 36.22 22.73 -2.97 -8.25 -27.23 96 41.51 50.59 13.24 9.84 -16.77 128 48.98 38.15 6.41 -.33 -22.80________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________1 7.62 -38.03 -26.26 -50.00 -33.33 2 28.95 20.46 21.62 0 -7.14 4 84.05 60.79 45.74 -2.43 -12.42 8 86.43 79.57 50.32 15.85 -3.10 16 88.63 99.48 58.17 9.47 -13.10 24 74.65 80.87 41.99 -1.81 -22.89 32 63.86 59.21 23.58 -18.13 -36.37 40 64.79 60.53 22.23 -15.77 -35.84 48 49.68 26.93 .51 -36.40 -49.61 64 54.69 36.50 5.41 -26.59 -43.23 80 45.06 12.72 -13.25 -37.79 -52.08 96 40.21 -3.16 -24.53 -39.92 -52.97 128 36.33 -33.19 -43.66 -5.68 -20.49________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%
Michael S. Tsirkin
maintainerQuoted 24 lines · On Wed, Feb 23, 2011 at 10:52:09AM +0530, Krishna Kumar2 wrote:
Simon Horman <[email protected]> wrote on 02/22/2011 01:17:09 PM:
Hi Simon,
Quoted 3 lines
I have a few questions about the results below:
1. Are the (%) comparisons between non-mq and mq virtio?
Yes - mainline kernel with transmit-only MQ patch.
Quoted 1 line
2. Was UDP or TCP used?
TCP. I had done some initial testing on UDP, but don't have the results now as it is really old. But I will be running it again.
Quoted 1 line
3. What was the transmit size (-m option to netperf)?
I didn't use the -m option, so it defaults to 16K. The script does:
netperf -t
TCP_STREAM-c -C -l 60 -H $SERVER
Quoted 7 lines
Quoted 2 lines
Also, I'm interested to know what the status of these patches is. Are you planing a fresh series?
Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?
Yes, please do.
Quoted 4 lines
The TX-only patch helped the guest TX path but didn't help
host->guestmuch (as tested usingTCP_MAERTSfrom the guest). But with the TX+RX patch, both directions are getting improvements.
Also, my hope is that with appropriate queue mapping, we might be able to do away with heuristics to detect single stream load that TX only code needs.
Quoted 1 line
Remote testing is still to be done.
Others might be able to help here once you post the patch.
Quoted 122 lines
Thanks,
- KK
Quoted 118 lines
Quoted 118 lines
Changes from rev2:
------------------
1. Define (invirtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:struct send_queue { struct virtqueue *svq; struct scatterlist tx_sg[MAX_SKB_FRAGS + 2]; } ____cacheline_aligned_in_smp;and to statically allocate '
VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:vhost=on -> #txqueues = 1 vhost=on,mq=on -> #txqueues = 4 vhost=on,mq=on,numtxqs=2 -> #txqueues = 2 vhost=on,mq=on,numtxqs=8 -> #txqueues = 8Performance (guest -> local host): -----------------------------------System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 4.46 -1.96 .19 -12.50 -6.06 2 4.93 -1.16 2.10 0 -2.38 4 46.17 64.77 33.72 19.51 -2.48 8 47.89 70.00 36.23 41.46 13.35 16 48.97 80.44 40.67 21.11 -5.46 24 49.03 78.78 41.22 20.51 -4.78 32 51.11 77.15 42.42 15.81 -6.87 40 51.60 71.65 42.43 9.75 -8.94 48 50.10 69.55 42.85 11.80 -5.81 64 46.24 68.42 42.67 14.18 -3.28 80 46.37 63.13 41.62 7.43 -6.73 96 46.40 63.31 42.20 9.36 -4.78 128 50.43 62.79 42.16 13.11 -1.23________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -.76 -1.56 2.33 0 3.03 2 17.41 11.11 11.41 0 -4.76 4 42.12 55.11 30.20 19.51 .62 8 54.69 80.00 39.22 24.39 -3.88 16 54.77 81.62 40.89 20.34 -6.58 24 54.66 79.68 41.57 15.49 -8.99 32 54.92 76.82 41.79 17.59 -5.70 40 51.79 68.56 40.53 15.31 -3.87 48 51.72 66.40 40.84 9.72 -7.13 64 51.11 63.94 41.10 5.93 -8.82 80 46.51 59.50 39.80 9.33 -4.18 96 47.72 57.75 39.84 4.20 -7.62 128 54.35 58.95 40.66 3.24 -8.63________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________1 -1.43 -3.52 1.55 0 3.03 2 33.09 21.63 20.12 -10.00 -9.52 4 67.17 94.60 44.28 19.51 -11.80 8 75.72 108.14 49.15 25.00 -10.71 16 80.34 101.77 52.94 25.93 -4.49 24 70.84 93.12 43.62 27.63 -5.03 32 69.01 94.16 47.33 29.68 -1.51 40 58.56 63.47 25.91 -3.92 -25.85 48 61.16 74.70 34.88 .89 -22.08 64 54.37 69.09 26.80 -6.68 -30.04 80 36.22 22.73 -2.97 -8.25 -27.23 96 41.51 50.59 13.24 9.84 -16.77 128 48.98 38.15 6.41 -.33 -22.80________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________1 7.62 -38.03 -26.26 -50.00 -33.33 2 28.95 20.46 21.62 0 -7.14 4 84.05 60.79 45.74 -2.43 -12.42 8 86.43 79.57 50.32 15.85 -3.10 16 88.63 99.48 58.17 9.47 -13.10 24 74.65 80.87 41.99 -1.81 -22.89 32 63.86 59.21 23.58 -18.13 -36.37 40 64.79 60.53 22.23 -15.77 -35.84 48 49.68 26.93 .51 -36.40 -49.61 64 54.69 36.50 5.41 -26.59 -43.23 80 45.06 12.72 -13.25 -37.79 -52.08 96 40.21 -3.16 -24.53 -39.92 -52.97 128 36.33 -33.19 -43.66 -5.68 -20.49________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%
Quoted 1 line
"Michael S. Tsirkin" <[email protected]> wrote on 02/23/2011 12:09:15 PM:
Hi Michael,
Quoted 6 lines
Quoted 4 lines
Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?
Yes, please do.
Sure, will get a build/test on latest bits and send in 1-2 days.
Quoted 8 lines
Quoted 4 lines
The TX-only patch helped the guest TX path but didn't help
host->guestmuch (as tested usingTCP_MAERTSfrom the guest). But with the TX+RX patch, both directions are getting improvements.Also, my hope is that with appropriate queue mapping, we might be able to do away with heuristics to detect single stream load that TX only code needs.
Yes, that whole stuff is removed, and the TX/RX path is unchanged with this patch (thankfully :)
Quoted 3 lines
Quoted 1 line
Remote testing is still to be done.
Others might be able to help here once you post the patch.
That's great, will appreciate any help.
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 18 lines · On Wed, Feb 23, 2011 at 12:18:36PM +0530, Krishna Kumar2 wrote:
Quoted 1 line
"Michael S. Tsirkin" <[email protected]> wrote on 02/23/2011 12:09:15 PM:
Hi Michael,
Quoted 6 lines
Quoted 4 lines
Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?
Yes, please do.
Sure, will get a build/test on latest bits and send in 1-2 days.
Quoted 5 lines
Quoted 4 lines
The TX-only patch helped the guest TX path but didn't help
host->guestmuch (as tested usingTCP_MAERTSfrom the guest). But with the TX+RX patch, both directions are getting improvements.
Quoted 6 lines
Quoted 3 lines
Also, my hope is that with appropriate queue mapping, we might be able to do away with heuristics to detect single stream load that TX only code needs.
Yes, that whole stuff is removed, and the TX/RX path is unchanged with this patch (thankfully :)
Cool. I was wondering whether in that case, we can do without host kernel changes at all,
and use a separate fd for each TX/RX pair. The advantage of that approach is that this way, the max fd limit naturally sets an upper bound on the amount of resources userspace can use up.
Thoughts?
In any case, pls don't let the above delay sending an RFC.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Remote testing is still to be done.
Others might be able to help here once you post the patch.
That's great, will appreciate any help.
Thanks,
- KK
"Michael S. Tsirkin" <[email protected]> wrote on 02/23/2011 09:25:34 PM:
Quoted 21 lines
Quoted 7 lines
Sure, will get a build/test on latest bits and send in 1-2 days.
Quoted 5 lines
Quoted 4 lines
The TX-only patch helped the guest TX path but didn't help
host->guestmuch (as tested usingTCP_MAERTSfrom the guest). But with the TX+RX patch, both directions are getting improvements.Quoted 6 lines
Quoted 3 lines
Also, my hope is that with appropriate queue mapping, we might be able to do away with heuristics to detect single stream load that TX only code needs.
Yes, that whole stuff is removed, and the TX/RX path is unchanged with this patch (thankfully :)
Cool. I was wondering whether in that case, we can do without host kernel changes at all,
and use a separate fd for each TX/RX pair. The advantage of that approach is that this way, the max fd limit naturally sets an upper bound on the amount of resources userspace can use up.
Quoted 4 lines
Thoughts?
In any case, pls don't let the above delay sending an RFC.
I will look into this also.
Please excuse the delay in sending the patch out faster - my bits are a little old, so it is taking some time to move to the latest kernel and get some initial TCP/UDP test results. I should have it ready by tomorrow.
Thanks,
- KK
Quoted 32 lines · On Wed, Feb 23, 2011 at 10:52:09AM +0530, Krishna Kumar2 wrote:
Simon Horman <[email protected]> wrote on 02/22/2011 01:17:09 PM:
Hi Simon,
Quoted 3 lines
I have a few questions about the results below:
1. Are the (%) comparisons between non-mq and mq virtio?
Yes - mainline kernel with transmit-only MQ patch.
Quoted 1 line
2. Was UDP or TCP used?
TCP. I had done some initial testing on UDP, but don't have the results now as it is really old. But I will be running it again.
Quoted 1 line
3. What was the transmit size (-m option to netperf)?
I didn't use the -m option, so it defaults to 16K. The script does:
netperf -t
TCP_STREAM-c -C -l 60 -H $SERVERQuoted 2 lines
Also, I'm interested to know what the status of these patches is. Are you planing a fresh series?
Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?
Quoted 4 lines
The TX-only patch helped the guest TX path but didn't help
host->guestmuch (as tested usingTCP_MAERTSfrom the guest). But with the TX+RX patch, both directions are getting improvements. Remote testing is still to be done.
Hi Krishna,
thanks for clarifying the test results.
I'm looking forward to the forthcoming RFC patches.