Skip to contentSkip to conversation

Implement multiqueue virtio-net

RFC PATCH In review Krishna Kumar 4 participants 35 messages Oct 20, 2010 – Feb 24, 2011 3 messages not in the archive

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v3 cover Implement multiqueue virtio-net

Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.

Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:

       struct send_queue {
               struct virtqueue *svq;
               struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
       } ____cacheline_aligned_in_smp;

and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.

Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:

        vhost=on                   ->   #txqueues = 1
        vhost=on,mq=on             ->   #txqueues = 4
        vhost=on,mq=on,numtxqs=2   ->   #txqueues = 2
        vhost=on,mq=on,numtxqs=8   ->   #txqueues = 8
                   Performance (guest -> local host):
                   -----------------------------------

System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).

______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1          4.46    -1.96     .19     -12.50   -6.06
2          4.93    -1.16    2.10      0       -2.38
4          46.17    64.77   33.72     19.51   -2.48
8          47.89    70.00   36.23     41.46    13.35
16         48.97    80.44   40.67     21.11   -5.46
24         49.03    78.78   41.22     20.51   -4.78
32         51.11    77.15   42.42     15.81   -6.87
40         51.60    71.65   42.43     9.75    -8.94
48         50.10    69.55   42.85     11.80   -5.81
64         46.24    68.42   42.67     14.18   -3.28
80         46.37    63.13   41.62     7.43    -6.73
96         46.40    63.31   42.20     9.36    -4.78
128        50.43    62.79   42.16     13.11   -1.23

________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%

______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -.76    -1.56     2.33      0        3.03
2           17.41    11.11    11.41     0       -4.76
4           42.12    55.11    30.20     19.51    .62
8           54.69    80.00    39.22     24.39    -3.88
16          54.77    81.62    40.89     20.34    -6.58
24          54.66    79.68    41.57     15.49    -8.99
32          54.92    76.82    41.79     17.59    -5.70
40          51.79    68.56    40.53     15.31    -3.87
48          51.72    66.40    40.84     9.72     -7.13
64          51.11    63.94    41.10     5.93     -8.82
80          46.51    59.50    39.80     9.33     -4.18
96          47.72    57.75    39.84     4.20     -7.62
128         54.35    58.95    40.66     3.24     -8.63

________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%

______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -1.43    -3.52    1.55      0          3.03
2           33.09     21.63   20.12    -10.00     -9.52
4           67.17     94.60   44.28     19.51     -11.80
8           75.72     108.14  49.15     25.00     -10.71
16          80.34     101.77  52.94     25.93     -4.49
24          70.84     93.12   43.62     27.63     -5.03
32          69.01     94.16   47.33     29.68     -1.51
40          58.56     63.47   25.91    -3.92      -25.85
48          61.16     74.70   34.88     .89       -22.08
64          54.37     69.09   26.80    -6.68      -30.04
80          36.22     22.73   -2.97    -8.25      -27.23
96          41.51     50.59   13.24     9.84      -16.77
128         48.98     38.15   6.41     -.33       -22.80

________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%

______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________

1            7.62     -38.03   -26.26  -50.00   -33.33
2            28.95     20.46    21.62   0       -7.14
4            84.05     60.79    45.74  -2.43    -12.42
8            86.43     79.57    50.32   15.85   -3.10
16           88.63     99.48    58.17   9.47    -13.10
24           74.65     80.87    41.99  -1.81    -22.89
32           63.86     59.21    23.58  -18.13   -36.37
40           64.79     60.53    22.23  -15.77   -35.84
48           49.68     26.93    .51    -36.40   -49.61
64           54.69     36.50    5.41   -26.59   -43.23
80           45.06     12.72   -13.25  -37.79   -52.08
96           40.21    -3.16    -24.53  -39.92   -52.97
128          36.33    -33.19   -43.66  -5.68    -20.49

________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%

---

#1 lore raw

RFC PATCH v3 1/4 Change virtqueue structure

Move queue_index from virtio_pci_vq_info to virtqueue. This allows callback handlers to figure out the queue number for the vq that needs attention.

---
 drivers/virtio/virtio_pci.c |   10 +++-------
 include/linux/virtio.h      |    1 +
 2 files changed, 4 insertions(+), 7 deletions(-)
include/linux/virtio.h +1 -0
diff -ruNp org/include/linux/virtio.h new.dynamic.optimize_vhost/include/linux/virtio.h
@@ -22,6 +22,7 @@ struct virtqueue {
 	void (*callback)(struct virtqueue *vq);
 	const char *name;
 	struct virtio_device *vdev;
+	int queue_index;	/* the index of the queue */
 	void *priv;
 };
 
drivers/virtio/virtio_pci.c +3 -7
diff -ruNp org/drivers/virtio/virtio_pci.c new.dynamic.optimize_vhost/drivers/virtio/virtio_pci.c
@@ -75,9 +75,6 @@ struct virtio_pci_vq_info
 	/* the number of entries in the queue */
 	int num;
 
-	/* the index of the queue */
-	int queue_index;
-
 	/* the virtual address of the ring queue */
 	void *queue;
 
@@ -185,11 +182,10 @@ static void vp_reset(struct virtio_devic
 static void vp_notify(struct virtqueue *vq)
 {
 	struct virtio_pci_device *vp_dev = to_vp_device(vq->vdev);
-	struct virtio_pci_vq_info *info = vq->priv;
 
 	/* we write the queue's selector into the notification register to
 	 * signal the other end */
-	iowrite16(info->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_NOTIFY);
+	iowrite16(vq->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_NOTIFY);
 }
 
 /* Handle a configuration change: Tell driver if it wants to know. */
@@ -385,7 +381,6 @@ static struct virtqueue *setup_vq(struct
 	if (!info)
 		return ERR_PTR(-ENOMEM);
 
-	info->queue_index = index;
 	info->num = num;
 	info->msix_vector = msix_vec;
 
@@ -408,6 +403,7 @@ static struct virtqueue *setup_vq(struct
 		goto out_activate_queue;
 	}
 
+	vq->queue_index = index;
 	vq->priv = info;
 	info->vq = vq;
 
@@ -446,7 +442,7 @@ static void vp_del_vq(struct virtqueue *
 	list_del(&info->node);
 	spin_unlock_irqrestore(&vp_dev->lock, flags);
 
-	iowrite16(info->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_SEL);
+	iowrite16(vq->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_SEL);
 
 	if (vp_dev->msix_enabled) {
 		iowrite16(VIRTIO_MSI_NO_VECTOR,
#2 lore raw

RFC PATCH v3 2/4 Changes for virtio-net

In reply to Krishna Kumar

Implement mq virtio-net driver.

Though struct virtio_net_config changes, it works with old qemu's since the last element is not accessed, unless qemu sets VIRTIO_NET_F_NUMTXQS. Patch also adds a macro for the maximum number of TX vq's (VIRTIO_MAX_SQ) that the user can specify.

---

 drivers/net/virtio_net.c   |  234 ++++++++++++++++++++++++++---------
 include/linux/virtio_net.h |    6 
 2 files changed, 185 insertions(+), 55 deletions(-)
include/linux/virtio_net.h +6 -0
diff -ruNp org/include/linux/virtio_net.h new.dynamic.optimize_vhost/include/linux/virtio_net.h
@@ -7,6 +7,9 @@
 #include <linux/virtio_config.h>
 #include <linux/if_ether.h>
 
+/* Maximum number of TX queues supported */
+#define VIRTIO_MAX_SQ 32
+
 /* The feature bitmap for virtio net */
 #define VIRTIO_NET_F_CSUM	0	/* Host handles pkts w/ partial csum */
 #define VIRTIO_NET_F_GUEST_CSUM	1	/* Guest handles pkts w/ partial csum */
@@ -26,6 +29,7 @@
 #define VIRTIO_NET_F_CTRL_RX	18	/* Control channel RX mode support */
 #define VIRTIO_NET_F_CTRL_VLAN	19	/* Control channel VLAN filtering */
 #define VIRTIO_NET_F_CTRL_RX_EXTRA 20	/* Extra RX mode control support */
+#define VIRTIO_NET_F_NUMTXQS	21	/* Device supports multiple TX queue */
 
 #define VIRTIO_NET_S_LINK_UP	1	/* Link is up */
 
@@ -34,6 +38,8 @@ struct virtio_net_config {
 	__u8 mac[6];
 	/* See VIRTIO_NET_F_STATUS and VIRTIO_NET_S_* above */
 	__u16 status;
+	/* number of transmit queues */
+	__u16 numtxqs;
 } __attribute__((packed));
 
 /* This is the first element of the scatter-gather list.  If you don't
drivers/net/virtio_net.c +179 -55

378 lines not on this page · raw message

#3 lore raw

RFC PATCH v3 3/4 Changes for vhost

In reply to Krishna Kumar

Changes for mq vhost.

vhost_net_open is changed to allocate a vhost_net and return. The remaining initializations are delayed till SET_OWNER. SET_OWNER is changed so that the argument is used to determine how many txqs to use. Unmodified qemu's will pass NULL, so this is recognized and handled as numtxqs=1.

Besides changing handle_tx to use 'vq', this patch also changes handle_rx to take vq as parameter. The mq RX patch requires this change, but till then it is consistent (and less confusing) to make the interfaces for handling rx and tx similar.

vhost thread handling for RX and TX is as follows. The first vhost thread handles RX traffic, while the remaining threads handles TX. The number of threads is <= #txqs, and threads handle more than one txq when #txqs is more than MAX_VHOST_THREADS (4). When guest is started with >1 txqs and there is only one stream of traffic from the guest, that is recognized and handled such that vhost[0] processes both RX and TX. This can change dynamically. vhost_poll has a new element - find_vq(), which allows optimizing some code for cases where numtxqs=1 or a packet on vhost[0] needs processing.

---
 drivers/vhost/net.c   |  284 ++++++++++++++++++++++++++--------------
 drivers/vhost/vhost.c |  275 ++++++++++++++++++++++++++++----------
 drivers/vhost/vhost.h |   42 +++++
 3 files changed, 430 insertions(+), 171 deletions(-)
drivers/vhost/vhost.h +36 -6
diff -ruNp org/drivers/vhost/vhost.h new/drivers/vhost/vhost.h
@@ -35,11 +35,13 @@ struct vhost_poll {
 	wait_queue_t              wait;
 	struct vhost_work	  work;
 	unsigned long		  mask;
-	struct vhost_dev	 *dev;
+	struct vhost_virtqueue	  *(*find_vq)(struct vhost_poll *poll);
+	struct vhost_virtqueue	  *vq;  /* points back to vq */
 };
 
 void vhost_poll_init(struct vhost_poll *poll, vhost_work_fn_t fn,
-		     unsigned long mask, struct vhost_dev *dev);
+		     unsigned long mask, struct vhost_virtqueue *vq,
+		     int single_queue);
 void vhost_poll_start(struct vhost_poll *poll, struct file *file);
 void vhost_poll_stop(struct vhost_poll *poll);
 void vhost_poll_flush(struct vhost_poll *poll);
@@ -108,6 +110,10 @@ struct vhost_virtqueue {
 	/* Log write descriptors */
 	void __user *log_base;
 	struct vhost_log *log;
+	struct task_struct *worker; /* vhost for this vq, can be shared */
+	spinlock_t *work_lock;
+	struct list_head *work_list;
+	int qnum;		/* 0 for RX, 1 -> n-1 for TX */
 };
 
 struct vhost_dev {
@@ -119,15 +125,39 @@ struct vhost_dev {
 	struct mutex mutex;
 	unsigned acked_features;
 	struct vhost_virtqueue *vqs;
+	unsigned long *jiffies;
 	int nvqs;
 	struct file *log_file;
 	struct eventfd_ctx *log_ctx;
-	spinlock_t work_lock;
-	struct list_head work_list;
-	struct task_struct *worker;
+	spinlock_t *work_lock;
+	struct list_head *work_list;
 };
 
-long vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue *vqs, int nvqs);
+/*
+ * Define maximum number of TX threads, and use that to have a maximum
+ * number of vhost threads to handle RX & TX. First thread handles RX.
+ * If guest is started with #txqs=1, only one vhost thread is started.
+ * Else, upto MAX_VHOST_THREADS are started where th[0] handles RX and
+ * remaining handles TX. However, vhost_poll_queue has an optimization
+ * where th[0] is selected for both RX & TX if there is only one flow.
+ */
+#define MAX_TXQ_THREADS		4
+#define MAX_VHOST_THREADS	(MAX_TXQ_THREADS + 1)
+
+static inline int get_nvhosts(int nvqs)
+{
+	int num_vhosts = nvqs - 1;
+
+	if (nvqs > 2)
+		num_vhosts = min_t(int, nvqs, MAX_VHOST_THREADS);
+
+	return num_vhosts;
+}
+
+int vhost_setup_vqs(struct vhost_dev *dev, int numtxqs);
+void vhost_free_vqs(struct vhost_dev *dev);
+long vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue *vqs, int nvqs,
+		    int nvhosts);
 long vhost_dev_check_owner(struct vhost_dev *);
 long vhost_dev_reset_owner(struct vhost_dev *);
 void vhost_dev_cleanup(struct vhost_dev *);
drivers/vhost/net.c +189 -95

465 lines not on this page · raw message

drivers/vhost/vhost.c +205 -70

411 lines not on this page · raw message

#4 lore raw

RFC PATCH v3 4/4 qemu changes

In reply to Krishna Kumar

Changes in qemu to support mq TX.

---

 hw/vhost.c      |    7 ++++--
 hw/vhost.h      |    2 -
 hw/vhost_net.c  |   16 +++++++++----
 hw/vhost_net.h  |    2 -
 hw/virtio-net.c |   53 ++++++++++++++++++++++++++++++++--------------
 hw/virtio-net.h |    2 +
 hw/virtio-pci.c |    2 +
 net.c           |   17 ++++++++++++++
 net.h           |    1 
 net/tap.c       |   34 ++++++++++++++++++++++++++---
 10 files changed, 107 insertions(+), 29 deletions(-)
hw/vhost.c +5 -2
diff -ruNp org3/hw/vhost.c new3/hw/vhost.c
@@ -580,7 +580,7 @@ static void vhost_virtqueue_cleanup(stru
                               0, virtio_queue_get_desc_size(vdev, idx));
 }
 
-int vhost_dev_init(struct vhost_dev *hdev, int devfd)
+int vhost_dev_init(struct vhost_dev *hdev, int devfd, int numtxqs)
 {
     uint64_t features;
     int r;
@@ -592,11 +592,14 @@ int vhost_dev_init(struct vhost_dev *hde
             return -errno;
         }
     }
-    r = ioctl(hdev->control, VHOST_SET_OWNER, NULL);
+
+    r = ioctl(hdev->control, VHOST_SET_OWNER, numtxqs);
     if (r < 0) {
         goto fail;
     }
 
+    hdev->nvqs = numtxqs + 1;
+
     r = ioctl(hdev->control, VHOST_GET_FEATURES, &features);
     if (r < 0) {
         goto fail;
hw/vhost.h +1 -1
diff -ruNp org3/hw/vhost.h new3/hw/vhost.h
@@ -40,7 +40,7 @@ struct vhost_dev {
     unsigned long long log_size;
 };
 
-int vhost_dev_init(struct vhost_dev *hdev, int devfd);
+int vhost_dev_init(struct vhost_dev *hdev, int devfd, int numtxqs);
 void vhost_dev_cleanup(struct vhost_dev *hdev);
 int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev);
 void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev);
hw/vhost_net.c +11 -5
diff -ruNp org3/hw/vhost_net.c new3/hw/vhost_net.c
@@ -36,7 +36,8 @@
 
 struct vhost_net {
     struct vhost_dev dev;
-    struct vhost_virtqueue vqs[2];
+    struct vhost_virtqueue *vqs;
+    int nvqs;
     int backend;
     VLANClientState *vc;
 };
@@ -81,7 +82,8 @@ static int vhost_net_get_fd(VLANClientSt
     }
 }
 
-struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd)
+struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd,
+				 int numtxqs)
 {
     int r;
     struct vhost_net *net = qemu_malloc(sizeof *net);
@@ -98,10 +100,14 @@ struct vhost_net *vhost_net_init(VLANCli
         (1 << VHOST_NET_F_VIRTIO_NET_HDR);
     net->backend = r;
 
-    r = vhost_dev_init(&net->dev, devfd);
+    r = vhost_dev_init(&net->dev, devfd, numtxqs);
     if (r < 0) {
         goto fail;
     }
+
+    net->nvqs = numtxqs + 1;
+    net->vqs = qemu_malloc(net->nvqs * (sizeof *net->vqs));
+
     if (!tap_has_vnet_hdr_len(backend,
                               sizeof(struct virtio_net_hdr_mrg_rxbuf))) {
         net->dev.features &= ~(1 << VIRTIO_NET_F_MRG_RXBUF);
@@ -131,7 +137,6 @@ int vhost_net_start(struct vhost_net *ne
                              sizeof(struct virtio_net_hdr_mrg_rxbuf));
     }
 
-    net->dev.nvqs = 2;
     net->dev.vqs = net->vqs;
     r = vhost_dev_start(&net->dev, dev);
     if (r < 0) {
@@ -188,7 +193,8 @@ void vhost_net_cleanup(struct vhost_net 
     qemu_free(net);
 }
 #else
-struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd)
+struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd,
+				 int nvqs)
 {
 	return NULL;
 }
hw/vhost_net.h +1 -1
diff -ruNp org3/hw/vhost_net.h new3/hw/vhost_net.h
@@ -6,7 +6,7 @@
 struct vhost_net;
 typedef struct vhost_net VHostNetState;
 
-VHostNetState *vhost_net_init(VLANClientState *backend, int devfd);
+VHostNetState *vhost_net_init(VLANClientState *backend, int devfd, int nvqs);
 
 int vhost_net_start(VHostNetState *net, VirtIODevice *dev);
 void vhost_net_stop(VHostNetState *net, VirtIODevice *dev);
hw/virtio-net.c +37 -16

156 lines not on this page · raw message

hw/virtio-net.h +2 -0
diff -ruNp org3/hw/virtio-net.h new3/hw/virtio-net.h
@@ -44,6 +44,7 @@
 #define VIRTIO_NET_F_CTRL_RX    18      /* Control channel RX mode support */
 #define VIRTIO_NET_F_CTRL_VLAN  19      /* Control channel VLAN filtering */
 #define VIRTIO_NET_F_CTRL_RX_EXTRA 20   /* Extra RX mode control support */
+#define VIRTIO_NET_F_NUMTXQS    21      /* Supports multiple TX queues */
 
 #define VIRTIO_NET_S_LINK_UP    1       /* Link is up */
 
@@ -72,6 +73,7 @@ struct virtio_net_config
     uint8_t mac[ETH_ALEN];
     /* See VIRTIO_NET_F_STATUS and VIRTIO_NET_S_* above */
     uint16_t status;
+    uint16_t numtxqs;	/* number of transmit queues */
 } __attribute__((packed));
 
 /* This is the first element of the scatter-gather list.  If you don't
hw/virtio-pci.c +2 -0
diff -ruNp org3/hw/virtio-pci.c new3/hw/virtio-pci.c
@@ -99,6 +99,7 @@ typedef struct {
     uint32_t addr;
     uint32_t class_code;
     uint32_t nvectors;
+    uint32_t mq;
     BlockConf block;
     NICConf nic;
     uint32_t host_features;
@@ -788,6 +789,7 @@ static PCIDeviceInfo virtio_info[] = {
         .romfile    = "pxe-virtio.bin",
         .qdev.props = (Property[]) {
             DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 3),
+	    DEFINE_PROP_UINT32("mq", VirtIOPCIProxy, mq, 1),
             DEFINE_VIRTIO_NET_FEATURES(VirtIOPCIProxy, host_features),
             DEFINE_NIC_PROPERTIES(VirtIOPCIProxy, nic),
             DEFINE_PROP_UINT32("x-txtimer", VirtIOPCIProxy,
net/tap.c +30 -4
diff -ruNp org3/net/tap.c new3/net/tap.c
@@ -320,13 +320,14 @@ static NetClientInfo net_tap_info = {
 static TAPState *net_tap_fd_init(VLANState *vlan,
                                  const char *model,
                                  const char *name,
-                                 int fd,
+                                 int fd, int numtxqs,
                                  int vnet_hdr)
 {
     VLANClientState *nc;
     TAPState *s;
 
     nc = qemu_new_net_client(&net_tap_info, vlan, NULL, model, name);
+    nc->numtxqs = numtxqs;
 
     s = DO_UPCAST(TAPState, nc, nc);
 
@@ -424,6 +425,27 @@ int net_init_tap(QemuOpts *opts, Monitor
 {
     TAPState *s;
     int fd, vnet_hdr = 0;
+    int vhost;
+    int numtxqs = 1;
+
+    vhost = qemu_opt_get_bool(opts, "vhost", 0);
+
+    /*
+     * We support multiple tx queues if:
+     *      1. smp > 1
+     *      2. vhost=on
+     *      3. mq=on
+     * In this case, #txqueues = #cpus. This value can be changed by
+     * using the "numtxqs" option.
+     */
+    if (vhost && smp_cpus > 1) {
+        if (qemu_opt_get_bool(opts, "mq", 0)) {
+#define VIRTIO_MAX_TXQS         32
+            int dflt = MIN(smp_cpus, VIRTIO_MAX_TXQS);
+
+            numtxqs = qemu_opt_get_number(opts, "numtxqs", dflt);
+        }
+    }
 
     if (qemu_opt_get(opts, "fd")) {
         if (qemu_opt_get(opts, "ifname") ||
@@ -457,7 +479,7 @@ int net_init_tap(QemuOpts *opts, Monitor
         }
     }
 
-    s = net_tap_fd_init(vlan, "tap", name, fd, vnet_hdr);
+    s = net_tap_fd_init(vlan, "tap", name, fd, numtxqs, vnet_hdr);
     if (!s) {
         close(fd);
         return -1;
@@ -486,7 +508,7 @@ int net_init_tap(QemuOpts *opts, Monitor
         }
     }
 
-    if (qemu_opt_get_bool(opts, "vhost", !!qemu_opt_get(opts, "vhostfd"))) {
+    if (vhost) {
         int vhostfd, r;
         if (qemu_opt_get(opts, "vhostfd")) {
             r = net_handle_fd_param(mon, qemu_opt_get(opts, "vhostfd"));
@@ -497,9 +519,13 @@ int net_init_tap(QemuOpts *opts, Monitor
         } else {
             vhostfd = -1;
         }
-        s->vhost_net = vhost_net_init(&s->nc, vhostfd);
+        s->vhost_net = vhost_net_init(&s->nc, vhostfd, numtxqs);
         if (!s->vhost_net) {
             error_report("vhost-net requested but could not be initialized");
+            if (numtxqs > 1) {
+                error_report("Need vhost support for numtxqs > 1, exiting...");
+                exit(1);
+            }
             return -1;
         }
     } else if (qemu_opt_get(opts, "vhostfd")) {
net.c +17 -0
diff -ruNp org3/net.c new3/net.c
@@ -849,6 +849,15 @@ static int net_init_nic(QemuOpts *opts,
         return -1;
     }
 
+    if (nd->netdev->numtxqs > 1 && nd->nvectors == DEV_NVECTORS_UNSPECIFIED) {
+        /*
+         * User specified mq for guest, but no "vectors=", tune
+         * it automatically to 'numtxqs' TX + 1 RX + 1 controlq.
+         */
+        nd->nvectors = nd->netdev->numtxqs + 1 + 1;
+        monitor_printf(mon, "nvectors tuned to %d\n", nd->nvectors);
+    }
+
     nd->used = 1;
     nb_nics++;
 
@@ -992,6 +1001,14 @@ static const struct {
             },
 #ifndef _WIN32
             {
+                .name = "mq",
+                .type = QEMU_OPT_BOOL,
+                .help = "enable multiqueue on network i/f",
+            }, {
+                .name = "numtxqs",
+                .type = QEMU_OPT_NUMBER,
+                .help = "optional number of TX queues, if mq is enabled",
+            }, {
                 .name = "fd",
                 .type = QEMU_OPT_STRING,
                 .help = "file descriptor of an already opened tap",
net.h +1 -0
diff -ruNp org3/net.h new3/net.h
@@ -62,6 +62,7 @@ struct VLANClientState {
     struct VLANState *vlan;
     VLANClientState *peer;
     NetQueue *send_queue;
+    int numtxqs;
     char *model;
     char *name;
     char info_str[256];
#5 lore raw
In reply to Krishna Kumar
Quoted 1 line

Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:

Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.

Some more test results:
_____________________________________________________

         Host->Guest BW (numtxqs=2)
#       BW%     CPU%    RCPU%   SD%     RSD%
_____________________________________________________
1       5.53    .31     .67     -5.88   0
2       -2.11   -1.01   -2.08   4.34    0
4       13.53   10.77   13.87   -1.96   0
8       34.22   22.80   30.53   -8.46   -2.50
16      30.89   24.06   35.17   -5.20   3.20
24      33.22   26.30   43.39   -5.17   7.58
32      30.85   27.27   47.74   -.59    15.51
40      33.80   27.33   48.00   -7.42   7.59
48      45.93   26.33   45.46   -12.24  1.10
64      33.51   27.11   45.00   -3.27   10.30
80      39.28   29.21   52.33   -4.88   12.17
96      32.05   31.01   57.72   -1.02   19.05
128     35.66   32.04   60.00   -.66    20.41

_____________________________________________________
BW: 23.5% CPU/RCPU: 28.6%,51.2% SD/RSD: -2.6%,15.8%

____________________________________________________
Guest->Host 512 byte (numtxqs=2):
# BW% CPU% RCPU% SD% RSD%
_____________________________________________________

1       3.02    -3.84   -4.76   -12.50  -7.69
2       52.77   -15.73  -8.66   -45.31  -40.33
4       -23.14  13.84   7.50    50.58   40.81
8       -21.44  28.08   16.32   63.06   47.43
16      33.53   46.50   27.19   7.61    -6.60
24      55.77   42.81   30.49   -8.65   -16.48
32      52.59   38.92   29.08   -9.18   -15.63
40      50.92   36.11   28.92   -10.59  -15.30
48      46.63   34.73   28.17   -7.83   -12.32
64      45.56   37.12   28.81   -5.05   -10.80
80      44.55   36.60   28.45   -4.95   -10.61
96      43.02   35.97   28.89   -.11    -5.31
128     38.54   33.88   27.19   -4.79   -9.54

_____________________________________________________
BW: 34.4% CPU/RCPU: 35.9%,27.8% SD/RSD: -4.1%,-9.3%

Thanks,

- KK

Quoted 127 lines

[v3 RFC PATCH 0/4] Implement multiqueue virtio-net

Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.

Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:

       struct send_queue {
               struct virtqueue *svq;
               struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
       } ____cacheline_aligned_in_smp;

and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.

Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:

        vhost=on                   ->   #txqueues = 1
        vhost=on,mq=on             ->   #txqueues = 4
        vhost=on,mq=on,numtxqs=2   ->   #txqueues = 2
        vhost=on,mq=on,numtxqs=8   ->   #txqueues = 8
                   Performance (guest -> local host):
                   -----------------------------------

System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).

______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1          4.46    -1.96     .19     -12.50   -6.06
2          4.93    -1.16    2.10      0       -2.38
4          46.17    64.77   33.72     19.51   -2.48
8          47.89    70.00   36.23     41.46    13.35
16         48.97    80.44   40.67     21.11   -5.46
24         49.03    78.78   41.22     20.51   -4.78
32         51.11    77.15   42.42     15.81   -6.87
40         51.60    71.65   42.43     9.75    -8.94
48         50.10    69.55   42.85     11.80   -5.81
64         46.24    68.42   42.67     14.18   -3.28
80         46.37    63.13   41.62     7.43    -6.73
96         46.40    63.31   42.20     9.36    -4.78
128        50.43    62.79   42.16     13.11   -1.23

________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%

______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -.76    -1.56     2.33      0        3.03
2           17.41    11.11    11.41     0       -4.76
4           42.12    55.11    30.20     19.51    .62
8           54.69    80.00    39.22     24.39    -3.88
16          54.77    81.62    40.89     20.34    -6.58
24          54.66    79.68    41.57     15.49    -8.99
32          54.92    76.82    41.79     17.59    -5.70
40          51.79    68.56    40.53     15.31    -3.87
48          51.72    66.40    40.84     9.72     -7.13
64          51.11    63.94    41.10     5.93     -8.82
80          46.51    59.50    39.80     9.33     -4.18
96          47.72    57.75    39.84     4.20     -7.62
128         54.35    58.95    40.66     3.24     -8.63

________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%

______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -1.43    -3.52    1.55      0          3.03
2           33.09     21.63   20.12    -10.00     -9.52
4           67.17     94.60   44.28     19.51     -11.80
8           75.72     108.14  49.15     25.00     -10.71
16          80.34     101.77  52.94     25.93     -4.49
24          70.84     93.12   43.62     27.63     -5.03
32          69.01     94.16   47.33     29.68     -1.51
40          58.56     63.47   25.91    -3.92      -25.85
48          61.16     74.70   34.88     .89       -22.08
64          54.37     69.09   26.80    -6.68      -30.04
80          36.22     22.73   -2.97    -8.25      -27.23
96          41.51     50.59   13.24     9.84      -16.77
128         48.98     38.15   6.41     -.33       -22.80

________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%

______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________

1            7.62     -38.03   -26.26  -50.00   -33.33
2            28.95     20.46    21.62   0       -7.14
4            84.05     60.79    45.74  -2.43    -12.42
8            86.43     79.57    50.32   15.85   -3.10
16           88.63     99.48    58.17   9.47    -13.10
24           74.65     80.87    41.99  -1.81    -22.89
32           63.86     59.21    23.58  -18.13   -36.37
40           64.79     60.53    22.23  -15.77   -35.84
48           49.68     26.93    .51    -36.40   -49.61
64           54.69     36.50    5.41   -26.59   -43.23
80           45.06     12.72   -13.25  -37.79   -52.08
96           40.21    -3.16    -24.53  -39.92   -52.97
128          36.33    -33.19   -43.66  -5.68    -20.49

________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%

#6 lore raw
Quoted 4 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line

Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:

Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.

I am trying to wrap my head around kernel/user interface here. E.g., will we need another incompatible change when we add multiple RX queues? Also need to think about how robust our single stream heuristic is, e.g. what are the chances it will misdetect a bidirectional
UDP stream as a single TCP?

Quoted 175 lines

Some more test results:
_____________________________________________________

         Host->Guest BW (numtxqs=2)
#       BW%     CPU%    RCPU%   SD%     RSD%
_____________________________________________________
1       5.53    .31     .67     -5.88   0
2       -2.11   -1.01   -2.08   4.34    0
4       13.53   10.77   13.87   -1.96   0
8       34.22   22.80   30.53   -8.46   -2.50
16      30.89   24.06   35.17   -5.20   3.20
24      33.22   26.30   43.39   -5.17   7.58
32      30.85   27.27   47.74   -.59    15.51
40      33.80   27.33   48.00   -7.42   7.59
48      45.93   26.33   45.46   -12.24  1.10
64      33.51   27.11   45.00   -3.27   10.30
80      39.28   29.21   52.33   -4.88   12.17
96      32.05   31.01   57.72   -1.02   19.05
128     35.66   32.04   60.00   -.66    20.41

_____________________________________________________
BW: 23.5% CPU/RCPU: 28.6%,51.2% SD/RSD: -2.6%,15.8%

____________________________________________________
Guest->Host 512 byte (numtxqs=2):
# BW% CPU% RCPU% SD% RSD%
_____________________________________________________

1       3.02    -3.84   -4.76   -12.50  -7.69
2       52.77   -15.73  -8.66   -45.31  -40.33
4       -23.14  13.84   7.50    50.58   40.81
8       -21.44  28.08   16.32   63.06   47.43
16      33.53   46.50   27.19   7.61    -6.60
24      55.77   42.81   30.49   -8.65   -16.48
32      52.59   38.92   29.08   -9.18   -15.63
40      50.92   36.11   28.92   -10.59  -15.30
48      46.63   34.73   28.17   -7.83   -12.32
64      45.56   37.12   28.81   -5.05   -10.80
80      44.55   36.60   28.45   -4.95   -10.61
96      43.02   35.97   28.89   -.11    -5.31
128     38.54   33.88   27.19   -4.79   -9.54

_____________________________________________________
BW: 34.4% CPU/RCPU: 35.9%,27.8% SD/RSD: -4.1%,-9.3%

Thanks,

- KK

Quoted 127 lines

[v3 RFC PATCH 0/4] Implement multiqueue virtio-net

Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.

Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:

       struct send_queue {
               struct virtqueue *svq;
               struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
       } ____cacheline_aligned_in_smp;

and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.

Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:

        vhost=on                   ->   #txqueues = 1
        vhost=on,mq=on             ->   #txqueues = 4
        vhost=on,mq=on,numtxqs=2   ->   #txqueues = 2
        vhost=on,mq=on,numtxqs=8   ->   #txqueues = 8
                   Performance (guest -> local host):
                   -----------------------------------

System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).

______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1          4.46    -1.96     .19     -12.50   -6.06
2          4.93    -1.16    2.10      0       -2.38
4          46.17    64.77   33.72     19.51   -2.48
8          47.89    70.00   36.23     41.46    13.35
16         48.97    80.44   40.67     21.11   -5.46
24         49.03    78.78   41.22     20.51   -4.78
32         51.11    77.15   42.42     15.81   -6.87
40         51.60    71.65   42.43     9.75    -8.94
48         50.10    69.55   42.85     11.80   -5.81
64         46.24    68.42   42.67     14.18   -3.28
80         46.37    63.13   41.62     7.43    -6.73
96         46.40    63.31   42.20     9.36    -4.78
128        50.43    62.79   42.16     13.11   -1.23

________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%

______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -.76    -1.56     2.33      0        3.03
2           17.41    11.11    11.41     0       -4.76
4           42.12    55.11    30.20     19.51    .62
8           54.69    80.00    39.22     24.39    -3.88
16          54.77    81.62    40.89     20.34    -6.58
24          54.66    79.68    41.57     15.49    -8.99
32          54.92    76.82    41.79     17.59    -5.70
40          51.79    68.56    40.53     15.31    -3.87
48          51.72    66.40    40.84     9.72     -7.13
64          51.11    63.94    41.10     5.93     -8.82
80          46.51    59.50    39.80     9.33     -4.18
96          47.72    57.75    39.84     4.20     -7.62
128         54.35    58.95    40.66     3.24     -8.63

________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%

______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -1.43    -3.52    1.55      0          3.03
2           33.09     21.63   20.12    -10.00     -9.52
4           67.17     94.60   44.28     19.51     -11.80
8           75.72     108.14  49.15     25.00     -10.71
16          80.34     101.77  52.94     25.93     -4.49
24          70.84     93.12   43.62     27.63     -5.03
32          69.01     94.16   47.33     29.68     -1.51
40          58.56     63.47   25.91    -3.92      -25.85
48          61.16     74.70   34.88     .89       -22.08
64          54.37     69.09   26.80    -6.68      -30.04
80          36.22     22.73   -2.97    -8.25      -27.23
96          41.51     50.59   13.24     9.84      -16.77
128         48.98     38.15   6.41     -.33       -22.80

________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%

______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________

1            7.62     -38.03   -26.26  -50.00   -33.33
2            28.95     20.46    21.62   0       -7.14
4            84.05     60.79    45.74  -2.43    -12.42
8            86.43     79.57    50.32   15.85   -3.10
16           88.63     99.48    58.17   9.47    -13.10
24           74.65     80.87    41.99  -1.81    -22.89
32           63.86     59.21    23.58  -18.13   -36.37
40           64.79     60.53    22.23  -15.77   -35.84
48           49.68     26.93    .51    -36.40   -49.61
64           54.69     36.50    5.41   -26.59   -43.23
80           45.06     12.72   -13.25  -37.79   -52.08
96           40.21    -3.16    -24.53  -39.92   -52.97
128          36.33    -33.19   -43.66  -5.68    -20.49

________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%

#7 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 10/25/2010 09:47:18 PM:

Quoted 6 lines
Quoted 2 lines

Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.

I am trying to wrap my head around kernel/user interface here. E.g., will we need another incompatible change when we add multiple RX queues?

Though I added a 'mq' option to qemu, there shouldn't be any incompatibility between old and new qemu's wrt vhost and virtio-net drivers. So the old qemu will run new host and new guest without issues, and new qemu can also run old host and old guest. Multiple RXQ will also not add any incompatibility.

With MQ RX, I will be able to remove the hueristic (idea from David Stevens). The idea is: Guest sends out packets on, say TXQ#2, vhost#2 processes the packets but packets going out from host to guest might be sent out on a different RXQ, say RXQ#4. Guest receives the packet on RXQ#4, and all future responses on that connection are sent on TXQ#4. Now vhost#4 processes both RX and TX packets for this connection. Without needing to hash on the connection, guest can make sure that the same vhost thread will handle a single connection.

Quoted 3 lines

Also need to think about how robust our single stream heuristic is, e.g. what are the chances it will misdetect a bidirectional UDP stream as a single TCP?

I think it should not happen. The hueristic code gets called for handling just the transmit packets, packets that vhost sends out to the guest skip this path.

I tested unidirectional and bidirectional UDP to confirm:

8 iterations of iperf tests, each iteration of 15 secs, result is the sum of all 8 iterations in Gbits/sec
__________________________________________
Uni-directional Bi-directional
Org New Org New
__________________________________________
71.78 71.77 71.74 72.07
__________________________________________

Thanks,

- KK

A message that is not in the archive (sent off-list or not yet received): OF5C53E9CF.FFDF2CE7-ON652577C8.00191D14-652577C8.001C2154@LocalDomain

#9 lore raw

Krishna Kumar2/India/IBM wrote on 10/26/2010 10:40:35 AM:

Quoted 32 lines
Quoted 3 lines

I am trying to wrap my head around kernel/user interface here. E.g., will we need another incompatible change when we add multiple RX queues?

Though I added a 'mq' option to qemu, there shouldn't be any incompatibility between old and new qemu's wrt vhost and virtio-net drivers. So the old qemu will run new host and new guest without issues, and new qemu can also run old host and old guest. Multiple RXQ will also not add any incompatibility.

With MQ RX, I will be able to remove the hueristic (idea from David Stevens). The idea is: Guest sends out packets on, say TXQ#2, vhost#2 processes the packets but packets going out from host to guest might be sent out on a different RXQ, say RXQ#4. Guest receives the packet on RXQ#4, and all future responses on that connection are sent on TXQ#4. Now vhost#4 processes both RX and TX packets for this connection. Without needing to hash on the connection, guest can make sure that the same vhost thread will handle a single connection.

Quoted 3 lines

Also need to think about how robust our single stream heuristic is, e.g. what are the chances it will misdetect a bidirectional UDP stream as a single TCP?

I think it should not happen. The hueristic code gets called for handling just the transmit packets, packets that vhost sends out to the guest skip this path.

I tested unidirectional and bidirectional UDP to confirm:

Quoted 8 lines

8 iterations of iperf tests, each iteration of 15 secs, result is the sum of all 8 iterations in Gbits/sec
__________________________________________
Uni-directional Bi-directional
Org New Org New
__________________________________________
71.78 71.77 71.74 72.07
__________________________________________

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

------ numtxqs=8, vhosts=5 ---------
# BW% CPU% SD%
------------------------------------

1     .49    1.07     0
2    23.51   52.51    26.66
4    75.17   72.43    8.57
8    86.54   80.21    27.85
16   92.37   85.99    6.27
24   91.37   84.91    8.41
32   89.78   82.90    3.31
48   89.85   79.95   -3.57
64   85.83   80.28    2.22
80   88.90   79.47   -23.18
96   90.12   79.98    14.71
128  86.13   80.60    4.42

------------------------------------
BW: 71.3%, CPU: 80.4%, SD: 1.2%

------ numtxqs=16, vhosts=5 --------
# BW% CPU% SD%
------------------------------------

1    1.80     0        0
2    19.81    50.68    26.66
4    57.31    52.77    8.57
8    108.44   88.19   -5.21
16   106.09   85.03   -4.44
24   102.34   84.23   -.82
32   102.77   82.71   -5.81
48   100.00   79.62   -7.29
64   96.86    79.75   -6.10
80   99.26    79.82   -27.34
96   94.79    80.02   -5.08
128  98.14    81.15   -15.25

------------------------------------
BW: 77.9%, CPU: 80.4%, SD: -13.6%

Thanks,

- KK

#10 lore raw
Quoted 3 lines · On Tue, Oct 26, 2010 at 02:38:53PM +0530, Krishna Kumar2 wrote:

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?

Signature · 1 line
MST
#11 lore raw
Quoted 9 lines

"Michael S. Tsirkin" <[email protected]>

Quoted 3 lines · On Tue, Oct 26, 2010 at 02:38:53PM +0530, Krishna Kumar2 wrote:

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?

Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding. I started binding vhost thread after Avi suggested it in response to my v1 patch (he suggested some more that I haven't done), and have been doing only this tuning ever since. This is part of his mail for the tuning:

Quoted 5 lines

vhost:
thread #0: CPU0
thread #1: CPU1
thread #2: CPU2
thread #3: CPU3

I simply bound each thread to CPU0-3 instead.

Thanks,

- KK

#12 lore raw
Quoted 7 lines · On Tue, Oct 26, 2010 at 03:31:39PM +0530, Krishna Kumar2 wrote:
Quoted 7 lines

"Michael S. Tsirkin" <[email protected]>

Quoted 3 lines · On Tue, Oct 26, 2010 at 02:38:53PM +0530, Krishna Kumar2 wrote:

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

Quoted 5 lines
Quoted 2 lines

Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?

Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.

If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.

Quoted 17 lines

I started binding vhost thread
after Avi suggested it in response to my v1 patch (he suggested some more that I haven't done), and have been doing only this tuning ever since. This is part of his mail for the tuning:

Quoted 5 lines

vhost:
thread #0: CPU0
thread #1: CPU1
thread #2: CPU2
thread #3: CPU3

I simply bound each thread to CPU0-3 instead.

Thanks,

- KK

#13 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 10/26/2010 04:39:13 PM:

(merging two posts into one)

Quoted 4 lines

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

For external host, there is a contention among different queues (vhosts) when packets are processed in tun/bridge, unless I implement MQ TX for macvtap (tun/bridge?). So my testing shows a small improvement (1 to 1.5% average) in BW and a rise in SD (between 10-15%). For remote host, I think tun/macvtap needs MQ TX support?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?

Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.

If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.

I will get a test run with and without binding and post the results later today.

Thanks,

- KK

#14 lore raw
Quoted 9 lines · On Thu, Oct 28, 2010 at 10:44:14AM +0530, Krishna Kumar2 wrote:

"Michael S. Tsirkin" <[email protected]> wrote on 10/26/2010 04:39:13 PM:

(merging two posts into one)

Quoted 4 lines

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

Quoted 6 lines

For external host, there is a contention among different queues (vhosts) when packets are processed in tun/bridge, unless I implement MQ TX for macvtap (tun/bridge?). So my testing shows a small improvement (1 to 1.5% average) in BW and a rise in SD (between 10-15%). For remote host, I think tun/macvtap needs MQ TX support?

Confused. I thought this *is* with a multiqueue tun/macvtap? bridge does not do any queueing AFAIK ...
I think we need to fix the contention. With migration what was guest to host a minute ago might become guest to external now ...

Quoted 19 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?

Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.

If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.

I will get a test run with and without binding and post the results later today.

Thanks,

- KK

#15 lore raw
Quoted 14 lines

"Michael S. Tsirkin" <[email protected]>

Quoted 11 lines
Quoted 4 lines

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

For external host, there is a contention among different queues (vhosts) when packets are processed in tun/bridge, unless I implement MQ TX for macvtap (tun/bridge?). So my testing shows a small improvement (1 to 1.5% average) in BW and a rise in SD (between 10-15%). For remote host, I think tun/macvtap needs MQ TX support?

Quoted 4 lines

Confused. I thought this *is* with a multiqueue tun/macvtap? bridge does not do any queueing AFAIK ...
I think we need to fix the contention. With migration what was guest to host a minute ago might become guest to external now ...

Macvtap RX is MQ but not TX. I don't think MQ TX support is required for macvtap, though. Is it enough for existing macvtap sendmsg to work, since it calls dev_queue_xmit which selects the txq for the outgoing device?

Thanks,

- KK

#16 lore raw
Quoted 19 lines · On Thu, Oct 28, 2010 at 11:42:05AM +0530, Krishna Kumar2 wrote:
Quoted 14 lines

"Michael S. Tsirkin" <[email protected]>

Quoted 11 lines
Quoted 4 lines

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

For external host, there is a contention among different queues (vhosts) when packets are processed in tun/bridge, unless I implement MQ TX for macvtap (tun/bridge?). So my testing shows a small improvement (1 to 1.5% average) in BW and a rise in SD (between 10-15%). For remote host, I think tun/macvtap needs MQ TX support?

Quoted 4 lines

Confused. I thought this *is* with a multiqueue tun/macvtap? bridge does not do any queueing AFAIK ...
I think we need to fix the contention. With migration what was guest to host a minute ago might become guest to external now ...

Quoted 8 lines

Macvtap RX is MQ but not TX. I don't think MQ TX support is required for macvtap, though. Is it enough for existing macvtap sendmsg to work, since it calls dev_queue_xmit which selects the txq for the outgoing device?

Thanks,

- KK

I think there would be an issue with using a single poll notifier and contention on send buffer atomic variable.
Is tun different than macvtap? We need to support both long term ...

Signature · 1 line
MST

A message that is not in the archive (sent off-list or not yet received): OFC29C4491.59069AD1-ON652577CA.00170F0D-652577CA.001C76C8@LocalDomain

#18 lore raw
Quoted 9 lines

Krishna Kumar2/India/IBM wrote on 10/28/2010 10:44:14 AM:

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.

If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.

I will get a test run with and without binding and post the results later today.

Correction: The result with binding is is much better for SD/CPU compared to without-binding:

_____________________________________________________
numtxqs=8,vhosts=5, Bind vs No-bind
# BW% CPU% RCPU% SD% RSD%
_____________________________________________________

1     11.25     10.77    1.89     0        -6.06
2     18.66     7.20     7.20    -14.28    -7.40
4     4.24     -1.27     1.56    -2.70     -.98
8     14.91    -3.79     5.46    -12.19    -3.76
16    12.32    -8.67     4.63    -35.97    -26.66
24    11.68    -7.83     5.10    -40.73    -32.37
32    13.09    -10.51    6.57    -51.52    -42.28
40    11.04    -4.12     11.23   -50.69    -42.81
48    8.61     -10.30    6.04    -62.38    -55.54
64    7.55     -6.05     6.41    -61.20    -56.04
80    8.74     -11.45    6.29    -72.65    -67.17
96    9.84     -6.01     9.87    -69.89    -64.78
128   5.57     -6.23     8.99    -75.03    -70.97

_____________________________________________________
BW: 10.4%, CPU/RCPU: -7.4%,7.7%, SD: -70.5%,-65.7%

Notes:
1. All my test results earlier was binding vhost to cpus 0-3 for both org and new kernel.
2. I am not using MST's use_mq patch, only mainline kernel. However, I reported earlier that I got better results with that patch. The result for MQ vs MQ+use_mm patch (from my earlier mail):

BW: 0 CPU/RCPU: -4.2,-6.1 SD/RSD: -13.1,-15.6

Thanks,

- KK

#19 lore raw
Quoted 12 lines · On Thu, Oct 28, 2010 at 12:48:57PM +0530, Krishna Kumar2 wrote:
Quoted 12 lines

Krishna Kumar2/India/IBM wrote on 10/28/2010 10:44:14 AM:

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?

Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.

I will get a test run with and without binding and post the results later today.

Correction: The result with binding is is much better for SD/CPU compared to without-binding:

Can you pls ty finding out why that is? Is some thread bouncing between CPUs? Does a wrong numa node get picked up?
In practice users are very unlikely to pin threads to CPUs.

Signature · 1 line
MST
#20 lore raw
Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · On Thu, Oct 28, 2010 at 12:48:57PM +0530, Krishna Kumar2 wrote:
Quoted 1 line

Krishna Kumar2/India/IBM wrote on 10/28/2010 10:44:14 AM:

In practice users are very unlikely to pin threads to CPUs.

I may be misunderstanding what you're referring to. It caught my attention since I'm working on a configuration to do what you say is unlikely, so I'll chime in for what it's worth.

An option in Vyatta allows assigning CPU affinity to network adapters, since apparently seperate L2 caches can have a significant impact on throughput.

Although much of their focus seems to be on commercial virtualization platforms, I do see quite a few forum posts with regard to KVM. Mabye this still qualifies as an edge case, but as for virtualized routing theirs seems to offer the most functionality.

http://www.vyatta.org/forum/viewtopic.php?t=2697

-cb

#21 lore raw
In reply to Krishna Kumar2
Quoted 12 lines · On Thu, Oct 28, 2010 at 12:48:57PM +0530, Krishna Kumar2 wrote:
Quoted 12 lines

Krishna Kumar2/India/IBM wrote on 10/28/2010 10:44:14 AM:

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Results for UDP BW tests (unidirectional, sum across 3 iterations, each iteration of 45 seconds, default netperf, vhosts bound to cpus 0-3; no other tuning):

Is binding vhost threads to CPUs really required?
What happens if we let the scheduler do its job?

Nothing drastic, I remember BW% and SD% both improved a bit as a result of binding.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

If there's a significant improvement this would mean that we need to rethink the vhost-net interaction with the scheduler.

I will get a test run with and without binding and post the results later today.

Correction: The result with binding is is much better for SD/CPU compared to without-binding:

Something that was suggested to me off-list is trying to set smp affinity for NIC: in host to guest case probably virtio-net, for external to guest the host NIC as well.

Signature · 1 line
MST
#22 lore raw
In reply to Krishna Kumar2
Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line

Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:

Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.

Some more test results:
_____________________________________________________

         Host->Guest BW (numtxqs=2)

# BW% CPU% RCPU% SD% RSD%
_____________________________________________________

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

Signature · 1 line
MST
#23 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 10/26/2010 02:27:09 PM:

Quoted 14 lines

Re: [v3 RFC PATCH 0/4] Implement multiqueue virtio-net

Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line

Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:

Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.

Some more test results:
_____________________________________________________

         Host->Guest BW (numtxqs=2)

# BW% CPU% RCPU% SD% RSD%
_____________________________________________________

Quoted 4 lines

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

I had to make a few changes to qemu (and a minor change in macvtap driver) to get multiple TXQ support using macvtap working. The NIC is a ixgbe card.

__________________________________________________________________________

            Org vs New (I/O: 512 bytes, #numtxqs=2, #vhosts=3)
#      BW1     BW2 (%)       SD1    SD2 (%)        RSD1    RSD2 (%)
__________________________________________________________________________
1      14367   13142 (-8.5)  56     62 (10.7)      8        8 (0)
2      3652    3855 (5.5)    37     35 (-5.4)      7        6 (-14.2)
4      12529   12059 (-3.7)  65     77 (18.4)      35       35 (0)
8      13912   14668 (5.4)   288    332 (15.2)     175      184 (5.1)
16     13433   14455 (7.6)   1218   1321 (8.4)     920      943 (2.5)
24     12750   13477 (5.7)   2876   2985 (3.7)     2514     2348 (-6.6)
32     11729   12632 (7.6)   5299   5332 (.6)      4934     4497 (-8.8)
40     11061   11923 (7.7)   8482   8364 (-1.3)    8374     7495 (-10.4)
48     10624   11267 (6.0)   12329  12258 (-.5)    12762    11538 (-9.5)
64     10524   10596 (.6)    21689  22859 (5.3)    23626    22403 (-5.1)
80     9856    10284 (4.3)   35769  36313 (1.5)    39932    36419 (-8.7)
96     9691    10075 (3.9)   52357  52259 (-.1)    58676    53463 (-8.8)
128    9351    9794 (4.7)    114707 94275 (-17.8)  114050   97337 (-14.6)

__________________________________________________________________________
Avg: BW: (3.3) SD: (-7.3) RSD: (-11.0)

__________________________________________________________________________

            Org vs New (I/O: 1K, #numtxqs=8, #vhosts=5)
#      BW1      BW2 (%)       SD1   SD2 (%)        RSD1   RSD2 (%)
__________________________________________________________________________
1      16509    15985 (-3.1)  45    47 (4.4)       7       7 (0)
2      6963     4499 (-35.3)  17    51 (200.0)     7       7 (0)
4      12932    11080 (-14.3) 49    74 (51.0)      35      35 (0)
8      13878    14095 (1.5)   223   292 (30.9)     175     181 (3.4)
16     13440    13698 (1.9)   980   1131 (15.4)    926     942 (1.7)
24     12680    12927 (1.9)   2387  2463 (3.1)     2526    2342 (-7.2)
32     11714    12261 (4.6)   4506  4486 (-.4)     4941    4463 (-9.6)
40     11059    11651 (5.3)   7244  7081 (-2.2)    8349    7437 (-10.9)
48     10580    11095 (4.8)   10811 10500 (-2.8)   12809   11403 (-10.9)
64     10569    10566 (0)     19194 19270 (.3)     23648   21717 (-8.1)
80     9827     10753 (9.4)   31668 29425 (-7.0)   39991   33824 (-15.4)
96     10043    10150 (1.0)   45352 44227 (-2.4)   57766   51131 (-11.4)
128    9360     9979 (6.6)    92058 79198 (-13.9)  114381  92873 (-18.8)

__________________________________________________________________________
Avg: BW: (-.5) SD: (-7.5) RSD: (-14.7)

Is there anything else you would like me to test/change, or shall I submit the next version (with the above macvtap changes)?

Thanks,

- KK

#24 lore raw
Quoted 65 lines · On Tue, Nov 09, 2010 at 10:08:21AM +0530, Krishna Kumar2 wrote:

"Michael S. Tsirkin" <[email protected]> wrote on 10/26/2010 02:27:09 PM:

Quoted 14 lines

Re: [v3 RFC PATCH 0/4] Implement multiqueue virtio-net

Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line

Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:

Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.

Some more test results:
_____________________________________________________

         Host->Guest BW (numtxqs=2)

# BW% CPU% RCPU% SD% RSD%
_____________________________________________________

Quoted 4 lines

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

I had to make a few changes to qemu (and a minor change in macvtap driver) to get multiple TXQ support using macvtap working. The NIC is a ixgbe card.

__________________________________________________________________________

            Org vs New (I/O: 512 bytes, #numtxqs=2, #vhosts=3)
#      BW1     BW2 (%)       SD1    SD2 (%)        RSD1    RSD2 (%)
__________________________________________________________________________
1      14367   13142 (-8.5)  56     62 (10.7)      8        8 (0)
2      3652    3855 (5.5)    37     35 (-5.4)      7        6 (-14.2)
4      12529   12059 (-3.7)  65     77 (18.4)      35       35 (0)
8      13912   14668 (5.4)   288    332 (15.2)     175      184 (5.1)
16     13433   14455 (7.6)   1218   1321 (8.4)     920      943 (2.5)
24     12750   13477 (5.7)   2876   2985 (3.7)     2514     2348 (-6.6)
32     11729   12632 (7.6)   5299   5332 (.6)      4934     4497 (-8.8)
40     11061   11923 (7.7)   8482   8364 (-1.3)    8374     7495 (-10.4)
48     10624   11267 (6.0)   12329  12258 (-.5)    12762    11538 (-9.5)
64     10524   10596 (.6)    21689  22859 (5.3)    23626    22403 (-5.1)
80     9856    10284 (4.3)   35769  36313 (1.5)    39932    36419 (-8.7)
96     9691    10075 (3.9)   52357  52259 (-.1)    58676    53463 (-8.8)
128    9351    9794 (4.7)    114707 94275 (-17.8)  114050   97337 (-14.6)

__________________________________________________________________________
Avg: BW: (3.3) SD: (-7.3) RSD: (-11.0)

__________________________________________________________________________

            Org vs New (I/O: 1K, #numtxqs=8, #vhosts=5)
#      BW1      BW2 (%)       SD1   SD2 (%)        RSD1   RSD2 (%)
__________________________________________________________________________
1      16509    15985 (-3.1)  45    47 (4.4)       7       7 (0)
2      6963     4499 (-35.3)  17    51 (200.0)     7       7 (0)
4      12932    11080 (-14.3) 49    74 (51.0)      35      35 (0)
8      13878    14095 (1.5)   223   292 (30.9)     175     181 (3.4)
16     13440    13698 (1.9)   980   1131 (15.4)    926     942 (1.7)
24     12680    12927 (1.9)   2387  2463 (3.1)     2526    2342 (-7.2)
32     11714    12261 (4.6)   4506  4486 (-.4)     4941    4463 (-9.6)
40     11059    11651 (5.3)   7244  7081 (-2.2)    8349    7437 (-10.9)
48     10580    11095 (4.8)   10811 10500 (-2.8)   12809   11403 (-10.9)
64     10569    10566 (0)     19194 19270 (.3)     23648   21717 (-8.1)
80     9827     10753 (9.4)   31668 29425 (-7.0)   39991   33824 (-15.4)
96     10043    10150 (1.0)   45352 44227 (-2.4)   57766   51131 (-11.4)
128    9360     9979 (6.6)    92058 79198 (-13.9)  114381  92873 (-18.8)

__________________________________________________________________________
Avg: BW: (-.5) SD: (-7.5) RSD: (-14.7)

Quoted 6 lines

Is there anything else you would like me to test/change, or shall I submit the next version (with the above macvtap changes)?

Thanks,

- KK

Something strange here, right?
1. You are consistently getting >10G/s here, and even with a single stream?
2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.

Is your card MQ?

Signature · 1 line
MST
#25 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 06:52:39 PM:

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines

Re: [v3 RFC PATCH 0/4] Implement multiqueue virtio-net

Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line

Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:

Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.

Some more test results:
_____________________________________________________

         Host->Guest BW (numtxqs=2)

# BW% CPU% RCPU% SD% RSD%
_____________________________________________________

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

Quoted 5 lines
Quoted 5 lines

I had to make a few changes to qemu (and a minor change in macvtap driver) to get multiple TXQ support using macvtap working. The NIC is a ixgbe card.

__________________________________________________________________________

Quoted 3 lines
Quoted 3 lines
            Org vs New (I/O: 512 bytes, #numtxqs=2, #vhosts=3)

# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)

__________________________________________________________________________

Quoted 8 lines
Quoted 8 lines
1      14367   13142 (-8.5)  56     62 (10.7)      8        8 (0)
2      3652    3855 (5.5)    37     35 (-5.4)      7        6 (-14.2)
4      12529   12059 (-3.7)  65     77 (18.4)      35       35 (0)
8      13912   14668 (5.4)   288    332 (15.2)     175      184 (5.1)
16     13433   14455 (7.6)   1218   1321 (8.4)     920      943 (2.5)
24     12750   13477 (5.7)   2876   2985 (3.7)     2514     2348 (-6.6)
32     11729   12632 (7.6)   5299   5332 (.6)      4934     4497 (-8.8)
40     11061   11923 (7.7)   8482   8364 (-1.3)    8374     7495

(-10.4)

Quoted 1 line
Quoted 1 line

48 10624 11267 (6.0) 12329 12258 (-.5) 12762 11538

(-9.5)

Quoted 1 line
Quoted 1 line

64 10524 10596 (.6) 21689 22859 (5.3) 23626 22403

(-5.1)

Quoted 1 line
Quoted 1 line

80 9856 10284 (4.3) 35769 36313 (1.5) 39932 36419

(-8.7)

Quoted 1 line
Quoted 1 line

96 9691 10075 (3.9) 52357 52259 (-.1) 58676 53463

(-8.8)

Quoted 1 line
Quoted 1 line

128 9351 9794 (4.7) 114707 94275 (-17.8) 114050 97337

(-14.6)

Quoted 1 line
Quoted 1 line

__________________________________________________________________________

Quoted 3 lines
Quoted 3 lines

Avg: BW: (3.3) SD: (-7.3) RSD: (-11.0)

__________________________________________________________________________

Quoted 3 lines
Quoted 3 lines
            Org vs New (I/O: 1K, #numtxqs=8, #vhosts=5)

# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)

__________________________________________________________________________

Quoted 9 lines
Quoted 9 lines
1      16509    15985 (-3.1)  45    47 (4.4)       7       7 (0)
2      6963     4499 (-35.3)  17    51 (200.0)     7       7 (0)
4      12932    11080 (-14.3) 49    74 (51.0)      35      35 (0)
8      13878    14095 (1.5)   223   292 (30.9)     175     181 (3.4)
16     13440    13698 (1.9)   980   1131 (15.4)    926     942 (1.7)
24     12680    12927 (1.9)   2387  2463 (3.1)     2526    2342 (-7.2)
32     11714    12261 (4.6)   4506  4486 (-.4)     4941    4463 (-9.6)
40     11059    11651 (5.3)   7244  7081 (-2.2)    8349    7437 (-10.9)
48     10580    11095 (4.8)   10811 10500 (-2.8)   12809   11403

(-10.9)

Quoted 2 lines
Quoted 2 lines
64     10569    10566 (0)     19194 19270 (.3)     23648   21717 (-8.1)
80     9827     10753 (9.4)   31668 29425 (-7.0)   39991   33824

(-15.4)

Quoted 1 line
Quoted 1 line

96 10043 10150 (1.0) 45352 44227 (-2.4) 57766 51131

(-11.4)

Quoted 1 line
Quoted 1 line

128 9360 9979 (6.6) 92058 79198 (-13.9) 114381 92873

(-18.8)

Quoted 1 line
Quoted 1 line

__________________________________________________________________________

Quoted 11 lines
Quoted 8 lines

Avg: BW: (-.5) SD: (-7.5) RSD: (-14.7)

Is there anything else you would like me to test/change, or shall I submit the next version (with the above macvtap changes)?

Thanks,

- KK

Something strange here, right?
1. You are consistently getting >10G/s here, and even with a single

stream?

Sorry, I should have mentioned this though I had stated in my earlier mails. Each test result has two iterations, each of 60 seconds, except when #netperfs is 1 for which I do 10 iteration (sum across 10 iterations). I started doing many more iterations for 1 netperf after finding the issue earlier with single stream. So the BW is only 4.5-7 Gbps.

Quoted 3 lines

2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.

(doubling happens consistently for guest -> host, but never for remote host) I tried 512/txqs=2 and 1024/txqs=8 to get a varied testing scenario. In first case, there is a slight improvement in BW and good reduction in SD. In the second case, only SD improves (though BW drops for 2 stream for some reason). In both cases, BW and SD improves as the number of sessions increase.

Quoted 1 line

Is your card MQ?

Yes, the card is MQ. ixgbe 10g card.

Thanks,

- KK

#26 lore raw
Quoted 88 lines · On Tue, Nov 09, 2010 at 08:58:44PM +0530, Krishna Kumar2 wrote:

"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 06:52:39 PM:

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines

Re: [v3 RFC PATCH 0/4] Implement multiqueue virtio-net

Quoted 10 lines · On Mon, Oct 25, 2010 at 09:20:38PM +0530, Krishna Kumar2 wrote:
Quoted 1 line

Krishna Kumar2/India/IBM@IBMIN wrote on 10/20/2010 02:24:52 PM:

Any feedback, comments, objections, issues or bugs about the patches? Please let me know if something needs to be done.

Some more test results:
_____________________________________________________

         Host->Guest BW (numtxqs=2)

# BW% CPU% RCPU% SD% RSD%
_____________________________________________________

I think we discussed the need for external to guest testing over 10G. For large messages we should not see any change but you should be able to get better numbers for small messages assuming a MQ NIC card.

Quoted 5 lines
Quoted 5 lines

I had to make a few changes to qemu (and a minor change in macvtap driver) to get multiple TXQ support using macvtap working. The NIC is a ixgbe card.

__________________________________________________________________________

Quoted 3 lines
Quoted 3 lines
            Org vs New (I/O: 512 bytes, #numtxqs=2, #vhosts=3)

# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)

__________________________________________________________________________

Quoted 8 lines
Quoted 8 lines
1      14367   13142 (-8.5)  56     62 (10.7)      8        8 (0)
2      3652    3855 (5.5)    37     35 (-5.4)      7        6 (-14.2)
4      12529   12059 (-3.7)  65     77 (18.4)      35       35 (0)
8      13912   14668 (5.4)   288    332 (15.2)     175      184 (5.1)
16     13433   14455 (7.6)   1218   1321 (8.4)     920      943 (2.5)
24     12750   13477 (5.7)   2876   2985 (3.7)     2514     2348 (-6.6)
32     11729   12632 (7.6)   5299   5332 (.6)      4934     4497 (-8.8)
40     11061   11923 (7.7)   8482   8364 (-1.3)    8374     7495

(-10.4)

Quoted 1 line
Quoted 1 line

48 10624 11267 (6.0) 12329 12258 (-.5) 12762 11538

(-9.5)

Quoted 1 line
Quoted 1 line

64 10524 10596 (.6) 21689 22859 (5.3) 23626 22403

(-5.1)

Quoted 1 line
Quoted 1 line

80 9856 10284 (4.3) 35769 36313 (1.5) 39932 36419

(-8.7)

Quoted 1 line
Quoted 1 line

96 9691 10075 (3.9) 52357 52259 (-.1) 58676 53463

(-8.8)

Quoted 1 line
Quoted 1 line

128 9351 9794 (4.7) 114707 94275 (-17.8) 114050 97337

(-14.6)

Quoted 1 line
Quoted 1 line

__________________________________________________________________________

Quoted 3 lines
Quoted 3 lines

Avg: BW: (3.3) SD: (-7.3) RSD: (-11.0)

__________________________________________________________________________

Quoted 3 lines
Quoted 3 lines
            Org vs New (I/O: 1K, #numtxqs=8, #vhosts=5)

# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)

__________________________________________________________________________

Quoted 9 lines
Quoted 9 lines
1      16509    15985 (-3.1)  45    47 (4.4)       7       7 (0)
2      6963     4499 (-35.3)  17    51 (200.0)     7       7 (0)
4      12932    11080 (-14.3) 49    74 (51.0)      35      35 (0)
8      13878    14095 (1.5)   223   292 (30.9)     175     181 (3.4)
16     13440    13698 (1.9)   980   1131 (15.4)    926     942 (1.7)
24     12680    12927 (1.9)   2387  2463 (3.1)     2526    2342 (-7.2)
32     11714    12261 (4.6)   4506  4486 (-.4)     4941    4463 (-9.6)
40     11059    11651 (5.3)   7244  7081 (-2.2)    8349    7437 (-10.9)
48     10580    11095 (4.8)   10811 10500 (-2.8)   12809   11403

(-10.9)

Quoted 2 lines
Quoted 2 lines
64     10569    10566 (0)     19194 19270 (.3)     23648   21717 (-8.1)
80     9827     10753 (9.4)   31668 29425 (-7.0)   39991   33824

(-15.4)

Quoted 1 line
Quoted 1 line

96 10043 10150 (1.0) 45352 44227 (-2.4) 57766 51131

(-11.4)

Quoted 1 line
Quoted 1 line

128 9360 9979 (6.6) 92058 79198 (-13.9) 114381 92873

(-18.8)

Quoted 1 line
Quoted 1 line

__________________________________________________________________________

Quoted 9 lines
Quoted 8 lines

Avg: BW: (-.5) SD: (-7.5) RSD: (-14.7)

Is there anything else you would like me to test/change, or shall I submit the next version (with the above macvtap changes)?

Thanks,

- KK

Quoted 8 lines
Quoted 2 lines

Something strange here, right?
1. You are consistently getting >10G/s here, and even with a single

stream?

Sorry, I should have mentioned this though I had stated in my earlier mails. Each test result has two iterations, each of 60 seconds, except when #netperfs is 1 for which I do 10 iteration (sum across 10 iterations).

So need to divide the number by 10?

Quoted 8 lines

I started doing many more iterations
for 1 netperf after finding the issue earlier with single stream. So the BW is only 4.5-7 Gbps.

Quoted 3 lines

2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.

Quoted 6 lines

(doubling happens consistently for guest -> host, but never for remote host) I tried 512/txqs=2 and 1024/txqs=8 to get a varied testing scenario. In first case, there is a slight improvement in BW and good reduction in SD. In the second case, only SD improves (though BW drops for 2 stream for some reason). In both cases, BW and SD improves as the number of sessions increase.

I guess this is another indication that something's wrong. We are quite far from line rate, the fact BW does not scale means there's some contention in the code.

Quoted 7 lines
Quoted 1 line

Is your card MQ?

Yes, the card is MQ. ixgbe 10g card.

Thanks,

- KK

#27 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 09:03:25 PM:

Quoted 10 lines
Quoted 8 lines
Quoted 2 lines

Something strange here, right?
1. You are consistently getting >10G/s here, and even with a single

stream?

Sorry, I should have mentioned this though I had stated in my earlier mails. Each test result has two iterations, each of 60 seconds, except when #netperfs is 1 for which I do 10 iteration (sum across 10 iterations).

So need to divide the number by 10?

Yes, that is what I get with 512/1K macvtap I/O size :)

Quoted 8 lines
Quoted 8 lines

I started doing many more iterations
for 1 netperf after finding the issue earlier with single stream. So the BW is only 4.5-7 Gbps.

Quoted 3 lines

2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.

Quoted 8 lines
Quoted 6 lines

(doubling happens consistently for guest -> host, but never for remote host) I tried 512/txqs=2 and 1024/txqs=8 to get a varied testing scenario. In first case, there is a slight improvement in BW and good reduction in SD. In the second case, only SD improves (though BW drops for 2 stream for some reason). In both cases, BW and SD improves as the number of sessions increase.

I guess this is another indication that something's wrong.

The patch - both virtio-net and vhost-net, doesn't have any locking/mutex's/ or any synchronization method. Guest -> host performance improvement of upto 100% shows the patch is not doing anything wrong.

Quoted 2 lines

We are quite far from line rate, the fact BW does not scale means there's some contention in the code.

Attaining line speed with macvtap seems to be a generic issue and unrelated to my patch specifically. IMHO if there is nothing wrong in the code (review) and is accepted, it will benefit as others can also help to find what needs to be implemented in vhost/macvtap/qemu to get line speed for guest->remote-host.

PS: bare-metal performance for host->remote-host is also

    2.7 Gbps and 2.8 Gbps for 512/1024 for the same card.

Thanks,

- KK

#28 lore raw
Quoted 30 lines · On Tue, Nov 09, 2010 at 10:54:57PM +0530, Krishna Kumar2 wrote:

"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 09:03:25 PM:

Quoted 10 lines
Quoted 8 lines
Quoted 2 lines

Something strange here, right?
1. You are consistently getting >10G/s here, and even with a single

stream?

Sorry, I should have mentioned this though I had stated in my earlier mails. Each test result has two iterations, each of 60 seconds, except when #netperfs is 1 for which I do 10 iteration (sum across 10 iterations).

So need to divide the number by 10?

Yes, that is what I get with 512/1K macvtap I/O size :)

Quoted 15 lines
Quoted 14 lines

I started doing many more iterations
for 1 netperf after finding the issue earlier with single stream. So the BW is only 4.5-7 Gbps.

Quoted 3 lines

2. With 2 streams, is where we get < 10G/s originally. Instead of doubling that we get a marginal improvement with 2 queues and about 30% worse with 1 queue.

(doubling happens consistently for guest -> host, but never for remote host) I tried 512/txqs=2 and 1024/txqs=8 to get a varied testing scenario. In first case, there is a slight improvement in BW and good reduction in SD. In the second case, only SD improves (though BW drops for 2 stream for some reason). In both cases, BW and SD improves as the number of sessions increase.

Quoted 6 lines
Quoted 1 line

I guess this is another indication that something's wrong.

The patch - both virtio-net and vhost-net, doesn't have any locking/mutex's/ or any synchronization method. Guest -> host performance improvement of upto 100% shows the patch is not doing anything wrong.

My concern is this: we don't seem to do anything in tap or macvtap to help packets from separate virtio queues get to separate queues in the hardware device and to avoid reordering when we do this.

- skb_tx_hash calculation will get different results
- hash math that e.g. tcp does will run on guest and seems to be discarded

etc

Maybe it's as simple as some tap/macvtap ioctls to set up the queue number in skbs. Or maybe we need to pass the skb hash from guest to host. It's this last option that should make us especially cautios as it'll affect guest/host interface.

Also see d5a9e24afb4ab38110ebb777588ea0bd0eacbd0a: if we have hardware which records an RX queue, it appears important to pass that info to guest and to use that in selecting the TX queue. Of course we won't see this in netperf runs but this needs to be given thought too - supporting this seems to suggest either sticking the hash in the virtio net header for both tx and rx, or using multiplease RX queues.

Quoted 8 lines
Quoted 2 lines

We are quite far from line rate, the fact BW does not scale means there's some contention in the code.

Attaining line speed with macvtap seems to be a generic issue and unrelated to my patch specifically. IMHO if there is nothing wrong in the code (review) and is accepted, it will benefit as others can also help to find what needs to be implemented in vhost/macvtap/qemu to get line speed for guest->remote-host.

No problem, I will queue these patches in some branch
to help enable cooperation, as well as help you
iterate with incremental patches instead of resending it all each time.

Quoted 4 lines

PS: bare-metal performance for host->remote-host is also

    2.7 Gbps and 2.8 Gbps for 512/1024 for the same card.

Thanks,

You mean native linux BW does not scale for your host with

# of connections either? I guess this just means need another
setup for testing?
Quoted 1 line

- KK

A message that is not in the archive (sent off-list or not yet received): OF24E08752.2087FFA4-ON652577D6.00532DF1-652577D6.0054B291@LocalDomain

#30 lore raw

MQ performance on other cards (cxgb3)

I had sent this mail to Michael last week - he agrees that I should share this information on the list:

On latest net-next-2.6, virtio-net (guest->host) results are:
______________________________________________________________

                         SQ vs MQ (#txqs=8)
#      BW1      BW2 (%)          CPU1     CPU2 (%)   RCPU1   RCPU2 (%)
_______________________________________________________________
1      105774  112256 (6.1)   257      255 (-.7)     532     549 (3.1)
2      20842   30674 (47.1)   107      150 (40.1)    208     279 (34.1)
4      22500   31953 (42.0)   241      409 (69.7)    467     619 (32.5)
8      22416   44507 (98.5)   477      1039 (117.8)  960     1459 (51.9)
16     22605   45372 (100.7)  905      2060 (127.6)  1895    2962 (56.3)
24     23192   44201 (90.5)   1360     3028 (122.6)  2833    4437 (56.6)
32     23158   43394 (87.3)   1811     3957 (118.4)  3770    5936 (57.4)
40     23322   42550 (82.4)   2276     4986 (119.0)  4711    7417 (57.4)
48     23564   41931 (77.9)   2757     5966 (116.3)  5653    8896 (57.3)
64     23949   41092 (71.5)   3788     7898 (108.5)  7609    11826 (55.4)
80     23256   41343 (77.7)   4597     9887 (115.0)  9503    14801 (55.7)
96     23310   40645 (74.3)   5588     11758 (110.4) 11381   17761 (56.0)
128    24095   41082 (70.5)   7587     15574 (105.2) 15029   23716 (57.8)

______________________________________________________________
Avg: BW: (58.3) CPU: (110.8) RCPU: (55.9)

It's true that average CPU% on guest is almost double that of the BW improvement. But I don't think this is due to the patch (driver does no synchronization, etc). To compare MQ vs SQ on a 10G card, I ran the same test from host to remote host across cxgb3. The results are somewhat similar:

(I changed cxgb_open on the client system to:

	netif_set_real_num_tx_queues(dev, 1);
	err = netif_set_real_num_rx_queues(dev, 1);

to simulate single queue (SQ))
_____________________________________
cxgb3 SQ vs cxgb3 MQ
# BW1 BW2 (%) CPU1 CPU2 (%)
_____________________________________

1     8301    8315 (.1)        5         4.66 (-6.6)
2     9395    9380 (-.1)      16        16 (0)
4     9411    9414 (0)        33        26 (-21.2)
8     9411    9398 (-.1)      60        62 (3.3)
16   9412    9413 (0)        116      117 (.8)
24   9442    9963 (5.5)     179      198 (10.6)
32   10031  10025 (0)       230     249 (8.2)
40   9953    10024 (.7)      300     312 (4.0)
48   10002  10015 (.1)      351     376 (7.1)
64   10022  10024 (0)       494     515 (4.2)
80   8894    10011 (12.5)   537    630 (17.3)
96   8465    9907 (17.0)     612    749 (22.3)
128  7541   9617 (27.5)     760    989 (30.1)

_____________________________________
Avg: BW: (3.8) CPU: (14.8)

(Each case runs runs once for 60 secs)

The BW increased modestly but CPU increased much more. I assume the change I made above to convert the driver from MQ to SQ is not incorrect.

Thanks,

- KK

#31 lore raw
In reply to Krishna Kumar
Quoted 3 lines · Krishna Kumar

Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.

Hi Krishna,

I have a few questions about the results below:

1. Are the (%) comparisons between non-mq and mq virtio?
2. Was UDP or TCP used?
3. What was the transmit size (-m option to netperf)?

Also, I'm interested to know what the status of these patches is. Are you planing a fresh series?

Quoted 128 lines

Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:

       struct send_queue {
               struct virtqueue *svq;
               struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
       } ____cacheline_aligned_in_smp;

and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.

Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:

        vhost=on                   ->   #txqueues = 1
        vhost=on,mq=on             ->   #txqueues = 4
        vhost=on,mq=on,numtxqs=2   ->   #txqueues = 2
        vhost=on,mq=on,numtxqs=8   ->   #txqueues = 8
                   Performance (guest -> local host):
                   -----------------------------------

System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).

______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1          4.46    -1.96     .19     -12.50   -6.06
2          4.93    -1.16    2.10      0       -2.38
4          46.17    64.77   33.72     19.51   -2.48
8          47.89    70.00   36.23     41.46    13.35
16         48.97    80.44   40.67     21.11   -5.46
24         49.03    78.78   41.22     20.51   -4.78
32         51.11    77.15   42.42     15.81   -6.87
40         51.60    71.65   42.43     9.75    -8.94
48         50.10    69.55   42.85     11.80   -5.81
64         46.24    68.42   42.67     14.18   -3.28
80         46.37    63.13   41.62     7.43    -6.73
96         46.40    63.31   42.20     9.36    -4.78
128        50.43    62.79   42.16     13.11   -1.23

________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%

______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -.76    -1.56     2.33      0        3.03
2           17.41    11.11    11.41     0       -4.76
4           42.12    55.11    30.20     19.51    .62
8           54.69    80.00    39.22     24.39    -3.88
16          54.77    81.62    40.89     20.34    -6.58
24          54.66    79.68    41.57     15.49    -8.99
32          54.92    76.82    41.79     17.59    -5.70
40          51.79    68.56    40.53     15.31    -3.87
48          51.72    66.40    40.84     9.72     -7.13
64          51.11    63.94    41.10     5.93     -8.82
80          46.51    59.50    39.80     9.33     -4.18
96          47.72    57.75    39.84     4.20     -7.62
128         54.35    58.95    40.66     3.24     -8.63

________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%

______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -1.43    -3.52    1.55      0          3.03
2           33.09     21.63   20.12    -10.00     -9.52
4           67.17     94.60   44.28     19.51     -11.80
8           75.72     108.14  49.15     25.00     -10.71
16          80.34     101.77  52.94     25.93     -4.49
24          70.84     93.12   43.62     27.63     -5.03
32          69.01     94.16   47.33     29.68     -1.51
40          58.56     63.47   25.91    -3.92      -25.85
48          61.16     74.70   34.88     .89       -22.08
64          54.37     69.09   26.80    -6.68      -30.04
80          36.22     22.73   -2.97    -8.25      -27.23
96          41.51     50.59   13.24     9.84      -16.77
128         48.98     38.15   6.41     -.33       -22.80

________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%

______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________

1            7.62     -38.03   -26.26  -50.00   -33.33
2            28.95     20.46    21.62   0       -7.14
4            84.05     60.79    45.74  -2.43    -12.42
8            86.43     79.57    50.32   15.85   -3.10
16           88.63     99.48    58.17   9.47    -13.10
24           74.65     80.87    41.99  -1.81    -22.89
32           63.86     59.21    23.58  -18.13   -36.37
40           64.79     60.53    22.23  -15.77   -35.84
48           49.68     26.93    .51    -36.40   -49.61
64           54.69     36.50    5.41   -26.59   -43.23
80           45.06     12.72   -13.25  -37.79   -52.08
96           40.21    -3.16    -24.53  -39.92   -52.97
128          36.33    -33.19   -43.66  -5.68    -20.49

________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%

---
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#32 lore raw

Simon Horman <[email protected]> wrote on 02/22/2011 01:17:09 PM:

Hi Simon,

Quoted 3 lines

I have a few questions about the results below:

1. Are the (%) comparisons between non-mq and mq virtio?

Yes - mainline kernel with transmit-only MQ patch.

Quoted 1 line

2. Was UDP or TCP used?

TCP. I had done some initial testing on UDP, but don't have the results now as it is really old. But I will be running it again.

Quoted 1 line

3. What was the transmit size (-m option to netperf)?

I didn't use the -m option, so it defaults to 16K. The script does:

netperf -t TCP_STREAM -c -C -l 60 -H $SERVER

Quoted 2 lines

Also, I'm interested to know what the status of these patches is. Are you planing a fresh series?

Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?

The TX-only patch helped the guest TX path but didn't help host->guest much (as tested using TCP_MAERTS from the guest). But with the TX+RX patch, both directions are getting improvements. Remote testing is still to be done.

Thanks,

- KK

Quoted 118 lines
Quoted 118 lines

Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:

       struct send_queue {
               struct virtqueue *svq;
               struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
       } ____cacheline_aligned_in_smp;

and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.

Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:

        vhost=on                   ->   #txqueues = 1
        vhost=on,mq=on             ->   #txqueues = 4
        vhost=on,mq=on,numtxqs=2   ->   #txqueues = 2
        vhost=on,mq=on,numtxqs=8   ->   #txqueues = 8
                   Performance (guest -> local host):
                   -----------------------------------

System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).

______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1          4.46    -1.96     .19     -12.50   -6.06
2          4.93    -1.16    2.10      0       -2.38
4          46.17    64.77   33.72     19.51   -2.48
8          47.89    70.00   36.23     41.46    13.35
16         48.97    80.44   40.67     21.11   -5.46
24         49.03    78.78   41.22     20.51   -4.78
32         51.11    77.15   42.42     15.81   -6.87
40         51.60    71.65   42.43     9.75    -8.94
48         50.10    69.55   42.85     11.80   -5.81
64         46.24    68.42   42.67     14.18   -3.28
80         46.37    63.13   41.62     7.43    -6.73
96         46.40    63.31   42.20     9.36    -4.78
128        50.43    62.79   42.16     13.11   -1.23

________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%

______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -.76    -1.56     2.33      0        3.03
2           17.41    11.11    11.41     0       -4.76
4           42.12    55.11    30.20     19.51    .62
8           54.69    80.00    39.22     24.39    -3.88
16          54.77    81.62    40.89     20.34    -6.58
24          54.66    79.68    41.57     15.49    -8.99
32          54.92    76.82    41.79     17.59    -5.70
40          51.79    68.56    40.53     15.31    -3.87
48          51.72    66.40    40.84     9.72     -7.13
64          51.11    63.94    41.10     5.93     -8.82
80          46.51    59.50    39.80     9.33     -4.18
96          47.72    57.75    39.84     4.20     -7.62
128         54.35    58.95    40.66     3.24     -8.63

________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%

______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -1.43    -3.52    1.55      0          3.03
2           33.09     21.63   20.12    -10.00     -9.52
4           67.17     94.60   44.28     19.51     -11.80
8           75.72     108.14  49.15     25.00     -10.71
16          80.34     101.77  52.94     25.93     -4.49
24          70.84     93.12   43.62     27.63     -5.03
32          69.01     94.16   47.33     29.68     -1.51
40          58.56     63.47   25.91    -3.92      -25.85
48          61.16     74.70   34.88     .89       -22.08
64          54.37     69.09   26.80    -6.68      -30.04
80          36.22     22.73   -2.97    -8.25      -27.23
96          41.51     50.59   13.24     9.84      -16.77
128         48.98     38.15   6.41     -.33       -22.80

________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%

______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________

1            7.62     -38.03   -26.26  -50.00   -33.33
2            28.95     20.46    21.62   0       -7.14
4            84.05     60.79    45.74  -2.43    -12.42
8            86.43     79.57    50.32   15.85   -3.10
16           88.63     99.48    58.17   9.47    -13.10
24           74.65     80.87    41.99  -1.81    -22.89
32           63.86     59.21    23.58  -18.13   -36.37
40           64.79     60.53    22.23  -15.77   -35.84
48           49.68     26.93    .51    -36.40   -49.61
64           54.69     36.50    5.41   -26.59   -43.23
80           45.06     12.72   -13.25  -37.79   -52.08
96           40.21    -3.16    -24.53  -39.92   -52.97
128          36.33    -33.19   -43.66  -5.68    -20.49

________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%

#33 lore raw
Quoted 24 lines · On Wed, Feb 23, 2011 at 10:52:09AM +0530, Krishna Kumar2 wrote:

Simon Horman <[email protected]> wrote on 02/22/2011 01:17:09 PM:

Hi Simon,

Quoted 3 lines

I have a few questions about the results below:

1. Are the (%) comparisons between non-mq and mq virtio?

Yes - mainline kernel with transmit-only MQ patch.

Quoted 1 line

2. Was UDP or TCP used?

TCP. I had done some initial testing on UDP, but don't have the results now as it is really old. But I will be running it again.

Quoted 1 line

3. What was the transmit size (-m option to netperf)?

I didn't use the -m option, so it defaults to 16K. The script does:

netperf -t TCP_STREAM -c -C -l 60 -H $SERVER

Quoted 7 lines
Quoted 2 lines

Also, I'm interested to know what the status of these patches is. Are you planing a fresh series?

Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?

Yes, please do.

Quoted 4 lines

The TX-only patch helped the guest TX path but didn't help host->guest much (as tested using TCP_MAERTS from the guest). But with the TX+RX patch, both directions are getting improvements.

Also, my hope is that with appropriate queue mapping, we might be able to do away with heuristics to detect single stream load that TX only code needs.

Quoted 1 line

Remote testing is still to be done.

Others might be able to help here once you post the patch.

Quoted 122 lines

Thanks,

- KK

Quoted 118 lines
Quoted 118 lines

Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:

       struct send_queue {
               struct virtqueue *svq;
               struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
       } ____cacheline_aligned_in_smp;

and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.

Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:

        vhost=on                   ->   #txqueues = 1
        vhost=on,mq=on             ->   #txqueues = 4
        vhost=on,mq=on,numtxqs=2   ->   #txqueues = 2
        vhost=on,mq=on,numtxqs=8   ->   #txqueues = 8
                   Performance (guest -> local host):
                   -----------------------------------

System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).

______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1          4.46    -1.96     .19     -12.50   -6.06
2          4.93    -1.16    2.10      0       -2.38
4          46.17    64.77   33.72     19.51   -2.48
8          47.89    70.00   36.23     41.46    13.35
16         48.97    80.44   40.67     21.11   -5.46
24         49.03    78.78   41.22     20.51   -4.78
32         51.11    77.15   42.42     15.81   -6.87
40         51.60    71.65   42.43     9.75    -8.94
48         50.10    69.55   42.85     11.80   -5.81
64         46.24    68.42   42.67     14.18   -3.28
80         46.37    63.13   41.62     7.43    -6.73
96         46.40    63.31   42.20     9.36    -4.78
128        50.43    62.79   42.16     13.11   -1.23

________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%

______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -.76    -1.56     2.33      0        3.03
2           17.41    11.11    11.41     0       -4.76
4           42.12    55.11    30.20     19.51    .62
8           54.69    80.00    39.22     24.39    -3.88
16          54.77    81.62    40.89     20.34    -6.58
24          54.66    79.68    41.57     15.49    -8.99
32          54.92    76.82    41.79     17.59    -5.70
40          51.79    68.56    40.53     15.31    -3.87
48          51.72    66.40    40.84     9.72     -7.13
64          51.11    63.94    41.10     5.93     -8.82
80          46.51    59.50    39.80     9.33     -4.18
96          47.72    57.75    39.84     4.20     -7.62
128         54.35    58.95    40.66     3.24     -8.63

________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%

______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -1.43    -3.52    1.55      0          3.03
2           33.09     21.63   20.12    -10.00     -9.52
4           67.17     94.60   44.28     19.51     -11.80
8           75.72     108.14  49.15     25.00     -10.71
16          80.34     101.77  52.94     25.93     -4.49
24          70.84     93.12   43.62     27.63     -5.03
32          69.01     94.16   47.33     29.68     -1.51
40          58.56     63.47   25.91    -3.92      -25.85
48          61.16     74.70   34.88     .89       -22.08
64          54.37     69.09   26.80    -6.68      -30.04
80          36.22     22.73   -2.97    -8.25      -27.23
96          41.51     50.59   13.24     9.84      -16.77
128         48.98     38.15   6.41     -.33       -22.80

________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%

______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________

1            7.62     -38.03   -26.26  -50.00   -33.33
2            28.95     20.46    21.62   0       -7.14
4            84.05     60.79    45.74  -2.43    -12.42
8            86.43     79.57    50.32   15.85   -3.10
16           88.63     99.48    58.17   9.47    -13.10
24           74.65     80.87    41.99  -1.81    -22.89
32           63.86     59.21    23.58  -18.13   -36.37
40           64.79     60.53    22.23  -15.77   -35.84
48           49.68     26.93    .51    -36.40   -49.61
64           54.69     36.50    5.41   -26.59   -43.23
80           45.06     12.72   -13.25  -37.79   -52.08
96           40.21    -3.16    -24.53  -39.92   -52.97
128          36.33    -33.19   -43.66  -5.68    -20.49

________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%

#34 lore raw
Quoted 1 line

"Michael S. Tsirkin" <[email protected]> wrote on 02/23/2011 12:09:15 PM:

Hi Michael,

Quoted 6 lines
Quoted 4 lines

Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?

Yes, please do.

Sure, will get a build/test on latest bits and send in 1-2 days.

Quoted 8 lines
Quoted 4 lines

The TX-only patch helped the guest TX path but didn't help host->guest much (as tested using TCP_MAERTS from the guest). But with the TX+RX patch, both directions are getting improvements.

Also, my hope is that with appropriate queue mapping, we might be able to do away with heuristics to detect single stream load that TX only code needs.

Yes, that whole stuff is removed, and the TX/RX path is unchanged with this patch (thankfully :)

Quoted 3 lines
Quoted 1 line

Remote testing is still to be done.

Others might be able to help here once you post the patch.

That's great, will appreciate any help.

Thanks,

- KK

#35 lore raw
Quoted 18 lines · On Wed, Feb 23, 2011 at 12:18:36PM +0530, Krishna Kumar2 wrote:
Quoted 1 line

"Michael S. Tsirkin" <[email protected]> wrote on 02/23/2011 12:09:15 PM:

Hi Michael,

Quoted 6 lines
Quoted 4 lines

Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?

Yes, please do.

Sure, will get a build/test on latest bits and send in 1-2 days.

Quoted 5 lines
Quoted 4 lines

The TX-only patch helped the guest TX path but didn't help host->guest much (as tested using TCP_MAERTS from the guest). But with the TX+RX patch, both directions are getting improvements.

Quoted 6 lines
Quoted 3 lines

Also, my hope is that with appropriate queue mapping, we might be able to do away with heuristics to detect single stream load that TX only code needs.

Yes, that whole stuff is removed, and the TX/RX path is unchanged with this patch (thankfully :)

Cool. I was wondering whether in that case, we can do without host kernel changes at all,
and use a separate fd for each TX/RX pair. The advantage of that approach is that this way, the max fd limit naturally sets an upper bound on the amount of resources userspace can use up.

Thoughts?

In any case, pls don't let the above delay sending an RFC.

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

Remote testing is still to be done.

Others might be able to help here once you post the patch.

That's great, will appreciate any help.

Thanks,

- KK

#36 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 02/23/2011 09:25:34 PM:

Quoted 21 lines
Quoted 7 lines

Sure, will get a build/test on latest bits and send in 1-2 days.

Quoted 5 lines
Quoted 4 lines

The TX-only patch helped the guest TX path but didn't help host->guest much (as tested using TCP_MAERTS from the guest). But with the TX+RX patch, both directions are getting improvements.

Quoted 6 lines
Quoted 3 lines

Also, my hope is that with appropriate queue mapping, we might be able to do away with heuristics to detect single stream load that TX only code needs.

Yes, that whole stuff is removed, and the TX/RX path is unchanged with this patch (thankfully :)

Cool. I was wondering whether in that case, we can do without host kernel changes at all,
and use a separate fd for each TX/RX pair. The advantage of that approach is that this way, the max fd limit naturally sets an upper bound on the amount of resources userspace can use up.

Quoted 4 lines

Thoughts?

In any case, pls don't let the above delay sending an RFC.

I will look into this also.

Please excuse the delay in sending the patch out faster - my bits are a little old, so it is taking some time to move to the latest kernel and get some initial TCP/UDP test results. I should have it ready by tomorrow.

Thanks,

- KK

#37 lore raw
In reply to Krishna Kumar2
Quoted 32 lines · On Wed, Feb 23, 2011 at 10:52:09AM +0530, Krishna Kumar2 wrote:

Simon Horman <[email protected]> wrote on 02/22/2011 01:17:09 PM:

Hi Simon,

Quoted 3 lines

I have a few questions about the results below:

1. Are the (%) comparisons between non-mq and mq virtio?

Yes - mainline kernel with transmit-only MQ patch.

Quoted 1 line

2. Was UDP or TCP used?

TCP. I had done some initial testing on UDP, but don't have the results now as it is really old. But I will be running it again.

Quoted 1 line

3. What was the transmit size (-m option to netperf)?

I didn't use the -m option, so it defaults to 16K. The script does:

netperf -t TCP_STREAM -c -C -l 60 -H $SERVER

Quoted 2 lines

Also, I'm interested to know what the status of these patches is. Are you planing a fresh series?

Yes. Michael Tsirkin had wanted to see how the MQ RX patch would look like, so I was in the process of getting the two working together. The patch is ready and is being tested. Should I send a RFC patch at this time?

Quoted 4 lines

The TX-only patch helped the guest TX path but didn't help host->guest much (as tested using TCP_MAERTS from the guest). But with the TX+RX patch, both directions are getting improvements. Remote testing is still to be done.

Hi Krishna,

thanks for clarifying the test results.
I'm looking forward to the forthcoming RFC patches.