Skip to contentSkip to conversation

Implement multiqueue virtio-net

RFC PATCH Superseded Krishna Kumar 5 participants 17 messages Sep 17, 2010 – Oct 6, 2010

Subsystems (6)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Implement multiqueue virtio-net

Following patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.

1. This feature was first implemented with a single vhost. Testing showed 3-8% performance gain for upto 8 netperf sessions (and sometimes 16), but BW dropped with more sessions. However, adding more vhosts improved BW significantly all the way to 128 sessions. Multiple vhost is implemented in-kernel by passing an argument to SET_OWNER (retaining backward compatibility). The vhost patch adds 173 source lines (incl comments).
2. BW -> CPU/SD equation: Average TCP performance increased 23% compared to almost 70% for earlier patch (with unrestricted #vhosts). SD improved -4.2% while it had increased 55% for the earlier patch. Increasing #vhosts has it's pros and cons, but this patch lays emphasis on reducing CPU utilization. Another option could be a tunable to select number of vhosts threads.
3. Interoperability: Many combinations, but not all, of qemu, host, guest tested together. Tested with multiple i/f's on guest, with both mq=on/off, vhost=on/off, etc.

Changes from rev1:
------------------
1. Move queue_index from virtio_pci_vq_info to virtqueue, and resulting changes to existing code and to the patch.
2. virtio-net probe uses virtio_config_val.
3. Remove constants: VIRTIO_MAX_TXQS, MAX_VQS, all arrays allocated on stack, etc.
4. Restrict number of vhost threads to 2 - I get much better cpu/sd results (without any tuning) with low number of vhost threads. Higher vhosts gives better average BW performance (from average of 45%), but SD increases significantly (90%).
5. Working of vhost threads changes, eg for numtxqs=4:

       vhost-0: handles RX
       vhost-1: handles TX[0]
       vhost-0: handles TX[1]
       vhost-1: handles TX[2]
       vhost-0: handles TX[3]

                  Enabling MQ on virtio:
                  -----------------------

When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:

        vhost=on                   ->   #txqueues = 1
        vhost=on,mq=on             ->   #txqueues = 4
        vhost=on,mq=on,numtxqs=8   ->   #txqueues = 8
        vhost=on,mq=on,numtxqs=2   ->   #txqueues = 2
                   Performance (guest -> local host):
                   -----------------------------------

System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory, numtxqs=4
All testing without any system tuning, and default netperf Results split across two tables to show SD and CPU usage:
________________________________________________________________________
TCP: BW vs CPU/Remote CPU utilization:
# BW1 BW2 (%) CPU1 CPU2 (%) RCPU1 RCPU2 (%)
________________________________________________________________________

1    69971  65376 (-6.56)  134   170  (26.86)   322    376   (16.77)
2    20911  24839 (18.78)  107   139  (29.90)   217    264   (21.65)
4    21431  28912 (34.90)  213   318  (49.29)   444    541   (21.84)
8    21857  34592 (58.26)  444   859  (93.46)   901    1247  (38.40)
16   22368  33083 (47.90)  899   1523 (69.41)   1813   2410  (32.92)
24   22556  32578 (44.43)  1347  2249 (66.96)   2712   3606  (32.96)
32   22727  30923 (36.06)  1806  2506 (38.75)   3622   3952  (9.11)
40   23054  29334 (27.24)  2319  2872 (23.84)   4544   4551  (.15)
48   23006  28800 (25.18)  2827  2990 (5.76)    5465   4718  (-13.66)
64   23411  27661 (18.15)  3708  3306 (-10.84)  7231   5218  (-27.83)
80   23175  27141 (17.11)  4796  4509 (-5.98)   9152   7182  (-21.52)
96   23337  26759 (14.66)  5603  4543 (-18.91)  10890  7162  (-34.23)
128  22726  28339 (24.69)  7559  6395 (-15.39)  14600  10169 (-30.34)

________________________________________________________________________
Summary: BW: 22.8% CPU: 1.9% RCPU: -17.0%
________________________________________________________________________
TCP: BW vs SD/Remote SD:
# BW1 BW2 (%) SD1 SD2 (%) RSD1 RSD2 (%)
________________________________________________________________________

1    69971  65376 (-6.56)  4       6     (50.00)    21      26     (23.80)
2    20911  24839 (18.78)  6       7     (16.66)    27      28     (3.70)
4    21431  28912 (34.90)  26      31    (19.23)    108     111    (2.77)
8    21857  34592 (58.26)  106     135   (27.35)    432     393    (-9.02)
16   22368  33083 (47.90)  431     577   (33.87)    1742    1828   (4.93)
24   22556  32578 (44.43)  972     1393  (43.31)    3915    4479   (14.40)
32   22727  30923 (36.06)  1723    2165  (25.65)    6908    6842   (-.95)
40   23054  29334 (27.24)  2774    2761  (-.46)     10874   8764   (-19.40)
48   23006  28800 (25.18)  4126    3847  (-6.76)    15953   12172  (-23.70)
64   23411  27661 (18.15)  7216    6035  (-16.36)   28146   19078  (-32.21)
80   23175  27141 (17.11)  11729   12454 (6.18)     44765   39750  (-11.20)
96   23337  26759 (14.66)  16745   15905 (-5.01)    65099   50261  (-22.79)
128  22726  28339 (24.69)  30571   27893 (-8.76)    118089  89994  (-23.79)

________________________________________________________________________
Summary: BW: 22.8% SD: -4.21% RSD: -21.06%
________________________________________________________________________
UDP: BW vs SD/CPU
# BW1 BW2 (%) CPU1 CPU2 (%) SD1 SD2 (%)
_____________________________________________________________________________

1      36521    37415 (2.44)   61     61    (0)      2      2      (0)
4      28585    46903 (64.08)  397    546   (37.53)  72     68     (-5.55)
8      26649    44694 (67.71)  851    1243  (46.06)  334    339    (1.49)
16     25905    43385 (67.47)  1740   2631  (51.20)  1409   1572   (11.56)
32     24980    40448 (61.92)  3502   5360  (53.05)  5881   6401   (8.84)
48     27439    39451 (43.77)  5410   8324  (53.86)  12475  14855  (19.07)
64     25682    39915 (55.42)  7165   10825 (51.08)  23404  25982  (11.01)
96     26205    40190 (53.36)  10855  16283 (50.00)  52124  75014  (43.91)
128    25741    40252 (56.37)  14448  22186 (53.55)  133922 96843  (-27.68)

____________________________________________________________________________
Summary: BW: 50.4 CPU: 51.8 SD: -27.68
_____________________________________________________________________________
N#: Number of netperf sessions, 60 sec runs BW1,SD1,RSD1: Bandwidth (sum across 2 runs in mbps), SD and Remote
SD for original code
BW2,SD2,RSD2: Bandwidth (sum across 2 runs in mbps), SD and Remote
SD for new code.
CPU1,CPU2,RCPU1,RCPU2: Similar to SD.

For 1 TCP netperf, I ran 7 iterations and summed it. Explanation for degradation for 1 stream case:
1. Without any tuning, BW falls -6.5%.
2. When vhosts on server were bound to CPU0, BW was as good as with original code.
3. When new code was started with numtxqs=1 (or mq=off, which is the default), there was no degradation.

Next steps:
-----------
1. MQ RX patch is also complete - plan to submit once TX is OK (as well as after identifying bandwidth degradations for some test cases).
2. Cache-align data structures: I didn't see any BW/SD improvement after making the sq's (and similarly for vhost) cache-aligned statically:

        struct virtnet_info {
                ...
                struct send_queue sq[16] ____cacheline_aligned_in_smp;
                ...
        };

3. Migration is not tested.

Review/feedback appreciated.

---

#1 lore raw

RFC PATCH v2 1/4 Change virtqueue structure

Move queue_index from virtio_pci_vq_info to virtqueue. This allows callback handlers to figure out the queue number for the vq that needs attention.

---
 drivers/virtio/virtio_pci.c |   10 +++-------
 include/linux/virtio.h      |    1 +
 2 files changed, 4 insertions(+), 7 deletions(-)
include/linux/virtio.h +1 -0
diff -ruNp org2/include/linux/virtio.h tx_only2/include/linux/virtio.h
@@ -22,6 +22,7 @@ struct virtqueue {
 	void (*callback)(struct virtqueue *vq);
 	const char *name;
 	struct virtio_device *vdev;
+	int queue_index;	/* the index of the queue */
 	void *priv;
 };
 
drivers/virtio/virtio_pci.c +3 -7
diff -ruNp org2/drivers/virtio/virtio_pci.c tx_only2/drivers/virtio/virtio_pci.c
@@ -75,9 +75,6 @@ struct virtio_pci_vq_info
 	/* the number of entries in the queue */
 	int num;
 
-	/* the index of the queue */
-	int queue_index;
-
 	/* the virtual address of the ring queue */
 	void *queue;
 
@@ -185,11 +182,10 @@ static void vp_reset(struct virtio_devic
 static void vp_notify(struct virtqueue *vq)
 {
 	struct virtio_pci_device *vp_dev = to_vp_device(vq->vdev);
-	struct virtio_pci_vq_info *info = vq->priv;
 
 	/* we write the queue's selector into the notification register to
 	 * signal the other end */
-	iowrite16(info->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_NOTIFY);
+	iowrite16(vq->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_NOTIFY);
 }
 
 /* Handle a configuration change: Tell driver if it wants to know. */
@@ -385,7 +381,6 @@ static struct virtqueue *setup_vq(struct
 	if (!info)
 		return ERR_PTR(-ENOMEM);
 
-	info->queue_index = index;
 	info->num = num;
 	info->msix_vector = msix_vec;
 
@@ -408,6 +403,7 @@ static struct virtqueue *setup_vq(struct
 		goto out_activate_queue;
 	}
 
+	vq->queue_index = index;
 	vq->priv = info;
 	info->vq = vq;
 
@@ -446,7 +442,7 @@ static void vp_del_vq(struct virtqueue *
 	list_del(&info->node);
 	spin_unlock_irqrestore(&vp_dev->lock, flags);
 
-	iowrite16(info->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_SEL);
+	iowrite16(vq->queue_index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_SEL);
 
 	if (vp_dev->msix_enabled) {
 		iowrite16(VIRTIO_MSI_NO_VECTOR,
#2 lore raw

RFC PATCH v2 2/4 Changes for virtio-net

In reply to Krishna Kumar

Implement mq virtio-net driver.

Though struct virtio_net_config changes, it works with old qemu's since the last element is not accessed, unless qemu sets VIRTIO_NET_F_NUMTXQS.

---
 drivers/net/virtio_net.c   |  213 ++++++++++++++++++++++++++---------
 include/linux/virtio_net.h |    3 
 2 files changed, 163 insertions(+), 53 deletions(-)
include/linux/virtio_net.h +3 -0
diff -ruNp org2/include/linux/virtio_net.h tx_only2/include/linux/virtio_net.h
@@ -26,6 +26,7 @@
 #define VIRTIO_NET_F_CTRL_RX	18	/* Control channel RX mode support */
 #define VIRTIO_NET_F_CTRL_VLAN	19	/* Control channel VLAN filtering */
 #define VIRTIO_NET_F_CTRL_RX_EXTRA 20	/* Extra RX mode control support */
+#define VIRTIO_NET_F_NUMTXQS	21	/* Device supports multiple TX queue */
 
 #define VIRTIO_NET_S_LINK_UP	1	/* Link is up */
 
@@ -34,6 +35,8 @@ struct virtio_net_config {
 	__u8 mac[6];
 	/* See VIRTIO_NET_F_STATUS and VIRTIO_NET_S_* above */
 	__u16 status;
+	/* number of transmit queues */
+	__u16 numtxqs;
 } __attribute__((packed));
 
 /* This is the first element of the scatter-gather list.  If you don't
drivers/net/virtio_net.c +160 -53

356 lines not on this page · raw message

#3 lore raw

Re: [v2 RFC PATCH 2/4] Changes for virtio-net

Quoted 36 lines · Krishna Kumar

Implement mq virtio-net driver.

Though struct virtio_net_config changes, it works with old qemu's since the last element is not accessed, unless qemu sets VIRTIO_NET_F_NUMTXQS.

---
 drivers/net/virtio_net.c   |  213 ++++++++++++++++++++++++++---------
 include/linux/virtio_net.h |    3 
 2 files changed, 163 insertions(+), 53 deletions(-)

diff -ruNp org2/include/linux/virtio_net.h tx_only2/include/linux/virtio_net.h

include/linux/virtio_net.h +3 -0
@@ -26,6 +26,7 @@
 #define VIRTIO_NET_F_CTRL_RX	18	/* Control channel RX mode support */
 #define VIRTIO_NET_F_CTRL_VLAN	19	/* Control channel VLAN filtering */
 #define VIRTIO_NET_F_CTRL_RX_EXTRA 20	/* Extra RX mode control support */
+#define VIRTIO_NET_F_NUMTXQS	21	/* Device supports multiple TX queue */
 
 #define VIRTIO_NET_S_LINK_UP	1	/* Link is up */
 
@@ -34,6 +35,8 @@ struct virtio_net_config {
 	__u8 mac[6];
 	/* See VIRTIO_NET_F_STATUS and VIRTIO_NET_S_* above */
 	__u16 status;
+	/* number of transmit queues */
+	__u16 numtxqs;
 } __attribute__((packed));
 
 /* This is the first element of the scatter-gather list.  If you don't

diff -ruNp org2/drivers/net/virtio_net.c tx_only2/drivers/net/virtio_net.c

drivers/net/virtio_net.c module_param(gso, bool, 0444); · line 49

Quoted 10 lines
drivers/net/virtio_net.c +7 -0
@@ -40,9 +40,20 @@ module_param(gso, bool, 0444);
 
 #define VIRTNET_SEND_COMMAND_SG_MAX    2
 
+/* Our representation of a send virtqueue */
+struct send_queue {
+	struct virtqueue *svq;
+
+	/* TX: fragments + linear part + virtio header */
+	struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
+};

You probably want ____cacheline_aligned_in_smp

drivers/net/virtio_net.c module_param(gso, bool, 0444); · line 57

Quoted 9 lines
drivers/net/virtio_net.c +5 -1
+
 struct virtnet_info {
 	struct virtio_device *vdev;
-	struct virtqueue *rvq, *svq, *cvq;
+	int numtxqs;			/* Number of tx queues */
+	struct send_queue *sq;
+	struct virtqueue *rvq;
+	struct virtqueue *cvq;
 	struct net_device *dev;

struct napi will probably be dirtied by RX processing

You should make sure it doesnt dirty cache line of above (read mostly) fields

Quoted 148 lines
drivers/net/virtio_net.c +28 -23
 	struct napi_struct napi;
 	unsigned int status;
@@ -62,9 +73,8 @@ struct virtnet_info {
 	/* Chain pages by the private ptr. */
 	struct page *pages;
 
-	/* fragments + linear part + virtio header */
+	/* RX: fragments + linear part + virtio header */
 	struct scatterlist rx_sg[MAX_SKB_FRAGS + 2];
-	struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
 };
 
 struct skb_vnet_hdr {
@@ -120,12 +130,13 @@ static struct page *get_a_page(struct vi
 static void skb_xmit_done(struct virtqueue *svq)
 {
 	struct virtnet_info *vi = svq->vdev->priv;
+	int qnum = svq->queue_index - 1;	/* 0 is RX vq */
 
 	/* Suppress further interrupts. */
 	virtqueue_disable_cb(svq);
 
 	/* We were probably waiting for more output buffers. */
-	netif_wake_queue(vi->dev);
+	netif_wake_subqueue(vi->dev, qnum);
 }
 
 static void set_skb_frag(struct sk_buff *skb, struct page *page,
@@ -495,12 +506,13 @@ again:
 	return received;
 }
 
-static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
+static unsigned int free_old_xmit_skbs(struct virtnet_info *vi,
+				       struct virtqueue *svq)
 {
 	struct sk_buff *skb;
 	unsigned int len, tot_sgs = 0;
 
-	while ((skb = virtqueue_get_buf(vi->svq, &len)) != NULL) {
+	while ((skb = virtqueue_get_buf(svq, &len)) != NULL) {
 		pr_debug("Sent skb %p\n", skb);
 		vi->dev->stats.tx_bytes += skb->len;
 		vi->dev->stats.tx_packets++;
@@ -510,7 +522,8 @@ static unsigned int free_old_xmit_skbs(s
 	return tot_sgs;
 }
 
-static int xmit_skb(struct virtnet_info *vi, struct sk_buff *skb)
+static int xmit_skb(struct virtnet_info *vi, struct sk_buff *skb,
+		    struct virtqueue *svq, struct scatterlist *tx_sg)
 {
 	struct skb_vnet_hdr *hdr = skb_vnet_hdr(skb);
 	const unsigned char *dest = ((struct ethhdr *)skb->data)->h_dest;
@@ -548,12 +561,12 @@ static int xmit_skb(struct virtnet_info 
 
 	/* Encode metadata header at front. */
 	if (vi->mergeable_rx_bufs)
-		sg_set_buf(vi->tx_sg, &hdr->mhdr, sizeof hdr->mhdr);
+		sg_set_buf(tx_sg, &hdr->mhdr, sizeof hdr->mhdr);
 	else
-		sg_set_buf(vi->tx_sg, &hdr->hdr, sizeof hdr->hdr);
+		sg_set_buf(tx_sg, &hdr->hdr, sizeof hdr->hdr);
 
-	hdr->num_sg = skb_to_sgvec(skb, vi->tx_sg + 1, 0, skb->len) + 1;
-	return virtqueue_add_buf(vi->svq, vi->tx_sg, hdr->num_sg,
+	hdr->num_sg = skb_to_sgvec(skb, tx_sg + 1, 0, skb->len) + 1;
+	return virtqueue_add_buf(svq, tx_sg, hdr->num_sg,
 					0, skb);
 }
 
@@ -561,31 +574,34 @@ static netdev_tx_t start_xmit(struct sk_
 {
 	struct virtnet_info *vi = netdev_priv(dev);
 	int capacity;
+	int qnum = skb_get_queue_mapping(skb);
+	struct virtqueue *svq = vi->sq[qnum].svq;
 
 	/* Free up any pending old buffers before queueing new ones. */
-	free_old_xmit_skbs(vi);
+	free_old_xmit_skbs(vi, svq);
 
 	/* Try to transmit */
-	capacity = xmit_skb(vi, skb);
+	capacity = xmit_skb(vi, skb, svq, vi->sq[qnum].tx_sg);
 
 	/* This can happen with OOM and indirect buffers. */
 	if (unlikely(capacity < 0)) {
 		if (net_ratelimit()) {
 			if (likely(capacity == -ENOMEM)) {
 				dev_warn(&dev->dev,
-					 "TX queue failure: out of memory\n");
+					 "TXQ (%d) failure: out of memory\n",
+					 qnum);
 			} else {
 				dev->stats.tx_fifo_errors++;
 				dev_warn(&dev->dev,
-					 "Unexpected TX queue failure: %d\n",
-					 capacity);
+					 "Unexpected TXQ (%d) failure: %d\n",
+					 qnum, capacity);
 			}
 		}
 		dev->stats.tx_dropped++;
 		kfree_skb(skb);
 		return NETDEV_TX_OK;
 	}
-	virtqueue_kick(vi->svq);
+	virtqueue_kick(svq);
 
 	/* Don't wait up for transmitted skbs to be freed. */
 	skb_orphan(skb);
@@ -594,13 +610,13 @@ static netdev_tx_t start_xmit(struct sk_
 	/* Apparently nice girls don't return TX_BUSY; stop the queue
 	 * before it gets out of hand.  Naturally, this wastes entries. */
 	if (capacity < 2+MAX_SKB_FRAGS) {
-		netif_stop_queue(dev);
-		if (unlikely(!virtqueue_enable_cb(vi->svq))) {
+		netif_stop_subqueue(dev, qnum);
+		if (unlikely(!virtqueue_enable_cb(svq))) {
 			/* More just got used, free them then recheck. */
-			capacity += free_old_xmit_skbs(vi);
+			capacity += free_old_xmit_skbs(vi, svq);
 			if (capacity >= 2+MAX_SKB_FRAGS) {
-				netif_start_queue(dev);
-				virtqueue_disable_cb(vi->svq);
+				netif_start_subqueue(dev, qnum);
+				virtqueue_disable_cb(svq);
 			}
 		}
 	}
@@ -871,10 +887,10 @@ static void virtnet_update_status(struct
 
 	if (vi->status & VIRTIO_NET_S_LINK_UP) {
 		netif_carrier_on(vi->dev);
-		netif_wake_queue(vi->dev);
+		netif_tx_wake_all_queues(vi->dev);
 	} else {
 		netif_carrier_off(vi->dev);
-		netif_stop_queue(vi->dev);
+		netif_tx_stop_all_queues(vi->dev);
 	}
 }
 
@@ -885,18 +901,112 @@ static void virtnet_config_changed(struc
 	virtnet_update_status(vi);
 }
 

drivers/net/virtio_net.c static void virtnet_config_changed(struc · line 913

Quoted 10 lines
drivers/net/virtio_net.c +10 -0
+#define MAX_DEVICE_NAME		16
+static int initialize_vqs(struct virtnet_info *vi, int numtxqs)
+{
+	vq_callback_t **callbacks;
+	struct virtqueue **vqs;
+	int i, err = -ENOMEM;
+	int totalvqs;
+	char **names;
+
+	/* Allocate send queues */

no check on numtxqs ? Hmm...

Please then use kcalloc(numtxqs, sizeof(*vi->sq), GFP_KERNEL) so that some check is done for you ;)

Quoted 22 lines
drivers/net/virtio_net.c +22 -0
+	vi->sq = kzalloc(numtxqs * sizeof(*vi->sq), GFP_KERNEL);
+	if (!vi->sq)
+		goto out;
+
+	/* setup initial send queue parameters */
+	for (i = 0; i < numtxqs; i++)
+		sg_init_table(vi->sq[i].tx_sg, ARRAY_SIZE(vi->sq[i].tx_sg));
+
+	/*
+	 * We expect 1 RX virtqueue followed by 'numtxqs' TX virtqueues, and
+	 * optionally one control virtqueue.
+	 */
+	totalvqs = 1 + numtxqs +
+		   virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ);
+
+	/* Setup parameters for find_vqs */
+	vqs = kmalloc(totalvqs * sizeof(*vqs), GFP_KERNEL);
+	callbacks = kmalloc(totalvqs * sizeof(*callbacks), GFP_KERNEL);
+	names = kzalloc(totalvqs * sizeof(*names), GFP_KERNEL);
+	if (!vqs || !callbacks || !names)
+		goto free_mem;
+
#4 lore raw

Re: [v2 RFC PATCH 2/4] Changes for virtio-net

Eric Dumazet <[email protected]> wrote on 09/17/2010 03:55:54 PM:

Quoted 9 lines

drivers/net/virtio_net.c module_param(gso, bool, 0444); · line 49

Quoted 7 lines
drivers/net/virtio_net.c +7 -0
+/* Our representation of a send virtqueue */
+struct send_queue {
+   struct virtqueue *svq;
+
+   /* TX: fragments + linear part + virtio header */
+   struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
+};

You probably want ____cacheline_aligned_in_smp

I had tried this and mentioned this in Patch 0/4:
"2. Cache-align data structures: I didn't see any BW/SD improvement
after making the sq's (and similarly for vhost) cache-aligned statically:

        struct virtnet_info {
                ...
                struct send_queue sq[16] ____cacheline_aligned_in_smp;
                ...
        };

"

I am not sure why this made no difference?

Quoted 10 lines

drivers/net/virtio_net.c module_param(gso, bool, 0444); · line 57

Quoted 9 lines
drivers/net/virtio_net.c +5 -1
+
 struct virtnet_info {
    struct virtio_device *vdev;
-   struct virtqueue *rvq, *svq, *cvq;
+   int numtxqs;         /* Number of tx queues */
+   struct send_queue *sq;
+   struct virtqueue *rvq;
+   struct virtqueue *cvq;
    struct net_device *dev;
Quoted 4 lines

struct napi will probably be dirtied by RX processing

You should make sure it doesnt dirty cache line of above (read mostly) fields

I am changing the layout of napi wrt other pointers in this patch, though the to-be-submitted RX patch does that. Should I do something for this TX-only patch?

Quoted 13 lines

drivers/net/virtio_net.c static void virtnet_config_changed(struc · line 913

Quoted 10 lines
drivers/net/virtio_net.c +10 -0
+#define MAX_DEVICE_NAME      16
+static int initialize_vqs(struct virtnet_info *vi, int numtxqs)
+{
+   vq_callback_t **callbacks;
+   struct virtqueue **vqs;
+   int i, err = -ENOMEM;
+   int totalvqs;
+   char **names;
+
+   /* Allocate send queues */

no check on numtxqs ? Hmm...

Quoted 2 lines

Please then use kcalloc(numtxqs, sizeof(*vi->sq), GFP_KERNEL) so that some check is done for you ;)

Right! I need to re-introduce some limit. Rusty, should I simply add a check for a constant (like 256) here?

Thanks for your review, Eric!

- KK

#5 lore raw

Re: [v2 RFC PATCH 2/4] Changes for virtio-net

Quoted 13 lines

Krishna Kumar2/India/IBM@IBMIN
Sent by: [email protected]

Quoted 10 lines
Quoted 9 lines
drivers/net/virtio_net.c +5 -1
+
 struct virtnet_info {
    struct virtio_device *vdev;
-   struct virtqueue *rvq, *svq, *cvq;
+   int numtxqs;         /* Number of tx queues */
+   struct send_queue *sq;
+   struct virtqueue *rvq;
+   struct virtqueue *cvq;
    struct net_device *dev;
Quoted 8 lines
Quoted 4 lines

struct napi will probably be dirtied by RX processing

You should make sure it doesnt dirty cache line of above (read mostly) fields

I am changing the layout of napi wrt other pointers in this patch, though the to-be-submitted RX patch does that. Should I do something for this TX-only patch?

Sorry, I think my sentence is not clear! I will make this change (and also cache-line align the send queues), test and let you know the result.

Thanks,

- KK

#6 lore raw

RFC PATCH v2 3/4 Changes for vhost

In reply to Krishna Kumar

Changes for mq vhost.

vhost_net_open is changed to allocate a vhost_net and return. The remaining initializations are delayed till SET_OWNER. SET_OWNER is changed so that the argument is used to figure out how many txqs to use. Unmodified qemu's will pass NULL, so this is recognized and handled as numtxqs=1.

Besides changing handle_tx to use 'vq', this patch also changes handle_rx to take vq as parameter. The mq RX patch requires this change, but till then it is consistent (and less confusing) to make the interfaces for handling rx and tx similar.

---
 drivers/vhost/net.c   |  273 ++++++++++++++++++++++++++--------------
 drivers/vhost/vhost.c |  186 +++++++++++++++++++--------
 drivers/vhost/vhost.h |   16 +-
 3 files changed, 324 insertions(+), 151 deletions(-)
drivers/vhost/vhost.h +11 -5
diff -ruNp org2/drivers/vhost/vhost.h tx_only2/drivers/vhost/vhost.h
@@ -40,11 +40,11 @@ struct vhost_poll {
 	wait_queue_t              wait;
 	struct vhost_work	  work;
 	unsigned long		  mask;
-	struct vhost_dev	 *dev;
+	struct vhost_virtqueue	  *vq;  /* points back to vq */
 };
 
 void vhost_poll_init(struct vhost_poll *poll, vhost_work_fn_t fn,
-		     unsigned long mask, struct vhost_dev *dev);
+		     unsigned long mask, struct vhost_virtqueue *vq);
 void vhost_poll_start(struct vhost_poll *poll, struct file *file);
 void vhost_poll_stop(struct vhost_poll *poll);
 void vhost_poll_flush(struct vhost_poll *poll);
@@ -110,6 +110,10 @@ struct vhost_virtqueue {
 	/* Log write descriptors */
 	void __user *log_base;
 	struct vhost_log log[VHOST_NET_MAX_SG];
+	struct task_struct *worker; /* vhost for this vq, shared btwn RX/TX */
+	spinlock_t *work_lock;
+	struct list_head *work_list;
+	int qnum;	/* 0 for RX, 1 -> n-1 for TX */
 };
 
 struct vhost_dev {
@@ -124,11 +128,13 @@ struct vhost_dev {
 	int nvqs;
 	struct file *log_file;
 	struct eventfd_ctx *log_ctx;
-	spinlock_t work_lock;
-	struct list_head work_list;
-	struct task_struct *worker;
+	spinlock_t *work_lock;
+	struct list_head *work_list;
 };
 
+int vhost_setup_vqs(struct vhost_dev *dev, int numtxqs);
+void vhost_free_vqs(struct vhost_dev *dev);
+int vhost_get_num_threads(int nvqs);
 long vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue *vqs, int nvqs);
 long vhost_dev_check_owner(struct vhost_dev *);
 long vhost_dev_reset_owner(struct vhost_dev *);
drivers/vhost/net.c +182 -91

449 lines not on this page · raw message

drivers/vhost/vhost.c +131 -55

316 lines not on this page · raw message

#7 lore raw

RFC PATCH v2 4/4 qemu changes

In reply to Krishna Kumar

Changes in qemu to support mq TX.

---
 hw/vhost.c      |    8 ++-
 hw/vhost.h      |    2 
 hw/vhost_net.c  |   16 +++++--
 hw/vhost_net.h  |    2 
 hw/virtio-net.c |   97 ++++++++++++++++++++++++++++++----------------
 hw/virtio-net.h |    2 
 hw/virtio-pci.c |    2 
 net.c           |   17 ++++++++
 net.h           |    1 
 net/tap.c       |   27 ++++++++++--
 10 files changed, 129 insertions(+), 45 deletions(-)
hw/vhost.c +6 -2
diff -ruNp org2/hw/vhost.c tx_only.rev2/hw/vhost.c
@@ -599,23 +599,27 @@ static void vhost_virtqueue_cleanup(stru
                               0, virtio_queue_get_desc_size(vdev, idx));
 }
 
-int vhost_dev_init(struct vhost_dev *hdev, int devfd)
+int vhost_dev_init(struct vhost_dev *hdev, int devfd, int numtxqs)
 {
     uint64_t features;
     int r;
     if (devfd >= 0) {
         hdev->control = devfd;
+        hdev->nvqs = 2;
     } else {
         hdev->control = open("/dev/vhost-net", O_RDWR);
         if (hdev->control < 0) {
             return -errno;
         }
     }
-    r = ioctl(hdev->control, VHOST_SET_OWNER, NULL);
+
+    r = ioctl(hdev->control, VHOST_SET_OWNER, numtxqs);
     if (r < 0) {
         goto fail;
     }
 
+    hdev->nvqs = numtxqs + 1;
+
     r = ioctl(hdev->control, VHOST_GET_FEATURES, &features);
     if (r < 0) {
         goto fail;
hw/vhost.h +1 -1
diff -ruNp org2/hw/vhost.h tx_only.rev2/hw/vhost.h
@@ -40,7 +40,7 @@ struct vhost_dev {
     unsigned long long log_size;
 };
 
-int vhost_dev_init(struct vhost_dev *hdev, int devfd);
+int vhost_dev_init(struct vhost_dev *hdev, int devfd, int nvqs);
 void vhost_dev_cleanup(struct vhost_dev *hdev);
 int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev);
 void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev);
hw/vhost_net.c +11 -5
diff -ruNp org2/hw/vhost_net.c tx_only.rev2/hw/vhost_net.c
@@ -36,7 +36,8 @@
 
 struct vhost_net {
     struct vhost_dev dev;
-    struct vhost_virtqueue vqs[2];
+    struct vhost_virtqueue *vqs;
+    int nvqs;
     int backend;
     VLANClientState *vc;
 };
@@ -76,7 +77,8 @@ static int vhost_net_get_fd(VLANClientSt
     }
 }
 
-struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd)
+struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd,
+				 int numtxqs)
 {
     int r;
     struct vhost_net *net = qemu_malloc(sizeof *net);
@@ -93,10 +95,14 @@ struct vhost_net *vhost_net_init(VLANCli
         (1 << VHOST_NET_F_VIRTIO_NET_HDR);
     net->backend = r;
 
-    r = vhost_dev_init(&net->dev, devfd);
+    r = vhost_dev_init(&net->dev, devfd, numtxqs);
     if (r < 0) {
         goto fail;
     }
+
+    net->nvqs = numtxqs + 1;
+    net->vqs = qemu_malloc(net->nvqs * (sizeof *net->vqs));
+
     if (~net->dev.features & net->dev.backend_features) {
         fprintf(stderr, "vhost lacks feature mask %" PRIu64 " for backend\n",
                 (uint64_t)(~net->dev.features & net->dev.backend_features));
@@ -118,7 +124,6 @@ int vhost_net_start(struct vhost_net *ne
     struct vhost_vring_file file = { };
     int r;
 
-    net->dev.nvqs = 2;
     net->dev.vqs = net->vqs;
     r = vhost_dev_start(&net->dev, dev);
     if (r < 0) {
@@ -166,7 +171,8 @@ void vhost_net_cleanup(struct vhost_net 
     qemu_free(net);
 }
 #else
-struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd)
+struct vhost_net *vhost_net_init(VLANClientState *backend, int devfd,
+				 int nvqs)
 {
 	return NULL;
 }
hw/vhost_net.h +1 -1
diff -ruNp org2/hw/vhost_net.h tx_only.rev2/hw/vhost_net.h
@@ -6,7 +6,7 @@
 struct vhost_net;
 typedef struct vhost_net VHostNetState;
 
-VHostNetState *vhost_net_init(VLANClientState *backend, int devfd);
+VHostNetState *vhost_net_init(VLANClientState *backend, int devfd, int nvqs);
 
 int vhost_net_start(VHostNetState *net, VirtIODevice *dev);
 void vhost_net_stop(VHostNetState *net, VirtIODevice *dev);
hw/virtio-net.c +65 -32

228 lines not on this page · raw message

hw/virtio-net.h +2 -0
diff -ruNp org2/hw/virtio-net.h tx_only.rev2/hw/virtio-net.h
@@ -44,6 +44,7 @@
 #define VIRTIO_NET_F_CTRL_RX    18      /* Control channel RX mode support */
 #define VIRTIO_NET_F_CTRL_VLAN  19      /* Control channel VLAN filtering */
 #define VIRTIO_NET_F_CTRL_RX_EXTRA 20   /* Extra RX mode control support */
+#define VIRTIO_NET_F_NUMTXQS    21      /* Supports multiple TX queues */
 
 #define VIRTIO_NET_S_LINK_UP    1       /* Link is up */
 
@@ -58,6 +59,7 @@ struct virtio_net_config
     uint8_t mac[ETH_ALEN];
     /* See VIRTIO_NET_F_STATUS and VIRTIO_NET_S_* above */
     uint16_t status;
+    uint16_t numtxqs;	/* number of transmit queues */
 } __attribute__((packed));
 
 /* This is the first element of the scatter-gather list.  If you don't
hw/virtio-pci.c +2 -0
diff -ruNp org2/hw/virtio-pci.c tx_only.rev2/hw/virtio-pci.c
@@ -99,6 +99,7 @@ typedef struct {
     uint32_t addr;
     uint32_t class_code;
     uint32_t nvectors;
+    uint32_t mq;
     BlockConf block;
     NICConf nic;
     uint32_t host_features;
@@ -722,6 +723,7 @@ static PCIDeviceInfo virtio_info[] = {
         .romfile    = "pxe-virtio.bin",
         .qdev.props = (Property[]) {
             DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 3),
+	    DEFINE_PROP_UINT32("mq", VirtIOPCIProxy, mq, 1),
             DEFINE_VIRTIO_NET_FEATURES(VirtIOPCIProxy, host_features),
             DEFINE_NIC_PROPERTIES(VirtIOPCIProxy, nic),
             DEFINE_PROP_END_OF_LIST(),
net/tap.c +23 -4
diff -ruNp org2/net/tap.c tx_only.rev2/net/tap.c
@@ -299,13 +299,14 @@ static NetClientInfo net_tap_info = {
 static TAPState *net_tap_fd_init(VLANState *vlan,
                                  const char *model,
                                  const char *name,
-                                 int fd,
+                                 int fd, int numtxqs,
                                  int vnet_hdr)
 {
     VLANClientState *nc;
     TAPState *s;
 
     nc = qemu_new_net_client(&net_tap_info, vlan, NULL, model, name);
+    nc->numtxqs = numtxqs;
 
     s = DO_UPCAST(TAPState, nc, nc);
 
@@ -403,6 +404,24 @@ int net_init_tap(QemuOpts *opts, Monitor
 {
     TAPState *s;
     int fd, vnet_hdr = 0;
+    int vhost;
+    int numtxqs = 1;
+
+    vhost = qemu_opt_get_bool(opts, "vhost", 0);
+
+    /*
+     * We support multiple tx queues if:
+     *      1. smp > 1
+     *      2. vhost=on
+     *      3. mq=on
+     * In this case, #txqueues = #cpus. This value can be changed by
+     * using the "numtxqs" option.
+     */
+    if (vhost && smp_cpus > 1) {
+        if (qemu_opt_get_bool(opts, "mq", 0)) {
+            numtxqs = qemu_opt_get_number(opts, "numtxqs", smp_cpus);
+        }
+    }
 
     if (qemu_opt_get(opts, "fd")) {
         if (qemu_opt_get(opts, "ifname") ||
@@ -436,7 +455,7 @@ int net_init_tap(QemuOpts *opts, Monitor
         }
     }
 
-    s = net_tap_fd_init(vlan, "tap", name, fd, vnet_hdr);
+    s = net_tap_fd_init(vlan, "tap", name, fd, numtxqs, vnet_hdr);
     if (!s) {
         close(fd);
         return -1;
@@ -465,7 +484,7 @@ int net_init_tap(QemuOpts *opts, Monitor
         }
     }
 
-    if (qemu_opt_get_bool(opts, "vhost", !!qemu_opt_get(opts, "vhostfd"))) {
+    if (vhost) {
         int vhostfd, r;
         if (qemu_opt_get(opts, "vhostfd")) {
             r = net_handle_fd_param(mon, qemu_opt_get(opts, "vhostfd"));
@@ -476,7 +495,7 @@ int net_init_tap(QemuOpts *opts, Monitor
         } else {
             vhostfd = -1;
         }
-        s->vhost_net = vhost_net_init(&s->nc, vhostfd);
+        s->vhost_net = vhost_net_init(&s->nc, vhostfd, numtxqs);
         if (!s->vhost_net) {
             error_report("vhost-net requested but could not be initialized");
             return -1;
net.c +17 -0
diff -ruNp org2/net.c tx_only.rev2/net.c
@@ -814,6 +814,15 @@ static int net_init_nic(QemuOpts *opts,
         return -1;
     }
 
+    if (nd->netdev->numtxqs > 1 && nd->nvectors == DEV_NVECTORS_UNSPECIFIED) {
+        /*
+         * User specified mq for guest, but no "vectors=", tune
+         * it automatically to 'numtxqs' TX + 1 RX + 1 controlq.
+         */
+        nd->nvectors = nd->netdev->numtxqs + 1 + 1;
+        monitor_printf(mon, "nvectors tuned to %d\n", nd->nvectors);
+    }
+
     nd->used = 1;
     nb_nics++;
 
@@ -957,6 +966,14 @@ static const struct {
             },
 #ifndef _WIN32
             {
+                .name = "mq",
+                .type = QEMU_OPT_BOOL,
+                .help = "enable multiqueue on network i/f",
+            }, {
+                .name = "numtxqs",
+                .type = QEMU_OPT_NUMBER,
+                .help = "optional number of TX queues, if mq is enabled",
+            }, {
                 .name = "fd",
                 .type = QEMU_OPT_STRING,
                 .help = "file descriptor of an already opened tap",
net.h +1 -0
diff -ruNp org2/net.h tx_only.rev2/net.h
@@ -62,6 +62,7 @@ struct VLANClientState {
     struct VLANState *vlan;
     VLANClientState *peer;
     NetQueue *send_queue;
+    int numtxqs;
     char *model;
     char *name;
     char info_str[256];
#8 lore raw
In reply to Krishna Kumar
Quoted 35 lines · Krishna Kumar

Following patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.

1. This feature was first implemented with a single vhost. Testing showed 3-8% performance gain for upto 8 netperf sessions (and sometimes 16), but BW dropped with more sessions. However, adding more vhosts improved BW significantly all the way to 128 sessions. Multiple vhost is implemented in-kernel by passing an argument to SET_OWNER (retaining backward compatibility). The vhost patch adds 173 source lines (incl comments).
2. BW -> CPU/SD equation: Average TCP performance increased 23% compared to almost 70% for earlier patch (with unrestricted #vhosts). SD improved -4.2% while it had increased 55% for the earlier patch. Increasing #vhosts has it's pros and cons, but this patch lays emphasis on reducing CPU utilization. Another option could be a tunable to select number of vhosts threads.
3. Interoperability: Many combinations, but not all, of qemu, host, guest tested together. Tested with multiple i/f's on guest, with both mq=on/off, vhost=on/off, etc.

Changes from rev1:
------------------
1. Move queue_index from virtio_pci_vq_info to virtqueue, and resulting changes to existing code and to the patch.
2. virtio-net probe uses virtio_config_val.
3. Remove constants: VIRTIO_MAX_TXQS, MAX_VQS, all arrays allocated on stack, etc.
4. Restrict number of vhost threads to 2 - I get much better cpu/sd results (without any tuning) with low number of vhost threads. Higher vhosts gives better average BW performance (from average of 45%), but SD increases significantly (90%).
5. Working of vhost threads changes, eg for numtxqs=4:

Quoted 5 lines
       vhost-0: handles RX
       vhost-1: handles TX[0]
       vhost-0: handles TX[1]
       vhost-1: handles TX[2]
       vhost-0: handles TX[3]

This doesn't look symmetrical.
TCP flows that go via TX(1,3) use the same vhost thread for RX packets, whereas flows via TX(0,2) use a different vhost thread.

Thanks
Sridhar

#9 lore raw
In reply to Krishna Kumar
Quoted 2 lines · Krishna Kumar

For 1 TCP netperf, I ran 7 iterations and summed it. Explanation for degradation for 1 stream case:

Could you document how exactly do you measure multistream bandwidth:
netperf flags, etc?

Quoted 1 line

1. Without any tuning, BW falls -6.5%.

Any idea where does this come from?
Do you see more TX interrupts? RX interrupts? Exits? Do interrupts bounce more between guest CPUs?

Quoted 13 lines

2. When vhosts on server were bound to CPU0, BW was as good as with original code.
3. When new code was started with numtxqs=1 (or mq=off, which is the default), there was no degradation.

Next steps:
-----------
1. MQ RX patch is also complete - plan to submit once TX is OK (as well as after identifying bandwidth degradations for some test cases).
2. Cache-align data structures: I didn't see any BW/SD improvement after making the sq's (and similarly for vhost) cache-aligned statically:

Quoted 6 lines
        struct virtnet_info {
                ...
                struct send_queue sq[16] ____cacheline_aligned_in_smp;
                ...
        };

3. Migration is not tested.

4. Identify reasons for single netperf BW regression.

5. Test perf in more scenarious:
small packets
host -> guest
guest <-> external
in last case:
find some other way to measure host CPU utilization, try multiqueue and single queue devices

6. Use above to figure out what is a sane default for numtxqs.

Quoted 5 lines

Review/feedback appreciated.

---

#10 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 09/19/2010 06:14:43 PM:

Quoted 2 lines

Could you document how exactly do you measure multistream bandwidth:
netperf flags, etc?

All results were without any netperf flags or system tuning:
for i in $list
do
netperf -c -C -l 60 -H 192.168.122.1 > /tmp/netperf.$$.$i &
done
wait
Another script processes the result files. It also displays the start time/end time of each iteration to make sure skew due to parallel netperfs is minimal.

I changed the vhost functionality once more to try to get the best model, the new model being:
1. #numtxqs=1 -> #vhosts=1, this thread handles both RX/TX.
2. #numtxqs>1 -> vhost[0] handles RX and vhost[1-MAX] handles TX[0-n], where MAX is 4. Beyond numtxqs=4, the remaining TX queues are handled by vhost threads in round-robin fashion.

Results from here on are with these changes, and only "tuning" is to set each vhost's affinity to CPUs[0-3] ("taskset -p f <vhost-pids>").

Quoted 4 lines

Any idea where does this come from?
Do you see more TX interrupts? RX interrupts? Exits? Do interrupts bounce more between guest CPUs?
4. Identify reasons for single netperf BW regression.

After testing various combinations of #txqs, #vhosts, #netperf sessions, I think the drop for 1 stream is due to TX and RX for a flow being processed on different cpus. I did two more tests:
1. Pin vhosts to same CPU:
- BW drop is much lower for 1 stream case (- 5 to -8% range)
- But performance is not so high for more sessions.
2. Changed vhost to be single threaded:
- No degradation for 1 session, and improvement for upto 8, sometimes 16 streams (5-12%).
- BW degrades after that, all the way till 128 netperf sessions.
- But overall CPU utilization improves.
Summary of the entire run (for 1-128 sessions):
txq=4: BW: (-2.3) CPU: (-16.5) RCPU: (-5.3)
txq=16: BW: (-1.9) CPU: (-24.9) RCPU: (-9.6)

I don't see any reasons mentioned above. However, for higher number of netperf sessions, I see a big increase in retransmissions:
_______________________________________
#netperf ORG NEW
BW (#retr) BW (#retr)
_______________________________________

1          70244 (0)     64102 (0)
4          21421 (0)     36570 (416)
8          21746 (0)     38604 (148)
16         21783 (0)     40632 (464)
32         22677 (0)     37163 (1053)
64         23648 (4)     36449 (2197)
128        23251 (2)     31676 (3185)

_______________________________________

Single netperf case didn't have any retransmissions so that is not the cause for drop. I tested ixgbe (MQ):
___________________________________________________________
#netperf ixgbe ixgbe (pin intrs to cpu#0 on
both server/client)
BW (#retr) BW (#retr)
___________________________________________________________

1           3567 (117)          6000 (251)
2           4406 (477)          6298 (725)
4           6119 (1085)         7208 (3387)
8           6595 (4276)         7381 (15296)
16          6651 (11651)        6856 (30394)

___________________________________________________________

Quoted 2 lines

5. Test perf in more scenarious:
small packets

512 byte packets - BW drop for upto 8 (sometimes 16) netperf sessions, but increases with #sessions:
_______________________________________________________________________________
# BW1 BW2 (%) CPU1 CPU2 (%) RCPU1 RCPU2 (%)
_______________________________________________________________________________

1       4043    3800 (-6.0)     50      50 (0)          86      98 (13.9)
2       8358    7485 (-10.4)    153     178 (16.3)      230     264 (14.7)
4       20664   13567 (-34.3)   448     490 (9.3)       530     624 (17.7)
8       25198   17590 (-30.1)   967     1021 (5.5)      1085    1257 (15.8)
16      23791   24057 (1.1)     1904    2220 (16.5)     2156    2578 (19.5)
24      23055   26378 (14.4)    2807    3378 (20.3)     3225    3901 (20.9)
32      22873   27116 (18.5)    3748    4525 (20.7)     4307    5239 (21.6)
40      22876   29106 (27.2)    4705    5717 (21.5)     5388    6591 (22.3)
48      23099   31352 (35.7)    5642    6986 (23.8)     6475    8085 (24.8)
64      22645   30563 (34.9)    7527    9027 (19.9)     8619    10656 (23.6)
80      22497   31922 (41.8)    9375    11390 (21.4)    10736   13485 (25.6)
96      22509   32718 (45.3)    11271   13710 (21.6)    12927   16269 (25.8)
128     22255   32397 (45.5)    15036   18093 (20.3)    17144   21608 (26.0)

_______________________________________________________________________________
SUM: BW: (16.7) CPU: (20.6) RCPU: (24.3)
_______________________________________________________________________________

Quoted 1 line

host -> guest

_______________________________________________________________________________
# BW1 BW2 (%) CPU1 CPU2 (%) RCPU1 RCPU2 (%)
_______________________________________________________________________________

*1      70706   90398 (27.8)    300     327 (9.0)       140     175 (25.0)
2       20951   21937 (4.7)     188     196 (4.2)       93      103 (10.7)
4       19952   25281 (26.7)    397     496 (24.9)      210     304 (44.7)
8       18559   24992 (34.6)    802     1010 (25.9)     439     659 (50.1)
16      18882   25608 (35.6)    1642    2082 (26.7)     953     1454 (52.5)
24      19012   26955 (41.7)    2465    3153 (27.9)     1452    2254 (55.2)
32      19846   26894 (35.5)    3278    4238 (29.2)     1914    3081 (60.9)
40      19704   27034 (37.2)    4104    5303 (29.2)     2409    3866 (60.4)
48      19721   26832 (36.0)    4924    6418 (30.3)     2898    4701 (62.2)
64      19650   26849 (36.6)    6595    8611 (30.5)     3975    6433 (61.8)
80      19432   26823 (38.0)    8244    10817 (31.2)    4985    8165 (63.7)
96      20347   27886 (37.0)    9913    13017 (31.3)    5982    9860 (64.8)
128     19108   27715 (45.0)    13254   17546 (32.3)    8153    13589 (66.6)

_______________________________________________________________________________
SUM: BW: (32.4) CPU: (30.4) RCPU: (62.6)
_______________________________________________________________________________
*: Sum over 7 iterations, remaining test cases are sum over 2 iterations

Quoted 1 line

guest <-> external

I haven't done this right now since I don't have a setup. I guess it would be limited by wire speed and gains may not be there. I will try to do this later when I get the setup.

Quoted 4 lines

in last case:
find some other way to measure host CPU utilization, try multiqueue and single queue devices
6. Use above to figure out what is a sane default for numtxqs

A. Summary for default I/O (16K):

#txqs=2 (#vhost=3):       BW: (37.6)      CPU: (69.2)     RCPU: (40.8)
#txqs=4 (#vhost=5):       BW: (36.9)      CPU: (60.9)     RCPU: (25.2)
#txqs=8 (#vhost=5):       BW: (41.8)      CPU: (50.0)     RCPU: (15.2)
#txqs=16 (#vhost=5):      BW: (40.4)      CPU: (49.9)     RCPU: (10.0)

B. Summary for 512 byte I/O:

#txqs=2 (#vhost=3):       BW: (31.6)      CPU: (35.7)     RCPU: (28.6)
#txqs=4 (#vhost=5):       BW: (5.7)       CPU: (27.2)     RCPU: (22.7)
#txqs=8 (#vhost=5):       BW: (-.6)       CPU: (25.1)     RCPU: (22.5)
#txqs=16 (#vhost=5):      BW: (-6.6)      CPU: (24.7)     RCPU: (21.7)

Summary:

1. Average BW increase for regular I/O is best for #txq=16 with the least CPU utilization increase.
2. The average BW for 512 byte I/O is best for lower #txq=2. For higher #txqs, BW increased only after a particular #netperf sessions - in my testing that limit was 32 netperf sessions.
3. Multiple txq for guest by itself doesn't seem to have any issues. Guest CPU% increase is slightly higher than BW improvement. I think it is true for all mq drivers since more paths run in parallel upto the device instead of sleeping and allowing one thread to send all packets via qdisc_restart.
4. Having high number of txqs gives better gains and reduces cpu util on the guest and the host.
5. MQ is intended for server loads. MQ should probably not be explicitly specified for client systems.
6. No regression with numtxqs=1 (or if mq option is not used) in any testing scenario.

I will send the v3 patch within a day after some more testing.

Thanks,

- KK

#11 lore raw
Quoted 25 lines · On Tue, Oct 05, 2010 at 04:10:00PM +0530, Krishna Kumar2 wrote:

"Michael S. Tsirkin" <[email protected]> wrote on 09/19/2010 06:14:43 PM:

Quoted 2 lines

Could you document how exactly do you measure multistream bandwidth:
netperf flags, etc?

All results were without any netperf flags or system tuning:
for i in $list
do
netperf -c -C -l 60 -H 192.168.122.1 > /tmp/netperf.$$.$i &
done
wait
Another script processes the result files. It also displays the start time/end time of each iteration to make sure skew due to parallel netperfs is minimal.

I changed the vhost functionality once more to try to get the best model, the new model being:
1. #numtxqs=1 -> #vhosts=1, this thread handles both RX/TX.
2. #numtxqs>1 -> vhost[0] handles RX and vhost[1-MAX] handles TX[0-n], where MAX is 4. Beyond numtxqs=4, the remaining TX queues are handled by vhost threads in round-robin fashion.

Results from here on are with these changes, and only "tuning" is to set each vhost's affinity to CPUs[0-3] ("taskset -p f <vhost-pids>").

Quoted 8 lines
Quoted 4 lines

Any idea where does this come from?
Do you see more TX interrupts? RX interrupts? Exits? Do interrupts bounce more between guest CPUs?
4. Identify reasons for single netperf BW regression.

After testing various combinations of #txqs, #vhosts, #netperf sessions, I think the drop for 1 stream is due to TX and RX for a flow being processed on different cpus.

Right. Can we fix it?

Quoted 13 lines

I did two more tests:
1. Pin vhosts to same CPU:
- BW drop is much lower for 1 stream case (- 5 to -8% range)
- But performance is not so high for more sessions.
2. Changed vhost to be single threaded:
- No degradation for 1 session, and improvement for upto 8, sometimes 16 streams (5-12%).
- BW degrades after that, all the way till 128 netperf sessions.
- But overall CPU utilization improves.
Summary of the entire run (for 1-128 sessions):
txq=4: BW: (-2.3) CPU: (-16.5) RCPU: (-5.3)
txq=16: BW: (-1.9) CPU: (-24.9) RCPU: (-9.6)

Quoted 2 lines

I don't see any reasons mentioned above. However, for higher number of netperf sessions, I see a big increase in retransmissions:

Hmm, ok, and do you see any errors?

Quoted 20 lines

_______________________________________
#netperf ORG NEW
BW (#retr) BW (#retr)
_______________________________________

1          70244 (0)     64102 (0)
4          21421 (0)     36570 (416)
8          21746 (0)     38604 (148)
16         21783 (0)     40632 (464)
32         22677 (0)     37163 (1053)
64         23648 (4)     36449 (2197)
128        23251 (2)     31676 (3185)

_______________________________________

Single netperf case didn't have any retransmissions so that is not the cause for drop. I tested ixgbe (MQ):
___________________________________________________________
#netperf ixgbe ixgbe (pin intrs to cpu#0 on
both server/client)
BW (#retr) BW (#retr)
___________________________________________________________

Quoted 5 lines
1           3567 (117)          6000 (251)
2           4406 (477)          6298 (725)
4           6119 (1085)         7208 (3387)
8           6595 (4276)         7381 (15296)
16          6651 (11651)        6856 (30394)

Interesting.
You are saying we get much more retransmissions with physical nic as well?

Quoted 49 lines

___________________________________________________________

Quoted 2 lines

5. Test perf in more scenarious:
small packets

512 byte packets - BW drop for upto 8 (sometimes 16) netperf sessions, but increases with #sessions:
_______________________________________________________________________________
# BW1 BW2 (%) CPU1 CPU2 (%) RCPU1 RCPU2 (%)
_______________________________________________________________________________

1       4043    3800 (-6.0)     50      50 (0)          86      98 (13.9)
2       8358    7485 (-10.4)    153     178 (16.3)      230     264 (14.7)
4       20664   13567 (-34.3)   448     490 (9.3)       530     624 (17.7)
8       25198   17590 (-30.1)   967     1021 (5.5)      1085    1257 (15.8)
16      23791   24057 (1.1)     1904    2220 (16.5)     2156    2578 (19.5)
24      23055   26378 (14.4)    2807    3378 (20.3)     3225    3901 (20.9)
32      22873   27116 (18.5)    3748    4525 (20.7)     4307    5239 (21.6)
40      22876   29106 (27.2)    4705    5717 (21.5)     5388    6591 (22.3)
48      23099   31352 (35.7)    5642    6986 (23.8)     6475    8085 (24.8)
64      22645   30563 (34.9)    7527    9027 (19.9)     8619    10656 (23.6)
80      22497   31922 (41.8)    9375    11390 (21.4)    10736   13485 (25.6)
96      22509   32718 (45.3)    11271   13710 (21.6)    12927   16269 (25.8)
128     22255   32397 (45.5)    15036   18093 (20.3)    17144   21608 (26.0)

_______________________________________________________________________________
SUM: BW: (16.7) CPU: (20.6) RCPU: (24.3)
_______________________________________________________________________________

Quoted 1 line

host -> guest

_______________________________________________________________________________
# BW1 BW2 (%) CPU1 CPU2 (%) RCPU1 RCPU2 (%)
_______________________________________________________________________________

*1      70706   90398 (27.8)    300     327 (9.0)       140     175 (25.0)
2       20951   21937 (4.7)     188     196 (4.2)       93      103 (10.7)
4       19952   25281 (26.7)    397     496 (24.9)      210     304 (44.7)
8       18559   24992 (34.6)    802     1010 (25.9)     439     659 (50.1)
16      18882   25608 (35.6)    1642    2082 (26.7)     953     1454 (52.5)
24      19012   26955 (41.7)    2465    3153 (27.9)     1452    2254 (55.2)
32      19846   26894 (35.5)    3278    4238 (29.2)     1914    3081 (60.9)
40      19704   27034 (37.2)    4104    5303 (29.2)     2409    3866 (60.4)
48      19721   26832 (36.0)    4924    6418 (30.3)     2898    4701 (62.2)
64      19650   26849 (36.6)    6595    8611 (30.5)     3975    6433 (61.8)
80      19432   26823 (38.0)    8244    10817 (31.2)    4985    8165 (63.7)
96      20347   27886 (37.0)    9913    13017 (31.3)    5982    9860 (64.8)
128     19108   27715 (45.0)    13254   17546 (32.3)    8153    13589 (66.6)

_______________________________________________________________________________
SUM: BW: (32.4) CPU: (30.4) RCPU: (62.6)
_______________________________________________________________________________
*: Sum over 7 iterations, remaining test cases are sum over 2 iterations

Quoted 5 lines
Quoted 1 line

guest <-> external

I haven't done this right now since I don't have a setup. I guess it would be limited by wire speed and gains may not be there. I will try to do this later when I get the setup.

OK but at least need to check that it does not hurt things.

Quoted 19 lines
Quoted 4 lines

in last case:
find some other way to measure host CPU utilization, try multiqueue and single queue devices
6. Use above to figure out what is a sane default for numtxqs

A. Summary for default I/O (16K):

#txqs=2 (#vhost=3):       BW: (37.6)      CPU: (69.2)     RCPU: (40.8)
#txqs=4 (#vhost=5):       BW: (36.9)      CPU: (60.9)     RCPU: (25.2)
#txqs=8 (#vhost=5):       BW: (41.8)      CPU: (50.0)     RCPU: (15.2)
#txqs=16 (#vhost=5):      BW: (40.4)      CPU: (49.9)     RCPU: (10.0)

B. Summary for 512 byte I/O:

#txqs=2 (#vhost=3):       BW: (31.6)      CPU: (35.7)     RCPU: (28.6)
#txqs=4 (#vhost=5):       BW: (5.7)       CPU: (27.2)     RCPU: (22.7)
#txqs=8 (#vhost=5):       BW: (-.6)       CPU: (25.1)     RCPU: (22.5)
#txqs=16 (#vhost=5):      BW: (-6.6)      CPU: (24.7)     RCPU: (21.7)

Summary:

Quoted 16 lines

1. Average BW increase for regular I/O is best for #txq=16 with the least CPU utilization increase.
2. The average BW for 512 byte I/O is best for lower #txq=2. For higher #txqs, BW increased only after a particular #netperf sessions - in my testing that limit was 32 netperf sessions.
3. Multiple txq for guest by itself doesn't seem to have any issues. Guest CPU% increase is slightly higher than BW improvement. I think it is true for all mq drivers since more paths run in parallel upto the device instead of sleeping and allowing one thread to send all packets via qdisc_restart.
4. Having high number of txqs gives better gains and reduces cpu util on the guest and the host.
5. MQ is intended for server loads. MQ should probably not be explicitly specified for client systems.
6. No regression with numtxqs=1 (or if mq option is not used) in any testing scenario.

Of course txq=1 can be considered a kind of fix, but if we know the issue is TX/RX flows getting bounced between CPUs, can we fix this? Workload-specific optimizations can only get us this far.

Quoted 6 lines

I will send the v3 patch within a day after some more testing.

Thanks,

- KK

#12 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 10/05/2010 11:53:23 PM:

Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

Any idea where does this come from?
Do you see more TX interrupts? RX interrupts? Exits? Do interrupts bounce more between guest CPUs?
4. Identify reasons for single netperf BW regression.

After testing various combinations of #txqs, #vhosts, #netperf sessions, I think the drop for 1 stream is due to TX and RX for a flow being processed on different cpus.

Right. Can we fix it?

I am not sure how to. My initial patch had one thread but gave small gains and ran into limitations once number of sessions became large.

Quoted 8 lines
Quoted 8 lines

I did two more tests:
1. Pin vhosts to same CPU:
- BW drop is much lower for 1 stream case (- 5 to -8% range)
- But performance is not so high for more sessions.
2. Changed vhost to be single threaded:
- No degradation for 1 session, and improvement for upto
8, sometimes 16 streams (5-12%).
- BW degrades after that, all the way till 128 netperf

sessions.

Quoted 9 lines
Quoted 7 lines

- But overall CPU utilization improves. Summary of the entire run (for 1-128 sessions):
txq=4: BW: (-2.3) CPU: (-16.5) RCPU: (-5.3)
txq=16: BW: (-1.9) CPU: (-24.9) RCPU: (-9.6)

I don't see any reasons mentioned above. However, for higher number of netperf sessions, I see a big increase in retransmissions:

Hmm, ok, and do you see any errors?

I haven't seen any in any statistics, messages, etc. Also no retranmissions for txq=1.

Quoted 13 lines
Quoted 7 lines

Single netperf case didn't have any retransmissions so that is not the cause for drop. I tested ixgbe (MQ):
___________________________________________________________
#netperf ixgbe ixgbe (pin intrs to cpu#0 on
both server/client)
BW (#retr) BW (#retr)
___________________________________________________________

Quoted 5 lines
1           3567 (117)          6000 (251)
2           4406 (477)          6298 (725)
4           6119 (1085)         7208 (3387)
8           6595 (4276)         7381 (15296)
16          6651 (11651)        6856 (30394)
Quoted 3 lines

Interesting.
You are saying we get much more retransmissions with physical nic as well?

Yes, with ixgbe. I re-ran with 16 netperfs running for 15 secs on both ixgbe and cxgb3 just now to reconfirm:

ixgbe: BW: 6186.85 SD/Remote: 135.711, 339.376 CPU/Remote: 79.99, 200.00,
Retrans: 545
cxgb3: BW: 8051.07 SD/Remote: 144.416, 260.487 CPU/Remote: 110.88, 200.00, Retrans: 0

However 64 netperfs for 30 secs gave:

ixgbe: BW: 6691.12 SD/Remote: 8046.617, 5259.992 CPU/Remote: 1223.86, 799.97, Retrans: 1424
cxgb3: BW: 7799.16 SD/Remote: 2589.875, 4317.013 CPU/Remote: 480.39 800.64, Retrans: 649

# ethtool -i eth4
driver: ixgbe
version: 2.0.84-k2
firmware-version: 0.9-3
bus-info: 0000:1f:00.1
# ifconfig output:
       RX packets:783241 errors:0 dropped:0 overruns:0 frame:0
       TX packets:689533 errors:0 dropped:0 overruns:0 carrier:0
       collisions:0 txqueuelen:1000
# lspci output:
1f:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network
Connec
tion (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2
        Flags: bus master, fast devsel, latency 0, IRQ 30
        Memory at 98900000 (64-bit, prefetchable) [size=512K]
        I/O ports at 2020 [size=32]
        Memory at 98a00000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-40-4a-b4
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
Quoted 5 lines
Quoted 3 lines

I haven't done this right now since I don't have a setup. I guess it would be limited by wire speed and gains may not be there. I will try to do this later when I get the setup.

OK but at least need to check that it does not hurt things.

Yes, sure.

Quoted 15 lines
Quoted 15 lines

Summary:

1. Average BW increase for regular I/O is best for #txq=16 with the least CPU utilization increase.
2. The average BW for 512 byte I/O is best for lower #txq=2. For higher #txqs, BW increased only after a particular #netperf sessions - in my testing that limit was 32 netperf sessions.
3. Multiple txq for guest by itself doesn't seem to have any issues. Guest CPU% increase is slightly higher than BW improvement. I think it is true for all mq drivers since more paths run in parallel upto the device instead of sleeping and allowing one thread to send all packets via qdisc_restart.
4. Having high number of txqs gives better gains and reduces cpu util on the guest and the host.
5. MQ is intended for server loads. MQ should probably not be

explicitly

Quoted 7 lines
Quoted 3 lines

specified for client systems.
6. No regression with numtxqs=1 (or if mq option is not used) in any testing scenario.

Of course txq=1 can be considered a kind of fix, but if we know the issue is TX/RX flows getting bounced between CPUs, can we fix this? Workload-specific optimizations can only get us this far.

I will test with your patch tomorrow night once I am back.

Thanks,

- KK

#13 lore raw
Quoted 11 lines · On Wed, Oct 06, 2010 at 11:13:31PM +0530, Krishna Kumar2 wrote:

"Michael S. Tsirkin" <[email protected]> wrote on 10/05/2010 11:53:23 PM:

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

Any idea where does this come from?
Do you see more TX interrupts? RX interrupts? Exits? Do interrupts bounce more between guest CPUs?
4. Identify reasons for single netperf BW regression.

After testing various combinations of #txqs, #vhosts, #netperf sessions, I think the drop for 1 stream is due to TX and RX for a flow being processed on different cpus.

Quoted 5 lines
Quoted 1 line

Right. Can we fix it?

I am not sure how to. My initial patch had one thread but gave small gains and ran into limitations once number of sessions became large.

Sure. We will need multiple RX queues, and have a single thread handle a TX and RX pair. Then we need to make sure packets from a given flow on TX land on the same thread on RX. As flows can be hashed differently, for this to work we'll have to expose this info in host/guest interface.
But since multiqueue implies host/guest ABI changes anyway, this point is moot.

BTW, an interesting approach could be using bonding and multiple virtio-net interfaces.
What are the disadvantages of such a setup? One advantage is it can be made to work in existing guests.

Quoted 14 lines
Quoted 8 lines
Quoted 8 lines

I did two more tests:
1. Pin vhosts to same CPU:
- BW drop is much lower for 1 stream case (- 5 to -8% range)
- But performance is not so high for more sessions.
2. Changed vhost to be single threaded:
- No degradation for 1 session, and improvement for upto
8, sometimes 16 streams (5-12%).
- BW degrades after that, all the way till 128 netperf

sessions.

Quoted 5 lines
Quoted 5 lines

- But overall CPU utilization improves. Summary of the entire run (for 1-128 sessions):
txq=4: BW: (-2.3) CPU: (-16.5) RCPU: (-5.3)
txq=16: BW: (-1.9) CPU: (-24.9) RCPU: (-9.6)

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

I don't see any reasons mentioned above. However, for higher number of netperf sessions, I see a big increase in retransmissions:

Hmm, ok, and do you see any errors?

I haven't seen any in any statistics, messages, etc.

Herbert, could you help out debugging this increase in retransmissions please? Older mail on netdev in this thread has some numbers that seem to imply that we start hitting retransmissions much more as # of flows goes up.

Quoted 2 lines

Also no
retranmissions for txq=1.

While it's nice that we have this parameter, the need to choose between single stream and multi stream performance when you start the vm makes this patch much less interesting IMHO.

Signature · 1 line
MST
#14 lore raw
In reply to Krishna Kumar2
Quoted 21 lines · On Tuesday 05 October 2010, Krishna Kumar2 wrote:

After testing various combinations of #txqs, #vhosts, #netperf sessions, I think the drop for 1 stream is due to TX and RX for a flow being processed on different cpus. I did two more tests:
1. Pin vhosts to same CPU:
- BW drop is much lower for 1 stream case (- 5 to -8% range)
- But performance is not so high for more sessions.
2. Changed vhost to be single threaded:
- No degradation for 1 session, and improvement for upto 8, sometimes 16 streams (5-12%).
- BW degrades after that, all the way till 128 netperf sessions.
- But overall CPU utilization improves.
Summary of the entire run (for 1-128 sessions):
txq=4: BW: (-2.3) CPU: (-16.5) RCPU: (-5.3)
txq=16: BW: (-1.9) CPU: (-24.9) RCPU: (-9.6)

I don't see any reasons mentioned above. However, for higher number of netperf sessions, I see a big increase in retransmissions:
_______________________________________
#netperf ORG NEW
BW (#retr) BW (#retr)
_______________________________________

Quoted 8 lines
1          70244 (0)     64102 (0)
4          21421 (0)     36570 (416)
8          21746 (0)     38604 (148)
16         21783 (0)     40632 (464)
32         22677 (0)     37163 (1053)
64         23648 (4)     36449 (2197)
128        23251 (2)     31676 (3185)

_______________________________________

This smells like it could be related to a problem that Ben Greear found recently (see "macvlan: Enable qdisc backoff logic"). When the hardware is busy, used to just drop the packet. With Ben's patch, we return -EAGAIN to qemu (or vhost-net) to trigger a resend.

I suppose what we really should do is feed that condition back to the guest network stack and implement the backoff in there.

Arnd

#15 lore raw

Arnd Bergmann <[email protected]> wrote on 10/06/2010 05:49:00 PM:

Quoted 16 lines
Quoted 6 lines

I don't see any reasons mentioned above. However, for higher number of netperf sessions, I see a big increase in retransmissions:
_______________________________________
#netperf ORG NEW
BW (#retr) BW (#retr)
_______________________________________

Quoted 8 lines
1          70244 (0)     64102 (0)
4          21421 (0)     36570 (416)
8          21746 (0)     38604 (148)
16         21783 (0)     40632 (464)
32         22677 (0)     37163 (1053)
64         23648 (4)     36449 (2197)
128        23251 (2)     31676 (3185)

_______________________________________

Quoted 3 lines

This smells like it could be related to a problem that Ben Greear found recently (see "macvlan: Enable qdisc backoff logic"). When the hardware is busy, used to just drop the packet. With Ben's patch, we return

-EAGAIN

Quoted 4 lines

to qemu (or vhost-net) to trigger a resend.

I suppose what we really should do is feed that condition back to the guest network stack and implement the backoff in there.

Thanks for the pointer. I will take a look at this as I hadn't seen this patch earlier. Is there any way to figure out if this is the issue?

Thanks,

- KK

#16 lore raw
Quoted 22 lines · On Wednesday 06 October 2010 19:14:42 Krishna Kumar2 wrote:

Arnd Bergmann <[email protected]> wrote on 10/06/2010 05:49:00 PM:

Quoted 16 lines
Quoted 6 lines

I don't see any reasons mentioned above. However, for higher number of netperf sessions, I see a big increase in retransmissions:
_______________________________________
#netperf ORG NEW
BW (#retr) BW (#retr)
_______________________________________

Quoted 8 lines
1          70244 (0)     64102 (0)
4          21421 (0)     36570 (416)
8          21746 (0)     38604 (148)
16         21783 (0)     40632 (464)
32         22677 (0)     37163 (1053)
64         23648 (4)     36449 (2197)
128        23251 (2)     31676 (3185)

_______________________________________

Quoted 3 lines

This smells like it could be related to a problem that Ben Greear found recently (see "macvlan: Enable qdisc backoff logic"). When the hardware is busy, used to just drop the packet. With Ben's patch, we return

-EAGAIN

Quoted 8 lines
Quoted 4 lines

to qemu (or vhost-net) to trigger a resend.

I suppose what we really should do is feed that condition back to the guest network stack and implement the backoff in there.

Thanks for the pointer. I will take a look at this as I hadn't seen this patch earlier. Is there any way to figure out if this is the issue?

I think a good indication would be if this changes with/without the patch, and if you see -EAGAIN in qemu with the patch applied.

Arnd