Skip to contentSkip to conversation

virtio-net: use mtu size as buffer length for big packets

PATCH In review Gavin Li 5 participants 54 messages Aug 2, 2022 – Aug 11, 2022

Subsystems (3)
lore
Share & tools
lore

Gavin Li

author
#0 lore raw

PATCH 1/1 virtio-net: use mtu size as buffer length for big packets

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

  1500        22.5     22.4
  9000        12.8     25.9
---
 drivers/net/virtio_net.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +16 -4
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
 	/* I like... big packets and I cannot lie! */
 	bool big_packets;
 
+	/* Indicates GSO support */
+	bool gso_is_supported;
+
 	/* Host will merge rx buffers for big packets (shake it! shake it!) */
 	bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
 static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 			   gfp_t gfp)
 {
+	unsigned int sg_num = MAX_SKB_FRAGS;
 	struct page *first, *list = NULL;
 	char *p;
 	int i, err, offset;
 
-	sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+	if (!vi->gso_is_supported) {
+		unsigned int mtu = vi->dev->mtu;
+
+		sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;
+	}
+
+	sg_init_table(rq->sg, sg_num + 2);
 
 	/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
-	for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+	for (i = sg_num + 1; i > 1; --i) {
 		first = get_a_page(rq, gfp);
 		if (!first) {
 			if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 
 	/* chain first in list head */
 	first->private = (unsigned long)list;
-	err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+	err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
 				  first, gfp);
 	if (err < 0)
 		give_pages(rq, first);
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
 	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
 	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
 		vi->big_packets = true;
+		vi->gso_is_supported = true;
+	}
 
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
 		vi->mergeable_rx_bufs = true;
Signature · 6 lines
2.31.1


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#1 lore raw
Quoted 76 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

  1500        22.5     22.4
  9000        12.8     25.9
---
 drivers/net/virtio_net.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +13 -3
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
        /* I like... big packets and I cannot lie! */
        bool big_packets;
 
+       /* Indicates GSO support */
+       bool gso_is_supported;
+
        /* Host will merge rx buffers for big packets (shake it! shake it!) */
        bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
 static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                           gfp_t gfp)
 {
+       unsigned int sg_num = MAX_SKB_FRAGS;
        struct page *first, *list = NULL;
        char *p;
        int i, err, offset;
 
-       sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+       if (!vi->gso_is_supported) {
+               unsigned int mtu = vi->dev->mtu;
+
+               sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;
+       }
+
+       sg_init_table(rq->sg, sg_num + 2);
 
        /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
-       for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+       for (i = sg_num + 1; i > 1; --i) {
                first = get_a_page(rq, gfp);
                if (!first) {
                        if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 
        /* chain first in list head */
        first->private = (unsigned long)list;
-       err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+       err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                  first, gfp);
        if (err < 0)
                give_pages(rq, first);

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586

Quoted 7 lines
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)
        if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
                vi->big_packets = true;
+               vi->gso_is_supported = true;

Why not simply re-use big_packets here?

Thanks

Quoted 8 lines
drivers/net/virtio_net.c +1 -1
+       }
 
        if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
                vi->mergeable_rx_bufs = true;
--

2.31.1

Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#2 lore raw
Quoted 87 lines · Jason Wang
Quoted 76 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

  1500        22.5     22.4
  9000        12.8     25.9
---
 drivers/net/virtio_net.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +13 -3
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
        /* I like... big packets and I cannot lie! */
        bool big_packets;
 
+       /* Indicates GSO support */
+       bool gso_is_supported;
+
        /* Host will merge rx buffers for big packets (shake it! shake it!) */
        bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
 static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                           gfp_t gfp)
 {
+       unsigned int sg_num = MAX_SKB_FRAGS;
        struct page *first, *list = NULL;
        char *p;
        int i, err, offset;
 
-       sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+       if (!vi->gso_is_supported) {
+               unsigned int mtu = vi->dev->mtu;
+
+               sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;
+       }
+
+       sg_init_table(rq->sg, sg_num + 2);
 
        /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
-       for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+       for (i = sg_num + 1; i > 1; --i) {
                first = get_a_page(rq, gfp);
                if (!first) {
                        if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 
        /* chain first in list head */
        first->private = (unsigned long)list;
-       err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+       err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                  first, gfp);
        if (err < 0)
                give_pages(rq, first);

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586

Quoted 7 lines
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)
        if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
                vi->big_packets = true;
+               vi->gso_is_supported = true;

Why not simply re-use big_packets here?

Quoted 1 line

Thanks

I don't get this question. The patch does use big_packets, it wants to figure out guest GSO is off so MTU limits the size. The name "gso_is_supported" is confusing, should be e.g. guest_gso.

Quoted 8 lines
Quoted 8 lines
drivers/net/virtio_net.c +1 -1
+       }
 
        if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
                vi->mergeable_rx_bufs = true;
--

2.31.1

Quoted 4 lines
Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#3 lore raw
Quoted 89 lines · Michael S. Tsirkin
Quoted 87 lines · Jason Wang
Quoted 76 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

  1500        22.5     22.4
  9000        12.8     25.9
---
 drivers/net/virtio_net.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +13 -3
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
        /* I like... big packets and I cannot lie! */
        bool big_packets;
 
+       /* Indicates GSO support */
+       bool gso_is_supported;
+
        /* Host will merge rx buffers for big packets (shake it! shake it!) */
        bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
 static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                           gfp_t gfp)
 {
+       unsigned int sg_num = MAX_SKB_FRAGS;
        struct page *first, *list = NULL;
        char *p;
        int i, err, offset;
 
-       sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+       if (!vi->gso_is_supported) {
+               unsigned int mtu = vi->dev->mtu;
+
+               sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;
+       }
+
+       sg_init_table(rq->sg, sg_num + 2);
 
        /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
-       for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+       for (i = sg_num + 1; i > 1; --i) {
                first = get_a_page(rq, gfp);
                if (!first) {
                        if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 
        /* chain first in list head */
        first->private = (unsigned long)list;
-       err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+       err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                  first, gfp);
        if (err < 0)
                give_pages(rq, first);

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586

Quoted 7 lines
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)
        if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
                vi->big_packets = true;
+               vi->gso_is_supported = true;

Why not simply re-use big_packets here?

Quoted 4 lines
Quoted 1 line

Thanks

I don't get this question. The patch does use big_packets, it wants to figure out guest GSO is off so MTU limits the size.

Yes.

Thanks

Quoted 16 lines

The name "gso_is_supported" is confusing, should be e.g. guest_gso.

Quoted 12 lines
Quoted 12 lines
drivers/net/virtio_net.c +1 -1
+       }
 
        if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
                vi->mergeable_rx_bufs = true;
--

2.31.1

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#4 lore raw
Quoted 91 lines · Jason Wang
Quoted 89 lines · Michael S. Tsirkin
Quoted 87 lines · Jason Wang
Quoted 76 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

  1500        22.5     22.4
  9000        12.8     25.9
---
 drivers/net/virtio_net.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +13 -3
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
        /* I like... big packets and I cannot lie! */
        bool big_packets;
 
+       /* Indicates GSO support */
+       bool gso_is_supported;
+
        /* Host will merge rx buffers for big packets (shake it! shake it!) */
        bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
 static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                           gfp_t gfp)
 {
+       unsigned int sg_num = MAX_SKB_FRAGS;
        struct page *first, *list = NULL;
        char *p;
        int i, err, offset;
 
-       sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+       if (!vi->gso_is_supported) {
+               unsigned int mtu = vi->dev->mtu;
+
+               sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;
+       }
+
+       sg_init_table(rq->sg, sg_num + 2);
 
        /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
-       for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+       for (i = sg_num + 1; i > 1; --i) {
                first = get_a_page(rq, gfp);
                if (!first) {
                        if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 
        /* chain first in list head */
        first->private = (unsigned long)list;
-       err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+       err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                  first, gfp);
        if (err < 0)
                give_pages(rq, first);

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586

Quoted 7 lines
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)
        if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
                vi->big_packets = true;
+               vi->gso_is_supported = true;

Why not simply re-use big_packets here?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Thanks

I don't get this question. The patch does use big_packets, it wants to figure out guest GSO is off so MTU limits the size.

Yes.

Thanks

I wonder if it's better to introduce the boolean here:

        /* TODO: size buffers correctly in this case. */
                if (dev->mtu > ETH_DATA_LEN)
                        vi->big_packets = true;

Thanks

Quoted 12 lines
Quoted 12 lines

The name "gso_is_supported" is confusing, should be e.g. guest_gso.

Quoted 8 lines
Quoted 8 lines
drivers/net/virtio_net.c +1 -1
+       }
 
        if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
                vi->mergeable_rx_bufs = true;
--

2.31.1

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#7 lore raw
In reply to Gavin Li
Quoted 3 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

Quoted 7 lines

it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

Quoted 37 lines

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

   1500        22.5     22.4
   9000        12.8     25.9
---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----
  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
  	/* I like... big packets and I cannot lie! */
  	bool big_packets;
 
+	/* Indicates GSO support */
+	bool gso_is_supported;
+
  	/* Host will merge rx buffers for big packets (shake it! shake it!) */
  	bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
  static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
  			   gfp_t gfp)
  {

drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, · line 1326

Quoted 10 lines
drivers/net/virtio_net.c +5 -1
+	unsigned int sg_num = MAX_SKB_FRAGS;
  	struct page *first, *list = NULL;
  	char *p;
  	int i, err, offset;
 
-	sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+	if (!vi->gso_is_supported) {
+		unsigned int mtu = vi->dev->mtu;
+
+		sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;

DIV_ROUND_UP() can be used?

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, · line 1331

Quoted 5 lines
drivers/net/virtio_net.c +3 -0
+	}
+
+	sg_init_table(rq->sg, sg_num + 2);
 
  	/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

Quoted 15 lines
drivers/net/virtio_net.c +2 -2
-	for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+	for (i = sg_num + 1; i > 1; --i) {
  		first = get_a_page(rq, gfp);
  		if (!first) {
  			if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 
  	/* chain first in list head */
  	first->private = (unsigned long)list;
-	err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+	err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
  				  first, gfp);
  	if (err < 0)
  		give_pages(rq, first);

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586

Quoted 7 lines
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)
  	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
  	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
  	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
  		vi->big_packets = true;
+		vi->gso_is_supported = true;

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

Thanks,
-Siwei

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3590

Quoted 4 lines
drivers/net/virtio_net.c +1 -0
+	}
 
  	if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
  		vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#8 lore raw
Quoted 64 lines · Si-Wei Liu
Quoted 3 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

Quoted 7 lines

  it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

Quoted 37 lines

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
              Before   After
   1500        22.5     22.4
   9000        12.8     25.9

---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----

  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +0 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
       /* I like... big packets and I cannot lie! */
       bool big_packets;

  +    /* Indicates GSO support */
+    bool gso_is_supported;
+
      /* Host will merge rx buffers for big packets (shake it! shake it!) */
      bool mergeable_rx_bufs;
  @@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                 gfp_t gfp)
  {
+    unsigned int sg_num = MAX_SKB_FRAGS;
      struct page *first, *list = NULL;

Quoted 8 lines

      char *p;
      int i, err, offset;
  -    sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+    if (!vi->gso_is_supported) {
+        unsigned int mtu = vi->dev->mtu;
+
+        sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;

DIV_ROUND_UP() can be used?

Quoted 1 line

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time

... , but can align with new mtu during .ndo_change_mtu(), too.

Quoted 1 line

) in struct virtnet_info?

Thanks,
-Siwei

Quoted 31 lines
Quoted 4 lines

+    }
+
+    sg_init_table(rq->sg, sg_num + 2);
        /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

Quoted 16 lines

-    for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+    for (i = sg_num + 1; i > 1; --i) {
          first = get_a_page(rq, gfp);
          if (!first) {
              if (list)

drivers/net/virtio_net.c +1 -1
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info 
 *vi, struct receive_queue *rq,
         /* chain first in list head */
       first->private = (unsigned long)list;
-    err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+    err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                     first, gfp);
       if (err < 0)
           give_pages(rq, first);
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device 
 *vdev)
Quoted 7 lines

      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) || -        virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +        virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) { vi->big_packets = true;
+        vi->gso_is_supported = true;

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

Quoted 6 lines

Thanks,
-Siwei

Quoted 3 lines

+    }
        if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#6 lore raw
In reply to Michael S. Tsirkin
Quoted 89 lines · Michael S. Tsirkin

External email: Use caution opening links or attachments

Quoted 85 lines · Jason Wang
Quoted 75 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

   1500        22.5     22.4
   9000        12.8     25.9
---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----
  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +13 -3
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
         /* I like... big packets and I cannot lie! */
         bool big_packets;
 
+       /* Indicates GSO support */
+       bool gso_is_supported;
+
         /* Host will merge rx buffers for big packets (shake it! shake it!) */
         bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
  static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                            gfp_t gfp)
  {
+       unsigned int sg_num = MAX_SKB_FRAGS;
         struct page *first, *list = NULL;
         char *p;
         int i, err, offset;
 
-       sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+       if (!vi->gso_is_supported) {
+               unsigned int mtu = vi->dev->mtu;
+
+               sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;
+       }
+
+       sg_init_table(rq->sg, sg_num + 2);
 
         /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
-       for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+       for (i = sg_num + 1; i > 1; --i) {
                 first = get_a_page(rq, gfp);
                 if (!first) {
                         if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 
         /* chain first in list head */
         first->private = (unsigned long)list;
-       err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+       err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                   first, gfp);
         if (err < 0)
                 give_pages(rq, first);

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586

Quoted 7 lines
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)
         if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
             virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
             virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
                 vi->big_packets = true;
+               vi->gso_is_supported = true;

Why not simply re-use big_packets here?

Quoted 4 lines
Quoted 1 line

Thanks

I don't get this question. The patch does use big_packets, it wants to figure out guest GSO is off so MTU limits the size. The name "gso_is_supported" is confusing, should be e.g. guest_gso.

ACK.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/net/virtio_net.c +1 -1
+       }
 
         if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
                 vi->mergeable_rx_bufs = true;
--

2.31.1

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#5 lore raw
In reply to Jason Wang
Quoted 96 lines · Jason Wang

External email: Use caution opening links or attachments

Quoted 92 lines · Jason Wang
Quoted 86 lines · Michael S. Tsirkin
Quoted 85 lines · Jason Wang
Quoted 75 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

   1500        22.5     22.4
   9000        12.8     25.9
---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----
  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +13 -3
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
         /* I like... big packets and I cannot lie! */
         bool big_packets;
 
+       /* Indicates GSO support */
+       bool gso_is_supported;
+
         /* Host will merge rx buffers for big packets (shake it! shake it!) */
         bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
  static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                            gfp_t gfp)
  {
+       unsigned int sg_num = MAX_SKB_FRAGS;
         struct page *first, *list = NULL;
         char *p;
         int i, err, offset;
 
-       sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+       if (!vi->gso_is_supported) {
+               unsigned int mtu = vi->dev->mtu;
+
+               sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;
+       }
+
+       sg_init_table(rq->sg, sg_num + 2);
 
         /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
-       for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+       for (i = sg_num + 1; i > 1; --i) {
                 first = get_a_page(rq, gfp);
                 if (!first) {
                         if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
 
         /* chain first in list head */
         first->private = (unsigned long)list;
-       err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+       err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                   first, gfp);
         if (err < 0)
                 give_pages(rq, first);

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586

Quoted 7 lines
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)
         if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
             virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
             virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
                 vi->big_packets = true;
+               vi->gso_is_supported = true;

Why not simply re-use big_packets here?

Quoted 3 lines
Quoted 1 line

Thanks

I don't get this question. The patch does use big_packets, it wants to figure out guest GSO is off so MTU limits the size.

Yes.

Quoted 8 lines
Quoted 1 line

Thanks

I wonder if it's better to introduce the boolean here:

         /* TODO: size buffers correctly in this case. */
                 if (dev->mtu > ETH_DATA_LEN)
                         vi->big_packets = true;

Thanks

This is not a safe or straightforward way to determine that guest GSO is not supported, as big_packets just indicates the packets can be big, not specifically guest GSO.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

The name "gso_is_supported" is confusing, should be e.g. guest_gso.

Quoted 8 lines
Quoted 8 lines
drivers/net/virtio_net.c +1 -1
+       }
 
         if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
                 vi->mergeable_rx_bufs = true;
--

2.31.1

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#10 lore raw
In reply to Si-Wei Liu
Quoted 9 lines · Si-Wei Liu

External email: Use caution opening links or attachments

Quoted 3 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

Quoted 9 lines
Quoted 8 lines

  it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

Quoted 44 lines
Quoted 44 lines

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
              Before   After
   1500        22.5     22.4
   9000        12.8     25.9

---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----

  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +4 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
       /* I like... big packets and I cannot lie! */
       bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
       /* Host will merge rx buffers for big packets (shake it! shake it!) */
       bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct 
 virtnet_info *vi, struct receive_queue *rq,
   static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                          gfp_t gfp)
   {
+     unsigned int sg_num = MAX_SKB_FRAGS;
       struct page *first, *list = NULL;
       char *p;
       int i, err, offset;
 
Quoted 7 lines

drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,

Quoted 6 lines
drivers/net/virtio_net.c +4 -1
-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu 
 / PAGE_SIZE;

DIV_ROUND_UP() can be used?

ACK

Quoted 4 lines

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Quoted 6 lines

drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, · line 1331

Quoted 5 lines
drivers/net/virtio_net.c +3 -0
+     }
+
+     sg_init_table(rq->sg, sg_num + 2);
 
       /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

Quoted 20 lines
Quoted 20 lines
drivers/net/virtio_net.c +1 -1
-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
               first = get_a_page(rq, gfp);

              if (!first) {
                      if (list)

drivers/net/virtio_net.c +1 -1
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info 
 *vi, struct receive_queue *rq,
 
       /* chain first in list head */
       first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                 first, gfp);
       if (err < 0)

              give_pages(rq, first);

drivers/net/virtio_net.c +0 -0
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device 
 *vdev)

      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||

Quoted 8 lines
Quoted 4 lines

-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) { vi->big_packets = true;
+             vi->gso_is_supported = true;

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

Quoted 9 lines

Thanks,
-Siwei

Quoted 4 lines

+     }

      if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#9 lore raw
In reply to Si-Wei Liu
Quoted 72 lines · Si-Wei Liu

External email: Use caution opening links or attachments

Quoted 68 lines · Si-Wei Liu
Quoted 3 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

Quoted 7 lines

  it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

Quoted 40 lines

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
              Before   After
   1500        22.5     22.4
   9000        12.8     25.9

---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----

  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +0 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
       /* I like... big packets and I cannot lie! */
       bool big_packets;

  +    /* Indicates GSO support */
+    bool gso_is_supported;
+
      /* Host will merge rx buffers for big packets (shake it! shake it!) */
      bool mergeable_rx_bufs;
  @@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
  static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                 gfp_t gfp)
  {
+    unsigned int sg_num = MAX_SKB_FRAGS;
      struct page *first, *list = NULL;

Quoted 8 lines

      char *p;
      int i, err, offset;
  -    sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+    if (!vi->gso_is_supported) {
+        unsigned int mtu = vi->dev->mtu;
+
+        sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;

DIV_ROUND_UP() can be used?

Quoted 4 lines
Quoted 3 lines

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time

... , but can align with new mtu during .ndo_change_mtu(), too.

ACK but don't know how to do this.

Quoted 46 lines
Quoted 1 line

) in struct virtnet_info?

Thanks,
-Siwei

Quoted 40 lines
Quoted 4 lines

+    }
+
+    sg_init_table(rq->sg, sg_num + 2);
        /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

Quoted 16 lines

-    for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+    for (i = sg_num + 1; i > 1; --i) {
          first = get_a_page(rq, gfp);
          if (!first) {
              if (list)

drivers/net/virtio_net.c +1 -1
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
         /* chain first in list head */
       first->private = (unsigned long)list;
-    err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+    err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                     first, gfp);
       if (err < 0)
           give_pages(rq, first);
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)
Quoted 7 lines

      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) || -        virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +        virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) { vi->big_packets = true;
+        vi->gso_is_supported = true;

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

Thanks,
-Siwei

Quoted 3 lines

+    }
        if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#11 lore raw
In reply to Gavin Li
Quoted 73 lines · Gavin Li
Quoted 9 lines · Si-Wei Liu

External email: Use caution opening links or attachments

Quoted 3 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

Quoted 9 lines
Quoted 8 lines

  it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

Quoted 44 lines
Quoted 44 lines

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
              Before   After
   1500        22.5     22.4
   9000        12.8     25.9

---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----

  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +4 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
       /* I like... big packets and I cannot lie! */
       bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
       /* Host will merge rx buffers for big packets (shake it! shake it!) */
       bool mergeable_rx_bufs;
 
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct 
 virtnet_info *vi, struct receive_queue *rq,
   static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                          gfp_t gfp)
   {
+     unsigned int sg_num = MAX_SKB_FRAGS;
       struct page *first, *list = NULL;
       char *p;
       int i, err, offset;
 
Quoted 7 lines

drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,

Quoted 6 lines
drivers/net/virtio_net.c +4 -1
-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu 
 / PAGE_SIZE;

DIV_ROUND_UP() can be used?

Quoted 6 lines

ACK

Quoted 4 lines

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.

Quoted 35 lines
Quoted 6 lines

drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, · line 1331

Quoted 5 lines
drivers/net/virtio_net.c +3 -0
+     }
+
+     sg_init_table(rq->sg, sg_num + 2);
 
       /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

Quoted 20 lines
Quoted 20 lines
drivers/net/virtio_net.c +1 -1
-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
               first = get_a_page(rq, gfp);

              if (!first) {
                      if (list)

drivers/net/virtio_net.c +1 -1
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info 
 *vi, struct receive_queue *rq,
 
       /* chain first in list head */
       first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                 first, gfp);
       if (err < 0)

              give_pages(rq, first);

drivers/net/virtio_net.c +0 -0
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device 
 *vdev)

      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||

Quoted 8 lines
Quoted 4 lines

-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) { vi->big_packets = true;
+             vi->gso_is_supported = true;

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

Quoted 1 line

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Quoted 9 lines
Quoted 9 lines

Thanks,
-Siwei

Quoted 4 lines

+     }

      if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#12 lore raw
Quoted 88 lines · Si-Wei Liu

External email: Use caution opening links or attachments

Quoted 77 lines · Gavin Li
Quoted 9 lines · Si-Wei Liu

External email: Use caution opening links or attachments

Quoted 3 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

Quoted 9 lines
Quoted 8 lines

  it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

Quoted 46 lines
Quoted 46 lines

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
              Before   After
   1500        22.5     22.4
   9000        12.8     25.9

---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----

  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
       /* I like... big packets and I cannot lie! */
       bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
       /* Host will merge rx buffers for big packets (shake it! shake

it!) */
      bool mergeable_rx_bufs;

drivers/net/virtio_net.c +0 -0
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct
 virtnet_info *vi, struct receive_queue *rq,

  static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                         gfp_t gfp)
  {
+     unsigned int sg_num = MAX_SKB_FRAGS;
      struct page *first, *list = NULL;
      char *p;
      int i, err, offset;

Quoted 7 lines
Quoted 6 lines

-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;

DIV_ROUND_UP() can be used?

ACK

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Quoted 8 lines

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.

a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.

Quoted 54 lines
Quoted 36 lines
Quoted 7 lines
Quoted 6 lines

+     }
+
+     sg_init_table(rq->sg, sg_num + 2);

      /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

Quoted 20 lines
Quoted 20 lines

-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
              first = get_a_page(rq, gfp);
              if (!first) {
                      if (list)

drivers/net/virtio_net.c +1 -1
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
 
       /* chain first in list head */
       first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                 first, gfp);
       if (err < 0)

              give_pages(rq, first);

drivers/net/virtio_net.c +0 -0
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)

      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||

Quoted 8 lines
Quoted 4 lines

-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) { vi->big_packets = true;
+             vi->gso_is_supported = true;

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

Quoted 3 lines

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Quoted 9 lines
Quoted 9 lines

Thanks,
-Siwei

Quoted 4 lines

+     }

      if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) vi->mergeable_rx_bufs = true;

#13 lore raw
Quoted 115 lines · Gavin Li

On 8/9/2022 7:56 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/8/2022 12:31 AM, Gavin Li wrote:

On 8/6/2022 6:11 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/1/2022 9:45 PM, Gavin Li wrote:

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
Before After

   1500        22.5     22.4
   9000        12.8     25.9
---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----
  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
      /* I like... big packets and I cannot lie! */
      bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
      /* Host will merge rx buffers for big packets (shake it! shake

it!) */

      bool mergeable_rx_bufs;
drivers/net/virtio_net.c +5 -1
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct
 virtnet_info *vi, struct receive_queue *rq,
  static int add_recvbuf_big(struct virtnet_info *vi, struct
 receive_queue *rq,
                         gfp_t gfp)
  {
+     unsigned int sg_num = MAX_SKB_FRAGS;
      struct page *first, *list = NULL;
      char *p;
      int i, err, offset;
 
-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu
 / PAGE_SIZE;
 

DIV_ROUND_UP() can be used?

ACK

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.

Quoted 1 line

a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.

Right, it could be done on top.

But another note is that, it would still be better to support GUEST GSO feature:

1) can work for the case for path MTU
2) (migration)compatibility with software backends

Quoted 37 lines
drivers/net/virtio_net.c +3 -0
+     }
+
+     sg_init_table(rq->sg, sg_num + 2);
 
      /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

drivers/net/virtio_net.c +2 -2
-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
              first = get_a_page(rq, gfp);
              if (!first) {
                      if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
 
      /* chain first in list head */
      first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                first, gfp);
      if (err < 0)
              give_pages(rq, first);
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)
      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
          virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
          virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
              vi->big_packets = true;
+             vi->gso_is_supported = true;
 
Quoted 4 lines

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

It is fine as long as

1) we don't implement ethtool API for changing guest offloads
2) big mode XDP is not enabled

So that code works only for XDP but we forbid big packets in the case of XDP right now.

Thanks

Quoted 16 lines

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Thanks,

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3590

Quoted 6 lines
drivers/net/virtio_net.c +1 -1
-Siwei
 
+     }
 
      if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
              vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#14 lore raw
Quoted 17 lines · Jason Wang
Quoted 10 lines
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)
      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
          virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
          virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
              vi->big_packets = true;
+             vi->gso_is_supported = true;
 
Quoted 4 lines

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

It is fine as long as

Quoted 7 lines

1) we don't implement ethtool API for changing guest offloads
2) big mode XDP is not enabled

So that code works only for XDP but we forbid big packets in the case of XDP right now.

Thanks

To put it another way, changing big_packets after probe requires a bunch of work as current code assumes this flag never changes. Adding a TODO to handle dynamic offload config is fine but
I don't think it should block this.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#16 lore raw
In reply to Jason Wang
Quoted 1 line · Jason Wang
Quoted 1 line

+ unsigned int mtu = vi->dev->mtu;

BTW should this not be max_mtu? Otherwise if user configures mtu that is too small we'll add buffers that are too small. some backends simply lock up if this happens (I think vhost does).
Maybe we should add a feature to allow packet drop if it's too small. And send mtu guest to host while we are at it?

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#15 lore raw
In reply to Michael S. Tsirkin
Quoted 25 lines · Michael S. Tsirkin
Quoted 23 lines · Jason Wang
Quoted 10 lines
(quoted diff) +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)
      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
          virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
          virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
              vi->big_packets = true;
+             vi->gso_is_supported = true;
 
Quoted 4 lines

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

It is fine as long as

1) we don't implement ethtool API for changing guest offloads
2) big mode XDP is not enabled

So that code works only for XDP but we forbid big packets in the case of XDP right now.

Quoted 6 lines
Quoted 1 line

Thanks

To put it another way, changing big_packets after probe requires a bunch of work as current code assumes this flag never changes. Adding a TODO to handle dynamic offload config is fine but
I don't think it should block this.

Yes, this is what I mean.

Thanks

Quoted 4 lines

--
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#17 lore raw
In reply to Michael S. Tsirkin
Quoted 5 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang
Quoted 1 line

+ unsigned int mtu = vi->dev->mtu;

BTW should this not be max_mtu?

Yes.

Quoted 3 lines

Otherwise if user configures mtu that
is too small we'll add buffers that are too small. some backends simply lock up if this happens (I think vhost does).

Probably not? If we run out of buffers, we will wait for the next kick. (Otherwise it would be an guest triggerable behaviour)

Quoted 1 line

Maybe we should add a feature to allow packet drop if it's too small.

I may miss something but isn't this what most devices will do? (e.g ethernet have a minimal packet length)

Thanks

Quoted 5 lines

And send mtu guest to host while we are at it?

--
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#53 lore raw
In reply to Gavin Li
Quoted 91 lines · Gavin Li
Quoted 88 lines · Si-Wei Liu

External email: Use caution opening links or attachments

Quoted 84 lines · Gavin Li
Quoted 9 lines · Si-Wei Liu

External email: Use caution opening links or attachments

Quoted 3 lines · Gavin Li

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

Quoted 9 lines
Quoted 8 lines

  it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

Quoted 46 lines
Quoted 46 lines

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
              Before   After
   1500        22.5     22.4
   9000        12.8     25.9

---
  drivers/net/virtio_net.c | 20 ++++++++++++++++----

  1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
       /* I like... big packets and I cannot lie! */
       bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
       /* Host will merge rx buffers for big packets (shake it! shake

it!) */
      bool mergeable_rx_bufs;

drivers/net/virtio_net.c +0 -0
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct
 virtnet_info *vi, struct receive_queue *rq,

  static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                         gfp_t gfp)
  {
+     unsigned int sg_num = MAX_SKB_FRAGS;
      struct page *first, *list = NULL;
      char *p;
      int i, err, offset;

Quoted 7 lines
Quoted 6 lines

-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;

DIV_ROUND_UP() can be used?

ACK

Quoted 4 lines

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Quoted 8 lines
Quoted 7 lines

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu()
should have validated the MTU value before coming down to it. And I
suspect you might want to do virtnet_close() and virtnet_open()
before/after changing the buffer size on the fly (the netif_running()
case), implementing .ndo_change_mtu() will be needed anyway.

a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.

Sure, it's fine to separate it out into another patch and optimize on top later on. Though the previous comment of avoiding repeatedly computing sg_num in datapath still holds: set sg_num only once in probe() time then, in which case the sg_num needed can be simply inferred from dev->max_mtu rather than dev->mtu.

-Siwei

Quoted 54 lines
Quoted 54 lines
Quoted 36 lines
Quoted 7 lines
Quoted 6 lines

+     }
+
+     sg_init_table(rq->sg, sg_num + 2);

      /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

Quoted 20 lines
Quoted 20 lines

-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
              first = get_a_page(rq, gfp);
              if (!first) {
                      if (list)

drivers/net/virtio_net.c +1 -1
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
 
       /* chain first in list head */
       first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                 first, gfp);
       if (err < 0)

              give_pages(rq, first);

drivers/net/virtio_net.c +0 -0
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)

      if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||

Quoted 8 lines
Quoted 4 lines

-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) { vi->big_packets = true;
+             vi->gso_is_supported = true;

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

Quoted 3 lines

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Quoted 9 lines
Quoted 9 lines

Thanks,
-Siwei

Quoted 4 lines

+     }

      if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) vi->mergeable_rx_bufs = true;

#18 lore raw
In reply to Jason Wang
Quoted 159 lines · Jason Wang
Quoted 114 lines · Gavin Li

On 8/9/2022 7:56 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/8/2022 12:31 AM, Gavin Li wrote:

On 8/6/2022 6:11 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/1/2022 9:45 PM, Gavin Li wrote:

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)

               Before   After
    1500        22.5     22.4
    9000        12.8     25.9
---
   drivers/net/virtio_net.c | 20 ++++++++++++++++----
   1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
       /* I like... big packets and I cannot lie! */
       bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
       /* Host will merge rx buffers for big packets (shake it! shake

it!) */

       bool mergeable_rx_bufs;
drivers/net/virtio_net.c +5 -1
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct
 virtnet_info *vi, struct receive_queue *rq,
   static int add_recvbuf_big(struct virtnet_info *vi, struct
 receive_queue *rq,
                          gfp_t gfp)
   {
+     unsigned int sg_num = MAX_SKB_FRAGS;
       struct page *first, *list = NULL;
       char *p;
       int i, err, offset;
 
-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu
 / PAGE_SIZE;
 

DIV_ROUND_UP() can be used?

ACK

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.

Quoted 1 line

a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.

Right, it could be done on top.

But another note is that, it would still be better to support GUEST GSO feature:

1) can work for the case for path MTU
2) (migration)compatibility with software backends

Quoted 36 lines
drivers/net/virtio_net.c +3 -0
+     }
+
+     sg_init_table(rq->sg, sg_num + 2);
 
       /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

drivers/net/virtio_net.c +2 -2
-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
               first = get_a_page(rq, gfp);
               if (!first) {
                       if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
 
       /* chain first in list head */
       first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                 first, gfp);
       if (err < 0)
               give_pages(rq, first);
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)
       if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
               vi->big_packets = true;
+             vi->gso_is_supported = true;
 
Quoted 7 lines
Quoted 4 lines

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

It is fine as long as

1) we don't implement ethtool API for changing guest offloads

Not sure if I missed something, but it looks the current virtnet_set_features() already supports toggling on/off GRO HW through commit a02e8964eaf9271a8a5fcc0c55bd13f933bafc56 (formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads() or the caller doesn't unset big_packet ...".

Quoted 1 line

2) big mode XDP is not enabled

Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Regards,
-Siwei

Quoted 20 lines

So that code works only for XDP but we forbid big packets in the case of XDP right now.

Thanks

Quoted 14 lines

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Thanks,
-Siwei

Quoted 7 lines
Quoted 7 lines

+ }

if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#19 lore raw
Quoted 6 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM

Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500 to send packets but want to receive 9K packets?

#20 lore raw
Quoted 6 lines · Parav Pandit
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM
Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500 to send packets but want to receive 9K packets?

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000. What I actually tried to say is that as our current use case (software virtio) supports XDP applications with 1500 guest mtu and mergeable buffer enabled on a 9000 MTU link, technically it's a legitimate use case regardless of mergeable buffer capability. Otherwise it's considered to be a usability regression in driver software side when migrating (not live migrating) existing user to vdpa, due to lack of hardware implementation of certain relevant feature.

Thanks,
-Siwei

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#21 lore raw
Quoted 10 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 6 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.

Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".

So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).

#22 lore raw
Quoted 7 lines · Parav Pandit
Quoted 7 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 5 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

Which application benefit by having asymmetry by lowering mtu to 1500

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.

For big_packet path, yes, we need improvement; for mergeable, it's adaptable to any incoming packet size so 1500 is what it is today.

Quoted 4 lines

Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".

Right, that's what it happens today on device side (i.e. vhost-net, btw mlx5 vdpa device seems to have a bug not pro-actively dropping packets that exceed the MTU size, causing guest panic in small packet path).

Quoted 3 lines

So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).

Usually, the use case I'm aware of would set the peer's MTU to 1500 (e.g. on a virtual network appliance), or it would rely on path mtu discovery to avoid packet drop across links.

-Siwei

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#23 lore raw
Quoted 32 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 4:33 PM

Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 7 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

Quoted 7 lines
Quoted 1 line

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.
For big_packet path, yes, we need improvement; for mergeable, it's adaptable to any incoming packet size so 1500 is what it is today.

Quoted 4 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500 bytes. This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".
Right, that's what it happens today on device side (i.e. vhost-net, btw mlx5 vdpa device seems to have a bug not pro-actively dropping packets that exceed the MTU size, causing guest panic in small packet path).

Quoted 1 line
Quoted 7 lines
Quoted 1 line

So, I am lost what virtio net device user application is trying to achieve by

sending smaller packets and dropping all receive packets.

Quoted 1 line

(it doesn’t have any relation to mergeable or otherwise).

Usually, the use case I'm aware of would set the peer's MTU to 1500 (e.g. on a virtual network appliance), or it would rely on path mtu discovery to avoid packet drop across links.

Ok. Somehow the application knows the mtu to set on (all) peer(s) and hope for the best. Understood.

#24 lore raw
Quoted 33 lines · Parav Pandit
Quoted 33 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 4:33 PM

Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 7 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

Quoted 7 lines
Quoted 1 line

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.
For big_packet path, yes, we need improvement; for mergeable, it's adaptable to any incoming packet size so 1500 is what it is today.

Quoted 4 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500 bytes. This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".
Right, that's what it happens today on device side (i.e. vhost-net, btw mlx5 vdpa device seems to have a bug not pro-actively dropping packets that exceed the MTU size, causing guest panic in small packet path).

Quoted 2 lines

So, I am lost what virtio net device user application is trying to achieve by

Quoted 8 lines
Quoted 6 lines

sending smaller packets and dropping all receive packets.

Quoted 1 line

(it doesn’t have any relation to mergeable or otherwise).

Usually, the use case I'm aware of would set the peer's MTU to 1500 (e.g. on a virtual network appliance), or it would rely on path mtu discovery to avoid packet drop across links.

Ok. Somehow the application knows the mtu to set on (all) peer(s) and hope for the best. Understood.

That's generally what one has to do with mtu, yes - it has to be set consistently across the LAN. While e.g. pMTU might help work around some misconfigured LANs with a mix of different MTUs it was never designed for that.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#49 lore raw
In reply to Si-Wei Liu
Quoted 168 lines · Si-Wei Liu
Quoted 165 lines · Jason Wang
Quoted 114 lines · Gavin Li

On 8/9/2022 7:56 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/8/2022 12:31 AM, Gavin Li wrote:

On 8/6/2022 6:11 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/1/2022 9:45 PM, Gavin Li wrote:

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)

               Before   After
    1500        22.5     22.4
    9000        12.8     25.9
---
   drivers/net/virtio_net.c | 20 ++++++++++++++++----
   1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
       /* I like... big packets and I cannot lie! */
       bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
       /* Host will merge rx buffers for big packets (shake it! shake

it!) */

       bool mergeable_rx_bufs;
drivers/net/virtio_net.c +5 -1
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct
 virtnet_info *vi, struct receive_queue *rq,
   static int add_recvbuf_big(struct virtnet_info *vi, struct
 receive_queue *rq,
                          gfp_t gfp)
   {
+     unsigned int sg_num = MAX_SKB_FRAGS;
       struct page *first, *list = NULL;
       char *p;
       int i, err, offset;
 
-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu
 / PAGE_SIZE;
 

DIV_ROUND_UP() can be used?

ACK

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.

Quoted 1 line

a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.

Right, it could be done on top.

But another note is that, it would still be better to support GUEST GSO feature:

1) can work for the case for path MTU
2) (migration)compatibility with software backends

Quoted 36 lines
drivers/net/virtio_net.c +3 -0
+     }
+
+     sg_init_table(rq->sg, sg_num + 2);
 
       /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

drivers/net/virtio_net.c +2 -2
-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
               first = get_a_page(rq, gfp);
               if (!first) {
                       if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
 
       /* chain first in list head */
       first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                 first, gfp);
       if (err < 0)
               give_pages(rq, first);
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)
       if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
               vi->big_packets = true;
+             vi->gso_is_supported = true;
 
Quoted 4 lines

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

It is fine as long as

Quoted 6 lines
Quoted 1 line

1) we don't implement ethtool API for changing guest offloads

Not sure if I missed something, but it looks the current virtnet_set_features() already supports toggling on/off GRO HW through commit a02e8964eaf9271a8a5fcc0c55bd13f933bafc56 (formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads() or the caller doesn't unset big_packet ...".

"we" here is the device, not the driver.

Quoted 27 lines
Quoted 1 line

2) big mode XDP is not enabled

Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Regards,
-Siwei

Quoted 20 lines

So that code works only for XDP but we forbid big packets in the case of XDP right now.

Thanks

Quoted 14 lines

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Thanks,
-Siwei

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ }

if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#25 lore raw
In reply to Parav Pandit
Quoted 13 lines · Parav Pandit
Quoted 10 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 6 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 8 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.

Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".

The "mtu" here is the device config field, which is

        /* Default maximum transmit unit advice */

there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

Quoted 2 lines

So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#50 lore raw
In reply to Michael S. Tsirkin
Quoted 166 lines · Michael S. Tsirkin
Quoted 165 lines · Si-Wei Liu
Quoted 163 lines · Jason Wang
Quoted 113 lines · Gavin Li

On 8/9/2022 7:56 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/8/2022 12:31 AM, Gavin Li wrote:

On 8/6/2022 6:11 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/1/2022 9:45 PM, Gavin Li wrote:

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)

                Before   After
     1500        22.5     22.4
     9000        12.8     25.9
---
    drivers/net/virtio_net.c | 20 ++++++++++++++++----
    1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
        /* I like... big packets and I cannot lie! */
        bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
        /* Host will merge rx buffers for big packets (shake it! shake

it!) */

        bool mergeable_rx_bufs;
drivers/net/virtio_net.c +5 -1
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct
 virtnet_info *vi, struct receive_queue *rq,
    static int add_recvbuf_big(struct virtnet_info *vi, struct
 receive_queue *rq,
                           gfp_t gfp)
    {
+     unsigned int sg_num = MAX_SKB_FRAGS;
        struct page *first, *list = NULL;
        char *p;
        int i, err, offset;
 
-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu
 / PAGE_SIZE;
 

DIV_ROUND_UP() can be used?

ACK

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.

Quoted 1 line

a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.

Right, it could be done on top.

But another note is that, it would still be better to support GUEST GSO feature:

1) can work for the case for path MTU
2) (migration)compatibility with software backends

Quoted 35 lines
drivers/net/virtio_net.c +3 -0
+     }
+
+     sg_init_table(rq->sg, sg_num + 2);
 
        /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

drivers/net/virtio_net.c +2 -2
-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
                first = get_a_page(rq, gfp);
                if (!first) {
                        if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
 
        /* chain first in list head */
        first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                  first, gfp);
        if (err < 0)
                give_pages(rq, first);
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)
        if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
                vi->big_packets = true;
+             vi->gso_is_supported = true;
 
Quoted 4 lines

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

It is fine as long as

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

1) we don't implement ethtool API for changing guest offloads

Not sure if I missed something, but it looks the current virtnet_set_features() already supports toggling on/off GRO HW through commit a02e8964eaf9271a8a5fcc0c55bd13f933bafc56 (formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads() or the caller doesn't unset big_packet ...".

"we" here is the device, not the driver.

What is the ethtool API from device level? VIRTIO_NET_F_CTRL_GUEST_OFFLOADS for sure, right?

It's implemented in software backend as far as I know. I see no reason *technically* this is infeasible, regardless what you name it, being a bug or TODO.

-Siwei

Quoted 27 lines
Quoted 27 lines
Quoted 2 lines

2) big mode XDP is not enabled

Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Regards,
-Siwei

Quoted 19 lines

So that code works only for XDP but we forbid big packets in the case of XDP right now.

Thanks

Quoted 14 lines

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Thanks,
-Siwei

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ }

if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#26 lore raw
In reply to Michael S. Tsirkin
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 4 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500 bytes. This is because device follows the spec " The device MUST NOT pass

Quoted 7 lines

received packets that exceed mtu".

The "mtu" here is the device config field, which is

        /* Default maximum transmit unit advice */

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.
When user overrides it, driver can be further optimized to honor such new value on rx buffer posting.

Quoted 3 lines

And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

If user space insists on small buffers, so be it. It only works when user exactly know what user is doing in the whole network. When user prefers to override the device RO field, device is in the dark and things work on best effort basis. This must be a reasonably advance user who has good knowledge of its network topology etc.

For such case, may be yes, driver should be further optimized.

#27 lore raw
Quoted 21 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 4 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500 bytes. This is because device follows the spec " The device MUST NOT pass

Quoted 7 lines

received packets that exceed mtu".

The "mtu" here is the device config field, which is

        /* Default maximum transmit unit advice */
Quoted 6 lines

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

yes

Quoted 1 line

When user overrides it, driver can be further optimized to honor such new value on rx buffer posting.

no, not without a feature bit promising device won't get wedged.

Quoted 4 lines
Quoted 3 lines

And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

If user space insists on small buffers, so be it.

If previously things worked, the "so be it" is a regression and blaming users won't help us.

Quoted 1 line

It only works when user exactly know what user is doing in the whole network.

If you want to claim this you need a new feature bit.

Quoted 1 line

When user prefers to override the device RO field, device is in the dark and things work on best effort basis.

Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.

Quoted 4 lines

This must be a reasonably advance user who has good knowledge of its network topology etc.

For such case, may be yes, driver should be further optimized.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#51 lore raw
In reply to Si-Wei Liu
Quoted 170 lines · Si-Wei Liu
Quoted 167 lines · Michael S. Tsirkin
Quoted 166 lines · Si-Wei Liu
Quoted 164 lines · Jason Wang
Quoted 113 lines · Gavin Li

On 8/9/2022 7:56 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/8/2022 12:31 AM, Gavin Li wrote:

On 8/6/2022 6:11 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/1/2022 9:45 PM, Gavin Li wrote:

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)

                Before   After
     1500        22.5     22.4
     9000        12.8     25.9
---
    drivers/net/virtio_net.c | 20 ++++++++++++++++----
    1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
        /* I like... big packets and I cannot lie! */
        bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
        /* Host will merge rx buffers for big packets (shake it! shake

it!) */

        bool mergeable_rx_bufs;
drivers/net/virtio_net.c +5 -1
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct
 virtnet_info *vi, struct receive_queue *rq,
    static int add_recvbuf_big(struct virtnet_info *vi, struct
 receive_queue *rq,
                           gfp_t gfp)
    {
+     unsigned int sg_num = MAX_SKB_FRAGS;
        struct page *first, *list = NULL;
        char *p;
        int i, err, offset;
 
-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu
 / PAGE_SIZE;
 

DIV_ROUND_UP() can be used?

ACK

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.

Quoted 1 line

a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.

Right, it could be done on top.

But another note is that, it would still be better to support GUEST GSO feature:

1) can work for the case for path MTU
2) (migration)compatibility with software backends

Quoted 35 lines
drivers/net/virtio_net.c +3 -0
+     }
+
+     sg_init_table(rq->sg, sg_num + 2);
 
        /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

drivers/net/virtio_net.c +2 -2
-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
                first = get_a_page(rq, gfp);
                if (!first) {
                        if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
 
        /* chain first in list head */
        first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                  first, gfp);
        if (err < 0)
                give_pages(rq, first);
drivers/net/virtio_net.c +2 -1
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)
        if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) ||
            virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) ||
-         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO))
+         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {
                vi->big_packets = true;
+             vi->gso_is_supported = true;
 
Quoted 4 lines

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

It is fine as long as

1) we don't implement ethtool API for changing guest offloads

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Not sure if I missed something, but it looks the current virtnet_set_features() already supports toggling on/off GRO HW through commit a02e8964eaf9271a8a5fcc0c55bd13f933bafc56 (formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads() or the caller doesn't unset big_packet ...".

"we" here is the device, not the driver.

What is the ethtool API from device level? VIRTIO_NET_F_CTRL_GUEST_OFFLOADS for sure, right?

Yes.

Quoted 5 lines

It's implemented in software backend as far as I know. I see no reason *technically* this is infeasible, regardless what you name it, being a bug or TODO.

-Siwei

It's feasible but it's more work. Whether we bother depends in particular on whether anyone cares.

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 2 lines

2) big mode XDP is not enabled

Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Regards,
-Siwei

Quoted 19 lines

So that code works only for XDP but we forbid big packets in the case of XDP right now.

Thanks

Quoted 14 lines

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Thanks,
-Siwei

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ }

if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#44 lore raw
In reply to Michael S. Tsirkin
Quoted 21 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 6 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 8 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.

Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".

Quoted 8 lines

The "mtu" here is the device config field, which is

         /* Default maximum transmit unit advice */

there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

It'd be nice to document this requirement or statement to the spec for clarity purpose. Otherwise various device implementations are hard to follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.

-Siwei

Quoted 9 lines
Quoted 4 lines

So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#45 lore raw
Quoted 26 lines · Si-Wei Liu
Quoted 23 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 6 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 8 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.

Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".

The "mtu" here is the device config field, which is

Quoted 8 lines
Quoted 6 lines
         /* Default maximum transmit unit advice */

there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

It'd be nice to document this requirement or statement to the spec for clarity purpose.

It's not a requirement, more of a bug. But it's been like this for years.

Quoted 6 lines

Otherwise various device implementations are hard to
follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.

-Siwei

Weird where do you see this in code? I see

                sock_len = vhost_net_rx_peek_head_len(net, sock->sk,
                                                      &busyloop_intr);
                if (!sock_len)
                        break;
                sock_len += sock_hlen;
                vhost_len = sock_len + vhost_hlen;
                headcount = get_rx_bufs(vq, vq->heads + nvq->done_idx,
                                        vhost_len, &in, vq_log, &log,
                                        likely(mergeable) ? UIO_MAXIOV : 1);
                /* On error, stop handling until the next kick. */
                if (unlikely(headcount < 0))
                        goto out;

so it does not drop a packet, it just stops processing the queue.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#28 lore raw
In reply to Michael S. Tsirkin
Quoted 45 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packets

Quoted 7 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 3 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500

bytes.

Quoted 9 lines
Quoted 8 lines
Quoted 1 line

This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".

The "mtu" here is the device config field, which is

        /* Default maximum transmit unit advice */
Quoted 6 lines

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

Quoted 7 lines

yes

Quoted 1 line

When user overrides it, driver can be further optimized to honor such new

value on rx buffer posting.

no, not without a feature bit promising device won't get wedged.

I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

If user space insists on small buffers, so be it.

If previously things worked, the "so be it" is a regression and blaming users won't help us.

I am not suggesting above.
This was Si-Wei's suggestion that somehow driver wants to post smaller buffers than the mtu because user knows what peer is doing. So may be driver can be extended to give more weight on user config.

Quoted 5 lines
Quoted 1 line

It only works when user exactly know what user is doing in the whole

network.

If you want to claim this you need a new feature bit.

Why is a new bit needed to tell device?
User is doing something its own config mismatching the buffers and mtu. A solid use case hasn't emerged for this yet.

If user wants to modify the mtu, we should just make virtio_net_config.mtu as RW field using new feature bit. Is that what you mean?
If so, yes, it makes things very neat where driver and device are aligned to each other, the way they are today. Only limitation is that its one-way. = device tells to driver.

Quoted 6 lines
Quoted 1 line

When user prefers to override the device RO field, device is in the dark and

things work on best effort basis.

Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.

Not sure, why things get stuck for ever. Maybe you have example to explain. I am mostly missing something.

Quoted 5 lines
Quoted 1 line

This must be a reasonably advance user who has good knowledge of its

network topology etc.

Quoted 3 lines

For such case, may be yes, driver should be further optimized.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#46 lore raw
In reply to Michael S. Tsirkin
Quoted 27 lines · Michael S. Tsirkin
Quoted 26 lines · Si-Wei Liu
Quoted 24 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 6 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 8 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.

Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".

The "mtu" here is the device config field, which is

          /* Default maximum transmit unit advice */
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

It'd be nice to document this requirement or statement to the spec for clarity purpose.

It's not a requirement, more of a bug. But it's been like this for years.

Well, I'm not sure how it may wedge the device if not capable of posting to smaller buffers, is there other option than drop? Truncate to what the buffer size may fit and deliver up? Seems even worse than drop...

Quoted 23 lines
Quoted 7 lines

Otherwise various device implementations are hard to
follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.

-Siwei

Weird where do you see this in code? I see

                 sock_len = vhost_net_rx_peek_head_len(net, sock->sk,
                                                       &busyloop_intr);
                 if (!sock_len)
                         break;
                 sock_len += sock_hlen;
                 vhost_len = sock_len + vhost_hlen;
                 headcount = get_rx_bufs(vq, vq->heads + nvq->done_idx,
                                         vhost_len, &in, vq_log, &log,
                                         likely(mergeable) ? UIO_MAXIOV : 1);
                 /* On error, stop handling until the next kick. */
                 if (unlikely(headcount < 0))
                         goto out;
Quoted 1 line

so it does not drop a packet, it just stops processing the queue.

Here

                /* On overrun, truncate and discard */
                if (unlikely(headcount > UIO_MAXIOV)) { iov_iter_init(&msg.msg_iter, READ, vq->iov, 1, 1); err = sock->ops->recvmsg(sock, &msg,
                                                 1, MSG_DONTWAIT | MSG_TRUNC); pr_debug("Discarded rx packet: len %zd\n", sock_len); continue;
                }

recvmsg(, , 1, ) is essentially to drop the oversized packet.

In get_rx_bufs(), overrun detection will return something larger than UIO_MAXIOV as indicator:

static int get_rx_bufs()
{
:
;
        /* Detect overrun */
        if (unlikely(datalen > 0)) {
                r = UIO_MAXIOV + 1;
                goto err;
        }
:
:

-Siwei

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#29 lore raw
In reply to Parav Pandit
Quoted 47 lines · Parav Pandit
Quoted 47 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packets

Quoted 7 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 3 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500

bytes.

Quoted 9 lines
Quoted 8 lines
Quoted 1 line

This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".

The "mtu" here is the device config field, which is

        /* Default maximum transmit unit advice */
Quoted 6 lines

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

yes

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

When user overrides it, driver can be further optimized to honor such new

value on rx buffer posting.

no, not without a feature bit promising device won't get wedged.

I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).

They just are because drivers did not do this.

Quoted 26 lines
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

If user space insists on small buffers, so be it.

If previously things worked, the "so be it" is a regression and blaming users won't help us.

I am not suggesting above.
This was Si-Wei's suggestion that somehow driver wants to post smaller buffers than the mtu because user knows what peer is doing. So may be driver can be extended to give more weight on user config.

Quoted 5 lines
Quoted 1 line

It only works when user exactly know what user is doing in the whole

network.

If you want to claim this you need a new feature bit.

Why is a new bit needed to tell device?
User is doing something its own config mismatching the buffers and mtu. A solid use case hasn't emerged for this yet.

If user wants to modify the mtu, we should just make virtio_net_config.mtu as RW field using new feature bit. Is that what you mean?
If so, yes, it makes things very neat where driver and device are aligned to each other, the way they are today. Only limitation is that its one-way. = device tells to driver.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

When user prefers to override the device RO field, device is in the dark and

things work on best effort basis.

Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.

Not sure, why things get stuck for ever. Maybe you have example to explain. I am mostly missing something.

I sent an explanation a bit earlier. It's more or less a bug.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

This must be a reasonably advance user who has good knowledge of its

network topology etc.

Quoted 3 lines

For such case, may be yes, driver should be further optimized.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#47 lore raw
In reply to Si-Wei Liu
Quoted 98 lines · Si-Wei Liu
Quoted 27 lines · Michael S. Tsirkin
Quoted 26 lines · Si-Wei Liu
Quoted 24 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 6 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 8 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.

Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".

The "mtu" here is the device config field, which is

          /* Default maximum transmit unit advice */
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

It'd be nice to document this requirement or statement to the spec for clarity purpose.

It's not a requirement, more of a bug. But it's been like this for years.

Well, I'm not sure how it may wedge the device if not capable of posting to smaller buffers, is there other option than drop? Truncate to what the buffer size may fit and deliver up? Seems even worse than drop...

Quoted 23 lines
Quoted 7 lines

Otherwise various device implementations are hard to
follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.

-Siwei

Weird where do you see this in code? I see

                 sock_len = vhost_net_rx_peek_head_len(net, sock->sk,
                                                       &busyloop_intr);
                 if (!sock_len)
                         break;
                 sock_len += sock_hlen;
                 vhost_len = sock_len + vhost_hlen;
                 headcount = get_rx_bufs(vq, vq->heads + nvq->done_idx,
                                         vhost_len, &in, vq_log, &log,
                                         likely(mergeable) ? UIO_MAXIOV : 1);
                 /* On error, stop handling until the next kick. */
                 if (unlikely(headcount < 0))
                         goto out;
Quoted 1 line

so it does not drop a packet, it just stops processing the queue.

Here

                /* On overrun, truncate and discard */
                if (unlikely(headcount > UIO_MAXIOV)) { iov_iter_init(&msg.msg_iter, READ, vq->iov, 1, 1); err = sock->ops->recvmsg(sock, &msg, 1, MSG_DONTWAIT | MSG_TRUNC);
                        pr_debug("Discarded rx packet: len %zd\n", sock_len);
                        continue;
                }

recvmsg(, , 1, ) is essentially to drop the oversized packet.

In get_rx_bufs(), overrun detection will return something larger than UIO_MAXIOV as indicator:

static int get_rx_bufs()
{
:
;
        /* Detect overrun */
        if (unlikely(datalen > 0)) {
                r = UIO_MAXIOV + 1;
                goto err;
        }
:
:

Quoted 1 line

-Siwei

Hmm you are right. I'll check but it seems I have misread the code. Sorry about wasting your time on this.
So maybe the approach is ok then.
It's late, I'll recheck tomorrow.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#30 lore raw
In reply to Parav Pandit
Quoted 8 lines · Parav Pandit
Quoted 6 lines
Quoted 1 line

When user prefers to override the device RO field, device is in the dark and

things work on best effort basis.

Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.

Not sure, why things get stuck for ever. Maybe you have example to explain. I am mostly missing something.

I'm no longer sure I'm right. Will recheck tomorrow, it's late here.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#31 lore raw
In reply to Parav Pandit
Quoted 43 lines · Parav Pandit
Quoted 43 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packets

Quoted 11 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 2 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500

bytes.

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".

The "mtu" here is the device config field, which is

Quoted 8 lines
Quoted 2 lines
         /* Default maximum transmit unit advice */

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

yes

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

When user overrides it, driver can be further optimized to honor such new

value on rx buffer posting.

no, not without a feature bit promising device won't get wedged.

I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).

I am also confused how the device can be wedged in this case.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

If user space insists on small buffers, so be it.

If previously things worked, the "so be it" is a regression and blaming users won't help us.

I am not suggesting above.
This was Si-Wei's suggestion that somehow driver wants to post smaller buffers than the mtu because user knows what peer is doing. So may be driver can be extended to give more weight on user config.

It's not me, it's from our customers with real use cases. Some of which have very dedicate network setup, and it's not at odd they know virtio internals quite well. At one point they even customized the driver to disable mergeable buffer ahead of us offering them the opt-out from device level. And their appliance indeed has assumption of 1460 mtu everywhere.

-Siwei

Quoted 15 lines
Quoted 6 lines
Quoted 2 lines

It only works when user exactly know what user is doing in the whole

network.

If you want to claim this you need a new feature bit.

Why is a new bit needed to tell device?
User is doing something its own config mismatching the buffers and mtu. A solid use case hasn't emerged for this yet.

If user wants to modify the mtu, we should just make virtio_net_config.mtu as RW field using new feature bit. Is that what you mean?
If so, yes, it makes things very neat where driver and device are aligned to each other, the way they are today. Only limitation is that its one-way. = device tells to driver.

Quoted 13 lines
Quoted 6 lines
Quoted 1 line

When user prefers to override the device RO field, device is in the dark and

things work on best effort basis.

Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.

Not sure, why things get stuck for ever. Maybe you have example to explain. I am mostly missing something.

Quoted 4 lines
Quoted 1 line

This must be a reasonably advance user who has good knowledge of its

network topology etc.

Quoted 2 lines

For such case, may be yes, driver should be further optimized.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#52 lore raw
In reply to Si-Wei Liu
Quoted 162 lines · Si-Wei Liu
Quoted 159 lines · Jason Wang
Quoted 114 lines · Gavin Li

On 8/9/2022 7:56 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/8/2022 12:31 AM, Gavin Li wrote:

On 8/6/2022 6:11 AM, Si-Wei Liu wrote:

External email: Use caution opening links or attachments

On 8/1/2022 9:45 PM, Gavin Li wrote:

Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,

GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.

ACK

   it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.

Therefore, if GSO is not supported,

Ditto.

ACK

use the MTU to calculate the
optimal amount of segments required.

Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.

MTU(Bytes)/Bandwidth (Gbit/s)
               Before   After
    1500        22.5     22.4
    9000        12.8     25.9

---
   drivers/net/virtio_net.c | 20 ++++++++++++++++----

   1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +3 -0
index ec8e1b3108c3..d36918c1809d 100644
@@ -222,6 +222,9 @@ struct virtnet_info {
        /* I like... big packets and I cannot lie! */
        bool big_packets;
 
+     /* Indicates GSO support */
+     bool gso_is_supported;
+
        /* Host will merge rx buffers for big packets (shake it! shake

it!) */
       bool mergeable_rx_bufs;

drivers/net/virtio_net.c +0 -0
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct
 virtnet_info *vi, struct receive_queue *rq,

   static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
                          gfp_t gfp)
   {
+     unsigned int sg_num = MAX_SKB_FRAGS;
       struct page *first, *list = NULL;
       char *p;
       int i, err, offset;

-     sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+     if (!vi->gso_is_supported) {
+             unsigned int mtu = vi->dev->mtu;
+
+             sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu / PAGE_SIZE;

DIV_ROUND_UP() can be used?

ACK

Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?

Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.

Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.

Quoted 1 line

a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.

Right, it could be done on top.

But another note is that, it would still be better to support GUEST GSO feature:

1) can work for the case for path MTU
2) (migration)compatibility with software backends

Quoted 36 lines

+     }
+
+     sg_init_table(rq->sg, sg_num + 2);

       /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */

Comment doesn't match code.

ACK

-     for (i = MAX_SKB_FRAGS + 1; i > 1; --i) {
+     for (i = sg_num + 1; i > 1; --i) {
               first = get_a_page(rq, gfp);
               if (!first) {
                       if (list)

drivers/net/virtio_net.c +1 -1
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info
 *vi, struct receive_queue *rq,
 
        /* chain first in list head */
        first->private = (unsigned long)list;
-     err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2,
+     err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2,
                                  first, gfp);
        if (err < 0)

               give_pages(rq, first);

drivers/net/virtio_net.c +0 -0
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device
 *vdev)

       if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) || -         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) { vi->big_packets = true;
+             vi->gso_is_supported = true;

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.

It is fine as long as

1) we don't implement ethtool API for changing guest offloads

Not sure if I missed something, but it looks the current virtnet_set_features() already supports toggling on/off GRO HW through commit a02e8964eaf9271a8a5fcc0c55bd13f933bafc56 (formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads() or the caller doesn't unset big_packet ...".

Yes, I miss that.

Quoted 3 lines
Quoted 2 lines

2) big mode XDP is not enabled

Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

AFAIK, this requires more changes since mergeable path allocates PAGE_SIZE while small path allocates ~1500. This is something that needs to be fixed.

Thanks

Quoted 23 lines

Regards,
-Siwei

Quoted 20 lines

So that code works only for XDP but we forbid big packets in the case of XDP right now.

Thanks

Quoted 14 lines

ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?

I think that it should be fine, though you may want to deal with the XDP path not to regress it.

-Siwei

Thanks,
-Siwei

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

+     }

       if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) vi->mergeable_rx_bufs = true;

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#48 lore raw
In reply to Si-Wei Liu
Quoted 75 lines · Si-Wei Liu
Quoted 27 lines · Michael S. Tsirkin
Quoted 26 lines · Si-Wei Liu
Quoted 24 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PM

Quoted 6 lines
Quoted 5 lines

From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.

Which application benefit by having asymmetry by lowering mtu to 1500

to send packets but want to receive 9K packets?

Below details doesn’t answer the question of asymmetry. :)

Quoted 8 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.

Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".

The "mtu" here is the device config field, which is

          /* Default maximum transmit unit advice */

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.

It'd be nice to document this requirement or statement to the spec for clarity purpose.

It's not a requirement, more of a bug. But it's been like this for years.

Well, I'm not sure how it may wedge the device if not capable of posting to smaller buffers, is there other option than drop? Truncate to what the buffer size may fit and deliver up? Seems even worse than drop...

Quoted 23 lines
Quoted 7 lines

Otherwise various device implementations are hard to
follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.

-Siwei

Weird where do you see this in code? I see

                 sock_len = vhost_net_rx_peek_head_len(net, sock->sk, &busyloop_intr);
                 if (!sock_len)
                         break;
                 sock_len += sock_hlen;
                 vhost_len = sock_len + vhost_hlen;
                 headcount = get_rx_bufs(vq, vq->heads + nvq->done_idx, vhost_len, &in, vq_log, &log, likely(mergeable) ? UIO_MAXIOV : 1); /* On error, stop handling until the next kick. */ if (unlikely(headcount < 0))
                         goto out;

Quoted 1 line

so it does not drop a packet, it just stops processing the queue.

Here

                /* On overrun, truncate and discard */
                if (unlikely(headcount > UIO_MAXIOV)) { iov_iter_init(&msg.msg_iter, READ, vq->iov, 1, 1); err = sock->ops->recvmsg(sock, &msg,
                                                 1, MSG_DONTWAIT | MSG_TRUNC); pr_debug("Discarded rx packet: len %zd\n", sock_len); continue;
                }

Quoted 1 line

recvmsg(, , 1, ) is essentially to drop the oversized packet.

It's not necessarily the oversized packet but the packet that has too many sgs.

This issues has been discussed in the past, (for example we disable large rx queue size for vhost-net in Qemu). Where it could be solved by doing piece-wise copy

Thanks

Quoted 17 lines

In get_rx_bufs(), overrun detection will return something larger than UIO_MAXIOV as indicator:

static int get_rx_bufs()
{
:
;
        /* Detect overrun */
        if (unlikely(datalen > 0)) {
                r = UIO_MAXIOV + 1;
                goto err;
        }
:
:

Quoted 12 lines

-Siwei

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#32 lore raw
In reply to Si-Wei Liu
Quoted 47 lines · Si-Wei Liu
Quoted 44 lines · Parav Pandit
Quoted 44 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packets

Quoted 11 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 2 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500

bytes.

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".

The "mtu" here is the device config field, which is

Quoted 8 lines
Quoted 2 lines
         /* Default maximum transmit unit advice */

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

yes

Quoted 1 line

When user overrides it, driver can be further optimized to honor such new

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

value on rx buffer posting.

no, not without a feature bit promising device won't get wedged.

I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).

I am also confused how the device can be wedged in this case.

Yea sorry. I misunderstood the code. It can't be.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#33 lore raw
Quoted 50 lines · Michael S. Tsirkin
Quoted 49 lines · Si-Wei Liu
Quoted 46 lines · Parav Pandit
Quoted 46 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packets

Quoted 11 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 2 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500

bytes.

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".

The "mtu" here is the device config field, which is

Quoted 8 lines
Quoted 2 lines
         /* Default maximum transmit unit advice */

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

yes

Quoted 1 line

When user overrides it, driver can be further optimized to honor such new

value on rx buffer posting.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

no, not without a feature bit promising device won't get wedged.

I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).

I am also confused how the device can be wedged in this case.

Yea sorry. I misunderstood the code. It can't be.

Here's a problem as I see it. Let's say we reduce mtu. Small buffers are added. Now we increase mtu.
Device will drop all packets until small buffers are consumed.

Should we make this depend on the vq reset ability maybe?

Quoted 2 lines
Signature · 1 line
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#34 lore raw
Quoted 59 lines · Michael S. Tsirkin
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Si-Wei Liu
Quoted 43 lines · Parav Pandit
Quoted 43 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packets

Quoted 11 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 2 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500

bytes.

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".

The "mtu" here is the device config field, which is

Quoted 8 lines
Quoted 2 lines
         /* Default maximum transmit unit advice */

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

yes

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

When user overrides it, driver can be further optimized to honor such new

value on rx buffer posting.

no, not without a feature bit promising device won't get wedged.

I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).

I am also confused how the device can be wedged in this case.

Quoted 7 lines
Quoted 1 line

Yea sorry. I misunderstood the code. It can't be.

Here's a problem as I see it. Let's say we reduce mtu. Small buffers are added. Now we increase mtu.
Device will drop all packets until small buffers are consumed.

Should we make this depend on the vq reset ability maybe?

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Thanks

Quoted 4 lines
Quoted 3 lines

--
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#35 lore raw
Quoted 66 lines · Jason Wang
Quoted 65 lines · Michael S. Tsirkin
Quoted 50 lines · Michael S. Tsirkin
Quoted 49 lines · Si-Wei Liu
Quoted 46 lines · Parav Pandit
Quoted 46 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packets

Quoted 11 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 2 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500

bytes.

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".

The "mtu" here is the device config field, which is

Quoted 8 lines
Quoted 2 lines
         /* Default maximum transmit unit advice */

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

yes

Quoted 1 line

When user overrides it, driver can be further optimized to honor such new

value on rx buffer posting.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

no, not without a feature bit promising device won't get wedged.

I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).

I am also confused how the device can be wedged in this case.

Yea sorry. I misunderstood the code. It can't be.

Here's a problem as I see it. Let's say we reduce mtu. Small buffers are added. Now we increase mtu.
Device will drop all packets until small buffers are consumed.

Quoted 6 lines
Quoted 1 line

Should we make this depend on the vq reset ability maybe?

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Thanks

Device reset is really annoying in that it loses all the state:
rx filters etc etc.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

--
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#36 lore raw
Quoted 6 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AM

Quoted 3 lines
Quoted 2 lines

Should we make this depend on the vq reset ability maybe?

Quoted 7 lines
Quoted 4 lines

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Thanks

Device reset is really annoying in that it loses all the state:
rx filters etc etc.

The elegant solution is let driver tell the new mtu to the device.
One way to do so is by using existing ctrl vq.
If merged buffer is done, and new mtu is > minimum posting size, no need to undergo vq reset. If merged buffer is not done, and buffer posted are smaller than new mtu, undergo vq reset optionally.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#37 lore raw
Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AM

Quoted 6 lines
Quoted 2 lines

Should we make this depend on the vq reset ability maybe?

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Thanks

Device reset is really annoying in that it loses all the state:
rx filters etc etc.

The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.

That will need a new feature bit.

Quoted 2 lines

If merged buffer is done, and new mtu is > minimum posting size, no need to undergo vq reset. If merged buffer is not done, and buffer posted are smaller than new mtu, undergo vq reset optionally.

This can be done with or without sending mtu to device.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#38 lore raw
Quoted 16 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PM

Quoted 11 lines · Parav Pandit
Quoted 11 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AM

Quoted 7 lines
Quoted 2 lines

Should we make this depend on the vq reset ability maybe?

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Device reset is really annoying in that it loses all the state:
rx filters etc etc.

The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.

That will need a new feature bit.

Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?

Quoted 6 lines
Quoted 1 line

If merged buffer is done, and new mtu is > minimum posting size, no need

to undergo vq reset.

Quoted 1 line

If merged buffer is not done, and buffer posted are smaller than new mtu,

undergo vq reset optionally.

This can be done with or without sending mtu to device.

Yes, telling mtu to device helps device to optimize and adhere to the spec line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#39 lore raw
Quoted 19 lines · Parav Pandit
Quoted 19 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PM

Quoted 14 lines · Parav Pandit
Quoted 13 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AM

Quoted 7 lines
Quoted 2 lines

Should we make this depend on the vq reset ability maybe?

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Thanks

Device reset is really annoying in that it loses all the state:
rx filters etc etc.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.

That will need a new feature bit.

Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?

We did feature bits for this in the past.

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

If merged buffer is done, and new mtu is > minimum posting size, no need

to undergo vq reset.

Quoted 1 line

If merged buffer is not done, and buffer posted are smaller than new mtu,

undergo vq reset optionally.

This can be done with or without sending mtu to device.

Yes, telling mtu to device helps device to optimize and adhere to the spec line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.

Again, that line refers to \field{mtu} which is the max mtu supported, irrespective to anything driver does.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
#40 lore raw
Quoted 32 lines · Michael S. Tsirkin
Quoted 19 lines · Parav Pandit
Quoted 19 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PM

Quoted 14 lines · Parav Pandit
Quoted 13 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AM

Quoted 7 lines
Quoted 2 lines

Should we make this depend on the vq reset ability maybe?

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Thanks

Device reset is really annoying in that it loses all the state:
rx filters etc etc.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.

That will need a new feature bit.

Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?

We did feature bits for this in the past.

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

If merged buffer is done, and new mtu is > minimum posting size, no need

to undergo vq reset.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

If merged buffer is not done, and buffer posted are smaller than new mtu,

undergo vq reset optionally.

This can be done with or without sending mtu to device.

Yes, telling mtu to device helps device to optimize and adhere to the spec line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.

Again, that line refers to \field{mtu} which is the max mtu supported, irrespective to anything driver does.

BTW with any such ctrl vq interface we need to think how cases such as increasing and decreasing MTU work. The normal behaviour for linux drivers is to limit this to when the link is down. Which reminds me, we do not have a command to bring link down, either.

Quoted 2 lines
Signature · 1 line
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#41 lore raw
In reply to Michael S. Tsirkin
Quoted 27 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:59 PM

Quoted 22 lines · Parav Pandit
Quoted 22 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PM

Quoted 16 lines · Parav Pandit
Quoted 6 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AM

Quoted 3 lines
Quoted 2 lines

Should we make this depend on the vq reset ability maybe?

Quoted 7 lines
Quoted 4 lines

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Thanks

Device reset is really annoying in that it loses all the state:
rx filters etc etc.

The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

That will need a new feature bit.

Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?

We did feature bits for this in the past.

Ok. Will try to draft the update for future.

Gavin should repost the patch to address comments unrelated to this future bit anyway. Right?

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

If merged buffer is done, and new mtu is > minimum posting size, no need

to undergo vq reset.

Quoted 2 lines

If merged buffer is not done, and buffer posted are smaller than new mtu,

undergo vq reset optionally.

This can be done with or without sending mtu to device.

Yes, telling mtu to device helps device to optimize and adhere to the spec

Quoted 8 lines

line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.

Again, that line refers to \field{mtu} which is the max mtu supported, irrespective to anything driver does.

--
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#42 lore raw
Quoted 32 lines · Parav Pandit
Quoted 32 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:59 PM

Quoted 26 lines · Parav Pandit
Quoted 16 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PM

Quoted 11 lines · Parav Pandit
Quoted 11 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AM

Quoted 7 lines
Quoted 2 lines

Should we make this depend on the vq reset ability maybe?

The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Device reset is really annoying in that it loses all the state:
rx filters etc etc.

The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.

That will need a new feature bit.

Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?

Quoted 6 lines
Quoted 2 lines

We did feature bits for this in the past.

Ok. Will try to draft the update for future.

Gavin should repost the patch to address comments unrelated to this future bit anyway. Right?

Right.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

If merged buffer is done, and new mtu is > minimum posting size, no need

to undergo vq reset.

Quoted 2 lines

If merged buffer is not done, and buffer posted are smaller than new mtu,

undergo vq reset optionally.

This can be done with or without sending mtu to device.

Yes, telling mtu to device helps device to optimize and adhere to the spec

Quoted 8 lines
Quoted 8 lines

line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.

Again, that line refers to \field{mtu} which is the max mtu supported, irrespective to anything driver does.

--
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#43 lore raw
In reply to Michael S. Tsirkin
Quoted 55 lines · Michael S. Tsirkin
Quoted 54 lines · Michael S. Tsirkin
Quoted 53 lines · Si-Wei Liu
Quoted 51 lines · Parav Pandit
Quoted 39 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packets

Quoted 11 lines · Parav Pandit
Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM

[..]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.

How? Driver reduced the mXu to 1500, say it is improved to post buffers of

1500 bytes.

Quoted 2 lines

Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500

bytes.

Quoted 12 lines
Quoted 8 lines
Quoted 1 line

This is because device follows the spec " The device MUST NOT pass

received packets that exceed mtu".

The "mtu" here is the device config field, which is

          /* Default maximum transmit unit advice */

It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.

Quoted 1 line

there is no guarantee device will not get a bigger packet.

Quoted 9 lines
Quoted 2 lines

Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.

yes

Quoted 1 line

When user overrides it, driver can be further optimized to honor such new

value on rx buffer posting.

no, not without a feature bit promising device won't get wedged.

I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

I am also confused how the device can be wedged in this case.

Yea sorry. I misunderstood the code. It can't be.

Here's a problem as I see it. Let's say we reduce mtu. Small buffers are added. Now we increase mtu.
Device will drop all packets until small buffers are consumed.

Should we make this depend on the vq reset ability maybe?

To be honest I am not sure if worth it, very few user changes MTU on the fly with traffic ongoing, for the most cases I've seen users just change it only once in deployment time. Even if they change it on the fly they may need to be aware of the consequence and implication of loss of packets. In real devices, mtu change could end up with link status change and in that case there's usually no guarantee arrived  packet will be kept during the window.

While I could understand this would slightly introduce regression on functionality, as the worst case for packet loss if device dropping packets, it would be all smaller buffers of the full queue size. For correctness and elegance I don't mind introducing specific feature that changes MTU, or even relying on vq_reset is fine. device_reset would be too overwhelmed for this special use case IMHO.

-Siwei

Quoted 3 lines
Quoted 3 lines
Signature · 2 lines
-- 
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]