virtio-net: use mtu size as buffer length for big packets
Subsystems (3)
PATCH 1/1 virtio-net: use mtu size as buffer length for big packets
Currently add_recvbuf_big() allocates MAX_SKB_FRAGS segments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.
Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.
Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After
1500 22.5 22.4 9000 12.8 25.9
--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/net/virtio_net.c +16 -4
@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big (struct virtnet_info *vi,struct receive_queue *rq,gfp_t gfp) { +unsigned int sg_num =MAX_SKB_FRAGS ;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ; + } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ -for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)
@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages (rq, first);
@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true; + }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true;
Signature · 6 lines
2.31.1 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 76 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.
Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +13
-3@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big (struct virtnet_info *vi,struct receive_queue *rq,gfp_t gfp) { +unsigned int sg_num =MAX_SKB_FRAGS ;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ; + } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ -for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages (rq, first);
drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586
Quoted 7 lines
drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;
Why not simply re-use big_packets here?
Thanks
Quoted 8 lines
drivers/net/virtio_net.c +1
-1+ }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true; --2.31.1
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 87 lines · Jason Wang
Quoted 76 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.
Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +13
-3@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big (struct virtnet_info *vi,struct receive_queue *rq,gfp_t gfp) { +unsigned int sg_num =MAX_SKB_FRAGS ;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ; + } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ -for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages (rq, first);drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586
Quoted 7 lines
drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Why not simply re-use
big_packetshere?
Quoted 1 line
Thanks
I don't get this question. The patch does use big_packets, it wants to figure out guest GSO is off so MTU limits the size. The name "gso_is_supported" is confusing, should be e.g. guest_gso.
Quoted 8 lines
Quoted 8 lines
drivers/net/virtio_net.c +1
-1+ }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true; --2.31.1
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 89 lines · Michael S. Tsirkin
Quoted 87 lines · Jason Wang
Quoted 76 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.
Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +13
-3@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big (struct virtnet_info *vi,struct receive_queue *rq,gfp_t gfp) { +unsigned int sg_num =MAX_SKB_FRAGS ;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ; + } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ -for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages (rq, first);drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586
Quoted 7 lines
drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Why not simply re-use
big_packetshere?
Quoted 4 lines
Quoted 1 line
Thanks
I don't get this question. The patch does use
big_packets, it wants to figure out guest GSO is off so MTU limits the size.
Yes.
Thanks
Quoted 16 lines
The name "
gso_is_supported" is confusing, should be e.g.guest_gso.Quoted 12 lines
Quoted 12 lines
drivers/net/virtio_net.c +1
-1+ }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true; --2.31.1
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 91 lines · Jason Wang
Quoted 89 lines · Michael S. Tsirkin
Quoted 87 lines · Jason Wang
Quoted 76 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.
Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +13
-3@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big (struct virtnet_info *vi,struct receive_queue *rq,gfp_t gfp) { +unsigned int sg_num =MAX_SKB_FRAGS ;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ; + } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ -for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages (rq, first);drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586
Quoted 7 lines
drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Why not simply re-use
big_packetshere?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Thanks
I don't get this question. The patch does use
big_packets, it wants to figure out guest GSO is off so MTU limits the size.Yes.
Thanks
I wonder if it's better to introduce the boolean here:
/* TODO: size buffers correctly in this case. */
if (dev->mtu > ETH_DATA_LEN)
vi->big_packets = true;Thanks
Quoted 12 lines
Quoted 12 lines
The name "
gso_is_supported" is confusing, should be e.g.guest_gso.Quoted 8 lines
Quoted 8 lines
drivers/net/virtio_net.c +1
-1+ }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true; --2.31.1
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 3 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,
GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
Quoted 7 lines
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
Quoted 37 lines
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big (struct virtnet_info *vi,struct receive_queue *rq,gfp_t gfp) {
drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, · line 1326
Quoted 10 lines
drivers/net/virtio_net.c +5
-1+unsigned int sg_num =MAX_SKB_FRAGS ;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ;
DIV_ROUND_UP() can be used?
Since this branch slightly adds up cost to the datapath, I wonder if this sg_num can be saved and set only once (generally in virtnet_probe time) in struct virtnet_info?
drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, · line 1331
Quoted 5 lines
drivers/net/virtio_net.c +3
-0+ } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
Comment doesn't match code.
Quoted 15 lines
drivers/net/virtio_net.c +2
-2-for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages (rq, first);
drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586
Quoted 7 lines
drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;
Please do the same for virtnet_clear_guest_offloads(), and correspondingly virtnet_restore_guest_offloads() as well. Not sure why virtnet_clear_guest_offloads() or the caller doesn't unset big_packet on successful return, seems like a bug to me.
Thanks,
-Siwei
drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3590
Quoted 4 lines
drivers/net/virtio_net.c +1
-0+ }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 64 lines · Si-Wei Liu
Quoted 3 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
Quoted 7 lines
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
Quoted 37 lines
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After
1500 22.5 22.4
9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>---
drivers/net/virtio_net.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +0
-0@@ -222,6 +222,9 @@ struct virtnet_info { /* I like... big packets and I cannot lie! */ bool big_packets;+ /* Indicates GSO support */
+ boolgso_is_supported;
+
/* Host will merge rx buffers for big packets (shake it! shake it!) */
boolmergeable_rx_bufs;
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(structvirtnet_info*vi, structreceive_queue*rq, static int add_recvbuf_big(structvirtnet_info*vi, structreceive_queue*rq,
gfp_tgfp)
{
+ unsigned intsg_num=MAX_SKB_FRAGS;
struct page *first, *list = NULL;Quoted 8 lines
char *p;
int i, err, offset;
-sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+ if (!vi->gso_is_supported) {
+ unsigned int mtu =vi->dev->mtu;
+
+sg_num= (mtu %PAGE_SIZE) ? mtu /PAGE_SIZE+ 1 : mtu /PAGE_SIZE;
DIV_ROUND_UP()can be used?
Quoted 1 line
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime
... , but can align with new mtu during .ndo_change_mtu(), too.
Quoted 1 line
) in struct
virtnet_info?
Thanks,
-Siwei
Quoted 31 lines
Quoted 4 lines
+ }
+
+sg_init_table(rq->sg, sg_num + 2);
/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */Comment doesn't match code.
Quoted 16 lines
- for (i =
MAX_SKB_FRAGS+ 1; i > 1; --i) {
+ for (i =sg_num+ 1; i > 1; --i) {
first =get_a_page(rq, gfp);
if (!first) {
if (list)drivers/net/virtio_net.c +1
-1@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, /* chain first in list head */ first->private = (unsigned long)list; - err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2, + err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2, first, gfp); if (err < 0) give_pages(rq, first);@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)Quoted 7 lines
if (
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN)|| -virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {vi->big_packets= true;
+vi->gso_is_supported= true;Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.
Quoted 6 lines
Thanks,
-SiweiQuoted 3 lines
+ }
if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 89 lines · Michael S. Tsirkin
External email: Use caution opening links or attachments
Quoted 85 lines · Jason Wang
Quoted 75 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.
Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +13
-3@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big (struct virtnet_info *vi,struct receive_queue *rq,gfp_t gfp) { +unsigned int sg_num =MAX_SKB_FRAGS ;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ; + } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ -for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages (rq, first);drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586
Quoted 7 lines
drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Why not simply re-use
big_packetshere?
Quoted 4 lines
Quoted 1 line
Thanks
I don't get this question. The patch does use
big_packets, it wants to figure out guest GSO is off so MTU limits the size. The name "gso_is_supported" is confusing, should be e.g.guest_gso.
ACK.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/net/virtio_net.c +1
-1+ }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true; --2.31.1
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 96 lines · Jason Wang
External email: Use caution opening links or attachments
Quoted 92 lines · Jason Wang
Quoted 86 lines · Michael S. Tsirkin
Quoted 85 lines · Jason Wang
Quoted 75 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported, it would be sufficient to allocate segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported, use the MTU to calculate the optimal amount of segments required.
Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +13
-3@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big (struct virtnet_info *vi,struct receive_queue *rq,gfp_t gfp) { +unsigned int sg_num =MAX_SKB_FRAGS ;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ; + } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ -for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages (rq, first);drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3586
Quoted 7 lines
drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Why not simply re-use
big_packetshere?Quoted 3 lines
Quoted 1 line
Thanks
I don't get this question. The patch does use
big_packets, it wants to figure out guest GSO is off so MTU limits the size.Yes.
Quoted 8 lines
Quoted 1 line
Thanks
I wonder if it's better to introduce the boolean here:
/* TODO: size buffers correctly in this case. */ if (dev->mtu > ETH_DATA_LEN) vi->big_packets = true;Thanks
This is not a safe or straightforward way to determine that guest GSO is not supported, as big_packets just indicates the packets can be big, not specifically guest GSO.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
The name "
gso_is_supported" is confusing, should be e.g.guest_gso.Quoted 8 lines
Quoted 8 lines
drivers/net/virtio_net.c +1
-1+ }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true; --2.31.1
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 9 lines · Si-Wei Liu
External email: Use caution opening links or attachments
Quoted 3 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
Quoted 9 lines
Quoted 8 lines
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
Quoted 44 lines
Quoted 44 lines
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After
1500 22.5 22.4
9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>---
drivers/net/virtio_net.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +4
-0@@ -222,6 +222,9 @@ struct virtnet_info { /* I like... big packets and I cannot lie! */ bool big_packets; + /* Indicates GSO support */ + bool gso_is_supported; + /* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, gfp_t gfp) { + unsigned int sg_num = MAX_SKB_FRAGS; struct page *first, *list = NULL; char *p; int i, err, offset;
Quoted 7 lines
drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
Quoted 6 lines
drivers/net/virtio_net.c +4
-1- sg_init_table(rq->sg, MAX_SKB_FRAGS + 2); + if (!vi->gso_is_supported) { + unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu /PAGE_SIZE ;
DIV_ROUND_UP()can be used?
ACK
Quoted 4 lines
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?
Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea? ndo_change_mtu might be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.
Quoted 6 lines
drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, · line 1331
Quoted 5 lines
drivers/net/virtio_net.c +3
-0+ } + + sg_init_table(rq->sg, sg_num + 2); /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */Comment doesn't match code.
ACK
Quoted 20 lines
Quoted 20 lines
drivers/net/virtio_net.c +1
-1- for (i = MAX_SKB_FRAGS + 1; i > 1; --i) { + for (i = sg_num + 1; i > 1; --i) { first = get_a_page(rq, gfp);if (!first) {
if (list)drivers/net/virtio_net.c +1
-1@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, /* chain first in list head */ first->private = (unsigned long)list; - err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2, + err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2, first, gfp); if (err < 0)
give_pages(rq, first);drivers/net/virtio_net.c +0
-0@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN)||
Quoted 8 lines
Quoted 4 lines
-
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {vi->big_packets= true;
+vi->gso_is_supported= true;Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.
ACK. The two calls virtnet_set_guest_offloads and virtnet_set_guest_offloads is also called by virtnet_set_features. Do you think if I can do this in virtnet_set_guest_offloads?
Quoted 9 lines
Thanks,
-SiweiQuoted 4 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 72 lines · Si-Wei Liu
External email: Use caution opening links or attachments
Quoted 68 lines · Si-Wei Liu
Quoted 3 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
Quoted 7 lines
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
Quoted 40 lines
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After
1500 22.5 22.4
9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>---
drivers/net/virtio_net.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +0
-0@@ -222,6 +222,9 @@ struct virtnet_info { /* I like... big packets and I cannot lie! */ bool big_packets;+ /* Indicates GSO support */
+ boolgso_is_supported;
+
/* Host will merge rx buffers for big packets (shake it! shake it!) */
boolmergeable_rx_bufs;
@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(structvirtnet_info*vi, structreceive_queue*rq,
static int add_recvbuf_big(structvirtnet_info*vi, structreceive_queue*rq,
gfp_tgfp)
{
+ unsigned intsg_num=MAX_SKB_FRAGS;
struct page *first, *list = NULL;Quoted 8 lines
char *p;
int i, err, offset;
-sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+ if (!vi->gso_is_supported) {
+ unsigned int mtu =vi->dev->mtu;
+
+sg_num= (mtu %PAGE_SIZE) ? mtu /PAGE_SIZE+ 1 : mtu /PAGE_SIZE;
DIV_ROUND_UP()can be used?
Quoted 4 lines
Quoted 3 lines
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime... , but can align with new mtu during .ndo_change_mtu(), too.
ACK but don't know how to do this.
Quoted 46 lines
Quoted 1 line
) in struct
virtnet_info?Thanks,
-SiweiQuoted 40 lines
Quoted 4 lines
+ }
+
+sg_init_table(rq->sg, sg_num + 2);
/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */Comment doesn't match code.
Quoted 16 lines
- for (i =
MAX_SKB_FRAGS+ 1; i > 1; --i) {
+ for (i =sg_num+ 1; i > 1; --i) {
first =get_a_page(rq, gfp);
if (!first) {
if (list)drivers/net/virtio_net.c +1
-1@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, /* chain first in list head */ first->private = (unsigned long)list; - err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2, + err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2, first, gfp); if (err < 0) give_pages(rq, first);@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)Quoted 7 lines
if (
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN)|| -virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {vi->big_packets= true;
+vi->gso_is_supported= true;Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.Thanks,
-SiweiQuoted 3 lines
+ }
if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 73 lines · Gavin Li
Quoted 9 lines · Si-Wei Liu
External email: Use caution opening links or attachments
Quoted 3 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
Quoted 9 lines
Quoted 8 lines
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
Quoted 44 lines
Quoted 44 lines
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After
1500 22.5 22.4
9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>---
drivers/net/virtio_net.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +4
-0@@ -222,6 +222,9 @@ struct virtnet_info { /* I like... big packets and I cannot lie! */ bool big_packets; + /* Indicates GSO support */ + bool gso_is_supported; + /* Host will merge rx buffers for big packets (shake it! shake it!) */ bool mergeable_rx_bufs;@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, gfp_t gfp) { + unsigned int sg_num = MAX_SKB_FRAGS; struct page *first, *list = NULL; char *p; int i, err, offset;Quoted 7 lines
drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
Quoted 6 lines
drivers/net/virtio_net.c +4
-1- sg_init_table(rq->sg, MAX_SKB_FRAGS + 2); + if (!vi->gso_is_supported) { + unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu % PAGE_SIZE) ? mtu / PAGE_SIZE + 1 : mtu /PAGE_SIZE ;
DIV_ROUND_UP()can be used?
Quoted 6 lines
ACK
Quoted 4 lines
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.
Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original HOST_MTU. In the vendor defined .ndo_change_mtu() op, dev_validate_mtu() should have validated the MTU value before coming down to it. And I suspect you might want to do virtnet_close() and virtnet_open() before/after changing the buffer size on the fly (the netif_running() case), implementing .ndo_change_mtu() will be needed anyway.
Quoted 35 lines
Quoted 6 lines
drivers/net/virtio_net.c static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, · line 1331
Quoted 5 lines
drivers/net/virtio_net.c +3
-0+ } + + sg_init_table(rq->sg, sg_num + 2); /* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */Comment doesn't match code.
ACK
Quoted 20 lines
Quoted 20 lines
drivers/net/virtio_net.c +1
-1- for (i = MAX_SKB_FRAGS + 1; i > 1; --i) { + for (i = sg_num + 1; i > 1; --i) { first = get_a_page(rq, gfp);if (!first) {
if (list)drivers/net/virtio_net.c +1
-1@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, /* chain first in list head */ first->private = (unsigned long)list; - err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2, + err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2, first, gfp); if (err < 0)
give_pages(rq, first);drivers/net/virtio_net.c +0
-0@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN)||Quoted 8 lines
Quoted 4 lines
-
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {vi->big_packets= true;
+vi->gso_is_supported= true;Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.
Quoted 1 line
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?
I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Quoted 9 lines
Quoted 9 lines
Thanks,
-SiweiQuoted 4 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 88 lines · Si-Wei Liu
External email: Use caution opening links or attachments
Quoted 77 lines · Gavin Li
Quoted 9 lines · Si-Wei Liu
External email: Use caution opening links or attachments
Quoted 3 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
Quoted 9 lines
Quoted 8 lines
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
Quoted 46 lines
Quoted 46 lines
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After
1500 22.5 22.4
9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>---
drivers/net/virtio_net.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info { /* I like... big packets and I cannot lie! */ bool big_packets; + /* Indicates GSO support */ + bool gso_is_supported; + /* Host will merge rx buffers for big packets (shake it! shakeit!) */
boolmergeable_rx_bufs;drivers/net/virtio_net.c +0
-0@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big(struct
virtnet_info*vi, structreceive_queue*rq,
gfp_tgfp)
{
+ unsigned intsg_num=MAX_SKB_FRAGS;
struct page *first, *list = NULL;
char *p;
int i, err, offset;Quoted 7 lines
Quoted 6 lines
-
sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+ if (!vi->gso_is_supported) {
+ unsigned int mtu =vi->dev->mtu;
+
+sg_num= (mtu %PAGE_SIZE) ? mtu /PAGE_SIZE+ 1 : mtu /PAGE_SIZE;
DIV_ROUND_UP()can be used?ACK
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.
Quoted 8 lines
Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op,dev_validate_mtu()should have validated the MTU value before coming down to it. And I suspect you might want to dovirtnet_close()andvirtnet_open()before/after changing the buffer size on the fly (thenetif_running()case), implementing .ndo_change_mtu() will be needed anyway.
a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing close() and open() sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.
Quoted 54 lines
Quoted 36 lines
Quoted 7 lines
Quoted 6 lines
+ }
+
+sg_init_table(rq->sg, sg_num + 2);/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
Comment doesn't match code.
ACK
Quoted 20 lines
Quoted 20 lines
- for (i =
MAX_SKB_FRAGS+ 1; i > 1; --i) {
+ for (i =sg_num+ 1; i > 1; --i) {
first =get_a_page(rq, gfp);
if (!first) {
if (list)drivers/net/virtio_net.c +1
-1@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, /* chain first in list head */ first->private = (unsigned long)list; - err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2, + err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2, first, gfp); if (err < 0)
give_pages(rq, first);drivers/net/virtio_net.c +0
-0@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN)||Quoted 8 lines
Quoted 4 lines
-
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {vi->big_packets= true;
+vi->gso_is_supported= true;Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.Quoted 3 lines
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Quoted 9 lines
Quoted 9 lines
Thanks,
-SiweiQuoted 4 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))vi->mergeable_rx_bufs= true;
Quoted 115 lines · Gavin Li
On 8/9/2022 7:56 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/8/2022 12:31 AM, Gavin Li wrote:
On 8/6/2022 6:11 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/1/2022 9:45 PM, Gavin Li wrote:
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */
bool mergeable_rx_bufs;drivers/net/virtio_net.c +5
-1@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, gfp_t gfp) { + unsigned int sg_num = MAX_SKB_FRAGS;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ;
DIV_ROUND_UP()can be used?ACK
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op,dev_validate_mtu()should have validated the MTU value before coming down to it. And I suspect you might want to dovirtnet_close()andvirtnet_open()before/after changing the buffer size on the fly (thenetif_running()case), implementing .ndo_change_mtu() will be needed anyway.
Quoted 1 line
a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing
close()andopen()sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.
Right, it could be done on top.
But another note is that, it would still be better to support GUEST GSO feature:
1) can work for the case for path MTU
2) (migration)compatibility with software backends
Quoted 37 lines
drivers/net/virtio_net.c +3
-0+ } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ Comment doesn't match code.
ACK
drivers/net/virtio_net.c +2
-2-for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages(rq, first);drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;
Quoted 4 lines
Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.
It is fine as long as
1) we don't implement ethtool API for changing guest offloads
2) big mode XDP is not enabled
So that code works only for XDP but we forbid big packets in the case of XDP right now.
Thanks
Quoted 16 lines
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Thanks,
drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 3590
Quoted 6 lines
drivers/net/virtio_net.c +1
-1-Siwei + }if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) vi->mergeable_rx_bufs = true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 17 lines · Jason Wang
Quoted 10 lines
drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Quoted 4 lines
Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.It is fine as long as
Quoted 7 lines
1) we don't implement ethtool API for changing guest offloads
2) big mode XDP is not enabledSo that code works only for XDP but we forbid big packets in the case of XDP right now.
Thanks
To put it another way, changing big_packets after probe requires a bunch of work as current code assumes this flag never changes. Adding a TODO to handle dynamic offload config is fine but
I don't think it should block this.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 1 line · Jason Wang
Quoted 1 line
+ unsigned int mtu =
vi->dev->mtu;
BTW should this not be max_mtu? Otherwise if user configures mtu that is too small we'll add buffers that are too small. some backends simply lock up if this happens (I think vhost does).
Maybe we should add a feature to allow packet drop if it's too small. And send mtu guest to host while we are at it?
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 25 lines · Michael S. Tsirkin
Quoted 23 lines · Jason Wang
Quoted 10 lines
(quoted diff) +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev) if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6) || virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN) || - virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) + virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) { vi->big_packets = true; + vi->gso_is_supported = true;Quoted 4 lines
Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.It is fine as long as
1) we don't implement ethtool API for changing guest offloads
2) big mode XDP is not enabledSo that code works only for XDP but we forbid big packets in the case of XDP right now.
Quoted 6 lines
Quoted 1 line
Thanks
To put it another way, changing
big_packetsafter probe requires a bunch of work as current code assumes this flag never changes. Adding a TODO to handle dynamic offload config is fine but
I don't think it should block this.
Yes, this is what I mean.
Thanks
Quoted 4 lines
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 5 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang
Quoted 1 line
+ unsigned int mtu =
vi->dev->mtu;BTW should this not be
max_mtu?
Yes.
Quoted 3 lines
Otherwise if user configures mtu that
is too small we'll add buffers that are too small. some backends simply lock up if this happens (I think vhost does).
Probably not? If we run out of buffers, we will wait for the next kick. (Otherwise it would be an guest triggerable behaviour)
Quoted 1 line
Maybe we should add a feature to allow packet drop if it's too small.
I may miss something but isn't this what most devices will do? (e.g ethernet have a minimal packet length)
Thanks
Quoted 5 lines
And send mtu guest to host while we are at it?
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 91 lines · Gavin Li
Quoted 88 lines · Si-Wei Liu
External email: Use caution opening links or attachments
Quoted 84 lines · Gavin Li
Quoted 9 lines · Si-Wei Liu
External email: Use caution opening links or attachments
Quoted 3 lines · Gavin Li
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
Quoted 9 lines
Quoted 8 lines
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
Quoted 46 lines
Quoted 46 lines
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After
1500 22.5 22.4
9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>---
drivers/net/virtio_net.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info { /* I like... big packets and I cannot lie! */ bool big_packets; + /* Indicates GSO support */ + bool gso_is_supported; + /* Host will merge rx buffers for big packets (shake it! shakeit!) */
boolmergeable_rx_bufs;drivers/net/virtio_net.c +0
-0@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big(struct
virtnet_info*vi, structreceive_queue*rq,
gfp_tgfp)
{
+ unsigned intsg_num=MAX_SKB_FRAGS;
struct page *first, *list = NULL;
char *p;
int i, err, offset;Quoted 7 lines
Quoted 6 lines
-
sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+ if (!vi->gso_is_supported) {
+ unsigned int mtu =vi->dev->mtu;
+
+sg_num= (mtu %PAGE_SIZE) ? mtu /PAGE_SIZE+ 1 : mtu /PAGE_SIZE;
DIV_ROUND_UP()can be used?ACK
Quoted 4 lines
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.
Quoted 8 lines
Quoted 7 lines
Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op,dev_validate_mtu()
should have validated the MTU value before coming down to it. And I
suspect you might want to dovirtnet_close()andvirtnet_open()
before/after changing the buffer size on the fly (thenetif_running()
case), implementing .ndo_change_mtu() will be needed anyway.a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing
close()andopen()sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.
Sure, it's fine to separate it out into another patch and optimize on top later on. Though the previous comment of avoiding repeatedly computing sg_num in datapath still holds: set sg_num only once in probe() time then, in which case the sg_num needed can be simply inferred from dev->max_mtu rather than dev->mtu.
-Siwei
Quoted 54 lines
Quoted 54 lines
Quoted 36 lines
Quoted 7 lines
Quoted 6 lines
+ }
+
+sg_init_table(rq->sg, sg_num + 2);/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
Comment doesn't match code.
ACK
Quoted 20 lines
Quoted 20 lines
- for (i =
MAX_SKB_FRAGS+ 1; i > 1; --i) {
+ for (i =sg_num+ 1; i > 1; --i) {
first =get_a_page(rq, gfp);
if (!first) {
if (list)drivers/net/virtio_net.c +1
-1@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, /* chain first in list head */ first->private = (unsigned long)list; - err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2, + err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2, first, gfp); if (err < 0)
give_pages(rq, first);drivers/net/virtio_net.c +0
-0@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN)||Quoted 8 lines
Quoted 4 lines
-
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {vi->big_packets= true;
+vi->gso_is_supported= true;Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.Quoted 3 lines
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Quoted 9 lines
Quoted 9 lines
Thanks,
-SiweiQuoted 4 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))vi->mergeable_rx_bufs= true;
Quoted 159 lines · Jason Wang
Quoted 114 lines · Gavin Li
On 8/9/2022 7:56 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/8/2022 12:31 AM, Gavin Li wrote:
On 8/6/2022 6:11 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/1/2022 9:45 PM, Gavin Li wrote:
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After 1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */
bool mergeable_rx_bufs;drivers/net/virtio_net.c +5
-1@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, gfp_t gfp) { + unsigned int sg_num = MAX_SKB_FRAGS;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ;
DIV_ROUND_UP()can be used?ACK
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op,dev_validate_mtu()should have validated the MTU value before coming down to it. And I suspect you might want to dovirtnet_close()andvirtnet_open()before/after changing the buffer size on the fly (thenetif_running()case), implementing .ndo_change_mtu() will be needed anyway.Quoted 1 line
a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing
close()andopen()sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.Right, it could be done on top.
But another note is that, it would still be better to support GUEST GSO feature:
1) can work for the case for path MTU
2) (migration)compatibility with software backendsQuoted 36 lines
drivers/net/virtio_net.c +3
-0+ } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ Comment doesn't match code.
ACK
drivers/net/virtio_net.c +2
-2-for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages(rq, first);drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;
Quoted 7 lines
Quoted 4 lines
Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.It is fine as long as
1) we don't implement ethtool API for changing guest offloads
Not sure if I missed something, but it looks the current virtnet_set_features() already supports toggling on/off GRO HW through commit a02e8964eaf9271a8a5fcc0c55bd13f933bafc56 (formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads() or the caller doesn't unset big_packet ...".
Quoted 1 line
2) big mode XDP is not enabled
Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.
Regards,
-Siwei
Quoted 20 lines
So that code works only for XDP but we forbid big packets in the case of XDP right now.
Thanks
Quoted 14 lines
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Thanks,
-Siwei
Quoted 7 lines
Quoted 7 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 6 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PMCurrently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.
Which application benefit by having asymmetry by lowering mtu to 1500 to send packets but want to receive 9K packets?
Quoted 6 lines · Parav Pandit
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM
Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500 to send packets but want to receive 9K packets?
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000. What I actually tried to say is that as our current use case (software virtio) supports XDP applications with 1500 guest mtu and mergeable buffer enabled on a 9000 MTU link, technically it's a legitimate use case regardless of mergeable buffer capability. Otherwise it's considered to be a usability regression in driver software side when migrating (not live migrating) existing user to vdpa, due to lack of hardware implementation of certain relevant feature.
Thanks,
-Siwei
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 10 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 6 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.
Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".
So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).
Quoted 7 lines · Parav Pandit
Quoted 7 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 5 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Which application benefit by having asymmetry by lowering mtu to 1500
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.
For big_packet path, yes, we need improvement; for mergeable, it's adaptable to any incoming packet size so 1500 is what it is today.
Quoted 4 lines
Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".
Right, that's what it happens today on device side (i.e. vhost-net, btw mlx5 vdpa device seems to have a bug not pro-actively dropping packets that exceed the MTU size, causing guest panic in small packet path).
Quoted 3 lines
So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).
Usually, the use case I'm aware of would set the peer's MTU to 1500 (e.g. on a virtual network appliance), or it would rely on path mtu discovery to avoid packet drop across links.
-Siwei
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 32 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 4:33 PMQuoted 9 lines · Parav Pandit
Quoted 9 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 7 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
Quoted 7 lines
Quoted 1 line
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Forbig_packetpath, yes, we need improvement; for mergeable, it's adaptable to any incoming packet size so 1500 is what it is today.Quoted 4 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500 bytes. This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
Right, that's what it happens today on device side (i.e. vhost-net, btw mlx5 vdpa device seems to have a bug not pro-actively dropping packets that exceed the MTU size, causing guest panic in small packet path).Quoted 1 line
Quoted 7 lines
Quoted 1 line
So, I am lost what virtio net device user application is trying to achieve by
sending smaller packets and dropping all receive packets.
Quoted 1 line
(it doesn’t have any relation to mergeable or otherwise).
Usually, the use case I'm aware of would set the peer's MTU to 1500 (e.g. on a virtual network appliance), or it would rely on path mtu discovery to avoid packet drop across links.
Ok. Somehow the application knows the mtu to set on (all) peer(s) and hope for the best. Understood.
Michael S. Tsirkin
maintainerQuoted 33 lines · Parav Pandit
Quoted 33 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 4:33 PMQuoted 9 lines · Parav Pandit
Quoted 9 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 7 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
Quoted 7 lines
Quoted 1 line
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Forbig_packetpath, yes, we need improvement; for mergeable, it's adaptable to any incoming packet size so 1500 is what it is today.Quoted 4 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500 bytes. This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
Right, that's what it happens today on device side (i.e. vhost-net, btw mlx5 vdpa device seems to have a bug not pro-actively dropping packets that exceed the MTU size, causing guest panic in small packet path).Quoted 2 lines
So, I am lost what virtio net device user application is trying to achieve by
Quoted 8 lines
Quoted 6 lines
sending smaller packets and dropping all receive packets.
Quoted 1 line
(it doesn’t have any relation to mergeable or otherwise).
Usually, the use case I'm aware of would set the peer's MTU to 1500 (e.g. on a virtual network appliance), or it would rely on path mtu discovery to avoid packet drop across links.
Ok. Somehow the application knows the mtu to set on (all) peer(s) and hope for the best. Understood.
That's generally what one has to do with mtu, yes - it has to be set consistently across the LAN. While e.g. pMTU might help work around some misconfigured LANs with a mix of different MTUs it was never designed for that.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 168 lines · Si-Wei Liu
Quoted 165 lines · Jason Wang
Quoted 114 lines · Gavin Li
On 8/9/2022 7:56 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/8/2022 12:31 AM, Gavin Li wrote:
On 8/6/2022 6:11 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/1/2022 9:45 PM, Gavin Li wrote:
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After 1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */
bool mergeable_rx_bufs;drivers/net/virtio_net.c +5
-1@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, gfp_t gfp) { + unsigned int sg_num = MAX_SKB_FRAGS;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ;
DIV_ROUND_UP()can be used?ACK
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op,dev_validate_mtu()should have validated the MTU value before coming down to it. And I suspect you might want to dovirtnet_close()andvirtnet_open()before/after changing the buffer size on the fly (thenetif_running()case), implementing .ndo_change_mtu() will be needed anyway.Quoted 1 line
a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing
close()andopen()sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.Right, it could be done on top.
But another note is that, it would still be better to support GUEST GSO feature:
1) can work for the case for path MTU
2) (migration)compatibility with software backendsQuoted 36 lines
drivers/net/virtio_net.c +3
-0+ } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ Comment doesn't match code.
ACK
drivers/net/virtio_net.c +2
-2-for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages(rq, first);drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Quoted 4 lines
Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.It is fine as long as
Quoted 6 lines
Quoted 1 line
1) we don't implement ethtool API for changing guest offloads
Not sure if I missed something, but it looks the current
virtnet_set_features()already supports toggling on/off GRO HW through commita02e8964eaf9271a8a5fcc0c55bd13f933bafc56(formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads()or the caller doesn't unsetbig_packet...".
"we" here is the device, not the driver.
Quoted 27 lines
Quoted 1 line
2) big mode XDP is not enabled
Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.
Regards,
-SiweiQuoted 20 lines
So that code works only for XDP but we forbid big packets in the case of XDP right now.
Thanks
Quoted 14 lines
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Thanks,
-Siwei
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 13 lines · Parav Pandit
Quoted 10 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 6 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 8 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.
Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".
The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */
there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
Quoted 2 lines
So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 166 lines · Michael S. Tsirkin
Quoted 165 lines · Si-Wei Liu
Quoted 163 lines · Jason Wang
Quoted 113 lines · Gavin Li
On 8/9/2022 7:56 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/8/2022 12:31 AM, Gavin Li wrote:
On 8/6/2022 6:11 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/1/2022 9:45 PM, Gavin Li wrote:
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After 1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */
bool mergeable_rx_bufs;drivers/net/virtio_net.c +5
-1@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, gfp_t gfp) { + unsigned int sg_num = MAX_SKB_FRAGS;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ;
DIV_ROUND_UP()can be used?ACK
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op,dev_validate_mtu()should have validated the MTU value before coming down to it. And I suspect you might want to dovirtnet_close()andvirtnet_open()before/after changing the buffer size on the fly (thenetif_running()case), implementing .ndo_change_mtu() will be needed anyway.Quoted 1 line
a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing
close()andopen()sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.Right, it could be done on top.
But another note is that, it would still be better to support GUEST GSO feature:
1) can work for the case for path MTU
2) (migration)compatibility with software backendsQuoted 35 lines
drivers/net/virtio_net.c +3
-0+ } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ Comment doesn't match code.
ACK
drivers/net/virtio_net.c +2
-2-for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages(rq, first);drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Quoted 4 lines
Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.It is fine as long as
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
1) we don't implement ethtool API for changing guest offloads
Not sure if I missed something, but it looks the current
virtnet_set_features()already supports toggling on/off GRO HW through commita02e8964eaf9271a8a5fcc0c55bd13f933bafc56(formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads()or the caller doesn't unsetbig_packet..."."we" here is the device, not the driver.
What is the ethtool API from device level? VIRTIO_NET_F_CTRL_GUEST_OFFLOADS for sure, right?
It's implemented in software backend as far as I know. I see no reason *technically* this is infeasible, regardless what you name it, being a bug or TODO.
-Siwei
Quoted 27 lines
Quoted 27 lines
Quoted 2 lines
2) big mode XDP is not enabled
Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.
Regards,
-SiweiQuoted 19 lines
So that code works only for XDP but we forbid big packets in the case of XDP right now.
Thanks
Quoted 14 lines
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Thanks,
-Siwei
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM
[..]
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 4 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500 bytes. This is because device follows the spec " The device MUST NOT pass
Quoted 7 lines
received packets that exceed mtu".
The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */
It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.
When user overrides it, driver can be further optimized to honor such new value on rx buffer posting.
Quoted 3 lines
And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
If user space insists on small buffers, so be it. It only works when user exactly know what user is doing in the whole network. When user prefers to override the device RO field, device is in the dark and things work on best effort basis. This must be a reasonably advance user who has good knowledge of its network topology etc.
For such case, may be yes, driver should be further optimized.
Michael S. Tsirkin
maintainerQuoted 21 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 4 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500 bytes. This is because device follows the spec " The device MUST NOT pass
Quoted 7 lines
received packets that exceed mtu".
The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */
Quoted 6 lines
It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.
yes
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new value on rx buffer posting.
no, not without a feature bit promising device won't get wedged.
Quoted 4 lines
Quoted 3 lines
And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
If user space insists on small buffers, so be it.
If previously things worked, the "so be it" is a regression and blaming users won't help us.
Quoted 1 line
It only works when user exactly know what user is doing in the whole network.
If you want to claim this you need a new feature bit.
Quoted 1 line
When user prefers to override the device RO field, device is in the dark and things work on best effort basis.
Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.
Quoted 4 lines
This must be a reasonably advance user who has good knowledge of its network topology etc.
For such case, may be yes, driver should be further optimized.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 170 lines · Si-Wei Liu
Quoted 167 lines · Michael S. Tsirkin
Quoted 166 lines · Si-Wei Liu
Quoted 164 lines · Jason Wang
Quoted 113 lines · Gavin Li
On 8/9/2022 7:56 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/8/2022 12:31 AM, Gavin Li wrote:
On 8/6/2022 6:11 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/1/2022 9:45 PM, Gavin Li wrote:
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After 1500 22.5 22.4 9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>--- drivers/net/virtio_net.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info {/* I like... big packets and I cannot lie! */ bool big_packets; +/* Indicates GSO support */ +bool gso_is_supported; +/* Host will merge rx buffers for big packets (shake it! shake it!) */
bool mergeable_rx_bufs;drivers/net/virtio_net.c +5
-1@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, gfp_t gfp) { + unsigned int sg_num = MAX_SKB_FRAGS;struct page *first, *list =NULL ;char *p;int i, err, offset; -sg_init_table (rq->sg,MAX_SKB_FRAGS +2 ); +if (!vi->gso_is_supported) { +unsigned int mtu = vi->dev->mtu; + + sg_num = (mtu %PAGE_SIZE ) ? mtu /PAGE_SIZE +1 : mtu /PAGE_SIZE ;
DIV_ROUND_UP()can be used?ACK
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op,dev_validate_mtu()should have validated the MTU value before coming down to it. And I suspect you might want to dovirtnet_close()andvirtnet_open()before/after changing the buffer size on the fly (thenetif_running()case), implementing .ndo_change_mtu() will be needed anyway.Quoted 1 line
a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing
close()andopen()sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.Right, it could be done on top.
But another note is that, it would still be better to support GUEST GSO feature:
1) can work for the case for path MTU
2) (migration)compatibility with software backendsQuoted 35 lines
drivers/net/virtio_net.c +3
-0+ } + +sg_init_table (rq->sg, sg_num +2 );/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */ Comment doesn't match code.
ACK
drivers/net/virtio_net.c +2
-2-for (i =MAX_SKB_FRAGS +1 ; i >1 ; --i) { +for (i = sg_num +1 ; i >1 ; --i) { first =get_a_page (rq, gfp);if (!first) {if (list)@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,/* chain first in list head */ first->private = (unsigned long )list; - err =virtqueue_add_inbuf (rq->vq, rq->sg,MAX_SKB_FRAGS +2 , + err =virtqueue_add_inbuf (rq->vq, rq->sg, sg_num +2 , first, gfp);if (err <0 )give_pages(rq, first);drivers/net/virtio_net.c +2
-1@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO4 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_TSO6 ) ||virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_ECN ) || -virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) +virtio_has_feature (vdev,VIRTIO_NET_F_GUEST_UFO )) { vi->big_packets = true; + vi->gso_is_supported = true;Quoted 4 lines
Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.It is fine as long as
1) we don't implement ethtool API for changing guest offloads
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Not sure if I missed something, but it looks the current
virtnet_set_features()already supports toggling on/off GRO HW through commita02e8964eaf9271a8a5fcc0c55bd13f933bafc56(formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads()or the caller doesn't unsetbig_packet..."."we" here is the device, not the driver.
What is the ethtool API from device level?
VIRTIO_NET_F_CTRL_GUEST_OFFLOADSfor sure, right?
Yes.
Quoted 5 lines
It's implemented in software backend as far as I know. I see no reason *technically* this is infeasible, regardless what you name it, being a bug or TODO.
-Siwei
It's feasible but it's more work. Whether we bother depends in particular on whether anyone cares.
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 2 lines
2) big mode XDP is not enabled
Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.
Regards,
-SiweiQuoted 19 lines
So that code works only for XDP but we forbid big packets in the case of XDP right now.
Thanks
Quoted 14 lines
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Thanks,
-Siwei
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))
vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 21 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 6 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 8 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.
Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".
Quoted 8 lines
The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
It'd be nice to document this requirement or statement to the spec for clarity purpose. Otherwise various device implementations are hard to follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.
-Siwei
Quoted 9 lines
Quoted 4 lines
So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 26 lines · Si-Wei Liu
Quoted 23 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 6 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 8 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.
Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".The "mtu" here is the device config field, which is
Quoted 8 lines
Quoted 6 lines
/* Default maximum transmit unit advice */there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
It'd be nice to document this requirement or statement to the spec for clarity purpose.
It's not a requirement, more of a bug. But it's been like this for years.
Quoted 6 lines
Otherwise various device implementations are hard to
follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.-Siwei
Weird where do you see this in code? I see
sock_len = vhost_net_rx_peek_head_len(net, sock->sk,
&busyloop_intr);
if (!sock_len)
break;
sock_len += sock_hlen;
vhost_len = sock_len + vhost_hlen;
headcount = get_rx_bufs(vq, vq->heads + nvq->done_idx,
vhost_len, &in, vq_log, &log,
likely(mergeable) ? UIO_MAXIOV : 1);
/* On error, stop handling until the next kick. */
if (unlikely(headcount < 0))
goto out;so it does not drop a packet, it just stops processing the queue.
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 45 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packetsQuoted 7 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 3 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500
bytes.
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */Quoted 6 lines
It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.
Quoted 7 lines
yes
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new
value on rx buffer posting.
no, not without a feature bit promising device won't get wedged.
I mean to say as_it_stands today, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
If user space insists on small buffers, so be it.
If previously things worked, the "so be it" is a regression and blaming users won't help us.
I am not suggesting above.
This was Si-Wei's suggestion that somehow driver wants to post smaller buffers than the mtu because user knows what peer is doing. So may be driver can be extended to give more weight on user config.
Quoted 5 lines
Quoted 1 line
It only works when user exactly know what user is doing in the whole
network.
If you want to claim this you need a new feature bit.
Why is a new bit needed to tell device?
User is doing something its own config mismatching the buffers and mtu. A solid use case hasn't emerged for this yet.
If user wants to modify the mtu, we should just make virtio_net_config.mtu as RW field using new feature bit. Is that what you mean?
If so, yes, it makes things very neat where driver and device are aligned to each other, the way they are today. Only limitation is that its one-way. = device tells to driver.
Quoted 6 lines
Quoted 1 line
When user prefers to override the device RO field, device is in the dark and
things work on best effort basis.
Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.
Not sure, why things get stuck for ever. Maybe you have example to explain. I am mostly missing something.
Quoted 5 lines
Quoted 1 line
This must be a reasonably advance user who has good knowledge of its
network topology etc.
Quoted 3 lines
For such case, may be yes, driver should be further optimized.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 27 lines · Michael S. Tsirkin
Quoted 26 lines · Si-Wei Liu
Quoted 24 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 6 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 8 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.
Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
It'd be nice to document this requirement or statement to the spec for clarity purpose.
It's not a requirement, more of a bug. But it's been like this for years.
Well, I'm not sure how it may wedge the device if not capable of posting to smaller buffers, is there other option than drop? Truncate to what the buffer size may fit and deliver up? Seems even worse than drop...
Quoted 23 lines
Quoted 7 lines
Otherwise various device implementations are hard to
follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.-Siwei
Weird where do you see this in code? I see
sock_len = vhost_net_rx_peek_head_len(net, sock->sk, &busyloop_intr); if (!sock_len) break; sock_len += sock_hlen; vhost_len = sock_len + vhost_hlen; headcount = get_rx_bufs(vq, vq->heads + nvq->done_idx, vhost_len, &in, vq_log, &log, likely(mergeable) ? UIO_MAXIOV : 1); /* On error, stop handling until the next kick. */ if (unlikely(headcount < 0)) goto out;
Quoted 1 line
so it does not drop a packet, it just stops processing the queue.
Here
/* On overrun, truncate and discard */
if (unlikely(headcount > UIO_MAXIOV)) { iov_iter_init(&msg.msg_iter, READ, vq->iov, 1, 1); err = sock->ops->recvmsg(sock, &msg,
1, MSG_DONTWAIT | MSG_TRUNC); pr_debug("Discarded rx packet: len %zd\n", sock_len); continue;
}
recvmsg(, , 1, ) is essentially to drop the oversized packet.
In get_rx_bufs(), overrun detection will return something larger than UIO_MAXIOV as indicator:
static int get_rx_bufs()
{
:
;
/* Detect overrun */
if (unlikely(datalen > 0)) {
r = UIO_MAXIOV + 1;
goto err;
}
:
:
-Siwei
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 47 lines · Parav Pandit
Quoted 47 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packetsQuoted 7 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 3 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500
bytes.
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */Quoted 6 lines
It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.yes
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new
value on rx buffer posting.
no, not without a feature bit promising device won't get wedged.
I mean to say
as_it_standstoday, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).
They just are because drivers did not do this.
Quoted 26 lines
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
If user space insists on small buffers, so be it.
If previously things worked, the "so be it" is a regression and blaming users won't help us.
I am not suggesting above.
This was Si-Wei's suggestion that somehow driver wants to post smaller buffers than the mtu because user knows what peer is doing. So may be driver can be extended to give more weight on user config.Quoted 5 lines
Quoted 1 line
It only works when user exactly know what user is doing in the whole
network.
If you want to claim this you need a new feature bit.
Why is a new bit needed to tell device?
User is doing something its own config mismatching the buffers and mtu. A solid use case hasn't emerged for this yet.If user wants to modify the mtu, we should just make virtio_net_config.mtu as RW field using new feature bit. Is that what you mean?
If so, yes, it makes things very neat where driver and device are aligned to each other, the way they are today. Only limitation is that its one-way. = device tells to driver.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
When user prefers to override the device RO field, device is in the dark and
things work on best effort basis.
Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.
Not sure, why things get stuck for ever. Maybe you have example to explain. I am mostly missing something.
I sent an explanation a bit earlier. It's more or less a bug.
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
This must be a reasonably advance user who has good knowledge of its
network topology etc.
Quoted 3 lines
For such case, may be yes, driver should be further optimized.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 98 lines · Si-Wei Liu
Quoted 27 lines · Michael S. Tsirkin
Quoted 26 lines · Si-Wei Liu
Quoted 24 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 6 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 8 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.
Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
It'd be nice to document this requirement or statement to the spec for clarity purpose.
It's not a requirement, more of a bug. But it's been like this for years.
Well, I'm not sure how it may wedge the device if not capable of posting to smaller buffers, is there other option than drop? Truncate to what the buffer size may fit and deliver up? Seems even worse than drop...
Quoted 23 lines
Quoted 7 lines
Otherwise various device implementations are hard to
follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.-Siwei
Weird where do you see this in code? I see
sock_len = vhost_net_rx_peek_head_len(net, sock->sk, &busyloop_intr); if (!sock_len) break; sock_len += sock_hlen; vhost_len = sock_len + vhost_hlen; headcount = get_rx_bufs(vq, vq->heads + nvq->done_idx, vhost_len, &in, vq_log, &log, likely(mergeable) ? UIO_MAXIOV : 1); /* On error, stop handling until the next kick. */ if (unlikely(headcount < 0)) goto out;Quoted 1 line
so it does not drop a packet, it just stops processing the queue.
Here
/* On overrun, truncate and discard */
if (unlikely(headcount > UIO_MAXIOV)) {iov_iter_init(&msg.msg_iter, READ, vq->iov, 1, 1); err =sock->ops->recvmsg(sock, &msg, 1, MSG_DONTWAIT | MSG_TRUNC);
pr_debug("Discarded rx packet: len %zd\n", sock_len);
continue;
}
recvmsg(, , 1, )is essentially to drop the oversized packet.In
get_rx_bufs(), overrun detection will return something larger thanUIO_MAXIOVas indicator:static int
get_rx_bufs()
{
:
;
/* Detect overrun */
if (unlikely(datalen > 0)) {
r =UIO_MAXIOV+ 1;
goto err;
}
:
:
Quoted 1 line
-Siwei
Hmm you are right. I'll check but it seems I have misread the code. Sorry about wasting your time on this.
So maybe the approach is ok then.
It's late, I'll recheck tomorrow.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 8 lines · Parav Pandit
Quoted 6 lines
Quoted 1 line
When user prefers to override the device RO field, device is in the dark and
things work on best effort basis.
Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.
Not sure, why things get stuck for ever. Maybe you have example to explain. I am mostly missing something.
I'm no longer sure I'm right. Will recheck tomorrow, it's late here.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 43 lines · Parav Pandit
Quoted 43 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packetsQuoted 11 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 2 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500
bytes.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
The "mtu" here is the device config field, which is
Quoted 8 lines
Quoted 2 lines
/* Default maximum transmit unit advice */It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.yes
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new
value on rx buffer posting.
no, not without a feature bit promising device won't get wedged.
I mean to say
as_it_standstoday, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).
I am also confused how the device can be wedged in this case.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
If user space insists on small buffers, so be it.
If previously things worked, the "so be it" is a regression and blaming users won't help us.
I am not suggesting above.
This was Si-Wei's suggestion that somehow driver wants to post smaller buffers than the mtu because user knows what peer is doing. So may be driver can be extended to give more weight on user config.
It's not me, it's from our customers with real use cases. Some of which have very dedicate network setup, and it's not at odd they know virtio internals quite well. At one point they even customized the driver to disable mergeable buffer ahead of us offering them the opt-out from device level. And their appliance indeed has assumption of 1460 mtu everywhere.
-Siwei
Quoted 15 lines
Quoted 6 lines
Quoted 2 lines
It only works when user exactly know what user is doing in the whole
network.
If you want to claim this you need a new feature bit.
Why is a new bit needed to tell device?
User is doing something its own config mismatching the buffers and mtu. A solid use case hasn't emerged for this yet.If user wants to modify the mtu, we should just make virtio_net_config.mtu as RW field using new feature bit. Is that what you mean?
If so, yes, it makes things very neat where driver and device are aligned to each other, the way they are today. Only limitation is that its one-way. = device tells to driver.
Quoted 13 lines
Quoted 6 lines
Quoted 1 line
When user prefers to override the device RO field, device is in the dark and
things work on best effort basis.
Dropping packets is best effort. Getting stuck forever isn't, that's a quality of implementation issue.
Not sure, why things get stuck for ever. Maybe you have example to explain. I am mostly missing something.
Quoted 4 lines
Quoted 1 line
This must be a reasonably advance user who has good knowledge of its
network topology etc.
Quoted 2 lines
For such case, may be yes, driver should be further optimized.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 162 lines · Si-Wei Liu
Quoted 159 lines · Jason Wang
Quoted 114 lines · Gavin Li
On 8/9/2022 7:56 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/8/2022 12:31 AM, Gavin Li wrote:
On 8/6/2022 6:11 AM, Si-Wei Liu wrote:
External email: Use caution opening links or attachments
On 8/1/2022 9:45 PM, Gavin Li wrote:
Currently
add_recvbuf_big()allocatesMAX_SKB_FRAGSsegments for big packets even when GUEST_* offloads are not present on the device. However, if GSO is not supported,GUEST GSO (virtio term), or GRO HW (netdev core term) it should have been be called.
ACK
it would be sufficient to allocate
segments to cover just up the MTU size and no further. Allocating the maximum amount of segments results in a large waste of buffer space in the queue, which limits the number of packets that can be buffered and can result in reduced performance.Therefore, if GSO is not supported,
Ditto.
ACK
use the MTU to calculate the
optimal amount of segments required.Below is the iperf TCP test results over a Mellanox NIC, using vDPA for 1 VQ, queue size 1024, before and after the change, with the iperf server running over the virtio-net interface.
MTU(Bytes)/Bandwidth (Gbit/s)
Before After
1500 22.5 22.4
9000 12.8 25.9Signed-off-by Gavin Li <[email protected]>Reviewed-by Gavi Teitz <[email protected]>Reviewed-by Parav Pandit <[email protected]>---
drivers/net/virtio_net.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)drivers/net/virtio_net.c +3
-0@@ -222,6 +222,9 @@ struct virtnet_info { /* I like... big packets and I cannot lie! */ bool big_packets; + /* Indicates GSO support */ + bool gso_is_supported; + /* Host will merge rx buffers for big packets (shake it! shakeit!) */
boolmergeable_rx_bufs;drivers/net/virtio_net.c +0
-0@@ -1312,14 +1315,21 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,static int add_recvbuf_big(struct
virtnet_info*vi, structreceive_queue*rq,
gfp_tgfp)
{
+ unsigned intsg_num=MAX_SKB_FRAGS;
struct page *first, *list = NULL;
char *p;
int i, err, offset;-
sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+ if (!vi->gso_is_supported) {
+ unsigned int mtu =vi->dev->mtu;
+
+sg_num= (mtu %PAGE_SIZE) ? mtu /PAGE_SIZE+ 1 : mtu /PAGE_SIZE;
DIV_ROUND_UP()can be used?ACK
Since this branch slightly adds up cost to the datapath, I wonder if this
sg_numcan be saved and set only once (generally invirtnet_probetime) in structvirtnet_info?Not sure how to do it and align it with align with new mtu during .ndo_change_mtu()---as you mentioned in the following mail. Any idea?
ndo_change_mtumight be in vendor specific code and unmanageable. In my case, the mtu can only be changed in the xml of the guest vm.Nope, for e.g. "ip link dev eth0 set mtu 1500" can be done from guest on a virtio-net device with 9000 MTU (as defined in guest xml). Basically guest user can set MTU to any valid value lower than the original
HOST_MTU. In the vendor defined .ndo_change_mtu() op,dev_validate_mtu()should have validated the MTU value before coming down to it. And I suspect you might want to dovirtnet_close()andvirtnet_open()before/after changing the buffer size on the fly (thenetif_running()case), implementing .ndo_change_mtu() will be needed anyway.Quoted 1 line
a guest VM driver changing mtu to smaller one is valid use case. However, current optimization suggested in the patch doesn't degrade any performance. Performing
close()andopen()sequence is good idea, that I would like to take up next after this patch as its going to be more than one patch to achieve it.Right, it could be done on top.
But another note is that, it would still be better to support GUEST GSO feature:
1) can work for the case for path MTU
2) (migration)compatibility with software backendsQuoted 36 lines
+ }
+
+sg_init_table(rq->sg, sg_num + 2);/* page in rq->sg[MAX_SKB_FRAGS + 1] is list tail */
Comment doesn't match code.
ACK
- for (i =
MAX_SKB_FRAGS+ 1; i > 1; --i) {
+ for (i =sg_num+ 1; i > 1; --i) {
first =get_a_page(rq, gfp);
if (!first) {
if (list)drivers/net/virtio_net.c +1
-1@@ -1350,7 +1360,7 @@ static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq, /* chain first in list head */ first->private = (unsigned long)list; - err = virtqueue_add_inbuf(rq->vq, rq->sg, MAX_SKB_FRAGS + 2, + err = virtqueue_add_inbuf(rq->vq, rq->sg, sg_num + 2, first, gfp); if (err < 0)
give_pages(rq, first);drivers/net/virtio_net.c +0
-0@@ -3571,8 +3581,10 @@ static int virtnet_probe(struct virtio_device *vdev)if (
virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)||virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_ECN)|| -virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) +virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UFO)) {vi->big_packets= true;
+vi->gso_is_supported= true;
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Please do the same for
virtnet_clear_guest_offloads(), and correspondinglyvirtnet_restore_guest_offloads()as well. Not sure whyvirtnet_clear_guest_offloads()or the caller doesn't unsetbig_packeton successful return, seems like a bug to me.It is fine as long as
1) we don't implement ethtool API for changing guest offloads
Not sure if I missed something, but it looks the current
virtnet_set_features()already supports toggling on/off GRO HW through commita02e8964eaf9271a8a5fcc0c55bd13f933bafc56(formerly misnamed as LRO). Sorry, I realized I had a typo in email: "virtnet_set_guest_offloads()or the caller doesn't unsetbig_packet...".
Yes, I miss that.
Quoted 3 lines
Quoted 2 lines
2) big mode XDP is not enabled
Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.
AFAIK, this requires more changes since mergeable path allocates PAGE_SIZE while small path allocates ~1500. This is something that needs to be fixed.
Thanks
Quoted 23 lines
Regards,
-SiweiQuoted 20 lines
So that code works only for XDP but we forbid big packets in the case of XDP right now.
Thanks
Quoted 14 lines
ACK. The two calls
virtnet_set_guest_offloadsandvirtnet_set_guest_offloadsis also called byvirtnet_set_features. Do you think if I can do this invirtnet_set_guest_offloads?I think that it should be fine, though you may want to deal with the XDP path not to regress it.
-Siwei
Thanks,
-Siwei
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
+ }
if (
virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF))vi->mergeable_rx_bufs= true;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 75 lines · Si-Wei Liu
Quoted 27 lines · Michael S. Tsirkin
Quoted 26 lines · Si-Wei Liu
Quoted 24 lines · Michael S. Tsirkin
Quoted 11 lines · Parav Pandit
Quoted 9 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 3:09 PMQuoted 6 lines
Quoted 5 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, August 9, 2022 2:39 PM Currently it is not. Not a single patch nor this patch, but the context for the eventual goal is to allow XDP on a MTU=9000 link when guest users intentionally lower down MTU to 1500.Which application benefit by having asymmetry by lowering mtu to 1500
to send packets but want to receive 9K packets?
Below details doesn’t answer the question of asymmetry. :)
Quoted 8 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of 1500 bytes.
Device doesn't know about it because mtu in config space is RO field.
Device keep dropping 9K packets because buffers posted are 1500 bytes.
This is because device follows the spec " The device MUST NOT pass received packets that exceed mtu".The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
there is no guarantee device will not get a bigger packet. And there is no guarantee such a packet will be dropped as opposed to wedging the device if userspace insists on adding smaller buffers.
It'd be nice to document this requirement or statement to the spec for clarity purpose.
It's not a requirement, more of a bug. But it's been like this for years.
Well, I'm not sure how it may wedge the device if not capable of posting to smaller buffers, is there other option than drop? Truncate to what the buffer size may fit and deliver up? Seems even worse than drop...
Quoted 23 lines
Quoted 7 lines
Otherwise various device implementations are hard to
follow. The capture is that vhost-net drops bigger packets while the driver only supplied smaller buffers. This is the status quo, and users seemingly have relied on this behavior for some while.-Siwei
Weird where do you see this in code? I see
sock_len=vhost_net_rx_peek_head_len(net, sock->sk, &busyloop_intr);
if (!sock_len)
break;
sock_len+=sock_hlen;
vhost_len=sock_len+vhost_hlen;
headcount =get_rx_bufs(vq, vq->heads + nvq->done_idx, vhost_len, &in, vq_log, &log, likely(mergeable) ? UIO_MAXIOV : 1); /* On error, stop handling until the next kick. */ if (unlikely(headcount < 0))
goto out;Quoted 1 line
so it does not drop a packet, it just stops processing the queue.
Here
/* On overrun, truncate and discard */
if (unlikely(headcount > UIO_MAXIOV)) {iov_iter_init(&msg.msg_iter, READ, vq->iov, 1, 1); err = sock->ops->recvmsg(sock, &msg,
1,MSG_DONTWAIT|MSG_TRUNC);pr_debug("Discarded rx packet: len %zd\n", sock_len); continue;
}
Quoted 1 line
recvmsg(, , 1, )is essentially to drop the oversized packet.
It's not necessarily the oversized packet but the packet that has too many sgs.
This issues has been discussed in the past, (for example we disable large rx queue size for vhost-net in Qemu). Where it could be solved by doing piece-wise copy
Thanks
Quoted 17 lines
In
get_rx_bufs(), overrun detection will return something larger thanUIO_MAXIOVas indicator:static int
get_rx_bufs()
{
:
;
/* Detect overrun */
if (unlikely(datalen > 0)) {
r =UIO_MAXIOV+ 1;
goto err;
}
:
:
Quoted 12 lines
-Siwei
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
So, I am lost what virtio net device user application is trying to achieve by sending smaller packets and dropping all receive packets. (it doesn’t have any relation to mergeable or otherwise).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 47 lines · Si-Wei Liu
Quoted 44 lines · Parav Pandit
Quoted 44 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packetsQuoted 11 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 2 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500
bytes.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
The "mtu" here is the device config field, which is
Quoted 8 lines
Quoted 2 lines
/* Default maximum transmit unit advice */It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.yes
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
value on rx buffer posting.
no, not without a feature bit promising device won't get wedged.
I mean to say
as_it_standstoday, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).I am also confused how the device can be wedged in this case.
Yea sorry. I misunderstood the code. It can't be.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 50 lines · Michael S. Tsirkin
Quoted 49 lines · Si-Wei Liu
Quoted 46 lines · Parav Pandit
Quoted 46 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packetsQuoted 11 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 2 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500
bytes.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
The "mtu" here is the device config field, which is
Quoted 8 lines
Quoted 2 lines
/* Default maximum transmit unit advice */It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.yes
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new
value on rx buffer posting.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
no, not without a feature bit promising device won't get wedged.
I mean to say
as_it_standstoday, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).I am also confused how the device can be wedged in this case.
Yea sorry. I misunderstood the code. It can't be.
Here's a problem as I see it. Let's say we reduce mtu. Small buffers are added. Now we increase mtu.
Device will drop all packets until small buffers are consumed.
Should we make this depend on the vq reset ability maybe?
Quoted 2 lines
Signature · 1 line
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 59 lines · Michael S. Tsirkin
Quoted 57 lines · Michael S. Tsirkin
Quoted 55 lines · Si-Wei Liu
Quoted 43 lines · Parav Pandit
Quoted 43 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packetsQuoted 11 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 2 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500
bytes.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
The "mtu" here is the device config field, which is
Quoted 8 lines
Quoted 2 lines
/* Default maximum transmit unit advice */It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.yes
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new
value on rx buffer posting.
no, not without a feature bit promising device won't get wedged.
I mean to say
as_it_standstoday, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).I am also confused how the device can be wedged in this case.
Quoted 7 lines
Quoted 1 line
Yea sorry. I misunderstood the code. It can't be.
Here's a problem as I see it. Let's say we reduce mtu. Small buffers are added. Now we increase mtu.
Device will drop all packets until small buffers are consumed.Should we make this depend on the vq reset ability maybe?
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Thanks
Quoted 4 lines
Quoted 3 lines
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 66 lines · Jason Wang
Quoted 65 lines · Michael S. Tsirkin
Quoted 50 lines · Michael S. Tsirkin
Quoted 49 lines · Si-Wei Liu
Quoted 46 lines · Parav Pandit
Quoted 46 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packetsQuoted 11 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 2 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500
bytes.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
The "mtu" here is the device config field, which is
Quoted 8 lines
Quoted 2 lines
/* Default maximum transmit unit advice */It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.yes
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new
value on rx buffer posting.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
no, not without a feature bit promising device won't get wedged.
I mean to say
as_it_standstoday, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).I am also confused how the device can be wedged in this case.
Yea sorry. I misunderstood the code. It can't be.
Here's a problem as I see it. Let's say we reduce mtu. Small buffers are added. Now we increase mtu.
Device will drop all packets until small buffers are consumed.
Quoted 6 lines
Quoted 1 line
Should we make this depend on the vq reset ability maybe?
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Thanks
Device reset is really annoying in that it loses all the state:
rx filters etc etc.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 6 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AMQuoted 3 lines
Quoted 2 lines
Should we make this depend on the vq reset ability maybe?
Quoted 7 lines
Quoted 4 lines
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Thanks
Device reset is really annoying in that it loses all the state:
rx filters etc etc.
The elegant solution is let driver tell the new mtu to the device.
One way to do so is by using existing ctrl vq.
If merged buffer is done, and new mtu is > minimum posting size, no need to undergo vq reset. If merged buffer is not done, and buffer posted are smaller than new mtu, undergo vq reset optionally.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 9 lines · Parav Pandit
Quoted 9 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AMQuoted 6 lines
Quoted 2 lines
Should we make this depend on the vq reset ability maybe?
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Thanks
Device reset is really annoying in that it loses all the state:
rx filters etc etc.The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.
That will need a new feature bit.
Quoted 2 lines
If merged buffer is done, and new mtu is > minimum posting size, no need to undergo vq reset. If merged buffer is not done, and buffer posted are smaller than new mtu, undergo vq reset optionally.
This can be done with or without sending mtu to device.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 16 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PMQuoted 11 lines · Parav Pandit
Quoted 11 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AMQuoted 7 lines
Quoted 2 lines
Should we make this depend on the vq reset ability maybe?
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Thanks
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Device reset is really annoying in that it loses all the state:
rx filters etc etc.The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.
That will need a new feature bit.
Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?
Quoted 6 lines
Quoted 1 line
If merged buffer is done, and new mtu is > minimum posting size, no need
to undergo vq reset.
Quoted 1 line
If merged buffer is not done, and buffer posted are smaller than new mtu,
undergo vq reset optionally.
This can be done with or without sending mtu to device.
Yes, telling mtu to device helps device to optimize and adhere to the spec line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 19 lines · Parav Pandit
Quoted 19 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PMQuoted 14 lines · Parav Pandit
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AMQuoted 7 lines
Quoted 2 lines
Should we make this depend on the vq reset ability maybe?
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Thanks
Device reset is really annoying in that it loses all the state:
rx filters etc etc.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.
That will need a new feature bit.
Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?
We did feature bits for this in the past.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
If merged buffer is done, and new mtu is > minimum posting size, no need
to undergo vq reset.
Quoted 1 line
If merged buffer is not done, and buffer posted are smaller than new mtu,
undergo vq reset optionally.
This can be done with or without sending mtu to device.
Yes, telling mtu to device helps device to optimize and adhere to the spec line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.
Again, that line refers to \field{mtu} which is the max mtu supported, irrespective to anything driver does.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 32 lines · Michael S. Tsirkin
Quoted 19 lines · Parav Pandit
Quoted 19 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PMQuoted 14 lines · Parav Pandit
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AMQuoted 7 lines
Quoted 2 lines
Should we make this depend on the vq reset ability maybe?
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Thanks
Device reset is really annoying in that it loses all the state:
rx filters etc etc.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.
That will need a new feature bit.
Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?
We did feature bits for this in the past.
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
If merged buffer is done, and new mtu is > minimum posting size, no need
to undergo vq reset.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
If merged buffer is not done, and buffer posted are smaller than new mtu,
undergo vq reset optionally.
This can be done with or without sending mtu to device.
Yes, telling mtu to device helps device to optimize and adhere to the spec line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.
Again, that line refers to \field{mtu} which is the max mtu supported, irrespective to anything driver does.
BTW with any such ctrl vq interface we need to think how cases such as increasing and decreasing MTU work. The normal behaviour for linux drivers is to limit this to when the link is down. Which reminds me, we do not have a command to bring link down, either.
Quoted 2 lines
Signature · 1 line
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 27 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:59 PMQuoted 22 lines · Parav Pandit
Quoted 22 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PMQuoted 16 lines · Parav Pandit
Quoted 6 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AMQuoted 3 lines
Quoted 2 lines
Should we make this depend on the vq reset ability maybe?
Quoted 7 lines
Quoted 4 lines
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Thanks
Device reset is really annoying in that it loses all the state:
rx filters etc etc.The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
That will need a new feature bit.
Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?
We did feature bits for this in the past.
Ok. Will try to draft the update for future.
Gavin should repost the patch to address comments unrelated to this future bit anyway. Right?
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
If merged buffer is done, and new mtu is > minimum posting size, no need
to undergo vq reset.
Quoted 2 lines
If merged buffer is not done, and buffer posted are smaller than new mtu,
undergo vq reset optionally.
This can be done with or without sending mtu to device.
Yes, telling mtu to device helps device to optimize and adhere to the spec
Quoted 8 lines
line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.
Again, that line refers to \field{mtu} which is the max mtu supported, irrespective to anything driver does.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Michael S. Tsirkin
maintainerQuoted 32 lines · Parav Pandit
Quoted 32 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:59 PMQuoted 26 lines · Parav Pandit
Quoted 16 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 12:05 PMQuoted 11 lines · Parav Pandit
Quoted 11 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, August 10, 2022 5:03 AMQuoted 7 lines
Quoted 2 lines
Should we make this depend on the vq reset ability maybe?
The advantage of this is to keep TX working. Or we can use device reset as a fallback if there's no vq reset.
Thanks
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Device reset is really annoying in that it loses all the state:
rx filters etc etc.The elegant solution is let driver tell the new mtu to the device. One way to do so is by using existing ctrl vq.
That will need a new feature bit.
Yes. ctrl vq can tell what all configuration does it allow. :) Or you prefer feature bit?
Quoted 6 lines
Quoted 2 lines
We did feature bits for this in the past.
Ok. Will try to draft the update for future.
Gavin should repost the patch to address comments unrelated to this future bit anyway. Right?
Right.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
If merged buffer is done, and new mtu is > minimum posting size, no need
to undergo vq reset.
Quoted 2 lines
If merged buffer is not done, and buffer posted are smaller than new mtu,
undergo vq reset optionally.
This can be done with or without sending mtu to device.
Yes, telling mtu to device helps device to optimize and adhere to the spec
Quoted 8 lines
Quoted 8 lines
line " The device MUST NOT pass received packets that exceed mtu" in section 5.1.4.1.
Again, that line refers to \field{mtu} which is the max mtu supported, irrespective to anything driver does.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 55 lines · Michael S. Tsirkin
Quoted 54 lines · Michael S. Tsirkin
Quoted 53 lines · Si-Wei Liu
Quoted 51 lines · Parav Pandit
Quoted 39 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 6:26 PM
To: Parav Pandit <[email protected]>
Cc: Si-Wei Liu <[email protected]>; Jason Wang <[email protected]>; Gavin Li <[email protected]>; Hemminger, Stephen <[email protected]>; davem
<[email protected]>; virtualization <[email protected] foundation.org>; Virtio-Dev <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gavi Teitz <[email protected]>
Subject: Re: [virtio-dev] [PATCH] virtio-net: use mtu size as buffer length for big packetsQuoted 11 lines · Parav Pandit
Quoted 2 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, August 9, 2022 5:38 PM[..]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I think virtio-net driver doesn't differentiate MTU and MRU, in which case the receive buffer will be reduced to fit the 1500B payload size when mtu is lowered down to 1500 from 9000.
How? Driver reduced the mXu to 1500, say it is improved to post buffers of
1500 bytes.
Quoted 2 lines
Device doesn't know about it because mtu in config space is RO field. Device keep dropping 9K packets because buffers posted are 1500
bytes.
Quoted 12 lines
Quoted 8 lines
Quoted 1 line
This is because device follows the spec " The device MUST NOT pass
received packets that exceed mtu".
The "mtu" here is the device config field, which is
/* Default maximum transmit unit advice */It is the field from struct virtio_net_config.mtu. right? This is RO field for driver.
Quoted 1 line
there is no guarantee device will not get a bigger packet.
Quoted 9 lines
Quoted 2 lines
Right. That is what I also hinted.
Hence, allocating buffers worth upto mtu is safer.yes
Quoted 1 line
When user overrides it, driver can be further optimized to honor such new
value on rx buffer posting.
no, not without a feature bit promising device won't get wedged.
I mean to say
as_it_standstoday, driver can decide to post smaller buffers with larger mtu. Why device should be affected with it?
( I am not proposing such weird configuration but asking for sake of correctness).
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
I am also confused how the device can be wedged in this case.
Yea sorry. I misunderstood the code. It can't be.
Here's a problem as I see it. Let's say we reduce mtu. Small buffers are added. Now we increase mtu.
Device will drop all packets until small buffers are consumed.Should we make this depend on the vq reset ability maybe?
To be honest I am not sure if worth it, very few user changes MTU on the fly with traffic ongoing, for the most cases I've seen users just change it only once in deployment time. Even if they change it on the fly they may need to be aware of the consequence and implication of loss of packets. In real devices, mtu change could end up with link status change and in that case there's usually no guarantee arrived packet will be kept during the window.
While I could understand this would slightly introduce regression on functionality, as the worst case for packet loss if device dropping packets, it would be all smaller buffers of the full queue size. For correctness and elegance I don't mind introducing specific feature that changes MTU, or even relying on vq_reset is fine. device_reset would be too overwhelmed for this special use case IMHO.
-Siwei
Quoted 3 lines
Quoted 3 lines
Signature · 2 lines
-- MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]