virtio-net: Set needed_headroom for virtio-net when VIRTIO_F_ANY_LAYOUT is true

PATCH Applied Zhangjie (HZ) 5 participants 5 messages Apr 29, 2014 – Apr 30, 2014

Subsystems (3)

Networking Network drivers virtio net driver

lore

Share & tools

lore

Zhangjie (HZ)

author

2014-04-29 10:44 UTC #0 lore raw

PATCH 1/1 virtio-net: Set needed_headroom for virtio-net when VIRTIO_F_ANY_LAYOUT is true

This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287 ("virtio-net: put virtio-net header inline with data"). TCP packages have enough room to put virtio-net header in, but UDP packages do not. By setting dev->needed_headroom for virtio-net device, UDP packages could have enough room.

For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);". The Macro is defined as follows:

#define LL_RESERVED_SPACE(dev) \
     ((((dev)->hard_header_len+(dev)->needed_headroom)\
     &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)

By default, for UDP packages, after skb is allocated, only 16 bytes reserved. And 2 bytes remained after mac header is set. That is not enough to put virtio-net header in. If we set dev->needed_headroom to 12 or 10 (according to mergeable_rx_bufs is on or off ), more room can be reserved. Then there is enough room for UDP packages to put the header in.

test result list as below:
guest and host: suse11sp3, netperf, intel 2.4GHz

+-------+---------+---------+---------+---------+
|       |   old             |   new             |
+-------+---------+---------+---------+---------+
| UDP   |  Gbit/s | pps     |  Gbit/s | pps     |
| 64    |  0.57   | 692232  |  0.61   | 742420  |
| 256   |  1.60   | 686860  |  1.71   | 733331  |
| 512   |  2.92   | 674576  |  3.07   | 710446  |
| 1024  |  4.99   | 598977  |  5.17   | 620821  |
| 1460  |  5.68   | 483757  |  7.16   | 610519  |
| 4096  |  6.98   | 637468  |  7.21   | 658471  |
+-------+---------+---------+---------+---------+

Signed-off-by Zhang Jie <[email protected]>

---
 drivers/net/virtio_net.c | 7 +++++++
 1 file changed, 7 insertions(+)

drivers/net/virtio_net.c +7 -0

index 8a852b5..b852bb3 100644
@@ -1724,6 +1724,13 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
 		vi->has_cvq = true;
 
+	if (vi->any_header_sg) {
+		if (vi->mergeable_rx_bufs)
+			dev->needed_headroom = sizeof(struct virtio_net_hdr_mrg_rxbuf);
+		else
+			dev->needed_headroom = sizeof(struct virtio_net_hdr);
+	}
+
 	/* Use single tx/rx queue pair as default */
 	vi->curr_queue_pairs = 1;
 	vi->max_queue_pairs = max_queue_pairs;

Signature · 1 line

1.9.0.msysgit.0

Rusty Russell

Acked-by on v1 1/1

2014-04-30 07:11 UTC #1 lore raw

Quoted 9 lines · Zhangjie (HZ)

This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287 ("virtio-net: put virtio-net header inline with data"). TCP packages have enough room to put virtio-net header in, but UDP packages do not. By setting dev->needed_headroom for virtio-net device, UDP packages could have enough room.
For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);". The Macro is defined as follows:

Quoted 8 lines

#define LL_RESERVED_SPACE(dev) \
     ((((dev)->hard_header_len+(dev)->needed_headroom)\
     &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
By default, for UDP packages, after skb is allocated, only 16 bytes reserved. And 2 bytes remained after mac header is set. That is not enough to put virtio-net header in. If we set dev->needed_headroom to 12 or 10 (according to mergeable_rx_bufs is on or off ), more room can be reserved. Then there is enough room for UDP packages to put the header in.

Seems like a no-brainer. Good find!

Acked-by Rusty Russell <[email protected]>

Thanks,
Rusty.

Quoted 47 lines

test result list as below:
guest and host: suse11sp3, netperf, intel 2.4GHz

+-------+---------+---------+---------+---------+
|       |   old             |   new             |
+-------+---------+---------+---------+---------+
| UDP   |  Gbit/s | pps     |  Gbit/s | pps     |
| 64    |  0.57   | 692232  |  0.61   | 742420  |
| 256   |  1.60   | 686860  |  1.71   | 733331  |
| 512   |  2.92   | 674576  |  3.07   | 710446  |
| 1024  |  4.99   | 598977  |  5.17   | 620821  |
| 1460  |  5.68   | 483757  |  7.16   | 610519  |
| 4096  |  6.98   | 637468  |  7.21   | 658471  |
+-------+---------+---------+---------+---------+

Signed-off-by Zhang Jie <[email protected]>

---
 drivers/net/virtio_net.c | 7 +++++++
 1 file changed, 7 insertions(+)

drivers/net/virtio_net.c +7 -0

index 8a852b5..b852bb3 100644
@@ -1724,6 +1724,13 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
 		vi->has_cvq = true;
 
+	if (vi->any_header_sg) {
+		if (vi->mergeable_rx_bufs)
+			dev->needed_headroom = sizeof(struct virtio_net_hdr_mrg_rxbuf);
+		else
+			dev->needed_headroom = sizeof(struct virtio_net_hdr);
+	}
+
 	/* Use single tx/rx queue pair as default */
 	vi->curr_queue_pairs = 1;
 	vi->max_queue_pairs = max_queue_pairs;

Signature · 8 lines

1.9.0.msysgit.0


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Michael S. Tsirkin

maintainer Acked-by on v1 1/1

2014-04-30 09:29 UTC #2 lore raw

In reply to Zhangjie (HZ)

Quoted 32 lines · Zhangjie (HZ)

This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287 ("virtio-net: put virtio-net header inline with data"). TCP packages have enough room to put virtio-net header in, but UDP packages do not. By setting dev->needed_headroom for virtio-net device, UDP packages could have enough room.
For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);". The Macro is defined as follows:
#define LL_RESERVED_SPACE(dev) \
     ((((dev)->hard_header_len+(dev)->needed_headroom)\
     &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
By default, for UDP packages, after skb is allocated, only 16 bytes reserved. And 2 bytes remained after mac header is set. That is not enough to put virtio-net header in. If we set dev->needed_headroom to 12 or 10 (according to mergeable_rx_bufs is on or off ), more room can be reserved. Then there is enough room for UDP packages to put the header in.
test result list as below:
guest and host: suse11sp3, netperf, intel 2.4GHz
+-------+---------+---------+---------+---------+
|       |   old             |   new             |
+-------+---------+---------+---------+---------+
| UDP   |  Gbit/s | pps     |  Gbit/s | pps     |
| 64    |  0.57   | 692232  |  0.61   | 742420  |
| 256   |  1.60   | 686860  |  1.71   | 733331  |
| 512   |  2.92   | 674576  |  3.07   | 710446  |
| 1024  |  4.99   | 598977  |  5.17   | 620821  |
| 1460  |  5.68   | 483757  |  7.16   | 610519  |
| 4096  |  6.98   | 637468  |  7.21   | 658471  |
+-------+---------+---------+---------+---------+

Quoted 1 line

Signed-off-by Zhang Jie <[email protected]>

Thanks for your patience!

Acked-by Michael S. Tsirkin <[email protected]>

Quoted 25 lines

---
 drivers/net/virtio_net.c | 7 +++++++
 1 file changed, 7 insertions(+)

drivers/net/virtio_net.c +7 -0

index 8a852b5..b852bb3 100644
@@ -1724,6 +1724,13 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
 		vi->has_cvq = true;
 
+	if (vi->any_header_sg) {
+		if (vi->mergeable_rx_bufs)
+			dev->needed_headroom = sizeof(struct virtio_net_hdr_mrg_rxbuf);
+		else
+			dev->needed_headroom = sizeof(struct virtio_net_hdr);
+	}
+
 	/* Use single tx/rx queue pair as default */
 	vi->curr_queue_pairs = 1;
 	vi->max_queue_pairs = max_queue_pairs;

Signature · 1 line

1.9.0.msysgit.0

Jason Wang

Acked-by on v1 1/1

2014-04-30 09:38 UTC #3 lore raw

In reply to Zhangjie (HZ)

Quoted 45 lines · Zhangjie (HZ)

This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287 ("virtio-net: put virtio-net header inline with data"). TCP packages have enough room to put virtio-net header in, but UDP packages do not. By setting dev->needed_headroom for virtio-net device, UDP packages could have enough room.
For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);". The Macro is defined as follows:
#define LL_RESERVED_SPACE(dev) \
     ((((dev)->hard_header_len+(dev)->needed_headroom)\
     &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
By default, for UDP packages, after skb is allocated, only 16 bytes reserved. And 2 bytes remained after mac header is set. That is not enough to put virtio-net header in. If we set dev->needed_headroom to 12 or 10 (according to mergeable_rx_bufs is on or off ), more room can be reserved. Then there is enough room for UDP packages to put the header in.
test result list as below:
guest and host: suse11sp3, netperf, intel 2.4GHz
+-------+---------+---------+---------+---------+
|       |   old             |   new             |
+-------+---------+---------+---------+---------+
| UDP   |  Gbit/s | pps     |  Gbit/s | pps     |
| 64    |  0.57   | 692232  |  0.61   | 742420  |
| 256   |  1.60   | 686860  |  1.71   | 733331  |
| 512   |  2.92   | 674576  |  3.07   | 710446  |
| 1024  |  4.99   | 598977  |  5.17   | 620821  |
| 1460  |  5.68   | 483757  |  7.16   | 610519  |
| 4096  |  6.98   | 637468  |  7.21   | 658471  |
+-------+---------+---------+---------+---------+
Signed-off-by Zhang Jie <[email protected]>
---
 drivers/net/virtio_net.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/net/virtio_net.c +0 -0
index 8a852b5..b852bb3 100644
@@ -1724,6 +1724,13 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
 		vi->has_cvq = true;
 

drivers/net/virtio_net.c static int virtnet_probe(struct virtio_device *vdev) · line 1736

Quoted 10 lines

drivers/net/virtio_net.c +7 -0

+	if (vi->any_header_sg) {
+		if (vi->mergeable_rx_bufs)
+			dev->needed_headroom = sizeof(struct virtio_net_hdr_mrg_rxbuf);
+		else
+			dev->needed_headroom = sizeof(struct virtio_net_hdr);
+	}
+
 	/* Use single tx/rx queue pair as default */
 	vi->curr_queue_pairs = 1;
 	vi->max_queue_pairs = max_queue_pairs;

Acked-by Jason Wang <[email protected]>

David Miller

maintainer Applied

2014-04-30 17:32 UTC #4 lore raw

In reply to Zhangjie (HZ)

From: "Zhangjie (HZ)" <[email protected]>
Date: Tue, 29 Apr 2014 18:43:22 +0800

Quoted 32 lines

This is a small supplement for commit e7428e95a06fb516fac1308bd0e176e27c0b9287 ("virtio-net: put virtio-net header inline with data"). TCP packages have enough room to put virtio-net header in, but UDP packages do not. By setting dev->needed_headroom for virtio-net device, UDP packages could have enough room.
For UDP packages, sk_buff is alloced in fun __ip_append_data. The size is "alloclen + hh_len + 15", and "hh_len = LL_RESERVED_SPACE(rt-dst.dev);". The Macro is defined as follows:
#define LL_RESERVED_SPACE(dev) \
     ((((dev)->hard_header_len+(dev)->needed_headroom)\
     &~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
By default, for UDP packages, after skb is allocated, only 16 bytes reserved. And 2 bytes remained after mac header is set. That is not enough to put virtio-net header in. If we set dev->needed_headroom to 12 or 10 (according to mergeable_rx_bufs is on or off ), more room can be reserved. Then there is enough room for UDP packages to put the header in.
test result list as below:
guest and host: suse11sp3, netperf, intel 2.4GHz
+-------+---------+---------+---------+---------+
|       |   old             |   new             |
+-------+---------+---------+---------+---------+
| UDP   |  Gbit/s | pps     |  Gbit/s | pps     |
| 64    |  0.57   | 692232  |  0.61   | 742420  |
| 256   |  1.60   | 686860  |  1.71   | 733331  |
| 512   |  2.92   | 674576  |  3.07   | 710446  |
| 1024  |  4.99   | 598977  |  5.17   | 620821  |
| 1460  |  5.68   | 483757  |  7.16   | 610519  |
| 4096  |  6.98   | 637468  |  7.21   | 658471  |
+-------+---------+---------+---------+---------+

Quoted 1 line

Signed-off-by Zhang Jie <[email protected]>

Applied.