VSOCK: support mergeable rx buffer in vhost-vsock

From the test results, the performance is improved obviously, and guest

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

This series of patches mainly did three things：
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big packet(64K) included guest->Host and Host->Guest. I test performance independently and the result as follows:

Before performance:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

In this case, max send pkt size is still limited to 4K, so Host->Guest performance will worse than before.

After performance increase the max send pkt size to 64K:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s

Quoted 1 line

From the test results, the performance is improved obviously, and guest

memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock, we need to add a qemu patch to support parse feature.

---
v1 -> v2:
* Addressed comments from Jason Wang.
* Add performance test result independently.
* Use Skb_page_frag_refill() which can use high order page and reduce the stress of page allocator.
* Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small size can't fill one full packet, we only 128 vq num now.
* Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx consistency.
* Add virtio_transport ops to get max pkt len, in order to be compatible with old version.
---

Yiwen Jiang (5):
VSOCK: support fill mergeable rx buffer in guest
VSOCK: support fill data to mergeable rx buffer in host
VSOCK: support receive mergeable rx buffer in guest
VSOCK: increase send pkt len in mergeable mode to improve performance
VSOCK: batch sending rx buffer to increase bandwidth

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

Signature · 1 line

1.8.3.1

Quoted 25 lines · jiangyiwen

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

This series of patches mainly did three things：
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big packet(64K) included guest->Host and Host->Guest. I test performance independently and the result as follows:

Before performance:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

Quoted 2 lines

In this case, max send pkt size is still limited to 4K, so Host->Guest performance will worse than before.

It's concerning though, what if application sends small packets? What is the source of the slowdown? Do you know?

Quoted 37 lines

After performance increase the max send pkt size to 64K:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s

Quoted 1 line

From the test results, the performance is improved obviously, and guest

memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock, we need to add a qemu patch to support parse feature.

---
v1 -> v2:
* Addressed comments from Jason Wang.
* Add performance test result independently.
* Use Skb_page_frag_refill() which can use high order page and reduce the stress of page allocator.
* Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small size can't fill one full packet, we only 128 vq num now.
* Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx consistency.
* Add virtio_transport ops to get max pkt len, in order to be compatible with old version.
---

Yiwen Jiang (5):
VSOCK: support fill mergeable rx buffer in guest
VSOCK: support fill data to mergeable rx buffer in host
VSOCK: support receive mergeable rx buffer in guest
VSOCK: increase send pkt len in mergeable mode to improve performance
VSOCK: batch sending rx buffer to increase bandwidth

Quoted 9 lines

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

Signature · 1 line

1.8.3.1

.

Quoted 25 lines · jiangyiwen

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

This series of patches mainly did three things：
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big packet(64K) included guest->Host and Host->Guest. I test performance independently and the result as follows:

Before performance:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

Quoted 2 lines

In this case, max send pkt size is still limited to 4K, so Host->Guest performance will worse than before.

It's concerning though, what if application sends small packets? What is the source of the slowdown? Do you know?

Quoted 44 lines

After performance increase the max send pkt size to 64K:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s

Quoted 1 line

From the test results, the performance is improved obviously, and guest

memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock, we need to add a qemu patch to support parse feature.

---
v1 -> v2:
* Addressed comments from Jason Wang.
* Add performance test result independently.
* Use Skb_page_frag_refill() which can use high order page and reduce the stress of page allocator.
* Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small size can't fill one full packet, we only 128 vq num now.
* Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx consistency.
* Add virtio_transport ops to get max pkt len, in order to be compatible with old version.
---

Yiwen Jiang (5):
VSOCK: support fill mergeable rx buffer in guest
VSOCK: support fill data to mergeable rx buffer in host
VSOCK: support receive mergeable rx buffer in guest
VSOCK: increase send pkt len in mergeable mode to improve performance
VSOCK: batch sending rx buffer to increase bandwidth

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

Quoted 2 lines

Signature · 1 line

1.8.3.1

.

Quoted 25 lines · jiangyiwen

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

This series of patches mainly did three things：
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big packet(64K) included guest->Host and Host->Guest. I test performance independently and the result as follows:

Before performance:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

Quoted 2 lines

In this case, max send pkt size is still limited to 4K, so Host->Guest performance will worse than before.

It's concerning though, what if application sends small packets? What is the source of the slowdown? Do you know?

Quoted 37 lines

After performance increase the max send pkt size to 64K:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s

Quoted 1 line

From the test results, the performance is improved obviously, and guest

memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock, we need to add a qemu patch to support parse feature.

---
v1 -> v2:
* Addressed comments from Jason Wang.
* Add performance test result independently.
* Use Skb_page_frag_refill() which can use high order page and reduce the stress of page allocator.
* Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small size can't fill one full packet, we only 128 vq num now.
* Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx consistency.
* Add virtio_transport ops to get max pkt len, in order to be compatible with old version.
---

Yiwen Jiang (5):
VSOCK: support fill mergeable rx buffer in guest
VSOCK: support fill data to mergeable rx buffer in host
VSOCK: support receive mergeable rx buffer in guest
VSOCK: increase send pkt len in mergeable mode to improve performance
VSOCK: batch sending rx buffer to increase bandwidth

Quoted 9 lines

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

Signature · 1 line

1.8.3.1

.

Quoted 26 lines · Michael S. Tsirkin

Quoted 25 lines · jiangyiwen

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

This series of patches mainly did three things：
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big packet(64K) included guest->Host and Host->Guest. I test performance independently and the result as follows:

Before performance:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

Quoted 6 lines

Quoted 2 lines

In this case, max send pkt size is still limited to 4K, so Host->Guest performance will worse than before.

It's concerning though, what if application sends small packets? What is the source of the slowdown? Do you know?

Hi Michael,

To the two cases, I test the results included small and big packets as follows:

64K packets performance comparison:
Single socket Multiple sockets
Host->Guest(before) 1352.60MB/s 1436.33MB/s

Host->Guest(only use mergeable rx buffer) 1290.08MB/s 1212.67MB/s

4K packets performance comparison:

                                              Single socket    Multiple sockets
Host->Guest(before)                           535.47MB/s       688.67MB/s
Host->Guest(only use mergeable rx buffer)     522.33MB/s       599.00MB/s

3K packets performance comparison:

                                              Single socket    Multiple sockets
Host->Guest(before)                           359.74MB/s       442.00MB/s
Host->Guest(only use mergeable rx buffer)     374.47MB/s       452.33MB/s

We can see an interesting thing, for 64K and 4K packets, using mergeable buffer has a poor performance, for 3K packet, both have the same performance.

I guess in mergeable mode, when host send a 4k packet to guest, we should call vhost_get_vq_desc() twice in host(hdr + 4k data), and in guest we also should call virtqueue_get_buf() twice. So when packet is smaller than (4k - hdr), it can be packed in a single page, so the performance is the same as before.

So in the mergeable mode, the performance may be
worse in ((4k - hdr), 4k] than before.

Thanks,
Yiwen.

Quoted 49 lines

Quoted 37 lines

After performance increase the max send pkt size to 64K:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s

Quoted 1 line

From the test results, the performance is improved obviously, and guest

memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock, we need to add a qemu patch to support parse feature.

---
v1 -> v2:
* Addressed comments from Jason Wang.
* Add performance test result independently.
* Use Skb_page_frag_refill() which can use high order page and reduce the stress of page allocator.
* Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small size can't fill one full packet, we only 128 vq num now.
* Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx consistency.
* Add virtio_transport ops to get max pkt len, in order to be compatible with old version.
---

Yiwen Jiang (5):
VSOCK: support fill mergeable rx buffer in guest
VSOCK: support fill data to mergeable rx buffer in host
VSOCK: support receive mergeable rx buffer in guest
VSOCK: increase send pkt len in mergeable mode to improve performance
VSOCK: batch sending rx buffer to increase bandwidth

Quoted 9 lines

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

Signature · 1 line

1.8.3.1

.

Quoted 26 lines · Michael S. Tsirkin

Quoted 25 lines · jiangyiwen

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

This series of patches mainly did three things：
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big packet(64K) included guest->Host and Host->Guest. I test performance independently and the result as follows:

Before performance:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

Quoted 6 lines

Quoted 2 lines

In this case, max send pkt size is still limited to 4K, so Host->Guest performance will worse than before.

It's concerning though, what if application sends small packets? What is the source of the slowdown? Do you know?

Hi Michael,

To the two cases, I test the results included small and big packets as follows:

64K packets performance comparison:
Single socket Multiple sockets
Host->Guest(before) 1352.60MB/s 1436.33MB/s

Host->Guest(only use mergeable rx buffer) 1290.08MB/s 1212.67MB/s

4K packets performance comparison:

                                              Single socket    Multiple sockets
Host->Guest(before)                           535.47MB/s       688.67MB/s
Host->Guest(only use mergeable rx buffer)     522.33MB/s       599.00MB/s

3K packets performance comparison:

                                              Single socket    Multiple sockets
Host->Guest(before)                           359.74MB/s       442.00MB/s
Host->Guest(only use mergeable rx buffer)     374.47MB/s       452.33MB/s

We can see an interesting thing, for 64K and 4K packets, using mergeable buffer has a poor performance, for 3K packet, both have the same performance.

I guess in mergeable mode, when host send a 4k packet to guest, we should call vhost_get_vq_desc() twice in host(hdr + 4k data), and in guest we also should call virtqueue_get_buf() twice. So when packet is smaller than (4k - hdr), it can be packed in a single page, so the performance is the same as before.

So in the mergeable mode, the performance may be
worse in ((4k - hdr), 4k] than before.

Thanks,
Yiwen.

Quoted 44 lines

Quoted 44 lines

After performance increase the max send pkt size to 64K:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s

Quoted 1 line

From the test results, the performance is improved obviously, and guest

memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock, we need to add a qemu patch to support parse feature.

---
v1 -> v2:
* Addressed comments from Jason Wang.
* Add performance test result independently.
* Use Skb_page_frag_refill() which can use high order page and reduce the stress of page allocator.
* Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small size can't fill one full packet, we only 128 vq num now.
* Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx consistency.
* Add virtio_transport ops to get max pkt len, in order to be compatible with old version.
---

Yiwen Jiang (5):
VSOCK: support fill mergeable rx buffer in guest
VSOCK: support fill data to mergeable rx buffer in host
VSOCK: support receive mergeable rx buffer in guest
VSOCK: increase send pkt len in mergeable mode to improve performance
VSOCK: batch sending rx buffer to increase bandwidth

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

Quoted 5 lines

Quoted 2 lines

Signature · 1 line

1.8.3.1

.

Quoted 25 lines · jiangyiwen

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

This series of patches mainly did three things：
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big packet(64K) included guest->Host and Host->Guest. I test performance independently and the result as follows:

Before performance:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

Quoted 2 lines

In this case, max send pkt size is still limited to 4K, so Host->Guest performance will worse than before.

It's concerning though, what if application sends small packets? What is the source of the slowdown? Do you know?

Quoted 44 lines

After performance increase the max send pkt size to 64K:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s

Quoted 1 line

From the test results, the performance is improved obviously, and guest

memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock, we need to add a qemu patch to support parse feature.

---
v1 -> v2:
* Addressed comments from Jason Wang.
* Add performance test result independently.
* Use Skb_page_frag_refill() which can use high order page and reduce the stress of page allocator.
* Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small size can't fill one full packet, we only 128 vq num now.
* Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx consistency.
* Add virtio_transport ops to get max pkt len, in order to be compatible with old version.
---

Yiwen Jiang (5):
VSOCK: support fill mergeable rx buffer in guest
VSOCK: support fill data to mergeable rx buffer in host
VSOCK: support receive mergeable rx buffer in guest
VSOCK: increase send pkt len in mergeable mode to improve performance
VSOCK: batch sending rx buffer to increase bandwidth

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

Quoted 2 lines

Signature · 1 line

1.8.3.1

.

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

This series of patches mainly did three things：
- mergeable buffer implementation
- increase the max send pkt size
- add used and signal guest in a batch

And I write a tool to test the vhost-vsock performance, mainly send big packet(64K) included guest->Host and Host->Guest. I test performance independently and the result as follows:

Before performance:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1450MB/s                ~1600MB/s

After performance only use implement mergeable rx buffer:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~400MB/s                 ~480MB/s
Host->Guest   ~1280MB/s                ~1350MB/s

In this case, max send pkt size is still limited to 4K, so Host->Guest performance will worse than before.

After performance increase the max send pkt size to 64K:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1500MB/s                ~2440MB/s

After performance all patches are used:

              Single socket            Multiple sockets(Max Bandwidth)
Guest->Host   ~1700MB/s                ~2900MB/s
Host->Guest   ~1700MB/s                ~2900MB/s

Quoted 1 line

From the test results, the performance is improved obviously, and guest

memory will not be wasted.

In addition, in order to support mergeable rx buffer in virtio-vsock, we need to add a qemu patch to support parse feature.

---
v1 -> v2:
* Addressed comments from Jason Wang.
* Add performance test result independently.
* Use Skb_page_frag_refill() which can use high order page and reduce the stress of page allocator.
* Still use fixed size(PAGE_SIZE) to fill rx buffer, because too small size can't fill one full packet, we only 128 vq num now.
* Use iovec to replace buf in struct virtio_vsock_pkt, keep tx and rx consistency.
* Add virtio_transport ops to get max pkt len, in order to be compatible with old version.
---

Yiwen Jiang (5):
VSOCK: support fill mergeable rx buffer in guest
VSOCK: support fill data to mergeable rx buffer in host
VSOCK: support receive mergeable rx buffer in guest
VSOCK: increase send pkt len in mergeable mode to improve performance
VSOCK: batch sending rx buffer to increase bandwidth

 drivers/vhost/vsock.c                   | 183 ++++++++++++++++++++-----
 include/linux/virtio_vsock.h            |  13 +-
 include/uapi/linux/virtio_vsock.h       |   5 +
 net/vmw_vsock/virtio_transport.c        | 229 +++++++++++++++++++++++++++-----
 net/vmw_vsock/virtio_transport_common.c |  66 ++++++---
 5 files changed, 411 insertions(+), 85 deletions(-)

Signature · 1 line

1.8.3.1

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

Quoted 5 lines · jiangyiwen

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

Sorry, I've been a bad maintainer. I was focussed on other projects and my email backlog is huge.

I like the idea of trying out optimizations on virtio-vsock, seeing if code can be shared with virtio-net, and maybe later switching to a virtio-net transport for vsock (if it turns out enough code can be shared).

Another optimization that could be interesting:

Userspace processes reading from a socket sleep in vsock_stream_recvmsg(). I wonder if we can bypass struct virtio_vsock_pkt and copying the payload into pkt->buf in this case. (This doesn't improve poll(2)/select(2) though!)

Imagine a userspace process waiting for data on a socket. When the virtqueue becomes ready, we can read in struct virtio_vsock_hdr and find the socket for that connection. Then we could copy the payload directly to userspace instead of creating a virtio_vsock_pkt and copying to pkt->buf first.

Quoted 5 lines · jiangyiwen

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.

Sorry, I've been a bad maintainer. I was focussed on other projects and my email backlog is huge.

I like the idea of trying out optimizations on virtio-vsock, seeing if code can be shared with virtio-net, and maybe later switching to a virtio-net transport for vsock (if it turns out enough code can be shared).

Another optimization that could be interesting:

Userspace processes reading from a socket sleep in vsock_stream_recvmsg(). I wonder if we can bypass struct virtio_vsock_pkt and copying the payload into pkt->buf in this case. (This doesn't improve poll(2)/select(2) though!)

Imagine a userspace process waiting for data on a socket. When the virtqueue becomes ready, we can read in struct virtio_vsock_hdr and find the socket for that connection. Then we could copy the payload directly to userspace instead of creating a virtio_vsock_pkt and copying to pkt->buf first.

Now vsock only support send/receive small packet, it can't achieve high performance. As previous discussed with Jason Wang, I revisit the idea of vhost-net about mergeable rx buffer and implement the mergeable rx buffer in vhost-vsock, it can allow big packet to be scattered in into different buffers and improve performance obviously.