VSOCK: The performance problem of vhost_vsock.

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Stefan, have you encountered these problems?

Thanks,
Yiwen.

Quoted 8 lines · jiangyiwen

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

TCP/IP is not a must for vhost-net.

How do you test and compare the performance?

Thanks

Quoted 10 lines

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Stefan, have you encountered these problems?

Thanks,
Yiwen.

.

Quoted 12 lines · Jason Wang

Quoted 8 lines · jiangyiwen

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

Quoted 6 lines

TCP/IP is not a must for vhost-net.

How do you test and compare the performance?

Thanks

I test the performance used my test tool, like follows:

Server Client
socket()
bind()
listen()

socket(AF_VSOCK) or socket(AF_INET)
Accept() <-------------->connect()
*======Start Record Time======* Call syscall sendfile()
Recv()
Send end
Receive end
Send(file_size)

                         Recv(file_size)
                         *======End Record Time======*

The test result, vhost-vsock is about 500MB/s, and vhost-net is about 2500MB/s.

By the way, vhost-net use single queue.

Thanks.

Quoted 14 lines

Quoted 10 lines

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Stefan, have you encountered these problems?

Thanks,
Yiwen.

.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Quoted 32 lines · jiangyiwen

Quoted 7 lines · Jason Wang

Quoted 5 lines · jiangyiwen

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

Quoted 9 lines

Quoted 3 lines

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

TCP/IP is not a must for vhost-net.

How do you test and compare the performance?

Thanks

I test the performance used my test tool, like follows:

Server Client
socket()
bind()
listen()

socket(AF_VSOCK) or socket(AF_INET)
Accept() <-------------->connect()
*======Start Record Time======* Call syscall sendfile()
Recv()
Send end
Receive end
Send(file_size)

Quoted 23 lines

                          Recv(file_size)
                          *======End Record Time======*

The test result, vhost-vsock is about 500MB/s, and vhost-net is about 2500MB/s.

By the way, vhost-net use single queue.

Thanks.

Quoted 13 lines

Quoted 10 lines

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Stefan, have you encountered these problems?

Thanks,
Yiwen.

.

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Thanks,
Yiwen.

Quoted 5 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Quoted 8 lines

Quoted 1 line

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 7 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Thanks,
Yiwen.

Quoted 21 lines · Jason Wang

Quoted 8 lines

Quoted 1 line

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 5 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Quoted 2 lines

Quoted 2 lines

Thanks,
Yiwen.

Btw, if you're using vsock for transferring large files, maybe it's more efficient to implement sendpage() for vsock to allow sendfile()/splice() work.

Thanks

.

Quoted 19 lines · Jason Wang

Quoted 18 lines · Jason Wang

Quoted 7 lines

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 4 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Quoted 7 lines

Quoted 2 lines

Quoted 2 lines

Thanks,
Yiwen.

Btw, if you're using vsock for transferring large files, maybe it's more efficient to implement sendpage() for vsock to allow sendfile()/splice() work.

Thanks

I can't agree more.

why vhost_vsock is still behind vhost_net?
Because I use sendfile() to test performance at first, and then I found vsock don't implement sendpage() and cause the bandwidth can't be increased. So I use read() and send() to replace sendfile(), it will increase some switch between kernel and user mode, and sendfile() can support zero copy. I think this is main reason.

Thanks.

Quoted 3 lines

.

Quoted 36 lines · jiangyiwen

Quoted 19 lines · Jason Wang

Quoted 18 lines · Jason Wang

Quoted 7 lines

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 4 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Quoted 7 lines

Quoted 2 lines

Quoted 2 lines

Thanks,
Yiwen.

Btw, if you're using vsock for transferring large files, maybe it's more efficient to implement sendpage() for vsock to allow sendfile()/splice() work.

Thanks

I can't agree more.

why vhost_vsock is still behind vhost_net?
Because I use sendfile() to test performance at first, and then I found vsock don't implement sendpage() and cause the bandwidth can't be increased. So I use read() and send() to replace sendfile(), it will increase some switch between kernel and user mode, and sendfile() can support zero copy. I think this is main reason.

Quoted 1 line

Thanks.

Want to post patches for this then :) ?

Thanks

Quoted 5 lines

Quoted 4 lines

.

.

Quoted 39 lines · Jason Wang

Quoted 27 lines · jiangyiwen

Quoted 18 lines · Jason Wang

Quoted 17 lines · Jason Wang

Quoted 7 lines

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 4 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Quoted 6 lines

Quoted 2 lines

Quoted 2 lines

Thanks,
Yiwen.

Btw, if you're using vsock for transferring large files, maybe it's more efficient to implement sendpage() for vsock to allow sendfile()/splice() work.

Thanks

I can't agree more.

Quoted 8 lines

why vhost_vsock is still behind vhost_net?
Because I use sendfile() to test performance at first, and then I found vsock don't implement sendpage() and cause the bandwidth can't be increased. So I use read() and send() to replace sendfile(), it will increase some switch between kernel and user mode, and sendfile() can support zero copy. I think this is main reason.

Thanks.

Want to post patches for this then :) ?

Quoted 2 lines

Thanks

I may not post patches at the moment because there are other tasks.

After a period of time, I will consider implement the feature.

Thanks.

Quoted 39 lines · Jason Wang

Quoted 27 lines · jiangyiwen

Quoted 18 lines · Jason Wang

Quoted 17 lines · Jason Wang

Quoted 7 lines

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 4 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Quoted 6 lines

Quoted 2 lines

Quoted 2 lines

Thanks,
Yiwen.

Btw, if you're using vsock for transferring large files, maybe it's more efficient to implement sendpage() for vsock to allow sendfile()/splice() work.

Thanks

I can't agree more.

Quoted 8 lines

why vhost_vsock is still behind vhost_net?
Because I use sendfile() to test performance at first, and then I found vsock don't implement sendpage() and cause the bandwidth can't be increased. So I use read() and send() to replace sendfile(), it will increase some switch between kernel and user mode, and sendfile() can support zero copy. I think this is main reason.

Thanks.

Want to post patches for this then :) ?

Quoted 2 lines

Thanks

I may not post patches at the moment because there are other tasks.

After a period of time, I will consider implement the feature.

Thanks.

Quoted 19 lines · Jason Wang

Quoted 18 lines · Jason Wang

Quoted 7 lines

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 4 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Quoted 7 lines

Quoted 2 lines

Quoted 2 lines

Thanks,
Yiwen.

Btw, if you're using vsock for transferring large files, maybe it's more efficient to implement sendpage() for vsock to allow sendfile()/splice() work.

Thanks

I can't agree more.

why vhost_vsock is still behind vhost_net?
Because I use sendfile() to test performance at first, and then I found vsock don't implement sendpage() and cause the bandwidth can't be increased. So I use read() and send() to replace sendfile(), it will increase some switch between kernel and user mode, and sendfile() can support zero copy. I think this is main reason.

Thanks.

Quoted 3 lines

.

Quoted 8 lines

Quoted 1 line

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 7 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Thanks,
Yiwen.

Quoted 56 lines · jiangyiwen

Quoted 32 lines · jiangyiwen

Quoted 7 lines · Jason Wang

Quoted 5 lines · jiangyiwen

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

Quoted 9 lines

Quoted 3 lines

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

TCP/IP is not a must for vhost-net.

How do you test and compare the performance?

Thanks

I test the performance used my test tool, like follows:

Server Client
socket()
bind()
listen()

socket(AF_VSOCK) or socket(AF_INET)
Accept() <-------------->connect()
*======Start Record Time======* Call syscall sendfile()
Recv()
Send end
Receive end
Send(file_size)

Quoted 23 lines

                          Recv(file_size)
                          *======End Record Time======*

The test result, vhost-vsock is about 500MB/s, and vhost-net is about 2500MB/s.

By the way, vhost-net use single queue.

Thanks.

Quoted 13 lines

Quoted 10 lines

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Stefan, have you encountered these problems?

Thanks,
Yiwen.

.

Quoted 7 lines

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 14 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Thanks,
Yiwen.

Quoted 5 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

.

Quoted 56 lines · jiangyiwen

Quoted 32 lines · jiangyiwen

Quoted 7 lines · Jason Wang

Quoted 5 lines · jiangyiwen

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

Quoted 9 lines

Quoted 3 lines

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

TCP/IP is not a must for vhost-net.

How do you test and compare the performance?

Thanks

I test the performance used my test tool, like follows:

Server Client
socket()
bind()
listen()

socket(AF_VSOCK) or socket(AF_INET)
Accept() <-------------->connect()
*======Start Record Time======* Call syscall sendfile()
Recv()
Send end
Receive end
Send(file_size)

Quoted 23 lines

                          Recv(file_size)
                          *======End Record Time======*

The test result, vhost-vsock is about 500MB/s, and vhost-net is about 2500MB/s.

By the way, vhost-net use single queue.

Thanks.

Quoted 13 lines

Quoted 10 lines

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Stefan, have you encountered these problems?

Thanks,
Yiwen.

.

Quoted 7 lines

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

Looks like the value was chosen for a balance between rx buffer size and performance. Allocating 64K always even for small packet is kind of waste and stress for guest memory. Virito-net try to avoid this by inventing the merge able rx buffer which allows big packet to be scattered in into different buffers. We can reuse this idea or revisit the idea of using virtio-net/vhost-net as a transport of vsock.

What interesting is the performance is still behind vhost-net.

Thanks

Quoted 8 lines

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Thanks,
Yiwen.

Quoted 6 lines

Quoted 5 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

.

Quoted 32 lines · jiangyiwen

Quoted 7 lines · Jason Wang

Quoted 5 lines · jiangyiwen

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

Quoted 9 lines

Quoted 3 lines

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

TCP/IP is not a must for vhost-net.

How do you test and compare the performance?

Thanks

I test the performance used my test tool, like follows:

Server Client
socket()
bind()
listen()

socket(AF_VSOCK) or socket(AF_INET)
Accept() <-------------->connect()
*======Start Record Time======* Call syscall sendfile()
Recv()
Send end
Receive end
Send(file_size)

Quoted 23 lines

                          Recv(file_size)
                          *======End Record Time======*

The test result, vhost-vsock is about 500MB/s, and vhost-net is about 2500MB/s.

By the way, vhost-net use single queue.

Thanks.

Quoted 13 lines

Quoted 10 lines

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Stefan, have you encountered these problems?

Thanks,
Yiwen.

.

Hi Jason and Stefan,

Maybe I find the reason of bad performance.

I found pkt_len is limited to VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE(4K), it will cause the bandwidth is limited to 500~600MB/s. And once I increase to 64k, it can improve about 3 times(~1500MB/s).

By the way, I send to 64K in application once, and I don't use sg_init_one and rewrite function to packet sg list because pkt_len include multiple pages.

Thanks,
Yiwen.

Quoted 5 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Quoted 8 lines · jiangyiwen

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

TCP/IP is not a must for vhost-net.

How do you test and compare the performance?

Thanks

Quoted 5 lines

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Quoted 5 lines

Stefan, have you encountered these problems?

Thanks,
Yiwen.

.

Hi Stefan & All:

Now I find vhost-vsock has two performance problems even if it is not designed for performance.

First, I think vhost-vsock should faster than vhost-net because it is no TCP/IP stack, but the real test result vhost-net is 5~10 times than vhost-vsock, currently I am looking for the reason.

Second, vhost-vsock only supports two vqs(tx and rx), that means if multiple sockets in the guest will use the same vq to transmit the message and get the response. So if there are multiple applications in the guest, we should support "Multiqueue" feature for Virtio-vsock.

Stefan, have you encountered these problems?

Thanks,
Yiwen.