updated: kvm networking todo wiki
Subsystems (4)
Michael S. Tsirkin
maintainerHey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.
----
I've put up a wiki page with a kvm networking todo list, mainly to avoid effort duplication, but also in the hope to draw attention to what I think we should try addressing in KVM:
http://www.linux-kvm.org/page/NetworkingTodo
This page could cover all networking related activity in KVM, currently most info is related to virtio-net.
Note: if there's no developer listed for an item, this just means I don't know of anyone actively working on an issue at the moment, not that no one intends to.
I would appreciate it if others working on one of the items on this list would add their names so we can communicate better. If others like this wiki page, please go ahead and add stuff you are working on if any.
It would be especially nice to add autotest projects:
there is just a short test matrix and a catch-all 'Cover test matrix with autotest', currently.
Currently there are some links to Red Hat bugzilla entries, feel free to add links to other bugzillas.
Thanks!
Signature · 1 line
MST
Quoted 21 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.----
I've put up a wiki page with a kvm networking todo list, mainly to avoid effort duplication, but also in the hope to draw attention to what I think we should try addressing in KVM:
http://www.linux-kvm.org/page/NetworkingTodo
This page could cover all networking related activity in KVM, currently most info is related to virtio-net.
Note: if there's no developer listed for an item, this just means I don't know of anyone actively working on an issue at the moment, not that no one intends to.
Quoted 7 lines
I would appreciate it if others working on one of the items on this list would add their names so we can communicate better. If others like this wiki page, please go ahead and add stuff you are working on if any.
It would be especially nice to add autotest projects:
there is just a short test matrix and a catch-all 'Cover test matrix with autotest', currently.
Ok, I'll take a look and fill in with currently available networking tests.
Quoted 5 lines
Currently there are some links to Red Hat bugzilla entries, feel free to add links to other bugzillas.
Thanks!
Quoted 21 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.----
I've put up a wiki page with a kvm networking todo list, mainly to avoid effort duplication, but also in the hope to draw attention to what I think we should try addressing in KVM:
http://www.linux-kvm.org/page/NetworkingTodo
This page could cover all networking related activity in KVM, currently most info is related to virtio-net.
Note: if there's no developer listed for an item, this just means I don't know of anyone actively working on an issue at the moment, not that no one intends to.
Quoted 7 lines
I would appreciate it if others working on one of the items on this list would add their names so we can communicate better. If others like this wiki page, please go ahead and add stuff you are working on if any.
It would be especially nice to add autotest projects:
there is just a short test matrix and a catch-all 'Cover test matrix with autotest', currently.
Ok, I'll take a look and fill in with currently available networking tests.
Quoted 5 lines
Currently there are some links to Red Hat bugzilla entries, feel free to add links to other bugzillas.
Thanks!
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.
Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
Quoted 29 lines
----
I've put up a wiki page with a kvm networking todo list, mainly to avoid effort duplication, but also in the hope to draw attention to what I think we should try addressing in KVM:
http://www.linux-kvm.org/page/NetworkingTodo
This page could cover all networking related activity in KVM, currently most info is related to virtio-net.
Note: if there's no developer listed for an item, this just means I don't know of anyone actively working on an issue at the moment, not that no one intends to.
I would appreciate it if others working on one of the items on this list would add their names so we can communicate better. If others like this wiki page, please go ahead and add stuff you are working on if any.
It would be especially nice to add autotest projects:
there is just a short test matrix and a catch-all 'Cover test matrix with autotest', currently.Currently there are some links to Red Hat bugzilla entries, feel free to add links to other bugzillas.
Thanks!
Michael S. Tsirkin
maintainerQuoted 10 lines · Jason Wang
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
It might have been interesting several years ago, but now that linux has vhost-net in kernel, the only point seems to be to
speed up networking on non-linux hosts. Since non-linux
does not have kvm, I doubt virtio is a bottleneck.
IMO yet another networking backend is a distraction,
and confusing to users.
In any case, I'd like to see virtio-blk dataplane replace non dataplane first. We don't want two copies of
virtio-net in qemu.
Quoted 29 lines
Quoted 29 lines
----
I've put up a wiki page with a kvm networking todo list, mainly to avoid effort duplication, but also in the hope to draw attention to what I think we should try addressing in KVM:
http://www.linux-kvm.org/page/NetworkingTodo
This page could cover all networking related activity in KVM, currently most info is related to virtio-net.
Note: if there's no developer listed for an item, this just means I don't know of anyone actively working on an issue at the moment, not that no one intends to.
I would appreciate it if others working on one of the items on this list would add their names so we can communicate better. If others like this wiki page, please go ahead and add stuff you are working on if any.
It would be especially nice to add autotest projects:
there is just a short test matrix and a catch-all 'Cover test matrix with autotest', currently.Currently there are some links to Red Hat bugzilla entries, feel free to add links to other bugzillas.
Thanks!
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.
Quoted 8 lines
Quoted 4 lines
Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
It might have been interesting several years ago, but now that linux has vhost-net in kernel, the only point seems to be to
speed up networking on non-linux hosts.
Data plane just means having a dedicated thread for virtqueue processing that doesn't hold qemu_mutex.
Of course we're going to do this in QEMU. It's a no brainer. But not as a separate device, just as an improvement to the existing userspace virtio-net.
Quoted 1 line
Since non-linux does not have kvm, I doubt virtio is a bottleneck.
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
Quoted 5 lines
IMO yet another networking backend is a distraction, and confusing to users.
In any case, I'd like to see virtio-blk dataplane replace non dataplane first. We don't want two copies of virtio-net in qemu.
100% agreed.
Regards,
Anthony Liguori
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
----
I've put up a wiki page with a kvm networking todo list, mainly to avoid effort duplication, but also in the hope to draw attention to what I think we should try addressing in KVM:
http://www.linux-kvm.org/page/NetworkingTodo
This page could cover all networking related activity in KVM, currently most info is related to virtio-net.
Note: if there's no developer listed for an item, this just means I don't know of anyone actively working on an issue at the moment, not that no one intends to.
I would appreciate it if others working on one of the items on this list would add their names so we can communicate better. If others like this wiki page, please go ahead and add stuff you are working on if any.
It would be especially nice to add autotest projects:
there is just a short test matrix and a catch-all 'Cover test matrix with autotest', currently.Currently there are some links to Red Hat bugzilla entries, feel free to add links to other bugzillas.
Thanks!
Michael S. Tsirkin
maintainerQuoted 25 lines · Anthony Liguori
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.Quoted 8 lines
Quoted 4 lines
Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
It might have been interesting several years ago, but now that linux has vhost-net in kernel, the only point seems to be to
speed up networking on non-linux hosts.Data plane just means having a dedicated thread for virtqueue processing that doesn't hold
qemu_mutex.Of course we're going to do this in QEMU. It's a no brainer. But not as a separate device, just as an improvement to the existing userspace virtio-net.
Quoted 8 lines
Quoted 1 line
Since non-linux does not have kvm, I doubt virtio is a bottleneck.
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
Great idea, that sounds very intresting.
I'll add it to the wiki.
In fact a bit of complexity in vhost was put there in the vague hope to support something like this: virtio rings are not translated through regular memory tables, instead, vhost gets a pointer to ring address.
This allows qemu acting as a man in the middle,
verifying the descriptors but not touching the
Anyone interested in working on such a project?
Quoted 42 lines
Quoted 5 lines
IMO yet another networking backend is a distraction, and confusing to users.
In any case, I'd like to see virtio-blk dataplane replace non dataplane first. We don't want two copies of virtio-net in qemu.100% agreed.
Regards,
Anthony Liguori
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
----
I've put up a wiki page with a kvm networking todo list, mainly to avoid effort duplication, but also in the hope to draw attention to what I think we should try addressing in KVM:
http://www.linux-kvm.org/page/NetworkingTodo
This page could cover all networking related activity in KVM, currently most info is related to virtio-net.
Note: if there's no developer listed for an item, this just means I don't know of anyone actively working on an issue at the moment, not that no one intends to.
I would appreciate it if others working on one of the items on this list would add their names so we can communicate better. If others like this wiki page, please go ahead and add stuff you are working on if any.
It would be especially nice to add autotest projects:
there is just a short test matrix and a catch-all 'Cover test matrix with autotest', currently.Currently there are some links to Red Hat bugzilla entries, feel free to add links to other bugzillas.
Thanks!
Quoted 39 lines · Michael S. Tsirkin
Quoted 25 lines · Anthony Liguori
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.Quoted 8 lines
Quoted 4 lines
Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
It might have been interesting several years ago, but now that linux has vhost-net in kernel, the only point seems to be to
speed up networking on non-linux hosts.Data plane just means having a dedicated thread for virtqueue processing that doesn't hold
qemu_mutex.Of course we're going to do this in QEMU. It's a no brainer. But not as a separate device, just as an improvement to the existing userspace virtio-net.
Quoted 8 lines
Quoted 1 line
Since non-linux does not have kvm, I doubt virtio is a bottleneck.
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
Great idea, that sounds very intresting.
I'll add it to the wiki.
Quoted 8 lines
In fact a bit of complexity in vhost was put there in the vague hope to support something like this: virtio rings are not translated through regular memory tables, instead, vhost gets a pointer to ring address.
This allows qemu acting as a man in the middle,
verifying the descriptors but not touching theAnyone interested in working on such a project?
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce. We already have two sets of host side ring code in the kernel (vhost and vringh, though they're being unified).
All an accelerator can offer on the tx side is zero copy and direct update of the used ring. On rx userspace could register the buffers and the accelerator could fill them and update the used ring. It still needs to deal with merged buffers, for example.
You avoid the address translation in the kernel, but I'm not convinced that's a key problem.
Cheers,
Rusty.
Quoted 44 lines · Rusty Russell
Quoted 43 lines · Michael S. Tsirkin
Quoted 25 lines · Anthony Liguori
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.Quoted 8 lines
Quoted 4 lines
Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
It might have been interesting several years ago, but now that linux has vhost-net in kernel, the only point seems to be to
speed up networking on non-linux hosts.Data plane just means having a dedicated thread for virtqueue processing that doesn't hold
qemu_mutex.Of course we're going to do this in QEMU. It's a no brainer. But not as a separate device, just as an improvement to the existing userspace virtio-net.
Quoted 8 lines
Quoted 1 line
Since non-linux does not have kvm, I doubt virtio is a bottleneck.
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
Great idea, that sounds very intresting.
I'll add it to the wiki.
In fact a bit of complexity in vhost was put there in the vague hope to support something like this: virtio rings are not translated through regular memory tables, instead, vhost gets a pointer to ring address.
Quoted 7 lines
Quoted 4 lines
This allows qemu acting as a man in the middle,
verifying the descriptors but not touching theAnyone interested in working on such a project?
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
Regards,
Anthony Liguori
Quoted 14 lines
We already have two
sets of host side ring code in the kernel (vhost and vringh, though they're being unified).All an accelerator can offer on the tx side is zero copy and direct update of the used ring. On rx userspace could register the buffers and the accelerator could fill them and update the used ring. It still needs to deal with merged buffers, for example.
You avoid the address translation in the kernel, but I'm not convinced that's a key problem.
Cheers,
Rusty.
Michael S. Tsirkin
maintainerQuoted 58 lines · Anthony Liguori
Quoted 44 lines · Rusty Russell
Quoted 43 lines · Michael S. Tsirkin
Quoted 25 lines · Anthony Liguori
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.Quoted 8 lines
Quoted 4 lines
Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
It might have been interesting several years ago, but now that linux has vhost-net in kernel, the only point seems to be to
speed up networking on non-linux hosts.Data plane just means having a dedicated thread for virtqueue processing that doesn't hold
qemu_mutex.Of course we're going to do this in QEMU. It's a no brainer. But not as a separate device, just as an improvement to the existing userspace virtio-net.
Quoted 8 lines
Quoted 1 line
Since non-linux does not have kvm, I doubt virtio is a bottleneck.
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
Great idea, that sounds very intresting.
I'll add it to the wiki.
In fact a bit of complexity in vhost was put there in the vague hope to support something like this: virtio rings are not translated through regular memory tables, instead, vhost gets a pointer to ring address.
Quoted 7 lines
Quoted 4 lines
This allows qemu acting as a man in the middle,
verifying the descriptors but not touching theAnyone interested in working on such a project?
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
Quoted 6 lines
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
Regards,
Anthony Liguori
It was also floated as an alternative way to do live migration.
Quoted 14 lines
Quoted 14 lines
We already have two
sets of host side ring code in the kernel (vhost and vringh, though they're being unified).All an accelerator can offer on the tx side is zero copy and direct update of the used ring. On rx userspace could register the buffers and the accelerator could fill them and update the used ring. It still needs to deal with merged buffers, for example.
You avoid the address translation in the kernel, but I'm not convinced that's a key problem.
Cheers,
Rusty.
Quoted 12 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
Quoted 6 lines
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Confused,
Rusty.
Quoted 22 lines · Rusty Russell
Quoted 12 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
Quoted 6 lines
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Quoted 7 lines
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Sounds like recvmmsg(2).
Stefan
Quoted 7 lines · Stefan Hajnoczi
Quoted 4 lines · Rusty Russell
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Sounds like recvmmsg(2).
Wow... the future is here, today!
Thanks,
Rusty.
Quoted 26 lines · Stefan Hajnoczi
Quoted 25 lines · Rusty Russell
Quoted 12 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
Quoted 6 lines
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
Quoted 6 lines
Quoted 4 lines
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Sounds like recvmmsg(2).
Could we map this to mergable rx buffers though?
Regards,
Anthony Liguori
Quoted 2 lines
Stefan
Michael S. Tsirkin
maintainerQuoted 33 lines · Anthony Liguori
Quoted 31 lines · Stefan Hajnoczi
Quoted 29 lines · Rusty Russell
Quoted 12 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
Quoted 6 lines
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Quoted 7 lines
Quoted 1 line
Sounds like recvmmsg(2).
Could we map this to mergable rx buffers though?
Regards,
Anthony Liguori
Yes because we don't have to complete buffers in order.
Quoted 2 lines
Quoted 2 lines
Stefan
Quoted 36 lines · Michael S. Tsirkin
Quoted 35 lines · Anthony Liguori
Quoted 32 lines · Stefan Hajnoczi
Quoted 22 lines · Rusty Russell
Quoted 12 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
Quoted 6 lines
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Quoted 7 lines
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Sounds like recvmmsg(2).
Quoted 7 lines
Quoted 5 lines
Could we map this to mergable rx buffers though?
Regards,
Anthony Liguori
Yes because we don't have to complete buffers in order.
What I meant though was for GRO, we don't know how large the received packet is going to be. Mergable rx buffers lets us allocate a pool of data for all incoming packets instead of allocating max packet size * max packets.
recvmmsg expects an array of msghdrs and I presume each needs to be given a fixed size. So this seems incompatible with mergable rx buffers.
Regards,
Anthony Liguori
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Stefan
Quoted 46 lines · Anthony Liguori
Quoted 36 lines · Michael S. Tsirkin
Quoted 35 lines · Anthony Liguori
Quoted 32 lines · Stefan Hajnoczi
Quoted 22 lines · Rusty Russell
Quoted 12 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
Quoted 6 lines
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Quoted 7 lines
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Sounds like recvmmsg(2).
Quoted 7 lines
Quoted 5 lines
Could we map this to mergable rx buffers though?
Regards,
Anthony Liguori
Yes because we don't have to complete buffers in order.
Quoted 8 lines
What I meant though was for GRO, we don't know how large the received packet is going to be. Mergable rx buffers lets us allocate a pool of data for all incoming packets instead of allocating max packet size * max packets.
recvmmsg expects an array of msghdrs and I presume each needs to be given a fixed size. So this seems incompatible with mergable rx buffers.
Good point. You'd need to build 64k buffers to pass to recvmmsg, then reuse the parts it didn't touch on the next call. This limits us to about a 16th of what we could do with an interface which understood buffer merging, but I don't know how much that would matter in practice. We'd need some benchmarks....
Cheers,
Rusty.
Quoted 22 lines · Rusty Russell
Quoted 12 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
Quoted 6 lines
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Quoted 7 lines
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Sounds like recvmmsg(2).
Stefan
Quoted 17 lines · Rusty Russell
Quoted 16 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
Quoted 7 lines
Quoted 2 lines
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
The only trouble with sendmsg would be doing batch submission and asynchronous completion.
A thread pool could certainly be used for this I guess.
Regards,
Anthony Liguori
Quoted 7 lines
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Confused,
Rusty.
Quoted 17 lines · Rusty Russell
Quoted 16 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
Quoted 7 lines
Quoted 2 lines
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
The only trouble with sendmsg would be doing batch submission and asynchronous completion.
A thread pool could certainly be used for this I guess.
Regards,
Anthony Liguori
Quoted 7 lines
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Confused,
Rusty.
Quoted 12 lines · Anthony Liguori
Quoted 10 lines · Rusty Russell
Quoted 7 lines · Anthony Liguori
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce.
Quoted 6 lines
The model is very interesting for QEMU because then we can use vhost as a backend for other types of network adapters (like vmxnet3 or even e1000).
It also helps for things like fault tolerance where we need to be able to control packet flow within QEMU.
(CC's reduced, context added, Dmitry Fleytman added for vmxnet3 thoughts).
Then I'm really confused as to what this would look like. A zero copy sendmsg? We should be able to implement that today.
On the receive side, what can we do better than readv? If we need to return to userspace to tell the guest that we've got a new packet, we don't win on latency. We might reduce syscall overhead with a multi-dimensional readv to read multiple packets at once?
Confused,
Rusty.
Quoted 39 lines · Michael S. Tsirkin
Quoted 25 lines · Anthony Liguori
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.Quoted 8 lines
Quoted 4 lines
Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
It might have been interesting several years ago, but now that linux has vhost-net in kernel, the only point seems to be to
speed up networking on non-linux hosts.Data plane just means having a dedicated thread for virtqueue processing that doesn't hold
qemu_mutex.Of course we're going to do this in QEMU. It's a no brainer. But not as a separate device, just as an improvement to the existing userspace virtio-net.
Quoted 8 lines
Quoted 1 line
Since non-linux does not have kvm, I doubt virtio is a bottleneck.
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
Great idea, that sounds very intresting.
I'll add it to the wiki.
Quoted 8 lines
In fact a bit of complexity in vhost was put there in the vague hope to support something like this: virtio rings are not translated through regular memory tables, instead, vhost gets a pointer to ring address.
This allows qemu acting as a man in the middle,
verifying the descriptors but not touching theAnyone interested in working on such a project?
It would be an interesting idea if we didn't already have the vhost model where we don't need the userspace bounce. We already have two sets of host side ring code in the kernel (vhost and vringh, though they're being unified).
All an accelerator can offer on the tx side is zero copy and direct update of the used ring. On rx userspace could register the buffers and the accelerator could fill them and update the used ring. It still needs to deal with merged buffers, for example.
You avoid the address translation in the kernel, but I'm not convinced that's a key problem.
Cheers,
Rusty.
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Michael S. Tsirkin
Hey guys,
I've updated the kvm networking todo wiki with current projects. Will try to keep it up to date more often.
Original announcement below.
Quoted 8 lines
Quoted 4 lines
Thanks a lot. I've added the tasks I'm currently working on to the wiki.
btw. I notice the virtio-net data plane were missed in the wiki. Is the project still being considered?
It might have been interesting several years ago, but now that linux has vhost-net in kernel, the only point seems to be to
speed up networking on non-linux hosts.
Data plane just means having a dedicated thread for virtqueue processing that doesn't hold qemu_mutex.
Of course we're going to do this in QEMU. It's a no brainer. But not as a separate device, just as an improvement to the existing userspace virtio-net.
Quoted 1 line
Since non-linux does not have kvm, I doubt virtio is a bottleneck.
FWIW, I think what's more interesting is using vhost-net as a networking backend with virtio-net in QEMU being what's guest facing.
In theory, this gives you the best of both worlds: QEMU acts as a first line of defense against a malicious guest while still getting the performance advantages of vhost-net (zero-copy).
Quoted 5 lines
IMO yet another networking backend is a distraction, and confusing to users.
In any case, I'd like to see virtio-blk dataplane replace non dataplane first. We don't want two copies of virtio-net in qemu.
100% agreed.
Regards,
Anthony Liguori
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
----
I've put up a wiki page with a kvm networking todo list, mainly to avoid effort duplication, but also in the hope to draw attention to what I think we should try addressing in KVM:
http://www.linux-kvm.org/page/NetworkingTodo
This page could cover all networking related activity in KVM, currently most info is related to virtio-net.
Note: if there's no developer listed for an item, this just means I don't know of anyone actively working on an issue at the moment, not that no one intends to.
I would appreciate it if others working on one of the items on this list would add their names so we can communicate better. If others like this wiki page, please go ahead and add stuff you are working on if any.
It would be especially nice to add autotest projects:
there is just a short test matrix and a catch-all 'Cover test matrix with autotest', currently.Currently there are some links to Red Hat bugzilla entries, feel free to add links to other bugzillas.
Thanks!