vhost: a kernel-level virtio server
Subsystems (5)
Michael S. Tsirkin
maintainerv2 cover vhost: a kernel-level virtio server
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
Changes from v1:
- Move use_mm/unuse_mm from fs/aio.c to mm instead of copying.
- Reorder code to avoid need for forward declarations
- Kill a couple of debugging printks
Michael S. Tsirkin (2):
mm: export use_mm/unuse_mm to modules
vhost_net: a kernel-level virtio server
MAINTAINERS | 10 + arch/x86/kvm/Kconfig | 1 + drivers/Makefile | 1 + drivers/vhost/Kconfig | 11 + drivers/vhost/Makefile | 2 + drivers/vhost/net.c | 411 +++++++++++++++++++++++++++ drivers/vhost/vhost.c | 663 +++++++++++++++++++++++++++++++++++++++++++ drivers/vhost/vhost.h | 108 +++++++ fs/aio.c | 47 +--- include/linux/Kbuild | 1 + include/linux/miscdevice.h | 1 + include/linux/mmu_context.h | 9 + include/linux/vhost.h | 100 +++++++ mm/Makefile | 2 +- mm/mmu_context.c | 58 ++++ 15 files changed, 1378 insertions(+), 47 deletions(-) create mode 100644 drivers/vhost/Kconfig create mode 100644 drivers/vhost/Makefile create mode 100644 drivers/vhost/net.c create mode 100644 drivers/vhost/vhost.c create mode 100644 drivers/vhost/vhost.h create mode 100644 include/linux/mmu_context.h create mode 100644 include/linux/vhost.h create mode 100644 mm/mmu_context.c
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Michael S. Tsirkin
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run?
Regards,
-Greg
Michael S. Tsirkin
maintainerQuoted 10 lines · Gregory Haskins
Quoted 9 lines · Michael S. Tsirkin
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
Quoted 9 lines
Quoted 2 lines
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run?
Regards,
-Greg
1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
2. disable tso,gso,lro with ethtool
3. add vhost=ethX
Signature · 7 lines
MST -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 16 lines · Michael S. Tsirkin
Quoted 15 lines · Gregory Haskins
Quoted 11 lines · Michael S. Tsirkin
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run?
Quoted 7 lines
Quoted 3 lines
Regards,
-Greg1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
Quoted 1 line
2. disable tso,gso,lro with ethtool
Out of curiosity, wouldnt you only need to disable LRO on the adapter, since the other two (IIUC) are transmit path and are therefore influenced by the skb's you generate in vhost?
Quoted 1 line
3. add vhost=ethX
You mean via "ip link" I assume?
Regards,
-Greg
Michael S. Tsirkin
maintainerQuoted 21 lines · Gregory Haskins
Quoted 20 lines · Michael S. Tsirkin
Quoted 18 lines · Gregory Haskins
Quoted 11 lines · Michael S. Tsirkin
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run?
Regards,
-Greg
Quoted 7 lines
Quoted 3 lines
1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Quoted 5 lines
Quoted 1 line
2. disable tso,gso,lro with ethtool
Out of curiosity, wouldnt you only need to disable LRO on the adapter, since the other two (IIUC) are transmit path and are therefore influenced by the skb's you generate in vhost?
Hmm, makes sense. I'll check this and let you know.
Quoted 4 lines
Quoted 2 lines
3. add vhost=ethX
You mean via "ip link" I assume?
No, that's a new flag for virtio in qemu:
-net nic,model=virtio,vhost=veth0
Quoted 3 lines
Regards,
-Greg
Quoted 2 lines · Michael S. Tsirkin
Quoted 1 line · Gregory Haskins
Michael S. Tsirkin wrote:
<snip>
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Makes sense. Got it.
I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge?
<snip>
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
3. add vhost=ethX
You mean via "ip link" I assume?
No, that's a new flag for virtio in qemu:
-net nic,model=virtio,vhost=veth0
Ah, ok. Even better.
Thanks!
-Greg
Quoted 4 lines · Gregory Haskins
Quoted 4 lines
Quoted 3 lines
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
Quoted 9 lines
Quoted 2 lines
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Makes sense. Got it.
I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge?
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2
It's a bit more complicated than it need to be, but should work fine.
Arnd <><
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michael S. Tsirkin
maintainerQuoted 18 lines · Arnd Bergmann
Quoted 4 lines · Gregory Haskins
Quoted 4 lines
Quoted 3 lines
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
Quoted 9 lines
Quoted 2 lines
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Makes sense. Got it.
I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge?
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
Quoted 7 lines
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2It's a bit more complicated than it need to be, but should work fine.
Arnd <><
Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michael S. Tsirkin
maintainerQuoted 12 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Quoted 8 lines
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><
Oh, hopefully macvlan will soon allow that.
Signature · 7 lines
MST -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michael S. Tsirkin
maintainerQuoted 12 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Quoted 8 lines
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><
Oh, hopefully macvlan will soon allow that.
Signature · 1 line
MST
Quoted 12 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Quoted 8 lines
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><
Yeah, this would be the config I would be interested in.
Regards,
-Greg
Michael S. Tsirkin
maintainerQuoted 16 lines · Gregory Haskins
Quoted 15 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Quoted 7 lines
Quoted 5 lines
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><Yeah, this would be the config I would be interested in.
Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest.
Here's one way to do this:
-net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22
In guests, for simplicity, configure eth1 and eth0 to use separate subnets.
Long term, I hope macvlan will be extended to support guest to guest.
Quoted 3 lines
Regards,
-Greg
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 37 lines · Michael S. Tsirkin
Quoted 15 lines · Gregory Haskins
Quoted 14 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Quoted 6 lines
Quoted 5 lines
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><Yeah, this would be the config I would be interested in.
Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest.
Here's one way to do this:
-net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22
Quoted 2 lines
In guests, for simplicity, configure eth1 and eth0 to use separate subnets.
I can try to do a few variations, but what I am interested is in performance in a real-world L2 configuration. This would generally mean
all hosts (virtual or physical) in the same L2 domain.
If I get a chance, though, I will try to also wire them up in isolation as another data point.
Regards,
-Greg
Michael S. Tsirkin
maintainerQuoted 45 lines · Gregory Haskins
Quoted 37 lines · Michael S. Tsirkin
Quoted 15 lines · Gregory Haskins
Quoted 14 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Quoted 6 lines
Quoted 5 lines
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><Yeah, this would be the config I would be interested in.
Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest.
Here's one way to do this:
-net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22Quoted 2 lines
In guests, for simplicity, configure eth1 and eth0 to use separate subnets.
I can try to do a few variations, but what I am interested is in performance in a real-world L2 configuration. This would generally mean
all hosts (virtual or physical) in the same L2 domain.
Quoted 7 lines
If I get a chance, though, I will try to also wire them up in isolation as another data point.
Regards,
-Greg
Or patch macvlan to support guest to guest:
http://markmail.org/message/sjy74g57qsvdo2wh
That patch needs to be updated to support guest to guest multiast, but it seems functional enough for your purposes.
Signature · 7 lines
MST -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michael S. Tsirkin
maintainerQuoted 45 lines · Gregory Haskins
Quoted 37 lines · Michael S. Tsirkin
Quoted 15 lines · Gregory Haskins
Quoted 14 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Quoted 6 lines
Quoted 5 lines
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><Yeah, this would be the config I would be interested in.
Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest.
Here's one way to do this:
-net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22Quoted 2 lines
In guests, for simplicity, configure eth1 and eth0 to use separate subnets.
I can try to do a few variations, but what I am interested is in performance in a real-world L2 configuration. This would generally mean
all hosts (virtual or physical) in the same L2 domain.
Quoted 7 lines
If I get a chance, though, I will try to also wire them up in isolation as another data point.
Regards,
-Greg
Or patch macvlan to support guest to guest:
http://markmail.org/message/sjy74g57qsvdo2wh
That patch needs to be updated to support guest to guest multiast, but it seems functional enough for your purposes.
Signature · 1 line
MST
Quoted 37 lines · Michael S. Tsirkin
Quoted 15 lines · Gregory Haskins
Quoted 14 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Quoted 6 lines
Quoted 5 lines
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><Yeah, this would be the config I would be interested in.
Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest.
Here's one way to do this:
-net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22
Quoted 2 lines
In guests, for simplicity, configure eth1 and eth0 to use separate subnets.
I can try to do a few variations, but what I am interested is in performance in a real-world L2 configuration. This would generally mean
all hosts (virtual or physical) in the same L2 domain.
If I get a chance, though, I will try to also wire them up in isolation as another data point.
Regards,
-Greg
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 16 lines · Gregory Haskins
Quoted 15 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Quoted 7 lines
Quoted 5 lines
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><Yeah, this would be the config I would be interested in.
Hmm, this wouldn't be the config to use for the benchmark though: there are just too many variables. If you want both guest to guest and guest to host, create 2 nics in the guest.
Here's one way to do this:
-net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth0 -redir tcp:8022::22 -net nic,model=virtio,vlan=0 -net user,vlan=0 -net nic,vlan=1,model=virtio,vhost=veth1 -redir tcp:8023::22
In guests, for simplicity, configure eth1 and eth0 to use separate subnets.
Long term, I hope macvlan will be extended to support guest to guest.
Quoted 3 lines
Regards,
-Greg
Quoted 12 lines · Arnd Bergmann
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Quoted 8 lines
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><
Yeah, this would be the config I would be interested in.
Regards,
-Greg
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Sure, but the setup I described is the one that I would expect to see in practice because it gives you external connectivity.
Measuring two guests communicating over a veth pair is interesting for finding the bottlenecks, but of little practical relevance.
Arnd <><
Michael S. Tsirkin
maintainerQuoted 18 lines · Arnd Bergmann
Quoted 4 lines · Gregory Haskins
Quoted 4 lines
Quoted 3 lines
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
Quoted 9 lines
Quoted 2 lines
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Makes sense. Got it.
I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge?
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
Quoted 7 lines
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2It's a bit more complicated than it need to be, but should work fine.
Arnd <><
Heh, you don't need a bridge in this picture:
guest 1 - vhost - veth0 - veth1 - vhost guest 2
Quoted 4 lines · Gregory Haskins
Quoted 4 lines
Quoted 3 lines
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
Quoted 9 lines
Quoted 2 lines
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Makes sense. Got it.
I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge?
If I understand it correctly, you can at least connect a veth pair to a bridge, right? Something like
veth0 - veth1 - vhost - guest 1
eth0 - br0-|
veth2 - veth3 - vhost - guest 2
It's a bit more complicated than it need to be, but should work fine.
Arnd <><
Michael S. Tsirkin
maintainerQuoted 18 lines · Gregory Haskins
Quoted 2 lines · Michael S. Tsirkin
Quoted 1 line · Gregory Haskins
Michael S. Tsirkin wrote:
<snip>
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Quoted 7 lines
Makes sense. Got it.
I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge?
<snip>
Presumably you mean on the same host? There were also some patches to enable local guest to guest for macvlan, that would be a nice software-only solution. For back to back, I just tried over veth, seems to work fine.
Quoted 12 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
3. add vhost=ethX
You mean via "ip link" I assume?
No, that's a new flag for virtio in qemu:
-net nic,model=virtio,vhost=veth0
Ah, ok. Even better.
Thanks!
-Greg
Michael S. Tsirkin
maintainerQuoted 18 lines · Gregory Haskins
Quoted 2 lines · Michael S. Tsirkin
Quoted 1 line · Gregory Haskins
Michael S. Tsirkin wrote:
<snip>
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Quoted 7 lines
Makes sense. Got it.
I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge?
<snip>
Presumably you mean on the same host? There were also some patches to enable local guest to guest for macvlan, that would be a nice software-only solution. For back to back, I just tried over veth, seems to work fine.
Quoted 12 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
3. add vhost=ethX
You mean via "ip link" I assume?
No, that's a new flag for virtio in qemu:
-net nic,model=virtio,vhost=veth0
Ah, ok. Even better.
Thanks!
-Greg
Quoted 2 lines · Michael S. Tsirkin
Quoted 1 line · Gregory Haskins
Michael S. Tsirkin wrote:
<snip>
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Makes sense. Got it.
I was going to add guest-to-guest to the test matrix, but I assume that is not supported with vhost unless you have something like a VEPA enabled bridge?
<snip>
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
3. add vhost=ethX
You mean via "ip link" I assume?
No, that's a new flag for virtio in qemu:
-net nic,model=virtio,vhost=veth0
Ah, ok. Even better.
Thanks!
-Greg
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 21 lines · Gregory Haskins
Quoted 20 lines · Michael S. Tsirkin
Quoted 18 lines · Gregory Haskins
Quoted 11 lines · Michael S. Tsirkin
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run?
Regards,
-Greg
Quoted 7 lines
Quoted 3 lines
1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
SRIOV is not a requirement. And you can also use a dedicated nic+programmed mac if you are so inclined.
Quoted 5 lines
Quoted 1 line
2. disable tso,gso,lro with ethtool
Out of curiosity, wouldnt you only need to disable LRO on the adapter, since the other two (IIUC) are transmit path and are therefore influenced by the skb's you generate in vhost?
Hmm, makes sense. I'll check this and let you know.
Quoted 4 lines
Quoted 2 lines
3. add vhost=ethX
You mean via "ip link" I assume?
No, that's a new flag for virtio in qemu:
-net nic,model=virtio,vhost=veth0
Quoted 3 lines
Regards,
-Greg
Quoted 16 lines · Michael S. Tsirkin
Quoted 15 lines · Gregory Haskins
Quoted 11 lines · Michael S. Tsirkin
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run?
Quoted 7 lines
Quoted 3 lines
Regards,
-Greg1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
Are you saying SRIOV is a requirement, and I can either program the SRIOV adapter with a mac or use promis? Or are you saying I can use SRIOV+programmed mac OR a regular nic + promisc (with a perf penalty).
Quoted 1 line
2. disable tso,gso,lro with ethtool
Out of curiosity, wouldnt you only need to disable LRO on the adapter, since the other two (IIUC) are transmit path and are therefore influenced by the skb's you generate in vhost?
Quoted 1 line
3. add vhost=ethX
You mean via "ip link" I assume?
Regards,
-Greg
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 10 lines · Gregory Haskins
Quoted 9 lines · Michael S. Tsirkin
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
Quoted 9 lines
Quoted 2 lines
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run?
Regards,
-Greg
1. use a dedicated network interface with SRIOV, program mac to match that of guest (for testing, you can set promisc mode, but that is bad for performance)
2. disable tso,gso,lro with ethtool
3. add vhost=ethX
Signature · 1 line
MST
Quoted 11 lines · Michael S. Tsirkin
This implements vhost: a kernel-level backend for virtio, The main motivation for this work is to reduce virtualization overhead for virtio by removing system calls on data path, without guest changes. For virtio-net, this removes up to 4 system calls per packet: vm exit for kick, reentry for kick, iothread wakeup for packet, interrupt injection for packet.
Some more detailed description attached to the patch itself.
The patches are against 2.6.31-rc4. I'd like them to go into linux-next and down the road 2.6.32 if possible. Please comment.
I will add this series to my benchmark run in the next day or so. Any specific instructions on how to set it up and run?
Regards,
-Greg
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization