performance of virtual functions compared to virtio

In general should virtual functions outperform virtio+vhost for networking performance - latency and throughput?

I have 2 VMs running on a host. Each VM has 2 nics -- one tied to a VF and the other going through virtio and a tap device like so:

   ------                  ----
  |      |----------------| VF |---
  |      |                 ----   |
  | VM 1 |                        |
  |      |    -----               |
  |      |---| tap |---           |
   ------     -----   |          ---
                     ---        | e |
                    | b |       | t |
                    | r |       | h |
                     ---        | 2 |
   ------     -----   |          ---
  |      |---| tap |---           |
  |      |    -----               |
  | VM 2 |                        |
  |      |                 ----   |
  |      |----------------| VF |---
   ------                  ----

The network arguments to qemu-kvm are:

(quoted diff) +0 -2

-netdev type=tap,vhost=on,ifname=tap2,id=netdev1
-device virtio-net-pci,mac=${mac},netdev=netdev1

where ${mac} is unique to each VM and for the VF:
-device pci-assign,host=${pciid}

netserver is running within the VMs, and the netperf commands I am running are:

netperf -p 12346 -H <ip> -l 20 -jcC -fM -v 2 -t TCP_RR -- -r 1024 netperf -p 12346 -H <ip> -l 20 -jcC -fM -v 2 -t TCP_STREAM

where <ip> changes depending on which interface I want to send the traffic through. To say the least results are a bit disappointing for the VF:

                  latency   throughput
              (usec/Tran)   (MB/sec)

Host-VM
over virtio 139.160 1199.40
over VF 488.124 209.22

VM-VM

 over virtio      322.056     773.54
 over VF          488.051     328.88

I am just getting started with VFs and could use some hints on how to improve the performance.

Device assignment via a VF provides the lowest latency and most bandwidth for *getting data off the host system*, though virtio/vhost is getting better. If all you care about is VM-VM on the same host or VM-host, then virtio is only limited by memory bandwidth/latency and

host processor cycles. Your processor has 25GB/s of memory bandwidth. On the other hand, the VF has to send data all the way out to the wire and all the way back up through the NIC to get to the other VM/host. You're using a 1Gb/s NIC. Your results actually seem to indicate you're getting better than wire rate, so maybe you're only passing through an internal switch on the NIC, in any case, VFs are not optimal for communication within the same physical system. They are optimal for off host communication. Thanks,

Note I think in both cases we can make significant improvements:
- for VFs, steer device interrupts to the cpus which run the vcpus that will receive the interrupts eventually (ISTR some work about this, but not sure)
- for virtio, use a DMA engine to copy data (I think there exists code in upstream which does this, but has this been enabled/tuned?)

Quoted 5 lines · Avi Kivity

Note I think in both cases we can make significant improvements:
- for VFs, steer device interrupts to the cpus which run the vcpus that will receive the interrupts eventually (ISTR some work about this, but not sure)
- for virtio, use a DMA engine to copy data (I think there exists code in upstream which does this, but has this been enabled/tuned?)

Which data copy in virtio? Is this a vhost-net specific thing you're thinking about?

Quoted 4 lines · Stefan Hajnoczi

Quoted 2 lines · Avi Kivity

Note I think in both cases we can make significant improvements:
- for VFs, steer device interrupts to the cpus which run the vcpus

that will

Quoted 8 lines

Quoted 1 line

receive the interrupts eventually (ISTR some work about this, but

not sure)

Quoted 1 line

- for virtio, use a DMA engine to copy data (I think there exists

code in

Quoted 1 line

upstream which does this, but has this been enabled/tuned?)

Which data copy in virtio? Is this a vhost-net specific thing you're thinking about?

There are several copies.

qemu's virtio-net implementation incurs a copy on tx and on rx when calling the kernel; in addition there is also an internal copy:

        /* copy in packet.  ugh */
        len = iov_from_buf(sg, elem.in_num,
                           buf + offset, size - offset);

In principle vhost-net can avoid the tx copy, but I think now we have 1 copy on rx and tx each.

If a host interface is dedicated to backing a vhost-net interface (say if you have an SR/IOV card) then you can in principle avoid the rx copy as well.

An alternative to avoiding the copies is to use a dma engine, like I mentioned.

Quoted 10 lines

There are several copies.

qemu's virtio-net implementation incurs a copy on tx and on rx when calling the kernel; in addition there is also an internal copy:

         /* copy in packet.  ugh */
         len = iov_from_buf(sg, elem.in_num,
                            buf + offset, size - offset);

Quoted 2 lines

In principle vhost-net can avoid the tx copy, but I think now we have 1

 copy on rx and tx each.

So there is a copy internal to qemu, then from qemu to the host tap device and then tap device to a physical NIC if the packet is leaving the host?

Is that what the zero-copy patch set is attempting - bypassing the transmit copy to the macvtap device?

Quoted 8 lines

If a host interface is dedicated to backing a vhost-net interface (say if you have an SR/IOV card) then you can in principle avoid the rx copy as well.

An alternative to avoiding the copies is to use a dma engine, like I mentioned.

How does the DMA engine differ from the zero-copy patch set?

Quoted 10 lines · David Ahern

Quoted 10 lines

There are several copies.

qemu's virtio-net implementation incurs a copy on tx and on rx when calling the kernel; in addition there is also an internal copy:

         /* copy in packet.  ugh */
         len = iov_from_buf(sg, elem.in_num,
                            buf + offset, size - offset);

Quoted 7 lines

Quoted 1 line

In principle vhost-net can avoid the tx copy, but I think now we

have 1

Quoted 1 line

 copy on rx and tx each.

So there is a copy internal to qemu, then from qemu to the host tap device and then tap device to a physical NIC if the packet is leaving the host?

There is no internal copy on tx, just rx.

So:

virtio-net: 1 internal rx, 1 kernel/user rx, 1 kernel/user tx
vhost-net: 1 internal rx, 1 internal tx

Quoted 3 lines

Is that what the zero-copy patch set is attempting - bypassing the transmit copy to the macvtap device?

Yes.

Quoted 11 lines

Quoted 3 lines

If a host interface is dedicated to backing a vhost-net interface (say if you have an SR/IOV card) then you can in principle avoid the rx

copy

Quoted 5 lines

as well.

An alternative to avoiding the copies is to use a dma engine, like I mentioned.

How does the DMA engine differ from the zero-copy patch set?

The DMA engine does not avoid the copy, it merely uses a device other than the cpu to perform it. It offloads the cpu but still loads the interconnect. True zero-copy avoids both the cpu load and the interconnect load.

Is the following depict where copies are done for virtio-net?

Quoted 4 lines · Alex Williamson

Device assignment via a VF provides the lowest latency and most bandwidth for *getting data off the host system*, though virtio/vhost is getting better. If all you care about is VM-VM on the same host or VM-host, then virtio is only limited by memory bandwidth/latency and

Quoted 9 lines

host processor cycles. Your processor has 25GB/s of memory bandwidth. On the other hand, the VF has to send data all the way out to the wire and all the way back up through the NIC to get to the other VM/host. You're using a 1Gb/s NIC. Your results actually seem to indicate you're getting better than wire rate, so maybe you're only passing through an internal switch on the NIC, in any case, VFs are not optimal for communication within the same physical system. They are optimal for off host communication. Thanks,

Note I think in both cases we can make significant improvements:
- for VFs, steer device interrupts to the cpus which run the vcpus that will receive the interrupts eventually (ISTR some work about this, but not sure)

- for virtio, use a DMA engine to copy data (I think there exists code in upstream which does this, but has this been enabled/tuned?)

Quoted 5 lines

Note I think in both cases we can make significant improvements:
- for VFs, steer device interrupts to the cpus which run the vcpus that will receive the interrupts eventually (ISTR some work about this, but not sure)

I don't understand your point here. I thought interrupts for the VF were only delivered to the guest, not the host.

Device assignment via a VF provides the lowest latency and most bandwidth for *getting data off the host system*, though virtio/vhost is getting better. If all you care about is VM-VM on the same host or VM-host, then virtio is only limited by memory bandwidth/latency and

host processor cycles. Your processor has 25GB/s of memory bandwidth. On the other hand, the VF has to send data all the way out to the wire and all the way back up through the NIC to get to the other VM/host. You're using a 1Gb/s NIC. Your results actually seem to indicate you're getting better than wire rate, so maybe you're only passing through an internal switch on the NIC, in any case, VFs are not optimal for communication within the same physical system. They are optimal for off host communication. Thanks,

Quoted 4 lines · Alex Williamson

Device assignment via a VF provides the lowest latency and most bandwidth for *getting data off the host system*, though virtio/vhost is getting better. If all you care about is VM-VM on the same host or VM-host, then virtio is only limited by memory bandwidth/latency and

Quoted 8 lines

host processor cycles. Your processor has 25GB/s of memory bandwidth. On the other hand, the VF has to send data all the way out to the wire and all the way back up through the NIC to get to the other VM/host. You're using a 1Gb/s NIC. Your results actually seem to indicate you're getting better than wire rate, so maybe you're only passing through an internal switch on the NIC, in any case, VFs are not optimal for communication within the same physical system. They are optimal for off host communication. Thanks,

Hi Alex:

Host-host was the next focus for the tests. I have 2 of the aforementioned servers, each configured identically. As a reminder:

Host:
Dell R410
2 quad core [email protected] GHz processors
16 GB RAM
Intel 82576 NIC (Gigabit ET Quad Port)
- devices eth2, eth3, eth4, eth5
Fedora 14
kernel: 2.6.35.12-88.fc14.x86_64
qemu-kvm.git, ffce28fe6 (18-April-11)

VMs:
Fedora 14
kernel 2.6.35.11-83.fc14.x86_64
2 vcpus
1GB RAM
2 NICs - 1 virtio, 1 VF

The virtio network arguments to qemu-kvm are:
-netdev type=tap,vhost=on,ifname=tap0,id=netdev1 -device virtio-net-pci,mac=${mac},netdev=netdev1

For this round of tests I have the following setup:

.======================================.

  | Host - A                             |
  |                                      |
  |      .-------------------------.     |
  |      |  Virtual Machine - C    |     |
  |      |                         |     |
  |      |  .------.     .------.  |     |
  |      '--| eth1 |-----| eth0 |--'     |
  |         '------'     '------'        |
  |   192.168. |             | 192.168.103.71
  |     102.71 |          .------.       |
  |            |          | tap0 |       |
  |            |          '------'       |
  |            |             |           |
  |            |          .------.       |
  |            |          |  br  | 192.168.103.79
  |            |          '------'       |
  |           {VF}           |           |
  |       .--------.      .------.       |
  '=======|  eth2  |======| eth3 |======='
          '--------'      '------'
192.168.102.79 |             |
               | point-to-   |
               |       point |
               | connections |
192.168.102.80 |             |
          .--------.      .------.
  .=======|  eth2  |======| eth3 |=======.
  |       '--------'      '------'       |
  |           {VF}           |           |
  |            |          .------.       |
  |            |          |  br  | 192.168.103.80
  |            |          '------'       |
  |            |             |           |
  |            |          .------.       |
  |            |          | tap0 |       |
  |   192.168. |          '------'       |
  |     102.81 |             | 192.168.103.81
  |         .------.     .------.        |
  |      .--| eth1 |-----| eth0 |--.     |
  |      |  '------'     '------'  |     |
  |      |                         |     |
  |      |  Virtual Machine - D    |     |
  |      '-------------------------'     |
  |                                      |
  | Host - B                             |

'======================================'

So, basically, 192.168.102 is the network where the VMs have a VF, and 192.168.103 is the network where the VMs use virtio for networking.

The netperf commands are all run on either Host-A or VM-C:

netperf -H $ip -jcC -v 2 -t TCP_RR -- -r 1024 -D L,R
netperf -H $ip -jcC -v 2 -t TCP_STREAM -- -m 1024 -D L,R

latency throughput
(usec) Mbps
cross-host:

  A-B, eth2          185            932
  A-B, eth3          185            935

same host, host-VM:

  A-C, using VF      488           1085 (seen as high as 1280's)
  A-C, virtio        150           4282

cross-host, host-VM:

  A-D, VF            489            938
  A-D, virtio        288            889

cross-host, VM-VM:

  C-D, VF            488            934
  C-D, virtio        490            933

While throughput for VFs is fine (near line-rate when crossing hosts),

the latency is horrible. Any options to improve that?

Quoted 9 lines

So, basically, 192.168.102 is the network where the VMs have a VF, and 192.168.103 is the network where the VMs use virtio for networking.

The netperf commands are all run on either Host-A or VM-C:

netperf -H $ip -jcC -v 2 -t TCP_RR -- -r 1024 -D L,R
netperf -H $ip -jcC -v 2 -t TCP_STREAM -- -m 1024 -D L,R

Quoted 5 lines

latency throughput
(usec) Mbps
cross-host:

  A-B, eth2          185            932
  A-B, eth3          185            935

This is actually PF-PF, right? It would be interesting to load igbvf on the hosts and determine VF-VF latency as well.

Quoted 4 lines

same host, host-VM:

  A-C, using VF      488           1085 (seen as high as 1280's)
  A-C, virtio        150           4282

We know virtio has a shorter path for this test.

Quoted 10 lines

cross-host, host-VM:

  A-D, VF            489            938
  A-D, virtio        288            889

cross-host, VM-VM:

  C-D, VF            488            934
  C-D, virtio        490            933

Quoted 1 line

While throughput for VFs is fine (near line-rate when crossing hosts),

FWIW, it's not too difficult to get line rate on a 1Gbps network, even some of the emulated NICs can do it. There will be a difference in host CPU power to get it though, where it should theoretically be emulated > virtio > pci-assign.

Quoted 2 lines

the latency is horrible. Any options to improve that?

If you don't mind testing, I'd like to see VF-VF between the hosts (to do this, don't assign eth2 an IP, just make sure it's up, then load the igbvf driver on the host and assign an IP to one of the VFs associated with the eth2 PF), and cross host testing using the PF for the guest instead of the VF. This should help narrow down how much of the latency is due to using the VF vs the PF, since all of the virtio tests are using the PF. I've been suspicious that the VF adds some latency, but haven't had a good test setup (or time) to dig very deep into it.

Thanks,

Alex

Quoted 9 lines · Alex Williamson

Quoted 9 lines

So, basically, 192.168.102 is the network where the VMs have a VF, and 192.168.103 is the network where the VMs use virtio for networking.

The netperf commands are all run on either Host-A or VM-C:

netperf -H $ip -jcC -v 2 -t TCP_RR -- -r 1024 -D L,R
netperf -H $ip -jcC -v 2 -t TCP_STREAM -- -m 1024 -D L,R

Quoted 8 lines

Quoted 5 lines

latency throughput
(usec) Mbps
cross-host:

  A-B, eth2          185            932
  A-B, eth3          185            935

This is actually PF-PF, right? It would be interesting to load igbvf on the hosts and determine VF-VF latency as well.

yes, PF-PF. eth3 has the added bridge layer, but from what I can see the overhead is noise. I added host-to-host to put the host-to-VM numbers in perspective.

Quoted 6 lines

Quoted 4 lines

same host, host-VM:

  A-C, using VF      488           1085 (seen as high as 1280's)
  A-C, virtio        150           4282

We know virtio has a shorter path for this test.

No complaints about the throughput numbers; the latency is the problem.

Quoted 10 lines

Quoted 10 lines

cross-host, host-VM:

  A-D, VF            489            938
  A-D, virtio        288            889

cross-host, VM-VM:

  C-D, VF            488            934
  C-D, virtio        490            933

Quoted 6 lines

Quoted 1 line

While throughput for VFs is fine (near line-rate when crossing hosts),

FWIW, it's not too difficult to get line rate on a 1Gbps network, even some of the emulated NICs can do it. There will be a difference in host CPU power to get it though, where it should theoretically be emulated > virtio > pci-assign.

10GB is the goal; 1GB offers a cheaper learning environment. ;-)

Quoted 11 lines

Quoted 2 lines

the latency is horrible. Any options to improve that?

If you don't mind testing, I'd like to see VF-VF between the hosts (to do this, don't assign eth2 an IP, just make sure it's up, then load the igbvf driver on the host and assign an IP to one of the VFs associated with the eth2 PF), and cross host testing using the PF for the guest instead of the VF. This should help narrow down how much of the latency is due to using the VF vs the PF, since all of the virtio tests are using the PF. I've been suspicious that the VF adds some latency, but haven't had a good test setup (or time) to dig very deep into it.

It's a quad nic, so I left eth2 and eth3 alone and added the VF-VF test using VFs on eth4.

Indeed latency is 488 usec and throughput is 925 Mbps. This is host-to-host using VFs.

So we're effectively getting host-host latency/throughput for the VF, it's just that in the 82576 implementation of SR-IOV, the VF takes a latency hit that puts it pretty close to virtio. Unfortunate. I think

you'll find that passing the PF to the guests should be pretty close to that 185us latency. I would assume (hope) the higher end NICs reduce

this, but it seems to be a hardware limitation, so it's hard to predict. Thanks,

Alex

Quoted 3 lines · Alex Williamson

So we're effectively getting host-host latency/throughput for the VF, it's just that in the 82576 implementation of SR-IOV, the VF takes a latency hit that puts it pretty close to virtio. Unfortunate. I think

For host-to-VM using VFs is worse than virtio which is counterintuitive.

Quoted 2 lines

you'll find that passing the PF to the guests should be pretty close to that 185us latency. I would assume (hope) the higher end NICs reduce

About that 185usec: do you know where the bottleneck is? It seems as if the packet is held in some queue waiting for an event/timeout before it is transmitted.

Quoted 7 lines · David Ahern

Quoted 3 lines · Alex Williamson

So we're effectively getting host-host latency/throughput for the VF, it's just that in the 82576 implementation of SR-IOV, the VF takes a latency hit that puts it pretty close to virtio. Unfortunate. I think

For host-to-VM using VFs is worse than virtio which is counterintuitive.

On the same host, just think about the data path of one versus the other. On the guest side, there's virtio vs a physical NIC. virtio is designed to be virtualization friendly, so hopefully has less context switches in setting up and processing transactions. Once the packet leaves the assigned physical NIC, it has to come back up the entire host I/O stack, while the virtio device is connected to an internal bridge and bypasses all but the upper level network routing.

Quoted 7 lines

Quoted 3 lines

you'll find that passing the PF to the guests should be pretty close to that 185us latency. I would assume (hope) the higher end NICs reduce

About that 185usec: do you know where the bottleneck is? It seems as if the packet is held in some queue waiting for an event/timeout before it is transmitted.

I don't know specifically, I don't do much network performance tuning. Interrupt coalescing could be a factor, along with various offload settings, and of course latency of the physical NIC hardware and interconnects. Thanks,

Alex

Quoted 9 lines · Alex Williamson

Quoted 7 lines · David Ahern

Quoted 3 lines · Alex Williamson

So we're effectively getting host-host latency/throughput for the VF, it's just that in the 82576 implementation of SR-IOV, the VF takes a latency hit that puts it pretty close to virtio. Unfortunate. I think

For host-to-VM using VFs is worse than virtio which is counterintuitive.

Quoted 7 lines

On the same host, just think about the data path of one versus the other. On the guest side, there's virtio vs a physical NIC. virtio is designed to be virtualization friendly, so hopefully has less context switches in setting up and processing transactions. Once the packet leaves the assigned physical NIC, it has to come back up the entire host I/O stack, while the virtio device is connected to an internal bridge and bypasses all but the upper level network routing.

I get the virtio path, but you lost me on the physical NIC. I thought the point of VFs is to bypass the host from having to touch the packet, so the processing path with a VM using a VF would be the same as a non-VM.

Quoted 19 lines · David Ahern

Quoted 9 lines · Alex Williamson

Quoted 7 lines · David Ahern

Quoted 3 lines · Alex Williamson

So we're effectively getting host-host latency/throughput for the VF, it's just that in the 82576 implementation of SR-IOV, the VF takes a latency hit that puts it pretty close to virtio. Unfortunate. I think

For host-to-VM using VFs is worse than virtio which is counterintuitive.

Quoted 7 lines

On the same host, just think about the data path of one versus the other. On the guest side, there's virtio vs a physical NIC. virtio is designed to be virtualization friendly, so hopefully has less context switches in setting up and processing transactions. Once the packet leaves the assigned physical NIC, it has to come back up the entire host I/O stack, while the virtio device is connected to an internal bridge and bypasses all but the upper level network routing.

Quoted 3 lines

I get the virtio path, but you lost me on the physical NIC. I thought the point of VFs is to bypass the host from having to touch the packet, so the processing path with a VM using a VF would be the same as a non-VM.

In the VF case, the host is only involved in processing the packet on it's end of the connection, but the packet still has to go all the way out to the physical device and all the way back. Handled on one end by the VM and the other end by the host.

An analogy might be sending a letter to an office coworker in a neighboring cube. You could just pass the letter over the wall (virtio) or you could go put it in the mailbox, signal the mail carrier, who comes and moves it to your neighbor's mailbox, who then gets signaled that they have a letter (device assignment).

Since the networks stacks are completely separate from one another, there's very little difference in data path whether you're talking to the host, a remote system, or a remote VM, which is reflected in your performance data. Hope that helps,

Alex

Quoted 35 lines · Alex Williamson

Quoted 19 lines · David Ahern

Quoted 9 lines · Alex Williamson

Quoted 7 lines · David Ahern

Quoted 3 lines · Alex Williamson

So we're effectively getting host-host latency/throughput for the VF, it's just that in the 82576 implementation of SR-IOV, the VF takes a latency hit that puts it pretty close to virtio. Unfortunate. I think

For host-to-VM using VFs is worse than virtio which is counterintuitive.

Quoted 7 lines

On the same host, just think about the data path of one versus the other. On the guest side, there's virtio vs a physical NIC. virtio is designed to be virtualization friendly, so hopefully has less context switches in setting up and processing transactions. Once the packet leaves the assigned physical NIC, it has to come back up the entire host I/O stack, while the virtio device is connected to an internal bridge and bypasses all but the upper level network routing.

Quoted 3 lines

I get the virtio path, but you lost me on the physical NIC. I thought the point of VFs is to bypass the host from having to touch the packet, so the processing path with a VM using a VF would be the same as a non-VM.

In the VF case, the host is only involved in processing the packet on it's end of the connection, but the packet still has to go all the way out to the physical device and all the way back. Handled on one end by the VM and the other end by the host.

An analogy might be sending a letter to an office coworker in a neighboring cube. You could just pass the letter over the wall (virtio) or you could go put it in the mailbox, signal the mail carrier, who comes and moves it to your neighbor's mailbox, who then gets signaled that they have a letter (device assignment).

Quoted 4 lines

Since the networks stacks are completely separate from one another, there's very little difference in data path whether you're talking to the host, a remote system, or a remote VM, which is reflected in your performance data. Hope that helps,

Got you. I was thinking host-VM as VM on separate host; I didn't make that clear. Thanks for clarifying - I like the letter example.

Quoted 3 lines · Alex Williamson

So we're effectively getting host-host latency/throughput for the VF, it's just that in the 82576 implementation of SR-IOV, the VF takes a latency hit that puts it pretty close to virtio. Unfortunate. I think

For host-to-VM using VFs is worse than virtio which is counterintuitive.

Quoted 2 lines

you'll find that passing the PF to the guests should be pretty close to that 185us latency. I would assume (hope) the higher end NICs reduce

About that 185usec: do you know where the bottleneck is? It seems as if the packet is held in some queue waiting for an event/timeout before it is transmitted.

David

Quoted 4 lines

this, but it seems to be a hardware limitation, so it's hard to predict. Thanks,

Alex

Quoted 5 lines

Quoted 3 lines

same host, host-VM:

  A-C, using VF      488           1085 (seen as high as 1280's)
  A-C, virtio        150           4282

We know virtio has a shorter path for this test.

No complaints about the throughput numbers; the latency is the problem.