vhost + multiqueue + RSS question.

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

--
Gleb.

Hi Michael,

I am playing with vhost multiqueue capability and have a question
about vhost multiqueue and RSS (receive side steering). My setup has Mellanox
ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

--
Gleb.

Hi Michael,

I am playing with vhost multiqueue capability and have a question
about vhost multiqueue and RSS (receive side steering). My setup has Mellanox
ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

--
Gleb.

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors? To see, can you try dumping struct kvm_irqfd that's passed to kvm?

Quoted 2 lines

--
Gleb.

I have a question related this topic. So How do you set the RSS Key on the Mellanox NIc? I mean from your Guest?

If it being set as part of Host driver, is there a way to set it from Guest? I mean my guest will choose a RSS Key and will try to set on the Physical NIC.

Thanks,
Venkatesh

Quoted 24 lines · Michael S. Tsirkin

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors? To see, can you try dumping struct kvm_irqfd that's passed to kvm?

Quoted 2 lines

Quoted 2 lines

--
Gleb.

This sounds like a regression, which kernel/qemu version did you use?

Quoted 24 lines · Michael S. Tsirkin

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors? To see, can you try dumping struct kvm_irqfd that's passed to kvm?

Quoted 2 lines

Quoted 2 lines

--
Gleb.

This sounds like a regression, which kernel/qemu version did you use?

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

Quoted 20 lines · Michael S. Tsirkin

Quoted 18 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

Quoted 2 lines

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

--
Gleb.

Quoted 23 lines · Gleb Natapov

Quoted 22 lines · Michael S. Tsirkin

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

Quoted 4 lines

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

Quoted 23 lines · Gleb Natapov

Quoted 22 lines · Michael S. Tsirkin

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

Quoted 4 lines

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

Quoted 29 lines · Michael S. Tsirkin

Quoted 27 lines · Gleb Natapov

Quoted 20 lines · Michael S. Tsirkin

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

Quoted 2 lines

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

--
Gleb.

Quoted 29 lines · Michael S. Tsirkin

Quoted 27 lines · Gleb Natapov

Quoted 20 lines · Michael S. Tsirkin

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

Quoted 2 lines

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

--
Gleb.

Quoted 30 lines · Gleb Natapov

Quoted 29 lines · Michael S. Tsirkin

Quoted 27 lines · Gleb Natapov

Quoted 20 lines · Michael S. Tsirkin

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

Quoted 2 lines

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 7 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Yes, what virtio has is not RSS - it's an accelerated RFS really.

Quoted 30 lines · Gleb Natapov

Quoted 29 lines · Michael S. Tsirkin

Quoted 27 lines · Gleb Natapov

Quoted 20 lines · Michael S. Tsirkin

Quoted 10 lines · Gleb Natapov

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

Quoted 2 lines

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 7 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Yes, what virtio has is not RSS - it's an accelerated RFS really.

Quoted 58 lines · Gleb Natapov

Quoted 30 lines · Michael S. Tsirkin

Quoted 29 lines · Gleb Natapov

Quoted 27 lines · Michael S. Tsirkin

Quoted 20 lines · Gleb Natapov

Quoted 10 lines

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

Quoted 8 lines

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

Quoted 2 lines

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 7 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Quoted 2 lines

Yes, what virtio has is not RSS - it's an accelerated RFS really.

OK, if what virtio has is RFS and not RSS my test results make sense. Thanks!

Quoted 30 lines · Gleb Natapov

Quoted 30 lines · Michael S. Tsirkin

Quoted 29 lines

Quoted 29 lines · Gleb Natapov

Quoted 27 lines

Quoted 27 lines · Michael S. Tsirkin

Quoted 20 lines

Quoted 20 lines · Gleb Natapov

Quoted 18 lines

Quoted 18 lines

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

Quoted 2 lines

Quoted 2 lines

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Yes, what virtio has is not RSS - it's an accelerated RFS really.

The point is to try and take application locality into account.

Quoted 56 lines · Michael S. Tsirkin

Quoted 55 lines · Gleb Natapov

Quoted 28 lines · Michael S. Tsirkin

Quoted 27 lines · Gleb Natapov

Quoted 26 lines · Michael S. Tsirkin

Quoted 13 lines · Gleb Natapov

Quoted 12 lines

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

Quoted 8 lines

Quoted 6 lines

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 7 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Yes, what virtio has is not RSS - it's an accelerated RFS really.

Quoted 2 lines

OK, if what virtio has is RFS and not RSS my test results make sense. Thanks!

I think the RSS emulation for virtio-mq NIC is implemented in tun_select_queue(), am I missing something?

Thanks,
Zhang Haoyu

Quoted 56 lines · Zhang Haoyu

Quoted 56 lines · Michael S. Tsirkin

Quoted 55 lines · Gleb Natapov

Quoted 28 lines · Michael S. Tsirkin

Quoted 27 lines · Gleb Natapov

Quoted 26 lines · Michael S. Tsirkin

Quoted 13 lines · Gleb Natapov

Quoted 12 lines

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

Quoted 8 lines

Quoted 6 lines

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 7 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Yes, what virtio has is not RSS - it's an accelerated RFS really.

Quoted 8 lines

Quoted 2 lines

OK, if what virtio has is RFS and not RSS my test results make sense. Thanks!

I think the RSS emulation for virtio-mq NIC is implemented in tun_select_queue(), am I missing something?

Thanks,
Zhang Haoyu

Yes, if RSS is the short for Receive Side Steering which is a generic technology. But RSS is usually short for Receive Side Scaling which was commonly technology used by Windows, it was implemented through a indirection table in the card which is obviously not supported in tun currently.

Quoted 56 lines · Zhang Haoyu

Quoted 56 lines · Michael S. Tsirkin

Quoted 55 lines · Gleb Natapov

Quoted 28 lines · Michael S. Tsirkin

Quoted 27 lines · Gleb Natapov

Quoted 26 lines · Michael S. Tsirkin

Quoted 13 lines · Gleb Natapov

Quoted 12 lines

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

Quoted 8 lines

Quoted 6 lines

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 7 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Yes, what virtio has is not RSS - it's an accelerated RFS really.

Quoted 8 lines

Quoted 2 lines

OK, if what virtio has is RFS and not RSS my test results make sense. Thanks!

I think the RSS emulation for virtio-mq NIC is implemented in tun_select_queue(), am I missing something?

Thanks,
Zhang Haoyu

Yes, if RSS is the short for Receive Side Steering which is a generic technology. But RSS is usually short for Receive Side Scaling which was commonly technology used by Windows, it was implemented through a indirection table in the card which is obviously not supported in tun currently.

Quoted 55 lines · Michael S. Tsirkin

Quoted 30 lines · Gleb Natapov

Quoted 30 lines · Michael S. Tsirkin

Quoted 29 lines

Quoted 29 lines · Gleb Natapov

Quoted 27 lines

Quoted 27 lines · Michael S. Tsirkin

Quoted 20 lines

Quoted 20 lines · Gleb Natapov

Quoted 18 lines

Quoted 18 lines

Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox ConnectX-3 NIC which supports multiqueue and RSS. Network related parameters for qemu are:

   -netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
   -device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is only one tcp stream I expect it to be handled by one queue only but this seams to be not the case. ethtool -S on a host shows that the stream is handled by one queue in the NIC, just like I would expect, but in a guest all 4 virtio-input interrupt are incremented. Am I missing any configuration?

Quoted 2 lines

Quoted 2 lines

I don't see anything obviously wrong with what you describe. Maybe, somehow, same irqfd got bound to multiple MSI vectors?

It does not look like this is what is happening judging by the way interrupts are distributed between queues. They are not distributed uniformly and often I see one queue gets most interrupt and others get much less and then it changes.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs. You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Quoted 1 line

Yes, what virtio has is not RSS - it's an accelerated RFS really.

Strictly speaking, not aRFS. aRFS requires a programmable filter and needs driver to fill the filter on demand. For virtio-net, this is done automatically in host side (tun/tap). There's no guest involvement.

Quoted 3 lines

The point is to try and take application locality into account.

Yes, the locality was done through (consider a N vcpu guest with N queue):

- virtio-net driver will provide a default 1:1 mapping between vcpu and txq through XPS
- virtio-net driver will suggest a default irq affinity hint also for a 1:1 mapping bettwen vcpu and txq/rxq

With all these, each vcpu get its private txq/rxq paris. And host side implementation (tun/tap) will make sure if the packets of a flow were received from queue N, if will also use queue N to transmit the packets of this flow to guest.

Quoted 56 lines · Jason Wang

Quoted 56 lines · Michael S. Tsirkin

Quoted 55 lines

Quoted 55 lines · Gleb Natapov

Quoted 30 lines

Quoted 30 lines

Quoted 30 lines

Quoted 30 lines · Michael S. Tsirkin

Quoted 29 lines

Quoted 29 lines

Quoted 29 lines

Quoted 29 lines · Gleb Natapov

On Sun, Nov 16, 2014 at 08:56:04PM +0200, Michael S. Tsirkin wrote:
On Sun, Nov 16, 2014 at 06:18:18PM +0200, Gleb Natapov wrote:
Hi Michael,

I am playing with vhost multiqueue capability and have a question about
vhost multiqueue and RSS (receive side steering). My setup has Mellanox
ConnectX-3 NIC which supports multiqueue and RSS. Network related
parameters for qemu are:

-netdev tap,id=hn0,script=qemu-ifup.sh,vhost=on,queues=4
-device virtio-net-pci,netdev=hn0,id=nic1,mq=on,vectors=10

In a guest I ran "ethtool -L eth0 combined 4" to enable multiqueue.

I am running one tcp stream into the guest using iperf. Since there is
only one tcp stream I expect it to be handled by one queue only but
this seams to be not the case. ethtool -S on a host shows that the
stream is handled by one queue in the NIC, just like I would expect,
but in a guest all 4 virtio-input interrupt are incremented. Am I
missing any configuration?

I don't see anything obviously wrong with what you describe.
Maybe, somehow, same irqfd got bound to multiple MSI vectors?
It does not look like this is what is happening judging by the way
interrupts are distributed between queues. They are not distributed
uniformly and often I see one queue gets most interrupt and others get
much less and then it changes.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Weird. It would happen if you transmitted from multiple CPUs.
You did pin iperf to a single CPU within guest, did you not?

No, I didn't because I didn't expect it to matter for input interrupts. When I run iperf on a host rx queue that receives all packets depends only on a connection itself, not on a cpu iperf is running on (I tested that).

This really depends on the type of networking card you have on the host, and how it's configured.

I think you will get something more closely resembling this behaviour if you enable RFS in host.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

When I pin iperf in a guest I do indeed see that all interrupts are arriving to the same irq vector. Is a number after virtio-input in /proc/interrupt any indication of a queue a packet arrived to (on a host I can use ethtool -S to check what queue receives packets, but unfortunately this does not work for virtio nic in a guest)?

I think it is.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Because if
it is the way RSS works in virtio is not how it works on a host and not what I would expect after reading about RSS. The queue a packets arrives to should be calculated by hashing fields from a packet header only.

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

Quoted 1 line

Yes, what virtio has is not RSS - it's an accelerated RFS really.

Strictly speaking, not aRFS. aRFS requires a programmable filter and needs driver to fill the filter on demand. For virtio-net, this is done automatically in host side (tun/tap). There's no guest involvement.

Well guest affects the filter by sending tx packets.