virtio: Large number of tcp connections, vhost_net seems to be a bottleneck

DISCUSSION Sahid Ferdjaoui 3 participants 4 messages Oct 20, 2013 – Oct 23, 2013 1 message not in the archive KVM

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Sahid Ferdjaoui

author

2013-10-20 08:13 UTC #1 lore raw

Hi all,

I'm working on create a large number of tcp connections on a guest; The environment is on OpenStack:

Host (dedicated compute node):
OS/Kernel: Ubuntu/3.2
Cpus: 24
Mems: 128GB

Guest (alone on the Host):
OS/Kernel: Ubuntu/3.2
Cpus: 4
Mems: 32GB

Currently a guest can handle about 700 000 established connections, the cpus are not loaded and 12giga of memory are used. I'm working to understand why I can go up...

On my host, after several tests with different versions of openvswitch and with linux bridge,
It look like the process vhost_net is the only process loaded to 100% and it seems vhost_net cannot use more than 1 cpu.

I would like to get more informations about vhost_net and if there is a solution to configure it to use more than 1 cpu?

Thanks a lot,
s.

Mike Dawson

2013-10-20 15:52 UTC #2 lore raw

Quoted 15 lines · Sahid Ferdjaoui

Hi all,
I'm working on create a large number of tcp connections on a guest; The environment is on OpenStack:
Host (dedicated compute node):
   OS/Kernel: Ubuntu/3.2
   Cpus: 24
   Mems: 128GB
Guest (alone on the Host):
   OS/Kernel: Ubuntu/3.2
   Cpus: 4
   Mems: 32GB

Quoted 7 lines

Currently a guest can handle about 700 000 established connections, the cpus are not loaded and 12giga of memory are used. I'm working to understand why I can go up...
On my host, after several tests with different versions of openvswitch and with linux bridge,
It look like the process vhost_net is the only process loaded to 100% and it seems vhost_net cannot use more than 1 cpu.
I would like to get more informations about vhost_net and if there is a solution to configure it to use more than 1 cpu?

Not sure if it is relevant in your case, but the newly-released Open vSwitch 2.0 is now multi-threaded:

http://openvswitch.org/releases/NEWS-2.0.0

@martin_casado said "This is a big deal. Multi-threading provides huge performance benefits on flow setup"

https://twitter.com/martin_casado/status/390384030488616960

If you give it a try, let us know.

- Mike

Quoted 8 lines

Thanks a lot,
s.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Sahid Ferdjaoui

author

2013-10-21 08:50 UTC #3 lore raw

Thanks for your back Mike, I did some tests with ovs2.0 and the number of tcp connections up to 1 000 000 easily. So if I understand well the deal is not in relation with vhost_net or it can have a link between them?

http://i.imgur.com/kJJWtOk.png

----- Original Message -----
From: "Mike Dawson" <[email protected]>
To: "Sahid Ferdjaoui" <[email protected]>, [email protected]
Sent: Sunday, October 20, 2013 5:52:36 PM
Subject: Re: virtio: Large number of tcp connections, vhost_net seems to be a bottleneck

Quoted 15 lines · Sahid Ferdjaoui

Hi all,
I'm working on create a large number of tcp connections on a guest; The environment is on OpenStack:
Host (dedicated compute node):
   OS/Kernel: Ubuntu/3.2
   Cpus: 24
   Mems: 128GB
Guest (alone on the Host):
   OS/Kernel: Ubuntu/3.2
   Cpus: 4
   Mems: 32GB

Quoted 7 lines

Currently a guest can handle about 700 000 established connections, the cpus are not loaded and 12giga of memory are used. I'm working to understand why I can go up...
On my host, after several tests with different versions of openvswitch and with linux bridge,
It look like the process vhost_net is the only process loaded to 100% and it seems vhost_net cannot use more than 1 cpu.
I would like to get more informations about vhost_net and if there is a solution to configure it to use more than 1 cpu?

Not sure if it is relevant in your case, but the newly-released Open vSwitch 2.0 is now multi-threaded:

http://openvswitch.org/releases/NEWS-2.0.0

@martin_casado said "This is a big deal. Multi-threading provides huge performance benefits on flow setup"

https://twitter.com/martin_casado/status/390384030488616960

If you give it a try, let us know.

- Mike

Quoted 8 lines

Thanks a lot,
s.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jason Wang

2013-10-23 12:30 UTC #4 lore raw

In reply to Sahid Ferdjaoui

Quoted 15 lines · Sahid Ferdjaoui

Hi all,
I'm working on create a large number of tcp connections on a guest; The environment is on OpenStack:
Host (dedicated compute node):
OS/Kernel: Ubuntu/3.2
Cpus: 24
Mems: 128GB
Guest (alone on the Host):
OS/Kernel: Ubuntu/3.2
Cpus: 4
Mems: 32GB

Quoted 7 lines

Currently a guest can handle about 700 000 established connections, the cpus are not loaded and 12giga of memory are used. I'm working to understand why I can go up...
On my host, after several tests with different versions of openvswitch and with linux bridge,
It look like the process vhost_net is the only process loaded to 100% and it seems vhost_net cannot use more than 1 cpu.
I would like to get more informations about vhost_net and if there is a solution to configure it to use more than 1 cpu?

You can if you enable the multiqueue support for virtio-net, it can use N threads when there's N queue pairs.

See http://www.linux-kvm.org/page/Multiqueue for more information.

Quoted 6 lines

Thanks a lot,
s.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html