Skip to contentSkip to conversation

virtio-net: enable multiqueue by default

PATCH Applied Jason Wang 5 participants 11 messages Nov 25, 2016 – Nov 28, 2016

Subsystems (4)
lore
Share & tools
lore

Jason Wang

author
#0 lore raw

PATCH 1/1 virtio-net: enable multiqueue by default

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>
---
 drivers/net/virtio_net.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +7 -2
index d4ac7a6..a21d93a 100644
@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (vi->any_header_sg)
 		dev->needed_headroom = vi->hdr_len;
 
-	/* Use single tx/rx queue pair as default */
-	vi->curr_queue_pairs = 1;
+	/* Enable multiqueue by default */
+	if (num_online_cpus() >= max_queue_pairs)
+		vi->curr_queue_pairs = max_queue_pairs;
+	else
+		vi->curr_queue_pairs = num_online_cpus();
 	vi->max_queue_pairs = max_queue_pairs;
 
 	/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
@@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev)
 		goto free_unregister_netdev;
 	}
 
+	virtnet_set_affinity(vi);
+
 	/* Assume link up if device can't report link status,
 	   otherwise get link status from config. */
 	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
Signature · 1 line
2.7.4
#1 lore raw
Quoted 15 lines · Jason Wang

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>

OK at some level but all uses of num_online_cpus() like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?

Quoted 33 lines
---
 drivers/net/virtio_net.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +7 -2
index d4ac7a6..a21d93a 100644
@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (vi->any_header_sg)
 		dev->needed_headroom = vi->hdr_len;
 
-	/* Use single tx/rx queue pair as default */
-	vi->curr_queue_pairs = 1;
+	/* Enable multiqueue by default */
+	if (num_online_cpus() >= max_queue_pairs)
+		vi->curr_queue_pairs = max_queue_pairs;
+	else
+		vi->curr_queue_pairs = num_online_cpus();
 	vi->max_queue_pairs = max_queue_pairs;
 
 	/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
@@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev)
 		goto free_unregister_netdev;
 	}
 
+	virtnet_set_affinity(vi);
+
 	/* Assume link up if device can't report link status,
 	   otherwise get link status from config. */
 	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
Signature · 1 line
2.7.4
author
#2 lore raw
Quoted 23 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang
Quoted 15 lines

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Quoted 7 lines
Quoted 7 lines
Cc Hannes Frederic Sowa<[email protected]>
Cc Michael S. Tsirkin<[email protected]>
Cc Neil Horman<[email protected]>
Cc Jeremy Eder<[email protected]>
Cc Marko Myllynen<[email protected]>
Cc Maxime Coquelin<[email protected]>
Quoted 4 lines

OK at some level but all uses of num_online_cpus() like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?

Not sure I get the point, do you mean adding get/put_online_cpus()? But is it a real bug? We don't do any cpu specific things so I believe it's not necessary (unless we want to keep #queues == #vcpus magically but I don't think so). Admin need to re-configure #queues after cpu hotplug if they wish.

Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#3 lore raw
In reply to Michael S. Tsirkin
Quoted 24 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>
Quoted 5 lines

OK at some level but all uses of num_online_cpus() like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?

Isn't the fix orthogonal to this use though? That is to say, you shoudl register a hotplug notifier first, and use the handler to adjust the number of queues on hotplug add/remove?

Neil

Quoted 34 lines
Quoted 34 lines
---
 drivers/net/virtio_net.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +7 -2
index d4ac7a6..a21d93a 100644
@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (vi->any_header_sg)
 		dev->needed_headroom = vi->hdr_len;
 
-	/* Use single tx/rx queue pair as default */
-	vi->curr_queue_pairs = 1;
+	/* Enable multiqueue by default */
+	if (num_online_cpus() >= max_queue_pairs)
+		vi->curr_queue_pairs = max_queue_pairs;
+	else
+		vi->curr_queue_pairs = num_online_cpus();
 	vi->max_queue_pairs = max_queue_pairs;
 
 	/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
@@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev)
 		goto free_unregister_netdev;
 	}
 
+	virtnet_set_affinity(vi);
+
 	/* Assume link up if device can't report link status,
 	   otherwise get link status from config. */
 	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
Signature · 1 line
2.7.4
#4 lore raw
In reply to Michael S. Tsirkin
Quoted 24 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>
Quoted 5 lines

OK at some level but all uses of num_online_cpus() like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?

Isn't the fix orthogonal to this use though? That is to say, you shoudl register a hotplug notifier first, and use the handler to adjust the number of queues on hotplug add/remove?

Neil

Quoted 34 lines
Quoted 34 lines
---
 drivers/net/virtio_net.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +7 -2
index d4ac7a6..a21d93a 100644
@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (vi->any_header_sg)
 		dev->needed_headroom = vi->hdr_len;
 
-	/* Use single tx/rx queue pair as default */
-	vi->curr_queue_pairs = 1;
+	/* Enable multiqueue by default */
+	if (num_online_cpus() >= max_queue_pairs)
+		vi->curr_queue_pairs = max_queue_pairs;
+	else
+		vi->curr_queue_pairs = num_online_cpus();
 	vi->max_queue_pairs = max_queue_pairs;
 
 	/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
@@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev)
 		goto free_unregister_netdev;
 	}
 
+	virtnet_set_affinity(vi);
+
 	/* Assume link up if device can't report link status,
 	   otherwise get link status from config. */
 	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
Signature · 1 line
2.7.4
maintainer
#5 lore raw
In reply to Michael S. Tsirkin

From: "Michael S. Tsirkin" <[email protected]>
Date: Fri, 25 Nov 2016 06:43:08 +0200

Quoted 24 lines · Jason Wang
Quoted 15 lines

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>
Quoted 4 lines

OK at some level but all uses of num_online_cpus() like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?

This is more being used like a heuristic in this scenerio, and in fact I would say one would keep the code this way even once proper hotplug handlers are installed to adjust the queued dynamically if there is a desired (which is also not necessarily the case).

I really don't think this change should be held on up on this issue. So can we please make some forward progress here?

Thanks.

#6 lore raw
Quoted 37 lines · David Miller

From: "Michael S. Tsirkin" <[email protected]>
Date: Fri, 25 Nov 2016 06:43:08 +0200

Quoted 24 lines · Jason Wang
Quoted 15 lines

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>
Quoted 4 lines

OK at some level but all uses of num_online_cpus() like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?

This is more being used like a heuristic in this scenerio, and in fact I would say one would keep the code this way even once proper hotplug handlers are installed to adjust the queued dynamically if there is a desired (which is also not necessarily the case).

Quoted 5 lines

I really don't think this change should be held on up on this issue. So can we please make some forward progress here?

Thanks.

Also it might be worth noting all the other multiqueue capable ethernet devices I checked use some variation of this heuristic. Typically related to what other features are enables RSS, DCB, etc. So it at least should be familiar to folks who do hotplug cpus on bare metal boxes.

.John

Acked-by on v1 1/1
#7 lore raw
In reply to Jason Wang
Quoted 56 lines · Jason Wang

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>
---
 drivers/net/virtio_net.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +7 -2
index d4ac7a6..a21d93a 100644
@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (vi->any_header_sg)
 		dev->needed_headroom = vi->hdr_len;
 
-	/* Use single tx/rx queue pair as default */
-	vi->curr_queue_pairs = 1;
+	/* Enable multiqueue by default */
+	if (num_online_cpus() >= max_queue_pairs)
+		vi->curr_queue_pairs = max_queue_pairs;
+	else
+		vi->curr_queue_pairs = num_online_cpus();
 	vi->max_queue_pairs = max_queue_pairs;
 
 	/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
@@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev)
 		goto free_unregister_netdev;
 	}
 
+	virtnet_set_affinity(vi);
+
 	/* Assume link up if device can't report link status,
 	   otherwise get link status from config. */
 	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
Signature · 1 line
2.7.4
Acked-by on v1 1/1
#8 lore raw
In reply to Jason Wang
Quoted 56 lines · Jason Wang

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>
---
 drivers/net/virtio_net.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +7 -2
index d4ac7a6..a21d93a 100644
@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (vi->any_header_sg)
 		dev->needed_headroom = vi->hdr_len;
 
-	/* Use single tx/rx queue pair as default */
-	vi->curr_queue_pairs = 1;
+	/* Enable multiqueue by default */
+	if (num_online_cpus() >= max_queue_pairs)
+		vi->curr_queue_pairs = max_queue_pairs;
+	else
+		vi->curr_queue_pairs = num_online_cpus();
 	vi->max_queue_pairs = max_queue_pairs;
 
 	/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
@@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev)
 		goto free_unregister_netdev;
 	}
 
+	virtnet_set_affinity(vi);
+
 	/* Assume link up if device can't report link status,
 	   otherwise get link status from config. */
 	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
Signature · 1 line
2.7.4
maintainer Acked-by on v1 1/1
#9 lore raw
In reply to Jason Wang
Quoted 15 lines · Jason Wang

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>

OK I stil htink we should handle cpu hotplug better but this can be done separately.

Quoted 33 lines
---
 drivers/net/virtio_net.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +7 -2
index d4ac7a6..a21d93a 100644
@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (vi->any_header_sg)
 		dev->needed_headroom = vi->hdr_len;
 
-	/* Use single tx/rx queue pair as default */
-	vi->curr_queue_pairs = 1;
+	/* Enable multiqueue by default */
+	if (num_online_cpus() >= max_queue_pairs)
+		vi->curr_queue_pairs = max_queue_pairs;
+	else
+		vi->curr_queue_pairs = num_online_cpus();
 	vi->max_queue_pairs = max_queue_pairs;
 
 	/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
@@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev)
 		goto free_unregister_netdev;
 	}
 
+	virtnet_set_affinity(vi);
+
 	/* Assume link up if device can't report link status,
 	   otherwise get link status from config. */
 	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
Signature · 1 line
2.7.4
maintainer Applied
#10 lore raw
In reply to Jason Wang

From: Jason Wang <[email protected]>
Date: Fri, 25 Nov 2016 12:37:26 +0800

Quoted 15 lines

We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:

- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases

So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.

Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Neil Horman <[email protected]>
Cc Jeremy Eder <[email protected]>
Cc Marko Myllynen <[email protected]>
Cc Maxime Coquelin <[email protected]>

Applied, thanks Jason.