virtio-net: enable multiqueue by default
Subsystems (4)
PATCH 1/1 virtio-net: enable multiqueue by default
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the cases
So this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
--- drivers/net/virtio_net.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +7 -2
@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->any_header_sg) dev->needed_headroom = vi->hdr_len; -/* Use single tx/rx queue pair as default */ - vi->curr_queue_pairs =1 ; +/* Enable multiqueue by default */ +if (num_online_cpus () >= max_queue_pairs) + vi->curr_queue_pairs = max_queue_pairs; +else + vi->curr_queue_pairs =num_online_cpus (); vi->max_queue_pairs = max_queue_pairs;/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
@@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev) goto free_unregister_netdev; } +virtnet_set_affinity (vi); +/* Assume link up if device can't report link status, otherwise get link status from config. */ if (virtio_has_feature (vi->vdev,VIRTIO_NET_F_STATUS )) {
Signature · 1 line
2.7.4
Michael S. Tsirkin
maintainerQuoted 15 lines · Jason Wang
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>
OK at some level but all uses of num_online_cpus() like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?
Quoted 33 lines
--- drivers/net/virtio_net.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +7
-2@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->any_header_sg) dev->needed_headroom = vi->hdr_len; -/* Use single tx/rx queue pair as default */ - vi->curr_queue_pairs =1 ; +/* Enable multiqueue by default */ +if (num_online_cpus () >= max_queue_pairs) + vi->curr_queue_pairs = max_queue_pairs; +else + vi->curr_queue_pairs =num_online_cpus (); vi->max_queue_pairs = max_queue_pairs;/* Allocate/initialize the rx/tx queues, and invoke find_vqs */ @@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev) goto free_unregister_netdev; } +virtnet_set_affinity (vi); +/* Assume link up if device can't report link status, otherwise get link status from config. */ if (virtio_has_feature (vi->vdev,VIRTIO_NET_F_STATUS )) {Signature · 1 line
2.7.4
Quoted 23 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang
Quoted 15 lines
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Quoted 7 lines
Quoted 7 lines
Cc Hannes Frederic Sowa<[email protected]>Cc Michael S. Tsirkin<[email protected]>Cc Neil Horman<[email protected]>Cc Jeremy Eder<[email protected]>Cc Marko Myllynen<[email protected]>Cc Maxime Coquelin<[email protected]>Signed-off-by Jason Wang<[email protected]>
Quoted 4 lines
OK at some level but all uses of
num_online_cpus()like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?
Not sure I get the point, do you mean adding get/put_online_cpus()? But is it a real bug? We don't do any cpu specific things so I believe it's not necessary (unless we want to keep #queues == #vcpus magically but I don't think so). Admin need to re-configure #queues after cpu hotplug if they wish.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 24 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>
Quoted 5 lines
OK at some level but all uses of
num_online_cpus()like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?
Isn't the fix orthogonal to this use though? That is to say, you shoudl register a hotplug notifier first, and use the handler to adjust the number of queues on hotplug add/remove?
Neil
Quoted 34 lines
Quoted 34 lines
--- drivers/net/virtio_net.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +7
-2@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->any_header_sg) dev->needed_headroom = vi->hdr_len; -/* Use single tx/rx queue pair as default */ - vi->curr_queue_pairs =1 ; +/* Enable multiqueue by default */ +if (num_online_cpus () >= max_queue_pairs) + vi->curr_queue_pairs = max_queue_pairs; +else + vi->curr_queue_pairs =num_online_cpus (); vi->max_queue_pairs = max_queue_pairs;/* Allocate/initialize the rx/tx queues, and invoke find_vqs */ @@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev) goto free_unregister_netdev; } +virtnet_set_affinity (vi); +/* Assume link up if device can't report link status, otherwise get link status from config. */ if (virtio_has_feature (vi->vdev,VIRTIO_NET_F_STATUS )) {Signature · 1 line
2.7.4
Quoted 24 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>
Quoted 5 lines
OK at some level but all uses of
num_online_cpus()like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?
Isn't the fix orthogonal to this use though? That is to say, you shoudl register a hotplug notifier first, and use the handler to adjust the number of queues on hotplug add/remove?
Neil
Quoted 34 lines
Quoted 34 lines
--- drivers/net/virtio_net.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +7
-2@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->any_header_sg) dev->needed_headroom = vi->hdr_len; -/* Use single tx/rx queue pair as default */ - vi->curr_queue_pairs =1 ; +/* Enable multiqueue by default */ +if (num_online_cpus () >= max_queue_pairs) + vi->curr_queue_pairs = max_queue_pairs; +else + vi->curr_queue_pairs =num_online_cpus (); vi->max_queue_pairs = max_queue_pairs;/* Allocate/initialize the rx/tx queues, and invoke find_vqs */ @@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev) goto free_unregister_netdev; } +virtnet_set_affinity (vi); +/* Assume link up if device can't report link status, otherwise get link status from config. */ if (virtio_has_feature (vi->vdev,VIRTIO_NET_F_STATUS )) {Signature · 1 line
2.7.4
David Miller
maintainerFrom: "Michael S. Tsirkin" <[email protected]>
Date: Fri, 25 Nov 2016 06:43:08 +0200
Quoted 24 lines · Jason Wang
Quoted 15 lines
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>
Quoted 4 lines
OK at some level but all uses of
num_online_cpus()like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?
This is more being used like a heuristic in this scenerio, and in fact I would say one would keep the code this way even once proper hotplug handlers are installed to adjust the queued dynamically if there is a desired (which is also not necessarily the case).
I really don't think this change should be held on up on this issue. So can we please make some forward progress here?
Thanks.
Quoted 37 lines · David Miller
From: "Michael S. Tsirkin" <[email protected]>
Date: Fri, 25 Nov 2016 06:43:08 +0200Quoted 24 lines · Jason Wang
Quoted 15 lines
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>Quoted 4 lines
OK at some level but all uses of
num_online_cpus()like this are racy versus hotplug.
I know we already have this bug but shouldn't we fix it before we add more?This is more being used like a heuristic in this scenerio, and in fact I would say one would keep the code this way even once proper hotplug handlers are installed to adjust the queued dynamically if there is a desired (which is also not necessarily the case).
Quoted 5 lines
I really don't think this change should be held on up on this issue. So can we please make some forward progress here?
Thanks.
Also it might be worth noting all the other multiqueue capable ethernet devices I checked use some variation of this heuristic. Typically related to what other features are enables RSS, DCB, etc. So it at least should be familiar to folks who do hotplug cpus on bare metal boxes.
.John
Quoted 56 lines · Jason Wang
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>--- drivers/net/virtio_net.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +7
-2@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->any_header_sg) dev->needed_headroom = vi->hdr_len; -/* Use single tx/rx queue pair as default */ - vi->curr_queue_pairs =1 ; +/* Enable multiqueue by default */ +if (num_online_cpus () >= max_queue_pairs) + vi->curr_queue_pairs = max_queue_pairs; +else + vi->curr_queue_pairs =num_online_cpus (); vi->max_queue_pairs = max_queue_pairs;/* Allocate/initialize the rx/tx queues, and invoke find_vqs */ @@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev) goto free_unregister_netdev; } +virtnet_set_affinity (vi); +/* Assume link up if device can't report link status, otherwise get link status from config. */ if (virtio_has_feature (vi->vdev,VIRTIO_NET_F_STATUS )) {Signature · 1 line
2.7.4
Quoted 56 lines · Jason Wang
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>--- drivers/net/virtio_net.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +7
-2@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->any_header_sg) dev->needed_headroom = vi->hdr_len; -/* Use single tx/rx queue pair as default */ - vi->curr_queue_pairs =1 ; +/* Enable multiqueue by default */ +if (num_online_cpus () >= max_queue_pairs) + vi->curr_queue_pairs = max_queue_pairs; +else + vi->curr_queue_pairs =num_online_cpus (); vi->max_queue_pairs = max_queue_pairs;/* Allocate/initialize the rx/tx queues, and invoke find_vqs */ @@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev) goto free_unregister_netdev; } +virtnet_set_affinity (vi); +/* Assume link up if device can't report link status, otherwise get link status from config. */ if (virtio_has_feature (vi->vdev,VIRTIO_NET_F_STATUS )) {Signature · 1 line
2.7.4
Quoted 15 lines · Jason Wang
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>
OK I stil htink we should handle cpu hotplug better but this can be done separately.
Quoted 33 lines
--- drivers/net/virtio_net.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +7
-2@@ -1886,8 +1886,11 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->any_header_sg) dev->needed_headroom = vi->hdr_len; -/* Use single tx/rx queue pair as default */ - vi->curr_queue_pairs =1 ; +/* Enable multiqueue by default */ +if (num_online_cpus () >= max_queue_pairs) + vi->curr_queue_pairs = max_queue_pairs; +else + vi->curr_queue_pairs =num_online_cpus (); vi->max_queue_pairs = max_queue_pairs;/* Allocate/initialize the rx/tx queues, and invoke find_vqs */ @@ -1918,6 +1921,8 @@ static int virtnet_probe(struct virtio_device *vdev) goto free_unregister_netdev; } +virtnet_set_affinity (vi); +/* Assume link up if device can't report link status, otherwise get link status from config. */ if (virtio_has_feature (vi->vdev,VIRTIO_NET_F_STATUS )) {Signature · 1 line
2.7.4
From: Jason Wang <[email protected]>
Date: Fri, 25 Nov 2016 12:37:26 +0800
Quoted 15 lines
We use single queue even if multiqueue is enabled and let admin to enable it through ethtool later. This is used to avoid possible regression (small packet TCP stream transmission). But looks like an overkill since:
- single queue user can disable multiqueue when launching qemu
- brings extra troubles for the management since it needs extra admin tool in guest to enable multiqueue
- multiqueue performs much better than single queue in most of the casesSo this patch enables multiqueue by default: if #queues is less than or equal to #vcpu, enable as much as queue pairs; if #queues is greater than #vcpu, enable #vcpu queue pairs.
Quoted 7 lines
Cc Hannes Frederic Sowa <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Neil Horman <[email protected]>Cc Jeremy Eder <[email protected]>Cc Marko Myllynen <[email protected]>Cc Maxime Coquelin <[email protected]>Signed-off-by Jason Wang <[email protected]>
Applied, thanks Jason.