Multiqueue macvtap driver
Subsystems (2)
PATCH 1/1 Multiqueue macvtap driver
From: Krishna Kumar <[email protected]>
Implement multiqueue facility for macvtap driver. The idea is that a macvtap device can be opened multiple times and the fd's can be used to register eg, as backend for vhost.
Please review.
--- drivers/net/macvtap.c | 85 ++++++++++++++++++++++++++++------- include/linux/if_macvlan.h | 11 ++++ 2 files changed, 78 insertions(+), 18 deletions(-)
include/linux/if_macvlan.h +10 -1
@@ -40,6 +40,14 @@ struct macvlan_rx_stats {unsigned long rx_errors; }; +#define MIN (x, y) (((x) < (y)) ? (x) : (y)) + +/* +* Maximum times a macvtap device can be opened. This can be used to +* configure the number of receive queue, e.g. for multiqueue virtio. +*/ +#define MAX_MACVTAP_QUEUES MIN(16, NR_CPUS) +struct macvlan_dev {struct net_device *dev;struct list_head list;
@@ -50,7 +58,8 @@ struct macvlan_dev {enum macvlan_mode mode;int (*receive)(struct sk_buff *skb);int (*forward)(struct net_device *dev,struct sk_buff *skb); -struct macvtap_queue *tap; +struct macvtap_queue *taps[MAX_MACVTAP_QUEUES ]; +int numvtaps; }; static inline void macvlan_count_rx(const struct macvlan_dev *vlan,
drivers/net/macvtap.c +68 -17
142 lines not on this page · raw message
On Sat, 2010-07-31 at 19:27 +0530, Krishna Kumar wrote:
[...]
drivers/net/macvtap.c static void macvtap_put_queue(struct mac · line 165
Quoted 10 lines
drivers/net/macvtap.c +4
-1@@ -136,39 +158,68 @@ static void macvtap_put_queue(struct mac }/* -* Since we only support one queue, just dereference the pointer. +* Select a queue based on the rxq of the device on which this packet +* arrived. If the incoming device is not mq, then use our cpu number +* to select a queue. vlan->numvtaps is cached in case it changes +* during the execution of this function. */
[...]
This can result in reordering if a single-queue device's RX interrupt's CPU affinity is changed. We generally try to avoid that. You should really use or generate a flow hash. There is code for this in net/core/dev.c:get_rps_cpu() which could be factored out into a separate function.
Ben.
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
David Miller
maintainerFrom: Ben Hutchings <[email protected]>
Date: Sat, 31 Jul 2010 20:18:27 +0100
Quoted 14 lines · Krishna Kumar
[...]
Quoted 10 lines
drivers/net/macvtap.c +4
-1@@ -136,39 +158,68 @@ static void macvtap_put_queue(struct mac }/* -* Since we only support one queue, just dereference the pointer. +* Select a queue based on the rxq of the device on which this packet +* arrived. If the incoming device is not mq, then use our cpu number +* to select a queue. vlan->numvtaps is cached in case it changes +* during the execution of this function. */ [...]
Quoted 5 lines
This can result in reordering if a single-queue device's RX interrupt's CPU affinity is changed. We generally try to avoid that. You should really use or generate a flow hash. There is code for this in net/core/dev.c:get_rps_cpu() which could be factored out into a separate function.
Agreed.
Thanks Ben & Dave. A question though - which of the following is preferable for the macvtap driver:
1. Calculate flow and use that to find a queue; or
2. First check if skb_rx_queue_recorded is true and if so use that; otherwise calculate the flow as in #1.
I guess #1 is better, since packets for a single flow will go to the same queue even if they arrive on different rxqs of a mq driver. But I want to make sure.
Thanks,
- KK
David Miller <[email protected]> wrote on 08/01/2010 01:04:06 PM:
Quoted 12 lines
David Miller <[email protected]>
08/01/2010 01:04 PMTo
cc
Krishna Kumar2/India/IBM@IBMIN, [email protected],
[email protected], [email protected]
Quoted 25 lines
Subject
Re: [PATCH] Multiqueue macvtap driver
From: Ben Hutchings <[email protected]>
Date: Sat, 31 Jul 2010 20:18:27 +0100Quoted 18 lines · Krishna Kumar
[...]
drivers/net/macvtap.c static void macvtap_put_queue(struct mac · line 165
Quoted 10 lines
drivers/net/macvtap.c +4
-1@@ -136,39 +158,68 @@ static void macvtap_put_queue(struct mac }/* -* Since we only support one queue, just dereference the pointer. +* Select a queue based on the rxq of the device on which this packet +* arrived. If the incoming device is not mq, then use our cpu number +* to select a queue. vlan->numvtaps is cached in case it changes +* during the execution of this function. */ [...]
This can result in reordering if a single-queue device's RX interrupt's CPU affinity is changed. We generally try to avoid that. You should really use or generate a flow hash. There is code for this in net/core/dev.c:get_rps_cpu() which could be factored out into a
separate
Quoted 3 lines
Quoted 1 line
function.
Agreed.
Quoted 10 lines · On Mon, 2010-08-02 at 18:07 +0530, Krishna Kumar2 wrote:
Thanks Ben & Dave. A question though - which of the following is preferable for the macvtap driver:
1. Calculate flow and use that to find a queue; or
2. First check ifskb_rx_queue_recordedis true and if so use that; otherwise calculate the flow as in #1.I guess #1 is better, since packets for a single flow will go to the same queue even if they arrive on different rxqs of a mq driver. But I want to make sure.
[...]
#2 is right. Just as macvtap should provide a stable flow to RX-queue mapping, so should the drivers it interfaces with.
Ben.
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.