Skip to contentSkip to conversation

Multiqueue macvtap driver

PATCH In review Krishna Kumar 3 participants 5 messages Jul 31, 2010 – Aug 2, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Multiqueue macvtap driver

From: Krishna Kumar <[email protected]>

Implement multiqueue facility for macvtap driver. The idea is that a macvtap device can be opened multiple times and the fd's can be used to register eg, as backend for vhost.

Please review.

---
 drivers/net/macvtap.c      |   85 ++++++++++++++++++++++++++++-------
 include/linux/if_macvlan.h |   11 ++++
 2 files changed, 78 insertions(+), 18 deletions(-)
include/linux/if_macvlan.h +10 -1
diff -ruNp org/include/linux/if_macvlan.h new/include/linux/if_macvlan.h
@@ -40,6 +40,14 @@ struct macvlan_rx_stats {
 	unsigned long		rx_errors;
 };
 
+#define MIN(x, y)		(((x) < (y)) ? (x) : (y))
+
+/*
+ * Maximum times a macvtap device can be opened. This can be used to
+ * configure the number of receive queue, e.g. for multiqueue virtio.
+ */
+#define MAX_MACVTAP_QUEUES	MIN(16, NR_CPUS)
+
 struct macvlan_dev {
 	struct net_device	*dev;
 	struct list_head	list;
@@ -50,7 +58,8 @@ struct macvlan_dev {
 	enum macvlan_mode	mode;
 	int (*receive)(struct sk_buff *skb);
 	int (*forward)(struct net_device *dev, struct sk_buff *skb);
-	struct macvtap_queue	*tap;
+	struct macvtap_queue	*taps[MAX_MACVTAP_QUEUES];
+	int			numvtaps;
 };
 
 static inline void macvlan_count_rx(const struct macvlan_dev *vlan,
drivers/net/macvtap.c +68 -17

142 lines not on this page · raw message

#1 lore raw

On Sat, 2010-07-31 at 19:27 +0530, Krishna Kumar wrote:
[...]

drivers/net/macvtap.c static void macvtap_put_queue(struct mac · line 165

Quoted 10 lines
drivers/net/macvtap.c +4 -1
@@ -136,39 +158,68 @@ static void macvtap_put_queue(struct mac
 }
 
 /*
- * Since we only support one queue, just dereference the pointer.
+ * Select a queue based on the rxq of the device on which this packet
+ * arrived. If the incoming device is not mq, then use our cpu number
+ * to select a queue. vlan->numvtaps is cached in case it changes
+ * during the execution of this function.
  */

[...]

This can result in reordering if a single-queue device's RX interrupt's CPU affinity is changed. We generally try to avoid that. You should really use or generate a flow hash. There is code for this in net/core/dev.c:get_rps_cpu() which could be factored out into a separate function.

Ben.

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
maintainer
#2 lore raw

From: Ben Hutchings <[email protected]>
Date: Sat, 31 Jul 2010 20:18:27 +0100

Quoted 14 lines · Krishna Kumar

[...]

Quoted 10 lines
drivers/net/macvtap.c +4 -1
@@ -136,39 +158,68 @@ static void macvtap_put_queue(struct mac
 }
 
 /*
- * Since we only support one queue, just dereference the pointer.
+ * Select a queue based on the rxq of the device on which this packet
+ * arrived. If the incoming device is not mq, then use our cpu number
+ * to select a queue. vlan->numvtaps is cached in case it changes
+ * during the execution of this function.
  */

[...]

Quoted 5 lines

This can result in reordering if a single-queue device's RX interrupt's CPU affinity is changed. We generally try to avoid that. You should really use or generate a flow hash. There is code for this in net/core/dev.c:get_rps_cpu() which could be factored out into a separate function.

Agreed.

#3 lore raw

Thanks Ben & Dave. A question though - which of the following is preferable for the macvtap driver:

1. Calculate flow and use that to find a queue; or
2. First check if skb_rx_queue_recorded is true and if so use that; otherwise calculate the flow as in #1.

I guess #1 is better, since packets for a single flow will go to the same queue even if they arrive on different rxqs of a mq driver. But I want to make sure.

Thanks,

- KK

David Miller <[email protected]> wrote on 08/01/2010 01:04:06 PM:

Quoted 12 lines

David Miller <[email protected]>
08/01/2010 01:04 PM

To

[email protected]

cc

Krishna Kumar2/India/IBM@IBMIN, [email protected],
[email protected], [email protected]

Quoted 25 lines

Subject

Re: [PATCH] Multiqueue macvtap driver

From: Ben Hutchings <[email protected]>
Date: Sat, 31 Jul 2010 20:18:27 +0100

Quoted 18 lines · Krishna Kumar

[...]

drivers/net/macvtap.c static void macvtap_put_queue(struct mac · line 165

Quoted 10 lines
drivers/net/macvtap.c +4 -1
@@ -136,39 +158,68 @@ static void macvtap_put_queue(struct mac
 }
 
 /*
- * Since we only support one queue, just dereference the pointer.
+ * Select a queue based on the rxq of the device on which this packet
+ * arrived. If the incoming device is not mq, then use our cpu number
+ * to select a queue. vlan->numvtaps is cached in case it changes
+ * during the execution of this function.
  */

[...]

This can result in reordering if a single-queue device's RX interrupt's CPU affinity is changed. We generally try to avoid that. You should really use or generate a flow hash. There is code for this in net/core/dev.c:get_rps_cpu() which could be factored out into a

separate

Quoted 3 lines
Quoted 1 line

function.

Agreed.

#4 lore raw
Quoted 10 lines · On Mon, 2010-08-02 at 18:07 +0530, Krishna Kumar2 wrote:

Thanks Ben & Dave. A question though - which of the following is preferable for the macvtap driver:

1. Calculate flow and use that to find a queue; or
2. First check if skb_rx_queue_recorded is true and if so use that; otherwise calculate the flow as in #1.

I guess #1 is better, since packets for a single flow will go to the same queue even if they arrive on different rxqs of a mq driver. But I want to make sure.

[...]

#2 is right. Just as macvtap should provide a stable flow to RX-queue mapping, so should the drivers it interfaces with.

Ben.

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.