Skip to contentSkip to conversation

RFS hardware acceleration

RFC PATCH In review Ben Hutchings 3 participants 12 messages Sep 20, 2010 – Sep 22, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover RFS hardware acceleration

Tom,

This patch series extends RFS to use hardware RX filters where available. Depending on the number of hardware RX queues and their IRQs' affinity, this should reduce the need for IPIs or at least get packets delivered to the right NUMA node.

I've implemented the driver side of this for our hardware, though I don't know whether you have any of that to test on. I would be very interested to know how much this can help in the sort of cases where you use RFS.

Ben.

Ben Hutchings (4):
IRQ: IRQ groups for multiqueue devices
net: RPS: Enable hardware acceleration
sfc: Implement RFS acceleration
sfc/RFS/irq_group debug output

 drivers/net/sfc/efx.c     |   49 +++++++++++---
 drivers/net/sfc/efx.h     |    9 +++
 drivers/net/sfc/filter.c  |  109 +++++++++++++++++++++++++++++
 include/linux/irq.h       |   52 ++++++++++++++
 include/linux/netdevice.h |   29 +++++++-
 kernel/irq/manage.c       |  170 +++++++++++++++++++++++++++++++++++++++++++++
 net/core/dev.c            |   88 ++++++++++++++++++++++--
 7 files changed, 488 insertions(+), 18 deletions(-)
Signature · 7 lines
1.7.2.1


-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#1 lore raw

RFC PATCH 1/4 IRQ: IRQ groups for multiqueue devices

When initiating I/O on multiqueue devices, we usually want to select a queue for which the response will be handled on the same or a nearby CPU. IRQ groups hold a mapping of CPU to IRQ which will be updated based on the inverse of IRQ CPU-affinities plus CPU topology information.

---
 include/linux/irq.h |   52 ++++++++++++++++++
 kernel/irq/manage.c |  149 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 201 insertions(+), 0 deletions(-)
include/linux/irq.h +52 -0
index c03243a..bbddd5f 100644
@@ -196,6 +196,8 @@ struct irq_desc {
 #ifdef CONFIG_SMP
 	cpumask_var_t		affinity;
 	const struct cpumask	*affinity_hint;
+	struct irq_group	*group;
+	u16			group_index;
 	unsigned int		node;
 #ifdef CONFIG_GENERIC_PENDING_IRQ
 	cpumask_var_t		pending_mask;
@@ -498,6 +500,33 @@ static inline void free_desc_masks(struct irq_desc *old_desc,
 #endif
 }
 
+/**
+ * struct irq_group - IRQ group for multiqueue devices
+ * @closest: For each CPU, the index and distance to the closest IRQ,
+ *	based on affinity masks
+ * @size: Size of the group
+ * @used: Number of IRQs currently included in the group
+ * @irq: Descriptors for IRQs in the group
+ */
+struct irq_group {
+	struct {
+		u16	index;
+		u16	dist;
+	} closest[NR_CPUS];
+	unsigned int	size, used;
+	struct irq_desc *irq[0];
+};
+#define IRQ_CPU_DIST_INF 0xffff
+
+extern struct irq_group *alloc_irq_group(unsigned int size, gfp_t flags);
+extern void free_irq_group(struct irq_group *group);
+extern void irq_group_add(struct irq_group *group, unsigned int irq);
+
+static inline u16 irq_group_get_index(struct irq_group *group, int cpu)
+{
+	return group->closest[cpu].index;
+}
+
 #else /* !CONFIG_SMP */
 
 static inline bool alloc_desc_masks(struct irq_desc *desc, int node,
@@ -519,6 +548,29 @@ static inline void free_desc_masks(struct irq_desc *old_desc,
 				   struct irq_desc *new_desc)
 {
 }
+
+struct irq_group {
+};
+
+static inline struct irq_group *alloc_irq_group(unsigned int size, gfp_t flags)
+{
+	static struct irq_group dummy;
+	return &dummy;
+}
+
+static inline void free_irq_group(struct irq_group *group)
+{
+}
+
+static inline void irq_group_add(struct irq_group *group, unsigned int irq)
+{
+}
+
+static inline u16 irq_group_get_index(struct irq_group *group, int cpu)
+{
+	return 0;
+}
+
 #endif	/* CONFIG_SMP */
 
 #endif /* _LINUX_IRQ_H */
kernel/irq/manage.c +149 -0

161 lines not on this page · raw message

Signature · 8 lines
1.7.2.1



-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#2 lore raw

Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices

Ben,

On Mon, 20 Sep 2010, Ben Hutchings wrote:

it would have been nice if I'd been cc'ed on that, but of course it's my fault that there is no entry in MAINTAINERS. No, it's not.

Quoted 1 line

When initiating I/O on multiqueue devices, we usually want to select a

"we usually" is pretty useless for people not familiar with the problem at hand. A [PATCH 0/4] intro would have been helpful along with the users (aka [PATCH 2-4/4]) of the new facility.

Don't take it personally and I'm sure that you're solving a real world problem, but I'm starting to get grumpy that especially networking folks (aside of various driver developers) believe that adding random stuff to kernel/irq is their private pleasure. Is it that hard to talk to me upfront ?

Quoted 4 lines

queue for which the response will be handled on the same or a nearby CPU. IRQ groups hold a mapping of CPU to IRQ which will be updated based on the inverse of IRQ CPU-affinities plus CPU topology information.

Can you please explain, why you need that reverse mapping including the below code ? What problem does this solve which can not be deduced by the exisiting information/infrastructure ? And why is that reverse mapping tied to interrupts and not something which we want to see in some generic available (library) code ?

More comments inline.

include/linux/irq.h struct irq_desc { · line 198

Quoted 13 lines
---
 include/linux/irq.h |   52 ++++++++++++++++++
 kernel/irq/manage.c |  149 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 201 insertions(+), 0 deletions(-)
include/linux/irq.h +0 -0
index c03243a..bbddd5f 100644
@@ -196,6 +196,8 @@ struct irq_desc {
 #ifdef CONFIG_SMP
 	cpumask_var_t		affinity;
 	const struct cpumask	*affinity_hint;

How about updating the kernel doc above the struct ?

include/linux/irq.h struct irq_desc { · line 199

Quoted 1 line
include/linux/irq.h +1 -0
+	struct irq_group	*group;

Grr, how does this compile ? That needs at least a forward declaration of struct irq_group. RFC is _NOT_ an excuse

include/linux/irq.h struct irq_desc { · line 200

Quoted 1 line
include/linux/irq.h +1 -0
+	u16			group_index;

What's group_index doing and what's the point of an u16 here ?

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 506

Quoted 11 lines
include/linux/irq.h +4 -0
 	unsigned int		node;
 #ifdef CONFIG_GENERIC_PENDING_IRQ
 	cpumask_var_t		pending_mask;
@@ -498,6 +500,33 @@ static inline void free_desc_masks(struct irq_desc *old_desc,
 #endif
 }
 
+/**
+ * struct irq_group - IRQ group for multiqueue devices
+ * @closest: For each CPU, the index and distance to the closest IRQ,
+ *	based on affinity masks

index of what ?

Btw, please follow the style of the other kernel doc comments in this file where the explanation is aligned for readability sake

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 509

Quoted 3 lines
include/linux/irq.h +3 -0
+ * @size: Size of the group
+ * @used: Number of IRQs currently included in the group
+ * @irq: Descriptors for IRQs in the group

That's an array of pointers to irq descriptors, right ?

Insert some sensible decription what irq groups are and for which problem space they are useful if at all.

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 516

Quoted 7 lines
include/linux/irq.h +7 -0
+ */
+struct irq_group {
+	struct {
+		u16	index;
+		u16	dist;
+	} closest[NR_CPUS];
+	unsigned int	size, used;

Separate lines please

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 518

Quoted 2 lines
include/linux/irq.h +2 -0
+	struct irq_desc *irq[0];
+};

Please separate this with a newline and add some useful comment about the meaning and purpose of this not selfexplaining constant.

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 529

Quoted 7 lines
include/linux/irq.h +6 -0
+#define IRQ_CPU_DIST_INF 0xffff
 
+static inline u16 irq_group_get_index(struct irq_group *group, int cpu)
+{
+	return group->closest[cpu].index;
+}
+

So you have an accessor function for closest[cpu].index. Are the other members meant to be accessible directly by random driver ?

Quoted 4 lines
include/linux/irq.h +0 -0
 #else /* !CONFIG_SMP */
 
 static inline bool alloc_desc_masks(struct irq_desc *desc, int node,

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 557

Quoted 10 lines
include/linux/irq.h +7 -0
@@ -519,6 +548,29 @@ static inline void free_desc_masks(struct irq_desc *old_desc,
 				   struct irq_desc *new_desc)
 {
 }
+
+struct irq_group {
+};
+
+static inline struct irq_group *alloc_irq_group(unsigned int size, gfp_t flags)
+{
+	static struct irq_group dummy;

That will create one static instance per callsite. Is that on purpose? If yes, it needs a damned good comment.

Quoted 16 lines
include/linux/irq.h +1 -0
+	return &dummy;
 
 #endif	/* CONFIG_SMP */
 
 #endif /* _LINUX_IRQ_H */
kernel/irq/manage.c +3 -0
index c3003e9..3f2b1a9 100644
@@ -100,6 +100,154 @@ void irq_set_thread_affinity(struct irq_desc *desc)
 	}
 }
 
+static void irq_group_update_neigh(struct irq_group *group,
+				   const struct cpumask *mask,
+				   u16 index, u16 dist)

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 115

Quoted 10 lines
kernel/irq/manage.c +10 -0
+{
+	int cpu;
+
+	for_each_cpu(cpu, mask) {
+		if (dist < group->closest[cpu].dist) {
+			group->closest[cpu].index = index;
+			group->closest[cpu].dist = dist;
+		}
+	}
+}

I have not the faintest idea how group, index and dist are related to each other. And I have no intention to decode the information about that piecewise by reverse engineering that obfuscated code.

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 129

Quoted 13 lines
kernel/irq/manage.c +13 -0
+static bool irq_group_copy_neigh(struct irq_group *group, int cpu,
+				 const struct cpumask *mask, u16 dist)
+{
+	int neigh;
+
+	for_each_cpu(neigh, mask) {
+		if (group->closest[neigh].dist <= dist) {
+			group->closest[cpu].index = group->closest[neigh].index;
+			group->closest[cpu].dist = dist;
+			return true;
+		}
+	}
+	return false;

What's the reason for copying or not ?

Quoted 42 lines
kernel/irq/manage.c +42 -0
+}
+
+/* Update the per-CPU closest IRQs following a change of affinity */
+static void
+irq_update_group(struct irq_desc *desc, const struct cpumask *affinity)
+{
+	struct irq_group *group = desc->group;
+	unsigned index = desc->group_index;
+	int cpu;
+
+	if (!group)
+		return;
+
+	/* Invalidate old distances to this IRQ */
+	for_each_online_cpu(cpu)
+		if (group->closest[cpu].index == index)
+			group->closest[cpu].dist = IRQ_CPU_DIST_INF;
+
+	/*
+	 * Set this as the closest IRQ for all CPUs in the affinity mask,
+	 * plus the following CPUs if they don't have a closer IRQ:
+	 * - all other threads in the same core (distance 1);
+	 * - all other cores in the same package (distance 2);
+	 * - all other packages in the same NUMA node (distance 3).
+	 */
+	for_each_cpu(cpu, affinity) {
+		group->closest[cpu].index = index;
+		group->closest[cpu].dist = 0;
+		irq_group_update_neigh(group, topology_thread_cpumask(cpu),
+				       index, 1);
+		irq_group_update_neigh(group, topology_core_cpumask(cpu),
+				       index, 2);
+		irq_group_update_neigh(group, cpumask_of_node(cpu_to_node(cpu)),
+				       index, 3);
+	}
+
+	/* Find new closest IRQ for any CPUs left with invalid distances */
+	for_each_online_cpu(cpu) {
+		if (!(group->closest[cpu].index == index &&
+		      group->closest[cpu].dist == IRQ_CPU_DIST_INF))
+			continue;
+		if (irq_group_copy_neigh(group, cpu,

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 181

Quoted 10 lines
kernel/irq/manage.c +10 -0
+					 topology_thread_cpumask(cpu), 1))
+			continue;
+		if (irq_group_copy_neigh(group, cpu,
+					 topology_core_cpumask(cpu), 2))
+			continue;
+		if (irq_group_copy_neigh(group, cpu,
+					 cpumask_of_node(cpu_to_node(cpu)), 3))
+			continue;
+		/* We could continue into NUMA node distances, but for now
+		 * we give up. */

What are the consequences of giving up ? Does not happen ? Should not happen ? Will break ? Don't care ? ....

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 181

Quoted 1 line
kernel/irq/manage.c +1 -0
+	}

This is called from irq_set_affinity() with desc->lock held and interrupts disabled. You're not serious about that, are you ?

Damned, there are two iterations over each online cpu and another one over the affinity mask. Did you ever extrapolate how long that runs on a really large machine ?

If CPU affinity of an IRQ changes, then it does not matter if one or two interrupts end up in the wrong space. Those changes are not happening every other interrupt.

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 187

Quoted 5 lines
kernel/irq/manage.c +5 -0
+}
+
+/**
+ *	alloc_irq_group - allocate IRQ group
+ *	@size:		Size of the group

size of what ? I guess number of interrupts, right ?

Quoted 4 lines
kernel/irq/manage.c +4 -0
+ *	@flags:		Allocation flags e.g. %GFP_KERNEL
+ */
+struct irq_group *alloc_irq_group(unsigned int size, gfp_t flags)
+{

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 201

Quoted 10 lines
kernel/irq/manage.c +10 -0
+	struct irq_group *group =
+		kzalloc(sizeof(*group) + size * sizeof(group->irq[0]), flags);
+	int cpu;
+
+	if (!group)
+		return NULL;
+
+	/* Initially assign CPUs to IRQs on a rota */
+	for (cpu = 0; cpu < NR_CPUS; cpu++) {
+		group->closest[cpu].index = cpu % size;

So here we randomly assign index with the lower cpu numbers no matter whether they are online or possible ?

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 208

Quoted 7 lines
kernel/irq/manage.c +7 -0
+		group->closest[cpu].dist = IRQ_CPU_DIST_INF;
+	}
+
+	group->size = size;
+	return group;
+}
+EXPORT_SYMBOL(alloc_irq_group);

EXPORT_SYMBOL_GPL if at all. Same for the other exports

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 212

Quoted 4 lines
kernel/irq/manage.c +4 -0
+
+/**
+ *	free_irq_group - free IRQ group
+ *	@group:		IRQ group allocated with alloc_irq_group(), or %NULL

How is this serialized or sanity checked against free_irq() ?

Quoted 18 lines
kernel/irq/manage.c +18 -0
+ */
+void free_irq_group(struct irq_group *group)
+{
+	struct irq_desc *desc;
+	unsigned int i;
+
+	if (!group)
+		return;
+
+	/* Remove all descriptors from the group */
+	for (i = 0; i < group->used; i++) {
+		desc = group->irq[i];
+		BUG_ON(desc->group != group || desc->group_index != i);
+		desc->group = NULL;
+	}
+
+	kfree(group);
+}

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 240

Quoted 10 lines
kernel/irq/manage.c +10 -0
+EXPORT_SYMBOL(free_irq_group);
+
+/**
+ *	irq_group_add - add IRQ to a group
+ *	@group:		IRQ group allocated with alloc_irq_group()
+ *	@irq:		Interrupt to add to group
+ */
+void irq_group_add(struct irq_group *group, unsigned int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);

Again, how is this serialized against anything else fiddling with irq_desc[irq]?

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 249

Quoted 8 lines
kernel/irq/manage.c +8 -0
+	BUG_ON(desc->group);
+	BUG_ON(group->used >= group->size);
+
+	desc->group = group;
+	desc->group_index = group->used;
+	group->irq[group->used++] = desc;
+}
+EXPORT_SYMBOL(irq_group_add);

Thanks,

tglx

#3 lore raw

Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices

Quoted 4 lines · Thomas Gleixner

Ben,

On Mon, 20 Sep 2010, Ben Hutchings wrote:

Quoted 8 lines

it would have been nice if I'd been cc'ed on that, but of course it's my fault that there is no entry in MAINTAINERS. No, it's not.

Quoted 1 line

When initiating I/O on multiqueue devices, we usually want to select a

"we usually" is pretty useless for people not familiar with the problem at hand. A [PATCH 0/4] intro would have been helpful along with the users (aka [PATCH 2-4/4]) of the new facility.

There was an intro on the netdev list
<http://article.gmane.org/gmane.linux.network/172427> but it probably doesn't tell you what you want to know anyway.

Quoted 11 lines

Don't take it personally and I'm sure that you're solving a real world problem, but I'm starting to get grumpy that especially networking folks (aside of various driver developers) believe that adding random stuff to kernel/irq is their private pleasure. Is it that hard to talk to me upfront ?

Quoted 4 lines

queue for which the response will be handled on the same or a nearby CPU. IRQ groups hold a mapping of CPU to IRQ which will be updated based on the inverse of IRQ CPU-affinities plus CPU topology information.

Quoted 5 lines

Can you please explain, why you need that reverse mapping including the below code ? What problem does this solve which can not be deduced by the exisiting information/infrastructure ? And why is that reverse mapping tied to interrupts and not something which we want to see in some generic available (library) code ?

Are you thinking of a per-CPU distance map? That seems like it would be useful to have. I wonder about the storage requirements on larger systems.

Quoted 18 lines

More comments inline.

include/linux/irq.h struct irq_desc { · line 198

Quoted 13 lines
---
 include/linux/irq.h |   52 ++++++++++++++++++
 kernel/irq/manage.c |  149 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 201 insertions(+), 0 deletions(-)
include/linux/irq.h +0 -0
index c03243a..bbddd5f 100644
@@ -196,6 +196,8 @@ struct irq_desc {
 #ifdef CONFIG_SMP
 	cpumask_var_t		affinity;
 	const struct cpumask	*affinity_hint;

How about updating the kernel doc above the struct ?

Quoted 4 lines

include/linux/irq.h struct irq_desc { · line 199

Quoted 1 line
include/linux/irq.h +1 -0
+	struct irq_group	*group;

Grr, how does this compile ? That needs at least a forward declaration of struct irq_group. RFC is _NOT_ an excuse

Naming a struct in a function parameter declarator is not a valid forward declaration, but this is.

Quoted 3 lines

include/linux/irq.h struct irq_desc { · line 200

Quoted 1 line
include/linux/irq.h +1 -0
+	u16			group_index;

What's group_index doing and what's the point of an u16 here ?

Index of this IRQ within the group.

Quoted 7 lines
Quoted 7 lines
include/linux/irq.h +0 -0
 	unsigned int		node;
 #ifdef CONFIG_GENERIC_PENDING_IRQ
 	cpumask_var_t		pending_mask;
@@ -498,6 +500,33 @@ static inline void free_desc_masks(struct irq_desc *old_desc,
 #endif
 }
 
Quoted 6 lines

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 506

Quoted 4 lines
include/linux/irq.h +4 -0
+/**
+ * struct irq_group - IRQ group for multiqueue devices
+ * @closest: For each CPU, the index and distance to the closest IRQ,
+ *	based on affinity masks

index of what ?

Index within the group of the closest IRQ for this CPU; also the queue index.

[...]

Quoted 8 lines

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 529

Quoted 5 lines
include/linux/irq.h +5 -0
+static inline u16 irq_group_get_index(struct irq_group *group, int cpu)
+{
+	return group->closest[cpu].index;
+}
+

So you have an accessor function for closest[cpu].index. Are the other members meant to be accessible directly by random driver ?

No, they should not be touched at all.

Quoted 17 lines
Quoted 4 lines
include/linux/irq.h +0 -0
 #else /* !CONFIG_SMP */
 
 static inline bool alloc_desc_masks(struct irq_desc *desc, int node,

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 557

Quoted 10 lines
include/linux/irq.h +7 -0
@@ -519,6 +548,29 @@ static inline void free_desc_masks(struct irq_desc *old_desc,
 				   struct irq_desc *new_desc)
 {
 }
+
+struct irq_group {
+};
+
+static inline struct irq_group *alloc_irq_group(unsigned int size, gfp_t flags)
+{
+	static struct irq_group dummy;

That will create one static instance per callsite. Is that on purpose? If yes, it needs a damned good comment.

Which uses no storage, or maybe 1 byte. I'm open to suggestions for a better dummy implementation.

Quoted 22 lines
Quoted 22 lines
include/linux/irq.h +1 -0
+	return &dummy;
 
 #endif	/* CONFIG_SMP */
 
 #endif /* _LINUX_IRQ_H */
kernel/irq/manage.c +9 -0
index c3003e9..3f2b1a9 100644
@@ -100,6 +100,154 @@ void irq_set_thread_affinity(struct irq_desc *desc)
 	}
 }
 
+static void irq_group_update_neigh(struct irq_group *group,
+				   const struct cpumask *mask,
+				   u16 index, u16 dist)
+{
+	int cpu;
+
+	for_each_cpu(cpu, mask) {
+		if (dist < group->closest[cpu].dist) {
+			group->closest[cpu].index = index;
Quoted 8 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 115

Quoted 4 lines
kernel/irq/manage.c +4 -0
+			group->closest[cpu].dist = dist;
+		}
+	}
+}

I have not the faintest idea how group, index and dist are related to each other. And I have no intention to decode the information about that piecewise by reverse engineering that obfuscated code.

I'm happy to write more comments.

index is the index of the IRQ being updated within the group. dist is the maximum distance of that IRQ from the CPUs in the mask.

Quoted 7 lines
Quoted 7 lines
kernel/irq/manage.c +7 -0
+static bool irq_group_copy_neigh(struct irq_group *group, int cpu,
+				 const struct cpumask *mask, u16 dist)
+{
+	int neigh;
+
+	for_each_cpu(neigh, mask) {
+		if (group->closest[neigh].dist <= dist) {
Quoted 8 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 129

Quoted 6 lines
kernel/irq/manage.c +6 -0
+			group->closest[cpu].index = group->closest[neigh].index;
+			group->closest[cpu].dist = dist;
+			return true;
+		}
+	}
+	return false;

What's the reason for copying or not ?

We're trying to find closer IRQs for the CPUs which were removed from the affinity of the IRQ currently being moved.

[...]

Quoted 5 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 181

Quoted 2 lines
kernel/irq/manage.c +2 -0
+		/* We could continue into NUMA node distances, but for now
+		 * we give up. */

What are the consequences of giving up ? Does not happen ? Should not happen ? Will break ? Don't care ? ....

Poor choice of closest IRQ. Nothing disastrous.

Quoted 8 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 131

Quoted 1 line
kernel/irq/manage.c +1 -0
+	}

This is called from irq_set_affinity() with desc->lock held and interrupts disabled. You're not serious about that, are you ?

Damned, there are two iterations over each online cpu and another one over the affinity mask. Did you ever extrapolate how long that runs on a really large machine ?

No, and I know this sucks. So, do you want to suggest anything better? Should I look at building those CPU distance maps (trading space for time)?

Quoted 11 lines

If CPU affinity of an IRQ changes, then it does not matter if one or two interrupts end up in the wrong space. Those changes are not happening every other interrupt.

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 187

Quoted 5 lines
kernel/irq/manage.c +5 -0
+}
+
+/**
+ *	alloc_irq_group - allocate IRQ group
+ *	@size:		Size of the group

size of what ? I guess number of interrupts, right ?

Right.

Quoted 9 lines
Quoted 9 lines
kernel/irq/manage.c +9 -0
+ *	@flags:		Allocation flags e.g. %GFP_KERNEL
+ */
+struct irq_group *alloc_irq_group(unsigned int size, gfp_t flags)
+{
+	struct irq_group *group =
+		kzalloc(sizeof(*group) + size * sizeof(group->irq[0]), flags);
+	int cpu;
+
+	if (!group)
Quoted 8 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 201

Quoted 5 lines
kernel/irq/manage.c +5 -0
+		return NULL;
+
+	/* Initially assign CPUs to IRQs on a rota */
+	for (cpu = 0; cpu < NR_CPUS; cpu++) {
+		group->closest[cpu].index = cpu % size;

So here we randomly assign index with the lower cpu numbers no matter whether they are online or possible ?

Doesn't matter, because we're mapping CPUs to IRQs not the other way round.

Quoted 10 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 208

Quoted 7 lines
kernel/irq/manage.c +7 -0
+		group->closest[cpu].dist = IRQ_CPU_DIST_INF;
+	}
+
+	group->size = size;
+	return group;
+}
+EXPORT_SYMBOL(alloc_irq_group);

EXPORT_SYMBOL_GPL if at all. Same for the other exports

Quoted 6 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 212

Quoted 4 lines
kernel/irq/manage.c +4 -0
+
+/**
+ *	free_irq_group - free IRQ group
+ *	@group:		IRQ group allocated with alloc_irq_group(), or %NULL

How is this serialized or sanity checked against free_irq() ?

free_irq() should be called first, for all IRQs in the group. The IRQs should then be really freed with pci_disable_msix() or similar.

Quoted 23 lines
Quoted 23 lines
kernel/irq/manage.c +23 -0
+ */
+void free_irq_group(struct irq_group *group)
+{
+	struct irq_desc *desc;
+	unsigned int i;
+
+	if (!group)
+		return;
+
+	/* Remove all descriptors from the group */
+	for (i = 0; i < group->used; i++) {
+		desc = group->irq[i];
+		BUG_ON(desc->group != group || desc->group_index != i);
+		desc->group = NULL;
+	}
+
+	kfree(group);
+}
+EXPORT_SYMBOL(free_irq_group);
+
+/**
+ *	irq_group_add - add IRQ to a group
+ *	@group:		IRQ group allocated with alloc_irq_group()
Quoted 8 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 240

Quoted 5 lines
kernel/irq/manage.c +5 -0
+ *	@irq:		Interrupt to add to group
+ */
+void irq_group_add(struct irq_group *group, unsigned int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);

Again, how is this serialized against anything else fiddling with irq_desc[irq]?

The driver should call this after allocating IRQs with pci_enable_msix() or similar and before setting the handlers with request_irq(). Is that sufficient?

Ben.

Quoted 12 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 249

Quoted 8 lines
kernel/irq/manage.c +8 -0
+	BUG_ON(desc->group);
+	BUG_ON(group->used >= group->size);
+
+	desc->group = group;
+	desc->group_index = group->used;
+	group->irq[group->used++] = desc;
+}
+EXPORT_SYMBOL(irq_group_add);

Thanks,

tglx

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#4 lore raw

Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices

Quoted 12 lines · Ben Hutchings
Quoted 10 lines · Thomas Gleixner
Quoted 4 lines

queue for which the response will be handled on the same or a nearby CPU. IRQ groups hold a mapping of CPU to IRQ which will be updated based on the inverse of IRQ CPU-affinities plus CPU topology information.

Can you please explain, why you need that reverse mapping including the below code ? What problem does this solve which can not be deduced by the exisiting information/infrastructure ? And why is that reverse mapping tied to interrupts and not something which we want to see in some generic available (library) code ?

Quoted 3 lines

Are you thinking of a per-CPU distance map? That seems like it would be useful to have. I wonder about the storage requirements on larger systems.

That might be a problem, but we should at least look into this.

But even if a prebuilt map is too expensive storage wise, then the functions which build the lookup map might be useful for similar distance lookup scenarios as well.

Quoted 11 lines
Quoted 6 lines

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 557

Quoted 3 lines
include/linux/irq.h +3 -0
+static inline struct irq_group *alloc_irq_group(unsigned int size, gfp_t flags)
+{
+	static struct irq_group dummy;

That will create one static instance per callsite. Is that on purpose? If yes, it needs a damned good comment.

Which uses no storage, or maybe 1 byte. I'm open to suggestions for a better dummy implementation.

Quoted 1 line

include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 558

Quoted 1 line
include/linux/irq.h +1 -0
+	return &dummy;

Why does it need to return a real struct instead of NULL ?

Quoted 5 lines
Quoted 5 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+	}

This is called from irq_set_affinity() with desc->lock held and interrupts disabled. You're not serious about that, are you ?

Quoted 7 lines
Quoted 3 lines

Damned, there are two iterations over each online cpu and another one over the affinity mask. Did you ever extrapolate how long that runs on a really large machine ?

No, and I know this sucks. So, do you want to suggest anything better? Should I look at building those CPU distance maps (trading space for time)?

The point is that there is no reason why this code needs to run in a spinlocked irq disabled reason. I know why your implementation needs it, because it protects you against module unload etc. Come on, we have enough infrastructure to control the lifetime of objects in sane ways and not by hijacking a heavy lock and disabling interrupts.

Btw, how does this code deal with concurrent affinity setting of multiple irqs in the group ? Random number generator ?

Quoted 1 line
Quoted 1 line
Quoted 1 line

+ * @flags: Allocation flags e.g. %GFP_KERNEL

Why do we need flags here? That code gets called in a device setup routine and not from random context.

Quoted 11 lines
Quoted 10 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 212

Quoted 8 lines

+ */

kernel/irq/manage.c +6 -0
+struct irq_group *alloc_irq_group(unsigned int size, gfp_t flags)
+{
 
+
+/**
+ *	free_irq_group - free IRQ group
+ *	@group:		IRQ group allocated with alloc_irq_group(), or %NULL

How is this serialized or sanity checked against free_irq() ?

Quoted 2 lines

free_irq() should be called first, for all IRQs in the group. The IRQs should then be really freed with pci_disable_msix() or similar.

I explained that several times, that I do not care about what should be called in which order and what not. Driver coders get it wrong all the time, then it explodes in the genirq code and I get the crap to debug. No, thanks.

Also if you call free_irq() first, then you still have a reference to that very irq descriptor in your group and a reference to the group in the irq descriptor. Brilliant, as the irq descriptor can be reused immediately after free_irq().

Also it's not only problematic against free_irq, it's damned racy against a concurrent affinity setting as well. And that's not controlled by "should be called" at all.

Quoted 12 lines
Quoted 11 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 240

Quoted 8 lines
kernel/irq/manage.c +8 -0
+/**
+ *	irq_group_add - add IRQ to a group
+ *	@group:		IRQ group allocated with alloc_irq_group()
+ *	@irq:		Interrupt to add to group
+ */
+void irq_group_add(struct irq_group *group, unsigned int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);

Again, how is this serialized against anything else fiddling with irq_desc[irq]?

Quoted 3 lines

The driver should call this after allocating IRQs with pci_enable_msix() or similar and before setting the handlers with request_irq(). Is that sufficient?

"should call" is _NEVER_ sufficient. fiddling with irq_desc w/o holding the lock is a nono, period.

Quoted 6 lines
Quoted 6 lines

kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 247

Quoted 6 lines
kernel/irq/manage.c +6 -0
+	BUG_ON(desc->group);
+	BUG_ON(group->used >= group->size);
+
+	desc->group = group;
+	desc->group_index = group->used;
+	group->irq[group->used++] = desc;

Forgot to ask yesterday. Why do we need to store irq desc in the group ? For free_irq_group() I guess, but I explained already why this code is broken. And it's even more broken when an irq is moved to a different node and CONFIG_NUMA_IRQ_DESC=y.

Lemme summarize what I understand so far. For multiqueue devices with several queues and interrupts you want to lookup the closest queue/irq to the cpu on which you are currently running to avoid expensive cross node memory access. Now you need to follow affinity changes for this lookup table.

The basic idea of your code is ok. The lookup table itself is fine, but the integration into the genirq code sucks. What needs to be done:

1) Remove the references to irq_desc in the group. They are not needed and will never work. The group does not need any information about the irq number and the irq descriptor.

2) Add proper refcounting to struct irq_group so it can be accessed outside of irq_desc->lock.

3) Move the update of the map outside of irq_desc->lock and the irq disabled region. Update the map in setup_affinity() as well. That code can be called from various atomic contexts, so it's probably the best thing to delegate the update to a workqueue or such unless you come up with a nice prebuilt lookup table.

4) Add comments so the code is understandable for mere mortals

5) Add proper install/free handling. Hint: affinity_hint

6) All modifications of irq_desc need a check whether irq_to_desc returned a valid pointer and must hold desc->lock

7) Deal with concurrent updates of multiple irqs in a group (at least a comment why the code does not need serialization)

8) Move the code to kernel/irq/irqgroup.c or some other sensible name and make it configurable so embedded folks don't have to carry it around as useless binary bloat.

Thanks,

tglx

#5 lore raw

Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices

Quoted 31 lines · Thomas Gleixner
Quoted 1 line · Ben Hutchings

On Mon, 2010-09-20 at 23:27 +0200, Thomas Gleixner wrote:

Lemme summarize what I understand so far. For multiqueue devices with several queues and interrupts you want to lookup the closest queue/irq to the cpu on which you are currently running to avoid expensive cross node memory access. Now you need to follow affinity changes for this lookup table.

The basic idea of your code is ok. The lookup table itself is fine, but the integration into the genirq code sucks. What needs to be done:

1) Remove the references to irq_desc in the group. They are not needed and will never work. The group does not need any information about the irq number and the irq descriptor.

2) Add proper refcounting to struct irq_group so it can be accessed outside of irq_desc->lock.

3) Move the update of the map outside of irq_desc->lock and the irq disabled region. Update the map in setup_affinity() as well. That code can be called from various atomic contexts, so it's probably the best thing to delegate the update to a workqueue or such unless you come up with a nice prebuilt lookup table.

4) Add comments so the code is understandable for mere mortals

5) Add proper install/free handling. Hint: affinity_hint

6) All modifications of irq_desc need a check whether irq_to_desc returned a valid pointer and must hold desc->lock

Quoted 6 lines

7) Deal with concurrent updates of multiple irqs in a group (at least a comment why the code does not need serialization)

8) Move the code to kernel/irq/irqgroup.c or some other sensible name and make it configurable so embedded folks don't have to carry it around as useless binary bloat.

Talked to Peter about it and we came to the conclusion, that we should just provide a callback infrastructure in the irq code which does not care about the action behind it. That's going to solve #1,#2,#3,#5,#6 and parts of #8

That queue/index map code should move to lib/ or some other appropriate place so it can be shared with storage or whatever is going to grow multiqueue. comments #4, #7, #8 (s@kernel/irq@lib/@) above still apply :)

The modification to the genirq code would be based on registering

struct irq_affinity_callback {
	unsigned int irq;
	struct kref kref;
	struct work work;
	void (*callback)(struct irq_affinity_callback *, const cpumask_t *mask);
	void (*release)(struct kref *ref);
};

for an interrupt via

int irq_set_affinity_callback(unsigned int irq,

    			      struct irq_affinity_callback *cb);

That function can be called with cb=NULL to remove the callback. if cb!=NULL, irq, kref and work are initialized.

The genirq code schedules work when affinity changes and the callback struct pointer is non NULL. The callback is then called from the worker thread with a copy of the irq affinity mask.

So on any affinity change, we do with desc->lock held:

   if (desc->affinity_callback) {
      	kref_get(&desc->affinity_callback->kref);
	schedule_work(&desc->affinity_callback->work);

}

The worker function in the genirq code does:

{
    struct irq_affinity_callback *cb;
    struct irq_desc *desc;
    cpu_mask_t cpumask;

    cb = container_of(work, struct irq_affinity_callback, work);

    desc = irq_to_desc(cb->irq);
    if (!desc)
       goto out;

    if (!alloc_cpumask_var(&cpumask, GFP_KERNEL)
       goto out;

    raw_spin_lock_irqsave(&desc->lock, flags);
    cpumask_copy(cpumask, desc->affinity);
    raw_spin_unlock_irqrestore(&desc->lock, flags);

    cb->callback(cb, cpumask);

    free_cpumask_var(cpumask);
out:
    kref_put(&cb->kref, cb->release);
}

That allows you to do all kind of magic in thread context, updating the queue map, reallocating queue memory when the node affinity changes (I know that you want to), go wild.

Thoughts ?

tglx

#6 lore raw

Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices

On Tue, 2010-09-21 at 21:04 +0200, Thomas Gleixner wrote:
[...]

Quoted 9 lines

Talked to Peter about it and we came to the conclusion, that we should just provide a callback infrastructure in the irq code which does not care about the action behind it. That's going to solve #1,#2,#3,#5,#6 and parts of #8

That queue/index map code should move to lib/ or some other appropriate place so it can be shared with storage or whatever is going to grow multiqueue. comments #4, #7, #8 (s@kernel/irq@lib/@) above still apply :)

OK.

Quoted 10 lines

The modification to the genirq code would be based on registering

struct irq_affinity_callback {
	unsigned int irq;
	struct kref kref;
	struct work work;
	void (*callback)(struct irq_affinity_callback *, const cpumask_t *mask);
	void (*release)(struct kref *ref);
};
Quoted 7 lines

for an interrupt via

int irq_set_affinity_callback(unsigned int irq,

    			      struct irq_affinity_callback *cb);

That function can be called with cb=NULL to remove the callback. if cb!=NULL, irq, kref and work are initialized.

When should it be called, relative to {request,free}_irq() and pci_{disable,enable}_msix()?

[...]

Quoted 3 lines

That allows you to do all kind of magic in thread context, updating the queue map, reallocating queue memory when the node affinity changes (I know that you want to), go wild.

I definitely don't want to reallocate queues if node affinity of the IRQ is changed by irqbalance, because this will disrupt traffic. So changing the node affinity of queues has to be a separate operation.

Quoted 1 line

Thoughts ?

This does look like something I can use, thanks.

Ben.

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#7 lore raw

Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices

Quoted 26 lines · Ben Hutchings

On Tue, 2010-09-21 at 21:04 +0200, Thomas Gleixner wrote:
[...]

Quoted 9 lines

Talked to Peter about it and we came to the conclusion, that we should just provide a callback infrastructure in the irq code which does not care about the action behind it. That's going to solve #1,#2,#3,#5,#6 and parts of #8

That queue/index map code should move to lib/ or some other appropriate place so it can be shared with storage or whatever is going to grow multiqueue. comments #4, #7, #8 (s@kernel/irq@lib/@) above still apply :)

OK.

Quoted 12 lines

The modification to the genirq code would be based on registering

struct irq_affinity_callback {
	unsigned int irq;
	struct kref kref;
	struct work work;
	void (*callback)(struct irq_affinity_callback *, const cpumask_t *mask);
	void (*release)(struct kref *ref);
};

for an interrupt via

Quoted 8 lines
Quoted 5 lines

int irq_set_affinity_callback(unsigned int irq,

    			      struct irq_affinity_callback *cb);

That function can be called with cb=NULL to remove the callback. if cb!=NULL, irq, kref and work are initialized.

When should it be called, relative to {request,free}_irq() and pci_{disable,enable}_msix()?

It should be called before request_irq and before free_irq. free_irq will warn when the pointer is !NULL.

Quoted 8 lines

[...]

Quoted 3 lines

That allows you to do all kind of magic in thread context, updating the queue map, reallocating queue memory when the node affinity changes (I know that you want to), go wild.

I definitely don't want to reallocate queues if node affinity of the IRQ is changed by irqbalance, because this will disrupt traffic. So changing the node affinity of queues has to be a separate operation.

Fair enough.

Quoted 3 lines
Quoted 1 line

Thoughts ?

This does look like something I can use, thanks.

Will look into it in the next days.

Thanks,

tglx

#8 lore raw

RFC PATCH 2/4 net: RPS: Enable hardware acceleration

In reply to Ben Hutchings

Allow drivers for multiqueue hardware with flow filter tables to accelerate RFS. The driver must:

1. Set net_device::rx_irq_group to an irq_group of the RX completion IRQs (in queue order). This will provide a mapping from CPUs to the queues for which completions are handled nearest to them.

2. Implement net_device_ops::ndo_rx_flow_steer. This operation adds or replaces a filter steering the given flow to the given RX queue, if possible.

3. Periodically remove filters for which rps_may_expire_flow() returns true.
---
The heuristic in rps_may_expire_flow() is quite possibly bogus. I'm not even sure whether expiry of flows should be triggered by the driver or from the RPS/RFS core. Any better ideas on how to do this?

Ben.

 include/linux/netdevice.h |   29 +++++++++++++--
 net/core/dev.c            |   88 +++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 108 insertions(+), 9 deletions(-)
include/linux/netdevice.h +26 -3
index f7f1302..897118f 100644
@@ -524,14 +524,16 @@ struct rps_map {
 #define RPS_MAP_SIZE(_num) (sizeof(struct rps_map) + (_num * sizeof(u16)))
 
 /*
- * The rps_dev_flow structure contains the mapping of a flow to a CPU and the
- * tail pointer for that CPU's input queue at the time of last enqueue.
+ * The rps_dev_flow structure contains the mapping of a flow to a CPU, the
+ * tail pointer for that CPU's input queue at the time of last enqueue, and
+ * a hardware filter index.
  */
 struct rps_dev_flow {
 	u16 cpu;
-	u16 fill;
+	u16 filter;
 	unsigned int last_qtail;
 };
+#define RPS_NO_FILTER 0xffff
 
 /*
  * The rps_dev_flow_table structure contains a table of flow mappings.
@@ -581,6 +583,9 @@ static inline void rps_reset_sock_flow(struct rps_sock_flow_table *table,
 
 extern struct rps_sock_flow_table *rps_sock_flow_table;
 
+extern bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,
+				u32 flow_id, u16 filter_id);
+
 /* This structure contains an instance of an RX queue. */
 struct netdev_rx_queue {
 	struct rps_map *rps_map;
@@ -701,6 +706,13 @@ struct netdev_rx_queue {
  * int (*ndo_set_vf_port)(struct net_device *dev, int vf,
  *			  struct nlattr *port[]);
  * int (*ndo_get_vf_port)(struct net_device *dev, int vf, struct sk_buff *skb);
+ *
+ *	RFS acceleration.
+ * int (*ndo_rx_flow_steer)(struct net_device *dev, const struct sk_buff *skb,
+ *			    u16 rxq_index, u32 flow_id);
+ *	Set hardware filter for RFS.  rxq_index is the target queue index;
+ *	flow_id is a flow ID to be passed to rps_may_expire_flow() later.
+ *	Return the filter ID on success, or a negative error code.
  */
 #define HAVE_NET_DEVICE_OPS
 struct net_device_ops {
@@ -773,6 +785,12 @@ struct net_device_ops {
 	int			(*ndo_fcoe_get_wwn)(struct net_device *dev,
 						    u64 *wwn, int type);
 #endif
+#ifdef CONFIG_RPS
+	int			(*ndo_rx_flow_steer)(struct net_device *dev,
+						     const struct sk_buff *skb,
+						     u16 rxq_index,
+						     u32 flow_id);
+#endif
 };
 
 /*
@@ -978,6 +996,11 @@ struct net_device {
 
 	/* Number of RX queues allocated at alloc_netdev_mq() time  */
 	unsigned int		num_rx_queues;
+
+	/* IRQ group for RX completion interrupts, indexed by RX queue
+	 * number.  Assigned by driver.  This must only be set if the
+	 * ndo_rx_flow_steer operation is defined. */
+	struct irq_group	*rx_irq_group;
 #endif
 
 	rx_handler_func_t	*rx_handler;
net/core/dev.c +82 -6

106 lines not on this page · raw message

Signature · 8 lines
1.7.2.1



-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#9 lore raw

RFC PATCH 3/4 sfc: Implement RFS acceleration

In reply to Ben Hutchings

---
This depends on today's patch series for net-next-2.6.

Ben.

 drivers/net/sfc/efx.c    |   49 ++++++++++++++++++----
 drivers/net/sfc/efx.h    |    9 ++++
 drivers/net/sfc/filter.c |  100 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 149 insertions(+), 9 deletions(-)
drivers/net/sfc/efx.c +40 -9
index 8a51c41..aea6283 100644
@@ -127,6 +127,8 @@ static int napi_weight = 64;
  * monitor.  On Falcon-based NICs, this will:
  * - Check the on-board hardware monitor;
  * - Poll the link state and reconfigure the hardware as necessary.
+ * If RFS is enabled, this will scan part of the RX IP filter table and
+ * remove filters for inactive flows.
  */
 unsigned int efx_monitor_interval = 1 * HZ;
 
@@ -1174,7 +1176,7 @@ static int efx_wanted_channels(void)
 /* Probe the number and type of interrupts we are able to obtain, and
  * the resulting numbers of channels and RX queues.
  */
-static void efx_probe_interrupts(struct efx_nic *efx)
+static int efx_probe_interrupts(struct efx_nic *efx)
 {
 	int max_channels =
 		min_t(int, efx->type->phys_addr_channels, EFX_MAX_CHANNELS);
@@ -1216,6 +1218,17 @@ static void efx_probe_interrupts(struct efx_nic *efx)
 				efx->n_tx_channels = efx->n_channels;
 				efx->n_rx_channels = efx->n_channels;
 			}
+#ifdef CONFIG_RPS
+			efx->net_dev->rx_irq_group =
+				alloc_irq_group(efx->n_rx_channels, GFP_KERNEL);
+			if (!efx->net_dev->rx_irq_group) {
+				pci_disable_msix(efx->pci_dev);
+				return -ENOMEM;
+			}
+			for (i = 0; i < efx->n_rx_channels; i++)
+				irq_group_add(efx->net_dev->rx_irq_group,
+					      xentries[i].vector);
+#endif
 			for (i = 0; i < n_channels; i++)
 				efx_get_channel(efx, i)->irq =
 					xentries[i].vector;
@@ -1249,6 +1262,8 @@ static void efx_probe_interrupts(struct efx_nic *efx)
 		efx->n_tx_channels = 1;
 		efx->legacy_irq = efx->pci_dev->irq;
 	}
+
+	return 0;
 }
 
 static void efx_remove_interrupts(struct efx_nic *efx)
@@ -1258,6 +1273,10 @@ static void efx_remove_interrupts(struct efx_nic *efx)
 	/* Remove MSI/MSI-X interrupts */
 	efx_for_each_channel(channel, efx)
 		channel->irq = 0;
+#ifdef CONFIG_RPS
+	free_irq_group(efx->net_dev->rx_irq_group);
+	efx->net_dev->rx_irq_group = NULL;
+#endif
 	pci_disable_msi(efx->pci_dev);
 	pci_disable_msix(efx->pci_dev);
 
@@ -1307,7 +1326,9 @@ static int efx_probe_nic(struct efx_nic *efx)
 
 	/* Determine the number of channels and queues by trying to hook
 	 * in MSI-X interrupts. */
-	efx_probe_interrupts(efx);
+	rc = efx_probe_interrupts(efx);
+	if (rc)
+		goto fail;
 
 	if (efx->n_channels > 1)
 		get_random_bytes(&efx->rx_hash_key, sizeof(efx->rx_hash_key));
@@ -1322,6 +1343,10 @@ static int efx_probe_nic(struct efx_nic *efx)
 	efx_init_irq_moderation(efx, tx_irq_mod_usec, rx_irq_mod_usec, true);
 
 	return 0;
+
+fail:
+	efx->type->remove(efx);
+	return rc;
 }
 
 static void efx_remove_nic(struct efx_nic *efx)
@@ -1419,13 +1444,15 @@ static void efx_start_all(struct efx_nic *efx)
 	if (efx->reset_pending != RESET_TYPE_NONE)
 		efx_mcdi_mode_poll(efx);
 
-	/* Start the hardware monitor if there is one. Otherwise (we're link
-	 * event driven), we have to poll the PHY because after an event queue
-	 * flush, we could have a missed a link state change */
-	if (efx->type->monitor != NULL) {
+	/* Start the periodic monitor if necessary */
+	if (efx->type->monitor || efx_filter_rfs_enabled())
 		queue_delayed_work(efx->workqueue, &efx->monitor_work,
 				   efx_monitor_interval);
-	} else {
+
+	/* If we normally rely on link state events, we have to poll
+	 * the PHY because after an event queue flush, we could have a
+	 * missed a link state change */
+	if (!efx->type->monitor) {
 		mutex_lock(&efx->mac_lock);
 		if (efx->phy_op->poll(efx))
 			efx_link_status_changed(efx);
@@ -1556,17 +1583,18 @@ static void efx_monitor(struct work_struct *data)
 	netif_vdbg(efx, timer, efx->net_dev,
 		   "hardware monitor executing on CPU %d\n",
 		   raw_smp_processor_id());
-	BUG_ON(efx->type->monitor == NULL);
 
 	/* If the mac_lock is already held then it is likely a port
 	 * reconfiguration is already in place, which will likely do
 	 * most of the work of monitor() anyway. */
-	if (mutex_trylock(&efx->mac_lock)) {
+	if (efx->type->monitor && mutex_trylock(&efx->mac_lock)) {
 		if (efx->port_enabled)
 			efx->type->monitor(efx);
 		mutex_unlock(&efx->mac_lock);
 	}
 
+	efx_filter_rfs_expire(efx);
+
 	queue_delayed_work(efx->workqueue, &efx->monitor_work,
 			   efx_monitor_interval);
 }
@@ -1849,6 +1877,9 @@ static const struct net_device_ops efx_netdev_ops = {
 #ifdef CONFIG_NET_POLL_CONTROLLER
 	.ndo_poll_controller = efx_netpoll,
 #endif
+#ifdef CONFIG_RPS
+	.ndo_rx_flow_steer	= efx_filter_rfs,
+#endif
 };
 
 static void efx_update_name(struct efx_nic *efx)
drivers/net/sfc/efx.h +9 -0
index f502b14..88a43f1 100644
@@ -77,6 +77,15 @@ extern int efx_filter_remove_filter(struct efx_nic *efx,
 extern void efx_filter_table_clear(struct efx_nic *efx,
 				   enum efx_filter_table_id table_id,
 				   enum efx_filter_priority priority);
+#ifdef CONFIG_RPS
+extern int efx_filter_rfs(struct net_device *net_dev, const struct sk_buff *skb,
+			  u16 rxq_index, u32 flow_id);
+extern void efx_filter_rfs_expire(struct efx_nic *efx);
+#define efx_filter_rfs_enabled() 1
+#else
+static inline void efx_filter_rfs_expire(struct efx_nic *efx) {}
+#define efx_filter_rfs_enabled() 0
+#endif
 
 /* Channels */
 extern void efx_process_channel_now(struct efx_channel *channel);
drivers/net/sfc/filter.c +100 -0

123 lines not on this page · raw message

Signature · 8 lines
1.7.2.1



-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#10 lore raw

RFC PATCH 4/4 sfc/RFS/irq_group debug output

In reply to Ben Hutchings

Just some logging I found useful.

Ben.

---
 drivers/net/sfc/filter.c |   11 ++++++++++-
 kernel/irq/manage.c      |   21 +++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletions(-)
drivers/net/sfc/filter.c +10 -1
index 349b5d1..db7fa46 100644
@@ -506,6 +506,11 @@ int efx_filter_rfs(struct net_device *net_dev, const struct sk_buff *skb,
 	rc = efx_filter_insert_filter(efx, &spec, true);
 	if (rc >= 0)
 		state->rps_flow_id[rc] = flow_id;
+	netif_info(efx, rx_status, efx->net_dev,
+		   "steering %s %pI4:%u:%pI4:%u to queue %u [flow %u filter %d]\n",
+		   (ip->protocol == IPPROTO_TCP) ? "TCP" : "UDP",
+		   &ip->saddr, ntohs(ports[0]), &ip->daddr, ntohs(ports[1]),
+		   rxq_index, flow_id, rc);
 
 	return rc;
 }
@@ -529,8 +534,12 @@ void efx_filter_rfs_expire(struct efx_nic *efx)
 		    table->spec[index].priority == EFX_FILTER_PRI_HINT &&
 		    rps_may_expire_flow(efx->net_dev,
 					table->spec[index].dmaq_id,
-					state->rps_flow_id[index], index))
+					state->rps_flow_id[index], index)) {
+			netif_info(efx, rx_status, efx->net_dev,
+				   "expiring filter %d [flow %u]\n",
+				   index, state->rps_flow_id[index]);
 			efx_filter_table_clear_entry(efx, table, index);
+		}
 		index = (index + 1) & mask;
 	}
 
kernel/irq/manage.c +21 -0
index 3f2b1a9..7199dde 100644
@@ -129,6 +129,21 @@ static bool irq_group_copy_neigh(struct irq_group *group, int cpu,
 	return false;
 }
 
+static void print_irq_group(const struct irq_group *group, const char *prefix)
+{
+	unsigned index;
+	int cpu;
+
+	pr_info("irq_group %p, %s:\n", group, prefix);
+
+	for_each_possible_cpu(cpu) {
+		index = group->closest[cpu].index;
+		pr_info("cpu %d -> index %u (IRQ %u; distance %u)\n",
+			cpu, index, group->irq[index]->irq,
+			group->closest[cpu].dist);
+	}
+}
+
 /* Update the per-CPU closest IRQs following a change of affinity */
 static void
 irq_update_group(struct irq_desc *desc, const struct cpumask *affinity)
@@ -145,6 +160,8 @@ irq_update_group(struct irq_desc *desc, const struct cpumask *affinity)
 		if (group->closest[cpu].index == index)
 			group->closest[cpu].dist = IRQ_CPU_DIST_INF;
 
+	print_irq_group(group, "after invalidating old distances");
+
 	/*
 	 * Set this as the closest IRQ for all CPUs in the affinity mask,
 	 * plus the following CPUs if they don't have a closer IRQ:
@@ -163,6 +180,8 @@ irq_update_group(struct irq_desc *desc, const struct cpumask *affinity)
 				       index, 3);
 	}
 
+	print_irq_group(group, "after updating neighbours");
+
 	/* Find new closest IRQ for any CPUs left with invalid distances */
 	for_each_online_cpu(cpu) {
 		if (!(group->closest[cpu].index == index &&
@@ -180,6 +199,8 @@ irq_update_group(struct irq_desc *desc, const struct cpumask *affinity)
 		/* We could continue into NUMA node distances, but for now
 		 * we give up. */
 	}
+
+	print_irq_group(group, "after copying neighbours");
 }
 
 /**
Signature · 7 lines
1.7.2.1


-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#11 lore raw
In reply to Ben Hutchings

Thanks Ben, this does look interesting. We'll try to take a look.

Quoted 39 lines · Ben Hutchings

Tom,

This patch series extends RFS to use hardware RX filters where available.  Depending on the number of hardware RX queues and their IRQs' affinity, this should reduce the need for IPIs or at least get packets delivered to the right NUMA node.

I've implemented the driver side of this for our hardware, though I don't know whether you have any of that to test on.  I would be very interested to know how much this can help in the sort of cases where you use RFS.

Ben.

Ben Hutchings (4):
 IRQ: IRQ groups for multiqueue devices
 net: RPS: Enable hardware acceleration
 sfc: Implement RFS acceleration
 sfc/RFS/irq_group debug output

 drivers/net/sfc/efx.c     |   49 +++++++++++--- drivers/net/sfc/efx.h     |    9 +++
 drivers/net/sfc/filter.c  |  109 +++++++++++++++++++++++++++++ include/linux/irq.h       |   52 ++++++++++++++ include/linux/netdevice.h |   29 +++++++-
 kernel/irq/manage.c       |  170 +++++++++++++++++++++++++++++++++++++++++++++ net/core/dev.c            |   88 ++++++++++++++++++++++--

 7 files changed, 488 insertions(+), 18 deletions(-)

--
1.7.2.1

--
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.