RFS hardware acceleration
Subsystems (3)
cover RFS hardware acceleration
Tom,
This patch series extends RFS to use hardware RX filters where available. Depending on the number of hardware RX queues and their IRQs' affinity, this should reduce the need for IPIs or at least get packets delivered to the right NUMA node.
I've implemented the driver side of this for our hardware, though I don't know whether you have any of that to test on. I would be very interested to know how much this can help in the sort of cases where you use RFS.
Ben.
Ben Hutchings (4):
IRQ: IRQ groups for multiqueue devices
net: RPS: Enable hardware acceleration
sfc: Implement RFS acceleration
sfc/RFS/irq_group debug output
drivers/net/sfc/efx.c | 49 +++++++++++--- drivers/net/sfc/efx.h | 9 +++ drivers/net/sfc/filter.c | 109 +++++++++++++++++++++++++++++ include/linux/irq.h | 52 ++++++++++++++ include/linux/netdevice.h | 29 +++++++- kernel/irq/manage.c | 170 +++++++++++++++++++++++++++++++++++++++++++++ net/core/dev.c | 88 ++++++++++++++++++++++-- 7 files changed, 488 insertions(+), 18 deletions(-)
Signature · 7 lines
1.7.2.1 -- Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
RFC PATCH 1/4 IRQ: IRQ groups for multiqueue devices
When initiating I/O on multiqueue devices, we usually want to select a queue for which the response will be handled on the same or a nearby CPU. IRQ groups hold a mapping of CPU to IRQ which will be updated based on the inverse of IRQ CPU-affinities plus CPU topology information.
--- include/linux/irq.h | 52 ++++++++++++++++++ kernel/irq/manage.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 201 insertions(+), 0 deletions(-)
include/linux/irq.h +52 -0
@@ -196,6 +196,8 @@ struct irq_desc {#ifdef CONFIG_SMP cpumask_var_t affinity;const struct cpumask *affinity_hint; +struct irq_group *group; +u16 group_index;unsigned int node;#ifdef CONFIG_GENERIC_PENDING_IRQ cpumask_var_t pending_mask;
@@ -498,6 +500,33 @@ static inline void free_desc_masks(struct irq_desc *old_desc,#endif } +/** +* struct irq_group - IRQ group for multiqueue devices +* @closest: For each CPU, the index and distance to the closest IRQ, +* based on affinity masks +* @size: Size of the group +* @used: Number of IRQs currently included in the group +* @irq: Descriptors for IRQs in the group +*/ +struct irq_group { +struct { +u16 index; +u16 dist; + } closest[NR_CPUS ]; +unsigned int size, used; +struct irq_desc *irq[0 ]; +}; +#define IRQ_CPU_DIST_INF 0xffff + +extern struct irq_group *alloc_irq_group (unsigned int size,gfp_t flags); +extern void free_irq_group (struct irq_group *group); +extern void irq_group_add (struct irq_group *group,unsigned int irq); + +static inline u16 irq_group_get_index (struct irq_group *group,int cpu) +{ +return group->closest[cpu].index; +} +#else /* !CONFIG_SMP */ static inline bool alloc_desc_masks(struct irq_desc *desc, int node,
@@ -519,6 +548,29 @@ static inline void free_desc_masks(struct irq_desc *old_desc, struct irq_desc *new_desc) { } + +struct irq_group { +}; + +static inline struct irq_group *alloc_irq_group (unsigned int size,gfp_t flags) +{ +static struct irq_group dummy; +return &dummy; +} + +static inline void free_irq_group (struct irq_group *group) +{ +} + +static inline void irq_group_add (struct irq_group *group,unsigned int irq) +{ +} + +static inline u16 irq_group_get_index (struct irq_group *group,int cpu) +{ +return 0 ; +} +#endif /* CONFIG_SMP */ #endif /* _LINUX_IRQ_H */
kernel/irq/manage.c +149 -0
161 lines not on this page · raw message
Signature · 8 lines
1.7.2.1 -- Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices
Ben,
On Mon, 20 Sep 2010, Ben Hutchings wrote:
it would have been nice if I'd been cc'ed on that, but of course it's my fault that there is no entry in MAINTAINERS. No, it's not.
Quoted 1 line
When initiating I/O on multiqueue devices, we usually want to select a
"we usually" is pretty useless for people not familiar with the problem at hand. A [PATCH 0/4] intro would have been helpful along with the users (aka [PATCH 2-4/4]) of the new facility.
Don't take it personally and I'm sure that you're solving a real world problem, but I'm starting to get grumpy that especially networking folks (aside of various driver developers) believe that adding random stuff to kernel/irq is their private pleasure. Is it that hard to talk to me upfront ?
Quoted 4 lines
queue for which the response will be handled on the same or a nearby CPU. IRQ groups hold a mapping of CPU to IRQ which will be updated based on the inverse of IRQ CPU-affinities plus CPU topology information.
Can you please explain, why you need that reverse mapping including the below code ? What problem does this solve which can not be deduced by the exisiting information/infrastructure ? And why is that reverse mapping tied to interrupts and not something which we want to see in some generic available (library) code ?
More comments inline.
include/linux/irq.h struct irq_desc { · line 198
Quoted 13 lines
--- include/linux/irq.h | 52 ++++++++++++++++++ kernel/irq/manage.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 201 insertions(+), 0 deletions(-)include/linux/irq.h +0
-0@@ -196,6 +196,8 @@ struct irq_desc {#ifdef CONFIG_SMP cpumask_var_t affinity;const struct cpumask *affinity_hint;
How about updating the kernel doc above the struct ?
include/linux/irq.h struct irq_desc { · line 199
Quoted 1 line
include/linux/irq.h +1
-0+struct irq_group *group;
Grr, how does this compile ? That needs at least a forward declaration of struct irq_group. RFC is _NOT_ an excuse
include/linux/irq.h struct irq_desc { · line 200
Quoted 1 line
include/linux/irq.h +1
-0+u16 group_index;
What's group_index doing and what's the point of an u16 here ?
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 506
Quoted 11 lines
include/linux/irq.h +4
-0unsigned int node;#ifdef CONFIG_GENERIC_PENDING_IRQ cpumask_var_t pending_mask;@@ -498,6 +500,33 @@ static inline void free_desc_masks(struct irq_desc *old_desc,#endif } +/** +* struct irq_group - IRQ group for multiqueue devices +* @closest: For each CPU, the index and distance to the closest IRQ, +* based on affinity masks
index of what ?
Btw, please follow the style of the other kernel doc comments in this file where the explanation is aligned for readability sake
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 509
Quoted 3 lines
include/linux/irq.h +3
-0+* @size: Size of the group +* @used: Number of IRQs currently included in the group +* @irq: Descriptors for IRQs in the group
That's an array of pointers to irq descriptors, right ?
Insert some sensible decription what irq groups are and for which problem space they are useful if at all.
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 516
Quoted 7 lines
include/linux/irq.h +7
-0+*/ +struct irq_group { +struct { +u16 index; +u16 dist; + } closest[NR_CPUS ]; +unsigned int size, used;
Separate lines please
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 518
Quoted 2 lines
include/linux/irq.h +2
-0+struct irq_desc *irq[0 ]; +};
Please separate this with a newline and add some useful comment about the meaning and purpose of this not selfexplaining constant.
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 529
Quoted 7 lines
include/linux/irq.h +6
-0+#define IRQ_CPU_DIST_INF 0xffff +static inline u16 irq_group_get_index (struct irq_group *group,int cpu) +{ +return group->closest[cpu].index; +} +
So you have an accessor function for closest[cpu].index. Are the other members meant to be accessible directly by random driver ?
Quoted 4 lines
include/linux/irq.h +0
-0#else /* !CONFIG_SMP */ static inline bool alloc_desc_masks(struct irq_desc *desc, int node,
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 557
Quoted 10 lines
include/linux/irq.h +7
-0@@ -519,6 +548,29 @@ static inline void free_desc_masks(struct irq_desc *old_desc, struct irq_desc *new_desc) { } + +struct irq_group { +}; + +static inline struct irq_group *alloc_irq_group (unsigned int size,gfp_t flags) +{ +static struct irq_group dummy;
That will create one static instance per callsite. Is that on purpose? If yes, it needs a damned good comment.
Quoted 16 lines
include/linux/irq.h +1
-0+return &dummy;#endif /* CONFIG_SMP */ #endif /* _LINUX_IRQ_H */ kernel/irq/manage.c +3
-0@@ -100,6 +100,154 @@ void irq_set_thread_affinity(struct irq_desc *desc) } } +static void irq_group_update_neigh (struct irq_group *group, +const struct cpumask *mask, +u16 index,u16 dist)
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 115
Quoted 10 lines
kernel/irq/manage.c +10
-0+{ +int cpu; + +for_each_cpu (cpu, mask) { +if (dist < group->closest[cpu].dist) { + group->closest[cpu].index = index; + group->closest[cpu].dist = dist; + } + } +}
I have not the faintest idea how group, index and dist are related to each other. And I have no intention to decode the information about that piecewise by reverse engineering that obfuscated code.
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 129
Quoted 13 lines
kernel/irq/manage.c +13
-0+static bool irq_group_copy_neigh (struct irq_group *group,int cpu, +const struct cpumask *mask,u16 dist) +{ +int neigh; + +for_each_cpu (neigh, mask) { +if (group->closest[neigh].dist <= dist) { + group->closest[cpu].index = group->closest[neigh].index; + group->closest[cpu].dist = dist; +return true; + } + } +return false;
What's the reason for copying or not ?
Quoted 42 lines
kernel/irq/manage.c +42
-0+} + +/* Update the per-CPU closest IRQs following a change of affinity */ +static void +irq_update_group (struct irq_desc *desc,const struct cpumask *affinity) +{ +struct irq_group *group = desc->group; +unsigned index = desc->group_index; +int cpu; + +if (!group) +return ; + +/* Invalidate old distances to this IRQ */ +for_each_online_cpu (cpu) +if (group->closest[cpu].index == index) + group->closest[cpu].dist =IRQ_CPU_DIST_INF ; + +/* +* Set this as the closest IRQ for all CPUs in the affinity mask, +* plus the following CPUs if they don't have a closer IRQ: +* - all other threads in the same core (distance 1); +* - all other cores in the same package (distance 2); +* - all other packages in the same NUMA node (distance 3). +*/ +for_each_cpu (cpu, affinity) { + group->closest[cpu].index = index; + group->closest[cpu].dist =0 ; +irq_group_update_neigh (group,topology_thread_cpumask (cpu), + index,1 ); +irq_group_update_neigh (group,topology_core_cpumask (cpu), + index,2 ); +irq_group_update_neigh (group,cpumask_of_node (cpu_to_node (cpu)), + index,3 ); + } + +/* Find new closest IRQ for any CPUs left with invalid distances */ +for_each_online_cpu (cpu) { +if (!(group->closest[cpu].index == index && + group->closest[cpu].dist ==IRQ_CPU_DIST_INF )) +continue ; + if (irq_group_copy_neigh(group, cpu,
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 181
Quoted 10 lines
kernel/irq/manage.c +10
-0+ topology_thread_cpumask(cpu), 1)) +continue ; +if (irq_group_copy_neigh (group, cpu, +topology_core_cpumask (cpu),2 )) +continue ; +if (irq_group_copy_neigh (group, cpu, +cpumask_of_node (cpu_to_node (cpu)),3 )) +continue ; +/* We could continue into NUMA node distances, but for now +* we give up. */
What are the consequences of giving up ? Does not happen ? Should not happen ? Will break ? Don't care ? ....
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 181
Quoted 1 line
kernel/irq/manage.c +1
-0+ }
This is called from irq_set_affinity() with desc->lock held and interrupts disabled. You're not serious about that, are you ?
Damned, there are two iterations over each online cpu and another one over the affinity mask. Did you ever extrapolate how long that runs on a really large machine ?
If CPU affinity of an IRQ changes, then it does not matter if one or two interrupts end up in the wrong space. Those changes are not happening every other interrupt.
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 187
Quoted 5 lines
kernel/irq/manage.c +5
-0+} + +/** +* alloc_irq_group - allocate IRQ group +* @size: Size of the group
size of what ? I guess number of interrupts, right ?
Quoted 4 lines
kernel/irq/manage.c +4
-0+* @flags: Allocation flags e.g. %GFP_KERNEL +*/ +struct irq_group *alloc_irq_group (unsigned int size,gfp_t flags) +{
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 201
Quoted 10 lines
kernel/irq/manage.c +10
-0+struct irq_group *group = +kzalloc (sizeof (*group) + size *sizeof (group->irq[0 ]), flags); +int cpu; + +if (!group) +return NULL ; + +/* Initially assign CPUs to IRQs on a rota */ +for (cpu =0 ; cpu <NR_CPUS ; cpu++) { + group->closest[cpu].index = cpu % size;
So here we randomly assign index with the lower cpu numbers no matter whether they are online or possible ?
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 208
Quoted 7 lines
kernel/irq/manage.c +7
-0+ group->closest[cpu].dist =IRQ_CPU_DIST_INF ; + } + + group->size = size; +return group; +} +EXPORT_SYMBOL (alloc_irq_group);
EXPORT_SYMBOL_GPL if at all. Same for the other exports
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 212
Quoted 4 lines
kernel/irq/manage.c +4
-0+ +/** +* free_irq_group - free IRQ group +* @group: IRQ group allocated with alloc_irq_group(), or %NULL
How is this serialized or sanity checked against free_irq() ?
Quoted 18 lines
kernel/irq/manage.c +18
-0+*/ +void free_irq_group (struct irq_group *group) +{ +struct irq_desc *desc; +unsigned int i; + +if (!group) +return ; + +/* Remove all descriptors from the group */ +for (i =0 ; i < group->used; i++) { + desc = group->irq[i]; +BUG_ON (desc->group != group || desc->group_index != i); + desc->group =NULL ; + } + +kfree (group); +}
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 240
Quoted 10 lines
kernel/irq/manage.c +10
-0+EXPORT_SYMBOL (free_irq_group); + +/** +* irq_group_add - add IRQ to a group +* @group: IRQ group allocated with alloc_irq_group() +* @irq: Interrupt to add to group +*/ +void irq_group_add (struct irq_group *group,unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq);
Again, how is this serialized against anything else fiddling with irq_desc[irq]?
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 249
Quoted 8 lines
kernel/irq/manage.c +8
-0+BUG_ON (desc->group); +BUG_ON (group->used >= group->size); + + desc->group = group; + desc->group_index = group->used; + group->irq[group->used++] = desc; +} +EXPORT_SYMBOL (irq_group_add);
Thanks,
tglx
Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices
Quoted 4 lines · Thomas Gleixner
Ben,
On Mon, 20 Sep 2010, Ben Hutchings wrote:
Quoted 8 lines
it would have been nice if I'd been cc'ed on that, but of course it's my fault that there is no entry in
MAINTAINERS. No, it's not.Quoted 1 line
When initiating I/O on multiqueue devices, we usually want to select a
"we usually" is pretty useless for people not familiar with the problem at hand. A [PATCH 0/4] intro would have been helpful along with the users (aka [PATCH 2-4/4]) of the new facility.
There was an intro on the netdev list
<http://article.gmane.org/gmane.linux.network/172427> but it probably doesn't tell you what you want to know anyway.
Quoted 11 lines
Don't take it personally and I'm sure that you're solving a real world problem, but I'm starting to get grumpy that especially networking folks (aside of various driver developers) believe that adding random stuff to kernel/irq is their private pleasure. Is it that hard to talk to me upfront ?
Quoted 4 lines
queue for which the response will be handled on the same or a nearby CPU. IRQ groups hold a mapping of CPU to IRQ which will be updated based on the inverse of IRQ CPU-affinities plus CPU topology information.
Quoted 5 lines
Can you please explain, why you need that reverse mapping including the below code ? What problem does this solve which can not be deduced by the exisiting information/infrastructure ? And why is that reverse mapping tied to interrupts and not something which we want to see in some generic available (library) code ?
Are you thinking of a per-CPU distance map? That seems like it would be useful to have. I wonder about the storage requirements on larger systems.
Quoted 18 lines
More comments inline.
include/linux/irq.h struct irq_desc { · line 198
Quoted 13 lines
--- include/linux/irq.h | 52 ++++++++++++++++++ kernel/irq/manage.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 201 insertions(+), 0 deletions(-)include/linux/irq.h +0
-0@@ -196,6 +196,8 @@ struct irq_desc {#ifdef CONFIG_SMP cpumask_var_t affinity;const struct cpumask *affinity_hint;How about updating the kernel doc above the struct ?
Quoted 4 lines
include/linux/irq.h struct irq_desc { · line 199
Quoted 1 line
include/linux/irq.h +1
-0+struct irq_group *group;Grr, how does this compile ? That needs at least a forward declaration of struct
irq_group. RFC is _NOT_ an excuse
Naming a struct in a function parameter declarator is not a valid forward declaration, but this is.
Quoted 3 lines
include/linux/irq.h struct irq_desc { · line 200
Quoted 1 line
include/linux/irq.h +1
-0+u16 group_index;What's
group_indexdoing and what's the point of an u16 here ?
Index of this IRQ within the group.
Quoted 7 lines
Quoted 7 lines
include/linux/irq.h +0
-0unsigned int node;#ifdef CONFIG_GENERIC_PENDING_IRQ cpumask_var_t pending_mask;@@ -498,6 +500,33 @@ static inline void free_desc_masks(struct irq_desc *old_desc,#endif }
Quoted 6 lines
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 506
Quoted 4 lines
include/linux/irq.h +4
-0+/** +* struct irq_group - IRQ group for multiqueue devices +* @closest: For each CPU, the index and distance to the closest IRQ, +* based on affinity masks index of what ?
Index within the group of the closest IRQ for this CPU; also the queue index.
[...]
Quoted 8 lines
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 529
Quoted 5 lines
include/linux/irq.h +5
-0+static inline u16 irq_group_get_index (struct irq_group *group,int cpu) +{ +return group->closest[cpu].index; +} +So you have an accessor function for closest[cpu].index. Are the other members meant to be accessible directly by random driver ?
No, they should not be touched at all.
Quoted 17 lines
Quoted 4 lines
include/linux/irq.h +0
-0#else /* !CONFIG_SMP */ static inline bool alloc_desc_masks(struct irq_desc *desc, int node,include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 557
Quoted 10 lines
include/linux/irq.h +7
-0@@ -519,6 +548,29 @@ static inline void free_desc_masks(struct irq_desc *old_desc, struct irq_desc *new_desc) { } + +struct irq_group { +}; + +static inline struct irq_group *alloc_irq_group (unsigned int size,gfp_t flags) +{ +static struct irq_group dummy;That will create one static instance per callsite. Is that on purpose? If yes, it needs a damned good comment.
Which uses no storage, or maybe 1 byte. I'm open to suggestions for a better dummy implementation.
Quoted 22 lines
Quoted 22 lines
include/linux/irq.h +1
-0+return &dummy;#endif /* CONFIG_SMP */ #endif /* _LINUX_IRQ_H */ kernel/irq/manage.c +9
-0@@ -100,6 +100,154 @@ void irq_set_thread_affinity(struct irq_desc *desc) } } +static void irq_group_update_neigh (struct irq_group *group, +const struct cpumask *mask, +u16 index,u16 dist) +{ +int cpu; + +for_each_cpu (cpu, mask) { +if (dist < group->closest[cpu].dist) { + group->closest[cpu].index = index;
Quoted 8 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 115
Quoted 4 lines
kernel/irq/manage.c +4
-0+ group->closest[cpu].dist = dist; + } + } +}I have not the faintest idea how group, index and dist are related to each other. And I have no intention to decode the information about that piecewise by reverse engineering that obfuscated code.
I'm happy to write more comments.
index is the index of the IRQ being updated within the group. dist is the maximum distance of that IRQ from the CPUs in the mask.
Quoted 7 lines
Quoted 7 lines
kernel/irq/manage.c +7
-0+static bool irq_group_copy_neigh (struct irq_group *group,int cpu, +const struct cpumask *mask,u16 dist) +{ +int neigh; + +for_each_cpu (neigh, mask) { +if (group->closest[neigh].dist <= dist) {
Quoted 8 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 129
Quoted 6 lines
kernel/irq/manage.c +6
-0+ group->closest[cpu].index = group->closest[neigh].index; + group->closest[cpu].dist = dist; +return true; + } + } +return false;What's the reason for copying or not ?
We're trying to find closer IRQs for the CPUs which were removed from the affinity of the IRQ currently being moved.
[...]
Quoted 5 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 181
Quoted 2 lines
kernel/irq/manage.c +2
-0+/* We could continue into NUMA node distances, but for now +* we give up. */ What are the consequences of giving up ? Does not happen ? Should not happen ? Will break ? Don't care ? ....
Poor choice of closest IRQ. Nothing disastrous.
Quoted 8 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 131
Quoted 1 line
kernel/irq/manage.c +1
-0+ }This is called from
irq_set_affinity()withdesc->lockheld and interrupts disabled. You're not serious about that, are you ?Damned, there are two iterations over each online cpu and another one over the affinity mask. Did you ever extrapolate how long that runs on a really large machine ?
No, and I know this sucks. So, do you want to suggest anything better? Should I look at building those CPU distance maps (trading space for time)?
Quoted 11 lines
If CPU affinity of an IRQ changes, then it does not matter if one or two interrupts end up in the wrong space. Those changes are not happening every other interrupt.
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 187
Quoted 5 lines
kernel/irq/manage.c +5
-0+} + +/** +* alloc_irq_group - allocate IRQ group +* @size: Size of the group size of what ? I guess number of interrupts, right ?
Right.
Quoted 9 lines
Quoted 9 lines
kernel/irq/manage.c +9
-0+* @flags: Allocation flags e.g. %GFP_KERNEL +*/ +struct irq_group *alloc_irq_group (unsigned int size,gfp_t flags) +{ +struct irq_group *group = +kzalloc (sizeof (*group) + size *sizeof (group->irq[0 ]), flags); +int cpu; + +if (!group)
Quoted 8 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 201
Quoted 5 lines
kernel/irq/manage.c +5
-0+return NULL ; + +/* Initially assign CPUs to IRQs on a rota */ +for (cpu =0 ; cpu <NR_CPUS ; cpu++) { + group->closest[cpu].index = cpu % size;So here we randomly assign index with the lower cpu numbers no matter whether they are online or possible ?
Doesn't matter, because we're mapping CPUs to IRQs not the other way round.
Quoted 10 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 208
Quoted 7 lines
kernel/irq/manage.c +7
-0+ group->closest[cpu].dist =IRQ_CPU_DIST_INF ; + } + + group->size = size; +return group; +} +EXPORT_SYMBOL (alloc_irq_group);
EXPORT_SYMBOL_GPLif at all. Same for the other exports
Quoted 6 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 212
Quoted 4 lines
kernel/irq/manage.c +4
-0+ +/** +* free_irq_group - free IRQ group +* @group: IRQ group allocated with alloc_irq_group(), or %NULL How is this serialized or sanity checked against
free_irq()?
free_irq() should be called first, for all IRQs in the group. The IRQs should then be really freed with pci_disable_msix() or similar.
Quoted 23 lines
Quoted 23 lines
kernel/irq/manage.c +23
-0+*/ +void free_irq_group (struct irq_group *group) +{ +struct irq_desc *desc; +unsigned int i; + +if (!group) +return ; + +/* Remove all descriptors from the group */ +for (i =0 ; i < group->used; i++) { + desc = group->irq[i]; +BUG_ON (desc->group != group || desc->group_index != i); + desc->group =NULL ; + } + +kfree (group); +} +EXPORT_SYMBOL (free_irq_group); + +/** +* irq_group_add - add IRQ to a group +* @group: IRQ group allocated with alloc_irq_group()
Quoted 8 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 240
Quoted 5 lines
kernel/irq/manage.c +5
-0+* @irq: Interrupt to add to group +*/ +void irq_group_add (struct irq_group *group,unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq);Again, how is this serialized against anything else fiddling with irq_desc[irq]?
The driver should call this after allocating IRQs with pci_enable_msix() or similar and before setting the handlers with request_irq(). Is that sufficient?
Ben.
Quoted 12 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 249
Quoted 8 lines
kernel/irq/manage.c +8
-0+BUG_ON (desc->group); +BUG_ON (group->used >= group->size); + + desc->group = group; + desc->group_index = group->used; + group->irq[group->used++] = desc; +} +EXPORT_SYMBOL (irq_group_add);Thanks,
tglx
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices
Quoted 12 lines · Ben Hutchings
Quoted 10 lines · Thomas Gleixner
Quoted 4 lines
queue for which the response will be handled on the same or a nearby CPU. IRQ groups hold a mapping of CPU to IRQ which will be updated based on the inverse of IRQ CPU-affinities plus CPU topology information.
Can you please explain, why you need that reverse mapping including the below code ? What problem does this solve which can not be deduced by the exisiting information/infrastructure ? And why is that reverse mapping tied to interrupts and not something which we want to see in some generic available (library) code ?
Quoted 3 lines
Are you thinking of a per-CPU distance map? That seems like it would be useful to have. I wonder about the storage requirements on larger systems.
That might be a problem, but we should at least look into this.
But even if a prebuilt map is too expensive storage wise, then the functions which build the lookup map might be useful for similar distance lookup scenarios as well.
Quoted 11 lines
Quoted 6 lines
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 557
Quoted 3 lines
include/linux/irq.h +3
-0+static inline struct irq_group *alloc_irq_group (unsigned int size,gfp_t flags) +{ +static struct irq_group dummy;That will create one static instance per callsite. Is that on purpose? If yes, it needs a damned good comment.
Which uses no storage, or maybe 1 byte. I'm open to suggestions for a better dummy implementation.
Quoted 1 line
include/linux/irq.h static inline void free_desc_masks(struct irq_desc *old_desc, · line 558
Quoted 1 line
include/linux/irq.h +1
-0+return &dummy;
Why does it need to return a real struct instead of NULL ?
Quoted 5 lines
Quoted 5 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ }This is called from
irq_set_affinity()withdesc->lockheld and interrupts disabled. You're not serious about that, are you ?
Quoted 7 lines
Quoted 3 lines
Damned, there are two iterations over each online cpu and another one over the affinity mask. Did you ever extrapolate how long that runs on a really large machine ?
No, and I know this sucks. So, do you want to suggest anything better? Should I look at building those CPU distance maps (trading space for time)?
The point is that there is no reason why this code needs to run in a spinlocked irq disabled reason. I know why your implementation needs it, because it protects you against module unload etc. Come on, we have enough infrastructure to control the lifetime of objects in sane ways and not by hijacking a heavy lock and disabling interrupts.
Btw, how does this code deal with concurrent affinity setting of multiple irqs in the group ? Random number generator ?
Quoted 1 line
Quoted 1 line
Quoted 1 line
+ * @flags: Allocation flags e.g. %GFP_KERNEL
Why do we need flags here? That code gets called in a device setup routine and not from random context.
Quoted 11 lines
Quoted 10 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 212
Quoted 8 lines
+ */
kernel/irq/manage.c +6
-0+struct irq_group *alloc_irq_group (unsigned int size,gfp_t flags) +{ + +/** +* free_irq_group - free IRQ group +* @group: IRQ group allocated with alloc_irq_group(), or %NULL How is this serialized or sanity checked against
free_irq()?
Quoted 2 lines
free_irq()should be called first, for all IRQs in the group. The IRQs should then be really freed withpci_disable_msix()or similar.
I explained that several times, that I do not care about what should be called in which order and what not. Driver coders get it wrong all the time, then it explodes in the genirq code and I get the crap to debug. No, thanks.
Also if you call free_irq() first, then you still have a reference to that very irq descriptor in your group and a reference to the group in the irq descriptor. Brilliant, as the irq descriptor can be reused immediately after free_irq().
Also it's not only problematic against free_irq, it's damned racy against a concurrent affinity setting as well. And that's not controlled by "should be called" at all.
Quoted 12 lines
Quoted 11 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 240
Quoted 8 lines
kernel/irq/manage.c +8
-0+/** +* irq_group_add - add IRQ to a group +* @group: IRQ group allocated with alloc_irq_group() +* @irq: Interrupt to add to group +*/ +void irq_group_add (struct irq_group *group,unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq);Again, how is this serialized against anything else fiddling with irq_desc[irq]?
Quoted 3 lines
The driver should call this after allocating IRQs with
pci_enable_msix()or similar and before setting the handlers withrequest_irq(). Is that sufficient?
"should call" is _NEVER_ sufficient. fiddling with irq_desc w/o holding the lock is a nono, period.
Quoted 6 lines
Quoted 6 lines
kernel/irq/manage.c void irq_set_thread_affinity(struct irq_desc *desc) · line 247
Quoted 6 lines
kernel/irq/manage.c +6
-0+BUG_ON (desc->group); +BUG_ON (group->used >= group->size); + + desc->group = group; + desc->group_index = group->used; + group->irq[group->used++] = desc;
Forgot to ask yesterday. Why do we need to store irq desc in the group ? For free_irq_group() I guess, but I explained already why this code is broken. And it's even more broken when an irq is moved to a different node and CONFIG_NUMA_IRQ_DESC=y.
Lemme summarize what I understand so far. For multiqueue devices with several queues and interrupts you want to lookup the closest queue/irq to the cpu on which you are currently running to avoid expensive cross node memory access. Now you need to follow affinity changes for this lookup table.
The basic idea of your code is ok. The lookup table itself is fine, but the integration into the genirq code sucks. What needs to be done:
1) Remove the references to irq_desc in the group. They are not needed and will never work. The group does not need any information about the irq number and the irq descriptor.
2) Add proper refcounting to struct irq_group so it can be accessed outside of irq_desc->lock.
3) Move the update of the map outside of irq_desc->lock and the irq disabled region. Update the map in setup_affinity() as well. That code can be called from various atomic contexts, so it's probably the best thing to delegate the update to a workqueue or such unless you come up with a nice prebuilt lookup table.
4) Add comments so the code is understandable for mere mortals
5) Add proper install/free handling. Hint: affinity_hint
6) All modifications of irq_desc need a check whether irq_to_desc returned a valid pointer and must hold desc->lock
7) Deal with concurrent updates of multiple irqs in a group (at least a comment why the code does not need serialization)
8) Move the code to kernel/irq/irqgroup.c or some other sensible name and make it configurable so embedded folks don't have to carry it around as useless binary bloat.
Thanks,
tglx
Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices
Quoted 31 lines · Thomas Gleixner
Quoted 1 line · Ben Hutchings
On Mon, 2010-09-20 at 23:27 +0200, Thomas Gleixner wrote:
Lemme summarize what I understand so far. For multiqueue devices with several queues and interrupts you want to lookup the closest queue/irq to the cpu on which you are currently running to avoid expensive cross node memory access. Now you need to follow affinity changes for this lookup table.
The basic idea of your code is ok. The lookup table itself is fine, but the integration into the genirq code sucks. What needs to be done:
1) Remove the references to
irq_descin the group. They are not needed and will never work. The group does not need any information about the irq number and the irq descriptor.2) Add proper refcounting to struct
irq_groupso it can be accessed outside ofirq_desc->lock.3) Move the update of the map outside of
irq_desc->lockand the irq disabled region. Update the map insetup_affinity()as well. That code can be called from various atomic contexts, so it's probably the best thing to delegate the update to a workqueue or such unless you come up with a nice prebuilt lookup table.4) Add comments so the code is understandable for mere mortals
5) Add proper install/free handling. Hint:
affinity_hint6) All modifications of
irq_descneed a check whetherirq_to_descreturned a valid pointer and must holddesc->lock
Quoted 6 lines
7) Deal with concurrent updates of multiple irqs in a group (at least a comment why the code does not need serialization)
8) Move the code to
kernel/irq/irqgroup.cor some other sensible name and make it configurable so embedded folks don't have to carry it around as useless binary bloat.
Talked to Peter about it and we came to the conclusion, that we should just provide a callback infrastructure in the irq code which does not care about the action behind it. That's going to solve #1,#2,#3,#5,#6 and parts of #8
That queue/index map code should move to lib/ or some other appropriate place so it can be shared with storage or whatever is going to grow multiqueue. comments #4, #7, #8 (s@kernel/irq@lib/@) above still apply :)
The modification to the genirq code would be based on registering
struct irq_affinity_callback {
unsigned int irq;
struct kref kref;
struct work work;
void (*callback)(struct irq_affinity_callback *, const cpumask_t *mask);
void (*release)(struct kref *ref);
};for an interrupt via
int irq_set_affinity_callback(unsigned int irq,
struct irq_affinity_callback *cb);
That function can be called with cb=NULL to remove the callback. if cb!=NULL, irq, kref and work are initialized.
The genirq code schedules work when affinity changes and the callback struct pointer is non NULL. The callback is then called from the worker thread with a copy of the irq affinity mask.
So on any affinity change, we do with desc->lock held:
if (desc->affinity_callback) {
kref_get(&desc->affinity_callback->kref);
schedule_work(&desc->affinity_callback->work);}
The worker function in the genirq code does:
{
struct irq_affinity_callback *cb;
struct irq_desc *desc;
cpu_mask_t cpumask;
cb = container_of(work, struct irq_affinity_callback, work);
desc = irq_to_desc(cb->irq);
if (!desc)
goto out;
if (!alloc_cpumask_var(&cpumask, GFP_KERNEL)
goto out;
raw_spin_lock_irqsave(&desc->lock, flags);
cpumask_copy(cpumask, desc->affinity);
raw_spin_unlock_irqrestore(&desc->lock, flags);
cb->callback(cb, cpumask);
free_cpumask_var(cpumask);
out:
kref_put(&cb->kref, cb->release);
}That allows you to do all kind of magic in thread context, updating the queue map, reallocating queue memory when the node affinity changes (I know that you want to), go wild.
Thoughts ?
tglx
Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices
On Tue, 2010-09-21 at 21:04 +0200, Thomas Gleixner wrote:
[...]
Quoted 9 lines
Talked to Peter about it and we came to the conclusion, that we should just provide a callback infrastructure in the irq code which does not care about the action behind it. That's going to solve #1,#2,#3,#5,#6 and parts of #8
That queue/index map code should move to
lib/or some other appropriate place so it can be shared with storage or whatever is going to grow multiqueue. comments #4, #7, #8 (s@kernel/irq@lib/@) above still apply :)
OK.
Quoted 10 lines
The modification to the genirq code would be based on registering
struct irq_affinity_callback { unsigned int irq; struct kref kref; struct work work; void (*callback)(struct irq_affinity_callback *, const cpumask_t *mask); void (*release)(struct kref *ref); };
Quoted 7 lines
for an interrupt via
int irq_set_affinity_callback(unsigned int irq,
struct irq_affinity_callback *cb);That function can be called with cb=NULL to remove the callback. if cb!=NULL, irq, kref and work are initialized.
When should it be called, relative to {request,free}_irq() and pci_{disable,enable}_msix()?
[...]
Quoted 3 lines
That allows you to do all kind of magic in thread context, updating the queue map, reallocating queue memory when the node affinity changes (I know that you want to), go wild.
I definitely don't want to reallocate queues if node affinity of the IRQ is changed by irqbalance, because this will disrupt traffic. So changing the node affinity of queues has to be a separate operation.
Quoted 1 line
Thoughts ?
This does look like something I can use, thanks.
Ben.
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Re: [RFC][PATCH 1/4] IRQ: IRQ groups for multiqueue devices
Quoted 26 lines · Ben Hutchings
On Tue, 2010-09-21 at 21:04 +0200, Thomas Gleixner wrote:
[...]Quoted 9 lines
Talked to Peter about it and we came to the conclusion, that we should just provide a callback infrastructure in the irq code which does not care about the action behind it. That's going to solve #1,#2,#3,#5,#6 and parts of #8
That queue/index map code should move to
lib/or some other appropriate place so it can be shared with storage or whatever is going to grow multiqueue. comments #4, #7, #8 (s@kernel/irq@lib/@) above still apply :)OK.
Quoted 12 lines
The modification to the genirq code would be based on registering
struct irq_affinity_callback { unsigned int irq; struct kref kref; struct work work; void (*callback)(struct irq_affinity_callback *, const cpumask_t *mask); void (*release)(struct kref *ref); };for an interrupt via
Quoted 8 lines
Quoted 5 lines
int irq_set_affinity_callback(unsigned int irq,
struct irq_affinity_callback *cb);That function can be called with cb=NULL to remove the callback. if cb!=NULL, irq, kref and work are initialized.
When should it be called, relative to {request,free}_irq() and pci_{disable,enable}_msix()?
It should be called before request_irq and before free_irq. free_irq will warn when the pointer is !NULL.
Quoted 8 lines
[...]
Quoted 3 lines
That allows you to do all kind of magic in thread context, updating the queue map, reallocating queue memory when the node affinity changes (I know that you want to), go wild.
I definitely don't want to reallocate queues if node affinity of the IRQ is changed by irqbalance, because this will disrupt traffic. So changing the node affinity of queues has to be a separate operation.
Fair enough.
Quoted 3 lines
Quoted 1 line
Thoughts ?
This does look like something I can use, thanks.
Will look into it in the next days.
Thanks,
tglx
RFC PATCH 2/4 net: RPS: Enable hardware acceleration
In reply to Ben HutchingsAllow drivers for multiqueue hardware with flow filter tables to accelerate RFS. The driver must:
1. Set net_device::rx_irq_group to an irq_group of the RX completion IRQs (in queue order). This will provide a mapping from CPUs to the queues for which completions are handled nearest to them.
2. Implement net_device_ops::ndo_rx_flow_steer. This operation adds or replaces a filter steering the given flow to the given RX queue, if possible.
3. Periodically remove filters for which rps_may_expire_flow() returns true.
---
The heuristic in rps_may_expire_flow() is quite possibly bogus. I'm not even sure whether expiry of flows should be triggered by the driver or from the RPS/RFS core. Any better ideas on how to do this?
Ben.
include/linux/netdevice.h | 29 +++++++++++++-- net/core/dev.c | 88 +++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 108 insertions(+), 9 deletions(-)
include/linux/netdevice.h +26 -3
@@ -524,14 +524,16 @@ struct rps_map {#define RPS_MAP_SIZE (_num) (sizeof(struct rps_map) + (_num * sizeof(u16)))/* -* The rps_dev_flow structure contains the mapping of a flow to a CPU and the -* tail pointer for that CPU's input queue at the time of last enqueue. +* The rps_dev_flow structure contains the mapping of a flow to a CPU, the +* tail pointer for that CPU's input queue at the time of last enqueue, and +* a hardware filter index. */ struct rps_dev_flow {u16 cpu; -u16 fill; +u16 filter;unsigned int last_qtail; }; +#define RPS_NO_FILTER 0xffff/* * The rps_dev_flow_table structure contains a table of flow mappings.
@@ -581,6 +583,9 @@ static inline void rps_reset_sock_flow(struct rps_sock_flow_table *table,extern struct rps_sock_flow_table *rps_sock_flow_table; +extern bool rps_may_expire_flow (struct net_device *dev,u16 rxq_index, +u32 flow_id,u16 filter_id); +/* This structure contains an instance of an RX queue. */ struct netdev_rx_queue {struct rps_map *rps_map;
@@ -701,6 +706,13 @@ struct netdev_rx_queue {* int (*ndo_set_vf_port)(struct net_device *dev, int vf, * struct nlattr *port[]); * int (*ndo_get_vf_port)(struct net_device *dev, int vf, struct sk_buff *skb); +* +* RFS acceleration. +* int (*ndo_rx_flow_steer)(struct net_device *dev, const struct sk_buff *skb, +* u16 rxq_index, u32 flow_id); +* Set hardware filter for RFS. rxq_index is the target queue index; +* flow_id is a flow ID to be passed to rps_may_expire_flow() later. +* Return the filter ID on success, or a negative error code. */ #define HAVE_NET_DEVICE_OPS struct net_device_ops {
@@ -773,6 +785,12 @@ struct net_device_ops {int (*ndo_fcoe_get_wwn)(struct net_device *dev,u64 *wwn,int type);#endif +#ifdef CONFIG_RPS +int (*ndo_rx_flow_steer)(struct net_device *dev, +const struct sk_buff *skb, +u16 rxq_index, +u32 flow_id); +#endif };/*
@@ -978,6 +996,11 @@ struct net_device {/* Number of RX queues allocated at alloc_netdev_mq() time */ unsigned int num_rx_queues; + +/* IRQ group for RX completion interrupts, indexed by RX queue +* number. Assigned by driver. This must only be set if the +* ndo_rx_flow_steer operation is defined. */ +struct irq_group *rx_irq_group;#endif rx_handler_func_t *rx_handler;
net/core/dev.c +82 -6
106 lines not on this page · raw message
Signature · 8 lines
1.7.2.1 -- Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
RFC PATCH 3/4 sfc: Implement RFS acceleration
In reply to Ben Hutchings---
This depends on today's patch series for net-next-2.6.
Ben.
drivers/net/sfc/efx.c | 49 ++++++++++++++++++---- drivers/net/sfc/efx.h | 9 ++++ drivers/net/sfc/filter.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 149 insertions(+), 9 deletions(-)
drivers/net/sfc/efx.c +40 -9
@@ -127,6 +127,8 @@ static int napi_weight = 64;* monitor. On Falcon-based NICs, this will: * - Check the on-board hardware monitor; * - Poll the link state and reconfigure the hardware as necessary. +* If RFS is enabled, this will scan part of the RX IP filter table and +* remove filters for inactive flows. */ unsigned int efx_monitor_interval =1 *HZ ;
@@ -1174,7 +1176,7 @@ static int efx_wanted_channels(void)/* Probe the number and type of interrupts we are able to obtain, and * the resulting numbers of channels and RX queues. */ -static void efx_probe_interrupts (struct efx_nic *efx) +static int efx_probe_interrupts (struct efx_nic *efx) {int max_channels =min_t (int, efx->type->phys_addr_channels,EFX_MAX_CHANNELS );
@@ -1216,6 +1218,17 @@ static void efx_probe_interrupts(struct efx_nic *efx) efx->n_tx_channels = efx->n_channels; efx->n_rx_channels = efx->n_channels; } +#ifdef CONFIG_RPS + efx->net_dev->rx_irq_group = +alloc_irq_group (efx->n_rx_channels,GFP_KERNEL ); +if (!efx->net_dev->rx_irq_group) { +pci_disable_msix (efx->pci_dev); +return -ENOMEM ; + } +for (i =0 ; i < efx->n_rx_channels; i++) +irq_group_add (efx->net_dev->rx_irq_group, + xentries[i].vector); +#endif for (i =0 ; i < n_channels; i++)efx_get_channel (efx, i)->irq = xentries[i].vector;
@@ -1249,6 +1262,8 @@ static void efx_probe_interrupts(struct efx_nic *efx) efx->n_tx_channels =1 ; efx->legacy_irq = efx->pci_dev->irq; } + +return 0 ; }static void efx_remove_interrupts (struct efx_nic *efx)
@@ -1258,6 +1273,10 @@ static void efx_remove_interrupts(struct efx_nic *efx)/* Remove MSI/MSI-X interrupts */ efx_for_each_channel (channel, efx) channel->irq =0 ; +#ifdef CONFIG_RPS +free_irq_group (efx->net_dev->rx_irq_group); + efx->net_dev->rx_irq_group =NULL ; +#endif pci_disable_msi (efx->pci_dev);pci_disable_msix (efx->pci_dev);
@@ -1307,7 +1326,9 @@ static int efx_probe_nic(struct efx_nic *efx)/* Determine the number of channels and queues by trying to hook * in MSI-X interrupts. */ -efx_probe_interrupts (efx); + rc =efx_probe_interrupts (efx); +if (rc) + goto fail;if (efx->n_channels >1 )get_random_bytes (&efx->rx_hash_key,sizeof (efx->rx_hash_key));
@@ -1322,6 +1343,10 @@ static int efx_probe_nic(struct efx_nic *efx)efx_init_irq_moderation (efx, tx_irq_mod_usec, rx_irq_mod_usec, true);return 0 ; + +fail: + efx->type->remove (efx); +return rc; }static void efx_remove_nic (struct efx_nic *efx)
@@ -1419,13 +1444,15 @@ static void efx_start_all(struct efx_nic *efx)if (efx->reset_pending !=RESET_TYPE_NONE )efx_mcdi_mode_poll (efx); -/* Start the hardware monitor if there is one. Otherwise (we're link -* event driven), we have to poll the PHY because after an event queue -* flush, we could have a missed a link state change */ -if (efx->type->monitor !=NULL ) { +/* Start the periodic monitor if necessary */ +if (efx->type->monitor ||efx_filter_rfs_enabled ())queue_delayed_work (efx->workqueue, &efx->monitor_work, efx_monitor_interval); - }else { + +/* If we normally rely on link state events, we have to poll +* the PHY because after an event queue flush, we could have a +* missed a link state change */ +if (!efx->type->monitor) {mutex_lock (&efx->mac_lock);if (efx->phy_op->poll (efx))efx_link_status_changed (efx);
@@ -1556,17 +1583,18 @@ static void efx_monitor(struct work_struct *data)netif_vdbg (efx, timer, efx->net_dev,"hardware monitor executing on CPU %d\n" ,raw_smp_processor_id ()); -BUG_ON (efx->type->monitor ==NULL );/* If the mac_lock is already held then it is likely a port * reconfiguration is already in place, which will likely do * most of the work of monitor() anyway. */ -if (mutex_trylock (&efx->mac_lock)) { +if (efx->type->monitor &&mutex_trylock (&efx->mac_lock)) {if (efx->port_enabled) efx->type->monitor (efx);mutex_unlock (&efx->mac_lock); } +efx_filter_rfs_expire (efx); +queue_delayed_work (efx->workqueue, &efx->monitor_work, efx_monitor_interval); }
@@ -1849,6 +1877,9 @@ static const struct net_device_ops efx_netdev_ops = {#ifdef CONFIG_NET_POLL_CONTROLLER .ndo_poll_controller = efx_netpoll,#endif +#ifdef CONFIG_RPS + .ndo_rx_flow_steer = efx_filter_rfs, +#endif };static void efx_update_name (struct efx_nic *efx)
drivers/net/sfc/efx.h +9 -0
@@ -77,6 +77,15 @@ extern int efx_filter_remove_filter(struct efx_nic *efx,extern void efx_filter_table_clear (struct efx_nic *efx,enum efx_filter_table_id table_id,enum efx_filter_priority priority); +#ifdef CONFIG_RPS +extern int efx_filter_rfs (struct net_device *net_dev,const struct sk_buff *skb, +u16 rxq_index,u32 flow_id); +extern void efx_filter_rfs_expire (struct efx_nic *efx); +#define efx_filter_rfs_enabled () 1 +#else +static inline void efx_filter_rfs_expire (struct efx_nic *efx) {} +#define efx_filter_rfs_enabled () 0 +#endif /* Channels */ extern void efx_process_channel_now (struct efx_channel *channel);
drivers/net/sfc/filter.c +100 -0
123 lines not on this page · raw message
Signature · 8 lines
1.7.2.1 -- Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
RFC PATCH 4/4 sfc/RFS/irq_group debug output
In reply to Ben HutchingsJust some logging I found useful.
Ben.
--- drivers/net/sfc/filter.c | 11 ++++++++++- kernel/irq/manage.c | 21 +++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletions(-)
drivers/net/sfc/filter.c +10 -1
@@ -506,6 +506,11 @@ int efx_filter_rfs(struct net_device *net_dev, const struct sk_buff *skb, rc =efx_filter_insert_filter (efx, &spec, true);if (rc >=0 ) state->rps_flow_id[rc] = flow_id; +netif_info (efx, rx_status, efx->net_dev, +"steering %s %pI4:%u:%pI4:%u to queue %u [flow %u filter %d]\n" , + (ip->protocol ==IPPROTO_TCP ) ?"TCP" :"UDP" , + &ip->saddr,ntohs (ports[0 ]), &ip->daddr,ntohs (ports[1 ]), + rxq_index, flow_id, rc);return rc; }
@@ -529,8 +534,12 @@ void efx_filter_rfs_expire(struct efx_nic *efx) table->spec[index].priority == EFX_FILTER_PRI_HINT && rps_may_expire_flow(efx->net_dev, table->spec[index].dmaq_id, - state->rps_flow_id[index], index)) + state->rps_flow_id[index], index)) { +netif_info (efx, rx_status, efx->net_dev, +"expiring filter %d [flow %u]\n" , + index, state->rps_flow_id[index]);efx_filter_table_clear_entry (efx, table, index); + } index = (index +1 ) & mask; }
kernel/irq/manage.c +21 -0
@@ -129,6 +129,21 @@ static bool irq_group_copy_neigh(struct irq_group *group, int cpu,return false; } +static void print_irq_group (const struct irq_group *group,const char *prefix) +{ +unsigned index; +int cpu; + +pr_info ("irq_group %p, %s:\n" , group, prefix); + +for_each_possible_cpu (cpu) { + index = group->closest[cpu].index; +pr_info ("cpu %d -> index %u (IRQ %u; distance %u)\n" , + cpu, index, group->irq[index]->irq, + group->closest[cpu].dist); + } +} +/* Update the per-CPU closest IRQs following a change of affinity */ static void irq_update_group (struct irq_desc *desc,const struct cpumask *affinity)
@@ -145,6 +160,8 @@ irq_update_group(struct irq_desc *desc, const struct cpumask *affinity)if (group->closest[cpu].index == index) group->closest[cpu].dist =IRQ_CPU_DIST_INF ; +print_irq_group (group,"after invalidating old distances" ); +/* * Set this as the closest IRQ for all CPUs in the affinity mask, * plus the following CPUs if they don't have a closer IRQ:
@@ -163,6 +180,8 @@ irq_update_group(struct irq_desc *desc, const struct cpumask *affinity) index, 3); } +print_irq_group (group,"after updating neighbours" ); +/* Find new closest IRQ for any CPUs left with invalid distances */ for_each_online_cpu (cpu) { if (!(group->closest[cpu].index == index &&
@@ -180,6 +199,8 @@ irq_update_group(struct irq_desc *desc, const struct cpumask *affinity)/* We could continue into NUMA node distances, but for now * we give up. */ } + +print_irq_group (group,"after copying neighbours" ); }/**
Signature · 7 lines
1.7.2.1 -- Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Thanks Ben, this does look interesting. We'll try to take a look.
Quoted 39 lines · Ben Hutchings
Tom,
This patch series extends RFS to use hardware RX filters where available. Depending on the number of hardware RX queues and their IRQs' affinity, this should reduce the need for IPIs or at least get packets delivered to the right NUMA node.
I've implemented the driver side of this for our hardware, though I don't know whether you have any of that to test on. I would be very interested to know how much this can help in the sort of cases where you use RFS.
Ben.
Ben Hutchings (4):
IRQ: IRQ groups for multiqueue devices
net: RPS: Enable hardware acceleration
sfc: Implement RFS acceleration
sfc/RFS/irq_group debug outputdrivers/net/sfc/efx.c | 49 +++++++++++---
drivers/net/sfc/efx.h| 9 +++
drivers/net/sfc/filter.c | 109 +++++++++++++++++++++++++++++include/linux/irq.h| 52 ++++++++++++++include/linux/netdevice.h| 29 +++++++-
kernel/irq/manage.c | 170 +++++++++++++++++++++++++++++++++++++++++++++net/core/dev.c| 88 ++++++++++++++++++++++--7 files changed, 488 insertions(+), 18 deletions(-)--
1.7.2.1--
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.