export irq_set/get_affinity() for multiqueue network drivers
Subsystems (3)
RFC 1/1 export irq_set/get_affinity() for multiqueue network drivers
Hello,
Is there any way to setup IRQ masks from within a driver? myri10ge currently relies on an external script (writing in /proc/irq/*/smp_affinity) to bind each queue/MSI-X to a different processor. By default, Linux will either:
* round-robin the interrupts (killing the benefit of DCA for instance)
* put all IRQs on the same CPU (killing much of the benefit of multislices)
With more and more drivers using multiqueues, I think we need a nice way to bind MSI-X from within the drivers. I am not sure what's best, the attached (untested) patch would just export the existing irq_set_affinity() and add irq_get_affinity(). Comments?
thanks,
Brice
[PATCH] export irq_set/get_affinity() to modules
Export irq_set_affinity() and add/export irq_get_affinity() so that network drivers may access MSI-X interrupt masks to bind multiqueues to different CPUs. Otherwise Linux will either:
* round-robin the interrupts (killing the benefit of DCA, for instance)
* put all IRQs on the same CPU (killing much of the benefit of multislices)
--- include/linux/interrupt.h | 3 +++ kernel/irq/manage.c | 15 +++++++++++++++ 2 files changed, 18 insertions(+)
include/linux/interrupt.h +3 -0
@@ -107,6 +107,7 @@ extern void enable_irq(unsigned int irq);extern cpumask_t irq_default_affinity;extern int irq_set_affinity (unsigned int irq,cpumask_t cpumask); +extern int irq_get_affinity (unsigned int irq,cpumask_t *cpumask);extern int irq_can_set_affinity (unsigned int irq);extern int irq_select_affinity (unsigned int irq);
@@ -117,6 +118,8 @@ static inline int irq_set_affinity(unsigned int irq, cpumask_t cpumask)return -EINVAL ; } +static inline int irq_get_affinity (unsigned int irq) {return 0 ; } +static inline int irq_can_set_affinity (unsigned int irq) {return 0 ;
kernel/irq/manage.c +15 -0
@@ -96,6 +96,21 @@ int irq_set_affinity(unsigned int irq, cpumask_t cpumask)#endif return 0 ; } +EXPORT_SYMBOL (irq_set_affinity); + +int irq_get_affinity (unsigned int irq,cpumask_t *cpumask) +{ +struct irq_desc *desc = irq_desc + irq; +cpumask_t *mask = &desc->affinity; + +#ifdef CONFIG_GENERIC_PENDING_IRQ +if (desc->status &IRQ_MOVE_PENDING ) + mask = &desc->pending_mask; +#endif + memcpy(cpumask, mask, sizeof(*mask); +return 0 ; +} +EXPORT_SYMBOL (irq_get_affinity);#ifndef CONFIG_AUTO_IRQ_AFFINITY /*
David Miller
maintainerFrom: Brice Goglin <[email protected]>
Date: Thu, 28 Aug 2008 22:21:53 +0200
Quoted 4 lines
With more and more drivers using multiqueues, I think we need a nice way to bind MSI-X from within the drivers. I am not sure what's best, the attached (untested) patch would just export the existing
irq_set_affinity()and addirq_get_affinity(). Comments?
I think we should rather have some kind of generic thing in the IRQ layer that allows specifying the usage model of the device's interrupts, so that the IRQ layer can choose a default affinities.
I never notice any of this complete insanity on sparc64 because we flat spread out all of the interrupts across the machine.
What we don't want it drivers choosing IRQ affinity settings, they have no idea about NUMA topology, what NUMA node the PCI controller sits behind, what cpus are there, etc. and without that kind of knowledge you cannot possible make affinity decisions properly.
Quoted 4 lines · David Miller
I think we should rather have some kind of generic thing in the IRQ layer that allows specifying the usage model of the device's interrupts, so that the IRQ layer can choose a default affinities.
Quoted 8 lines
I never notice any of this complete insanity on sparc64 because we flat spread out all of the interrupts across the machine.
What we don't want it drivers choosing IRQ affinity settings, they have no idea about NUMA topology, what NUMA node the PCI controller sits behind, what cpus are there, etc. and without that kind of knowledge you cannot possible make affinity decisions properly.
As long as we get something better than the current behavior, I am fine with it :)
Brice
Quoted 8 lines · On Thu, 28 Aug 2008 22:21:53 +0200 Brice Goglin <[email protected]> wrote:
Hello,
Is there any way to setup IRQ masks from within a driver? myri10ge currently relies on an external script (writing in
/proc/irq/*/smp_affinity) to bind each queue/MSI-X to a different processor. By default, Linux will either:
* round-robin the interrupts (killing the benefit of DCA for instance)
* put all IRQs on the same CPU (killing much of th
* do the right thing with the userspace irq balancer
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 5 lines · Arjan van de Ven
Quoted 2 lines · On Thu, 28 Aug 2008 22:21:53 +0200 Brice Goglin <[email protected]> wrote:
Hello,
Quoted 8 lines
Quoted 6 lines
Is there any way to setup IRQ masks from within a driver? myri10ge currently relies on an external script (writing in
/proc/irq/*/smp_affinity) to bind each queue/MSI-X to a different processor. By default, Linux will either:
* round-robin the interrupts (killing the benefit of DCA for instance)
* put all IRQs on the same CPU (killing much of th* do the right thing with the userspace irq balancer
It probably also needs to be hooked up the sched_mc_power_savings When the switch is on the interrupts shouldn't be spread out over that many sockets.
Does it need callbacks to change the interrupts when that variable changes?
Also I suspect handling SMT explicitely is a good idea. e.g. I would always set the affinity to all thread siblings in a core, not just a single one, because context switch is very cheap between them.
-Andi
Signature · 1 line
[email protected]
Quoted 15 lines · On Fri, 29 Aug 2008 18:48:12 +0200 Andi Kleen <[email protected]> wrote:
Quoted 13 lines · Arjan van de Ven
Quoted 9 lines · On Thu, 28 Aug 2008 22:21:53 +0200 Brice Goglin <[email protected]> wrote:
Hello,
Is there any way to setup IRQ masks from within a driver? myri10ge currently relies on an external script (writing in
/proc/irq/*/smp_affinity) to bind each queue/MSI-X to a different processor. By default, Linux will either:
* round-robin the interrupts (killing the benefit of DCA for instance)
* put all IRQs on the same CPU (killing much of th
Quoted 5 lines
Quoted 1 line
* do the right thing with the userspace irq balancer
It probably also needs to be hooked up the
sched_mc_power_savingsWhen the switch is on the interrupts shouldn't be spread out over that many sockets.
that's what irqbalance already does today.
Quoted 4 lines
Also I suspect handling SMT explicitely is a good idea. e.g. I would always set the affinity to all thread siblings in a core, not just a single one, because context switch is very cheap between them.
that is what irqbalance already does today, at least for what it considers somewhat slower irqs.
for networking it still sucks because the packet reordering logic is per logical cpu so you still don't want to receive packets from the same "stream" over multiple logical cpus.
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 2 lines
Also I suspect handling SMT explicitely is a good idea. e.g. I would
always set the affinity to all thread siblings in a core, not just a single one, because context switch is very cheap between them.
That is true, but don't they also "compete" for pipeline resources?
rick jones