notify user-level IRQ balancer when kernel assigns IRQ affinity
We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).
The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).
I know this needs work, but would like to get comments on the idea before doing more coding and testing.
---
arch/Kconfig +11 -0
@@ -158,4 +158,15 @@ config HAVE_PERF_EVENTS_NMI subsystem. Also has support for calculating CPU cycle events to determine how many clock cycles in a given period. +config NOTIFY_USER_IRQ_BALANCER + bool "Notify user-level IRQ balancer (EXPERIMENTAL)" + default n + depends on NET && X86 + help + When the kernel sets a default CPU affinity for an interrupt, + emit a notification message over a netlink socket so that a + user-level IRQ balancer may re-balance IRQs the way it prefers. + + If unsure, say N. + source "kernel/gcov/Kconfig"
include/linux/irq_notify.h new file +33 -0
@@ -0,0 +1,33 @@ +#ifndef __IRQ_NOTIFY_H +#define __IRQ_NOTIFY_H + +enum { +IRQ_NOTIFY_TYPE_UNSPEC , +IRQ_NOTIFY_TYPE_NOTICE , + __IRQ_NOTIFY_TYPE_MAX, +}; + +#define IRQ_NOTIFY_TYPE_MAX (__IRQ_NOTIFY_TYPE_MAX - 1) + +enum { +IRQ_NOTIFY_CMD_ATTR_UNSPEC , +IRQ_NOTIFY_CMD_ATTR_IRQ , + __IRQ_NOTIFY_CMD_ATTR_MAX, +}; + +#define IRQ_NOTIFY_CMD_ATTR_MAX (__IRQ_NOTIFY_CMD_ATTR_MAX - 1) + +#define IRQ_NOTIFY_GNL_NAME "IRQ_NOTIFY" +#define IRQ_NOTIFY_GNL_VERSION 1 + +#define IRQ_NOTIFY_GRP_ID 1 + +#ifdef CONFIG_NOTIFY_USER_IRQ_BALANCER +extern int irq_notify (int ); +#else +static inline int irq_notify (int ) +{ +} +#endif /* CONFIG_NOTIFY_USER_IRQ_BALANCER */ + +#endif /* __IRQ_NOTIFY_H */
kernel/irq/Makefile +1 -0
@@ -5,3 +5,4 @@ obj-$(CONFIG_PROC_FS) += proc.o obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o obj-$(CONFIG_NUMA_IRQ_DESC) += numa_migrate.o obj-$(CONFIG_PM_SLEEP) += pm.o +obj-$(CONFIG_NOTIFY_USER_IRQ_BALANCER) += irq_notify.o
kernel/irq/irq_notify.c new file +103 -0
103 lines not on this page · raw message
kernel/irq/manage.c +2 -0
@@ -14,6 +14,7 @@#include <linux/interrupt.h> #include <linux/slab.h> #include <linux/sched.h> +#include <linux/irq_notify.h> #include "internals.h"
@@ -178,6 +179,7 @@ static int setup_affinity(unsigned int irq, struct irq_desc *desc)cpumask_and (desc->affinity, cpu_online_mask, irq_default_affinity); set_affinity: desc->chip->set_affinity (irq, desc->affinity); +irq_notify (irq);return 0 ; }
Quoted 25 lines · Arthur Kepner
We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).
The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).
I know this needs work, but would like to get comments on the idea before doing more coding and testing.
Signed-off-by Arthur Kepner<[email protected]>---
arch/Kconfig +0
-0
arch/Kconfig config HAVE_PERF_EVENTS_NMI · line 164
Quoted 7 lines
arch/Kconfig +4
-0@@ -158,4 +158,15 @@ config HAVE_PERF_EVENTS_NMI subsystem. Also has support for calculating CPU cycle events to determine how many clock cycles in a given period. +config NOTIFY_USER_IRQ_BALANCER + bool "Notify user-level IRQ balancer (EXPERIMENTAL)" + default n + depends on NET&& X86
Missing a space before that '&&', but more importantly, what part of the patch is x86 specific?
David Daney
Quoted 10 lines · David Daney
Quoted 10 lines
---
arch/Kconfig +0
-0@@ -158,4 +158,15 @@ config HAVE_PERF_EVENTS_NMI subsystem. Also has support for calculating CPU cycle events to determine how many clock cycles in a given period.
Quoted 9 lines
arch/Kconfig config HAVE_PERF_EVENTS_NMI
Quoted 4 lines
arch/Kconfig +4
-0+config NOTIFY_USER_IRQ_BALANCER + bool "Notify user-level IRQ balancer (EXPERIMENTAL)" + default n + depends on NET&& X86Missing a space before that '&&', but more importantly, what part of the patch is x86 specific?
Hmmm. I see the space in my copy of the message.
Anyway, the X86 dependency is bogus. For a while there was such a dependency, but that's gone now. Just forgot to fix that up. Will do that now.
Signature · 1 line
Arthur
From: Arthur Kepner <[email protected]>
Date: Fri, 3 Sep 2010 14:05:54 -0700
Quoted 12 lines
We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).
The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).
Quoted 4 lines
I know this needs work, but would like to get comments on the idea before doing more coding and testing.
Signed-off-by Arthur Kepner <[email protected]>
As I replied to you on netdev, this can't possibly work.
Interrupt allocation needs to work properly before userspace even starts up. So the allocation management must be inside of the kernel entirely.
On Fri, 3 Sep 2010, Arthur Kepner wrote:
It might be helpful to cc the maintainer of kernel/irq on patches which affect that area.
Quoted 11 lines
We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).
The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).
Errm. If the irq balancer has not been run, then why should some new irq balancer be up and running at this time. In the worst case you need to sort that out during boot _before_ you even hit userspace.
Further this adds just redundant information as user space is already notified about a new device being setup via udev, so what's the point of adding another notification mechanism.
Thanks,
tglx
Quoted 4 lines · Thomas Gleixner
On Fri, 3 Sep 2010, Arthur Kepner wrote:
It might be helpful to cc the maintainer of kernel/irq on patches which affect that area.
It wasn't (and after revieweing the MAINTAINERS file, still isn't) clear to me that you're the maintainer there, but I'll cc you on this topic in the future.
Quoted 13 lines
Quoted 6 lines
We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).
Quoted 6 lines
The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).
Quoted 3 lines
Errm. If the irq balancer has not been run, then why should some new irq balancer be up and running at this time. In the worst case you need to sort that out during boot _before_ you even hit userspace.
David M. made the same point.
I've got a new scheme in the works here:
http://marc.info/?l=linux-kernel&m=128381632020999&w=2
When I get some time on a big NUMA system, I'll test the latest iteration of that, and post an update.
Signature · 1 line
Arthur
PATCH 1/1 MAINTAINERS: Add IRQ SUBSYSTEM
Quoted 7 lines · Arthur Kepner
Quoted 3 lines · Thomas Gleixner
On Fri, 3 Sep 2010, Arthur Kepner wrote:
It might be helpful to cc the maintainer of kernel/irq on patches which affect that area.It wasn't (and after reviewing the
MAINTAINERSfile, still isn't) clear to me that you're the maintainer there, but I'll cc you on this topic in the future.
To make it easier on others, maybe Thomas should add his name to MAINTAINERS because kernel/irq currently has no listed MAINTAINER.
$ ./scripts/get_maintainer.pl --nogit --roles -f kernel/irq [email protected] (open list)
$ ./scripts/get_maintainer.pl --git --rolestats -f kernel/irq Thomas Gleixner <[email protected]> (commit_signer:19/39=49%) Jiri Kosina <[email protected]> (commit_signer:5/39=13%) "H. Peter Anvin" <[email protected]> (commit_signer:4/39=10%) Yinghai Lu <[email protected]> (commit_signer:4/39=10%) Ingo Molnar <[email protected]> (commit_signer:3/39=8%) [email protected] (open list)
--- MAINTAINERS | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
MAINTAINERS +8 -0
@@ -3194,6 +3194,14 @@ F: drivers/net/irda/ F: include/net/irda/ F: net/irda/ +IRQ SUBSYSTEM +M: Thomas Gleixner <[email protected]> +S: Maintained +F: Documentation/IRQ*.txt +F: include/linux/interrupt.h +F: include/linux/irq.h +F: kernel/irq/ + ISAPNP M: Jaroslav Kysela <[email protected]> S: Maintained