Skip to contentSkip to conversation

notify user-level IRQ balancer when kernel assigns IRQ affinity

RFC/PATCH New Arthur Kepner 5 participants 7 messages Sep 3, 2010 – Sep 9, 2010 IRQ

lore
Share & tools
lore
#0 lore raw

We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).

The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).

I know this needs work, but would like to get comments on the idea before doing more coding and testing.

---

arch/Kconfig +11 -0
index 4877a8c..65c79c7 100644
@@ -158,4 +158,15 @@ config HAVE_PERF_EVENTS_NMI
 	  subsystem.  Also has support for calculating CPU cycle events
 	  to determine how many clock cycles in a given period.
 
+config NOTIFY_USER_IRQ_BALANCER
+	bool "Notify user-level IRQ balancer (EXPERIMENTAL)"
+	default n
+	depends on NET && X86
+	help
+	  When the kernel sets a default CPU affinity for an interrupt, 
+          emit a notification message over a netlink socket so that a 
+          user-level IRQ balancer may re-balance IRQs the way it prefers.
+
+	  If unsure, say N.
+
 source "kernel/gcov/Kconfig"
include/linux/irq_notify.h new file +33 -0
new file mode 100644
index 0000000..b1bbdee
@@ -0,0 +1,33 @@
+#ifndef __IRQ_NOTIFY_H
+#define __IRQ_NOTIFY_H
+
+enum {
+	IRQ_NOTIFY_TYPE_UNSPEC,
+	IRQ_NOTIFY_TYPE_NOTICE,
+	__IRQ_NOTIFY_TYPE_MAX,
+};
+
+#define IRQ_NOTIFY_TYPE_MAX (__IRQ_NOTIFY_TYPE_MAX - 1)
+
+enum {
+	IRQ_NOTIFY_CMD_ATTR_UNSPEC,
+	IRQ_NOTIFY_CMD_ATTR_IRQ,
+	__IRQ_NOTIFY_CMD_ATTR_MAX,
+};
+
+#define IRQ_NOTIFY_CMD_ATTR_MAX (__IRQ_NOTIFY_CMD_ATTR_MAX - 1)
+
+#define IRQ_NOTIFY_GNL_NAME "IRQ_NOTIFY"
+#define IRQ_NOTIFY_GNL_VERSION 1
+
+#define IRQ_NOTIFY_GRP_ID 1
+
+#ifdef CONFIG_NOTIFY_USER_IRQ_BALANCER
+extern int irq_notify(int);
+#else
+static inline int irq_notify(int)
+{
+}
+#endif /* CONFIG_NOTIFY_USER_IRQ_BALANCER */
+
+#endif /* __IRQ_NOTIFY_H */
kernel/irq/Makefile +1 -0
index 7d04780..b74c75e 100644
@@ -5,3 +5,4 @@ obj-$(CONFIG_PROC_FS) += proc.o
 obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
 obj-$(CONFIG_NUMA_IRQ_DESC) += numa_migrate.o
 obj-$(CONFIG_PM_SLEEP) += pm.o
+obj-$(CONFIG_NOTIFY_USER_IRQ_BALANCER) += irq_notify.o
kernel/irq/irq_notify.c new file +103 -0

103 lines not on this page · raw message

kernel/irq/manage.c +2 -0
index c3003e9..2fa65e2 100644
@@ -14,6 +14,7 @@
 #include <linux/interrupt.h>
 #include <linux/slab.h>
 #include <linux/sched.h>
+#include <linux/irq_notify.h>
 
 #include "internals.h"
 
@@ -178,6 +179,7 @@ static int setup_affinity(unsigned int irq, struct irq_desc *desc)
 	cpumask_and(desc->affinity, cpu_online_mask, irq_default_affinity);
 set_affinity:
 	desc->chip->set_affinity(irq, desc->affinity);
+	irq_notify(irq);
 
 	return 0;
 }
#1 lore raw
Quoted 25 lines · Arthur Kepner

We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).

The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).

I know this needs work, but would like to get comments on the idea before doing more coding and testing.

---

arch/Kconfig +0 -0
index 4877a8c..65c79c7 100644

arch/Kconfig config HAVE_PERF_EVENTS_NMI · line 164

Quoted 7 lines
arch/Kconfig +4 -0
@@ -158,4 +158,15 @@ config HAVE_PERF_EVENTS_NMI
  	  subsystem.  Also has support for calculating CPU cycle events
  	  to determine how many clock cycles in a given period.
 
+config NOTIFY_USER_IRQ_BALANCER
+	bool "Notify user-level IRQ balancer (EXPERIMENTAL)"
+	default n
+	depends on NET&&  X86

Missing a space before that '&&', but more importantly, what part of the patch is x86 specific?

David Daney

#2 lore raw
Quoted 10 lines · David Daney
Quoted 10 lines

---

arch/Kconfig +0 -0
index 4877a8c..65c79c7 100644
@@ -158,4 +158,15 @@ config HAVE_PERF_EVENTS_NMI
  	  subsystem.  Also has support for calculating CPU cycle events
  	  to determine how many clock cycles in a given period.
 
Quoted 9 lines

arch/Kconfig config HAVE_PERF_EVENTS_NMI

Quoted 4 lines
arch/Kconfig +4 -0
+config NOTIFY_USER_IRQ_BALANCER
+	bool "Notify user-level IRQ balancer (EXPERIMENTAL)"
+	default n
+	depends on NET&&  X86

Missing a space before that '&&', but more importantly, what part of the patch is x86 specific?

Hmmm. I see the space in my copy of the message.

Anyway, the X86 dependency is bogus. For a while there was such a dependency, but that's gone now. Just forgot to fix that up. Will do that now.

Signature · 1 line
Arthur
#3 lore raw
In reply to Arthur Kepner

From: Arthur Kepner <[email protected]>
Date: Fri, 3 Sep 2010 14:05:54 -0700

Quoted 12 lines

We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).

The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).

Quoted 4 lines

I know this needs work, but would like to get comments on the idea before doing more coding and testing.

As I replied to you on netdev, this can't possibly work.

Interrupt allocation needs to work properly before userspace even starts up. So the allocation management must be inside of the kernel entirely.

#4 lore raw
In reply to Arthur Kepner

On Fri, 3 Sep 2010, Arthur Kepner wrote:

It might be helpful to cc the maintainer of kernel/irq on patches which affect that area.

Quoted 11 lines

We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).

The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).

Errm. If the irq balancer has not been run, then why should some new irq balancer be up and running at this time. In the worst case you need to sort that out during boot _before_ you even hit userspace.

Further this adds just redundant information as user space is already notified about a new device being setup via udev, so what's the point of adding another notification mechanism.

Thanks,

tglx

#5 lore raw
Quoted 4 lines · Thomas Gleixner

On Fri, 3 Sep 2010, Arthur Kepner wrote:

It might be helpful to cc the maintainer of kernel/irq on patches which affect that area.

It wasn't (and after revieweing the MAINTAINERS file, still isn't) clear to me that you're the maintainer there, but I'll cc you on this topic in the future.

Quoted 13 lines
Quoted 6 lines

We've run into situations where a CPU runs out of interrupt vectors, because all the interrupts are getting the default affinity (and the interrupt balancer hasn't yet run).

Quoted 6 lines

The following emits a netlink message whenever an interrupt is given a default CPU affinity. A user-level IRQ balancer can use those messages to decide if, and how to reassign affinities. This should allow us to avoid running out of vectors on any particular CPU (or at least make it far less likely).

Quoted 3 lines

Errm. If the irq balancer has not been run, then why should some new irq balancer be up and running at this time. In the worst case you need to sort that out during boot _before_ you even hit userspace.

David M. made the same point.

I've got a new scheme in the works here:

http://marc.info/?l=linux-kernel&m=128381632020999&w=2

When I get some time on a big NUMA system, I'll test the latest iteration of that, and post an update.

Signature · 1 line
Arthur
#6 lore raw

PATCH 1/1 MAINTAINERS: Add IRQ SUBSYSTEM

Quoted 7 lines · Arthur Kepner
Quoted 3 lines · Thomas Gleixner

On Fri, 3 Sep 2010, Arthur Kepner wrote:
It might be helpful to cc the maintainer of kernel/irq on patches which affect that area.

It wasn't (and after reviewing the MAINTAINERS file, still isn't) clear to me that you're the maintainer there, but I'll cc you on this topic in the future.

To make it easier on others, maybe Thomas should add his name to MAINTAINERS because kernel/irq currently has no listed MAINTAINER.

$ ./scripts/get_maintainer.pl --nogit --roles -f kernel/irq
[email protected] (open list)
$ ./scripts/get_maintainer.pl --git --rolestats -f kernel/irq
Thomas Gleixner <[email protected]> (commit_signer:19/39=49%)
Jiri Kosina <[email protected]> (commit_signer:5/39=13%)
"H. Peter Anvin" <[email protected]> (commit_signer:4/39=10%)
Yinghai Lu <[email protected]> (commit_signer:4/39=10%)
Ingo Molnar <[email protected]> (commit_signer:3/39=8%)
[email protected] (open list)
---
 MAINTAINERS |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
MAINTAINERS +8 -0
index 087912a..1dd4c9f 100644
@@ -3194,6 +3194,14 @@ F:	drivers/net/irda/
 F:	include/net/irda/
 F:	net/irda/
 
+IRQ SUBSYSTEM
+M:	Thomas Gleixner <[email protected]>
+S:	Maintained
+F:	Documentation/IRQ*.txt
+F:	include/linux/interrupt.h
+F:	include/linux/irq.h
+F:	kernel/irq/
+
 ISAPNP
 M:	Jaroslav Kysela <[email protected]>
 S:	Maintained