IRQ threads

I have attached a patch for implementing IRQ handlers in threads, for latency-reduction purposes. It requires that softirqs must be run in threads (or else they could end up running inside the IRQ threads, which will at the very least trigger bugs due to in_irq() being set). I've tested it with Ingo's voluntary-preempt J7 patch, and it should work with the TimeSys softirq thread patch as well (though you might get a conflict with the PF_IRQHANDLER definition; just merge them into one).

Some notes:

1. This may not work properly with some interrupt controller code, which doesn't do the obvious thing with mask_and_ack() and end(). This includes the IO-APIC code, which has an empty end() for edge triggered interrupts and an empty mask_and_ack() for level-triggered interrupts. The mask_and_ack() needs to really mask the interrupt, as otherwise the hardware will not deliver lower-priority (to it) interrupts, which may have a higher-priority thread.

2. This patch does not disable local interrupts when running a threaded handler. SMP-safe drivers shouldn't be directly bothered by this (as the interrupt could as easily have happened on another CPU), but there may be some interactions with softirqs and per-cpu data, if a softirq thread preempts an IRQ thread, or an IRQ thread gets migrated to a different CPU. I'm particularly worried about the network code. If possible, I'd like to find and fix such breakages rather than use local_irq_disable(), as that would prevent IRQ proritization from working, and prevent IRQ threads from being used to isolate the rest of the system from long-running IRQs (such as non-DMA IDE).

3. The i8042 driver had to be marked SA_NOTHREAD, as there are non-preemptible regions where it spins, waiting for an interrupt. Ideally, this driver (and others like it) should be fixed to either do a cond_resched() or use a wait queue.

4. This might be a good time to get around to moving the bulk of the arch/whatever/kernel/irq.c into generic code, as the code said was supposed to happen in 2.5. This patch is currently only for x86 (though we've run IRQ threads on many different platforms in the past).

5. Is there any reason why an IRQ controller might want to have its end() called even if IRQ_DISABLED or IRQ_INPROGRESS is set? It'd be nice to merge those checks in with the IRQ_THREADPENDING/IRQ_THREADRUNNING checks.

6. This patch causes in_irq() to return true if an IRQ thread is running, as some drivers use it in common code to determine how to act. in_interrupt(), however, will return false in such a case. The exact meaning of these macros in the presence of IRQ threads isn't very well defined, and I hope this results in sane behavior.

diff -urN linux-2.6.8-rc2/arch/i386/kernel/i386_ksyms.c linux-2.6.8-rc2-irq-threads/arch/i386/kernel/i386_ksyms.c
@@ -146,7 +146,6 @@
 EXPORT_SYMBOL_NOVERS(__read_lock_failed);
 
 /* Global SMP stuff */
-EXPORT_SYMBOL(synchronize_irq);
 EXPORT_SYMBOL(smp_call_function);
 
 /* TLB flushing */

@@ -154,6 +153,10 @@
 EXPORT_SYMBOL_GPL(flush_tlb_all);
 #endif
 
+#if defined(CONFIG_SMP) || defined(CONFIG_IRQ_THREADS) 
+EXPORT_SYMBOL(synchronize_irq);
+#endif
+
 #ifdef CONFIG_X86_IO_APIC
 EXPORT_SYMBOL(IO_APIC_get_PCI_irq_vector);
 #endif

diff -urN linux-2.6.8-rc2/arch/i386/kernel/i8259.c linux-2.6.8-rc2-irq-threads/arch/i386/kernel/i8259.c
@@ -332,7 +332,8 @@
  * New motherboards sometimes make IRQ 13 be a PCI interrupt,
  * so allow interrupt sharing.
  */
-static struct irqaction fpu_irq = { math_error_irq, 0, CPU_MASK_NONE, "fpu", NULL, NULL };
+static struct irqaction fpu_irq = 
+	{ math_error_irq, SA_NOTHREAD, CPU_MASK_NONE, "fpu", NULL, NULL };
 
 void __init init_ISA_irqs (void)
 {

diff -urN linux-2.6.8-rc2/arch/i386/mach-default/setup.c linux-2.6.8-rc2-irq-threads/arch/i386/mach-default/setup.c
@@ -27,7 +27,8 @@
 /*
  * IRQ2 is cascade interrupt to second interrupt controller
  */
-static struct irqaction irq2 = { no_action, 0, CPU_MASK_NONE, "cascade", NULL, NULL};
+static struct irqaction irq2 = 
+	{ no_action, SA_NOTHREAD, CPU_MASK_NONE, "cascade", NULL, NULL};
 
 /**
  * intr_init_hook - post gate setup interrupt initialisation

@@ -71,7 +72,9 @@
 {
 }
 
-static struct irqaction irq0  = { timer_interrupt, SA_INTERRUPT, CPU_MASK_NONE, "timer", NULL, NULL};
+static struct irqaction irq0  = 
+	{ timer_interrupt, SA_INTERRUPT | SA_NOTHREAD, CPU_MASK_NONE, 
+	  "timer", NULL, NULL};
 
 /**
  * time_init_hook - do any specific initialisations for the system timer.

diff -urN linux-2.6.8-rc2/arch/i386/mach-visws/setup.c linux-2.6.8-rc2-irq-threads/arch/i386/mach-visws/setup.c
@@ -112,7 +112,7 @@
 
 static struct irqaction irq0 = {
 	.handler =	timer_interrupt,
-	.flags =	SA_INTERRUPT,
+	.flags =	SA_INTERRUPT | SA_NOTHREAD,
 	.name =		"timer",
 };
 

diff -urN linux-2.6.8-rc2/arch/i386/mach-visws/visws_apic.c linux-2.6.8-rc2-irq-threads/arch/i386/mach-visws/visws_apic.c
@@ -261,11 +261,13 @@
 static struct irqaction master_action = {
 	.handler =	piix4_master_intr,
 	.name =		"PIIX4-8259",
+	.flags =        SA_NOTHREAD,
 };
 
 static struct irqaction cascade_action = {
 	.handler = 	no_action,
 	.name =		"cascade",
+	.flags =        SA_NOTHREAD,
 };
 
 

diff -urN linux-2.6.8-rc2/arch/i386/mach-voyager/setup.c linux-2.6.8-rc2-irq-threads/arch/i386/mach-voyager/setup.c
@@ -17,7 +17,8 @@
 /*
  * IRQ2 is cascade interrupt to second interrupt controller
  */
-static struct irqaction irq2 = { no_action, 0, CPU_MASK_NONE, "cascade", NULL, NULL};
+static struct irqaction irq2 = 
+	{ no_action, SA_NOTHREAD, CPU_MASK_NONE, "cascade", NULL, NULL};
 
 void __init intr_init_hook(void)
 {

@@ -40,7 +41,9 @@
 {
 }
 
-static struct irqaction irq0  = { timer_interrupt, SA_INTERRUPT, CPU_MASK_NONE, "timer", NULL, NULL};
+static struct irqaction irq0  = 
+	{ timer_interrupt, SA_INTERRUPT | SA_NOTHREAD, 
+	  CPU_MASK_NONE, "timer", NULL, NULL};
 
 void __init time_init_hook(void)
 {

diff -urN linux-2.6.8-rc2/drivers/input/serio/i8042.c linux-2.6.8-rc2-irq-threads/drivers/input/serio/i8042.c
@@ -277,7 +277,7 @@
 			return 0;
 
 	if (request_irq(values->irq, i8042_interrupt,
-			SA_SHIRQ, "i8042", i8042_request_irq_cookie)) {
+			SA_SHIRQ | SA_NOTHREAD, "i8042", i8042_request_irq_cookie)) {
 		printk(KERN_ERR "i8042.c: Can't get irq %d for %s, unregistering the port.\n", values->irq, values->name);
 		goto irq_fail;
 	}

@@ -571,7 +571,7 @@
  * in trying to detect AUX presence.
  */
 
-	if (request_irq(values->irq, i8042_interrupt, SA_SHIRQ,
+	if (request_irq(values->irq, i8042_interrupt, SA_SHIRQ | SA_NOTHREAD,
 				"i8042", &i8042_check_aux_cookie))
                 return -1;
 	free_irq(values->irq, &i8042_check_aux_cookie);

diff -urN linux-2.6.8-rc2/include/asm-i386/hardirq.h linux-2.6.8-rc2-irq-threads/include/asm-i386/hardirq.h
@@ -64,7 +64,12 @@
  * Are we doing bottom half or hardware interrupt processing?
  * Are we in a softirq context? Interrupt context?
  */
+#ifdef CONFIG_IRQ_THREADS
+#define in_irq()     (hardirq_count() || (current->flags & PF_IRQHANDLER))
+#else
 #define in_irq()		(hardirq_count())
+#endif
+
 #define in_softirq()		(softirq_count())
 #define in_interrupt()		(irq_count())
 

@@ -92,7 +97,7 @@
 		preempt_enable_no_resched();				\
 } while (0)
 
-#ifndef CONFIG_SMP
+#if !defined(CONFIG_SMP) && !defined(CONFIG_IRQ_THREADS)
 # define synchronize_irq(irq)	barrier()
 #else
   extern void synchronize_irq(unsigned int irq);

diff -urN linux-2.6.8-rc2/include/asm-i386/irq.h linux-2.6.8-rc2-irq-threads/include/asm-i386/irq.h
@@ -27,6 +27,8 @@
 extern void release_x86_irqs(struct task_struct *);
 extern int can_request_irq(unsigned int, unsigned long flags);
 
+#define get_irq_desc(irq) (&irq_desc[irq])
+
 #ifdef CONFIG_X86_LOCAL_APIC
 #define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif

diff -urN linux-2.6.8-rc2/include/asm-i386/signal.h linux-2.6.8-rc2-irq-threads/include/asm-i386/signal.h
@@ -122,6 +122,7 @@
 #define SA_PROBE		SA_ONESHOT
 #define SA_SAMPLE_RANDOM	SA_RESTART
 #define SA_SHIRQ		0x04000000
+#define SA_NOTHREAD             0x01000000
 #endif
 
 #define SIG_BLOCK          0	/* for blocking signals */

diff -urN linux-2.6.8-rc2/include/linux/interrupt.h linux-2.6.8-rc2-irq-threads/include/linux/interrupt.h
@@ -51,7 +51,7 @@
 /*
  * Temporary defines for UP kernels, until all code gets fixed.
  */
-#ifndef CONFIG_SMP
+#if !defined(CONFIG_SMP) && !defined(CONFIG_IRQ_THREADS)
 # define cli()			local_irq_disable()
 # define sti()			local_irq_enable()
 # define save_flags(x)		local_save_flags(x)

diff -urN linux-2.6.8-rc2/include/linux/sched.h linux-2.6.8-rc2-irq-threads/include/linux/sched.h
@@ -555,6 +555,7 @@
 #define PF_SWAPOFF	0x00080000	/* I am in swapoff */
 #define PF_LESS_THROTTLE 0x00100000	/* Throttle me less: I clean memory */
 #define PF_SYNCWRITE	0x00200000	/* I am doing a sync write */
+#define PF_IRQHANDLER   0x00400000      /* in_irq() should return true */
 
 #ifdef CONFIG_SMP
 #define SCHED_LOAD_SCALE	128UL	/* increase resolution of load */

diff -urN linux-2.6.8-rc2/init/Kconfig linux-2.6.8-rc2-irq-threads/init/Kconfig
@@ -294,6 +294,18 @@
 
 	  If unsure, say N.
 
+config IRQ_THREADS
+  bool "Run all IRQs in threads by default"
+  depends on PREEMPT
+  help
+    This option creates a thread for each IRQ, which runs at high
+    real-time priority, unless the SA_NOTHREAD option is passed to
+    request_irq().  This allows these IRQs to be prioritized, so as
+    to avoid preempting very high priority real-time tasks.  This
+    also allows spinlocks used by threaded IRQs to be converted
+    into sleeping mutexes, for further reduction of latency (however,
+    this is not done automatically).
+
 endmenu		# General setup
 
 

diff -urN linux-2.6.8-rc2/init/main.c linux-2.6.8-rc2-irq-threads/init/main.c
@@ -668,6 +668,8 @@
 	smp_init();
 	sched_init_smp();
 
+	spawn_irq_threads();
+
 	/*
 	 * Do this before initcalls, because some drivers want to access
 	 * firmware files.

diff -urN linux-2.6.8-rc2/kernel/Makefile linux-2.6.8-rc2-irq-threads/kernel/Makefile
@@ -23,6 +23,7 @@
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
 obj-$(CONFIG_AUDIT) += audit.o
 obj-$(CONFIG_AUDITSYSCALL) += auditsc.o
+obj-$(CONFIG_IRQ_THREADS) += irq.o
 
 ifneq ($(CONFIG_IA64),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

hardirqs (unlike softirqs) we might attempt to make preemptable. It really should work because if they run with interrupts enabled then they must be prepared to see interrupts coming from a similar device using the same softirq facilities.

migration between CPUs we can prevent easily. While it's not common for a driver to rely on per-CPU-ness _outside_ of a spinlocked region, it could in theory happen.

for now i've gone the more conservative route of keeping irqs atomic still, and applying lock-break methods to them.

this is not enough: DMA-IDE creates big latencies too, and those latencies happen under a spinlock - so with your patch it would be non-preemptible still.

yeah, i fixed it via cond_resched_softirq(), _after_ making sure it's safe to preempt there - softirqs are not generally preemption safe because they are fundamentally per-CPU. (meanwhile Dmitry Torokhov has posted a patch that fixes the atkbd.c problem for real by using workqueues. The psmouse-base.c problem needs a similar fix.)

agreed. I punted this one for the time being as it's clearly separate from the issue of latencies and it's deeply intrusive to 2.6.

e.g. in the IO-APIC case if we ack the local APIC only in the end() function then we must do that - an un-acked local APIC prevents other IRQs from being delivered. We do this for level-triggered IO-APIC irqs.

this is an incorrect change, just grep for in_interrupt() in linux/drivers/ ...

I agree with the concept of using multiple threads for interrupts - i'll add that to the voluntary-preempt patch too. This is an essential feature to prioritize interrupts.

what do you think about making the i8259A's interrupt priorities configurable? (a'la rtirq patch) Does it make any sense, given how early we mask the source irq and ack the interrupt controller [hence giving all other interrupts a fair chance to arrive ASAP]?

Bernhard Kuhn's rtirq patch is for IO-APIC/APICs, but i think the latency issues could be equally well fixed by not keeping the local APIC un-ACK-ed during level triggered irqs, but doing the mask & ack thing. This will be slightly slower but should make them both redirectable and more symmetric and fair.

Ingo

Not sure why don't want to use something as architecture-specific as the rtirq patch. We've been developing the Adeos nanokernel for 2 years now, and it provides a uniform functionality regardless of the architecture you are running it on. Adeos now runs on x86, PowerPC, ARM (mmu-less and mmu-full), and IA64. Plus, RTAI now uses Adeos to run side-by-side with Linux on at least the x86 and the PowerPC. If you're looking at prioritizing interrupts, then Adeos' interrupt pipeline is certainly the most compelling method available at this point in time.

Karim

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

grr... typo:

               ^^^^^ you'd

Karim

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

That way I picture the problem permits those threads to migration across CPUs and therefore kill interrupt performance from cache thrashing. Do you have a solution for that ? I like the way you're doing it now with irqd() in that it's CPU-local, but as you know it's not priority sensitive.

I think it's pretty complementary to what's going on, but it's also driven by application demand, how it could possibly hook into the scheduler or something else like that. There isn't a clear precedence for how to use something like this potentially. LynxOS doesn't have priorities above interrupt priorities, that mask interrupts somehow, but are folks that need that kind of stuff and could certainly be added in a relatively trivial manner.

My answer would be something like yes it's needed, but not now.

bill

Wouldn't the IRQ threads be subject to the same heuristics that the scheduler uses with ordinary threads, in order to avoid unnecessary CPU migration? Plus, IRQs ordinarily get distributed across CPUs, and in most cases shouldn't have a very large cache footprint (especially data; the code can be in multiple CPU caches at once), so I don't think this is a susbtantial degradation from the way things already are.

If desired by the user, an IRQ thread could be bound to a specific CPU to avoid such problems (in which case, they'd probably want to set the smp_affinity of the hard IRQ stub to the same CPU).

-Scott

I get a number of gripes from SMP aware folks that the context switching overhead is significant as well as cache issues. That's what the concern is about.

Yeah, this is an obvious next step in order to get better performance. Do you have number showing how this logic effects overall SMP and interrupt performance ?

I wish I could help with this, but I'm doing other things at the moment.

bill

i fixed this problem in -M5 the other way around: the IRQ threads follow the affinity settings. They will bind themselves to the first CPU in the affinity mask and they migrate only at 'safe' points (between hardirqs).

this way e.g. user-space irqbalance will automatically move the IRQ threads around too.

Ingo

The intrusiveness is somewhat mitigated by not having to convert all architectures at once, though; the generic code would only be included for those architectures that set the relevant CONFIG_. At that point, it basically is just moving code from one file to another (for x86), and doing minor tweaks for those architectures that are close to what x86 does.

Yes, but the IO-APIC code needs to be changed anyway to work properly with IRQ threads.

It could be useful for SA_NOTHREAD interrupts, but I don't think it buys much for threaded interrupts (as you'd have to wait until normal IRQs are enabled to schedule the handler thread anyway).

I agree. How much slower would it be (is it worth an #ifdef for irq threads)? Hopefully PIC operations aren't as slow as they are on the 8259...

-Scott

i'm wondering about a couple of details. Why were the changes to note_interrupt() necessary? Also, why the enable_irq() change? What do you think about the simpler approach in my patch which keeps the irq masked until the thread runs?

Ingo

The intent was to defer the calling of note_interrupt() until after all handlers (threaded and non-threaded) had run. However, it's probably not strictly necessary, as it requires the vast majority of IRQs to be unhandled (whereas threads will only give you 50%).

It might be nice to add a note_interrupt() to the no-action case as well, in case an interrupt with no handlers at all fails to be masked properly.

Oh, and the calling of report_bad_irq() seems to have disappeared from run_irq_thread in that patch; there should be a:

if (!noirqdebug) {
	if (retval == IRQ_HANDLED)
		desc->status &= ~IRQ_UNHANDLED;
	else
		report_bad_irq(irq, desc, retval);
}

before the "desc->status &= ~IRQ_THREADRUNNING;" line in kernel/irq.c.

If you mean the do_startup_irq() change, it was mainly to manage the IRQ_DELAYEDSTARTUP flag, which prevents an IRQ from being unmasked before the thread has been created (or more accurately, reminds run_irq_thread to call do_startup_irq()).

That way works as well (the desc would just have IRQ_THREADPENDING marked until the thread runs for the first time, if an IRQ does happen before the thread starts).

I've attached a new version of the patch that eliminates the note_interrupt() and startup/shutdown changes, adds the missing note_interrupt to run_irq_thread(), and makes in_interrupt respond positively to IRQ threads (it didn't do so in 2.4 because a lot of places used in_interrupt as in_atomic, but that shouldn't be a problem now, at least in core code).

diff -urN linux-2.6.8-rc2/arch/i386/kernel/i386_ksyms.c linux-2.6.8-rc2-irq-threads/arch/i386/kernel/i386_ksyms.c
@@ -146,7 +146,6 @@
 EXPORT_SYMBOL_NOVERS(__read_lock_failed);
 
 /* Global SMP stuff */
-EXPORT_SYMBOL(synchronize_irq);
 EXPORT_SYMBOL(smp_call_function);
 
 /* TLB flushing */

@@ -154,6 +153,10 @@
 EXPORT_SYMBOL_GPL(flush_tlb_all);
 #endif
 
+#if defined(CONFIG_SMP) || defined(CONFIG_IRQ_THREADS) 
+EXPORT_SYMBOL(synchronize_irq);
+#endif
+
 #ifdef CONFIG_X86_IO_APIC
 EXPORT_SYMBOL(IO_APIC_get_PCI_irq_vector);
 #endif

diff -urN linux-2.6.8-rc2/arch/i386/kernel/i8259.c linux-2.6.8-rc2-irq-threads/arch/i386/kernel/i8259.c
@@ -332,7 +332,8 @@
  * New motherboards sometimes make IRQ 13 be a PCI interrupt,
  * so allow interrupt sharing.
  */
-static struct irqaction fpu_irq = { math_error_irq, 0, CPU_MASK_NONE, "fpu", NULL, NULL };
+static struct irqaction fpu_irq = 
+	{ math_error_irq, SA_NOTHREAD, CPU_MASK_NONE, "fpu", NULL, NULL };
 
 void __init init_ISA_irqs (void)
 {

diff -urN linux-2.6.8-rc2/arch/i386/mach-default/setup.c linux-2.6.8-rc2-irq-threads/arch/i386/mach-default/setup.c
@@ -27,7 +27,8 @@
 /*
  * IRQ2 is cascade interrupt to second interrupt controller
  */
-static struct irqaction irq2 = { no_action, 0, CPU_MASK_NONE, "cascade", NULL, NULL};
+static struct irqaction irq2 = 
+	{ no_action, SA_NOTHREAD, CPU_MASK_NONE, "cascade", NULL, NULL};
 
 /**
  * intr_init_hook - post gate setup interrupt initialisation

@@ -71,7 +72,9 @@
 {
 }
 
-static struct irqaction irq0  = { timer_interrupt, SA_INTERRUPT, CPU_MASK_NONE, "timer", NULL, NULL};
+static struct irqaction irq0  = 
+	{ timer_interrupt, SA_INTERRUPT | SA_NOTHREAD, CPU_MASK_NONE, 
+	  "timer", NULL, NULL};
 
 /**
  * time_init_hook - do any specific initialisations for the system timer.

diff -urN linux-2.6.8-rc2/arch/i386/mach-visws/setup.c linux-2.6.8-rc2-irq-threads/arch/i386/mach-visws/setup.c
@@ -112,7 +112,7 @@
 
 static struct irqaction irq0 = {
 	.handler =	timer_interrupt,
-	.flags =	SA_INTERRUPT,
+	.flags =	SA_INTERRUPT | SA_NOTHREAD,
 	.name =		"timer",
 };
 

diff -urN linux-2.6.8-rc2/arch/i386/mach-visws/visws_apic.c linux-2.6.8-rc2-irq-threads/arch/i386/mach-visws/visws_apic.c
@@ -261,11 +261,13 @@
 static struct irqaction master_action = {
 	.handler =	piix4_master_intr,
 	.name =		"PIIX4-8259",
+	.flags =        SA_NOTHREAD,
 };
 
 static struct irqaction cascade_action = {
 	.handler = 	no_action,
 	.name =		"cascade",
+	.flags =        SA_NOTHREAD,
 };
 
 

diff -urN linux-2.6.8-rc2/arch/i386/mach-voyager/setup.c linux-2.6.8-rc2-irq-threads/arch/i386/mach-voyager/setup.c
@@ -17,7 +17,8 @@
 /*
  * IRQ2 is cascade interrupt to second interrupt controller
  */
-static struct irqaction irq2 = { no_action, 0, CPU_MASK_NONE, "cascade", NULL, NULL};
+static struct irqaction irq2 = 
+	{ no_action, SA_NOTHREAD, CPU_MASK_NONE, "cascade", NULL, NULL};
 
 void __init intr_init_hook(void)
 {

@@ -40,7 +41,9 @@
 {
 }
 
-static struct irqaction irq0  = { timer_interrupt, SA_INTERRUPT, CPU_MASK_NONE, "timer", NULL, NULL};
+static struct irqaction irq0  = 
+	{ timer_interrupt, SA_INTERRUPT | SA_NOTHREAD, 
+	  CPU_MASK_NONE, "timer", NULL, NULL};
 
 void __init time_init_hook(void)
 {

diff -urN linux-2.6.8-rc2/drivers/input/serio/i8042.c linux-2.6.8-rc2-irq-threads/drivers/input/serio/i8042.c
@@ -277,7 +277,7 @@
 			return 0;
 
 	if (request_irq(values->irq, i8042_interrupt,
-			SA_SHIRQ, "i8042", i8042_request_irq_cookie)) {
+			SA_SHIRQ | SA_NOTHREAD, "i8042", i8042_request_irq_cookie)) {
 		printk(KERN_ERR "i8042.c: Can't get irq %d for %s, unregistering the port.\n", values->irq, values->name);
 		goto irq_fail;
 	}

@@ -571,7 +571,7 @@
  * in trying to detect AUX presence.
  */
 
-	if (request_irq(values->irq, i8042_interrupt, SA_SHIRQ,
+	if (request_irq(values->irq, i8042_interrupt, SA_SHIRQ | SA_NOTHREAD,
 				"i8042", &i8042_check_aux_cookie))
                 return -1;
 	free_irq(values->irq, &i8042_check_aux_cookie);

diff -urN linux-2.6.8-rc2/include/asm-i386/hardirq.h linux-2.6.8-rc2-irq-threads/include/asm-i386/hardirq.h
@@ -64,10 +64,19 @@
  * Are we doing bottom half or hardware interrupt processing?
  * Are we in a softirq context? Interrupt context?
  */
+#ifdef CONFIG_IRQ_THREADS
+#define in_irq() (hardirq_count() || (current->flags & PF_IRQHANDLER))
+#else
 #define in_irq()		(hardirq_count())
+#endif
+
 #define in_softirq()		(softirq_count())
-#define in_interrupt()		(irq_count())
 
+#ifdef CONFIG_IRQ_THREADS
+#define in_interrupt() (irq_count() || (current->flags & PF_IRQHANDLER))
+#else
+#define in_interrupt()		(irq_count())
+#endif
 
 #define hardirq_trylock()	(!in_interrupt())
 #define hardirq_endlock()	do { } while (0)

@@ -92,7 +101,7 @@
 		preempt_enable_no_resched();				\
 } while (0)
 
-#ifndef CONFIG_SMP
+#if !defined(CONFIG_SMP) && !defined(CONFIG_IRQ_THREADS)
 # define synchronize_irq(irq)	barrier()
 #else
   extern void synchronize_irq(unsigned int irq);

diff -urN linux-2.6.8-rc2/include/asm-i386/irq.h linux-2.6.8-rc2-irq-threads/include/asm-i386/irq.h
@@ -27,6 +27,8 @@
 extern void release_x86_irqs(struct task_struct *);
 extern int can_request_irq(unsigned int, unsigned long flags);
 
+#define get_irq_desc(irq) (&irq_desc[irq])
+
 #ifdef CONFIG_X86_LOCAL_APIC
 #define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif

diff -urN linux-2.6.8-rc2/include/asm-i386/signal.h linux-2.6.8-rc2-irq-threads/include/asm-i386/signal.h
@@ -122,6 +122,7 @@
 #define SA_PROBE		SA_ONESHOT
 #define SA_SAMPLE_RANDOM	SA_RESTART
 #define SA_SHIRQ		0x04000000
+#define SA_NOTHREAD             0x01000000
 #endif
 
 #define SIG_BLOCK          0	/* for blocking signals */

diff -urN linux-2.6.8-rc2/include/linux/interrupt.h linux-2.6.8-rc2-irq-threads/include/linux/interrupt.h
@@ -51,7 +51,7 @@
 /*
  * Temporary defines for UP kernels, until all code gets fixed.
  */
-#ifndef CONFIG_SMP
+#if !defined(CONFIG_SMP) && !defined(CONFIG_IRQ_THREADS)
 # define cli()			local_irq_disable()
 # define sti()			local_irq_enable()
 # define save_flags(x)		local_save_flags(x)

@@ -247,4 +247,15 @@
 extern int probe_irq_off(unsigned long);	/* returns 0 or negative on failure */
 extern unsigned int probe_irq_mask(unsigned long);	/* returns mask of ISA interrupts */
 
+#ifdef CONFIG_IRQ_THREADS
+
+/* This is under CONFIG_IRQ_THREADS for now, so it doesn't break other
+   architectures where it's still static.  It has to be here rather
+   than in irq.h, because it depends on irqreturn_t, and including
+   this file from irq.h apparently causes a loop. */
+
+void note_interrupt(int irq, irq_desc_t *desc, irqreturn_t action_ret);
+extern int noirqdebug;
+
+#endif
 #endif

diff -urN linux-2.6.8-rc2/include/linux/sched.h linux-2.6.8-rc2-irq-threads/include/linux/sched.h
@@ -555,6 +555,7 @@
 #define PF_SWAPOFF	0x00080000	/* I am in swapoff */
 #define PF_LESS_THROTTLE 0x00100000	/* Throttle me less: I clean memory */
 #define PF_SYNCWRITE	0x00200000	/* I am doing a sync write */
+#define PF_IRQHANDLER   0x00400000      /* in_irq() should return true */
 
 #ifdef CONFIG_SMP
 #define SCHED_LOAD_SCALE	128UL	/* increase resolution of load */

diff -urN linux-2.6.8-rc2/init/Kconfig linux-2.6.8-rc2-irq-threads/init/Kconfig
@@ -294,6 +294,18 @@
 
 	  If unsure, say N.
 
+config IRQ_THREADS
+  bool "Run all IRQs in threads by default"
+  depends on PREEMPT
+  help
+    This option creates a thread for each IRQ, which runs at high
+    real-time priority, unless the SA_NOTHREAD option is passed to
+    request_irq().  This allows these IRQs to be prioritized, so as
+    to avoid preempting very high priority real-time tasks.  This
+    also allows spinlocks used by threaded IRQs to be converted
+    into sleeping mutexes, for further reduction of latency (however,
+    this is not done automatically).
+
 endmenu		# General setup
 
 

diff -urN linux-2.6.8-rc2/init/main.c linux-2.6.8-rc2-irq-threads/init/main.c
@@ -668,6 +668,8 @@
 	smp_init();
 	sched_init_smp();
 
+	spawn_irq_threads();
+
 	/*
 	 * Do this before initcalls, because some drivers want to access
 	 * firmware files.

diff -urN linux-2.6.8-rc2/kernel/Makefile linux-2.6.8-rc2-irq-threads/kernel/Makefile
@@ -23,6 +23,7 @@
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
 obj-$(CONFIG_AUDIT) += audit.o
 obj-$(CONFIG_AUDITSYSCALL) += auditsc.o
+obj-$(CONFIG_IRQ_THREADS) += irq.o
 
 ifneq ($(CONFIG_IA64),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

i mean the change below - why do irqthreads necessiate it?

Ingo

@@ -395,7 +402,14 @@ void enable_irq(unsigned int irq)
desc->status = status | IRQ_REPLAY; hw_resend_irq(desc->handler,irq);
}
- desc->handler->enable(irq);
+
+ /* Don't unmask the IRQ if it's in progress, or else you
+ could re-enter the IRQ handler. As it is now enabled,
+ the IRQ will be unmasked when the handler is finished. */
+
+ if (!(desc->status & (IRQ_INPROGRESS | IRQ_THREADRUNNING |
+ IRQ_THREADPENDING)))
+ desc->handler->enable(irq);
/* fall-through */
}
default:

The intent is to make enable_irq() robust against calls while the thread is still running/pending (such as if the thread has lower priority than the task that calls enable_irq()). This implies that the preceding disable was of the _nosync() variety.

I believe we saw drivers/net/8390.c doing this, and it was causing an interrupt storm because, at the time (this was over a year ago), actionless irqs (which this IRQ was, because the IRQ was still in progress) had end() called, unmasking it again. The IRQ was level triggered, and thus the handler never got a chance to run.

That wouldn't happen with the current code, because it checks for THREADPENDING || THREADRUNNING before calling end() in really_do_IRQ(), so now all the check does is save the wasted time of one bad interrupt each time it happens.

-Scott

8390 does a disable_irq_nosync having previously cleared the IRQ on the controller. This is neccessary because IRQ arrival on PC hardware is asynchronous to all other busses and can take incredibly long times on SMP hardware prior to PIV. Thus it happens now and then that the controller emits an IRQ, we clear the source, the clear is done and later the IRQ arrives that has already been cleared down on the original IRQ source. Most drivers just use spinlocks but the 8390 is so slow that is has to pull other stunts or even things like serial ports and the 1Khz clock slide.

Alan

My experience with clients who have been using TimeSys' stuff has been abysmal. The fact of the of the matter is that most people who used this were practically locked-in to TimeSys' services, unable to download anything "standard" off the net and using it with their kernel. In one example, we had to ditch the kernel the client got from TimeSys because we had spent 10+ hours trying to get LTT to work on it without any success whatsoever.

As I had said on other lists before, I don't see the point of creating that much complexity in the kernel in order to try to shave-off a little bit more off of the kernel's interrupt response time. The fact of the matter is that neither this patch nor most of the other patches suggested makes the kernel truely hard-rt. These patches only make the kernel respond "faster". If you really need hard-rt, then you should be using the Adeos nanokernel. With Adeos, you can even get a hard-rt driver without using RTAI or any of the other rt derivatives.

Karim

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

This is obvious FUD from someone who is selling something. Please keep this crap off LKML.

Lee

"selling something" ??? I don't know what you're talking about. There is an entire history behind Adeos and its endorsement by a slew of open source organizations and individuals. You probably want to go read the LKML archives on the topic.

Please keep your uneducated self off the LKML.

Thanks,

Karim

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

I am familiar with Adeos, as well as other hard-RT solutions for Linux. I did my homework before deciding that I do not in fact need hard-RT, so I really am not interested in your flamewars, keep them on your RT mailing lists.

The part that was obvious commercially motivated FUD (and which you omitted) t in which you badmouth TimeSys and its services, then Your .sig states that you are a consultant specializing in realtime and embedded Linux.

And your point is ... ??? Should everyone on this list who's working for a company that is involved in a certain speciality field stop commenting on the solutions proposed by people working for other companies?

This style of attack is called sophism, and it is total crap for the educated crowd. Attacking someone based on who he works for and what he does has nothing to do with actually presenting a case against his arguments.

The personal experiences I've stated before have happened. Lest you suggest that those of us who have had bad experiences with certain pieces of technology should shut-up and not voice our complaints for fear of offending someone out there? Sorry, I don't live in Disneyland.

If you're not interested in hard-rt, that's you're business. Others on this list have expressed interest in the past for this, and you are welcome to ignore any traffic in this regard. For my part, I will get involved whenever I see someone suggesting something that relates to real-time because this is a topic I'm interested in. What a surprise that I'm actually working in a field I'm interested in ... sheesh ...

Karim

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

No, of course not. But please be more specific than 'everything I've seen from TimeSys is crap, I have talked to some of their clients who had $FOO problem'. Your complaint was so general as to not be refutable.

I could not agree more. Your original post had a strong ad hominem flavor, which was my objection. Of course you are free to attack anything at any time, on its technical merits, this is what engineers do.

I am interested in RT as well, I did not mean to imply that I don't find it a valid topic for discussion on LKML, but you have to admit that your post bordered on a troll.

Lee

I didn't say their stuff was crap. I said that "__My__ experience with clients ...". IOW, I'm talking about cases where I was called on by customers of mine who were using "stuff" (i.e. threaded int handlers and mutexed-locks), and in practice I have found that the only entity who could service this was the one that provided my clients with said kernel. And from my point of view, this is indeed "abysmal" because the whole point of using Linux is to be able to service your car at any retailer.

You are correct in stating that the lack of details made the argument hard to defend against, but I had a very nasty choice in writing this:
a) Either give enough detail, in which case my clients' confidentiality would be in question.
b) Or shut up and let this go unanswered on the LKML.

I can see that it could be intrepreted as such. But keep in mind that I was replying to Scott's _public_ posting of TimeSys' patch. IOW, there's no way that I could claim that there would be lock-in in the future if the patch did indeed make it in the kernel. The past tense was used on purpose, and TimeSys have been very up-front about their intent of getting this into the mainline.

My real argument was best summarized in the second paragraph, and what I'm saying is that these approaches make the kernel's dynamic behavior extremely complicated. And while they do contribute to making the kernel's response time faster, they do not provided hard-rt, which is what everyone is trying to get in the end anyway (either intentionally or unintentionally.)

With that, let me respond to Bill's discussion on signle vs. N kernels as that thread is the most likely to be fruitfull. I hope you'll agree.

Karim

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

Yes, agreed. I am glad this did not escalate, and I hope you can understand how I would have overlooked your actual argument due to my perceiving the first paragraph as vaguely ad hominem.

Lee

Yes I do understand, and thanks for taking the time to sort this out :)

Karim

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

With that said, there's really two camps that are emerging in the real time Linux field, dual and single kernel. The single kernel work that's current being done could very well get Linux to being hard RT, assuming that you solve all of the technical problems with things like RCU, etc... in 2.6.

The dual kernels folks would be in less of position to VAR their own stuff and sell proprietary products if Linux were to get native hard RT performance if you accept that economic criteria. Who knows what the actual results will be.

It could be that all of this work with Linux could bury prioprietary OS product (such as LynxOS here) or it could open doors to other things unknown things that were never possible previous to Linux getting some kind of hard RT capability. It's certainly a scary notion to think about with many variables to consider. Linux getting hard RT is inevitable. It's just a question of how it'll be handled by proprietary OS vendors, witness IBM for a positive example. A negative one would be Sun.

Now that Windriver System (the idiot folks that never understood Linux before laying off tons of folks and disbanned the rather famous BSD/OS group which I was apart of, etc...) and Red Hat is in the picture, it's all starting to cook up.

All things to think about.

bill

As I understand it there will still be a place for the current hard-RT Linux solutions, because even if I can get five nines latency better than N, this is not good enough for hard RT, as you need to be able to mathematically demonstrate that you can *never* miss a deadline.

Or are you saying that the latest developments in the stock kernel make this possible?

Lee

Not quite with this thread and in this stage of development, but this is quite possible if certain concurrency problems are solved in Linux. RCU is a potential pain as well as other things. BSD/OS-FreeBSD-current are not new to this kind of conversion, so this is certainly very possible with very finite software engineering problems to solve.

Scary ain't it ? It makes me wonder some times.

bill

Two things:
- What I'm suggesting is a nanokernel-based N kernel scheme, not the dual kernel scheme (which is patented BTW).
- There's only one company out there that is known to sell proprietary products around the dual kernel approach, and it isn't mine.

Indeed. So the question now becomes: is it worth introducing that much complexity inside the kernel to solve a problem that matters only marginally to server and workstation users? It's already difficult as it is to manage the preemption, do we really want to go the full way with threaded int handlers and mutexes instead of locks?

What I'm suggesting is a very simple model that solves 90% of the time- sensitive problems I have seen with Linux: Use the Adeos nanokernel to implement the real-time component of drivers as a priority domain the interrupt pipeline. Hence, instead of the current driver model:
1- Int handler
2- BH/SoftIRQ/etc.
We get:
1- Hard-rt handler
2- Normal Linux handler
3- BH/SoftIRQ/etc.

This is even without any RTAI/RTL or anything of that kind.

And for the rest, then you want to have a look at Philippe Gerum's ongoing RTAI-fusion work. With RTAI-fusion, you get the same normal Linux ABI for all user-space tasks, but you get hard-rt for free. Practically, normal Linux calls are caught and run through RTAI instead of Linux. Philippe has already got the standard nanosleep() running in hard-rt. So the question is therefore straight-forward:
should we engineer a path for converting the entire kernel to hard-rt or do we keep the kernel as it was designed and add the necessary mechanisms for obtaining hard-rt using things that were made to provide it by design?

I personally believe that the added complexity does not benefit Linux. I may yet be shown wrong, but with what we can currently do with plain vanilla Adeos, and where RTAI/fusion is heading, the problem space of applications which can't be serviced by this combination is getting increasingly limited.

Karim

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

I have to say that I didn't really carefully read the papers for your stuff and I still don't exactly understand what's going on with your nanokernel. It sounds interesting, but it doesn't solve what I think are inescapable preemption problems with the Linux kernel.

Complexity is a kind of relative terms in this conversation. It's more of a function of what one is uses here verses what other folks have been doing outside of Linux. I personally don't think it's THAT hard. It's just that folks inside Linux have really gotten comfortable using spinlocks for everything instead of another style of thinking. It's all learnable after you get over the shock and not as hard as folks think it would be.

I don't immediately see why this is going to give us hard RT for Linux applications. I'd like a short explanation if you have time to reply to this.

I don't really understand what RTAI fusion is and I would like some pointers to this. I've read the web page for this, but I wasn't able to extract a good chunk of information that would give me a clear picture of what it does.

Well, you have to think of it in a wider scope for a bit. If you want rate-guarateed streaming partition support from SGI's XFS, touching dcache, VM, etc... then you have to dealing with non-preemptable locking throughout the Linux kernel, no choice. Having a pervasive model is a superior solution to dual kernels since you have the full support of Linux verses the constrained semantics and APIs of a controlled sub-domain. That typically needs to be temporarly buffered (and API) both in and out of the a Linux sub-system. I fail to see how a nano-kernel, RTAI and other supporting kernel sub-systems can get around this problem without solving preemption issues within Linux.

When you think of things like this you have to think of what's happening now with applications that include the traditional RT stuff as well as media driven apps, which are becoming more and prevalent as little things like cell phone devices get more and more bandwidth. For media driven applications, none of the secondary/dual kernel or things like that can properly deal with things like IO thread/channel bonding, which depends on various IO layers that touch many SMP locked critical sections/systems in the kernel. Running into Linux's SMP locking logic in these cases are inescapable, so I don't see a choice but doing some kind of full lock conversion.

Anytime you have an application that uses a full suite of Linux facilities will have this problem. It just hasn't been done for a 2.6 kernel just yet.

bill

<snip>

With the ego power switch and name-calling amplifier lowered to the minimum, maybe there could be a third approach, like cooperation between both through a better integration. At least, the signal / noise ratio would improve...

The hard RT people I know of and work with want to be able 1) to get microsecond level bounded interrupt latency with no exception to this rule, and 2) to be able to choose the right level of dispatch latency on a thread-by-thread basis, from a few microseconds to a few hundreds of, but in any case _bounded_ and predictable in the worst case. For this to happen, they are willing to accept stringent limitations functionaly-wise if need be to obtain the first, but still get access to the regular Linux programming model and APIs if the second fits their apps. They already know how they could mix both properly in what would look like a single system from the application pov.

For these people, the current undergoing work aimed at improving the current determinism of the vanilla kernel is everything but a danger:
it's fundamental and very good news, because it could make 2) a reality sooner or later. However, point 1) remains an issue, and unless you find a solution for mixing fire and water, i.e. determinism which requires unfairness by design and throughput seeking fairness on the average case, you would likely end up considering that the Linux RT people's radical approach of using a dual-kernel does not make them uneducated bozos (Ok, except me perhaps, but this is not my point). To get microsecond level guaranteed interrupt latencies, the problem is far beyond solving random latency spots here and there: it's an architectural issue.

To achieve this, we (i.e. the educated ones like Karim helping the uneducated bozo like myself; yep, this is a teamwork) have come to the conclusion that we needed a portable infrastructure that allows a complete prioritization of interrupts, and hw events of interest in general (e.g. traps/exceptions). Some infrastructure that exposes the same interface regardless of the platform it runs on. It's called Adeos, the source code identation is terrible (after 20 years practicing it, I still
find means to worsen my coding style, funky eh?!) but it's a working example of such kind of infrastructure. The advantage of such kind of thin layer is that you can plug any hard real-time core over it. This layer can remain silent when unused, it can be configured out, it is just an enabler. You don't have to put the hell of a havoc in a stable GPOS core to modify some key architectural characteristics of the Linux kernel in order to buy hard RT capabilities to everyone, which could be construed as smashing a squadron of flies with nukes.

There is evidence that the GPL side of the hard RT universe does too. Indeed.

Philippe.