Skip to contentSkip to conversation

genirq: Provide adaptive irq oneshot functionality

RFC PATCH New Thomas Gleixner 1 participant 5 messages Dec 15, 2010 IRQ

lore
Share & tools
lore
#0 lore raw

cover genirq: Provide adaptive irq oneshot functionality

Warning: This is completely untested. Jan managed to confuse me even way beyond my base confusion level, so testing seems to be the appropriate reparation.

The final goal of these modifications is to allow an adaptive oneshot mode for possibly shared interrupts. If the interrupt is not shared then oneshot mode should be used as it is more efficient than masking/unmasking at the device level. When the interrupt becomes shared then the oneshot mode needs to be disabled and device level masking must be done.

That requires transitioning from one state to the other which needs the help of the interrupt handler of the device which asked for this feature.

This has been implemented before by Jan Kiszka in several iterations and I have to admit that I led Jan down the wrong road when I suggested to put the shared status into irq_data.

Seems my brain tricked me as I'm about to add a status field to irq_data for irq chips consumption, but of course that does not help interrupt handlers as they need to retrieve that information on every handler invocation, which is involves a radix tree lookup in the worst case.

Though avoiding the initialy proposed notifier ugliness in the first place and (ab^H^Hre)using the interrupt handler which has to be aware about the shared/non shared state already turned out to be a not too bad idea. Just my init idea sucked a bit.

Instead of retrieving irq_data we encode the shared state and the transition to it into the lower two bits of the dev_id pointer which is the second argument to the interrupt handler if and only if the driver requested the interrupt with the new IRQF_ADAPTIVE_SHARED flag set.

Thanks,

tglx

#1 lore raw

RFC PATCH 1/4 genirq: Globally serialize request/free_irq

request/free_irq is not a hotpath, so we can afford to serialize it with a global lock. That allows us to shorten critical sections as the action chain of an irq descriptor becomes protected by the global lock. It's also a prerequisite for further changes which provide adaptive oneshot functionality for possibly shared interrupts.

Cc Tom Lyon <[email protected]>
Cc Alex Williamson <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Marcelo Tosatti <[email protected]>
Cc Jan Kiszka <[email protected]>
Cc Jan Kiszka <[email protected]>
---
 kernel/irq/manage.c |   19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
kernel/irq/manage.c +18 -1
Index: linux-2.6-tip/kernel/irq/manage.c
===================================================================
@@ -14,9 +14,12 @@
 #include <linux/interrupt.h>
 #include <linux/slab.h>
 #include <linux/sched.h>
+#include <linux/mutex.h>
 
 #include "internals.h"
 
+static DEFINE_MUTEX(register_lock);
+
 /**
  *	synchronize_irq - wait for pending IRQ handlers (on other CPUs)
  *	@irq: interrupt number to wait for
@@ -870,8 +873,12 @@ out_thread:
 int setup_irq(unsigned int irq, struct irqaction *act)
 {
 	struct irq_desc *desc = irq_to_desc(irq);
+	int ret;
 
-	return __setup_irq(irq, desc, act);
+	mutex_lock(&register_lock);
+	ret = __setup_irq(irq, desc, act);
+	mutex_unlock(&register_lock);
+	return ret;
 }
 EXPORT_SYMBOL_GPL(setup_irq);
 
@@ -977,7 +984,9 @@ static struct irqaction *__free_irq(unsi
  */
 void remove_irq(unsigned int irq, struct irqaction *act)
 {
+	mutex_lock(&register_lock);
 	__free_irq(irq, act->dev_id);
+	mutex_unlock(&register_lock);
 }
 EXPORT_SYMBOL_GPL(remove_irq);
 
@@ -1002,9 +1011,13 @@ void free_irq(unsigned int irq, void *de
 	if (!desc)
 		return;
 
+	mutex_lock(&register_lock);
+
 	chip_bus_lock(desc);
 	kfree(__free_irq(irq, dev_id));
 	chip_bus_sync_unlock(desc);
+
+	mutex_unlock(&register_lock);
 }
 EXPORT_SYMBOL(free_irq);
 
@@ -1091,10 +1104,14 @@ int request_threaded_irq(unsigned int ir
 	action->name = devname;
 	action->dev_id = dev_id;
 
+	mutex_lock(&register_lock);
+
 	chip_bus_lock(desc);
 	retval = __setup_irq(irq, desc, action);
 	chip_bus_sync_unlock(desc);
 
+	mutex_unlock(&register_lock);
+
 	if (retval)
 		kfree(action);
 
#2 lore raw

RFC PATCH 2/4 genirq: Move action walk outside of desc->lock held region

In reply to Thomas Gleixner

The global serialization of request/free_irq allows us to reduce the spinlocked and interrupt disabled regions as the global lock protects the action chain and basic setup functions.

Cc Tom Lyon <[email protected]>
Cc Alex Williamson <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Marcelo Tosatti <[email protected]>
Cc Jan Kiszka <[email protected]>
Cc Jan Kiszka <[email protected]>
---
 kernel/irq/manage.c |   73 +++++++++++++++++++++++++++-------------------------
 1 file changed, 39 insertions(+), 34 deletions(-)
kernel/irq/manage.c +39 -34
Index: linux-2.6-tip/kernel/irq/manage.c
===================================================================
@@ -658,8 +658,8 @@ __setup_irq(unsigned int irq, struct irq
 {
 	struct irqaction *old, **old_ptr;
 	const char *old_name = NULL;
+	bool nested, shared = false;
 	unsigned long flags;
-	int nested, shared = 0;
 	int ret;
 
 	if (!desc)
@@ -705,30 +705,10 @@ __setup_irq(unsigned int irq, struct irq
 	}
 
 	/*
-	 * Create a handler thread when a thread function is supplied
-	 * and the interrupt does not nest into another interrupt
-	 * thread.
+	 * This is called with register_lock held, so we can walk the
+	 * action chain w/o holding desc->lock. Nothing can add or
+	 * remove an action.
 	 */
-	if (new->thread_fn && !nested) {
-		struct task_struct *t;
-
-		t = kthread_create(irq_thread, new, "irq/%d-%s", irq,
-				   new->name);
-		if (IS_ERR(t))
-			return PTR_ERR(t);
-		/*
-		 * We keep the reference to the task struct even if
-		 * the thread dies to avoid that the interrupt code
-		 * references an already freed task_struct.
-		 */
-		get_task_struct(t);
-		new->thread = t;
-	}
-
-	/*
-	 * The following block of code has to be executed atomically
-	 */
-	raw_spin_lock_irqsave(&desc->lock, flags);
 	old_ptr = &desc->action;
 	old = *old_ptr;
 	if (old) {
@@ -756,14 +736,36 @@ __setup_irq(unsigned int irq, struct irq
 			old_ptr = &old->next;
 			old = *old_ptr;
 		} while (old);
-		shared = 1;
+		shared = true;
+	}
+
+	/*
+	 * Create a handler thread when a thread function is supplied
+	 * and the interrupt does not nest into another interrupt
+	 * thread.
+	 */
+	if (new->thread_fn && !nested) {
+		struct task_struct *t;
+
+		t = kthread_create(irq_thread, new, "irq/%d-%s", irq,
+				   new->name);
+		if (IS_ERR(t))
+			return PTR_ERR(t);
+		/*
+		 * We keep the reference to the task struct even if
+		 * the thread dies to avoid that the interrupt code
+		 * references an already freed task_struct.
+		 */
+		get_task_struct(t);
+		new->thread = t;
 	}
 
 	if (!shared) {
 		irq_chip_set_defaults(desc->irq_data.chip);
-
 		init_waitqueue_head(&desc->wait_for_threads);
 
+		raw_spin_lock_irqsave(&desc->lock, flags);
+
 		/* Setup the type (level, edge polarity) if configured: */
 		if (new->flags & IRQF_TRIGGER_MASK) {
 			ret = __irq_set_trigger(desc, irq,
@@ -773,6 +775,7 @@ __setup_irq(unsigned int irq, struct irq
 				goto out_thread;
 		} else
 			compat_irq_chip_set_default_handler(desc);
+
 #if defined(CONFIG_IRQ_PER_CPU)
 		if (new->flags & IRQF_PERCPU)
 			desc->status |= IRQ_PER_CPU;
@@ -799,13 +802,17 @@ __setup_irq(unsigned int irq, struct irq
 		/* Set default affinity mask once everything is setup */
 		setup_affinity(irq, desc);
 
-	} else if ((new->flags & IRQF_TRIGGER_MASK)
-			&& (new->flags & IRQF_TRIGGER_MASK)
-				!= (desc->status & IRQ_TYPE_SENSE_MASK)) {
+	} else {
+		raw_spin_lock_irqsave(&desc->lock, flags);
+
+		if ((new->flags & IRQF_TRIGGER_MASK)
+		    && (new->flags & IRQF_TRIGGER_MASK)
+		    != (desc->status & IRQ_TYPE_SENSE_MASK)) {
 		/* hope the handler works with the actual trigger mode... */
 		pr_warning("IRQ %d uses trigger mode %d; requested %d\n",
 				irq, (int)(desc->status & IRQ_TYPE_SENSE_MASK),
 				(int)(new->flags & IRQF_TRIGGER_MASK));
+		}
 	}
 
 	new->irq = irq;
@@ -848,7 +855,7 @@ mismatch:
 		dump_stack();
 	}
 #endif
-	ret = -EBUSY;
+	return -EBUSY;
 
 out_thread:
 	raw_spin_unlock_irqrestore(&desc->lock, flags);
@@ -897,8 +904,6 @@ static struct irqaction *__free_irq(unsi
 	if (!desc)
 		return NULL;
 
-	raw_spin_lock_irqsave(&desc->lock, flags);
-
 	/*
 	 * There can be multiple actions per IRQ descriptor, find the right
 	 * one based on the dev_id:
@@ -909,8 +914,6 @@ static struct irqaction *__free_irq(unsi
 
 		if (!action) {
 			WARN(1, "Trying to free already-free IRQ %d\n", irq);
-			raw_spin_unlock_irqrestore(&desc->lock, flags);
-
 			return NULL;
 		}
 
@@ -919,6 +922,8 @@ static struct irqaction *__free_irq(unsi
 		action_ptr = &action->next;
 	}
 
+	raw_spin_lock_irqsave(&desc->lock, flags);
+
 	/* Found it - now remove it from the list of entries: */
 	*action_ptr = action->next;
 
#3 lore raw

RFC PATCH 3/4 genirq: Add notification mechanism for adaptive shared irqs

In reply to Thomas Gleixner

The final goal of these modifications is to allow an adaptive oneshot mode for possibly shared interrupts. If the interrupt is not shared then oneshot mode should be used as it is more efficient than masking/unmasking at the device level. When the interrupt becomes shared then the oneshot mode needs to be disabled and device level masking must be done.

That requires transitioning from one state to the other which needs the help of the interrupt handler of the device which asked for this feature.

First of all the interrupt handler needs to know whether it needs to mask at the device level or if it can avoid that and rely on the line level masking. To avoid a lookup of irq_data/desc in every interrupt we use the lower 2 bits of the device id pointer to encode the shared state and a transitional state. This reduces the runtime overhead to almost zero. The encoding of the lower two bits of dev_id depends on IRQF_ADAPTIVE_SHARED set, so there is no conflict with existing drivers. Driver which use the IRQF_ADAPTIVE_SHARED flag need to decode dev_id to gain this functionality.

The two bits which are encoded are:
IRQ_DEV_ID_SHARED and IRQ_DEV_ID_PREPARE_SHARED

IRQ_DEV_ID_SHARED indicates that the irq line is shared.

IRQ_DEV_ID_PREPARE_SHARED indicates the transition from unshared to shared. Handler must mask at the device level, when an interrupt is on the fly. This requires local state tracking in the device handler.

The transition from shared to non shared requires just the correct order of removing the IRQ_DEV_ID_SHARED bit from the dev_id vs. removing the last shared action handler from the action chain. In the worst case (if an interrupt is on the fly) the device mask is kept until the on the fly interrupt is finished. After that the line based masking takes place.

Cc Tom Lyon <[email protected]>
Cc Alex Williamson <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Marcelo Tosatti <[email protected]>
Cc Jan Kiszka <[email protected]>
Cc Jan Kiszka <[email protected]>
---
 include/linux/interrupt.h |   34 +++++++++++++++++++++++
 kernel/irq/manage.c       |   66 +++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 94 insertions(+), 6 deletions(-)
include/linux/interrupt.h +34 -0
Index: linux-2.6-tip/include/linux/interrupt.h
===================================================================
@@ -55,6 +55,8 @@
  *                Used by threaded interrupts which need to keep the
  *                irq line disabled until the threaded handler has been run.
  * IRQF_NO_SUSPEND - Do not disable this IRQ during suspend
+ * IRQF_ADAPTIVE_SHARED - Request notification about interrupt line
+ *                        sharing in the dev_id argument
  *
  */
 #define IRQF_DISABLED		0x00000020
@@ -67,6 +69,7 @@
 #define IRQF_IRQPOLL		0x00001000
 #define IRQF_ONESHOT		0x00002000
 #define IRQF_NO_SUSPEND		0x00004000
+#define IRQF_ADAPTIVE_SHARED	0x00008000
 
 #define IRQF_TIMER		(__IRQF_TIMER | IRQF_NO_SUSPEND)
 
@@ -126,6 +129,37 @@ struct irqaction {
 
 extern irqreturn_t no_action(int cpl, void *dev_id);
 
+/*
+ * Macros and functions for interrupt handlers which need to know
+ * about the sharing status of the interrupt line
+ * (IRQF_SHARED_ADAPTIVE)
+ */
+#define IRQ_DEV_ID_SHARED		0x01UL
+#define IRQ_DEV_ID_PREPARE_SHARED	0x02UL
+#define IRQ_DEV_ID_MASK			0x03UL
+
+static inline void *irq_real_dev_id(void *dev_id)
+{
+	unsigned long id = ((unsigned long) dev_id) & ~IRQ_DEV_ID_MASK;
+	return (void *)id;
+}
+
+static inline void *irq_modify_dev_id(void *dev_id, unsigned long mask)
+{
+	unsigned long id = ((unsigned long) dev_id) | mask;
+	return (void *)id;
+}
+
+static inline bool irq_dev_id_is_shared(void *dev_id)
+{
+	return ((unsigned long) dev_id) & IRQ_DEV_ID_SHARED;
+}
+
+static inline bool irq_dev_id_prepare_shared(void *dev_id)
+{
+	return ((unsigned long) dev_id) & IRQ_DEV_ID_PREPARE_SHARED;
+}
+
 #ifdef CONFIG_GENERIC_HARDIRQS
 extern int __must_check
 request_threaded_irq(unsigned int irq, irq_handler_t handler,
kernel/irq/manage.c +60 -6

107 lines not on this page · raw message

#4 lore raw

RFC PATCH 4/4 genirq: Add support for IRQF_COND_ONESHOT

In reply to Thomas Gleixner

From: Jan Kiszka <[email protected]>

Provide an adaptive version of IRQF_ONESHOT: If the line is exclusively used, IRQF_COND_ONESHOT provides the same semantics as IRQF_ONESHOT. If it is shared, the line will be unmasked directly after the hardirq handler, just as if IRQF_COND_ONESHOT was not provided.

[ tglx: adapted to the dev_id based notification mechanism ]

Cc Tom Lyon <[email protected]>
Cc Alex Williamson <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Marcelo Tosatti <[email protected]>
---
 include/linux/interrupt.h |    3 +++
 kernel/irq/manage.c       |   25 +++++++++++++++++--------
 2 files changed, 20 insertions(+), 8 deletions(-)
include/linux/interrupt.h +3 -0
Index: linux-2.6-tip/include/linux/interrupt.h
===================================================================
@@ -57,6 +57,8 @@
  * IRQF_NO_SUSPEND - Do not disable this IRQ during suspend
  * IRQF_ADAPTIVE_SHARED - Request notification about interrupt line
  *                        sharing in the dev_id argument
+ * IRQF_COND_ONESHOT - If line is not shared, keep interrupt disabled after
+ *                     hardirq handler finshed.
  *
  */
 #define IRQF_DISABLED		0x00000020
@@ -70,6 +72,7 @@
 #define IRQF_ONESHOT		0x00002000
 #define IRQF_NO_SUSPEND		0x00004000
 #define IRQF_ADAPTIVE_SHARED	0x00008000
+#define IRQF_COND_ONESHOT	0x00010000
 
 #define IRQF_TIMER		(__IRQF_TIMER | IRQF_NO_SUSPEND)
 
kernel/irq/manage.c +17 -8
Index: linux-2.6-tip/kernel/irq/manage.c
===================================================================
@@ -583,7 +583,7 @@ static int irq_thread(void *data)
 	struct sched_param param = { .sched_priority = MAX_USER_RT_PRIO/2, };
 	struct irqaction *action = data;
 	struct irq_desc *desc = irq_to_desc(action->irq);
-	int wake, oneshot = desc->status & IRQ_ONESHOT;
+	int wake, oneshot;
 
 	sched_setscheduler(current, SCHED_FIFO, &param);
 	current->irqaction = action;
@@ -606,6 +606,7 @@ static int irq_thread(void *data)
 			desc->status |= IRQ_PENDING;
 			raw_spin_unlock_irq(&desc->lock);
 		} else {
+			oneshot = desc->status & IRQ_ONESHOT;
 			raw_spin_unlock_irq(&desc->lock);
 
 			action->thread_fn(action->irq, action->dev_id);
@@ -657,8 +658,9 @@ void exit_irq_thread(void)
  * in, as new actions have the shared bit set already before they are
  * added to the action chain.
  */
-static void irq_notify_shared(unsigned int irq, struct irqaction *action)
+static void irq_notify_shared(unsigned int irq, struct irq_desc *desc)
 {
+	struct irqaction *action = desc->action;
 	unsigned long flags;
 
 	if (!(action->flags & IRQF_ADAPTIVE_SHARED) ||
@@ -667,15 +669,19 @@ static void irq_notify_shared(unsigned i
 
 	disable_irq(irq);
 	action->dev_id = irq_modify_dev_id(action->dev_id, IRQ_DEV_ID_SHARED);
+
 	local_irq_save(flags);
 	action->handler(irq, irq_modify_dev_id(action->dev_id,
 					       IRQ_DEV_ID_PREPARE_SHARED));
-	local_irq_restore(flags);
+
+	/* Clear the oneshot flag */
+	raw_spin_lock(&desc->lock);
+	desc->status &= ~IRQ_ONESHOT;
+	raw_spin_unlock_irqrestore(&desc->lock, flags);
 
 	/*
-	 * This also unmasks an eventually masked irq line. The
-	 * handler has masked the device if an interrupt was on the
-	 * fly.
+	 * Reenable the interrupt if desc->depth == 1. This is safe
+	 * now as the handler has masked at the device level.
 	 */
 	enable_irq(irq);
 }
@@ -815,7 +821,7 @@ __setup_irq(unsigned int irq, struct irq
 		desc->status &= ~(IRQ_AUTODETECT | IRQ_WAITING | IRQ_ONESHOT |
 				  IRQ_INPROGRESS | IRQ_SPURIOUS_DISABLED);
 
-		if (new->flags & IRQF_ONESHOT)
+		if (new->flags & (IRQF_ONESHOT | IRQF_COND_ONESHOT))
 			desc->status |= IRQ_ONESHOT;
 
 		if (!(desc->status & IRQ_NOAUTOEN)) {
@@ -835,7 +841,7 @@ __setup_irq(unsigned int irq, struct irq
 
 	} else {
 		/* Take care of adaptive shared interrupts */
-		irq_notify_shared(irq, desc->action);
+		irq_notify_shared(irq, desc);
 		if (new->flags & IRQF_ADAPTIVE_SHARED)
 			new->dev_id = irq_modify_dev_id(new->dev_id,
 							IRQ_DEV_ID_SHARED);
@@ -983,6 +989,9 @@ static struct irqaction *__free_irq(unsi
 			desc->irq_data.chip->irq_shutdown(&desc->irq_data);
 		else
 			desc->irq_data.chip->irq_disable(&desc->irq_data);
+	} else if (!desc->action->next) {
+		if (desc->action->flags & IRQF_COND_ONESHOT)
+			desc->status |= IRQ_ONESHOT;
 	}
 
 #ifdef CONFIG_SMP