Skip to contentSkip to conversation

Clean up SMP IPI support

PATCH New Russell King - ARM Linux 2 participants 17 messages Nov 15, 2010 – Nov 17, 2010

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover Clean up SMP IPI support

This series is aimed at cleaning up our SMP IPI support. As has been recently pointed out, we have a spinlock in percpu data, which is not very nice.

If we switch to using the GIC SGIs, one for each IPI function, then we don't need the mask of ipi bits indicating what work needs to be done. This allows us to get rid of the troublesome spinlock in send_ipi_message().

However, we end up with slightly more expense on the IPI receiver side, as they no longer have a word to read to process a set of pending IPIs. This isn't much of a down-side as you hardly end up with masses of pending IPIs. Modern Cortex A9 platforms usually only clock up the reschedule IPI.

This leaves us with just ipi_count in the ipi per-cpu data structure. Other platforms put this in the global irq_stat array, and break this down into individual IPIs. So, let's do the same.

One down-side of this patchset is that platforms which use SGI 1 to wake their secondary CPUs will receive an "Unknown IPI message" kernel critical message at boot time - as SGI 1 is no longer recognised as a Linux IPI interrupt. We _could_ make this 'IPI_BOOT', something which is only used for booting and therefore simply ignored.

#1 lore raw

PATCH 1/9 ARM: remove obsolete smp_cross_call_done()

smp_cross_call_done() was removed long ago (see 78d236c - remove useless smp_cross_call_done()). Remove those which have been subsequently merged.

---
 arch/arm/mach-tegra/include/mach/smp.h |    7 -------
 1 files changed, 0 insertions(+), 7 deletions(-)
arch/arm/mach-tegra/include/mach/smp.h +0 -7
index e4a34a3..d3cd265 100644
@@ -12,11 +12,4 @@ static inline void smp_cross_call(const struct cpumask *mask)
 	gic_raise_softirq(mask, 1);
 }
 
-/*
- * Do nothing on MPcore.
- */
-static inline void smp_cross_call_done(cpumask_t callmap)
-{
-}
-
 #endif
Signature · 1 line
1.6.2.5
#2 lore raw

PATCH 2/9 ARM: smp: pass an ipi number to smp_cross_call()

In reply to Russell King - ARM Linux

This allows us to use smp_cross_call() to trigger a number of different software generated interrupts, rather than combining them all on one SGI. Recover the SGI number via do_IPI.

---
 arch/arm/include/asm/smp.h                |    4 ++--
 arch/arm/kernel/entry-armv.S              |    2 +-
 arch/arm/kernel/smp.c                     |   10 ++--------
 arch/arm/mach-msm/include/mach/smp.h      |    4 ++--
 arch/arm/mach-omap2/omap-smp.c            |    2 +-
 arch/arm/mach-realview/include/mach/smp.h |    4 ++--
 arch/arm/mach-realview/platsmp.c          |    2 +-
 arch/arm/mach-s5pv310/include/mach/smp.h  |    4 ++--
 arch/arm/mach-s5pv310/platsmp.c           |    2 +-
 arch/arm/mach-tegra/include/mach/smp.h    |    4 ++--
 arch/arm/mach-ux500/include/mach/smp.h    |    4 ++--
 arch/arm/mach-ux500/platsmp.c             |    2 +-
 arch/arm/mach-vexpress/include/mach/smp.h |    4 ++--
 arch/arm/mach-vexpress/platsmp.c          |    2 +-
 arch/arm/plat-omap/include/plat/smp.h     |    4 ++--
 15 files changed, 24 insertions(+), 30 deletions(-)
arch/arm/include/asm/smp.h +2 -2
index 3d05190..da7e7ca 100644
@@ -38,7 +38,7 @@ extern void show_ipi_list(struct seq_file *p);
 /*
  * Called from assembly code, this handles an IPI.
  */
-asmlinkage void do_IPI(struct pt_regs *regs);
+asmlinkage void do_IPI(int ipinr, struct pt_regs *regs);
 
 /*
  * Setup the set of possible CPUs (via set_cpu_possible)
@@ -53,7 +53,7 @@ extern void smp_store_cpu_info(unsigned int cpuid);
 /*
  * Raise an IPI cross call on CPUs in callmap.
  */
-extern void smp_cross_call(const struct cpumask *mask);
+extern void smp_cross_call(const struct cpumask *mask, int ipi);
 
 /*
  * Boot a secondary CPU, and assign it the specified idle task.
arch/arm/kernel/entry-armv.S +1 -1
index c09e357..955cf5f 100644
@@ -48,7 +48,7 @@
 	 */
 	ALT_SMP(test_for_ipi r0, r6, r5, lr)
 	ALT_UP_B(9997f)
-	movne	r0, sp
+	movne	r1, sp
 	adrne	lr, BSYM(1b)
 	bne	do_IPI
 
arch/arm/kernel/smp.c +2 -8
index 8c19595..7a236db 100644
@@ -404,7 +404,7 @@ static void send_ipi_message(const struct cpumask *mask, enum ipi_msg_type msg)
 	/*
 	 * Call the platform specific cross-CPU call function.
 	 */
-	smp_cross_call(mask);
+	smp_cross_call(mask, 1);
 
 	local_irq_restore(flags);
 }
@@ -537,14 +537,8 @@ static void ipi_cpu_stop(unsigned int cpu)
 
 /*
  * Main handler for inter-processor interrupts
- *
- * For ARM, the ipimask now only identifies a single
- * category of IPI (Bit 1 IPIs have been replaced by a
- * different mechanism):
- *
- *  Bit 0 - Inter-processor function call
  */
-asmlinkage void __exception do_IPI(struct pt_regs *regs)
+asmlinkage void __exception do_IPI(int ipinr, struct pt_regs *regs)
 {
 	unsigned int cpu = smp_processor_id();
 	struct ipi_data *ipi = &per_cpu(ipi_data, cpu);
arch/arm/mach-msm/include/mach/smp.h +2 -2
index 3ff7bf5..a95f7b9 100644
@@ -31,9 +31,9 @@
 
 #include <asm/hardware/gic.h>
 
-static inline void smp_cross_call(const struct cpumask *mask)
+static inline void smp_cross_call(const struct cpumask *mask, int ipi)
 {
-	gic_raise_softirq(mask, 1);
+	gic_raise_softirq(mask, ipi);
 }
 
 #endif
arch/arm/mach-omap2/omap-smp.c +1 -1
index 9e9f70e..56a8bce 100644
@@ -76,7 +76,7 @@ int __cpuinit boot_secondary(unsigned int cpu, struct task_struct *idle)
 	omap_modify_auxcoreboot0(0x200, 0xfffffdff);
 	flush_cache_all();
 	smp_wmb();
-	smp_cross_call(cpumask_of(cpu));
+	smp_cross_call(cpumask_of(cpu), 1);
 
 	/*
 	 * Now the secondary core is starting up let it run its
arch/arm/mach-realview/include/mach/smp.h +2 -2
index d3cd265..d1aa704 100644
@@ -7,9 +7,9 @@
 /*
  * We use IRQ1 as the IPI
  */
-static inline void smp_cross_call(const struct cpumask *mask)
+static inline void smp_cross_call(const struct cpumask *mask, int ipi)
 {
-	gic_raise_softirq(mask, 1);
+	gic_raise_softirq(mask, ipi);
 }
 
 #endif
arch/arm/mach-realview/platsmp.c +1 -1
index 0092658..af3d909 100644
@@ -116,7 +116,7 @@ int __cpuinit boot_secondary(unsigned int cpu, struct task_struct *idle)
 	 * Use smp_cross_call() for this, since there's little
 	 * point duplicating the code here
 	 */
-	smp_cross_call(cpumask_of(cpu));
+	smp_cross_call(cpumask_of(cpu), 1);
 
 	timeout = jiffies + (1 * HZ);
 	while (time_before(jiffies, timeout)) {
arch/arm/mach-s5pv310/include/mach/smp.h +2 -2
index b7ec252..2897747 100644
@@ -14,9 +14,9 @@ extern void __iomem *gic_cpu_base_addr;
 /*
  * We use IRQ1 as the IPI
  */
-static inline void smp_cross_call(const struct cpumask *mask)
+static inline void smp_cross_call(const struct cpumask *mask, int ipi)
 {
-	gic_raise_softirq(mask, 1);
+	gic_raise_softirq(mask, ipi);
 }
 
 #endif
arch/arm/mach-s5pv310/platsmp.c +1 -1
index d357c19..d474426 100644
@@ -97,7 +97,7 @@ int __cpuinit boot_secondary(unsigned int cpu, struct task_struct *idle)
 	 * the boot monitor to read the system wide flags register,
 	 * and branch to the address found there.
 	 */
-	smp_cross_call(cpumask_of(cpu));
+	smp_cross_call(cpumask_of(cpu), 1);
 
 	timeout = jiffies + (1 * HZ);
 	while (time_before(jiffies, timeout)) {
arch/arm/mach-tegra/include/mach/smp.h +2 -2
index d3cd265..d1aa704 100644
@@ -7,9 +7,9 @@
 /*
  * We use IRQ1 as the IPI
  */
-static inline void smp_cross_call(const struct cpumask *mask)
+static inline void smp_cross_call(const struct cpumask *mask, int ipi)
 {
-	gic_raise_softirq(mask, 1);
+	gic_raise_softirq(mask, ipi);
 }
 
 #endif
arch/arm/mach-ux500/include/mach/smp.h +2 -2
index 197e841..bd57c50 100644
@@ -18,8 +18,8 @@ extern void u8500_secondary_startup(void);
 /*
  * We use IRQ1 as the IPI
  */
-static inline void smp_cross_call(const struct cpumask *mask)
+static inline void smp_cross_call(const struct cpumask *mask, int ipi)
 {
-	gic_raise_softirq(mask, 1);
+	gic_raise_softirq(mask, ipi);
 }
 #endif
arch/arm/mach-ux500/platsmp.c +1 -1
index 9e4c678..b8987bd 100644
@@ -78,7 +78,7 @@ int __cpuinit boot_secondary(unsigned int cpu, struct task_struct *idle)
 	__cpuc_flush_dcache_area((void *)&pen_release, sizeof(pen_release));
 	outer_clean_range(__pa(&pen_release), __pa(&pen_release) + 1);
 
-	smp_cross_call(cpumask_of(cpu));
+	smp_cross_call(cpumask_of(cpu), 1);
 
 	timeout = jiffies + (1 * HZ);
 	while (time_before(jiffies, timeout)) {
arch/arm/mach-vexpress/include/mach/smp.h +2 -2
index 5a6da4f..721be0f 100644
@@ -7,8 +7,8 @@
 /*
  * We use IRQ1 as the IPI
  */
-static inline void smp_cross_call(const struct cpumask *mask)
+static inline void smp_cross_call(const struct cpumask *mask, int ipi)
 {
-	gic_raise_softirq(mask, 1);
+	gic_raise_softirq(mask, ipi);
 }
 #endif
arch/arm/mach-vexpress/platsmp.c +1 -1
index 6709706..276f916 100644
@@ -92,7 +92,7 @@ int __cpuinit boot_secondary(unsigned int cpu, struct task_struct *idle)
 	 * the boot monitor to read the system wide flags register,
 	 * and branch to the address found there.
 	 */
-	smp_cross_call(cpumask_of(cpu));
+	smp_cross_call(cpumask_of(cpu), 1);
 
 	timeout = jiffies + (1 * HZ);
 	while (time_before(jiffies, timeout)) {
arch/arm/plat-omap/include/plat/smp.h +2 -2
index ecd6a48..e5541e5 100644
@@ -29,9 +29,9 @@ extern u32 omap_read_auxcoreboot0(void);
 /*
  * We use Soft IRQ1 as the IPI
  */
-static inline void smp_cross_call(const struct cpumask *mask)
+static inline void smp_cross_call(const struct cpumask *mask, int ipi)
 {
-	gic_raise_softirq(mask, 1);
+	gic_raise_softirq(mask, ipi);
 }
 
 #endif
Signature · 1 line
1.6.2.5
#3 lore raw

PATCH 2/9 ARM: smp: pass an ipi number to smp_cross_call()

arch/arm/mach-msm/include/mach/smp.h line 37

Quoted 12 lines · Russell King - ARM Linux
arch/arm/mach-msm/include/mach/smp.h +2 -2
@@ -31,9 +31,9 @@
 
 #include <asm/hardware/gic.h>
 
-static inline void smp_cross_call(const struct cpumask *mask)
+static inline void smp_cross_call(const struct cpumask *mask, int ipi)
 {
-       gic_raise_softirq(mask, 1);
+       gic_raise_softirq(mask, ipi);
 }

BTW, why don't we make smp_cross_call() generic, it looks like all platforms define the same thing.

Unless we get SMP systems in the future without a GIC.

Signature · 1 line
Catalin
#4 lore raw

PATCH 3/9 ARM: smp: avoid using bitmasks and locks for IPIs, use hardware instead

In reply to Russell King - ARM Linux

Avoid using bitmasks and locks in the percpu area for IPIs, and instead use individual software generated interrupts to identify the reason for the IPI. This avoids the problems of having spinlocks in the percpu area.

---
 arch/arm/kernel/smp.c |   87 ++++++++++++++----------------------------------
 1 files changed, 26 insertions(+), 61 deletions(-)
arch/arm/kernel/smp.c +26 -61
index 7a236db..78d55c6 100644
@@ -48,20 +48,15 @@ struct secondary_data secondary_data;
 
 /*
  * structures for inter-processor calls
- * - A collection of single bit ipi messages.
  */
 struct ipi_data {
-	spinlock_t lock;
 	unsigned long ipi_count;
-	unsigned long bits;
 };
 
-static DEFINE_PER_CPU(struct ipi_data, ipi_data) = {
-	.lock	= SPIN_LOCK_UNLOCKED,
-};
+static DEFINE_PER_CPU(struct ipi_data, ipi_data);
 
 enum ipi_msg_type {
-	IPI_TIMER,
+	IPI_TIMER = 2,
 	IPI_RESCHEDULE,
 	IPI_CALL_FUNC,
 	IPI_CALL_FUNC_SINGLE,
@@ -389,22 +384,13 @@ void __init smp_prepare_boot_cpu(void)
 static void send_ipi_message(const struct cpumask *mask, enum ipi_msg_type msg)
 {
 	unsigned long flags;
-	unsigned int cpu;
 
 	local_irq_save(flags);
 
-	for_each_cpu(cpu, mask) {
-		struct ipi_data *ipi = &per_cpu(ipi_data, cpu);
-
-		spin_lock(&ipi->lock);
-		ipi->bits |= 1 << msg;
-		spin_unlock(&ipi->lock);
-	}
-
 	/*
 	 * Call the platform specific cross-CPU call function.
 	 */
-	smp_cross_call(mask, 1);
+	smp_cross_call(mask, msg);
 
 	local_irq_restore(flags);
 }
@@ -546,56 +532,35 @@ asmlinkage void __exception do_IPI(int ipinr, struct pt_regs *regs)
 
 	ipi->ipi_count++;
 
-	for (;;) {
-		unsigned long msgs;
-
-		spin_lock(&ipi->lock);
-		msgs = ipi->bits;
-		ipi->bits = 0;
-		spin_unlock(&ipi->lock);
-
-		if (!msgs)
-			break;
-
-		do {
-			unsigned nextmsg;
-
-			nextmsg = msgs & -msgs;
-			msgs &= ~nextmsg;
-			nextmsg = ffz(~nextmsg);
-
-			switch (nextmsg) {
-			case IPI_TIMER:
-				ipi_timer();
-				break;
+	switch (ipinr) {
+	case IPI_TIMER:
+		ipi_timer();
+		break;
 
-			case IPI_RESCHEDULE:
-				/*
-				 * nothing more to do - eveything is
-				 * done on the interrupt return path
-				 */
-				break;
+	case IPI_RESCHEDULE:
+		/*
+		 * nothing more to do - eveything is
+		 * done on the interrupt return path
+		 */
+		break;
 
-			case IPI_CALL_FUNC:
-				generic_smp_call_function_interrupt();
-				break;
+	case IPI_CALL_FUNC:
+		generic_smp_call_function_interrupt();
+		break;
 
-			case IPI_CALL_FUNC_SINGLE:
-				generic_smp_call_function_single_interrupt();
-				break;
+	case IPI_CALL_FUNC_SINGLE:
+		generic_smp_call_function_single_interrupt();
+		break;
 
-			case IPI_CPU_STOP:
-				ipi_cpu_stop(cpu);
-				break;
+	case IPI_CPU_STOP:
+		ipi_cpu_stop(cpu);
+		break;
 
-			default:
-				printk(KERN_CRIT "CPU%u: Unknown IPI message 0x%x\n",
-				       cpu, nextmsg);
-				break;
-			}
-		} while (msgs);
+	default:
+		printk(KERN_CRIT "CPU%u: Unknown IPI message 0x%x\n",
+		       cpu, ipinr);
+		break;
 	}
-
 	set_irq_regs(old_regs);
 }
 
Signature · 1 line
1.6.2.5
#5 lore raw

PATCH 3/9 ARM: smp: avoid using bitmasks and locks for IPIs, use hardware instead

Quoted 14 lines · Russell King - ARM Linux

Avoid using bitmasks and locks in the percpu area for IPIs, and instead use individual software generated interrupts to identify the reason for the IPI. This avoids the problems of having spinlocks in the percpu area.

---
 arch/arm/kernel/smp.c |   87 ++++++++++++++----------------------------------
 1 files changed, 26 insertions(+), 61 deletions(-)
arch/arm/kernel/smp.c +0 -0
index 7a236db..78d55c6 100644

[...]

arch/arm/kernel/smp.c struct secondary_data secondary_data; · line 62

Quoted 6 lines

enum ipi_msg_type {

arch/arm/kernel/smp.c +1 -1
-       IPI_TIMER,
+       IPI_TIMER = 2,
        IPI_RESCHEDULE,
        IPI_CALL_FUNC,
        IPI_CALL_FUNC_SINGLE,

Maybe a comment here on why this starts at 2. People may wonder in the future.

Signature · 1 line
Catalin
#6 lore raw

PATCH 3/9 ARM: smp: avoid using bitmasks and locks for IPIs, use hardware instead

Quoted 23 lines · Catalin Marinas
Quoted 14 lines · Russell King - ARM Linux

Avoid using bitmasks and locks in the percpu area for IPIs, and instead use individual software generated interrupts to identify the reason for the IPI. This avoids the problems of having spinlocks in the percpu area.

---
 arch/arm/kernel/smp.c |   87 ++++++++++++++----------------------------------
 1 files changed, 26 insertions(+), 61 deletions(-)
arch/arm/kernel/smp.c +0 -0
index 7a236db..78d55c6 100644

[...]

arch/arm/kernel/smp.c struct secondary_data secondary_data; · line 62

Quoted 6 lines

enum ipi_msg_type {

arch/arm/kernel/smp.c +1 -1
-       IPI_TIMER,
+       IPI_TIMER = 2,
        IPI_RESCHEDULE,
        IPI_CALL_FUNC,
        IPI_CALL_FUNC_SINGLE,
Quoted 2 lines

Maybe a comment here on why this starts at 2. People may wonder in the future.

I would like to get rid of the 'starts at 2' but we can only do this once we're sure that the use of SGI1 to wake up the CPUs is not something that is a hard requirement for boot loaders.

I've not yet tested whether Realview and Versatile Express are fine with a different SGI number.

#7 lore raw

PATCH 3/9 ARM: smp: avoid using bitmasks and locks for IPIs, use hardware instead

Quoted 27 lines · Russell King - ARM Linux
Quoted 23 lines · Catalin Marinas
Quoted 14 lines · Russell King - ARM Linux

Avoid using bitmasks and locks in the percpu area for IPIs, and instead use individual software generated interrupts to identify the reason for the IPI. This avoids the problems of having spinlocks in the percpu area.

---
 arch/arm/kernel/smp.c |   87 ++++++++++++++----------------------------------
 1 files changed, 26 insertions(+), 61 deletions(-)
arch/arm/kernel/smp.c +0 -0
index 7a236db..78d55c6 100644

[...]

arch/arm/kernel/smp.c struct secondary_data secondary_data; · line 62

Quoted 6 lines

enum ipi_msg_type {

arch/arm/kernel/smp.c +1 -1
-       IPI_TIMER,
+       IPI_TIMER = 2,
        IPI_RESCHEDULE,
        IPI_CALL_FUNC,
        IPI_CALL_FUNC_SINGLE,
Quoted 2 lines

Maybe a comment here on why this starts at 2. People may wonder in the future.

Quoted 6 lines

I would like to get rid of the 'starts at 2' but we can only do this once we're sure that the use of SGI1 to wake up the CPUs is not something that is a hard requirement for boot loaders.

I've not yet tested whether Realview and Versatile Express are fine with a different SGI number.

AFAIK, the ARM boot monitor doesn't care about the IPI number, it just needs something to take it out of the WFI.

BTW, in boot_secondary() we could issue a SEV as well from the primary CPU. That's mainly useful when you don't have a full boot monitor to set up the GIC and it uses WFE in a loop (I have some simple wrapper for this, mainly used on software models).

Signature · 1 line
Catalin
#8 lore raw

PATCH 4/9 ARM: include local timer irq stats only when local timers configured

In reply to Russell King - ARM Linux
---
 arch/arm/include/asm/hardirq.h |    2 ++
 arch/arm/kernel/irq.c          |    2 ++
 arch/arm/kernel/smp.c          |   24 ++++++++++++------------
 3 files changed, 16 insertions(+), 12 deletions(-)
arch/arm/include/asm/hardirq.h +2 -0
index 6d7485a..522fb23 100644
@@ -7,7 +7,9 @@
 
 typedef struct {
 	unsigned int __softirq_pending;
+#ifdef CONFIG_LOCAL_TIMERS
 	unsigned int local_timer_irqs;
+#endif
 } ____cacheline_aligned irq_cpustat_t;
 
 #include <linux/irq_cpustat.h>	/* Standard mappings for irq_cpustat_t above */
arch/arm/kernel/irq.c +2 -0
index 36ad3be..ea29721 100644
@@ -93,6 +93,8 @@ unlock:
 #endif
 #ifdef CONFIG_SMP
 		show_ipi_list(p);
+#endif
+#ifdef CONFIG_LOCAL_TIMERS
 		show_local_irqs(p);
 #endif
 		seq_printf(p, "Err: %10lu\n", irq_err_count);
arch/arm/kernel/smp.c +12 -12
index 78d55c6..b58b6c4 100644
@@ -417,18 +417,6 @@ void show_ipi_list(struct seq_file *p)
 	seq_putc(p, '\n');
 }
 
-void show_local_irqs(struct seq_file *p)
-{
-	unsigned int cpu;
-
-	seq_printf(p, "LOC: ");
-
-	for_each_present_cpu(cpu)
-		seq_printf(p, "%10u ", irq_stat[cpu].local_timer_irqs);
-
-	seq_putc(p, '\n');
-}
-
 /*
  * Timer (local or broadcast) support
  */
@@ -455,6 +443,18 @@ asmlinkage void __exception do_local_timer(struct pt_regs *regs)
 
 	set_irq_regs(old_regs);
 }
+
+void show_local_irqs(struct seq_file *p)
+{
+	unsigned int cpu;
+
+	seq_printf(p, "LOC: ");
+
+	for_each_present_cpu(cpu)
+		seq_printf(p, "%10u ", irq_stat[cpu].local_timer_irqs);
+
+	seq_putc(p, '\n');
+}
 #endif
 
 #ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST
Signature · 1 line
1.6.2.5
#9 lore raw

PATCH 5/9 ARM: smp: provide accessors for irq_stat data

In reply to Russell King - ARM Linux

Provide __inc_irq_stat() and __get_irq_stat() to increment and read the irq stat counters.

---
 arch/arm/include/asm/hardirq.h |    3 +++
 arch/arm/kernel/smp.c          |    4 ++--
 2 files changed, 5 insertions(+), 2 deletions(-)
arch/arm/include/asm/hardirq.h +3 -0
index 522fb23..923eaa1 100644
@@ -14,6 +14,9 @@ typedef struct {
 
 #include <linux/irq_cpustat.h>	/* Standard mappings for irq_cpustat_t above */
 
+#define __inc_irq_stat(cpu, member)	__IRQ_STAT(cpu, member)++
+#define __get_irq_stat(cpu, member)	__IRQ_STAT(cpu, member)
+
 #if NR_IRQS > 512
 #define HARDIRQ_BITS	10
 #elif NR_IRQS > 256
arch/arm/kernel/smp.c +2 -2
index b58b6c4..f276336 100644
@@ -437,7 +437,7 @@ asmlinkage void __exception do_local_timer(struct pt_regs *regs)
 	int cpu = smp_processor_id();
 
 	if (local_timer_ack()) {
-		irq_stat[cpu].local_timer_irqs++;
+		__inc_irq_stat(cpu, local_timer_irqs);
 		ipi_timer();
 	}
 
@@ -451,7 +451,7 @@ void show_local_irqs(struct seq_file *p)
 	seq_printf(p, "LOC: ");
 
 	for_each_present_cpu(cpu)
-		seq_printf(p, "%10u ", irq_stat[cpu].local_timer_irqs);
+		seq_printf(p, "%10u ", __get_irq_stat(cpu, local_timer_irqs));
 
 	seq_putc(p, '\n');
 }
Signature · 1 line
1.6.2.5
#10 lore raw

PATCH 6/9 ARM: smp: move ipi_count into irq_stat structure

In reply to Russell King - ARM Linux

Move the ipi_count into irq_stat, which allows the ipi_data structure to be entirely removed.

---
 arch/arm/include/asm/hardirq.h |    3 +++
 arch/arm/kernel/smp.c          |   14 ++------------
 2 files changed, 5 insertions(+), 12 deletions(-)
arch/arm/include/asm/hardirq.h +3 -0
index 923eaa1..824b08a 100644
@@ -10,6 +10,9 @@ typedef struct {
 #ifdef CONFIG_LOCAL_TIMERS
 	unsigned int local_timer_irqs;
 #endif
+#ifdef CONFIG_SMP
+	unsigned int ipi_irqs;
+#endif
 } ____cacheline_aligned irq_cpustat_t;
 
 #include <linux/irq_cpustat.h>	/* Standard mappings for irq_cpustat_t above */
arch/arm/kernel/smp.c +2 -12
index f276336..edb8fd5 100644
@@ -46,15 +46,6 @@
  */
 struct secondary_data secondary_data;
 
-/*
- * structures for inter-processor calls
- */
-struct ipi_data {
-	unsigned long ipi_count;
-};
-
-static DEFINE_PER_CPU(struct ipi_data, ipi_data);
-
 enum ipi_msg_type {
 	IPI_TIMER = 2,
 	IPI_RESCHEDULE,
@@ -412,7 +403,7 @@ void show_ipi_list(struct seq_file *p)
 	seq_puts(p, "IPI:");
 
 	for_each_present_cpu(cpu)
-		seq_printf(p, " %10lu", per_cpu(ipi_data, cpu).ipi_count);
+		seq_printf(p, " %10u", __get_irq_stat(cpu, ipi_irqs));
 
 	seq_putc(p, '\n');
 }
@@ -527,10 +518,9 @@ static void ipi_cpu_stop(unsigned int cpu)
 asmlinkage void __exception do_IPI(int ipinr, struct pt_regs *regs)
 {
 	unsigned int cpu = smp_processor_id();
-	struct ipi_data *ipi = &per_cpu(ipi_data, cpu);
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
-	ipi->ipi_count++;
+	__inc_irq_stat(cpu, ipi_irqs);
 
 	switch (ipinr) {
 	case IPI_TIMER:
Signature · 1 line
1.6.2.5
#11 lore raw

PATCH 7/9 ARM: fix /proc/interrupts formatting

In reply to Russell King - ARM Linux

As per x86, align the initial column according to how many IRQs we have. Also, provide an english explaination for the 'LOC:' and 'IPI:' lines.

---
 arch/arm/include/asm/mach/irq.h |    2 +-
 arch/arm/include/asm/smp.h      |    4 ++--
 arch/arm/kernel/fiq.c           |    5 +++--
 arch/arm/kernel/irq.c           |   16 ++++++++++------
 arch/arm/kernel/smp.c           |   14 +++++++-------
 5 files changed, 23 insertions(+), 18 deletions(-)
arch/arm/include/asm/mach/irq.h +1 -1
index ce3eee9..2bc47fb 100644
@@ -20,7 +20,7 @@ struct seq_file;
 extern unsigned int arch_nr_irqs;
 extern void (*init_arch_irq)(void);
 extern void init_FIQ(void);
-extern int show_fiq_list(struct seq_file *, void *);
+extern int show_fiq_list(struct seq_file *, int);
 
 /*
  * This is for easy migration, but should be changed in the source
arch/arm/include/asm/smp.h +2 -2
index da7e7ca..f93d0a6 100644
@@ -33,7 +33,7 @@ struct seq_file;
 /*
  * generate IPI list text
  */
-extern void show_ipi_list(struct seq_file *p);
+extern void show_ipi_list(struct seq_file *, int);
 
 /*
  * Called from assembly code, this handles an IPI.
@@ -97,6 +97,6 @@ extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
 /*
  * show local interrupt info
  */
-extern void show_local_irqs(struct seq_file *);
+extern void show_local_irqs(struct seq_file *, int);
 
 #endif /* ifndef __ASM_ARM_SMP_H */
arch/arm/kernel/fiq.c +3 -2
index 6ff7919..47837b8 100644
@@ -67,10 +67,11 @@ static struct fiq_handler default_owner = {
 
 static struct fiq_handler *current_fiq = &default_owner;
 
-int show_fiq_list(struct seq_file *p, void *v)
+int show_fiq_list(struct seq_file *p, int prec)
 {
 	if (current_fiq != &default_owner)
-		seq_printf(p, "FIQ:              %s\n", current_fiq->name);
+		seq_printf(p, "%*s:              %s\n", prec, "FIQ",
+			current_fiq->name);
 
 	return 0;
 }
arch/arm/kernel/irq.c +10 -6
index ea29721..4e7a7d2 100644
@@ -57,11 +57,15 @@ int show_interrupts(struct seq_file *p, void *v)
 	struct irq_desc *desc;
 	struct irqaction * action;
 	unsigned long flags;
+	int prec, n;
+
+	for (prec = 3, n = 1000; prec < 10 && n <= nr_irqs; prec++)
+		n *= 10;
 
 	if (i == 0) {
 		char cpuname[12];
 
-		seq_printf(p, "    ");
+		seq_printf(p, "%*s ", prec, "");
 		for_each_present_cpu(cpu) {
 			sprintf(cpuname, "CPU%d", cpu);
 			seq_printf(p, " %10s", cpuname);
@@ -76,7 +80,7 @@ int show_interrupts(struct seq_file *p, void *v)
 		if (!action)
 			goto unlock;
 
-		seq_printf(p, "%3d: ", i);
+		seq_printf(p, "%*d: ", prec, i);
 		for_each_present_cpu(cpu)
 			seq_printf(p, "%10u ", kstat_irqs_cpu(i, cpu));
 		seq_printf(p, " %10s", desc->chip->name ? : "-");
@@ -89,15 +93,15 @@ unlock:
 		raw_spin_unlock_irqrestore(&desc->lock, flags);
 	} else if (i == nr_irqs) {
 #ifdef CONFIG_FIQ
-		show_fiq_list(p, v);
+		show_fiq_list(p, prec);
 #endif
 #ifdef CONFIG_SMP
-		show_ipi_list(p);
+		show_ipi_list(p, prec);
 #endif
 #ifdef CONFIG_LOCAL_TIMERS
-		show_local_irqs(p);
+		show_local_irqs(p, prec);
 #endif
-		seq_printf(p, "Err: %10lu\n", irq_err_count);
+		seq_printf(p, "%*s: %10lu\n", prec, "Err", irq_err_count);
 	}
 	return 0;
 }
arch/arm/kernel/smp.c +7 -7
index edb8fd5..d0e36e3 100644
@@ -396,16 +396,16 @@ void arch_send_call_function_single_ipi(int cpu)
 	send_ipi_message(cpumask_of(cpu), IPI_CALL_FUNC_SINGLE);
 }
 
-void show_ipi_list(struct seq_file *p)
+void show_ipi_list(struct seq_file *p, int prec)
 {
 	unsigned int cpu;
 
-	seq_puts(p, "IPI:");
+	seq_printf(p, "%*s: ", prec, "IPI");
 
 	for_each_present_cpu(cpu)
-		seq_printf(p, " %10u", __get_irq_stat(cpu, ipi_irqs));
+		seq_printf(p, "%10u ", __get_irq_stat(cpu, ipi_irqs));
 
-	seq_putc(p, '\n');
+	seq_printf(p, " Inter-processor interrupts\n");
 }
 
 /*
@@ -435,16 +435,16 @@ asmlinkage void __exception do_local_timer(struct pt_regs *regs)
 	set_irq_regs(old_regs);
 }
 
-void show_local_irqs(struct seq_file *p)
+void show_local_irqs(struct seq_file *p, int prec)
 {
 	unsigned int cpu;
 
-	seq_printf(p, "LOC: ");
+	seq_printf(p, "%*s: ", prec, "LOC");
 
 	for_each_present_cpu(cpu)
 		seq_printf(p, "%10u ", __get_irq_stat(cpu, local_timer_irqs));
 
-	seq_putc(p, '\n');
+	seq_printf(p, " Local timer interrupts\n");
 }
 #endif
 
Signature · 1 line
1.6.2.5
#12 lore raw

PATCH 8/9 ARM: smp: provide individual IPI interrupt statistics

In reply to Russell King - ARM Linux

This separates out the individual IPI interrupt counts from the total IPI count, which allows better visibility of what IPIs are being used for.

---
 arch/arm/include/asm/hardirq.h |    4 +++-
 arch/arm/kernel/irq.c          |    5 +++++
 arch/arm/kernel/smp.c          |   25 +++++++++++++++++++------
 3 files changed, 27 insertions(+), 7 deletions(-)
arch/arm/include/asm/hardirq.h +3 -1
index 824b08a..c52e250 100644
@@ -5,13 +5,15 @@
 #include <linux/threads.h>
 #include <asm/irq.h>
 
+#define NR_IPI	5
+
 typedef struct {
 	unsigned int __softirq_pending;
 #ifdef CONFIG_LOCAL_TIMERS
 	unsigned int local_timer_irqs;
 #endif
 #ifdef CONFIG_SMP
-	unsigned int ipi_irqs;
+	unsigned int ipi_irqs[NR_IPI];
 #endif
 } ____cacheline_aligned irq_cpustat_t;
 
arch/arm/kernel/irq.c +5 -0
index 4e7a7d2..6276f01 100644
@@ -62,6 +62,11 @@ int show_interrupts(struct seq_file *p, void *v)
 	for (prec = 3, n = 1000; prec < 10 && n <= nr_irqs; prec++)
 		n *= 10;
 
+#ifdef CONFIG_SMP
+	if (prec < 4)
+		prec = 4;
+#endif
+
 	if (i == 0) {
 		char cpuname[12];
 
arch/arm/kernel/smp.c +19 -6
index d0e36e3..c088485 100644
@@ -396,16 +396,28 @@ void arch_send_call_function_single_ipi(int cpu)
 	send_ipi_message(cpumask_of(cpu), IPI_CALL_FUNC_SINGLE);
 }
 
+static const char *ipi_types[NR_IPI] = {
+#define S(x,s)	[x - IPI_TIMER] = s
+	S(IPI_TIMER, "Timer broadcast interrupts"),
+	S(IPI_RESCHEDULE, "Rescheduling interrupts"),
+	S(IPI_CALL_FUNC, "Function call interrupts"),
+	S(IPI_CALL_FUNC_SINGLE, "Single function call interrupts"),
+	S(IPI_CPU_STOP, "CPU stop interrupts"),
+};
+
 void show_ipi_list(struct seq_file *p, int prec)
 {
-	unsigned int cpu;
+	unsigned int cpu, i;
 
-	seq_printf(p, "%*s: ", prec, "IPI");
+	for (i = 0; i < NR_IPI; i++) {
+		seq_printf(p, "%*s%u: ", prec - 1, "IPI", i);
 
-	for_each_present_cpu(cpu)
-		seq_printf(p, "%10u ", __get_irq_stat(cpu, ipi_irqs));
+		for_each_present_cpu(cpu)
+			seq_printf(p, "%10u ",
+				   __get_irq_stat(cpu, ipi_irqs[i]));
 
-	seq_printf(p, " Inter-processor interrupts\n");
+		seq_printf(p, " %s\n", ipi_types[i]);
+	}
 }
 
 /*
@@ -520,7 +532,8 @@ asmlinkage void __exception do_IPI(int ipinr, struct pt_regs *regs)
 	unsigned int cpu = smp_processor_id();
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
-	__inc_irq_stat(cpu, ipi_irqs);
+	if (ipinr >= IPI_TIMER && ipinr < IPI_TIMER + NR_IPI)
+		__inc_irq_stat(cpu, ipi_irqs[ipinr - IPI_TIMER]);
 
 	switch (ipinr) {
 	case IPI_TIMER:
Signature · 1 line
1.6.2.5
#13 lore raw

PATCH 9/9 ARM: smp: collect IPI and local timer IRQs for /proc/stat

In reply to Russell King - ARM Linux

The IPI and local timer interrupts weren't being properly accounted for in /proc/stat. Collect them from the irq_stat structure, and return their sum.

---
 arch/arm/include/asm/hardirq.h |    8 ++++++++
 arch/arm/kernel/smp.c          |   15 +++++++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)
arch/arm/include/asm/hardirq.h +8 -0
index c52e250..89ad180 100644
@@ -22,6 +22,14 @@ typedef struct {
 #define __inc_irq_stat(cpu, member)	__IRQ_STAT(cpu, member)++
 #define __get_irq_stat(cpu, member)	__IRQ_STAT(cpu, member)
 
+#ifdef CONFIG_SMP
+u64 smp_irq_stat_cpu(unsigned int cpu);
+#else
+#define smp_irq_stat_cpu(cpu)	0
+#endif
+
+#define arch_irq_stat_cpu	smp_irq_stat_cpu
+
 #if NR_IRQS > 512
 #define HARDIRQ_BITS	10
 #elif NR_IRQS > 256
arch/arm/kernel/smp.c +15 -0
index c088485..5d0dc16 100644
@@ -420,6 +420,21 @@ void show_ipi_list(struct seq_file *p, int prec)
 	}
 }
 
+u64 smp_irq_stat_cpu(unsigned int cpu)
+{
+	u64 sum = 0;
+	int i;
+
+	for (i = 0; i < NR_IPI; i++)
+		sum += __get_irq_stat(cpu, ipi_irqs[i]);
+
+#ifdef CONFIG_LOCAL_TIMERS
+	sum += __get_irq_stat(cpu, local_timer_irqs);
+#endif
+
+	return sum;
+}
+
 /*
  * Timer (local or broadcast) support
  */
Signature · 1 line
1.6.2.5
#14 lore raw

cover Clean up SMP IPI support

In reply to Russell King - ARM Linux
Quoted 8 lines · Russell King - ARM Linux

This series is aimed at cleaning up our SMP IPI support. As has been recently pointed out, we have a spinlock in percpu data, which is not very nice.

If we switch to using the GIC SGIs, one for each IPI function, then we don't need the mask of ipi bits indicating what work needs to be done. This allows us to get rid of the troublesome spinlock in send_ipi_message().

FWIW, the patches look fine to me (though I have tested them):

#15 lore raw

cover Clean up SMP IPI support

Quoted 11 lines · Catalin Marinas
Quoted 8 lines · Russell King - ARM Linux

This series is aimed at cleaning up our SMP IPI support. As has been recently pointed out, we have a spinlock in percpu data, which is not very nice.

If we switch to using the GIC SGIs, one for each IPI function, then we don't need the mask of ipi bits indicating what work needs to be done. This allows us to get rid of the troublesome spinlock in send_ipi_message().

FWIW, the patches look fine to me (though I have tested them):

                                              ^^^^

I meant I have *not* tested them.

Signature · 1 line
Catalin
#16 lore raw

cover Clean up SMP IPI support

Quoted 6 lines · Catalin Marinas
Quoted 5 lines · Catalin Marinas
Quoted 4 lines · Russell King - ARM Linux

This series is aimed at cleaning up our SMP IPI support. As has been recently pointed out, we have a spinlock in percpu data, which is not very nice.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

If we switch to using the GIC SGIs, one for each IPI function, then we don't need the mask of ipi bits indicating what work needs to be done. This allows us to get rid of the troublesome spinlock in send_ipi_message().

FWIW, the patches look fine to me (though I have tested them):

                                              ^^^^

I meant I have *not* tested them.

They work fine on my Versatile Express, but I've not checked the Realview EB yet.