Skip to contentSkip to conversation

nohz: Fix nohz kick irq work on tick v2

PATCH In review Frederic Weisbecker 3 participants 14 messages Sep 10, 2014 – Sep 13, 2014

Subsystems (20)
Arm (32-bit) x86 PowerPC MIPS +16 more subsystems
lore
Share & tools
lore
#0 lore raw

cover nohz: Fix nohz kick irq work on tick v2

Hi,

The nohz kick uses irq work to restart the tick but it relies on irq work to run on its own IPI. Otherwise the kick runs on the tick and bad things happen when it tries to cancel the tick hrtimer from itself.

That 2nd version addresses Peterz review.

Note I only tested x86. Just waiting for general agreement before testing more seriously.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v2

HEAD: 863bea6acd4c9151f9be2fa623e8fee588d4e58c

Thanks,
Frederic
---

Frederic Weisbecker (7):
nohz: Move nohz full init call to tick init
irq_work: Force raised irq work to run on irq work interrupt
x86: Tell irq work about self IPI support
arm: Tell irq work about self IPI support
arm64: Tell irq work about self IPI support
nohz: Consolidate nohz full init code
nohz: nohz full depends on irq work self IPI support

Peter Zijlstra (1):
irq_work: Introduce arch_irq_work_has_interrupt()

 arch/alpha/include/asm/Kbuild      |  1 +
 arch/arc/include/asm/Kbuild        |  1 +
 arch/arm/include/asm/irq_work.h    | 11 ++++++++
 arch/arm/kernel/smp.c              |  2 +-
 arch/arm64/include/asm/Kbuild      |  2 +-
 arch/arm64/include/asm/irq_work.h  | 11 ++++++++
 arch/arm64/include/asm/smp.h       |  2 ++
 arch/arm64/kernel/smp.c            |  2 +-
 arch/avr32/include/asm/Kbuild      |  1 +
 arch/blackfin/include/asm/Kbuild   |  1 +
 arch/c6x/include/asm/Kbuild        |  1 +
 arch/cris/include/asm/Kbuild       |  1 +
 arch/frv/include/asm/Kbuild        |  1 +
 arch/hexagon/include/asm/Kbuild    |  1 +
 arch/ia64/include/asm/Kbuild       |  1 +
 arch/m32r/include/asm/Kbuild       |  1 +
 arch/m68k/include/asm/Kbuild       |  1 +
 arch/metag/include/asm/Kbuild      |  1 +
 arch/microblaze/include/asm/Kbuild |  1 +
 arch/mips/include/asm/Kbuild       |  1 +
 arch/mn10300/include/asm/Kbuild    |  1 +
 arch/openrisc/include/asm/Kbuild   |  1 +
 arch/parisc/include/asm/Kbuild     |  1 +
 arch/powerpc/include/asm/Kbuild    |  1 +
 arch/s390/include/asm/Kbuild       |  1 +
 arch/score/include/asm/Kbuild      |  1 +
 arch/sh/include/asm/Kbuild         |  1 +
 arch/sparc/include/asm/Kbuild      |  1 +
 arch/tile/include/asm/Kbuild       |  1 +
 arch/um/include/asm/Kbuild         |  1 +
 arch/unicore32/include/asm/Kbuild  |  1 +
 arch/x86/include/asm/irq_work.h    | 11 ++++++++
 arch/x86/kernel/irq_work.c         |  2 +-
 arch/xtensa/include/asm/Kbuild     |  1 +
 include/asm-generic/irq_work.h     | 10 ++++++++
 include/linux/irq_work.h           |  3 +++
 include/linux/tick.h               |  2 --
 init/main.c                        |  1 -
 kernel/irq_work.c                  | 13 ++++++++--
 kernel/time/tick-common.c          |  1 +
 kernel/time/tick-internal.h        |  7 +++++
 kernel/time/tick-sched.c           | 52 ++++++++++++++++++++++++--------------
 kernel/time/timer.c                |  2 +-
 43 files changed, 131 insertions(+), 29 deletions(-)
#1 lore raw

PATCH 1/8 nohz: Move nohz full init call to tick init

This way we unbloat a bit main.c and more importantly we initialize nohz full after init_IRQ(). This dependency will be needed in further patches because nohz full needs irq work to raise its own IRQ. Information about the support for this ability on ARM64 is obtained on init_IRQ() which initialize the pointer to __smp_call_function.

Since tick_init() is called right after init_IRQ(), this is a good place to call tick_nohz_init() and prepare for that dependency.

Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/tick.h        | 2 --
 init/main.c                 | 1 -
 kernel/time/tick-common.c   | 1 +
 kernel/time/tick-internal.h | 7 +++++++
 4 files changed, 8 insertions(+), 3 deletions(-)
include/linux/tick.h +0 -2
index 0590523..97fb83e 100644
@@ -181,7 +181,6 @@ static inline bool tick_nohz_full_cpu(int cpu)
 	return cpumask_test_cpu(cpu, tick_nohz_full_mask);
 }
 
-extern void tick_nohz_init(void);
 extern void __tick_nohz_full_check(void);
 extern void tick_nohz_full_kick_cpu(int cpu);
 
@@ -193,7 +192,6 @@ static inline void tick_nohz_full_kick(void)
 extern void tick_nohz_full_kick_all(void);
 extern void __tick_nohz_task_switch(struct task_struct *tsk);
 #else
-static inline void tick_nohz_init(void) { }
 static inline bool tick_nohz_full_enabled(void) { return false; }
 static inline bool tick_nohz_full_cpu(int cpu) { return false; }
 static inline void __tick_nohz_full_check(void) { }
init/main.c +0 -1
index bb1aed9..8af2f1a 100644
@@ -577,7 +577,6 @@ asmlinkage __visible void __init start_kernel(void)
 		local_irq_disable();
 	idr_init_cache();
 	rcu_init();
-	tick_nohz_init();
 	context_tracking_init();
 	radix_tree_init();
 	/* init some links before init_ISA_irqs() */
kernel/time/tick-common.c +1 -0
index 0a0608e..052b4b5 100644
@@ -400,4 +400,5 @@ void tick_resume(void)
 void __init tick_init(void)
 {
 	tick_broadcast_init();
+	tick_nohz_init();
 }
kernel/time/tick-internal.h +7 -0
index c19c1d8..366aeb4 100644
@@ -99,6 +99,13 @@ static inline int tick_broadcast_oneshot_active(void) { return 0; }
 static inline bool tick_broadcast_oneshot_available(void) { return false; }
 #endif /* !TICK_ONESHOT */
 
+/* NO_HZ_FULL internal */
+#ifdef CONFIG_NO_HZ_FULL
+extern void tick_nohz_init(void);
+# else
+static inline void tick_nohz_init(void) { }
+#endif
+
 /*
  * Broadcasting support
  */
Signature · 1 line
2.1.0
#2 lore raw

PATCH 2/8 irq_work: Introduce arch_irq_work_has_interrupt()

In reply to Frederic Weisbecker

From: Peter Zijlstra <[email protected]>

The nohz full code needs irq work to trigger its own interrupt so that the subsystem can work even when the tick is stopped.

Lets introduce arch_irq_work_has_interrupt() that archs can override to tell about their support for this ability.

Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 arch/alpha/include/asm/Kbuild      |  1 +
 arch/arc/include/asm/Kbuild        |  1 +
 arch/arm/include/asm/Kbuild        |  1 +
 arch/arm64/include/asm/Kbuild      |  3 ++-
 arch/avr32/include/asm/Kbuild      |  1 +
 arch/blackfin/include/asm/Kbuild   |  1 +
 arch/c6x/include/asm/Kbuild        |  1 +
 arch/cris/include/asm/Kbuild       |  1 +
 arch/frv/include/asm/Kbuild        |  1 +
 arch/hexagon/include/asm/Kbuild    |  1 +
 arch/ia64/include/asm/Kbuild       |  1 +
 arch/m32r/include/asm/Kbuild       |  1 +
 arch/m68k/include/asm/Kbuild       |  1 +
 arch/metag/include/asm/Kbuild      |  1 +
 arch/microblaze/include/asm/Kbuild |  1 +
 arch/mips/include/asm/Kbuild       |  1 +
 arch/mn10300/include/asm/Kbuild    |  1 +
 arch/openrisc/include/asm/Kbuild   |  1 +
 arch/parisc/include/asm/Kbuild     |  1 +
 arch/powerpc/include/asm/Kbuild    |  1 +
 arch/s390/include/asm/Kbuild       |  1 +
 arch/score/include/asm/Kbuild      |  1 +
 arch/sh/include/asm/Kbuild         |  1 +
 arch/sparc/include/asm/Kbuild      |  1 +
 arch/tile/include/asm/Kbuild       |  1 +
 arch/um/include/asm/Kbuild         |  1 +
 arch/unicore32/include/asm/Kbuild  |  1 +
 arch/x86/include/asm/Kbuild        |  1 +
 arch/xtensa/include/asm/Kbuild     |  1 +
 include/asm-generic/irq_work.h     | 10 ++++++++++
 include/linux/irq_work.h           |  2 ++
 31 files changed, 42 insertions(+), 1 deletion(-)
 create mode 100644 include/asm-generic/irq_work.h
arch/alpha/include/asm/Kbuild +1 -0
index e858aa0..a52cbf1 100644
@@ -4,6 +4,7 @@ generic-y += clkdev.h
 generic-y += cputime.h
 generic-y += exec.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += preempt.h
 generic-y += scatterlist.h
arch/arc/include/asm/Kbuild +1 -0
index e76fd79..b8fffc1 100644
@@ -18,6 +18,7 @@ generic-y += ioctl.h
 generic-y += ioctls.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kmap_types.h
 generic-y += kvm_para.h
 generic-y += local.h
arch/arm/include/asm/Kbuild +1 -0
index 70cd84e..202905e 100644
@@ -11,6 +11,7 @@ generic-y += hash.h
 generic-y += ioctl.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += local.h
 generic-y += local64.h
arch/arm64/include/asm/Kbuild +2 -1
index 0b3fcf8..d617789 100644
@@ -9,8 +9,8 @@ generic-y += current.h
 generic-y += delay.h
 generic-y += div64.h
 generic-y += dma.h
-generic-y += emergency-restart.h
 generic-y += early_ioremap.h
+generic-y += emergency-restart.h
 generic-y += errno.h
 generic-y += ftrace.h
 generic-y += hash.h
@@ -19,6 +19,7 @@ generic-y += ioctl.h
 generic-y += ioctls.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += kvm_para.h
arch/avr32/include/asm/Kbuild +1 -0
index 00a0f3c..2a71b1c 100644
@@ -9,6 +9,7 @@ generic-y += exec.h
 generic-y += futex.h
 generic-y += hash.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
arch/blackfin/include/asm/Kbuild +1 -0
index 0d93b9a..46ed6bb 100644
@@ -15,6 +15,7 @@ generic-y += hw_irq.h
 generic-y += ioctl.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += kvm_para.h
arch/c6x/include/asm/Kbuild +1 -0
index 8dbdce8..e77e0c1 100644
@@ -22,6 +22,7 @@ generic-y += ioctl.h
 generic-y += ioctls.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += local.h
arch/cris/include/asm/Kbuild +1 -0
index 31742df..802b94c4 100644
@@ -8,6 +8,7 @@ generic-y += clkdev.h
 generic-y += cputime.h
 generic-y += exec.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += kvm_para.h
 generic-y += linkage.h
 generic-y += mcs_spinlock.h
arch/frv/include/asm/Kbuild +1 -0
index 5b73921..3caf05c 100644
@@ -3,6 +3,7 @@ generic-y += clkdev.h
 generic-y += cputime.h
 generic-y += exec.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += preempt.h
 generic-y += scatterlist.h
arch/hexagon/include/asm/Kbuild +1 -0
index 0e69796..5f234a5 100644
@@ -23,6 +23,7 @@ generic-y += ioctls.h
 generic-y += iomap.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += local.h
arch/ia64/include/asm/Kbuild +1 -0
index e8317d2..747320b 100644
@@ -2,6 +2,7 @@
 generic-y += clkdev.h
 generic-y += exec.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += kvm_para.h
 generic-y += mcs_spinlock.h
 generic-y += preempt.h
arch/m32r/include/asm/Kbuild +1 -0
index accc10a..e02448b 100644
@@ -3,6 +3,7 @@ generic-y += clkdev.h
 generic-y += cputime.h
 generic-y += exec.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += module.h
 generic-y += preempt.h
arch/m68k/include/asm/Kbuild +1 -0
index c67c94a..dbaf9f3 100644
@@ -11,6 +11,7 @@ generic-y += hw_irq.h
 generic-y += ioctl.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += kvm_para.h
arch/metag/include/asm/Kbuild +1 -0
index c29ead8..7b8111c 100644
@@ -19,6 +19,7 @@ generic-y += ioctl.h
 generic-y += ioctls.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += kvm_para.h
arch/microblaze/include/asm/Kbuild +1 -0
index 27a3acd..448143b 100644
@@ -5,6 +5,7 @@ generic-y += cputime.h
 generic-y += device.h
 generic-y += exec.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += preempt.h
 generic-y += scatterlist.h
arch/mips/include/asm/Kbuild +1 -0
index 335e529..57012ef 100644
@@ -3,6 +3,7 @@ generic-y += cputime.h
 generic-y += current.h
 generic-y += emergency-restart.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
 generic-y += mutex.h
arch/mn10300/include/asm/Kbuild +1 -0
index ecbd667..77eb1a6 100644
@@ -4,6 +4,7 @@ generic-y += clkdev.h
 generic-y += cputime.h
 generic-y += exec.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += preempt.h
 generic-y += scatterlist.h
arch/openrisc/include/asm/Kbuild +1 -0
index 480af0d..89b61d7 100644
@@ -31,6 +31,7 @@ generic-y += ioctl.h
 generic-y += ioctls.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += kvm_para.h
arch/parisc/include/asm/Kbuild +1 -0
index ecf25e6..ffb024b 100644
@@ -10,6 +10,7 @@ generic-y += exec.h
 generic-y += hash.h
 generic-y += hw_irq.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kvm_para.h
 generic-y += local.h
arch/powerpc/include/asm/Kbuild +1 -0
index 7f23f16..31e8f59 100644
@@ -1,6 +1,7 @@
 
 generic-y += clkdev.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += preempt.h
 generic-y += rwsem.h
arch/s390/include/asm/Kbuild +1 -0
index b3fea07..773f866 100644
@@ -2,6 +2,7 @@
 
 generic-y += clkdev.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += preempt.h
 generic-y += scatterlist.h
arch/score/include/asm/Kbuild +1 -0
index aad2091..1909d2a 100644
@@ -6,6 +6,7 @@ generic-y += barrier.h
 generic-y += clkdev.h
 generic-y += cputime.h
 generic-y += hash.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += preempt.h
 generic-y += scatterlist.h
arch/sh/include/asm/Kbuild +1 -0
index c19e47d..5a6c9ac 100644
@@ -12,6 +12,7 @@ generic-y += hash.h
 generic-y += ioctl.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kvm_para.h
 generic-y += local.h
 generic-y += local64.h
arch/sparc/include/asm/Kbuild +1 -0
index cdd1b44..f5f94ce 100644
@@ -8,6 +8,7 @@ generic-y += emergency-restart.h
 generic-y += exec.h
 generic-y += hash.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += linkage.h
 generic-y += local.h
 generic-y += local64.h
arch/tile/include/asm/Kbuild +1 -0
index 0aa5675..e6462b8 100644
@@ -17,6 +17,7 @@ generic-y += ioctl.h
 generic-y += ioctls.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
arch/um/include/asm/Kbuild +1 -0
index 7bd64aa..244b12c 100644
@@ -14,6 +14,7 @@ generic-y += hash.h
 generic-y += hw_irq.h
 generic-y += io.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += mcs_spinlock.h
 generic-y += mutex.h
arch/unicore32/include/asm/Kbuild +1 -0
index 1e5fb87..5a2bb53 100644
@@ -22,6 +22,7 @@ generic-y += ioctl.h
 generic-y += ioctls.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += local.h
arch/x86/include/asm/Kbuild +1 -0
index 3bf000f..8fa909c 100644
@@ -7,5 +7,6 @@ genhdr-y += unistd_x32.h
 generic-y += clkdev.h
 generic-y += cputime.h
 generic-y += early_ioremap.h
+generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += scatterlist.h
arch/xtensa/include/asm/Kbuild +1 -0
index c3d20ba..105d389 100644
@@ -12,6 +12,7 @@ generic-y += hardirq.h
 generic-y += hash.h
 generic-y += ioctl.h
 generic-y += irq_regs.h
+generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += kvm_para.h
include/asm-generic/irq_work.h new file +10 -0
new file mode 100644
index 0000000..a44f452
@@ -0,0 +1,10 @@
+#ifndef __ASM_IRQ_WORK_H
+#define __ASM_IRQ_WORK_H
+
+static inline bool arch_irq_work_has_interrupt(void)
+{
+	return false;
+}
+
+#endif /* __ASM_IRQ_WORK_H */
+
include/linux/irq_work.h +2 -0
index bf9422c..6b47b2e 100644
@@ -42,6 +42,8 @@ void irq_work_run(void);
 void irq_work_sync(struct irq_work *work);
 
 #ifdef CONFIG_IRQ_WORK
+#include <asm/irq_work.h>
+
 bool irq_work_needs_cpu(void);
 #else
 static inline bool irq_work_needs_cpu(void) { return false; }
Signature · 1 line
2.1.0
#3 lore raw

PATCH 3/8 irq_work: Force raised irq work to run on irq work interrupt

In reply to Frederic Weisbecker

The nohz full kick, which restarts the tick when any resource depend on it, can't be executed anywhere given the operation it does on timers. If it is called from the scheduler or timers code, chances are that we run into a deadlock.

This is why we run the nohz full kick from an irq work. That way we make sure that the kick runs on a virgin context.

However if that's the case when irq work runs in its own dedicated self-ipi, things are different for the big bunch of archs that don't support the self triggered way. In order to support them, irq works are also handled by the timer interrupt as fallback.

Now when irq works run on the timer interrupt, the context isn't blank. More precisely, they can run in the context of the hrtimer that runs the tick. But the nohz kick cancels and restarts this hrtimer and cancelling an hrtimer from itself isn't allowed. This is why we run in an endless loop:

	Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
	CPU: 2 PID: 7538 Comm: kworker/u8:8 Not tainted 3.16.0+ #34
	Workqueue: btrfs-endio-write normal_work_helper [btrfs]
	 ffff880244c06c88 000000001b486fe1 ffff880244c06bf0 ffffffff8a7f1e37
	 ffffffff8ac52a18 ffff880244c06c78 ffffffff8a7ef928 0000000000000010
	 ffff880244c06c88 ffff880244c06c20 000000001b486fe1 0000000000000000
	Call Trace:
	 <NMI[<ffffffff8a7f1e37>] dump_stack+0x4e/0x7a
	 [<ffffffff8a7ef928>] panic+0xd4/0x207
	 [<ffffffff8a1450e8>] watchdog_overflow_callback+0x118/0x120
	 [<ffffffff8a186b0e>] __perf_event_overflow+0xae/0x350
	 [<ffffffff8a184f80>] ? perf_event_task_disable+0xa0/0xa0
	 [<ffffffff8a01a4cf>] ? x86_perf_event_set_period+0xbf/0x150
	 [<ffffffff8a187934>] perf_event_overflow+0x14/0x20
	 [<ffffffff8a020386>] intel_pmu_handle_irq+0x206/0x410
	 [<ffffffff8a01937b>] perf_event_nmi_handler+0x2b/0x50
	 [<ffffffff8a007b72>] nmi_handle+0xd2/0x390
	 [<ffffffff8a007aa5>] ? nmi_handle+0x5/0x390
	 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0
	 [<ffffffff8a008062>] default_do_nmi+0x72/0x1c0
	 [<ffffffff8a008268>] do_nmi+0xb8/0x100
	 [<ffffffff8a7ff66a>] end_repeat_nmi+0x1e/0x2e
	 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0
	 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0
	 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0
	 <<EOE><IRQ[<ffffffff8a0ccd2f>] lock_acquired+0xaf/0x450
	 [<ffffffff8a0f74c5>] ? lock_hrtimer_base.isra.20+0x25/0x50
	 [<ffffffff8a7fc678>] _raw_spin_lock_irqsave+0x78/0x90
	 [<ffffffff8a0f74c5>] ? lock_hrtimer_base.isra.20+0x25/0x50
	 [<ffffffff8a0f74c5>] lock_hrtimer_base.isra.20+0x25/0x50
	 [<ffffffff8a0f7723>] hrtimer_try_to_cancel+0x33/0x1e0
	 [<ffffffff8a0f78ea>] hrtimer_cancel+0x1a/0x30
	 [<ffffffff8a109237>] tick_nohz_restart+0x17/0x90
	 [<ffffffff8a10a213>] __tick_nohz_full_check+0xc3/0x100
	 [<ffffffff8a10a25e>] nohz_full_kick_work_func+0xe/0x10
	 [<ffffffff8a17c884>] irq_work_run_list+0x44/0x70
	 [<ffffffff8a17c8da>] irq_work_run+0x2a/0x50
	 [<ffffffff8a0f700b>] update_process_times+0x5b/0x70
	 [<ffffffff8a109005>] tick_sched_handle.isra.21+0x25/0x60
	 [<ffffffff8a109b81>] tick_sched_timer+0x41/0x60
	 [<ffffffff8a0f7aa2>] __run_hrtimer+0x72/0x470
	 [<ffffffff8a109b40>] ? tick_sched_do_timer+0xb0/0xb0
	 [<ffffffff8a0f8707>] hrtimer_interrupt+0x117/0x270
	 [<ffffffff8a034357>] local_apic_timer_interrupt+0x37/0x60
	 [<ffffffff8a80010f>] smp_apic_timer_interrupt+0x3f/0x50
	 [<ffffffff8a7fe52f>] apic_timer_interrupt+0x6f/0x80

To fix this we force non-lazy irq works to run on irq work self-IPIs when available. That ability of the arch to trigger irq work self IPIs is available with arch_irq_work_has_interrupt().

Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/irq_work.h |  1 +
 kernel/irq_work.c        | 13 +++++++++++--
 kernel/time/timer.c      |  2 +-
 3 files changed, 13 insertions(+), 3 deletions(-)
include/linux/irq_work.h +1 -0
index 6b47b2e..bf3fe71 100644
@@ -39,6 +39,7 @@ bool irq_work_queue_on(struct irq_work *work, int cpu);
 #endif
 
 void irq_work_run(void);
+void irq_work_tick(void);
 void irq_work_sync(struct irq_work *work);
 
 #ifdef CONFIG_IRQ_WORK
kernel/irq_work.c +11 -2
index e6bcbe7..353967d 100644
@@ -115,8 +115,10 @@ bool irq_work_needs_cpu(void)
 
 	raised = &__get_cpu_var(raised_list);
 	lazy = &__get_cpu_var(lazy_list);
-	if (llist_empty(raised) && llist_empty(lazy))
-		return false;
+
+	if (llist_empty(raised) || arch_irq_work_has_interrupt())
+		if (llist_empty(lazy))
+			return false;
 
 	/* All work should have been flushed before going offline */
 	WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));
@@ -171,6 +173,13 @@ void irq_work_run(void)
 }
 EXPORT_SYMBOL_GPL(irq_work_run);
 
+void irq_work_tick(void)
+{
+	if (!arch_irq_work_has_interrupt())
+		irq_work_run_list(&__get_cpu_var(raised_list));
+	irq_work_run_list(&__get_cpu_var(lazy_list));
+}
+
 /*
  * Synchronize against the irq_work @entry, ensures the entry is not
  * currently in use.
kernel/time/timer.c +1 -1
index aca5dfe..9bbb834 100644
@@ -1385,7 +1385,7 @@ void update_process_times(int user_tick)
 	rcu_check_callbacks(cpu, user_tick);
 #ifdef CONFIG_IRQ_WORK
 	if (in_irq())
-		irq_work_run();
+		irq_work_tick();
 #endif
 	scheduler_tick();
 	run_posix_cpu_timers(p);
Signature · 1 line
2.1.0
#4 lore raw

PATCH 4/8 x86: Tell irq work about self IPI support

In reply to Frederic Weisbecker

x86 supports irq work self-IPIs when local apic is available. This is partly known on runtime so lets implement arch_irq_work_has_interrupt() accordingly.

This should be safely called after setup_arch().

Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 arch/x86/include/asm/Kbuild     |  1 -
 arch/x86/include/asm/irq_work.h | 11 +++++++++++
 arch/x86/kernel/irq_work.c      |  2 +-
 3 files changed, 12 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/irq_work.h
arch/x86/include/asm/Kbuild +0 -1
index 8fa909c..3bf000f 100644
@@ -7,6 +7,5 @@ genhdr-y += unistd_x32.h
 generic-y += clkdev.h
 generic-y += cputime.h
 generic-y += early_ioremap.h
-generic-y += irq_work.h
 generic-y += mcs_spinlock.h
 generic-y += scatterlist.h
arch/x86/include/asm/irq_work.h new file +11 -0
new file mode 100644
index 0000000..4c01cab
@@ -0,0 +1,11 @@
+#ifndef _ASM_IRQ_WORK_H
+#define _ASM_IRQ_WORK_H
+
+#include <asm/processor.h>
+
+static inline bool arch_irq_work_has_interrupt(void)
+{
+	return static_cpu_has(X86_FEATURE_APIC);
+}
+
+#endif /* _ASM_IRQ_WORK_H */
arch/x86/kernel/irq_work.c +1 -1
index 1de84e3..15d741d 100644
@@ -41,7 +41,7 @@ __visible void smp_trace_irq_work_interrupt(struct pt_regs *regs)
 void arch_irq_work_raise(void)
 {
 #ifdef CONFIG_X86_LOCAL_APIC
-	if (!cpu_has_apic)
+	if (!arch_irq_work_has_interrupt())
 		return;
 
 	apic->send_IPI_self(IRQ_WORK_VECTOR);
Signature · 1 line
2.1.0
#5 lore raw

PATCH 5/8 arm: Tell irq work about self IPI support

In reply to Frederic Weisbecker

ARM irq work IPI support depends on SMP support. That information is partly known at early boottime. Lets implement arch_irq_work_has_interrupt() accordingly.

Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Russell King <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 arch/arm/include/asm/Kbuild     |  1 -
 arch/arm/include/asm/irq_work.h | 11 +++++++++++
 arch/arm/kernel/smp.c           |  2 +-
 3 files changed, 12 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/include/asm/irq_work.h
arch/arm/include/asm/Kbuild +0 -1
index 202905e..70cd84e 100644
@@ -11,7 +11,6 @@ generic-y += hash.h
 generic-y += ioctl.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
-generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += local.h
 generic-y += local64.h
arch/arm/include/asm/irq_work.h new file +11 -0
new file mode 100644
index 0000000..712d03e
@@ -0,0 +1,11 @@
+#ifndef __ASM_ARM_IRQ_WORK_H
+#define __ASM_ARM_IRQ_WORK_H
+
+#include <asm/smp_plat.h>
+
+static inline bool arch_irq_work_has_interrupt(void)
+{
+	return is_smp();
+}
+
+#endif /* _ASM_ARM_IRQ_WORK_H */
arch/arm/kernel/smp.c +1 -1
index 9388a3d..bbe22fc 100644
@@ -503,7 +503,7 @@ void arch_send_call_function_single_ipi(int cpu)
 #ifdef CONFIG_IRQ_WORK
 void arch_irq_work_raise(void)
 {
-	if (is_smp())
+	if (arch_irq_work_has_interrupt())
 		smp_cross_call(cpumask_of(smp_processor_id()), IPI_IRQ_WORK);
 }
 #endif
Signature · 1 line
2.1.0
#6 lore raw

PATCH 6/8 arm64: Tell irq work about self IPI support

In reply to Frederic Weisbecker

ARM64 irq work self-IPI support depends on __smp_cross_call to point to some relevant IRQ controller operations. This information should be available after the call to init_IRQ().

Lets implement arch_irq_work_has_interrupt() accordingly.

Cc Catalin Marinas <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/Kbuild     |  1 -
 arch/arm64/include/asm/irq_work.h | 11 +++++++++++
 arch/arm64/include/asm/smp.h      |  2 ++
 arch/arm64/kernel/smp.c           |  2 +-
 4 files changed, 14 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm64/include/asm/irq_work.h
arch/arm64/include/asm/Kbuild +0 -1
index d617789..c196847 100644
@@ -19,7 +19,6 @@ generic-y += ioctl.h
 generic-y += ioctls.h
 generic-y += ipcbuf.h
 generic-y += irq_regs.h
-generic-y += irq_work.h
 generic-y += kdebug.h
 generic-y += kmap_types.h
 generic-y += kvm_para.h
arch/arm64/include/asm/irq_work.h new file +11 -0
new file mode 100644
index 0000000..8e24ef3
@@ -0,0 +1,11 @@
+#ifndef __ASM_IRQ_WORK_H
+#define __ASM_IRQ_WORK_H
+
+#include <asm/smp.h>
+
+static inline bool arch_irq_work_has_interrupt(void)
+{
+	return !!__smp_cross_call;
+}
+
+#endif /* __ASM_IRQ_WORK_H */
arch/arm64/include/asm/smp.h +2 -0
index a498f2c..780f82c 100644
@@ -48,6 +48,8 @@ extern void smp_init_cpus(void);
  */
 extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
 
+extern void (*__smp_cross_call)(const struct cpumask *, unsigned int);
+
 /*
  * Called from the secondary holding pen, this is the secondary CPU entry point.
  */
arch/arm64/kernel/smp.c +1 -1
index 4743397..b06d1d9 100644
@@ -470,7 +470,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 	}
 }
 
-static void (*__smp_cross_call)(const struct cpumask *, unsigned int);
+void (*__smp_cross_call)(const struct cpumask *, unsigned int);
 
 void __init set_smp_cross_call(void (*fn)(const struct cpumask *, unsigned int))
 {
Signature · 1 line
2.1.0
maintainer Acked-by on v1 6/8
#7 lore raw

Re: [PATCH 6/8] arm64: Tell irq work about self IPI support

Quoted 6 lines · Frederic Weisbecker

ARM64 irq work self-IPI support depends on __smp_cross_call to point to some relevant IRQ controller operations. This information should be available after the call to init_IRQ().

Lets implement arch_irq_work_has_interrupt() accordingly.

Quoted 7 lines
Cc Catalin Marinas <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Will Deacon <[email protected]>

For the arm64 bits:

#8 lore raw

Re: [PATCH 6/8] arm64: Tell irq work about self IPI support

Quoted 15 lines · Catalin Marinas
Quoted 6 lines · Frederic Weisbecker

ARM64 irq work self-IPI support depends on __smp_cross_call to point to some relevant IRQ controller operations. This information should be available after the call to init_IRQ().

Lets implement arch_irq_work_has_interrupt() accordingly.

Quoted 7 lines
Cc Catalin Marinas <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Will Deacon <[email protected]>
Quoted 3 lines

For the arm64 bits:

Thanks a lot!

#9 lore raw

PATCH 7/8 nohz: Consolidate nohz full init code

In reply to Frederic Weisbecker

The supports for CONFIG_NO_HZ_FULL_ALL=y and the nohz_full= kernel parameter both have their own way to do the same thing: allocate full dynticks cpumasks, fill them and initialize some state variables.

Lets consolidate that all in the same place.

While at it, convert some regular printk message to warnings when fundamental allocations fail.

Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/time/tick-sched.c | 38 +++++++++++++++++++-------------------
 1 file changed, 19 insertions(+), 19 deletions(-)
kernel/time/tick-sched.c +19 -19
index 99aa6ee..45cdfbb 100644
@@ -281,22 +281,12 @@ out:
 /* Parse the boot-time nohz CPU list from the kernel parameters. */
 static int __init tick_nohz_full_setup(char *str)
 {
-	int cpu;
-
 	alloc_bootmem_cpumask_var(&tick_nohz_full_mask);
-	alloc_bootmem_cpumask_var(&housekeeping_mask);
 	if (cpulist_parse(str, tick_nohz_full_mask) < 0) {
 		pr_warning("NOHZ: Incorrect nohz_full cpumask\n");
+		free_bootmem_cpumask_var(tick_nohz_full_mask);
 		return 1;
 	}
-
-	cpu = smp_processor_id();
-	if (cpumask_test_cpu(cpu, tick_nohz_full_mask)) {
-		pr_warning("NO_HZ: Clearing %d from nohz_full range for timekeeping\n", cpu);
-		cpumask_clear_cpu(cpu, tick_nohz_full_mask);
-	}
-	cpumask_andnot(housekeeping_mask,
-		       cpu_possible_mask, tick_nohz_full_mask);
 	tick_nohz_full_running = true;
 
 	return 1;
@@ -335,18 +325,11 @@ static int tick_nohz_init_all(void)
 
 #ifdef CONFIG_NO_HZ_FULL_ALL
 	if (!alloc_cpumask_var(&tick_nohz_full_mask, GFP_KERNEL)) {
-		pr_err("NO_HZ: Can't allocate full dynticks cpumask\n");
-		return err;
-	}
-	if (!alloc_cpumask_var(&housekeeping_mask, GFP_KERNEL)) {
-		pr_err("NO_HZ: Can't allocate not-full dynticks cpumask\n");
+		WARN(1, "NO_HZ: Can't allocate full dynticks cpumask\n");
 		return err;
 	}
 	err = 0;
 	cpumask_setall(tick_nohz_full_mask);
-	cpumask_clear_cpu(smp_processor_id(), tick_nohz_full_mask);
-	cpumask_clear(housekeeping_mask);
-	cpumask_set_cpu(smp_processor_id(), housekeeping_mask);
 	tick_nohz_full_running = true;
 #endif
 	return err;
@@ -361,6 +344,23 @@ void __init tick_nohz_init(void)
 			return;
 	}
 
+	if (!alloc_cpumask_var(&housekeeping_mask, GFP_KERNEL)) {
+		WARN(1, "NO_HZ: Can't allocate not-full dynticks cpumask\n");
+		cpumask_clear(tick_nohz_full_mask);
+		tick_nohz_full_running = false;
+		return;
+	}
+
+	cpu = smp_processor_id();
+
+	if (cpumask_test_cpu(cpu, tick_nohz_full_mask)) {
+		pr_warning("NO_HZ: Clearing %d from nohz_full range for timekeeping\n", cpu);
+		cpumask_clear_cpu(cpu, tick_nohz_full_mask);
+	}
+
+	cpumask_andnot(housekeeping_mask,
+		       cpu_possible_mask, tick_nohz_full_mask);
+
 	for_each_cpu(cpu, tick_nohz_full_mask)
 		context_tracking_cpu_set(cpu);
 
Signature · 1 line
2.1.0
#10 lore raw

PATCH 8/8 nohz: nohz full depends on irq work self IPI support

In reply to Frederic Weisbecker

The nohz full functionality depends on IRQ work to trigger its own interrupts. As it's used to restart the tick, we can't rely on the tick fallback for irq work callbacks, ie: we can't use the tick to restart the tick itself.

Lets reject the full dynticks initialization if that arch support isn't available.

As a side effect, this makes sure that nohz kick is never called from the tick. That otherwise would result in illegal hrtimer self-cancellation and lockup.

Cc Ingo Molnar <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/time/tick-sched.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
kernel/time/tick-sched.c +14 -0
index 45cdfbb..d0f0785 100644
@@ -351,6 +351,20 @@ void __init tick_nohz_init(void)
 		return;
 	}
 
+	/*
+	 * Full dynticks uses irq work to drive the tick rescheduling on safe
+	 * locking contexts. But then we need irq work to raise its own
+	 * interrupts to avoid circular dependency on the tick
+	 */
+	if (!arch_irq_work_has_interrupt()) {
+		pr_warning("NO_HZ: Can't run full dynticks because arch doesn't "
+			   "support irq work self-IPIs\n");
+		cpumask_clear(tick_nohz_full_mask);
+		cpumask_copy(housekeeping_mask, cpu_possible_mask);
+		tick_nohz_full_running = false;
+		return;
+	}
+
 	cpu = smp_processor_id();
 
 	if (cpumask_test_cpu(cpu, tick_nohz_full_mask)) {
Signature · 1 line
2.1.0
Acked-by on the v1 series
#11 lore raw
In reply to Frederic Weisbecker
Quoted 11 lines · Frederic Weisbecker

Frederic Weisbecker (7):
nohz: Move nohz full init call to tick init
irq_work: Force raised irq work to run on irq work interrupt
x86: Tell irq work about self IPI support
arm: Tell irq work about self IPI support
arm64: Tell irq work about self IPI support
nohz: Consolidate nohz full init code
nohz: nohz full depends on irq work self IPI support

Peter Zijlstra (1):
irq_work: Introduce arch_irq_work_has_interrupt()

Looks ok to me.

#12 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Frederic Weisbecker

Frederic Weisbecker (7):
nohz: Move nohz full init call to tick init
irq_work: Force raised irq work to run on irq work interrupt
x86: Tell irq work about self IPI support
arm: Tell irq work about self IPI support
arm64: Tell irq work about self IPI support
nohz: Consolidate nohz full init code
nohz: nohz full depends on irq work self IPI support

Quoted 6 lines
Quoted 2 lines

Peter Zijlstra (1):
irq_work: Introduce arch_irq_work_has_interrupt()

Looks ok to me.

Intel? :-) Wow.

Thanks a lot!

Note, I fear I'll have to turn the static_cpu_has into cpu_has. The alternatives code which initialize all static_cpu_has() things is called from check_bugs() in start_kernel() which is way after IRQs are enabled for the 1st time. irq_work_tick() has many opportunities to run in between.

#13 lore raw
Quoted 3 lines · Frederic Weisbecker
Quoted 1 line

Intel? :-) Wow.

Yeah, moved about a month ago :-)

Quoted 5 lines

Note, I fear I'll have to turn the static_cpu_has into cpu_has. The alternatives code which initialize all static_cpu_has() things is called from check_bugs() in start_kernel() which is way after IRQs are enabled for the 1st time. irq_work_tick() has many opportunities to run in between.

Ah well. It was worth a try I suppose.