nohz: Fix nohz kick irq work on tick v2
Subsystems (20)
cover nohz: Fix nohz kick irq work on tick v2
Hi,
The nohz kick uses irq work to restart the tick but it relies on irq work to run on its own IPI. Otherwise the kick runs on the tick and bad things happen when it tries to cancel the tick hrtimer from itself.
That 2nd version addresses Peterz review.
Note I only tested x86. Just waiting for general agreement before testing more seriously.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v2
HEAD: 863bea6acd4c9151f9be2fa623e8fee588d4e58c
Thanks,
Frederic
---
Frederic Weisbecker (7):
nohz: Move nohz full init call to tick init
irq_work: Force raised irq work to run on irq work interrupt
x86: Tell irq work about self IPI support
arm: Tell irq work about self IPI support
arm64: Tell irq work about self IPI support
nohz: Consolidate nohz full init code
nohz: nohz full depends on irq work self IPI support
Peter Zijlstra (1):
irq_work: Introduce arch_irq_work_has_interrupt()
arch/alpha/include/asm/Kbuild | 1 + arch/arc/include/asm/Kbuild | 1 + arch/arm/include/asm/irq_work.h | 11 ++++++++ arch/arm/kernel/smp.c | 2 +- arch/arm64/include/asm/Kbuild | 2 +- arch/arm64/include/asm/irq_work.h | 11 ++++++++ arch/arm64/include/asm/smp.h | 2 ++ arch/arm64/kernel/smp.c | 2 +- arch/avr32/include/asm/Kbuild | 1 + arch/blackfin/include/asm/Kbuild | 1 + arch/c6x/include/asm/Kbuild | 1 + arch/cris/include/asm/Kbuild | 1 + arch/frv/include/asm/Kbuild | 1 + arch/hexagon/include/asm/Kbuild | 1 + arch/ia64/include/asm/Kbuild | 1 + arch/m32r/include/asm/Kbuild | 1 + arch/m68k/include/asm/Kbuild | 1 + arch/metag/include/asm/Kbuild | 1 + arch/microblaze/include/asm/Kbuild | 1 + arch/mips/include/asm/Kbuild | 1 + arch/mn10300/include/asm/Kbuild | 1 + arch/openrisc/include/asm/Kbuild | 1 + arch/parisc/include/asm/Kbuild | 1 + arch/powerpc/include/asm/Kbuild | 1 + arch/s390/include/asm/Kbuild | 1 + arch/score/include/asm/Kbuild | 1 + arch/sh/include/asm/Kbuild | 1 + arch/sparc/include/asm/Kbuild | 1 + arch/tile/include/asm/Kbuild | 1 + arch/um/include/asm/Kbuild | 1 + arch/unicore32/include/asm/Kbuild | 1 + arch/x86/include/asm/irq_work.h | 11 ++++++++ arch/x86/kernel/irq_work.c | 2 +- arch/xtensa/include/asm/Kbuild | 1 + include/asm-generic/irq_work.h | 10 ++++++++ include/linux/irq_work.h | 3 +++ include/linux/tick.h | 2 -- init/main.c | 1 - kernel/irq_work.c | 13 ++++++++-- kernel/time/tick-common.c | 1 + kernel/time/tick-internal.h | 7 +++++ kernel/time/tick-sched.c | 52 ++++++++++++++++++++++++-------------- kernel/time/timer.c | 2 +- 43 files changed, 131 insertions(+), 29 deletions(-)
PATCH 1/8 nohz: Move nohz full init call to tick init
This way we unbloat a bit main.c and more importantly we initialize nohz full after init_IRQ(). This dependency will be needed in further patches because nohz full needs irq work to raise its own IRQ. Information about the support for this ability on ARM64 is obtained on init_IRQ() which initialize the pointer to __smp_call_function.
Since tick_init() is called right after init_IRQ(), this is a good place to call tick_nohz_init() and prepare for that dependency.
--- include/linux/tick.h | 2 -- init/main.c | 1 - kernel/time/tick-common.c | 1 + kernel/time/tick-internal.h | 7 +++++++ 4 files changed, 8 insertions(+), 3 deletions(-)
include/linux/tick.h +0 -2
@@ -181,7 +181,6 @@ static inline bool tick_nohz_full_cpu(int cpu)return cpumask_test_cpu (cpu, tick_nohz_full_mask); } -extern void tick_nohz_init (void );extern void __tick_nohz_full_check (void );extern void tick_nohz_full_kick_cpu (int cpu);
@@ -193,7 +192,6 @@ static inline void tick_nohz_full_kick(void)extern void tick_nohz_full_kick_all (void );extern void __tick_nohz_task_switch (struct task_struct *tsk);#else -static inline void tick_nohz_init (void ) { }static inline bool tick_nohz_full_enabled (void ) {return false; }static inline bool tick_nohz_full_cpu (int cpu) {return false; }static inline void __tick_nohz_full_check (void ) { }
init/main.c +0 -1
@@ -577,7 +577,6 @@ asmlinkage __visible void __init start_kernel(void)local_irq_disable ();idr_init_cache ();rcu_init (); -tick_nohz_init ();context_tracking_init ();radix_tree_init ();/* init some links before init_ISA_irqs() */
kernel/time/tick-common.c +1 -0
@@ -400,4 +400,5 @@ void tick_resume(void)void __init tick_init (void ) {tick_broadcast_init (); +tick_nohz_init (); }
kernel/time/tick-internal.h +7 -0
@@ -99,6 +99,13 @@ static inline int tick_broadcast_oneshot_active(void) { return 0; }static inline bool tick_broadcast_oneshot_available (void ) {return false; }#endif /* !TICK_ONESHOT */ +/* NO_HZ_FULL internal */ +#ifdef CONFIG_NO_HZ_FULL +extern void tick_nohz_init (void ); +# else +static inline void tick_nohz_init (void ) { } +#endif +/* * Broadcasting support */
Signature · 1 line
2.1.0
PATCH 2/8 irq_work: Introduce arch_irq_work_has_interrupt()
In reply to Frederic WeisbeckerFrom: Peter Zijlstra <[email protected]>
The nohz full code needs irq work to trigger its own interrupt so that the subsystem can work even when the tick is stopped.
Lets introduce arch_irq_work_has_interrupt() that archs can override to tell about their support for this ability.
--- arch/alpha/include/asm/Kbuild | 1 + arch/arc/include/asm/Kbuild | 1 + arch/arm/include/asm/Kbuild | 1 + arch/arm64/include/asm/Kbuild | 3 ++- arch/avr32/include/asm/Kbuild | 1 + arch/blackfin/include/asm/Kbuild | 1 + arch/c6x/include/asm/Kbuild | 1 + arch/cris/include/asm/Kbuild | 1 + arch/frv/include/asm/Kbuild | 1 + arch/hexagon/include/asm/Kbuild | 1 + arch/ia64/include/asm/Kbuild | 1 + arch/m32r/include/asm/Kbuild | 1 + arch/m68k/include/asm/Kbuild | 1 + arch/metag/include/asm/Kbuild | 1 + arch/microblaze/include/asm/Kbuild | 1 + arch/mips/include/asm/Kbuild | 1 + arch/mn10300/include/asm/Kbuild | 1 + arch/openrisc/include/asm/Kbuild | 1 + arch/parisc/include/asm/Kbuild | 1 + arch/powerpc/include/asm/Kbuild | 1 + arch/s390/include/asm/Kbuild | 1 + arch/score/include/asm/Kbuild | 1 + arch/sh/include/asm/Kbuild | 1 + arch/sparc/include/asm/Kbuild | 1 + arch/tile/include/asm/Kbuild | 1 + arch/um/include/asm/Kbuild | 1 + arch/unicore32/include/asm/Kbuild | 1 + arch/x86/include/asm/Kbuild | 1 + arch/xtensa/include/asm/Kbuild | 1 + include/asm-generic/irq_work.h | 10 ++++++++++ include/linux/irq_work.h | 2 ++ 31 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 include/asm-generic/irq_work.h
arch/alpha/include/asm/Kbuild +1 -0
@@ -4,6 +4,7 @@ generic-y += clkdev.h generic-y += cputime.h generic-y += exec.h generic-y += hash.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += preempt.h generic-y += scatterlist.h
arch/arc/include/asm/Kbuild +1 -0
@@ -18,6 +18,7 @@ generic-y += ioctl.h generic-y += ioctls.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kmap_types.h generic-y += kvm_para.h generic-y += local.h
arch/arm/include/asm/Kbuild +1 -0
@@ -11,6 +11,7 @@ generic-y += hash.h generic-y += ioctl.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += local.h generic-y += local64.h
arch/arm64/include/asm/Kbuild +2 -1
@@ -9,8 +9,8 @@ generic-y += current.h generic-y += delay.h generic-y += div64.h generic-y += dma.h -generic-y += emergency-restart.h generic-y += early_ioremap.h +generic-y += emergency-restart.h generic-y += errno.h generic-y += ftrace.h generic-y += hash.h
@@ -19,6 +19,7 @@ generic-y += ioctl.h generic-y += ioctls.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += kvm_para.h
arch/avr32/include/asm/Kbuild +1 -0
@@ -9,6 +9,7 @@ generic-y += exec.h generic-y += futex.h generic-y += hash.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += local.h generic-y += local64.h generic-y += mcs_spinlock.h
arch/blackfin/include/asm/Kbuild +1 -0
@@ -15,6 +15,7 @@ generic-y += hw_irq.h generic-y += ioctl.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += kvm_para.h
arch/c6x/include/asm/Kbuild +1 -0
@@ -22,6 +22,7 @@ generic-y += ioctl.h generic-y += ioctls.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += local.h
arch/cris/include/asm/Kbuild +1 -0
@@ -8,6 +8,7 @@ generic-y += clkdev.h generic-y += cputime.h generic-y += exec.h generic-y += hash.h +generic-y += irq_work.h generic-y += kvm_para.h generic-y += linkage.h generic-y += mcs_spinlock.h
arch/frv/include/asm/Kbuild +1 -0
@@ -3,6 +3,7 @@ generic-y += clkdev.h generic-y += cputime.h generic-y += exec.h generic-y += hash.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += preempt.h generic-y += scatterlist.h
arch/hexagon/include/asm/Kbuild +1 -0
@@ -23,6 +23,7 @@ generic-y += ioctls.h generic-y += iomap.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += local.h
arch/ia64/include/asm/Kbuild +1 -0
@@ -2,6 +2,7 @@ generic-y += clkdev.h generic-y += exec.h generic-y += hash.h +generic-y += irq_work.h generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += preempt.h
arch/m32r/include/asm/Kbuild +1 -0
@@ -3,6 +3,7 @@ generic-y += clkdev.h generic-y += cputime.h generic-y += exec.h generic-y += hash.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += module.h generic-y += preempt.h
arch/m68k/include/asm/Kbuild +1 -0
@@ -11,6 +11,7 @@ generic-y += hw_irq.h generic-y += ioctl.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += kvm_para.h
arch/metag/include/asm/Kbuild +1 -0
@@ -19,6 +19,7 @@ generic-y += ioctl.h generic-y += ioctls.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += kvm_para.h
arch/microblaze/include/asm/Kbuild +1 -0
@@ -5,6 +5,7 @@ generic-y += cputime.h generic-y += device.h generic-y += exec.h generic-y += hash.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += preempt.h generic-y += scatterlist.h
arch/mips/include/asm/Kbuild +1 -0
@@ -3,6 +3,7 @@ generic-y += cputime.h generic-y += current.h generic-y += emergency-restart.h generic-y += hash.h +generic-y += irq_work.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += mutex.h
arch/mn10300/include/asm/Kbuild +1 -0
@@ -4,6 +4,7 @@ generic-y += clkdev.h generic-y += cputime.h generic-y += exec.h generic-y += hash.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += preempt.h generic-y += scatterlist.h
arch/openrisc/include/asm/Kbuild +1 -0
@@ -31,6 +31,7 @@ generic-y += ioctl.h generic-y += ioctls.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += kvm_para.h
arch/parisc/include/asm/Kbuild +1 -0
@@ -10,6 +10,7 @@ generic-y += exec.h generic-y += hash.h generic-y += hw_irq.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kvm_para.h generic-y += local.h
arch/powerpc/include/asm/Kbuild +1 -0
@@ -1,6 +1,7 @@ generic-y += clkdev.h generic-y += hash.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += preempt.h generic-y += rwsem.h
arch/s390/include/asm/Kbuild +1 -0
@@ -2,6 +2,7 @@ generic-y += clkdev.h generic-y += hash.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += preempt.h generic-y += scatterlist.h
arch/score/include/asm/Kbuild +1 -0
@@ -6,6 +6,7 @@ generic-y += barrier.h generic-y += clkdev.h generic-y += cputime.h generic-y += hash.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += preempt.h generic-y += scatterlist.h
arch/sh/include/asm/Kbuild +1 -0
@@ -12,6 +12,7 @@ generic-y += hash.h generic-y += ioctl.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kvm_para.h generic-y += local.h generic-y += local64.h
arch/sparc/include/asm/Kbuild +1 -0
@@ -8,6 +8,7 @@ generic-y += emergency-restart.h generic-y += exec.h generic-y += hash.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += linkage.h generic-y += local.h generic-y += local64.h
arch/tile/include/asm/Kbuild +1 -0
@@ -17,6 +17,7 @@ generic-y += ioctl.h generic-y += ioctls.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += local.h generic-y += local64.h generic-y += mcs_spinlock.h
arch/um/include/asm/Kbuild +1 -0
@@ -14,6 +14,7 @@ generic-y += hash.h generic-y += hw_irq.h generic-y += io.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += mcs_spinlock.h generic-y += mutex.h
arch/unicore32/include/asm/Kbuild +1 -0
@@ -22,6 +22,7 @@ generic-y += ioctl.h generic-y += ioctls.h generic-y += ipcbuf.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += local.h
arch/x86/include/asm/Kbuild +1 -0
@@ -7,5 +7,6 @@ genhdr-y += unistd_x32.h generic-y += clkdev.h generic-y += cputime.h generic-y += early_ioremap.h +generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += scatterlist.h
arch/xtensa/include/asm/Kbuild +1 -0
@@ -12,6 +12,7 @@ generic-y += hardirq.h generic-y += hash.h generic-y += ioctl.h generic-y += irq_regs.h +generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += kvm_para.h
include/asm-generic/irq_work.h new file +10 -0
@@ -0,0 +1,10 @@ +#ifndef __ASM_IRQ_WORK_H +#define __ASM_IRQ_WORK_H + +static inline bool arch_irq_work_has_interrupt (void ) +{ +return false; +} + +#endif /* __ASM_IRQ_WORK_H */ +
include/linux/irq_work.h +2 -0
@@ -42,6 +42,8 @@ void irq_work_run(void);void irq_work_sync (struct irq_work *work);#ifdef CONFIG_IRQ_WORK +#include <asm/irq_work.h> +bool irq_work_needs_cpu (void );#else static inline bool irq_work_needs_cpu (void ) {return false; }
Signature · 1 line
2.1.0
PATCH 3/8 irq_work: Force raised irq work to run on irq work interrupt
In reply to Frederic WeisbeckerThe nohz full kick, which restarts the tick when any resource depend on it, can't be executed anywhere given the operation it does on timers. If it is called from the scheduler or timers code, chances are that we run into a deadlock.
This is why we run the nohz full kick from an irq work. That way we make sure that the kick runs on a virgin context.
However if that's the case when irq work runs in its own dedicated self-ipi, things are different for the big bunch of archs that don't support the self triggered way. In order to support them, irq works are also handled by the timer interrupt as fallback.
Now when irq works run on the timer interrupt, the context isn't blank. More precisely, they can run in the context of the hrtimer that runs the tick. But the nohz kick cancels and restarts this hrtimer and cancelling an hrtimer from itself isn't allowed. This is why we run in an endless loop:
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2 CPU: 2 PID: 7538 Comm: kworker/u8:8 Not tainted 3.16.0+ #34 Workqueue: btrfs-endio-write normal_work_helper [btrfs] ffff880244c06c88 000000001b486fe1 ffff880244c06bf0 ffffffff8a7f1e37 ffffffff8ac52a18 ffff880244c06c78 ffffffff8a7ef928 0000000000000010 ffff880244c06c88 ffff880244c06c20 000000001b486fe1 0000000000000000 Call Trace: <NMI[<ffffffff8a7f1e37>] dump_stack+0x4e/0x7a [<ffffffff8a7ef928>] panic+0xd4/0x207 [<ffffffff8a1450e8>] watchdog_overflow_callback+0x118/0x120 [<ffffffff8a186b0e>] __perf_event_overflow+0xae/0x350 [<ffffffff8a184f80>] ? perf_event_task_disable+0xa0/0xa0 [<ffffffff8a01a4cf>] ? x86_perf_event_set_period+0xbf/0x150 [<ffffffff8a187934>] perf_event_overflow+0x14/0x20 [<ffffffff8a020386>] intel_pmu_handle_irq+0x206/0x410 [<ffffffff8a01937b>] perf_event_nmi_handler+0x2b/0x50 [<ffffffff8a007b72>] nmi_handle+0xd2/0x390 [<ffffffff8a007aa5>] ? nmi_handle+0x5/0x390 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0 [<ffffffff8a008062>] default_do_nmi+0x72/0x1c0 [<ffffffff8a008268>] do_nmi+0xb8/0x100 [<ffffffff8a7ff66a>] end_repeat_nmi+0x1e/0x2e [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0 <<EOE><IRQ[<ffffffff8a0ccd2f>] lock_acquired+0xaf/0x450 [<ffffffff8a0f74c5>] ? lock_hrtimer_base.isra.20+0x25/0x50 [<ffffffff8a7fc678>] _raw_spin_lock_irqsave+0x78/0x90 [<ffffffff8a0f74c5>] ? lock_hrtimer_base.isra.20+0x25/0x50 [<ffffffff8a0f74c5>] lock_hrtimer_base.isra.20+0x25/0x50 [<ffffffff8a0f7723>] hrtimer_try_to_cancel+0x33/0x1e0 [<ffffffff8a0f78ea>] hrtimer_cancel+0x1a/0x30 [<ffffffff8a109237>] tick_nohz_restart+0x17/0x90 [<ffffffff8a10a213>] __tick_nohz_full_check+0xc3/0x100 [<ffffffff8a10a25e>] nohz_full_kick_work_func+0xe/0x10 [<ffffffff8a17c884>] irq_work_run_list+0x44/0x70 [<ffffffff8a17c8da>] irq_work_run+0x2a/0x50 [<ffffffff8a0f700b>] update_process_times+0x5b/0x70 [<ffffffff8a109005>] tick_sched_handle.isra.21+0x25/0x60 [<ffffffff8a109b81>] tick_sched_timer+0x41/0x60 [<ffffffff8a0f7aa2>] __run_hrtimer+0x72/0x470 [<ffffffff8a109b40>] ? tick_sched_do_timer+0xb0/0xb0 [<ffffffff8a0f8707>] hrtimer_interrupt+0x117/0x270 [<ffffffff8a034357>] local_apic_timer_interrupt+0x37/0x60 [<ffffffff8a80010f>] smp_apic_timer_interrupt+0x3f/0x50 [<ffffffff8a7fe52f>] apic_timer_interrupt+0x6f/0x80
To fix this we force non-lazy irq works to run on irq work self-IPIs when available. That ability of the arch to trigger irq work self IPIs is available with arch_irq_work_has_interrupt().
--- include/linux/irq_work.h | 1 + kernel/irq_work.c | 13 +++++++++++-- kernel/time/timer.c | 2 +- 3 files changed, 13 insertions(+), 3 deletions(-)
include/linux/irq_work.h +1 -0
@@ -39,6 +39,7 @@ bool irq_work_queue_on(struct irq_work *work, int cpu);#endif void irq_work_run (void ); +void irq_work_tick (void );void irq_work_sync (struct irq_work *work);#ifdef CONFIG_IRQ_WORK
kernel/irq_work.c +11 -2
@@ -115,8 +115,10 @@ bool irq_work_needs_cpu(void) raised = &__get_cpu_var (raised_list); lazy = &__get_cpu_var (lazy_list); -if (llist_empty (raised) &&llist_empty (lazy)) -return false; + +if (llist_empty (raised) ||arch_irq_work_has_interrupt ()) +if (llist_empty (lazy)) +return false;/* All work should have been flushed before going offline */ WARN_ON_ONCE (cpu_is_offline (smp_processor_id ()));
@@ -171,6 +173,13 @@ void irq_work_run(void) }EXPORT_SYMBOL_GPL (irq_work_run); +void irq_work_tick (void ) +{ +if (!arch_irq_work_has_interrupt ()) +irq_work_run_list (&__get_cpu_var (raised_list)); +irq_work_run_list (&__get_cpu_var (lazy_list)); +} +/* * Synchronize against the irq_work @entry, ensures the entry is not * currently in use.
kernel/time/timer.c +1 -1
@@ -1385,7 +1385,7 @@ void update_process_times(int user_tick)rcu_check_callbacks (cpu, user_tick);#ifdef CONFIG_IRQ_WORK if (in_irq ()) -irq_work_run (); +irq_work_tick ();#endif scheduler_tick ();run_posix_cpu_timers (p);
Signature · 1 line
2.1.0
PATCH 4/8 x86: Tell irq work about self IPI support
In reply to Frederic Weisbeckerx86 supports irq work self-IPIs when local apic is available. This is partly known on runtime so lets implement arch_irq_work_has_interrupt() accordingly.
This should be safely called after setup_arch().
--- arch/x86/include/asm/Kbuild | 1 - arch/x86/include/asm/irq_work.h | 11 +++++++++++ arch/x86/kernel/irq_work.c | 2 +- 3 files changed, 12 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/irq_work.h
arch/x86/include/asm/Kbuild +0 -1
@@ -7,6 +7,5 @@ genhdr-y += unistd_x32.h generic-y += clkdev.h generic-y += cputime.h generic-y += early_ioremap.h -generic-y += irq_work.h generic-y += mcs_spinlock.h generic-y += scatterlist.h
arch/x86/include/asm/irq_work.h new file +11 -0
@@ -0,0 +1,11 @@ +#ifndef _ASM_IRQ_WORK_H +#define _ASM_IRQ_WORK_H + +#include <asm/processor.h> + +static inline bool arch_irq_work_has_interrupt (void ) +{ +return static_cpu_has (X86_FEATURE_APIC ); +} + +#endif /* _ASM_IRQ_WORK_H */
arch/x86/kernel/irq_work.c +1 -1
@@ -41,7 +41,7 @@ __visible void smp_trace_irq_work_interrupt(struct pt_regs *regs)void arch_irq_work_raise (void ) {#ifdef CONFIG_X86_LOCAL_APIC -if (!cpu_has_apic) +if (!arch_irq_work_has_interrupt ())return ; apic->send_IPI_self (IRQ_WORK_VECTOR );
Signature · 1 line
2.1.0
PATCH 5/8 arm: Tell irq work about self IPI support
In reply to Frederic WeisbeckerARM irq work IPI support depends on SMP support. That information is partly known at early boottime. Lets implement arch_irq_work_has_interrupt() accordingly.
--- arch/arm/include/asm/Kbuild | 1 - arch/arm/include/asm/irq_work.h | 11 +++++++++++ arch/arm/kernel/smp.c | 2 +- 3 files changed, 12 insertions(+), 2 deletions(-) create mode 100644 arch/arm/include/asm/irq_work.h
arch/arm/include/asm/Kbuild +0 -1
@@ -11,7 +11,6 @@ generic-y += hash.h generic-y += ioctl.h generic-y += ipcbuf.h generic-y += irq_regs.h -generic-y += irq_work.h generic-y += kdebug.h generic-y += local.h generic-y += local64.h
arch/arm/include/asm/irq_work.h new file +11 -0
@@ -0,0 +1,11 @@ +#ifndef __ASM_ARM_IRQ_WORK_H +#define __ASM_ARM_IRQ_WORK_H + +#include <asm/smp_plat.h> + +static inline bool arch_irq_work_has_interrupt (void ) +{ +return is_smp (); +} + +#endif /* _ASM_ARM_IRQ_WORK_H */
arch/arm/kernel/smp.c +1 -1
@@ -503,7 +503,7 @@ void arch_send_call_function_single_ipi(int cpu)#ifdef CONFIG_IRQ_WORK void arch_irq_work_raise (void ) { -if (is_smp ()) +if (arch_irq_work_has_interrupt ())smp_cross_call (cpumask_of (smp_processor_id ()),IPI_IRQ_WORK ); }#endif
Signature · 1 line
2.1.0
PATCH 6/8 arm64: Tell irq work about self IPI support
In reply to Frederic WeisbeckerARM64 irq work self-IPI support depends on __smp_cross_call to point to some relevant IRQ controller operations. This information should be available after the call to init_IRQ().
Lets implement arch_irq_work_has_interrupt() accordingly.
--- arch/arm64/include/asm/Kbuild | 1 - arch/arm64/include/asm/irq_work.h | 11 +++++++++++ arch/arm64/include/asm/smp.h | 2 ++ arch/arm64/kernel/smp.c | 2 +- 4 files changed, 14 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/irq_work.h
arch/arm64/include/asm/Kbuild +0 -1
@@ -19,7 +19,6 @@ generic-y += ioctl.h generic-y += ioctls.h generic-y += ipcbuf.h generic-y += irq_regs.h -generic-y += irq_work.h generic-y += kdebug.h generic-y += kmap_types.h generic-y += kvm_para.h
arch/arm64/include/asm/irq_work.h new file +11 -0
@@ -0,0 +1,11 @@ +#ifndef __ASM_IRQ_WORK_H +#define __ASM_IRQ_WORK_H + +#include <asm/smp.h> + +static inline bool arch_irq_work_has_interrupt (void ) +{ +return !!__smp_cross_call; +} + +#endif /* __ASM_IRQ_WORK_H */
arch/arm64/include/asm/smp.h +2 -0
@@ -48,6 +48,8 @@ extern void smp_init_cpus(void);*/ extern void set_smp_cross_call (void (*)(const struct cpumask *,unsigned int )); +extern void (*__smp_cross_call)(const struct cpumask *,unsigned int ); +/* * Called from the secondary holding pen, this is the secondary CPU entry point. */
arch/arm64/kernel/smp.c +1 -1
@@ -470,7 +470,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus) } } -static void (*__smp_cross_call)(const struct cpumask *,unsigned int ); +void (*__smp_cross_call)(const struct cpumask *,unsigned int );void __init set_smp_cross_call (void (*fn)(const struct cpumask *,unsigned int )) {
Signature · 1 line
2.1.0
Re: [PATCH 6/8] arm64: Tell irq work about self IPI support
Quoted 6 lines · Frederic Weisbecker
ARM64 irq work self-IPI support depends on
__smp_cross_callto point to some relevant IRQ controller operations. This information should be available after the call toinit_IRQ().Lets implement
arch_irq_work_has_interrupt()accordingly.
Quoted 7 lines
Cc Catalin Marinas <[email protected]>Cc Ingo Molnar <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>
For the arm64 bits:
Re: [PATCH 6/8] arm64: Tell irq work about self IPI support
Quoted 15 lines · Catalin Marinas
Quoted 6 lines · Frederic Weisbecker
ARM64 irq work self-IPI support depends on
__smp_cross_callto point to some relevant IRQ controller operations. This information should be available after the call toinit_IRQ().Lets implement
arch_irq_work_has_interrupt()accordingly.Quoted 7 lines
Cc Catalin Marinas <[email protected]>Cc Ingo Molnar <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>
Quoted 3 lines
For the arm64 bits:
Acked-by Catalin Marinas <[email protected]>
Thanks a lot!
PATCH 7/8 nohz: Consolidate nohz full init code
In reply to Frederic WeisbeckerThe supports for CONFIG_NO_HZ_FULL_ALL=y and the nohz_full= kernel parameter both have their own way to do the same thing: allocate full dynticks cpumasks, fill them and initialize some state variables.
Lets consolidate that all in the same place.
While at it, convert some regular printk message to warnings when fundamental allocations fail.
--- kernel/time/tick-sched.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
kernel/time/tick-sched.c +19 -19
@@ -281,22 +281,12 @@ out:/* Parse the boot-time nohz CPU list from the kernel parameters. */ static int __init tick_nohz_full_setup (char *str) { -int cpu; -alloc_bootmem_cpumask_var (&tick_nohz_full_mask); -alloc_bootmem_cpumask_var (&housekeeping_mask);if (cpulist_parse (str, tick_nohz_full_mask) <0 ) {pr_warning ("NOHZ: Incorrect nohz_full cpumask\n" ); +free_bootmem_cpumask_var (tick_nohz_full_mask);return 1 ; } - - cpu =smp_processor_id (); -if (cpumask_test_cpu (cpu, tick_nohz_full_mask)) { -pr_warning ("NO_HZ: Clearing %d from nohz_full range for timekeeping\n" , cpu); -cpumask_clear_cpu (cpu, tick_nohz_full_mask); - } -cpumask_andnot (housekeeping_mask, - cpu_possible_mask, tick_nohz_full_mask); tick_nohz_full_running = true;return 1 ;
@@ -335,18 +325,11 @@ static int tick_nohz_init_all(void)#ifdef CONFIG_NO_HZ_FULL_ALL if (!alloc_cpumask_var (&tick_nohz_full_mask,GFP_KERNEL )) { -pr_err ("NO_HZ: Can't allocate full dynticks cpumask\n" ); -return err; - } -if (!alloc_cpumask_var (&housekeeping_mask,GFP_KERNEL )) { -pr_err ("NO_HZ: Can't allocate not-full dynticks cpumask\n" ); +WARN (1 ,"NO_HZ: Can't allocate full dynticks cpumask\n" );return err; } err =0 ;cpumask_setall (tick_nohz_full_mask); -cpumask_clear_cpu (smp_processor_id (), tick_nohz_full_mask); -cpumask_clear (housekeeping_mask); -cpumask_set_cpu (smp_processor_id (), housekeeping_mask); tick_nohz_full_running = true;#endif return err;
@@ -361,6 +344,23 @@ void __init tick_nohz_init(void)return ; } +if (!alloc_cpumask_var (&housekeeping_mask,GFP_KERNEL )) { +WARN (1 ,"NO_HZ: Can't allocate not-full dynticks cpumask\n" ); +cpumask_clear (tick_nohz_full_mask); + tick_nohz_full_running = false; +return ; + } + + cpu =smp_processor_id (); + +if (cpumask_test_cpu (cpu, tick_nohz_full_mask)) { +pr_warning ("NO_HZ: Clearing %d from nohz_full range for timekeeping\n" , cpu); +cpumask_clear_cpu (cpu, tick_nohz_full_mask); + } + +cpumask_andnot (housekeeping_mask, + cpu_possible_mask, tick_nohz_full_mask); +for_each_cpu (cpu, tick_nohz_full_mask)context_tracking_cpu_set (cpu);
Signature · 1 line
2.1.0
PATCH 8/8 nohz: nohz full depends on irq work self IPI support
In reply to Frederic WeisbeckerThe nohz full functionality depends on IRQ work to trigger its own interrupts. As it's used to restart the tick, we can't rely on the tick fallback for irq work callbacks, ie: we can't use the tick to restart the tick itself.
Lets reject the full dynticks initialization if that arch support isn't available.
As a side effect, this makes sure that nohz kick is never called from the tick. That otherwise would result in illegal hrtimer self-cancellation and lockup.
--- kernel/time/tick-sched.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
kernel/time/tick-sched.c +14 -0
@@ -351,6 +351,20 @@ void __init tick_nohz_init(void)return ; } +/* +* Full dynticks uses irq work to drive the tick rescheduling on safe +* locking contexts. But then we need irq work to raise its own +* interrupts to avoid circular dependency on the tick +*/ +if (!arch_irq_work_has_interrupt ()) { +pr_warning ("NO_HZ: Can't run full dynticks because arch doesn't " +"support irq work self-IPIs\n" ); +cpumask_clear (tick_nohz_full_mask); +cpumask_copy (housekeeping_mask, cpu_possible_mask); + tick_nohz_full_running = false; +return ; + } + cpu =smp_processor_id ();if (cpumask_test_cpu (cpu, tick_nohz_full_mask)) {
Signature · 1 line
2.1.0
Quoted 11 lines · Frederic Weisbecker
Frederic Weisbecker (7):
nohz: Move nohz full init call to tick init
irq_work: Force raised irq work to run on irq work interrupt
x86: Tell irq work about self IPI support
arm: Tell irq work about self IPI support
arm64: Tell irq work about self IPI support
nohz: Consolidate nohz full init code
nohz: nohz full depends on irq work self IPI supportPeter Zijlstra (1):
irq_work: Introducearch_irq_work_has_interrupt()
Looks ok to me.
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Frederic Weisbecker
Frederic Weisbecker (7):
nohz: Move nohz full init call to tick init
irq_work: Force raised irq work to run on irq work interrupt
x86: Tell irq work about self IPI support
arm: Tell irq work about self IPI support
arm64: Tell irq work about self IPI support
nohz: Consolidate nohz full init code
nohz: nohz full depends on irq work self IPI support
Quoted 6 lines
Quoted 2 lines
Peter Zijlstra (1):
irq_work: Introducearch_irq_work_has_interrupt()Looks ok to me.
Intel? :-) Wow.
Thanks a lot!
Note, I fear I'll have to turn the static_cpu_has into cpu_has. The alternatives code which initialize all static_cpu_has() things is called from check_bugs() in start_kernel() which is way after IRQs are enabled for the 1st time. irq_work_tick() has many opportunities to run in between.
Quoted 3 lines · Frederic Weisbecker
Quoted 1 line
Intel? :-) Wow.
Yeah, moved about a month ago :-)
Quoted 5 lines
Note, I fear I'll have to turn the
static_cpu_hasintocpu_has. The alternatives code which initialize allstatic_cpu_has()things is called fromcheck_bugs()instart_kernel()which is way after IRQs are enabled for the 1st time.irq_work_tick()has many opportunities to run in between.
Ah well. It was worth a try I suppose.