printk: Make it usable on nohz CPUs v3
Subsystems (22)
cover printk: Make it usable on nohz CPUs v3
Hi,
Changes since v2:
* Use an irq_work flag instead of a queue parameter to define whether we want an IPI or not. This way it becomes a work property that can't change during its lifecyle. It more simple that way.
* Fix some build errors reported by Wu Fengguang.
* Send an IPI even on non-empty queues, in case it's only filled with lazy works.
* Fix a race when claiming is concurrent with another CPU.
Only boot-tested on x86 for now.
Testing can be made by pulling:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/printk-v3
Thanks.
Frederic Weisbecker (9):
irq_work: Fix racy check on work pending flag
irq_work: Remove CONFIG_HAVE_IRQ_WORK
irq_work: Move irq_work_raise() declaration/default definition to
arch headers
irq_work: Let the arch tell us about self-IPI support
x86: Implement arch_irq_work_has_ipi()
nohz: Add API to check tick state
irq_work: Make self-IPIs optable
irq_work: Handle queuing without IPI support in dyntick idle mode
printk: Wake up klogd using irq_work
arch/alpha/Kconfig | 1 - arch/alpha/include/asm/irq_work.h | 9 +++ arch/alpha/kernel/time.c | 2 +- arch/arm/Kconfig | 1 - arch/arm/include/asm/irq_work.h | 1 + arch/arm64/Kconfig | 1 - arch/arm64/include/asm/irq_work.h | 1 + arch/avr32/include/asm/irq_work.h | 1 + arch/blackfin/Kconfig | 1 - arch/blackfin/include/asm/irq_work.h | 1 + arch/c6x/include/asm/irq_work.h | 1 + arch/cris/include/asm/irq_work.h | 1 + arch/frv/Kconfig | 1 - arch/frv/include/asm/irq_work.h | 1 + arch/h8300/include/asm/irq_work.h | 1 + arch/hexagon/Kconfig | 1 - arch/hexagon/include/asm/irq_work.h | 1 + arch/ia64/include/asm/irq_work.h | 1 + arch/m32r/include/asm/irq_work.h | 1 + arch/m68k/include/asm/irq_work.h | 1 + arch/microblaze/include/asm/irq_work.h | 1 + arch/mips/Kconfig | 1 - arch/mips/include/asm/irq_work.h | 1 + arch/mn10300/include/asm/irq_work.h | 1 + arch/openrisc/include/asm/irq_work.h | 1 + arch/parisc/Kconfig | 1 - arch/parisc/include/asm/irq_work.h | 1 + arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/irq_work.h | 8 +++ arch/powerpc/kernel/time.c | 2 +- arch/s390/Kconfig | 1 - arch/s390/include/asm/irq_work.h | 1 + arch/score/include/asm/irq_work.h | 1 + arch/sh/Kconfig | 1 - arch/sh/include/asm/irq_work.h | 1 + arch/sparc/Kconfig | 1 - arch/sparc/include/asm/irq_work.h | 8 +++ arch/sparc/kernel/pcr.c | 2 +- arch/tile/include/asm/irq_work.h | 1 + arch/um/include/asm/irq_work.h | 1 + arch/unicore32/include/asm/irq_work.h | 1 + arch/x86/Kconfig | 1 - arch/x86/include/asm/irq_work.h | 15 +++++ arch/x86/kernel/irq_work.c | 6 +- arch/xtensa/include/asm/irq_work.h | 1 + drivers/staging/iio/trigger/Kconfig | 1 - include/asm-generic/irq_work.h | 23 ++++++++ include/linux/irq_work.h | 32 ++++++++++++ include/linux/printk.h | 3 - include/linux/tick.h | 17 ++++++- init/Kconfig | 5 +-- kernel/irq_work.c | 88 ++++++++++++++++++++++---------- kernel/printk.c | 12 ++--- kernel/time/tick-sched.c | 7 ++- kernel/timer.c | 1 - 55 files changed, 211 insertions(+), 67 deletions(-) create mode 100644 arch/alpha/include/asm/irq_work.h create mode 100644 arch/arm/include/asm/irq_work.h create mode 100644 arch/arm64/include/asm/irq_work.h create mode 100644 arch/avr32/include/asm/irq_work.h create mode 100644 arch/blackfin/include/asm/irq_work.h create mode 100644 arch/c6x/include/asm/irq_work.h create mode 100644 arch/cris/include/asm/irq_work.h create mode 100644 arch/frv/include/asm/irq_work.h create mode 100644 arch/h8300/include/asm/irq_work.h create mode 100644 arch/hexagon/include/asm/irq_work.h create mode 100644 arch/ia64/include/asm/irq_work.h create mode 100644 arch/m32r/include/asm/irq_work.h create mode 100644 arch/m68k/include/asm/irq_work.h create mode 100644 arch/microblaze/include/asm/irq_work.h create mode 100644 arch/mips/include/asm/irq_work.h create mode 100644 arch/mn10300/include/asm/irq_work.h create mode 100644 arch/openrisc/include/asm/irq_work.h create mode 100644 arch/parisc/include/asm/irq_work.h create mode 100644 arch/powerpc/include/asm/irq_work.h create mode 100644 arch/s390/include/asm/irq_work.h create mode 100644 arch/score/include/asm/irq_work.h create mode 100644 arch/sh/include/asm/irq_work.h create mode 100644 arch/sparc/include/asm/irq_work.h create mode 100644 arch/tile/include/asm/irq_work.h create mode 100644 arch/um/include/asm/irq_work.h create mode 100644 arch/unicore32/include/asm/irq_work.h create mode 100644 arch/x86/include/asm/irq_work.h create mode 100644 arch/xtensa/include/asm/irq_work.h create mode 100644 include/asm-generic/irq_work.h
Signature · 1 line
1.7.5.4
RFC PATCH 1/9 irq_work: Fix racy check on work pending flag
Context requirements on irq work claim are not entirely clear. But it appears that we can try to claim a work that may be already claimed by another CPU.
If so then the early check on IRQ_WORK_PENDING in irq_work_claim() is racy because another CPU may be changing the flags concurrently and we have nothing to synchronize against that. So the value we deal with may be stale for a while already.
To fix this, start with our best wish as the initial value for the work flags and feed cmpxchg with it. But only do the check against IRQ_WORK_PENDING flag with the cmpxchg result.
Nonetheless, if the work is not pending but our best wish was wrong, restart with the old value returned by cmpxchg.
--- kernel/irq_work.c | 17 ++++++++++++----- 1 files changed, 12 insertions(+), 5 deletions(-)
kernel/irq_work.c +12 -5
@@ -34,15 +34,22 @@ static DEFINE_PER_CPU(struct llist_head, irq_work_list);*/ static bool irq_work_claim (struct irq_work *work) { -unsigned long flags, nflags; +unsigned long flags, oflags, nflags; +/* +* Can't check IRQ_WORK_PENDING bit right now because the work +* can be running on another CPU and we are not sync with its +* changes to work flags. Only cmpxchg can reliably check for us. +*/ + flags = work->flags & ~IRQ_WORK_PENDING ;for (;;) { - flags = work->flags; -if (flags &IRQ_WORK_PENDING ) -return false; nflags = flags |IRQ_WORK_FLAGS ; -if (cmpxchg (&work->flags, flags, nflags) == flags) + oflags =cmpxchg (&work->flags, flags, nflags); +if (oflags == flags)break ; +if (oflags &IRQ_WORK_PENDING ) +return false; + flags = oflags;cpu_relax (); }
Signature · 1 line
1.7.5.4
Steven Rostedt
reviewerRe: [RFC PATCH 1/9] irq_work: Fix racy check on work pending flag
Quoted 38 lines · Frederic Weisbecker
Context requirements on irq work claim are not entirely clear. But it appears that we can try to claim a work that may be already claimed by another CPU.
If so then the early check on
IRQ_WORK_PENDINGinirq_work_claim()is racy because another CPU may be changing the flags concurrently and we have nothing to synchronize against that. So the value we deal with may be stale for a while already.To fix this, start with our best wish as the initial value for the work flags and feed cmpxchg with it. But only do the check against
IRQ_WORK_PENDINGflag with the cmpxchg result.Nonetheless, if the work is not pending but our best wish was wrong, restart with the old value returned by cmpxchg.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Andrew Morton <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul Gortmaker <[email protected]>--- kernel/irq_work.c | 17 ++++++++++++----- 1 files changed, 12 insertions(+), 5 deletions(-)kernel/irq_work.c +0
-1@@ -34,15 +34,22 @@ static DEFINE_PER_CPU(struct llist_head, irq_work_list);*/ static bool irq_work_claim (struct irq_work *work) { -unsigned long flags, nflags;
kernel/irq_work.c static DEFINE_PER_CPU(struct llist_head, irq_work_list); · line 40
Quoted 10 lines
kernel/irq_work.c +7
-1+unsigned long flags, oflags, nflags; +/* +* Can't check IRQ_WORK_PENDING bit right now because the work +* can be running on another CPU and we are not sync with its +* changes to work flags. Only cmpxchg can reliably check for us. +*/ + flags = work->flags & ~IRQ_WORK_PENDING ;for (;;) { - flags = work->flags;
I wonder if the bug is just a memory barrier missing here? But that also suggests that the other CPU used a memory barrier too (or cmpxchg() which implies one).
But this change looks fine too.
-- Steve
Quoted 13 lines
kernel/irq_work.c +5
-3-if (flags &IRQ_WORK_PENDING ) -return false; nflags = flags |IRQ_WORK_FLAGS ; -if (cmpxchg (&work->flags, flags, nflags) == flags) + oflags =cmpxchg (&work->flags, flags, nflags); +if (oflags == flags)break ; +if (oflags &IRQ_WORK_PENDING ) +return false; + flags = oflags;cpu_relax (); }
Re: [RFC PATCH 1/9] irq_work: Fix racy check on work pending flag
2012/10/29 Steven Rostedt <[email protected]>:
Quoted 5 lines
I wonder if the bug is just a memory barrier missing here? But that also suggests that the other CPU used a memory barrier too (or
cmpxchg()which implies one).But this change looks fine too.
I fear a memory barrier wouldn't be much helpful here.
Thing is I'm not sure exactly what strict semantics is expected there:
do we allow for concurrent claiming only locally or also across CPUs? Given that we only disable preemption on work queueing, I assumed we allow for concurrent claiming across CPUs. Also we have irq work users that may claim on random CPUs already and thus this can result in cross-CPUs concurrent claiming. Task wide (not per cpu) perf event are such examples. May be drivers/acpi/apei/ghes.c and perhaps others.
Also if we allow for cross-CPUs concurrent claiming, we should set IRQ_WORK_BUSY with cmpxchg() (or xchg() if it's atomic, because nobody can modify it concurrently at this stage as it's claimed). Otherwise a CPU can fail its claim whereas the other CPU that claimed it may actually have executed the work already, or is in the middle of doing so. The race window stops when we clear the IRQ_WORK_BUSY flag with cmpxchg so it's very tight. But still it's there.
Also irq_work_sync() should poll with ACCESS_ONCE(). Same problem with foggy semantics here: do we allow remote sync?
RFC PATCH 2/9 irq_work: Remove CONFIG_HAVE_IRQ_WORK
In reply to Frederic Weisbeckerirq work is supposed to work everywhere because of the irq work hook in the generic timer tick function.
I might be missing something though...
--- arch/alpha/Kconfig | 1 - arch/arm/Kconfig | 1 - arch/arm64/Kconfig | 1 - arch/blackfin/Kconfig | 1 - arch/frv/Kconfig | 1 - arch/hexagon/Kconfig | 1 - arch/mips/Kconfig | 1 - arch/parisc/Kconfig | 1 - arch/powerpc/Kconfig | 1 - arch/s390/Kconfig | 1 - arch/sh/Kconfig | 1 - arch/sparc/Kconfig | 1 - arch/x86/Kconfig | 1 - drivers/staging/iio/trigger/Kconfig | 1 - init/Kconfig | 4 ---- 15 files changed, 0 insertions(+), 18 deletions(-)
arch/alpha/Kconfig +0 -1
@@ -5,7 +5,6 @@ config ALPHA select HAVE_IDE select HAVE_OPROFILE select HAVE_SYSCALL_WRAPPERS - select HAVE_IRQ_WORK select HAVE_PCSPKR_PLATFORM select HAVE_PERF_EVENTS select HAVE_DMA_ATTRS
arch/arm/Kconfig +0 -1
@@ -30,7 +30,6 @@ config ARM select HAVE_KERNEL_LZO select HAVE_KERNEL_LZMA select HAVE_KERNEL_XZ - select HAVE_IRQ_WORK select HAVE_PERF_EVENTS select PERF_USE_VMALLOC select HAVE_REGS_AND_STACK_ACCESS_API
arch/arm64/Kconfig +0 -1
@@ -17,7 +17,6 @@ config ARM64 select HAVE_GENERIC_DMA_COHERENT select HAVE_GENERIC_HARDIRQS select HAVE_HW_BREAKPOINT if PERF_EVENTS - select HAVE_IRQ_WORK select HAVE_MEMBLOCK select HAVE_PERF_EVENTS select HAVE_SPARSE_IRQ
arch/blackfin/Kconfig +0 -1
@@ -24,7 +24,6 @@ config BLACKFIN select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_TRACE_MCOUNT_TEST select HAVE_IDE - select HAVE_IRQ_WORK select HAVE_KERNEL_GZIP if RAMKERNEL select HAVE_KERNEL_BZIP2 if RAMKERNEL select HAVE_KERNEL_LZMA if RAMKERNEL
arch/frv/Kconfig +0 -1
@@ -3,7 +3,6 @@ config FRV default y select HAVE_IDE select HAVE_ARCH_TRACEHOOK - select HAVE_IRQ_WORK select HAVE_PERF_EVENTS select HAVE_UID16 select HAVE_GENERIC_HARDIRQS
arch/hexagon/Kconfig +0 -1
@@ -14,7 +14,6 @@ config HEXAGON # select HAVE_CLK # select IRQ_PER_CPU # select GENERIC_PENDING_IRQ if SMP - select HAVE_IRQ_WORK select GENERIC_ATOMIC64 select HAVE_PERF_EVENTS select HAVE_GENERIC_HARDIRQS
arch/mips/Kconfig +0 -1
@@ -4,7 +4,6 @@ config MIPS select HAVE_GENERIC_DMA_COHERENT select HAVE_IDE select HAVE_OPROFILE - select HAVE_IRQ_WORK select HAVE_PERF_EVENTS select PERF_USE_VMALLOC select HAVE_ARCH_KGDB
arch/parisc/Kconfig +0 -1
@@ -9,7 +9,6 @@ config PARISC select RTC_DRV_GENERIC select INIT_ALL_POSSIBLE select BUG - select HAVE_IRQ_WORK select HAVE_PERF_EVENTS select GENERIC_ATOMIC64 if !64BIT select HAVE_GENERIC_HARDIRQS
arch/powerpc/Kconfig +0 -1
@@ -118,7 +118,6 @@ config PPC select HAVE_SYSCALL_WRAPPERS if PPC64 select GENERIC_ATOMIC64 if PPC32 select ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE - select HAVE_IRQ_WORK select HAVE_PERF_EVENTS select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_HW_BREAKPOINT if PERF_EVENTS && PPC_BOOK3S_64
arch/s390/Kconfig +0 -1
@@ -78,7 +78,6 @@ config S390 select HAVE_KVM if 64BIT select HAVE_ARCH_TRACEHOOK select INIT_ALL_POSSIBLE - select HAVE_IRQ_WORK select HAVE_PERF_EVENTS select ARCH_HAVE_NMI_SAFE_CMPXCHG select HAVE_DEBUG_KMEMLEAK
arch/sh/Kconfig +0 -1
@@ -11,7 +11,6 @@ config SUPERH select HAVE_ARCH_TRACEHOOK select HAVE_DMA_API_DEBUG select HAVE_DMA_ATTRS - select HAVE_IRQ_WORK select HAVE_PERF_EVENTS select HAVE_DEBUG_BUGVERBOSE select ARCH_HAVE_CUSTOM_GPIO_H
arch/sparc/Kconfig +0 -1
@@ -22,7 +22,6 @@ config SPARC select ARCH_WANT_OPTIONAL_GPIOLIB select RTC_CLASS select RTC_DRV_M48T59 - select HAVE_IRQ_WORK select HAVE_DMA_ATTRS select HAVE_DMA_API_DEBUG select HAVE_ARCH_JUMP_LABEL
arch/x86/Kconfig +0 -1
@@ -26,7 +26,6 @@ config X86 select HAVE_OPROFILE select HAVE_PCSPKR_PLATFORM select HAVE_PERF_EVENTS - select HAVE_IRQ_WORK select HAVE_IOREMAP_PROT select HAVE_KPROBES select HAVE_MEMBLOCK
drivers/staging/iio/trigger/Kconfig +0 -1
@@ -21,7 +21,6 @@ config IIO_GPIO_TRIGGER config IIO_SYSFS_TRIGGER tristate "SYSFS trigger" depends on SYSFS - depends on HAVE_IRQ_WORK select IRQ_WORK help Provides support for using SYSFS entry as IIO triggers.
init/Kconfig +0 -4
@@ -20,12 +20,8 @@ config CONSTRUCTORS bool depends on !UML -config HAVE_IRQ_WORK - bool - config IRQ_WORK bool - depends on HAVE_IRQ_WORK config BUILDTIME_EXTABLE_SORT bool
Signature · 1 line
1.7.5.4
RFC PATCH 3/9 irq_work: Move irq_work_raise() declaration/default definition to arch headers
In reply to Frederic WeisbeckerThis optimization doesn't matter much. But this prepares the arch headers that we need to add a new API in order to detect when the arch can trigger self IPIs to implement the irq work.
This is necessary later to make printk working in nohz CPUs.
--- arch/alpha/include/asm/irq_work.h | 6 ++++++ arch/arm/include/asm/irq_work.h | 1 + arch/arm64/include/asm/irq_work.h | 1 + arch/avr32/include/asm/irq_work.h | 1 + arch/blackfin/include/asm/irq_work.h | 1 + arch/c6x/include/asm/irq_work.h | 1 + arch/cris/include/asm/irq_work.h | 1 + arch/frv/include/asm/irq_work.h | 1 + arch/h8300/include/asm/irq_work.h | 1 + arch/hexagon/include/asm/irq_work.h | 1 + arch/ia64/include/asm/irq_work.h | 1 + arch/m32r/include/asm/irq_work.h | 1 + arch/m68k/include/asm/irq_work.h | 1 + arch/microblaze/include/asm/irq_work.h | 1 + arch/mips/include/asm/irq_work.h | 1 + arch/mn10300/include/asm/irq_work.h | 1 + arch/openrisc/include/asm/irq_work.h | 1 + arch/parisc/include/asm/irq_work.h | 1 + arch/powerpc/include/asm/irq_work.h | 6 ++++++ arch/s390/include/asm/irq_work.h | 1 + arch/score/include/asm/irq_work.h | 1 + arch/sh/include/asm/irq_work.h | 1 + arch/sparc/include/asm/irq_work.h | 6 ++++++ arch/tile/include/asm/irq_work.h | 1 + arch/um/include/asm/irq_work.h | 1 + arch/unicore32/include/asm/irq_work.h | 1 + arch/x86/include/asm/irq_work.h | 10 ++++++++++ arch/x86/kernel/irq_work.c | 4 ++-- arch/xtensa/include/asm/irq_work.h | 1 + include/asm-generic/irq_work.h | 9 +++++++++ include/linux/irq_work.h | 1 + kernel/irq_work.c | 7 ------- 32 files changed, 64 insertions(+), 9 deletions(-) create mode 100644 arch/alpha/include/asm/irq_work.h create mode 100644 arch/arm/include/asm/irq_work.h create mode 100644 arch/arm64/include/asm/irq_work.h create mode 100644 arch/avr32/include/asm/irq_work.h create mode 100644 arch/blackfin/include/asm/irq_work.h create mode 100644 arch/c6x/include/asm/irq_work.h create mode 100644 arch/cris/include/asm/irq_work.h create mode 100644 arch/frv/include/asm/irq_work.h create mode 100644 arch/h8300/include/asm/irq_work.h create mode 100644 arch/hexagon/include/asm/irq_work.h create mode 100644 arch/ia64/include/asm/irq_work.h create mode 100644 arch/m32r/include/asm/irq_work.h create mode 100644 arch/m68k/include/asm/irq_work.h create mode 100644 arch/microblaze/include/asm/irq_work.h create mode 100644 arch/mips/include/asm/irq_work.h create mode 100644 arch/mn10300/include/asm/irq_work.h create mode 100644 arch/openrisc/include/asm/irq_work.h create mode 100644 arch/parisc/include/asm/irq_work.h create mode 100644 arch/powerpc/include/asm/irq_work.h create mode 100644 arch/s390/include/asm/irq_work.h create mode 100644 arch/score/include/asm/irq_work.h create mode 100644 arch/sh/include/asm/irq_work.h create mode 100644 arch/sparc/include/asm/irq_work.h create mode 100644 arch/tile/include/asm/irq_work.h create mode 100644 arch/um/include/asm/irq_work.h create mode 100644 arch/unicore32/include/asm/irq_work.h create mode 100644 arch/x86/include/asm/irq_work.h create mode 100644 arch/xtensa/include/asm/irq_work.h create mode 100644 include/asm-generic/irq_work.h
arch/alpha/include/asm/irq_work.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _ALPHA_IRQ_WORK_H +#define _ALPHA_IRQ_WORK_H + +extern void arch_irq_work_raise (void ); + +#endif
arch/arm/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/arm64/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/avr32/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/blackfin/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/c6x/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/cris/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/frv/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/h8300/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/hexagon/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/ia64/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/m32r/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/m68k/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/microblaze/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/mips/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/mn10300/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/openrisc/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/parisc/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/powerpc/include/asm/irq_work.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _ASM_POWERPC_IRQ_WORK_H +#define _ASM_POWERPC_IRQ_WORK_H + +extern void arch_irq_work_raise (void ); + +#endif
arch/s390/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/score/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/sh/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/sparc/include/asm/irq_work.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef ___ASM_SPARC_IRQ_H +#define ___ASM_SPARC_IRQ_H + +extern void arch_irq_work_raise (void ); + +#endif
arch/tile/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/um/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/unicore32/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
arch/x86/include/asm/irq_work.h new file +10 -0
@@ -0,0 +1,10 @@ +#ifndef _ASM_X86_IRQ_WORK_H +#define _ASM_X86_IRQ_WORK_H + +#ifndef CONFIG_X86_LOCAL_APIC +#include <asm-generic/irq_work.h> +# else +extern void arch_irq_work_raise (void ); +#endif + +#endif
arch/x86/kernel/irq_work.c +2 -2
@@ -18,13 +18,13 @@ void smp_irq_work_interrupt(struct pt_regs *regs)irq_exit (); } +#ifdef CONFIG_X86_LOCAL_APIC void arch_irq_work_raise (void ) { -#ifdef CONFIG_X86_LOCAL_APIC if (!cpu_has_apic)return ; apic->send_IPI_self (IRQ_WORK_VECTOR );apic_wait_icr_idle (); -#endif } +#endif
arch/xtensa/include/asm/irq_work.h new file +1 -0
@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
include/asm-generic/irq_work.h new file +9 -0
@@ -0,0 +1,9 @@ +#ifndef __ASM_GENERIC_ATOMIC_H +#define __ASM_GENERIC_ATOMIC_H + +/* +* Lame architectures will get the timer tick callback +*/ +static inline void arch_irq_work_raise (void ) { } + +#endif
include/linux/irq_work.h +1 -0
@@ -2,6 +2,7 @@#define _LINUX_IRQ_WORK_H#include <linux/llist.h> +#include <asm/irq_work.h> struct irq_work {unsigned long flags;
kernel/irq_work.c +0 -7
@@ -56,13 +56,6 @@ static bool irq_work_claim(struct irq_work *work)return true; } -void __weak arch_irq_work_raise (void ) -{ -/* -* Lame architectures will get the timer tick callback -*/ -} -/* * Queue the entry and raise the IPI if needed. */
Signature · 1 line
1.7.5.4
Steven Rostedt
reviewerRe: [RFC PATCH 3/9] irq_work: Move irq_work_raise() declaration/default definition to arch headers
Quoted 95 lines · Frederic Weisbecker
This optimization doesn't matter much. But this prepares the arch headers that we need to add a new API in order to detect when the arch can trigger self IPIs to implement the irq work.
This is necessary later to make printk working in nohz CPUs.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Andrew Morton <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul Gortmaker <[email protected]>--- arch/alpha/include/asm/irq_work.h | 6 ++++++ arch/arm/include/asm/irq_work.h | 1 + arch/arm64/include/asm/irq_work.h | 1 + arch/avr32/include/asm/irq_work.h | 1 + arch/blackfin/include/asm/irq_work.h | 1 + arch/c6x/include/asm/irq_work.h | 1 + arch/cris/include/asm/irq_work.h | 1 + arch/frv/include/asm/irq_work.h | 1 + arch/h8300/include/asm/irq_work.h | 1 + arch/hexagon/include/asm/irq_work.h | 1 + arch/ia64/include/asm/irq_work.h | 1 + arch/m32r/include/asm/irq_work.h | 1 + arch/m68k/include/asm/irq_work.h | 1 + arch/microblaze/include/asm/irq_work.h | 1 + arch/mips/include/asm/irq_work.h | 1 + arch/mn10300/include/asm/irq_work.h | 1 + arch/openrisc/include/asm/irq_work.h | 1 + arch/parisc/include/asm/irq_work.h | 1 + arch/powerpc/include/asm/irq_work.h | 6 ++++++ arch/s390/include/asm/irq_work.h | 1 + arch/score/include/asm/irq_work.h | 1 + arch/sh/include/asm/irq_work.h | 1 + arch/sparc/include/asm/irq_work.h | 6 ++++++ arch/tile/include/asm/irq_work.h | 1 + arch/um/include/asm/irq_work.h | 1 + arch/unicore32/include/asm/irq_work.h | 1 + arch/x86/include/asm/irq_work.h | 10 ++++++++++ arch/x86/kernel/irq_work.c | 4 ++-- arch/xtensa/include/asm/irq_work.h | 1 + include/asm-generic/irq_work.h | 9 +++++++++ include/linux/irq_work.h | 1 + kernel/irq_work.c | 7 ------- 32 files changed, 64 insertions(+), 9 deletions(-) create mode 100644 arch/alpha/include/asm/irq_work.h create mode 100644 arch/arm/include/asm/irq_work.h create mode 100644 arch/arm64/include/asm/irq_work.h create mode 100644 arch/avr32/include/asm/irq_work.h create mode 100644 arch/blackfin/include/asm/irq_work.h create mode 100644 arch/c6x/include/asm/irq_work.h create mode 100644 arch/cris/include/asm/irq_work.h create mode 100644 arch/frv/include/asm/irq_work.h create mode 100644 arch/h8300/include/asm/irq_work.h create mode 100644 arch/hexagon/include/asm/irq_work.h create mode 100644 arch/ia64/include/asm/irq_work.h create mode 100644 arch/m32r/include/asm/irq_work.h create mode 100644 arch/m68k/include/asm/irq_work.h create mode 100644 arch/microblaze/include/asm/irq_work.h create mode 100644 arch/mips/include/asm/irq_work.h create mode 100644 arch/mn10300/include/asm/irq_work.h create mode 100644 arch/openrisc/include/asm/irq_work.h create mode 100644 arch/parisc/include/asm/irq_work.h create mode 100644 arch/powerpc/include/asm/irq_work.h create mode 100644 arch/s390/include/asm/irq_work.h create mode 100644 arch/score/include/asm/irq_work.h create mode 100644 arch/sh/include/asm/irq_work.h create mode 100644 arch/sparc/include/asm/irq_work.h create mode 100644 arch/tile/include/asm/irq_work.h create mode 100644 arch/um/include/asm/irq_work.h create mode 100644 arch/unicore32/include/asm/irq_work.h create mode 100644 arch/x86/include/asm/irq_work.h create mode 100644 arch/xtensa/include/asm/irq_work.h create mode 100644 include/asm-generic/irq_work.harch/alpha/include/asm/irq_work.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef _ALPHA_IRQ_WORK_H +#define _ALPHA_IRQ_WORK_H + +extern void arch_irq_work_raise (void ); + +#endif arch/arm/include/asm/irq_work.h new file +0
-0
arch/arm/include/asm/irq_work.h line 1
Quoted 1 line
arch/arm/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h>
Don't add stub files. We don't do that anymore.
The correct way of doing this is to modify:
arch/*/include/asm/Kbuild
and add:
generic-y += irq_work.h
-- Steve
Quoted 266 lines
arch/arm64/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/avr32/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/blackfin/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/c6x/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/cris/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/frv/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/h8300/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/hexagon/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/ia64/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/m32r/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/m68k/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/microblaze/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/mips/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/mn10300/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/openrisc/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/parisc/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/powerpc/include/asm/irq_work.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef _ASM_POWERPC_IRQ_WORK_H +#define _ASM_POWERPC_IRQ_WORK_H + +extern void arch_irq_work_raise (void ); + +#endif arch/s390/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/score/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/sh/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/sparc/include/asm/irq_work.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef ___ASM_SPARC_IRQ_H +#define ___ASM_SPARC_IRQ_H + +extern void arch_irq_work_raise (void ); + +#endif arch/tile/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/um/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/unicore32/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> arch/x86/include/asm/irq_work.h new file +10
-0@@ -0,0 +1,10 @@ +#ifndef _ASM_X86_IRQ_WORK_H +#define _ASM_X86_IRQ_WORK_H + +#ifndef CONFIG_X86_LOCAL_APIC +#include <asm-generic/irq_work.h> +# else +extern void arch_irq_work_raise (void ); +#endif + +#endif arch/x86/kernel/irq_work.c +2
-2@@ -18,13 +18,13 @@ void smp_irq_work_interrupt(struct pt_regs *regs)irq_exit (); } +#ifdef CONFIG_X86_LOCAL_APIC void arch_irq_work_raise (void ) { -#ifdef CONFIG_X86_LOCAL_APIC if (!cpu_has_apic)return ; apic->send_IPI_self (IRQ_WORK_VECTOR );apic_wait_icr_idle (); -#endif } +#endif arch/xtensa/include/asm/irq_work.h new file +1
-0@@ -0,0 +1 @@ +#include <asm-generic/irq_work.h> include/asm-generic/irq_work.h new file +9
-0@@ -0,0 +1,9 @@ +#ifndef __ASM_GENERIC_ATOMIC_H +#define __ASM_GENERIC_ATOMIC_H + +/* +* Lame architectures will get the timer tick callback +*/ +static inline void arch_irq_work_raise (void ) { } + +#endif include/linux/irq_work.h +1
-0@@ -2,6 +2,7 @@#define _LINUX_IRQ_WORK_H#include <linux/llist.h> +#include <asm/irq_work.h> struct irq_work {unsigned long flags;kernel/irq_work.c +0
-7@@ -56,13 +56,6 @@ static bool irq_work_claim(struct irq_work *work)return true; } -void __weak arch_irq_work_raise (void ) -{ -/* -* Lame architectures will get the timer tick callback -*/ -} -/* * Queue the entry and raise the IPI if needed. */
Re: [RFC PATCH 3/9] irq_work: Move irq_work_raise() declaration/default definition to arch headers
2012/10/29 Steven Rostedt <[email protected]>:
Quoted 9 lines
Don't add stub files. We don't do that anymore.
The correct way of doing this is to modify:
arch/*/include/asm/Kbuildand add:
generic-y +=
irq_work.h
Ah, ok!
RFC PATCH 4/9 irq_work: Let the arch tell us about self-IPI support
In reply to Frederic WeisbeckerThis prepares us to make printk working on nohz CPUs using irq work.
--- arch/alpha/include/asm/irq_work.h | 5 ++++- arch/alpha/kernel/time.c | 2 +- arch/powerpc/include/asm/irq_work.h | 4 +++- arch/powerpc/kernel/time.c | 2 +- arch/sparc/include/asm/irq_work.h | 4 +++- arch/sparc/kernel/pcr.c | 2 +- arch/x86/include/asm/irq_work.h | 9 +++++---- arch/x86/kernel/irq_work.c | 2 +- include/asm-generic/irq_work.h | 14 ++++++++++++++ 9 files changed, 33 insertions(+), 11 deletions(-)
arch/alpha/include/asm/irq_work.h +4 -1
@@ -1,6 +1,9 @@#ifndef _ALPHA_IRQ_WORK_H#define _ALPHA_IRQ_WORK_H -extern void arch_irq_work_raise (void ); +extern void __arch_irq_work_raise (void ); +#define arch_irq_work_raise __arch_irq_work_raise + +#include <asm-generic/irq_work.h> #endif
arch/alpha/kernel/time.c +1 -1
@@ -90,7 +90,7 @@ DEFINE_PER_CPU(u8, irq_work_pending);#define test_irq_work_pending () __get_cpu_var(irq_work_pending)#define clear_irq_work_pending () __get_cpu_var(irq_work_pending) = 0 -void arch_irq_work_raise (void ) +void __arch_irq_work_raise (void ) {set_irq_work_pending_flag (); }
arch/powerpc/include/asm/irq_work.h +3 -1
@@ -1,6 +1,8 @@#ifndef _ASM_POWERPC_IRQ_WORK_H#define _ASM_POWERPC_IRQ_WORK_H -extern void arch_irq_work_raise (void ); +extern void __arch_irq_work_raise (void ); +#define arch_irq_work_raise __arch_irq_work_raise +#include <asm-generic/irq_work.h> #endif
arch/powerpc/kernel/time.c +1 -1
@@ -466,7 +466,7 @@ DEFINE_PER_CPU(u8, irq_work_pending);#endif /* 32 vs 64 bit */ -void arch_irq_work_raise (void ) +void __arch_irq_work_raise (void ) {preempt_disable ();set_irq_work_pending_flag ();
arch/sparc/include/asm/irq_work.h +3 -1
@@ -1,6 +1,8 @@#ifndef ___ASM_SPARC_IRQ_H#define ___ASM_SPARC_IRQ_H -extern void arch_irq_work_raise (void ); +extern void __arch_irq_work_raise (void ); +#define arch_irq_work_raise __arch_irq_work_raise +#include <asm-generic/irq_work.h> #endif
arch/sparc/kernel/pcr.c +1 -1
@@ -43,7 +43,7 @@ void __irq_entry deferred_pcr_work_irq(int irq, struct pt_regs *regs)set_irq_regs (old_regs); } -void arch_irq_work_raise (void ) +void __arch_irq_work_raise (void ) {set_softint (1 <<PIL_DEFERRED_PCR_WORK ); }
arch/x86/include/asm/irq_work.h +5 -4
@@ -1,10 +1,11 @@#ifndef _ASM_X86_IRQ_WORK_H#define _ASM_X86_IRQ_WORK_H -#ifndef CONFIG_X86_LOCAL_APIC -#include <asm-generic/irq_work.h> -# else -extern void arch_irq_work_raise (void ); +#ifdef CONFIG_X86_LOCAL_APIC +extern void __arch_irq_work_raise (void ); +#define arch_irq_work_raise __arch_irq_work_raise#endif +#include <asm-generic/irq_work.h> +#endif
arch/x86/kernel/irq_work.c +1 -1
@@ -19,7 +19,7 @@ void smp_irq_work_interrupt(struct pt_regs *regs) }#ifdef CONFIG_X86_LOCAL_APIC -void arch_irq_work_raise (void ) +void __arch_irq_work_raise (void ) {if (!cpu_has_apic)return ;
include/asm-generic/irq_work.h +14 -0
@@ -4,6 +4,20 @@/* * Lame architectures will get the timer tick callback */ +#ifndef arch_irq_work_raisestatic inline void arch_irq_work_raise (void ) { } +#endif + +/* +* Unless told otherwise, consider the arch doesn't implement irq work +* using self IPIs but through another way like defaulting to the hook +* on the sched tick. +*/ +#ifndef arch_irq_work_has_ipi +static inline bool arch_irq_work_has_ipi (void ) +{ +return false; +} +#endif #endif
Signature · 1 line
1.7.5.4
Steven Rostedt
reviewerRe: [RFC PATCH 4/9] irq_work: Let the arch tell us about self-IPI support
Quoted 2 lines · Frederic Weisbecker
This prepares us to make printk working on nohz CPUs using irq work.
-ENOTENOUGHINFO
Please state how this prepares printk for nohz CPUS using irq_work.
Thanks,
-- Steve
Quoted 9 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Andrew Morton <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul Gortmaker <[email protected]>---
Re: [RFC PATCH 4/9] irq_work: Let the arch tell us about self-IPI support
2012/10/29 Steven Rostedt <[email protected]>:
Quoted 7 lines · Frederic Weisbecker
Quoted 2 lines
This prepares us to make printk working on nohz CPUs using irq work.
-ENOTENOUGHINFOPlease state how this prepares printk for nohz CPUS using
irq_work.
RIght, I'll add the details in the next version.
RFC PATCH 5/9 x86: Implement arch_irq_work_has_ipi()
In reply to Frederic WeisbeckerMost of the time, x86 can trigger self-IPIs. Tell
irq work subsystem about it.
--- arch/x86/include/asm/irq_work.h | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)
arch/x86/include/asm/irq_work.h +4 -0
@@ -2,8 +2,12 @@#define _ASM_X86_IRQ_WORK_H#ifdef CONFIG_X86_LOCAL_APIC +#include <asm/cpufeature.h> +extern void __arch_irq_work_raise (void );#define arch_irq_work_raise __arch_irq_work_raise + +#define arch_irq_work_has_ipi () (cpu_has_apic)#endif #include <asm-generic/irq_work.h>
Signature · 1 line
1.7.5.4
RFC PATCH 6/9 nohz: Add API to check tick state
In reply to Frederic WeisbeckerWe need some quick way to check if the CPU has stopped its tick. This will be useful to implement the printk tick using the irq work subsystem.
--- include/linux/tick.h | 17 ++++++++++++++++- kernel/time/tick-sched.c | 2 +- 2 files changed, 17 insertions(+), 2 deletions(-)
include/linux/tick.h +16 -1
@@ -8,6 +8,8 @@#include <linux/clockchips.h> #include <linux/irqflags.h> +#include <linux/percpu.h> +#include <linux/hrtimer.h> #ifdef CONFIG_GENERIC_CLOCKEVENTS
@@ -122,13 +124,26 @@ static inline int tick_oneshot_mode_active(void) { return 0; }#endif /* !CONFIG_GENERIC_CLOCKEVENTS */ # ifdef CONFIG_NO_HZ +DECLARE_PER_CPU(struct tick_sched, tick_cpu_sched); + +static inline int tick_nohz_tick_stopped (void ) +{ +return __this_cpu_read (tick_cpu_sched.tick_stopped); +} +extern void tick_nohz_idle_enter (void );extern void tick_nohz_idle_exit (void );extern void tick_nohz_irq_exit (void );extern ktime_t tick_nohz_get_sleep_length (void );extern u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time);extern u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time); -# else + +# else /* !CONFIG_NO_HZ */ +static inline int tick_nohz_tick_stopped (void ) +{ +return 0 ; +} +static inline void tick_nohz_idle_enter (void ) { }static inline void tick_nohz_idle_exit (void ) { }
kernel/time/tick-sched.c +1 -1
@@ -28,7 +28,7 @@/* * Per cpu nohz control structure */ -static DEFINE_PER_CPU(struct tick_sched, tick_cpu_sched); +DEFINE_PER_CPU(struct tick_sched, tick_cpu_sched);/* * The time, when the last jiffy update happened. Protected by xtime_lock.
Signature · 1 line
1.7.5.4
RFC PATCH 7/9 irq_work: Make self-IPIs optable
In reply to Frederic WeisbeckerOn irq work initialization, let the user choose to define it as "lazy" or not. "Lazy" means that we don't want to send an IPI (provided the arch can anyway) when we enqueue this work but we rather prefer to wait for the next timer tick to execute our work if possible.
This is going to be a benefit for non-urgent enqueuers (like printk in the future) that may prefer not to raise an IPI storm in case of frequent enqueuing on short periods of time.
--- include/linux/irq_work.h | 31 ++++++++++++++++++++++++++ kernel/irq_work.c | 53 ++++++++++++++++++++++++++++++++------------- kernel/time/tick-sched.c | 3 +- 3 files changed, 70 insertions(+), 17 deletions(-)
include/linux/irq_work.h +31 -0
@@ -4,6 +4,20 @@#include <linux/llist.h> #include <asm/irq_work.h> +/* +* An entry can be in one of four states: +* +* free NULL, 0 -> {claimed} : free to be used +* claimed NULL, 3 -> {pending} : claimed to be enqueued +* pending next, 3 -> {busy} : queued, pending callback +* busy NULL, 2 -> {free, claimed} : callback in progress, can be claimed +*/ + +#define IRQ_WORK_PENDING 1UL +#define IRQ_WORK_BUSY 2UL +#define IRQ_WORK_FLAGS 3UL +#define IRQ_WORK_LAZY 4UL/* Doesn't want IPI, wait for tick */ +struct irq_work {unsigned long flags;struct llist_node llnode;
@@ -17,8 +31,25 @@ void init_irq_work(struct irq_work *work, void (*func)(struct irq_work *)) work->func = func; } +#define DEFINE_IRQ_WORK (w, f) \ + struct irq_work w = { \ + .func = f, \ + } + +#define DEFINE_IRQ_WORK_LAZY (w, f) \ + struct irq_work w = { \ + .flags = IRQ_WORK_LAZY, \ + .func = f, \ + } +bool irq_work_queue (struct irq_work *work);void irq_work_run (void );void irq_work_sync (struct irq_work *work); +#ifdef CONFIG_IRQ_WORK +bool irq_work_needs_cpu (void ); +#else +static bool irq_work_needs_cpu (void ) {return false; } +#endif +#endif /* _LINUX_IRQ_WORK_H */
kernel/irq_work.c +37 -16
92 lines not on this page · raw message
kernel/time/tick-sched.c +2 -1
@@ -20,6 +20,7 @@#include <linux/profile.h> #include <linux/sched.h> #include <linux/module.h> +#include <linux/irq_work.h> #include <asm/irq_regs.h>
@@ -289,7 +290,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, }while (read_seqretry (&xtime_lock, seq));if (rcu_needs_cpu (cpu, &rcu_delta_jiffies) ||printk_needs_cpu (cpu) || -arch_needs_cpu (cpu)) { +arch_needs_cpu (cpu) ||irq_work_needs_cpu ()) { next_jiffies = last_jiffies +1 ; delta_jiffies =1 ; }else {
Signature · 1 line
1.7.5.4
Steven Rostedt
reviewerRe: [RFC PATCH 7/9] irq_work: Make self-IPIs optable
Quoted 29 lines · Frederic Weisbecker
On irq work initialization, let the user choose to define it as "lazy" or not. "Lazy" means that we don't want to send an IPI (provided the arch can anyway) when we enqueue this work but we rather prefer to wait for the next timer tick to execute our work if possible.
This is going to be a benefit for non-urgent enqueuers (like printk in the future) that may prefer not to raise an IPI storm in case of frequent enqueuing on short periods of time.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Andrew Morton <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul Gortmaker <[email protected]>--- include/linux/irq_work.h | 31 ++++++++++++++++++++++++++ kernel/irq_work.c | 53 ++++++++++++++++++++++++++++++++------------- kernel/time/tick-sched.c | 3 +- 3 files changed, 70 insertions(+), 17 deletions(-)include/linux/irq_work.h +0
-0
include/linux/irq_work.h line 9
Quoted 6 lines
include/linux/irq_work.h +3
-0@@ -4,6 +4,20 @@#include <linux/llist.h> #include <asm/irq_work.h> +/* +* An entry can be in one of four states: +*
Can you add a comment to what the pointer value is. I know you just moved it to the header, but it's still confusing.
Quoted 70 lines
include/linux/irq_work.h +28
-0+* free NULL, 0 -> {claimed} : free to be used +* claimed NULL, 3 -> {pending} : claimed to be enqueued +* pending next, 3 -> {busy} : queued, pending callback +* busy NULL, 2 -> {free, claimed} : callback in progress, can be claimed +*/ + +#define IRQ_WORK_PENDING 1UL +#define IRQ_WORK_BUSY 2UL +#define IRQ_WORK_FLAGS 3UL +#define IRQ_WORK_LAZY 4UL/* Doesn't want IPI, wait for tick */ +struct irq_work {unsigned long flags;struct llist_node llnode;@@ -17,8 +31,25 @@ void init_irq_work(struct irq_work *work, void (*func)(struct irq_work *)) work->func = func; } +#define DEFINE_IRQ_WORK (w, f) \ + struct irq_work w = { \ + .func = f, \ + } + +#define DEFINE_IRQ_WORK_LAZY (w, f) \ + struct irq_work w = { \ + .flags = IRQ_WORK_LAZY, \ + .func = f, \ + } +bool irq_work_queue (struct irq_work *work);void irq_work_run (void );void irq_work_sync (struct irq_work *work); +#ifdef CONFIG_IRQ_WORK +bool irq_work_needs_cpu (void ); +#else +static bool irq_work_needs_cpu (void ) {return false; } +#endif +#endif /* _LINUX_IRQ_WORK_H */ kernel/irq_work.c +2
-12@@ -12,20 +12,10 @@#include <linux/percpu.h> #include <linux/hardirq.h> #include <linux/irqflags.h> +#include <linux/tick.h> +#include <linux/sched.h> #include <asm/processor.h> -/* -* An entry can be in one of four states: -* -* free NULL, 0 -> {claimed} : free to be used -* claimed NULL, 3 -> {pending} : claimed to be enqueued -* pending next, 3 -> {busy} : queued, pending callback -* busy NULL, 2 -> {free, claimed} : callback in progress, can be claimed -*/ - -#define IRQ_WORK_PENDING 1UL -#define IRQ_WORK_BUSY 2UL -#define IRQ_WORK_FLAGS 3UL static DEFINE_PER_CPU(struct llist_head, irq_work_list);@@ -66,10 +56,28 @@ static void __irq_work_queue(struct irq_work *work)preempt_disable ();
kernel/irq_work.c static void __irq_work_queue(struct irq_work *work) · line 65
Quoted 10 lines
kernel/irq_work.c +6
-2empty =llist_add (&work->llnode, &__get_cpu_var (irq_work_list)); -/* The list was empty, raise self-interrupt to start processing. */ -if (empty) + +/* +* In any case, raise an IPI if requested and possible in case +* the queue is empty or it's filled with lazy works. +*/ +if (!(work->flags &IRQ_WORK_LAZY ) &&arch_irq_work_has_ipi ()) {arch_irq_work_raise ();
Doesn't this mean that now if we queue up a bunch of work (say in tracing), that we will send out an IPI for each queue? We only want to send out an IPI if the list isn't empty. Perhaps we should make two lists. One for lazy work and one for immediate work. Then, when adding a non-lazy work item, we can check the empty variable for that. No need to check the result for the lazy queue. That will be done during tick.
Perhaps then do:
empty = false;
if (work->flags & IRQ_WORK_LAZY)
llist_add(&work->llnode, &__get_cpu_var(irq_work_lazy_list));
else
empty = llist_add(&work->llnode, &__get_cpu_var(irq_work_list));
if (empty) {
}You can add your arch_irq_work_has_ipi() stuff as well.
-- Steve
Quoted 18 lines
kernel/irq_work.c +14
-0+ goto out; + }llist_add (&work->llnode, &__get_cpu_var (irq_work_list)); +if (empty) { +/* +* If the arch uses some other obscure way than IPI to raise +* an irq work, just raise and don't think further. +* Now if it can send an IPI, although not requested, and the tick +* is stopped, send it nonetheless otherwise we may wait a long while +* before that lazy work can run. +*/ +if (!arch_irq_work_has_ipi () ||tick_nohz_tick_stopped ()) +arch_irq_work_raise (); + } +out:preempt_enable (); }
Re: [RFC PATCH 7/9] irq_work: Make self-IPIs optable
2012/10/29 Steven Rostedt <[email protected]>:
Quoted 33 lines · Frederic Weisbecker
Quoted 32 lines
On irq work initialization, let the user choose to define it as "lazy" or not. "Lazy" means that we don't want to send an IPI (provided the arch can anyway) when we enqueue this work but we rather prefer to wait for the next timer tick to execute our work if possible.
This is going to be a benefit for non-urgent enqueuers (like printk in the future) that may prefer not to raise an IPI storm in case of frequent enqueuing on short periods of time.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Andrew Morton <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul Gortmaker <[email protected]>--- include/linux/irq_work.h | 31 ++++++++++++++++++++++++++ kernel/irq_work.c | 53 ++++++++++++++++++++++++++++++++------------- kernel/time/tick-sched.c | 3 +- 3 files changed, 70 insertions(+), 17 deletions(-)include/linux/irq_work.h +0
-0@@ -4,6 +4,20 @@#include <linux/llist.h> #include <asm/irq_work.h>
Quoted 6 lines
include/linux/irq_work.h line 9
Quoted 3 lines
include/linux/irq_work.h +3
-0+/* +* An entry can be in one of four states: +* Can you add a comment to what the pointer value is. I know you just moved it to the header, but it's still confusing.
Which pointer value? You mean the flag? You mean the below need more details or?
Quoted 11 lines
Quoted 11 lines
include/linux/irq_work.h +10
-0+* free NULL, 0 -> {claimed} : free to be used +* claimed NULL, 3 -> {pending} : claimed to be enqueued +* pending next, 3 -> {busy} : queued, pending callback +* busy NULL, 2 -> {free, claimed} : callback in progress, can be claimed +*/ + +#define IRQ_WORK_PENDING 1UL +#define IRQ_WORK_BUSY 2UL +#define IRQ_WORK_FLAGS 3UL +#define IRQ_WORK_LAZY 4UL/* Doesn't want IPI, wait for tick */
[...]
Quoted 12 lines
Quoted 12 lines
include/linux/irq_work.h +6
-2@@ -66,10 +56,28 @@ static void __irq_work_queue(struct irq_work *work)preempt_disable (); empty =llist_add (&work->llnode, &__get_cpu_var (irq_work_list)); -/* The list was empty, raise self-interrupt to start processing. */ -if (empty) + +/* +* In any case, raise an IPI if requested and possible in case +* the queue is empty or it's filled with lazy works. +*/ +if (!(work->flags &IRQ_WORK_LAZY ) &&arch_irq_work_has_ipi ()) {
Quoted 8 lines
include/linux/irq_work.h static void __irq_work_queue(struct irq_work *work) · line 65
Quoted 1 line
include/linux/irq_work.h +0
-0arch_irq_work_raise ();Doesn't this mean that now if we queue up a bunch of work (say in tracing), that we will send out an IPI for each queue? We only want to send out an IPI if the list isn't empty. Perhaps we should make two lists. One for lazy work and one for immediate work. Then, when adding a non-lazy work item, we can check the empty variable for that. No need to check the result for the lazy queue. That will be done during tick.
Indeed, if an IPI is pending while we queue another work, we'll raise another one. I would prefer to avoid the complication of adding another queue though. Perhaps a per cpu "ipi_pending" flag would be enough. I'll try something.
RFC PATCH 8/9 irq_work: Handle queuing without IPI support in dyntick idle mode
In reply to Frederic WeisbeckerIf we enqueue a work while in dyntick idle mode and the arch doesn't have self-IPI support, we may not find an opportunity to run the work before a while.
In this case, exit the idle loop to re-evaluate irq_work_needs_cpu() and restart the tick.
--- kernel/irq_work.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)
kernel/irq_work.c +11 -0
@@ -76,6 +76,17 @@ static void __irq_work_queue(struct irq_work *work)*/ if (!arch_irq_work_has_ipi () ||tick_nohz_tick_stopped ())arch_irq_work_raise (); + +/* +* If we rely on the timer tick or some obscure way to run the work +* while the CPU is in dyntick idle mode, we may not have an opportunity +* to do so before a while. Let's just exit the idle loop and hope we +* haven't yet reached the last need_resched() check before the CPU goes +* to low power mode. +*/ +if (!arch_irq_work_has_ipi () &&tick_nohz_tick_stopped () + &&is_idle_task (current)) +set_need_resched (); } out:preempt_enable ();
Signature · 1 line
1.7.5.4
RFC PATCH 9/9 printk: Wake up klogd using irq_work
In reply to Frederic Weisbeckerklogd is woken up asynchronously from the tick in order to do it safely.
However if printk is called when the tick is stopped, the reader won't be woken up until the next interrupt, which might not fire before a while. As a result, the user may miss some message.
To fix this, lets implement the printk tick using a lazy irq work. This subsystem takes care of the timer tick state and can fix up accordingly.
--- include/linux/printk.h | 3 --- init/Kconfig | 1 + kernel/printk.c | 12 +++++------- kernel/time/tick-sched.c | 2 +- kernel/timer.c | 1 - 5 files changed, 7 insertions(+), 12 deletions(-)
include/linux/printk.h +0 -3
@@ -98,9 +98,6 @@ int no_printk(const char *fmt, ...) extern asmlinkage __printf(1, 2) void early_printk(const char *fmt, ...); -extern int printk_needs_cpu(int cpu); -extern void printk_tick(void); - #ifdef CONFIG_PRINTK asmlinkage __printf(5, 0) int vprintk_emit(int facility, int level,
init/Kconfig +1 -0
@@ -1184,6 +1184,7 @@ config HOTPLUG config PRINTK default y bool "Enable support for printk" if EXPERT + select IRQ_WORK help This option enables normal printk support. Removing it eliminates most of the message strings from the kernel image
kernel/printk.c +5 -7
@@ -42,6 +42,7 @@#include <linux/notifier.h> #include <linux/rculist.h> #include <linux/poll.h> +#include <linux/irq_work.h> #include <asm/uaccess.h>
@@ -1956,7 +1957,7 @@ int is_console_locked(void) static DEFINE_PER_CPU(int, printk_pending); static DEFINE_PER_CPU(char [PRINTK_BUF_SIZE], printk_sched_buf); -void printk_tick (void ) +static void wake_up_klogd_work_func (struct irq_work *irq_work) {if (__this_cpu_read (printk_pending)) {int pending =__this_cpu_xchg (printk_pending,0 );
@@ -1969,17 +1970,14 @@ void printk_tick(void) } } -int printk_needs_cpu (int cpu) -{ -if (cpu_is_offline (cpu)) -printk_tick (); -return __this_cpu_read (printk_pending); -} +static DEFINE_IRQ_WORK_LAZY(wake_up_klogd_work, wake_up_klogd_work_func);void wake_up_klogd (void ) {if (waitqueue_active (&log_wait))this_cpu_or (printk_pending,PRINTK_PENDING_WAKEUP ); + +irq_work_queue (&wake_up_klogd_work); }static void console_cont_flush (char *text,size_t size)
kernel/time/tick-sched.c +1 -1
@@ -289,7 +289,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, time_delta =timekeeping_max_deferment (); }while (read_seqretry (&xtime_lock, seq)); -if (rcu_needs_cpu (cpu, &rcu_delta_jiffies) ||printk_needs_cpu (cpu) || +if (rcu_needs_cpu (cpu, &rcu_delta_jiffies) ||arch_needs_cpu (cpu) ||irq_work_needs_cpu ()) { next_jiffies = last_jiffies +1 ; delta_jiffies =1 ;
kernel/timer.c +0 -1
@@ -1349,7 +1349,6 @@ void update_process_times(int user_tick)account_process_tick (p, user_tick);run_local_timers ();rcu_check_callbacks (cpu, user_tick); -printk_tick ();#ifdef CONFIG_IRQ_WORK if (in_irq ())irq_work_run ();
Signature · 1 line
1.7.5.4
Steven Rostedt
reviewerRe: [RFC PATCH 9/9] printk: Wake up klogd using irq_work
Quoted 6 lines · Frederic Weisbecker
klogd is woken up asynchronously from the tick in order to do it safely.
However if printk is called when the tick is stopped, the reader won't be woken up until the next interrupt, which might not fire before a while. As a result, the user may miss some message.
Just a grammar nit (and goes for your previous patch as well). We say "might not fire for a while." or you could say ".. for some time", but not "before a while" ;-)
I wounder what the French translation of that is.
-- Steve
Quoted 6 lines
To fix this, lets implement the printk tick using a lazy irq work. This subsystem takes care of the timer tick state and can fix up accordingly.
Signed-off-by Frederic Weisbecker <[email protected]>
Re: [RFC PATCH 9/9] printk: Wake up klogd using irq_work
2012/10/29 Steven Rostedt <[email protected]>:
Quoted 11 lines · Frederic Weisbecker
Quoted 6 lines
klogd is woken up asynchronously from the tick in order to do it safely.
However if printk is called when the tick is stopped, the reader won't be woken up until the next interrupt, which might not fire before a while. As a result, the user may miss some message.
Just a grammar nit (and goes for your previous patch as well). We say "might not fire for a while." or you could say ".. for some time", but not "before a while" ;-)
Right!
Quoted 2 lines
I wounder what the French translation of that is.
You don't really want to know ;)
But as you guess, this is quite close to something like "before a while" ;)