posix-cpu-timers: Move expiry into task work context
Subsystems (4)
v3 cover posix-cpu-timers: Move expiry into task work context
This is the 3rd installment of the series to move posix timer expiry heavy lifting out of hard interrupt context.
Running posix CPU timers in hard interrupt context has a few downsides:
- For PREEMPT_RT it cannot work as the expiry code needs to take sighand lock, which is a 'sleeping spinlock' in RT. The original RT approach of offloading the posix CPU timer handling into a high priority thread was clumsy and provided no real benefit in general.
- For fine grained accounting it's just wrong to run this in context of the timer interrupt because that way a process specific cpu time is accounted to the timer interrupt.
- Long running timer interrupts caused by a large amount of expiring timers which can be created and armed by unpriviledged user space.
There is no hard requirement to expire them in interrupt context.
If the signal is targeted at the task itself then it won't be delivered before the task returns to user space anyway. If the signal is targeted at a supervisor process then it might be slightly delayed, but posix CPU timers are inaccurate anyway due to the fact that they are tied to the tick.
This series has no code dependency on the entry/KVM work, but a functional dependency vs. KVM handling task work before going into guest mode exists. It applies on mainline and passes all tests - except when KVM is active and timers are armed on the KVM threads. This particular issue is solved when the entry changes are applied as well. See below.
The previous version can be found here:
https://lore.kernel.org/r/[email protected]
The changes versus V2 are:
- Address the ordering issues vs. RT (Peter)
- Move task work head into task struct (Oleg)
- Drop the last patch which is an optimization and needs more thought now with the reworked state handling.
The series is standalone except for the fact that KVM does not handle task work before entering a guest. The necessary changes for this are in
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry
and the whole lot (entry + posix CPU timers) is available from:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git timers/posixtimer
Thanks,
tglx
--- arch/x86/Kconfig | 1 include/linux/posix-timers.h | 17 +++ include/linux/sched.h | 4 include/linux/seccomp.h | 3 kernel/entry/common.c | 4 kernel/time/Kconfig | 9 + kernel/time/posix-cpu-timers.c | 216 ++++++++++++++++++++++++++++++++++++----- kernel/time/timer.c | 1 8 files changed, 227 insertions(+), 28 deletions(-)
PATCH v3 1/3 posix-cpu-timers: Split run_posix_cpu_timers()
From: Thomas Gleixner <[email protected]>
Split it up as a preparatory step to move the heavy lifting out of interrupt context.
--- kernel/time/posix-cpu-timers.c | 43 ++++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 19 deletions(-)
kernel/time/posix-cpu-timers.c +24 -19
@@ -1080,32 +1080,15 @@ static inline bool fastpath_timer_check(return false; } -/* -* This is called from the timer interrupt handler. The irq handler has -* already updated our counts. We need to check if any timers fire now. -* Interrupts are disabled. -*/ -void run_posix_cpu_timers (void ) +static void __run_posix_cpu_timers (struct task_struct *tsk) { -struct task_struct *tsk = current;struct k_itimer *timer, *next;unsigned long flags;LIST_HEAD (firing); -lockdep_assert_irqs_disabled (); - -/* -* The fast path checks that there are no expired thread or thread -* group timers. If that's so, just return. -*/ -if (!fastpath_timer_check (tsk)) +if (!lock_task_sighand (tsk, &flags))return ; -lockdep_posixtimer_enter (); -if (!lock_task_sighand (tsk, &flags)) { -lockdep_posixtimer_exit (); -return ; - }/* * Here we take off tsk->signal->cpu_timers[N] and * tsk->cpu_timers[N] all the timers that are firing, and
@@ -1147,6 +1130,28 @@ void run_posix_cpu_timers(void)cpu_timer_fire (timer);spin_unlock (&timer->it_lock); } +} + +/* +* This is called from the timer interrupt handler. The irq handler has +* already updated our counts. We need to check if any timers fire now. +* Interrupts are disabled. +*/ +void run_posix_cpu_timers (void ) +{ +struct task_struct *tsk = current; + +lockdep_assert_irqs_disabled (); + +/* +* The fast path checks that there are no expired thread or thread +* group timers. If that's so, just return. +*/ +if (!fastpath_timer_check (tsk)) +return ; + +lockdep_posixtimer_enter (); +__run_posix_cpu_timers (tsk);lockdep_posixtimer_exit (); }
[tip: timers/core] posix-cpu-timers: Split run_posix_cpu_timers()
The following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 820903c784a01bf6e143253418508da4f5790cff Gitweb: https://git.kernel.org/tip/820903c784a01bf6e143253418508da4f5790cff Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 30 Jul 2020 12:14:05 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Thu, 06 Aug 2020 16:50:59 +02:00
posix-cpu-timers: Split run_posix_cpu_timers()
Split it up as a preparatory step to move the heavy lifting out of interrupt context.
--- kernel/time/posix-cpu-timers.c | 43 ++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 19 deletions(-)
kernel/time/posix-cpu-timers.c +24 -19
@@ -1080,32 +1080,15 @@ static inline bool fastpath_timer_check(struct task_struct *tsk)return false; } -/* -* This is called from the timer interrupt handler. The irq handler has -* already updated our counts. We need to check if any timers fire now. -* Interrupts are disabled. -*/ -void run_posix_cpu_timers (void ) +static void __run_posix_cpu_timers (struct task_struct *tsk) { -struct task_struct *tsk = current;struct k_itimer *timer, *next;unsigned long flags;LIST_HEAD (firing); -lockdep_assert_irqs_disabled (); - -/* -* The fast path checks that there are no expired thread or thread -* group timers. If that's so, just return. -*/ -if (!fastpath_timer_check (tsk)) +if (!lock_task_sighand (tsk, &flags))return ; -lockdep_posixtimer_enter (); -if (!lock_task_sighand (tsk, &flags)) { -lockdep_posixtimer_exit (); -return ; - }/* * Here we take off tsk->signal->cpu_timers[N] and * tsk->cpu_timers[N] all the timers that are firing, and
@@ -1147,6 +1130,28 @@ void run_posix_cpu_timers(void)cpu_timer_fire (timer);spin_unlock (&timer->it_lock); } +} + +/* +* This is called from the timer interrupt handler. The irq handler has +* already updated our counts. We need to check if any timers fire now. +* Interrupts are disabled. +*/ +void run_posix_cpu_timers (void ) +{ +struct task_struct *tsk = current; + +lockdep_assert_irqs_disabled (); + +/* +* The fast path checks that there are no expired thread or thread +* group timers. If that's so, just return. +*/ +if (!fastpath_timer_check (tsk)) +return ; + +lockdep_posixtimer_enter (); +__run_posix_cpu_timers (tsk);lockdep_posixtimer_exit (); }
PATCH v3 2/3 posix-cpu-timers: Provide mechanisms to defer timer handling to task_work
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Running posix CPU timers in hard interrupt context has a few downsides:
- For PREEMPT_RT it cannot work as the expiry code needs to take sighand lock, which is a 'sleeping spinlock' in RT. The original RT approach of offloading the posix CPU timer handling into a high priority thread was clumsy and provided no real benefit in general.
- For fine grained accounting it's just wrong to run this in context of the timer interrupt because that way a process specific cpu time is accounted to the timer interrupt.
- Long running timer interrupts caused by a large amount of expiring timers which can be created and armed by unpriviledged user space.
There is no hard requirement to expire them in interrupt context.
If the signal is targeted at the task itself then it won't be delivered before the task returns to user space anyway. If the signal is targeted at a supervisor process then it might be slightly delayed, but posix CPU timers are inaccurate anyway due to the fact that they are tied to the tick.
Provide infrastructure to schedule task work which allows splitting the posix CPU timer code into a quick check in interrupt context and a thread context expiry and signal delivery function. This has to be enabled by architectures as it requires that the architecture specific KVM implementation handles pending task work before exiting to guest mode.
---
V3: Move the task work head to task_struct and just initialize it on
init_task (Oleg)
Address the ordering issues vs. RT and add comments documenting it (Peter)
Drop atomic ops as they are not required anymore
--- include/linux/posix-timers.h | 17 +++ include/linux/sched.h | 4 kernel/time/Kconfig | 9 + kernel/time/posix-cpu-timers.c | 185 ++++++++++++++++++++++++++++++++++++++--- kernel/time/timer.c | 1 5 files changed, 204 insertions(+), 12 deletions(-)
include/linux/posix-timers.h +17 -0
@@ -6,6 +6,7 @@#include <linux/list.h> #include <linux/alarmtimer.h> #include <linux/timerqueue.h> +#include <linux/task_work.h> struct kernel_siginfo ;struct task_struct ;
@@ -125,6 +126,16 @@ struct posix_cputimers {unsigned int expiry_active; }; +/** +* posix_cputimers_work - Container for task work based posix CPU timer expiry +* @work: The task work to be scheduled +* @scheduled: @work has been scheduled already, no further processing +*/ +struct posix_cputimers_work { +struct callback_head work; +unsigned int scheduled; +}; +static inline void posix_cputimers_init (struct posix_cputimers *pct) {memset (pct,0 ,sizeof (*pct));
@@ -165,6 +176,12 @@ static inline void posix_cputimers_group u64 cpu_limit) { }#endif +#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK +void posix_cputimers_init_work (void ); +#else +static inline void posix_cputimers_init_work (void ) { } +#endif +#define REQUEUE_PENDING 1/**
include/linux/sched.h +4 -0
@@ -882,6 +882,10 @@ struct task_struct {/* Empty if CONFIG_POSIX_CPUTIMERS=n */ struct posix_cputimers posix_cputimers; +#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK +struct posix_cputimers_work posix_cputimers_work; +#endif +/* Process credentials: */ /* Tracer's credentials at attach: */
kernel/time/Kconfig +9 -0
@@ -52,6 +52,15 @@ config GENERIC_CLOCKEVENTS_MIN_ADJUST config GENERIC_CMOS_UPDATE bool +# Select to handle posix CPU timers from task_work +# and not from the timer interrupt context +config HAVE_POSIX_CPU_TIMERS_TASK_WORK + bool + +config POSIX_CPU_TIMERS_TASK_WORK + bool + default y if POSIX_TIMERS && HAVE_POSIX_CPU_TIMERS_TASK_WORK + if GENERIC_CLOCKEVENTS menu "Timers subsystem"
kernel/time/posix-cpu-timers.c +173 -12
232 lines not on this page · raw message
kernel/time/timer.c +1 -0
@@ -2046,6 +2046,7 @@ static void __init init_timer_cpus(void)void __init init_timers (void ) {init_timer_cpus (); +posix_cputimers_init_work ();open_softirq (TIMER_SOFTIRQ , run_timer_softirq); }
[tip: timers/core] posix-cpu-timers: Provide mechanisms to defer timer handling to task_work
The following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 1fb497dd003009be95ce67689ac800c446b7acc5 Gitweb: https://git.kernel.org/tip/1fb497dd003009be95ce67689ac800c446b7acc5 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 30 Jul 2020 12:14:06 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Thu, 06 Aug 2020 16:50:59 +02:00
posix-cpu-timers: Provide mechanisms to defer timer handling to task_work
Running posix CPU timers in hard interrupt context has a few downsides:
- For PREEMPT_RT it cannot work as the expiry code needs to take sighand lock, which is a 'sleeping spinlock' in RT. The original RT approach of offloading the posix CPU timer handling into a high priority thread was clumsy and provided no real benefit in general.
- For fine grained accounting it's just wrong to run this in context of the timer interrupt because that way a process specific CPU time is accounted to the timer interrupt.
- Long running timer interrupts caused by a large amount of expiring timers which can be created and armed by unpriviledged user space.
There is no hard requirement to expire them in interrupt context.
If the signal is targeted at the task itself then it won't be delivered before the task returns to user space anyway. If the signal is targeted at a supervisor process then it might be slightly delayed, but posix CPU timers are inaccurate anyway due to the fact that they are tied to the tick.
Provide infrastructure to schedule task work which allows splitting the posix CPU timer code into a quick check in interrupt context and a thread context expiry and signal delivery function. This has to be enabled by architectures as it requires that the architecture specific KVM implementation handles pending task work before exiting to guest mode.
--- include/linux/posix-timers.h | 17 +++- include/linux/sched.h | 4 +- kernel/time/Kconfig | 9 ++- kernel/time/posix-cpu-timers.c | 185 +++++++++++++++++++++++++++++--- kernel/time/timer.c | 1 +- 5 files changed, 204 insertions(+), 12 deletions(-)
include/linux/posix-timers.h +17 -0
@@ -6,6 +6,7 @@#include <linux/list.h> #include <linux/alarmtimer.h> #include <linux/timerqueue.h> +#include <linux/task_work.h> struct kernel_siginfo ;struct task_struct ;
@@ -125,6 +126,16 @@ struct posix_cputimers {unsigned int expiry_active; }; +/** +* posix_cputimers_work - Container for task work based posix CPU timer expiry +* @work: The task work to be scheduled +* @scheduled: @work has been scheduled already, no further processing +*/ +struct posix_cputimers_work { +struct callback_head work; +unsigned int scheduled; +}; +static inline void posix_cputimers_init (struct posix_cputimers *pct) {memset (pct,0 ,sizeof (*pct));
@@ -165,6 +176,12 @@ static inline void posix_cputimers_group_init(struct posix_cputimers *pct, u64 cpu_limit) { }#endif +#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK +void posix_cputimers_init_work (void ); +#else +static inline void posix_cputimers_init_work (void ) { } +#endif +#define REQUEUE_PENDING 1/**
include/linux/sched.h +4 -0
@@ -889,6 +889,10 @@ struct task_struct {/* Empty if CONFIG_POSIX_CPUTIMERS=n */ struct posix_cputimers posix_cputimers; +#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK +struct posix_cputimers_work posix_cputimers_work; +#endif +/* Process credentials: */ /* Tracer's credentials at attach: */
kernel/time/Kconfig +9 -0
@@ -52,6 +52,15 @@ config GENERIC_CLOCKEVENTS_MIN_ADJUST config GENERIC_CMOS_UPDATE bool +# Select to handle posix CPU timers from task_work +# and not from the timer interrupt context +config HAVE_POSIX_CPU_TIMERS_TASK_WORK + bool + +config POSIX_CPU_TIMERS_TASK_WORK + bool + default y if POSIX_TIMERS && HAVE_POSIX_CPU_TIMERS_TASK_WORK + if GENERIC_CLOCKEVENTS menu "Timers subsystem"
kernel/time/posix-cpu-timers.c +173 -12
232 lines not on this page · raw message
kernel/time/timer.c +1 -0
@@ -2017,6 +2017,7 @@ static void __init init_timer_cpus(void)void __init init_timers (void ) {init_timer_cpus (); +posix_cputimers_init_work ();open_softirq (TIMER_SOFTIRQ , run_timer_softirq); }
PATCH v3 3/3 x86: Select POSIX_CPU_TIMERS_TASK_WORK
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Move POSIX CPU timer expiry and signal delivery into task context.
--- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -209,6 +209,7 @@ config X86 select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT + select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION select HAVE_FUNCTION_ARG_ACCESS_API
[tip: timers/core] x86: Select POSIX_CPU_TIMERS_TASK_WORK
The following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 0099808553ad4f9c04ad7afd966f6d7f470f247f Gitweb: https://git.kernel.org/tip/0099808553ad4f9c04ad7afd966f6d7f470f247f Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 30 Jul 2020 12:14:07 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Thu, 06 Aug 2020 16:50:59 +02:00
x86: Select POSIX_CPU_TIMERS_TASK_WORK
Move POSIX CPU timer expiry and signal delivery into task context.
--- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -209,6 +209,7 @@ config X86 select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT + select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION select HAVE_FUNCTION_ARG_ACCESS_API
Quoted 10 lines · Thomas Gleixner
arch/x86/Kconfig | 1 include/linux/posix-timers.h | 17 +++ include/linux/sched.h | 4 include/linux/seccomp.h | 3 kernel/entry/common.c | 4 kernel/time/Kconfig | 9 + kernel/time/posix-cpu-timers.c | 216 ++++++++++++++++++++++++++++++++++++----- kernel/time/timer.c | 1 8 files changed, 227 insertions(+), 28 deletions(-)
FWIW,
Quoted 9 lines · Thomas Gleixner
arch/x86/Kconfig | 1 include/linux/posix-timers.h | 17 +++ include/linux/sched.h | 4 include/linux/seccomp.h | 3 kernel/entry/common.c | 4 kernel/time/Kconfig | 9 + kernel/time/posix-cpu-timers.c | 216 ++++++++++++++++++++++++++++++++++++----- kernel/time/timer.c | 1 8 files changed, 227 insertions(+), 28 deletions(-)