Skip to contentSkip to conversation

posix-cpu-timers: Move expiry into task work context

PATCH Applied Thomas Gleixner 4 participants 9 messages Jul 30, 2020 – Aug 6, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v3 cover posix-cpu-timers: Move expiry into task work context

This is the 3rd installment of the series to move posix timer expiry heavy lifting out of hard interrupt context.

Running posix CPU timers in hard interrupt context has a few downsides:

- For PREEMPT_RT it cannot work as the expiry code needs to take sighand lock, which is a 'sleeping spinlock' in RT. The original RT approach of offloading the posix CPU timer handling into a high priority thread was clumsy and provided no real benefit in general.

- For fine grained accounting it's just wrong to run this in context of the timer interrupt because that way a process specific cpu time is accounted to the timer interrupt.

- Long running timer interrupts caused by a large amount of expiring timers which can be created and armed by unpriviledged user space.

There is no hard requirement to expire them in interrupt context.

If the signal is targeted at the task itself then it won't be delivered before the task returns to user space anyway. If the signal is targeted at a supervisor process then it might be slightly delayed, but posix CPU timers are inaccurate anyway due to the fact that they are tied to the tick.

This series has no code dependency on the entry/KVM work, but a functional dependency vs. KVM handling task work before going into guest mode exists. It applies on mainline and passes all tests - except when KVM is active and timers are armed on the KVM threads. This particular issue is solved when the entry changes are applied as well. See below.

The previous version can be found here:

    https://lore.kernel.org/r/[email protected]

The changes versus V2 are:

- Address the ordering issues vs. RT (Peter)
- Move task work head into task struct (Oleg)
- Drop the last patch which is an optimization and needs more thought now with the reworked state handling.

The series is standalone except for the fact that KVM does not handle task work before entering a guest. The necessary changes for this are in

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry

and the whole lot (entry + posix CPU timers) is available from:

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git timers/posixtimer

Thanks,

tglx

---
 arch/x86/Kconfig               |    1 
 include/linux/posix-timers.h   |   17 +++
 include/linux/sched.h          |    4 
 include/linux/seccomp.h        |    3 
 kernel/entry/common.c          |    4 
 kernel/time/Kconfig            |    9 +
 kernel/time/posix-cpu-timers.c |  216 ++++++++++++++++++++++++++++++++++++-----
 kernel/time/timer.c            |    1 
 8 files changed, 227 insertions(+), 28 deletions(-)
#1 lore raw

PATCH v3 1/3 posix-cpu-timers: Split run_posix_cpu_timers()

From: Thomas Gleixner <[email protected]>

Split it up as a preparatory step to move the heavy lifting out of interrupt context.

---
 kernel/time/posix-cpu-timers.c |   43 ++++++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 19 deletions(-)
kernel/time/posix-cpu-timers.c +24 -19
@@ -1080,32 +1080,15 @@ static inline bool fastpath_timer_check(
 	return false;
 }
 
-/*
- * This is called from the timer interrupt handler.  The irq handler has
- * already updated our counts.  We need to check if any timers fire now.
- * Interrupts are disabled.
- */
-void run_posix_cpu_timers(void)
+static void __run_posix_cpu_timers(struct task_struct *tsk)
 {
-	struct task_struct *tsk = current;
 	struct k_itimer *timer, *next;
 	unsigned long flags;
 	LIST_HEAD(firing);
 
-	lockdep_assert_irqs_disabled();
-
-	/*
-	 * The fast path checks that there are no expired thread or thread
-	 * group timers.  If that's so, just return.
-	 */
-	if (!fastpath_timer_check(tsk))
+	if (!lock_task_sighand(tsk, &flags))
 		return;
 
-	lockdep_posixtimer_enter();
-	if (!lock_task_sighand(tsk, &flags)) {
-		lockdep_posixtimer_exit();
-		return;
-	}
 	/*
 	 * Here we take off tsk->signal->cpu_timers[N] and
 	 * tsk->cpu_timers[N] all the timers that are firing, and
@@ -1147,6 +1130,28 @@ void run_posix_cpu_timers(void)
 			cpu_timer_fire(timer);
 		spin_unlock(&timer->it_lock);
 	}
+}
+
+/*
+ * This is called from the timer interrupt handler.  The irq handler has
+ * already updated our counts.  We need to check if any timers fire now.
+ * Interrupts are disabled.
+ */
+void run_posix_cpu_timers(void)
+{
+	struct task_struct *tsk = current;
+
+	lockdep_assert_irqs_disabled();
+
+	/*
+	 * The fast path checks that there are no expired thread or thread
+	 * group timers.  If that's so, just return.
+	 */
+	if (!fastpath_timer_check(tsk))
+		return;
+
+	lockdep_posixtimer_enter();
+	__run_posix_cpu_timers(tsk);
 	lockdep_posixtimer_exit();
 }
 
#2 lore raw

[tip: timers/core] posix-cpu-timers: Split run_posix_cpu_timers()

The following commit has been merged into the timers/core branch of tip:

The following commit has been merged into the timers/core branch of tip:

Commit-ID:     820903c784a01bf6e143253418508da4f5790cff
Gitweb:        https://git.kernel.org/tip/820903c784a01bf6e143253418508da4f5790cff
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 30 Jul 2020 12:14:05 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 06 Aug 2020 16:50:59 +02:00

posix-cpu-timers: Split run_posix_cpu_timers()

Split it up as a preparatory step to move the heavy lifting out of interrupt context.

---
 kernel/time/posix-cpu-timers.c | 43 ++++++++++++++++++---------------
 1 file changed, 24 insertions(+), 19 deletions(-)
kernel/time/posix-cpu-timers.c +24 -19
index 1651179..e5ad873 100644
@@ -1080,32 +1080,15 @@ static inline bool fastpath_timer_check(struct task_struct *tsk)
 	return false;
 }
 
-/*
- * This is called from the timer interrupt handler.  The irq handler has
- * already updated our counts.  We need to check if any timers fire now.
- * Interrupts are disabled.
- */
-void run_posix_cpu_timers(void)
+static void __run_posix_cpu_timers(struct task_struct *tsk)
 {
-	struct task_struct *tsk = current;
 	struct k_itimer *timer, *next;
 	unsigned long flags;
 	LIST_HEAD(firing);
 
-	lockdep_assert_irqs_disabled();
-
-	/*
-	 * The fast path checks that there are no expired thread or thread
-	 * group timers.  If that's so, just return.
-	 */
-	if (!fastpath_timer_check(tsk))
+	if (!lock_task_sighand(tsk, &flags))
 		return;
 
-	lockdep_posixtimer_enter();
-	if (!lock_task_sighand(tsk, &flags)) {
-		lockdep_posixtimer_exit();
-		return;
-	}
 	/*
 	 * Here we take off tsk->signal->cpu_timers[N] and
 	 * tsk->cpu_timers[N] all the timers that are firing, and
@@ -1147,6 +1130,28 @@ void run_posix_cpu_timers(void)
 			cpu_timer_fire(timer);
 		spin_unlock(&timer->it_lock);
 	}
+}
+
+/*
+ * This is called from the timer interrupt handler.  The irq handler has
+ * already updated our counts.  We need to check if any timers fire now.
+ * Interrupts are disabled.
+ */
+void run_posix_cpu_timers(void)
+{
+	struct task_struct *tsk = current;
+
+	lockdep_assert_irqs_disabled();
+
+	/*
+	 * The fast path checks that there are no expired thread or thread
+	 * group timers.  If that's so, just return.
+	 */
+	if (!fastpath_timer_check(tsk))
+		return;
+
+	lockdep_posixtimer_enter();
+	__run_posix_cpu_timers(tsk);
 	lockdep_posixtimer_exit();
 }
 
#3 lore raw

PATCH v3 2/3 posix-cpu-timers: Provide mechanisms to defer timer handling to task_work

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Running posix CPU timers in hard interrupt context has a few downsides:

- For PREEMPT_RT it cannot work as the expiry code needs to take sighand lock, which is a 'sleeping spinlock' in RT. The original RT approach of offloading the posix CPU timer handling into a high priority thread was clumsy and provided no real benefit in general.

- For fine grained accounting it's just wrong to run this in context of the timer interrupt because that way a process specific cpu time is accounted to the timer interrupt.

- Long running timer interrupts caused by a large amount of expiring timers which can be created and armed by unpriviledged user space.

There is no hard requirement to expire them in interrupt context.

If the signal is targeted at the task itself then it won't be delivered before the task returns to user space anyway. If the signal is targeted at a supervisor process then it might be slightly delayed, but posix CPU timers are inaccurate anyway due to the fact that they are tied to the tick.

Provide infrastructure to schedule task work which allows splitting the posix CPU timer code into a quick check in interrupt context and a thread context expiry and signal delivery function. This has to be enabled by architectures as it requires that the architecture specific KVM implementation handles pending task work before exiting to guest mode.

---
V3: Move the task work head to task_struct and just initialize it on
init_task (Oleg)
Address the ordering issues vs. RT and add comments documenting it (Peter)
Drop atomic ops as they are not required anymore

---
 include/linux/posix-timers.h   |   17 +++
 include/linux/sched.h          |    4 
 kernel/time/Kconfig            |    9 +
 kernel/time/posix-cpu-timers.c |  185 ++++++++++++++++++++++++++++++++++++++---
 kernel/time/timer.c            |    1 
 5 files changed, 204 insertions(+), 12 deletions(-)
include/linux/posix-timers.h +17 -0
@@ -6,6 +6,7 @@
 #include <linux/list.h>
 #include <linux/alarmtimer.h>
 #include <linux/timerqueue.h>
+#include <linux/task_work.h>
 
 struct kernel_siginfo;
 struct task_struct;
@@ -125,6 +126,16 @@ struct posix_cputimers {
 	unsigned int			expiry_active;
 };
 
+/**
+ * posix_cputimers_work - Container for task work based posix CPU timer expiry
+ * @work:	The task work to be scheduled
+ * @scheduled:  @work has been scheduled already, no further processing
+ */
+struct posix_cputimers_work {
+	struct callback_head	work;
+	unsigned int		scheduled;
+};
+
 static inline void posix_cputimers_init(struct posix_cputimers *pct)
 {
 	memset(pct, 0, sizeof(*pct));
@@ -165,6 +176,12 @@ static inline void posix_cputimers_group
 					      u64 cpu_limit) { }
 #endif
 
+#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK
+void posix_cputimers_init_work(void);
+#else
+static inline void posix_cputimers_init_work(void) { }
+#endif
+
 #define REQUEUE_PENDING 1
 
 /**
include/linux/sched.h +4 -0
@@ -882,6 +882,10 @@ struct task_struct {
 	/* Empty if CONFIG_POSIX_CPUTIMERS=n */
 	struct posix_cputimers		posix_cputimers;
 
+#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK
+	struct posix_cputimers_work	posix_cputimers_work;
+#endif
+
 	/* Process credentials: */
 
 	/* Tracer's credentials at attach: */
kernel/time/Kconfig +9 -0
@@ -52,6 +52,15 @@ config GENERIC_CLOCKEVENTS_MIN_ADJUST
 config GENERIC_CMOS_UPDATE
 	bool
 
+# Select to handle posix CPU timers from task_work
+# and not from the timer interrupt context
+config HAVE_POSIX_CPU_TIMERS_TASK_WORK
+	bool
+
+config POSIX_CPU_TIMERS_TASK_WORK
+	bool
+	default y if POSIX_TIMERS && HAVE_POSIX_CPU_TIMERS_TASK_WORK
+
 if GENERIC_CLOCKEVENTS
 menu "Timers subsystem"
 
kernel/time/posix-cpu-timers.c +173 -12

232 lines not on this page · raw message

kernel/time/timer.c +1 -0
@@ -2046,6 +2046,7 @@ static void __init init_timer_cpus(void)
 void __init init_timers(void)
 {
 	init_timer_cpus();
+	posix_cputimers_init_work();
 	open_softirq(TIMER_SOFTIRQ, run_timer_softirq);
 }
 
#4 lore raw

[tip: timers/core] posix-cpu-timers: Provide mechanisms to defer timer handling to task_work

The following commit has been merged into the timers/core branch of tip:

The following commit has been merged into the timers/core branch of tip:

Commit-ID:     1fb497dd003009be95ce67689ac800c446b7acc5
Gitweb:        https://git.kernel.org/tip/1fb497dd003009be95ce67689ac800c446b7acc5
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 30 Jul 2020 12:14:06 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 06 Aug 2020 16:50:59 +02:00

posix-cpu-timers: Provide mechanisms to defer timer handling to task_work

Running posix CPU timers in hard interrupt context has a few downsides:

- For PREEMPT_RT it cannot work as the expiry code needs to take sighand lock, which is a 'sleeping spinlock' in RT. The original RT approach of offloading the posix CPU timer handling into a high priority thread was clumsy and provided no real benefit in general.

- For fine grained accounting it's just wrong to run this in context of the timer interrupt because that way a process specific CPU time is accounted to the timer interrupt.

- Long running timer interrupts caused by a large amount of expiring timers which can be created and armed by unpriviledged user space.

There is no hard requirement to expire them in interrupt context.

If the signal is targeted at the task itself then it won't be delivered before the task returns to user space anyway. If the signal is targeted at a supervisor process then it might be slightly delayed, but posix CPU timers are inaccurate anyway due to the fact that they are tied to the tick.

Provide infrastructure to schedule task work which allows splitting the posix CPU timer code into a quick check in interrupt context and a thread context expiry and signal delivery function. This has to be enabled by architectures as it requires that the architecture specific KVM implementation handles pending task work before exiting to guest mode.

---
 include/linux/posix-timers.h   |  17 +++-
 include/linux/sched.h          |   4 +-
 kernel/time/Kconfig            |   9 ++-
 kernel/time/posix-cpu-timers.c | 185 +++++++++++++++++++++++++++++---
 kernel/time/timer.c            |   1 +-
 5 files changed, 204 insertions(+), 12 deletions(-)
include/linux/posix-timers.h +17 -0
index e3f0f85..896c16d 100644
@@ -6,6 +6,7 @@
 #include <linux/list.h>
 #include <linux/alarmtimer.h>
 #include <linux/timerqueue.h>
+#include <linux/task_work.h>
 
 struct kernel_siginfo;
 struct task_struct;
@@ -125,6 +126,16 @@ struct posix_cputimers {
 	unsigned int			expiry_active;
 };
 
+/**
+ * posix_cputimers_work - Container for task work based posix CPU timer expiry
+ * @work:	The task work to be scheduled
+ * @scheduled:  @work has been scheduled already, no further processing
+ */
+struct posix_cputimers_work {
+	struct callback_head	work;
+	unsigned int		scheduled;
+};
+
 static inline void posix_cputimers_init(struct posix_cputimers *pct)
 {
 	memset(pct, 0, sizeof(*pct));
@@ -165,6 +176,12 @@ static inline void posix_cputimers_group_init(struct posix_cputimers *pct,
 					      u64 cpu_limit) { }
 #endif
 
+#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK
+void posix_cputimers_init_work(void);
+#else
+static inline void posix_cputimers_init_work(void) { }
+#endif
+
 #define REQUEUE_PENDING 1
 
 /**
include/linux/sched.h +4 -0
index 06ec604..e9942ce 100644
@@ -889,6 +889,10 @@ struct task_struct {
 	/* Empty if CONFIG_POSIX_CPUTIMERS=n */
 	struct posix_cputimers		posix_cputimers;
 
+#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK
+	struct posix_cputimers_work	posix_cputimers_work;
+#endif
+
 	/* Process credentials: */
 
 	/* Tracer's credentials at attach: */
kernel/time/Kconfig +9 -0
index fcc4235..a09b1d6 100644
@@ -52,6 +52,15 @@ config GENERIC_CLOCKEVENTS_MIN_ADJUST
 config GENERIC_CMOS_UPDATE
 	bool
 
+# Select to handle posix CPU timers from task_work
+# and not from the timer interrupt context
+config HAVE_POSIX_CPU_TIMERS_TASK_WORK
+	bool
+
+config POSIX_CPU_TIMERS_TASK_WORK
+	bool
+	default y if POSIX_TIMERS && HAVE_POSIX_CPU_TIMERS_TASK_WORK
+
 if GENERIC_CLOCKEVENTS
 menu "Timers subsystem"
 
kernel/time/posix-cpu-timers.c +173 -12

232 lines not on this page · raw message

kernel/time/timer.c +1 -0
index ae5029f..a16764b 100644
@@ -2017,6 +2017,7 @@ static void __init init_timer_cpus(void)
 void __init init_timers(void)
 {
 	init_timer_cpus();
+	posix_cputimers_init_work();
 	open_softirq(TIMER_SOFTIRQ, run_timer_softirq);
 }
 
#5 lore raw

PATCH v3 3/3 x86: Select POSIX_CPU_TIMERS_TASK_WORK

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Move POSIX CPU timer expiry and signal delivery into task context.

---
 arch/x86/Kconfig |    1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -209,6 +209,7 @@ config X86
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
 	select MMU_GATHER_RCU_TABLE_FREE		if PARAVIRT
+	select HAVE_POSIX_CPU_TIMERS_TASK_WORK
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_RELIABLE_STACKTRACE		if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
 	select HAVE_FUNCTION_ARG_ACCESS_API
#6 lore raw

[tip: timers/core] x86: Select POSIX_CPU_TIMERS_TASK_WORK

The following commit has been merged into the timers/core branch of tip:

The following commit has been merged into the timers/core branch of tip:

Commit-ID:     0099808553ad4f9c04ad7afd966f6d7f470f247f
Gitweb:        https://git.kernel.org/tip/0099808553ad4f9c04ad7afd966f6d7f470f247f
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 30 Jul 2020 12:14:07 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 06 Aug 2020 16:50:59 +02:00

x86: Select POSIX_CPU_TIMERS_TASK_WORK

Move POSIX CPU timer expiry and signal delivery into task context.

---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
index fd03cef..a82e715 100644
@@ -209,6 +209,7 @@ config X86
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
 	select MMU_GATHER_RCU_TABLE_FREE		if PARAVIRT
+	select HAVE_POSIX_CPU_TIMERS_TASK_WORK
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_RELIABLE_STACKTRACE		if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
 	select HAVE_FUNCTION_ARG_ACCESS_API
Reviewed-by on the v3 series
#7 lore raw
In reply to Thomas Gleixner
Quoted 10 lines · Thomas Gleixner
 arch/x86/Kconfig               |    1 
 include/linux/posix-timers.h   |   17 +++
 include/linux/sched.h          |    4 
 include/linux/seccomp.h        |    3 
 kernel/entry/common.c          |    4 
 kernel/time/Kconfig            |    9 +
 kernel/time/posix-cpu-timers.c |  216 ++++++++++++++++++++++++++++++++++++-----
 kernel/time/timer.c            |    1 
 8 files changed, 227 insertions(+), 28 deletions(-)

FWIW,

maintainer Acked-by on the v3 series
#8 lore raw
In reply to Thomas Gleixner
Quoted 9 lines · Thomas Gleixner
 arch/x86/Kconfig               |    1 
 include/linux/posix-timers.h   |   17 +++
 include/linux/sched.h          |    4 
 include/linux/seccomp.h        |    3 
 kernel/entry/common.c          |    4 
 kernel/time/Kconfig            |    9 +
 kernel/time/posix-cpu-timers.c |  216 ++++++++++++++++++++++++++++++++++++-----
 kernel/time/timer.c            |    1 
 8 files changed, 227 insertions(+), 28 deletions(-)