Skip to contentSkip to conversation

timer: Improve itimers scalability

PATCH In review Jason Low 7 participants 17 messages Oct 14, 2015 – Oct 16, 2015

Subsystems (4)
lore
Share & tools
lore

Jason Low

author
#0 lore raw

v2 cover timer: Improve itimers scalability

While running a database workload on a 16 socket machine, there were scalability issues related to itimers. The following link contains a more detailed summary of the issues at the application level.

https://lkml.org/lkml/2015/8/26/737

Commit 1018016c706f addressed the issue with the thread_group_cputimer spinlock taking up a significant portion of total run time. This patch series addresses the secondary issue where a lot of time is spent trying to acquire the sighand lock. It was found in some cases that 200+ threads were simultaneously contending for the same sighand lock, reducing throughput by more than 30%.

With this patch set (along with commit 1018016c706f mentioned above), the performance hit of itimers almost completely goes away on the 16 socket system.

Jason Low (4):
timer: Optimize fastpath_timer_check()
timer: Check thread timers only when there are active thread timers
timer: Convert cputimer->running to bool
timer: Reduce unnecessary sighand lock contention

 include/linux/init_task.h      |    3 +-
 include/linux/sched.h          |    9 ++++--
 kernel/fork.c                  |    2 +-
 kernel/time/posix-cpu-timers.c |   63 ++++++++++++++++++++++++++++-----------
 4 files changed, 54 insertions(+), 23 deletions(-)
Signature · 1 line
1.7.2.5
author
#1 lore raw

PATCH v2 1/4 timer: Optimize fastpath_timer_check()

In fastpath_timer_check(), the task_cputime() function is always called to compute the utime and stime values. However, this is not necessary if there are no per-thread timers to check for. This patch modifies the code such that we compute the task_cputime values only when there are per-thread timers set.

---
 kernel/time/posix-cpu-timers.c |   11 +++--------
 1 files changed, 3 insertions(+), 8 deletions(-)
kernel/time/posix-cpu-timers.c +3 -8
index 892e3da..aa4b6f4 100644
@@ -1117,17 +1117,12 @@ static inline int task_cputime_expired(const struct task_cputime *sample,
 static inline int fastpath_timer_check(struct task_struct *tsk)
 {
 	struct signal_struct *sig;
-	cputime_t utime, stime;
-
-	task_cputime(tsk, &utime, &stime);
 
 	if (!task_cputime_zero(&tsk->cputime_expires)) {
-		struct task_cputime task_sample = {
-			.utime = utime,
-			.stime = stime,
-			.sum_exec_runtime = tsk->se.sum_exec_runtime
-		};
+		struct task_cputime task_sample;
 
+		task_cputime(tsk, &task_sample.utime, &task_sample.stime);
+		task_sample.sum_exec_runtime = tsk->se.sum_exec_runtime;
 		if (task_cputime_expired(&task_sample, &tsk->cputime_expires))
 			return 1;
 	}
Signature · 1 line
1.7.2.5
#2 lore raw

[tip:timers/core] posix_cpu_timer: Optimize fastpath_timer_check( )

posix_cpu_timer: Optimize fastpath_timer_check()
Commit-ID:  7c177d994eb9637302b79e80d331f48dfbe26368
Gitweb:     http://git.kernel.org/tip/7c177d994eb9637302b79e80d331f48dfbe26368
Author:     Jason Low <[email protected]>
AuthorDate: Wed, 14 Oct 2015 12:07:53 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 15 Oct 2015 11:23:41 +0200

posix_cpu_timer: Optimize fastpath_timer_check()

In fastpath_timer_check(), the task_cputime() function is always called to compute the utime and stime values. However, this is not necessary if there are no per-thread timers to check for. This patch modifies the code such that we compute the task_cputime values only when there are per-thread timers set.

---
 kernel/time/posix-cpu-timers.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)
kernel/time/posix-cpu-timers.c +3 -8
index 892e3da..aa4b6f4 100644
@@ -1117,17 +1117,12 @@ static inline int task_cputime_expired(const struct task_cputime *sample,
 static inline int fastpath_timer_check(struct task_struct *tsk)
 {
 	struct signal_struct *sig;
-	cputime_t utime, stime;
-
-	task_cputime(tsk, &utime, &stime);
 
 	if (!task_cputime_zero(&tsk->cputime_expires)) {
-		struct task_cputime task_sample = {
-			.utime = utime,
-			.stime = stime,
-			.sum_exec_runtime = tsk->se.sum_exec_runtime
-		};
+		struct task_cputime task_sample;
 
+		task_cputime(tsk, &task_sample.utime, &task_sample.stime);
+		task_sample.sum_exec_runtime = tsk->se.sum_exec_runtime;
 		if (task_cputime_expired(&task_sample, &tsk->cputime_expires))
 			return 1;
 	}
author
#3 lore raw

PATCH v2 2/4 timer: Check thread timers only when there are active thread timers

In reply to Jason Low

The fastpath_timer_check() contains logic to check for if any timers are set by checking if !task_cputime_zero(). Similarly, we can do this before calling check_thread_timers(). In the case where there are only process-wide timers, this will skip all of the computations for per-thread timers when there are no per-thread timers.

As suggested by George, we can put the task_cputime_zero() check in check_thread_timers(), since that is more of an optization to the function. Similarly, we move the existing check of cputimer->running to check_process_timers().

---
 kernel/time/posix-cpu-timers.c |   22 ++++++++++++++++------
 1 files changed, 16 insertions(+), 6 deletions(-)
kernel/time/posix-cpu-timers.c +16 -6
index aa4b6f4..6f6e252 100644
@@ -864,6 +864,13 @@ static void check_thread_timers(struct task_struct *tsk,
 	unsigned long long expires;
 	unsigned long soft;
 
+	/*
+	 * If cputime_expires is zero, then there are no active
+	 * per thread CPU timers.
+	 */
+	if (task_cputime_zero(&tsk->cputime_expires))
+		return;
+
 	expires = check_timers_list(timers, firing, prof_ticks(tsk));
 	tsk_expires->prof_exp = expires_to_cputime(expires);
 
@@ -962,6 +969,13 @@ static void check_process_timers(struct task_struct *tsk,
 	unsigned long soft;
 
 	/*
+	 * If cputimer is not running, then there are no active
+	 * process wide timers (POSIX 1.b, itimers, RLIMIT_CPU).
+	 */
+	if (!READ_ONCE(tsk->signal->cputimer.running))
+		return;
+
+	/*
 	 * Collect the current process totals.
 	 */
 	thread_group_cputimer(tsk, &cputime);
@@ -1169,12 +1183,8 @@ void run_posix_cpu_timers(struct task_struct *tsk)
 	 * put them on the firing list.
 	 */
 	check_thread_timers(tsk, &firing);
-	/*
-	 * If there are any active process wide timers (POSIX 1.b, itimers,
-	 * RLIMIT_CPU) cputimer must be running.
-	 */
-	if (READ_ONCE(tsk->signal->cputimer.running))
-		check_process_timers(tsk, &firing);
+
+	check_process_timers(tsk, &firing);
 
 	/*
 	 * We must release these locks before taking any timer's lock.
Signature · 1 line
1.7.2.5
#4 lore raw

[tip:timers/core] posix_cpu_timer: Check thread timers only when there are active thread timers

posix_cpu_timer: Check thread timers only when there are active thread timers
Commit-ID:  934715a191e4be0c602d39455a7a74316f274d60
Gitweb:     http://git.kernel.org/tip/934715a191e4be0c602d39455a7a74316f274d60
Author:     Jason Low <[email protected]>
AuthorDate: Wed, 14 Oct 2015 12:07:54 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 15 Oct 2015 11:23:41 +0200

posix_cpu_timer: Check thread timers only when there are active thread timers

The fastpath_timer_check() contains logic to check for if any timers are set by checking if !task_cputime_zero(). Similarly, we can do this before calling check_thread_timers(). In the case where there are only process-wide timers, this will skip all of the computations for per-thread timers when there are no per-thread timers.

As suggested by George, we can put the task_cputime_zero() check in check_thread_timers(), since that is more of an optization to the function. Similarly, we move the existing check of cputimer->running to check_process_timers().

---
 kernel/time/posix-cpu-timers.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
kernel/time/posix-cpu-timers.c +16 -6
index aa4b6f4..6f6e252 100644
@@ -864,6 +864,13 @@ static void check_thread_timers(struct task_struct *tsk,
 	unsigned long long expires;
 	unsigned long soft;
 
+	/*
+	 * If cputime_expires is zero, then there are no active
+	 * per thread CPU timers.
+	 */
+	if (task_cputime_zero(&tsk->cputime_expires))
+		return;
+
 	expires = check_timers_list(timers, firing, prof_ticks(tsk));
 	tsk_expires->prof_exp = expires_to_cputime(expires);
 
@@ -962,6 +969,13 @@ static void check_process_timers(struct task_struct *tsk,
 	unsigned long soft;
 
 	/*
+	 * If cputimer is not running, then there are no active
+	 * process wide timers (POSIX 1.b, itimers, RLIMIT_CPU).
+	 */
+	if (!READ_ONCE(tsk->signal->cputimer.running))
+		return;
+
+	/*
 	 * Collect the current process totals.
 	 */
 	thread_group_cputimer(tsk, &cputime);
@@ -1169,12 +1183,8 @@ void run_posix_cpu_timers(struct task_struct *tsk)
 	 * put them on the firing list.
 	 */
 	check_thread_timers(tsk, &firing);
-	/*
-	 * If there are any active process wide timers (POSIX 1.b, itimers,
-	 * RLIMIT_CPU) cputimer must be running.
-	 */
-	if (READ_ONCE(tsk->signal->cputimer.running))
-		check_process_timers(tsk, &firing);
+
+	check_process_timers(tsk, &firing);
 
 	/*
 	 * We must release these locks before taking any timer's lock.
author
#5 lore raw

PATCH v2 3/4 timer: Convert cputimer->running to bool

In reply to Jason Low

In the next patch in this series, a new field 'checking_timer' will be added to 'struct thread_group_cputimer'. Both this and the existing 'running' integer field are just used as boolean values. To save space in the structure, we can make both of these fields booleans.

This is a preparatory patch to convert the existing running integer field to a boolean.

---
 include/linux/init_task.h      |    2 +-
 include/linux/sched.h          |    6 +++---
 kernel/fork.c                  |    2 +-
 kernel/time/posix-cpu-timers.c |    4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)
include/linux/init_task.h +1 -1
index e38681f..c43b80f 100644
@@ -59,7 +59,7 @@ extern struct fs_struct init_fs;
 	.rlim		= INIT_RLIMITS,					\
 	.cputimer	= { 						\
 		.cputime_atomic	= INIT_CPUTIME_ATOMIC,			\
-		.running	= 0,					\
+		.running	= false,				\
 	},								\
 	INIT_PREV_CPUTIME(sig)						\
 	.cred_guard_mutex =						\
include/linux/sched.h +3 -3
index 23ca455..35a9c46 100644
@@ -623,15 +623,15 @@ struct task_cputime_atomic {
 /**
  * struct thread_group_cputimer - thread group interval timer counts
  * @cputime_atomic:	atomic thread group interval timers.
- * @running:		non-zero when there are timers running and
- *			@cputime receives updates.
+ * @running:		true when there are timers running and
+ *			@cputime_atomic receives updates.
  *
  * This structure contains the version of task_cputime, above, that is
  * used for thread group CPU timer calculations.
  */
 struct thread_group_cputimer {
 	struct task_cputime_atomic cputime_atomic;
-	int running;
+	bool running;
 };
 
 #include <linux/rwsem.h>
kernel/fork.c +1 -1
index 83dea82..fe3f371 100644
@@ -1101,7 +1101,7 @@ static void posix_cpu_timers_init_group(struct signal_struct *sig)
 	cpu_limit = READ_ONCE(sig->rlim[RLIMIT_CPU].rlim_cur);
 	if (cpu_limit != RLIM_INFINITY) {
 		sig->cputime_expires.prof_exp = secs_to_cputime(cpu_limit);
-		sig->cputimer.running = 1;
+		sig->cputimer.running = true;
 	}
 
 	/* The timer lists. */
kernel/time/posix-cpu-timers.c +2 -2
index 6f6e252..2d58153 100644
@@ -249,7 +249,7 @@ void thread_group_cputimer(struct task_struct *tsk, struct task_cputime *times)
 		 * but barriers are not required because update_gt_cputime()
 		 * can handle concurrent updates.
 		 */
-		WRITE_ONCE(cputimer->running, 1);
+		WRITE_ONCE(cputimer->running, true);
 	}
 	sample_cputime_atomic(times, &cputimer->cputime_atomic);
 }
@@ -918,7 +918,7 @@ static inline void stop_process_timers(struct signal_struct *sig)
 	struct thread_group_cputimer *cputimer = &sig->cputimer;
 
 	/* Turn off cputimer->running. This is done without locking. */
-	WRITE_ONCE(cputimer->running, 0);
+	WRITE_ONCE(cputimer->running, false);
 }
 
 static u32 onecputick;
Signature · 1 line
1.7.2.5
#6 lore raw

[tip:timers/core] posix_cpu_timer: Convert cputimer-> running to bool

posix_cpu_timer: Convert cputimer->running to bool
Commit-ID:  d5c373eb5610686162ff50429f63f4c00c554799
Gitweb:     http://git.kernel.org/tip/d5c373eb5610686162ff50429f63f4c00c554799
Author:     Jason Low <[email protected]>
AuthorDate: Wed, 14 Oct 2015 12:07:55 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 15 Oct 2015 11:23:41 +0200

posix_cpu_timer: Convert cputimer->running to bool

In the next patch in this series, a new field 'checking_timer' will be added to 'struct thread_group_cputimer'. Both this and the existing 'running' integer field are just used as boolean values. To save space in the structure, we can make both of these fields booleans.

This is a preparatory patch to convert the existing running integer field to a boolean.

Reviewed: George Spelvin <[email protected]>

---
 include/linux/init_task.h      | 2 +-
 include/linux/sched.h          | 6 +++---
 kernel/fork.c                  | 2 +-
 kernel/time/posix-cpu-timers.c | 4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)
include/linux/init_task.h +1 -1
index e38681f..c43b80f 100644
@@ -59,7 +59,7 @@ extern struct fs_struct init_fs;
 	.rlim		= INIT_RLIMITS,					\
 	.cputimer	= { 						\
 		.cputime_atomic	= INIT_CPUTIME_ATOMIC,			\
-		.running	= 0,					\
+		.running	= false,				\
 	},								\
 	INIT_PREV_CPUTIME(sig)						\
 	.cred_guard_mutex =						\
include/linux/sched.h +3 -3
index b7b9501..6c8504a 100644
@@ -617,15 +617,15 @@ struct task_cputime_atomic {
 /**
  * struct thread_group_cputimer - thread group interval timer counts
  * @cputime_atomic:	atomic thread group interval timers.
- * @running:		non-zero when there are timers running and
- * 			@cputime receives updates.
+ * @running:		true when there are timers running and
+ *			@cputime_atomic receives updates.
  *
  * This structure contains the version of task_cputime, above, that is
  * used for thread group CPU timer calculations.
  */
 struct thread_group_cputimer {
 	struct task_cputime_atomic cputime_atomic;
-	int running;
+	bool running;
 };
 
 #include <linux/rwsem.h>
kernel/fork.c +1 -1
index 2845623..6ac8942 100644
@@ -1101,7 +1101,7 @@ static void posix_cpu_timers_init_group(struct signal_struct *sig)
 	cpu_limit = READ_ONCE(sig->rlim[RLIMIT_CPU].rlim_cur);
 	if (cpu_limit != RLIM_INFINITY) {
 		sig->cputime_expires.prof_exp = secs_to_cputime(cpu_limit);
-		sig->cputimer.running = 1;
+		sig->cputimer.running = true;
 	}
 
 	/* The timer lists. */
kernel/time/posix-cpu-timers.c +2 -2
index 6f6e252..2d58153 100644
@@ -249,7 +249,7 @@ void thread_group_cputimer(struct task_struct *tsk, struct task_cputime *times)
 		 * but barriers are not required because update_gt_cputime()
 		 * can handle concurrent updates.
 		 */
-		WRITE_ONCE(cputimer->running, 1);
+		WRITE_ONCE(cputimer->running, true);
 	}
 	sample_cputime_atomic(times, &cputimer->cputime_atomic);
 }
@@ -918,7 +918,7 @@ static inline void stop_process_timers(struct signal_struct *sig)
 	struct thread_group_cputimer *cputimer = &sig->cputimer;
 
 	/* Turn off cputimer->running. This is done without locking. */
-	WRITE_ONCE(cputimer->running, 0);
+	WRITE_ONCE(cputimer->running, false);
 }
 
 static u32 onecputick;
author
#7 lore raw

PATCH v2 4/4 timer: Reduce unnecessary sighand lock contention

In reply to Jason Low

It was found while running a database workload on large systems that significant time was spent trying to acquire the sighand lock.

The issue was that whenever an itimer expired, many threads ended up simultaneously trying to send the signal. Most of the time, nothing happened after acquiring the sighand lock because another thread had just already sent the signal and updated the "next expire" time. The fastpath_timer_check() didn't help much since the "next expire" time was updated after the threads exit fastpath_timer_check().

This patch addresses this by having the thread_group_cputimer structure maintain a boolean to signify when a thread in the group is already checking for process wide timers, and adds extra logic in the fastpath to check the boolean.

---
 include/linux/init_task.h      |    1 +
 include/linux/sched.h          |    3 +++
 kernel/time/posix-cpu-timers.c |   26 ++++++++++++++++++++++++--
 3 files changed, 28 insertions(+), 2 deletions(-)
include/linux/init_task.h +1 -0
index c43b80f..810a34f 100644
@@ -60,6 +60,7 @@ extern struct fs_struct init_fs;
 	.cputimer	= { 						\
 		.cputime_atomic	= INIT_CPUTIME_ATOMIC,			\
 		.running	= false,				\
+		.checking_timer = false,				\
 	},								\
 	INIT_PREV_CPUTIME(sig)						\
 	.cred_guard_mutex =						\
include/linux/sched.h +3 -0
index 35a9c46..0f1ebec 100644
@@ -625,6 +625,8 @@ struct task_cputime_atomic {
  * @cputime_atomic:	atomic thread group interval timers.
  * @running:		true when there are timers running and
  *			@cputime_atomic receives updates.
+ * @checking_timer:	true when a thread in the group is in the
+ *			process of checking for thread group timers.
  *
  * This structure contains the version of task_cputime, above, that is
  * used for thread group CPU timer calculations.
@@ -632,6 +634,7 @@ struct task_cputime_atomic {
 struct thread_group_cputimer {
 	struct task_cputime_atomic cputime_atomic;
 	bool running;
+	bool checking_timer;
 };
 
 #include <linux/rwsem.h>
kernel/time/posix-cpu-timers.c +24 -2
index 2d58153..957fbae 100644
@@ -975,6 +975,12 @@ static void check_process_timers(struct task_struct *tsk,
 	if (!READ_ONCE(tsk->signal->cputimer.running))
 		return;
 
+        /*
+	 * Signify that a thread is checking for process timers.
+	 * Write access to this field is protected by the sighand lock.
+	 */
+	sig->cputimer.checking_timer = true;
+
 	/*
 	 * Collect the current process totals.
 	 */
@@ -1029,6 +1035,8 @@ static void check_process_timers(struct task_struct *tsk,
 	sig->cputime_expires.sched_exp = sched_expires;
 	if (task_cputime_zero(&sig->cputime_expires))
 		stop_process_timers(sig);
+
+	sig->cputimer.checking_timer = false;
 }
 
 /*
@@ -1142,8 +1150,22 @@ static inline int fastpath_timer_check(struct task_struct *tsk)
 	}
 
 	sig = tsk->signal;
-	/* Check if cputimer is running. This is accessed without locking. */
-	if (READ_ONCE(sig->cputimer.running)) {
+	/*
+	 * Check if thread group timers expired when the cputimer is
+	 * running and no other thread in the group is already checking
+	 * for thread group cputimers. These fields are read without the
+	 * sighand lock. However, this is fine because this is meant to
+	 * be a fastpath heuristic to determine whether we should try to
+	 * acquire the sighand lock to check/handle timers.
+	 * 
+	 * In the worst case scenario, if 'running' or 'checking_timer' gets
+	 * set but the current thread doesn't see the change yet, we'll wait
+	 * until the next thread in the group gets a scheduler interrupt to
+	 * handle the timer. This isn't an issue in practice because these
+	 * types of delays with signals actually getting sent are expected.
+	 */
+	if (READ_ONCE(sig->cputimer.running) &&
+	    !READ_ONCE(sig->cputimer.checking_timer)) {
 		struct task_cputime group_sample;
 
 		sample_cputime_atomic(&group_sample, &sig->cputimer.cputime_atomic);
Signature · 1 line
1.7.2.5
#8 lore raw

[tip:timers/core] posix_cpu_timer: Reduce unnecessary sighand lock contention

posix_cpu_timer: Reduce unnecessary sighand lock contention
Commit-ID:  c8d75aa47dd585c9538a8205e9bb9847e12cfb84
Gitweb:     http://git.kernel.org/tip/c8d75aa47dd585c9538a8205e9bb9847e12cfb84
Author:     Jason Low <[email protected]>
AuthorDate: Wed, 14 Oct 2015 12:07:56 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 15 Oct 2015 11:23:41 +0200

posix_cpu_timer: Reduce unnecessary sighand lock contention

It was found while running a database workload on large systems that significant time was spent trying to acquire the sighand lock.

The issue was that whenever an itimer expired, many threads ended up simultaneously trying to send the signal. Most of the time, nothing happened after acquiring the sighand lock because another thread had just already sent the signal and updated the "next expire" time. The fastpath_timer_check() didn't help much since the "next expire" time was updated after the threads exit fastpath_timer_check().

This patch addresses this by having the thread_group_cputimer structure maintain a boolean to signify when a thread in the group is already checking for process wide timers, and adds extra logic in the fastpath to check the boolean.

---
 include/linux/init_task.h      |  1 +
 include/linux/sched.h          |  3 +++
 kernel/time/posix-cpu-timers.c | 26 ++++++++++++++++++++++++--
 3 files changed, 28 insertions(+), 2 deletions(-)
include/linux/init_task.h +1 -0
index c43b80f..810a34f 100644
@@ -60,6 +60,7 @@ extern struct fs_struct init_fs;
 	.cputimer	= { 						\
 		.cputime_atomic	= INIT_CPUTIME_ATOMIC,			\
 		.running	= false,				\
+		.checking_timer = false,				\
 	},								\
 	INIT_PREV_CPUTIME(sig)						\
 	.cred_guard_mutex =						\
include/linux/sched.h +3 -0
index 6c8504a..f87559d 100644
@@ -619,6 +619,8 @@ struct task_cputime_atomic {
  * @cputime_atomic:	atomic thread group interval timers.
  * @running:		true when there are timers running and
  *			@cputime_atomic receives updates.
+ * @checking_timer:	true when a thread in the group is in the
+ *			process of checking for thread group timers.
  *
  * This structure contains the version of task_cputime, above, that is
  * used for thread group CPU timer calculations.
@@ -626,6 +628,7 @@ struct task_cputime_atomic {
 struct thread_group_cputimer {
 	struct task_cputime_atomic cputime_atomic;
 	bool running;
+	bool checking_timer;
 };
 
 #include <linux/rwsem.h>
kernel/time/posix-cpu-timers.c +24 -2
index 2d58153..f5e86d2 100644
@@ -975,6 +975,12 @@ static void check_process_timers(struct task_struct *tsk,
 	if (!READ_ONCE(tsk->signal->cputimer.running))
 		return;
 
+        /*
+	 * Signify that a thread is checking for process timers.
+	 * Write access to this field is protected by the sighand lock.
+	 */
+	sig->cputimer.checking_timer = true;
+
 	/*
 	 * Collect the current process totals.
 	 */
@@ -1029,6 +1035,8 @@ static void check_process_timers(struct task_struct *tsk,
 	sig->cputime_expires.sched_exp = sched_expires;
 	if (task_cputime_zero(&sig->cputime_expires))
 		stop_process_timers(sig);
+
+	sig->cputimer.checking_timer = false;
 }
 
 /*
@@ -1142,8 +1150,22 @@ static inline int fastpath_timer_check(struct task_struct *tsk)
 	}
 
 	sig = tsk->signal;
-	/* Check if cputimer is running. This is accessed without locking. */
-	if (READ_ONCE(sig->cputimer.running)) {
+	/*
+	 * Check if thread group timers expired when the cputimer is
+	 * running and no other thread in the group is already checking
+	 * for thread group cputimers. These fields are read without the
+	 * sighand lock. However, this is fine because this is meant to
+	 * be a fastpath heuristic to determine whether we should try to
+	 * acquire the sighand lock to check/handle timers.
+	 *
+	 * In the worst case scenario, if 'running' or 'checking_timer' gets
+	 * set but the current thread doesn't see the change yet, we'll wait
+	 * until the next thread in the group gets a scheduler interrupt to
+	 * handle the timer. This isn't an issue in practice because these
+	 * types of delays with signals actually getting sent are expected.
+	 */
+	if (READ_ONCE(sig->cputimer.running) &&
+	    !READ_ONCE(sig->cputimer.checking_timer)) {
 		struct task_cputime group_sample;
 
 		sample_cputime_atomic(&group_sample, &sig->cputimer.cputime_atomic);
Reviewed-by on the v2 series
#9 lore raw
In reply to Jason Low

I'm going to give 4/4 a closer look to see if the races with timer expiration make more sense to me than last time around. (E.g. do CPU time signals even work in CONFIG_NO_HZ_FULL?)

But although I haven't yet convinced myself the current code is right, the changes don't seem to make it any worse. So consider all four

Thank you!

#10 lore raw
Quoted 3 lines · George Spelvin

I'm going to give 4/4 a closer look to see if the races with timer expiration make more sense to me than last time around. (E.g. do CPU time signals even work in CONFIG_NO_HZ_FULL?)

Those enqueued with timer_settime() do work. But itimers, and rlimits (RLIMIT_RTTIME, RLIMIT_CPU) aren't supported well. I need to rework that.

Quoted 7 lines

But although I haven't yet convinced myself the current code is right, the changes don't seem to make it any worse. So consider all four

Thank you!

#11 lore raw
In reply to George Spelvin
Quoted 8 lines · George Spelvin

I'm going to give 4/4 a closer look to see if the races with timer expiration make more sense to me than last time around. (E.g. do CPU time signals even work in CONFIG_NO_HZ_FULL?)

But although I haven't yet convinced myself the current code is right, the changes don't seem to make it any worse. So consider all four

Thanks George!

#12 lore raw
In reply to Jason Low
Quoted 23 lines · Jason Low

While running a database workload on a 16 socket machine, there were scalability issues related to itimers. The following link contains a more detailed summary of the issues at the application level.

https://lkml.org/lkml/2015/8/26/737

Commit 1018016c706f addressed the issue with the thread_group_cputimer spinlock taking up a significant portion of total run time. This patch series addresses the secondary issue where a lot of time is spent trying to acquire the sighand lock. It was found in some cases that 200+ threads were simultaneously contending for the same sighand lock, reducing throughput by more than 30%.

With this patch set (along with commit 1018016c706f mentioned above), the performance hit of itimers almost completely goes away on the 16 socket system.

Jason Low (4):
timer: Optimize fastpath_timer_check()
timer: Check thread timers only when there are active thread timers
timer: Convert cputimer->running to bool
timer: Reduce unnecessary sighand lock contention

Quoted 5 lines
 include/linux/init_task.h      |    3 +-
 include/linux/sched.h          |    9 ++++--
 kernel/fork.c                  |    2 +-
 kernel/time/posix-cpu-timers.c |   63 ++++++++++++++++++++++++++++-----------
 4 files changed, 54 insertions(+), 23 deletions(-)

Is there some itimers benchmark that can be used to measure the effects of these changes?

Thanks,

Ingo

#13 lore raw
Quoted 25 lines · Ingo Molnar
Quoted 23 lines · Jason Low

While running a database workload on a 16 socket machine, there were scalability issues related to itimers. The following link contains a more detailed summary of the issues at the application level.

https://lkml.org/lkml/2015/8/26/737

Commit 1018016c706f addressed the issue with the thread_group_cputimer spinlock taking up a significant portion of total run time. This patch series addresses the secondary issue where a lot of time is spent trying to acquire the sighand lock. It was found in some cases that 200+ threads were simultaneously contending for the same sighand lock, reducing throughput by more than 30%.

With this patch set (along with commit 1018016c706f mentioned above), the performance hit of itimers almost completely goes away on the 16 socket system.

Jason Low (4):
timer: Optimize fastpath_timer_check()
timer: Check thread timers only when there are active thread timers
timer: Convert cputimer->running to bool
timer: Reduce unnecessary sighand lock contention

Quoted 8 lines
Quoted 5 lines
 include/linux/init_task.h      |    3 +-
 include/linux/sched.h          |    9 ++++--
 kernel/fork.c                  |    2 +-
 kernel/time/posix-cpu-timers.c |   63 ++++++++++++++++++++++++++++-----------
 4 files changed, 54 insertions(+), 23 deletions(-)

Is there some itimers benchmark that can be used to measure the effects of these changes?

Yes, we also wrote a micro benchmark which generates cache misses and measures the average cost of each cache miss (with itimers enabled). We used this while writing and testing patches, since it takes a bit longer to set up and run the database.

Jason

#14 lore raw
Quoted 16 lines · Jason Low
Quoted 16 lines
Quoted 15 lines

With this patch set (along with commit 1018016c706f mentioned above), the performance hit of itimers almost completely goes away on the 16 socket system.

Jason Low (4):
timer: Optimize fastpath_timer_check()
timer: Check thread timers only when there are active thread timers
timer: Convert cputimer->running to bool
timer: Reduce unnecessary sighand lock contention

 include/linux/init_task.h      |    3 +-
 include/linux/sched.h          |    9 ++++--
 kernel/fork.c                  |    2 +-
 kernel/time/posix-cpu-timers.c |   63 ++++++++++++++++++++++++++++-----------
 4 files changed, 54 insertions(+), 23 deletions(-)
Quoted 7 lines
Quoted 2 lines

Is there some itimers benchmark that can be used to measure the effects of these changes?

Yes, we also wrote a micro benchmark which generates cache misses and measures the average cost of each cache miss (with itimers enabled). We used this while writing and testing patches, since it takes a bit longer to set up and run the database.

Mind posting it, so that people can stick it into a new 'perf bench timer' subcommand, and/or reproduce your results with it?

Thanks,

Ingo

#15 lore raw
Quoted 21 lines · Ingo Molnar
Quoted 19 lines · Jason Low
Quoted 18 lines
Quoted 10 lines

With this patch set (along with commit 1018016c706f mentioned above), the performance hit of itimers almost completely goes away on the 16 socket system.

Jason Low (4):
timer: Optimize fastpath_timer_check()
timer: Check thread timers only when there are active thread timers
timer: Convert cputimer->running to bool
timer: Reduce unnecessary sighand lock contention

Quoted 5 lines
 include/linux/init_task.h      |    3 +-
 include/linux/sched.h          |    9 ++++--
 kernel/fork.c                  |    2 +-
 kernel/time/posix-cpu-timers.c |   63 ++++++++++++++++++++++++++++-----------
 4 files changed, 54 insertions(+), 23 deletions(-)

Is there some itimers benchmark that can be used to measure the effects of these changes?

Quoted 7 lines
Quoted 4 lines

Yes, we also wrote a micro benchmark which generates cache misses and measures the average cost of each cache miss (with itimers enabled). We used this while writing and testing patches, since it takes a bit longer to set up and run the database.

Mind posting it, so that people can stick it into a new 'perf bench timer' subcommand, and/or reproduce your results with it?

Yes, sure. At the moment, this micro benchmark is written in C++ and integrated with the database code. We can look into rewriting it into a more general program so that it can be included in perf.

Thanks,
Jason

#16 lore raw

Removing dependency to the database code is trivial. It's just 100 lines that launch lots of threads and do NUMA-aware memory accesses so that remote NUMA access cost does not affect the benchmark.

It's just a bit tedious to convert the C++11 code into C/pthread. C++11 really spoiled me.
Still, not much work. Let me know where to post the code.

Quoted 10 lines · Jason Low
Quoted 2 lines

Mind posting it, so that people can stick it into a new 'perf bench timer' subcommand, and/or reproduce your results with it?

Yes, sure. At the moment, this micro benchmark is written in C++ and integrated with the database code. We can look into rewriting it into a more general program so that it can be included in perf.

Thanks,
Jason

Signature · 1 line
Hideaki Kimura