clockevents: Per cpu tick skew boot option

PATCH Superseded Mike Galbraith 4 participants 9 messages May 6, 2012 – May 24, 2012

Subsystems (2)

Documentation Nohz, dynticks support

lore

Share & tools

lore

Mike Galbraith

author

2012-05-06 12:58 UTC #0 lore raw

PATCH 1/1 clockevents: Per cpu tick skew boot option

Let the user decide whether power consumption or jitter is the more important consideration for their machines.

Quoting removal commit af5ab277ded04bd9bc6b048c5a2f0e7d70ef0867 Historically, Linux has tried to make the regular timer tick on the various CPUs not happen at the same time, to avoid contention on xtime_lock.

Nowadays, with the tickless kernel, this contention no longer happens since time keeping and updating are done differently. In addition, this skew is actually hurting power consumption in a measurable way on many-core systems.
End quote

Problems:

- Contrary to the above, systems do encounter contention on both xtime_lock and RCU structure locks when the tick is synchronized.

- Moderate sized RT systems suffer intolerable jitter due to the tick being synchronized.

- SGI reports the same for their large systems.

- Fully utilized systems reap no power saving benefit from skew removal, but do suffer from resulting induced lock contention.

Signed-off-by Mike Galbraith <[email protected]>

---
 Documentation/kernel-parameters.txt |    5 +++++
 kernel/time/tick-sched.c            |   19 +++++++++++++++++++
 2 files changed, 24 insertions(+)

Documentation/kernel-parameters.txt +5 -0

@@ -2426,6 +2426,11 @@ bytes respectively. Such letter suffixes
 
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems.  Note: increases
+			power consumption, and should only be enabled if running
+			jitter sensitive (HPC/RT) workloads.
+
 	security=	[SECURITY] Choose a security module to enable at boot.
 			If this boot parameter is not specified, only the first
 			security module asking for security registration will be

kernel/time/tick-sched.c +19 -0

@@ -814,6 +814,8 @@ static enum hrtimer_restart tick_sched_t
 	return HRTIMER_RESTART;
 }
 
+static int sched_skew_tick;
+
 /**
  * tick_setup_sched_timer - setup the tick emulation timer
  */

@@ -831,6 +833,14 @@ void tick_setup_sched_timer(void)
 	/* Get the next period (per cpu) */
 	hrtimer_set_expires(&ts->sched_timer, tick_init_jiffy_update());
 
+	/* Offset the tick to avert xtime_lock contention. */
+	if (sched_skew_tick) {
+		u64 offset = ktime_to_ns(tick_period) >> 1;
+		do_div(offset, num_possible_cpus());
+		offset *= smp_processor_id();
+		hrtimer_add_expires_ns(&ts->sched_timer, offset);
+	}
+
 	for (;;) {
 		hrtimer_forward(&ts->sched_timer, now, tick_period);
 		hrtimer_start_expires(&ts->sched_timer,

@@ -910,3 +920,12 @@ int tick_check_oneshot_change(int allow_
 	tick_nohz_switch_to_nohz();
 	return 0;
 }
+
+static int __init skew_tick(char *str)
+{
+	get_option(&str, &sched_skew_tick);
+
+	return 0;
+}
+early_param("skew_tick", skew_tick);
+

Mike Galbraith

2012-05-06 13:10 UTC #1 lore raw

Quoted 4 lines · Mike Galbraith

- 0209f649 rcu: limit rcu_node leaf-level fanout
This patch was born to combat lock contention which testing showed to have been _induced by_ skew removal. Skew the tick, contention disappeared virtually completely.

P.S. This boot option patch will complement a proposed patch by Paul McKenney to allow boot time fanout configuration. Between the two, users will then be able to boot time select the tradeoff they need for optimal performance of their systems.

-Mike

Thomas Gleixner

2012-05-07 19:17 UTC #2 lore raw

In reply to Mike Galbraith

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes · line 2433

Quoted 6 lines · Mike Galbraith

Documentation/kernel-parameters.txt +5 -0

+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems.  Note: increases
+			power consumption, and should only be enabled if running
+			jitter sensitive (HPC/RT) workloads.
+

The "=" is wrong as skew_tick should not take parameters. It's disabled by default. So "skew_tick" simply enables it, right ?

Mike Galbraith

author

2012-05-08 03:20 UTC #3 lore raw

Quoted 10 lines · Thomas Gleixner

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes · line 2433
Quoted 6 lines · Mike Galbraith
Documentation/kernel-parameters.txt +5 -0
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems.  Note: increases
+			power consumption, and should only be enabled if running
+			jitter sensitive (HPC/RT) workloads.
+
The "=" is wrong as skew_tick should not take parameters. It's disabled by default. So "skew_tick" simply enables it, right ?

Unless as I have RT set up, it's turned on by default, so '=' lets the user turn it back off.

-Mike

Thomas Gleixner

2012-05-08 09:44 UTC #4 lore raw

Quoted 12 lines · Mike Galbraith

Quoted 10 lines · Thomas Gleixner
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes · line 2433
Quoted 6 lines · Mike Galbraith
Documentation/kernel-parameters.txt +5 -0
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems.  Note: increases
+			power consumption, and should only be enabled if running
+			jitter sensitive (HPC/RT) workloads.
+
The "=" is wrong as skew_tick should not take parameters. It's disabled by default. So "skew_tick" simply enables it, right ?

Quoted 2 lines

Unless as I have RT set up, it's turned on by default, so '=' lets the user turn it back off.

Then the doc should say what's the parameter after the "+" is :)

Mike Galbraith

author

2012-05-08 10:21 UTC #5 lore raw

Quoted 14 lines · Thomas Gleixner

Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Thomas Gleixner
Review
Quoted 6 lines · Mike Galbraith
(quoted diff) +5 -0
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems.  Note: increases
+			power consumption, and should only be enabled if running
+			jitter sensitive (HPC/RT) workloads.
+
The "=" is wrong as skew_tick should not take parameters. It's disabled by default. So "skew_tick" simply enables it, right ?

Quoted 4 lines

Quoted 2 lines
Unless as I have RT set up, it's turned on by default, so '=' lets the user turn it back off.
Then the doc should say what's the parameter after the "+" is :)

I only put anything there because boss said "Document", I was hiding it along with fugly but damn useful <koff> HPC/RT cpuset patch ;-)

Let the user decide whether power consumption or jitter is the more important consideration for their machines.

Problems:

- Contrary to the above, systems do encounter contention on both xtime_lock and RCU structure locks when the tick is synchronized.

- Moderate sized RT systems suffer intolerable jitter due to the tick being synchronized.

- SGI reports the same for their large systems.

- Fully utilized systems reap no power saving benefit from skew removal, but do suffer from resulting induced lock contention.

Signed-off-by Mike Galbraith <[email protected]>

---
 Documentation/kernel-parameters.txt |    9 +++++++++
 kernel/time/tick-sched.c            |   19 +++++++++++++++++++
 2 files changed, 28 insertions(+)

Documentation/kernel-parameters.txt +9 -0

@@ -2426,6 +2426,15 @@ bytes respectively. Such letter suffixes
 
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems, and/or RCU lock
+			contention on all systems with CONFIG_MAXSMP set.
+			Format: { "0" | "1" }
+			0 -- disable. (may be 1 via CONFIG_CMDLINE="skew_tick=1"
+			1 -- enable.
+			Note: increases power consumption, thus should only be
+			enabled if running jitter sensitive (HPC/RT) workloads.
+
 	security=	[SECURITY] Choose a security module to enable at boot.
 			If this boot parameter is not specified, only the first
 			security module asking for security registration will be

kernel/time/tick-sched.c +19 -0

@@ -814,6 +814,8 @@ static enum hrtimer_restart tick_sched_t
 	return HRTIMER_RESTART;
 }
 
+static int sched_skew_tick;
+
 /**
  * tick_setup_sched_timer - setup the tick emulation timer
  */

@@ -831,6 +833,14 @@ void tick_setup_sched_timer(void)
 	/* Get the next period (per cpu) */
 	hrtimer_set_expires(&ts->sched_timer, tick_init_jiffy_update());
 
+	/* Offset the tick to avert xtime_lock contention. */
+	if (sched_skew_tick) {
+		u64 offset = ktime_to_ns(tick_period) >> 1;
+		do_div(offset, num_possible_cpus());
+		offset *= smp_processor_id();
+		hrtimer_add_expires_ns(&ts->sched_timer, offset);
+	}
+
 	for (;;) {
 		hrtimer_forward(&ts->sched_timer, now, tick_period);
 		hrtimer_start_expires(&ts->sched_timer,

@@ -910,3 +920,12 @@ int tick_check_oneshot_change(int allow_
 	tick_nohz_switch_to_nohz();
 	return 0;
 }
+
+static int __init skew_tick(char *str)
+{
+	get_option(&str, &sched_skew_tick);
+
+	return 0;
+}
+early_param("skew_tick", skew_tick);
+

Paul E. McKenney

Acked-by

2012-05-10 18:17 UTC #6 lore raw

Quoted 65 lines · Mike Galbraith

Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Thomas Gleixner
Review
Quoted 6 lines · Mike Galbraith
(quoted diff) +5 -0
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems.  Note: increases
+			power consumption, and should only be enabled if running
+			jitter sensitive (HPC/RT) workloads.
+
The "=" is wrong as skew_tick should not take parameters. It's disabled by default. So "skew_tick" simply enables it, right ?
Quoted 4 lines
Quoted 2 lines
Unless as I have RT set up, it's turned on by default, so '=' lets the user turn it back off.
Then the doc should say what's the parameter after the "+" is :)
I only put anything there because boss said "Document", I was hiding it along with fugly but damn useful <koff> HPC/RT cpuset patch ;-)
Let the user decide whether power consumption or jitter is the more important consideration for their machines.
Quoting removal commit af5ab277ded04bd9bc6b048c5a2f0e7d70ef0867 Historically, Linux has tried to make the regular timer tick on the various CPUs not happen at the same time, to avoid contention on xtime_lock.
Nowadays, with the tickless kernel, this contention no longer happens since time keeping and updating are done differently. In addition, this skew is actually hurting power consumption in a measurable way on many-core systems.
End quote
Problems:
- Contrary to the above, systems do encounter contention on both xtime_lock and RCU structure locks when the tick is synchronized.
- Moderate sized RT systems suffer intolerable jitter due to the tick being synchronized.
- SGI reports the same for their large systems.
- Fully utilized systems reap no power saving benefit from skew removal, but do suffer from resulting induced lock contention.
- 0209f649 rcu: limit rcu_node leaf-level fanout
This patch was born to combat lock contention which testing showed to have been _induced by_ skew removal. Skew the tick, contention disappeared virtually completely.
Signed-off-by Mike Galbraith <[email protected]>
---
 Documentation/kernel-parameters.txt |    9 +++++++++
 kernel/time/tick-sched.c            |   19 +++++++++++++++++++
 2 files changed, 28 insertions(+)

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes · line 2431

Quoted 6 lines

Documentation/kernel-parameters.txt +3 -0

@@ -2426,6 +2426,15 @@ bytes respectively. Such letter suffixes
 
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems, and/or RCU lock
+			contention on all systems with CONFIG_MAXSMP set.

Suggest instead:

contention on systems with large CONFIG_RCU_FANOUT values.

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes · line 2433

Quoted 2 lines

Documentation/kernel-parameters.txt +2 -0

+			Format: { "0" | "1" }
+			0 -- disable. (may be 1 via CONFIG_CMDLINE="skew_tick=1"

Suggest simply:

0 -- disable (default for typical kernel builds).

With these changes:

Acked-by Paul E. McKenney <[email protected]>

Quoted 54 lines

Documentation/kernel-parameters.txt +4 -0

+			1 -- enable.
+			Note: increases power consumption, thus should only be
+			enabled if running jitter sensitive (HPC/RT) workloads.
+
 	security=	[SECURITY] Choose a security module to enable at boot.
 			If this boot parameter is not specified, only the first
 			security module asking for security registration will be

kernel/time/tick-sched.c +19 -1

@@ -814,6 +814,8 @@ static enum hrtimer_restart tick_sched_t
 	return HRTIMER_RESTART;
 }
 
+static int sched_skew_tick;
+
 /**
  * tick_setup_sched_timer - setup the tick emulation timer
  */

@@ -831,6 +833,14 @@ void tick_setup_sched_timer(void)
 	/* Get the next period (per cpu) */
 	hrtimer_set_expires(&ts->sched_timer, tick_init_jiffy_update());
 
+	/* Offset the tick to avert xtime_lock contention. */
+	if (sched_skew_tick) {
+		u64 offset = ktime_to_ns(tick_period) >> 1;
+		do_div(offset, num_possible_cpus());
+		offset *= smp_processor_id();
+		hrtimer_add_expires_ns(&ts->sched_timer, offset);
+	}
+
 	for (;;) {
 		hrtimer_forward(&ts->sched_timer, now, tick_period);
 		hrtimer_start_expires(&ts->sched_timer,

@@ -910,3 +920,12 @@ int tick_check_oneshot_change(int allow_
 	tick_nohz_switch_to_nohz();
 	return 0;
 }
+
+static int __init skew_tick(char *str)
+{
+	get_option(&str, &sched_skew_tick);
+
+	return 0;
+}
+early_param("skew_tick", skew_tick);
+
 
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Mike Galbraith

author

2012-05-23 15:23 UTC #7 lore raw

PATCH v3 1/1 clockevents: Per cpu tick skew boot option

Quoted 20 lines · Paul E. McKenney

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes · line 2431
Quoted 9 lines
Documentation/kernel-parameters.txt +3 -0
@@ -2426,6 +2426,15 @@ bytes respectively. Such letter suffixes
 
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems, and/or RCU lock
+			contention on all systems with CONFIG_MAXSMP set.
Suggest instead:
contention on systems with large CONFIG_RCU_FANOUT values.
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes · line 2433
Quoted 2 lines
Documentation/kernel-parameters.txt +2 -0
+			Format: { "0" | "1" }
+			0 -- disable. (may be 1 via CONFIG_CMDLINE="skew_tick=1"
Suggest simply:

Quoted 5 lines

0 -- disable (default for typical kernel builds).
With these changes:
Acked-by Paul E. McKenney <[email protected]>

Hunted down round-tuit.

clockevents: Per cpu tick skew boot option

Problems:
- Contrary to the above, all systems do encounter contention on xtime_lock and RCU structure locks when the tick is synchronized.

- Large systems and moderate sized RT systems suffer intolerable jitter with the tick synchronized.

- Fully utilized systems reap no power saving benefit, but do suffer from synchronized tick lock contention.

- 0209f649 rcu: limit rcu_node leaf-level fanout
This patch was born to combat lock contention which testing showed to have been _induced by_ skew removal. Skew the tick, contention disappeared virtually completely. Measured latency on 48 core box was >330us. Revert, amd restore skew, it dropped back to ~70us. We absorbed a 400% latency increase to combat induced contention.

Let the user decide whether power consumption or jitter is the more important consideration for their machines.

Signed-off-by Mike Galbraith <[email protected]>

Acked-by Paul E. McKenney <[email protected]>

---
 Documentation/kernel-parameters.txt |    9 +++++++++
 kernel/time/tick-sched.c            |   19 +++++++++++++++++++
 2 files changed, 28 insertions(+)

Documentation/kernel-parameters.txt +9 -0

@@ -2426,6 +2426,15 @@ bytes respectively. Such letter suffixes
 
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems, and/or RCU lock
+			contention on systems with large CONFIG_RCU_FANOUT values.
+			Format: { "0" | "1" }
+			0 -- disable (default for typical kernel builds).
+			1 -- enable.
+			Note: increases power consumption, thus should only be
+			enabled if running jitter sensitive (HPC/RT) workloads.
+
 	security=	[SECURITY] Choose a security module to enable at boot.
 			If this boot parameter is not specified, only the first
 			security module asking for security registration will be

kernel/time/tick-sched.c +19 -0

@@ -814,6 +814,8 @@ static enum hrtimer_restart tick_sched_t
 	return HRTIMER_RESTART;
 }
 
+static int sched_skew_tick;
+
 /**
  * tick_setup_sched_timer - setup the tick emulation timer
  */

@@ -831,6 +833,14 @@ void tick_setup_sched_timer(void)
 	/* Get the next period (per cpu) */
 	hrtimer_set_expires(&ts->sched_timer, tick_init_jiffy_update());
 
+	/* Offset the tick to avert xtime_lock contention. */
+	if (sched_skew_tick) {
+		u64 offset = ktime_to_ns(tick_period) >> 1;
+		do_div(offset, num_possible_cpus());
+		offset *= smp_processor_id();
+		hrtimer_add_expires_ns(&ts->sched_timer, offset);
+	}
+
 	for (;;) {
 		hrtimer_forward(&ts->sched_timer, now, tick_period);
 		hrtimer_start_expires(&ts->sched_timer,

@@ -910,3 +920,12 @@ int tick_check_oneshot_change(int allow_
 	tick_nohz_switch_to_nohz();
 	return 0;
 }
+
+static int __init skew_tick(char *str)
+{
+	get_option(&str, &sched_skew_tick);
+
+	return 0;
+}
+early_param("skew_tick", skew_tick);
+

tip-bot for Mike Galbraith

author

2012-05-24 23:52 UTC #8 lore raw

[tip:timers/core] tick: Add tick skew boot option

In reply to Mike Galbraith

Commit-ID:  5307c9556bc17e3cd26d4e94fc3b2565921834de
Gitweb:     http://git.kernel.org/tip/5307c9556bc17e3cd26d4e94fc3b2565921834de
Author:     Mike Galbraith <[email protected]>
AuthorDate: Tue, 8 May 2012 12:20:58 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 25 May 2012 01:44:50 +0200

tick: Add tick skew boot option

Let the user decide whether power consumption or jitter is the more important consideration for their machines.

Quoting removal commit af5ab277ded04bd9bc6b048c5a2f0e7d70ef0867:

"Historically, Linux has tried to make the regular timer tick on the
various CPUs not happen at the same time, to avoid contention on xtime_lock.

Problems:

- Contrary to the above, systems do encounter contention on both xtime_lock and RCU structure locks when the tick is synchronized.

- Moderate sized RT systems suffer intolerable jitter due to the tick being synchronized.

- SGI reports the same for their large systems.

- Fully utilized systems reap no power saving benefit from skew removal, but do suffer from resulting induced lock contention.

Signed-off-by Mike Galbraith <[email protected]>

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Thomas Gleixner <[email protected]>

---
 Documentation/kernel-parameters.txt |    9 +++++++++
 kernel/time/tick-sched.c            |   18 ++++++++++++++++++
 2 files changed, 27 insertions(+), 0 deletions(-)

Documentation/kernel-parameters.txt +9 -0

index b69cfdc12..ea38cd1 100644
@@ -2532,6 +2532,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
+	skew_tick=	[KNL] Offset the periodic timer tick per cpu to mitigate
+			xtime_lock contention on larger systems, and/or RCU lock
+			contention on all systems with CONFIG_MAXSMP set.
+			Format: { "0" | "1" }
+			0 -- disable. (may be 1 via CONFIG_CMDLINE="skew_tick=1"
+			1 -- enable.
+			Note: increases power consumption, thus should only be
+			enabled if running jitter sensitive (HPC/RT) workloads.
+
 	security=	[SECURITY] Choose a security module to enable at boot.
 			If this boot parameter is not specified, only the first
 			security module asking for security registration will be

kernel/time/tick-sched.c +18 -0

index 6a3a5b9..4eddbb5 100644
@@ -814,6 +814,8 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)
 	return HRTIMER_RESTART;
 }
 
+static int sched_skew_tick;
+
 /**
  * tick_setup_sched_timer - setup the tick emulation timer
  */

@@ -831,6 +833,14 @@ void tick_setup_sched_timer(void)
 	/* Get the next period (per cpu) */
 	hrtimer_set_expires(&ts->sched_timer, tick_init_jiffy_update());
 
+	/* Offset the tick to avert xtime_lock contention. */
+	if (sched_skew_tick) {
+		u64 offset = ktime_to_ns(tick_period) >> 1;
+		do_div(offset, num_possible_cpus());
+		offset *= smp_processor_id();
+		hrtimer_add_expires_ns(&ts->sched_timer, offset);
+	}
+
 	for (;;) {
 		hrtimer_forward(&ts->sched_timer, now, tick_period);
 		hrtimer_start_expires(&ts->sched_timer,

@@ -910,3 +920,11 @@ int tick_check_oneshot_change(int allow_nohz)
 	tick_nohz_switch_to_nohz();
 	return 0;
 }
+
+static int __init skew_tick(char *str)
+{
+	get_option(&str, &sched_skew_tick);
+
+	return 0;
+}
+early_param("skew_tick", skew_tick);