Skip to contentSkip to conversation

Full dynticks, CONFIG_NO_HZ_FULL feature

GIT PULL Ingo Molnar 4 participants 8 messages May 5, 2013 – May 8, 2013

Subsystems (11)
lore
Share & tools
lore

Ingo Molnar

author maintainer
#0 lore raw

Linus,

Please consider pulling the latest timers-nohz-for-linus git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers-nohz-for-linus

HEAD: 265f22a975c1e4cc3a4d1f94a3ec53ffbb6f5b9f sched: Keep at least 1 tick per second for active dynticks tasks

This tree from Frederic Weisbecker adds a new, (exciting! :-) core kernel feature to the timer and scheduler subsystems: 'full dynticks', or CONFIG_NO_HZ_FULL=y.

This feature extends the nohz variable-size timer tick feature from idle to busy CPUs (running at most one task) as well, potentially reducing the number of timer interrupts significantly.

This feature got motivated by real-time folks and the -rt tree, but the general utility and motivation of full-dynticks runs wider than that:

- HPC workloads get faster: CPUs running a single task should be able to utilize a maximum amount of CPU power. A periodic timer tick at HZ=1000 can cause a constant overhead of up to 1.0%. This feature removes that overhead - and speeds up the system by 0.5%-1.0% on typical distro configs even on modern systems.

- Real-time workload latency reduction: CPUs running critical tasks should experience as little jitter as possible. The last remaining source of kernel-related jitter was the periodic timer tick.

- A single task executing on a CPU is a pretty common situation, especially with an increasing number of cores/CPUs, so this feature helps desktop and mobile workloads as well.

The cost of the feature is mainly related to increased timer-reprogramming overhead when a CPU switches its tick period, and thus slightly longer to-idle and from-idle latency.

Configuration-wise a third mode of operation is added to the existing two NOHZ kconfig modes:

- CONFIG_HZ_PERIODIC: [formerly !CONFIG_NO_HZ], now explicitly named as a config option. This is the traditional Linux periodic tick design:
there's a HZ tick going on all the time, regardless of whether a CPU is idle or not.

- CONFIG_NO_HZ_IDLE: [formerly CONFIG_NO_HZ=y], this turns off the periodic tick when a CPU enters idle mode.

- CONFIG_NO_HZ_FULL: this new mode, in addition to turning off the tick when a CPU is idle, also slows the tick down to 1 Hz (one timer interrupt per second) when only a single task is running on a CPU.

The .config behavior is compatible: existing !CONFIG_NO_HZ and CONFIG_NO_HZ=y settings get translated to the new values, without the user having to configure anything. CONFIG_NO_HZ_FULL is turned off by default.

This feature is based on a lot of infrastructure work that has been steadily going upstream in the last 2-3 cycles: related RCU support and non-periodic cputime support in particular is upstream already.

This tree adds the final pieces and activates the feature. The pull request is marked RFC because:

- it's marked 64-bit only at the moment - the 32-bit support patch is small but did not get ready in time.

- it has a number of fresh commits that came in after the merge window. The overwhelming majority of commits are from before the merge window, but still some aspects of the tree are fresh and so I marked it RFC.

- it's a pretty wide-reaching feature with lots of effects - and while the components have been in testing for some time, the full combination is still not very widely used. That it's default-off should reduce its regression abilities and obviously there are no known regressions with CONFIG_NO_HZ_FULL=y enabled either.

- the feature is not completely idempotent: there is no 100% equivalent replacement for a periodic scheduler/timer tick. In particular there's ongoing work to map out and reduce its effects on scheduler load-balancing and statistics. This should not impact correctness though, there are no known regressions related to this feature at this point.

- it's a pretty ambitious feature that with time will likely be enabled by most Linux distros, and we'd like you to make input on its design/implementation, if you dislike some aspect we missed. Without flaming us to crisp! :-)

Future plans:

- there's ongoing work to reduce 1Hz to 0Hz, to essentially shut off the periodic tick altogether when there's a single busy task on a CPU. We'd first like 1 Hz to be exposed more widely before we go for the 0 Hz target though.

- once we reach 0 Hz we can and remove the periodic tick assumption from nr_running>=2 as well, by essentially interrupting busy tasks only as frequently as the sched_latency constraints require us to do - once every 4-40 msecs, depending on nr_running.

I am personally leaning towards biting the bullet and doing this in v3.10, like the -rt tree this effort has been going on for too long - but the final word is up to you as usual.

More technical details can be found in Documentation/timers/NO_HZ.txt.

Thanks,

Ingo

------------------>
Frederic Weisbecker (34):
nohz: Basic full dynticks interface
nohz: Assign timekeeping duty to a CPU outside the full dynticks range
nohz: Wake up full dynticks CPUs when a timer gets enqueued
nohz: Unhide full dynticks feature from its dependencies
nohz: Rename CONFIG_NO_HZ to CONFIG_NO_HZ_COMMON
nohz: Pack nohz Kconfig option in a menu of choices
nohz: Print final full dynticks CPUs range on boot
nohz: Fix old dynticks idle Kconfig backward compatibility
nohz: Switch from "extended nohz" to "full nohz" based naming
nohz: Align periodic tick Kconfig with other choices' naming convention
nohz: Improve a bit the full dynticks Kconfig documentation
nohz: New APIs to re-evaluate the tick on full dynticks CPUs
nohz: Force boot CPU outside full dynticks range
nohz: Ensure full dynticks CPUs are RCU nocbs
nohz: New option to default all CPUs in full dynticks range
posix_timers: Kick full dynticks CPUs when a posix cpu timer is armed
posix_timers: New API to prevent from stopping the tick when timers are running
posix_timers: Fix pre-condition to stop the tick on full dynticks
perf: Kick full dynticks CPU if events rotation is needed
perf: New helper to prevent full dynticks CPUs from stopping tick
sched: Kick full dynticks CPU that have more than one task enqueued.
sched: New helper to prevent from stopping the tick in full dynticks
nohz: Re-evaluate the tick from the scheduler IPI
nohz: Implement full dynticks kick
nohz: Prepare to stop the tick on irq exit
nohz: Re-evaluate the tick for the new task after a context switch
nohz: Disable the tick when irq resume in full dynticks CPU
nohz: Select wide RCU nocb for full dynticks
nohz: Add basic tracing
nohz: Fix unavailable tick_stop tracepoint in dynticks idle
nohz: Remove full dynticks' superfluous dependency on RCU tree
nohz: Select VIRT_CPU_ACCOUNTING_GEN from full dynticks config
rcu: Fix full dynticks' dependency on wide RCU nocb mode
sched: Keep at least 1 tick per second for active dynticks tasks

Ingo Molnar (1):
nohz: Reduce overhead under high-freq idling patterns

Kevin Hilman (1):
cputime_nsecs: use math64.h for nsec resolution conversion helpers

Li Zhong (1):
nohz: Protect smp_processor_id() in tick_nohz_task_switch()

Paul E. McKenney (2):
rcu: Kick adaptive-ticks CPUs that are holding up RCU grace periods
nohz_full: Add documentation.

 Documentation/RCU/stallwarn.txt         |   2 +-
 Documentation/cpu-freq/governors.txt    |   4 +-
 Documentation/kernel-parameters.txt     |   8 +
 Documentation/timers/NO_HZ.txt          | 273 +++++++++++++++++++++++++++++
 arch/um/include/shared/common-offsets.h |   4 +-
 arch/um/os-Linux/time.c                 |   2 +-
 include/asm-generic/cputime_nsecs.h     |  28 ++-
 include/linux/perf_event.h              |   6 +
 include/linux/posix-timers.h            |   2 +
 include/linux/rcupdate.h                |   7 +
 include/linux/sched.h                   |  19 +-
 include/linux/tick.h                    |  25 ++-
 include/trace/events/timer.h            |  21 +++
 init/Kconfig                            |  12 +-
 init/main.c                             |   1 +
 kernel/events/core.c                    |  17 +-
 kernel/hrtimer.c                        |   4 +-
 kernel/posix-cpu-timers.c               |  76 ++++++--
 kernel/rcutree.c                        |  16 +-
 kernel/rcutree.h                        |   2 +-
 kernel/rcutree_plugin.h                 |  29 +++-
 kernel/sched/core.c                     |  92 ++++++++--
 kernel/sched/fair.c                     |  10 +-
 kernel/sched/idle_task.c                |   1 +
 kernel/sched/sched.h                    |  25 ++-
 kernel/softirq.c                        |  19 +-
 kernel/time/Kconfig                     |  80 ++++++++-
 kernel/time/tick-broadcast.c            |   3 +-
 kernel/time/tick-common.c               |   5 +-
 kernel/time/tick-sched.c                | 296 ++++++++++++++++++++++++++++++--
 kernel/timer.c                          |  16 +-
 31 files changed, 988 insertions(+), 117 deletions(-)
 create mode 100644 Documentation/timers/NO_HZ.txt
Documentation/RCU/stallwarn.txt +1 -1
index e38b8df..8e9359d 100644
@@ -191,7 +191,7 @@ o	A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that
 o	A hardware or software issue shuts off the scheduler-clock
 	interrupt on a CPU that is not in dyntick-idle mode.  This
 	problem really has happened, and seems to be most likely to
-	result in RCU CPU stall warnings for CONFIG_NO_HZ=n kernels.
+	result in RCU CPU stall warnings for CONFIG_NO_HZ_COMMON=n kernels.
 
 o	A bug in the RCU implementation.
 
Documentation/cpu-freq/governors.txt +2 -2
index c7a2eb8..e3e5d9a 100644
@@ -131,8 +131,8 @@ sampling_rate_min:
 The sampling rate is limited by the HW transition latency:
 transition_latency * 100
 Or by kernel restrictions:
-If CONFIG_NO_HZ is set, the limit is 10ms fixed.
-If CONFIG_NO_HZ is not set or nohz=off boot parameter is used, the
+If CONFIG_NO_HZ_COMMON is set, the limit is 10ms fixed.
+If CONFIG_NO_HZ_COMMON is not set or nohz=off boot parameter is used, the
 limits depend on the CONFIG_HZ option:
 HZ=1000: min=20000us  (20ms)
 HZ=250:  min=80000us  (80ms)
Documentation/kernel-parameters.txt +8 -0
index de12397..7d55ebb 100644
@@ -1951,6 +1951,14 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			Valid arguments: on, off
 			Default: on
 
+	nohz_full=	[KNL,BOOT]
+			In kernels built with CONFIG_NO_HZ_FULL=y, set
+			the specified list of CPUs whose tick will be stopped
+			whenever possible. The boot CPU will be forced outside
+			the range to maintain the timekeeping.
+			The CPUs in this range must also be included in the
+			rcu_nocbs= set.
+
 	noiotrap	[SH] Disables trapped I/O port accesses.
 
 	noirqdebug	[X86-32] Disables the code which attempts to detect and
Documentation/timers/NO_HZ.txt new file +273 -0

273 lines not on this page · raw message

arch/um/include/shared/common-offsets.h +2 -2
index 2df313b..c923068 100644
@@ -30,8 +30,8 @@ DEFINE(UM_NSEC_PER_USEC, NSEC_PER_USEC);
 #ifdef CONFIG_PRINTK
 DEFINE(UML_CONFIG_PRINTK, CONFIG_PRINTK);
 #endif
-#ifdef CONFIG_NO_HZ
-DEFINE(UML_CONFIG_NO_HZ, CONFIG_NO_HZ);
+#ifdef CONFIG_NO_HZ_COMMON
+DEFINE(UML_CONFIG_NO_HZ_COMMON, CONFIG_NO_HZ_COMMON);
 #endif
 #ifdef CONFIG_UML_X86
 DEFINE(UML_CONFIG_UML_X86, CONFIG_UML_X86);
arch/um/os-Linux/time.c +1 -1
index fac388c..e9824d5 100644
@@ -79,7 +79,7 @@ long long os_nsecs(void)
 	return timeval_to_ns(&tv);
 }
 
-#ifdef UML_CONFIG_NO_HZ
+#ifdef UML_CONFIG_NO_HZ_COMMON
 static int after_sleep_interval(struct timespec *ts)
 {
 	return 0;
include/asm-generic/cputime_nsecs.h +19 -9
index a8ece9a..2c9e62c 100644
@@ -16,21 +16,27 @@
 #ifndef _ASM_GENERIC_CPUTIME_NSECS_H
 #define _ASM_GENERIC_CPUTIME_NSECS_H
 
+#include <linux/math64.h>
+
 typedef u64 __nocast cputime_t;
 typedef u64 __nocast cputime64_t;
 
 #define cputime_one_jiffy		jiffies_to_cputime(1)
 
+#define cputime_div(__ct, divisor)  div_u64((__force u64)__ct, divisor)
+#define cputime_div_rem(__ct, divisor, remainder) \
+	div_u64_rem((__force u64)__ct, divisor, remainder);
+
 /*
  * Convert cputime <-> jiffies (HZ)
  */
 #define cputime_to_jiffies(__ct)	\
-	((__force u64)(__ct) / (NSEC_PER_SEC / HZ))
+	cputime_div(__ct, NSEC_PER_SEC / HZ)
 #define cputime_to_scaled(__ct)		(__ct)
 #define jiffies_to_cputime(__jif)	\
 	(__force cputime_t)((__jif) * (NSEC_PER_SEC / HZ))
 #define cputime64_to_jiffies64(__ct)	\
-	((__force u64)(__ct) / (NSEC_PER_SEC / HZ))
+	cputime_div(__ct, NSEC_PER_SEC / HZ)
 #define jiffies64_to_cputime64(__jif)	\
 	(__force cputime64_t)((__jif) * (NSEC_PER_SEC / HZ))
 
@@ -45,7 +51,7 @@ typedef u64 __nocast cputime64_t;
  * Convert cputime <-> microseconds
  */
 #define cputime_to_usecs(__ct)		\
-	((__force u64)(__ct) / NSEC_PER_USEC)
+	cputime_div(__ct, NSEC_PER_USEC)
 #define usecs_to_cputime(__usecs)	\
 	(__force cputime_t)((__usecs) * NSEC_PER_USEC)
 #define usecs_to_cputime64(__usecs)	\
@@ -55,7 +61,7 @@ typedef u64 __nocast cputime64_t;
  * Convert cputime <-> seconds
  */
 #define cputime_to_secs(__ct)		\
-	((__force u64)(__ct) / NSEC_PER_SEC)
+	cputime_div(__ct, NSEC_PER_SEC)
 #define secs_to_cputime(__secs)		\
 	(__force cputime_t)((__secs) * NSEC_PER_SEC)
 
@@ -69,8 +75,10 @@ static inline cputime_t timespec_to_cputime(const struct timespec *val)
 }
 static inline void cputime_to_timespec(const cputime_t ct, struct timespec *val)
 {
-	val->tv_sec  = (__force u64) ct / NSEC_PER_SEC;
-	val->tv_nsec = (__force u64) ct % NSEC_PER_SEC;
+	u32 rem;
+
+	val->tv_sec = cputime_div_rem(ct, NSEC_PER_SEC, &rem);
+	val->tv_nsec = rem;
 }
 
 /*
@@ -83,15 +91,17 @@ static inline cputime_t timeval_to_cputime(const struct timeval *val)
 }
 static inline void cputime_to_timeval(const cputime_t ct, struct timeval *val)
 {
-	val->tv_sec = (__force u64) ct / NSEC_PER_SEC;
-	val->tv_usec = ((__force u64) ct % NSEC_PER_SEC) / NSEC_PER_USEC;
+	u32 rem;
+
+	val->tv_sec = cputime_div_rem(ct, NSEC_PER_SEC, &rem);
+	val->tv_usec = rem / NSEC_PER_USEC;
 }
 
 /*
  * Convert cputime <-> clock (USER_HZ)
  */
 #define cputime_to_clock_t(__ct)	\
-	((__force u64)(__ct) / (NSEC_PER_SEC / USER_HZ))
+	cputime_div(__ct, (NSEC_PER_SEC / USER_HZ))
 #define clock_t_to_cputime(__x)		\
 	(__force cputime_t)((__x) * (NSEC_PER_SEC / USER_HZ))
 
include/linux/perf_event.h +6 -0
index e0373d2..f463a46 100644
@@ -788,6 +788,12 @@ static inline int __perf_event_disable(void *info)			{ return -1; }
 static inline void perf_event_task_tick(void)				{ }
 #endif
 
+#if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_NO_HZ_FULL)
+extern bool perf_event_can_stop_tick(void);
+#else
+static inline bool perf_event_can_stop_tick(void)			{ return true; }
+#endif
+
 #if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_CPU_SUP_INTEL)
 extern void perf_restore_debug_store(void);
 #else
include/linux/posix-timers.h +2 -0
index 042058f..3698d9d 100644
@@ -122,6 +122,8 @@ void run_posix_cpu_timers(struct task_struct *task);
 void posix_cpu_timers_exit(struct task_struct *task);
 void posix_cpu_timers_exit_group(struct task_struct *task);
 
+bool posix_cpu_timers_can_stop_tick(struct task_struct *tsk);
+
 void set_process_cpu_timer(struct task_struct *task, unsigned int clock_idx,
 			   cputime_t *newval, cputime_t *oldval);
 
include/linux/rcupdate.h +7 -0
index 9ed2c9a..4ccd68e 100644
@@ -1000,4 +1000,11 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
 #define kfree_rcu(ptr, rcu_head)					\
 	__kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head))
 
+#ifdef CONFIG_RCU_NOCB_CPU
+extern bool rcu_is_nocb_cpu(int cpu);
+#else
+static inline bool rcu_is_nocb_cpu(int cpu) { return false; }
+#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
+
+
 #endif /* __LINUX_RCUPDATE_H */
include/linux/sched.h +13 -6
index 981ab68..af008d7 100644
@@ -231,7 +231,7 @@ extern void init_idle_bootup_task(struct task_struct *idle);
 
 extern int runqueue_is_locked(int cpu);
 
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
 extern void nohz_balance_enter_idle(int cpu);
 extern void set_cpu_sd_state_idle(void);
 extern int get_nohz_timer_target(void);
@@ -1762,13 +1762,13 @@ static inline int set_cpus_allowed_ptr(struct task_struct *p,
 }
 #endif
 
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 void calc_load_enter_idle(void);
 void calc_load_exit_idle(void);
 #else
 static inline void calc_load_enter_idle(void) { }
 static inline void calc_load_exit_idle(void) { }
-#endif /* CONFIG_NO_HZ */
+#endif /* CONFIG_NO_HZ_COMMON */
 
 #ifndef CONFIG_CPUMASK_OFFSTACK
 static inline int set_cpus_allowed(struct task_struct *p, cpumask_t new_mask)
@@ -1854,10 +1854,17 @@ extern void idle_task_exit(void);
 static inline void idle_task_exit(void) {}
 #endif
 
-#if defined(CONFIG_NO_HZ) && defined(CONFIG_SMP)
-extern void wake_up_idle_cpu(int cpu);
+#if defined(CONFIG_NO_HZ_COMMON) && defined(CONFIG_SMP)
+extern void wake_up_nohz_cpu(int cpu);
 #else
-static inline void wake_up_idle_cpu(int cpu) { }
+static inline void wake_up_nohz_cpu(int cpu) { }
+#endif
+
+#ifdef CONFIG_NO_HZ_FULL
+extern bool sched_can_stop_tick(void);
+extern u64 scheduler_tick_max_deferment(void);
+#else
+static inline bool sched_can_stop_tick(void) { return false; }
 #endif
 
 #ifdef CONFIG_SCHED_AUTOGROUP
include/linux/tick.h +21 -4
index 553272e..9180f4b 100644
@@ -82,7 +82,7 @@ extern int tick_program_event(ktime_t expires, int force);
 extern void tick_setup_sched_timer(void);
 # endif
 
-# if defined CONFIG_NO_HZ || defined CONFIG_HIGH_RES_TIMERS
+# if defined CONFIG_NO_HZ_COMMON || defined CONFIG_HIGH_RES_TIMERS
 extern void tick_cancel_sched_timer(int cpu);
 # else
 static inline void tick_cancel_sched_timer(int cpu) { }
@@ -123,7 +123,7 @@ static inline void tick_check_idle(int cpu) { }
 static inline int tick_oneshot_mode_active(void) { return 0; }
 #endif /* !CONFIG_GENERIC_CLOCKEVENTS */
 
-# ifdef CONFIG_NO_HZ
+# ifdef CONFIG_NO_HZ_COMMON
 DECLARE_PER_CPU(struct tick_sched, tick_cpu_sched);
 
 static inline int tick_nohz_tick_stopped(void)
@@ -138,7 +138,7 @@ extern ktime_t tick_nohz_get_sleep_length(void);
 extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
 extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 
-# else /* !CONFIG_NO_HZ */
+# else /* !CONFIG_NO_HZ_COMMON */
 static inline int tick_nohz_tick_stopped(void)
 {
 	return 0;
@@ -155,7 +155,24 @@ static inline ktime_t tick_nohz_get_sleep_length(void)
 }
 static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; }
 static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; }
-# endif /* !NO_HZ */
+# endif /* !CONFIG_NO_HZ_COMMON */
+
+#ifdef CONFIG_NO_HZ_FULL
+extern void tick_nohz_init(void);
+extern int tick_nohz_full_cpu(int cpu);
+extern void tick_nohz_full_check(void);
+extern void tick_nohz_full_kick(void);
+extern void tick_nohz_full_kick_all(void);
+extern void tick_nohz_task_switch(struct task_struct *tsk);
+#else
+static inline void tick_nohz_init(void) { }
+static inline int tick_nohz_full_cpu(int cpu) { return 0; }
+static inline void tick_nohz_full_check(void) { }
+static inline void tick_nohz_full_kick(void) { }
+static inline void tick_nohz_full_kick_all(void) { }
+static inline void tick_nohz_task_switch(struct task_struct *tsk) { }
+#endif
+
 
 # ifdef CONFIG_CPU_IDLE_GOV_MENU
 extern void menu_hrtimer_cancel(void);
include/trace/events/timer.h +21 -0
index 425bcfe..e967dd8 100644
@@ -323,6 +323,27 @@ TRACE_EVENT(itimer_expire,
 		  (int) __entry->pid, (unsigned long long)__entry->now)
 );
 
+#ifdef CONFIG_NO_HZ_COMMON
+TRACE_EVENT(tick_stop,
+
+	TP_PROTO(int success, char *error_msg),
+
+	TP_ARGS(success, error_msg),
+
+	TP_STRUCT__entry(
+		__field( int ,		success	)
+		__string( msg, 		error_msg )
+	),
+
+	TP_fast_assign(
+		__entry->success	= success;
+		__assign_str(msg, error_msg);
+	),
+
+	TP_printk("success=%s msg=%s",  __entry->success ? "yes" : "no", __get_str(msg))
+);
+#endif
+
 #endif /*  _TRACE_TIMER_H */
 
 /* This part must be outside protection */
init/Kconfig +6 -6
index 4367e13..b3fdf9a 100644
@@ -302,7 +302,7 @@ choice
 # Kind of a stub config for the pure tick based cputime accounting
 config TICK_CPU_ACCOUNTING
 	bool "Simple tick based cputime accounting"
-	depends on !S390
+	depends on !S390 && !NO_HZ_FULL
 	help
 	  This is the basic tick based cputime accounting that maintains
 	  statistics about user, system and idle time spent on per jiffies
@@ -312,7 +312,7 @@ config TICK_CPU_ACCOUNTING
 
 config VIRT_CPU_ACCOUNTING_NATIVE
 	bool "Deterministic task and CPU time accounting"
-	depends on HAVE_VIRT_CPU_ACCOUNTING
+	depends on HAVE_VIRT_CPU_ACCOUNTING && !NO_HZ_FULL
 	select VIRT_CPU_ACCOUNTING
 	help
 	  Select this option to enable more accurate task and CPU time
@@ -342,7 +342,7 @@ config VIRT_CPU_ACCOUNTING_GEN
 
 config IRQ_TIME_ACCOUNTING
 	bool "Fine granularity task level IRQ time accounting"
-	depends on HAVE_IRQ_TIME_ACCOUNTING
+	depends on HAVE_IRQ_TIME_ACCOUNTING && !NO_HZ_FULL
 	help
 	  Select this option to enable fine granularity task irq time
 	  accounting. This is done by reading a timestamp on each
@@ -576,7 +576,7 @@ config RCU_FANOUT_EXACT
 
 config RCU_FAST_NO_HZ
 	bool "Accelerate last non-dyntick-idle CPU's grace periods"
-	depends on NO_HZ && SMP
+	depends on NO_HZ_COMMON && SMP
 	default n
 	help
 	  This option permits CPUs to enter dynticks-idle state even if
@@ -687,7 +687,7 @@ choice
 
 config RCU_NOCB_CPU_NONE
 	bool "No build_forced no-CBs CPUs"
-	depends on RCU_NOCB_CPU
+	depends on RCU_NOCB_CPU && !NO_HZ_FULL
 	help
 	  This option does not force any of the CPUs to be no-CBs CPUs.
 	  Only CPUs designated by the rcu_nocbs= boot parameter will be
@@ -695,7 +695,7 @@ config RCU_NOCB_CPU_NONE
 
 config RCU_NOCB_CPU_ZERO
 	bool "CPU 0 is a build_forced no-CBs CPU"
-	depends on RCU_NOCB_CPU
+	depends on RCU_NOCB_CPU && !NO_HZ_FULL
 	help
 	  This option forces CPU 0 to be a no-CBs CPU.  Additional CPUs
 	  may be designated as no-CBs CPUs using the rcu_nocbs= boot
init/main.c +1 -0
index 12c3669..1952bf2 100644
@@ -545,6 +545,7 @@ asmlinkage void __init start_kernel(void)
 	idr_init_cache();
 	perf_event_init();
 	rcu_init();
+	tick_nohz_init();
 	radix_tree_init();
 	/* init some links before init_ISA_irqs() */
 	early_irq_init();
kernel/events/core.c +16 -1
index 3820e3c..6b41c18 100644
@@ -18,6 +18,7 @@
 #include <linux/poll.h>
 #include <linux/slab.h>
 #include <linux/hash.h>
+#include <linux/tick.h>
 #include <linux/sysfs.h>
 #include <linux/dcache.h>
 #include <linux/percpu.h>
@@ -685,8 +686,12 @@ static void perf_pmu_rotate_start(struct pmu *pmu)
 
 	WARN_ON(!irqs_disabled());
 
-	if (list_empty(&cpuctx->rotation_list))
+	if (list_empty(&cpuctx->rotation_list)) {
+		int was_empty = list_empty(head);
 		list_add(&cpuctx->rotation_list, head);
+		if (was_empty)
+			tick_nohz_full_kick();
+	}
 }
 
 static void get_ctx(struct perf_event_context *ctx)
@@ -2591,6 +2596,16 @@ done:
 		list_del_init(&cpuctx->rotation_list);
 }
 
+#ifdef CONFIG_NO_HZ_FULL
+bool perf_event_can_stop_tick(void)
+{
+	if (list_empty(&__get_cpu_var(rotation_list)))
+		return true;
+	else
+		return false;
+}
+#endif
+
 void perf_event_task_tick(void)
 {
 	struct list_head *head = &__get_cpu_var(rotation_list);
kernel/hrtimer.c +2 -2
index 14be27f..abfd89d 100644
@@ -161,7 +161,7 @@ struct hrtimer_clock_base *lock_hrtimer_base(const struct hrtimer *timer,
  */
 static int hrtimer_get_target(int this_cpu, int pinned)
 {
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 	if (!pinned && get_sysctl_timer_migration() && idle_cpu(this_cpu))
 		return get_nohz_timer_target();
 #endif
@@ -1107,7 +1107,7 @@ ktime_t hrtimer_get_remaining(const struct hrtimer *timer)
 }
 EXPORT_SYMBOL_GPL(hrtimer_get_remaining);
 
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 /**
  * hrtimer_get_next_event - get the time until next expiry event
  *
kernel/posix-cpu-timers.c +60 -16

124 lines not on this page · raw message

kernel/rcutree.c +13 -3
index d853430..16ea679 100644
@@ -799,6 +799,16 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)
 		rdp->offline_fqs++;
 		return 1;
 	}
+
+	/*
+	 * There is a possibility that a CPU in adaptive-ticks state
+	 * might run in the kernel with the scheduling-clock tick disabled
+	 * for an extended time period.  Invoke rcu_kick_nohz_cpu() to
+	 * force the CPU to restart the scheduling-clock tick in this
+	 * CPU is in this state.
+	 */
+	rcu_kick_nohz_cpu(rdp->cpu);
+
 	return 0;
 }
 
@@ -1820,7 +1830,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,
 			  struct rcu_node *rnp, struct rcu_data *rdp)
 {
 	/* No-CBs CPUs do not have orphanable callbacks. */
-	if (is_nocb_cpu(rdp->cpu))
+	if (rcu_is_nocb_cpu(rdp->cpu))
 		return;
 
 	/*
@@ -2892,10 +2902,10 @@ static void _rcu_barrier(struct rcu_state *rsp)
 	 * corresponding CPU's preceding callbacks have been invoked.
 	 */
 	for_each_possible_cpu(cpu) {
-		if (!cpu_online(cpu) && !is_nocb_cpu(cpu))
+		if (!cpu_online(cpu) && !rcu_is_nocb_cpu(cpu))
 			continue;
 		rdp = per_cpu_ptr(rsp->rda, cpu);
-		if (is_nocb_cpu(cpu)) {
+		if (rcu_is_nocb_cpu(cpu)) {
 			_rcu_barrier_trace(rsp, "OnlineNoCB", cpu,
 					   rsp->n_barrier_done);
 			atomic_inc(&rsp->barrier_cpu_count);
kernel/rcutree.h +1 -1
index 14ee407..da77a8f 100644
@@ -530,13 +530,13 @@ static int rcu_nocb_needs_gp(struct rcu_state *rsp);
 static void rcu_nocb_gp_set(struct rcu_node *rnp, int nrq);
 static void rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp);
 static void rcu_init_one_nocb(struct rcu_node *rnp);
-static bool is_nocb_cpu(int cpu);
 static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,
 			    bool lazy);
 static bool rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp,
 				      struct rcu_data *rdp);
 static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp);
 static void rcu_spawn_nocb_kthreads(struct rcu_state *rsp);
+static void rcu_kick_nohz_cpu(int cpu);
 static bool init_nocb_callback_list(struct rcu_data *rdp);
 
 #endif /* #ifndef RCU_TREE_NONCORE */
kernel/rcutree_plugin.h +21 -8
index d084ae3..71bd733 100644
@@ -28,6 +28,7 @@
 #include <linux/gfp.h>
 #include <linux/oom.h>
 #include <linux/smpboot.h>
+#include <linux/tick.h>
 
 #define RCU_KTHREAD_PRIO 1
 
@@ -2052,7 +2053,7 @@ static void rcu_init_one_nocb(struct rcu_node *rnp)
 }
 
 /* Is the specified CPU a no-CPUs CPU? */
-static bool is_nocb_cpu(int cpu)
+bool rcu_is_nocb_cpu(int cpu)
 {
 	if (have_rcu_nocb_mask)
 		return cpumask_test_cpu(cpu, rcu_nocb_mask);
@@ -2110,7 +2111,7 @@ static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,
 			    bool lazy)
 {
 
-	if (!is_nocb_cpu(rdp->cpu))
+	if (!rcu_is_nocb_cpu(rdp->cpu))
 		return 0;
 	__call_rcu_nocb_enqueue(rdp, rhp, &rhp->next, 1, lazy);
 	if (__is_kfree_rcu_offset((unsigned long)rhp->func))
@@ -2134,7 +2135,7 @@ static bool __maybe_unused rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp,
 	long qll = rsp->qlen_lazy;
 
 	/* If this is not a no-CBs CPU, tell the caller to do it the old way. */
-	if (!is_nocb_cpu(smp_processor_id()))
+	if (!rcu_is_nocb_cpu(smp_processor_id()))
 		return 0;
 	rsp->qlen = 0;
 	rsp->qlen_lazy = 0;
@@ -2306,11 +2307,6 @@ static void rcu_init_one_nocb(struct rcu_node *rnp)
 {
 }
 
-static bool is_nocb_cpu(int cpu)
-{
-	return false;
-}
-
 static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,
 			    bool lazy)
 {
@@ -2337,3 +2333,20 @@ static bool init_nocb_callback_list(struct rcu_data *rdp)
 }
 
 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
+
+/*
+ * An adaptive-ticks CPU can potentially execute in kernel mode for an
+ * arbitrarily long period of time with the scheduling-clock tick turned
+ * off.  RCU will be paying attention to this CPU because it is in the
+ * kernel, but the CPU cannot be guaranteed to be executing the RCU state
+ * machine because the scheduling-clock tick has been disabled.  Therefore,
+ * if an adaptive-ticks CPU is failing to respond to the current grace
+ * period and has not be idle from an RCU perspective, kick it.
+ */
+static void rcu_kick_nohz_cpu(int cpu)
+{
+#ifdef CONFIG_NO_HZ_FULL
+	if (tick_nohz_full_cpu(cpu))
+		smp_send_reschedule(cpu);
+#endif /* #ifdef CONFIG_NO_HZ_FULL */
+}
kernel/sched/core.c +81 -11

183 lines not on this page · raw message

kernel/sched/fair.c +5 -5
index 8bf7081..c61a614 100644
@@ -5355,7 +5355,7 @@ out_unlock:
 	return 0;
 }
 
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 /*
  * idle load balancing details
  * - When one of the busy CPUs notice that there may be an idle rebalancing
@@ -5572,9 +5572,9 @@ out:
 		rq->next_balance = next_balance;
 }
 
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 /*
- * In CONFIG_NO_HZ case, the idle balance kickee will do the
+ * In CONFIG_NO_HZ_COMMON case, the idle balance kickee will do the
  * rebalancing for all the cpus for whom scheduler ticks are stopped.
  */
 static void nohz_idle_balance(int this_cpu, enum cpu_idle_type idle)
@@ -5717,7 +5717,7 @@ void trigger_load_balance(struct rq *rq, int cpu)
 	if (time_after_eq(jiffies, rq->next_balance) &&
 	    likely(!on_null_domain(cpu)))
 		raise_softirq(SCHED_SOFTIRQ);
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 	if (nohz_kick_needed(rq, cpu) && likely(!on_null_domain(cpu)))
 		nohz_balancer_kick(cpu);
 #endif
@@ -6187,7 +6187,7 @@ __init void init_sched_fair_class(void)
 #ifdef CONFIG_SMP
 	open_softirq(SCHED_SOFTIRQ, run_rebalance_domains);
 
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 	nohz.next_balance = jiffies;
 	zalloc_cpumask_var(&nohz.idle_cpus_mask, GFP_NOWAIT);
 	cpu_notifier(sched_ilb_notifier, 0);
kernel/sched/idle_task.c +1 -0
index b8ce773..d8da010 100644
@@ -17,6 +17,7 @@ select_task_rq_idle(struct task_struct *p, int sd_flag, int flags)
 static void pre_schedule_idle(struct rq *rq, struct task_struct *prev)
 {
 	idle_exit_fair(rq);
+	rq_last_tick_reset(rq);
 }
 
 static void post_schedule_idle(struct rq *rq)
kernel/sched/sched.h +23 -2
index 4c225c4..ce39224 100644
@@ -5,6 +5,7 @@
 #include <linux/mutex.h>
 #include <linux/spinlock.h>
 #include <linux/stop_machine.h>
+#include <linux/tick.h>
 
 #include "cpupri.h"
 #include "cpuacct.h"
@@ -405,10 +406,13 @@ struct rq {
 	#define CPU_LOAD_IDX_MAX 5
 	unsigned long cpu_load[CPU_LOAD_IDX_MAX];
 	unsigned long last_load_update_tick;
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 	u64 nohz_stamp;
 	unsigned long nohz_flags;
 #endif
+#ifdef CONFIG_NO_HZ_FULL
+	unsigned long last_sched_tick;
+#endif
 	int skip_clock_update;
 
 	/* capture load from *all* tasks on this cpu: */
@@ -1072,6 +1076,16 @@ static inline u64 steal_ticks(u64 steal)
 static inline void inc_nr_running(struct rq *rq)
 {
 	rq->nr_running++;
+
+#ifdef CONFIG_NO_HZ_FULL
+	if (rq->nr_running == 2) {
+		if (tick_nohz_full_cpu(rq->cpu)) {
+			/* Order rq->nr_running write against the IPI */
+			smp_wmb();
+			smp_send_reschedule(rq->cpu);
+		}
+       }
+#endif
 }
 
 static inline void dec_nr_running(struct rq *rq)
@@ -1079,6 +1093,13 @@ static inline void dec_nr_running(struct rq *rq)
 	rq->nr_running--;
 }
 
+static inline void rq_last_tick_reset(struct rq *rq)
+{
+#ifdef CONFIG_NO_HZ_FULL
+	rq->last_sched_tick = jiffies;
+#endif
+}
+
 extern void update_rq_clock(struct rq *rq);
 
 extern void activate_task(struct rq *rq, struct task_struct *p, int flags);
@@ -1299,7 +1320,7 @@ extern void init_rt_rq(struct rt_rq *rt_rq, struct rq *rq);
 
 extern void account_cfs_bandwidth_used(int enabled, int was_enabled);
 
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 enum rq_nohz_flag_bits {
 	NOHZ_TICK_STOPPED,
 	NOHZ_BALANCE_KICK,
kernel/softirq.c +14 -5
index 14d7758..51a09d5 100644
@@ -329,6 +329,19 @@ static inline void invoke_softirq(void)
 		wakeup_softirqd();
 }
 
+static inline void tick_irq_exit(void)
+{
+#ifdef CONFIG_NO_HZ_COMMON
+	int cpu = smp_processor_id();
+
+	/* Make sure that timer wheel updates are propagated */
+	if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
+		if (!in_interrupt())
+			tick_nohz_irq_exit();
+	}
+#endif
+}
+
 /*
  * Exit an interrupt context. Process softirqs if needed and possible:
  */
@@ -346,11 +359,7 @@ void irq_exit(void)
 	if (!in_interrupt() && local_softirq_pending())
 		invoke_softirq();
 
-#ifdef CONFIG_NO_HZ
-	/* Make sure that timer wheel updates are propagated */
-	if (idle_cpu(smp_processor_id()) && !in_interrupt() && !need_resched())
-		tick_nohz_irq_exit();
-#endif
+	tick_irq_exit();
 	rcu_irq_exit();
 }
 
kernel/time/Kconfig +74 -6

94 lines not on this page · raw message

kernel/time/tick-broadcast.c +2 -1
index 7f32fe0..40c1050 100644
@@ -574,7 +574,8 @@ void tick_broadcast_setup_oneshot(struct clock_event_device *bc)
 		bc->event_handler = tick_handle_oneshot_broadcast;
 
 		/* Take the do_timer update */
-		tick_do_timer_cpu = cpu;
+		if (!tick_nohz_full_cpu(cpu))
+			tick_do_timer_cpu = cpu;
 
 		/*
 		 * We must be careful here. There might be other CPUs
kernel/time/tick-common.c +4 -1
index b1600a6..83f2bd9 100644
@@ -163,7 +163,10 @@ static void tick_setup_device(struct tick_device *td,
 		 * this cpu:
 		 */
 		if (tick_do_timer_cpu == TICK_DO_TIMER_BOOT) {
-			tick_do_timer_cpu = cpu;
+			if (!tick_nohz_full_cpu(cpu))
+				tick_do_timer_cpu = cpu;
+			else
+				tick_do_timer_cpu = TICK_DO_TIMER_NONE;
 			tick_next_period = ktime_get();
 			tick_period = ktime_set(0, NSEC_PER_SEC / HZ);
 		}
kernel/time/tick-sched.c +280 -16

390 lines not on this page · raw message

kernel/timer.c +8 -8
index dbf7a78..1b7489f 100644
@@ -738,7 +738,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires,
 
 	cpu = smp_processor_id();
 
-#if defined(CONFIG_NO_HZ) && defined(CONFIG_SMP)
+#if defined(CONFIG_NO_HZ_COMMON) && defined(CONFIG_SMP)
 	if (!pinned && get_sysctl_timer_migration() && idle_cpu(cpu))
 		cpu = get_nohz_timer_target();
 #endif
@@ -930,14 +930,14 @@ void add_timer_on(struct timer_list *timer, int cpu)
 	debug_activate(timer, timer->expires);
 	internal_add_timer(base, timer);
 	/*
-	 * Check whether the other CPU is idle and needs to be
-	 * triggered to reevaluate the timer wheel when nohz is
-	 * active. We are protected against the other CPU fiddling
+	 * Check whether the other CPU is in dynticks mode and needs
+	 * to be triggered to reevaluate the timer wheel.
+	 * We are protected against the other CPU fiddling
 	 * with the timer by holding the timer base lock. This also
-	 * makes sure that a CPU on the way to idle can not evaluate
-	 * the timer wheel.
+	 * makes sure that a CPU on the way to stop its tick can not
+	 * evaluate the timer wheel.
 	 */
-	wake_up_idle_cpu(cpu);
+	wake_up_nohz_cpu(cpu);
 	spin_unlock_irqrestore(&base->lock, flags);
 }
 EXPORT_SYMBOL_GPL(add_timer_on);
@@ -1188,7 +1188,7 @@ static inline void __run_timers(struct tvec_base *base)
 	spin_unlock_irq(&base->lock);
 }
 
-#ifdef CONFIG_NO_HZ
+#ifdef CONFIG_NO_HZ_COMMON
 /*
  * Find out when the next timer event is due to happen. This
  * is used on S/390 to stop all activity when a CPU is idle.
#1 lore raw
Quoted 2 lines · Ingo Molnar

Please consider pulling the latest timers-nohz-for-linus git tree from:

Ok, it seems to work for me, so pulled.

However, by "work for me" I mean "doesn't actually seem to make any difference for me". Maybe I'm odd, but the most common situation is either a fairly idle machine (in which case the old NOHZ did fine) or a fairly over-crowded one when I'm running something sufficiently threaded (in which case the new NOHZ_FULL doesn't do anything either).

So I really hope the "cpu has more than one running thread" case is getting a lot of attention. Not for 3.10, but right now it seems to still result in the same old 1kHz timer interrupts..

So I haven't actually found a real load where any of this makes a noticeable *difference*.

Linus

#2 lore raw
Quoted 12 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar

Please consider pulling the latest timers-nohz-for-linus git tree from:

Ok, it seems to work for me, so pulled.

However, by "work for me" I mean "doesn't actually seem to make any difference for me". Maybe I'm odd, but the most common situation is either a fairly idle machine (in which case the old NOHZ did fine) or a fairly over-crowded one when I'm running something sufficiently threaded (in which case the new NOHZ_FULL doesn't do anything either).

Quoted 6 lines

So I really hope the "cpu has more than one running thread" case is getting a lot of attention. Not for 3.10, but right now it seems to still result in the same old 1kHz timer interrupts..

So I haven't actually found a real load where any of this makes a noticeable *difference*.

The workloads where we expect the most noticeable differences are HPC workloads with short iterations and a HPC-style barrier between each interation on the one hand and real-time workloads on the other. My guess is that you aren't doing too much of either.

We do have some measurements taken on an early prototype of this patchset, which are on slides 5 and 6 of:

http://linuxplumbersconf.org/2009/slides/Josh-Triplett-painless-kernel.pdf

This is for an HPC workload with a 100-microsecond iteration time.

Thanx, Paul

author maintainer
#3 lore raw
Quoted 17 lines · Paul E. McKenney
Quoted 16 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar

Please consider pulling the latest timers-nohz-for-linus git tree from:

Ok, it seems to work for me, so pulled.

However, by "work for me" I mean "doesn't actually seem to make any difference for me". Maybe I'm odd, but the most common situation is either a fairly idle machine (in which case the old NOHZ did fine) or a fairly over-crowded one when I'm running something sufficiently threaded (in which case the new NOHZ_FULL doesn't do anything either).

So I really hope the "cpu has more than one running thread" case is getting a lot of attention. Not for 3.10, but right now it seems to still result in the same old 1kHz timer interrupts..

Quoted 7 lines
Quoted 2 lines

So I haven't actually found a real load where any of this makes a noticeable *difference*.

The workloads where we expect the most noticeable differences are HPC workloads with short iterations and a HPC-style barrier between each interation on the one hand and real-time workloads on the other. My guess is that you aren't doing too much of either.

I think Linus might have referred to my 'future plans' entry:

| Future plans:
|
| - there's ongoing work to reduce 1Hz to 0Hz, to essentially shut
|   off the periodic tick altogether when there's a single busy task on a
|   CPU. We'd first like 1 Hz to be exposed more widely before we go for
|   the 0 Hz target though.
|
| - once we reach 0 Hz we can and remove the periodic tick assumption from
|   nr_running>=2 as well, by essentially interrupting busy tasks only as
|   frequently as the sched_latency constraints require us to do - once
|   every 4-40 msecs, depending on nr_running.

and indicated that in practice desktop and developer workload will see the full win from reduced HZ only once we implement those two points and extend the scope of dynticks even more and make HZ truly variable regardless of rq->nr_running.

Quoted 6 lines

We do have some measurements taken on an early prototype of this patchset, which are on slides 5 and 6 of:

http://linuxplumbersconf.org/2009/slides/Josh-Triplett-painless-kernel.pdf

This is for an HPC workload with a 100-microsecond iteration time.

Interesting that HZ=1000 caused 8% overhead there. On a regular x86 server PC I've measured the HZ=1000 overhead to pure user-space execution to be around 1% (sometimes a bit less, sometimes a bit more).

But even 1% is worth it.

Thanks,

Ingo

#4 lore raw
Quoted 42 lines · Ingo Molnar
Quoted 17 lines · Paul E. McKenney
Quoted 16 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar

Please consider pulling the latest timers-nohz-for-linus git tree from:

Ok, it seems to work for me, so pulled.

However, by "work for me" I mean "doesn't actually seem to make any difference for me". Maybe I'm odd, but the most common situation is either a fairly idle machine (in which case the old NOHZ did fine) or a fairly over-crowded one when I'm running something sufficiently threaded (in which case the new NOHZ_FULL doesn't do anything either).

So I really hope the "cpu has more than one running thread" case is getting a lot of attention. Not for 3.10, but right now it seems to still result in the same old 1kHz timer interrupts..

Quoted 7 lines
Quoted 2 lines

So I haven't actually found a real load where any of this makes a noticeable *difference*.

The workloads where we expect the most noticeable differences are HPC workloads with short iterations and a HPC-style barrier between each interation on the one hand and real-time workloads on the other. My guess is that you aren't doing too much of either.

I think Linus might have referred to my 'future plans' entry:

| Future plans:
|
| - there's ongoing work to reduce 1Hz to 0Hz, to essentially shut
|   off the periodic tick altogether when there's a single busy task on a
|   CPU. We'd first like 1 Hz to be exposed more widely before we go for
|   the 0 Hz target though.
|
| - once we reach 0 Hz we can and remove the periodic tick assumption from
|   nr_running>=2 as well, by essentially interrupting busy tasks only as
|   frequently as the sched_latency constraints require us to do - once
|   every 4-40 msecs, depending on nr_running.
Quoted 4 lines

and indicated that in practice desktop and developer workload will see the full win from reduced HZ only once we implement those two points and extend the scope of dynticks even more and make HZ truly variable regardless of rq->nr_running.

You are right, that would make more sense given his response. I guess I should read these things more carefully before replying. :-/

Quoted 5 lines
Quoted 5 lines

We do have some measurements taken on an early prototype of this patchset, which are on slides 5 and 6 of:

http://linuxplumbersconf.org/2009/slides/Josh-Triplett-painless-kernel.pdf

Quoted 7 lines
Quoted 1 line

This is for an HPC workload with a 100-microsecond iteration time.

Interesting that HZ=1000 caused 8% overhead there. On a regular x86 server PC I've measured the HZ=1000 overhead to pure user-space execution to be around 1% (sometimes a bit less, sometimes a bit more).

But even 1% is worth it.

I believe that the difference is tick skew -- the data above was collected with it enabled, but it now is disabled by default (but can be enabled via the skew_tick= boot parameter). Large systems benefit from tick skew due to reduced lock contention, but Frederic's patches allow them to avoid the contention when there are multiple runnable processes per CPU on the one hand but also to avoid OS jitter when there is but one runnable process per CPU on the other hand.

Thanx, Paul

#5 lore raw
Quoted 2 lines · Paul E. McKenney
Quoted 2 lines

I think Linus might have referred to my 'future plans' entry:

Indeed. I feel that HPC is entirely irrelevant to anybody, *especially* HPC benchmarks. In real life, even HPC doesn't tend to have the nice behavior their much-touted benchmarks have.

So as long as the NOHZ is for HPC-style loads, then quite frankly, I don't feel it is worth it. The _only_ thing that makes it worth it is that "future plans" part where it would actually help real loads.

Quoted 8 lines
Quoted 6 lines

Interesting that HZ=1000 caused 8% overhead there. On a regular x86 server PC I've measured the HZ=1000 overhead to pure user-space execution to be around 1% (sometimes a bit less, sometimes a bit more).

But even 1% is worth it.

I believe that the difference is tick skew

Quite possibly it is also virtualization.

The VM people are the one who complain the loudest about how certain things make their performance go down the toilet. And interrupts tend to be high on that list, and unless you have hardware support for virtual timer interrupts I can easily see a factor of four cost or more.

And the VM people then flail around wildly to always blame everybody else. *Anybody* else than the VM overhead itself.

It also depends a lot on architecture. The ia64 people had much bigger problems with the timer interrupt than x86 ever did. Again, they saw this mainly on the HPC benchmarks, because the benchmarks were carefully tuned to have huge-page support and were doing largely irrelevant things like big LINPACK runs, and the timer irq ended up blowing their carefully tuned caches and TLB's out.

Never mind that nobody sane ever *cared*. Afaik, no real HPC load has anything like that behavior, much less anything else. But they had numbers to prove how bad it was, and it was a load with very stable numbers.

Combine the two (bad HPC benchmarks and VM), and you can make an argument for just about anything. And people have.

I am personally less than impressed with some of the benchmarks I've seen, if it wasn't clear.

Linus

author maintainer
#6 lore raw
Quoted 45 lines · Linus Torvalds
Quoted 2 lines · Paul E. McKenney
Quoted 2 lines

I think Linus might have referred to my 'future plans' entry:

Indeed. I feel that HPC is entirely irrelevant to anybody, *especially* HPC benchmarks. In real life, even HPC doesn't tend to have the nice behavior their much-touted benchmarks have.

So as long as the NOHZ is for HPC-style loads, then quite frankly, I don't feel it is worth it. The _only_ thing that makes it worth it is that "future plans" part where it would actually help real loads.

Quoted 8 lines
Quoted 6 lines

Interesting that HZ=1000 caused 8% overhead there. On a regular x86 server PC I've measured the HZ=1000 overhead to pure user-space execution to be around 1% (sometimes a bit less, sometimes a bit more).

But even 1% is worth it.

I believe that the difference is tick skew

Quite possibly it is also virtualization.

The VM people are the one who complain the loudest about how certain things make their performance go down the toilet. And interrupts tend to be high on that list, and unless you have hardware support for virtual timer interrupts I can easily see a factor of four cost or more.

And the VM people then flail around wildly to always blame everybody else. *Anybody* else than the VM overhead itself.

It also depends a lot on architecture. The ia64 people had much bigger problems with the timer interrupt than x86 ever did. Again, they saw this mainly on the HPC benchmarks, because the benchmarks were carefully tuned to have huge-page support and were doing largely irrelevant things like big LINPACK runs, and the timer irq ended up blowing their carefully tuned caches and TLB's out.

Never mind that nobody sane ever *cared*. Afaik, no real HPC load has anything like that behavior, much less anything else. But they had numbers to prove how bad it was, and it was a load with very stable numbers.

Quoted 5 lines

Combine the two (bad HPC benchmarks and VM), and you can make an argument for just about anything. And people have.

I am personally less than impressed with some of the benchmarks I've seen, if it wasn't clear.

Okay.

I never actually ran HPC benchmarks to characterise the overhead - the 0.5%-1.0% figure was the 'worst case' improvement on native hardware with a couple of cores, running a plain infinite loop with no cache footprint.

The per CPU timer/scheduler irq takes 5-10 usecs to execute, and with HZ=1000 which most distros use that happens once every 1000 usecs, which is measurable overhead.

So this feature, in the nr_running=1 case, will produce at minimum a 0.5%-1.0% speedup of user-space workloads (on typical x86).

That alone makes it worth it I think - but we also want to generalize it to nr_running >= 2 as well to cover make -jX workloads, etc.

Thanks,

Ingo

#7 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Paul E. McKenney
Quoted 2 lines

I think Linus might have referred to my 'future plans' entry:

Quoted 7 lines

Indeed. I feel that HPC is entirely irrelevant to anybody, *especially* HPC benchmarks. In real life, even HPC doesn't tend to have the nice behavior their much-touted benchmarks have.

So as long as the NOHZ is for HPC-style loads, then quite frankly, I don't feel it is worth it. The _only_ thing that makes it worth it is that "future plans" part where it would actually help real loads.

The work is very relevant to a lot of the actual customer applications that Tilera chips are sold into: running very low latency userspace applications that handle packet processing (or, to a lesser extent, video frame processing). For packet processing in particular, you really want to be able to guarantee that you can set up a core handling packets in userspace and get NO INTERRUPTS at all, ever. If you do get an interrupt, you end up dropping a bunch of packets on the floor.

We are still using code I developed internally for this (for the curious, in my dataplane branch on kernel.org), but I expect in the relatively near future we will be trying to switch to the NOHZ stuff instead. I'm really pleased to see it start getting merged up.

Signature · 2 lines
Chris Metcalf, Tilera Corp.
http://www.tilera.com