Skip to contentSkip to conversation

timekeeping: Missing timekeeping update detection

RFC PATCH In review Frederic Weisbecker 3 participants 11 messages Aug 21, 2013 – Aug 30, 2013

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover timekeeping: Missing timekeeping update detection

Hi,

With the timekeeping going to be maintained by full system idle detection patchset from Paul, it seems that the guarantees that enforce timekeeping progression are going to grow in complexity enough to deserve some automated checking.

So here is a proposition in the form of a timekeeping watchdog. It uses periodic NMIs that poll on any suspicious drift between jiffies and a global cpu clock progression.

Thanks.

Frederic Weisbecker (6):
sched: Let arch tell us if sched clock is NMI-safe
x86: nsecs to cycles conversion
x86: Tell that sched clock is callable in nmi
seqlock: Add raw_seqbegin() for non-waiting readers
jiffies: Add jiffies_to_nsecs
timekeeping: Debug missing timekeeping updates

 arch/Kconfig                       |    8 ++
 arch/x86/Kconfig                   |    2 +
 arch/x86/include/asm/cycles.h      |   11 +++
 arch/x86/kernel/apic/hw_nmi.c      |    7 --
 include/linux/jiffies.h            |    6 ++
 include/linux/seqlock.h            |    5 ++
 include/linux/time.h               |   11 +++
 kernel/time/Makefile               |    1 +
 kernel/time/tick-sched.c           |    4 +
 kernel/time/timekeeping.c          |    1 +
 kernel/time/timekeeping_selftest.c |  125 ++++++++++++++++++++++++++++++++++++
 kernel/watchdog.c                  |    3 +-
 lib/Kconfig.debug                  |   13 ++++
 13 files changed, 189 insertions(+), 8 deletions(-)
 create mode 100644 arch/x86/include/asm/cycles.h
 create mode 100644 kernel/time/timekeeping_selftest.c
Signature · 1 line
1.7.5.4
#1 lore raw

RFC PATCH 1/6 sched: Let arch tell us if sched clock is NMI-safe

sched_clock() should be fast, scalable and not use any lock. As a resuly it should be safely called from NMIs.

Now just in case there might be some implementation details proper to some archs that make sched_clock() not reliable or not safe in NMIs, lets provide a way through Kconfig for archs to testify about that support.

Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc John Stultz <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Don Zickus <[email protected]>
---
 arch/Kconfig |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
arch/Kconfig +5 -0
index 1feb169..52ad235 100644
@@ -393,6 +393,11 @@ config HAVE_UNDERSCORE_SYMBOL_PREFIX
 	  Some architectures generate an _ in front of C symbols; things like
 	  module loading and assembly files need to know about this.
 
+config HAVE_SCHED_CLOCK_NMI
+	bool
+	help
+	  Architecture's sched_clock() implementation is safely callable from  NMIs.
+
 #
 # ABI hall of shame
 #
Signature · 1 line
1.7.5.4
#2 lore raw

RFC PATCH 2/6 x86: nsecs to cycles conversion

In reply to Frederic Weisbecker

hw_nmi_get_sample_period() is simply a conversion from a period to cycles. Lets generalize the API naming so that it can be used for wider purpose than just watchdog perf event settings. Also it makes the function name less opaque about what it really does.

Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc John Stultz <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Don Zickus <[email protected]>
---
 arch/Kconfig                  |    3 +++
 arch/x86/Kconfig              |    1 +
 arch/x86/include/asm/cycles.h |   11 +++++++++++
 arch/x86/kernel/apic/hw_nmi.c |    7 -------
 kernel/watchdog.c             |    3 ++-
 lib/Kconfig.debug             |    1 +
 6 files changed, 18 insertions(+), 8 deletions(-)
 create mode 100644 arch/x86/include/asm/cycles.h
arch/Kconfig +3 -0
index 52ad235..cc4d14a 100644
@@ -398,6 +398,9 @@ config HAVE_SCHED_CLOCK_NMI
 	help
 	  Architecture's sched_clock() implementation is safely callable from  NMIs.
 
+config HAVE_NSECS_TO_CYCLES
+	bool
+
 #
 # ABI hall of shame
 #
arch/x86/Kconfig +1 -0
index b32ebf9..7fbda5c 100644
@@ -124,6 +124,7 @@ config X86
 	select COMPAT_OLD_SIGACTION if IA32_EMULATION
 	select RTC_LIB
 	select HAVE_DEBUG_STACKOVERFLOW
+	select HAVE_NSECS_TO_CYCLES
 
 config INSTRUCTION_DECODER
 	def_bool y
arch/x86/include/asm/cycles.h new file +11 -0
new file mode 100644
index 0000000..7dedeb3
@@ -0,0 +1,11 @@
+#ifndef _ASM_X86_CYCLES_H
+#define _ASM_X86_CYCLES_H
+
+#include <linux/time.h>
+#include <asm/tsc.h>
+
+static inline u64 nsecs_to_cycles(u64 nsecs)
+{
+	return (u64)((cpu_khz) * nsecs) / NSEC_PER_MSEC;
+}
+#endif /* #ifndef _ASM_X86_CYCLES_H */
arch/x86/kernel/apic/hw_nmi.c +0 -7
index a698d71..4015906 100644
@@ -19,13 +19,6 @@
 #include <linux/module.h>
 #include <linux/delay.h>
 
-#ifdef CONFIG_HARDLOCKUP_DETECTOR
-u64 hw_nmi_get_sample_period(int watchdog_thresh)
-{
-	return (u64)(cpu_khz) * 1000 * watchdog_thresh;
-}
-#endif
-
 #ifdef arch_trigger_all_cpu_backtrace
 /* For reliability, we're prepared to waste bits here. */
 static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly;
kernel/watchdog.c +2 -1
index 1241d8c..e04887c 100644
@@ -26,6 +26,7 @@
 #include <linux/sched/rt.h>
 
 #include <asm/irq_regs.h>
+#include <asm/cycles.h>
 #include <linux/kvm_para.h>
 #include <linux/perf_event.h>
 
@@ -417,7 +418,7 @@ static int watchdog_nmi_enable(unsigned int cpu)
 		goto out_enable;
 
 	wd_attr = &wd_hw_attr;
-	wd_attr->sample_period = hw_nmi_get_sample_period(watchdog_thresh);
+	wd_attr->sample_period = nsecs_to_cycles(watchdog_thresh * NSEC_PER_SEC);
 
 	/* Try to register using hardware perf events */
 	event = perf_event_create_kernel_counter(wd_attr, cpu, NULL, watchdog_overflow_callback, NULL);
lib/Kconfig.debug +1 -0
index 1501aa5..568179d 100644
@@ -634,6 +634,7 @@ config HARDLOCKUP_DETECTOR
 	def_bool y
 	depends on LOCKUP_DETECTOR && !HAVE_NMI_WATCHDOG
 	depends on PERF_EVENTS && HAVE_PERF_EVENTS_NMI
+	depends on HAVE_NSECS_TO_CYCLES
 
 config BOOTPARAM_HARDLOCKUP_PANIC
 	bool "Panic (Reboot) On Hard Lockups"
Signature · 1 line
1.7.5.4
#3 lore raw

Re: [RFC PATCH 2/6] x86: nsecs to cycles conversion

Quoted 5 lines · Frederic Weisbecker

hw_nmi_get_sample_period() is simply a conversion from a period to cycles. Lets generalize the API naming so that it can be used for wider purpose than just watchdog perf event settings. Also it makes the function name less opaque about what it really does.

<snip>

Quoted 40 lines
arch/x86/include/asm/cycles.h new file +11 -0
new file mode 100644
index 0000000..7dedeb3
@@ -0,0 +1,11 @@
+#ifndef _ASM_X86_CYCLES_H
+#define _ASM_X86_CYCLES_H
+
+#include <linux/time.h>
+#include <asm/tsc.h>
+
+static inline u64 nsecs_to_cycles(u64 nsecs)
+{
+	return (u64)((cpu_khz) * nsecs) / NSEC_PER_MSEC;
+}
+#endif /* #ifndef _ASM_X86_CYCLES_H */
arch/x86/kernel/apic/hw_nmi.c +0 -7
index a698d71..4015906 100644
@@ -19,13 +19,6 @@
 #include <linux/module.h>
 #include <linux/delay.h>
 
-#ifdef CONFIG_HARDLOCKUP_DETECTOR
-u64 hw_nmi_get_sample_period(int watchdog_thresh)
-{
-	return (u64)(cpu_khz) * 1000 * watchdog_thresh;
-}
-#endif
-
 #ifdef arch_trigger_all_cpu_backtrace
 /* For reliability, we're prepared to waste bits here. */
 static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly;
kernel/watchdog.c +0 -0
index 1241d8c..e04887c 100644

kernel/watchdog.c line 31

Quoted 6 lines
kernel/watchdog.c +1 -0
@@ -26,6 +26,7 @@
 #include <linux/sched/rt.h>
 
 #include <asm/irq_regs.h>
+#include <asm/cycles.h>
 #include <linux/kvm_para.h>
 #include <linux/perf_event.h>

Do you have to wrap the above with #ifdef CONFIG_HARDLOCKUP_DETECTOR? Otherwise non-x86 boxes won't compile unless I missed stub nsecs_to_cycles() somewhere.

Cheers,
Don

#4 lore raw

Re: [RFC PATCH 2/6] x86: nsecs to cycles conversion

Quoted 51 lines · Don Zickus
Quoted 5 lines · Frederic Weisbecker

hw_nmi_get_sample_period() is simply a conversion from a period to cycles. Lets generalize the API naming so that it can be used for wider purpose than just watchdog perf event settings. Also it makes the function name less opaque about what it really does.

<snip>

Quoted 42 lines
arch/x86/include/asm/cycles.h new file +11 -0
new file mode 100644
index 0000000..7dedeb3
@@ -0,0 +1,11 @@
+#ifndef _ASM_X86_CYCLES_H
+#define _ASM_X86_CYCLES_H
+
+#include <linux/time.h>
+#include <asm/tsc.h>
+
+static inline u64 nsecs_to_cycles(u64 nsecs)
+{
+	return (u64)((cpu_khz) * nsecs) / NSEC_PER_MSEC;
+}
+#endif /* #ifndef _ASM_X86_CYCLES_H */
arch/x86/kernel/apic/hw_nmi.c +0 -7
index a698d71..4015906 100644
@@ -19,13 +19,6 @@
 #include <linux/module.h>
 #include <linux/delay.h>
 
-#ifdef CONFIG_HARDLOCKUP_DETECTOR
-u64 hw_nmi_get_sample_period(int watchdog_thresh)
-{
-	return (u64)(cpu_khz) * 1000 * watchdog_thresh;
-}
-#endif
-
 #ifdef arch_trigger_all_cpu_backtrace
 /* For reliability, we're prepared to waste bits here. */
 static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly;
kernel/watchdog.c +0 -0
index 1241d8c..e04887c 100644
@@ -26,6 +26,7 @@
 #include <linux/sched/rt.h>
 
Quoted 8 lines

kernel/watchdog.c line 31

Quoted 4 lines
kernel/watchdog.c +1 -0
 #include <asm/irq_regs.h>
+#include <asm/cycles.h>
 #include <linux/kvm_para.h>
 #include <linux/perf_event.h>

Do you have to wrap the above with #ifdef CONFIG_HARDLOCKUP_DETECTOR? Otherwise non-x86 boxes won't compile unless I missed stub nsecs_to_cycles() somewhere.

Good catch, I forgot that.

Thanks.

Quoted 3 lines

Cheers,
Don

#5 lore raw

RFC PATCH 3/6 x86: Tell that sched clock is callable in nmi

In reply to Frederic Weisbecker

The x86 version of sched_clock() seem to be safely callable from NMIs, tell about that.

Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc John Stultz <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Don Zickus <[email protected]>
---
 arch/x86/Kconfig |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
arch/x86/Kconfig +1 -0
index 7fbda5c..fc7c34c 100644
@@ -125,6 +125,7 @@ config X86
 	select RTC_LIB
 	select HAVE_DEBUG_STACKOVERFLOW
 	select HAVE_NSECS_TO_CYCLES
+	select HAVE_SCHED_CLOCK_NMI
 
 config INSTRUCTION_DECODER
 	def_bool y
Signature · 1 line
1.7.5.4
#6 lore raw

RFC PATCH 4/6 seqlock: Add raw_seqbegin() for non-waiting readers

In reply to Frederic Weisbecker

Add the seqlock version of raw_seqcount_begin() for readers that can't wait for writers to complete their update and who can live with seqlock read acquirement failure.

Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc John Stultz <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Don Zickus <[email protected]>
---
 include/linux/seqlock.h |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
include/linux/seqlock.h +5 -0
index 1829905..aa845de 100644
@@ -203,6 +203,11 @@ static inline unsigned read_seqbegin(const seqlock_t *sl)
 	return read_seqcount_begin(&sl->seqcount);
 }
 
+static inline unsigned raw_seqbegin(const seqlock_t *sl)
+{
+	return raw_seqcount_begin(&sl->seqcount);
+}
+
 static inline unsigned read_seqretry(const seqlock_t *sl, unsigned start)
 {
 	return read_seqcount_retry(&sl->seqcount, start);
Signature · 1 line
1.7.5.4
#7 lore raw

RFC PATCH 5/6 jiffies: Add jiffies_to_nsecs

In reply to Frederic Weisbecker

This is going to be used by the timekeeping debugger.

It's a first naive version, at least to make sure that future ad hoc implementations don't poliferate.

Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc John Stultz <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Don Zickus <[email protected]>
---
 include/linux/jiffies.h |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
include/linux/jiffies.h +6 -0
index 97ba4e7..77a24da 100644
@@ -294,6 +294,12 @@ extern unsigned long preset_lpj;
  */
 extern unsigned int jiffies_to_msecs(const unsigned long j);
 extern unsigned int jiffies_to_usecs(const unsigned long j);
+
+static inline u64 jiffies_to_nsecs(const unsigned long j)
+{
+	return jiffies_to_usecs(j) * NSEC_PER_USEC;
+}
+
 extern unsigned long msecs_to_jiffies(const unsigned int m);
 extern unsigned long usecs_to_jiffies(const unsigned int u);
 extern unsigned long timespec_to_jiffies(const struct timespec *value);
Signature · 1 line
1.7.5.4
#8 lore raw

RFC PATCH 6/6 timekeeping: Debug missing timekeeping updates

In reply to Frederic Weisbecker

With the full dynticks feature and the tricky full system idle detection code that is coming soon, it becomes necessary to have some debug code that makes sure that the timekeeping is always maintained and moving forward as expected.

This provides a simple detection of missing timekeeping updates inspired by the lockup detector's use of CPU cycles clock.

The jiffies are compared to the cpu clock after several snapshots taken from NMIs that trigger after arbitrary CPU cycles period overflow.

If the jiffies progression appears to drift too far away from the CPU clock's, this triggers a warning.

We just make sure not to account the tiny code on irq entry that may have stale jiffies values before tick_check_nohz() is called after the CPU is woken up while the system went full idle for some time.

Same goes for idle exit in case the tick were stopped but idle was polling on need_resched().

Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc John Stultz <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Don Zickus <[email protected]>
---
 include/linux/time.h               |   11 +++
 kernel/time/Makefile               |    1 +
 kernel/time/tick-sched.c           |    4 +
 kernel/time/timekeeping.c          |    1 +
 kernel/time/timekeeping_selftest.c |  125 ++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug                  |   12 ++++
 6 files changed, 154 insertions(+), 0 deletions(-)
 create mode 100644 kernel/time/timekeeping_selftest.c
include/linux/time.h +11 -0
index d5d229b..39d8493 100644
@@ -181,6 +181,17 @@ extern s32 timekeeping_get_tai_offset(void);
 extern void timekeeping_set_tai_offset(s32 tai_offset);
 extern void timekeeping_clocktai(struct timespec *ts);
 
+#ifdef CONFIG_DEBUG_TIMEKEEPING
+extern void timekeeping_selftest_update(void);
+extern void timekeeping_selftest_start(void);
+extern void timekeeping_selftest_stop(void);
+
+#else
+static inline void timekeeping_selftest_update(void) { }
+static inline void timekeeping_selftest_start(void) { }
+static inline void timekeeping_selftest_stop(void) { }
+#endif
+
 struct tms;
 extern void do_sys_times(struct tms *);
 
kernel/time/Makefile +1 -0
index 9250130..cbea072 100644
@@ -9,3 +9,4 @@ obj-$(CONFIG_TICK_ONESHOT)			+= tick-oneshot.o
 obj-$(CONFIG_TICK_ONESHOT)			+= tick-sched.o
 obj-$(CONFIG_TIMER_STATS)			+= timer_stats.o
 obj-$(CONFIG_DEBUG_FS)				+= timekeeping_debug.o
+obj-$(CONFIG_DEBUG_TIMEKEEPING)			+= timekeeping_selftest.o
kernel/time/tick-sched.c +4 -0
index e77edc9..dcba0e1 100644
@@ -771,6 +771,8 @@ static void __tick_nohz_idle_enter(struct tick_sched *ts)
 
 		if (!was_stopped && ts->tick_stopped)
 			ts->idle_jiffies = ts->last_jiffies;
+
+		timekeeping_selftest_stop();
 	}
 }
 
@@ -937,6 +939,7 @@ void tick_nohz_idle_exit(void)
 	if (ts->tick_stopped) {
 		tick_nohz_restart_sched_tick(ts, now);
 		tick_nohz_account_idle_ticks(ts);
+		timekeeping_selftest_start();
 	}
 
 	local_irq_enable();
@@ -1049,6 +1052,7 @@ static inline void tick_check_nohz(int cpu)
 	if (ts->tick_stopped) {
 		tick_nohz_update_jiffies(now);
 		tick_nohz_kick_tick(cpu, now);
+		timekeeping_selftest_start();
 	}
 }
 
kernel/time/timekeeping.c +1 -0
index 48b9fff..1dcb216 100644
@@ -1583,6 +1583,7 @@ struct timespec get_monotonic_coarse(void)
 void do_timer(unsigned long ticks)
 {
 	jiffies_64 += ticks;
+	timekeeping_selftest_update();
 	update_wall_time();
 	calc_global_load(ticks);
 }
kernel/time/timekeeping_selftest.c new file +125 -0

125 lines not on this page · raw message

lib/Kconfig.debug +12 -0
index 568179d..419f2e9 100644
@@ -788,6 +788,18 @@ config TIMER_STATS
 	  (it defaults to deactivated on bootup and will only be activated
 	  if some application like powertop activates it explicitly).
 
+config DEBUG_TIMEKEEPING
+	bool "Debug timekeeping"
+	depends on HAVE_SCHED_CLOCK_NMI
+	depends on PERF_EVENTS && HAVE_PERF_EVENTS_NMI && HAVE_NSECS_TO_CYCLES
+	help
+	  This enables detection of missing timekeeping updates. This option
+	  is mostly useful to check the correctness of the full system idle
+	  detection used by full dynticks to adaptively run the timekeeper
+	  in order to keep reasonable power consumption. But it can also be
+	  used anytime to ensure that timekeeping is correctly moving foward
+	  when it is expected to.
+
 config DEBUG_PREEMPT
 	bool "Debug preemptible kernel"
 	depends on DEBUG_KERNEL && PREEMPT && TRACE_IRQFLAGS_SUPPORT
Signature · 1 line
1.7.5.4
#9 lore raw

Re: [RFC PATCH 6/6] timekeeping: Debug missing timekeeping updates

Quoted 14 lines · Frederic Weisbecker

With the full dynticks feature and the tricky full system idle detection code that is coming soon, it becomes necessary to have some debug code that makes sure that the timekeeping is always maintained and moving forward as expected.

This provides a simple detection of missing timekeeping updates inspired by the lockup detector's use of CPU cycles clock.

The jiffies are compared to the cpu clock after several snapshots taken from NMIs that trigger after arbitrary CPU cycles period overflow.

If the jiffies progression appears to drift too far away from the CPU clock's, this triggers a warning.

Quoted 7 lines

We just make sure not to account the tiny code on irq entry that may have stale jiffies values before tick_check_nohz() is called after the CPU is woken up while the system went full idle for some time.

Same goes for idle exit in case the tick were stopped but idle was polling on need_resched().

So you're using sched_clock to try to detect timekeeping inconsistencies. Hrm.. Do you have some examples of where this debug infrastructure helped out?

A few thoughts:

1) Why are you using jiffies as the timekeeping reference instead of reading some of actual timekeeping values? Jiffies usage has been intentionally on the decline, and since the dynticks infrastructure landed, jiffies are just derived from the timekeeping core, so its so its sort of strange to see it used for this.

2) This seems very similar to the old lost-ticks compensation code we had prior to the clocksource infrastructure, and seems like it might suffer from some of the issues seen there. For instance, sched_clock has been historically looser in its correctness requirements then the timekeeping code, so using it to validate the more strict timekeeping code, makes me worry we might see cases of false positives.

3) I'm also curious (maybe skeptical) as if sched_clock is reliable enough to use for validating time, then we likely are using that same hardware as the timekeeping clocksource. Thus cases where I'd suspect you'd see likely issues w/ nohz, like clocksource counter overflows being missed on quick wrapping clcoksources wouldn't really apply.

Personally, I've been thinking the timekeeping update code could use some improvements/warnings around cases where update delay is larger then the clocksource max_deferment - possibly falling back to a slower overflow-proof multiply as is done in the CLOCK_SOURCE_SUSPEND_NONSTOP resume case. This would allow more robust behaivor in cases like kvm guests being paused for unreasonable lengths of time, and could also provide very similar NOHZ debug warnings (assuming the clocksource doesn't wrap quickly - but again, in those cases, I'm not confident we can trust sched_clock either).

thanks
-john

#10 lore raw

Re: [RFC PATCH 6/6] timekeeping: Debug missing timekeeping updates

Quoted 20 lines · John Stultz
Quoted 19 lines · Frederic Weisbecker

With the full dynticks feature and the tricky full system idle detection code that is coming soon, it becomes necessary to have some debug code that makes sure that the timekeeping is always maintained and moving forward as expected.

This provides a simple detection of missing timekeeping updates inspired by the lockup detector's use of CPU cycles clock.

The jiffies are compared to the cpu clock after several snapshots taken from NMIs that trigger after arbitrary CPU cycles period overflow.

If the jiffies progression appears to drift too far away from the CPU clock's, this triggers a warning.

We just make sure not to account the tiny code on irq entry that may have stale jiffies values before tick_check_nohz() is called after the CPU is woken up while the system went full idle for some time.

Quoted 6 lines
Quoted 2 lines

Same goes for idle exit in case the tick were stopped but idle was polling on need_resched().

So you're using sched_clock to try to detect timekeeping inconsistencies. Hrm.. Do you have some examples of where this debug infrastructure helped out?

Yeah, currently the full dynticks implies to keep a CPU with the tick alive all the time. This way we make sure that busy CPUs have a reliable timekeeping even when they run tickless.

We could make the timekeeper go to sleep when the system is fully idle and wake it up when there is at least a full dynticks CPU alive. But the simple concept of full idle detection is actually not that obvious. We need to maintain some atomic counter of busy CPUs, send an IPI to the timekeeper when a single CPU wakes up, and order all that correctly.

But maintaining such atomic counter is bad for scalability.

So Paul MckKenney is working on a full system idle detection that reuse the RCU extended grace period detection infrastructure. He enhanced it with a state machine based on smp barriers and atomic ops. The thing appears to be much more scalable that a single busy CPU counters.

All in one it makes sure that timekeeping is always well maintained while keeping reasonable power consumption by allowing the timekeeper to sleep when it considers that it's time to do so.

I don't understand the state macchine completely though, so my brain takes it as a lemma. And in any case it's a quite complicated piece on which the timekeeping progression depends.

So I thought we really need to start adding some automated detection of missing timekeeping.

Quoted 8 lines

A few thoughts:

1) Why are you using jiffies as the timekeeping reference instead of reading some of actual timekeeping values? Jiffies usage has been intentionally on the decline, and since the dynticks infrastructure landed, jiffies are just derived from the timekeeping core, so its so its sort of strange to see it used for this.

That's because it's a simple counter so it's simpler to compute diffs and comparisons on top of it. I prefered to use it instead of gtod values to keep the NMI fast check path simple enough.

Quoted 7 lines

2) This seems very similar to the old lost-ticks compensation code we had prior to the clocksource infrastructure, and seems like it might suffer from some of the issues seen there. For instance, sched_clock has been historically looser in its correctness requirements then the timekeeping code, so using it to validate the more strict timekeeping code, makes me worry we might see cases of false positives.

Yeah I was worried about that too. That's why that detection allows large drifts between jiffies and sched clock (around 0.5 secs).

Do you think of other clock base I could use instead? sched clock also has the advantage to be fast and NMI-safe, at least in x86.

Quoted 6 lines

3) I'm also curious (maybe skeptical) as if sched_clock is reliable enough to use for validating time, then we likely are using that same hardware as the timekeeping clocksource. Thus cases where I'd suspect you'd see likely issues w/ nohz, like clocksource counter overflows being missed on quick wrapping clcoksources wouldn't really apply.

Hmm, but I thought timekeeping_max_deferment would takes care of clocksource overflows?

Quoted 11 lines

Personally, I've been thinking the timekeeping update code could use some improvements/warnings around cases where update delay is larger then the clocksource max_deferment - possibly falling back to a slower overflow-proof multiply as is done in the CLOCK_SOURCE_SUSPEND_NONSTOP resume case. This would allow more robust behaivor in cases like kvm guests being paused for unreasonable lengths of time, and could also provide very similar NOHZ debug warnings (assuming the clocksource doesn't wrap quickly - but again, in those cases, I'm not confident we can trust sched_clock either).

May be. I don't know enough the details of timekeeping to debate there :)

Thanks.