Skip to contentSkip to conversation

clocksource: Warn if too many missing ticks are detected

PATCH Superseded Waiman Long 3 participants 5 messages Sep 19, 2018 – Sep 24, 2018 Timekeeping, clocksource core, NTP, alarmtimer

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 clocksource: Warn if too many missing ticks are detected

The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when nohz_full is specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.

That particular bug is fixed by the 4.19 commit 7059b36636beab ("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.

---
 kernel/time/clocksource.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
kernel/time/clocksource.c +12 -0
index 0e6e97a..64b05ab 100644
@@ -213,6 +213,18 @@ static void clocksource_watchdog(struct timer_list *unused)
 	if (!watchdog_running)
 		goto out;
 
+	/*
+	 * When the timer tick is incorrectly stopped on a CPU with
+	 * pending events, for example, it is possible that the
+	 * clocksource watchdog will stop running for a sufficiently
+	 * long enough time to cause overflow in the delta
+	 * computation leading to incorrect report of unstable clock
+	 * source. So print a warning if there is unusually large
+	 * delay (> 0.5s) in the invocation of the watchdog. That
+	 * can indicate a hidden bug in the timer tick code.
+	 */
+	WARN_ON_ONCE(jiffies - watchdog_timer.expires > WATCHDOG_INTERVAL);
+
 	reset_pending = atomic_read(&watchdog_reset_pending);
 
 	list_for_each_entry(cs, &watchdog_list, wd_list) {
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 7 lines · Waiman Long

The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when nohz_full is specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.

Quoted 5 lines

That particular bug is fixed by the 4.19 commit 7059b36636beab ("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.

Second thoughts on this. Putting the check into the clocksource watchdog is the wrong place as it's just checking at a place where the symptom shows. What about putting it right to the source, i.e. in the timer wheel as it does not depend on the clocksource watchdog being active. The clocksource watchdog triggering is just one of the symptoms, but in general timers being massively late is not a good thing.

Thanks,

tglx

#2 lore raw
Quoted 15 lines · Thomas Gleixner
Quoted 7 lines · Waiman Long

The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when nohz_full is specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.

Quoted 5 lines

That particular bug is fixed by the 4.19 commit 7059b36636beab ("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.

Quoted 6 lines

Second thoughts on this. Putting the check into the clocksource watchdog is the wrong place as it's just checking at a place where the symptom shows. What about putting it right to the source, i.e. in the timer wheel as it does not depend on the clocksource watchdog being active. The clocksource watchdog triggering is just one of the symptoms, but in general timers being massively late is not a good thing.

Just make sure to think of the virt case; virt can cause all kind of 'fun' lateness.

#3 lore raw
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Waiman Long

The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when nohz_full is specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.

That particular bug is fixed by the 4.19 commit 7059b36636beab ("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.

Quoted 8 lines
Quoted 6 lines

Second thoughts on this. Putting the check into the clocksource watchdog is the wrong place as it's just checking at a place where the symptom shows. What about putting it right to the source, i.e. in the timer wheel as it does not depend on the clocksource watchdog being active. The clocksource watchdog triggering is just one of the symptoms, but in general timers being massively late is not a good thing.

Just make sure to think of the virt case; virt can cause all kind of 'fun' lateness.

So the question now is how much "lateness" is abnormal? My current patch just use a 1/2 s threshold. Is that too small or too big?

Cheers,
Longman

#4 lore raw
In reply to Thomas Gleixner
Quoted 21 lines · Thomas Gleixner
Quoted 7 lines · Waiman Long

The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when nohz_full is specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.

Quoted 5 lines

That particular bug is fixed by the 4.19 commit 7059b36636beab ("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.

Second thoughts on this. Putting the check into the clocksource watchdog is the wrong place as it's just checking at a place where the symptom shows. What about putting it right to the source, i.e. in the timer wheel as it does not depend on the clocksource watchdog being active. The clocksource watchdog triggering is just one of the symptoms, but in general timers being massively late is not a good thing.

Quoted 4 lines

Thanks,

tglx

It looks like there are more complications to deal with in order to do the proper check. I will leave the check in the watchdog for the time being and see how thing works out.

-Longman