clocksource: Warn if too many missing ticks are detected
PATCH v2 1/1 clocksource: Warn if too many missing ticks are detected
The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when nohz_full is specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.
That particular bug is fixed by the 4.19 commit 7059b36636beab ("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.
--- kernel/time/clocksource.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/time/clocksource.c +12 -0
@@ -213,6 +213,18 @@ static void clocksource_watchdog(struct timer_list *unused)if (!watchdog_running) goto out; +/* +* When the timer tick is incorrectly stopped on a CPU with +* pending events, for example, it is possible that the +* clocksource watchdog will stop running for a sufficiently +* long enough time to cause overflow in the delta +* computation leading to incorrect report of unstable clock +* source. So print a warning if there is unusually large +* delay (> 0.5s) in the invocation of the watchdog. That +* can indicate a hidden bug in the timer tick code. +*/ +WARN_ON_ONCE (jiffies - watchdog_timer.expires >WATCHDOG_INTERVAL ); + reset_pending =atomic_read (&watchdog_reset_pending);list_for_each_entry (cs, &watchdog_list, wd_list) {
Signature · 1 line
1.8.3.1
Quoted 7 lines · Waiman Long
The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when
nohz_fullis specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.
Quoted 5 lines
That particular bug is fixed by the 4.19 commit
7059b36636beab("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.
Second thoughts on this. Putting the check into the clocksource watchdog is the wrong place as it's just checking at a place where the symptom shows. What about putting it right to the source, i.e. in the timer wheel as it does not depend on the clocksource watchdog being active. The clocksource watchdog triggering is just one of the symptoms, but in general timers being massively late is not a good thing.
Thanks,
tglx
Quoted 15 lines · Thomas Gleixner
Quoted 7 lines · Waiman Long
The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when
nohz_fullis specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.Quoted 5 lines
That particular bug is fixed by the 4.19 commit
7059b36636beab("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.
Quoted 6 lines
Second thoughts on this. Putting the check into the clocksource watchdog is the wrong place as it's just checking at a place where the symptom shows. What about putting it right to the source, i.e. in the timer wheel as it does not depend on the clocksource watchdog being active. The clocksource watchdog triggering is just one of the symptoms, but in general timers being massively late is not a good thing.
Just make sure to think of the virt case; virt can cause all kind of 'fun' lateness.
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Waiman Long
The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when
nohz_fullis specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.That particular bug is fixed by the 4.19 commit
7059b36636beab("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.
Quoted 8 lines
Quoted 6 lines
Second thoughts on this. Putting the check into the clocksource watchdog is the wrong place as it's just checking at a place where the symptom shows. What about putting it right to the source, i.e. in the timer wheel as it does not depend on the clocksource watchdog being active. The clocksource watchdog triggering is just one of the symptoms, but in general timers being massively late is not a good thing.
Just make sure to think of the virt case; virt can cause all kind of 'fun' lateness.
So the question now is how much "lateness" is abnormal? My current patch just use a 1/2 s threshold. Is that too small or too big?
Cheers,
Longman
Quoted 21 lines · Thomas Gleixner
Quoted 7 lines · Waiman Long
The clocksource watchdog, when running, is scheduled on all the CPUs in the system sequentially on a round-robin fashion with a period of 0.5s. A bug in the 4.18 kernel is causing missing ticks when
nohz_fullis specified. Under some circumstances, this causes the watchdog to incorrectly state that the TSC is unstable because of counter overflow in the hpet watchdog clock source after a few minutes delay.Quoted 5 lines
That particular bug is fixed by the 4.19 commit
7059b36636beab("sched:
idle: Avoid retaining the tick when it has been stopped"). To make it easier to catch this kind of bug in the future, a check is added to see if there is too much delay in the invocation of the watchdog callback and print a warning once if it happens.Second thoughts on this. Putting the check into the clocksource watchdog is the wrong place as it's just checking at a place where the symptom shows. What about putting it right to the source, i.e. in the timer wheel as it does not depend on the clocksource watchdog being active. The clocksource watchdog triggering is just one of the symptoms, but in general timers being massively late is not a good thing.
Quoted 4 lines
Thanks,
tglx
It looks like there are more complications to deal with in order to do the proper check. I will leave the check in the watchdog for the time being and see how thing works out.
-Longman