Skip to contentSkip to conversation

clocksource: improve unstable clocksource detection

PATCH In review Shaohua Li 2 participants 4 messages Aug 5, 2015 – Aug 17, 2015 1 message not in the archive Timekeeping, clocksource core, NTP, alarmtimer

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Shaohua Li

author
#1 lore raw

PATCH 1/2 clocksource: improve unstable clocksource detection

Quoted 1 line

From time to time we saw TSC is marked as unstable in our systems, while

the CPUs declare to have stable TSC. Looking at the clocksource unstable detection, there are two problems:
- watchdog clock source wrap. HPET is the most common watchdog clock source. It's 32-bit and runs in 14.3Mhz. That means the hpet counter can wrap in about 5 minutes.
- threshold isn't scaled against interval. The threshold is 0.0625s in 0.5s interval. What if the actual interval is bigger than 0.5s?

The watchdog runs in a timer bh, so hard/soft irq can defer its running. Heavy network stack softirq can hog a cpu. IPMI driver can disable interrupt for a very long time. The first problem is mostly we are suffering I think.

Here is a simple patch to fix the issues. If the waterdog doesn't run for a long time, we ignore the detection. This should work for the two problems. For the second one, we probably doen't need to scale if the interval isn't very long.

Cc John Stultz <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/time/clocksource.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/time/clocksource.c +5 -2
index 841b72f..8417c83 100644
@@ -122,9 +122,10 @@ static int clocksource_watchdog_kthread(void *data);
 static void __clocksource_change_rating(struct clocksource *cs, int rating);
 
 /*
- * Interval: 0.5sec Threshold: 0.0625s
+ * Interval: 0.5sec MaxInterval: 1s Threshold: 0.0625s
  */
 #define WATCHDOG_INTERVAL (HZ >> 1)
+#define WATCHDOG_MAX_INTERVAL_NS (NSEC_PER_SEC)
 #define WATCHDOG_THRESHOLD (NSEC_PER_SEC >> 4)
 
 static void clocksource_watchdog_work(struct work_struct *work)
@@ -217,7 +218,9 @@ static void clocksource_watchdog(unsigned long data)
 			continue;
 
 		/* Check the deviation from the watchdog clocksource. */
-		if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD)) {
+		if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD) &&
+		    cs_nsec < WATCHDOG_MAX_INTERVAL_NS &&
+		    wd_nsec < WATCHDOG_MAX_INTERVAL_NS) {
 			pr_warn("timekeeping watchdog: Marking clocksource '%s' as unstable because the skew is too large:\n",
 				cs->name);
 			pr_warn("                      '%s' wd_now: %llx wd_last: %llx mask: %llx\n",
Signature · 1 line
1.8.5.6
author
#2 lore raw

ping, any comments?

Quoted 57 lines · Shaohua Li

From time to time we saw TSC is marked as unstable in our systems, while the CPUs declare to have stable TSC. Looking at the clocksource unstable detection, there are two problems:
- watchdog clock source wrap. HPET is the most common watchdog clock source. It's 32-bit and runs in 14.3Mhz. That means the hpet counter can wrap in about 5 minutes.
- threshold isn't scaled against interval. The threshold is 0.0625s in 0.5s interval. What if the actual interval is bigger than 0.5s?

The watchdog runs in a timer bh, so hard/soft irq can defer its running. Heavy network stack softirq can hog a cpu. IPMI driver can disable interrupt for a very long time. The first problem is mostly we are suffering I think.

Here is a simple patch to fix the issues. If the waterdog doesn't run for a long time, we ignore the detection. This should work for the two problems. For the second one, we probably doen't need to scale if the interval isn't very long.

Cc John Stultz <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/time/clocksource.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/time/clocksource.c +5 -2
index 841b72f..8417c83 100644
@@ -122,9 +122,10 @@ static int clocksource_watchdog_kthread(void *data);
 static void __clocksource_change_rating(struct clocksource *cs, int rating);
 
 /*
- * Interval: 0.5sec Threshold: 0.0625s
+ * Interval: 0.5sec MaxInterval: 1s Threshold: 0.0625s
  */
 #define WATCHDOG_INTERVAL (HZ >> 1)
+#define WATCHDOG_MAX_INTERVAL_NS (NSEC_PER_SEC)
 #define WATCHDOG_THRESHOLD (NSEC_PER_SEC >> 4)
 
 static void clocksource_watchdog_work(struct work_struct *work)
@@ -217,7 +218,9 @@ static void clocksource_watchdog(unsigned long data)
 			continue;
 
 		/* Check the deviation from the watchdog clocksource. */
-		if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD)) {
+		if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD) &&
+		    cs_nsec < WATCHDOG_MAX_INTERVAL_NS &&
+		    wd_nsec < WATCHDOG_MAX_INTERVAL_NS) {
 			pr_warn("timekeeping watchdog: Marking clocksource '%s' as unstable because the skew is too large:\n",
 				cs->name);
 			pr_warn("                      '%s' wd_now: %llx wd_last: %llx mask: %llx\n",
Signature · 1 line
1.8.5.6
#3 lore raw
Quoted 49 lines · Shaohua Li

ping, any comments?

Quoted 46 lines · Shaohua Li

From time to time we saw TSC is marked as unstable in our systems, while the CPUs declare to have stable TSC. Looking at the clocksource unstable detection, there are two problems:
- watchdog clock source wrap. HPET is the most common watchdog clock source. It's 32-bit and runs in 14.3Mhz. That means the hpet counter can wrap in about 5 minutes.
- threshold isn't scaled against interval. The threshold is 0.0625s in 0.5s interval. What if the actual interval is bigger than 0.5s?

The watchdog runs in a timer bh, so hard/soft irq can defer its running. Heavy network stack softirq can hog a cpu. IPMI driver can disable interrupt for a very long time. The first problem is mostly we are suffering I think.

Here is a simple patch to fix the issues. If the waterdog doesn't run for a long time, we ignore the detection. This should work for the two problems. For the second one, we probably doen't need to scale if the interval isn't very long.

Cc John Stultz <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/time/clocksource.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/time/clocksource.c +2 -1
index 841b72f..8417c83 100644
@@ -122,9 +122,10 @@ static int clocksource_watchdog_kthread(void *data);
 static void __clocksource_change_rating(struct clocksource *cs, int rating);
 
 /*
- * Interval: 0.5sec Threshold: 0.0625s
+ * Interval: 0.5sec MaxInterval: 1s Threshold: 0.0625s
  */
 #define WATCHDOG_INTERVAL (HZ >> 1)
+#define WATCHDOG_MAX_INTERVAL_NS (NSEC_PER_SEC)
 #define WATCHDOG_THRESHOLD (NSEC_PER_SEC >> 4)
 
 static void clocksource_watchdog_work(struct work_struct *work)
@@ -217,7 +218,9 @@ static void clocksource_watchdog(unsigned long data)
                      continue;
 
Quoted 8 lines

kernel/time/clocksource.c static void clocksource_watchdog(unsigned long data) · line 226

Quoted 8 lines
kernel/time/clocksource.c +3 -1
              /* Check the deviation from the watchdog clocksource. */
-             if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD)) {
+             if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD) &&
+                 cs_nsec < WATCHDOG_MAX_INTERVAL_NS &&
+                 wd_nsec < WATCHDOG_MAX_INTERVAL_NS) {
                      pr_warn("timekeeping watchdog: Marking clocksource '%s' as unstable because the skew is too large:\n",
                              cs->name);
                      pr_warn("                      '%s' wd_now: %llx wd_last: %llx mask: %llx\n",

Sorry. I got back from vacation last week and haven't finished queuing items from my inbox. It looks reasonable to me. I'm hoping to put together my patchset for 4.3, today and this hopefully should make it unless I run into anything concerning.

thanks
-john

author
#4 lore raw

PATCH 2/2 clocksource: sanity check watchdog clocksource

In reply to a message not in the archive

Add a sanity check to make sure watchdog clocksource doesn't wrap too quickly.

Cc John Stultz <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/time/clocksource.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/time/clocksource.c +2 -1
index 8417c83..64e4629 100644
@@ -327,7 +327,8 @@ static void clocksource_enqueue_watchdog(struct clocksource *cs)
 		if (cs->flags & CLOCK_SOURCE_IS_CONTINUOUS)
 			cs->flags |= CLOCK_SOURCE_VALID_FOR_HRES;
 		/* Pick the best watchdog. */
-		if (!watchdog || cs->rating > watchdog->rating) {
+		if (cs->max_idle_ns > WATCHDOG_MAX_INTERVAL_NS &&
+		    (!watchdog || cs->rating > watchdog->rating)) {
 			watchdog = cs;
 			/* Reset watchdog cycles */
 			clocksource_reset_watchdog();
Signature · 1 line
1.8.5.6