Skip to contentSkip to conversation

x86/tsc: future proof native_calibrate_tsc()

PATCH In review Len Brown 2 participants 3 messages Dec 15, 2017 – Jan 2, 2018 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Len Brown

author
#1 lore raw

PATCH 1/3 x86/tsc: future proof native_calibrate_tsc()

From: Len Brown <[email protected]>

Linux-4.10 added X86_FEATURE_TSC_KNOWN_FREQ.

commit 4ca4df0b7eb0
("x86/tsc: Mark TSC frequency determined by CPUID as known")

When it is set, run-time refined TSC calibration is disabled.

But native_calibrate_tsc() sets it on all systems with CPUID.15, even if crystal_khz can not be discovered via CPUID.15 or via built-in table. eg. When a new Intel processor comes out that does not return CPUID.15.crystal_khz, and does not have its model# and crystal_khz listed in native_calibate_tsc().

This results in (tsc_khz = cpu_khz) and TSC refined calibration disabled. But for machines with the TSC and CPU in different clock domains (and those support CPUID.15), almost all such configurations will experience measurable negative timeofday clock drift.

---
 arch/x86/kernel/tsc.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kernel/tsc.c +2 -0
index 8ea117f8142e..49d772672367 100644
@@ -609,6 +609,8 @@ unsigned long native_calibrate_tsc(void)
 		case INTEL_FAM6_ATOM_GOLDMONT:
 			crystal_khz = 19200;	/* 19.2 MHz */
 			break;
+		default:
+			return 0;	/* no X86_FEATURE_TSC_KNOWN_FREQ */
 		}
 	}
 
Signature · 1 line
2.14.0-rc0
author
#2 lore raw

PATCH 2/3 x86/tsc: Fix erroneous TSC rate on Skylake Xeon

In reply to a message not in the archive

From: Len Brown <[email protected]>

Linux-4.9 added INTEL_FAM6_SKYLAKE_X to native_calibrate_tsc(), allowing the kernel to calculate the TSC frequency, instead of calibrating:

commit 6baf3d61821f
("x86/tsc: Add additional Intel CPU models to the crystal quirk list")

There are several problems with doing this.

The first is that while SKX servers use a 25 MHz crystal, SKX workstations (with same model #) use a 24 MHz crystal. This results in a -4.0% time drift rate on SKX workstations.

But even SKX servers have a problem.
SKX subjects the crystal to an EMI reduction circuit that reduces its actual value by (approximately) -0.25%. This results in -1 second per 10 minute time drift as compared to network time on SKX servers.

This also results in an even more subtle symptom on systems that use the LAPIC timer (versus the TSC deadline timer). Clock ticks scheduled with the LAPIC timer arrive a few usec before the time they are expected (according to the slow TSC). This causes Linux to poll-idle, when it should be in an idle power saving state. The idle and clock code do not graciously recover from this error, sometimes resulting in significant polling and significant power impact.

So stop using native_calibrate_tsc() for INTEL_FAM6_SKYLAKE_X. native_calibrate_tsc() will return 0, boot will run with tsc_khz = cpu_khz, and the TSC refined calibration will update tsc_khz to correct for the difference.

Yes, a future patch can detect 24 vs 25 MHz crystals, and it can estimate a correction for the EMI circuit impact. However, that would effect boot-time only, as we'd probably choose to keep the refined TSC calibration, because the EMI correction estimate may not be precise.

Cc Prarit Bhargava <[email protected]>
Cc <[email protected]> # v4.9+
---
 arch/x86/kernel/tsc.c | 1 -
 1 file changed, 1 deletion(-)
arch/x86/kernel/tsc.c +0 -1
index 49d772672367..19ca0004caf6 100644
@@ -602,7 +602,6 @@ unsigned long native_calibrate_tsc(void)
 		case INTEL_FAM6_KABYLAKE_DESKTOP:
 			crystal_khz = 24000;	/* 24.0 MHz */
 			break;
-		case INTEL_FAM6_SKYLAKE_X:
 		case INTEL_FAM6_ATOM_DENVERTON:
 			crystal_khz = 25000;	/* 25.0 MHz */
 			break;
Signature · 1 line
2.14.0-rc0
#3 lore raw

Re: [PATCH 2/3] x86/tsc: Fix erroneous TSC rate on Skylake Xeon

Quoted 54 lines · Len Brown

From: Len Brown <[email protected]>

Linux-4.9 added INTEL_FAM6_SKYLAKE_X to native_calibrate_tsc(), allowing the kernel to calculate the TSC frequency, instead of calibrating:

commit 6baf3d61821f
("x86/tsc: Add additional Intel CPU models to the crystal quirk list")

There are several problems with doing this.

The first is that while SKX servers use a 25 MHz crystal, SKX workstations (with same model #) use a 24 MHz crystal. This results in a -4.0% time drift rate on SKX workstations.

But even SKX servers have a problem.
SKX subjects the crystal to an EMI reduction circuit that reduces its actual value by (approximately) -0.25%. This results in -1 second per 10 minute time drift as compared to network time on SKX servers.

This also results in an even more subtle symptom on systems that use the LAPIC timer (versus the TSC deadline timer). Clock ticks scheduled with the LAPIC timer arrive a few usec before the time they are expected (according to the slow TSC). This causes Linux to poll-idle, when it should be in an idle power saving state. The idle and clock code do not graciously recover from this error, sometimes resulting in significant polling and significant power impact.

So stop using native_calibrate_tsc() for INTEL_FAM6_SKYLAKE_X. native_calibrate_tsc() will return 0, boot will run with tsc_khz = cpu_khz, and the TSC refined calibration will update tsc_khz to correct for the difference.

Yes, a future patch can detect 24 vs 25 MHz crystals, and it can estimate a correction for the EMI circuit impact. However, that would effect boot-time only, as we'd probably choose to keep the refined TSC calibration, because the EMI correction estimate may not be precise.

Cc Prarit Bhargava <[email protected]>
Cc <[email protected]> # v4.9+
---
 arch/x86/kernel/tsc.c | 1 -
 1 file changed, 1 deletion(-)
arch/x86/kernel/tsc.c +0 -0
index 49d772672367..19ca0004caf6 100644

arch/x86/kernel/tsc.c unsigned long native_calibrate_tsc(void) · line 607

Quoted 7 lines
arch/x86/kernel/tsc.c +0 -1
@@ -602,7 +602,6 @@ unsigned long native_calibrate_tsc(void)
 		case INTEL_FAM6_KABYLAKE_DESKTOP:
 			crystal_khz = 24000;	/* 24.0 MHz */
 			break;
-		case INTEL_FAM6_SKYLAKE_X:
 		case INTEL_FAM6_ATOM_DENVERTON:
 			crystal_khz = 25000;	/* 25.0 MHz */
 			break;

Len, I lifted this code directly from tools/power/x86/turbostat/turbostat.c:4155 -- is that code incorrect too?

P.