x86: Export tsc related information in sysfs
Subsystems (2)
Quoted 2 lines · Ingo Molnar
The point is for the kernel to not be complicit in practices that are technically not reliable.
One usecase that hasn't been discussed is when userspace needs this info to calibrate the TSC.
Take NTP as an example. It does a pretty good job of observing the drift in gettimeofday() against a reference clock and correcting for it. This seems to work well even when GTOD uses the TSC. But, it assumes that the drift changes slowly.
That goes out the window on reboot, because the kernel only spends 25ms on TSC<->PIT calibration and the value of tsc_khz can vary a lot from boot to boot. Then NTP starts up and reads a drift value from /var/lib/ntp/ntp.drift that it *thinks* is accurate. In our experience, it'll then spend up to 48 hours doing god knows what to the clock until it converges on the real drift at the new tsc_khz. initscripts could correct for the kernel's recalibration, but tsc_khz isn't exported.
So it's too bad that it can't be exported somehow. The TSC on our machines has proven to be stable for all intents and purposes; I just checked 25 of my machines, most have uptime of >200 days, all of them still have current_clocksource=tsc. After NTP or PTPd has been running for a while, things converge, but being unable to reboot is a headache. Using the HPET for gettimeofday() would be impractical for performance reasons.
Quoted 13 lines · Brian Bloniarz
Take NTP as an example. It does a pretty good job of observing the drift in
gettimeofday()against a reference clock and correcting for it. This seems to work well even when GTOD uses the TSC. But, it assumes that the drift changes slowly.That goes out the window on reboot, because the kernel only spends 25ms on TSC<->PIT calibration and the value of
tsc_khzcan vary a lot from boot to boot. Then NTP starts up and reads a drift value from/var/lib/ntp/ntp.driftthat it *thinks* is accurate. In our experience, it'll then spend up to 48 hours doing god knows what to the clock until it converges on the real drift at the newtsc_khz. initscripts could correct for the kernel's recalibration, buttsc_khzisn't exported.
Quoted 7 lines
So it's too bad that it can't be exported somehow. The TSC on our machines has proven to be stable for all intents and purposes; I just checked 25 of my machines, most have uptime of >200 days, all of them still have current_clocksource=tsc. After NTP or PTPd has been running for a while, things converge, but being unable to reboot is a headache. Using the HPET for
gettimeofday()would be impractical for performance reasons.
Yea, the relative instability of the tsc calibration at boot is an issue for folks who want very very precise timekeeping immediately after a reboot.
I proposed a solution to this awhile back via a boot option users could use to specify the tsc_khz freq, so it would be consistent from boot to boot. See: https://patchwork.kernel.org/patch/22492/
It didn't really go anywhere due to a lack of public interest. However, if you're interested in playing with it, I can try to revive the patch.
thanks
-john
H. Peter Anvin
reviewerQuoted 5 lines · john stultz
Yea, the relative instability of the tsc calibration at boot is an issue for folks who want very very precise timekeeping immediately after a reboot.
Quoted 8 lines
I proposed a solution to this awhile back via a boot option users could use to specify the
tsc_khzfreq, so it would be consistent from boot to boot. See: https://patchwork.kernel.org/patch/22492/It didn't really go anywhere due to a lack of public interest. However, if you're interested in playing with it, I can try to revive the patch.
The point about NTP is a very good one; with NTP you *really* want what you're adjusting to be directly related to the crystal oscillator -- as what it is; precision over accuracy as NTP takes care of accuracy but not precision.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 6 lines
From: john stultz [mailto:[email protected]]
Yea, the relative instability of the tsc calibration at boot is an issue for folks who want very very precise timekeeping immediately after a reboot.
Quoted 7 lines
I proposed a solution to this awhile back via a boot option users could use to specify the
tsc_khzfreq, so it would be consistent from boot to boot. See: https://patchwork.kernel.org/patch/22492/It didn't really go anywhere due to a lack of public interest. However, if you're interested in playing with it, I can try to revive the patch.
Another possibility: Optionally trust the stamped rate for the part?
I understand that on Nehalem this value is available in MSR_PLATFORM_INFO[15:8] (google for MSR_PLATFORM_INFO 15 8), but I don't know if this MSR is available on older (or AMD) processors.
Just wondering: If one were to put an ultra-precise scope on a processor, how far off would the calibrated value be? I'd imagine the process of calibrating one unknown crystal against a second crystal which has a known-but-not-highly-precise frequency, though good enough for most purposes, is not particularly accurate. In other words, maybe the stamped rate is more accurate than the calibrated rate anyway?
H. Peter Anvin
reviewerQuoted 7 lines · Dan Magenheimer
Just wondering: If one were to put an ultra-precise scope on a processor, how far off would the calibrated value be? I'd imagine the process of calibrating one unknown crystal against a second crystal which has a known-but-not-highly-precise frequency, though good enough for most purposes, is not particularly accurate. In other words, maybe the stamped rate is more accurate than the calibrated rate anyway?
No. Not even close.
A spread-spectrum clock is inaccurate by entire percentage points. A non-spread clock is typically ±50 ppm with typical consumer PC oscillators, ±1 ppm with non-crappy but still cheap oscillators (e.g. used in cell phones.)
-hpa
Quoted 14 lines · Dan Magenheimer
Quoted 6 lines
From: john stultz [mailto:[email protected]]
Yea, the relative instability of the tsc calibration at boot is an issue for folks who want very very precise timekeeping immediately after a reboot.
Quoted 7 lines
I proposed a solution to this awhile back via a boot option users could use to specify the
tsc_khzfreq, so it would be consistent from boot to boot. See: https://patchwork.kernel.org/patch/22492/It didn't really go anywhere due to a lack of public interest. However, if you're interested in playing with it, I can try to revive the patch.
Quoted 6 lines
Another possibility: Optionally trust the stamped rate for the part?
I understand that on Nehalem this value is available in MSR_PLATFORM_INFO[15:8] (google for
MSR_PLATFORM_INFO15 8), but I don't know if this MSR is available on older (or AMD) processors.
Hmmm. That could be an option for newer cpus that I wouldn't oppose.
While Peter is correct that the stamped value is probably not very accurate, atleast it would be constant from boot to boot, and NTP's calculated drift value would be correct.
We'd need a check to make sure its not way off, since NTP will give up if its outside 500ppm. So as long as its close to the calibrated value, we probably could use it.
thanks
-john
H. Peter Anvin
reviewerQuoted 11 lines · john stultz
Hmmm. That could be an option for newer cpus that I wouldn't oppose.
While Peter is correct that the stamped value is probably not very accurate, atleast it would be constant from boot to boot, and NTP's calculated drift value would be correct.
We'd need a check to make sure its not way off, since NTP will give up if its outside 500ppm. So as long as its close to the calibrated value, we probably could use it.
Is that still the case? I thought newer versions of NTP could deal with large values. Inaccuracies of way more than 500 ppm are everyday.
-hpa
Quoted 8 lines · H. Peter Anvin
Quoted 7 lines · john stultz
Hmmm. That could be an option for newer cpus that I wouldn't oppose.
While Peter is correct that the stamped value is probably not very accurate, atleast it would be constant from boot to boot, and NTP's calculated drift value would be correct.
Quoted 7 lines
Quoted 4 lines
We'd need a check to make sure its not way off, since NTP will give up if its outside 500ppm. So as long as its close to the calibrated value, we probably could use it.
Is that still the case? I thought newer versions of NTP could deal with large values. Inaccuracies of way more than 500 ppm are everyday.
That's scary.
Yea, in the kernel the ntp freq correction tops out at 500ppm. Almost all the systems I see tend to fall in the +/- 200ppm range (if there's not something terribly wrong with the hardware).
So maybe things aren't so bad out there? Or is that wishful thinking?
thanks
-john
H. Peter Anvin
reviewerQuoted 17 lines · john stultz
Quoted 8 lines · H. Peter Anvin
Quoted 7 lines · john stultz
Hmmm. That could be an option for newer cpus that I wouldn't oppose.
While Peter is correct that the stamped value is probably not very accurate, atleast it would be constant from boot to boot, and NTP's calculated drift value would be correct.
Quoted 7 lines
Quoted 4 lines
We'd need a check to make sure its not way off, since NTP will give up if its outside 500ppm. So as long as its close to the calibrated value, we probably could use it.
Is that still the case? I thought newer versions of NTP could deal with large values. Inaccuracies of way more than 500 ppm are everyday.
Quoted 8 lines
That's scary.
Yea, in the kernel the ntp freq correction tops out at 500ppm. Almost all the systems I see tend to fall in the +/- 200ppm range (if there's not something terribly wrong with the hardware).
So maybe things aren't so bad out there? Or is that wishful thinking?
In the kernel, yes; I thought the ntp daemon itself now handled the exceptions (basically it detects if the PLL consistently veers off the rails and adjusts the timing constants.)
However, you're comparing apples to oranges: you're talking about current kernels, which means a calibrated TSC, which means you're comparing to the non-spread 14.31818 MHz clock (which feeds into the HPET, PMTMR and 8254 on a standard PC platform.) In most PCs this is a separate oscillator from the bus clock which is spread spectrum. As a result, it should be in the ±50 ppm range in theory; in practice as you observe the range is wider.
-hpa
Quoted 11 lines
Quoted 1 line
Is that still the case? I thought newer versions of NTP could deal
with
Quoted 1 line
large values. Inaccuracies of way more than 500 ppm are everyday.
That's scary.
Yea, in the kernel the ntp freq correction tops out at 500ppm. Almost all the systems I see tend to fall in the +/- 200ppm range (if there's not something terribly wrong with the hardware).
So maybe things aren't so bad out there? Or is that wishful thinking?
Since Brian's concern is at boot-time at which point there is no network or ntp, and assuming that it would be unwise to vary tsc_khz dynamically on a clocksource==tsc machine (is it?), would optionally lengthening the TSC<->PIT calibration beyond 25ms result in a more consistent tsc_khz between boots? Or is the relative instability an unavoidable result of skew between the PIT and the fixed constant PIT_TICK_RATE combined with algorithmic/arithmetic error? Or is the jitter of the (spread-spectrum) TSC too extreme? Or ???
If better more consistent calibration is possible, offering that as an optional kernel parameter seems better than specifying a fixed tsc_khz (stamped or user-specified) which may or may not be ignored due to "too different from measured tsc_khz". Even an (*optional*) extra second or two of boot time might be perfectly OK if it resulted in an additional five or six bits of tsc_khz precision.
Thoughts, Brian?
H. Peter Anvin
reviewerQuoted 29 lines · Dan Magenheimer
Quoted 11 lines
Quoted 1 line
Is that still the case? I thought newer versions of NTP could deal
with
Quoted 1 line
large values. Inaccuracies of way more than 500 ppm are everyday.
That's scary.
Yea, in the kernel the ntp freq correction tops out at 500ppm. Almost all the systems I see tend to fall in the +/- 200ppm range (if there's not something terribly wrong with the hardware).
So maybe things aren't so bad out there? Or is that wishful thinking?
Since Brian's concern is at boot-time at which point there is no network or ntp, and assuming that it would be unwise to vary
tsc_khzdynamically on a clocksource==tsc machine (is it?), would optionally lengthening the TSC<->PIT calibration beyond 25ms result in a more consistenttsc_khzbetween boots? Or is the relative instability an unavoidable result of skew between the PIT and the fixed constantPIT_TICK_RATEcombined with algorithmic/arithmetic error? Or is the jitter of the (spread-spectrum) TSC too extreme? Or ???If better more consistent calibration is possible, offering that as an optional kernel parameter seems better than specifying a fixed
tsc_khz(stamped or user-specified) which may or may not be ignored due to "too different from measuredtsc_khz". Even an (*optional*) extra second or two of boot time might be perfectly OK if it resulted in an additional five or six bits oftsc_khzprecision.
Quoted 1 line
Thoughts, Brian?
Making the calibration time longer should give a more precise result, but of course at the expense of longer boot time.
A longer sample would make sense if the goal is to freeze it into a kernel command line variable, but the real question is how many people would actually do that (and how many people would then suffer problems because they upgraded their CPU/mobo and got massive failures on post-boot.)
-hpa
Quoted 32 lines · H. Peter Anvin
Quoted 29 lines · Dan Magenheimer
Quoted 11 lines
Quoted 1 line
Is that still the case? I thought newer versions of NTP could deal
with
Quoted 1 line
large values. Inaccuracies of way more than 500 ppm are everyday.
That's scary.
Yea, in the kernel the ntp freq correction tops out at 500ppm. Almost all the systems I see tend to fall in the +/- 200ppm range (if there's not something terribly wrong with the hardware).
So maybe things aren't so bad out there? Or is that wishful thinking?
Since Brian's concern is at boot-time at which point there is no network or ntp, and assuming that it would be unwise to vary
tsc_khzdynamically on a clocksource==tsc machine (is it?), would optionally lengthening the TSC<->PIT calibration beyond 25ms result in a more consistenttsc_khzbetween boots? Or is the relative instability an unavoidable result of skew between the PIT and the fixed constantPIT_TICK_RATEcombined with algorithmic/arithmetic error? Or is the jitter of the (spread-spectrum) TSC too extreme? Or ???If better more consistent calibration is possible, offering that as an optional kernel parameter seems better than specifying a fixed
tsc_khz(stamped or user-specified) which may or may not be ignored due to "too different from measuredtsc_khz". Even an (*optional*) extra second or two of boot time might be perfectly OK if it resulted in an additional five or six bits oftsc_khzprecision.Quoted 1 line
Thoughts, Brian?
Quoted 7 lines
Making the calibration time longer should give a more precise result, but of course at the expense of longer boot time.
A longer sample would make sense if the goal is to freeze it into a kernel command line variable, but the real question is how many people would actually do that (and how many people would then suffer problems because they upgraded their CPU/mobo and got massive failures on post-boot.)
I'll admit its a feature for a minority of users. Probably why its not included.
And the upgraded system issue was something I tried to address by using the calibrated value if it was off by some unreasonable amount, however folks protested that, figuring since if its explicitly stated kernel should not override it (ie: for the use case of where the calibration is broken and folks want to force the value).
Also, you don't really need extra accuracy, you just need it to be the same from boot to boot. NTP keeps the correction factor persistent from boot to boot via the drift file. The boot argument is just trying to save the time (possibly hours depending on ntp config) after a reboot for NTP to correct for the new error introduced by calibration.
thanks
-john
Quoted 6 lines
From: john stultz [mailto:[email protected]]
Also, you don't really need extra accuracy, you just need it to be the same from boot to boot. NTP keeps the correction factor persistent from boot to boot via the drift file. The boot argument is just trying to save the time (possibly hours depending on ntp config) after a reboot for NTP to correct for the new error introduced by calibration.
I was assuming that extra accuracy would decrease the ntp convergence time by about the same factor (5-6 bits of extra accuracy would decrease ntp convergence time by 32-64x). Is that an incorrect assumption?
Quoted 6 lines
From: H. Peter Anvin [mailto:[email protected]]
A longer sample would make sense if the goal is to freeze it into a kernel command line variable, but the real question is how many people would actually do that (and how many people would then suffer problems because they upgraded their CPU/mobo and got massive failures on post- boot.)
Not sure why upgraded mobo's would fail due to a longer sample?
As more and more systems become dependent on clocksource==tsc and more and more people assume nanosecond-class measurements are relatively accurate, I'd expect the accuracy of tsc_khz to become more important. While desktop users might bristle at an extra second of boot delay, I'll bet many server farm administrators would gladly pay that upfront cost if they know an option exists.
H. Peter Anvin
reviewerQuoted 7 lines · Dan Magenheimer
Quoted 6 lines
From: john stultz [mailto:[email protected]]
Also, you don't really need extra accuracy, you just need it to be the same from boot to boot. NTP keeps the correction factor persistent from boot to boot via the drift file. The boot argument is just trying to save the time (possibly hours depending on ntp config) after a reboot for NTP to correct for the new error introduced by calibration.
Quoted 5 lines
I was assuming that extra accuracy would decrease the ntp convergence time by about the same factor (5-6 bits of extra accuracy would decrease ntp convergence time by 32-64x). Is that an incorrect assumption?
Yes.
Quoted 8 lines
Quoted 6 lines
From: H. Peter Anvin [mailto:[email protected]]
A longer sample would make sense if the goal is to freeze it into a kernel command line variable, but the real question is how many people would actually do that (and how many people would then suffer problems because they upgraded their CPU/mobo and got massive failures on post- boot.)Not sure why upgraded mobo's would fail due to a longer sample?
Not due to a longer sample, but a frozen sample.
Quoted 7 lines
As more and more systems become dependent on clocksource==tsc and more and more people assume nanosecond-class measurements are relatively accurate, I'd expect the accuracy of
tsc_khzto become more important. While desktop users might bristle at an extra second of boot delay, I'll bet many server farm administrators would gladly pay that upfront cost if they know an option exists.
Not really. The delta measurements aren't the issue here, but rather walltime convergence.
-hpa
Quoted 11 lines · Dan Magenheimer
Quoted 6 lines
From: john stultz [mailto:[email protected]]
Also, you don't really need extra accuracy, you just need it to be the same from boot to boot. NTP keeps the correction factor persistent from boot to boot via the drift file. The boot argument is just trying to save the time (possibly hours depending on ntp config) after a reboot for NTP to correct for the new error introduced by calibration.I was assuming that extra accuracy would decrease the ntp convergence time by about the same factor (5-6 bits of extra accuracy would decrease ntp convergence time by 32-64x). Is that an incorrect assumption?
Sorry, this is sort of mixing points. I was saying you don't need more accuracy (as opposed to what H. Peter mentioned below) when setting the tsc_khz= option I proposed. Since it will be constant from boot to boot, and thus will reduce the ntp convergence time.
However, without such a boot option, more accuracy from an increased calibration time would help. However, the tradeoff of a longer boot time is one not many will probably want.
Quoted 8 lines
Quoted 6 lines
From: H. Peter Anvin [mailto:[email protected]]
A longer sample would make sense if the goal is to freeze it into a kernel command line variable, but the real question is how many people would actually do that (and how many people would then suffer problems because they upgraded their CPU/mobo and got massive failures on post- boot.)Not sure why upgraded mobo's would fail due to a longer sample?
Again, this is mixing the discussion. The concern was users of a tsc_khz= boot option might have problems when they upgrade, as the actual TSC freq might not match what was specified at boot.
Quoted 7 lines
As more and more systems become dependent on clocksource==tsc and more and more people assume nanosecond-class measurements are relatively accurate, I'd expect the accuracy of
tsc_khzto become more important. While desktop users might bristle at an extra second of boot delay, I'll bet many server farm administrators would gladly pay that upfront cost if they know an option exists.
Maybe something like a tsc_long_calibration=1 option would allow for this?
However, I really do like the idea of pulling the stamped value from the MSR and if its close to what we quickly calibrated, use that.
thanks
-john
Quoted 10 lines
Quoted 7 lines
As more and more systems become dependent on clocksource==tsc and more and more people assume nanosecond-class measurements are relatively accurate, I'd expect the accuracy of
tsc_khzto become more important. While desktop users might bristle at an extra second of boot delay, I'll bet many server farm administrators would gladly pay that upfront cost if they know an option exists.Maybe something like a tsc_long_calibration=1 option would allow for this?
On a system with synchronized TSC and multiple cores you could also simply do a longer calibration on another core in the background after a quick "fast calibration"
-Andi
Quoted 2 lines
Sorry, this is sort of mixing points
Again, this is mixing the discussion.
Oops, sorry, missed that. :-}
Quoted 2 lines
Maybe something like a tsc_long_calibration=1 option would allow for this?
Sounds good to me. If it's non-obvious what value to choose for the new calibration, maybe specifying it in MS (per MAX_QUICK_PIT_MS in arch/x86/kernel/tsc.c) would be nice.
Quoted 2 lines
However, I really do like the idea of pulling the stamped value from the MSR and if its close to what we quickly calibrated, use that.
On a quick sample of two machines looking at the TSC calibration done by Xen (which exposes the equivalent of tsc_khz), it appears that the stamped value is different from the calibration by about 1000ppm. YMMV.
H. Peter Anvin
reviewerQuoted 4 lines · Dan Magenheimer
On a quick sample of two machines looking at the TSC calibration done by Xen (which exposes the equivalent of
tsc_khz), it appears that the stamped value is different from the calibration by about 1000ppm. YMMV.
So pretty seriously different. Less than I would have expected, but not out of the ballpark.
-hpa
I still sorta think the clearest thing is to keep the kernel's quick, dynamic calibration at boot, and expose the tsc_khz it decides on. People who don't care about clocksync get their quick boot, NTP knows enough to correct its drift estimate when it starts, and users don't need to learn all these details to get stable clocksync (*).
If we're being strict, something like NTP needs to know exactly what's driving gettimeofday(). If the clocksource changes, it needs to know that so it could correct or trash its drift estimate. If there's one-time calibration, it needs to know what the result was. If there's continuous calibration, it either needs to be notified, or have the ability to disable it. Right? So I think exporting tsc_khz in some form is a step in the right direction.
So what's wrong with just adding a /sys/devices/system/clocksource/clocksource0/tsc_khz? Maybe Thomas Gleixner's suggestion of a vget_tsc_raw() would also suffice, I'm not sure I understand the details enough.
Any of the other fixes people have discussed (tsc_khz= bootopt, tsc_long_calibration=1) would be enough to make me happy though :)
(*) Though they still need to learn enough to coax the kernel into giving them a fast gettimeofday(). That's a price you gotta pay if you care enough :)
Quoted 2 lines · Brian Bloniarz
So what's wrong with just adding a
/sys/devices/system/clocksource/clocksource0/tsc_khz?
As an RFC:
Add clocksource.sys_register & sys_unregister so the current clocksource can add supplemental information to /sys/devices/system/clocksource/clocksource0/
Export tsc_khz when current_clocksource==tsc so that daemons like NTP can account for the variability of calibration results.
arch/x86/kernel/tsc.c +26 -0
@@ -76,6 +76,11 @@ unsigned long long sched_clock(void) __attribute__((alias("native_sched_clock")));#endif +int sysfs_tsc_register (struct sys_device *clocksource_dev, +struct clocksource *cs); +void sysfs_tsc_unregister (struct sys_device *clocksource_dev, +struct clocksource *cs); +int check_tsc_unstable (void ) {return tsc_unstable;
@@ -757,6 +762,8 @@ static struct clocksource clocksource_tsc = {#ifdef CONFIG_X86_64 .vread = vread_tsc, #endif + .sys_register = sysfs_tsc_register, + .sys_unregister = sysfs_tsc_unregister, };void mark_tsc_unstable (char *reason)
@@ -967,3 +974,22 @@ void __init tsc_init(void)init_tsc_clocksource (); } +static ssize_t show_tsc_khz ( +struct sys_device *dev,struct sysdev_attribute *attr,char *buf) +{ +return sprintf (buf,"%u\n" , tsc_khz); +} + +static SYSDEV_ATTR(tsc_khz, 0444, show_tsc_khz, NULL); + +int sysfs_tsc_register (struct sys_device *clocksource_dev, +struct clocksource *cs) +{ +return sysdev_create_file (clocksource_dev, &attr_tsc_khz); +} + +void sysfs_tsc_unregister (struct sys_device *clocksource_dev, +struct clocksource *cs) +{ +sysdev_remove_file (clocksource_dev, &attr_tsc_khz); +}
include/linux/clocksource.h +7 -0
@@ -15,6 +15,7 @@#include <linux/cache.h> #include <linux/timer.h> #include <linux/init.h> +#include <linux/sysdev.h> #include <asm/div64.h> #include <asm/io.h>
@@ -156,6 +157,8 @@ extern u64 timecounter_cyc2time(struct timecounter *tc,* @vread: vsyscall based read * @suspend: suspend function for the clocksource, if necessary * @resume: resume function for the clocksource, if necessary +* @sys_register: optional, register additional sysfs attributes +* @sys_unregister: optional, unregister sysfs attributes */ struct clocksource {/*
@@ -194,6 +197,10 @@ struct clocksource {struct list_head wd_list;cycle_t wd_last;#endif +int (*sys_register)(struct sys_device *clocksource_dev, +struct clocksource *cs); +void (*sys_unregister)(struct sys_device *clocksource_dev, +struct clocksource *cs); };/*
kernel/time/clocksource.c +29 -0
@@ -41,6 +41,8 @@ void timecounter_init(struct timecounter *tc, }EXPORT_SYMBOL_GPL (timecounter_init); +void sysfs_alter_clocksource (struct clocksource *old,struct clocksource *new); +/** * timecounter_read_delta - get nanoseconds since last call of this function * @tc: Pointer to time counter
@@ -572,6 +574,7 @@ static void clocksource_select(void) }if (curr_clocksource != best) {printk (KERN_INFO "Switching to clocksource %s\n" , best->name); +sysfs_alter_clocksource (curr_clocksource, best); curr_clocksource = best;timekeeping_notify (curr_clocksource); }
@@ -834,6 +837,8 @@ static struct sys_device device_clocksource = { .cls = &clocksource_sysclass, }; +static int sysfs_active =0 ; +static int __init init_clocksource_sysfs (void ) {int error =sysdev_class_register (&clocksource_sysclass);
@@ -848,10 +853,34 @@ static int __init init_clocksource_sysfs(void) error =sysdev_create_file ( &device_clocksource, &attr_available_clocksource); + +if (!error) + { +mutex_lock (&clocksource_mutex); +if (curr_clocksource->sys_register) + error = curr_clocksource->sys_register ( + &device_clocksource, curr_clocksource); +mutex_unlock (&clocksource_mutex); + } + +if (!error) + sysfs_active =1 ; return error; }device_initcall (init_clocksource_sysfs); + +void sysfs_alter_clocksource (struct clocksource *old, +struct clocksource *new) +{ +if (!sysfs_active) +return ; +if (old->sys_unregister) + old->sys_unregister (&device_clocksource, old); +if (new->sys_register) + new->sys_register (&device_clocksource, new); +} +#endif /* CONFIG_SYSFS */ /**
Quoted 6 lines · Brian Bloniarz
Quoted 2 lines · Brian Bloniarz
So what's wrong with just adding a
/sys/devices/system/clocksource/clocksource0/tsc_khz?As an RFC:
Quoted 7 lines
Add clocksource.sys_register &
sys_unregisterso the current clocksource can add supplemental information to/sys/devices/system/clocksource/clocksource0/Export
tsc_khzwhen current_clocksource==tsc so that daemons like NTP can account for the variability of calibration results.
I think this is a bad idea, as it creates an ABI that is arch AND machine specific, which will cause portability problems in applications that expect the interface to be there.
thanks
-john
Quoted 10 lines · john stultz
Quoted 9 lines · Brian Bloniarz
Quoted 2 lines · Brian Bloniarz
So what's wrong with just adding a
/sys/devices/system/clocksource/clocksource0/tsc_khz?As an RFC:
Add clocksource.sys_register &
sys_unregisterso the current clocksource can add supplemental information to/sys/devices/system/clocksource/clocksource0/
Quoted 7 lines
Quoted 3 lines
Export
tsc_khzwhen current_clocksource==tsc so that daemons like NTP can account for the variability of calibration results.I think this is a bad idea, as it creates an ABI that is arch AND machine specific, which will cause portability problems in applications that expect the interface to be there.
It's an arch-independent ABI that returns ENOENT on unsupported platforms ;)
Could you please explain what you envision as an
arch-independent solution to this problem?
I guess the tsc_long_calibration=1 alternative is
one.
Quoted 3 lines · Brian Bloniarz
Quoted 2 lines · Brian Bloniarz
So what's wrong with just adding a
/sys/devices/system/clocksource/clocksource0/tsc_khz?
It's wrong because TSC is an x86'ism.
Quoted 9 lines
As an RFC:
Add clocksource.sys_register &
sys_unregisterso the current clocksource can add supplemental information to/sys/devices/system/clocksource/clocksource0/Export
tsc_khzwhen current_clocksource==tsc so that daemons like NTP can account for the variability of calibration results.
I'd rather see a generic solution which provides the information of the current (and possibly those of the available) clock source(s).
This x86 centric TSC world view is horrible.
Thanks,
tglx
Quoted 19 lines · Brian Bloniarz
Quoted 10 lines · john stultz
Quoted 9 lines · Brian Bloniarz
Quoted 2 lines · Brian Bloniarz
So what's wrong with just adding a
/sys/devices/system/clocksource/clocksource0/tsc_khz?As an RFC:
Add clocksource.sys_register &
sys_unregisterso the current clocksource can add supplemental information to/sys/devices/system/clocksource/clocksource0/Quoted 7 lines
Quoted 3 lines
Export
tsc_khzwhen current_clocksource==tsc so that daemons like NTP can account for the variability of calibration results.I think this is a bad idea, as it creates an ABI that is arch AND machine specific, which will cause portability problems in applications that expect the interface to be there.
Quoted 7 lines
It's an arch-independent ABI that returns ENOENT on unsupported platforms ;)
Could you please explain what you envision as an
arch-independent solution to this problem?
I guess the tsc_long_calibration=1 alternative is
one.
Arch independent solution is to provide information about the current clock source in general. This is _NOT_ a TSC specific problem, you have the same trouble with any other clocksource which gets calibrated and does not take it's frequency as a constant value from boot loader, configuration or some CPU/chipset register. The only missing piece is a frequency member in struct clocksource which needs to be filled in by the arch/machine specific code.
Thanks,
tglx
Quoted 32 lines
From: Thomas Gleixner [mailto:[email protected]]
Quoted 11 lines · Brian Bloniarz
Quoted 10 lines · john stultz
Quoted 9 lines · Brian Bloniarz
Quoted 2 lines · Brian Bloniarz
So what's wrong with just adding a
/sys/devices/system/clocksource/clocksource0/tsc_khz?As an RFC:
Add clocksource.sys_register &
sys_unregisterso the current clocksource can add supplemental information to/sys/devices/system/clocksource/clocksource0/Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Export
tsc_khzwhen current_clocksource==tsc so that daemons like NTP can account for the variability of calibration results.I think this is a bad idea, as it creates an ABI that is arch AND machine specific, which will cause portability problems in
applications
Quoted 9 lines
Quoted 1 line
that expect the interface to be there.
It's an arch-independent ABI that returns ENOENT on unsupported platforms ;)
Could you please explain what you envision as an
arch-independent solution to this problem?
I guess the tsc_long_calibration=1 alternative is
one.
Quoted 7 lines
Arch independent solution is to provide information about the current clock source in general. This is _NOT_ a TSC specific problem, you have the same trouble with any other clocksource which gets calibrated and does not take it's frequency as a constant value from boot loader, configuration or some CPU/chipset register. The only missing piece is a frequency member in struct clocksource which needs to be filled in by the
arch/machinespecific code.
Actually there is already a frequency in struct clocksource except it's represented by the two components: mult and shift. Maybe it would be best to expose these instead of khz (for all clocksources) so as to limit abuse by naive users.
So, Thomas and John, if Brian's patch is modified to provide:
/sys/devices/system/clocksource/clocksource0/current_mult /sys/devices/system/clocksource/clocksource0/current_shift and/or /sys/devices/system/clocksource/clocksource0/current_khz
is that an acceptable arch-independent patch? (And which do you prefer?)
Quoted 10 lines · Dan Magenheimer
Quoted 9 lines
From: Thomas Gleixner [mailto:[email protected]]
Arch independent solution is to provide information about the current clock source in general. This is _NOT_ a TSC specific problem, you have the same trouble with any other clocksource which gets calibrated and does not take it's frequency as a constant value from boot loader, configuration or some CPU/chipset register. The only missing piece is a frequency member in struct clocksource which needs to be filled in by the
arch/machinespecific code.
Quoted 4 lines
Actually there is already a frequency in struct clocksource except it's represented by the two components: mult and shift. Maybe it would be best to expose these instead of khz (for all clocksources) so as to limit abuse by naive users.
You mean the TSC user space ones ?
Quoted 9 lines
So, Thomas and John, if Brian's patch is modified to provide:
/sys/devices/system/clocksource/clocksource0/current_mult/sys/devices/system/clocksource/clocksource0/current_shiftand/or/sys/devices/system/clocksource/clocksource0/current_khzis that an acceptable arch-independent patch? (And which do you prefer?)
I'd rather prefer the frequency interface for a simple reason. It allows to add a commandline option which provides the NTP folks with a sensible solution to their calibration problem as I don't see that a longer calibration time will reliably fix it.
So we'd get a "clocksource_freq=XXX" option which would be applied to the clocksource which is selected on the command line with "clocksource=NNN".
John ?
Thanks,
tglx
Quoted 15 lines · Brian Bloniarz
Quoted 14 lines · john stultz
Quoted 12 lines · Brian Bloniarz
Quoted 2 lines · Brian Bloniarz
So what's wrong with just adding a
/sys/devices/system/clocksource/clocksource0/tsc_khz?As an RFC:
Add clocksource.sys_register &
sys_unregisterso the current clocksource can add supplemental information to/sys/devices/system/clocksource/clocksource0/Export
tsc_khzwhen current_clocksource==tsc so that daemons like NTP can account for the variability of calibration results.
Quoted 6 lines
Quoted 3 lines
I think this is a bad idea, as it creates an ABI that is arch AND machine specific, which will cause portability problems in applications that expect the interface to be there.
It's an arch-independent ABI that returns ENOENT on unsupported platforms ;)
Right but having applications add "Linux on x86 where the TSC is being used" logic is pretty poor solution. Its an issue that should be addressed from the kernel side.
And really, if apps really wanted this info, they can fish it out of /proc/cpuinfo.
Quoted 4 lines
Could you please explain what you envision as an
arch-independent solution to this problem?
I guess the tsc_long_calibration=1 alternative is
one.
...and the tsc_khz= patch I posted earlier.
thanks
-john
Quoted 3 lines · john stultz
Right but having applications add "Linux on x86 where the TSC is being used" logic is pretty poor solution. Its an issue that should be addressed from the kernel side.
Yeah, sure.
Quoted 2 lines
And really, if apps really wanted this info, they can fish it out of
/proc/cpuinfo.
Really? I was under the impression that tsc_khz can differ from cpu_mhz (invariant tsc?), and cpu_mhz can differ from what shows up in /proc/cpuinfo cpuMHz due to cpufreq scaling. I was also under the impression that knowing or controlling tsc_khz is what NTP needs to ensure stability (assuming the TSC is otherwise stable, i.e. no halts-in-idle, NMI etc etc weirdness).
Quoted 4 lines · Dan Magenheimer
/sys/devices/system/clocksource/clocksource0/current_khzis that an acceptable arch-independent patch? (And which do you prefer?)
Thomas Gleixner:
Quoted 2 lines
I'd rather see a generic solution which provides the information of the current (and possibly those of the available) clock source(s).
Another possibility:
$ cd /sys/devices/system/clocksource/clocksource0/
$ ls -lR available_clocksource current_clocksource current_clocksource_ln -> tsc tsc/ tsc/calibration tsc/calibrated_master -> ../hpet tsc/khz hpet/ hpet/calibration hpet/khz
$ cat tsc/calibration slave # there has been a one-time calibration against a reference at boot time, # the source clock is in calibrated_master and and the khz is calculated # from that
$ cat hpet/calibration constant # takes its value from constant value from boot loader, configuration # or some CPU/chipset register
Would this be workable? I need to look deeper at how the other clocksources work, for example the virtualized ones. I'm also wondering if NICs with their own clocks & IEEE-1588 support are going to become part of the clocksource infrastructure (see e.g. http://patchwork.ozlabs.org/patch/52626/)
Thanks everyone for the guidance.
Quoted 7 lines · Brian Bloniarz
Quoted 3 lines · john stultz
Right but having applications add "Linux on x86 where the TSC is being used" logic is pretty poor solution. Its an issue that should be addressed from the kernel side.
Yeah, sure.
Quoted 8 lines
Quoted 2 lines
And really, if apps really wanted this info, they can fish it out of
/proc/cpuinfo.Really? I was under the impression that
tsc_khzcan differ fromcpu_mhz(invariant tsc?), andcpu_mhzcan differ from what shows up in/proc/cpuinfocpuMHz due to cpufreq scaling. I was also under the impression that knowing or controllingtsc_khzis what NTP needs to ensure stability (assuming the TSC is otherwise stable, i.e. no halts-in-idle, NMI etc etc weirdness).
Bah. You're right. I shouldn't be emailing this early :)
Even so, I'm still not a fan of the "expose raw details so userland apps can hack around the kernel's inadequacies" approach.
Quoted 9 lines
Quoted 4 lines · Dan Magenheimer
/sys/devices/system/clocksource/clocksource0/current_khzis that an acceptable arch-independent patch? (And which do you prefer?)
Thomas Gleixner:
Quoted 2 lines
I'd rather see a generic solution which provides the information of the current (and possibly those of the available) clock source(s).
While I'm not a huge fan of it, Thomas' way would be a bit more palatable.
NTP can check the initial freq the clocksource was registered and if its different from the last boot decide if it can recalculate that into a new correction factor, or just throw out the drift file value.
Brian: is this something the NTPd folks actually want? Has anyone checked with them before we hand down the solution from high upon on lkml mountain?
Personally I think NTPd should be a little more savvy about how far it trusts the drift file when it starts up. Since I believe its fast-startup mode can quickly estimate the drift well within 100ppm, which is about the maximum variance I've seen from the calibration code.
thanks
-john
H. Peter Anvin
reviewerQuoted 10 lines · john stultz
Brian: is this something the NTPd folks actually want? Has anyone checked with them before we hand down the solution from high upon on lkml mountain?
Personally I think NTPd should be a little more savvy about how far it trusts the drift file when it starts up. Since I believe its fast-startup mode can quickly estimate the drift well within 100ppm, which is about the maximum variance I've seen from the calibration code.
Engaging with them is probably a good idea. In the past, the NTP core folks have been extremely anti-Linux and pro-BSD and therefore unwilling to talk, but that has at least in part been due to what they perceive as unilateral actions on our part.
-hpa
Quoted 3 lines · john stultz
Brian: is this something the NTPd folks actually want? Has anyone checked with them before we hand down the solution from high upon on lkml mountain?
I haven't checked, it's been a while since I dealt with
this problem. The NTP maintainers definitely complain about the quick TSC calibration code like it's a bug:
(e.g. http://www.mail-archive.com/[email protected]/msg02079.html).
Anyway I'll reach out before I spend any time investing in
a solution that they don't want (and you don't like :).
Quoted 4 lines
Personally I think NTPd should be a little more savvy about how far it trusts the drift file when it starts up. Since I believe its fast-startup mode can quickly estimate the drift well within 100ppm, which is about the maximum variance I've seen from the calibration code.
The workaround we went with was to remove the drift file on every reboot. But in our experience, even with iburst, converging takes a long time. I don't have hard numbers since it's been a long time since I investigated the problem, but we defined failure as >1ms offset syncing to a server in our LAN, and a cold NTP boot takes 10-20 hours to get there.
I was hoping that being able to reuse the drift information across boots would shorten convergence time. I think that in principle it's a nice thing to be able to do. Though as far as I'm aware, neither chrony nor PTPd (IEEE 1588) attempts to do this.
H. Peter Anvin
reviewerQuoted 5 lines · Brian Bloniarz
Quoted 3 lines · john stultz
Brian: is this something the NTPd folks actually want? Has anyone checked with them before we hand down the solution from high upon on lkml mountain?
Quoted 7 lines
I haven't checked, it's been a while since I dealt with
this problem. The NTP maintainers definitely complain about the quick TSC calibration code like it's a bug:
(e.g. http://www.mail-archive.com/[email protected]/msg02079.html).
Anyway I'll reach out before I spend any time investing in
a solution that they don't want (and you don't like :).
Yes, Prof. Mills in particular (for those who don't know, he's "Mr. NTP") really gets upset about the way Linux does timekeeping. Unfortunately it's not clear to me that he's willing to work with us as opposed to wanting things to work exactly like BSD, and swive the non-NTP users.
-hpa
Quoted 17 lines · Brian Bloniarz
Quoted 3 lines · john stultz
Brian: is this something the NTPd folks actually want? Has anyone checked with them before we hand down the solution from high upon on lkml mountain?
I haven't checked, it's been a while since I dealt with
this problem. The NTP maintainers definitely complain about the quick TSC calibration code like it's a bug:
(e.g. http://www.mail-archive.com/[email protected]/msg02079.html).
Anyway I'll reach out before I spend any time investing in
a solution that they don't want (and you don't like :).Quoted 4 lines
Personally I think NTPd should be a little more savvy about how far it trusts the drift file when it starts up. Since I believe its fast-startup mode can quickly estimate the drift well within 100ppm, which is about the maximum variance I've seen from the calibration code.
Quoted 6 lines
The workaround we went with was to remove the drift file on every reboot. But in our experience, even with iburst, converging takes a long time. I don't have hard numbers since it's been a long time since I investigated the problem, but we defined failure as >1ms offset syncing to a server in our LAN, and a cold NTP boot takes 10-20 hours to get there.
Ok. If its been awhile, you may find recent kernels (2.6.31+) are much faster to converge due to adjustments made to the SHIFT_PLL constant. This was done explicitly to address issues similar to what you describe above.
thanks
-john
Quoted 14 lines · H. Peter Anvin
Quoted 5 lines · Brian Bloniarz
Quoted 3 lines · john stultz
Brian: is this something the NTPd folks actually want? Has anyone checked with them before we hand down the solution from high upon on lkml mountain?
Quoted 7 lines
I haven't checked, it's been a while since I dealt with
this problem. The NTP maintainers definitely complain about the quick TSC calibration code like it's a bug:
(e.g. http://www.mail-archive.com/[email protected]/msg02079.html).
Anyway I'll reach out before I spend any time investing in
a solution that they don't want (and you don't like :).
Quoted 5 lines
Yes, Prof. Mills in particular (for those who don't know, he's "Mr. NTP") really gets upset about the way Linux does timekeeping. Unfortunately it's not clear to me that he's willing to work with us as opposed to wanting things to work exactly like BSD, and swive the non-NTP users.
Although I suspect his dislike for Linux is historical, as Roman reworked the ntp code to follow the NTPv4 reference implementation back in the 2.6.19 timeframe.
However I'd be more then happy to try to address any specific deficiencies with Linux's NTP implementation if someone can better express Prof Mills' critiques are.
thanks
-john
Quoted 25 lines · john stultz
Quoted 17 lines · Brian Bloniarz
Quoted 3 lines · john stultz
Brian: is this something the NTPd folks actually want? Has anyone checked with them before we hand down the solution from high upon on lkml mountain?
I haven't checked, it's been a while since I dealt with
this problem. The NTP maintainers definitely complain about the quick TSC calibration code like it's a bug:
(e.g. http://www.mail-archive.com/[email protected]/msg02079.html).
Anyway I'll reach out before I spend any time investing in
a solution that they don't want (and you don't like :).Quoted 4 lines
Personally I think NTPd should be a little more savvy about how far it trusts the drift file when it starts up. Since I believe its fast-startup mode can quickly estimate the drift well within 100ppm, which is about the maximum variance I've seen from the calibration code.
Quoted 6 lines
The workaround we went with was to remove the drift file on every reboot. But in our experience, even with iburst, converging takes a long time. I don't have hard numbers since it's been a long time since I investigated the problem, but we defined failure as >1ms offset syncing to a server in our LAN, and a cold NTP boot takes 10-20 hours to get there.
Quoted 4 lines
Ok. If its been awhile, you may find recent kernels (2.6.31+) are much faster to converge due to adjustments made to the
SHIFT_PLLconstant. This was done explicitly to address issues similar to what you describe above.
My tests were pre-2.6.31, this is really good to know. I'll take another look on recent kernels.
H. Peter Anvin
reviewerQuoted 7 lines · john stultz
Quoted 6 lines
Yes, Prof. Mills in particular (for those who don't know, he's "Mr. NTP") really gets upset about the way Linux does timekeeping. Unfortunately it's not clear to me that he's willing to work with us as opposed to wanting things to work exactly like BSD, and swive the non-NTP users.
Quoted 8 lines
Although I suspect his dislike for Linux is historical, as Roman reworked the ntp code to follow the NTPv4 reference implementation back in the 2.6.19 timeframe.
However I'd be more then happy to try to address any specific deficiencies with Linux's NTP implementation if someone can better express Prof Mills' critiques are.
That's the $10M question... I think it starts with asking the NTP community for advice.
-hpa