Skip to contentSkip to conversation

x86: Export tsc related information in sysfs

PATCH In review Venkatesh Pallipadi 10 participants 44 messages May 15, 2010 – Jun 4, 2010 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86: Export tsc related information in sysfs

From: Dan Magenheimer <[email protected]>

Kernel information about calibrated value of tsc_khz and tsc_stability (result of tsc warp test) are useful bits of information for any app that wants to use TSC directly. Export this read_only information in sysfs.

---
 arch/x86/kernel/tsc.c |   76 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 76 insertions(+), 0 deletions(-)
arch/x86/kernel/tsc.c +76 -0
index 9faf91a..24dd484 100644
@@ -10,6 +10,7 @@
 #include <linux/clocksource.h>
 #include <linux/percpu.h>
 #include <linux/timex.h>
+#include <linux/sysdev.h>
 
 #include <asm/hpet.h>
 #include <asm/timer.h>
@@ -857,6 +858,81 @@ static void __init init_tsc_clocksource(void)
 	clocksource_register(&clocksource_tsc);
 }
 
+#ifdef CONFIG_SYSFS
+/*
+ * Export TSC related info to user land. This reflects kernel usage of TSC
+ * as hints to userspace users of TSC. The read_only info provided here:
+ * - tsc_stable: 1 implies system has TSC that always counts at a constant
+ *   rate, sync across CPUs and has passed the kernel warp test.
+ * - tsc_khz: TSC frequency in khz.
+ * - tsc_mult and tsc_shift: multiplier and shift to optimally convert
+ *   TSC delta to ns; ns = ((u64) delta * mult) >> shift
+ */
+
+#define define_show_var_function(_name, _var) \
+static ssize_t show_##_name( \
+	struct sys_device *dev, struct sysdev_attribute *attr, char *buf) \
+{ \
+	return sprintf(buf, "%u\n", (unsigned int) _var);\
+}
+
+define_show_var_function(tsc_stable, !tsc_unstable);
+define_show_var_function(tsc_khz, tsc_khz);
+define_show_var_function(tsc_mult, clocksource_tsc.mult);
+define_show_var_function(tsc_shift, clocksource_tsc.shift);
+
+static SYSDEV_ATTR(tsc_stable, 0444, show_tsc_stable, NULL);
+static SYSDEV_ATTR(tsc_khz, 0444, show_tsc_khz, NULL);
+static SYSDEV_ATTR(tsc_mult, 0444, show_tsc_mult, NULL);
+static SYSDEV_ATTR(tsc_shift, 0444, show_tsc_shift, NULL);
+
+static struct sysdev_attribute *tsc_attrs[] = {
+	&attr_tsc_stable,
+	&attr_tsc_khz,
+	&attr_tsc_mult,
+	&attr_tsc_shift,
+};
+
+static struct sysdev_class tsc_sysclass = {
+	.name = "tsc",
+};
+
+static struct sys_device device_tsc = {
+	.id = 0,
+	.cls = &tsc_sysclass,
+};
+
+static int __init init_tsc_sysfs(void)
+{
+	int err, i = 0;
+
+	err = sysdev_class_register(&tsc_sysclass);
+	if (err)
+		return err;
+
+	err = sysdev_register(&device_tsc);
+	if (err)
+		goto fail;
+
+	for (i = 0; i < ARRAY_SIZE(tsc_attrs); i++) {
+		err = sysdev_create_file(&device_tsc, tsc_attrs[i]);
+		if (err)
+			goto fail;
+	}
+
+	return 0;
+
+fail:
+	while (--i >= 0)
+		sysdev_remove_file(&device_tsc, tsc_attrs[i]);
+
+	sysdev_unregister(&device_tsc);
+	sysdev_class_unregister(&tsc_sysclass);
+	return err;
+}
+device_initcall(init_tsc_sysfs);
+#endif
+
 #ifdef CONFIG_X86_64
 /*
  * calibrate_cpu is used on systems with fixed rate TSCs to determine
Signature · 1 line
1.7.0.1
#1 lore raw
Quoted 6 lines · Venkatesh Pallipadi

From: Dan Magenheimer <[email protected]>

Kernel information about calibrated value of tsc_khz and tsc_stability (result of tsc warp test) are useful bits of information for any app that wants to use TSC directly. Export this read_only information in sysfs.

Is this really a good idea? It will encourage the applications to use RDTSC directly, but there are all kinds of constraints on that. Even the kernel has a hard time with them, how likely is it that applications will get all that right?

It would be better to fix them to use the vsyscalls instead. Or if they can't use the vsyscalls for some reason today fix them.

This way if anything changes again in TSC the kernel could shield the applications.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#2 lore raw
Quoted 10 lines

From: Andi Kleen [mailto:[email protected]]

Quoted 2 lines

Kernel information about calibrated value of tsc_khz and tsc_stability (result of tsc warp test) are useful bits of

information

Quoted 2 lines

for any app that wants to use TSC directly. Export this read_only information in sysfs.

Is this really a good idea? It will encourage the applications to use RDTSC directly, but there are all kinds of constraints on

Indeed, that is what it is intended to do.

Quoted 2 lines

that. Even the kernel has a hard time with them, how likely is it that applications will get all that right?

That's the point of exposing the tsc_reliable kernel data. If the processor has Invariant TSC and the system has successfully passed Ingo's warp test and, as a result the kernel is using TSC as a clocksource, why not enable userland apps that need to obtain timestamp data tens or hundreds of thousands of times per second to also use the TSC directly?

Quoted 2 lines

It would be better to fix them to use the vsyscalls instead. Or if they can't use the vsyscalls for some reason today fix them.

The problem is from an app point-of-view there is no vsyscall. There are two syscalls: gettimeofday and clock_gettime. Sometimes, if it gets lucky, they turn out to be very fast and sometimes it doesn't get lucky and they are VERY slow (resulting in a performance hit of 10% or more), depending on a number of factors completely out of the control of the app and even undetectable to the app.

Note also that even vsyscall with TSC as the clocksource will still be significantly slower than rdtsc, especially in the common case where a timestamp is directly stored and the delta between two timestamps is later evaluated; in the vsyscall case, each timestamp is a function call and a convert to nsec but in the TSC case, each timestamp is a single instruction.

Quoted 2 lines

This way if anything changes again in TSC the kernel could shield the applications.

If tsc_reliable is 1, the system and the kernel are guaranteeing to the app that nothing will change in the TSC. In an Invariant TSC system that has passed Ingo's warp test (to eliminate the possibility of a fixed interprocessor TSC gap due to a broken BIOS in a multi-node NUMA system), if anything changes in the clock signal that drives the TSC, the system is badly broken and far worse things -- like inter-processor cache incoherency -- may happen.

Is it finally possible to get past the horrible SMP TSC problems of the past and allow apps, under the right conditions, to be able to use rdtsc again? This patch argues "yes".

#3 lore raw
Quoted 16 lines · Dan Magenheimer
Quoted 10 lines

From: Andi Kleen [mailto:[email protected]]

Quoted 2 lines

Kernel information about calibrated value of tsc_khz and tsc_stability (result of tsc warp test) are useful bits of

information

Quoted 2 lines

for any app that wants to use TSC directly. Export this read_only information in sysfs.

Is this really a good idea?  It will encourage the applications to use RDTSC directly, but there are all kinds of constraints on

Indeed, that is what it is intended to do.

Quoted 2 lines

that. Even the kernel has a hard time with them, how likely is it that applications will get all that right?

Quoted 8 lines

That's the point of exposing the tsc_reliable kernel data. If the processor has Invariant TSC and the system has successfully passed Ingo's warp test and, as a result the kernel is using TSC as a clocksource, why not enable userland apps that need to obtain timestamp data tens or hundreds of thousands of times per second to also use the TSC directly?

I am a little concerned about applications getting this right, with rdtsc and related barriers etc. May be that calls for a userspace library.

Despite of that, it is useful information to human user to know whether tsc is stable and what TSC freq is. It will be very hard for userspace to do the TSC calibration. And to know whether TSC warp test passed or whether TSC is marked unstable by the kernel, there is no one place to get these answers today. Users have to look at clocksource, dmesg, etc.

Thanks,
Venki

#4 lore raw
In reply to Dan Magenheimer
Quoted 10 lines · On Sat, 15 May 2010 06:29:25 -0700 (PDT) Dan Magenheimer…
Quoted 2 lines

It would be better to fix them to use the vsyscalls instead. Or if they can't use the vsyscalls for some reason today fix them.

The problem is from an app point-of-view there is no vsyscall. There are two syscalls: gettimeofday and clock_gettime. Sometimes, if it gets lucky, they turn out to be very fast and sometimes it doesn't get lucky and they are VERY slow (resulting in a performance hit of 10% or more), depending on a number of factors completely out of the control of the app and even undetectable to the app.

But the point is.. in the case you get that 10% hit.... that is exactly the case where tsc would not work either!!!

Quoted 7 lines

If tsc_reliable is 1, the system and the kernel are guaranteeing to the app that nothing will change in the TSC. In an Invariant TSC system that has passed Ingo's warp test (to eliminate the possibility of a fixed interprocessor TSC gap due to a broken BIOS in a multi-node NUMA system), if anything changes in the clock

just when we're trying to get rid of this constraint by allowing a per cpu offset... (this is needed to cope with cpus not powering on at the exact same time... including hotplug cpu etc etc)

oh and.. what notification mechanism do you have to notify the application that the tsc now is no longer reliable? Such conditions can exist... for example due to a CPU being hotplugged, or some SMM screwing around and the kernel detecting that or .. or ...

really. Use the vsyscall. If the vsyscall does not do exactly what you want, make a better vsyscall.

But friends don't let friends use rdtsc in application code.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#5 lore raw
Quoted 1 line

From: Arjan van de Ven [mailto:[email protected]]

(Arjan comments reordered somewhat)

Quoted 1 line

But friends don't let friends use rdtsc in application code.

Um, I realize that many people have been burned by this many times over the years so it is a "hot stove". I also realize that there are many environments where using rdtsc is risking stepping on landmines. But I (we?) also know there are many environments now where using rdtsc is NOT risky at all... and with the vast majority of new systems soon shipping with Invariant TSC and a single socket (and even most multiple-socket systems with non-broken BIOSes passing a warp test), why should past burns outlaw userland use of a very fast, very useful CPU feature? After all, CPU designers at both Intel and AMD have spent a great deal of design effort and transistors to FINALLY provide an Invariant TSC.

Quoted 10 lines
Quoted 7 lines

The problem is from an app point-of-view there is no vsyscall. There are two syscalls: gettimeofday and clock_gettime. Sometimes, if it gets lucky, they turn out to be very fast and sometimes it doesn't get lucky and they are VERY slow (resulting in a performance hit of 10% or more), depending on a number of factors completely out of the control of the app and even undetectable to the app.

But the point is.. in the case you get that 10% hit.... that is exactly the case where tsc would not work either!!!

Yes, understood. But the kernel doesn't expose a "gettimeofday performance sucks" flag either. If it did (or in the case of the patch, if tsc_reliable is zero) the application could at least choose to turn off the 10000-100000 timestamps/second and log a message saying "you are running on old hardware so you get fewer features".

Quoted 8 lines

just when we're trying to get rid of this constraint by allowing a per cpu offset... (this is needed to cope with cpus not powering on at the exact same time... including hotplug cpu etc etc)

oh and.. what notification mechanism do you have to notify the application that the tsc now is no longer reliable? Such conditions can exist... for example due to a CPU being hotplugged, or some SMM screwing around and the kernel detecting that or .. or ...

The proposal doesn't provide a notification mechanism (though I'm not against it)... if the tsc can EVER become unreliable, tsc_reliable should be 0.

A CPU-hotplugable system is a good example of a case where the kernel should expose that tsc_reliable is 0. (I've heard anecdotally that CPU hotplug into a QPI or Hypertransport system will have some other interesting challenges, so may require some special kernel parameters anyway.) Even if tsc_reliable were only enabled if a "no-cpu_hotplug" kernel parameter is set, that is still useful. And with cores-per-socket (and even nodes-per-socket) going up seemingly every day, multi-socket systems will likely be an ever smaller percentage of new systems.

A virtual machine where live migration to another physical machine may occur is another good example where tsc_reliable should be 0. Xen now has a VM config feature that says "migration is disallowed" for this reason; the Invariant TSC flag is always off for a VM unless this "no_migrate" flag is set (or rdtsc is emulated).

Quoted 2 lines

really. Use the vsyscall. If the vsyscall does not do exactly what you want, make a better vsyscall.

If this discussion results in a better vsyscall and/or a way for applications to easily determine (and report loudly) that the system does NOT provide a good way to do a fast timestamp, that may be sufficient. But please propose how that will be done as the current software choices are inadequate and the CPU designers have finally fixed the problem for the vast majority of systems. I am already aware of some enterprise software that is doing its best to guess whether TSC is reliable by looking at CPU families and socket counts, but this is doomed to failure in userland and is something that the kernel knows and should now expose.

Thanks,
Dan

#6 lore raw
Quoted 5 lines · On Sat, 15 May 2010 15:32:51 -0700 (PDT) Dan Magenheimer…
Quoted 1 line

From: Arjan van de Ven [mailto:[email protected]]

(Arjan comments reordered somewhat)

Quoted 1 line

But friends don't let friends use rdtsc in application code.

Quoted 8 lines

Um, I realize that many people have been burned by this many times over the years so it is a "hot stove". I also realize that there are many environments where using rdtsc is risking stepping on landmines.

But I (we?) also
know there are many environments now where using rdtsc is NOT risky at all...

I see a lot of Intel hardware.. (stuff that you likely don't see yet ;-) and I have not yet seen a system where the kernel would be able to give the guarantee as you describe it in your email.

If you want a sysfs variable that is always 0... go wild.

Quoted 4 lines

and with the vast majority of new
systems soon shipping with Invariant TSC and a single socket (and even most multiple-socket systems with non-broken BIOSes passing a warp test),

(the warp test is going away)

on multisocket that passes a wrap test you can still get skew over time.. due to things like SMM, thermal throttling etc etc.

Quoted 5 lines

why should past burns outlaw
userland use of a very fast, very useful CPU feature? After all, CPU designers at both Intel and AMD have spent a great deal of design effort and transistors to FINALLY provide an Invariant TSC.

sadly even with all these transistors no system that I know of today can guarantee the guarantee by the rules you state.

Quoted 8 lines
Quoted 4 lines

oh and.. what notification mechanism do you have to notify the application that the tsc now is no longer reliable? Such conditions can exist... for example due to a CPU being hotplugged, or some SMM screwing around and the kernel detecting that or .. or ...

The proposal doesn't provide a notification mechanism (though I'm not against it)... if the tsc can EVER become unreliable, tsc_reliable should be 0.

then it should be 0 always on all of todays hardware. SMM, thermal overload, etc etc ... you name it. Things the kernel will get notified about...

Quoted 5 lines

A CPU-hotplugable system is a good example of a case where the kernel should expose that tsc_reliable is 0. (I've heard anecdotally that CPU hotplug into a QPI or Hypertransport system will have some other interesting challenges, so may require some special kernel parameters anyway.)

eh no.
hot add works just fine.

(hot remove is a very different ballgame)

Quoted 10 lines
Quoted 2 lines

really. Use the vsyscall. If the vsyscall does not do exactly what you want, make a better vsyscall.

If this discussion results in a better vsyscall and/or a way for applications to easily determine (and report loudly) that the system does NOT provide a good way to do a fast timestamp, that may be sufficient. But please propose how that will be done as the current software choices are inadequate and the CPU designers have finally fixed the problem for the vast majority of systems.

*cough*

Quoted 5 lines

I am already aware of some enterprise software
that is doing its best to guess whether TSC is reliable by looking at CPU families and socket counts, but this is doomed to failure in userland and is something that the kernel knows and should now expose.

can you name said "enterprise" software by name please? We need a huge advertisement to let people know not to trust their important data to it..

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#7 lore raw
Quoted 17 lines · Arjan van de Ven
Quoted 5 lines · On Sat, 15 May 2010 15:32:51 -0700 (PDT) Dan Magenheimer…
Quoted 1 line

From: Arjan van de Ven [mailto:[email protected]]

(Arjan comments reordered somewhat)

Quoted 1 line

But friends don't let friends use rdtsc in application code.

Quoted 8 lines

Um, I realize that many people have been burned by this many times over the years so it is a "hot stove". I also realize that there are many environments where using rdtsc is risking stepping on landmines.

But I (we?) also
know there are many environments now where using rdtsc is NOT risky at all...

Quoted 5 lines

I see a lot of Intel hardware.. (stuff that you likely don't see yet ;-) and I have not yet seen a system where the kernel would be able to give the guarantee as you describe it in your email.

If you want a sysfs variable that is always 0... go wild.

Nah, there are systems which will have it set to 1:

Dig out your good old Pentium-I box and enjoy.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

oh and.. what notification mechanism do you have to notify the application that the tsc now is no longer reliable? Such conditions can exist... for example due to a CPU being hotplugged, or some SMM screwing around and the kernel detecting that or .. or ...

Quoted 7 lines
Quoted 3 lines

The proposal doesn't provide a notification mechanism (though I'm not against it)... if the tsc can EVER become unreliable, tsc_reliable should be 0.

then it should be 0 always on all of todays hardware. SMM, thermal overload, etc etc ... you name it. Things the kernel will get notified about...

What we could expose is an estimate about the performance of gettimeofday/clock_gettime. The kernel has all the information to do that, but this still does not solve the notification problem when we need to switch to a different clock source.

Thanks,

tglx

#8 lore raw
Quoted 4 lines

From: Thomas Gleixner [mailto:[email protected]]
What we can talk about is a vget_tsc_raw() interface along with a vconvert_tsc_delta() interface, where vget_tsc_raw() returns you an nasty error code for everything which is not usable.

I'm open to something like that provided:

1) It works (whenever possible) without changing privilege levels or causing vmexits or other "hidden slowness" problems when used both in bare-metal Linux and in a virtual machine.
2) The "transformation" performed by the kernel on the TSC does not require some hidden pcpu number that won't work in a virtual machine.

If TSC is indeed reliable (see below), it is both faster AND meets the above constraints.

Quoted 6 lines
Quoted 2 lines

From: Arjan van de Ven [mailto:[email protected]] If you want a sysfs variable that is always 0... go wild.

From: Thomas Gleixner [mailto:[email protected]] Nah, there are systems which will have it set to 1:
Dig out your good old Pentium-I box and enjoy.

Hot stove syndrome again? Are you truly saying that there are NO single-socket multi-core systems that don't have stupid firmware (SMI and/or BIOS)? Or are you saying that significant TSC clock skew occurs even between the cores on a single-socket Nehalem system?

If things are this bad, why on earth would the kernel itself EVER use TSC even as its own internal clocksource? Or even to provide additional precision to a slow platform timer?

Or are you saying that many systems (and especially large multi-socket systems) DO exist where the kernel isn't able to proactively determine that the firmware is broken and/or significant thermal variation may occur across sockets? This I believe.

I understand that you both are involved in pushing the limits of large systems and that time synchronization is a very hard problem, perhaps effectively unsolvable, in these systems.

But that doesn't mean the vast majority of latest generation single-socket systems can't set "tsc_reliable" to 1. Or that the kernel is responsible for detecting and/or correcting every system with buggy firmware.

Maybe the best way to solve the "buggy firmware problem" is exactly by encouraging enterprise apps to use TSC and to expose and *blacklist* systems and/or system vendors who ship boxes with crappy firmware!

Quoted 5 lines

From: Thomas Gleixner [mailto:[email protected]]
What we could expose is an estimate about the performance of gettimeofday/clock_gettime. The kernel has all the information to do that, but this still does not solve the notification problem when we need to switch to a different clock source.

This would at least be a big step in the right direction.

But if we go with a vget_raw_tsc() or direct TSC solution, you have convinced me of the need for notification. Maybe this is a perfect use for (at least one bit in) the TSC_AUX register and the rdtscp instruction?

And I do agree with Venki that some user library (or at least published sample code) should be made available to demonstrate proper usage and to dampen out the worst of the "broken user problem".

Quoted 4 lines
Quoted 4 lines

From: Arjan van de Ven [mailto:[email protected]]
can you name said "enterprise" software by name please? We need a huge advertisement to let people know not to trust their important data to it..

For obvious reasons I can't do that, but I can point to enterprise *operating systems* that have long since solved this same problem one way or another: Solaris on x86 and HP-UX (the latter admittedly on ia64). Enterprise app vendors are quite happy with requiring conformance to a very completely specified software/hardware/firmware stack before providing support to an app customer. I'm just trying to ensure that Linux can be part of that spec.

#9 lore raw

Dan,

Quoted 5 lines · Dan Magenheimer
Quoted 4 lines

From: Thomas Gleixner [mailto:[email protected]]
What we can talk about is a vget_tsc_raw() interface along with a vconvert_tsc_delta() interface, where vget_tsc_raw() returns you an nasty error code for everything which is not usable.

Quoted 8 lines

I'm open to something like that provided:

1) It works (whenever possible) without changing privilege levels or causing vmexits or other "hidden slowness" problems when used both in bare-metal Linux and in a virtual machine.
2) The "transformation" performed by the kernel on the TSC does not require some hidden pcpu number that won't work in a virtual machine.

What I have in mind and what I'm working on for quite a while is going to work on both bare metal and VMs w/o hidden slowness.

Quoted 11 lines

If TSC is indeed reliable (see below), it is both faster AND meets the above constraints.

Quoted 6 lines
Quoted 2 lines

From: Arjan van de Ven [mailto:[email protected]] If you want a sysfs variable that is always 0... go wild.

From: Thomas Gleixner [mailto:[email protected]] Nah, there are systems which will have it set to 1:
Dig out your good old Pentium-I box and enjoy.

Hot stove syndrome again? Are you truly saying that there

Kinda hot stove, yes. I'm unfortunately forced to deal with the 500+ different variants of borked timers and that makes me very reluctant to believe anything what chip/board/bios vendors promise. It's not the one time hot stove experience, it's the constant exposure to the never ending supply of hot stoves, which makes me nervous.

I wish I could say something different.

Quoted 2 lines

are NO single-socket multi-core systems that don't have stupid firmware (SMI and/or BIOS)? Or are you saying that

There are single socket multi-core x86 systems with a sane BIOS, but there is no reliable way to tell which ones belong into that category.

Quoted 2 lines

significant TSC clock skew occurs even between the cores on a single-socket Nehalem system?

There is no clock skew between the cores of a package - at least we are not aware of such a problem. Though I wouldn't rely on that
forever: they also said that the Titanic was unsinkable :)

Quoted 2 lines

If things are this bad, why on earth would the kernel itself EVER use TSC even as its own internal clocksource? Or

We try to use it for performance sake, but the kernel does at least it's very best to find out when it goes bad. We then switch back to a hpet or pm-timer which is horrible performance wise but does not screw up timekeeping and everything which relies on it completely.

Quoted 1 line

even to provide additional precision to a slow platform timer?

We don't do that anymore.

Quoted 5 lines

Or are you saying that many systems (and especially large multi-socket systems) DO exist where the kernel isn't able to proactively determine that the firmware is broken and/or significant thermal variation may occur across sockets? This I believe.

As I said, we try our very best to determine when things go awry, but there are small errors which occur either sporadic or after longer uptime which we cannot yet detect reliably. Multi-socket falls into that category, but we are working on that.

Quoted 4 lines

I understand that you both are involved in pushing the limits of large systems and that time synchronization is a very hard problem, perhaps effectively unsolvable, in these systems.

Well, it would be solvable in hardware and it has been done in hardware more than 20 years ago. Just not there where it would have been important: inside of x86 cpus. Hint: there are other architectures which got that right from the very beginning even on multi-socket systems.

Admitted, x86 made progress, but we are still some steps away from something which I would consider reliable under all circumstances.

But you are right, some of the problems with the existing hardware are just unsolvable and I spent a serious amount of time on trying to convince myself otherwise.

The nasty thing about the subtle wreckage is that it is really hard to investigate and debug and I wasted a whole week recently to figure out what caused the time going backwards problem on a dual socket westmere. Not fun !

Quoted 4 lines

But that doesn't mean the vast majority of latest generation single-socket systems can't set "tsc_reliable" to 1. Or that the kernel is responsible for detecting and/or correcting every system with buggy firmware.

It _IS_ responsible to detect buggy firmware otherwise we would just drain in bug reports about broken timekeeping. We've been there, no way to go back to this.

Quoted 4 lines

Maybe the best way to solve the "buggy firmware problem" is exactly by encouraging enterprise apps to use TSC and to expose and *blacklist* systems and/or system vendors who ship boxes with crappy firmware!

Blacklists are the last resort if a problem is not detectable by the kernel. We usually detect the non usability of TSC and emit a prominent warning into dmesg. Those warnings are there for years, but the number of systems with BIOS caused TSC wreckage has grown.

Quoted 7 lines
Quoted 5 lines

From: Thomas Gleixner [mailto:[email protected]]
What we could expose is an estimate about the performance of gettimeofday/clock_gettime. The kernel has all the information to do that, but this still does not solve the notification problem when we need to switch to a different clock source.

This would at least be a big step in the right direction.

Ok.

Quoted 4 lines

But if we go with a vget_raw_tsc() or direct TSC solution, you have convinced me of the need for notification. Maybe this is a perfect use for (at least one bit in) the TSC_AUX register and the rdtscp instruction?

Uurgh, no. The vsyscall will return a proper error code when shit happens. And really, we don't want to encourage the direct use of rdtsc at all. Also rdtscp is a full serializing instruction, which is probably not what you want to get fast timestamps.

Quoted 4 lines

And I do agree with Venki that some user library (or at least published sample code) should be made available to demonstrate proper usage and to dampen out the worst of the "broken user problem".

Using a vsyscall is the best way to achieve that. Simple function call interface with a well defined ABI and a proper return code. If the user ignores the return code - none of my problems.

Further it allows us

- to keep the various CPU generation specific quirks well confined in the kernel and we can even do fixups for correctable wreckage.

- to expose coarser grained fast timestamps when the TSC is not usable. [So the best name for it would be vget_timestamp(), which btw. allows us to provide the same interface to non x86 as well ]

Thoughts ?

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

From: Arjan van de Ven [mailto:[email protected]]
can you name said "enterprise" software by name please? We need a huge advertisement to let people know not to trust their important data to it..

For obvious reasons I can't do that, but I can point to enterprise *operating systems* that have long since solved this same problem one way or another: Solaris on x86 and

On a well selected subset of the machines which they control themself.

Quoted 1 line

HP-UX (the latter admittedly on ia64). Enterprise app

But that's probably just a property of ia64 and not the merit of HP, as their x86 machines have a proven track record of BIOS/SMI problems.

Quoted 4 lines

vendors are quite happy with requiring conformance to a very completely specified software/hardware/firmware stack before providing support to an app customer. I'm just trying to ensure that Linux can be part of that spec.

I understand that and I'm willing to help, but in a sane and controlled way which does me not expose to a new category of unfixable bugreports and complaints.

Thanks,

tglx

#10 lore raw
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

From: Thomas Gleixner [mailto:[email protected]]
What we can talk about is a vget_tsc_raw() interface along with a

What I have in mind and what I'm working on for quite a while is going to work on both bare metal and VMs w/o hidden slowness.

Well, if this can be done today/soon and is fast enough (say <2x the cycles of a rdtsc), I am very interested and "won't let my friends use rdtsc" :-) Anything I can do to help?

Quoted 11 lines

From: Thomas Gleixner [mailto:[email protected]]
We try to use it for performance sake, but the kernel does at least it's very best to find out when it goes bad. We then switch back to a hpet or pm-timer which is horrible performance wise but does not screw up timekeeping and everything which relies on it completely.
:
As I said, we try our very best to determine when things go awry, but there are small errors which occur either sporadic or after longer uptime which we cannot yet detect reliably. Multi-socket falls into that category, but we are working on that.

Quoted 3 lines

From: Arjan van de Ven [mailto:[email protected]]
Why do you think we do extensive and continuous validation of the tsc (and soon, continuous recalibration)

So the kernel has the ability to detect that the TSC is "OK for now", but must use some kind of polling (periodic warp test) to recognize that TSC has gone "bad". As long as TSC is good AND a sophisticated enterprise app understands that TSC might go bad at some point in the future AND if the kernel exposes "goodness" information AND the app (like the kernel) is resilient** to the possibility that there might be some period of time that obtained timestamps might be "bad" before the app polls the kernel to find out that the kernel says they are indeed "bad"... why should it be forbidden for an app to use TSC?

(** e.g. increments its own tsc_last to ensure time never goes backwards)

It seems like the only advantages the kernel has here over a reasonably intelligent app is that: 1) the kernel can run a warp test and the app can't, and 2) the kernel can estimate the frequency of the TSC and the app can't. AND, in the case of a virtual machine, the kernel has neither of these advantages anyway.

So though I now understand and agree that neither the kernel nor an app can guarantee that TSC won't unexpectedly go from "good" to "bad", I still don't understand why "TSC goodness" information shouldn't be exposed to userland, where an intelligent enterprise app can choose to use TSC when it is good (for the same reason the kernel does: "for performance sake") and choose to stop using it when it goes bad (for the same reason the kernel does: to "not screw up timekeeping").

It sounds as if you are saying that "the kernel is allowed to use a rope because if it accidentally gets the rope around its neck, it has a knife to ensure it doesn't hang itself" BUT "the app isn't allowed to use a rope because it might hang itself and we'll be damned if we loan our knife to an app because, well... because it doesn't need a knife because we said it shouldn't use the rope".

I think you can understand why this isn't a very satisfying explanation.

P.S. Thanks for taking the time to discuss this!

#11 lore raw
Quoted 5 lines · On Sun, 16 May 2010 18:31:30 -0700 (PDT) Dan Magenheimer…

It seems like the only advantages the kernel has here over a reasonably intelligent app is that: 1) the kernel can run a warp test and the app can't, and 2) the kernel can estimate the frequency of the TSC and the app can't.

and 3) the kernel gets thermal interrupts and the app does not and 4) the kernel decides which power management to use when and 5) the kernel can find out if SMI's happened, and the app cannot. and 6) the kernel can access tsc and a per cpu offset/frequency data atomically, without being scheduled to another CPU. The app cannot [well it can ask the kernel to be pinned, and that's a 99.99% thing, but still]

[snipped a bunch of twists of my argument that are not correct]

look we're not disabling ring 3 tsc. We could, but we don't. we're just telling you that WE as kernel cannot tell you, in an architectural and long term (multiple kernel versions and hardware generations) stable way, when the tsc is "usable". Because WE know it is barely if any so. We continuously add workarounds, calibrations and tweaks for this, and stop using it at runtime when something smells funny and defeats our logic.

If you want to find out yourself if the tsc is good enough for you that is one thing.... but if you want the kernel to have an official interface for it.... the kernel has to live by that commitment. We cannot put in that interface "oh and you need to implement the same workarounds, scaling and offsets as the kernel does", because that's in a huge flux, and will change from kernel version to kernel version. The only shot you could get is some vsyscall/vdso function that gives you a unit (but that is not easy given per cpu offset/frequency/etc.. but at least the kernel can try)

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#12 lore raw
Quoted 1 line · Arjan van de Ven

look we're not disabling ring 3 tsc. We could, but we don't.

Maybe we should.

#13 lore raw
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Arjan van de Ven

look we're not disabling ring 3 tsc. We could, but we don't.

Maybe we should.

We could then trap and emulate it with something sensible like clock monotonic.

/me runs :-)

#14 lore raw
In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Arjan van de Ven

look we're not disabling ring 3 tsc. We could, but we don't.

Maybe we should.

That would kill the vsyscall too. Remember it's running in ring 3.

That is in theory you could disable it on systems where the vsyscall doesn't use it, but then you would likely break huge amounts of software, unless you emulate it.

Emulation would be a possibility, but I'm not sure it would make anyone really happy. It would be certainly slow.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#15 lore raw
Quoted 11 lines · Andi Kleen
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Arjan van de Ven

look we're not disabling ring 3 tsc. We could, but we don't.

Maybe we should.

That would kill the vsyscall too. Remember it's running in ring 3.

That is in theory you could disable it on systems where the vsyscall doesn't use it, but then you would likely break huge amounts of software, unless you emulate it.

Well, software shouldn't use it, so breaking it sounds like a fine idea ;-)

Also, a slow emulation is an incentive to actually do the right thing.

#16 lore raw
Quoted 12 lines

From: Peter Zijlstra [mailto:[email protected]]
Subject: Re: [PATCH] x86: Export tsc related information in sysfs

Quoted 8 lines · Andi Kleen
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Arjan van de Ven

look we're not disabling ring 3 tsc. We could, but we don't.

Maybe we should.

That would kill the vsyscall too. Remember it's running in ring 3.

Quoted 7 lines
Quoted 2 lines

That is in theory you could disable it on systems where the vsyscall doesn't use it, but then you would likely break huge amounts of

software,

Quoted 1 line

unless you emulate it.

Well, software shouldn't use it, so breaking it sounds like a fine idea ;-)

Last fall, I discovered that EVERY program on RHEL5 (U2?) uses rdtsc because RHEL5 ld.so uses rdtsc. The uses are few and harmless but are there nonetheless. Clearly that can be fixed, but you might be surprised how big "huge amounts of software" is.

Quoted 1 line

Also, a slow emulation is an incentive to actually do the right thing.

Emulation is not particularly slow, especially compared to accessing the HPET. If the kernel deems TSC is unsafe, the ring 3 vsyscall shouldn't be using rdtsc either so the additional trap overhead might be in the noise.

As long as there is a sysfs file that can override the setting and there is a counter (accessible via sysfs) that can count the number of emulated rdtsc/rdtscp instructions (possibly optionally by pid so the "offending" userland threads can be tracked down), setting CR4.TSD whenever the kernel deems TSC is unsafe and emulating rdtsc might be a reasonable solution. Infrequent rdtsc users won't know or care, and frequent users will at least be able to learn the frequency of their "sin".

And to help Thomas/Arjan/Ingo/Andi educate users, every read or write to any of these sysfs files could also result in a printk of "Use of rdtsc is deprecated... use vsyscalls instead. See Documentation/friends_dont_let_friends_use_rdtsc." (Half ;-)

And the sysfs file could have a "strict" setting which kills any thread that uses rdtsc, so Thomas can tell future problem
reporters: "Set the rdtsc setting to strict and if you still have problems, call me back." (Other half of ;-)

#17 lore raw
In reply to Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Andi Kleen
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Arjan van de Ven

look we're not disabling ring 3 tsc. We could, but we don't.

Maybe we should.

That would kill the vsyscall too. Remember it's running in ring 3.

Quoted 8 lines
Quoted 3 lines

That is in theory you could disable it on systems where the vsyscall doesn't use it, but then you would likely break huge amounts of software, unless you emulate it.

Well, software shouldn't use it, so breaking it sounds like a fine idea ;-)

Also, a slow emulation is an incentive to actually do the right thing.

The problem is that you throw the baby (vsyscall) out with the bathwater (user rdtsc).

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#18 lore raw
Quoted 3 lines · H. Peter Anvin

The problem is that you throw the baby (vsyscall) out with the bathwater (user rdtsc).

Well, we could only flip the CR4 bit when we mark the TSC unsuitable for gtod. That should be plenty good to tag all userspace trying to use it, since more than half my machines don't use TSC for clocksource.

#19 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · H. Peter Anvin

The problem is that you throw the baby (vsyscall) out with the bathwater (user rdtsc).

Well, we could only flip the CR4 bit when we mark the TSC unsuitable for gtod. That should be plenty good to tag all userspace trying to use it, since more than half my machines don't use TSC for clocksource.

This might be an option, although it would have to be an *option*. There are restricted uses of the TSC in userspace which are still useful (mainly involving performance analysis and/or CPU-locked processes).

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#20 lore raw
Quoted 17 lines

From: H. Peter Anvin [mailto:[email protected]]
Sent: Tuesday, May 18, 2010 11:04 AM
To: Peter Zijlstra
Cc: Andi Kleen; Arjan van de Ven; Dan Magenheimer; Thomas Gleixner; Venkatesh Pallipadi; Ingo Molnar; Chris Mason; linux- [email protected]
Subject: Re: [PATCH] x86: Export tsc related information in sysfs

Quoted 3 lines · Peter Zijlstra
Quoted 2 lines · H. Peter Anvin

The problem is that you throw the baby (vsyscall) out with the

bathwater

Quoted 3 lines
Quoted 1 line

(user rdtsc).

Well, we could only flip the CR4 bit when we mark the TSC unsuitable

for

Quoted 8 lines
Quoted 1 line

gtod. That should be plenty good to tag all userspace trying to use

it,

Quoted 1 line

since more than half my machines don't use TSC for clocksource.

This might be an option, although it would have to be an *option*. There are restricted uses of the TSC in userspace which are still useful
(mainly involving performance analysis and/or CPU-locked processes).

(Though I expect tglx/arjan/andi/mingo to disagree with this proposal for similar reasons as the original one that started this thread...)

Proposal:

/sys/devices/system/tsc/native (writable by root):

0 = (default) Kernel dynamically controls TSC emulation.
When the kernel deems TSC usable as a clocksource, rdtsc will be executed directly by the CPU. When the kernel deems TSC unsafe to use, rdtsc will be trapped and emulated.
1 = TSC emulation is never enabled. Programs using rdtsc
directly are subject to the many known and sometimes rare and subtle vagaries of TSC.
2 = TSC emulation is always enabled (for debug only) 3 = Processes using TSC will be treated as if they executed
an illegal instruction. [? Can the kernel recognize use of rdtsc in a vsyscall and emulate so that, even though vsyscall is slower, all other rdtsc
in userspace are illegal?] [? Can/should this be enforced only on non-root processes?]

/sys/devices/system/tsc/system_count (writable by root):

Contains a count of all TSC emulations, system-wide. Writable to allow reset to zero.

/sys/devices/system/tsc/pid_counters (writable by root):

0 = (default) TSC counts are system-wide only
1 = TSC counted per pid (at performance penalty)
counters in /proc/PID/tsc_count

/proc/PID/tsc_count (readonly):

If /sys/devices/system/tsc_pid_counters is 1,
contains the count of rdtsc instructions emulated
for this PID.

(Note: except for the actual instruction emulation which will be faithful, rdtscp will be treated and counted as a rdtsc.)

#21 lore raw
Quoted 4 lines · Dan Magenheimer

(Though I expect tglx/arjan/andi/mingo to disagree with this proposal for similar reasons as the original one that started this thread...)

I'll add another reason to disagree: exporting these as sysfs variables is non-atomic, but these are really only useful when atomically read as a unit.

-hpa

#22 lore raw
Quoted 11 lines

From: H. Peter Anvin [mailto:[email protected]]
Subject: Re: [PATCH] x86: Export tsc related information in sysfs

Quoted 3 lines · Dan Magenheimer

(Though I expect tglx/arjan/andi/mingo to disagree with this proposal for similar reasons as the original one that started this thread...)

I'll add another reason to disagree: exporting these as sysfs variables is non-atomic, but these are really only useful when atomically read as a unit.

Which "these"? The counters? I would think the primary use of the counters is to diagnose extreme problem cases, not to differentiate whether a system or process did exactly 27 vs 28 rdtsc's, so I don't see why atomic read is at all necessary.

Or did I miss your point entirely?

#23 lore raw
Quoted 18 lines

From: Dan Magenheimer

Quoted 6 lines

From: H. Peter Anvin [mailto:[email protected]]
Subject: Re: [PATCH] x86: Export tsc related information in sysfs

Quoted 2 lines · Dan Magenheimer

(Though I expect tglx/arjan/andi/mingo to disagree with this

proposal

Quoted 1 line
Quoted 1 line

for similar reasons as the original one that started this

thread...)

Quoted 2 lines

I'll add another reason to disagree: exporting these as sysfs

variables

Quoted 1 line

is non-atomic, but these are really only useful when atomically read

as

Quoted 1 line

a unit.

Quoted 7 lines

Which "these"? The counters? I would think the primary use of the counters is to diagnose extreme problem cases, not to differentiate whether a system or process did exactly 27 vs 28 rdtsc's, so I don't see why atomic read is at all necessary.

Or did I miss your point entirely?

Oops, hit send too soon.

And the reason I expect tglx/arjan/andi/mingo to disagree is because their position is that there is NO safe use for rdtsc in userspace EVER! Whereas your position stated earlier:

Quoted 3 lines

There are restricted uses of the TSC in userspace which are still useful
(mainly involving performance analysis and/or CPU-locked processes).

says there are.

While the engineer in me agrees with tglx/arjan/andi/mingo, the realist in me agrees with you.

#24 lore raw
Quoted 7 lines · Dan Magenheimer

Oops, hit send too soon.

And the reason I expect tglx/arjan/andi/mingo to disagree is because their position is that there is NO safe use for rdtsc in userspace EVER! Whereas your position stated earlier:

Quoted 8 lines
Quoted 3 lines

There are restricted uses of the TSC in userspace which are still useful
(mainly involving performance analysis and/or CPU-locked processes).

says there are.

While the engineer in me agrees with tglx/arjan/andi/mingo, the realist in me agrees with you.

I should have added "that are not related to wall time" to the statement above.

Furthermore, vsyscalls are user space from a CPU perspective.

-hpa

#25 lore raw
Quoted 13 lines

From: H. Peter Anvin [mailto:[email protected]]
Subject: Re: [PATCH] x86: Export tsc related information in sysfs

Quoted 2 lines · Dan Magenheimer

And the reason I expect tglx/arjan/andi/mingo to disagree is because their position is that there is NO safe use for rdtsc in userspace

EVER!

Quoted 6 lines

Whereas your position stated earlier:

Quoted 3 lines

There are restricted uses of the TSC in userspace which are still useful
(mainly involving performance analysis and/or CPU-locked processes).

Quoted 7 lines
Quoted 4 lines

says there are.

While the engineer in me agrees with tglx/arjan/andi/mingo, the realist in me agrees with you.

I should have added "that are not related to wall time" to the statement above.

Yes... that still puts your opinion at odds with tglx/etc. All of the cases I am concerned with ARE performance analysis uses, not wall time uses.

Quoted 1 line

Furthermore, vsyscalls are user space from a CPU perspective.

Yes, understood, a minor semantic issue. From a kernel perspective vsyscalls are kernelspace, so IIUC this is OK with tglx/etc.

Since vsyscall shouldn't be using rdtsc when the kernel doesn't trust TSC, it doesn't matter if CR4.TSD is enabled when the kernel doesn't trust TSC.

I'm still not sure if you are in favor of optionally emulating PL3 rdtsc instructions or not? I thought my proposal was just filling out some details of your proposal and suggesting a default.

#26 lore raw
Quoted 8 lines · Dan Magenheimer

Yes, understood, a minor semantic issue. From a kernel perspective vsyscalls are kernelspace, so IIUC this is OK with tglx/etc.

Since vsyscall shouldn't be using rdtsc when the kernel doesn't trust TSC, it doesn't matter if CR4.TSD is enabled when the kernel doesn't trust TSC.

That is correct.

Quoted 4 lines

I'm still not sure if you are in favor of optionally emulating PL3 rdtsc instructions or not? I thought my proposal was just filling out some details of your proposal and suggesting a default.

I'm not in favor of emulating rdtsc instructions. I would consider letting them SIGILL (actually SIGSEGV since RDTSC #GP in userspace) when the TSC is unavailable, though.

It's not clear to me that it's possible, though, since that also affects RDTSCP.

-hpa

#27 lore raw
Quoted 10 lines
Quoted 4 lines

I'm still not sure if you are in favor of optionally emulating PL3 rdtsc instructions or not? I thought my proposal was just filling out some details of your proposal and suggesting a default.

I'm not in favor of emulating rdtsc instructions. I would consider letting them SIGILL (actually SIGSEGV since RDTSC #GP in userspace)

It's not clear to me that it's possible, though, since that also affects RDTSCP.

(All the variations are boggling so hard to discuss in a linear email thread.)

IIUC, tglx/arjan consider RDTSC and RDTSCP to be in the same category. RDTSCP simply eliminates one large class of TSC problems, but not all the possible system TSC problems that the kernel can (or can't) detect. So userspace (non-vsyscall) shouldn't use either one

Further, this one redeeming feature of RDTSCP can be useless and/or misleading in a virtual machine the way the kernel sets up TSC_AUX.

Quoted 1 line

when the TSC is unavailable, though.

Do you mean "when the processor doesn't support a TSC instruction" (very rare nowadays AFAIK) or "when the kernel determines that TSC is not safe to use as a clocksource"?

#28 lore raw
In reply to H. Peter Anvin
Quoted 3 lines

I'm not in favor of emulating rdtsc instructions. I would consider letting them SIGILL (actually SIGSEGV since RDTSC #GP in userspace) when the TSC is unavailable, though.

I doubt we could ever do that, it would likely break just too much code. Yes the code is already broken likely on some system, but there's a big difference between wrong time and crash.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#29 lore raw
Quoted 6 lines · Andi Kleen
Quoted 3 lines

I'm not in favor of emulating rdtsc instructions. I would consider letting them SIGILL (actually SIGSEGV since RDTSC #GP in userspace) when the TSC is unavailable, though.

I doubt we could ever do that, it would likely break just too much code. Yes the code is already broken likely on some system, but there's

s/some/most/

Quoted 1 line

a big difference between wrong time and crash.

Maybe start with a patch that logs all users and start sending patches to the respective projects to clean them up.

Once we get most of userspace running fine, we can switch it to generating faults.

Of course closed source stuff will have to deal with it themselves, but who cares about that anyway ;-)

#30 lore raw
In reply to Dan Magenheimer
Quoted 8 lines · Dan Magenheimer
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

From: Thomas Gleixner [mailto:[email protected]]
What we can talk about is a vget_tsc_raw() interface along with a

What I have in mind and what I'm working on for quite a while is going to work on both bare metal and VMs w/o hidden slowness.

Well, if this can be done today/soon and is fast enough (say <2x the cycles of a rdtsc), I am very interested and

Are you going to measure that with rdtsc() ? :)

Quoted 2 lines

"won't let my friends use rdtsc" :-) Anything I can do to help?

Yes, stop trying to convince me that rdtsc in apps is a good idea. :)

Quoted 10 lines

It sounds as if you are saying that "the kernel is allowed to use a rope because if it accidentally gets the rope around its neck, it has a knife to ensure it doesn't hang itself" BUT "the app isn't allowed to use a rope because it might hang itself and we'll be damned if we loan our knife to an app because, well... because it doesn't need a knife because we said it shouldn't use the rope".

I think you can understand why this isn't a very satisfying explanation.

What I understand is that you want us to give out the rope only and when things go wrong let us kernel developers deal with the bugreports about the missing knife.

Please understand that once we expose that tsc_reliable information we are responsible for its correctness. People will use it whether the enterprise entity who wants this feature has qualified that particular piece of hardware or not. And while the support of that enity refuses to help on non qualified hardware (your own words), we'll end up with the mess which was created to help that very entity.

I think you understand that I have no intention to put a ticking time bomb into the code I'm responsible for. I really have better things to do than shooting myself in the foot.

Thanks,

tglx

#31 lore raw
In reply to Dan Magenheimer
Quoted 7 lines · On Sun, 16 May 2010 09:42:40 -0700 (PDT) Dan Magenheimer…
Quoted 3 lines

From: Thomas Gleixner [mailto:[email protected]] Nah, there are systems which will have it set to 1:
Dig out your good old Pentium-I box and enjoy.

Hot stove syndrome again? Are you truly saying that there are NO single-socket multi-core systems that don't have stupid firmware (SMI and/or BIOS)?

there are no systems *where we can know* this.
Some of the stupid SMI only triggers on higher temperature situations etc. Impossible to know upfront.

Quoted 2 lines

If things are this bad, why on earth would the kernel itself EVER use TSC even as its own internal clocksource?

Why do you think we do extensive and continuous validation of the tsc (and soon, continuous recalibration)

Quoted 4 lines

But that doesn't mean the vast majority of latest generation single-socket systems can't set "tsc_reliable" to 1. Or that the kernel is responsible for detecting and/or correcting every system with buggy firmware.

sadly this also shows up on single socket systems... much more than we like.

This is why I really really hate having apps run tsc directly. A VDSO call at least gives the kernel the option to ensure correctness... even if it starts out fast and goes slow suddenly after 3 weeks when the AC in the datacenter got maintenance for an hour.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#32 lore raw
In reply to Dan Magenheimer
Quoted 6 lines

Yes, understood. But the kernel doesn't expose a "gettimeofday performance sucks" flag either. If it did (or in the case of the patch, if tsc_reliable is zero) the application could at least choose to turn off the 10000-100000 timestamps/second and log a message saying "you are running on old hardware so you get fewer features".

I don't think anyone would object to exporting such a flag if it's cleanly designed.

Getting the semantics right for that might be somewhat tricky though. How is "slow" defined?

Quoted 2 lines

A CPU-hotplugable system is a good example of a case where the kernel should expose that tsc_reliable is 0. (I've heard

That would mean that a large class of systems which are always hotplug capable (even if it's not used) would never get fast TSC time.

Wasn't the goal here to be faster?

Quoted 8 lines

anecdotally that CPU hotplug into a QPI or Hypertransport system will have some other interesting challenges, so may require some special kernel parameters anyway.) Even if tsc_reliable were only enabled if a "no-cpu_hotplug" kernel parameter is set, that is still useful. And with cores-per-socket (and even nodes-per-socket) going up seemingly every day, multi-socket systems will likely be an ever smaller percentage of new systems.

Still the people running them will expect as good performance as possible.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#33 lore raw

Hi!

Quoted 7 lines
Quoted 6 lines

Yes, understood. But the kernel doesn't expose a "gettimeofday performance sucks" flag either. If it did (or in the case of the patch, if tsc_reliable is zero) the application could at least choose to turn off the 10000-100000 timestamps/second and log a message saying "you are running on old hardware so you get fewer features".

Quoted 5 lines

I don't think anyone would object to exporting such a flag if it's cleanly designed.

Getting the semantics right for that might be somewhat tricky though. How is "slow" defined?

Well... if you want to know how fast gettimeofday is, perhaps doing

gettimeofday(); gettimeofday();

is good enough?

If not, perhaps you can export 'how many clocks is gettimeofday
expected to take' variable somewhere, but...

Pavel

Quoted 23 lines
Quoted 2 lines

A CPU-hotplugable system is a good example of a case where the kernel should expose that tsc_reliable is 0. (I've heard

That would mean that a large class of systems which are always hotplug capable (even if it's not used) would never get fast TSC time.

Wasn't the goal here to be faster?

Quoted 8 lines

anecdotally that CPU hotplug into a QPI or Hypertransport system will have some other interesting challenges, so may require some special kernel parameters anyway.) Even if tsc_reliable were only enabled if a "no-cpu_hotplug" kernel parameter is set, that is still useful. And with cores-per-socket (and even nodes-per-socket) going up seemingly every day, multi-socket systems will likely be an ever smaller percentage of new systems.

Still the people running them will expect as good performance as possible.

-Andi

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#34 lore raw
In reply to Dan Magenheimer

Dan,

Quoted 4 lines · Dan Magenheimer
Quoted 4 lines

From: Andi Kleen [mailto:[email protected]]

Quoted 2 lines

Kernel information about calibrated value of tsc_khz and tsc_stability (result of tsc warp test) are useful bits of

Quoted 8 lines
Quoted 6 lines

information

Quoted 2 lines

for any app that wants to use TSC directly. Export this read_only information in sysfs.

Is this really a good idea? It will encourage the applications to use RDTSC directly, but there are all kinds of constraints on

Indeed, that is what it is intended to do.

And you better do not.

Short story: TSC sucks in all aspects. Never ever let an application rely on it how tempting it may be.

Quoted 4 lines
Quoted 2 lines

that. Even the kernel has a hard time with them, how likely is it that applications will get all that right?

That's the point of exposing the tsc_reliable kernel data.

The tsc_reliable bit is useless outside of the kernel.

Quoted 6 lines

If the processor has Invariant TSC and the system has successfully passed Ingo's warp test and, as a result the kernel is using TSC as a clocksource, why not enable userland apps that need to obtain timestamp data tens or hundreds of thousands of times per second to also use the TSC directly?

Simply because at the time of this writing there is no single reliable TSC instance available.

Yeah, the CPU has that "P and C state invariant feature bit", but it's _not_ worth a penny.

Lemme explain some of the reasons in random order:

1) SMI:

We have proof that SMIs fiddle with the TSC to hide the fact that they happened. Yes, that's stupid, but a matter of fact. We have no reliable way to detect that shit in the kernel yet, but we are working on it. Some of those "intelligent" BIOS fkcups can be detected already and all we can do is disable TSC.

That's going to be easier once the TSC is not longer writeable and instead we get an writeable per cpu offset register. That way we can observe the SMI tricks way easier, but even then we cannot reliably undo them before some TSC user which is out of the kernels control can access it.

2) Boot offset / hotplug

Even if the TSC is completely in sync frequency wise there is no way to prevent per core/HT offsets. I'm writing this from a box where a perfectly in sync TSC (with the nice "I'm stable and reliable" bit set) is hosed by some BIOS magic which manages to offset the non boot cpu TSCs by > 300k cycles.

3) Multi socket

The "reliable" TSCs of a package are driven by the same clock, but on multi socket systems this is not the case. Each socket derives its TSC clock via a PLL from a global distributed clock at least in theory. But there is no guarantee that a board manufacturer really distributes that global base clock and instead uses a separate "global" clock on each socket.

Aside of that even if all the PLLs are driven by the same global clock there is no guarantee that the resulting PLL'ed clocks are in sync. They are not, and they never ever will be. The PLL accuracy differs in the ppm range and is also prone to temperature variations. The result over time is that the TSCs of different sockets diverge via drift in an observable way. We have bug reports about resulting user space observable time going backwards problems already.

Quoted 9 lines
Quoted 2 lines

It would be better to fix them to use the vsyscalls instead. Or if they can't use the vsyscalls for some reason today fix them.

The problem is from an app point-of-view there is no vsyscall. There are two syscalls: gettimeofday and clock_gettime. Sometimes, if it gets lucky, they turn out to be very fast and sometimes it doesn't get lucky and they are VERY slow (resulting in a performance hit of 10% or more), depending on a number of factors completely out of the control of the app and even undetectable to the app.

And they get slow for a reason: simply because the stupid hardware is not reliable whether it has some "I claim to be reliable tag" on it or not.

Quoted 7 lines

Note also that even vsyscall with TSC as the clocksource will still be significantly slower than rdtsc, especially in the common case where a timestamp is directly stored and the delta between two timestamps is later evaluated; in the vsyscall case, each timestamp is a function call and a convert to nsec but in the TSC case, each timestamp is a single instruction.

That is all understandable, but as long as we do not have some really reliable hardware I'm going to NACK any exposure of the gory details to user space simply because I have to deal with the fallout of this.

What we can talk about is a vget_tsc_raw() interface along with a vconvert_tsc_delta() interface, where vget_tsc_raw() returns you an nasty error code for everything which is not usable.

Quoted 4 lines
Quoted 2 lines

This way if anything changes again in TSC the kernel could shield the applications.

If tsc_reliable is 1, the system and the kernel are guaranteeing

Wrong. The kernel is not guaranteeing anything. See above.

Quoted 10 lines

to the app that nothing will change in the TSC. In an Invariant TSC system that has passed Ingo's warp test (to eliminate the possibility of a fixed interprocessor TSC gap due to a broken BIOS in a multi-node NUMA system), if anything changes in the clock signal that drives the TSC, the system is badly broken and far worse things -- like inter-processor cache incoherency -- may happen.

Is it finally possible to get past the horrible SMP TSC problems of the past and allow apps, under the right conditions, to be able to use rdtsc again? This patch argues "yes".

Dream on while working with the 2 machines at your desk which represent about 90% of the sane subset in the x86 universe!

We are working on solutions to get the TSC reliably usable in the case of "P/C state invariant" feature bit set, but that will be restricted to a vsyscall and you won't be able to use it realiably in the way you envision until either

- chip manufacturers finally grasp that reliable and fast access to timestamps is something important

- BIOS tinkeres finally grasp that fiddling with time is a NONO - or chip manufactures prevent them from doing so

or until we get something which myself an others proposed > 10years ago:

A simple master clock driven 1MHZ == resolution 1us counter which can be synced / preset by simple mechanisms and which was btw. developed in 1990es cluster computing environments.

Thanks,

tglx

#35 lore raw
In reply to Dan Magenheimer

Hi dan,

Quoted 6 lines · Dan Magenheimer

The problem is from an app point-of-view there is no vsyscall. There are two syscalls: gettimeofday and clock_gettime. Sometimes, if it gets lucky, they turn out to be very fast and sometimes it doesn't get lucky and they are VERY slow (resulting in a performance hit of 10% or more), depending on a number of factors completely out of the control of the app and even undetectable to the app.

What would the application do in the 10% case?

(Assuming modern kernels, I know older kernels had trouble sometimes):

That's the case when the TSC doesn't work reliably, so if it uses it anyways it won't get good time.

It seems to me you're bordering on violating Steinberg's rule of system programming here :-)

Quoted 8 lines

Note also that even vsyscall with TSC as the clocksource will still be significantly slower than rdtsc, especially in the common case where a timestamp is directly stored and the delta between two timestamps is later evaluated; in the vsyscall case, each timestamp is a function call and a convert to nsec but in the TSC case, each timestamp is a single instruction.

First the single instruction is typically quite slow. Then to really get monotonous time you need a barrier anyways.

When I originally wrote vsyscalls that overhead wasn't that big with all that compared to open coding. The only thing that could be stripped might be the unit conversion. In principle a new vsyscall could be added for that (what units do you need?)

I remember when they were converted to clocksources they got somewhat slower, but I suspect with some tuning work that could be also fixed again.

I think glibc also still does a unnecessary indirect jump (might hurt you if your CPU cannot predict that), but that could be fixed too. I think I have an old patch for that in fact, if you're still willing to use the old style vsyscalls.

Quoted 9 lines
Quoted 3 lines

This way if anything changes again in TSC the kernel could shield the applications.

If tsc_reliable is 1, the system and the kernel are guaranteeing to the app that nothing will change in the TSC. In an Invariant TSC system that has passed Ingo's warp test (to eliminate the possibility of a fixed interprocessor TSC gap due to a broken BIOS in a multi-node NUMA system), if anything changes in the clock

That only handles cases visible at boot. If the TSC breaks longer term the kernel catches it with its watchdog, but your user application won't.

Quoted 2 lines

signal that drives the TSC, the system is badly broken and far worse things -- like inter-processor cache incoherency -- may happen.

I don't think that's true. There are various large systems with non synchronized TSC and I haven't heard of any unique cache coherency problems on that.

Also often the TSC is actually synchronized, but unfortunately runs with a offset.

Quoted 4 lines

Is it finally possible to get past the horrible SMP TSC problems of the past and allow apps, under the right conditions, to be able to use rdtsc again? This patch argues "yes".

Yes but why not let them use vsyscalls?

I know vsyscalls still have some issues today, but these would be better fixed than worked around like this.

e.g.

If the idea is to use the TSC on not fully synchronized systems?

I haven't fully kept track, but at some point there was an attempt to have more POSIX clocks with loser semantics (like per thread monotonous). If you use that you'll get fast time (well not day time, but perhaps useful time) which might be good enough without hacks like this?

If the semantics are not exactly right I think more POSIX clocks could be added too.

Or if the time conversion is a problem we could add a posix_gettime_otherunit() or so (e.g. with a second vsyscall that converts units so you don't need to do it in the fast path)

A long time ago there was also the idea to export the information if gettimeofday()/clock_gettime() was fast or not. If this helps this could be probably revisited. But I'm not sure what the application should really do in this case.

32bit doesn't have a fast ring 3 gtod() today but that could be also fixed.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#36 lore raw

(note important alternate proposal suggested below, please don't get bored and stop reading yet ;-)

Quoted 5 lines

From: Thomas Gleixner [mailto:[email protected]]

Quoted 2 lines

"won't let my friends use rdtsc" :-) Anything I can do to help?

Yes, stop trying to convince me that rdtsc in apps is a good idea. :)

You've convinced ME that it is NOT a good idea in many many cases and that some of those cases are very very hard to detect. Unfortunately, convincing ME is not sufficient. The problem is that Pandora's box is already open and it is getting ever more wide open. (Each of you has implied that TSC on future systems is getting even more likely to be reliable.) Not only is rdtsc unprivileged but so is the CPUID Invariant TSC bit. And the kernel already publishes current_clocksource and puts a tempting MHz rate in its logs. While we all might hope that every system programmer will read this long thread and also be convinced never to use rdtsc, the problem is there is a high and increasing chance that any given systems programmer will write code that uses rdtsc and, in all of his/her test machines will NEVER see a problem. But, sadly, some of the customers using that app WILL see the problem BUT neither the customers nor the systems programmer may ever know what really went wrong (unless they hire Thomas to debug it :-)

So let me suggest inverting the logic of the patch and maybe it will serve all of us better. (see proposal below)

Quoted 11 lines

From: Arjan van de Ven [mailto:[email protected]]
If you want to find out yourself if the tsc is good enough for you that is one thing.... but if you want the kernel to have an official interface for it.... the kernel has to live by that commitment. We cannot put in that interface "oh and you need to implement the same workarounds, scaling and offsets as the kernel does", because that's in a huge flux, and will change from kernel version to kernel version. The only shot you could get is some vsyscall/vdso function that gives you a unit (but that is not easy given per cpu offset/frequency/etc.. but at least the kernel can try)

Quoted 7 lines

From: Thomas Gleixner [mailto:[email protected]]
Please understand that once we expose that tsc_reliable information we are responsible for its correctness. People will use it whether the enterprise entity who wants this feature has qualified that particular piece of hardware or not. And while the support of that enity refuses to help on non qualified hardware (your own words), we'll end up with the mess which was created to help that very entity.

OK, so let's invert the sense of the sysfs file and call it (for now) "tsc_detected_as_UNreliable". Then anytime the kernel detects a failed warp test (or any other suspicious condition), it changes the bit from 0 to 1 effectively saying "if you are using rdtsc against our recommendation, we told you that it might go bad and it has, so consider yourself warned that some of the timestamps you've taken since the last time you've checked this flag may be b*rked"

IMHO, addressing the issue directly and clearly documenting it (instead of trying to hide the dirty laundry in the kernel) will result in far better education of systems programmers and far fewer end user problems. Which raises another good analogy:

You are telling teenagers to abstain and I am proposing that we instead encourage them to use a condom.

You are simply not going to stop systems programmers from using rdtsc... let's at least allow them to practice "safe TSC usage" which, to extend the analogy, we all know is still not 100.0% effective but, if they are going to do "it" anyway, is still far better than completely unprotected TSC usage.

(The remainder of this response is discussion of individual points raised, so can be skipped by most readers.)

====

Quoted 8 lines

From: Arjan van de Ven [mailto:[email protected]]
and 3) the kernel gets thermal interrupts and the app does not and 4) the kernel decides which power management to use when and 5) the kernel can find out if SMI's happened, and the app cannot. and 6) the kernel can access tsc and a per cpu offset/frequency data atomically, without being scheduled to another CPU. The app cannot [well it can ask the kernel to be pinned, and that's a 99.99% thing, but still]

These are all very good reasons where the kernel might turn on a "tsc_detected_as_unreliable" bit. And CPU-hotplug_add too.

And, by the way, are any of these valid in a virtual machine?

Quoted 2 lines

From: Andi Kleen [mailto:[email protected]]
What would the application do in the 10% case?

(and in the "tsc_detected_as_unreliable" == 1 case) Well, some possibilities are:
- Disable the heavy TSC usage and log/notify that it has been disabled because the TSC is not reliable.
- Switch to gettimeofday and log/notify that, sorry, overall performance has become slower due to the fact that the TSC has become reliable
- Log a message telling the user to confirm that the hardware they are using is on the app's supported/compatibility list and that they are using the latest firmware and that their thermal logs are OK, because something might not be quite right with their system

Quoted 3 lines

From: Andi Kleen [mailto:[email protected]]
32bit doesn't have a fast ring 3 gtod() today but that could be also fixed.

That might have helped, because the enterprise app I mentioned earlier was 32-bit... but I'll bet the genie is out of the bottle now and the app has already shipped using rdtsc.

Quoted 3 lines

From: Andi Kleen [mailto:[email protected]]
It seems to me you're bordering on violating Steinberg's rule of system programming here :-)

<Embarrassed to not know this rule, Dan goes off and googles but fails to find any good matches before his TSC goes bad>

Quoted 3 lines

From: Andi Kleen [mailto:[email protected]] First the single instruction is typically quite slow. Then to really get monotonous time you need a barrier anyways.

Agreed, I've measured 30-ish and 60-ish cycles on a couple of machines.

Quoted 6 lines

From: Andi Kleen [mailto:[email protected]]
When I originally wrote vsyscalls that overhead wasn't that big with all that compared to open coding. The only thing that could be stripped might be the unit conversion. In principle a new vsyscall could be added for that (what units do you need?)

Quoted 8 lines

I remember when they were converted to clocksources they got somewhat slower, but I suspect with some tuning work that could be also fixed again.

I think glibc also still does a unnecessary indirect jump (might hurt you if your CPU cannot predict that), but that could be fixed too. I think I have an old patch for that in fact, if you're still willing to use the old style vsyscalls.

I think vsyscall is an excellent idea and I'm very much in favor of continuing to improve it and encouraging people to use it. But until it either "always" "just works" in "all" of the software/hardware environments used by a systems programmer in their development and testing and in the systems the apps get deployed on (including virtual machines) OR until it clearly provides an indicator that it is hiding dirty performance laundry, IMHO it won't convince the (admittedly undereducated) pro-TSC crowd.

Quoted 7 lines
Quoted 2 lines

signal that drives the TSC, the system is badly broken and far worse things -- like inter-processor cache incoherency -- may happen.

From: Andi Kleen [mailto:[email protected]]
I don't think that's true. There are various large systems with non synchronized TSC and I haven't heard of any unique cache coherency problems on that.

Here I was referring to clock skew/drift, not the "fixed offset" problem. I'm far from an expert in PLL's etc, but I think if the clock signal is delayed far enough to cause TSC to skew significantly, eventually some critical cache coherency protocol is eventually going to miss a beat and screw up and corrupt data.

Quoted 1 line

If the idea is to use the TSC on not fully synchronized systems?

That has never been my intent, though others might be interested in that.

Quoted 15 lines

From: Andi Kleen [mailto:[email protected]]
I haven't fully kept track, but at some point there was an attempt to have more POSIX clocks with loser semantics (like per thread monotonous). If you use that you'll get fast time (well not day time, but perhaps useful time) which might be good enough without hacks like this?

If the semantics are not exactly right I think more POSIX clocks could be added too.

Or if the time conversion is a problem we could add a posix_gettime_otherunit()
or so (e.g. with a second vsyscall that converts units so you don't need to do it in the fast path)

Quoted 8 lines

A long time ago there was also the idea to export the information if gettimeofday()/clock_gettime() was fast or not. If this helps this could
be probably revisited. But I'm not sure what the application should really do in this case.

32bit doesn't have a fast ring 3 gtod() today but that could be also fixed.

I believe (and this is strictly a personal opinion based on my view of human psychology) that adding more obscure interfaces and more obscure options is a losing battle.

Quoted 4 lines

From: Thomas Gleixner [mailto:[email protected]]
I think you understand that I have no intention to put a ticking time bomb into the code I'm responsible for. I really have better things to do than shooting myself in the foot.

As we both know, the ticking time bomb is already there. IMHO this revised proposal provides you with "plausible deniability"** "The kernel now advertises when it detects the TSC is bad or has gone bad... did your app vendor check the kernel-provided info?"

** http://en.wikipedia.org/wiki/Plausible_deniability

#37 lore raw
Quoted 4 lines · Dan Magenheimer

will read this long thread and also be convinced never to use rdtsc, the problem is there is a high and increasing chance that any given systems programmer will write code that uses rdtsc and, in all of his/her test machines will NEVER see a problem. But,

It would be probably a good idea to document all this somewhere central, agreed.

In fact I remember wanting to write such a paper once. Possibly even with some title borrowing from Swift. It seems it never happened :)

Quoted 7 lines
Quoted 3 lines

From: Andi Kleen [mailto:[email protected]]
32bit doesn't have a fast ring 3 gtod() today but that could be also fixed.

That might have helped, because the enterprise app I mentioned earlier was 32-bit... but I'll bet the genie is out of the bottle now and the app has already shipped using rdtsc.

Well the enterprise will have to live with wrong timing now and then (or perhaps totally broken timing on some system) then.

Quoted 6 lines
Quoted 3 lines

From: Andi Kleen [mailto:[email protected]]
It seems to me you're bordering on violating Steinberg's rule of system programming here :-)

<Embarrassed to not know this rule, Dan goes off and googles but fails to find any good matches before his TSC goes bad>

Sorry it's Steinbach's rule. Never test for an error condition you don't know how to handle.

Quoted 7 lines
Quoted 4 lines

From: Andi Kleen [mailto:[email protected]] First the single instruction is typically quite slow. Then to really get monotonous time you need a barrier anyways.

Agreed, I've measured 30-ish and 60-ish cycles on a couple of machines.

It can be worse.

Also for correct behaviour you need the barrier on many systems. And some other workarounds.

Quoted 4 lines

I think vsyscall is an excellent idea and I'm very much in favor of continuing to improve it and encouraging people to use it. But until it either "always" "just works" in "all"

On 64bit it should just work, although it could be made somewhat (but not dramatically) faster.

On older kernels vsyscall got a bad name because it was not always as clever as it could be to chose when to use TSC and when not, but that is long fixed. It worked in any case, just was slower[1]

[1] That is some specific TSC misbehaviours were only fixed in later kernels, but it's very unlikely that any other RDTSC user got that right either.

Quoted 10 lines
Quoted 4 lines

From: Andi Kleen [mailto:[email protected]]
I don't think that's true. There are various large systems with non synchronized TSC and I haven't heard of any unique cache coherency problems on that.

Here I was referring to clock skew/drift, not the "fixed offset" problem. I'm far from an expert in PLL's etc, but I think if the clock signal is delayed far enough to cause TSC to skew significantly, eventually some critical cache coherency protocol is eventually going to miss a beat and screw up and corrupt data.

Generally small systems run on the same clock (and any drift you see has other reasons), but large systems built out of multiple boards run on different clocks and the interconnect
does appropiate timing between the different domains for itself.

The interconnects also have checksums and other error checking and recovery mechanisms and tend to do something appropiate when there is a transmission problem. It does not usually lead to data corruption.

Quoted 3 lines

I believe (and this is strictly a personal opinion based on my view of human psychology) that adding more obscure interfaces and more obscure options is a losing battle.

It sounds like you're arguing against your own patch here.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#38 lore raw
In reply to Dan Magenheimer
Quoted 10 lines · Dan Magenheimer

OK, so let's invert the sense of the sysfs file and call it (for now) "tsc_detected_as_UNreliable". Then anytime the kernel detects a failed warp test (or any other suspicious condition), it changes the bit from 0 to 1 effectively saying "if you are using rdtsc against our recommendation, we told you that it might go bad and it has, so consider yourself warned that some of the timestamps you've taken since the last time you've checked this flag may be b*rked"

Quoted 7 lines

IMHO, addressing the issue directly and clearly documenting it (instead of trying to hide the dirty laundry in the kernel) will result in far better education of systems programmers and far fewer end user problems. Which raises another good analogy:

You are telling teenagers to abstain and I am proposing that we instead encourage them to use a condom.

Wrong. A vsyscall _is_ the protection which you want them to pull over the rdtsc.

You are basically telling them: Go ahead, but keep in mind to look for that well hidden tag behind the left earlobe which might change suddenly from "no disease" to "infectous".

Thanks,

tglx

#39 lore raw
Quoted 21 lines

From: Thomas Gleixner [mailto:[email protected]]

Quoted 10 lines · Dan Magenheimer

OK, so let's invert the sense of the sysfs file and call it (for now) "tsc_detected_as_UNreliable". Then anytime the kernel detects a failed warp test (or any other suspicious condition), it changes the bit from 0 to 1 effectively saying "if you are using rdtsc against our recommendation, we told you that it might go bad and it has, so consider yourself warned that some of the timestamps you've taken since the last time you've checked this flag may be b*rked"

Quoted 7 lines

IMHO, addressing the issue directly and clearly documenting it (instead of trying to hide the dirty laundry in the kernel) will result in far better education of systems programmers and far fewer end user problems. Which raises another good analogy:

You are telling teenagers to abstain and I am proposing that we instead encourage them to use a condom.

Quoted 6 lines

Wrong. A vsyscall _is_ the protection which you want them to pull over the rdtsc.

You are basically telling them: Go ahead, but keep in mind to look for that well hidden tag behind the left earlobe which might change suddenly from "no disease" to "infectous".

OK, well ignoring the metaphor, it's clear we disagree on a point that neither one of us can prove: You think your decision to avoid sharing kernel information will stop system programmers from using rdtsc, and I think some are going to use rdtsc anyway and blame Linux when something eventually and silently breaks.

Given that I'm not going to win this argument even by pointing to examples, let's move forward with your solution. Can you say more about your vget_tsc_raw() directions? Or at least describe the API if there is anything special? I'm getting beaten on for an answer *today*, they say (and I quote from an email today) they "have been using vsyscalls for a while and still have a performance headache", and rdtsc looks awfully tempting, even if it is not 100% perfect. So what can I give them as an alternative?

(And, yes, I understand this is not your problem, but I do worry that it is perceived as *Linux's* problem. And as we all have painfully learned in our careers, it's almost impossible to beat back a perception by presenting a set of byzantine facts that only a handful of people in the world truly understand.)

Thanks!
Dan

#40 lore raw
Quoted 7 lines · Dan Magenheimer

OK, well ignoring the metaphor, it's clear we disagree on a point that neither one of us can
prove: You think your decision to avoid sharing kernel information will stop system programmers from using rdtsc, and I think some are going to use rdtsc anyway and blame Linux when something eventually and silently breaks.

Applications can do various unreliable things, the kernel cannot do anything about that.

The point is for the kernel to not be complicit in practices that are technically not reliable.

So the kernel wont 'signal' that something is safe to use if it is not safe to use.

One suggestion in this thread makes sense i think: to signal via sysfs that gettimeofday is slow.

Plus lets hope that we really can figure out a fast, TSC based gettimeofday implementation. If that is possible then user-space will get a fast gettimeofday right out of box.

Thanks,

Ingo

#41 lore raw
In reply to Dan Magenheimer
Quoted 2 lines · Dan Magenheimer

Given that I'm not going to win this argument even by pointing to examples, [...]

You could win the argument by coming up with a patch that changes gettimeofday to make use of the TSC in a reliable manner.

I really mean it - and it might be possible - but we have not found it yet.

Thanks,

Ingo

#42 lore raw
In reply to Venkatesh Pallipadi

Hello,

Quoted 74 lines · Venkatesh Pallipadi

From: Dan Magenheimer <[email protected]>

Kernel information about calibrated value of tsc_khz and tsc_stability (result of tsc warp test) are useful bits of information for any app that wants to use TSC directly. Export this read_only information in sysfs.

---
 arch/x86/kernel/tsc.c |   76 +++++++++++++++++++++++++++++++++++++++++++++++++

 1 files changed, 76 insertions(+), 0 deletions(-)
arch/x86/kernel/tsc.c +46 -0
index 9faf91a..24dd484 100644
@@ -10,6 +10,7 @@
  #include <linux/clocksource.h>
  #include <linux/percpu.h>
  #include <linux/timex.h>
+#include <linux/sysdev.h>
 
  #include <asm/hpet.h>
  #include <asm/timer.h>
@@ -857,6 +858,81 @@ static void __init init_tsc_clocksource(void)
        clocksource_register(&clocksource_tsc);
  }
 
+#ifdef CONFIG_SYSFS
+/*
+ * Export TSC related info to user land. This reflects kernel usage of TSC
+ * as hints to userspace users of TSC. The read_only info provided here:
+ * - tsc_stable: 1 implies system has TSC that always counts at a constant
+ *   rate, sync across CPUs and has passed the kernel warp test.
+ * - tsc_khz: TSC frequency in khz.
+ * - tsc_mult and tsc_shift: multiplier and shift to optimally convert
+ *   TSC delta to ns; ns = ((u64) delta * mult) >> shift
+ */
+
+#define define_show_var_function(_name, _var) \
+static ssize_t show_##_name( \
+       struct sys_device *dev, struct sysdev_attribute *attr, char *buf) \
+{ \
+       return sprintf(buf, "%u\n", (unsigned int) _var);\
+}
+
+define_show_var_function(tsc_stable, !tsc_unstable);
+define_show_var_function(tsc_khz, tsc_khz);
+define_show_var_function(tsc_mult, clocksource_tsc.mult);
+define_show_var_function(tsc_shift, clocksource_tsc.shift);
+
+static SYSDEV_ATTR(tsc_stable, 0444, show_tsc_stable, NULL);
+static SYSDEV_ATTR(tsc_khz, 0444, show_tsc_khz, NULL);
+static SYSDEV_ATTR(tsc_mult, 0444, show_tsc_mult, NULL);
+static SYSDEV_ATTR(tsc_shift, 0444, show_tsc_shift, NULL);
+
+static struct sysdev_attribute *tsc_attrs[] = {
+       &attr_tsc_stable,
+       &attr_tsc_khz,
+       &attr_tsc_mult,
+       &attr_tsc_shift,
+};
+
+static struct sysdev_class tsc_sysclass = {
+       .name = "tsc",
+};
+
+static struct sys_device device_tsc = {
+       .id = 0,
+       .cls = &tsc_sysclass,
+};
+
+static int __init init_tsc_sysfs(void)

arch/x86/kernel/tsc.c static void __init init_tsc_clocksource(void) · line 915

Quoted 10 lines
arch/x86/kernel/tsc.c +10 -0
+{
+       int err, i = 0;
+
+       err = sysdev_class_register(&tsc_sysclass);
+       if (err)
+               return err;
+
+       err = sysdev_register(&device_tsc);
+       if (err)
+               goto fail;

fail will call sysdev_unregister(&device_tsc) which is not appropriate, please fix this goto.

Thanks,
--
Jaswinder Singh.

Quoted 32 lines
arch/x86/kernel/tsc.c +20 -1
+
+       for (i = 0; i < ARRAY_SIZE(tsc_attrs); i++) {
+               err = sysdev_create_file(&device_tsc, tsc_attrs[i]);
+               if (err)
+                       goto fail;
+       }
+
+       return 0;
+
+fail:
+       while (--i >= 0)
+               sysdev_remove_file(&device_tsc, tsc_attrs[i]);
+
+       sysdev_unregister(&device_tsc);
+       sysdev_class_unregister(&tsc_sysclass);
+       return err;
+}
+device_initcall(init_tsc_sysfs);
+#endif
+
  #ifdef CONFIG_X86_64
  /*
  * calibrate_cpu is used on systems with fixed rate TSCs to determine
--

1.7.0.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

#43 lore raw
Quoted 17 lines · Jaswinder Singh Rajput

Hello,

arch/x86/kernel/tsc.c static void __init init_tsc_clocksource(void) · line 915

Quoted 12 lines
arch/x86/kernel/tsc.c +12 -0
+
+static int __init init_tsc_sysfs(void)
+{
+       int err, i = 0;
+
+       err = sysdev_class_register(&tsc_sysclass);
+       if (err)
+               return err;
+
+       err = sysdev_register(&device_tsc);
+       if (err)
+               goto fail;

fail will call sysdev_unregister(&device_tsc) which is not appropriate, please fix this goto.

Thanks for catching this. Will fix this in the patch refresh.

Thanks,
Venki