tsc-disable_B9

Marcelo,
This patch (which applies on top of my tsc-disable_B9 patch as well as
James Cleverdon's summit patch), is a performance improvement for multi-CEC IBM x440 systems which suffer from drifting TSCs. Rather then forcing do_gettimeofday to call do_slow_gettimeoffset and access the PIT (as my tsc-disable patch does), passing "cyclone" as a boot option will make do_gettimeofday use a 100Mhz performance counter found in the Summit chipset.

Quoted 8 lines · john stultz

Marcelo,
This patch (which applies on top of my tsc-disable_B9 patch as well as
James Cleverdon's summit patch), is a performance improvement for multi-CEC IBM x440 systems which suffer from drifting TSCs. Rather then forcing do_gettimeofday to call do_slow_gettimeoffset and access the PIT (as my tsc-disable patch does), passing "cyclone" as a boot option will make do_gettimeofday use a 100Mhz performance counter found in the Summit chipset.

Why not probe for the summit chipset ?

already seen on another cpu. This patch allows people compiling w/ 'Multi-node NUMA support' to pass "notsc" or "bad-tsc" as boot parameters. "notsc" disables rdtsc calls, and forces the kernel to use the PIT for gettimeofday calucluations (as normally expected w/ i386 compiled kernels). While "bad-tsc" forces the kernel to use the PIT for gettimeofday, but does not disable TSC access.

Quoted 6 lines · john stultz

already seen on another cpu. This patch allows people compiling w/ 'Multi-node NUMA support' to pass "notsc" or "bad-tsc" as boot parameters. "notsc" disables rdtsc calls, and forces the kernel to use the PIT for gettimeofday calucluations (as normally expected w/ i386 compiled kernels). While "bad-tsc" forces the kernel to use the PIT for gettimeofday, but does not disable TSC access.

I've done a version of this for -ac which uses the NUMA autodetect, and also provides the hook so I can disable tsc on split on x86 smp with variable multipliers some time. The only comment I really have is please

use "badtsc" not "bad-tsc" - to match "notsc"

This is how I did it - barring the code that is -ac specific to automatically flip the mode when NUMA hw is detected

diff -u --exclude-from /usr/src/exclude --new-file --recursive linux.20pre1/include/asm-i386/processor.h linux.20pre1-ac2/include/asm-i386/processor.h

include/asm-i386/processor.h +1 -1

@@ -96,7 +96,7 @@
 
 extern void identify_cpu(struct cpuinfo_x86 *);
 extern void print_cpu_info(struct cpuinfo_x86 *);
-extern void dodgy_tsc(void);
+extern int dodgy_tsc(void);
 
 /*
  * EFLAGS bits

diff -u --exclude-from /usr/src/exclude --new-file --recursive linux.20pre1/arch/i386/kernel/setup.c linux.20pre1-ac2/arch/i386/kernel/setup.c

arch/i386/kernel/setup.c +9 -2

@@ -1193,19 +1126,51 @@
 }
 __setup("cachesize=", cachesize_setup);
 
-
 #ifndef CONFIG_X86_TSC
+
 static int tsc_disable __initdata = 0;
 
-static int __init tsc_setup(char *str)
+void disable_tsc(void)
+{
+	if(tsc_disable == 0)
+	{
+		printk(KERN_INFO "Disabling use of TSC for time counting.\n");
+		tsc_disable = 1;
+	}
+}

arch/i386/kernel/setup.c +5 -1

+/* Allow TSC use but don't use it for gettimeofday  */
+
+static int __init badtsc_setup(char *str)
 {
 	tsc_disable = 1;
 	return 1;
 }
 
-__setup("notsc", tsc_setup);
+__setup("badtsc", badtsc_setup);
+

Not sure I followed that, do you mean per-cpu TSC management for gettimeofday?

Quoted 2 lines · john stultz

Not sure I followed that, do you mean per-cpu TSC management for gettimeofday?

We have some x86 setups where people plug say a 300MHhz and a 450MHz celeron into the same board. This works because they are same FSB, different multiplier (works and intel certify being two different things)

Needless to say tsc does not work well on such boxes. Thats why I don't trust the tsc at all in such cases. Since you'll have the nice cyclone timer for the Summit it seems best not to trust it, and on the summit to use the cyclone for udelay as well ?

I agree dodgy_tsc needs to change name. Perhaps we actually want

	int tsc = select_tsc();

	switch(tsc)
	{
		case TSC_CYCLONE:
		case TSC_PROCESSOR:
		case TSC_NONE:
		..
	}

Quoted 8 lines · Alan Cox

Quoted 2 lines · john stultz

Not sure I followed that, do you mean per-cpu TSC management for gettimeofday?

We have some x86 setups where people plug say a 300MHhz and a 450MHz celeron into the same board. This works because they are same FSB, different multiplier (works and intel certify being two different things)

Oh yes, with the old NUMAQ hardware here, one can mix nodes of different speed cpus. Once I get a chance, I'm going to begin working on this issue for 2.5. My plan right now is to keep per-cpu last_tsc_low and fast_gettimeoffset_quotient values, then round robin the timer interrupt.

Quoted 8 lines

Needless to say tsc does not work well on such boxes. Thats why I don't trust the tsc at all in such cases. Since you'll have the nice cyclone timer for the Summit it seems best not to trust it, and on the summit to use the cyclone for udelay as well ?

I agree dodgy_tsc needs to change name. Perhaps we actually want

Quoted 9 lines

      int tsc = select_tsc();

      switch(tsc)
      {
              case TSC_CYCLONE:
              case TSC_PROCESSOR:
              case TSC_NONE:
              ..
      }

Sounds good. I'll re-work my patch and resubmit.

thanks!

arch/i386/kernel/time.c +0 -2

-john
 
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

An interesting approach, however, could you take a look at the high-res-timers patch (see signature). In that code (in

the TSC version), we use the TSC to update jiffies and _sub_jiffie (which is TSC counts into the next jiffie). We also want to be able to "grab" a new TSC and figure the time quickly, without updating either jiffies or _sub_jiffie. Your approach would, I think, mean that both jiffies and _sub_jiffie would be per cpu values, not impossible, but, well, hard.

On the other hand, the high-res-timers patch also allows one to use the ACPI pm timer, and ignore TSC completely :)

(also, doesn't MS have some sort of high res performance counter standard they're pushing for in hardware?), ...

Oh yes, with the old NUMAQ hardware here, one can mix nodes of different speed cpus. Once I get a chance, I'm going to begin working on this issue for 2.5. My plan right now is to keep per-cpu last_tsc_low and fast_gettimeoffset_quotient values, then round robin the timer interrupt.

Quoted 11 lines · Alan Cox

Quoted 6 lines · john stultz

already seen on another cpu. This patch allows people compiling w/ 'Multi-node NUMA support' to pass "notsc" or "bad-tsc" as boot parameters. "notsc" disables rdtsc calls, and forces the kernel to use the PIT for gettimeofday calucluations (as normally expected w/ i386 compiled kernels). While "bad-tsc" forces the kernel to use the PIT for gettimeofday, but does not disable TSC access.

I've done a version of this for -ac which uses the NUMA autodetect, and also provides the hook so I can disable tsc on split on x86 smp with variable multipliers some time. The only comment I really have is please

Not sure I followed that, do you mean per-cpu TSC management for gettimeofday?

Quoted 1 line

use "badtsc" not "bad-tsc" - to match "notsc"

Sounds good. Changed in my tree.

Quoted 2 lines

This is how I did it - barring the code that is -ac specific to automatically flip the mode when NUMA hw is detected

Interesting. See my comments below. Auto-detection has been on my list for the last week, so I'm eagerly awaiting a peek at the next ac release :)

sorry but I don't see the point of badtsc only in kernel.

If the TSC is bad that will be in particular bad from userspace where there's no hope to know what CPU you're running on.

How do you detect the NUMA hw? That would be a nice addition so the numa-Q users won't be required to add notsc to the append lilo line.

Quoted 4 lines · Andrea Arcangeli

sorry but I don't see the point of badtsc only in kernel.

If the TSC is bad that will be in particular bad from userspace where there's no hope to know what CPU you're running on.

You can still do meaningful measurements for things like profiling because the cpu hop is statistically uninteresting.

Quoted 6 lines · Alan Cox

Quoted 4 lines · Andrea Arcangeli

sorry but I don't see the point of badtsc only in kernel.

If the TSC is bad that will be in particular bad from userspace where there's no hope to know what CPU you're running on.

Quoted 2 lines

You can still do meaningful measurements for things like profiling because the cpu hop is statistically uninteresting.

There are kernel extensions around that handle process migration across CPUs while providing virtualised per-process TSC counts, and for which the TSCs do NOT need to be in perfect sync.

Disabling user-space RDTSC just because the TSCs aren't in sync is stupid.

However here the point is that the TSC was left _enabled_ (not disabled and emulated as you are advocating) despite it was not in sync. That cannot make any sense, except if you use the tsc as a random number generator.

Quoted 4 lines · Andrea Arcangeli

However here the point is that the TSC was left _enabled_ (not disabled and emulated as you are advocating) despite it was not in sync. That cannot make any sense, except if you use the tsc as a random number generator.

Totally untrue. When you are doing profiling the data is very useful because CPU switches are trivial to filter and statistically rare

But silenty breaking apps and not allowing in any way to apps to learn if the tsc is returning random or if it's returning something significant (I understand that's the way you did it in -ac) is a no-way by default IMHO.

Quoted 4 lines · Andrea Arcangeli

But silenty breaking apps and not allowing in any way to apps to learn if the tsc is returning random or if it's returning something significant (I understand that's the way you did it in -ac) is a no-way by default IMHO.

All such apps and libraries are already broken have been silently broken since about 1999 and will continue to be broken. Thats been true since speedstep cpus appeared if not before.

certainly fair enough argument in theory, but in practice you're not going to risk running those apps in a laptop or in general with any power management that will decrease the frequency of the cpu anytime.

Furthmore the speedstep right now today can crash any laptop that boots at reduced mhz and that switches to higher mhz at runtime, that change

of the tsc frequency simply make udelay run faster, and it'll break drivers easily. I suspect there's even an unfixable race condition in the speedstep hardware since it's not the kernel asking for the change

significant info via the tsc to userspace, and there's no way to know that your app isn't breaking because of numa, unless you disable the tsc to userspace.

Quoted 3 lines

certainly fair enough argument in theory, but in practice you're not going to risk running those apps in a laptop or in general with any power management that will decrease the frequency of the cpu anytime.

Any PIII with speedstep, any Athlon and PIV.

Quoted 2 lines

Furthmore the speedstep right now today can crash any laptop that boots at reduced mhz and that switches to higher mhz at runtime, that change

Actually the reduced loops in the kernel seem to work fine

Quoted 3 lines

of the tsc frequency simply make udelay run faster, and it'll break drivers easily. I suspect there's even an unfixable race condition in the speedstep hardware since it's not the kernel asking for the change

Fixed in the -ac tree for the non APM triggered case because we use cpufreq code

Quoted 4 lines

significant info via the tsc to userspace, and there's no way to know that your app isn't breaking because of numa, unless you disable the tsc to userspace.

And you can test that with notsc. Oh and you might also want the code that makes notsc on a tsc only kernel print a warning btw. badtsc lets you say "I have a brain cell" notsc lets you select "clueless app checking mode"

Quoted 2 lines

Fixed in the -ac tree for the non APM triggered case because we use cpufreq code

if the reduced loops is supposed to work fine what was there left to fix?

Quoted 4 lines

And you can test that with notsc. Oh and you might also want the code that makes notsc on a tsc only kernel print a warning btw. badtsc lets you say "I have a brain cell" notsc lets you select "clueless app checking mode"

what's wrong with a sysctl to specify you have a brain cell? I'm not advocating not to let you specify you have a brain cell, I'm only don't see why we should assume it when we very know apps will break silenty, and that it will be not noticeable except with subtle breakage after some runtime in -ac.

Quoted 8 lines

Quoted 4 lines

But silenty breaking apps and not allowing in any way to apps to learn if the tsc is returning random or if it's returning something significant (I understand that's the way you did it in -ac) is a no-way by default IMHO.

All such apps and libraries are already broken have been silently broken since about 1999 and will continue to be broken. Thats been true since speedstep cpus appeared if not before.

certainly fair enough argument in theory, but in practice you're not going to risk running those apps in a laptop or in general with any

And following your argument that these apps have been silenty broken since 1999, if there's no broken app out there, nobody will ever get the instruction fault. If there's any app broken out there we probably like

Quoted 3 lines

And following your argument that these apps have been silenty broken since 1999, if there's no broken app out there, nobody will ever get the instruction fault. If there's any app broken out there we probably like

No. rdtsc is still usefull if you are clever and statistically filter out. Also rdtsc provides you number of cycles, so if you want to know how many cycles mov %eax,%ebx takes, you can do that even on speedstep. Anything that correlates rdtsc to real time is broken, however.

Quoted 8 lines

Quoted 3 lines

And following your argument that these apps have been silenty broken since 1999, if there's no broken app out there, nobody will ever get the instruction fault. If there's any app broken out there we probably like

No. rdtsc is still usefull if you are clever and statistically filter out. Also rdtsc provides you number of cycles, so if you want to know how many cycles mov %eax,%ebx takes, you can do that even on speedstep. Anything that correlates rdtsc to real time is broken, however.

It's not correlating it to real time that's the problem. It's getting resceduled inbetween calls that hurts. Take your example.

rdtsc
mov %eax,%ebx
<- get rescheduled here
rdtsc

Broken. May even take negative "time".

It's not correlating it to real time that's the problem. It's getting resceduled inbetween calls that hurts. Take your example.

rdtsc
mov %eax,%ebx
<- get rescheduled here
rdtsc

Broken. May even take negative "time".

Quoted 8 lines

It's not correlating it to real time that's the problem. It's getting resceduled inbetween calls that hurts. Take your example.

rdtsc
mov %eax,%ebx
<- get rescheduled here
rdtsc

Quoted 1 line

Broken. May even take negative "time".

Statistically irrelevant. When you have 100,000 samples all the pre-emption ones drop into the dud sample filter with IRQ disturbance and so on.

Quoted 8 lines

Quoted 3 lines

And following your argument that these apps have been silenty broken since 1999, if there's no broken app out there, nobody will ever get the instruction fault. If there's any app broken out there we probably like

No. rdtsc is still usefull if you are clever and statistically filter out. Also rdtsc provides you number of cycles, so if you want to know how many cycles mov %eax,%ebx takes, you can do that even on speedstep. Anything that correlates rdtsc to real time is broken, however.

It's not correlating it to real time that's the problem. It's getting resceduled inbetween calls that hurts. Take your example.

rdtsc
mov %eax,%ebx
<- get rescheduled here
rdtsc

Broken. May even take negative "time".

M.

Quoted 17 lines

Quoted 8 lines

Quoted 3 lines

And following your argument that these apps have been silenty broken since 1999, if there's no broken app out there, nobody will ever get the instruction fault. If there's any app broken out there we probably like

No. rdtsc is still usefull if you are clever and statistically filter out. Also rdtsc provides you number of cycles, so if you want to know how many cycles mov %eax,%ebx takes, you can do that even on speedstep. Anything that correlates rdtsc to real time is broken, however.

It's not correlating it to real time that's the problem. It's getting resceduled inbetween calls that hurts. Take your example.

rdtsc
mov %eax,%ebx
<- get rescheduled here
rdtsc

Quoted 3 lines

Broken. May even take negative "time".

M.

The CPU counters are synchronized on SMP machines. How can you ever get negative time? Even GHz machines take several months to wrap the count.

The CPU counters are synchronized on SMP machines. How can you ever get negative time? Even GHz machines take several months to wrap the count.

Quoted 8 lines

Quoted 3 lines

And following your argument that these apps have been silenty broken since 1999, if there's no broken app out there, nobody will ever get the instruction fault. If there's any app broken out there we probably like

No. rdtsc is still usefull if you are clever and statistically filter out. Also rdtsc provides you number of cycles, so if you want to know how many cycles mov %eax,%ebx takes, you can do that even on speedstep. Anything that correlates rdtsc to real time is broken, however.

It's not correlating it to real time that's the problem. It's getting resceduled inbetween calls that hurts. Take your example.

rdtsc
mov %eax,%ebx
<- get rescheduled here
rdtsc

Broken. May even take negative "time".